Skip to main content
Version: 3.16

ScalarDB Cluster Configurations

This document describes the configurations for ScalarDB Cluster. ScalarDB Cluster consists of multiple cluster nodes, each of which needs to be configured. The configurations need to be specified in the properties file.

Cluster configurations​

This section describes the configurations for ScalarDB Cluster.

General configurations​

The following general configurations are available for ScalarDB Cluster.

Transaction management configurations​

NameDescriptionDefault.
scalar.db.transaction_managerTransaction manager of ScalarDB. Specify consensus-commit to use Consensus Commit or single-crud-operation to run non-transactional storage operations. Note that the configurations under the scalar.db.consensus_commit prefix are ignored if you use single-crud-operation.consensus-commit
scalar.db.consensus_commit.isolation_levelIsolation level used for Consensus Commit. Either SNAPSHOT, SERIALIZABLE, or READ_COMMITTED can be specified.SNAPSHOT
scalar.db.consensus_commit.coordinator.namespaceNamespace name of Coordinator tables used for Consensus Commit.coordinator

Node configurations​

NameDescription γ€€Default
scalar.db.cluster.membership.typeMembership type. Currently, only KUBERNETES can be specified.KUBERNETES
scalar.db.cluster.membership.kubernetes.endpoint.namespace_nameThis configuration is for the KUBERNETES membership type. Namespace name for the endpoint resource.default
scalar.db.cluster.membership.kubernetes.endpoint.nameThis configuration is for the KUBERNETES membership type. Name of the endpoint resource to get the membership info.
scalar.db.cluster.node.decommissioning_duration_secsDecommissioning duration in seconds.30
scalar.db.cluster.node.grpc.max_inbound_message_sizeMaximum message size allowed to be received.The gRPC default value
scalar.db.cluster.node.grpc.max_inbound_metadata_sizeMaximum size of metadata allowed to be received.The gRPC default value
scalar.db.cluster.node.portPort number of the ScalarDB Cluster node.60053
scalar.db.cluster.node.prometheus_exporter_portPort number of the Prometheus exporter.9080
scalar.db.cluster.grpc.deadline_duration_millisDeadline duration for gRPC in milliseconds.60000 (60 seconds)
scalar.db.cluster.node.standalone_mode.enabledWhether standalone mode is enabled. Note that if standalone mode is enabled, the membership configurations (scalar.db.cluster.membership.*) will be ignored.false
scalar.db.transaction.enabledWhether the transaction feature is enabled. For example, if you use only the embedding feature, you can set this property to false.true
scalar.db.cluster.node.scanner_management.expiration_time_millisScalarDB Cluster nodes maintain in-progress scanners. This process expires scanners that have been idle for an extended period to prevent resource leaks. This configuration specifies the expiration time of this scanner management feature in milliseconds.60000 (60 seconds)

The following performance-related configurations are available for the Consensus Commit transaction manager:

NameDescriptionDefault
scalar.db.consensus_commit.parallel_executor_countNumber of executors (threads) for parallel execution. This number refers to the total number of threads across transactions in a ScalarDB Cluster node or a ScalarDB process.128
scalar.db.consensus_commit.parallel_preparation.enabledWhether or not the preparation phase is executed in parallel.true
scalar.db.consensus_commit.parallel_validation.enabledWhether or not the validation phase (in EXTRA_READ) is executed in parallel.The value of scalar.db.consensus_commit.parallel_commit.enabled
scalar.db.consensus_commit.parallel_commit.enabledWhether or not the commit phase is executed in parallel.true
scalar.db.consensus_commit.parallel_rollback.enabledWhether or not the rollback phase is executed in parallel.The value of scalar.db.consensus_commit.parallel_commit.enabled
scalar.db.consensus_commit.async_commit.enabledWhether or not the commit phase is executed asynchronously.false
scalar.db.consensus_commit.async_rollback.enabledWhether or not the rollback phase is executed asynchronously.The value of scalar.db.consensus_commit.async_commit.enabled
scalar.db.consensus_commit.parallel_implicit_pre_read.enabledWhether or not implicit pre-read is executed in parallel.true
scalar.db.consensus_commit.one_phase_commit.enabledWhether or not the one-phase commit optimization is enabled.false
scalar.db.consensus_commit.coordinator.write_omission_on_read_only.enabledWhether or not the Coordinator write omission optimization is enabled for read-only transactions. This optimization is useful for read-only transactions that do not modify any data, as it avoids unnecessary writes to the Coordinator tables.true
scalar.db.consensus_commit.coordinator.group_commit.enabledWhether or not committing the transaction state is executed in batch mode. This feature can't be used with a two-phase commit interface.false
scalar.db.consensus_commit.coordinator.group_commit.slot_capacityMaximum number of slots in a group for the group commit feature. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.120
scalar.db.consensus_commit.coordinator.group_commit.group_size_fix_timeout_millisTimeout to fix the size of slots in a group. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.140
scalar.db.consensus_commit.coordinator.group_commit.delayed_slot_move_timeout_millisTimeout to move delayed slots from a group to another isolated group to prevent the original group from being affected by delayed transactions. A large value improves the efficiency of group commit, but may also increase the latency and the likelihood of transaction conflicts.11200
scalar.db.consensus_commit.coordinator.group_commit.old_group_abort_timeout_millisTimeout to abort an old ongoing group. A small value reduces resource consumption through aggressive aborts, but may also increase the likelihood of unnecessary aborts for long-running transactions.60000
scalar.db.consensus_commit.coordinator.group_commit.timeout_check_interval_millisInterval for checking the group commit–related timeouts.20
scalar.db.consensus_commit.coordinator.group_commit.metrics_monitor_log_enabledWhether or not the metrics of the group commit are logged periodically.false

ScalarDB has a storage (database) abstraction layer that supports multiple storage implementations. You can specify the storage implementation by using the scalar.db.storage property.

Select a database to see the configurations available for each storage.

The following configurations are available for JDBC databases:

NameDescriptionDefault
scalar.db.storagejdbc must be specified.-
scalar.db.contact_pointsJDBC connection URL.
scalar.db.usernameUsername to access the database.
scalar.db.passwordPassword to access the database.
scalar.db.jdbc.connection_pool.min_idleMinimum number of idle connections in the connection pool.20
scalar.db.jdbc.connection_pool.max_idleMaximum number of connections that can remain idle in the connection pool.50
scalar.db.jdbc.connection_pool.max_totalMaximum total number of idle and borrowed connections that can be active at the same time for the connection pool. Use a negative value for no limit.100
scalar.db.jdbc.prepared_statements_pool.enabledSetting this property to true enables prepared-statement pooling.false
scalar.db.jdbc.prepared_statements_pool.max_openMaximum number of open statements that can be allocated from the statement pool at the same time. Use a negative value for no limit.-1
scalar.db.jdbc.isolation_levelIsolation level for JDBC. READ_UNCOMMITTED, READ_COMMITTED, REPEATABLE_READ, or SERIALIZABLE can be specified.Underlying-database specific
scalar.db.jdbc.table_metadata.connection_pool.min_idleMinimum number of idle connections in the connection pool for the table metadata.5
scalar.db.jdbc.table_metadata.connection_pool.max_idleMaximum number of connections that can remain idle in the connection pool for the table metadata.10
scalar.db.jdbc.table_metadata.connection_pool.max_totalMaximum total number of idle and borrowed connections that can be active at the same time for the connection pool for the table metadata. Use a negative value for no limit.25
scalar.db.jdbc.admin.connection_pool.min_idleMinimum number of idle connections in the connection pool for admin.5
scalar.db.jdbc.admin.connection_pool.max_idleMaximum number of connections that can remain idle in the connection pool for admin.10
scalar.db.jdbc.admin.connection_pool.max_totalMaximum total number of idle and borrowed connections that can be active at the same time for the connection pool for admin. Use a negative value for no limit.25
scalar.db.jdbc.db2.variable_key_column_sizeColumn size for TEXT and BLOB columns in IBM Db2 when they are used as a primary key or secondary key. Minimum 64 bytes.128
scalar.db.jdbc.db2.time_column.default_date_componentValue of the date component used for storing TIME data in IBM Db2. Since the IBM Db2 TIMESTAMP type is used to store ScalarDB TIME type data because it provides fractional-second precision, ScalarDB stores TIME data with the same date component value for ease of comparison and sorting.1970-01-01
note

If you're using SQLite3 as a JDBC database, you must set scalar.db.contact_points as follows:

scalar.db.contact_points=jdbc:sqlite:<SQLITE_DB_FILE_PATH>?busy_timeout=10000

Unlike other JDBC databases, SQLite3 doesn't fully support concurrent access. To avoid frequent errors caused internally by SQLITE_BUSY, we recommend setting a busy_timeout parameter.

Multi-storage support​

ScalarDB supports using multiple storage implementations simultaneously. You can use multiple storages by specifying multi-storage as the value for the scalar.db.storage property.

For details about using multiple storages, see Multi-Storage Transactions.

Cross-partition scan configurations​

By enabling the cross-partition scan option as described below, the Scan operation can retrieve all records across partitions. In addition, you can specify arbitrary conditions and orderings in the cross-partition Scan operation by enabling cross_partition_scan.filtering and cross_partition_scan.ordering, respectively. Currently, the cross-partition scan with ordering option is available only for JDBC databases. To enable filtering and ordering, scalar.db.cross_partition_scan.enabled must be set to true.

For details on how to use cross-partition scan, see Scan operation.

warning

For non-JDBC databases, we do not recommend enabling cross-partition scan with the SERIALIAZABLE isolation level because transactions could be executed at a lower isolation level (that is, SNAPSHOT). When using non-JDBC databases, use cross-partition scan at your own risk only if consistency does not matter for your transactions.

NameDescriptionDefault
scalar.db.cross_partition_scan.enabledEnable cross-partition scan.false
scalar.db.cross_partition_scan.filtering.enabledEnable filtering in cross-partition scan.false
scalar.db.cross_partition_scan.ordering.enabledEnable ordering in cross-partition scan.false

Scan fetch size​

You can configure the fetch size for storage scan operations by using the following property:

NameDescriptionDefault
scalar.db.scan_fetch_sizeSpecifies the number of records to fetch in a single batch during a storage scan operation. A larger value can improve performance for a large result set by reducing round trips to the storage, but it also increases memory usage. A smaller value uses less memory but may increase latency.10

The configurations for ScalarDB Cluster GraphQL are as follows:

NameDescriptionDefault
scalar.db.graphql.enabledWhether ScalarDB Cluster GraphQL is enabled.false
scalar.db.graphql.portPort number of the GraphQL server.8080
scalar.db.graphql.pathPath component of the URL of the GraphQL endpoint./graphql
scalar.db.graphql.namespacesComma-separated list of namespaces of tables for which the GraphQL server generates a schema. Note that at least one namespace is required.
scalar.db.graphql.graphiqlWhether the GraphQL server serves GraphiQL IDE.true
scalar.db.graphql.schema_checking_interval_millisInterval in milliseconds at which GraphQL server will rebuild the GraphQL schema if any change is detected in the ScalarDB schema.30000 (30 seconds)

Creating or modifying the ScalarDB schema when the server is running​

Since the GraphQL schema is statically built at server startup, if the ScalarDB schema is modified (for example, if a table is added, altered, or deleted), then the corresponding GraphQL schema won't reflect the changes unless it is rebuilt. To address this, the GraphQL server provides two mechanisms: a periodic check and an on-demand check.

Run periodic checks​

The server periodically checks if changes in the ScalarDB schema occur and rebuilds the corresponding GraphQL schema if necessary. By default, the check occurs every 30 seconds, but the interval can be configured by using the scalar.db.graphql.schema_checking_interval_millis property.

If you don't need to run periodic checks, you can disable it by setting the property value to -1.

Run on-demand checks​

You can also request the server to check changes in the ScalarDB schema and rebuild the corresponding GraphQL schema if necessary by performing a POST request to the /update-graphql-schema endpoint of the HTTP API.

For example, if the HTTP API is running on localhost:8080 and the scalar.db.graphql.path property is set to /graphql, this endpoint can be called by running the following command:

curl -X POST http://localhost:8080/graphql/update-graphql-schema

The configurations for ScalarDB Cluster SQL are as follows:

NameDescriptionDefault
scalar.db.sql.enabledWhether ScalarDB Cluster SQL is enabled.false
scalar.db.sql.statement_cache.enabledEnable the statement cache.false
scalar.db.sql.statement_cache.sizeMaximum number of cached statements.100
scalar.db.sql.default_transaction_modeDefault transaction mode. TRANSACTION or TWO_PHASE_COMMIT_TRANSACTION can be set.TRANSACTION
scalar.db.sql.default_namespace_nameDefault namespace name. If you don't specify a namespace name in your SQL statement, this value is used.

Other ScalarDB Cluster configurations​

The following are additional configurations available for ScalarDB Cluster:

NameDescriptionDefault
scalar.db.metadata.cache_expiration_time_secsScalarDB has a metadata cache to reduce the number of requests to the database. This setting specifies the expiration time of the cache in seconds. If you specify -1, the cache will never expire.60
scalar.db.active_transaction_management.expiration_time_millisScalarDB maintains in-progress transactions, which can be resumed by using a transaction ID. This process expires transactions that have been idle for an extended period to prevent resource leaks. This setting specifies the expiration time of this transaction management feature in milliseconds.60000 (60 seconds)
scalar.db.consensus_commit.include_metadata.enabledWhen using Consensus Commit, if this is set to true, Get and Scan operations results will contain transaction metadata. To see the transaction metadata columns details for a given table, you can use the DistributedTransactionAdmin.getTableMetadata() method, which will return the table metadata augmented with the transaction metadata columns. Using this configuration can be useful to investigate transaction-related issues.false
scalar.db.default_namespace_nameThe given namespace name will be used by operations that do not already specify a namespace.

Client configurations​

This section describes the general configurations for the ScalarDB Cluster client.

Configurations for the primitive interface​

The following table shows the general configurations for the ScalarDB Cluster client.

NameDescriptionDefault
scalar.db.transaction_managercluster should be specified.-
scalar.db.contact_pointsContact point of the cluster. If you use the indirect client mode, specify the IP address of the load balancer in front of your cluster nodes by using the format indirect:<the load balancer IP address>. If you use the direct-kubernetes client mode, specify the namespace name (optional) and the name of the endpoint resource to get the membership information by using the format direct-kubernetes:<namespace name>/<endpoint name> or just direct-kubernetes:<endpoint name>. If you don't specify the namespace name, the client will use the default namespace.
scalar.db.contact_portPort number for the contact point.60053
scalar.db.cluster.grpc.deadline_duration_millisDeadline duration for gRPC in millis.60000 (60 seconds)
scalar.db.cluster.grpc.max_inbound_message_sizeMaximum message size allowed for a single gRPC frame.The gRPC default value
scalar.db.cluster.grpc.max_inbound_metadata_sizeMaximum size of metadata allowed to be received.The gRPC default value
scalar.db.cluster.client.scan_fetch_sizeThe fetch size used for Scanner to fetch data from the cluster. This is the number of records that Scanner fetches at once from the cluster. A larger value can improve performance by reducing the number of round trips to the cluster, but it may also increase memory usage.10

For example, if you use the indirect client mode and the load balancer IP address is 192.168.10.1, you can configure the client as follows:

scalar.db.transaction_manager=cluster
scalar.db.contact_points=indirect:192.168.10.1

Or if you use the direct-kubernetes client mode, with the namespace of the endpoint as ns and the endpoint name as scalardb-cluster, you can configure the client as follows:

scalar.db.transaction_manager=cluster
scalar.db.contact_points=direct-kubernetes:ns/scalardb-cluster

Configurations for the SQL interface​

The following table shows the configurations for the ScalarDB Cluster SQL client.

NameDescriptionDefault
scalar.db.sql.connection_modecluster should be specified.-
scalar.db.sql.cluster_mode.contact_pointsContact point of the cluster. If you use the indirect client mode, specify the IP address of the load balancer in front of your cluster nodes by using the format indirect:<the load balancer IP address>. If you use the direct-kubernetes client mode, specify the namespace name (optional) and the name of the endpoint resource to get the membership information by using the format direct-kubernetes:<namespace name>/<endpoint name> or just direct-kubernetes:<endpoint name>. If you don't specify the namespace name, the client will use the default namespace.
scalar.db.sql.cluster_mode.contact_portPort number for the contact point.60053
scalar.db.sql.default_transaction_modeDefault transaction mode. TRANSACTION or TWO_PHASE_COMMIT_TRANSACTION can be set.TRANSACTION
scalar.db.sql.default_namespace_nameDefault namespace name. If you don't specify a namespace name in your SQL statement, this value is used.
scalar.db.cluster.grpc.deadline_duration_millisDeadline duration for gRPC in millis.60000 (60 seconds)
scalar.db.cluster.grpc.max_inbound_message_sizeMaximum message size allowed for a single gRPC frame.The gRPC default value
scalar.db.cluster.grpc.max_inbound_metadata_sizeMaximum size of metadata allowed to be received.The gRPC default value

For example, if you use the indirect client mode and the load balancer IP address is 192.168.10.1, you can configure the client as follows:

scalar.db.sql.connection_mode=cluster
scalar.db.sql.cluster_mode.contact_points=indirect:192.168.10.1

Or if you use the direct-kubernetes client mode, with the namespace of the endpoint as ns and the endpoint name as scalardb-cluster, you can configure the client as follows:

scalar.db.sql.connection_mode=cluster
scalar.db.sql.cluster_mode.contact_points=direct-kubernetes:ns/scalardb-cluster

For details about how to configure ScalarDB JDBC, see JDBC connection URL.

For details about how to configure Spring Data JDBC for ScalarDB, see Configurations.

Configuration example - App, ScalarDB Cluster, and database​

In this example configuration, the app (ScalarDB library with gRPC) connects to an underlying storage or database (in this case, Cassandra) through ScalarDB Cluster, which is a component that is available only in the ScalarDB Enterprise edition.

note

This configuration is acceptable for production use because ScalarDB Cluster implements the Scalar Admin interface, which enables you to take transactionally consistent backups for ScalarDB by pausing ScalarDB Cluster.

The following is an example of the configuration for connecting the app to the underlying database through ScalarDB Cluster:

# Transaction manager implementation.
scalar.db.transaction_manager=cluster

# Contact point of the cluster.
scalar.db.contact_points=indirect:<SCALARDB_CLUSTER_CONTACT_POINT>

For details about client configurations, see the ScalarDB Cluster client configurations.

Footnotes​

  1. It's worth benchmarking the performance with a few variations (for example, 75% and 125% of the default value) on the same underlying storage that your application uses, considering your application's access pattern, to determine the optimal configuration as it really depends on those factors. Also, it's important to benchmark combinations of these parameters (for example, first, slot_capacity:20 and group_size_fix_timeout_millis:40; second, slot_capacity:30 and group_size_fix_timeout_millis:40; and third, slot_capacity:20 and group_size_fix_timeout_millis:80) to determine the optimal combination. ↩ ↩2 ↩3