# ScalarDB Documentation - Full Repository Context # Generated by using GitIngest for AI/LLM consumption # Universal hybrid transaction/analytical processing (HTAP) engine for diverse databases # Website: https://scalardb.scalar-labs.com Directory: home/runner/work/docs-scalardb/docs-scalardb Files analyzed: 214 Estimated tokens: 542.1k Directory structure: └── docs-scalardb/ ├── docs/ │ ├── add-scalardb-to-your-build.mdx │ ├── api-guide.mdx │ ├── backup-restore.mdx │ ├── configurations.mdx │ ├── consensus-commit.mdx │ ├── data-loader.mdx │ ├── data-modeling.mdx │ ├── database-adapters.mdx │ ├── database-configurations.mdx │ ├── deploy-overview.mdx │ ├── design.mdx │ ├── develop-overview.mdx │ ├── develop-run-analytical-queries-overview.mdx │ ├── develop-run-non-transactional-operations-overview.mdx │ ├── develop-run-transactions-overview.mdx │ ├── features.mdx │ ├── getting-started-with-benchmarking-scalardb.mdx │ ├── getting-started-with-scalardb-by-using-kotlin.mdx │ ├── getting-started-with-scalardb.mdx │ ├── glossary.mdx │ ├── index.mdx │ ├── learning-paths.mdx │ ├── libraries-and-tools.mdx │ ├── manage-backup-and-restore.mdx │ ├── manage-monitor-overview.mdx │ ├── manage-overview.mdx │ ├── migrate-overview.mdx │ ├── multi-storage-transactions.mdx │ ├── onboarding.mdx │ ├── overview.mdx │ ├── quickstart-overview.mdx │ ├── quickstart-scalardb-analytics-overview.mdx │ ├── quickstart-scalardb-cluster-overview.mdx │ ├── quickstart-scalardb-core-overview.mdx │ ├── requirements.mdx │ ├── roadmap.mdx │ ├── run-non-transactional-storage-operations-through-library.mdx │ ├── run-non-transactional-storage-operations-through-primitive-crud-interface.mdx │ ├── run-transactions-through-scalardb-core-library.mdx │ ├── scalardb-core-status-codes.mdx │ ├── scalardb-data-loader-status-codes.mdx │ ├── scalardb-schema-loader-status-codes.mdx │ ├── schema-loader-import.mdx │ ├── schema-loader.mdx │ ├── two-phase-commit-transactions.mdx │ ├── components/ │ │ ├── _getting-started-load-schema.mdx │ │ └── _getting-started-setup-storage.mdx │ ├── helm-charts/ │ │ ├── configure-custom-values-envoy.mdx │ │ ├── configure-custom-values-file.mdx │ │ ├── configure-custom-values-scalar-admin-for-kubernetes.mdx │ │ ├── configure-custom-values-scalar-manager.mdx │ │ ├── configure-custom-values-scalardb-analytics-server.mdx │ │ ├── configure-custom-values-scalardb-cluster.mdx │ │ ├── configure-custom-values-scalardb-graphql.mdx │ │ ├── configure-custom-values-scalardb.mdx │ │ ├── configure-custom-values-scalardl-auditor.mdx │ │ ├── configure-custom-values-scalardl-ledger.mdx │ │ ├── configure-custom-values-scalardl-schema-loader.mdx │ │ ├── getting-started-logging.mdx │ │ ├── getting-started-monitoring.mdx │ │ ├── getting-started-scalar-helm-charts.mdx │ │ ├── getting-started-scalar-manager.mdx │ │ ├── getting-started-scalardb-cluster-tls-cert-manager.mdx │ │ ├── getting-started-scalardb-cluster-tls.mdx │ │ ├── getting-started-scalardb.mdx │ │ ├── getting-started-scalardl-auditor-tls-cert-manager.mdx │ │ ├── getting-started-scalardl-auditor-tls.mdx │ │ ├── getting-started-scalardl-auditor.mdx │ │ ├── getting-started-scalardl-ledger.mdx │ │ ├── how-to-deploy-scalar-admin-for-kubernetes.mdx │ │ ├── how-to-deploy-scalar-products.mdx │ │ ├── how-to-deploy-scalardb-cluster.mdx │ │ ├── how-to-deploy-scalardb-graphql.mdx │ │ ├── how-to-deploy-scalardb.mdx │ │ ├── how-to-deploy-scalardl-auditor.mdx │ │ ├── how-to-deploy-scalardl-ledger.mdx │ │ ├── mount-files-or-volumes-on-scalar-pods.mdx │ │ └── use-secret-for-credentials.mdx │ ├── releases/ │ │ ├── release-notes.mdx │ │ └── release-support-policy.mdx │ ├── scalar-kubernetes/ │ │ ├── AccessScalarProducts.mdx │ │ ├── AwsMarketplaceGuide.mdx │ │ ├── BackupNoSQL.mdx │ │ ├── BackupRDB.mdx │ │ ├── BackupRestoreGuide.mdx │ │ ├── CreateAKSClusterForScalarDB.mdx │ │ ├── CreateAKSClusterForScalarDL.mdx │ │ ├── CreateAKSClusterForScalarDLAuditor.mdx │ │ ├── CreateAKSClusterForScalarProducts.mdx │ │ ├── CreateBastionServer.mdx │ │ ├── CreateEKSClusterForScalarDB.mdx │ │ ├── CreateEKSClusterForScalarDBCluster.mdx │ │ ├── CreateEKSClusterForScalarDL.mdx │ │ ├── CreateEKSClusterForScalarDLAuditor.mdx │ │ ├── CreateEKSClusterForScalarProducts.mdx │ │ ├── HowToCreateKeyAndCertificateFiles.mdx │ │ ├── HowToGetContainerImages.mdx │ │ ├── HowToScaleScalarDB.mdx │ │ ├── HowToScaleScalarDL.mdx │ │ ├── HowToUpgradeScalarDB.mdx │ │ ├── HowToUpgradeScalarDL.mdx │ │ ├── HowToUseContainerImages.mdx │ │ ├── K8sLogCollectionGuide.mdx │ │ ├── K8sMonitorGuide.mdx │ │ ├── ManualDeploymentGuideScalarDBClusterOnEKS.mdx │ │ ├── ManualDeploymentGuideScalarDBServerOnAKS.mdx │ │ ├── ManualDeploymentGuideScalarDBServerOnEKS.mdx │ │ ├── ManualDeploymentGuideScalarDLAuditorOnAKS.mdx │ │ ├── ManualDeploymentGuideScalarDLAuditorOnEKS.mdx │ │ ├── ManualDeploymentGuideScalarDLOnAKS.mdx │ │ ├── ManualDeploymentGuideScalarDLOnEKS.mdx │ │ ├── NetworkPeeringForScalarDLAuditor.mdx │ │ ├── ProductionChecklistForScalarDBCluster.mdx │ │ ├── ProductionChecklistForScalarDLAuditor.mdx │ │ ├── ProductionChecklistForScalarDLLedger.mdx │ │ ├── ProductionChecklistForScalarProducts.mdx │ │ ├── RegularCheck.mdx │ │ ├── RestoreDatabase.mdx │ │ ├── SetupDatabase.mdx │ │ ├── SetupDatabaseForAWS.mdx │ │ ├── SetupDatabaseForAzure.mdx │ │ └── alerts/ │ │ ├── README.mdx │ │ ├── Envoy.mdx │ │ └── Ledger.mdx │ ├── scalar-licensing/ │ │ ├── commercial.mdx │ │ ├── index.mdx │ │ └── trial.mdx │ ├── scalar-manager/ │ │ ├── how-to-use-scalar-manager.mdx │ │ ├── metrics-reference.mdx │ │ └── overview.mdx │ ├── scalardb-analytics/ │ │ ├── _README.mdx │ │ ├── authentication-and-authorization.mdx │ │ ├── configurations.mdx │ │ ├── create-scalardb-analytics-catalog.mdx │ │ ├── deploy-scalardb-analytics-server.mdx │ │ ├── deployment-local.mdx │ │ ├── deployment.mdx │ │ ├── design.mdx │ │ ├── quickstart.mdx │ │ ├── reference-cli-command.mdx │ │ ├── reference-data-source.mdx │ │ └── run-analytical-queries.mdx │ ├── scalardb-benchmarks/ │ │ └── README.mdx │ ├── scalardb-cluster/ │ │ ├── authorize-with-abac.mdx │ │ ├── compatibility.mdx │ │ ├── control-access-via-oidc-based-jwt-tokens.mdx │ │ ├── deploy-scalardb-cluster-google-cloud-marketplace.mdx │ │ ├── deployment-patterns-for-microservices.mdx │ │ ├── developer-guide-for-scalardb-cluster-with-java-api.mdx │ │ ├── encrypt-data-at-rest.mdx │ │ ├── encrypt-wire-communications.mdx │ │ ├── getting-started-with-scalardb-cluster-dotnet.mdx │ │ ├── getting-started-with-scalardb-cluster-graphql.mdx │ │ ├── getting-started-with-scalardb-cluster-sql-dotnet.mdx │ │ ├── getting-started-with-scalardb-cluster-sql-jdbc.mdx │ │ ├── getting-started-with-scalardb-cluster-sql-linq.mdx │ │ ├── getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx │ │ ├── getting-started-with-scalardb-cluster.mdx │ │ ├── getting-started-with-using-go-for-scalardb-cluster.mdx │ │ ├── getting-started-with-using-python-for-scalardb-cluster.mdx │ │ ├── getting-started-with-vector-search.mdx │ │ ├── index.mdx │ │ ├── remote-replication.mdx │ │ ├── run-non-transactional-storage-operations-through-scalardb-cluster.mdx │ │ ├── run-non-transactional-storage-operations-through-sql-interface.mdx │ │ ├── run-transactions-through-scalardb-cluster-sql.mdx │ │ ├── run-transactions-through-scalardb-cluster.mdx │ │ ├── scalardb-abac-status-codes.mdx │ │ ├── scalardb-auth-status-codes.mdx │ │ ├── scalardb-auth-with-sql.mdx │ │ ├── scalardb-cluster-configurations.mdx │ │ ├── scalardb-cluster-grpc-api-guide.mdx │ │ ├── scalardb-cluster-sql-grpc-api-guide.mdx │ │ ├── scalardb-cluster-status-codes.mdx │ │ ├── scalardb-embedding-store-status-codes.mdx │ │ ├── scalardb-encryption-status-codes.mdx │ │ ├── scalardb-remote-replication-status-codes.mdx │ │ ├── setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx │ │ └── standalone-mode.mdx │ ├── scalardb-cluster-dotnet-client-sdk/ │ │ ├── common-reference.mdx │ │ ├── exception-handling.mdx │ │ ├── getting-started-with-admin-api.mdx │ │ ├── getting-started-with-aspnet-and-di.mdx │ │ ├── getting-started-with-auth.mdx │ │ ├── getting-started-with-distributed-sql-transactions.mdx │ │ ├── getting-started-with-distributed-transactions.mdx │ │ ├── getting-started-with-linq.mdx │ │ ├── getting-started-with-scalardb-tables-as-csharp-classes.mdx │ │ ├── getting-started-with-two-phase-commit-transactions.mdx │ │ └── index.mdx │ ├── scalardb-data-loader/ │ │ ├── getting-started-export.mdx │ │ └── getting-started-import.mdx │ ├── scalardb-graphql/ │ │ ├── how-to-run-two-phase-commit-transaction.mdx │ │ ├── index.mdx │ │ └── scalardb-graphql-status-codes.mdx │ ├── scalardb-mcp-server/ │ │ ├── getting-started-with-scalardb-mcp-server.mdx │ │ └── tools-reference.mdx │ ├── scalardb-samples/ │ │ ├── README.mdx │ │ ├── dotnet-microservice-transactions-sample-with-shared-cluster-with-linq/ │ │ │ └── README.mdx │ │ ├── microservice-transaction-sample/ │ │ │ └── README.mdx │ │ ├── microservice-transactions-sample-with-shared-cluster-with-jdbc/ │ │ │ └── README.mdx │ │ ├── multi-storage-transaction-sample/ │ │ │ └── README.mdx │ │ ├── spring-data-microservice-transaction-sample/ │ │ │ └── README.mdx │ │ └── spring-data-multi-storage-transaction-sample/ │ │ └── README.mdx │ └── scalardb-sql/ │ ├── index.mdx │ ├── jdbc-guide.mdx │ ├── migration-guide.mdx │ ├── scalardb-sql-status-codes.mdx │ ├── spring-data-guide.mdx │ └── sql-api-guide.mdx └── src/ └── components/ └── en-us/ ├── _certificate-management.mdx ├── _helm-command-usage.mdx ├── _prerequisites-jdk-versions.mdx └── _warning-license-key-contact.mdx ================================================ FILE: docs/add-scalardb-to-your-build.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Add ScalarDB to Your Build import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; The ScalarDB library is available on the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb). You can add the library as a build dependency to your application by using Gradle or Maven. ## Configure your application based on your build tool Select your build tool, and follow the instructions to add the build dependency for ScalarDB to your application. To add the build dependency for ScalarDB by using Gradle, add the following to `build.gradle` in your application, replacing `` with the version of ScalarDB that you want to use: ```gradle dependencies { implementation 'com.scalar-labs:scalardb:' } ``` To add the build dependency for ScalarDB by using Maven, add the following to `pom.xml` in your application, replacing `` with the version of ScalarDB that you want to use: ```xml com.scalar-labs scalardb ``` ================================================ FILE: docs/api-guide.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Java API Guide import JavadocLink from '/src/theme/JavadocLink.js'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; The ScalarDB Java API is mainly composed of the Administrative API and Transactional API. This guide briefly explains what kinds of APIs exist, how to use them, and related topics like how to handle exceptions. ## Administrative API This section explains how to execute administrative operations programmatically by using the Administrative API in ScalarDB. :::warning When an Administrative API call writes to the underlying databases, it triggers several write operations. However, these operations are not executed atomically, meaning that if the call fails midway, you may encounter inconsistent states. To resolve this inconsistency issue, you can repair the table. For details, see the following pages: - [Repair a table](#repair-a-table) by using the Java API - [Repair tables](schema-loader.mdx#repair-tables) by using ScalarDB Schema Loader ::: :::note Another method for executing administrative operations is to use [Schema Loader](schema-loader.mdx). ::: ### Get a `DistributedTransactionAdmin` instance You first need to get a `DistributedTransactionAdmin` instance to execute administrative operations. To get a `DistributedTransactionAdmin` instance, you can use `TransactionFactory` as follows: ```java TransactionFactory transactionFactory = TransactionFactory.create(""); DistributedTransactionAdmin admin = transactionFactory.getTransactionAdmin(); ``` For details about configurations, see [ScalarDB Configurations](configurations.mdx). After you have executed all administrative operations, you should close the `DistributedTransactionAdmin` instance as follows: ```java admin.close(); ``` ### Create a namespace Before creating tables, namespaces must be created since a table belongs to one namespace. You can create a namespace as follows: ```java // Create the namespace "ns". If the namespace already exists, an exception will be thrown. admin.createNamespace("ns"); // Create the namespace only if it does not already exist. boolean ifNotExists = true; admin.createNamespace("ns", ifNotExists); // Create the namespace with options. Map options = ...; admin.createNamespace("ns", options); ``` #### Creation options In the namespace creation operations, you can specify options that are maps of option names and values (`Map`). By using the options, you can set storage adapter–specific configurations. Select your database to see the options available: No options are available. No options are available. | Name | Description | Default | |------------|-----------------------------------------------------|---------| | ru | Base resource unit. | 400 | | no-scaling | Disable auto-scaling for Cosmos DB for NoSQL. | false | | Name | Description | Default | |----------------------|----------------------------------------------------------------------------------------|------------------| | replication-strategy | Cassandra replication strategy. Must be `SimpleStrategy` or `NetworkTopologyStrategy`. | `SimpleStrategy` | | replication-factor | Cassandra replication factor. | 1 | No options are available. ### Create a table When creating a table, you should define the table metadata and then create the table. To define the table metadata, you can use `TableMetadata`. The following shows how to define the columns, partition key, clustering key including clustering orders, and secondary indexes of a table: ```java // Define the table metadata. TableMetadata tableMetadata = TableMetadata.newBuilder() .addColumn("c1", DataType.INT) .addColumn("c2", DataType.TEXT) .addColumn("c3", DataType.BIGINT) .addColumn("c4", DataType.FLOAT) .addColumn("c5", DataType.DOUBLE) .addPartitionKey("c1") .addClusteringKey("c2", Scan.Ordering.Order.DESC) .addClusteringKey("c3", Scan.Ordering.Order.ASC) .addSecondaryIndex("c4") .build(); ``` For details about the data model of ScalarDB, see [Data Model](design.mdx#data-model). Then, create a table as follows: ```java // Create the table "ns.tbl". If the table already exists, an exception will be thrown. admin.createTable("ns", "tbl", tableMetadata); // Create the table only if it does not already exist. boolean ifNotExists = true; admin.createTable("ns", "tbl", tableMetadata, ifNotExists); // Create the table with options. Map options = ...; admin.createTable("ns", "tbl", tableMetadata, options); ``` #### Creation options In the table creation operations, you can specify options that are maps of option names and values (`Map`). By using the options, you can set storage adapter–specific configurations. Select your database to see the options available: | Name | Description | Default | |------------|-----------------------------------------|---------| | transaction-metadata-decoupling | Enable [transaction metadata decoupling](./consensus-commit.mdx#transaction-metadata-decoupling) when using Consensus Commit, which manages the transaction metadata in a separate table from application data. | false | | Name | Description | Default | |------------|-----------------------------------------|---------| | no-scaling | Disable auto-scaling for DynamoDB. | false | | no-backup | Disable continuous backup for DynamoDB. | false | | ru | Base resource unit. | 10 | No options are available. | Name | Description | Default | |----------------------|----------------------------------------------------------------------------------------|------------------| | compaction-strategy | Cassandra compaction strategy, Must be `LCS`, `STCS`, or `TWCS`. | `STCS` | No options are available. ### Create a secondary index You can create a secondary index as follows: ```java // Create a secondary index on column "c5" for table "ns.tbl". If a secondary index already exists, an exception will be thrown. admin.createIndex("ns", "tbl", "c5"); // Create the secondary index only if it does not already exist. boolean ifNotExists = true; admin.createIndex("ns", "tbl", "c5", ifNotExists); // Create the secondary index with options. Map options = ...; admin.createIndex("ns", "tbl", "c5", options); ``` :::warning When using Consensus Commit, `createIndex()` on a non-primary-key column also creates a companion before-image secondary index. For details, see [Correctness of index-based reads](consensus-commit.mdx#correctness-of-index-based-reads). ::: #### Creation options In the secondary index creation operations, you can specify options that are maps of option names and values (`Map`). By using the options, you can set storage adapter–specific configurations. Select your database to see the options available: No options are available for JDBC databases. | Name | Description | Default | |------------|-----------------------------------------|---------| | no-scaling | Disable auto-scaling for DynamoDB. | false | | ru | Base resource unit. | 10 | No options are available. No options are available. No options are available. ### Add a new column to a table You can add a new, non-partition key column to a table as follows: ```java // Add a new column "c6" with the INT data type to the table "ns.tbl". admin.addNewColumnToTable("ns", "tbl", "c6", DataType.INT); // Add the new column only if it does not already exist. boolean ifNotExists = true; admin.addNewColumnToTable("ns", "tbl", "c6", DataType.INT, false, ifNotExists); ``` :::warning You should carefully consider adding a new column to a table because the execution time may vary greatly depending on the underlying storage. Please plan accordingly and consider the following, especially if the database runs in production: - **For Cosmos DB for NoSQL and DynamoDB:** Adding a column is almost instantaneous as the table schema is not modified. Only the table metadata stored in a separate table is updated. - **For Cassandra:** Adding a column will only update the schema metadata and will not modify the existing schema records. The cluster topology is the main factor for the execution time. Changes to the schema metadata are shared to each cluster node via a gossip protocol. Because of this, the larger the cluster, the longer it will take for all nodes to be updated. - **For relational databases (MySQL, Oracle, etc.):** Adding a column can cause a table rebuild depending on the database engine. In such cases, it can take a long time to execute. ::: ### Drop a column from a table You can drop a column from a table as follows: ```java // Drop the column "c6" from the table "ns.tbl". admin.dropColumnFromTable("ns", "tbl", "c6"); // Drop the column only if it exists. boolean ifExists = true; admin.dropColumnFromTable("ns", "tbl", "c6", ifExists); ``` :::note You cannot drop a column from a table in the following cases: - The column is part of the partition key or clustering key. - The table is on a non-JDBC database except for Cassandra. ::: :::warning You should carefully consider dropping a column from a table because the execution time may vary greatly depending on the underlying storage. Please plan accordingly and consider the following, especially if the database runs in production: - **For Cassandra:** Dropping a column will only update the schema metadata and the actual data will be removed during the next compaction. The cluster topology is the main factor for the execution time. Changes to the schema metadata are shared to each cluster node via a gossip protocol. Because of this, the larger the cluster, the longer it will take for all nodes to be updated. - **For relational databases (MySQL, Oracle, etc.):** Dropping a column issues `ALTER TABLE ... DROP COLUMN` to the underlying relational databases and could trigger a table rebuild depending on the database. In such cases, it can take a long time to execute. ::: ### Rename a column of a table You can rename a column of a table as follows: ```java // Rename the column "c6" to "c66" in the table "ns.tbl". admin.renameColumnInTable("ns", "tbl", "c6", "c66"); ``` :::note You cannot rename a column of a table in the following cases: - The table is on a non-JDBC database except for Cassandra. - For Cassandra, the column is not part of the partition key or clustering key. - For Db2, the column is part of the partition key, clustering key, or secondary index key. ::: ### Rename a table You can rename a table as follows: ```java // Rename the table "ns.tbl" to "ns.new_tbl". admin.renameTable("ns", "tbl", "new_tbl"); ``` :::note You cannot rename a table on non-JDBC databases. ::: ### Alter a column data type of a table You can alter a column data type of a table as follows: ```java // Alter the data type of the column "c6" to BIGINT in the table "ns.tbl". admin.alterColumnDataType("ns", "tbl", "c6", DataType.BIGINT); ``` :::note You cannot alter a column data type of a table in the following cases: - The column is part of the partition key, clustering key, or secondary index key. - The table is on a non-JDBC database or on SQLite. - The conversions other than from INT to BIGINT, FLOAT to DOUBLE, and from any data to TEXT are specified. - For Oracle, the conversions except for from INT to BIGINT are specified. - For Db2 and TiDB, the conversion from BLOB to TEXT is specified. ::: :::warning You should carefully consider altering a column type because the execution time may vary greatly depending on the underlying storage. Please plan accordingly and consider the following, especially if the database runs in production: - **For relational databases (MySQL, Oracle, etc.):** Altering a column issues `ALTER TABLE ... ALTER COLUMN` or `ALTER TABLE ... MODIFY` to the underlying relational databases and could trigger a table rebuild depending on the database. In such cases, it can take a long time to execute. ::: ### Truncate a table You can truncate a table as follows: ```java // Truncate the table "ns.tbl". admin.truncateTable("ns", "tbl"); ``` ### Drop a secondary index You can drop a secondary index as follows: ```java // Drop the secondary index on column "c5" from table "ns.tbl". If the secondary index does not exist, an exception will be thrown. admin.dropIndex("ns", "tbl", "c5"); // Drop the secondary index only if it exists. boolean ifExists = true; admin.dropIndex("ns", "tbl", "c5", ifExists); ``` :::warning When using Consensus Commit, `dropIndex()` also drops the companion before-image secondary index. For details, see [Correctness of index-based reads](consensus-commit.mdx#correctness-of-index-based-reads). ::: ### Drop a table You can drop a table as follows: ```java // Drop the table "ns.tbl". If the table does not exist, an exception will be thrown. admin.dropTable("ns", "tbl"); // Drop the table only if it exists. boolean ifExists = true; admin.dropTable("ns", "tbl", ifExists); ``` ### Drop a namespace You can drop a namespace as follows: ```java // Drop the namespace "ns". If the namespace does not exist, an exception will be thrown. admin.dropNamespace("ns"); // Drop the namespace only if it exists. boolean ifExists = true; admin.dropNamespace("ns", ifExists); ``` ### Get existing namespaces You can get the existing namespaces as follows: ```java Set namespaces = admin.getNamespaceNames(); ``` :::note This method extracts the namespace names of user tables dynamically. As a result, only namespaces that contain tables are returned. Starting from ScalarDB 4.0, we plan to improve the design to remove this limitation. ::: ### Get the tables of a namespace You can get the tables of a namespace as follows: ```java // Get the tables of the namespace "ns". Set tables = admin.getNamespaceTableNames("ns"); ``` ### Get table metadata You can get table metadata as follows: ```java // Get the table metadata for "ns.tbl". TableMetadata tableMetadata = admin.getTableMetadata("ns", "tbl"); ``` ### Repair a table You can repair the table metadata of an existing table as follows: ```java // Repair the table "ns.tbl" with options. TableMetadata tableMetadata = TableMetadata.newBuilder() ... .build(); Map options = ...; admin.repairTable("ns", "tbl", tableMetadata, options); ``` :::warning When using Consensus Commit, after upgrading to ScalarDB 3.16.5, 3.17.3, or 3.18.0 from an earlier version, you must run `repairTable()` on each existing table to create the companion before-image secondary indexes that index-based reads require. For details, see [Correctness of index-based reads](consensus-commit.mdx#correctness-of-index-based-reads). ::: ### Specify operations for the Coordinator table The Coordinator table is used by the [Transactional API](#transactional-api) to track the statuses of transactions. When using a transaction manager, you must create the Coordinator table to execute transactions. In addition to creating the table, you can truncate and drop the Coordinator table. #### Create the Coordinator table You can create the Coordinator table as follows: ```java // Create the Coordinator table. admin.createCoordinatorTables(); // Create the Coordinator table only if one does not already exist. boolean ifNotExist = true; admin.createCoordinatorTables(ifNotExist); // Create the Coordinator table with options. Map options = ...; admin.createCoordinatorTables(options); ``` #### Truncate the Coordinator table You can truncate the Coordinator table as follows: ```java // Truncate the Coordinator table. admin.truncateCoordinatorTables(); ``` #### Drop the Coordinator table You can drop the Coordinator table as follows: ```java // Drop the Coordinator table. admin.dropCoordinatorTables(); // Drop the Coordinator table if one exist. boolean ifExist = true; admin.dropCoordinatorTables(ifExist); ``` ### Import a table You can import an existing table to ScalarDB as follows: ```java // Import the table "ns.tbl". If the table is already managed by ScalarDB, the target table does not // exist, or the table does not meet the requirements of the ScalarDB table, an exception will be thrown. admin.importTable("ns", "tbl", options, overrideColumnsType); ``` :::warning When using Consensus Commit, you should carefully plan to import a table to ScalarDB in production because it will add transaction metadata columns to your database tables and the ScalarDB metadata tables. In this case, there would also be several differences between your database and ScalarDB, as well as some limitations. For details, see [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](./schema-loader-import.mdx). You can also enable *transaction metadata decoupling* by specifying the `transaction-metadata-decoupling` option to `true` to store transaction metadata in a separate table from application data. For details, see [Transaction Metadata Decoupling](./schema-loader.mdx#transaction-metadata-decoupling). ::: ## Transactional API This section explains how to execute transactional operations by using the Transactional API in ScalarDB. ### Get a `DistributedTransactionManager` instance You first need to get a `DistributedTransactionManager` instance to execute transactional operations. To get a `DistributedTransactionManager` instance, you can use `TransactionFactory` as follows: ```java TransactionFactory transactionFactory = TransactionFactory.create(""); DistributedTransactionManager transactionManager = transactionFactory.getTransactionManager(); ``` After you have executed all transactional operations, you should close the `DistributedTransactionManager` instance as follows: ```java transactionManager.close(); ``` ### Execute transactions This subsection explains how to execute transactions with multiple CRUD operations. #### Begin or start a transaction Before executing transactional CRUD operations, you need to begin or start a transaction. You can begin a transaction as follows: ```java // Begin a transaction. DistributedTransaction transaction = transactionManager.begin(); ``` Or, you can start a transaction as follows: ```java // Start a transaction. DistributedTransaction transaction = transactionManager.start(); ``` Alternatively, you can use the `begin` method for a transaction by specifying a transaction ID as follows: ```java // Begin a transaction by specifying a transaction ID. DistributedTransaction transaction = transactionManager.begin(""); ``` Or, you can use the `start` method for a transaction by specifying a transaction ID as follows: ```java // Start a transaction by specifying a transaction ID. DistributedTransaction transaction = transactionManager.start(""); ``` :::note Specifying a transaction ID is useful when you want to link external systems to ScalarDB. Otherwise, you should use the `begin()` method or the `start()` method. When you specify a transaction ID, make sure you specify a unique ID (for example, UUID v4) throughout the system since ScalarDB depends on the uniqueness of transaction IDs for correctness. ::: ##### Begin or start a transaction in read-only mode You can also begin or start a transaction in read-only mode. In this case, the transaction will not allow any write operations, and it will be optimized for read operations. :::note Using read-only transactions for read-only operations is strongly recommended to improve performance and reduce resource usage. ::: You can begin or start a transaction in read-only mode as follows: ```java // Begin a transaction in read-only mode. DistributedTransaction transaction = transactionManager.beginReadOnly(); ``` ```java // Start a transaction in read-only mode. DistributedTransaction transaction = transactionManager.startReadOnly(); ``` Alternatively, you can use the `beginReadOnly` and `startReadOnly` methods by specifying a transaction ID as follows: ```java // Begin a transaction in read-only mode by specifying a transaction ID. DistributedTransaction transaction = transactionManager.beginReadOnly(""); ``` ```java // Start a transaction in read-only mode by specifying a transaction ID. DistributedTransaction transaction = transactionManager.startReadOnly(""); ``` :::note Specifying a transaction ID is useful when you want to link external systems to ScalarDB. Otherwise, you should use the `beginReadOnly()` method or the `startReadOnly()` method. When you specify a transaction ID, make sure you specify a unique ID (for example, UUID v4) throughout the system since ScalarDB depends on the uniqueness of transaction IDs for correctness. ::: ##### Begin or start a transaction with attributes You can specify a map of transaction-scoped attributes when beginning or starting a transaction. The specified attributes are merged into every operation issued within the transaction, so configuration that should apply uniformly across the whole transaction can be specified once at begin time. If an operation already has the same attribute key, the attribute on the operation takes precedence. You can begin or start a transaction with attributes as follows: ```java // Specify attributes for the transaction. Map attributes = ...; // Begin a transaction with attributes. DistributedTransaction transaction = transactionManager.begin(attributes); ``` ```java // Start a transaction with attributes. DistributedTransaction transaction = transactionManager.start(attributes); ``` You can also specify a transaction ID together with attributes as follows: ```java // Begin a transaction with attributes by specifying a transaction ID. DistributedTransaction transaction = transactionManager.begin("", attributes); ``` ```java // Start a transaction with attributes by specifying a transaction ID. DistributedTransaction transaction = transactionManager.start("", attributes); ``` The `beginReadOnly` and `startReadOnly` methods also have overloads that accept attributes: ```java // Begin a transaction in read-only mode with attributes. DistributedTransaction transaction = transactionManager.beginReadOnly(attributes); ``` ```java // Start a transaction in read-only mode with attributes. DistributedTransaction transaction = transactionManager.startReadOnly(attributes); ``` ```java // Begin a transaction in read-only mode with attributes by specifying a transaction ID. DistributedTransaction transaction = transactionManager.beginReadOnly("", attributes); ``` ```java // Start a transaction in read-only mode with attributes by specifying a transaction ID. DistributedTransaction transaction = transactionManager.startReadOnly("", attributes); ``` For the list of available attributes, see [Operation attributes](#operation-attributes). #### Join a transaction Joining a transaction is particularly useful in a stateful application where a transaction spans multiple client requests. In such a scenario, the application can start a transaction during the first client request. Then, in subsequent client requests, the application can join the ongoing transaction by using the `join()` method. You can join an ongoing transaction that has already begun by specifying the transaction ID as follows: ```java // Join a transaction. DistributedTransaction transaction = transactionManager.join(""); ``` :::note To get the transaction ID with `getId()`, you can specify the following: ```java tx.getId(); ``` ::: #### Resume a transaction Resuming a transaction is particularly useful in a stateful application where a transaction spans multiple client requests. In such a scenario, the application can start a transaction during the first client request. Then, in subsequent client requests, the application can resume the ongoing transaction by using the `resume()` method. You can resume an ongoing transaction that you have already begun by specifying a transaction ID as follows: ```java // Resume a transaction. DistributedTransaction transaction = transactionManager.resume(""); ``` :::note To get the transaction ID with `getId()`, you can specify the following: ```java tx.getId(); ``` ::: #### Implement CRUD operations The following sections describe key construction and CRUD operations. :::note Although all the builders of the CRUD operations can specify consistency by using the `consistency()` methods, those methods are ignored. Instead, the `LINEARIZABLE` consistency level is always used in transactions. ::: ##### Key construction Most CRUD operations need to specify `Key` objects (partition-key, clustering-key, etc.). So, before moving on to CRUD operations, the following explains how to construct a `Key` object. For a single column key, you can use `Key.of()` methods to construct the key as follows: ```java // For a key that consists of a single column of INT. Key key1 = Key.ofInt("col1", 1); // For a key that consists of a single column of BIGINT. Key key2 = Key.ofBigInt("col1", 100L); // For a key that consists of a single column of DOUBLE. Key key3 = Key.ofDouble("col1", 1.3d); // For a key that consists of a single column of TEXT. Key key4 = Key.ofText("col1", "value"); ``` For a key that consists of two to five columns, you can use the `Key.of()` method to construct the key as follows. Similar to `ImmutableMap.of()` in Guava, you need to specify column names and values in turns: ```java // For a key that consists of two to five columns. Key key1 = Key.of("col1", 1, "col2", 100L); Key key2 = Key.of("col1", 1, "col2", 100L, "col3", 1.3d); Key key3 = Key.of("col1", 1, "col2", 100L, "col3", 1.3d, "col4", "value"); Key key4 = Key.of("col1", 1, "col2", 100L, "col3", 1.3d, "col4", "value", "col5", false); ``` For a key that consists of more than five columns, we can use the builder to construct the key as follows: ```java // For a key that consists of more than five columns. Key key = Key.newBuilder() .addInt("col1", 1) .addBigInt("col2", 100L) .addDouble("col3", 1.3d) .addText("col4", "value") .addBoolean("col5", false) .addInt("col6", 100) .build(); ``` ##### `Get` operation `Get` is an operation to retrieve a single record specified by a primary key. You need to create a `Get` object first, and then you can execute the object by using the `transaction.get()` method as follows: ```java // Create a `Get` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Get get = Get.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .projections("c1", "c2", "c3", "c4") .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) .build(); // Execute the `Get` operation. Optional result = transaction.get(get); ``` You can specify projections to choose which columns are returned. ###### Use the `WHERE` clause You can also specify arbitrary conditions by using the `where()` method. If the retrieved record does not match the conditions specified by the `where()` method, `Option.empty()` will be returned. As an argument of the `where()` method, you can specify a condition, an AND-wise condition set, or an OR-wise condition set. After calling the `where()` method, you can add more conditions or condition sets by using the `and()` method or `or()` method as follows: ```java // Create a `Get` operation with condition sets. Get get = Get.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .where( ConditionSetBuilder.condition(ConditionBuilder.column("c1").isLessThanInt(10)) .or(ConditionBuilder.column("c1").isGreaterThanInt(20)) .build()) .and( ConditionSetBuilder.condition(ConditionBuilder.column("c2").isLikeText("a%")) .or(ConditionBuilder.column("c2").isLikeText("b%")) .build()) .build(); ``` :::note In the `where()` condition method chain, the conditions must be an AND-wise junction of `ConditionalExpression` or `OrConditionSet` (known as conjunctive normal form) like the above example or an OR-wise junction of `ConditionalExpression` or `AndConditionSet` (known as disjunctive normal form). ::: For more details about available conditions and condition sets, see the and pages in the Javadoc. ###### Handle `Result` objects The `Get` operation and `Scan` operation return `Result` objects. The following shows how to handle `Result` objects. You can get a column value of a result by using `get("")` methods as follows: ```java // Get the BOOLEAN value of a column. boolean booleanValue = result.getBoolean(""); // Get the INT value of a column. int intValue = result.getInt(""); // Get the BIGINT value of a column. long bigIntValue = result.getBigInt(""); // Get the FLOAT value of a column. float floatValue = result.getFloat(""); // Get the DOUBLE value of a column. double doubleValue = result.getDouble(""); // Get the TEXT value of a column. String textValue = result.getText(""); // Get the BLOB value of a column as a `ByteBuffer`. ByteBuffer blobValue = result.getBlob(""); // Get the BLOB value of a column as a `byte` array. byte[] blobValueAsBytes = result.getBlobAsBytes(""); // Get the DATE value of a column as a `LocalDate`. LocalDate dateValue = result.getDate(""); // Get the TIME value of a column as a `LocalTime`. LocalTime timeValue = result.getTime(""); // Get the TIMESTAMP value of a column as a `LocalDateTime`. LocalDateTime timestampValue = result.getTimestamp(""); // Get the TIMESTAMPTZ value of a column as a `Instant`. Instant timestampTZValue = result.getTimestampTZ(""); ``` And if you need to check if a value of a column is null, you can use the `isNull("")` method. ``` java // Check if a value of a column is null. boolean isNull = result.isNull(""); ``` For more details, see the page in the Javadoc. ###### Execute `Get` by using a secondary index You can execute a `Get` operation by using a secondary index. Instead of specifying a partition key, you can specify an index key (indexed column) to use a secondary index as follows: ```java // Create a `Get` operation by using a secondary index. Key indexKey = Key.ofFloat("c4", 1.23F); Get get = Get.newBuilder() .namespace("ns") .table("tbl") .indexKey(indexKey) .projections("c1", "c2", "c3", "c4") .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) .build(); // Execute the `Get` operation. Optional result = transaction.get(get); ``` You can also specify arbitrary conditions by using the `where()` method. For details, see [Use the `WHERE` clause](#use-the-where-clause). :::note If the result has more than one record, `transaction.get()` will throw an exception. If you want to handle multiple results, see [Execute `Scan` by using a secondary index](#execute-scan-by-using-a-secondary-index). ::: ##### `Scan` operation `Scan` is an operation to retrieve multiple records within a partition. You can specify clustering-key boundaries and orderings for clustering-key columns in `Scan` operations. To execute a `Scan` operation, you can use the `transaction.scan()` method or the `transaction.getScanner()` method: - `transaction.scan()`: - This method immediately executes the given `Scan` operation and returns a list of all matching records. It is suitable when the result set is expected to be small enough to fit in memory. - `transaction.getScanner()`: - This method returns a `Scanner` object that allows you to iterate over the result set lazily. It is useful when the result set may be large, as it avoids loading all records into memory at once. You need to create a `Scan` object first, and then you can execute the object by using the `transaction.scan()` method or the `transaction.getScanner()` method as follows: ```java // Create a `Scan` operation. Key partitionKey = Key.ofInt("c1", 10); Key startClusteringKey = Key.of("c2", "aaa", "c3", 100L); Key endClusteringKey = Key.of("c2", "aaa", "c3", 300L); Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .start(startClusteringKey, true) // Include startClusteringKey .end(endClusteringKey, false) // Exclude endClusteringKey .projections("c1", "c2", "c3", "c4") .orderings(Scan.Ordering.desc("c2"), Scan.Ordering.asc("c3")) .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) .limit(10) .build(); // Execute the `Scan` operation by using the `transaction.scan()` method. List results = transaction.scan(scan); // Or, execute the `Scan` operation by using the `transaction.getScanner()` method. try (TransactionCrudOperable.Scanner scanner = transaction.getScanner(scan)) { // Fetch the next result from the scanner Optional result = scanner.one(); // Fetch all remaining results from the scanner List allResults = scanner.all(); } ``` You can omit the clustering-key boundaries or specify either a `start` boundary or an `end` boundary. If you don't specify `orderings`, you will get results ordered by the clustering order that you defined when creating the table. In addition, you can specify `projections` to choose which columns are returned and use `limit` to specify the number of records to return in `Scan` operations. ###### Use the `WHERE` clause You can also specify arbitrary conditions by using the `where()` method to filter scanned records. As an argument of the `where()` method, you can specify a condition, an AND-wise condition set, or an OR-wise condition set. After calling the `where()` method, you can add more conditions or condition sets by using the `and()` method or `or()` method as follows: ```java // Create a `Scan` operation with condition sets. Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .all() .where( ConditionSetBuilder.condition(ConditionBuilder.column("c1").isLessThanInt(10)) .or(ConditionBuilder.column("c1").isGreaterThanInt(20)) .build()) .and( ConditionSetBuilder.condition(ConditionBuilder.column("c2").isLikeText("a%")) .or(ConditionBuilder.column("c2").isLikeText("b%")) .build()) .limit(10) .build(); ``` :::note In the `where()` condition method chain, the conditions must be an AND-wise junction of `ConditionalExpression` or `OrConditionSet` (known as conjunctive normal form) like the above example or an OR-wise junction of `ConditionalExpression` or `AndConditionSet` (known as disjunctive normal form). ::: For more details about available conditions and condition sets, see the and pages in the Javadoc. ###### Execute `Scan` by using a secondary index You can execute a `Scan` operation by using a secondary index. Instead of specifying a partition key, you can specify an index key (indexed column) to use a secondary index as follows: ```java // Create a `Scan` operation by using a secondary index. Key indexKey = Key.ofFloat("c4", 1.23F); Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .indexKey(indexKey) .projections("c1", "c2", "c3", "c4") .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) .limit(10) .build(); // Execute the `Scan` operation. List results = transaction.scan(scan); ``` You can also specify arbitrary conditions using the `where()` method. For details, see [Use the `WHERE` clause](#use-the-where-clause-1). :::note You can't specify clustering-key boundaries and orderings in `Scan` by using a secondary index. ::: ###### Execute cross-partition `Scan` without specifying a partition key to retrieve all the records of a table You can execute a `Scan` operation across all partitions, which we call *cross-partition scan*, without specifying a partition key by enabling the following configuration in the ScalarDB properties file. ```properties scalar.db.cross_partition_scan.enabled=true ``` :::warning For non-JDBC databases, transactions could be executed at read-committed snapshot isolation (`SNAPSHOT`), which is a lower isolation level, even if you enable cross-partition scan with the `SERIALIZABLE` isolation level. When using non-JDBC databases, use cross-partition scan only if consistency does not matter for your transactions. ::: Instead of calling the `partitionKey()` method in the builder, you can call the `all()` method to scan a table without specifying a partition key as follows: ```java // Create a `Scan` operation without specifying a partition key. Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .all() .projections("c1", "c2", "c3", "c4") .limit(10) .build(); // Execute the `Scan` operation. List results = transaction.scan(scan); ``` :::note You can't specify any orderings in cross-partition `Scan` when using non-JDBC databases. For details on how to use cross-partition `Scan` with filtering or ordering, see [Execute cross-partition `Scan` with filtering and ordering](#execute-cross-partition-scan-with-filtering-and-ordering). ::: ###### Execute cross-partition `Scan` with filtering and ordering By enabling the cross-partition scan option with filtering and ordering as follows, you can execute a cross-partition `Scan` operation with flexible conditions and orderings: ```properties scalar.db.cross_partition_scan.enabled=true scalar.db.cross_partition_scan.filtering.enabled=true scalar.db.cross_partition_scan.ordering.enabled=true ``` :::note You can't enable `scalar.db.cross_partition_scan.ordering` in non-JDBC databases. ::: You can call the `where()` and `ordering()` methods after calling the `all()` method to specify arbitrary conditions and orderings as follows: ```java // Create a `Scan` operation with arbitrary conditions and orderings. Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .all() .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) .projections("c1", "c2", "c3", "c4") .orderings(Scan.Ordering.desc("c3"), Scan.Ordering.asc("c4")) .limit(10) .build(); // Execute the `Scan` operation. List results = transaction.scan(scan); ``` For details about the `WHERE` clause, see [Use the `WHERE` clause](#use-the-where-clause-1). ###### Enable cross-partition scan per operation Instead of enabling cross-partition scan globally through the configurations above, you can enable it for a specific operation by setting cross-partition scan attributes on the operation. When set on the operation, the attribute takes precedence over the corresponding configuration value for that operation. The following example enables cross-partition scan, filtering, and ordering for a specific operation: ```java // Enable cross-partition scan for this specific `Scan` operation. Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .all() .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) .orderings(Scan.Ordering.asc("c3")) .attribute(DatabaseOperationAttributes.CROSS_PARTITION_SCAN_ENABLED, "true") .attribute(DatabaseOperationAttributes.CROSS_PARTITION_SCAN_FILTERING_ENABLED, "true") .attribute(DatabaseOperationAttributes.CROSS_PARTITION_SCAN_ORDERING_ENABLED, "true") .build(); ``` For details on these attributes, see [Cross-partition scan attributes](#cross-partition-scan-attributes). ##### `Put` operation :::note The `Put` operation is deprecated as of ScalarDB 3.13 and will be removed in a future release. Instead of using the `Put` operation, use the `Insert` operation, the `Upsert` operation, or the `Update` operation. ::: `Put` is an operation to put a record specified by a primary key. The operation behaves as an upsert operation for a record, in which the operation updates the record if the record exists or inserts the record if the record does not exist. :::note When you update an existing record, you need to read the record by using `Get` or `Scan` before using a `Put` operation. Otherwise, the operation will fail due to a conflict. This occurs because of the specification of ScalarDB to manage transactions properly. Instead of reading the record explicitly, you can enable implicit pre-read. For details, see [Enable implicit pre-read for `Put` operations](#enable-implicit-pre-read-for-put-operations). ::: You need to create a `Put` object first, and then you can execute the object by using the `transaction.put()` method as follows: ```java // Create a `Put` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); // Execute the `Put` operation. transaction.put(put); ``` You can also put a record with `null` values as follows: ```java Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", null) .doubleValue("c5", null) .build(); ``` ###### Enable implicit pre-read for `Put` operations In Consensus Commit, an application must read a record before mutating the record with `Put` and `Delete` operations to obtain the latest states of the record if the record exists. Instead of reading the record explicitly, you can enable *implicit pre-read*. By enabling implicit pre-read, if an application does not read the record explicitly in a transaction, ScalarDB will read the record on behalf of the application before committing the transaction. You can enable implicit pre-read for a `Put` operation by specifying `enableImplicitPreRead()` in the `Put` operation builder as follows: ```java Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .enableImplicitPreRead() .build(); ``` :::note If you are certain that a record you are trying to mutate does not exist, you should not enable implicit pre-read for the `Put` operation for better performance. For example, if you load initial data, you should not enable implicit pre-read. A `Put` operation without implicit pre-read is faster than `Put` operation with implicit pre-read because the operation skips an unnecessary read. ::: ##### `Insert` operation `Insert` is an operation to insert an entry into the underlying storage through a transaction. If the entry already exists, a conflict error will occur. You need to create an `Insert` object first, and then you can execute the object by using the `transaction.insert()` method as follows: ```java // Create an `Insert` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Insert insert = Insert.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); // Execute the `Insert` operation. transaction.insert(insert); ``` ##### `Upsert` operation `Upsert` is an operation to insert an entry into or update an entry in the underlying storage through a transaction. If the entry already exists, it will be updated; otherwise, the entry will be inserted. You need to create an `Upsert` object first, and then you can execute the object by using the `transaction.upsert()` method as follows: ```java // Create an `Upsert` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Upsert upsert = Upsert.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); // Execute the `Upsert` operation. transaction.upsert(upsert); ``` ##### `Update` operation `Update` is an operation to update an entry in the underlying storage through a transaction. If the entry does not exist, the operation will not make any changes. You need to create an `Update` object first, and then you can execute the object by using the `transaction.update()` method as follows: ```java // Create an `Update` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Update update = Update.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); // Execute the `Update` operation. transaction.update(update); ``` ##### `Delete` operation `Delete` is an operation to delete a record specified by a primary key. :::note When you delete a record, you don't have to read the record beforehand because implicit pre-read is always enabled for `Delete` operations. ::: You need to create a `Delete` object first, and then you can execute the object by using the `transaction.delete()` method as follows: ```java // Create a `Delete` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .build(); // Execute the `Delete` operation. transaction.delete(delete); ``` ##### `Put`, `Delete`, and `Update` with a condition You can write arbitrary conditions (for example, a bank account balance must be equal to or more than zero) that you require a transaction to meet before being committed by implementing logic that checks the conditions in the transaction. Alternatively, you can write simple conditions in a mutation operation, such as `Put`, `Delete`, and `Update`. When a `Put`, `Delete`, or `Update` operation includes a condition, the operation is executed only if the specified condition is met. If the condition is not met when the operation is executed, an exception called `UnsatisfiedConditionException` will be thrown. :::note When you specify a condition in a `Put` operation, you need to read the record beforehand or enable implicit pre-read. ::: ###### Conditions for `Put` You can specify a condition in a `Put` operation as follows: ```java // Build a condition. MutationCondition condition = ConditionBuilder.putIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) .build(); Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .condition(condition) // condition .build(); // Execute the `Put` operation. transaction.put(put); ``` In addition to using the `putIf` condition, you can specify the `putIfExists` and `putIfNotExists` conditions as follows: ```java // Build a `putIfExists` condition. MutationCondition putIfExistsCondition = ConditionBuilder.putIfExists(); // Build a `putIfNotExists` condition. MutationCondition putIfNotExistsCondition = ConditionBuilder.putIfNotExists(); ``` ###### Conditions for `Delete` You can specify a condition in a `Delete` operation as follows: ```java // Build a condition. MutationCondition condition = ConditionBuilder.deleteIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) .build(); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .condition(condition) // condition .build(); // Execute the `Delete` operation. transaction.delete(delete); ``` In addition to using the `deleteIf` condition, you can specify the `deleteIfExists` condition as follows: ```java // Build a `deleteIfExists` condition. MutationCondition deleteIfExistsCondition = ConditionBuilder.deleteIfExists(); ``` ###### Conditions for `Update` You can specify a condition in an `Update` operation as follows: ```java // Build a condition. MutationCondition condition = ConditionBuilder.updateIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) .build(); Update update = Update.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .condition(condition) // condition .build(); // Execute the `Update` operation. transaction.update(update); ``` In addition to using the `updateIf` condition, you can specify the `updateIfExists` condition as follows: ```java // Build a `updateIfExists` condition. MutationCondition updateIfExistsCondition = ConditionBuilder.updateIfExists(); ``` ##### Mutate operation Mutate is an operation to execute multiple mutations for `Put`, `Insert`, `Upsert`, `Update`, and `Delete`. You need to create mutation objects first, and then you can execute the mutations by using the `transaction.mutate()` method as follows: ```java // Create `Put` and `Delete` operations. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKeyForPut = Key.of("c2", "aaa", "c3", 100L); Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForPut) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); Key clusteringKeyForDelete = Key.of("c2", "bbb", "c3", 200L); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForDelete) .build(); // Execute the mutations. transaction.mutate(Arrays.asList(put, delete)); ``` ##### Batch operation Batch is an operation to execute multiple operations for `Get`, `Scan`, `Put`, `Insert`, `Upsert`, `Update`, and `Delete`. You need to create operation objects first, and then you can execute the operations by using the `transaction.batch()` method as follows: ```java // Create operation objects. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKeyForGet = Key.of("c2", "aaa", "c3", 100L); Get get = Get.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForGet) .build(); Scan scan = Scan.newBuilder().namespace("ns").table("tbl").partitionKey(partitionKey).build(); Key clusteringKeyForInsert = Key.of("c2", "bbb", "c3", 200L); Insert insert = Insert.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForInsert) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); Key clusteringKeyForDelete = Key.of("c2", "ccc", "c3", 300L); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForDelete) .build(); // Execute the operations. List batchResults = transaction.batch(Arrays.asList(get, scan, insert, delete)); Optional getResult = batchResults.get(0).getGetResult(); // Result of the get ... List scanResult = batchResults.get(1).getScanResult(); // Result of the scan ... ``` ##### Default namespace for CRUD operations A default namespace for all CRUD operations can be set by using a property in the ScalarDB configuration. ```properties scalar.db.default_namespace_name= ``` Any operation that does not specify a namespace will use the default namespace set in the configuration. ```java // This operation will target the default namespace. Scan scanUsingDefaultNamespace = Scan.newBuilder() .table("tbl") .all() .build(); // This operation will target the "ns" namespace. Scan scanUsingSpecifiedNamespace = Scan.newBuilder() .namespace("ns") .table("tbl") .all() .build(); ``` ##### Operation attributes An operation attribute is a key-value pair that can be used to store additional information about an operation. You can set operation attributes either on individual operations by using the `attribute()` or `attributes()` method in the operation builder as operation-scoped attributes, or at transaction begin time as transaction-scoped attributes that are merged into every operation in the transaction. If the same attribute is set at both levels, the operation-scoped value takes precedence over the transaction-scoped value. For details on transaction-scoped attributes, see [Begin or start a transaction with attributes](#begin-or-start-a-transaction-with-attributes). The following examples show how to set operation attributes on individual operations: ```java // Set operation attributes in the `Get` operation. Get get = Get.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .attribute("attribute1", "value1") .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) .build(); // Set operation attributes in the `Scan` operation. Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .projections("c1", "c2", "c3", "c4") .attribute("attribute1", "value1") .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) .build(); // Set operation attributes in the `Insert` operation. Insert insert = Insert.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .attribute("attribute1", "value1") .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) .build(); // Set operation attributes in the `Upsert` operation. Upsert upsert = Upsert.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .attribute("attribute1", "value1") .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) .build(); // Set operation attributes in the `Update` operation. Update update = Update.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .attribute("attribute1", "value1") .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) .build(); // Set operation attributes in the `Delete` operation. Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .attribute("attribute1", "value1") .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) .build(); ``` The following attributes are available. Setting an attribute on an operation that does not support it has no effect. ###### Cross-partition scan attributes The following attributes apply to a cross-partition `Scan` operation (a `Scan` operation without a partition key). When set on the operation, they take precedence over the corresponding `scalar.db.cross_partition_scan.*` configuration values for that operation. For details on the configurations, see [Cross-partition scan configurations](configurations.mdx#cross-partition-scan-configurations). - **`db-cross-partition-scan-enabled`:** Whether cross-partition scan is enabled for the operation. The value must be `true` or `false`. - **`db-cross-partition-scan-filtering-enabled`:** Whether cross-partition scan with filtering is enabled for the operation. The value must be `true` or `false`. - **`db-cross-partition-scan-ordering-enabled`:** Whether cross-partition scan with ordering is enabled for the operation. The value must be `true` or `false`. This attribute is available only for JDBC databases. You can use the utility class to set these attributes. ###### Consensus Commit attributes The following attribute applies to transactions executed under [Consensus Commit](consensus-commit.mdx). - **`cc-transaction-isolation`:** The isolation level for the transaction. The value must be `SNAPSHOT`, `SERIALIZABLE`, or `READ_COMMITTED`. When set, this overrides the configured default isolation level for the transaction. For details on isolation levels, see [Isolation levels](consensus-commit.mdx#isolation-levels). :::warning Unlike the other operation attributes, `cc-transaction-isolation` cannot be applied per operation within a transaction. The attribute is interpreted only at transaction begin time. To use it, either specify it when [beginning or starting a transaction with attributes](#begin-or-start-a-transaction-with-attributes), or, when [executing transactions without beginning or starting a transaction](#execute-transactions-without-beginning-or-starting-a-transaction), set it on the operation, which ScalarDB then uses for the transaction that it implicitly begins. ::: You can use the utility class to set this attribute. #### Commit a transaction After executing CRUD operations, you need to commit a transaction to finish it. You can commit a transaction as follows: ```java // Commit a transaction. transaction.commit(); ``` #### Roll back or abort a transaction If an error occurs when executing a transaction, you can roll back or abort the transaction. You can roll back a transaction as follows: ```java // Roll back a transaction. transaction.rollback(); ``` Or, you can abort a transaction as follows: ```java // Abort a transaction. transaction.abort(); ``` For details about how to handle exceptions in ScalarDB, see [How to handle exceptions](#how-to-handle-exceptions). ### Execute transactions without beginning or starting a transaction You can execute transactional operations without beginning or starting a transaction. In this case, ScalarDB will automatically begin a transaction before executing the operations and commit the transaction after executing the operations. This section explains how to execute transactions without beginning or starting a transaction. #### Execute `Get` operation `Get` is an operation to retrieve a single record specified by a primary key. You need to create a `Get` object first, and then you can execute the object by using the `transactionManager.get()` method as follows: ```java // Create a `Get` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Get get = Get.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .projections("c1", "c2", "c3", "c4") .build(); // Execute the `Get` operation. Optional result = transactionManager.get(get); ``` For details about the `Get` operation, see [`Get` operation](#get-operation). #### Execute `Scan` operation `Scan` is an operation to retrieve multiple records within a partition. You can specify clustering-key boundaries and orderings for clustering-key columns in `Scan` operations. To execute a `Scan` operation, you can use the `transactionManager.scan()` method or the `transactionManager.getScanner()` method: - `transactionManager.scan()`: - This method immediately executes the given `Scan` operation and returns a list of all matching records. It is suitable when the result set is expected to be small enough to fit in memory. - `transactionManager.getScanner()`: - This method returns a `Scanner` object that allows you to iterate over the result set lazily. It is useful when the result set may be large, as it avoids loading all records into memory at once. You need to create a `Scan` object first, and then you can execute the object by using the `transactionManager.scan()` method or the `transactionManager.getScanner()` method as follows: ```java // Create a `Scan` operation. Key partitionKey = Key.ofInt("c1", 10); Key startClusteringKey = Key.of("c2", "aaa", "c3", 100L); Key endClusteringKey = Key.of("c2", "aaa", "c3", 300L); Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .start(startClusteringKey, true) // Include startClusteringKey .end(endClusteringKey, false) // Exclude endClusteringKey .projections("c1", "c2", "c3", "c4") .orderings(Scan.Ordering.desc("c2"), Scan.Ordering.asc("c3")) .limit(10) .build(); // Execute the `Scan` operation by using the `transactionManager.scan()` method. List results = transactionManager.scan(scan); // Or, execute the `Scan` operation by using the `transactionManager.getScanner()` method. try (TransactionManagerCrudOperable.Scanner scanner = transactionManager.getScanner(scan)) { // Fetch the next result from the scanner Optional result = scanner.one(); // Fetch all remaining results from the scanner List allResults = scanner.all(); } ``` For details about the `Scan` operation, see [`Scan` operation](#scan-operation). #### Execute `Put` operation :::note The `Put` operation is deprecated as of ScalarDB 3.13 and will be removed in a future release. Instead of using the `Put` operation, use the `Insert` operation, the `Upsert` operation, or the `Update` operation. ::: You need to create a `Put` object first, and then you can execute the object by using the `transactionManager.put()` method as follows: ```java // Create a `Put` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); // Execute the `Put` operation. transactionManager.put(put); ``` For details about the `Put` operation, see [`Put` operation](#put-operation). #### Execute `Insert` operation `Insert` is an operation to insert an entry into the underlying storage through a transaction. If the entry already exists, a conflict error will occur. You need to create an `Insert` object first, and then you can execute the object by using the `transactionManager.insert()` method as follows: ```java // Create an `Insert` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Insert insert = Insert.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); // Execute the `Insert` operation. transactionManager.insert(insert); ``` For details about the `Insert` operation, see [`Insert` operation](#insert-operation). #### Execute `Upsert` operation `Upsert` is an operation to insert an entry into or update an entry in the underlying storage through a transaction. If the entry already exists, it will be updated; otherwise, the entry will be inserted. You need to create an `Upsert` object first, and then you can execute the object by using the `transactionManager.upsert()` method as follows: ```java // Create an `Upsert` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Upsert upsert = Upsert.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); // Execute the `Upsert` operation. transactionManager.upsert(upsert); ``` For details about the `Insert` operation, see [`Upsert` operation](#upsert-operation). #### Execute `Update` operation `Update` is an operation to update an entry in the underlying storage through a transaction. If the entry does not exist, the operation will not make any changes. You need to create an `Update` object first, and then you can execute the object by using the `transactionManager.update()` method as follows: ```java // Create an `Update` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Update update = Update.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); // Execute the `Update` operation. transactionManager.update(update); ``` For details about the `Update` operation, see [`Update` operation](#update-operation). #### Execute `Delete` operation `Delete` is an operation to delete a record specified by a primary key. You need to create a `Delete` object first, and then you can execute the object by using the `transaction.delete()` method as follows: ```java // Create a `Delete` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .build(); // Execute the `Delete` operation. transactionManager.delete(delete); ``` For details about the `Delete` operation, see [`Delete` operation](#delete-operation). #### Execute Mutate operation Mutate is an operation to execute multiple mutations (`Put`, `Insert`, `Upsert`, `Update`, and `Delete` operations). You need to create mutation objects first, and then you can execute the mutations by using the `transactionManager.mutate()` method as follows: ```java // Create `Put` and `Delete` operations. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKeyForPut = Key.of("c2", "aaa", "c3", 100L); Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForPut) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); Key clusteringKeyForDelete = Key.of("c2", "bbb", "c3", 200L); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForDelete) .build(); // Execute the mutations. transactionManager.mutate(Arrays.asList(put, delete)); ``` For details about the Mutate operation, see [Mutate operation](#mutate-operation). In addition, for details about how to handle exceptions in ScalarDB, see [How to handle exceptions](#how-to-handle-exceptions). #### Execute Batch operation Batch is an operation to execute multiple operations for `Get`, `Scan`, `Put`, `Insert`, `Upsert`, `Update`, and `Delete`. You need to create operation objects first, and then you can execute the operations by using the `transactionManager.batch()` method as follows: ```java // Create operation objects. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKeyForGet = Key.of("c2", "aaa", "c3", 100L); Get get = Get.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForGet) .build(); Scan scan = Scan.newBuilder().namespace("ns").table("tbl").partitionKey(partitionKey).build(); Key clusteringKeyForInsert = Key.of("c2", "bbb", "c3", 200L); Insert insert = Insert.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForInsert) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); Key clusteringKeyForDelete = Key.of("c2", "ccc", "c3", 300L); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForDelete) .build(); // Execute the operations. List batchResults = transactionManager.batch(Arrays.asList(get, scan, insert, delete)); Optional getResult = batchResults.get(0).getGetResult(); // Result of the get ... List scanResult = batchResults.get(1).getScanResult(); // Result of the scan ... ``` For details about the Batch operation, see [Batch operation](#batch-operation). ## How to handle exceptions When executing a transaction, you will also need to handle exceptions properly. :::warning If you don't handle exceptions properly, you may face anomalies or data inconsistency. ::: The following sample code shows how to handle exceptions: ```java public class Sample { public static void main(String[] args) throws Exception { TransactionFactory factory = TransactionFactory.create(""); DistributedTransactionManager transactionManager = factory.getTransactionManager(); int retryCount = 0; TransactionException lastException = null; while (true) { if (retryCount++ > 0) { // Retry the transaction three times maximum. if (retryCount >= 3) { // Throw the last exception if the number of retries exceeds the maximum. throw lastException; } // Sleep 100 milliseconds before retrying the transaction. TimeUnit.MILLISECONDS.sleep(100); } DistributedTransaction transaction = null; try { // Begin a transaction. transaction = transactionManager.begin(); // Execute CRUD operations in the transaction. Optional result = transaction.get(...); List results = transaction.scan(...); transaction.put(...); transaction.delete(...); // Commit the transaction. transaction.commit(); } catch (UnsatisfiedConditionException e) { // You need to handle `UnsatisfiedConditionException` only if a mutation operation specifies a condition. // This exception indicates the condition for the mutation operation is not met. try { transaction.rollback(); } catch (RollbackException ex) { // Rolling back the transaction failed. Since the transaction should eventually recover, // you don't need to do anything further. You can simply log the occurrence here. } // You can handle the exception here, according to your application requirements. return; } catch (UnknownTransactionStatusException e) { // If you catch `UnknownTransactionStatusException` when committing the transaction, // it indicates that the status of the transaction, whether it was successful or not, is unknown. // In such a case, you need to check if the transaction is committed successfully or not and // retry the transaction if it failed. How to identify a transaction status is delegated to users. return; } catch (TransactionException e) { // For other exceptions, you can try retrying the transaction. // For `CrudConflictException`, `CommitConflictException`, and `TransactionNotFoundException`, // you can basically retry the transaction. However, for the other exceptions, the transaction // will still fail if the cause of the exception is non-transient. In such a case, you will // exhaust the number of retries and throw the last exception. if (transaction != null) { try { transaction.rollback(); } catch (RollbackException ex) { // Rolling back the transaction failed. The transaction should eventually recover, // so you don't need to do anything further. You can simply log the occurrence here. } } lastException = e; } } } } ``` ### `TransactionException` and `TransactionNotFoundException` The `begin()` API could throw `TransactionException` or `TransactionNotFoundException`: - If you catch `TransactionException`, this exception indicates that the transaction has failed to begin due to transient or non-transient faults. You can try retrying the transaction, but you may not be able to begin the transaction due to non-transient faults. - If you catch `TransactionNotFoundException`, this exception indicates that the transaction has failed to begin due to transient faults. In this case, you can retry the transaction. The `join()` API could also throw `TransactionNotFoundException`. You can handle this exception in the same way that you handle the exceptions for the `begin()` API. ### `CrudException` and `CrudConflictException` The APIs for CRUD operations (`get()`, `scan()`, `put()`, `delete()`, and `mutate()`) could throw `CrudException` or `CrudConflictException`: - If you catch `CrudException`, this exception indicates that the transaction CRUD operation has failed due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is non-transient. - If you catch `CrudConflictException`, this exception indicates that the transaction CRUD operation has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. ### `UnsatisfiedConditionException` The APIs for mutation operations (`put()`, `delete()`, and `mutate()`) could also throw `UnsatisfiedConditionException`. If you catch `UnsatisfiedConditionException`, this exception indicates that the condition for the mutation operation is not met. You can handle this exception according to your application requirements. ### `CommitException`, `CommitConflictException`, and `UnknownTransactionStatusException` The `commit()` API could throw `CommitException`, `CommitConflictException`, or `UnknownTransactionStatusException`: - If you catch `CommitException`, this exception indicates that committing the transaction fails due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is non-transient. - If you catch `CommitConflictException`, this exception indicates that committing the transaction has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. - If you catch `UnknownTransactionStatusException`, this exception indicates that the status of the transaction, whether it was successful or not, is unknown. In this case, you need to check if the transaction is committed successfully and retry the transaction if it has failed. How to identify a transaction status is delegated to users. You may want to create a transaction status table and update it transactionally with other application data so that you can get the status of a transaction from the status table. ### Notes about some exceptions Although not illustrated in the sample code, the `resume()` API could also throw `TransactionNotFoundException`. This exception indicates that the transaction associated with the specified ID was not found and/or the transaction might have expired. In either case, you can retry the transaction from the beginning since the cause of this exception is basically transient. In the sample code, for `UnknownTransactionStatusException`, the transaction is not retried because the application must check if the transaction was successful to avoid potential duplicate operations. For other exceptions, the transaction is retried because the cause of the exception is transient or non-transient. If the cause of the exception is transient, the transaction may succeed if you retry it. However, if the cause of the exception is non-transient, the transaction will still fail even if you retry it. In such a case, you will exhaust the number of retries. :::note In the sample code, the transaction is retried three times maximum and sleeps for 100 milliseconds before it is retried. But you can choose a retry policy, such as exponential backoff, according to your application requirements. ::: ## Group commit for the Coordinator table The Coordinator table that is used for Consensus Commit transactions is a vital data store, and using robust storage for it is recommended. However, utilizing more robust storage options, such as internally leveraging multi-AZ or multi-region replication, may lead to increased latency when writing records to the storage, resulting in poor throughput performance. ScalarDB provides a group commit feature for the Coordinator table that groups multiple record writes into a single write operation, improving write throughput. In this case, latency may increase or decrease, depending on the underlying database and the workload. To enable the group commit feature, add the following configuration: ```properties # By default, this configuration is set to `false`. scalar.db.consensus_commit.coordinator.group_commit.enabled=true # These properties are for tuning the performance of the group commit feature. # scalar.db.consensus_commit.coordinator.group_commit.group_size_fix_timeout_millis=40 # scalar.db.consensus_commit.coordinator.group_commit.delayed_slot_move_timeout_millis=800 # scalar.db.consensus_commit.coordinator.group_commit.old_group_abort_timeout_millis=30000 # scalar.db.consensus_commit.coordinator.group_commit.timeout_check_interval_millis=10 # scalar.db.consensus_commit.coordinator.group_commit.metrics_monitor_log_enabled=true ``` ### Limitations This section describes the limitations of the group commit feature. #### Custom transaction ID passed by users The group commit feature implicitly generates an internal value and uses it as a part of transaction ID. Therefore, a custom transaction ID manually passed by users via `com.scalar.db.transaction.consensuscommit.ConsensusCommitManager.begin(String txId)` or `com.scalar.db.transaction.consensuscommit.TwoPhaseConsensusCommitManager.begin(String txId)` can't be used as is for later API calls. You need to use a transaction ID returned from`com.scalar.db.transaction.consensuscommit.ConsensusCommit.getId()` or `com.scalar.db.transaction.consensuscommit.TwoPhaseConsensusCommit.getId()` instead. ```java // This custom transaction ID needs to be used for ScalarDB transactions. String myTxId = UUID.randomUUID().toString(); ... DistributedTransaction transaction = manager.begin(myTxId); ... // When the group commit feature is enabled, a custom transaction ID passed by users can't be used as is. // logger.info("The transaction state: {}", manager.getState(myTxId)); logger.info("The transaction state: {}", manager.getState(transaction.getId())); ``` #### Prohibition of use with a two-phase commit interface The group commit feature manages all ongoing transactions in memory. If this feature is enabled with a two-phase commit interface, the information must be solely maintained by the coordinator service to prevent conflicts caused by participant services' inconsistent writes to the Coordinator table, which may contain different transaction distributions over groups. This limitation introduces some complexities and inflexibilities related to application development. Therefore, combining the use of the group commit feature with a two-phase commit interface is currently prohibited. ##### Enabling the feature on existing applications is not supported The group commit feature uses a new column in the Coordinator table. The current [Schema Loader](schema-loader.mdx), as of ScalarDB 3, doesn't support table schema migration for the Coordinator table. Therefore, enabling the group commit feature on existing applications where any transactions have been executed is not supported. To use this feature, you'll need to start your application in a clean state. Coordinator table schema migration in [Schema Loader](schema-loader.mdx) is expected to be supported in ScalarDB 4.0. ## Investigating Consensus Commit transaction manager errors To investigate errors when using the Consensus Commit transaction manager, you can enable a configuration that will return table metadata augmented with transaction metadata columns, which can be helpful when investigating transaction-related issues. This configuration, which is only available when troubleshooting the Consensus Commit transaction manager, enables you to see transaction metadata column details for a given table by using the `DistributedTransactionAdmin.getTableMetadata()` method. By adding the following configuration, `Get` and `Scan` operations results will contain [transaction metadata](schema-loader.mdx#internal-metadata-for-consensus-commit): ```properties # By default, this configuration is set to `false`. scalar.db.consensus_commit.include_metadata.enabled=true ``` ================================================ FILE: docs/backup-restore.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to Back Up and Restore Databases Used Through ScalarDB import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; Since ScalarDB provides transaction capabilities on top of non-transactional or transactional databases non-invasively, you need to take special care to back up and restore the databases in a transactionally consistent way. This guide describes how to back up and restore the databases that ScalarDB supports. ## Create a backup How you create a backup depends on which database you're using and whether or not you're using multiple databases. The following decision tree shows which approach you should take. ```mermaid flowchart TD A[Are you using a single database with ScalarDB?] A -->|Yes| B[Does the database have transaction support?] B -->|Yes| C[Perform back up without explicit pausing] B ---->|No| D[Perform back up with explicit pausing] A ---->|No| D ``` ### Back up without explicit pausing If you're using ScalarDB with a single database with support for transactions, you can create a backup of the database even while ScalarDB continues to accept transactions. :::warning Before creating a backup, you should consider the safest way to create a transactionally consistent backup of your databases and understand any risks that are associated with the backup process. ::: One requirement for creating a backup in ScalarDB is that backups for all the ScalarDB-managed tables (including the Coordinator table) need to be transactionally consistent or automatically recoverable to a transactionally consistent state. That means that you need to create a consistent backup by dumping all tables in a single transaction. How you create a transactionally consistent backup depends on the type of database that you're using. Select a database to see how to create a transactionally consistent backup for ScalarDB. :::note The backup methods by database listed below are just examples of some of the databases that ScalarDB supports. ::: You can restore to any point within the backup retention period by using the automated backup feature. Use the `mysqldump` command with the `--single-transaction` option. Use the `pg_dump` command. Use the `.backup` command with the `.timeout` command as specified in [Special commands to sqlite3 (dot-commands)](https://www.sqlite.org/cli.html#special_commands_to_sqlite3_dot_commands_) For an example, see [BASH: SQLite3 .backup command](https://stackoverflow.com/questions/23164445/bash-sqlite3-backup-command). Clusters are backed up automatically based on the backup policy, and these backups are retained for a specific duration. You can also perform on-demand backups. For details on performing backups, see [YugabyteDB Managed: Back up and restore clusters](https://docs.yugabyte.com/preview/yugabyte-cloud/cloud-clusters/backup-clusters/). Use the `backup` command. For details, on performing backups, see [Db2: Backup overview](https://www.ibm.com/docs/en/db2/12.1.0?topic=recovery-backup). ### Back up with explicit pausing Another way to create a transactionally consistent backup is to create a backup while a cluster of ScalarDB instances does not have any outstanding transactions. Creating the backup depends on the following: - If the underlying database has a point-in-time snapshot or backup feature, you can create a backup during the period when no outstanding transactions exist. - If the underlying database has a point-in-time restore or recovery (PITR) feature, you can set a restore point to a time (preferably the mid-time) in the pause duration period when no outstanding transactions exist. :::note When using a PITR feature, you should minimize the clock drifts between clients and servers by using clock synchronization, such as NTP. Otherwise, the time you get as the paused duration might be too different from the time in which the pause was actually conducted, which could restore the backup to a point where ongoing transactions exist. In addition, you should pause for a sufficient amount of time (for example, five seconds) and use the mid-time of the paused duration as a restore point since clock synchronization cannot perfectly synchronize clocks between nodes. ::: To make ScalarDB drain outstanding requests and stop accepting new requests so that a pause duration can be created, you should implement the [Scalar Admin](https://github.com/scalar-labs/scalar-admin) interface properly in your application that uses ScalarDB or use [ScalarDB Cluster](scalardb-cluster/index.mdx), which implements the Scalar Admin interface. By using the [Scalar Admin client tool](https://github.com/scalar-labs/scalar-admin/blob/main/README.md#scalar-admin-client-tool), you can pause nodes, servers, or applications that implement the Scalar Admin interface without losing ongoing transactions. How you create a transactionally consistent backup depends on the type of database that you're using. Select a database to see how to create a transactionally consistent backup for ScalarDB. :::note The backup methods by database listed below are just examples of some of the databases that ScalarDB supports. ::: You must enable the PITR feature for DynamoDB tables. If you're using [ScalarDB Schema Loader](schema-loader.mdx) to create schemas, the tool enables the PITR feature for tables by default. To specify a transactionally consistent restore point, pause your ScalarDB Cluster or your application that is using ScalarDB as described in [Back up with explicit pausing](#back-up-with-explicit-pausing). You must create a Cosmos DB for NoSQL account with a continuous backup policy that has the PITR feature enabled. After enabling the feature, backups are created continuously. To specify a transactionally consistent restore point, pause your ScalarDB Cluster or your application that is using ScalarDB as described in [Back up with explicit pausing](#back-up-with-explicit-pausing). Cassandra has a built-in replication feature, so you do not always have to create a transactionally consistent backup. For example, if the replication factor is set to `3` and only the data of one of the nodes in a Cassandra cluster is lost, you won't need a transactionally consistent backup (snapshot) because the node can be recovered by using a normal, transactionally inconsistent backup (snapshot) and the repair feature. However, if the quorum of cluster nodes loses their data, you will need a transactionally consistent backup (snapshot) to restore the cluster to a certain transactionally consistent point. To create a transactionally consistent cluster-wide backup (snapshot), pause your ScalarDB Cluster or your application that is using ScalarDB and create backups (snapshots) of the nodes as described in [Back up with explicit pausing](#back-up-with-explicit-pausing) or stop the Cassandra cluster, take copies of all the data in the nodes, and start the cluster. You can perform on-demand backups or scheduled backups during a paused duration. For details on performing backups, see [YugabyteDB Managed: Back up and restore clusters](https://docs.yugabyte.com/preview/yugabyte-cloud/cloud-clusters/backup-clusters/). You can create a backup of your bucket by using AWS Backup during a pause duration. For details, see [Amazon S3 backups](https://docs.aws.amazon.com/aws-backup/latest/devguide/s3-backups.html). To specify a transactionally consistent restore point, pause your ScalarDB Cluster or your application that is using ScalarDB as described in [Back up with explicit pausing](#back-up-with-explicit-pausing). You can create a backup of your container by using Azure Backup during a pause duration. For details, see [Overview of Azure Blob backup](https://learn.microsoft.com/en-us/azure/backup/blob-backup-overview). To specify a transactionally consistent restore point, pause your ScalarDB Cluster or your application that is using ScalarDB as described in [Back up with explicit pausing](#back-up-with-explicit-pausing). You can create a copy of your bucket, for example by using [Storage Transfer Service](https://docs.cloud.google.com/storage-transfer/docs/overview). For details, see [Transfer between Cloud Storage buckets](https://docs.cloud.google.com/storage-transfer/docs/cloud-storage-to-cloud-storage). To specify a transactionally consistent restore point, pause your ScalarDB Cluster or your application that is using ScalarDB as described in [Back up with explicit pausing](#back-up-with-explicit-pausing). ## Restore a backup How you restore a transactionally consistent backup depends on the type of database that you're using. Select a database to see how to create a transactionally consistent backup for ScalarDB. :::note The restore methods by database listed below are just examples of some of the databases that ScalarDB supports. ::: You can restore to any point within the backup retention period by using the automated backup feature. First, stop all the nodes of the Cassandra cluster. Then, clean the `data`, `commitlog`, and `hints` directories, and place the backups (snapshots) in each node. After placing the backups (snapshots) in each node, start all the nodes of the Cassandra Cluster. Follow the official Azure documentation for [restore an account by using Azure portal](https://docs.microsoft.com/en-us/azure/cosmos-db/restore-account-continuous-backup#restore-account-portal). After restoring a backup, [configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level) of the restored databases to `STRONG`. In addition, you should use the mid-time of the paused duration as the restore point as previously explained. ScalarDB implements the Cosmos DB adapter by using its stored procedures, which are installed when creating schemas by using ScalarDB Schema Loader. However, the PITR feature of Cosmos DB doesn't restore stored procedures. Because of this, you need to re-install the required stored procedures for all tables after restoration. You can do this by using ScalarDB Schema Loader with the `--repair-all` option. For details, see [Repair tables](schema-loader.mdx#repair-tables). Follow the official AWS documentation for [restoring a DynamoDB table to a point in time](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/PointInTimeRecovery.Tutorial.html), but keep in mind that a table can only be restored with an alias. Because of this, you will need to restore the table with an alias, delete the original table, and rename the alias to the original name to restore the tables with the same name. To do this procedure: 1. Create a backup. 1. Select the mid-time of the paused duration as the restore point. 2. Restore by using the PITR of table A to table B. 3. Create a backup of the restored table B (assuming that the backup is named backup B). 4. Remove table B. 2. Restore the backup. 1. Remove table A. 2. Create a table named A by using backup B. :::note * You must do the steps mentioned above for each table because tables can only be restored one at a time. * Configurations such as PITR and auto-scaling policies are reset to the default values for restored tables, so you must manually configure the required settings. For details, see the official AWS documentation for [How to restore DynamoDB tables with DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/CreateBackup.html#CreateBackup_HowItWorks-restore). ::: If you used `mysqldump` to create the backup file, use the `mysql` command to restore the backup as specified in [Reloading SQL-Format Backups](https://dev.mysql.com/doc/mysql-backup-excerpt/8.0/en/reloading-sql-format-dumps.html). If you used `pg_dump` to create the backup file, use the `psql` command to restore the backup as specified in [Restoring the Dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE). Use the `.restore` command as specified in [Special commands to sqlite3 (dot-commands)](https://www.sqlite.org/cli.html#special_commands_to_sqlite3_dot_commands_). You can restore from the scheduled or on-demand backup within the backup retention period. For details on performing backups, see [YugabyteDB Managed: Back up and restore clusters](https://docs.yugabyte.com/preview/yugabyte-cloud/cloud-clusters/backup-clusters/). Use the `restore` command. For details, on restoring the database, see [Db2: Restore overview](https://www.ibm.com/docs/en/db2/12.1.0?topic=recovery-restore). If you used AWS Backup to create a backup of your bucket, follow [Restore S3 data using AWS Backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/restoring-s3.html). When you restore the backup to another bucket, you need to update the value of the `scalar.db.contact_points` property in the ScalarDB configuration to point to the new bucket. If you used Azure Backup to create a backup of your container, follow [Restore Azure Blobs](https://learn.microsoft.com/en-us/azure/backup/blob-restore?tabs=operational-backup). When you restore the backup to another container, you need to update the value of the `scalar.db.contact_points` property in the ScalarDB configuration to point to the new container. Use a copied bucket as a new bucket by updating the value of the `scalar.db.contact_points` property in the ScalarDB configuration. ================================================ FILE: docs/configurations.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Core Configurations import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; This page describes the available configurations for ScalarDB Core. :::tip If you are using ScalarDB Cluster, please refer to [ScalarDB Cluster Configurations](./scalardb-cluster/scalardb-cluster-configurations.mdx) instead. ::: ## General configurations The following configurations are available for the Consensus Commit transaction manager. ### `transaction_manager` - **Field:** `scalar.db.transaction_manager` - **Description:** Transaction manager of ScalarDB. Specify `consensus-commit` to use [Consensus Commit](./consensus-commit.mdx) or `single-crud-operation` to [run non-transactional storage operations](./run-non-transactional-storage-operations-through-library.mdx). Note that the configurations under the `scalar.db.consensus_commit` prefix are ignored if you use `single-crud-operation`. - **Default value:** `consensus-commit` ### `isolation_level` - **Field:** `scalar.db.consensus_commit.isolation_level` - **Description:** Isolation level used for Consensus Commit. Either `SNAPSHOT`, `SERIALIZABLE`, or `READ_COMMITTED` can be specified. - **Default value:** `SNAPSHOT` ### `coordinator.namespace` - **Field:** `scalar.db.consensus_commit.coordinator.namespace` - **Description:** Namespace name of Coordinator tables used for Consensus Commit. - **Default value:** `coordinator` ## Performance-related configurations The following performance-related configurations are available for the Consensus Commit transaction manager. ### `parallel_executor_count` - **Field:** `scalar.db.consensus_commit.parallel_executor_count` - **Description:** Number of executors (threads) for parallel execution. This number refers to the total number of threads across transactions in a ScalarDB Cluster node or a ScalarDB Core process. - **Default value:** `128` ### `parallel_preparation.enabled` - **Field:** `scalar.db.consensus_commit.parallel_preparation.enabled` - **Description:** Whether or not the preparation phase is executed in parallel. - **Default value:** `true` ### `parallel_validation.enabled` - **Field:** `scalar.db.consensus_commit.parallel_validation.enabled` - **Description:** Whether or not the validation phase (in `EXTRA_READ`) is executed in parallel. - **Default value:** The value of `scalar.db.consensus_commit.parallel_commit.enabled` ### `parallel_commit.enabled` - **Field:** `scalar.db.consensus_commit.parallel_commit.enabled` - **Description:** Whether or not the commit phase is executed in parallel. - **Default value:** `true` ### `parallel_rollback.enabled` - **Field:** `scalar.db.consensus_commit.parallel_rollback.enabled` - **Description:** Whether or not the rollback phase is executed in parallel. - **Default value:** The value of `scalar.db.consensus_commit.parallel_commit.enabled` ### `async_commit.enabled` - **Field:** `scalar.db.consensus_commit.async_commit.enabled` - **Description:** Whether or not the commit phase is executed asynchronously. - **Default value:** `false` ### `async_rollback.enabled` - **Field:** `scalar.db.consensus_commit.async_rollback.enabled` - **Description:** Whether or not the rollback phase is executed asynchronously. - **Default value:** The value of `scalar.db.consensus_commit.async_commit.enabled` ### `parallel_implicit_pre_read.enabled` - **Field:** `scalar.db.consensus_commit.parallel_implicit_pre_read.enabled` - **Description:** Whether or not implicit pre-read is executed in parallel. - **Default value:** `true` ### `one_phase_commit.enabled` - **Field:** `scalar.db.consensus_commit.one_phase_commit.enabled` - **Description:** Whether or not the one-phase commit optimization is enabled. - **Default value:** `false` ### `coordinator.write_omission_on_read_only.enabled` - **Field:** `scalar.db.consensus_commit.coordinator.write_omission_on_read_only.enabled` - **Description:** Whether or not the Coordinator write omission optimization is enabled for read-only transactions. This optimization is useful for read-only transactions that do not modify any data, as it avoids unnecessary writes to the Coordinator tables. - **Default value:** `true` ### `coordinator.group_commit.enabled` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.enabled` - **Description:** Whether or not committing the transaction state is executed in batch mode. This feature can't be used with a two-phase commit interface. - **Default value:** `false` ### `coordinator.group_commit.slot_capacity` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.slot_capacity` - **Description:** Maximum number of slots in a group for the group commit feature. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.[^1] - **Default value:** `20` ### `coordinator.group_commit.group_size_fix_timeout_millis` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.group_size_fix_timeout_millis` - **Description:** Timeout to fix the size of slots in a group. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.[^1] - **Default value:** `40` ### `coordinator.group_commit.delayed_slot_move_timeout_millis` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.delayed_slot_move_timeout_millis` - **Description:** Timeout to move delayed slots from a group to another isolated group to prevent the original group from being affected by delayed transactions. A large value improves the efficiency of group commit, but may also increase the latency and the likelihood of transaction conflicts.[^1] - **Default value:** `1200` ### `coordinator.group_commit.old_group_abort_timeout_millis` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.old_group_abort_timeout_millis` - **Description:** Timeout to abort an old ongoing group. A small value reduces resource consumption through aggressive aborts, but may also increase the likelihood of unnecessary aborts for long-running transactions. - **Default value:** `60000` ### `coordinator.group_commit.timeout_check_interval_millis` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.timeout_check_interval_millis` - **Description:** Interval for checking the group commit–related timeouts. - **Default value:** `20` ### `coordinator.group_commit.metrics_monitor_log_enabled` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.metrics_monitor_log_enabled` - **Description:** Whether or not the metrics of the group commit are logged periodically. - **Default value:** `false` ## Storage-related configurations ScalarDB has a storage (database) abstraction layer that supports multiple storage implementations. You can specify the storage implementation by using the `scalar.db.storage` property. :::note For details about using multiple storages, see [Multi-storage configurations](#multi-storage-configurations). ::: Select a database to see the configurations available for each storage. The following configurations are available for JDBC databases.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `jdbc` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** JDBC connection URL. - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** Username to access the database. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Password to access the database. - **Default value:** empty :::note The following properties have been removed and will be ignored if set. If these properties are still in your configuration, please remove them to avoid warning messages. - `scalar.db.jdbc.connection_pool.max_idle` - `scalar.db.jdbc.table_metadata.connection_pool.max_idle` - `scalar.db.jdbc.admin.connection_pool.max_idle` - `scalar.db.jdbc.prepared_statements_pool.enabled` - `scalar.db.jdbc.prepared_statements_pool.max_open` :::

`jdbc.connection_pool.min_idle`

- **Field:** `scalar.db.jdbc.connection_pool.min_idle` - **Description:** Minimum number of idle connections in the connection pool. - **Default value:** `20`

`jdbc.connection_pool.max_total`

- **Field:** `scalar.db.jdbc.connection_pool.max_total` - **Description:** Maximum total number of idle and active connections in the connection pool. - **Default value:** `200`

`jdbc.connection_pool.connection_timeout_millis`

- **Field:** `scalar.db.jdbc.connection_pool.connection_timeout_millis` - **Description:** Maximum time in milliseconds to wait for a connection from the pool. - **Default value:** `30000`

`jdbc.connection_pool.idle_timeout_millis`

- **Field:** `scalar.db.jdbc.connection_pool.idle_timeout_millis` - **Description:** Maximum time in milliseconds that a connection is allowed to sit idle in the pool. This setting only applies when `min_idle` is less than `max_total`. A value of `0` means idle connections are never removed. - **Default value:** `600000`

`jdbc.connection_pool.max_lifetime_millis`

- **Field:** `scalar.db.jdbc.connection_pool.max_lifetime_millis` - **Description:** Maximum lifetime in milliseconds of a connection in the pool. Connections that exceed this lifetime will be retired. This value should be set to a few seconds shorter than any database or infrastructure-imposed connection timeout. A value of `0` means no maximum lifetime. - **Default value:** `1800000`

`jdbc.connection_pool.keepalive_time_millis`

- **Field:** `scalar.db.jdbc.connection_pool.keepalive_time_millis` - **Description:** Interval in milliseconds at which the pool will attempt to keep connections alive to prevent them from being timed out by the database or network infrastructure. This value must be less than `max_lifetime_millis`. A value of `0` disables keepalive. - **Default value:** `0`

`jdbc.isolation_level`

- **Field:** `scalar.db.jdbc.isolation_level` - **Description:** Isolation level for JDBC. `READ_COMMITTED`, `REPEATABLE_READ`, or `SERIALIZABLE` can be specified. - **Default value:** Underlying-database specific

`jdbc.table_metadata.connection_pool.min_idle`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.min_idle` - **Description:** Minimum number of idle connections in the connection pool for the table metadata. - **Default value:** `5`

`jdbc.table_metadata.connection_pool.max_total`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.max_total` - **Description:** Maximum total number of idle and active connections in the connection pool for the table metadata. - **Default value:** `25`

`jdbc.table_metadata.connection_pool.connection_timeout_millis`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.connection_timeout_millis` - **Description:** Same as `jdbc.connection_pool.connection_timeout_millis`, but for the table metadata connection pool. - **Default value:** `30000`

`jdbc.table_metadata.connection_pool.idle_timeout_millis`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.idle_timeout_millis` - **Description:** Same as `jdbc.connection_pool.idle_timeout_millis`, but for the table metadata connection pool. - **Default value:** `600000`

`jdbc.table_metadata.connection_pool.max_lifetime_millis`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.max_lifetime_millis` - **Description:** Same as `jdbc.connection_pool.max_lifetime_millis`, but for the table metadata connection pool. - **Default value:** `1800000`

`jdbc.table_metadata.connection_pool.keepalive_time_millis`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.keepalive_time_millis` - **Description:** Same as `jdbc.connection_pool.keepalive_time_millis`, but for the table metadata connection pool. - **Default value:** `0`

`jdbc.admin.connection_pool.min_idle`

- **Field:** `scalar.db.jdbc.admin.connection_pool.min_idle` - **Description:** Minimum number of idle connections in the connection pool for admin. - **Default value:** `5`

`jdbc.admin.connection_pool.max_total`

- **Field:** `scalar.db.jdbc.admin.connection_pool.max_total` - **Description:** Maximum total number of idle and active connections in the connection pool for admin. - **Default value:** `25`

`jdbc.admin.connection_pool.connection_timeout_millis`

- **Field:** `scalar.db.jdbc.admin.connection_pool.connection_timeout_millis` - **Description:** Same as `jdbc.connection_pool.connection_timeout_millis`, but for the admin connection pool. - **Default value:** `30000`

`jdbc.admin.connection_pool.idle_timeout_millis`

- **Field:** `scalar.db.jdbc.admin.connection_pool.idle_timeout_millis` - **Description:** Same as `jdbc.connection_pool.idle_timeout_millis`, but for the admin connection pool. - **Default value:** `600000`

`jdbc.admin.connection_pool.max_lifetime_millis`

- **Field:** `scalar.db.jdbc.admin.connection_pool.max_lifetime_millis` - **Description:** Same as `jdbc.connection_pool.max_lifetime_millis`, but for the admin connection pool. - **Default value:** `1800000`

`jdbc.admin.connection_pool.keepalive_time_millis`

- **Field:** `scalar.db.jdbc.admin.connection_pool.keepalive_time_millis` - **Description:** Same as `jdbc.connection_pool.keepalive_time_millis`, but for the admin connection pool. - **Default value:** `0`

`jdbc.mysql.variable_key_column_size`

- **Field:** `scalar.db.jdbc.mysql.variable_key_column_size` - **Description:** Column size for TEXT and BLOB columns in MySQL when they are used as a primary key or secondary key. Minimum 64 bytes. - **Default value:** `128`

`jdbc.oracle.variable_key_column_size`

- **Field:** `scalar.db.jdbc.oracle.variable_key_column_size` - **Description:** Column size for TEXT and BLOB columns in Oracle when they are used as a primary key or secondary key. Minimum 64 bytes. - **Default value:** `128`

`jdbc.oracle.time_column.default_date_component`

- **Field:** `scalar.db.jdbc.oracle.time_column.default_date_component` - **Description:** Value of the date component used for storing `TIME` data in Oracle. Since Oracle has no data type to only store a time without a date component, ScalarDB stores `TIME` data with the same date component value for ease of comparison and sorting. - **Default value:** `1970-01-01`

`jdbc.db2.variable_key_column_size`

- **Field:** `scalar.db.jdbc.db2.variable_key_column_size` - **Description:** Column size for TEXT and BLOB columns in IBM Db2 when they are used as a primary key or secondary key. Minimum 64 bytes. - **Default value:** `128`

`jdbc.db2.time_column.default_date_component`

- **Field:** `scalar.db.jdbc.db2.time_column.default_date_component` - **Description:** Value of the date component used for storing `TIME` data in IBM Db2. Since the IBM Db2 TIMESTAMP type is used to store ScalarDB `TIME` type data because it provides fractional-second precision, ScalarDB stores `TIME` data with the same date component value for ease of comparison and sorting. - **Default value:** `1970-01-01`

`jdbc.spanner.time_column.default_date_component`

- **Field:** `scalar.db.jdbc.spanner.time_column.default_date_component` - **Description:** Value of the date component used for storing `TIME` data in Spanner. Because Spanner's PostgreSQL dialect has no native TIME type, ScalarDB stores `TIME` data as Spanner `TIMESTAMP WITH TIME ZONE` data with a fixed date component to enable comparison and sorting. - **Default value:** `1970-01-01` :::note **SQLite3** If you're using SQLite3 as a JDBC database, you must set `scalar.db.contact_points` as follows: ```properties scalar.db.contact_points=jdbc:sqlite:?busy_timeout=10000 ``` Unlike other JDBC databases, [SQLite3 doesn't fully support concurrent access](https://www.sqlite.org/lang_transaction.html). To avoid frequent errors caused internally by [`SQLITE_BUSY`](https://www.sqlite.org/rescode.html#busy), setting a [`busy_timeout`](https://www.sqlite.org/c3ref/busy_timeout.html) parameter is recommended. **YugabyteDB** If you're using YugabyteDB as a JDBC database, you can specify multiple endpoints in `scalar.db.contact_points` as follows: ```properties scalar.db.contact_points=jdbc:yugabytedb://127.0.0.1:5433\\,127.0.0.2:5433\\,127.0.0.3:5433/?load-balance=true ``` Multiple endpoints should be separated by escaped commas. For information on YugabyteDB's smart driver and load balancing, see [YugabyteDB smart drivers for YSQL](https://docs.yugabyte.com/preview/drivers-orms/smart-drivers/). **AlloyDB** If you are using AlloyDB on Google Cloud as a JDBC database and want to connect with the [Java connector](https://docs.cloud.google.com/alloydb/docs/connect-language-connectors#configure-connectors), you need to add additional properties in `scalar.db.contact_points` as follows: ```properties scalar.db.contact_points=jdbc:postgresql:///?socketFactory=com.google.cloud.alloydb.SocketFactory&alloydbInstanceName=&alloydbIpType=PUBLIC ``` **Spanner** Authentication to Spanner requires using a Google Cloud service account key in JSON format. Set `scalar.db.password` to the full content of the service account key file as a single line JSON. The `scalar.db.username` property is unused for Spanner. ScalarDB also sets the JVM system property `ENABLE_CREDENTIALS_PROVIDER=true`, which is required by the Spanner JDBC driver to authenticate. For example: ```properties scalar.db.storage=jdbc scalar.db.contact_points=jdbc:cloudspanner:/projects//instances//databases/ scalar.db.username= scalar.db.password= ``` ::: The following configurations are available for DynamoDB.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `dynamo` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** AWS region with which ScalarDB should communicate (for example, `us-east-1`). - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** AWS access key used to identify the user interacting with AWS. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** AWS secret access key used to authenticate the user interacting with AWS. - **Default value:** empty

`dynamo.endpoint_override`

- **Field:** `scalar.db.dynamo.endpoint_override` - **Description:** Amazon DynamoDB endpoint with which ScalarDB should communicate. This is primarily used for testing with a local instance instead of an AWS service. - **Default value:** empty

`dynamo.namespace.prefix`

- **Field:** `scalar.db.dynamo.namespace.prefix` - **Description:** Prefix for the user namespaces and metadata namespace names. Since AWS requires having unique tables names in a single AWS region, this is useful if you want to use multiple ScalarDB environments (development, production, etc.) in a single AWS region. - **Default value:** empty The following configurations are available for CosmosDB for NoSQL.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `cosmos` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** Azure Cosmos DB for NoSQL endpoint with which ScalarDB should communicate. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Either a master or read-only key used to perform authentication for accessing Azure Cosmos DB for NoSQL. - **Default value:** empty

`cosmos.consistency_level`

- **Field:** `scalar.db.cosmos.consistency_level` - **Description:** Consistency level used for Cosmos DB operations. `STRONG` or `BOUNDED_STALENESS` can be specified. - **Default value:** `STRONG` The following configurations are available for Cassandra.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `cassandra` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** Comma-separated contact points. - **Default value:** empty

`contact_port`

- **Field:** `scalar.db.contact_port` - **Description:** Port number for all the contact points. - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** Username to access the database. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Password to access the database. - **Default value:** empty The following configurations are available for S3.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `s3` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** '/'-separated region and S3 bucket name (for example, `us-east-1/my-bucket`). - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** AWS access key. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** AWS secret access key. - **Default value:** empty

`s3.multipart_upload_part_size_bytes`

- **Field:** `scalar.db.s3.multipart_upload_part_size_bytes` - **Description:** The part size in bytes for multipart upload. - **Default value:** The default value of [`minimumPartSizeInBytes`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/multipart/MultipartConfiguration.html#minimumPartSizeInBytes()) in the AWS SDK.

`s3.multipart_upload_max_concurrency`

- **Field:** `scalar.db.s3.multipart_upload_max_concurrency` - **Description:** The maximum number of concurrent requests allowed for multipart upload. - **Default value:** The default value of [`maxConcurrency`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/crt/AwsCrtAsyncHttpClient.Builder.html#maxConcurrency(java.lang.Integer)) in the AWS SDK.

`s3.multipart_upload_threshold_size_bytes`

- **Field:** `scalar.db.s3.multipart_upload_threshold_size_bytes` - **Description:** The threshold size in bytes to enable multipart upload. If the object size is greater than or equal to this value, multipart upload is used. - **Default value:** The default value of [`thresholdInBytes`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/multipart/MultipartConfiguration.html#thresholdInBytes()) in the AWS SDK.

`s3.request_timeout_secs`

- **Field:** `scalar.db.s3.request_timeout_secs` - **Description:** The request timeout in seconds for S3 operations set to [`apiCallTimeout`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/client/config/ClientOverrideConfiguration.Builder.html#apiCallTimeout(java.time.Duration)) in the AWS SDK. - **Default value:** empty (no timeout) The following configurations are available for Blob Storage.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `blob-storage` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** Blob Storage endpoint URL including the container name (for example, `https://.blob.core.windows.net/my-container`). - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** Azure Storage account name. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Azure Storage account key. - **Default value:** empty

`blob_storage.parallel_upload_block_size_bytes`

- **Field:** `scalar.db.blob_storage.parallel_upload_block_size_bytes` - **Description:** The block size in bytes for parallel upload. - **Default value:** The default value of [`setBlockSizeLong`](https://learn.microsoft.com/en-us/java/api/com.azure.storage.blob.models.paralleltransferoptions?view=azure-java-stable#com-azure-storage-blob-models-paralleltransferoptions-setblocksizelong(java-lang-long)) in the Azure SDK.

`blob_storage.parallel_upload_max_concurrency`

- **Field:** `scalar.db.blob_storage.parallel_upload_max_concurrency` - **Description:** The maximum number of concurrent requests allowed for parallel upload. - **Default value:** The default value of [`setMaxConcurrency`](https://learn.microsoft.com/en-us/java/api/com.azure.storage.blob.models.paralleltransferoptions?view=azure-java-stable#com-azure-storage-blob-models-paralleltransferoptions-setmaxconcurrency(java-lang-integer)) in the Azure SDK.

`blob_storage.parallel_upload_threshold_size_bytes`

- **Field:** `scalar.db.blob_storage.parallel_upload_threshold_size_bytes` - **Description:** The threshold size in bytes to enable parallel upload. If the object size is greater than this value, parallel upload is used. - **Default value:** The default value of [`setMaxSingleUploadSizeLong`](https://learn.microsoft.com/en-us/java/api/com.azure.storage.blob.models.paralleltransferoptions?view=azure-java-stable#com-azure-storage-blob-models-paralleltransferoptions-setmaxsingleuploadsizelong(java-lang-long)) in the Azure SDK.

`blob_storage.request_timeout_secs`

- **Field:** `scalar.db.blob_storage.request_timeout_secs` - **Description:** The request timeout in seconds for Blob Storage operations. - **Default value:** empty (no timeout) The following configurations are available for Cloud Storage.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `cloud-storage` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** Cloud Storage bucket name. - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** Google Cloud project ID. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Full content of the Google Cloud service account key file as a single-line JSON. - **Default value:** empty

`cloud_storage.upload_chunk_size_bytes`

- **Field:** `scalar.db.cloud_storage.upload_chunk_size_bytes` - **Description:** The chunk size in bytes for upload. - **Default value:** The default value of [`setChunkSize`](https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.WriteChannel#com_google_cloud_WriteChannel_setChunkSize_int_) in the Google Cloud SDK. ### Multi-storage configurations ScalarDB supports using multiple storage implementations simultaneously. For details about using multiple storages, see [Multi-Storage Transactions](./multi-storage-transactions.mdx). #### `storage` - **Field:** `scalar.db.storage` - **Description:** `multi-storage` must be specified. #### `multi_storage.storages` - **Field:** `scalar.db.multi_storage.storages` - **Description:** Comma-separated storage names (for example, `cassandra,mysql`). These storage names will be used in the `scalar.db.multi_storage.namespace_mapping` property to map namespaces to storages. - **Default value:** empty #### `multi_storage.default_storage` - **Field:** `scalar.db.multi_storage.default_storage` - **Description:** Default storage name. This storage will be used for any namespace that doesn't have mapping defined in the `scalar.db.multi_storage.namespace_mapping` property. - **Default value:** empty #### `multi_storage.namespace_mapping` - **Field:** `scalar.db.multi_storage.namespace_mapping` - **Description:** Mapping of namespaces to storages (for example, `user:my_cassandra,coordinator:my_mysql`). - **Default value:** empty :::tip The storage names (``) are arbitrary values that you need to define. You can use any names that you like as long as they are consistent across the multi-storage configurations. ::: #### `multi_storage.storages..` For configuring specific storages, use `scalar.db.multi_storage.storages..`, with `` being one of the storage names specified in the `scalar.db.multi_storage.storages` property and `` being the property name for the specific storage. For example, if you've defined [namespace mapping](#multi_storagenamespace_mapping) as `scalar.db.multi_storage.namespace_mapping=user:my_cassandra,coordinator:my_mysql`, with `my_cassandra` and `my_mysql` being the storage names for the `user` and `coordinator` namespaces, respectively: - You can specify the contact points for Cassandra by using `scalar.db.multi_storage.storages.my_cassandra.contact_points`. - You can specify the minimum number of idle connections in the connection pool for MySQL by using `scalar.db.multi_storage.storages.my_mysql.jdbc.connection_pool.min_idle`. For details about the properties available for each storage, see [Storage-related configurations](#storage-related-configurations). ### Cross-partition scan configurations By enabling the cross-partition scan option as described below, the `Scan` operation can retrieve all records across partitions. In addition, you can specify arbitrary conditions and orderings in the cross-partition `Scan` operation by enabling `cross_partition_scan.filtering` and `cross_partition_scan.ordering`, respectively. Currently, the cross-partition scan with ordering option is available only for JDBC databases. To enable filtering and ordering, `scalar.db.cross_partition_scan.enabled` must be set to `true`. For details on how to use cross-partition scan, see [Scan operation](./api-guide.mdx#scan-operation). :::warning For non-JDBC databases, transactions could be executed at read-committed snapshot isolation (`SNAPSHOT`), which is a lower isolation level, even if you enable cross-partition scan with the `SERIALIZABLE` isolation level. When using non-JDBC databases, use cross-partition scan only if consistency does not matter for your transactions. ::: #### `cross_partition_scan.enabled` - **Field:** `scalar.db.cross_partition_scan.enabled` - **Description:** Enable cross-partition scan. - **Default value:** `true` #### `cross_partition_scan.filtering.enabled` - **Field:** `scalar.db.cross_partition_scan.filtering.enabled` - **Description:** Enable filtering in cross-partition scan. - **Default value:** `false` #### `cross_partition_scan.ordering.enabled` - **Field:** `scalar.db.cross_partition_scan.ordering.enabled` - **Description:** Enable ordering in cross-partition scan. - **Default value:** `false` ### Scan configurations You can configure the fetch size for storage scan operations by using the following property. #### `scan_fetch_size` - **Field:** `scalar.db.scan_fetch_size` - **Description:** Specifies the number of records to fetch in a single batch during a storage scan operation. A larger value can improve performance for a large result set by reducing round trips to the storage, but it also increases memory usage. A smaller value uses less memory but may increase latency. - **Default value:** `10` ## Other ScalarDB configurations The following are additional configurations available for ScalarDB. ### `metadata.cache_expiration_time_secs` - **Field:** `scalar.db.metadata.cache_expiration_time_secs` - **Description:** ScalarDB has a metadata cache to reduce the number of requests to the database. This setting specifies the expiration time of the cache in seconds. If you specify `-1`, the cache will never expire. - **Default value:** `60` ### `active_transaction_management.expiration_time_millis` - **Field:** `scalar.db.active_transaction_management.expiration_time_millis` - **Description:** ScalarDB maintains in-progress transactions, which can be resumed by using a transaction ID. This process expires transactions that have been idle for an extended period to prevent resource leaks. This setting specifies the expiration time of this transaction management feature in milliseconds. - **Default value:** `-1` (no expiration) ### `consensus_commit.include_metadata.enabled` - **Field:** `scalar.db.consensus_commit.include_metadata.enabled` - **Description:** When using Consensus Commit, if this is set to `true`, `Get` and `Scan` operations results will contain transaction metadata. To see the transaction metadata columns details for a given table, you can use the `DistributedTransactionAdmin.getTableMetadata()` method, which will return the table metadata augmented with the transaction metadata columns. Using this configuration can be useful to investigate transaction-related issues. - **Default value:** `false` ### `consensus_commit.index.eventually_consistent_read.enabled` - **Field:** `scalar.db.consensus_commit.index.eventually_consistent_read.enabled` - **Description:** When using Consensus Commit, if this is set to `true`, the before-image index check will be skipped, and index-based reads may miss records whose indexed column is being concurrently updated. - **Default value:** `false` :::warning This is a backward-compatibility option and is **not recommended for new workloads**. For details, see [Correctness of index-based reads](consensus-commit.mdx#correctness-of-index-based-reads). ::: ### `default_namespace_name` - **Field:** `scalar.db.default_namespace_name` - **Description:** The given namespace name will be used by operations that do not already specify a namespace. - **Default value:** empty ## Placeholder usage You can use placeholders in the values, and they are replaced with environment variables (`${env:}`) or system properties (`${sys:}`). You can also specify default values in placeholders like `${sys::-}`. The following is an example of a configuration that uses placeholders: ```properties scalar.db.username=${env:SCALAR_DB_USERNAME:-admin} scalar.db.password=${env:SCALAR_DB_PASSWORD} ``` In this example configuration, ScalarDB reads the username and password from environment variables. If the environment variable `SCALAR_DB_USERNAME` does not exist, ScalarDB uses the default value `admin`. ## Configuration example - App and database ```mermaid flowchart LR app["App
(ScalarDB library with
Consensus Commit)"] db[(Underlying storage or database)] app --> db ``` In this example configuration, the app (ScalarDB library with Consensus Commit) connects to an underlying storage or database (in this case, Cassandra) directly. :::warning This configuration exists only for development purposes and isn't suitable for a production environment. This is because the app needs to implement the [Scalar Admin](https://github.com/scalar-labs/scalar-admin) interface to take transactionally consistent backups for ScalarDB, which requires additional configurations. ::: The following is an example of the configuration for connecting the app to the underlying database through ScalarDB: ```properties # Transaction manager implementation. scalar.db.transaction_manager=consensus-commit # Storage implementation. scalar.db.storage=cassandra # Comma-separated contact points. scalar.db.contact_points= # Credential information to access the database. scalar.db.username= scalar.db.password= ``` ================================================ FILE: docs/consensus-commit.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Consensus Commit Protocol import JavadocLink from '/src/theme/JavadocLink.js'; Consensus Commit is the transaction protocol used in ScalarDB and is designed for executing transactions spanning multiple diverse databases. Its uniqueness is that the protocol achieves ACID transactions without relying on the transaction capabilities of the underlying databases, unlike X/Open XA-based solutions. This document explains the details of the protocol, including how it works, the guaranteed isolation levels, the interfaces, the performance optimization that it employs, and its limitations. ## The protocol This section explains how the Consensus Commit protocol works. The Consensus Commit protocol uses a concurrency control protocol to guarantee isolation and an atomic commitment protocol to guarantee atomicity and durability. ### Concurrency control protocol The Consensus Commit protocol employs optimistic concurrency control (OCC) as its concurrency control protocol. OCC operates under the assumption that conflicts are rare, allowing transactions to proceed without the need for locks and resolving conflicts only when they actually occur. Therefore, OCC performs great in low-contention environments. It is also particularly beneficial in distributed environments, where managing locks is tricky. :::note Pessimistic concurrency control (PCC), on the other hand, assumes conflicts are common and takes locks on resources when they are used to avoid interference. Therefore, PCC performs great in high-contention environments. ::: The OCC protocol of ScalarDB has three phases, as the commonly used OCC protocols, each of which does the following: * Read phase: * ScalarDB tracks the read and write sets of transactions. ScalarDB copies every record that a transaction accesses from databases to its local workspace and stores its writes in the local workspace. * Validation phase: * ScalarDB checks if the committing transaction conflicts with other transactions. ScalarDB uses backward validation; it goes to the write phase only if other transactions have not written what the transaction reads and writes, which are called read validation and write validation, respectively. * Write phase: * ScalarDB propagates the changes in the transaction's write set to the database and makes them visible to other transactions. As described next, ScalarDB provides an isolation mode (isolation level) where it skips the read validation in the validation phase to allow for more performance for some workloads that don't require the read validation for correctness. :::note The OCC of ScalarDB without the read validation works similarly to snapshot isolation. However, it works with a single version and causes read-skew anomalies because it does not create global snapshots. ::: ### Atomic commitment protocol The Consensus Commit protocol employs a variant of the two-phase commit protocol as an atomic commitment protocol (ACP). The ACP of ScalarDB comprises two phases, each of which has two sub-phases, and briefly works as follows: * Prepare phase (prepare-records phase \+ validate-records phase): * In the prepare-records phase, ScalarDB runs the write validation of the OCC protocol for all the records written by the transaction by updating the statuses of the records to PREPARED and moves on to the next phase if all the records are successfully validated. * In the validate-records phase, ScalarDB runs the read validation of the OCC protocol for all the records read by the transaction and moves on to the next phase if all the records are successfully validated. * Commit phase (commit-state phase \+ commit-records phase): * In the commit-state phase, ScalarDB commits the transaction by writing a COMMITTED state to a special table called a coordinator table. * In the commit-records phase, ScalarDB runs the write phase of the OCC protocol for all the records written by the transaction by updating the statuses of the records to COMMITTED. :::note In case of deleting records, the statuses of the records are first changed to DELETED in the prepare phase and later physically deleted in the commit phase. ::: #### How it works in more detail Let's see how the protocol works in each phase in more detail. ##### Before the prepare phase First, a transaction begins when a client accesses ScalarDB (or a ScalarDB Cluster node) and issues a `begin` command. When a transaction begins, ScalarDB acts as a transaction coordinator, accessing the underlying databases, and first generates a transaction ID (TxID) with UUID version 4. Then, when the client is ready to commit the transaction after performing operations such as reading and writing records, it calls a `commit` command to request ScalarDB to commit the transaction and enters the prepare phase. As described previously, ScalarDB holds the read set (readSet) and write set (writeSet) of the transaction in its local workspace at the time of committing. ##### Prepare phase ScalarDB first prepares the records of the write set by propagating the records, including transaction logs like TxID as described later, with PREPARED states to the underlying databases as the prepare-records phase. Here, we assume a write set maintains updated records composed of the original records and updated columns. If any preparation fails, it aborts the transaction by writing an ABORTED state record to a Coordinator table, where all the transactions’ final states are determined and managed. We explain the Coordinator table in more detail later in this section. :::note ScalarDB checks conflicting preparations by using linearizable conditional writes. A transaction updates a record if the record has not been updated by another transaction since the transaction read it by checking if the TxID of the record has not been changed. ::: ScalarDB then moves on to the validate-records phase as necessary. The validate-records phase is only necessary if the isolation level is set to SERIALIZABLE. In this phase, ScalarDB re-reads all the records in the read set to see if other transactions have written the records that the transaction has read before. If the read set has not been changed, the transaction can go to the commit-state phase since there are no anti-dependencies; otherwise, it aborts the transaction. ##### Commit phase If all the validations in the prepare phase are done successfully, ScalarDB commits the transaction by writing a COMMITTED state record to the Coordinator table as the commit-state phase. :::note * ScalarDB uses linearizable conditional writes to coordinate concurrent writes to the Coordinator table, creating a state record with a TxID if there is no record for the TxID. Once the COMMITTED state is correctly written to the Coordinator table, the transaction is regarded as committed. * By default, if a transaction contains only read operations, ScalarDB skips the commit-state phase. However, you can configure ScalarDB to write a COMMITTED state record to the Coordinator table even for read-only transactions by setting the following parameter to `false`: * `scalar.db.consensus_commit.coordinator.write_omission_on_read_only.enabled` ::: Then, ScalarDB commits all the validated (prepared) records by changing the states of the records to COMMITTED as the commit-records phase. #### Distributed WAL ScalarDB stores transaction logs (transaction metadata), which are for write-ahead logging (WAL), in the underlying database records that it manages. Specifically, as shown in the following figure, ScalarDB manages special columns for the log information in a record in addition to the columns that an application manages. The log information comprises, for example, a transaction ID (TxID) that has updated the corresponding record most recently, a record version number (Version), a record state (TxState) (for example, COMMITTED or PREPARED), timestamps (not shown in the diagram), and a before image that comprises the previous version's application data and its metadata. ScalarDB also manages transaction states separately from the application records in the Coordinator table. The Coordinator table determines and manages transaction states as a single source of truth. The Coordinator table can be collocated with application-managed tables or located in a separate dedicated database. ![Distributed WAL](images/scalardb-metadata.png) :::note The Coordinator table can be replicated for high availability by using the replication and consensus capabilities of the underlying databases. For example, if you manage the Coordinator table by using Cassandra with a replication factor of three, you can make the transaction coordination of ScalarDB tolerate one replica crash. Hence, you can make the atomic commitment protocol of ScalarDB perform like the Paxos Commit protocol; it could mitigate liveness issues (for example, blocking problems) without sacrificing safety. ::: #### Transaction metadata decoupling ScalarDB manages transaction metadata in the same table as application data by default. However, ScalarDB also provides an option to enable managing transaction metadata separately from the application data, which we call transaction metadata decoupling. This capability is particularly useful when importing existing database tables into ScalarDB, as it allows users to maintain the original schema without any modifications. :::note Transaction metadata decoupling is currently in Private Preview. For details, please [contact us](https://www.scalar-labs.com/contact) or wait for this feature to become publicly available in a future version. Currently, transaction metadata decoupling is supported only for JDBC databases. ::: If transaction metadata decoupling is enabled, ScalarDB internally creates a virtual table that combines the application data table with the transaction metadata table. The Consensus Commit protocol operates on this virtual table, functioning in the same manner as it does when transaction metadata decoupling is not enabled. However, ScalarDB requires twice as many accesses to the underlying databases when reading and writing records, which could result in performance overhead. ScalarDB minimizes the performance overhead by grouping and pushing down as many operations as possible when accessing the underlying databases. #### Lazy recovery Transactions can crash at any time and could leave records in an uncommitted state. ScalarDB recovers uncommitted records lazily when it reads them, depending on the transaction states of the Coordinator table. Specifically, if a record is in the PREPARED state, but the transaction that updated the record has expired or been aborted, the record will be rolled back. If a record is in the PREPARED state and the transaction that updated the record is committed, the record will be rolled forward. A transaction expires after a certain amount of time (currently 15 seconds). When ScalarDB observes a record that has been prepared by an expired transaction, ScalarDB writes the ABORTED state for the transaction to the Coordinator table (with retries). If ScalarDB successfully writes the ABORTED state to the Coordinator table, the transaction is aborted. Otherwise, the transaction will be committed by the original process that is slow but still alive for some reason, or it will remain in the UNKNOWN state until it is either aborted or committed. #### Correctness of index-based reads To ensure the correctness of index-based `Get`, `Scan`, and `ScanAll` operations, ScalarDB performs an additional check called a *before-image index check* when reading records through a secondary index. For every user-defined secondary index on a non-primary-key column, ScalarDB stores the before-image in an internal column named `before_` and maintains a companion secondary index on that before-image column in the underlying storage. The check uses this companion index to find records whose committed indexed-column value matches the query but whose current indexed value has been changed by another transaction that is not yet committed (that is, records in the PREPARED or DELETED state). Any such record is then recovered (rolled back or rolled forward) before results are returned; therefore, ScalarDB throws an exception if the record hasn't expired, otherwise it recovers the transaction. Under the SERIALIZABLE isolation level, ScalarDB also uses the same before-image index during the read validation phase so that concurrent updates on indexed columns are correctly detected as anti-dependencies. As a result, index-based `Get`, `Scan`, and `ScanAll` operations return every record whose committed indexed-column value matches the query. :::warning - Tables created by a ScalarDB version that predates this check do not have the companion before-image indexes required for it. The check is available starting from ScalarDB 3.16.5, 3.17.3, and 3.18.0. After upgrading from an earlier version, you must run [`repairTable()`](api-guide.mdx#repair-a-table) on each existing table to create the missing companion indexes. Until `repairTable()` is run, ScalarDB will log a warning at startup, and index-based reads may miss records whose indexed column is being concurrently updated. - Setting `scalar.db.consensus_commit.index.eventually_consistent_read.enabled` to `true` skips the before-image index check. As a result, index-based reads may miss records whose indexed column is being concurrently updated. This option exists for backward compatibility and is not recommended for new workloads. ::: ## Isolation levels The Consensus Commit protocol supports three isolation levels: read-committed snapshot isolation (a weaker variant of snapshot isolation), serializable, and read-committed, each of which has the following characteristics: * Read-committed snapshot isolation (SNAPSHOT - default) * Possible anomalies: read skew, write skew, read only * Faster than serializable, but guarantees weaker correctness. * Serializable (SERIALIZABLE) * Possible anomalies: None * Slower than read-committed snapshot isolation, but guarantees stronger (strongest) correctness. * Read-committed (READ_COMMITTED) * Possible anomalies: read skew, write skew, read only * Faster than read-committed snapshot isolation because it could return non-latest committed records. As described above, serializable is preferable from a correctness perspective, but slower than read-committed snapshot isolation. Choose the appropriate one based on your application requirements and workload. For details on how to configure read-committed snapshot isolation, serializable, and read-committed, see [ScalarDB Configuration](configurations.mdx#basic-configurations). You can also override the configured default isolation level on a per-transaction basis by using the `cc-transaction-isolation` attribute when beginning or starting a transaction. For details, see [Consensus Commit attributes](api-guide.mdx#consensus-commit-attributes). :::note The Consensus Commit protocol of ScalarDB requires each underlying database to provide linearizable operations, as described in [Configurations for the Underlying Databases of ScalarDB](database-configurations.mdx#transactions); thus, it guarantees strict serializability. ::: :::warning Scanning records without specifying a partition key (for example, or `SELECT * FROM table`) for non-JDBC databases does not always guarantee serializability, even if `SERIALIZABLE` is specified. Therefore, you should do so at your own discretion and consider updating the schemas if possible. For more details, refer to [Cross-partition scan configurations](configurations.mdx#cross-partition-scan-configurations). ::: ## Interfaces The Consensus Commit protocol provides two interfaces: [a one-phase commit interface and a two-phase commit interface](scalardb-cluster/run-transactions-through-scalardb-cluster.mdx#run-transactions). The one-phase commit interface is a simple interface that provides only a single `commit` method, where all the phases of the atomic commitment protocol are executed in the method. On the other hand, the two-phase commit interface exposes each phase of the protocol with `prepare`, `validate`, and `commit` methods. :::note The `prepare` method is for the prepare-records phase, and the `validate` method is for the validate-records phase. ::: In most cases, using the one-phase commit interface is recommended since it is easier to use and handle errors. But the two-phase commit interface is useful when running a transaction across multiple applications or services without directly accessing databases from ScalarDB, such as maintaining the consistency of databases in microservices. ## Performance optimization The Consensus Commit protocol employs several performance optimizations. ### Parallel execution Consensus Commit executes each phase of the atomic commitment protocol in parallel, using intra-transaction parallelism without sacrificing correctness. Specifically, it tries to execute the prepare-records phase by writing records with PREPARED status in parallel. Likewise, it uses a similar parallel execution for the validate-records phase, the commit-records phase, and the rollback process. You can enable respective parallel execution by using the following parameters: * Prepare-records phase * `scalar.db.consensus_commit.parallel_preparation.enabled` * Validate-records phase * `scalar.db.consensus_commit.parallel_validation.enabled` * Commit-records phase * `scalar.db.consensus_commit.parallel_commit.enabled` * Rollback processing * `scalar.db.consensus_commit.parallel_rollback.enabled` You can also configure the execution parallelism by using the following parameter: * `scalar.db.consensus_commit.parallel_executor_count` For details about the configuration, refer to [Performance-related configurations](configurations.mdx#performance-related-configurations). ### Asynchronous execution Since a transaction is regarded as committed if the commit-state phase completes successfully, it can also return to the client without waiting for the completion of the commit-records phase, executing the phase asynchronously. Likewise, when a transaction fails for some reason and does a rollback, the rollback process can also be executed asynchronously without waiting for its completion. You can enable respective asynchronous execution by using the following parameters: * Commit-records phase * `scalar.db.consensus_commit.async_commit.enabled` * Rollback processing * `scalar.db.consensus_commit.async_rollback.enabled` ### One-phase commit With one-phase commit optimization, ScalarDB can omit the prepare-records and commit-state phases without sacrificing correctness, provided that the transaction only updates records that the underlying database can atomically update. You can enable one-phase commit optimization by using the following parameter: * `scalar.db.consensus_commit.one_phase_commit.enabled` ### Group commit Consensus Commit provides a group-commit feature to execute the commit-state phase of multiple transactions in a batch, reducing the number of writes for the commit-state phase. It is especially useful when writing to a Coordinator table is slow, for example, when the Coordinator table is deployed in a multi-region environment for high availability. You can enable group commit by using the following parameter: * `scalar.db.consensus_commit.coordinator.group_commit.enabled` Group commit has several other parameters. For more details, refer to [Performance-related configurations](configurations.mdx#performance-related-configurations). ## Limitations ScalarDB has several limitations in achieving database-agnostic transactions. ### Applications must access ScalarDB to access the underlying databases Since ScalarDB with the Consensus Commit protocol handles transactions in its layer without depending on the transactional capability of the underlying databases, your applications cannot bypass ScalarDB. Bypassing it will cause unexpected behavior, mostly resulting in facing some database anomalies. Even for read operations, accessing the underlying databases of ScalarDB directly will give you inconsistent data with the transaction metadata, so it is not allowed. However, for tables that are not managed or touched by ScalarDB transactions, you can read from and write to the tables. For example, it is OK to check tables' metadata information, such as information schema, by directly accessing the tables without going through ScalarDB. Also, there are several other cases where you can access databases directly without going through ScalarDB. The basic criterion is whether or not you update the data of the underlying databases. If you are sure that you do not write to the databases, you can access the databases directly. For example, it is OK to take a backup of databases by using database-native tools. :::note If you take backups from multiple databases or from non-transactional databases, you need to pause your applications or ScalarDB Cluster. For more details, refer to [How to Back Up and Restore Databases Used Through ScalarDB](backup-restore.mdx). ::: ### Executing particular operations in a certain sequence is prohibited for correctness In the current implementation, ScalarDB throws an exception when executing scan operations after write (Put, Insert, Update, Upsert, Delete) operations for the same record in a transaction. ## See also You can learn more about the Consesnus Commit protocol by seeing the following presentation and YouTube video, which summarizes, visually, how the protocol works: - **Speaker Deck presentation:** [ScalarDB: Universal Transaction Manager](https://speakerdeck.com/scalar/scalar-db-universal-transaction-manager) - **YouTube (Japanese):** [How ScalarDB runs transactions (a part of DBSJ lecture)](https://www.youtube.com/watch?v=s6Q7QQccDTc) In addition, more details about the protocol, including the background, the challenges, and the novelty, are discussed in the following research paper and its presentation: - **Research paper:** [ScalarDB: Universal Transaction Manager for Polystores](https://www.vldb.org/pvldb/vol16/p3768-yamada.pdf) - **Speaker Deck presentation:** [ScalarDB: Universal Transaction Manager for Polystores](https://speakerdeck.com/scalar/scalardb-universal-transaction-manager-for-polystores-vldb23) ================================================ FILE: docs/data-loader.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Data Loader import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; ScalarDB Data Loader is a utility tool enabling you to import and export data with ScalarDB Core easily. If you're using ScalarDB Cluster, you can use ScalarDB Cluster Data Loader, which is a version of Data Loader that you can use to either access backend databases directly or connect to the cluster to perform import and export operations through it. Data Loader provides structured import and export processes with validation, error handling, and detailed logging to help you safely move data in and out of ScalarDB. ## Choose the right configuration based on your use case Use the following decision tree to determine which configuration pattern is appropriate for your use case: ```mermaid flowchart LR A["Using Consensus Commit or Single CRUD?"] -->|Consensus Commit| B["Using ScalarDB Cluster?"] A -->|Single CRUD| C["Using ScalarDB Cluster?"] B -->|Yes| D["Import/Export large amount of data?"] B -->|No| E["C) Access databases directly"] D -->|Yes| F["A) Access databases directly"] D -->|No| G["B) Access databases through ScalarDB Cluster"] C -->|Yes| H["Import/Export large amount of data?"] C -->|No| I["F) Access databases directly"] H -->|Yes| J["D) Access databases directly"] H -->|No| K["E) Access databases through ScalarDB Cluster"] click E "?config-pattern=pattern-ac#configuration-patterns" "Go to Pattern A/C" click F "?config-pattern=pattern-ac#configuration-patterns" "Go to Pattern A/C" click G "?config-pattern=pattern-b#configuration-patterns" "Go to Pattern B" click I "?config-pattern=pattern-df#configuration-patterns" "Go to Pattern D/F" click J "?config-pattern=pattern-df#configuration-patterns" "Go to Pattern D/F" click K "?config-pattern=pattern-e#configuration-patterns" "Go to Pattern E" classDef clickable fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#1a73e8 class E,F,G,I,J,K clickable ``` :::note When using Consensus Commit, if you want SERIALIZABLE isolation, stop the cluster. If you can accept READ COMMITTED isolation, you don't need to stop the cluster. ::: ### Configuration patterns {#configuration-patterns} Based on the decision tree, select your configuration pattern: Use this pattern when you need Consensus Commit transactions and are accessing databases directly (either because you don't have ScalarDB Cluster, or you have large amounts of data to import/export). Rows are imported by using transactional operations, ensuring ACID properties. By default, up to 100 put operations are grouped into a single transaction. You can adjust this by using the `--transaction-size` option. **Client configuration** (scalardb.properties for Data Loader): ```properties # Transaction manager for direct database access with Consensus Commit scalar.db.transaction_manager=consensus-commit # Storage configuration (example for PostgreSQL) scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://:5432/ scalar.db.username= scalar.db.password= ``` For other database configurations, see [ScalarDB Configurations](./configurations.mdx). When running import commands, use `--mode TRANSACTION`. The `--mode` argument is not required for export commands. :::note When using the Consensus Commit transaction manager, each transaction group (100 records by default) meets ACID guarantees, but the overall import or export operation is not atomic. If interrupted, some groups may be committed while others are not. Use the log files to identify and retry failed records. ::: :::warning To ensure data consistency: - **With ScalarDB Cluster (Pattern A):** Stop the cluster during the import or export operation. - **Without ScalarDB Cluster (Pattern C):** Stop other processes that update the databases during the operation. ::: Use this pattern when you have ScalarDB Cluster configured for Consensus Commit transactions and are not importing or exporting large amounts of data. **Client configuration** (scalardb.properties for ScalarDB Cluster Data Loader): ```properties # Transaction manager for connecting to ScalarDB Cluster scalar.db.transaction_manager=cluster # Contact point of the cluster (use your load balancer address) scalar.db.contact_points=indirect: # Optional: Port number (default is 60053) scalar.db.contact_port=60053 ``` Replace `` with your ScalarDB Cluster endpoint (for example, `localhost` or `192.168.10.1`). **Cluster configuration** (scalardb-cluster-node.properties): ```properties # Transaction manager on the cluster side scalar.db.transaction_manager=consensus-commit # Isolation level (SNAPSHOT, SERIALIZABLE, or READ_COMMITTED) scalar.db.consensus_commit.isolation_level=SNAPSHOT # Storage configuration (example for PostgreSQL) scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://:5432/ scalar.db.username= scalar.db.password= ``` For other database configurations, see [ScalarDB Cluster Configurations](./scalardb-cluster/scalardb-cluster-configurations.mdx). When running import commands, use `--mode TRANSACTION`. The `--mode` argument is not required for export commands. :::warning To ensure data consistency, stop other processes that update the databases through ScalarDB Cluster during the import or export operation. ::: Use this pattern when you need non-transactional storage operations and are accessing databases directly (either because you don't have ScalarDB Cluster, or you have large amounts of data to import/export). **Client configuration** (scalardb.properties for Data Loader): ```properties # Transaction manager for direct database access (non-transactional) scalar.db.transaction_manager=single-crud-operation # Storage configuration (example for PostgreSQL) scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://:5432/ scalar.db.username= scalar.db.password= ``` For other database configurations, see [ScalarDB Configurations](./configurations.mdx). When running import commands, use `--mode STORAGE`. The `--mode` argument is not required for export commands. Use this pattern when you have ScalarDB Cluster configured for non-transactional storage operations and are not importing or exporting large amounts of data. **Client configuration** (scalardb.properties for ScalarDB Cluster Data Loader): ```properties # Transaction manager for connecting to ScalarDB Cluster scalar.db.transaction_manager=cluster # Contact point of the cluster (use your load balancer address) scalar.db.contact_points=indirect: # Optional: Port number (default is 60053) scalar.db.contact_port=60053 ``` Replace `` with your ScalarDB Cluster endpoint (for example, `localhost` or `192.168.10.1`). **Cluster configuration** (scalardb-cluster-node.properties): ```properties # Transaction manager on the cluster side (non-transactional) scalar.db.transaction_manager=single-crud-operation # Storage configuration (example for PostgreSQL) scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://:5432/ scalar.db.username= scalar.db.password= ``` For other database configurations, see [ScalarDB Cluster Configurations](./scalardb-cluster/scalardb-cluster-configurations.mdx). When running import commands, use `--mode STORAGE`. The `--mode` argument is not required for export commands. ## Prerequisites Before using Data Loader with ScalarDB Cluster, make sure you have the following: - One of the following Java Development Kits (JDKs): - A valid **`scalardb.properties`** file configured with ScalarDB Cluster connection settings (cluster endpoint and port) - A running ScalarDB Cluster instance and network access to cluster endpoints Before using Data Loader with direct database access, make sure you have the following: - One of the following Java Development Kits (JDKs): - A valid **`scalardb.properties`** file configured with direct database connection settings - Database permissions for read and write operations (see [Database permission requirements](./requirements.mdx#database-permission-requirements)) ## Set up Data Loader Select your preferred method to set up Data Loader, and follow the instructions. Download **`scalardb-cluster-data-loader--all.jar`** from the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page. Verify the installation by running the following command, replacing `` with the version number: ```console java -jar scalardb-cluster-data-loader--all.jar --help ``` If successful, you'll see the list of available commands and options. You can pull the Docker image from the [Scalar container registry](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-data-loader-cli) by running the following command, replacing the contents in the angle brackets as described: ```console docker pull ghcr.io/scalar-labs/scalardb-cluster-data-loader-cli: ``` You can run Data Loader commands by using the container. The following example shows how to verify the installation: ```console docker run --rm ghcr.io/scalar-labs/scalardb-cluster-data-loader-cli: --help ``` If successful, you'll see the list of available commands and options. :::note All command examples in this documentation use the JAR file syntax. You can run the same commands with the container by replacing `java -jar scalardb-cluster-data-loader--all.jar` with the Docker equivalent and mounting your local files as volumes. For example: ```console # JAR syntax java -jar scalardb-cluster-data-loader--all.jar import \ --config scalardb.properties --file data.json ... # Docker equivalent docker run --rm \ -v ./scalardb.properties:/scalardb.properties \ -v ./data.json:/data.json \ ghcr.io/scalar-labs/scalardb-cluster-data-loader-cli: \ import --config /scalardb.properties --file /data.json ... ``` ::: Select your preferred method to set up Data Loader, and follow the instructions. Download **`scalardb-data-loader-.jar`** from the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page. Verify the installation by running the following command, replacing `` with the version number: ```console java -jar scalardb-data-loader-.jar --help ``` If successful, you'll see the list of available commands and options. You can pull the Docker image from the [Scalar container registry](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-data-loader-cli) by running the following command, replacing the contents in the angle brackets as described: ```console docker pull ghcr.io/scalar-labs/scalardb-data-loader-cli: ``` You can run Data Loader commands by using the container. The following example shows how to verify the installation: ```console docker run --rm ghcr.io/scalar-labs/scalardb-data-loader-cli: --help ``` If successful, you'll see the list of available commands and options. :::note All command examples in this documentation use the JAR file syntax. You can run the same commands with the container by replacing `java -jar scalardb-data-loader-.jar` with the Docker equivalent and mounting your local files as volumes. For example: ```console # JAR syntax java -jar scalardb-data-loader-.jar import \ --config scalardb.properties --file data.json ... # Docker equivalent docker run --rm \ -v ./scalardb.properties:/scalardb.properties \ -v ./data.json:/data.json \ ghcr.io/scalar-labs/scalardb-data-loader-cli: \ import --config /scalardb.properties --file /data.json ... ``` ::: ## Importing data This section explains how to use the import function in Data Loader. ### Basic import example The simplest way to import data is with automatic field mapping, where Data Loader matches source file fields to table columns by name. Data Loader supports three file formats: JSON, JSONL (JSON Lines), and CSV. The following examples show how to import each format. **Import a JSON file with automatic mapping** To import a JSON file into your table, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .json \ --format JSON ``` ```console java -jar scalardb-data-loader-.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .json \ --format JSON ``` This command imports the JSON file into the specified table using default settings (INSERT mode, automatic field mapping). **Example JSON file format:** ```json [ { "id": 1, "name": "Product A", "price": 100 }, { "id": 2, "name": "Product B", "price": 200 } ] ``` **Import a JSONL (JSON Lines) file with automatic mapping** To import a JSONL file into your table, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .jsonl \ --format JSONL ``` ```console java -jar scalardb-data-loader-.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .jsonl \ --format JSONL ``` This command imports the JSONL file into the specified table using default settings (INSERT mode, automatic field mapping). **Example JSONL file format:** ```json {"id": 1, "name": "Product A", "price": 100} {"id": 2, "name": "Product B", "price": 200} ``` **Import a CSV file with automatic mapping** To import a CSV file into your table, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .csv \ --format CSV ``` ```console java -jar scalardb-data-loader-.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .csv \ --format CSV ``` This command imports the CSV file into the specified table using default settings (INSERT mode, automatic field mapping). **Example CSV file format:** ```csv id,name,price 1,Product A,100 2,Product B,200 ``` :::note The CSV file must include a header row with column names that match your table columns. If your CSV file doesn't have a header row, use the `--header` flag to specify column names. ::: :::warning When importing data by using direct database access, keep the following in mind to ensure data consistency: - **With ScalarDB Cluster in your environment:** Stop the cluster during the operation. - **Without ScalarDB Cluster:** Stop other processes that update the databases during the operation. ::: ### Common import scenarios This section describes common import scenarios. #### Update existing records instead of inserting new ones To update existing records instead of inserting new ones, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .json \ --format JSON \ --import-mode UPDATE ``` ```console java -jar scalardb-data-loader-.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .json \ --format JSON \ --import-mode UPDATE ``` #### Import with custom field mapping using a control file If your source file fields don't match your table column names, you can use a control file to define custom mapping rules. For details on creating control files and mapping configurations, see [Custom data mapping](#custom-data-mapping). To import with custom field mapping using a control file, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --file .json \ --format JSON \ --control-file .json ``` ```console java -jar scalardb-data-loader-.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --file .json \ --format JSON \ --control-file .json ``` #### Import CSV data with a custom delimiter To import CSV data with a custom delimiter, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .csv \ --format CSV \ --delimiter ";" ``` ```console java -jar scalardb-data-loader-.jar import \ --config scalardb.properties \ --mode TRANSACTION \ --namespace \ --table \ --file .csv \ --format CSV \ --delimiter ";" ``` ### Configuring your import For more control over the import process, you can configure various options: #### Import modes Choose the appropriate import mode based on your use case: - **INSERT** (default): Insert new records only. Fails if data already exists based on partition and clustering keys. - **UPDATE**: Update existing records only. Fails if data doesn't exist. - **UPSERT**: Insert new records or update existing ones based on partition and clustering keys. :::note When using INSERT mode, you must have matching fields in the source file for each target column (via automatic or custom data mapping). This requirement also applies when an UPSERT operation results in an INSERT operation. ::: ### Command-line flags The following is a list of flags (options) that can be used with the import function in Data Loader: | Flag | Description | Usage | | ---------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------- | | `--mode` | The access mode for Data Loader. Required. Supported modes are `STORAGE` (single CRUD) and `TRANSACTION` (Consensus Commit). When using ScalarDB Cluster, the mode must match the `scalar.db.transaction_manager` setting in the Cluster configuration. When accessing databases directly, the mode must match the `scalar.db.transaction_manager` setting. | `scalardb-data-loader --mode TRANSACTION` | | `--config` | The path to the `.properties` file for ScalarDB. This file should contain either cluster connection settings or direct database connection settings, depending on your chosen access pattern. If omitted, the tool looks for a file named `scalardb.properties` in the current folder. | `scalardb-data-loader --config scalardb.properties` | | `--namespace` | The namespace to import table data to. Required when no control file is provided. | `scalardb-data-loader --namespace namespace` | | `--table` | The name of the table to import data to. Required when no control file is provided. | `scalardb-data-loader --table tableName` | | `--import-mode` | Mode to import data into the ScalarDB table. Supported modes are `INSERT`, `UPDATE`, and `UPSERT`. Optional. The default value is `INSERT`. | `scalardb-data-loader --import-mode UPDATE` | | `--require-all-columns` | If set, data rows cannot be imported if they are missing columns. Optional. The default value is `false`. | `scalardb-data-loader --require-all-columns` | | `--file` | The path to the file that will be imported. Required. | `scalardb-data-loader --file ` | | `--log-dir` | Directory where log files should be stored. Optional. The default value is `logs`. | `scalardb-data-loader --log-dir ` | | `--log-success` | Enable logging of successfully processed records. Optional. The default value is `false`. | `scalardb-data-loader --log-success` | | `--log-raw-record` | Include the original source record in the log file output. Optional. The default value is `false`. | `scalardb-data-loader --log-raw-record` | | `--max-threads` | Maximum number of threads to use for parallel processing. The default value is the number of available processors. | `scalardb-data-loader --max-threads 10` | | `--format` | The format of the import file. Supported formats are `JSON`, `JSONL`, and `CSV`. Optional. The default value is `JSON`. | `scalardb-data-loader --format CSV` | | `--ignore-nulls` | Ignore null values in the source file during import. This means that existing data will not be overwritten by null values. Optional. The default value is `false`. | `scalardb-data-loader --ignore-nulls` | | `--pretty-print` | **(JSON/JSONL only)** Enable pretty printing for JSON output in log files. Optional. The default value is `false`. | `scalardb-data-loader --pretty-print` | | `--control-file` | The path to the JSON control file that specifies the rules for custom data mapping and/or multi-table import. | `scalardb-data-loader --control-file control.json` | | `--control-file-validation` | The validation level for the control file. Supported levels are `MAPPED`, `KEYS`, and `FULL`. Optional. The default level is `MAPPED`. | `scalardb-data-loader --control-file-validation FULL` | | `--delimiter` | **(CSV only)** Delimiter character used in the CSV import file. The default delimiter is a comma. | `scalardb-data-loader --delimiter ";"` | | `--header` | **(CSV only)** Specify the header row when the import file contains CSV data and does not have a header row. Provide the column names as a single, delimiter-separated list. If you change `--delimiter`, use the same delimiter in the header value. | `scalardb-data-loader --header id,name,price` | | `--data-chunk-size` | Number of records to load into memory for processing before moving to the next batch. This controls memory usage, not transaction boundaries. Optional. The default value is `500`. | `scalardb-data-loader --data-chunk-size 1000` | | `--data-chunk-queue-size` | Maximum queue size for loaded records waiting to be processed. Optional. The default value is `256`. | `scalardb-data-loader --data-chunk-queue-size 100` | | `--split-log-mode` | Split log file into multiple files based on data chunks. Optional. The default value is `false`. | `scalardb-data-loader --split-log-mode` | | `--transaction-size` | Group size of put operations per transaction commit. Specifies how many records are committed together in a single transaction. Only supported when using Consensus Commit. Optional. The default value is `100`. | `scalardb-data-loader --transaction-size 200` | ### Data mapping This section explains the two data-mapping types: automatic data mapping and custom data mapping. #### Automatic data mapping If no control file is provided, Data Loader will automatically map the fields in the source data to the available columns in the ScalarDB table. If the name doesn't match, and if all columns are required, a validation error will occur. If that occurs, importing the record will fail and the result will be added to the failed output log. #### Custom data mapping If the source fields don't match the target column name, you must use a control file. In the control file, you will need to specify the custom mapping rules for the field names. For example, the following control file maps the field `source_field_name` in the source file to `target_column_name` in the target table: ```json { "tables": [{ "namespace": "", "table_name": "", "mappings": [{ "source_field": "", "target_column": "" }] } ] } ``` ### Control file To allow for custom data mapping or multi-table importing, Data Loader supports configuration via a JSON control file. This file needs to be passed in via the `--control-file` argument when starting Data Loader. #### Control file validation levels To enforce validation on the control file, Data Loader allows you to specify the validation level. Based on the set level, Data Loader will run a pre-check and validate the control file based on the level rules. The following levels are supported: | Level | What It Validates | When to Use | | ----- | ----------------- | ----------- | | FULL | All table columns have mappings | Ensuring your control file covers every column | | KEYS | Only partition and clustering keys have mappings | Partial updates where you only care about key columns | | MAPPED (default) | Only the mappings you specify are valid | You trust your control file and want minimal validation | The validation level is optional and can be set via the `--control-file-validation` argument when starting Data Loader. :::note This validation is run as a pre-check and doesn't mean that the import process will automatically succeed. For example, if the level is set to MAPPED and the control file doesn't contain mappings for each column for an INSERT operation, the import process will still fail because all columns are required to be mapped for an INSERT operation. ::: ### Multi-table import Data Loader supports multi-table target importing, allowing you to import a single row from a JSON, JSON Lines, or CSV file into multiple tables by specifying table-mapping rules in the control file. :::note Multi-table import requires a control file. This feature is not supported without a control file. ::: When using multi-table import in ScalarDB `TRANSACTION` mode, a separate transaction is created for each table that a source row is imported into. For example, if a source row is mapped to two tables in the control file, two separate transactions will be created. **Example: Import one source row into multiple tables** A JSON source record with multiple fields: ```json [{ "field1": "value1", "field2": "value2", "field3": "value3" }] ``` Can be imported into multiple tables by using a control file that maps different fields to different tables: ```json { "tables": [{ "namespace": "", "table_name": "", "mappings": [{ "source_field": "field1", "target_column": "" }, { "source_field": "field2", "target_column": "" }] }, { "namespace": "", "table_name": "", "mappings": [{ "source_field": "field1", "target_column": "" }, { "source_field": "field3", "target_column": "" }] } ] } ``` This configuration imports `field1` and `field2` into ``, and `field1` and `field3` into ``. ### Output logs Data Loader creates detailed log files for every import operation, tracking both successful and failed records. #### Log file locations By default, Data Loader generates two log files in the `logs/` directory: - **Success log:** Contains all successfully imported records. - **Failure log:** Contains records that failed to import with error details. You can change the log directory using the `--log-dir` flag. #### Understanding the logs Both log files include a `data_loader_import_status` field added to each record: **In the success log:** - Shows whether each record was inserted (new) or updated (existing). - Includes transaction details when running in `TRANSACTION` mode. **In the failure log:** - Explains why each record failed to import. - Lists specific validation errors or constraint violations. #### Retrying failed imports The failure log is designed for easy recovery: 1. **Edit the failed records** in the failure log to fix the issues (for example, adding missing columns and correcting invalid values). 2. **Use the edited file directly** as input for a new import operation. 3. **No cleanup needed** since the `data_loader_import_status` field is automatically ignored during re-import. :::tip Enable `--log-success` to log successfully imported records, and use `--log-raw-record` to include the original source data in the log output. ::: #### Log format | Field | Description | | -------------- | ------------------------------------------------------------ | | `action` | The result of the import process for the data record: UPDATE, INSERT, or FAILED_DURING_VALIDATION. | | `namespace` | The name of the namespace of the table that the data is imported into. | | `tableName` | The name of the table that the data is imported into. | | `is_data_mapped` | Whether custom data mapping was applied or not based on an available control file. | | `tx_id` | The transaction ID. Only available if Data Loader is run in `TRANSACTION` mode. | | `value` | The final value, after optional data mapping, that Data Loader uses in the `PUT` operation. | | `row_number` | The line number or record number of the source data. | | `errors` | A list of validation or other errors for operations that failed during the import process. | The following is an example of a JSON-formatted log file that shows a successful import: ```json [{ "column_1": 1, "column_2": 2, "column_n": 3, "data_loader_import_status": { "results": [{ "action": "UPDATE", "namespace": "namespace1", "tableName": "table1", "is_data_mapped": true, "tx_id": "value", "value": "value", "row_number": "value" }] } }] ``` The following shows an example of a JSON-formatted log file of a failed import: ```json [{ "column_1": 1, "column_2": 2, "column_n": 3, "data_loader_import_status": { "results": [{ "action": "FAILED_DURING_VALIDATION", "namespace": "namespace1", "tableName": "table1", "is_data_mapped": false, "value": "value", "row_number": "value", "errors": [ "missing columns found during validation" ] }] } }] ``` ### Duplicate data :::warning Make sure your import file doesn't contain duplicate records with the same partition keys and/or clustering keys. Data Loader does not detect or prevent duplicates in the source file. ::: In ScalarDB `TRANSACTION` mode, attempting to update the same target data in fast succession will result in `No Mutation` errors. Data Loader does not handle these errors automatically. Failed data rows will be logged to the failed import result output file, where you can review and re-import them later if needed. ## Exporting data This section explains how to use the export function in Data Loader. :::note Export operations use the same access patterns and configuration as imports. See the [Configuration patterns](#configuration-patterns) section for details on configuring Data Loader for ScalarDB Cluster access or direct database access. ::: ### Basic export example The simplest way to export data is to export an entire table. Data Loader performs a ScalarDB scan operation and exports the results to a file. Data Loader supports three export formats: JSON, JSONL (JSON Lines), and CSV. The following examples show how to export to each format. **Export an entire table to JSON** To export a table to JSON format, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar export \ --config scalardb.properties \ --namespace \ --table \ --format JSON ``` ```console java -jar scalardb-data-loader-.jar export \ --config scalardb.properties \ --namespace \ --table \ --format JSON ``` This command exports all data from the specified table to a JSON file in the current directory. The output file will be automatically named by using the format `export..
..json`. **Example JSON output format:** ```json [ { "id": 1, "name": "Product A", "price": 100 }, { "id": 2, "name": "Product B", "price": 200 } ] ``` **Export an entire table to JSONL** To export a table to JSONL (JSON Lines) format, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar export \ --config scalardb.properties \ --namespace \ --table \ --format JSONL ``` ```console java -jar scalardb-data-loader-.jar export \ --config scalardb.properties \ --namespace \ --table \ --format JSONL ``` This command exports all data from the specified table to a JSONL file in the current directory. The output file will be automatically named by using the format `export..
..jsonl`. **Example JSONL output format:** ```json {"id": 1, "name": "Product A", "price": 100} {"id": 2, "name": "Product B", "price": 200} ``` **Export an entire table to CSV** To export a table to CSV format, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar export \ --config scalardb.properties \ --namespace \ --table \ --format CSV ``` ```console java -jar scalardb-data-loader-.jar export \ --config scalardb.properties \ --namespace \ --table \ --format CSV ``` This command exports all data from the specified table to a CSV file in the current directory. The output file will be automatically named by using the format `export..
..csv`. **Example CSV output format:** ```csv id,name,price 1,Product A,100 2,Product B,200 ``` :::note By default, CSV exports include a header row with column names. Use the `--no-header` flag to exclude the header row if needed. ::: :::warning When exporting data by using direct database access, keep the following in mind to ensure data consistency: - **With ScalarDB Cluster in your environment:** Stop the cluster during the operation. - **Without ScalarDB Cluster:** Stop other processes that update the databases during the operation. ::: :::warning[For full table exports only] To export an entire table without specifying a partition key (full table scan), you must enable cross-partition scanning: - **With ScalarDB Cluster:** Enable cross-partition scanning in your ScalarDB Cluster configuration. - **Without ScalarDB Cluster (direct database access):** Enable cross-partition scanning in your `scalardb.properties` file: ```properties scalar.db.cross_partition_scan.enabled=true ``` If this setting is not enabled, full table exports will fail. For details about this configuration, see [Cross-partition scan configurations](./configurations.mdx#cross-partition-scan-configurations). When exporting a specific partition by using `--partition-key`, cross-partition scanning is not needed. ::: ### Common export scenarios The following are some common data-exporting scenarios. #### Export data to a specific file and format To export data to a specific file and format, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar export \ --config scalardb.properties \ --namespace \ --table \ --output-file .csv \ --format CSV ``` ```console java -jar scalardb-data-loader-.jar export \ --config scalardb.properties \ --namespace \ --table \ --output-file .csv \ --format CSV ``` #### Export specific columns only To export specific columns only, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar export \ --config scalardb.properties \ --namespace \ --table \ --projection ,, ``` ```console java -jar scalardb-data-loader-.jar export \ --config scalardb.properties \ --namespace \ --table \ --projection ,, ``` #### Export data for a specific partition key To export data for a specific partition key, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar export \ --config scalardb.properties \ --namespace \ --table \ --partition-key = ``` ```console java -jar scalardb-data-loader-.jar export \ --config scalardb.properties \ --namespace \ --table \ --partition-key = ``` #### Export with a row limit To export with a row limit, run the following command, replacing the contents of the angle brackets as described: ```console java -jar scalardb-cluster-data-loader--all.jar export \ --config scalardb.properties \ --namespace \ --table \ --limit 1000 ``` ```console java -jar scalardb-data-loader-.jar export \ --config scalardb.properties \ --namespace \ --table \ --limit 1000 ``` ### Command-line flags The following is a list of flags (options) that can be used with the export function in Data Loader: | Flag | Description | Usage | | ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------ | | `--config` | The path to the `.properties` file for ScalarDB. This file should contain either cluster connection settings or direct database connection settings, depending on your chosen access pattern. If omitted, the tool looks for a file named `scalardb.properties` in the current folder. | `scalardb-data-loader --config scalardb.properties` | | `--namespace` | The namespace to export table data from. Required. | `scalardb-data-loader --namespace namespace` | | `--table` | The name of the table to export data from. Required. | `scalardb-data-loader --table tableName` | | `--partition-key` | A specific partition key to export data from. Specify in the format `key=value`. By default, this option exports all data from the specified table. | `scalardb-data-loader --partition-key id=100` | | `--sort-by` | Clustering key sorting order. Supported values are `asc` and `desc`. This flag is only applicable when using `--partition-key`. | `scalardb-data-loader --sort-by asc` | | `--projection` | Columns to include in the export. Provide as a comma-separated list. You can also repeat the argument to provide multiple projections. | `scalardb-data-loader --projection column1,column2` | | `--start-key` | Clustering key and value to mark the start of the scan. Specify in the format `key=value`. This flag is only applicable when using `--partition-key`. | `scalardb-data-loader --start-key timestamp=1000` | | `--start-inclusive` | Make the start key inclusive. The default value is `true`. This flag is only applicable when using `--partition-key`. | `scalardb-data-loader --start-inclusive false` | | `--end-key` | Clustering key and value to mark the end of the scan. Specify in the format `key=value`. This flag is only applicable when using `--partition-key`. | `scalardb-data-loader --end-key timestamp=9999` | | `--end-inclusive` | Make the end key inclusive. The default value is `true`. This flag is only applicable when using `--partition-key`. | `scalardb-data-loader --end-inclusive false` | | `--limit` | Maximum number of rows to export. If omitted, there is no limit. | `scalardb-data-loader --limit 1000` | | `--output-dir` | Directory where the exported file should be saved. The default is the current directory.

Note: Data Loader doesn't create the output directory for you, so the directory needs to already exist. | `scalardb-data-loader --output-dir ./exports` | | `--output-file` | The name of the output file for the exported data. If omitted, the tool will save the file with the following name format:
`export..
..` | `scalardb-data-loader --output-file output.json` | | `--format` | Format of the exported data file. Supported formats are `JSON`, `JSONL`, and `CSV`. The default value is `JSON`. | `scalardb-data-loader --format CSV` | | `--delimiter` | **(CSV only)** Delimiter character for CSV files. The default delimiter is a comma. | `scalardb-data-loader --delimiter ";"` | | `--no-header` | **(CSV only)** Exclude header row in CSV files. The default value is `false`. | `scalardb-data-loader --no-header` | | `--pretty-print` | **(JSON/JSONL only)** Pretty-print JSON output. The default value is `false`. | `scalardb-data-loader --pretty-print` | | `--data-chunk-size` | Number of records to load into memory for processing before moving to the next batch. This controls memory usage. The default value is `200`. | `scalardb-data-loader --data-chunk-size 500` | | `--max-threads` | Maximum number of threads to use for parallel processing. The default value is the number of available processors. | `scalardb-data-loader --max-threads 10` | ================================================ FILE: docs/data-modeling.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Model Your Data Data modeling (or in other words, designing your database schemas) is the process of conceptualizing and visualizing how data will be stored and used by identifying the patterns used to access data and the types of queries to be performed within business operations. This page first explains the ScalarDB data model and then describes how to design your database schemas based on the data model. ## ScalarDB data model ScalarDB's data model is an extended key-value model inspired by the Bigtable data model. It is similar to the relational model but differs in several ways, as described below. The data model is chosen to abstract various databases, such as relational databases, NoSQL databases, and NewSQL databases. The following diagram shows an example of ScalarDB tables, each of which is a collection of records. This section first explains what objects, such as tables and records, ScalarDB defines and then describes how to locate the records. ![ScalarDB data model](images/scalardb_data_model.png) ### Objects in ScalarDB The ScalarDB data model has several objects. #### Namespace A namespace is a collection of tables analogous to an SQL namespace or database. #### Table A table is a collection of partitions. A namespace most often contains one or more tables, each identified by a name. #### Partition A partition is a collection of records and a unit of distribution to nodes, whether logical or physical. Therefore, records within the same partition are placed in the same node. ScalarDB assumes multiple partitions are distributed by hashing. #### Record / row A record or row is a set of columns that is uniquely identifiable among all other records. #### Column A column is a fundamental data element and does not need to be broken down any further. Each record is composed of one or more columns. Each column has a data type. For details about the data type, refer to [Database Adapters](database-adapters.mdx). #### Secondary index A secondary index is a sorted copy of a column in a single base table. Each index entry is linked to a corresponding table partition. ScalarDB currently doesn't support multi-column indexes, so it can create indexes with only one column. ### How to locate records This section discusses how to locate records from a table. #### Primary key A primary key uniquely identifies each record; no two records can have the same primary key. Therefore, you can locate a record by specifying a primary key. A primary key comprises a partition key and, optionally, a clustering key. #### Partition key A partition key uniquely identifies a partition. A partition key comprises a set of columns, which are called partition key columns. When you specify only a partition key, you can get a set of records that belong to the partition. #### Clustering key A clustering key uniquely identifies a record within a partition. It comprises a set of columns called clustering-key columns. When you want to specify a clustering key, you should specify a partition key for efficient lookups. When you specify a clustering key without a partition key, you end up scanning all the partitions. Scanning all the partitions is time consuming, especially when the amount of data is large, so only do so at your own discretion. Records within a partition are assumed to be sorted by clustering-key columns, specified as a clustering order. Therefore, you can specify a part of clustering-key columns in the defined order to narrow down the results to be returned. #### Index key An index key identifies records by looking up the key in indexes. An index key lookup spans all the partitions, so it is not necessarily efficient, especially if the selectivity of a lookup is not low. ## How to design your database schemas You can design your database schemas similarly to the relational model, but there is a basic principle and are a few best practices to follow. ### Query-driven data modeling In relational databases, data is organized in normalized tables with foreign keys used to reference related data in other tables. The queries that the application will make are structured by the tables, and the related data is queried as table joins. Although ScalarDB supports join operations in ScalarDB SQL, data modeling should be more query-driven, like NoSQL databases. The data access patterns and application queries should determine the structure and organization of tables. ### Best practices This section describes best practices for designing your database schemas. #### Consider data distribution Preferably, you should try to balance loads to partitions by properly selecting partition and clustering keys. For example, in a banking application, if you choose an account ID as a partition key, you can perform any account operations for a specific account within the partition to which the account belongs. So, if you operate on different account IDs, you will access different partitions. On the other hand, if you choose a branch ID as a partition key and an account ID as a clustering key, all the accesses to a branch's account IDs go to the same partition, causing an imbalance in loads and data sizes. In addition, you should choose a high-cardinality column as a partition key because creating a small number of large partitions also causes an imbalance in loads and data sizes. #### Try to read a single partition Because of the data model characteristics, single partition lookup is most efficient. If you need to issue a scan or select a request that requires multi-partition lookups or scans, which you can [enable with cross-partition scan](configurations.mdx#cross-partition-scan-configurations), do so at your own discretion and consider updating the schemas if possible. For example, in a banking application, if you choose email as a partition key and an account ID as a clustering key, and issue a query that specifies an account ID, the query will span all the partitions because it cannot identify the corresponding partition efficiently. In such a case, you should always look up the table with an account ID. :::note If you read multiple partitions on a relational database with proper indexes, your query might be efficient because the query is pushed down to the database. ::: #### Try to avoid using secondary indexes Similarly to the above, if you need to issue a scan or select a request that uses a secondary index, the request will span all the partitions of a table. Therefore, you should try to avoid using secondary indexes. If you need to use a secondary index, use it through a low-selectivity query, which looks up a small portion. As an alternative to secondary indexes, you can create another table that works as a clustered index of a base table. For example, assume there is a table with three columns: `table1(A, B, C)`, with the primary key `A`. Then, you can create a table like `index-table1(C, A, B)` with `C` as the primary key so that you can look up a single partition by specifying a value for `C`. This approach could speed up read queries but might create more load to write queries because you need to write to two tables by using ScalarDB transactions. :::note There are plans to have a table-based secondary-index feature in ScalarDB in the future. ::: #### Consider data is assumed to be distributed by hashing In the current ScalarDB data model, data is assumed to be distributed by hashing. Therefore, you can't perform range queries efficiently without a partition key. If you want to issue range queries efficiently, you need to do so within a partition. However, if you follow this approach, you must specify a partition key. This can pose scalability issues as the range queries always go to the same partition, potentially overloading it. This limitation is not specific to ScalarDB but to databases where data is distributed by hashing for scalability. :::note If you run ScalarDB on a relational database with proper indexes, your range query might be efficient because the query is pushed down to the database. ::: ================================================ FILE: docs/database-adapters.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Database Adapters ScalarDB provides a database-agnostic abstraction layer that enables applications to perform ACID transactions across different databases without being tied to any specific database product. To achieve this, ScalarDB uses database adapters that translate its unified data model into the native constructs of each supported database. This document describes how each adapter maps the logical data model of ScalarDB—namespaces, tables, partition keys, clustering keys, and columns—to the underlying database and what limitations apply to each adapter. :::note When you use Consensus Commit as the transaction protocol, ScalarDB adds metadata columns to each table in the underlying database. These columns are managed by the transaction protocol, not the adapter. For details, see [Consensus Commit](consensus-commit.mdx). ::: For the background on the logical data model of ScalarDB, see [Data Modeling](data-modeling.mdx). For supported database versions, see [Requirements](requirements.mdx). For configuration guidance for each database, see [Database Configurations](database-configurations.mdx). ## JDBC adapter The JDBC adapter supports relational databases through JDBC connections. The following databases are supported: MySQL, MariaDB, TiDB, PostgreSQL, YugabyteDB, AlloyDB, Amazon Aurora (MySQL-compatible and PostgreSQL-compatible), Oracle Database, SQL Server, IBM Db2, Spanner (PostgreSQL dialect), and SQLite. MariaDB, TiDB, and Amazon Aurora MySQL-compatible edition follow the same mapping as MySQL. YugabyteDB, AlloyDB, and Amazon Aurora PostgreSQL-compatible edition follow the same mapping as PostgreSQL. ### Namespace and table mapping How ScalarDB namespaces map to native database constructs depends on the RDBMS. The following table summarizes the mapping for each supported database. | RDBMS | ScalarDB namespace maps to | Notes | |-------|---------------------------|-------| | MySQL, MariaDB, TiDB | Database | In MySQL, a database and a schema are synonymous. | | PostgreSQL, YugabyteDB, AlloyDB | Schema | Created within the connected database. | | Oracle Database | Schema (User) | In Oracle, creating a user automatically creates a schema of the same name. ScalarDB creates a dedicated user for each namespace, and tables are stored in the corresponding schema. | | SQL Server | Schema | Created within the connected database. | | IBM Db2 | Schema | Created within the connected database. | | Spanner (PostgreSQL dialect) | Schema | Created within the connected database. | | SQLite | Table name prefix | Since SQLite is a single-file database, the namespace is prepended to the table name with a `$` separator (for example, `my_namespace$my_table`). | ScalarDB table names map directly to database table names (with the exception of SQLite, which uses the prefixed format described above). :::warning For SQLite, namespace names and table names cannot contain the `$` character because ScalarDB uses it as a separator. ::: ### Key and index mapping The ScalarDB partition key and clustering key columns together form the primary key of the underlying database table. ScalarDB secondary indexes are created as standard database indexes. ### Data-type mapping The following table shows how ScalarDB data types map to native column types in each JDBC database. | ScalarDB | MySQL, MariaDB, TiDB | PostgreSQL, YugabyteDB, AlloyDB | Oracle | Spanner (PostgreSQL dialect) | SQL Server | Db2 | SQLite | |----------|----------------------|------------------------|--------|---------|------------|-----|--------| | BOOLEAN | BOOLEAN | BOOLEAN | NUMBER(1) | BOOLEAN | BIT | BOOLEAN | BOOLEAN | | INT | INT | INT | NUMBER(10) | BIGINT | INT | INT | INT | | BIGINT | BIGINT | BIGINT | NUMBER(19) | BIGINT | BIGINT | BIGINT | BIGINT | | FLOAT | REAL | REAL | BINARY_FLOAT | FLOAT | FLOAT(24) | REAL | FLOAT | | DOUBLE | DOUBLE | DOUBLE PRECISION | BINARY_DOUBLE | DOUBLE PRECISION | FLOAT | DOUBLE | DOUBLE | | TEXT | LONGTEXT | TEXT | VARCHAR2(4000) | TEXT | VARCHAR(8000) | VARCHAR(32672) | TEXT | | BLOB | LONGBLOB | BYTEA | BLOB | BYTEA | VARBINARY(8000) | BLOB(2G) | BLOB | | DATE | DATE | DATE | DATE | DATE | DATE | DATE | INT | | TIME | TIME(6) | TIME | TIMESTAMP(6) | TIMESTAMP WITH TIME ZONE | TIME(6) | TIMESTAMP(6) | BIGINT | | TIMESTAMP | DATETIME(3) | TIMESTAMP | TIMESTAMP(3) | TIMESTAMP WITH TIME ZONE | DATETIME2(3) | TIMESTAMP(3) | BIGINT | | TIMESTAMPTZ | DATETIME(3) | TIMESTAMP WITH TIME ZONE | TIMESTAMP(3) WITH TIME ZONE | TIMESTAMP WITH TIME ZONE | DATETIMEOFFSET(3) | TIMESTAMP(3) | BIGINT | When TEXT or BLOB columns are used as a partition key, clustering key, or secondary index key, some databases use a smaller, fixed-size type instead of the default type shown above. | Database | TEXT key column type | BLOB key column type | |----------|---------------------|----------------------| | MySQL, MariaDB, TiDB | VARCHAR(*size*) | VARBINARY(*size*) | | PostgreSQL, YugabyteDB, AlloyDB | VARCHAR(10485760) | BYTEA (no conversion) | | Oracle | VARCHAR2(*size*) | Not supported as key | | Db2 | VARCHAR(*size*) | Not supported as key | The *size* in the table above defaults to 128 and can be configured per database through the following properties. The minimum allowed value is 64. - `scalar.db.jdbc.mysql.variable_key_column_size` — Applies to MySQL, MariaDB, and TiDB. - `scalar.db.jdbc.oracle.variable_key_column_size` — Applies to Oracle Database. - `scalar.db.jdbc.db2.variable_key_column_size` — Applies to IBM Db2. For the enforced value ranges of each ScalarDB data type, see [Value ranges and precision](#value-ranges-and-precision). ### Limitations The following limitations apply to specific databases when using the JDBC adapter. **IBM Db2:** - BLOB cannot be used as a partition key, clustering key, secondary index key, or ordering column in a cross-partition scan. - FLOAT / DOUBLE minimum values are Float.MIN_NORMAL / Double.MIN_NORMAL, not Java's MIN_VALUE. - Rename is not supported for partition-key, clustering-key, or secondary-index columns (non-key columns can be renamed). - ALTER COLUMN TYPE from BLOB → TEXT is not supported. **Oracle Database:** - BLOB cannot be used as a partition key, clustering key, secondary index key, condition column in a Get or Scan operation, or ordering column in a cross-partition scan. - ALTER COLUMN TYPE supports only INT → BIGINT. All other conversions, including widening FLOAT → DOUBLE and converting to TEXT, will throw an exception. **YugabyteDB:** - FLOAT and DOUBLE cannot be used as a partition key, clustering key, or secondary index key. - Import table is unsupported. **SQLite:** - SQLite does not fully support concurrent access. Use SQLite only for development and testing purposes, not for production workloads. - Import table and virtual table are entirely unsupported. **TiDB:** - TiDB does not support the SERIALIZABLE isolation level. ScalarDB uses REPEATABLE READ instead. - TiDB does not support ALTER COLUMN TYPE from BLOB to TEXT. **Spanner (PostgreSQL dialect):** - FLOAT columns cannot be used as a partition key, clustering key, or secondary index key. - Table and column renaming are not supported. - ALTER COLUMN TYPE is supported only for BLOB → TEXT conversions. - The LIKE expression escape character is fixed to `\` and cannot be configured or disabled. ## DynamoDB adapter The DynamoDB adapter maps ScalarDB operations to Amazon DynamoDB. ### Namespace and table mapping DynamoDB does not have a native namespace concept. ScalarDB represents each table as a DynamoDB table whose name combines the namespace and table name with a dot (`.`) separator. For example, a ScalarDB table named `orders` in the namespace `my_app` becomes a DynamoDB table named `my_app.orders`. You can optionally configure a namespace prefix through the `scalar.db.dynamo.namespace.prefix` property. When a prefix is set, it is prepended to the table name. For example, with the prefix `prod_`, the DynamoDB table name becomes `prod_my_app.orders`. ### Key and index mapping Since DynamoDB supports only a single partition key (hash key) and a single sort key per table, ScalarDB handles multi-column keys by encoding and concatenating all partition key columns into a single binary hash key, and all clustering key columns into a single binary sort key that preserves the specified sort order. ScalarDB creates a Global Secondary Index (GSI) for each secondary index defined in the schema. Non-key columns are stored as individual DynamoDB attributes. ### Data-type mapping The following table shows how ScalarDB data types map to DynamoDB attribute types. | ScalarDB | DynamoDB | Notes | |----------|----------|-------| | BOOLEAN | BOOL | | | INT | N (Number) | | | BIGINT | N (Number) | | | FLOAT | N (Number) | | | DOUBLE | N (Number) | | | TEXT | S (String) | | | BLOB | B (Binary) | | | DATE | N (Number) | Stored as epoch day (days since 1970-01-01). | | TIME | N (Number) | Stored as nano of day (nanoseconds since midnight). | | TIMESTAMP | N (Number) | Stored as a packed value of epoch second and millisecond of second. | | TIMESTAMPTZ | N (Number) | Stored as a packed value of epoch second and millisecond of second in UTC. | For the enforced value ranges of each ScalarDB data type, see [Value ranges and precision](#value-ranges-and-precision). ### Limitations - The DynamoDB item size limit of 400 KB applies to each ScalarDB record, including any transaction metadata columns added by Consensus Commit. - BLOB cannot be used as a partition key or clustering key except for the last column of a composite partition key. - BOOLEAN cannot be used as a secondary index key. - BOOLEAN conditional mutations are limited to EQ, NE, IS NULL, IS NOT NULL. - You must designate a single primary region. Do not read from asynchronously replicated non-primary regions. - Cross-partition scan ordering on non-primary-key columns is not supported. Users who rely on ordering in ScanAll will hit an error. - Schema evolution DDLs, such as drop column, rename column, alter column type, and rename table, are not supported. - Secondary-index columns cannot be set to an empty string or empty BLOB because GSIs don't allow empty keys. ## Cosmos DB for NoSQL adapter The Cosmos DB for NoSQL adapter maps ScalarDB operations to Azure Cosmos DB for NoSQL. ### Namespace and table mapping Each ScalarDB namespace maps to a Cosmos DB database, and each ScalarDB table maps to a Cosmos DB container within that database. ### Key and index mapping Since a Cosmos DB container supports only a single partition key path, ScalarDB handles multi-column partition keys by concatenating all partition key column values into a single string, using a colon (`:`) as the delimiter. For example, if a table has a partition key with columns `tenant_id = "A"` and `user_id = 123`, the concatenated partition key value stored in Cosmos DB is `A:123`. Clustering key columns and non-key columns are stored as properties within each JSON document. ScalarDB configures composite indexes on the container to support efficient ordering by clustering key columns. For secondary indexes, ScalarDB adds the corresponding paths to the container's indexing policy. :::warning Because the colon (`:`) is used as the partition key delimiter, text values in partition key columns must not contain colons. Using colons in partition key text values causes incorrect behavior. ::: ### Data-type mapping The following table shows how ScalarDB data types are represented in Cosmos DB JSON documents. | ScalarDB | Cosmos DB JSON type | Notes | |----------|---------------------|-------| | BOOLEAN | boolean | | | INT | number | | | BIGINT | number | | | FLOAT | number | | | DOUBLE | number | | | TEXT | string | | | BLOB | string | Stored as a Base64-encoded string. | | DATE | number | Stored as epoch day (days since 1970-01-01). | | TIME | number | Stored as nano of day (nanoseconds since midnight). | | TIMESTAMP | number | Stored as a packed value of epoch second and millisecond of second. | | TIMESTAMPTZ | number | Stored as a packed value of epoch second and millisecond of second in UTC. | For the enforced value ranges of each ScalarDB data type, see [Value ranges and precision](#value-ranges-and-precision). ### Limitations - The Cosmos DB document size limit of 2 MB applies to each ScalarDB record, including any transaction metadata added by Consensus Commit. - BIGINT values are restricted to the range -2^53 to 2^53 because Cosmos DB stores numbers internally as double-precision floating-point, which can only represent integers exactly up to 2^53. - BLOB cannot be used as a clustering key. - Text values in partition key columns must not contain colons (`:`). - Primary key column values must not contain the following characters: `/`, `\`, `?`, `#`. These characters are [restricted by Cosmos DB resource IDs](https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.databaseproperties.id?view=azure-dotnet#remarks). - BLOB conditional mutations are limited to EQ, NE, IS NULL, IS NOT NULL (enforced by CosmosOperationChecker). Comparison operators like GT/LT will throw an exception. - The consistency level must be set to Strong or Bounded Staleness. For details, see [Database Configurations](database-configurations.mdx). - You must use a single-region write configuration. - Cross-partition scan ordering on non-primary-key columns is not supported. Users who rely on ordering in ScanAll will hit an error. - Schema evolution DDLs, such as drop column, rename column, alter column type, and rename table, are not supported. ## Cassandra adapter The Cassandra adapter maps ScalarDB operations to Apache Cassandra. ### Namespace and table mapping Each ScalarDB namespace maps directly to a Cassandra keyspace, and each ScalarDB table maps to a Cassandra table within that keyspace. This is the most natural mapping among all adapters because the data model of Cassandra closely mirrors the model of ScalarDB. When ScalarDB creates a keyspace, it uses SimpleStrategy with a replication factor of 1 by default. You can configure the replication strategy and factor through namespace creation options. ### Key and index mapping ScalarDB partition key columns map directly to Cassandra partition key columns, and ScalarDB clustering key columns map to Cassandra clustering columns with the same sort order (ascending or descending). ScalarDB secondary indexes are created as Cassandra secondary indexes. ### Data-type mapping The following table shows how ScalarDB data types map to Cassandra native types. | ScalarDB | Cassandra | Notes | |----------|-----------|-------| | BOOLEAN | boolean | | | INT | int | | | BIGINT | bigint | | | FLOAT | float | | | DOUBLE | double | | | TEXT | text | | | BLOB | blob | | | DATE | date | | | TIME | time | | | TIMESTAMP | — | **Not supported.** Use TIMESTAMPTZ instead. | | TIMESTAMPTZ | timestamp | Cassandra's `timestamp` type stores an absolute instant (epoch-based), which aligns with the TIMESTAMPTZ semantics (a date-time on the UTC time zone) of ScalarDB. | For the enforced value ranges of each ScalarDB data type, see [Value ranges and precision](#value-ranges-and-precision). ### Limitations - The TIMESTAMP data type (without time zone) is not supported with the Cassandra adapter. You must use TIMESTAMPTZ instead. Attempting to create a table with a TIMESTAMP column results in an error. - BLOB size per mutation is capped at 16 MB by default. (configurable with max_mutation_size) - Using PutIf with an IS NULL condition on a non-existing record does not throw NoMutationException. It silently succeeds, which differs from every other adapter. - You must use a single primary cluster. Do not read from asynchronously replicated non-primary clusters. - The Cassandra commit log must be configured for batch or group sync mode. For details, see [Database Configurations](database-configurations.mdx). - ScalarDB uses Cassandra lightweight transactions (LWT) to achieve linearizable operations, which has performance implications compared to regular Cassandra operations. - Cross-partition scan ordering on non-primary-key columns is not supported. Users who rely on ordering in ScanAll will hit an error. - Schema evolution DDLs, such as alter column type and rename table, are not supported. Drop column is supported. Rename column is supported only for primary-key columns (partition key and clustering key columns). ## Object storage adapter The object storage adapter maps ScalarDB operations to object storage services. The following backends are supported: Amazon S3, Azure Blob Storage, and Google Cloud Storage. ### Namespace and table mapping Unlike the other adapters, the object storage adapter stores all data in a single configured bucket (or container, in the case of Azure Blob Storage). Namespaces are not mapped to separate buckets. Instead, ScalarDB uses a hierarchical object key in the format `/
/` to organize data logically within the bucket. ### Key and index mapping Each ScalarDB partition is stored as a single object. The object key follows the format `/
/`, where the partition key portion encodes the partition key column values. For tables with composite partition keys, the individual column values are concatenated by using `!` as the delimiter. The object content is a JSON document that contains all records in the partition. Each record includes the partition key columns, clustering key columns, and non-key column values. Records within the partition are sorted by clustering key. Because each partition is stored as a single object, consider the following when designing table schemas: - Avoid defining clustering keys if the sizes of records are large. - Place frequently read records in the same partition to reduce the number of read operations to object storage. - Place frequently written records in different partitions to avoid write conflicts in object storage. ### Data-type mapping All ScalarDB data types are serialized into JSON when stored in object storage. The mapping follows the same pattern as the Cosmos DB adapter: numeric types and temporal types are stored as JSON numbers, TEXT is stored as a JSON string, BOOLEAN is stored as a JSON boolean, and BLOB is stored as a Base64-encoded string. For the enforced value ranges of each ScalarDB data type, see [Value ranges and precision](#value-ranges-and-precision). ### Limitations - Primary-key text values cannot contain `/` or `!`. - Secondary indexes are not supported at all. - BIGINT values are restricted to the range -2^53 to 2^53 because numbers are serialized as JSON numbers, which can only represent integers exactly up to 2^53. - BLOB data larger than 1.5 GiB cannot be stored. - You must use a single region for the storage bucket or container. - Only certain storage classes are supported: S3 Standard for Amazon S3, Hot tier for Azure Blob Storage, and Standard for Google Cloud Storage. For details, see [Database Configurations](database-configurations.mdx). - All data is stored in a single bucket, so namespace isolation is logical rather than physical. - Cross-partition scan ordering on non-primary-key columns is not supported. Users who rely on ordering in ScanAll will hit an error. - Schema evolution DDLs, such as drop column, rename column, alter column type, and rename table, are not supported. ## Value ranges and precision ScalarDB enforces consistent value ranges across adapters in most cases. The following table shows the supported range and precision for each data type. | Data type | Range | Precision | Notes | |-----------|-------|-----------|-------| | BOOLEAN | `true` or `false` | | | | INT | -2^31 to 2^31 - 1 | | | | BIGINT | -2^63 to 2^63 - 1 | | Restricted to -2^53 to 2^53 in the Cosmos DB for NoSQL and object storage adapters. | | FLOAT | -3.4028235E+38 to 3.4028235E+38 | ~7 decimal digits | | | DOUBLE | -1.7976931348623157E+308 to 1.7976931348623157E+308 | ~15 decimal digits | | | TEXT | Unlimited length | | Maximum size depends on the underlying database. | | BLOB | Unlimited length | | Maximum size depends on the underlying database. | | DATE | 1000-01-01 to 9999-12-31 | Day | | | TIME | 00:00:00.000000 to 23:59:59.999999 | Microsecond | | | TIMESTAMP | 1000-01-01T00:00:00.000 to 9999-12-31T23:59:59.999 | Millisecond | Without time zone. | | TIMESTAMPTZ | 1000-01-01T00:00:00.000Z to 9999-12-31T23:59:59.999Z | Millisecond | On the UTC time zone. | TIMESTAMP represents a date and time without time zone information. TIMESTAMPTZ represents a date and time on the UTC time zone. Despite having similar names, these types have different semantics and are not interchangeable. ## See also - [Model Your Data](data-modeling.mdx) - [Requirements](requirements.mdx) - [Configurations for the Underlying Databases of ScalarDB](database-configurations.mdx) - [ScalarDB Schema Loader](schema-loader.mdx) - [Consensus Commit Protocol](consensus-commit.mdx) - [ScalarDB Design](design.mdx) ================================================ FILE: docs/database-configurations.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Configurations for the Underlying Databases of ScalarDB import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; This document explains how to configure the underlying databases of ScalarDB to make applications that use ScalarDB work correctly and efficiently. ## General requirements for the underlying databases ScalarDB requires each underlying database to provide certain capabilities to run transactions and analytics on the databases. This document explains the general requirements and how to configure each database to achieve the requirements. ### Transactions ScalarDB requires each underlying database to provide at least the following capabilities to run transactions on the databases: - Linearizable read and conditional mutations (write and delete) on a single database record. - Durability of written database records. - Ability to store arbitrary data beside application data in each database record. ### Analytics ScalarDB requires each underlying database to provide the following capability to run analytics on the databases: - Ability to return only committed records. :::note You need to have database accounts that have enough privileges to access the databases through ScalarDB since ScalarDB runs on the underlying databases not only for CRUD operations but also for performing operations like creating or altering schemas, tables, or indexes. ScalarDB basically requires a fully privileged account to access the underlying databases. ::: ## How to configure databases to achieve the general requirements Select your database for details on how to configure it to achieve the general requirements.

Transactions

- Use a single primary server or synchronized multi-primary servers for all operations (no read operations on read replicas that are asynchronously replicated from a primary database). - Use read-committed or stricter isolation levels.

Analytics

- Use read-committed or stricter isolation levels. Select your NoSQL database.

Transactions

- Use a single primary cluster for all operations (no read or write operations in non-primary clusters). - Use `batch` or `group` for `commitlog_sync`. - If you're using Cassandra-compatible databases, those databases must properly support lightweight transactions (LWT).

Analytics

- Not applicable. Cassandra always returns committed records, so there are no Cassandra-specific requirements.

Transactions

- Use a single primary region for all operations with `Strong` or `Bounded Staleness` consistency.

Analytics

- Not applicable. Cosmos DB always returns committed records, so there are no Cosmos DB–specific requirements.

Transactions

- Use a single primary region for all operations. (No read and write operations on global tables in non-primary regions.) - There is no concept for primary regions in DynamoDB, so you must designate a primary region by yourself.

Analytics

- Not applicable. DynamoDB always returns committed records, so there are no DynamoDB-specific requirements.

Transactions

- Use a single region for all operations. For Blob Storage, use a single primary region. - Choose the following storage classes or access tiers for each storage: - **S3**: [S3 Standard](https://aws.amazon.com/s3/storage-classes/) - **Blob Storage**: [Standard general-purpose v2, Hot tier](https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview) - **Cloud Storage**: [Standard storage](https://docs.cloud.google.com/storage/docs/storage-classes#standard) :::note Other storage classes or access tiers can be used, but the ones listed above are verified and supported. ::: ## Recommendations Properly configuring each underlying database of ScalarDB for high performance and high availability is recommended. The following recommendations include some knobs and configurations to update. :::note ScalarDB can be seen as an application of underlying databases, so you may want to try updating other knobs and configurations that are commonly used to improve efficiency. ::: - Use read-committed isolation for better performance. - Follow the performance optimization best practices for each database. For example, increasing the buffer size (for example, `shared_buffers` in PostgreSQL) and increasing the number of connections (for example, `max_connections` in PostgreSQL) are usually recommended for better performance. Select your NoSQL database. - Increase `concurrent_reads` and `concurrent_writes` for high throughput. For details, see the official Cassandra documentation about [`concurrent_writes`](https://cassandra.apache.org/doc/stable/cassandra/configuration/cass_yaml_file.html#concurrent_writes). - Increase the number of Request Units (RUs) for high throughput. - Enable point-in-time restore (PITR). - Enable availability zones. - Increase the number of read capacity units (RCUs) and write capacity units (WCUs) for high throughput. - Enable point-in-time recovery (PITR). :::note Since DynamoDB stores data in multiple availability zones by default, you don’t need to adjust any configurations to improve availability. ::: - Configure a lifecycle rule to delete incomplete multipart uploads as described in [Configuring a bucket lifecycle configuration to delete incomplete multipart uploads](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html) when using S3. - For schema design recommendations specific to object storage, see [Database Adapters](database-adapters.mdx#key-and-index-mapping-4). ================================================ FILE: docs/deploy-overview.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Deploy Overview In this category, you can follow guides to help you become more familiar with deploying ScalarDB, specifically ScalarDB Cluster and ScalarDB Analytics, in local and cloud-based Kubernetes environments. ## Deploy ScalarDB Cluster in a local Kubernetes environment To learn how to deploy ScalarDB Cluster in a local Kubernetes environment by using a Helm Chart and a PostgreSQL database, see [Deploy ScalarDB Cluster Locally](scalardb-cluster/setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). ## Deploy ScalarDB Cluster in a cloud-based Kubernetes environment To learn how to deploy ScalarDB Cluster in a cloud-based Kubernetes environment by using a Helm Chart, see [Deploy ScalarDB Cluster on Amazon Elastic Kubernetes Service (EKS)](scalar-kubernetes/ManualDeploymentGuideScalarDBClusterOnEKS.mdx). ## Deploy ScalarDB Analytics in a local Kubernetes environment To learn how to deploy ScalarDB Analytics in a local Kubernetes environment by using a Helm Chart, see [Deploy ScalarDB Analytics Locally](scalardb-analytics/deployment-local.mdx). ## Deploy ScalarDB Analytics in a public cloud-based environment To learn how to deploy ScalarDB Analytics in a public cloud-based environment, see [Deploy ScalarDB Analytics in Public Cloud Environments](scalardb-analytics/deployment.mdx). ================================================ FILE: docs/design.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Design This document briefly explains the design and implementation of ScalarDB. For what ScalarDB is and its use cases, see [ScalarDB Overview](./overview.mdx). ## Overall architecture ScalarDB is hybrid transaction/analytical processing (HTAP) middleware that sits in between applications and databases. As shown in the following figure, ScalarDB consists of three components: Core, Cluster, and Analytics. ScalarDB basically employs a layered architecture, so the Cluster and Analytics components use the Core component to interact with underlying databases but sometimes bypass the Core component for performance optimization without sacrificing correctness. Likewise, each component also consists of several layers. ![ScalarDB architecture](images/scalardb-architecture.png) ## Components The following subsections explain each component one by one. ### Core ScalarDB Core, which is provided as open-source software under the Apache 2 License, is an integral part of ScalarDB. Core provides a database manager that has an abstraction layer that abstracts underlying databases and adapters (or shims) that implement the abstraction for each database. In addition, it provides a transaction manager on top of the database abstraction that achieves database-agnostic transaction management based on Scalar's novel distributed transaction protocol called [Consensus Commit](./consensus-commit.mdx). Core is provided as a library that offers a simple CRUD interface. ### Cluster ScalarDB Cluster, which is licensed under a commercial license, is a component that provides a clustering solution for the Core component to work as a clustered server. Cluster is mainly designed for OLTP workloads, which have many small, transactional and non-transactional reads and writes. In addition, it provides several enterprise features such as authentication, authorization, encryption at rest, and fine-grained access control (attribute-based access control). Not only does Cluster offer the same CRUD interface as the Core component, but it also offers SQL and GraphQL interfaces. Furthermore, it offers a vector store interface to interact with several vector stores. Since Cluster is provided as a container in a Kubernetes Pod, you can increase performance and availability by having more containers. ### Analytics ScalarDB Analytics, which is licensed under a commercial license, is a component that provides scalable analytical processing for the data managed by the Core component or managed by applications that don’t use ScalarDB. Analytics is mainly designed for OLAP workloads, which have a small number of large, analytical read queries. In addition, it offers a SQL and DataSet API through Spark. Since the Analytics component is provided as a Java package that can be installed on Apache Spark engines, you can increase performance by having more Spark worker nodes. ## Metadata tables ScalarDB manages various types of metadata in the underlying databases to provide its capabilities. The following table summarizes the metadata managed by each component. | Component | Metadata tables | Purpose | Location | | --------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | Core | `scalardb.metadata` | For database schema information | In all the databases under ScalarDB | | Core | `coordinator.state` | For transaction statuses | In one designated database specified to store the Coordinator table | | Core | Application-managed tables | For WAL information | In all the tables accessed by Consensus Commit | | Cluster | `scalardb.users`, `scalardb.namespace_privileges`, `scalardb.table_privileges`, `scalardb.auth_tokens` | For [authentication and authorization](./scalardb-cluster/scalardb-auth-with-sql.mdx) | In one designated database specified to store the scalardb system namespace | | Cluster | `scalardb.encrypted_columns` | For [encryption at rest](./scalardb-cluster/encrypt-data-at-rest.mdx) | In one designated database specified to store the scalardb system namespace | | Cluster | `scalardb.abac_*` | For [attribute-based access control](./scalardb-cluster/authorize-with-abac.mdx) | In one designated database specified to store the scalardb system namespace | | Cluster | Remote replication-managed tables (`transaction_groups`) | For buffering write operations from the primary site in the [remote replication](./scalardb-cluster/remote-replication.mdx) | In the replication database | | Cluster | Remote replication-managed tables (suffixed with `__records`) | For managing the replication states of the backup site database tables in the [remote replication](./scalardb-cluster/remote-replication.mdx) | In the backup site database | | Analytics | All the tables managed by the catalog server | For [data catalog](./scalardb-analytics/design.mdx#universal-data-catalog) | In the catalog server database | :::note If you need to take backups of the databases accessed by ScalarDB, you will also need to take backups of the metadata managed by ScalarDB. For more details, see [How to Back Up and Restore Databases Used Through ScalarDB](./backup-restore.mdx). ::: ## Limitations ScalarDB operates between applications and databases, which leads to certain limitations. This section summarizes the limitations of ScalarDB. ### Applications cannot bypass ScalarDB to run transactions and analytical queries ScalarDB Core offers a database-agnostic transaction capability that operates outside of databases. Therefore, applications must interact with ScalarDB to execute transactions; otherwise, ScalarDB cannot ensure transaction correctness, such as snapshot and serializable isolation. For more details, see [Consensus Commit](./consensus-commit.mdx). Likewise, ScalarDB Analytics offers a scalable analytical query processing capability that operates outside of databases. Therefore, applications must interact with ScalarDB Analytics to execute analytical queries; otherwise, ScalarDB cannot ensure correctness, such as read-committed isolation. For more details, see [ScalarDB Analytics Design](./scalardb-analytics/design.mdx). ### Applications cannot use all the capabilities of the underlying databases ScalarDB serves as an abstraction layer over the underlying databases, which means that applications cannot use all the capabilities and data types of these databases. For instance, ScalarDB does not support database-specific features such as Oracle PL/SQL. ScalarDB has been enhanced to provide features that are commonly found in most supported databases. For a list of features, see [ScalarDB Features](./features.mdx). To learn about the features planned for future releases, see [Roadmap](./roadmap.mdx). ## Further reading For more details about the design and implementation of ScalarDB, see the following documents: - **Speaker Deck presentation:** [ScalarDB: Universal Transaction Manager](https://speakerdeck.com/scalar/scalar-db-universal-transaction-manager) In addition, the following materials were presented at the VLDB 2023 conference: - **Speaker Deck presentation:** [ScalarDB: Universal Transaction Manager for Polystores](https://speakerdeck.com/scalar/scalardb-universal-transaction-manager-for-polystores-vldb23) - **Detailed paper:** [ScalarDB: Universal Transaction Manager for Polystores](https://www.vldb.org/pvldb/vol16/p3768-yamada.pdf) ================================================ FILE: docs/develop-overview.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Develop Overview In this category, you can follow guides to help you become more familiar with ScalarDB, specifically with how to run transactions, analytical queries, and non-transactional storage operations. To get started with developing applications for ScalarDB, see the following sub-categories. ## Run transactions In this sub-category, you can learn how to model your data based on the ScalarDB data model and create schemas. Then, you can learn how to run transactions through the ScalarDB Core library and ScalarDB Cluster, a gRPC server that wraps the Core library. For an overview of this sub-category, see [Run Transactions Overview](develop-run-transactions-overview.mdx). ## Run non-transactional operations In this sub-category, you can learn how to run such non-transactional storage operations. For an overview of this sub-category, see [Run Non-Transactional Operations Overview](develop-run-non-transactional-operations-overview.mdx). ## Run analytical queries To learn how to run analytical queries by using ScalarDB Analytics, see [Run Analytical Queries Through ScalarDB Analytics](scalardb-analytics/run-analytical-queries.mdx). ## Run sample applications In this sub-category, you can learn how to run various sample applications that take advantage of ScalarDB. For an overview of this sub-category, see [Run Sample Applications Overview](scalardb-samples/README.mdx). ================================================ FILE: docs/develop-run-analytical-queries-overview.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Run Analytical Queries Overview In this sub-category, you can learn how to set up and configure ScalarDB Analytics, an analytics component of ScalarDB. After setting it up, you can run analytical queries over ScalarDB-managed databases, which are updated through ScalarDB transactions, and non-ScalarDB-managed databases. To learn how to run analytical queries, see the following: - [Run Analytical Queries Through ScalarDB Analytics](scalardb-analytics/run-analytical-queries.mdx) ================================================ FILE: docs/develop-run-non-transactional-operations-overview.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Run Non-Transactional Storage Operations Overview ScalarDB was initially designed to provide a unified abstraction between diverse databases and transactions across such databases. However, there are cases where you only need the unified abstraction to simplify your applications that use multiple, possibly diverse, databases. ScalarDB can be configured to provide only the unified abstraction, without transaction capabilities, so that it only runs non-transactional operations on the underlying database and storage. Since ScalarDB in this configuration doesn't guarantee ACID across multiple operations, you can perform operations with better performance. In this sub-category, you can learn how to run such non-transactional storage operations. - Run Through the CRUD Interface - [Use the ScalarDB Core Library](run-non-transactional-storage-operations-through-library.mdx) - [Use ScalarDB Cluster](scalardb-cluster/run-non-transactional-storage-operations-through-scalardb-cluster.mdx) - [Run Through the SQL Interface](scalardb-cluster/run-non-transactional-storage-operations-through-sql-interface.mdx) - [Run Through the Primitive CRUD Interface](run-non-transactional-storage-operations-through-primitive-crud-interface.mdx) ================================================ FILE: docs/develop-run-transactions-overview.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Run Transactions Overview In this sub-category, you can learn how to model your data based on the ScalarDB data model and create schemas. Then, you can learn how to run transactions through the ScalarDB Core library and ScalarDB Cluster, a gRPC server that wraps the Core library. - [Model Your Data](data-modeling.mdx) - Run Through the CRUD Interface - [Use the ScalarDB Core Library](run-transactions-through-scalardb-core-library.mdx) - [Use ScalarDB Cluster](scalardb-cluster/run-transactions-through-scalardb-cluster.mdx) - [Run Through the SQL Interface](scalardb-cluster/run-transactions-through-scalardb-cluster-sql.mdx) ================================================ FILE: docs/features.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Features This document briefly explains which features are available in which editions of ScalarDB. | | ScalarDB Core (Community) | ScalarDB Cluster (Enterprise Standard) | ScalarDB Cluster (Enterprise Premium) | ScalarDB Analytics (Enterprise) | |-------------------------------------------------------------------------------------------------------------------------------------|---------------------------|----------------------------------------|------------------------------------------------------------|---------------------------------| | [Transaction processing across databases with primitive interfaces](getting-started-with-scalardb.mdx) | ✅ | ✅ | ✅ | – | | [Clustering](scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx) | - | ✅ | ✅ | – | | [Non-transactional storage operations](develop-run-non-transactional-operations-overview.mdx) | – | ✅ (3.14+) | ✅ (3.14+) | – | | [Authentication/authorization](scalardb-cluster/scalardb-auth-with-sql.mdx) | – | ✅ | ✅ | – | | [Encryption](scalardb-cluster/encrypt-data-at-rest.mdx) | – | – | ✅ (3.14+) | – | | [Attribute-based access control](scalardb-cluster/authorize-with-abac.mdx) | – | – | ✅ (3.15+) (Enterprise Premium Option*, Private Preview**) | – | | [SQL interface (SQL API, JDBC, Spring Data JDBC, and LINQ)](scalardb-sql/index.mdx) | – | – | ✅ | – | | [GraphQL interface](scalardb-graphql/index.mdx) | – | – | ✅ | – | | [Vector search interface](scalardb-cluster/getting-started-with-vector-search.mdx) | – | – | ✅ (3.15+) (Private Preview**) | – | | [Analytical query processing across ScalarDB-managed data sources](scalardb-analytics/quickstart.mdx) | – | – | – | ✅ (3.14+) | | [Analytical query processing across non-ScalarDB-managed data sources](scalardb-analytics/quickstart.mdx) | – | – | – | ✅ (3.15+) | | [Remote replication](scalardb-cluster/remote-replication.mdx) | – | – | ✅ (3.16+) (Private Preview**) | – | \* This feature is not available in the Enterprise Premium edition. If you want to use this feature, please [contact us](https://www.scalar-labs.com/contact). \*\* This feature is currently in Private Preview. For details, please [contact us](https://www.scalar-labs.com/contact) or wait for this feature to become publicly available in a future version. ================================================ FILE: docs/getting-started-with-benchmarking-scalardb.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Benchmarking ScalarDB This guide walks you through planning, running, and analyzing benchmarks for ScalarDB. A structured benchmarking methodology produces reliable, meaningful performance measurements that you can use to make informed decisions about system configuration, resource sizing, and technology selection. ## Plan the benchmark Careful planning before running benchmarks prevents wasted effort and ensures that your results are actionable. ### Define your benchmarking objectives Start by clarifying what you want to learn from the benchmark. Your objective determines the workload, metrics, environment, and configuration decisions that follow. Common benchmarking objectives include the following: - **System comparison:** Compare ScalarDB against other systems that provide similar capabilities to determine which best fits your requirements. - **Scalability validation:** Confirm that ScalarDB scales as expected when you add nodes or increase concurrency under a specific workload. - **Cost estimation:** Map the performance you need to the resources required to achieve it. - **Performance verification:** Confirm that ScalarDB meets a minimum performance threshold under a specific workload. ### Understand performance metrics The two fundamental performance metrics for benchmarking are throughput and latency. Throughput measures the number of transactions the system completes per unit of time, expressed as transactions per second (TPS). Latency measures the time elapsed between submitting a transaction and receiving a response, expressed in milliseconds (ms). Throughput and latency are related but distinct. A system may achieve high throughput while individual transactions experience high latency, or vice versa. Measure both to get a complete picture of system performance. :::tip When reporting latency, include percentile distributions (p50, p95, p99) rather than averages alone. Average latency can mask tail latency issues that affect user experience. ::: ### Choose a workload Performance varies significantly depending on the workload. A throughput number is meaningful only when paired with the workload that produced it. Consider two systems that both achieve 1,000 TPS. If one system performs 10 read-modify-write (RMW) operations per transaction while the other performs a single RMW, the first system is doing 10 times more work despite reporting the same TPS value. When designing your workload, consider the following factors: - **Operation mix:** The ratio of reads to writes. - **Operations per transaction:** The number of operations within a single transaction. - **Access pattern:** Uniform versus skewed toward hotspots. - **Concurrency level:** The number of concurrent threads. - **Dataset size:** The total amount of data the benchmark operates on. - **Record size:** The size of each record the benchmark operates on. Vary concurrency and access skew across experiments to understand how the system behaves under different conditions. ### Define the comparison baseline When comparing ScalarDB against other systems, ensure that the comparison is fair and the results are interpretable. First, ensure a fair comparison by using systems that provide the same guarantees. If multi-database transactions spanning multiple databases are a requirement, comparing ScalarDB against a system that does not support such transactions is not meaningful because the systems solve fundamentally different problems. However, if multi-database transactions are optional rather than required, comparing the two to measure the overhead of coordination is valid. In that case, state explicitly that the comparison measures coordination overhead, not general system performance. Second, change one variable at a time. When comparing configurations or systems, vary only one parameter per experiment. If you change multiple variables simultaneously, you cannot determine which variable caused the observed performance difference. :::warning Comparing systems with different consistency guarantees (for example, eventual consistency versus linearizable consistency) without accounting for the difference leads to misleading conclusions. Always document the guarantees each system provides alongside the performance results. ::: ### Estimate expected performance Before running benchmarks, estimate the expected performance based on your understanding of the workload and the system. Benchmarking is the combination of performance estimation and performance measurement. Measurement without estimation is difficult to validate. To create an estimate, first identify the operations your workload performs (reads, writes, commits). Then, review the phases of the [Consensus Commit protocol](./consensus-commit.mdx) that ScalarDB executes for each transaction and sum the per-phase latencies based on the storage backend and network characteristics to estimate the latency of a single transaction. Finally, divide the expected concurrency by this single-transaction latency to derive a rough throughput estimate. When modeling ScalarDB application performance, the dominant cost factor depends on your environment. In most modern environments where storage I/O is fast, the number of network hops is the primary factor that determines transaction latency, so counting the network round trips each transaction phase requires gives a reasonable latency estimate. However, in environments with large datasets and I/O-intensive workloads where disk throughput is a bottleneck, the number of disk reads and writes per transaction is a better basis for the estimate. The estimate does not need to be precise. Even a rough order-of-magnitude estimate helps detect misconfigurations. For example, if a configuration error is suppressing performance, having no estimate means you have no way to detect the problem from the measurement results alone. Conversely, if the estimate and measurement agree, both are likely correct. If they disagree, investigating the discrepancy often reveals errors in either the estimate or the system configuration. ## Prepare the environment This section explains how to set up the infrastructure, establish baseline measurements, and prepare the benchmarking tool. ### Prepare the infrastructure The testing environment directly affects benchmark results. Use stable compute resources and avoid cloud instance types that introduce performance variance. Cloud virtual machines share physical hardware with other tenants, so performance can vary by time of day. Do not use burstable instances (for example, AWS T-series), because these instances throttle CPU performance after credits are exhausted, producing inconsistent results. Similarly, do not use spot instances or preemptible instances, as these use surplus capacity and may be reclaimed during a benchmark run. In addition to choosing stable instance types, match the resources to your workload. For example, if your workload is I/O-intensive, provision instances with high I/O throughput (such as instances backed by NVMe SSDs). Otherwise, I/O bottlenecks may mask the actual system performance. :::note For production-grade benchmarks of ScalarDB Cluster, a minimum of 3 worker nodes with 4 vCPU and 8 GB memory each is recommended, with ScalarDB Cluster pods limited to 2 vCPU and 4 GB memory per pod. Place nodes in different availability zones for resilience testing. For details, see [Production checklist for ScalarDB Cluster](./scalar-kubernetes/ProductionChecklistForScalarDBCluster.mdx). ::: ### Measure baseline environment performance Before benchmarking ScalarDB, measure the raw performance of your environment to establish a baseline and detect infrastructure issues early. The following tools are useful for measuring baseline performance: - [sysbench](https://github.com/akopytov/sysbench) — CPU and memory throughput. - [fio](https://github.com/axboe/fio) — Sequential and random I/O throughput and latency. - [iperf](https://github.com/esnet/iperf) — Network bandwidth and latency between nodes. These baselines help you determine whether the infrastructure is performing as expected and whether any bottleneck you observe during ScalarDB benchmarks originates from the infrastructure rather than from ScalarDB. ### Prepare the benchmarking tool If an existing benchmarking tool supports your workload, use it rather than building a custom tool. Using an established tool reduces the risk of measurement errors and makes it easier for others to reproduce your results. Regardless of which approach you choose, understanding what the workload does remains important. [ScalarDB benchmarks](./scalardb-benchmarks/README.mdx) is the official benchmarking tool for ScalarDB. It provides standard workloads (TPC-C and YCSB variants) built on the [Kelpie](https://github.com/scalar-labs/kelpie) framework. If your benchmarking scenario requires a workload that ScalarDB benchmarks does not support, you can build a custom benchmark. In that case, consider reusing the measurement infrastructure from Kelpie to ensure accurate timing and reporting. For instructions on setting up ScalarDB benchmarks, including cloning the repository, building the benchmark JAR, downloading Kelpie, loading schemas, and creating benchmark configuration files, see the [ScalarDB benchmarks repository](https://github.com/scalar-labs/scalardb-benchmarks). For the full list of workload-specific parameters, see [ScalarDB benchmarks](./scalardb-benchmarks/README.mdx). ## Configure ScalarDB for benchmarking Before running benchmarks, configure ScalarDB with the performance parameters that match your benchmarking objectives. The connection mode subsection below is specific to ScalarDB Cluster. The performance-related properties apply to both ScalarDB Core and ScalarDB Cluster, but the configuration location differs (see [Tune performance-related properties](#tune-performance-related-properties)). ### Choose a deployment pattern Choose a deployment pattern based on how your application uses ScalarDB. ScalarDB supports the following deployment patterns: - **ScalarDB Core (embedded):** The application or benchmark program embeds ScalarDB as a library. This pattern is the simplest to set up because it requires no separate server infrastructure. Use this pattern when you want to benchmark ScalarDB without the overhead of a cluster deployment. - **ScalarDB Cluster with a single cluster:** A single ScalarDB Cluster instance serves the application. This is the most common deployment pattern when using ScalarDB Cluster. Use this pattern for most benchmarking scenarios because it is simpler to manage and requires fewer resources. In microservice use cases, this pattern is called the *shared cluster pattern*, where all services share the single cluster instance and use the one-phase commit interface. - **ScalarDB Cluster with multiple clusters:** Each application or service has its own dedicated ScalarDB Cluster instance. Transactions that span multiple clusters use the two-phase commit interface, which adds complexity to transaction handling and error recovery but provides stronger resource isolation. Use this pattern only if you specifically need to evaluate cross-cluster transaction performance with the two-phase commit interface. In microservice use cases, this pattern is called the *separated cluster pattern*. For details on microservice deployment patterns, see [ScalarDB Cluster deployment patterns for microservices](./scalardb-cluster/deployment-patterns-for-microservices.mdx). ### Choose a connection mode When using ScalarDB Cluster, choose a client connection mode. ScalarDB Cluster supports two modes: `direct-kubernetes` mode and `indirect` mode. In **`direct-kubernetes` mode**, the client uses the Kubernetes API and a membership-based routing algorithm to connect directly to the correct cluster node. This eliminates an extra network hop, resulting in lower latency and higher throughput. However, the client application must run inside the same Kubernetes cluster as ScalarDB Cluster. In **`indirect` mode**, the client sends requests to any cluster node through a load balancer. There are two routing options within `indirect` mode: - **Via Service (ClusterIP) (recommended):** The client connects through an L4 load balancer to a Kubernetes Service, which routes to a cluster node. This option avoids the extra hop through Envoy, resulting in lower latency. - **Via Envoy:** The client connects through an L4 load balancer to an Envoy proxy, which routes to a cluster node. This option provides better load balancing across ScalarDB Cluster nodes because Envoy can distribute requests at the L7 layer, but it introduces an additional network hop. `indirect` mode allows clients to run outside the Kubernetes cluster but adds extra network hops per request. Of the two options, connecting via Service is generally recommended for lower latency, while connecting via Envoy may be preferable when load distribution across cluster nodes is a priority. From a performance perspective, `direct-kubernetes` mode is recommended over both options within `indirect` mode. To configure `direct-kubernetes` mode, set the following properties in your ScalarDB configuration file: ```properties scalar.db.transaction_manager=cluster scalar.db.contact_points=direct-kubernetes:/ scalar.db.contact_port=60053 ``` To configure `indirect` mode, replace the `contact_points` value: ```properties scalar.db.contact_points=indirect: ``` ### Tune performance-related properties ScalarDB provides several configuration properties that affect performance. The following sections describe the most relevant properties for benchmarking. :::note If you use ScalarDB Core, set these properties in the application's ScalarDB configuration file. If you use ScalarDB Cluster, set them in the ScalarDB Cluster configuration. The exception is the client-side optimizations described in [Client-side optimizations](#client-side-optimizations), which are ScalarDB Cluster-specific and configured on the client. ::: #### Consensus Commit optimizations The [Consensus Commit protocol](./consensus-commit.mdx) supports several optimizations that improve performance. The parallel execution properties (like `parallel_preparation` and `parallel_commit`) are enabled by default and execute protocol phases in parallel for multi-record transactions. Async commit (`async_commit`) returns to the client after the commit-state phase completes without waiting for the commit-records phase, which increases throughput at the cost of a slightly wider recovery window. One-phase commit (`one_phase_commit`) skips the prepare and commit-state phases entirely when all mutations can be applied atomically, significantly reducing round trips for simple transactions. | Property | Default | Description | |----------|---------|-------------| | `scalar.db.consensus_commit.parallel_preparation.enabled` | `true` | Execute the prepare phase in parallel for multi-record transactions. | | `scalar.db.consensus_commit.parallel_commit.enabled` | `true` | Execute the commit-records phase in parallel. | | `scalar.db.consensus_commit.async_commit.enabled` | `false` | Return to the client after the commit-state phase without waiting for commit-records. Note that it might cause a slowdown under a medium-to-high contention workload due to excessive recovery. | | `scalar.db.consensus_commit.one_phase_commit.enabled` | `false` | Skip prepare and commit-state phases when mutations can be applied atomically. | For details on how each optimization affects the Consensus Commit protocol, see [Performance optimization](./consensus-commit.mdx#performance-optimization). #### Isolation level The `scalar.db.consensus_commit.isolation_level` property controls the transaction isolation level. The available levels are `SNAPSHOT`, `SERIALIZABLE`, and `READ_COMMITTED`. Higher isolation levels increase overhead because the protocol must perform additional checks to prevent anomalies. Choose the isolation level based on the anomalies your application can accept, not based on performance alone. Each level permits different types of anomalies, and the correct choice depends on your application's correctness requirements. :::warning Do not weaken the isolation level solely to improve benchmark throughput. If your application requires `SERIALIZABLE` isolation, benchmarking with `READ_COMMITTED` produces throughput numbers that are irrelevant to your production workload. Always benchmark with the isolation level you intend to use in production. ::: #### Group commit Group commit batches multiple transactions' coordinator state writes into a single operation, which is particularly beneficial when the coordinator table resides in a remote or high-latency storage backend. Group commit is incompatible with the two-phase commit interface. | Property | Default | Description | |----------|---------|-------------| | `scalar.db.consensus_commit.coordinator.group_commit.enabled` | `false` | Enable group commit for coordinator state writes. | | `scalar.db.consensus_commit.coordinator.group_commit.slot_capacity` | `20` | Maximum number of transaction slots per group. | | `scalar.db.consensus_commit.coordinator.group_commit.group_size_fix_timeout_millis` | `40` | Milliseconds to wait for a group to fill before processing. | When tuning group commit, benchmark combinations of `slot_capacity` and `group_size_fix_timeout_millis` together to find the balance that works best for your workload and storage backend. For example, try `20` and `40`; `30` and `40`, and `20` and `80`, respectively. #### Client-side optimizations The following properties are specific to ScalarDB Cluster and reduce network round trips without changing transaction semantics. The `piggyback_begin` property eliminates the dedicated begin RPC call by piggybacking the transaction begin onto the first CRUD operation, saving one round trip per transaction. The `write_buffering` property buffers non-conditional write operations (inserts, upserts, unconditional puts, updates, and deletes) and executes them in batches, reducing the total number of RPC calls for write-heavy transactions. | Property | Default | Description | |----------|---------|-------------| | `scalar.db.cluster.client.piggyback_begin.enabled` | `false` | Eliminate the dedicated begin RPC by piggybacking it onto the first CRUD operation. | | `scalar.db.cluster.client.write_buffering.enabled` | `false` | Buffer non-conditional write operations and execute them in batches. | :::warning * If `piggyback_begin` is enabled, you will get an `IllegalStateException` when calling the `DistributedTransaction.getId()` method until the actual begin operation is executed. * If `piggyback_begin` or `write_buffering` is enabled, you will always get an `IllegalStateException` when calling the `DistributedTransactionManager.resume()` and `DistributedTransactionManager.join()` methods. ::: Similar optimizations are available for SQL transactions. The `piggyback_begin` property works the same way as the primitive API version, eliminating the dedicated begin RPC call. The `write_buffering` property buffers `INSERT` and `UPSERT` statements on the client side and executes them in batches. | Property | Default | Description | |----------|---------|-------------| | `scalar.db.sql.cluster_mode.client.piggyback_begin.enabled` | `false` | Eliminate the dedicated begin RPC by piggybacking it onto the first execute operation. | | `scalar.db.sql.cluster_mode.client.write_buffering.enabled` | `false` | Buffer `INSERT` and `UPSERT` statements and execute them in batches. | :::warning * If `piggyback_begin` is enabled, `SqlSession.getTransactionId()` returns `Optional.empty()` until the actual begin operation is executed. * If `piggyback_begin` or `write_buffering` is enabled, you will always get an `IllegalStateException` when calling the `SqlSession.resume()` and `SqlSession.join()` methods. ::: #### JDBC connection pool If you use a JDBC database as the storage backend, tune the connection pool to match your expected concurrency. The `max_total` property controls the maximum number of connections in the pool. | Property | Default | Description | |----------|---------|-------------| | `scalar.db.jdbc.connection_pool.min_idle` | `20` | Minimum idle connections in the pool. | | `scalar.db.jdbc.connection_pool.max_total` | `200` | Maximum total connections (idle + active). | | `scalar.db.jdbc.connection_pool.connection_timeout_millis` | `30000` | Maximum time in milliseconds to wait for a connection from the pool. | | `scalar.db.jdbc.connection_pool.idle_timeout_millis` | `600000` | Maximum time in milliseconds that a connection is allowed to sit idle in the pool. | | `scalar.db.jdbc.connection_pool.max_lifetime_millis` | `1800000` | Maximum lifetime in milliseconds of a connection in the pool. | | `scalar.db.jdbc.connection_pool.keepalive_time_millis` | `0` | Interval in milliseconds for connection keepalive. `0` disables keepalive. | For the complete list of configuration properties, see [ScalarDB Core Configurations](./configurations.mdx) and [ScalarDB Cluster Configurations](./scalardb-cluster/scalardb-cluster-configurations.mdx). ## Run the benchmark This section outlines practices for running benchmarks that produce consistent and reliable measurements. ### Warm up the target system Before measuring performance, warm up the system to eliminate initialization overhead such as JVM class loading, JIT compilation, connection pool establishment, and cache population. ScalarDB benchmarks supports a ramp-up period through the `ramp_for_sec` parameter. Set this to at least 30 seconds. Data collected during the ramp-up period is excluded from the results. ### Run for a sufficient duration Run the benchmark for at least 60 seconds after the ramp-up period. Short runs are more susceptible to transient variations such as garbage collection pauses and network fluctuations that can skew results. For specific objectives, adjust the duration accordingly. For quick validation, 1 to 5 minutes is typically sufficient. For performance comparisons that require statistical rigor, run for 5 to 15 minutes. For stability testing or degradation detection, run for several hours or days to observe long-term behavior. ### Run multiple iterations Cloud environments exhibit performance variance due to shared physical infrastructure. Run each benchmark configuration at least three times and aggregate the results by averaging. If results vary significantly across iterations, use a trimmed mean (discarding the highest and lowest values) or report results with standard deviation to convey the variance. If the standard deviation exceeds 10% of the mean (the coefficient of variation is greater than 0.1), it is advisable to investigate the cause of the variance before drawing conclusions from the data. ## Validate and analyze the results This section describes how to validate benchmark results against your estimates and iterate on the configuration to ensure the measurements are accurate and actionable. ### Compare with estimated performance After each benchmark run, compare the measured results against the estimates you prepared before running the benchmark. If the results match your estimates, both the estimate and the measurement are likely correct. If the results fall significantly below your estimates, a misconfiguration, resource bottleneck, or environmental issue may be suppressing performance. Investigate before accepting the results. If the results exceed your estimates, the estimate may have overlooked an optimization, or the measurement may be incorrect (for example, measuring only successful transactions while ignoring failures). :::warning Do not publish or act on benchmark results that you cannot explain. Unexplained results may stem from misconfigurations that invalidate the measurements. ::: ### Tune and re-run If measured performance falls short of estimates, the system may not be optimally configured. Review the configuration properties described in the previous sections, adjust one parameter at a time, and re-run the benchmark after each change. Repeat this cycle until the measured results and estimates converge. Common tuning actions include the following: - Tuning the retry backoff interval to reduce wasted work from repeated conflicts under high-contention workloads, since ScalarDB uses optimistic concurrency control (OCC). For example, the TPC-C workload in ScalarDB benchmarks provides a `backoff` parameter for this purpose. - Enabling async commit for high-throughput workloads. - Enabling and tuning group commit for workloads with high coordinator write traffic. - Adjusting the JDBC connection pool size to match the concurrency level. ### Analyze the results Once measured results and estimates are in agreement, you can be confident that the results are valid. Summarize the throughput, latency distribution (p50, p95, p99), and any error rates for each configuration you tested. Use your understanding of the workload and system to explain why each configuration produced the results it did. Based on your original objective, determine the deployment configuration, resource sizing, or system choice that best meets your requirements. ## Next steps - [ScalarDB Benchmarking Tools](./scalardb-benchmarks/README.mdx) — Full reference for the official ScalarDB benchmarking tool, including all workload parameters. - [ScalarDB Core Configurations](./configurations.mdx) — Complete list of ScalarDB Core configuration properties. - [ScalarDB Cluster Configurations](./scalardb-cluster/scalardb-cluster-configurations.mdx) — Cluster-specific configuration properties, including client-side optimizations. - [Consensus Commit Protocol](./consensus-commit.mdx) — How ScalarDB achieves ACID transactions across databases. - [ScalarDB Cluster Deployment Patterns for Microservices](./scalardb-cluster/deployment-patterns-for-microservices.mdx) — Shared versus separated cluster patterns. - [Production Checklist for ScalarDB Cluster](./scalar-kubernetes/ProductionChecklistForScalarDBCluster.mdx) — Resource sizing and deployment recommendations for production. ================================================ FILE: docs/getting-started-with-scalardb-by-using-kotlin.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB by Using Kotlin import StorageSetupTabs from './components/_getting-started-setup-storage.mdx'; import LoadSchemaTabs from './components/_getting-started-load-schema.mdx'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This getting started tutorial explains how to configure your preferred database in ScalarDB and set up a basic electronic money application by using Kotlin. Since Kotlin has Java interoperability, you can use ScalarDB directly from Kotlin. :::warning The electronic money application is simplified for this tutorial and isn't suitable for a production environment. ::: ## Prerequisites for this sample application - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ## Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/scalardb-kotlin-sample ``` ## Set up your database for ScalarDB Follow the instructions below to configure your database for ScalarDB. For a list of databases that ScalarDB supports, see [Databases](requirements.mdx#databases). ## Load the database schema You need to define the database schema (the method in which the data will be organized) in the application. For details about the supported data types, see [Database Adapters](database-adapters.mdx). For this tutorial, a file named **schema.json** already exists in the `scalardb-samples/scalardb-kotlin-sample` directory. To apply the schema, go to the [`scalardb` Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you are using to the `scalardb-samples/scalardb-kotlin-sample` directory. Then, based on your database, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: ## Execute transactions and retrieve data in the basic electronic money application After loading the schema, you can execute transactions and retrieve data in the basic electronic money application that is included in the repository that you cloned. The application supports the following types of transactions: - Create an account. - Add funds to an account. - Send funds between two accounts. - Get an account balance. :::note When you first execute a Gradle command, Gradle will automatically install the necessary libraries. ::: ### Create an account with a balance You need an account with a balance so that you can send funds between accounts. To create an account for **customer1** that has a balance of **500**, run the following command: ```console ./gradlew run --args="-action charge -amount 500 -to customer1" ``` ### Create an account without a balance After setting up an account that has a balance, you need another account for sending funds to. To create an account for **merchant1** that has a balance of **0**, run the following command: ```console ./gradlew run --args="-action charge -amount 0 -to merchant1" ``` ### Add funds to an account You can add funds to an account in the same way that you created and added funds to an account in [Create an account with a balance](#create-an-account-with-a-balance). To add **500** to the account for **customer1**, run the following command: ```console ./gradlew run --args="-action charge -amount 500 -to customer1" ``` The account for **customer1** will now have a balance of **1000**. ### Send electronic money between two accounts Now that you have created two accounts, with at least one of those accounts having a balance, you can send funds from one account to the other account. To have **customer1** pay **100** to **merchant1**, run the following command: ```console ./gradlew run --args="-action pay -amount 100 -from customer1 -to merchant1" ``` ### Get an account balance After sending funds from one account to the other, you can check the balance of each account. To get the balance of **customer1**, run the following command: ```console ./gradlew run --args="-action getBalance -id customer1" ``` You should see the following output: ```console ... The balance for customer1 is 900 ... ``` To get the balance of **merchant1**, run the following command: ```console ./gradlew run --args="-action getBalance -id merchant1" ``` You should see the following output: ```console ... The balance for merchant1 is 100 ... ``` ## Stop the database To stop the database, stop the Docker container by running the following command: ```console docker compose down ``` ## Reference To see the source code for the electronic money application used in this tutorial, see [`ElectronicMoney.kt`](https://github.com/scalar-labs/scalardb-samples/blob/main/scalardb-kotlin-sample/src/main/kotlin/sample/ElectronicMoney.kt). ================================================ FILE: docs/getting-started-with-scalardb.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB import StorageSetupTabs from './components/_getting-started-setup-storage.mdx'; import LoadSchemaTabs from './components/_getting-started-load-schema.mdx'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This getting started tutorial explains how to configure your preferred database in ScalarDB and illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a credit card by using ScalarDB. The sample e-commerce application shows how users can order and pay for items by using a line of credit. :::warning Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling in ScalarDB, see [How to handle exceptions](api-guide.mdx#how-to-handle-exceptions). ::: ## Prerequisites for this sample application - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ## Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/scalardb-sample ``` ## Set up your database for ScalarDB Follow the instructions below to configure your database for ScalarDB. For a list of databases that ScalarDB supports, see [Databases](requirements.mdx#databases). ## Load the database schema You need to define the database schema (the method in which the data will be organized) in the application. For details about the supported data types, see [Database Adapters](database-adapters.mdx). For this tutorial, a file named **schema.json** already exists in the `scalardb-samples/scalardb-sample` directory. To apply the schema, go to the [`scalardb` Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you are using to the `scalardb-samples/scalardb-sample` directory. Then, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: ### Schema details As shown in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-sample/schema.json) for the sample application, all the tables are created in the `sample` namespace. - `sample.customers`: a table that manages customer information - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit - `credit_total`: the amount of money that the customer has spent from their line of credit - `sample.orders`: a table that manages order information - `sample.statements`: a table that manages order statement information - `sample.items`: a table that manages information for items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/getting-started-ERD.png) ### Load the initial data Before running the sample application, you need to load the initial data by running the following command: ```console ./gradlew run --args="LoadInitialData" ``` After the initial data has loaded, the following records should be stored in the tables. **`sample.customers` table** | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | **`sample.items` table** | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Execute transactions and retrieve data in the sample application The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. ### Get customer information Start with getting information about the customer whose ID is `1` by running the following command: ```console ./gradlew run --args="GetCustomerInfo 1" ``` You should see the following output: ```console ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0} ... ``` ### Place an order Then, have customer ID `1` place an order for three apples and two oranges by running the following command: :::note The order format in this command is `./gradlew run --args="PlaceOrder :,:,..."`. ::: ```console ./gradlew run --args="PlaceOrder 1 1:3,2:2" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e"} ... ``` ### Check the order details Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: ```console ./gradlew run --args="GetOrder " ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: ```console ... {"order": {"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} ... ``` ### Place another order Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: ```console ./gradlew run --args="PlaceOrder 1 5:1" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d"} ... ``` ### Check the order history Get the history of all orders for customer ID `1` by running the following command: ```console ./gradlew run --args="GetOrders 1" ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: ```console ... {"order": [{"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000},{"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d","timestamp": 1650948412766,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000}]} ... ``` ### Check the credit total Get the credit total for customer ID `1` by running the following command: ```console ./gradlew run --args="GetCustomerInfo 1" ``` You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: ```console ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} ... ``` Try to place an order for one grape and one mango by running the following command: ```console ./gradlew run --args="PlaceOrder 1 3:1,4:1" ``` You should see the following output, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. ```console ... java.lang.RuntimeException: Credit limit exceeded at sample.Sample.placeOrder(Sample.java:205) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:33) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:8) at picocli.CommandLine.executeUserObject(CommandLine.java:1783) at picocli.CommandLine.access$900(CommandLine.java:145) at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975) at picocli.CommandLine.execute(CommandLine.java:1904) at sample.command.SampleCommand.main(SampleCommand.java:35) ... ``` ### Make a payment To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. Make a payment by running the following command: ```console ./gradlew run --args="Repayment 1 8000" ``` Then, check the `credit_total` amount for customer ID `1` by running the following command: ```console ./gradlew run --args="GetCustomerInfo 1" ``` You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: ```console ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000} ... ``` Now that customer ID `1` has made a payment, place an order for one grape and one mango by running the following command: ```console ./gradlew run --args="PlaceOrder 1 3:1,4:1" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "8911cab3-1c2b-4322-9386-adb1c024e078"} ... ``` ## Stop the database To stop the database, stop the Docker container by running the following command: ```console docker compose down ``` ## Reference To see the source code for the e-commerce application used in this tutorial, see [`Sample.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/scalardb-sample/src/main/java/sample/Sample.java). ================================================ FILE: docs/glossary.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Glossary This glossary includes database and distributed-system terms that are often used when using ScalarDB. ## ACID Atomicity, consistency, isolation, and durability (ACID) is a set of properties that ensure database transactions are processed reliably, maintaining integrity even in cases of errors or system failures. ## concurrency control Concurrency control in databases ensures that multiple transactions can occur simultaneously without causing data inconsistency, usually through mechanisms like locking or timestamp ordering. ## consensus Consensus in distributed systems refers to the process of achieving agreement among multiple computers or nodes on a single data value or system state. ## data federation Data federation is the process of integrating data from different sources without moving the data, creating a unified view for querying and analysis. ## data mesh A data mesh is a decentralized data architecture that enables each business domain within a company to autonomously manage data and use it efficiently. ## data virtualization Data virtualization is similar to data federation in many aspects, meaning that it virtualizes multiple data sources into a unified view, simplifying queries without moving the data. ## database anomalies Database anomalies are inconsistencies or errors in data that can occur when operations such as insertions, updates, or deletions are performed without proper transaction management. ## federation engine A federation engine facilitates data integration and querying across multiple disparate data sources, often as part of a data federation architecture. ## global transaction A global transaction spans multiple databases or distributed systems and ensures that all involved systems commit or roll back changes as a single unit. ## heterogeneous databases Heterogeneous databases refer to systems composed of different database technologies that may have distinct data models, query languages, and transaction mechanisms. ## HTAP Hybrid transactional/analytical processing (HTAP) refers to a system that can handle both transactional and analytical workloads concurrently on the same data set, removing the need for separate databases. ## JDBC Java Database Connectivity (JDBC) is an API that allows Java applications to interact with databases, providing methods for querying and updating data in relational databases. ## linearizability Linearizability is a strong consistency model in distributed systems where operations appear to occur atomically in some order, and each operation takes effect between its start and end. ## NoSQL database A NoSQL database is a non-relational databases designed for specific data models, such as document, key-value, wide-column, or graph stores, often used for handling large-scale, distributed data. ## Paxos Paxos is a family of protocols used in distributed systems to achieve consensus, even in the presence of node failures. ## PITR Point-in-time recovery (PITR) allows a database to be restored to a previous state at any specific time, usually after an unintended event like data corruption. ## polystores Polystores are database architectures that allow users to interact with multiple, heterogeneous data stores, each optimized for a specific workload or data type, as if they were a single system. ## read-committed isolation Read-committed isolation is an isolation level where each transaction sees only committed data, preventing dirty reads but allowing non-repeatable reads. ## relational database A relational database stores data in tables with rows and columns, using a structured query language (SQL) to define, query, and manipulate the data. ## replication Replication in databases involves copying and distributing data across multiple machines or locations to ensure reliability, availability, and fault tolerance. ## Saga The Saga pattern is a method for managing long-running transactions in a distributed system, where each operation in the transaction is followed by a compensating action in case of failure. ## serializable isolation Serializable isolation (serializability) is the highest isolation level in transactional systems, ensuring that the outcome of concurrently executed transactions is the same as if they were executed sequentially. ## snapshot isolation Snapshot isolation is an isolation level that allows transactions to read a consistent snapshot of the database, protecting them from seeing changes made by other transactions until they complete. ## TCC Try-Confirm/Cancel (TCC) is a pattern for distributed transactions that splits an operation into three steps, allowing for coordination and recovery across multiple systems. ## transaction A transaction in databases is a sequence of operations treated as a single logical unit of work, ensuring consistency and integrity, typically conforming to ACID properties. ## transaction manager A transaction manager coordinates the execution of transactions across multiple systems or databases, ensuring that all steps of the transaction succeed or fail as a unit. ## two-phase commit Two-phase commit is a protocol for ensuring all participants in a distributed transaction either commit or roll back the transaction, ensuring consistency across systems. ================================================ FILE: docs/index.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish image: img/scalardb-social-card-preview.png hide_table_of_contents: true title: "" --- import CategoryGrid from '/src/components/Cards/3.18'; ================================================ FILE: docs/learning-paths.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Learning Paths This guide provides learning paths for different roles. Depending on your role, follow the appropriate sequence of documents to gain a comprehensive understanding of ScalarDB. ## Architects This path is designed for architects who want to design applications that use ScalarDB. - [About ScalarDB category](./overview.mdx) - [Quickstart category](./quickstart-overview.mdx) - [Develop category](./develop-overview.mdx) (as necessary) ## Application developers This path is designed for application developers who want to build applications that use ScalarDB. - [About ScalarDB category](./overview.mdx) - [Quickstart category](./quickstart-overview.mdx) - [Develop category](./develop-overview.mdx) - [Migrate category](./migrate-overview.mdx) (as necessary) ## Infrastructure engineers This path is designed for infrastructure engineers who want to deploy and manage ScalarDB in various environments. - [About ScalarDB category](./overview.mdx) - [Quickstart category](./quickstart-overview.mdx) - [Deploy category](./deploy-overview.mdx) - [Manage category](./manage-overview.mdx) ## Business decision makers This path is designed for business decision makers who want to understand the strategic direction and future plans of ScalarDB. - [ScalarDB Overview](./overview.mdx) - [ScalarDB Roadmap](./roadmap.mdx) ================================================ FILE: docs/libraries-and-tools.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium - Enterprise Option displayed_sidebar: docsEnglish --- # Libraries and Tools for ScalarDB ScalarDB provides various libraries and tools to help you build and operate scalable and reliable applications. Below are some key libraries and tools available. ## Core and Cluster This section lists the libraries and tools available for ScalarDB Core and Cluster. ### Libraries The following libraries are available for ScalarDB Core and Cluster. | Library | Edition | Maven Package | Container Image | Reference | |----------------------------------|------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|-----------------|--------------------------------------------------------------------------------------------| | ScalarDB Core Java API library | Community, Enterprise Standard, & Enterprise Premium | [Maven Central Repository](https://central.sonatype.com/artifact/com.scalar-labs/scalardb) | N/A | [Documentation](./api-guide.mdx) | | ScalarDB Cluster Java Client SDK | Enterprise Standard & Enterprise Premium | [Maven Central Repository](https://central.sonatype.com/artifact/com.scalar-labs/scalardb-cluster-java-client-sdk) | N/A | [Documentation](./scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx) | | ScalarDB SQL | Enterprise Standard & Enterprise Premium | [Maven Central Repository](https://central.sonatype.com/artifact/com.scalar-labs/scalardb-sql) | N/A | [Documentation](./scalardb-sql/sql-api-guide.mdx) | | JDBC driver for ScalarDB SQL | Enterprise Standard & Enterprise Premium | [Maven Central Repository](https://central.sonatype.com/artifact/com.scalar-labs/scalardb-sql-jdbc) | N/A | [Documentation](./scalardb-sql/jdbc-guide.mdx) | | Spring Data JDBC for ScalarDB | Enterprise Standard & Enterprise Premium | [Maven Central Repository](https://central.sonatype.com/artifact/com.scalar-labs/scalardb-sql-spring-data) | N/A | [Documentation](./scalardb-sql/spring-data-guide.mdx) | ### Tools The following tools are available for ScalarDB Core and Cluster. | Tool | Edition | Maven Package | JAR File | Container Image | Reference | |----------------------------------|------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------| | ScalarDB Schema Loader | Community, Enterprise Standard, & Enterprise Premium | [Maven Central Repository](https://central.sonatype.com/artifact/com.scalar-labs/scalardb-schema-loader) | [GitHub Releases](https://github.com/scalar-labs/scalardb/releases) | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-schema-loader) | [Documentation](./schema-loader.mdx) | | ScalarDB Cluster Schema Loader | Enterprise Standard & Enterprise Premium | N/A | [GitHub Releases](https://github.com/scalar-labs/scalardb/releases) | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-schema-loader) | [Documentation](./scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#schema-loader-for-cluster) | | ScalarDB Data Loader CLI | Community, Enterprise Standard, & Enterprise Premium | [Maven Central Repository](https://central.sonatype.com/artifact/com.scalar-labs/scalardb-data-loader-cli) | [GitHub Releases](https://github.com/scalar-labs/scalardb/releases) | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-data-loader-cli) | [Documentation](./data-loader.mdx) | | ScalarDB Cluster Data Loader CLI | Enterprise Standard, & Enterprise Premium | N/A | [GitHub Releases](https://github.com/scalar-labs/scalardb/releases) | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-data-loader-cli) | [Documentation](./data-loader.mdx) | | ScalarDB Cluster SQL CLI | Enterprise Premium | N/A | [GitHub Releases](https://github.com/scalar-labs/scalardb/releases) | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-sql-cli) | [Documentation](./scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli) | | Replication CLI | Enterprise Premium | N/A | [GitHub Releases](https://github.com/scalar-labs/scalardb/releases) | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-replication-cli) | [Documentation](./scalardb-cluster/remote-replication.mdx#replication-cli) | | ScalarDB MCP Server | Community, Enterprise Standard, & Enterprise Premium | N/A | N/A | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-mcp-server) | [Documentation](./scalardb-mcp-server/getting-started-with-scalardb-mcp-server.mdx) | | Helm Charts | Community, Enterprise Standard, & Enterprise Premium | N/A | N/A | N/A | [Documentation](./helm-charts/getting-started-scalar-helm-charts.mdx) | | Scalar Admin for Kubernetes | Enterprise Standard & Enterprise Premium | [Maven Central Repository](https://central.sonatype.com/artifact/com.scalar-labs/scalar-admin-for-kubernetes) | N/A | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalar-admin-for-kubernetes) | [Documentation](./helm-charts/how-to-deploy-scalar-admin-for-kubernetes.mdx) | ### Cluster components The following components are available for ScalarDB Cluster. | Component | Edition | Container Image | Reference | |----------------------------------|---------------------|------------------------------------------------------------------------------------------------------------------|-----------------------------| | ScalarDB Cluster Node (BYOL) | Enterprise Premium | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-node-byol-premium) | Documentation (coming soon) | | ScalarDB Cluster Node (BYOL) | Enterprise Standard | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-node-byol-standard) | Documentation (coming soon) | | ScalarDB Cluster Node (UBI BYOL) | Enterprise Premium | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-node-ubi-byol-premium) | Documentation (coming soon) | | ScalarDB Cluster Node (UBI BYOL) | Enterprise Standard | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-node-ubi-byol-standard) | Documentation (coming soon) | ## Analytics This section lists the libraries and tools available for ScalarDB Analytics. ### Libraries The following libraries are available for ScalarDB Analytics. | Library | Edition | Maven Package | Container Image | Reference | |---------------------------|-------------------|------------------------------------------------------------------------------------------------|-----------------|-------------------------------------------------------------------------------------------------------------| | ScalarDB Analytics | Enterprise Option | [Maven Central Repository](https://central.sonatype.com/search?q=scalardb-analytics-spark-all) | N/A | [Documentation](./scalardb-analytics/run-analytical-queries.mdx#build-configuration-for-spark-applications) | ### Tools The following tools are available for ScalarDB Analytics. | Tool | Edition | Maven Package | JAR File | Container Image | Reference | |---------------------------|------------------------------------------------------|---------------|---------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------| | ScalarDB Analytics server | Enterprise Option | N/A | N/A | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-analytics-server-byol) | [Documentation](./scalardb-analytics/deploy-scalardb-analytics-server.mdx) | | ScalarDB Analytics CLI | Enterprise Option | N/A | [GitHub Releases](https://github.com/scalar-labs/scalardb/releases) | [GitHub](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-analytics-cli) | [Documentation](./scalardb-analytics/reference-cli-command.mdx) | | Helm Charts | Community, Enterprise Standard, & Enterprise Premium | N/A | N/A | N/A | [Documentation](./helm-charts/getting-started-scalar-helm-charts.mdx) | ================================================ FILE: docs/manage-backup-and-restore.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Back Up and Restore Databases This guide explains how to back up and restore databases that are used by ScalarDB. ## Basic guidelines to back up and restore databases Before performing a backup, be sure to read [How to Back Up and Restore Databases Used Through ScalarDB](backup-restore.mdx). ## Back up databases when using ScalarDB in a Kubernetes environment For details on how to back up databases in a Kubernetes environment, see [Back up a NoSQL database in a Kubernetes environment](scalar-kubernetes/BackupNoSQL.mdx). ## Restore databases when using ScalarDB in a Kubernetes environment For details on how to restore databases in a Kubernetes environment, see [Restore databases in a Kubernetes environment](scalar-kubernetes/RestoreDatabase.mdx). ================================================ FILE: docs/manage-monitor-overview.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Monitor Overview Scalar Manager is a centralized management and monitoring solution for ScalarDB within Kubernetes cluster environments that allows you to: - Check the availability of ScalarDB. - Schedule or execute pausing jobs that create transactionally consistent periods in the databases used by ScalarDB. - Check the time-series metrics and logs of ScalarDB through Grafana dashboards. For more details about Scalar Manager, see [Scalar Manager Overview](scalar-manager/overview.mdx). ## Deploy Scalar Manager You can deploy Scalar Manager by using a Helm Chart. For details on how to deploy Scalar Manager, see [Deploy Scalar Manager](helm-charts/getting-started-scalar-manager.mdx). ================================================ FILE: docs/manage-overview.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Manage Overview In this category, you can follow guides to help you manage ScalarDB. - For details on how to scale ScalarDB, see [Scale](scalar-kubernetes/HowToScaleScalarDB.mdx). - For details on how to upgrade ScalarDB, see [Upgrade](scalar-kubernetes/HowToUpgradeScalarDB.mdx). ## Monitor In this sub-category, you can learn how to monitor your ScalarDB deployment. For an overview of this sub-category, see [Monitor Overview](manage-monitor-overview.mdx). ## Back up and restore In this sub-category, you can learn how to back up and restore the databases that are connected to your ScalarDB deployment. For an overview of this sub-category, see [Back Up and Restore Databases](manage-backup-and-restore.mdx). ================================================ FILE: docs/migrate-overview.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Migrate Overview For details on importing your tables or migrating your applications and databases to a ScalarDB-based environment, see the following guides: - [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](schema-loader-import.mdx) - [How to Migrate Your Applications and Databases into a ScalarDB-Based Environment](scalardb-sql/migration-guide.mdx) ================================================ FILE: docs/multi-storage-transactions.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Multi-Storage Transactions ScalarDB transactions can span multiple storages or databases while maintaining ACID compliance by using a feature called *multi-storage transactions*. This page explains how multi-storage transactions work and how to configure the feature in ScalarDB. ## How multi-storage transactions work in ScalarDB In ScalarDB, the `multi-storage` implementation holds multiple storage instances and has mappings from a namespace name to a proper storage instance. When an operation is executed, the multi-storage transactions feature chooses a proper storage instance from the specified namespace by using the namespace-storage mapping and uses that storage instance. ## How to configure ScalarDB to support multi-storage transactions To enable multi-storage transactions, you need to specify `consensus-commit` as the value for `scalar.db.transaction_manager`, `multi-storage` as the value for `scalar.db.storage`, and configure your databases in the ScalarDB properties file. The following is an example of configurations for multi-storage transactions: ```properties # Consensus Commit is required to support multi-storage transactions. scalar.db.transaction_manager=consensus-commit # Multi-storage implementation is used for Consensus Commit. scalar.db.storage=multi-storage # Define storage names by using a comma-separated format. # In this case, "cassandra" and "mysql" are used. scalar.db.multi_storage.storages=cassandra,mysql # Define the "cassandra" storage. # When setting storage properties, such as `storage`, `contact_points`, `username`, and `password`, for multi-storage transactions, the format is `scalar.db.multi_storage.storages..`. # For example, to configure the `scalar.db.contact_points` property for Cassandra, specify `scalar.db.multi_storage.storages.cassandra.contact_point`. scalar.db.multi_storage.storages.cassandra.storage=cassandra scalar.db.multi_storage.storages.cassandra.contact_points=localhost scalar.db.multi_storage.storages.cassandra.username=cassandra scalar.db.multi_storage.storages.cassandra.password=cassandra # Define the "mysql" storage. # When defining JDBC-specific configurations for multi-storage transactions, you can follow a similar format of `scalar.db.multi_storage.storages..`. # For example, to configure the `scalar.db.jdbc.connection_pool.min_idle` property for MySQL, specify `scalar.db.multi_storage.storages.mysql.jdbc.connection_pool.min_idle`. scalar.db.multi_storage.storages.mysql.storage=jdbc scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://localhost:3306/ scalar.db.multi_storage.storages.mysql.username=root scalar.db.multi_storage.storages.mysql.password=mysql # Define the JDBC-specific configurations for the "mysql" storage. scalar.db.multi_storage.storages.mysql.jdbc.connection_pool.min_idle=5 scalar.db.multi_storage.storages.mysql.jdbc.connection_pool.max_total=25 # Define namespace mapping from a namespace name to a storage. # The format is ":,...". scalar.db.multi_storage.namespace_mapping=user:cassandra,coordinator:mysql # Define the default storage that's used if a specified table doesn't have any mapping. scalar.db.multi_storage.default_storage=cassandra ``` For additional configurations, see [Multi-storage configurations](configurations.mdx#multi-storage-configurations). ## Hands-on tutorial For a hands-on tutorial, see [Create a Sample Application That Supports Multi-Storage Transactions](scalardb-samples/multi-storage-transaction-sample/README.mdx). ================================================ FILE: docs/onboarding.mdx ================================================ --- id: onboarding title: Redirecting ... --- import {useEffect} from 'react'; import {useLocation} from '@docusaurus/router'; export default function Redirect() { const location = useLocation(); useEffect(() => { let path = location.pathname; if (path.startsWith('/en-us/')) { path = path.replace('/en-us/', '/ja-jp/'); } else if (!path.startsWith('/ja-jp/')) { path = `/ja-jp${path}`; } // 🔥 Force full reload (fixes broken page issue) window.location.replace(path); }, [location]); return

This page doesn't exist in English. Redirecting to Japanese version...

; } ================================================ FILE: docs/overview.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Overview This page describes what ScalarDB is and its primary use cases. ## What is ScalarDB? ScalarDB is a universal hybrid transaction/analytical processing (HTAP) engine for diverse databases. It runs as middleware on databases and virtually unifies diverse databases by achieving ACID transactions and real-time analytics across them to simplify the complexity of managing multiple databases or multiple instances of a single database. ![How ScalarDB simplifies complex data management architecture.](images/scalardb.png) As a versatile solution, ScalarDB supports a range of databases, including: - Relational databases that support JDBC, such as IBM Db2, MariaDB, Microsoft SQL Server, MySQL, Oracle Database, PostgreSQL, SQLite, and their compatible databases, like Amazon Aurora and YugabyteDB. - NoSQL databases like Amazon DynamoDB, Apache Cassandra, and Azure Cosmos DB. For details on which databases ScalarDB supports, refer to [Databases](requirements.mdx#databases). ## Why ScalarDB? Several solutions, such as global transaction managers, data federation engines, and HTAP systems, have similar goals, but they are limited in the following perspectives: - Global transaction managers (like Oracle MicroTx and Atomikos) are designed to run transactions across a limited set of heterogeneous databases (like only XA-compliant databases). - Data federation engines (like Denodo and Starburst) are designed to run analytical queries across heterogeneous databases. - HTAP systems (like TiDB and SingleStore) run both transactions and analytical queries only on homogeneous databases. In other words, they virtually unify databases, but with limitations. For example, with data federation engines, users can run read-only analytical queries on a virtualized view across multiple databases. However, they often need to run update queries separately for each database. Unlike other solutions, ScalarDB stands out by offering the ability to run both transactional and analytical queries on heterogeneous databases, which can significantly simplify database management. The following table summarizes how ScalarDB is different from the other solutions. | | Transactions across heterogeneous databases | Analytics across heterogeneous databases | | :------------------------------------------------------------: | :------------------------------------------------------------------: | :--------------------------------------: | | Global transaction managers (like Oracle MicroTx and Atomikos) | Yes (but existing solutions support only a limited set of databases) | No | | Data federation engines (like Denodo and Starburst) | No | Yes | | HTAP systems (like TiDB and SingleStore) | No (support homogeneous databases only) | No (support homogeneous databases only) | | **ScalarDB** | **Yes (supports various databases)** | **Yes** | ## ScalarDB use cases ScalarDB can be used in various ways. Here are the three primary use cases of ScalarDB. ### Managing siloed databases easily Many enterprises comprise several organizations, departments, and business units to support agile business operations, which often leads to siloed information systems. In particular, different organizations likely manage different applications with different databases. Managing such siloed databases is challenging because applications must communicate with each database separately and properly deal with the differences between databases. ScalarDB simplifies the management of siloed databases with a unified interface, enabling users to treat the databases as if they were a single database. For example, users can run (analytical) join queries over multiple databases without interacting with the databases respectively. ### Managing consistency between multiple database Modern architectures, like the microservice architecture, encourage a system to separate a service and its database into smaller subsets to increase system modularity and development efficiency. However, managing diverse databases, especially of different kinds, is challenging because applications must ensure the correct states (or, in other words, consistencies) of those databases, even using transaction management patterns like Saga and TCC. ScalarDB simplifies managing such diverse databases with a correctness guarantee (or, in other words, ACID with strict serializability), enabling you to focus on application development without worrying about guaranteeing consistency between databases. ### Simplifying data management in a data mesh Enterprises have been investing their time in building [data meshes](https://martinfowler.com/articles/data-mesh-principles.html) to streamline and scale data utilization. However, constructing a data mesh is not necessarily easy. For example, there are many technical issues in how to manage decentralized data. ScalarDB simplifies the management of decentralized databases in a data mesh, for example, by providing a unified API for all the databases in a data mesh to align with the data-as-a-product principle easily. ### Reducing database migration hurdles Applications tend to be locked into using a certain database because of the specific capabilities that the database provides. Such database lock-in discourages upgrading or changing the database because doing so often requires rewriting the application. ScalarDB provides a unified interface for diverse databases. Thus, once an application is written by using the ScalarDB interface, it becomes portable, which helps to achieve seamless database migration without rewriting the application. ## Further reading - [ScalarDB Technical Overview](https://speakerdeck.com/scalar/scalar-db-universal-transaction-manager) - [ScalarDB Research Paper [VLDB'23]](https://dl.acm.org/doi/10.14778/3611540.3611563) ================================================ FILE: docs/quickstart-overview.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Quickstart Overview In this category, you can follow quickstart tutorials for how to get started with running transactions and queries through ScalarDB. ## Try running transactions through the ScalarDB Core library In this sub-category, you can follow tutorials on how to run ACID transactions through the ScalarDB Core library, which is publicly available under the Apache 2 License. For an overview of this sub-category, see [ScalarDB Core Quickstart Overview](quickstart-scalardb-core-overview.mdx). ## Try running transactions through ScalarDB Cluster In this sub-category, you can see tutorials on how to run ACID transactions through ScalarDB Cluster, which is a [gRPC](https://grpc.io/) server that wraps the ScalarDB Core library. For an overview of this sub-category, see [ScalarDB Cluster Quickstart Overview](quickstart-scalardb-cluster-overview.mdx). :::note ScalarDB Cluster is available only in the Enterprise edition. ::: ## Try running analytical queries through ScalarDB Analytics In this sub-category, you can see tutorials on how to run analytical queries over the databases that you write through ScalarDB by using a component called ScalarDB Analytics. ScalarDB Analytics targets both ScalarDB-managed databases, which are updated through ScalarDB transactions, and non-ScalarDB-managed databases. For an overview of this sub-category, see [ScalarDB Analytics Quickstart Overview](quickstart-scalardb-analytics-overview.mdx). ================================================ FILE: docs/quickstart-scalardb-analytics-overview.mdx ================================================ --- tags: - Community - Enterprise Option displayed_sidebar: docsEnglish --- # ScalarDB Analytics Quickstart Overview In this sub-category, you can see tutorials on how to run analytical queries over the databases that you write through ScalarDB by using a component called ScalarDB Analytics. - To try running analytical queries through Spark, see [Getting Started with ScalarDB Analytics](scalardb-analytics/quickstart.mdx). ================================================ FILE: docs/quickstart-scalardb-cluster-overview.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster Quickstart Overview In this sub-category, you can see tutorials on how to run ACID transactions through ScalarDB Cluster, which is a [gRPC](https://grpc.io/) server that wraps the ScalarDB Core library. - To try running transactions, see [Getting Started with ScalarDB Cluster](scalardb-cluster/getting-started-with-scalardb-cluster.mdx). - To try running transactions through the SQL interface via JDBC, see [Getting Started with ScalarDB Cluster SQL via JDBC](scalardb-cluster/getting-started-with-scalardb-cluster-sql-jdbc.mdx). - To try running transactions through the SQL interface via Spring Data JDBC, see [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](scalardb-cluster/getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx). - To try running transactions through the GraphQL interface, see [Getting Started with ScalarDB Cluster GraphQL](scalardb-cluster/getting-started-with-scalardb-cluster-graphql.mdx). ================================================ FILE: docs/quickstart-scalardb-core-overview.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Core Quickstart Overview In this sub-category, you can follow tutorials on how to run ACID transactions through the ScalarDB Core library, which is publicly available under the Apache 2 license. - To try running transactions, see [Getting Started with ScalarDB](getting-started-with-scalardb.mdx). - To try running transactions by using Kotlin, see [Getting Started with ScalarDB by Using Kotlin](getting-started-with-scalardb-by-using-kotlin.mdx). ================================================ FILE: docs/requirements.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Requirements import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This page outlines the requirements for using each ScalarDB component, including the programming languages and their versions, supported databases and their versions, and the necessary configurations. ## Core ScalarDB Core is a key component of ScalarDB, providing a database manager with an abstraction layer that abstracts underlying databases. For more information, see [ScalarDB Design](./design.mdx). ### Languages and runtimes ScalarDB Core provides a Java client SDK for interacting with ScalarDB. It also includes tools, such as Schema Loader and Data Loader, which run on the Java Virtual Machine (JVM). #### Java The ScalarDB Core library is available on the Maven Central Repository. You can add the library as a build dependency to your application by using Gradle or Maven. For more details, see [Add ScalarDB to Your Build](./add-scalardb-to-your-build.mdx). For building applications that integrate with the library, the following Java Development Kits (JDKs) are verified and supported. Java Runtime Environments (JREs) of these JDKs are also supported for running the tools. ### Databases ScalarDB runs on top of the following databases and their versions. #### Relational databases | Version | Aurora MySQL 3 | Aurora MySQL 2 | | :---------------- | :------------- | :------------- | | **ScalarDB 3.18** | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | | **ScalarDB 3.16** | ✅ | ✅ | | **ScalarDB 3.15** | ✅ | ✅ | | **ScalarDB 3.14** | ✅ | ✅ | | Version | Aurora PostgreSQL 17 | Aurora PostgreSQL 16 | Aurora PostgreSQL 15 | Aurora PostgreSQL 14 | Aurora PostgreSQL 13 | |:------------------|:---------------------|:---------------------|:---------------------|:---------------------|:---------------------| | **ScalarDB 3.18** | ✅ | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.16** | ✅ | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.15** | ✅ | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.14** | ✅ | ✅ | ✅ | ✅ | ✅ | | Version | Db2 12.1 | Db2 11.5 | | :---------------- | :------- | :------- | | **ScalarDB 3.18** | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | | **ScalarDB 3.16** | ✅ | ✅ | | **ScalarDB 3.15** | ❌ | ❌ | | **ScalarDB 3.14** | ❌ | ❌ | :::note Only Linux, UNIX, and Windows versions of Db2 are supported. The z/OS version is not currently supported. ::: | Version | MariaDB 11.4 | MariaDB 10.11 | | :---------------- | :----------- | :------------ | | **ScalarDB 3.18** | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | | **ScalarDB 3.16** | ✅ | ✅ | | **ScalarDB 3.15** | ✅ | ✅ | | **ScalarDB 3.14** | ✅ | ✅ | | Version | MySQL 8.4 | MySQL 8.0 | | :---------------- | :-------- | :-------- | | **ScalarDB 3.18** | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | | **ScalarDB 3.16** | ✅ | ✅ | | **ScalarDB 3.15** | ✅ | ✅ | | **ScalarDB 3.14** | ✅ | ✅ | | Version | Oracle Database 23ai | Oracle Database 21c | Oracle Database 19c | | :---------------- | :------------------- | :------------------ | :------------------ | | **ScalarDB 3.18** | ✅ | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | ✅ | | **ScalarDB 3.16** | ✅ | ✅ | ✅ | | **ScalarDB 3.15** | ✅ | ✅ | ✅ | | **ScalarDB 3.14** | ✅ | ✅ | ✅ | | Version | PostgreSQL 17 | PostgreSQL 16 | PostgreSQL 15 | PostgreSQL 14 | PostgreSQL 13 | | :---------------- | :------------ | :------------ | :------------ | :------------ | ------------- | | **ScalarDB 3.18** | ✅ | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.16** | ✅ | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.15** | ✅ | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.14** | ✅ | ✅ | ✅ | ✅ | ✅ | | Version | SQL Server 2022 | SQL Server 2019 | SQL Server 2017 | | :---------------- | :-------------- | :-------------- | :-------------- | | **ScalarDB 3.18** | ✅ | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | ✅ | | **ScalarDB 3.16** | ✅ | ✅ | ✅ | | **ScalarDB 3.15** | ✅ | ✅ | ✅ | | **ScalarDB 3.14** | ✅ | ✅ | ✅ | | Version | SQLite 3 | | :---------------- | :------- | | **ScalarDB 3.18** | ✅ | | **ScalarDB 3.17** | ✅ | | **ScalarDB 3.16** | ✅ | | **ScalarDB 3.15** | ✅ | | **ScalarDB 3.14** | ✅ | #### NewSQL databases | Version | AlloyDB 16 | AlloyDB 15 | |:------------------|:-----------|:-----------| | **ScalarDB 3.18** | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | | **ScalarDB 3.16** | ❌ | ❌ | | **ScalarDB 3.15** | ❌ | ❌ | | **ScalarDB 3.14** | ❌ | ❌ | | Version | Spanner (PostgreSQL dialect) | |:------------------|:--------| | **ScalarDB 3.18** | ✅ | | **ScalarDB 3.17** | ❌ | | **ScalarDB 3.16** | ❌ | | **ScalarDB 3.15** | ❌ | | **ScalarDB 3.14** | ❌ | :::note ScalarDB works only for [Spanner database with the PostgreSQL dialect](https://docs.cloud.google.com/spanner/docs/create-manage-databases). ::: | Version | TiDB 8.5 | TiDB 7.5 | TiDB 6.5 | |:------------------|:---------|:---------|----------| | **ScalarDB 3.18** | ✅ | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | ✅ | | **ScalarDB 3.16** | ❌ | ❌ | ❌ | | **ScalarDB 3.15** | ❌ | ❌ | ❌ | | **ScalarDB 3.14** | ❌ | ❌ | ❌ | | Version | YugabyteDB 2 | | :---------------- | :----------- | | **ScalarDB 3.18** | ✅ | | **ScalarDB 3.17** | ✅ | | **ScalarDB 3.16** | ✅ | | **ScalarDB 3.15** | ✅ | | **ScalarDB 3.14** | ✅ | #### NoSQL databases | Version | DynamoDB | | :---------------- | :------- | | **ScalarDB 3.18** | ✅ | | **ScalarDB 3.17** | ✅ | | **ScalarDB 3.16** | ✅ | | **ScalarDB 3.15** | ✅ | | **ScalarDB 3.14** | ✅ | | Version | Cassandra 5.0 | Cassandra 4.1 | Cassandra 3.11 | Cassandra 3.0 | | :---------------- |:--------------|:--------------| :------------- | :------------ | | **ScalarDB 3.18** | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.17** | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.16** | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.15** | ✅ | ✅ | ✅ | ✅ | | **ScalarDB 3.14** | ✅ | ✅ | ✅ | ✅ | | Version | Cosmos DB for NoSQL | | :---------------- | :------------------ | | **ScalarDB 3.18** | ✅ | | **ScalarDB 3.17** | ✅ | | **ScalarDB 3.16** | ✅ | | **ScalarDB 3.15** | ✅ | | **ScalarDB 3.14** | ✅ | #### Object Storage :::note Object Storage support is currently in Private Preview. For more details, please [contact us](https://www.scalar-labs.com/contact) or wait for this feature to become publicly available in a future version. ::: | Version | S3 | | :---------------- | :------------------ | | **ScalarDB 3.18** | ✅ | | **ScalarDB 3.17** | ✅ | | **ScalarDB 3.16** | ❌ | | **ScalarDB 3.15** | ❌ | | **ScalarDB 3.14** | ❌ | | Version | Blob Storage | | :---------------- | :------------------ | | **ScalarDB 3.18** | ✅ | | **ScalarDB 3.17** | ✅ | | **ScalarDB 3.16** | ❌ | | **ScalarDB 3.15** | ❌ | | **ScalarDB 3.14** | ❌ | | Version | Cloud Storage | | :---------------- | :------------------ | | **ScalarDB 3.18** | ✅ | | **ScalarDB 3.17** | ✅ | | **ScalarDB 3.16** | ❌ | | **ScalarDB 3.15** | ❌ | | **ScalarDB 3.14** | ❌ | :::note For details on how to configure each database, see [Configurations for the Underlying Databases of ScalarDB](./database-configurations.mdx). ::: ### Database permission requirements ScalarDB requires specific permissions to perform its operations on the underlying databases. :::note ScalarDB assumes that the same underlying database user account is used for all administrative and CRUD operations. ::: #### Relational databases This section describes the permission requirements for relational databases. If you're using Db2, the following authority must be granted. - `DBADM` - `DBADM` If you're using MariaDB, the following privileges must be granted. - `CREATE` - `CREATE VIEW` - `DROP` - `INDEX` - `ALTER` - `SELECT` - `INSERT` - `UPDATE` - `DELETE` - `CREATE` - `CREATE VIEW` - `DROP` - `INDEX` - `ALTER` - `SELECT` - `INSERT` - `UPDATE` - `DELETE` If you're using MySQL, the following privileges must be granted. - `CREATE` - `CREATE VIEW` - `DROP` - `INDEX` - `ALTER` - `SELECT` - `INSERT` - `UPDATE` - `DELETE` - `CREATE` - `CREATE VIEW` - `DROP` - `INDEX` - `ALTER` - `SELECT` - `INSERT` - `UPDATE` - `DELETE` If you're using Oracle Database, the following privileges must be granted. - `CREATE SESSION` - `CREATE USER` - `DROP USER` - `ALTER USER` - `CREATE ANY TABLE` - `DROP ANY TABLE` - `CREATE ANY INDEX` - `ALTER ANY INDEX` - `DROP ANY INDEX` - `ALTER ANY TABLE` - `SELECT ANY TABLE` - `INSERT ANY TABLE` - `UPDATE ANY TABLE` - `DELETE ANY TABLE` - `CREATE ANY VIEW` - `DROP ANY VIEW` - `CREATE SESSION` - `CREATE USER` - `DROP USER` - `ALTER USER` - `CREATE ANY TABLE` - `DROP ANY TABLE` - `CREATE ANY INDEX` - `ALTER ANY INDEX` - `DROP ANY INDEX` - `ALTER ANY TABLE` - `SELECT ANY TABLE` - `INSERT ANY TABLE` - `UPDATE ANY TABLE` - `DELETE ANY TABLE` - `CREATE ANY VIEW` - `DROP ANY VIEW` - `CREATE SESSION` - `CREATE USER` - `DROP USER` - `ALTER USER` - `CREATE ANY TABLE` - `DROP ANY TABLE` - `CREATE ANY INDEX` - `ALTER ANY INDEX` - `DROP ANY INDEX` - `ALTER ANY TABLE` - `SELECT ANY TABLE` - `INSERT ANY TABLE` - `UPDATE ANY TABLE` - `DELETE ANY TABLE` - `CREATE ANY VIEW` - `DROP ANY VIEW` If you're using PostgreSQL, the following database privilege must be granted. - `CREATE` - `CREATE` - `CREATE` - `CREATE` - `CREATE` If you're using SQL Server, the following database permissions must be granted. - `CREATE SCHEMA` - `CREATE TABLE` - `CREATE VIEW` - `CREATE SCHEMA` - `CREATE TABLE` - `CREATE VIEW` - `CREATE SCHEMA` - `CREATE TABLE` - `CREATE VIEW` #### NewSQL databases This section describes the permission requirements for NewSQL databases. If you're using AlloyDB, the following database privilege must be granted. - `CREATE` - `CREATE` If you're using Spanner, the following role must be granted: - `roles/spanner.databaseUser` If you're using TiDB, the following privileges must be granted. - `CREATE` - `CREATE VIEW` - `DROP` - `INDEX` - `ALTER` - `SELECT` - `INSERT` - `UPDATE` - `DELETE` - `CREATE` - `CREATE VIEW` - `DROP` - `INDEX` - `ALTER` - `SELECT` - `INSERT` - `UPDATE` - `DELETE` - `CREATE` - `CREATE VIEW` - `DROP` - `INDEX` - `ALTER` - `SELECT` - `INSERT` - `UPDATE` - `DELETE` If you're using YugabyteDB, the following database privilege must be granted. - `CREATE` #### NoSQL databases This section describes the permission requirements for NoSQL databases. If you're using Amazon DynamoDB, the following actions must be granted. - `dynamodb:ConditionCheckItem` - `dynamodb:PutItem` - `dynamodb:ListTables` - `dynamodb:DeleteItem` - `dynamodb:Scan` - `dynamodb:Query` - `dynamodb:UpdateItem` - `dynamodb:DeleteTable` - `dynamodb:UpdateContinuousBackups` - `dynamodb:CreateTable` - `dynamodb:DescribeTable` - `dynamodb:GetItem` - `dynamodb:DescribeContinuousBackups` - `dynamodb:UpdateTable` - `application-autoscaling:RegisterScalableTarget` - `application-autoscaling:DeleteScalingPolicy` - `application-autoscaling:PutScalingPolicy` - `application-autoscaling:DeregisterScalableTarget` - `application-autoscaling:TagResource` If you're using Apache Cassandra, the following privileges must be granted. - `CREATE` - `DROP` - `ALTER` - `SELECT` - `MODIFY` - `CREATE` - `DROP` - `ALTER` - `SELECT` - `MODIFY` - `CREATE` - `DROP` - `ALTER` - `SELECT` - `MODIFY` - `CREATE` - `DROP` - `ALTER` - `SELECT` - `MODIFY` ScalarDB requires an Azure Cosmos DB key, which has full access to Azure Cosmos DB for NoSQL, for authentication. #### Object Storage This section describes the permission requirements for object storage. If you're using Amazon S3 as Object Storage, the following actions must be granted: - `s3:PutObject` - `s3:GetObject` - `s3:DeleteObject` - `s3:ListBucket` ScalarDB requires an access key, which has full access to Azure Blob Storage, for authentication. If you're using Google Cloud Storage as Object Storage, the following role must be granted: - `Storage Object Admin (roles/storage.objectAdmin)` ## Cluster ScalarDB Cluster is a component that provides a clustering solution for the Core component to work as a clustered server. For more information, see [ScalarDB Design](./design.mdx). ### Languages and runtimes ScalarDB Cluster provides Java and .NET client SDKs that wrap gRPC-generated clients for ease of use. #### Java The Java client SDK for ScalarDB Cluster is available on the Maven Central Repository. You can add the library as a build dependency to your application by using Gradle or Maven. For more details, see [Add ScalarDB Cluster Java Client SDK to your build](scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#add-scalardb-cluster-java-client-sdk-to-your-build). For building applications that integrate with the library, the following Java Development Kits (JDKs) are verified and supported: :::note The ScalarDB Cluster Embedding Java Client SDK library requires Java 17 or 21 from one of the vendors listed above. ::: #### .NET The .NET client SDK for ScalarDB Cluster is available as a NuGet package. For more details, see [Install the SDK](scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-transactions.mdx#install-the-sdk). For building applications that integrate with the library, the following .NET versions are verified and supported: - [.NET 8.0](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) - [.NET 6.0](https://dotnet.microsoft.com/en-us/download/dotnet/6.0) #### Other languages Since ScalarDB Cluster uses gRPC, you can also create your own client in your preferred language by using the generated clients from the proto file. If you need the proto file, please [contact support](https://www.scalar-labs.com/support). ### Databases Since ScalarDB Cluster uses Core to interact with databases, the requirements for databases are the same as those for Core. For more information, see [Databases](#databases). ### Required ports ScalarDB Cluster requires the following ports to be accessible. These default port numbers can be configured via the corresponding properties as needed: - 60053 (Administrative API / Transactional API / SQL API / administrator service; configurable via `scalar.db.cluster.node.port`) - 60054 (internal gRPC communication between cluster nodes; configurable via `scalar.db.cluster.internal.node.port`. This port only needs to be reachable between cluster nodes and should typically be restricted from client and public networks.) - 8080 (GraphQL; configurable via `scalar.db.graphql.port`) - 9080 (metrics; configurable via `scalar.db.cluster.node.prometheus_exporter_port`) :::note If you configure `scalar.db.cluster.node.admin.port` to run the administrator service (`pause`, `unpause`, and `checkPaused`) on a dedicated port, that dedicated port will also need to be accessible. In this case, only the administrator service runs on that port instead of the port specified by `scalar.db.cluster.node.port`; the port specified by `scalar.db.cluster.node.port` must still remain accessible for the other gRPC APIs, including the Administrative API, Transactional API, and SQL API. ::: ### Kubernetes ScalarDB Cluster is provided as a cluster consisting of one or more Pods on the Kubernetes platform in production environments. ScalarDB Cluster supports the following platforms and tools. #### Platform - **[Kubernetes](https://kubernetes.io/):** 1.32 - 1.35 - **[Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/)** - **[Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service)** - **[Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift):** 4.18 - 4.21 :::note - ScalarDB Cluster does not support Extended Update Support (EUS) versions of **Red Hat OpenShift**. - ScalarDB Cluster for Red Hat OpenShift is provided as a container image based on Red Hat Universal Base Images (UBI), which uses Red Hat Build of OpenJDK. - If you plan to use Red Hat OpenShift, it is your responsibility to ensure you have a valid subscription. Please contact Red Hat directly for more information. Red Hat offers various subscription options for Red Hat OpenShift, and you may also need to purchase a separate subscription for Red Hat Build of OpenJDK. ::: #### Package manager - **[Helm](https://helm.sh/):** 3.5+ ## Analytics ScalarDB Analytics is a component that provides scalable analytical processing for the data managed by the Core component or managed by applications that don't use ScalarDB. For more information, see [ScalarDB Design](./design.mdx). ### Spark ScalarDB Analytics uses [Apache Spark](https://spark.apache.org/) for the query engine. It supports the following versions of Spark. Cloud-managed Spark services, such as Amazon EMR, Azure Synapse Analytics, and Databricks, are also supported as long as they use the Spark versions listed in the following table. | ScalarDB Analytics Version | Spark Versions | Scala Versions | | :------------------------- | :------------- | :------------- | | 3.18 | 3.5, 3.4 | 2.13, 2.12 | | 3.17 | 3.5, 3.4 | 2.13, 2.12 | | 3.16 | 3.5, 3.4 | 2.13, 2.12 | | 3.15 | 3.5, 3.4 | 2.13, 2.12 | | 3.14 | 3.5, 3.4 | 2.13, 2.12 | ### Languages and runtimes ScalarDB Analytics provides a library for running federated queries on Spark. It also provides a tool called ScalarDB Analytics CLI, which runs on the Java Virtual Machine (JVM). The library is available on the Maven Central Repository. You need to specify the library when setting up Spark. For more details, see [Set up ScalarDB Analytics in the Spark configuration](scalardb-analytics/run-analytical-queries.mdx#set-up-scalardb-analytics-in-the-spark-configuration). :::note Since Spark and Scala may be incompatible among different minor versions, the library offers different artifacts for various Spark and Scala versions, named in the format `scalardb-analytics-spark-all-_`. Make sure that you select the artifact matching the Spark and Scala versions you're using. For example, if you're using Spark 3.5 with Scala 2.13, you must specify `scalardb-analytics-spark-all-3.5_2.13`. ::: #### Java For using the library on Spark, the following JREs are verified and supported: :::note The Java runtime versions available for Spark on each cloud service are based on those supported by Apache Spark. For reference, the following shows the Spark service versions and corresponding Java runtime versions on major cloud services: - **Amazon EMR:** EMR 7.x: Spark 3.5 (Java 17); EMR 6.x: Spark 3.4 (Java 8) - **Azure Synapse Analytics:** Spark 3.5 (Java 17); Spark 3.4 (Java 11) - **Databricks:** Runtime 16.4 LTS: Spark 3.5 (Java 17); Runtime 15.4 LTS: Spark 3.5 (Java 8); Runtime 14.3 LTS: Spark 3.5 (Java 8); Runtime 13.3 LTS: Spark 3.4 (Java 8) For detailed information, please refer to the documentation for the cloud service you are using. ::: For running ScalarDB Analytics CLI, the following JREs are verified and supported: ### Databases ScalarDB Analytics runs on top of the following databases and their versions. #### ScalarDB ScalarDB Analytics can run analytical queries on the databases managed by ScalarDB Core and Cluster. It uses the ScalarDB Core library of the same version to interact with these databases, as shown below. | ScalarDB Analytics version | ScalarDB Core version | | :------------------------- | :-------------------- | | 3.18 | 3.18 | | 3.17 | 3.17 | | 3.16 | 3.16 | | 3.15 | 3.15 | | 3.14 | 3.14 | For the supported databases and their versions, see [Databases](#databases). #### Relational databases ScalarDB Analytics can run analytical queries on the following relational databases **not** managed by ScalarDB Core and Cluster. | Version | MySQL 8.4 | MySQL 8.0 | | :-------------------------- | :-------- | :-------- | | **ScalarDB Analytics 3.18** | ✅ | ✅ | | **ScalarDB Analytics 3.17** | ✅ | ✅ | | **ScalarDB Analytics 3.16** | ❌ | ✅ | | **ScalarDB Analytics 3.15** | ❌ | ✅ | | **ScalarDB Analytics 3.14** | ❌ | ✅ | | Version | Oracle Database 23ai | Oracle Database 21c | Oracle Database 19c | | :-------------------------- | :------------------- | :------------------ | :------------------ | | **ScalarDB Analytics 3.18** | ✅ | ✅ | ✅ | | **ScalarDB Analytics 3.17** | ✅ | ✅ | ✅ | | **ScalarDB Analytics 3.16** | ✅ | ❌ | ❌ | | **ScalarDB Analytics 3.15** | ✅ | ❌ | ❌ | | **ScalarDB Analytics 3.14** | ✅ | ❌ | ❌ | | Version | PostgreSQL 18 | PostgreSQL 17 | PostgreSQL 16 | | :-------------------------- | :------------ | :------------ | :------------ | | **ScalarDB Analytics 3.18** | ✅ | ✅ | ✅ | | **ScalarDB Analytics 3.17** | ✅ | ✅ | ✅ | | **ScalarDB Analytics 3.16** | ❌ | ❌ | ✅ | | **ScalarDB Analytics 3.15** | ❌ | ❌ | ✅ | | **ScalarDB Analytics 3.14** | ❌ | ❌ | ✅ | | Version | SQL Server 2022 | SQL Server 2019 | SQL Server 2017 | | :-------------------------- | :-------------- | :-------------- | :-------------- | | **ScalarDB Analytics 3.18** | ✅ | ✅ | ✅ | | **ScalarDB Analytics 3.17** | ✅ | ✅ | ✅ | | **ScalarDB Analytics 3.16** | ❌ | ✅ | ❌ | | **ScalarDB Analytics 3.15** | ❌ | ✅ | ❌ | | **ScalarDB Analytics 3.14** | ❌ | ✅ | ❌ | #### NoSQL databases ScalarDB Analytics can run analytical queries on the following NoSQL databases **not** managed by ScalarDB Core and Cluster. | Version | DynamoDB | | :-------------------------- | :------- | | **ScalarDB Analytics 3.18** | ✅ | | **ScalarDB Analytics 3.17** | ✅ | | **ScalarDB Analytics 3.16** | ✅ | | **ScalarDB Analytics 3.15** | ✅ | | **ScalarDB Analytics 3.14** | ✅ | #### Analytical platforms ScalarDB Analytics can run analytical queries on the following analytical platforms **not** managed by ScalarDB Core and Cluster. | Version | Databricks | | :-------------------------- | :--------- | | **ScalarDB Analytics 3.18** | ✅ | | **ScalarDB Analytics 3.17** | ✅ | | **ScalarDB Analytics 3.16** | ✅ | | **ScalarDB Analytics 3.15** | ❌ | | **ScalarDB Analytics 3.14** | ❌ | :::note ScalarDB Analytics supports running queries on Databricks by using both SQL warehouses and compute. For compute, all default compute policies are supported. ::: | Version | Snowflake | | :-------------------------- | :-------- | | **ScalarDB Analytics 3.18** | ✅ | | **ScalarDB Analytics 3.17** | ✅ | | **ScalarDB Analytics 3.16** | ✅ | | **ScalarDB Analytics 3.15** | ❌ | | **ScalarDB Analytics 3.14** | ❌ | #### ScalarDB Analytics server database The ScalarDB Analytics server manages catalog information in its database using ScalarDB Core. You can use any database supported by ScalarDB Core for this purpose. For the supported databases and their versions, see [Databases](#databases). ### Database permission requirements ScalarDB Analytics requires read permissions to perform its operations on the underlying databases. For databases managed under ScalarDB Core and Cluster, the databases are already configured according to [Database permission requirements](#database-permission-requirements), so no additional configuration is required. For databases **not** managed under ScalarDB Core and Cluster, make sure you register your data sources with users who have read permission on the data sources. For instructions on registering your data sources, see [Create your catalog](scalardb-analytics/create-scalardb-analytics-catalog.mdx#create-your-catalog). Since ScalarDB Analytics server uses ScalarDB Core to manage catalog information, you need to configure a database user for ScalarDB Analytics server according to the [Database permission requirements](#database-permission-requirements) documentation for ScalarDB Core. ### Required ports ScalarDB Analytics requires the following ports to be accessible. These default port numbers can be configured as needed: - 11051 (catalog service) - 11052 (metering service) - The port number that Apache Spark uses depends on how you deploy the Spark cluster. For details on which ports you need to make accessible, please refer to your Spark service provider's documentation. ### Kubernetes The server component of ScalarDB Analytics (ScalarDB Analytics server) is provided as a Pod on the Kubernetes platform in production environments. ScalarDB Analytics supports the following platforms and tools. #### Platform - **[Kubernetes](https://kubernetes.io/):** 1.32 - 1.35 - **[Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/)** - **[Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service)** - **[Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift):** TBD #### Package manager - **[Helm](https://helm.sh/):** 3.5+ ================================================ FILE: docs/roadmap.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Roadmap This roadmap provides a look into the proposed future of ScalarDB. The purpose of this roadmap is to provide visibility into what changes may be coming so that you can more closely follow progress, learn about key milestones, and give feedback during development. This roadmap will be updated as new versions of ScalarDB are released. :::warning During the course of development, this roadmap is subject to change based on user needs and feedback. **Do not schedule your release plans according to the contents of this roadmap.** If you have a feature request or want to prioritize feature development, please create an issue in [GitHub](https://github.com/scalar-labs/scalardb/issues). ::: ### CY2026 Q2 #### New components - **ScalarDB Saga** - Users will be able to use ScalarDB Saga, a distributed transaction coordinator that implements the Saga pattern, to manage long-running transactions across multiple services and databases in a microservices architecture. - **ScalarDB 2PC coordinator** - Users will be able to use ScalarDB 2PC coordinator, a distributed transaction coordinator that implements the two-phase commit protocol, to manage distributed transactions across multiple services and databases in a microservices architecture. #### New features - **Authentication with OIDC** - Users will be able to authenticate to ScalarDB Cluster and ScalarDB Analytics by using OpenID Connect (OIDC). - **Universal authentication and authorization** - Users will be able to be given access to ScalarDB Cluster and ScalarDB Analytics by using a unified authentication and authorization method. - **Coordinator-based redo logging for pause-less backup** - Users will be able to perform pause-less backup by using coordinator-based redo logging, which allows users to back up data without pausing the system. - **Universal catalog** - Users will be able to manage metadata, including schemas and semantic information, for operational and analytical databases across separate business domains in a unified manner. #### Usability - **Strongly consistent secondary index access** - Users will be able to access secondary indexes with strong consistency so that they can get the most up-to-date data from secondary indexes. - **Support more SQL syntax/features** - Users will be able to use more SQL syntax/features in ScalarDB SQL so that they can express their intentions in a simpler way. For example, users will be able to use the IN operator in ScalarDB SQL. - **Addition of DECIMAL data types** - Users will be able to use DECIMAL data types so that users can handle decimal numbers with high precision. - **Elimination of out-of-memory errors due to large scans** - Users will be able to issue large scans without experiencing out-of-memory errors. #### Performance - **Reduction of storage space needed for managing ScalarDB metadata** - Users will likely use less storage space to run ScalarDB. ScalarDB will remove the before image of committed transactions after they are committed. However, whether or not those committed transactions will impact actual storage space depends on the underlying databases. - **Predicate pushdown** - Users will be able to benefit from predicate pushdown in ScalarDB Analytics so that filtering operations are pushed down to the underlying databases to reduce data transfer and improve query performance. #### Cloud support - **Google Cloud Marketplace support for ScalarDB Cluster and ScalarDB Analytics** - Users will be able to deploy ScalarDB Cluster and ScalarDB Analytics by using the Google Cloud Marketplace offering, which enables users to use a pay-as-you-go subscription model. - **Red Hat Ecosystem Catalog integration for ScalarDB Cluster** - Users will be able to deploy ScalarDB Cluster from Red Hat Ecosystem Catalog, which enables users to use ScalarDB Cluster as Red Hat-certified third-party products and services. ### CY2026 Q3 #### New features - **Audit logging** - Users will be able to view and manage the access logs of ScalarDB Cluster and Analytics, mainly for auditing purposes. - **Native secondary index** - Users will be able to define flexible secondary indexes. The existing secondary index is limited because it is implemented based on the common capabilities of the supported databases' secondary indexes. Therefore, for example, users cannot define a multi-column index. The new secondary index will be created at the ScalarDB layer so that users can create more flexible indexes, like a multi-column index. - **Views** - Users will be able to define views so that they can manage multiple different databases in an easier and simplified way. - **Two-layered query engines** - Users will be able to use two-layered query engines to better federate analytical queries across domains. The first-layer query engine is responsible for federating analytical queries across various domains, while the second-layer query engine federates analytical queries within a specific domain across its corresponding databases. - **LangGraph4j integration** - Users will be able to use LangGraph4j to build AI applications that can interact with ScalarDB. #### Cloud support - **Azure Marketplace support for ScalarDB Cluster and ScalarDB Analytics** - Users will be able to deploy ScalarDB Cluster and ScalarDB Analytics by using the Azure Marketplace offering, which enables users to use a pay-as-you-go subscription model. #### Integration - **Kong integration** - Users will be able to integrate ScalarDB Cluster with Kong Ingress Controller so that they can manage and secure access to ScalarDB Cluster more easily. ### CY2026 Q4 #### New features - **Stored procedures** - Users will be able to define stored procedures so that they can execute a set of operations with a complex logic inside ScalarDB Cluster. - **Triggers** - Users will be able to define triggers so that they can automatically execute a set of operations when a specific event occurs in ScalarDB Cluster. - **Queue interface** - Users will be able to use ScalarDB as a message queue so that they can build event-driven architectures more easily. #### Performance - **Adaptive caching** - Users will be able to benefit from adaptive caching in ScalarDB Analytics so that frequently accessed data is cached automatically to improve query performance. ### CY2027 #### New features - **User-defined functions (UDFs)** - Users will be able to define functions so that they can use functions in SQLs to express complex logic in a simpler way. - **GraphDB support** - Users will be able to use GraphDBs as underlying databases through ScalarDB Cluster. ================================================ FILE: docs/run-non-transactional-storage-operations-through-library.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Run Non-Transactional Storage Operations Through the Core Library import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import StorageSetupTabs from './components/_getting-started-setup-storage.mdx'; This guide explains how to run non-transactional storage operations through the ScalarDB Core library. ## Preparation For the purpose of this guide, you will set up a database and ScalarDB by using a sample in the ScalarDB samples repository. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the necessary files by running the following command: ```console cd scalardb-samples/scalardb-sample ``` ## Set up a database Follow the instructions below to configure your database for ScalarDB. For a list of databases that ScalarDB supports, see [Databases](requirements.mdx#databases). For a comprehensive list of configurations for ScalarDB, see [ScalarDB Configurations](configurations.mdx). ## Configure ScalarDB to run non-transactional storage operations To run non-transactional storage operations, you need to configure the `scalar.db.transaction_manager` property to `single-crud-operation` in the configuration file **database.properties**: ```properties scalar.db.transaction_manager=single-crud-operation ``` ## Create or import a schema ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. - **Need to create a database schema?** See [ScalarDB Schema Loader](schema-loader.mdx). - **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](schema-loader-import.mdx). ## Load initial data as necessary ScalarDB Data Loader is a utility for importing and exporting data with ScalarDB. - **Need to import data into your database?** See [Importing data](data-loader.mdx#importing-data). - **Need to export data from your database?** See [Exporting data](data-loader.mdx#exporting-data). ## Create your Java application This section describes how to add the ScalarDB Core library to your project and how to configure it to run non-transactional storage operations by using Java. ### Add ScalarDB to your project The ScalarDB library is available on the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb). You can add the library as a build dependency to your application by using Gradle or Maven. Select your build tool, and follow the instructions to add the build dependency for ScalarDB to your application. To add the build dependency for ScalarDB by using Gradle, add the following to `build.gradle` in your application: ```gradle dependencies { implementation 'com.scalar-labs:scalardb:3.18.0' } ``` To add the build dependency for ScalarDB by using Maven, add the following to `pom.xml` in your application: ```xml com.scalar-labs scalardb 3.18.0 ``` ### Use the Java API For details about the Java API, see [ScalarDB Java API Guide](api-guide.mdx). :::note The following limitations apply to non-transactional storage operations: - Beginning a transaction is not supported. For more details, see [Execute transactions without beginning or starting a transaction](api-guide.mdx#execute-transactions-without-beginning-or-starting-a-transaction). - Executing multiple mutations in a single transaction is not supported. ::: ### Learn more - [Javadoc](https://javadoc.io/doc/com.scalar-labs/scalardb/3.18.0/index.html) ================================================ FILE: docs/run-non-transactional-storage-operations-through-primitive-crud-interface.mdx ================================================ --- tags: - Community displayed_sidebar: docsEnglish --- # Run Non-Transactional Storage Operations Through the Primitive CRUD Interface This page explains how to run non-transactional storage operations through the primitive CRUD interface, also known as the Storage API. This guide assumes that you have an advanced understanding of ScalarDB. One of the keys to achieving storage-agnostic or database-agnostic ACID transactions on top of existing storage and database systems is the storage abstraction capabilities that ScalarDB provides. Storage abstraction defines a [data model](design.mdx#data-model) and the APIs (Storage API) that issue operations on the basis of the data model. Although you will likely use the [Transactional API](api-guide.mdx#transactional-api) in most cases, another option is to use the Storage API. The benefits of using the Storage API include the following: - As with the Transactional API, you can write your application code without worrying too much about the underlying storage implementation. - If you don't need transactions for some of the data in your application, you can use the Storage API to partially avoid transactions, which results in faster execution. :::warning Directly using the Storage API or mixing the Transactional API and the Storage API could cause unexpected behavior. For example, since the Storage API cannot provide transaction capability, the API could cause anomalies or data inconsistency if failures occur when executing operations. Therefore, you should be *very* careful about using the Storage API and use it only if you know exactly what you are doing. ::: ## Storage API Example This section explains how the Storage API can be used in a basic electronic money application. :::warning The electronic money application is simplified for this example and isn’t suitable for a production environment. ::: ### ScalarDB configuration Before you begin, you should configure ScalarDB in the same way mentioned in [Getting Started with ScalarDB](getting-started-with-scalardb.mdx). With that in mind, this Storage API example assumes that the configuration file `scalardb.properties` exists. ### Set up the database schema You need to define the database schema (the method in which the data will be organized) in the application. For details about the supported data types, see [Database Adapters](database-adapters.mdx). For this example, create a file named `emoney-storage.json` in the `scalardb/docs/getting-started` directory. Then, add the following JSON code to define the schema. :::note In the following JSON, the `transaction` field is set to `false`, which indicates that you should use this table with the Storage API. ::: ```json { "emoney.account": { "transaction": false, "partition-key": [ "id" ], "clustering-key": [], "columns": { "id": "TEXT", "balance": "INT" } } } ``` To apply the schema, go to the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you are using to the `getting-started` folder. Then, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: ```console java -jar scalardb-schema-loader-.jar --config scalardb.properties -f emoney-storage.json ``` ### Example code The following is example source code for the electronic money application that uses the Storage API. :::warning As previously mentioned, since the Storage API cannot provide transaction capability, the API could cause anomalies or data inconsistency if failures occur when executing operations. Therefore, you should be *very* careful about using the Storage API and use it only if you know exactly what you are doing. ::: ```java public class ElectronicMoney { private static final String SCALARDB_PROPERTIES = System.getProperty("user.dir") + File.separator + "scalardb.properties"; private static final String NAMESPACE = "emoney"; private static final String TABLENAME = "account"; private static final String ID = "id"; private static final String BALANCE = "balance"; private final DistributedStorage storage; public ElectronicMoney() throws IOException { StorageFactory factory = StorageFactory.create(SCALARDB_PROPERTIES); storage = factory.getStorage(); } public void charge(String id, int amount) throws ExecutionException { // Retrieve the current balance for id Get get = Get.newBuilder() .namespace(NAMESPACE) .table(TABLENAME) .partitionKey(Key.ofText(ID, id)) .build(); Optional result = storage.get(get); // Calculate the balance int balance = amount; if (result.isPresent()) { int current = result.get().getInt(BALANCE); balance += current; } // Update the balance Put put = Put.newBuilder() .namespace(NAMESPACE) .table(TABLENAME) .partitionKey(Key.ofText(ID, id)) .intValue(BALANCE, balance) .build(); storage.put(put); } public void pay(String fromId, String toId, int amount) throws ExecutionException { // Retrieve the current balances for ids Get fromGet = Get.newBuilder() .namespace(NAMESPACE) .table(TABLENAME) .partitionKey(Key.ofText(ID, fromId)) .build(); Get toGet = Get.newBuilder() .namespace(NAMESPACE) .table(TABLENAME) .partitionKey(Key.ofText(ID, toId)) .build(); Optional fromResult = storage.get(fromGet); Optional toResult = storage.get(toGet); // Calculate the balances (it assumes that both accounts exist) int newFromBalance = fromResult.get().getInt(BALANCE) - amount; int newToBalance = toResult.get().getInt(BALANCE) + amount; if (newFromBalance < 0) { throw new RuntimeException(fromId + " doesn't have enough balance."); } // Update the balances Put fromPut = Put.newBuilder() .namespace(NAMESPACE) .table(TABLENAME) .partitionKey(Key.ofText(ID, fromId)) .intValue(BALANCE, newFromBalance) .build(); Put toPut = Put.newBuilder() .namespace(NAMESPACE) .table(TABLENAME) .partitionKey(Key.ofText(ID, toId)) .intValue(BALANCE, newToBalance) .build(); storage.put(fromPut); storage.put(toPut); } public int getBalance(String id) throws ExecutionException { // Retrieve the current balances for id Get get = Get.newBuilder() .namespace(NAMESPACE) .table(TABLENAME) .partitionKey(Key.ofText(ID, id)) .build(); Optional result = storage.get(get); int balance = -1; if (result.isPresent()) { balance = result.get().getInt(BALANCE); } return balance; } public void close() { storage.close(); } } ``` ## Storage API guide The Storage API is composed of the Administrative API and CRUD API. ### Administrative API You can execute administrative operations programmatically as described in this section. :::note Another method that you could use to execute administrative operations is by using [Schema Loader](schema-loader.mdx). ::: #### Get a `DistributedStorageAdmin` instance To execute administrative operations, you first need to get a `DistributedStorageAdmin` instance. You can obtain the `DistributedStorageAdmin` instance from `StorageFactory` as follows: ```java StorageFactory storageFactory = StorageFactory.create(""); DistributedStorageAdmin admin = storageFactory.getStorageAdmin(); ``` For details about configurations, see [ScalarDB Configurations](configurations.mdx). After you have executed all administrative operations, you should close the `DistributedStorageAdmin` instance as follows: ```java admin.close(); ``` #### Create a namespace Before creating tables, namespaces must be created since a table belongs to one namespace. You can create a namespace as follows: ```java // Create the namespace "ns". If the namespace already exists, an exception will be thrown. admin.createNamespace("ns"); // Create the namespace only if it does not already exist. boolean ifNotExists = true; admin.createNamespace("ns", ifNotExists); // Create the namespace with options. Map options = ...; admin.createNamespace("ns", options); ``` For details about creation options, see [Creation options](./api-guide.mdx#creation-options). #### Create a table When creating a table, you should define the table metadata and then create the table. To define the table metadata, you can use `TableMetadata`. The following shows how to define the columns, partition key, clustering key including clustering orders, and secondary indexes of a table: ```java // Define the table metadata. TableMetadata tableMetadata = TableMetadata.newBuilder() .addColumn("c1", DataType.INT) .addColumn("c2", DataType.TEXT) .addColumn("c3", DataType.BIGINT) .addColumn("c4", DataType.FLOAT) .addColumn("c5", DataType.DOUBLE) .addPartitionKey("c1") .addClusteringKey("c2", Scan.Ordering.Order.DESC) .addClusteringKey("c3", Scan.Ordering.Order.ASC) .addSecondaryIndex("c4") .build(); ``` For details about the data model of ScalarDB, see [Data Model](design.mdx#data-model). Then, create a table as follows: ```java // Create the table "ns.tbl". If the table already exists, an exception will be thrown. admin.createTable("ns", "tbl", tableMetadata); // Create the table only if it does not already exist. boolean ifNotExists = true; admin.createTable("ns", "tbl", tableMetadata, ifNotExists); // Create the table with options. Map options = ...; admin.createTable("ns", "tbl", tableMetadata, options); ``` For details about creation options, see [Creation options](./api-guide.mdx#creation-options-1). #### Create a secondary index You can create a secondary index as follows: ```java // Create a secondary index on column "c5" for table "ns.tbl". If a secondary index already exists, an exception will be thrown. admin.createIndex("ns", "tbl", "c5"); // Create the secondary index only if it does not already exist. boolean ifNotExists = true; admin.createIndex("ns", "tbl", "c5", ifNotExists); // Create the secondary index with options. Map options = ...; admin.createIndex("ns", "tbl", "c5", options); ``` For details about creation options, see [Creation options](./api-guide.mdx#creation-options-2). #### Add a new column to a table You can add a new, non-partition key column to a table as follows: ```java // Add a new column "c6" with the INT data type to the table "ns.tbl". admin.addNewColumnToTable("ns", "tbl", "c6", DataType.INT) ``` :::warning You should carefully consider adding a new column to a table because the execution time may vary greatly depending on the underlying storage. Please plan accordingly and consider the following, especially if the database runs in production: - **For Cosmos DB for NoSQL and DynamoDB:** Adding a column is almost instantaneous as the table schema is not modified. Only the table metadata stored in a separate table is updated. - **For Cassandra:** Adding a column will only update the schema metadata and will not modify the existing schema records. The cluster topology is the main factor for the execution time. Changes to the schema metadata are shared to each cluster node via a gossip protocol. Because of this, the larger the cluster, the longer it will take for all nodes to be updated. - **For relational databases (MySQL, Oracle, etc.):** Adding a column shouldn't take a long time to execute. ::: #### Truncate a table You can truncate a table as follows: ```java // Truncate the table "ns.tbl". admin.truncateTable("ns", "tbl"); ``` #### Drop a secondary index You can drop a secondary index as follows: ```java // Drop the secondary index on column "c5" from table "ns.tbl". If the secondary index does not exist, an exception will be thrown. admin.dropIndex("ns", "tbl", "c5"); // Drop the secondary index only if it exists. boolean ifExists = true; admin.dropIndex("ns", "tbl", "c5", ifExists); ``` #### Drop a table You can drop a table as follows: ```java // Drop the table "ns.tbl". If the table does not exist, an exception will be thrown. admin.dropTable("ns", "tbl"); // Drop the table only if it exists. boolean ifExists = true; admin.dropTable("ns", "tbl", ifExists); ``` #### Drop a namespace You can drop a namespace as follows: ```java // Drop the namespace "ns". If the namespace does not exist, an exception will be thrown. admin.dropNamespace("ns"); // Drop the namespace only if it exists. boolean ifExists = true; admin.dropNamespace("ns", ifExists); ``` #### Get existing namespaces You can get the existing namespaces as follows: ```java Set namespaces = admin.getNamespaceNames(); ``` :::note This method extracts the namespace names of user tables dynamically. As a result, only namespaces that contain tables are returned. Starting from ScalarDB 4.0, we plan to improve the design to remove this limitation. ::: #### Get the tables of a namespace You can get the tables of a namespace as follows: ```java // Get the tables of the namespace "ns". Set tables = admin.getNamespaceTableNames("ns"); ``` #### Get table metadata You can get table metadata as follows: ```java // Get the table metadata for "ns.tbl". TableMetadata tableMetadata = admin.getTableMetadata("ns", "tbl"); ``` #### Repair a table You can repair the table metadata of an existing table as follows: ```java // Repair the table "ns.tbl" with options. TableMetadata tableMetadata = TableMetadata.newBuilder() ... .build(); Map options = ...; admin.repairTable("ns", "tbl", tableMetadata, options); ``` ### Implement CRUD operations The following sections describe CRUD operations. #### Get a `DistributedStorage` instance To execute CRUD operations in the Storage API, you need to get a `DistributedStorage` instance. You can get an instance as follows: ```java StorageFactory storageFactory = StorageFactory.create(""); DistributedStorage storage = storageFactory.getStorage(); ``` After you have executed all CRUD operations, you should close the `DistributedStorage` instance as follows: ```java storage.close(); ``` #### `Get` operation `Get` is an operation to retrieve a single record specified by a primary key. You need to create a `Get` object first, and then you can execute the object by using the `storage.get()` method as follows: ```java // Create a `Get` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Get get = Get.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .projections("c1", "c2", "c3", "c4") .build(); // Execute the `Get` operation. Optional result = storage.get(get); ``` You can also specify projections to choose which columns are returned. For details about how to construct `Key` objects, see [Key construction](api-guide.mdx#key-construction). And, for details about how to handle `Result` objects, see [Handle Result objects](api-guide.mdx#handle-result-objects). ##### Specify a consistency level You can specify a consistency level in each operation (`Get`, `Scan`, `Put`, and `Delete`) in the Storage API as follows: ```java Get get = Get.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .consistency(Consistency.LINEARIZABLE) // Consistency level .build(); ``` The following table describes the three consistency levels: | Consistency level | Description | | ------------------- | ----------- | | `SEQUENTIAL` | Sequential consistency assumes that the underlying storage implementation makes all operations appear to take effect in some sequential order and the operations of each individual process appear in this sequence. | | `EVENTUAL` | Eventual consistency assumes that the underlying storage implementation makes all operations take effect eventually. | | `LINEARIZABLE` | Linearizable consistency assumes that the underlying storage implementation makes each operation appear to take effect atomically at some point between its invocation and completion. | ##### Execute `Get` by using a secondary index You can execute a `Get` operation by using a secondary index. Instead of specifying a partition key, you can specify an index key (indexed column) to use a secondary index as follows: ```java // Create a `Get` operation by using a secondary index. Key indexKey = Key.ofFloat("c4", 1.23F); Get get = Get.newBuilder() .namespace("ns") .table("tbl") .indexKey(indexKey) .projections("c1", "c2", "c3", "c4") .build(); // Execute the `Get` operation. Optional result = storage.get(get); ``` :::note If the result has more than one record, `storage.get()` will throw an exception. ::: #### `Scan` operation `Scan` is an operation to retrieve multiple records within a partition. You can specify clustering-key boundaries and orderings for clustering-key columns in `Scan` operations. You need to create a `Scan` object first, and then you can execute the object by using the `storage.scan()` method as follows: ```java // Create a `Scan` operation. Key partitionKey = Key.ofInt("c1", 10); Key startClusteringKey = Key.of("c2", "aaa", "c3", 100L); Key endClusteringKey = Key.of("c2", "aaa", "c3", 300L); Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .start(startClusteringKey, true) // Include startClusteringKey .end(endClusteringKey, false) // Exclude endClusteringKey .projections("c1", "c2", "c3", "c4") .orderings(Scan.Ordering.desc("c2"), Scan.Ordering.asc("c3")) .limit(10) .build(); // Execute the `Scan` operation. Scanner scanner = storage.scan(scan); ``` You can omit the clustering-key boundaries or specify either a `start` boundary or an `end` boundary. If you don't specify `orderings`, you will get results ordered by the clustering order that you defined when creating the table. In addition, you can specify `projections` to choose which columns are returned and use `limit` to specify the number of records to return in `Scan` operations. ##### Handle `Scanner` objects A `Scan` operation in the Storage API returns a `Scanner` object. If you want to get results one by one from the `Scanner` object, you can use the `one()` method as follows: ```java Optional result = scanner.one(); ``` Or, if you want to get a list of all results, you can use the `all()` method as follows: ```java List results = scanner.all(); ``` In addition, since `Scanner` implements `Iterable`, you can use `Scanner` in a for-each loop as follows: ```java for (Result result : scanner) { ... } ``` Remember to close the `Scanner` object after getting the results: ```java scanner.close(); ``` Or you can use `try`-with-resources as follows: ```java try (Scanner scanner = storage.scan(scan)) { ... } ``` ##### Execute `Scan` by using a secondary index You can execute a `Scan` operation by using a secondary index. Instead of specifying a partition key, you can specify an index key (indexed column) to use a secondary index as follows: ```java // Create a `Scan` operation by using a secondary index. Key indexKey = Key.ofFloat("c4", 1.23F); Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .indexKey(indexKey) .projections("c1", "c2", "c3", "c4") .limit(10) .build(); // Execute the `Scan` operation. Scanner scanner = storage.scan(scan); ``` :::note You can't specify clustering-key boundaries and orderings in `Scan` by using a secondary index. ::: ##### Execute `Scan` without specifying a partition key to retrieve all the records of a table You can execute a `Scan` operation without specifying a partition key. Instead of calling the `partitionKey()` method in the builder, you can call the `all()` method to scan a table without specifying a partition key as follows: ```java // Create a `Scan` operation without specifying a partition key. Key partitionKey = Key.ofInt("c1", 10); Key startClusteringKey = Key.of("c2", "aaa", "c3", 100L); Key endClusteringKey = Key.of("c2", "aaa", "c3", 300L); Scan scan = Scan.newBuilder() .namespace("ns") .table("tbl") .all() .projections("c1", "c2", "c3", "c4") .limit(10) .build(); // Execute the `Scan` operation. Scanner scanner = storage.scan(scan); ``` :::note You can't specify clustering-key boundaries and orderings in `Scan` without specifying a partition key. ::: #### `Put` operation `Put` is an operation to put a record specified by a primary key. The operation behaves as an upsert operation for a record, in which the operation updates the record if the record exists or inserts the record if the record does not exist. You need to create a `Put` object first, and then you can execute the object by using the `storage.put()` method as follows: ```java // Create a `Put` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); // Execute the `Put` operation. storage.put(put); ``` You can also put a record with `null` values as follows: ```java Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", null) .doubleValue("c5", null) .build(); ``` :::note If you specify `enableImplicitPreRead()`, `disableImplicitPreRead()`, or `implicitPreReadEnabled()` in the `Put` operation builder, they will be ignored. ::: #### `Delete` operation `Delete` is an operation to delete a record specified by a primary key. You need to create a `Delete` object first, and then you can execute the object by using the `storage.delete()` method as follows: ```java // Create a `Delete` operation. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .build(); // Execute the `Delete` operation. storage.delete(delete); ``` #### `Put` and `Delete` with a condition You can write arbitrary conditions (for example, a bank account balance must be equal to or more than zero) that you require an operation to meet before being executed by implementing logic that checks the conditions. Alternatively, you can write simple conditions in a mutation operation, such as `Put` and `Delete`. When a `Put` or `Delete` operation includes a condition, the operation is executed only if the specified condition is met. If the condition is not met when the operation is executed, an exception called `NoMutationException` will be thrown. ##### Conditions for `Put` In a `Put` operation in the Storage API, you can specify a condition that causes the `Put` operation to be executed only when the specified condition matches. This operation is like a compare-and-swap operation where the condition is compared and the update is performed atomically. You can specify a condition in a `Put` operation as follows: ```java // Build a condition. MutationCondition condition = ConditionBuilder.putIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) .build(); Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .condition(condition) // condition .build(); ``` Other than the `putIf` condition, you can specify the `putIfExists` and `putIfNotExists` conditions as follows: ```java // Build a `putIfExists` condition. MutationCondition putIfExistsCondition = ConditionBuilder.putIfExists(); // Build a `putIfNotExists` condition. MutationCondition putIfNotExistsCondition = ConditionBuilder.putIfNotExists(); ``` ##### Conditions for `Delete` Similar to a `Put` operation, you can specify a condition in a `Delete` operation in the Storage API. You can specify a condition in a `Delete` operation as follows: ```java // Build a condition. MutationCondition condition = ConditionBuilder.deleteIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) .build(); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKey) .condition(condition) // condition .build(); ``` In addition to using the `deleteIf` condition, you can specify the `deleteIfExists` condition as follows: ```java // Build a `deleteIfExists` condition. MutationCondition deleteIfExistsCondition = ConditionBuilder.deleteIfExists(); ``` #### Mutate operation Mutate is an operation to execute multiple mutations (`Put` and `Delete` operations) in a single partition. You need to create mutation objects first, and then you can execute the objects by using the `storage.mutate()` method as follows: ```java // Create `Put` and `Delete` operations. Key partitionKey = Key.ofInt("c1", 10); Key clusteringKeyForPut = Key.of("c2", "aaa", "c3", 100L); Put put = Put.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForPut) .floatValue("c4", 1.23F) .doubleValue("c5", 4.56) .build(); Key clusteringKeyForDelete = Key.of("c2", "bbb", "c3", 200L); Delete delete = Delete.newBuilder() .namespace("ns") .table("tbl") .partitionKey(partitionKey) .clusteringKey(clusteringKeyForDelete) .build(); // Execute the operations. storage.mutate(Arrays.asList(put, delete)); ``` :::note A Mutate operation only accepts mutations for a single partition; otherwise, an exception will be thrown. In addition, if you specify multiple conditions in a Mutate operation, the operation will be executed only when all the conditions match. ::: #### Default namespace for CRUD operations A default namespace for all CRUD operations can be set by using a property in the ScalarDB configuration. ```properties scalar.db.default_namespace_name= ``` Any operation that does not specify a namespace will use the default namespace set in the configuration. ```java // This operation will target the default namespace. Scan scanUsingDefaultNamespace = Scan.newBuilder() .table("tbl") .all() .build(); // This operation will target the "ns" namespace. Scan scanUsingSpecifiedNamespace = Scan.newBuilder() .namespace("ns") .table("tbl") .all() .build(); ``` ================================================ FILE: docs/run-transactions-through-scalardb-core-library.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Run Transactions Through the ScalarDB Core Library import StorageSetupTabs from './components/_getting-started-setup-storage.mdx'; This guide explains how to configure your ScalarDB properties file and create schemas to run transactions through a one-phase or a two-phase commit interface by using the ScalarDB Core library. ## Preparation For the purpose of this guide, you will set up a database and ScalarDB by using a sample in the ScalarDB samples repository. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the necessary files by running the following command: ```console cd scalardb-samples/scalardb-sample ``` ## Set up a database Follow the instructions below to configure your database for ScalarDB. For a list of databases that ScalarDB supports, see [Databases](requirements.mdx#databases). For a comprehensive list of configurations for ScalarDB, see [ScalarDB Configurations](configurations.mdx). ## Create or import a schema ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. - **Need to create a database schema?** See [ScalarDB Schema Loader](schema-loader.mdx). - **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](schema-loader-import.mdx). ## Load initial data as necessary ScalarDB Data Loader is a utility for importing and exporting data with ScalarDB. - **Need to import data into your database?** See [Importing data](data-loader.mdx#importing-data). - **Need to export data from your database?** See [Exporting data](data-loader.mdx#exporting-data). ## Run transactions by using Java - **Want to run transactions by using a one-phase commit interface?** See the [ScalarDB Java API Guide](api-guide.mdx#transactional-api). - **Want to run transactions by using a two-phase commit interface?** See [Transactions with a Two-Phase Commit Interface](two-phase-commit-transactions.mdx). ================================================ FILE: docs/scalardb-core-status-codes.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Core Error Codes This page provides a list of error codes in ScalarDB Core. ## Error code classes and descriptions | Class | Description | |:----------------|:---------------------------------------------------------| | `DB-CORE-1xxxx` | Errors for the user error category | | `DB-CORE-2xxxx` | Errors for the concurrency error category | | `DB-CORE-3xxxx` | Errors for the internal error category | | `DB-CORE-4xxxx` | Errors for the unknown transaction status error category | ## `DB-CORE-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-CORE-10000` **Message** ```markdown Only a single-column index is supported. Operation: %s ``` ### `DB-CORE-10001` **Message** ```markdown The column of the specified index key is not indexed. Operation: %s ``` ### `DB-CORE-10002` **Message** ```markdown The index key is not properly specified. Operation: %s ``` ### `DB-CORE-10003` **Message** ```markdown Clustering keys cannot be specified when using an index. Operation: %s ``` ### `DB-CORE-10004` **Message** ```markdown Orderings cannot be specified when using an index. Operation: %s ``` ### `DB-CORE-10005` **Message** ```markdown The limit cannot be negative. Operation: %s ``` ### `DB-CORE-10006` **Message** ```markdown Cross-partition scan is not enabled. Operation: %s ``` ### `DB-CORE-10007` **Message** ```markdown Cross-partition scan ordering is not enabled. Operation: %s ``` ### `DB-CORE-10008` **Message** ```markdown Cross-partition scan filtering is not enabled. Operation: %s ``` ### `DB-CORE-10009` **Message** ```markdown The specified projection is not found. Projection: %s, Operation: %s ``` ### `DB-CORE-10010` **Message** ```markdown The clustering key boundary is not properly specified. Operation: %s ``` ### `DB-CORE-10011` **Message** ```markdown The start clustering key is not properly specified. Operation: %s ``` ### `DB-CORE-10012` **Message** ```markdown The end clustering key is not properly specified. Operation: %s ``` ### `DB-CORE-10013` **Message** ```markdown Orderings are not properly specified. Operation: %s ``` ### `DB-CORE-10014` **Message** ```markdown The specified ordering column is not found. Ordering: %s, Operation: %s ``` ### `DB-CORE-10015` **Message** ```markdown The condition is not properly specified. Operation: %s ``` ### `DB-CORE-10016` **Message** ```markdown The table does not exist. Table: %s ``` ### `DB-CORE-10017` **Message** ```markdown The column value is not properly specified. Column: %s, Operation: %s ``` ### `DB-CORE-10018` **Message** ```markdown The mutations are empty ``` ### `DB-CORE-10019` **Message** ```markdown The storage does not support mutations across multiple partitions. Storage: %s; Mutations: %s ``` ### `DB-CORE-10020` **Message** ```markdown The partition key is not properly specified. Operation: %s ``` ### `DB-CORE-10021` **Message** ```markdown The clustering key is not properly specified. Operation: %s ``` ### `DB-CORE-10022` **Message** ```markdown The authentication and authorization feature is not enabled. To use this feature, you must enable it. Note that this feature is supported only in the ScalarDB Enterprise edition ``` ### `DB-CORE-10023` **Message** ```markdown This condition is not allowed for the PutIf operation. Condition: %s ``` ### `DB-CORE-10024` **Message** ```markdown This condition is not allowed for the DeleteIf operation. Condition: %s ``` ### `DB-CORE-10025` **Message** ```markdown Operator must be LIKE or NOT_LIKE. Operator: %s ``` ### `DB-CORE-10026` **Message** ```markdown An escape character must be a string of a single character or an empty string ``` ### `DB-CORE-10027` **Message** ```markdown The LIKE pattern must not be null ``` ### `DB-CORE-10028` **Message** ```markdown The LIKE pattern must not include only an escape character ``` ### `DB-CORE-10029` **Message** ```markdown The LIKE pattern must not end with an escape character ``` ### `DB-CORE-10030` **Message** ```markdown The column %s does not exist ``` ### `DB-CORE-10031` **Message** ```markdown This operation is not supported when getting records of a database without using an index ``` ### `DB-CORE-10032` **Message** ```markdown This operation is not supported when getting records of a database by using an index ``` ### `DB-CORE-10033` **Message** ```markdown This operation is not supported when scanning all the records of a database or scanning records of a database by using an index ``` ### `DB-CORE-10034` **Message** ```markdown This operation is supported only when scanning records of a database by using an index ``` ### `DB-CORE-10035` **Message** ```markdown This operation is not supported when scanning records of a database by using an index ``` ### `DB-CORE-10037` **Message** ```markdown This operation is supported only when no conditions are specified. If you want to modify a condition, please use clearConditions() to remove all existing conditions first ``` ### `DB-CORE-10038` **Message** ```markdown One or more columns must be specified ``` ### `DB-CORE-10039` **Message** ```markdown One or more partition keys must be specified ``` ### `DB-CORE-10040` **Message** ```markdown The column definition must be specified since %s is specified as a partition key ``` ### `DB-CORE-10041` **Message** ```markdown The column definition must be specified since %s is specified as a clustering key ``` ### `DB-CORE-10042` **Message** ```markdown Invalid ID specified. ID: %d ``` ### `DB-CORE-10043` **Message** ```markdown The transaction is not active. Status: %s ``` ### `DB-CORE-10044` **Message** ```markdown The transaction has already been committed. Status: %s ``` ### `DB-CORE-10045` **Message** ```markdown The transaction has not been prepared. Status: %s ``` ### `DB-CORE-10046` **Message** ```markdown The transaction has not been prepared or validated. Status: %s ``` ### `DB-CORE-10047` **Message** ```markdown The transaction already exists ``` ### `DB-CORE-10048` **Message** ```markdown A transaction associated with the specified transaction ID is not found. The transaction might have expired ``` ### `DB-CORE-10049` **Message** ```markdown %s is the system namespace name ``` ### `DB-CORE-10050` **Message** ```markdown The namespace already exists. Namespace: %s ``` ### `DB-CORE-10051` **Message** ```markdown The namespace does not exist. Namespace: %s ``` ### `DB-CORE-10052` **Message** ```markdown The table already exists. Table: %s ``` ### `DB-CORE-10053` **Message** ```markdown The namespace is not empty. Namespace: %s; Tables in the namespace: %s ``` ### `DB-CORE-10054` **Message** ```markdown The column does not exist. Table: %s; Column: %s ``` ### `DB-CORE-10055` **Message** ```markdown The index already exists. Table: %s; Column: %s ``` ### `DB-CORE-10056` **Message** ```markdown The index does not exist. Table: %s; Column: %s ``` ### `DB-CORE-10057` **Message** ```markdown The column already exists. Table: %s; Column: %s ``` ### `DB-CORE-10058` **Message** ```markdown The operation does not have the target namespace or table name. Operation: %s ``` ### `DB-CORE-10059` **Message** ```markdown The specified value of the property '%s' is not a number. Value: %s ``` ### `DB-CORE-10060` **Message** ```markdown The specified value of the property '%s' is not a boolean. Value: %s ``` ### `DB-CORE-10061` **Message** ```markdown Reading the file failed. File: %s ``` ### `DB-CORE-10062` **Message** ```markdown The property 'scalar.db.cross_partition_scan.enabled' must be set to true to use cross-partition scan with filtering or ordering ``` ### `DB-CORE-10063` **Message** ```markdown This column value is out of range for BigInt. Value: %s ``` ### `DB-CORE-10064` **Message** ```markdown This type is not supported. Name: %s, Type: %s ``` ### `DB-CORE-10065` **Message** ```markdown Storage '%s' is not found ``` ### `DB-CORE-10066` **Message** ```markdown Transaction manager '%s' is not found ``` ### `DB-CORE-10068` **Message** ```markdown Please use scan() for non-exact match selection. Operation: %s ``` ### `DB-CORE-10069` **Message** ```markdown Import-related functionality is not supported in Cassandra ``` ### `DB-CORE-10070` **Message** ```markdown The %s network strategy does not exist ``` ### `DB-CORE-10071` **Message** ```markdown The property 'scalar.db.contact_port' must be greater than or equal to zero ``` ### `DB-CORE-10073` **Message** ```markdown The BLOB type is not supported for clustering keys in Cosmos DB. Column: %s ``` ### `DB-CORE-10074` **Message** ```markdown Import-related functionality is not supported in Cosmos DB ``` ### `DB-CORE-10075` **Message** ```markdown The property 'scalar.db.contact_points' must not be empty ``` ### `DB-CORE-10076` **Message** ```markdown Cosmos DB supports only EQ, NE, IS_NULL, and IS_NOT_NULL operations for the BLOB type in conditions. Mutation: %s ``` ### `DB-CORE-10077` **Message** ```markdown The specified consistency level is not supported. Consistency level: %s ``` ### `DB-CORE-10078` **Message** ```markdown 0x00 bytes are not accepted in BLOB values in DESC order ``` ### `DB-CORE-10079` **Message** ```markdown Cannot encode a Text value that contains '\u0000' ``` ### `DB-CORE-10081` **Message** ```markdown An index column cannot be set to null or an empty value for Text or Blob in DynamoDB. Operation: %s ``` ### `DB-CORE-10082` **Message** ```markdown DynamoDB supports only EQ, NE, IS_NULL, and IS_NOT_NULL operations for the BOOLEAN type in conditions. Mutation: %s ``` ### `DB-CORE-10083` **Message** ```markdown Nested multi-storage definitions are not supported. Storage: %s ``` ### `DB-CORE-10084` **Message** ```markdown Storage not found. Storage: %s ``` ### `DB-CORE-10085` **Message** ```markdown The namespace name is not acceptable in SQLite. Namespace: %s ``` ### `DB-CORE-10086` **Message** ```markdown The table name is not acceptable in SQLite. Table: %s ``` ### `DB-CORE-10087` **Message** ```markdown Importing tables is not allowed in SQLite ``` ### `DB-CORE-10088` **Message** ```markdown The %s table must have a primary key ``` ### `DB-CORE-10089` **Message** ```markdown The RDB engine is not supported. JDBC connection URL: %s ``` ### `DB-CORE-10090` **Message** ```markdown Data type %s(%d) is not supported: %s ``` ### `DB-CORE-10091` **Message** ```markdown Data type %s is not supported: %s ``` ### `DB-CORE-10092` **Message** ```markdown Getting a transaction state is not supported in JDBC transactions ``` ### `DB-CORE-10093` **Message** ```markdown Rolling back a transaction is not supported in JDBC transactions ``` ### `DB-CORE-10094` **Message** ```markdown Coordinator tables already exist ``` ### `DB-CORE-10095` **Message** ```markdown Coordinator tables do not exist ``` ### `DB-CORE-10096` **Message** ```markdown The namespace %s is reserved. Any operations on this namespace are not allowed ``` ### `DB-CORE-10097` **Message** ```markdown Mutating transaction metadata columns is not allowed. Table: %s; Column: %s ``` ### `DB-CORE-10098` **Message** ```markdown A %s condition is not allowed on Put operations ``` ### `DB-CORE-10099` **Message** ```markdown A %s condition is not allowed on Delete operations ``` ### `DB-CORE-10100` **Message** ```markdown The condition is not allowed to target transaction metadata columns. Table: %s; Column: %s ``` ### `DB-CORE-10101` **Message** ```markdown The column '%s' is reserved as transaction metadata ``` ### `DB-CORE-10102` **Message** ```markdown Non-primary key columns with the 'before_' prefix, '%s', are reserved as transaction metadata ``` ### `DB-CORE-10103` **Message** ```markdown Put cannot have a condition when the target record is unread and implicit pre-read is disabled. Please read the target record beforehand or enable implicit pre-read: %s ``` ### `DB-CORE-10104` **Message** ```markdown Writing data already-deleted by the same transaction is not allowed ``` ### `DB-CORE-10106` **Message** ```markdown Scanning data already-written or already-deleted by the same transaction is not allowed ``` ### `DB-CORE-10107` **Message** ```markdown The transaction is not validated. When using the SERIALIZABLE isolation level, you need to call validate() before calling commit() ``` ### `DB-CORE-10108` **Message** ```markdown DynamoDB cannot batch more than 100 mutations at once ``` ### `DB-CORE-10126` **Message** ```markdown The mutation type is not supported. Only the Put or Delete type is supported. Mutation: %s ``` ### `DB-CORE-10127` **Message** ```markdown This condition is not allowed for the UpdateIf operation. Condition: %s ``` ### `DB-CORE-10128` **Message** ```markdown Cross-partition scan with ordering is not supported in Cassandra ``` ### `DB-CORE-10129` **Message** ```markdown Cross-partition scan with ordering is not supported in Cosmos DB ``` ### `DB-CORE-10130` **Message** ```markdown Cross-partition scan with ordering is not supported in DynamoDB ``` ### `DB-CORE-10136` **Message** ```markdown Getting a transaction state is not supported in single CRUD operation transactions ``` ### `DB-CORE-10137` **Message** ```markdown Rolling back a transaction is not supported in single CRUD operation transactions ``` ### `DB-CORE-10138` **Message** ```markdown Multiple mutations are not supported in single CRUD operation transactions ``` ### `DB-CORE-10139` **Message** ```markdown Beginning a transaction is not allowed in single CRUD operation transactions ``` ### `DB-CORE-10140` **Message** ```markdown Resuming a transaction is not allowed in single CRUD operation transactions ``` ### `DB-CORE-10141` **Message** ```markdown Using the group commit feature on the Coordinator table with a two-phase commit interface is not allowed ``` ### `DB-CORE-10142` **Message** ```markdown This operation is supported only when no conditions are specified. If you want to modify a condition, please use clearConditions() to remove all existing conditions first ``` ### `DB-CORE-10143` **Message** ```markdown The encryption feature is not enabled. To encrypt data at rest, you must enable this feature. Note that this feature is supported only in the ScalarDB Enterprise edition ``` ### `DB-CORE-10144` **Message** ```markdown The variable key column size must be greater than or equal to 64 ``` ### `DB-CORE-10145` **Message** ```markdown The value of the column %s in the primary key contains an illegal character. Primary-key columns must not contain any of the following characters in Cosmos DB: ':', '/', '\', '#', '?'. Value: %s ``` ### `DB-CORE-10146` **Message** ```markdown Inserting data already-written by the same transaction is not allowed ``` ### `DB-CORE-10147` **Message** ```markdown Deleting data already-inserted by the same transaction is not allowed ``` ### `DB-CORE-10152` **Message** ```markdown The attribute-based access control feature is not enabled. To use this feature, you must enable it. Note that this feature is supported only in the ScalarDB Enterprise edition ``` ### `DB-CORE-10158` **Message** ```markdown This DATE column value is out of the valid range. It must be between 1000-01-01 and 9999-12-12. Value: %s ``` ### `DB-CORE-10159` **Message** ```markdown This TIME column value precision cannot be shorter than one microsecond. Value: %s ``` ### `DB-CORE-10160` **Message** ```markdown This TIMESTAMP column value is out of the valid range. It must be between 1000-01-01T00:00:00.000 and 9999-12-31T23:59:59.999. Value: %s ``` ### `DB-CORE-10161` **Message** ```markdown This TIMESTAMP column value precision cannot be shorter than one millisecond. Value: %s ``` ### `DB-CORE-10162` **Message** ```markdown This TIMESTAMPTZ column value is out of the valid range. It must be between 1000-01-01T00:00:00.000Z to 9999-12-31T23:59:59.999Z. Value: %s ``` ### `DB-CORE-10163` **Message** ```markdown This TIMESTAMPTZ column value precision cannot be shorter than one millisecond. Value: %s ``` ### `DB-CORE-10164` **Message** ```markdown The underlying-storage data type %s is not supported as the ScalarDB %s data type: %s ``` ### `DB-CORE-10188` **Message** ```markdown The replication feature is not enabled. To use this feature, you must enable it. Note that this feature is supported only in the ScalarDB Enterprise edition ``` ### `DB-CORE-10205` **Message** ```markdown Some scanners were not closed. All scanners must be closed before committing the transaction ``` ### `DB-CORE-10206` **Message** ```markdown Some scanners were not closed. All scanners must be closed before preparing the transaction ``` ### `DB-CORE-10211` **Message** ```markdown Mutations are not allowed in read-only transactions. Transaction ID: %s ``` ### `DB-CORE-10212` **Message** ```markdown The storage does not support mutations across multiple records. Storage: %s; Mutations: %s ``` ### `DB-CORE-10213` **Message** ```markdown The storage does not support mutations across multiple tables. Storage: %s; Mutations: %s ``` ### `DB-CORE-10214` **Message** ```markdown The storage does not support mutations across multiple namespaces. Storage: %s; Mutations: %s ``` ### `DB-CORE-10215` **Message** ```markdown Mutations across multiple storages are not allowed. Mutations: %s ``` ### `DB-CORE-10216` **Message** ```markdown Primary key columns cannot be dropped. Table: %s; Column: %s ``` ### `DB-CORE-10217` **Message** ```markdown Cosmos DB does not support the feature for dropping columns ``` ### `DB-CORE-10218` **Message** ```markdown DynamoDB does not support the feature for dropping columns ``` ### `DB-CORE-10219` **Message** ```markdown Cosmos DB does not support the feature for renaming columns ``` ### `DB-CORE-10220` **Message** ```markdown DynamoDB does not support the feature for renaming columns ``` ### `DB-CORE-10221` **Message** ```markdown Cassandra does not support renaming non-primary key columns ``` ### `DB-CORE-10222` **Message** ```markdown Db2 does not support renaming primary key or index key columns ``` ### `DB-CORE-10223` **Message** ```markdown Import-related functionality is not supported in DynamoDB ``` ### `DB-CORE-10224` **Message** ```markdown The BLOB type is supported only for the last column in the partition key in DynamoDB. Column: %s ``` ### `DB-CORE-10225` **Message** ```markdown The BLOB type is not supported for clustering keys in DynamoDB. Column: %s ``` ### `DB-CORE-10226` **Message** ```markdown The BOOLEAN type is not supported for index columns in DynamoDB. Column: %s ``` ### `DB-CORE-10227` **Message** ```markdown The TIMESTAMP type is not supported in Cassandra. Column: %s ``` ### `DB-CORE-10228` **Message** ```markdown With Db2, using a BLOB column as partition key, clustering key or secondary index is not supported. ``` ### `DB-CORE-10229` **Message** ```markdown With Db2, setting an ordering on a BLOB column when using a cross partition scan operation is not supported. Ordering: %s ``` ### `DB-CORE-10230` **Message** ```markdown Renaming tables is not supported in Cassandra ``` ### `DB-CORE-10231` **Message** ```markdown Renaming tables is not supported in Cosmos DB ``` ### `DB-CORE-10232` **Message** ```markdown Renaming tables is not supported in DynamoDB ``` ### `DB-CORE-10233` **Message** ```markdown Altering primary key or index key column types is not supported. Table: %s; Column: %s ``` ### `DB-CORE-10234` **Message** ```markdown Invalid column type conversion from %s to %s. Column: %s ``` ### `DB-CORE-10235` **Message** ```markdown Cassandra does not support the feature for altering column types ``` ### `DB-CORE-10236` **Message** ```markdown Cosmos DB does not support the feature for altering column types ``` ### `DB-CORE-10237` **Message** ```markdown DynamoDB does not support the feature for altering column types ``` ### `DB-CORE-10238` **Message** ```markdown SQLite does not support the feature for altering column types ``` ### `DB-CORE-10239` **Message** ```markdown Oracle does not support column type conversion from %s to %s ``` ### `DB-CORE-10240` **Message** ```markdown Db2 does not support column type conversion from %s to %s ``` ### `DB-CORE-10241` **Message** ```markdown The operations are empty ``` ### `DB-CORE-10242` **Message** ```markdown Multiple operations are not supported in single CRUD operation transactions ``` ### `DB-CORE-10243` **Message** ```markdown This batch result doesn't have a get result ``` ### `DB-CORE-10244` **Message** ```markdown This batch result doesn't have a scan result ``` ### `DB-CORE-10245` **Message** ```markdown TiDB does not support column type conversion from %s to %s ``` ### `DB-CORE-10246` **Message** ```markdown With Oracle, using a BLOB column as partition key, clustering key or secondary index is not supported. ``` ### `DB-CORE-10247` **Message** ```markdown With Oracle, setting an ordering on a BLOB column when using a cross partition scan operation is not supported. Ordering: %s ``` ### `DB-CORE-10248` **Message** ```markdown With Oracle, setting a condition on a BLOB column when using a selection operation is not supported. Condition: %s ``` ### `DB-CORE-10249` **Message** ```markdown The namespace has non-ScalarDB tables and cannot be dropped. Namespace: %s; Tables in the namespace: %s ``` ### `DB-CORE-10250` **Message** ```markdown Import-related functionality is not supported in Object Storage ``` ### `DB-CORE-10251` **Message** ```markdown Index-related functionality is not supported in Object Storage ``` ### `DB-CORE-10252` **Message** ```markdown Object Storage does not support the feature for dropping columns ``` ### `DB-CORE-10253` **Message** ```markdown Object Storage does not support the feature for renaming columns ``` ### `DB-CORE-10254` **Message** ```markdown Renaming tables is not supported in Object Storage ``` ### `DB-CORE-10255` **Message** ```markdown Object Storage does not support the feature for altering column types ``` ### `DB-CORE-10256` **Message** ```markdown Cross-partition scan with ordering is not supported in Object Storage ``` ### `DB-CORE-10257` **Message** ```markdown The value of the column %s in the primary key contains an illegal character. Value: %s ``` ### `DB-CORE-10258` **Message** ```markdown Specifying transaction metadata columns in the projection is not allowed. Table: %s; Column: %s ``` ### `DB-CORE-10259` **Message** ```markdown Specifying transaction metadata columns in the ordering is not allowed. Table: %s; Column: %s ``` ### `DB-CORE-10260` **Message** ```markdown Get operations by using an index is not allowed in the SERIALIZABLE isolation level ``` ### `DB-CORE-10261` **Message** ```markdown Scan operations by using an index is not allowed in the SERIALIZABLE isolation level ``` ### `DB-CORE-10262` **Message** ```markdown Conditions on indexed columns in cross-partition scan operations are not allowed in the SERIALIZABLE isolation level ``` ### `DB-CORE-10263` **Message** ```markdown The service account key for Cloud Storage was not found. ``` ### `DB-CORE-10264` **Message** ```markdown Failed to load the service account key for Cloud Storage. ``` ### `DB-CORE-10265` **Message** ```markdown To support virtual tables, the atomicity unit of the storage must be at least at the namespace level. Storage: %s; Atomicity unit: %s ``` ### `DB-CORE-10266` **Message** ```markdown The source tables must reside within the atomicity unit of the storage. Storage: %s; Atomicity unit: %s; Left source table: %s; Right source table: %s ``` ### `DB-CORE-10267` **Message** ```markdown The virtual table functionality is not supported in DynamoDB ``` ### `DB-CORE-10268` **Message** ```markdown The source tables must have the same primary key. Left source table: %s; Right source table: %s ``` ### `DB-CORE-10269` **Message** ```markdown The source tables must have the same data types for primary key column. Column: %s; Left source table: %s; Right source table: %s ``` ### `DB-CORE-10270` **Message** ```markdown The source tables must have the same clustering orders for clustering key column. Column: %s; Left source table: %s; Right source table: %s ``` ### `DB-CORE-10271` **Message** ```markdown The source tables have conflicting non-key column names. Left source table: %s; Right source table: %s; Conflicting columns: %s ``` ### `DB-CORE-10272` **Message** ```markdown Virtual tables cannot be used as source tables. Source table: %s ``` ### `DB-CORE-10273` **Message** ```markdown The source tables must be in the same storage. Left source table: %s; Right source table: %s ``` ### `DB-CORE-10274` **Message** ```markdown The virtual table must be in the same storage as its source tables. Virtual table: %s; Left source table: %s; Right source table: %s ``` ### `DB-CORE-10275` **Message** ```markdown Source tables cannot be dropped while virtual tables depending on them exist. Source table: %s; Virtual tables: %s ``` ### `DB-CORE-10276` **Message** ```markdown The DeleteIf IS_NULL condition for right source table columns is not allowed in LEFT_OUTER virtual tables. Virtual table: %s ``` ### `DB-CORE-10277` **Message** ```markdown The transaction metadata decoupling feature is not supported in the storage. Storage %s ``` ### `DB-CORE-10278` **Message** ```markdown The storage does not guarantee consistent reads for virtual tables. Depending on the storage configuration, you may be able to adjust the settings to enable consistent reads. Please refer to the storage configuration for details. Storage: %s ``` ## `DB-CORE-2xxxx` status codes The following are status codes and messages for the concurrency error category. ### `DB-CORE-20000` **Message** ```markdown No mutation was applied ``` ### `DB-CORE-20001` **Message** ```markdown Logging failed in the batch ``` ### `DB-CORE-20002` **Message** ```markdown The operation failed in the batch with type %s ``` ### `DB-CORE-20003` **Message** ```markdown An error occurred in the batch. Details: %s ``` ### `DB-CORE-20004` **Message** ```markdown A Paxos phase in the CAS operation failed ``` ### `DB-CORE-20005` **Message** ```markdown The learn phase in the CAS operation failed ``` ### `DB-CORE-20006` **Message** ```markdown A simple write operation failed ``` ### `DB-CORE-20007` **Message** ```markdown An error occurred in the mutation. Details: %s ``` ### `DB-CORE-20008` **Message** ```markdown A RetryWith error occurred in the mutation. Details: %s ``` ### `DB-CORE-20009` **Message** ```markdown A transaction conflict occurred in the mutation. Details: %s ``` ### `DB-CORE-20010` **Message** ```markdown A transaction conflict occurred in the mutation. Details: %s ``` ### `DB-CORE-20011` **Message** ```markdown A conflict occurred. Please try restarting the transaction. Details: %s ``` ### `DB-CORE-20012` **Message** ```markdown The %s condition of the %s operation is not satisfied. Targeting column(s): %s ``` ### `DB-CORE-20013` **Message** ```markdown The record being prepared already exists. Details: %s ``` ### `DB-CORE-20014` **Message** ```markdown A conflict occurred when preparing records. Details: %s ``` ### `DB-CORE-20015` **Message** ```markdown The committing state in the coordinator failed. The transaction has been aborted. Details: %s ``` ### `DB-CORE-20016` **Message** ```markdown A conflict occurred during implicit pre-read. Details: %s ``` ### `DB-CORE-20017` **Message** ```markdown This record needs to be recovered. Table: %s; Partition Key: %s; Clustering Key: %s; Transaction ID that wrote the record: %s ``` ### `DB-CORE-20018` **Message** ```markdown The record does not exist, so the %s condition is not satisfied ``` ### `DB-CORE-20019` **Message** ```markdown The record exists, so the %s condition is not satisfied ``` ### `DB-CORE-20020` **Message** ```markdown The condition on the column '%s' is not satisfied ``` ### `DB-CORE-20022` **Message** ```markdown An anti-dependency was found. The transaction has been aborted ``` ### `DB-CORE-20023` **Message** ```markdown A transaction conflict occurred in the Insert operation ``` ### `DB-CORE-20024` **Message** ```markdown The %s condition of the %s operation is not satisfied. Targeting column(s): %s ``` ### `DB-CORE-20025` **Message** ```markdown A transaction conflict occurred in the Insert operation ``` ### `DB-CORE-20026` **Message** ```markdown A conflict occurred when committing records. Details: %s ``` ### `DB-CORE-20027` **Message** ```markdown A transaction conflict occurred in the mutation. Details: %s ``` ## `DB-CORE-3xxxx` status codes The following are status codes and messages for the internal error category. ### `DB-CORE-30000` **Message** ```markdown Creating the namespace failed. Namespace: %s ``` ### `DB-CORE-30001` **Message** ```markdown Dropping the namespace failed. Namespace: %s ``` ### `DB-CORE-30002` **Message** ```markdown Creating the table failed. Table: %s ``` ### `DB-CORE-30003` **Message** ```markdown Dropping the table failed. Table: %s ``` ### `DB-CORE-30004` **Message** ```markdown Truncating the table failed. Table: %s ``` ### `DB-CORE-30005` **Message** ```markdown Creating the index failed. Table: %s, Column: %s ``` ### `DB-CORE-30006` **Message** ```markdown Dropping the index failed. Table: %s, Column: %s ``` ### `DB-CORE-30007` **Message** ```markdown Getting the table metadata failed. Table: %s ``` ### `DB-CORE-30008` **Message** ```markdown Getting the table names in the namespace failed. Namespace: %s ``` ### `DB-CORE-30009` **Message** ```markdown Checking the namespace existence failed. Namespace: %s ``` ### `DB-CORE-30010` **Message** ```markdown Checking the table existence failed. Table: %s ``` ### `DB-CORE-30011` **Message** ```markdown Checking the index existence failed. Table: %s; Column: %s ``` ### `DB-CORE-30012` **Message** ```markdown Repairing the namespace failed. Namespace: %s ``` ### `DB-CORE-30013` **Message** ```markdown Repairing the table failed. Table: %s ``` ### `DB-CORE-30014` **Message** ```markdown Adding a new column to the table failed. Table: %s; Column: %s; ColumnType: %s ``` ### `DB-CORE-30015` **Message** ```markdown Getting the namespace names failed ``` ### `DB-CORE-30016` **Message** ```markdown Getting the table metadata of the table being imported failed. Table: %s ``` ### `DB-CORE-30017` **Message** ```markdown Importing the table failed. Table: %s ``` ### `DB-CORE-30018` **Message** ```markdown Adding the raw column to the table failed. Table: %s; Column: %s; ColumnType: %s ``` ### `DB-CORE-30019` **Message** ```markdown Upgrading the ScalarDB environment failed ``` ### `DB-CORE-30020` **Message** ```markdown Something wrong because WriteType is neither CAS nor SIMPLE ``` ### `DB-CORE-30021` **Message** ```markdown An error occurred in the selection. Details: %s ``` ### `DB-CORE-30022` **Message** ```markdown An error occurred in the mutation. Details: %s ``` ### `DB-CORE-30023` **Message** ```markdown An error occurred in the selection. Details: %s ``` ### `DB-CORE-30024` **Message** ```markdown An error occurred in the mutation. Details: %s ``` ### `DB-CORE-30025` **Message** ```markdown An error occurred in the selection. Details: %s ``` ### `DB-CORE-30026` **Message** ```markdown An error occurred in the mutation. Details: %s ``` ### `DB-CORE-30027` **Message** ```markdown An error occurred in the selection. Details: %s ``` ### `DB-CORE-30028` **Message** ```markdown Fetching the next result failed. Details: %s ``` ### `DB-CORE-30029` **Message** ```markdown Rolling back the transaction failed. Details: %s ``` ### `DB-CORE-30030` **Message** ```markdown Committing the transaction failed. Details: %s ``` ### `DB-CORE-30031` **Message** ```markdown The Get operation failed. Details: %s ``` ### `DB-CORE-30032` **Message** ```markdown The Scan operation failed. Details: %s ``` ### `DB-CORE-30033` **Message** ```markdown The Put operation failed. Details: %s ``` ### `DB-CORE-30034` **Message** ```markdown The Delete operation failed. Details: %s ``` ### `DB-CORE-30035` **Message** ```markdown Beginning a transaction failed. Details: %s ``` ### `DB-CORE-30036` **Message** ```markdown Preparing records failed. Details: %s ``` ### `DB-CORE-30037` **Message** ```markdown Validation failed. Details: %s ``` ### `DB-CORE-30038` **Message** ```markdown Executing implicit pre-read failed. Details: %s ``` ### `DB-CORE-30039` **Message** ```markdown Reading a record from the underlying storage failed. Details: %s ``` ### `DB-CORE-30040` **Message** ```markdown Scanning records from the underlying storage failed. Details: %s ``` ### `DB-CORE-30041` **Message** ```markdown Rollback failed because the transaction has already been committed ``` ### `DB-CORE-30042` **Message** ```markdown Rollback failed ``` ### `DB-CORE-30043` **Message** ```markdown The Insert operation failed. Details: %s ``` ### `DB-CORE-30044` **Message** ```markdown The Upsert operation failed. Details: %s ``` ### `DB-CORE-30045` **Message** ```markdown The Update operation failed. Details: %s ``` ### `DB-CORE-30046` **Message** ```markdown Handling the before-preparation hook failed. Details: %s ``` ### `DB-CORE-30054` **Message** ```markdown Getting the scanner failed. Details: %s ``` ### `DB-CORE-30055` **Message** ```markdown Closing the scanner failed. Details: %s ``` ### `DB-CORE-30056` **Message** ```markdown Getting the storage information failed. Namespace: %s ``` ### `DB-CORE-30057` **Message** ```markdown Recovering records failed. Details: %s ``` ### `DB-CORE-30058` **Message** ```markdown Committing records failed. Details: %s ``` ### `DB-CORE-30059` **Message** ```markdown Dropping a column from the table failed. Table: %s; Column: %s ``` ### `DB-CORE-30060` **Message** ```markdown Renaming a column failed. Table: %s; Old column name: %s; New column name: %s ``` ### `DB-CORE-30061` **Message** ```markdown Renaming a table failed. Old table name: %s; New table name: %s ``` ### `DB-CORE-30062` **Message** ```markdown Altering a column type failed. Table: %s; Column: %s; New column type: %s ``` ### `DB-CORE-30063` **Message** ```markdown Getting the MySQL JDBC connection metadata failed. Details: %s ``` ### `DB-CORE-30064` **Message** ```markdown An error occurred in the selection. Details: %s ``` ### `DB-CORE-30065` **Message** ```markdown An error occurred in the mutation. Details: %s ``` ### `DB-CORE-30066` **Message** ```markdown Creating the virtual table failed. Virtual table: %s; Left source table: %s; Right source table: %s ``` ### `DB-CORE-30067` **Message** ```markdown Getting the virtual table information failed. Table: %s ``` ## `DB-CORE-4xxxx` status codes The following are status codes and messages for the unknown transaction status error category. ### `DB-CORE-40000` **Message** ```markdown Rolling back the transaction failed. Details: %s ``` ### `DB-CORE-40001` **Message** ```markdown Committing state failed with NoMutationException, but the coordinator status does not exist. Details: %s ``` ### `DB-CORE-40002` **Message** ```markdown The coordinator status cannot be retrieved. Details: %s ``` ### `DB-CORE-40003` **Message** ```markdown The coordinator status is unknown. Details: %s ``` ### `DB-CORE-40004` **Message** ```markdown Aborting state failed with NoMutationException, but the coordinator status does not exist. Details: %s ``` ### `DB-CORE-40005` **Message** ```markdown One-phase committing records failed. Details: %s ``` ================================================ FILE: docs/scalardb-data-loader-status-codes.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Data Loader Error Codes This page provides a list of error codes in ScalarDB Data Loader. ## Error code classes and descriptions | Class | Description | |:-----------------------|:---------------------------------------| | `DB-DATA-LOADER-1xxxx` | Errors for the user error category | | `DB-DATA-LOADER-3xxxx` | Errors for the internal error category | ## `DB-DATA-LOADER-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-DATA-LOADER-10000` **Message** ```markdown Data chunk queue size must be greater than 0 ``` ### `DB-DATA-LOADER-10001` **Message** ```markdown The directory '%s' does not have write permissions. Please ensure that the current user has write access to the directory ``` ### `DB-DATA-LOADER-10002` **Message** ```markdown Failed to create the directory '%s'. Please check if you have sufficient permissions and if there are any file system restrictions. Details: %s ``` ### `DB-DATA-LOADER-10003` **Message** ```markdown Directory path cannot be null or empty ``` ### `DB-DATA-LOADER-10004` **Message** ```markdown No file extension was found in the provided file name %s ``` ### `DB-DATA-LOADER-10005` **Message** ```markdown Invalid file extension: %s. Allowed extensions are: %s ``` ### `DB-DATA-LOADER-10006` **Message** ```markdown Invalid key: Column %s does not exist in the table %s in namespace %s ``` ### `DB-DATA-LOADER-10007` **Message** ```markdown Invalid base64 encoding for blob value '%s' for column %s in table %s in namespace %s ``` ### `DB-DATA-LOADER-10008` **Message** ```markdown Invalid number '%s' specified for column %s in table %s in namespace %s ``` ### `DB-DATA-LOADER-10009` **Message** ```markdown Method null argument not allowed ``` ### `DB-DATA-LOADER-10010` **Message** ```markdown The provided clustering key %s was not found ``` ### `DB-DATA-LOADER-10011` **Message** ```markdown The column '%s' was not found ``` ### `DB-DATA-LOADER-10012` **Message** ```markdown The provided partition key is incomplete. Required key: %s ``` ### `DB-DATA-LOADER-10013` **Message** ```markdown The provided clustering-key order does not match the table schema. Required order: %s ``` ### `DB-DATA-LOADER-10014` **Message** ```markdown The provided partition-key order does not match the table schema. Required order: %s ``` ### `DB-DATA-LOADER-10015` **Message** ```markdown Missing namespace or table: %s, %s ``` ### `DB-DATA-LOADER-10016` **Message** ```markdown Failed to retrieve table metadata. Details: %s ``` ### `DB-DATA-LOADER-10017` **Message** ```markdown Duplicate data mappings found for table '%s' in the control file ``` ### `DB-DATA-LOADER-10018` **Message** ```markdown No mapping found for column '%s' in table '%s' in the control file. Control file validation set at 'FULL'. All columns need to be mapped ``` ### `DB-DATA-LOADER-10019` **Message** ```markdown The control file is missing data mappings ``` ### `DB-DATA-LOADER-10020` **Message** ```markdown The target column '%s' for source field '%s' could not be found in table '%s' ``` ### `DB-DATA-LOADER-10021` **Message** ```markdown The required partition key '%s' is missing in the control file mapping for table '%s' ``` ### `DB-DATA-LOADER-10022` **Message** ```markdown The required clustering key '%s' is missing in the control file mapping for table '%s' ``` ### `DB-DATA-LOADER-10023` **Message** ```markdown Duplicated data mappings found for column '%s' in table '%s' ``` ### `DB-DATA-LOADER-10024` **Message** ```markdown Missing required field or column mapping for clustering key %s ``` ### `DB-DATA-LOADER-10025` **Message** ```markdown Missing required field or column mapping for partition key %s ``` ### `DB-DATA-LOADER-10026` **Message** ```markdown Missing field or column mapping for %s ``` ### `DB-DATA-LOADER-10027` **Message** ```markdown Something went wrong while converting the ScalarDB values to strings. The table metadata and Value datatype probably do not match. Details: %s ``` ### `DB-DATA-LOADER-10028` **Message** ```markdown The provided file format is not supported : %s ``` ### `DB-DATA-LOADER-10029` **Message** ```markdown Could not find the partition key ``` ### `DB-DATA-LOADER-10030` **Message** ```markdown The source record needs to contain all fields if the UPSERT turns into an INSERT ``` ### `DB-DATA-LOADER-10031` **Message** ```markdown Record already exists ``` ### `DB-DATA-LOADER-10032` **Message** ```markdown Record was not found ``` ### `DB-DATA-LOADER-10033` **Message** ```markdown Could not find the clustering key ``` ### `DB-DATA-LOADER-10034` **Message** ```markdown No table metadata found ``` ### `DB-DATA-LOADER-10035` **Message** ```markdown The data mapping source field '%s' for table '%s' is missing in the JSON data record ``` ### `DB-DATA-LOADER-10036` **Message** ```markdown The CSV row: %s does not match header: %s ``` ### `DB-DATA-LOADER-10037` **Message** ```markdown Expected JSON file content to be an array ``` ### `DB-DATA-LOADER-10038` **Message** ```markdown Missing option: either the '--namespace' and '--table' options or the '--control-file' option must be specified ``` ### `DB-DATA-LOADER-10039` **Message** ```markdown The file '%s' specified by the argument '%s' does not exist ``` ### `DB-DATA-LOADER-10040` **Message** ```markdown Cannot write to the log directory: %s ``` ### `DB-DATA-LOADER-10041` **Message** ```markdown Failed to create the log directory: %s ``` ### `DB-DATA-LOADER-10042` **Message** ```markdown Failed to parse the control file: %s ``` ### `DB-DATA-LOADER-10043` **Message** ```markdown No permission to create or write files in the directory: %s ``` ### `DB-DATA-LOADER-10044` **Message** ```markdown Failed to create the directory: %s ``` ### `DB-DATA-LOADER-10045` **Message** ```markdown Path exists but is not a directory: %s ``` ### `DB-DATA-LOADER-10046` **Message** ```markdown File path must not be blank ``` ### `DB-DATA-LOADER-10047` **Message** ```markdown File not found: %s ``` ### `DB-DATA-LOADER-10048` **Message** ```markdown Invalid date time value '%s' specified for column %s in table %s in namespace %s ``` ### `DB-DATA-LOADER-10049` **Message** ```markdown Key value cannot be null or empty ``` ### `DB-DATA-LOADER-10050` **Message** ```markdown Invalid key-value format: %s ``` ### `DB-DATA-LOADER-10051` **Message** ```markdown Value must not be null ``` ### `DB-DATA-LOADER-10052` **Message** ```markdown Delimiter must not be null ``` ### `DB-DATA-LOADER-10053` **Message** ```markdown Config file path must not be blank ``` ### `DB-DATA-LOADER-10054` **Message** ```markdown Data chunk size must be greater than 0 ``` ### `DB-DATA-LOADER-10055` **Message** ```markdown Transaction size must be greater than 0 ``` ### `DB-DATA-LOADER-10056` **Message** ```markdown Number of max threads must be greater than 0 ``` ### `DB-DATA-LOADER-10057` **Message** ```markdown Cannot specify both deprecated option '%s' and new option '%s'. Please use only '%s' ``` ### `DB-DATA-LOADER-10058` **Message** ```markdown TRANSACTION mode is not compatible with the current configuration. Please try with STORAGE mode or check your ScalarDB configuration. Details: %s ``` ### `DB-DATA-LOADER-10059` **Message** ```markdown Failed to validate TRANSACTION mode. Details: %s ``` ## `DB-DATA-LOADER-3xxxx` status codes The following are status codes and messages for the internal error category. ### `DB-DATA-LOADER-30000` **Message** ```markdown A problem occurred while trying to save the data. Details: %s ``` ### `DB-DATA-LOADER-30001` **Message** ```markdown A problem occurred while scanning. Are you sure you are running in the correct transaction mode? Details: %s ``` ### `DB-DATA-LOADER-30002` **Message** ```markdown Failed to read CSV file. Details: %s ``` ### `DB-DATA-LOADER-30003` **Message** ```markdown Failed to CSV read header line. Details: %s ``` ### `DB-DATA-LOADER-30004` **Message** ```markdown Data chunk processing was interrupted. Details: %s ``` ### `DB-DATA-LOADER-30005` **Message** ```markdown Failed to read JSON file. Details: %s ``` ### `DB-DATA-LOADER-30006` **Message** ```markdown Failed to read JSON Lines file. Details: %s ``` ================================================ FILE: docs/scalardb-schema-loader-status-codes.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Schema Loader Error Codes This page provides a list of error codes in ScalarDB Schema Loader. ## Error code classes and descriptions | Class | Description | |:-------------------------|:-----------------------------------| | `DB-SCHEMA-LOADER-1xxxx` | Errors for the user error category | ## `DB-SCHEMA-LOADER-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-SCHEMA-LOADER-10000` **Message** ```markdown The table does not exist. Table: %s ``` ### `DB-SCHEMA-LOADER-10001` **Message** ```markdown The partition keys for the table %s.%s were modified, but altering partition keys is not supported ``` ### `DB-SCHEMA-LOADER-10002` **Message** ```markdown The clustering keys for the table %s.%s were modified, but altering clustering keys is not supported ``` ### `DB-SCHEMA-LOADER-10003` **Message** ```markdown The clustering order of the table %s.%s were modified, but altering the clustering order is not supported ``` ### `DB-SCHEMA-LOADER-10004` **Message** ```markdown The column %s in the table %s.%s has been deleted. Column deletion is not supported when altering a table ``` ### `DB-SCHEMA-LOADER-10005` **Message** ```markdown The data type for the column %s in the table %s.%s was modified, but altering data types is not supported ``` ### `DB-SCHEMA-LOADER-10006` **Message** ```markdown Specifying the '--schema-file' option is required when using the '--repair-all' option ``` ### `DB-SCHEMA-LOADER-10007` **Message** ```markdown Specifying the '--schema-file' option is required when using the '--alter' option ``` ### `DB-SCHEMA-LOADER-10008` **Message** ```markdown Specifying the '--schema-file' option is required when using the '--import' option ``` ### `DB-SCHEMA-LOADER-10009` **Message** ```markdown Specifying the '--coordinator' option with the '--import' option is not allowed. Create Coordinator tables separately ``` ### `DB-SCHEMA-LOADER-10010` **Message** ```markdown Reading the configuration file failed. File: %s ``` ### `DB-SCHEMA-LOADER-10011` **Message** ```markdown Reading the schema file failed. File: %s ``` ### `DB-SCHEMA-LOADER-10012` **Message** ```markdown Parsing the schema JSON failed. Details: %s ``` ### `DB-SCHEMA-LOADER-10013` **Message** ```markdown The table name must contain the namespace and the table. Table: %s ``` ### `DB-SCHEMA-LOADER-10014` **Message** ```markdown The partition key must be specified. Table: %s ``` ### `DB-SCHEMA-LOADER-10015` **Message** ```markdown Invalid clustering-key format. The clustering key must be in the format of 'column_name' or 'column_name ASC/DESC'. Table: %s; Clustering key: %s ``` ### `DB-SCHEMA-LOADER-10016` **Message** ```markdown Columns must be specified. Table: %s ``` ### `DB-SCHEMA-LOADER-10017` **Message** ```markdown Invalid column type. Table: %s; Column: %s; Type: %s ``` ================================================ FILE: docs/schema-loader-import.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; You might want to use ScalarDB (e.g., for database-spanning transactions) with your existing databases. In that case, you can import those databases under the ScalarDB control using ScalarDB Schema Loader. ScalarDB Schema Loader automatically adds ScalarDB-internal metadata columns in each existing table and metadata tables to enable various ScalarDB functionalities including transaction management across multiple databases. ## Before you begin :::warning You should carefully plan to import a table to ScalarDB in production because it will add transaction metadata columns to your database tables and the ScalarDB metadata tables. In this case, there would also be several differences between your database and ScalarDB, as well as some limitations. ::: ### What will be added to your databases - **ScalarDB metadata tables:** ScalarDB manages namespace names and table metadata in a namespace (schema or database in underlying databases) called 'scalardb'. - **Transaction metadata columns:** The Consensus Commit transaction manager requires metadata (for example, transaction ID, record version, and transaction status) stored along with the actual records to handle transactions properly. Thus, this tool adds the metadata columns if you use the Consensus Commit transaction manager. :::note This tool only changes database metadata. Thus, the processing time does not increase in proportion to the database size and usually takes only several seconds. ::: ### Requirements - [JDBC databases](./requirements.mdx#relational-databases), except for SQLite, can be imported. - Each table must have primary key columns. (Composite primary keys can be available.) - Target tables must only have columns with supported data types. For details, see [Data-type mapping from JDBC databases to ScalarDB](#data-type-mapping-from-jdbc-databases-to-scalardb). - ScalarDB assumes that the same underlying database user account is used for all administrative and CRUD operations. Therefore, if the table owner is different from the user account used for ScalarDB, you will likely need additional permissions beyond those mentioned in [Database permission requirements](./requirements.mdx#database-permission-requirements). These requirements are based on the assumption that the user account used by ScalarDB is also the table owner. ### Set up Schema Loader To set up Schema Loader for importing existing tables, see [Set up Schema Loader](./schema-loader.mdx#set-up-schema-loader). ## Run Schema Loader for importing existing tables You can import an existing table in JDBC databases to ScalarDB by using the `--import` option and an import-specific schema file. To import tables, run the following command, replacing the contents in the angle brackets as described: ```console java -jar scalardb-schema-loader-.jar --config -f --import ``` - ``: Version of ScalarDB Schema Loader that you set up. - ``: Path to a properties file for ScalarDB. For a sample properties file, see [`database.properties`](https://github.com/scalar-labs/scalardb/blob/master/conf/database.properties). - ``: Path to an import schema file. For a sample, see [Sample import schema file](#sample-import-schema-file). If you use the Consensus Commit transaction manager after importing existing tables, run the following command separately, replacing the contents in the angle brackets as described: ```console java -jar scalardb-schema-loader-.jar --config --coordinator ``` ## Sample import schema file The following is a sample schema for importing tables. For the sample schema file, see [`import_schema_sample.json`](https://github.com/scalar-labs/scalardb/blob/master/schema-loader/sample/import_schema_sample.json). ```json { "sample_namespace1.sample_table1": { "transaction": true, "override-columns-type": { "c3": "TIME", "c5": "TIMESTAMP" } }, "sample_namespace1.sample_table2": { "transaction": true }, "sample_namespace2.sample_table3": { "transaction": false } } ``` The import table schema consists of a namespace name, a table name, a `transaction` field, and an optional `override-columns-type` field: - The `transaction` field indicates whether or not the table will be imported for transactions. If you set the `transaction` field to `true` or don't specify the `transaction` field, this tool will create a table with transaction metadata, if needed. If you set the `transaction` field to `false`, this tool will import a table without adding transaction metadata (that is, for a table using the [Storage API](run-non-transactional-storage-operations-through-primitive-crud-interface.mdx)). - The `override-columns-type` field indicates the columns for which you wish to override the default data-type mapping. This field is optional and only needs to be set with the columns requiring a type override. ## Data-type mapping from JDBC databases to ScalarDB The following table shows the supported data types in each JDBC database and their mapping to the ScalarDB data types. Select your database and check if your existing tables can be imported. | MySQL/MariaDB/TiDB | ScalarDB | Notes | |--------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------| | bigint | BIGINT | | | binary | BLOB | | | bit | BOOLEAN | | | blob | BLOB | See warning [1](#1) below. | | char | TEXT | See warning [1](#1) below. | | date | DATE | | | datetime | TIMESTAMP (default) and TIMESTAMPTZ | When importing as TIMESTAMPTZ, ScalarDB will assume the data to be on the UTC time zone. See warning [5](#5) below. | | double | DOUBLE | | | float | FLOAT | | | int | INT | | | int unsigned | BIGINT | See warning [1](#1) below. | | integer | INT | | | longblob | BLOB | | | longtext | TEXT | | | mediumblob | BLOB | See warning [1](#1) below. | | mediumint | INT | See warning [1](#1) below. | | mediumtext | TEXT | See warning [1](#1) below. | | smallint | INT | See warning [1](#1) below. | | text | TEXT | See warning [1](#1) below. | | time | TIME | | | timestamp | TIMESTAMPTZ | | | tinyblob | BLOB | See warning [1](#1) below. | | tinyint | INT | See warning [1](#1) below. | | tinyint(1) | BOOLEAN | | | tinytext | TEXT | See warning [1](#1) below. | | varbinary | BLOB | See warning [1](#1) below. | | varchar | TEXT | See warning [1](#1) below. | Data types not listed above are not supported. The following are some common data types that are not supported: - bigint unsigned - bit(n) (n > 1) - decimal - enum - geometry - json - numeric - set - year | PostgreSQL/YugabyteDB/AlloyDB | ScalarDB | Notes | |--------------------------|-------------|----------------------------| | bigint | BIGINT | | | boolean | BOOLEAN | | | bytea | BLOB | | | character | TEXT | See warning [1](#1) below. | | character varying | TEXT | See warning [1](#1) below. | | date | DATE | | | double precision | DOUBLE | | | integer | INT | | | real | FLOAT | | | smallint | INT | See warning [1](#1) below. | | text | TEXT | | | time | TIME | | | timestamp | TIMESTAMP | | | timestamp with time zone | TIMESTAMPTZ | | Data types not listed above are not supported. The following are some common data types that are not supported: - bigserial - bit - box - cidr - circle - inet - interval - json - jsonb - line - lseg - macaddr - macaddr8 - money - numeric - path - pg_lsn - pg_snapshot - point - polygon - serial - smallserial - time with time zone - tsquery - tsvector - txid_snapshot - uuid - xml | Oracle | ScalarDB | Notes | |--------------------------------|-------------------------------------|----------------------------| | binary_double | DOUBLE | | | binary_float | FLOAT | | | blob | BLOB | See warning [2](#2) below. | | char | TEXT | See warning [1](#1) below. | | clob | TEXT | | | date | DATE (default), TIME, and TIMESTAMP | See warning [5](#5) below. | | float | DOUBLE | See warning [3](#3) below. | | long | TEXT | | | long raw | BLOB | | | nchar | TEXT | See warning [1](#1) below. | | nclob | TEXT | | | number(p,s), with p ≠ 1 | BIGINT / DOUBLE | See warning [4](#4) below. | | number(1,0) | BIGINT (default), BOOLEAN | See warning [5](#5) below. | | nvarchar2 | TEXT | See warning [1](#1) below. | | raw | BLOB | See warning [1](#1) below. | | timestamp | TIMESTAMP (default) and TIME | See warning [5](#5) below. | | timestamp with time zone | TIMESTAMPTZ | | | timestamp with local time zone | TIMESTAMPTZ | | | varchar2 | TEXT | See warning [1](#1) below. | Data types not listed above are not supported. The following are some common data types that are not supported: - interval - rowid - urowid - bfile - json | SQL Server | ScalarDB | Notes | |----------------|-------------|----------------------------| | bigint | BIGINT | | | binary | BLOB | See warning [1](#1) below. | | bit | BOOLEAN | | | char | TEXT | See warning [1](#1) below. | | date | DATE | | | datetime | TIMESTAMP | | datetime2 | TIMESTAMP | | | float | DOUBLE | | | image | BLOB | See warning [6](#6) below. | | int | INT | | | nchar | TEXT | See warning [1](#1) below. | | ntext | TEXT | | | nvarchar | TEXT | See warning [1](#1) below. | | offsetdatetime | TIMESTAMPTZ | | | real | FLOAT | | | smalldatetime | TIMESTAMP | | | smallint | INT | See warning [1](#1) below. | | text | TEXT | | | time | TIME | | | tinyint | INT | See warning [1](#1) below. | | varbinary | BLOB | See warning [1](#1) below. | | varchar | TEXT | See warning [1](#1) below. | Data types not listed above are not supported. The following are some common data types that are not supported: - cursor - decimal - geography - geometry - hierarchyid - money - numeric - rowversion - smallmoney - sql_variant - uniqueidentifier - xml | Db2 | ScalarDB | Notes | |-----------------------|----------------------------------------|----------------------------| | BIGINT | BIGINT | | | BINARY | BLOB | | | BLOB | BLOB | | | BOOLEAN | BOOLEAN | | | CHAR | TEXT | | | CHAR FOR BIT DATA | BLOB | | | CLOB | TEXT | | | DATE | DATE | | | DOUBLE | DOUBLE | See warning [1](#1) below. | | FLOAT(p), with p ≤ 24 | FLOAT | See warning [1](#1) below. | | FLOAT(p), with p ≥ 25 | DOUBLE | See warning [1](#1) below. | | GRAPHIC | TEXT | | | INT | INT | | | NCHAR | TEXT | | | NCLOB | TEXT | | | NVARCHAR | TEXT | | | REAL | FLOAT | See warning [1](#1) below. | | SMALLINT | INT | | | TIME | TIME | | | TIMESTAMP | TIMESTAMP (default), TIME, TIMESTAMPTZ | See warning [5](#5) below. | | VARBINARY | BLOB | | | VARCHAR | TEXT | | | VARCHAR FOR BIT DATA | BLOB | | | VARGRAPHIC | TEXT | | Data types not listed above are not supported. The following are some common data types that are not supported: - decimal - decfloat - xml | Spanner | ScalarDB | Notes | |--------------------------|----------------------------------------|----------------------------| | bigint | BIGINT | | | boolean | BOOLEAN | | | bytea | BLOB | | | date | DATE | | | double precision | DOUBLE | | | real | FLOAT | | | text | TEXT | | | timestamp with time zone | TIMESTAMPTZ (default), TIME, TIMESTAMP | See warning [6](#6) below. | Data types not listed above are not supported. The following are some common data types that are not supported: - array - decimal - interval - jsonb - serial - uuid :::warning
  1. For certain data types noted above, ScalarDB may map a data type larger than that of the underlying database. In that case, you will see errors when inserting a value larger than the underlying column's limit.
  2. The maximum size of `BLOB` in ScalarDB is about 2GB (precisely 2^31-1 bytes). In contrast, Oracle `blob` can have (4GB-1)*(number of blocks). Thus, if data larger than 2GB exists in the imported table, ScalarDB cannot read it.
  3. ScalarDB does not support Oracle `float` columns that have a higher precision than `DOUBLE` in ScalarDB.
  4. ScalarDB does not support Oracle `numeric(p, s)` columns (`p` is precision and `s` is scale) when `p` is larger than 18 due to the maximum size of the data type in ScalarDB. Note that ScalarDB maps the column to `BIGINT` if `s` is zero; otherwise ScalarDB will map the column to `DOUBLE`. For the latter case, be aware that round-up or round-off can happen in the underlying database since the floating-point value will be cast to a fixed-point value.
  5. The underlying storage type can be mapped to several ScalarDB data types. To override the default mapping, use the `override-columns-type` field in the import schema file. For an example, see [Sample import schema file](#sample-import-schema-file).
  6. ScalarDB does not support altering SQL Server `image` columns imported as ScalarDB `BLOB` columns to change their data types to `TEXT`.
::: ## Decoupling transaction metadata You can separately manage the transaction metadata from application data by enabling [transaction metadata decoupling](./consensus-commit.mdx#transaction-metadata-decoupling). To decouple transaction metadata for an imported table, add the `transaction-metadata-decoupling` field with a value of `true` in the import schema file, as shown in the following example: ```json { "sample_namespace.sample_table": { "transaction-metadata-decoupling": true } } ``` :::note The imported table name is the original table name with the `_scalardb` suffix appended, so you can access it as `_scalardb`. ::: For details about transaction metadata decoupling, see [Transaction metadata decoupling](./schema-loader.mdx#transaction-metadata-decoupling). ## Use import function in your application You can use the import function in your application by using the following interfaces: - [ScalarDB Admin API](./api-guide.mdx#import-a-table) - [ScalarDB Schema Loader API](./schema-loader.mdx#use-schema-loader-in-your-application) ================================================ FILE: docs/schema-loader.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Schema Loader import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. In addition, ScalarDB stores internal metadata, such as transaction IDs, record versions, and transaction statuses, to manage transaction logs and statuses when you use the Consensus Commit transaction manager. Since managing the schema mapping and metadata for transactions can be difficult, you can use ScalarDB Schema Loader, which is a tool to create schemas that doesn't require you to need in-depth knowledge about schema mapping or metadata. You have two options to specify general CLI options in Schema Loader: - Pass the ScalarDB properties file and database-specific or storage-specific options. - Pass database-specific or storage-specific options without the ScalarDB properties file. (Deprecated) :::note This tool supports only basic options to create, delete, repair, or alter a table. If you want to use the advanced features of a database, you must alter your tables with a database-specific tool after creating the tables with this tool. ::: ## Set up Schema Loader Select your preferred method to set up Schema Loader, and follow the instructions. You can download the release versions of Schema Loader from the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page. You can pull the Docker image from the [Scalar container registry](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-schema-loader) by running the following command, replacing the contents in the angle brackets as described: ```console docker run --rm -v :/scalardb.properties -v :/schema.json ghcr.io/scalar-labs/scalardb-schema-loader: --config /scalardb.properties --schema-file /schema.json ``` :::note You can specify the same command arguments even if you use the fat JAR or the container. In the [Available commands](#available-commands) section, the JAR is used, but you can run the commands by using the container in the same way by replacing `java -jar scalardb-schema-loader-.jar` with `docker run --rm -v : [-v :] ghcr.io/scalar-labs/scalardb-schema-loader:`. ::: ## Run Schema Loader This section explains how to run Schema Loader. ### Available commands Select how you would like to configure Schema Loader for your database. The preferred method is to use the properties file since other, database-specific methods are deprecated. The following commands are available when using the properties file: ```console Usage: java -jar scalardb-schema-loader-.jar [-D] [--coordinator] [--no-backup] [--no-scaling] -c= [--compaction-strategy=] [-f=] [--replication-factor=] [--replication-strategy=] [--ru=] Create/Delete schemas in the storage defined in the config file -A, --alter Alter tables : it will add new columns and create/delete secondary index for existing tables. It compares the provided table schema to the existing schema to decide which columns need to be added and which indexes need to be created or deleted -c, --config= Path to the config file of ScalarDB --compaction-strategy= The compaction strategy, must be LCS, STCS or TWCS (supported in Cassandra) --coordinator Create/delete/repair Coordinator tables -D, --delete-all Delete tables -f, --schema-file= -I, --import Import tables : it will import existing non-ScalarDB tables to ScalarDB. Path to the schema json file --no-backup Disable continuous backup (supported in DynamoDB) --no-scaling Disable auto-scaling (supported in DynamoDB, Cosmos DB) --repair-all Repair tables : it repairs the table metadata of existing tables. When using Cosmos DB, it additionally repairs stored procedure attached to each table --replication-factor= The replication factor (supported in Cassandra) --replication-strategy= The replication strategy, must be SimpleStrategy or NetworkTopologyStrategy (supported in Cassandra) --ru= Base resource unit (supported in DynamoDB, Cosmos DB) ``` For a sample properties file, see [`database.properties`](https://github.com/scalar-labs/scalardb/blob/master/conf/database.properties). :::note The following database-specific methods have been deprecated. Please use the [commands for configuring the properties file](#available-commands) instead. ```console Usage: java -jar scalardb-schema-loader-.jar --jdbc [-D] -f= -j= -p= -u= Create/Delete JDBC schemas -A, --alter Alter tables : it will add new columns and create/delete secondary index for existing tables. It compares the provided table schema to the existing schema to decide which columns need to be added and which indexes need to be created or deleted -D, --delete-all Delete tables -f, --schema-file= Path to the schema json file -j, --jdbc-url= JDBC URL -p, --password= JDBC password --repair-all Repair tables : it repairs the table metadata of existing tables -u, --user= JDBC user ``` ```console Usage: java -jar scalardb-schema-loader-.jar --dynamo [-D] [--no-backup] [--no-scaling] [--endpoint-override=] -f= -p= [-r=] --region= -u= Create/Delete DynamoDB schemas -A, --alter Alter tables : it will add new columns and create/delete secondary index for existing tables. It compares the provided table schema to the existing schema to decide which columns need to be added and which indexes need to be created or deleted -D, --delete-all Delete tables --endpoint-override= Endpoint with which the DynamoDB SDK should communicate -f, --schema-file= Path to the schema json file --no-backup Disable continuous backup for DynamoDB --no-scaling Disable auto-scaling for DynamoDB -p, --password= AWS access secret key -r, --ru= Base resource unit --region= AWS region --repair-all Repair tables : it repairs the table metadata of existing tables -u, --user= AWS access key ID ``` ```console Usage: java -jar scalardb-schema-loader-.jar --cosmos [-D] [--no-scaling] -f= -h= -p= [-r=] Create/Delete Cosmos DB schemas -A, --alter Alter tables : it will add new columns and create/delete secondary index for existing tables. It compares the provided table schema to the existing schema to decide which columns need to be added and which indexes need to be created or deleted -D, --delete-all Delete tables -f, --schema-file= Path to the schema json file -h, --host= Cosmos DB account URI --no-scaling Disable auto-scaling for Cosmos DB -p, --password= Cosmos DB key -r, --ru= Base resource unit --repair-all Repair tables : it repairs the table metadata of existing tables and repairs stored procedure attached to each table ``` ```console Usage: java -jar scalardb-schema-loader-.jar --cassandra [-D] [-c=] -f= -h= [-n=] [-p=] [-P=] [-R=] [-u=] Create/Delete Cassandra schemas -A, --alter Alter tables : it will add new columns and create/delete secondary index for existing tables. It compares the provided table schema to the existing schema to decide which columns need to be added and which indexes need to be created or deleted -c, --compaction-strategy= Cassandra compaction strategy, must be LCS, STCS or TWCS -D, --delete-all Delete tables -f, --schema-file= Path to the schema json file -h, --host= Cassandra host IP -n, --network-strategy= Cassandra network strategy, must be SimpleStrategy or NetworkTopologyStrategy -p, --password= Cassandra password -P, --port= Cassandra Port -R, --replication-factor= Cassandra replication factor --repair-all Repair tables : it repairs the table metadata of existing tables -u, --user= Cassandra user ``` ::: ### Create namespaces and tables To create namespaces and tables by using a properties file, run the following command, replacing the contents in the angle brackets as described: ```console java -jar scalardb-schema-loader-.jar --config -f [--coordinator] ``` If `--coordinator` is specified, a [Coordinator table](api-guide.mdx#specify-operations-for-the-coordinator-table) will be created. When the [group commit feature for the Coordinator table](api-guide.mdx#group-commit-for-coordinator-table) is enabled, this command creates a Coordinator table with additional columns for the group commit feature. :::note The following database-specific CLI arguments have been deprecated. Please use the CLI arguments for configuring the properties file instead. ```console java -jar scalardb-schema-loader-.jar --jdbc -j -u -p -f ``` ```console java -jar scalardb-schema-loader-.jar --dynamo -u -p --region -f [-r BASE_RESOURCE_UNIT] ``` - `` should be a string to specify an AWS region like `ap-northeast-1`. - `-r` option is almost the same as Cosmos DB for NoSQL option. However, the unit means DynamoDB capacity unit. The read and write capacity units are set the same value. ```console java -jar scalardb-schema-loader-.jar --cosmos -h -p -f [-r BASE_RESOURCE_UNIT] ``` - `` you can use a primary key or a secondary key. - `-r BASE_RESOURCE_UNIT` is an option. You can specify the RU of each database. The maximum RU in tables in the database will be set. If you don't specify RU of tables, the database RU will be set with this option. By default, it's 400. ```console java -jar scalardb-schema-loader-.jar --cassandra -h [-P ] [-u ] [-p ] -f [-n ] [-R ] ``` - If `-P ` is not supplied, it defaults to `9042`. - If `-u ` is not supplied, it defaults to `cassandra`. - If `-p ` is not supplied, it defaults to `cassandra`. - `` should be `SimpleStrategy` or `NetworkTopologyStrategy`. ::: ### Alter tables You can use a command to add new columns to and create or delete a secondary index for existing tables. This command compares the provided table schema to the existing schema to decide which columns need to be added and which indexes need to be created or deleted. To add new columns to and create or delete a secondary index for existing tables, run the following command, replacing the contents in the angle brackets as described: ```console java -jar scalardb-schema-loader-.jar --config -f --alter ``` :::note The following database-specific CLI arguments have been deprecated. Please use the CLI arguments for configuring the properties file instead. ```console java -jar scalardb-schema-loader-.jar --jdbc -j -u -p -f --alter ``` ```console java -jar scalardb-schema-loader-.jar --dynamo -u -p --region -f --alter ``` ```console java -jar scalardb-schema-loader-.jar --cosmos -h -p -f --alter ``` ```console java -jar scalardb-schema-loader-.jar --cassandra -h [-P ] [-u ] [-p ] -f --alter ``` ::: ### Delete tables You can delete tables by using the properties file. To delete tables, run the following command, replacing the contents in the angle brackets as described: ```console java -jar scalardb-schema-loader-.jar --config -f [--coordinator] -D ``` If `--coordinator` is specified, the Coordinator table will be deleted as well. :::note The following database-specific CLI arguments have been deprecated. Please use the CLI arguments for configuring the properties file instead. ```console java -jar scalardb-schema-loader-.jar --jdbc -j -u -p -f -D ``` ```console java -jar scalardb-schema-loader-.jar --dynamo -u -p --region -f -D ``` ```console java -jar scalardb-schema-loader-.jar --cosmos -h -p -f -D ``` ```console java -jar scalardb-schema-loader-.jar --cassandra -h [-P ] [-u ] [-p ] -f -D ``` ::: ### Repair tables You can repair the table metadata of existing tables by using the properties file. To repair table metadata of existing tables, run the following command, replacing the contents in the angle brackets as described: ```console java -jar scalardb-schema-loader-.jar --config -f [--coordinator] --repair-all ``` :::warning Before executing this command, you should confirm the following configurations are the same as those that were last applied: - The schema configuration - Whether the [group commit feature for the Coordinator table](api-guide.mdx#group-commit-for-coordinator-table) is enabled or not, if the `--coordinator` option described below is specified ::: If `--coordinator` is specified, the Coordinator table will be repaired as well. In addition, if you're using Cosmos DB for NoSQL, running this command will also repair stored procedures attached to each table. :::note The following database-specific CLI arguments have been deprecated. Please use the CLI arguments for configuring the properties file instead. ```console java -jar scalardb-schema-loader-.jar --jdbc -j -u -p -f --repair-all ``` ```console java -jar scalardb-schema-loader-.jar --dynamo -u -p --region [--no-backup] -f --repair-all ``` ```console java -jar scalardb-schema-loader-.jar --cosmos -h -p -f --repair-all ``` ```console java -jar scalardb-schema-loader-.jar --cassandra -h [-P ] [-u ] [-p ] -f --repair-all ``` ::: ### Import tables You can import an existing table in JDBC databases to ScalarDB by using the `--import` option and an import-specific schema file. For details, see [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](./schema-loader-import.mdx). ### Sample schema file The following is a sample schema. For a sample schema file, see [`schema_sample.json`](https://github.com/scalar-labs/scalardb/blob/master/schema-loader/sample/schema_sample.json). ```json { "sample_db.sample_table": { "transaction": false, "partition-key": [ "c1" ], "clustering-key": [ "c4 ASC", "c6 DESC" ], "columns": { "c1": "INT", "c2": "TEXT", "c3": "BLOB", "c4": "INT", "c5": "BOOLEAN", "c6": "INT" }, "secondary-index": [ "c2", "c4" ] }, "sample_db.sample_table1": { "transaction": true, "partition-key": [ "c1" ], "clustering-key": [ "c4" ], "columns": { "c1": "INT", "c2": "TEXT", "c3": "INT", "c4": "INT", "c5": "BOOLEAN" } }, "sample_db.sample_table2": { "transaction": false, "partition-key": [ "c1" ], "clustering-key": [ "c4", "c3" ], "columns": { "c1": "INT", "c2": "TEXT", "c3": "INT", "c4": "INT", "c5": "BOOLEAN" } } } ``` The schema has table definitions that include `columns`, `partition-key`, `clustering-key`, `secondary-index`, and `transaction` fields. - The `columns` field defines columns of the table and their data types. - The `partition-key` field defines which columns the partition key is composed of. - The `clustering-key` field defines which columns the clustering key is composed of. - The `secondary-index` field defines which columns are indexed. - The `transaction` field indicates whether the table is for transactions or not. - If you set the `transaction` field to `true` or don't specify the `transaction` field, this tool creates a table with transaction metadata if needed. - If you set the `transaction` field to `false`, this tool creates a table without any transaction metadata (that is, for a table with [Storage API](run-non-transactional-storage-operations-through-primitive-crud-interface.mdx)). You can also specify database or storage-specific options in the table definition as follows: ```json { "sample_db.sample_table3": { "partition-key": [ "c1" ], "columns": { "c1": "INT", "c2": "TEXT", "c3": "BLOB" }, "compaction-strategy": "LCS", "ru": 5000 } } ``` The database or storage-specific options you can specify are as follows: No options are available for JDBC databases. The `ru` option stands for Request Units. For details, see [RUs](#rus). The `ru` option stands for Request Units. For details, see [RUs](#rus). The `compaction-strategy` option is the compaction strategy used. This option should be `STCS` (SizeTieredCompaction), `LCS` (LeveledCompactionStrategy), or `TWCS` (TimeWindowCompactionStrategy). ## Scale for performance when using Cosmos DB for NoSQL or DynamoDB When using Cosmos DB for NoSQL or DynamoDB, you can scale by using Request Units (RUs) or auto-scaling. ### RUs You can scale the throughput of Cosmos DB for NoSQL and DynamoDB by specifying the `--ru` option. When specifying this option, scaling applies to all tables or the `ru` parameter for each table. If the `--ru` option is not set, the default values will be `400` for Cosmos DB for NoSQL and `10` for DynamoDB. :::note - Schema Loader abstracts [Request Units](https://docs.microsoft.com/azure/cosmos-db/request-units) for Cosmos DB for NoSQL and [Capacity Units](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.ProvisionedThroughput.Manual) for DynamoDB with `RU`. Therefore, be sure to set an appropriate value depending on the database implementation. - Be aware that Schema Loader sets the same value to both read capacity unit and write capacity unit for DynamoDB. ::: ### Auto-scaling By default, Schema Loader enables auto-scaling of RUs for all tables: RUs scale between 10 percent and 100 percent of a specified RU depending on the workload. For example, if you specify `-r 10000`, the RUs of each table auto-scales between `1000` and `10000`. :::note Auto-scaling for Cosmos DB for NoSQL is enabled only when this option is set to `4000` or more. ::: ## Internal transaction metadata for Consensus Commit The Consensus Commit transaction manager manages transaction metadata (for example, transaction ID, record version, and transaction status) stored along with the actual records to handle transactions properly. Thus, along with any columns that the application requires, additional columns for the metadata need to be defined in the schema. Additionally, this tool creates a table with the metadata if you use the Consensus Commit transaction manager. ### Decoupling transaction metadata You can separately manage the transaction metadata from application data by enabling [transaction metadata decoupling](./consensus-commit.mdx#transaction-metadata-decoupling). #### Create a new table with transaction metadata decoupling When creating a new table with transaction metadata decoupling, you first need to enable the option by adding the `"transaction-metadata-decoupling": true` to your schema file, and then create a table. The following is an example schema file: ```json { "sample_db.sample_table": { "transaction-metadata-decoupling": true, "partition-key": [ "c1" ], "clustering-key": [ "c4" ], "columns": { "c1": "INT", "c2": "TEXT", "c3": "INT", "c4": "INT" } } } ``` ScalarDB internally creates two tables per table defined in the schema: an application table and a transaction metadata table as follows: | Table | Description | |----------------------------|--------------------------------------------------------------------------------------------------------------------------| | `_data` | Application table that stores user data | | `_tx_metadata` | Transaction metadata table that stores transaction-related metadata and before-image columns for non-primary key columns | Then, you can access the table `` as a virtual table that combines the data table and transaction metadata table by using `INNER JOIN`. #### Import an existing table with transaction metadata decoupling When importing an existing table with transaction metadata decoupling, you also need to enable the option by adding the `"transaction-metadata-decoupling": true` to your schema file, and then import a table. The following is an example schema file: ```json { "sample_db.sample_table": { "transaction-metadata-decoupling": true } } ``` ScalarDB assumes that the existing table is the data table and creates only the transaction metadata table as follows: | Table | Description | |----------------------------|--------------------------------------------------------------------------------------------------------------------------| | `` | Original application table (unchanged) | | `_tx_metadata` | Transaction metadata table that stores transaction-related metadata and before-image columns for non-primary key columns | Then, you can access the table `_scalardb` as a virtual table that combines the data table and transaction metadata table by using `LEFT OUTER JOIN`. #### Limitations Currently, when using transaction metadata decoupling, the following operations are not supported: - Repairing tables - Adding or dropping columns - Renaming tables - Altering column types ## Use Schema Loader in your application You can check the version of Schema Loader from the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb-schema-loader). For example in Gradle, you can add the following dependency to your `build.gradle` file, replacing `` with the version of Schema Loader that you want to use: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-schema-loader:' } ``` ### Create, alter, repair, or delete tables You can create, alter, delete, or repair tables that are defined in the schema by using Schema Loader. To do this, you can pass a ScalarDB properties file, schema, and additional options, if needed, as shown below: ```java public class SchemaLoaderSample { public static int main(String... args) throws SchemaLoaderException { Path configFilePath = Paths.get("database.properties"); // "sample_schema.json" and "altered_sample_schema.json" can be found in the "/sample" directory. Path schemaFilePath = Paths.get("sample_schema.json"); Path alteredSchemaFilePath = Paths.get("altered_sample_schema.json"); boolean createCoordinatorTables = true; // whether to create the Coordinator table or not boolean deleteCoordinatorTables = true; // whether to delete the Coordinator table or not boolean repairCoordinatorTables = true; // whether to repair the Coordinator table or not Map tableCreationOptions = new HashMap<>(); tableCreationOptions.put( CassandraAdmin.REPLICATION_STRATEGY, ReplicationStrategy.SIMPLE_STRATEGY.toString()); tableCreationOptions.put(CassandraAdmin.COMPACTION_STRATEGY, CompactionStrategy.LCS.toString()); tableCreationOptions.put(CassandraAdmin.REPLICATION_FACTOR, "1"); tableCreationOptions.put(DynamoAdmin.REQUEST_UNIT, "1"); tableCreationOptions.put(DynamoAdmin.NO_SCALING, "true"); tableCreationOptions.put(DynamoAdmin.NO_BACKUP, "true"); Map indexCreationOptions = new HashMap<>(); indexCreationOptions.put(DynamoAdmin.NO_SCALING, "true"); Map tableReparationOptions = new HashMap<>(); indexCreationOptions.put(DynamoAdmin.NO_BACKUP, "true"); // Create tables. SchemaLoader.load(configFilePath, schemaFilePath, tableCreationOptions, createCoordinatorTables); // Alter tables. SchemaLoader.alterTables(configFilePath, alteredSchemaFilePath, indexCreationOptions); // Repair tables. SchemaLoader.repairTables(configFilePath, schemaFilePath, tableReparationOptions, repairCoordinatorTables); // Delete tables. SchemaLoader.unload(configFilePath, schemaFilePath, deleteCoordinatorTables); return 0; } } ``` You can also create, delete, or repair a schema by passing a serialized-schema JSON string (the raw text of a schema file) as shown below: ```java // Create tables. SchemaLoader.load(configFilePath, serializedSchemaJson, tableCreationOptions, createCoordinatorTables); // Alter tables. SchemaLoader.alterTables(configFilePath, serializedAlteredSchemaFilePath, indexCreationOptions); // Repair tables. SchemaLoader.repairTables(configFilePath, serializedSchemaJson, tableReparationOptions, repairCoordinatorTables); // Delete tables. SchemaLoader.unload(configFilePath, serializedSchemaJson, deleteCoordinatorTables); ``` When configuring ScalarDB, you can use a `Properties` object as well, as shown below: ```java // Create tables. SchemaLoader.load(properties, serializedSchemaJson, tableCreationOptions, createCoordinatorTables); // Alter tables. SchemaLoader.alterTables(properties, serializedAlteredSchemaFilePath, indexCreationOptions); // Repair tables. SchemaLoader.repairTables(properties, serializedSchemaJson, tableReparationOptions, repairCoordinatorTables); // Delete tables. SchemaLoader.unload(properties, serializedSchemaJson, deleteCoordinatorTables); ``` ### Import tables You can import an existing JDBC database table to ScalarDB by using the `--import` option and an import-specific schema file, in a similar manner as shown in [Sample schema file](#sample-schema-file). For details, see [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](./schema-loader-import.mdx). :::warning You should carefully plan to import a table to ScalarDB in production because it will add transaction metadata columns to your database tables and the ScalarDB metadata tables. In this case, there would also be several differences between your database and ScalarDB, as well as some limitations. ::: The following is an import sample: ```java public class SchemaLoaderImportSample { public static int main(String... args) throws SchemaLoaderException { Path configFilePath = Paths.get("database.properties"); // "import_sample_schema.json" can be found in the "/sample" directory. Path schemaFilePath = Paths.get("import_sample_schema.json"); Map tableImportOptions = new HashMap<>(); // Import tables. // You can also use a Properties object instead of configFilePath and a serialized-schema JSON // string instead of schemaFilePath. SchemaLoader.importTables(configFilePath, schemaFilePath, tableImportOptions); return 0; } } ``` ================================================ FILE: docs/two-phase-commit-transactions.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Transactions with a Two-Phase Commit Interface ScalarDB supports executing transactions with a two-phase commit interface. With the two-phase commit interface, you can execute a transaction that spans multiple processes or applications, like in a microservice architecture. This page explains how transactions with a two-phase commit interface work in ScalarDB and how to configure and execute them in ScalarDB. ## How transactions with a two-phase commit interface work in ScalarDB ScalarDB normally executes transactions in a single transaction manager instance with a one-phase commit interface. In transactions with a one-phase commit interface, you begin a transaction, execute CRUD operations, and commit the transaction in the same transaction manager instance. In ScalarDB, you can execute transactions with a two-phase commit interface that span multiple transaction manager instances. The transaction manager instances can be in the same process or application, or the instances can be in different processes or applications. For example, if you have transaction manager instances in multiple microservices, you can execute a transaction that spans multiple microservices. In transactions with a two-phase commit interface, there are two roles—Coordinator and a participant—that collaboratively execute a single transaction. The Coordinator process and the participant processes all have different transaction manager instances. The Coordinator process first begins or starts a transaction, and the participant processes join the transaction. After executing CRUD operations, the Coordinator process and the participant processes commit the transaction by using the two-phase interface. ## How to execute transactions with a two-phase commit interface To execute a two-phase commit transaction, you must get the transaction manager instance. Then, the Coordinator process can begin or start the transaction, and the participant can process the transaction. ### Get a `TwoPhaseCommitTransactionManager` instance You first need to get a `TwoPhaseCommitTransactionManager` instance to execute transactions with a two-phase commit interface. To get a `TwoPhaseCommitTransactionManager` instance, you can use `TransactionFactory` as follows: ```java TransactionFactory factory = TransactionFactory.create(""); TwoPhaseCommitTransactionManager transactionManager = factory.getTwoPhaseCommitTransactionManager(); ``` ### Begin or start a transaction (for Coordinator) For the process or application that begins the transaction to act as Coordinator, you should use the following `begin` method: ```java // Begin a transaction. TwoPhaseCommitTransaction tx = transactionManager.begin(); ``` Or, for the process or application that begins the transaction to act as Coordinator, you should use the following `start` method: ```java // Start a transaction. TwoPhaseCommitTransaction tx = transactionManager.start(); ``` Alternatively, you can use the `begin` method for a transaction by specifying a transaction ID as follows: ```java // Begin a transaction by specifying a transaction ID. TwoPhaseCommitTransaction tx = transactionManager.begin(""); ``` Or, you can use the `start` method for a transaction by specifying a transaction ID as follows: ```java // Start a transaction by specifying a transaction ID. TwoPhaseCommitTransaction tx = transactionManager.start(""); ``` ### Join a transaction (for participants) For participants, you can join a transaction by specifying the transaction ID associated with the transaction that Coordinator has started or begun as follows: ```java TwoPhaseCommitTransaction tx = transactionManager.join(""); ``` :::note To get the transaction ID with `getId()`, you can specify the following: ```java tx.getId(); ``` ::: ### CRUD operations for the transaction The CRUD operations for `TwoPhaseCommitTransacton` are the same as the operations for `DistributedTransaction`. For details, see [CRUD operations](api-guide.mdx#crud-operations). The following is example code for CRUD operations in transactions with a two-phase commit interface: ```java TwoPhaseCommitTransaction tx = ... // Retrieve the current balances by ID. Get fromGet = Get.newBuilder() .namespace(NAMESPACE) .table(TABLE) .partitionKey(new Key(ID, fromId)) .build(); Get toGet = Get.newBuilder() .namespace(NAMESPACE) .table(TABLE) .partitionKey(new Key(ID, toId)) .build(); Optional fromResult = tx.get(fromGet); Optional toResult = tx.get(toGet); // Calculate the balances (assuming that both accounts exist). int newFromBalance = fromResult.get().getInt(BALANCE) - amount; int newToBalance = toResult.get().getInt(BALANCE) + amount; // Update the balances. Put fromPut = Put.newBuilder() .namespace(NAMESPACE) .table(TABLE) .partitionKey(new Key(ID, fromId)) .intValue(BALANCE, newFromBalance) .build(); Put toPut = Put.newBuilder() .namespace(NAMESPACE) .table(TABLE) .partitionKey(new Key(ID, toId)) .intValue(BALANCE, newToBalance) .build(); tx.put(fromPut); tx.put(toPut); ``` ### Prepare, commit, or roll back a transaction After finishing CRUD operations, you need to commit the transaction. As with the standard two-phase commit protocol, there are two phases: prepare and commit. In all the Coordinator and participant processes, you need to prepare and then commit the transaction as follows: ```java TwoPhaseCommitTransaction tx = ... try { // Execute CRUD operations in the Coordinator and participant processes. ... // Prepare phase: Prepare the transaction in all the Coordinator and participant processes. tx.prepare(); ... // Commit phase: Commit the transaction in all the Coordinator and participant processes. tx.commit(); ... } catch (TransactionException e) { // If an error happens, you will need to roll back the transaction in all the Coordinator and participant processes. tx.rollback(); ... } ``` For `prepare()`, if any of the Coordinator or participant processes fail to prepare the transaction, you will need to call `rollback()` (or `abort()`) in all the Coordinator and participant processes. For `commit()`, if any of the Coordinator or participant processes successfully commit the transaction, you can consider the transaction as committed. When a transaction has been committed, you can ignore any errors in the other Coordinator and participant processes. If all the Coordinator and participant processes fail to commit the transaction, you will need to call `rollback()` (or `abort()`) in all the Coordinator and participant processes. For better performance, you can call `prepare()`, `commit()`, and `rollback()` in the Coordinator and participant processes in parallel, respectively. #### Validate the transaction Depending on the concurrency control protocol, you need to call `validate()` in all the Coordinator and participant processes after `prepare()` and before `commit()`, as shown below: ```java // Prepare phase 1: Prepare the transaction in all the Coordinator and participant processes. tx.prepare(); ... // Prepare phase 2: Validate the transaction in all the Coordinator and participant processes. tx.validate(); ... // Commit phase: Commit the transaction in all the Coordinator and participant processes. tx.commit(); ... ``` Similar to `prepare()`, if any of the Coordinator or participant processes fail to validate the transaction, you will need to call `rollback()` (or `abort()`) in all the Coordinator and participant processes. In addition, you can call `validate()` in the Coordinator and participant processes in parallel for better performance. :::note When using the [Consensus Commit](configurations.mdx#use-consensus-commit-directly) transaction manager with `EXTRA_READ` set as the value for `scalar.db.consensus_commit.serializable_strategy` and `SERIALIZABLE` set as the value for `scalar.db.consensus_commit.isolation_level`, you need to call `validate()`. However, if you are not using Consensus Commit, specifying `validate()` will not have any effect. ::: ### Execute a transaction by using multiple transaction manager instances By using the APIs described above, you can execute a transaction by using multiple transaction manager instances. The following is an example of how that works. :::warning The following sample code is a simplified version that solely illustrates how to use the two-phase commit interface and doesn't represent a real-world use case. In actual applications, it is assumed that a transaction manager instance is created per process or service. For a real-world example, see the [microservice transactions sample tutorial](scalardb-samples/microservice-transaction-sample/README.mdx). ::: ```java TransactionFactory factory1 = TransactionFactory.create(""); TwoPhaseCommitTransactionManager transactionManager1 = factory1.getTwoPhaseCommitTransactionManager(); TransactionFactory factory2 = TransactionFactory.create(""); TwoPhaseCommitTransactionManager transactionManager2 = factory2.getTwoPhaseCommitTransactionManager(); TwoPhaseCommitTransaction transaction1 = null; TwoPhaseCommitTransaction transaction2 = null; try { // Begin a transaction. transaction1 = transactionManager1.begin(); // Join the transaction begun by `transactionManager1` by getting the transaction ID. transaction2 = transactionManager2.join(transaction1.getId()); // Execute CRUD operations in the transaction. Optional result = transaction1.get(...); List results = transaction2.scan(...); transaction1.put(...); transaction2.delete(...); // Prepare the transaction. transaction1.prepare(); transaction2.prepare(); // Validate the transaction. transaction1.validate(); transaction2.validate(); // Commit the transaction. If any of the transactions successfully commit, // you can regard the transaction as committed. AtomicReference exception = new AtomicReference<>(); boolean anyMatch = Stream.of(transaction1, transaction2) .anyMatch( t -> { try { t.commit(); return true; } catch (TransactionException e) { exception.set(e); return false; } }); // If all the transactions fail to commit, throw the exception and roll back the transaction. if (!anyMatch) { throw exception.get(); } } catch (TransactionException e) { // Roll back the transaction. if (transaction1 != null) { try { transaction1.rollback(); } catch (RollbackException e1) { // Handle the exception. } } if (transaction2 != null) { try { transaction2.rollback(); } catch (RollbackException e1) { // Handle the exception. } } } ``` For simplicity, the above example code doesn't handle the exceptions that the APIs may throw. For details about handling exceptions, see [How to handle exceptions](#how-to-handle-exceptions). As previously mentioned, for `commit()`, if any of the Coordinator or participant processes succeed in committing the transaction, you can consider the transaction as committed. Also, for better performance, you can execute `prepare()`, `validate()`, and `commit()` in parallel, respectively. ### Resume or re-join a transaction Given that processes or applications that use transactions with a two-phase commit interface usually involve multiple request and response exchanges, you might need to execute a transaction across various endpoints or APIs. For such scenarios, you can use `resume()` or `join()` to get a transaction object (an instance of `TwoPhaseCommitTransaction`) for the transaction that you previously joined. The following shows how `resume()` and `join()` work: ```java // Join (or begin) the transaction. TwoPhaseCommitTransaction tx = transactionManager.join(""); ... // Resume the transaction by using the transaction ID. TwoPhaseCommitTransaction tx1 = transactionManager.resume(""); // Or you can re-join the transaction by using the transaction ID. TwoPhaseCommitTransaction tx2 = transactionManager.join(""); ``` :::note To get the transaction ID with `getId()`, you can specify the following: ```java tx.getId(); ``` In addition, when using `join()` to re-join a transaction, if you have not joined the transaction before, a new transaction object will be returned. On the other hand, when using `resume()` to resume a transaction, if you have not joined the transaction before, `TransactionNotFoundException` will be thrown. ::: The following is an example of two services that have multiple endpoints: ```java interface ServiceA { void facadeEndpoint() throws Exception; } interface ServiceB { void endpoint1(String txId) throws Exception; void endpoint2(String txId) throws Exception; void prepare(String txId) throws Exception; void commit(String txId) throws Exception; void rollback(String txId) throws Exception; } ``` The following is an example of a client calling `ServiceA.facadeEndpoint()` that begins a transaction that spans the two services (`ServiceA` and `ServiceB`): ```java public class ServiceAImpl implements ServiceA { private TwoPhaseCommitTransactionManager transactionManager = ...; private ServiceB serviceB = ...; ... @Override public void facadeEndpoint() throws Exception { TwoPhaseCommitTransaction tx = transactionManager.begin(); try { ... // Call `ServiceB` `endpoint1`. serviceB.endpoint1(tx.getId()); ... // Call `ServiceB` `endpoint2`. serviceB.endpoint2(tx.getId()); ... // Prepare. tx.prepare(); serviceB.prepare(tx.getId()); // Commit. tx.commit(); serviceB.commit(tx.getId()); } catch (Exception e) { // Roll back. tx.rollback(); serviceB.rollback(tx.getId()); } } } ``` As shown above, the facade endpoint in `ServiceA` calls multiple endpoints (`endpoint1()`, `endpoint2()`, `prepare()`, `commit()`, and `rollback()`) of `ServiceB`. In addition, in transactions with a two-phase commit interface, you need to use the same transaction object across the endpoints. In this situation, you can resume the transaction. The implementation of `ServiceB` is as follows: ```java public class ServiceBImpl implements ServiceB { private TwoPhaseCommitTransactionManager transactionManager = ...; ... @Override public void endpoint1(String txId) throws Exception { // Join the transaction. TwoPhaseCommitTransaction tx = transactionManager.join(txId); ... } @Override public void endpoint2(String txId) throws Exception { // Resume the transaction that you joined in `endpoint1()`. TwoPhaseCommitTransaction tx = transactionManager.resume(txId); // Or re-join the transaction that you joined in `endpoint1()`. // TwoPhaseCommitTransaction tx = transactionManager.join(txId); ... } @Override public void prepare(String txId) throws Exception { // Resume the transaction. TwoPhaseCommitTransaction tx = transactionManager.resume(txId); // Or re-join the transaction. // TwoPhaseCommitTransaction tx = transactionManager.join(txId); ... // Prepare. tx.prepare(); } @Override public void commit(String txId) throws Exception { // Resume the transaction. TwoPhaseCommitTransaction tx = transactionManager.resume(txId); // Or re-join the transaction. // TwoPhaseCommitTransaction tx = transactionManager.join(txId); ... // Commit. tx.commit(); } @Override public void rollback(String txId) throws Exception { // Resume the transaction. TwoPhaseCommitTransaction tx = transactionManager.resume(txId); // Or re-join the transaction. // TwoPhaseCommitTransaction tx = transactionManager.join(txId); ... // Roll back. tx.rollback(); } } ``` As shown above, by resuming or re-joining the transaction, you can share the same transaction object across multiple endpoints in `ServiceB`. ## How to handle exceptions When executing a transaction by using multiple transaction manager instances, you will also need to handle exceptions properly. :::warning If you don't handle exceptions properly, you may face anomalies or data inconsistency. ::: For instance, in the example code in [Execute a transaction by using multiple transaction manager instances](#execute-a-transaction-by-using-multiple-transaction-manager-instances), multiple transaction managers (`transactionManager1` and `transactionManager2`) are used in a single process for ease of explanation. However, that example code doesn't include a way to handle exceptions. The following example code shows how to handle exceptions in transactions with a two-phase commit interface: ```java public class Sample { public static void main(String[] args) throws Exception { TransactionFactory factory1 = TransactionFactory.create(""); TwoPhaseCommitTransactionManager transactionManager1 = factory1.getTwoPhaseCommitTransactionManager(); TransactionFactory factory2 = TransactionFactory.create(""); TwoPhaseCommitTransactionManager transactionManager2 = factory2.getTwoPhaseCommitTransactionManager(); int retryCount = 0; TransactionException lastException = null; while (true) { if (retryCount++ > 0) { // Retry the transaction three times maximum in this sample code. if (retryCount >= 3) { // Throw the last exception if the number of retries exceeds the maximum. throw lastException; } // Sleep 100 milliseconds before retrying the transaction in this sample code. TimeUnit.MILLISECONDS.sleep(100); } TwoPhaseCommitTransaction transaction1 = null; TwoPhaseCommitTransaction transaction2 = null; try { // Begin a transaction. transaction1 = transactionManager1.begin(); // Join the transaction that `transactionManager1` begun by using the transaction ID. transaction2 = transactionManager2.join(transaction1.getId()); // Execute CRUD operations in the transaction. Optional result = transaction1.get(...); List results = transaction2.scan(...); transaction1.put(...); transaction2.delete(...); // Prepare the transaction. prepare(transaction1, transaction2); // Validate the transaction. validate(transaction1, transaction2); // Commit the transaction. commit(transaction1, transaction2); } catch (UnknownTransactionStatusException e) { // If you catch `UnknownTransactionStatusException` when committing the transaction, // it indicates that the status of the transaction, whether it was successful or not, is unknown. // In such a case, you need to check if the transaction is committed successfully or not and // retry the transaction if it failed. How to identify a transaction status is delegated to users. return; } catch (TransactionException e) { // For other exceptions, you can try retrying the transaction. // For `CrudConflictException`, `PreparationConflictException`, `ValidationConflictException`, // `CommitConflictException`, and `TransactionNotFoundException`, you can basically retry the // transaction. However, for the other exceptions, the transaction will still fail if the cause of // the exception is non-transient. In such a case, you will exhaust the number of retries and // throw the last exception. rollback(transaction1, transaction2); lastException = e; } } } private static void prepare(TwoPhaseCommitTransaction... transactions) throws TransactionException { // You can execute `prepare()` in parallel. List exceptions = Stream.of(transactions) .parallel() .map( t -> { try { t.prepare(); return null; } catch (TransactionException e) { return e; } }) .filter(Objects::nonNull) .collect(Collectors.toList()); // If any of the transactions failed to prepare, throw the exception. if (!exceptions.isEmpty()) { throw exceptions.get(0); } } private static void validate(TwoPhaseCommitTransaction... transactions) throws TransactionException { // You can execute `validate()` in parallel. List exceptions = Stream.of(transactions) .parallel() .map( t -> { try { t.validate(); return null; } catch (TransactionException e) { return e; } }) .filter(Objects::nonNull) .collect(Collectors.toList()); // If any of the transactions failed to validate, throw the exception. if (!exceptions.isEmpty()) { throw exceptions.get(0); } } private static void commit(TwoPhaseCommitTransaction... transactions) throws TransactionException { // You can execute `commit()` in parallel. List exceptions = Stream.of(transactions) .parallel() .map( t -> { try { t.commit(); return null; } catch (TransactionException e) { return e; } }) .filter(Objects::nonNull) .collect(Collectors.toList()); // If any of the transactions successfully committed, you can regard the transaction as committed. if (exceptions.size() < transactions.length) { if (!exceptions.isEmpty()) { // You can log the exceptions here if you want. } return; // Commit was successful. } // // If all the transactions failed to commit: // // If any of the transactions failed to commit due to `UnknownTransactionStatusException`, throw // it because you should not retry the transaction in such a case. Optional unknownTransactionStatusException = exceptions.stream().filter(e -> e instanceof UnknownTransactionStatusException).findFirst(); if (unknownTransactionStatusException.isPresent()) { throw unknownTransactionStatusException.get(); } // Otherwise, throw the first exception. throw exceptions.get(0); } private static void rollback(TwoPhaseCommitTransaction... transactions) { Stream.of(transactions) .parallel() .filter(Objects::nonNull) .forEach( t -> { try { t.rollback(); } catch (RollbackException e) { // Rolling back the transaction failed. The transaction should eventually recover, // so you don't need to do anything further. You can simply log the occurrence here. } }); } } ``` ### `TransactionException` and `TransactionNotFoundException` The `begin()` API could throw `TransactionException` or `TransactionNotFoundException`: - If you catch `TransactionException`, this exception indicates that the transaction has failed to begin due to transient or non-transient faults. You can try retrying the transaction, but you may not be able to begin the transaction due to non-transient faults. - If you catch `TransactionNotFoundException`, this exception indicates that the transaction has failed to begin due to transient faults. In this case, you can retry the transaction. The `join()` API could also throw `TransactionNotFoundException`. You can handle this exception in the same way that you handle the exceptions for the `begin()` API. ### `CrudException` and `CrudConflictException` The APIs for CRUD operations (`get()`, `scan()`, `put()`, `delete()`, and `mutate()`) could throw `CrudException` or `CrudConflictException`: - If you catch `CrudException`, this exception indicates that the transaction CRUD operation has failed due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction will still fail if the cause is non-transient. - If you catch `CrudConflictException`, this exception indicates that the transaction CRUD operation has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. ### `PreparationException` and `PreparationConflictException` The `prepare()` API could throw `PreparationException` or `PreparationConflictException`: - If you catch `PreparationException`, this exception indicates that preparing the transaction fails due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction will still fail if the cause is non-transient. - If you catch `PreparationConflictException`, this exception indicates that preparing the transaction has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. ### `ValidationException` and `ValidationConflictException` The `validate()` API could throw `ValidationException` or `ValidationConflictException`: - If you catch `ValidationException`, this exception indicates that validating the transaction fails due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction will still fail if the cause is non-transient. - If you catch `ValidationConflictException`, this exception indicates that validating the transaction has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. ### `CommitException`, `CommitConflictException`, and `UnknownTransactionStatusException` The `commit()` API could throw `CommitException`, `CommitConflictException`, or `UnknownTransactionStatusException`: - If you catch `CommitException`, this exception indicates that committing the transaction fails due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction will still fail if the cause is non-transient. - If you catch `CommitConflictException`, this exception indicates that committing the transaction has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. - If you catch `UnknownTransactionStatusException`, this exception indicates that the status of the transaction, whether it was successful or not, is unknown. In this case, you need to check if the transaction is committed successfully and retry the transaction if it has failed. How to identify a transaction status is delegated to users. You may want to create a transaction status table and update it transactionally with other application data so that you can get the status of a transaction from the status table. ### Notes about some exceptions Although not illustrated in the example code, the `resume()` API could also throw `TransactionNotFoundException`. This exception indicates that the transaction associated with the specified ID was not found and/or the transaction might have expired. In either case, you can retry the transaction from the beginning since the cause of this exception is basically transient. In the sample code, for `UnknownTransactionStatusException`, the transaction is not retried because the application must check if the transaction was successful to avoid potential duplicate operations. For other exceptions, the transaction is retried because the cause of the exception is transient or non-transient. If the cause of the exception is transient, the transaction may succeed if you retry it. However, if the cause of the exception is non-transient, the transaction will still fail even if you retry it. In such a case, you will exhaust the number of retries. :::note If you begin a transaction by specifying a transaction ID, you must use a different ID when you retry the transaction. In addition, in the sample code, the transaction is retried three times maximum and sleeps for 100 milliseconds before it is retried. But you can choose a retry policy, such as exponential backoff, according to your application requirements. ::: ## Request routing in transactions with a two-phase commit interface Services that use transactions with a two-phase commit interface usually execute a transaction by exchanging multiple requests and responses, as shown in the following diagram: ![Sequence diagram for transactions with a two-phase commit interface](images/two_phase_commit_sequence_diagram.png) In addition, each service typically has multiple servers (or hosts) for scalability and availability and uses server-side (proxy) or client-side load balancing to distribute requests to the servers. In such a case, since transaction processing in transactions with a two-phase commit interface is stateful, requests in a transaction must be routed to the same servers while different transactions need to be distributed to balance the load, as shown in the following diagram: ![Load balancing for transactions with a two-phase commit interface](images/two_phase_commit_load_balancing.png) There are several approaches to achieve load balancing for transactions with a two-phase commit interface depending on the protocol between the services. Some approaches for this include using gRPC, HTTP/1.1, and [ScalarDB Cluster](scalardb-cluster/index.mdx), which is a component that is available only in the ScalarDB Enterprise edition. ### gRPC When you use a client-side load balancer, you can use the same gRPC connection to send requests in a transaction, which guarantees that the requests go to the same servers. When you use a server-side (proxy) load balancer, solutions are different between an L3/L4 (transport-level) load balancer and an L7 (application-level) load balancer: - When using an L3/L4 load balancer, you can use the same gRPC connection to send requests in a transaction, similar to when you use a client-side load balancer. In this case, requests in the same gRPC connection always go to the same server. - When using an L7 load balancer, since requests in the same gRPC connection don't necessarily go to the same server, you need to use cookies or similar method to route requests to the correct server. - For example, if you use [Envoy](https://www.envoyproxy.io/), you can use session affinity (sticky session) for gRPC. Alternatively, you can use [bidirectional streaming RPC in gRPC](https://grpc.io/docs/what-is-grpc/core-concepts/#bidirectional-streaming-rpc) since the L7 load balancer distributes requests in the same stream to the same server. For more details about load balancing in gRPC, see [gRPC Load Balancing](https://grpc.io/blog/grpc-load-balancing/). ### HTTP/1.1 Typically, you use a server-side (proxy) load balancer with HTTP/1.1: - When using an L3/L4 load balancer, you can use the same HTTP connection to send requests in a transaction, which guarantees the requests go to the same server. - When using an L7 load balancer, since requests in the same HTTP connection don't necessarily go to the same server, you need to use cookies or similar method to route requests to the correct server. You can use session affinity (sticky session) in that case. ### ScalarDB Cluster ScalarDB Cluster addresses request routing by providing a routing mechanism that is capable of directing requests to the appropriate cluster node within the cluster. For details about ScalarDB Cluster, see [ScalarDB Cluster](scalardb-cluster/index.mdx). ## Hands-on tutorial One of the use cases for transactions with a two-phase commit interface is microservice transactions. For a hands-on tutorial, see [Create a Sample Application That Supports Microservice Transactions](scalardb-samples/microservice-transaction-sample/README.mdx). ================================================ FILE: docs/components/_getting-started-load-schema.mdx ================================================ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; import Admonition from '@theme/Admonition'; import Link from '@docusaurus/Link'; {/* Reusable Schema Loader command tabs for "Load the database schema" sections. Used by the ScalarDB Core (Java and Kotlin) getting-started guides, which share the same `database.properties` config file and the same `schema.json` schema. ScalarDB Cluster guides use a different schema-loader binary and are not handled by this component. The UI uses the same two-level nested-tabs design as : - Outer tabs (groupId="storage-category") pick a category: "Relational databases" | "NewSQL databases" | "NoSQL databases" - Inner tabs (groupId="databases") pick a specific storage in that category, listed in alphabetical order. Takes no props. */} export const baseCommand = (extraFlags) => `java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator${extraFlags ? ' ' + extraFlags : ''}`; export const loaderCommand = (extraFlags) => ( {baseCommand(extraFlags)} ); export const CoordinatorNote = ({ children }) => (

The --coordinator option is specified because a table with transaction set to true exists in the schema. For details about configuring and loading a schema, see ScalarDB Schema Loader.

{children} ); Select your relational database. {loaderCommand()} {loaderCommand()} {loaderCommand()} {loaderCommand()} {loaderCommand()} {loaderCommand()} {loaderCommand()} Select your NewSQL database. {loaderCommand()} {loaderCommand()} {loaderCommand()} {loaderCommand()} Select your NoSQL database. {loaderCommand('--replication-factor=1')}

In addition, the --replication-factor=1 option has an effect only when using Cassandra. The default replication factor is 3, but to facilitate the setup in this tutorial, 1 is used so that you only need to prepare a cluster with one node instead of three nodes. However, keep in mind that a replication factor of 1 is not suited for production.

{loaderCommand()} {loaderCommand('--no-backup --no-scaling')}

Also, --no-backup and --no-scaling options are specified because Amazon DynamoDB Local does not support continuous backup and auto-scaling.

================================================ FILE: docs/components/_getting-started-setup-storage.mdx ================================================ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock'; import Admonition from '@theme/Admonition'; {/* Reusable storage setup tabs for "Set up a database" sections. The UI uses two-level nested tabs: - Outer tabs (groupId="storage-category") pick a category: "Relational databases" | "NewSQL databases" | "NoSQL databases" | "Object storage" - Inner tabs (groupId="databases") pick a specific storage in that category, listed in alphabetical order. Props: - mode: "core-java" | "core-kotlin" | "cluster" Switches properties filename, "Configure ScalarDB" vs "Configure ScalarDB Cluster", host names in JDBC URLs, and the samples directory under scalardb-samples/. Also controls which mode-specific storage tabs are shown: SQLite is core-only (because it is not recommend to use with ScalarDB Cluster), Object Storage is cluster-only (because they do not support secondary-index and the ScalarDB sample use secondary-index for some of its commands) */} export const samplesDir = (props) => { switch (props.mode) { case 'cluster': return 'scalardb-cluster-standalone-mode'; case 'core-kotlin': return 'scalardb-kotlin-sample'; case 'core-java': return 'scalardb-sample'; default: throw new Error( ` requires a "mode" prop set to one of: "core-java", "core-kotlin", "cluster". Received: ${JSON.stringify(props.mode)}` ); } }; export const propertiesFile = (props) => props.mode === 'cluster' ? 'scalardb-cluster-node.properties' : 'database.properties'; export const configureHeading = (props) => props.mode === 'cluster' ? 'Configure ScalarDB Cluster' : 'Configure ScalarDB'; export const host = (props, service) => props.mode === 'cluster' ? `${service}-1` : 'localhost'; export const dedent = (strings, ...values) => { // Strips the shared leading indentation (and surrounding blank lines) from a // code block, so the source can be indented to match the surrounding JSX // while the rendered output stays flush-left. const raw = strings.reduce( (acc, str, i) => acc + (i ? values[i - 1] : '') + str, '' ); const lines = raw.replace(/^\n/, '').replace(/\n[ \t]*$/, '').split('\n'); const min = Math.min( ...lines.filter((l) => l.trim()).map((l) => l.match(/^[ \t]*/)[0].length) ); return lines.map((l) => l.slice(min)).join('\n'); }; Select your relational database.

Run Db2 locally

You can run IBM Db2 in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start IBM Db2, run the following command: docker compose up -d db2

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for Db2 in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For Db2 scalar.db.storage=jdbc scalar.db.contact_points=jdbc:db2://${host(props, 'db2')}:50000/sample scalar.db.username=db2inst1 scalar.db.password=db2inst1 `}

Run MariaDB locally

You can run MariaDB in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start MariaDB, run the following command: docker compose up -d mariadb

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for MariaDB in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For MariaDB scalar.db.storage=jdbc scalar.db.contact_points=jdbc:mariadb://${host(props, 'mariadb')}:3306 scalar.db.username=root scalar.db.password=mariadb `}

Run MySQL locally

You can run MySQL in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start MySQL, run the following command: docker compose up -d mysql

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for MySQL in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For MySQL scalar.db.storage=jdbc scalar.db.contact_points=jdbc:mysql://${host(props, 'mysql')}:3306/ scalar.db.username=root scalar.db.password=mysql `}

Run Oracle Database locally

You can run Oracle Database in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start Oracle Database, run the following command: docker compose up -d oracle

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for Oracle Database in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For Oracle scalar.db.storage=jdbc scalar.db.contact_points=jdbc:oracle:thin:@//${host(props, 'oracle')}:1521/FREEPDB1 scalar.db.username=SYSTEM scalar.db.password=Oracle `}

Run PostgreSQL locally

You can run PostgreSQL in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start PostgreSQL, run the following command: docker compose up -d postgres

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for PostgreSQL in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For PostgreSQL scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://${host(props, 'postgres')}:5432/ scalar.db.username=postgres scalar.db.password=postgres `}

Run SQL Server locally

You can run SQL Server in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start SQL Server, run the following command: docker compose up -d sqlserver

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for SQL Server in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For SQL Server scalar.db.storage=jdbc scalar.db.contact_points=jdbc:sqlserver://${host(props, 'sqlserver')}:1433;encrypt=true;trustServerCertificate=true scalar.db.username=sa scalar.db.password=SqlServer22 `} {props.mode !== 'cluster' && (

{configureHeading(props)}

SQLite is an embedded, file-based database, so there is no separate server to start. ScalarDB creates the database file automatically when it is first used. The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for SQLite in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For Sqlite scalar.db.storage=jdbc scalar.db.contact_points=jdbc:sqlite:scalardb-sample.sqlite3?busy_timeout=10000 scalar.db.username= scalar.db.password= `} )} Select your NewSQL database.

Run AlloyDB locally

You can run AlloyDB Omni in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start AlloyDB Omni, run the following command: docker compose up -d alloydb

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for AlloyDB in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For AlloyDB scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://${host(props, 'alloydb')}:5432/ scalar.db.username=postgres scalar.db.password=postgres `}

Run Spanner locally

You can run Spanner Omni in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start Spanner Omni, run the following command: docker compose up -d spanner spanner-init

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for Spanner in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For Spanner scalar.db.storage=jdbc scalar.db.contact_points=jdbc:cloudspanner://${host(props, 'spanner')}:15000/databases/test-db;isExperimentalHost=true;usePlainText=true scalar.db.username= scalar.db.password= `}

Run TiDB locally

You can run TiDB locally by using the TiUP tool. For installation instructions, see [Install TiUP](https://docs.pingcap.com/tidb/stable/tiup-overview/#install-tiup). To start TiDB, run the following command: tiup playground v8.5 --without-monitor

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for TiDB in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For TiDB scalar.db.storage=jdbc scalar.db.contact_points=jdbc:mysql://${props.mode === 'cluster' ? 'host.docker.internal' : 'localhost'}:4000/ scalar.db.username=root scalar.db.password= `}

Run YugabyteDB locally

You can run YugabyteDB in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start YugabyteDB, run the following command: docker compose up -d yugabyte

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for YugabyteDB in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For Yugabyte scalar.db.storage=jdbc scalar.db.contact_points=jdbc:yugabytedb://${host(props, 'yugabyte')}:5433/postgres scalar.db.username=yugabyte scalar.db.password=yugabyte `} Select your NoSQL database.

Run Cassandra locally

You can run Apache Cassandra in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start Apache Cassandra, run the following command: docker compose up -d cassandra

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for Cassandra in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For Cassandra scalar.db.storage=cassandra scalar.db.contact_points=${host(props, 'cassandra')} scalar.db.username=cassandra scalar.db.password=cassandra `} To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account).

Configure Cosmos DB for NoSQL

Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level).

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Be sure to change the values for scalar.db.contact_points and scalar.db.password as described. {dedent` # For Cosmos DB scalar.db.storage=cosmos scalar.db.contact_points= scalar.db.password= `} You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for scalar.db.password.

Run Amazon DynamoDB Local

You can run Amazon DynamoDB Local in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start Amazon DynamoDB Local, run the following command: docker compose up -d dynamodb

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for Amazon DynamoDB Local in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For DynamoDB Local scalar.db.storage=dynamo scalar.db.contact_points=sample scalar.db.username=sample scalar.db.password=sample scalar.db.dynamo.endpoint_override=http://${host(props, 'dynamodb')}:8000 `} {props.mode === 'cluster' && ( Select your object storage.

Run Azurite

You can run Azurite, which is a local emulator for Blob Storage, in Docker Compose by using the docker-compose.yml file in the scalardb-samples/{samplesDir(props)} directory. To start Blob Storage, run the following command: docker compose up -d blobstorage Then, create a container named test-container by running the following command: docker compose up blobstorage-container-creator

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Please uncomment the properties for Blob Storage in the {propertiesFile(props)} file so that the configuration looks as follows: {dedent` # For Blob Storage scalar.db.storage=blob-storage scalar.db.contact_points=http://${host(props, 'blobstorage')}:10000/test/test-container scalar.db.username=test scalar.db.password=test `}

To use Google Cloud Storage, you must have a Google Cloud account. If you don't have a Google Cloud account, visit Get started with Google Cloud.

Configure Cloud Storage

Create a Cloud Storage bucket. For instructions on creating a bucket, see Create buckets. You also need a service account key for authentication. For details, see Create a service account key.

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Be sure to change the values for scalar.db.contact_points, scalar.db.username, and scalar.db.password as described.

{dedent` # For Cloud Storage scalar.db.storage=cloud-storage scalar.db.contact_points= scalar.db.username= scalar.db.password= `} Set scalar.db.password to the full content of your Google Cloud service account key file as a single-line JSON.

To use Amazon S3, you must have an AWS account. If you don't have an AWS account, visit Create an AWS account.

Configure Amazon S3

Create an S3 bucket. For instructions on creating a bucket, see Creating a bucket. You also need an access key for authentication. For details, see Creating access keys.

{configureHeading(props)}

The {propertiesFile(props)} file in the scalardb-samples/{samplesDir(props)} directory contains database configurations. Be sure to change the values for scalar.db.contact_points, scalar.db.username, and scalar.db.password as described.

{dedent` # For S3 scalar.db.storage=s3 scalar.db.contact_points=/ scalar.db.username= scalar.db.password= `} The format of scalar.db.contact_points is <REGION>/<S3_BUCKET_NAME> (for example, us-east-1/my-bucket). )} ================================================ FILE: docs/helm-charts/configure-custom-values-envoy.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Configure a custom values file for Scalar Envoy import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CertificateManagement from '/src/components/en-us/_certificate-management.mdx'; This document explains how to create your custom values file for the Scalar Envoy chart. If you want to know the details of the parameters, please refer to the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/envoy/README.md) of the Scalar Envoy chart. ## Configure custom values for Scalar Envoy chart The Scalar Envoy chart is used via other charts (scalardb, scalardb-cluster, scalardl, and scalardl-audit), so you don't need to create a custom values file for the Scalar Envoy chart. If you want to configure Scalar Envoy, you need to add the `envoy.*` configuration to the other charts. For example, if you want to configure the Scalar Envoy for ScalarDB Server, you can configure some Scalar Envoy configurations in the custom values file of ScalarDB as follows. * Example (scalardb-custom-values.yaml) ```yaml envoy: configurationsForScalarEnvoy: ... scalardb: configurationsForScalarDB: ... ``` ## Required configurations ### Service configurations You must set `envoy.service.type` to specify the Service resource type of Kubernetes. If you accept client requests from inside of the Kubernetes cluster only (for example, if you deploy your client applications on the same Kubernetes cluster as Scalar products), you can set `envoy.service.type` to `ClusterIP`. This configuration doesn't create any load balancers provided by cloud service providers. ```yaml envoy: service: type: ClusterIP ``` If you want to use a load balancer provided by a cloud service provider to accept client requests from outside of the Kubernetes cluster, you need to set `envoy.service.type` to `LoadBalancer`. ```yaml envoy: service: type: LoadBalancer ``` If you want to configure the load balancer via annotations, you can also set annotations to `envoy.service.annotations`. ```yaml envoy: service: type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: "nlb" ``` ## Optional configurations ### Resource configurations (Recommended in the production environment) If you want to control pod resources using the requests and limits of Kubernetes, you can use `envoy.resources`. You can configure them using the same syntax as the requests and limits of Kubernetes. So, please refer to the official document [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for more details on the requests and limits of Kubernetes. ```yaml envoy: resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 2000m memory: 4Gi ``` ### Affinity configurations (Recommended in the production environment) If you want to control pod deployment using the affinity and anti-affinity of Kubernetes, you can use `envoy.affinity`. You can configure them using the same syntax as the affinity of Kubernetes. So, please refer to the official document [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) for more details on the affinity configuration of Kubernetes. ```yaml envoy: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - scalardb-cluster - key: app.kubernetes.io/app operator: In values: - envoy topologyKey: kubernetes.io/hostname weight: 50 ``` ### Prometheus and Grafana configurations (Recommended in production environments) If you want to monitor Scalar Envoy pods using [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), you can deploy a ConfigMap, a ServiceMonitor, and a PrometheusRule resource for kube-prometheus-stack using `envoy.grafanaDashboard.enabled`, `envoy.serviceMonitor.enabled`, and `envoy.prometheusRule.enabled`. ```yaml envoy: grafanaDashboard: enabled: true namespace: monitoring serviceMonitor: enabled: true namespace: monitoring interval: 15s prometheusRule: enabled: true namespace: monitoring ``` ### SecurityContext configurations (Default value is recommended) If you want to set SecurityContext and PodSecurityContext for Scalar Envoy pods, you can use `envoy.securityContext` and `envoy.podSecurityContext`. You can configure them using the same syntax as SecurityContext and PodSecurityContext of Kubernetes. So, please refer to the official document [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for more details on the SecurityContext and PodSecurityContext configurations of Kubernetes. ```yaml envoy: podSecurityContext: seccompProfile: type: RuntimeDefault securityContext: capabilities: drop: - ALL runAsNonRoot: true allowPrivilegeEscalation: false ``` ### Image configurations (Default value is recommended) If you want to change the image repository and version, you can use `envoy.image.repository` to specify the container repository information of the Scalar Envoy container image that you want to pull. ```yaml envoy: image: repository: ``` If you're using AWS, refer to [How to install Scalar products through AWS Marketplace](../scalar-kubernetes/AwsMarketplaceGuide.mdx). ### TLS configurations (optional based on your environment) You can enable TLS in: - Downstream connections between the client and Scalar Envoy. - Upstream connections between Scalar Envoy and Scalar products. #### Enable TLS in downstream connections You can enable TLS in downstream connections by using the following configurations: ```yaml envoy: tls: downstream: enabled: true ``` ##### Use your private key and certificate files You can set your private key and certificate files by using the following configurations: ```yaml envoy: tls: downstream: enabled: true certChainSecret: "envoy-tls-cert" privateKeySecret: "envoy-tls-key" ``` In this case, you have to create secret resources that include private key and certificate files for Scalar Envoy as follows, replacing the contents in the angle brackets as described: ```console kubectl create secret generic envoy-tls-cert --from-file=tls.crt=/ -n kubectl create secret generic envoy-tls-key --from-file=tls.key=/ -n ``` For more details on how to prepare private key and certificate files, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx). ##### Use a trusted CA with cert-manager to manage your private key and certificate files You can manage your private key and certificate files with cert-manager by using the following configurations, replacing the content in the angle brackets as described: :::note * If you want to use cert-manager, you must deploy cert-manager and prepare the `Issuers` resource. For details, see the cert-manager documentation, [Installation](https://cert-manager.io/docs/installation/) and [Issuer Configuration](https://cert-manager.io/docs/configuration/). * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. The default certificate configuration is recommended, but if you use a custom certificate configuration, you must satisfy the certificate requirements of Scalar products. For details, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml envoy: tls: downstream: enabled: true certManager: enabled: true issuerRef: name: dnsNames: - envoy.scalar.example.com ``` In this case, cert-manager issues your private key and certificate files by using your trusted issuer. By using cert-manager, you don't need to mount your private key and certificate files manually. ##### Use a self-signed CA with cert-manager to manage your private key and certificate files You can manage your private key and self-signed certificate files with cert-manager by using the following configurations: :::note * If you want to use cert-manager, you must deploy cert-manager. For details, see the cert-manager documentation, [Installation](https://cert-manager.io/docs/installation/). * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. The default certificate configuration is recommended, but if you use a custom certificate configuration, you must satisfy the certificate requirements of Scalar products. For details, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml envoy: tls: downstream: enabled: true certManager: enabled: true selfSigned: enabled: true dnsNames: - envoy.scalar.example.com ``` In this case, Scalar Helm Charts and cert-manager issue your private key and self-signed certificate files. You don't need to mount your private key and certificate files manually. #### Enable TLS in upstream connections You can enable TLS in upstream connections by using the following configurations: ```yaml envoy: tls: upstream: enabled: true ``` Also, you must set root CA certificate file of upstream Scalar products. To determine which approach you should take, refer to the following decision tree: ```mermaid flowchart TD A[Are you using cert-manager?] A -->|Yes| B A -->|No| D B[Are you using a self-signed CA with cert-manager?] B -->|No| C[Are you using the same trusted CA for Envoy and
upstream Scalar products with cert-manager?] C -->|No| D[You must set upstream Scalar products'
root CA certificate manually.] C ---->|Yes| E[Scalar Helm Chart automatically sets the root CA certificate. You
don't need to set `envoy.tls.upstream.caRootCertSecret` explicitly.] B ---->|Yes| E ``` ##### Set your root CA certificate file of upstream Scalar products You can set your root CA certificate file by using the following configurations: ```yaml envoy: tls: upstream: enabled: true caRootCertSecret: "envoy-upstream-scalardb-cluster-root-ca" ``` In this case, you have to create secret resources that include CA certificate files as follows. You must set the root CA certificate file based on the upstream that you use (ScalarDB Cluster, ScalarDL Ledger, or ScalarDL Auditor). Be sure to replace the contents in the angle brackets as described. ```console kubectl create secret generic envoy-upstream-scalardb-cluster-root-ca --from-file=ca.crt=/ -n ``` ```console kubectl create secret generic envoy-upstream-scalardl-ledger-root-ca --from-file=ca.crt=/ -n ``` ```console kubectl create secret generic envoy-upstream-scalardl-auditor-root-ca --from-file=ca.crt=/ -n ``` For more details on how to prepare private key and certificate files, see [How to create key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx). ##### Set custom authority for TLS communications You can set the custom authority for TLS communications by using `envoy.tls.upstream.overrideAuthority`. This value doesn't change what host is actually connected. This value is intended for testing but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set by using `scalardbCluster.tls.certChainSecret`, `ledger.tls.certChainSecret`, or `auditor.tls.certChainSecret`, depending on which product you're using. Envoy uses this value for verifying the certificate of the TLS connection with ScalarDB Cluster or ScalarDL. ```yaml envoy: tls: upstream: enabled: true overrideAuthority: "cluster.scalardb.example.com" ``` ### Replica configurations (Optional based on your environment) You can specify the number of replicas (pods) of Scalar Envoy using `envoy.replicaCount`. ```yaml envoy: replicaCount: 3 ``` ### Taint and toleration configurations (Optional based on your environment) If you want to control pod deployment by using the taints and tolerations in Kubernetes, you can use `envoy.tolerations`. You can configure taints and tolerations by using the same syntax as the tolerations in Kubernetes. For details on configuring tolerations in Kubernetes, see the official Kubernetes documentation [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). ```yaml envoy: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardb ``` ================================================ FILE: docs/helm-charts/configure-custom-values-file.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Configure a custom values file for Scalar Helm Charts When you deploy Scalar products using Scalar Helm Charts, you must prepare your custom values file based on your environment. Please refer to the following documents for more details on how to a create custom values file for each product. * [ScalarDB Cluster](configure-custom-values-scalardb-cluster.mdx) * [ScalarDL Ledger](configure-custom-values-scalardl-ledger.mdx) * [ScalarDL Auditor](configure-custom-values-scalardl-auditor.mdx) * [ScalarDL Schema Loader](configure-custom-values-scalardl-schema-loader.mdx) * [Scalar Admin for Kubernetes](configure-custom-values-scalar-admin-for-kubernetes.mdx) * [Scalar Manager](configure-custom-values-scalar-manager.mdx) * [Envoy](configure-custom-values-envoy.mdx) * [[Deprecated] ScalarDB Server](configure-custom-values-scalardb.mdx) * [[Deprecated] ScalarDB GraphQL](configure-custom-values-scalardb-graphql.mdx) ================================================ FILE: docs/helm-charts/configure-custom-values-scalar-admin-for-kubernetes.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Configure a custom values file for Scalar Admin for Kubernetes This document explains how to create your custom values file for the Scalar Admin for Kubernetes chart. For details on the parameters, see the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/scalar-admin-for-kubernetes/README.md) of the Scalar Admin for Kubernetes chart. ## Required configurations This section explains the required configurations when setting up a custom values file for Scalar Admin for Kubernetes. ### Flag configurations You must specify several flags to `scalarAdminForKubernetes.commandArgs` as an array to run Scalar Admin for Kubernetes. For more details on the flags, see [README](https://github.com/scalar-labs/scalar-admin-for-kubernetes/blob/main/README.md) of Scalar Admin for Kubernetes. ```yaml scalarAdminForKubernetes: commandArgs: - -r - - -n - - -d - - -z - ``` ## Optional configurations This section explains the optional configurations when setting up a custom values file for Scalar Admin for Kubernetes. ### CronJob configurations (optional based on your environment) By default, the Scalar Admin for Kubernetes chart creates a [Job](https://kubernetes.io/docs/concepts/workloads/controllers/job/) resource to run the Scalar Admin for Kubernetes CLI tool once. If you want to run the Scalar Admin for Kubernetes CLI tool periodically by using [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/), you can set `scalarAdminForKubernetes.jobType` to `cronjob`. Also, you can set some configurations for the CronJob resource. ```yaml scalarAdminForKubernetes: cronJob: timeZone: "Etc/UTC" schedule: "0 0 * * *" ``` ### Resource configurations (recommended in production environments) To control pod resources by using requests and limits in Kubernetes, you can use `scalarAdminForKubernetes.resources`. You can configure requests and limits by using the same syntax as requests and limits in Kubernetes. For more details on requests and limits in Kubernetes, see [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). ```yaml scalarAdminForKubernetes: resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 2000m memory: 4Gi ``` ### SecurityContext configurations (default value is recommended) To set SecurityContext and PodSecurityContext for Scalar Admin for Kubernetes pods, you can use `scalarAdminForKubernetes.securityContext` and `scalarAdminForKubernetes.podSecurityContext`. You can configure SecurityContext and PodSecurityContext by using the same syntax as SecurityContext and PodSecurityContext in Kubernetes. For more details on the SecurityContext and PodSecurityContext configurations in Kubernetes, see [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). ```yaml scalarAdminForKubernetes: podSecurityContext: seccompProfile: type: RuntimeDefault securityContext: capabilities: drop: - ALL runAsNonRoot: true allowPrivilegeEscalation: false ``` ### Image configurations (default value is recommended) If you want to change the image repository, you can use `scalarAdminForKubernetes.image.repository` to specify the container repository information of the Scalar Admin for Kubernetes image that you want to pull. ```yaml scalarAdminForKubernetes: image: repository: ``` ### Taint and toleration configurations (optional based on your environment) If you want to control pod deployment by using taints and tolerations in Kubernetes, you can use `scalarAdminForKubernetes.tolerations`. You can configure taints and tolerations by using the same syntax as the tolerations in Kubernetes. For details on configuring tolerations in Kubernetes, see the official Kubernetes documentation [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). ```yaml scalarAdminForKubernetes: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardb-cluster ``` ### TLS configurations (optional based on your environment) You can enable TLS between Scalar Admin for Kubernetes and the pause targets (ScalarDB Cluster or ScalarDL) by using the following configurations: ```yaml scalarAdminForKubernetes: commandArgs: - (omit other options) - --tls - --ca-root-cert-path - /tls/certs/ca.crt - --override-authority - cluster.scalardb.example.com ``` You can mount the `/tls/certs/ca.crt` file on a pod by using a secret resource. To mount the file, specify the name of the secret resource that includes the root CA certificate file to `scalarAdminForKubernetes.tls.caRootCertSecret` as follows: ```yaml scalarAdminForKubernetes: tls: caRootCertSecret: "scalar-admin-tls-ca" ``` In this case, you have to create a secret resource that includes the root CA certificate file for the pause targets (ScalarDB Cluster or ScalarDL) as follows: ```console kubectl create secret generic scalar-admin-tls-ca --from-file=ca.crt=/path/to/your/ca/certificate/file -n ``` ================================================ FILE: docs/helm-charts/configure-custom-values-scalar-manager.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Configure a Custom Values File for Scalar Manager This document provides instructions on how to configure a custom values file for the Scalar Manager Helm Chart. For details about the available parameters, see the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/scalar-manager/README.md) in the Scalar Manager chart repository. ## Required configurations This section describes the service, image, and Scalar Manager configurations that you must include in the Scalar Manager values file. ### Service configurations You must configure `web.service.type` to define the Kubernetes Service resource type. To use a load balancer that cloud service providers offer for exposing the service, set `web.service.type` to `LoadBalancer`. ```yaml web: service: type: LoadBalancer # other web configurations ``` #### Security considerations for exposing Scalar Manager Setting `web.service.type` to `LoadBalancer` exposes Scalar Manager externally via `HTTP` by default, which creates security risks on untrusted networks due to unencrypted traffic. If external access is not required, using a private network or properly configuring network access to your Kubernetes cluster is recommended. Scalar Manager supports authentication and authorization mechanisms. You can configure these mechanisms to ensure authorized actions for features like scheduling jobs to pause Scalar products. For details, see [Authentication configuration for Scalar Manager](#authentication-configuration-for-scalar-manager). ### Container image configurations You must configure `api.image.repository` and `web.image.repository`. These settings specify the Scalar Manager container images, ensuring you can pull them from the container repository. ```yaml api: image: repository: web: image: repository: ``` ## Optional configurations This section describes optional configurations for customizing the Scalar Manager values file. ### Scalar Manager configurations (optional based on your environment) You can override the `api.applicationProperties` setting to modify the default Scalar Manager configurations. ```yaml api: applicationProperties: | prometheus.kubernetes-service-label-name="app" prometheus.kubernetes-service-label-value="kube-prometheus-stack-prometheus" prometheus.kubernetes-service-port-name="http-web" # other application properties ``` Scalar Manager includes default configurations to discover Scalar product deployments and the Prometheus service within the cluster. In most scenarios, especially when following the guides to deploy `kube-prometheus-stack` and `loki-stack`, these default configurations are sufficient and do not require modification. #### Configurable properties in `api.applicationProperties` The configurations for Scalar Manager are in the format of Java application properties, which are `key=value` pairs. These application properties can be set by using the `api.applicationProperties` custom value in the Scalar Manager Helm Chart. | Name | Description | Default Value | |:--------------------------------------------------------------------|:----------------------------------------------------------------------------|:---------------------------------------------------------------------------| | `prometheus.kubernetes-service-label-name` | The label name used to discover the Prometheus service in Kubernetes | `app` | | `prometheus.kubernetes-service-label-value` | The label value corresponding to `prometheus.kubernetes-service-label-name` | `kube-prometheus-stack-prometheus` | | `prometheus.kubernetes-service-port-name` | The port name used to discover the Prometheus service port in Kubernetes | `http-web` | | `springdoc.swagger-ui.enabled` | Whether to enable the Swagger UI or not | `false` | | `springdoc.swagger-ui.path` | The path of the Swagger UI | `/swagger-ui.html` | | `app.cors.allowed-origins` | The allowed origins for CORS | `*` | | `app.cors.allowed-methods` | The allowed methods for CORS | `*` | | `app.cors.allowed-headers` | The allowed headers for CORS | `*` | | `authentication.providers.static-jwt.secret` | Secret key used for signing JWT tokens; minimum 32 characters | `example-jwt-secret-with-minimum-32-characters` | | `authentication.providers.static-jwt.issuer-uri` | The issuer URI of the JWT tokens | `https://scalar-manager.example.com` | | `authentication.providers.static-jwt.access-token-expiration-time` | The expiration time of the access token | `1h` | | `authentication.providers.static-jwt.refresh-token-expiration-time` | The expiration time of the refresh token | `3d` | | `app.initial-admin-user.enabled` | Whether to enable the initial admin user or not | `true` | | `app.initial-admin-user.email` | The email address of the initial admin user | `admin@example.com` | | `app.initial-admin-user.name` | The name of the initial admin user | `Administrator` | | `app.initial-admin-user.password` | The password of the initial admin user | `Password@123!` | | `spring.jpa.hibernate.ddl-auto` | The DDL mode for Hibernate | `update` | | `spring.jpa.show-sql` | Whether to show the SQL query | `false` | | `spring.jpa.properties.hibernate.format_sql` | Whether to format the SQL query | `false` | | `spring.datasource.url` | The URL of the database | `jdbc:postgresql://scalar-manager-postgres-postgresql:5432/scalar-manager` | | `spring.datasource.username` | The username of the database | `scalar-manager` | | `spring.datasource.password` | The password of the database | `scalar-manager` | | `spring.datasource.driver-class-name` | The driver class name of the database | `org.postgresql.Driver` | :::note There are more configurations that you can set in `api.applicationProperties` regarding the JPA, Hibernate, and Spring Data. If you're familiar with these configurations, you can set them to customize the database connection and the behavior of Scalar Manager. ::: ##### Authentication configuration for Scalar Manager By default, to access Scalar Manager, you need to authenticate by using a username and password. The following are the prerequisites for setting up authentication: - You need to have a PostgreSQL database, either your own or one that a cloud service provider hosts. For example, you can use the [Bitnami package for PostgreSQL](https://artifacthub.io/packages/helm/bitnami/postgresql) to deploy a PostgreSQL database in your Kubernetes cluster. - You must set the `authentication.providers.static-jwt.secret` configuration. This configuration is used for signing JWT tokens, and the minimum length of the secret is 32 characters. The following is an example of the additional configurations you need to set in the `api.applicationProperties` to apply the above prerequisites. Be sure to change the configurations to match your environment. ```properties # JWT configuration # Secret key used for signing JWT tokens, minimum 32 characters authentication.providers.static-jwt.secret=${AUTHENTICATION_PROVIDERS_STATIC_JWT_SECRET:example-jwt-secret-with-minimum-32-characters} authentication.providers.static-jwt.issuer-uri=${AUTHENTICATION_PROVIDERS_STATIC_JWT_ISSUER_URI:https://scalar-manager.example.com} authentication.providers.static-jwt.access-token-expiration-time=${AUTHENTICATION_PROVIDERS_STATIC_JWT_ACCESS_TOKEN_EXPIRATION_TIME:1h} authentication.providers.static-jwt.refresh-token-expiration-time=${AUTHENTICATION_PROVIDERS_STATIC_JWT_REFRESH_TOKEN_EXPIRATION_TIME:3d} # Initial admin configuration app.initial-admin-user.enabled=${APP_INITIAL_ADMIN_USER_ENABLED:true} app.initial-admin-user.email=${APP_INITIAL_ADMIN_USER_EMAIL:admin@example.com} app.initial-admin-user.name=${APP_INITIAL_ADMIN_USER_NAME:Administrator} app.initial-admin-user.password=${APP_INITIAL_ADMIN_USER_PASSWORD:Password@123!} # JPA configuration spring.jpa.hibernate.ddl-auto=${SPRING_JPA_HIBERNATE_DDL_AUTO:update} spring.jpa.show-sql=${SPRING_JPA_SHOW_SQL:false} spring.jpa.properties.hibernate.format_sql=${SPRING_JPA_PROPERTIES_HIBERNATE_FORMAT_SQL:false} # Database configuration spring.datasource.url=jdbc:postgresql://${DATABASE_HOST:scalar-manager-postgres-postgresql}:${DATABASE_PORT:5432}/${DATABASE_NAME:scalar-manager} spring.datasource.username=${DATABASE_USERNAME:scalar-manager} spring.datasource.password=${DATABASE_PASSWORD:scalar-manager} spring.datasource.driver-class-name=org.postgresql.Driver ``` ##### Service discovery Scalar Manager uses labels to discover the Prometheus service in Kubernetes, and then uses the port name to connect to them. You can modify the labels and the port name by setting the `prometheus.kubernetes-service-label-name`, `prometheus.kubernetes-service-label-value`, and `prometheus.kubernetes-service-port-name` configurations. In general, you don't need to modify these configurations. However, if you customized the labels or port names of the Prometheus service when installing their Helm Charts, you should adjust these configurations to match your customizations. #### Configurable environment variables in `web.env` | Name | Description | Default Value | |:---------------------|:---------------------------------------------------------|:---------------------------------------------------------------------| | `GRAFANA_SERVER_URL` | The URL of the Grafana service in the Kubernetes cluster | `http://scalar-monitoring-grafana.monitoring.svc.cluster.local:3000` | Currently, the `GRAFANA_SERVER_URL` variable can be set in `web.env` to customize the proxy from the Scalar Manager web UI to the Grafana UI. By default, the variable is set to the Grafana service `scalar-monitoring-grafana` installed in the `monitoring` namespace. If you have installed Grafana in different namespace or have changed the name of the Grafana service, you will need to update the `GRAFANA_SERVER_URL` variable accordingly. ================================================ FILE: docs/helm-charts/configure-custom-values-scalardb-analytics-server.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Configure a custom values file for ScalarDB Analytics server import CertificateManagement from '/src/components/en-us/_certificate-management.mdx'; This document explains how to create your custom values file for the ScalarDB Analytics server chart. For details on the parameters, see the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/scalardb-analytics-server/README.md) of the ScalarDB Analytics server chart. ## Required configurations This section describes the required image, database, and service configurations. ### Image configurations You must set `scalarDbAnalyticsServer.image.repository`. Be sure to specify the ScalarDB Analytics server container image so that you can pull the image from the container repository. ```yaml scalarDbAnalyticsServer: image: repository: ``` ### Database configurations You must set `scalarDbAnalyticsServer.properties`. For details about configuring the value of this parameter, see [ScalarDB Analytics server configuration](https://scalardb.scalar-labs.com/docs/latest/scalardb-analytics/configuration). ```yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.db.url=jdbc:postgresql://localhost:5432/scalardb_analytics scalar.db.analytics.server.db.username=analytics_user scalar.db.analytics.server.db.password=your_secure_password ``` ### Service configurations You must set `scalarDbAnalyticsServer.service.type` to specify the Service resource type of Kubernetes. If the ScalarDB Analytics server accepts client requests from inside of the Kubernetes cluster only (for example, if you deploy your client applications on the same Kubernetes cluster as Scalar products), you can set `scalarDbAnalyticsServer.service.type` to `ClusterIP`. This configuration doesn't create any load balancers provided by cloud service providers. ```yaml scalarDbAnalyticsServer: service: type: ClusterIP ``` If you want to use a load balancer provided by a cloud service provider to accept client requests from outside of the Kubernetes cluster, you need to set `scalarDbAnalyticsServer.service.type` to `LoadBalancer`. ```yaml scalarDbAnalyticsServer: service: type: LoadBalancer ``` If you want to configure the load balancer via annotations, you can also set annotations to `scalarDbAnalyticsServer.service.annotations`. ```yaml scalarDbAnalyticsServer: service: type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: "nlb" ``` ## Optional configurations This section describes the optional configurations. ### Secret configurations (recommended in production environments) To use environment variables to set some properties (for example, credentials), you can use `scalarDbAnalyticsServer.secretName` to specify the Secret resource that includes some credentials. For example, you can set credentials for a backend database (`scalar.db.analytics.server.db.username` and `scalar.db.analytics.server.db.password`) by using environment variables, which makes your pods more secure. ```yaml scalarDbAnalyticsServer: secretName: "scalardb-analytics-server-credentials-secret" ``` :::tip The ScalarDB Analytics server automatically loads configurations from specific environment variables. The naming rule for the environment variables is as follows: - Capitalize all characters of the property name. - Replace periods (`.`) with underscores (`_`). For example, if you want to set `scalar.db.analytics.server.db.username` and `scalar.db.analytics.server.db.password` via environment variables, you must set environment variables `SCALAR_DB_ANALYTICS_SERVER_DB_USERNAME` and `SCALAR_DB_ANALYTICS_SERVER_DB_PASSWORD`. In this case, you don't need to set `scalar.db.analytics.server.db.username` and `scalar.db.analytics.server.db.password` in `scalarDbAnalyticsServer.properties`. Setting only the environment variables is enough. For example, you can create such a secret resource that includes `SCALAR_DB_ANALYTICS_SERVER_DB_USERNAME` and `SCALAR_DB_ANALYTICS_SERVER_DB_PASSWORD` as follows: ```console kubectl create secret generic scalardb-analytics-server-credentials-secret \ --from-literal=SCALAR_DB_ANALYTICS_SERVER_DB_USERNAME=analytics_user \ --from-literal=SCALAR_DB_ANALYTICS_SERVER_DB_PASSWORD=your_secure_password ``` ::: ### SecurityContext configurations (the default value is recommended) To set SecurityContext and PodSecurityContext for ScalarDB Analytics server pods, you can use `scalarDbAnalyticsServer.securityContext` and `scalarDbAnalyticsServer.podSecurityContext`. You can configure SecurityContext and PodSecurityContext by using the same syntax as SecurityContext and PodSecurityContext in Kubernetes. For more details on the SecurityContext and PodSecurityContext configurations in Kubernetes, see [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). ```yaml scalarDbAnalyticsServer: podSecurityContext: seccompProfile: type: RuntimeDefault securityContext: capabilities: drop: - ALL runAsNonRoot: true allowPrivilegeEscalation: false ``` ### TLS configurations (optional based on your environment) You can enable TLS in: - The communications between the ScalarDB Analytics server and its client. #### Enable TLS You can enable TLS in all ScalarDB Analytics server connections by using the following configurations: ```yaml scalarDbAnalyticsServer: properties: | ...(omit)... scalar.db.analytics.server.tls.enabled=true scalar.db.analytics.server.tls.cert_chain_path=/tls/scalardb-analytics-server/certs/tls.crt scalar.db.analytics.server.tls.private_key_path=/tls/scalardb-analytics-server/certs/tls.key tls: enabled: true ``` :::note Based on the specification of the private key and certificate that are created by cert-manager and the specification of this chart, you must set the fixed file path and file name when you enable the TLS feature. Please set the above file paths and file names as is for `scalar.db.analytics.server.tls.cert_chain_path` and `scalar.db.analytics.server.tls.private_key_path`. ::: ##### Use your private key and certificate files You can set your private key and certificate files by using the following configurations: ```yaml scalarDbAnalyticsServer: tls: enabled: true caRootCertSecret: "scalardb-analytics-server-tls-ca" certChainSecret: "scalardb-analytics-server-tls-cert" privateKeySecret: "scalardb-analytics-server-tls-key" ``` In this case, you have to create secret resources that include private key and certificate files for the ScalarDB Analytics server as follows, replacing the contents in the angle brackets as described: ```console kubectl create secret generic scalardb-analytics-server-tls-ca --from-file=ca.crt= -n kubectl create secret generic scalardb-analytics-server-tls-cert --from-file=tls.crt= -n kubectl create secret generic scalardb-analytics-server-tls-key --from-file=tls.key= -n ``` For more details on how to prepare private key and certificate files, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx). ##### Use a trusted CA with cert-manager to manage your private key and certificate files You can manage your private key and certificate files with cert-manager by using the following configurations, replacing the content in the angle brackets as described: :::note * If you want to use cert-manager, you must deploy cert-manager and prepare the `Issuers` resource. For details, see the cert-manager documentation, [Installation](https://cert-manager.io/docs/installation/) and [Issuer Configuration](https://cert-manager.io/docs/configuration/). * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. The default certificate configuration is recommended, but if you use a custom certificate configuration, you must satisfy the certificate requirements of Scalar products. For details, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml scalarDbAnalyticsServer: tls: enabled: true certManager: enabled: true issuerRef: name: dnsNames: - server.analytics.scalardb.example.com ``` In this case, cert-manager issues your private key and certificate files by using your trusted issuer. You don't need to mount your private key and certificate files manually. ##### Use a self-signed CA with cert-manager to manage your private key and certificate files You can manage your private key and self-signed certificate files with cert-manager by using the following configurations: :::note * If you want to use cert-manager, you must deploy cert-manager. For more details on how to deploy cert-manager, see [Installation](https://cert-manager.io/docs/installation/) in the official documentation for cert-manager. * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. The default certificate configuration is recommended, but if you use a custom certificate configuration, you must satisfy the certificate requirements of Scalar products. See [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml scalarDbAnalyticsServer: tls: enabled: true certManager: enabled: true selfSigned: enabled: true dnsNames: - server.analytics.scalardb.example.com ``` In this case, Scalar Helm Charts and cert-manager issue your private key and self-signed certificate files. You don't need to mount your private key and certificate files manually. ##### Set custom authority for TLS communications You can set the custom authority for TLS communications by using `scalarDbAnalyticsServer.tls.overrideAuthority`. This value doesn't change what host is actually connected. This value is intended for testing but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set by using `scalarDbAnalyticsServer.tls.certChainSecret`. This chart uses this value for health check requests (`startupProbe` and `livenessProbe`). ```yaml scalarDbAnalyticsServer: tls: enabled: true overrideAuthority: "server.analytics.scalardb.example.com" ``` ### Affinity configurations (optional based on your environment) To control pod deployment by using affinity and anti-affinity in Kubernetes, you can use `scalarDbAnalyticsServer.affinity`. You can configure affinity and anti-affinity by using the same syntax for affinity and anti-affinity in Kubernetes. For more details on configuring affinity in Kubernetes, see [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). ```yaml scalarDbAnalyticsServer: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - scalardb-analytics-server - key: app.kubernetes.io/app operator: In values: - scalardb-analytics-server topologyKey: kubernetes.io/hostname weight: 50 ``` ### Taint and toleration configurations (optional based on your environment) If you want to control pod deployment by using the taints and tolerations in Kubernetes, you can use `scalarDbAnalyticsServer.tolerations`. You can configure taints and tolerations by using the same syntax as the tolerations in Kubernetes. For details on configuring tolerations in Kubernetes, see the official Kubernetes documentation [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). ```yaml scalarDbAnalyticsServer: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardb-analytics-server ``` ================================================ FILE: docs/helm-charts/configure-custom-values-scalardb-cluster.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Configure a custom values file for ScalarDB Cluster import CertificateManagement from '/src/components/en-us/_certificate-management.mdx'; This document explains how to create your custom values file for the ScalarDB Cluster chart. For details on the parameters, see the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/scalardb-cluster/README.md) of the ScalarDB Cluster chart. ## Required configurations ### Image configurations You must set `scalardbCluster.image.repository`. Be sure to specify the ScalarDB Cluster container image so that you can pull the image from the container repository. ```yaml scalardbCluster: image: repository: ``` ### Database configurations You must set `scalardbCluster.scalardbClusterNodeProperties`. Please set `scalardb-cluster-node.properties` to this parameter. For more details on the configurations of ScalarDB Cluster, see [ScalarDB Cluster Configurations](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/scalardb-cluster-configurations/). ```yaml scalardbCluster: scalardbClusterNodeProperties: | scalar.db.cluster.membership.type=KUBERNETES scalar.db.cluster.membership.kubernetes.endpoint.namespace_name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAMESPACE_NAME} scalar.db.cluster.membership.kubernetes.endpoint.name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAME} scalar.db.contact_points=localhost scalar.db.username=${env:SCALAR_DB_USERNAME} scalar.db.password=${env:SCALAR_DB_PASSWORD} scalar.db.storage=cassandra ``` Note that you must always set the following three properties if you deploy ScalarDB Cluster in a Kubernetes environment by using Scalar Helm Chart. These properties are fixed values. Since the properties don't depend on individual environments, you can set the same values by copying the following values and pasting them in `scalardbCluster.scalardbClusterNodeProperties`. ```yaml scalardbCluster: scalardbClusterNodeProperties: | scalar.db.cluster.membership.type=KUBERNETES scalar.db.cluster.membership.kubernetes.endpoint.namespace_name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAMESPACE_NAME} scalar.db.cluster.membership.kubernetes.endpoint.name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAME} ``` ## Optional configurations ### Resource configurations (recommended in production environments) To control pod resources by using requests and limits in Kubernetes, you can use `scalardbCluster.resources`. Note that it is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. Also, if you use the pay-as-you-go (PAYG) containers that the AWS Marketplace provides, you will not be able to run any containers that exceed the 2vCPU / 4GB memory configuration in `resources.limits`. If you exceed this resource limitation, the pods will automatically stop. You can configure requests and limits by using the same syntax as requests and limits in Kubernetes. For more details on requests and limits in Kubernetes, see [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/). ```yaml scalardbCluster: resources: requests: cpu: 2000m memory: 4Gi limits: cpu: 2000m memory: 4Gi ``` ### Secret configurations (recommended in production environments) To use environment variables to set some properties (e.g., credentials) in `scalardbCluster.scalardbClusterNodeProperties`, you can use `scalardbCluster.secretName` to specify the Secret resource that includes some credentials. For example, you can set credentials for a backend database (`scalar.db.username` and `scalar.db.password`) by using environment variables, which makes your pods more secure. For more details on how to use a Secret resource, see [How to use Secret resources to pass the credentials as the environment variables into the properties file](use-secret-for-credentials.mdx). ```yaml scalardbCluster: secretName: "scalardb-cluster-credentials-secret" ``` ### Affinity configurations (recommended in production environments) To control pod deployment by using affinity and anti-affinity in Kubernetes, you can use `scalardbCluster.affinity`. You can configure affinity and anti-affinity by using the same syntax for affinity and anti-affinity in Kubernetes. For more details on configuring affinity in Kubernetes, see [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). ```yaml scalardbCluster: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - scalardb-cluster - key: app.kubernetes.io/app operator: In values: - scalardb-cluster topologyKey: kubernetes.io/hostname weight: 50 ``` ### Prometheus and Grafana configurations (recommended in production environments) To monitor ScalarDB Cluster pods by using [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), you can set `scalardbCluster.grafanaDashboard.enabled`, `scalardbCluster.serviceMonitor.enabled`, and `scalardbCluster.prometheusRule.enabled` to `true`. When you set these configurations to `true`, the chart deploys the necessary resources and kube-prometheus-stack starts monitoring automatically. ```yaml scalardbCluster: grafanaDashboard: enabled: true namespace: monitoring serviceMonitor: enabled: true namespace: monitoring interval: 15s prometheusRule: enabled: true namespace: monitoring ``` ### SecurityContext configurations (default value is recommended) To set SecurityContext and PodSecurityContext for ScalarDB Cluster pods, you can use `scalardbCluster.securityContext` and `scalardbCluster.podSecurityContext`. You can configure SecurityContext and PodSecurityContext by using the same syntax as SecurityContext and PodSecurityContext in Kubernetes. For more details on the SecurityContext and PodSecurityContext configurations in Kubernetes, see [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). ```yaml scalardbCluster: podSecurityContext: seccompProfile: type: RuntimeDefault securityContext: capabilities: drop: - ALL runAsNonRoot: true allowPrivilegeEscalation: false ``` ### TLS configurations (optional based on your environment) You can enable TLS in: - The communications between the ScalarDB Cluster node and clients. - The communications between all ScalarDB Cluster nodes (the cluster's internal communications). #### Enable TLS You can enable TLS in all ScalarDB Cluster connections by using the following configurations: ```yaml scalardbCluster: scalardbClusterNodeProperties: | ...(omit)... scalar.db.cluster.tls.enabled=true scalar.db.cluster.tls.ca_root_cert_path=/tls/scalardb-cluster/certs/ca.crt scalar.db.cluster.node.tls.cert_chain_path=/tls/scalardb-cluster/certs/tls.crt scalar.db.cluster.node.tls.private_key_path=/tls/scalardb-cluster/certs/tls.key scalar.db.cluster.tls.override_authority= tls: enabled: true ``` ##### Use your private key and certificate files You can set your private key and certificate files by using the following configurations: ```yaml scalardbCluster: tls: enabled: true caRootCertSecret: "scalardb-cluster-tls-ca" certChainSecret: "scalardb-cluster-tls-cert" privateKeySecret: "scalardb-cluster-tls-key" ``` In this case, you have to create secret resources that include private key and certificate files for ScalarDB Cluster as follows, replacing the contents in the angle brackets as described: ```console kubectl create secret generic scalardb-cluster-tls-ca --from-file=ca.crt=/ -n kubectl create secret generic scalardb-cluster-tls-cert --from-file=tls.crt=/ -n kubectl create secret generic scalardb-cluster-tls-key --from-file=tls.key=/ -n ``` For more details on how to prepare private key and certificate files, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx). ##### Use a trusted CA with cert-manager to manage your private key and certificate files You can manage your private key and certificate files with cert-manager by using the following configurations, replacing the content in the angle brackets as described: :::note * If you want to use cert-manager, you must deploy cert-manager and prepare the `Issuers` resource. For details, see the cert-manager documentation, [Installation](https://cert-manager.io/docs/installation/) and [Issuer Configuration](https://cert-manager.io/docs/configuration/). * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. The default certificate configuration is recommended, but if you use a custom certificate configuration, you must satisfy the certificate requirements of Scalar products. For details, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml scalardbCluster: tls: enabled: true certManager: enabled: true issuerRef: name: dnsNames: - cluster.scalardb.example.com ``` In this case, cert-manager issues your private key and certificate files by using your trusted issuer. You don't need to mount your private key and certificate files manually. ##### Use a self-signed CA with cert-manager to manage your private key and certificate files You can manage your private key and self-signed certificate files with cert-manager by using the following configurations: :::note * If you want to use cert-manager, you must deploy cert-manager. For more details on how to deploy cert-manager, see the [Installation](https://cert-manager.io/docs/installation/) in the cert-manager official document. * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. We recommend the default certificate configuration, but if you custom certificate configuration, you must satisfy the certificate requirements of Scalar products. See [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml scalardbCluster: tls: enabled: true certManager: enabled: true selfSigned: enabled: true dnsNames: - cluster.scalardb.example.com ``` In this case, Scalar Helm Charts and cert-manager issue your private key and self-signed certificate files. You don't need to mount your private key and certificate files manually. ##### Set custom authority for TLS communications You can set the custom authority for TLS communications by using `scalardbCluster.tls.overrideAuthority`. This value doesn't change what host is actually connected. This value is intended for testing but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set by using `scalardbCluster.tls.certChainSecret`. This chart uses this value for `startupProbe` and `livenessProbe`. ```yaml scalardbCluster: tls: enabled: true overrideAuthority: "cluster.scalardb.example.com" ``` ##### Set a root CA certificate for Prometheus Operator If you set `scalardbCluster.serviceMonitor.enabled=true` and `scalardbCluster.tls.enabled=true` (in other words, if you monitor ScalarDB Cluster with TLS configuration by using Prometheus Operator), you must set the secret name to `scalardbCluster.tls.caRootCertSecretForServiceMonitor`. ```yaml scalardbCluster: tls: enabled: true caRootCertSecretForServiceMonitor: "scalardb-cluster-tls-ca-for-prometheus" ``` In this case, you have to create secret resources that include a root CA certificate for ScalarDB Cluster in the same namespace as Prometheus as follows: ```console kubectl create secret generic scalardb-cluster-tls-ca-for-prometheus --from-file=ca.crt=/path/to/your/ca/certificate/file -n ``` ### Replica configurations (optional based on your environment) You can specify the number of ScalarDB Cluster replicas (pods) by using `scalardbCluster.replicaCount`. ```yaml scalardbCluster: replicaCount: 3 ``` ### Logging configurations (optional based on your environment) To change the ScalarDB Cluster log level, you can use `scalardbCluster.logLevel`. ```yaml scalardbCluster: logLevel: INFO ``` ### GraphQL configurations (optional based on your environment) To use the GraphQL feature in ScalarDB Cluster, you can set `scalardbCluster.graphql.enabled` to `true` to deploy some resources for the GraphQL feature. Note that you also need to set `scalar.db.graphql.enabled=true` in `scalardbCluster.scalardbClusterNodeProperties` when using the GraphQL feature. ```yaml scalardbCluster: graphql: enabled: true ``` Also, you can configure the `Service` resource that accepts GraphQL requests from clients. ```yaml scalardbCluster: graphql: service: type: ClusterIP annotations: {} ports: graphql: port: 8080 targetPort: 8080 protocol: TCP ``` ### SQL configurations (optional based on your environment) To use the SQL feature in ScalarDB Cluster, there is no configuration necessary for custom values files. You can use the feature by setting `scalar.db.sql.enabled=true` in `scalardbCluster.scalardbClusterNodeProperties`. ### Scalar Envoy configurations (optional based on your environment) To use ScalarDB Cluster with `indirect` mode, you must enable Envoy as follows. ```yaml envoy: enabled: true ``` Also, you must set the Scalar Envoy configurations in the custom values file for ScalarDB Cluster. This is because clients need to send requests to ScalarDB Cluster via Scalar Envoy as the load balancer of gRPC requests if you deploy ScalarDB Cluster in a Kubernetes environment with `indirect` mode. For more details on Scalar Envoy configurations, see [Configure a custom values file for Scalar Envoy](configure-custom-values-envoy.mdx). ```yaml envoy: configurationsForScalarEnvoy: ... scalardbCluster: configurationsForScalarDbCluster: ... ``` ### Taint and toleration configurations (optional based on your environment) If you want to control pod deployment by using the taints and tolerations in Kubernetes, you can use `scalardbCluster.tolerations`. You can configure taints and tolerations by using the same syntax as the tolerations in Kubernetes. For details on configuring tolerations in Kubernetes, see the official Kubernetes documentation [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). ```yaml scalardbCluster: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardb-cluster ``` ### Encryption configurations (optional based on your environment) You can enable [encryption at rest](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/encrypt-data-at-rest/) to protect the data in the backend databases. When you use the encryption feature, you have the following two deployment options: 1. Use HashiCorp Vault (HashiCorp Cloud Platform (HCP) Vault Dedicated) to manage and store the DEKs. 1. Use ScalarDB Cluster to manage the DEK, and store it in Kubernetes Secrets. #### Use HashiCorp Vault You can use HashiCorp Vault (HCP Vault Dedicated) to encrypt data as follows, replacing the contents in the angle brackets as described: ```yaml scalardbCluster: scalardbClusterNodeProperties: | ...(omit)... scalar.db.cluster.encryption.enabled=true scalar.db.cluster.encryption.type=vault scalar.db.cluster.encryption.vault.address=https://: scalar.db.cluster.encryption.vault.token= scalar.db.cluster.encryption.vault.transit_secrets_engine_path= encryption: enabled: true type: "vault" ``` #### Use ScalarDB Cluster and Kubernetes Secrets You can use ScalarDB Cluster and Kubernetes Secrets to encrypt data as follows, replacing the contents in the angle brackets as described: ```yaml scalardbCluster: scalardbClusterNodeProperties: | ...(omit)... scalar.db.cluster.encryption.enabled=true scalar.db.cluster.encryption.type=self scalar.db.cluster.encryption.self.kubernetes.secret.namespace_name=${env:SCALAR_DB_CLUSTER_ENCRYPTION_SELF_KUBERNETES_SECRET_NAMESPACE_NAME} encryption: enabled: true type: "self" ``` In this case, you don't need to replace `${env:SCALAR_DB_CLUSTER_ENCRYPTION_SELF_KUBERNETES_SECRET_NAMESPACE_NAME}` since the Helm Chart for ScalarDB Cluster automatically sets the namespace information as an environment variable. Because of this, you can keep the value `${env:SCALAR_DB_CLUSTER_ENCRYPTION_SELF_KUBERNETES_SECRET_NAMESPACE_NAME}` as is. ================================================ FILE: docs/helm-charts/configure-custom-values-scalardb-graphql.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # [Deprecated] Configure a custom values file for ScalarDB GraphQL :::note ScalarDB GraphQL Server is now deprecated. Please use [ScalarDB Cluster](configure-custom-values-scalardb-cluster.mdx) instead. ::: This document explains how to create your custom values file for the ScalarDB GraphQL chart. If you want to know the details of the parameters, please refer to the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/scalardb-graphql/README.md) of the ScalarDB GraphQL chart. ## Required configurations ### Ingress configuration You must set `ingress` to listen the client requests. When you deploy multiple GraphQL servers, session affinity is required to handle transactions properly. This is because GraphQL servers keep the transactions in memory, so GraphQL queries that use continued transactions must be routed to the same server that started the transaction. For example, if you use NGINX Ingress Controller, you can set ingress configurations as follows. ```yaml ingress: enabled: true className: nginx annotations: nginx.ingress.kubernetes.io/session-cookie-path: / nginx.ingress.kubernetes.io/affinity: cookie nginx.ingress.kubernetes.io/session-cookie-name: INGRESSCOOKIE nginx.ingress.kubernetes.io/session-cookie-hash: sha1 nginx.ingress.kubernetes.io/session-cookie-max-age: "300" hosts: - host: "" paths: - path: /graphql pathType: Exact ``` If you use ALB of AWS, you can set ingress configurations as follows. ```yaml ingress: enabled: true className: alb annotations: alb.ingress.kubernetes.io/scheme: internal alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60 alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/healthcheck-path: /graphql?query=%7B__typename%7D hosts: - host: "" paths: - path: /graphql pathType: Exact ``` ### Image configurations You must set `image.repository`. Be sure to specify the ScalarDB GraphQL container image so that you can pull the image from the container repository. ```yaml image: repository: ``` If you're using AWS, refer to [How to install Scalar products through AWS Marketplace](../scalar-kubernetes/AwsMarketplaceGuide.mdx). ### Database configurations You must set `scalarDbGraphQlConfiguration`. If you use ScalarDB Server with ScalarDB GraphQL (recommended), you must set the configuration to access the ScalarDB Server pods. ```yaml scalarDbGraphQlConfiguration: contactPoints: contactPort: 60051 storage: "grpc" transactionManager: "grpc" namespaces: ``` ## Optional configurations ### Resource configurations (Recommended in the production environment) If you want to control pod resources using the requests and limits of Kubernetes, you can use `resources`. Note that it is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. Also, if you use the pay-as-you-go (PAYG) containers that the AWS Marketplace provides, you will not be able to run any containers that exceed the 2vCPU / 4GB memory configuration in `resources.limits`. If you exceed this resource limitation, the pods will automatically stop. You can configure them using the same syntax as the requests and limits of Kubernetes. So, please refer to the official document [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for more details on the requests and limits of Kubernetes. ```yaml resources: requests: cpu: 2000m memory: 4Gi limits: cpu: 2000m memory: 4Gi ``` ### Affinity configurations (Recommended in the production environment) If you want to control pod deployment using the affinity and anti-affinity of Kubernetes, you can use `affinity`. You can configure them using the same syntax as the affinity of Kubernetes. So, please refer to the official document [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) for more details on the affinity configuration of Kubernetes. ```yaml affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/app operator: In values: - scalardb-graphql topologyKey: kubernetes.io/hostname weight: 50 ``` ### Prometheus/Grafana configurations (Recommended in the production environment) If you want to monitor ScalarDB GraphQL pods using [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), you can deploy a ConfigMap, a ServiceMonitor, and a PrometheusRule resource for kube-prometheus-stack using `grafanaDashboard.enabled`, `serviceMonitor.enabled`, and `prometheusRule.enabled`. ```yaml grafanaDashboard: enabled: true namespace: monitoring serviceMonitor: enabled: true namespace: monitoring interval: 15s prometheusRule: enabled: true namespace: monitoring ``` ### SecurityContext configurations (Default value is recommended) If you want to set SecurityContext and PodSecurityContext for ScalarDB GraphQL pods, you can use `securityContext` and `podSecurityContext`. You can configure them using the same syntax as SecurityContext and PodSecurityContext of Kubernetes. So, please refer to the official document [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for more details on the SecurityContext and PodSecurityContext configurations of Kubernetes. ```yaml podSecurityContext: seccompProfile: type: RuntimeDefault securityContext: capabilities: drop: - ALL runAsNonRoot: true allowPrivilegeEscalation: false ``` ### GraphQL Server configurations (Optional based on your environment) If you want to change the path to run the graphql queries, you can use `scalarDbGraphQlConfiguration.path`. By default, you can run the graphql queries using `http://:80/graphql`. You can also enable/disable [GraphiQL](https://github.com/graphql/graphiql/tree/main/packages/graphiql) using `scalarDbGraphQlConfiguration.graphiql`. ```yaml scalarDbGraphQlConfiguration: path: /graphql graphiql: "true" ``` ### TLS configurations (Optional based on your environment) If you want to use TLS between the client and the ingress, you can use `ingress.tls`. You must create a Secret resource that includes a secret key and a certificate file. Please refer to the official document [Ingress - TLS](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls) for more details on the Secret resource for Ingress. ```yaml ingress: tls: - hosts: - foo.example.com - bar.example.com - bax.example.com secretName: graphql-ingress-tls ``` ### Replica configurations (Optional based on your environment) You can specify the number of replicas (pods) of ScalarDB GraphQL using `replicaCount`. ```yaml replicaCount: 3 ``` ### Logging configurations (Optional based on your environment) If you want to change the log level of ScalarDB GraphQL, you can use `scalarDbGraphQlConfiguration.logLevel`. ```yaml scalarDbGraphQlConfiguration: logLevel: INFO ``` ### Taint and toleration configurations (Optional based on your environment) If you want to control pod deployment by using the taints and tolerations in Kubernetes, you can use `tolerations`. You can configure taints and tolerations by using the same syntax as the tolerations in Kubernetes. For details on configuring tolerations in Kubernetes, see the official Kubernetes documentation [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). ```yaml tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardb ``` ================================================ FILE: docs/helm-charts/configure-custom-values-scalardb.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Deprecated displayed_sidebar: docsEnglish --- # [Deprecated] Configure a custom values file for ScalarDB Server :::note ScalarDB Server is now deprecated. Please use [ScalarDB Cluster](configure-custom-values-scalardb-cluster.mdx) instead. ::: This document explains how to create your custom values file for the ScalarDB Server chart. If you want to know the details of the parameters, please refer to the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/scalardb/README.md) of the ScalarDB Server chart. ## Required configurations ### Scalar Envoy configurations You must set the Scalar Envoy configurations in the custom values file for ScalarDB Server. This is because client requests are sent to ScalarDB Server via Scalar Envoy as the load balancer of gRPC requests if you deploy ScalarDB Server on a Kubernetes environment. Please refer to the document [Configure a custom values file for Scalar Envoy](configure-custom-values-envoy.mdx) for more details on the Scalar Envoy configurations. ```yaml envoy: configurationsForScalarEnvoy: ... scalardb: configurationsForScalarDB: ... ``` ### Image configurations You must set `scalardb.image.repository`. Be sure to specify the ScalarDB Server container image so that you can pull the image from the container repository. ```yaml scalardb: image: repository: ``` If you're using AWS, refer to [How to install Scalar products through AWS Marketplace](../scalar-kubernetes/AwsMarketplaceGuide.mdx). ### Database configurations You must set `scalardb.databaseProperties`. Please set your `database.properties` to this parameter. Please refer to the [Configure ScalarDB Server](https://scalardb.scalar-labs.com/docs/latest/scalardb-server#configure-scalardb-server) for more details on the configuration of ScalarDB Server. ```yaml scalardb: databaseProperties: | scalar.db.server.port=60051 scalar.db.server.prometheus_exporter_port=8080 scalar.db.server.grpc.max_inbound_message_size= scalar.db.server.grpc.max_inbound_metadata_size= scalar.db.contact_points=localhost scalar.db.username=cassandra scalar.db.password=cassandra scalar.db.storage=cassandra scalar.db.transaction_manager=consensus-commit scalar.db.consensus_commit.isolation_level=SNAPSHOT scalar.db.consensus_commit.serializable_strategy= scalar.db.consensus_commit.include_metadata.enabled=false ``` ## Optional configurations ### Resource configurations (Recommended in the production environment) If you want to control pod resources using the requests and limits of Kubernetes, you can use `scalardb.resources`. Note that it is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. Also, if you use the pay-as-you-go (PAYG) containers that the AWS Marketplace provides, you will not be able to run any containers that exceed the 2vCPU / 4GB memory configuration in `resources.limits`. If you exceed this resource limitation, the pods will automatically stop. You can configure them using the same syntax as the requests and limits of Kubernetes. So, please refer to the official document [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for more details on the requests and limits of Kubernetes. ```yaml scalardb: resources: requests: cpu: 2000m memory: 4Gi limits: cpu: 2000m memory: 4Gi ``` ### Secret configurations (Recommended in the production environment) If you want to use environment variables to set some properties (e.g., credentials) in the `scalardb.databaseProperties`, you can use `scalardb.secretName` to specify the Secret resource that includes some credentials. For example, you can set credentials for a backend database (`scalar.db.username` and `scalar.db.password`) using environment variables, which makes your pods more secure. Please refer to the document [How to use Secret resources to pass the credentials as the environment variables into the properties file](use-secret-for-credentials.mdx) for more details on how to use a Secret resource. ```yaml scalardb: secretName: "scalardb-credentials-secret" ``` ### Affinity configurations (Recommended in the production environment) If you want to control pod deployment using the affinity and anti-affinity of Kubernetes, you can use `scalardb.affinity`. You can configure them using the same syntax as the affinity of Kubernetes. So, please refer to the official document [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) for more details on the affinity configuration of Kubernetes. ```yaml scalardb: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - scalardb - key: app.kubernetes.io/app operator: In values: - scalardb topologyKey: kubernetes.io/hostname weight: 50 ``` ### Prometheus/Grafana configurations (Recommended in the production environment) If you want to monitor ScalarDB Server pods using [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), you can deploy a ConfigMap, a ServiceMonitor, and a PrometheusRule resource for kube-prometheus-stack using `scalardb.grafanaDashboard.enabled`, `scalardb.serviceMonitor.enabled`, and `scalardb.prometheusRule.enabled`. ```yaml scalardb: grafanaDashboard: enabled: true namespace: monitoring serviceMonitor: enabled: true namespace: monitoring interval: 15s prometheusRule: enabled: true namespace: monitoring ``` ### SecurityContext configurations (Default value is recommended) If you want to set SecurityContext and PodSecurityContext for ScalarDB Server pods, you can use `scalardb.securityContext` and `scalardb.podSecurityContext`. You can configure them using the same syntax as SecurityContext and PodSecurityContext of Kubernetes. So, please refer to the official document [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for more details on the SecurityContext and PodSecurityContext configurations of Kubernetes. ```yaml scalardb: podSecurityContext: seccompProfile: type: RuntimeDefault securityContext: capabilities: drop: - ALL runAsNonRoot: true allowPrivilegeEscalation: false ``` ### Replica configurations (Optional based on your environment) You can specify the number of replicas (pods) of ScalarDB Server using `scalardb.replicaCount`. ```yaml scalardb: replicaCount: 3 ``` ### Logging configurations (Optional based on your environment) If you want to change the log level of ScalarDB Server, you can use `scalardb.storageConfiguration.dbLogLevel`. ```yaml scalardb: storageConfiguration: dbLogLevel: INFO ``` ### Taint and toleration configurations (Optional based on your environment) If you want to control pod deployment by using the taints and tolerations in Kubernetes, you can use `scalardb.tolerations`. You can configure taints and tolerations by using the same syntax as the tolerations in Kubernetes. For details on configuring tolerations in Kubernetes, see the official Kubernetes documentation [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). ```yaml scalardb: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardb ``` ================================================ FILE: docs/helm-charts/configure-custom-values-scalardl-auditor.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Configure a custom values file for ScalarDL Auditor import CertificateManagement from '/src/components/en-us/_certificate-management.mdx'; This document explains how to create your custom values file for the ScalarDL Auditor chart. If you want to know the details of the parameters, please refer to the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/scalardl-audit/README.md) of the ScalarDL Auditor chart. ## Required configurations ### Scalar Envoy configurations You must set the Scalar Envoy configurations in the custom values file for ScalarDL Auditor. This is because client requests are sent to ScalarDL Auditor via Scalar Envoy as the load balancer of gRPC requests if you deploy ScalarDL Auditor on a Kubernetes environment. Please refer to the document [Configure a custom values file for Scalar Envoy](configure-custom-values-envoy.mdx) for more details on the Scalar Envoy configurations. ```yaml envoy: configurationsForScalarEnvoy: ... auditor: configurationsForScalarDLAuditor: ... ``` ### Image configurations You must set `auditor.image.repository`. Be sure to specify the ScalarDL Auditor container image so that you can pull the image from the container repository. ```yaml auditor: image: repository: ``` For more details on the container repository for Scalar products, see [How to get the container images of Scalar products](../scalar-kubernetes/HowToGetContainerImages.mdx). ### Auditor/Database configurations You must set `auditor.auditorProperties`. Please set your `auditor.properties` to this parameter. Please refer to the [auditor.properties](https://github.com/scalar-labs/scalar/blob/master/auditor/conf/auditor.properties) for more details on the configuration of ScalarDL Auditor. ```yaml auditor: auditorProperties: | scalar.db.contact_points=localhost scalar.db.username=cassandra scalar.db.password=cassandra scalar.db.storage=cassandra scalar.dl.auditor.ledger.host= scalar.dl.auditor.private_key_path=/keys/auditor-key-file ``` ### Private key configurations You must set a private key file to `scalar.dl.auditor.private_key_path`. You must also mount the private key file on the ScalarDL Auditor pod. For more details on how to mount the private key file, refer to [Mount a private key file on a pod in ScalarDL Helm Charts](mount-files-or-volumes-on-scalar-pods.mdx#mount-a-private-key-file-on-a-pod-in-scalardl-helm-charts). ## Optional configurations ### Resource configurations (Recommended in the production environment) If you want to control pod resources using the requests and limits of Kubernetes, you can use `auditor.resources`. Note that it is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. Also, if you use the pay-as-you-go (PAYG) containers that the AWS Marketplace provides, you will not be able to run any containers that exceed the 2vCPU / 4GB memory configuration in `resources.limits`. If you exceed this resource limitation, the pods will automatically stop. You can configure them using the same syntax as the requests and limits of Kubernetes. So, please refer to the official document [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for more details on the requests and limits of Kubernetes. ```yaml auditor: resources: requests: cpu: 2000m memory: 4Gi limits: cpu: 2000m memory: 4Gi ``` ### Secret configurations If you want to use environment variables to set some properties (e.g., credentials) in the `auditor.auditorProperties`, you can use `auditor.secretName` to specify the Secret resource that includes some credentials. For example, you can set credentials for a backend database (`scalar.db.username` and `scalar.db.password`) using environment variables, which makes your pods more secure. Please refer to the document [How to use Secret resources to pass the credentials as the environment variables into the properties file](use-secret-for-credentials.mdx) for more details on how to use a Secret resource. ```yaml auditor: secretName: "auditor-credentials-secret" ``` ### Affinity configurations (Recommended in the production environment) If you want to control pod deployment using the affinity and anti-affinity of Kubernetes, you can use `auditor.affinity`. You can configure them using the same syntax as the affinity of Kubernetes. So, please refer to the official document [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) for more details on the affinity configuration of Kubernetes. ```yaml auditor: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - scalardl-audit - key: app.kubernetes.io/app operator: In values: - auditor topologyKey: kubernetes.io/hostname weight: 50 ``` ### Prometheus/Grafana configurations (Recommended in the production environment) If you want to monitor ScalarDL Auditor pods using [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), you can deploy a ConfigMap, a ServiceMonitor, and a PrometheusRule resource for kube-prometheus-stack using `auditor.grafanaDashboard.enabled`, `auditor.serviceMonitor.enabled`, and `auditor.prometheusRule.enabled`. ```yaml auditor: grafanaDashboard: enabled: true namespace: monitoring serviceMonitor: enabled: true namespace: monitoring interval: 15s prometheusRule: enabled: true namespace: monitoring ``` ### SecurityContext configurations (Default value is recommended) If you want to set SecurityContext and PodSecurityContext for ScalarDL Auditor pods, you can use `auditor.securityContext` and `auditor.podSecurityContext`. You can configure them using the same syntax as SecurityContext and PodSecurityContext of Kubernetes. So, please refer to the official document [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for more details on the SecurityContext and PodSecurityContext configurations of Kubernetes. ```yaml auditor: podSecurityContext: seccompProfile: type: RuntimeDefault securityContext: capabilities: drop: - ALL runAsNonRoot: true allowPrivilegeEscalation: false ``` ### TLS configurations (optional based on your environment) You can enable TLS in: - The communications between the ScalarDL Auditor and clients. - The communications between the ScalarDL Ledger and ScalarDL Auditor. #### Enable TLS You can enable TLS in all ScalarDL Auditor connections by using the following configurations: ```yaml auditor: auditorProperties: | ...(omit)... scalar.dl.auditor.server.tls.enabled=true scalar.dl.auditor.server.tls.cert_chain_path=/tls/scalardl-auditor/certs/tls.crt scalar.dl.auditor.server.tls.private_key_path=/tls/scalardl-auditor/certs/tls.key scalar.dl.auditor.tls.enabled=true scalar.dl.auditor.tls.ca_root_cert_path=/tls/scalardl-ledger/certs/ca.crt scalar.dl.auditor.tls.override_authority=envoy.scalar.example.com tls: enabled: true ``` ##### Use your private key and certificate files You can set your private key and certificate files by using the following configurations: ```yaml auditor: tls: enabled: true caRootCertSecret: "scalardl-auditor-tls-ca" certChainSecret: "scalardl-auditor-tls-cert" privateKeySecret: "scalardl-auditor-tls-key" ``` In this case, you have to create secret resources that include private key and certificate files for ScalarDL Ledger and ScalarDL Auditor as follows, replacing the contents in the angle brackets as described: ```console kubectl create secret generic scalardl-auditor-tls-ca --from-file=ca.crt=/ -n kubectl create secret generic scalardl-auditor-tls-cert --from-file=tls.crt=/ -n kubectl create secret generic scalardl-auditor-tls-key --from-file=tls.key=/ -n kubectl create secret generic scalardl-auditor-tls-ca-for-ledger --from-file=ca.crt=/ -n ``` For more details on how to prepare private key and certificate files, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx). ##### Use a trusted CA with cert-manager to manage your private key and certificate files You can manage your private key and certificate files with cert-manager by using the following configurations, replacing the content in the angle brackets as described: :::note * If you want to use cert-manager, you must deploy cert-manager and prepare the `Issuers` resource. For details, see the cert-manager documentation, [Installation](https://cert-manager.io/docs/installation/) and [Issuer Configuration](https://cert-manager.io/docs/configuration/). * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. The default certificate configuration is recommended, but if you use a custom certificate configuration, you must satisfy the certificate requirements of Scalar products. For details, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml auditor: tls: enabled: true certManager: enabled: true issuerRef: name: dnsNames: - auditor.scalardl.example.com ``` In this case, cert-manager issues your private key and certificate files by using your trusted issuer. You don't need to mount private key and certificate files manually. ##### Use a self-signed CA with cert-manager to manage your private key and certificate files You can manage your private key and self-signed certificate files with cert-manager by using the following configurations: :::note * If you want to use cert-manager, you must deploy cert-manager. For details, see the cert-manager documentation, [Installation](https://cert-manager.io/docs/installation/). * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. The default certificate configuration is recommended, but if you use a custom certificate configuration, you must satisfy the certificate requirements of Scalar products. For details, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml auditor: tls: enabled: true certManager: enabled: true selfSigned: enabled: true dnsNames: - auditor.scalardl.example.com ``` In this case, Scalar Helm Charts and cert-manager issue your private key and self-signed certificate files. You don't need to mount private key and certificate files manually. #### Set a root CA certificate for ScalarDL Ledger If you enable TLS on the ScalarDL Ledger side, you must set a root CA certificate file for Envoy in front of ScalarDL Ledger to access it from ScalarDL Auditor. To determine which approach you should take, refer to the following decision tree: ```mermaid flowchart TD A[Are you using cert-manager?] A -->|Yes| B A -->|No| D B[Are you using a self-signed CA with cert-manager?] B -->|No| C[Are you using the same trusted CA for ScalarDL
Ledger and ScalarDL Auditor with cert-manager?] C -->|No| D[You must set the root
CA certificate of Envoy for ScalarDL Ledger manually.] C ---->|Yes| E[Scalar Helm Chart automatically sets the root CA certificate. You
don't need to set `auditor.tls.upstream.caRootCertSecret` explicitly.] ``` If you need to set the root CA certificate file of Envoy manually, you can set it by using the following configurations: ```yaml auditor: tls: enabled: true caRootCertForLedgerSecret: "scalardl-auditor-tls-ca-for-ledger" ``` In this case, you have to create secret resources that include root CA certificate files as follows, replacing the contents in the angle brackets as described: ```console kubectl create secret generic scalardl-auditor-tls-ca-for-ledger --from-file=ca.crt=//scalardl-ledger -n ``` ##### Set custom authority for TLS communications You can set the custom authority for TLS communications by using `auditor.tls.overrideAuthority`. This value doesn't change what host is actually connected. This value is intended for testing but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set by using `auditor.tls.certChainSecret`. This chart uses this value for `startupProbe` and `livenessProbe`. ##### Set a root CA certificate for Prometheus Operator If you set `auditor.serviceMonitor.enabled=true` and `auditor.tls.enabled=true` (in other words, if you monitor ScalarDL Auditor with TLS configuration by using Prometheus Operator), you must set the secret name to `auditor.tls.caRootCertSecretForServiceMonitor`. ```yaml auditor: tls: enabled: true caRootCertSecretForServiceMonitor: "scalardl-auditor-tls-ca-for-prometheus" ``` In this case, you have to create secret resources that include a root CA certificate for ScalarDL Auditor in the same namespace as Prometheus as follows: ```console kubectl create secret generic scalardl-auditor-tls-ca-for-prometheus --from-file=ca.crt=/path/to/your/ca/certificate/file -n ``` ### Replica configurations (Optional based on your environment) You can specify the number of replicas (pods) of ScalarDL Auditor using `auditor.replicaCount`. ```yaml auditor: replicaCount: 3 ``` ### Logging configurations (Optional based on your environment) If you want to change the log level of ScalarDL Auditor, you can use `auditor.scalarAuditorConfiguration.auditorLogLevel`. ```yaml auditor: scalarAuditorConfiguration: auditorLogLevel: INFO ``` ### Taint and toleration configurations (Optional based on your environment) If you want to control pod deployment by using the taints and tolerations in Kubernetes, you can use `auditor.tolerations`. You can configure taints and tolerations by using the same syntax as the tolerations in Kubernetes. For details on configuring tolerations in Kubernetes, see the official Kubernetes documentation [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). ```yaml auditor: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardl-auditor ``` ================================================ FILE: docs/helm-charts/configure-custom-values-scalardl-ledger.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Configure a custom values file for ScalarDL Ledger import CertificateManagement from '/src/components/en-us/_certificate-management.mdx'; This document explains how to create your custom values file for the ScalarDL Ledger chart. If you want to know the details of the parameters, please refer to the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/scalardl/README.md) of the ScalarDL Ledger chart. ## Required configurations ### Scalar Envoy configurations You must set the Scalar Envoy configurations in the custom values file for ScalarDL Ledger. This is because client requests are sent to ScalarDL Ledger via Scalar Envoy as the load balancer of gRPC requests if you deploy ScalarDL Ledger on a Kubernetes environment. Please refer to the document [Configure a custom values file for Scalar Envoy](configure-custom-values-envoy.mdx) for more details on the Scalar Envoy configurations. ```yaml envoy: configurationsForScalarEnvoy: ... ledger: configurationsForScalarDLLedger: ... ``` ### Image configurations You must set `ledger.image.repository`. Be sure to specify the ScalarDL Ledger container image so that you can pull the image from the container repository. ```yaml ledger: image: repository: ``` For more details on the container repository for Scalar products, see [How to get the container images of Scalar products](../scalar-kubernetes/HowToGetContainerImages.mdx). ### Ledger/Database configurations You must set `ledger.ledgerProperties`. Please set your `ledger.properties` to this parameter. Please refer to the [ledger.properties](https://github.com/scalar-labs/scalar/blob/master/ledger/conf/ledger.properties) for more details on the configuration of ScalarDL Ledger. ```yaml ledger: ledgerProperties: | scalar.db.contact_points=localhost scalar.db.username=cassandra scalar.db.password=cassandra scalar.db.storage=cassandra scalar.dl.ledger.proof.enabled=true scalar.dl.ledger.auditor.enabled=true scalar.dl.ledger.proof.private_key_path=/keys/ledger-key-file ``` ### Key/Certificate configurations If you set `scalar.dl.ledger.proof.enabled` to `true` (this configuration is required if you use ScalarDL Auditor), you must set a private key file to `scalar.dl.ledger.proof.private_key_path`. In this case, you must mount the private key file on the ScalarDL Ledger pod. For more details on how to mount the private key file, refer to [Mount key and certificate files on a pod in ScalarDL Helm Charts](mount-files-or-volumes-on-scalar-pods.mdx#mount-key-and-certificate-files-on-a-pod-in-scalardl-helm-charts). ## Optional configurations ### Resource configurations (Recommended in the production environment) If you want to control pod resources using the requests and limits of Kubernetes, you can use `ledger.resources`. Note that it is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. Also, if you use the pay-as-you-go (PAYG) containers that the AWS Marketplace provides, you will not be able to run any containers that exceed the 2vCPU / 4GB memory configuration in `resources.limits`. If you exceed this resource limitation, the pods will automatically stop. You can configure them using the same syntax as the requests and limits of Kubernetes. So, please refer to the official document [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for more details on the requests and limits of Kubernetes. ```yaml ledger: resources: requests: cpu: 2000m memory: 4Gi limits: cpu: 2000m memory: 4Gi ``` ### Secret configurations (Recommended in the production environment) If you want to use environment variables to set some properties (e.g., credentials) in the `ledger.ledgerProperties`, you can use `ledger.secretName` to specify the Secret resource that includes some credentials. For example, you can set credentials for a backend database (`scalar.db.username` and `scalar.db.password`) using environment variables, which makes your pods more secure. Please refer to the document [How to use Secret resources to pass the credentials as the environment variables into the properties file](use-secret-for-credentials.mdx) for more details on how to use a Secret resource. ```yaml ledger: secretName: "ledger-credentials-secret" ``` ### Affinity configurations (Recommended in the production environment) If you want to control pod deployment using the affinity and anti-affinity of Kubernetes, you can use `ledger.affinity`. You can configure them using the same syntax as the affinity of Kubernetes. So, please refer to the official document [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) for more details on the affinity configuration of Kubernetes. ```yaml ledger: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - scalardl - key: app.kubernetes.io/app operator: In values: - ledger topologyKey: kubernetes.io/hostname weight: 50 ``` ### Prometheus/Grafana configurations (Recommended in the production environment) If you want to monitor ScalarDL Ledger pods using [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), you can deploy a ConfigMap, a ServiceMonitor, and a PrometheusRule resource for kube-prometheus-stack using `ledger.grafanaDashboard.enabled`, `ledger.serviceMonitor.enabled`, and `ledger.prometheusRule.enabled`. ```yaml ledger: grafanaDashboard: enabled: true namespace: monitoring serviceMonitor: enabled: true namespace: monitoring interval: 15s prometheusRule: enabled: true namespace: monitoring ``` ### SecurityContext configurations (Default value is recommended) If you want to set SecurityContext and PodSecurityContext for ScalarDL Ledger pods, you can use `ledger.securityContext` and `ledger.podSecurityContext`. You can configure them using the same syntax as SecurityContext and PodSecurityContext of Kubernetes. So, please refer to the official document [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) for more details on the SecurityContext and PodSecurityContext configurations of Kubernetes. ```yaml ledger: podSecurityContext: seccompProfile: type: RuntimeDefault securityContext: capabilities: drop: - ALL runAsNonRoot: true allowPrivilegeEscalation: false ``` ### TLS configurations (optional based on your environment) You can enable TLS in: - The communications between the ScalarDL Ledger and clients. - The communications between the ScalarDL Ledger and ScalarDL Auditor. #### Enable TLS You can enable TLS in all ScalarDL Ledger connections by using the following configurations: ```yaml ledger: ledgerProperties: | ...(omit)... scalar.dl.ledger.server.tls.enabled=true scalar.dl.ledger.server.tls.cert_chain_path=/tls/scalardl-ledger/certs/tls.crt scalar.dl.ledger.server.tls.private_key_path=/tls/scalardl-ledger/certs/tls.key tls: enabled: true ``` ##### Use your private key and certificate files You can set your private key and certificate files by using the following configurations: ```yaml ledger: tls: enabled: true caRootCertSecret: "scalardl-ledger-tls-ca" certChainSecret: "scalardl-ledger-tls-cert" privateKeySecret: "scalardl-ledger-tls-key" ``` In this case, you have to create secret resources that include private key and certificate files for ScalarDL Ledger as follows, replacing the contents in the angle brackets as described: ```console kubectl create secret generic scalardl-ledger-tls-ca --from-file=ca.crt=/ -n kubectl create secret generic scalardl-ledger-tls-cert --from-file=tls.crt=/ -n kubectl create secret generic scalardl-ledger-tls-key --from-file=tls.key=/ -n ``` For more details on how to prepare private key and certificate files, see [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx). ##### Use a trusted CA with cert-manager to manage your private key and certificate files You can manage private key and certificate with cert-manager by using the following configurations: :::note * If you want to use cert-manager, you must deploy cert-manager and prepare the `Issuers` resource. For more details on cert-manager, see the [Installation](https://cert-manager.io/docs/installation/) and [Issuer Configuration](https://cert-manager.io/docs/configuration/) in the cert-manager official document. * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. We recommend the default certificate configuration, but if you custom certificate configuration, you must satisfy the certificate requirements of Scalar products. See [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml ledger: tls: enabled: true certManager: enabled: true issuerRef: name: your-trusted-ca dnsNames: - ledger.scalardl.example.com ``` In this case, cert-manager issues private key and certificate by using your trusted issuer. You don't need to mount private key and certificate files manually. ##### Use a self-signed CA with cert-manager to manage your private key and certificate files You can manage private key and self-signed certificate with cert-manager by using the following configurations: :::note * If you want to use cert-manager, you must deploy cert-manager. For more details on how to deploy cert-manager, see the [Installation](https://cert-manager.io/docs/installation/) in the cert-manager official document. * By default, Scalar Helm Chart creates a `Certificate` resource that satisfies the certificate requirements of Scalar products. We recommend the default certificate configuration, but if you custom certificate configuration, you must satisfy the certificate requirements of Scalar products. See [How to create private key and certificate files for Scalar products](../scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx#certificate-requirements). ::: ```yaml ledger: tls: enabled: true certManager: enabled: true selfSigned: enabled: true dnsNames: - ledger.scalardl.example.com ``` In this case, Scalar Helm Charts and cert-manager issue private key and self-signed certificate. You don't need to mount private key and certificate files manually. ##### Set custom authority for TLS communications You can set the custom authority for TLS communications by using `ledger.tls.overrideAuthority`. This value doesn't change what host is actually connected. This value is intended for testing but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set by using `ledger.tls.certChainSecret`. This chart uses this value for `startupProbe` and `livenessProbe`. ```yaml ledger: tls: enabled: true overrideAuthority: "ledger.scalardl.example.com" ``` ##### Set a root CA certificate for Prometheus Operator If you set `ledger.serviceMonitor.enabled=true` and `ledger.tls.enabled=true` (in other words, if you monitor ScalarDL Ledger with TLS configuration by using Prometheus Operator), you must set the secret name to `ledger.tls.caRootCertSecretForServiceMonitor`. ```yaml ledger: tls: enabled: true caRootCertSecretForServiceMonitor: "scalardl-ledger-tls-ca-for-prometheus" ``` In this case, you have to create secret resources that include a root CA certificate for ScalarDL Ledger in the same namespace as Prometheus as follows: ```console kubectl create secret generic scalardl-ledger-tls-ca-for-prometheus --from-file=ca.crt=/path/to/your/ca/certificate/file -n ``` ### Replica configurations (optional based on your environment) You can specify the number of replicas (pods) of ScalarDL Ledger using `ledger.replicaCount`. ```yaml ledger: replicaCount: 3 ``` ### Logging configurations (Optional based on your environment) If you want to change the log level of ScalarDL Ledger, you can use `ledger.scalarLedgerConfiguration.ledgerLogLevel`. ```yaml ledger: scalarLedgerConfiguration: ledgerLogLevel: INFO ``` ### Taint and toleration configurations (Optional based on your environment) If you want to control pod deployment by using the taints and tolerations in Kubernetes, you can use `ledger.tolerations`. You can configure taints and tolerations by using the same syntax as the tolerations in Kubernetes. For details on configuring tolerations in Kubernetes, see the official Kubernetes documentation [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). ```yaml ledger: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardl-ledger ``` ================================================ FILE: docs/helm-charts/configure-custom-values-scalardl-schema-loader.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Configure a custom values file for ScalarDL Schema Loader This document explains how to create your custom values file for the ScalarDL Schema Loader chart. If you want to know the details of the parameters, please refer to the [README](https://github.com/scalar-labs/helm-charts/blob/main/charts/schema-loading/README.md) of the ScalarDL Schema Loader chart. ## Required configurations ### Database configurations You must set `schemaLoading.databaseProperties`. Please set your `database.properties` to access the backend database to this parameter. Please refer to the [Getting Started with ScalarDB](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb) for more details on the database configuration of ScalarDB. ```yaml schemaLoading: databaseProperties: | scalar.db.contact_points=cassandra scalar.db.contact_port=9042 scalar.db.username=cassandra scalar.db.password=cassandra scalar.db.storage=cassandra ``` ### Schema type configurations You must set `schemaLoading.schemaType`. If you create the schema of ScalarDL Ledger, please set `ledger`. ```yaml schemaLoading: schemaType: ledger ``` If you create the schema of ScalarDL Auditor, please set `auditor`. ```yaml schemaLoading: schemaType: auditor ``` ## Optional configurations ### Secret configurations (Recommended in the production environment) If you want to use environment variables to set some properties (e.g., credentials) in the `schemaLoading.databaseProperties`, you can use `schemaLoading.secretName` to specify the Secret resource that includes some credentials. For example, you can set credentials for a backend database (`scalar.db.username` and `scalar.db.password`) using environment variables, which makes your pods more secure. Please refer to the document [How to use Secret resources to pass the credentials as the environment variables into the properties file](use-secret-for-credentials.mdx) for more details on how to use a Secret resource. ```yaml schemaLoading: secretName: "schema-loader-credentials-secret" ``` ### Image configurations (Default value is recommended) If you want to change the image repository, you can use `schemaLoading.image.repository` to specify which repository you want to use to pull the ScalarDL Schema Loader container image from. ```yaml schemaLoading: image: repository: ``` ### Flags configurations (Optional based on your environment) You can specify several flags as an array. Please refer to the document [ScalarDB Schema Loader](https://scalardb.scalar-labs.com/docs/latest/schema-loader) for more details on the flags. ```yaml schemaLoading: commandArgs: - "--alter" - "--compaction-strategy" - "" - "--delete-all" - "--no-backup" - "--no-scaling" - "--repair-all" - "--replication-factor" - "" - "--replication-strategy" - "" - "--ru" - "" ``` ================================================ FILE: docs/helm-charts/getting-started-logging.mdx ================================================ --- tags: - Community displayed_sidebar: docsEnglish --- # Getting Started with Helm Charts (Logging using Loki Stack) This document explains how to get started with log aggregation for Scalar products on Kubernetes using Grafana Loki (with Promtail). We assume that you have already read the [getting-started with monitoring](getting-started-monitoring.mdx) for Scalar products and installed kube-prometheus-stack. ## What we create We will deploy the following components on a Kubernetes cluster as follows. ``` +--------------------------------------------------------------------------------------------------+ | +------------------------------------+ | | | loki-stack | | | | | +-----------------+ | | | +--------------+ +--------------+ | <-----------------(Log)-------------- | Scalar Products | | | | | Loki | | Promtail | | | | | | | +--------------+ +--------------+ | | +-----------+ | | | +------------------------------------+ | | ScalarDB | | | | | +-----------+ | | | +------------------------------------------------------+ | | | | | kube-prometheus-stack | | +-----------+ | | | | | | | ScalarDL | | | | | +--------------+ +--------------+ +--------------+ | -----(Monitor)----> | +-----------+ | | | | | Prometheus | | Alertmanager | | Grafana | | +-----------------+ | | | +-------+------+ +------+-------+ +------+-------+ | | | | | | | | | | | +----------------+-----------------+ | | | | | | | | +--------------------------+---------------------------+ | | | | | | Kubernetes | +----------------------------+---------------------------------------------------------------------+ | <- expose to localhost (127.0.0.1) or use load balancer etc to access | (Access Dashboard through HTTP) | +----+----+ | Browser | +---------+ ``` ## Step 1. Prepare a custom values file 1. Get the sample file [scalar-loki-stack-custom-values.yaml](conf/scalar-loki-stack-custom-values.yaml) for the `loki-stack` helm chart. ## Step 2. Deploy `loki-stack` 1. Add the `grafana` helm repository. ```console helm repo add grafana https://grafana.github.io/helm-charts ``` 1. Deploy the `loki-stack` helm chart. ```console helm install scalar-logging-loki grafana/loki-stack -n monitoring -f scalar-loki-stack-custom-values.yaml ``` ## Step 3. Add a Loki data source in the Grafana configuration 1. Add a configuration of the Loki data source in the `scalar-prometheus-custom-values.yaml` file. ```yaml grafana: additionalDataSources: - name: Loki type: loki uid: loki url: http://scalar-logging-loki:3100/ access: proxy editable: false isDefault: false ``` 1. Apply the configuration (upgrade the deployment of `kube-prometheus-stack`). ```console helm upgrade scalar-monitoring prometheus-community/kube-prometheus-stack -n monitoring -f scalar-prometheus-custom-values.yaml ``` ## Step 4. Access the Grafana dashboard 1. Add Loki as a data source - Go to Grafana http://localhost:3000 (If you use minikube) - Go to `Explore` to find the added Loki - You can see the collected logs in the `Explore` page ## Step 5. Delete the `loki-stack` helm chart 1. Uninstall `loki-stack`. ```console helm uninstall scalar-logging-loki -n monitoring ``` ================================================ FILE: docs/helm-charts/getting-started-monitoring.mdx ================================================ --- tags: - Community displayed_sidebar: docsEnglish --- # Getting Started with Helm Charts (Monitoring using Prometheus Operator) This document explains how to get started with Scalar products monitoring on Kubernetes using Prometheus Operator (kube-prometheus-stack). Here, we assume that you already have a Mac or Linux environment for testing. We use **Minikube** in this document, but the steps we will show should work in any Kubernetes cluster. ## What we create We will deploy the following components on a Kubernetes cluster as follows. ``` +--------------------------------------------------------------------------------------------------+ | +------------------------------------------------------+ +-----------------+ | | | kube-prometheus-stack | | Scalar Products | | | | | | | | | | +--------------+ +--------------+ +--------------+ | -----(Monitor)----> | +-----------+ | | | | | Prometheus | | Alertmanager | | Grafana | | | | ScalarDB | | | | | +-------+------+ +------+-------+ +------+-------+ | | +-----------+ | | | | | | | | | +-----------+ | | | | +----------------+-----------------+ | | | ScalarDL | | | | | | | | +-----------+ | | | +--------------------------+---------------------------+ +-----------------+ | | | | | | Kubernetes | +----------------------------+---------------------------------------------------------------------+ | <- expose to localhost (127.0.0.1) or use load balancer etc to access | (Access Dashboard through HTTP) | +----+----+ | Browser | +---------+ ``` ## Step 1. Start a Kubernetes cluster First, you need to prepare a Kubernetes cluster. If you use a **minikube** environment, please refer to the [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx). If you have already started a Kubernetes cluster, you can skip this step. ## Step 2. Prepare a custom values file 1. Save the sample file [scalar-prometheus-custom-values.yaml](conf/scalar-prometheus-custom-values.yaml) for `kube-prometheus-stack`. 1. Add custom values in the `scalar-prometheus-custom-values.yaml` as follows. * settings * `prometheus.service.type` to `LoadBalancer` * `alertmanager.service.type` to `LoadBalancer` * `grafana.service.type` to `LoadBalancer` * `grafana.service.port` to `3000` * Example ```yaml alertmanager: service: type: LoadBalancer ... grafana: service: type: LoadBalancer port: 3000 ... prometheus: service: type: LoadBalancer ... ``` * Note: * If you want to customize the Prometheus Operator deployment by using Helm Charts, you'll need to set the following configurations to monitor Scalar products: * Set `serviceMonitorSelectorNilUsesHelmValues` and `ruleSelectorNilUsesHelmValues` to `false` (`true` by default) so that Prometheus Operator can detect `ServiceMonitor` and `PrometheusRule` for Scalar products. * If you want to use Scalar Manager, you'll need to set the following configurations to enable Scalar Manager to collect CPU and memory resources: * Set `kubeStateMetrics.enabled`, `nodeExporter.enabled`, and `kubelet.enabled` to `true`. * If you want to use Scalar Manager, you'll need to set the following configurations to enable Scalar Manager to embed Grafana: * Set `grafana.ini.security.allow_embedding` and `grafana.ini.auth.anonymous.enabled` to `true`. * Set `grafana.ini.auth.anonymous.org_name` to the organization you are using. If you're using the sample custom values, the value is `Main Org.`. * Set `grafana.ini.auth.anonymous.org_role` to `Editor`. ## Step 3. Deploy `kube-prometheus-stack` 1. Add the `prometheus-community` helm repository. ```console helm repo add prometheus-community https://prometheus-community.github.io/helm-charts ``` 1. Create a namespace `monitoring` on the Kubernetes. ```console kubectl create namespace monitoring ``` 1. Deploy the `kube-prometheus-stack`. ```console helm install scalar-monitoring prometheus-community/kube-prometheus-stack -n monitoring -f scalar-prometheus-custom-values.yaml ``` ## Step 4. Deploy (or Upgrade) Scalar products using Helm Charts * Note: * The following explains the minimum steps. If you want to know more details about the deployment of ScalarDB and ScalarDL, please refer to the following documents. * [Getting Started with Helm Charts (ScalarDB Server)](getting-started-scalardb.mdx) * [Getting Started with Helm Charts (ScalarDL Ledger / Ledger only)](getting-started-scalardl-ledger.mdx) * [Getting Started with Helm Charts (ScalarDL Ledger and Auditor / Auditor mode)](getting-started-scalardl-auditor.mdx) 1. To enable Prometheus monitoring of Scalar products, set `true` to the following configurations in the custom values file. * Configurations * `*.prometheusRule.enabled` * `*.grafanaDashboard.enabled` * `*.serviceMonitor.enabled` * Sample configuration files * ScalarDB (scalardb-custom-values.yaml) ```yaml envoy: prometheusRule: enabled: true grafanaDashboard: enabled: true serviceMonitor: enabled: true scalardb: prometheusRule: enabled: true grafanaDashboard: enabled: true serviceMonitor: enabled: true ``` * ScalarDL Ledger (scalardl-ledger-custom-values.yaml) ```yaml envoy: prometheusRule: enabled: true grafanaDashboard: enabled: true serviceMonitor: enabled: true ledger: prometheusRule: enabled: true grafanaDashboard: enabled: true serviceMonitor: enabled: true ``` * ScalarDL Auditor (scalardl-auditor-custom-values.yaml) ```yaml envoy: prometheusRule: enabled: true grafanaDashboard: enabled: true serviceMonitor: enabled: true auditor: prometheusRule: enabled: true grafanaDashboard: enabled: true serviceMonitor: enabled: true ``` 1. Deploy (or Upgrade) Scalar products using Helm Charts with the above custom values file. * Examples * ScalarDB ```console helm install scalardb scalar-labs/scalardb -f ./scalardb-custom-values.yaml ``` ```console helm upgrade scalardb scalar-labs/scalardb -f ./scalardb-custom-values.yaml ``` * ScalarDL Ledger ```console helm install scalardl-ledger scalar-labs/scalardl -f ./scalardl-ledger-custom-values.yaml ``` ```console helm upgrade scalardl-ledger scalar-labs/scalardl -f ./scalardl-ledger-custom-values.yaml ``` * ScalarDL Auditor ```console helm install scalardl-auditor scalar-labs/scalardl-audit -f ./scalardl-auditor-custom-values.yaml ``` ```console helm upgrade scalardl-auditor scalar-labs/scalardl-audit -f ./scalardl-auditor-custom-values.yaml ``` ## Step 5. Access Dashboards ### If you use minikube 1. To expose each service resource as your `localhost (127.0.0.1)`, open another terminal, and run the `minikube tunnel` command. ```console minikube tunnel ``` After running the `minikube tunnel` command, you can see the EXTERNAL-IP of each service resource as `127.0.0.1`. ```console kubectl get svc -n monitoring scalar-monitoring-kube-pro-prometheus scalar-monitoring-kube-pro-alertmanager scalar-monitoring-grafana ``` [Command execution result] ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalar-monitoring-kube-pro-prometheus LoadBalancer 10.98.11.12 127.0.0.1 9090:30550/TCP 26m scalar-monitoring-kube-pro-alertmanager LoadBalancer 10.98.151.66 127.0.0.1 9093:31684/TCP 26m scalar-monitoring-grafana LoadBalancer 10.103.19.4 127.0.0.1 3000:31948/TCP 26m ``` 1. Access each Dashboard. * Prometheus ```console http://localhost:9090/ ``` * Alertmanager ```console http://localhost:9093/ ``` * Grafana ```console http://localhost:3000/ ``` * Note: * You can see the user and password of Grafana as follows. * user ```console kubectl get secrets scalar-monitoring-grafana -n monitoring -o jsonpath='{.data.admin-user}' | base64 -d ``` * password ```console kubectl get secrets scalar-monitoring-grafana -n monitoring -o jsonpath='{.data.admin-password}' | base64 -d ``` ### If you use other Kubernetes than minikube If you use a Kubernetes cluster other than minikube, you need to access the LoadBalancer service according to the manner of each Kubernetes cluster. For example, using a Load Balancer provided by cloud service or the `kubectl port-forward` command. ## Step 6. Delete all resources After completing the Monitoring tests on the Kubernetes cluster, remove all resources. 1. Terminate the `minikube tunnel` command. (If you use minikube) ```console Ctrl + C ``` 1. Uninstall `kube-prometheus-stack`. ```console helm uninstall scalar-monitoring -n monitoring ``` 1. Delete minikube. (Optional / If you use minikube) ```console minikube delete --all ``` * Note: * If you deploy the ScalarDB or ScalarDL, you need to remove them before deleting minikube. ================================================ FILE: docs/helm-charts/getting-started-scalar-helm-charts.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Scalar Helm Charts This document explains how to get started with Scalar Helm Chart on a Kubernetes cluster as a test environment. Here, we assume that you already have a Mac or Linux environment for testing. We use **Minikube** in this document, but the steps we will show should work in any Kubernetes cluster. ## Tools We will use the following tools for testing. 1. minikube (If you use other Kubernetes distributions, minikube is not necessary.) 1. kubectl 1. Helm 1. cfssl / cfssljson ## Step 1. Install tools First, you need to install the following tools used in this guide. 1. Install the `minikube` command according to the [minikube documentation](https://minikube.sigs.k8s.io/docs/start/) 1. Install the `kubectl` command according to the [Kubernetes documentation](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) 1. Install the `helm` command according to the [Helm documentation](https://helm.sh/docs/intro/install/) 1. Install the `cfssl` and `cfssljson` according to the [CFSSL documentation](https://github.com/cloudflare/cfssl) :::note You need to install the `cfssl` and `cfssljson` command when following these getting started guides: * [ScalarDB Cluster with TLS](getting-started-scalardb-cluster-tls.mdx) * [ScalarDL Ledger and Auditor with TLS (Auditor mode)](getting-started-scalardl-auditor-tls.mdx) * [ScalarDL Ledger (Ledger only)](getting-started-scalardl-ledger.mdx) * [ScalarDL Ledger and Auditor (Auditor mode)](getting-started-scalardl-auditor.mdx) ::: ## Step 2. Start minikube with docker driver (Optional / If you use minikube) 1. Start minikube. ```console minikube start ``` 1. Check the status of the minikube and pods. ```console kubectl get pod -A ``` [Command execution result] ```console NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-64897985d-lbsfr 1/1 Running 1 (20h ago) 21h kube-system etcd-minikube 1/1 Running 1 (20h ago) 21h kube-system kube-apiserver-minikube 1/1 Running 1 (20h ago) 21h kube-system kube-controller-manager-minikube 1/1 Running 1 (20h ago) 21h kube-system kube-proxy-gsl6j 1/1 Running 1 (20h ago) 21h kube-system kube-scheduler-minikube 1/1 Running 1 (20h ago) 21h kube-system storage-provisioner 1/1 Running 2 (19s ago) 21h ``` If the minikube starts properly, you can see some pods are **Running** in the kube-system namespace. ## Step 3. After the Kubernetes cluster starts, you can try each Scalar Helm Charts on it. Please refer to the following documents for more details. * [ScalarDB Cluster with TLS](getting-started-scalardb-cluster-tls.mdx) * [ScalarDB Cluster with TLS by Using cert-manager](getting-started-scalardb-cluster-tls-cert-manager.mdx) * [ScalarDL Ledger and Auditor with TLS (Auditor mode)](getting-started-scalardl-auditor-tls.mdx) * [ScalarDL Ledger and Auditor with TLS by Using cert-manager (Auditor mode)](getting-started-scalardl-auditor-tls-cert-manager.mdx) * [ScalarDL Ledger (Ledger only)](getting-started-scalardl-ledger.mdx) * [ScalarDL Ledger and Auditor (Auditor mode)](getting-started-scalardl-auditor.mdx) * [Monitoring using Prometheus Operator](getting-started-monitoring.mdx) * [Logging using Loki Stack](getting-started-logging.mdx) * [Scalar Manager](getting-started-scalar-manager.mdx) * [[Deprecated] ScalarDB Server](getting-started-scalardb.mdx) ================================================ FILE: docs/helm-charts/getting-started-scalar-manager.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Deploy Scalar Manager import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; [Scalar Manager](../scalar-manager/overview.mdx) is a centralized management and monitoring solution for ScalarDB and ScalarDL in Kubernetes clusters. It enables you to: - Monitor the availability of ScalarDB and ScalarDL. - Schedule and execute pausing jobs to create transactionally consistent periods in the databases used by ScalarDB and ScalarDL. - Monitor ScalarDB and ScalarDL time-series metrics and logs through Grafana dashboards. This guide explains how to deploy and access Scalar Manager on a Kubernetes cluster using Scalar Helm Charts. ## Prerequisites Before deploying Scalar Manager, you must do the following: - Install the tools mentioned in [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx). - Deploy `kube-prometheus-stack` as instructed in [Getting Started with Helm Charts (Monitoring using Prometheus Operator)](getting-started-monitoring.mdx). - Deploy `loki-stack` as instructed in [Getting Started with Helm Charts (Logging using Loki Stack)](getting-started-logging.mdx). - Have a running PostgreSQL database, either self-managed or from a cloud service provider. This database stores the data of Scalar Manager. For example, you can use the [Bitnami package for PostgreSQL](https://artifacthub.io/packages/helm/bitnami/postgresql) to deploy a PostgreSQL database in your Kubernetes cluster. ## Deployment architecture diagram The following is an architecture diagram for the components deployed in a Kubernetes cluster. ``` +----------------------------------------------------------------------------------------------------------------------+ | +----------------------------+ | | | scalar-manager | | | | | | | | +------------------+ | ---------------------------------(Manage)--------------------------+ | | +---+--->| Scalar Manager | | | | | | | +---+--------------+ | | | | | | | | | | | | +--------+-------------------+ | | | | | | | | | +----+------------------------------------------+ | | | | | | | | | | +--------+------------------------------------------+---------+ | | | | | | kube-prometheus-stack | | V | | | | V V | +-----------------+ | | | | +--------------+ +--------------+ +--------------+ | -----(Monitor)----> | Scalar Products | | | | | | Prometheus | <---+ | Alertmanager | | Grafana | | | | | | | | +------+-------+ | +--------------+ +------+-------+ | | +-----------+ | | | | | | | | | | ScalarDB | | | | | | +----------------------------+ | | +-----------+ | | | | | | | | | | | | +---------------------------------------------------+---------+ | +-----------+ | | | | | | | ScalarDL | | | | | +------------------------------------------+ +---------- | +-----------+ | | | | | | +-----------------+ | | | +--------+---------------------------+ | | | | | | loki-stack | | | | | | V | | | | | | +--------------+ +--------------+ | <----------------(Log)-----------+ | | | | | Loki | | Promtail | | | | | | +--------------+ +--------------+ | | | | +------------------------------------+ | | | | | | Kubernetes | +----+-----------------------------------------------------------------------------------------------------------------+ | Expose the environment to localhost (127.0.0.1) or use a load balancer to access it | (Access the dashboard through HTTP) | +----+----+ | Browser | +---------+ ``` ## Step 1. Start minikube Open **Terminal**, and start minikube by running the following command: ```console minikube start ``` ## Step 2. Upgrade `kube-prometheus-stack` to enable Grafana authentication with auth proxy To allow users to access Grafana after logging in to Scalar Manager, you must enable Grafana authentication with auth proxy. In your custom values file for `kube-prometheus-stack` (for example, `scalar-prometheus-custom-values.yaml`), add or revise the following configurations: ```yaml kubeStateMetrics: enabled: true nodeExporter: enabled: true kubelet: enabled: true grafana: grafana.ini: users: allow_sign_up: false auto_assign_org: true auto_assign_org_role: Editor auth.proxy: enabled: true header_name: X-WEBAUTH-USER header_property: username auto_sign_up: true server: root_url: "%(protocol)s://%(domain)s:%(http_port)s/grafana" ``` Then, upgrade the Helm installation by running the following command: ```console helm upgrade scalar-monitoring prometheus-community/kube-prometheus-stack -n monitoring -f scalar-prometheus-custom-values.yaml ``` ## Step 3. Set environment variables Set the following environment variables for Scalar Manager, replacing the contents in the angle brackets as described: ```console SCALAR_MANAGER_RELEASE_NAME= SCALAR_MANAGER_NAMESPACE= SCALAR_MANAGER_CUSTOM_VALUES_FILE= SCALAR_MANAGER_CHART_VERSION= ``` ## Step 4. Prepare a custom values file Prepare a custom values file for Scalar Manager: 1. Create an empty file named `scalar-manager-custom-values.yaml`. 2. Follow the instructions in [Configure a custom values file for Scalar Manager](configure-custom-values-scalar-manager.mdx). ## Step 5. Deploy Deploy the `scalar-manager` Helm Chart by running the following command: ```console helm install ${SCALAR_MANAGER_RELEASE_NAME} scalar-labs/scalar-manager -n ${SCALAR_MANAGER_NAMESPACE} -f ${SCALAR_MANAGER_CUSTOM_VALUES_FILE} --version ${SCALAR_MANAGER_CHART_VERSION} ``` ## Step 6. Access Scalar Manager The method to access Scalar Manager depends on your Kubernetes cluster. To expose Scalar Manager on localhost (127.0.0.1), open another terminal and run the `minikube tunnel` command: ```console minikube tunnel ``` Then, access Scalar Manager at http://localhost:8000. If you're using a Kubernetes cluster other than minikube, access the `LoadBalancer` service according to your cluster's instructions. For example, use a load balancer from your cloud service provider or use the `kubectl port-forward` command. ## Additional details This section provides additional details related to configurations and resource discovery. ### Upgrade Scalar Manager To upgrade Scalar Manager, run the following command: ```console helm upgrade ${SCALAR_MANAGER_RELEASE_NAME} scalar-labs/scalar-manager -n ${SCALAR_MANAGER_NAMESPACE} -f ${SCALAR_MANAGER_CUSTOM_VALUES_FILE} --version ${SCALAR_MANAGER_CHART_VERSION} ``` ### Uninstall Scalar Manager To uninstall Scalar Manager, run the following command: ```console helm uninstall ${SCALAR_MANAGER_RELEASE_NAME} -n ${SCALAR_MANAGER_NAMESPACE} ``` ### Optional Scalar Manager configurations For optional configurations that you can set for Scalar Manager, see [Optional configurations](./configure-custom-values-scalar-manager.mdx#optional-configurations) ### Resource discovery Scalar Manager discovers the following Kubernetes resources in a cluster by using specific label selectors: - Dependencies - Prometheus service - Targets - ScalarDB Cluster deployments - ScalarDL Ledger deployments - ScalarDL Auditor deployments The following sections explain how Scalar Manager discovers these resources. #### Dependencies Scalar Manager searches for the default labels and values set in the [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) Helm Chart. For more information on the default labels and values that Scalar Manager uses to discover dependencies, see [Properties that you can set in `api.applicationProperties`](./configure-custom-values-scalar-manager.mdx#properties-that-you-can-set-in-apiapplicationProperties). Also, if you customized any values when installing `kube-prometheus-stack`, you will need to update the label selectors in the Scalar Manager custom value `api.applicationProperties`. #### Targets Scalar Manager searches for ScalarDB Cluster, ScalarDL Ledger, and ScalarDL Auditor deployments by using the following labels and values: - **ScalarDB Cluster:** `app.kubernetes.io/app=scalardb-cluster` - **ScalarDL Ledger:** `app.kubernetes.io/app=ledger` - **ScalarDL Auditor:** `app.kubernetes.io/app=auditor` Scalar Helm Charts use fixed labels and values for ScalarDB Cluster, ScalarDL Ledger, and ScalarDL Auditor deployments so that if you install ScalarDB and ScalarDL by using [Scalar Helm Charts](https://github.com/scalar-labs/helm-charts), Scalar Manager will automatically discover these deployments. ================================================ FILE: docs/helm-charts/getting-started-scalardb-cluster-tls-cert-manager.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Helm Charts (ScalarDB Cluster with TLS by Using cert-manager) This tutorial explains how to get started with ScalarDB Cluster with TLS configurations by using Helm Charts and cert-manager on a Kubernetes cluster in a test environment. Before starting, you should already have a Mac or Linux environment for testing. In addition, although this tutorial mentions using **minikube**, the steps described should work in any Kubernetes cluster. ## Requirements * You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). * You need to use ScalarDB Cluster 3.12 or later, which supports TLS. ## What you'll create In this tutorial, you'll deploy the following components on a Kubernetes cluster in the following way: ``` +----------------------------------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | [Pod] [Pod] [Pod] | | | | +-------+ +------------------------+ | | +---> | Envoy | ---+ +---> | ScalarDB Cluster node | ---+ | | [Pod] | +-------+ | | +------------------------+ | | | | | | | | | +-----------+ +---------+ | +-------+ | +--------------------+ | +------------------------+ | +---------------+ | | | Client | ---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | ScalarDB Cluster node | ---+---> | PostgreSQL | | | | (SQL CLI) | | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | +------------------------+ | | (For Ledger) | | | +-----------+ +---------+ | | +--------------------+ | | +---------------+ | | | +-------+ | | +------------------------+ | | | +---> | Envoy | ---+ +---> | ScalarDB Cluster node | ---+ | | +-------+ +------------------------+ | | | | +----------------------------------------------------------------------------------+ +---------------------+ | | | cert-manager (create private key and certificate for Envoy and ScalarDB Cluster) | | Issuer (Private CA) | | | +----------------------------------------------------------------------------------+ +---------------------+ | | | +----------------------------------------------------------------------------------------------------------------------------------------------------+ ``` cert-manager automatically creates the following private key and certificate files for TLS connections. ``` +----------------------+ +---> | For Scalar Envoy | | +----------------------+ | | tls.key | | | tls.crt | +-------------------------+ | +----------------------+ | Issuer (Self-signed CA) | ---(Sign certificates)---+ +-------------------------+ | +----------------------+ | tls.key | +---> | For ScalarDB Cluster | | tls.crt | +----------------------+ | ca.crt | | tls.key | +-------------------------+ | tls.crt | +----------------------+ ``` Scalar Helm Charts automatically mount each private key and certificate file for Envoy and ScalarDB Cluster as follows to enable TLS in each connection. You'll manually mount a root CA certificate file on the client. ``` +-------------------------------------+ +------------------------------------------------+ +--------------------------------+ | Client | ---(CRUD/SQL requests)---> | Envoy for ScalarDB Cluster | ---> | ScalarDB Cluster nodes | +-------------------------------------+ +------------------------------------------------+ +--------------------------------+ | ca.crt (to verify tls.crt of Envoy) | | tls.key | | tls.key | +-------------------------------------+ | tls.crt | | tls.crt | | ca.crt (to verify tls.crt of ScalarDB Cluster) | | ca.crt (to check health) | +------------------------------------------------+ +--------------------------------+ ``` The following connections exist amongst the ScalarDB Cluster–related components: * **`Client - Envoy for ScalarDB Cluster`:** When you execute a CRUD API or SQL API function, the client accesses Envoy for ScalarDB Cluster. * **`Envoy for ScalarDB Cluster - ScalarDB Cluster`:** Envoy works as an L7 (gRPC) load balancer in front of ScalarDB Cluster. * **`ScalarDB Cluster node - ScalarDB Cluster node`:** A ScalarDB Cluster node accesses other ScalarDB Cluster nodes. In other words, the cluster's internal communications exist amongst all ScalarDB Cluster nodes. ## Step 1. Start a Kubernetes cluster and install tools You need to prepare a Kubernetes cluster and install some tools (`kubectl`, `helm`, `cfssl`, and `cfssljson`). For more details on how to install them, see [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx). ## Step 2. Start the PostgreSQL containers ScalarDB Cluster must use some type of database system as a backend database. In this tutorial, you'll use PostgreSQL. You can deploy PostgreSQL on the Kubernetes cluster as follows: 1. Add the Bitnami helm repository. ```console helm repo add bitnami https://charts.bitnami.com/bitnami ``` 1. Deploy PostgreSQL for ScalarDB Cluster. ```console helm install postgresql-scalardb-cluster bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false \ -n default ``` 1. Check if the PostgreSQL containers are running. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-scalardb-cluster-0 1/1 Running 0 34s ``` ## Step 3. Create a working directory You'll create some configuration files locally. Be sure to create a working directory for those files. 1. Create a working directory. ```console mkdir -p ${HOME}/scalardb-cluster-test/ ``` ## Step 4. Deploy cert-manager and issuer resource This tutorial uses cert-manager to issue and manage your private keys and certificates. You can deploy cert-manager on the Kubernetes cluster as follows: 1. Add the Jetstack helm repository. ```console helm repo add jetstack https://charts.jetstack.io ``` 1. Deploy cert-manager. ```console helm install cert-manager jetstack/cert-manager \ --create-namespace \ --set installCRDs=true \ -n cert-manager ``` 1. Check if the cert-manager containers are running. ```console kubectl get pod -n cert-manager ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE cert-manager-6dc66985d4-6lvtt 1/1 Running 0 26s cert-manager-cainjector-c7d4dbdd9-xlrpn 1/1 Running 0 26s cert-manager-webhook-847d7676c9-ckcz2 1/1 Running 0 26s ``` 1. Change the working directory to `${HOME}/scalardb-cluster-test/`. ```console cd ${HOME}/scalardb-cluster-test/ ``` 1. Create a custom values file for the private CA (`private-ca-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardb-cluster-test/private-ca-custom-values.yaml apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: self-signed-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: self-signed-ca-cert spec: isCA: true commonName: self-signed-ca secretName: self-signed-ca-cert-secret privateKey: algorithm: ECDSA size: 256 issuerRef: name: self-signed-issuer kind: Issuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: self-signed-ca spec: ca: secretName: self-signed-ca-cert-secret EOF ``` 1. Deploy a self-signed CA. ```console kubectl apply -f ./private-ca-custom-values.yaml ``` 1. Check if the issuer resources are `True`. ```console kubectl get issuer ``` [Command execution result] ```console NAME READY AGE self-signed-ca True 6s self-signed-issuer True 6s ``` ## Step 5. Deploy ScalarDB Cluster on the Kubernetes cluster by using Helm Charts 1. Add the Scalar Helm Charts repository. ```console helm repo add scalar-labs https://scalar-labs.github.io/helm-charts ``` 1. Set your license key and certificate as environment variables. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). For details about the value of ``, see [How to Configure a License Key](../scalar-licensing/index.mdx). ```console SCALAR_DB_CLUSTER_LICENSE_KEY='' SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM='' ``` 1. Create a custom values file for ScalarDB Cluster (`scalardb-cluster-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardb-cluster-test/scalardb-cluster-custom-values.yaml envoy: enabled: true tls: downstream: enabled: true certManager: enabled: true issuerRef: name: self-signed-ca dnsNames: - envoy.scalar.example.com upstream: enabled: true overrideAuthority: "cluster.scalardb.example.com" scalardbCluster: image: repository: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium" scalardbClusterNodeProperties: | ### Necessary configurations for deployment on Kuberetes scalar.db.cluster.membership.type=KUBERNETES scalar.db.cluster.membership.kubernetes.endpoint.namespace_name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAMESPACE_NAME} scalar.db.cluster.membership.kubernetes.endpoint.name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAME} ### Storage configurations scalar.db.contact_points=jdbc:postgresql://postgresql-scalardb-cluster.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DB_CLUSTER_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DB_CLUSTER_POSTGRES_PASSWORD} scalar.db.storage=jdbc ### SQL configurations scalar.db.sql.enabled=true ### Auth configurations scalar.db.cluster.auth.enabled=true scalar.db.cross_partition_scan.enabled=true ### TLS configurations scalar.db.cluster.tls.enabled=true scalar.db.cluster.tls.ca_root_cert_path=/tls/scalardb-cluster/certs/ca.crt scalar.db.cluster.node.tls.cert_chain_path=/tls/scalardb-cluster/certs/tls.crt scalar.db.cluster.node.tls.private_key_path=/tls/scalardb-cluster/certs/tls.key scalar.db.cluster.tls.override_authority=cluster.scalardb.example.com ### License key configurations scalar.db.cluster.node.licensing.license_key=${env:SCALAR_DB_CLUSTER_LICENSE_KEY} scalar.db.cluster.node.licensing.license_check_cert_pem=${env:SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM} tls: enabled: true overrideAuthority: "cluster.scalardb.example.com" certManager: enabled: true issuerRef: name: self-signed-ca dnsNames: - cluster.scalardb.example.com secretName: "scalardb-credentials-secret" EOF ``` 1. Create a secret resource named `scalardb-credentials-secret` that includes credentials and license keys. ```console kubectl create secret generic scalardb-credentials-secret \ --from-literal=SCALAR_DB_CLUSTER_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DB_CLUSTER_POSTGRES_PASSWORD=postgres \ --from-literal=SCALAR_DB_CLUSTER_LICENSE_KEY="${SCALAR_DB_CLUSTER_LICENSE_KEY}" \ --from-file=SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM=<(echo ${SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM} | sed 's/\\n/\ /g') \ -n default ``` 1. Set the chart version of ScalarDB Cluster. ```console SCALAR_DB_CLUSTER_VERSION=3.12.2 SCALAR_DB_CLUSTER_CHART_VERSION=$(helm search repo scalar-labs/scalardb-cluster -l | grep -F "${SCALAR_DB_CLUSTER_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` 1. Deploy ScalarDB Cluster. ```console helm install scalardb-cluster scalar-labs/scalardb-cluster -f ${HOME}/scalardb-cluster-test/scalardb-cluster-custom-values.yaml --version ${SCALAR_DB_CLUSTER_CHART_VERSION} -n default ``` 1. Check if the ScalarDB Cluster pods are deployed. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-scalardb-cluster-0 1/1 Running 0 4m30s scalardb-cluster-envoy-7cc948dfb-4rb8l 1/1 Running 0 18s scalardb-cluster-envoy-7cc948dfb-hwt96 1/1 Running 0 18s scalardb-cluster-envoy-7cc948dfb-rzbrx 1/1 Running 0 18s scalardb-cluster-node-7c6959c79d-445kj 1/1 Running 0 18s scalardb-cluster-node-7c6959c79d-4z54q 1/1 Running 0 18s scalardb-cluster-node-7c6959c79d-vcv96 1/1 Running 0 18s ``` If the ScalarDB Cluster pods are deployed properly, the `STATUS` column for those pods will be displayed as `Running`. 1. Check if the ScalarDB Cluster services are deployed. ```console kubectl get svc -n default ``` [Command execution result] ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 7h34m postgresql-scalardb-cluster ClusterIP 10.96.92.27 5432/TCP 4m52s postgresql-scalardb-cluster-hl ClusterIP None 5432/TCP 4m52s scalardb-cluster-envoy ClusterIP 10.96.250.175 60053/TCP 40s scalardb-cluster-envoy-metrics ClusterIP 10.96.40.197 9001/TCP 40s scalardb-cluster-headless ClusterIP None 60053/TCP 40s scalardb-cluster-metrics ClusterIP 10.96.199.135 9080/TCP 40s ``` If the ScalarDB Cluster services are deployed properly, you can see private IP addresses in the `CLUSTER-IP` column. :::note The `CLUSTER-IP` values for `postgresql-scalardb-cluster-hl` and `scalardb-cluster-headless` are `None` since they have no IP addresses. ::: ## Step 6. Start a client container You'll use the CA certificate file in a client container. Therefore, you'll need to create a secret resource and mount it to the client container. 1. Create a secret resource named `client-ca-cert`. ```console kubectl create secret generic client-ca-cert --from-file=ca.crt=<(kubectl get secret self-signed-ca-cert-secret -o "jsonpath={.data['ca\.crt']}" | base64 -d) -n default ``` 1. Create a manifest file for a client pod (`scalardb-cluster-client-pod.yaml`). ```console cat << 'EOF' > ${HOME}/scalardb-cluster-test/scalardb-cluster-client-pod.yaml apiVersion: v1 kind: Pod metadata: name: "scalardb-cluster-client" spec: containers: - name: scalardb-cluster-client image: eclipse-temurin:8-jre-jammy command: ['sleep'] args: ['inf'] env: - name: SCALAR_DB_CLUSTER_VERSION value: SCALAR_DB_CLUSTER_CLIENT_POD_SCALAR_DB_CLUSTER_VERSION volumeMounts: - name: "client-ca-cert" mountPath: "/certs/" readOnly: true volumes: - name: "client-ca-cert" secret: secretName: "client-ca-cert" restartPolicy: Never EOF ``` 1. Set the ScalarDB Cluster version in the manifest file. ```console sed -i s/SCALAR_DB_CLUSTER_CLIENT_POD_SCALAR_DB_CLUSTER_VERSION/${SCALAR_DB_CLUSTER_VERSION}/ ${HOME}/scalardb-cluster-test/scalardb-cluster-client-pod.yaml ``` 1. Deploy the client pod. ```console kubectl apply -f ${HOME}/scalardb-cluster-test/scalardb-cluster-client-pod.yaml -n default ``` 1. Check if the client container is running. ```console kubectl get pod scalardb-cluster-client -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE scalardb-cluster-client 1/1 Running 0 26s ``` ## Step 7. Run the ScalarDB Cluster SQL CLI in the client container 1. Run bash in the client container. ```console kubectl exec -it scalardb-cluster-client -n default -- bash ``` The commands in the following steps must be run in the client container. 1. Download the ScalarDB Cluster SQL CLI from [Releases](https://github.com/scalar-labs/scalardb/releases). ```console curl -OL https://github.com/scalar-labs/scalardb/releases/download/v${SCALAR_DB_CLUSTER_VERSION}/scalardb-cluster-sql-cli-${SCALAR_DB_CLUSTER_VERSION}-all.jar ``` 1. Create a `database.properties` file and add configurations. ```console cat << 'EOF' > /database.properties # ScalarDB Cluster configurations scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points=indirect:scalardb-cluster-envoy.default.svc.cluster.local # Auth configurations scalar.db.cluster.auth.enabled=true scalar.db.sql.cluster_mode.username=admin scalar.db.sql.cluster_mode.password=admin # TLS configurations scalar.db.cluster.tls.enabled=true scalar.db.cluster.tls.ca_root_cert_path=/certs/ca.crt scalar.db.cluster.tls.override_authority=envoy.scalar.example.com EOF ``` 1. Run the ScalarDB Cluster SQL CLI. ```console java -jar /scalardb-cluster-sql-cli-${SCALAR_DB_CLUSTER_VERSION}-all.jar --config /database.properties ``` 1. Create a sample namespace named `ns`. ```sql CREATE NAMESPACE ns; ``` 1. Create a sample table named `tbl` under the namespace `ns`. ```sql CREATE TABLE ns.tbl (a INT, b INT, c INT, PRIMARY KEY(a, b)); ``` 1. Insert sample records. ```sql INSERT INTO ns.tbl VALUES (1,2,3), (4,5,6), (7,8,9); ``` 1. Select the sample records that you inserted. ```sql SELECT * FROM ns.tbl; ``` [Command execution result] ```sql 0: scalardb> SELECT * FROM ns.tbl; +---+---+---+ | a | b | c | +---+---+---+ | 7 | 8 | 9 | | 1 | 2 | 3 | | 4 | 5 | 6 | +---+---+---+ 3 rows selected (0.059 seconds) ``` 1. Press `Ctrl + D` to exit from ScalarDB Cluster SQL CLI. ```console ^D ``` 1. Exit from the client container. ```console exit ``` ## Step 8. Delete all resources After completing the ScalarDB Cluster tests on the Kubernetes cluster, remove all resources. 1. Uninstall ScalarDB Cluster and PostgreSQL. ```console helm uninstall -n default scalardb-cluster postgresql-scalardb-cluster ``` 1. Remove the self-signed CA. ``` kubectl delete -f ./private-ca-custom-values.yaml ``` 1. Uninstall cert-manager. ```console helm uninstall -n cert-manager cert-manager ``` 1. Remove the client container. ``` kubectl delete pod scalardb-cluster-client --grace-period 0 -n default ``` 1. Remove the secret resources. ``` kubectl delete secrets scalardb-credentials-secret self-signed-ca-cert-secret scalardb-cluster-envoy-tls-cert scalardb-cluster-tls-cert client-ca-cert ``` 1. Remove the namespace `cert-manager`. ``` kubectl delete ns cert-manager ``` 1. Remove the working directory and the sample configuration files. ```console cd ${HOME} ``` ```console rm -rf ${HOME}/scalardb-cluster-test/ ``` ## Further reading You can see how to get started with monitoring or logging for Scalar products in the following tutorials: * [Getting Started with Helm Charts (Monitoring using Prometheus Operator)](getting-started-monitoring.mdx) * [Getting Started with Helm Charts (Logging using Loki Stack)](getting-started-logging.mdx) * [Getting Started with Helm Charts (Scalar Manager)](getting-started-scalar-manager.mdx) ================================================ FILE: docs/helm-charts/getting-started-scalardb-cluster-tls.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Helm Charts (ScalarDB Cluster with TLS) This tutorial explains how to get started with ScalarDB Cluster with TLS configurations by using Helm Charts on a Kubernetes cluster in a test environment. Before starting, you should already have a Mac or Linux environment for testing. In addition, although this tutorial mentions using **minikube**, the steps described should work in any Kubernetes cluster. ## Requirements * You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). * You need to use ScalarDB Cluster 3.12 or later, which supports TLS. ## What you'll create In this tutorial, you'll deploy the following components on a Kubernetes cluster in the following way: ``` +----------------------------------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | [Pod] [Pod] [Pod] | | | | +-------+ +------------------------+ | | +---> | Envoy | ---+ +---> | ScalarDB Cluster node | ---+ | | [Pod] | +-------+ | | +------------------------+ | | | | | | | | | +-----------+ +---------+ | +-------+ | +--------------------+ | +------------------------+ | +---------------+ | | | Client | ---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | ScalarDB Cluster node | ---+---> | PostgreSQL | | | | (SQL CLI) | | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | +------------------------+ | | (For Ledger) | | | +-----------+ +---------+ | | +--------------------+ | | +---------------+ | | | +-------+ | | +------------------------+ | | | +---> | Envoy | ---+ +---> | ScalarDB Cluster node | ---+ | | +-------+ +------------------------+ | | | +----------------------------------------------------------------------------------------------------------------------------------------------------+ ``` You'll also create the following private key and certificate files for TLS connections. ``` +-------------------------------+ +---> | For Scalar Envoy | | +-------------------------------+ | | envoy-key.pem | | | envoy.pem | +----------------------+ | +-------------------------------+ | Self-signed CA | ---(Sign certificates)---+ +----------------------+ | +-------------------------------+ | ca-key.pem | +---> | For ScalarDB Cluster | | ca.pem | +-------------------------------+ +----------------------+ | scalardb-cluster-key.pem | | scalardb-cluster.pem | +-------------------------------+ ``` You'll set each private key and certificate file as follows to enable TLS in each connection. ``` +--------------------------------+ +-----------------------------------------+ +-----------------------------------------+ | Client | ---(CRUD/SQL requests)---> | Envoy for ScalarDB Cluster | ---> | ScalarDB Cluster nodes | +--------------------------------+ +-----------------------------------------+ +-----------------------------------------+ | ca.pem (to verify envoy.pem) | | envoy-key.pem | | scalardb-cluster-key.pem | +--------------------------------+ | envoy.pem | | scalardb-cluster.pem | | ca.pem (to verify scalardb-cluster.pem) | | ca.pem (used for health check) | +-----------------------------------------+ +-----------------------------------------+ ``` The following connections exist amongst the ScalarDB Cluster–related components: * **`Client - Envoy for ScalarDB Cluster`:** When you execute a CRUD API or SQL API function, the client accesses Envoy for ScalarDB Cluster. * **`Envoy for ScalarDB Cluster - ScalarDB Cluster`:** Envoy works as an L7 (gRPC) load balancer in front of ScalarDB Cluster. * **`ScalarDB Cluster node - ScalarDB Cluster node`:** A ScalarDB Cluster node accesses other ScalarDB Cluster nodes. In other words, the cluster's internal communications exist amongst all ScalarDB Cluster nodes. ## Step 1. Start a Kubernetes cluster and install tools You need to prepare a Kubernetes cluster and install some tools (`kubectl`, `helm`, `cfssl`, and `cfssljson`). For more details on how to install them, see [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx). ## Step 2. Start the PostgreSQL containers ScalarDB Cluster must use some type of database system as a backend database. In this tutorial, you'll use PostgreSQL. You can deploy PostgreSQL on the Kubernetes cluster as follows: 1. Add the Bitnami helm repository. ```console helm repo add bitnami https://charts.bitnami.com/bitnami ``` 1. Deploy PostgreSQL for ScalarDB Cluster. ```console helm install postgresql-scalardb-cluster bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false \ -n default ``` 1. Check if the PostgreSQL containers are running. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-scalardb-cluster-0 1/1 Running 0 34s ``` ## Step 3. Create a working directory You'll create some configuration files and private key and certificate files locally. Be sure to create a working directory for those files. 1. Create a working directory. ```console mkdir -p ${HOME}/scalardb-cluster-test/certs/ ``` ## Step 4. Create private key and certificate files You'll create private key and a certificate files. 1. Change the working directory to `${HOME}/scalardb-cluster-test/certs/`. ```console cd ${HOME}/scalardb-cluster-test/certs/ ``` 1. Create a JSON file that includes CA information. ```console cat << 'EOF' > ${HOME}/scalardb-cluster-test/certs/ca.json { "CN": "scalar-test-ca", "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "JP", "ST": "Tokyo", "L": "Shinjuku", "O": "Scalar Test CA" } ] } EOF ``` 1. Create the CA private key and certificate files. ```console cfssl gencert -initca ca.json | cfssljson -bare ca ``` 1. Create a JSON file that includes CA configurations. ```console cat << 'EOF' > ${HOME}/scalardb-cluster-test/certs/ca-config.json { "signing": { "default": { "expiry": "87600h" }, "profiles": { "scalar-test-ca": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth" ] } } } } EOF ``` 1. Create a JSON file that includes Envoy information. ```console cat << 'EOF' > ${HOME}/scalardb-cluster-test/certs/envoy.json { "CN": "scalar-envoy", "hosts": [ "envoy.scalar.example.com", "localhost" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "JP", "ST": "Tokyo", "L": "Shinjuku", "O": "Scalar Envoy Test" } ] } EOF ``` 1. Create a JSON file that includes ScalarDB Cluster information. ```console cat << 'EOF' > ${HOME}/scalardb-cluster-test/certs/scalardb-cluster.json { "CN": "scalardb-cluster", "hosts": [ "cluster.scalardb.example.com", "localhost" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "JP", "ST": "Tokyo", "L": "Shinjuku", "O": "ScalarDB Cluster Test" } ] } EOF ``` 1. Create private key and certificate files for Envoy. ```console cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile scalar-test-ca envoy.json | cfssljson -bare envoy ``` 1. Create private key and certificate files for ScalarDB Cluster. ```console cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile scalar-test-ca scalardb-cluster.json | cfssljson -bare scalardb-cluster ``` 1. Confirm that the private key and certificate files were created. ```console ls -1 ``` [Command execution result] ```console ca-config.json ca-key.pem ca.csr ca.json ca.pem envoy-key.pem envoy.csr envoy.json envoy.pem scalardb-cluster-key.pem scalardb-cluster.csr scalardb-cluster.json scalardb-cluster.pem ``` ## Step 5. Deploy ScalarDB Cluster on the Kubernetes cluster by using Helm Charts 1. Add the Scalar Helm Charts repository. ```console helm repo add scalar-labs https://scalar-labs.github.io/helm-charts ``` 1. Set your license key and certificate as environment variables. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). For details about the value of ``, see [How to Configure a License Key](../scalar-licensing/index.mdx). ```console SCALAR_DB_CLUSTER_LICENSE_KEY='' SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM='' ``` 1. Create a custom values file for ScalarDB Cluster (`scalardb-cluster-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardb-cluster-test/scalardb-cluster-custom-values.yaml envoy: enabled: true tls: downstream: enabled: true certChainSecret: "envoy-tls-cert" privateKeySecret: "envoy-tls-key" upstream: enabled: true overrideAuthority: "cluster.scalardb.example.com" caRootCertSecret: "scalardb-cluster-tls-ca" scalardbCluster: image: repository: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium" scalardbClusterNodeProperties: | ### Necessary configurations for deployment on Kuberetes scalar.db.cluster.membership.type=KUBERNETES scalar.db.cluster.membership.kubernetes.endpoint.namespace_name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAMESPACE_NAME} scalar.db.cluster.membership.kubernetes.endpoint.name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAME} ### Storage configurations scalar.db.contact_points=jdbc:postgresql://postgresql-scalardb-cluster.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DB_CLUSTER_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DB_CLUSTER_POSTGRES_PASSWORD} scalar.db.storage=jdbc ### SQL configurations scalar.db.sql.enabled=true ### Auth configurations scalar.db.cluster.auth.enabled=true scalar.db.cross_partition_scan.enabled=true ### TLS configurations scalar.db.cluster.tls.enabled=true scalar.db.cluster.tls.ca_root_cert_path=/tls/scalardb-cluster/certs/ca.crt scalar.db.cluster.node.tls.cert_chain_path=/tls/scalardb-cluster/certs/tls.crt scalar.db.cluster.node.tls.private_key_path=/tls/scalardb-cluster/certs/tls.key scalar.db.cluster.tls.override_authority=cluster.scalardb.example.com ### License key configurations scalar.db.cluster.node.licensing.license_key=${env:SCALAR_DB_CLUSTER_LICENSE_KEY} scalar.db.cluster.node.licensing.license_check_cert_pem=${env:SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM} tls: enabled: true overrideAuthority: "cluster.scalardb.example.com" caRootCertSecret: "scalardb-cluster-tls-ca" certChainSecret: "scalardb-cluster-tls-cert" privateKeySecret: "scalardb-cluster-tls-key" secretName: "scalardb-credentials-secret" EOF ``` 1. Create a secret resource named `scalardb-credentials-secret` that includes credentials and license keys. ```console kubectl create secret generic scalardb-credentials-secret \ --from-literal=SCALAR_DB_CLUSTER_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DB_CLUSTER_POSTGRES_PASSWORD=postgres \ --from-literal=SCALAR_DB_CLUSTER_LICENSE_KEY="${SCALAR_DB_CLUSTER_LICENSE_KEY}" \ --from-file=SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM=<(echo ${SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM} | sed 's/\\n/\ /g') \ -n default ``` 1. Create secret resources that include the private key and certificate files for Envoy. ```console kubectl create secret generic envoy-tls-cert --from-file=tls.crt=${HOME}/scalardb-cluster-test/certs/envoy.pem -n default kubectl create secret generic envoy-tls-key --from-file=tls.key=${HOME}/scalardb-cluster-test/certs/envoy-key.pem -n default ``` 1. Create secret resources that include the key, certificate, and CA certificate files for ScalarDB Cluster. ```console kubectl create secret generic scalardb-cluster-tls-ca --from-file=ca.crt=${HOME}/scalardb-cluster-test/certs/ca.pem -n default kubectl create secret generic scalardb-cluster-tls-cert --from-file=tls.crt=${HOME}/scalardb-cluster-test/certs/scalardb-cluster.pem -n default kubectl create secret generic scalardb-cluster-tls-key --from-file=tls.key=${HOME}/scalardb-cluster-test/certs/scalardb-cluster-key.pem -n default ``` 1. Set the chart version of ScalarDB Cluster. ```console SCALAR_DB_CLUSTER_VERSION=3.12.2 SCALAR_DB_CLUSTER_CHART_VERSION=$(helm search repo scalar-labs/scalardb-cluster -l | grep -F "${SCALAR_DB_CLUSTER_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` 1. Deploy ScalarDB Cluster. ```console helm install scalardb-cluster scalar-labs/scalardb-cluster -f ${HOME}/scalardb-cluster-test/scalardb-cluster-custom-values.yaml --version ${SCALAR_DB_CLUSTER_CHART_VERSION} -n default ``` 1. Check if the ScalarDB Cluster pods are deployed. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-scalardb-cluster-0 1/1 Running 0 4m30s scalardb-cluster-envoy-7cc948dfb-4rb8l 1/1 Running 0 18s scalardb-cluster-envoy-7cc948dfb-hwt96 1/1 Running 0 18s scalardb-cluster-envoy-7cc948dfb-rzbrx 1/1 Running 0 18s scalardb-cluster-node-7c6959c79d-445kj 1/1 Running 0 18s scalardb-cluster-node-7c6959c79d-4z54q 1/1 Running 0 18s scalardb-cluster-node-7c6959c79d-vcv96 1/1 Running 0 18s ``` If the ScalarDB Cluster pods are deployed properly, the `STATUS` column for those pods will be displayed as `Running`. 1. Check if the ScalarDB Cluster services are deployed. ```console kubectl get svc -n default ``` [Command execution result] ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 7h34m postgresql-scalardb-cluster ClusterIP 10.96.92.27 5432/TCP 4m52s postgresql-scalardb-cluster-hl ClusterIP None 5432/TCP 4m52s scalardb-cluster-envoy ClusterIP 10.96.250.175 60053/TCP 40s scalardb-cluster-envoy-metrics ClusterIP 10.96.40.197 9001/TCP 40s scalardb-cluster-headless ClusterIP None 60053/TCP 40s scalardb-cluster-metrics ClusterIP 10.96.199.135 9080/TCP 40s ``` If the ScalarDB Cluster services are deployed properly, you can see private IP addresses in the `CLUSTER-IP` column. :::note The `CLUSTER-IP` values for `postgresql-scalardb-cluster-hl` and `scalardb-cluster-headless` are `None` since they have no IP addresses. ::: ## Step 6. Start a client container You'll use the CA certificate file in a client container. Therefore, you'll need to create a secret resource and mount it to the client container. 1. Create a secret resource named `client-ca-cert`. ```console kubectl create secret generic client-ca-cert --from-file=ca.crt=${HOME}/scalardb-cluster-test/certs/ca.pem -n default ``` 1. Create a manifest file for a client pod (`scalardb-cluster-client-pod.yaml`). ```console cat << 'EOF' > ${HOME}/scalardb-cluster-test/scalardb-cluster-client-pod.yaml apiVersion: v1 kind: Pod metadata: name: "scalardb-cluster-client" spec: containers: - name: scalardb-cluster-client image: eclipse-temurin:8-jre-jammy command: ['sleep'] args: ['inf'] env: - name: SCALAR_DB_CLUSTER_VERSION value: SCALAR_DB_CLUSTER_CLIENT_POD_SCALAR_DB_CLUSTER_VERSION volumeMounts: - name: "client-ca-cert" mountPath: "/certs/" readOnly: true volumes: - name: "client-ca-cert" secret: secretName: "client-ca-cert" restartPolicy: Never EOF ``` 1. Set the ScalarDB Cluster version in the manifest file. ```console sed -i s/SCALAR_DB_CLUSTER_CLIENT_POD_SCALAR_DB_CLUSTER_VERSION/${SCALAR_DB_CLUSTER_VERSION}/ ${HOME}/scalardb-cluster-test/scalardb-cluster-client-pod.yaml ``` 1. Deploy the client pod. ```console kubectl apply -f ${HOME}/scalardb-cluster-test/scalardb-cluster-client-pod.yaml -n default ``` 1. Check if the client container is running. ```console kubectl get pod scalardb-cluster-client -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE scalardb-cluster-client 1/1 Running 0 26s ``` ## Step 7. Run the ScalarDB Cluster SQL CLI in the client container 1. Run bash in the client container. ```console kubectl exec -it scalardb-cluster-client -n default -- bash ``` The commands in the following steps must be run in the client container. 1. Download the ScalarDB Cluster SQL CLI from [Releases](https://github.com/scalar-labs/scalardb/releases). ```console curl -OL https://github.com/scalar-labs/scalardb/releases/download/v${SCALAR_DB_CLUSTER_VERSION}/scalardb-cluster-sql-cli-${SCALAR_DB_CLUSTER_VERSION}-all.jar ``` 1. Create a `database.properties` file and add configurations. ```console cat << 'EOF' > /database.properties # ScalarDB Cluster configurations scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points=indirect:scalardb-cluster-envoy.default.svc.cluster.local # Auth configurations scalar.db.cluster.auth.enabled=true scalar.db.sql.cluster_mode.username=admin scalar.db.sql.cluster_mode.password=admin # TLS configurations scalar.db.cluster.tls.enabled=true scalar.db.cluster.tls.ca_root_cert_path=/certs/ca.crt scalar.db.cluster.tls.override_authority=envoy.scalar.example.com EOF ``` 1. Run the ScalarDB Cluster SQL CLI. ```console java -jar /scalardb-cluster-sql-cli-${SCALAR_DB_CLUSTER_VERSION}-all.jar --config /database.properties ``` 1. Create a sample namespace named `ns`. ```sql CREATE NAMESPACE ns; ``` 1. Create a sample table named `tbl` under the namespace `ns`. ```sql CREATE TABLE ns.tbl (a INT, b INT, c INT, PRIMARY KEY(a, b)); ``` 1. Insert sample records. ```sql INSERT INTO ns.tbl VALUES (1,2,3), (4,5,6), (7,8,9); ``` 1. Select the sample records that you inserted. ```sql SELECT * FROM ns.tbl; ``` [Command execution result] ```sql 0: scalardb> SELECT * FROM ns.tbl; +---+---+---+ | a | b | c | +---+---+---+ | 7 | 8 | 9 | | 1 | 2 | 3 | | 4 | 5 | 6 | +---+---+---+ 3 rows selected (0.059 seconds) ``` 1. Press `Ctrl + D` to exit from ScalarDB Cluster SQL CLI. ```console ^D ``` 1. Exit from the client container. ```console exit ``` ## Step 8. Delete all resources After completing the ScalarDB Cluster tests on the Kubernetes cluster, remove all resources. 1. Uninstall ScalarDB Cluster and PostgreSQL. ```console helm uninstall -n default scalardb-cluster postgresql-scalardb-cluster ``` 1. Remove the client container. ``` kubectl delete pod scalardb-cluster-client --grace-period 0 -n default ``` 1. Remove the secret resources. ``` kubectl delete secrets scalardb-credentials-secret scalardb-cluster-tls-key scalardb-cluster-tls-cert scalardb-cluster-tls-ca envoy-tls-key envoy-tls-cert client-ca-cert ``` 1. Remove the working directory and sample files (configuration file, private key, and certificate). ```console cd ${HOME} ``` ```console rm -rf ${HOME}/scalardb-cluster-test/ ``` ## Further reading You can see how to get started with monitoring or logging for Scalar products in the following tutorials: * [Getting Started with Helm Charts (Monitoring using Prometheus Operator)](getting-started-monitoring.mdx) * [Getting Started with Helm Charts (Logging using Loki Stack)](getting-started-logging.mdx) * [Getting Started with Helm Charts (Scalar Manager)](getting-started-scalar-manager.mdx) ================================================ FILE: docs/helm-charts/getting-started-scalardb.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Deprecated displayed_sidebar: docsEnglish --- # [Deprecated] Getting Started with Helm Charts (ScalarDB Server) :::note ScalarDB Server is now deprecated. Please use [ScalarDB Cluster](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/setup-scalardb-cluster-on-kubernetes-by-using-helm-chart) instead. ::: This document explains how to get started with ScalarDB Server using Helm Chart on a Kubernetes cluster as a test environment. Here, we assume that you already have a Mac or Linux environment for testing. We use **Minikube** in this document, but the steps we will show should work in any Kubernetes cluster. ## Requirement * You need to subscribe to ScalarDB in the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-rzbuhxgvqf4d2) to get container images (`scalardb-server` and `scalardb-envoy`). For more details, refer to [How to install Scalar products through AWS Marketplace](../scalar-kubernetes/AwsMarketplaceGuide.mdx). ## What we create We will deploy the following components on a Kubernetes cluster as follows. ``` +--------------------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | | | [Pod] [Pod] [Pod] [Pod] | | | | +-------+ +-----------------+ | | +---> | Envoy | ---+ +---> | ScalarDB Server | ---+ | | | +-------+ | | +-----------------+ | | | | | | | | | +--------+ +---------+ | +-------+ | +-------------------+ | +-----------------+ | +------------+ | | | Client | ---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | ScalarDB Server | ---+---> | PostgreSQL | | | +--------+ | (Envoy) | | +-------+ | | (ScalarDB Server) | | +-----------------+ | +------------+ | | +---------+ | | +-------------------+ | | | | | +-------+ | | +-----------------+ | | | +---> | Envoy | ---+ +---> | ScalarDB Server | ---+ | | +-------+ +-----------------+ | | | +--------------------------------------------------------------------------------------------------------------------------------------+ ``` ## Step 1. Start a Kubernetes cluster First, you need to prepare a Kubernetes cluster. If you use a **minikube** environment, please refer to the [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx). If you have already started a Kubernetes cluster, you can skip this step. ## Step 2. Start a PostgreSQL container ScalarDB uses some kind of database system as a backend database. In this document, we use PostgreSQL. You can deploy PostgreSQL on the Kubernetes cluster as follows. 1. Add the Bitnami helm repository. ```console helm repo add bitnami https://charts.bitnami.com/bitnami ``` 1. Deploy PostgreSQL. ```console helm install postgresql-scalardb bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false ``` 1. Check if the PostgreSQL container is running. ```console kubectl get pod ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-scalardb-0 1/1 Running 0 2m42s ``` ## Step 3. Deploy ScalarDB Server on the Kubernetes cluster using Helm Charts 1. Add the Scalar helm repository. ```console helm repo add scalar-labs https://scalar-labs.github.io/helm-charts ``` 1. Create a secret resource to pull the ScalarDB container images from AWS. * AWS Marketplace ```console kubectl create secret docker-registry reg-ecr-mp-secrets \ --docker-server=709825985650.dkr.ecr.us-east-1.amazonaws.com \ --docker-username=AWS \ --docker-password=$(aws ecr get-login-password --region us-east-1) ``` For more details, refer to [How to install Scalar products through AWS Marketplace](../scalar-kubernetes/AwsMarketplaceGuide.mdx). 1. Create a custom values file for ScalarDB Server (scalardb-custom-values.yaml). * AWS Marketplace ```console cat << 'EOF' > scalardb-custom-values.yaml envoy: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-envoy" version: "1.3.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" scalardb: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-server" tag: "3.7.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" databaseProperties: | scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://postgresql-scalardb.default.svc.cluster.local:5432/postgres scalar.db.username={{ default .Env.SCALAR_DB_POSTGRES_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DB_POSTGRES_PASSWORD "" }} secretName: "scalardb-credentials-secret" EOF ``` 1. Create a Secret resource that includes a username and password for PostgreSQL. ```console kubectl create secret generic scalardb-credentials-secret \ --from-literal=SCALAR_DB_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DB_POSTGRES_PASSWORD=postgres ``` 1. Deploy ScalarDB Server. ```console helm install scalardb scalar-labs/scalardb -f ./scalardb-custom-values.yaml ``` 1. Check if the ScalarDB Server pods are deployed. ```console kubectl get pod ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-scalardb-0 1/1 Running 0 9m48s scalardb-765598848b-75csp 1/1 Running 0 6s scalardb-765598848b-w864f 1/1 Running 0 6s scalardb-765598848b-x8rqj 1/1 Running 0 6s scalardb-envoy-84c475f77b-kpz2p 1/1 Running 0 6s scalardb-envoy-84c475f77b-n74tk 1/1 Running 0 6s scalardb-envoy-84c475f77b-zbrwz 1/1 Running 0 6s ``` If the ScalarDB Server Pods are deployed properly, you can see the STATUS are **Running**. 1. Check if the ScalarDB Server services are deployed. ```console kubectl get svc ``` [Command execution result] ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 47d postgresql-scalardb ClusterIP 10.109.118.122 5432/TCP 10m postgresql-scalardb-hl ClusterIP None 5432/TCP 10m scalardb-envoy ClusterIP 10.110.110.250 60051/TCP 41s scalardb-envoy-metrics ClusterIP 10.107.98.227 9001/TCP 41s scalardb-headless ClusterIP None 60051/TCP 41s scalardb-metrics ClusterIP 10.108.188.10 8080/TCP 41s ``` If the ScalarDB Server services are deployed properly, you can see private IP addresses in the CLUSTER-IP column. (Note: `scalardb-headless` has no CLUSTER-IP.) ## Step 4. Start a Client container 1. Start a Client container on the Kubernetes cluster. ```console kubectl run scalardb-client --image eclipse-temurin:8-jdk --command sleep inf ``` 1. Check if the Client container is running. ```console kubectl get pod scalardb-client ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE scalardb-client 1/1 Running 0 23s ``` ## Step 5. Run ScalarDB sample applications in the Client container The following explains the minimum steps. If you want to know more details about ScalarDB, please refer to the [Getting Started with ScalarDB](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb). 1. Run bash in the Client container. ```console kubectl exec -it scalardb-client -- bash ``` After this step, run each command in the Client container. 1. Install the git and curl commands in the Client container. ```console apt update && apt install -y git curl ``` 1. Clone ScalarDB git repository. ```console git clone https://github.com/scalar-labs/scalardb.git ``` 1. Change the directory to `scalardb/`. ```console cd scalardb/ ``` ```console pwd ``` [Command execution result] ```console /scalardb ``` 1. Change branch to arbitrary version. ```console git checkout -b v3.7.0 refs/tags/v3.7.0 ``` ```console git branch ``` [Command execution result] ```console master * v3.7.0 ``` If you want to use another version, please specify the version (tag) you want to use. 1. Change the directory to `docs/getting-started/`. ```console cd docs/getting-started/ ``` ```console pwd ``` [Command execution result] ```console /scalardb/docs/getting-started ``` 1. Download Schema Loader from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases). ```console curl -OL https://github.com/scalar-labs/scalardb/releases/download/v3.7.0/scalardb-schema-loader-3.7.0.jar ``` You need to use the same version of ScalarDB and Schema Loader. 1. Create a configuration file (scalardb.properties) to access ScalarDB Server on the Kubernetes cluster. ```console cat << 'EOF' > scalardb.properties scalar.db.contact_points=scalardb-envoy.default.svc.cluster.local scalar.db.contact_port=60051 scalar.db.storage=grpc scalar.db.transaction_manager=grpc EOF ``` 1. Create a JSON file (emoney-transaction.json) that defines DB Schema for the sample applications. ```console cat << 'EOF' > emoney-transaction.json { "emoney.account": { "transaction": true, "partition-key": [ "id" ], "clustering-key": [], "columns": { "id": "TEXT", "balance": "INT" } } } EOF ``` 1. Run Schema Loader (Create sample TABLE). ```console java -jar ./scalardb-schema-loader-3.7.0.jar --config ./scalardb.properties -f emoney-transaction.json --coordinator ``` 1. Run the sample applications. * Charge `1000` to `user1`: ```console ./gradlew run --args="-action charge -amount 1000 -to user1" ``` * Charge `0` to `merchant1` (Just create an account for `merchant1`): ```console ./gradlew run --args="-action charge -amount 0 -to merchant1" ``` * Pay `100` from `user1` to `merchant1`: ```console ./gradlew run --args="-action pay -amount 100 -from user1 -to merchant1" ``` * Get the balance of `user1`: ```console ./gradlew run --args="-action getBalance -id user1" ``` * Get the balance of `merchant1`: ```console ./gradlew run --args="-action getBalance -id merchant1" ``` 1. (Optional) You can see the inserted and modified (INSERT/UPDATE) data through the sample applications using the following command. (This command needs to run on your localhost, not on the Client container.) ```console kubectl exec -it postgresql-scalardb-0 -- bash -c 'export PGPASSWORD=postgres && psql -U postgres -d postgres -c "SELECT * FROM emoney.account"' ``` [Command execution result] ```sql id | balance | tx_id | tx_state | tx_version | tx_prepared_at | tx_committed_at | before_tx_id | before_tx_state | before_tx_version | before_tx_prepared_at | before_tx_committed_at | before_balance -----------+---------+--------------------------------------+----------+------------+----------------+-----------------+--------------------------------------+-----------------+-------------------+-----------------------+------------------------+---------------- merchant1 | 100 | 65a90225-0846-4e97-b729-151f76f6ca2f | 3 | 2 | 1667361909634 |1667361909679 | 3633df99-a8ed-4301-a8b9-db1344807d7b | 3 | 1 | 1667361902466 | 1667361902485 | 0 user1 | 900 | 65a90225-0846-4e97-b729-151f76f6ca2f | 3 | 2 | 1667361909634 |1667361909679 | 5520cba4-625a-4886-b81f-6089bf846d18 | 3 | 1 | 1667361897283 | 1667361897317 | 1000 (2 rows) ``` * Note: * Usually, you need to access data (records) through ScalarDB. The above command is used to explain and confirm the working of the sample applications. ## Step 6. Delete all resources After completing the ScalarDB Server tests on the Kubernetes cluster, remove all resources. 1. Uninstall ScalarDB Server and PostgreSQL. ```console helm uninstall scalardb postgresql-scalardb ``` 1. Remove the Client container. ``` kubectl delete pod scalardb-client --force --grace-period 0 ``` ## Further reading You can see how to get started with monitoring or logging for Scalar products in the following documents. * [Getting Started with Helm Charts (Monitoring using Prometheus Operator)](getting-started-monitoring.mdx) * [Getting Started with Helm Charts (Logging using Loki Stack)](getting-started-logging.mdx) * [Getting Started with Helm Charts (Scalar Manager)](getting-started-scalar-manager.mdx) ================================================ FILE: docs/helm-charts/getting-started-scalardl-auditor-tls-cert-manager.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Getting Started with Helm Charts (ScalarDL Ledger and Auditor with TLS by Using cert-manager / Auditor Mode) This tutorial explains how to get started with ScalarDL Ledger and ScalarDL Auditor with TLS configurations by using Helm Charts and cert-manager on a Kubernetes cluster as a test environment. Before starting, you should already have a Mac or Linux environment for testing. In addition, although this tutorial mentions using **minikube**, the steps described should work in any Kubernetes cluster. ## Requirements * You need to have a license key (trial license or commercial license) for ScalarDL. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). * You need to use ScalarDL 3.9 or later, which supports TLS. :::note To make Byzantine-fault detection with auditing work properly, ScalarDL Ledger and ScalarDL Auditor should be deployed and managed in different administrative domains. However, in this tutorial, we will deploy ScalarDL Ledger and ScalarDL Auditor in the same Kubernetes cluster to make the test easier. ::: ## What you'll create In this tutorial, you'll deploy the following components on a Kubernetes cluster in the following way: ``` +-----------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | [Pod] [Pod] [Pod] | | | | +-------+ +---------+ | | +---> | Envoy | ---+ +---> | Ledger | ---+ | | | +-------+ | | +---------+ | | | | | | | | | +---------+ | +-------+ | +-----------+ | +---------+ | +---------------+ | | +---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | Ledger | ---+---> | PostgreSQL | | | | | (Envoy) | | +-------+ | | (Ledger) | | +---------+ | | (For Ledger) | | | | +---------+ | | +-----------+ | | +---------------+ | | [Pod] | | +-------+ | | +---------+ | | | | +---> | Envoy | ---+ +---> | Ledger | ---+ | | +--------+ | +-------+ +---------+ | | | Client | ---+ | | +--------+ | +-------+ +---------+ | | | +---> | Envoy | ---+ +---> | Auditor | ---+ | | | | +-------+ | | +---------+ | | | | | | | | | | | +---------+ | +-------+ | +-----------+ | +---------+ | +---------------+ | | +---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | Auditor | ---+---> | PostgreSQL | | | | (Envoy) | | +-------+ | | (Auditor) | | +---------+ | | (For Auditor) | | | +---------+ | | +-----------+ | | +---------------+ | | | +-------+ | | +---------+ | | | +---> | Envoy | ---+ +---> | Auditor | ---+ | | +-------+ +---------+ | | | | +--------------------------------------------------------------------------+ +---------------------+ | | | cert-manager (create private key and certificate for Envoy and ScalarDL) | | Issuer (Private CA) | | | +--------------------------------------------------------------------------+ +---------------------+ | | | +-----------------------------------------------------------------------------------------------------------------------------+ ``` cert-manager automatically creates the following private key and certificate files for TLS connections. ``` +----------------------+ +---> | For Scalar Envoy | | +----------------------+ | | tls.key | | | tls.crt | | +----------------------+ | +-------------------------+ | +----------------------+ | Issuer (Self-signed CA) | ---(Sign certificates)---+---> | For ScalarDL Ledger | +-------------------------+ | +----------------------+ | tls.key | | | tls.key | | tls.crt | | | tls.crt | | ca.crt | | +----------------------+ +-------------------------+ | | +----------------------+ +---> | For ScalarDL Auditor | +----------------------+ | tls.key | | tls.crt | +----------------------+ ``` Scalar Helm Charts automatically mount each private key and certificate file for Envoy and ScalarDL as follows to enable TLS in each connection. You'll manually mount a root CA certificate file on the client. ``` +------------------------------------------------+ +--------------------------------------+ +-------(Normal request)-----> | Envoy for ScalarDL Ledger | ---> | ScalarDL Ledger | | +------------------------------------------------+ +--------------------------------------+ | +---(Recovery request)---> | tls.key | ---> | tls.key | | | | tls.crt | | tls.crt | | | | ca.crt (to verify tls.crt of ScalarDL Ledger) | | ca.crt (to check health) | | | +------------------------------------------------+ +--------------------------------------+ +---------------------------------------+ | | | Client | ---+ | +---------------------------------------+ | +------------------------------------------------------------------------------------------------------------------------------+ | ca.crt (to verify tls.crt of Envoy) | | | +---------------------------------------+ | | | +------------------------------------------------+ +--------------------------------------+ | +-------(Normal request)-----> | Envoy for ScalarDL Auditor | ---> | ScalarDL Auditor | ---+ +------------------------------------------------+ +--------------------------------------+ | tls.key | | tls.key | | tls.crt | | tls.crt | | ca.crt (to verify tls.crt of ScalarDL Auditor) | | ca.crt (to check health) | +------------------------------------------------+ | ca.crt (to verify tls.crt of Envoy) | +--------------------------------------+ ``` The following connections exist amongst the ScalarDL-related components: * **`Client - Envoy for ScalarDL Ledger`:** When you execute a ScalarDL API function, the client accesses Envoy for ScalarDL Ledger. * **`Client - Envoy for ScalarDL Auditor`:** When you execute a ScalarDL API function, the client accesses Envoy for ScalarDL Auditor. * **`Envoy for ScalarDL Ledger - ScalarDL Ledger`:** Envoy works as an L7 (gRPC) load balancer in front of ScalarDL Ledger. * **`Envoy for ScalarDL Auditor - ScalarDL Auditor`:** Envoy works as an L7 (gRPC) load balancer in front of ScalarDL Auditor. * **`ScalarDL Auditor - Envoy for ScalarDL Ledger (ScalarDL Ledger)`:** When ScalarDL needs to run the recovery process to keep data consistent, ScalarDL Auditor runs the request against ScalarDL Ledger via Envoy. ## Step 1. Start a Kubernetes cluster and install tools You need to prepare a Kubernetes cluster and install some tools (`kubectl`, `helm`, `cfssl`, and `cfssljson`). For more details on how to install them, see [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx). ## Step 2. Start the PostgreSQL containers ScalarDL Ledger and ScalarDL Auditor must use some type of database system as a backend database. In this tutorial, you'll use PostgreSQL. You can deploy PostgreSQL on the Kubernetes cluster as follows: 1. Add the Bitnami helm repository. ```console helm repo add bitnami https://charts.bitnami.com/bitnami ``` 1. Deploy PostgreSQL for Ledger. ```console helm install postgresql-ledger bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false \ -n default ``` 1. Deploy PostgreSQL for Auditor. ```console helm install postgresql-auditor bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false \ -n default ``` 1. Check if the PostgreSQL containers are running. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-auditor-0 1/1 Running 0 11s postgresql-ledger-0 1/1 Running 0 16s ``` ## Step 3. Create a working directory You'll create some configuration files and private key and certificate files locally. Be sure to create a working directory for those files. 1. Create a working directory. ```console mkdir -p ${HOME}/scalardl-test/ ``` ## Step 4. Deploy cert-manager and issuer resource This tutorial uses cert-manager to issue and manage private keys and certificates. You can deploy cert-manager on the Kubernetes cluster as follows: 1. Add the Jetstack helm repository. ```console helm repo add jetstack https://charts.jetstack.io ``` 1. Deploy cert-manager. ```console helm install cert-manager jetstack/cert-manager \ --create-namespace \ --set installCRDs=true \ -n cert-manager ``` 1. Check if the cert-manager containers are running. ```console kubectl get pod -n cert-manager ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE cert-manager-6dc66985d4-6lvtt 1/1 Running 0 26s cert-manager-cainjector-c7d4dbdd9-xlrpn 1/1 Running 0 26s cert-manager-webhook-847d7676c9-ckcz2 1/1 Running 0 26s ``` 1. Change the working directory to `${HOME}/scalardl-test/`. ```console cd ${HOME}/scalardl-test/ ``` 1. Create a custom values file for private CA (`private-ca-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/private-ca-custom-values.yaml apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: self-signed-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: self-signed-ca-cert spec: isCA: true commonName: self-signed-ca secretName: self-signed-ca-cert-secret privateKey: algorithm: ECDSA size: 256 issuerRef: name: self-signed-issuer kind: Issuer group: cert-manager.io --- apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: self-signed-ca spec: ca: secretName: self-signed-ca-cert-secret EOF ``` 1. Deploy self-signed CA. ```console kubectl apply -f ./private-ca-custom-values.yaml ``` 1. Check if the issuer resources are `True`. ```console kubectl get issuer ``` [Command execution result] ```console NAME READY AGE self-signed-ca True 6s self-signed-issuer True 6s ``` ## Step 5. Create database schemas for ScalarDL Ledger and ScalarDL Auditor by using Helm Charts You'll deploy two ScalarDL Schema Loader pods on the Kubernetes cluster by using Helm Charts. The ScalarDL Schema Loader will create the database schemas for ScalarDL Ledger and Auditor in PostgreSQL. 1. Add the Scalar Helm Charts repository. ```console helm repo add scalar-labs https://scalar-labs.github.io/helm-charts ``` 1. Create a custom values file for ScalarDL Schema Loader for Ledger (`schema-loader-ledger-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/schema-loader-ledger-custom-values.yaml schemaLoading: schemaType: "ledger" databaseProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-ledger.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DL_LEDGER_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DL_LEDGER_POSTGRES_PASSWORD} scalar.db.storage=jdbc secretName: "schema-ledger-credentials-secret" EOF ``` 1. Create a custom values file for ScalarDL Schema Loader for Auditor (`schema-loader-auditor-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/schema-loader-auditor-custom-values.yaml schemaLoading: schemaType: "auditor" databaseProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-auditor.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DL_AUDITOR_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DL_AUDITOR_POSTGRES_PASSWORD} scalar.db.storage=jdbc secretName: "schema-auditor-credentials-secret" EOF ``` 1. Create a secret resource named `schema-ledger-credentials-secret` that includes a username and password for PostgreSQL for ScalarDL Ledger. ```console kubectl create secret generic schema-ledger-credentials-secret \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_PASSWORD=postgres \ -n default ``` 1. Create a secret resource named `schema-auditor-credentials-secret` that includes a username and password for PostgreSQL for ScalarDL Auditor. ```console kubectl create secret generic schema-auditor-credentials-secret \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_PASSWORD=postgres \ -n default ``` 1. Set the chart version of ScalarDL Schema Loader. ```console SCALAR_DL_VERSION=3.9.1 SCALAR_DL_SCHEMA_LOADER_CHART_VERSION=$(helm search repo scalar-labs/schema-loading -l | grep -F "${SCALAR_DL_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` 1. Deploy ScalarDL Schema Loader for ScalarDL Ledger. ```console helm install schema-loader-ledger scalar-labs/schema-loading -f ${HOME}/scalardl-test/schema-loader-ledger-custom-values.yaml --version ${SCALAR_DL_SCHEMA_LOADER_CHART_VERSION} -n default ``` 1. Deploy ScalarDL Schema Loader for ScalarDL Auditor. ```console helm install schema-loader-auditor scalar-labs/schema-loading -f ${HOME}/scalardl-test/schema-loader-auditor-custom-values.yaml --version ${SCALAR_DL_SCHEMA_LOADER_CHART_VERSION} -n default ``` 1. Check if the ScalarDL Schema Loader pods are deployed with the status `Completed`. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-auditor-0 1/1 Running 0 2m56s postgresql-ledger-0 1/1 Running 0 3m1s schema-loader-auditor-schema-loading-dvc5r 0/1 Completed 0 6s schema-loader-ledger-schema-loading-mtllb 0/1 Completed 0 10s ``` If the status of the ScalarDL Schema Loader pods are **ContainerCreating** or **Running**, wait for the `STATUS` column for those pods to show as `Completed`. ## Step 6. Deploy ScalarDL Ledger and ScalarDL Auditor on the Kubernetes cluster by using Helm Charts 1. Set your license key and certificate as environment variables. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). For details about the value of `` and ``, see [How to Configure a License Key](../scalar-licensing/index.mdx). ```console SCALAR_DL_LEDGER_LICENSE_KEY='' SCALAR_DL_LEDGER_LICENSE_CHECK_CERT_PEM='' SCALAR_DL_AUDITOR_LICENSE_KEY='' SCALAR_DL_AUDITOR_LICENSE_CHECK_CERT_PEM='' ``` 1. Create a custom values file for ScalarDL Ledger (`scalardl-ledger-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/scalardl-ledger-custom-values.yaml envoy: tls: downstream: enabled: true certManager: enabled: true issuerRef: name: self-signed-ca dnsNames: - envoy.scalar.example.com upstream: enabled: true overrideAuthority: "ledger.scalardl.example.com" ledger: image: repository: "ghcr.io/scalar-labs/scalardl-ledger-byol" ledgerProperties: | ### Storage configurations scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://postgresql-ledger.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DL_LEDGER_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DL_LEDGER_POSTGRES_PASSWORD} ### Ledger configurations scalar.dl.ledger.proof.enabled=true scalar.dl.ledger.auditor.enabled=true scalar.dl.ledger.authentication.method=hmac scalar.dl.ledger.authentication.hmac.cipher_key=${env:SCALAR_DL_LEDGER_HMAC_CIPHER_KEY} scalar.dl.ledger.servers.authentication.hmac.secret_key=${env:SCALAR_DL_LEDGER_HMAC_SECRET_KEY} ### TLS configurations scalar.dl.ledger.server.tls.enabled=true scalar.dl.ledger.server.tls.cert_chain_path=/tls/scalardl-ledger/certs/tls.crt scalar.dl.ledger.server.tls.private_key_path=/tls/scalardl-ledger/certs/tls.key ### License key configurations scalar.dl.licensing.license_key=${env:SCALAR_DL_LEDGER_LICENSE_KEY} scalar.dl.licensing.license_check_cert_pem=${env:SCALAR_DL_LEDGER_LICENSE_CHECK_CERT_PEM} tls: enabled: true overrideAuthority: "ledger.scalardl.example.com" certManager: enabled: true issuerRef: name: self-signed-ca dnsNames: - ledger.scalardl.example.com secretName: "ledger-credentials-secret" EOF ``` 1. Create a custom values file for ScalarDL Auditor (`scalardl-auditor-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/scalardl-auditor-custom-values.yaml envoy: tls: downstream: enabled: true certManager: enabled: true issuerRef: name: self-signed-ca dnsNames: - envoy.scalar.example.com upstream: enabled: true overrideAuthority: "auditor.scalardl.example.com" auditor: image: repository: "ghcr.io/scalar-labs/scalardl-auditor-byol" auditorProperties: | ### Storage configurations scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://postgresql-auditor.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DL_AUDITOR_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DL_AUDITOR_POSTGRES_PASSWORD} ### Auditor configurations scalar.dl.auditor.ledger.host=scalardl-ledger-envoy.default.svc.cluster.local scalar.dl.auditor.authentication.method=hmac scalar.dl.auditor.authentication.hmac.cipher_key=${env:SCALAR_DL_AUDITOR_HMAC_CIPHER_KEY} scalar.dl.auditor.servers.authentication.hmac.secret_key=${env:SCALAR_DL_AUDITOR_HMAC_SECRET_KEY} ### TLS configurations scalar.dl.auditor.server.tls.enabled=true scalar.dl.auditor.server.tls.cert_chain_path=/tls/scalardl-auditor/certs/tls.crt scalar.dl.auditor.server.tls.private_key_path=/tls/scalardl-auditor/certs/tls.key scalar.dl.auditor.tls.enabled=true scalar.dl.auditor.tls.ca_root_cert_path=/tls/scalardl-ledger/certs/ca.crt scalar.dl.auditor.tls.override_authority=envoy.scalar.example.com ### License key configurations scalar.dl.licensing.license_key=${env:SCALAR_DL_AUDITOR_LICENSE_KEY} scalar.dl.licensing.license_check_cert_pem=${env:SCALAR_DL_AUDITOR_LICENSE_CHECK_CERT_PEM} tls: enabled: true overrideAuthority: "auditor.scalardl.example.com" certManager: enabled: true issuerRef: name: self-signed-ca dnsNames: - auditor.scalardl.example.com secretName: "auditor-credentials-secret" EOF ``` 1. Create a secret resource named `ledger-credentials-secret` that includes credentials and a license key. ```console kubectl create secret generic ledger-credentials-secret \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_PASSWORD=postgres \ --from-literal=SCALAR_DL_LEDGER_HMAC_CIPHER_KEY=ledger-hmac-cipher-key \ --from-literal=SCALAR_DL_LEDGER_HMAC_SECRET_KEY=scalardl-hmac-secret-key \ --from-literal=SCALAR_DL_LEDGER_LICENSE_KEY="${SCALAR_DL_LEDGER_LICENSE_KEY}" \ --from-file=SCALAR_DL_LEDGER_LICENSE_CHECK_CERT_PEM=<(echo ${SCALAR_DL_LEDGER_LICENSE_CHECK_CERT_PEM} | sed 's/\\n/\ /g') \ -n default ``` 1. Create a secret resource named `auditor-credentials-secret` that includes credentials and a license key. ```console kubectl create secret generic auditor-credentials-secret \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_PASSWORD=postgres \ --from-literal=SCALAR_DL_AUDITOR_HMAC_CIPHER_KEY=auditor-hmac-cipher-key \ --from-literal=SCALAR_DL_AUDITOR_HMAC_SECRET_KEY=scalardl-hmac-secret-key \ --from-literal=SCALAR_DL_AUDITOR_LICENSE_KEY="${SCALAR_DL_AUDITOR_LICENSE_KEY}" \ --from-file=SCALAR_DL_AUDITOR_LICENSE_CHECK_CERT_PEM=<(echo ${SCALAR_DL_AUDITOR_LICENSE_CHECK_CERT_PEM} | sed 's/\\n/\ /g') \ -n default ``` 1. Create a secret resource named `auditor-keys` to disable the `digital-signature` authentication method. In this tutorial, you'll use the `hmac` authentication method instead of `digital-signature`. ```console kubectl create secret generic auditor-keys \ --from-literal=tls.key=dummy-data-to-disable-digital-signature-method \ --from-literal=certificate=dummy-data-to-disable-digital-signature-method \ -n default ``` Note: If you use `hmac` as an authentication method, you have to create a dummy secret `auditor-key` to disable `digital-signature` on the Helm Chart side. 1. Set the chart version of ScalarDL Ledger and ScalarDL Auditor. ```console SCALAR_DL_LEDGER_CHART_VERSION=$(helm search repo scalar-labs/scalardl -l | grep -v -e "scalar-labs/scalardl-audit" | grep -F "${SCALAR_DL_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) SCALAR_DL_AUDITOR_CHART_VERSION=$(helm search repo scalar-labs/scalardl-audit -l | grep -F "${SCALAR_DL_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` 1. Deploy ScalarDL Ledger. ```console helm install scalardl-ledger scalar-labs/scalardl -f ${HOME}/scalardl-test/scalardl-ledger-custom-values.yaml --version ${SCALAR_DL_LEDGER_CHART_VERSION} -n default ``` 1. Deploy ScalarDL Auditor. ```console helm install scalardl-auditor scalar-labs/scalardl-audit -f ${HOME}/scalardl-test/scalardl-auditor-custom-values.yaml --version ${SCALAR_DL_AUDITOR_CHART_VERSION} -n default ``` 1. Check if the ScalarDL Ledger and ScalarDL Auditor pods are deployed. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-auditor-0 1/1 Running 0 14m postgresql-ledger-0 1/1 Running 0 14m scalardl-auditor-auditor-5b885ff4c8-fwkpf 1/1 Running 0 18s scalardl-auditor-auditor-5b885ff4c8-g69cb 1/1 Running 0 18s scalardl-auditor-auditor-5b885ff4c8-nsmnq 1/1 Running 0 18s scalardl-auditor-envoy-689bcbdf65-5mn6v 1/1 Running 0 18s scalardl-auditor-envoy-689bcbdf65-fpq8j 1/1 Running 0 18s scalardl-auditor-envoy-689bcbdf65-lsz2t 1/1 Running 0 18s scalardl-ledger-envoy-547bbf7546-n7p5x 1/1 Running 0 26s scalardl-ledger-envoy-547bbf7546-p8nwp 1/1 Running 0 26s scalardl-ledger-envoy-547bbf7546-pskpb 1/1 Running 0 26s scalardl-ledger-ledger-6db5dc8774-5zsbj 1/1 Running 0 26s scalardl-ledger-ledger-6db5dc8774-vnmrw 1/1 Running 0 26s scalardl-ledger-ledger-6db5dc8774-wpjvs 1/1 Running 0 26s schema-loader-auditor-schema-loading-dvc5r 0/1 Completed 0 11m schema-loader-ledger-schema-loading-mtllb 0/1 Completed 0 11m ``` If the ScalarDL Ledger and ScalarDL Auditor pods are deployed properly, the `STATUS` column for those pods will be displayed as `Running`. 1. Check if the ScalarDL Ledger and ScalarDL Auditor services are deployed. ```console kubectl get svc -n default ``` [Command execution result] ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 47d postgresql-auditor ClusterIP 10.107.9.78 5432/TCP 15m postgresql-auditor-hl ClusterIP None 5432/TCP 15m postgresql-ledger ClusterIP 10.108.241.181 5432/TCP 15m postgresql-ledger-hl ClusterIP None 5432/TCP 15m scalardl-auditor-envoy ClusterIP 10.100.61.202 40051/TCP,40052/TCP 55s scalardl-auditor-envoy-metrics ClusterIP 10.99.6.227 9001/TCP 55s scalardl-auditor-headless ClusterIP None 40051/TCP,40053/TCP,40052/TCP 55s scalardl-auditor-metrics ClusterIP 10.108.1.147 8080/TCP 55s scalardl-ledger-envoy ClusterIP 10.101.191.116 50051/TCP,50052/TCP 61s scalardl-ledger-envoy-metrics ClusterIP 10.106.52.103 9001/TCP 61s scalardl-ledger-headless ClusterIP None 50051/TCP,50053/TCP,50052/TCP 61s scalardl-ledger-metrics ClusterIP 10.99.122.106 8080/TCP 61s ``` If the ScalarDL Ledger and ScalarDL Auditor services are deployed properly, you can see private IP addresses in the `CLUSTER-IP` column. :::note The `CLUSTER-IP` values for `scalardl-ledger-headless`, `scalardl-auditor-headless`, `postgresql-ledger-hl`, and `postgresql-auditor-hl` are `None` since they have no IP addresses. ::: ## Step 7. Start a client container You'll use the CA certificate file in a client container. Therefore, you'll need to create a secret resource and mount it to the client container. 1. Create a secret resource named `client-ca-cert`. ```console kubectl create secret generic client-ca-cert --from-file=ca.crt=<(kubectl get secret self-signed-ca-cert-secret -o "jsonpath={.data['ca\.crt']}" | base64 -d) -n default ``` 1. Create a manifest file for a client pod (`scalardl-client-pod.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/scalardl-client-pod.yaml apiVersion: v1 kind: Pod metadata: name: "scalardl-client" spec: containers: - name: scalardl-client image: eclipse-temurin:8-jdk command: ['sleep'] args: ['inf'] env: - name: SCALAR_DL_VERSION value: SCALAR_DL_CLIENT_POD_SCALAR_DL_VERSION volumeMounts: - name: "client-ca-cert" mountPath: "/certs/" readOnly: true volumes: - name: "client-ca-cert" secret: secretName: "client-ca-cert" restartPolicy: Never EOF ``` 1. Set the ScalarDL version in the manifest file. ```console sed -i s/SCALAR_DL_CLIENT_POD_SCALAR_DL_VERSION/${SCALAR_DL_VERSION}/ ${HOME}/scalardl-test/scalardl-client-pod.yaml ``` 1. Deploy the client pod. ```console kubectl apply -f ${HOME}/scalardl-test/scalardl-client-pod.yaml -n default ``` 1. Check if the client container is running. ```console kubectl get pod scalardl-client -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE scalardl-client 1/1 Running 0 4s ``` ## Step 8. Run ScalarDL sample contracts in the client container The following explains the minimum steps needed to run sample contracts. For more details about ScalarDL Ledger and ScalarDL Auditor, see the following: * [Getting Started with ScalarDL](https://scalardl.scalar-labs.com/docs/latest/getting-started/) * [Getting Started with ScalarDL Auditor](https://scalardl.scalar-labs.com/docs/latest/getting-started-auditor/) 1. Run bash in the client container. ```console kubectl exec -it scalardl-client -n default -- bash ``` The commands in the following steps must be run in the client container. 1. Install the git, curl, and unzip commands in the client container. ```console apt update && apt install -y git curl unzip ``` 1. Clone the ScalarDL Java Client SDK git repository. ```console git clone https://github.com/scalar-labs/scalardl-java-client-sdk.git ``` 1. Change the working directory to `scalardl-java-client-sdk/`. ```console cd scalardl-java-client-sdk/ ``` ```console pwd ``` [Command execution result] ```console /scalardl-java-client-sdk ``` 1. Change the branch to the version you're using. ```console git checkout -b v${SCALAR_DL_VERSION} refs/tags/v${SCALAR_DL_VERSION} ``` 1. Build the sample contracts. ```console ./gradlew assemble ``` 1. Download the CLI tools for ScalarDL from [ScalarDL Java Client SDK Releases](https://github.com/scalar-labs/scalardl-java-client-sdk/releases). ```console curl -OL https://github.com/scalar-labs/scalardl-java-client-sdk/releases/download/v${SCALAR_DL_VERSION}/scalardl-java-client-sdk-${SCALAR_DL_VERSION}.zip ``` 1. Unzip the `scalardl-java-client-sdk-${SCALAR_DL_VERSION}.zip` file. ```console unzip ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}.zip ``` 1. Create a configuration file named `client.properties` to access ScalarDL Ledger and ScalarDL Auditor on the Kubernetes cluster. ```console cat << 'EOF' > client.properties # Ledger configuration scalar.dl.client.server.host=scalardl-ledger-envoy.default.svc.cluster.local scalar.dl.client.tls.enabled=true scalar.dl.client.tls.ca_root_cert_path=/certs/ca.crt scalar.dl.client.tls.override_authority=envoy.scalar.example.com # Auditor configuration scalar.dl.client.auditor.enabled=true scalar.dl.client.auditor.host=scalardl-auditor-envoy.default.svc.cluster.local scalar.dl.client.auditor.tls.enabled=true scalar.dl.client.auditor.tls.ca_root_cert_path=/certs/ca.crt scalar.dl.client.auditor.tls.override_authority=envoy.scalar.example.com # Client configuration scalar.dl.client.authentication_method=hmac scalar.dl.client.entity.id=client scalar.dl.client.entity.identity.hmac.secret_key=scalardl-hmac-client-secert-key EOF ``` 1. Register the client secret. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl register-secret --config ./client.properties ``` 1. Register the sample contract `StateUpdater`. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl register-contract --config ./client.properties --contract-id StateUpdater --contract-binary-name com.org1.contract.StateUpdater --contract-class-file ./build/classes/java/main/com/org1/contract/StateUpdater.class ``` 1. Register the sample contract `StateReader`. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl register-contract --config ./client.properties --contract-id StateReader --contract-binary-name com.org1.contract.StateReader --contract-class-file ./build/classes/java/main/com/org1/contract/StateReader.class ``` 1. Register the contract `ValidateLedger` to execute a validate request. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl register-contract --config ./client.properties --contract-id validate-ledger --contract-binary-name com.scalar.dl.client.contract.ValidateLedger --contract-class-file ./build/classes/java/main/com/scalar/dl/client/contract/ValidateLedger.class ``` 1. Execute the contract `StateUpdater`. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl execute-contract --config ./client.properties --contract-id StateUpdater --contract-argument '{"asset_id": "test_asset", "state": 3}' ``` This sample contract updates the `state` (value) of the asset named `test_asset` to `3`. 1. Execute the contract `StateReader`. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl execute-contract --config ./client.properties --contract-id StateReader --contract-argument '{"asset_id": "test_asset"}' ``` [Command execution result] ```console Contract result: { "id" : "test_asset", "age" : 0, "output" : { "state" : 3 } } ``` ### Reference * If the asset data is not tampered with, running the `execute-contract` command to request contract execution will return `OK` as a result. * If the asset data is tampered with (for example, if the `state` value in the database is tampered with), running the `execute-contract` command to request contract execution will return a value other than `OK` (for example, `INCONSISTENT_STATES`) as a result. See the following as an example for how ScalarDL detects data tampering. [Command execution result (if the asset data is tampered with)] ```console { "status_code" : "INCONSISTENT_STATES", "error_message" : "The results from Ledger and Auditor don't match" } ``` 1. Execute a validation request for the asset. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl validate-ledger --config ./client.properties --asset-id "test_asset" ``` [Command execution result] ```console { "status_code" : "OK", "Ledger" : { "id" : "test_asset", "age" : 0, "nonce" : "3533427d-03cf-41d1-bf95-4d31eb0cb24d", "hash" : "FiquvtPMKLlxKf4VGoccSAGsi9ptn4ozYVVTwdSzEQ0=", "signature" : "MEYCIQDiiXqzw6K+Ml4uvn8rK43o5wHWESU3hoXnZPi6/OeKVwIhAM+tFBcapl6zg47Uq0Uc8nVNGWNHZLBDBGve3F0xkzTR" }, "Auditor" : { "id" : "test_asset", "age" : 0, "nonce" : "3533427d-03cf-41d1-bf95-4d31eb0cb24d", "hash" : "FiquvtPMKLlxKf4VGoccSAGsi9ptn4ozYVVTwdSzEQ0=", "signature" : "MEUCIQDLsfUR2PmxSvfpL3YvHJUkz00RDpjCdctkroZKXE8d5QIgH73FQH2e11jfnynD00Pp9DrIG1vYizxDsvxUsMPo9IU=" } } ``` ### Reference * If the asset data is not tampered with, running the `validate-ledger` command to request validation will return `OK` as the result. * If the asset data is tampered with (for example, if the `state` value in the database is tampered with), running the `validate-ledger` command to request validation will return a value other than `OK` (for example, `INVALID_OUTPUT`) as a result. See the following as an example for how ScalarDL detects data tampering. [Command execution result (if the asset data is tampered with)] ```console { "status_code" : "INCONSISTENT_STATES", "error_message" : "The results from Ledger and Auditor don't match" } ``` 1. Exit from the client container. ```console exit ``` ## Step 9. Delete all resources After completing the ScalarDL Ledger and ScalarDL Auditor tests on the Kubernetes cluster, remove all resources. 1. Uninstall ScalarDL Ledger, ScalarDL Auditor, ScalarDL Schema Loader, and PostgreSQL. ```console helm uninstall -n default scalardl-ledger schema-loader-ledger postgresql-ledger scalardl-auditor schema-loader-auditor postgresql-auditor ``` 1. Remove the self-signed CA. ``` kubectl delete -f ./private-ca-custom-values.yaml ``` 1. Uninstall cert-manager. ```console helm uninstall -n cert-manager cert-manager ``` 1. Remove the client container. ``` kubectl delete pod scalardl-client --grace-period 0 -n default ``` 1. Remove the secret resources. ``` kubectl delete secrets self-signed-ca-cert-secret schema-ledger-credentials-secret schema-auditor-credentials-secret scalardl-ledger-tls-cert scalardl-ledger-envoy-tls-cert scalardl-auditor-tls-cert scalardl-auditor-envoy-tls-cert ledger-credentials-secret auditor-credentials-secret client-ca-cert auditor-keys ``` 1. Remove the namespace `cert-manager`. ``` kubectl delete ns cert-manager ``` 1. Remove the working directory and sample files (configuration files). ```console cd ${HOME} ``` ```console rm -rf ${HOME}/scalardl-test/ ``` ## Further reading You can see how to get started with monitoring or logging for Scalar products in the following tutorials: * [Getting Started with Helm Charts (Monitoring using Prometheus Operator)](getting-started-monitoring.mdx) * [Getting Started with Helm Charts (Logging using Loki Stack)](getting-started-logging.mdx) * [Getting Started with Helm Charts (Scalar Manager)](getting-started-scalar-manager.mdx) ================================================ FILE: docs/helm-charts/getting-started-scalardl-auditor-tls.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Getting Started with Helm Charts (ScalarDL Ledger and Auditor with TLS / Auditor Mode) This tutorial explains how to get started with ScalarDL Ledger and ScalarDL Auditor with TLS configurations by using Helm Charts on a Kubernetes cluster as a test environment. Before starting, you should already have a Mac or Linux environment for testing. In addition, although this tutorial mentions using **minikube**, the steps described should work in any Kubernetes cluster. ## Requirements * You need to have a license key (trial license or commercial license) for ScalarDL. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). * You need to use ScalarDL 3.9 or later, which supports TLS. :::note To make Byzantine fault detection with auditing work properly, ScalarDL Ledger and ScalarDL Auditor should be deployed and managed in different administrative domains. However, in this tutorial, we will deploy ScalarDL Ledger and ScalarDL Auditor in the same Kubernetes cluster to make the test easier. ::: ## What you'll create In this tutorial, you'll deploy the following components on a Kubernetes cluster in the following way: ``` +-----------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | [Pod] [Pod] [Pod] | | | | +-------+ +---------+ | | +---> | Envoy | ---+ +---> | Ledger | ---+ | | | +-------+ | | +---------+ | | | | | | | | | +---------+ | +-------+ | +-----------+ | +---------+ | +---------------+ | | +---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | Ledger | ---+---> | PostgreSQL | | | | | (Envoy) | | +-------+ | | (Ledger) | | +---------+ | | (For Ledger) | | | | +---------+ | | +-----------+ | | +---------------+ | | [Pod] | | +-------+ | | +---------+ | | | | +---> | Envoy | ---+ +---> | Ledger | ---+ | | +--------+ | +-------+ +---------+ | | | Client | ---+ | | +--------+ | +-------+ +---------+ | | | +---> | Envoy | ---+ +---> | Auditor | ---+ | | | | +-------+ | | +---------+ | | | | | | | | | | | +---------+ | +-------+ | +-----------+ | +---------+ | +---------------+ | | +---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | Auditor | ---+---> | PostgreSQL | | | | (Envoy) | | +-------+ | | (Auditor) | | +---------+ | | (For Auditor) | | | +---------+ | | +-----------+ | | +---------------+ | | | +-------+ | | +---------+ | | | +---> | Envoy | ---+ +---> | Auditor | ---+ | | +-------+ +---------+ | | | +-----------------------------------------------------------------------------------------------------------------------------+ ``` You'll also create the following private key and certificate files for TLS connections. ``` +----------------------+ +---> | For Scalar Envoy | | +----------------------+ | | envoy-key.pem | | | envoy.pem | | +----------------------+ | +----------------------+ | +----------------------+ | Self-signed CA | ---(Sign certificates)---+---> | For ScalarDL Ledger | +----------------------+ | +----------------------+ | ca-key.pem | | | ledger-key.pem | | ca.pem | | | ledger.pem | +----------------------+ | +----------------------+ | | +----------------------+ +---> | For ScalarDL Auditor | +----------------------+ | auditor-key.pem | | auditor.pem | +----------------------+ ``` You'll set each private key and certificate file as follows to enable TLS in each connection. ``` +--------------------------------+ +--------------------------------+ +-------(Normal request)-----> | Envoy for ScalarDL Ledger | ---> | ScalarDL Ledger | | +--------------------------------+ +--------------------------------+ | +---(Recovery request)---> | envoy-key.pem | ---> | ledger-key.pem | | | | envoy.pem | | ledger.pem | | | | ca.pem (to verify ledger.pem) | | ca.pem (used for health check) | | | +--------------------------------+ +--------------------------------+ +--------------------------------+ | | | Client | ---+ | +--------------------------------+ | +--------------------------------------------------------------------------------------------------------+ | ca.pem (to verify envoy.pem) | | | +--------------------------------+ | | | +--------------------------------+ +--------------------------------+ | +-------(Normal request)-----> | Envoy for ScalarDL Auditor | ---> | ScalarDL Auditor | ---+ +--------------------------------+ +--------------------------------+ | envoy-key.pem | | auditor-key.pem | | envoy.pem | | auditor.pem | | ca.pem (to verify auditor.pem) | | ca.pem (used for health check) | +--------------------------------+ | ca.pem (to verify ledger.pem) | +--------------------------------+ ``` The following connections exist amongst the ScalarDL-related components: * **`Client - Envoy for ScalarDL Ledger`:** When you execute a ScalarDL API function, the client accesses Envoy for ScalarDL Ledger. * **`Client - Envoy for ScalarDL Auditor`:** When you execute a ScalarDL API function, the client accesses Envoy for ScalarDL Auditor. * **`Envoy for ScalarDL Ledger - ScalarDL Ledger`:** Envoy works as an L7 (gRPC) load balancer in front of ScalarDL Ledger. * **`Envoy for ScalarDL Auditor - ScalarDL Auditor`:** Envoy works as an L7 (gRPC) load balancer in front of ScalarDL Auditor. * **`ScalarDL Auditor - Envoy for ScalarDL Ledger (ScalarDL Ledger)`:** When ScalarDL needs to run the recovery process to keep data consistent, ScalarDL Auditor runs the request against ScalarDL Ledger via Envoy. ## Step 1. Start a Kubernetes cluster and install tools You need to prepare a Kubernetes cluster and install some tools (`kubectl`, `helm`, `cfssl`, and `cfssljson`). For more details on how to install them, see [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx). ## Step 2. Start the PostgreSQL containers ScalarDL Ledger and ScalarDL Auditor must use some type of database system as a backend database. In this tutorial, you'll use PostgreSQL. You can deploy PostgreSQL on the Kubernetes cluster as follows: 1. Add the Bitnami helm repository. ```console helm repo add bitnami https://charts.bitnami.com/bitnami ``` 1. Deploy PostgreSQL for Ledger. ```console helm install postgresql-ledger bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false \ -n default ``` 1. Deploy PostgreSQL for Auditor. ```console helm install postgresql-auditor bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false \ -n default ``` 1. Check if the PostgreSQL containers are running. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-auditor-0 1/1 Running 0 11s postgresql-ledger-0 1/1 Running 0 16s ``` ## Step 3. Create a working directory You'll create some configuration files and private key and certificate files locally. Be sure to create a working directory for those files. 1. Create a working directory. ```console mkdir -p ${HOME}/scalardl-test/certs/ ``` ## Step 4. Create private key and certificate files You'll create private key and a certificate files. 1. Change the working directory to `${HOME}/scalardl-test/certs/`. ```console cd ${HOME}/scalardl-test/certs/ ``` 1. Create a JSON file that includes CA information. ```console cat << 'EOF' > ${HOME}/scalardl-test/certs/ca.json { "CN": "scalar-test-ca", "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "JP", "ST": "Tokyo", "L": "Shinjuku", "O": "Scalar Test CA" } ] } EOF ``` 1. Create the CA private key and certificate files. ```console cfssl gencert -initca ca.json | cfssljson -bare ca ``` 1. Create a JSON file that includes CA configurations. ```console cat << 'EOF' > ${HOME}/scalardl-test/certs/ca-config.json { "signing": { "default": { "expiry": "87600h" }, "profiles": { "scalar-test-ca": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth" ] } } } } EOF ``` 1. Create a JSON file that includes Envoy information. ```console cat << 'EOF' > ${HOME}/scalardl-test/certs/envoy.json { "CN": "scalar-envoy", "hosts": [ "envoy.scalar.example.com", "localhost" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "JP", "ST": "Tokyo", "L": "Shinjuku", "O": "Scalar Envoy Test" } ] } EOF ``` 1. Create a JSON file that includes ScalarDL Ledger information. ```console cat << 'EOF' > ${HOME}/scalardl-test/certs/ledger.json { "CN": "scalardl-ledger", "hosts": [ "ledger.scalardl.example.com", "localhost" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "JP", "ST": "Tokyo", "L": "Shinjuku", "O": "ScalarDL Ledger Test" } ] } EOF ``` 1. Create a JSON file that includes ScalarDL Auditor information. ```console cat << 'EOF' > ${HOME}/scalardl-test/certs/auditor.json { "CN": "scalardl-auditor", "hosts": [ "auditor.scalardl.example.com", "localhost" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "JP", "ST": "Tokyo", "L": "Shinjuku", "O": "ScalarDL Auditor Test" } ] } EOF ``` 1. Create private key and certificate files for Envoy. ```console cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile scalar-test-ca envoy.json | cfssljson -bare envoy ``` 1. Create private key and certificate files for ScalarDL Ledger. ```console cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile scalar-test-ca ledger.json | cfssljson -bare ledger ``` 1. Create private key and certificate files for ScalarDL Auditor. ```console cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile scalar-test-ca auditor.json | cfssljson -bare auditor ``` 1. Confirm that the private key and certificate files were created. ```console ls -1 ``` [Command execution result] ```console auditor-key.pem auditor.csr auditor.json auditor.pem ca-config.json ca-key.pem ca.csr ca.json ca.pem envoy-key.pem envoy.csr envoy.json envoy.pem ledger-key.pem ledger.csr ledger.json ledger.pem ``` ## Step 5. Create database schemas for ScalarDL Ledger and ScalarDL Auditor by using Helm Charts You'll deploy two ScalarDL Schema Loader pods on the Kubernetes cluster by using Helm Charts. The ScalarDL Schema Loader will create the database schemas for ScalarDL Ledger and Auditor in PostgreSQL. 1. Change the working directory to `${HOME}/scalardl-test/`. ```console cd ${HOME}/scalardl-test/ ``` 1. Add the Scalar Helm Charts repository. ```console helm repo add scalar-labs https://scalar-labs.github.io/helm-charts ``` 1. Create a custom values file for ScalarDL Schema Loader for Ledger (`schema-loader-ledger-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/schema-loader-ledger-custom-values.yaml schemaLoading: schemaType: "ledger" databaseProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-ledger.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DL_LEDGER_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DL_LEDGER_POSTGRES_PASSWORD} scalar.db.storage=jdbc secretName: "schema-ledger-credentials-secret" EOF ``` 1. Create a custom values file for ScalarDL Schema Loader for Auditor (`schema-loader-auditor-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/schema-loader-auditor-custom-values.yaml schemaLoading: schemaType: "auditor" databaseProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-auditor.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DL_AUDITOR_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DL_AUDITOR_POSTGRES_PASSWORD} scalar.db.storage=jdbc secretName: "schema-auditor-credentials-secret" EOF ``` 1. Create a secret resource named `schema-ledger-credentials-secret` that includes a username and password for PostgreSQL for ScalarDL Ledger. ```console kubectl create secret generic schema-ledger-credentials-secret \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_PASSWORD=postgres \ -n default ``` 1. Create a secret resource named `schema-auditor-credentials-secret` that includes a username and password for PostgreSQL for ScalarDL Auditor. ```console kubectl create secret generic schema-auditor-credentials-secret \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_PASSWORD=postgres \ -n default ``` 1. Set the chart version of ScalarDL Schema Loader. ```console SCALAR_DL_VERSION=3.9.1 SCALAR_DL_SCHEMA_LOADER_CHART_VERSION=$(helm search repo scalar-labs/schema-loading -l | grep -F "${SCALAR_DL_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` 1. Deploy ScalarDL Schema Loader for ScalarDL Ledger. ```console helm install schema-loader-ledger scalar-labs/schema-loading -f ${HOME}/scalardl-test/schema-loader-ledger-custom-values.yaml --version ${SCALAR_DL_SCHEMA_LOADER_CHART_VERSION} -n default ``` 1. Deploy ScalarDL Schema Loader for ScalarDL Auditor. ```console helm install schema-loader-auditor scalar-labs/schema-loading -f ${HOME}/scalardl-test/schema-loader-auditor-custom-values.yaml --version ${SCALAR_DL_SCHEMA_LOADER_CHART_VERSION} -n default ``` 1. Check if the ScalarDL Schema Loader pods are deployed with the status `Completed`. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-auditor-0 1/1 Running 0 2m56s postgresql-ledger-0 1/1 Running 0 3m1s schema-loader-auditor-schema-loading-dvc5r 0/1 Completed 0 6s schema-loader-ledger-schema-loading-mtllb 0/1 Completed 0 10s ``` If the status of the ScalarDL Schema Loader pods are **ContainerCreating** or **Running**, wait for the `STATUS` column for those pods to show as `Completed`. ## Step 6. Deploy ScalarDL Ledger and ScalarDL Auditor on the Kubernetes cluster by using Helm Charts 1. Set your license key and certificate as environment variables. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). For details about the value of `` and ``, see [How to Configure a License Key](../scalar-licensing/index.mdx). ```console SCALAR_DL_LEDGER_LICENSE_KEY='' SCALAR_DL_LEDGER_LICENSE_CHECK_CERT_PEM='' SCALAR_DL_AUDITOR_LICENSE_KEY='' SCALAR_DL_AUDITOR_LICENSE_CHECK_CERT_PEM='' ``` 1. Create a custom values file for ScalarDL Ledger (`scalardl-ledger-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/scalardl-ledger-custom-values.yaml envoy: tls: downstream: enabled: true certChainSecret: "envoy-tls-cert" privateKeySecret: "envoy-tls-key" upstream: enabled: true overrideAuthority: "ledger.scalardl.example.com" caRootCertSecret: "scalardl-ledger-tls-ca" ledger: image: repository: "ghcr.io/scalar-labs/scalardl-ledger-byol" ledgerProperties: | ### Storage configurations scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://postgresql-ledger.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DL_LEDGER_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DL_LEDGER_POSTGRES_PASSWORD} ### Ledger configurations scalar.dl.ledger.proof.enabled=true scalar.dl.ledger.auditor.enabled=true scalar.dl.ledger.authentication.method=hmac scalar.dl.ledger.authentication.hmac.cipher_key=${env:SCALAR_DL_LEDGER_HMAC_CIPHER_KEY} scalar.dl.ledger.servers.authentication.hmac.secret_key=${env:SCALAR_DL_LEDGER_HMAC_SECRET_KEY} ### TLS configurations scalar.dl.ledger.server.tls.enabled=true scalar.dl.ledger.server.tls.cert_chain_path=/tls/scalardl-ledger/certs/tls.crt scalar.dl.ledger.server.tls.private_key_path=/tls/scalardl-ledger/certs/tls.key ### License key configurations scalar.dl.licensing.license_key=${env:SCALAR_DL_LEDGER_LICENSE_KEY} scalar.dl.licensing.license_check_cert_pem=${env:SCALAR_DL_LEDGER_LICENSE_CHECK_CERT_PEM} tls: enabled: true overrideAuthority: "ledger.scalardl.example.com" caRootCertSecret: "scalardl-ledger-tls-ca" certChainSecret: "scalardl-ledger-tls-cert" privateKeySecret: "scalardl-ledger-tls-key" secretName: "ledger-credentials-secret" EOF ``` 1. Create a custom values file for ScalarDL Auditor (`scalardl-auditor-custom-values.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/scalardl-auditor-custom-values.yaml envoy: tls: downstream: enabled: true certChainSecret: "envoy-tls-cert" privateKeySecret: "envoy-tls-key" upstream: enabled: true overrideAuthority: "auditor.scalardl.example.com" caRootCertSecret: "scalardl-auditor-tls-ca" auditor: image: repository: "ghcr.io/scalar-labs/scalardl-auditor-byol" auditorProperties: | ### Storage configurations scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://postgresql-auditor.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DL_AUDITOR_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DL_AUDITOR_POSTGRES_PASSWORD} ### Auditor configurations scalar.dl.auditor.ledger.host=scalardl-ledger-envoy.default.svc.cluster.local scalar.dl.auditor.authentication.method=hmac scalar.dl.auditor.authentication.hmac.cipher_key=${env:SCALAR_DL_AUDITOR_HMAC_CIPHER_KEY} scalar.dl.auditor.servers.authentication.hmac.secret_key=${env:SCALAR_DL_AUDITOR_HMAC_SECRET_KEY} ### TLS configurations scalar.dl.auditor.server.tls.enabled=true scalar.dl.auditor.server.tls.cert_chain_path=/tls/scalardl-auditor/certs/tls.crt scalar.dl.auditor.server.tls.private_key_path=/tls/scalardl-auditor/certs/tls.key scalar.dl.auditor.tls.enabled=true scalar.dl.auditor.tls.ca_root_cert_path=/tls/scalardl-ledger/certs/ca.crt scalar.dl.auditor.tls.override_authority=envoy.scalar.example.com ### License key configurations scalar.dl.licensing.license_key=${env:SCALAR_DL_AUDITOR_LICENSE_KEY} scalar.dl.licensing.license_check_cert_pem=${env:SCALAR_DL_AUDITOR_LICENSE_CHECK_CERT_PEM} tls: enabled: true overrideAuthority: "auditor.scalardl.example.com" caRootCertSecret: "scalardl-auditor-tls-ca" certChainSecret: "scalardl-auditor-tls-cert" privateKeySecret: "scalardl-auditor-tls-key" caRootCertForLedgerSecret: "scalardl-auditor-tls-ca-for-ledger" secretName: "auditor-credentials-secret" EOF ``` 1. Create a secret resource named `ledger-credentials-secret` that includes credentials and a license key. ```console kubectl create secret generic ledger-credentials-secret \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_PASSWORD=postgres \ --from-literal=SCALAR_DL_LEDGER_HMAC_CIPHER_KEY=ledger-hmac-cipher-key \ --from-literal=SCALAR_DL_LEDGER_HMAC_SECRET_KEY=scalardl-hmac-secret-key \ --from-literal=SCALAR_DL_LEDGER_LICENSE_KEY="${SCALAR_DL_LEDGER_LICENSE_KEY}" \ --from-file=SCALAR_DL_LEDGER_LICENSE_CHECK_CERT_PEM=<(echo ${SCALAR_DL_LEDGER_LICENSE_CHECK_CERT_PEM} | sed 's/\\n/\ /g') \ -n default ``` 1. Create a secret resource named `auditor-credentials-secret` that includes credentials and a license key. ```console kubectl create secret generic auditor-credentials-secret \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_PASSWORD=postgres \ --from-literal=SCALAR_DL_AUDITOR_HMAC_CIPHER_KEY=auditor-hmac-cipher-key \ --from-literal=SCALAR_DL_AUDITOR_HMAC_SECRET_KEY=scalardl-hmac-secret-key \ --from-literal=SCALAR_DL_AUDITOR_LICENSE_KEY="${SCALAR_DL_AUDITOR_LICENSE_KEY}" \ --from-file=SCALAR_DL_AUDITOR_LICENSE_CHECK_CERT_PEM=<(echo ${SCALAR_DL_AUDITOR_LICENSE_CHECK_CERT_PEM} | sed 's/\\n/\ /g') \ -n default ``` 1. Create secret resources that include the private key and certificate files for Envoy. ```console kubectl create secret generic envoy-tls-cert --from-file=tls.crt=${HOME}/scalardl-test/certs/envoy.pem -n default kubectl create secret generic envoy-tls-key --from-file=tls.key=${HOME}/scalardl-test/certs/envoy-key.pem -n default ``` 1. Create secret resources that include the private key, certificate, and CA certificate files for ScalarDL Ledger. ```console kubectl create secret generic scalardl-ledger-tls-ca --from-file=ca.crt=${HOME}/scalardl-test/certs/ca.pem -n default kubectl create secret generic scalardl-ledger-tls-cert --from-file=tls.crt=${HOME}/scalardl-test/certs/ledger.pem -n default kubectl create secret generic scalardl-ledger-tls-key --from-file=tls.key=${HOME}/scalardl-test/certs/ledger-key.pem -n default ``` 1. Create secret resources that include the private key, certificate, and CA certificate files for ScalarDL Auditor. ```console kubectl create secret generic scalardl-auditor-tls-ca --from-file=ca.crt=${HOME}/scalardl-test/certs/ca.pem -n default kubectl create secret generic scalardl-auditor-tls-cert --from-file=tls.crt=${HOME}/scalardl-test/certs/auditor.pem -n default kubectl create secret generic scalardl-auditor-tls-key --from-file=tls.key=${HOME}/scalardl-test/certs/auditor-key.pem -n default kubectl create secret generic scalardl-auditor-tls-ca-for-ledger --from-file=ca.crt=${HOME}/scalardl-test/certs/ca.pem -n default ``` 1. Create a secret resource named `auditor-keys` to disable the `digital-signature` authentication method. In this tutorial, you'll use the `hmac` authentication method instead of `digital-signature`. ```console kubectl create secret generic auditor-keys \ --from-literal=tls.key=dummy-data-to-disable-digital-signature-method \ --from-literal=certificate=dummy-data-to-disable-digital-signature-method \ -n default ``` Note: If you use `hmac` as an authentication method, you have to create a dummy secret `auditor-key` to disable `digital-signature` on the helm chart side. 1. Set the chart version of ScalarDL Ledger and ScalarDL Auditor. ```console SCALAR_DL_LEDGER_CHART_VERSION=$(helm search repo scalar-labs/scalardl -l | grep -v -e "scalar-labs/scalardl-audit" | grep -F "${SCALAR_DL_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) SCALAR_DL_AUDITOR_CHART_VERSION=$(helm search repo scalar-labs/scalardl-audit -l | grep -F "${SCALAR_DL_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` 1. Deploy ScalarDL Ledger. ```console helm install scalardl-ledger scalar-labs/scalardl -f ${HOME}/scalardl-test/scalardl-ledger-custom-values.yaml --version ${SCALAR_DL_LEDGER_CHART_VERSION} -n default ``` 1. Deploy ScalarDL Auditor. ```console helm install scalardl-auditor scalar-labs/scalardl-audit -f ${HOME}/scalardl-test/scalardl-auditor-custom-values.yaml --version ${SCALAR_DL_AUDITOR_CHART_VERSION} -n default ``` 1. Check if the ScalarDL Ledger and ScalarDL Auditor pods are deployed. ```console kubectl get pod -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-auditor-0 1/1 Running 0 14m postgresql-ledger-0 1/1 Running 0 14m scalardl-auditor-auditor-5b885ff4c8-fwkpf 1/1 Running 0 18s scalardl-auditor-auditor-5b885ff4c8-g69cb 1/1 Running 0 18s scalardl-auditor-auditor-5b885ff4c8-nsmnq 1/1 Running 0 18s scalardl-auditor-envoy-689bcbdf65-5mn6v 1/1 Running 0 18s scalardl-auditor-envoy-689bcbdf65-fpq8j 1/1 Running 0 18s scalardl-auditor-envoy-689bcbdf65-lsz2t 1/1 Running 0 18s scalardl-ledger-envoy-547bbf7546-n7p5x 1/1 Running 0 26s scalardl-ledger-envoy-547bbf7546-p8nwp 1/1 Running 0 26s scalardl-ledger-envoy-547bbf7546-pskpb 1/1 Running 0 26s scalardl-ledger-ledger-6db5dc8774-5zsbj 1/1 Running 0 26s scalardl-ledger-ledger-6db5dc8774-vnmrw 1/1 Running 0 26s scalardl-ledger-ledger-6db5dc8774-wpjvs 1/1 Running 0 26s schema-loader-auditor-schema-loading-dvc5r 0/1 Completed 0 11m schema-loader-ledger-schema-loading-mtllb 0/1 Completed 0 11m ``` If the ScalarDL Ledger and ScalarDL Auditor pods are deployed properly, the `STATUS` column for those pods will be displayed as `Running`. 1. Check if the ScalarDL Ledger and ScalarDL Auditor services are deployed. ```console kubectl get svc -n default ``` [Command execution result] ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 47d postgresql-auditor ClusterIP 10.107.9.78 5432/TCP 15m postgresql-auditor-hl ClusterIP None 5432/TCP 15m postgresql-ledger ClusterIP 10.108.241.181 5432/TCP 15m postgresql-ledger-hl ClusterIP None 5432/TCP 15m scalardl-auditor-envoy ClusterIP 10.100.61.202 40051/TCP,40052/TCP 55s scalardl-auditor-envoy-metrics ClusterIP 10.99.6.227 9001/TCP 55s scalardl-auditor-headless ClusterIP None 40051/TCP,40053/TCP,40052/TCP 55s scalardl-auditor-metrics ClusterIP 10.108.1.147 8080/TCP 55s scalardl-ledger-envoy ClusterIP 10.101.191.116 50051/TCP,50052/TCP 61s scalardl-ledger-envoy-metrics ClusterIP 10.106.52.103 9001/TCP 61s scalardl-ledger-headless ClusterIP None 50051/TCP,50053/TCP,50052/TCP 61s scalardl-ledger-metrics ClusterIP 10.99.122.106 8080/TCP 61s ``` If the ScalarDL Ledger and ScalarDL Auditor services are deployed properly, you can see private IP addresses in the `CLUSTER-IP` column. :::note The `CLUSTER-IP` values for `scalardl-ledger-headless`, `scalardl-auditor-headless`, `postgresql-ledger-hl`, and `postgresql-auditor-hl` are `None` since they have no IP addresses. ::: ## Step 7. Start a client container You'll use the CA certificate file in a client container. Therefore, you'll need to create a secret resource and mount it to the client container. 1. Create a secret resource named `client-ca-cert`. ```console kubectl create secret generic client-ca-cert --from-file=ca.crt=${HOME}/scalardl-test/certs/ca.pem -n default ``` 1. Create a manifest file for a client pod (`scalardl-client-pod.yaml`). ```console cat << 'EOF' > ${HOME}/scalardl-test/scalardl-client-pod.yaml apiVersion: v1 kind: Pod metadata: name: "scalardl-client" spec: containers: - name: scalardl-client image: eclipse-temurin:8-jdk command: ['sleep'] args: ['inf'] env: - name: SCALAR_DL_VERSION value: SCALAR_DL_CLIENT_POD_SCALAR_DL_VERSION volumeMounts: - name: "client-ca-cert" mountPath: "/certs/" readOnly: true volumes: - name: "client-ca-cert" secret: secretName: "client-ca-cert" restartPolicy: Never EOF ``` 1. Set the ScalarDL version in the manifest file. ```console sed -i s/SCALAR_DL_CLIENT_POD_SCALAR_DL_VERSION/${SCALAR_DL_VERSION}/ ${HOME}/scalardl-test/scalardl-client-pod.yaml ``` 1. Deploy the client pod. ```console kubectl apply -f ${HOME}/scalardl-test/scalardl-client-pod.yaml -n default ``` 1. Check if the client container is running. ```console kubectl get pod scalardl-client -n default ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE scalardl-client 1/1 Running 0 4s ``` ## Step 8. Run ScalarDL sample contracts in the client container The following explains the minimum steps needed to run sample contracts. For more details about ScalarDL Ledger and ScalarDL Auditor, see the following: * [Getting Started with ScalarDL](https://scalardl.scalar-labs.com/docs/latest/getting-started/) * [Getting Started with ScalarDL Auditor](https://scalardl.scalar-labs.com/docs/latest/getting-started-auditor/) 1. Run bash in the client container. ```console kubectl exec -it scalardl-client -n default -- bash ``` The commands in the following steps must be run in the client container. 1. Install the git, curl, and unzip commands in the client container. ```console apt update && apt install -y git curl unzip ``` 1. Clone the ScalarDL Java Client SDK git repository. ```console git clone https://github.com/scalar-labs/scalardl-java-client-sdk.git ``` 1. Change the working directory to `scalardl-java-client-sdk/`. ```console cd scalardl-java-client-sdk/ ``` ```console pwd ``` [Command execution result] ```console /scalardl-java-client-sdk ``` 1. Change the branch to the version you're using. ```console git checkout -b v${SCALAR_DL_VERSION} refs/tags/v${SCALAR_DL_VERSION} ``` 1. Build the sample contracts. ```console ./gradlew assemble ``` 1. Download the CLI tools for ScalarDL from [ScalarDL Java Client SDK Releases](https://github.com/scalar-labs/scalardl-java-client-sdk/releases). ```console curl -OL https://github.com/scalar-labs/scalardl-java-client-sdk/releases/download/v${SCALAR_DL_VERSION}/scalardl-java-client-sdk-${SCALAR_DL_VERSION}.zip ``` You need to use the same version of CLI tools and ScalarDL Ledger. 1. Unzip the `scalardl-java-client-sdk-${SCALAR_DL_VERSION}.zip` file. ```console unzip ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}.zip ``` 1. Create a configuration file named `client.properties` to access ScalarDL Ledger and ScalarDL Auditor on the Kubernetes cluster. ```console cat << 'EOF' > client.properties # Ledger configuration scalar.dl.client.server.host=scalardl-ledger-envoy.default.svc.cluster.local scalar.dl.client.tls.enabled=true scalar.dl.client.tls.ca_root_cert_path=/certs/ca.crt scalar.dl.client.tls.override_authority=envoy.scalar.example.com # Auditor configuration scalar.dl.client.auditor.enabled=true scalar.dl.client.auditor.host=scalardl-auditor-envoy.default.svc.cluster.local scalar.dl.client.auditor.tls.enabled=true scalar.dl.client.auditor.tls.ca_root_cert_path=/certs/ca.crt scalar.dl.client.auditor.tls.override_authority=envoy.scalar.example.com # Client configuration scalar.dl.client.authentication_method=hmac scalar.dl.client.entity.id=client scalar.dl.client.entity.identity.hmac.secret_key=scalardl-hmac-client-secert-key EOF ``` 1. Register the client secret. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl register-secret --config ./client.properties ``` 1. Register the sample contract `StateUpdater`. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl register-contract --config ./client.properties --contract-id StateUpdater --contract-binary-name com.org1.contract.StateUpdater --contract-class-file ./build/classes/java/main/com/org1/contract/StateUpdater.class ``` 1. Register the sample contract `StateReader`. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl register-contract --config ./client.properties --contract-id StateReader --contract-binary-name com.org1.contract.StateReader --contract-class-file ./build/classes/java/main/com/org1/contract/StateReader.class ``` 1. Register the contract `ValidateLedger` to execute a validate request. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl register-contract --config ./client.properties --contract-id validate-ledger --contract-binary-name com.scalar.dl.client.contract.ValidateLedger --contract-class-file ./build/classes/java/main/com/scalar/dl/client/contract/ValidateLedger.class ``` 1. Execute the contract `StateUpdater`. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl execute-contract --config ./client.properties --contract-id StateUpdater --contract-argument '{"asset_id": "test_asset", "state": 3}' ``` This sample contract updates the `state` (value) of the asset named `test_asset` to `3`. 1. Execute the contract `StateReader`. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl execute-contract --config ./client.properties --contract-id StateReader --contract-argument '{"asset_id": "test_asset"}' ``` [Command execution result] ```console Contract result: { "id" : "test_asset", "age" : 0, "output" : { "state" : 3 } } ``` ### Reference * If the asset data is not tampered with, running the `execute-contract` command to request contract execution will return `OK` as a result. * If the asset data is tampered with (for example, if the `state` value in the database is tampered with), running the `execute-contract` command to request contract execution will return a value other than `OK` (for example, `INCONSISTENT_STATES`) as a result. See the following as an example for how ScalarDL detects data tampering. [Command execution result (if the asset data is tampered with)] ```console { "status_code" : "INCONSISTENT_STATES", "error_message" : "The results from Ledger and Auditor don't match" } ``` 1. Execute a validation request for the asset. ```console ./scalardl-java-client-sdk-${SCALAR_DL_VERSION}/bin/scalardl validate-ledger --config ./client.properties --asset-id "test_asset" ``` [Command execution result] ```console { "status_code" : "OK", "Ledger" : { "id" : "test_asset", "age" : 0, "nonce" : "3533427d-03cf-41d1-bf95-4d31eb0cb24d", "hash" : "FiquvtPMKLlxKf4VGoccSAGsi9ptn4ozYVVTwdSzEQ0=", "signature" : "MEYCIQDiiXqzw6K+Ml4uvn8rK43o5wHWESU3hoXnZPi6/OeKVwIhAM+tFBcapl6zg47Uq0Uc8nVNGWNHZLBDBGve3F0xkzTR" }, "Auditor" : { "id" : "test_asset", "age" : 0, "nonce" : "3533427d-03cf-41d1-bf95-4d31eb0cb24d", "hash" : "FiquvtPMKLlxKf4VGoccSAGsi9ptn4ozYVVTwdSzEQ0=", "signature" : "MEUCIQDLsfUR2PmxSvfpL3YvHJUkz00RDpjCdctkroZKXE8d5QIgH73FQH2e11jfnynD00Pp9DrIG1vYizxDsvxUsMPo9IU=" } } ``` ### Reference * If the asset data is not tampered with, running the `validate-ledger` command to request validation will return `OK` as the result. * If the asset data is tampered with (for example, if the `state` value in the database is tampered with), running the `validate-ledger` command to request validation will return a value other than `OK` (for example, `INVALID_OUTPUT`) as a result. See the following as an example for how ScalarDL detects data tampering. [Command execution result (if the asset data is tampered with)] ```console { "status_code" : "INCONSISTENT_STATES", "error_message" : "The results from Ledger and Auditor don't match" } ``` 1. Exit from the client container. ```console exit ``` ## Step 9. Delete all resources After completing the ScalarDL Ledger and ScalarDL Auditor tests on the Kubernetes cluster, remove all resources. 1. Uninstall ScalarDL Ledger, ScalarDL Auditor, ScalarDL Schema Loader, and PostgreSQL. ```console helm uninstall -n default scalardl-ledger schema-loader-ledger postgresql-ledger scalardl-auditor schema-loader-auditor postgresql-auditor ``` 1. Remove the client container. ``` kubectl delete pod scalardl-client --grace-period 0 -n default ``` 1. Remove the secret resources. ``` kubectl delete secrets envoy-tls-key envoy-tls-cert schema-ledger-credentials-secret schema-auditor-credentials-secret ledger-credentials-secret scalardl-ledger-tls-ca scalardl-ledger-tls-cert scalardl-ledger-tls-key auditor-credentials-secret auditor-keys scalardl-auditor-tls-ca scalardl-auditor-tls-cert scalardl-auditor-tls-key scalardl-auditor-tls-ca-for-ledger client-ca-cert ``` 1. Remove the working directory and sample files (configuration file, private key, and certificate). ```console cd ${HOME} ``` ```console rm -rf ${HOME}/scalardl-test/ ``` ## Further reading You can see how to get started with monitoring or logging for Scalar products in the following tutorials: * [Getting Started with Helm Charts (Monitoring using Prometheus Operator)](getting-started-monitoring.mdx) * [Getting Started with Helm Charts (Logging using Loki Stack)](getting-started-logging.mdx) * [Getting Started with Helm Charts (Scalar Manager)](getting-started-scalar-manager.mdx) ================================================ FILE: docs/helm-charts/getting-started-scalardl-auditor.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Getting Started with Helm Charts (ScalarDL Ledger and Auditor / Auditor mode) This document explains how to get started with ScalarDL Ledger and Auditor using Helm Chart on a Kubernetes cluster as a test environment. Here, we assume that you already have a Mac or Linux environment for testing. We use **Minikube** in this document, but the steps we will show should work in any Kubernetes cluster. ## Requirement You need to subscribe to ScalarDL Ledger and ScalarDL Auditor in the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-rzbuhxgvqf4d2) to get the following container images. * AWS Marketplace * scalar-ledger * scalar-ledger-envoy * scalardl-schema-loader-ledger * scalar-auditor * scalar-auditor-envoy * scalardl-schema-loader-auditor For more details, refer to [How to install Scalar products through AWS Marketplace](../scalar-kubernetes/AwsMarketplaceGuide.mdx). ## Note To make Byzantine fault detection with auditing work properly, Ledger and Auditor should be deployed and managed in different administrative domains. However, in this guide, we will deploy Ledger and Auditor in the same Kubernetes cluster to make the test easier. ## What we create We will deploy the following components on a Kubernetes cluster as follows. ``` +-----------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | [Pod] [Pod] [Pod] | | | | +-------+ +---------+ | | +---> | Envoy | ---+ +---> | Ledger | ---+ | | | +-------+ | | +---------+ | | | | | | | | | +---------+ | +-------+ | +-----------+ | +---------+ | +---------------+ | | +---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | Ledger | ---+---> | PostgreSQL | | | | | (Envoy) | | +-------+ | | (Ledger) | | +---------+ | | (For Ledger) | | | | +---------+ | | +-----------+ | | +---------------+ | | | | +-------+ | | +---------+ | | | | +---> | Envoy | ---+ +---> | Ledger | ---+ | | +--------+ | +-------+ +---------+ | | | Client | ---+ | | +--------+ | +-------+ +---------+ | | | +---> | Envoy | ---+ +---> | Auditor | ---+ | | | | +-------+ | | +---------+ | | | | | | | | | | | +---------+ | +-------+ | +-----------+ | +---------+ | +---------------+ | | +---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | Auditor | ---+---> | PostgreSQL | | | | (Envoy) | | +-------+ | | (Auditor) | | +---------+ | | (For Auditor) | | | +---------+ | | +-----------+ | | +---------------+ | | | +-------+ | | +---------+ | | | +---> | Envoy | ---+ +---> | Auditor | ---+ | | +-------+ +---------+ | | | +-----------------------------------------------------------------------------------------------------------------------------+ ``` ## Step 1. Start a Kubernetes cluster First, you need to prepare a Kubernetes cluster. If you use a **minikube** environment, please refer to the [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx). If you have already started a Kubernetes cluster, you can skip this step. ## Step 2. Start PostgreSQL containers ScalarDL Ledger and Auditor use some kind of database system as a backend database. In this document, we use PostgreSQL. You can deploy PostgreSQL on the Kubernetes cluster as follows. 1. Add the Bitnami helm repository. ```console helm repo add bitnami https://charts.bitnami.com/bitnami ``` 1. Deploy PostgreSQL for Ledger. ```console helm install postgresql-ledger bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false ``` 1. Deploy PostgreSQL for Auditor. ```console helm install postgresql-auditor bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false ``` 1. Check if the PostgreSQL containers are running. ```console kubectl get pod ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-auditor-0 1/1 Running 0 11s postgresql-ledger-0 1/1 Running 0 16s ``` ## Step 3. Create a working directory We will create some configuration files and key/certificate files locally. So, create a working directory for them. 1. Create a working directory. ```console mkdir -p ~/scalardl-test/certs/ ``` ## Step 4. Create key/certificate files Note: In this guide, we will use self-sign certificates for the test. However, it is strongly recommended that these certificates NOT be used in production. 1. Change the working directory to `~/scalardl-test/certs/` directory. ```console cd ~/scalardl-test/certs/ ``` 1. Create a JSON file that includes Ledger information. ```console cat << 'EOF' > ~/scalardl-test/certs/ledger.json { "CN": "ledger", "hosts": ["example.com","*.example.com"], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "O": "ledger", "OU": "test team", "L": "Shinjuku", "ST": "Tokyo", "C": "JP" } ] } EOF ``` 1. Create a JSON file that includes Auditor information. ```console cat << 'EOF' > ~/scalardl-test/certs/auditor.json { "CN": "auditor", "hosts": ["example.com","*.example.com"], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "O": "auditor", "OU": "test team", "L": "Shinjuku", "ST": "Tokyo", "C": "JP" } ] } EOF ``` 1. Create a JSON file that includes Client information. ```console cat << 'EOF' > ~/scalardl-test/certs/client.json { "CN": "client", "hosts": ["example.com","*.example.com"], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "O": "client", "OU": "test team", "L": "Shinjuku", "ST": "Tokyo", "C": "JP" } ] } EOF ``` 1. Create key/certificate files for the Ledger. ```console cfssl selfsign "" ./ledger.json | cfssljson -bare ledger ``` 1. Create key/certificate files for the Auditor. ```console cfssl selfsign "" ./auditor.json | cfssljson -bare auditor ``` 1. Create key/certificate files for the Client. ```console cfssl selfsign "" ./client.json | cfssljson -bare client ``` 1. Confirm key/certificate files are created. ```console ls -1 ``` [Command execution result] ```console auditor-key.pem auditor.csr auditor.json auditor.pem client-key.pem client.csr client.json client.pem ledger-key.pem ledger.csr ledger.json ledger.pem ``` ## Step 5. Create DB schemas for ScalarDL Ledger and ScalarDL Auditor using Helm Charts We will deploy two ScalarDL Schema Loader pods on the Kubernetes cluster using Helm Charts. The ScalarDL Schema Loader will create the DB schemas for ScalarDL Ledger and Auditor in PostgreSQL. 1. Change the working directory to `~/scalardl-test/`. ```console cd ~/scalardl-test/ ``` 1. Add the Scalar helm repository. ```console helm repo add scalar-labs https://scalar-labs.github.io/helm-charts ``` 1. Create a secret resource to pull the ScalarDL container images from AWS. * AWS Marketplace ```console kubectl create secret docker-registry reg-ecr-mp-secrets \ --docker-server=709825985650.dkr.ecr.us-east-1.amazonaws.com \ --docker-username=AWS \ --docker-password=$(aws ecr get-login-password --region us-east-1) ``` For more details, refer to [How to install Scalar products through AWS Marketplace](../scalar-kubernetes/AwsMarketplaceGuide.mdx). 1. Create a custom values file for ScalarDL Schema Loader for Ledger (schema-loader-ledger-custom-values.yaml). * AWS Marketplace ```console cat << 'EOF' > ~/scalardl-test/schema-loader-ledger-custom-values.yaml schemaLoading: schemaType: "ledger" image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardl-schema-loader-ledger" version: "3.6.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" databaseProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-ledger.default.svc.cluster.local:5432/postgres scalar.db.username={{ default .Env.SCALAR_DL_LEDGER_POSTGRES_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DL_LEDGER_POSTGRES_PASSWORD "" }} scalar.db.storage=jdbc secretName: "ledger-credentials-secret" EOF ``` 1. Create a custom values file for ScalarDL Schema Loader for Auditor (schema-loader-auditor-custom-values.yaml). * AWS Marketplace ```console cat << 'EOF' > ~/scalardl-test/schema-loader-auditor-custom-values.yaml schemaLoading: schemaType: "auditor" image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardl-schema-loader-auditor" version: "3.6.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" databaseProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-auditor.default.svc.cluster.local:5432/postgres scalar.db.username={{ default .Env.SCALAR_DL_AUDITOR_POSTGRES_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DL_AUDITOR_POSTGRES_PASSWORD "" }} scalar.db.storage=jdbc secretName: "auditor-credentials-secret" EOF ``` 1. Create a secret resource that includes a username and password for PostgreSQL for Ledger. ```console kubectl create secret generic ledger-credentials-secret \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_PASSWORD=postgres ``` 1. Create a secret resource that includes a username and password for PostgreSQL for Auditor. ```console kubectl create secret generic auditor-credentials-secret \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_AUDITOR_POSTGRES_PASSWORD=postgres ``` 1. Deploy the ScalarDL Schema Loader for Ledger. ```console helm install schema-loader-ledger scalar-labs/schema-loading -f ./schema-loader-ledger-custom-values.yaml ``` 1. Deploy the ScalarDL Schema Loader for Auditor. ```console helm install schema-loader-auditor scalar-labs/schema-loading -f ./schema-loader-auditor-custom-values.yaml ``` 1. Check if the ScalarDL Schema Loader pods are deployed and completed. ```console kubectl get pod ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-auditor-0 1/1 Running 0 2m56s postgresql-ledger-0 1/1 Running 0 3m1s schema-loader-auditor-schema-loading-dvc5r 0/1 Completed 0 6s schema-loader-ledger-schema-loading-mtllb 0/1 Completed 0 10s ``` If the ScalarDL Schema Loader pods are **ContainerCreating** or **Running**, wait for the process will be completed (The STATUS will be **Completed**). ## Step 6. Deploy ScalarDL Ledger and Auditor on the Kubernetes cluster using Helm Charts 1. Create a custom values file for ScalarDL Ledger (scalardl-ledger-custom-values.yaml). * AWS Marketplace ```console cat << 'EOF' > ~/scalardl-test/scalardl-ledger-custom-values.yaml envoy: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-ledger-envoy" version: "1.3.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" ledger: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-ledger" version: "3.6.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" ledgerProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-ledger.default.svc.cluster.local:5432/postgres scalar.db.username={{ default .Env.SCALAR_DL_LEDGER_POSTGRES_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DL_LEDGER_POSTGRES_PASSWORD "" }} scalar.db.storage=jdbc scalar.dl.ledger.proof.enabled=true scalar.dl.ledger.auditor.enabled=true scalar.dl.ledger.proof.private_key_path=/keys/private-key secretName: "ledger-credentials-secret" extraVolumes: - name: "ledger-keys" secret: secretName: "ledger-keys" extraVolumeMounts: - name: "ledger-keys" mountPath: "/keys" readOnly: true EOF ``` 1. Create a custom values file for ScalarDL Auditor (scalardl-auditor-custom-values.yaml). * AWS Marketplace ```console cat << 'EOF' > ~/scalardl-test/scalardl-auditor-custom-values.yaml envoy: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-auditor-envoy" version: "1.3.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" auditor: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-auditor" version: "3.6.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" auditorProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-auditor.default.svc.cluster.local:5432/postgres scalar.db.username={{ default .Env.SCALAR_DL_AUDITOR_POSTGRES_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DL_AUDITOR_POSTGRES_PASSWORD "" }} scalar.db.storage=jdbc scalar.dl.auditor.ledger.host=scalardl-ledger-envoy.default.svc.cluster.local scalar.dl.auditor.private_key_path=/keys/private-key secretName: "auditor-credentials-secret" extraVolumes: - name: "auditor-keys" secret: secretName: "auditor-keys" extraVolumeMounts: - name: "auditor-keys" mountPath: "/keys" readOnly: true EOF ``` 1. Create secret resource `ledger-keys`. ```console kubectl create secret generic ledger-keys --from-file=certificate=./certs/ledger.pem --from-file=private-key=./certs/ledger-key.pem ``` 1. Create secret resource `auditor-keys`. ```console kubectl create secret generic auditor-keys --from-file=certificate=./certs/auditor.pem --from-file=private-key=./certs/auditor-key.pem ``` 1. Deploy the ScalarDL Ledger. ```console helm install scalardl-ledger scalar-labs/scalardl -f ./scalardl-ledger-custom-values.yaml ``` 1. Deploy the ScalarDL Auditor. ```console helm install scalardl-auditor scalar-labs/scalardl-audit -f ./scalardl-auditor-custom-values.yaml ``` 1. Check if the ScalarDL Ledger and Auditor pods are deployed. ```console kubectl get pod ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-auditor-0 1/1 Running 0 14m postgresql-ledger-0 1/1 Running 0 14m scalardl-auditor-auditor-5b885ff4c8-fwkpf 1/1 Running 0 18s scalardl-auditor-auditor-5b885ff4c8-g69cb 1/1 Running 0 18s scalardl-auditor-auditor-5b885ff4c8-nsmnq 1/1 Running 0 18s scalardl-auditor-envoy-689bcbdf65-5mn6v 1/1 Running 0 18s scalardl-auditor-envoy-689bcbdf65-fpq8j 1/1 Running 0 18s scalardl-auditor-envoy-689bcbdf65-lsz2t 1/1 Running 0 18s scalardl-ledger-envoy-547bbf7546-n7p5x 1/1 Running 0 26s scalardl-ledger-envoy-547bbf7546-p8nwp 1/1 Running 0 26s scalardl-ledger-envoy-547bbf7546-pskpb 1/1 Running 0 26s scalardl-ledger-ledger-6db5dc8774-5zsbj 1/1 Running 0 26s scalardl-ledger-ledger-6db5dc8774-vnmrw 1/1 Running 0 26s scalardl-ledger-ledger-6db5dc8774-wpjvs 1/1 Running 0 26s schema-loader-auditor-schema-loading-dvc5r 0/1 Completed 0 11m schema-loader-ledger-schema-loading-mtllb 0/1 Completed 0 11m ``` If the ScalarDL Ledger and Auditor pods are deployed properly, you can see the STATUS are **Running**. 1. Check if the ScalarDL Ledger and Auditor services are deployed. ```console kubectl get svc ``` [Command execution result] ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 47d postgresql-auditor ClusterIP 10.107.9.78 5432/TCP 15m postgresql-auditor-hl ClusterIP None 5432/TCP 15m postgresql-ledger ClusterIP 10.108.241.181 5432/TCP 15m postgresql-ledger-hl ClusterIP None 5432/TCP 15m scalardl-auditor-envoy ClusterIP 10.100.61.202 40051/TCP,40052/TCP 55s scalardl-auditor-envoy-metrics ClusterIP 10.99.6.227 9001/TCP 55s scalardl-auditor-headless ClusterIP None 40051/TCP,40053/TCP,40052/TCP 55s scalardl-auditor-metrics ClusterIP 10.108.1.147 8080/TCP 55s scalardl-ledger-envoy ClusterIP 10.101.191.116 50051/TCP,50052/TCP 61s scalardl-ledger-envoy-metrics ClusterIP 10.106.52.103 9001/TCP 61s scalardl-ledger-headless ClusterIP None 50051/TCP,50053/TCP,50052/TCP 61s scalardl-ledger-metrics ClusterIP 10.99.122.106 8080/TCP 61s ``` If the ScalarDL Ledger and Auditor services are deployed properly, you can see private IP addresses in the CLUSTER-IP column. (Note: `scalardl-ledger-headless` and `scalardl-auditor-headless` have no CLUSTER-IP.) ## Step 7. Start a Client container We will use certificate files in a Client container. So, we create a secret resource and mount it to a Client container. 1. Create secret resource `client-keys`. ``` kubectl create secret generic client-keys --from-file=certificate=./certs/client.pem --from-file=private-key=./certs/client-key.pem ``` 1. Start a Client container on the Kubernetes cluster. ```console cat << 'EOF' | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: "scalardl-client" spec: containers: - name: scalardl-client image: eclipse-temurin:8-jdk command: ['sleep'] args: ['inf'] volumeMounts: - name: "ledger-keys" mountPath: "/keys/ledger" readOnly: true - name: "auditor-keys" mountPath: "/keys/auditor" readOnly: true - name: "client-keys" mountPath: "/keys/client" readOnly: true volumes: - name: "ledger-keys" secret: secretName: "ledger-keys" - name: "auditor-keys" secret: secretName: "auditor-keys" - name: "client-keys" secret: secretName: "client-keys" restartPolicy: Never EOF ``` 1. Check if the Client container is running. ```console kubectl get pod scalardl-client ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE scalardl-client 1/1 Running 0 4s ``` ## Step 8. Run ScalarDL sample contracts in the Client container The following explains the minimum steps. If you want to know more details about ScalarDL Ledger and Auditor, please refer to the following documents. * [Getting Started with ScalarDL](https://scalardl.scalar-labs.com/docs/latest/getting-started) * [Getting Started with ScalarDL Auditor](https://scalardl.scalar-labs.com/docs/latest/getting-started-auditor) When you use Auditor, you need to register the certificate for the Ledger and Auditor before starting the client application. Ledger needs to register its certificate to Auditor, and Auditor needs to register its certificate to Ledger. 1. Run bash in the Client container. ```console kubectl exec -it scalardl-client -- bash ``` After this step, run each command in the Client container. 1. Install the git, curl and unzip commands in the Client container. ```console apt update && apt install -y git curl unzip ``` 1. Clone ScalarDL Java Client SDK git repository. ```console git clone https://github.com/scalar-labs/scalardl-java-client-sdk.git ``` 1. Change the directory to `scalardl-java-client-sdk/`. ```console cd scalardl-java-client-sdk/ ``` ```console pwd ``` [Command execution result] ```console /scalardl-java-client-sdk ``` 1. Change branch to arbitrary version. ```console git checkout -b v3.6.0 refs/tags/v3.6.0 ``` ```console git branch ``` [Command execution result] ```console master * v3.6.0 ``` If you want to use another version, please specify the version (tag) you want to use. You need to use the same version of ScalarDL Ledger and ScalarDL Java Client SDK. 1. Build the sample contracts. ```console ./gradlew assemble ``` 1. Download CLI tools of ScalarDL from [ScalarDL Java Client SDK Releases](https://github.com/scalar-labs/scalardl-java-client-sdk/releases). ```console curl -OL https://github.com/scalar-labs/scalardl-java-client-sdk/releases/download/v3.6.0/scalardl-java-client-sdk-3.6.0.zip ``` You need to use the same version of CLI tools and ScalarDL Ledger. 1. Unzip the `scalardl-java-client-sdk-3.6.0.zip` file. ```console unzip ./scalardl-java-client-sdk-3.6.0.zip ``` 1. Create a configuration file (ledger.as.client.properties) to register the certificate of Ledger to Auditor. ```console cat << 'EOF' > ledger.as.client.properties # Ledger scalar.dl.client.server.host=scalardl-ledger-envoy.default.svc.cluster.local # Auditor scalar.dl.client.auditor.enabled=true scalar.dl.client.auditor.host=scalardl-auditor-envoy.default.svc.cluster.local # Certificate scalar.dl.client.cert_holder_id=ledger scalar.dl.client.cert_path=/keys/ledger/certificate scalar.dl.client.private_key_path=/keys/ledger/private-key EOF ``` 1. Create a configuration file (auditor.as.client.properties) to register the certificate of Auditor to Ledger. ```console cat << 'EOF' > auditor.as.client.properties # Ledger scalar.dl.client.server.host=scalardl-ledger-envoy.default.svc.cluster.local # Auditor scalar.dl.client.auditor.enabled=true scalar.dl.client.auditor.host=scalardl-auditor-envoy.default.svc.cluster.local # Certificate scalar.dl.client.cert_holder_id=auditor scalar.dl.client.cert_path=/keys/auditor/certificate scalar.dl.client.private_key_path=/keys/auditor/private-key EOF ``` 1. Create a configuration file (client.properties) to access ScalarDL Ledger on the Kubernetes cluster. ```console cat << 'EOF' > client.properties # Ledger scalar.dl.client.server.host=scalardl-ledger-envoy.default.svc.cluster.local # Auditor scalar.dl.client.auditor.enabled=true scalar.dl.client.auditor.host=scalardl-auditor-envoy.default.svc.cluster.local # Certificate scalar.dl.client.cert_holder_id=client scalar.dl.client.cert_path=/keys/client/certificate scalar.dl.client.private_key_path=/keys/client/private-key EOF ``` 1. Register the certificate file of Ledger. ```console ./scalardl-java-client-sdk-3.6.0/bin/register-cert --properties ./ledger.as.client.properties ``` 1. Register the certificate file of Auditor. ```console ./scalardl-java-client-sdk-3.6.0/bin/register-cert --properties ./auditor.as.client.properties ``` 1. Register the certificate file of client. ```console ./scalardl-java-client-sdk-3.6.0/bin/register-cert --properties ./client.properties ``` 1. Register the sample contract `StateUpdater`. ```console ./scalardl-java-client-sdk-3.6.0/bin/register-contract --properties ./client.properties --contract-id StateUpdater --contract-binary-name com.org1.contract.StateUpdater --contract-class-file ./build/classes/java/main/com/org1/contract/StateUpdater.class ``` 1. Register the sample contract `StateReader`. ```console ./scalardl-java-client-sdk-3.6.0/bin/register-contract --properties ./client.properties --contract-id StateReader --contract-binary-name com.org1.contract.StateReader --contract-class-file ./build/classes/java/main/com/org1/contract/StateReader.class ``` 1. Register the contract `ValdateLedger` to execute a validate request. ```console ./scalardl-java-client-sdk-3.6.0/bin/register-contract --properties ./client.properties --contract-id validate-ledger --contract-binary-name com.scalar.dl.client.contract.ValidateLedger --contract-class-file ./build/classes/java/main/com/scalar/dl/client/contract/ValidateLedger.class ``` 1. Execute the contract `StateUpdater`. ```console ./scalardl-java-client-sdk-3.6.0/bin/execute-contract --properties ./client.properties --contract-id StateUpdater --contract-argument '{"asset_id": "test_asset", "state": 3}' ``` This sample contract updates the `state` (value) of the asset named `test_asset` to `3`. 1. Execute the contract `StateReader`. ```console ./scalardl-java-client-sdk-3.6.0/bin/execute-contract --properties ./client.properties --contract-id StateReader --contract-argument '{"asset_id": "test_asset"}' ``` [Command execution result] ```console Contract result: { "id" : "test_asset", "age" : 0, "output" : { "state" : 3 } } ``` * Reference information * If the asset data is not tampered with, the contract execution request (execute-contract command) returns `OK` as a result. * If the asset data is tampered with (e.g. the `state` value in the DB is tampered with), the contract execution request (execute-contract command) returns a value other than `OK` (e.g. `INCONSISTENT_STATES`) as a result, like the following. [Command execution result (If the asset data is tampered with)] ```console { "status_code" : "INCONSISTENT_STATES", "error_message" : "The results from Ledger and Auditor don't match" } ``` * In this way, the ScalarDL can detect data tampering. 1. Execute a validation request for the asset. ```console ./scalardl-java-client-sdk-3.6.0/bin/validate-ledger --properties ./client.properties --asset-id "test_asset" ``` [Command execution result] ```console { "status_code" : "OK", "Ledger" : { "id" : "test_asset", "age" : 0, "nonce" : "3533427d-03cf-41d1-bf95-4d31eb0cb24d", "hash" : "FiquvtPMKLlxKf4VGoccSAGsi9ptn4ozYVVTwdSzEQ0=", "signature" : "MEYCIQDiiXqzw6K+Ml4uvn8rK43o5wHWESU3hoXnZPi6/OeKVwIhAM+tFBcapl6zg47Uq0Uc8nVNGWNHZLBDBGve3F0xkzTR" }, "Auditor" : { "id" : "test_asset", "age" : 0, "nonce" : "3533427d-03cf-41d1-bf95-4d31eb0cb24d", "hash" : "FiquvtPMKLlxKf4VGoccSAGsi9ptn4ozYVVTwdSzEQ0=", "signature" : "MEUCIQDLsfUR2PmxSvfpL3YvHJUkz00RDpjCdctkroZKXE8d5QIgH73FQH2e11jfnynD00Pp9DrIG1vYizxDsvxUsMPo9IU=" } } ``` * Reference information * If the asset data is not tampered with, the validation request (validate-ledger command) returns `OK` as a result. * If the asset data is tampered with (e.g. the `state` value in the DB is tampered with), the validation request (validate-ledger command) returns a value other than `OK` (e.g. `INVALID_OUTPUT`) as a result, like the following. [Command execution result (If the asset data is tampered with)] ```console { "status_code" : "INCONSISTENT_STATES", "error_message" : "The results from Ledger and Auditor don't match" } ``` * In this way, the ScalarDL Ledger can detect data tampering. ## Step 9. Delete all resources After completing the ScalarDL Ledger tests on the Kubernetes cluster, remove all resources. 1. Uninstall ScalarDL Ledger, ScalarDL Schema Loader, and PostgreSQL. ```console helm uninstall scalardl-ledger schema-loader-ledger postgresql-ledger scalardl-auditor schema-loader-auditor postgresql-auditor ``` 1. Remove the Client container. ``` kubectl delete pod scalardl-client --force --grace-period 0 ``` 1. Remove the working directory and sample files (configuration file, key, and certificate). ```console cd ~ ``` ```console rm -rf ~/scalardl-test/ ``` ## Further reading You can see how to get started with monitoring or logging for Scalar products in the following documents. * [Getting Started with Helm Charts (Monitoring using Prometheus Operator)](getting-started-monitoring.mdx) * [Getting Started with Helm Charts (Logging using Loki Stack)](getting-started-logging.mdx) * [Getting Started with Helm Charts (Scalar Manager)](getting-started-scalar-manager.mdx) ================================================ FILE: docs/helm-charts/getting-started-scalardl-ledger.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Getting Started with Helm Charts (ScalarDL Ledger / Ledger only) This document explains how to get started with ScalarDL Ledger using Helm Chart on a Kubernetes cluster as a test environment. Here, we assume that you already have a Mac or Linux environment for testing. We use **Minikube** in this document, but the steps we will show should work in any Kubernetes cluster. ## Requirement You need to subscribe to ScalarDL Ledger in the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-rzbuhxgvqf4d2) to get the following container images. * AWS Marketplace * scalar-ledger * scalar-ledger-envoy * scalardl-schema-loader-ledger For more details, refer to [How to install Scalar products through AWS Marketplace](../scalar-kubernetes/AwsMarketplaceGuide.mdx). ## What we create We will deploy the following components on a Kubernetes cluster as follows. ``` +--------------------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | | | [Pod] [Pod] [Pod] [Pod] | | | | +-------+ +-----------------+ | | +---> | Envoy | ---+ +---> | ScalarDL Ledger | ---+ | | | +-------+ | | +-----------------+ | | | | | | | | | +--------+ +---------+ | +-------+ | +-------------------+ | +-----------------+ | +------------+ | | | Client | ---> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | ScalarDL Ledger | ---+---> | PostgreSQL | | | +--------+ | (Envoy) | | +-------+ | | (ScalarDL Ledger) | | +-----------------+ | +------------+ | | +---------+ | | +-------------------+ | | | | | +-------+ | | +-----------------+ | | | +---> | Envoy | ---+ +---> | ScalarDL Ledger | ---+ | | +-------+ +-----------------+ | | | +--------------------------------------------------------------------------------------------------------------------------------------+ ``` ## Step 1. Start a Kubernetes cluster First, you need to prepare a Kubernetes cluster. If you use a **minikube** environment, please refer to the [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx). If you have already started a Kubernetes cluster, you can skip this step. ## Step 2. Start a PostgreSQL container ScalarDL Ledger uses some kind of database system as a backend database. In this document, we use PostgreSQL. You can deploy PostgreSQL on the Kubernetes cluster as follows. 1. Add the Bitnami helm repository. ```console helm repo add bitnami https://charts.bitnami.com/bitnami ``` 1. Deploy PostgreSQL. ```console helm install postgresql-ledger bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false ``` 1. Check if the PostgreSQL container is running. ```console kubectl get pod ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-ledger-0 1/1 Running 0 11s ``` ## Step 3. Create a working directory We will create some configuration files and key/certificate files locally. So, create a working directory for them. 1. Create a working directory. ```console mkdir -p ~/scalardl-test/certs/ ``` ## Step 4. Create key/certificate files Note: In this guide, we will use self-sign certificates for the test. However, it is strongly recommended that these certificates NOT be used in production. 1. Change the working directory to `~/scalardl-test/certs/` directory. ```console cd ~/scalardl-test/certs/ ``` 1. Create a JSON file that includes Ledger information. ```console cat << 'EOF' > ~/scalardl-test/certs/ledger.json { "CN": "ledger", "hosts": ["example.com","*.example.com"], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "O": "ledger", "OU": "test team", "L": "Shinjuku", "ST": "Tokyo", "C": "JP" } ] } EOF ``` 1. Create a JSON file that includes Client information. ```console cat << 'EOF' > ~/scalardl-test/certs/client.json { "CN": "client", "hosts": ["example.com","*.example.com"], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "O": "client", "OU": "test team", "L": "Shinjuku", "ST": "Tokyo", "C": "JP" } ] } EOF ``` 1. Create key/certificate files for the Ledger. ```console cfssl selfsign "" ./ledger.json | cfssljson -bare ledger ``` 1. Create key/certificate files for the Client. ```console cfssl selfsign "" ./client.json | cfssljson -bare client ``` 1. Confirm key/certificate files are created. ```console ls -1 ``` [Command execution result] ```console client-key.pem client.csr client.json client.pem ledger-key.pem ledger.csr ledger.json ledger.pem ``` ## Step 5. Create DB schemas for ScalarDL Ledger using Helm Charts We will deploy a ScalarDL Schema Loader on the Kubernetes cluster using Helm Charts. The ScalarDL Schema Loader will create the DB schemas for ScalarDL Ledger in PostgreSQL. 1. Change the working directory to `~/scalardl-test/`. ```console cd ~/scalardl-test/ ``` 1. Add the Scalar helm repository. ```console helm repo add scalar-labs https://scalar-labs.github.io/helm-charts ``` 1. Create a secret resource to pull the ScalarDL container images from AWS. * AWS Marketplace ```console kubectl create secret docker-registry reg-ecr-mp-secrets \ --docker-server=709825985650.dkr.ecr.us-east-1.amazonaws.com \ --docker-username=AWS \ --docker-password=$(aws ecr get-login-password --region us-east-1) ``` For more details, refer to [How to install Scalar products through AWS Marketplace](../scalar-kubernetes/AwsMarketplaceGuide.mdx). 1. Create a custom values file for ScalarDL Schema Loader (schema-loader-ledger-custom-values.yaml). * AWS Marketplace ```console cat << 'EOF' > ~/scalardl-test/schema-loader-ledger-custom-values.yaml schemaLoading: schemaType: "ledger" image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardl-schema-loader-ledger" version: "3.6.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" databaseProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-ledger.default.svc.cluster.local:5432/postgres scalar.db.username={{ default .Env.SCALAR_DL_LEDGER_POSTGRES_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DL_LEDGER_POSTGRES_PASSWORD "" }} scalar.db.storage=jdbc secretName: "ledger-credentials-secret" EOF ``` 1. Create a secret resource that includes a username and password for PostgreSQL. ```console kubectl create secret generic ledger-credentials-secret \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DL_LEDGER_POSTGRES_PASSWORD=postgres ``` 1. Deploy the ScalarDL Schema Loader. ```console helm install schema-loader-ledger scalar-labs/schema-loading -f ./schema-loader-ledger-custom-values.yaml ``` 1. Check if the ScalarDL Schema Loader pod is deployed and completed. ```console kubectl get pod ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-ledger-0 1/1 Running 0 11m schema-loader-ledger-schema-loading-46rcr 0/1 Completed 0 3s ``` If the ScalarDL Schema Loader pod is **ContainerCreating** or **Running**, wait for the process will be completed (The STATUS will be **Completed**). ## Step 6. Deploy ScalarDL Ledger on the Kubernetes cluster using Helm Charts 1. Create a custom values file for ScalarDL Ledger (scalardl-ledger-custom-values.yaml). * AWS Marketplace ```console cat << 'EOF' > ~/scalardl-test/scalardl-ledger-custom-values.yaml envoy: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-ledger-envoy" version: "1.3.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" ledger: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-ledger" version: "3.6.0" imagePullSecrets: - name: "reg-ecr-mp-secrets" ledgerProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-ledger.default.svc.cluster.local:5432/postgres scalar.db.username={{ default .Env.SCALAR_DL_LEDGER_POSTGRES_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DL_LEDGER_POSTGRES_PASSWORD "" }} scalar.db.storage=jdbc scalar.dl.ledger.proof.enabled=true scalar.dl.ledger.proof.private_key_path=/keys/private-key secretName: "ledger-credentials-secret" extraVolumes: - name: "ledger-keys" secret: secretName: "ledger-keys" extraVolumeMounts: - name: "ledger-keys" mountPath: "/keys" readOnly: true EOF ``` 1. Create secret resource `ledger-keys`. ```console kubectl create secret generic ledger-keys --from-file=private-key=./certs/ledger-key.pem ``` 1. Deploy the ScalarDL Ledger. ```console helm install scalardl-ledger scalar-labs/scalardl -f ./scalardl-ledger-custom-values.yaml ``` 1. Check if the ScalarDL Ledger pods are deployed. ```console kubectl get pod ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE postgresql-ledger-0 1/1 Running 0 14m scalardl-ledger-envoy-547bbf7546-6cn88 1/1 Running 0 52s scalardl-ledger-envoy-547bbf7546-rpg5p 1/1 Running 0 52s scalardl-ledger-envoy-547bbf7546-x2vlg 1/1 Running 0 52s scalardl-ledger-ledger-9bdf7f8bd-29bzm 1/1 Running 0 52s scalardl-ledger-ledger-9bdf7f8bd-9fklw 1/1 Running 0 52s scalardl-ledger-ledger-9bdf7f8bd-9tw5x 1/1 Running 0 52s schema-loader-ledger-schema-loading-46rcr 0/1 Completed 0 3m38s ``` If the ScalarDL Ledger pods are deployed properly, you can see the STATUS are **Running**. 1. Check if the ScalarDL Ledger services are deployed. ```console kubectl get svc ``` [Command execution result] ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 47d postgresql-ledger ClusterIP 10.109.253.150 5432/TCP 15m postgresql-ledger-hl ClusterIP None 5432/TCP 15m scalardl-ledger-envoy ClusterIP 10.106.141.153 50051/TCP,50052/TCP 83s scalardl-ledger-envoy-metrics ClusterIP 10.108.36.136 9001/TCP 83s scalardl-ledger-headless ClusterIP None 50051/TCP,50053/TCP,50052/TCP 83s scalardl-ledger-metrics ClusterIP 10.98.4.217 8080/TCP 83s ``` If the ScalarDL Ledger services are deployed properly, you can see private IP addresses in the CLUSTER-IP column. (Note: `scalardl-ledger-headless` has no CLUSTER-IP.) ## Step 7. Start a Client container We will use certificate files in a Client container. So, we create a secret resource and mount it to a Client container. 1. Create secret resource `client-keys`. ``` kubectl create secret generic client-keys --from-file=certificate=./certs/client.pem --from-file=private-key=./certs/client-key.pem ``` 1. Start a Client container on the Kubernetes cluster. ```console cat << 'EOF' | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: "scalardl-client" spec: containers: - name: scalardl-client image: eclipse-temurin:8-jdk command: ['sleep'] args: ['inf'] volumeMounts: - name: "client-keys" mountPath: "/keys" readOnly: true volumes: - name: "client-keys" secret: secretName: "client-keys" restartPolicy: Never EOF ``` 1. Check if the Client container is running. ```console kubectl get pod scalardl-client ``` [Command execution result] ```console NAME READY STATUS RESTARTS AGE scalardl-client 1/1 Running 0 11s ``` ## Step 8. Run ScalarDL sample contracts in the Client container The following explains the minimum steps. If you want to know more details about ScalarDL and the contract, please refer to the [Getting Started with ScalarDL](https://scalardl.scalar-labs.com/docs/latest/getting-started). 1. Run bash in the Client container. ```console kubectl exec -it scalardl-client -- bash ``` After this step, run each command in the Client container. 1. Install the git, curl and unzip commands in the Client container. ```console apt update && apt install -y git curl unzip ``` 1. Clone ScalarDL Java Client SDK git repository. ```console git clone https://github.com/scalar-labs/scalardl-java-client-sdk.git ``` 1. Change the directory to `scalardl-java-client-sdk/`. ```console cd scalardl-java-client-sdk/ ``` ```console pwd ``` [Command execution result] ```console /scalardl-java-client-sdk ``` 1. Change branch to arbitrary version. ```console git checkout -b v3.6.0 refs/tags/v3.6.0 ``` ```console git branch ``` [Command execution result] ```console master * v3.6.0 ``` If you want to use another version, please specify the version (tag) you want to use. You need to use the same version of ScalarDL Ledger and ScalarDL Java Client SDK. 1. Build the sample contracts. ```console ./gradlew assemble ``` 1. Download CLI tools of ScalarDL from [ScalarDL Java Client SDK Releases](https://github.com/scalar-labs/scalardl-java-client-sdk/releases). ```console curl -OL https://github.com/scalar-labs/scalardl-java-client-sdk/releases/download/v3.6.0/scalardl-java-client-sdk-3.6.0.zip ``` You need to use the same version of CLI tools and ScalarDL Ledger. 1. Unzip the `scalardl-java-client-sdk-3.6.0.zip` file. ```console unzip ./scalardl-java-client-sdk-3.6.0.zip ``` 1. Create a configuration file (client.properties) to access ScalarDL Ledger on the Kubernetes cluster. ```console cat << 'EOF' > client.properties scalar.dl.client.server.host=scalardl-ledger-envoy.default.svc.cluster.local scalar.dl.client.cert_holder_id=client scalar.dl.client.cert_path=/keys/certificate scalar.dl.client.private_key_path=/keys/private-key EOF ``` 1. Register the certificate file of the client. ```console ./scalardl-java-client-sdk-3.6.0/bin/register-cert --properties ./client.properties ``` 1. Register the sample contract `StateUpdater`. ```console ./scalardl-java-client-sdk-3.6.0/bin/register-contract --properties ./client.properties --contract-id StateUpdater --contract-binary-name com.org1.contract.StateUpdater --contract-class-file ./build/classes/java/main/com/org1/contract/StateUpdater.class ``` 1. Register the sample contract `StateReader`. ```console ./scalardl-java-client-sdk-3.6.0/bin/register-contract --properties ./client.properties --contract-id StateReader --contract-binary-name com.org1.contract.StateReader --contract-class-file ./build/classes/java/main/com/org1/contract/StateReader.class ``` 1. Execute the contract `StateUpdater`. ```console ./scalardl-java-client-sdk-3.6.0/bin/execute-contract --properties ./client.properties --contract-id StateUpdater --contract-argument '{"asset_id": "test_asset", "state": 3}' ``` This sample contract updates the `state` (value) of the asset named `test_asset` to `3`. 1. Execute the contract `StateReader`. ```console ./scalardl-java-client-sdk-3.6.0/bin/execute-contract --properties ./client.properties --contract-id StateReader --contract-argument '{"asset_id": "test_asset"}' ``` [Command execution result] ```console Contract result: { "id" : "test_asset", "age" : 0, "output" : { "state" : 3 } } ``` 1. Execute a validation request for the asset. ```console ./scalardl-java-client-sdk-3.6.0/bin/validate-ledger --properties ./client.properties --asset-id "test_asset" ``` [Command execution result] ```console { "status_code" : "OK", "Ledger" : { "id" : "test_asset", "age" : 0, "nonce" : "f31599c6-e6b9-4b77-adc3-61cb5f119bd3", "hash" : "9ExfFl5Lg9IQwdXdW9b87Bi+PWccn3OSNRbhmI/dboo=", "signature" : "MEQCIG6Xa4WOWGMIIbA3PnCje4aAapYfCMerF54xRW0gaUuzAiBCA1nCAPoFWgxArB34/u9b+KeoxQBMALI/pOzMNoLExg==" }, "Auditor" : null } ``` * Reference information * If the asset data is not tampered with, the validation request (validate-ledger command) returns `OK` as a result. * If the asset data is tampered with (e.g. the `state` value in the DB is tampered with), the validation request (validate-ledger command) returns a value other than `OK` (e.g. `INVALID_OUTPUT`) as a result, like the following. [Command execution result (If the asset data is tampered with)] ```console { "status_code" : "INVALID_OUTPUT", "Ledger" : { "id" : "test_asset", "age" : 0, "nonce" : "f31599c6-e6b9-4b77-adc3-61cb5f119bd3", "hash" : "9ExfFl5Lg9IQwdXdW9b87Bi+PWccn3OSNRbhmI/dboo=", "signature" : "MEQCIGtJerW7N93c/bvIBy/7NXxoQwGFznHMmV6RzsgHQg0dAiBu+eBxkfmMQKJY2d9fLNvCH+4b+9rl7gZ3OXJ2NYeVsA==" }, "Auditor" : null } ``` * In this way, the ScalarDL Ledger can detect data tampering. ## Step 9. Delete all resources After completing the ScalarDL Ledger tests on the Kubernetes cluster, remove all resources. 1. Uninstall ScalarDL Ledger, ScalarDL Schema Loader, and PostgreSQL. ```console helm uninstall scalardl-ledger schema-loader-ledger postgresql-ledger ``` 1. Remove the Client container. ``` kubectl delete pod scalardl-client --force --grace-period 0 ``` 1. Remove the working directory and sample files (configuration file, key, and certificate). ```console cd ~ ``` ```console rm -rf ~/scalardl-test/ ``` ## Further reading You can see how to get started with monitoring or logging for Scalar products in the following documents. * [Getting Started with Helm Charts (Monitoring using Prometheus Operator)](getting-started-monitoring.mdx) * [Getting Started with Helm Charts (Logging using Loki Stack)](getting-started-logging.mdx) * [Getting Started with Helm Charts (Scalar Manager)](getting-started-scalar-manager.mdx) ================================================ FILE: docs/helm-charts/how-to-deploy-scalar-admin-for-kubernetes.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to deploy Scalar Admin for Kubernetes This document explains how to deploy Scalar Admin for Kubernetes by using Scalar Helm Charts. For details on the custom values file for Scalar Admin for Kubernetes, see [Configure a custom values file for Scalar Admin for Kubernetes](configure-custom-values-scalar-admin-for-kubernetes.mdx). ## Deploy Scalar Admin for Kubernetes To deploy Scalar Admin for Kubernetes, run the following command, replacing the contents in the angle brackets as described: ```console helm install scalar-labs/scalar-admin-for-kubernetes -n -f / --version ``` ## Upgrade a Scalar Admin for Kubernetes job To upgrade a Scalar Admin for Kubernetes job, run the following command, replacing the contents in the angle brackets as described: ```console helm upgrade scalar-labs/scalar-admin-for-kubernetes -n -f / --version ``` ## Delete a Scalar Admin for Kubernetes job To delete a Scalar Admin for Kubernetes job, run the following command, replacing the contents in the angle brackets as described: ```console helm uninstall -n ``` ================================================ FILE: docs/helm-charts/how-to-deploy-scalar-products.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Deploy Scalar products using Scalar Helm Charts This document explains how to deploy Scalar products using Scalar Helm Charts. If you want to test Scalar products on your local environment using a minikube cluster, please refer to the following getting started guide. * [Getting Started with Scalar Helm Charts](getting-started-scalar-helm-charts.mdx) ## Prerequisites ### Install the helm command You must install the helm command to use Scalar Helm Charts. Please install the helm command according to the [Helm document](https://helm.sh/docs/intro/install/). ### Add the Scalar Helm Charts repository ```console helm repo add scalar-labs https://scalar-labs.github.io/helm-charts ``` ```console helm repo update scalar-labs ``` ### Prepare a Kubernetes cluster You must prepare a Kubernetes cluster for the deployment of Scalar products. If you use EKS (Amazon Elastic Kubernetes Service) or AKS (Azure Kubernetes Service) in the production environment. Please refer to the following document for more details. - [Guidelines for creating an Amazon EKS cluster for Scalar products](../scalar-kubernetes/CreateEKSClusterForScalarProducts.mdx) - [Guidelines for creating an AKS cluster for Scalar products](../scalar-kubernetes/CreateAKSClusterForScalarProducts.mdx) You must prepare a supported version of Kubernetes. For versions that Scalar Helm Charts supports, see [Kubernetes](https://scalardb.scalar-labs.com/docs/latest/requirements/#kubernetes). ### Prepare a database (ScalarDB, ScalarDL Ledger, ScalarDL Auditor) You must prepare a database as a backend storage of ScalarDB/ScalarDL. You can see the supported databases by ScalarDB/ScalarDL in the following document. * [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases) ### Prepare a custom values file You must prepare your custom values file based on your environment. Please refer to the following documents for more details on how to create a custom values file. * [Configure a custom values file for Scalar Helm Charts](configure-custom-values-file.mdx) ### Get the container images If you're using commercially licensed Scalar products, you must get the container images of those products. For details, see [How to get the container images of Scalar products](../scalar-kubernetes/HowToGetContainerImages.mdx). If you're using any of the following products from the public container repository, you can get the container images from the public container repository with the default configuration of Scalar Helm Chart: * Scalar Envoy (deploy with ScalarDB Cluster, ScalarDL Ledger, or ScalarDL Auditor) * ScalarDL Schema Loader * Scalar Admin for Kubernetes ## Deploy Scalar products Please refer to the following documents for more details on how to deploy each product. * [ScalarDB Cluster](how-to-deploy-scalardb-cluster.mdx) * [ScalarDL Ledger](how-to-deploy-scalardl-ledger.mdx) * [ScalarDL Auditor](how-to-deploy-scalardl-auditor.mdx) * [Scalar Admin for Kubernetes](how-to-deploy-scalar-admin-for-kubernetes.mdx) * [Scalar Manager](getting-started-scalar-manager.mdx) * [[Deprecated] ScalarDB Server](how-to-deploy-scalardb.mdx) * [[Deprecated] ScalarDB GraphQL](how-to-deploy-scalardb-graphql.mdx) ================================================ FILE: docs/helm-charts/how-to-deploy-scalardb-cluster.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to deploy ScalarDB Cluster This document explains how to deploy ScalarDB Cluster by using Scalar Helm Charts. For details on the custom values file for ScalarDB Cluster, see [Configure a custom values file for ScalarDB Cluster](configure-custom-values-scalardb-cluster.mdx). ## Deploy ScalarDB Cluster ```console helm install scalar-labs/scalardb-cluster -n -f / --version ``` ## Upgrade a ScalarDB Cluster deployment ```console helm upgrade scalar-labs/scalardb-cluster -n -f / --version ``` ## Delete a ScalarDB Cluster deployment ```console helm uninstall -n ``` ## Deploy your client application on Kubernetes with `direct-kubernetes` mode If you use ScalarDB Cluster with `direct-kubernetes` mode, you must: 1. Deploy your application pods on the same Kubernetes cluster as ScalarDB Cluster. 2. Create three Kubernetes resources (`Role`, `RoleBinding`, and `ServiceAccount`). 3. Mount the `ServiceAccount` on your application pods. This method is necessary because the ScalarDB Cluster client library with `direct-kubernetes` mode runs the Kubernetes API from inside of your application pods to get information about the ScalarDB Cluster pods. * Role ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: scalardb-cluster-client-role namespace: rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "watch", "list"] ``` * RoleBinding ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: scalardb-cluster-client-rolebinding namespace: subjects: - kind: ServiceAccount name: scalardb-cluster-client-sa roleRef: kind: Role name: scalardb-cluster-client-role apiGroup: rbac.authorization.k8s.io ``` * ServiceAccount ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: scalardb-cluster-client-sa namespace: ``` ================================================ FILE: docs/helm-charts/how-to-deploy-scalardb-graphql.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # [Deprecated] How to deploy ScalarDB GraphQL :::note ScalarDB GraphQL Server is now deprecated. Please use [ScalarDB Cluster](how-to-deploy-scalardb-cluster.mdx) instead. ::: This document explains how to deploy ScalarDB GraphQL using Scalar Helm Charts. You must prepare your custom values file. Please refer to the following document for more details on the custom values file for ScalarDB GraphQL. * [[Deprecated] Configure a custom values file for ScalarDB GraphQL](configure-custom-values-scalardb-graphql.mdx) ## Deploy ScalarDB Server (recommended option) When you deploy ScalarDB GraphQL, it is recommended to deploy ScalarDB Server between ScalarDB GraphQL and backend databases as follows. ``` [Client] ---> [ScalarDB GraphQL] ---> [ScalarDB Server] ---> [Backend databases] ``` Please deploy ScalarDB Server before you deploy ScalarDB GraphQL according to the document [How to deploy ScalarDB Server](how-to-deploy-scalardb.mdx). ## Deploy ScalarDB GraphQL ```console helm install scalar-labs/scalardb-graphql -n -f / --version ``` ## Upgrade the deployment of ScalarDB GraphQL ```console helm upgrade scalar-labs/scalardb-graphql -n -f / --version ``` ## Delete the deployment of ScalarDB GraphQL ```console helm uninstall -n ``` ================================================ FILE: docs/helm-charts/how-to-deploy-scalardb.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Deprecated displayed_sidebar: docsEnglish --- # [Deprecated] How to deploy ScalarDB Server :::note ScalarDB Server is now deprecated. Please use [ScalarDB Cluster](how-to-deploy-scalardb-cluster.mdx) instead. ::: This document explains how to deploy ScalarDB Server using Scalar Helm Charts. You must prepare your custom values file. Please refer to the following document for more details on the custom values file for ScalarDB Server. * [[Deprecated] Configure a custom values file for ScalarDB Server](configure-custom-values-scalardb.mdx) ## Deploy ScalarDB Server ```console helm install scalar-labs/scalardb -n -f / --version ``` ## Upgrade the deployment of ScalarDB Server ```console helm upgrade scalar-labs/scalardb -n -f / --version ``` ## Delete the deployment of ScalarDB Server ```console helm uninstall -n ``` ================================================ FILE: docs/helm-charts/how-to-deploy-scalardl-auditor.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # How to deploy ScalarDL Auditor This document explains how to deploy ScalarDL Auditor using Scalar Helm Charts. You must prepare your custom values file. Please refer to the following document for more details on the custom values file for ScalarDL Auditor and ScalarDL Schema Loader. * [Configure a custom values file for ScalarDL Auditor](configure-custom-values-scalardl-auditor.mdx) * [Configure a custom values file for ScalarDL Schema Loader](configure-custom-values-scalardl-schema-loader.mdx) ## Prepare a private key file and a certificate file When you deploy ScalarDL Auditor, you must create a Secrete resource to mount the private key file and the certificate file on the ScalarDL Auditor pods. For more details on how to mount the key and certificate files on the ScalarDL pods, refer to [Mount key and certificate files on a pod in ScalarDL Helm Charts](mount-files-or-volumes-on-scalar-pods.mdx#mount-key-and-certificate-files-on-a-pod-in-scalardl-helm-charts). ## Create schemas for ScalarDL Auditor (Deploy ScalarDL Schema Loader) Before you deploy ScalarDL Auditor, you must create schemas for ScalarDL Auditor on the backend database. ```console helm install scalar-labs/schema-loading -n -f / --version ``` ## Deploy ScalarDL Auditor ```console helm install scalar-labs/scalardl-audit -n -f / --version ``` ## Upgrade the deployment of ScalarDL Auditor ```console helm upgrade scalar-labs/scalardl-audit -n -f / --version ``` ## Delete the deployment of ScalarDL Auditor and ScalarDL Schema Loader ```console helm uninstall -n ``` ================================================ FILE: docs/helm-charts/how-to-deploy-scalardl-ledger.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # How to deploy ScalarDL Ledger This document explains how to deploy ScalarDL Ledger using Scalar Helm Charts. You must prepare your custom values file. Please refer to the following document for more details on the custom values file for ScalarDL Ledger and ScalarDL Schema Loader. * [Configure a custom values file for ScalarDL Ledger](configure-custom-values-scalardl-ledger.mdx) * [Configure a custom values file for ScalarDL Schema Loader](configure-custom-values-scalardl-schema-loader.mdx) ## Prepare a private key file (optional / it is necessary if you use ScalarDL Auditor) If you use the [asset proofs](https://scalardl.scalar-labs.com/docs/latest/how-to-write-applications#what-is-asset-proof) of ScalarDL Ledger, you must create a Secrete resource to mount the private key file on the ScalarDL Ledger pods. If you use ScalarDL Auditor, asset proof is necessary. Please refer to the following document for more details on how to mount the key/certificate files on the ScalarDL pods. * [Mount key and certificate files on a pod in ScalarDL Helm Charts](mount-files-or-volumes-on-scalar-pods.mdx#mount-key-and-certificate-files-on-a-pod-in-scalardl-helm-charts) ## Create schemas for ScalarDL Ledger (Deploy ScalarDL Schema Loader) Before you deploy ScalarDL Ledger, you must create schemas for ScalarDL Ledger on the backend database. ```console helm install scalar-labs/schema-loading -n -f / --version ``` ## Deploy ScalarDL Ledger ```console helm install scalar-labs/scalardl -n -f / --version ``` ## Upgrade the deployment of ScalarDL Ledger ```console helm upgrade scalar-labs/scalardl -n -f / --version ``` ## Delete the deployment of ScalarDL Ledger and ScalarDL Schema Loader ```console helm uninstall -n ``` ================================================ FILE: docs/helm-charts/mount-files-or-volumes-on-scalar-pods.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Mount any files or volumes on Scalar product pods You can mount any files or volumes on Scalar product pods when you use ScalarDB Server, ScalarDB Cluster, or ScalarDL Helm Charts (ScalarDL Ledger and ScalarDL Auditor). ## Mount a private key file on a pod in ScalarDL Helm Charts You must mount the private key file to run ScalarDL Auditor. * Configuration example * ScalarDL Ledger ```yaml ledger: ledgerProperties: | ... scalar.dl.ledger.proof.enabled=true scalar.dl.ledger.auditor.enabled=true scalar.dl.ledger.proof.private_key_path=/keys/private-key ``` * ScalarDL Auditor ```yaml auditor: auditorProperties: | ... scalar.dl.auditor.private_key_path=/keys/private-key ``` In this example, you need to mount a **private-key** file under the `/keys` directory in the container. And, you need to mount a file named `private-key`. You can use `extraVolumes` and `extraVolumeMounts` to mount this file. 1. Set `extraVolumes` and `extraVolumeMounts` in the custom values file using the same syntax of Kubernetes manifest. You need to specify the directory name to the key `mountPath`. * Example * ScalarDL Ledger ```yaml ledger: extraVolumes: - name: ledger-keys secret: secretName: ledger-keys extraVolumeMounts: - name: ledger-keys mountPath: /keys readOnly: true ``` * ScalarDL Auditor ```yaml auditor: extraVolumes: - name: auditor-keys secret: secretName: auditor-keys extraVolumeMounts: - name: auditor-keys mountPath: /keys readOnly: true ``` 1. Create a `Secret` resource that includes a private key file. You need to specify the file name as keys of `Secret`. * Example * ScalarDL Ledger ```console kubectl create secret generic ledger-keys \ --from-file=private-key=./ledger-key.pem ``` * ScalarDL Auditor ```console kubectl create secret generic auditor-keys \ --from-file=private-key=./auditor-key.pem ``` 1. Deploy Scalar products with the above custom values file. After deploying Scalar products, the private key file is mounted under the `/keys` directory as follows. * Example * ScalarDL Ledger ```console ls -l /keys/ ``` You should see the following output: ```console total 0 lrwxrwxrwx 1 root root 18 Jun 27 03:12 private-key -> ..data/private-key ``` * ScalarDL Auditor ```console ls -l /keys/ ``` You should see the following output: ```console total 0 lrwxrwxrwx 1 root root 18 Jun 27 03:16 private-key -> ..data/private-key ``` ## Mount emptyDir to get a heap dump file You can mount emptyDir to Scalar product pods by using the following keys in your custom values file. For example, you can use this volume to get a heap dump of Scalar products. * Keys * `scalardb.extraVolumes` / `scalardb.extraVolumeMounts` (ScalarDB Server) * `scalardbCluster.extraVolumes` / `scalardbCluster.extraVolumeMounts` (ScalarDB Cluster) * `ledger.extraVolumes` / `ledger.extraVolumeMounts` (ScalarDL Ledger) * `auditor.extraVolumes` / `auditor.extraVolumeMounts` (ScalarDL Auditor) * Example (ScalarDB Server) ```yaml scalardb: extraVolumes: - name: heap-dump emptyDir: {} extraVolumeMounts: - name: heap-dump mountPath: /dump ``` In this example, you can see the mounted volume in the ScalarDB Server pod as follows. ```console ls -ld /dump ``` You should see the following output: ```console drwxrwxrwx 2 root root 4096 Feb 6 07:43 /dump ``` ================================================ FILE: docs/helm-charts/use-secret-for-credentials.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to use Secret resources to pass credentials as environment variables into the properties file import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; You can pass credentials like **username** or **password** as environment variables via a `Secret` resource in Kubernetes. The docker images for previous versions of Scalar products use the `dockerize` command for templating properties files. The docker images for the latest versions of Scalar products get values directly from environment variables. Note: You cannot use the following environment variable names in your custom values file since these are used in the Scalar Helm Chart internal. ```console HELM_SCALAR_DB_CONTACT_POINTS HELM_SCALAR_DB_CONTACT_PORT HELM_SCALAR_DB_USERNAME HELM_SCALAR_DB_PASSWORD HELM_SCALAR_DB_STORAGE HELM_SCALAR_DL_LEDGER_PROOF_ENABLED HELM_SCALAR_DL_LEDGER_AUDITOR_ENABLED HELM_SCALAR_DL_LEDGER_PROOF_PRIVATE_KEY_PATH HELM_SCALAR_DL_AUDITOR_SERVER_PORT HELM_SCALAR_DL_AUDITOR_SERVER_PRIVILEGED_PORT HELM_SCALAR_DL_AUDITOR_SERVER_ADMIN_PORT HELM_SCALAR_DL_AUDITOR_LEDGER_HOST HELM_SCALAR_DL_AUDITOR_CERT_HOLDER_ID HELM_SCALAR_DL_AUDITOR_CERT_VERSION HELM_SCALAR_DL_AUDITOR_CERT_PATH HELM_SCALAR_DL_AUDITOR_PRIVATE_KEY_PATH SCALAR_DB_LOG_LEVEL SCALAR_DL_LEDGER_LOG_LEVEL SCALAR_DL_AUDITOR_LOG_LEVEL SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAMESPACE_NAME SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAME ``` 1. Set environment variable name to the properties configuration in the custom values file. See the following examples based on the product you're using. ```yaml scalardbCluster: scalardbClusterNodeProperties: | ... scalar.db.username=${env:SCALAR_DB_USERNAME} scalar.db.password=${env:SCALAR_DB_PASSWORD} ... ```

ScalarDB Server 3.8 or later (Apache Commons Text syntax)

```yaml scalardb: databaseProperties: | ... scalar.db.username=${env:SCALAR_DB_USERNAME} scalar.db.password=${env:SCALAR_DB_PASSWORD} ... ```

ScalarDB Server 3.7 or earlier (Go template syntax)

```yaml scalardb: databaseProperties: | ... scalar.db.username={{ default .Env.SCALAR_DB_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DB_PASSWORD "" }} ... ```

ScalarDL Ledger 3.8 or later (Apache Commons Text syntax)

```yaml ledger: ledgerProperties: | ... scalar.db.username=${env:SCALAR_DB_USERNAME} scalar.db.password=${env:SCALAR_DB_PASSWORD} ... ```

ScalarDL Ledger 3.7 or earlier (Go template syntax)

```yaml ledger: ledgerProperties: | ... scalar.db.username={{ default .Env.SCALAR_DB_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DB_PASSWORD "" }} ... ```

ScalarDL Auditor 3.8 or later (Apache Commons Text syntax)

```yaml auditor: auditorProperties: | ... scalar.db.username=${env:SCALAR_DB_USERNAME} scalar.db.password=${env:SCALAR_DB_PASSWORD} ... ```

ScalarDL Auditor 3.7 or earlier (Go template syntax)

```yaml auditor: auditorProperties: | ... scalar.db.username={{ default .Env.SCALAR_DB_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DB_PASSWORD "" }} ... ```

ScalarDL Schema Loader 3.8 or later (Apache Commons Text syntax)

```yaml schemaLoading: databaseProperties: | ... scalar.db.username=${env:SCALAR_DB_USERNAME} scalar.db.password=${env:SCALAR_DB_PASSWORD} ... ```

ScalarDL Schema Loader 3.7 or earlier (Go template syntax)

```yaml schemaLoading: databaseProperties: | ... scalar.db.username={{ default .Env.SCALAR_DB_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DB_PASSWORD "" }} ... ``` 1. Create a `Secret` resource that includes credentials. You need to specify the environment variable name as keys of the `Secret`. * Example ```console kubectl create secret generic scalardb-credentials-secret \ --from-literal=SCALAR_DB_USERNAME=postgres \ --from-literal=SCALAR_DB_PASSWORD=postgres ``` 1. Set the `Secret` name to the following keys in the custom values file. See the following examples based on the product you're using. **Key:** `scalardbCluster.secretName` ```yaml scalardbCluster: secretName: "scalardb-cluster-credentials-secret" ``` **Key:** `scalardb.secretName` ```yaml scalardb: secretName: "scalardb-credentials-secret" ``` **Key:** `ledger.secretName` ```yaml ledger: secretName: "ledger-credentials-secret" ``` **Key:** `auditor.secretName` ```yaml auditor: secretName: "auditor-credentials-secret" ``` **Key:** `schemaLoading.secretName` ```yaml schemaLoading: secretName: "schema-loader-ledger-credentials-secret" ``` 1. Deploy Scalar products with the above custom values file. After deploying Scalar products, the Go template strings (environment variables) are replaced by the values of the `Secret`. * Example * Custom values file ```yaml scalardb: databaseProperties: | scalar.db.contact_points=jdbc:postgresql://postgresql-scalardb.default.svc.cluster.local:5432/postgres scalar.db.username={{ default .Env.SCALAR_DB_USERNAME "" }} scalar.db.password={{ default .Env.SCALAR_DB_PASSWORD "" }} scalar.db.storage=jdbc ``` * Properties file in containers ```properties scalar.db.contact_points=jdbc:postgresql://postgresql-scalardb.default.svc.cluster.local:5432/postgres scalar.db.username=postgres scalar.db.password=postgres scalar.db.storage=jdbc ``` If you use Apache Commons Text syntax, Scalar products get values directly from environment variables. ================================================ FILE: docs/releases/release-notes.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB 3.18 Release Notes This page includes a list of release notes for ScalarDB 3.18. ## v3.18.0 **Release date:** May 1, 2026 ### Summary This release includes a lot of enhancements, improvements, security issue fixes, and bug fixes. :::warning Backward-incompatible changes - **JDBC connection pool migrated from Apache Commons DBCP2 to HikariCP:** - The following configuration properties are no longer supported. They are ignored, with a warning logged, if set: - `scalar.db.jdbc.connection_pool.max_idle` - `scalar.db.jdbc.table_metadata.connection_pool.max_idle` - `scalar.db.jdbc.admin.connection_pool.max_idle` - `scalar.db.jdbc.prepared_statements_pool.enabled` - `scalar.db.jdbc.prepared_statements_pool.max_open` - Prepared statement pooling is no longer available because HikariCP has no equivalent feature. - Connection pool default behavior follows HikariCP rather than DBCP2, which may change runtime behavior even without configuration changes (for example, connection acquisition timeout and idle eviction). - **Existing tables require `repairTable()` to create the new before-image secondary index.** Index-based `Get`, `Scan`, and `ScanAll` in Consensus Commit now rely on a companion `before_` index. Until you run `repairTable()` on each existing table, ScalarDB logs a warning at startup and index-based reads fall back to the previous behavior, which may miss records whose indexed column is being concurrently updated. The configuration `scalar.db.consensus_commit.index.eventually_consistent_read.enabled` is provided as an opt-out but is not recommended for new workloads. - **MySQL Connector/J replaced with MariaDB Connector/J.** SSL is no longer enabled by default. If you connect to MySQL 8.0 or later with `caching_sha2_password` authentication, add `sslMode=REQUIRED` or `allowPublicKeyRetrieval=true` to your JDBC URL. No action is required for `jdbc:mysql://` URLs because ScalarDB automatically appends `permitMysqlScheme` for compatibility. - **`BigIntColumn.MAX_VALUE` and `BigIntColumn.MIN_VALUE` constants have been removed.** Code that referenced them must use `Long.MAX_VALUE` and `Long.MIN_VALUE` instead. ::: ### Community edition #### Enhancements - Added `AuthAdmin.getRole(roleName)`. ([#3238](https://github.com/scalar-labs/scalardb/pull/3238)) - Added operation-level attributes to allow cross-partition scan, filtering, and ordering on a per-operation basis without requiring global configuration. ([#3425](https://github.com/scalar-labs/scalardb/pull/3425)) - Added `DistributedTransactionAdmin.hasPrivilege()`. ([#3432](https://github.com/scalar-labs/scalardb/pull/3432)) - Added `AuthenticationMethod` enum and authentication-method-aware user management APIs to `AuthAdmin` to support OIDC authentication. ([#3433](https://github.com/scalar-labs/scalardb/pull/3433)) - Added `attributes` parameter to `DistributedTransactionManager` `begin` and `start` methods, enabling transaction-scoped attribute configuration. CRUD methods on `DistributedTransactionManager` also pass the operation's attributes to the internally begun transaction. Also added support for specifying the isolation level per transaction in Consensus Commit via the `cc-transaction-isolation` attribute. ([#3466](https://github.com/scalar-labs/scalardb/pull/3466) [#3517](https://github.com/scalar-labs/scalardb/pull/3517) [#3540](https://github.com/scalar-labs/scalardb/pull/3540)) - Added support for Google Cloud Spanner as a JDBC storage, via its PostgreSQL-compatible dialect. ([#3510](https://github.com/scalar-labs/scalardb/pull/3510)) #### Improvements - Extended the applicability of one-phase commit optimization in the Consensus Commit protocol. This allows one-phase commit to be used even in SERIALIZABLE isolation level when the transaction only reads records that it subsequently updates, improving performance for read-modify-write workloads. ([#3295](https://github.com/scalar-labs/scalardb/pull/3295)) - Added batch execution support for mutate operations in the JDBC adapter to improve performance when executing multiple mutations. Mutations with the same SQL statement are now grouped and executed as a batch. ([#3304](https://github.com/scalar-labs/scalardb/pull/3304)) - Migrated the JDBC adapter's connection pooling library from Apache Commons DBCP2 to HikariCP for improved performance and reliability. ([#3305](https://github.com/scalar-labs/scalardb/pull/3305)) - Changed the behavior of Consensus Commit transactions to allow Insert/Upsert/Update operations after Delete on the same record within the same transaction. Previously, this operation threw an `IllegalArgumentException`, but now it properly handles the case by treating it as a new record insertion with null values for unspecified columns. ([#3319](https://github.com/scalar-labs/scalardb/pull/3319)) - Added @CheckReturnValue annotation to Get/Scan builders. ([#3320](https://github.com/scalar-labs/scalardb/pull/3320)) - Added support for setting null values on secondary index columns in DynamoDB. When a null value is set, the attribute is removed from the item and the record will not appear in secondary index scans. ([#3326](https://github.com/scalar-labs/scalardb/pull/3326)) - Inserting or updating records with TIME (microsecond precision), TIMESTAMP (millisecond precision), and TIMESTAMPTZ (millisecond precision) column values will truncate out-of-range precision rather than throwing an exception. ([#3393](https://github.com/scalar-labs/scalardb/pull/3393)) - Fixed a correctness issue in index-based Get, Scan, and ScanAll operations in Consensus Commit where records whose indexed column was being concurrently updated by another transaction could be missed. ScalarDB now maintains a companion before-image secondary index for each user-defined secondary index on a non-primary-key column and uses it to recover PREPARED/DELETED records during index reads. ([#3419](https://github.com/scalar-labs/scalardb/pull/3419) [#3463](https://github.com/scalar-labs/scalardb/pull/3463)) - Replaced MySQL Connector/J with MariaDB Connector/J to resolve GPLv2 licensing concerns. ([#3428](https://github.com/scalar-labs/scalardb/pull/3428)) - Shortened JDBC index names using a hash when they exceed the maximum identifier length supported by the underlying database. ([#3481](https://github.com/scalar-labs/scalardb/pull/3481)) - Relaxed the BigInt value range restriction (`-2^53` to `2^53`) that previously applied to all storage backends. This restriction now only applies to Cosmos DB and Object Storage. Other backends (JDBC, Cassandra, DynamoDB) now support the full Java long range for BigInt columns. ([#3490](https://github.com/scalar-labs/scalardb/pull/3490)) - Changed the Oracle BIGINT column type mapping from NUMBER(16) to NUMBER(19) to support the full Java long range. ([#3507](https://github.com/scalar-labs/scalardb/pull/3507)) - Changed the deprecation policy so that APIs and configurations marked as deprecated will now be removed in 4.0.0 instead of 5.0.0. ([#3520](https://github.com/scalar-labs/scalardb/pull/3520)) #### Bug fixes - Fixed option issues in Object Storage adapter. ([#3237](https://github.com/scalar-labs/scalardb/pull/3237)) - On Oracle, when importing a table with a column using the `NUMBER(1)` data type, which is usually used for BOOLEAN data, that column can now be mapped to ScalarDB BOOLEAN using ScalarDB Schema Loader `override-columns-type` setting. ([#3239](https://github.com/scalar-labs/scalardb/pull/3239)) - Fix to increase the maximum allowed string length with Object Storage. ([#3248](https://github.com/scalar-labs/scalardb/pull/3248)) - Updated the upper limit value displayed in the error message for data size limitation in Object Storage adapter. ([#3264](https://github.com/scalar-labs/scalardb/pull/3264)) - Added explicit commits for Oracle database when using SERIALIZABLE isolation level to ensure snapshot updates after each operation. ([#3294](https://github.com/scalar-labs/scalardb/pull/3294)) - Upgraded the Jackson library to fix a security issue: [GHSA-72hv-8253-57qq](https://github.com/advisories/GHSA-72hv-8253-57qq "GHSA-72hv-8253-57qq") ([#3394](https://github.com/scalar-labs/scalardb/pull/3394)) - Fix `dropColumnFromTable()` dropping the secondary index even when column drop is unsupported. ([#3450](https://github.com/scalar-labs/scalardb/pull/3450)) - Upgraded the Netty library to fix security issues: [CVE-2026-33870](https://github.com/advisories/GHSA-pwqr-wmgm-9rr8 "CVE-2026-33870") and [CVE-2026-33871](https://github.com/advisories/GHSA-w9fj-cfpg-grvv "CVE-2026-33871") ([#3452](https://github.com/scalar-labs/scalardb/pull/3452)) - Fixed a bug where index-based Get and Scan operations could return incorrect results after lazy recovery rolled back a PREPARED record whose after-image index value matched the query but whose before-image (restored) value did not. ([#3488](https://github.com/scalar-labs/scalardb/pull/3488)) ### Enterprise edition #### Enhancements ##### ScalarDB Cluster - Added `getRole()` API and equivalent to retrieve a single role by name. - Added support for starting replication with existing backup site tables. - Added support for the `executeBatch` API in ScalarDB Cluster SQL transactions, enabling batch execution of multiple SQL statements in a single call. - Added client-side optimizations for SQL transactions (piggyback_begin and write_buffering) to reduce RPC overhead between the client and the cluster. - Added support for configuring a separate gRPC port for the admin service by using the `scalar.db.cluster.node.admin.port` property. - Added `DistributedTransactionAdmin.hasPrivilege()` to check if a specified user has a specific privilege on a table. - Added support for the attributes parameter in transaction begin/start methods, enabling transaction-scoped configuration such as isolation level. - Added OIDC JWT authentication support. - Added ThreadLocal-based user-password authentication support via `CredentialsHolder`, allowing multiple users to share a single `TransactionManager`. - Added property-based OIDC JWT authentication support for the client, allowing JWT access tokens to be passed via configuration properties. ##### ScalarDB SQL - Added `Metadata.getRole(roleName)` - Added support for Spring Boot 4 in Spring Data JDBC for ScalarDB. - Added support for extending `AbstractJdbcConfiguration` for custom bean configuration in Spring Data JDBC for ScalarDB. - Changed the `PreparedStatement` API to use `bind()` methods that return a `BoundStatement` for parameter binding, instead of directly setting values on `PreparedStatement`. - Added `executeBatch` API to execute multiple statements in a single call, with support in the core SQL API, direct-mode, and JDBC driver. - Added support for BLOB literals using X'hex' syntax in SQL statements. - Added support for AuthenticationMethod in `CREATE/ALTER USER` and `SHOW USERS` - Added support for specifying transaction-scoped attributes in `BEGIN` and `START TRANSACTION` SQL statements using the `WITH` clause (e.g., `BEGIN WITH 'cc-transaction-isolation' = 'SNAPSHOT'`). Also added `begin(Map)`, and `beginReadOnly(Map)` methods to `SqlSession` for programmatic attribute specification. - Updated `SqlJdbcDatabaseMetaData.getUserName()` to return the authenticated user name via `Metadata.getCurrentUser()`. - Added support for specifying ABAC read and write tags in the `WITH` clause of `BEGIN` and `START TRANSACTION` statements. #### Improvements ##### ScalarDB Cluster - Refactored internal auth token handling to use a type-safe class hierarchy instead of string prefix parsing. - Auth, ABAC, and Encryption modules no longer require `scalar.db.cross_partition_scan.enabled=true` to be enabled globally. These modules now use operation-level cross-partition scan attributes and run internal metadata transactions with Snapshot Isolation. - Changed the deprecation policy so that APIs marked as deprecated will now be removed in 4.0.0 instead of 5.0.0. ##### ScalarDB GraphQL - Relaxed the range of values accepted by the GraphQL `BigInt` scalar to the full Java `long` range, aligning with the corresponding relaxation in ScalarDB. Backend-specific limits (Cosmos DB and Object Storage still restrict BigInt to `-2^53` to `2^53`) are now enforced by ScalarDB at the storage layer rather than by the GraphQL layer. ##### ScalarDB SQL - Improve thread pool management in two-phase commit interface transaction to avoid potential starvation issues in Spring Data JDBC for ScalarDB. - Added one-operation mode support for one-shot query execution to improve performance by skipping explicit transaction begin/commit when the SQL query can be expressed as a single operation. - Inserting or updating records with TIME (microsecond precision), TIMESTAMP (millisecond precision), and TIMESTAMPTZ (millisecond precision) column values will truncate out-of-range precision rather than throwing an exception. - Fixed an issue where the shadow jar was unnecessarily published to GitHub Packages for the CLI module. - Update the TIMESTAMPTZ literal to make optional the space character before the UTC timezone `Z` character. For example, `2021-03-04 12:30:45.123Z` is now accepted, in addition to the current format `2021-03-04 12:30:45.123 Z`. Also, when selecting a TIMESTAMPTZ column, the value is printed without a space before the `Z`. - Refactored the SQL statement builder APIs for improved type safety and ergonomics. `SelectStatementBuilder` now enforces SQL clause ordering at the type level so that `having()` is only callable after `groupBy()`, and HAVING supports the same fluent `.and(...)` / `.or(...)` chain as WHERE; `Having.of(...)` factories have been consolidated into `Having.create(...)`; `CreateUserStatementBuilder` / `AlterUserStatementBuilder` expose explicit `withPassword()` / `withSuperuser()` / `withNoSuperuser()` methods in place of the previous overloaded `with(...)`; the `UserOption` enum has been removed in favor of a nullable `superuser` boolean on `CreateUserStatement` / `AlterUserStatement`; and `FunctionRef` now provides convenience factories (`count()`, `sum(...)`, `avg(...)`, `min(...)`, `max(...)`) for common aggregates. #### Bug fixes ##### ScalarDB Cluster - Made `GRANT ROLE` command idempotent, allowing duplicate grants and upgrading to `WITH ADMIN OPTION` when re-granting. - Fixed a bug where ScalarDB Cluster cannot be deployed in the Omnistrate environment by upgrading `scalar-metering`. - Fixed a bug where the pause functionality did not work correctly when transactions expired. - Fixed an issue where the batch operation with piggyback commit threw `CrudException` instead of `CrudConflictException` when a commit conflict occurred. This allows clients to properly detect and handle commit conflicts. - Fixed a bug where `batch()` with piggyback commit threw `CrudException` instead of `UnknownTransactionStatusException` on unexpected gRPC errors. This could cause incorrect error handling on the client side, as the transaction status was actually unknown when piggyback commit was enabled. - Upgraded `grpc_health_probe` to fix security issues: [CVE-2025-68121](https://github.com/advisories/GHSA-h355-32pf-p2xm "CVE-2025-68121"), [CVE-2025-61726](https://github.com/advisories/GHSA-gm9r-q53w-2gh4 "CVE-2025-61726"), [CVE-2025-61728](https://github.com/advisories/GHSA-g9q4-qjx4-2v7q "CVE-2025-61728"), [CVE-2025-61729](https://github.com/advisories/GHSA-7c64-f9jr-v9h2 "CVE-2025-61729"), and [CVE-2025-61730](https://github.com/advisories/GHSA-gr56-3gp6-6gmj "CVE-2025-61730") - Excluded `com.microsoft.azure:adal4j` from the Kubernetes Java Client to fix security issues: [CVE-2023-52428](https://github.com/advisories/GHSA-gvpg-vgmx-xg6w "CVE-2023-52428"), [CVE-2021-31684](https://github.com/advisories/GHSA-fg2v-w576-w4v3 "CVE-2021-31684"), and [CVE-2023-1370](https://github.com/advisories/GHSA-493p-pfq6-5258 "CVE-2023-1370") - Upgraded the Jackson library to fix a security issue: [GHSA-72hv-8253-57qq](https://github.com/advisories/GHSA-72hv-8253-57qq "GHSA-72hv-8253-57qq") - Upgraded `grpc_health_probe` to fix security issues: [CVE-2025-59250](https://github.com/advisories/GHSA-m494-w24q-6f7w "CVE-2025-59250") and [CVE-2026-25679](https://github.com/advisories/GHSA-j3gx-2473-5fp8 "CVE-2026-25679") - Upgraded the Netty library to fix security issues: [CVE-2026-33870](https://github.com/advisories/GHSA-pwqr-wmgm-9rr8 "CVE-2026-33870") and [CVE-2026-33871](https://github.com/advisories/GHSA-w9fj-cfpg-grvv "CVE-2026-33871") - Fixed a bug where one-shot batch operations bypassed pause control in the GateKept transaction managers. - Upgraded `grpc_health_probe` to fix a security issue: [CVE-2026-34986](https://github.com/advisories/GHSA-78h2-9frx-2jm8 "CVE-2026-34986"). ##### ScalarDB SQL - Ambiguous column names in ORDER BY and HAVING clauses were detected. - Fixed a `ClassCastException` that occurred in `StatementUtils.appendTerm()` when handling `DATE`, `TIME`, `TIMESTAMP`, and `TIMESTAMPTZ` values. These values are now correctly formatted as string literals instead of being incorrectly cast to String. - Fixed `SelectStatement.toSql()` to correctly generate ORDER BY clauses with aggregate functions such as `SUM()` and `COUNT()`. Previously, only column-based orderings were handled, causing incorrect SQL output when function-based orderings were used. - Fixed a bug where duplicate column names were allowed in CREATE TABLE statements. - Fixed missing strict date validation on time-related type formatters, which could allow invalid dates (e.g., February 30) to be silently accepted instead of rejected. - `ResultSet.getObject` on BLOB columns now returns a `java.sql.Blob`, and `ResultSet.getBlob`, `ResultSet.getBinaryStream`, and several `PreparedStatement.setBlob` / `setBinaryStream` overloads are now supported. `ResultSetMetaData.getColumnType` for FLOAT columns now returns `Types.REAL` (previously `Types.FLOAT`). `ResultSet.getString` now returns Java `null` for SQL NULL instead of the literal string `"null"`, in line with the JDBC spec. As a side effect, the SQL CLI now renders BLOB values as `X'...'` hex literals instead of `[B@xxxxxxxx`. - Fixed an issue where `CachedMetadata#invalidateNamespaceNamesCache()` did not actually invalidate the cached list of namespaces, causing `SHOW NAMESPACES` and related operations to potentially return stale results until the cache TTL expired. ### Enterprise Options #### ScalarDB Analytics ##### Enhancements - Password authentication with access token support. - Internal user directory management. - gRPC authentication interceptor. - SDK and CLI authentication with token caching. - Spark data source authentication support. - ScalarDB Cluster password authentication backend. - gRPC retry with exponential backoff for Client SDK. ##### Improvements - Set scan fetch size to 4096 for JDBC and ScalarDB data sources. ##### Breaking changes - Unify ScalarDB namespace to `scalardb_analytics` and add domain prefixes to table names (`registry_`, `auth_`) to avoid PostgreSQL identifier length limits. ##### Bug fixes - Use TIMESTAMPTZ instead of TIMESTAMP for temporal columns. ================================================ FILE: docs/releases/release-support-policy.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Release Support Policy This page describes Scalar's support policy for major and minor version releases of ScalarDB. ## Terms and definitions - **Maintenance Support:** Scalar will provide product updates, including code fixes and documentation, and technical support through its [support portal](https://support.scalar-labs.com/) to customers with a commercial license, until the date specified. - **Assistance Support:** Scalar will provide limited technical support for non-code-related questions in the form of FAQs and inquiries through its [support portal](https://support.scalar-labs.com/) to customers with a commercial license until the date specified. - **Extended Support:** Extended Support is available as an add-on for customers with a commercial license who want support for a version that is no longer under Maintenance Support or Assistance Support. ## Release support timelines
Version Release Date Maintenance Support Ends Assistance Support Ends Extended Support
3.18 2026-05-01 TBD* TBD* Contact us
3.17 2025-11-26 2027-05-01 2027-10-28 Contact us
3.16 2025-06-23 2026-11-26 2027-05-26 Contact us
3.15 2025-02-20 2026-06-23 2026-12-20 Contact us
3.14 2024-11-22 2026-02-20 2026-08-18 Contact us
3.13** 2024-07-08 2025-11-22 2026-05-21 Contact us
3.12** 2024-02-17 2025-07-08 2026-01-04 Contact us
3.11** 2023-12-27 2025-02-16 2025-08-15 Contact us
3.10** 2023-07-20 2024-12-26 2025-06-24 Contact us
3.9** 2023-04-27 2024-07-19 2025-01-15 Contact us
3.8** 2023-01-17 2024-04-26 2024-10-23 Contact us
3.7** 2022-09-03 2024-01-17 2024-07-15 Contact us
3.6** 2022-07-08 2023-09-03 2024-03-01 Contact us
3.5** 2022-02-16 2023-07-08 2024-01-04 Contact us
3.4** 2021-12-02 2023-02-16 2023-08-15 Contact us
\* "TBD" will be replaced with a date after the next minor version is released.
\*\* This product version is no longer supported under Maintenance Support or Assistance Support. ================================================ FILE: docs/scalar-kubernetes/AccessScalarProducts.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Make ScalarDB or ScalarDL deployed in a Kubernetes cluster environment available from applications This document explains how to make ScalarDB or ScalarDL deployed in a Kubernetes cluster environment available from applications. To make ScalarDB or ScalarDL available from applications, you can use Scalar Envoy via a Kubernetes service resource named `-envoy`. You can use `-envoy` in several ways, such as: * Directly from inside the same Kubernetes cluster as ScalarDB or ScalarDL. * Via a load balancer from outside the Kubernetes cluster. * From a bastion server by using the `kubectl port-forward` command (for testing purposes only). The resource name `-envoy` is decided based on the helm release name. You can see the helm release name by running the following command: ```console helm list -n ns-scalar ``` You should see the following output: ```console NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION scalardb ns-scalar 1 2023-02-09 19:31:40.527130674 +0900 JST deployed scalardb-2.5.0 3.8.0 scalardl-auditor ns-scalar 1 2023-02-09 19:32:03.008986045 +0900 JST deployed scalardl-audit-2.5.1 3.7.1 scalardl-ledger ns-scalar 1 2023-02-09 19:31:53.459548418 +0900 JST deployed scalardl-4.5.1 3.7.1 ``` You can also see the envoy service name `-envoy` by running the following command: ```console kubectl get service -n ns-scalar ``` You should see the following output: ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-envoy LoadBalancer 10.99.245.143 60051:31110/TCP 2m2s scalardb-envoy-metrics ClusterIP 10.104.56.87 9001/TCP 2m2s scalardb-headless ClusterIP None 60051/TCP 2m2s scalardb-metrics ClusterIP 10.111.213.194 8080/TCP 2m2s scalardl-auditor-envoy LoadBalancer 10.111.141.43 40051:31553/TCP,40052:31171/TCP 99s scalardl-auditor-envoy-metrics ClusterIP 10.104.245.188 9001/TCP 99s scalardl-auditor-headless ClusterIP None 40051/TCP,40053/TCP,40052/TCP 99s scalardl-auditor-metrics ClusterIP 10.105.119.158 8080/TCP 99s scalardl-ledger-envoy LoadBalancer 10.96.239.167 50051:32714/TCP,50052:30857/TCP 109s scalardl-ledger-envoy-metrics ClusterIP 10.97.204.18 9001/TCP 109s scalardl-ledger-headless ClusterIP None 50051/TCP,50053/TCP,50052/TCP 109s scalardl-ledger-metrics ClusterIP 10.104.216.189 8080/TCP 109s ``` ## Run application (client) requests to ScalarDB or ScalarDL via service resources directly from inside the same Kubernetes cluster If you deploy your application (client) in the same Kubernetes cluster as ScalarDB or ScalarDL (for example, if you deploy your application [client] on another node group or pool in the same Kubernetes cluster), the application can access ScalarDB or ScalarDL by using Kubernetes service resources. The format of the service resource name (FQDN) is `-envoy..svc.cluster.local`. The following are examples of ScalarDB and ScalarDL deployments on the `ns-scalar` namespace: * **ScalarDB Server** ```console scalardb-envoy.ns-scalar.svc.cluster.local ``` * **ScalarDL Ledger** ```console scalardl-ledger-envoy.ns-scalar.svc.cluster.local ``` * **ScalarDL Auditor** ```console scalardl-auditor-envoy.ns-scalar.svc.cluster.local ``` When using the Kubernetes service resource, you must set the above FQDN in the properties file for the application (client) as follows: * **Client properties file for ScalarDB Server** ```properties scalar.db.contact_points=-envoy..svc.cluster.local scalar.db.contact_port=60051 scalar.db.storage=grpc scalar.db.transaction_manager=grpc ``` * **Client properties file for ScalarDL Ledger** ```properties scalar.dl.client.server.host=-envoy..svc.cluster.local scalar.dl.ledger.server.port=50051 scalar.dl.ledger.server.privileged_port=50052 ``` * **Client properties file for ScalarDL Ledger with ScalarDL Auditor mode enabled** ```properties # Ledger scalar.dl.client.server.host=-envoy..svc.cluster.local scalar.dl.ledger.server.port=50051 scalar.dl.ledger.server.privileged_port=50052 # Auditor scalar.dl.client.auditor.enabled=true scalar.dl.client.auditor.host=-envoy..svc.cluster.local scalar.dl.auditor.server.port=40051 scalar.dl.auditor.server.privileged_port=40052 ``` ## Run application (client) requests to ScalarDB or ScalarDL via load balancers from outside the Kubernetes cluster If you deploy your application (client) in an environment outside the Kubernetes cluster for ScalarDB or ScalarDL (for example, if you deploy your application [client] on another Kubernetes cluster, container platform, or server), the application can access ScalarDB or ScalarDL by using a load balancer that each cloud service provides. You can create a load balancer by setting `envoy.service.type` to `LoadBalancer` in your custom values file. After configuring the custom values file, you can use Scalar Envoy through a Kubernetes service resource by using the load balancer. You can also set the load balancer configurations by using annotations. For more details on how to configure your custom values file, see [Service configurations](../helm-charts/configure-custom-values-envoy.mdx#service-configurations). When using a load balancer, you must set the FQDN or IP address of the load balancer in the properties file for the application (client) as follows. * **Client properties file for ScalarDB Server** ```properties scalar.db.contact_points= scalar.db.contact_port=60051 scalar.db.storage=grpc scalar.db.transaction_manager=grpc ``` * **Client properties file for ScalarDL Ledger** ```properties scalar.dl.client.server.host= scalar.dl.ledger.server.port=50051 scalar.dl.ledger.server.privileged_port=50052 ``` * **Client properties file for ScalarDL Ledger with ScalarDL Auditor mode enabled** ```properties # Ledger scalar.dl.client.server.host= scalar.dl.ledger.server.port=50051 scalar.dl.ledger.server.privileged_port=50052 # Auditor scalar.dl.client.auditor.enabled=true scalar.dl.client.auditor.host= scalar.dl.auditor.server.port=40051 scalar.dl.auditor.server.privileged_port=40052 ``` The concrete implementation of the load balancer and access method depend on the Kubernetes cluster. If you are using a managed Kubernetes cluster, see the following official documentation based on your cloud service provider: * **Amazon Elastic Kubernetes Service (EKS)** * [Network load balancing on Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html) * **Azure Kubernetes Service (AKS)** * [Use a public standard load balancer in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/load-balancer-standard) * [Use an internal load balancer with Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/internal-lb) ## Run client requests to ScalarDB or ScalarDL from a bastion server (for testing purposes only; not recommended in a production environment) You can run client requests to ScalarDB or ScalarDL from a bastion server by running the `kubectl port-forward` command. If you create a ScalarDL Auditor mode environment, however, you must run two `kubectl port-forward` commands with different kubeconfig files from one bastion server to access two Kubernetes clusters. 1. **(ScalarDL Auditor mode only)** In the bastion server for ScalarDL Ledger, configure an existing kubeconfig file or add a new kubeconfig file to access the Kubernetes cluster for ScalarDL Auditor. For details on how to configure the kubeconfig file of each managed Kubernetes cluster, see [Configure kubeconfig](CreateBastionServer.mdx#configure-kubeconfig). 2. Configure port forwarding to each service from the bastion server. * **ScalarDB Server** ```console kubectl port-forward -n svc/-envoy 60051:60051 ``` * **ScalarDL Ledger** ```console kubectl --context port-forward -n svc/-envoy 50051:50051 kubectl --context port-forward -n svc/-envoy 50052:50052 ``` * **ScalarDL Auditor** ```console kubectl --context port-forward -n svc/-envoy 40051:40051 kubectl --context port-forward -n svc/-envoy 40052:40052 ``` 3. Configure the properties file to access ScalarDB or ScalarDL via `localhost`. * **Client properties file for ScalarDB Server** ```properties scalar.db.contact_points=localhost scalar.db.contact_port=60051 scalar.db.storage=grpc scalar.db.transaction_manager=grpc ``` * **Client properties file for ScalarDL Ledger** ```properties scalar.dl.client.server.host=localhost scalar.dl.ledger.server.port=50051 scalar.dl.ledger.server.privileged_port=50052 ``` * **Client properties file for ScalarDL Ledger with ScalarDL Auditor mode enabled** ```properties # Ledger scalar.dl.client.server.host=localhost scalar.dl.ledger.server.port=50051 scalar.dl.ledger.server.privileged_port=50052 # Auditor scalar.dl.client.auditor.enabled=true scalar.dl.client.auditor.host=localhost scalar.dl.auditor.server.port=40051 scalar.dl.auditor.server.privileged_port=40052 ``` ================================================ FILE: docs/scalar-kubernetes/AwsMarketplaceGuide.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to install Scalar products through AWS Marketplace import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; Scalar products (ScalarDB, ScalarDL, and their tools) are available in the AWS Marketplace as container images. This guide explains how to install Scalar products through the AWS Marketplace. :::note - Some Scalar products are available under commercial licenses, and the AWS Marketplace provides those products as pay-as-you-go (PAYG) pricing. When you use pay-as-you-go pricing, AWS will charge you the Scalar product license fee based on your usage. - Previously, a bring-your-own-license (BYOL) option was offered in the AWS Marketplace. However, that option has been deprecated and removed, so it is no longer supported in the AWS Marketplace. - A BYOL option is provided in the following public container repositories outside of the AWS Marketplace. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact-us). - [ScalarDB Cluster Enterprise Standard](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-node-byol-standard) - [ScalarDB Cluster Enterprise Premium](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-node-byol-premium) - [ScalarDB Analytics Server](https://github.com/scalar-labs/scalardb-analytics/pkgs/container/scalardb-analytics-server-byol) - [ScalarDL Ledger](https://github.com/orgs/scalar-labs/packages/container/package/scalardl-ledger-byol) - [ScalarDL Auditor](https://github.com/orgs/scalar-labs/packages/container/package/scalardl-auditor-byol) ::: ## Subscribe to Scalar products from AWS Marketplace 1. Select your Scalar product to see the links to the AWS Marketplace. Select your edition of ScalarDB Enterprise. | PAYG | BYOL (Deprecated) | |:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| | [ScalarDB Cluster](https://aws.amazon.com/marketplace/pp/prodview-jx6qxatkxuwm4) | [ScalarDB Cluster](https://aws.amazon.com/marketplace/pp/prodview-alcwrmw6v4cfy) | | PAYG | BYOL (Deprecated) | |:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------| | [ScalarDB Cluster](https://aws.amazon.com/marketplace/pp/prodview-djqw3zk6dwyk6) | [ScalarDB Cluster](https://aws.amazon.com/marketplace/pp/prodview-alcwrmw6v4cfy) | | PAYG | |:-------------------------------------------------------------------------------------------| | [ScalarDB Analytics Server](https://aws.amazon.com/marketplace/pp/prodview-53ik57autkmci) | | PAYG | BYOL (Deprecated) | |:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| | [ScalarDL Ledger](https://aws.amazon.com/marketplace/pp/prodview-wttioaezp5j6e) | [ScalarDL Ledger](https://aws.amazon.com/marketplace/pp/prodview-3jdwfmqonx7a2) | | PAYG | BYOL (Deprecated) | |:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| | [ScalarDL Auditor](https://aws.amazon.com/marketplace/pp/prodview-ke3yiw4mhriuu) | [ScalarDL Auditor](https://aws.amazon.com/marketplace/pp/prodview-tj7svy75gu7m6) | 1. Select **Continue to Subscribe**. 1. Sign in to AWS Marketplace using your IAM user. If you have already signed in, this step will be skipped automatically. 1. Read the **Terms and Conditions** and select **Accept Terms**. It takes some time. When it's done, you can see the current date in the **Effective date** column. Also, you can see our products on the [Manage subscriptions](https://us-east-1.console.aws.amazon.com/marketplace/home#/subscriptions) page of AWS Console. ## **[Pay-As-You-Go]** Deploy containers on EKS (Amazon Elastic Kubernetes Service) from AWS Marketplace using Scalar Helm Charts By subscribing to Scalar products in the AWS Marketplace, you can pull the container images of Scalar products from the private container registry ([ECR](https://aws.amazon.com/ecr/)) of the AWS Marketplace. This section explains how to deploy Scalar products with pay-as-you-go pricing in your [EKS](https://aws.amazon.com/eks/) cluster from the private container registry. 1. Create an OIDC provider. You must create an identity and access management (IAM) OpenID Connect (OIDC) provider to run the AWS Marketplace Metering Service from ScalarDL pods. ```console eksctl utils associate-iam-oidc-provider --region --cluster --approve ``` For details, see [Creating an IAM OIDC provider for your cluster](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html). 1. Create a service account. To allow your pods to run the AWS Marketplace Metering Service, you can use [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). ```console eksctl create iamserviceaccount \ --name \ --namespace \ --region \ --cluster \ --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringFullAccess \ --approve \ --override-existing-serviceaccounts ``` 1. Update the custom values file of the Helm Chart for the Scalar product that you want to install. You need to specify the private container registry (ECR) of the AWS Marketplace as the value for `[].image.repository` in the custom values file. You also need to specify the service account name that you created in the previous step as the value for `[].serviceAccount.serviceAccountName` and set `[].serviceAccount.automountServiceAccountToken` to `true`. See the following examples based on the product you're using. Select your edition of ScalarDB Enterprise. In the `scalardb-cluster-standard-custom-values.yaml` file: ```yaml scalardbCluster: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-cluster-node-aws-payg-standard" serviceAccount: serviceAccountName: "" automountServiceAccountToken: true ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDB Cluster](../helm-charts/configure-custom-values-scalardb-cluster.mdx). ::: In the `scalardb-cluster-premium-custom-values.yaml` file: ```yaml scalardbCluster: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-cluster-node-aws-payg-premium" serviceAccount: serviceAccountName: "" automountServiceAccountToken: true ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDB Cluster](../helm-charts/configure-custom-values-scalardb-cluster.mdx). :::

ScalarDB Analytics server

In the `scalardb-analytics-server-custom-values.yaml` file: ```yaml scalarDbAnalyticsServer: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-analytics-server-aws-payg" serviceAccount: serviceAccountName: "" automountServiceAccountToken: true ``` :::note For more details on the configurations, see [Configure a Custom Values File for ScalarDB Analytics Server](../helm-charts/configure-custom-values-scalardb-analytics-server.mdx). :::

ScalarDL Ledger

In the `scalardl-ledger-custom-values.yaml` file: ```yaml ledger: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardl-ledger-aws-payg" serviceAccount: serviceAccountName: "" automountServiceAccountToken: true ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Ledger](../helm-charts/configure-custom-values-scalardl-ledger.mdx). :::

ScalarDL Schema Loader for Ledger

You don't need to update the `[].image.repository` configuration in your `schema-loader-ledger-custom-values.yaml` file. The container image of ScalarDL Schema Loader is provided in the [public container repository](https://github.com/orgs/scalar-labs/packages/container/package/scalardl-schema-loader). :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Schema Loader](../helm-charts/configure-custom-values-scalardl-schema-loader.mdx). :::

ScalarDL Auditor

In the `scalardl-auditor-custom-values.yaml` file: ```yaml auditor: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardl-auditor-aws-payg" serviceAccount: serviceAccountName: "" automountServiceAccountToken: true ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Auditor](../helm-charts/configure-custom-values-scalardl-auditor.mdx). :::

ScalarDL Schema Loader for Auditor

You don't need to update the `[].image.repository` configuration in your `schema-loader-auditor-custom-values.yaml` file. The container image of ScalarDL Schema Loader is provided in the [public container repository](https://github.com/orgs/scalar-labs/packages/container/package/scalardl-schema-loader). :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Schema Loader](../helm-charts/configure-custom-values-scalardl-schema-loader.mdx). ::: 1. Deploy Scalar products by using Helm Charts in conjunction with the above custom values files. See the following examples based on the product you're using. Select your edition of ScalarDB Enterprise. ```console helm install scalardb-cluster-standard scalar-labs/scalardb-cluster -f scalardb-cluster-standard-custom-values.yaml ``` ```console helm install scalardb-cluster-premium scalar-labs/scalardb-cluster -f scalardb-cluster-premium-custom-values.yaml ```

ScalarDB Analytics server

```console helm install scalardb-analytics-server scalar-labs/scalardb-analytics-server -f scalardb-analytics-server-custom-values.yaml ```

ScalarDL Ledger

```console helm install scalardl-ledger scalar-labs/scalardl -f scalardl-ledger-custom-values.yaml ```

ScalarDL Schema Loader for Ledger

```console helm install schema-loader scalar-labs/schema-loading -f schema-loader-ledger-custom-values.yaml ```

ScalarDL Auditor

```console helm install scalardl-auditor scalar-labs/scalardl-audit -f scalardl-auditor-custom-values.yaml ```

ScalarDL Schema Loader for Auditor

```console helm install schema-loader scalar-labs/schema-loading -f schema-loader-auditor-custom-values.yaml ``` ## **[Deprecated] [BYOL]** Deploy containers on EKS (Amazon Elastic Kubernetes Service) from AWS Marketplace using Scalar Helm Charts By subscribing to Scalar products in the AWS Marketplace, you can pull the container images of Scalar products from the private container registry ([ECR](https://aws.amazon.com/ecr/)) of the AWS Marketplace. This section explains how to deploy Scalar products with the BYOL option in your [EKS](https://aws.amazon.com/eks/) cluster from the private container registry. 1. Update the custom values file of the Helm Chart for the Scalar product that you want to install. You need to specify the private container registry (ECR) of AWS Marketplace as the value of `[].image.repository` in the custom values file. See the following examples based on the product you're using. ```yaml scalardbCluster: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-cluster-node-aws-byol" ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDB Cluster](../helm-charts/configure-custom-values-scalardb-cluster.mdx). :::

ScalarDL Ledger

In the `scalardl-ledger-custom-values.yaml` file: ```yaml ledger: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-ledger" ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Ledger](../helm-charts/configure-custom-values-scalardl-ledger.mdx). :::

ScalarDL Schema Loader for Ledger

You don't need to update the `[].image.repository` configuration in your `schema-loader-ledger-custom-values.yaml` file. The container image of ScalarDL Schema Loader is provided in the [public container repository](https://github.com/orgs/scalar-labs/packages/container/package/scalardl-schema-loader). :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Schema Loader](../helm-charts/configure-custom-values-scalardl-schema-loader.mdx). :::

ScalarDL Auditor

In the `scalardl-auditor-custom-values.yaml` file: ```yaml auditor: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-auditor" ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Auditor](../helm-charts/configure-custom-values-scalardl-auditor.mdx). :::

ScalarDL Schema Loader for Auditor

You don't need to update the `[].image.repository` configuration in your `schema-loader-auditor-custom-values.yaml` file. The container image of ScalarDL Schema Loader is provided in the [public container repository](https://github.com/orgs/scalar-labs/packages/container/package/scalardl-schema-loader). :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Schema Loader](../helm-charts/configure-custom-values-scalardl-schema-loader.mdx). ::: 1. Deploy the Scalar products using the Helm Chart with the above custom values files. See the following examples based on the product you're using. See the following examples based on the product you're using. ```console helm install scalardb-cluster scalar-labs/scalardb-cluster -f scalardb-cluster-custom-values.yaml ```

ScalarDL Ledger

```console helm install scalardl-ledger scalar-labs/scalardl -f scalardl-ledger-custom-values.yaml ```

ScalarDL Schema Loader for Ledger

```console helm install schema-loader scalar-labs/schema-loading -f schema-loader-ledger-custom-values.yaml ```

ScalarDL Auditor

```console helm install scalardl-auditor scalar-labs/scalardl-audit -f scalardl-auditor-custom-values.yaml ```

ScalarDL Schema Loader for Auditor

```console helm install schema-loader scalar-labs/schema-loading -f schema-loader-auditor-custom-values.yaml ``` ## **[Deprecated] [BYOL]** Deploy containers on Kubernetes other than EKS from AWS Marketplace using Scalar Helm Charts 1. Install the `aws` command according to the [AWS Official Document (Installing or updating the latest version of the AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). 1. Configure the AWS CLI with your credentials according to the [AWS Official Document (Configuration basics)](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html). 1. Create a `reg-ecr-mp-secrets` secret resource for pulling the container images from the ECR of AWS Marketplace. ```console kubectl create secret docker-registry reg-ecr-mp-secrets \ --docker-server=709825985650.dkr.ecr.us-east-1.amazonaws.com \ --docker-username=AWS \ --docker-password=$(aws ecr get-login-password --region us-east-1) ``` 1. Update the custom values file of the Helm Chart for the Scalar product that you want to install. You need to specify the private container registry (ECR) of AWS Marketplace as the value of `[].image.repository` in the custom values file. Also, you need to specify the `reg-ecr-mp-secrets` as the value of `[].imagePullSecrets`. See the following examples based on the product you're using. ```yaml scalardbCluster: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-cluster-node-aws-byol" imagePullSecrets: - name: "reg-ecr-mp-secrets" ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDB Cluster](../helm-charts/configure-custom-values-scalardb-cluster.mdx). :::

ScalarDL Ledger

In the `scalardl-ledger-custom-values.yaml` file: ```yaml ledger: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-ledger" imagePullSecrets: - name: "reg-ecr-mp-secrets" ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Ledger](../helm-charts/configure-custom-values-scalardl-ledger.mdx). :::

ScalarDL Schema Loader for Ledger

You don't need to update the `[].image.repository` configuration in your `schema-loader-ledger-custom-values.yaml` file. The container image of ScalarDL Schema Loader is provided in the [public container repository](https://github.com/orgs/scalar-labs/packages/container/package/scalardl-schema-loader). :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Schema Loader](../helm-charts/configure-custom-values-scalardl-schema-loader.mdx). :::

ScalarDL Auditor

In the `scalardl-auditor-custom-values.yaml` file: ```yaml auditor: image: repository: "709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalar-auditor" imagePullSecrets: - name: "reg-ecr-mp-secrets" ``` :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Auditor](../helm-charts/configure-custom-values-scalardl-auditor.mdx). :::

ScalarDL Schema Loader for Auditor

You don't need to update the `[].image.repository` configuration in your `schema-loader-auditor-custom-values.yaml` file. The container image of ScalarDL Schema Loader is provided in the [public container repository](https://github.com/orgs/scalar-labs/packages/container/package/scalardl-schema-loader). :::note For more details on the configurations, see [Configure a custom values file for ScalarDL Schema Loader](../helm-charts/configure-custom-values-scalardl-schema-loader.mdx). ::: 1. Deploy the Scalar products using the Helm Chart with the above custom values files. * Examples Please refer to the **[Deprecated] [BYOL] Deploy containers on EKS (Amazon Elastic Kubernetes Service) from AWS Marketplace using Scalar Helm Charts** section of this document. ================================================ FILE: docs/scalar-kubernetes/BackupNoSQL.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Back up a NoSQL database in a Kubernetes environment This guide explains how to create a transactionally consistent backup of managed databases that ScalarDB or ScalarDL uses in a Kubernetes environment. Please note that, when using a NoSQL database or multiple databases, you **must** pause ScalarDB or ScalarDL to create a transactionally consistent backup. For details on how ScalarDB backs up databases, see [A Guide on How to Backup and Restore Databases Used Through ScalarDB](https://scalardb.scalar-labs.com/docs/latest/backup-restore/). In this guide, we assume that you are using point-in-time recovery (PITR) or its equivalent features. Therefore, we must create a period where there are no ongoing transactions for restoration. You can then restore data to that specific period by using PITR. If you restore data to a time without creating a period where there are no ongoing transactions, the restored data could be transactionally inconsistent, causing ScalarDB or ScalarDL to not work properly with the data. ## Create a period to restore data, and perform a backup 1. Check the following four points by running the `kubectl get pod` command before starting the backup operation: * **The number of ScalarDB or ScalarDL pods.** Write down the number of pods so that you can compare that number with the number of pods after performing the backup. * **The ScalarDB or ScalarDL pod names in the `NAME` column.** Write down the pod names so that you can compare those names with the pod names after performing the backup. * **The ScalarDB or ScalarDL pod status is `Running` in the `STATUS` column.** Confirm that the pods are running before proceeding with the backup. You will need to pause the pods in the next step. * **The restart count of each pod in the `RESTARTS` column.** Write down the restart count of each pod so that you can compare the count with the restart counts after performing the backup. 2. Pause the ScalarDB or ScalarDL pods by using `scalar-admin`. For details on how to pause the pods, see the [Details on using `scalar-admin`](BackupNoSQL.mdx#details-on-using-scalar-admin) section in this guide. 3. Write down the `pause completed` time. You will need to refer to that time when restoring the data by using the PITR feature. 4. Back up each database by using the backup feature. If you have enabled the automatic backup and PITR features, the managed databases will perform back up automatically. Please note that you should wait for approximately 10 seconds so that you can create a sufficiently long period to avoid a clock skew issue between the client clock and the database clock. This 10-second period is the exact period in which you can restore data by using the PITR feature. 5. Unpause ScalarDB or ScalarDL pods by using `scalar-admin`. For details on how to unpause the pods, see the [Details on using `scalar-admin`](BackupNoSQL.mdx#details-on-using-scalar-admin) section in this guide. 6. Check the `unpause started` time. You must check the `unpause started` time to confirm the exact period in which you can restore data by using the PITR feature. 7. Check the pod status after performing the backup. You must check the following four points by using the `kubectl get pod` command after the backup operation is completed. * **The number of ScalarDB or ScalarDL pods.** Confirm this number matches the number of pods that you wrote down before performing the backup. * **The ScalarDB or ScalarDL pod names in the `NAME` column.** Confirm the names match the pod names that you wrote down before performing the backup. * **The ScalarDB or ScalarDL pod status is `Running` in the `STATUS` column.** * **The restart count of each pod in the `RESTARTS` column.** Confirm the counts match the restart counts that you wrote down before performing the backup **If any of the two values are different, you must retry the backup operation from the beginning.** The reason for the different values may be caused by some pods being added or restarted while performing the backup. In such case, those pods will run in the `unpause` state. Pods in the `unpause` state will cause the backup data to be transactionally inconsistent. 8. **(Amazon DynamoDB only)** If you use the PITR feature of DynamoDB, you will need to perform additional steps to create a backup because the feature restores data with another name table by using PITR. For details on the additional steps after creating the exact period in which you can restore the data, please see [Restore databases in a Kubernetes environment](RestoreDatabase.mdx#amazon-dynamodb). ## Back up multiple databases If you have two or more databases that the [Multi-storage Transactions](https://scalardb.scalar-labs.com/docs/latest/multi-storage-transactions/) or [Two-phase Commit Transactions](https://scalardb.scalar-labs.com/docs/latest/two-phase-commit-transactions/) feature uses, you must pause all instances of ScalarDB or ScalarDL and create the same period where no ongoing transactions exist in the databases. To ensure consistency between multiple databases, you must restore the databases to the same point in time by using the PITR feature. ## Details on using `scalar-admin` ### Check the Kubernetes resource name You must specify the SRV service URL to the `-s (--srv-service-url)` flag. In Kubernetes environments, the format of the SRV service URL is `_my-port-name._my-port-protocol.my-svc.my-namespace.svc.cluster.local`. If you use Scalar Helm Charts to deploy ScalarDB or ScalarDL, the `my-svc` and `my-namespace` may vary depending on your environment. You must specify the headless service name as `my-svc` and the namespace as `my-namespace`. * Example * ScalarDB Server ```console _scalardb._tcp.-headless..svc.cluster.local ``` * ScalarDL Ledger ```console _scalardl-admin._tcp.-headless..svc.cluster.local ``` * ScalarDL Auditor ```console _scalardl-auditor-admin._tcp.-headless..svc.cluster.local ``` The helm release name decides the headless service name `-headless`. You can see the helm release name by running the following command: ```console helm list -n ns-scalar ``` You should see the following output: ```console NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION scalardb ns-scalar 1 2023-02-09 19:31:40.527130674 +0900 JST deployed scalardb-2.5.0 3.8.0 scalardl-auditor ns-scalar 1 2023-02-09 19:32:03.008986045 +0900 JST deployed scalardl-audit-2.5.1 3.7.1 scalardl-ledger ns-scalar 1 2023-02-09 19:31:53.459548418 +0900 JST deployed scalardl-4.5.1 3.7.1 ``` You can also see the headless service name `-headless` by running the `kubectl get service` command. ```console kubectl get service -n ns-scalar ``` You should see the following output: ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-envoy LoadBalancer 10.99.245.143 60051:31110/TCP 2m2s scalardb-envoy-metrics ClusterIP 10.104.56.87 9001/TCP 2m2s scalardb-headless ClusterIP None 60051/TCP 2m2s scalardb-metrics ClusterIP 10.111.213.194 8080/TCP 2m2s scalardl-auditor-envoy LoadBalancer 10.111.141.43 40051:31553/TCP,40052:31171/TCP 99s scalardl-auditor-envoy-metrics ClusterIP 10.104.245.188 9001/TCP 99s scalardl-auditor-headless ClusterIP None 40051/TCP,40053/TCP,40052/TCP 99s scalardl-auditor-metrics ClusterIP 10.105.119.158 8080/TCP 99s scalardl-ledger-envoy LoadBalancer 10.96.239.167 50051:32714/TCP,50052:30857/TCP 109s scalardl-ledger-envoy-metrics ClusterIP 10.97.204.18 9001/TCP 109s scalardl-ledger-headless ClusterIP None 50051/TCP,50053/TCP,50052/TCP 109s scalardl-ledger-metrics ClusterIP 10.104.216.189 8080/TCP 109s ``` ### Pause You can send a pause request to ScalarDB or ScalarDL pods in a Kubernetes environment. * Example * ScalarDB Server ```console kubectl run scalar-admin-pause --image=ghcr.io/scalar-labs/scalar-admin: --restart=Never -it -- -c pause -s _scalardb._tcp.-headless..svc.cluster.local ``` * ScalarDL Ledger ```console kubectl run scalar-admin-pause --image=ghcr.io/scalar-labs/scalar-admin: --restart=Never -it -- -c pause -s _scalardl-admin._tcp.-headless..svc.cluster.local ``` * ScalarDL Auditor ```console kubectl run scalar-admin-pause --image=ghcr.io/scalar-labs/scalar-admin: --restart=Never -it -- -c pause -s _scalardl-auditor-admin._tcp.-headless..svc.cluster.local ``` ### Unpause You can send an unpause request to ScalarDB or ScalarDL pods in a Kubernetes environment. * Example * ScalarDB Server ```console kubectl run scalar-admin-unpause --image=ghcr.io/scalar-labs/scalar-admin: --restart=Never -it -- -c unpause -s _scalardb._tcp.-headless..svc.cluster.local ``` * ScalarDL Ledger ```console kubectl run scalar-admin-unpause --image=ghcr.io/scalar-labs/scalar-admin: --restart=Never -it -- -c unpause -s _scalardl-admin._tcp.-headless..svc.cluster.local ``` * ScalarDL Auditor ```console kubectl run scalar-admin-unpause --image=ghcr.io/scalar-labs/scalar-admin: --restart=Never -it -- -c unpause -s _scalardl-auditor-admin._tcp.-headless..svc.cluster.local ``` ### Check the `pause completed` time and `unpause started` time The `scalar-admin` pods output the `pause completed` time and `unpause started` time to stdout. You can also see those times by running the `kubectl logs` command. ```console kubectl logs scalar-admin-pause ``` ```console kubectl logs scalar-admin-unpause ``` ================================================ FILE: docs/scalar-kubernetes/BackupRDB.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Back up an RDB in a Kubernetes environment This guide explains how to create a backup of a single relational database (RDB) that ScalarDB or ScalarDL uses in a Kubernetes environment. Please note that this guide assumes that you are using a managed database from a cloud services provider. If you have two or more RDBs that the [Multi-storage Transactions](https://scalardb.scalar-labs.com/docs/latest/multi-storage-transactions/) or [Two-phase Commit Transactions](https://scalardb.scalar-labs.com/docs/latest/two-phase-commit-transactions/) feature uses, you must follow the instructions in [Back up a NoSQL database in a Kubernetes environment](BackupNoSQL.mdx) instead. ## Perform a backup To perform backups, you should enable the automated backup feature available in the managed databases. By enabling this feature, you do not need to perform any additional backup operations. For details on the backup configurations in each managed database, see the following guides: * [Set up a database for ScalarDB/ScalarDL deployment on AWS](SetupDatabaseForAWS.mdx) * [Set up a database for ScalarDB/ScalarDL deployment on Azure](SetupDatabaseForAzure.mdx) Because the managed RDB keeps backup data consistent from a transactions perspective, you can restore backup data to any point in time by using the point-in-time recovery (PITR) feature in the managed RDB. ================================================ FILE: docs/scalar-kubernetes/BackupRestoreGuide.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Back up and restore ScalarDB or ScalarDL data in a Kubernetes environment This guide explains how to backup and restore ScalarDB or ScalarDL data in a Kubernetes environment. Please note that this guide assumes that you are using a managed database from a cloud services provider as the backend database for ScalarDB or ScalarDL. The following is a list of the managed databases that this guide assumes you might be using: * NoSQL: does not support transactions * Amazon DynamoDB * Azure Cosmos DB for NoSQL * Relational database (RDB): supports transactions * Amazon RDS * MySQL * Oracle * PostgreSQL * SQL Server * Amazon Aurora * MySQL * PostgreSQL * Azure Database * MySQL * PostgreSQL For details on how to back up and restore databases used with ScalarDB in a transactionally consistent way, see [A Guide on How to Backup and Restore Databases Used Through ScalarDB](https://scalardb.scalar-labs.com/docs/latest/backup-restore/). ## Perform a backup ### Confirm the type of database and number of databases you are using How you perform backup and restore depends on the type of database (NoSQL or RDB) and the number of databases you are using. #### NoSQL or multiple databases If you are using a NoSQL database, or if you have two or more databases that the [Multi-storage Transactions](https://scalardb.scalar-labs.com/docs/latest/multi-storage-transactions/) or [Two-phase Commit Transactions](https://scalardb.scalar-labs.com/docs/latest/two-phase-commit-transactions/) feature uses, please see [Back up a NoSQL database in a Kubernetes environment](BackupNoSQL.mdx) for details on how to perform a backup. #### Single RDB If you are using a single RDB, please see [Back up an RDB in a Kubernetes environment](BackupRDB.mdx) for details on how to perform a backup. If you have two or more RDBs that the [Multi-storage Transactions](https://scalardb.scalar-labs.com/docs/latest/multi-storage-transactions/) or [Two-phase Commit Transactions](https://scalardb.scalar-labs.com/docs/latest/two-phase-commit-transactions/) feature uses, you must follow the instructions in [Back up a NoSQL database in a Kubernetes environment](BackupNoSQL.mdx) instead. ## Restore a database For details on how to restore data from a managed database, please see [Restore databases in a Kubernetes environment](RestoreDatabase.mdx). ================================================ FILE: docs/scalar-kubernetes/CreateAKSClusterForScalarDB.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Deprecated displayed_sidebar: docsEnglish --- # Guidelines for creating an AKS cluster for ScalarDB Server This document explains the requirements and recommendations for creating an Azure Kubernetes Service (AKS) cluster for ScalarDB Server deployment. For details on how to deploy ScalarDB Server on an AKS cluster, see [Deploy ScalarDB Server on AKS](ManualDeploymentGuideScalarDBServerOnAKS.mdx). ## Before you begin You must create an AKS cluster based on the following requirements, recommendations, and your project's requirements. For specific details about how to create an AKS cluster, refer to the following official Microsoft documentation based on the tool you use in your environment: * [Azure CLI](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-cli) * [PowerShell](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-powershell) * [Azure portal](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-portal) ## Requirements When deploying ScalarDB Server, you must: * Create the AKS cluster by using a [supported Kubernetes version](https://scalardb.scalar-labs.com/docs/latest/requirements/#kubernetes). * Configure the AKS cluster based on the version of Kubernetes and your project's requirements. ## Recommendations (optional) The following are some recommendations for deploying ScalarDB Server. These recommendations are not required, so you can choose whether or not to apply these recommendations based on your needs. ### Create at least three worker nodes and three pods To ensure that the AKS cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardb-custom-values.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different [availability zones](https://learn.microsoft.com/en-us/azure/availability-zones/az-overview) (AZs), you can withstand an AZ failure. ::: ### Use 4vCPU / 8GB memory nodes for the worker node in the ScalarDB Server node pool It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition to the ScalarDB Server pod, Kubernetes could deploy some of the following components to each worker node: * ScalarDB Server pod (2vCPU / 4GB) * Envoy proxy * Your application pods (if you choose to run your application's pods on the same worker node) * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [Create at least three worker nodes and three pods](#create-at-least-three-worker-nodes-and-three-pods). However, three nodes with at least 4vCPU / 8GB memory resources per node is the minimum for production environment. You should also consider the resources of the AKS cluster (for example, the number of worker nodes, vCPUs per node, memory per node, ScalarDB Server pods, and pods for your application), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node when deciding the worker node resources. ### Create a node pool for ScalarDB Server pods AKS creates one system node pool named **agentpool** that is preferred for system pods (used to keep AKS running) by default. We recommend creating another node pool with **user** mode for ScalarDB Server pods and deploying ScalarDB Server pods on this additional node pool. ### Configure cluster autoscaler in AKS If you want to scale ScalarDB Server pods automatically by using [Horizontal Pod Autoscaler](https://learn.microsoft.com/en-us/azure/aks/concepts-scale#horizontal-pod-autoscaler), you should configure cluster autoscaler in AKS too. For details, refer to the official Microsoft documentation at [Cluster autoscaler](https://learn.microsoft.com/en-us/azure/aks/concepts-scale#cluster-autoscaler). In addition, if you configure cluster autoscaler, you should create a subnet in a virtual network (VNet) for AKS to ensure a sufficient number of IPs exist so that AKS can work without network issues after scaling. The required number of IPs varies depending on the networking plug-in. For more details about the number of IPs required, refer to the following: * [Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-kubenet) * [Configure Azure CNI networking in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni) ### Create the AKS cluster on a private network You should create the AKS cluster on a private network (private subnet in a VNet) since ScalarDB Server does not provide any services to users directly via internet access. We recommend accessing ScalarDB Server via a private network from your applications. ### Create the AKS cluster by using Azure CNI, if necessary The AKS default networking plug-in is [kubenet](https://learn.microsoft.com/en-us/azure/aks/configure-kubenet). If your requirement does not match kubenet, you should use [Azure Container Networking Interface (CNI)](https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni). For example, if you want to deploy multiple ScalarDB Server environments on one AKS cluster (e.g., deploy a multi-tenant ScalarDB Server) and you want to control the connection between each tenant by using [Kubernetes NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/), kubenet supports only the Calico Network Policy, which the [Azure support team does not support](https://learn.microsoft.com/en-us/azure/aks/use-network-policies#differences-between-azure-network-policy-manager-and-calico-network-policy-and-their-capabilities). Please note that the Calico Network Policy is supported only by the Calico community or through additional paid support. The Azure support and engineering teams, however, do support Azure CNI. So, if you want to use Kubernetes NetworkPolicies and receive support from the Azure support team, you should use Azure CNI. For more details about the differences between kubenet and Azure CNI, refer to the following official Microsoft documentation: * [Network concepts for applications in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/concepts-network) * [Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-kubenet) * [Configure Azure CNI networking in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni) ### Restrict connections by using some security features based on your requirements You should restrict unused connections in ScalarDB Server. To restrict unused connections, you can use some security features in Azure, like [network security groups](https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview). The connections (ports) that ScalarDB Server uses by default are as follows: * ScalarDB Server * 60051/TCP (accepts requests from a client) * 8080/TCP (accepts monitoring requests) * Scalar Envoy (used with ScalarDB Server) * 60051/TCP (load balancing for ScalarDB Server) * 9001/TCP (accepts monitoring requests for Scalar Envoy itself) :::note - If you change the default listening port for ScalarDB Server in the configuration file (`database.properties`), you must allow connections by using the port that you configured. - You must also allow the connections that AKS uses itself. For more details about AKS traffic requirements, refer to [Control egress traffic using Azure Firewall in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic). ::: ================================================ FILE: docs/scalar-kubernetes/CreateAKSClusterForScalarDL.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Guidelines for creating an AKS cluster for ScalarDL Ledger This document explains the requirements and recommendations for creating an Azure Kubernetes Service (AKS) cluster for ScalarDL Ledger deployment. For details on how to deploy ScalarDL Ledger on an AKS cluster, see [Deploy ScalarDL Ledger on AKS](ManualDeploymentGuideScalarDLOnAKS.mdx). ## Before you begin You must create an AKS cluster based on the following requirements, recommendations, and your project's requirements. For specific details about how to create an AKS cluster, refer to the following official Microsoft documentation based on the tool you use in your environment: * [Azure CLI](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-cli) * [PowerShell](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-powershell) * [Azure portal](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-portal) ## Requirements When deploying ScalarDL Ledger, you must: * Create the AKS cluster by using a [supported Kubernetes version](https://scalardb.scalar-labs.com/docs/latest/requirements/#kubernetes). * Configure the AKS cluster based on the version of Kubernetes and your project's requirements. :::warning For Byzantine fault detection in ScalarDL to work properly, do not deploy your application pods on the same AKS cluster as the ScalarDL Ledger deployment. ::: ## Recommendations (optional) The following are some recommendations for deploying ScalarDL Ledger. These recommendations are not required, so you can choose whether or not to apply these recommendations based on your needs. ### Create at least three worker nodes and three pods To ensure that the AKS cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardl-custom-values.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different [availability zones](https://learn.microsoft.com/en-us/azure/availability-zones/az-overview) (AZs), you can withstand an AZ failure. ::: ### Use 4vCPU / 8GB memory nodes for the worker node in the ScalarDL Ledger node pool It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition to the ScalarDL Ledger pod, Kubernetes could deploy some of the following components to each worker node: * ScalarDL Ledger pod (2vCPU / 4GB) * Envoy proxy * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [Create at least three worker nodes and three pods](#create-at-least-three-worker-nodes-and-three-pods). However, three nodes with at least 4vCPU / 8GB memory resources per node is the minimum environment for production. You should also consider the resources of the AKS cluster (for example, the number of worker nodes, vCPUs per node, memory per node, and ScalarDL Ledger pods), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node when deciding the worker node resources. ### Create a node pool for ScalarDL Ledger pods AKS creates one system node pool named **agentpool** that is preferred for system pods (used to keep AKS running) by default. We recommend creating another node pool with **user** mode for ScalarDL Ledger pods and deploying ScalarDL Ledger pods on this additional node pool. ### Configure cluster autoscaler in AKS If you want to scale ScalarDL Ledger pods automatically by using [Horizontal Pod Autoscaler](https://learn.microsoft.com/en-us/azure/aks/concepts-scale#horizontal-pod-autoscaler), you should configure cluster autoscaler in AKS too. For details, refer to the official Microsoft documentation at [Cluster autoscaler](https://learn.microsoft.com/en-us/azure/aks/concepts-scale#cluster-autoscaler). In addition, if you configure cluster autoscaler, you should create a subnet in a virtual network (VNet) for AKS to ensure a sufficient number of IPs exist so that AKS can work without network issues after scaling. The required number of IPs varies depending on the networking plug-in. For more details about the number of IPs required, refer to the following: * [Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-kubenet) * [Configure Azure CNI networking in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni) ### Create the AKS cluster on a private network You should create the AKS cluster on a private network (private subnet in a VNet) since ScalarDL Ledger does not provide any services to users directly via internet access. We recommend accessing ScalarDL Ledger via a private network from your applications. ### Create the AKS cluster by using Azure CNI, if necessary The AKS default networking plug-in is [kubenet](https://learn.microsoft.com/en-us/azure/aks/configure-kubenet). If your requirement does not match kubenet, you should use [Azure Container Networking Interface (CNI)](https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni). For example, if you want to deploy multiple ScalarDL Ledger environments on one AKS cluster (e.g., deploy multi-tenant ScalarDL Ledger) and you want to control the connection between each tenant by using [Kubernetes NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/), kubenet supports only the Calico Network Policy, which the [Azure support team does not support](https://learn.microsoft.com/en-us/azure/aks/use-network-policies#differences-between-azure-network-policy-manager-and-calico-network-policy-and-their-capabilities). Please note that the Calico Network Policy is supported only by the Calico community or through additional paid support. The Azure support and engineering teams, however, do support Azure CNI. So, if you want to use Kubernetes NetworkPolicies and receive support from the Azure support team, you should use Azure CNI. For more details about the differences between kubenet and Azure CNI, refer to the following official Microsoft documentation: * [Network concepts for applications in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/concepts-network) * [Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-kubenet) * [Configure Azure CNI networking in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni) ### Restrict connections by using some security features based on your requirements You should restrict unused connections in ScalarDL Ledger. To restrict unused connections, you can use some security features in Azure, like [network security groups](https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview). The connections (ports) that ScalarDL Ledger uses by default are as follows: * ScalarDL Ledger * 50051/TCP (accepts requests from a client) * 50052/TCP (accepts privileged requests from a client) * 50053/TCP (accepts pause and unpause requests from a scalar-admin client tool) * 8080/TCP (accepts monitoring requests) * Scalar Envoy (used with ScalarDL Ledger) * 50051/TCP (load balancing for ScalarDL Ledger) * 50052/TCP (load balancing for ScalarDL Ledger) * 9001/TCP (accepts monitoring requests for Scalar Envoy itself) :::note - If you change the default listening port for ScalarDL Ledger in the configuration file (`ledger.properties`), you must allow connections by using the port that you configured. - You must also allow the connections that AKS uses itself. For more details about AKS traffic requirements, refer to [Control egress traffic using Azure Firewall in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic). ::: ================================================ FILE: docs/scalar-kubernetes/CreateAKSClusterForScalarDLAuditor.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Guidelines for creating an AKS cluster for ScalarDL Ledger and ScalarDL Auditor This document explains the requirements and recommendations for creating an Azure Kubernetes Service (AKS) cluster for ScalarDL Ledger and ScalarDL Auditor deployment. For details on how to deploy ScalarDL Ledger and ScalarDL Auditor on an AKS cluster, see [Deploy ScalarDL Ledger and ScalarDL Auditor on AKS](ManualDeploymentGuideScalarDLAuditorOnAKS.mdx). ## Before you begin You must create an AKS cluster based on the following requirements, recommendations, and your project's requirements. For specific details about how to create an AKS cluster, refer to the following official Microsoft documentation based on the tool you use in your environment: * [Azure CLI](https://learn.microsoft.com/ja-jp/azure/aks/learn/quick-kubernetes-deploy-cli) * [PowerShell](https://learn.microsoft.com/ja-jp/azure/aks/learn/quick-kubernetes-deploy-powershell) * [Azure portal](https://learn.microsoft.com/ja-jp/azure/aks/learn/quick-kubernetes-deploy-portal) ## Requirements When deploying ScalarDL Ledger and ScalarDL Auditor, you must: * Create two AKS clusters by using a [supported Kubernetes version](https://scalardb.scalar-labs.com/docs/latest/requirements/#kubernetes). * One AKS cluster for ScalarDL Ledger * One AKS cluster for ScalarDL Auditor * Configure the AKS clusters based on the version of Kubernetes and your project's requirements. * Configure a virtual network (VNet) as follows. * Connect the **VNet of AKS (for Ledger)** and the **VNet of AKS (for Auditor)** by using [virtual network peering](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-manage-peering). To do so, you must specify the different IP ranges for the **VNet of AKS (for Ledger)** and the **VNet of AKS (for Auditor)** when you create those VNets. * Allow **connections between Ledger and Auditor** to make ScalarDL (Auditor mode) work properly. * For more details about these network requirements, refer to [Configure Network Peering for ScalarDL Auditor Mode](NetworkPeeringForScalarDLAuditor.mdx). :::warning For Byzantine fault detection in ScalarDL to work properly, do not deploy your application pods on the same AKS clusters as the ScalarDL Ledger and ScalarDL Auditor deployments. ::: ## Recommendations (optional) The following are some recommendations for deploying ScalarDL Ledger and ScalarDL Auditor. These recommendations are not required, so you can choose whether or not to apply these recommendations based on your needs. ### Create at least three worker nodes and three pods per AKS cluster To ensure that the AKS cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [ScalarDL Ledger sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardl-custom-values.yaml) and [ScalarDL Auditor sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardl-audit-custom-values.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different [availability zones](https://learn.microsoft.com/en-us/azure/availability-zones/az-overview) (AZs), you can withstand an AZ failure. ::: ### Use 4vCPU / 8GB memory nodes for the worker node in the ScalarDL Ledger and ScalarDL Auditor node pool It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition to the ScalarDL Ledger and ScalarDL Auditor pods, Kubernetes could deploy some of the following components to each worker node: * AKS cluster for ScalarDL Ledger * ScalarDL Ledger pod (2vCPU / 4GB) * Envoy proxy * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components * AKS cluster for ScalarDL Auditor * ScalarDL Auditor pod (2vCPU / 4GB) * Envoy proxy * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [Create at least three worker nodes and three pods](#create-at-least-three-worker-nodes-and-three-pods-per-aks-cluster). And remember, for Byzantine fault detection to work properly, you cannot deploy your application pods on the same AKS clusters as the ScalarDL Ledger and ScalarDL Auditor deployments. However, three nodes with at least 4vCPU / 8GB memory resources per node is the minimum environment for production. You should also consider the resources of the AKS cluster (for example, the number of worker nodes, vCPUs per node, memory per node, ScalarDL Ledger pods, and ScalarDL Auditor pods), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node when deciding the worker node resources. ### Create node pools for ScalarDL Ledger and ScalarDL Auditor pods AKS creates one system node pool named **agentpool** that is preferred for system pods (used to keep AKS running) by default. We recommend creating additional node pools with **user** mode for ScalarDL Ledger and ScalarDL Auditor pods and deploying ScalarDL Ledger and ScalarDL Auditor pods on those additional node pools. ### Configure cluster autoscaler in AKS If you want to scale ScalarDL Ledger and ScalarDL Auditor pods automatically by using [Horizontal Pod Autoscaler)](https://learn.microsoft.com/en-us/azure/aks/concepts-scale#horizontal-pod-autoscaler), you should configure cluster autoscaler in AKS too. For details, refer to the official Microsoft documentation at [Cluster autoscaler](https://learn.microsoft.com/en-us/azure/aks/concepts-scale#cluster-autoscaler). In addition, if you configure cluster autoscaler, you should create a subnet in a VNet for AKS to ensure a sufficient number of IPs exist so that AKS can work without network issues after scaling. The required number of IPs varies depending on the networking plug-in. For more details about the number of IPs required, refer to the following: * [Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-kubenet) * [Configure Azure CNI networking in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni) ### Create the AKS cluster on a private network You should create the AKS cluster on a private network (private subnet in a VNet) since ScalarDL Ledger and ScalarDL Auditor do not provide any services to users directly via internet access. We recommend accessing ScalarDL Ledger and ScalarDL Auditor via a private network from your applications. ### Create the AKS cluster by using Azure CNI, if necessary The AKS default networking plug-in is [kubenet](https://learn.microsoft.com/en-us/azure/aks/configure-kubenet). If your requirement does not match kubenet, you should use [Azure Container Networking Interface (CNI)](https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni). For example, if you want to deploy multiple ScalarDL Ledger and ScalarDL Auditor environments on only one AKS cluster instead of two AKS clusters (e.g., deploy multi-tenant ScalarDL) and control the connection between each tenant by using [Kubernetes NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/), kubenet supports only the Calico Network Policy, which the [Azure support team does not support](https://learn.microsoft.com/en-us/azure/aks/use-network-policies#differences-between-azure-network-policy-manager-and-calico-network-policy-and-their-capabilities). Please note that the Calico Network Policy is supported only by the Calico community or through additional paid support. The Azure support and engineering teams, however, do support Azure CNI. So, if you want to use Kubernetes NetworkPolicies and receive support from the Azure support team, you should use Azure CNI. For more details about the differences between kubenet and Azure CNI, refer to the following official Microsoft documentation: * [Network concepts for applications in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/concepts-network) * [Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-kubenet) * [Configure Azure CNI networking in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni) ### Restrict connections by using some security features based on your requirements You should restrict unused connections in ScalarDL and ScalarDL Auditor. To restrict unused connections, you can use some security features of Azure, like [network security groups](https://learn.microsoft.com/en-us/azure/virtual-network/network-security-groups-overview). The connections (ports) that ScalarDL Ledger and ScalarDL Auditor use by default are as follows: * ScalarDL Ledger * 50051/TCP (accepts requests from a client and ScalarDL Auditor) * 50052/TCP (accepts privileged requests from a client and ScalarDL Auditor) * 50053/TCP (accepts pause/unpause requests from a scalar-admin client tool) * 8080/TCP (accepts monitoring requests) * ScalarDL Auditor * 40051/TCP (accepts requests from a client) * 40052/TCP (accepts privileged requests from a client) * 40053/TCP (accepts pause and unpause requests from a scalar-admin client tool) * 8080/TCP (accepts monitoring requests) * Scalar Envoy (used with ScalarDL Ledger and ScalarDL Auditor) * 50051/TCP (load balancing for ScalarDL Ledger) * 50052/TCP (load balancing for ScalarDL Ledger) * 40051/TCP (load balancing for ScalarDL Auditor) * 40052/TCP (load balancing for ScalarDL Auditor) * 9001/TCP (accepts monitoring requests for Scalar Envoy itself) :::note - If you change the default listening port for ScalarDL Ledger and ScalarDL Auditor in their configuration files (`ledger.properties` and `auditor.properties`, respectively), you must allow connections by using the port that you configured. - You must also allow the connections that AKS uses itself. For more details about AKS traffic requirements, refer to [Control egress traffic using Azure Firewall in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/limit-egress-traffic). ::: ================================================ FILE: docs/scalar-kubernetes/CreateAKSClusterForScalarProducts.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Guidelines for creating an AKS cluster for Scalar products To create an Azure Kubernetes Service (AKS) cluster for Scalar products, refer to the following: * [Guidelines for creating an AKS cluster for ScalarDB Server](CreateAKSClusterForScalarDB.mdx) * [Guidelines for creating an AKS cluster for ScalarDL Ledger](CreateAKSClusterForScalarDL.mdx) * [Guidelines for creating an AKS cluster for ScalarDL Ledger and ScalarDL Auditor](CreateAKSClusterForScalarDLAuditor.mdx) To deploy Scalar products on AKS, refer to the following: * [Deploy ScalarDB Server on AKS](ManualDeploymentGuideScalarDBServerOnAKS.mdx) * [Deploy ScalarDL Ledger on AKS](ManualDeploymentGuideScalarDLOnAKS.mdx) * [Deploy ScalarDL Ledger and ScalarDL Auditor on AKS](ManualDeploymentGuideScalarDLAuditorOnAKS.mdx) ================================================ FILE: docs/scalar-kubernetes/CreateBastionServer.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Create a bastion server This document explains how to create a bastion server and install some tools for the deployment of Scalar products. ## Create a server on the same private network as a Kubernetes cluster It is recommended to create a Kubernetes cluster for Scalar products on a private network. If you create a Kubernetes cluster on a private network, you should create a bastion server on the same private network to access your Kubernetes cluster. ## Install tools Please install the following tools on the bastion server according to their official documents. * [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) * [helm](https://helm.sh/docs/intro/install/) ## Configure kubeconfig After you install the kubectl command, you must configure a **kubeconfig** to access your Kubernetes cluster. Please refer to the following official document for more details on how to configure kubeconfig in each managed Kubernetes. If you use Amazon EKS (Amazon Elastic Kubernetes Service), you must install the **AWS CLI** according to the official document [Installing or updating the latest version of the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). After that, you can see how to configure kubeconfig in [Creating or updating a kubeconfig file for an Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html). If you use AKS (Azure Kubernetes Service), you must install the **Azure CLI** according to the official document [How to install the Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). After that, you can see how to configure kubeconfig in [az aks get-credentials](https://learn.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-get-credentials). ## Check installation You can check if the tools are installed as follows. * kubectl ```console kubectl version --client ``` * helm ```console helm version ``` You can also check if your kubeconfig is properly configured as follows. If you see a URL response, kubectl is correctly configured to access your cluster. ```console kubectl cluster-info ``` ================================================ FILE: docs/scalar-kubernetes/CreateEKSClusterForScalarDB.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Deprecated displayed_sidebar: docsEnglish --- # (Deprecated) Guidelines for creating an EKS cluster for ScalarDB Server :::warning ScalarDB Server is now deprecated. Please use [ScalarDB Cluster](ManualDeploymentGuideScalarDBClusterOnEKS.mdx) instead. ::: This document explains the requirements and recommendations for creating an Amazon Elastic Kubernetes Service (EKS) cluster for ScalarDB Server deployment. For details on how to deploy ScalarDB Server on an EKS cluster, see [Deploy ScalarDB Server on Amazon EKS](ManualDeploymentGuideScalarDBServerOnEKS.mdx). ## Before you begin You must create an EKS cluster based on the following requirements, recommendations, and your project's requirements. For specific details about how to create an EKS cluster, see the official Amazon documentation at [Creating an Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html). ## Requirements When deploying ScalarDB Server, you must: * Create the EKS cluster by using a [supported Kubernetes version](https://scalardb.scalar-labs.com/docs/latest/requirements/#kubernetes). * Configure the EKS cluster based on the version of Kubernetes and your project's requirements. ## Recommendations (optional) The following are some recommendations for deploying ScalarDB Server. These recommendations are not required, so you can choose whether or not to apply these recommendations based on your needs. ### Create at least three worker nodes and three pods To ensure that the EKS cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardb-custom-values.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different [availability zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) (AZs), you can withstand an AZ failure. ::: ### Use 4vCPU / 8GB memory nodes for the worker node in the ScalarDB Server node group It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition to the ScalarDB Server pod, Kubernetes could deploy some of the following components to each worker node: * ScalarDB Server pod (2vCPU / 4GB) * Envoy proxy * Your application pods (if you choose to run your application's pods on the same worker node) * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [Create at least three worker nodes and three pods](#create-at-least-three-worker-nodes-and-three-pods). However, three nodes with at least 4vCPU / 8GB memory resources per node is the minimum for production environment. You should also consider the resources of the EKS cluster (for example, the number of worker nodes, vCPUs per node, memory per node, ScalarDB Server pods, and pods for your application), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node when deciding the worker node resources. ### Configure Cluster Autoscaler in EKS If you want to scale ScalarDB Server pods automatically by using [Horizontal Pod Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/horizontal-pod-autoscaler.html), you should configure Cluster Autoscaler in EKS too. For details, see the official Amazon documentation at [Autoscaling](https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html#cluster-autoscaler). In addition, if you configure Cluster Autoscaler, you should create a subnet in an Amazon Virtual Private Cloud (VPC) for EKS with the prefix (e.g., `/24`) to ensure a sufficient number of IPs exist so that EKS can work without network issues after scaling. ### Create the EKS cluster on a private network You should create the EKS cluster on a private network (private subnet in a VPC) since ScalarDB Server does not provide any services to users directly via internet access. We recommend accessing ScalarDB Server via a private network from your applications. ### Restrict connections by using some security features based on your requirements You should restrict unused connections in ScalarDB Server. To restrict unused connections, you can use some security features in AWS, like [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) and [network access control lists](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html). The connections (ports) that ScalarDB Server uses by default are as follows: * ScalarDB Server * 60051/TCP (accepts requests from a client) * 8080/TCP (accepts monitoring requests) * Scalar Envoy (used with ScalarDB Server) * 60051/TCP (load balancing for ScalarDB Server) * 9001/TCP (accepts monitoring requests for Scalar Envoy itself) :::note - If you change the default listening port for ScalarDB Server in the configuration file (`database.properties`), you must allow connections by using the port that you configured. - You must also allow the connections that EKS uses itself. For more details about Amazon EKS security group requirements, refer to [Amazon EKS security group requirements and considerations](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html). ::: ================================================ FILE: docs/scalar-kubernetes/CreateEKSClusterForScalarDBCluster.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Guidelines for creating an EKS cluster for ScalarDB Cluster This document explains the requirements and recommendations for creating an Amazon Elastic Kubernetes Service (EKS) cluster for ScalarDB Cluster deployment. For details on how to deploy ScalarDB Cluster on an EKS cluster, see [Deploy ScalarDB Cluster on Amazon EKS](ManualDeploymentGuideScalarDBClusterOnEKS.mdx). ## Before you begin You must create an EKS cluster based on the following requirements, recommendations, and your project's requirements. For specific details about how to create an EKS cluster, see the official Amazon documentation at [Creating an Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html). ## Requirements When deploying ScalarDB Cluster, you must: * Create the EKS cluster by using a [supported Kubernetes version](https://scalardb.scalar-labs.com/docs/latest/requirements/#kubernetes). * Configure the EKS cluster based on the version of Kubernetes and your project's requirements. ## Recommendations (optional) The following are some recommendations for deploying ScalarDB Cluster. These recommendations are not required, so you can choose whether or not to apply these recommendations based on your needs. ### Create at least three worker nodes and three pods To ensure that the EKS cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardb-cluster-custom-values-indirect-mode.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different [availability zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) (AZs), you can withstand an AZ failure. ::: ### Use 4vCPU / 8GB memory nodes for the worker node in the ScalarDB Cluster node group It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition to the ScalarDB Cluster pod, Kubernetes could deploy some of the following components to each worker node: * ScalarDB Cluster pod (2vCPU / 4GB) * Envoy proxy (if you use `indirect` client mode or use a programming language other than Java) * Your application pods (if you choose to run your application's pods on the same worker node) * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components :::note You do not need to deploy an Envoy pod when using `direct-kubernetes` mode. ::: With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [Create at least three worker nodes and three pods](#create-at-least-three-worker-nodes-and-three-pods). However, three nodes with at least 4vCPU / 8GB memory resources per node is the minimum for production environment. You should also consider the resources of the EKS cluster (for example, the number of worker nodes, vCPUs per node, memory per node, ScalarDB Cluster pods, and pods for your application), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node when deciding the worker node resources. ### Configure Cluster Autoscaler in EKS If you want to scale ScalarDB Cluster pods automatically by using [Horizontal Pod Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/horizontal-pod-autoscaler.html), you should configure Cluster Autoscaler in EKS too. For details, see the official Amazon documentation at [Autoscaling](https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html#cluster-autoscaler). In addition, if you configure Cluster Autoscaler, you should create a subnet in an Amazon Virtual Private Cloud (VPC) for EKS with the prefix (e.g., `/24`) to ensure a sufficient number of IPs exist so that EKS can work without network issues after scaling. ### Create the EKS cluster on a private network You should create the EKS cluster on a private network (private subnet in a VPC) since ScalarDB Cluster does not provide any services to users directly via internet access. We recommend accessing ScalarDB Cluster via a private network from your applications. ### Restrict connections by using some security features based on your requirements You should restrict unused connections in ScalarDB Cluster. To restrict unused connections, you can use some security features in AWS, like [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) and [network access control lists](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html). The connections (ports) that ScalarDB Cluster uses by default are as follows: * ScalarDB Cluster * 60053/TCP (accepts gRPC or SQL requests from a client) * 8080/TCP (accepts GraphQL requests from a client) * 9080/TCP (accepts monitoring requests) * Scalar Envoy (used with ScalarDB Cluster `indirect` mode) * 60053/TCP (load balancing for ScalarDB Cluster) * 9001/TCP (accepts monitoring requests for Scalar Envoy itself) :::note - If you change the default listening port for ScalarDB Cluster in the configuration file (`scalardb-cluster-node.properties`), you must allow connections by using the port that you configured. - You must also allow the connections that EKS uses itself. For more details about Amazon EKS security group requirements, refer to [Amazon EKS security group requirements and considerations](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html). ::: ================================================ FILE: docs/scalar-kubernetes/CreateEKSClusterForScalarDL.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Guidelines for creating an EKS cluster for ScalarDL Ledger This document explains the requirements and recommendations for creating an Amazon Elastic Kubernetes Service (EKS) cluster for ScalarDL Ledger deployment. For details on how to deploy ScalarDL Ledger on an EKS cluster, see [Deploy ScalarDL Ledger on Amazon EKS](ManualDeploymentGuideScalarDLOnEKS.mdx). ## Before you begin You must create an EKS cluster based on the following requirements, recommendations, and your project's requirements. For specific details about how to create an EKS cluster, see the official Amazon documentation at [Creating an Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html). ## Requirements When deploying ScalarDL Ledger, you must: * Create the EKS cluster by using a [supported Kubernetes version](https://scalardb.scalar-labs.com/docs/latest/requirements/#kubernetes). * Configure the EKS cluster based on the version of Kubernetes and your project's requirements. :::warning For Byzantine fault detection in ScalarDL to work properly, do not deploy your application pods on the same EKS cluster as the ScalarDL Ledger deployment. ::: ## Recommendations (optional) The following are some recommendations for deploying ScalarDL Ledger. These recommendations are not required, so you can choose whether or not to apply these recommendations based on your needs. ### Create at least three worker nodes and three pods To ensure that the EKS cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardl-custom-values.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different [availability zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) (AZs), you can withstand an AZ failure. ::: ### Use 4vCPU / 8GB memory nodes for the worker node in the ScalarDL Ledger node group It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition to the ScalarDL Ledger pod, Kubernetes could deploy some of the following components to each worker node: * ScalarDL Ledger pod (2vCPU / 4GB) * Envoy proxy * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [Create at least three worker nodes and three pods](#create-at-least-three-worker-nodes-and-three-pods). However, three nodes with at least 4vCPU / 8GB memory resources per node is the minimum environment for production. You should also consider the resources of the EKS cluster (for example, the number of worker nodes, vCPUs per node, memory per node, and ScalarDL Ledger pods), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node when deciding the worker node resources. ### Configure Cluster Autoscaler in EKS If you want to scale ScalarDL Ledger pods automatically by using [Horizontal Pod Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/horizontal-pod-autoscaler.html), you should configure Cluster Autoscaler in EKS too. For details, see the official Amazon documentation at [Autoscaling](https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html#cluster-autoscaler). In addition, if you configure Cluster Autoscaler, you should create a subnet in an Amazon Virtual Private Cloud (VPC) for EKS with the prefix (e.g., `/24`) to ensure a sufficient number of IPs exist so that EKS can work without network issues after scaling. ### Create the EKS cluster on a private network You should create the EKS cluster on a private network (private subnet in a VPC) since ScalarDL Ledger does not provide any services to users directly via internet access. We recommend accessing ScalarDL Ledger via a private network from your applications. ### Restrict connections by using some security features based on your requirements You should restrict unused connections in ScalarDL Ledger. To restrict unused connections, you can use some security features in AWS, like [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) and [network access control lists](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html). The connections (ports) that ScalarDL Ledger uses by default are as follows: * ScalarDL Ledger * 50051/TCP (Accept the requests from a client) * 50052/TCP (accepts privileged requests from a client) * 50053/TCP (accepts pause and unpause requests from a scalar-admin client tool) * 8080/TCP (accepts monitoring requests) * Scalar Envoy (used with ScalarDL Ledger) * 50051/TCP (load balancing for ScalarDL Ledger) * 50052/TCP (load balancing for ScalarDL Ledger) * 9001/TCP (accepts monitoring requests for Scalar Envoy itself) :::note - If you change the default listening port for ScalarDL Ledger in the configuration file (`ledger.properties`), you must allow connections by using the port that you configured. - You must also allow the connections that EKS uses itself. For more details about Amazon EKS security group requirements, refer to [Amazon EKS security group requirements and considerations](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html). ::: ================================================ FILE: docs/scalar-kubernetes/CreateEKSClusterForScalarDLAuditor.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Guidelines for creating an EKS cluster for ScalarDL Ledger and ScalarDL Auditor This document explains the requirements and recommendations for creating an Amazon Elastic Kubernetes Service (EKS) cluster for ScalarDL Ledger and ScalarDL Auditor deployment. For details on how to deploy ScalarDL Ledger and ScalarDL Auditor on an EKS cluster, see [Deploy ScalarDL Ledger and ScalarDL Auditor on Amazon EKS](ManualDeploymentGuideScalarDLAuditorOnEKS.mdx). ## Before you begin You must create an EKS cluster based on the following requirements, recommendations, and your project's requirements. For specific details about how to create an EKS cluster, see the official Amazon documentation at [Creating an Amazon EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html). ## Requirements When deploying ScalarDL Ledger and ScalarDL Auditor, you must: * Create two EKS clusters by using a [supported Kubernetes version](https://scalardb.scalar-labs.com/docs/latest/requirements/#kubernetes). * One EKS cluster for ScalarDL Ledger * One EKS cluster for ScalarDL Auditor * Configure the EKS clusters based on the version of Kubernetes and your project's requirements. * Configure an Amazon Virtual Private Cloud (VPC) as follows. * Connect the **VPC of EKS (for Ledger)** and the **VPC of EKS (for Auditor)** by using [VPC peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html). To do so, you must specify the different IP ranges for the **VPC of EKS (for Ledger)** and the **VPC of EKS (for Auditor)** when you create those VPCs. * Allow **connections between Ledger and Auditor** to make ScalarDL (Auditor mode) work properly. * For more details about these network requirements, refer to [Configure Network Peering for ScalarDL Auditor Mode](NetworkPeeringForScalarDLAuditor.mdx). :::warning For Byzantine fault detection in ScalarDL to work properly, do not deploy your application pods on the same EKS clusters as the ScalarDL Ledger and ScalarDL Auditor deployments. ::: ## Recommendations (optional) The following are some recommendations for deploying ScalarDL Ledger and ScalarDL Auditor. These recommendations are not required, so you can choose whether or not to apply these recommendations based on your needs. ### Create at least three worker nodes and three pods per EKS cluster To ensure that the EKS cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [ScalarDL Ledger sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardl-custom-values.yaml) and [ScalarDL Auditor sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardl-audit-custom-values.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different [availability zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) (AZs), you can withstand an AZ failure. ::: ### Use 4vCPU / 8GB memory nodes for the worker node in the ScalarDL Ledger and ScalarDL Auditor node group It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition to the ScalarDL Ledger and ScalarDL Auditor pods, Kubernetes could deploy some of the following components to each worker node: * EKS cluster for ScalarDL Ledger * ScalarDL Ledger pod (2vCPU / 4GB) * Envoy proxy * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components * EKS cluster for ScalarDL Auditor * ScalarDL Auditor pod (2vCPU / 4GB) * Envoy proxy * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [Create at least three worker nodes and three pods](#create-at-least-three-worker-nodes-and-three-pods-per-eks-cluster). And remember, for Byzantine fault detection to work properly, you cannot deploy your application pods on the same EKS clusters as the ScalarDL Ledger and ScalarDL Auditor deployments. However, three nodes with at least 4vCPU / 8GB memory resources per node is a minimum environment for production. You should also consider the resources of the EKS cluster (for example, the number of worker nodes, vCPUs per node, memory per node, ScalarDL Ledger pods, and ScalarDL Auditor pods), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node when deciding the worker node resources. ### Configure Cluster Autoscaler in EKS If you want to scale ScalarDL Ledger or ScalarDL Auditor pods automatically by using [Horizontal Pod Autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/horizontal-pod-autoscaler.html), you should configure Cluster Autoscaler in EKS too. For details, see the official Amazon documentation at [Autoscaling](https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html#cluster-autoscaler). In addition, if you configure Cluster Autoscaler, you should create a subnet in a VPC for EKS with the prefix (e.g., `/24`) to ensure a sufficient number of IPs exist so that EKS can work without network issues after scaling. ### Create the EKS cluster on a private network You should create the EKS cluster on a private network (private subnet in a VPC) since ScalarDL Ledger and ScalarDL Auditor do not provide any services to users directly via internet access. We recommend accessing ScalarDL Ledger and ScalarDL Auditor via a private network from your applications. ### Restrict connections by using some security features based on your requirements You should restrict unused connections in ScalarDL Ledger and ScalarDL Auditor. To restrict unused connections, you can use some security features in AWS, like [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) and [network access control lists](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html). The connections (ports) that ScalarDL Ledger and ScalarDL Auditor use by default are as follows: * ScalarDL Ledger * 50051/TCP (accepts requests from a client and ScalarDL Auditor) * 50052/TCP (accepts privileged requests from a client and ScalarDL Auditor) * 50053/TCP (accepts pause and unpause requests from a scalar-admin client tool) * 8080/TCP (accepts monitoring requests) * ScalarDL Auditor * 40051/TCP (accepts requests from a client) * 40052/TCP (accepts privileged requests from a client) * 40053/TCP (accepts pause and unpause requests from a scalar-admin client tool) * 8080/TCP (accepts monitoring requests) * Scalar Envoy (used with ScalarDL Ledger and ScalarDL Auditor) * 50051/TCP (load balancing for ScalarDL Ledger) * 50052/TCP (load balancing for ScalarDL Ledger) * 40051/TCP (load balancing for ScalarDL Auditor) * 40052/TCP (load balancing for ScalarDL Auditor) * 9001/TCP (accepts monitoring requests for Scalar Envoy itself) :::note - If you change the default listening port for ScalarDL Ledger and ScalarDL Auditor in their configuration files (`ledger.properties` and `auditor.properties`, respectively), you must allow the connections by using the port that you configured. - You must also allow the connections that EKS uses itself. For more details about Amazon EKS security group requirements, refer to [Amazon EKS security group requirements and considerations](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html). ::: ================================================ FILE: docs/scalar-kubernetes/CreateEKSClusterForScalarProducts.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Guidelines for creating an Amazon EKS cluster for Scalar products To create an Amazon Elastic Kubernetes Service (EKS) cluster for Scalar products, refer to the following: * [Guidelines for creating an EKS cluster for ScalarDB Cluster](CreateEKSClusterForScalarDBCluster.mdx) * [(Deprecated) Guidelines for creating an EKS cluster for ScalarDB Server](CreateEKSClusterForScalarDB.mdx) * [Guidelines for creating an EKS cluster for ScalarDL Ledger](CreateEKSClusterForScalarDL.mdx) * [Guidelines for creating an EKS cluster for ScalarDL Ledger and ScalarDL Auditor](CreateEKSClusterForScalarDLAuditor.mdx) To deploy Scalar products on Amazon EKS, refer to the following: * [Deploy ScalarDB Server on Amazon EKS (Amazon Elastic Kubernetes Service)](ManualDeploymentGuideScalarDBServerOnEKS.mdx) * [Deploy ScalarDL Ledger on Amazon EKS (Amazon Elastic Kubernetes Service)](ManualDeploymentGuideScalarDLOnEKS.mdx) * [Deploy ScalarDL Ledger and ScalarDL Auditor on Amazon EKS (Amazon Elastic Kubernetes Service)](ManualDeploymentGuideScalarDLAuditorOnEKS.mdx) ================================================ FILE: docs/scalar-kubernetes/HowToCreateKeyAndCertificateFiles.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to Create Private Key and Certificate Files for TLS Connections in Scalar Products This guide explains how to create private key and certificate files for TLS connections in ScalarDB Cluster and ScalarDL. When you enable the TLS feature, you must prepare private key and certificate files. ## Certificate requirements * You can use only `RSA` or `ECDSA` as an algorithm for private key and certificate files. * For ScalarDB Analytics server, you must use `PKCS #8` as the private key format based on the default setting of gRPC. ## Example steps to create sample private key and certificate files In this example, you'll create sample private key and certificate files by using `cfssl` and `cfssljson`. If you don't have those tools installed, please install `cfssl` and `cfssljson` to run this example. :::note * You can use other tools, like `openssl`, to create the private key and certificate files. Alternatively, you can ask a third-party CA or the administrator of your private CA to create the private key and certificate for your production environment. * This example creates a self-signed certificate. However, it is strongly recommended that these certificates **not** be used in production. Please ask trusted issuers (a public CA or your private CA) to create certificate files for your production environment based on your security requirements. ::: 1. Create a working directory. ```console mkdir -p ${HOME}/scalar/example/certs/ ``` 1. Change the working directory to `${HOME}/scalar/example/certs/`. ```console cd ${HOME}/scalar/example/certs/ ``` 1. Create a JSON file that includes CA information. ```console cat << 'EOF' > ${HOME}/scalar/example/certs/ca.json { "CN": "scalar-example-ca", "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "JP", "ST": "Tokyo", "L": "Shinjuku", "O": "Scalar Example CA" } ] } EOF ``` 1. Create the CA private key and certificate files. ```console cfssl gencert -initca ca.json | cfssljson -bare ca ``` 1. Create a JSON file that includes CA configurations. ```console cat << 'EOF' > ${HOME}/scalar/example/certs/ca-config.json { "signing": { "default": { "expiry": "87600h" }, "profiles": { "scalar-example-ca": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth" ] } } } } EOF ``` 1. Create a JSON file that includes server information. ```console cat << 'EOF' > ${HOME}/scalar/example/certs/server.json { "CN": "scalar-example-server", "hosts": [ "server.scalar.example.com", "localhost" ], "key": { "algo": "ecdsa", "size": 256 }, "names": [ { "C": "JP", "ST": "Tokyo", "L": "Shinjuku", "O": "Scalar Example Server" } ] } EOF ``` 1. Create the private key and certificate files for the server. ```console cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile scalar-example-ca server.json | cfssljson -bare server ``` 1. Confirm that the private key and certificate files were created. ```console ls -1 ``` [Command execution result] ```console ca-config.json ca-key.pem ca.csr ca.json ca.pem server-key.pem server.csr server.json server.pem ``` In this case: * `server-key.pem` is the private key file. * `server.pem` is the certificate file. * `ca.pem` is the root CA certificate file. ================================================ FILE: docs/scalar-kubernetes/HowToGetContainerImages.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to get the container images of Scalar products import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; You can get the container images of Scalar products in several ways. Please choose one of the following methods. You can get the container images from the public container repository if you have a commercial license. For more details on how to use container images, see [How to use the container images](./HowToUseContainerImages.mdx). For details on how to get Scalar products from AWS Marketplace, see [How to install Scalar products through AWS Marketplace](./AwsMarketplaceGuide.mdx). Scalar products are currently not available in Azure Marketplace. Please get the container images from one of the other methods. ================================================ FILE: docs/scalar-kubernetes/HowToScaleScalarDB.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to Scale ScalarDB Cluster This guide explains how to scale ScalarDB Cluster. The contents of this guide assume that you used [Scalar Helm Chart](https://github.com/scalar-labs/helm-charts) to deploy ScalarDB Cluster, which is the recommended way. :::note You might be able to resolve some performance issues by scaling ScalarDB Cluster if a bottleneck exists on the ScalarDB Cluster side. However, sometimes a performance issue is caused by a bottleneck in the backend databases. In such cases, scaling ScalarDB Cluster will not resolve the performance issue. Instead, please check where the bottleneck exists. If the bottleneck exists in the backend databases, consider scaling the backend databases. ::: 1. Add the following to your custom values file, replacing `` with the number of pods you want to scale: ```yaml scalardbCluster: replicaCount: ``` 1. Upgrade your ScalarDB Cluster deployment by running the following `helm upgrade` command, which uses the updated custom values file. Be sure to replace the contents in the angle brackets as described: ```console helm upgrade scalar-labs/scalardb-cluster -n -f / --version ``` ================================================ FILE: docs/scalar-kubernetes/HowToScaleScalarDL.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # How to Scale ScalarDL import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; This guide explains how to scale ScalarDL. The contents of this guide assume that you used [Scalar Helm Chart](https://github.com/scalar-labs/helm-charts) to deploy ScalarDL, which is the recommended way. :::note You might be able to resolve some performance issues by scaling ScalarDL if a bottleneck exists on the ScalarDL side. However, sometimes a performance issue is caused by a bottleneck in the backend databases. In such cases, scaling ScalarDL will not resolve the performance issue. Instead, please check where the bottleneck exists. If the bottleneck exists in the backend databases, consider scaling the backend databases. ::: 1. Add the following to your custom values file, replacing `` with the number of pods you want to scale: ```yaml ledger: replicaCount: ``` 1. Upgrade your ScalarDL Ledger deployment by running the following `helm upgrade` command, which uses the updated custom values file. Be sure to replace the contents in the angle brackets as described: ```console helm upgrade scalar-labs/scalardl -n -f / --version ``` 1. Add the following to your custom values file, replacing `` with the number of pods you want to scale: ```yaml auditor: replicaCount: ``` 1. Upgrade your ScalarDL Auditor deployment by running the following `helm upgrade` command, which uses the updated custom values file. Be sure to replace the contents in the angle brackets as described: ```console helm upgrade scalar-labs/scalardl-audit -n -f / --version ``` ================================================ FILE: docs/scalar-kubernetes/HowToUpgradeScalarDB.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to Upgrade ScalarDB import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; This guide explains how to upgrade to a newer version of ScalarDB. ## Before you begin Before you upgrade to a new version, please check the [ScalarDB Cluster Compatibility Matrix](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/compatibility/) to ensure compatibility between ScalarDB Cluster and the client SDKs. ## Upgrade versions To learn about upgrading your version of ScalarDB, select the type of upgrade you want to do. Major versions do **not** keep backward compatibility. So, you might need to do special operations when you upgrade from one major version to another major version. For example: - Update the database schema on the backend database side. - Update the API in your application. For details on what you need when you upgrade to a major version, please refer to the release notes for the major version that you want to upgrade to. Minor versions keep backward compatibility. So, you can upgrade ScalarDB from one minor version to another minor version in the same major version without doing any special operations. For example, you don't need to update the database schema on the backend database side or update the API in your application. If you use [Scalar Helm Chart](https://github.com/scalar-labs/helm-charts) to deploy ScalarDB Cluster, you can upgrade your ScalarDB Cluster deployment as follows: 1. Set the ScalarDB Cluster Helm Chart version as an environment variable. You can do this by running the following command to put the chart version into the environment variable `SCALAR_DB_CLUSTER_CHART_VERSION`: ```console SCALAR_DB_CLUSTER_CHART_VERSION=1.5.0 ``` :::tip You can search for the chart version that corresponds to the ScalarDB Cluster version, run the following command: ```console helm search repo scalar-labs/scalardb-cluster -l ``` The following command might be helpful, but please make sure to replace the contents in the angle brackets with your version of ScalarDB Cluster: ```console SCALAR_DB_CLUSTER_VERSION=..; SCALAR_DB_CLUSTER_CHART_VERSION=$(helm search repo scalar-labs/scalardb-cluster -l | grep -F "${SCALAR_DB_CLUSTER_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` ::: 1. Upgrade your ScalarDB Cluster deployment by replacing the contents in the angle brackets as described: ```console helm upgrade scalar-labs/scalardb-cluster -n -f / --version ${SCALAR_DB_CLUSTER_CHART_VERSION} ``` After you upgrade the ScalarDB Cluster deployment, you should consider upgrading the version of the [ScalarDB Cluster Java Client SDK](https://mvnrepository.com/artifact/com.scalar-labs/scalardb-cluster-java-client-sdk) or the [ScalarDB Cluster .NET Client SDK](https://www.nuget.org/packages/ScalarDB.Net.Client) on your application side. ScalarDB Core is provided as a Java library. So, you can update the dependencies of your Java project and rebuild your application to upgrade ScalarDB versions. Patch versions keep backward compatibility. So, you can upgrade ScalarDB from one patch version to another patch version in the same major version and minor version without doing any special operations. For example, you don't need to update the database schema on the backend database side or update the API in your application. The method for upgrading to a patch version is the same as for upgrading to a minor version. For details on how to upgrade, see the [Upgrade to a minor version](?versions=upgrade-minor-version) tab. :::warning ScalarDB does **not** support downgrading to a previous version (major, minor, or patch). You can only upgrade to a newer version. ::: ================================================ FILE: docs/scalar-kubernetes/HowToUpgradeScalarDL.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # How to Upgrade ScalarDL import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; This guide explains how to upgrade to a newer version of ScalarDL. ## Before you begin Before you upgrade to a new version, please check the [ScalarDL Compatibility Matrix](https://scalardl.scalar-labs.com/docs/latest/compatibility/) to ensure compatibility between ScalarDL and the client SDKs. ## Upgrade versions To learn about upgrading your version of ScalarDL, select the type of upgrade you want to do. Major versions do **not** keep backward compatibility. So, you might need to do special operations when you upgrade from one major version to another major version. For example: - Update the database schema on the backend database side. - Update the API in your application. For details on what you need when you upgrade to a major version, please refer to the release notes for the major version that you want to upgrade to. Minor versions keep backward compatibility. So, you can upgrade ScalarDL from one minor version to another minor version in the same major version without doing any special operations. For example, you don't need to update the database schema on the backend database side or update the API in your application. If you use [Scalar Helm Chart](https://github.com/scalar-labs/helm-charts) to deploy ScalarDL Ledger, you can upgrade your ScalarDL Ledger deployment as follows: 1. Set the ScalarDL Ledger Helm Chart version as an environment variable. You can do this by running the following command to put the chart version into the environment variable `SCALAR_DL_LEDGER_CHART_VERSION`: ```console SCALAR_DL_LEDGER_CHART_VERSION=4.8.0 ``` :::tip You can search for the chart version that corresponds to the ScalarDL Ledger version as follows: ```console helm search repo scalar-labs/scalardl -l ``` The following command might be helpful, but please make sure to replace the contents in the angle brackets with your version of ScalarDL Ledger: ```console SCALAR_DL_VERSION=..; SCALAR_DL_LEDGER_CHART_VERSION=$(helm search repo scalar-labs/scalardl -l | grep -v -e "scalar-labs/scalardl-audit" | grep -F "${SCALAR_DL_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` ::: 1. Upgrade your ScalarDL Ledger deployment by replacing the contents in the angle brackets as described: ```console helm upgrade scalar-labs/scalardl -n -f / --version ${SCALAR_DL_LEDGER_CHART_VERSION} ``` After you upgrade the ScalarDL Ledger deployment (and the ScalarDL Auditor deployment if you use Auditor mode), you should consider upgrading the version of the [ScalarDL Java Client SDK](https://mvnrepository.com/artifact/com.scalar-labs/scalardl-java-client-sdk) on your application side. If you use [Scalar Helm Chart](https://github.com/scalar-labs/helm-charts) to deploy ScalarDL Auditor, you can upgrade your ScalarDL Auditor deployment as follows: 1. Set the ScalarDL Auditor Helm Chart version as an environment variable. You can do this by running the following command to put the chart version into the environment variable `SCALAR_DL_AUDITOR_CHART_VERSION`: ```console SCALAR_DL_AUDITOR_CHART_VERSION=2.8.0 ``` :::tip You can search for the chart version that corresponds to the ScalarDL Auditor version as follows: ```console helm search repo scalar-labs/scalardl-audit -l ``` The following command might be helpful, but please make sure to replace the contents in the angle brackets with your version of ScalarDL Auditor: ```console SCALAR_DL_VERSION=..; SCALAR_DL_AUDITOR_CHART_VERSION=$(helm search repo scalar-labs/scalardl-audit -l | grep -F "${SCALAR_DL_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` ::: 1. Upgrade your ScalarDL Auditor deployment by replacing the contents in the angle brackets as described: ```console helm upgrade scalar-labs/scalardl-audit -n -f / --version ${SCALAR_DL_AUDITOR_CHART_VERSION} ``` After you upgrade the ScalarDL Auditor deployment and the ScalarDL Ledger deployment, you should consider upgrading the version of the [ScalarDL Java Client SDK](https://mvnrepository.com/artifact/com.scalar-labs/scalardl-java-client-sdk) on your application side. Patch versions keep backward compatibility. So, you can upgrade ScalarDL from one patch version to another patch version in the same major version and minor version without doing any special operations. For example, you don't need to update the database schema on the backend database side or update the API in your application. The method for upgrading to a patch version is the same as for upgrading to a minor version. For details on how to upgrade, see the [Upgrade to a minor version](?versions=upgrade-minor-version) tab. :::warning ScalarDL does **not** support downgrading to a previous version (major, minor, or patch). You can only upgrade to a newer version. ::: ================================================ FILE: docs/scalar-kubernetes/HowToUseContainerImages.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to use the container images import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; You can pull the container images from the public container repository. You must configure the license key and the certificate in your `.properties` file if you use the container images. ## Prerequisites The public container images are available for the following products and versions: * ScalarDB Cluster v3.12 or later * ScalarDL v3.9 or later ## Pull the container images from the public container repository You can pull the container image of each product from the public container repository. To pull a container image, select your Scalar product to see the link to the container image. Select your edition of ScalarDB Enterprise. https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-node-byol-standard https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-node-byol-premium https://github.com/orgs/scalar-labs/packages/container/package/scalardl-ledger-byol https://github.com/orgs/scalar-labs/packages/container/package/scalardl-auditor-byol If you're using Scalar Helm Charts, you must set `*.image.repository` in the custom values file for the product that you're using. Select your Scalar product to see how to set `*.image.repository`. Select your edition of ScalarDB Enterprise. ```yaml scalardbCluster: image: repository: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-standard" ``` ```yaml scalardbCluster: image: repository: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium" ``` ```yaml ledger: image: repository: "ghcr.io/scalar-labs/scalardl-ledger-byol" ``` ```yaml auditor: image: repository: "ghcr.io/scalar-labs/scalardl-auditor-byol" ``` ## Set the license key in the `.properties` file To run the container images, you must set `license key` and `certificate` in your `.properties` file. Select your Scalar product to see how to set `license key` and `certificate`. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). ```properties scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` ```properties scalar.dl.licensing.license_key= scalar.dl.licensing.license_check_cert_pem= ``` ```properties scalar.dl.licensing.license_key= scalar.dl.licensing.license_check_cert_pem= ``` If you're using Scalar Helm Charts, you must set the properties in the custom values file for the product that you're using. Select your Scalar product to see how to set the properties in the custom values file. ```yaml scalardbCluster: scalardbClusterNodeProperties: | scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` ```yaml ledger: ledgerProperties: | scalar.dl.licensing.license_key= scalar.dl.licensing.license_check_cert_pem= ``` ```yaml auditor: auditorProperties: | scalar.dl.licensing.license_key= scalar.dl.licensing.license_check_cert_pem= ``` ================================================ FILE: docs/scalar-kubernetes/K8sLogCollectionGuide.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Collecting logs from Scalar products on a Kubernetes cluster import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; This document explains how to deploy Grafana Loki and Promtail on Kubernetes with Helm. After following this document, you can collect logs of Scalar products on your Kubernetes environment. If you use a managed Kubernetes cluster and you want to use the cloud service features for monitoring and logging, please refer to the following document. * [Logging and monitoring on Amazon EKS](https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/amazon-eks-logging-monitoring.html) * [Monitoring Azure Kubernetes Service (AKS) with Azure Monitor](https://learn.microsoft.com/en-us/azure/aks/monitor-aks) ## Prerequisites * Create a Kubernetes cluster. * [Create an EKS cluster for Scalar products](CreateEKSClusterForScalarProducts.mdx) * [Create an AKS cluster for Scalar products](CreateAKSClusterForScalarProducts.mdx) * Create a Bastion server and set `kubeconfig`. * [Create a bastion server](CreateBastionServer.mdx) * Deploy Prometheus Operator (we use Grafana to explore collected logs) * [Monitoring Scalar products on the Kubernetes cluster](K8sMonitorGuide.mdx) ## Add the grafana helm repository This document uses Helm for the deployment of Prometheus Operator. ```console helm repo add grafana https://grafana.github.io/helm-charts ``` ```console helm repo update ``` ## Prepare a custom values file Please get the sample file [scalar-loki-stack-custom-values.yaml](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalar-loki-stack-custom-values.yaml) for loki-stack. For the logging of Scalar products, this sample file's configuration is recommended. ### Set nodeSelector in the custom values file (Optional) You might need to set nodeSelector in the custom values file (scalar-loki-stack-custom-values.yaml) as follows if you add labels to your Kubernetes worker node. See the following examples based on the product you're using. Select the ScalarDB product you're using. ```yaml promtail: nodeSelector: scalar-labs.com/dedicated-node: scalardb-cluster ``` ```yaml promtail: nodeSelector: scalar-labs.com/dedicated-node: scalardb ``` Select the ScalarDL product you're using. ```yaml promtail: nodeSelector: scalar-labs.com/dedicated-node: scalardl-ledger ``` ```yaml promtail: nodeSelector: scalar-labs.com/dedicated-node: scalardl-auditor ``` ### Set tolerations in the custom values file (Optional) You might need to set tolerations in the custom values file (scalar-loki-stack-custom-values.yaml) as follows if you add taints to your Kubernetes worker node. See the following examples based on the product you're using. Select the ScalarDB product you're using. ```yaml promtail: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardb-cluster ``` ```yaml promtail: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardb ``` Select the ScalarDL product you're using. ```yaml promtail: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardl-ledger ``` ```yaml promtail: tolerations: - effect: NoSchedule key: scalar-labs.com/dedicated-node operator: Equal value: scalardl-auditor ``` ## Deploy Loki and Promtail It is recommended to deploy Loki and Promtail on the same namespace `monitoring` as Prometheus and Grafana. You have already created the `monitoring` namespace in the document [Monitoring Scalar products on the Kubernetes cluster](K8sMonitorGuide.mdx). ```console helm install scalar-logging-loki grafana/loki-stack -n monitoring -f scalar-loki-stack-custom-values.yaml ``` ## Check if Loki and Promtail are deployed If the Loki and Promtail pods are deployed properly, you can see the `STATUS` is `Running` using the `kubectl get pod -n monitoring` command. Since promtail pods are deployed as DaemonSet, the number of promtail pods depends on the number of Kubernetes nodes. In the following example, there are three worker nodes for Scalar products in the Kubernetes cluster. ```console kubectl get pod -n monitoring ``` You should see the following output: ```console NAME READY STATUS RESTARTS AGE scalar-logging-loki-0 1/1 Running 0 35m scalar-logging-loki-promtail-2fnzn 1/1 Running 0 32m scalar-logging-loki-promtail-2pwkx 1/1 Running 0 30m scalar-logging-loki-promtail-gfx44 1/1 Running 0 32m ``` ## View log in Grafana dashboard You can see the collected logs in the Grafana dashboard as follows. 1. Access the Grafana dashboard 1. Go to the `Explore` page 1. Select `Loki` from the top left pull-down 1. Set conditions to query logs 1. Select the `Run query` button at the top right Please refer to the [Monitoring Scalar products on the Kubernetes cluster](K8sMonitorGuide.mdx) for more details on how to access the Grafana dashboard. ================================================ FILE: docs/scalar-kubernetes/K8sMonitorGuide.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Monitoring Scalar products on a Kubernetes cluster This document explains how to deploy Prometheus Operator on Kubernetes with Helm. After following this document, you can use Prometheus, Alertmanager, and Grafana for monitoring Scalar products on your Kubernetes environment. If you use a managed Kubernetes cluster and you want to use the cloud service features for monitoring and logging, please refer to the following document. * [Logging and monitoring on Amazon EKS](https://docs.aws.amazon.com/prescriptive-guidance/latest/implementing-logging-monitoring-cloudwatch/amazon-eks-logging-monitoring.html) * [Monitoring Azure Kubernetes Service (AKS) with Azure Monitor](https://learn.microsoft.com/en-us/azure/aks/monitor-aks) ## Prerequisites * Create a Kubernetes cluster. * [Create an EKS cluster for Scalar products](CreateEKSClusterForScalarProducts.mdx) * [Create an AKS cluster for Scalar products](CreateAKSClusterForScalarProducts.mdx) * Create a Bastion server and set `kubeconfig`. * [Create a bastion server](CreateBastionServer.mdx) ## Add the prometheus-community helm repository This document uses Helm for the deployment of Prometheus Operator. ```console helm repo add prometheus-community https://prometheus-community.github.io/helm-charts ``` ```console helm repo update ``` ## Prepare a custom values file Please get the sample file [scalar-prometheus-custom-values.yaml](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalar-prometheus-custom-values.yaml) for kube-prometheus-stack. For the monitoring of Scalar products, this sample file's configuration is recommended. In this sample file, the Service resources are not exposed to access from outside of a Kubernetes cluster. If you want to access dashboards from outside of your Kubernetes cluster, you must set `*.service.type` to `LoadBalancer` or `*.ingress.enabled` to `true`. Please refer to the following official document for more details on the configurations of kube-prometheus-stack. * [kube-prometheus-stack - Configuration](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#configuration) ## Deploy Prometheus Operator Scalar products assume the Prometheus Operator is deployed in the `monitoring` namespace by default. So, please create the namespace `monitoring` and deploy Prometheus Operator in the `monitoring` namespace. 1. Create a namespace `monitoring` on Kubernetes. ```console kubectl create namespace monitoring ``` 1. Deploy the kube-prometheus-stack. ```console helm install scalar-monitoring prometheus-community/kube-prometheus-stack -n monitoring -f scalar-prometheus-custom-values.yaml ``` ## Check if the Prometheus Operator is deployed If the Prometheus Operator (includes Prometheus, Alertmanager, and Grafana) pods are deployed properly, you can see the `STATUS` is `Running` using the following command: ```console kubectl get pod -n monitoring ``` You should see the following output: ```console NAME READY STATUS RESTARTS AGE alertmanager-scalar-monitoring-kube-pro-alertmanager-0 2/2 Running 0 55s prometheus-scalar-monitoring-kube-pro-prometheus-0 2/2 Running 0 55s scalar-monitoring-grafana-cb4f9f86b-jmkpz 3/3 Running 0 62s scalar-monitoring-kube-pro-operator-865bbb8454-9ppkc 1/1 Running 0 62s ``` ## Deploy (or Upgrade) Scalar products using Helm Charts 1. To enable Prometheus monitoring for Scalar products, you must set `true` to the following configurations in the custom values file. * Configurations * `*.prometheusRule.enabled` * `*.grafanaDashboard.enabled` * `*.serviceMonitor.enabled` Please refer to the following documents for more details on the custom values file of each Scalar product. * [ScalarDB Cluster](../helm-charts/configure-custom-values-scalardb-cluster.mdx#prometheus-and-grafana-configurations-recommended-in-production-environments) * [(Deprecated) ScalarDB Server](../helm-charts/configure-custom-values-scalardb.mdx#prometheusgrafana-configurations-recommended-in-the-production-environment) * [(Deprecated) ScalarDB GraphQL](../helm-charts/configure-custom-values-scalardb-graphql.mdx#prometheusgrafana-configurations-recommended-in-the-production-environment) * [ScalarDL Ledger](../helm-charts/configure-custom-values-scalardl-ledger.mdx#prometheusgrafana-configurations-recommended-in-the-production-environment) * [ScalarDL Auditor](../helm-charts/configure-custom-values-scalardl-auditor.mdx#prometheusgrafana-configurations-recommended-in-the-production-environment) 1. Deploy (or Upgrade) Scalar products using Helm Charts with the above custom values file. Please refer to the following documents for more details on how to deploy/upgrade Scalar products. * [ScalarDB Cluster](../helm-charts/how-to-deploy-scalardb-cluster.mdx) * [(Deprecated) ScalarDB Server](../helm-charts/how-to-deploy-scalardb.mdx) * [(Deprecated) ScalarDB GraphQL](../helm-charts/how-to-deploy-scalardb-graphql.mdx) * [ScalarDL Ledger](../helm-charts/how-to-deploy-scalardl-ledger.mdx) * [ScalarDL Auditor](../helm-charts/how-to-deploy-scalardl-auditor.mdx) ## How to access dashboards When you set `*.service.type` to `LoadBalancer` or `*.ingress.enabled` to `true`, you can access dashboards via Service or Ingress of Kubernetes. The concrete implementation and access method depend on the Kubernetes cluster. If you use a managed Kubernetes cluster, please refer to the cloud provider's official document for more details. * EKS * [Network load balancing on Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html) * [Application load balancing on Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html) * AKS * [Use a public standard load balancer in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/load-balancer-standard) * [Create an ingress controller in Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/ingress-basic) ## Access the dashboard from your local machine (For testing purposes only / Not recommended in the production environment) You can access each dashboard from your local machine using the `kubectl port-forward` command. 1. Port forwarding to each service from your local machine. * Prometheus ```console kubectl port-forward -n monitoring svc/scalar-monitoring-kube-pro-prometheus 9090:9090 ``` * Alertmanager ```console kubectl port-forward -n monitoring svc/scalar-monitoring-kube-pro-alertmanager 9093:9093 ``` * Grafana ```console kubectl port-forward -n monitoring svc/scalar-monitoring-grafana 3000:3000 ``` 1. Access each Dashboard. * Prometheus ```console http://localhost:9090/ ``` * Alertmanager ```console http://localhost:9093/ ``` * Grafana ```console http://localhost:3000/ ``` * Note: * You can see the user and password of Grafana as follows. * user ```console kubectl get secrets scalar-monitoring-grafana -n monitoring -o jsonpath='{.data.admin-user}' | base64 -d ``` * password ```console kubectl get secrets scalar-monitoring-grafana -n monitoring -o jsonpath='{.data.admin-password}' | base64 -d ``` ================================================ FILE: docs/scalar-kubernetes/ManualDeploymentGuideScalarDBClusterOnEKS.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Deploy ScalarDB Cluster on Amazon Elastic Kubernetes Service (EKS) This guide explains how to deploy ScalarDB Cluster on Amazon Elastic Kubernetes Service (EKS). In this guide, you will create one of the following two environments in your AWS environment. The environments differ depending on which [client mode](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api#client-modes) you use: * **[`direct-kubernetes` client mode](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api#direct-kubernetes-client-mode).** In this mode, you deploy your application in the same EKS cluster as your ScalarDB Cluster deployment. ![image](images/png/EKS_ScalarDB_Cluster_Direct_Kubernetes_Mode.drawio.png) * **[`indirect` client mode](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api#indirect-client-mode).** In this mode, you deploy your application in an environment that is different from the EKS cluster that contains your ScalarDB Cluster deployment. ![image](images/png/EKS_ScalarDB_Cluster_Indirect_Mode.drawio.png) ## Step 1. Subscribe to ScalarDB Cluster in AWS Marketplace You must get the ScalarDB Cluster container image by visiting AWS Marketplace and subscribing to [ScalarDB Cluster Standard Edition (Pay-As-You-Go)](https://aws.amazon.com/marketplace/pp/prodview-jx6qxatkxuwm4) or [ScalarDB Cluster Premium Edition (Pay-As-You-Go)](https://aws.amazon.com/marketplace/pp/prodview-djqw3zk6dwyk6). For details on how to subscribe to ScalarDB Cluster in AWS Marketplace, see [Subscribe to Scalar products from AWS Marketplace](AwsMarketplaceGuide.mdx#subscribe-to-scalar-products-from-aws-marketplace). ## Step 2. Create an EKS cluster You must create an EKS cluster for the ScalarDB Cluster deployment. For details, see [Guidelines for creating an Amazon EKS cluster for Scalar products](CreateEKSClusterForScalarProducts.mdx). ## Step 3. Set up a database for ScalarDB Cluster You must prepare a database before deploying ScalarDB Cluster. To see which types of databases ScalarDB supports, refer to [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases). For details on setting up a database, see [Set up a database for ScalarDB/ScalarDL deployment on AWS](SetupDatabaseForAWS.mdx). ## Step 4. Create a bastion server To execute some tools for deploying and managing ScalarDB Cluster on EKS, you must prepare a bastion server in the same Amazon Virtual Private Cloud (VPC) of the EKS cluster that you created in **Step 2**. For details, see [Create a Bastion Server](CreateBastionServer.mdx). ## Step 5. Prepare a custom values file for the Scalar Helm Chart To perform tasks, like accessing information in the database that you created in **Step 3**, you must configure a custom values file for the Scalar Helm Chart for ScalarDB Cluster based on your environment. For details, see [Configure a custom values file for Scalar Helm Charts](../helm-charts/configure-custom-values-file.mdx). **Note:** If you deploy your application in an environment that is different from the EKS cluster that has your ScalarDB Cluster deployment (i.e., you use `indirect` client mode), you must set the `envoy.enabled` parameter to `true` and the `envoy.service.type` parameter to `LoadBalancer` to access Scalar Envoy from your application. ## Step 6. Deploy ScalarDB Cluster by using the Scalar Helm Chart Deploy ScalarDB Cluster on your EKS cluster by using the Helm Chart for ScalarDB Cluster. For details, see [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx). **Note:** We recommend creating a dedicated namespace by using the `kubectl create ns scalardb-cluster` command and deploying ScalarDB Cluster in the namespace by using the `-n scalardb-cluster` option with the `helm install` command. ## Step 7. Check the status of your ScalarDB Cluster deployment After deploying ScalarDB Cluster in your EKS cluster, you must check the status of each component. For details, see [Components to Regularly Check When Running in a Kubernetes Environment](RegularCheck.mdx). ## Step 8. Monitor your ScalarDB Cluster deployment After deploying ScalarDB Cluster in your EKS cluster, we recommend monitoring the deployed components and collecting their logs, especially in production. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ## Step 9. Deploy your application If you use [`direct-kubernetes` client mode](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api#direct-kubernetes-client-mode), you must deploy additional Kubernetes resources. For details, see [Deploy your client application on Kubernetes with `direct-kubernetes` mode](../helm-charts/how-to-deploy-scalardb-cluster.mdx#deploy-your-client-application-on-kubernetes-with-direct-kubernetes-mode). ## Remove ScalarDB Cluster from EKS If you want to remove the environment that you created, please remove all the resources in reverse order from which you created them in. ================================================ FILE: docs/scalar-kubernetes/ManualDeploymentGuideScalarDBServerOnAKS.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Deprecated displayed_sidebar: docsEnglish --- # [Deprecated] Deploy ScalarDB Server on Azure Kubernetes Service (AKS) This guide explains how to deploy ScalarDB Server on Azure Kubernetes Service (AKS). In this guide, you will create one of the following two environments in your Azure environment. The difference between the two environments is how you plan to deploy the application: * Deploy your application in the same AKS cluster as your ScalarDB Server deployment. In this case, you don't need to use the load balancers that Azure provides to access Scalar Envoy from your application. ![image](images/png/AKS_ScalarDB_Server_App_In_Cluster.drawio.png) * Deploy your application in an environment that is different from the AKS cluster that contains your ScalarDB Server deployment. In this case, you must use the load balancers that Azure provides to access Scalar Envoy from your application. ![image](images/png/AKS_ScalarDB_Server_App_Out_Cluster.drawio.png) ## Step 1. Create an AKS cluster You must create an AKS cluster for the ScalarDB Server deployment. For details, see [Guidelines for creating an AKS cluster for Scalar products](CreateAKSClusterForScalarProducts.mdx). ## Step 2. Set up a database for ScalarDB Server You must prepare a database before deploying ScalarDB Server. To see which types of databases ScalarDB supports, refer to [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases). For details on setting up a database, see [Set up a database for ScalarDB/ScalarDL deployment in Azure](SetupDatabaseForAzure.mdx). ## Step 3. Create a bastion server To execute some tools for deploying and managing ScalarDB Server on AKS, you must prepare a bastion server in the same Azure Virtual Network (VNet) of the AKS cluster that you created in **Step 1**. For details, see [Create a Bastion Server](CreateBastionServer.mdx). ## Step 4. Prepare a custom values file for the Scalar Helm Chart To perform tasks, like accessing information in the database that you created in **Step 2**, you must configure a custom values file for the Scalar Helm Chart for ScalarDB Server based on your environment. For details, see [Configure a custom values file of Scalar Helm Chart](../helm-charts/configure-custom-values-file.mdx). **Note:** If you deploy your application in an environment that is different from the AKS cluster that has your ScalarDB Server deployment, you must set the `envoy.service.type` parameter to `LoadBalancer` to access Scalar Envoy from your application. ## Step 5. Deploy ScalarDB Server by using the Scalar Helm Chart Deploy ScalarDB Server on your AKS cluster by using the Helm Chart for ScalarDB Server. For details, see [Deploy Scalar Products using Scalar Helm Chart](../helm-charts/how-to-deploy-scalar-products.mdx). **Note:** We recommend creating a dedicated namespace by using the `kubectl create ns scalardb` command and deploying ScalarDB Server in the namespace by using the `-n scalardb` option with the `helm install` command. ## Step 6. Check the status of your ScalarDB Server deployment After deploying ScalarDB Server in your AKS cluster, you must check the status of each component. For details, see [Components to Regularly Check When Running in a Kubernetes Environment](RegularCheck.mdx). ## Step 7. Monitor your ScalarDB Server deployment After deploying ScalarDB Server in your AKS cluster, we recommend monitoring the deployed components and collecting their logs, especially in production. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ## Remove ScalarDB Server from AKS If you want to remove the environment that you created, please remove all the resources in reverse order from which you created them in. ================================================ FILE: docs/scalar-kubernetes/ManualDeploymentGuideScalarDBServerOnEKS.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Deprecated displayed_sidebar: docsEnglish --- # Deploy ScalarDB Server on Amazon Elastic Kubernetes Service (EKS) This guide explains how to deploy ScalarDB Server on Amazon Elastic Kubernetes Service (EKS). In this guide, you will create one of the following two environments in your AWS environment. The difference between the two environments is how you plan to deploy the application: * Deploy your application in the same EKS cluster as your ScalarDB Server deployment. In this case, you don't need to use the load balancers that AWS provides to access Scalar Envoy from your application. ![image](images/png/EKS_ScalarDB_Server_App_In_Cluster.drawio.png) * Deploy your application in an environment that is different from the EKS cluster that contains your ScalarDB Server deployment. In this case, you must use the load balancers that AWS provides to access Scalar Envoy from your application. ![image](images/png/EKS_ScalarDB_Server_App_Out_Cluster.drawio.png) ## Step 1. Subscribe to ScalarDB Server in AWS Marketplace You must get the ScalarDB Server container image by visiting [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-rzbuhxgvqf4d2) and subscribing to ScalarDB Server. For details on how to subscribe to ScalarDB Server in AWS Marketplace, see [Subscribe to Scalar products from AWS Marketplace](AwsMarketplaceGuide.mdx#subscribe-to-scalar-products-from-aws-marketplace). ## Step 2. Create an EKS cluster You must create an EKS cluster for the ScalarDB Server deployment. For details, see [Guidelines for creating an Amazon EKS cluster for Scalar products](CreateEKSClusterForScalarProducts.mdx). ## Step 3. Set up a database for ScalarDB Server You must prepare a database before deploying ScalarDB Server. To see which types of databases ScalarDB supports, refer to [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases). For details on setting up a database, see [Set up a database for ScalarDB/ScalarDL deployment on AWS](SetupDatabaseForAWS.mdx). ## Step 4. Create a bastion server To execute some tools for deploying and managing ScalarDB Server on EKS, you must prepare a bastion server in the same Amazon Virtual Private Cloud (VPC) of the EKS cluster that you created in **Step 2**. For details, see [Create a Bastion Server](CreateBastionServer.mdx). ## Step 5. Prepare a custom values file for the Scalar Helm Chart To perform tasks, like accessing information in the database that you created in **Step 3**, you must configure a custom values file for the Scalar Helm Chart for ScalarDB Server based on your environment. For details, see [Configure a custom values file for Scalar Helm Charts](../helm-charts/configure-custom-values-file.mdx). **Note:** If you deploy your application in an environment that is different from the EKS cluster that has your ScalarDB Server deployment, you must set the `envoy.service.type` parameter to `LoadBalancer` to access Scalar Envoy from your application. ## Step 6. Deploy ScalarDB Server by using the Scalar Helm Chart Deploy ScalarDB Server on your EKS cluster by using the Helm Chart for ScalarDB Server. For details, see [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx). **Note:** We recommend creating a dedicated namespace by using the `kubectl create ns scalardb` command and deploying ScalarDB Server in the namespace by using the `-n scalardb` option with the `helm install` command. ## Step 7. Check the status of your ScalarDB Server deployment After deploying ScalarDB Server in your EKS cluster, you must check the status of each component. For details, see [Components to Regularly Check When Running in a Kubernetes Environment](RegularCheck.mdx). ## Step 8. Monitor your ScalarDB Server deployment After deploying ScalarDB Server in your EKS cluster, we recommend monitoring the deployed components and collecting their logs, especially in production. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ## Remove ScalarDB Server from EKS If you want to remove the environment that you created, please remove all the resources in reverse order from which you created them in. ================================================ FILE: docs/scalar-kubernetes/ManualDeploymentGuideScalarDLAuditorOnAKS.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Deploy ScalarDL Ledger and ScalarDL Auditor on Azure Kubernetes Service (AKS) This guide explains how to deploy ScalarDL Ledger and ScalarDL Auditor on Azure Kubernetes Service (AKS). In this guide, you will create one of the following three environments in your Azure environment. To make Byzantine fault detection work properly, we recommend deploying ScalarDL Ledger and ScalarDL Auditor on different administrative domains (i.e., separate environments). * Use different Azure accounts (most recommended way) ![image](images/png/AKS_ScalarDL_Auditor_Multi_Account.drawio.png) * Use different Azure Virtual Networks (VNets) (second recommended way) ![image](images/png/AKS_ScalarDL_Auditor_Multi_VNet.drawio.png) * Use different namespaces (third recommended way) ![image](images/png/AKS_ScalarDL_Auditor_Multi_Namespace.drawio.png) **Note:** This guide follows the second recommended way, "Use different VNets." ## Step 1. Get the ScalarDL Ledger and ScalarDL Auditor container images You must get the ScalarDL Ledger and ScalarDL Auditor container images. For details, see [How to get the container images of Scalar products](HowToGetContainerImages.mdx). ## Step 2. Create an AKS cluster for ScalarDL Ledger You must create an AKS cluster for the ScalarDL Ledger deployment. For details, see [Guidelines for creating an AKS cluster for Scalar products](CreateAKSClusterForScalarProducts.mdx). ## Step 3. Create an AKS cluster for ScalarDL Auditor You must also create an AKS cluster for the ScalarDL Auditor deployment. For details, see [Guidelines for creating an AKS cluster for Scalar products](CreateAKSClusterForScalarProducts.mdx). ## Step 4. Set up a database for ScalarDL Ledger You must prepare a database before deploying ScalarDL Ledger. Because ScalarDL Ledger uses ScalarDB internally to access databases, refer to [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases) to see which types of databases ScalarDB supports. For details on setting up a database, see [Set up a database for ScalarDB/ScalarDL deployment in Azure](SetupDatabaseForAzure.mdx). ## Step 5. Set up a database for ScalarDL Auditor You must also prepare a database before deploying ScalarDL Auditor. Because ScalarDL Auditor uses ScalarDB internally to access databases, refer to [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases) to see which types of databases ScalarDB supports. For details on setting up a database, see [Set up a database for ScalarDB/ScalarDL deployment in Azure](SetupDatabaseForAzure.mdx). ## Step 6. Create a bastion server for ScalarDL Ledger To execute some tools for deploying and managing ScalarDL Ledger on AKS, you must prepare a bastion server in the same VNet of the AKS cluster that you created in **Step 2**. For details, see [Create a Bastion Server](CreateBastionServer.mdx). ## Step 7. Create a bastion server for ScalarDL Auditor To execute some tools for deploying and managing ScalarDL Auditor on AKS, you must prepare a bastion server in the same VNet of the AKS cluster that you created in **Step 3**. For details, see [Create a Bastion Server](CreateBastionServer.mdx). ## Step 8. Create network peering between two AKS clusters To make ScalarDL work properly, ScalarDL Ledger and ScalarDL Auditor need to connect to each other. You must connect two VNets by using [virtual network peering](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview). For details, see [Configure Network Peering for ScalarDL Auditor Mode](NetworkPeeringForScalarDLAuditor.mdx). ## Step 9. Prepare custom values files for the Scalar Helm Charts for both ScalarDL Ledger and ScalarDL Schema Loader To perform tasks, like accessing information in the database that you created in **Step 4**, you must configure custom values files for the Scalar Helm Charts for both ScalarDL Ledger and ScalarDL Schema Loader (for Ledger) based on your environment. For details, see [Configure a custom values file for Scalar Helm Charts](../helm-charts/configure-custom-values-file.mdx). ## Step 10. Deploy ScalarDL Ledger by using the Scalar Helm Chart Deploy ScalarDL Ledger on your AKS cluster by using the Helm Chart for ScalarDL Ledger. For details, see [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx). **Note:** We recommend creating a dedicated namespace by using the `kubectl create ns scalardl-ledger` command and deploying ScalarDL Ledger in the namespace by using the `-n scalardl-ledger` option with the `helm install` command. ## Step 11. Prepare custom values files for the Scalar Helm Charts for both ScalarDL Auditor and ScalarDL Schema Loader To perform tasks, like accessing information in the database that you created in **Step 5**, you must also configure a custom values files for the Scalar Helm Chart for both ScalarDL Auditor and ScalarDL Schema Loader (for Auditor) based on your environment. For details, see [Configure a custom values file for Scalar Helm Charts](../helm-charts/configure-custom-values-file.mdx). ## Step 12. Deploy ScalarDL Auditor by using the Scalar Helm Chart Deploy ScalarDL Auditor on your AKS cluster by using the Helm Chart for ScalarDL Auditor. For details, see [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx). **Note:** We recommend creating a dedicated namespace by using the `kubectl create ns scalardl-auditor` command and deploying ScalarDL Auditor in the namespace by using the `-n scalardl-auditor` option with the `helm install` command. ## Step 13. Check the status of your ScalarDL Ledger deployment After deploying ScalarDL Ledger in your AKS cluster, you must check the status of each component. For details, see [Components to Regularly Check When Running in a Kubernetes Environment](RegularCheck.mdx). ## Step 14. Check the status of your ScalarDL Auditor deployment After deploying ScalarDL Auditor in your AKS cluster, you must check the status of each component. For details, see [Components to Regularly Check When Running in a Kubernetes Environment](RegularCheck.mdx). ## Step 15. Monitor your ScalarDL Ledger deployment After deploying ScalarDL Ledger in your AKS cluster, we recommend monitoring the deployed components and collecting their logs, especially in production. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ## Step 16. Monitor your ScalarDL Auditor deployment After deploying ScalarDL Auditor in your AKS cluster, we recommend monitoring the deployed components and collecting their logs, especially in production. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ## Remove ScalarDL Ledger and ScalarDL Auditor from AKS If you want to remove the environment that you created, please remove all the resources in reverse order from which you created them in. ================================================ FILE: docs/scalar-kubernetes/ManualDeploymentGuideScalarDLAuditorOnEKS.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Deploy ScalarDL Ledger and ScalarDL Auditor on Amazon Elastic Kubernetes Service (EKS) This guide explains how to deploy ScalarDL Ledger and ScalarDL Auditor on Amazon Elastic Kubernetes Service (EKS). In this guide, you will create one of the following three environments in your AWS environment. To make Byzantine fault detection work properly, we recommend deploying ScalarDL Ledger and ScalarDL Auditor on different administrative domains (i.e., separate environments). * Use different AWS accounts (most recommended way) ![image](images/png/EKS_ScalarDL_Auditor_Multi_Account.drawio.png) * Use different Amazon Virtual Private Clouds (VPCs) (second recommended way) ![image](images/png/EKS_ScalarDL_Auditor_Multi_VPC.drawio.png) * Use different namespaces (third recommended way) ![image](images/png/EKS_ScalarDL_Auditor_Multi_Namespace.drawio.png) **Note:** This guide follows the second recommended way, "Use different VPCs." ## Step 1. Subscribe to ScalarDL Ledger and ScalarDL Auditor in AWS Marketplace You must get the ScalarDL Ledger and ScalarDL Auditor container images from [AWS Marketplace](https://aws.amazon.com/marketplace/seller-profile?id=bd4cd7de-49cd-433f-97ba-5cf71d76ec7b) and subscribe to ScalarDL Ledger and ScalarDL Auditor. For details on how to subscribe to ScalarDL Ledger and ScalarDL Auditor in AWS Marketplace, see [Subscribe to Scalar products from AWS Marketplace](AwsMarketplaceGuide.mdx#subscribe-to-scalar-products-from-aws-marketplace). ## Step 2. Create an EKS cluster for ScalarDL Ledger You must create an EKS cluster for the ScalarDL Ledger deployment. For details, see [Guidelines for creating an Amazon EKS cluster for Scalar products](CreateEKSClusterForScalarProducts.mdx). ## Step 3. Create an EKS cluster for ScalarDL Auditor You must also create an EKS cluster for the ScalarDL Auditor deployment. For details, see [Guidelines for creating an Amazon EKS cluster for Scalar products](CreateEKSClusterForScalarProducts.mdx). ## Step 4. Set up a database for ScalarDL Ledger You must prepare a database before deploying ScalarDL Ledger. Because ScalarDL Ledger uses ScalarDB internally to access databases, refer to [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases) to see which types of databases ScalarDB supports. For details on setting up a database, see [Set up a database for ScalarDB/ScalarDL deployment on AWS](SetupDatabaseForAWS.mdx). ## Step 5. Set up a database for ScalarDL Auditor You must also prepare a database before deploying ScalarDL Auditor. Because ScalarDL Auditor uses ScalarDB internally to access databases, refer to [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases) to see which types of databases ScalarDB supports. For details on setting up a database, see [Set up a database for ScalarDB/ScalarDL deployment on AWS](SetupDatabaseForAWS.mdx). ## Step 6. Create a bastion server for ScalarDL Ledger To execute some tools for deploying and managing ScalarDL Ledger on EKS, you must prepare a bastion server in the same VPC of the EKS cluster that you created in **Step 2**. For details, see [Create a Bastion Server](CreateBastionServer.mdx). ## Step 7. Create a bastion server for ScalarDL Auditor To execute some tools for deploying and managing ScalarDL Auditor on EKS, you must prepare a bastion server in the same VPC of the EKS cluster that you created in **Step 3**. For details, see [Create a Bastion Server](CreateBastionServer.mdx). ## Step 8. Create network peering between two EKS clusters To make ScalarDL work properly, ScalarDL Ledger and ScalarDL Auditor need to connect to each other. You must connect two VPCs by using [VPC peering](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html). For details, see [Configure network peering for ScalarDL Auditor mode](NetworkPeeringForScalarDLAuditor.mdx). ## Step 9. Prepare custom values files for the Scalar Helm Charts for ScalarDL Ledger and ScalarDL Schema Loader To perform tasks, like accessing information in the database that you created in **Step 4**, you must configure custom values files for the Scalar Helm Charts for ScalarDL Ledger and ScalarDL Schema Loader (for Ledger) based on your environment. For details, see [Configure a custom values file for Scalar Helm Charts](../helm-charts/configure-custom-values-file.mdx). ## Step 10. Deploy ScalarDL Ledger by using the Scalar Helm Chart Deploy ScalarDL Ledger in your EKS cluster by using the Helm Chart for ScalarDL Ledger. For details, see [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx). **Note:** We recommend creating a dedicated namespace by using the `kubectl create ns scalardl-ledger` command and deploying ScalarDL Ledger in the namespace by using the `-n scalardl-ledger` option with the `helm install` command. ## Step 11. Prepare custom values files for the Scalar Helm Charts for both ScalarDL Auditor and ScalarDL Schema Loader To perform tasks, like accessing information in the database that you created in **Step 5**, you must configure custom values files for the Scalar Helm Charts for both ScalarDL Auditor and ScalarDL Schema Loader (for Auditor) based on your environment. For details, see [Configure a custom values file for Scalar Helm Charts](../helm-charts/configure-custom-values-file.mdx). ## Step 12. Deploy ScalarDL Auditor by using the Scalar Helm Chart Deploy ScalarDL Auditor in your EKS cluster by using the Helm Chart for ScalarDL Auditor. For details , see [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx). **Note:** We recommend creating a dedicated namespace by using the `kubectl create ns scalardl-auditor` command and deploying ScalarDL Auditor in the namespace by using the `-n scalardl-auditor` option with the `helm install` command. ## Step 13. Check the status of your ScalarDL Ledger deployment After deploying ScalarDL Ledger in your EKS cluster, you must check the status of each component. For details, see [Components to Regularly Check When Running in a Kubernetes Environment](RegularCheck.mdx) for more details. ## Step 14. Check the status of your ScalarDL Auditor deployment After deploying ScalarDL Auditor on your EKS cluster, you need to check the status of each component. See [Components to Regularly Check When Running in a Kubernetes Environment](RegularCheck.mdx) for more details. ## Step 15. Monitor your ScalarDL Ledger deployment After deploying ScalarDL Ledger in your EKS cluster, we recommend monitoring the deployed components and collecting their logs, especially in production. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ## Step 16. Monitor your ScalarDL Auditor deployment After deploying ScalarDL Auditor in your EKS cluster, we recommend monitoring the deployed components and collecting their logs, especially in production. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ## Remove ScalarDL Ledger and ScalarDL Auditor from EKS If you want to remove the environment you created, please remove all the resources in reverse order from which you created them in. ================================================ FILE: docs/scalar-kubernetes/ManualDeploymentGuideScalarDLOnAKS.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Deploy ScalarDL Ledger on Azure Kubernetes Service (AKS) This document explains how to deploy **ScalarDL Ledger** on Azure Kubernetes Service (AKS). In this guide, you will create the following environment in your Azure environment. ![image](images/png/AKS_ScalarDL_Ledger.drawio.png) ## Step 1. Get the ScalarDL Ledger container image You must get the ScalarDL Ledger container image. For details, see [How to get the container images of Scalar products](HowToGetContainerImages.mdx). ## Step 2. Create an AKS cluster You must create an AKS cluster for the ScalarDL Ledger deployment. For details, see [Guidelines for creating an AKS cluster for Scalar products](CreateAKSClusterForScalarProducts.mdx). ## Step 3. Set up a database for ScalarDL Ledger You must prepare a database before deploying ScalarDL Ledger. Because ScalarDL Ledger uses ScalarDB internally to access databases, refer to [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases) to see which types of databases ScalarDB supports. For details on setting up a database, see [Set up a database for ScalarDB/ScalarDL deployment in Azure](SetupDatabaseForAzure.mdx). ## Step 4. Create a bastion server To execute some tools for deploying and managing ScalarDL Ledger on AKS, you must prepare a bastion server in the same Azure Virtual Network (VNet) of the AKS cluster that you created in **Step 2**. For details, see [Create a Bastion Server](CreateBastionServer.mdx). ## Step 5. Prepare custom values files for the Scalar Helm Charts for both ScalarDL Ledger and ScalarDL Schema Loader To perform tasks, like accessing information in the database that you created in **Step 3**, you must configure custom values files for the Scalar Helm Charts for both ScalarDL Ledger and ScalarDL Schema Loader (for Ledger) based on your environment. For details, see [Configure a custom values file for Scalar Helm Charts](../helm-charts/configure-custom-values-file.mdx). ## Step 6. Deploy ScalarDL Ledger by using the Scalar Helm Chart Deploy ScalarDL Ledger in your AKS cluster by using the Helm Chart for ScalarDL Ledger. For details, see [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx). **Note:** We recommend creating a dedicated namespace by using the `kubectl create ns scalardl-ledger` command and deploying ScalarDL Ledger in the namespace by using the `-n scalardl-ledger` option with the `helm install` command. ## Step 7. Check the status your ScalarDL Ledger deployment After deploying ScalarDL Ledger in your AKS cluster, you must check the status of each component. For details, see [Components to Regularly Check When Running in a Kubernetes Environment](RegularCheck.mdx). ## Step 8. Monitor your ScalarDL Ledger deployment After deploying ScalarDL Ledger in your AKS cluster, we recommend monitoring the deployed components and collecting their logs, especially in production. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ## Remove ScalarDL Ledger from AKS If you want to remove the environment that you created, please remove all the resources in reverse order from which you created them in. ================================================ FILE: docs/scalar-kubernetes/ManualDeploymentGuideScalarDLOnEKS.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Deploy ScalarDL Ledger on Amazon Elastic Kubernetes Service (EKS) This document explains how to deploy **ScalarDL Ledger** on Amazon Elastic Kubernetes Service (EKS). In this guide, you will create the following environment in your AWS environment account. ![image](images/png/EKS_ScalarDL_Ledger.drawio.png) ## Step 1. Subscribe to ScalarDL Ledger in AWS Marketplace You must get the ScalarDL Ledger container image from [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-3jdwfmqonx7a2) and subscribe to ScalarDL. For details on how to subscribe to ScalarDL Ledger in AWS Marketplace, see [Subscribe to Scalar products from AWS Marketplace](AwsMarketplaceGuide.mdx#subscribe-to-scalar-products-from-aws-marketplace). ## Step 2. Create an EKS cluster You must create an EKS cluster for the ScalarDL Ledger deployment. For details, see [Guidelines for creating an Amazon EKS cluster for Scalar products](CreateEKSClusterForScalarProducts.mdx). ## Step 3. Set up a database for ScalarDL Ledger You must prepare a database before deploying ScalarDL Ledger. Because ScalarDL Ledger uses ScalarDB internally to access databases, refer to [ScalarDB Supported Databases](https://scalardb.scalar-labs.com/docs/latest/requirements#databases) to see which types of databases ScalarDB supports. For details on setting up a database, see [Set up a database for ScalarDB/ScalarDL deployment on AWS](SetupDatabaseForAWS.mdx). ## Step 4. Create a bastion server To execute some tools for deploying and managing ScalarDL Ledger on EKS, you must prepare a bastion server in the same Amazon Virtual Private Cloud (VPC) of the EKS cluster you created in **Step 2**. For details, see [Create a Bastion Server](CreateBastionServer.mdx). ## Step 5. Prepare custom values files for the Scalar Helm Charts for both ScalarDL Ledger and ScalarDL Schema Loader To perform tasks, like accessing information in the database that you created in **Step 3**, you must configure custom values files for the Scalar Helm Charts for both ScalarDL Ledger and ScalarDL Schema Loader (for Ledger) based on your environment. For details, see [Configure a custom values file for Scalar Helm Charts](../helm-charts/configure-custom-values-file.mdx). ## Step 6. Deploy ScalarDL Ledger by using the Scalar Helm Chart Deploy ScalarDL Ledger in your EKS cluster by using the Helm Chart for ScalarDL Ledger. For details, see [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx). **Note:** We recommend creating a dedicated namespace by using the `kubectl create ns scalardl-ledger` command and deploying ScalarDL Ledger in the namespace by using the `-n scalardl-ledger` option with the `helm install` command. ## Step 7. Check the status of your ScalarDL Ledger deployment After deploying ScalarDL Ledger in your EKS cluster, you must check the status of each component. For details, see [Components to Regularly Check When Running in a Kubernetes Environment](RegularCheck.mdx). ## Step 8. Monitor your ScalarDL Ledger deployment After deploying ScalarDL Ledger in your EKS cluster, we recommend monitoring the deployed components and collecting their logs, especially in production. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ## Remove ScalarDL Ledger from EKS If you want to remove the environment that you created, please remove all the resources in reverse order from which you created them in. ================================================ FILE: docs/scalar-kubernetes/NetworkPeeringForScalarDLAuditor.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Configure Network Peering for ScalarDL Auditor Mode This document explains how to connect multiple private networks for ScalarDL Auditor mode to perform network peering. For ScalarDL Auditor mode to work properly, you must connect ScalarDL Ledger to ScalarDL Auditor. ## What network you must connect To make ScalarDL Auditor mode (Byzantine fault detection) work properly, you must connect three private networks. * [ScalarDL Ledger network] ↔ [ScalarDL Auditor network] * [ScalarDL Ledger network] ↔ [application (client) network] * [ScalarDL Auditor network] ↔ [application (client) network] ## Network requirements ### IP address ranges To avoid conflicting IP addresses between the private networks, you must have private networks with different IP address ranges. For example: * **Private network for ScalarDL Ledger:** 10.1.0.0/16 * **Private network for ScalarDL Auditor:** 10.2.0.0/16 * **Private network for application (client):** 10.3.0.0/16 ### Connections The default network ports for connecting ScalarDL Ledger, ScalarDL Auditor, and the application (client) by default are as follows. You must allow these connections between each private network. * **ScalarDL Ledger** * **50051/TCP:** Accept requests from an application (client) and ScalarDL Auditor via Scalar Envoy. * **50052/TCP:** Accept privileged requests from an application (client) and ScalarDL Auditor via Scalar Envoy. * **ScalarDL Auditor** * **40051/TCP:** Accept requests from an application (client) and ScalarDL Ledger via Scalar Envoy. * **40052/TCP:** Accept privileged requests from an application (client) and ScalarDL Ledger via Scalar Envoy. * **Scalar Envoy** (used with ScalarDL Ledger and ScalarDL Auditor) * **50051/TCP:** Accept requests for ScalarDL Ledger from an application (client) and ScalarDL Auditor. * **50052/TCP:** Accept privileged requests for ScalarDL Ledger from an application (client) and ScalarDL Auditor. * **40051/TCP:** Accept requests for ScalarDL Auditor from an application (client) and ScalarDL Ledger. * **40052/TCP:** Accept privileged requests for ScalarDL Auditor from an application (client) and ScalarDL Ledger. Note that, if you change the listening port for ScalarDL in the configuration file (ledger.properties or auditor.properties) from the default, you must allow the connections by using the port that you configured. ## Private-network peering For details on how to connect private networks in each cloud, see official documents. ### Amazon VPC peering For details on how to peer virtual private clouds (VPCs) in an Amazon Web Services (AWS) environment, see the official documentation from Amazon at [Create a VPC peering connection](https://docs.aws.amazon.com/vpc/latest/peering/create-vpc-peering-connection.html). ### Azure VNet peering For details on how to peer virtual networks in an Azure environment, see the official documentation from Microsoft at [Virtual network peering](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview). ================================================ FILE: docs/scalar-kubernetes/ProductionChecklistForScalarDBCluster.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Production checklist for ScalarDB Cluster This checklist provides recommendations when deploying ScalarDB Cluster in a production environment. ## Before you begin In this checklist, we assume that you are deploying ScalarDB Cluster on a managed Kubernetes cluster, which is recommended. ## Production checklist: ScalarDB Cluster The following is a checklist of recommendations when setting up ScalarDB Cluster in a production environment. ### Number of pods and Kubernetes worker nodes To ensure that the Kubernetes cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardb-cluster-custom-values-indirect-mode.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different availability zones (AZs), you can withstand an AZ failure. ::: ### Worker node specifications It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition, some pods other than ScalarDB Cluster pods exist on the worker nodes. In other words, the following components could run on one worker node: * ScalarDB Cluster pod (2vCPU / 4GB) * Envoy proxy (if you use `indirect` client mode or use a programming language other than Java) * Your application pods (if you choose to run your application's pods on the same worker node) * Monitoring components (if you deploy monitoring components such `kube-prometheus-stack`) * Kubernetes components :::note You do not need to deploy an Envoy pod when using `direct-kubernetes` mode. ::: With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [Number of pods and Kubernetes worker nodes](ProductionChecklistForScalarDBCluster.mdx#number-of-pods-and-kubernetes-worker-nodes). However, three nodes with at least 4vCPU / 8GB memory resources per node is the minimum for a production environment. You should also consider the resources of the Kubernetes cluster (for example, the number of worker nodes, vCPUs per node, memories per node, ScalarDB Cluster pods, and pods for your application), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node to decide on the worker node resources. ### Network You should create the Kubernetes cluster on a private network since ScalarDB Cluster does not provide any services to users directly via internet access. We recommend accessing ScalarDB Cluster via a private network from your applications. ### Monitoring and logging You should monitor the deployed components and collect their logs. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ### Backup and restore You should enable the automatic backup feature and point-in-time recovery (PITR) feature in the backend database. For details, see [Set up a database for ScalarDB/ScalarDL deployment](SetupDatabase.mdx). ## Production checklist: Client applications that access ScalarDB Cluster The following is a checklist of recommendations when setting up a client application that accesses ScalarDB Cluster in a production environment. ### Client mode (Java client library only) When using Java for your application, you can use an official Java client library. In this case, you can choose one of the two client modes: [`direct-kubernetes mode`](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api#direct-kubernetes-client-mode) or [`indirect mode`](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api#indirect-client-mode). From the perspective of performance, we recommend using `direct-kubernetes` mode. To use `direct-kubernetes` mode, you must deploy your application pods on the same Kubernetes cluster as ScalarDB Cluster pods. In this case, you don't need to deploy Envoy pods. If you can't deploy your Java application pods on the same Kubernetes cluster as ScalarDB Cluster pods for some reason, you must use `indirect` mode. In this case, you must deploy Envoy pods. :::note The client mode configuration is dedicated to the Java client library. If you use a programming language other than Java for your application (essentially, if you use the [gRPC API](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/scalardb-cluster-grpc-api-guide) or [gRPC SQL API](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/scalardb-cluster-sql-grpc-api-guide) directly from the programming language), no such configuration exists. In this case, you must deploy Envoy pods. ::: ### Transaction manager configuration (Java client library only) The client application must always access the database through ScalarDB Cluster. To ensure requests are running properly, check the properties file for your client application and confirm that `scalar.db.transaction_manager=cluster` is configured when using the CRUD API. #### Recommended for production environments ```mermaid flowchart LR app["App
ScalarDB Cluster Library with gRPC"] server["ScalarDB Cluster
ScalarDB Library with
Consensus Commit"] db[(Underlying storage or database)] app --> server --> db ``` #### Not recommended for production environments (for testing purposes only) ```mermaid flowchart LR app["App
ScalarDB Cluster Library with
Consensus Commit"] db[(Underlying storage or database)] app --> db ``` ### SQL connection configuration (Java client library only) The client application must always access the database through ScalarDB Cluster. To ensure requests are running properly, check the properties file for your client application and confirm that `scalar.db.sql.connection_mode=cluster` is configured when using the SQL API. #### Recommended for production environments ```mermaid flowchart LR app["App
ScalarDB SQL Library (Cluster mode)"] server["ScalarDB Cluster
ScalarDB Library with
Consensus Commit"] db[(Underlying storage or database)] app --> server --> db ``` #### Not recommended for production environments (for testing purposes only) ```mermaid flowchart LR app["App
ScalarDB SQL Library (Direct mode)"] db[(Underlying storage or database)] app --> db ``` ### Deployment of the client application when using `direct-kubernetes` client mode (Java client library only) If you use [`direct-kubernetes` client mode](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api#direct-kubernetes-client-mode), you must deploy your client application on the same Kubernetes cluster as the ScalarDB Cluster deployment. Also, when using `direct-kubernetes` client mode, you must deploy additional Kubernetes resources to make your client application work properly. For details, see [Deploy your client application on Kubernetes with `direct-kubernetes` mode](../helm-charts/how-to-deploy-scalardb-cluster.mdx#deploy-your-client-application-on-kubernetes-with-direct-kubernetes-mode). ### Transaction handling (Java client library and gRPC API) You must make sure that your application always runs [`commit()`](https://scalardb.scalar-labs.com/docs/latest/api-guide#commit-a-transaction) or [`rollback()`](https://scalardb.scalar-labs.com/docs/latest/api-guide#roll-back-or-abort-a-transaction) after you [`begin()`](https://scalardb.scalar-labs.com/docs/latest/api-guide#begin-or-start-a-transaction) a transaction. If the application does not run `commit()` or `rollback()`, your application might experience unexpected issues or read inconsistent data from the backend database. :::note If you use the [gRPC API](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/scalardb-cluster-grpc-api-guide) or [SQL gRPC API](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/scalardb-cluster-sql-grpc-api-guide), your application should call a `Commit` or `Rollback` service after you call a `Begin` service to begin a transaction. ::: ### Exception handling (Java client library and gRPC API) You must make sure that your application handles transaction exceptions. For details, see the document for the API that you are using: * [Handle exceptions (Transactional API)](https://scalardb.scalar-labs.com/docs/latest/api-guide#handle-exceptions). * [Handle exceptions (two-phase commit transactions API)](https://scalardb.scalar-labs.com/docs/latest/two-phase-commit-transactions#handle-exceptions) * [Execute transactions (ScalarDB SQL API)](https://scalardb.scalar-labs.com/docs/latest/scalardb-sql/sql-api-guide#execute-transactions) * [Handle SQLException (ScalarDB JDBC)](https://scalardb.scalar-labs.com/docs/latest/scalardb-sql/jdbc-guide#handle-sqlexception) * [Error handling (ScalarDB Cluster gRPC API)](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/scalardb-cluster-grpc-api-guide#error-handling-1) * [Error handling (ScalarDB Cluster SQL gRPC API)](https://scalardb.scalar-labs.com/docs/latest/scalardb-cluster/scalardb-cluster-sql-grpc-api-guide#error-handling-1) ================================================ FILE: docs/scalar-kubernetes/ProductionChecklistForScalarDLAuditor.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Production checklist for ScalarDL Auditor This checklist provides recommendations when deploying ScalarDL Auditor in a production environment. ## Before you begin In this checklist, we assume that you are deploying ScalarDL Auditor on a managed Kubernetes cluster, which is recommended. ## Production checklist: ScalarDL Auditor The following is a checklist of recommendations when setting up ScalarDL Auditor in a production environment. ### ScalarDL availability To ensure that the Kubernetes cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardl-audit-custom-values.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different availability zones (AZs), you can withstand an AZ failure. ::: ### Resources It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition to the ScalarDL Auditor pod, Kubernetes could deploy some of the following components to each worker node: * ScalarDL Auditor pod (2vCPU / 4GB) * Envoy proxy * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [ScalarDL availability](#scalardl-availability). However, three nodes with at least 4vCPU / 8GB memory resources per node is the minimum environment for production. You should also consider the resources of the Kubernetes cluster (for example, the number of worker nodes, vCPUs per node, memory per node, and ScalarDL Auditor pods), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node when deciding the worker node resources. ### Network You should create the Kubernetes cluster on a private network since ScalarDL Auditor does not provide any services to users directly via internet access. We recommend accessing ScalarDL Auditor via a private network from your applications. ### Monitoring and logging You should monitor the deployed components and collect their logs. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ### Backup and restore You should enable the automatic backup feature and point-in-time recovery (PITR) feature in the backend database. For details, see [Set up a database for ScalarDB/ScalarDL deployment](SetupDatabase.mdx). ### ScalarDL Auditor deployment For Byzantine fault detection in ScalarDL to work properly, do not deploy ScalarDL Auditor pods on the same Kubernetes clusters as the ScalarDL Ledger deployment. Instead, you must deploy ScalarDL Auditor pods in an environment other than the administrative domain (other than the Kubernetes cluster) for the ScalarDL Ledger deployment. #### Required for production environments ```mermaid graph LR subgraph "ScalarDL" subgraph "Administrative domain 1" subgraph "Kubernetes cluster for Ledger" B-1[ScalarDL Ledger] end end subgraph "Administrative domain 2" subgraph "Kubernetes cluster for Auditor" C-1[ScalarDL Auditor] end end end ``` #### Not recommended for production environments (for testing purposes only) ```mermaid graph LR subgraph "Kubernetes cluster" direction LR A-1[ScalarDL Ledger] A-2[ScalarDL Auditor] end ``` ### Connection between ScalarDL Ledger and ScalarDL Auditor For ScalarDL Auditor mode to work properly, you must allow the connection between ScalarDL Ledger and ScalarDL Auditor. ```mermaid graph LR subgraph "Kubernetes cluster for Ledger" A-1[ScalarDL Ledger] end subgraph "Kubernetes cluster for Auditor" B-1[ScalarDL Auditor] end A-1 --- B-1 ``` ScalarDL uses the following ports for the connections between ScalarDL Ledger and ScalarDL Auditor. You must allow these connections between ScalarDL Ledger and ScalarDL Auditor: * ScalarDL Ledger * 50051/TCP * 50052/TCP * ScalarDL Auditor * 40051/TCP * 40052/TCP ### Private key and certificate When you use PKI for authentication, you must make sure that private keys and certificates that you register to ScalarDL Ledger and ScalaDL Auditor match the following requirements: ```console Algorithm : ECDSA Hash function : SHA256 Curve parameter : P-256 ``` For details, see [How to get a certificate](https://scalardl.scalar-labs.com/docs/latest/ca/caclient-getting-started). ## Production checklist: Client applications that access ScalarDL Auditor The following is a checklist of recommendations when setting up a client application that accesses ScalarDL Auditor in a production environment. ### Client application deployment For Byzantine fault detection in ScalarDL to work properly, do not deploy your application pods on the same Kubernetes clusters as the ScalarDL deployment. Instead, you must deploy your application in an environment other than the administrative domain (other than the Kubernetes cluster) for the ScalarDL deployment. #### Required for production environments ```mermaid graph LR subgraph "Administrative domain 1" subgraph "Another environment" A-1[User application] end end subgraph "ScalarDL" subgraph "Administrative domain 2" subgraph "Kubernetes cluster for Ledger" B-1[ScalarDL Ledger] end end subgraph "Administrative domain 3" subgraph "Kubernetes cluster for Auditor" C-1[ScalarDL Auditor] end end end A-1 --> B-1 A-1 --> C-1 ``` #### Not recommended for production environments (for testing purposes only) ```mermaid graph LR subgraph "Kubernetes cluster" direction LR A-1[User application] A-2[ScalarDL Ledger] A-3[ScalarDL Auditor] end A-1 --> A-2 A-1 --> A-3 ``` ### Client application checklist You must also make sure that you satisfy the [Production checklist: Client applications that access ScalarDL Ledger](ProductionChecklistForScalarDLLedger.mdx#production-checklist-client-applications-that-access-scalardl-ledger). ================================================ FILE: docs/scalar-kubernetes/ProductionChecklistForScalarDLLedger.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Production checklist for ScalarDL Ledger This checklist provides recommendations when deploying ScalarDL Ledger in a production environment. ## Before you begin In this checklist, we assume that you are deploying ScalarDL Ledger on a managed Kubernetes cluster, which is recommended. ## Production checklist: ScalarDL Ledger The following is a checklist of recommendations when setting up ScalarDL Ledger in a production environment. ### ScalarDL availability To ensure that the Kubernetes cluster has high availability, you should use at least three worker nodes and deploy at least three pods spread across the worker nodes. You can see the [sample configurations](https://github.com/scalar-labs/scalar-kubernetes/blob/master/conf/scalardl-custom-values.yaml) of `podAntiAffinity` for making three pods spread across the worker nodes. :::note If you place the worker nodes in different availability zones (AZs), you can withstand an AZ failure. ::: ### Resources It is recommended to set at least 2vCPU / 4GB memory if you use the bring-your-own-license (BYOL) containers. In addition to the ScalarDL Ledger pod, Kubernetes could deploy some of the following components to each worker node: * ScalarDL Ledger pod (2vCPU / 4GB) * Envoy proxy * Monitoring components (if you deploy monitoring components such as `kube-prometheus-stack`) * Kubernetes components With this in mind, you should use a worker node that has at least 4vCPU / 8GB memory resources and use at least three worker nodes for availability, as mentioned in [ScalarDL availability](#scalardl-availability). However, three nodes with at least 4vCPU / 8GB memory resources per node is the minimum environment for production. You should also consider the resources of the Kubernetes cluster (for example, the number of worker nodes, vCPUs per node, memory per node, and ScalarDL Ledger pods), which depend on your system's workload. In addition, if you plan to scale the pods automatically by using some features like [Horizontal Pod Autoscaling (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/), you should consider the maximum number of pods on the worker node when deciding the worker node resources. ### Network You should create the Kubernetes cluster on a private network since ScalarDL Ledger does not provide any services to users directly via internet access. We recommend accessing ScalarDL Ledger via a private network from your applications. ### Monitoring and logging You should monitor the deployed components and collect their logs. For details, see [Monitoring Scalar products on a Kubernetes cluster](K8sMonitorGuide.mdx) and [Collecting logs from Scalar products on a Kubernetes cluster](K8sLogCollectionGuide.mdx). ### Backup and restore You should enable the automatic backup feature and point-in-time recovery (PITR) feature in the backend database. For details, see [Set up a database for ScalarDB/ScalarDL deployment](SetupDatabase.mdx). ## Production checklist: Client applications that access ScalarDL Ledger The following is a checklist of recommendations when setting up a client application that accesses ScalarDL Ledger in a production environment. ### Client application deployment For Byzantine fault detection in ScalarDL to work properly, do not deploy your application pods on the same Kubernetes clusters as the ScalarDL Ledger deployment. Instead, you must deploy your application in an environment other than the administrative domain (other than the Kubernetes cluster) for the ScalarDL Ledger deployment. #### Required for production environments ```mermaid graph LR subgraph "Administrative domain 1" subgraph "Another environment" A-1[User application] end end subgraph "Administrative domain 2" subgraph "Kubernetes cluster" B-1[ScalarDL Ledger] end end A-1 --> B-1 ``` #### Not recommended for production environments (for testing purposes only) ```mermaid graph LR subgraph "Kubernetes cluster" direction LR A-1[User application] --> A-2[ScalarDL Ledger] end ``` ### Contract and function To check if your contract and function follow the guidelines, see the following: * [A Guide on How to Write a Good Contract for ScalarDL](https://scalardl.scalar-labs.com/docs/latest/how-to-write-contract) * [A Guide on How to Write Function for ScalarDL](https://scalardl.scalar-labs.com/docs/latest/how-to-write-function) ### Contract versioning After you register a contract, you cannot overwrite that existing contract. So, you should consider the versioning of contracts. We recommend one of the following two methods. #### Versioning by using `Class Name` ```console Contract ID : FooV1 Binary Name : com.example.contract.FooV1 Class file (Class Name) : src/main/java/com/example/contract/FooV1.class --- Contract ID : FooV2 Binary Name : com.example.contract.FooV2 Class file (Class Name) : src/main/java/com/example/contract/FooV2.class ``` #### Versioning by using `Package Name` ```console Contract ID : FooV3 Binary Name : com.example.contract.v3.Foo Class file (Class Name) : src/main/java/com/example/contract/v3/Foo.class --- Contract ID : FooV4 Binary Name : com.example.contract.v4.Foo Class file (Class Name) : src/main/java/com/example/contract/v4/Foo.class ``` ### Contract limitations If the binary name, package name, and class name are different when you register the contract, you cannot execute that contract after registering it. #### Binary name and class name are different (you cannot execute this contract) ```console Contract ID : FooV5 Binary Name : com.example.contract.FooV5 Class file (Class Name) : src/main/java/com/example/contract/FooV6.class ``` #### Binary name and package name are different (you cannot execute this contract) ```console Contract ID : FooV7 Binary Name : com.example.contract.v7.Foo Class file (Class Name) : src/main/java/com/example/contract/v8/Foo.class ``` ### Private key and certificate When you use PKI for authentication, you must make sure that private keys and certificates that you register to ScalarDL Ledger match the following requirements: ```console Algorithm : ECDSA Hash function : SHA256 Curve parameter : P-256 ``` For details, see [How to get a certificate](https://scalardl.scalar-labs.com/docs/latest/ca/caclient-getting-started). ### Exception handling You must make sure that your application handles exceptions. For details, see [A Guide on How to Handle Errors in ScalarDL](https://scalardl.scalar-labs.com/docs/latest/how-to-write-applications#handle-errors). ================================================ FILE: docs/scalar-kubernetes/ProductionChecklistForScalarProducts.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Production checklist for Scalar products To make your deployment ready for production, refer to the following: * [Production checklist for ScalarDB Cluster](ProductionChecklistForScalarDBCluster.mdx) * [Production checklist for ScalarDL Ledger](ProductionChecklistForScalarDLLedger.mdx) * [Production checklist for ScalarDL Auditor](ProductionChecklistForScalarDLAuditor.mdx) ================================================ FILE: docs/scalar-kubernetes/RegularCheck.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Components to Regularly Check When Running in a Kubernetes Environment Most of the components deployed by manual deployment guides are self-healing with the help of the managed Kubernetes services and Kubernetes self-healing capability. There are also configured alerts that occur when some unexpected behavior happens. Thus, there shouldn't be so many things to do day by day for the deployment of Scalar products on the managed Kubernetes cluster. However, it is recommended to check the status of a system on a regular basis to see if everything is working fine. Here is the list of things you might want to do on a regular basis. ## Kubernetes resources ### Check if Pods are all healthy statues Please check the Kubernetes namespaces: * `default` (or specified namespace when you deploy Scalar products) for the Scalar product deployment * `monitoring` for the Prometheus Operator and Loki What to check: * `STATUS` is all `Running` * Pods are evenly distributed on the different nodes ```console kubectl get pod -o wide -n ``` You should see the following output: ```console NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES scalardb-7876f595bd-2jb28 1/1 Running 0 2m35s 10.244.2.6 k8s-worker2 scalardb-7876f595bd-rfvk6 1/1 Running 0 2m35s 10.244.1.8 k8s-worker scalardb-7876f595bd-xfkv4 1/1 Running 0 2m35s 10.244.3.8 k8s-worker3 scalardb-envoy-84c475f77b-cflkn 1/1 Running 0 2m35s 10.244.1.7 k8s-worker scalardb-envoy-84c475f77b-tzmc9 1/1 Running 0 2m35s 10.244.3.7 k8s-worker3 scalardb-envoy-84c475f77b-vztqr 1/1 Running 0 2m35s 10.244.2.5 k8s-worker2 ``` ```console kubectl get pod -n monitoring -o wide ``` You should see the following output: ```console NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES alertmanager-scalar-monitoring-kube-pro-alertmanager-0 2/2 Running 1 (11m ago) 12m 10.244.2.4 k8s-worker2 prometheus-scalar-monitoring-kube-pro-prometheus-0 2/2 Running 0 12m 10.244.1.5 k8s-worker scalar-logging-loki-0 1/1 Running 0 13m 10.244.2.2 k8s-worker2 scalar-logging-loki-promtail-2c4k9 0/1 Running 0 13m 10.244.0.5 k8s-control-plane scalar-logging-loki-promtail-8r48b 1/1 Running 0 13m 10.244.3.2 k8s-worker3 scalar-logging-loki-promtail-b26c6 1/1 Running 0 13m 10.244.2.3 k8s-worker2 scalar-logging-loki-promtail-sks56 1/1 Running 0 13m 10.244.1.2 k8s-worker scalar-monitoring-grafana-77c4dbdd85-4mrn7 3/3 Running 0 12m 10.244.3.4 k8s-worker3 scalar-monitoring-kube-pro-operator-7575dd8bbd-bxhrc 1/1 Running 0 12m 10.244.1.3 k8s-worker ``` ### Check if Nodes are all healthy statuses What to check: * `STATUS` is all `Ready` ```console kubectl get nodes ``` You should see the following output: ```console NAME STATUS ROLES AGE VERSION k8s-control-plane Ready control-plane 16m v1.25.3 k8s-worker Ready 15m v1.25.3 k8s-worker2 Ready 15m v1.25.3 k8s-worker3 Ready 15m v1.25.3 ``` ## Prometheus dashboard (Alerts of Scalar products) Access to the Prometheus dashboard according to the document [Monitoring Scalar products on the Kubernetes cluster](K8sMonitorGuide.mdx). In the **Alerts** tab, you can see the alert status. What to check: * All alerts are **green (Inactive)** If some issue is occurring, it shows you **red (Firing)** status. ## Grafana dashboard (metrics of Scalar products) Access to the Grafana dashboard according to the document [Monitoring Scalar products on the Kubernetes cluster](K8sMonitorGuide.mdx). In the **Dashboards** tab, you can see the dashboard of Scalar products. In these dashboards, you can see some metrics of Scalar products. Those dashboards cannot address issues directly, but you can see changes from normal (e.g., increasing transaction errors) to get hints for investigating issues. ================================================ FILE: docs/scalar-kubernetes/RestoreDatabase.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Restore databases in a Kubernetes environment This guide explains how to restore databases that ScalarDB or ScalarDL uses in a Kubernetes environment. Please note that this guide assumes that you are using a managed database from a cloud services provider as the backend database for ScalarDB or ScalarDL. ## Procedure to restore databases 1. Scale in ScalarDB or ScalarDL pods to **0** to stop requests to the backend databases. You can scale in the pods to **0** by using the `--set *.replicaCount=0` flag in the helm command. * ScalarDB Server ```console helm upgrade scalar-labs/scalardb -n -f /path/to/ --set scalardb.replicaCount=0 ``` * ScalarDL Ledger ```console helm upgrade scalar-labs/scalardl -n -f /path/to/ --set ledger.replicaCount=0 ``` * ScalarDL Auditor ```console helm upgrade scalar-labs/scalardl-audit -n -f /path/to/ --set auditor.replicaCount=0 ``` 2. Restore the databases by using the point-in-time recovery (PITR) feature. For details on how to restore the databases based on your managed database, please refer to the [Supplemental procedures to restore databases based on managed database](RestoreDatabase.mdx#supplemental-procedures-to-restore-databases-based-on-managed-database) section in this guide. If you are using NoSQL or multiple databases, you should specify the middle point of the pause duration period that you created when following the backup procedure in [Back up a NoSQL database in a Kubernetes environment](BackupNoSQL.mdx). 3. Update **database.properties**, **ledger.properties**, or **auditor.properties** based on the newly restored database. Because the PITR feature restores databases as another instance, you must update the endpoint information in the custom values file of ScalarDB or ScalarDL to access the newly restored databases. For details on how to configure the custom values file, see [Configure a custom values file for Scalar Helm Charts](../helm-charts/configure-custom-values-file.mdx). Please note that, if you are using Amazon DynamoDB, your data will be restored with another table name instead of another instance. In other words, the endpoint will not change after restoring the data. Instead, you will need to restore the data by renaming the tables in Amazon DynamoDB. For details on how to restore data with the same table name, please see the [Amazon DynamoDB](RestoreDatabase.mdx#amazon-dynamodb) section in this guide. 4. Scale out the ScalarDB or ScalarDL pods to **1** or more to start accepting requests from clients by using the `--set *.replicaCount=N` flag in the helm command. * ScalarDB Server ```console helm upgrade scalar-labs/scalardb -n -f /path/to/ --set scalardb.replicaCount=3 ``` * ScalarDL Ledger ```console helm upgrade scalar-labs/scalardl -n -f /path/to/ --set ledger.replicaCount=3 ``` * ScalarDL Auditor ```console helm upgrade scalar-labs/scalardl-audit -n -f /path/to/ --set auditor.replicaCount=3 ``` ## Supplemental procedures to restore data based on managed database ### Amazon DynamoDB When using the PITR feature, Amazon DynamoDB restores data with another table name. Therefore, you must follow additional steps to restore data with the same table name. #### Steps 1. Create a backup. 1. Select the middle point of the pause duration period as the restore point. 2. Use PITR to restore table A to table B. 3. Perform a backup of the restored table B. Then, confirm the backup is named appropriately for backup B. 4. Remove table B. For details on how to restore DynamoDB tables by using PITR and how to perform a backup of DynamoDB tables manually, see the following official documentation from Amazon: * [Restoring a DynamoDB table to a point in time](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/PointInTimeRecovery.Tutorial.html) * [Backing up a DynamoDB table](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Backup.Tutorial.html) You can do this **Create a backup** step as a part of backup operations in the [Back up a NoSQL database in a Kubernetes environment](BackupNoSQL.mdx#create-a-period-to-restore-data-and-perform-a-backup). 2. Restore from the backup. 1. Remove table A. 2. Create a table named A by using backup B. 3. Update the table configuration if necessary, depending on your environment. Some configurations, like autoscaling policies, are not set after restoring, so you may need to manually set those configurations depending on your needs. For details, see the official documentation from Amazon at [Backing up and restoring DynamoDB tables with DynamoDB: How it works](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/CreateBackup.html). For example, if you are using ScalarDB Schema Loader or ScalarDL Schema Loader to create tables, autoscaling is enabled by default. Therefore, you will need to manually enable autoscaling for the restored tables in DynamoDB. For details on how to enable autoscaling in DynamoDB, see the official documentation from Amazon at [Enabling DynamoDB auto scaling on existing tables](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.Console.html#AutoScaling.Console.ExistingTable). In addition, after restoring the databases, the PITR feature will be disabled and the read/write capacity mode is reset to the default value. If necessary, depending on your environment, you will need to manually set these configurations. For some configurations for restored tables, see [Set up a database for ScalarDB/ScalarDL deployment on AWS (Amazon DynamoDB)](SetupDatabaseForAWS.mdx#amazon-dynamodb). ### Azure Cosmos DB for NoSQL When using the PITR feature, Azure Cosmos DB restores data by using another account. Therefore, you must update the endpoint configuration in the custom values file. #### Steps 1. Restore the account. For details on how to restore an Azure Cosmos DB account by using PITR, see [Restore an Azure Cosmos DB account that uses continuous backup mode](https://learn.microsoft.com/en-us/azure/cosmos-db/restore-account-continuous-backup). 2. Change the **default consistency level** for the restored account from the default value to **Strong**. For details on how to change this value, see the official documentation from Microsoft a [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). 3. Update **database.properties** for ScalarDB Schema Loader or ScalarDL Schema Loader based on the newly restored account. ScalarDB implements the Cosmos DB adapter by using its stored procedures, which are installed when creating schemas by using ScalarDB Schema Loader or ScalarDL Schema Loader. However, the PITR feature in Cosmos DB does not restore stored procedures, so you will need to reinstall the required stored procedures for all tables after restoration. You can reinstall the required stored procedures by using the `--repair-all` option in ScalarDB Schema Loader or ScalarDL Schema Loader. * **ScalarDB tables:** For details on how to configure **database.properties** for ScalarDB Schema Loader, see [Configure ScalarDB for Cosmos DB for NoSQL](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb#set-up-your-database-for-scalardb). * **ScalarDL tables:** For details on how to configure the custom values file for ScalarDL Schema Loader, see [Configure a custom values file for ScalarDL Schema Loader](../helm-charts/configure-custom-values-scalardl-schema-loader.mdx). 4. Re-create the stored procedures by using the `--repair-all` flag in ScalarDB Schema Loader or ScalarDL Schema Loader as follows: * ScalarDB tables ```console java -jar scalardb-schema-loader-.jar --config /path/to/ -f /path/to/ [--coordinator] --repair-all ``` * ScalarDL Ledger tables ```console helm install repair-schema-ledger scalar-labs/schema-loading -n -f /path/to/ --set "schemaLoading.commandArgs={--repair-all}" ``` * ScalarDL Auditor tables ```console helm install repair-schema-auditor scalar-labs/schema-loading -n -f /path/to/ --set "schemaLoading.commandArgs={--repair-all}" ``` For more details on repairing tables in ScalarDB Schema Loader, see [Repair tables](https://scalardb.scalar-labs.com/docs/latest/schema-loader#repair-tables). 5. Update the table configuration if necessary, depending on your environment. For some configurations for restored accounts, see [Set up a database for ScalarDB/ScalarDL deployment on Azure (Azure Cosmos DB for NoSQL)](SetupDatabaseForAzure.mdx#azure-cosmos-db-for-nosql). ### Amazon RDS When using the PITR feature, Amazon RDS restores data by using another database instance. Therefore, you must update the endpoint configuration in the custom values file. #### Steps 1. Restore the database instance. For details on how to restore the Amazon RDS instance by using PITR, see the following official documentation from Amazon: * [Restoring a DB instance to a specified time](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.html) * [Restoring a Multi-AZ DB cluster to a specified time](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PIT.MultiAZDBCluster.html) 2. Update the table configuration if necessary, depending on your environment. For some configurations for the restored database instance, see [Set up a database for ScalarDB/ScalarDL deployment on AWS (Amazon RDS for MySQL, PostgreSQL, Oracle, and SQL Server)](SetupDatabaseForAWS.mdx#amazon-rds-for-mysql-postgresql-oracle-and-sql-server). ### Amazon Aurora When using the PITR feature, Amazon Aurora restores data by using another database cluster. Therefore, you must update the endpoint configuration in the custom values file. #### Steps 1. Restore the database cluster. For details on how to restore an Amazon Aurora cluster by using PITR. see the official documentation from Amazon at [Restoring a DB cluster to a specified time](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-pitr.html). 2. Add a replica (reader) to make the database cluster a Multi-AZ cluster if necessary, depending on your environment. The PITR feature in Amazon Aurora cannot restore a database cluster by using a Multi-AZ configuration. If you want to restore the database cluster as a Multi-AZ cluster, you must add a reader after restoring the database cluster. For details on how to add a reader, see the official documentation from Amazon at [Adding Aurora Replicas to a DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-replicas-adding.html). 3. Update the table configuration if necessary, depending on your environment. For some configurations for the restored database cluster, see [Set up a database for ScalarDB/ScalarDL deployment on AWS (Amazon Aurora MySQL and Amazon Aurora PostgreSQL)](SetupDatabaseForAWS.mdx#amazon-aurora-mysql-and-amazon-aurora-postgresql). ### Azure Database for MySQL/PostgreSQL When using the PITR feature, Azure Database for MySQL/PostgreSQL restores data by using another server. Therefore, you must update the endpoint configuration in the custom values file. #### Steps 1. Restore the database server. For details on how to restore an Azure Database for MySQL/PostgreSQL server by using PITR, see the following: * [Point-in-time restore of a Azure Database for MySQL Flexible Server using Azure portal](https://learn.microsoft.com/en-us/azure/mysql/flexible-server/how-to-restore-server-portal) * [Backup and restore in Azure Database for PostgreSQL - Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-backup-restore) 2. Update the table configuration if necessary, depending on your environment. For some configurations for the restored database server, see the following: * [Set up a database for ScalarDB/ScalarDL deployment on Azure (Azure Database for MySQL)](SetupDatabaseForAzure.mdx#azure-database-for-mysql) * [Set up a database for ScalarDB/ScalarDL deployment on Azure (Azure Database for PostgreSQL)](SetupDatabaseForAzure.mdx#azure-database-for-postgresql) ================================================ FILE: docs/scalar-kubernetes/SetupDatabase.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Set up a database for ScalarDB/ScalarDL deployment This guide explains how to set up a database for ScalarDB/ScalarDL deployment on cloud services. * [Set up a database for ScalarDB/ScalarDL deployment on AWS](SetupDatabaseForAWS.mdx) * [Set up a database for ScalarDB/ScalarDL deployment on Azure](SetupDatabaseForAzure.mdx) ================================================ FILE: docs/scalar-kubernetes/SetupDatabaseForAWS.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Set up a database for ScalarDB/ScalarDL deployment on AWS This guide explains how to set up a database for ScalarDB/ScalarDL deployment on AWS. ## Amazon DynamoDB ### Authentication method When you use DynamoDB, you must set `REGION`, `ACCESS_KEY_ID`, and `SECRET_ACCESS_KEY` in the ScalarDB/ScalarDL properties file as follows. ```properties scalar.db.contact_points= scalar.db.username= scalar.db.password= scalar.db.storage=dynamo ``` Please refer to the following document for more details on the properties for DynamoDB. * [Configure ScalarDB for DynamoDB](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb#configure-scalardb-2) ### Required configuration/steps DynamoDB is available for use in AWS by default. You do not need to set up anything manually to use it. ### Optional configurations/steps #### Enable point-in-time recovery (Recommended in the production environment) You can enable PITR as a backup/restore method for DynamoDB. If you use [ScalarDB Schema Loader](https://scalardb.scalar-labs.com/docs/latest/schema-loader) for creating schema, it enables the PITR feature for tables by default. Please refer to the official document for more details. * [Point-in-time recovery for DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/PointInTimeRecovery.html) It is recommended since the point-in-time recovery feature automatically and continuously takes backups so that you can reduce downtime (pause duration) for backup operations. Please refer to the following document for more details on how to backup/restore Scalar product data. * [Backup restore guide for Scalar products](BackupRestoreGuide.mdx) #### Configure monitoring (Recommended in the production environment) You can configure the monitoring and logging of DynamoDB using its native feature. Please refer to the official document for more details. * [Monitoring and logging](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/monitoring.html) It is recommended since the metrics and logs help you to investigate some issues in the production environment when they happen. #### Use VPC endpoint (Recommended in the production environment) // Note that We have not yet tested this feature with Scalar products. // TODO: We need to test this feature with Scalar products. * [Using Amazon VPC endpoints to access DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/vpc-endpoints-dynamodb.html) It is recommended since the private internal connections not via WAN can make a system more secure. #### Configure Read/Write Capacity (Optional based on your environment) You can configure the **Read/Write Capacity** of DynamoDB tables based on your requirements. Please refer to the official document for more details on Read/Write Capacity. * [Read/write capacity mode](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html) You can configure Read/Write Capacity using ScalarDB/DL Schema Loader when you create a table. Please refer to the following document for more details on how to configure Read/Write Capacity (RU) using ScalarDB/DL Schema Loader. * [ScalarDB Schema Loader](https://scalardb.scalar-labs.com/docs/latest/schema-loader) ## Amazon RDS for MySQL, PostgreSQL, Oracle, and SQL Server ### Authentication method When you use RDS, you must set `JDBC_URL`, `USERNAME`, and `PASSWORD` in the ScalarDB/ScalarDL properties file as follows. ```properties scalar.db.contact_points= scalar.db.username= scalar.db.password= scalar.db.storage=jdbc ``` Please refer to the following document for more details on the properties for RDS (JDBC databases). * [Configure ScalarDB for JDBC databases](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb#set-up-your-database-for-scalardb) ### Required configuration/steps #### Create an RDS database instance You must create an RDS database instance. Please refer to the official document for more details. * [Configuring an Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_RDS_Configuring.html) ### Optional configurations/steps #### Enable automated backups (Recommended in the production environment) You can enable automated backups. Please refer to the official document for more details. * [Working with backups](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html) It is recommended since the automated backups feature enables a point-in-time recovery feature. It can recover data to a specific point in time. It can reduce downtime (pause duration) for backup operations when you use multi databases under Scalar products. Please refer to the following document for more details on how to backup/restore the Scalar product data. * [Backup restore guide for Scalar products](BackupRestoreGuide.mdx) #### Configure monitoring (Recommended in the production environment) You can configure the monitoring and logging of RDS using its native feature. Please refer to the official documents for more details. * [Monitoring metrics in an Amazon RDS instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Monitoring.html) * [Monitoring events, logs, and streams in an Amazon RDS DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Monitor_Logs_Events.html) It is recommended since the metrics and logs help you to investigate some issues in the production environment when they happen. #### Disable public access (Recommended in the production environment) Public access is disabled by default. You can access the RDS database instance from the Scalar product pods on your EKS cluster as follows. * Create the RDS database instance on the same VPC as your EKS cluster. * Connect the VPC for the RDS and the VPC for the EKS cluster for the Scalar product deployment using [VPC peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html). (// TODO: We need to test this feature with Scalar products.) It is recommended since the private internal connections not via WAN can make a system more secure. ## Amazon Aurora MySQL and Amazon Aurora PostgreSQL ### Authentication method When you use Amazon Aurora, you must set `JDBC_URL`, `USERNAME`, and `PASSWORD` in the ScalarDB/ScalarDL properties file as follows. ```properties scalar.db.contact_points= scalar.db.username= scalar.db.password= scalar.db.storage=jdbc ``` Please refer to the following document for more details on the properties for Amazon Aurora (JDBC databases). * [Configure ScalarDB for JDBC databases](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb#set-up-your-database-for-scalardb) ### Required configuration/steps #### Create an Amazon Aurora DB cluster You must create an Amazon Aurora DB cluster. Please refer to the official document for more details. * [Configuring your Amazon Aurora DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_AuroraSettingUp.html) ### Optional configurations/steps #### Configure backup configurations (Optional based on your environment) Amazon Aurora automatically gets a backup by default. You do not need to enable the backup feature manually. If you want to change some backup configurations like the backup retention period and backup window, you can configure them. Please refer to the official document for more details. * [Backing up and restoring an Amazon Aurora DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/BackupRestoreAurora.html) Please refer to the following document for more details on how to backup/restore the Scalar product data. * [Backup restore guide for Scalar products](BackupRestoreGuide.mdx) #### Configure monitoring (Recommended in the production environment) You can configure the monitoring and logging of Amazon Aurora using its native feature. Please refer to the official documents for more details. * [Monitoring metrics in an Amazon Aurora cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/MonitoringAurora.html) * [Monitoring events, logs, and streams in an Amazon Aurora DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_Monitor_Logs_Events.html) It is recommended since the metrics and logs help you to investigate some issues in the production environment when they happen. #### Disable public access (Recommended in the production environment) Public access is disabled by default. You can access the Amazon Aurora DB cluster from the Scalar product pods on your EKS cluster as follows. * Create the Amazon Aurora DB cluster on the same VPC as your EKS cluster. * Connect the VPC for the Amazon Aurora DB cluster and the VPC for the EKS cluster for the Scalar product deployment using [VPC peering](https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html). (// TODO: We need to test this feature with Scalar products.) It is recommended since the private internal connections not via WAN can make a system more secure. ================================================ FILE: docs/scalar-kubernetes/SetupDatabaseForAzure.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Set up a database for ScalarDB/ScalarDL deployment on Azure This guide explains how to set up a database for ScalarDB/ScalarDL deployment on Azure. ## Azure Cosmos DB for NoSQL ### Authentication method When you use Cosmos DB for NoSQL, you must set `COSMOS_DB_URI` and `COSMOS_DB_KEY` in the ScalarDB/ScalarDL properties file as follows. ```properties scalar.db.contact_points= scalar.db.password= scalar.db.storage=cosmos ``` Please refer to the following document for more details on the properties for Cosmos DB for NoSQL. * [Configure ScalarDB for Cosmos DB for NoSQL](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb#set-up-your-database-for-scalardb) ### Required configuration/steps #### Create an Azure Cosmos DB account You must create an Azure Cosmos DB account with the NoSQL (core) API. You must set the **Capacity mode** as **Provisioned throughput** when you create it. Please refer to the official document for more details. * [Quickstart: Create an Azure Cosmos DB account, database, container, and items from the Azure portal](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal) #### Configure a default consistency configuration You must set the **Default consistency level** as **Strong**. Please refer to the official document for more details. * [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#config/ure-the-default-consistency-level) ### Optional configurations/steps #### Configure backup configurations (Recommended in the production environment) You can configure **Backup modes** as **Continuous backup mode** for PITR. Please refer to the official document for more details. * [Backup modes](https://learn.microsoft.com/en-us/azure/cosmos-db/online-backup-and-restore#backup-modes) It is recommended since the continuous backup mode automatically and continuously gets backups so that we can reduce downtime (pause duration) for backup operations. Please refer to the following document for more details on how to backup/restore the Scalar product data. * [Backup restore guide for Scalar products](BackupRestoreGuide.mdx) #### Configure monitoring (Recommended in the production environment) You can configure the monitoring of Cosmos DB using its native feature. Please refer to the official document for more details. * [Monitor Azure Cosmos DB](https://learn.microsoft.com/en-us/azure/cosmos-db/monitor) It is recommended since the metrics and logs help you to investigate some issues in the production environment when they happen. #### Enable service endpoint (Recommended in the production environment) You can configure the Azure Cosmos DB account to allow access only from a specific subnet of a virtual network (VNet). Please refer to the official document for more details. * [Configure access to Azure Cosmos DB from virtual networks (VNet)](https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-configure-vnet-service-endpoint) It is recommended since the private internal connections not via WAN can make a system more secure. #### Configure the Request Units (Optional based on your environment) You can configure the **Request Units** of Cosmos DB based on your requirements. Please refer to the official document for more details on the request units. * [Request Units in Azure Cosmos DB](https://learn.microsoft.com/en-us/azure/cosmos-db/request-units) You can configure Request Units using ScalarDB/DL Schema Loader when you create a table. Please refer to the following document for more details on how to configure Request Units (RU) using ScalarDB/DL Schema Loader. * [ScalarDB Schema Loader](https://scalardb.scalar-labs.com/docs/latest/schema-loader) ## Azure Database for MySQL ### Authentication method When you use Azure Database for MySQL, you must set `JDBC_URL`, `USERNAME`, and `PASSWORD` in the ScalarDB/ScalarDL properties file as follows. ```properties scalar.db.contact_points= scalar.db.username= scalar.db.password= scalar.db.storage=jdbc ``` Please refer to the following document for more details on the properties for Azure Database for MySQL (JDBC databases). * [Configure ScalarDB for JDBC databases](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb#set-up-your-database-for-scalardb) ### Required configuration/steps #### Create a database server You must create a database server. Please refer to the official document for more details. * [Quickstart: Use the Azure portal to create an Azure Database for MySQL Flexible Server](https://learn.microsoft.com/en-us/azure/mysql/flexible-server/quickstart-create-server-portal) You can choose **Single Server** or **Flexible Server** for your deployment. However, Flexible Server is recommended in Azure. This document assumes that you use Flexible Server. Please refer to the official documents for more details on the deployment models. * [What is Azure Database for MySQL?](https://learn.microsoft.com/en-us/azure/mysql/single-server/overview#deployment-models) ### Optional configurations/steps #### Configure backup configurations (Optional based on your environment) Azure Database for MySQL gets a backup by default. You do not need to enable the backup feature manually. If you want to change some backup configurations like the backup retention period, you can configure it. Please refer to the official document for more details. * [Backup and restore in Azure Database for MySQL Flexible Server](https://learn.microsoft.com/en-us/azure/mysql/flexible-server/concepts-backup-restore) Please refer to the following document for more details on how to backup/restore the Scalar product data. * [Backup restore guide for Scalar products](BackupRestoreGuide.mdx) #### Configure monitoring (Recommended in the production environment) You can configure the monitoring of Azure Database for MySQL using its native feature. Please refer to the official document for more details. * [Monitor Azure Database for MySQL Flexible Server](https://learn.microsoft.com/en-us/azure/mysql/flexible-server/concepts-monitoring) It is recommended since the metrics and logs help you to investigate some issues in the production environment when they happen. #### Disable public access (Recommended in the production environment) You can configure **Private access (VNet Integration)** as a **Connectivity method**. Please refer to the official document for more details. * [Connectivity and networking concepts for Azure Database for MySQL - Flexible Server](https://learn.microsoft.com/en-us/azure/mysql/flexible-server/concepts-networking) You can access the database server from the Scalar product pods on your AKS cluster as follows. * Create the database server on the same VNet as your AKS cluster. * Connect the VNet for the database server and the VNet for the AKS cluster for the Scalar product deployment using [Virtual network peering](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview). (// TODO: We need to test this feature with Scalar products.) It is recommended since the private internal connections not via WAN can make a system more secure. ## Azure Database for PostgreSQL ### Authentication method When you use Azure Database for PostgreSQL, you must set `JDBC_URL`, `USERNAME`, and `PASSWORD` in the ScalarDB/ScalarDL properties file as follows. ```properties scalar.db.contact_points= scalar.db.username= scalar.db.password= scalar.db.storage=jdbc ``` Please refer to the following document for more details on the properties for Azure Database for PostgreSQL (JDBC databases). * [Configure ScalarDB for JDBC databases](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb#set-up-your-database-for-scalardb) ### Required configuration/steps #### Create a database server You must create a database server. Please refer to the official document for more details. * [Quickstart: Create an Azure Database for PostgreSQL - Flexible Server in the Azure portal](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/quickstart-create-server-portal) You can choose **Single Server** or **Flexible Server** for your deployment. However, Flexible Server is recommended in Azure. This document assumes that you use Flexible Server. Please refer to the official documents for more details on the deployment models. * [What is Azure Database for PostgreSQL?](https://learn.microsoft.com/en-us/azure/postgresql/single-server/overview#deployment-models) ### Optional configurations/steps #### Configure backup configurations (Optional based on your environment) Azure Database for PostgreSQL gets a backup by default. You do not need to enable the backup feature manually. If you want to change some backup configurations like the backup retention period, you can configure it. Please refer to the official document for more details. * [Backup and restore in Azure Database for PostgreSQL - Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-backup-restore) Please refer to the following document for more details on how to backup/restore the Scalar product data. * [Backup restore guide for Scalar products](BackupRestoreGuide.mdx) #### Configure monitoring (Recommended in the production environment) You can configure the monitoring of Azure Database for PostgreSQL using its native feature. Please refer to the official document for more details. * [Monitor metrics on Azure Database for PostgreSQL - Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-monitoring) It is recommended since the metrics and logs help you to investigate some issues in the production environment when they happen. #### Disable public access (Recommended in the production environment) You can configure **Private access (VNet Integration)** as a **Connectivity method**. Please refer to the official document for more details. * [Networking overview for Azure Database for PostgreSQL - Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-networking) You can access the database server from the Scalar product pods on your AKS cluster as follows. * Create the database server on the same VNet as your AKS cluster. * Connect the VNet for the database server and the VNet for the AKS cluster for the Scalar product deployment using [Virtual network peering](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview). (// TODO: We need to test this feature with Scalar products.) It is recommended since the private internal connections not via WAN can make a system more secure. ================================================ FILE: docs/scalar-kubernetes/alerts/README.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Scalar Alerts This section covers the types of alerts and what actions need to be taken. * [Envoy Alerts](Envoy.mdx) * [Ledger Alerts](Ledger.mdx) ================================================ FILE: docs/scalar-kubernetes/alerts/Envoy.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Envoy Alerts ## EnvoyClusterDown This is the most critical alert and indicates that an Envoy cluster is not able to process requests. This alert should be handled with the highest priority. ### Example Alert #### Firing ``` [FIRING:1] EnvoyClusterDown - critical Alert: Envoy cluster is down - critical Description: Envoy cluster is down, no resquest can be process Details: • alertname: EnvoyClusterDown • deployment: prod-scalardl-envoy ``` #### Resolved ``` [RESOLVED] EnvoyClusterDown - critical Alert: Envoy cluster is down - critical Description: Envoy cluster is down, no resquest can be process Details: • alertname: EnvoyClusterDown • deployment: prod-scalardl-envoy ``` ### Action Needed * Check the number of replicas set `kubectl get deployments. prod-scalardl-envoy` * Check the number of replicas set `kubectl describe deployments. prod-scalardl-envoy` * Check nodes statuses with `kubectl get node -o wide` * Check the log server to pinpoint the root cause of a failure with kubernetes logs on the monitor server `/log/kubernetes//-/kube.log` * Check a cloud provider to see if there is any known issue. For example, you can check statues [here](https://status.azure.com/en-us/status) in Azure. ## EnvoyClusterDegraded This alert lets you know if a kubernetes cluster cannot start envoy pods, which means that the cluster does not have enough resource or lost of one or many kubernetes nodes to run the deployment. ### Example Alert #### Firing ``` [FIRING:1] EnvoyClusterDegraded - warning Alert: Envoy cluster is running in a degraded mode - warning Description: Envoy cluster is running in a degraded mode, some of the Envoy pods are not healthy Details: • alertname: EnvoyClusterDegraded • deployment: prod-scalardl-envoy ``` #### Resolved ``` [RESOLVED] EnvoyClusterDegraded - warning Alert: Envoy cluster is running in a degraded mode - warning Description: Envoy cluster is running in a degraded mode, some of the Envoy pods are not healthy Details: • alertname: EnvoyClusterDegraded • deployment: prod-scalardl-envoy ``` ### Action Needed * Check the log server to pinpoint the root cause of a failure with kubernetes logs on the monitor server `/log/kubernetes//-/kube.log` or `kubectl logs prod-scalardl-envoy-xxxx-yyyy` * Check kubernetes deployment with `kubectl describe deployments prod-scalardl-envoy` * Check replica set with `kubectl get replicasets.apps` * Check nodes statuses with `kubectl get node -o wide` * Check a cloud provider to see if there is any known issue. For example, you can check statues [here](https://status.azure.com/en-us/status) in Azure. ## EnvoyPodsPending This alert lets you know if a kubernetes cluster cannot start envoy pods, which means that the cluster does not have the enough resource. ### Example Alert #### Firing ``` [FIRING:1] EnvoyPodsPending - warning Alert: Pod prod-scalardl-envoy-xxxx-yyyy in namespace default in pending status - warning Description: Pod prod-scalardl-envoy-xxxx-yyyy in namespace default has been in pending status for more than 1 minute. Details: • alertname: EnvoyPodsPending • deployment: prod-scalardl-envoy ``` #### Resolved ``` [RESOLVED:1] EnvoyPodsPending - warning Alert: Pod prod-scalardl-envoy-xxxx-yyyy in namespace default in pending status - warning Description: Pod prod-scalardl-envoy-xxxx-yyyy in namespace default has been in pending status for more than 1 minute. Details: • alertname: EnvoyPodsPending • deployment: prod-scalardl-envoy ``` ### Action Needed * Check log server to pinpoint the root cause of a failure with kubernetes logs on the monitor server `/log/kube//*.log` * Check a kubernetes deployment with `kubectl describe pod prod-scalardl-envoy-xxxx-yyyy` ## EnvoyPodsError This alert lets you know if a kubernetes cluster cannot start envoy pods for one of the following reasons: * CrashLoopBackOff * CreateContainerConfigError * CreateContainerError * ErrImagePull * ImagePullBackOff * InvalidImageName ### Example Alert #### Firing ``` [FIRING:1] EnvoyPodsError - warning Alert: Pod prod-scalardl-envoy-xxxx-yyyy in namespace default has an error status - warning Description: Pod prod-scalardl-envoy-xxxx-yyyy in namespace default has been in pending status for more than 1 minutes. Details: • alertname: EnvoyPodsError • deployment: prod-scalardl-envoy ``` #### Resolved ``` [RESOLVED:1] EnvoyPodsError - warning Alert: Pod prod-scalardl-envoy-xxxx-yyyy in namespace default has an error status - warning Description: Pod prod-scalardl-envoy-xxxx-yyyy in namespace default has been in pending status for more than 1 minutes. Details: • alertname: EnvoyPodsError • deployment: prod-scalardl-envoy ``` ### Action Needed * Check a kubernetes deployment with `kubectl describe pod prod-scalardl-envoy-xxxx-yyyy` * Check the log server to pinpoint the root cause of a failure with kubernetes logs on the monitor server `/log/kubernetes//-/kube.log` ================================================ FILE: docs/scalar-kubernetes/alerts/Ledger.mdx ================================================ --- tags: - Enterprise displayed_sidebar: docsEnglish --- # Ledger Alerts ## LedgerClusterDown This is the most critical alert and indicates that an Ledger cluster is not able to process requests. This alert should be handled with the highest priority. ### Example Alert #### Firing ``` [FIRING:1] LedgerClusterDown - critical Alert: Ledger cluster is down - critical Description: Ledger cluster is down, no resquest can be process. Details: • alertname: LedgerClusterDown • deployment: prod-scalardl-ledger ``` #### Resolved ``` [RESOLVED] LedgerClusterDown - critical Alert: Ledger cluster is down - critical Description: Ledger cluster is down, no resquest can be process. Details: • alertname: LedgerClusterDown • deployment: prod-scalardl-ledger ``` ### Action Needed * Check the number of replicas set `kubectl get deployments. prod-scalardl-ledger` * Check the number of replicas set `kubectl describe deployments. prod-scalardl-ledger` * Check nodes statuses with `kubectl get node -o wide` * Check the log server to pinpoint the root cause of a failure with kubernetes logs on the monitor server `/log/kubernetes//-/kube.log` * Check a cloud provider to see if there is any known issue. For example, you can check statues [here](https://status.azure.com/en-us/status) in Azure. ## LedgerClusterDegraded This alert lets you know if a kubernetes cluster cannot start ledger pods, which means that the cluster does not have enough resource or lost of one or many kubernetes nodes to run the deployment. ### Example Alert #### Firing ``` [FIRING:1] LedgerClusterDegraded - warning Alert: Ledger cluster is running in a degraded mode - warning Description: Ledger cluster is running in a degraded mode, some of the Ledger pods are not healthy. Details: • alertname: LedgerClusterDegraded • deployment: prod-scalardl-ledger ``` #### Resolved ``` [RESOLVED] LedgerClusterDegraded - warning Alert: Ledger cluster is running in a degraded mode - warning Description: Ledger cluster is running in a degraded mode, some of the Ledger pods are not healthy. Details: • alertname: LedgerClusterDegraded • deployment: prod-scalardl-ledger ``` ### Action Needed * Check the log server to pinpoint the root cause of a failure with kubernetes logs on the monitor server `/log/kubernetes//-/kube.log` * Check kubernetes deployment with `kubectl describe deployments prod-scalardl-ledger` * Check replica set with `kubectl get replicasets.apps` * Check nodes statuses with `kubectl get node -o wide` * Check a cloud provider to see if there is any known issue. For example, you can check statues [here](https://status.azure.com/en-us/status) in Azure. ## LedgerPodsPending This alert lets you know if a kubernetes cluster cannot start ledger pods, which means that the cluster does not have the enough resource. ### Example Alert #### Firing ``` [FIRING:1] LedgerPodsPending - warning Alert: Pod prod-scalardl-ledger-xxxx-yyyy in namespace default in pending status - warning Description: Pod prod-scalardl-ledger-xxxx-yyyy in namespace default has been in pending status for more than 1 minute. Details: • alertname: LedgerPodsPending • deployment: prod-scalardl-ledger ``` #### Resolved ``` [RESOLVED:1] LedgerPodsPending - warning Alert: Pod prod-scalardl-ledger-xxxx-yyyy in namespace default in pending status - warning Description: Pod prod-scalardl-ledger-xxxx-yyyy in namespace default has been in pending status for more than 1 minute. Details: • alertname: LedgerPodsPending • deployment: prod-scalardl-ledger ``` ### Action Needed * Check log server to pinpoint root cause of failure with the kubernetes logs on the monitor server `/log/kubernetes//-/kube.log` * Check the kubernetes deployment with `kubectl describe pod prod-scalardl-ledger-xxxx-yyyy` ## LedgerPodsError This alert lets you know if a kubernetes cluster cannot start ledger pods for one of the following reasons: * CrashLoopBackOff * CreateContainerConfigError * CreateContainerError * ErrImagePull * ImagePullBackOff * InvalidImageName ### Example Alert #### Firing ``` [FIRING:1] LedgerPodsError - warning Alert: Pod prod-scalardl-ledger-xxxx-yyyy in namespace default has an error status - warning Description: Pod prod-scalardl-ledger-xxxx-yyyy in namespace default has been in pending status for more than 1 minutes. Details: • alertname: LedgerPodsError • deployment: prod-scalardl-ledger ``` #### Resolved ``` [RESOLVED:1] LedgerPodsError - warning Alert: Pod prod-scalardl-ledger-xxxx-yyyy in namespace default has an error status - warning Description: Pod prod-scalardl-ledger-xxxx-yyyy in namespace default has been in pending status for more than 1 minutes. Details: • alertname: LedgerPodsError • deployment: prod-scalardl-ledger ``` ### Action Needed * Check the kubernetes deployment with `kubectl describe pod prod-scalardl-ledger-xxxx-yyyy` * Check log server to pinpoint root cause of failure with the kubernetes logs on the monitor server `/log/kubernetes//-/kube.log` ================================================ FILE: docs/scalar-licensing/commercial.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Enterprise Option displayed_sidebar: docsEnglish --- # How to Configure a Commercial License Key To run ScalarDB Enterprise Standard/Premium or ScalarDB Analytics, you must create a `.properties` file and add your commercial license key and a certificate to the file. In your `.properties` file, copy one of the following configurations, based on the product you're using, and paste the contents in the `.properties` file, replacing `` with your license key. :::note - If you don't have a license key, please see [How to Configure a Trial License Key](./trial.mdx) for ready-to-use keys or [contact us](https://www.scalar-labs.com/contact) to obtain a commercial license. - The licensing mechanism for ScalarDB verifies the signature, expiration date, and edition of your license key. However, it doesn't check deployment details such as the number of running Pods or contract-specific limits. Because of this, you may technically be able to deploy more Pods than your contract allows, so make sure you use the number you are allowed. ::: ## ScalarDB Enterprise Standard/Premium ```properties scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIICKzCCAdKgAwIBAgIIBXxj3s8NU+owCgYIKoZIzj0EAwIwbDELMAkGA1UEBhMC\nSlAxDjAMBgNVBAgTBVRva3lvMREwDwYDVQQHEwhTaGluanVrdTEVMBMGA1UEChMM\nU2NhbGFyLCBJbmMuMSMwIQYDVQQDExplbnRlcnByaXNlLnNjYWxhci1sYWJzLmNv\nbTAeFw0yMzExMTYwNzExNTdaFw0yNDAyMTUxMzE2NTdaMGwxCzAJBgNVBAYTAkpQ\nMQ4wDAYDVQQIEwVUb2t5bzERMA8GA1UEBxMIU2hpbmp1a3UxFTATBgNVBAoTDFNj\nYWxhciwgSW5jLjEjMCEGA1UEAxMaZW50ZXJwcmlzZS5zY2FsYXItbGFicy5jb20w\nWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATJx5gvAr+GZAHcBpUvDFDsUlFo4GNw\npRfsntzwStIP8ac3dew7HT4KbGBWei0BvIthleaqpv0AEP7JT6eYAkNvo14wXDAO\nBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwG\nA1UdEwEB/wQCMAAwHQYDVR0OBBYEFMIe+XuuZcnDX1c3TmUPlu3kNv/wMAoGCCqG\nSM49BAMCA0cAMEQCIGGlqKpgv+KW+Z1ZkjfMHjSGeUZKBLwfMtErVyc9aTdIAiAy\nvsZyZP6Or9o40x3l3pw/BT7wvy93Jm0T4vtVQH6Zuw==\n-----END CERTIFICATE----- ``` ## ScalarDB Analytics ```apacheconf spark.sql.catalog.scalardb_catalog.license.key spark.sql.catalog.scalardb_catalog.license.cert_pem -----BEGIN CERTIFICATE-----\nMIICKzCCAdKgAwIBAgIIBXxj3s8NU+owCgYIKoZIzj0EAwIwbDELMAkGA1UEBhMC\nSlAxDjAMBgNVBAgTBVRva3lvMREwDwYDVQQHEwhTaGluanVrdTEVMBMGA1UEChMM\nU2NhbGFyLCBJbmMuMSMwIQYDVQQDExplbnRlcnByaXNlLnNjYWxhci1sYWJzLmNv\nbTAeFw0yMzExMTYwNzExNTdaFw0yNDAyMTUxMzE2NTdaMGwxCzAJBgNVBAYTAkpQ\nMQ4wDAYDVQQIEwVUb2t5bzERMA8GA1UEBxMIU2hpbmp1a3UxFTATBgNVBAoTDFNj\nYWxhciwgSW5jLjEjMCEGA1UEAxMaZW50ZXJwcmlzZS5zY2FsYXItbGFicy5jb20w\nWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATJx5gvAr+GZAHcBpUvDFDsUlFo4GNw\npRfsntzwStIP8ac3dew7HT4KbGBWei0BvIthleaqpv0AEP7JT6eYAkNvo14wXDAO\nBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwG\nA1UdEwEB/wQCMAAwHQYDVR0OBBYEFMIe+XuuZcnDX1c3TmUPlu3kNv/wMAoGCCqG\nSM49BAMCA0cAMEQCIGGlqKpgv+KW+Z1ZkjfMHjSGeUZKBLwfMtErVyc9aTdIAiAy\nvsZyZP6Or9o40x3l3pw/BT7wvy93Jm0T4vtVQH6Zuw==\n-----END CERTIFICATE----- ``` ================================================ FILE: docs/scalar-licensing/index.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Enterprise Option displayed_sidebar: docsEnglish --- # How to Configure a License Key To run ScalarDB Enterprise Standard/Premium or ScalarDB Analytics, you need to configure a license key. - If you have a commercial license key for ScalarDB Enterprise Standard/Premium edition or ScalarDB Analytics, please refer to [How to Configure a Commercial License Key](./commercial.mdx) to configure your license key. - If you want to evaluate ScalarDB Enterprise Standard/Premium edition or ScalarDB Analytics, you can use the trial license keys provided in [How to Configure a Trial License Key](./trial.mdx). ================================================ FILE: docs/scalar-licensing/trial.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium - Enterprise Option displayed_sidebar: docsEnglish --- # How to Configure a Trial License Key You can use the following trial license keys for ScalarDB Enterprise Standard/Premium and ScalarDB Analytics. If you have a commercial license key, please refer to [How to Configure a Commercial License Key](./commercial.mdx) to configure your license key. To run ScalarDB Enterprise Standard/Premium or ScalarDB Analytics, you must create a `.properties` file and add the trial license key and the certificate to the file. In your `.properties` file, copy one of the following configurations, based on the product you're using, and paste the contents in the `.properties` file. :::warning - These trial license keys are for non-production, evaluation purposes only. - These trial licenses are provided "as-is" without any warranty, and Scalar shall not be liable for any damages arising from their use. - When using a trial license, ScalarDB Cluster and/or ScalarDB Analytics must be connected to the Internet to validate the license and check its expiration. - Redistribution or reverse engineering of these license keys is strictly prohibited. - These trial license keys are updated periodically. For production use, please [contact us](https://www.scalar-labs.com/contact) to obtain a commercial license. ::: :::note ScalarDB Core is available as open-source software under the Apache 2.0 License on [GitHub](https://github.com/scalar-labs/scalardb). ::: ## ScalarDB Enterprise Standard/Premium ```properties scalar.db.cluster.node.licensing.license_key={"organization_name":"Trial","product_name":"ScalarDB Cluster","product_version":3,"license_type":"trial","signature":"MEUCIQDzRbDIaQR2yOBpl6GkB3AJWu9jHyUeGmGfOV6xxr2VlAIgJoDcCTN80m52m8JRbQcaJ+hgw7Os76lMHnn/wVdC/Oo=","expiration_date_time":"2026-08-30T10:44:23.496+09:00[Asia/Tokyo]"} scalar.db.cluster.node.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIICIzCCAcigAwIBAgIIKT9LIGX1TJQwCgYIKoZIzj0EAwIwZzELMAkGA1UEBhMC\nSlAxDjAMBgNVBAgTBVRva3lvMREwDwYDVQQHEwhTaGluanVrdTEVMBMGA1UEChMM\nU2NhbGFyLCBJbmMuMR4wHAYDVQQDExV0cmlhbC5zY2FsYXItbGFicy5jb20wHhcN\nMjMxMTE2MDcxMDM5WhcNMjQwMjE1MTMxNTM5WjBnMQswCQYDVQQGEwJKUDEOMAwG\nA1UECBMFVG9reW8xETAPBgNVBAcTCFNoaW5qdWt1MRUwEwYDVQQKEwxTY2FsYXIs\nIEluYy4xHjAcBgNVBAMTFXRyaWFsLnNjYWxhci1sYWJzLmNvbTBZMBMGByqGSM49\nAgEGCCqGSM49AwEHA0IABBSkIYAk7r5FRDf5qRQ7dbD3ib5g3fb643h4hqCtK+lC\nwM4AUr+PPRoquAy+Ey2sWEvYrWtl2ZjiYyyiZw8slGCjXjBcMA4GA1UdDwEB/wQE\nAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIw\nADAdBgNVHQ4EFgQUbFyOWFrsjkkOvjw6vK3gGUADGOcwCgYIKoZIzj0EAwIDSQAw\nRgIhAKwigOb74z9BdX1+dUpeVG8WrzLTIqdIU0w+9jhAueXoAiEA6cniJ3qsP4j7\nsck62kHnFpH1fCUOc/b/B8ZtfeXI2Iw=\n-----END CERTIFICATE----- ``` ## ScalarDB Analytics ```apacheconf spark.sql.catalog.scalardb_catalog.license.key {"organization_name":"Trial","product_name":"ScalarDB Analytics","product_version":3,"license_type":"trial","signature":"MEUCIFG1ecsXAbKafnfE2FSMjYDB8w15/HntGrC8RHXAUtc7AiEAr39FEbDIEr39kB+w7+rJETH25k3Ex/TnWNM9Wm1zYSc=","expiration_date_time":"2026-08-30T10:44:25.095+09:00[Asia/Tokyo]"} spark.sql.catalog.scalardb_catalog.license.cert_pem -----BEGIN CERTIFICATE-----\nMIICIzCCAcigAwIBAgIIKT9LIGX1TJQwCgYIKoZIzj0EAwIwZzELMAkGA1UEBhMC\nSlAxDjAMBgNVBAgTBVRva3lvMREwDwYDVQQHEwhTaGluanVrdTEVMBMGA1UEChMM\nU2NhbGFyLCBJbmMuMR4wHAYDVQQDExV0cmlhbC5zY2FsYXItbGFicy5jb20wHhcN\nMjMxMTE2MDcxMDM5WhcNMjQwMjE1MTMxNTM5WjBnMQswCQYDVQQGEwJKUDEOMAwG\nA1UECBMFVG9reW8xETAPBgNVBAcTCFNoaW5qdWt1MRUwEwYDVQQKEwxTY2FsYXIs\nIEluYy4xHjAcBgNVBAMTFXRyaWFsLnNjYWxhci1sYWJzLmNvbTBZMBMGByqGSM49\nAgEGCCqGSM49AwEHA0IABBSkIYAk7r5FRDf5qRQ7dbD3ib5g3fb643h4hqCtK+lC\nwM4AUr+PPRoquAy+Ey2sWEvYrWtl2ZjiYyyiZw8slGCjXjBcMA4GA1UdDwEB/wQE\nAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIw\nADAdBgNVHQ4EFgQUbFyOWFrsjkkOvjw6vK3gGUADGOcwCgYIKoZIzj0EAwIDSQAw\nRgIhAKwigOb74z9BdX1+dUpeVG8WrzLTIqdIU0w+9jhAueXoAiEA6cniJ3qsP4j7\nsck62kHnFpH1fCUOc/b/B8ZtfeXI2Iw=\n-----END CERTIFICATE----- ``` ================================================ FILE: docs/scalar-manager/how-to-use-scalar-manager.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # How to Use Scalar Manager Scalar Manager is a centralized management and monitoring solution for ScalarDB in Kubernetes environments. It simplifies operational tasks by providing a graphical user interface (GUI) that combines functionalities previously managed through separate command-line tools and third-party solutions. This guide explains how to use Scalar Manager to monitor, manage, and maintain your ScalarDB deployments. :::note For instructions on how to set up and configure Scalar Manager in your environment, see [Deploy Scalar Manager](../helm-charts/getting-started-scalar-manager.mdx). ::: ## System requirements Scalar Manager is a web-based application that can be accessed from the following supported web browsers: - Google Chrome (latest version) - Mozilla Firefox (latest version) - Microsoft Edge (latest version) - Safari (latest version) For the best experience: - Ensure that you have JavaScript enabled. - Confirm that you're connected to your Scalar Manager instance. - Disable pop-up blockers for the application domain. :::note This application is designed for desktop and tablet browser use. While it may load on mobile devices, functionality is not guaranteed or supported at this time. ::: ## User authentication This section describes how to log in, manage your password, and use single sign-on (SSO) in Scalar Manager. ### How to log in to Scalar Manager 1. In a web browser, open the Scalar Manager link that your system administrator provided. 2. In the **Email** field, enter your email address. 3. In the **Password** field, enter your password and select **Log In**. After logging in, you'll be redirected to the dashboard. :::note If you see an error message, double-check your email address and password, and try again. ::: ![login-page](images/login-page.png) ### How to manage your password The following describes how to change your password, the password requirements, and what to do if you forget your password. #### Change your password 1. Log in to Scalar Manager. 2. Go to your profile page. 3. In the **Set New Password** section, enter the required information: - Your current password - Your new password 4. Select **Save** to apply the changes. ![my-profile-page](images/my-profile.png) #### Password requirements When creating a new password, ensure that it meets the following criteria: - Minimum 8 characters long - Include at least: - 1 uppercase letter - 1 lowercase letter - 1 number - 1 special character #### What to do if you forget your password 1. Contact your system administrator. 2. Wait for your system administrator to reset your password and provide you with a temporary password. 3. Use the temporary password to log in. 4. Change your password immediately after logging in. :::note Scalar Manager does not support self-service password recovery. ::: ### How to use SSO with Grafana Scalar Manager provides seamless authentication with Grafana by using your existing credentials. With SSO integration, you can access any Grafana dashboard directly through the Scalar Manager interface. Authentication happens automatically when using your Scalar Manager credentials. :::note - SSO integration requires proper configuration by your system administrator. - If you encounter login prompts, please contact your system administrator. ::: ## User roles This section describes the roles that you can assign to users in Scalar Manager. ### Available roles and permissions The system has three fixed roles that cannot be extended or customized: | Role | Description | |---------------|---------------------------| | Administrator | Full system access | | Writer | Cluster management access | | Reader | View-only access | ### Role-based feature access Understanding which features are available to each role helps in assigning appropriate roles to users: | Feature | Administrator | Writer | Reader | |--------------------------|---------------|--------|--------| | User Management | ✅ | – | – | | Cluster Operations | ✅ | ✅ | – | | Execute Pauses | ✅ | ✅ | – | | View Cluster Information | ✅ | ✅ | ✅ | | View Metrics | ✅ | ✅ | ✅ | | View Logs | ✅ | ✅ | ✅ | :::note Role assignments take effect immediately after saving. ::: ## User management Users with the Administrator role can create, modify, and remove user accounts through Scalar Manager. ### How to assign roles to users Only users with the Administrator role can assign or modify user roles. Roles can be assigned in two ways: #### 1. Assigning a role during user creation 1. Go to the user list by selecting the **Users** menu item under **Admin Settings**. 2. Create a new user by selecting **Add User**. A sidebar will appear on the right side of the page. 3. Assign a role to the newly created user by doing the following: 1. Fill in the user information (name, email address, and password). 2. In the **Role** dropdown menu, select one of the three roles: - Administrator - Writer - Reader 3. Select **Add User** to create the user with the assigned role. #### 2. Modifying an existing user's role 1. Go to the user list by selecting the **Users** menu item under **Admin Settings**. You'll see a list of all users in the system. 2. Select the user whose role you want to modify. A sidebar will appear that shows the user's current information. 3. In the **Role** dropdown, select the new role that you want to assign the user. 4. Select **Save Changes** to apply your changes. ### Creating a new user 1. **Access the user management page** - Click on the **Users** menu item in the admin settings at the bottom of the page - You'll be taken to the user list page 2. **Using the user list** - The user list shows all users in the system - You can filter users by role using the role filter - Search for specific users by name or email address using the search bar 3. **Creating a new user** - Click the **Add User** button - A sidebar will appear on the right side of the page - Fill in the required user information: - Name - Email address - Select a role for the user - Enter an initial password - Click the **Add User** button to create the user :::note The system does not currently support email notifications. New users will need to be informed of their credentials through other means. ::: ![user-create-page](images/user-create.png) ### Modifying user details 1. Go to the user list by selecting the **Users** menu item under **Admin Settings**. You'll see a list of all users in the system. 2. Select the user that you want to modify from the list. A sidebar that shows the user's current information will appear on the right side of the page. 3. In the sidebar, choose what information you want to modify: - User's name - Email address - Role assignment - Password (if needed) 4. Select **Save Changes** to apply your changes. :::note Changes to user details take effect immediately after saving. ::: ![edit-user-page](images/edit-user.png) ### Deactivating/removing users :::note The system does not support temporarily deactivating user accounts. User accounts can only be permanently deleted. ::: 1. Go to the user list by selecting the **Users** menu item under **Admin Settings**. You'll see a list of all users in the system. 2. Find the user you want to delete in the list, then select **...** (context menu) next to the user's name. 3. Select **Delete** from the menu. 4. Confirm the deletion in the dialog window that appears. :::note When a user is deleted: - Their account is permanently removed. - Their authentication token will no longer work. - This action cannot be undone. ::: ![delete-user-page](images/delete-user.png) ### Reset a user's password 1. Go to the **Users** section in the administrator settings. You'll see a list of all users in the system. 2. Find the user in the list, and select their account to open their profile. 3. In the **Password** field, enter a password that the user will temporarily use. 4. Save the changes. 5. Notify the user of their temporary password. 6. Instruct the user to change the password after logging in with that temporary password. ## Cluster management This section describes how to view your cluster dashboard in Scalar Manager and how to access detailed release information. ### How to view your cluster dashboard After logging in, you'll see the main dashboard that helps you monitor and manage your Kubernetes cluster and ScalarDB deployments. #### View overall cluster health In the upper section of the dashboard, you'll see the following: - Kubernetes cluster availability status - Total CPU utilization - Total memory utilization - Current RPS (requests per second) #### Monitor release status The dashboard displays a list of all your ScalarDB deployments grouped by releases. For each row, you'll see the following: - Namespace name - Release name - ScalarDB deployment name and component type - Pod availability - Current RPS (requests per second) - Resource usage (CPU and memory) ### How to access detailed release information To get detailed information about a specific release, go to the list on the dashboard and select any row in the list to open the details page for that release. - **View release metrics.** The details page will show the following: - Overall availability status - Total CPU utilization - Total memory usage - Current RPS - **Monitor individual pods.** The pods list will show the following: - Pod status (Running, Pending, Failed) - Application name - Pod name and IP address - Uptime duration - Restart count - Individual pod resource usage - **Analyze pod health.** Use pod information to do the following: - Identify problematic pods (high restart count or failed status) - Monitor resource distribution - Track individual pod performance - **Filter and search pods.** Use the search bar at the top of the pods list to filter by the following: - Application name - Pod name - IP address ![release-detail-page](images/release-detail.png) ## Monitoring Scalar Manager provides comprehensive metrics in the following categories: - **Total Requests:** Overall success and failure rates - **Distributed Transaction Admin Service:** Table management operations - **Distributed Transaction Service:** Transaction operations for standard transactions - **Two Phase Commit Transaction Service:** Transaction operations for two-phase commit (2PC) transactions - **GraphQL Service:** GraphQL API performance - **SQL Total Requests:** SQL interface usage - **SQL Distributed Transaction Service:** SQL transaction operations - **SQL Two Phase Commit Transaction Service:** SQL 2PC transaction operations - **SQL Metadata Service:** SQL schema operations For detailed descriptions and interpretations of each metric, see [Scalar Manager Metrics Reference](metrics-reference.mdx). ### How to view metrics The metrics dashboard is pre-configured with product-specific metrics for ScalarDB. You can: - Select from pre-registered dashboards for different products and components. - View real-time performance metrics. - Analyze historical data trends. - Monitor system health indicators. To access the metrics dashboard: 1. Select **Metrics** in the side menu. 2. Select **Open metrics dashboard in Grafana**. Grafana will open with your cluster metrics displayed. Authentication is handled seamlessly by using your Scalar Manager credentials. ![grafana-dashboard](images/metrics.png) ## Log management This section describes how to access logs. ### How to access logs To access the logs dashboard: 1. Select **Logs** in the side menu. 2. Select **Open logs in Grafana**. Grafana will open with the logs from your cluster's pods displayed. The logs dashboard is pre-configured with your cluster's pod information. You can: - Filter logs by pod labels. - Search for specific log entries. - View real-time log streams. - Analyze historical log data. ![grafana-logs-dashboard](images/logs.png) ## Pausing and resuming clusters Pausing is a process that temporarily stops new transactions from being accepted to ensure transactional consistency during operations like backups. This helps maintain data integrity by ensuring that all transactions are completed before the operation begins. For detailed information about pausing and when to use it, see [How to Back Up and Restore Databases Used Through ScalarDB](../backup-restore.mdx). ### How to execute a pausing job immediately To execute a pausing job immediately: 1. Go to **Backup & Restore**. 2. Select **Create Pauses**. 3. Select the namespace from the dropdown menu. 4. Select the release from the dropdown menu. 5. Select **Execute**. This will pause all jobs in the release. ![execute-pause-page](images/execute-pause.png) ### How to schedule pausing jobs 1. Go to **Backup & Restore**. 2. Select **Create Pauses**. 3. Select **+ Schedule**. A pop-up window will appear for you to configure the schedule. 4. Select the namespace and release from the dropdown menus. 5. Choose the scheduling type (Daily, Weekly, or Monthly). 6. Set the time parameters (hour and minutes), timezone, and pause duration. 7. Select **Schedule** to create the schedule. - To discard the schedule, select **Cancel**. ![schedule-pause-popup](images/schedule-pause-popup.png) ### How to view and manage scheduled pauses Pausing jobs appear in the scheduled pause job list. To see the list of scheduled pausing jobs: 1. Go to **Backup & Restore**. 2. Select **Scheduled Pauses**. The list will display all scheduled pausing jobs with details such as namespace, release name, product, component, schedule time, timezone, and pause duration. To delete a scheduled pausing job: 1. Select **Delete** next to the scheduled pausing job. 2. Confirm the deletion when prompted. You can see specific pausing jobs by: - Using the search bar to find scheduled pauses by keyword. - Filtering the list by namespace or release. ![schedule-pauses-page](images/schedule-pauses.png) ### How to check pause results The Check Pauses page shows the results of executed pauses, providing information about when pauses were executed properly: 1. Go to **Backup & Restore**. 2. Select **Check Pauses**. The Check Pauses page is a read-only page that displays the results of executed pausing jobs, with details including namespace, release, start/end times, and timezone. You can see specific pausing jobs by: - Using the search bar to find scheduled pauses by keyword. - Filtering the list by namespace or release. :::note This page shows only the results of pauses that have been executed. To [schedule a pausing job](#how-to-schedule-pausing-jobs) or [execute a pausing job](#how-to-execute-a-pausing-job-immediately), you need to go to the Create Pauses page as described in the previous sections. ::: ![check-pauses-page](images/check-pauses.png) ================================================ FILE: docs/scalar-manager/metrics-reference.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Scalar Manager Metrics Reference This document provides a list of explanations for all metrics available in Scalar Manager, via Grafana, to help you monitor the performance, health, and operational status of your ScalarDB deployments. ## Understanding the metrics Before exploring the specific metrics, it's important to understand how they are measured: - **Rate metrics (per one second):** These metrics measure the frequency of operations, indicating how many times a specific operation occurs each second. - **Execution time metrics (percentile):** These metrics measure how long operations take to complete, presented as percentiles (p50, p90, p99). Execution times are measured in milliseconds. For example: - **p50 (median):** 50% of operations complete faster than this time - **p90:** 90% of operations complete faster than this time - **p99:** 99% of operations complete faster than this time Higher percentile values (especially p99) help identify worst-case performance scenarios that might affect user experience even when average performance seems acceptable. In the tables below, related rate and execution time metrics are grouped together for clarity. ## Total Requests | Metric | Description | |-------------------------------------|---------------------------------------------------------| | **Success Requests per one second** | The number of successful requests processed per second. | | **Failure Requests per one second** | The number of failed requests per second. | | **Create table per one second / execution time (percentile)** | Number of table creation operations per second and the time taken to execute them, measured in percentiles. | | **Drop table per one second / execution time (percentile)** | Number of table drop operations per second and the time taken to execute them, measured in percentiles. | | **Truncate table per one second / execution time (percentile)** | Number of table truncate operations per second and the time taken to execute them, measured in percentiles. | | **Get table metadata per one second / execution time (percentile)** | Number of metadata retrieval operations per second and the time taken to retrieve them, measured in percentiles. | ## Distributed Transaction Service | Metric | Description | |--------|-------------| | **Transaction begin per one second / execution time (percentile)** | Number of transaction start operations per second and the time taken to start them, measured in percentiles. | | **Transaction get per one second / execution time (percentile)** | Number of data retrieval operations within transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction scan per one second / execution time (percentile)** | Number of scan (range read) operations within transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction put per one second / execution time (percentile)** | Number of data write operations within transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction delete per one second / execution time (percentile)** | Number of data deletion operations within transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction mutate per one second / execution time (percentile)** | Number of batch mutation operations within transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction commit per one second / execution time (percentile)** | Number of transaction commit operations per second and the time taken to execute them, measured in percentiles. | | **Transaction rollback per one second / execution time (percentile)** | Number of transaction rollback operations per second and the time taken to execute them, measured in percentiles. | ## Two Phase Commit Transaction Service | Metric | Description | |--------|-------------| | **Transaction begin per one second / execution time (percentile)** | Number of 2PC transaction start operations per second and the time taken to start them, measured in percentiles. | | **Transaction join per one second / execution time (percentile)** | Number of transaction join operations per second and the time taken to execute them, measured in percentiles. | | **Transaction get per one second / execution time (percentile)** | Number of data retrieval operations within 2PC transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction scan per one second / execution time (percentile)** | Number of scan operations within 2PC transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction put per one second / execution time (percentile)** | Number of data write operations within 2PC transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction delete per one second / execution time (percentile)** | Number of data deletion operations within 2PC transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction mutate per one second / execution time (percentile)** | Number of batch mutation operations within 2PC transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction prepare per one second / execution time (percentile)** | Number of transaction prepare operations per second and the time taken to execute them, measured in percentiles. | | **Transaction validate per one second / execution time (percentile)** | Number of transaction validation operations per second and the time taken to execute them, measured in percentiles. | | **Transaction commit per one second / execution time (percentile)** | Number of 2PC transaction commit operations per second and the time taken to execute them, measured in percentiles. | | **Transaction rollback per one second / execution time (percentile)** | Number of 2PC transaction rollback operations per second and the time taken to execute them, measured in percentiles. | ## GraphQL Service | Metric | Description | |--------|-------------| | **Success/Failure HTTP Requests per one second** | Number of successful and failed HTTP requests to the GraphQL service per second. | | **Success/Failure GraphQL Queries per one second** | Number of successful and failed GraphQL queries executed per second. | | **GraphQL execution time (percentile)** | Time taken to execute GraphQL queries, measured in percentiles. | ## SQL Total Requests | Metric | Description | |-------------------------------------------------|--------------------------------------------------------------------| | **Success/Failure SQL Requests per one second** | Number of successful and failed SQL requests processed per second. | ## SQL Distributed Transaction Service | Metric | Description | |-----------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------| | **Transaction begin per one second / execution time (percentile)** | Number of SQL transaction start operations per second and the time taken to start them, measured in percentiles. | | **Transaction execute per one second / execution time (percentile)** | Number of SQL statement execution operations per second and the time taken to execute them, measured in percentiles. | | **Transaction commit per one second / execution time (percentile)** | Number of SQL transaction commit operations per second and the time taken to execute them, measured in percentiles. | | **Transaction rollback per one second / execution time (percentile)** | Number of SQL transaction rollback operations per second and the time taken to execute them, measured in percentiles. | ## SQL Two Phase Commit Transaction Service | Metric | Description | |-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------| | **Transaction begin per one second / execution time (percentile)** | Number of SQL 2PC transaction start operations per second and the time taken to start them, measured in percentiles. | | **Transaction join per one second / execution time (percentile)** | Number of SQL transaction join operations per second and the time taken to execute them, measured in percentiles. | | **Transaction execute per one second / execution time (percentile)** | Number of SQL statement execution operations within 2PC transactions per second and the time taken to execute them, measured in percentiles. | | **Transaction prepare per one second / execution time (percentile)** | Number of SQL transaction prepare operations per second and the time taken to execute them, measured in percentiles. | | **Transaction validate per one second / execution time (percentile)** | Number of SQL transaction validation operations per second and the time taken to execute them, measured in percentiles. | | **Transaction commit per one second / execution time (percentile)** | Number of SQL 2PC transaction commit operations per second and the time taken to execute them, measured in percentiles. | | **Transaction rollback per one second / execution time (percentile)** | Number of SQL 2PC transaction rollback operations per second and the time taken to execute them, measured in percentiles. | ## SQL Metadata Service | Metric | Description | |-----------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------| | **Get namespace metadata per one second / execution time (percentile)** | Number of namespace metadata retrieval operations per second and the time taken to retrieve them, measured in percentiles. | | **Get table metadata per one second / execution time (percentile)** | Number of SQL table metadata retrieval operations per second and the time taken to retrieve them, measured in percentiles. | | **List table metadata in namespace per one second / execution time (percentile)** | Number of operations listing all tables in a namespace per second and the time taken to list them, measured in percentiles. | ================================================ FILE: docs/scalar-manager/overview.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Scalar Manager Overview Scalar Manager is a centralized management and monitoring solution for ScalarDB within Kubernetes cluster environments. It simplifies the operational tasks associated with these products by aggregating essential functionalities into a graphical user interface (GUI). ## Why Scalar Manager? Before Scalar Manager was released, you would need to use various command-line tools and third-party solutions individually to manage and monitor ScalarDB deployments. For example, `kubectl` is often used to check deployment status, the Prometheus stack for monitoring metrics, the Loki stack for log analysis, and Scalar's proprietary CLI tool for pausing ScalarDB to ensure transactional consistency between multiple databases. This constellation of tools presented a steep learning curve and lacked a unified interface, resulting in inefficient workflows for performing routine management tasks or troubleshooting issues. Scalar Manager mitigates these pain points by aggregating essential functionalities into a single, user-friendly GUI. With Scalar Manager, you can reduce the time and effort needed for management and monitoring, allowing you to focus on business development and operations. ## Key features At its core, Scalar Manager provides the following features. ### Centralized cluster visualization You can quickly gain real-time metrics about cluster health, pod logs, hardware usage, performance metrics like requests per second, and deep visibility into time-series data via the Grafana dashboards. ![dashboard-cluster](images/dashboard-cluster.png) ![dashboard-pod-list](images/dashboard-pod-list.png) With the Grafana dashboards, you can also view pod logs and metrics in real-time or in time series. ![logs](images/logs.png) ![metrics](images/metrics2.png) ### Streamlined pausing job management You can execute or schedule pausing jobs to ensure transactional consistency, review and manage scheduled jobs, and monitor paused states within an intuitive GUI. ![create-pauses](images/backup-and-restore-create-pauses.png) ![check-pauses](images/backup-and-restore-check-pauses.png) ### User management Scalar Manager includes authentication capabilities, allowing for secure access control to your deployment. The system provides user management functionalities that enable administrators to create, modify, and remove user accounts through an intuitive interface. ### Authentication and authorization By using the authorization feature, administrators can define and assign specific roles to users, controlling their access permissions within the Scalar Manager environment. This control ensures that users only have access to the functionalities relevant to their responsibilities. ### Integrated authentication with Grafana Scalar Manager now offers seamless authentication integration between your Grafana instance and other components of the system. This single-sign-on capability eliminates the need for multiple authentication processes, streamlining the user experience and enhancing security by reducing credential management overhead. ## Required port Scalar Manager requires port 13000 to be accessible. ================================================ FILE: docs/scalardb-analytics/_README.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # ScalarDB Analytics import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; **ScalarDB Analytics** is the analytical component of ScalarDB. Similar to ScalarDB, it unifies diverse data sources - ranging from RDBMSs like PostgreSQL and MySQL to NoSQL databases such as Cassandra and DynamoDB - into a single logical database. While ScalarDB focuses on operational workloads with strong transactional consistency across multiple databases, ScalarDB Analytics is optimized for analytical workloads. It supports a wide range of queries, including complex joins, aggregations, and window functions. ScalarDB Analytics operates seamlessly on both ScalarDB-managed data sources and non-ScalarDB-managed ones, enabling advanced analytical queries across various datasets. The current version of ScalarDB Analytics leverages **Apache Spark** as its execution engine. It provides a unified view of ScalarDB-managed and non-ScalarDB-managed data sources by utilizing a Spark custom catalog. Using ScalarDB Analytics, you can treat tables from these data sources as native Spark tables. This allows you to execute arbitrary Spark SQL queries seamlessly. For example, you can join a table stored in Cassandra with a table in PostgreSQL to perform a cross-database analysis with ease. ## Further reading This section provides links to various ScalarDB Analytics–related documentation. ### Getting started * [Getting Started with ScalarDB Analytics](./quickstart.mdx) - A quick tutorial to set up ScalarDB Analytics and run federated queries ### Key documentation * [Deploy ScalarDB Analytics in Public Cloud Environments](./deployment.mdx) - Deploy on Amazon EMR, Databricks, and other platforms * [Create a ScalarDB Analytics Catalog](./create-scalardb-analytics-catalog.mdx) - Create catalogs and add data sources * [Run Analytical Queries Through ScalarDB Analytics](./run-analytical-queries.mdx) - Execute queries across multiple databases * [ScalarDB Analytics Configurations](./configurations.mdx) - Configure Spark and data sources ### Technical details * [ScalarDB Analytics Design](./design.mdx) - Deep dive into the technical architecture * [Spark](../requirements.mdx#spark) - Supported Spark and Scala versions ================================================ FILE: docs/scalardb-analytics/authentication-and-authorization.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Authenticate and Authorize Users in ScalarDB Analytics ScalarDB Analytics can authenticate and authorize users with resource-level access control. You can create users and grant or revoke permissions on catalogs, data sources, namespaces, and tables. Roles can also be created to group permissions and can be granted to users. This guide describes how to use authentication and authorization in ScalarDB Analytics. ## Authentication ScalarDB Analytics authenticates users by verifying their username and password against a configured authentication backend. After successful authentication, the server issues an access token that the client uses for subsequent requests. Each user can have only one active access token at a time; authenticating again invalidates the previous token. ### Authentication backends ScalarDB Analytics supports the following authentication backends: - **Internal:** Manages user credentials directly within ScalarDB Analytics. Users are registered by using the `user register` CLI command, and their passwords are stored (hashed) in the ScalarDB Analytics metadata database. This is the default backend. - **ScalarDB Cluster:** Delegates credential verification to an external ScalarDB Cluster instance. When a user logs in for the first time, ScalarDB Analytics automatically creates a corresponding user record (just-in-time provisioning). The `user register` and `user unregister` commands are not available with this backend because the user lifecycle is managed in ScalarDB Cluster. You can configure the authentication backend by using the `scalar.db.analytics.server.auth.password.backend` server property. For details, see [Configurations](#configurations). ### Users A user is an entity that can authenticate with ScalarDB Analytics and perform operations based on assigned permissions. Users can log in to ScalarDB Analytics with a username and a password and execute operations if they have the required permissions. Authentication and authorization support two types of users: - **Superusers:** This type of user has all permissions. Only superusers can create catalogs and manage users and roles. - **Normal users:** This type of user initially doesn't have any permissions, so they need to be granted permissions by a superuser or a user with catalog-level admin permission. ### Initial admin user When you enable authentication and authorization, ScalarDB Analytics creates an initial admin user and assigns a built-in superuser role to that user. The behavior differs depending on the authentication backend. #### Internal backend When the server starts, ScalarDB Analytics creates the user specified by `scalar.db.analytics.server.auth.initial_admin_username` with the password specified by `scalar.db.analytics.server.auth.password.internal.initial_admin_password` and assigns the SUPERADMIN role to that user. This assignment occurs only if no users are already assigned to the SUPERADMIN role. You can log in with this user and create other users if necessary. #### ScalarDB Cluster backend The server does not create any users at startup. Instead, when the user specified by `scalar.db.analytics.server.auth.initial_admin_username` logs in for the first time, ScalarDB Analytics automatically registers that user through just-in-time provisioning and assigns the SUPERADMIN role. This assignment occurs only if no users are already assigned to the SUPERADMIN role. Other users are also provisioned on first login, but they are not assigned the SUPERADMIN role. ### User management When using the internal backend, you can register and unregister users by using the CLI: ```console scalardb-analytics-cli user register -u alice -p scalardb-analytics-cli user unregister -u alice ``` The following commands are available regardless of the authentication backend: ```console scalardb-analytics-cli user list scalardb-analytics-cli user describe -u alice ``` ## Authorization ScalarDB Analytics provides resource-level access control through roles and permissions. This section describes how to manage roles, permissions, and access control. ### Roles A role is a named collection of permissions that can be granted to users. Using roles provides a convenient way to manage permissions for multiple users, rather than granting individual permissions to each user. Only superusers can create, delete, or list roles. Only superusers can assign roles to users or remove role assignments. #### SUPERADMIN role SUPERADMIN is a built-in role that is automatically created when the server starts. Users assigned to the SUPERADMIN role bypass all access control checks and can perform any operation. In API responses, users assigned to the SUPERADMIN role are represented with the `is_superuser` flag set to `true`. #### Role management You can manage roles by using the following CLI commands: ```console scalardb-analytics-cli role create -n analyst scalardb-analytics-cli role delete -n analyst scalardb-analytics-cli role list scalardb-analytics-cli role grant -r analyst -u alice scalardb-analytics-cli role revoke -r analyst -u alice ``` ### Permissions ScalarDB Analytics uses resource-level access control. Permissions are granted on a specific resource and apply to the operations associated with the resource type. When granting or revoking permissions through the CLI, you specify the resource type as a subcommand and the permission level with the `-p` option. The valid `-p` values depend on the resource type: - **Catalog:** `read`, `write`, or `admin` - **Data source:** `read` or `admin` - **Namespace:** `read` - **Table:** `read` The following permissions are available: | Permission | Resource type | `-p` value | Description | |:--------------------|:--------------|:-----------|:-----------------------------------------------------------------------------------| | `CATALOG_READ` | Catalog | `read` | Read catalog metadata. | | `CATALOG_WRITE` | Catalog | `write` | Create or modify resources within a catalog (for example, register a data source). | | `CATALOG_ADMIN` | Catalog | `admin` | Delete a catalog and manage permissions on resources within the catalog. | | `DATA_SOURCE_READ` | Data source | `read` | Read data source metadata. | | `DATA_SOURCE_ADMIN` | Data source | `admin` | Delete a data source. | | `NAMESPACE_READ` | Namespace | `read` | Read namespace metadata. | | `TABLE_READ` | Table | `read` | Read table metadata. | #### Resource hierarchy Resources in ScalarDB Analytics are organized in a hierarchy. Permissions granted at a higher level in the hierarchy apply to all resources below that level. ``` System (SUPERADMIN only) └── Catalog └── Data source └── Namespace └── Table ``` Permissions granted at a higher level propagate to all resources below that level. For example, granting `CATALOG_READ` on a catalog allows the user to read metadata for all data sources, namespaces, and tables within that catalog. However, the specific permissions required for each operation are defined individually. For details, see [Which permissions are required for each type of operation](#which-permissions-are-required-for-each-type-of-operation). #### Direct and role-based grants Permissions can be granted in two ways: - **Direct grants:** A permission is granted directly to a specific user on a specific resource. - **Role-based grants:** A permission is granted to a role on a specific resource. All users assigned to that role receive the permission. A user's effective permissions are the union of all directly granted permissions and all permissions from assigned roles. #### Permission management You can grant and revoke permissions by specifying the resource type, the target resource, and the permission level. Permissions can be granted to a user (`--user`) or a role (`--role`). Grant a permission to a user: ```console scalardb-analytics-cli permission grant catalog --catalog my_catalog --user alice -p read ``` Grant a permission to a role: ```console scalardb-analytics-cli permission grant catalog --catalog my_catalog --role analyst -p read ``` For data sources, namespaces, and tables, specify the parent resources: ```console scalardb-analytics-cli permission grant data-source \ --catalog my_catalog --data-source my_ds --user alice -p read scalardb-analytics-cli permission grant namespace \ --catalog my_catalog --data-source my_ds --namespace my_ns --user alice -p read scalardb-analytics-cli permission grant table \ --catalog my_catalog --data-source my_ds --namespace my_ns --table my_tbl \ --user alice -p read ``` Revoke a permission: ```console scalardb-analytics-cli permission revoke catalog --catalog my_catalog --user alice -p read ``` List the effective permissions for a user: ```console scalardb-analytics-cli permission list -u alice ``` Users who hold `CATALOG_ADMIN` on a catalog can grant and revoke permissions on any resource within that catalog. Superusers can manage permissions on all resources. #### Which permissions are required for each type of operation The following table shows which permissions satisfy the authorization requirement for each operation. For operations that involve a resource hierarchy, the system checks permissions at each level, starting from the target resource and moving up to the catalog level. A match at any level is sufficient. :::note When [ScalarDB authorization delegation](#scalardb-authorization-delegation) is enabled, namespace-level and table-level authorization for ScalarDB data sources (databases that ScalarDB Cluster manages) is delegated to ScalarDB Cluster instead of following the permissions listed in the table below. ::: | Operation | Satisfying permissions | |:--------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------| | Create a catalog | SUPERADMIN only | | Find a catalog | `CATALOG_READ` or `CATALOG_ADMIN` on the catalog | | List catalogs | Same as find a catalog (results are filtered) | | Initialize a catalog | `CATALOG_WRITE` or `CATALOG_ADMIN` on the catalog | | Delete a catalog | `CATALOG_ADMIN` on the catalog | | Register a data source | `CATALOG_WRITE` or `CATALOG_ADMIN` on the parent catalog | | Find a data source | `DATA_SOURCE_READ` or `DATA_SOURCE_ADMIN` on the data source, or `CATALOG_READ` or `CATALOG_ADMIN` on the parent catalog | | List data sources | Same as find a data source (results are filtered) | | Delete a data source | `DATA_SOURCE_ADMIN` on the data source, or `CATALOG_ADMIN` on the parent catalog | | Describe a namespace | `NAMESPACE_READ` on the namespace, `DATA_SOURCE_READ` or `DATA_SOURCE_ADMIN` on the parent data source, or `CATALOG_READ` or `CATALOG_ADMIN` on the parent catalog | | List namespaces | Same as describe a namespace (results are filtered) | | Describe a table | `TABLE_READ` on the table, `NAMESPACE_READ` on the parent namespace, `DATA_SOURCE_READ` or `DATA_SOURCE_ADMIN` on the parent data source, or `CATALOG_READ` or `CATALOG_ADMIN` on the parent catalog | | List tables | Same as describe a table (results are filtered) | | Register or unregister a user | SUPERADMIN only | | Create, delete, or list roles | SUPERADMIN only | | Grant or revoke a role | SUPERADMIN only | | Grant or revoke a permission | `CATALOG_ADMIN` on the target catalog, or SUPERADMIN | | List permissions | Same as the corresponding read operation for the resource | ### ScalarDB authorization delegation When you use ScalarDB Cluster as the authentication backend, you can optionally delegate namespace-level and table-level authorization for ScalarDB data sources (databases that ScalarDB Cluster manages) to ScalarDB Cluster. This means that the privilege system in ScalarDB Cluster becomes the single source of truth for access control on ScalarDB data, eliminating the need to manage permissions in two places. #### How delegation works When authorization delegation is enabled, ScalarDB Analytics checks ScalarDB Cluster privileges (specifically, the `READ` privilege) instead of its own access control entries for namespace and table operations on ScalarDB data sources. Catalog-level and data source-level authorization remains under ScalarDB Analytics access control regardless of delegation settings. #### Prerequisites To use ScalarDB authorization delegation, the following conditions must be met: - The authentication backend must be set to `scalardb-cluster`. - The `scalar.db.analytics.server.auth.password.scalardb_cluster.acl_delegation` property must be set to `true`. - The user must have been authenticated through the ScalarDB Cluster backend. #### Authorization responsibility by resource level The following table shows which system is responsible for authorization at each resource level when authorization delegation is enabled or disabled. | Resource level | Delegation enabled | Delegation disabled | |:------------------------------|:-------------------|:--------------------| | Catalog | ScalarDB Analytics | ScalarDB Analytics | | Data source | ScalarDB Analytics | ScalarDB Analytics | | Namespace (managed by ScalarDB Cluster) | ScalarDB Cluster | ScalarDB Analytics | | Table (managed by ScalarDB Cluster) | ScalarDB Cluster | ScalarDB Analytics | | Namespace (not managed by ScalarDB Cluster) | ScalarDB Analytics | ScalarDB Analytics | | Table (not managed by ScalarDB Cluster) | ScalarDB Analytics | ScalarDB Analytics | :::note Only ScalarDB data sources support authorization delegation. Other data source types always use ScalarDB Analytics access control. ::: ## Configurations This section describes the available configurations for authentication and authorization. ### ScalarDB Analytics server configurations To enable authentication and authorization, you need to set `scalar.db.analytics.server.auth.enabled` to `true`. #### `enabled` - **Field:** `scalar.db.analytics.server.auth.enabled` - **Description:** Whether authentication and authorization are enabled. - **Default value:** `false` You can also set the following configurations: #### `initial_admin_username` - **Field:** `scalar.db.analytics.server.auth.initial_admin_username` - **Description:** The username of the initial admin user who will be assigned the SUPERADMIN role. - **Default value:** None #### `password.backend` - **Field:** `scalar.db.analytics.server.auth.password.backend` - **Description:** The authentication backend to use (`internal` or `scalardb-cluster`). - **Default value:** `internal` #### `password.token_ttl_seconds` - **Field:** `scalar.db.analytics.server.auth.password.token_ttl_seconds` - **Description:** The time-to-live (TTL) of access tokens in seconds. - **Default value:** `86400` #### `password.internal.initial_admin_password` - **Field:** `scalar.db.analytics.server.auth.password.internal.initial_admin_password` - **Description:** The initial password for the admin user. Only used when `auth.password.backend` is set to `internal`. - **Default value:** None If you use `scalardb-cluster` as the authentication backend, you can also set the following configurations: #### `password.scalardb_cluster.host` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.host` - **Description:** The hostname or IP address of the ScalarDB Cluster instance. - **Default value:** `localhost` #### `password.scalardb_cluster.port` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.port` - **Description:** The port number of the ScalarDB Cluster instance. - **Default value:** `60053` #### `password.scalardb_cluster.deadline_millis` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.deadline_millis` - **Description:** The gRPC deadline in milliseconds for authentication requests to ScalarDB Cluster. - **Default value:** `5000` #### `password.scalardb_cluster.acl_delegation` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.acl_delegation` - **Description:** Whether to delegate namespace-level and table-level authorization for ScalarDB data sources to ScalarDB Cluster. For details, see [ScalarDB authorization delegation](#scalardb-authorization-delegation). - **Default value:** `false` #### `password.scalardb_cluster.tls.enabled` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.tls.enabled` - **Description:** Whether to enable TLS for the connection to ScalarDB Cluster. - **Default value:** `false` #### `password.scalardb_cluster.tls.ca_root_cert_path` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.tls.ca_root_cert_path` - **Description:** Path to the CA root certificate file for verifying the ScalarDB Cluster server certificate. - **Default value:** None #### `password.scalardb_cluster.tls.override_authority` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.tls.override_authority` - **Description:** Override the server authority for TLS verification of the ScalarDB Cluster connection. - **Default value:** None ### CLI client configurations To authenticate with the ScalarDB Analytics server, configure the client credentials by using a properties file or environment variables. #### `username` - **Field:** `scalar.db.analytics.client.auth.username` - **Description:** The username for authenticating with the ScalarDB Analytics server. - **Default value:** None #### `password` - **Field:** `scalar.db.analytics.client.auth.password` - **Description:** The password for authenticating with the ScalarDB Analytics server. - **Default value:** None You can also set credentials by using environment variables, which take precedence over configuration file properties. #### `SCALAR_DB_ANALYTICS_CLIENT_AUTH_USERNAME` - **Description:** The username for authentication. #### `SCALAR_DB_ANALYTICS_CLIENT_AUTH_PASSWORD` - **Description:** The password for authentication. ### Spark configurations To authenticate Spark applications with the ScalarDB Analytics server, add the following properties to your Spark configuration. Replace `` with the name of your catalog. #### `username` - **Field:** `spark.sql.catalog..server.auth.username` - **Description:** The username for authenticating the Spark application with the ScalarDB Analytics server. - **Default value:** None #### `password` - **Field:** `spark.sql.catalog..server.auth.password` - **Description:** The password for authenticating the Spark application with the ScalarDB Analytics server. - **Default value:** None ## Wire encryption If you enable authentication and authorization, you must also enable TLS to protect user credentials in transit. For details about TLS configuration, see [ScalarDB Analytics Configurations](configurations.mdx). ## Next steps - [ScalarDB Analytics Configurations](configurations.mdx) - View all available configuration options - [Create a ScalarDB Analytics Catalog](create-scalardb-analytics-catalog.mdx) - Learn how to create catalogs and add data sources ================================================ FILE: docs/scalardb-analytics/configurations.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # ScalarDB Analytics Configurations This page provides a comprehensive reference for configuring all components of ScalarDB Analytics. ## Overview ScalarDB Analytics consists of three main components that require configuration: 1. **ScalarDB Analytics server** - The server that hosts the catalog information and metering services 2. **CLI client** - The command-line interface for managing catalogs and data sources 3. **Spark integration** - Configuration for using ScalarDB Analytics with Apache Spark ## ScalarDB Analytics server configuration The server is configured using a standard Java properties file (for example, `scalardb-analytics-server.properties`) that defines database connections, network settings, licensing, and optional features. ### Metadata database configurations Configure the metadata database that stores catalog information. #### `db.contact_points` - **Field:** `scalar.db.analytics.server.db.contact_points` - **Description:** The JDBC URL for the metadata database used by ScalarDB Analytics. #### `db.username` - **Field:** `scalar.db.analytics.server.db.username` - **Description:** The username for connecting to the metadata database. #### `db.password` - **Field:** `scalar.db.analytics.server.db.password` - **Description:** The password for the metadata database user. ### Server network configuration Configure network settings including service ports and TLS/SSL encryption. #### `catalog.port` - **Field:** `scalar.db.analytics.server.catalog.port` - **Description:** Port for the catalog service. - **Default value:** `11051` #### `metering.port` - **Field:** `scalar.db.analytics.server.metering.port` - **Description:** Port for the metering service. - **Default value:** `11052` #### `tls.enabled` - **Field:** `scalar.db.analytics.server.tls.enabled` - **Description:** Enable TLS/SSL for secure communication. - **Default value:** `false` #### `tls.cert_chain_path` - **Field:** `scalar.db.analytics.server.tls.cert_chain_path` - **Description:** Path to the server certificate chain file. Required when `tls.enabled` is `true`. #### `tls.private_key_path` - **Field:** `scalar.db.analytics.server.tls.private_key_path` - **Description:** Path to the server private key file. Required when `tls.enabled` is `true`. ### License configuration Configure your ScalarDB Analytics license. #### `licensing.license_key` - **Field:** `scalar.db.analytics.server.licensing.license_key` - **Description:** Your ScalarDB Analytics license key. #### `licensing.license_check_cert_pem` - **Field:** `scalar.db.analytics.server.licensing.license_check_cert_pem` - **Description:** License verification certificate as PEM string. Either this or `license_check_cert_path` must be specified. #### `licensing.license_check_cert_path` - **Field:** `scalar.db.analytics.server.licensing.license_check_cert_path` - **Description:** Path to license verification certificate file. Either this or `license_check_cert_pem` must be specified. ### Authentication and authorization configurations Configure authentication and authorization for the ScalarDB Analytics server. For more details, see [Authenticate and Authorize Users in ScalarDB Analytics](authentication-and-authorization.mdx). #### `auth.enabled` - **Field:** `scalar.db.analytics.server.auth.enabled` - **Description:** Whether authentication and authorization are enabled. - **Default value:** `false` #### `auth.initial_admin_username` - **Field:** `scalar.db.analytics.server.auth.initial_admin_username` - **Description:** The username of the initial admin user who will be assigned the SUPERADMIN role when no users have already been assigned to the SUPERADMIN role. #### `auth.password.backend` - **Field:** `scalar.db.analytics.server.auth.password.backend` - **Description:** The authentication backend to use (`internal` or `scalardb-cluster`). - **Default value:** `internal` #### `auth.password.token_ttl_seconds` - **Field:** `scalar.db.analytics.server.auth.password.token_ttl_seconds` - **Description:** The time-to-live (TTL) of access tokens in seconds. - **Default value:** `86400` (24 hours) #### `auth.password.internal.initial_admin_password` - **Field:** `scalar.db.analytics.server.auth.password.internal.initial_admin_password` - **Description:** The initial password for the admin user. Only used when `auth.password.backend` is `internal`. #### `auth.password.scalardb_cluster.host` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.host` - **Description:** The hostname or IP address of the ScalarDB Cluster instance. Required when `auth.password.backend` is `scalardb-cluster`. - **Default value:** `localhost` #### `auth.password.scalardb_cluster.port` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.port` - **Description:** The port number of the ScalarDB Cluster instance. Required when `auth.password.backend` is `scalardb-cluster`. - **Default value:** `60053` #### `auth.password.scalardb_cluster.deadline_millis` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.deadline_millis` - **Description:** The gRPC deadline in milliseconds for authentication requests to ScalarDB Cluster. - **Default value:** `5000` #### `auth.password.scalardb_cluster.acl_delegation` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.acl_delegation` - **Description:** Whether to delegate namespace-level and table-level authorization for ScalarDB data sources to ScalarDB Cluster. For details, see [Authenticate and Authorize Users in ScalarDB Analytics](authentication-and-authorization.mdx#scalardb-authorization-delegation). - **Default value:** `false` #### `auth.password.scalardb_cluster.tls.enabled` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.tls.enabled` - **Description:** Whether to enable TLS for the connection to ScalarDB Cluster. - **Default value:** `false` #### `auth.password.scalardb_cluster.tls.ca_root_cert_path` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.tls.ca_root_cert_path` - **Description:** Path to the CA root certificate file for verifying the ScalarDB Cluster server certificate. #### `auth.password.scalardb_cluster.tls.override_authority` - **Field:** `scalar.db.analytics.server.auth.password.scalardb_cluster.tls.override_authority` - **Description:** Override the server authority for TLS verification of the ScalarDB Cluster connection. ### Metering storage configuration Configure storage for metering data. #### `metering.storage.provider` - **Field:** `scalar.db.analytics.server.metering.storage.provider` - **Description:** Storage provider for metering data (`filesystem`, `aws-s3`, `azureblob`, `google-cloud-storage`). #### `metering.storage.containerName` - **Field:** `scalar.db.analytics.server.metering.storage.containerName` - **Description:** Container/bucket name for cloud storage. - **Default value:** `metering` #### `metering.storage.path` - **Field:** `scalar.db.analytics.server.metering.storage.path` - **Description:** Local directory path. Required when provider is `filesystem`. #### `metering.storage.accessKeyId` - **Field:** `scalar.db.analytics.server.metering.storage.accessKeyId` - **Description:** Access key ID for cloud storage providers. Required for `aws-s3`, `azureblob`, and `google-cloud-storage`. #### `metering.storage.secretAccessKey` - **Field:** `scalar.db.analytics.server.metering.storage.secretAccessKey` - **Description:** Secret access key for cloud storage providers. Required for `aws-s3`, `azureblob`, and `google-cloud-storage`. #### `metering.storage.prefix` - **Field:** `scalar.db.analytics.server.metering.storage.prefix` - **Description:** Optional prefix for all storage paths. ## CLI client configuration The CLI client requires connection settings to communicate with the ScalarDB Analytics server using a Java properties file (for example, `client.properties`). ## Configuration properties This section describes the configuration properties. ### Server connection configuration The following is a list of configurations for connecting to the server. #### `server.host` - **Field:** `scalar.db.analytics.client.server.host` - **Description:** Hostname or IP address of the ScalarDB Analytics server. #### `server.catalog.port` - **Field:** `scalar.db.analytics.client.server.catalog.port` - **Description:** Port number for the catalog service. - **Default value:** `11051` #### `server.metering.port` - **Field:** `scalar.db.analytics.client.server.metering.port` - **Description:** Port number for the metering service. - **Default value:** `11052` ### Authentication configuration The following is a list of configurations for authentication. For more details, see [Authenticate and Authorize Users in ScalarDB Analytics](authentication-and-authorization.mdx). #### `auth.username` - **Field:** `scalar.db.analytics.client.auth.username` - **Description:** The username for authenticating with the ScalarDB Analytics server. #### `auth.password` - **Field:** `scalar.db.analytics.client.auth.password` - **Description:** The password for authenticating with the ScalarDB Analytics server. :::tip You can also set the credentials by using environment variables, which take precedence over the configuration file properties. | Environment variable | Description | |:----------------------------------------------|:-------------------------------| | `SCALAR_DB_ANALYTICS_CLIENT_AUTH_USERNAME` | The username for authentication. | | `SCALAR_DB_ANALYTICS_CLIENT_AUTH_PASSWORD` | The password for authentication. | ::: ### TLS configuration The following is a list of configurations for TLS. #### `server.tls.enabled` - **Field:** `scalar.db.analytics.client.server.tls.enabled` - **Description:** Enable TLS/SSL for server connections. - **Default value:** `false` #### `server.tls.ca_root_cert_path` - **Field:** `scalar.db.analytics.client.server.tls.ca_root_cert_path` - **Description:** Path to the CA certificate file for verifying server certificates. Required when `tls.enabled` is `true`. #### `server.tls.override_authority` - **Field:** `scalar.db.analytics.client.server.tls.override_authority` - **Description:** Override the server authority for TLS verification (useful for testing). ## Spark integration configuration To use ScalarDB Analytics with Apache Spark, configure your Spark application by adding the necessary settings to your Spark configuration file (`spark-defaults.conf`). ### Spark Core configuration The following is a list of configurations for Spark Core. #### `spark.jars.packages` - **Field:** `spark.jars.packages` - **Description:** Maven coordinates for ScalarDB Analytics dependencies. #### `spark.extraListeners` - **Field:** `spark.extraListeners` - **Description:** Register the ScalarDB Analytics metering listener. ### Catalog configuration The following is a list of configurations for the catalog. #### `spark.sql.catalog.` - **Field:** `spark.sql.catalog.` - **Description:** Register the ScalarDB Analytics catalog implementation. Replace `` with the exact name of the catalog created in ScalarDB Analytics server. Use `com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog` as the value. :::important The `` must match the catalog name created in ScalarDB Analytics server using the CLI. For example, if you created a catalog named `production` in the server, use `spark.sql.catalog.production`. ::: ### Server connection configurations The following is a list of configurations for the server connection. #### `spark.sql.catalog..server.host` - **Field:** `spark.sql.catalog..server.host` - **Description:** Hostname or IP address of the ScalarDB Analytics server. #### `spark.sql.catalog..server.catalog.port` - **Field:** `spark.sql.catalog..server.catalog.port` - **Description:** Port number for the catalog service. - **Default value:** `11051` #### `spark.sql.catalog..server.metering.port` - **Field:** `spark.sql.catalog..server.metering.port` - **Description:** Port number for the metering service. - **Default value:** `11052` ### TLS/SSL configurations The following is a list of configurations for TLS/SSL. #### `spark.sql.catalog..server.tls.enabled` - **Field:** `spark.sql.catalog..server.tls.enabled` - **Description:** Enable TLS/SSL for server connections. - **Default value:** `false` #### `spark.sql.catalog..server.tls.ca_root_cert_path` - **Field:** `spark.sql.catalog..server.tls.ca_root_cert_path` - **Description:** Path to the CA certificate file for verifying server certificates. Required when `tls.enabled` is `true`. #### `spark.sql.catalog..server.tls.override_authority` - **Field:** `spark.sql.catalog..server.tls.override_authority` - **Description:** Override the server authority for TLS verification. Replace `` with your chosen catalog name (for example, `analytics`). ### Authentication configurations The following is a list of configurations for authentication when connecting to a server with authentication enabled. For more details, see [Authenticate and Authorize Users in ScalarDB Analytics](authentication-and-authorization.mdx). #### `spark.sql.catalog..server.auth.username` - **Field:** `spark.sql.catalog..server.auth.username` - **Description:** The username for authenticating the Spark application with the ScalarDB Analytics server. #### `spark.sql.catalog..server.auth.password` - **Field:** `spark.sql.catalog..server.auth.password` - **Description:** The password for authenticating the Spark application with the ScalarDB Analytics server. ## Configuration examples This section provides some configuration examples. ### Basic development configuration The following are examples of configurations for the server, CLI client, and Spark. #### Server configuration (`scalardb-analytics-server.properties`) ```properties # Metadata database scalar.db.analytics.server.db.contact_points=jdbc:postgresql://localhost:5432/scalardb_analytics scalar.db.analytics.server.db.username=dev_user scalar.db.analytics.server.db.password=dev_password # License scalar.db.analytics.server.licensing.license_key=YOUR_DEV_LICENSE_KEY scalar.db.analytics.server.licensing.license_check_cert_path=/path/to/license_cert.pem # Metering storage (filesystem for development) scalar.db.analytics.server.metering.storage.provider=filesystem scalar.db.analytics.server.metering.storage.path=/tmp/scalardb-analytics-metering ``` #### CLI client configuration (`client.properties`) ```properties scalar.db.analytics.client.server.host=localhost ``` #### Spark configuration (`spark-defaults.conf`) ```properties spark.jars.packages com.scalar-labs:scalardb-analytics-spark-all-3.5_2.12:3.18.0 spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener spark.sql.catalog.analytics com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog spark.sql.catalog.analytics.server.host localhost ``` ### Configuration with authentication enabled The following are examples of configurations with authentication and authorization enabled. #### Server configuration (`scalardb-analytics-server.properties`) ```properties # Metadata database scalar.db.analytics.server.db.contact_points=jdbc:postgresql://localhost:5432/scalardb_analytics scalar.db.analytics.server.db.username=dev_user scalar.db.analytics.server.db.password=dev_password # Authentication and authorization scalar.db.analytics.server.auth.enabled=true scalar.db.analytics.server.auth.initial_admin_username= scalar.db.analytics.server.auth.password.internal.initial_admin_password= # TLS (required when authentication is enabled) scalar.db.analytics.server.tls.enabled=true scalar.db.analytics.server.tls.cert_chain_path=/path/to/server.crt scalar.db.analytics.server.tls.private_key_path=/path/to/server.key # License scalar.db.analytics.server.licensing.license_key=YOUR_DEV_LICENSE_KEY scalar.db.analytics.server.licensing.license_check_cert_path=/path/to/license_cert.pem # Metering storage (filesystem for development) scalar.db.analytics.server.metering.storage.provider=filesystem scalar.db.analytics.server.metering.storage.path=/tmp/scalardb-analytics-metering ``` #### CLI client configuration (`client.properties`) ```properties scalar.db.analytics.client.server.host=localhost scalar.db.analytics.client.server.tls.enabled=true scalar.db.analytics.client.server.tls.ca_root_cert_path=/path/to/ca.crt scalar.db.analytics.client.auth.username= scalar.db.analytics.client.auth.password= ``` ### Configuration with ScalarDB Cluster authentication backend The following is an example of a server configuration that uses ScalarDB Cluster as the authentication backend with ACL delegation enabled. #### Server configuration (`scalardb-analytics-server.properties`) ```properties # Metadata database scalar.db.analytics.server.db.contact_points=jdbc:postgresql://db.internal:5432/scalardb_analytics scalar.db.analytics.server.db.username=analytics_user scalar.db.analytics.server.db.password=your_secure_password # Authentication and authorization scalar.db.analytics.server.auth.enabled=true scalar.db.analytics.server.auth.initial_admin_username=admin scalar.db.analytics.server.auth.password.backend=scalardb-cluster scalar.db.analytics.server.auth.password.scalardb_cluster.host=cluster.example.com scalar.db.analytics.server.auth.password.scalardb_cluster.port=60053 scalar.db.analytics.server.auth.password.scalardb_cluster.acl_delegation=true # TLS (required when authentication is enabled) scalar.db.analytics.server.tls.enabled=true scalar.db.analytics.server.tls.cert_chain_path=/path/to/server.crt scalar.db.analytics.server.tls.private_key_path=/path/to/server.key # License scalar.db.analytics.server.licensing.license_key=YOUR_LICENSE_KEY scalar.db.analytics.server.licensing.license_check_cert_path=/path/to/license_cert.pem # Metering storage scalar.db.analytics.server.metering.storage.provider=filesystem scalar.db.analytics.server.metering.storage.path=/tmp/scalardb-analytics-metering ``` ### Production configuration with TLS The following are examples of configurations for TLS, CLI client, and Spark in production environments. #### Server configuration (`scalardb-analytics-server.properties`) ```properties # Metadata database scalar.db.analytics.server.db.contact_points=jdbc:postgresql://db.internal:5432/scalardb_analytics_prod scalar.db.analytics.server.db.username=analytics_prod scalar.db.analytics.server.db.password=your_secure_password # gRPC ports scalar.db.analytics.server.catalog.port=11051 scalar.db.analytics.server.metering.port=11052 # TLS scalar.db.analytics.server.tls.enabled=true scalar.db.analytics.server.tls.cert_chain_path=/path/to/server.crt scalar.db.analytics.server.tls.private_key_path=/path/to/server.key # License scalar.db.analytics.server.licensing.license_key=YOUR_LICENSE_KEY scalar.db.analytics.server.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIID...certificate content...\n-----END CERTIFICATE----- # Metering storage (S3) scalar.db.analytics.server.metering.storage.provider=aws-s3 scalar.db.analytics.server.metering.storage.containerName=analytics-metering scalar.db.analytics.server.metering.storage.accessKeyId=AKIAIOSFODNN7EXAMPLE scalar.db.analytics.server.metering.storage.secretAccessKey=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY scalar.db.analytics.server.metering.storage.prefix=prod/ ``` #### CLI Client configuration (`client.properties`) ```properties scalar.db.analytics.client.server.host=analytics.example.com scalar.db.analytics.client.server.tls.enabled=true scalar.db.analytics.client.server.tls.ca_root_cert_path=/path/to/cert.pem ``` #### Spark configuration (`spark-defaults.conf`) ```properties spark.jars.packages com.scalar-labs:scalardb-analytics-spark-all-3.5_2.12:3.18.0 spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener spark.sql.catalog.analytics com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog spark.sql.catalog.analytics.server.host analytics.example.com spark.sql.catalog.analytics.server.tls.enabled true spark.sql.catalog.analytics.server.tls.ca_root_cert_path /path/to/cert.pem ``` ## Next steps - [Authenticate and Authorize Users in ScalarDB Analytics](authentication-and-authorization.mdx) - Set up authentication and authorization - [Create a ScalarDB Analytics Catalog](./create-scalardb-analytics-catalog.mdx) - Learn how to create catalogs and add data sources - [Run Analytical Queries Through ScalarDB Analytics](run-analytical-queries.mdx) - Start running queries with your configuration - [Deploy ScalarDB Analytics in Public Cloud Environments](deployment.mdx) - Deploy ScalarDB Analytics in production ================================================ FILE: docs/scalardb-analytics/create-scalardb-analytics-catalog.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Create a ScalarDB Analytics Catalog import WarningLicenseKeyContact from "/src/components/en-us/_warning-license-key-contact.mdx"; This guide explains how to create a ScalarDB Analytics catalog. The ScalarDB Analytics catalog serves as the central hub that organizes information from various underlying data sources, including database schemas and contact points, enabling you to run analytical queries across these data sources through a unified interface. This information is referred to as catalog information. ## Set up ScalarDB Analytics server Catalog information is managed by a component called a ScalarDB Analytics server. So, you first need to set up a ScalarDB Analytics server. The ScalarDB Analytics server also performs several other tasks, such as collecting usage metering information and storing it in a file system or cloud blob storage. ### Prerequisites The ScalarDB Analytics server requires a database to store catalog information. This database is referred to as the **metadata database** throughout this documentation. ScalarDB Analytics supports the following databases for the metadata database: - PostgreSQL - MySQL - SQL Server - Oracle Create a database and user with appropriate privileges before starting the ScalarDB Analytics server. The specific commands vary by database type. ### Configure the ScalarDB Analytics server Create a ScalarDB Analytics server configuration file (for example, `scalardb-analytics-server.properties`). The following example uses PostgreSQL as the metadata database: ```properties # Metadata database configuration (required) scalar.db.analytics.server.db.contact_points=jdbc:postgresql://localhost:5432/scalardb_analytics scalar.db.analytics.server.db.username=analytics_user scalar.db.analytics.server.db.password=your_secure_password # gRPC server configuration (optional) scalar.db.analytics.server.catalog.port=11051 # default scalar.db.analytics.server.metering.port=11052 # default # TLS configuration (optional but recommended for production) scalar.db.analytics.server.tls.enabled=true scalar.db.analytics.server.tls.cert_chain_path=/path/to/server.crt scalar.db.analytics.server.tls.private_key_path=/path/to/server.key # License configuration (required) scalar.db.analytics.server.licensing.license_key= scalar.db.analytics.server.licensing.license_check_cert_pem= # Metering storage configuration (required) scalar.db.analytics.server.metering.storage.provider=filesystem scalar.db.analytics.server.metering.storage.path=/var/scalardb-analytics/metering ``` :::note For production deployments, configure metering storage to use object storage (for example, Amazon S3, Google Cloud Storage, or Azure Blob Storage) instead of the local filesystem. For detailed configuration options, see [ScalarDB Analytics Configurations](./configurations.mdx). ::: ### Start the ScalarDB Analytics server Start the ScalarDB Analytics server with your configuration: ```console docker run -d \ --name scalardb-analytics-server \ -p 11051:11051 \ -p 11052:11052 \ -v /path/to/scalardb-analytics-server.properties:/scalardb-analytics-server/server.properties \ ghcr.io/scalar-labs/scalardb-analytics-server: ``` Replace `` with the ScalarDB Analytics version you want to use. You can find available versions at the [container registry page](https://github.com/scalar-labs/scalardb-analytics/pkgs/container/scalardb-analytics-server-byol). The container uses the configuration file at `/scalardb-analytics-server/server.properties` by default. The ScalarDB Analytics server will perform the following during startup: 1. Validate the license 2. Connect to the metadata database 3. Start gRPC services on the configured ports 4. Begin accepting client connections :::tip Keep note of your server configuration (hostname and ports) as you will need this information later when configuring Spark applications to connect to your catalog. ::: ### Check server health (optional) If you want to verify the server is running properly, you can use grpc-health-probe (included in the container image): ```console # Check catalog service health docker exec scalardb-analytics-server grpc-health-probe -addr=localhost:11051 # Check metering service health docker exec scalardb-analytics-server grpc-health-probe -addr=localhost:11052 # For TLS-enabled servers docker exec scalardb-analytics-server grpc-health-probe -addr=localhost:11051 -tls -tls-ca-cert=/path/to/ca.crt ``` ## Set up ScalarDB Analytics CLI ScalarDB Analytics CLI is a command-line tool that communicates with the ScalarDB Analytics server to manage catalogs, register data sources, and perform administrative tasks. For details, see the [ScalarDB Analytics CLI Command Reference](./reference-cli-command.mdx) ### Install the CLI The `scalardb-analytics-cli` tool is available as a container image: ```console # Pull the CLI image docker pull ghcr.io/scalar-labs/scalardb-analytics-cli: ``` Replace `` with the ScalarDB Analytics version you want to use. Available versions can be found at the [container registry page](https://github.com/scalar-labs/scalardb-analytics/pkgs/container/scalardb-analytics-cli). To run CLI commands, you'll need to mount your configuration file into the container: ```console # Example: List catalogs docker run --rm \ -v $(pwd)/client.properties:/config/client.properties:ro \ ghcr.io/scalar-labs/scalardb-analytics-cli: \ -c /config/client.properties \ catalog list ``` ### Configure the client Create a configuration file named `client.properties` in your current directory: ```properties # Server connection scalar.db.analytics.client.server.host=localhost scalar.db.analytics.client.server.catalog.port=11051 scalar.db.analytics.client.server.metering.port=11052 # TLS/SSL configuration (if enabled on server) scalar.db.analytics.client.server.tls.enabled=true scalar.db.analytics.client.server.tls.ca_root_cert_path=/path/to/ca.crt scalar.db.analytics.client.server.tls.override_authority=analytics.example.com ``` For detailed configuration options, see [ScalarDB Analytics Configurations](./configurations.mdx). ### Set up an alias (optional) For convenience, you can create an alias to avoid typing the long Docker command each time: ```console alias scalardb-analytics-cli='docker run --rm -v $(pwd)/client.properties:/config/client.properties:ro ghcr.io/scalar-labs/scalardb-analytics-cli: -c /config/client.properties' ``` With this alias, you can run commands more simply: ```console scalardb-analytics-cli catalog list ``` ## Create your catalog This section describes how to create a catalog container, add data sources to your catalog, and verify your catalog. ### Create a catalog container A catalog serves as a logical container for organizing data sources. Create your first catalog: ```console scalardb-analytics-cli catalog create --catalog production ``` :::important Remember the catalog name you choose here (for example, `production`). You will need to use this exact same name when configuring your Spark applications to connect to this catalog. ::: Verify the catalog was created: ```console scalardb-analytics-cli catalog list ``` ### Add data sources to your catalog Create a data source registration file for your database. Here's an example for PostgreSQL: Create `postgres-datasource.json`: ```json { "type": "postgresql", "host": "postgres.example.com", "port": 5432, "username": "analytics_user", "password": "secure_password", "database": "customers" } ``` For detailed configuration options and examples for other database types (MySQL, ScalarDB, Oracle, SQL Server, DynamoDB), see the [Data Source Reference](./reference-data-source.mdx). Register the data source: ```console scalardb-analytics-cli data-source register --catalog production --data-source postgres_customers --provider-file postgres-datasource.json ``` ### Verify your catalog List all data sources in your catalog: ```console scalardb-analytics-cli data-source list --catalog production ``` List namespaces in your catalog: ```console scalardb-analytics-cli namespace list --catalog production ``` List tables in your catalog: ```console scalardb-analytics-cli table list --catalog production ``` ## Next steps You now have a fully functional ScalarDB Analytics catalog with registered data sources. To develop analytical applications using this catalog: 1. **Run analytical queries:** See [Run Analytical Queries Through ScalarDB Analytics](./run-analytical-queries.mdx) 2. **Add more data sources:** See [Data Source Reference](./reference-data-source.mdx) 3. **Deploy in public clouds:** See [Deploy ScalarDB Analytics in Public Cloud Environments](./deployment.mdx) 4. **Explore configuration details:** See [ScalarDB Analytics Configurations](./configurations.mdx) ================================================ FILE: docs/scalardb-analytics/deploy-scalardb-analytics-server.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; import HelmCommandUsage from '/src/components/en-us/_helm-command-usage.mdx'; # Deploy a ScalarDB Analytics server This document explains how to deploy a ScalarDB Analytics server in your local or production environment. ## Step 1. Select a billing plan for ScalarDB Analytics You can get the ScalarDB Analytics server in several ways: You can use ScalarDB Analytics in a pay-as-you-go (PAYG) plan. In this case, you will pay the license fee based on your query usage. You can use ScalarDB Analytics in a PAYG plan in AWS Marketplace. To deploy the ScalarDB Analytics server from AWS Marketplace with a PAYG plan: 1. Go to the AWS Marketplace page [ScalarDB Analytics server](https://aws.amazon.com/marketplace/pp/prodview-53ik57autkmci). 1. Subscribe to the ScalarDB Analytics server. 1. Select **View purchase options**. 1. Select **Subscribe**. :::tip After subscribing, you'll have permission to pull the container image of the ScalarDB Analytics server from the following container registry. You will specify this container registry and pull the container image in a later step, so keep note of it. ```console 709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-analytics-server-aws-payg ``` ::: You can use ScalarDB Analytics in a bring-your-own-license (BYOL) plan. In this case, you will pay the license fee based on your contract, with an upper limit on the queries you can run. You can use ScalarDB Analytics in a BYOL plan on supported Kubernetes platforms. You can see the supported Kubernetes platforms in [Requirements](../requirements.mdx#kubernetes). You can deploy the ScalarDB Analytics server by using a container image with a license key that is provided in a BYOL plan. You can pull the container image of the ScalarDB Analytics server from the following container registry. :::note You will specify this container registry in a later step, so keep note of it. ::: ```console ghcr.io/scalar-labs/scalardb-analytics-server-byol ``` If you want to use other platforms, please [contact us](https://www.scalar-labs.com/contact-us). ## Step 2. Deploy a Kubernetes cluster Deploy a cluster on your preferred Kubernetes platform based on the following requirements and checkpoints: 1. Decide which Kubernetes platform to use based on the billing plan and purpose. - If you chose **PAYG (container offer - AWS Marketplace)** in [Step 1. Select a billing plan for ScalarDB Analytics](#step-1-select-a-billing-plan-for-scalardb-analytics), you need to deploy Amazon Elastic Kubernetes Service (EKS) in the supported regions. The supported regions will be referred to in a later step. - If you chose **BYOL (container offer - supported Kubernetes platform)** in [Step 1. Select a billing plan for ScalarDB Analytics](#step-1-select-a-billing-plan-for-scalardb-analytics), you can use the supported Kubernetes platforms. :::note You should use minikube for testing or development purposes only. minikube is not recommended for production use. ::: 1. Check the general recommendations and requirements of the Kubernetes cluster for the ScalarDB Analytics server. - Recommendations - You should use a worker node that has at least 2 CPUs and 4 GB of memory. - Currently, the ScalarDB Analytics server does not have a clustering feature. Therefore, only one worker node is enough. - If you want to make the Kubernetes cluster itself highly available, you can deploy it with multiple worker nodes. - Requirements - You must allow your Spark application to connect to the ScalarDB Analytics server deployed on the Kubernetes cluster from a network perspective. To see which port the ScalarDB Analytics server uses, see [Requirements](../requirements.mdx). - You must allow the ScalarDB Analytics server to read from and write to the backend database to store the catalog information. These procedures will be described in detail in [Step 3. Deploy a backend database](#step-3-deploy-a-backend-database). - You must allow the ScalarDB Analytics server to read from and write to the object storage to store metering information. These procedures will be described in detail in [Step 4. Deploy an object storage](#step-4-deploy-an-object-storage). 1. Deploy a Kubernetes cluster for the ScalarDB Analytics server. For testing or development purposes, you can use minikube as a local Kubernetes cluster. For details on how to install and start minikube, see the [official minikube documentation](https://minikube.sigs.k8s.io/docs/start/). For production environments, please deploy the Kubernetes cluster based on the above requirements of the ScalarDB Analytics server and your system's requirements, for example, security, availability, backup/restore, cost, and scalability amongst your other requirements. - If you chose **BYOL (container offer - supported Kubernetes platform)**, you can use Amazon Elastic Kubernetes Service (EKS). - If you chose **PAYG (container offer - AWS Marketplace)** in [Step 1. Select a billing plan for ScalarDB Analytics](#step-1-select-a-billing-plan-for-scalardb-analytics), you need to do the following: 1. Deploy EKS in supported regions that are described in the AWS documentation [MeterUsage Region support for Amazon ECS and Amazon EKS](https://docs.aws.amazon.com/marketplace/latest/APIReference/metering-regions.html#meterusage-region-support-ecs-eks). 1. Run the following two commands after you deploy EKS: - `eksctl utils associate-iam-oidc-provider` ```console eksctl utils associate-iam-oidc-provider --region --cluster --approve ``` - `eksctl create iamserviceaccount` ```console eksctl create iamserviceaccount \ --name \ --namespace \ --region \ --cluster \ --attach-policy-arn arn:aws:iam::aws:policy/AWSMarketplaceMeteringFullAccess \ --approve \ --override-existing-serviceaccounts ``` You can set an arbitrary name to `SERVICE_ACCOUNT_NAME` based on the [Kubernetes resource naming rule](https://kubernetes.io/docs/concepts/overview/working-with-objects/names/). :::note Keep note of the value that you set for `SERVICE_ACCOUNT_NAME` because you will specify this service account name in a later step. ::: :::important For production environments, you must use the supported Kubernetes platform. You can see the supported Kubernetes platforms in [Requirements](../requirements.mdx#kubernetes). ::: ## Step 3. Deploy a backend database Deploy your preferred backend database based on the following requirements and checkpoints: 1. Decide which backend database to use. - You can see the supported backend database for the ScalarDB Analytics server in [Requirements](../requirements.mdx). - Unless you have a special reason not to, you should use a database that you are familiar with. 1. Check the backend database requirements for the ScalarDB Analytics server. - You can see the requirements of each backend database in the [Requirements](../requirements.mdx) page. 1. Deploy the backend database in your environment. For testing or development purposes, you can deploy a backend database in the Kubernetes cluster as a Pod. For example, if you use PostgreSQL, you can deploy it as follows: 1. Add the Bitnami Helm repository by running the following command: ```console helm repo add bitnami https://charts.bitnami.com/bitnami ``` 1. Deploy PostgreSQL by running the following command: ```console helm install postgresql-scalardb-cluster bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false ``` 1. Check if the PostgreSQL container is running by running the following command: ```console kubectl get pod ``` You should see the following output: ```console NAME READY STATUS RESTARTS AGE postgresql-scalardb-cluster-0 1/1 Running 0 17s ``` For production environments, please deploy the backend database based on the above requirements of the ScalarDB Analytics server and your system's requirements, for example, security, availability, backup/restore, cost, and scalability amongst your other requirements. ## Step 4. Deploy an object storage Deploy an object storage based on the following requirements and checkpoints: 1. Decide which object storage to use. - You can use [Amazon S3](https://aws.amazon.com/s3/), [Azure Blob Storage](https://azure.microsoft.com/products/storage/blobs), or [Google's Cloud Storage](https://cloud.google.com/storage) as a data store for metering information for the ScalarDB Analytics server. - You should use the object storage that is provided by the same cloud service provider as the Kubernetes cluster that you chose in [Step 2. Deploy a Kubernetes cluster](#step-2-deploy-a-kubernetes-cluster). For example, if you chose EKS, you should use Amazon S3. 1. Check the object storage requirements for the ScalarDB Analytics server. - You must allow the ScalarDB Analytics server to read from and write to the object storage. 1. Deploy the object storage in your environment. For testing or development purposes, you can store metering information on the filesystem in the ScalarDB Analytics server container. In other words, you don't need to use the object storage. In this case, you need to set `scalar.db.analytics.server.metering.storage.provider=filesystem` in the properties file. For more details, see [Step 5. Create a custom values file](#step-5-create-a-custom-values-file). For production environments, please deploy the object storage based on the above requirements of the ScalarDB Analytics server and your system's requirements, for example, security, availability, backup/restore, cost, and scalability amongst your other requirements. ## Step 5. Create a custom values file Create your custom values file `scalardb-analytics-server.yaml` based on your environment and your decisions in the previous steps. ### Set the required configurations 1. Set the container image and the license configurations Based on the billing plan you chose in [Step 1. Select a billing plan for ScalarDB Analytics](#step-1-select-a-billing-plan-for-scalardb-analytics), set the container image configuration to `scalarDbAnalyticsServer.image.repository`. Select one of the following billing plans to see an example of this configuration. ```yaml scalarDbAnalyticsServer: image: repository: 709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-analytics-server-aws-payg ``` ```yaml scalarDbAnalyticsServer: image: repository: ghcr.io/scalar-labs/scalardb-analytics-server-byol properties: | scalar.db.analytics.server.licensing.license_key= scalar.db.analytics.server.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIID...certificate content...\n-----END CERTIFICATE----- ``` 1. Set the service account configurations Based on the billing plan you chose in [Step 1. Select a billing plan for ScalarDB Analytics](#step-1-select-a-billing-plan-for-scalardb-analytics), set the service account configurations to `scalarDbAnalyticsServer.serviceAccount`. Select one of the following billing plans to see an example of this configuration. ```yaml scalarDbAnalyticsServer: serviceAccount: serviceAccountName: automountServiceAccountToken: true ``` :::note Change `` to the name of the service account that you created by using the `eksctl create iamserviceaccount` command in [Step 2. Deploy a Kubernetes cluster](#step-2-deploy-a-kubernetes-cluster). ::: You don't need to set a service account configuration. 1. Set the database configurations Based on the backend database you chose in [Step 3. Deploy a backend database](#step-3-deploy-a-backend-database), set the database configurations in `scalarDbAnalyticsServer.properties`. Select one of the following databases to see an example of these configurations. ```yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.db.contact_points=jdbc:postgresql://:/ scalar.db.analytics.server.db.username= scalar.db.analytics.server.db.password= ``` ```yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.db.contact_points=jdbc:mysql://:/ scalar.db.analytics.server.db.username= scalar.db.analytics.server.db.password= ``` ```yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.db.contact_points=jdbc:sqlserver://:;databaseName=;encrypt=true;trustServerCertificate=true scalar.db.analytics.server.db.username= scalar.db.analytics.server.db.password= ``` ```yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.db.contact_points=jdbc:oracle:thin:@//:/ scalar.db.analytics.server.db.username= scalar.db.analytics.server.db.password= ``` 1. Set the object storage configurations Based on the object storage you chose in [Step 4. Deploy an object storage](#step-4-deploy-an-object-storage), please set object storage configurations in `scalarDbAnalyticsServer.properties`. Select one of the following object storages to see an example of these configurations. ```yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.metering.storage.provider=aws-s3 scalar.db.analytics.server.metering.storage.accessKeyId= scalar.db.analytics.server.metering.storage.secretAccessKey= ``` ```yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.metering.storage.provider=azureblob scalar.db.analytics.server.metering.storage.accessKeyId= scalar.db.analytics.server.metering.storage.secretAccessKey= ``` ```yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.metering.storage.provider=google-cloud-storage scalar.db.analytics.server.metering.storage.accessKeyId= scalar.db.analytics.server.metering.storage.secretAccessKey= ``` :::note You can use `filesystem` for testing or development purposes only. Filesystem is not recommended for production use. ::: ```yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.metering.storage.provider=filesystem scalar.db.analytics.server.metering.storage.path=/tmp/scalardb-analytics-metering ``` 1. Set the service configurations Based on the connectivity of the ScalarDB Analytics server, you need to set `scalarDbAnalyticsServer.service.type`. Select one of the following types of connections to see an example of this configuration. If your Spark application accesses the ScalarDB Analytics server from outside of the Kubernetes cluster, set `scalarDbAnalyticsServer.service.type` to `LoadBalancer`. ```yaml scalarDbAnalyticsServer: service: type: "LoadBalancer" ``` If your Spark application accesses the ScalarDB Analytics server from inside of the Kubernetes cluster, set `scalarDbAnalyticsServer.service.type` to `ClusterIP`. ```yaml scalarDbAnalyticsServer: service: type: "ClusterIP" ``` 1. Check the required configurations After completing the above steps, you should have the following configurations, depending on your environment, for example: :::note These configurations are just examples. The actual configurations may be different from these examples. Please make sure to set configurations based on your environment. ::: ```yaml scalarDbAnalyticsServer: image: repository: ghcr.io/scalar-labs/scalardb-analytics-server-byol properties: | # License configurations scalar.db.analytics.server.licensing.license_key= scalar.db.analytics.server.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIID...certificate content...\n-----END CERTIFICATE----- # Database configurations scalar.db.analytics.server.db.contact_points=jdbc:postgresql://:/ scalar.db.analytics.server.db.username= scalar.db.analytics.server.db.password= # Object storage configurations scalar.db.analytics.server.metering.storage.provider=azureblob scalar.db.analytics.server.metering.storage.accessKeyId= scalar.db.analytics.server.metering.storage.secretAccessKey= service: type: "LoadBalancer" ``` ```yaml scalarDbAnalyticsServer: image: repository: 709825985650.dkr.ecr.us-east-1.amazonaws.com/scalar/scalardb-analytics-server-aws-payg properties: | # Database configurations scalar.db.analytics.server.db.contact_points=jdbc:mysql://:/ scalar.db.analytics.server.db.username= scalar.db.analytics.server.db.password= # Object storage configurations scalar.db.analytics.server.metering.storage.provider=aws-s3 scalar.db.analytics.server.metering.storage.accessKeyId= scalar.db.analytics.server.metering.storage.secretAccessKey= service: type: "ClusterIP" serviceAccount: serviceAccountName: "scalardb-analytics-payg-sa" automountServiceAccountToken: true ``` :::note You can use `filesystem` for testing or development purposes only. Filesystem is not recommended for production use. ::: ```yaml scalarDbAnalyticsServer: image: repository: ghcr.io/scalar-labs/scalardb-analytics-server-byol properties: | # License configurations scalar.db.analytics.server.licensing.license_key= scalar.db.analytics.server.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIID...certificate content...\n-----END CERTIFICATE----- # Database configurations scalar.db.analytics.server.db.contact_points=jdbc:sqlserver://:;databaseName=;encrypt=true;trustServerCertificate=true scalar.db.analytics.server.db.username= scalar.db.analytics.server.db.password= # Filesystem configurations scalar.db.analytics.server.metering.storage.provider=filesystem scalar.db.analytics.server.metering.storage.path=/tmp/scalardb-analytics-metering service: type: "ClusterIP" ``` ### Set the optional configurations You can see the optional configurations in [Optional configurations](../helm-charts/configure-custom-values-scalardb-analytics-server.mdx#optional-configurations). Set the optional configurations based on your environment if necessary. ## Step 6. Deploy a ScalarDB Analytics server by using Helm Chart Deploy, upgrade, or uninstall the ScalarDB Analytics server deployment by using the `helm` command with your custom values file `scalardb-analytics-server.yaml` that you created in [Step 5. Create a custom values file](#step-5-create-a-custom-values-file). ## Step 7. Check your deployment After deploying the ScalarDB Analytics server or upgrading it, you should check the following points: 1. Check if the pod status is `Running` by running the following command: ```console kubectl get pod --namespace ``` :::note For the `--namespace` option, change `` to the name of the Kubernetes namespace that you deployed the ScalarDB Analytics server to. ::: For example, you can see `Running` in the `STATUS` column and `1/1` in the `READY` column as follows: ```console $ kubectl get pod NAME READY STATUS RESTARTS AGE scalardb-analytics-server-86767fff4c-p6nkq 1/1 Running 0 22m ``` 1. Check if the service is exported. ```console kubectl get svc --namespace ``` :::note For the `--namespace` option, change `` to the name of the Kubernetes namespace that you deployed the ScalarDB Analytics server to. ::: If you set `scalarDbAnalyticsServer.service.type` to `LoadBalancer` in [Step 5. Create a custom values file](#step-5-create-a-custom-values-file), you'll see the IP address or FQDN (depending on Kubernetes cluster) in the `EXTERNAL-IP` column as follows: ```console $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 4h54m scalardb-analytics-server LoadBalancer 10.98.116.121 127.0.0.1 11051:32619/TCP,11052:32598/TCP 2m43s ``` :::note If you're using minikube for testing or development purposes, you'll need to run the [minikube tunnel](https://minikube.sigs.k8s.io/docs/commands/tunnel/) command to expose the `LoadBalancer` service. ::: If you set `scalarDbAnalyticsServer.service.type` to `ClusterIP` in [Step 5. Create a custom values file](#step-5-create-a-custom-values-file), you'll see the IP address in the `CLUSTER-IP` column as follows: ```console $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 4h56m scalardb-analytics-server ClusterIP 10.102.141.240 11051/TCP,11052/TCP 3s ``` ================================================ FILE: docs/scalardb-analytics/deployment-local.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Deploy ScalarDB Analytics Locally import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; This guide explains how to deploy ScalarDB Analytics to a local Kubernetes cluster, specifically designed for testing purposes, by using a Helm Chart. ## Prerequisites Before deploying ScalarDB Analytics to a local environment, ensure that you have the following tools installed: - Kubernetes cluster (this guide assumes you're using [minikube](https://minikube.sigs.k8s.io/docs/start/)) - [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) - [Helm](https://helm.sh/docs/intro/install/) ## Example architecture The following is an example architecture described in this guide. ```mermaid flowchart TB User(("User")) subgraph K8s["Kubernetes cluster"] subgraph ControlPlane["Kubernetes control plane"] APIServer("Kubernetes API server") end subgraph PVC["PVC"] JAR[📄 Application's JAR] end Client["Client"] AnalyticsServer["Analytics server"] CLITool["CLI tool"] subgraph Spark["Spark"] Driver["Driver"] Executor1["Executor"] end PostgreSQLBackend[("PostgreSQL
(Analytics server backend)")] PostgreSQLNonManaged[("PostgreSQL
(Non-ScalarDB managed)")] MySQLManaged[("MySQL
(ScalarDB managed)")] end User --> Client User -->|"Deploy via Helm Chart"| AnalyticsServer User --> CLITool Client -->|"spark-submit, spark-sql"| APIServer APIServer -->|"Create"| Driver APIServer -->|"Create"| Executor1 Driver -->|"gRPC"| AnalyticsServer CLITool -->|"Create catalogs,
Register data sources,
etc."| AnalyticsServer JAR --> Driver Driver <--> Executor1 Executor1 --> PostgreSQLNonManaged Executor1 --> MySQLManaged AnalyticsServer --> PostgreSQLBackend style K8s fill:#F5F5F5 style User fill:#FFFFFF style ControlPlane fill:#FFFFFF style PVC fill:#FFFFFF style APIServer fill:#FFFFFF style JAR fill:#FFFFFF style Spark fill:#F5F5FF ``` This guide assumes a Kubernetes cluster running on minikube. In this setup, PostgreSQL is treated as an external data source not managed by ScalarDB transactions, while MySQL is treated as a data source managed by ScalarDB transactions (ScalarDB-managed data source). Both of these data sources store user data. The ScalarDB Analytics server is deployed as a Pod by using a Helm Chart. The ScalarDB Analytics server uses a dedicated backend database to store catalog data. A separate Pod is also created to serve as the client for running Spark commands. Additionally, the CLI tool used to operate the ScalarDB Analytics server is provided as a container image and runs on a separate Pod. :::note Please set up each data source yourself, referring to resources such as [How to Deploy ScalarDB Cluster Locally](../scalardb-cluster/setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx) for guidance. ::: ## Step 1: Set up the Kubernetes environment You first need to set up the Kubernetes environment where all components will be deployed. ### Create ServiceAccount and RoleBinding Create a service account (`ServiceAccount`) and a role binding (`RoleBinding`) to allow Spark jobs to manage resources within the Kubernetes cluster. ```shell NAMESPACE=default SERVICE_ACCOUNT_NAME=spark cat < analytics-server-custom-values.yaml scalarDbAnalyticsServer: properties: | scalar.db.analytics.server.catalog.port=11051 scalar.db.analytics.server.metering.port=11052 scalar.db.analytics.server.db.contact_points= scalar.db.analytics.server.db.username= scalar.db.analytics.server.db.password= scalar.db.analytics.server.metering.storage.provider=filesystem scalar.db.analytics.server.metering.storage.container_name=metering scalar.db.analytics.server.metering.storage.path=/tmp scalar.db.analytics.server.licensing.license_key= scalar.db.analytics.server.licensing.license_check_cert_pem= EOF ``` The following describes what you should change the content in the angle brackets to: - ``: JDBC connection string for the backend database of the ScalarDB Analytics server. - ``: The username of the backend database. - ``: The password of the backend database. - ``: The license key for the ScalarDB Analytics server. - ``: The PEM encoded license certificate for the ScalarDB Analytics server. For guidance on how to set up your license, including both commercial and trial licenses, see [How to Configure a License Key](../scalar-licensing/index.mdx). :::note The metering-related property values (`scalar.db.analytics.server.metering.storage.*`) can be used as shown in the example. For more details on metering configuration, see the [Configuration reference](./configurations.mdx). ::: ### Deploy the Analytics server Deploy the Analytics Server by running the following command. ```shell helm install scalardb-analytics-server scalar-labs/scalardb-analytics-server -f analytics-server-custom-values.yaml ``` For more details on deploying the ScalarDB Analytics server, see [Deploy a ScalarDB Analytics server](./deploy-scalardb-analytics-server.mdx). ## Step 3: Configure the catalog and data sources by using the CLI tool To create catalogs and register data sources on the ScalarDB Analytics server, use the CLI tool, which is provided as a container image. As an example, this section shows how to set up a Pod for the CLI tool and run commands from it. ### Set up a Pod for the CLI tool Create a manifest file for the CLI tool Pod. ```shell cat < analytics-server-cli.yaml apiVersion: v1 kind: Pod metadata: name: analytics-server-cli spec: containers: - name: analytics-server-cli image: ghcr.io/scalar-labs/scalardb-analytics-cli:3.18.0 command: ['sleep'] args: ['inf'] restartPolicy: Never EOF ``` You can change `metadata.name` and `spec.containers[*].name` to any values you like. Then, create the Pod for the CLI tool by running the following command: ```shell kubectl apply -f analytics-server-cli.yaml ``` Once the Pod is deployed, access it through the shell by running the following command. All following steps in this section should be performed inside this Pod. ```shell kubectl exec -it analytics-server-cli -- bash ``` Set up an alias for the CLI tool to simplify command execution by running the following command: ```shell alias scalardb-analytics-cli="java -jar /scalardb-analytics-cli/scalardb-analytics-cli.jar" ``` ### Prepare a configuration file for the ScalarDB data source When you use a data source under ScalarDB transaction management, you need to provide properties for ScalarDB. This guide explains how to register the ScalarDB properties file to the ScalarDB Analytics server. This example uses only MySQL but is intentionally configured as a multi-storage setup to simplify adding other databases later. As configured, the storage name `mysql` is assigned to the MySQL data source, and the namespace `nsmy` is mapped to it. ```shell cat < scalardb.properties # Storage scalar.db.storage=multi-storage # Multi-storage settings scalar.db.multi_storage.storages=mysql # Namespace mapping scalar.db.multi_storage.namespace_mapping=nsmy:mysql # Default storage scalar.db.multi_storage.default_storage=mysql # Multi-storage: Define MySQL scalar.db.multi_storage.storages.mysql.storage=jdbc scalar.db.multi_storage.storages.mysql.contact_points= scalar.db.multi_storage.storages.mysql.username= scalar.db.multi_storage.storages.mysql.password= EOF ``` :::note For details about multi-storage configurations, see [Multi-Storage Transactions](../multi-storage-transactions.mdx#how-to-configure-scalardb-to-support-multi-storage-transactions). ::: ### Prepare data source definition files You must define the data sources that ScalarDB Analytics accesses in JSON format. The following is an example of defining a data source managed by ScalarDB. When using a ScalarDB-managed data source, you must set the `type` item to `scalardb` and specify the path to the properties file by using the `${file:}` syntax. Please Replace `` with the actual path to the ScalarDB properties file, as shown below: ```shell cat <<"EOF" > data_source_scalardb.json { "type": "scalardb", "configs": "${file:./scalardb.properties}" } EOF ``` :::note You can specify `` as either an absolute path or a relative path. ::: The following is an example of defining a PostgreSQL data source that is not managed by ScalarDB. You must specify `postgresql` as the value of `type` item when using PostgreSQL as the data source. Then, replace each placeholder value enclosed in angle brackets in the command below with the appropriate value for your PostgreSQL data source and run the command: ```shell cat < data_source_postgres.json { "type": "postgresql", "host": , "port": "5432", "username": , "password": , "database": } EOF ``` ### Create a configuration file for the CLI tool Create a configuration file (`client.properties`) for the ScalarDB Analytics CLI tool. To connect to the ScalarDB Analytics server from the CLI tool, you need the hostname or IP address of the server. You can get the IP address by checking the `CLUSTER-IP` value of the ScalarDB Analytics server service by running the following command: ```shell $ kubectl get svc scalardb-analytics-server NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-analytics-server ClusterIP 10.97.81.28 11051/TCP,11052/TCP 40s ``` Then, create the configuration file by running the following command, replacing `` with the `CLUSTER-IP` value you retrieved: ```shell cat < client.properties scalar.db.analytics.client.server.host= scalar.db.analytics.client.server.catalog.port=11051 EOF ``` ### Register the catalog and data sources This section describes how to register a catalog and data sources using the CLI tool. #### Create a catalog First, create a catalog by using the following command. Replace `` with your desired catalog name. ```shell scalardb-analytics-cli -c client.properties catalog create --catalog ``` #### Register data sources Next, register the data sources for both ScalarDB-managed and non–ScalarDB-managed. Register a ScalarDB-managed data source by using the following command. Replace `` and `` with your desired catalog and data source name. ```shell scalardb-analytics-cli -c client.properties data-source register \ --catalog= --data-source= --provider-file=./data_source_scalardb.json ``` Register a non-ScalarDB-managed data source by using the following command. Replace `` and `` with your desired catalog and data source name. ```shell scalardb-analytics-cli -c client.properties data-source register \ --catalog= --data-source= --provider-file=./data_source_postgres.json ``` #### Additional CLI Commands The CLI tool provides additional commands for managing catalogs and data sources. For detailed instructions, refer to the [ScalarDB Analytics CLI tool documentation](./reference-cli-command.mdx). ## Step 4: Deploy a Spark client Pod In this step, you will deploy a Spark client Pod and set it up to run Spark jobs. ### Create a Spark client Pod Create a manifest file for the Spark client Pod. In the following example, the service account name is set to `spark`. Configure the Spark client Pod by running the following command: ```shell cat <<'EOF' > spark-client.yaml apiVersion: v1 kind: Pod metadata: name: "spark-client" spec: serviceAccountName: spark containers: - name: spark-client image: eclipse-temurin:21 command: ['sleep'] args: ['inf'] restartPolicy: Never terminationGracePeriodSeconds: 0 EOF ``` Create the Spark client Pod by running the following command: ```shell kubectl apply -f spark-client.yaml ``` ### Set up the Spark client Pod Access the Spark client Pod via a shell session by running the following command: ```shell kubectl exec -it spark-client -- bash ``` Install the Spark binary files and navigate to their directory by running the following command: ```shell VERSION=3.5.7 curl -O https://dlcdn.apache.org/spark/spark-${VERSION}/spark-${VERSION}-bin-hadoop3.tgz tar xzf spark-${VERSION}-bin-hadoop3.tgz cd spark-${VERSION}-bin-hadoop3 ``` Create a `spark-defaults.conf` file by changing the content in the angle brackets and then running the following command: ```shell cat < ./conf/spark-defaults.conf spark.jars.packages com.scalar-labs:scalardb-analytics-spark-all-_: spark.sql.catalog. com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog spark.sql.catalog..server.host spark.sql.catalog..server.catalog.port 11051 spark.sql.catalog..server.metering.port 11052 spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener EOF ``` The following describes what you should change the content in the angle brackets to: - ``: The version of Spark. - ``: The version of Scala used to build Spark. - ``: The version of ScalarDB Analytics. - ``: The name of the catalog. - ``: The `CLUSTER-IP` value of the ScalarDB Analytics server service. For more details, refer to [Set up ScalarDB Analytics in the Spark configuration](./run-analytical-queries.mdx#set-up-scalardb-analytics-in-the-spark-configuration). ## Step 5: Run Spark jobs from the client Pod At this point, the Spark client Pod has been set up and is ready to run Spark jobs. This step shows examples of how to run analytical queries as Spark jobs using the following two methods. - Using Spark SQL - Submitting jobs by using the `spark-submit` command :::note ScalarDB Analytics currently uses Apache Spark as its query engine. It can leverage Spark's native Kubernetes deployment mode, which enables dynamic provisioning of Spark driver and executor Pods at runtime. To use the Kubernetes deployment mode, you need to specify the Kubernetes API server (`k8s://...`) in the `--master` option of the spark commands. :::

Use the `spark-sql` command to run Spark SQL

You can run Spark SQL by running a command like the following: ```shell ./bin/spark-sql \ --master k8s://https://kubernetes.default.svc \ --conf spark.kubernetes.container.image=apache/spark:3.5.7-scala2.12-java11-python3-r-ubuntu \ --conf spark.driver.host=$(hostname -i) ```

Use the `spark-submit` command to run a Spark job

This section describes registering an application JAR, creating a temporary Pod, creating a Pod template, and executing `spark-submit`.

Register the application JAR to PVC

To run an application as a Spark job, you need to prepare the application's JAR file and execute the `spark-submit` command by running the following command. The JAR file must be located at a path accessible from the Spark driver. There are several ways to achieve this, and this guide demonstrates how to use a persistent volume claim (PVC). ```shell PVC_NAME=spark-app-pvc cat <Create a temporary Pod and copy the file Create a temporary Pod to store the application JAR in the PVC by running the following command: ```shell cat <Create a Pod template To create a Pod template for the dynamically generated Spark driver and executor Pods, log in to the Spark client Pod and run the following command: ```shell PVC_NAME=spark-app-pvc cat < spark-pod-template.yaml apiVersion: v1 kind: Pod metadata: name: spark-pod-template spec: volumes: - name: spark-jar-volume persistentVolumeClaim: claimName: ${PVC_NAME} containers: - name: spark-kubernetes-container volumeMounts: - mountPath: /opt/spark-jars name: spark-jar-volume EOF ```

Execute `spark-submit`

Run the application as a Spark job by using a command like the following: ```shell ./bin/spark-submit \ --master k8s://https://kubernetes.default.svc \ --deploy-mode cluster \ --name analytics-sample-job \ --class com.example.TestApp \ --conf spark.kubernetes.container.image=apache/spark:3.5.7-scala2.12-java11-python3-r-ubuntu \ --conf spark.kubernetes.namespace=default \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.driver.podTemplateFile=./spark-pod-template.yaml \ --conf spark.kubernetes.executor.podTemplateFile=./spark-pod-template.yaml \ --conf spark.jars.ivy=/tmp/.ivy2 \ --conf spark.jars.repositories=https://repo1.maven.org/maven2,https://packages.confluent.io/maven/ \ --properties-file ./conf/spark-defaults.conf \ local:///opt/spark-jars/app.jar ``` ## Clean up deployed resources This section shows how to clean up the resources you deployed in the Kubernetes environment. Remove the ScalarDB Analytics server by running the following command: ```shell helm uninstall scalardb-analytics-server postgresql-scalardb-analytics ``` Additionally, you can remove the Pods you deployed by running the following command: ```shell kubectl delete pod spark-client analytics-server-cli ``` Also, you can remove the other Kubernetes resources you created by running the following commands: ```shell # Delete the `spark` service account kubectl delete serviceaccount spark # Delete the `spark-app-pvc` PVC kubectl delete pvc spark-app-pvc ``` ================================================ FILE: docs/scalardb-analytics/deployment.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; # Deploy ScalarDB Analytics in Public Cloud Environments This guide explains how to deploy ScalarDB Analytics in a public cloud environment. ScalarDB Analytics consists of two main components: a ScalarDB Analytics server and Apache Spark. In this guide, you can choose either Amazon EMR, Databricks, Azure Synapse Analytics, or Google Cloud Dataproc for the Spark environment. For details about ScalarDB Analytics, refer to [ScalarDB Analytics Design](./design.mdx). ## Deploy ScalarDB Analytics server ScalarDB Analytics requires a catalog server to manage metadata and data source connections. The catalog server should be deployed by using Helm Charts on a Kubernetes cluster. For detailed deployment instructions, see [Deploy ScalarDB Analytics Server](./deploy-scalardb-analytics-server.mdx). After deploying the catalog server, note the following information for Spark configuration: - Catalog server host address - Catalog port (default: 11051) - Metering port (default: 11052) ## Deploy Spark with ScalarDB Analytics After deploying the catalog server, you can configure and deploy Spark with ScalarDB Analytics by using managed Spark services. ### Supported managed Spark services and their application types ScalarDB Analytics supports the following managed Spark services and application types. | Public Cloud Service | Spark Driver | Spark Connect | Notebook | JDBC | | ----------------------- | ------------ | ------------- | -------- | ---- | | Amazon EMR (EMR on EC2) | ✅ | ✅ | ✅ | ❌ | | Databricks | ✅ | ❌ | ✅ | ✅ | | Azure Synapse Analytics | ✅ | ❌ | ✅ | ❌ | | Google Cloud Dataproc | ✅ | ✅ | ✅ | ❌ | ### Configure and deploy Select your public cloud environment, and follow the instructions to set up and deploy Spark with ScalarDB Analytics. You can use Amazon EMR (EMR on EC2) to run analytical queries through ScalarDB Analytics. For the basics to launch an EMR cluster, please refer to the [AWS EMR on EC2 documentation](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan.html).

ScalarDB Analytics configuration

To enable ScalarDB Analytics, you need to add the following configuration to the Software setting when you launch an EMR cluster. Be sure to replace the content in the angle brackets: ```json [ { "Classification": "spark-defaults", "Properties": { "spark.jars.packages": "com.scalar-labs:scalardb-analytics-spark-all-_:", "spark.extraListeners": "com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener", "spark.sql.catalog.": "com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog", "spark.sql.catalog..server.host": "", "spark.sql.catalog..server.catalog.port": "11051", "spark.sql.catalog..server.metering.port": "11052" } } ] ``` The following describes what you should change the content in the angle brackets to: - ``: The version of Spark (`3.5` or `3.4`). - ``: The version of Scala used to build Spark (`2.13` or `2.12`). - ``: The version of ScalarDB Analytics. - ``: The name of the catalog. This must match a catalog created on the ScalarDB Analytics server. - ``: The host address of your ScalarDB Analytics server. For more details, refer to [Set up ScalarDB Analytics in the Spark configuration](./run-analytical-queries.mdx#set-up-scalardb-analytics-in-the-spark-configuration).

Run analytical queries via the Spark driver

After the EMR Spark cluster has launched, you can use ssh to connect to the primary node of the EMR cluster and run your Spark application. For details on how to create a Spark driver application, refer to [Spark driver application](./run-analytical-queries.mdx?spark-application-type=spark-driver#develop-a-spark-application).

Run analytical queries via Spark Connect

You can use Spark Connect to run your Spark application remotely by using the EMR cluster that you launched. You first need to configure the Software setting in the same way as the [Spark driver application](./run-analytical-queries.mdx?spark-application-type=spark-driver#develop-a-spark-application). You also need to set the following configuration to enable Spark Connect.

Allow inbound traffic for a Spark Connect server

1. Create a security group to allow inbound traffic for a Spark Connect server. (Port 15001 is the default). 2. Allow the role of "Amazon EMR service role" to attach the security group to the primary node of the EMR cluster. 3. Add the security group to the primary node of the EMR cluster as "Additional security groups" when you launch the EMR cluster.

Launch the Spark Connect server via a bootstrap action

1. Create a script file to launch the Spark Connect server as follows: ```bash #!/usr/bin/env bash set -eu -o pipefail cd /var/lib/spark sudo -u spark /usr/lib/spark/sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_:,com.scalar-labs:scalardb-analytics-spark-all-_: ``` The following describes what you should change the content in the angle brackets to: - ``: The major and minor version of Scala that matches your Spark installation (2.12 or 2.13). - ``: The full version of Spark you are using (such as 3.5.3). - ``: The major and minor version of Spark you are using (3.4 or 3.5). - ``: The version of ScalarDB Analytics. 2. Upload the script file to S3. 3. Allow the role of "EC2 instance profile for Amazon EMR" to access the uploaded script file in S3. 4. Add the uploaded script file to "Bootstrap actions" when you launch the EMR cluster.

Run analytical queries

You can run your Spark application via Spark Connect from anywhere by using the remote URL of the Spark Connect server, which is `sc://:15001`. For details on how to create a Spark application by using Spark Connect, refer to [Spark Connect application](./run-analytical-queries.mdx?spark-application-type=spark-connect#develop-a-spark-application).

Run interactive queries by using notebooks

You can also use EMR Studio notebooks to run interactive queries through ScalarDB Analytics. EMR Studio provides a managed Jupyter notebook environment that can be attached to your EMR cluster. For details on how to set up and use EMR Studio notebooks, refer to the [Amazon EMR Studio documentation](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio.html). After attaching a notebook to the EMR cluster that is configured with ScalarDB Analytics, you can run SQL queries by using the `%%sql` magic command. To list the available catalogs, run the following command: ```sql %%sql -- List available catalogs SHOW CATALOGS ``` To list the databases in the ScalarDB catalog, run the following command, replacing `` with your actual catalog name configured in the ScalarDB Analytics server. ```sql %%sql -- List databases in ScalarDB catalog SHOW DATABASES IN ``` To query the data, run the following command, replacing the placeholders in angle brackets with your actual catalog, data source, namespace, and table names configured in the ScalarDB Analytics server. ```sql %%sql -- Query data SELECT * FROM ... LIMIT 10 ``` You can use Databricks to run analytical queries through ScalarDB Analytics. :::note Note that Databricks provides a modified version of Apache Spark, which works differently from the original Apache Spark. :::

Prepare an init script for loading the ScalarDB Analytics library JAR

1. Download the ScalarDB Analytics library JAR file from the Maven repository. Choose the appropriate JAR file based on your Spark, Scala, and ScalarDB versions: - [scalardb-analytics-spark-all-3.4_2.13 (Spark v3.4, Scala v2.13)](https://repo1.maven.org/maven2/com/scalar-labs/scalardb-analytics-spark-all-3.4_2.13/) - [scalardb-analytics-spark-all-3.5_2.13 (Spark v3.5, Scala v2.13)](https://repo1.maven.org/maven2/com/scalar-labs/scalardb-analytics-spark-all-3.5_2.13/) - [scalardb-analytics-spark-all-3.4_2.12 (Spark v3.4, Scala v2.12)](https://repo1.maven.org/maven2/com/scalar-labs/scalardb-analytics-spark-all-3.4_2.12/) - [scalardb-analytics-spark-all-3.5_2.12 (Spark v3.5, Scala v2.12)](https://repo1.maven.org/maven2/com/scalar-labs/scalardb-analytics-spark-all-3.5_2.12/) 2. Upload the JAR file to the Databricks workspace. 3. Create an init script as follows, replacing `` with the path to your JAR file in the Databricks workspace: ```bash #!/bin/bash # Target directories TARGET_DIRECTORIES=("/databricks/jars" "/databricks/hive_metastore_jars") JAR_PATH="" # Copy the JAR file to the target directories for TARGET_DIR in "${TARGET_DIRECTORIES[@]}"; do mkdir -p "$TARGET_DIR" cp "$JAR_PATH" "$TARGET_DIR/" done ``` 4. Upload the init script to the Databricks workspace.

Launch Databricks compute

ScalarDB Analytics works with all-purpose compute on Databricks. When you launch compute, you need to configure the compute to enable ScalarDB Analytics as follows: 1. Select `Create compute` in the `Compute` menu. 2. Select `Unrestricted` from the `Policy` dropdown menu. 3. Select an appropriate Databricks runtime version that supports Spark 3.4 or 3.5. 4. Go to the `Advanced` section. In the `Access mode` tab, select `Manual` as the access mode, and choose `No isolation shared`. 5. In the `Advanced` section, select the `Spark` tab, and enter the following configurations in `Spark config`: ``` spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener spark.sql.catalog. com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog spark.sql.catalog..server.host spark.sql.catalog..server.catalog.port 11051 spark.sql.catalog..server.metering.port 11052 ``` Replace the placeholders: - ``: The name of the catalog. This must match a catalog created on the ScalarDB Analytics server. - ``: The host address of your ScalarDB Analytics server. 6. In the `Advanced` section, select the `init scripts` tab, and specify the path to the init script in the workspace you uploaded. 7. Select `Create`.

Run analytical queries via the Spark driver

You can run your Spark application on the properly configured Databricks compute with Databricks Jobs to access the tables in ScalarDB Analytics. ScalarDB Analytics works with Databricks Jobs task types for Python, JAR, SQL, and Notebook. See the following section for details on using notebooks. For more details on how to use Databricks Jobs, refer to the [Databricks Jobs documentation](https://docs.databricks.com/en/jobs/index.html).

Run interactive queries by using notebooks

You can also use Databricks notebooks to run interactive queries through ScalarDB Analytics. After launching the compute that is configured with ScalarDB Analytics, you can create a notebook and attach it to the compute. For details on how to use Databricks notebooks, refer to the [Databricks notebooks documentation](https://docs.databricks.com/en/notebooks/index.html). After attaching a notebook to the compute, you can run SQL queries as follows. To list the available catalogs, run the following command: ```sql -- List available catalogs SHOW CATALOGS ``` To list the databases in the ScalarDB catalog, run the following command, replacing `` with your actual catalog name configured in the ScalarDB Analytics server. ```sql -- List databases in ScalarDB catalog SHOW DATABASES IN ``` To query the data, run the following command, replacing the placeholders in angle brackets with your actual catalog, data source, namespace, and table names configured in the ScalarDB Analytics server. ```sql -- Query data SELECT * FROM ... LIMIT 10 ```

Run analytical queries via the JDBC driver

Databricks supports JDBC to run SQL jobs on compute. After compute is launched, you can get the JDBC URL of the compute in the `Advanced` > `JDBC/ODBC` tab. To connect to the compute by using JDBC, you need to add the Databricks JDBC driver to your application dependencies. For example, if you are using Gradle, you can add the following dependency to your `build.gradle` file after replacing `` with the version of the Databricks JDBC driver you want to use: ```groovy implementation("com.databricks:databricks-jdbc:") ``` Then, you can connect to the compute by using JDBC with the JDBC URL (``), as is common with JDBC applications. ```java Class.forName("com.databricks.client.jdbc.Driver"); String url = ""; Connection conn = DriverManager.getConnection(url); ``` For more details on how to use JDBC with Databricks, refer to the [Databricks JDBC Driver documentation](https://docs.databricks.com/en/integrations/jdbc/index.html).

Use Azure Synapse Analytics

You can use Azure Synapse Analytics to run analytical queries through ScalarDB Analytics. For the basics of Azure Synapse Analytics, please refer to the [Azure Synapse Analytics documentation](https://learn.microsoft.com/en-us/azure/synapse-analytics/). :::note Azure Synapse Analytics uses serverless Apache Spark pools to run Spark workloads. For more details about Spark pools, see [Apache Spark in Azure Synapse Analytics](https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-overview). :::

Prerequisites

Before configuring ScalarDB Analytics on Azure Synapse, ensure the following requirements are met.
Synapse workspace
A Synapse workspace is a collaboration environment for analytics in Azure. You must create a Synapse workspace before you run Spark workloads. For more details about creating a Synapse workspace, see [Quickstart: Create a Synapse workspace](https://learn.microsoft.com/en-us/azure/synapse-analytics/quickstart-create-workspace). :::note This guide assumes that you have created a Synapse workspace with a Managed Virtual Network enabled. A Managed Virtual Network is recommended for network isolation and security. For more details, see [Azure Synapse Analytics Managed Virtual Network](https://learn.microsoft.com/en-us/azure/synapse-analytics/security/synapse-workspace-managed-vnet). :::
Storage permissions
If you plan to run a Spark driver application, both the **Synapse workspace managed identity** and the **user account** that submits the job require the `Storage Blob Data Contributor` role on the storage account. This role must be assigned by a subscription owner or user access administrator.

ScalarDB Analytics configuration

To enable ScalarDB Analytics, add the following configuration to the Spark pool configuration in Synapse Studio. 1. Go to **Synapse Studio**, select manage **Manage**, and choose **Apache Spark pools**. 2. Select your Spark pool, and then select **Apache Spark configuration**. 3. Create a new configuration or edit an existing one with the following properties: ```text spark.jars.packages com.scalar-labs:scalardb-analytics-spark-all-_: spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener spark.sql.catalog. com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog spark.sql.catalog..server.host spark.sql.catalog..server.catalog.port 11051 spark.sql.catalog..server.metering.port 11052 ``` The following describes what you should change the content in the angle brackets to: - ``: The version of Spark (`3.4` or `3.5`). - ``: The version of Scala used to build Spark (`2.12` or `2.13`). - ``: The version of ScalarDB Analytics. - ``: The name of the catalog. This must match a catalog created on the ScalarDB Analytics server. - ``: The IP address of your Managed Private Endpoint for the ScalarDB Analytics server.

Set up private connectivity

ScalarDB Analytics requires network connectivity from the Managed Virtual Network of the Synapse workspace to both the ScalarDB Analytics server and data sources. To establish this connectivity, you must use a Private Link Service and Managed Private Endpoints as follows: 1. **Connectivity between the Synapse workspace and the ScalarDB Analytics server:** Create a Private Link Service for the Internal Load Balancer in Azure Kubernetes Service (AKS), then create a Managed Private Endpoint in the Managed Virtual Network of the Synapse workspace. 2. **Connectivity between the Synapse workspace and data sources:** Create a Managed Private Endpoint in the Managed Virtual Network of the Synapse workspace directly to each data source (for example, Azure Database for PostgreSQL). :::caution If you are using Azure Database for PostgreSQL Flexible Server, it must be created with **Public access** mode. VNet integration mode does not support adding private endpoints after creation. For more details, see the [Networking](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-networking) section of the Azure Database for PostgreSQL documentation. :::
Create a Private Link Service for the ScalarDB Analytics server
To create an Internal Load Balancer for the ScalarDB Analytics server, set `scalarDbAnalyticsServer.service.type` to `LoadBalancer` in your custom values file for the Helm chart. :::note The Helm chart includes the `service.beta.kubernetes.io/azure-load-balancer-internal: "true"` annotation by default, so the Load Balancer will be created as an Internal Load Balancer. ::: After deploying with this configuration, proceed with the following steps. 1. Verify that an internal IP address is assigned to the ScalarDB Analytics server service: ```console kubectl get svc -n ``` Replace `` with the namespace where ScalarDB Analytics server is deployed and `` with the name of the service (for example, `scalardb-analytics-server`). Confirm that the service has a private IP address (for example, `10.x.x.x`) assigned in the `EXTERNAL-IP` column. For Internal Load Balancer services, the `EXTERNAL-IP` shows a private IP address instead of a public IP. 2. Create a Private Link Service in Azure Portal: 1. Go to **Private Link**, select **Private Link Services**, and choose **Create**. 2. Select the Internal Load Balancer for the ScalarDB Analytics server service. 3. Configure the Source NAT subnet and access security. 3. Create a Managed Private Endpoint in Synapse Studio: 1. Go to **Manage**, select **Managed private endpoints**, and choose **New**. 2. Select **Private Link Service**, and then choose the Private Link Service you created. 3. After creation, approve the connection in the Private Link Service settings. 4. Note the IP address of the Managed Private Endpoint for use in Spark configuration.
Create a Managed Private Endpoint for the data source
1. In Synapse Studio, go to **Manage**, select **Managed private endpoints**, and choose **New**. 2. Select your database type (for example, **Azure Database for PostgreSQL Flexible Server**). 3. Select your database server, and then create the endpoint. 4. Approve the connection in the database server's networking settings: 1. Go to **Azure Portal**, select **Your database server**, choose **Networking**, and select **Private access**. 2. Select the pending connection, and then approve it.

Run analytical queries via the Spark driver

To run Spark driver applications: 1. Upload your Spark driver application JAR to Azure Data Lake Storage Gen2. 2. Go to **Develop**, and select **New Spark job definition**. 3. Configure the job: - **Language:** Spark (Scala/Java) - **Main class:** Your application's main class - **Main definition file:** Path to your JAR file (for example, `abfss://@.dfs.core.windows.net/path/to/app.jar`) - ``: Your Azure Data Lake Storage Gen2 container name - ``: Your Azure storage account name - **Spark pool:** Your configured Spark pool 4. Submit the job. :::note Make sure the required permissions are set as described in the [Prerequisites](#prerequisites) section. :::

Run interactive queries by using notebooks

You can also use Azure Synapse Analytics notebooks to run interactive queries through ScalarDB Analytics. After configuring the Spark pool, you can create a notebook and attach it to the Spark pool. For details on how to use Azure Synapse Analytics notebooks, refer to the [Azure Synapse Analytics notebooks documentation](https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-notebook-concept). After attaching a notebook to the Spark pool, you can run SQL queries as follows. To list the available catalogs, run the following command: ```sql -- List available catalogs SHOW CATALOGS ``` To list the databases in the ScalarDB catalog, run the following command, replacing `` with your actual catalog name configured in the ScalarDB Analytics server. ```sql -- List databases in ScalarDB catalog ``` To query the data, run the following command, replacing the placeholders in angle brackets with your actual catalog, data source, namespace, and table names configured in the ScalarDB Analytics server. ```sql -- Query data SELECT * FROM ... LIMIT 10 ``` You can use Google Cloud Dataproc to run analytical queries through ScalarDB Analytics. Dataproc provides both Compute-based clusters and Serverless, and ScalarDB Analytics supports both. For the basics of Dataproc, refer to the [Google Cloud Dataproc documentation](https://cloud.google.com/dataproc/docs).

Set up a Google Cloud environment

To use ScalarDB Analytics with Dataproc, you first need to set up a Google Cloud environment: create a VPC, create a database instance for the ScalarDB Analytics server, create a Google Kubernetes Engine (GKE) cluster, and deploy the ScalarDB Analytics server.

Create a VPC

By default, the ScalarDB Analytics server Helm Chart configures the LoadBalancer service to use a private IP from the VPC for its external IP. Additionally, it's recommended to access Cloud SQL via private IP. Therefore, you need to create a VPC for a private IP access. Allow the following ports in the firewall rules for the VPC: - **For Dataproc internal communication:** TCP 0-65535, UDP 0-65535, ICMP (within the subnet only) - **For SSH via Identity-Aware Proxy (IAP):** TCP 22 - **For Cloud SQL:** The port corresponding to your database (within the subnet only) The following is an example of creating a VPC, subnet, and firewall rules: ```bash # Create a VPC gcloud compute networks create \ --subnet-mode=custom \ --project= # Create a subnet gcloud compute networks subnets create \ --network= \ --region= \ --range=10.0.0.0/20 \ --project= # For Dataproc internal communication gcloud compute firewall-rules create -allow-internal \ --network= \ --allow=tcp:0-65535,udp:0-65535,icmp \ --source-ranges=10.0.0.0/20 \ --project= # For SSH via IAP gcloud compute firewall-rules create -allow-iap-ssh \ --network= \ --allow=tcp:22 \ --source-ranges=35.235.240.0/20 \ --project= # For Cloud SQL (PostgreSQL example) gcloud compute firewall-rules create -allow-postgres \ --network= \ --allow=tcp:5432 \ --source-ranges=10.0.0.0/20 \ --project= ``` Replace the following placeholders: - ``: Your VPC name - ``: Your subnet name - ``: Your Google Cloud project ID - ``: The region (for example, `us-west1`) For details on VPC and firewall rules, refer to [Create and manage VPC networks](https://cloud.google.com/vpc/docs/create-modify-vpc-networks) and [Using IAP for TCP forwarding](https://cloud.google.com/iap/docs/using-tcp-forwarding).

Create a database instance for the ScalarDB Analytics server

The ScalarDB Analytics server requires a backend database. This section explains how to set one up by using Cloud SQL. You can use other databases depending on your requirements. Google Cloud recommends connecting via a private IP. This guide enables private IP connections. To use a private IP, you need to configure private service access in the VPC beforehand. For details, refer to [Configuring private services access](https://cloud.google.com/vpc/docs/configure-private-services-access). The following is an example of creating a Cloud SQL instance with a private IP enabled: ```bash gcloud sql instances create \ --database-version=POSTGRES_15 \ --tier=db-custom-2-4096 \ --region= \ --network=projects//global/networks/ \ --no-assign-ip \ --project= ``` Replace the following placeholders: - ``: Your Cloud SQL instance name - ``: The region (for example, `us-west1`) - ``: Your Google Cloud project ID - ``: Your VPC name After creating the instance, create a database and a user: ```bash # Create a database gcloud sql databases create \ --instance= \ --project= # Create a user gcloud sql users create \ --instance= \ --password= \ --project= ``` For details, refer to [Configuring private IP for Cloud SQL](https://cloud.google.com/sql/docs/postgres/configure-private-ip).

Deploy the ScalarDB Analytics server on GKE

Follow these steps for deploying the ScalarDB Analytics server on GKE. **1. Create a GKE Autopilot cluster** Create a GKE Autopilot cluster to deploy the ScalarDB Analytics server. Using the subnet that you created in the VPC section ensures that the LoadBalancer's external IP will be a private IP within the VPC. ```bash gcloud container clusters create-auto \ --region= \ --network= \ --subnetwork= \ --project= ``` Replace the following placeholders: - ``: Your GKE cluster name - ``: The region (for example, `us-west1`) - ``: The VPC name created in the VPC section - ``: The subnet name created in the VPC section - ``: Your Google Cloud project ID After creating the cluster, get the credentials so that kubectl can connect to the cluster: ```bash gcloud container clusters get-credentials \ --region= \ --project= ``` **2. Install Cloud SQL Auth Proxy Operator** When using Cloud SQL as the backend database, Google Cloud recommends connecting via Cloud SQL Auth Proxy. Since the ScalarDB Analytics server Helm Chart does not support adding sidecar containers, this guide uses Cloud SQL Auth Proxy Operator to deploy the sidecar container. To install Cloud SQL Auth Proxy Operator on the GKE cluster, follow [Connect using the Cloud SQL Proxy Operator](https://cloud.google.com/sql/docs/postgres/connect-proxy-operator) . **3. Create an AuthProxyWorkload resource** Create an AuthProxyWorkload resource to inject the Cloud SQL Auth Proxy sidecar into the ScalarDB Analytics server deployment. When a Deployment matching the `workloadSelector` of this resource is deployed, a Cloud SQL Auth Proxy sidecar container will be automatically added. The following is an example AuthProxyWorkload configuration: ```yaml apiVersion: cloudsql.cloud.google.com/v1 kind: AuthProxyWorkload metadata: name: scalardb-analytics-server-cloudsql-auth-proxy spec: workloadSelector: kind: "Deployment" name: "scalardb-analytics-server" instances: - connectionString: "" portEnvName: "DB_PORT" hostEnvName: "INSTANCE_HOST" ``` Replace the following placeholder: - ``: The Cloud SQL instance connection name. You can get it by running the following command: ```bash gcloud sql instances describe \ --project= \ --format="value(connectionName)" ``` Create the AuthProxyWorkload resource with the following command: ```bash kubectl apply -f auth-proxy-workload.yaml ``` **4. Deploy the ScalarDB Analytics server** To deploy the ScalarDB Analytics server by using Helm, follow [Deploy a ScalarDB Analytics server](./deploy-scalardb-analytics-server.mdx). When using Cloud SQL Auth Proxy, database connections are made through the sidecar container. Use the environment variables `INSTANCE_HOST` and `DB_PORT` configured in AuthProxyWorkload to specify the connection destination. The following is an example Helm values file: ```yaml scalarDbAnalyticsServer: image: repository: ghcr.io/scalar-labs/scalardb-analytics-server-byol properties: | # License configuration scalar.db.analytics.server.licensing.license_key= scalar.db.analytics.server.licensing.license_check_cert_pem= # Database configuration scalar.db.analytics.server.db.contact_points=jdbc:postgresql://${INSTANCE_HOST}:${DB_PORT}/ scalar.db.analytics.server.db.username= scalar.db.analytics.server.db.password= # Metering storage configuration scalar.db.analytics.server.metering.storage.provider=google-cloud-storage scalar.db.analytics.server.metering.storage.bucket_name= scalar.db.analytics.server.metering.storage.accessKeyId= scalar.db.analytics.server.metering.storage.secretAccessKey= serviceAccount: serviceAccountName: scalardb-analytics-server ``` Replace the following placeholders: - ``: Your license key - ``: Your license verification certificate. For guidance on how to set up your license, including both commercial and trial licenses, see [How to Configure a License Key](../scalar-licensing/index.mdx). - ``: The database name created in the Cloud SQL section - ``: The database username - ``: The database password - ``: The Cloud Storage bucket name for storing metering information. For how to create a bucket, refer to [Create buckets](https://cloud.google.com/storage/docs/creating-buckets). - ``: The Cloud Storage HMAC access key - ``: The Cloud Storage HMAC secret key. For how to create HMAC keys, refer to [Manage HMAC keys](https://cloud.google.com/storage/docs/authentication/managing-hmackeys). **5. Verify the ScalarDB Analytics server host address** After deployment, verify the LoadBalancer's external IP of the ScalarDB Analytics server by running the following command. This IP address will be used as `` in the subsequent Dataproc section. ```bash kubectl get svc scalardb-analytics-server ``` Example output: ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-analytics-server LoadBalancer 10.98.116.121 203.0.113.10 11051:32619/TCP,11052:32598/TCP 2m43s ``` The IP address shown in the `EXTERNAL-IP` column (for example, `203.0.113.10`) is the value for ``.

Set up a Dataproc cluster

You can create a Dataproc Compute cluster by running the following command: ```bash gcloud dataproc clusters create \ --region= \ --subnet=projects//regions//subnetworks/ \ --properties=spark:spark.jars.packages=com.scalar-labs:scalardb-analytics-spark-all-_: \ --properties=spark:spark.extraListeners=com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener \ --properties=spark:spark.sql.catalog.=com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog \ --properties=spark:spark.sql.catalog..server.host= \ --properties=spark:spark.sql.catalog..server.catalog.port=11051 \ --properties=spark:spark.sql.catalog..server.metering.port=11052 \ --project= ``` Replace the following placeholders: - ``: Your Dataproc cluster name - ``: Your catalog name. This must match the catalog name created on the ScalarDB Analytics server. - ``: The region (for example, `us-west1`) - ``: Your Google Cloud project ID - ``: The subnet name created in the VPC section - ``: The Spark version (for example, `3.5` or `3.4`). Refer to [Dataproc version list](https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-version-clusters). - ``: The Scala version used to build Spark (for example, `2.13` or `2.12`) - ``: The ScalarDB Analytics version (for example, `3.18.0`) - ``: The ScalarDB Analytics server host address verified in the GKE section After creating the cluster, you can connect via SSH through IAP and run analytical queries by using `spark-shell` or `spark-sql` on the cluster: ```bash gcloud compute ssh -m \ --zone= \ --tunnel-through-iap \ --project= ``` Replace the following placeholders: - ``: The Dataproc cluster name you created - ``: The zone where the cluster's primary node exists (for example, `us-west1-a`) - ``: Your Google Cloud project ID Since Dataproc Compute clusters provide a standard Apache Spark environment, you can run Spark applications by using standard methods. For more information, refer to the [Apache Spark documentation](https://spark.apache.org/docs/latest/submitting-applications.html). For details on Spark driver applications, refer to [Spark driver application](./run-analytical-queries.mdx?spark-application-type=spark-driver#develop-a-spark-application).

Run interactive queries by using notebooks

You can also use Jupyter notebooks on Dataproc Compute clusters to run interactive queries through ScalarDB Analytics. To enable Jupyter notebooks, add the `--optional-components=JUPYTER` flag and enable the Component Gateway when creating the cluster. For details on how to use Jupyter notebooks on Dataproc, refer to the [Dataproc Jupyter component documentation](https://cloud.google.com/dataproc/docs/concepts/components/jupyter). After accessing the Jupyter notebook through the Component Gateway, you can run SQL queries by using the `%%sql` magic command. To list the available catalogs, run the following command: ```sql %%sql -- List available catalogs SHOW CATALOGS ``` To list the databases in the ScalarDB catalog, run the following command, replacing `` with your actual catalog name configured in the ScalarDB Analytics server. ```sql %%sql -- List databases in ScalarDB catalog SHOW DATABASES IN ``` To query the data, run the following command, replacing the placeholders in angle brackets with your actual catalog, data source, namespace, and table names configured in the ScalarDB Analytics server. ```sql %%sql -- Query data SELECT * FROM ... LIMIT 10 ```

Set up Dataproc Serverless

Dataproc Serverless is a Spark Connect-based service. Currently, only Python clients are provided, so you must use it from notebooks or Python clients. The following is an example configuration in Python: ```python from google.cloud.dataproc_spark_connect import DataprocSparkSession from google.cloud.dataproc_v1 import Session session_config = Session() session_config.environment_config.execution_config.subnetwork_uri = "projects//regions//subnetworks/" spark = ( DataprocSparkSession.builder.projectId("") .location("") .dataprocSessionConfig(session_config) .config( "spark.jars.packages", "com.scalar-labs:scalardb-analytics-spark-all-_:", ) .config( "spark.extraListeners", "com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener", ) .config( "spark.sql.catalog.", "com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog", ) .config("spark.sql.catalog..server.host", "") .config("spark.sql.catalog..server.catalog.port", "11051") .config("spark.sql.catalog..server.metering.port", "11052") .getOrCreate() ) ``` Replace the following placeholders: - ``: Your catalog name. This must match the catalog name created on the ScalarDB Analytics server. - ``: Your Google Cloud project ID - ``: The region (for example, `us-west1`) - ``: The subnet name created in the VPC section - ``: The Spark version (for example, `3.5` or `3.4`) - ``: The Scala version used to build Spark (for example, `2.13` or `2.12`) - ``: The ScalarDB Analytics version (for example, `3.18.0`) - ``: The ScalarDB Analytics server host address verified in the GKE section For details on Spark Connect applications, refer to [Spark Connect application](./run-analytical-queries.mdx?spark-application-type=spark-connect#develop-a-spark-application) and [Dataproc Serverless for Spark documentation](https://cloud.google.com/dataproc-serverless/docs). ================================================ FILE: docs/scalardb-analytics/design.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # ScalarDB Analytics Design and Implementation ScalarDB Analytics is the analytical component of ScalarDB. Similar to ScalarDB, it unifies diverse data sources—ranging from RDBMSs like PostgreSQL and MySQL to NoSQL databases like Cassandra and DynamoDB—into a single logical database. This enables you to perform analytical queries across multiple databases seamlessly. ## Design ScalarDB Analytics consists of two main components: a universal data catalog and a query engine: - **Universal data catalog.** The universal data catalog is a flexible metadata management system that handles multiple catalog spaces. Each catalog space provides an independent logical grouping of data sources, enabling organized management of diverse data environments. - **Query engine.** The query engine executes queries against the universal data catalog. ScalarDB Analytics provides appropriate data connectors to interface with the underlying data sources. ScalarDB Analytics employs a decoupled architecture where the data catalog and query engine are separate components. This design allows for integration with various existing query engines through an extensible architecture. As a result, you can select different query engines to execute queries against the same data catalog based on your specific requirements. ### Universal data catalog The universal data catalog is composed of several levels and is structured as follows: ```mermaid graph TD C[Catalog] --> D[Data Source] C[Catalog] --> D2[Data Source] subgraph " " D --> N[Namespace] D --> N2[Namespace] N --> T[Table] N --> T2[Table] T --> TC[Column] T --> TC2[Column] D2 end ``` The following are definitions for those levels: - **Catalog** is a folder that contains all your data source information. For example, you might have one catalog called `analytics_catalog` for your analytics data and another called `operational_catalog` for your day-to-day operations. - **Data source** represents each data source you connect to. For each data source, ScalarDB Analytics stores important information like: - What kind of data source it is (PostgreSQL, Cassandra, etc.) - How to connect to it (connection details and passwords) - Special features the data source supports (like transactions) - **Namespace** is like a subfolder within your data source that groups related tables together. In PostgreSQL these are called schemas, in Cassandra they're called keyspaces. You can have multiple levels of namespaces, similar to having folders within folders. - **Table** is where your actual data lives. For each table, ScalarDB Analytics keeps track of: - What columns it has - What type of data each column can store - Whether columns can be empty (null) #### Data source integration When registering a data source to ScalarDB Analytics, two types of mappings occur: 1. **Catalog structure mapping:** The data source's catalog information (namespaces, tables, and columns) is resolved and mapped to the universal data catalog structure. 2. **Data type mapping:** Native data types from each data source are mapped to the [supported data types](#supported-data-types). These mappings ensure compatibility and consistency across different database systems. For detailed information about how specific databases are mapped, see [Catalog structure mappings by data source](./reference-data-source.mdx#catalog-structure-mappings-by-data-source). ### Query engine A query engine is an independent component along with the universal data catalog, which is responsible for executing queries against the data sources registered in the universal data catalog and returning the results to the user. ScalarDB Analytics does not currently provide a built-in query engine. Instead, it is designed to be integrated with existing query engines, normally provided as a plugin of the query engine. When you run a query, the ScalarDB Analytics query engine plugin works as follows: 1. Fetches the catalog metadata by calling the universal data catalog API, like the data source location, the table object identifier, and the table schema. 2. Sets up the data source connectors to the data sources by using the catalog metadata. 3. Provides the query optimization information to the query engine based on the catalog metadata. 4. Reads the data from the data sources by using the data source connectors. ScalarDB Analytics manages these processes internally. You can simply run a query against the universal data catalog by using the query engine API in the same way that you would normally run a query. ScalarDB Analytics currently supports Apache Spark as its query engine. For details on how to use ScalarDB Analytics with Spark, see [Run Analytical Queries Through ScalarDB Analytics](./run-analytical-queries.mdx). ## Implementation ScalarDB Analytics provides the following components to implement the design described above. ### ScalarDB Analytics server The ScalarDB Analytics server is a server-side component that manages the universal data catalog. It stores and manages catalog metadata, including data source configurations, namespace structures, and table schemas. The server exposes APIs that query engines and CLI tools use to interact with the universal data catalog. For details on how to set up the server and create a catalog, see [Create a ScalarDB Analytics Catalog](./create-scalardb-analytics-catalog.mdx). #### Supported data types ScalarDB Analytics supports a wide range of data types across different data sources. The universal data catalog maps these data types to a common set of types to ensure compatibility and consistency across sources. The following list shows the supported data types in ScalarDB Analytics: - `BYTE` - `SMALLINT` - `INT` - `BIGINT` - `FLOAT` - `DOUBLE` - `DECIMAL` - `TEXT` - `BLOB` - `BOOLEAN` - `DATE` - `TIME` - `TIMESTAMP` - `TIMESTAMPTZ` - `DURATION` - `INTERVAL` These data types are used across all data sources and provide a unified type system for querying heterogeneous databases. ### ScalarDB Analytics CLI The ScalarDB Analytics CLI is a command-line tool for managing the universal data catalog. You can use it to create catalogs, register data sources, and explore the structure of your registered databases. The CLI communicates with the ScalarDB Analytics server through its catalog API. You configure the server connection by specifying the server host and port in a properties file. For details on CLI configuration and available commands, see [CLI client configuration](./configurations.mdx#cli-client-configuration) and [ScalarDB Analytics CLI Command Reference](./reference-cli-command.mdx). ### Query engine integrations ScalarDB Analytics integrates with existing query engines to execute queries against the universal data catalog. #### Apache Spark plugin ScalarDB Analytics currently supports Apache Spark as its query engine. The integration is provided through the ScalarDB Analytics query engine plugin for Apache Spark, which implements a custom Spark catalog (`ScalarDbAnalyticsCatalog`) that connects to the ScalarDB Analytics server. This plugin exposes data sources registered in the universal data catalog as Spark tables, enabling you to query them by using Spark SQL. With this integration, you can execute arbitrary Spark SQL queries across multiple heterogeneous data sources seamlessly. For details on how to configure and use ScalarDB Analytics with Spark, see [Run Analytical Queries Through ScalarDB Analytics](./run-analytical-queries.mdx). ================================================ FILE: docs/scalardb-analytics/quickstart.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB Analytics import WarningLicenseKeyContact from "/src/components/en-us/_warning-license-key-contact.mdx"; This getting-started tutorial guide explains how to set up ScalarDB Analytics and run federated queries across different databases, including PostgreSQL, MySQL, and Cassandra. For an overview of ScalarDB Analytics and its key benefits, refer to the [ScalarDB Overview](../overview.mdx) and [ScalarDB Design](../design.mdx) pages. ## What you'll build In this tutorial, you'll set up a sample e-commerce analytics environment where: - Customer data resides in PostgreSQL - Order data is managed by ScalarDB in MySQL - Line item details are stored in Cassandra, which are updated through ScalarDB transactions You'll run analytical queries that join data across all three databases to gain business insights. The source code is available at [https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-analytics-sample](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-analytics-sample). ## Prerequisites - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ## Step 1: Set up the environment This section describes how to set up a ScalarDB Analytics environment. ### Clone the repository Open **Terminal**, and clone the ScalarDB samples repository: ```console git clone https://github.com/scalar-labs/scalardb-samples cd scalardb-samples/scalardb-analytics-sample ``` ### Configure your license To add your ScalarDB Analytics license, open `config/scalardb-analytics-server.properties`. Then, uncomment and update the license configuration lines, replacing `` and `` with your actual license information: ```properties # License configuration (required for production) scalar.db.analytics.server.licensing.license_key= scalar.db.analytics.server.licensing.license_check_cert_pem= ``` ## Step 2: Set up the sample databases To set up the sample databases, run the following command: ```console docker compose up -d --wait ``` This command starts the following services locally: - **ScalarDB Analytics components:** - **ScalarDB Analytics server:** Manages metadata about all data sources and provides a unified interface for querying. - **Sample databases:** - **PostgreSQL:** Used as a non-ScalarDB-managed database (accessed directly) - **Cassandra and MySQL:** Used as ScalarDB-managed databases (accessed through ScalarDB's transaction layer) In this guide, PostgreSQL is referred to as a **non-ScalarDB-managed database**, which is not managed by ScalarDB transactions, while Cassandra and MySQL are referred to as **ScalarDB-managed databases**, which are managed by ScalarDB transactions. The sample data is automatically loaded into all databases during the initial setup. After completing the setup, the following tables should be available: - In PostgreSQL: - `sample_ns.customer` - In ScalarDB (backed by Cassandra): - `cassandrans.lineitem` - In ScalarDB (backed by MySQL): - `mysqlns.orders` According to the above, within ScalarDB, `cassandrans` and `mysqlns` are mapped to Cassandra and MySQL, respectively. For details about the table schema, including column definitions and data types, refer to [Schema details](#schema-details). Ensure that the sample data has been successfully loaded into these tables. ## Step 3: Register data sources by using the ScalarDB Analytics CLI Before running analytical queries, you need to register the data sources with the ScalarDB Analytics server. You can do this by using the ScalarDB Analytics CLI. ### Create a catalog First, create a new catalog to organize your data sources: ```console docker compose run --rm scalardb-analytics-cli catalog create --catalog sample_catalog ``` ### Register ScalarDB as a data source Register the ScalarDB-managed databases: ```console docker compose run --rm scalardb-analytics-cli data-source register \ --catalog=sample_catalog --data-source=scalardb --provider-file=/config/data-sources/scalardb.json ``` This registers tables from both Cassandra and MySQL, which are managed by ScalarDB. ### Register PostgreSQL as a data source Register the PostgreSQL database: ```console docker compose run --rm scalardb-analytics-cli data-source register \ --catalog=sample_catalog --data-source=postgres --provider-file=/config/data-sources/postgres.json ``` ## Step 4: Launch the Spark SQL console To launch the Spark SQL console, run the following command: ```console docker compose run --rm spark-sql ``` While launching the Spark SQL console, the ScalarDB Analytics catalog is initialized with the configuration in **spark-defaults.conf** and is registered as a Spark catalog named `sample_catalog`. ### Namespace mapping The following tables in the configured data sources are mapped to Spark SQL tables, allowing seamless querying across different data sources: - For PostgreSQL: - `sample_catalog.postgres.sample_ns.customer` - For ScalarDB (backed by Cassandra): - `sample_catalog.scalardb.cassandrans.lineitem` - For ScalarDB (backed by MySQL): - `sample_catalog.scalardb.mysqlns.orders` ## Step 5: Run analytical queries Now that you've set up your ScalarDB Analytics environment, you can run analytical queries on the sample data by using the Spark SQL console. ### Query 1: Analyze shipping performance and returns The SQL query below demonstrates basic analytical capabilities by examining line item data from Cassandra. The query helps answer business questions like: - What percentage of items are returned versus shipped successfully? - What's the financial impact of returns? - How does pricing vary between different order statuses? The query calculates key metrics grouped by return status and line status: ```sql SELECT l_returnflag, l_linestatus, sum(l_quantity) AS sum_qty, sum(l_extendedprice) AS sum_base_price, sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, avg(l_quantity) AS avg_qty, avg(l_extendedprice) AS avg_price, avg(l_discount) AS avg_disc, count(*) AS count_order FROM sample_catalog.scalardb.cassandrans.lineitem WHERE to_date(l_shipdate, 'yyyy-MM-dd') <= date '1998-12-01' - 3 GROUP BY l_returnflag, l_linestatus ORDER BY l_returnflag, l_linestatus; ``` You should see the following output: ```console A F 1519 2374824.6560278563 1387364.2207725341 1962763.4654265852 26.649122807017545 41663.590456629056 0.41501802923479575 57 N F 98 146371.2295412012 85593.96776336085 121041.55837332775 32.666666666666664 48790.409847067065 0.40984706454007996 3 N O 5374 8007373.247086477 4685647.785126835 6624210.945739046 24.427272727272726 36397.15112312035 0.4147594809559689 220 R F 1461 2190869.9676265526 1284178.4378283697 1814151.2807494882 25.189655172413794 37773.62013149229 0.41323493790730753 58 ``` ### Query 2: Cross-database analysis for revenue optimization The following SQL query showcases the key capability of ScalarDB Analytics: joining data across different databases without data movement. Specifically, this query joins the customer table in PostgreSQL, the order table in MySQL, and the line items in Cassandra without requiring data movements by, for example, ETL pipelines. The query helps answer business questions like: - To prioritize fulfillment, what are the high-value orders from specific customer segments that haven't shipped yet? The query finds AUTOMOBILE segment customers with unshipped orders, ranked by revenue: ```sql SELECT l_orderkey, sum(l_extendedprice * (1 - l_discount)) AS revenue, o_orderdate, o_shippriority FROM sample_catalog.postgres.sample_ns.customer, sample_catalog.scalardb.mysqlns.orders, sample_catalog.scalardb.cassandrans.lineitem WHERE c_mktsegment = 'AUTOMOBILE' AND c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderdate < '1995-03-15' AND l_shipdate > '1995-03-15' GROUP BY l_orderkey, o_orderdate, o_shippriority ORDER BY revenue DESC, o_orderdate, l_orderkey LIMIT 10; ``` You should see the following output: ```console 1071617 128186.99915996166 1995-03-10 0 1959075 33104.51278645416 1994-12-23 0 430243 19476.115819260962 1994-12-24 0 ``` The result indicates that the shipment of the order with the order key `1071617` should be prioritized. :::note You can also run any arbitrary query that Apache Spark and Spark SQL support on the imported tables in this sample tutorial. Since ScalarDB Analytics supports all queries that Spark SQL supports, you can do not only selections (filtering), joins, aggregations, and ordering, as shown in the example, but also window functions, lateral joins, and other various operations. To see which types of queries Spark SQL supports, see the [Spark SQL documentation](https://spark.apache.org/docs/latest/sql-ref.html). ::: ## Step 6: Stop the sample application To stop the sample application and remove all associated volumes, run the following command. This action shuts down all services and deletes any persisted data stored in the volumes, resetting the application state: ```console docker compose down -v ``` ## Reference ### Schema details The following entity relationship diagram illustrates the relationships between the tables across PostgreSQL, MySQL, and Cassandra, with foreign keys linking customers, orders, and line items. ```mermaid erDiagram "postgres.sample_ns.customer" ||--|{ "scalardb.mysqlns.orders" : "custkey" "postgres.sample_ns.customer" { int c_custkey text c_name text c_address int c_nationkey text c_phone double c_acctbal text c_mktsegment text c_comment } "scalardb.mysqlns.orders" ||--|{ "scalardb.cassandrans.lineitem" : "orderkey" "scalardb.mysqlns.orders" { int o_orderkey int o_custkey text o_orderstatus double o_totalprice text o_orderdate text o_orderpriority text o_clerk int o_shippriority text o_comment } "scalardb.cassandrans.lineitem" { int l_orderkey int l_partkey int l_suppkey int l_linenumber double l_quantity double l_extendedprice double l_discount double l_tax text l_returnflag text l_linestatus text l_shipdate text l_commitdate text l_receiptdate text l_shipinstruct text l_shipmode text l_comment } ``` - `postgres.sample_ns.customer` comes from PostgreSQL, which is not managed by ScalarDB. - `scalardb.mysqlns.orders` and `scalardb.cassandrans.lineitem` come from ScalarDB, which are backed by MySQL and Cassandra, respectively. The following are brief descriptions of the tables: - **`postgres.sample_ns.customer`.** A table that represents information about customers. This table includes attributes like customer key, name, address, phone number, and account balance. - **`scalardb.mysqlns.orders`.** A table that contains information about orders that customers have placed. This table includes attributes like order key, customer key, order status, order date, and order priority. - **`scalardb.cassandrans.lineitem`.** A table that represents line items associated with orders. This table includes attributes such as order key, part key, supplier key, quantity, price, and shipping date. ================================================ FILE: docs/scalardb-analytics/reference-cli-command.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- # ScalarDB Analytics CLI Command Reference The ScalarDB Analytics CLI uses a hierarchical command structure: ``` scalardb-analytics-cli [options] ``` Available resources: - **catalog:** Top-level containers for organizing data sources - **data-source:** External databases registered within catalogs - **namespace:** Database-specific organizational units - **table:** Database tables within namespaces ## Catalog operations This section describes how to create a new catalog, list all catalogs, show catalog details, and delete a catalog. ### Create a new catalog Creating a new catalog can be done as follows. Please replace `` with the name of a catalog you create. ``` scalardb-analytics-cli catalog create --catalog ``` ### List all catalogs Display all existing catalogs in the system. ``` scalardb-analytics-cli catalog list ``` ### Show catalog details Display detailed information about a specific catalog. You can specify the catalog either by its name or by its UUID. To specify by catalog name: ``` scalardb-analytics-cli catalog describe --catalog ``` Please replace `` with the name of the catalog you want to describe. To specify by catalog ID: ``` scalardb-analytics-cli catalog describe --catalog-id ``` Please replace `` with the UUID of the catalog you want to describe. ### Delete a catalog Remove a catalog from the system. The operation fails if the catalog contains data sources unless you use the `--cascade` option to delete all contents. You can specify the catalog either by its name or by its UUID. To delete an empty catalog by name: ``` scalardb-analytics-cli catalog delete --catalog ``` Please replace `` with the name of the catalog you want to delete. To delete a catalog by ID: ``` scalardb-analytics-cli catalog delete --catalog-id ``` Please replace `` with the UUID of the catalog you want to delete. To delete a catalog and all its contents (data sources and their children): ``` scalardb-analytics-cli catalog delete --catalog --cascade ``` ## Data source operations This section describes how to register a new data source, list all data sources, show data source details, and delete a data source. ### Register a new data source Add a new data source to a catalog by specifying the catalog name, data source name, and provider configuration. ``` scalardb-analytics-cli data-source register --catalog --data-source [schema-option] ``` Please replace: - `` with the name of the catalog that will own the data source. - `` with the name of the data source to register. #### Provider options You must specify the provider configuration by using one of the following options: - `--provider-json `: Inline JSON payload that describes the data source provider. - `--provider-file `: Path to a JSON file that describes the data source provider. - `--provider-stdin`: Read the provider JSON payload from standard input. #### Schema options For the providers that do not support automatic schema resolution, such as DynamoDB, you must manually specify the schema definition by using one of the following methods: - `--schema-json `: Inline JSON payload that describes the data source schema. - `--schema-file `: Path to a JSON file that describes the data source schema. :::note Whether a schema is required depends on the provider type. Some providers, such as PostgreSQL and MySQL, resolve schemas automatically and do not accept manual schema input. Other providers, such as DynamoDB, require a schema to be provided through one of the available schema options. ::: #### Examples To register a data source by using a provider file: ``` scalardb-analytics-cli data-source register --catalog my_catalog --data-source my_datasource --provider-file /path/to/provider.json ``` Please replace `/path/to/provider.json` with the path to the provider file. To register a data source by using inline JSON: ``` scalardb-analytics-cli data-source register --catalog my_catalog --data-source my_datasource --provider-json '{"type":"postgresql","host":"localhost","port":5432,"database":"mydb","user":"user","password":"pass"}' ``` To register a data source by using standard input: ``` cat provider.json | scalardb-analytics-cli data-source register --catalog my_catalog --data-source my_datasource --provider-stdin ``` The provider JSON format is described in the [Data Source Reference](reference-data-source.mdx). ### List all data sources Display all data sources within a specific catalog. ``` scalardb-analytics-cli data-source list --catalog ``` Please replace `` with the name of the catalog whose data sources you want to list. ### Show data source details Display detailed information about a specific data source. You can specify the data source either by its name within a catalog or by its UUID. To specify by catalog and data source name: ``` scalardb-analytics-cli data-source describe --catalog --data-source ``` Please replace: - `` with the name of the catalog containing the data source - `` with the name of the data source you want to describe To specify by data source ID: ``` scalardb-analytics-cli data-source describe --data-source-id ``` Please replace `` with the UUID of the data source you want to describe. ### Delete a data source Remove a data source from a catalog. The operation fails if the data source contains namespaces unless you use the `--cascade` option to delete all contents. You can specify the data source either by its name within a catalog or by its UUID. To delete an empty data source by name: ``` scalardb-analytics-cli data-source delete --catalog --data-source ``` Please replace: - `` with the name of the catalog containing the data source - `` with the name of the data source you want to delete To delete a data source by ID: ``` scalardb-analytics-cli data-source delete --data-source-id ``` Please replace `` with the UUID of the data source you want to delete. To delete a data source and all its contents (namespaces, tables, and columns): ``` scalardb-analytics-cli data-source delete --catalog --data-source --cascade ``` ## Namespace operations This section describes how to list all namespaces and show namespace details. ### List all namespaces Display all namespaces within a specific catalog. ``` scalardb-analytics-cli namespace list --catalog ``` Please replace: - `` with the name of the catalog whose namespaces you want to list ### Show namespace details Display detailed information about a specific namespace. You can specify the namespace either by its name within a data source or by its UUID. For nested namespaces, use `.` as a separator (for example, `--namespace parent.child`). To specify by catalog, data source, and namespace name: ``` scalardb-analytics-cli namespace describe --catalog --data-source --namespace ``` Please replace: - `` with the name of the catalog containing the data source - `` with the name of the data source containing the namespace - `` with the name of the namespace you want to describe To specify by namespace ID: ``` scalardb-analytics-cli namespace describe --namespace-id ``` Please replace `` with the UUID of the namespace you want to describe. ## Table operations This section describes how to list all tables and show the table schema. ### List all tables Display all tables within a specific catalog. ``` scalardb-analytics-cli table list --catalog ``` Please replace: - `` with the name of the catalog containing the data source ### Show the table schema Display the schema information including all columns for a specific table. You can specify the table either by its name within a namespace or by its UUID. For nested namespaces, use `.` as a separator (for example, `--namespace parent.child`). To specify by catalog, data source, namespace, and table name: ``` scalardb-analytics-cli table describe --catalog --data-source --namespace --table ``` Please replace: - `` with the name of the catalog containing the data source - `` with the name of the data source containing the namespace - `` with the name of the namespace containing the table - `` with the name of the table you want to describe To specify by table ID: ``` scalardb-analytics-cli table describe --table-id ``` Please replace `` with the UUID of the table you want to describe. ================================================ FILE: docs/scalardb-analytics/reference-data-source.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; # Data Source Reference import WarningLicenseKeyContact from "/src/components/en-us/_warning-license-key-contact.mdx"; This reference guide provides detailed information about data source configuration formats, provider-specific settings, and data type mappings for ScalarDB Analytics. ## Data source registration file format Data sources are registered to catalogs by using the CLI with provider configuration files. These files define the connection settings for each data source type. For CLI command details, see [CLI command reference](./reference-cli-command.mdx). The provider configuration file has the following structure: ```json { "type": "", // Database type: postgresql, mysql, scalardb, sqlserver, oracle, dynamodb, databricks, snowflake // Type-specific connection configuration // Configuration varies by database type } ``` The catalog name and data source name are specified as CLI options. (`--catalog` and `--data-source`) :::tip File reference syntax You can use the `${file:path}` syntax to load configuration values from an external file. This is useful for reusing existing configuration files or separating sensitive information. **Supported file formats:** - **`.properties` files**: Loaded and converted to a JSON object with string values - **`.json` files**: Loaded as-is (any valid JSON structure) For example: ```json { "type": "scalardb", "configs": "${file:/path/to/scalardb.properties}" } ``` ::: ## Provider configuration by type The following sections show the provider configuration for each supported database type:

Configurations

The following configuration is for ScalarDB.

`configs`

- **Field:** `configs` - **Description:** A map of ScalarDB configuration properties. These are the same properties that would be specified in a ScalarDB configuration file. You can specify the configuration inline as a JSON object or use a file reference with the `${file:path}` syntax.

Example

**Inline configuration:** ```json { "type": "scalardb", "configs": { "scalar.db.contact_points": "localhost", "scalar.db.username": "admin", "scalar.db.password": "admin", "scalar.db.storage": "jdbc" } } ``` **Using file reference with a properties file:** ```json { "type": "scalardb", "configs": "${file:/path/to/scalardb.properties}" } ```

Data access method

ScalarDB Analytics reads data from ScalarDB by using the ScalarDB Core library directly, not through ScalarDB Cluster. As a result, features that are available only in ScalarDB Cluster (such as encryption) cannot be used with the ScalarDB data source.

Scan behavior

Internally, the ScalarDB data source uses the [`Scan` operation with `all()`](../api-guide.mdx#scan-operation) to read data. This operation requires [cross-partition scan](../configurations.mdx#cross-partition-scan-configurations) to be enabled. Filtering and ordering are not applied at the ScalarDB level. The relevant settings are as follows: - `scalar.db.cross_partition_scan.enabled` must be `true` (the default is `true`). - `scalar.db.cross_partition_scan.filtering.enabled` has no effect. - `scalar.db.cross_partition_scan.ordering.enabled` has no effect. :::note Filter push-down and other optimizations may be supported in future releases. :::

ScalarDB Core configuration overrides

[ScalarDB Core configuration properties](../configurations.mdx) are generally respected when used as a ScalarDB data source. However, the following properties are overridden by ScalarDB Analytics:
  • scalar.db.scan_fetch_size: If not explicitly set by the user, defaults to 4096 instead of the ScalarDB Core default of 10.
  • scalar.db.consensus_commit.isolation_level: Always overridden to READ_COMMITTED, regardless of the user-specified value.

Configuration

The following configurations are for PostgreSQL.

`host`

- **Field:** `host` - **Description:** PostgreSQL server hostname.

`port`

- **Field:** `port` - **Description:** Port number.

`username`

- **Field:** `username` - **Description:** Database user.

`password`

- **Field:** `password` - **Description:** Database password.

`database`

- **Field:** `database` - **Description:** Database name to connect to.

Example

```json { "type": "postgresql", "host": "postgres.example.com", "port": 5432, "username": "analytics_user", "password": "secure_password", "database": "customers" } ```

Configuration

The following configurations are for MySQL.

`host`

- **Field:** `host` - **Description:** MySQL server hostname.

`port`

- **Field:** `port` - **Description:** Port number.

`username`

- **Field:** `username` - **Description:** Database user.

`password`

- **Field:** `password` - **Description:** Database password.

`database`

- **Field:** `database` - **Description:** Specific database to import. If omitted, all databases will be imported. - **Default value:** None (imports all databases)

Example

```json { "type": "mysql", "host": "mysql.example.com", "port": 3306, "username": "analytics_user", "password": "secure_password", "database": "orders" // Optional - if omitted, all databases will be imported } ```

Configuration

The following configurations are for Oracle.

`host`

- **Field:** `host` - **Description:** Oracle server hostname.

`port`

- **Field:** `port` - **Description:** Port number.

`username`

- **Field:** `username` - **Description:** Database user.

`password`

- **Field:** `password` - **Description:** Database password.

`serviceName`

- **Field:** `serviceName` - **Description:** Oracle service name.

Example

```json { "type": "oracle", "host": "oracle.example.com", "port": 1521, "username": "analytics_user", "password": "secure_password", "serviceName": "ORCL" } ```

Configuration

The following configurations are for SQL Server.

`host`

- **Field:** `host` - **Description:** SQL Server hostname.

`port`

- **Field:** `port` - **Description:** Port number.

`username`

- **Field:** `username` - **Description:** Database user.

`password`

- **Field:** `password` - **Description:** Database password.

`database`

- **Field:** `database` - **Description:** Specific database to connect to. - **Default value:** None (connects to default database)

`secure`

- **Field:** `secure` - **Description:** Enable encryption. - **Default value:** `false`

Example

```json { "type": "sqlserver", "host": "sqlserver.example.com", "port": 1433, "username": "sa", "password": "secure_password", "database": "analytics", // Optional - if specified, only this database will be imported "secure": true // Optional - enable encryption } ```

Configuration

The following configurations are for Databricks (Databricks SQL/JDBC).

`host`

- **Field:** `host` - **Description:** Databricks workspace hostname (for example, `adb-1234567890123.4.azuredatabricks.net`).

`port`

- **Field:** `port` - **Description:** Port number. - **Default value:** Driver default. (Optional)

`httpPath`

- **Field:** `httpPath` - **Description:** HTTP path of your SQL warehouse or cluster (for example, `/sql/1.0/warehouses/xxxxxxxxxxxxxx`).

`oAuthClientId`

- **Field:** `oAuthClientId` - **Description:** OAuth machine-to-machine (M2M) service principal's UUID or Application ID for Databricks SQL/JDBC authentication.

`oAuthSecret`

- **Field:** `oAuthSecret` - **Description:** OAuth M2M service principal's secret for Databricks SQL/JDBC authentication.

`catalog`

- **Field:** `catalog` - **Description:** Default catalog to use. (Optional)

Example

```json { "type": "databricks", "host": "adb-1234567890123.4.azuredatabricks.net", "port": 443, "httpPath": "/sql/1.0/warehouses/xxxxxxxxxxxxxx", "oAuthClientId": "YOUR_CLIENT_ID", "oAuthSecret": "YOUR_CLIENT_SECRET", "catalog": "main" } ```

Configuration

The following configurations are for Snowflake.

`account`

- **Field:** `account` - **Description:** Snowflake account identifier (for example, `xy12345.ap-northeast-1`).

`username`

- **Field:** `username` - **Description:** A Snowflake user.

`password`

- **Field:** `password` - **Description:** A Snowflake user's programmatic access token.

`database`

- **Field:** `database` - **Description:** Default database to resolve/import. (Optional)

Example

```json { "type": "snowflake", "account": "YOUR-ACCOUNT", "username": "analytics_user", "password": "secure_password", "database": "ANALYTICS" } ```

Configuration

The following configurations are for DynamoDB. :::note AWS Credentials DynamoDB authentication uses the standard AWS SDK credential provider chain. Credentials can be configured through: - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) - AWS credentials file (`~/.aws/credentials`) - IAM roles (when running on EC2, ECS, or Lambda) - AWS SSO or other credential providers supported by the AWS SDK For more information, see the [AWS SDK documentation on credential providers](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html). :::

`region`

- **Field:** `region` - **Description:** AWS region (for example, `us-east-1`). Either `region` or `endpoint` must be specified (not both).

`endpoint`

- **Field:** `endpoint` - **Description:** Custom endpoint URL. Either `region` or `endpoint` must be specified (not both).

Schema definition

Since DynamoDB is schema-less, you must provide a schema definition separately by using the `--schema-json` or `--schema-file` CLI option. The schema cannot be automatically resolved.

Schema structure

The schema file must contain the following structure:
`namespaces[]`
- **Field:** `namespaces[]` - **Description:** Array of namespace definitions.
`namespaces[].names[]`
- **Field:** `namespaces[].names[]` - **Description:** Array of namespace name components (strings). For single-level namespaces, use a single-element array.
`namespaces[].tables[]`
- **Field:** `namespaces[].tables[]` - **Description:** Array of table definitions.
`namespaces[].tables[].name`
- **Field:** `namespaces[].tables[].name` - **Description:** Table name. Must match the DynamoDB table name.
`namespaces[].tables[].columns[]`
- **Field:** `namespaces[].tables[].columns[]` - **Description:** Array of column definitions.
`namespaces[].tables[].columns[].name`
- **Field:** `namespaces[].tables[].columns[].name` - **Description:** Column name. Must match the DynamoDB attribute name.
`namespaces[].tables[].columns[].type`
- **Field:** `namespaces[].tables[].columns[].type` - **Description:** Data type.
`namespaces[].tables[].columns[].nullable`
- **Field:** `namespaces[].tables[].columns[].nullable` - **Description:** Whether the column can contain null values. - **Default value:** `true`

Example

**Provider configuration file (`provider.json`):** ```json { "type": "dynamodb", "region": "us-east-1" } ``` **Schema definition file (`schema.json`):** ```json { "namespaces": [ { "names": ["production"], "tables": [ { "name": "user_events", "columns": [ { "name": "user_id", "type": "TEXT", "nullable": false }, { "name": "event_time", "type": "TIMESTAMP", "nullable": false }, { "name": "event_type", "type": "TEXT" }, { "name": "event_data", "type": "TEXT" } ] } ] } ] } ``` **CLI command:** ```console scalardb-analytics datasource register \ --catalog production \ --data-source dynamodb_events \ --provider-file provider.json \ --schema-file schema.json ``` ## Catalog information reference This section describes catalog structure mappings by data source and data type mappings. ### Catalog structure mappings by data source When registering a data source to ScalarDB Analytics, the catalog structure of the data source, that is, namespaces, tables, and columns, are resolved and registered to the universal data catalog. To resolve the catalog structure of the data source, a particular object on the data sources side are mapped to the universal data catalog object. #### Catalog-level mappings The catalog-level mappings are the mappings of the namespace names, table names, and column names from the data sources to the universal data catalog. To see the catalog-level mappings in each data source, select a data source. The catalog structure of ScalarDB is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: - The ScalarDB namespace is mapped to the namespace. Therefore, the namespace of the ScalarDB data source is always single level, consisting of only the namespace name. - The ScalarDB table is mapped to the table. - The ScalarDB column is mapped to the column. The catalog structure of PostgreSQL is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: - The PostgreSQL schema is mapped to the namespace. Therefore, the namespace of the PostgreSQL data source is always single level, consisting of only the schema name. - Only user-defined schemas are mapped to namespaces. The following system schemas are ignored: - `information_schema` - `pg_catalog` - The PostgreSQL table is mapped to the table. - The PostgreSQL column is mapped to the column. The catalog structure of MySQL is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: - The MySQL database is mapped to the namespace. Therefore, the namespace of the MySQL data source is always single level, consisting of only the database name. - Only user-defined databases are mapped to namespaces. The following system databases are ignored: - `mysql` - `sys` - `information_schema` - `performance_schema` - The MySQL table is mapped to the table. - The MySQL column is mapped to the column. The catalog structure of Oracle is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: - The Oracle schema is mapped to the namespace. Therefore, the namespace of the Oracle data source is always single level, consisting of only schema name. - Only user-defined schemas are mapped to namespaces. The following system schemas are ignored: - `ANONYMOUS` - `APPQOSSYS` - `AUDSYS` - `CTXSYS` - `DBSNMP` - `DGPDB_INT` - `DBSFWUSER` - `DVF` - `DVSYS` - `GGSYS` - `GSMADMIN_INTERNAL` - `GSMCATUSER` - `GSMROOTUSER` - `GSMUSER` - `LBACSYS` - `MDSYS` - `OJVMSYS` - `ORDDATA` - `ORDPLUGINS` - `ORDSYS` - `OUTLN` - `REMOTE_SCHEDULER_AGENT` - `SI_INFORMTN_SCHEMA` - `SYS` - `SYS$UMF` - `SYSBACKUP` - `SYSDG` - `SYSKM` - `SYSRAC` - `SYSTEM` - `WMSYS` - `XDB` - `DIP` - `MDDATA` - `ORACLE_OCM` - `XS$NULL` The catalog structure of SQL Server is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: - Each SQL Server database-schema pair is mapped to a namespace in ScalarDB Analytics. Therefore, the namespace of the SQL Server data source is always two-level, consisting of the database name and the schema name. - Only user-defined databases are mapped to namespaces. The following system databases are ignored: - `master` - `model` - `msdb` - `tempdb` - Only user-defined schemas are mapped to namespaces. The following system schemas are ignored: - `sys` - `guest` - `INFORMATION_SCHEMA` - `db_accessadmin` - `db_backupoperator` - `db_datareader` - `db_datawriter` - `db_ddladmin` - `db_denydatareader` - `db_denydatawriter` - `db_owner` - `db_securityadmin` - The SQL Server table is mapped to the table. - The SQL Server column is mapped to the column. The catalog structure of Databricks is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: - Each Databricks catalog-schema pair is mapped to a namespace in ScalarDB Analytics. Therefore, the namespace of the Databricks data source always has two levels, consisting of the catalog name and the schema name. - The following system catalogs/schemas are ignored: - **Catalogs:** `system` - **Schemas:** `information_schema`, `global_temp`, `sys`, `routines` - The Databricks table is mapped to the table. - The Databricks column is mapped to the column. The catalog structure of Snowflake is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: - Each Snowflake database-schema pair is mapped to a namespace in ScalarDB Analytics. Therefore, the namespace of the Snowflake data source always has two levels, consisting of the database name and the schema name. - The following system databases/schemas are ignored: - **Databases:** `SNOWFLAKE` - **Schemas:** `INFORMATION_SCHEMA` - The Snowflake table is mapped to the table. - The Snowflake column is mapped to the column. Since DynamoDB is schema-less, you need to specify the catalog structure explicitly when registering a DynamoDB data source by using the following format JSON: ```json { "namespaces": [ { "name": "", "tables": [ { "name": "", "columns": [ { "name": "", "type": "" }, ... ] }, ... ] }, ... ] } ``` In the specified JSON, you can use any arbitrary namespace names, but the table names must match the table names in DynamoDB and column name and type must match field names and types in DynamoDB. ### Data type mappings The following sections show how native types from each data source are mapped to ScalarDB Analytics types: :::warning Columns with data types that are not included in the mapping tables below will be ignored during data source registration. These columns will not appear in the ScalarDB Analytics catalog and cannot be queried. Information about ignored columns is logged in the ScalarDB Analytics server logs. ::: | **ScalarDB Data Type** | **ScalarDB Analytics Data Type** | | :--------------------- | :------------------------------- | | `BOOLEAN` | `BOOLEAN` | | `INT` | `INT` | | `BIGINT` | `BIGINT` | | `FLOAT` | `FLOAT` | | `DOUBLE` | `DOUBLE` | | `TEXT` | `TEXT` | | `BLOB` | `BLOB` | | `DATE` | `DATE` | | `TIME` | `TIME` | | `TIMESTAMP` | `TIMESTAMP` | | `TIMESTAMPTZ` | `TIMESTAMPTZ` | | **PostgreSQL Data Type** | **ScalarDB Analytics Data Type** | | :---------------------------- | :------------------------------- | | `integer` | `INT` | | `bigint` | `BIGINT` | | `real` | `FLOAT` | | `double precision` | `DOUBLE` | | `smallserial` | `SMALLINT` | | `serial` | `INT` | | `bigserial` | `BIGINT` | | `char` | `TEXT` | | `varchar` | `TEXT` | | `text` | `TEXT` | | `bpchar` | `TEXT` | | `boolean` | `BOOLEAN` | | `bytea` | `BLOB` | | `date` | `DATE` | | `time` | `TIME` | | `time with time zone` | `TIME` | | `time without time zone` | `TIME` | | `timestamp` | `TIMESTAMP` | | `timestamp with time zone` | `TIMESTAMPTZ` | | `timestamp without time zone` | `TIMESTAMP` | | **MySQL Data Type** | **ScalarDB Analytics Data Type** | | :------------------- | :------------------------------- | | `bit` | `BOOLEAN` | | `bit(1)` | `BOOLEAN` | | `bit(x)` if _x >= 2_ | `BLOB` | | `tinyint` | `SMALLINT` | | `tinyint(1)` | `BOOLEAN` | | `boolean` | `BOOLEAN` | | `smallint` | `SMALLINT` | | `smallint unsigned` | `INT` | | `mediumint` | `INT` | | `mediumint unsigned` | `INT` | | `int` | `INT` | | `int unsigned` | `BIGINT` | | `bigint` | `BIGINT` | | `float` | `FLOAT` | | `double` | `DOUBLE` | | `real` | `DOUBLE` | | `char` | `TEXT` | | `varchar` | `TEXT` | | `text` | `TEXT` | | `binary` | `BLOB` | | `varbinary` | `BLOB` | | `blob` | `BLOB` | | `date` | `DATE` | | `time` | `TIME` | | `datetime` | `TIMESTAMP` | | `timestamp` | `TIMESTAMPTZ` | | **Oracle Data Type** | **ScalarDB Analytics Data Type** | | :------------------------------- | :------------------------------- | | `NUMBER` if _scale = 0_ | `BIGINT` | | `NUMBER` if _scale > 0_ | `DOUBLE` | | `FLOAT` if _precision ≤ 53_ | `DOUBLE` | | `BINARY_FLOAT` | `FLOAT` | | `BINARY_DOUBLE` | `DOUBLE` | | `CHAR` | `TEXT` | | `NCHAR` | `TEXT` | | `VARCHAR2` | `TEXT` | | `NVARCHAR2` | `TEXT` | | `CLOB` | `TEXT` | | `NCLOB` | `TEXT` | | `BLOB` | `BLOB` | | `BOOLEAN` | `BOOLEAN` | | `DATE` | `DATE` | | `TIMESTAMP` | `TIMESTAMPTZ` | | `TIMESTAMP WITH TIME ZONE` | `TIMESTAMPTZ` | | `TIMESTAMP WITH LOCAL TIME ZONE` | `TIMESTAMP` | | `RAW` | `BLOB` | | **SQL Server Data Type** | **ScalarDB Analytics Data Type** | | :----------------------- | :------------------------------- | | `bit` | `BOOLEAN` | | `tinyint` | `SMALLINT` | | `smallint` | `SMALLINT` | | `int` | `INT` | | `bigint` | `BIGINT` | | `real` | `FLOAT` | | `float` | `DOUBLE` | | `float(n)` if _n ≤ 24_ | `FLOAT` | | `float(n)` if _n ≥ 25_ | `DOUBLE` | | `binary` | `BLOB` | | `varbinary` | `BLOB` | | `char` | `TEXT` | | `varchar` | `TEXT` | | `nchar` | `TEXT` | | `nvarchar` | `TEXT` | | `ntext` | `TEXT` | | `text` | `TEXT` | | `date` | `DATE` | | `time` | `TIME` | | `datetime` | `TIMESTAMP` | | `datetime2` | `TIMESTAMP` | | `smalldatetime` | `TIMESTAMP` | | `datetimeoffset` | `TIMESTAMPTZ` | | **Databricks SQL Data Type** | **ScalarDB Analytics Data Type** | | :--------------------------- | :--------------------------------------------------------------------------------------------- | | `TINYINT` | `SMALLINT` | | `SMALLINT` | `SMALLINT` | | `INT` / `INTEGER` | `INT` | | `BIGINT` | `BIGINT` | | `FLOAT` | `FLOAT` | | `DOUBLE` | `DOUBLE` | | `DECIMAL(p,s)` if _s = 0_ | `BYTE` (p ≤ 2), `SMALLINT` (p 3–4), `INT` (p 5–9), `BIGINT` (p 10–18), `DECIMAL(p,0)` (p > 18) | | `DECIMAL(p,s)` if _s ≠ 0_ | `DECIMAL(p,s)` | | `STRING` / `VARCHAR` | `TEXT` | | `BINARY` | `BLOB` | | `BOOLEAN` | `BOOLEAN` | | `DATE` | `DATE` | | `TIMESTAMP` | `TIMESTAMPTZ` | | `TIMESTAMP_NTZ` | `TIMESTAMP` | :::note - For `DECIMAL` types, when precision and scale are not specified during table creation, the default values of precision = 38 and scale = 0 are applied. - For `DECIMAL` types with scale = 0 and small precision, optimized integer types (`BYTE`, `SMALLINT`, `INT`, `BIGINT`) are used to improve storage and performance. ::: | **Snowflake Data Type** | **ScalarDB Analytics Data Type** | | :--------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | | `NUMBER(p,0)` / `INT` / `INTEGER` / `BIGINT` / `SMALLINT` / `TINYINT` / `BYTEINT` | `BYTE` (p ≤ 2), `SMALLINT` (p 3–4), `INT` (p 5–9), `BIGINT` (p 10–18), `DECIMAL(p,0)` (p > 18) | | `NUMBER(p,s)` / `NUMERIC` / `DECIMAL` if _s ≠ 0_ | `DECIMAL(p,s)` | | `FLOAT` / `FLOAT4` / `FLOAT8` / `DOUBLE` / `DOUBLE PRECISION` / `REAL` | `DOUBLE` | | `VARCHAR` / `STRING` / `TEXT` / `NVARCHAR` / `NVARCHAR2` / `CHAR VARYING` / `NCHAR VARYING` / `CHAR` / `CHARACTER` / `NCHAR` | `TEXT` | | `BINARY` / `VARBINARY` | `BLOB` | | `BOOLEAN` | `BOOLEAN` | | `DATE` | `DATE` | | `TIME` | `TIME` | | `TIMESTAMP_NTZ` / `DATETIME` | `TIMESTAMP` | | `TIMESTAMP_LTZ` | `TIMESTAMPTZ` | | `TIMESTAMP_TZ` | `TIMESTAMPTZ` | :::note - For `NUMBER` and `DECIMAL` types, when precision and scale are not specified during table creation, the default values of precision = 38 and scale = 0 are applied. - For `NUMBER` and `DECIMAL` types with scale = 0 and small precision, optimized integer types (`BYTE`, `SMALLINT`, `INT`, `BIGINT`) are used to improve storage and performance. ::: | **DynamoDB Data Type** | **ScalarDB Analytics Data Type** | | :--------------------- | :------------------------------- | | `String` | `TEXT` | | `Number` | `DOUBLE` | | `Binary` | `BLOB` | | `Boolean` | `BOOLEAN` | | `Null` | `NULL` | | `String Set` | `TEXT` | | `Number Set` | `TEXT` | | `Binary Set` | `TEXT` | | `List` | `TEXT` | | `Map` | `TEXT` | :::note DynamoDB complex data types (String Set, Number Set, Binary Set, List, Map) are mapped to `TEXT` for compatibility. The actual values are serialized as JSON strings in ScalarDB Analytics queries. ::: ================================================ FILE: docs/scalardb-analytics/run-analytical-queries.mdx ================================================ --- tags: - Enterprise Option displayed_sidebar: docsEnglish --- import Tabs from "@theme/Tabs"; import TabItem from "@theme/TabItem"; # Run Analytical Queries Through ScalarDB Analytics This guide explains how to develop ScalarDB Analytics applications. For details on the architecture and design, see [ScalarDB Analytics Design](./design.mdx) ScalarDB Analytics currently uses Spark as an execution engine and provides a Spark custom catalog plugin to provide a unified view of ScalarDB-managed and non-ScalarDB-managed data sources as Spark tables. This allows you to execute arbitrary Spark SQL queries seamlessly. ## Preparation This section describes the prerequisites, setting up ScalarDB Analytics in the Spark configuration, and adding the ScalarDB Analytics dependency. ### Prerequisites - **ScalarDB Analytics server:** A running instance that manages catalog information and connects to your data sources. The server must be set up with at least one data source registered. For registering data sources, see [Create a ScalarDB Analytics Catalog](./create-scalardb-analytics-catalog.mdx). - **Apache Spark:** A compatible version of Apache Spark. For supported versions, see [Spark](../requirements.mdx#spark). If you don't have Spark installed yet, please download the Spark distribution from [Apache's website](https://spark.apache.org/downloads.html). ### Set up ScalarDB Analytics in the Spark configuration ScalarDB Analytics requires specific Spark configurations to integrate with ScalarDB Analytics server. #### Required Spark configurations To use ScalarDB Analytics with Spark, you need to configure: 1. **ScalarDB Analytics package**: Add the JAR dependency that matches your Spark and Scala versions 2. **Metering listener**: Register the listener to track resource usage for billing 3. **Catalog registration**: Register a Spark catalog that connects to your ScalarDB Analytics server When configuring Spark, you must specify a catalog name that matches the catalog created on your ScalarDB Analytics server. This ensures Spark can correctly access the data sources managed by that catalog. #### Example configuration The following is a complete example configuration: ```conf # 1. ScalarDB Analytics package spark.jars.packages com.scalar-labs:scalardb-analytics-spark-all-_: # 2. Metering listener spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener # 3. Catalog registration spark.sql.catalog.myanalytics com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog spark.sql.catalog.myanalytics.server.host analytics-server.example.com spark.sql.catalog.myanalytics.server.catalog.port 11051 spark.sql.catalog.myanalytics.server.metering.port 11052 ``` The following describes what you should change the content in the angle brackets to: - ``: Your Spark version (for example, `3.5` or `3.4`) - ``: Your Scala version (for example, `2.13` or `2.12`) - ``: The ScalarDB Analytics version (for example, `3.18.0`) In this example: - The catalog name `myanalytics` must match a catalog that exists on your ScalarDB Analytics server. - The ScalarDB Analytics server is running at `analytics-server.example.com`. - Tables will be accessed using the format: `myanalytics...`. :::important The catalog name in your Spark configuration must match the name of a catalog created on the ScalarDB Analytics server by using the CLI. For example, if you created a catalog named `production` on the server, you must use `production` as the catalog name in your Spark configuration properties (for example, `spark.sql.catalog.production`, `spark.sql.catalog.production.server.host`, etc.). ::: :::note Data source configurations are managed by ScalarDB Analytics server. For information on configuring data sources in ScalarDB Analytics server, see [Create a ScalarDB Analytics Catalog](./create-scalardb-analytics-catalog.mdx). ::: ### Build configuration for Spark applications When developing Spark applications that use ScalarDB Analytics, you can add the dependency to your build configuration. For example, with Gradle: ```kotlin dependencies { implementation("com.scalar-labs:scalardb-analytics-spark-all-_:") } ``` :::note If you bundle your application in a fat JAR by using plugins like Gradle Shadow or Maven Shade, exclude ScalarDB Analytics from the fat JAR by using configurations such as `provided` or `shadow`. ::: ## Develop a Spark application In this section, you will learn how to develop a Spark application that uses ScalarDB Analytics in Java. There are three ways to develop Spark applications with ScalarDB Analytics: 1. **Spark driver application**: A traditional Spark application that runs within the cluster 2. **Spark Connect application**: A remote application that uses the Spark Connect protocol 3. **JDBC application**: A remote application that uses the JDBC interface :::note Depending on your environment, you may not be able to use all the methods mentioned above. For details about supported features and deployment options, refer to [Supported managed Spark services and their application types](./deployment.mdx#supported-managed-spark-services-and-their-application-types). ::: With all these methods, you can refer to tables in ScalarDB Analytics by using the same table identifier format. For details about how ScalarDB Analytics maps catalog information from data sources, see [Catalog structure mappings by data source](./reference-data-source.mdx#catalog-structure-mappings-by-data-source). You can use a commonly used `SparkSession` class for ScalarDB Analytics. Additionally, you can use any type of cluster deployment that Spark supports, such as YARN, Kubernetes, standalone, or local mode. To read data from tables in ScalarDB Analytics, you can use the `spark.sql` or `spark.read.table` function in the same way as when reading a normal Spark table. First, you need to set up your Java project. For example, if you are using Gradle, you can add the following to your `build.gradle.kts` file: ```kotlin dependencies { implementation("com.scalar-labs:scalardb-analytics-spark-all-_:") } ``` Below is an example of a Spark driver application: ```java import org.apache.spark.sql.SparkSession; public class MyApp { public static void main(String[] args) { // Create a SparkSession try (SparkSession spark = SparkSession.builder().getOrCreate()) { // Read data from a table in ScalarDB Analytics spark.sql("SELECT * FROM my_catalog.my_data_source.my_namespace.my_table").show(); } } } ``` Then, you can build and run your application by using the `spark-submit` command. :::note You may need to build a fat JAR file for your application, as is usual for normal Spark applications. ::: ```console spark-submit --class MyApp --master local[*] my-spark-application-all.jar ``` :::tip You can also use other CLI tools that Spark provides, such as `spark-sql` and `spark-shell`, to interact with ScalarDB Analytics tables. ::: You can use [Spark Connect](https://spark.apache.org/spark-connect/) to interact with ScalarDB Analytics. By using Spark Connect, you can access a remote Spark cluster and read data in the same way as a Spark driver application. The following briefly describes how to use Spark Connect. First, you need to start a Spark Connect server in the remote Spark cluster by running the following command: ```console ./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_:,com.scalar-labs:scalardb-analytics-spark-all-_: ``` The following describes what you should change the content in the angle brackets to: - ``: The major and minor version of Scala that matches your Spark installation (`2.12` or `2.13`) - ``: The full version of Spark you are using (for example, `3.5.7`) - ``: The major and minor version of Spark you are using (`3.4` or `3.5`) - ``: The version of ScalarDB Analytics (for example, `3.18.0`) :::note The versions of the packages must match the versions of Spark and ScalarDB Analytics that you are using. ::: You also need to include the Spark Connect client package in your application. For example, if you are using Gradle, you can add the following to your `build.gradle.kts` file: ```kotlin implementation("org.apache.spark:spark-connect-client-jvm_2.12:3.5.7") ``` Then, you can write a Spark Connect client application to connect to the server and read data. ```java import org.apache.spark.sql.SparkSession; public class MyApp { public static void main(String[] args) { try (SparkSession spark = SparkSession.builder() .remote("sc://:") .getOrCreate()) { // Read data from a table in ScalarDB Analytics spark.sql("SELECT * FROM my_catalog.my_data_source.my_namespace.my_table").show(); } } } ``` You can run your Spark Connect client application as a normal Java application by running the following command: ```console java -jar my-spark-connect-client.jar ``` For details about how you can use Spark Connect, refer to the [Spark Connect documentation](https://spark.apache.org/docs/latest/spark-connect-overview.html). Unfortunately, Spark Thrift JDBC server does not support the Spark features that are necessary for ScalarDB Analytics, so you cannot use JDBC to read data from ScalarDB Analytics in your Apache Spark environment. JDBC application is referred to here because some managed Spark services provide different ways to interact with a Spark cluster via the JDBC interface. For more details, refer to [Supported application types](./deployment.mdx#supported-managed-spark-services-and-their-application-types). ## Catalog information mapping ScalarDB Analytics manages its own catalog, containing data sources, namespaces, tables, and columns. That information is automatically mapped to the Spark catalog. In this section, you will learn how ScalarDB Analytics maps its catalog information to the Spark catalog. For details about how information in the raw data sources is mapped to the ScalarDB Analytics catalog, see [Catalog information mappings by data source](./design.mdx#catalog-information-mappings-by-data-source). ### Catalog structure mapping ScalarDB Analytics maps catalog structure from data sources to Spark catalogs. Tables from data sources in the ScalarDB Analytics catalog are mapped to Spark tables by using the following format: ```console ... ``` The following describes what you should change the content in the angle brackets to: - ``: The name of the catalog. - ``: The name of the data source. - ``: The names of the namespaces. If the namespace names are multi-level, they are concatenated with a dot (`.`) as the separator. - ``: The name of the table. For example, if you have a ScalarDB catalog named `my_catalog` that contains a data source named `my_data_source` and a schema named `my_schema`, you can refer to the table named `my_table` in that schema as `my_catalog.my_data_source.my_schema.my_table`. ### Data-type mapping ScalarDB Analytics maps data types in its catalog to Spark data types. The following table shows how the data types are mapped: | ScalarDB Analytics Data Type | Spark Data Type | | :--------------------------- | :----------------- | | `BYTE` | `Byte` | | `SMALLINT` | `Short` | | `INT` | `Integer` | | `BIGINT` | `Long` | | `FLOAT` | `Float` | | `DOUBLE` | `Double` | | `DECIMAL` | `Decimal` | | `TEXT` | `String` | | `BLOB` | `Binary` | | `BOOLEAN` | `Boolean` | | `DATE` | `Date` | | `TIME` | `Timestamp` | | `TIMESTAMP` | `Timestamp` | | `TIMESTAMPTZ` | `Timestamp` | | `DURATION` | `Long` | | `INTERVAL` | `String` | ================================================ FILE: docs/scalardb-benchmarks/README.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Benchmarking Tools import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to run benchmarking tools for ScalarDB. Database benchmarking is helpful for evaluating how databases perform against a set of standards. ## Benchmark workloads - TPC-C - YCSB (Workloads A, C, and F) - Multi-storage YCSB (Workloads C and F) - This YCSB variant is for a multi-storage environment that uses ScalarDB. - Workers in a multi-storage YCSB execute the same number of read and write operations in two namespaces: `ycsb_primary` and `ycsb_secondary`. ## Prerequisites - One of the following Java Development Kits (JDKs): - Gradle - [Kelpie](https://github.com/scalar-labs/kelpie) - Kelpie is a framework for performing end-to-end testing, such as system benchmarking and verification. Get the latest version from [Kelpie Releases](https://github.com/scalar-labs/kelpie), and unzip the archive file. - A client to run the benchmarking tools - A target database - For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). ## Set up the benchmarking tools The following sections describe how to set up the benchmarking tools. ### Clone the ScalarDB benchmarks repository Open **Terminal**, then clone the ScalarDB benchmarks repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-benchmarks ``` Then, go to the directory that contains the benchmarking files by running the following command: ```console cd scalardb-benchmarks ``` ### Build the tools To build the benchmarking tools, run the following command: ```console ./gradlew shadowJar ``` ### Load the schema Before loading the initial data, the tables must be defined by using the [ScalarDB Schema Loader](../schema-loader.mdx). You can download the ScalarDB Schema Loader on the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page. Select the Schema Loader based on how you access ScalarDB: - **Using the ScalarDB Core library (Community edition)?:** Choose `scalardb-schema-loader-.jar` for the version of ScalarDB that you're using. Then, save the `.jar` file in the `scalardb-benchmarks` root directory. - **Using ScalarDB Cluster (Enterprise edition)?:** Choose `scalardb-cluster-schema-loader--all.jar` for the version of ScalarDB Cluster that you're using. Then, save the `.jar` file in the `scalardb-benchmarks` root directory. In addition, you need a properties file to access ScalarDB via the Java CRUD interface. For details about configuring the ScalarDB properties file, see [ScalarDB Core Configurations](../configurations.mdx) or the [Java Client SDK configurations section in ScalarDB Cluster Configurations](../scalardb-cluster/scalardb-cluster-configurations.mdx#java-client-sdk-configurations). After applying the schema and configuring the properties file, select a benchmark and follow the instructions to create the tables. To create tables for TPC-C benchmarking ([`tpcc-schema.json`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/tpcc-schema.json)), run the following command, replacing the contents in the angle brackets as described: ```console java -jar scalardb-schema-loader-.jar --config -f tpcc-schema.json --coordinator ``` If you are using ScalarDB Cluster, run the following command instead: ```console java -jar scalardb-cluster-schema-loader--all.jar --config -f tpcc-schema.json --coordinator ``` To create tables for YCSB benchmarking ([`ycsb-schema.json`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/ycsb-schema.json)), run the following command, replacing the contents in the angle brackets as described: ```console java -jar scalardb-schema-loader-.jar --config -f ycsb-schema.json --coordinator ``` If you are using ScalarDB Cluster, run the following command instead: ```console java -jar scalardb-cluster-schema-loader--all.jar --config -f ycsb-schema.json --coordinator ``` To create tables for multi-storage YCSB benchmarking ([`ycsb-multi-storage-schema.json`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/ycsb-multi-storage-schema.json)), run the following command, replacing the contents in the angle brackets as described: ```console java -jar scalardb-schema-loader-.jar --config -f ycsb-multi-storage-schema.json --coordinator ``` If you are using ScalarDB Cluster, run the following command instead: ```console java -jar scalardb-cluster-schema-loader--all.jar --config -f ycsb-multi-storage-schema.json --coordinator ``` ### Prepare a benchmarking configuration file To run a benchmark, you must first prepare a benchmarking configuration file. The configuration file requires at least the locations of the workload modules to run and the database configuration. The following is an example configuration for running the TPC-C benchmark. The ScalarDB properties file specified for `config_file` should be the properties file that you created as one of the steps in [Load the schema](#load-the-schema). :::note Alternatively, instead of using the ScalarDB properties file, you can specify each database configuration item in the `.toml` file. If `config_file` is specified, all other configurations under `[database_config]` will be ignored even if they are uncommented. ::: ```toml [modules] [modules.preprocessor] name = "com.scalar.db.benchmarks.tpcc.TpccLoader" path = "./build/libs/scalardb-benchmarks-all.jar" [modules.processor] name = "com.scalar.db.benchmarks.tpcc.TpccBench" path = "./build/libs/scalardb-benchmarks-all.jar" [modules.postprocessor] name = "com.scalar.db.benchmarks.tpcc.TpccReporter" path = "./build/libs/scalardb-benchmarks-all.jar" [database_config] config_file = "" #contact_points = "localhost" #contact_port = 9042 #username = "cassandra" #password = "cassandra" #storage = "cassandra" ``` You can define parameters to pass to modules in the configuration file. For details, see the sample configuration files below and available parameters in [Common parameters](#common-parameters): - **TPC-C:** [`tpcc-benchmark-config.toml`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/tpcc-benchmark-config.toml) - **YCSB:** [`ycsb-benchmark-config.toml`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/ycsb-benchmark-config.toml) - **Multi-storage YCSB:** [`ycsb-multi-storage-benchmark-config.toml`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/ycsb-multi-storage-benchmark-config.toml) ## Run a benchmark Select a benchmark, and follow the instructions to run the benchmark. To run the TPC-C benchmark, run the following command, replacing `` with the path to the Kelpie directory: ```console //bin/kelpie --config tpcc-benchmark-config.toml ``` To run the YCSB benchmark, run the following command, replacing `` with the path to the Kelpie directory: ```console //bin/kelpie --config ycsb-benchmark-config.toml ``` To run the multi-storage YCSB benchmark, run the following command, replacing `` with the path to the Kelpie directory: ```console //bin/kelpie --config ycsb-multi-storage-benchmark-config.toml ``` In addition, the following options are available: - `--only-pre`. Only loads the data. - `--only-process`. Only runs the benchmark. - `--except-pre` Runs a job without loading the data. - `--except-process`. Runs a job without running the benchmark. ## Common parameters The following parameters are common to all workloads. ### `concurrency` - **Description:** Number of worker threads that concurrently execute benchmark transactions against the database. This parameter controls the level of parallelism during the actual benchmark execution phase. Increasing this value simulates more concurrent client accesses and higher workload intensity. - **Default value:** `1` ### `run_for_sec` - **Description:** Duration of the benchmark execution phase (in seconds). This parameter defines how long the benchmark will run and submit transactions to the database. - **Default value:** `60` ### `ramp_for_sec` - **Description:** Duration of the ramp-up period before the benchmark measurement phase begins (in seconds). During this warm-up period, the system executes transactions but does not record performance metrics. This allows the system to reach a steady state before collecting benchmark results. - **Default value:** `0` ## Workload-specific parameters Select a benchmark to see its available workload parameters.

`num_warehouses`

- **Description:** Number of warehouses to create for the TPC-C benchmark workload. This value is the scale factor that determines the dataset size. Increasing this value creates a larger working set and enables enterprise-scale testing. - **Default value:** `1`

`load_concurrency`

- **Description:** Number of parallel threads used to load initial benchmark data into the database. This parameter controls how fast the data-loading phase completes. Increasing this value can significantly reduce data-loading time, especially for larger numbers of warehouses. This is separate from the `concurrency` parameter used during benchmark execution. - **Default value:** `1`

`load_start_warehouse`

- **Description:** Start ID of the warehouse range to load. This option can be useful with `skip_item_load` when loading large-scale data with multiple clients or adding additional warehouses. - **Default value:** `1`

`load_end_warehouse`

- **Description:** End ID of the warehouse range to load. You can use either `num_warehouses` or `load_end_warehouse` to specify the number of loading warehouses. - **Default value:** `1`

`skip_item_load`

- **Description:** If set to `true`, skips loading the item table. This can be useful when loading data with multiple clients, as the item table is shared across all warehouses and only needs to be loaded once. - **Default value:** `false`

`use_table_index`

- **Description:** If set to `true`, uses a table-based secondary index instead of the secondary index of ScalarDB. The table-based secondary index builds its index using regular ScalarDB tables, allowing efficient lookups of specific customers or orders via partition keys. In contrast, the ScalarDB secondary index leverages the native secondary index feature of the underlying database, and thus its behavior depends on its implementation. Since secondary indexes in NoSQL databases such as Cassandra often require careful handling from a performance perspective, this option allows you to observe the performance characteristics of your target database when running workloads that involve secondary indexes. - **Default value:** `false`

`np_only`

- **Description:** If set to `true`, runs the benchmark with only New-Order and Payment transactions (50% each), excluding other TPC-C transaction types. This setting can be useful for focused performance testing under a write-heavy workload without having long-running reads. - **Default value:** `false`

`rate_new_order`

- **Description:** Percentage of New-Order transactions in the transaction mix. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100%. - **Default value:** N/A

`rate_payment`

- **Description:** Percentage of Payment transactions in the transaction mix. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100%. - **Default value:** N/A

`rate_order_status`

- **Description:** Percentage of Order-Status transactions in the transaction mix. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100%. - **Default value:** N/A

`rate_delivery`

- **Description:** Percentage of Delivery transactions in the transaction mix. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100%. - **Default value:** N/A

`rate_stock_level`

- **Description:** Percentage of Stock-Level transactions in the transaction mix. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100%. - **Default value:** N/A

`backoff`

- **Description:** Sleep time in milliseconds inserted after a transaction is aborted due to a conflict. This parameter can help reduce contention by introducing a delay before retrying failed transactions. - **Default value:** `0`

`load_concurrency`

- **Description:** Number of parallel threads used to load initial benchmark data into the database. This parameter controls how fast the initial data-loading phase completes. Increasing this value can significantly reduce data-loading time for large datasets. This is separate from the `concurrency` parameter used during benchmark execution. - **Default value:** `1`

`load_batch_size`

- **Description:** Number of records to insert within a single transaction during the initial data-loading phase. Larger batch sizes can improve loading performance by reducing the number of transactions, but may increase the execution time of each transaction. - **Default value:** `1`

`load_overwrite`

- **Description:** If set to `true`, overwrites existing records when loading data. When set to `true`, the initial data-loading phase will update existing records instead of failing on conflicts. - **Default value:** `false`

`ops_per_tx`

- **Description:** Number of read or write operations to execute within a single transaction. This parameter affects transaction size and execution time. Higher values create longer-running transactions. - **Default value:** `2` (Workloads A and C), `1` (Workload F)

`record_count`

- **Description:** Number of records to create for the YCSB benchmark workload. This parameter determines the size of the dataset and affects the working-set size during benchmark execution. - **Default value:** `1000`

`use_read_modify_write`

- **Description:** If set to `true`, uses read-modify-write operations instead of blind writes in Workload A. The default value is `true` because ScalarDB doesn't allow a blind write for an existing record when you're using the default transaction manager, Consensus Commit. Note that the original Workload A assumes `false`. - **Default value:** `true` ================================================ FILE: docs/scalardb-cluster/authorize-with-abac.mdx ================================================ --- tags: - Enterprise Premium Option - Private Preview displayed_sidebar: docsEnglish --- # Control User Access in a Fine-Grained Manner :::info - This feature is currently available only to customers in Japan. If you're a customer in Japan, please see the Japanese version of this page. - If you need more details about this feature in English, please [contact support](https://www.scalar-labs.com/support). ::: ScalarDB Cluster can authorize users in a fine-grained manner with a mechanism called attributed-based access control (ABAC). This page explains how to use ABAC in ScalarDB Cluster. ## What is ABAC? ABAC is a fine-grained access control mechanism in ScalarDB Cluster, allowing for record-level access control instead of just table-level access control, done through [simple authorization](./scalardb-auth-with-sql.mdx). With ABAC, a user can access a particular record only if the user's attributes and the record's attributes match. For example, you can restrict access to some highly confidential records to only users with the required privileges. This mechanism is also useful when multiple applications share the same table but need to access different segments based on their respective privileges. ## Why use ABAC? Enterprise databases often provide row-level security or similar alternatives to allow for controlling access to rows in a database table. However, if a system comprises several databases, you need to configure each database one by one in the same way. If different kinds of databases are used, you have to configure each database by understanding the differences in the capabilities of each database. Such configuration causes too much burden and is error-prone. With ABAC, you can just configure it once, even though you manage several databases under ScalarDB. Row-level security features in most databases often require you to implement matching logic through functions like stored procedures. This can sometimes lead to writing lots of code to achieve the desired logic, which can become burdensome. In contrast, ABAC allows you to configure matching logic by using attributes known as tags. With ABAC, you only need to define these tags and assign them to users and records, eliminating the need for coding. Tags consist of several components that enable you to specify matching logic in a flexible and straightforward manner. ================================================ FILE: docs/scalardb-cluster/compatibility.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster Compatibility Matrix This document shows the compatibility of ScalarDB Cluster versions among client SDK versions. ## ScalarDB Cluster compatibility with client SDKs | ScalarDB Cluster version | ScalarDB Cluster Java Client SDK version | ScalarDB Cluster .NET Client SDK version | |:-------------------------|:-----------------------------------------|:-----------------------------------------| | 3.18 | 3.14 - 3.18 | 3.14* - 3.18 | | 3.17 | 3.14 - 3.17 | 3.14* - 3.17 | | 3.16 | 3.14 - 3.16 | 3.14* - 3.16 | | 3.15 | 3.14 - 3.15 | 3.14* - 3.15 | | 3.14 | 3.14 | 3.14* | \* This version is in private preview, which means that future versions might have backward-incompatible updates. :::note - You can consider the client tools (for example, [ScalarDB Cluster SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli) and [ScalarDB Cluster Schema Loader](developer-guide-for-scalardb-cluster-with-java-api.mdx#schema-loader-for-cluster)) to be the same as the ScalarDB Cluster Java Client SDK. In other words, you can apply the same compatibility rules to client tools as the ScalarDB Cluster Java Client SDK. - When you access backend databases by using ScalarDB Cluster Data Loader, you must use a version of ScalarDB Cluster Data Loader that is compatible with the version of ScalarDB Cluster that you're using. In this case, the supported version of ScalarDB Cluster Data Loader is the same as the version of the ScalarDB Cluster Java Client SDK shown in the matrix above. Note that you can have ScalarDB Cluster Data Loader either access backend databases directly or connect to ScalarDB Cluster to access backend databases. - If you use a new feature that ScalarDB Cluster provides in a new minor version, you may need to use the same or a later version of the client tools or re-create (or update) existing schemas. For details, please refer to the relevant documentation about each feature. - For Scalar Admin and Scalar Admin for Kubernetes, using the latest versions of these tools is recommended to ensure compatibility with the version of ScalarDB Cluster that you are using. ::: ## Version skew policy :::note Versions are expressed as `x.y.z`, where `x` represents the major version, `y` represents the minor version, and `z` represents the patch version. This format follows [Semantic Versioning](https://semver.org/). ::: - If the **major** versions are different between ScalarDB Cluster and a client SDK, they are **not** compatible and are **not** supported. - If the **major** versions are the same and the **minor** versions are different between ScalarDB Cluster and a client SDK, the version of ScalarDB Cluster must be greater than or equal to the client SDK version. For example: - **Supported:** Combination of ScalarDB Cluster 3.17 and client SDK 3.14 - **Not supported:** Combination of ScalarDB Cluster 3.14 and client SDK 3.17 - If the **major** versions and the **minor** versions are the same, you can use different **patch** versions between ScalarDB Cluster and a client SDK. For example: - **Supported:** Combination of ScalarDB Cluster 3.17.2 and client SDK 3.17.0 - **Supported:** Combination of ScalarDB Cluster 3.17.0 and client SDK 3.17.2 ================================================ FILE: docs/scalardb-cluster/control-access-via-oidc-based-jwt-tokens.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Control User Access via OIDC-Based JWT Access Tokens import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; ScalarDB Cluster can control user access based on JWT access tokens issued by an OpenID Connect (OIDC) provider (for example, Keycloak), as an alternative to password-based authentication, allowing client applications to authenticate requests without directly managing ScalarDB passwords. ## How OIDC-based access control works The following sections describe the use case, authentication flow, and validation rules for OIDC-based access control. ### Use case OIDC-based access control is designed for scenarios where one or more OIDC users map to a single ScalarDB service user. The OIDC client application authenticates users through the OIDC Provider, obtains a JWT access token containing the ScalarDB username and access scope, and sends the token with each request to ScalarDB Cluster. ```mermaid flowchart LR OP["OIDC Provider
(e.g., Keycloak)"] CA["Client application"] SC["ScalarDB Cluster"] DB["Databases"] OP -- "issued JWTs" --> CA CA -- "JWT access token" --> SC SC -- "restricted operations
based on token claims" --> DB ``` The OIDC Provider issues JWT access tokens to the client application. The client sends the token with each request to ScalarDB Cluster, which validates the token, maps it to a ScalarDB user, and restricts access based on the token claims. ### Authentication flow When a client sends a request with a JWT access token, ScalarDB Cluster performs the following steps: 1. **Fetches the OIDC Provider configuration.** ScalarDB Cluster retrieves the OpenID Provider configuration from `{issuer_url}/.well-known/openid-configuration` and caches the result. 2. **Fetches the JWKS.** ScalarDB Cluster extracts the JSON Web Key Set (JWKS) URL from the provider configuration, fetches the keys, and caches them. 3. **Validates the JWT.** ScalarDB Cluster verifies the token signature and standard claims per [RFC 9068](https://datatracker.ietf.org/doc/html/rfc9068#name-validating-jwt-access-token). 4. **Maps the token to a ScalarDB user.** ScalarDB Cluster extracts the ScalarDB username from the configured claim and looks up the user record. 5. **Validates the authentication method.** ScalarDB Cluster confirms that the user is permitted to use OIDC authentication and that the issuer is trusted. OIDC authentication is enabled for a user when the user is created with the `AUTH_METHOD OIDC` option. 6. **Executes the request.** ScalarDB Cluster executes the request as the mapped ScalarDB user with the applicable restrictions. ### JWT access token validation ScalarDB Cluster validates JWT access tokens following [RFC 9068](https://datatracker.ietf.org/doc/html/rfc9068). Specifically, it performs the following checks: - **`typ` header:** By default, the `typ` header must be `at+jwt` or `application/at+jwt`. You can disable this check by setting `require_at_jwt_typ` to `false`. - **Signature:** The token signature is verified by using the keys from the OIDC Provider's JWKS endpoint. Only the following algorithms are accepted: RSASSA-PKCS-v1_5, RSASSA-PSS, and ECDSA. - **`iss` claim:** The issuer must match the value configured in `trusted_issuers`. - **`aud` claim:** The audience must contain the value configured in `audience.name`. - **`exp` claim:** The token must not be expired. A configurable clock skew tolerance is applied. ### User mapping ScalarDB Cluster identifies the ScalarDB user by extracting the value of the claim specified in `username.claim_name` from the validated JWT. ScalarDB uses this value to look up the corresponding user record in the `users` metadata table. :::warning Choose the username claim carefully. Unintentionally or incorrectly sharing a ScalarDB user across OIDC users may cause a security issue. ::: ## Configurations This section describes the configurations for OIDC-based access control. For general authentication and authorization configurations, see [Authenticate and Authorize Users](./scalardb-auth-with-sql.mdx). ### Server-side configurations The following are the minimum required server-side configurations for OIDC-based access control. You must also set `scalar.db.cluster.auth.enabled` to `true`. | Property | Description | Default value | |---------------------------------------------------------|------------------------------------------------------------------------------------------------------|---------------| | `scalar.db.cluster.auth.oidc.trusted_issuers` | The trusted OIDC issuer URL. Tokens whose `iss` claim does not match this value are rejected. This property must be specified when using OIDC-based access control. | empty | | `scalar.db.cluster.auth.oidc.username.claim_name` | The JWT claim name used to extract the ScalarDB username. This property must be specified when using OIDC-based access control. | empty | | `scalar.db.cluster.auth.oidc.audience.name` | The expected value in the JWT `aud` claim. | `scalardb` | For additional server-side configurations (cache TTL, clock skew, etc.), see [OIDC configurations](./scalardb-cluster-configurations.mdx#oidc-configurations) in the ScalarDB Cluster configurations. ### Client-side configurations The following are the client-side configurations for OIDC-based access control. When you use `oidc_jwt`, the `scalar.db.username` and `scalar.db.password` properties are not required. | Property | Description | Default value | |---------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------|---------------| | `scalar.db.cluster.client.auth.type` | The authentication type for the primitive interface. Set to `oidc_jwt` for OIDC. | empty (treated as `userpass`) | | `scalar.db.sql.cluster_mode.auth.type` | The authentication type for the SQL interface. Set to `oidc_jwt` for OIDC. | empty (treated as `userpass`) | :::note You can pass the JWT access token programmatically by using `OidcJwtAccessTokenHolder`, which is the recommended option when handling many OIDC users. Alternatively, you can supply the token via the access token property (`scalar.db.cluster.client.auth.oidc_jwt.access_token` for the primitive interface or `scalar.db.sql.cluster_mode.auth.oidc_jwt.access_token` for the SQL interface), which is easy to start with, especially for testing with tools like the SQL CLI. However, since the token is set at initialization time, it cannot be refreshed. ::: For additional client-side configurations, see the configurations for the [primitive interface](./scalardb-cluster-configurations.mdx#configurations-for-the-primitive-interface) and the [SQL interface](./scalardb-cluster-configurations.mdx#configurations-for-the-sql-interface) in the ScalarDB Cluster configurations. ## Troubleshooting Detailed error messages are shown in the ScalarDB Cluster node log. Check the log for the root cause when a request fails. | Error | Possible cause | Resolution | |-------|---------------|------------| | JWT validation error | The `trusted_issuers` configuration does not match the Keycloak issuer URL. | Check the Keycloak realm settings and verify the issuer URL by visiting `{keycloak_url}/realms/{realm}/.well-known/openid-configuration`. | | User not found | The claim specified in `username.claim_name` is missing from the JWT, or the ScalarDB user does not exist. | Decode the JWT to verify the claim, and run `SHOW USERS` in the SQL CLI to confirm the user exists. | | Audience mismatch | The `audience.name` configuration does not match the `aud` claim in the JWT. | Verify that the Keycloak audience mapper and the `audience.name` property use the same value. | | `typ` header error | The JWT `typ` header is not `at+jwt`. | Verify that the Keycloak client is configured to use `at+jwt` as the access token header type, or set `require_at_jwt_typ` to `false` for development purposes. | | Authentication method error | The ScalarDB user is not permitted to use OIDC, or the user is a superuser. | Verify that the user was created with `AUTH_METHOD OIDC` and that the user is not a superuser. | ## See also - [Authenticate and Authorize Users](./scalardb-auth-with-sql.mdx) - [ScalarDB Cluster Configurations](./scalardb-cluster-configurations.mdx) ================================================ FILE: docs/scalardb-cluster/deploy-scalardb-cluster-google-cloud-marketplace.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Deploy ScalarDB Cluster Through Google Cloud Marketplace This document explains how to deploy ScalarDB Cluster in your Google Cloud environment through Google Cloud Marketplace. ## Prerequisites - You must create a Google Cloud project to deploy ScalarDB Cluster. - You must prepare the backend databases that you want to use under ScalarDB Cluster. ## Deploy ScalarDB Cluster 1. Decide which edition of ScalarDB Cluster you want to use, based on the [features](../features.mdx) of each edition. 1. Go to the [ScalarDB page on Google Cloud Marketplace](https://console.cloud.google.com/marketplace/product/scalarlabs-public/scalardb). 1. Subscribe to **ScalarDB Enterprise Edition Standard** or **ScalarDB Enterprise Edition Premium**. ![Select edition in Google Cloud Marketplace](images/google-marketplace/select-edition-google-marketplace.png) 1. On the **New ScalarDB subscription** page, read the content under **Additional terms** and select the checkbox. Then, select the **Subscribe** button. ![Subscribe to ScalarDB](images/google-marketplace/subscribe-to-scalardb.png) 1. When the **Your order request has been sent to Scalar, Inc.** pop-up window is displayed, select the **Go to product page** button. ![Select go to product page](images/google-marketplace/select-go-to-product-page.png) The **Purchase pending provider approval** message will be displayed at the top of the product page. Please wait for the team at Scalar to approve your request. ![Pending provider approval](images/google-marketplace/pending-provider-approval.png) 1. After your request has been approved, the **Manage on provider** button will be displayed at the top of the product page. Select the **Manage on provider** button. ![Manage on provider](images/google-marketplace/manage-on-provider.png) 1. When the pop-up window asking you to confirm leaving Google Cloud Marketplace is displayed, select the **OK** button. ![Leaving Google](images/google-marketplace/leaving-google.png) 1. When the Scalar portal login page is displayed, enter your email address and select the **Next** button. ![Scalar portal login](images/google-marketplace/scalar-portal-login.png) 1. Select the **Sign up with a password** button. ![Select sign up with a password](images/google-marketplace/sign-up-with-password.png) 1. On the **Get Started Today** page, enter your account information. ![Enter account information](images/google-marketplace/enter-account-information.png) 1. When the **Verify Your Email to Activate Your Account** page is displayed, check your email and select the link in the message that you received to activate your account. ![Verify email to activate](images/google-marketplace/verify-email-to-activate.png) 1. After you verify your account, you will see the login page again. Please log in to the Scalar portal. ![Scalar portal login after verification](images/google-marketplace/scalar-portal-login-after-verification.png) After you log in, the **Deployment Instances** page will be displayed. ![Top page of Scalar portal](images/google-marketplace/top-page-of-scalar-portal.png) 1. In the sidebar navigation, select **Cloud Accounts**. Then, select the **Create** button on the **Cloud Accounts** screen. ![Cloud account create](images/google-marketplace/cloud-account-create.png) 1. On the **Cloud Accounts** page, select the **Subscribe** button. :::important Please make sure that you select the same edition that you selected on the Google Cloud Marketplace page. ::: ![Select edition in the Scalar portal](images/google-marketplace/select-edition-scalar-portal.png) 1. After you select the **Subscribe** button, the button will show the message **Pending Approval**. Please wait for the team at Scalar to approve your request. ![Pending approval in the Scalar portal](images/google-marketplace/pending-approval-scalar-portal.png) After your request has been approved, you will get an email with the title **Your ScalarDB Enterprise Edition - [EDITION_NAME] subscription request approved** from the following email address: marketplace-notifications@scalar-labs.com. Also, after your request has been approved, the **Subscribe** button on the **Cloud Accounts** page of the Scalar portal will be grayed out. ![Grayed out subscribe button](images/google-marketplace/grayed-out-subscribe-button.png) 1. On the **Cloud Accounts** page, enter the **Project ID** and **Project number** of your Google Cloud project that you want to use for the ScalarDB Cluster deployment. You can use the default values for the other configurations like **Subscription**, **Cloud Provider**, and **Account Configuration Method**. After you enter the **Project ID** and **Project number**, select the **Create** button. ![Enter project ID and project number](images/google-marketplace/enter-project-id-and-project-number.png) 1. When the **Account Configuration Instructions** pop-up window is displayed, copy the bash command and select the link **Google Cloud Shell** to open the Google Cloud Shell. ![Account configuration instructions](images/google-marketplace/account-configuration-instructions.png) 1. In the Google Cloud Shell, paste the bash command that you copied in the **Account Configuration Instructions** pop-up window and execute the command. You will be asked **Do you want to proceed?**, like in the example output below. Enter **yes**. ```console ============================================= Google Cloud Setup Script ============================================= This script will configure your GCP project. ✔ Enable required GCP APIs ✔ Create & configure service accounts ✔ Assign IAM roles ✔ Setup Workload Identity Pool and OIDC Provider ----------------------------------------------- Do you want to proceed? (yes/no): ``` If the bash command is successful, a message like the following will be displayed. ```console YYYY-MM-DD HH:MM:SS - [INFO] - Script completed successfully. ``` 1. On the **Cloud Accounts** page in the Scalar portal, the status will be **Ready**. ![Cloud account lifecycle status ready](images/google-marketplace/cloud-account-lifecycle-status-ready.png) 1. In the sidebar navigation, select **Instances**. Then, select the **Create** button on the **Deployment Instances** screen. ![Deployment instance create](images/google-marketplace/deployment-instance-create.png) 1. On the **Create Deployment Instance** page, enter the configurations as follows: - **Subscription Plan:** Select the edition that you subscribed to. - **Subscription:** Use the default value. (You don't need to update this value.) - **Cloud Provider:** Use the default value. (You don't need to update this value.) - **Region:** Select the region that you want to deploy ScalarDB Cluster in. - **Tags:** Set any arbitrary tags. - **Network Type:** Select **Public**. - **Cloud Provider Account Config ID:** Select your cloud account that you previously registered on the **Cloud Accounts** page. - **ScalarDB Cluster Node Replica Count:** Enter the number of ScalarDB Cluster node replicas. - **Database Properties:** Enter the ScalarDB Cluster properties. For details, please see [ScalarDB Cluster Configurations](./scalardb-cluster-configurations.mdx). - **Database Properties (Sensitive Information):** Enter the ScalarDB Cluster properties, like `scalar.db.username` and `scalar.db.password`. These values are invisible in the Scalar portal after you deploy ScalarDB Cluster. :::note In this text box, you can enter (copy and paste) the properties in the same format as normal database properties. For example: ```properties scalar.db.username= scalar.db.password= ``` These values are automatically masked when you enter them into this text box. ::: Example: ![Create deployment instance](images/google-marketplace/create-deployment-instance.png) 1. After you enter the configurations for your deployment, select the **Create** button. The **Launching Your Instance** pop-up window will be displayed. ![Launching your instance](images/google-marketplace/launching-your-instance.png) 1. After your instance has been launched, your deployed instance will be displayed with **Lifecycle Status** as **Running**. ![Deployed your instance](images/google-marketplace/deployed-your-instance.png) :::note Deploying a new instance will take time. In particular, if this is the first time that your ScalarDB Cluster instance is deployed to the region that you selected, a GKE cluster will be created first, which will take a significant amount of time. ::: 1. To see the endpoint information to access the ScalarDB Cluster endpoint and the Grafana dashboard, select an instance under **Instance ID** and go to the **Connectivity** tab. ![Connectivity](images/google-marketplace/connectivity.png) ================================================ FILE: docs/scalardb-cluster/deployment-patterns-for-microservices.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster Deployment Patterns for Microservices When building microservice applications that use ScalarDB Cluster, there are two patterns you can choose for how to deploy ScalarDB Cluster: shared-cluster pattern and separated-cluster pattern. This document first explains those patterns, how they differ, and the basic guidelines on which one to choose in which cases. Also, this document assumes that your microservice applications are created based on the database-per-service pattern, where each microservice manages its database, and a microservice needs to access another microservice's database via APIs between the microservices. ## ScalarDB Cluster deployment patterns In the shared-cluster pattern, microservices share one ScalarDB Cluster instance, which is a cluster of ScalarDB Cluster nodes, in a system, so they access the same ScalarDB Cluster instance to interact with their databases. On the other hand, in the separated-cluster pattern, microservices use several ScalarDB Cluster instances. Typically, one microservice accesses one ScalarDB Cluster instance to interact with its database. The following diagram shows the patterns. (MS stands for microservice.) ![ScalarDB Cluster deployment patterns for microservices.](images/scalardb-deployment-patterns.png) :::note You also need to manage the Coordinator table in either pattern in addition to the databases required for microservices. ::: ## Pros and cons One obvious difference is the amount of resources for ScalarDB Cluster instances. With the separated-cluster pattern, you need more resources to manage your applications. This also incurs more maintenance burden and costs. In addition, the ScalarDB Cluster APIs that you would need to use are different. Specifically, for the shared-cluster pattern, you need to use the [one-phase commit interface](../api-guide.mdx#transactional-api), where only one microservice needs to call `commit` to commit a transaction after microservices read and write records. For the separated-cluster pattern, you need to use the [two-phase commit interface](../two-phase-commit-transactions.mdx), where all the microservices first need to call `prepare` and then call `commit` if all the prepare calls are successful. Therefore, microservices with the separated-cluster pattern will likely be more complex than microservices with the shared-cluster pattern because they need to handle transactions and their errors in a more fine-grained manner. Moreover, the level of resource isolation is different. Microservices should be well-isolated for better maintainability and development efficiency, but the shared-cluster pattern brings weaker resource isolation. Weak resource isolation might also bring weak security. However, security risks can be mitigated by using the security features of ScalarDB Cluster, like authentication and authorization. Similarly, there is a difference in how systems are administrated. Specifically, in the shared-cluster pattern, a team must be tasked with managing a ScalarDB Cluster instance on behalf of the other teams. Typically, the central data team can manage it, but issues may arise if no such team exists. With the separated-cluster pattern, administration is more balanced but has a similar issue for the Coordinator table. The issue can be addressed by having a microservice for coordination and making a team manage the microservice. The following is a summary of the pros and cons of the patterns. ### Shared-cluster pattern - **Pros:** - Simple transaction and error handling because of the one-phase commit interface. (Backup operations for databases can also be simple.) - Less resource usage because it uses one ScalarDB Cluster instance. - **Cons:** - Weak resource isolation between microservices. - Unbalanced administration. (One team needs to manage a ScalarDB Cluster instance on behalf of the others.) ### Separated-cluster pattern - **Pros:** - Better resource isolation. - More balanced administration. (A team manages one microservice and one ScalarDB Cluster instance. Also, a team must be tasked with managing the Coordinator table.) - **Cons:** - Complex transaction and error handling due to the two-phase commit interface. (Backup operations for databases can also be complex.) - More resource usage because of several ScalarDB Cluster instances. ## Which pattern to choose Using the shared-cluster pattern is recommended whenever possible. Although the shared-cluster pattern has some disadvantages, as described above, its simplicity and ease of management outweigh those disadvantages. Moreover, since ScalarDB Cluster stores all critical states in their underlying databases and does not hold any critical states in its memory, it can be seen as just a path to the databases. Therefore, we believe a system with the shared-cluster pattern still complies with the database-per-service pattern and does not violate the microservice philosophy much. If the cons of the shared-cluster pattern are not acceptable, you can still use the separated-cluster pattern. However, you should use that pattern only if you properly understand the mechanism and usage of the two-phase commit interface. Otherwise, you might face some issues, like database anomalies. ## Limitations ScalarDB provides several APIs, such as CRUD, SQL, and Spring Data JDBC. Although the CRUD and SQL interfaces support both the shared-cluster and separated-cluster patterns, the Spring Data JDBC interface does not support the shared-cluster pattern. This is because its one-phase commit interface currently assumes an application is monolithic, where it is not divided into microservices that interact with each other. The Spring Data JDBC interface supports the two-phase commit interface and the separated-cluster pattern, just as the other APIs do. ## See also - [Transactions with a Two-Phase Commit Interface](../two-phase-commit-transactions.mdx) ================================================ FILE: docs/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Developer Guide for ScalarDB Cluster with the Java API ScalarDB Cluster provides a Java API for developing applications. This document explains how to use the Java API. ## Add ScalarDB Cluster Java Client SDK to your build The ScalarDB Cluster Java Client SDK is available in the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb-cluster-java-client-sdk). To add a dependency on the ScalarDB Cluster Java Client SDK by using Gradle, use the following: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.18.0' } ``` To add a dependency by using Maven, use the following: ```xml com.scalar-labs scalardb-cluster-java-client-sdk 3.18.0 ``` ## Client modes The ScalarDB Cluster Java Client SDK supports two client modes: `indirect` and `direct-kubernetes`. The following describes the client modes. ### `indirect` client mode This mode simply sends a request to any cluster node (typically via a load balancer, such as Envoy), and the cluster node receiving the request routes the request to the appropriate cluster node that has the transaction state. ![ScalarDB Cluster architecture](images/indirect-client-mode.png) The advantage of this mode is that we can keep the client thin. The disadvantage is that we need an additional hop to reach the correct cluster node, which may affect performance. You can use this connection mode even if your application is running on a different Kubernetes cluster and your application can't access the Kubernetes API and each cluster node. If your application is running on the same Kubernetes cluster as your ScalarDB Cluster nodes, you can use the `direct-kubernetes` client mode. ### `direct-kubernetes` client mode In this mode, the client uses the membership logic (using the Kubernetes API) and the distribution logic (consistent hashing algorithm) to find the right cluster node that has the transaction state. The client then sends a request to the cluster node directly. ![ScalarDB Cluster architecture](images/direct-kubernetes-client-mode.png) The advantage of this mode is that we can reduce the hop count to reach the proper cluster node, which will improve the performance. The disadvantage of this mode is that we need to make the client fat because the client needs to have membership logic and request-routing logic. Since this connection mode needs to access the Kubernetes API and each cluster node, you can use this connection mode only if your application is running on the same Kubernetes cluster as your ScalarDB Cluster nodes. If your application is running on a different Kubernetes cluster, use the `indirect` client mode. For details about how to deploy your application on Kubernetes with `direct-kubernetes` client mode, see [Deploy your client application on Kubernetes with `direct-kubernetes` mode](../helm-charts/how-to-deploy-scalardb-cluster.mdx#deploy-your-client-application-on-kubernetes-with-direct-kubernetes-mode). ## ScalarDB Cluster Java API The ScalarDB Cluster Java Client SDK provides a Java API for applications to access ScalarDB Cluster. The following diagram shows the architecture of the ScalarDB Cluster Java API. ``` +------------------+ | User/Application | +------------------+ ↓ Java API +--------------+ | ScalarDB API | +--------------+ ↓ gRPC +------------------+ | ScalarDB Cluster | +------------------+ ↓ DB vendor–specific protocol +----+ | DB | +----+ ``` Using the ScalarDB Cluster Java API is almost the same as using the ScalarDB Java API except the client configurations and Schema Loader are different. For details, see [ScalarDB Java API Guide](../api-guide.mdx). The following section describes the Schema Loader for ScalarDB Cluster. ### Schema Loader for Cluster To load a schema via ScalarDB Cluster, you need to use the dedicated Schema Loader for ScalarDB Cluster (Schema Loader for Cluster). Using the Schema Loader for Cluster is basically the same as using the [ScalarDB Schema Loader](../schema-loader.mdx) except the name of the JAR file is different. You can download the Schema Loader for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.18.0). After downloading the JAR file, you can run Schema Loader for Cluster with the following command: ```console java -jar scalardb-cluster-schema-loader-3.18.0-all.jar --config --schema-file --coordinator ``` You can also pull the Docker image from the [Scalar container registry](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-schema-loader) by running the following command, replacing the contents in the angle brackets as described: ```console docker run --rm -v :/scalardb.properties -v :/schema.json ghcr.io/scalar-labs/scalardb-cluster-schema-loader:3.18.0 --config /scalardb.properties --schema-file /schema.json --coordinator ``` ## ScalarDB Cluster SQL ScalarDB Cluster SQL can be accessed via JDBC and Spring Data JDBC for ScalarDB in Java as follows: ``` +-----------------------------------------+ | User/Application | +-----------------------------------------+ ↓ ↓ Java API Java API ↓ +-------------------------------+ (JDBC) ↓ | Spring Data JDBC for ScalarDB | ↓ +-------------------------------+ +----------------------------------------------+ | ScalarDB JDBC (ScalarDB SQL) | +----------------------------------------------+ ↓ gRPC +----------------------+ | ScalarDB Cluster SQL | +----------------------+ ↓ DB vendor–specific protocol +----+ | DB | +----+ ``` This section describes how to use ScalarDB Cluster SQL though JDBC and Spring Data JDBC for ScalarDB. ### ScalarDB Cluster SQL via JDBC Using ScalarDB Cluster SQL via JDBC is almost the same using [ScalarDB JDBC](../scalardb-sql/jdbc-guide.mdx) except for how to add the JDBC driver to your project. In addition to adding the ScalarDB Cluster Java Client SDK as described in [Add ScalarDB Cluster Java Client SDK to your build](#add-scalardb-cluster-java-client-sdk-to-your-build), you need to add the following dependencies to your project: To add the dependencies on the ScalarDB Cluster JDBC driver by using Gradle, use the following: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-sql-jdbc:3.18.0' implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.18.0' } ``` To add the dependencies by using Maven, use the following: ```xml com.scalar-labs scalardb-sql-jdbc 3.18.0 com.scalar-labs scalardb-cluster-java-client-sdk 3.18.0 ``` Other than that, using ScalarDB Cluster SQL via JDBC is the same as using ScalarDB JDBC. For details about ScalarDB JDBC, see [ScalarDB JDBC Guide](../scalardb-sql/jdbc-guide.mdx). ### ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB Similar to ScalarDB Cluster SQL via JDBC, using ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB is almost the same as using [Spring Data JDBC for ScalarDB](../scalardb-sql/spring-data-guide.mdx) except for how to add it to your project. In addition to adding the ScalarDB Cluster Java Client SDK as described in [Add ScalarDB Cluster Java Client SDK to your build](#add-scalardb-cluster-java-client-sdk-to-your-build), you need to add the following dependencies to your project: To add the dependencies by using Gradle, use the following: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-sql-spring-data:3.18.0' implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.18.0' } ``` To add the dependencies by using Maven, use the following: ```xml com.scalar-labs scalardb-sql-spring-data 3.18.0 com.scalar-labs scalardb-cluster-java-client-sdk 3.18.0 ``` Other than that, using ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB is the same as using Spring Data JDBC for ScalarDB. For details about Spring Data JDBC for ScalarDB, see [Guide of Spring Data JDBC for ScalarDB](../scalardb-sql/spring-data-guide.mdx). ### SQL CLI Like other SQL databases, ScalarDB SQL also provides a CLI tool where you can issue SQL statements interactively in a command-line shell. You can download the SQL CLI for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.18.0). After downloading the JAR file, you can run the SQL CLI with the following command: ```console java -jar scalardb-cluster-sql-cli-3.18.0-all.jar --config ``` You can also pull the Docker image from the [Scalar container registry](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-sql-cli) by running the following command, replacing the contents in the angle brackets as described: ```console docker run --rm -it -v :/scalardb-sql.properties ghcr.io/scalar-labs/scalardb-cluster-sql-cli:3.18.0 --config /scalardb-sql.properties ``` #### Usage You can see the CLI usage with the `-h` option as follows: ```console java -jar scalardb-cluster-sql-cli-3.18.0-all.jar -h Usage: scalardb-sql-cli [-hs] -c=PROPERTIES_FILE [-e=COMMAND] [-f=FILE] [-l=LOG_FILE] [-o=] [-p=PASSWORD] [-u=USERNAME] Starts ScalarDB SQL CLI. -c, --config=PROPERTIES_FILE A configuration file in properties format. -e, --execute=COMMAND A command to execute. -f, --file=FILE A script file to execute. -h, --help Display this help message. -l, --log=LOG_FILE A file to write output. -o, --output-format= Format mode for result display. You can specify table/vertical/csv/tsv/xmlattrs/xmlelements/json/a nsiconsole. -p, --password=PASSWORD A password to connect. -s, --silent Reduce the amount of informational messages displayed. -u, --username=USERNAME A username to connect. ``` ## Further reading If you want to use ScalarDB Cluster in programming languages other than Java, you can use the ScalarDB Cluster gRPC API. For details about the ScalarDB Cluster gRPC API, refer to the following: * [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) * [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) JavaDocs are also available: * [ScalarDB Cluster Java Client SDK](https://javadoc.io/doc/com.scalar-labs/scalardb-cluster-java-client-sdk/3.18.0/index.html) * [ScalarDB Cluster Common](https://javadoc.io/doc/com.scalar-labs/scalardb-cluster-common/3.18.0/index.html) * [ScalarDB Cluster RPC](https://javadoc.io/doc/com.scalar-labs/scalardb-cluster-rpc/3.18.0/index.html) ================================================ FILE: docs/scalardb-cluster/encrypt-data-at-rest.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Encrypt Data at Rest import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This document explains how to encrypt data at rest in ScalarDB. ## Overview ScalarDB can encrypt data stored through it. The encryption feature is similar to transparent data encryption (TDE) in major database systems; therefore, it is transparent to applications. ScalarDB encrypts data before writing it to the backend databases and decrypts it when reading from them. Currently, ScalarDB supports column-level encryption, allowing specific columns in a table to be encrypted. ## Configurations To enable the encryption feature, you need to configure `scalar.db.cluster.encryption.enabled` to `true` in the ScalarDB Cluster node configuration file. | Name | Description | Default | |----------------------------------------|-----------------------------------------|---------| | `scalar.db.cluster.encryption.enabled` | Whether ScalarDB encrypts data at rest. | `false` | :::note Since encryption is transparent to the client, you don't need to change the client configuration. ::: The other configurations depend on the encryption implementation you choose. Currently, ScalarDB supports the following encryption implementations: - HashiCorp Vault encryption - Self-encryption The following sections explain how to configure each encryption implementation. ### HashiCorp Vault encryption In HashiCorp Vault encryption, ScalarDB uses the [encryption as a service](https://developer.hashicorp.com/vault/tutorials/encryption-as-a-service/eaas-transit) of HashiCorp Vault to encrypt and decrypt data. In this implementation, ScalarDB delegates the management of encryption keys, as well as the encryption and decryption of data, to HashiCorp Vault. To use HashiCorp Vault encryption, you need to set the property `scalar.db.cluster.encryption.type` to `vault` in the ScalarDB Cluster node configuration file: | Name | Description | Default | |-------------------------------------|-------------------------------------------------------------|---------| | `scalar.db.cluster.encryption.type` | Should be set to `vault` to use HashiCorp Vault encryption. | | You also need to configure the following properties: | Name | Description | Default | |------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------| | `scalar.db.cluster.encryption.vault.key_type` | The key type. Currently, `aes128-gcm96`, `aes256-gcm96`, and `chacha20-poly1305` are supported. For details about the key types, see [Key types](https://developer.hashicorp.com/vault/docs/secrets/transit#key-types). | `aes128-gcm96` | | `scalar.db.cluster.encryption.vault.associated_data_required` | Whether associated data is required for AEAD encryption. | `false` | | `scalar.db.cluster.encryption.vault.address` | The address of the HashiCorp Vault server. | | | `scalar.db.cluster.encryption.vault.token` | The token to authenticate with HashiCorp Vault. | | | `scalar.db.cluster.encryption.vault.namespace` | The namespace of the HashiCorp Vault. This configuration is optional. | | | `scalar.db.cluster.encryption.vault.transit_secrets_engine_path` | The path of the transit secrets engine. | `transit` | | `scalar.db.cluster.encryption.vault.column_batch_size` | The number of columns to be included in a single request to the HashiCorp Vault server. | `64` | :::note Currently, ScalarDB supports only the token auth method. There are plans to implement additional, more secure auth methods, such as AppRole, OIDC, and Kubernetes auth, in the future. ::: ### Self-encryption In self-encryption, ScalarDB manages data encryption keys (DEKs) and performs encryption and decryption. ScalarDB generates a DEK for each table when creating the table and stores it in Kubernetes Secrets. To use self-encryption, you need to set the property `scalar.db.cluster.encryption.type` to `self` in the ScalarDB Cluster node configuration file: | Name | Description | Default | |-------------------------------------|-------------------------------------------------|---------| | `scalar.db.cluster.encryption.type` | Should be set to `self` to use self-encryption. | | You also need to configure the following properties: | Name | Description | Default | |-------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------| | `scalar.db.cluster.encryption.self.key_type` | The key type. Currently, `AES128_GCM`, `AES256_GCM`, `AES128_EAX`, `AES256_EAX`, `AES128_CTR_HMAC_SHA256`, `AES256_CTR_HMAC_SHA256`, `CHACHA20_POLY1305`, and `XCHACHA20_POLY1305` are supported. For details about the key types, see [Choose a key type](https://developers.google.com/tink/aead#choose_a_key_type). | `AES128_GCM` | | `scalar.db.cluster.encryption.self.associated_data_required` | Whether associated data is required for AEAD encryption. | `false` | | `scalar.db.cluster.encryption.self.kubernetes.secret.namespace_name` | The namespace name of the Kubernetes Secrets. | `default` | | `scalar.db.cluster.encryption.self.data_encryption_key_cache_expiration_time` | The expiration time of the DEK cache in milliseconds. | `60000` (60 seconds) | ### Delete the DEK when dropping a table By default, ScalarDB does not delete the data encryption key (DEK) associated with a table when the table is dropped. However, you can configure ScalarDB to delete the DEK when dropping a table. To enable this, set the property `scalar.db.cluster.encryption.delete_data_encryption_key_on_drop_table.enabled` to `true` in the ScalarDB Cluster node configuration file: | Name | Description | Default | |---------------------------------------------------------------------------------|------------------------------------------------------------------|---------| | `scalar.db.cluster.encryption.delete_data_encryption_key_on_drop_table.enabled` | Whether to delete the DEK when dropping a table. | `false` | ## Limitations There are some limitations to the encryption feature: - Primary-key columns (partition-key columns and clustering-key columns) cannot be encrypted. - Secondary-index columns cannot be encrypted. - Encrypted columns cannot be specified in the WHERE clauses or ORDER BY clauses. - Encrypted columns are stored in the underlying database as the BLOB type, so encrypted columns that are larger than the maximum size of the BLOB type cannot be stored. To see if the database you're using has a maximum size for the BLOB type, see [Database Adapters](../database-adapters.mdx). - Encrypted columns cannot be renamed. - Encrypted columns cannot be altered to change their data types. ## Wire encryption If you enable the encryption feature, enabling wire encryption to protect your data is strongly recommended, especially in production environments. For details about wire encryption, see [Encrypt Wire Communications](encrypt-wire-communications.mdx). ## Tutorial - Encrypt data by configuring HashiCorp Vault encryption This tutorial explains how to encrypt data stored through ScalarDB by using HashiCorp Vault encryption. ### Prerequisites - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ### Step 1. Install HashiCorp Vault Install HashiCorp Vault by referring to the official HashiCorp documentation, [Install Vault](https://developer.hashicorp.com/vault/tutorials/getting-started/getting-started-install). ### Step 2. Create the ScalarDB Cluster configuration file Create the following configuration file as `scalardb-cluster-node.properties`, replacing `` and `` with your ScalarDB license key and license check certificate values. For more information about the license key and certificate, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). ```properties scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://postgresql:5432/postgres scalar.db.username=postgres scalar.db.password=postgres scalar.db.cluster.node.standalone_mode.enabled=true scalar.db.sql.enabled=true # Enable cross-partition scan to perform a full scan by using the SELECT statements in this tutorial. # This is not required for encryption itself. scalar.db.cross_partition_scan.enabled=true # Encryption configurations scalar.db.cluster.encryption.enabled=true scalar.db.cluster.encryption.type=vault scalar.db.cluster.encryption.vault.address=http://vault:8200 scalar.db.cluster.encryption.vault.token=root # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` ### Step 3. Create the Docker Compose configuration file Create the following configuration file as `docker-compose.yaml`. ```yaml services: vault: container_name: "vault" image: "hashicorp/vault:1.17.3" ports: - 8200:8200 environment: - VAULT_DEV_ROOT_TOKEN_ID=root - VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200 cap_add: - IPC_LOCK postgresql: container_name: "postgresql" image: "postgres:15" ports: - 5432:5432 environment: - POSTGRES_PASSWORD=postgres healthcheck: test: ["CMD-SHELL", "pg_isready || exit 1"] interval: 1s timeout: 10s retries: 60 start_period: 30s scalardb-cluster-standalone: container_name: "scalardb-cluster-node" image: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium:3.18.0" ports: - 60053:60053 - 9080:9080 volumes: - ./scalardb-cluster-node.properties:/scalardb-cluster/node/scalardb-cluster-node.properties depends_on: postgresql: condition: service_healthy ``` ### Step 4. Start the HashiCorp Vault server Run the following command to start the HashiCorp Vault server in development mode. ```console docker compose up vault -d ``` Once the HashiCorp Vault server is running, set its environment variables by running the following commands. ```console export VAULT_ADDR="http://127.0.0.1:8200" export VAULT_TOKEN=root ``` ### Step 5. Enable the transit secrets engine on the HashiCorp Vault server Run the following command to enable the transit secrets engine on the HashiCorp Vault server. ```console vault secrets enable transit ``` ### Step 6. Start PostgreSQL and ScalarDB Cluster Run the following command to start PostgreSQL and ScalarDB Cluster in standalone mode. ```console docker compose up postgresql scalardb-cluster-standalone -d ``` It may take a few minutes for ScalarDB Cluster to fully start. ### Step 7. Connect to ScalarDB Cluster To connect to ScalarDB Cluster, this tutorial uses the SQL CLI, a tool for connecting to ScalarDB Cluster and executing SQL queries. You can download the SQL CLI from the [ScalarDB releases page](https://github.com/scalar-labs/scalardb/releases). Create a configuration file named `scalardb-cluster-sql-cli.properties`. This file will be used to connect to ScalarDB Cluster by using the SQL CLI. ```properties scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points=indirect:localhost ``` Then, start the SQL CLI by running the following command. ```console java -jar scalardb-cluster-sql-cli-3.18.0-all.jar --config scalardb-cluster-sql-cli.properties ``` To begin, create the Coordinator tables required for ScalarDB transaction execution. ```sql CREATE COORDINATOR TABLES IF NOT EXISTS; ``` Now you're ready to use the database with the encryption feature enabled in ScalarDB Cluster. ### Step 8. Create a table Before creating a table, you need to create a namespace. ```sql CREATE NAMESPACE ns; ``` Next, create a table. ```sql CREATE TABLE ns.tbl ( id INT PRIMARY KEY, col1 TEXT ENCRYPTED, col2 INT ENCRYPTED, col3 INT); ``` By using the `ENCRYPTED` keyword, the data in the specified columns will be encrypted. In this example, the data in `col1` and `col2` will be encrypted. ### Step 9. Insert data into the table To insert data into the table, execute the following SQL query. ```sql INSERT INTO ns.tbl (id, col1, col2, col3) VALUES (1, 'data1', 123, 456); ``` To verify the inserted data, run the following SQL query. ```sql SELECT * FROM ns.tbl; ``` ```console +----+-------+------+------+ | id | col1 | col2 | col3 | +----+-------+------+------+ | 1 | data1 | 123 | 456 | +----+-------+------+------+ ``` ### Step 10. Verify data encryption To verify that the data is encrypted, connect directly to PostgreSQL and check the data. :::warning Reading or writing data from the backend database directly is not supported in ScalarDB. In such a case, ScalarDB cannot guarantee data consistency. This guide accesses the backend database directly for testing purposes, however, you cannot do this in a production environment. ::: Run the following command to connect to PostgreSQL. ```console docker exec -it postgresql psql -U postgres ``` Next, execute the following SQL query to check the data in the table. ```sql SELECT id, col1, col2, col3 FROM ns.tbl; ``` You should see a similar output as below, which confirms that the data in `col1` and `col2` are encrypted. ```console id | col1 | col2 | col3 ----+--------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+------ 1 | \x7661756c743a76313a6b6f76455062316a676e6a4a596b643743765539315a49714d625564545a61697152666c7967367837336e66 | \x7661756c743a76313a4b6244543162764678676d44424b526d7037794f5176423569616e615635304c473079664354514b3866513d | 456 ``` ================================================ FILE: docs/scalardb-cluster/encrypt-wire-communications.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Encrypt Wire Communications ScalarDB can encrypt wire communications by using Transport Layer Security (TLS). This document explains the configurations for wire encryption in ScalarDB. The wire encryption feature encrypts: * The communications between the ScalarDB Cluster node and clients. * The communications between all the ScalarDB Cluster nodes (the cluster's internal communications). This feature uses TLS support in gRPC. For details, see the official gRPC [Security Policy](https://github.com/grpc/grpc-java/blob/master/SECURITY.md). :::note Enabling wire encryption between the ScalarDB Cluster nodes and the underlying databases in production environments is strongly recommended. For instructions on how to enable wire encryption between the ScalarDB Cluster nodes and the underlying databases, please refer to the product documentation for your underlying databases. ::: ## Configurations This section describes the available configurations for wire encryption. ### ScalarDB Cluster node configurations To enable wire encryption in the ScalarDB Cluster nodes, you need to set `scalar.db.cluster.tls.enabled` to `true`. | Name | Description | Default | |---------------------------------|-------------------------------------------|---------| | `scalar.db.cluster.tls.enabled` | Whether wire encryption (TLS) is enabled. | `false` | You also need to set the following configurations: | Name | Description | Default | |-----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------| | `scalar.db.cluster.tls.ca_root_cert_pem` | The custom CA root certificate (PEM data) for TLS communication. | | | `scalar.db.cluster.tls.ca_root_cert_path` | The custom CA root certificate (file path) for TLS communication. | | | `scalar.db.cluster.tls.override_authority` | The custom authority for TLS communication. This doesn't change what host is actually connected. This is intended for testing, but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set for `scalar.db.cluster.node.tls.cert_chain_path`. | | | `scalar.db.cluster.node.tls.cert_chain_path` | The certificate chain file used for TLS communication. | | | `scalar.db.cluster.node.tls.private_key_path` | The private key file used for TLS communication. | | To specify the certificate authority (CA) root certificate, you should set either `scalar.db.cluster.tls.ca_root_cert_pem` or `scalar.db.cluster.tls.ca_root_cert_path`. If you set both, `scalar.db.cluster.tls.ca_root_cert_pem` will be used. ### Client configurations To enable wire encryption on the client side by using the ScalarDB Cluster Java client SDK, you need to set `scalar.db.cluster.tls.enabled` to `true`. | Name | Description | Default | |---------------------------------|-------------------------------------------|---------| | `scalar.db.cluster.tls.enabled` | Whether wire encryption (TLS) is enabled. | `false` | You also need to set the following configurations: | Name | Description | Default | |--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------| | `scalar.db.cluster.tls.ca_root_cert_pem` | The custom CA root certificate (PEM data) for TLS communication. | | | `scalar.db.cluster.tls.ca_root_cert_path` | The custom CA root certificate (file path) for TLS communication. | | | `scalar.db.cluster.tls.override_authority` | The custom authority for TLS communication. This doesn't change what host is actually connected. This is intended for testing, but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set for `scalar.db.cluster.node.tls.cert_chain_path`. | | To specify the CA root certificate, you should set either `scalar.db.cluster.tls.ca_root_cert_pem` or `scalar.db.cluster.tls.ca_root_cert_path`. If you set both, `scalar.db.cluster.tls.ca_root_cert_pem` will be used. ================================================ FILE: docs/scalardb-cluster/getting-started-with-scalardb-cluster-dotnet.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB Cluster via .NET This tutorial describes how to create a sample application that uses [ScalarDB Cluster](./index.mdx) through the .NET API. ## Overview This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB. :::note Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling, see [Exception Handling in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/exception-handling.mdx). ::: The following diagram shows the system architecture of the sample application: ```mermaid stateDiagram-v2 state "Sample application using the .NET API" as SA state "Kubernetes Cluster" as KC state "Service (Envoy)" as SE state "Pod" as P1 state "Pod" as P2 state "Pod" as P3 state "Envoy" as E1 state "Envoy" as E2 state "Envoy" as E3 state "Service (ScalarDB Cluster)" as SSC state "ScalarDB Cluster" as SC1 state "ScalarDB Cluster" as SC2 state "ScalarDB Cluster" as SC3 state "PostgreSQL" as PSQL SA --> SE state KC { SE --> E1 SE --> E2 SE --> E3 state P1 { E1 --> SSC E2 --> SSC E3 --> SSC } SSC --> SC1 SSC --> SC2 SSC --> SC3 state P2 { SC1 --> PSQL SC1 --> SC2 SC1 --> SC3 SC2 --> PSQL SC2 --> SC1 SC2 --> SC3 SC3 --> PSQL SC3 --> SC1 SC3 --> SC2 } state P3 { PSQL } } ``` ### What you can do in this sample application The sample application supports the following types of transactions: - Get customer information. - Place an order by using a line of credit. - Checks if the cost of the order is below the customer's credit limit. - If the check passes, records the order history and updates the amount the customer has spent. - Get order information by order ID. - Get order information by customer ID. - Make a payment. - Reduces the amount the customer has spent. ## Prerequisites for this sample application - [.NET SDK 8.0](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) - ScalarDB Cluster running on a Kubernetes cluster - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). :::note .NET SDK 8.0 is the version used to create the sample application. For information about all supported versions, see [Requirements](../requirements.mdx#net) ::: ## Set up ScalarDB Cluster The following sections describe how to set up the sample e-commerce application. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sample ``` ### Update the referenced version of the ScalarDB.Client package To use ScalarDB Cluster, open `ScalarDbClusterSample.csproj` in your preferred text editor. Then, update the version of the referenced `ScalarDB.Client` package, replacing `.` with the version of the deployed ScalarDB Cluster: ```xml ``` ### Modify `scalardb-options.json` You need to modify `scalardb-options.json` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the Envoy service resource (`scalardb-cluster-envoy`). To get the service resource, run the following command: ```console kubectl get svc scalardb-cluster-envoy ``` You should see a similar output as below, with different values for `CLUSTER-IP`, `PORT(S)`, and `AGE`: ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`. Open `scalardb-options.json` by running the following command: ```console vim scalardb-options.json ``` Then, modify `scalardb-options.json` as follows: ```json { "ScalarDbOptions": { "Address": "http://localhost:60053" } } ``` ### Load the initial data Before running the sample application, you need to load the initial data by running the following command: ```console dotnet run LoadInitialData ``` #### Schema details Running the command above will also apply the schema. All the tables are created in the `sample` namespace. - `sample.customers`: a table that manages customer information - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit - `credit_total`: the amount of money that the customer has spent from their line of credit - `sample.orders`: a table that manages order information - `sample.statements`: a table that manages order statement information - `sample.items`: a table that manages information for items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/getting-started-ERD.png) #### Initial data After the initial data has loaded, the following records should be stored in the tables. **`sample.customers` table** | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | **`sample.items` table** | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Execute transactions and retrieve data in the sample application The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. ### Get customer information Start with getting information about the customer whose ID is `1` by running the following command: ```console dotnet run GetCustomerInfo 1 ``` You should see the following output: ```console { "id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0 } ``` ### Place an order Then, have customer ID `1` place an order for three apples and two oranges by running the following command: :::note The order format in this command is `dotnet run PlaceOrder :,:,..."`. ::: ```console dotnet run PlaceOrder 1 1:3,2:2 ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "order_id": "5a22150b-1944-403f-b02c-77183e705d1b" } ``` ### Check the order details Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: ```console dotnet run GetOrder ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: ```console { "order": { "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", "timestamp": 1743143358216, "customer_id": 1, "customer_name": "Yamada Taro", "statements": [ { "item_id": 1, "item_name": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "item_id": 2, "item_name": "Orange", "price": 2000, "count": 2, "total": 4000 } ], "total": 7000 } } ``` ### Place another order Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: ```console dotnet run PlaceOrder 1 5:1 ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8" } ``` ### Check the order history Get the history of all orders for customer ID `1` by running the following command: ```console dotnet run GetOrders 1 ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1`: ```console { "orders": [ { "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", "timestamp": 1743143358216, "customer_id": 1, "customer_name": "Yamada Taro", "statements": [ { "item_id": 1, "item_name": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "item_id": 2, "item_name": "Orange", "price": 2000, "count": 2, "total": 4000 } ], "total": 7000 }, { "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8", "timestamp": 1743143505436, "customer_id": 1, "customer_name": "Yamada Taro", "statements": [ { "item_id": 5, "item_name": "Melon", "price": 3000, "count": 1, "total": 3000 } ], "total": 3000 } ] } ``` ### Check the credit total Get the credit total for customer ID `1` by running the following command: ```console dotnet run GetCustomerInfo 1 ``` You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: ```console { "id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000 } ``` Try to place an order for one grape and one mango by running the following command: ```console dotnet run PlaceOrder 1 3:1,4:1 ``` You should see a similar output as below, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. ```console Unhandled exception: System.Exception: Credit limit exceeded (17500 > 10000) at ScalarDbClusterSample.Sample.PlaceOrder(Int32 customerId, IReadOnlyDictionary`2 itemCounts) in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sample/Sample.cs:line 254 at ScalarDbClusterSample.Commands.PlaceOrderCommand.<>c.<b__6_0>d.MoveNext() in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sample/Commands/PlaceOrderCommand.cs:line 47 --- End of stack trace from previous location --- ... ``` ### Make a payment To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. Make a payment by running the following command: ```console dotnet run Repayment 1 8000 ``` Then, check the `credit_total` amount for customer ID `1` by running the following command: ```console dotnet run GetCustomerInfo 1 ``` You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: ```console { "id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000 } ``` Now that customer ID `1` has made a payment, place an order for one grape and one mango by running the following command: ```console dotnet run PlaceOrder 1 3:1,4:1 ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "order_id": "ecd68f46-e248-4f2e-b581-620e9019bf5b" } ``` ## See also For details about developing applications that use ScalarDB Cluster with the .NET API, refer to the following: - [ScalarDB Cluster .NET Client SDK Overview](../scalardb-cluster-dotnet-client-sdk/index.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/getting-started-with-scalardb-cluster-graphql.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB Cluster GraphQL import JavadocLink from '/src/theme/JavadocLink.js'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to use ScalarDB Cluster GraphQL. ## Prerequisites - One of the following Java Development Kits (JDKs): - ScalarDB Cluster running on a Kubernetes cluster - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). ## Sample application This tutorial illustrates the process of creating an electronic money application, where money can be transferred between accounts. The following diagram shows the system architecture of the sample application: ``` +----------------------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | | | [Pod] [Pod] [Pod] | | | | +-------+ | | +---> | Envoy | ---+ | | | +-------+ | | | | | | +------------------------+ | +---------+ | +-------+ | +--------------------+ | | Schema Loader | --+-> | Service | ---+---> | Envoy | ---+---------> | Service | ---+ | | (indirect client mode) | | | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | | +------------------------+ | +---------+ | | +--------------------+ | +-----------------------+ | | | +-------+ | | +---> | ScalarDB Cluster Node | ---+ | | +---> | Envoy | ---+ | | +-----------------------+ | | | +-------+ | | | | | | | +-----------------------+ | +------------+ | | +---+---> | ScalarDB Cluster Node | ---+---> | PostgreSQL | | | | | +-----------------------+ | +------------+ | | | | | | | | | +-----------------------+ | | | | +---> | ScalarDB Cluster Node | ---+ | | | +-----------------------+ | +------------+ | +----------------------------+ | | | Browser | ------+---------------------------------------> | Service | ---+ | | (GraphiQL) | | | (ScalarDB Cluster GraphQL) | | +------------+ | +----------------------------+ | | | +----------------------------------------------------------------------------------------------------------------------------------------+ ``` ## Step 1. Create `schema.json` The following is a simple example schema. Create `schema.json`, and add the following to the file: ```json { "emoney.account": { "transaction": true, "partition-key": [ "id" ], "clustering-key": [], "columns": { "id": "TEXT", "balance": "INT" } } } ``` ## Step 2. Create `database.properties` You need to create `database.properties` for the Schema Loader for ScalarDB Cluster. But first, you need to get the `EXTERNAL-IP` address of the service resource of Envoy (`scalardb-cluster-envoy`). To see the `EXTERNAL-IP` address, run the following command: ```console kubectl get svc scalardb-cluster-envoy NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`. Then, create `database.properties`, and add the following to the file: ```properties scalar.db.transaction_manager=cluster scalar.db.contact_points=indirect:localhost ``` To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.transaction_manager` property. In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). ## Step 3. Load a schema To load a schema via ScalarDB Cluster, you need to use the dedicated Schema Loader for ScalarDB Cluster (Schema Loader for Cluster). Using the Schema Loader for Cluster is basically the same as using the [Schema Loader for ScalarDB](../schema-loader.mdx) except the name of the JAR file is different. You can download the Schema Loader for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.18.0). After downloading the JAR file, you can run the Schema Loader for Cluster with the following command: ```console java -jar scalardb-cluster-schema-loader-3.18.0-all.jar --config database.properties -f schema.json --coordinator ``` ## Step 4. Run operations from GraphiQL In ScalarDB Cluster, if the `scalar.db.graphql.graphiql` property is set to `true` (`true` is the default value), the GraphiQL IDE will be available. To get the `EXTERNAL-IP` address of the service resource of ScalarDB Cluster GraphQL (`scalardb-cluster-graphql`), run the following command: ```console kubectl get svc scalardb-cluster-graphql NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-graphql LoadBalancer 10.105.74.214 localhost 8080:30514/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`, and the endpoint URL of GraphiQL IDE is `http://localhost:8080/graphql`. Opening that URL with your web browser will take you to the GraphiQL screen. Let's insert the first record. In the left pane, paste the following mutation, then push the triangle-shaped `Execute Query` button at the top of the window. ```graphql mutation PutUser1 { account_put(put: {key: {id: "user1"}, values: {balance: 1000}}) } ``` ScalarDB GraphQL always runs queries with transactions. The above query starts a new transaction, executes a ScalarDB Put command, and commits the transaction at the end of the execution. The following response from the GraphQL server will appear in the right pane: ```json { "data": { "account_put": true } } ``` The `"data"` field contains the result of the execution. This response shows the `account_put` field of the mutation was successful. The result type of mutations is `Boolean!`, which indicates whether the operation succeeded or not. Next, let's get the record you just inserted. Paste the following query next to the previous mutation in the left pane, and click the `Execute Query` button. Since you don't delete the `mutation PutUser1` above, a pull-down menu will appear below the button, and you can choose which operation should be executed. Choose `GetUser1`, as shown below: ```graphql query GetUser1 { account_get(get: {key: {id: "user1"}}) { account { id balance } } } ``` You should get the following result in the right pane: ```json { "data": { "account_get": { "account": { "id": "user1", "balance": 1000 } } } } ``` ### Mappings between GraphQL API and ScalarDB Java API The automatically generated GraphQL schema defines queries, mutations, and object types for input/output to allow you to run CRUD operations for all the tables in the target namespaces. These operations are designed to match the ScalarDB APIs defined in the interface. Assuming you have an `account` table in a namespace, the following queries and mutations will be generated: | ScalarDB API | GraphQL root type | GraphQL field | |--------------------------------------------------------|-------------------|------------------------------------------------------------------------------------| | `get(Get get)` | `Query` | `account_get(get: account_GetInput!): account_GetPayload` | | `scan(Scan scan)` | `Query` | `account_scan(scan: account_ScanInput!): account_ScanPayload` | | `put(Put put)` | `Mutation` | `account_put(put: account_PutInput!): Boolean!` | | `put(java.util.List puts)` | `Mutation` | `account_bulkPut(put: [account_PutInput!]!): Boolean!` | | `delete(Delete delete)` | `Mutation` | `account_delete(delete: account_DeleteInput!): Boolean!` | | `delete(java.util.List deletes)` | `Mutation` | `account_bulkDelete(delete: [account_DeleteInput!]!): Boolean!` | | `mutate(java.util.List mutations)` | `Mutation` | `account_mutate(put: [account_PutInput!]delete: [account_DeleteInput!]): Boolean!` | Note that the `scan` field is not generated for a table with no clustering key. This is the reason why the `account_scan` field is not available in this electronic money sample application. You can see all generated GraphQL types in GraphiQL's Documentation Explorer (the `< Docs` link at the top-left corner). ## Step 5. Run a transaction across multiple requests from GraphiQL Let's run a transaction that spans multiple GraphQL requests. The generated schema provides the `@transaction` directive that allows you to identify transactions. You can use this directive with both queries and mutations. Before starting a transaction, you need to insert the necessary record with the following mutation: ```graphql mutation PutUser2 { account_put(put: {key: {id: "user2"}, values: {balance: 1000}}) } ``` ### Start a transaction before running an operation Running the following to add a `@transaction` directive with no arguments to a query or mutation directs the execution to start a new transaction: ```graphql query GetAccounts @transaction { user1: account_get(get: {key: {id: "user1"}}) { account { balance } } user2: account_get(get: {key: {id: "user2"}}) { account { balance } } } ``` After running the above command, you will get a result with a transaction ID in the `extensions` field. The `id` value in the extensions is the transaction ID in which the operation in the request was run. In this case, the following is the new ID of the transaction just started by the request: ```json { "data": { "user1": { "account": { "balance": 1000 } }, "user2": { "account": { "balance": 1000 } } }, "extensions": { "transaction": { "id": "c88da8a6-a13f-4857-82fe-45f1ab4150f9" } } } ``` ### Run an operation in a continued transaction To run the next queries or mutations in the transaction you started, specify the transaction ID as the `id` argument of the `@transaction`. The following example updates two accounts you retrieved in the previous example by transferring a balance from user1's account to user2's account in the same transaction: ```graphql mutation Transfer @transaction(id: "c88da8a6-a13f-4857-82fe-45f1ab4150f9") { user1: account_put(put: {key: {id: "user1"}, values: {balance: 750}}) user2: account_put(put: {key: {id: "user2"}, values: {balance: 1250}}) } ``` Note that a transaction started with GraphQL has a timeout of 1 minute (by default) and will be aborted automatically if it exceeds the timeout. ### Commit a transaction To commit the continued transaction, specify both the `id` and the `commit: true` flag as arguments of the `@transaction` directive: ```graphql query GetAndCommit @transaction(id: "c88da8a6-a13f-4857-82fe-45f1ab4150f9", commit: true) { user1: account_get(get: {key: {id: "user1"}}) { account { balance } } user2: account_get(get: {key: {id: "user2"}}) { account { balance } } } ``` **Note:** If you specify a `commit: true` flag without an `id` argument like `@transaction(commit: true)`, a new transaction will start and be committed just for one operation. This behavior is exactly the same as not specifying the `@transaction` directive, as seen in the above examples using GraphiQL. In other words, you can omit the directive itself when `@transaction(commit: true)` is specified. ### Abort or roll back a transaction If you need to abort or roll back a transaction explicitly, you can use the `abort` or `rollback` mutation fields interchangeably (both have the same effect and usage). Note that you cannot mix these fields with any other operations, so you must specify only the `abort` or `rollback` mutation field as follows: ```graphql mutation AbortTx @transaction(id: "c88da8a6-a13f-4857-82fe-45f1ab4150f9") { abort } ``` Or: ```graphql mutation RollbackTx @transaction(id: "c88da8a6-a13f-4857-82fe-45f1ab4150f9") { rollback } ``` ## See also For other ScalarDB Cluster tutorials, see the following: - [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) - [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) - [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) - [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) - [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: - [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/getting-started-with-scalardb-cluster-sql-dotnet.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB Cluster SQL via .NET This tutorial describes how to create a sample application that uses [ScalarDB Cluster](./index.mdx) SQL through the .NET API. ## Overview This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB. :::note Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling, see [Exception Handling in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/exception-handling.mdx). ::: The following diagram shows the system architecture of the sample application: ```mermaid stateDiagram-v2 state "Sample application using the .NET API" as SA state "Kubernetes Cluster" as KC state "Service (Envoy)" as SE state "Pod" as P1 state "Pod" as P2 state "Pod" as P3 state "Envoy" as E1 state "Envoy" as E2 state "Envoy" as E3 state "Service (ScalarDB Cluster)" as SSC state "ScalarDB Cluster" as SC1 state "ScalarDB Cluster" as SC2 state "ScalarDB Cluster" as SC3 state "PostgreSQL" as PSQL SA --> SE state KC { SE --> E1 SE --> E2 SE --> E3 state P1 { E1 --> SSC E2 --> SSC E3 --> SSC } SSC --> SC1 SSC --> SC2 SSC --> SC3 state P2 { SC1 --> PSQL SC1 --> SC2 SC1 --> SC3 SC2 --> PSQL SC2 --> SC1 SC2 --> SC3 SC3 --> PSQL SC3 --> SC1 SC3 --> SC2 } state P3 { PSQL } } ``` ### What you can do in this sample application The sample application supports the following types of transactions: - Get customer information. - Place an order by using a line of credit. - Checks if the cost of the order is below the customer's credit limit. - If the check passes, records the order history and updates the amount the customer has spent. - Get order information by order ID. - Get order information by customer ID. - Make a payment. - Reduces the amount the customer has spent. ## Prerequisites for this sample application - [.NET SDK 8.0](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) - ScalarDB Cluster running on a Kubernetes cluster - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [How to Deploy ScalarDB Cluster Locally](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). :::note .NET SDK 8.0 is the version used to create the sample application. For information about all supported versions, see [Requirements](../requirements.mdx#net) ::: ## Set up ScalarDB Cluster The following sections describe how to set up the sample e-commerce application. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sql-sample ``` ### Update the referenced version of the ScalarDB.Client package To use ScalarDB Cluster, open `ScalarDbClusterSample.csproj` in your preferred text editor. Then, update the version of the referenced `ScalarDB.Client` package, replacing `.` with the version of the deployed ScalarDB Cluster: ```xml ``` ### Modify `scalardb-options.json` You need to modify `scalardb-options.json` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the Envoy service resource (`scalardb-cluster-envoy`). To get the service resource, run the following command: ```console kubectl get svc scalardb-cluster-envoy ``` You should see a similar output as below, with different values for `CLUSTER-IP`, `PORT(S)`, and `AGE`: ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`. Open `scalardb-options.json` by running the following command: ```console vim scalardb-options.json ``` Then, modify `scalardb-options.json` as follows: ```json { "ScalarDbOptions": { "Address": "http://localhost:60053" } } ``` ### Load the initial data Before running the sample application, you need to load the initial data by running the following command: ```console dotnet run LoadInitialData ``` #### Schema details Running the command above will also apply the schema. All the tables are created in the `sample` namespace. - `sample.customers`: a table that manages customer information - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit - `credit_total`: the amount of money that the customer has spent from their line of credit - `sample.orders`: a table that manages order information - `sample.statements`: a table that manages order statement information - `sample.items`: a table that manages information for items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/getting-started-ERD.png) #### Initial data After the initial data has loaded, the following records should be stored in the tables. **`sample.customers` table** | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | **`sample.items` table** | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Execute transactions and retrieve data in the sample application The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. ### Get customer information Start with getting information about the customer whose ID is `1` by running the following command: ```console dotnet run GetCustomerInfo 1 ``` You should see the following output: ```console { "id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0 } ``` ### Place an order Then, have customer ID `1` place an order for three apples and two oranges by running the following command: :::note The order format in this command is `dotnet run PlaceOrder :,:,..."`. ::: ```console dotnet run PlaceOrder 1 1:3,2:2 ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "order_id": "5a22150b-1944-403f-b02c-77183e705d1b" } ``` ### Check the order details Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: ```console dotnet run GetOrder ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: ```console { "order": { "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", "timestamp": 1743143358216, "customer_id": 1, "customer_name": "Yamada Taro", "statements": [ { "item_id": 1, "item_name": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "item_id": 2, "item_name": "Orange", "price": 2000, "count": 2, "total": 4000 } ], "total": 7000 } } ``` ### Place another order Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: ```console dotnet run PlaceOrder 1 5:1 ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8" } ``` ### Check the order history Get the history of all orders for customer ID `1` by running the following command: ```console dotnet run GetOrders 1 ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1`: ```console { "orders": [ { "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", "timestamp": 1743143358216, "customer_id": 1, "customer_name": "Yamada Taro", "statements": [ { "item_id": 1, "item_name": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "item_id": 2, "item_name": "Orange", "price": 2000, "count": 2, "total": 4000 } ], "total": 7000 }, { "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8", "timestamp": 1743143505436, "customer_id": 1, "customer_name": "Yamada Taro", "statements": [ { "item_id": 5, "item_name": "Melon", "price": 3000, "count": 1, "total": 3000 } ], "total": 3000 } ] } ``` ### Check the credit total Get the credit total for customer ID `1` by running the following command: ```console dotnet run GetCustomerInfo 1 ``` You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: ```console { "id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000 } ``` Try to place an order for one grape and one mango by running the following command: ```console dotnet run PlaceOrder 1 3:1,4:1 ``` You should see a similar output as below, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. ```console Unhandled exception: System.Exception: Credit limit exceeded (17500 > 10000) at ScalarDbClusterSqlSample.Sample.PlaceOrder(Int32 customerId, IReadOnlyDictionary`2 itemCounts) in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sql-sample/Sample.cs:line 237 at ScalarDbClusterSqlSample.Commands.PlaceOrderCommand.<>c.<b__6_0>d.MoveNext() in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sql-sample/Commands/PlaceOrderCommand.cs:line 47 --- End of stack trace from previous location --- ... ``` ### Make a payment To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. Make a payment by running the following command: ```console dotnet run Repayment 1 8000 ``` Then, check the `credit_total` amount for customer ID `1` by running the following command: ```console dotnet run GetCustomerInfo 1 ``` You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: ```console { "id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000 } ``` Now that customer ID `1` has made a payment, place an order for one grape and one mango by running the following command: ```console dotnet run PlaceOrder 1 3:1,4:1 ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "order_id": "ecd68f46-e248-4f2e-b581-620e9019bf5b" } ``` ## See also For details about developing applications that use ScalarDB Cluster with the .NET API, refer to the following: - [ScalarDB Cluster .NET Client SDK Overview](../scalardb-cluster-dotnet-client-sdk/index.mdx) - [Getting Started with Distributed SQL Transactions in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-sql-transactions.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/getting-started-with-scalardb-cluster-sql-jdbc.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB Cluster SQL via JDBC import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to create a sample application by using ScalarDB Cluster SQL via JDBC. ## Prerequisites for this sample application - One of the following Java Development Kits (JDKs): - ScalarDB Cluster running on a Kubernetes cluster - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). ## Sample application This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB JDBC. The following diagram shows the system architecture of the sample application: ``` +------------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | | | [Pod] [Pod] [Pod] | +------------------------+ | | | SQL CLI | | +-------+ +-----------------------+ | | (indirect client mode) | --+ | +---> | Envoy | ---+ +---> | ScalarDB Cluster Node | ---+ | +------------------------+ | | | +-------+ | | +-----------------------+ | | | | | | | | | | | +---------+ | +-------+ | +--------------------+ | +-----------------------+ | +------------+ | +--+-> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | ScalarDB Cluster Node | ---+---> | PostgreSQL | | | | | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | +-----------------------+ | +------------+ | +------------------------+ | | +---------+ | | +--------------------+ | | | | Sample application | | | | +-------+ | | +-----------------------+ | | | with ScalarDB JDBC | --+ | +---> | Envoy | ---+ +---> | ScalarDB Cluster Node | ---+ | | (indirect client mode) | | +-------+ +-----------------------+ | +------------------------+ | | +------------------------------------------------------------------------------------------------------------------------------+ ``` ## Step 1. Clone the ScalarDB Samples repository ```console git clone https://github.com/scalar-labs/scalardb-samples.git cd scalardb-samples/scalardb-sql-jdbc-sample ``` ## Step 2. Modify `scalardb-sql.properties` You need to modify `scalardb-sql.properties` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the service resource of Envoy (`scalardb-cluster-envoy`) as follows: ```console kubectl get svc scalardb-cluster-envoy NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`. Next, open `scalardb-sql.properties`: ```console vim scalardb-sql.properties ``` Then, modify `scalardb-sql.properties` as follows: ```properties scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points=indirect:localhost ``` To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.sql.connection_mode` property. In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). ## Step 3. Load a schema To load a schema, you need to use [the SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli). You can download the SQL CLI from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.18.0). After downloading the JAR file, you can use SQL CLI for Cluster by running the following command: ```console java -jar scalardb-cluster-sql-cli-3.18.0-all.jar --config scalardb-sql.properties --file schema.sql ``` ## Step 4. Load the initial data Before running the sample application, you need to load the initial data by running the following command: ```console ./gradlew run --args="LoadInitialData" ``` After the initial data has loaded, the following records should be stored in the tables: - For the `sample.customers` table: | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | - For the `sample.items` table: | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Step 5. Run the sample application Let's start with getting information about the customer whose ID is `1`: ```console ./gradlew run --args="GetCustomerInfo 1" ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0} ... ``` Then, place an order for three apples and two oranges by using customer ID `1`. Note that the order format is `:,:,...`: ```console ./gradlew run --args="PlaceOrder 1 1:3,2:2" ... {"order_id": "454f9c97-f456-44fd-96da-f527187fe39b"} ... ``` You can see that running this command shows the order ID. Let's check the details of the order by using the order ID: ```console ./gradlew run --args="GetOrder 454f9c97-f456-44fd-96da-f527187fe39b" ... {"order": {"order_id": "454f9c97-f456-44fd-96da-f527187fe39b","timestamp": 1685602722821,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1, "name": "Apple", "price": 1000, "count": 3},{"item_id": 2, "name": "Orange", "price": 2000, "count": 2}],"total": 7000}} ... ``` Then, let's place another order and get the order history of customer ID `1`: ```console ./gradlew run --args="PlaceOrder 1 5:1" ... {"order_id": "3f40c718-59ec-48aa-a6fe-2fdaf12ad094"} ... ./gradlew run --args="GetOrders 1" ... {"order": [{"order_id": "454f9c97-f456-44fd-96da-f527187fe39b","timestamp": 1685602722821,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1, "name": "Apple", "price": 1000, "count": 3},{"item_id": 2, "name": "Orange", "price": 2000, "count": 2}],"total": 7000},{"order_id": "3f40c718-59ec-48aa-a6fe-2fdaf12ad094","timestamp": 1685602811718,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5, "name": "Melon", "price": 3000, "count": 1}],"total": 3000}]} ... ``` This order history is shown in descending order by timestamp. The customer's current `credit_total` is `10000`. Since the customer has now reached their `credit_limit`, which was shown when retrieving their information, they cannot place anymore orders. ```console ./gradlew run --args="GetCustomerInfo 1" ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} ... ./gradlew run --args="PlaceOrder 1 3:1,4:1" ... java.lang.RuntimeException: Credit limit exceeded at sample.Sample.placeOrder(Sample.java:184) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:32) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:8) at picocli.CommandLine.executeUserObject(CommandLine.java:2041) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) at picocli.CommandLine.execute(CommandLine.java:2170) at sample.command.SampleCommand.main(SampleCommand.java:35) ... ``` After making a payment, the customer will be able to place orders again. ```console ./gradlew run --args="Repayment 1 8000" ... ./gradlew run --args="GetCustomerInfo 1" ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000} ... ./gradlew run --args="PlaceOrder 1 3:1,4:1" ... {"order_id": "fb71279d-88ea-4974-a102-0ec4e7d65e25"} ... ``` ## Source code of the sample application To learn more about ScalarDB Cluster SQL JDBC, you can check the [source code of the sample application](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-sql-jdbc-sample/src/main/java/sample). ## See also For other ScalarDB Cluster tutorials, see the following: - [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) - [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) - [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) - [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) - [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: - [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/getting-started-with-scalardb-cluster-sql-linq.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB Cluster SQL via .NET and LINQ This tutorial describes how to create a sample application that uses [ScalarDB Cluster](./index.mdx) SQL through LINQ. ## Overview This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB. :::note Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling, see [Exception Handling in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/exception-handling.mdx). ::: The following diagram shows the system architecture of the sample application: ```mermaid stateDiagram-v2 state "Sample application using the .NET API" as SA state "Kubernetes Cluster" as KC state "Service (Envoy)" as SE state "Pod" as P1 state "Pod" as P2 state "Pod" as P3 state "Envoy" as E1 state "Envoy" as E2 state "Envoy" as E3 state "Service (ScalarDB Cluster)" as SSC state "ScalarDB Cluster" as SC1 state "ScalarDB Cluster" as SC2 state "ScalarDB Cluster" as SC3 state "PostgreSQL" as PSQL SA --> SE state KC { SE --> E1 SE --> E2 SE --> E3 state P1 { E1 --> SSC E2 --> SSC E3 --> SSC } SSC --> SC1 SSC --> SC2 SSC --> SC3 state P2 { SC1 --> PSQL SC1 --> SC2 SC1 --> SC3 SC2 --> PSQL SC2 --> SC1 SC2 --> SC3 SC3 --> PSQL SC3 --> SC1 SC3 --> SC2 } state P3 { PSQL } } ``` ### What you can do in this sample application The sample application supports the following types of transactions: - Get customer information. - Place an order by using a line of credit. - Checks if the cost of the order is below the customer's credit limit. - If the check passes, records the order history and updates the amount the customer has spent. - Get order information by order ID. - Get order information by customer ID. - Make a payment. - Reduces the amount the customer has spent. ## Prerequisites for this sample application - [.NET SDK 8.0](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) - ScalarDB Cluster running on a Kubernetes cluster - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [How to Deploy ScalarDB Cluster Locally](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). :::note .NET SDK 8.0 is the version used to create the sample application. For information about all supported versions, see [Requirements](../requirements.mdx#net) ::: ## Set up ScalarDB Cluster The following sections describe how to set up the sample e-commerce application. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-linq-sample ``` ### Update the referenced version of the ScalarDB.Client package To use ScalarDB Cluster, open `ScalarDbClusterSample.csproj` in your preferred text editor. Then, update the version of the referenced `ScalarDB.Client` package, replacing `.` with the version of the deployed ScalarDB Cluster: ```xml ``` ### Modify `scalardb-options.json` You need to modify `scalardb-options.json` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the Envoy service resource (`scalardb-cluster-envoy`). To get the service resource, run the following command: ```console kubectl get svc scalardb-cluster-envoy ``` You should see a similar output as below, with different values for `CLUSTER-IP`, `PORT(S)`, and `AGE`: ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`. Open `scalardb-options.json` by running the following command: ```console vim scalardb-options.json ``` Then, modify `scalardb-options.json` as follows: ```json { "ScalarDbOptions": { "Address": "http://localhost:60053" } } ``` ### Load the initial data Before running the sample application, you need to load the initial data by running the following command: ```console dotnet run LoadInitialData ``` #### Schema details Running the command above will also apply the schema. All the tables are created in the `sample` namespace. - `sample.customers`: a table that manages customer information - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit - `credit_total`: the amount of money that the customer has spent from their line of credit - `sample.orders`: a table that manages order information - `sample.statements`: a table that manages order statement information - `sample.items`: a table that manages information for items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/getting-started-ERD.png) #### Initial data After the initial data has loaded, the following records should be stored in the tables. **`sample.customers` table** | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | **`sample.items` table** | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Execute transactions and retrieve data in the sample application The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. ### Get customer information Start with getting information about the customer whose ID is `1` by running the following command: ```console dotnet run GetCustomerInfo 1 ``` You should see the following output: ```console { "id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0 } ``` ### Place an order Then, have customer ID `1` place an order for three apples and two oranges by running the following command: :::note The order format in this command is `dotnet run PlaceOrder :,:,..."`. ::: ```console dotnet run PlaceOrder 1 1:3,2:2 ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "order_id": "5a22150b-1944-403f-b02c-77183e705d1b" } ``` ### Check the order details Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: ```console dotnet run GetOrder ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: ```console { "order": { "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", "timestamp": 1743143358216, "customer_id": 1, "customer_name": "Yamada Taro", "statements": [ { "item_id": 1, "item_name": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "item_id": 2, "item_name": "Orange", "price": 2000, "count": 2, "total": 4000 } ], "total": 7000 } } ``` ### Place another order Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: ```console dotnet run PlaceOrder 1 5:1 ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8" } ``` ### Check the order history Get the history of all orders for customer ID `1` by running the following command: ```console dotnet run GetOrders 1 ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1`: ```console { "orders": [ { "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", "timestamp": 1743143358216, "customer_id": 1, "customer_name": "Yamada Taro", "statements": [ { "item_id": 1, "item_name": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "item_id": 2, "item_name": "Orange", "price": 2000, "count": 2, "total": 4000 } ], "total": 7000 }, { "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8", "timestamp": 1743143505436, "customer_id": 1, "customer_name": "Yamada Taro", "statements": [ { "item_id": 5, "item_name": "Melon", "price": 3000, "count": 1, "total": 3000 } ], "total": 3000 } ] } ``` ### Check the credit total Get the credit total for customer ID `1` by running the following command: ```console dotnet run GetCustomerInfo 1 ``` You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: ```console { "id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000 } ``` Try to place an order for one grape and one mango by running the following command: ```console dotnet run PlaceOrder 1 3:1,4:1 ``` You should see a similar output as below, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. ```console Unhandled exception: System.Exception: Credit limit exceeded (17500 > 10000) at ScalarDbClusterLinqSample.Sample.PlaceOrder(Int32 customerId, IReadOnlyDictionary`2 itemCounts) in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-linq-sample/Sample.cs:line 145 at ScalarDbClusterLinqSample.Commands.PlaceOrderCommand.<>c.<b__6_0>d.MoveNext() in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-linq-sample/Commands/PlaceOrderCommand.cs:line 47 --- End of stack trace from previous location --- ... ``` ### Make a payment To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. Make a payment by running the following command: ```console dotnet run Repayment 1 8000 ``` Then, check the `credit_total` amount for customer ID `1` by running the following command: ```console dotnet run GetCustomerInfo 1 ``` You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: ```console { "id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000 } ``` Now that customer ID `1` has made a payment, place an order for one grape and one mango by running the following command: ```console dotnet run PlaceOrder 1 3:1,4:1 ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "order_id": "ecd68f46-e248-4f2e-b581-620e9019bf5b" } ``` ## See also For details about developing applications that use ScalarDB Cluster with the .NET API, refer to the following: - [ScalarDB Cluster .NET Client SDK Overview](../scalardb-cluster-dotnet-client-sdk/index.mdx) - [Getting Started with LINQ in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-linq.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to create a sample application by using ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB. ## Prerequisites for this sample application - One of the following Java Development Kits (JDKs): - ScalarDB Cluster running on a Kubernetes cluster - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). ## Sample application This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using Spring Data JDBC for ScalarDB. The following diagram shows the system architecture of the sample application: ``` +------------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | | | [Pod] [Pod] [Pod] | +------------------------+ | | | SQL CLI | | +-------+ +-----------------------+ | | (indirect client mode) | --+ | +---> | Envoy | ---+ +---> | ScalarDB Cluster Node | ---+ | +------------------------+ | | | +-------+ | | +-----------------------+ | | | | | | | | | | | +---------+ | +-------+ | +--------------------+ | +-----------------------+ | +------------+ | +--+-> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | ScalarDB Cluster Node | ---+---> | PostgreSQL | | +------------------------+ | | | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | +-----------------------+ | +------------+ | | Sample application | | | +---------+ | | +--------------------+ | | | | with Spring Data JDBC | | | | +-------+ | | +-----------------------+ | | | for ScalarDB | --+ | +---> | Envoy | ---+ +---> | ScalarDB Cluster Node | ---+ | | (indirect client mode) | | +-------+ +-----------------------+ | +------------------------+ | | +------------------------------------------------------------------------------------------------------------------------------+ ``` ## Step 1. Clone the ScalarDB Samples repository ```console git clone https://github.com/scalar-labs/scalardb-samples.git cd scalardb-samples/spring-data-sample ``` ## Step 2. Modify `scalardb-sql.properties` You need to modify `scalardb-sql.properties` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the service resource of Envoy (`scalardb-cluster-envoy`) as follows: ```console kubectl get svc scalardb-cluster-envoy NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`. Next, open `scalardb-sql.properties`: ```console vim scalardb-sql.properties ``` Then, modify `scalardb-sql.properties` as follows: ```properties scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points=indirect:localhost ``` To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.sql.connection_mode` property. In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). ## Step 3. Load a schema To load a schema, you need to use [the SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli). You can download the SQL CLI from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.18.0). After downloading the JAR file, you can use SQL CLI for Cluster by running the following command: ```console java -jar scalardb-cluster-sql-cli-3.18.0-all.jar --config scalardb-sql.properties --file schema.sql ``` ## Step 4. Modify `application.properties` Then, you need to modify `application.properties` to connect to ScalarDB Cluster as well: ```console vim src/main/resources/application.properties ``` Similar to `scalardb-sql.properties`, you need to specify `cluster` for the `scalar.db.sql.connection_mode` property and use the `indirect` client mode. To do so, modify `application.properties` as follows: ```properties spring.datasource.driver-class-name=com.scalar.db.sql.jdbc.SqlJdbcDriver spring.datasource.url=jdbc:scalardb:\ ?scalar.db.sql.connection_mode=cluster\ &scalar.db.sql.cluster_mode.contact_points=indirect:localhost\ &scalar.db.consensus_commit.isolation_level=SERIALIZABLE\ &scalar.db.sql.default_namespace_name=sample ``` ## Step 5. Load the initial data Before running the sample application, you need to load the initial data by running the following command: ```console ./gradlew run --args="LoadInitialData" ``` After the initial data has loaded, the following records should be stored in the tables: - For the `sample.customers` table: | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | - For the `sample.items` table: | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Step 6. Run the sample application Let's start with getting information about the customer whose ID is `1`: ```console ./gradlew run --args="GetCustomerInfo 1" ... {"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":0} ... ``` Then, place an order for three apples and two oranges by using customer ID `1`. Note that the order format is `:,:,...`: ```console ./gradlew run --args="PlaceOrder 1 1:3,2:2" ... {"order_id":"2358ab35-5819-4f8f-acb1-12e73d97d34e","customer_id":1,"timestamp":1677478005400} ... ``` You can see that running this command shows the order ID. Let's check the details of the order by using the order ID: ```console ./gradlew run --args="GetOrder 2358ab35-5819-4f8f-acb1-12e73d97d34e" ... {"order_id":"2358ab35-5819-4f8f-acb1-12e73d97d34e","timestamp":1677478005400,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":1,"item_name":"Apple","price":1000,"count":3,"total":3000},{"item_id":2,"item_name":"Orange","price":2000,"count":2,"total":4000}],"total":7000} ... ``` Then, let's place another order and get the order history of customer ID `1`: ```console ./gradlew run --args="PlaceOrder 1 5:1" ... {"order_id":"46062b16-b71b-46f9-a9ff-dc6b0991259b","customer_id":1,"timestamp":1677478201428} ... ./gradlew run --args="GetOrders 1" ... [{"order_id":"46062b16-b71b-46f9-a9ff-dc6b0991259b","timestamp":1677478201428,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":5,"item_name":"Melon","price":3000,"count":1,"total":3000}],"total":3000},{"order_id":"2358ab35-5819-4f8f-acb1-12e73d97d34e","timestamp":1677478005400,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":1,"item_name":"Apple","price":1000,"count":3,"total":3000},{"item_id":2,"item_name":"Orange","price":2000,"count":2,"total":4000}],"total":7000}] ... ``` This order history is shown in descending order by timestamp. The customer's current `credit_total` is `10000`. Since the customer has now reached their `credit_limit`, which was shown when retrieving their information, they cannot place anymore orders. ```console ./gradlew run --args="GetCustomerInfo 1" ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} ... ./gradlew run --args="PlaceOrder 1 3:1,4:1" ... java.lang.RuntimeException: Credit limit exceeded. limit:10000, total:17500 at sample.SampleService.placeOrder(SampleService.java:102) at sample.SampleService$$FastClassBySpringCGLIB$$1123c447.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at sample.SampleService$$EnhancerBySpringCGLIB$$a94e1d9.placeOrder() at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:37) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:13) at picocli.CommandLine.executeUserObject(CommandLine.java:2041) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) at picocli.CommandLine.execute(CommandLine.java:2170) at sample.SampleApp.run(SampleApp.java:26) at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768) at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752) at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292) at sample.SampleApp.main(SampleApp.java:35) ... ``` After making a payment, the customer will be able to place orders again. ```console ./gradlew run --args="Repayment 1 8000" ... ./gradlew run --args="GetCustomerInfo 1" ... {"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":2000} ... ./gradlew run --args="PlaceOrder 1 3:1,4:1" ... {"order_id":"0350947a-9003-46f2-870e-6aa4b2df0f1f","customer_id":1,"timestamp":1677478728134} ... ``` ## Source code of the sample application To learn more about Spring Data JDBC for ScalarDB, you can check the [source code of the sample application](https://github.com/scalar-labs/scalardb-samples/tree/main/spring-data-sample/src/main). ## See also For other ScalarDB Cluster tutorials, see the following: - [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) - [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) - [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) - [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) - [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: - [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/getting-started-with-scalardb-cluster.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB Cluster import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to create a sample application that uses [ScalarDB Cluster](./index.mdx) through the Java API. ## Overview This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB. :::note Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling in ScalarDB, see [Handle exceptions](../api-guide.mdx#how-to-handle-exceptions). ::: The following diagram shows the system architecture of the sample application: ```mermaid stateDiagram-v2 state "Schema Loader
(indirect client mode)" as SL state "Sample application using the Java API
(indirect client mode)" as SA state "Kubernetes Cluster" as KC state "Service (Envoy)" as SE state "Pod" as P1 state "Pod" as P2 state "Pod" as P3 state "Envoy" as E1 state "Envoy" as E2 state "Envoy" as E3 state "Service (ScalarDB Cluster)" as SSC state "ScalarDB Cluster" as SC1 state "ScalarDB Cluster" as SC2 state "ScalarDB Cluster" as SC3 state "PostgreSQL" as PSQL SL --> SE SA --> SE state KC { SE --> E1 SE --> E2 SE --> E3 state P1 { E1 --> SSC E2 --> SSC E3 --> SSC } SSC --> SC1 SSC --> SC2 SSC --> SC3 state P2 { SC1 --> PSQL SC1 --> SC2 SC1 --> SC3 SC2 --> PSQL SC2 --> SC1 SC2 --> SC3 SC3 --> PSQL SC3 --> SC1 SC3 --> SC2 } state P3 { PSQL } } ``` ### What you can do in this sample application The sample application supports the following types of transactions: - Get customer information. - Place an order by using a line of credit. - Checks if the cost of the order is below the customer's credit limit. - If the check passes, records the order history and updates the amount the customer has spent. - Get order information by order ID. - Get order information by customer ID. - Make a payment. - Reduces the amount the customer has spent. ## Prerequisites for this sample application - One of the following Java Development Kits (JDKs): - ScalarDB Cluster running on a Kubernetes cluster - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). ## Set up ScalarDB Cluster The following sections describe how to set up the sample e-commerce application. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/scalardb-sample ``` ### Modify `build.gradle` To use ScalarDB Cluster, open `build.gradle` in your preferred text editor. Then, delete the existing dependency for `com.scalar-labs:scalardb` from the `dependencies` section, and add the following dependency to the `dependencies` section: ```gradle dependencies { ... implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.18.0' } ``` ### Modify `database.properties` You need to modify `database.properties` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the Envoy service resource (`scalardb-cluster-envoy`). To get the service resource, run the following command: ```console kubectl get svc scalardb-cluster-envoy ``` You should see a similar output as below, with different values for `CLUSTER-IP`, `PORT(S)`, and `AGE`: ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`. In `database.properties`, you need to specify `cluster` for the `scalar.db.transaction_manager` property and use `indirect` as the client mode for `scalar.db.contact_points` to connect to the Envoy service resource. Open `database.properties` by running the following command: ```console vim database.properties ``` Then, modify `database.properties` as follows: ```properties scalar.db.transaction_manager=cluster scalar.db.contact_points=indirect:localhost ``` :::note For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). ::: ### Load the schema The database schema (the method in which the data will be organized) for the sample application has already been defined in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-sample/schema.json). To apply the schema, go to [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.18.0) and download the ScalarDB Cluster Schema Loader to the `scalardb-samples/scalardb-sample` folder. Then, run the following command: ```console java -jar scalardb-cluster-schema-loader-3.18.0-all.jar --config database.properties -f schema.json --coordinator ``` #### Schema details As shown in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-sample/schema.json) for the sample application, all the tables are created in the `sample` namespace. - `sample.customers`: a table that manages customer information - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit - `credit_total`: the amount of money that the customer has spent from their line of credit - `sample.orders`: a table that manages order information - `sample.statements`: a table that manages order statement information - `sample.items`: a table that manages information for items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/getting-started-ERD.png) ### Load the initial data Before running the sample application, you need to load the initial data by running the following command: ```console ./gradlew run --args="LoadInitialData" ``` After the initial data has loaded, the following records should be stored in the tables. **`sample.customers` table** | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | **`sample.items` table** | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Execute transactions and retrieve data in the sample application The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. ### Get customer information Start with getting information about the customer whose ID is `1` by running the following command: ```console ./gradlew run --args="GetCustomerInfo 1" ``` You should see the following output: ```console ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0} ... ``` ### Place an order Then, have customer ID `1` place an order for three apples and two oranges by running the following command: :::note The order format in this command is `./gradlew run --args="PlaceOrder :,:,..."`. ::: ```console ./gradlew run --args="PlaceOrder 1 1:3,2:2" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e"} ... ``` ### Check the order details Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: ```console ./gradlew run --args="GetOrder " ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: ```console ... {"order": {"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} ... ``` ### Place another order Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: ```console ./gradlew run --args="PlaceOrder 1 5:1" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d"} ... ``` ### Check the order history Get the history of all orders for customer ID `1` by running the following command: ```console ./gradlew run --args="GetOrders 1" ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: ```console ... {"order": [{"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000},{"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d","timestamp": 1650948412766,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000}]} ... ``` ### Check the credit total Get the credit total for customer ID `1` by running the following command: ```console ./gradlew run --args="GetCustomerInfo 1" ``` You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: ```console ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} ... ``` Try to place an order for one grape and one mango by running the following command: ```console ./gradlew run --args="PlaceOrder 1 3:1,4:1" ``` You should see the following output, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. ```console ... java.lang.RuntimeException: Credit limit exceeded at sample.Sample.placeOrder(Sample.java:205) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:33) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:8) at picocli.CommandLine.executeUserObject(CommandLine.java:1783) at picocli.CommandLine.access$900(CommandLine.java:145) at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975) at picocli.CommandLine.execute(CommandLine.java:1904) at sample.command.SampleCommand.main(SampleCommand.java:35) ... ``` ### Make a payment To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. Make a payment by running the following command: ```console ./gradlew run --args="Repayment 1 8000" ``` Then, check the `credit_total` amount for customer ID `1` by running the following command: ```console ./gradlew run --args="GetCustomerInfo 1" ``` You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: ```console ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000} ... ``` Now that customer ID `1` has made a payment, place an order for one grape and one mango by running the following command: ```console ./gradlew run --args="PlaceOrder 1 3:1,4:1" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "8911cab3-1c2b-4322-9386-adb1c024e078"} ... ``` ## See also For other ScalarDB Cluster tutorials, see the following: - [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) - [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) - [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) - [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) - [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: - [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/getting-started-with-using-go-for-scalardb-cluster.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Using Go for ScalarDB Cluster import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; This document explains how to write gRPC client code for ScalarDB Cluster by using Go. ## Prerequisites - [Go](https://go.dev/dl/) (any one of the three latest major releases) - ScalarDB Cluster running on a Kubernetes cluster - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). ## Sample application This tutorial illustrates the process of creating an electronic money application, where money can be transferred between accounts. ## Step 1. Create `schema.json` The following is a simple example schema. Create `schema.json`, and add the following to the file: ```json { "emoney.account": { "transaction": true, "partition-key": [ "id" ], "clustering-key": [], "columns": { "id": "TEXT", "balance": "INT" } } } ``` ## Step 2. Create `database.properties` You need to create `database.properties` for the Schema Loader for ScalarDB Cluster. But first, you need to get the `EXTERNAL-IP` address of the service resource of the `LoadBalancer` service (`scalardb-cluster-envoy`). To see the `EXTERNAL-IP` address, run the following command: ```console kubectl get svc scalardb-cluster-envoy NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`. Then, create `database.properties`, and add the following to the file: ```properties scalar.db.transaction_manager=cluster scalar.db.contact_points=indirect:localhost ``` To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.transaction_manager` property. In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). ## Step 3. Load a schema To load a schema via ScalarDB Cluster, you need to use the dedicated Schema Loader for ScalarDB Cluster (Schema Loader for Cluster). Using the Schema Loader for Cluster is basically the same as using the [Schema Loader for ScalarDB](../schema-loader.mdx) except the name of the JAR file is different. You can download the Schema Loader for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.18.0). After downloading the JAR file, you can run the Schema Loader for Cluster with the following command: ```console java -jar scalardb-cluster-schema-loader-3.18.0-all.jar --config database.properties -f schema.json --coordinator ``` ## Step 4. Set up a Go environment Follow the [Prerequisites](https://grpc.io/docs/languages/go/quickstart/#prerequisites) section in the gRPC quick-start document to install the following components: - Go - Protocol buffer compiler, `protoc`, version 3.15 or later - Go plugins for the protocol compiler ## Step 5. Generate the stub code for ScalarDB Cluster gRPC To communicate with the gRPC server for ScalarDB Cluster, you will need to generate the stub code from the proto file. First, in a new working directory, create a directory named `scalardb-cluster`, which you will use to generate the gRPC code from, by running the following command: ```console mkdir scalardb-cluster ``` Then, download the `scalardb-cluster.proto` file and save it in the directory that you created. For ScalarDB Cluster users who have a commercial license, please [contact support](https://www.scalar-labs.com/support) if you need the `scalardb-cluster.proto` file. Generate the gRPC code by running the following command: ```console protoc --go_out=. --go_opt=paths=source_relative \ --go_opt=Mscalardb-cluster/scalardb-cluster.proto=example.com/scalardb-cluster \ --go-grpc_out=. --go-grpc_opt=paths=source_relative \ --go-grpc_opt=Mscalardb-cluster/scalardb-cluster.proto=example.com/scalardb-cluster \ scalardb-cluster/scalardb-cluster.proto ``` After running the command, you should see two files in the `scalardb-cluster` subdirectory: `scalardb-cluster.pb.go` and `scalardb-cluster_grpc.pb.go`. ## Step 6. Write a sample application The following is the program that uses the gRPC code. Save it as `main.go` in the working directory. This program does the same thing as the `ElectronicMoney.java` program in [Getting Started with ScalarDB](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb/). Note that you have to update the value of `SERVER_ADDRESS` based on the `EXTERNAL-IP` value of the ScalarDB Cluster `LoadBalancer` service in your environment. ```go package main import ( "context" "errors" "flag" "fmt" "log" "os" "time" pb "emoney/scalardb-cluster" "google.golang.org/grpc" "google.golang.org/grpc/credentials/insecure" ) const ( SERVER_ADDRESS = "localhost:60053" NAMESPACE = "emoney" TABLENAME = "account" ID = "id" BALANCE = "balance" ) var requestHeader = pb.RequestHeader{HopLimit: 10} type TxFn func(ctx context.Context, client pb.DistributedTransactionClient, transactionId string) error func withTransaction(fn TxFn) error { ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() // Set up a connection to the server. conn, err := grpc.Dial(SERVER_ADDRESS, grpc.WithTransportCredentials(insecure.NewCredentials())) if err != nil { return err } defer conn.Close() client := pb.NewDistributedTransactionClient(conn) // Begin a transaction beginResponse, err := client.Begin(ctx, &pb.BeginRequest{RequestHeader: &requestHeader}) if err != nil { return err } transactionId := beginResponse.TransactionId // Execute the function err = fn(ctx, client, transactionId) if err != nil { // Rollback the transaction if there is an error client.Rollback(ctx, &pb.RollbackRequest{TransactionId: transactionId}) return err } // Commit the transaction _, err = client.Commit(ctx, &pb.CommitRequest{RequestHeader: &requestHeader, TransactionId: transactionId}) return err } func charge(ctx context.Context, client pb.DistributedTransactionClient, transactionId string, id string, amount int) error { partitionKey := pb.Key{Columns: []*pb.Column{{Name: ID, Value: &pb.Column_TextValue_{TextValue: &pb.Column_TextValue{Value: &id}}}}} // Retrieve the current balance for id get := pb.Get{ NamespaceName: NAMESPACE, TableName: TABLENAME, PartitionKey: &partitionKey, ClusteringKey: nil, GetType: pb.Get_GET_TYPE_GET, } getResponse, err := client.Get(ctx, &pb.GetRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Get: &get}) if err != nil { return err } // Calculate the balance balance := int32(amount) if result := getResponse.GetResult(); result != nil { for _, column := range result.GetColumns() { if column.Name == BALANCE { balance += column.GetIntValue().GetValue() break } } } // Update the balance put := pb.Put{ NamespaceName: NAMESPACE, TableName: TABLENAME, PartitionKey: &partitionKey, ClusteringKey: nil, Columns: []*pb.Column{ {Name: BALANCE, Value: &pb.Column_IntValue_{IntValue: &pb.Column_IntValue{Value: &balance}}}, }, } _, err = client.Put(ctx, &pb.PutRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Puts: []*pb.Put{&put}}) return err } func pay(ctx context.Context, client pb.DistributedTransactionClient, transactionId string, fromId string, toId string, amount int) error { fromPartitionKey := pb.Key{Columns: []*pb.Column{{Name: ID, Value: &pb.Column_TextValue_{TextValue: &pb.Column_TextValue{Value: &fromId}}}}} toPartitionKey := pb.Key{Columns: []*pb.Column{{Name: ID, Value: &pb.Column_TextValue_{TextValue: &pb.Column_TextValue{Value: &toId}}}}} // Retrieve the current balances for ids fromGet := pb.Get{ NamespaceName: NAMESPACE, TableName: TABLENAME, PartitionKey: &fromPartitionKey, ClusteringKey: nil, GetType: pb.Get_GET_TYPE_GET, } fromGetResponse, err := client.Get(ctx, &pb.GetRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Get: &fromGet}) if err != nil { return err } toGet := pb.Get{ NamespaceName: NAMESPACE, TableName: TABLENAME, PartitionKey: &toPartitionKey, ClusteringKey: nil, GetType: pb.Get_GET_TYPE_GET, } toGetResponse, err := client.Get(ctx, &pb.GetRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Get: &toGet}) if err != nil { return err } // Calculate the balances (it assumes that both accounts exist) var ( fromBalance int32 toBalance int32 ) for _, column := range fromGetResponse.GetResult().GetColumns() { if column.Name == BALANCE { fromBalance = column.GetIntValue().GetValue() break } } for _, column := range toGetResponse.GetResult().GetColumns() { if column.Name == BALANCE { toBalance = column.GetIntValue().GetValue() break } } newFromBalance := fromBalance - int32(amount) newToBalance := toBalance + int32(amount) if newFromBalance < 0 { return errors.New(fromId + " doesn't have enough balance.") } // Update the balances fromPut := pb.Put{ NamespaceName: NAMESPACE, TableName: TABLENAME, PartitionKey: &fromPartitionKey, ClusteringKey: nil, Columns: []*pb.Column{ {Name: BALANCE, Value: &pb.Column_IntValue_{IntValue: &pb.Column_IntValue{Value: &newFromBalance}}}, }, } toPut := pb.Put{ NamespaceName: NAMESPACE, TableName: TABLENAME, PartitionKey: &toPartitionKey, ClusteringKey: nil, Columns: []*pb.Column{ {Name: BALANCE, Value: &pb.Column_IntValue_{IntValue: &pb.Column_IntValue{Value: &newToBalance}}}, }, } _, err = client.Put(ctx, &pb.PutRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Puts: []*pb.Put{&fromPut, &toPut}}) return err } func getBalance(ctx context.Context, client pb.DistributedTransactionClient, transactionId string, id string) (int, error) { // Retrieve the current balance for id get := pb.Get{ NamespaceName: NAMESPACE, TableName: TABLENAME, PartitionKey: &pb.Key{Columns: []*pb.Column{{Name: ID, Value: &pb.Column_TextValue_{TextValue: &pb.Column_TextValue{Value: &id}}}}}, ClusteringKey: nil, GetType: pb.Get_GET_TYPE_GET, } getResponse, err := client.Get(ctx, &pb.GetRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Get: &get}) if err != nil { return 0, err } if getResponse.GetResult() == nil || len(getResponse.GetResult().GetColumns()) == 0 { return 0, errors.New("Account " + id + " doesn't exist.") } var balance int for _, column := range getResponse.GetResult().GetColumns() { if column.Name == BALANCE { balance = int(column.GetIntValue().GetValue()) break } } return balance, nil } func main() { var ( action = flag.String("action", "", "Action to perform: charge / pay / getBalance") fromId = flag.String("from", "", "From account (needed for pay)") toId = flag.String("to", "", "To account (needed for charge and pay)") id = flag.String("id", "", "Account id (needed for getBalance)") ) var amount int flag.IntVar(&amount, "amount", 0, "Amount to transfer (needed for charge and pay)") flag.Parse() if *action == "charge" { if *toId == "" || amount < 0 { printUsageAndExit() } err := withTransaction(func(ctx context.Context, client pb.DistributedTransactionClient, txId string) error { return charge(ctx, client, txId, *toId, amount) }) if err != nil { log.Fatalf("error: %v", err) } } else if *action == "pay" { if *toId == "" || *fromId == "" || amount < 0 { printUsageAndExit() } err := withTransaction(func(ctx context.Context, client pb.DistributedTransactionClient, txId string) error { return pay(ctx, client, txId, *fromId, *toId, amount) }) if err != nil { log.Fatalf("error: %v", err) } } else if *action == "getBalance" { if *id == "" { printUsageAndExit() } var balance int err := withTransaction(func(ctx context.Context, client pb.DistributedTransactionClient, txId string) error { var err error balance, err = getBalance(ctx, client, txId, *id) return err }) if err != nil { log.Fatalf("error: %v", err) } fmt.Println(balance) } else { fmt.Fprintln(os.Stderr, "Unknown action "+*action) printUsageAndExit() } } func printUsageAndExit() { flag.Usage() os.Exit(1) } ``` After creating the `main.go` file, you need to create the `go.mod` file by running the following commands: ```console go mod init emoney go mod tidy ``` Now, the directory structure should be as follows: ```text . ├── go.mod ├── go.sum ├── main.go └── scalardb-cluster ├── scalardb-cluster.pb.go ├── scalardb-cluster.proto └── scalardb-cluster_grpc.pb.go ``` You can then run the program as follows: - Charge `1000` to `user1`: ```console go run main.go -action charge -amount 1000 -to user1 ``` - Charge `0` to `merchant1` (Just create an account for `merchant1`): ```console go run main.go -action charge -amount 0 -to merchant1 ``` - Pay `100` from `user1` to `merchant1`: ```console go run main.go -action pay -amount 100 -from user1 -to merchant1 ``` - Get the balance of `user1`: ```console go run main.go -action getBalance -id user1 ``` - Get the balance of `merchant1`: ```console go run main.go -action getBalance -id merchant1 ``` Note that you can also use `go build` to get the binary and then run it: ```console go build ./emoney -action getBalance -id user1 ``` ## See also For other ScalarDB Cluster tutorials, see the following: - [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) - [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) - [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) - [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) - [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: - [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/getting-started-with-using-python-for-scalardb-cluster.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Using Python for ScalarDB Cluster import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; This document explains how to write gRPC client code for ScalarDB Cluster by using Python. ## Prerequisites - [Python](https://www.python.org/downloads) 3.7 or later - ScalarDB Cluster running on a Kubernetes cluster - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). ## Sample application This tutorial illustrates the process of creating an electronic money application, where money can be transferred between accounts. ## Step 1. Create `schema.json` The following is a simple example schema. Create `schema.json`, and add the following to the file: ```json { "emoney.account": { "transaction": true, "partition-key": [ "id" ], "clustering-key": [], "columns": { "id": "TEXT", "balance": "INT" } } } ``` ## Step 2. Create `database.properties` You need to create `database.properties` for the Schema Loader for ScalarDB Cluster. But first, you need to get the `EXTERNAL-IP` address of the service resource of the `LoadBalancer` service (`scalardb-cluster-envoy`). To see the `EXTERNAL-IP` address, run the following command: ```console kubectl get svc scalardb-cluster-envoy NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h ``` In this case, the `EXTERNAL-IP` address is `localhost`. Then, create `database.properties`, and add the following to the file: ```properties scalar.db.transaction_manager=cluster scalar.db.contact_points=indirect:localhost ``` To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.transaction_manager` property. In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). ## Step 3. Load a schema To load a schema via ScalarDB Cluster, you need to use the dedicated Schema Loader for ScalarDB Cluster (Schema Loader for Cluster). Using the Schema Loader for Cluster is basically the same as using the [Schema Loader for ScalarDB](../schema-loader.mdx) except the name of the JAR file is different. You can download the Schema Loader for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.18.0). After downloading the JAR file, you can run the Schema Loader for Cluster with the following command: ```console java -jar scalardb-cluster-schema-loader-3.18.0-all.jar --config database.properties -f schema.json --coordinator ``` ## Step 4. Set up a Python environment You can choose any way you like to manage your Python environment. For the purpose of this guide, we assume that your Python application is running in an environment by using `venv`. Create a working directory anywhere, and go there. Then, run the following command to activate `venv` by running the following command: ```console python3 -m venv venv source venv/bin/activate ``` Let's install the gRPC packages with the `pip` command: ```console pip install grpcio grpcio-tools ``` ## Step 5. Generate the stub code for ScalarDB Cluster gRPC To communicate with the gRPC server for ScalarDB Cluster, you will need to generate the stub code from the proto file. First, download the `scalardb-cluster.proto` file, then save it in the working directory. For ScalarDB Cluster users who have a commercial license, please [contact support](https://www.scalar-labs.com/support) if you need the `scalardb-cluster.proto` file. You can generate the stub code by running the following command: ```console python -m grpc_tools.protoc -I . --python_out=. --pyi_out=. --grpc_python_out=. scalardb-cluster.proto ``` The following files will be generated: - `scalardb_cluster_pb2.py` - `scalardb_cluster_pb2.pyi` - `scalardb_cluster_pb2_grpc.py` ## Step 6. Write a sample application The following is the sample Python application (`electronic_money.py`) that uses the stub code. This program does the same thing as the `ElectronicMoney.java` program in [Getting Started with ScalarDB](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb/). Note that you have to update the value of `SERVER_ADDRESS` based on the `EXTERNAL-IP` value of the ScalarDB Cluster `LoadBalancer` service in your environment. ```python import argparse from typing import Optional import grpc import scalardb_cluster_pb2_grpc from scalardb_cluster_pb2 import ( BeginRequest, BeginResponse, Column, CommitRequest, Get, GetRequest, GetResponse, Key, Put, PutRequest, RequestHeader, RollbackRequest, ) SERVER_ADDRESS = "localhost:60053" NAMESPACE = "emoney" TABLENAME = "account" ID = "id" BALANCE = "balance" request_header = RequestHeader(hop_limit=10) def charge(id: str, amount: int) -> None: with grpc.insecure_channel(SERVER_ADDRESS) as channel: stub = scalardb_cluster_pb2_grpc.DistributedTransactionStub(channel) begin_response: BeginResponse = stub.Begin( BeginRequest(request_header=request_header) ) transaction_id = begin_response.transaction_id try: pkey = Key( columns=[ Column( name=ID, text_value=Column.TextValue(value=id), ) ] ) # Retrieve the current balance for id get = Get( namespace_name=NAMESPACE, table_name=TABLENAME, get_type=Get.GetType.GET_TYPE_GET, partition_key=pkey, ) get_response: GetResponse = stub.Get( GetRequest( request_header=request_header, transaction_id=transaction_id, get=get, ) ) # Calculate the balance balance = amount if get_response.result.columns: balance_column = next( c for c in get_response.result.columns if c.name == BALANCE ) current = balance_column.int_value.value balance += current # Update the balance put = Put( namespace_name=NAMESPACE, table_name=TABLENAME, partition_key=pkey, columns=[ Column(name=BALANCE, int_value=Column.IntValue(value=balance)) ], ) stub.Put( PutRequest( request_header=request_header, transaction_id=transaction_id, puts=[put], ) ) # Commit the transaction stub.Commit( CommitRequest( request_header=request_header, transaction_id=transaction_id, ) ) except Exception as e: # Rollback the transaction stub.Rollback( RollbackRequest( request_header=request_header, transaction_id=transaction_id, ) ) raise e def pay(from_id: str, to_id: str, amount: int) -> None: with grpc.insecure_channel(SERVER_ADDRESS) as channel: stub = scalardb_cluster_pb2_grpc.DistributedTransactionStub(channel) begin_response: BeginResponse = stub.Begin( BeginRequest(request_header=request_header) ) transaction_id = begin_response.transaction_id try: from_pkey = Key( columns=[ Column( name=ID, text_value=Column.TextValue(value=from_id), ) ] ) to_pkey = Key( columns=[ Column( name=ID, text_value=Column.TextValue(value=to_id), ) ] ) # Retrieve the current balances for ids from_get = Get( namespace_name=NAMESPACE, table_name=TABLENAME, get_type=Get.GetType.GET_TYPE_GET, partition_key=from_pkey, ) from_get_response: GetResponse = stub.Get( GetRequest( request_header=request_header, transaction_id=transaction_id, get=from_get, ) ) to_get = Get( namespace_name=NAMESPACE, table_name=TABLENAME, get_type=Get.GetType.GET_TYPE_GET, partition_key=to_pkey, ) to_get_response: GetResponse = stub.Get( GetRequest( request_header=request_header, transaction_id=transaction_id, get=to_get, ) ) # Calculate the balances (it assumes that both accounts exist) new_from_balance = ( next( c for c in from_get_response.result.columns if c.name == BALANCE ).int_value.value - amount ) new_to_balance = ( next( c for c in to_get_response.result.columns if c.name == BALANCE ).int_value.value + amount ) if new_from_balance < 0: raise RuntimeError(from_id + " doesn't have enough balance.") # Update the balances from_put = Put( namespace_name=NAMESPACE, table_name=TABLENAME, partition_key=from_pkey, columns=[ Column( name=BALANCE, int_value=Column.IntValue(value=new_from_balance) ) ], ) to_put = Put( namespace_name=NAMESPACE, table_name=TABLENAME, partition_key=to_pkey, columns=[ Column( name=BALANCE, int_value=Column.IntValue(value=new_to_balance) ) ], ) stub.Put( PutRequest( request_header=request_header, transaction_id=transaction_id, puts=[from_put, to_put], ) ) # Commit the transaction (records are automatically recovered in case of failure) stub.Commit( CommitRequest( request_header=request_header, transaction_id=transaction_id, ) ) except Exception as e: # Rollback the transaction stub.Rollback( RollbackRequest( request_header=request_header, transaction_id=transaction_id, ) ) raise e def get_balance(id: str) -> Optional[int]: with grpc.insecure_channel(SERVER_ADDRESS) as channel: stub = scalardb_cluster_pb2_grpc.DistributedTransactionStub(channel) begin_response: BeginResponse = stub.Begin( BeginRequest(request_header=request_header) ) transaction_id = begin_response.transaction_id try: # Retrieve the current balance for id get = Get( namespace_name=NAMESPACE, table_name=TABLENAME, get_type=Get.GetType.GET_TYPE_GET, partition_key=Key( columns=[ Column( name=ID, text_value=Column.TextValue(value=id), ) ] ), ) get_response: GetResponse = stub.Get( GetRequest( request_header=request_header, transaction_id=transaction_id, get=get, ) ) balance = None if get_response.result.columns: balance_column = next( c for c in get_response.result.columns if c.name == BALANCE ) balance = balance_column.int_value.value # Commit the transaction stub.Commit( CommitRequest( request_header=request_header, transaction_id=transaction_id, ) ) return balance except Exception as e: # Rollback the transaction stub.Rollback( RollbackRequest( request_header=request_header, transaction_id=transaction_id, ) ) raise e if __name__ == "__main__": parser = argparse.ArgumentParser() subparsers = parser.add_subparsers(required=True) parser_charge = subparsers.add_parser("charge") parser_charge.add_argument("-amount", type=int, required=True) parser_charge.add_argument("-to", type=str, required=True, dest="to_id") parser_charge.set_defaults(func=lambda args: charge(args.to_id, args.amount)) parser_pay = subparsers.add_parser("pay") parser_pay.add_argument("-amount", type=int, required=True) parser_pay.add_argument("-from", type=str, required=True, dest="from_id") parser_pay.add_argument("-to", type=str, required=True, dest="to_id") parser_pay.set_defaults( func=lambda args: pay(args.from_id, args.to_id, args.amount) ) parser_get_balance = subparsers.add_parser("get-balance") parser_get_balance.add_argument("-id", type=str, required=True) parser_get_balance.set_defaults(func=lambda args: print(get_balance(args.id))) args = parser.parse_args() args.func(args) ``` You can then run the program as follows: - Charge `1000` to `user1`: ```console python electronic_money.py charge -amount 1000 -to user1 ``` - Charge `0` to `merchant1` (Just create an account for `merchant1`): ```console python electronic_money.py charge -amount 0 -to merchant1 ``` - Pay `100` from `user1` to `merchant1`: ```console python electronic_money.py pay -amount 100 -from user1 -to merchant1 ``` - Get the balance of `user1`: ```console python electronic_money.py get-balance -id user1 ``` - Get the balance of `merchant1`: ```console python electronic_money.py get-balance -id merchant1 ``` ## See also For other ScalarDB Cluster tutorials, see the following: - [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) - [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) - [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) - [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) - [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: - [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/getting-started-with-vector-search.mdx ================================================ --- tags: - Enterprise Premium - Public Preview displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB Cluster for Vector Search import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; ScalarDB Cluster provides a vector store abstraction to help applications interact with vector stores (embedding stores) in a unified way. This getting-started tutorial explains how to run vector search in ScalarDB Cluster. ## What is the vector store abstraction? ScalarDB Cluster provides an abstraction for various vector stores, similar to how it abstracts different types of databases, including relational databases, NoSQL databases, and NewSQL databases. With this vector store abstraction, you can develop applications that interact with vector stores in a unified manner, making your applications independent of specific vector store implementations and ensuring their portability. Additionally, since the integration of vector stores is built into ScalarDB Cluster, your applications can take advantage of its scalability. The current implementation of the vector store abstraction leverages [LangChain4j](https://docs.langchain4j.dev/) and supports the following vector stores and embedding models. Vector stores: - In-memory - OpenSearch - Azure Cosmos DB NoSQL - Azure AI Search - pgvector Embedding models: - In-process - Amazon Bedrock - Azure OpenAI - Google Vertex AI - OpenAI ## Why use the vector store abstraction? In the era of generative AI, one of the challenges organizations face when deploying large language models (LLMs) is enabling these models to understand their enterprise data. Retrieval-augmented generation (RAG) is a key technique used to enhance LLMs with specific enterprise knowledge. For example, to ensure that chatbots powered by LLMs provide accurate and relevant responses, companies use RAG to integrate domain-specific information from user manuals and support documents. RAG relies on vector stores, which are typically created by extracting data from databases, converting that data into vectors, and then loading those vectors. By using vector store and database abstraction in ScalarDB Cluster, you can implement the entire process seamlessly. This approach significantly simplifies the workflow and code, eliminating the need to write complex applications that depend on specific vector stores and databases. ## Tutorial This tutorial explains how to run vector search in ScalarDB Cluster. ### Prerequisites - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ### 1. Create the ScalarDB Cluster configuration file Create the following configuration file as `scalardb-cluster-node.properties`, replacing `` and `` with your ScalarDB license key and license check certificate values. For more information about the license key and certificate, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). ```yaml scalar.db.transaction.enabled=false # Enable the standalone mode scalar.db.cluster.node.standalone_mode.enabled=true # Enable the embedding feature scalar.db.embedding.enabled=true # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` Additionally, you need to add the properties for the embedding store and the embedding model to the configuration file, depending on the embedding store and the embedding model you want to use. :::note ScalarDB Cluster supports multiple embedding stores and embedding models. You can define multiple named instances and select which ones to use at runtime. To configure multiple instances, specify a comma-separated list of names in `scalar.db.embedding.stores` and `scalar.db.embedding.models`. Then, configure each instance by using `scalar.db.embedding.stores..` or `scalar.db.embedding.models..` as a prefix. For example: ```properties # Define multiple embedding stores. scalar.db.embedding.stores=store1,store2 # Configure the first store. scalar.db.embedding.stores.store1.type=in-memory # Configure the second store. scalar.db.embedding.stores.store2.type=opensearch scalar.db.embedding.stores.store2.opensearch.server_url= ... # Define multiple embedding models. scalar.db.embedding.models=model1,model2 # Configure the first model. scalar.db.embedding.models.model1.type=in-process # Configure the second model. scalar.db.embedding.models.model2.type=open-ai scalar.db.embedding.models.model2.open-ai.api_key= ... ``` ::: Select the embedding store that you want to use, and follow the instructions to configure it. The in-memory embedding store is a basic in-memory implementation. This embedding store is useful for fast prototyping and simple use cases. To use the in-memory embedding store, add the following properties to the configuration file: ```properties # Define embedding store names (comma-separated list for multiple stores). scalar.db.embedding.stores=my_store # Configure the embedding store type. scalar.db.embedding.stores.my_store.type=in-memory ``` The OpenSearch embedding store is an embedding store that uses OpenSearch as the backend. Select whether your OpenSearch implementation is running locally or running on AWS, and follow the instructions to configure it. For OpenSearch clusters that are running locally and are reachable on the network, add the following properties to the configuration file: ```properties # Define embedding store names (comma-separated list for multiple stores). scalar.db.embedding.stores=my_store # Configure the embedding store type. scalar.db.embedding.stores.my_store.type=opensearch # OpenSearch Server URL. scalar.db.embedding.stores.my_store.opensearch.server_url= # OpenSearch API key (optional). scalar.db.embedding.stores.my_store.opensearch.api_key= # OpenSearch username (optional). scalar.db.embedding.stores.my_store.opensearch.user_name= # OpenSearch password (optional). scalar.db.embedding.stores.my_store.opensearch.password= # OpenSearch index name. scalar.db.embedding.stores.my_store.opensearch.index_name= ``` For OpenSearch clusters that are running as a fully managed service on AWS, add the following properties to the configuration file: ```properties # Define embedding store names (comma-separated list for multiple stores). scalar.db.embedding.stores=my_store # Configure the embedding store type. scalar.db.embedding.stores.my_store.type=opensearch # OpenSearch Server URL. scalar.db.embedding.stores.my_store.opensearch.server_url= # The AWS signing service name, one of `es` (Amazon OpenSearch) or `aoss` (Amazon OpenSearch Serverless). scalar.db.embedding.stores.my_store.opensearch.service_name= # The AWS region for which requests will be signed. This should typically match the region in `server_url`. scalar.db.embedding.stores.my_store.opensearch.region= # The AWS access key ID. scalar.db.embedding.stores.my_store.opensearch.access_key_id= # The AWS secret access key. scalar.db.embedding.stores.my_store.opensearch.secret_access_key= # OpenSearch index name. scalar.db.embedding.stores.my_store.opensearch.index_name= ``` The Azure Cosmos DB for NoSQL embedding store is an embedding store that uses Azure Cosmos DB as the backend. To use the Azure Cosmos DB for NoSQL embedding store, add the following properties to the configuration file: ```properties # Define embedding store names (comma-separated list for multiple stores). scalar.db.embedding.stores=my_store # Configure the embedding store type. scalar.db.embedding.stores.my_store.type=azure-cosmos-nosql # The Azure Cosmos DB endpoint that the SDK will connect to. scalar.db.embedding.stores.my_store.azure-cosmos-nosql.endpoint= # A master key used to perform authentication for accessing resources. A read-only key can also be used only for read-only operations. scalar.db.embedding.stores.my_store.azure-cosmos-nosql.key= # The database name to be used. scalar.db.embedding.stores.my_store.azure-cosmos-nosql.database_name= # The container name to be used. scalar.db.embedding.stores.my_store.azure-cosmos-nosql.container_name= # The dimensions of the embeddings. scalar.db.embedding.stores.my_store.azure-cosmos-nosql.dimensions= ``` The Azure AI Search embedding store is an embedding store that uses Azure AI Search as the backend. To use the Azure AI Search embedding store, add the following properties to the configuration file: ```properties # Define embedding store names (comma-separated list for multiple stores). scalar.db.embedding.stores=my_store # Configure the embedding store type. scalar.db.embedding.stores.my_store.type=azure-ai-search # The Azure AI Search endpoint. scalar.db.embedding.stores.my_store.azure-ai-search.endpoint= # The Azure AI Search API key. scalar.db.embedding.stores.my_store.azure-ai-search.api_key= # The name of the index to be used. If no index is provided, the default index name will be used. scalar.db.embedding.stores.my_store.azure-ai-search.index_name= # The dimensions of the embeddings. scalar.db.embedding.stores.my_store.azure-ai-search.dimensions= ``` The pgvector embedding store is an embedding store that uses pgvector, which is a Postgres extension for vector similarity search, as the backend. To use the pgvector embedding store, add the following properties to the configuration file: ```properties # Define embedding store names (comma-separated list for multiple stores). scalar.db.embedding.stores=my_store # Configure the embedding store type. scalar.db.embedding.stores.my_store.type=pgvector # The database host. scalar.db.embedding.stores.my_store.pgvector.host= # The database port. scalar.db.embedding.stores.my_store.pgvector.port= # The database user. scalar.db.embedding.stores.my_store.pgvector.user= # The database password. scalar.db.embedding.stores.my_store.pgvector.password= # The database name. scalar.db.embedding.stores.my_store.pgvector.database= # The table name. scalar.db.embedding.stores.my_store.pgvector.table=
# The dimensions of the embeddings. scalar.db.embedding.stores.my_store.pgvector.dimensions= ``` Select the embedding model that you want to use, and follow the instructions to configure it. The in-process embedding model is a local embedding model powered by [ONNX runtime](https://onnxruntime.ai/docs/get-started/with-java.html) and is running in the ScalarDB Cluster process. This embedding model is useful for fast prototyping and simple use cases. To use the in-process embedding model, add the following properties to the configuration file: ```properties # Define embedding model names (comma-separated list for multiple models). scalar.db.embedding.models=my_model # Configure the embedding model type. scalar.db.embedding.models.my_model.type=in-process ``` The Amazon Bedrock embedding model is an embedding model that uses Amazon Bedrock as the backend. To use the Amazon Bedrock embedding model, add the following properties to the configuration file: ```properties # Define embedding model names (comma-separated list for multiple models). scalar.db.embedding.models=my_model # Configure the embedding model type. scalar.db.embedding.models.my_model.type=bedrock-titan # The AWS region for which requests will be signed. scalar.db.embedding.models.my_model.bedrock-titan.region= # The AWS access key ID. scalar.db.embedding.models.my_model.bedrock-titan.access_key_id= # The AWS secret access key. scalar.db.embedding.models.my_model.bedrock-titan.secret_access_key= # The model. Either `amazon.titan-embed-text-v1` or `amazon.titan-embed-text-v2:0`. scalar.db.embedding.models.my_model.bedrock-titan.model= # The dimensions. scalar.db.embedding.models.my_model.bedrock-titan.dimensions= ``` The Azure OpenAI embedding model is an embedding model that uses Azure OpenAI as the backend. To use the Azure OpenAI embedding model, add the following properties to the configuration file: ```properties # Define embedding model names (comma-separated list for multiple models). scalar.db.embedding.models=my_model # Configure the embedding model type. scalar.db.embedding.models.my_model.type=azure-open-ai # The Azure OpenAI endpoint. scalar.db.embedding.models.my_model.azure-open-ai.endpoint= # The Azure OpenAI API key. scalar.db.embedding.models.my_model.azure-open-ai.api_key= # The deployment name in Azure OpenAI. scalar.db.embedding.models.my_model.azure-open-ai.deployment_name= # The dimensions. scalar.db.embedding.models.my_model.azure-open-ai.dimensions= ``` The Google Vertex AI embedding model is an embedding model that uses Google Vertex AI as the backend. To use the Google Vertex AI embedding model, add the following properties to the configuration file: ```properties # Define embedding model names (comma-separated list for multiple models). scalar.db.embedding.models=my_model # Configure the embedding model type. scalar.db.embedding.models.my_model.type=vertex-ai # The Google Cloud project. scalar.db.embedding.models.my_model.vertex-ai.project= # The Google Cloud location. scalar.db.embedding.models.my_model.vertex-ai.location= # The endpoint. scalar.db.embedding.models.my_model.vertex-ai.endpoint= # The publisher. scalar.db.embedding.models.my_model.vertex-ai.publisher= # The model name. scalar.db.embedding.models.my_model.vertex-ai.model_name= # The output dimensionality. scalar.db.embedding.models.my_model.vertex-ai.output_dimensionality= ``` The OpenAI embedding model is an embedding model that uses OpenAI as the backend. To use the OpenAI embedding model, add the following properties to the configuration file: ```properties # Define embedding model names (comma-separated list for multiple models). scalar.db.embedding.models=my_model # Configure the embedding model type. scalar.db.embedding.models.my_model.type=open-ai # The OpenAI API key. scalar.db.embedding.models.my_model.open-ai.api_key= # The model name. scalar.db.embedding.models.my_model.open-ai.model_name= # The base URL. scalar.db.embedding.models.my_model.open-ai.base_url= # The organization ID. scalar.db.embedding.models.my_model.open-ai.organization_id= # The dimensions. scalar.db.embedding.models.my_model.open-ai.dimensions= # The user. scalar.db.embedding.models.my_model.open-ai.user= ``` ### 2. Create the Docker Compose file Create the following configuration file as `docker-compose.yaml`. ```yaml services: scalardb-cluster-standalone: container_name: "scalardb-cluster-node" image: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium:3.18.0" ports: - 60053:60053 - 9080:9080 volumes: - ./scalardb-cluster-node.properties:/scalardb-cluster/node/scalardb-cluster-node.properties ``` ### 3. Start ScalarDB Cluster Run the following command to start ScalarDB Cluster in standalone mode. ```console docker compose up -d ``` It may take a few minutes for ScalarDB Cluster to fully start. ### 4. Add the Java Client SDK for the embedding store abstraction to your project The ScalarDB Cluster Embedding Java Client SDK library is available on the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb-cluster-embedding-java-client-sdk). You can add the library as a build dependency to your application by using Gradle or Maven. Select your build tool, and follow the instructions to add the build dependency for the ScalarDB Cluster Embedding Java Client SDK to your application. To add the build dependency for the ScalarDB Cluster Embedding Java Client SDK by using Gradle, add the following to `build.gradle` in your application: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-cluster-embedding-java-client-sdk:3.18.0' } ``` To add the build dependency for the ScalarDB Cluster Embedding Java Client SDK by using Maven, add the following to `pom.xml` in your application: ```xml com.scalar-labs scalardb-cluster-embedding-java-client-sdk 3.18.0 ``` ### 5. Run the sample code Create a new Java class and add the following code to run the sample code. ```java try (ScalarDbEmbeddingClientFactory scalarDbEmbeddingClientFactory = ScalarDbEmbeddingClientFactory.builder() .withProperty("scalar.db.embedding.client.contact_points", "indirect:localhost") .withProperty("scalar.db.embedding.client.store", "my_store") .withProperty("scalar.db.embedding.client.model", "my_model") .build()) { // Create an embedding store and an embedding model. EmbeddingStore scalarDbEmbeddingStore = scalarDbEmbeddingClientFactory.createEmbeddingStore(); EmbeddingModel scalarDbEmbeddingModel = scalarDbEmbeddingClientFactory.createEmbeddingModel(); // Add embeddings to the embedding store. TextSegment segment1 = TextSegment.from("I like football."); Embedding embedding1 = scalarDbEmbeddingModel.embed(segment1).content(); scalarDbEmbeddingStore.add(embedding1, segment1); TextSegment segment2 = TextSegment.from("The weather is good today."); Embedding embedding2 = scalarDbEmbeddingModel.embed(segment2).content(); scalarDbEmbeddingStore.add(embedding2, segment2); // Search for embeddings. Embedding queryEmbedding = scalarDbEmbeddingModel.embed("What is your favourite sport?").content(); EmbeddingSearchResult result = scalarDbEmbeddingStore.search( EmbeddingSearchRequest.builder() .queryEmbedding(queryEmbedding) .maxResults(1) .build()); // Print the search result. List> matches = result.matches(); EmbeddingMatch embeddingMatch = matches.get(0); System.out.println(embeddingMatch.embedded().text()); System.out.println(embeddingMatch.score()); } ``` This sample code demonstrates how to create an embedding store and an embedding model, add embeddings to the embedding store, and search for embeddings. Except for the part of the code that creates an embedding store and an embedding model, the usage is the same as LangChain4j. For more information about LangChain4j, see the following: - [LangChain4j](https://docs.langchain4j.dev/) - [Embedding Model](https://docs.langchain4j.dev/tutorials/rag#embedding-model) - [Embedding Store](https://docs.langchain4j.dev/tutorials/rag#embedding-store) #### About `ScalarDbEmbeddingClientFactory` As shown in the code snippet, the `ScalarDbEmbeddingClientFactory` class provides a builder to create an instance of the factory. The builder allows you to set properties for the factory. In this example, the `withProperty()` method is used to set the contact points, store name, and model name for the factory as follows: ```java ScalarDbEmbeddingClientFactory scalarDbEmbeddingClientFactory = ScalarDbEmbeddingClientFactory.builder() .withProperty("scalar.db.embedding.client.contact_points", "indirect:localhost") .withProperty("scalar.db.embedding.client.store", "my_store") .withProperty("scalar.db.embedding.client.model", "my_model") .build(); ``` The `scalar.db.embedding.client.store` and `scalar.db.embedding.client.model` properties specify which named embedding store and embedding model instances to use, as defined in the server configuration. You can also set a properties file by using the `withPropertiesFile()` method. If you want to interact with multiple vector stores, create multiple instances of the factory. Then, you can create an embedding store and an embedding model by using the factory as follows: ```java EmbeddingStore scalarDbEmbeddingStore = scalarDbEmbeddingClientFactory.createEmbeddingStore(); EmbeddingModel scalarDbEmbeddingModel = scalarDbEmbeddingClientFactory.createEmbeddingModel(); ``` Their methods internally connect to ScalarDB Cluster, which interacts with the embedding store by using the embedding model, both of which are specified in the configuration. You should reuse the `scalarDbEmbeddingStore` and `scalarDbEmbeddingModel` instances to interact with vector stores in an application. :::note The `ScalarDbEmbeddingClientFactory` instance should be closed after use to release the connection to ScalarDB Cluster. ::: ## Additional details The vector search feature is currently in Public Preview. For more details, please [contact us](https://www.scalar-labs.com/contact). - [Javadoc](https://javadoc.io/doc/com.scalar-labs/scalardb-cluster-embedding-java-client-sdk/3.18.0/index.html) ================================================ FILE: docs/scalardb-cluster/index.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster ScalarDB Cluster is a clustering solution for [ScalarDB](../overview.mdx) that consists of a set of cluster nodes, each of which provides ScalarDB functionality. Each cluster node has a routing mechanism that directs transaction requests to the appropriate cluster node within the cluster. ## Why ScalarDB Cluster? When executing a transaction that spans multiple client requests, such as in microservice transactions, all requests for the transaction must be processed on the same server due to the stateful nature of transaction processing. However, in a distributed environment, routing requests to the same server isn't straightforward because a service typically runs on multiple servers (or hosts) for scalability and availability. In this scenario, all requests within a transaction must be routed to the same server, while different transactions should be distributed to ensure load balancing. To address this challenge, a routing mechanism such as session affinity (also known as sticky sessions) needs to be configured. This strategy ensures that requests within a transaction are consistently routed to the same server. Alternatively, you can leverage a bidirectional-streaming RPC by using gRPC. However, it's important to note that implementing these configurations typically requires significant time and effort. In addition, specific configuration adjustments may be required depending on the load balancer product you are using. For more details on this topic, see [Request routing in transactions with a two-phase commit interface](../two-phase-commit-transactions.mdx#request-routing-in-transactions-with-a-two-phase-commit-interface). ScalarDB Cluster addresses this issue by providing a routing mechanism capable of directing requests to the appropriate cluster node within the cluster. Thus, when a cluster node receives a request, the node can route that request to the correct cluster node in the cluster. ## Architecture ScalarDB Cluster consists of a set of cluster nodes, each equipped with ScalarDB functionality. By using this solution, each cluster node can execute transactions independently. A notable feature of ScalarDB Cluster is the distribution of transaction requests by using a routing mechanism. When a cluster node receives a request, the node determines whether it's the appropriate cluster node to process the request. If it's not the appropriate node, the node routes the request to the appropriate cluster node within the cluster. To determine the appropriate cluster node, ScalarDB Cluster uses a consistent hashing algorithm. Membership management plays a critical role in ScalarDB Cluster. When a cluster node either joins or leaves the cluster, the configuration of the cluster is automatically adjusted to reflect this change. ScalarDB Cluster currently retrieves membership information by using the Kubernetes API. :::note Currently, ScalarDB Cluster supports running on Kubernetes only. ::: ![ScalarDB Cluster architecture](images/scalardb-cluster-architecture.png) ## Getting started Before you start the tutorials, you need to set up ScalarDB Cluster. To set up ScalarDB Cluster, see [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). For tutorials on getting started with ScalarDB Cluster, see the following: * [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) * [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) * [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) * [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) * [Getting Started with ScalarDB Cluster via .NET](getting-started-with-scalardb-cluster-dotnet.mdx) * [Getting Started with ScalarDB Cluster SQL via .NET](getting-started-with-scalardb-cluster-sql-dotnet.mdx) * [Getting Started with ScalarDB Cluster SQL via .NET and LINQ](getting-started-with-scalardb-cluster-sql-linq.mdx) ## References For details about the ScalarDB Cluster Helm Chart, refer to the following: * [ScalarDB Cluster Helm Chart](https://github.com/scalar-labs/helm-charts/tree/main/charts/scalardb-cluster) * [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx) * [How to deploy ScalarDB Cluster](../helm-charts/how-to-deploy-scalardb-cluster.mdx) For details about the configurations for ScalarDB Cluster, refer to the following: * [ScalarDB Cluster Configurations](scalardb-cluster-configurations.mdx) For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: * [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) For details about the ScalarDB Cluster gRPC API, refer to the following: * [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) * [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) ================================================ FILE: docs/scalardb-cluster/remote-replication.mdx ================================================ --- tags: - Enterprise Premium - Private Preview displayed_sidebar: docsEnglish --- # Replicate Data for High Availability ScalarDB Cluster can replicate its managed data to remote sites for high availability and workload distribution. The remote replication feature provides near-real-time replication of write operations from a primary site to one or more backup sites. This feature ensures business continuity by enabling failover to a backup site in the event of disasters or critical failures affecting the primary site. Additionally, the backup sites can function as read replicas, helping to offload analytical queries, reporting, and business intelligence workloads. ## What is remote replication in ScalarDB? Remote replication in ScalarDB uses a hybrid approach, combining synchronous and asynchronous replication. This ensures zero data loss (a recovery point objective, or RPO, of zero) while minimizing performance impact at the primary site. The recovery time objective (RTO) can be flexibly adjusted by controlling the amount of computing resources. This feature is built on top of ScalarDB Cluster, making it cloud-agnostic and database-agnostic. This allows replication from one database in one cloud vendor to another, possibly different kind of database in a different cloud vendor. ## Key benefits Remote replication provides several key advantages: - Guarantees zero data loss (RPO of 0) for all committed transactions. - Minimizes performance impact through the combination of synchronous and asynchronous processing. - Enables backup site deployment in different regions, availability zones, or data centers from the primary site. - Supports replication between different cloud service providers and database types. - Provides built-in crash tolerance and automatic recovery mechanisms. ## Architecture overview The following diagram illustrates the remote replication architecture: ```mermaid flowchart LR subgraph "Primary site" App["Client applications"] subgraph PrimaryCluster1["ScalarDB Cluster node 1"] LogWriter1["LogWriter"] end subgraph PrimaryCluster2["ScalarDB Cluster node 2"] LogWriter2["LogWriter"] end PrimaryDB["Primary site database"] App --> PrimaryCluster1 App --> PrimaryCluster2 PrimaryCluster1 --> PrimaryDB PrimaryCluster2 --> PrimaryDB end subgraph "Shared components" CoordDB["Coordinator database"] ReplDB["Replication database"] end subgraph "Backup site" subgraph BackupCluster1["ScalarDB Cluster node 1"] LogApplier1["LogApplier"] end subgraph BackupCluster2["ScalarDB Cluster node 2"] LogApplier2["LogApplier"] end BackupDB["Backup site database"] ReplRecordTables["Replication record metadata tables"] end LogWriter1 --> ReplDB PrimaryCluster1 --> CoordDB LogApplier1 --> CoordDB LogApplier1 --> ReplDB LogApplier1 --> BackupDB LogApplier1 --> ReplRecordTables LogApplier2 --> BackupDB LogApplier2 --> ReplRecordTables classDef primary fill:#e1f5fe classDef shared fill:#f3e5f5 classDef backup fill:#e8f5e8 class App,PrimaryCluster1,PrimaryCluster2,LogWriter1,LogWriter2,PrimaryDB primary class CoordDB,ReplDB shared class BackupCluster1,BackupCluster2,LogApplier1,LogApplier2,BackupDB,ReplRecordTables backup %% Invisible connections to force left-to-right layout of subgraphs %% These links are made invisible using linkStyle below PrimaryDB ~~~ CoordDB ReplDB ~~~ BackupDB ``` Remote replication consists of the components and tools listed in this section. ### Primary site components The primary site comprises three components: primary site database, client applications, and ScalarDB Cluster nodes. Each runs as follows: - A primary site database contains the application tables used by client applications via ScalarDB Cluster. - Client applications perform database operations. - ScalarDB Cluster nodes manage transaction states in the Coordinator database and use a module called LogWriter to capture transaction operations and write them to the replication database. ### Shared components (between primary and backup sites) Two components span the primary and backup sites: the Coordinator database and the replication database. These databases can be hosted in single database instances. However, because they maintain transaction information, it is required to replicate them across several sites as follows: - A Coordinator database manages transaction states across the sites in a highly available way. - A replication database stores transaction groups containing write operations from the primary site in a highly available way. ### Backup site components The backup site comprises two components: backup site database and ScalarDB Cluster nodes. Each runs as follows: - A backup site database contains the same application tables as the primary site. It also contains replication record metadata tables, which are internal tables that track replication metadata and unapplied operations. These tables are located in the same namespaces as the application tables (with the suffix `__records` by default). - ScalarDB Cluster nodes use a module called LogApplier to apply replicated data. Specifically, LogApplier checks the Coordinator database for transaction states, reads and removes write operations from the replication database, calculates dependencies by using the replication record metadata tables, and applies operations to the backup site tables. ### Administrative tools Remote replication uses the following administrative tools: Schema Loader and Replication CLI. Each runs as follows: - Schema Loader creates the replication tables in the replication database by using the `--replication-tables` option via ScalarDB Cluster endpoints. - Replication CLI monitors and administers ScalarDB Cluster nodes that replicate data through remote replication. ## How remote replication works Remote replication employs a hybrid approach that combines synchronous and asynchronous replication to ensure zero data loss (RPO = 0) with minimal impact on performance at the primary site. It comprises two phases, the synchronous phase and the asynchronous phase, as follows: - In the synchronous phase, write operations are copied from the primary site to the replication database during transaction commit. - In the asynchronous phase, these operations are processed from the replication database and applied to the backup site tables. The replication process follows these steps: 1. When a transaction commits on the primary site, the LogWriter captures all write operations and stores them in the replication database. 2. The LogApplier on the backup site continuously scans the replication database for new transaction data. 3. The LogApplier checks the Coordinator database to verify transaction completion. 4. The LogApplier orders and applies write operations based on transaction dependencies at the record level by using the replication record metadata tables. 5. The LogApplier applies the processed operations to the backup site tables, with the replication record metadata tables updated to track progress. ## Limitations and characteristics This section describes the limitations and characteristics of remote replication. ### Private preview limitations The current private preview version has the following limitations, but they are going to be relaxed when it becomes public preview or general availability (GA): - The specification may be changed in future releases. - Multiple backup sites are not supported. - Starting remote replication with restored data is not supported. Both primary and backup sites need to start from the beginning. - This feature does not work with the one-phase commit optimization. This optimization must be disabled for replication to function properly. - Creating the replication tables via ScalarDB SQL is not supported. - The combination of the [encryption feature](./encrypt-data-at-rest.mdx) and remote replication is not officially supported because it has not been verified. ### Architectural limitations Remote replication has the following architectural limitations, which are inherently challenging to relax due to the architecture: - Only [transactions in read-only mode](../api-guide.mdx#begin-or-start-a-transaction-in-read-only-mode) with the [read-committed isolation level](../consensus-commit.mdx#isolation-levels) are permitted on backup sites until failover. - DDL operations are not replicated. Schema changes must be applied manually to both primary and backup sites. - You cannot use the two-phase commit interface if this feature is enabled. - There may be a slight performance impact on the primary site, depending on replication database specifications and configurations. ## Failure handling Remote replication is designed to handle various failure scenarios gracefully, ensuring data consistency and system availability where possible. ### Primary site failure When the primary site becomes unavailable, all committed transactions are safely stored in the replication database or the backup site databases, ensuring zero data loss. The backup site can be promoted to serve application traffic once the LogApplier has processed all pending write operations from the replication database. ### Backup site failure If the backup site fails, the primary site continues operating normally without any disruption to application traffic. However, write operations will accumulate in the replication database since LogApplier cannot process them. During extended backup site downtime, there is a risk that the replication database may reach capacity limits, so monitoring is important. Once the backup site is restored and LogApplier resumes, replication will catch up with the accumulated operations. ### Coordinator database failure The impact of Coordinator database failure depends on its scope. In non-critical failures (such as a single region failure in a multi-region setup), remote replication continues operating normally. However, if the Coordinator database becomes completely unavailable, both ScalarDB Cluster in the primary site and LogApplier are affected: ScalarDB Cluster in the primary site cannot commit new transactions, and LogApplier cannot verify transaction states to proceed with replication. ### Replication database failure Similar to Coordinator database failures, the impact depends on the scope of the failure. Non-critical failures (like single region failures in multi-region deployments) don't affect the remote replication operation. However, complete replication database unavailability prevents ScalarDB Cluster in the primary site from committing new transactions and blocks LogApplier from reading write operations, effectively halting both primary operations and replication processing. ## Configuration This section describes the configuration options for remote replication in ScalarDB Cluster. ### Base replication configurations These configurations apply to the overall replication setup: #### `partition_count` - **Field:** `scalar.db.replication.partition_count` - **Description:** Number of partitions for the `transaction_groups` table. The tables in the replication database are partitioned for performance and scalability, and write operations are distributed evenly across partitions. This field must be identical between primary and backup sites. Changing the partition count requires restarting the ScalarDB Clusters in both sites. - **Default value:** `256` #### `repl_db.namespace` - **Field:** `scalar.db.replication.repl_db.namespace` - **Description:** Namespace name of replication tables. This field must be identical between primary and backup sites. - **Default value:** `replication` #### `record_table_suffix` - **Field:** `scalar.db.replication.record_table_suffix` - **Description:** Suffix for replication record metadata tables. - **Default value:** `__records` ### LogWriter configurations (primary site) LogWriter configurations control how write operations are captured and stored in the replication database during transaction commits. #### `log_writer.enabled` - **Field:** `scalar.db.replication.log_writer.enabled` - **Description:** Enable or disable LogWriter functionality. - **Default value:** `false` #### `log_writer.compression_type` - **Field:** `scalar.db.replication.log_writer.compression_type` - **Description:** Compression type for stored write operations in the replication database. Available values: `NONE`, `GZIP`. - **Default value:** `GZIP` #### `log_writer.group_commit.retention.time_millis` - **Field:** `scalar.db.replication.log_writer.group_commit.retention.time_millis` - **Description:** Maximum time to wait before committing a transaction group for the replication database. - **Default value:** `100` #### `log_writer.group_commit.retention.values` - **Field:** `scalar.db.replication.log_writer.group_commit.retention.values` - **Description:** Maximum number of transactions to batch together for the replication database. - **Default value:** `32` #### `log_writer.group_commit.timeout_check_interval_millis` - **Field:** `scalar.db.replication.log_writer.group_commit.timeout_check_interval_millis` - **Description:** Interval for checking group commit timeouts for the replication database. - **Default value:** `20` #### `log_writer.group_commit.max_thread_pool_size` - **Field:** `scalar.db.replication.log_writer.group_commit.max_thread_pool_size` - **Description:** Maximum thread pool size for group commit processing for the replication database. - **Default value:** `4096` ### LogApplier configurations (backup site) LogApplier configurations control how replication data is processed and applied to the backup site tables. #### `log_applier.enabled` - **Field:** `scalar.db.replication.log_applier.enabled` - **Description:** Enable or disable LogApplier functionality. - **Default value:** `false` #### `log_applier.transaction.expiration_millis` - **Field:** `scalar.db.replication.log_applier.transaction.expiration_millis` - **Description:** Transaction expiration time for cleanup. - **Default value:** `30000` #### `log_applier.transaction_group_scanner.threads` - **Field:** `scalar.db.replication.log_applier.transaction_group_scanner.threads` - **Description:** Number of scanner threads for transaction groups in the replication database. - **Default value:** `16` #### `log_applier.transaction_group_scanner.fetch_size` - **Field:** `scalar.db.replication.log_applier.transaction_group_scanner.fetch_size` - **Description:** Number of transaction group records to fetch in each scan operation from the replication database. - **Default value:** `32` #### `log_applier.transaction_group_scanner.wait_millis` - **Field:** `scalar.db.replication.log_applier.transaction_group_scanner.wait_millis` - **Description:** Wait time in milliseconds between scans of transaction groups in the replication database. Increasing this value reduces scan frequency but may increase replication latency. - **Default value:** `1000` #### `log_applier.transaction_group_scanner.dedup.expiration_millis` - **Field:** `scalar.db.replication.log_applier.transaction_group_scanner.dedup.expiration_millis` - **Description:** Expiration time for transaction group deduplication cache. LogApplier uses this cache to avoid reprocessing the same transaction groups from the replication database. - **Default value:** `10000` #### `log_applier.transaction_handler.threads` - **Field:** `scalar.db.replication.log_applier.transaction_handler.threads` - **Description:** Number of threads for LogApplier transaction handling. Increasing this value may improve replication performance but increases resource consumption. - **Default value:** `128` #### `log_applier.max_record_version` - **Field:** `scalar.db.replication.log_applier.max_record_version` - **Description:** Maximum record version to prevent integer overflow (for development use). - **Default value:** `Integer.MAX_VALUE - 1000` #### `log_applier.write_operation.expiration_millis` - **Field:** `scalar.db.replication.log_applier.write_operation.expiration_millis` - **Description:** Write operation expiration time for garbage collection. - **Default value:** `86400000` (1 day) #### `log_applier.replication_status_service.threads` - **Field:** `scalar.db.replication.log_applier.replication_status_service.threads` - **Description:** Number of threads for replication status service. - **Default value:** `16` #### `log_applier.coordinator_state_cache.expiration_millis` - **Field:** `scalar.db.replication.log_applier.coordinator_state_cache.expiration_millis` - **Description:** Expiration time for coordinator state cache. - **Default value:** `30000` #### `log_applier.coordinator_state_cache.size` - **Field:** `scalar.db.replication.log_applier.coordinator_state_cache.size` - **Description:** Maximum size of the coordinator state cache. - **Default value:** `1000` ## Important configuration notes This section describes capacity planning and failure resilience, database selection, and performance tuning. ### Partition count planning for capacity planning and failure resilience When planning the partition count ([`scalar.db.replication.partition_count`](#partition_count)), consider that partitions may have capacity limits depending on the underlying database. For example, [Cosmos DB partitions can contain a maximum of 20 GB of data](https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits). If the backup site is down for extended periods, many write operations accumulate in the replication database, potentially reaching partition limits. Plan the partition count based on estimated write operations from the primary site and potential backup site downtime duration. Increasing partition count may mitigate capacity risks, but could require more LogApplier resources. ### Database selection If Coordinator and replication databases (shared components) are unavailable, the primary site cannot commit transactions, and LogApplier cannot process replication data. Therefore, select databases ([`scalar.db.multi_storage.storages.coordinator.*`](../configurations.mdx#multi-storage-support) and [`scalar.db.multi_storage.storages.replication.*`](../configurations.mdx#multi-storage-support)) that are highly available to meet your disaster tolerance requirements and that must be replicated synchronously (for example, Cosmos DB with Strong consistency) across multiple AZs or regions, depending on where you replicate your data. ### Performance tuning When using remote replication, you may need to tune performance based on your workload characteristics and infrastructure setup. This section describes key configuration parameters for optimizing replication performance. #### LogWriter performance tuning For the primary site, tune group commit settings ([`scalar.db.replication.log_writer.group_commit.retention.time_millis`](#log_writergroup_commitretentiontime_millis) and [`scalar.db.replication.log_writer.group_commit.retention.values`](#log_writergroup_commitretentionvalues)) to balance latency and throughput based on your workload characteristics. #### LogApplier performance tuning For the backup site, tune thread counts ([`scalar.db.replication.log_applier.transaction_group_scanner.threads`](#log_appliertransaction_group_scannerthreads) and [`scalar.db.replication.log_applier.transaction_handler.threads`](#log_appliertransaction_handlerthreads)) to balance parallel processing performance and resource consumption. #### High-latency database environments Multi-region databases may have higher latency. Consider using [group commit configurations](../api-guide.mdx#group-commit-for-the-coordinator-table) to improve throughput on high-latency Coordinator databases. The Coordinator group commit feature batches multiple transactions together, reducing the impact of network latency on overall throughput. For replication database tuning, see the [LogWriter performance tuning](#logwriter-performance-tuning) section above. ## Tutorial This section walks you through setting up remote replication for ScalarDB Cluster. :::tip If you're new to ScalarDB Cluster, you may want to familiarize yourself with these topics first: - [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) - [How to Deploy ScalarDB Cluster Locally](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx) - [ScalarDB Cluster Configurations](scalardb-cluster-configurations.mdx) - [Run Transactions Through ScalarDB Cluster](run-transactions-through-scalardb-cluster.mdx) ::: ### Prerequisites Before setting up remote replication, ensure you have set up the following in your environment: - Kubernetes environment (for example, Minikube) - `helm` command for Kubernetes package management - `kubectl` command for Kubernetes cluster management ### Deployment topology This guide demonstrates a two-site setup, which can be deployed within a single Kubernetes cluster: - The primary site is where your application is running. - The backup site is where replicated data is stored for disaster recovery. :::note Kubernetes contexts If your primary and backup sites are deployed in different Kubernetes clusters, you'll need to switch kubectl contexts between sites. Use `kubectl config get-contexts` to list available contexts and `kubectl config use-context ` to switch between them. ::: ### Step 1: Set up the required databases Create the primary site, backup site, Coordinator, and Replication databases that are needed for the remote replication deployment. #### Primary and backup site databases Set up your primary and backup site databases by using any [ScalarDB-supported database](../requirements.mdx#databases). #### Coordinator and Replication databases (shared components) Set up these databases by using any ScalarDB-supported database based on your disaster tolerance requirements. Use a highly available database if needed. ##### Example database setups For this tutorial, you can use any ScalarDB-supported database such as PostgreSQL or MySQL. For production deployments, consider highly available databases (for example, Cosmos DB with multi-region configuration). ### Step 2: Configure the primary site In this step, you'll update the ScalarDB Cluster configuration, deploy a primary site cluster, and create primary site tables. #### 2.1 Update primary site ScalarDB Cluster configuration Add the remote replication configurations to your primary site ScalarDB Cluster that uses a multi-storage setup. The following is an example configuration: ```properties # Multi-storage configuration scalar.db.storage=multi-storage scalar.db.multi_storage.storages=default,coord,repl scalar.db.multi_storage.default_storage=default scalar.db.multi_storage.namespace_mapping=coordinator:coord,replication:repl # Primary site database scalar.db.multi_storage.storages.default.storage=jdbc scalar.db.multi_storage.storages.default.contact_points=jdbc:postgresql://primary-db:5432/ scalar.db.multi_storage.storages.default.username=postgres scalar.db.multi_storage.storages.default.password=postgres # Coordinator database (shared component) scalar.db.multi_storage.storages.coord.storage=jdbc scalar.db.multi_storage.storages.coord.contact_points=jdbc:postgresql://coordinator-db:5432/ scalar.db.multi_storage.storages.coord.username=postgres scalar.db.multi_storage.storages.coord.password=postgres # Replication database (shared component) scalar.db.multi_storage.storages.repl.storage=jdbc scalar.db.multi_storage.storages.repl.contact_points=jdbc:postgresql://replication-db:5432/ scalar.db.multi_storage.storages.repl.username=postgres scalar.db.multi_storage.storages.repl.password=postgres # Enable LogWriter scalar.db.replication.log_writer.enabled=true # Enable SQL interface (if you want to test with SQL CLI) scalar.db.sql.enabled=true ``` #### 2.2 Deploy a primary site cluster Deploy a primary site ScalarDB Cluster with the remote replication configuration: ```bash helm install scalar-labs/scalardb-cluster \ -f \ -n ``` Replace: - ``: A name for your primary site Helm release. - ``: Your Helm values file with the primary site configuration. - ``: Your Kubernetes namespace. Verify the primary site deployment: ```bash # Check primary site logs kubectl logs -n ``` Replace `` with your actual Pod name. If there are no errors, you should see a message indicating that LogWriter is properly initialized: ```console 2025-07-03 08:56:10,162 [INFO com.scalar.db.cluster.replication.logwriter.LogWriterSnapshotHook] LogWriter is initialized ``` #### 2.3 Create primary site tables Create a Kubernetes Job to run Schema Loader for the primary site: ```yaml # schema-loader-primary.yaml apiVersion: v1 kind: ConfigMap metadata: name: schema-config-primary data: application-schema.json: | { "test_namespace.test_table": { "transaction": true, "partition-key": [ "id" ], "columns": { "id": "INT", "name": "TEXT", "value": "INT" } } } --- apiVersion: v1 kind: ConfigMap metadata: name: schema-loader-config-primary data: primary-schema-loader.properties: | scalar.db.transaction_manager=cluster scalar.db.contact_points= --- apiVersion: batch/v1 kind: Job metadata: name: schema-loader-primary spec: template: spec: restartPolicy: Never containers: - name: schema-loader image: ghcr.io/scalar-labs/scalardb-cluster-schema-loader: args: - "--config" - "/config/primary-schema-loader.properties" - "--schema-file" - "/schema/application-schema.json" - "--coordinator" volumeMounts: - name: config mountPath: /config - name: schema mountPath: /schema volumes: - name: config configMap: name: schema-loader-config-primary - name: schema configMap: name: schema-config-primary ``` Replace `` with your primary site cluster contact points (same format as [ScalarDB Cluster Java Client SDK configurations](scalardb-cluster-configurations.mdx#java-client-sdk-configurations)) and `` with the ScalarDB Cluster version that you're using. Apply and run the Schema Loader job: ```bash # Apply the job kubectl apply -f schema-loader-primary.yaml -n # Check job status kubectl get jobs -n # Check logs if needed kubectl logs job/schema-loader-primary -n # Clean up after successful completion kubectl delete -f schema-loader-primary.yaml -n ``` ### Step 3: Configure the backup site In this step, you'll update the ScalarDB Cluster configuration for table schema migration, deploy a backup site cluster, and create backup site tables. #### 3.1 Update backup site ScalarDB Cluster configuration for table schema migration Add the remote replication configurations to your backup site ScalarDB Cluster that uses a multi-storage setup. For the initial deployment, enable only the replication admin feature but disable LogApplier to prevent it from trying to scan non-existent replication tables before Schema Loader creates them. The following is an example configuration: ```properties # Multi-storage configuration scalar.db.storage=multi-storage scalar.db.multi_storage.storages=default,coord,repl scalar.db.multi_storage.default_storage=default scalar.db.multi_storage.namespace_mapping=coordinator:coord,replication:repl # Backup site database scalar.db.multi_storage.storages.default.storage=jdbc scalar.db.multi_storage.storages.default.contact_points=jdbc:postgresql://backup-db:5432/ scalar.db.multi_storage.storages.default.username=postgres scalar.db.multi_storage.storages.default.password=postgres # Coordinator database (shared component - same configuration as primary site) scalar.db.multi_storage.storages.coord.storage=jdbc scalar.db.multi_storage.storages.coord.contact_points=jdbc:postgresql://coordinator-db:5432/ scalar.db.multi_storage.storages.coord.username=postgres scalar.db.multi_storage.storages.coord.password=postgres # Replication database (shared component - same configuration as primary site) scalar.db.multi_storage.storages.repl.storage=jdbc scalar.db.multi_storage.storages.repl.contact_points=jdbc:postgresql://replication-db:5432/ scalar.db.multi_storage.storages.repl.username=postgres scalar.db.multi_storage.storages.repl.password=postgres # Enable replication admin (required for Schema Loader to create replication tables) scalar.db.replication.admin.enabled=true # Enable SQL interface (if you want to test with SQL CLI) scalar.db.sql.enabled=true ``` #### 3.2 Deploy a backup site cluster Deploy a backup site ScalarDB Cluster with the remote replication configuration: ```bash helm install scalar-labs/scalardb-cluster \ -f \ -n ``` Replace: - ``: A name for your backup site Helm release. - ``: Your Helm values file with the backup site configuration. - ``: Your Kubernetes namespace. #### 3.3 Create backup site tables Create a Kubernetes Job to run Schema Loader for the backup site: ```yaml # schema-loader-backup.yaml apiVersion: v1 kind: ConfigMap metadata: name: schema-config-backup data: application-schema.json: | { "test_namespace.test_table": { "transaction": true, "partition-key": [ "id" ], "columns": { "id": "INT", "name": "TEXT", "value": "INT" } } } --- apiVersion: v1 kind: ConfigMap metadata: name: schema-loader-config-backup data: backup-schema-loader.properties: | scalar.db.transaction_manager=cluster scalar.db.contact_points= --- apiVersion: batch/v1 kind: Job metadata: name: schema-loader-backup spec: template: spec: restartPolicy: Never containers: - name: schema-loader image: ghcr.io/scalar-labs/scalardb-cluster-schema-loader: args: - "--config" - "/config/backup-schema-loader.properties" - "--schema-file" - "/schema/application-schema.json" - "--replication-tables" volumeMounts: - name: config mountPath: /config - name: schema mountPath: /schema volumes: - name: config configMap: name: schema-loader-config-backup - name: schema configMap: name: schema-config-backup ``` Replace `` with your backup site cluster contact points (same format as [ScalarDB Cluster Java Client SDK configurations](scalardb-cluster-configurations.mdx#java-client-sdk-configurations)) and `` with the ScalarDB Cluster version that you're using. Apply and run the Schema Loader job: ```bash # Apply the job kubectl apply -f schema-loader-backup.yaml -n # Check job status kubectl get jobs -n # Check logs if needed kubectl logs job/schema-loader-backup -n # Clean up after successful completion kubectl delete -f schema-loader-backup.yaml -n ``` The `--replication-tables` option creates all necessary tables in one operation, including the application tables and the replication tables. The replication record metadata tables corresponding to the application tables are automatically created when `scalar.db.replication.admin.enabled` or `scalar.db.replication.log_applier.enabled` is set to `true`. #### 3.4 Update backup site ScalarDB Cluster configuration for LogApplier deployment After the Schema Loader job completes successfully, update the backup site ScalarDB Cluster configuration to enable LogApplier and disable replication admin. This is the second deployment phase that activates the actual replication processing: ```properties # Multi-storage configuration scalar.db.storage=multi-storage scalar.db.multi_storage.storages=default,coord,repl scalar.db.multi_storage.default_storage=default scalar.db.multi_storage.namespace_mapping=coordinator:coord,replication:repl # Backup site database scalar.db.multi_storage.storages.default.storage=jdbc scalar.db.multi_storage.storages.default.contact_points=jdbc:postgresql://backup-db:5432/ scalar.db.multi_storage.storages.default.username=postgres scalar.db.multi_storage.storages.default.password=postgres # Coordinator database (shared component - same configuration as primary site) scalar.db.multi_storage.storages.coord.storage=jdbc scalar.db.multi_storage.storages.coord.contact_points=jdbc:postgresql://coordinator-db:5432/ scalar.db.multi_storage.storages.coord.username=postgres scalar.db.multi_storage.storages.coord.password=postgres # Replication database (shared component - same configuration as primary site) scalar.db.multi_storage.storages.repl.storage=jdbc scalar.db.multi_storage.storages.repl.contact_points=jdbc:postgresql://replication-db:5432/ scalar.db.multi_storage.storages.repl.username=postgres scalar.db.multi_storage.storages.repl.password=postgres # Enable LogApplier (now that replication tables exist) scalar.db.replication.log_applier.enabled=true # Enable SQL interface (if you want to test with SQL CLI) scalar.db.sql.enabled=true ``` Upgrade the backup site cluster with the updated configuration: ```bash helm upgrade scalar-labs/scalardb-cluster \ -f \ -n ``` Replace: - ``: Your backup site Helm release name. - ``: Your updated Helm values file with LogApplier enabled. - ``: Your Kubernetes namespace. Verify the backup site deployment with LogApplier: ```bash # Check backup site logs kubectl logs -n ``` Replace `` with your actual Pod name. Ensure there are no errors. You should see a message indicating that LogApplier is properly initialized: ```console 2025-07-03 03:28:27,725 [INFO com.scalar.db.cluster.replication.logapplier.LogApplier] Starting LogApplier processing. Partition range: Range{startInclusive=0, endExclusive=256} ``` ### Step 4: Test replication #### 4.1 Set up the SQL CLI for the primary site and test replication To test replication between sites, you should use the ScalarDB SQL CLI. Create a Kubernetes Pod to run the SQL CLI for the primary site: ```yaml # sql-cli-primary.yaml apiVersion: v1 kind: ConfigMap metadata: name: sql-cli-config-primary data: primary-sql-cli.properties: | scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points= --- apiVersion: v1 kind: Pod metadata: name: sql-cli-primary spec: restartPolicy: Never containers: - name: sql-cli image: ghcr.io/scalar-labs/scalardb-cluster-sql-cli: args: - "--config" - "/config/primary-sql-cli.properties" stdin: true tty: true volumeMounts: - name: config mountPath: /config volumes: - name: config configMap: name: sql-cli-config-primary ``` Replace `` with your primary site cluster contact points and `` with the ScalarDB Cluster version that you're using. Create and connect to the SQL CLI Pod by running the following commands: ```bash # Create the SQL CLI Pod kubectl apply -f sql-cli-primary.yaml -n # Attach to the running SQL CLI kubectl attach -it sql-cli-primary -n ``` ##### Test data replication Insert test data into the primary site by running the following commands: ```sql INSERT INTO test_namespace.test_table (id, name, value) VALUES (1, 'test_record', 100); SELECT * FROM test_namespace.test_table WHERE id = 1; ``` Detach from the session by pressing `Ctrl + P`, then `Ctrl + Q`. Then, create a similar Pod for the backup site: ```yaml # sql-cli-backup.yaml apiVersion: v1 kind: ConfigMap metadata: name: sql-cli-config-backup data: backup-sql-cli.properties: | scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points= --- apiVersion: v1 kind: Pod metadata: name: sql-cli-backup spec: restartPolicy: Never containers: - name: sql-cli image: ghcr.io/scalar-labs/scalardb-cluster-sql-cli: args: - "--config" - "/config/backup-sql-cli.properties" stdin: true tty: true volumeMounts: - name: config mountPath: /config volumes: - name: config configMap: name: sql-cli-config-backup ``` Replace `` with your backup site cluster contact points and `` with the ScalarDB Cluster version that you're using. Create and connect to the SQL CLI Pod by running the following commands: ```bash # Create the SQL CLI Pod kubectl apply -f sql-cli-backup.yaml -n # Attach to the running SQL CLI kubectl attach -it sql-cli-backup -n ``` Check the record is replicated by running the following command: ```sql SELECT * FROM test_namespace.test_table WHERE id = 1; ``` You should see the same data on both sites, confirming that replication is working correctly. You can insert additional records in the primary site and verify they appear in the backup site as well. To detach from the session, press `Ctrl + P`, then `Ctrl + Q`. Clean up the SQL CLI Pods when done: ```bash kubectl delete -f sql-cli-primary.yaml -n kubectl delete -f sql-cli-backup.yaml -n ``` ### Step 5: Monitor the replication state In this step, you'll monitor the replication status by using Replication CLI and Prometheus metrics. #### Replication CLI Replication CLI can get the status of LogApplier. This includes the number of partitions that contain remaining unapplied write operations in the replication database. This information is important because, if there are zero partitions, it means that all write operations have been successfully replicated and applied to the backup site database. In this case, you can use the synchronized backup site database as a new primary site database. Create a Kubernetes Pod to run Replication CLI for the backup site: ```yaml # repl-cli-backup.yaml apiVersion: v1 kind: Pod metadata: name: repl-cli-backup spec: restartPolicy: Never containers: - name: repl-cli-backup image: ghcr.io/scalar-labs/scalardb-cluster-replication-cli: args: - "--contact-points" - "" - "status" ``` Replace `` with your backup site cluster contact points (in the same format as [ScalarDB Cluster Java Client SDK configurations](scalardb-cluster-configurations.mdx#java-client-sdk-configurations)) and `` with the ScalarDB Cluster version that you're using. Ensure no new writes are being made to the primary site database to get an accurate synchronization point. Then, apply and run Replication CLI, and check the output: ```bash # Apply the Pod kubectl apply -f repl-cli-backup.yaml -n # Check the status kubectl get pod repl-cli-backup -n # Check the output from the Pod kubectl logs repl-cli-backup -n ``` If there are no errors, you should see a JSON output that includes the number of partitions containing the remaining unapplied write operations in the replication database: ```json {"remainingTransactionGroupPartitions":0} ``` If `remainingTransactionGroupPartitions` is more than 0, it indicates unapplied write operations still remain and you need to wait until it becomes 0 before using the backup site database as a new primary database. Clean up the Replication CLI Pod when done: ```bash kubectl delete -f repl-cli-backup.yaml -n ``` #### Prometheus metrics You can monitor LogApplier by using metrics. ScalarDB Cluster exposes many Prometheus format metrics, including LogApplier metrics, which can be monitored by using any tool that supports the format. For example, one option is using [Prometheus Operator (kube-prometheus-stack)](helm-charts/getting-started-monitoring.mdx). While LogApplier provides many metrics, the following metric is the most important for monitoring overall replication health: - **scalardb_cluster_stats_transaction_group_repo_oldest_record_age_millis:** The age (milliseconds) of the oldest transaction data in the replication database scanned by LogApplier. If this metric increases continuously, it indicates one of the following issues, which requires immediate investigation: - LogApplier is failing to process stored write operations (for example, the backup site database is down). - LogApplier cannot keep up with the primary site's throughput. ## Additional details Remote replication is currently in Private Preview. This feature and documentation are subject to change. For more details, please [contact us](https://www.scalar-labs.com/contact) or wait for this feature to become public preview or GA. ================================================ FILE: docs/scalardb-cluster/run-non-transactional-storage-operations-through-scalardb-cluster.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Run Non-Transactional Storage Operations Through ScalarDB Cluster import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import StorageSetupTabs from '../components/_getting-started-setup-storage.mdx'; This guide explains how to run non-transactional storage operations through ScalarDB Cluster. :::warning You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). ::: ## Preparation For the purpose of this guide, you will set up a database and ScalarDB Cluster in standalone mode by using a sample in the ScalarDB samples repository. :::note ScalarDB Cluster in standalone mode is primarily for development and testing purposes. ::: ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the necessary files by running the following command: ```console cd scalardb-samples/scalardb-cluster-standalone-mode ``` ## Set up a database Follow the instructions below to configure your database for ScalarDB Cluster. For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). For a comprehensive list of configurations for ScalarDB, see [ScalarDB Configurations](../configurations.mdx). ## Set up ScalarDB Cluster in standalone mode To set up ScalarDB Cluster in standalone mode, you'll need to configure ScalarDB Cluster to run non-transactional storage operations, set a license key, and then start ScalarDB Cluster. ### Configure ScalarDB Cluster to run non-transactional storage operations To run non-transactional storage operations, you need to configure the `scalar.db.transaction_manager` property to `single-crud-operation` in the configuration file `scalardb-cluster-node.properties`: ```properties scalar.db.transaction_manager=single-crud-operation ``` ### Set the license key Set the license key (trial license or commercial license) for the ScalarDB Clusters in the properties file. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). ### Start ScalarDB Cluster in standalone mode To start ScalarDB Cluster in standalone mode, run the following command: :::note If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. ::: ```console docker compose up -d scalardb-cluster-node ``` ## Create or import a schema ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. - **Need to create a database schema?** See [Schema Loader for Cluster](developer-guide-for-scalardb-cluster-with-java-api.mdx#schema-loader-for-cluster). - **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx). ## Load initial data as necessary ScalarDB Cluster Data Loader is a utility for importing and exporting data with ScalarDB Cluster. - **Need to import data into your database?** See [Importing data](../data-loader.mdx#importing-data). - **Need to export data from your database?** See [Exporting data](../data-loader.mdx#exporting-data). ## Create your Java application This section describes how to add the ScalarDB Cluster Java Client SDK to your project and how to configure it to run non-transactional storage operations by using Java. ### Add the ScalarDB Cluster Java Client SDK to your build The ScalarDB Cluster Java Client SDK is available in the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb-cluster-java-client-sdk). You can add the SDK as a build dependency to your application by using Gradle or Maven. Select your build tool, and follow the instructions to add the build dependency for the ScalarDB Cluster Java Client SDK to your application. To add the build dependency for the ScalarDB Cluster Java Client SDK by using Gradle, add the following to `build.gradle` in your application: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.18.0' } ``` To add the build dependency for the ScalarDB Cluster Java Client SDK by using Maven, add the following to `pom.xml` in your application: ```xml com.scalar-labs scalardb-cluster-java-client-sdk 3.18.0 ``` ### Configure the ScalarDB Cluster Java SDK For details about configuring the ScalarDB Cluster Java SDK, see [Client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#client-configurations) ### Use the Java API For details about the Java API, see [ScalarDB Java API Guide](../api-guide.mdx). :::note The following limitations apply to non-transactional storage operations: - Beginning a transaction is not supported. For more details, see [Execute transactions without beginning or starting a transaction](../api-guide.mdx#execute-transactions-without-beginning-or-starting-a-transaction). - Executing multiple mutations in a single transaction is not supported. ::: ### Learn more - [Javadoc](https://javadoc.io/doc/com.scalar-labs/scalardb/3.18.0/index.html) - [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) ================================================ FILE: docs/scalardb-cluster/run-non-transactional-storage-operations-through-sql-interface.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Run Non-Transactional Storage Operations Through the SQL Interface import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import StorageSetupTabs from '../components/_getting-started-setup-storage.mdx'; This guide explains how to run non-transactional storage operations through the SQL interface for ScalarDB Cluster. :::warning You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). ::: ## Preparation For the purpose of this guide, you will set up a database and ScalarDB Cluster in standalone mode by using a sample in the ScalarDB samples repository. :::note ScalarDB Cluster in standalone mode is primarily for development and testing purposes. ::: ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the necessary files by running the following command: ```console cd scalardb-samples/scalardb-cluster-standalone-mode ``` ## Set up a database Follow the instructions below to configure your database for ScalarDB Cluster. For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). For a comprehensive list of configurations for ScalarDB, see [ScalarDB Configurations](../configurations.mdx). ## Set up ScalarDB Cluster in standalone mode To set up ScalarDB Cluster in standalone mode, you'll need to configure ScalarDB Cluster to run non-transactional storage operations, set a license key, and then start ScalarDB Cluster. ### Configure ScalarDB Cluster to run non-transactional storage operations To run non-transactional storage operations, you need to configure the `scalar.db.transaction_manager` property to `single-crud-operation` in the configuration file `scalardb-cluster-node.properties`: ```properties scalar.db.transaction_manager=single-crud-operation ``` ### Set the license key Set the license key (trial license or commercial license) for the ScalarDB Clusters in the properties file. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). ### Start ScalarDB Cluster in standalone mode To start ScalarDB Cluster in standalone mode, run the following command: :::note If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. ::: ```console docker compose up -d scalardb-cluster-node ``` ## Create or import a schema ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. - **Need to create a database schema?** See [SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli). - **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx). Also, for a list of supported DDLs, see [ScalarDB SQL Grammar](../scalardb-sql/grammar.mdx). ## Load initial data as necessary ScalarDB Cluster Data Loader is a utility for importing and exporting data with ScalarDB Cluster. - **Need to import data into your database?** See [Importing data](../data-loader.mdx#importing-data). - **Need to export data from your database?** See [Exporting data](../data-loader.mdx#exporting-data). ## Create your application

Configure your JDBC application

This section describes how to add the ScalarDB JDBC driver to your project and how to configure it to run non-transactional storage operations by using Java.

Add the ScalarDB JDBC driver to your project

You can add the ScalarDB JDBC driver as a build dependency to your application by using Gradle or Maven. Select your build tool, and follow the instructions to add the build dependency for the ScalarDB JDBC driver to your application. To add the build dependency for the ScalarDB JDBC driver by using Gradle, add the following to `build.gradle` in your application: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-sql-jdbc:3.18.0' implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.18.0' } ``` To add the build dependency for the ScalarDB SQL API by using Maven, add the following to `pom.xml` in your application: ```xml com.scalar-labs scalardb-sql-jdbc 3.18.0 com.scalar-labs scalardb-cluster-java-client-sdk 3.18.0 ```

Configure the ScalarDB Cluster Java SDK for the SQL interface

For details about configuring the ScalarDB Cluster Java SDK for the SQL interface, see [ScalarDB Cluster SQL client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations).

Use the JDBC API

For details about the JDBC API, see [ScalarDB JDBC Guide](../scalardb-sql/jdbc-guide.mdx). :::note The following limitations apply to non-transactional storage operations: - Beginning a transaction is not supported. - Executing multiple mutations in a single SQL statement is not supported. - The isolation level is always `READ_COMMITTED`. :::

Learn more

- [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) - [Java JDBC API](https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/)

Configure your Java application

This section describes how to add the ScalarDB SQL API to your project and how to configure it to run non-transactional storage operations by using Java.

Add ScalarDB SQL API to your project

You can add the ScalarDB SQL API as a build dependency to your application by using Gradle or Maven. Select your build tool, and follow the instructions to add the build dependency for the ScalarDB SQL API to your application. To add the build dependency for the ScalarDB SQL API by using Gradle, add the following to `build.gradle` in your application: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-sql:3.18.0' implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.18.0' } ``` To add the build dependency for the ScalarDB SQL API by using Maven, add the following to `pom.xml` in your application: ```xml com.scalar-labs scalardb-sql 3.18.0 com.scalar-labs scalardb-cluster-java-client-sdk 3.18.0 ```

Configure the ScalarDB Cluster Java SDK for the SQL interface

For details about configuring the ScalarDB Cluster Java SDK for the SQL interface, see [ScalarDB Cluster SQL client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations).

Use the Java API

For details about the SQL API, see [ScalarDB SQL API Guide](../scalardb-sql/sql-api-guide.mdx). :::note The following limitations apply to non-transactional storage operations: - Beginning a transaction is not supported. - Executing multiple mutations in a single SQL statement is not supported. - The isolation level is always `READ_COMMITTED`. :::

Learn more

- [Javadoc](https://javadoc.io/doc/com.scalar-labs/scalardb-sql/3.18.0/index.html) ================================================ FILE: docs/scalardb-cluster/run-transactions-through-scalardb-cluster-sql.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Run Transactions Through ScalarDB Cluster SQL import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import StorageSetupTabs from '../components/_getting-started-setup-storage.mdx'; This guide explains how to configure your ScalarDB properties file and creating schemas to run transactions through a one-phase or a two-phase commit interface by using ScalarDB Cluster SQL. :::warning You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). ::: ## Preparation For the purpose of this guide, you will set up a database and ScalarDB Cluster in standalone mode by using a sample in the ScalarDB samples repository. :::note ScalarDB Cluster in standalone mode is primarily for development and testing purposes. ::: ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the necessary files by running the following command: ```console cd scalardb-samples/scalardb-cluster-standalone-mode ``` ## Set up a database Follow the instructions below to configure your database for ScalarDB Cluster. For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). For a comprehensive list of configurations for ScalarDB Cluster SQL, see [ScalarDB Cluster SQL client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations). ## Set up ScalarDB Cluster in standalone mode To set up ScalarDB Cluster in standalone mode, you'll need to set a license key and then start ScalarDB Cluster. ### Set the license key Set the license key (trial license or commercial license) for the ScalarDB Clusters in the properties file. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). ### Start ScalarDB Cluster in standalone mode To start ScalarDB Cluster in standalone mode, run the following command: :::note If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. ::: ```console docker compose up -d scalardb-cluster-node ``` ## Create or import a schema ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. - **Need to create a database schema?** See [SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli). - **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx). ## Load initial data as necessary ScalarDB Cluster Data Loader is a utility for importing and exporting data with ScalarDB Cluster. - **Need to import data into your database?** See [Importing data](../data-loader.mdx#importing-data). - **Need to export data from your database?** See [Exporting data](../data-loader.mdx#exporting-data). ## Run transactions You can run transactions by using a one-phase or a two-phase commit interface. Select your method for running transactions.

One-phase commit interface

For details about how to run transactions by using a one-phase commit interface, see the [ScalarDB SQL JDBC Guide](../scalardb-sql/jdbc-guide.mdx). :::note Documentation for how to run transactions in a two-phase commit interface is coming soon. :::

One-phase commit interface

For details about how to run transactions by using a one-phase commit interface, see the [ScalarDB SQL API Guide](../scalardb-sql/sql-api-guide.mdx). :::note Documentation for how to run transactions in a two-phase commit interface is coming soon. :::

Learn more

To learn more about running transactions by using ScalarDB Cluster SQL, see the following: - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx)

One-phase or two-phase commit interface

For details about how to run transactions by using a one-phase or a two-phase commit interface, see the [Getting Started with LINQ in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-linq.mdx#manage-transactions).

One-phase commit interface

For details about how to run transactions by using a one-phase commit interface, see the [Getting Started with Distributed SQL Transactions in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-sql-transactions.mdx). :::note Documentation for how to run transactions in a two-phase commit interface is coming soon. For now, please refer to [Getting Started with Distributed Transactions with a Two-Phase Commit Interface in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-two-phase-commit-transactions.mdx). ::: ================================================ FILE: docs/scalardb-cluster/run-transactions-through-scalardb-cluster.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Run Transactions Through ScalarDB Cluster import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import StorageSetupTabs from '../components/_getting-started-setup-storage.mdx'; This guide explains how to configure your ScalarDB properties file and create schemas to run transactions through a one-phase or a two-phase commit interface by using ScalarDB Cluster. :::warning You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). ::: ## Preparation For the purpose of this guide, you will set up a database and ScalarDB Cluster in standalone mode by using a sample in the ScalarDB samples repository. :::note ScalarDB Cluster in standalone mode is primarily for development and testing purposes. ::: ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the necessary files by running the following command: ```console cd scalardb-samples/scalardb-cluster-standalone-mode ``` ## Set up a database Follow the instructions below to configure your database for ScalarDB Cluster. For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). For a comprehensive list of configurations for ScalarDB Cluster, see [ScalarDB Cluster Configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#client-configurations). ## Set up ScalarDB Cluster in standalone mode To set up ScalarDB Cluster in standalone mode, you'll need to set a license key and then start ScalarDB Cluster. ### Set the license key Set the license key (trial license or commercial license) for the ScalarDB Clusters in the properties file. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). ### Start ScalarDB Cluster in standalone mode To start ScalarDB Cluster in standalone mode, run the following command: :::note If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. ::: ```console docker compose up -d scalardb-cluster-node ``` ## Create or import a schema ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. - **Need to create a database schema?** See [ScalarDB Schema Loader](../schema-loader.mdx). - **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx). ## Load initial data as necessary ScalarDB Cluster Data Loader is a utility for importing and exporting data with ScalarDB Cluster. - **Need to import data into your database?** See [Importing data](../data-loader.mdx#importing-data). - **Need to export data from your database?** See [Exporting data](../data-loader.mdx#exporting-data). ## Run transactions You can run transactions by using a one-phase or a two-phase commit interface. Select your method for running transactions. :::note If you are building a monolithic application, you should use the one-phase commit interface. However, if you are building a microservice application, see [ScalarDB Cluster Deployment Patterns for Microservices](./deployment-patterns-for-microservices.mdx) to decide which interface to use. :::

One-phase commit interface

For details about how to run transactions by using a one-phase commit interface, see the [ScalarDB Java API Guide](../api-guide.mdx#transactional-api). :::note To try running transactions by using a one-phase commit interface, see the following sample tutorials: - [Create a Sample Application That Supports Multi-Storage Transactions](../scalardb-samples/multi-storage-transaction-sample/README.mdx) - [Sample application of Spring Data JDBC for ScalarDB with Multi-storage Transactions](../scalardb-samples/spring-data-multi-storage-transaction-sample/README.mdx) :::

Two-phase commit interface

For details about how to run transactions by using a two-phase commit interface, see [Transactions with a Two-Phase Commit Interface](../two-phase-commit-transactions.mdx). :::note To try running transactions by using a two-phase commit interface, see the following sample tutorials: - [Create a Sample Application That Supports Microservice Transactions](../scalardb-samples/microservice-transaction-sample/README.mdx) - [Sample application of Spring Data JDBC for ScalarDB with Microservice Transactions](../scalardb-samples/spring-data-microservice-transaction-sample/README.mdx) :::

Learn more

To learn more about running transactions by using ScalarDB Cluster, see the following: - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx)

One-phase commit interface

For details about how to run transactions by using a one-phase commit interface, see the [Getting Started with Distributed Transactions in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-transactions.mdx).

Two-phase commit interface

For details about how to run transactions by using a two-phase commit interface, see [Getting Started with Distributed Transactions with a Two-Phase Commit Interface in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-two-phase-commit-transactions.mdx). ================================================ FILE: docs/scalardb-cluster/scalardb-abac-status-codes.mdx ================================================ --- tags: - Enterprise Premium Option - Private Preview displayed_sidebar: docsEnglish --- # Attribute-Based Access Control Error Codes This page provides a list of error codes related to attribute-based access control. ## Error code classes and descriptions | Class | Description | |:----------------|:----------------------------------------------| | `DB-ABAC-1xxxx` | Errors for the user error category | | `DB-ABAC-2xxxx` | Errors for the concurrency error category | | `DB-ABAC-3xxxx` | Errors for the internal error category | ## `DB-ABAC-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-ABAC-10000` **Message** ```markdown The put operation is not supported. Use insert, upsert, or update instead ``` ### `DB-ABAC-10001` **Message** ```markdown The default level must be less than or equal to the level. Default-level short name: %s; Default-level number: %d; Level short name: %s; Level number: %d ``` ### `DB-ABAC-10002` **Message** ```markdown The row level must be less than or equal to the level. Row-level short name: %s; Row-level number: %d; Level short name: %s; Level number: %d ``` ### `DB-ABAC-10003` **Message** ```markdown The operation does not have the target namespace or table name. Operation: %s ``` ### `DB-ABAC-10004` **Message** ```markdown The policy does not exist. Policy: %s ``` ### `DB-ABAC-10005` **Message** ```markdown The level does not exist. Policy: %s; Level short name: %s ``` ### `DB-ABAC-10006` **Message** ```markdown The compartment does not exist. Policy: %s; Compartment short name: %s ``` ### `DB-ABAC-10007` **Message** ```markdown The group does not exist. Policy: %s; Group short name: %s ``` ### `DB-ABAC-10008` **Message** ```markdown The policy already exists. Policy: %s ``` ### `DB-ABAC-10009` **Message** ```markdown The data tag column name is already in use in the policy %s. Policy: %s; Data tag column name: %s ``` ### `DB-ABAC-10010` **Message** ```markdown The level already exists. Policy: %s; Level short name: %s ``` ### `DB-ABAC-10011` **Message** ```markdown The compartment already exists. Policy: %s; Compartment short name: %s ``` ### `DB-ABAC-10012` **Message** ```markdown The group already exists. Policy: %s; Group short name: %s ``` ### `DB-ABAC-10013` **Message** ```markdown The parent group does not exist. Policy: %s; Group short name: %s; Parent-group short name: %s ``` ### `DB-ABAC-10014` **Message** ```markdown The parent group is the same as the child group. Policy: %s; Group short name: %s; Parent-group short name: %s ``` ### `DB-ABAC-10015` **Message** ```markdown The group has child groups. Policy: %s; Group short name: %s ``` ### `DB-ABAC-10016` **Message** ```markdown The compartment cannot be a row compartment for read-only access mode. Policy: %s; User: %s; Compartment short name: %s ``` ### `DB-ABAC-10017` **Message** ```markdown The compartment is already assigned to the user. Policy: %s; User: %s; Compartment short name: %s ``` ### `DB-ABAC-10018` **Message** ```markdown The group cannot be a row group for read-only access mode. Policy: %s; User: %s; Group short name: %s ``` ### `DB-ABAC-10019` **Message** ```markdown The group is already assigned to the user. Policy: %s; User: %s; Group short name: %s ``` ### `DB-ABAC-10020` **Message** ```markdown The namespace does not exist. Namespace: %s ``` ### `DB-ABAC-10021` **Message** ```markdown The namespace policy already exists. NamespacePolicy: %s ``` ### `DB-ABAC-10022` **Message** ```markdown The namespace policy does not exist. NamespacePolicy: %s ``` ### `DB-ABAC-10023` **Message** ```markdown The table does not exist. Table: %s ``` ### `DB-ABAC-10024` **Message** ```markdown The table policy already exists. TablePolicy: %s ``` ### `DB-ABAC-10025` **Message** ```markdown The table policy does not exist. TablePolicy: %s ``` ### `DB-ABAC-10026` **Message** ```markdown The short name must not be empty. Short name: %s ``` ### `DB-ABAC-10027` **Message** ```markdown The short name must be 30 characters or less. Short name: %s ``` ### `DB-ABAC-10028` **Message** ```markdown The short name must not contain a colon. Short name: %s ``` ### `DB-ABAC-10029` **Message** ```markdown The short name must not contain a comma. Short name: %s ``` ### `DB-ABAC-10030` **Message** ```markdown The data tag is invalid. Data tag: %s ``` ### `DB-ABAC-10031` **Message** ```markdown The level must be specified in the data tag. Data tag: %s ``` ### `DB-ABAC-10032` **Message** ```markdown The user tag is invalid. User tag: %s ``` ### `DB-ABAC-10033` **Message** ```markdown The level must be specified in the user tag. User tag: %s ``` ### `DB-ABAC-10034` **Message** ```markdown The specified user tag is not allowed. User tag: %s ``` ### `DB-ABAC-10035` **Message** ```markdown The data tag column type should be TEXT. Policy: %s; Data tag column: %s ``` ### `DB-ABAC-10036` **Message** ```markdown The specified data tag is not allowed. Data tag: %s ``` ### `DB-ABAC-10037` **Message** ```markdown The data tag column cannot be used in conditions. Operation: %s; Data tag column: %s ``` ### `DB-ABAC-10038` **Message** ```markdown The operation is not allowed for the policy. Policy: %s; Operation: %s ``` ### `DB-ABAC-10039` **Message** ```markdown Access denied: Invalid auth token ``` ### `DB-ABAC-10040` **Message** ```markdown The authentication and authorization feature must be enabled to use the attribute-based access control feature ``` ### `DB-ABAC-10041` **Message** ```markdown The single CRUD operation transaction manager does not support the attribute-based access control feature ``` ### `DB-ABAC-10042` **Message** ```markdown The data tag column cannot be used in ordering. Operation: %s; Data tag column: %s ``` ### `DB-ABAC-10043` **Message** ```markdown The namespace policy for the policy and namespace already exists. Policy: %s; Namespace: %s ``` ### `DB-ABAC-10044` **Message** ```markdown The table policy for the policy and table already exists. Policy: %s; Table: %s ``` ### `DB-ABAC-10045` **Message** ```markdown The user does not exist. Username: %s ``` ### `DB-ABAC-10046` **Message** ```markdown The table already exists. Table: %s ``` ## `DB-ABAC-2xxxx` status codes The following are status codes and messages for the concurrency error category. ### `DB-ABAC-20000` **Message** ```markdown The record does not exist, so the %s condition is not satisfied ``` ## `DB-ABAC-3xxxx` status codes The following are status codes and messages for the internal error category. ### `DB-ABAC-30000` **Message** ```markdown Creating a policy failed. Policy: %s; Data tag column: %s ``` ### `DB-ABAC-30001` **Message** ```markdown Enabling the policy failed. Policy: %s ``` ### `DB-ABAC-30002` **Message** ```markdown Disabling the policy failed. Policy: %s ``` ### `DB-ABAC-30003` **Message** ```markdown Getting the policy failed. Policy: %s ``` ### `DB-ABAC-30004` **Message** ```markdown Getting the policies failed ``` ### `DB-ABAC-30005` **Message** ```markdown Creating a level failed. Policy: %s; Level short name: %s; Level long name: %s; Level number: %d ``` ### `DB-ABAC-30006` **Message** ```markdown Dropping the level failed. Policy: %s; Level short name: %s ``` ### `DB-ABAC-30007` **Message** ```markdown Getting the level failed. Policy: %s; Level short name: %s ``` ### `DB-ABAC-30008` **Message** ```markdown Getting the levels failed. Policy: %s ``` ### `DB-ABAC-30009` **Message** ```markdown Creating a compartment failed. Policy: %s; Compartment short name: %s; Compartment long name: %s ``` ### `DB-ABAC-30010` **Message** ```markdown Dropping the compartment failed. Policy: %s; Compartment short name: %s ``` ### `DB-ABAC-30011` **Message** ```markdown Getting the compartment failed. Policy: %s; Compartment short name: %s ``` ### `DB-ABAC-30012` **Message** ```markdown Getting the compartments failed. Policy: %s ``` ### `DB-ABAC-30013` **Message** ```markdown Creating a group failed. Policy: %s; Group short name: %s; Group long name: %s; Parent-group short name: %s ``` ### `DB-ABAC-30014` **Message** ```markdown Dropping the group failed. Policy: %s; Group short name: %s ``` ### `DB-ABAC-30015` **Message** ```markdown Getting the group failed. Policy: %s; Group short name: %s ``` ### `DB-ABAC-30016` **Message** ```markdown Getting groups failed. Policy: %s ``` ### `DB-ABAC-30017` **Message** ```markdown Getting child groups failed. Policy: %s; Parent-group short name: %s ``` ### `DB-ABAC-30018` **Message** ```markdown Setting the levels to the user failed. Policy: %s; User: %s; Level short name: %s; Default-level short name: %s; Row-level short name: %s ``` ### `DB-ABAC-30019` **Message** ```markdown Adding the compartment to the user failed. Policy: %s; User: %s; Compartment short name: %s ``` ### `DB-ABAC-30020` **Message** ```markdown Removing the compartment from the user failed. Policy: %s; User: %s; Compartment short name: %s ``` ### `DB-ABAC-30021` **Message** ```markdown Adding the group to the user failed. Policy: %s; User: %s; Group short name: %s ``` ### `DB-ABAC-30022` **Message** ```markdown Removing the group from the user failed. Policy: %s; User: %s; Group short name: %s ``` ### `DB-ABAC-30023` **Message** ```markdown Dropping the user tag information from the user failed. Policy: %s; User: %s ``` ### `DB-ABAC-30024` **Message** ```markdown Getting the user tag information failed. Policy: %s; User: %s ``` ### `DB-ABAC-30025` **Message** ```markdown Creating a namespace policy failed. NamespacePolicy: %s; Policy: %s; Namespace: %s ``` ### `DB-ABAC-30026` **Message** ```markdown Enabling the namespace policy failed. NamespacePolicy: %s ``` ### `DB-ABAC-30027` **Message** ```markdown Disabling the namespace policy failed. NamespacePolicy: %s ``` ### `DB-ABAC-30028` **Message** ```markdown Removing the namespace policies assigned to the namespace failed. Namespace: %s ``` ### `DB-ABAC-30029` **Message** ```markdown Getting the namespace policies failed ``` ### `DB-ABAC-30030` **Message** ```markdown Creating a table policy failed. TablePolicy: %s; Policy: %s; Table: %s ``` ### `DB-ABAC-30031` **Message** ```markdown Enabling the table policy failed. TablePolicy: %s ``` ### `DB-ABAC-30032` **Message** ```markdown Disabling the table policy failed. TablePolicy: %s ``` ### `DB-ABAC-30033` **Message** ```markdown Removing the table policies assigned to the table failed. Table: %s ``` ### `DB-ABAC-30034` **Message** ```markdown Getting the table policies failed ``` ### `DB-ABAC-30035` **Message** ```markdown Getting the policies assigned to the namespace failed. Namespace: %s ``` ### `DB-ABAC-30036` **Message** ```markdown Getting the policies assigned to the table failed. Table: %s ``` ### `DB-ABAC-30037` **Message** ```markdown Registering the data tag failed. Policy: %s; Data tag: %s ``` ### `DB-ABAC-30038` **Message** ```markdown Getting the data tags failed. Policy: %s ``` ### `DB-ABAC-30039` **Message** ```markdown Getting the namespace policy failed. NamespacePolicy: %s ``` ### `DB-ABAC-30040` **Message** ```markdown Getting the table policy failed. TablePolicy: %s ``` ### `DB-ABAC-30041` **Message** ```markdown Renaming the table in the table policies failed. Old table name: %s; New table name: %s ``` ================================================ FILE: docs/scalardb-cluster/scalardb-auth-status-codes.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Authentication and Authorization Error Codes This page provides a list of error codes related to authentication and authorization. ## Error code classes and descriptions | Class | Description | |:----------------|:---------------------------------------| | `DB-AUTH-1xxxx` | Errors for the user error category | | `DB-AUTH-3xxxx` | Errors for the internal error category | ## `DB-AUTH-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-AUTH-10000` **Message** ```markdown The user already exists. Username: %s ``` ### `DB-AUTH-10001` **Message** ```markdown The user does not exist. Username: %s ``` ### `DB-AUTH-10003` **Message** ```markdown The namespace does not exist. Namespace: %s ``` ### `DB-AUTH-10004` **Message** ```markdown The table does not exist. Table: %s ``` ### `DB-AUTH-10005` **Message** ```markdown Invalid username or password ``` ### `DB-AUTH-10006` **Message** ```markdown Access denied: Invalid auth token ``` ### `DB-AUTH-10007` **Message** ```markdown Access denied: You need the %s privilege on the namespace %s to execute this operation ``` ### `DB-AUTH-10008` **Message** ```markdown Access denied: You need the %s privilege on the table %s to execute this operation ``` ### `DB-AUTH-10009` **Message** ```markdown Access denied: You must be a superuser to execute this operation ``` ### `DB-AUTH-10010` **Message** ```markdown Access denied: You can't access information about the user %s ``` ### `DB-AUTH-10011` **Message** ```markdown Access denied: You can't alter the user %s ``` ### `DB-AUTH-10012` **Message** ```markdown Access denied: You must be a superuser to change the SUPERUSER attribute ``` ### `DB-AUTH-10013` **Message** ```markdown You can't change the SUPERUSER attribute for the current user %s ``` ### `DB-AUTH-10014` **Message** ```markdown You can't drop the current user %s ``` ### `DB-AUTH-10015` **Message** ```markdown Access denied: You can't grant the %s privilege because you don't have the same privilege with GRANT OPTION on the table %s ``` ### `DB-AUTH-10016` **Message** ```markdown Access denied: You can't grant the %s privilege because you don't have the same privilege with GRANT OPTION on the namespace %s ``` ### `DB-AUTH-10017` **Message** ```markdown Access denied: You can't revoke the %s privilege because you don't have the same privilege with GRANT OPTION on the table %s ``` ### `DB-AUTH-10018` **Message** ```markdown Access denied: You can't revoke the %s privilege because you don't have the same privilege with GRANT OPTION on the namespace %s ``` ### `DB-AUTH-10019` **Message** ```markdown The operation does not have the target namespace or table name. Operation: %s ``` ### `DB-AUTH-10020` **Message** ```markdown The table already exists. Table: %s ``` ### `DB-AUTH-10021` **Message** ```markdown The role already exists. Role: %s ``` ### `DB-AUTH-10022` **Message** ```markdown The role does not exist. Role: %s ``` ### `DB-AUTH-10023` **Message** ```markdown Role not created because it would cause a role cycle to occur. Role: %s; GrantedRole: %s ``` ### `DB-AUTH-10024` **Message** ```markdown Access denied: You must be a superuser or have the admin option to execute this operation. Role: %s ``` ### `DB-AUTH-10025` **Message** ```markdown Access denied: You must be associated with the role %s ``` ### `DB-AUTH-10026` **Message** ```markdown Access denied: You can't grant the %s privilege to the role %s because you don't have the same privilege with GRANT OPTION on the table %s ``` ### `DB-AUTH-10027` **Message** ```markdown Access denied: You can't grant the %s privilege to the role %s because you don't have the same privilege with GRANT OPTION on the namespace %s ``` ### `DB-AUTH-10028` **Message** ```markdown Access denied: You can't revoke the %s privilege from the role %s because you don't have the same privilege with GRANT OPTION on the table %s ``` ### `DB-AUTH-10029` **Message** ```markdown Access denied: You can't revoke the %s privilege from the role %s because you don't have the same privilege with GRANT OPTION on the namespace %s ``` ## `DB-AUTH-3xxxx` status codes The following are status codes and messages for the internal error category. ### `DB-AUTH-30000` **Message** ```markdown Getting auth token information failed ``` ### `DB-AUTH-30001` **Message** ```markdown Getting the user failed. Username: %s ``` ### `DB-AUTH-30002` **Message** ```markdown Creating a user failed. Username: %s ``` ### `DB-AUTH-30003` **Message** ```markdown Altering the user failed. Username: %s ``` ### `DB-AUTH-30004` **Message** ```markdown Dropping the user failed. Username: %s ``` ### `DB-AUTH-30005` **Message** ```markdown Granting privileges failed. Username: %s; Namespace: %s; Privileges: %s ``` ### `DB-AUTH-30006` **Message** ```markdown Granting privileges failed. Username: %s; Table: %s; Privileges: %s ``` ### `DB-AUTH-30007` **Message** ```markdown Revoking privileges failed. Username: %s; Namespace: %s; Privileges: %s ``` ### `DB-AUTH-30008` **Message** ```markdown Revoking privileges failed. Username: %s; Table: %s; Privileges: %s ``` ### `DB-AUTH-30009` **Message** ```markdown Getting users failed ``` ### `DB-AUTH-30010` **Message** ```markdown Getting privileges failed. Username: %s; Namespace: %s ``` ### `DB-AUTH-30011` **Message** ```markdown Getting privileges failed. Username: %s; Table: %s ``` ### `DB-AUTH-30012` **Message** ```markdown Deleting privileges failed. Namespace: %s ``` ### `DB-AUTH-30013` **Message** ```markdown Deleting privileges failed. Table: %s ``` ### `DB-AUTH-30014` **Message** ```markdown Logging in failed. Username: %s ``` ### `DB-AUTH-30015` **Message** ```markdown Logging out failed ``` ### `DB-AUTH-30016` **Message** ```markdown Renaming table in privileges failed. Old table: %s; New table: %s ``` ### `DB-AUTH-30017` **Message** ```markdown Creating a role failed. Role: %s ``` ### `DB-AUTH-30018` **Message** ```markdown Dropping the role failed. Role: %s ``` ### `DB-AUTH-30019` **Message** ```markdown Granting the role to the user failed. User: %s; Role: %s ``` ### `DB-AUTH-30020` **Message** ```markdown Revoking the role from the user failed. User: %s; Role: %s ``` ### `DB-AUTH-30021` **Message** ```markdown Revoking the admin option from the user failed. User: %s; Role: %s ``` ### `DB-AUTH-30022` **Message** ```markdown Granting the role to the role failed. Role: %s; Granted role: %s ``` ### `DB-AUTH-30023` **Message** ```markdown Revoking role from role failed. Role: %s; Granted role: %s ``` ### `DB-AUTH-30024` **Message** ```markdown Revoking admin option from role failed. Role: %s; Granted role: %s ``` ### `DB-AUTH-30025` **Message** ```markdown Getting the role failed. RoleName: %s ``` ### `DB-AUTH-30026` **Message** ```markdown Getting roles failed ``` ### `DB-AUTH-30027` **Message** ```markdown Getting role hierarchies failed. Role: %s ``` ### `DB-AUTH-30028` **Message** ```markdown Getting users for role failed. Role: %s ``` ### `DB-AUTH-30029` **Message** ```markdown Getting roles for the user failed. Username: %s ``` ### `DB-AUTH-30030` **Message** ```markdown Getting role privileges failed. RoleName: %s; Namespace: %s ``` ### `DB-AUTH-30031` **Message** ```markdown Getting role privileges failed. RoleName: %s; Table: %s ``` ### `DB-AUTH-30032` **Message** ```markdown Granting privileges to the role failed. Role: %s; Namespace: %s; Privileges: %s ``` ### `DB-AUTH-30033` **Message** ```markdown Granting privileges to the role failed. Role: %s; Table: %s; Privileges: %s ``` ### `DB-AUTH-30034` **Message** ```markdown Revoking privileges from the role failed. Role: %s; Namespace: %s; Privileges: %s ``` ### `DB-AUTH-30035` **Message** ```markdown Revoking privileges from the role failed. Role: %s; Table: %s; Privileges: %s ``` ================================================ FILE: docs/scalardb-cluster/scalardb-auth-with-sql.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Authenticate and Authorize Users import JavadocLink from '/src/theme/JavadocLink.js'; import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; ScalarDB Cluster can authenticate and authorize users in a coarse-grained manner. You can create users and grant or revoke their privileges. Roles can also be created to group privileges and can be granted to users or other roles. This guide describes how to use authentication and authorization in ScalarDB SQL. For more details about the grammar, see [DCL](../scalardb-sql/grammar.mdx#dcl). :::tip You can also do authentication and authorization by using the primitive interface. For details, see , which implements . ::: ## Authentication methods ScalarDB Cluster supports the following authentication methods: - **Username and password (`USERPASS`):** Users authenticate with a username and password. This is the default method described in this guide. - **OIDC (`OIDC`):** Client applications pass JWT access tokens from an OIDC provider (for example, Keycloak) instead of passwords. For details, see [Control User Access via OIDC-Based JWT Access Tokens](./control-access-via-oidc-based-jwt-tokens.mdx). ## Users Users can log in to ScalarDB Cluster with a username and a password and execute SQL statements if they have the required privileges. Authentication and authorization support two types of users: - **Superusers:** This type of user has all privileges. Only superusers can create or drop other users and namespaces. - **Normal users:** This type of user initially doesn't have any privileges, so they need to be granted privileges by a superuser or another user who has the `GRANT` privilege. ### Initial user When you enable authentication and authorization, the initial user `admin` is created and the initial password of that user is `admin`. This user is a superuser and has all privileges. You can log in with this user and create other users if necessary. :::warning For security purposes, be sure to change the password of the initial user, especially before deploying to a production environment. ::: ## Roles A role is a named collection of privileges that can be granted to users or other roles. Using roles provides a convenient way to manage privileges for multiple users, rather than granting individual privileges to each user. Only superusers can create or drop roles. Users who have the `GRANT` privilege can grant their privileges to roles. When a role is granted to a user, the user can use all privileges granted to that role. If the role has other roles granted to it (role hierarchy), the user can also use the privileges from those roles. When granting a role, you can optionally specify `WITH ADMIN OPTION` to allow the grantee to grant the same role to others. ## Privileges The following privileges are available when using authentication and authorization: - `SELECT` - `INSERT` - `UPDATE` - `DELETE` - `CREATE` - `DROP` - `TRUNCATE` - `ALTER` - `GRANT` ### Which privileges are required for each type of operation The following tables show which privileges are required for each type of operation: #### DDL | Command | Superuser required | Required privileges | |-------------------------------|--------------------|---------------------| | `CREATE NAMESPACE` | `true` | | | `DROP NAMESPACE` | `true` | | | `CREATE TABLE` | | `CREATE` | | `DROP TABLE` | | `DROP` | | `CREATE INDEX` | | `CREATE` | | `DROP INDEX` | | `DROP` | | `TRUNCATE TABLE` | | `TRUNCATE` | | `ALTER TABLE` | | `ALTER` | | `CREATE COORDINATOR TABLES` | `true` | | | `DROP COORDINATOR TABLES` | `true` | | | `TRUNCATE COORDINATOR TABLES` | `true` | | #### DML | Command | Superuser required | Required privileges | |----------|--------------------|---------------------------------| | `SELECT` | | `SELECT` | | `INSERT` | | `SELECT`, `INSERT`, and `UPDATE` | | `UPSERT` | | `SELECT`, `INSERT`, and `UPDATE` | | `UPDATE` | | `SELECT`, `INSERT`, and `UPDATE` | | `DELETE` | | `SELECT` and `DELETE` | :::note ScalarDB initially offered only the `Put` operation, which corresponds to `UPSERT`, for writing data. As a result, there is only one internal write permission. Consequently, the permissions needed for `INSERT`, `UPDATE`, and `UPSERT` are the same; all of them require the `SELECT` privilege. In the future, ScalarDB plans to provide finer-grained write permissions. ::: #### DCL | Command | Superuser required | Required privileges | |------------------------|-----------------------------------------------|--------------------------------------------------------------------------------| | `CREATE USER` | `true` | | | `ALTER USER` | `true` (Users can change their own password.) | | | `DROP USER` | `true` | | | `GRANT` | | `GRANT` (Users can grant only the privileges that they have.) | | `REVOKE` | | `GRANT` (Users can revoke only the privileges that they have.) | | `CREATE ROLE` | `true` | | | `DROP ROLE` | `true` | | | `GRANT ... TO ROLE` | | `GRANT` (Users can grant only the privileges that they have.) | | `REVOKE ... FROM ROLE` | | `GRANT` (Users can revoke only the privileges that they have.) | | `GRANT ROLE` | | `ADMIN OPTION` on the role (Users can grant only those roles.) | | `REVOKE ROLE` | | `ADMIN OPTION` on the role (Users can revoke only those roles.) | | `REVOKE ADMIN OPTION` | | `ADMIN OPTION` on the role (Users can revoke `ADMIN OPTION` only for those roles.) | ## Configurations This section describes the available configurations for authentication and authorization. ### ScalarDB Cluster node configurations To enable authentication and authorization, you need to set `scalar.db.cluster.auth.enabled` to `true`. | Name | Description | Default | |----------------------------------|------------------------------------|---------| | `scalar.db.cluster.auth.enabled` | Whether authentication and authorization are enabled. | `false` | You can also set the following configurations: | Name | Description | Default | |----------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|--------------------| | `scalar.db.cluster.auth.cache_expiration_time_millis` | Cache expiration time for authentication and authorization information in milliseconds. | `60000` (1 minute) | | `scalar.db.cluster.auth.auth_token_expiration_time_minutes` | Authentication and authorization token expiration time in minutes. | `1440` (1 day) | | `scalar.db.cluster.auth.auth_token_gc_thread_interval_minutes` | Authentication and authorization token garbage collection (GC) thread interval in minutes. | `360` (6 hours) | | `scalar.db.cluster.auth.pepper` | A secret value added to a password before hashing. If not specified, the password is hashed without pepper. | | ### ScalarDB Cluster Java client SDK configurations To enable authentication and authorization on the client side, you need to set `scalar.db.cluster.auth.enabled` to `true`. | Name | Description | Default | |----------------------------------|-----------------------------------|---------| | `scalar.db.cluster.auth.enabled` | Whether authentication and authorization are enabled. | `false` | In addition to the configuration in the [ScalarDB Cluster SQL client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations) section, you also need to set `scalar.db.sql.cluster_mode.username` and `scalar.db.sql.cluster_mode.password` to specify the username and password of the client. | Name | Description | Default | |---------------------------------------|-----------------------------|---------| | `scalar.db.sql.cluster_mode.username` | The username of the client. | | | `scalar.db.sql.cluster_mode.password` | The password of the client. | | ## Wire encryption If you enable authentication and authorization, enabling wire encryption to protect the user credentials is strongly recommended, especially in production environments. For details about wire encryption, see [Encrypt Wire Communications](encrypt-wire-communications.mdx). ## Tutorial - Authenticate and authorize users This tutorial explains how to use authentication and authorization. ### Prerequisites - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ### 1. Create the ScalarDB Cluster configuration file Create the following configuration file as `scalardb-cluster-node.properties`, replacing `` and `` with your ScalarDB license key and license check certificate values. For more information about the license key and certificate, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). ```properties scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://postgresql:5432/postgres scalar.db.username=postgres scalar.db.password=postgres scalar.db.cluster.node.standalone_mode.enabled=true scalar.db.sql.enabled=true # Enable cross-partition scan to perform a full scan by using the SELECT statements in this tutorial. # This is not required for authentication and authorization itself. scalar.db.cross_partition_scan.enabled=true # Enable authentication and authorization scalar.db.cluster.auth.enabled=true # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` ### 2. Create the Docker Compose file Create the following configuration file as `docker-compose.yaml`. ```yaml services: postgresql: container_name: "postgresql" image: "postgres:15" ports: - 5432:5432 environment: - POSTGRES_PASSWORD=postgres healthcheck: test: ["CMD-SHELL", "pg_isready || exit 1"] interval: 1s timeout: 10s retries: 60 start_period: 30s scalardb-cluster-standalone: container_name: "scalardb-cluster-node" image: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium:3.18.0" ports: - 60053:60053 - 9080:9080 volumes: - ./scalardb-cluster-node.properties:/scalardb-cluster/node/scalardb-cluster-node.properties depends_on: postgresql: condition: service_healthy ``` ### 3. Start PostgreSQL and ScalarDB Cluster Run the following command to start PostgreSQL and ScalarDB Cluster in standalone mode. ```console docker compose up -d ``` It may take a few minutes for ScalarDB Cluster to fully start. ### 4. Connect to ScalarDB Cluster To connect to ScalarDB Cluster, this tutorial uses the SQL CLI, a tool for connecting to ScalarDB Cluster and executing SQL queries. You can download the SQL CLI from the [ScalarDB releases page](https://github.com/scalar-labs/scalardb/releases). Create a configuration file named `scalardb-cluster-sql-cli.properties`. This file will be used to connect to ScalarDB Cluster by using the SQL CLI. ```properties scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points=indirect:localhost # Enable authentication and authorization scalar.db.cluster.auth.enabled=true ``` Then, start the SQL CLI by running the following command. ```console java -jar scalardb-cluster-sql-cli-3.18.0-all.jar --config scalardb-cluster-sql-cli.properties ``` Enter the username and password as `admin` and `admin`, respectively. Now you're ready to use the database with authentication and authorization enabled in ScalarDB Cluster. ### 5. Create namespaces and a table Create namespaces. ```sql CREATE NAMESPACE ns1; CREATE NAMESPACE ns2; ``` Next, create a table in the `ns1` namespaces. ```sql CREATE TABLE ns1.tbl ( id INT PRIMARY KEY, col1 TEXT, col2 INT); ``` ### 6. Create a user Create a user named `user1`. ```sql CREATE USER user1 WITH PASSWORD 'user1'; ``` To check the user, run the following command. ```sql SHOW USERS; ``` ```console +----------+-------------+-----------------------+ | username | isSuperuser | authenticationMethods | +----------+-------------+-----------------------+ | user1 | false | USERPASS | | admin | true | USERPASS | +----------+-------------+-----------------------+ ``` You can see that the `user1` user has been created. ### 7. Grant privileges Grant the `SELECT`, `INSERT`, and `UPDATE` privileges to `user1` on the `ns1.tbl` table. ```sql GRANT SELECT, INSERT, UPDATE ON ns1.tbl TO user1; ``` Then, grant the `SELECT` privilege to `user1` on the `ns2` namespace. ```sql GRANT SELECT ON NAMESPACE ns2 TO user1; ``` To check the privileges, run the following command. ```sql SHOW GRANTS FOR user1; ``` ```console +---------+-----------+-----------+---------------+-------------------------+ | name | type | privilege | grantedToUser | rolesProvidingPrivilege | +---------+-----------+-----------+---------------+-------------------------+ | ns2 | NAMESPACE | SELECT | true | | | ns1.tbl | TABLE | SELECT | true | | | ns1.tbl | TABLE | INSERT | true | | | ns1.tbl | TABLE | UPDATE | true | | +---------+-----------+-----------+---------------+-------------------------+ ``` You can see that `user1` has been granted the `SELECT`, `INSERT`, and `UPDATE` privileges on the `ns1.tbl` table, and the `SELECT` privilege on the `ns2` namespace. ### 8. Log in as `user1` Log in as `user1` and execute SQL statements. ```console java -jar scalardb-cluster-sql-cli-3.18.0-all.jar --config scalardb-cluster-sql-cli.properties ``` Enter the username and password as `user1` and `user1`, respectively. Now you can execute SQL statements as `user1`. ### 9. Execute DML statements Execute the following `INSERT` statement as `user1`. ```sql INSERT INTO ns1.tbl VALUES (1, 'a', 1); ``` Then, execute the following `SELECT` statement as `user1`. ```sql SELECT * FROM ns1.tbl; ``` ```console +----+------+------+ | id | col1 | col2 | +----+------+------+ | 1 | a | 1 | +----+------+------+ ``` You can see that `user1` can execute `INSERT` and `SELECT` statements. Next, try executing the following `DELETE` statement as `user1`. ```sql DELETE FROM ns1.tbl WHERE id = 1; ``` ```console Error: Authorization error (PERMISSION_DENIED: SQL-10021: Access denied: You need the DELETE privilege on the table ns1.tbl to execute this operation) (state=SDB11,code=9911) ``` You will see the above error message because `user1` doesn't have the `DELETE` privilege on the `ns1.tbl` table. ### 10. Use roles to manage privileges Log in as `admin` to create and manage roles. ```console java -jar scalardb-cluster-sql-cli-3.18.0-all.jar --config scalardb-cluster-sql-cli.properties ``` Enter the username and password as `admin` and `admin`, respectively. Create a role named `cleanup_role`. ```sql CREATE ROLE cleanup_role; ``` To verify the role has been created, run the following command. ```sql SHOW ROLES; ``` ```console +--------------+--------------+ | roleName | grantedRoles | +--------------+--------------+ | cleanup_role | | +--------------+--------------+ ``` Grant the `SELECT`, `DELETE`, and `TRUNCATE` privileges on the `ns1.tbl` table to the role. ```sql GRANT SELECT, DELETE, TRUNCATE ON ns1.tbl TO ROLE cleanup_role; ``` To verify the privileges granted to the role, run the following command. ```sql SHOW ROLE GRANTS FOR cleanup_role; ``` ```console +---------+-------+-----------+ | name | type | privilege | +---------+-------+-----------+ | ns1.tbl | TABLE | SELECT | | ns1.tbl | TABLE | DELETE | | ns1.tbl | TABLE | TRUNCATE | +---------+-------+-----------+ ``` Grant the role to `user1`. ```sql GRANT ROLE cleanup_role TO user1; ``` To verify the privileges of `user1`, run the following command. ```sql SHOW GRANTS FOR user1; ``` ```console +---------+-----------+-----------+---------------+-------------------------+ | name | type | privilege | grantedToUser | rolesProvidingPrivilege | +---------+-----------+-----------+---------------+-------------------------+ | ns2 | NAMESPACE | SELECT | true | | | ns1.tbl | TABLE | SELECT | true | cleanup_role | | ns1.tbl | TABLE | INSERT | true | | | ns1.tbl | TABLE | UPDATE | true | | | ns1.tbl | TABLE | DELETE | false | cleanup_role | | ns1.tbl | TABLE | TRUNCATE | false | cleanup_role | +---------+-----------+-----------+---------------+-------------------------+ ``` Now, log in as `user1` and try the `DELETE` statement again. ```console java -jar scalardb-cluster-sql-cli-3.18.0-all.jar --config scalardb-cluster-sql-cli.properties ``` Enter the username and password as `user1` and `user1`, respectively. ```sql DELETE FROM ns1.tbl WHERE id = 1; ``` This time, the statement succeeds because `user1` now has the `DELETE` privilege through the `cleanup_role` role. ## See also For more information about using RBAC, see the role-related sections in the [ScalarDB SQL Grammar](../scalardb-sql/grammar.mdx) reference. ================================================ FILE: docs/scalardb-cluster/scalardb-cluster-configurations.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster Configurations import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; This document describes the configurations for ScalarDB Cluster. ScalarDB Cluster consists of multiple cluster nodes, each of which needs to be configured. The configurations need to be specified in the properties file. ## Cluster configurations This section describes the configurations for ScalarDB Cluster. ### General configurations The following general configurations are available for ScalarDB Cluster. #### Transaction management configurations The following transaction management-related configurations are available for ScalarDB Cluster. ##### `transaction_manager` - **Field:** `scalar.db.transaction_manager` - **Description:** Transaction manager of ScalarDB. Specify `consensus-commit` to use [Consensus Commit](../consensus-commit.mdx) or `single-crud-operation` to [run non-transactional storage operations](./run-non-transactional-storage-operations-through-scalardb-cluster.mdx). Note that the configurations under the `scalar.db.consensus_commit` prefix are ignored if you use `single-crud-operation`. - **Default value:** `consensus-commit` ##### `isolation_level` - **Field:** `scalar.db.consensus_commit.isolation_level` - **Description:** Isolation level used for Consensus Commit. Either `SNAPSHOT`, `SERIALIZABLE`, or `READ_COMMITTED` can be specified. - **Default value:** `SNAPSHOT` ##### `coordinator.namespace` - **Field:** `scalar.db.consensus_commit.coordinator.namespace` - **Description:** Namespace name of Coordinator tables used for Consensus Commit. - **Default value:** `coordinator` #### Node configurations The following node-related configurations are available for ScalarDB Cluster. ##### `cluster.membership.type` - **Field:** `scalar.db.cluster.membership.type` - **Description:** Membership type. Currently, only `KUBERNETES` can be specified. - **Default value:** `KUBERNETES` ##### `cluster.membership.kubernetes.endpoint.namespace_name` - **Field:** `scalar.db.cluster.membership.kubernetes.endpoint.namespace_name` - **Description:** This configuration is for the `KUBERNETES` membership type. Namespace name for the [endpoint resource](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints). - **Default value:** `default` ##### `cluster.membership.kubernetes.endpoint.name` - **Field:** `scalar.db.cluster.membership.kubernetes.endpoint.name` - **Description:** This configuration is for the `KUBERNETES` membership type. Name of the [endpoint resource](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints) to get the membership info. - **Default value:** empty ##### `cluster.node.decommissioning_duration_secs` - **Field:** `scalar.db.cluster.node.decommissioning_duration_secs` - **Description:** Duration in seconds until a ScalarDB Cluster node is actually decommissioned when shutting down. - **Default value:** `30` ##### `cluster.node.grpc.max_inbound_message_size` - **Field:** `scalar.db.cluster.node.grpc.max_inbound_message_size` - **Description:** Maximum message size allowed to be received. - **Default value:** The gRPC default value ##### `cluster.node.grpc.max_inbound_metadata_size` - **Field:** `scalar.db.cluster.node.grpc.max_inbound_metadata_size` - **Description:** Maximum size of metadata allowed to be received. - **Default value:** The gRPC default value ##### `cluster.node.port` - **Field:** `scalar.db.cluster.node.port` - **Description:** Port number of the ScalarDB Cluster node. - **Default value:** `60053` ##### `cluster.internal.node.port` - **Field:** `scalar.db.cluster.internal.node.port` - **Description:** Port number of the gRPC server used for internal communication between ScalarDB Cluster nodes. - **Default value:** `60054` ##### `cluster.node.admin.port` - **Field:** `scalar.db.cluster.node.admin.port` - **Description:** Port number of the administrative gRPC server of the ScalarDB Cluster node. If this property is set, the administrator service (`pause`, `unpause`, and `checkPaused`) will run on a dedicated gRPC server on this port, which is useful for isolating administrative operations from other gRPC traffic. If this property is not set, the administrator service will run on the same port as the other gRPC services (`scalar.db.cluster.node.port`). - **Default value:** empty (uses `scalar.db.cluster.node.port`) ##### `cluster.node.prometheus_exporter_port` - **Field:** `scalar.db.cluster.node.prometheus_exporter_port` - **Description:** Port number of the Prometheus exporter. - **Default value:** `9080` ##### `cluster.grpc.deadline_duration_millis` - **Field:** `scalar.db.cluster.grpc.deadline_duration_millis` - **Description:** Deadline duration for gRPC in milliseconds. - **Default value:** `60000` (60 seconds) ##### `cluster.node.standalone_mode.enabled` - **Field:** `scalar.db.cluster.node.standalone_mode.enabled` - **Description:** Whether standalone mode is enabled. Note that if standalone mode is enabled, the membership configurations (`scalar.db.cluster.membership.*`) will be ignored. - **Default value:** `false` ##### `transaction.enabled` - **Field:** `scalar.db.transaction.enabled` - **Description:** Whether the transaction feature is enabled. For example, if you use only the embedding feature, you can set this property to `false`. - **Default value:** `true` ##### `cluster.node.scanner_management.expiration_time_millis` - **Field:** `scalar.db.cluster.node.scanner_management.expiration_time_millis` - **Description:** ScalarDB Cluster nodes maintain in-progress scanners. This process expires scanners that have been idle for an extended period to prevent resource leaks. This configuration specifies the expiration time of this scanner management feature in milliseconds. - **Default value:** `60000` (60 seconds) ##### `cluster.node.grpc.max_connection_age_millis` - **Field:** `scalar.db.cluster.node.grpc.max_connection_age_millis` - **Description:** Maximum time that a channel may exist. It helps proactively close and refresh old connections to prevent imbalance across servers. - **Default value:** `Integer.MAX_VALUE` (Infinite) ##### `cluster.node.grpc.max_connection_age_grace_millis` - **Field:** `scalar.db.cluster.node.grpc.max_connection_age_grace_millis` - **Description:** Grace period after the channel reaches its max age. It provides a grace period for ongoing RPCs to complete before the connection is closed. - **Default value:** `Integer.MAX_VALUE` (Infinite) ### Performance-related configurations The following performance-related configurations are available for the Consensus Commit transaction manager. #### `parallel_executor_count` - **Field:** `scalar.db.consensus_commit.parallel_executor_count` - **Description:** Number of executors (threads) for parallel execution. This number refers to the total number of threads across transactions in a ScalarDB Cluster node or a ScalarDB Core process. - **Default value:** `128` #### `parallel_preparation.enabled` - **Field:** `scalar.db.consensus_commit.parallel_preparation.enabled` - **Description:** Whether or not the preparation phase is executed in parallel. - **Default value:** `true` #### `parallel_validation.enabled` - **Field:** `scalar.db.consensus_commit.parallel_validation.enabled` - **Description:** Whether or not the validation phase (in `EXTRA_READ`) is executed in parallel. - **Default value:** The value of `scalar.db.consensus_commit.parallel_commit.enabled` #### `parallel_commit.enabled` - **Field:** `scalar.db.consensus_commit.parallel_commit.enabled` - **Description:** Whether or not the commit phase is executed in parallel. - **Default value:** `true` #### `parallel_rollback.enabled` - **Field:** `scalar.db.consensus_commit.parallel_rollback.enabled` - **Description:** Whether or not the rollback phase is executed in parallel. - **Default value:** The value of `scalar.db.consensus_commit.parallel_commit.enabled` #### `async_commit.enabled` - **Field:** `scalar.db.consensus_commit.async_commit.enabled` - **Description:** Whether or not the commit phase is executed asynchronously. - **Default value:** `false` #### `async_rollback.enabled` - **Field:** `scalar.db.consensus_commit.async_rollback.enabled` - **Description:** Whether or not the rollback phase is executed asynchronously. - **Default value:** The value of `scalar.db.consensus_commit.async_commit.enabled` #### `parallel_implicit_pre_read.enabled` - **Field:** `scalar.db.consensus_commit.parallel_implicit_pre_read.enabled` - **Description:** Whether or not implicit pre-read is executed in parallel. - **Default value:** `true` #### `one_phase_commit.enabled` - **Field:** `scalar.db.consensus_commit.one_phase_commit.enabled` - **Description:** Whether or not the one-phase commit optimization is enabled. - **Default value:** `false` #### `coordinator.write_omission_on_read_only.enabled` - **Field:** `scalar.db.consensus_commit.coordinator.write_omission_on_read_only.enabled` - **Description:** Whether or not the Coordinator write omission optimization is enabled for read-only transactions. This optimization is useful for read-only transactions that do not modify any data, as it avoids unnecessary writes to the Coordinator tables. - **Default value:** `true` #### `coordinator.group_commit.enabled` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.enabled` - **Description:** Whether or not committing the transaction state is executed in batch mode. This feature can't be used with a two-phase commit interface. - **Default value:** `false` #### `coordinator.group_commit.slot_capacity` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.slot_capacity` - **Description:** Maximum number of slots in a group for the group commit feature. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.[^1] - **Default value:** `20` #### `coordinator.group_commit.group_size_fix_timeout_millis` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.group_size_fix_timeout_millis` - **Description:** Timeout to fix the size of slots in a group. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.[^1] - **Default value:** `40` #### `coordinator.group_commit.delayed_slot_move_timeout_millis` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.delayed_slot_move_timeout_millis` - **Description:** Timeout to move delayed slots from a group to another isolated group to prevent the original group from being affected by delayed transactions. A large value improves the efficiency of group commit, but may also increase the latency and the likelihood of transaction conflicts.[^1] - **Default value:** `1200` #### `coordinator.group_commit.old_group_abort_timeout_millis` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.old_group_abort_timeout_millis` - **Description:** Timeout to abort an old ongoing group. A small value reduces resource consumption through aggressive aborts, but may also increase the likelihood of unnecessary aborts for long-running transactions. - **Default value:** `60000` #### `coordinator.group_commit.timeout_check_interval_millis` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.timeout_check_interval_millis` - **Description:** Interval for checking the group commit–related timeouts. - **Default value:** `20` #### `coordinator.group_commit.metrics_monitor_log_enabled` - **Field:** `scalar.db.consensus_commit.coordinator.group_commit.metrics_monitor_log_enabled` - **Description:** Whether or not the metrics of the group commit are logged periodically. - **Default value:** `false` ### Storage-related configurations ScalarDB has a storage (database) abstraction layer that supports multiple storage implementations. You can specify the storage implementation by using the `scalar.db.storage` property. :::note For details about using multiple storages, see [Multi-storage configurations](#multi-storage-configurations). ::: Select a database to see the configurations available for each storage. The following configurations are available for JDBC databases.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `jdbc` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** JDBC connection URL. - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** Username to access the database. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Password to access the database. - **Default value:** empty :::note The following properties have been removed and will be ignored if set. If these properties are still in your configuration, please remove them to avoid warning messages. - `scalar.db.jdbc.connection_pool.max_idle` - `scalar.db.jdbc.table_metadata.connection_pool.max_idle` - `scalar.db.jdbc.admin.connection_pool.max_idle` - `scalar.db.jdbc.prepared_statements_pool.enabled` - `scalar.db.jdbc.prepared_statements_pool.max_open` :::

`jdbc.connection_pool.min_idle`

- **Field:** `scalar.db.jdbc.connection_pool.min_idle` - **Description:** Minimum number of idle connections in the connection pool. - **Default value:** `20`

`jdbc.connection_pool.max_total`

- **Field:** `scalar.db.jdbc.connection_pool.max_total` - **Description:** Maximum total number of idle and active connections in the connection pool. - **Default value:** `200`

`jdbc.connection_pool.connection_timeout_millis`

- **Field:** `scalar.db.jdbc.connection_pool.connection_timeout_millis` - **Description:** Maximum time in milliseconds to wait for a connection from the pool. - **Default value:** `30000`

`jdbc.connection_pool.idle_timeout_millis`

- **Field:** `scalar.db.jdbc.connection_pool.idle_timeout_millis` - **Description:** Maximum time in milliseconds that a connection is allowed to sit idle in the pool. This setting only applies when `min_idle` is less than `max_total`. A value of `0` means idle connections are never removed. - **Default value:** `600000`

`jdbc.connection_pool.max_lifetime_millis`

- **Field:** `scalar.db.jdbc.connection_pool.max_lifetime_millis` - **Description:** Maximum lifetime in milliseconds of a connection in the pool. Connections that exceed this lifetime will be retired. This value should be set to a few seconds shorter than any database or infrastructure-imposed connection timeout. A value of `0` means no maximum lifetime. - **Default value:** `1800000`

`jdbc.connection_pool.keepalive_time_millis`

- **Field:** `scalar.db.jdbc.connection_pool.keepalive_time_millis` - **Description:** Interval in milliseconds at which the pool will attempt to keep connections alive to prevent them from being timed out by the database or network infrastructure. This value must be less than `max_lifetime_millis`. A value of `0` disables keepalive. - **Default value:** `0`

`jdbc.isolation_level`

- **Field:** `scalar.db.jdbc.isolation_level` - **Description:** Isolation level for JDBC. `READ_COMMITTED`, `REPEATABLE_READ`, or `SERIALIZABLE` can be specified. - **Default value:** Underlying-database specific

`jdbc.table_metadata.connection_pool.min_idle`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.min_idle` - **Description:** Minimum number of idle connections in the connection pool for the table metadata. - **Default value:** `5`

`jdbc.table_metadata.connection_pool.max_total`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.max_total` - **Description:** Maximum total number of idle and active connections in the connection pool for the table metadata. - **Default value:** `25`

`jdbc.table_metadata.connection_pool.connection_timeout_millis`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.connection_timeout_millis` - **Description:** Same as `jdbc.connection_pool.connection_timeout_millis`, but for the table metadata connection pool. - **Default value:** `30000`

`jdbc.table_metadata.connection_pool.idle_timeout_millis`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.idle_timeout_millis` - **Description:** Same as `jdbc.connection_pool.idle_timeout_millis`, but for the table metadata connection pool. - **Default value:** `600000`

`jdbc.table_metadata.connection_pool.max_lifetime_millis`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.max_lifetime_millis` - **Description:** Same as `jdbc.connection_pool.max_lifetime_millis`, but for the table metadata connection pool. - **Default value:** `1800000`

`jdbc.table_metadata.connection_pool.keepalive_time_millis`

- **Field:** `scalar.db.jdbc.table_metadata.connection_pool.keepalive_time_millis` - **Description:** Same as `jdbc.connection_pool.keepalive_time_millis`, but for the table metadata connection pool. - **Default value:** `0`

`jdbc.admin.connection_pool.min_idle`

- **Field:** `scalar.db.jdbc.admin.connection_pool.min_idle` - **Description:** Minimum number of idle connections in the connection pool for admin. - **Default value:** `5`

`jdbc.admin.connection_pool.max_total`

- **Field:** `scalar.db.jdbc.admin.connection_pool.max_total` - **Description:** Maximum total number of idle and active connections in the connection pool for admin. - **Default value:** `25`

`jdbc.admin.connection_pool.connection_timeout_millis`

- **Field:** `scalar.db.jdbc.admin.connection_pool.connection_timeout_millis` - **Description:** Same as `jdbc.connection_pool.connection_timeout_millis`, but for the admin connection pool. - **Default value:** `30000`

`jdbc.admin.connection_pool.idle_timeout_millis`

- **Field:** `scalar.db.jdbc.admin.connection_pool.idle_timeout_millis` - **Description:** Same as `jdbc.connection_pool.idle_timeout_millis`, but for the admin connection pool. - **Default value:** `600000`

`jdbc.admin.connection_pool.max_lifetime_millis`

- **Field:** `scalar.db.jdbc.admin.connection_pool.max_lifetime_millis` - **Description:** Same as `jdbc.connection_pool.max_lifetime_millis`, but for the admin connection pool. - **Default value:** `1800000`

`jdbc.admin.connection_pool.keepalive_time_millis`

- **Field:** `scalar.db.jdbc.admin.connection_pool.keepalive_time_millis` - **Description:** Same as `jdbc.connection_pool.keepalive_time_millis`, but for the admin connection pool. - **Default value:** `0`

`jdbc.db2.variable_key_column_size`

- **Field:** `scalar.db.jdbc.db2.variable_key_column_size` - **Description:** Column size for TEXT and BLOB columns in IBM Db2 when they are used as a primary key or secondary key. Minimum 64 bytes. - **Default value:** `128`

`jdbc.db2.time_column.default_date_component`

- **Field:** `scalar.db.jdbc.db2.time_column.default_date_component` - **Description:** Value of the date component used for storing `TIME` data in IBM Db2. Since the IBM Db2 TIMESTAMP type is used to store ScalarDB `TIME` type data because it provides fractional-second precision, ScalarDB stores `TIME` data with the same date component value for ease of comparison and sorting. - **Default value:** `1970-01-01`

`jdbc.spanner.time_column.default_date_component`

- **Field:** `scalar.db.jdbc.spanner.time_column.default_date_component` - **Description:** Value of the date component used for storing `TIME` data in Spanner. Because Spanner's PostgreSQL dialect has no native TIME type, ScalarDB stores `TIME` data as Spanner `TIMESTAMP WITH TIME ZONE` data with a fixed date component to enable comparison and sorting. - **Default value:** `1970-01-01` :::note **SQLite3** If you're using SQLite3 as a JDBC database, you must set `scalar.db.contact_points` as follows: ```properties scalar.db.contact_points=jdbc:sqlite:?busy_timeout=10000 ``` Unlike other JDBC databases, [SQLite3 doesn't fully support concurrent access](https://www.sqlite.org/lang_transaction.html). To avoid frequent errors caused internally by [`SQLITE_BUSY`](https://www.sqlite.org/rescode.html#busy), we recommend setting a [`busy_timeout`](https://www.sqlite.org/c3ref/busy_timeout.html) parameter. **Spanner** Authentication to Spanner requires using a Google Cloud service account key in JSON format. Set `scalar.db.password` to the full content of the service account key file as a single line JSON. The `scalar.db.username` property is unused for Spanner. ScalarDB also sets the JVM system property `ENABLE_CREDENTIALS_PROVIDER=true`, which is required by the Spanner JDBC driver to authenticate. For example: ```properties scalar.db.storage=jdbc scalar.db.contact_points=jdbc:cloudspanner:/projects//instances//databases/ scalar.db.username= scalar.db.password= ``` ::: The following configurations are available for DynamoDB.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `dynamo` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** AWS region with which ScalarDB should communicate (for example, `us-east-1`). - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** AWS access key used to identify the user interacting with AWS. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** AWS secret access key used to authenticate the user interacting with AWS. - **Default value:** empty

`dynamo.endpoint_override`

- **Field:** `scalar.db.dynamo.endpoint_override` - **Description:** Amazon DynamoDB endpoint with which ScalarDB should communicate. This is primarily used for testing with a local instance instead of an AWS service. - **Default value:** empty

`dynamo.namespace.prefix`

- **Field:** `scalar.db.dynamo.namespace.prefix` - **Description:** Prefix for the user namespaces and metadata namespace names. Since AWS requires having unique tables names in a single AWS region, this is useful if you want to use multiple ScalarDB environments (development, production, etc.) in a single AWS region. - **Default value:** empty The following configurations are available for Cosmos DB for NoSQL.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `cosmos` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** Azure Cosmos DB for NoSQL endpoint with which ScalarDB should communicate. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Either a master or read-only key used to perform authentication for accessing Azure Cosmos DB for NoSQL. - **Default value:** empty

`cosmos.consistency_level`

- **Field:** `scalar.db.cosmos.consistency_level` - **Description:** Consistency level used for Cosmos DB operations. `STRONG` or `BOUNDED_STALENESS` can be specified. - **Default value:** `STRONG` The following configurations are available for Cassandra.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `cassandra` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** Comma-separated contact points. - **Default value:** empty

`contact_port`

- **Field:** `scalar.db.contact_port` - **Description:** Port number for all the contact points. - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** Username to access the database. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Password to access the database. - **Default value:** empty The following configurations are available for S3.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `s3` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** '/'-separated region and S3 bucket name (for example, `us-east-1/my-bucket`). - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** AWS access key. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** AWS secret access key. - **Default value:** empty

`s3.multipart_upload_part_size_bytes`

- **Field:** `scalar.db.s3.multipart_upload_part_size_bytes` - **Description:** The part size in bytes for multipart upload. - **Default value:** The default value of [`minimumPartSizeInBytes`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/multipart/MultipartConfiguration.html#minimumPartSizeInBytes()) in the AWS SDK.

`s3.multipart_upload_max_concurrency`

- **Field:** `scalar.db.s3.multipart_upload_max_concurrency` - **Description:** The maximum number of concurrent requests allowed for multipart upload. - **Default value:** The default value of [`maxConcurrency`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/http/crt/AwsCrtAsyncHttpClient.Builder.html#maxConcurrency(java.lang.Integer)) in the AWS SDK.

`s3.multipart_upload_threshold_size_bytes`

- **Field:** `scalar.db.s3.multipart_upload_threshold_size_bytes` - **Description:** The threshold size in bytes to enable multipart upload. If the object size is greater than or equal to this value, multipart upload is used. - **Default value:** The default value of [`thresholdInBytes`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/multipart/MultipartConfiguration.html#thresholdInBytes()) in the AWS SDK.

`s3.request_timeout_secs`

- **Field:** `scalar.db.s3.request_timeout_secs` - **Description:** The request timeout in seconds for S3 operations set to [`apiCallTimeout`](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/client/config/ClientOverrideConfiguration.Builder.html#apiCallTimeout(java.time.Duration)) in the AWS SDK. - **Default value:** empty (no timeout) The following configurations are available for Blob Storage.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `blob-storage` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** Blob Storage endpoint URL including the container name (for example, `https://.blob.core.windows.net/my-container`). - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** Azure Storage account name. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Azure Storage account key. - **Default value:** empty

`blob_storage.parallel_upload_block_size_bytes`

- **Field:** `scalar.db.blob_storage.parallel_upload_block_size_bytes` - **Description:** The block size in bytes for parallel upload. - **Default value:** The default value of [`setBlockSizeLong`](https://learn.microsoft.com/en-us/java/api/com.azure.storage.blob.models.paralleltransferoptions?view=azure-java-stable#com-azure-storage-blob-models-paralleltransferoptions-setblocksizelong(java-lang-long)) in the Azure SDK.

`blob_storage.parallel_upload_max_concurrency`

- **Field:** `scalar.db.blob_storage.parallel_upload_max_concurrency` - **Description:** The maximum number of concurrent requests allowed for parallel upload. - **Default value:** The default value of [`setMaxConcurrency`](https://learn.microsoft.com/en-us/java/api/com.azure.storage.blob.models.paralleltransferoptions?view=azure-java-stable#com-azure-storage-blob-models-paralleltransferoptions-setmaxconcurrency(java-lang-integer)) in the Azure SDK.

`blob_storage.parallel_upload_threshold_size_bytes`

- **Field:** `scalar.db.blob_storage.parallel_upload_threshold_size_bytes` - **Description:** The threshold size in bytes to enable parallel upload. If the object size is greater than this value, parallel upload is used. - **Default value:** The default value of [`setMaxSingleUploadSizeLong`](https://learn.microsoft.com/en-us/java/api/com.azure.storage.blob.models.paralleltransferoptions?view=azure-java-stable#com-azure-storage-blob-models-paralleltransferoptions-setmaxsingleuploadsizelong(java-lang-long)) in the Azure SDK.

`blob_storage.request_timeout_secs`

- **Field:** `scalar.db.blob_storage.request_timeout_secs` - **Description:** The request timeout in seconds for Blob Storage operations. - **Default value:** empty (no timeout) The following configurations are available for Cloud Storage.

`storage`

- **Field:** `scalar.db.storage` - **Description:** `cloud-storage` must be specified.

`contact_points`

- **Field:** `scalar.db.contact_points` - **Description:** Cloud Storage bucket name. - **Default value:** empty

`username`

- **Field:** `scalar.db.username` - **Description:** Google Cloud project ID. - **Default value:** empty

`password`

- **Field:** `scalar.db.password` - **Description:** Full content of the Google Cloud service account key file as a single-line JSON. - **Default value:** empty

`cloud_storage.upload_chunk_size_bytes`

- **Field:** `scalar.db.cloud_storage.upload_chunk_size_bytes` - **Description:** The chunk size in bytes for upload. - **Default value:** The default value of [`setChunkSize`](https://docs.cloud.google.com/java/docs/reference/google-cloud-core/latest/com.google.cloud.WriteChannel#com_google_cloud_WriteChannel_setChunkSize_int_) in the Google Cloud SDK. #### Multi-storage configurations ScalarDB supports using multiple storage implementations simultaneously. For details about using multiple storages, see [Multi-Storage Transactions](../multi-storage-transactions.mdx). ##### `storage` - **Field:** `scalar.db.storage` - **Description:** `multi-storage` must be specified. ##### `multi_storage.storages` - **Field:** `scalar.db.multi_storage.storages` - **Description:** Comma-separated storage names (for example, `cassandra,mysql`). These storage names will be used in the `scalar.db.multi_storage.namespace_mapping` property to map namespaces to storages. - **Default value:** empty ##### `multi_storage.default_storage` - **Field:** `scalar.db.multi_storage.default_storage` - **Description:** Default storage name. This storage will be used for any namespace that doesn't have mapping defined in the `scalar.db.multi_storage.namespace_mapping` property. - **Default value:** empty ##### `multi_storage.namespace_mapping` - **Field:** `scalar.db.multi_storage.namespace_mapping` - **Description:** Mapping of namespaces to storages (for example, `user:my_cassandra,coordinator:my_mysql`). - **Default value:** empty :::tip The storage names (``) are arbitrary values that you need to define. You can use any names that you like as long as they are consistent across the multi-storage configurations. ::: ##### `multi_storage.storages..` For configuring specific storages, use `scalar.db.multi_storage.storages..`, with `` being one of the storage names specified in the `scalar.db.multi_storage.storages` property and `` being the property name for the specific storage. For example, if you've defined [namespace mapping](#multi_storagenamespace_mapping) as `scalar.db.multi_storage.namespace_mapping=user:my_cassandra,coordinator:my_mysql`, with `my_cassandra` and `my_mysql` being the storage names for the `user` and `coordinator` namespaces, respectively: - You can specify the contact points for Cassandra by using `scalar.db.multi_storage.storages.my_cassandra.contact_points`. - You can specify the minimum number of idle connections in the connection pool for MySQL by using `scalar.db.multi_storage.storages.my_mysql.jdbc.connection_pool.min_idle`. For details about the properties available for each storage, see [Storage-related configurations](#storage-related-configurations). #### Cross-partition scan configurations By enabling the cross-partition scan option as described below, the `Scan` operation can retrieve all records across partitions. In addition, you can specify arbitrary conditions and orderings in the cross-partition `Scan` operation by enabling `cross_partition_scan.filtering` and `cross_partition_scan.ordering`, respectively. Currently, the cross-partition scan with ordering option is available only for JDBC databases. To enable filtering and ordering, `scalar.db.cross_partition_scan.enabled` must be set to `true`. For details on how to use cross-partition scan, see [Scan operation](../api-guide.mdx#scan-operation). :::warning For non-JDBC databases, we do not recommend enabling cross-partition scan with the `SERIALIAZABLE` isolation level because transactions could be executed at a lower isolation level (that is, `SNAPSHOT`). When using non-JDBC databases, use cross-partition scan at your own risk only if consistency does not matter for your transactions. ::: ##### `cross_partition_scan.enabled` - **Field:** `scalar.db.cross_partition_scan.enabled` - **Description:** Enable cross-partition scan. - **Default value:** `true` ##### `cross_partition_scan.filtering.enabled` - **Field:** `scalar.db.cross_partition_scan.filtering.enabled` - **Description:** Enable filtering in cross-partition scan. - **Default value:** `false` ##### `cross_partition_scan.ordering.enabled` - **Field:** `scalar.db.cross_partition_scan.ordering.enabled` - **Description:** Enable ordering in cross-partition scan. - **Default value:** `false` #### Scan configurations You can configure the fetch size for storage scan operations by using the following property. ##### `scan_fetch_size` - **Field:** `scalar.db.scan_fetch_size` - **Description:** Specifies the number of records to fetch in a single batch during a storage scan operation. A larger value can improve performance for a large result set by reducing round trips to the storage, but it also increases memory usage. A smaller value uses less memory but may increase latency. - **Default value:** `10` ### GraphQL-related configurations The configurations for ScalarDB Cluster GraphQL are as follows: #### `graphql.enabled` - **Field:** `scalar.db.graphql.enabled` - **Description:** Whether ScalarDB Cluster GraphQL is enabled. - **Default value:** `false` #### `graphql.port` - **Field:** `scalar.db.graphql.port` - **Description:** Port number of the GraphQL server. - **Default value:** `8080` #### `graphql.path` - **Field:** `scalar.db.graphql.path` - **Description:** Path component of the URL of the GraphQL endpoint. - **Default value:** `/graphql` #### `graphql.namespaces` - **Field:** `scalar.db.graphql.namespaces` - **Description:** Comma-separated list of namespaces of tables for which the GraphQL server generates a schema. If not specified, the GraphQL server generates a schema for all tables in all ScalarDB-managed namespaces. - **Default value:** empty #### `graphql.graphiql` - **Field:** `scalar.db.graphql.graphiql` - **Description:** Whether the GraphQL server serves [GraphiQL](https://github.com/graphql/graphiql) IDE. - **Default value:** `true` #### `graphql.schema_checking_interval_millis` - **Field:** `scalar.db.graphql.schema_checking_interval_millis` - **Description:** Interval in milliseconds at which GraphQL server will rebuild the GraphQL schema if any change is detected in the ScalarDB schema. - **Default value:** `30000` (30 seconds) #### Creating or modifying the ScalarDB schema when the server is running Since the GraphQL schema is statically built at server startup, if the ScalarDB schema is modified (for example, if a table is added, altered, or deleted), then the corresponding GraphQL schema won't reflect the changes unless it is rebuilt. To address this, the GraphQL server provides two mechanisms: a periodic check and an on-demand check. ##### Run periodic checks The server periodically checks if changes in the ScalarDB schema occur and rebuilds the corresponding GraphQL schema if necessary. By default, the check occurs every 30 seconds, but the interval can be configured by using the `scalar.db.graphql.schema_checking_interval_millis` property. If you don't need to run periodic checks, you can disable it by setting the property value to `-1`. ##### Run on-demand checks You can also request the server to check changes in the ScalarDB schema and rebuild the corresponding GraphQL schema if necessary by performing a POST request to the `/update-graphql-schema` endpoint of the HTTP API. For example, if the HTTP API is running on `localhost:8080` and the `scalar.db.graphql.path` property is set to `/graphql`, this endpoint can be called by running the following command: ```console curl -X POST http://localhost:8080/graphql/update-graphql-schema ``` ### SQL-related configurations The configurations for ScalarDB Cluster SQL are as follows: #### `sql.enabled` - **Field:** `scalar.db.sql.enabled` - **Description:** Whether ScalarDB Cluster SQL is enabled. - **Default value:** `false` #### `sql.statement_cache.enabled` - **Field:** `scalar.db.sql.statement_cache.enabled` - **Description:** Enable the statement cache. - **Default value:** `false` #### `sql.statement_cache.size` - **Field:** `scalar.db.sql.statement_cache.size` - **Description:** Maximum number of cached statements. - **Default value:** `100` #### `sql.default_transaction_mode` - **Field:** `scalar.db.sql.default_transaction_mode` - **Description:** Default transaction mode. `TRANSACTION` or `TWO_PHASE_COMMIT_TRANSACTION` can be set. - **Default value:** `TRANSACTION` #### `sql.default_namespace_name` - **Field:** `scalar.db.sql.default_namespace_name` - **Description:** Default namespace name. If you don't specify a namespace name in your SQL statement, this value is used. - **Default value:** empty ### Authentication and authorization configurations The following shows the authentication and authorization configurations for ScalarDB Cluster. #### `auth.enabled` - **Field:** `scalar.db.cluster.auth.enabled` - **Description:** Whether authentication and authorization are enabled. - **Default value:** `false` #### `auth.cache_expiration_time_millis` - **Field:** `scalar.db.cluster.auth.cache_expiration_time_millis` - **Description:** Cache expiration time for authentication and authorization information in milliseconds. - **Default value:** `60000` (1 minute) #### `auth.auth_token_expiration_time_minutes` - **Field:** `scalar.db.cluster.auth.auth_token_expiration_time_minutes` - **Description:** Authentication and authorization token expiration time in minutes. - **Default value:** `1440` (1 day) #### `auth.auth_token_gc_thread_interval_minutes` - **Field:** `scalar.db.cluster.auth.auth_token_gc_thread_interval_minutes` - **Description:** Authentication and authorization token garbage collection (GC) thread interval in minutes. - **Default value:** `360` (6 hours) #### `auth.pepper` - **Field:** `scalar.db.cluster.auth.pepper` - **Description:** A secret value added to a password before hashing. If not specified, the password is hashed without pepper. - **Default value:** empty ### OIDC configurations The following configurations are available for the OIDC integration in ScalarDB Cluster. For details about the OIDC integration, see [Control User Access via OIDC-Based JWT Access Tokens](./control-access-via-oidc-based-jwt-tokens.mdx). #### `auth.oidc.trusted_issuers` - **Field:** `scalar.db.cluster.auth.oidc.trusted_issuers` - **Description:** The trusted OIDC issuer URL. ScalarDB Cluster rejects tokens whose `iss` claim does not exactly match this value. This property must be specified when using the OIDC integration. Currently, only a single issuer URL is supported. For example, `http://localhost:8080/realms/my-realm` for Keycloak. - **Default value:** empty #### `auth.oidc.audience.name` - **Field:** `scalar.db.cluster.auth.oidc.audience.name` - **Description:** The expected value in the JWT `aud` claim. ScalarDB Cluster rejects tokens whose `aud` does not contain this value. - **Default value:** `scalardb` #### `auth.oidc.username.claim_name` - **Field:** `scalar.db.cluster.auth.oidc.username.claim_name` - **Description:** The JWT claim name used to extract the ScalarDB username. This property must be specified when using the OIDC integration. - **Default value:** empty #### `auth.oidc.jwt.access_token.require_at_jwt_typ` - **Field:** `scalar.db.cluster.auth.oidc.jwt.access_token.require_at_jwt_typ` - **Description:** Whether to require the JWT `typ` header to be `at+jwt` or `application/at+jwt` per RFC 9068. Set to `false` only for development purposes. - **Default value:** `true` #### `auth.oidc.jwt.jwks.url_cache.expiration_seconds` - **Field:** `scalar.db.cluster.auth.oidc.jwt.jwks.url_cache.expiration_seconds` - **Description:** The cache expiration time, in seconds, for the JWKS URL obtained from the OIDC provider configuration. - **Default value:** `86400` (24 hours) #### `auth.oidc.jwt.jwks.content_cache.expiration_seconds` - **Field:** `scalar.db.cluster.auth.oidc.jwt.jwks.content_cache.expiration_seconds` - **Description:** The cache expiration time, in seconds, for the fetched JWKS content. - **Default value:** `86400` (24 hours) #### `auth.oidc.jwt.max_clock_skew_seconds` - **Field:** `scalar.db.cluster.auth.oidc.jwt.max_clock_skew_seconds` - **Description:** The maximum clock skew tolerance, in seconds, for JWT expiration validation. - **Default value:** `10` ### Data-at-rest encryption configurations The following configurations are available for encrypting data at rest in ScalarDB Cluster. For details about encrypting data at rest, see [Encrypt Data at Rest](./encrypt-data-at-rest.mdx). #### `cluster.encryption.enabled` - **Field:** `scalar.db.cluster.encryption.enabled` - **Description:** Whether ScalarDB encrypts data at rest. - **Default value:** `false` #### `cluster.encryption.type` - **Field:** `scalar.db.cluster.encryption.type` - **Description:** Encryption implementation type. Either `vault` (for HashiCorp Vault encryption) or `self` (for self-encryption) can be specified. - **Default value:** empty #### `cluster.encryption.delete_data_encryption_key_on_drop_table.enabled` - **Field:** `scalar.db.cluster.encryption.delete_data_encryption_key_on_drop_table.enabled` - **Description:** Whether to delete the data encryption key (DEK) when dropping a table. - **Default value:** `false` #### HashiCorp Vault encryption configurations The following configurations are available when using HashiCorp Vault encryption (`scalar.db.cluster.encryption.type=vault`). ##### `cluster.encryption.vault.key_type` - **Field:** `scalar.db.cluster.encryption.vault.key_type` - **Description:** The key type. Currently, `aes128-gcm96`, `aes256-gcm96`, and `chacha20-poly1305` are supported. For details about the key types, see [Key types](https://developer.hashicorp.com/vault/docs/secrets/transit#key-types). - **Default value:** `aes128-gcm96` ##### `cluster.encryption.vault.associated_data_required` - **Field:** `scalar.db.cluster.encryption.vault.associated_data_required` - **Description:** Whether associated data is required for AEAD encryption. - **Default value:** `false` ##### `cluster.encryption.vault.address` - **Field:** `scalar.db.cluster.encryption.vault.address` - **Description:** The address of the HashiCorp Vault server. - **Default value:** empty ##### `cluster.encryption.vault.token` - **Field:** `scalar.db.cluster.encryption.vault.token` - **Description:** The token to authenticate with HashiCorp Vault. - **Default value:** empty ##### `cluster.encryption.vault.namespace` - **Field:** `scalar.db.cluster.encryption.vault.namespace` - **Description:** The namespace of the HashiCorp Vault. This configuration is optional. - **Default value:** empty ##### `cluster.encryption.vault.transit_secrets_engine_path` - **Field:** `scalar.db.cluster.encryption.vault.transit_secrets_engine_path` - **Description:** The path of the transit secrets engine. - **Default value:** `transit` ##### `cluster.encryption.vault.column_batch_size` - **Field:** `scalar.db.cluster.encryption.vault.column_batch_size` - **Description:** The number of columns to be included in a single request to the HashiCorp Vault server. - **Default value:** `64` #### Self-encryption configurations The following configurations are available when using self-encryption (`scalar.db.cluster.encryption.type=self`). ##### `cluster.encryption.self.key_type` - **Field:** `scalar.db.cluster.encryption.self.key_type` - **Description:** The key type. Currently, `AES128_GCM`, `AES256_GCM`, `AES128_EAX`, `AES256_EAX`, `AES128_CTR_HMAC_SHA256`, `AES256_CTR_HMAC_SHA256`, `CHACHA20_POLY1305`, and `XCHACHA20_POLY1305` are supported. For details about the key types, see [Choose a key type](https://developers.google.com/tink/aead#choose_a_key_type). - **Default value:** `AES128_GCM` ##### `cluster.encryption.self.associated_data_required` - **Field:** `scalar.db.cluster.encryption.self.associated_data_required` - **Description:** Whether associated data is required for AEAD encryption. - **Default value:** `false` ##### `cluster.encryption.self.kubernetes.secret.namespace_name` - **Field:** `scalar.db.cluster.encryption.self.kubernetes.secret.namespace_name` - **Description:** The namespace name of the Kubernetes Secrets. - **Default value:** `default` ##### `cluster.encryption.self.data_encryption_key_cache_expiration_time` - **Field:** `scalar.db.cluster.encryption.self.data_encryption_key_cache_expiration_time` - **Description:** The expiration time of the DEK cache in milliseconds. - **Default value:** `60000` (60 seconds) ### Wire encryption configurations The following configurations are available for encrypting wire communications in ScalarDB Cluster. For details about encrypting wire communications, see [Encrypt Wire Communications](./encrypt-wire-communications.mdx). #### `cluster.tls.enabled` - **Field:** `scalar.db.cluster.tls.enabled` - **Description:** Whether wire encryption (TLS) is enabled. - **Default value:** `false` #### `cluster.tls.ca_root_cert_pem` - **Field:** `scalar.db.cluster.tls.ca_root_cert_pem` - **Description:** The custom CA root certificate (PEM data) for TLS communication. - **Default value:** empty #### `cluster.tls.ca_root_cert_path` - **Field:** `scalar.db.cluster.tls.ca_root_cert_path` - **Description:** The custom CA root certificate (file path) for TLS communication. - **Default value:** empty #### `cluster.tls.override_authority` - **Field:** `scalar.db.cluster.tls.override_authority` - **Description:** The custom authority for TLS communication. This doesn't change what host is actually connected. This is intended for testing, but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set for `scalar.db.cluster.node.tls.cert_chain_path`. - **Default value:** empty #### `cluster.node.tls.cert_chain_path` - **Field:** `scalar.db.cluster.node.tls.cert_chain_path` - **Description:** The certificate chain file used for TLS communication. - **Default value:** empty #### `cluster.node.tls.private_key_path` - **Field:** `scalar.db.cluster.node.tls.private_key_path` - **Description:** The private key file used for TLS communication. - **Default value:** empty ### Remote replication configurations The following configurations are available for remote replication in ScalarDB Cluster. Remote replication enables data replication to remote sites for high availability and workload distribution. For details about using remote replication, see [Replicate Data for High Availability](./remote-replication.mdx). #### Base replication configurations The following configurations apply to the overall remote replication setup in ScalarDB Cluster. ##### `partition_count` - **Field:** `scalar.db.replication.partition_count` - **Description:** Number of partitions for the `transaction_groups` table. The tables in the replication database are partitioned for performance and scalability, and write operations are distributed evenly across partitions. This field must be identical between primary and backup sites. - **Default value:** `256` :::warning Changing the partition count requires restarting ScalarDB Clusters in both sites. ::: ##### `repl_db.namespace` - **Field:** `scalar.db.replication.repl_db.namespace` - **Description:** Namespace name of replication tables. This field must be identical between primary and backup sites. - **Default value:** `replication` ##### `record_table_suffix` - **Field:** `scalar.db.replication.record_table_suffix` - **Description:** Suffix for replication record metadata tables. - **Default value:** `__records` #### LogWriter configurations (primary site) LogWriter configurations control how write operations are captured and stored in the replication database during transaction commits. ##### `log_writer.enabled` - **Field:** `scalar.db.replication.log_writer.enabled` - **Description:** Enable or disable LogWriter functionality. - **Default value:** `false` ##### `log_writer.compression_type` - **Field:** `scalar.db.replication.log_writer.compression_type` - **Description:** Compression type for stored write operations in the replication database. Available values: `NONE`, `GZIP`. - **Default value:** `GZIP` ##### `log_writer.group_commit.retention.time_millis` - **Field:** `scalar.db.replication.log_writer.group_commit.retention.time_millis` - **Description:** Maximum time to wait before committing a transaction group for the replication database. - **Default value:** `100` (100 milliseconds) ##### `log_writer.group_commit.retention.values` - **Field:** `scalar.db.replication.log_writer.group_commit.retention.values` - **Description:** Maximum number of transactions to batch together for the replication database. - **Default value:** `32` ##### `log_writer.group_commit.timeout_check_interval_millis` - **Field:** `scalar.db.replication.log_writer.group_commit.timeout_check_interval_millis` - **Description:** Interval for checking group commit timeouts for the replication database. - **Default value:** `20` (20 milliseconds) ##### `log_writer.group_commit.max_thread_pool_size` - **Field:** `scalar.db.replication.log_writer.group_commit.max_thread_pool_size` - **Description:** Maximum thread pool size for group commit processing for the replication database. - **Default value:** `4096` #### LogApplier configurations (backup site) LogApplier configurations control how replication data is processed and applied to the backup site tables. ##### `log_applier.enabled` - **Field:** `scalar.db.replication.log_applier.enabled` - **Description:** Enable or disable LogApplier functionality. - **Default value:** `false` ##### `log_applier.transaction.expiration_millis` - **Field:** `scalar.db.replication.log_applier.transaction.expiration_millis` - **Description:** Expiration time in milliseconds for transactions replicated to the replication database from the primary site. - **Default value:** `30000` (30 seconds) ##### `log_applier.transaction_group_scanner.threads` - **Field:** `scalar.db.replication.log_applier.transaction_group_scanner.threads` - **Description:** Number of scanner threads for transaction groups in the replication database. - **Default value:** `16` ##### `log_applier.transaction_group_scanner.fetch_size` - **Field:** `scalar.db.replication.log_applier.transaction_group_scanner.fetch_size` - **Description:** Number of transaction group records to fetch in each scan operation from the replication database. - **Default value:** `32` ##### `log_applier.transaction_group_scanner.wait_millis` - **Field:** `scalar.db.replication.log_applier.transaction_group_scanner.wait_millis` - **Description:** Wait time in milliseconds between scans of transaction groups in the replication database. Increasing this value reduces scan frequency but may increase replication latency. - **Default value:** `1000` (1 second) ##### `log_applier.transaction_group_scanner.dedup.expiration_millis` - **Field:** `scalar.db.replication.log_applier.transaction_group_scanner.dedup.expiration_millis` - **Description:** Expiration time for transaction group deduplication cache. LogApplier uses this cache to avoid reprocessing the same transaction groups from the replication database. - **Default value:** `10000` (10 seconds) ##### `log_applier.transaction_handler.threads` - **Field:** `scalar.db.replication.log_applier.transaction_handler.threads` - **Description:** Number of threads for LogApplier transaction handling. Increasing this value may improve replication performance but increases resource consumption. - **Default value:** `128` ##### `log_applier.max_record_version` - **Field:** `scalar.db.replication.log_applier.max_record_version` - **Description:** Maximum record version to prevent integer overflow (for development use). - **Default value:** `Integer.MAX_VALUE - 1000` (where `Integer.MAX_VALUE` is 2,147,483,647) ##### `log_applier.write_operation.expiration_millis` - **Field:** `scalar.db.replication.log_applier.write_operation.expiration_millis` - **Description:** Expiration time in milliseconds for write operation metadata stored in the replication record metadata tables. After this time, expired metadata is eligible for garbage collection. - **Default value:** `86400000` (1 day) ##### `log_applier.replication_status_service.threads` - **Field:** `scalar.db.replication.log_applier.replication_status_service.threads` - **Description:** Number of threads for replication status service. - **Default value:** `16` ##### `log_applier.coordinator_state_cache.expiration_millis` - **Field:** `scalar.db.replication.log_applier.coordinator_state_cache.expiration_millis` - **Description:** Expiration time in milliseconds for the coordinator state cache. LogApplier caches transaction states from the Coordinator database to reduce lookups. A lower value means more frequent checks against the Coordinator database. - **Default value:** `30000` (30 seconds) ##### `log_applier.coordinator_state_cache.size` - **Field:** `scalar.db.replication.log_applier.coordinator_state_cache.size` - **Description:** Maximum number of entries in the coordinator state cache. This cache stores transaction states to optimize lookups to the Coordinator database. - **Default value:** `1000` ### Other ScalarDB Cluster configurations The following are additional configurations available for ScalarDB Cluster. #### `metadata.cache_expiration_time_secs` - **Field:** `scalar.db.metadata.cache_expiration_time_secs` - **Description:** ScalarDB has a metadata cache to reduce the number of requests to the database. This setting specifies the expiration time of the cache in seconds. If you specify `-1`, the cache will never expire. - **Default value:** `60` #### `active_transaction_management.expiration_time_millis` - **Field:** `scalar.db.active_transaction_management.expiration_time_millis` - **Description:** ScalarDB maintains in-progress transactions, which can be resumed by using a transaction ID. This process expires transactions that have been idle for an extended period to prevent resource leaks. This setting specifies the expiration time of this transaction management feature in milliseconds. - **Default value:** `60000` (60 seconds) #### `consensus_commit.include_metadata.enabled` - **Field:** `scalar.db.consensus_commit.include_metadata.enabled` - **Description:** When using Consensus Commit, if this is set to `true`, `Get` and `Scan` operations results will contain transaction metadata. To see the transaction metadata columns details for a given table, you can use the `DistributedTransactionAdmin.getTableMetadata()` method, which will return the table metadata augmented with the transaction metadata columns. Using this configuration can be useful to investigate transaction-related issues. - **Default value:** `false` #### `consensus_commit.index.eventually_consistent_read.enabled` - **Field:** `scalar.db.consensus_commit.index.eventually_consistent_read.enabled` - **Description:** When using Consensus Commit, if this is set to `true`, the before-image index check will be skipped, and index-based reads may miss records whose indexed column is being concurrently updated. - **Default value:** `false` :::warning This is a backward-compatibility option and is **not recommended for new workloads**. For details, see [Correctness of index-based reads](../consensus-commit.mdx#correctness-of-index-based-reads). ::: #### `default_namespace_name` - **Field:** `scalar.db.default_namespace_name` - **Description:** The given namespace name will be used by operations that do not already specify a namespace. - **Default value:** empty This section describes the configurations for the ScalarDB Cluster Java Client SDK. ## Java Client SDK configurations This section describes the configurations for the ScalarDB Cluster Java Client SDK. ### Configurations for the primitive interface The following shows the general configurations for the Java Client SDK when using the primitive interface. #### `transaction_manager` - **Field:** `scalar.db.transaction_manager` - **Description:** `cluster` should be specified. - **Default value:** empty #### `contact_points` - **Field:** `scalar.db.contact_points` - **Description:** Contact point of the cluster. If you use the `indirect` client mode, specify the IP address of the load balancer in front of your cluster nodes by using the format `indirect:`. If you use the `direct-kubernetes` client mode, specify the namespace name (optional) and the name of the [endpoint resource](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints) to get the membership information by using the format `direct-kubernetes:/` or just `direct-kubernetes:`. If you don't specify the namespace name, the Java Client SDK will use the `default` namespace. - **Default value:** empty :::note For example, if you use the `indirect` client mode and the load balancer IP address is `192.168.10.1`, you can configure the Java Client SDK as follows: ```properties scalar.db.transaction_manager=cluster scalar.db.contact_points=indirect:192.168.10.1 ``` Or if you use the `direct-kubernetes` client mode, with the namespace of the endpoint as `ns` and the endpoint name as `scalardb-cluster`, you can configure the Java Client SDK as follows: ```properties scalar.db.transaction_manager=cluster scalar.db.contact_points=direct-kubernetes:ns/scalardb-cluster ``` ::: #### `contact_port` - **Field:** `scalar.db.contact_port` - **Description:** Port number for the contact point. - **Default value:** `60053` #### `cluster.grpc.deadline_duration_millis` - **Field:** `scalar.db.cluster.grpc.deadline_duration_millis` - **Description:** Deadline duration for gRPC in millis. - **Default value:** `60000` (60 seconds) #### `cluster.grpc.max_inbound_message_size` - **Field:** `scalar.db.cluster.grpc.max_inbound_message_size` - **Description:** Maximum message size allowed for a single gRPC frame. - **Default value:** The gRPC default value #### `cluster.grpc.max_inbound_metadata_size` - **Field:** `scalar.db.cluster.grpc.max_inbound_metadata_size` - **Description:** Maximum size of metadata allowed to be received. - **Default value:** The gRPC default value #### `cluster.client.scan_fetch_size` - **Field:** `scalar.db.cluster.client.scan_fetch_size` - **Description:** The fetch size used for `Scanner` to fetch data from the cluster. This is the number of records that `Scanner` fetches at once from the cluster. A larger value can improve performance by reducing the number of round trips to the cluster, but it may also increase memory usage. - **Default value:** `10` #### `cluster.client.piggyback_begin.enabled` - **Field:** `scalar.db.cluster.client.piggyback_begin.enabled` - **Description:** Whether the piggyback-begin feature is enabled. It delays transaction begin until the first CRUD operation, piggybacking the begin with the first operation to eliminate a dedicated begin RPC call. - **Default value:** `false` #### `cluster.client.write_buffering.enabled` - **Field:** `scalar.db.cluster.client.write_buffering.enabled` - **Description:** Whether the write-buffering feature is enabled. It buffers non-conditional write operations (inserts, upserts, unconditional puts/updates/deletes) and executes them in batches to reduce the number of RPC calls. - **Default value:** `false` :::warning * If `piggyback_begin` is enabled, you will get an `IllegalStateException` when calling the `DistributedTransaction.getId()` method until the actual begin operation is executed. * If `piggyback_begin` or `write_buffering` is enabled, you will always get an `IllegalStateException` when calling the `DistributedTransactionManager.resume()` and `DistributedTransactionManager.join()` methods. ::: #### Authentication and authorization configurations The following shows the authentication and authorization configurations for the Java Client SDK when using the primitive interface. When you run applications or tools that use the Java Client SDK and the primitive interface (such as the [Schema Loader for Cluster](./developer-guide-for-scalardb-cluster-with-java-api.mdx#schema-loader-for-cluster) or [ScalarDB Benchmarking Tools](../scalardb-benchmarks/README.mdx)), you need to set the following configurations in your client-side `database.properties` file, especially if authentication and authorization are enabled on the cluster. ##### `auth.enabled` - **Field:** `scalar.db.cluster.auth.enabled` - **Description:** Whether authentication and authorization are enabled. - **Default value:** `false` ##### `username` - **Field:** `scalar.db.username` - **Description:** Username to access ScalarDB Cluster. Not required when using OIDC JWT authentication. - **Default value:** empty ##### `password` - **Field:** `scalar.db.password` - **Description:** Password to access ScalarDB Cluster. Not required when using OIDC JWT authentication. - **Default value:** empty ##### `cluster.client.auth.type` - **Field:** `scalar.db.cluster.client.auth.type` - **Description:** The authentication type. `userpass` uses username and password authentication. `oidc_jwt` uses OIDC JWT-based access control. For details about `oidc_jwt`, see [Control User Access via OIDC-Based JWT Access Tokens](./control-access-via-oidc-based-jwt-tokens.mdx). - **Default value:** empty (treated as `userpass`) #### `auth.oidc_jwt.access_token` - **Field:** `scalar.db.cluster.client.auth.oidc_jwt.access_token` - **Description:** The JWT access token for OIDC access control. Since the token is set at initialization time, it cannot be refreshed and is only valid for the duration of the JWT access token's expiration. - **Default value:** empty ### Configurations for wire encryption The following wire encryption configurations are available for the Java Client SDK. For details about encrypting wire communications, see [Encrypt Wire Communications](./encrypt-wire-communications.mdx). #### `cluster.tls.enabled` - **Field:** `scalar.db.cluster.tls.enabled` - **Description:** Whether wire encryption (TLS) is enabled for the Java Client SDK. - **Default value:** `false` #### `cluster.tls.ca_root_cert_pem` - **Field:** `scalar.db.cluster.tls.ca_root_cert_pem` - **Description:** The custom CA root certificate (PEM data) for TLS communication. - **Default value:** empty #### `cluster.tls.ca_root_cert_path` - **Field:** `scalar.db.cluster.tls.ca_root_cert_path` - **Description:** The custom CA root certificate (file path) for TLS communication. - **Default value:** empty #### `cluster.tls.override_authority` - **Field:** `scalar.db.cluster.tls.override_authority` - **Description:** The custom authority for TLS communication. This doesn't change what host is actually connected. This is intended for testing, but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set for `scalar.db.cluster.node.tls.cert_chain_path`, which is the path for the cluster's TLS certificate. - **Default value:** empty ### Configurations for the SQL interface The following shows the configurations for the Java Client SDK when using the SQL interface. #### `sql.connection_mode` - **Field:** `scalar.db.sql.connection_mode` - **Description:** `cluster` should be specified. - **Default value:** empty #### `sql.cluster_mode.contact_points` - **Field:** `scalar.db.sql.cluster_mode.contact_points` - **Description:** Contact point of the cluster. If you use the `indirect` client mode, specify the IP address of the load balancer in front of your cluster nodes by using the format `indirect:`. If you use the `direct-kubernetes` client mode, specify the namespace name (optional) and the name of the [endpoint resource](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints) to get the membership information by using the format `direct-kubernetes:/` or just `direct-kubernetes:`. If you don't specify the namespace name, the Java Client SDK will use the `default` namespace. - **Default value:** empty :::note For example, if you use the `indirect` client mode and the load balancer IP address is `192.168.10.1`, you can configure the Java Client SDK as follows: ```properties scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points=indirect:192.168.10.1 ``` Or if you use the `direct-kubernetes` client mode, with the namespace of the endpoint as `ns` and the endpoint name as `scalardb-cluster`, you can configure the Java Client SDK as follows: ```properties scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points=direct-kubernetes:ns/scalardb-cluster ``` For details about how to configure ScalarDB JDBC, see [JDBC connection URL](../scalardb-sql/jdbc-guide.mdx#jdbc-connection-url). For details about how to configure Spring Data JDBC for ScalarDB, see [Configurations](../scalardb-sql/spring-data-guide.mdx#configurations). ::: #### `sql.cluster_mode.contact_port` - **Field:** `scalar.db.sql.cluster_mode.contact_port` - **Description:** Port number for the contact point. - **Default value:** `60053` #### `sql.default_transaction_mode` - **Field:** `scalar.db.sql.default_transaction_mode` - **Description:** Default transaction mode. `TRANSACTION` or `TWO_PHASE_COMMIT_TRANSACTION` can be set. - **Default value:** `TRANSACTION` #### `sql.default_namespace_name` - **Field:** `scalar.db.sql.default_namespace_name` - **Description:** Default namespace name. If you don't specify a namespace name in your SQL statement, this value is used. - **Default value:** empty #### `cluster.grpc.deadline_duration_millis` - **Field:** `scalar.db.cluster.grpc.deadline_duration_millis` - **Description:** Deadline duration for gRPC in millis. - **Default value:** `60000` (60 seconds) #### `cluster.grpc.max_inbound_message_size` - **Field:** `scalar.db.cluster.grpc.max_inbound_message_size` - **Description:** Maximum message size allowed for a single gRPC frame. - **Default value:** The gRPC default value #### `cluster.grpc.max_inbound_metadata_size` - **Field:** `scalar.db.cluster.grpc.max_inbound_metadata_size` - **Description:** Maximum size of metadata allowed to be received. - **Default value:** The gRPC default value #### `sql.cluster_mode.client.piggyback_begin.enabled` - **Field:** `scalar.db.sql.cluster_mode.client.piggyback_begin.enabled` - **Description:** Whether the piggyback-begin feature is enabled for SQL transactions. It delays transaction begin until the first execute operation, piggybacking the begin with the first operation to eliminate a dedicated begin RPC call. - **Default value:** `false` #### `sql.cluster_mode.client.write_buffering.enabled` - **Field:** `scalar.db.sql.cluster_mode.client.write_buffering.enabled` - **Description:** Whether the write-buffering feature is enabled for SQL transactions. It buffers `INSERT` and `UPSERT` statements on the client side and executes them in batches to reduce the number of RPC calls. - **Default value:** `false` :::warning * If `piggyback_begin` is enabled, `SqlSession.getTransactionId()` returns `Optional.empty()` until the actual begin operation is executed. * If `piggyback_begin` or `write_buffering` is enabled, you will always get an `IllegalStateException` when calling the `SqlSession.resume()` and `SqlSession.join()` methods. ::: ### Authentication and authorization configurations The following shows the authentication and authorization configurations for the Java Client SDK when using the SQL interface. #### `auth.enabled` - **Field:** `scalar.db.cluster.auth.enabled` - **Description:** Whether authentication and authorization are enabled. - **Default value:** `false` #### `auth.username` - **Field:** `scalar.db.sql.cluster_mode.username` - **Description:** Username to access ScalarDB Cluster. Not required when using OIDC JWT authentication. - **Default value:** empty #### `auth.password` - **Field:** `scalar.db.sql.cluster_mode.password` - **Description:** Password to access ScalarDB Cluster. Not required when using OIDC JWT authentication. - **Default value:** empty #### `auth.type` - **Field:** `scalar.db.sql.cluster_mode.auth.type` - **Description:** The authentication type. `userpass` uses username and password authentication. `oidc_jwt` uses OIDC JWT-based access control. For details about `oidc_jwt`, see [Control User Access via OIDC-Based JWT Access Tokens](./control-access-via-oidc-based-jwt-tokens.mdx). - **Default value:** empty (treated as `userpass`) #### `auth.oidc_jwt.access_token` - **Field:** `scalar.db.sql.cluster_mode.auth.oidc_jwt.access_token` - **Description:** The JWT access token for OIDC access control. Since the token is set at initialization time, it cannot be refreshed and is only valid for the duration of the JWT access token's expiration. - **Default value:** empty ## Configuration example - App, ScalarDB Cluster, and database ```mermaid flowchart LR app["App -
ScalarDB library with gRPC"] cluster["ScalarDB Cluster -
(ScalarDB library with
Consensus Commit)"] db[(Underlying storage or database)] app --> cluster --> db ``` In this example configuration, the app (ScalarDB library with gRPC) connects to an underlying storage or database (in this case, Cassandra) through ScalarDB Cluster, which is a component that is available only in the ScalarDB Enterprise edition. :::note This configuration is acceptable for production use because ScalarDB Cluster implements the [Scalar Admin](https://github.com/scalar-labs/scalar-admin) interface, which enables you to take transactionally consistent backups for ScalarDB by pausing ScalarDB Cluster. ::: The following is an example of the configuration for connecting the app to the underlying database through ScalarDB Cluster: ```properties # Transaction manager implementation. scalar.db.transaction_manager=cluster # Contact point of the cluster. scalar.db.contact_points=indirect: ``` For details about the Java Client SDK configurations, see [ScalarDB Cluster Java Client SDK configurations](#java-client-sdk-configurations). [^1]: It's worth benchmarking the performance with a few variations (for example, 75% and 125% of the default value) on the same underlying storage that your application uses, considering your application's access pattern, to determine the optimal configuration as it really depends on those factors. Also, it's important to benchmark combinations of these parameters (for example, first, `slot_capacity:20` and `group_size_fix_timeout_millis:40`; second, `slot_capacity:30` and `group_size_fix_timeout_millis:40`; and third, `slot_capacity:20` and `group_size_fix_timeout_millis:80`) to determine the optimal combination. ================================================ FILE: docs/scalardb-cluster/scalardb-cluster-grpc-api-guide.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster gRPC API Guide import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; This document describes the ScalarDB Cluster gRPC API. ScalarDB Cluster provides a Java API that uses the gRPC API internally. If you use Java or a JVM language, you can use the Java API instead of the ScalarDB Cluster gRPC API directly. For details about the Java API, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). For details about the services and messages for the ScalarDB Cluster gRPC API, see the definitions in the `scalardb-cluster.proto` file. For ScalarDB Cluster users who have a commercial license, please [contact support](https://www.scalar-labs.com/support) if you need the `scalardb-cluster.proto` file. ScalarDB Cluster gRPC API is composed of the following services: - `scalardb.cluster.rpc.v1.DistributedTransaction`: Provides a distributed transaction capability for ScalarDB Cluster. - `scalardb.cluster.rpc.v1.TwoPhaseCommitTransaction`: Provides a two-phase commit transaction capability for ScalarDB Cluster. - `scalardb.cluster.rpc.v1.DistributedTransactionAdmin`: Provides comprehensive administrative operations. The following sections describe how to use each service. ## Overview of error handling in ScalarDB Cluster gRPC API Before describing how to use each service, this section explains how error handling works in ScalarDB Cluster gRPC API. ScalarDB Cluster gRPC API employs [Richer error model](https://grpc.io/docs/guides/error/#richer-error-model) for error handling. This model enables servers to return and enables clients to consume additional error details expressed as one or more protobuf messages. ScalarDB Cluster gRPC API uses `google.rpc.ErrorInfo`, which is one of the [standard set of error message types](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto), and puts additional error details in `ErrorInfo` fields. `ErrorInfo` has the following fields: - `reason`: A string that provides a short description of the error. The following sections describe the possible values of `reason` in each service. - `domain`: A string that indicates the error's origin. In ScalarDB Cluster gRPC API, this string is always set to `com.scalar.db.cluster`. - `metadata`: A map of metadata for the specific error. In ScalarDB Cluster gRPC API, a transaction ID with the `transactionId` key in the map is put if the error is related to a transaction. If you encounter an error, you can retrieve `ErrorInfo` from `google.rpc.Status` in the gRPC response, but the method for doing so depends on the programming language. Please refer to the appropriate documentation to understand how to get `ErrorInfo` in your specific programming language. ## How to use the `DistributedTransaction` service The `DistributedTransaction` service provides the following RPCs: - `Begin`: Begins a transaction. - `Get`: Retrieves a record. - `Scan`: Scans records. - `Put`: Puts a record. - `Delete`: Deletes a record. - `Mutate` Mutates (puts and deletes) multiple records. - `Commit`: Commits a transaction. - `Rollback`: Rolls back a transaction. First, you call `Begin` to initiate a transaction. Then, you can call `Get` and `Scan` to read records, `Put` and `Mutate` to write records, and `Delete` and `Mutate` to delete records. To finalize the transaction, call `Commit`. Alternatively, you can call `Rollback` at any time before the transaction is committed to cancel it. By calling `Begin`, you receive a transaction ID in the response, which you can then use to call `Get`, `Scan`, `Put`, `Delete`, `Mutate`, `Commit`, and `Rollback`. When you call `Begin`, you can optionally specify a transaction ID. If you specify a transaction ID, the user is responsible for guaranteeing the uniqueness of the ID. If you do not specify a transaction ID, ScalarDB Cluster will generate a transaction ID for the transaction. You need to set `RequestHeader` for each RPC request. `RequestHeader` contains a `hop_limit` field, which restricts the number of hops for a request. The purpose of the `hop_limit` is to prevent infinite loops within the cluster. Each time a request is forwarded to another cluster node, the `hop_limit` decreases by one. If the `hop_limit` reaches zero, the request will be rejected. ### Error handling The table below shows the status code and the possible values of `reason` in `ErrorInfo` in each RPC in the `DistributedTransaction` service: | RPC | Status code | `reason` in `ErrorInfo` | Description | |--------------------------------|---------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Begin | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Begin | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Begin | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Begin | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Get, Scan, Put, Delete, Mutate | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Get, Scan, Put, Delete, Mutate | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Get, Scan, Put, Delete, Mutate | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Get, Scan, Put, Delete, Mutate | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Get, Scan, Put, Delete, Mutate | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Get, Scan, Put, Delete, Mutate | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Put, Delete, Mutate | FAILED_PRECONDITION | UNSATISFIED_CONDITION | The mutation condition is not satisfied. | | Commit | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Commit | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Commit | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Commit | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Commit | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Commit | INTERNAL | UNKNOWN_TRANSACTION_STATUS | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. The responsibility for determining the transaction status rests with the users. It may be beneficial to create a transaction status table and update it in conjunction with other application data so that you can determine the status of a transaction from the table itself. | | Commit | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Rollback | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Rollback | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Rollback | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | | Rollback | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | | Rollback | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | If you encounter an error, you should roll back the transaction, except in the case of `Begin`. Then, you can retry the transaction from the beginning for the errors that can be resolved by retrying. Besides the errors listed above, you may encounter errors returned by the gRPC library. In these cases, the response will not contain `ErrorInfo`. For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). You can set a deadline for each RPC in gRPC. If the deadline is exceeded, you will receive a `DEADLINE_EXCEEDED` error. In general, you should roll back the transaction in this situation, unless the RPC is `Begin` or `Commit`. In the case of `Commit`, the situation is equivalent to `UNKNOWN_TRANSACTION_STATUS` (it is uncertain whether the transaction was successfully committed or not), and you must handle the error in the same way. ## How to use the `TwoPhaseCommitTransaction` service The `TwoPhaseCommitTransaction` service provides the following RPCs: - `Begin`: Begins a transaction. - `Join`: Joins a transaction. - `Get`: Retrieves a record. - `Scan`: Scans records. - `Put`: Puts a record. - `Delete`: Deletes a record. - `Mutate`: Mutates (puts and deletes) multiple records. - `Prepare`: Prepares a transaction. - `Validate`: Validates a transaction. - `Commit`: Commits a transaction. - `Rollback`: Rolls back a transaction. First, you call `Begin` to initiate a transaction if you are the coordinator process. Alternatively, if you are a participant process, you can call `Join` to take part in a transaction that the coordinator has already begun. Then, you can call `Get` and `Scan` to read records, `Put` and `Mutate` to write records, and `Delete` and `Mutate` to delete records. To finalize the transaction, call `Prepare`, `Validate`, and then `Commit` in order. Alternatively, you can call `Rollback` at any time before the transaction is committed to cancel it. By calling `Begin` or `Join`, you receive a transaction ID in the response, which you can then use to call `Get`, `Scan`, `Put`, `Delete`, `Mutate`, `Prepare`, `Validate`, `Commit`, and `Rollback`. When you call `Begin`, you can optionally specify a transaction ID. If you specify a transaction ID, the user is responsible for guaranteeing the uniqueness of the ID. If you do not specify a transaction ID, ScalarDB Cluster will generate a transaction ID for the transaction. You need to set `RequestHeader` for each RPC request. `RequestHeader` contains a `hop_limit` field, which restricts the number of hops for a request. The purpose of the `hop_limit` is to prevent infinite loops within the cluster. Each time a request is forwarded to another cluster node, the `hop_limit` decreases by one. If the `hop_limit` reaches zero, the request will be rejected. ### Error handling The table below shows the status code and the possible values of `reason` in `ErrorInfo` in each RPC in the `TwoPhaseCommitTransaction` service: | RPC | Status code | `reason` in `ErrorInfo` | Description | |--------------------------------|---------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Begin, Join | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Begin, Join | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Begin, Join | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Begin, Join | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Get, Scan, Put, Delete, Mutate | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Get, Scan, Put, Delete, Mutate | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Get, Scan, Put, Delete, Mutate | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Get, Scan, Put, Delete, Mutate | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Get, Scan, Put, Delete, Mutate | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Get, Scan, Put, Delete, Mutate | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Put, Delete, Mutate | FAILED_PRECONDITION | UNSATISFIED_CONDITION | The mutation condition is not satisfied. | | Prepare, Validate | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Prepare, Validate | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Prepare, Validate | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Prepare, Validate | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Prepare, Validate | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Prepare, Validate | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Commit | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Commit | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Commit | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Commit | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Commit | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Commit | INTERNAL | UNKNOWN_TRANSACTION_STATUS | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. The responsibility for determining the transaction status rests with the users. It may be beneficial to create a transaction status table and update it in conjunction with other application data so that you can determine the status of a transaction from the table itself. | | Commit | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Rollback | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Rollback | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Rollback | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | | Rollback | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | | Rollback | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | If you encounter an error, you should roll back the transaction, except in the case of `Begin` or `Join`. Then, you can retry the transaction from the beginning for the errors that can be resolved by retrying. Besides the errors listed above, you may encounter errors returned by the gRPC library. In these cases, the response will not contain `ErrorInfo`. For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). You can set a deadline for each RPC in gRPC. If the deadline is exceeded, you will receive a `DEADLINE_EXCEEDED` error. In general, you should roll back the transaction in this situation, unless the RPC is `Begin`, `Join`, or `Commit`. In the case of `Commit`, the situation is equivalent to `UNKNOWN_TRANSACTION_STATUS` (it is uncertain whether the transaction was successfully committed or not), and you must handle the error in the same way. ## How to use the `DistributedTransactionAdmin` service The `DistributedTransactionAdmin` service provides the following RPCs: - `CreateNamespace`: Creates a namespace. - `DropNamespace`: Drops a namespace. - `NamespaceExists`: Returns whether the specified namespace exists or not. - `CreateTable`: Creates a table. - `DropTable`: Drops a table. - `TruncateTable`: Truncates a table. - `TableExists`: Returns whether the specified table exists or not. - `CreateIndex`: Creates an index. - `DropIndex`: Drops an index. - `IndexExists`: Returns whether the specified index exists or not. - `RepairTable`: Repairs a namespace that may be in an unknown state. - `AddNewColumnToTable`: Adds a new column to a table. - `CreateCoordinatorTables`: Creates the Coordinator tables. - `DropCoordinatorTables`: Drops the Coordinator tables. - `TruncateCoordinatorTables`: Truncates the Coordinator tables. - `CoordinatorTablesExist`: Returns whether the Coordinator tables exist or not. - `RepairCoordinatorTables`: Repairs the Coordinator tables. - `GetTableMetadata`: Returns table metadata of the specified table. - `GetNamespaceTableNames`: Returns tables in the specified namespace. - `ImportTable`: Imports an existing table that is not managed by ScalarDB. ### Error handling The table below shows the status code and the possible values of `reason` in `ErrorInfo` for all RPCs in the `DistributedTransactionAdmin` service: | Status code | `reason` in `ErrorInfo` | Description | |---------------------|----------------------------|-------------------------------------------------| | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | INTERNAL | INTERNAL_ERROR | The operation has failed. | Besides the errors listed above, you may encounter errors returned by the gRPC library. In these cases, the response will not contain `ErrorInfo`. For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). ================================================ FILE: docs/scalardb-cluster/scalardb-cluster-sql-grpc-api-guide.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster SQL gRPC API Guide import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; This document describes the ScalarDB Cluster SQL gRPC API. ScalarDB Cluster SQL provides a Java API that uses the gRPC API internally. If you use Java or a JVM language, you can use the Java API instead of the ScalarDB Cluster SQL gRPC API directly. For details about the Java API, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). For details about the services and messages for the ScalarDB Cluster SQL gRPC API, see the definitions in the `scalardb-cluster-sql.proto` file. For ScalarDB Cluster users who have a commercial license, please [contact support](https://www.scalar-labs.com/support) if you need the `scalardb-cluster-sql.proto` file. ScalarDB Cluster SQL gRPC API is composed of the following services: - `scalardb.cluster.rpc.v1.sql.SqlTransaction`: Provides a transaction capability for ScalarDB Cluster SQL. - `scalardb.cluster.rpc.v1.sql.SqlTwoPhaseCommitTransaction`: Provides a two-phase commit transaction capability for ScalarDB Cluster SQL. - `scalardb.cluster.rpc.v1.sql.Metadata`: Provides a metadata view of ScalarDB Cluster SQL. The following sections describe how to use each service. ## Overview of error handling in ScalarDB Cluster SQL gRPC API Before describing how to use each service, this section explains how error handling works in ScalarDB Cluster SQL gRPC API. ScalarDB Cluster SQL gRPC API employs [Richer error model](https://grpc.io/docs/guides/error/#richer-error-model) for error handling. This model enables servers to return and enables clients to consume additional error details expressed as one or more protobuf messages. ScalarDB Cluster SQL gRPC API uses `google.rpc.ErrorInfo`, which is one of the [standard set of error message types](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto), and puts additional error details in `ErrorInfo` fields. `ErrorInfo` has the following fields: - `reason`: A string that provides a short description of the error. The following sections describe the possible values of `reason` in each service. - `domain`: A string that indicates the error's origin. In ScalarDB Cluster SQL gRPC API, this string is always set to `com.scalar.db.cluster.sql`. - `metadata`: A map of metadata for the specific error. In ScalarDB Cluster SQL gRPC API, a transaction ID with the `transactionId` key in the map is put if the error is related to a transaction. If you encounter an error, you can retrieve `ErrorInfo` from `google.rpc.Status` in the gRPC response, but the method for doing so depends on the programming language. Please refer to the appropriate documentation to understand how to get `ErrorInfo` in your specific programming language. ## How to use the `SqlTransaction` service The `SqlTransaction` service provides the following RPCs: - `Begin`: Begins a transaction. - `Execute` Executes a SQL statement. - `Commit`: Commits a transaction. - `Rollback`: Rolls back a transaction. First, you call `Begin` to initiate a transaction. Following that, you can call `Execute` to read, write, and delete records. To finalize the transaction, call `Commit`. Alternatively, you can call `Rollback` at any time before the transaction is committed to cancel it. By calling `Begin`, you receive a transaction ID in the response, which you can then use to call `Execute`, `Commit`, and `Rollback`. Also, you can call `Execute` without a transaction ID to execute a one-shot transaction. In this case, the transaction is automatically committed after it is executed. You can use this method to execute DDL statements as well. For details on the supported SQL statements, refer to [ScalarDB SQL Grammar](../scalardb-sql/grammar.mdx). Please note, however, that `Execute` supports only DML and DDL statements. When you call `Begin`, you can optionally specify a transaction ID. If you specify a transaction ID, the user is responsible for guaranteeing the uniqueness of the ID. If you do not specify a transaction ID, ScalarDB Cluster will generate a transaction ID for the transaction. You need to set `RequestHeader` for each RPC request. `RequestHeader` contains a `hop_limit` field, which restricts the number of hops for a request. The purpose of the `hop_limit` is to prevent infinite loops within the cluster. Each time a request is forwarded to another cluster node, the `hop_limit` decreases by one. If the `hop_limit` reaches zero, the request will be rejected. ### Error handling The table below shows the status code and the possible values of `reason` in `ErrorInfo` in each RPC in the `SqlTransaction` service: | RPC | Status code | `reason` in `ErrorInfo` | Description | |----------|---------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Begin | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Begin | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Begin | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Begin | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Execute | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Execute | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Execute | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Execute | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Execute | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Execute | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Commit | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Commit | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Commit | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Commit | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Commit | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Commit | INTERNAL | UNKNOWN_TRANSACTION_STATUS | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. The responsibility for determining the transaction status rests with the users. It may be beneficial to create a transaction status table and update it in conjunction with other application data so that you can determine the status of a transaction from the table itself. | | Commit | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Rollback | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Rollback | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Rollback | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | | Rollback | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | | Rollback | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | If you encounter an error, you should roll back the transaction, except in the case of `Begin`. Then, you can retry the transaction from the beginning for the errors that can be resolved by retrying. Besides the errors listed above, you may encounter errors returned by the gRPC library. In these cases, the response will not contain `ErrorInfo`. For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). You can set a deadline for each RPC in gRPC. If the deadline is exceeded, you will receive a `DEADLINE_EXCEEDED` error. In general, you should roll back the transaction in this situation, unless the RPC is `Begin` or `Commit`. In the case of `Commit`, the situation is equivalent to `UNKNOWN_TRANSACTION_STATUS` (it is uncertain whether the transaction was successfully committed or not), and you must handle the error in the same way. ## How to use the `SqlTwoPhaseCommitTransaction` service The `SqlTwoPhaseCommitTransaction` service provides the following RPCs: - `Begin`: Begins a transaction. - `Join`: Joins a transaction. - `Execute` Executes a SQL statement. - `Prepare`: Prepares a transaction. - `Validate`: Validates a transaction. - `Commit`: Commits a transaction. - `Rollback`: Rolls back a transaction. First, you call `Begin` to initiate a transaction if you are the coordinator process. Alternatively, if you are a participant process, you can call `Join` to take part in a transaction that the coordinator has already begun. Following that, you can call `Execute` to read, write, and delete records. To finalize the transaction, call `Prepare`, `Validate`, and then `Commit` in order. Alternatively, you can call `Rollback` at any time before the transaction is committed to cancel it. By calling `Begin` or `Join`, you receive a transaction ID in the response, which you can then use to call `Execute`, `Prepare`, `Validate`, `Commit`, and `Rollback`. In addition, you can call `Execute` without a transaction ID to execute a one-shot transaction. In this case, the transaction is automatically committed after it is executed. You can use this method to execute DDL statements as well. For details on the supported SQL statements, refer to [ScalarDB SQL Grammar](../scalardb-sql/grammar.mdx). Please note, however, that `Execute` supports only DML and DDL statements. When you call `Begin`, you can optionally specify a transaction ID. If you specify a transaction ID, the user is responsible for guaranteeing the uniqueness of the ID. If you do not specify a transaction ID, ScalarDB Cluster will generate a transaction ID for the transaction. You need to set `RequestHeader` for each RPC request. `RequestHeader` contains a `hop_limit` field, which restricts the number of hops for a request. The purpose of the `hop_limit` is to prevent infinite loops within the cluster. Each time a request is forwarded to another cluster node, the `hop_limit` decreases by one. If the `hop_limit` reaches zero, the request will be rejected. ### Error handling The table below shows the status code and the possible values of `reason` in `ErrorInfo` in each RPC in the `SqlTwoPhaseCommitTransaction` service: | RPC | Status code | `reason` in `ErrorInfo` | Description | |-------------------|---------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Begin, Join | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Begin, Join | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Begin, Join | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Begin, Join | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Execute | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Execute | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Execute | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Execute | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Execute | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Execute | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Prepare, Validate | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Prepare, Validate | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Prepare, Validate | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Prepare, Validate | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Prepare, Validate | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Prepare, Validate | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Commit | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Commit | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Commit | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | Commit | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | Commit | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | Commit | INTERNAL | UNKNOWN_TRANSACTION_STATUS | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. The responsibility for determining the transaction status rests with the users. It may be beneficial to create a transaction status table and update it in conjunction with other application data so that you can determine the status of a transaction from the table itself. | | Commit | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | Rollback | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | Rollback | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | Rollback | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | | Rollback | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | | Rollback | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | If you encounter an error, you should roll back the transaction, except in the case of `Begin` or `Join`. Then, you can retry the transaction from the beginning for the errors that can be resolved by retrying. Besides the errors listed above, you may encounter errors returned by the gRPC library. In these cases, the response will not contain `ErrorInfo`. For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). You can set a deadline for each RPC in gRPC. If the deadline is exceeded, you will receive a `DEADLINE_EXCEEDED` error. In general, you should roll back the transaction in this situation, unless the RPC is `Begin`, `Join`, or `Commit`. In the case of `Commit`, the situation is equivalent to `UNKNOWN_TRANSACTION_STATUS` (it is uncertain whether the transaction was successfully committed or not), and you must handle the error in the same way. ## How to use the `Metadata` service The `Metadata` service provides the following RPCs: - `GetNamespaceMetadata`: Retrieves namespace metadata of the specified namespace. - `ListTableMetadataInNamespace`: Retrieves table metadata of tables in the specified namespace. - `GetTableMetadata`: Retrieves table metadata of the specified table. ### Error handling The table below shows the status code and the possible values of `reason` in `ErrorInfo` for all RPCs in the `Metadata` service: | Status code | `reason` in `ErrorInfo` | Description | |---------------------|----------------------------|-------------------------------------------------| | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | | INTERNAL | INTERNAL_ERROR | The operation has failed. | Besides the errors listed above, you may encounter errors returned by the gRPC library. In these cases, the response will not contain `ErrorInfo`. For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). ================================================ FILE: docs/scalardb-cluster/scalardb-cluster-status-codes.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster Error Codes This page provides a list of error codes in ScalarDB Cluster. ## Error code classes and descriptions | Class | Description | |:-------------------|:------------------------------------------| | `DB-CLUSTER-1xxxx` | Errors for the user error category | | `DB-CLUSTER-2xxxx` | Errors for the concurrency error category | | `DB-CLUSTER-3xxxx` | Errors for the internal error category | ## `DB-CLUSTER-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-CLUSTER-10000` **Message** ```markdown The namespace does not exist. Namespace: %s ``` ### `DB-CLUSTER-10001` **Message** ```markdown The table does not exist. Table: %s ``` ### `DB-CLUSTER-10002` **Message** ```markdown The user does not exist. User: %s ``` ### `DB-CLUSTER-10004` **Message** ```markdown The get type is unspecified ``` ### `DB-CLUSTER-10005` **Message** ```markdown The get type is unrecognized ``` ### `DB-CLUSTER-10006` **Message** ```markdown The value of the column is not set. Column: %s ``` ### `DB-CLUSTER-10007` **Message** ```markdown The scan type is unspecified ``` ### `DB-CLUSTER-10008` **Message** ```markdown The scan type is unrecognized ``` ### `DB-CLUSTER-10009` **Message** ```markdown The order is unspecified ``` ### `DB-CLUSTER-10010` **Message** ```markdown The order is unrecognized ``` ### `DB-CLUSTER-10011` **Message** ```markdown The clustering order is unspecified ``` ### `DB-CLUSTER-10012` **Message** ```markdown The clustering order is unrecognized ``` ### `DB-CLUSTER-10013` **Message** ```markdown The put condition type is unspecified ``` ### `DB-CLUSTER-10014` **Message** ```markdown The put condition type is unrecognized ``` ### `DB-CLUSTER-10015` **Message** ```markdown The delete condition type is unspecified ``` ### `DB-CLUSTER-10016` **Message** ```markdown The delete condition type is unrecognized ``` ### `DB-CLUSTER-10017` **Message** ```markdown The operator is unspecified ``` ### `DB-CLUSTER-10018` **Message** ```markdown The operator is unrecognized ``` ### `DB-CLUSTER-10019` **Message** ```markdown The mutation is not set ``` ### `DB-CLUSTER-10020` **Message** ```markdown The data type is unspecified ``` ### `DB-CLUSTER-10021` **Message** ```markdown The data type is unrecognized ``` ### `DB-CLUSTER-10022` **Message** ```markdown The user option is unspecified ``` ### `DB-CLUSTER-10023` **Message** ```markdown The user option is unrecognized ``` ### `DB-CLUSTER-10024` **Message** ```markdown The privilege is unspecified ``` ### `DB-CLUSTER-10025` **Message** ```markdown The privilege is unrecognized ``` ### `DB-CLUSTER-10026` **Message** ```markdown The username is not set ``` ### `DB-CLUSTER-10027` **Message** ```markdown This feature is not supported in ScalarDB Cluster ``` ### `DB-CLUSTER-10028` **Message** ```markdown The property 'scalar.db.contact_points' must not be empty ``` ### `DB-CLUSTER-10029` **Message** ```markdown The property 'scalar.db.contact_points' must be prefixed with 'indirect:' or 'direct-kubernetes:' ``` ### `DB-CLUSTER-10030` **Message** ```markdown The format of the property 'scalar.db.contact_points' for direct-kubernetes client mode is 'direct-kubernetes:/' or 'direct-kubernetes:' ``` ### `DB-CLUSTER-10031` **Message** ```markdown The property 'scalar.db.sql.cluster_mode.contact_points' must not be empty ``` ### `DB-CLUSTER-10032` **Message** ```markdown The property 'scalar.db.sql.cluster_mode.contact_points' must be prefixed with 'indirect:' or 'direct-kubernetes:' ``` ### `DB-CLUSTER-10033` **Message** ```markdown The format of the property 'scalar.db.sql.cluster_mode.contact_points' for direct-kubernetes client mode is 'direct-kubernetes:/' or 'direct-kubernetes:' ``` ### `DB-CLUSTER-10035` **Message** ```markdown The update condition type is unspecified ``` ### `DB-CLUSTER-10036` **Message** ```markdown The update condition type is unrecognized ``` ### `DB-CLUSTER-10037` **Message** ```markdown The two-phase commit interface is not supported ``` ### `DB-CLUSTER-10039` **Message** ```markdown The policy state is unspecified ``` ### `DB-CLUSTER-10040` **Message** ```markdown The policy state is unrecognized ``` ### `DB-CLUSTER-10041` **Message** ```markdown The access mode is unspecified ``` ### `DB-CLUSTER-10042` **Message** ```markdown The access mode is unrecognized ``` ### `DB-CLUSTER-10043` **Message** ```markdown The service does not exist. Service Class: %s ``` ### `DB-CLUSTER-10044` **Message** ```markdown The policy does not exist. Policy: %s ``` ### `DB-CLUSTER-10057` **Message** ```markdown The operation is not set ``` ### `DB-CLUSTER-10058` **Message** ```markdown The batch result is not set ``` ### `DB-CLUSTER-10059` **Message** ```markdown Resuming a transaction is not allowed when piggyback-begin is enabled ``` ### `DB-CLUSTER-10060` **Message** ```markdown Resuming a transaction is not allowed when write-buffering is enabled ``` ### `DB-CLUSTER-10061` **Message** ```markdown The transaction has not begun yet. This situation may occur when piggyback-begin is enabled ``` ## `DB-CLUSTER-2xxxx` status codes The following are status codes and messages for the concurrency error category. ### `DB-CLUSTER-20000` **Message** ```markdown The hop limit is exceeded ``` ### `DB-CLUSTER-20001` **Message** ```markdown A transaction associated with the specified transaction ID is not found. The transaction might have expired, or the cluster node that handled the transaction might have been restarted. Transaction ID: %s ``` ### `DB-CLUSTER-20002` **Message** ```markdown A scanner associated with the specified scanner ID is not found. The scanner might have expired, or the cluster node that handled the scanner might have been restarted. Transaction ID: %s; Scanner ID: %s ``` ## `DB-CLUSTER-3xxxx` status codes The following are status codes and messages for the internal error category. ### `DB-CLUSTER-30000` **Message** ```markdown Getting local IP addresses failed ``` ### `DB-CLUSTER-30001` **Message** ```markdown Getting a cluster node object from the cache failed. Cluster Node IP Address: %s ``` ### `DB-CLUSTER-30002` **Message** ```markdown The ring is empty ``` ### `DB-CLUSTER-30003` **Message** ```markdown Getting the Kubernetes API client failed ``` ### `DB-CLUSTER-30004` **Message** ```markdown Reading the Kubernetes endpoint failed. Namespace: %s; Name: %s; Code: %d; Response Headers: %s; Response Body: %s ``` ### `DB-CLUSTER-30005` **Message** ```markdown Configuring TLS failed ``` ### `DB-CLUSTER-30006` **Message** ```markdown No nearest cluster nodes are found ``` ================================================ FILE: docs/scalardb-cluster/scalardb-embedding-store-status-codes.mdx ================================================ --- tags: - Enterprise Premium - Private Preview displayed_sidebar: docsEnglish --- # Embedding Store Error Codes This page provides a list of error codes related to embedding stores. ## Error code classes and descriptions | Class | Description | |:---------------------|:-----------------------------------| | `DB-EMBEDDING-1xxxx` | Errors for the user error category | ## `DB-EMBEDDING-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-EMBEDDING-10001` **Message** ```markdown The embedding store name "scalar.db.embedding.client.store" is not specified ``` ### `DB-EMBEDDING-10002` **Message** ```markdown The embedding model name "scalar.db.embedding.client.model" is not specified ``` ### `DB-EMBEDDING-10003` **Message** ```markdown The embedding store is not found. Store: %s ``` ### `DB-EMBEDDING-10004` **Message** ```markdown The embedding model is not found. Model: %s ``` ### `DB-EMBEDDING-10005` **Message** ```markdown The property 'scalar.db.embedding.client.contact_points' must not be empty ``` ### `DB-EMBEDDING-10006` **Message** ```markdown The property 'scalar.db.embedding.client.contact_points' must be prefixed with 'indirect:' or 'direct-kubernetes:' ``` ### `DB-EMBEDDING-10007` **Message** ```markdown The format of the property 'scalar.db.embedding.client.contact_points' for direct-kubernetes client mode is 'direct-kubernetes:/' or 'direct-kubernetes:' ``` ### `DB-EMBEDDING-10008` **Message** ```markdown The embeddings must be provided ``` ### `DB-EMBEDDING-10009` **Message** ```markdown Only one embedding can be added with an embedding ID ``` ### `DB-EMBEDDING-10010` **Message** ```markdown Text segments cannot be provided when adding an embedding with an embedding ID ``` ### `DB-EMBEDDING-10011` **Message** ```markdown Both embedding IDs and a filter cannot be provided ``` ### `DB-EMBEDDING-10012` **Message** ```markdown Unsupported embedding store type. Type: %s ``` ### `DB-EMBEDDING-10013` **Message** ```markdown Unsupported embedding model type. Type: %s ``` ### `DB-EMBEDDING-10014` **Message** ```markdown The filter is not set ``` ### `DB-EMBEDDING-10015` **Message** ```markdown Unsupported metadata value type. Type: %s ``` ### `DB-EMBEDDING-10016` **Message** ```markdown The metadata value is not set ``` ================================================ FILE: docs/scalardb-cluster/scalardb-encryption-status-codes.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Encryption Error Codes This page provides a list of error codes related to encryption. ## Error code classes and descriptions | Class | Description | |:----------------------|:---------------------------------------| | `DB-ENCRYPTION-1xxxx` | Errors for the user error category | | `DB-ENCRYPTION-3xxxx` | Errors for the internal error category | ## `DB-ENCRYPTION-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-ENCRYPTION-10000` **Message** ```markdown The partition key column cannot be encrypted. Column: %s ``` ### `DB-ENCRYPTION-10001` **Message** ```markdown The clustering key column cannot be encrypted. Column: %s ``` ### `DB-ENCRYPTION-10002` **Message** ```markdown The indexed column cannot be encrypted. Column: %s ``` ### `DB-ENCRYPTION-10003` **Message** ```markdown The encrypted column cannot be specified as an index column. Column: %s ``` ### `DB-ENCRYPTION-10004` **Message** ```markdown The operation does not have the target namespace or table name. Operation: %s ``` ### `DB-ENCRYPTION-10005` **Message** ```markdown The column value is not properly specified. Column: %s, Operation: %s ``` ### `DB-ENCRYPTION-10006` **Message** ```markdown The property for the encryption type ("scalar.db.cluster.encryption.type") is not set ``` ### `DB-ENCRYPTION-10007` **Message** ```markdown Unknown encryption type: %s ``` ### `DB-ENCRYPTION-10008` **Message** ```markdown The property for the address of the Vault server ("scalar.db.cluster.encryption.vault.address") is not set ``` ### `DB-ENCRYPTION-10009` **Message** ```markdown The property for the token for the Vault server ("scalar.db.cluster.encryption.vault.token") is not set ``` ### `DB-ENCRYPTION-10010` **Message** ```markdown The encrypted column cannot be specified in the condition. Column: %s, Operation: %s ``` ### `DB-ENCRYPTION-10011` **Message** ```markdown The encrypted column cannot be specified in the ordering. Column: %s, Operation: %s ``` ### `DB-ENCRYPTION-10012` **Message** ```markdown The key type specified by the property "scalar.db.cluster.encryption.vault.key_type" is not supported. The supported key types are "aes128-gcm96", "aes256-gcm96", and "chacha20-poly1305". Key type: %s ``` ### `DB-ENCRYPTION-10013` **Message** ```markdown The key type specified by the property "scalar.db.cluster.encryption.self.key_type" is not supported. The supported key types are "AES128_GCM", "AES256_GCM", "AES128_EAX", "AES256_EAX", "AES128_CTR_HMAC_SHA256", "AES256_CTR_HMAC_SHA256", "CHACHA20_POLY1305", and "XCHACHA20_POLY1305". Key type: %s ``` ### `DB-ENCRYPTION-10014` **Message** ```markdown The encrypted column cannot be renamed. Column: %s ``` ### `DB-ENCRYPTION-10015` **Message** ```markdown The table with encrypted columns cannot be renamed. Table: %s ``` ### `DB-ENCRYPTION-10016` **Message** ```markdown The encrypted column cannot be altered the type. Column: %s ``` ## `DB-ENCRYPTION-3xxxx` status codes The following are status codes and messages for the internal error category. ### `DB-ENCRYPTION-30000` **Message** ```markdown Retrieving encrypted columns failed. Table: %s ``` ### `DB-ENCRYPTION-30001` **Message** ```markdown Registering encrypted columns failed. Columns: %s, Table: %s ``` ### `DB-ENCRYPTION-30002` **Message** ```markdown Unregistering encrypted columns failed. Table: %s ``` ### `DB-ENCRYPTION-30003` **Message** ```markdown Creating a data encryption key failed. Details: %s ``` ### `DB-ENCRYPTION-30004` **Message** ```markdown Checking the existence of a data encryption key failed. Details: %s ``` ### `DB-ENCRYPTION-30005` **Message** ```markdown Updating the configuration of a data encryption key failed. Details: %s ``` ### `DB-ENCRYPTION-30006` **Message** ```markdown Deleting a data encryption key failed. Details: %s ``` ### `DB-ENCRYPTION-30007` **Message** ```markdown Encrypting data failed. Details: %s ``` ### `DB-ENCRYPTION-30008` **Message** ```markdown Decrypting data failed. Details: %s ``` ### `DB-ENCRYPTION-30009` **Message** ```markdown HTTP GET request failed. Details: %s ``` ### `DB-ENCRYPTION-30010` **Message** ```markdown HTTP POST request failed. Details: %s ``` ### `DB-ENCRYPTION-30011` **Message** ```markdown HTTP DELETE request failed. Details: %s ``` ### `DB-ENCRYPTION-30012` **Message** ```markdown Registering the AEAD configuration failed. Details: %s ``` ### `DB-ENCRYPTION-30013` **Message** ```markdown Getting the AEAD primitive failed. Details: %s ``` ### `DB-ENCRYPTION-30014` **Message** ```markdown Getting the Kubernetes API client failed ``` ### `DB-ENCRYPTION-30015` **Message** ```markdown Registering a data encryption key to the Kubernetes secret failed. Namespace: %s; Name: %s; Code: %d; Response Headers: %s; Response Body: %s ``` ### `DB-ENCRYPTION-30016` **Message** ```markdown Checking the existence of a data encryption key in the Kubernetes secret failed. Namespace: %s; Name: %s; Code: %d; Response Headers: %s; Response Body: %s ``` ### `DB-ENCRYPTION-30017` **Message** ```markdown Deleting a data encryption key in the Kubernetes secret failed. Namespace: %s; Name: %s; Code: %d; Response Headers: %s; Response Body: %s ``` ### `DB-ENCRYPTION-30018` **Message** ```markdown Parsing a data encryption key failed. Details: %s ``` ### `DB-ENCRYPTION-30019` **Message** ```markdown Reading a data encryption key in the Kubernetes secret failed. Namespace: %s; Name: %s; Code: %d; Response Headers: %s; Response Body: %s ``` ### `DB-ENCRYPTION-30020` **Message** ```markdown Unregistering encrypted columns failed. Table: %s; ColumnNames: %s ``` ================================================ FILE: docs/scalardb-cluster/scalardb-remote-replication-status-codes.mdx ================================================ --- tags: - Enterprise Premium - Private Preview displayed_sidebar: docsEnglish --- # Remote Replication Error Codes This page provides a list of error codes related to remote replication. ## Error code classes and descriptions | Class | Description | |:----------------|:-----------------------------------| | `DB-REPL-1xxxx` | Errors for the user error category | ## `DB-REPL-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-REPL-10057` **Message** ```markdown Replication tables already exist ``` ### `DB-REPL-10058` **Message** ```markdown Replication tables do not exist ``` ### `DB-REPL-10059` **Message** ```markdown The namespace %s is reserved for the replication feature. Any operations on this namespace are not allowed ``` ### `DB-REPL-10060` **Message** ```markdown One-phase commit is not supported in the remote replication feature ``` ================================================ FILE: docs/scalardb-cluster/setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # How to Deploy ScalarDB Cluster Locally import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; This guide provides instructions on how to deploy ScalarDB Cluster by using a Helm Chart on a local Kubernetes cluster, specifically designed for a test environment. ## Prerequisites - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later - Kubernetes cluster (either [minikube](https://minikube.sigs.k8s.io/docs/start/) or [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation)) - [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) - [Helm](https://helm.sh/docs/intro/install/) ## What you will create You will be deploying the following components on a local Kubernetes cluster as depicted below: ``` +---------------------------------------------------------------------------------------------------------------------------------------+ | [Kubernetes Cluster] | | | | [Pod] [Pod] [Pod] | | | | +-------+ | | +---> | Envoy | ---+ | | | +-------+ | | | | | | | +---------+ | +-------+ | +--------------------+ | | | Service | ---+---> | Envoy | ---+---------> | Service | ---+ | | | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | | | +---------+ | | +--------------------+ | +-----------------------+ | | | +-------+ | | +---> | ScalarDB Cluster Node | ---+ | | +---> | Envoy | ---+ | | +-----------------------+ | | | +-------+ | | | | | | | +-----------------------+ | +------------+ | | +---+---> | ScalarDB Cluster Node | ---+---> | PostgreSQL | | | | | +-----------------------+ | +------------+ | | | | | | | | | +-----------------------+ | | | | +---> | ScalarDB Cluster Node | ---+ | | | +-----------------------+ | | +----------------------------+ | | | | Service | ---+ | | | (ScalarDB Cluster GraphQL) | | | +----------------------------+ | | | +---------------------------------------------------------------------------------------------------------------------------------------+ ``` ## Step 1. Start a PostgreSQL container ScalarDB Cluster must use some kind of database system as its backend database. The database that is used in this guide is PostgreSQL. You can deploy PostgreSQL on the Kubernetes cluster as follows. 1. Add the Bitnami Helm repository by running the following command: ```console helm repo add bitnami https://charts.bitnami.com/bitnami ``` 2. Deploy PostgreSQL by running the following command: ```console helm install postgresql-scalardb-cluster bitnami/postgresql \ --set auth.postgresPassword=postgres \ --set primary.persistence.enabled=false ``` 3. Check if the PostgreSQL container is running by running the following command: ```console kubectl get pod ``` You should see the following output: ```console NAME READY STATUS RESTARTS AGE postgresql-scalardb-cluster-0 1/1 Running 0 17s ``` ## Step 2. Deploy ScalarDB Cluster on the Kubernetes cluster by using a Helm Chart 1. Add the Scalar Helm Charts repository by running the following command: ```console helm repo add scalar-labs https://scalar-labs.github.io/helm-charts ``` 2. Set your license key and certificate as environment variables. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). For details about the value for ``, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). ```console SCALAR_DB_CLUSTER_LICENSE_KEY='' SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM='' ``` 3. Create a custom values file for ScalarDB Cluster (`scalardb-cluster-custom-values.yaml`) by running the following command: ```console cat << 'EOF' > scalardb-cluster-custom-values.yaml envoy: enabled: true service: type: "LoadBalancer" scalardbCluster: image: repository: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium" scalardbClusterNodeProperties: | # ScalarDB Cluster configurations scalar.db.cluster.membership.type=KUBERNETES scalar.db.cluster.membership.kubernetes.endpoint.namespace_name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAMESPACE_NAME} scalar.db.cluster.membership.kubernetes.endpoint.name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAME} # Storage configurations scalar.db.storage=jdbc scalar.db.contact_points=jdbc:postgresql://postgresql-scalardb-cluster.default.svc.cluster.local:5432/postgres scalar.db.username=${env:SCALAR_DB_CLUSTER_POSTGRES_USERNAME} scalar.db.password=${env:SCALAR_DB_CLUSTER_POSTGRES_PASSWORD} # For ScalarDB Cluster GraphQL tutorial. scalar.db.graphql.enabled=true scalar.db.graphql.namespaces=emoney # For ScalarDB Cluster SQL tutorial. scalar.db.sql.enabled=true ### License key configurations scalar.db.cluster.node.licensing.license_key=${env:SCALAR_DB_CLUSTER_LICENSE_KEY} scalar.db.cluster.node.licensing.license_check_cert_pem=${env:SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM} graphql: enabled: true service: type: "LoadBalancer" secretName: "scalardb-credentials-secret" EOF ``` :::note For the purpose of this guide, the service type for ScalarDB Cluster GraphQL and Envoy is set to `LoadBalancer`. ::: 4. Create a secret resource named `scalardb-credentials-secret` that includes credentials and license keys. ```console kubectl create secret generic scalardb-credentials-secret \ --from-literal=SCALAR_DB_CLUSTER_POSTGRES_USERNAME=postgres \ --from-literal=SCALAR_DB_CLUSTER_POSTGRES_PASSWORD=postgres \ --from-literal=SCALAR_DB_CLUSTER_LICENSE_KEY="${SCALAR_DB_CLUSTER_LICENSE_KEY}" \ --from-file=SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM=<(echo ${SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM} | sed 's/\\n/\ /g') \ -n default ``` 5. Set the chart version of ScalarDB Cluster. ```console SCALAR_DB_CLUSTER_VERSION=3.18.0 SCALAR_DB_CLUSTER_CHART_VERSION=$(helm search repo scalar-labs/scalardb-cluster -l | grep -F "${SCALAR_DB_CLUSTER_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) ``` 6. Deploy ScalarDB Cluster. ```console helm install scalardb-cluster scalar-labs/scalardb-cluster -f scalardb-cluster-custom-values.yaml --version ${SCALAR_DB_CLUSTER_CHART_VERSION} -n default ``` 7. Check if the ScalarDB Cluster pods are deployed: ```console kubectl get pod ``` You should see the following output: ```console NAME READY STATUS RESTARTS AGE postgresql-scalardb-cluster-0 1/1 Running 0 84s scalardb-cluster-envoy-59899dc588-477tg 1/1 Running 0 35s scalardb-cluster-envoy-59899dc588-dpvhx 1/1 Running 0 35s scalardb-cluster-envoy-59899dc588-lv9hx 1/1 Running 0 35s scalardb-cluster-node-866c756c79-5v2tk 1/1 Running 0 35s scalardb-cluster-node-866c756c79-9zhq5 1/1 Running 0 35s scalardb-cluster-node-866c756c79-t6v86 1/1 Running 0 35s ``` If the ScalarDB Cluster Node Pods and the Envoy Pods are deployed properly, the `STATUS` for each pod will be `Running`. 6. Check if the service resources of ScalarDB Cluster are deployed by running the following command: ```console kubectl get svc ``` You should see the following output: ```console NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 443/TCP 260d postgresql-scalardb-cluster ClusterIP 10.110.97.40 5432/TCP 86s postgresql-scalardb-cluster-hl ClusterIP None 5432/TCP 86s scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 49s scalardb-cluster-envoy-metrics ClusterIP 10.111.131.189 9001/TCP 49s scalardb-cluster-graphql LoadBalancer 10.105.74.214 localhost 8080:30514/TCP 49s scalardb-cluster-headless ClusterIP None 60053/TCP 49s scalardb-cluster-metrics ClusterIP 10.110.132.22 9080/TCP 49s ``` If the service resources of ScalarDB Cluster and Envoy are deployed properly, the private IP addresses in the `CLUSTER-IP` column will be displayed. :::note `scalardb-cluster-headless` has no `CLUSTER-IP` address. ::: You can also see `EXTERNAL-IP` addresses assigned to the service resource of ScalarDB Cluster GraphQL (`scalardb-cluster-graphql`) and the service resource of Envoy (`scalardb-cluster-envoy`) with `TYPE` set to `LoadBalancer`. In addition, the access method to the `LoadBalancer` service from your environment depends on each Kubernetes distribution. For example: - If you're using minikube, you can use the [`minikube tunnel` command](https://minikube.sigs.k8s.io/docs/commands/tunnel/). - If you're using kind, you can use [Cloud Provider KIND](https://kind.sigs.k8s.io/docs/user/loadbalancer/). For details on how to access the `LoadBalancer` service, see the official documentation for the Kubernetes distribution that you're using. ## Delete all resources You can delete all resources created in this guide by running the following command: ```console helm uninstall scalardb-cluster postgresql-scalardb-cluster ``` ## Learn more To get familiar with other use cases for ScalarDB Cluster, try the following tutorials: - [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) - [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) - [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) - [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) ================================================ FILE: docs/scalardb-cluster/standalone-mode.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster Standalone Mode import StorageSetupTabs from '../components/_getting-started-setup-storage.mdx'; import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; Instead of setting up a Kubernetes cluster and deploying ScalarDB Cluster on top of it by using a Helm Chart, you can run ScalarDB Cluster in standalone mode, which simplifies development and testing processes. A primary use case for this would be when you want to start ScalarDB Cluster in standalone mode via Docker on your local machine and use it for development and testing. To run ScalarDB Cluster in standalone mode, you need to set the `scalar.db.cluster.node.standalone_mode.enabled` property to `true`: ```properties scalar.db.cluster.node.standalone_mode.enabled=true ``` ## Run ScalarDB Cluster in standalone mode on Docker Compose This section explains how to start ScalarDB Cluster in standalone mode on Docker Compose. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the necessary files by running the following command: ```console cd scalardb-samples/scalardb-cluster-standalone-mode/ ``` ### Set up your database for ScalarDB Cluster Follow the instructions below to configure your database for ScalarDB Cluster. For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). ### Set the license key Set the license key (trial license or commercial license) for the ScalarDB Clusters in the configuration file `scalardb-cluster-node.properties`. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). ### Start ScalarDB Cluster in standalone mode To start ScalarDB Cluster in standalone mode, run the following command: :::note If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. ::: ```console docker compose up -d scalardb-cluster-node ``` ## Client configurations for the ScalarDB Cluster Java API You can use the `indirect` client mode to connect to ScalarDB Cluster in standalone mode. For details about client configurations for the ScalarDB Cluster Java API, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/common-reference.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster .NET Client SDK Reference This reference provides details on how the ScalarDB Cluster .NET Client SDK works. ## Client configuration The client can be configured by using the following: - A settings file, like `appsettings.json` or a custom JSON file - Environment variables - The `ScalarDbOptions` object If you use the SDK with ASP.NET Core, you can configure an app in more ways. For details, see [Configuration in ASP.NET Core](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-8.0). For a list of options that you can configure, see [Available options](common-reference.mdx#available-options). ### Using a settings file The SDK supports both the standard `appsettings.json` and custom JSON setting files. To configure the client in a JSON file, add the `ScalarDbOptions` section and configure the options that you need. For example: ```json { "ScalarDbOptions": { "Address": "http://localhost:60053", "HopLimit": 10 } } ``` Then, create a configured `TransactionFactory` object as follows: ```c# // If appsettings.json is used, call the Create() method without parameters. var factory = TransactionFactory.Create(); // Or, if a custom file is used, call the Create() method that is passed in the path to the custom file as a parameter. factory = TransactionFactory.Create("scalardb-options.json"); ``` If you use the SDK with ASP.NET Core, the settings from `appsettings.json` will be applied automatically when the registered transaction managers and/or `ScalarDbContext` are created. If you want to use a custom JSON file, add it to the configuration framework as follows: ```c# var builder = WebApplication.CreateBuilder(args); // ... builder.Configuration.AddJsonFile("scalardb-options.json"); ``` :::warning Because the custom JSON file is applied after all standard configuration providers, the values from the custom file will override values from other sources. ::: ### Using environment variables To configure the client to use environment variables, you can use the prefix `ScalarDbOptions__`. For example: ```console export ScalarDbOptions__Address="http://localhost:60053" export ScalarDbOptions__HopLimit=10 ``` :::warning Values from environment variables will override values from settings files. ::: ### Using the `ScalarDbOptions` object You can configure the client at runtime by using the `ScalarDbOptions` object as follows: ```c# var options = new ScalarDbOptions() { Address = "http://localhost:60053", HopLimit = 10 }; var factory = TransactionFactory.Create(options); ``` You can also initialize the `ScalarDbOptions` object with values from JSON files and/or environment variables, and then set any remaining values at runtime as follows: ```c# // If appsettings.json is used, call the Load() method without parameters. var options = ScalarDbOptions.Load(); // Or, if a custom file is used, call the Load() method that is passed in the path to the custom file as a parameter. options = ScalarDbOptions.Load("scalardb-options.json"); options.HopLimit = 10; var factory = TransactionFactory.Create(options); ``` If you use the SDK with ASP.NET Core, a lambda function of `AddScalarDb` and/or `AddScalarDbContext` can be used as follows: ```c# var builder = WebApplication.CreateBuilder(args); //... builder.Services.AddScalarDb(options => { options.Address = "http://localhost:60053"; options.HopLimit = 10; }); builder.Services.AddScalarDbContext(options => { options.Address = "http://localhost:60053"; options.HopLimit = 10; }); ``` By using this configuration, the `ScalarDbOptions` object that is passed to the lambda function (named `options` in the example above) is initialized with values from the JSON files, environment variables, and other sources. ### Available options The following options are available: | Name | Description | Default | |-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------| | `Address` | **Required:** Address of the cluster in the following format: `://:`. ``: `https` if wire encryption (TLS) is enabled; `http` otherwise. ``: The FQDN or the IP address of the cluster. ``: The port number (`60053` by default) of the cluster. | - | | `HopLimit` | Number of hops for a request to the cluster. The purpose of `HopLimit` is to prevent infinite loops within the cluster. Each time a request is forwarded to another cluster node, `HopLimit` decreases by one. If `HopLimit` reaches zero, the request will be rejected. | `3` | | `RetryCount` | How many times a client can try to connect to the cluster if it's unavailable. | `10` | | `AuthEnabled` | Whether authentication and authorization are enabled. | `false` | | `Username` | Username for authentication and authorization. | | | `Password` | Password for authentication. If this isn't set, authentication is conducted without a password. | | | `AuthTokenExpirationTime` | Time after which the authentication token should be refreshed. If the time set for `AuthTokenExpirationTime` is greater than the expiration time on the cluster, the authentication token will be refreshed when an authentication error is received. If the authentication token is successfully refreshed, the authentication error won't be propagated to the client code. Instead, the operation that has failed with the authentication error will be retried automatically. If more than one operation is running in parallel, all these operations will fail once with the authentication error before the authentication token is refreshed. | `00:00:00` (The authentication token expiration time received from the cluster is used.) | | `TlsRootCertPem` | Custom CA root certificate (PEM data) for TLS communication. | | | `TlsRootCertPath` | File path to the custom CA root certificate for TLS communication. | | | `TlsOverrideAuthority` | Custom authority for TLS communication. This doesn't change what host is actually connected. This is mainly intended for testing. For example, you can specify the hostname presented in the cluster's certificate (the `scalar.db.cluster.node.tls.cert_chain_path` parameter of the cluster). If there's more than one hostname in the cluster's certificate, only the first hostname will be checked. | | | `LogSensitiveData` | If set to `true`, information like username, password, and authentication token will be logged as is without masking when logging gRPC requests and responses. | `false` | | `GrpcRequestTimeout` | Timeout for gRPC requests. Internally, the timeout's value is used to calculate and set a deadline for each gRPC request to the cluster. If the set deadline is exceeded, the request is cancelled and `DeadlineExceededException` is thrown. If the timeout is set to `0`, no deadline will be set. | `00:01:00` | | `GrpcMaxReceiveMessageSize` | The maximum message size in bytes that can be received by the client. When set to `0`, the message size is unlimited. | `4 MB` | | `GrpcMaxSendMessageSize` | The maximum message size in bytes that can be sent from the client. When set to `0`, the message size is unlimited. | `0` (Unlimited) | ## How ScalarDB column types are converted to and from .NET types When using [LINQ](getting-started-with-linq.mdx#set-up-classes) or extension methods for the [Transactional API](getting-started-with-scalardb-tables-as-csharp-classes.mdx#create-classes-for-all-scalardb-tables), [SQL API](getting-started-with-distributed-sql-transactions.mdx#execute-sql-queries), or [Administrative API](getting-started-with-scalardb-tables-as-csharp-classes.mdx#use-the-administrative-api), a column's value received from the cluster is automatically converted to a corresponding .NET type. Likewise, a value of a .NET property is automatically converted to a corresponding cluster's type when an object is being saved to the cluster. In the following table, you can find how types are converted: | ScalarDB type | .NET type | C# alias | |---------------|----------------------------|----------| | TEXT | System.String | string | | INT | System.Int32 | int | | BIGINT | System.Int64 | long | | FLOAT | System.Single | float | | DOUBLE | System.Double | double | | BOOLEAN | System.Boolean | bool | | BLOB | Google.Protobuf.ByteString | | | DATE | NodaTime.LocalDate | | | TIME | NodaTime.LocalTime | | | TIMESTAMP | NodaTime.LocalDateTime | | | TIMESTAMPTZ | NodaTime.Instant | | :::note The ScalarDB Cluster .NET Client SDK uses [Google.Protobuf](https://www.nuget.org/packages/Google.Protobuf) for `BLOB` type and [NodaTime](https://www.nuget.org/packages/NodaTime) for time-related types. ::: :::warning The precision of time-related types in .NET is greater than supported by ScalarDB. Therefore, you should be careful when saving time-related values received from external sources. The ScalarDB Cluster .NET Client SDK includes `WithScalarDbPrecision` extension methods that you can use to lower the precision of time-related values in the following manner: ```c# using ScalarDB.Client.Extensions; // ... var updatedAt = Instant.FromDateTimeUtc(DateTime.UtcNow) .WithScalarDbPrecision(); // using NodaTime to get current instant updatedAt = clockInstance.GetCurrentInstant() .WithScalarDbPrecision(); ``` For details about value ranges and precision in ScalarDB, see [Database Adapters](../database-adapters.mdx#value-ranges-and-precision). ::: ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/exception-handling.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Exception Handling in the ScalarDB Cluster .NET Client SDK When executing a transaction, you will also need to handle exceptions properly. :::warning If you don't handle exceptions properly, you may face anomalies or data inconsistency. ::: :::note The Transactional API is used in this example, but exceptions can be handled in the same way when using the SQL API or `ScalarDbContext`. ::: The following sample code shows how to handle exceptions: ```c# using System.ComponentModel.DataAnnotations.Schema; using ScalarDB.Client; using ScalarDB.Client.DataAnnotations; using ScalarDB.Client.Exceptions; using ScalarDB.Client.Extensions; var options = new ScalarDbOptions { Address = "http://:"}; var factory = TransactionFactory.Create(options); using var manager = factory.GetTransactionManager(); var retryCount = 0; TransactionException? lastException = null; while (true) { if (retryCount++ > 0) { // Retry the transaction three times maximum in this sample code if (retryCount > 3) // Throw the last exception if the number of retries exceeds the maximum throw lastException!; // Sleep 100 milliseconds before retrying the transaction in this sample code await Task.Delay(100); } // Begin a transaction var tran = await manager.BeginAsync(); try { // Execute CRUD operations in the transaction var getKeys = new Dictionary { { nameof(Item.Id), 1 } }; var result = await tran.GetAsync(getKeys); var scanKeys = new Dictionary { { nameof(Item.Id), 1 } }; await foreach (var item in tran.ScanAsync(scanKeys, null)) Console.WriteLine($"{item.Id}, {item.Name}, {item.Price}"); await tran.InsertAsync(new Item { Id = 1, Name = "Watermelon", Price = 4500 }); await tran.DeleteAsync(new Item { Id = 1 }); // Commit the transaction await tran.CommitAsync(); return; } catch (UnsatisfiedConditionException) { // You need to handle `UnsatisfiedConditionException` only if a mutation operation specifies // a condition. This exception indicates the condition for the mutation operation is not met. // InsertAsync/UpdateAsync implicitlly sets IfNotExists/IfExists condition try { await tran.RollbackAsync(); } catch (TransactionException ex) { // Rolling back the transaction failed. As the transaction should eventually recover, you // don't need to do anything further. You can simply log the occurrence here Console.WriteLine($"Rollback error: {ex}"); } // You can handle the exception here, according to your application requirements return; } catch (UnknownTransactionStatusException) { // If you catch `UnknownTransactionStatusException` when committing the transaction, it // indicates that the status of the transaction, whether it has succeeded or not, is // unknown. In such a case, you need to check if the transaction is committed successfully // or not and retry it if it failed. How to identify a transaction status is delegated to users return; } catch (TransactionException ex) { // For other exceptions, you can try retrying the transaction. // For `TransactionConflictException` and `TransactionNotFoundException`, // you can basically retry the transaction. However, for the other exceptions, // the transaction may still fail if the cause of the exception is nontransient. // In such a case, you will exhaust the number of retries and throw the last exception try { await tran.RollbackAsync(); } catch (TransactionException e) { // Rolling back the transaction failed. As the transaction should eventually recover, // you don't need to do anything further. You can simply log the occurrence here Console.WriteLine($"Rollback error: {e}"); } lastException = ex; } } [Table("order_service.items")] public class Item { [PartitionKey] [Column("item_id", Order = 0)] public int Id { get; set; } [Column("name", Order = 1)] public string Name { get; set; } = String.Empty; [Column("price", Order = 2)] public int Price { get; set; } } ``` :::note In the sample code, the transaction is retried a maximum of three times and sleeps for 100 milliseconds before it is retried. You can choose a retry policy, such as exponential backoff, according to your application requirements. ::: ### Exception details The table below shows transaction exceptions that can occur when communicating with the cluster: | Exception | Operations | Description | |-----------------------------------|--------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | AuthenticationErrorException | All | The authentication failed because of a wrong username and/or password when calling the cluster. | | AuthorizationErrorException | Put, Insert, Update, Delete, Mutate, Execute, Administrative | The authorization failed because of a lack of permissions. | | HopLimitExceededException | All | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | | IllegalArgumentException | All | The argument in the request message is invalid. | | IllegalStateException | All | The RPC was called in an invalid state. | | InternalErrorException | All | The operation failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | | TransactionConflictException | All except Begin, Join, Rollback | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | | TransactionNotFoundException | All except Begin, Join | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | | UnavailableException | All | ScalarDB Cluster is unavailable even after trying to connect multiple times. | | UnknownTransactionStatusException | Commit | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. You are responsible for determining the transaction status. You may benefit from creating a transaction status table and updating it in conjunction with other application data. Doing so may help you determine the status of a transaction from the table itself. | | UnsatisfiedConditionException | Put, Insert, Update, Delete, Mutate | The mutation condition is not satisfied. | If you encounter an exception, you should roll back the transaction, except in the case of `Begin`. After rolling back the transaction, you can retry the transaction from the beginning for the exceptions that can be resolved by retrying. Besides the exceptions listed above, you may encounter exceptions thrown by the gRPC library. In such cases, you can check the `RpcException` property for more information. Also, `ScalarDbContext` will throw a `TransactionException` type exception in the following cases: - If `BeginTransaction` or `JoinTransaction` were called when there was already an active transaction - If `CommitTransaction` or `RollbackTransaction` were called without an active transaction - If `PrepareTransaction` or `ValidateTransaction` were called without an active two-phase commit transaction ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/getting-started-with-admin-api.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with the Administrative API in the ScalarDB Cluster .NET Client SDK The ScalarDB Cluster .NET Client SDK supports the Administrative API of ScalarDB Cluster. By using this API, you can manage ScalarDB Cluster from .NET applications. :::note Although we recommend using asynchronous methods as in the following examples, you can use synchronous methods instead. ::: ## Install the SDK Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: ```console dotnet add package ScalarDB.Client --version '..*' ``` ## Create a settings file Create a `scalardb-options.json` file and add the following, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: ```json { "ScalarDbOptions": { "Address": "http://:", "HopLimit": 10 } } ``` For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). ## Get a transaction manager You need to get an object for interacting with the Administrative API. To get the object, you can use `TransactionFactory` as follows: ```c# // Pass the path to the settings file created in the previous step. var factory = TransactionFactory.Create("scalardb-options.json"); using var admin = factory.GetTransactionAdmin(); ``` ## Manage ScalarDB Cluster The following operations can be performed by using the ScalarDB Cluster .NET Client SDK. ### Create a new namespace ```c# await admin.CreateNamespaceAsync("ns", ifNotExists: true); ``` ### Drop a namespace ```c# await admin.DropNamespaceAsync("ns", ifExists: true); ``` ### Check if a namespace exists ```c# var namespaceExists = await admin.IsNamespacePresentAsync("ns"); ``` ### Create a new table ```c# // ... using ScalarDB.Client.Builders.Admin; using ScalarDB.Client.Core; // ... var tableMetadata = new TableMetadataBuilder() .AddPartitionKey("pk", DataType.Int) .AddClusteringKey("ck", DataType.Double) .AddSecondaryIndex("index", DataType.Float) .AddColumn("ordinary", DataType.Text) .Build(); await admin.CreateTableAsync("ns", "table_name", tableMetadata, ifNotExists: true); ``` ### Drop a table ```c# await admin.DropTableAsync("ns", "table_name", ifExists: true); ``` ### Checking if a table exists ```c# var tableExists = await admin.IsTablePresentAsync("ns", "table_name"); ``` ### Get the names of existing tables ```c# var tablesList = await admin.GetTableNamesAsync("ns"); ``` ### Create the Coordinator table ```c# await admin.CreateCoordinatorTablesAsync(); ``` ### Drop the Coordinator table ```c# await admin.DropCoordinatorTablesAsync(); ``` ### Check if the Coordinator table exist ```c# var exists = await admin.AreCoordinatorTablesPresentAsync(); ``` ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/getting-started-with-aspnet-and-di.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ASP.NET Core and Dependency Injection in the ScalarDB Cluster .NET Client SDK The ScalarDB Cluster .NET Client SDK supports dependency injection (DI) in frameworks like ASP.NET Core. ## Install the SDK Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: ```console dotnet add package ScalarDB.Client --version '..*' ``` ## Add client settings Add the `ScalarDbOptions` section to the `appsettings.json` file of your ASP.NET Core app, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: ```json { "ScalarDbOptions": { "Address": "http://:", "HopLimit": 10 } } ``` For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). ## Set up the transaction managers You can register the ScalarDB transaction managers in the DI container as follows: ```c# using ScalarDB.Client.Extensions; //... var builder = WebApplication.CreateBuilder(args); //... builder.Services.AddScalarDb(); ``` :::note The ScalarDB transaction managers will be registered as transient services. For details about service lifetimes, see [.NET dependency injection - Service lifetimes](https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#service-lifetimes). ::: After registering the transaction managers, they can be injected into the controller's constructor as follows: ```c# [ApiController] public class OrderController: ControllerBase { private readonly IDistributedTransactionManager _manager; private readonly ISqlTransactionManager _sqlManager; private readonly ITwoPhaseCommitTransactionManager _twoPhaseManager; private readonly ISqlTwoPhaseCommitTransactionManager _sqlTwoPhaseManager; private readonly IDistributedTransactionAdmin _admin; public OrderController(IDistributedTransactionManager manager, ISqlTransactionManager sqlManager, ITwoPhaseCommitTransactionManager twoPhaseManager, ISqlTwoPhaseCommitTransactionManager sqlTwoPhaseManager, IDistributedTransactionAdmin admin) { _manager = manager; _sqlManager = sqlManager; _twoPhaseManager = twoPhaseManager; _sqlTwoPhaseManager = sqlTwoPhaseManager; _admin = admin; } } ``` Although these examples are for WebApi projects, the examples will work in a similar way in GrpcService projects. ## Use read-only transactions ScalarDB Cluster supports read-only transactions for distributed and SQL transaction managers. After injecting `IDistributedTransactionManager` or `ISqlTransactionManager`, call `BeginReadOnlyAsync`/`BeginReadOnly` to hint to the server that only read operations will be performed: ```c# var tran = await distributedTransactionManager.BeginReadOnlyAsync(); // Execute only read operations inside this transaction await tran.GetAsync(get); await tran.CommitAsync(); ``` :::note Read-only mode is *not* supported for two-phase commit transaction managers (`ITwoPhaseCommitTransactionManager`, `ISqlTwoPhaseCommitTransactionManager`). Calling `BeginReadOnly*` on those managers throws `NotSupportedException`. ::: ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/getting-started-with-auth.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Authentication and Authorization by Using ScalarDB Cluster .NET Client SDK The ScalarDB Cluster .NET Client SDK supports [authentication and authorization](../scalardb-cluster/scalardb-auth-with-sql.mdx), which allows you to authenticate and authorize your requests to ScalarDB Cluster. ## Install the SDK Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: ```console dotnet add package ScalarDB.Client --version '..*' ``` ## Set credentials in the settings file You need to set credentials in the settings file as follows, replacing the contents in the angle brackets as described: ```json { "ScalarDbOptions": { "Address": "http://:", "HopLimit": 10, "AuthEnabled": true, "Username": "", "Password": "" } } ``` For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). ## Get a transaction manager You need to get a transaction manager or transaction admin object by using `TransactionFactory` as follows. Be sure to replace `` with `GetTransactionManager()`, `GetTwoPhaseCommitTransactionManager()`, `GetSqlTransactionManager()`, or `GetSqlTwoPhaseCommitTransactionManager()`. ```c# // Pass the path to the settings file. var factory = TransactionFactory.Create("scalardb-options.json"); // To get a transaction manager using var manager = factory.(); // To get a transaction admin using var admin = factory.GetTransactionAdmin(); ``` A transaction manager or transaction admin object created from `TransactionFactory` with the provided credentials will automatically log in to ScalarDB Cluster and can communicate with it. ## Wire encryption [Wire encryption](../scalardb-cluster/scalardb-auth-with-sql.mdx#wire-encryption) is also supported. It can be turned on by setting `Address` to the URL starting with `https` as follows: ```json { "ScalarDbOptions": { "Address": "https://:" } } ``` For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-sql-transactions.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Distributed SQL Transactions in the ScalarDB Cluster .NET Client SDK The ScalarDB Cluster .NET Client SDK supports the distributed SQL transaction functionality of ScalarDB Cluster. The SDK includes transaction and manager abstractions for easier communication within a cluster. :::note Although we recommend using asynchronous methods, as in the following examples, you can use synchronous methods instead. ::: For details about distributed non-SQL transactions, see [Getting Started with Distributed Transactions in the ScalarDB Cluster .NET Client SDK](getting-started-with-distributed-transactions.mdx). ## Install the SDK Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: ```console dotnet add package ScalarDB.Client --version '..*' ``` ## Create a settings file Create a `scalardb-options.json` file and add the following, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: ```json { "ScalarDbOptions": { "Address": "http://:", "HopLimit": 10 } } ``` For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). ## Get a transaction manager You need to get a transaction manager object for distributed SQL transactions. To get the transaction manager object, you can use `TransactionFactory` as follows: ```c# // Pass the path to the settings file created in the previous step. var factory = TransactionFactory.Create("scalardb-options.json"); using var manager = factory.GetSqlTransactionManager(); ``` ## Execute SQL queries To execute a SQL statement, you need an `ISqlStatement` object, which can be created by using a builder as follows: ```c# using ScalarDB.Client.Builders.Sql; // ... var sqlStatement = new SqlStatementBuilder() .SetSql("SELECT * FROM order_service.statements WHERE item_id = :item_id") .AddParam("item_id", 2) .Build(); ``` A single SQL statement can be executed directly by using the transaction manager as follows: ```c# var resultSet = await manager.ExecuteAsync(sqlStatement); ``` The result from the `ExecuteAsync` method will contain records received from the cluster. The value of the specific column can be retrieved in the following manner: ```c# foreach (var record in resultSet.Records) { // Getting an integer value from the "item_id" column. // If it fails, an exception will be thrown. var itemId = record.GetValue("item_id"); // Trying to get a string value from the "order_id" column. // If it fails, no exception will be thrown. if (record.TryGetValue("order_id", out var orderId)) Console.WriteLine($"order_id: {orderId}"); // Checking if the "count" column is null. if (record.IsNull("count")) Console.WriteLine("'count' is null"); } ``` For details about which type should be used in `GetValue` and `TryGetValue`, see [How ScalarDB Column Types Are Converted to and from .NET Types](common-reference.mdx#how-scalardb-column-types-are-converted-to-and-from-net-types). ## Execute SQL queries in a transaction To execute multiple SQL statements as part of a single transaction, you need a transaction object. You can create a transaction object by using the transaction manager as follows: ```c# var transaction = await manager.BeginAsync(); ``` You can also resume a transaction that has already been started as follows: ```c# var transaction = manager.Resume(transactionIdString); ``` :::note The `Resume` method doesn't have an asynchronous version because it only creates a transaction object. Because of this, resuming a transaction by using the wrong ID is possible. ::: ### Begin a read-only transaction If you perform only read operations (that is, `SELECT` statements) in a transaction, you can begin a read-only transaction as follows: ```c# var transaction = await manager.BeginReadOnlyAsync(); ``` :::note Read-only transactions are processed more efficiently than read transactions begun without the read-only mode by assuming no write operations are issued within them. Therefore, attempting to perform write operations (`INSERT`, `UPDATE`, or `DELETE` statements) in a read-only transaction will result in an error. ::: The transaction has the same `ExecuteAsync` method as the transaction manager. That method can be used to execute SQL statements. When a transaction is ready to be committed, you can call the `CommitAsync` method of the transaction as follows: ```c# await transaction.CommitAsync(); ``` To roll back the transaction, you can use the `RollbackAsync` method: ```c# await transaction.RollbackAsync(); ``` ## Execute SQL statements in a batch The `ExecuteBatchAsync` method allows you to execute multiple SQL statements in a single RPC call, reducing round-trip overhead when performing batch operations. You can use this method with both the transaction manager and the transaction object. ### Execute a batch by using the transaction manager You can execute a batch of SQL statements directly by using the transaction manager as follows: ```c# using ScalarDB.Client.Builders.Sql; // ... var statements = new[] { new SqlStatementBuilder() .SetSql("INSERT INTO ns.statements (order_id, item_id, count) VALUES (:order_id, :item_id, :count)") .AddParam("order_id", "1") .AddParam("item_id", 1) .AddParam("count", 5) .Build(), new SqlStatementBuilder() .SetSql("INSERT INTO ns.statements (order_id, item_id, count) VALUES (:order_id, :item_id, :count)") .AddParam("order_id", "2") .AddParam("item_id", 2) .AddParam("count", 10) .Build() }; var resultSets = await manager.ExecuteBatchAsync(statements); ``` ### Execute a batch in a transaction You can also execute a batch of SQL statements as part of a transaction as follows: ```c# var transaction = await manager.BeginAsync(); var resultSets = await transaction.ExecuteBatchAsync(statements); await transaction.CommitAsync(); ``` The results are returned as `IReadOnlyList` in the same order as the input statements. For `SELECT` statements, the result set contains the retrieved records. For `INSERT`, `UPDATE`, and `DELETE` statements, the result set is empty. The following example executes multiple `SELECT` statements in a batch and processes the results: ```c# var selectStatements = new[] { new SqlStatementBuilder() .SetSql("SELECT * FROM ns.statements WHERE order_id = :order_id") .AddParam("order_id", "1") .Build(), new SqlStatementBuilder() .SetSql("SELECT * FROM ns.statements WHERE order_id = :order_id") .AddParam("order_id", "2") .Build() }; var resultSets = await transaction.ExecuteBatchAsync(selectStatements); for (var i = 0; i < resultSets.Count; i++) { foreach (var record in resultSets[i].Records) { var itemId = record.GetValue("item_id"); Console.WriteLine($"item_id: {itemId}"); } } ``` ### Use positional parameters In addition to named parameters, you can use positional parameters as follows: ```c# var statements = new[] { new SqlStatementBuilder() .SetSql("SELECT * FROM ns.statements WHERE order_id = ? AND item_id = ?") .AddParam("1") .AddParam(1) .Build(), new SqlStatementBuilder() .SetSql("SELECT * FROM ns.statements WHERE order_id = ? AND item_id = ?") .AddParam("2") .AddParam(2) .Build() }; var resultSets = await transaction.ExecuteBatchAsync(statements); ``` :::note If an empty collection is provided to `ExecuteBatchAsync`, no RPC call is made and an empty list is returned. ::: ## Get Metadata You can retrieve ScalarDB's metadata with the Metadata property as follows: ```c# // namespaces, tables metadata var namespaceNames = new List(); await foreach (var ns in manager.Metadata.GetNamespacesAsync()) { namespaceNames.Add(ns.Name); Console.WriteLine($"Namespace: {ns.Name}"); await foreach (var tbl in ns.GetTablesAsync()) { Console.WriteLine($" Table: {tbl.Name}"); Console.WriteLine($" Columns:"); foreach (var col in tbl.Columns) Console.WriteLine($" {col.Name} [{col.DataType}]"); Console.WriteLine($" PartitionKey:"); foreach (var col in tbl.PartitionKey) Console.WriteLine($" {col.Name}"); Console.WriteLine($" ClusteringKey:"); foreach (var col in tbl.ClusteringKey) Console.WriteLine($" {col.Name} [{col.ClusteringOrder}]"); Console.WriteLine($" Indexes:"); foreach (var index in tbl.Indexes) Console.WriteLine($" {index.ColumnName}"); Console.WriteLine(); } } // users metadata await foreach (var user in manager.Metadata.GetUsersAsync()) { Console.WriteLine($"User: {user.Name} [IsSuperuser: {user.IsSuperuser}]"); foreach (var nsName in namespaceNames) { Console.WriteLine($" Namespace: {nsName}"); Console.WriteLine($" Privileges:"); foreach (var privilege in await user.GetPrivilegesAsync(nsName)) Console.WriteLine($" {privilege}"); } Console.WriteLine(); } ``` :::note To use LINQ methods with `IAsyncEnumerable`, you can install [System.Linq.Async](https://www.nuget.org/packages/System.Linq.Async/) package. ::: ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-transactions.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Distributed Transactions in the ScalarDB Cluster .NET Client SDK The ScalarDB Cluster .NET Client SDK supports the distributed transaction functionality of ScalarDB Cluster. The SDK includes transaction and manager abstractions for easier communication within a cluster. :::note Although we recommend using asynchronous methods as in the following examples, you can use synchronous versions instead. ::: For details about distributed SQL transactions, see [Getting Started with Distributed SQL Transactions in the ScalarDB Cluster .NET Client SDK](getting-started-with-distributed-sql-transactions.mdx). ## Install the SDK Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: ```console dotnet add package ScalarDB.Client --version '..*' ``` ## Create a settings file Create a `scalardb-options.json` file and add the following, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: ```json { "ScalarDbOptions": { "Address": "http://:", "HopLimit": 10 } } ``` For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). ## Get a transaction manager You need to get a transaction manager for distributed transactions. To get the transaction manager, you can use `TransactionFactory` as follows: ```c# // Pass the path to the settings file created in the previous step. var factory = TransactionFactory.Create("scalardb-options.json"); using var manager = factory.GetTransactionManager(); ``` ## Manage transactions To execute multiple CRUD operations as part of a single transaction, first, you need to begin a transaction. You can begin a transaction by using the transaction manager as follows: ```c# var transaction = await manager.BeginAsync(); ``` You can also resume a transaction that is already being executed as follows: ```c# var transaction = manager.Resume(transactionIdString); ``` :::note The `Resume` method doesn't have an asynchronous version because it only creates a transaction object. Because of this, resuming a transaction by using the wrong ID is possible. ::: ### Begin a read-only transaction If you perform only read operations (that is, `GetAsync` and `ScanAsync`) in a transaction, you can begin a read-only transaction as follows: ```c# var transaction = await manager.BeginReadOnlyAsync(); ``` :::note Read-only transactions are optimized for read operations only. Attempting to perform write operations (`InsertAsync`, `UpsertAsync`, `UpdateAsync`, `DeleteAsync`, `MutateAsync`, or `BatchAsync` with write operations) in a read-only transaction will result in an error. ::: When a transaction is ready to be committed, you can call the `CommitAsync` method of the transaction as follows: ```c# await transaction.CommitAsync(); ``` To roll back the transaction, you can use the `RollbackAsync` method: ```c# await transaction.RollbackAsync(); ``` ## Execute CRUD operations A transaction has `GetAsync`, `ScanAsync`, `InsertAsync`, `UpsertAsync`, `UpdateAsync`, `DeleteAsync`, and `MutateAsync` methods to execute CRUD operations against the cluster. As a parameter, these methods have an operation object. An operation object can be created by using the builders listed in this section. :::note CRUD operations can be executed in a one-shot transaction manner without needing to explicitly create a transaction. For that, a manager object has the same CRUD methods as a transaction object. ::: To use builders, add the following namespace to the `using` section: ```c# using ScalarDB.Client.Builders; ``` :::note The cluster does not support parallel execution of commands inside one transaction, so make sure to use `await` for asynchronous methods. ::: ### `GetAsync` method example To retrieve a single record, you can use the `GetAsync` method as follows: ```c# var get = new GetBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "1") .AddClusteringKey("item_id", 2) .SetProjections("item_id", "count") .Build(); var getResult = await transaction.GetAsync(get); ``` It is possible to retrieve a record by using an index instead of a partition key. To do that, you need to set the type of operation to `GetWithIndex` as follows: ```c# // ... using ScalarDB.Client.Core; // ... var get = new GetBuilder() // ... .SetGetType(GetOperationType.GetWithIndex) .AddPartitionKey("index_column", "1") .Build(); ``` You can also specify arbitrary conditions that a retrieved record must meet, or it won't be returned. The conditions can be set as conjunctions of conditions as follows: ```c# var get = new GetBuilder() // ... .AddConjunction(c => c.AddCondition("cost", 1000, Operator.LessThan)) .AddConjunction(c => { c.AddCondition("cost", 10000, Operator.LessThan); c.AddCondition("in_stock", true, Operator.Equal); }) .Build(); ``` In the above example, a record will be returned only if its `cost` is less than `1000`, or if its `cost` is less than `10000` and `in_stock` is true. #### Handle `IResult` objects The `GetAsync` and `ScanAsync` methods return `IResult` objects. An `IResult` object contains columns of the retrieved record. The value of the specific column can be retrieved in the following manner: ```c# // Getting an integer value from the "item_id" column. // If it fails, an exception will be thrown. var itemId = result.GetValue("item_id"); // Trying to get a string value from the "order_id" column. // If it fails, no exception will be thrown. if (result.TryGetValue("order_id", out var orderId)) Console.WriteLine($"order_id: {orderId}"); // Checking if the "count" column is null. if (result.IsNull("count")) Console.WriteLine("'count' is null"); ``` For details about which type should be used in `GetValue` and `TryGetValue`, see [How ScalarDB Column Types Are Converted to and from .NET Types](common-reference.mdx#how-scalardb-column-types-are-converted-to-and-from-net-types). ### `ScanAsync` method example To retrieve a range of records, you can use the `ScanAsync` method as follows: ```c# var scan = new ScanBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "1") .AddStartClusteringKey("item_id", 2) .SetStartInclusive(true) .AddEndClusteringKey("item_id", 8) .SetEndInclusive(true) .SetProjections("item_id", "count") .Build(); var scanResult = await transaction.ScanAsync(scan); ``` It is possible to retrieve a record by using an index instead of a partition key. To do that, you need to set the type of operation to `ScanWithIndex` as follows: ```c# // ... using ScalarDB.Client.Core; // ... var scan = new ScanBuilder() // ... .SetScanType(ScanOperationType.ScanWithIndex) .AddPartitionKey("index_column", "1") .Build(); ``` The arbitrary conditions that a retrieved record must meet can also be set for a scan operation in the same way as for a [get operation](getting-started-with-distributed-transactions.mdx#getasync-method-example). ### Use a scanner to retrieve large datasets The `ScanAsync` method returns all results at once, which can be memory intensive for large datasets. To handle large datasets more efficiently, you can use a scanner that streams results from the cluster and enables incremental processing. #### Create a scanner You can create a scanner from a transaction by using the `CreateScannerAsync` method: ```c# using ScalarDB.Client.Builders; using ScalarDB.Client.Core; // ... var scan = new ScanBuilder() .SetNamespaceName("ns") .SetTableName("items") .SetScanType(ScanOperationType.ScanAll) .Build(); await using var scanner = await transaction.CreateScannerAsync(scan); ``` :::note The scanner must be disposed of after use to release server-side resources. If the scanner's lifetime is limited to a single scope, use `await using` (or `using` for synchronous code) for automatic cleanup. If you need to return a scanner from a method or manage its lifetime across multiple scopes, call `DisposeAsync()` (or `Dispose()`) explicitly when done. ::: #### Retrieve results one by one Use the `OneAsync` method to retrieve results one at a time: ```c# await using var scanner = await transaction.CreateScannerAsync(scan); while (true) { var result = await scanner.OneAsync(); if (result == null) break; // Process the result. var itemId = result.GetValue("item_id"); Console.WriteLine($"Item ID: {itemId}"); } ``` If there are no more results, `OneAsync` returns `null`. Subsequent calls to `OneAsync` after receiving `null` will continue to return `null`. #### Retrieve all remaining results Use the `AllAsync` method to retrieve all remaining results at once: ```c# await using var scanner = await transaction.CreateScannerAsync(scan); var results = await scanner.AllAsync(); foreach (var result in results) { // Process each result. var itemId = result.GetValue("item_id"); Console.WriteLine($"Item ID: {itemId}"); } ``` :::note If you have already retrieved some results by using `OneAsync`, the `AllAsync` method will return only the remaining results that have not yet been retrieved. ::: #### Configure scanner fetch size You can configure the default fetch size for scanner operations by using `ScalarDbOptions`. The fetch size determines how many records are fetched from the server in each batch: ```c# var scalarDbOptions = new ScalarDbOptions { Address = "http://:", ScannerFetchSize = 20 // Default is 10. }; ``` #### Use a scanner with scan options The scanner supports all scan options available in `ScanBuilder`. ##### Scan with limit ```c# using ScalarDB.Client.Builders; using ScalarDB.Client.Core; // ... var scanWithLimit = new ScanBuilder() .SetNamespaceName("ns") .SetTableName("items") .SetScanType(ScanOperationType.ScanAll) .SetLimit(100) // Limit to 100 results. .Build(); await using var scanner = await transaction.CreateScannerAsync(scanWithLimit); var results = await scanner.AllAsync(); ``` ##### Scan with clustering key range ```c# using ScalarDB.Client.Builders; using ScalarDB.Client.Core; // ... var scanWithRange = new ScanBuilder() .SetNamespaceName("ns") .SetTableName("statements") .SetScanType(ScanOperationType.Scan) .AddPartitionKey("order_id", "1") .AddStartClusteringKey("item_id", 2) .SetStartInclusive(true) .AddEndClusteringKey("item_id", 8) .SetEndInclusive(true) .Build(); await using var scanner = await transaction.CreateScannerAsync(scanWithRange); ``` #### Close the scanner The scanner implements both `IDisposable` and `IAsyncDisposable`. The recommended approach is to use `await using` (or `using` for synchronous code) for automatic disposal: ```c# await using var scanner = await transaction.CreateScannerAsync(scan); // The scanner is automatically closed when leaving the scope. ``` If you need to close the scanner explicitly, you can use the `CloseAsync` method: ```c# var scanner = await transaction.CreateScannerAsync(scan); try { var result = await scanner.OneAsync(); // Process result... } finally { await scanner.CloseAsync(); } ``` #### Synchronous API All scanner methods have synchronous versions: ```c# using var scanner = transaction.CreateScanner(scan); // Retrieve one result. var result = scanner.One(); // Retrieve all remaining results. var results = scanner.All(); // Close the scanner. scanner.Close(); ``` #### Scanner vs. ScanAsync | Feature | Scanner | ScanAsync | |---------|---------|-----------| | Memory usage | Low (streaming) | Higher (all results in memory) | | Use case | Large datasets | Small to medium datasets | | Result retrieval | Incremental (`OneAsync`) or all at once (`AllAsync`) | All at once | | Resource management | Requires explicit disposal | Automatic | :::note For details on handling exceptions during scanner and other transaction operations, see [Exception Handling in the ScalarDB Cluster .NET Client SDK](exception-handling.mdx). ::: ### `InsertAsync` method example To insert a new record, you can use the `InsertAsync` method as follows: ```c# var insert = new InsertBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "1") .AddClusteringKey("item_id", 2) .AddColumn("count", 11) .Build(); await transaction.InsertAsync(insert); ``` ### `UpsertAsync` method example To upsert a record (update an existing record or insert a new one), you can use the `UpsertAsync` method as follows: ```c# var upsert = new UpsertBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "1") .AddClusteringKey("item_id", 2) .AddColumn("count", 11) .Build(); await transaction.UpsertAsync(upsert); ``` ### `UpdateAsync` method example To update an existing record, you can use the `UpdateAsync` method as follows: ```c# // ... using ScalarDB.Client.Core; // ... var update = new UpdateBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "1") .AddClusteringKey("item_id", 2) .AddColumn("count", 11) .AddCondition("processed", false, Operator.Equal) .Build(); await transaction.UpdateAsync(update); ``` ### `DeleteAsync` method example To delete a record, you can use the `DeleteAsync` method as follows: ```c# // ... using ScalarDB.Client.Core; // ... var delete = new DeleteBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "1") .AddClusteringKey("item_id", 2) .AddCondition("processed", false, Operator.Equal) .Build(); await transaction.DeleteAsync(delete); ``` ### `MutateAsync` method example The `MutateAsync` method allows you to execute more than one mutation operation in a single call to the cluster. You can do this in the following manner: ```c# // ... using ScalarDB.Client.Core; // ... var mutations = new IMutation[] { new InsertBuilder() // ... .Build(), new UpsertBuilder() // ... .Build(), new UpdateBuilder() // ... .Build(), new DeleteBuilder() // ... .Build() }; await transaction.MutateAsync(mutations); ``` ### `BatchAsync` method example The `BatchAsync` method allows you to execute multiple operations, including both read and write operations, in a single request. This can improve performance by reducing network round trips. You can use this method as follows: ```c# using ScalarDB.Client.Core; // ... var operations = new IOperation[] { new GetBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "1") .AddClusteringKey("item_id", 2) .Build(), new ScanBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "2") .Build(), new InsertBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "3") .AddClusteringKey("item_id", 1) .AddColumn("count", 5) .Build(), new DeleteBuilder() .SetNamespaceName("ns") .SetTableName("statements") .AddPartitionKey("order_id", "4") .AddClusteringKey("item_id", 1) .Build() }; var results = await transaction.BatchAsync(operations); ``` The results are returned in the same order as the input operations. You can inspect each result by using the type-checking properties as follows: ```c# for (var i = 0; i < results.Count; i++) { var result = results[i]; if (result.IsGetResult) { // GetResult is null if the record was not found. if (result.GetResult != null) { var count = result.GetResult.GetValue("count"); } } else if (result.IsScanResult) { foreach (var record in result.ScanResults) { var itemId = record.GetValue("item_id"); } } else if (result.IsInsertResult || result.IsUpsertResult || result.IsUpdateResult || result.IsDeleteResult) { // Mutation operations don't return data. } } ``` The supported operation types for `BatchAsync` are `Get`, `Scan`, `Insert`, `Upsert`, `Update`, and `Delete`. ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/getting-started-with-linq.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with LINQ in the ScalarDB Cluster .NET Client SDK The ScalarDB Cluster .NET Client SDK supports querying the cluster with LINQ and some Entity Framework-like functionality. :::note This SDK doesn't support [Entity Framework](https://learn.microsoft.com/en-us/ef/). Instead, this SDK implements functionality that is similar to Entity Framework. ::: :::note SQL support must be enabled on the cluster to use LINQ. ::: ## Install the SDK Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: ```console dotnet add package ScalarDB.Client --version '..*' ``` ## Add client settings Add the `ScalarDbOptions` section to the `appsettings.json` file of your ASP.NET Core app, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: ```json { "ScalarDbOptions": { "Address": "http://:", "HopLimit": 10 } } ``` For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). ## Set up classes After confirming that SQL support is enabled, create a C# class for each ScalarDB table that you want to use. For example: ```c# using System.ComponentModel.DataAnnotations.Schema; using ScalarDB.Client.DataAnnotations; // ... [Table("ns.statements")] public class Statement { [PartitionKey] [Column("statement_id", Order = 0)] public int Id { get; set; } [SecondaryIndex] [Column("order_id", Order = 1)] public string OrderId { get; set; } = String.Empty; [SecondaryIndex] [Column("item_id", Order = 2)] public int ItemId { get; set; } [Column("quantity", Order = 3)] public int Quantity { get; set; } } [Table("order_service.items")] public class Item { [PartitionKey] [Column("item_id", Order = 0)] public int Id { get; set; } [Column("name", Order = 1)] public string Name { get; set; } = String.Empty; [Column("price", Order = 2)] public int Price { get; set; } } ``` If a partition key, clustering key, or secondary index consists of more than one column, the `Order` property of `ColumnAttribute` will decide the order inside the key or index. For details about which types should be used for properties, see [How ScalarDB Column Types Are Converted to and from .NET Types](common-reference.mdx#how-scalardb-column-types-are-converted-to-and-from-net-types). Create a context class that has properties for all the tables you want to use. For example: ```c# public class MyDbContext: ScalarDbContext { public ScalarDbSet Statements { get; set; } public ScalarDbSet Items { get; set; } } ``` After all the classes are created, you need to register the created context in the dependency injection container. For example: ```c# using ScalarDB.Client.Extensions; //... var builder = WebApplication.CreateBuilder(args); //... builder.Services.AddScalarDbContext(); ``` :::note The context class will be registered as a transient service. For details about service lifetimes, see [.NET dependency injection - Service lifetimes](https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#service-lifetimes). ::: The context can be injected into the controller's constructor as follows: ```c# [ApiController] public class OrderController: ControllerBase { private readonly MyDbContext _myDbContext; public OrderController(MyDbContext myDbContext) { _myDbContext = myDbContext; } } ``` ## Use LINQ to query properties After receiving `MyDbContext` in your controller, you can query its properties by using LINQ. For example: ### Use query syntax ```c# from stat in _myDbContext.Statements join item in _myDbContext.Items on stat.ItemId equals item.Id where stat.Quantity > 2 && item.Name.Contains("apple") orderby stat.Quantity descending, stat.ItemId select new { item.Name, stat.Quantity }; ``` ### Use method syntax ```c# _myDbContext.Statements .Where(stat => stat.OrderId == "1") .Skip(1) .Take(2); ``` ### Use the `First` method to get one `Statement` by its partition key ```c# _myDbContext.Statements.First(stat => stat.OrderId == "1"); ``` ### Use the `DefaultIfEmpty` method to perform left outer join ```c# from stat in _myDbContext.Statements join item in _myDbContext.Items on stat.ItemId equals item.Id into items from i in items.DefaultIfEmpty() select new { ItemName = i != null ? i.Name : "" } ``` ### Use the `GroupBy` method to group and aggregate data ```c# // Group by single key with count _myDbContext.Statements .GroupBy(stat => stat.ItemId) .Select(g => new { ItemId = g.Key, Count = g.Count() }); ``` ```c# // Group by composite key _myDbContext.Statements .GroupBy(stat => new { stat.ItemId, stat.OrderId }) .Select(g => new { ItemId = g.Key.ItemId, OrderId = g.Key.OrderId, Count = g.Count() }); ``` ```c# // With WHERE before GROUP BY _myDbContext.Statements .Where(stat => stat.Quantity > 0) .GroupBy(stat => stat.ItemId) .Select(g => new { ItemId = g.Key, Total = g.Sum(x => x.Quantity) }); ``` ```c# // With HAVING clause (Where after GroupBy filters groups) _myDbContext.Statements .GroupBy(stat => stat.ItemId) .Where(g => g.Count() > 5) .Select(g => new { ItemId = g.Key, Count = g.Count() }); ``` ```c# // With multiple aggregate functions _myDbContext.Statements .GroupBy(stat => stat.ItemId) .Select(g => new { ItemId = g.Key, Count = g.Count(), TotalQuantity = g.Sum(x => x.Quantity), AverageQuantity = g.Average(x => x.Quantity), MinQuantity = g.Min(x => x.Quantity), MaxQuantity = g.Max(x => x.Quantity) }); ``` ```c# // Aggregate all data by using a constant key to retrieve multiple aggregates in a single query _myDbContext.Statements .GroupBy(_ => 1) .Select(g => new { Count = g.Count(), TotalQuantity = g.Sum(x => x.Quantity), AverageQuantity = g.Average(x => x.Quantity), MinQuantity = g.Min(x => x.Quantity), MaxQuantity = g.Max(x => x.Quantity) }); ``` ### Use standalone aggregate methods without `GroupBy` ```c# // Count all records _myDbContext.Statements.Count(); ``` ```c# // Count with a condition _myDbContext.Statements.Count(stat => stat.Quantity > 0); ``` ```c# // Other standalone aggregate functions _myDbContext.Statements.Sum(stat => stat.Quantity); _myDbContext.Statements.Average(stat => stat.Quantity); _myDbContext.Statements.Min(stat => stat.Quantity); _myDbContext.Statements.Max(stat => stat.Quantity); ``` The following methods are translated into SQL and executed remotely on the cluster. For chainable methods, any unsupported methods that follow will fall back to being executed locally in memory by using LINQ to Objects. Terminal methods like `First`/`FirstOrDefault` and standalone aggregate methods return a result directly. - `Select` - `Where` - `Join` - `GroupJoin` - `GroupBy` - `Skip` - `Take` - `OrderBy`/`OrderByDescending` - `ThenBy`/`ThenByDescending` - `First`/`FirstOrDefault` - `Count`/`LongCount` - `Sum` - `Average` - `Min` - `Max` The following aggregate functions are supported inside `Select` after `GroupBy`: - `Count`/`LongCount` - `Sum` - `Average` - `Min` - `Max` The following `String` methods are supported inside the predicates of `Where` and `First`/`FirstOrDefault` methods: - `Contains` - `StartsWith` - `EndsWith` Unsupported LINQ methods can be used after the supported methods. For example: ```c# _myDbContext.Statements .Where(stat => stat.OrderId == "1") // Will be executed remotely on the cluster. .Distinct() // Will be executed locally in the app. .Where(stat => stat.ItemId < 5); // Will be executed locally. ``` :::note If `Skip` is specified before `Take` or `First`/`FirstOrDefault`, the number that is passed to `Skip` will be added to the `LIMIT` number in the SQL query. By itself, `Skip` won't change the resulting SQL query. ::: ## Limitations when using LINQ against `ScalarDbSet{T}` objects - All method calls are supported inside `Select`. For example: ```c# .Select(stat => convertToSomething(stat.ItemId)) //... .Select(stat => stat.ItemId * getSomeNumber()) ``` - Method calls, except for calls against the querying object, are also supported inside `Where` and `First`/`FirstOrDefault`. For example: ```c# .Where(stat => stat.ItemId == getItemId()) // is OK //... .Where(stat => stat.ItemId.ToString() == "1") // is not supported ``` - All method calls are supported inside the result-selecting lambda of `Join` and `GroupJoin`. For example: ```c# .Join(_myDbContext.Items, stat => stat.ItemId, item => item.Id, (stat, item) => new { ItemName = convertToSomething(item.Name), ItemQuantity = stat.Quantity.ToString() }) ``` - Method calls are not supported inside the key-selecting lambdas of `Join` and `GroupJoin`. - Custom equality comparers are not supported. The `comparer` argument in `Join` and `GroupJoin` methods will be ignored if the argument has been passed. - More than one `from` directly in one query is not supported, except when the `DefaultIfEmpty` method is used to perform left outer join. Each subsequent `from` is considered to be a separate query. ```c# var firstQuery = from stat in _myDbContext.Statements where stat.Quantity > 2 select new { stat.Quantity }; var secondQuery = from item in _myDbContext.Items where item.Price > 6 select new { item.Name }; var finalQuery = from first in firstQuery from second in secondQuery select new { first.Quantity, second.Name }; // 1. firstQuery will be executed against the cluster. // 2. secondQuery will be executed against the cluster for each object (row) from 1. // 3. finalQuery will be executed locally with the results from 1 and 2. var result = finalQuery.ToArray(); ``` - Method calls are not supported inside `OrderBy`/`OrderByDescending` or `ThenBy`/`ThenByDescending`. - Only overloads of `Contains`, `StartsWith`, and `EndsWith` methods that have a single string argument are supported inside `Where` and `First`/`FirstOrDefault`. - The following methods are not translated to SQL after `GroupBy` and will fall back to being executed locally in memory by using LINQ to Objects: `OrderBy`/`OrderByDescending`, `ThenBy`/`ThenByDescending`, `Skip`, `Take`, `First`/`FirstOrDefault`, `Join`, and `GroupJoin`. - Nested `GroupBy` (calling `GroupBy` after another `GroupBy`) is not translated to SQL. The second `GroupBy` will fall back to being executed locally in memory by using LINQ to Objects. - `Count` with a predicate (for example, `g.Count(x => x.Quantity > 5)`) is not supported inside `Select` or `Where` (HAVING) after `GroupBy`. To filter rows before grouping, use `Where` before `GroupBy` instead. ## Modify data in a cluster by using `ScalarDbContext` The properties of the class inherited from `ScalarDbContext` can be used to modify data. ### Add a new object by using the `AddAsync` method ```c# var statement = new Statement { OrderId = "2", ItemId = 4, Quantity = 8 }; await _myDbContext.Statements.AddAsync(statement); ``` ### Update an object by using the `UpdateAsync` method ```c# var statement = _myDbContext.Statements.First(stat => stat.Id == 1); // ... statement.Quantity = 10; await _myDbContext.Statements.UpdateAsync(statement); ``` ### Remove an object by using the `RemoveAsync` method ```c# var statement = _myDbContext.Statements.First(stat => stat.Id == 1); // ... await _myDbContext.Statements.RemoveAsync(statement); ``` ## Manage transactions LINQ queries and `AddAsync`, `UpdateAsync`, and `RemoveAsync` methods can be executed without an explicitly started transaction. However, to execute multiple queries and methods as part of a single transaction, the transaction must be explicitly started and committed. `ScalarDbContext` supports both ordinary transactions and transactions with the two-phase commit interface in ScalarDB. ### Begin a new transaction ```c# await _myDbContext.BeginTransactionAsync(); ``` ### Begin a new read-only transaction If you perform only read operations (that is, LINQ queries without `AddAsync`, `UpdateAsync`, or `RemoveAsync`) in a transaction, you can begin a read-only transaction as follows: ```c# await _myDbContext.BeginReadOnlyTransactionAsync(); ``` :::note Read-only transactions are processed more efficiently than read transactions begun without the read-only mode by assuming no write operations are issued within them. Therefore, attempting to perform write operations (`AddAsync`, `UpdateAsync`, or `RemoveAsync`) in a read-only transaction will result in an error. Read-only mode is not supported for transactions with the two-phase commit interface. ::: ### Begin a new transaction with the two-phase commit interface ```c# await _myDbContext.BeginTwoPhaseCommitTransactionAsync(); ``` ### Get the ID of a currently active transaction ```c# var transactionId = _myDbContext.CurrentTransactionId; ``` ### Join an existing transaction with the two-phase commit interface ```c# await _myDbContext.JoinTwoPhaseCommitTransactionAsync(transactionId); ``` ### Resume an existing transaction ```c# await _myDbContext.ResumeTransaction(transactionId); ``` ### Resume an existing transaction with the two-phase commit interface ```c# await _myDbContext.ResumeTwoPhaseCommitTransaction(transactionId); ``` :::note The `ResumeTransaction`/`ResumeTwoPhaseCommitTransaction` methods don't have asynchronous versions because they only initialize the transaction data in the `ScalarDbContext` inheriting object without querying the cluster. Because of this, resuming a transaction by using the wrong ID is possible. ::: ### Commit a transaction (ordinary or two-phase commit) ```c# await _myDbContext.CommitTransactionAsync(); ``` ### Roll back a transaction (ordinary or two-phase commit) ```c# await _myDbContext.RollbackTransactionAsync(); ``` ### Prepare a transaction with the two-phase commit interface for the commit ```c# await _myDbContext.PrepareTransactionAsync(); ``` ### Validate a transaction with the two-phase commit interface before the commit ```c# await _myDbContext.ValidateTransactionAsync(); ``` ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/getting-started-with-scalardb-tables-as-csharp-classes.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Tables as C# Classes in the ScalarDB Cluster .NET Client SDK The ScalarDB Cluster .NET Client SDK helps you write code to access a cluster by abstracting ScalarDB tables as C# objects. After defining a class that represents a table in the cluster, you can ensure that a column name or its type won't be mixed up when querying the cluster. In addition, if a table's structure changes, you can apply the changes to the code by using the refactoring feature in your IDE. :::note Although we recommend using asynchronous methods, as in the following examples, you can use synchronous methods instead. ::: ## Install the SDK Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: ```console dotnet add package ScalarDB.Client --version '..*' ``` ## Create classes for all ScalarDB tables To work with ScalarDB tables as C# objects, you must create a class for each table that you want to use. For example: ```c# using System.ComponentModel.DataAnnotations.Schema; using ScalarDB.Client.DataAnnotations; // ... [Table("ns.statements")] public class Statement { [PartitionKey] [Column("order_id", Order = 0)] public string OrderId { get; set; } = String.Empty; [ClusteringKey] [Column("item_id", Order = 1)] public int ItemId { get; set; } [Column("count", Order = 2)] public int Count { get; set; } } ``` For details about which types should be used for properties, see [How ScalarDB Column Types Are Converted to and from .NET Types](common-reference.mdx#how-scalardb-column-types-are-converted-to-and-from-net-types). ## Execute CRUD operations After creating a class for each table, you can use the classes as objects by using the generic `GetAsync`, `ScanAsync`, `InsertAsync`, `UpdateAsync`, `DeleteAsync`, `UpsertAsync`, or `MutateAsync` method of `ITransactionCrudOperable`. To use these generic methods, add the following namespace to the `using` section: ```c# using ScalarDB.Client.Extensions; ``` ### Get one object by using the `GetAsync` method ```c# var keys = new Dictionary { { nameof(Statement.OrderId), "1" } }; var statement = await transaction.GetAsync(keys); Console.WriteLine($"ItemId: {statement.ItemId}, Count: {statement.Count}"); ``` ### Get multiple objects by using the `ScanAsync` method ```c# var startKeys = new Dictionary { { nameof(Statement.OrderId), "1" }, { nameof(Statement.ItemId), 3 } }; var endKeys = new Dictionary { { nameof(Statement.ItemId), 6} }; await foreach (var s in transaction.ScanAsync(startKeys, endKeys)) Console.WriteLine($"ItemId: {s.ItemId}, Count: {s.Count}"); ``` :::note To use LINQ methods with `IAsyncEnumerable`, you can install [System.Linq.Async](https://www.nuget.org/packages/System.Linq.Async/) package. ::: ### Insert a new object by using the `InsertAsync` method ```c# var statement = new Statement { OrderId = "2", ItemId = 4, Count = 8 }; await transaction.InsertAsync(statement); ``` ### Update an object by using the `UpdateAsync` method ```c# // ... statement.ItemId = 4; statement.Count = 8; await transaction.UpdateAsync(statement); ``` ### Delete an object by using the `DeleteAsync` method ```c# // ... await transaction.DeleteAsync(statement); ``` ### Upsert an object by using the `UpsertAsync` method ```c# var statement = new Statement { OrderId = "2", ItemId = 4, Count = 8 }; await transaction.UpsertAsync(statement); ``` ### Upsert and delete multiple objects at once by using the `MutateAsync` method ```c# var statement = new Statement { OrderId = "2", ItemId = 4, Count = 16 }; // ... await transaction.MutateAsync(objectsToUpsert: new[] { statement }, objectsToDelete: new[] { statement2 }); ``` :::note To modify objects by using the `UpdateAsync`, `DeleteAsync`, `UpsertAsync`, or `MutateAsync` method, the objects must be retrieved first by using the `GetAsync` or `ScanAsync` method. ::: ## Use the Administrative API C# objects also can be used with the Administrative API. To use generic Administrative API methods, add the following namespace to the `using` section: ```c# using ScalarDB.Client.Extensions; ``` ### Create a new namespace ```c# await admin.CreateNamespaceAsync(); ``` ### Drop an existing namespace ```c# await admin.DropNamespaceAsync(); ``` ### Check if a namespace exists ```c# var namespaceExists = await admin.IsNamespacePresentAsync(); ``` ### Create a new table ```c# await admin.CreateTableAsync(); ``` ### Drop an existing table ```c# await admin.DropTableAsync(); ``` ### Check if a table exists ```c# var tableExists = await admin.IsTablePresentAsync(); ``` ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/getting-started-with-two-phase-commit-transactions.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with Distributed Transactions with a Two-Phase Commit Interface in the ScalarDB Cluster .NET Client SDK The ScalarDB Cluster .NET Client SDK supports transactions with the two-phase commit interface in ScalarDB. The SDK includes transaction and manager abstractions for enhanced communication within a cluster. :::note Although we recommend using asynchronous methods as in the following examples, you can use synchronous methods instead. ::: ## About transactions with the two-phase commit interface By using the SDK, you can execute transactions with the two-phase commit interface that span multiple applications. For example, if you have multiple microservices, you can create a transaction manager in each of them and execute a transaction that spans those microservices. In transactions with the two-phase commit interface, there are two roles—coordinator and a participant—that collaboratively execute a single transaction. The coordinator process first begins a transaction and sends the ID of the transaction to all the participants, and the participant processes join the transaction. After executing CRUD or SQL operations, the coordinator process and the participant processes commit the transaction by using the two-phase interface. ## Install the SDK Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: ```console dotnet add package ScalarDB.Client --version '..*' ``` ## Create a settings file Create a `scalardb-options.json` file and add the following, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: ```json { "ScalarDbOptions": { "Address": "http://:", "HopLimit": 10 } } ``` For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). ## Get a transaction manager (for coordinator and participants) You need to get a transaction manager for distributed transactions with the two-phase commit interface. To get the transaction manager, you can use `TransactionFactory` as follows: ```c# // Pass the path to the settings file created in the previous step. var factory = TransactionFactory.Create("scalardb-options.json"); using var manager = factory.GetTwoPhaseCommitTransactionManager(); ``` Alternatively, you can use SQL instead of CRUD operations for transactions with the two-phase commit interface by specifying the following transaction manager: ```c# using var manager = factory.GetSqlTwoPhaseCommitTransactionManager(); ``` ## Begin a transaction (for coordinator) You can begin a transaction with the two-phase commit interface in the coordinator as follows: ```c# var transaction = await manager.BeginAsync(); ``` The ID of the started transaction can be obtained with the following code: ```c# var transactionId = transaction.Id; ``` ## Join a transaction (for participants) You can join a transaction with the two-phase commit interface in a participant as follows: ```c# var transaction = await manager.JoinAsync(transactionId); ``` ## Resume a transaction (for coordinator and participants) Usually, a transaction with the two-phase commit interface involves multiple request and response exchanges. In scenarios where you need to work with a transaction that has been begun or joined in the previous request, you can resume such transaction as follows: ```c# var transaction = manager.Resume(transactionId); ``` :::note The `Resume` method doesn't have an asynchronous version because it only creates a transaction object. Because of this, resuming a transaction by using the wrong ID is possible. ::: ## Roll back a transaction If a transaction fails to commit, you can roll back the transaction as follows: ```c# await transaction.RollbackAsync(); ``` ## Commit a transaction (for coordinator and participants) After completing CRUD or SQL operations, you must commit the transaction. However, for transactions with the two-phase commit interface, you must prepare the transaction in the coordinator and all the participants first. ```c# await transaction.PrepareAsync(); ``` Next, depending on the concurrency control protocol, you may need to validate the transaction in the coordinator and all the participants as follows: ```c# await transaction.ValidateAsync(); ``` Finally, you can commit the transaction in the coordinator and all the participants as follows: ```c# await transaction.CommitAsync(); ``` If the coordinator or any of the participants failed to prepare or validate the transaction, you will need to call `RollbackAsync` in the coordinator and all the participants. In addition, if the coordinator and all the participants failed to commit the transaction, you will need to call `RollbackAsync` in the coordinator and all the participants. However, if the coordinator or only some of the participants failed to commit the transaction, the transaction will be regarded as committed as long as the coordinator or any one of the participants has succeeded in committing the transaction. ## Execute CRUD operations The two-phase commit interface of the transaction has the same methods for CRUD operations as ordinary transactions. For details, see [Execute CRUD operations](getting-started-with-distributed-transactions.mdx#execute-crud-operations). ## Execute SQL statements The two-phase commit interface of the SQL transaction has the same methods for executing SQL queries as ordinary SQL transactions. For details, see [Execute SQL queries](getting-started-with-distributed-sql-transactions.mdx#execute-sql-queries). ================================================ FILE: docs/scalardb-cluster-dotnet-client-sdk/index.mdx ================================================ --- tags: - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB Cluster .NET Client SDK Overview The ScalarDB Cluster .NET Client SDK enables applications to connect to ScalarDB Cluster by using gRPC. To use the ScalarDB Cluster .NET Client SDK, see the following getting started guides: * [Getting Started with Distributed Transactions](getting-started-with-distributed-transactions.mdx) * [Getting Started with Distributed SQL Transactions](getting-started-with-distributed-sql-transactions.mdx) * [Getting Started with the Administrative API](getting-started-with-admin-api.mdx) * [Getting Started with ScalarDB Tables as C# Classes](getting-started-with-scalardb-tables-as-csharp-classes.mdx) * [Getting Started with ASP.NET Core and Dependency Injection](getting-started-with-aspnet-and-di.mdx) * [Getting Started with LINQ](getting-started-with-linq.mdx) * [Getting Started with Distributed Transactions with a Two-Phase Commit Interface](getting-started-with-two-phase-commit-transactions.mdx) * [Getting Started with Authentication and Authorization](getting-started-with-auth.mdx) * [Exception Handling](exception-handling.mdx) ================================================ FILE: docs/scalardb-data-loader/getting-started-export.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting started with Export This document explains how you can get started with ScalarDB Data Loader Export function. ## Features ScalarDB Data Loader allows you to export data in the following formats: - JSON - JSON Lines - CSV Each export will run a ScalarDB scan operation based on the provided CLI arguments when running Data Loader. ## Usage Data Loader export function can be started with the following minimal configuration: ```console ./scalardb-data-loader export --config scalardb.properties --namespace namespace --table tableName ``` - --config: the path to the ScalarDB connection properties file - --namespace: the namespace of the table that contains the data - --table: name of the table that contains the data By default, Data Loader will create the output file in the working directory if the `--output-file` argument is omitted as well. ### Command-line flags Here is a list of flags (options) that can be used with ScalarDB Data Loader. | Flag | Description | Usage | | ----------------- | ------------------------------------------------------------ | ------------------------------------------------------ | | --config | The path to the scalardb.properties file. If omitted the tool looks for a file named `scalardb.properties` in the current folder | `scalardb-data-loader --config scalardb.properties` | | --namespace | Namespace to export table data from. Required. | `scalardb-data-loader --namespace namespace` | | --table | Name of table to export data from. Required. | `scalardb-data-loader --table tableName` | | --key | Export data of specific Partition key. By default, it exports all data from the specified table. | `scalardb-data-loader --key columnName=value` | | --sort | Specify a column to sort on. The column needs to be a clustering key. The argument can be repeated to provide multiple sortings. This flag is only applicable to `--key`. | `scalardb-data-loader --sort columnName=desc` | | --projection | Limit the columns that are exported by providing a projection. The argument can be repeated to provide multiple projections. | `scalardb-data-loader --projection columnName` | | --start | Clustering key to mark scan start. This flag is only applicable to `--key`. | `scalardb-data-loader --start columnName=value` | | --start-exclusive | Is the scan start exclusive or not. If omitted, the default value is `false`. This flag is only applicable to `--key` | `scalardb-data-loader --start-exclusive` | | --end | Clustering key to mark scan end. This flag is only applicable to `--key`. | `scalardb-data-loader --end columnName=value` | | --end-exclusive | Is the scan start exclusive or not. If omitted, the default value is `false`. This flag is only applicable to `--key` | `scalardb-data-loader --end-exclusive` | | --limit | Limit the results of the scan. If omitted, the default value is `0` which means there is no limit. | `scalardb-data-loader --limit 1000` | | --output-file | The name and path of the output file. If omitted, the tool will save the file in the current folder with the following name format:
`export_namespace.tableName_timestamp.json` or `export_namespace.tableName_timestamp.csv`

The ouput folder needs to exists. The dataloader does not create the output folder for you. | `scalardb-data-loader --output-file ./out/output.json` | | --format | The output format. By default `json` is selected. | `scalardb-data-loader --format json` | | --metadata | When set to true the transaction metadata is included in the export. By default this is set to `false` | `scalardb-data-loader --metadata` | | --delimiter | The delimiter used in CSV files. Default value is `;` | `scalardb-data-loader --delimiter ;` | | --no-headers | Exclude header row in CSV file. Default is `false` | `scalardb-data-loader --no-headers` | | --threads | Thread count for concurrent processing. The default value is the number of available processors. | `scalardb-data-loader --threads 500` | ================================================ FILE: docs/scalardb-data-loader/getting-started-import.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting started with Import This document explains how you can get started with the ScalarDB Data Loader Import function. ## Features - Import data from JSON or JSON Lines files - Automatic data mapping based on source field name mapping - Custom Data mapping via a JSON control file - Import data from one record or line into multiple tables - Support for INSERT, UPDATE and UPSERT ## Usage The Data Loader import function can be started with the following minimal configuration: ```console ./scalardb-data-loader import --config scalardb.properties --namespace namespace --table tableName ``` The above configuration starts an import process where no control file is used and the data mapping is applied automatically. Execute the following steps to successfully import new or existing data - Prepare a source file containing data that needs to be imported. - Choose the right import mode. By default, the import is done in `upsert` mode which means that data will be inserted if new or updated if the partition key and/or clustering key is found. Other options are `insert` mode or `update` mode. - Find the correct `namespace` and `table` name to import data to. - Determine if you want to run an `all required columns` check for each data row. If enabled, data rows with missing columns will be treated as failed and not imported. - Specify the path names for the `success` and `failed` output files. By default, Data Loader creates the files in the working directory. - When dealing with JSON data, determine if you want the JSON output for the success or failed log files to be in `pretty print` or not. By default, this option is disabled for performance - Optionally specify the `threads` argument to tweak performance - Run the import from the command line to start importing your data. Make sure to run the ScalarDB Data Loader in the correct `storage` or `transaction` mode depending on your running ScalarDB instance. ### Command-line flags Here is a list of flags (options) that can be used with Data Loader: | Flag | Description | Usage | |---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------| | --mode | The mode in which ScalarDB is running. If omitted, the default value is `storage` | `scalardb-data-loader --mode transaction` | | --config | the path to the scalardb.properties file. If omitted the tool looks for a file named `scalardb.properties` in the current folder | `scalardb-data-loader --config scalardb.properties` | | --namespace | Namespace to export table data from. Required when no control file is provided. | `scalardb-data-loader --namespace namespace` | | --table | name of the table to export data from. Required when no control file is provided. | `scalardb-data-loader --table tableName` | | --import-mode | Mode to import the data into the ScalarDB table. Supported modes are `insert`, `update` and `upsert`. Optional. Default the value is set to `upsert` | `scalardb-data-loader --import-mode=upsert` | | --all-columns-required | If set, data rows cannot be imported if they are missing columns. Optional. By default, the check is not executed. | `scalardb-data-loader --all-columns-required` | | --file | Specify the path to the file that will be imported. Required | `scalardb-data-loader --file ` | | --success | The path to the file that is created to write the succeed import results to. Both succeed and failed import results will be written to a different file.
Optional. By default, the a new file will be created in the current working directory.

Note: if the file already exists, it will be overridden. | `scalardb-data-loader --success ` | | --failed | The path to the file that will be created to write the failed import results to.
Optional. By default, the a new file will be created in the current working directory.

Note: if the file already exists, it will be overridden. | `scalardb-data-loader --failed ` | | --threads | Thread count for concurrent processing. The default value is the number of available processors. | `scalardb-data-loader --threads 500` | | --format | The format of the import file. `json` and `jsonl` files are supported. Optional, default the value `json` is selected. | `scalardb-data-loader --format json` | | --ignore-null | The null values in the source file will be ignored, which means that the existing data will not be overwritten. Optional, default the value is `false`. | `scalardb-data-loader --ignore-null` | | --pretty | When set, the output to the success and failed files is done in `pretty print` mode. By default the option is not enabled. | `scalardb-data-loader --pretty` | | --control-file | The path to the JSON control file specifying the rules for the custom data mapping and/or multi-table import. | `scalardb-data-loader --control-file control.json` | | --control-file-validation-level | The validation level for the control file. `MAPPED`, `KEYS` or` FULL`.

Optional and by default the level is set to `MAPPED` | `scalardb-data-loader --control-file-validation-level FULL` | | --log-put-value | Whether the value that was used in the ScalarDB `PUT` operation is included in the log files or not.
Optional and disabled by default. | `scalardb-data-loader --log-put-value` | | --error-file-required | To export an optional error file of type JSON when the import file contains CSV data. By default, this option is disabled. | `scalardb-data-loader --error-file-required` | | --error | To specify an optional error file when the import file contains CSV data. | `scalardb-data-loader --error ` | | --delimiter | To specify a custom delimiter if the import file contains CSV data. | `scalardb-data-loader --delimiter ` | | --header | To specify the header row data if the import file contains CSV data and does not have a header row. | `scalardb-data-loader --header ` | ## Import mode Data Loader supports the following import modes: | Mode | Description | | ------ | ------------------------------------------------------------ | | INSERT | Each source record is treated as new data. If the data already exists in the ScalarDB table, based on the partition and clustering key, the import for this source data will fail. | | UPDATE | Each source record is treated as an update for existing data in the ScalarDB table. If the data does not exist in the table, based on the partition key and clustering key, the import for this source data will fail. | | UPSERT | If the target ScalarDB table already contains the data, the import will be done via an UPDATE. If the target data is missing, it will be treated as an INSERT. | *Note*: In the case of `INSERT`, it is required to have matching fields in the source files for each target column via automatic or custom mapping via the control file. This also applies to an `UPSERT` that turns into an `INSERT`. ## Data mapping ### Automatic mapping When no control file is provided, Data Loader will automatically map the fields in the source JSON data to the available columns in the ScalarDB table. If the name does not match, and if all columns are required, it will be treated as a validation error. In this case, the import for this record will fail and the result will be added to the failed output log. ### Custom mapping When the source fields do not match the target column name, it is necessary to use a control file. In this control, file you can specify the custom mapping rules for the field names. e.g. the following control file to map the field `custom_id` in the source file to `id` in the target table. ```json { "tables": [{ "namespace": "sample", "table_name": "table1", "mappings": [{ "source_field": "custom_id", "target_column": "id" }] } ] } ``` ## Control file To allow for custom data mapping or multi-table importing, Data Loader supports configuration via a JSON control file. This file needs to be passed in via the `--control-file` argument when starting Data Loader. ### Control file validation levels To enforce validation on the control file, Data Loader allows you to specify the validation level. Based on the set level, Data Loader will run a pre-check and validate the control file based on the level rules. The following levels are supported: | Level | Description | | ------ | ------------------------------------------------------------ | | FULL | This validation makes sure that the control file has mappings for each column in the target ScalarDB table. | | KEYS | This validation makes sure that mappings are available for each ScalarDB table partition and, if available, clustering keys columns in the control file mappings. | | MAPPED | The validation makes sure that the provided source fields and target columns exist for only the mappings that are provided in the control file.
No other fields are checked. | The validation level is optional and can be set via the `--control-file-validation-level` argument when starting Data Loader. *Note*: This validation is run as a pre-check and does not mean the import process will automatically succeed. e.g. If the level is set to mapped and the control file does not contain mappings for each column for an INSERT, the import process will still fail as all columns are required to be mapped for an INSERT. ## Multi-table import Data Loader supports multi-table target importing. One single row in a JSON or JSON Lines file can be imported into multiple tables by specifying table mapping rules in the control file. Currently, multi-table import is not supported without a control file. When using multi-table import in ScalarDB transaction mode, a transaction is created for each table import. e.g. If the source record is mapped to 3 tables in the control file, 3 separate transactions are created. e.g. The import the following source record into `table1` and `table2` we execute the following steps: | Table1 | Table2 | | ------ | ------ | | Id | Id | | price | amount | **Source record** ```json [{ "custom_id": 1, "custom_price": 1000, "custom_amount": 100 }] ``` **Control file** ```json { "tables": [{ "namespace": "sample", "table_name": "table1", "mappings": [{ "source_field": "custom_id", "target_column": "id" }, { "source_field": "custom_price", "target_column": "price" }] }, { "namespace": "sample", "table_name": "table2", "mappings": [{ "source_field": "custom_id", "target_column": "id" }, { "source_field": "custom_amount", "target_column": "amount" }] } ] } ``` ## Output logs When starting an import task, Data Loader logs the import results in two files. One file contains the import data that is successfully imported and one file contains the data that cannot be imported. The failed data will contain an added field that explains why the data could not be imported. This field is called `data_loader_import_status`. The file containing the failed imports can be edited to correct the problems and used as the source file for a new import task as is. It is not required to first remove the `data_loader_import_status` field containing the error. This field will be ignored during the import process and the original value will not be included in the new version of the success or failed output file. The file with the successfully imported data also contains the `data_loader_import_status` field. In this file, each imported data row has a status message for the data. Whether a new row was created or existing data was updated. ### Log format | Field | Description | | -------------- | ------------------------------------------------------------ | | action | The result of the import process for the data record. UPDATE, INSERT or FAILED_DURING_VALIDATION | | namespace | The name of the namespace of the table that the data is imported in | | table_name | The name of the table that the data is imported in | | is_data_mapped | Whether custom data mapping was applied or not based on an available control file. | | tx_id | The transaction ID. Only available if Data Loader is run in `transaction` mode. | | value | The final value, after optional data mapping, that Data Loader uses in the `PUT` operation. | | row_number | The line number or record number of the source data. | | Errors | A list of validation or other errors for things that went wrong during the import process. | Example of a JSON formatted success log file: ```json [{ "column_1": 1, "column_2": 2, "column_n": 3, "data_loader_import_status": { "results": [{ "action": "UPDATE", "namespace": "namespace1", "table_name": "table1", "is_data_mapped": true, "tx_id": "value", "value": "value", "row_number": "value" }] } }] ``` Example of a JSON formatted failed log file: ```json [{ "column_1": 1, "column_2": 2, "column_n": 3, "data_loader_import_status": { "results": [{ "action": "FAILED_DURING_VALIDATION", "namespace": "namespace1", "table_name": "table1", "is_data_mapped": false, "value": "value", "row_number": "value", "errors": [ "missing columns found during validation" ] }] } }] ``` ## Data duplicates Data Loader does not handle duplicates by itself. In ScalarDB transaction mode, trying to update the same target data in fast succession will cause `No Mutation` errors and these are not handled by Data Loader. These failed data rows will be added to the failed import result output file and can be re-tried for import later. However, it is recommended to make sure the import file does not contain updates or inserts on the same partition keys and/or clustering keys as the correct state cannot be guaranteed by the data loader. ## Storage vs transaction mode ScalarDB supports both storage and transaction mode and this support is included in Data Loader import process. When the loader is started in storage mode, each import is executed in a non-transactional way. Starting the loader in transaction mode will use transactions to import the data. Currently, each row is imported via a separate transaction. When importing a single record into multiple tables, a separate transaction is created for each table import. ================================================ FILE: docs/scalardb-graphql/how-to-run-two-phase-commit-transaction.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # How to run two-phase commit transaction ScalarDB GraphQL supports two-phase commit style transactions called [Two-phase Commit Transactions](../two-phase-commit-transactions.mdx). With Two-phase Commit Transactions, you can execute a transaction that spans multiple processes/applications (e.g., Microservices). We name the application that starts a transaction "coordinator" while the applications that join the transaction are named "participants". Every two-phase commit operation requires annotating the mutation or query operation with a `@twoPhaseCommit` directive. Below is a description of such operations. ## Start a transaction To start a transaction, add the `@twoPhaseCommit` directive without setting parameters. ```graphql query some_query @twoPhaseCommit { # some query } ``` The transaction ID of the started transaction will be returned in the extensions object that is part of the result. ```json { "data": { ... }, "extensions": { "transaction": { "id": "the_transaction_id" } } } ``` ## Join a transaction (for participants) In a participant application, to join the transaction started by a coordinator application, set the transaction ID with the `id` parameter and set the `join` parameter to true. ```graphql query some_query_from_participant @twoPhaseCommit(id:"the_transaction_id", join:true) { # some query } ``` ## Resume a transaction To continue executing operations in the started or joined transaction, set the transaction ID value in the `id` parameter of `@twoPhaseCommit` directive. ```graphql mutation some_mutation @twoPhaseCommit(id:"the_transaction_id") { # some mutation } ``` ## Prepare, validate and commit a transaction After finishing the query and mutation operations, you need to commit the transaction. Like a well-known two-phase commit protocol, there are two phases: prepare and commit. You first need to prepare the transaction in all the coordinator/participant applications, and then you need to commit the transaction in all the coordinator/participant applications. If the Consensus Commit transaction manager is configured with the `EXTRA_READ` serializable strategy in `SERIALIZABLE` isolation level, an extra "validate" phase is required between prepare and commit phases. Similarly to prepare and commit, validate need to be executed in all the coordinator/participants applications. Prepare, validate and commit can be executed in parallel with all the coordinator/participants applications. ### Prepare a transaction Two options are possible to prepare a two-phase commit transaction. #### Via the directive parameter By using the `prepare` parameter of the directive, the transaction will be prepared after the execution of the operation fields and only if they do not raise an error. ```graphql mutation some_mutation_then_prepare_tx @twoPhaseCommit(id:"the_transaction_id", prepare:true) { mutation1 : ... mutation2 : ... # the transaction will be prepared after the execution of the mutation1 and mutation2 fields } ``` #### Via the mutation field Add a `prepare` field in a mutation operation. This field will trigger the transaction preparation. ```graphql mutation prepare_tx @twoPhaseCommit(id:"the_transaction_id") { prepare } ``` ### Validate a transaction Add a `validate` field in a mutation operation. This field will trigger the transaction validation. ```graphql mutation validate_tx @twoPhaseCommit(id:"the_transaction_id") { validate } ``` ### Commit a transaction Add a `commit` field in a mutation operation. This field will trigger the transaction commit. ```graphql mutation commit_tx @twoPhaseCommit(id:"the_transaction_id") { commit } ``` ### Abort/Rollback a transaction When you need to abort/rollback a transaction explicitly, you can use the `abort` or `rollback` mutation fields interchangeably (both have the same effect and usage). Note that you cannot mix it with any other operations, so you must specify it alone. ```graphql mutation AbortTx @twoPhaseCommit(id: "the_transaction_id") { abort } ``` or ```graphql mutation RollbackTx @twoPhaseCommit(id: "the_transaction_id") { rollback } ``` ## Error handling If an exception is thrown by a `@twoPhaseCommit` operation, ScalarDB GraphQL triggers a rollback procedure that recovers the transaction. For more details about the exception handling in two-phase commit transaction, please refer to the [exception handling guide for ScalarDB two-phase commit transaction](../two-phase-commit-transactions.mdx#handle-exceptions). ================================================ FILE: docs/scalardb-graphql/index.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB GraphQL Overview ScalarDB GraphQL is an interface layer that allows client applications to communicate with ScalarDB Cluster by using GraphQL. It enables you to take advantage the benefits of GraphQL, such as flexible data retrieval and type safety, while benefiting from the transaction management and data access features in ScalarDB. By using ScalarDB GraphQL, you can create GraphQL schemas automatically based on the ScalarDB schemas, perform CRUD operations, and execute complex transactions across multiple databases. The interface simplifies backend development by providing a unified querying mechanism, making it particularly useful for modern applications expecting advanced and responsive data interactions. ## Getting started with GraphQL in ScalarDB Cluster ScalarDB GraphQL is designed to be intuitive and user-friendly, enabling developers to easily create GraphQL schemas automatically based on the ScalarDB schemas and interact with the underlying databases. For details on how to set up ScalarDB Cluster with GraphQL support, see [Getting Started with ScalarDB Cluster GraphQL](../scalardb-cluster/getting-started-with-scalardb-cluster-graphql.mdx). ## Transactions with a two-phase commit ScalarDB GraphQL supports executing transactions with a two-phase commit interface. By using the two-phase commit interface, you can execute a transaction that spans multiple processes/applications (for example, microservices applications). For details on how to execute transactions by using the two-phase commit interface in ScalarDB GraphQL, see [How to Run Two-Phase Commit Transactions](./how-to-run-two-phase-commit-transaction.mdx). ================================================ FILE: docs/scalardb-graphql/scalardb-graphql-status-codes.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB GraphQL Error Codes This page provides a list of error codes in ScalarDB GraphQL. ## Error code classes and descriptions | Class | Description | |:-------------------|:-----------------------------------| | `DB-GRAPHQL-1xxxx` | Errors for the user error category | ## `DB-GRAPHQL-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-GRAPHQL-10000` **Message** ```markdown A long value was expected ``` ### `DB-GRAPHQL-10001` **Message** ```markdown The value is out of range for BigIntValue ``` ### `DB-GRAPHQL-10002` **Message** ```markdown A long, integer, or string value was expected ``` ### `DB-GRAPHQL-10003` **Message** ```markdown The AST type `IntValue` was expected ``` ### `DB-GRAPHQL-10004` **Message** ```markdown A float value was expected ``` ### `DB-GRAPHQL-10005` **Message** ```markdown An integer or float value was expected ``` ### `DB-GRAPHQL-10006` **Message** ```markdown The AST type `IntValue` or `FloatValue` was expected ``` ### `DB-GRAPHQL-10007` **Message** ```markdown The type is not supported. Type: %s ``` ### `DB-GRAPHQL-10008` **Message** ```markdown The field `%s` requires a `@transaction` or `@twoPhaseCommit` directive with proper arguments ``` ### `DB-GRAPHQL-10009` **Message** ```markdown The field `%s` cannot be used together with other fields ``` ### `DB-GRAPHQL-10010` **Message** ```markdown The `@twoPhaseCommit` directive with the `id` argument is required to `%s` the transaction ``` ### `DB-GRAPHQL-10011` **Message** ```markdown `%s` and `prepare` cannot be run simultaneously ``` ### `DB-GRAPHQL-10012` **Message** ```markdown `%s` and `join` cannot be run simultaneously ``` ### `DB-GRAPHQL-10013` **Message** ```markdown The `@transaction` directive with the `id` argument is required to `%s` the transaction ``` ### `DB-GRAPHQL-10014` **Message** ```markdown `%s` and `commit` cannot be run simultaneously ``` ### `DB-GRAPHQL-10015` **Message** ```markdown An object cannot be annotated with both `@transaction` and `@twoPhaseCommit` directives ``` ### `DB-GRAPHQL-10016` **Message** ```markdown The `join` argument of the `@twoPhaseCommit` directive requires a transaction `id` argument ``` ### `DB-GRAPHQL-10017` **Message** ```markdown `%s` requires the mutation object to be annotated with a `@twoPhaseCommit` directive ``` ### `DB-GRAPHQL-10018` **Message** ```markdown The `%s` clustering key must have only one of the following: %s ``` ### `DB-GRAPHQL-10019` **Message** ```markdown A string variable is expected but got %s ``` ### `DB-GRAPHQL-10020` **Message** ```markdown Unexpected value of id: %s ``` ### `DB-GRAPHQL-10021` **Message** ```markdown A Boolean variable is expected but got %s ``` ### `DB-GRAPHQL-10022` **Message** ```markdown Unexpected value of %s: %s ``` ### `DB-GRAPHQL-10023` **Message** ```markdown Invalid column. Column: %s; Type: %s ``` ### `DB-GRAPHQL-10024` **Message** ```markdown Unexpected value of type: %s ``` ### `DB-GRAPHQL-10025` **Message** ```markdown Only one of the following can be specified: %s ``` ### `DB-GRAPHQL-10026` **Message** ```markdown Unexpected mutation field: %s ``` ### `DB-GRAPHQL-10027` **Message** ```markdown Invalid type: %s ``` ================================================ FILE: docs/scalardb-mcp-server/getting-started-with-scalardb-mcp-server.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Getting Started with ScalarDB MCP Server import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; ScalarDB MCP Server is a [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) implementation that enables LLMs to access and manage your data through ScalarDB. By leveraging LLMs, you can use natural language to search and update across multiple, potentially siloed, databases. ScalarDB MCP Server works with both single and multiple storage configurations. Unlike traditional approaches that require separate MCP servers for each database, ScalarDB MCP Server takes advantage of the [multi-storage capabilities of ScalarDB](../multi-storage-transactions.mdx) to provide unified access to heterogeneous databases (PostgreSQL, MySQL, Cosmos DB, DynamoDB, etc.) through a single MCP server. By simply submitting queries in natural language, the server automatically executes the appropriate operations across your databases, improving and accelerating decision-making processes. ## Architecture and key features The following diagram shows how ScalarDB MCP Server differs from traditional approaches. Instead of requiring separate MCP servers for each database, you connect once to ScalarDB MCP Server to access all your databases through ScalarDB. ![ScalarDB MCP server architecture](../images/scalardb-mcp-server-architecture.png) At its core, ScalarDB MCP Server provides the following capabilities. ### ScalarDB connectivity The MCP server internally uses the ScalarDB Core library or connects to your ScalarDB Cluster through the client library, depending on settings. This means users don't need to know how to use the libraries to interact with ScalarDB. ### Transactional operations ScalarDB MCP Server supports ACID-compliant transactions, allowing an LLM to execute multiple operations safely. When the LLM determines that operations should be grouped together, the MCP server ensures the operations either all succeed or all fail together, maintaining data integrity across your databases. ### Operational mode ScalarDB MCP Server supports two operational modes that match your ScalarDB component: SQL mode and CRUD mode. #### SQL mode SQL mode with ScalarDB Cluster provides a SQL interface for database operations. When you make natural language requests, the LLM automatically generates and executes SQL commands through the supported SQL operations in ScalarDB and handles transactions by using standard SQL syntax (`BEGIN`, `COMMIT`, `ROLLBACK`). This mode could be more efficient because the LLM only needs to use one tool to perform all operations. SQL mode is available only with ScalarDB Cluster. #### CRUD mode CRUD mode is used when you want programmatic control over your operations. Since ScalarDB Core doesn't include the SQL interface, this mode uses the native SDK operations in ScalarDB instead. The LLM converts your natural language requests into appropriate SDK calls by using individual tools for schema management, CRUD operations, and explicit transaction control. This mode could be less efficient because the LLM has to work with multiple tools to complete operations. ### Deployment limitations :::note The ScalarDB MCP Server currently runs in STDIO mode for local deployment only. Remote server deployment via Server-Sent Events (SSE) is not yet supported but is planned for future releases. **What this means:** - ✅ The MCP server runs locally alongside your AI client (Claude Desktop, Visual Studio Code, etc.). - ✅ Perfect for development, testing, and single-user scenarios. - ❌ Cannot deploy the MCP server on remote servers for multi-user access. - ❌ No web-based or cloud deployment options yet. ::: ## Example workflow Here's how you interact with ScalarDB MCP Server through natural language: **Querying data (SQL mode):** ```markdown You: "Show me all users from the customer table" 🤖 LLM automatically uses: scalardb_execute_sql tool SQL generated: SELECT * FROM customer Result: Customer data displayed with columns and values ``` **Querying data (CRUD mode):** ```markdown You: "Show me all users from the customer table" 🤖 LLM automatically uses: scalardb_scan tool Result: Customer data displayed with columns and values ``` **Creating database structures (CRUD mode):** ```markdown You: "Create a new table called products with columns id, name, and price" 🤖 LLM automatically uses: scalardb_create_table tool Result: ✅ Table 'products' created successfully ``` **Cross-database operations (multi-storage):** ```markdown You: "Get user profile and order history for user ID 123" 🤖 LLM automatically uses: scalardb_get tool (queries across multiple databases) Result: Combined user profile (from PostgreSQL) and order history (from DynamoDB) ``` The LLM automatically selects the appropriate tools based on your request—you don't need to know which specific tools exist or how to use them. ## Tutorial The following configuration samples use the same Cassandra and MySQL multi-storage configuration as the [Multi-Storage Transaction Sample](../scalardb-samples/multi-storage-transaction-sample/README.mdx). You can follow that hands-on tutorial to set up your databases, then use the same setup to test the MCP server with this tutorial. Configurations may vary depending on your specific MCP client and database environment. Refer to your MCP client's documentation for detailed setup instructions on how to add a connection to an MCP server. ### Setup Follow these steps to set up ScalarDB MCP Server. #### Prerequisites Ensure you have the following: - (For JAR distribution) One of the following Java Development Kits (JDKs): - (For Docker distribution) Docker 20.10 or later - (For this tutorial's examples) Cassandra and MySQL databases must be running - (For SQL mode) ScalarDB Cluster must also be running - MCP-compatible client (Claude Desktop, Visual Studio Code with Cline, etc.) #### Step 1: Choose your MCP client type Select the configuration method that matches your MCP client. If you're using the Claude Code CLI or similar tools that support command-line MCP server management, choose **CLI tools**. If you're using Claude Desktop or other clients that require manual JSON configuration files, choose **Manual configuration files**. For MCP clients with command-line server management (for example, the Claude Code CLI). #### Step 2: Choose your distribution method Docker images are available from the [ScalarDB MCP Server container registry](https://github.com/scalar-labs/scalardb-mcp-server/pkgs/container/scalardb-mcp-server). You can pull the Docker image from the container registry by running the following command. Be sure to replace `` with the version that you want to use. ```bash docker pull ghcr.io/scalar-labs/scalardb-mcp-server: ``` #### Step 3: Choose your ScalarDB deployment type Run the following command to add the MCP server: ```bash claude mcp add scalardb -- docker run --rm -i \ --name scalardb-mcp-server \ ghcr.io/scalar-labs/scalardb-mcp-server: \ --scalar.mcp.db.server.tool.mode=SQL \ --scalar.db.transaction_manager=cluster \ --scalar.db.contact_points=indirect:host.docker.internal \ --scalar.db.contact_port=60053 ``` This configuration uses SQL mode, which is recommended for ScalarDB Cluster as it provides a more efficient single-tool approach. :::note **ScalarDB Cluster configuration** The configuration above shows how the MCP server connects to ScalarDB Cluster as a client. The ScalarDB Cluster itself must be separately configured. For example, your ScalarDB Cluster configuration with multi-storage support would include: ```properties scalar.db.transaction_manager=consensus-commit scalar.db.storage=multi-storage scalar.db.multi_storage.storages=cassandra,mysql scalar.db.multi_storage.storages.cassandra.storage=cassandra scalar.db.multi_storage.storages.cassandra.contact_points=localhost scalar.db.multi_storage.storages.cassandra.username=cassandra scalar.db.multi_storage.storages.cassandra.password=cassandra scalar.db.multi_storage.storages.mysql.storage=jdbc scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://localhost:3306/ scalar.db.multi_storage.storages.mysql.username=root scalar.db.multi_storage.storages.mysql.password=mysql scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra scalar.db.multi_storage.default_storage=cassandra scalar.db.sql.enabled=true # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` For complete ScalarDB Cluster deployment and configuration instructions, see [ScalarDB Cluster Configurations](../scalardb-cluster/scalardb-cluster-configurations.mdx). For a hands-on setup guide with multi-storage, see the [Multi-Storage Transaction Sample](../scalardb-samples/multi-storage-transaction-sample/README.mdx). ::: :::important Docker flags: - `--rm`: Required to automatically remove the container after the MCP client disconnects - `--name`: Required to prevent dangling container instances from accumulating ::: Run the following command to add the MCP server: ```bash claude mcp add scalardb -- docker run --rm -i \ --name scalardb-mcp-server \ ghcr.io/scalar-labs/scalardb-mcp-server: \ --scalar.mcp.db.server.tool.mode=CRUD \ --scalar.db.transaction_manager=consensus-commit \ --scalar.db.storage=multi-storage \ --scalar.db.multi_storage.storages=cassandra,mysql \ --scalar.db.multi_storage.storages.cassandra.storage=cassandra \ --scalar.db.multi_storage.storages.cassandra.contact_points=host.docker.internal \ --scalar.db.multi_storage.storages.cassandra.username=cassandra \ --scalar.db.multi_storage.storages.cassandra.password=cassandra \ --scalar.db.multi_storage.storages.mysql.storage=jdbc \ --scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://host.docker.internal:3306/ \ --scalar.db.multi_storage.storages.mysql.username=root \ --scalar.db.multi_storage.storages.mysql.password=mysql \ --scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra \ --scalar.db.multi_storage.default_storage=cassandra ``` This configuration uses CRUD mode, which is required for ScalarDB Core as it doesn't include the SQL interface. :::note The example above demonstrates a multi-storage configuration. For other ScalarDB Core configuration options, see [ScalarDB Configurations](../configurations.mdx). ::: :::important Docker flags: - `--rm`: Required to automatically remove the container after the MCP client disconnects - `--name`: Required to prevent dangling container instances from accumulating ::: Download the latest JAR file from the [ScalarDB MCP Server releases page](https://github.com/scalar-labs/scalardb-mcp-server/releases/latest). #### Step 3: Choose your ScalarDB deployment type Run the following command to add the MCP server: ```bash claude mcp add scalardb \ -- java -jar /path/to/scalardb-mcp-server-.jar \ --scalar.mcp.db.server.tool.mode=SQL \ --scalar.db.transaction_manager=cluster \ --scalar.db.contact_points=indirect:localhost \ --scalar.db.contact_port=60053 ``` This configuration uses SQL mode, which is recommended for ScalarDB Cluster as it provides a more efficient single-tool approach. :::note **ScalarDB Cluster configuration** The configuration above shows how the MCP server connects to ScalarDB Cluster as a client. The ScalarDB Cluster itself must be separately configured. For example, your ScalarDB Cluster configuration with multi-storage support would include: ```properties scalar.db.transaction_manager=consensus-commit scalar.db.storage=multi-storage scalar.db.multi_storage.storages=cassandra,mysql scalar.db.multi_storage.storages.cassandra.storage=cassandra scalar.db.multi_storage.storages.cassandra.contact_points=localhost scalar.db.multi_storage.storages.cassandra.username=cassandra scalar.db.multi_storage.storages.cassandra.password=cassandra scalar.db.multi_storage.storages.mysql.storage=jdbc scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://localhost:3306/ scalar.db.multi_storage.storages.mysql.username=root scalar.db.multi_storage.storages.mysql.password=mysql scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra scalar.db.multi_storage.default_storage=cassandra scalar.db.sql.enabled=true # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` For complete ScalarDB Cluster deployment and configuration instructions, see [ScalarDB Cluster Configurations](../scalardb-cluster/scalardb-cluster-configurations.mdx). For a hands-on setup guide with multi-storage, see the [Multi-Storage Transaction Sample](../scalardb-samples/multi-storage-transaction-sample/README.mdx). ::: Run the following command to add the MCP server: ```bash claude mcp add scalardb \ -- java -jar /path/to/scalardb-mcp-server-.jar \ --scalar.mcp.db.server.tool.mode=CRUD \ --scalar.db.transaction_manager=consensus-commit \ --scalar.db.storage=multi-storage \ --scalar.db.multi_storage.storages=cassandra,mysql \ --scalar.db.multi_storage.storages.cassandra.storage=cassandra \ --scalar.db.multi_storage.storages.cassandra.contact_points=localhost \ --scalar.db.multi_storage.storages.cassandra.username=cassandra \ --scalar.db.multi_storage.storages.cassandra.password=cassandra \ --scalar.db.multi_storage.storages.mysql.storage=jdbc \ --scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://localhost:3306/ \ --scalar.db.multi_storage.storages.mysql.username=root \ --scalar.db.multi_storage.storages.mysql.password=mysql \ --scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra \ --scalar.db.multi_storage.default_storage=cassandra ``` This configuration uses CRUD mode, which is required for ScalarDB Core as it doesn't include the SQL interface. :::note The example above demonstrates a multi-storage configuration. For other ScalarDB Core configuration options, see [ScalarDB Configurations](../configurations.mdx). ::: The examples below use the configuration format for Claude Desktop, but most MCP clients use the same JSON structure. Refer to your specific client's documentation for the exact configuration file location. #### Step 2: Choose your distribution method Docker images are available from the [ScalarDB MCP Server container registry](https://github.com/scalar-labs/scalardb-mcp-server/pkgs/container/scalardb-mcp-server). You can pull the Docker image from the container registry by running the following command. Be sure to replace `` with the version that you want to use. ```bash docker pull ghcr.io/scalar-labs/scalardb-mcp-server: ``` #### Step 3: Choose your ScalarDB deployment type Add the following to your MCP client configuration file: ```json { "mcpServers": { "scalardb": { "command": "docker", "args": [ "run", "-i", "--rm", "--name", "scalardb-mcp-server", "ghcr.io/scalar-labs/scalardb-mcp-server:", "--scalar.db.transaction_manager=cluster", "--scalar.db.contact_points=indirect:host.docker.internal", "--scalar.db.contact_port=60053", "--scalar.mcp.db.server.tool.mode=SQL" ] } } } ``` This configuration uses SQL mode, which is recommended for ScalarDB Cluster as it provides a more efficient single-tool approach. :::note **ScalarDB Cluster configuration** The configuration above shows how the MCP server connects to ScalarDB Cluster as a client. The ScalarDB Cluster itself must be separately configured. For example, your ScalarDB Cluster configuration with multi-storage support would include: ```properties scalar.db.transaction_manager=consensus-commit scalar.db.storage=multi-storage scalar.db.multi_storage.storages=cassandra,mysql scalar.db.multi_storage.storages.cassandra.storage=cassandra scalar.db.multi_storage.storages.cassandra.contact_points=localhost scalar.db.multi_storage.storages.cassandra.username=cassandra scalar.db.multi_storage.storages.cassandra.password=cassandra scalar.db.multi_storage.storages.mysql.storage=jdbc scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://localhost:3306/ scalar.db.multi_storage.storages.mysql.username=root scalar.db.multi_storage.storages.mysql.password=mysql scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra scalar.db.multi_storage.default_storage=cassandra scalar.db.sql.enabled=true # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` For complete ScalarDB Cluster deployment and configuration instructions, see [ScalarDB Cluster Configurations](../scalardb-cluster/scalardb-cluster-configurations.mdx). For a hands-on setup guide with multi-storage, see the [Multi-Storage Transaction Sample](../scalardb-samples/multi-storage-transaction-sample/README.mdx). ::: :::important **Docker flags** - `--rm`: Required to automatically remove the container after the MCP client disconnects - `--name`: Required to prevent dangling container instances from accumulating ::: Add the following to your MCP client configuration file: ```json { "mcpServers": { "scalardb": { "command": "docker", "args": [ "run", "-i", "--rm", "--name", "scalardb-mcp-server", "ghcr.io/scalar-labs/scalardb-mcp-server:", "--scalar.mcp.db.server.tool.mode=CRUD", "--scalar.db.transaction_manager=consensus-commit", "--scalar.db.storage=multi-storage", "--scalar.db.multi_storage.storages=cassandra,mysql", "--scalar.db.multi_storage.storages.cassandra.storage=cassandra", "--scalar.db.multi_storage.storages.cassandra.contact_points=host.docker.internal", "--scalar.db.multi_storage.storages.cassandra.username=cassandra", "--scalar.db.multi_storage.storages.cassandra.password=cassandra", "--scalar.db.multi_storage.storages.mysql.storage=jdbc", "--scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://host.docker.internal:3306/", "--scalar.db.multi_storage.storages.mysql.username=root", "--scalar.db.multi_storage.storages.mysql.password=mysql", "--scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra", "--scalar.db.multi_storage.default_storage=cassandra" ] } } } ``` This configuration uses CRUD mode, which is required for ScalarDB Core as it doesn't include the SQL interface. :::note The example above demonstrates a multi-storage configuration. For other ScalarDB Core configuration options, see [ScalarDB Configurations](../configurations.mdx). ::: :::important **Docker flags** - `--rm`: Required to automatically remove the container after the MCP client disconnects - `--name`: Required to prevent dangling container instances from accumulating ::: Download the latest JAR file from the [ScalarDB MCP Server releases page](https://github.com/scalar-labs/scalardb-mcp-server/releases/latest). #### Step 3: Choose your ScalarDB deployment type Add the following to your MCP client configuration file: ```json { "mcpServers": { "scalardb": { "command": "java", "args": [ "-jar", "/path/to/scalardb-mcp-server-.jar", "--scalar.mcp.db.server.tool.mode=SQL", "--scalar.db.transaction_manager=cluster", "--scalar.db.contact_points=indirect:localhost", "--scalar.db.contact_port=60053" ], } } } ``` This configuration uses SQL mode, which is recommended for ScalarDB Cluster as it provides a more efficient single-tool approach. :::note **ScalarDB Cluster configuration** The configuration above shows how the MCP server connects to ScalarDB Cluster as a client. The ScalarDB Cluster itself must be separately configured. For example, your ScalarDB Cluster configuration with multi-storage support would include: ```properties scalar.db.transaction_manager=consensus-commit scalar.db.storage=multi-storage scalar.db.multi_storage.storages=cassandra,mysql scalar.db.multi_storage.storages.cassandra.storage=cassandra scalar.db.multi_storage.storages.cassandra.contact_points=localhost scalar.db.multi_storage.storages.cassandra.username=cassandra scalar.db.multi_storage.storages.cassandra.password=cassandra scalar.db.multi_storage.storages.mysql.storage=jdbc scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://localhost:3306/ scalar.db.multi_storage.storages.mysql.username=root scalar.db.multi_storage.storages.mysql.password=mysql scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra scalar.db.multi_storage.default_storage=cassandra scalar.db.sql.enabled=true # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` For complete ScalarDB Cluster deployment and configuration instructions, see [ScalarDB Cluster Configurations](../scalardb-cluster/scalardb-cluster-configurations.mdx). For a hands-on setup guide with multi-storage, see the [Multi-Storage Transaction Sample](../scalardb-samples/multi-storage-transaction-sample/README.mdx). ::: Add the following to your MCP client configuration file: ```json { "mcpServers": { "scalardb": { "command": "java", "args": [ "-jar", "/path/to/scalardb-mcp-server-.jar", "--scalar.mcp.db.server.tool.mode=CRUD", "--scalar.db.transaction_manager=consensus-commit", "--scalar.db.storage=multi-storage", "--scalar.db.multi_storage.storages=cassandra,mysql", "--scalar.db.multi_storage.storages.cassandra.storage=cassandra", "--scalar.db.multi_storage.storages.cassandra.contact_points=localhost", "--scalar.db.multi_storage.storages.cassandra.username=cassandra", "--scalar.db.multi_storage.storages.cassandra.password=cassandra", "--scalar.db.multi_storage.storages.mysql.storage=jdbc", "--scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://localhost:3306/", "--scalar.db.multi_storage.storages.mysql.username=root", "--scalar.db.multi_storage.storages.mysql.password=mysql", "--scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra", "--scalar.db.multi_storage.default_storage=cassandra" ] } } } ``` This configuration uses CRUD mode, which is required for ScalarDB Core as it doesn't include the SQL interface. :::note The example above demonstrates a multi-storage configuration. For other ScalarDB Core configuration options, see [ScalarDB Configurations](../configurations.mdx). ::: ### ScalarDB MCP Server configuration Configure the MCP server by providing command-line arguments with lowercase dot notations when starting the server. #### ScalarDB MCP Server–specific configuration These properties control how the ScalarDB MCP server operates: ##### `scalar.mcp.db.server.tool.mode` - **Property:** `scalar.mcp.db.server.tool.mode` - **Description:** Tool availability mode. - **Default value:** `CRUD` - **Options:** `SQL`, `CRUD` ##### `scalar.mcp.db.server.connection.health_check_interval_seconds` - **Property:** `scalar.mcp.db.server.connection.health_check_interval_seconds` - **Description:** Health check interval in seconds. - **Default value:** `30` - **Options:** Any positive integer value ##### `scalar.mcp.db.server.logging.file.name` - **Property:** `scalar.mcp.db.server.logging.file.name` - **Description:** Enable file logging by specifying a log file path. - **Default value:** No file logging - **Example:** `scalardb-mcp-server.log` ##### `scalar.mcp.db.server.logging.level` - **Property:** `scalar.mcp.db.server.logging.level` - **Description:** Set the logger level for the MCP server. - **Default value:** `INFO` - **Options:** `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` #### ScalarDB connection configuration The MCP server uses ScalarDB client configuration properties to connect to your ScalarDB deployment (either ScalarDB Cluster or ScalarDB Core). These properties are passed as command-line arguments with lowercase dot notations. See the setup examples above for complete configuration examples for both connection types. ## Available tools The ScalarDB MCP Server provides comprehensive database operations through specialized MCP tools. The LLM automatically selects and uses the appropriate tools based on your natural language requests. For complete tool documentation including all available operations, parameters, and examples, see the [ScalarDB MCP Server Tools Reference](./tools-reference.mdx). ## ScalarDB version compatibility | ScalarDB MCP Server | ScalarDB Core | ScalarDB Cluster | Java Version | Notes | |-------------------|-------------------|------------------|--------------|--------| | 0.9.x | 3.16+ | 3.16+ | 17+ | Initial release | ================================================ FILE: docs/scalardb-mcp-server/tools-reference.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB MCP Server Tools Reference The ScalarDB MCP Server provides comprehensive database operations through more than 20 specialized MCP tools. You can interact with the LLM by using natural language, and the LLM automatically selects and uses the appropriate tools to fulfill requests. Understanding these tools helps you know what database operations the LLM can perform on your behalf. ## Connection tools Monitor and verify your ScalarDB connection status and configuration. The following tool is available in both CRUD and SQL mode. | Tool | Description | |------|-------------| | `scalardb_connection_info` | Get the current connection status, configuration details, and health check results. | ## Schema management tools Create, modify, and inspect database structures including namespaces, tables, and indexes. The following tools are available in CRUD mode. | Tool | Description | |------|-------------| | `scalardb_create_namespace` | Create a new namespace/keyspace for organizing tables. | | `scalardb_drop_namespace` | Drop an existing namespace and all its tables. | | `scalardb_list_namespaces` | List all available namespaces in the database. | | `scalardb_create_table` | Create a new table with complete schema definition including partition keys, clustering keys, and columns. | | `scalardb_drop_table` | Drop an existing table and all its data. | | `scalardb_truncate_table` | Remove all data from a table while keeping the schema. | | `scalardb_describe_table` | Get detailed table schema including columns, keys, and metadata. | | `scalardb_list_tables` | List all tables within a specific namespace. | | `scalardb_add_new_column` | Add a new column to an existing table schema. | | `scalardb_create_index` | Create secondary indexes on table columns for faster queries. | | `scalardb_drop_index` | Drop an existing secondary index. | ## CRUD operation tools Perform data manipulation operations by using the ScalarDB Java client SDK for granular control and type safety. The following tools are available in CRUD mode. | Tool | Description | |------|-------------| | `scalardb_get` | Retrieve specific records by using partition keys, clustering keys, or secondary indexes. | | `scalardb_scan` | Scan records with flexible filtering, ordering, and pagination capabilities. | | `scalardb_insert` | Insert new records with automatic conflict detection. | | `scalardb_update` | Update existing records with conditional operations. | | `scalardb_upsert` | Insert new records or update existing ones (insert if they don't exist; update if they do exist). | | `scalardb_delete` | Delete records by using primary keys or conditional logic. | ## Transaction management tools Control ACID transactions by using the ScalarDB Java client SDK with proper isolation and consistency guarantees. Available in CRUD mode. | Tool | Description | |------|-------------| | `scalardb_begin_transaction` | Start a new read-write transaction with ACID guarantees. | | `scalardb_begin_readonly_transaction` | Start an optimized read-only transaction for queries. | | `scalardb_commit_transaction` | Commit a transaction and make all changes permanent. | | `scalardb_rollback_transaction` | Roll back a transaction and undo all changes. | ## SQL tools (ScalarDB Cluster only) Execute SQL commands directly through the ScalarDB SQL interface. The following tool is available in SQL mode. | Tool | Description | |------|-------------| | `scalardb_execute_sql` | Execute SQL queries directly (`SELECT`, `INSERT`, `UPDATE`, `DELETE`) with full SQL syntax support. | ## Coordinator tools Manage distributed transaction Coordinator tables for multi-database consistency. These tools are available in CRUD mode. | Tool | Description | |------|-------------| | `scalardb_create_coordinator_tables` | Create Coordinator tables required for distributed transactions. | | `scalardb_drop_coordinator_tables` | Drop Coordinator tables when no longer needed. | | `scalardb_truncate_coordinator_tables` | Clear Coordinator tables while preserving structure. | ## Tool availability by mode Different tools are available depending on the operational mode you choose. ### SQL mode - **Connection tools:** Monitor connection status and health. - **SQL tools:** Execute SQL queries directly through the ScalarDB SQL interface. - **Use case:** Best for ScalarDB Cluster deployments when you prefer using SQL syntax. ### CRUD mode - **Connection tools:** Monitor connection status and health. - **Schema management tools:** Create and manage namespaces, tables, and indexes. - **CRUD operation tools:** Perform data manipulation with the ScalarDB Java client SDK. - **Transaction management tools:** Control ACID transactions programmatically. - **Coordinator tools:** Manage distributed transaction coordination. - **Use case:** Required for ScalarDB Core. ================================================ FILE: docs/scalardb-samples/README.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Run Sample Applications Overview In this sub-category, you can learn how to run various sample applications that take advantage of ScalarDB. To set up and run the sample applications, see the following guides: ## Java - [Multi-storage Transaction Sample](multi-storage-transaction-sample/README.mdx) - [Microservice Transaction Sample](microservice-transaction-sample/README.mdx) - [Spring Data JDBC for ScalarDB with Multi-storage Transaction Sample](spring-data-multi-storage-transaction-sample/README.mdx) - [Spring Data JDBC for ScalarDB with Microservice Transaction Sample](spring-data-microservice-transaction-sample/README.mdx) - [Microservice Transactions with Shared ScalarDB Cluster with ScalarDB JDBC](microservice-transactions-sample-with-shared-cluster-with-jdbc/README.mdx) ## .NET - [Microservice Transactions with Shared ScalarDB Cluster with LINQ](dotnet-microservice-transactions-sample-with-shared-cluster-with-linq/README.mdx) ================================================ FILE: docs/scalardb-samples/dotnet-microservice-transactions-sample-with-shared-cluster-with-linq/README.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Create an Application That Supports Microservice Transactions in a Shared ScalarDB Cluster Environment by Using LINQ import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; This tutorial describes how to create a sample e-commerce application that supports microservice transactions and follows the shared-cluster pattern for the ScalarDB Cluster .NET Client SDK and LINQ. For details about the shared-cluster pattern, see [ScalarDB Cluster Deployment Patterns for Microservices](../../scalardb-cluster/deployment-patterns-for-microservices.mdx#shared-cluster-pattern). This section describes how to create a sample e-commerce application that follows the shared-cluster pattern by using the ScalarDB Cluster .NET Client SDK and LINQ. ## Overview of the sample microservice application The sample e-commerce application shows how users can order and pay for items by using a line of credit. The sample application has two microservices called the *Customer Service* and the *Order Service* based on the [database-per-service pattern](https://microservices.io/patterns/data/database-per-service.html): - The **Customer Service** manages customer information, including line-of-credit information, credit limit, and credit total. - The **Order Service** is responsible for order operations like placing an order and getting order histories. Each service has gRPC endpoints. Clients call the endpoints, and the services call each endpoint as well. The databases that you will be using in the sample application are Cassandra and MySQL. The Customer Service and the Order Service use Cassandra and MySQL, respectively, through ScalarDB Cluster. ![Overview](images/overview.png) As shown in the diagram, ScalarDB Cluster has a small Coordinator database used for the Consensus Commit protocol. The database is service independent and exists for managing transaction metadata for Consensus Commit in a highly available manner. In the sample application, for ease of setup and explanation, you co-locate the Coordinator database in the same Cassandra instance of the Order Service. Alternatively, you can manage the Coordinator database as a separate database. :::note Since the focus of the sample application is to demonstrate using ScalarDB Cluster, application-specific error handling, authentication processing, and similar functions are not included in the sample application. ::: ### Service endpoints The endpoints defined in the services are as follows: - Customer Service - `GetCustomerInfo` - `Payment` - `Repayment` - Order Service - `PlaceOrder` - `GetOrder` - `GetOrders` ### What you can do in this sample application The sample application supports the following types of transactions: - Get customer information through the `GetCustomerInfo` endpoint of the Customer Service. - Place an order by using a line of credit through the `PlaceOrder` endpoint of the Order Service and the `Payment` endpoint of the Customer Service. - Checks if the cost of the order is below the customer's credit limit. - If the check passes, records the order history and updates the amount the customer has spent. - Get order information by order ID through the `GetOrder` endpoint of the Order Service and the `GetCustomerInfo` endpoint of the Customer Service. - Get order information by customer ID through the `GetOrders` endpoint of the Order Service and the `GetCustomerInfo` endpoint of the Customer Service. - Make a payment through the `Repayment` endpoint of the Customer Service. - Reduces the amount the customer has spent. :::note The `GetCustomerInfo` endpoint works as a participant service endpoint when receiving a transaction ID from the coordinator. ::: ## Prerequisites - [.NET SDK 8.0](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later :::note .NET SDK 8.0 is the version used to create the sample application. For information about all supported versions, see [Requirements](../../requirements.mdx#net) ::: ## Set up ScalarDB Cluster The following sections describe how to set up the sample application that supports microservice transactions with ScalarDB Cluster. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/scalardb-dotnet-samples/microservice-transactions-sample-with-shared-cluster-with-linq ``` ### Set the license key Set the license key (trial license or commercial license) for the ScalarDB Cluster deployment to the configuration file [`scalardb-cluster-node.properties`](https://github.com/scalar-labs/scalardb-samples/tree/main/dotnet/microservice-transactions-sample-with-shared-cluster-with-linq/scalardb-cluster-node.properties). For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). ### Start containers The configuration file for ScalarDB Cluster is [`scalardb-cluster-node.properties`](https://github.com/scalar-labs/scalardb-samples/tree/main/dotnet/microservice-transactions-sample-with-shared-cluster-with-linq/scalardb-cluster-node.properties). The Customer Service and the Order Service use a standard `appsettings.json` file for their configurations. Cassandra and MySQL are already configured with the multi-storage setting, as shown in the configuration. For details about configuring the multi-storage transactions feature in ScalarDB, see [How to configure ScalarDB to support multi-storage transactions](../../multi-storage-transactions.mdx#how-to-configure-scalardb-to-support-multi-storage-transactions). For the sake of quickly setting up this sample application, you'll run ScalarDB Cluster in standalone mode. For details about running ScalarDB Cluster in standalone mode, see [ScalarDB Cluster Standalone Mode](../../scalardb-cluster/standalone-mode.mdx). Also, authentication is enabled to implement access control for each microservice. For details about authentication in ScalarDB, see [Authenticate Users](../../scalardb-cluster/scalardb-auth-with-sql.mdx). :::note For the sake of quickly setting up this sample application, wire encryption is not enabled. However, we recommend enabling wire encryption in a production environment to secure the communication between the client and the ScalarDB Cluster nodes, and amongst the ScalarDB Cluster nodes themselves. For details about wire encryption, see [Wire encryption](../../scalardb-cluster/encrypt-wire-communications.mdx). ::: To start the containers for the sample application, run the following command: ```console docker compose up -d --build ``` :::note Starting the Docker container may take more than one minute depending on your development environment. ::: ### Apply the schema, create users, grant privileges, and load the initial data The database schema (the method in which the data will be organized) for the sample application is defined as C# objects in [Common project](https://github.com/scalar-labs/scalardb-samples/tree/main/dotnet/microservice-transactions-sample-with-shared-cluster-with-linq/). To apply the schema, create users, grant privileges, and load the initial data, run the following command: ```console dotnet run --project DataLoader/DataLoader.csproj --config scalardb-options.json ``` #### Schema details All the tables for the Customer Service are created in the `customer_service` namespace. - `customer_service.customers`: a table that manages customers' information - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a line of credit - `credit_total`: the amount of money that each customer has already spent by using their line of credit Also, all the tables for the Order Service are created in the `order_service` namespace. - `order_service.orders`: a table that manages order information - `order_service.statements`: a table that manages order statement information - `order_service.items`: a table that manages information of items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/ERD.png) ### Initial data The following records should be stored in the `customer_service.customers` table: | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | And the following records should be stored in the `order_service.items` table: | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ### Users and privileges A `customer-service` user with `READ`, `CREATE`, `WRITE`, and `DELETE` privileges for the `customer_service` namespace, and an `order-service` user with `READ`, `CREATE`, `WRITE`, and `DELETE` privileges for the `order_service` namespace are created. ## Run the sample application The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. ### Get customer information Start with getting information about the customer whose ID is `1` by running the following command: ```console dotnet run --project Client/Client.csproj GetCustomerInfo 1 ``` You should see the following output: ```console { "id": 1, "name": "Yamada Taro", "creditLimit": 10000 } ``` At this time, `creditTotal` isn't shown, which means the current value of `creditTotal` is `0`. ### Place an order Then, have customer ID `1` place an order for three apples and two oranges by running the following command: :::note The order format in this command is `PlaceOrder :,:,...`. ::: ```console dotnet run --project Client/Client.csproj PlaceOrder 1 1:3,2:2 ``` You should see a similar output as below, with a different UUID for `orderId`, which confirms that the order was successful: ```console { "orderId": "4b076074-797f-4fdb-b357-59531f0aec12" } ``` ### Check the order details Check details about the order by running the following command, replacing `` with the UUID for the `orderId` that was shown after running the previous command: ```console dotnet run --project Client/Client.csproj GetOrder ``` You should see a similar output as below, with different UUIDs for `orderId` and `timestamp`: ```console { "order": { "orderId": "4b076074-797f-4fdb-b357-59531f0aec12", "timestamp": "63825948620680", "customerId": 1, "customerName": "Yamada Taro", "statement": [ { "itemId": 1, "itemName": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "itemId": 2, "itemName": "Orange", "price": 2000, "count": 2, "total": 4000 } ], "total": 7000 } } ``` ### Place another order Place an order for one melon that uses the remaining amount in `creditTotal` for customer ID `1` by running the following command: ```console dotnet run --project Client/Client.csproj PlaceOrder 1 5:1 ``` You should see a similar output as below, with a different UUID for `orderId`, which confirms that the order was successful: ```console { "orderId": "6c7750c8-10ad-4f02-aa3c-e30621d95151" } ``` ### Check the order history Get the history of all orders for customer ID `1` by running the following command: ```console dotnet run --project Client/Client.csproj GetOrders 1 ``` You should see a similar output as below, with different UUIDs for `orderId` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: ```console { "order": [ { "orderId": "4b076074-797f-4fdb-b357-59531f0aec12", "timestamp": "63825948620680", "customerId": 1, "customerName": "Yamada Taro", "statement": [ { "itemId": 1, "itemName": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "itemId": 2, "itemName": "Orange", "price": 2000, "count": 2, "total": 4000 } ], "total": 7000 }, { "orderId": "6c7750c8-10ad-4f02-aa3c-e30621d95151", "timestamp": "63825948672045", "customerId": 1, "customerName": "Yamada Taro", "statement": [ { "itemId": 5, "itemName": "Melon", "price": 3000, "count": 1, "total": 3000 } ], "total": 3000 } ] } ``` ### Check the credit total Get the credit total for customer ID `1` by running the following command: ```console dotnet run --project Client/Client.csproj GetCustomerInfo 1 ``` You should see the following output, which shows that customer ID `1` has reached their `creditLimit` amount in `creditTotal` and cannot place anymore orders: ```console { "id": 1, "name": "Yamada Taro", "creditLimit": 10000, "creditTotal": 10000 } ``` Try to place an order for one grape and one mango by running the following command: ```console dotnet run --project Client/Client.csproj PlaceOrder 1 3:1,4:1 ``` You should see the following output, which shows that the order failed because the `creditTotal` amount would exceed the `creditLimit` amount: ```console Unhandled exception: Grpc.Core.RpcException: Status(StatusCode="FailedPrecondition", Detail="Credit limit exceeded (17500 > 10000)") at MicroserviceTransactionsSample.Client.Commands.PlaceOrderCommand.placeOrder(Int32 customerId, Dictionary`2 orders, OrderServiceClient client) ... ``` ### Make a payment To continue making orders, customer ID `1` must make a payment to reduce the `creditTotal` amount. Make a payment by running the following command: ```console dotnet run --project Client/Client.csproj Repayment 1 8000 ``` Then, check the `creditTotal` amount for customer ID `1` by running the following command: ```console dotnet run --project Client/Client.csproj GetCustomerInfo 1 ``` You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `creditTotal` amount: ```console { "id": 1, "name": "Yamada Taro", "creditLimit": 10000, "creditTotal": 2000 } ``` Now that customer ID `1` has made a payment, place an order for one grape and one mango by running the following command: ```console dotnet run --project Client/Client.csproj PlaceOrder 1 3:1,4:1 ``` You should see a similar output as below, with a different UUID for `orderId`, which confirms that the order was successful: ```console { "orderId": "5e26a530-0b54-4205-bb74-a06675570934" } ``` ## Stop the sample application To stop the sample application, you need to stop the Docker containers that are running Cassandra, MySQL, ScalarDB Cluster, and the microservices. To stop the Docker containers, run the following command: ```console docker compose down -v ``` ================================================ FILE: docs/scalardb-samples/microservice-transaction-sample/README.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Create a Sample Application That Supports Microservice Transactions import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to create a sample application that supports microservice transactions in ScalarDB. ## Overview This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit through [transactions with a two-phase commit interface](../../two-phase-commit-transactions.mdx) in ScalarDB. The sample application has two microservices called the *Customer Service* and the *Order Service* based on the [database-per-service pattern](https://microservices.io/patterns/data/database-per-service.html): - The **Customer Service** manages customer information, including line-of-credit information, credit limit, and credit total. - The **Order Service** is responsible for order operations like placing an order and getting order histories. Each service has gRPC endpoints. Clients call the endpoints, and the services call each endpoint as well. The databases that you will be using in the sample application are Cassandra and MySQL. The Customer Service and the Order Service use Cassandra and MySQL, respectively, through ScalarDB. ![Overview](images/overview.png) As shown in the diagram, both services access a small Coordinator database used for the Consensus Commit protocol. The database is service-independent and exists for managing transaction metadata for Consensus Commit in a highly available manner. In the sample application, for ease of setup and explanation, we co-locate the Coordinator database in the same Cassandra instance of the Order Service. Alternatively, you can manage the Coordinator database as a separate database. :::note Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling in ScalarDB, see [How to handle exceptions](../../api-guide.mdx#how-to-handle-exceptions). Additionally, for the purpose of the sample application, each service has one container so that you can avoid using request routing between the services. However, for production use, because each service typically has multiple servers or hosts for scalability and availability, you should consider request routing between the services in transactions with a two-phase commit interface. For details about request routing, see [Request routing in transactions with a two-phase commit interface](../../two-phase-commit-transactions.mdx#request-routing-in-transactions-with-a-two-phase-commit-interface). ::: ### Service endpoints The endpoints defined in the services are as follows: - Customer Service - `getCustomerInfo` - `payment` - `prepare` - `validate` - `commit` - `rollback` - `repayment` - Order Service - `placeOrder` - `getOrder` - `getOrders` ### What you can do in this sample application The sample application supports the following types of transactions: - Get customer information through the `getCustomerInfo` endpoint of the Customer Service. - Place an order by using a line of credit through the `placeOrder` endpoint of the Order Service and the `payment`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. - Checks if the cost of the order is below the customer's credit limit. - If the check passes, records the order history and updates the amount the customer has spent. - Get order information by order ID through the `getOrder` endpoint of the Order Service and the `getCustomerInfo`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. - Get order information by customer ID through the `getOrders` endpoint of the Order Service and the `getCustomerInfo`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. - Make a payment through the `repayment` endpoint of the Customer Service. - Reduces the amount the customer has spent. :::note The `getCustomerInfo` endpoint works as a participant service endpoint when receiving a transaction ID from the coordinator. ::: ## Prerequisites for this sample application - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ## Set up ScalarDB The following sections describe how to set up the sample application that supports microservices transactions in ScalarDB. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/microservice-transaction-sample ``` ### Start Cassandra and MySQL Cassandra and MySQL are already configured for the sample application, as shown in [`database-cassandra.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/database-cassandra.properties) and [`database-mysql.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/database-mysql.properties), respectively. For details about configuring the multi-storage transactions feature in ScalarDB, see [How to configure ScalarDB to support multi-storage transactions](../../multi-storage-transactions.mdx#how-to-configure-scalardb-to-support-multi-storage-transactions). To start Cassandra and MySQL, which are included in the Docker container for the sample application, run the following command: ```console docker-compose up -d mysql cassandra ``` :::note Starting the Docker container may take more than one minute depending on your development environment. ::: ### Load the schema The database schema (the method in which the data will be organized) for the sample application has already been defined in [`customer-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service-schema.json) for the Customer Service and [`order-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service-schema.json) for the Order Service. To apply the schema, go to the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you want to use to the `scalardb-samples/microservice-transaction-sample` folder. #### MySQL To load the schema for [`customer-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service-schema.json) into MySQL, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: ```console java -jar scalardb-schema-loader-.jar --config database-mysql.properties --schema-file customer-service-schema.json ``` #### Cassandra To load the schema for [`order-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service-schema.json) into Cassandra, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: ```console java -jar scalardb-schema-loader-.jar --config database-cassandra.properties --schema-file order-service-schema.json --coordinator ``` #### Schema details As shown in [`customer-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service-schema.json), all the tables for the Customer Service are created in the `customer_service` namespace. - `customer_service.customers`: a table that manages customers' information - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a line of credit - `credit_total`: the amount of money that each customer has already spent by using their line of credit As shown in [`order-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service-schema.json), all the tables for the Order Service are created in the `order_service` namespace. - `order_service.orders`: a table that manages order information - `order_service.statements`: a table that manages order statement information - `order_service.items`: a table that manages information of items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/ERD.png) ### Load the initial data by starting the microservices Before starting the microservices, build the Docker images of the sample application by running the following command: ```console ./gradlew docker ``` Then, start the microservices by running the following command: ```console docker-compose up -d customer-service order-service ``` After starting the microservices and the initial data has loaded, the following records should be stored in the `customer_service.customers` table: | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | And the following records should be stored in the `order_service.items` table: | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Execute transactions and retrieve data in the sample application The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. ### Get customer information Start with getting information about the customer whose ID is `1` by running the following command: ```console ./gradlew :client:run --args="GetCustomerInfo 1" ``` You should see the following output: ```console ... {"id": 1,"name": "Yamada Taro","credit_limit": 10000} ... ``` At this time, `credit_total` isn't shown, which means the current value of `credit_total` is `0`. ### Place an order Then, have customer ID `1` place an order for three apples and two oranges by running the following command: :::note The order format in this command is `./gradlew run --args="PlaceOrder :,:,..."`. ::: ```console ./gradlew :client:run --args="PlaceOrder 1 1:3,2:2" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "4ccdb21c-ac03-4b48-bcb7-cad57eac1e79"} ... ``` ### Check the order details Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: ```console ./gradlew :client:run --args="GetOrder " ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: ```console ... {"order": {"order_id": "4ccdb21c-ac03-4b48-bcb7-cad57eac1e79","timestamp": 1631605253126,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} ... ``` ### Place another order Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: ```console ./gradlew :client:run --args="PlaceOrder 1 5:1" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "0b10db66-faa6-4323-8a7a-474e8534a7ee"} ... ``` ### Check the order history Get the history of all orders for customer ID `1` by running the following command: ```console ./gradlew :client:run --args="GetOrders 1" ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: ```console ... {"order": [{"order_id": "0b10db66-faa6-4323-8a7a-474e8534a7ee","timestamp": 1631605501485,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000},{"order_id": "4ccdb21c-ac03-4b48-bcb7-cad57eac1e79","timestamp": 1631605253126,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}]} ... ``` ### Check the credit total Get the credit total for customer ID `1` by running the following command: ```console ./gradlew :client:run --args="GetCustomerInfo 1" ``` You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: ```console ... {"id": 1,"name": "Yamada Taro","credit_limit": 10000,"credit_total": 10000} ... ``` Try to place an order for one grape and one mango by running the following command: ```console ./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" ``` You should see the following output, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount: ```console ... io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Credit limit exceeded at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) at sample.rpc.OrderServiceGrpc$OrderServiceBlockingStub.placeOrder(OrderServiceGrpc.java:296) at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:38) at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:12) at picocli.CommandLine.executeUserObject(CommandLine.java:2041) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) at picocli.CommandLine.execute(CommandLine.java:2170) at sample.client.Client.main(Client.java:39) ... ``` ### Make a payment To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. Make a payment by running the following command: ```console ./gradlew :client:run --args="Repayment 1 8000" ``` Then, check the `credit_total` amount for customer ID `1` by running the following command: ```console ./gradlew :client:run --args="GetCustomerInfo 1" ``` You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: ```console ... {"id": 1,"name": "Yamada Taro","credit_limit": 10000,"credit_total": 2000} ... ``` Now that customer ID `1` has made a payment, place an order for one grape and one mango by running the following command: ```console ./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "dd53dd9d-aaf4-41db-84b2-56951fed6425"} ... ``` ## Stop the sample application To stop the sample application, you need to stop the Docker containers that are running Cassandra, MySQL, and the microservices. To stop the Docker containers, run the following command: ```console docker-compose down ``` ## Reference - How the microservice transaction is achieved The transactions for placing an order, getting a single order, and getting the history of orders achieve the microservice transaction. This section focuses on how the transactions that span the Customer Service and the Order Service are implemented by placing an order as an example. The following sequence diagram shows the transaction for placing an order: ![Microservice transaction sequence diagram](images/sequence_diagram.png) ### 1. Transaction with a two-phase commit interface is started When a client sends a request to place an order to the Order Service, `OrderService.placeOrder()` is called, and the microservice transaction starts. At first, the Order Service starts a transaction with a two-phase commit interface with `start()` as follows. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). ```java transaction = twoPhaseCommitTransactionManager.start(); ``` ### 2. CRUD operations are executed After the transaction with a two-phase commit interface starts, CRUD operations are executed. The Order Service puts the order information in the `order_service.orders` table and the detailed information in the `order_service.statements` table as follows. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java): ```java // Put the order info into the `orders` table. Order.put(transaction, orderId, request.getCustomerId(), System.currentTimeMillis()); int amount = 0; for (ItemOrder itemOrder : request.getItemOrderList()) { // Put the order statement into the `statements` table. Statement.put(transaction, orderId, itemOrder.getItemId(), itemOrder.getCount()); // Retrieve the item info from the `items` table. Optional item = Item.get(transaction, itemOrder.getItemId()); if (!item.isPresent()) { responseObserver.onError( Status.NOT_FOUND.withDescription("Item not found").asRuntimeException()); return; } // Calculate the total amount. amount += item.get().price * itemOrder.getCount(); } ``` Then, the Order Service calls the `payment` gRPC endpoint of the Customer Service along with the transaction ID. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). ```java customerServiceStub.payment( PaymentRequest.newBuilder() .setTransactionId(transactionId) .setCustomerId(customerId) .setAmount(amount) .build()); ``` The `payment` endpoint of the Customer Service first joins the transaction with `join()` as follows. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java private void execOperationsAsParticipant(String funcName, String transactionId, TransactionFunction operations, StreamObserver responseObserver) { try { // Join the transaction TwoPhaseCommitTransaction transaction = twoPhaseCommitTransactionManager.join(transactionId); // Execute operations T response = operations.apply(transaction); ``` The endpoint then gets the customer information and checks if the customer's credit total exceeds the credit limit after the payment. If the credit total does not exceed the credit limit, the endpoint updates the customer's credit total. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java execOperationsAsParticipant("Payment", request.getTransactionId(), transaction -> { // Retrieve the customer info for the customer ID Optional result = Customer.get(transaction, request.getCustomerId()); if (!result.isPresent()) { throw Status.NOT_FOUND.withDescription("Customer not found").asRuntimeException(); } int updatedCreditTotal = result.get().creditTotal + request.getAmount(); // Check if the credit total exceeds the credit limit after payment if (updatedCreditTotal > result.get().creditLimit) { throw Status.FAILED_PRECONDITION .withDescription("Credit limit exceeded") .asRuntimeException(); } // Update `credit_total` for the customer Customer.updateCreditTotal(transaction, request.getCustomerId(), updatedCreditTotal); return Empty.getDefaultInstance(); }, responseObserver ); ``` ### 3. Transaction is committed by using the two-phase commit protocol After the Order Service receives the update that the payment succeeded, the Order Service tries to commit the transaction. To commit the transaction, the Order Service starts with preparing the transaction. The Order Service calls `prepare()` from its transaction object and calls the `prepare` gRPC endpoint of the Customer Service. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java): ```java transaction.prepare(); callPrepareEndpoint(transaction.getId()); ``` In this endpoint, the Customer Service resumes the transaction and calls `prepare()` from its transaction object, as well. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java): ```java // Resume the transaction. transaction = twoPhaseCommitTransactionManager.resume(request.getTransactionId()); // Prepare the transaction. transaction.prepare(); ``` Similarly, the Order Service and the Customer Service call `validate()` from their transaction objects. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java) and [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). For details about `validate()`, see [Validate the transaction](../../two-phase-commit-transactions.mdx#validate-the-transaction). After preparing and validating the transaction succeeds in both the Order Service and the Customer Service, the transaction can be committed. In this case, the Order Service calls `commit()` from its transaction object and then calls the `commit` gRPC endpoint of the Customer Service. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). ```java transaction.commit(); callCommitEndpoint(transaction.getId()); ``` In this endpoint, the Customer Service resumes the transaction and calls `commit()` from its transaction object, as well. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java // Resume the transaction. transaction = twoPhaseCommitTransactionManager.resume(request.getTransactionId()); // Commit the transaction. transaction.commit(); ``` ### Error handling If an error happens while executing a transaction, you will need to roll back the transaction. In this case, the Order Service calls `rollback()` from its transaction object and then calls the `rollback` gRPC endpoint of the Customer Service. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). ```java transaction.rollback(); callRollbackEndpoint(transaction.getId()); ``` In this endpoint, the Customer Service resumes the transaction and calls `rollback()` from its transaction object, as well. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java // Resume the transaction. TwoPhaseCommitTransaction transaction = twoPhaseCommitTransactionManager.resume(request.getTransactionId()); // Roll back the transaction. transaction.rollback(); ``` For details about how to handle exceptions in ScalarDB, see [How to handle exceptions](../../api-guide.mdx#how-to-handle-exceptions). ================================================ FILE: docs/scalardb-samples/microservice-transactions-sample-with-shared-cluster-with-jdbc/README.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Create an Application That Supports Microservice Transactions in a Shared ScalarDB Cluster Environment by Using ScalarDB JDBC import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to create a sample e-commerce application that supports microservice transactions and follows the shared-cluster pattern for ScalarDB Cluster by using ScalarDB JDBC. For details about the shared-cluster pattern, see [ScalarDB Cluster Deployment Patterns for Microservices](../../scalardb-cluster/deployment-patterns-for-microservices.mdx#shared-cluster-pattern). ## Overview of the sample microservice application The sample e-commerce application shows how users can order and pay for items by using a line of credit. The sample application has two microservices called the *Customer Service* and the *Order Service* based on the [database-per-service pattern](https://microservices.io/patterns/data/database-per-service.html): - The **Customer Service** manages customer information, including line-of-credit information, credit limit, and credit total. - The **Order Service** is responsible for order operations like placing an order and getting order histories. Each service has gRPC endpoints. Clients call the endpoints, and the services call each endpoint as well. The databases that you will be using in the sample application are Cassandra and MySQL. The Customer Service and the Order Service use Cassandra and MySQL, respectively, through ScalarDB Cluster. ![Overview](./images/overview.png) As shown in the diagram, ScalarDB Cluster has a small Coordinator database used for the Consensus Commit protocol. The database is service-independent and exists for managing transaction metadata for Consensus Commit in a highly available manner. In the sample application, for ease of setup and explanation, you co-locate the Coordinator database in the same Cassandra instance of the Order Service. Alternatively, you can manage the Coordinator database as a separate database. :::note Since the focus of the sample application is to demonstrate using ScalarDB Cluster, application-specific error handling, authentication processing, and similar functions are not included in the sample application. ::: ### Service endpoints The endpoints defined in the services are as follows: - Customer Service - `getCustomerInfo` - `payment` - `repayment` - Order Service - `placeOrder` - `getOrder` - `getOrders` ### What you can do in this sample application The sample application supports the following types of transactions: - Get customer information through the `getCustomerInfo` endpoint of the Customer Service. - Place an order by using a line of credit through the `placeOrder` endpoint of the Order Service and the `payment` endpoint of the Customer Service. - Checks if the cost of the order is below the customer's credit limit. - If the check passes, records the order history and updates the amount the customer has spent. - Get order information by order ID through the `getOrder` endpoint of the Order Service and the `getCustomerInfo` endpoint of the Customer Service. - Get order information by customer ID through the `getOrders` endpoint of the Order Service and the `getCustomerInfo` endpoint of the Customer Service. - Make a payment through the `repayment` endpoint of the Customer Service. - Reduces the amount the customer has spent. :::note The `getCustomerInfo` endpoint works as a participant service endpoint when receiving a transaction ID from the coordinator. ::: ## Prerequisites - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later Also, you need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). ## Set up ScalarDB Cluster The following sections describe how to set up the sample application that supports microservice transactions with ScalarDB Cluster. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/microservice-transaction-sample-with-shared-cluster-with-jdbc/ ``` ### Set the license key Set the license key (trial license or commercial license) for the ScalarDB Cluster deployment to the configuration file [`scalardb-cluster-node.properties`](https://github.com/scalar-labs/scalardb-samples/tree/main/microservice-transaction-sample-with-shared-cluster-with-jdbc/scalardb-cluster-node.properties). For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). ### Start Cassandra, MySQL, and ScalarDB Cluster The configuration file for ScalarDB Cluster is [`scalardb-cluster-node.properties`](https://github.com/scalar-labs/scalardb-samples/tree/main/microservice-transaction-sample-with-shared-cluster-with-jdbc/scalardb-cluster-node.properties). Cassandra and MySQL are already configured with the multi-storage setting, as shown in the configuration. For details about configuring the multi-storage transactions feature in ScalarDB, see [How to configure ScalarDB to support multi-storage transactions](../../multi-storage-transactions.mdx#how-to-configure-scalardb-to-support-multi-storage-transactions). For the sake of quickly setting up this sample application, you'll run ScalarDB Cluster in standalone mode. For details about running ScalarDB Cluster in standalone mode, see [ScalarDB Cluster Standalone Mode](../../scalardb-cluster/standalone-mode.mdx). Also, ScalarDB Auth is enabled to implement access control for each microservice. For details about ScalarDB Auth, see [ScalarDB Auth with SQL](../../scalardb-cluster/scalardb-auth-with-sql.mdx). :::note For the sake of quickly setting up this sample application, wire encryption is not enabled. However, we recommend enabling wire encryption in a production environment to secure the communication between the client and the ScalarDB Cluster nodes, and amongst the ScalarDB Cluster nodes themselves. For details about wire encryption, see [Wire encryption](../../scalardb-cluster/encrypt-wire-communications.mdx). ::: To start Cassandra, MySQL, and ScalarDB Cluster, which are included in the Docker container for the sample application, run the following command: ```console docker compose up -d mysql cassandra scalardb-cluster-node ``` :::note Starting the Docker container may take more than one minute depending on your development environment. ::: ### Load the schema The database schema (the method in which the data will be organized) for the sample application has already been defined in [`schema.sql`](https://github.com/scalar-labs/scalardb-samples/tree/main/microservice-transaction-sample-with-shared-cluster-with-jdbc/schema.sql). To apply the schema, go to the [Releases](https://github.com/scalar-labs/scalardb/releases) of ScalarDB and download the SQL CLI tool (`scalardb-cluster-sql-cli--all.jar`) for the version of ScalarDB Cluster that you want to use. Then, run the following command, replacing `` with the version of the SQL CLI tool that you downloaded: ```console java -jar scalardb-cluster-sql-cli--all.jar --config scalardb-cluster-sql-cli.properties --file schema.sql ``` #### Schema details As shown in [`schema.sql`](https://github.com/scalar-labs/scalardb-samples/tree/main/microservice-transaction-sample-with-shared-cluster-with-jdbc/schema.sql) for the sample application, all the tables for the Customer Service are created in the `customer_service` namespace. - `customer_service.customers`: a table that manages customers' information - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a line of credit - `credit_total`: the amount of money that each customer has already spent by using their line of credit Also, all the tables for the Order Service are created in the `order_service` namespace. - `order_service.orders`: a table that manages order information - `order_service.statements`: a table that manages order statement information - `order_service.items`: a table that manages information of items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](./images/ERD.png) ### Create users for the services and grant privileges to them To implement access control for each microservice, you need to create users for the services and grant privileges to them. Run the following command, replacing `` with the version of the SQL CLI tool that you downloaded: ```console java -jar scalardb-cluster-sql-cli--all.jar --config scalardb-cluster-sql-cli.properties --file user-privileges.sql ``` As shown in [`user-privileges.sql`](https://github.com/scalar-labs/scalardb-samples/tree/main/microservice-transaction-sample-with-shared-cluster-with-jdbc/user-privileges.sql), you create users for Customer Service and Order Service and grant privileges to them. ## Start the microservices The configuration files for Customer Service and Order Service are [`customer-service.properties`](https://github.com/scalar-labs/scalardb-samples/tree/main/microservice-transaction-sample-with-shared-cluster-with-jdbc/customer-service/customer-service.properties) and [`order-service.properties`](https://github.com/scalar-labs/scalardb-samples/tree/main/microservice-transaction-sample-with-shared-cluster-with-jdbc/order-service/order-service.properties), respectively. Before starting the microservices, build the Docker images of the sample application by running the following command: ```console ./gradlew docker ``` Then, start the microservices by running the following command: ```console docker compose up -d customer-service order-service ``` After starting the microservices and the initial data has loaded, the following records should be stored in the `customer_service.customers` table: | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | And the following records should be stored in the `order_service.items` table: | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Execute transactions and retrieve data in the sample application The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. ### Get customer information Start with getting information about the customer whose ID is `1` by running the following command: ```console ./gradlew :client:run --args="GetCustomerInfo 1" -q ``` You should see the following output: ```console { "id": 1, "name": "Yamada Taro", "creditLimit": 10000 } ``` At this time, `credit_total` isn't shown, which means the current value of `credit_total` is `0`. ### Place an order Then, have customer ID `1` place an order for three apples and two oranges by running the following command: :::note The order format in this command is `./gradlew run --args="PlaceOrder :,:,..."`. ::: ```console ./gradlew :client:run --args="PlaceOrder 1 1:3,2:2" -q ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "orderId": "2dab1af1-a008-45b2-92e3-f30ff2a4ae3e" } ``` ### Check the order details Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: ```console ./gradlew :client:run --args="GetOrder " -q ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: ```console { "order": { "orderId": "2dab1af1-a008-45b2-92e3-f30ff2a4ae3e", "timestamp": "1708566933602", "customerId": 1, "customerName": "Yamada Taro", "statement": [{ "itemId": 1, "itemName": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "itemId": 2, "itemName": "Orange", "price": 2000, "count": 2, "total": 4000 }], "total": 7000 } } ``` ### Place another order Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: ```console ./gradlew :client:run --args="PlaceOrder 1 5:1" -q ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "orderId": "abc6ed14-ac3d-4d4c-9a83-7fa0e70c1d4e" } ``` ### Check the order history Get the history of all orders for customer ID `1` by running the following command: ```console ./gradlew :client:run --args="GetOrders 1" -q ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: ```console { "order": [{ "orderId": "2dab1af1-a008-45b2-92e3-f30ff2a4ae3e", "timestamp": "1708566933602", "customerId": 1, "customerName": "Yamada Taro", "statement": [{ "itemId": 1, "itemName": "Apple", "price": 1000, "count": 3, "total": 3000 }, { "itemId": 2, "itemName": "Orange", "price": 2000, "count": 2, "total": 4000 }], "total": 7000 }, { "orderId": "abc6ed14-ac3d-4d4c-9a83-7fa0e70c1d4e", "timestamp": "1708566978052", "customerId": 1, "customerName": "Yamada Taro", "statement": [{ "itemId": 5, "itemName": "Melon", "price": 3000, "count": 1, "total": 3000 }], "total": 3000 }] } ``` ### Check the credit total Get the credit total for customer ID `1` by running the following command: ```console ./gradlew :client:run --args="GetCustomerInfo 1" -q ``` You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: ```console { "id": 1, "name": "Yamada Taro", "creditLimit": 10000, "creditTotal": 10000 } ``` Try to place an order for one grape and one mango by running the following command: ```console ./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" -q ``` You should see the following output, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount: ```console io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Credit limit exceeded at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167) at sample.rpc.OrderServiceGrpc$OrderServiceBlockingStub.placeOrder(OrderServiceGrpc.java:288) at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:38) at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:12) at picocli.CommandLine.executeUserObject(CommandLine.java:2041) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) at picocli.CommandLine.execute(CommandLine.java:2170) at sample.client.Client.main(Client.java:39) ``` ### Make a payment To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. Make a payment by running the following command: ```console ./gradlew :client:run --args="Repayment 1 8000" -q ``` Then, check the `credit_total` amount for customer ID `1` by running the following command: ```console ./gradlew :client:run --args="GetCustomerInfo 1" -q ``` You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: ```console { "id": 1, "name": "Yamada Taro", "creditLimit": 10000, "creditTotal": 2000 } ``` Now that customer ID `1` has made a payment, place an order for one grape and one mango by running the following command: ```console ./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" -q ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console { "orderId": "e29a5fd2-58f6-4bc5-9fef-8852461232e8" } ``` ## Stop the sample application To stop the sample application, you need to stop the Docker containers that are running Cassandra, MySQL, ScalarDB Cluster, and the microservices. To stop the Docker containers, run the following command: ```console docker compose down ``` ================================================ FILE: docs/scalardb-samples/multi-storage-transaction-sample/README.mdx ================================================ --- tags: - Community - Enterprise Standard - Enterprise Premium displayed_sidebar: docsEnglish --- # Create a Sample Application That Supports Multi-Storage Transactions import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to create a sample application that supports the multi-storage transactions feature in ScalarDB. ## Overview This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using the [multi-storage transactions](../../multi-storage-transactions.mdx) feature in ScalarDB. In this tutorial, you will build an application that uses both Cassandra and MySQL. By using the multi-storage transactions feature in ScalarDB, you can execute a transaction that spans both Cassandra and MySQL. ![Overview](images/overview.png) :::note Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling in ScalarDB, see [How to handle exceptions](../../api-guide.mdx#how-to-handle-exceptions). ::: ### What you can do in this sample application The sample application supports the following types of transactions: - Get customer information. - Place an order by using a line of credit. - Checks if the cost of the order is below the customer's credit limit. - If the check passes, records the order history and updates the amount the customer has spent. - Get order information by order ID. - Get order information by customer ID. - Make a payment. - Reduces the amount the customer has spent. ## Prerequisites for this sample application - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ## Set up ScalarDB The following sections describe how to set up the sample application that supports the multi-storage transactions feature in ScalarDB. ### Clone the ScalarDB samples repository Open **Terminal**, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory that contains the sample application by running the following command: ```console cd scalardb-samples/multi-storage-transaction-sample ``` ### Start Cassandra and MySQL Cassandra and MySQL are already configured for the sample application, as shown in [`database.properties`](https://github.com/scalar-labs/scalardb-samples/tree/master/multi-storage-transaction-sample/database.properties). For details about configuring the multi-storage transactions feature in ScalarDB, see [How to configure ScalarDB to support multi-storage transactions](../../multi-storage-transactions.mdx#how-to-configure-scalardb-to-support-multi-storage-transactions). To start Cassandra and MySQL, which are included in the Docker container for the sample application, make sure Docker is running and then run the following command: ```console docker-compose up -d ``` :::note Starting the Docker container may take more than one minute depending on your development environment. ::: ### Load the schema The database schema (the method in which the data will be organized) for the sample application has already been defined in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/master/multi-storage-transaction-sample/schema.json). To apply the schema, go to the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you want to use to the `scalardb-samples/multi-storage-transaction-sample` folder. Then, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: ```console java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator ``` #### Schema details As shown in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/master/multi-storage-transaction-sample/schema.json) for the sample application, all the tables are created in the `customer` and `order` namespaces. - `customer.customers`: a table that manages customers' information - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a line of credit - `credit_total`: the amount of money that each customer has already spent by using their line of credit - `order.orders`: a table that manages order information - `order.statements`: a table that manages order statement information - `order.items`: a table that manages information of items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/ERD.png) ### Load the initial data After the Docker container has started, load the initial data by running the following command: ```console ./gradlew run --args="LoadInitialData" ``` After the initial data has loaded, the following records should be stored in the tables. **`customer.customers` table** | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | **`order.items` table** | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Execute transactions and retrieve data in the sample application The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. ### Get customer information Start with getting information about the customer whose ID is `1` by running the following command: ```console ./gradlew run --args="GetCustomerInfo 1" ``` You should see the following output: ```console ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0} ... ``` ### Place an order Then, have customer ID `1` place an order for three apples and two oranges by running the following command: :::note The order format in this command is `./gradlew run --args="PlaceOrder :,:,..."`. ::: ```console ./gradlew run --args="PlaceOrder 1 1:3,2:2" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e"} ... ``` ### Check the order details Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: ```console ./gradlew run --args="GetOrder " ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: ```console ... {"order": {"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} ... ``` ### Place another order Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: ```console ./gradlew run --args="PlaceOrder 1 5:1" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d"} ... ``` ### Check the order history Get the history of all orders for customer ID `1` by running the following command: ```console ./gradlew run --args="GetOrders 1" ``` You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: ```console ... {"order": [{"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000},{"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d","timestamp": 1650948412766,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000}]} ... ``` ### Check the credit total Get the credit total for customer ID `1` by running the following command: ```console ./gradlew run --args="GetCustomerInfo 1" ``` You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: ```console ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} ... ``` Try to place an order for one grape and one mango by running the following command: ```console ./gradlew run --args="PlaceOrder 1 3:1,4:1" ``` You should see the following output, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount: ```console ... java.lang.RuntimeException: Credit limit exceeded at sample.Sample.placeOrder(Sample.java:205) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:33) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:8) at picocli.CommandLine.executeUserObject(CommandLine.java:1783) at picocli.CommandLine.access$900(CommandLine.java:145) at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975) at picocli.CommandLine.execute(CommandLine.java:1904) at sample.command.SampleCommand.main(SampleCommand.java:35) ... ``` ### Make a payment To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. Make a payment by running the following command: ```console ./gradlew run --args="Repayment 1 8000" ``` Then, check the `credit_total` amount for customer ID `1` by running the following command: ```console ./gradlew run --args="GetCustomerInfo 1" ``` You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: ```console ... {"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000} ... ``` Now that customer ID `1` has made a payment, place an order for one grape and one mango by running the following command: ```console ./gradlew run --args="PlaceOrder 1 3:1,4:1" ``` You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: ```console ... {"order_id": "8911cab3-1c2b-4322-9386-adb1c024e078"} ... ``` ## Stop the sample application To stop the sample application, stop the Docker container by running the following command: ```console docker-compose down ``` ================================================ FILE: docs/scalardb-samples/spring-data-microservice-transaction-sample/README.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Sample application of Spring Data JDBC for ScalarDB with Microservice Transactions import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to create a sample Spring Boot application for microservice transactions by using Spring Data JDBC for ScalarDB. For details about these features, see [Two-phase Commit Transactions](../../two-phase-commit-transactions.mdx) and [Guide of Spring Data JDBC for ScalarDB](../../scalardb-sql/spring-data-guide.mdx). ## Prerequisites for this sample application - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ## Sample application ### Overview This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit through [transactions with a two-phase commit interface](../../two-phase-commit-transactions.mdx) in ScalarDB. There are two microservices called the *Customer Service* and the *Order Service* based on the [*Database-per-service* pattern](https://microservices.io/patterns/data/database-per-service.html) in this sample application. The Customer Service manages customers' information including credit card information like a credit limit and a credit total. The Order Service is responsible for order operations like placing an order and getting order histories. Each service has gRPC endpoints. Clients call the endpoints, and the services call the endpoints each other as well. The Customer Service and the Order Service use MySQL and Cassandra through ScalarDB, respectively. ![Overview](images/overview.png) Each service accesses the databases through its own dedicated ScalarDB Cluster. :::note Both ScalarDB Clusters access a small coordinator database used for the Consensus Commit protocol. In this sample application, for ease of setup and explanation, the coordinator database is co-located in the same Cassandra instance of the Order Service, but of course, the coordinator database can be managed as a separate database. Also, application-specific error handling, authentication processing, etc. are omitted in the sample application since it focuses on explaining how to use ScalarDB. For details about handling exceptions in ScalarDB, see [How to handle exceptions](../../two-phase-commit-transactions.mdx#how-to-handle-exceptions). ::: ### Service endpoints The endpoints defined in the services are as follows: - Customer Service - `getCustomerInfo` - `payment` - `prepare` - `validate` - `commit` - `rollback` - `repayment` - Order Service - `placeOrder` - `getOrder` - `getOrders` ### What you can do in this sample application The sample application supports the following types of transactions: - Get customer information through the `getCustomerInfo` endpoint of the Customer Service. - Place an order by using a line of credit through the `placeOrder` endpoint of the Order Service and the `payment`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. - Checks if the cost of the order is below the customer's credit limit. - If the check passes, records the order history and updates the amount the customer has spent. - Get order information by order ID through the `getOrder` endpoint of the Order Service and the `getCustomerInfo`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. - Get order information by customer ID through the `getOrders` endpoint of the Order Service and the `getCustomerInfo`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. - Make a payment through the `repayment` endpoint of the Customer Service. - Reduces the amount the customer has spent. :::note The `getCustomerInfo` endpoint works as a participant service endpoint when receiving a transaction ID from the coordinator. ::: ## Configuration for ScalarDB Cluster [The configuration for ScalarDB Cluster for the Customer Service](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/scalardb-cluster-node-for-customer-service.properties) is as follows: ```properties scalar.db.storage=multi-storage scalar.db.multi_storage.storages=cassandra,mysql scalar.db.multi_storage.storages.cassandra.storage=cassandra scalar.db.multi_storage.storages.cassandra.contact_points=cassandra-1 scalar.db.multi_storage.storages.cassandra.username=cassandra scalar.db.multi_storage.storages.cassandra.password=cassandra scalar.db.multi_storage.storages.mysql.storage=jdbc scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://mysql-1:3306/ scalar.db.multi_storage.storages.mysql.username=root scalar.db.multi_storage.storages.mysql.password=mysql scalar.db.multi_storage.namespace_mapping=customer_service:mysql,coordinator:cassandra scalar.db.multi_storage.default_storage=mysql scalar.db.consensus_commit.isolation_level=SERIALIZABLE scalar.db.cluster.node.standalone_mode.enabled=true scalar.db.sql.enabled=true # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` - `scalar.db.sql.connection_mode`: This configuration decides how to connect to ScalarDB. - `scalar.db.storage`: Specifying `multi-storage` is necessary to use Multi-storage Transactions in ScalarDB. - `scalar.db.multi_storage.storages`: Your storage names must be defined here. - `scalar.db.multi_storage.storages.cassandra.*`: These configurations are for the `cassandra` storage, which is one of the storage names defined in `scalar.db.multi_storage.storages`. You can configure all the `scalar.db.*` properties for the `cassandra` storage here. - `scalar.db.multi_storage.storages.mysql.*`: These configurations are for the `mysql` storage, which is one of the storage names defined in `scalar.db.multi_storage.storages`. You can configure all the `scalar.db.*` properties for the `mysql` storage here. - `scalar.db.multi_storage.namespace_mapping`: This configuration maps the namespaces to the storage. In this sample application, operations for `customer_service` namespace tables are mapped to the `mysql` storage and operations for `order_service` namespace tables are mapped to the `cassandra` storage. You can also define which storage is mapped for the `coordinator` namespace that is used in Consensus Commit transactions. - `scalar.db.multi_storage.default_storage`: This configuration sets the default storage that is used for operations on unmapped namespace tables. - `scalar.db.sql.default_transaction_mode`: Specifying `two_phase_commit_transaction` is necessary to use Two-Phase Commit Transactions mode in ScalarDB. - `scalar.db.consensus_commit.isolation_level`: This configuration decides the isolation level used for ConsensusCommit. For details, see [Multi-Storage Transactions](../../multi-storage-transactions.mdx). [The configuration for ScalarDB Cluster for the Order Service](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/scalardb-cluster-node-for-order-service.properties) is as follows: ```properties scalar.db.storage=cassandra scalar.db.contact_points=cassandra-1 scalar.db.username=cassandra scalar.db.password=cassandra scalar.db.consensus_commit.isolation_level=SERIALIZABLE scalar.db.cluster.node.standalone_mode.enabled=true scalar.db.sql.enabled=true # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` - `scalar.db.storage`: `cassandra` is specified since this servcise uses only Cassandra as an underlying database. - `scalar.db.contact_points`: This configuration specifies the contact points (e.g., host) for connecting to Cassandra. - `scalar.db.username`: This configuration specifies the username for connecting to Cassandra. - `scalar.db.password`: This configuration specifies the password for connecting to Cassandra. - `scalar.db.sql.default_namespace_name`: This configuration sets the default namespace to `order_service`, eliminating the need for the application to specify namespaces. - `scalar.db.sql.default_transaction_mode`: Specifying `two_phase_commit_transaction` is necessary to use Two-Phase Commit Transactions mode in ScalarDB. - `scalar.db.consensus_commit.isolation_level`: This configuration decides the isolation level used for ConsensusCommit. In this sample application, the ScalarDB Clusters are running in standalone mode (`scalar.db.cluster.node.standalone_mode.enabled=true`). Also, you need to set the license key (trial license or commercial license) for the ScalarDB Clusters in the configuration file. For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). ## Setup ### Clone the ScalarDB samples repository Open Terminal, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory with this sample by running the following command: ```console cd scalardb-samples/spring-data-microservice-transaction-sample ``` ### Set the license key Set the license key (trial license or commercial license) for the ScalarDB Clusters in the configuration files [`scalardb-cluster-node-for-customer-service.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/scalardb-cluster-node-for-customer-service.properties) and [`scalardb-cluster-node-for-order-service.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/scalardb-cluster-node-for-order-service.properties). For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). ### Start Cassandra, MySQL, and ScalarDB Clusters To start Cassandra, MySQL, and ScalarDB Clusters, you need to run the following `docker-compose` command: ```console docker-compose up -d cassandra mysql scalardb-cluster-node-for-customer-service scalardb-cluster-node-for-order-service ``` :::note You will need to wait more than one minute for the containers to be fully started. ::: ### Load schema The database schema (the method in which the data will be organized) for the sample application has already been defined in [`schema-for-customer-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-customer-service.sql) for the Customer Service and [`schema-for-order-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-order-service.sql) for the Order Service. To apply the schema, go to the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page and download the SQL CLI tool that matches the version of ScalarDB that you want to use. #### MySQL To load the schema for [`schema-for-customer-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-customer-service.sql) into MySQL, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: ```console java -jar scalardb-cluster-sql-cli--all.jar --config scalardb-cluster-node-for-customer-service-client.properties --file schema-for-customer-service.sql ``` #### Cassandra To load the schema for [`schema-for-order-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-order-service.sql) into Cassandra, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: ```console java -jar scalardb-cluster-sql-cli--all.jar --config scalardb-cluster-node-for-order-service-client.properties --file schema-for-order-service.sql ``` #### Schema details As shown in [`schema-for-customer-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-customer-service.sql), all the tables for the Customer Service are created in the `customer_service` namespace. - `customer_service.customers`: a table that manages customers' information - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a line of credit - `credit_total`: the amount of money that each customer has already spent by using their line of credit As shown in [`schema-for-order-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-order-service.sql), all the tables for the Order Service are created in the `order_service` namespace. - `order_service.orders`: a table that manages order information - `order_service.statements`: a table that manages order statement information - `order_service.items`: a table that manages information of items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/ERD.png) ### Start Microservices First, you need to build the docker images of the sample application with the following command: ```console ./gradlew docker ``` Then, you can start the microservices with the following `docker-compose` command: ```console docker-compose up -d customer-service order-service ``` ### Initial data When the microservices start, the initial data is loaded automatically. After the initial data has loaded, the following records should be stored in the tables: - For the `customer_service.customers` table: | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | - For the `order_service.items` table: | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Run the sample application Let's start with getting information about the customer whose ID is `1`: ```console ./gradlew :client:run --args="GetCustomerInfo 1" ... {"id": 1,"name": "Yamada Taro","credit_limit": 10000} ... ``` At this time, `credit_total` isn't shown, which means the current value of `credit_total` is `0`. Then, place an order for three apples and two oranges by using customer ID `1`. Note that the order format is `:,:,...`: ```console ./gradlew :client:run --args="PlaceOrder 1 1:3,2:2" ... {"order_id": "415a453b-cfee-4c48-b8f6-d103d3e10bdb"} ... ``` You can see that running this command shows the order ID. Let's check the details of the order by using the order ID: ```console ./gradlew :client:run --args="GetOrder 415a453b-cfee-4c48-b8f6-d103d3e10bdb" ... {"order": {"order_id": "415a453b-cfee-4c48-b8f6-d103d3e10bdb","timestamp": 1686555272435,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": $ ,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} ... ``` Then, let's place another order and get the order history of customer ID `1`: ```console ./gradlew :client:run --args="PlaceOrder 1 5:1" ... {"order_id": "069be075-98f7-428c-b2e0-6820693fc41b"} ... ./gradlew :client:run --args="GetOrders 1" ... {"order": [{"order_id": "069be075-98f7-428c-b2e0-6820693fc41b","timestamp": 1686555279366,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000},{"order_id": "415a453b-cfee-4c48-b8f6-d103d3e10bdb","timestamp": 1686555272435,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}]} ... ``` This order history is shown in descending order by timestamp. The customer's current `credit_total` is `10000`. Since the customer has now reached their `credit_limit`, which was shown when retrieving their information, they cannot place anymore orders. ```console ./gradlew :client:run --args="GetCustomerInfo 1" ... {"id": 1,"name": "Yamada Taro","credit_limit": 10000,"credit_total": 10000} ... ./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" ... io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Credit limit exceeded. creditTotal:10000, payment:7500 at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) at sample.rpc.OrderServiceGrpc$OrderServiceBlockingStub.placeOrder(OrderServiceGrpc.java:296) at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:38) at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:12) at picocli.CommandLine.executeUserObject(CommandLine.java:2041) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) at picocli.CommandLine.execute(CommandLine.java:2170) at sample.client.Client.main(Client.java:39) ... ``` After making a payment, the customer will be able to place orders again. ```console ./gradlew :client:run --args="Repayment 1 8000" ... ./gradlew :client:run --args="GetCustomerInfo 1" ... {"id": 1,"name": "Yamada Taro","credit_limit": 10000,"credit_total": 2000} ... ./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" ... {"order_id": "b6adabd8-0a05-4109-9618-3420fea3161f"} ... ``` ## Clean up To stop Cassandra, MySQL and the Microservices, run the following command: ```console docker-compose down ``` ## Reference - How the microservice transaction is achieved The transactions for placing an order, getting a single order, and getting the history of orders achieve the microservice transaction. This section focuses on how the transactions that span the Customer Service and the Order Service are implemented by placing an order as an example. The following sequence diagram shows the transaction for placing an order: ![Sequence Diagram](images/sequence_diagram.png) ### 1. Transaction with a two-phase commit interface is started When a client sends a request to place an order to the Order Service, `OrderService.placeOrder()` is called, and the microservice transaction starts. At first, the Order Service starts a transaction with a two-phase commit interface with `ScalarDbTwoPcRepository.executeTwoPcTransaction()` as follows. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). ```java // Start a two-phase commit interface transaction TwoPcResult result = orderRepository.executeTwoPcTransaction(txId -> { ... }, ...); ``` The actions in [CRUD operations are executed](#2-crud-operations-are-executed), [Transaction is committed by using the two-phase commit protocol](#3-transaction-is-committed-by-using-the-two-phase-commit-protocol), and [Error handling](#error-handling) are automatically performed by the API. ### 2. CRUD operations are executed After the transaction with a two-phase commit interface starts, CRUD operations are executed by `ScalarDbTwoPcRepository.executeTwoPcTransaction()`. The Order Service puts the order information in the `order_service.orders` table and the detailed information in the `order_service.statements` table as follows. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). ```java // Put the order info into the `orders` table orderRepository.insert(order); AtomicInteger amount = new AtomicInteger(); for (ItemOrder itemOrder : request.getItemOrderList()) { int itemId = itemOrder.getItemId(); int count = itemOrder.getCount(); // Retrieve the item info from the `items` table Optional itemOpt = itemRepository.findById(itemId); if (!itemOpt.isPresent()) { String message = "Item not found: " + itemId; responseObserver.onError( Status.NOT_FOUND.withDescription(message).asRuntimeException()); throw new ScalarDbNonTransientException(message); } Item item = itemOpt.get(); int cost = item.price * count; // Put the order statement into the `statements` table statementRepository.insert(new Statement(itemId, orderId, count)); // Calculate the total amount amount.addAndGet(cost); } ``` Then, the Order Service calls the `payment` gRPC endpoint of the Customer Service along with the transaction ID. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). ```java customerServiceStub.payment( PaymentRequest.newBuilder() .setTransactionId(transactionId) .setCustomerId(customerId) .setAmount(amount) .build()); ``` The `payment` endpoint of the Customer Service first joins the transaction with `ScalarDbTwoPcRepository.joinTransactionOnParticipant()` as follows. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java customerRepository.joinTransactionOnParticipant(request.getTransactionId(), ...); ``` The endpoint then gets the customer information and checks if the customer's credit total exceeds the credit limit after the payment. If the credit total does not exceed the credit limit, the endpoint updates the customer's credit total. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java Customer customer = getCustomer(responseObserver, request.getCustomerId()); int updatedCreditTotal = customer.creditTotal + request.getAmount(); // Check if the credit total exceeds the credit limit after payment if (updatedCreditTotal > customer.creditLimit) { String message = String.format( "Credit limit exceeded. creditTotal:%d, payment:%d", customer.creditTotal, request.getAmount()); responseObserver.onError( Status.FAILED_PRECONDITION.withDescription(message).asRuntimeException()); throw new ScalarDbNonTransientException(message); } // Reduce `credit_total` for the customer customerRepository.update(customer.withCreditTotal(updatedCreditTotal)); ``` ### 3. Transaction is committed by using the two-phase commit protocol After the Order Service receives the update that the payment succeeded, the Order Service tries to commit the transaction. The `ScalarDbTwoPcRepository.executeTwoPcTransaction()` API, which called on the Order Service, automatically performs preparations, validations, and commits of both the local Order Service and the remote Customer Service. These steps are executed sequentially after the above CRUD operations successfully finish. The implementations to invoke `prepare`, `validate`, and `commit` gRPC endpoints of the Customer Service need to be passed as parameters to the API. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). ```java TwoPcResult result = orderRepository.executeTwoPcTransaction(txId ->{ ... }, Collections.singletonList( RemotePrepareCommitPhaseOperations.createSerializable( this::callPrepareEndpoint, this::callValidateEndpoint, this::callCommitEndpoint, this::callRollbackEndpoint ) ) ); ``` ![Sequence Diagram of High Level 2PC API](images/seq-diagram-high-level-2pc-api.png) In the `prepare` endpoint of the Customer Service, the endpoint resumes and prepares the transaction by using `ScalarDbTwoPcRepository.prepareTransactionOnParticipant()`. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java customerRepository.prepareTransactionOnParticipant(request.getTransactionId()); ``` In the `validate` endpoint of the Customer Service, the endpoint resumes and validates the transaction by using `ScalarDbTwoPcRepository.validateTransactionOnParticipant()`. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java customerRepository.validateTransactionOnParticipant(request.getTransactionId()); ``` In the `commit` endpoint of the Customer Service, the endpoint resumes and commits the transaction by using `ScalarDbTwoPcRepository.commitTransactionOnParticipant()`. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java customerRepository.commitTransactionOnParticipant(request.getTransactionId()); ``` ### Error handling If an error happens while executing a transaction, `ScalarDbTwoPcRepository.executeTwoPcTransaction()` will automatically roll back the transaction in both the local Order Service and the remote Customer Service. The implementation to invoke the `rollback` gRPC endpoint of the Customer Service also needs to be passed as a parameter to the API with other ones. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). ```java TwoPcResult result = orderRepository.executeTwoPcTransaction(txId ->{ ... }, Collections.singletonList( RemotePrepareCommitPhaseOperations.createSerializable( this::callPrepareEndpoint, this::callValidateEndpoint, this::callCommitEndpoint, this::callRollbackEndpoint ) ) ); ``` In the `rollback` endpoint of the Customer Service, the endpoint resumes and rolls back the transaction. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). ```java customerRepository.rollbackTransactionOnParticipant(request.getTransactionId()); ``` For details about how to handle exceptions in ScalarDB, see [How to handle exceptions](../../two-phase-commit-transactions.mdx#how-to-handle-exceptions). ================================================ FILE: docs/scalardb-samples/spring-data-multi-storage-transaction-sample/README.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Sample application of Spring Data JDBC for ScalarDB with Multi-storage Transactions import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; import JDKVersions from '/src/components/en-us/_prerequisites-jdk-versions.mdx'; This tutorial describes how to create a sample Spring Boot application by using Spring Data JDBC for ScalarDB with Multi-storage Transactions. ## Prerequisites for this sample application - One of the following Java Development Kits (JDKs): - [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later ## Sample application ### Overview This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using the [multi-storage transactions](../../multi-storage-transactions.mdx) feature in ScalarDB. :::note Application-specific error handling, authentication processing, etc. are omitted in the sample application since this tutorial focuses on explaining how to use Spring Data JDBC for ScalarDB with multi-storage transactions. For details, see [Guide of Spring Data JDBC for ScalarDB](../../scalardb-sql/spring-data-guide.mdx). ::: ![Overview](images/overview.png) The application accesses the databases through ScalarDB Cluster. ### Schema [The schema](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-multi-storage-transaction-sample/schema.sql) is as follows: ```sql CREATE COORDINATOR TABLES IF NOT EXIST; CREATE NAMESPACE IF NOT EXISTS customer; CREATE TABLE IF NOT EXISTS customer.customers ( customer_id INT PRIMARY KEY, name TEXT, credit_limit INT, credit_total INT ); CREATE NAMESPACE IF NOT EXISTS "order"; CREATE TABLE IF NOT EXISTS "order".orders ( customer_id INT, timestamp BIGINT, order_id TEXT, PRIMARY KEY (customer_id, timestamp) ); CREATE INDEX IF NOT EXISTS ON "order".orders (order_id); CREATE TABLE IF NOT EXISTS "order".statements ( order_id TEXT, item_id INT, count INT, PRIMARY KEY (order_id, item_id) ); CREATE TABLE IF NOT EXISTS "order".items ( item_id INT PRIMARY KEY, name TEXT, price INT ); ``` All the tables are created in the `customer` and `order` namespaces. - `customer.customers`: a table that manages customers' information - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a credit card - `credit_total`: the amount of money that each customer has already spent by using the credit card - `order.orders`: a table that manages order information - `order.statements`: a table that manages order statement information - `order.items`: a table that manages information of items to be ordered The Entity Relationship Diagram for the schema is as follows: ![ERD](images/ERD.png) ### Transactions The following five transactions are implemented in this sample application: 1. Getting customer information 2. Placing an order by credit card (checks if the cost of the order is below the credit limit, then records order history and updates the `credit_total` if the check passes) 3. Getting order information by order ID 4. Getting order information by customer ID 5. Repayment (reduces the amount in the `credit_total`) ## Configuration for ScalarDB Cluster [The configuration for ScalarDB Cluster](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-multi-storage-transaction-sample/scalardb-cluster-node.properties) are as follows: ```properties scalar.db.storage=multi-storage scalar.db.multi_storage.storages=cassandra,mysql scalar.db.multi_storage.storages.cassandra.storage=cassandra scalar.db.multi_storage.storages.cassandra.contact_points=cassandra-1 scalar.db.multi_storage.storages.cassandra.username=cassandra scalar.db.multi_storage.storages.cassandra.password=cassandra scalar.db.multi_storage.storages.mysql.storage=jdbc scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://mysql-1:3306/ scalar.db.multi_storage.storages.mysql.username=root scalar.db.multi_storage.storages.mysql.password=mysql scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra scalar.db.multi_storage.default_storage=cassandra scalar.db.cluster.node.standalone_mode.enabled=true scalar.db.sql.enabled=true # License key configurations scalar.db.cluster.node.licensing.license_key= scalar.db.cluster.node.licensing.license_check_cert_pem= ``` - `scalar.db.storage`: Specifying `multi-storage` is necessary to use Multi-storage Transactions in ScalarDB. - `scalar.db.multi_storage.storages`: Your storage names must be defined here. - `scalar.db.multi_storage.storages.cassandra.*`: These configurations are for the `cassandra` storage, which is one of the storage names defined in `scalar.db.multi_storage.storages`. You can configure all the `scalar.db.*` properties for the `cassandra` storage here. - `scalar.db.multi_storage.storages.mysql.*`: These configurations are for the `mysql` storage, which is one of the storage names defined in `scalar.db.multi_storage.storages`. You can configure all the `scalar.db.*` properties for the `mysql` storage here. - `scalar.db.multi_storage.namespace_mapping`: This configuration maps the namespaces to the storage. In this sample application, operations for `customer` namespace tables are mapped to the `mysql` storage and operations for `order` namespace tables are mapped to the `cassandra` storage. You can also define which storage is mapped for the `coordinator` namespace that is used in Consensus Commit transactions. - `scalar.db.multi_storage.default_storage`: This configuration sets the default storage that is used for operations on unmapped namespace tables. For details, see [Multi-Storage Transactions](../../multi-storage-transactions.mdx). In this sample application, ScalarDB Cluster is running in standalone mode (`scalar.db.cluster.node.standalone_mode.enabled=true`). Also, you need to set the license key (trial license or commercial license) for ScalarDB Cluster in the configuration file. For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). ## Client Configuration [The client configuration](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-multi-storage-transaction-sample/scalardb-sql.properties) is as follows: ```properties scalar.db.sql.connection_mode=cluster scalar.db.sql.cluster_mode.contact_points=indirect:localhost ``` ## Setup ### Clone the ScalarDB samples repository Open Terminal, then clone the ScalarDB samples repository by running the following command: ```console git clone https://github.com/scalar-labs/scalardb-samples ``` Then, go to the directory with this sample by running the following command: ```console cd scalardb-samples/spring-data-multi-storage-transaction-sample ``` ### Set the license key Set the license key (trial license or commercial license) for the ScalarDB Clusters in the configuration file [`scalardb-cluster-node.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-multi-storage-transaction-sample/scalardb-cluster-node.properties). For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). ### Start Cassandra, MySQL, and ScalarDB Cluster To start Cassandra, MySQL, and ScalarDB Cluster, you need to run the following `docker-compose` command: ```console docker-compose up -d ``` Please note that starting the containers may take more than one minute. ### Load schema You then need to apply the schema with the following command. To download the SQL CLI tool, `scalardb-cluster-sql-cli--all.jar`, see the [Releases](https://github.com/scalar-labs/scalardb/releases) of ScalarDB and download the version that you want to use. ```console java -jar scalardb-cluster-sql-cli--all.jar --config scalardb-sql.properties --file schema.sql ``` ### Load initial data After the containers have started, you need to load the initial data by running the following command: ```console ./gradlew run --args="LoadInitialData" ``` After the initial data has loaded, the following records should be stored in the tables: - For the `customer.customers` table: | customer_id | name | credit_limit | credit_total | |-------------|---------------|--------------|--------------| | 1 | Yamada Taro | 10000 | 0 | | 2 | Yamada Hanako | 10000 | 0 | | 3 | Suzuki Ichiro | 10000 | 0 | - For the `order.items` table: | item_id | name | price | |---------|--------|-------| | 1 | Apple | 1000 | | 2 | Orange | 2000 | | 3 | Grape | 2500 | | 4 | Mango | 5000 | | 5 | Melon | 3000 | ## Run the sample application Let's start with getting information about the customer whose ID is `1`: ```console ./gradlew run --args="GetCustomerInfo 1" ... {"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":0} ... ``` Then, place an order for three apples and two oranges by using customer ID `1`. Note that the order format is `:,:,...`: ```console ./gradlew run --args="PlaceOrder 1 1:3,2:2" ... {"order_id":"5d49eb62-fcb9-4dd2-9ae5-e714d989937f","customer_id":1,"timestamp":1677564659810} ... ``` You can see that running this command shows the order ID. Let's check the details of the order by using the order ID: ```console ./gradlew run --args="GetOrder 5d49eb62-fcb9-4dd2-9ae5-e714d989937f" ... {"order_id":"5d49eb62-fcb9-4dd2-9ae5-e714d989937f","timestamp":1677564659810,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":1,"item_name":"Apple","price":1000,"count":3,"total":3000},{"item_id":2,"item_name":"Orange","price":2000,"count":2,"total":4000}],"total":7000} ... ``` Then, let's place another order and get the order history of customer ID `1`: ```console ./gradlew run --args="PlaceOrder 1 5:1" ... {"order_id":"ccd97d75-ee57-4393-a0bb-5230c4a8c68a","customer_id":1,"timestamp":1677564776069} ... ./gradlew run --args="GetOrders 1" ... [{"order_id":"ccd97d75-ee57-4393-a0bb-5230c4a8c68a","timestamp":1677564776069,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":5,"item_name":"Melon","price":3000,"count":1,"total":3000}],"total":3000},{"order_id":"5d49eb62-fcb9-4dd2-9ae5-e714d989937f","timestamp":1677564659810,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":1,"item_name":"Apple","price":1000,"count":3,"total":3000},{"item_id":2,"item_name":"Orange","price":2000,"count":2,"total":4000}],"total":7000}] ... ``` This order history is shown in descending order by timestamp. The customer's current `credit_total` is `10000`. Since the customer has now reached their `credit_limit`, which was shown when retrieving their information, they cannot place anymore orders. ```console ./gradlew run --args="GetCustomerInfo 1" ... {"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":10000} ... ./gradlew run --args="PlaceOrder 1 3:1,4:1" ... java.lang.RuntimeException: Credit limit exceeded. limit:10000, total:17500 at sample.SampleService.placeOrder(SampleService.java:102) at sample.SampleService$$FastClassBySpringCGLIB$$1123c447.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at sample.SampleService$$EnhancerBySpringCGLIB$$1cb0cc8c.placeOrder() at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:37) at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:13) at picocli.CommandLine.executeUserObject(CommandLine.java:2041) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) at picocli.CommandLine.execute(CommandLine.java:2170) at sample.SampleApp.run(SampleApp.java:26) at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768) at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752) at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292) at sample.SampleApp.main(SampleApp.java:35) ... ``` After making a payment, the customer will be able to place orders again. ```console ./gradlew run --args="Repayment 1 8000" ... ./gradlew run --args="GetCustomerInfo 1" ... {"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":2000} ... ./gradlew run --args="PlaceOrder 1 3:1,4:1" ... {"order_id":"3ac4a1bf-a724-4f26-b948-9f03281a971e","customer_id":1,"timestamp":1677565028204} ... ``` ## Cleanup To stop Cassandra, MySQL, and ScalarDB Cluster, run the following command: ```console docker-compose down ``` ================================================ FILE: docs/scalardb-sql/index.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB SQL Overview ScalarDB SQL is an interface layer that allows client applications to communicate with ScalarDB Cluster by using SQL. :::note ScalarDB SQL is not fully compatible with standard SQL, but it offers a large subset of the SQL language. ::: ## Types of SQL interfaces ScalarDB SQL has three types of SQL interfaces. ### JDBC The JDBC interface lets you connect to ScalarDB Cluster by using the standard JDBC API. This is useful for applications that already use JDBC. For details on how to set up and use the JDBC interface, see the [ScalarDB JDBC Guide](./jdbc-guide.mdx). ### SQL API The SQL API lets you connect to ScalarDB Cluster by using the proprietary and modern Java SQL API. This is useful for applications that do not need to rely on the JDBC interface. For details on how to set up and use the SQL API, see the [ScalarDB SQL API Guide](./sql-api-guide.mdx). ### Spring Data JDBC The Spring Data JDBC interface lets you interact with ScalarDB Cluster via Spring Data JDBC repositories and entities. This is useful for applications that already use Spring Data or when you want to integrate ScalarDB Cluster into Spring applications. For details on how to set up and use the Sprign Data JDBC interface, see the [Guide of Spring Data JDBC for ScalarDB](./spring-data-guide.mdx). ================================================ FILE: docs/scalardb-sql/jdbc-guide.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB JDBC Guide The usage of ScalarDB JDBC basically follows [Java JDBC API](https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/). This guide describes several important topics that are specific to ScalarDB JDBC. ## Add ScalarDB JDBC driver to your project To add the dependencies for the ScalarDB JDBC driver by using Gradle, use the following, replacing `` with the versions of the ScalarDB JDBC driver and the related library, respectively, that you are using: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-sql-jdbc:' implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:' } ``` To add the dependencies by using Maven, use the following, replacing `...` with the version of the ScalarDB JDBC driver that you are using: ```xml com.scalar-labs scalardb-sql-jdbc ... com.scalar-labs scalardb-cluster-java-client-sdk ... ``` ## JDBC connection URL The JDBC connection URL format of ScalarDB JDBC is as follows: ```console jdbc:scalardb:?=&=&... ``` For example: Only specify configuration file path: ```console jdbc:scalardb:/path/to/database.properties ``` Only specify properties: ```console jdbc:scalardb:?scalar.db.contact_points=localhost&scalar.db.username=cassandra&scalar.db.password=cassandra&scalar.db.storage=cassandra ``` Specify configuration file path and properties: ```console jdbc:scalardb:/path/to/database.properties?scalar.db.metadata.cache_expiration_time_secs=0 ``` ## Configurations for ScalarDB JDBC Please see [ScalarDB Cluster SQL client configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations) for details on the configurations. In addition, the ScalarDB JDBC specific configurations are as follows: | name | description | default | |---------------------------------------------------------------------|-----------------------------------------------------------------------------|---------| | scalar.db.sql.jdbc.default_auto_commit | The default auto-commit mode for connections. | true | | scalar.db.sql.jdbc.default_read_only | The default read-only state for connections. | false | | scalar.db.sql.jdbc.sql_session_factory_cache.expiration_time_millis | The expiration time in milliseconds for the cache of SQL session factories. | 10000 | ## Data type mapping between ScalarDB and JDBC Since ScalarDB doesn't support all the data types defined in JDBC, the following explains the data type mapping between ScalarDB and JDBC. The data type mapping between ScalarDB and JDBC is as follows: | ScalarDB Type | JDBC (Java) Type | |---------------|-------------------------| | BOOLEAN | boolean or Boolean | | INT | int or Integer | | BIGINT | long or Long | | FLOAT | float or Float | | DOUBLE | double or Double | | TEXT | String | | BLOB | byte[] or java.sql.Blob | | DATE | java.time.LocalDate | | TIME | java.time.LocalTime | | TIMESTAMP | java.time.LocalDateTime | | TIMESTAMPTZ | java.time.Instant | For BLOB columns, `java.io.InputStream` is also accepted when writing. See the `PreparedStatement` example below. How to get the data from a `java.sql.ResultSet` object for each data type is as follows: ```java try (ResultSet resultSet = ...) { resultSet.next(); // Get a BOOLEAN value of a column boolean booleanValue = resultSet.getBoolean(""); // Get an INT value of a column int intValue = resultSet.getInt(""); // Get a BIGINT value of a column long bigIntValue = resultSet.getLong(""); // Get a FLOAT value of a column float floatValue = resultSet.getFloat(""); // Get a DOUBLE value of a column double doubleValue = resultSet.getDouble(""); // Get a TEXT value of a column String textValue = resultSet.getString(""); // Get a BLOB value of a column as a byte array byte[] blobBytes = resultSet.getBytes(""); // Get a BLOB value of a column as a java.sql.Blob // (this is also what resultSet.getObject returns for BLOB columns) Blob blob = resultSet.getBlob(""); // Get a BLOB value of a column as a stream InputStream blobStream = resultSet.getBinaryStream(""); // Get a DATE value of a column LocalDate dateValue = resultSet.getObject("", LocalDate.class); // Get a TIME value of a column LocalTime timeValue = resultSet.getObject("", LocalTime.class); // Get a TIMESTAMP value of a column LocalDateTime timestampValue = resultSet.getObject("", LocalDateTime.class); // Get a TIMESTAMPTZ value of a column Instant timestampTZValue = resultSet.getObject("", Instant.class); } ``` How to set the data as a parameter for each data type for a `java.sql.PreparedStatement` object is as follows: ```java try (PreparedStatement preparedStatement = ...) { // Set a BOOLEAN value as parameter preparedStatement.setBoolean(1, ); // Set an INT value as parameter preparedStatement.setInt(2, ); // Set a BIGINT value as parameter preparedStatement.setLong(3, ); // Set a FLOAT value as parameter preparedStatement.setFloat(4, ); // Set a DOUBLE value as parameter preparedStatement.setDouble(5, ); // Set a TEXT value as parameter preparedStatement.setString(6, ""); // Set a BLOB value as parameter (byte array) preparedStatement.setBytes(7, ); // Alternatively, if you have a java.sql.Blob (for example, one obtained from ResultSet.getBlob), // use preparedStatement.setBlob(7, ); // Set a DATE value as parameter preparedStatement.setObject(8, ); // Set a TIME value as parameter preparedStatement.setObject(9, ); // Set a TIMESTAMP value as parameter preparedStatement.setObject(10, ); // Set a TIMESTAMPTZ value as parameter preparedStatement.setObject(11, ); preparedStatement.execute(); } ``` For BLOB columns, the driver also supports writing directly from a stream. The following overloads are available in addition to `setBytes(int, byte[])` and `setBlob(int, java.sql.Blob)`: - `setBlob(int, java.io.InputStream)` and `setBlob(int, java.io.InputStream, long)`, which reads bytes from the stream (the overload with a length reads exactly that many bytes). - `setBinaryStream(int, java.io.InputStream)`, `setBinaryStream(int, java.io.InputStream, int)`, and `setBinaryStream(int, java.io.InputStream, long)`, which follow the same pattern. ## Execute batch statements The ScalarDB JDBC driver supports batch execution for mutation statements, following standard JDBC conventions. Use this when you need to apply many inserts, updates, or deletes in a single round trip. Using `java.sql.Statement`: ```java try (Statement statement = connection.createStatement()) { statement.addBatch("INSERT INTO tbl (c1, c2) VALUES (1, 'a')"); statement.addBatch("UPDATE tbl SET c2 = 'b' WHERE c1 = 2"); statement.addBatch("DELETE FROM tbl WHERE c1 = 3"); int[] updateCounts = statement.executeBatch(); } ``` Using `java.sql.PreparedStatement`: ```java try (PreparedStatement preparedStatement = connection.prepareStatement("INSERT INTO tbl (c1, c2) VALUES (?, ?)")) { preparedStatement.setInt(1, 1); preparedStatement.setString(2, "a"); preparedStatement.addBatch(); preparedStatement.setInt(1, 2); preparedStatement.setString(2, "b"); preparedStatement.addBatch(); int[] updateCounts = preparedStatement.executeBatch(); } ``` Batch execution is restricted to mutation statements (`INSERT`, `UPSERT`, `UPDATE`, and `DELETE`). Passing a `SELECT`, DDL, or DCL statement to `addBatch(...)` causes the subsequent `executeBatch()` call to throw a `SQLException`. If one or more statements in a batch fail, `executeBatch()` throws a `java.sql.BatchUpdateException`. You can inspect the per-statement update counts by calling `BatchUpdateException.getUpdateCounts()`. ## Handle SQLException The exception handling is basically the same as ScalarDB SQL API as follows: ```java // If you execute multiple statements in a transaction, you need to set auto-commit to false. connection.setAutoCommit(false); try { // Execute statements (SELECT/INSERT/UPDATE/DELETE) in the transaction ... // Commit the transaction connection.commit(); } catch (SQLException e) { if (e.getErrorCode() == 301) { // The error code 301 indicates that you catch `UnknownTransactionStatusException`. // If you catch `UnknownTransactionStatusException`, it indicates that the status of the // transaction, whether it has succeeded or not, is unknown. In such a case, you need to check // if the transaction is committed successfully or not and retry it if it failed. How to // identify a transaction status is delegated to users } else { // For other cases, you can try retrying the transaction // Rollback the transaction connection.rollback(); // The cause of the exception can be `TransactionRetryableException` or the other // exceptions. For `TransactionRetryableException`, you can basically retry the transaction. // However, for the other exceptions, the transaction may still fail if the cause of the // exception is nontransient. For such a case, you need to limit the number of retries and // give up retrying } } ``` Please see also [ScalarDB SQL API Guide](sql-api-guide.mdx) for more details on exception handling. ## References - [Java JDBC API](https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/) - [ScalarDB SQL API Guide](sql-api-guide.mdx) - [Javadoc for ScalarDB JDBC](https://javadoc.io/doc/com.scalar-labs/scalardb-sql-jdbc/3.18.0/index.html) ================================================ FILE: docs/scalardb-sql/migration-guide.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # How to Migrate Your Applications and Databases into a ScalarDB-Based Environment This guide describes how to migrate your existing applications and relational databases into ScalarDB-based applications and ScalarDB-managed databases, respectively. ## Target audience The target audiences for this guide are application developers and database administrators. The purpose of this guide is to help you understand how to migrate your existing applications and databases and in what conditions. ## What you will learn - Requirements for the migration - Steps to migrate your application - Changes to make in your application for the migration ## Steps to migrate your application ```mermaid flowchart LR subgraph AM[After migration] direction TB C[Application - updated] -->|ScalarDB SQL| D[ScalarDB] --> E[Relational/JDBC databases - schema updated] end subgraph BM[Before migration] direction TB A[Application] ---->|SQL| B[Relational/JDBC databases] end BM===>AM ``` 1. Verify the items in the checklist. - See [Migration checklist](#migration-checklist) to confirm your database is migratable. 2. Migrate your application (if necessary). - ScalarDB provides selection, projection, and join operations with dedicated SQL grammar. Thus, some SQL statements in your application might have to be changed for ScalarDB SQL, for example, at a grammar level or a logic level like aggregation processing. - For details, see [How to migrate your application](#how-to-migrate-your-application). 3. Back up your database. - Although ScalarDB Schema Loader, which you will use to import your database, only changes the metadata of your database when importing it to ScalarDB, you must back it up to avoid unexpected accidents. - Follow the administration guide of your database. 4. Set up a ScalarDB environment. - Prepare a configuration file so that ScalarDB can access target databases. - For details about ScalarDB configurations, see [ScalarDB Configurations](../configurations.mdx). 5. Import your database to ScalarDB. - Prepare an import schema file that defines target schemas and tables. The schemas and tables will be mapped to ScalarDB namespaces and tables, respectively. Note that "schema" is a synonym for "database" in some database systems. - Run the ScalarDB Schema Loader with the import option, the ScalarDB configuration file that you created, and the schema file that you prepared. - For details on how to use Schema Loader, see [Run Schema Loader for importing existing tables](../schema-loader-import.mdx#run-schema-loader-for-importing-existing-tables). 6. Switch your application and check the behavior. - Now, you can switch your application to the ScalarDB-based application. ## Migration checklist Before starting the migration, check the following questions. If the answer to any of these questions is "No", you must address them before proceeding with the migration. - Are your target database and version one of the [supported relational databases (called JDBC databases in ScalarDB) and versions](../requirements.mdx#relational-databases)? - Do you have a fully privileged account that can manage the target database? For details, see [the general requirements](../database-configurations.mdx#general-requirements). - Do all target tables have primary keys? - Is the data type of each column supported in ScalarDB? For supported data types and how they are mapped to ScalarDB data types, see [Data-type mapping from JDBC databases to ScalarDB](../schema-loader-import.mdx#data-type-mapping-from-jdbc-databases-to-scalardb). - Do the functionality and grammar of the queries in your application comply with the [ScalarDB SQL specifications](./grammar.mdx)? Or, for non-compliant queries, can you re-write them for ScalarDB? For examples of re-writes, see [How to migrate your application](#how-to-migrate-your-application). - After migrating your applications and databases into ScalarDB applications and ScalarDB-managed databases, respectively, can you stop accessing the databases directly? In other words, is it acceptable for you to always access the databases through ScalarDB? ## How to migrate your application Depending on your application environment, you may need to migrate your application in the following three aspects: - Change connection settings. - Modify SQL statements based on the ScalarDB SQL grammar. - Modify application logic if there is no available SQL modification workaround. ### Change connection settings If your application is based on Java, you can use the ScalarDB JDBC driver when migrating. For details on how to add dependencies for the ScalarDB JDBC driver and rewrite the connection URL, see the [ScalarDB JDBC Guide](./jdbc-guide.mdx). If your application is not based on Java, you can connect ScalarDB and issue SQL via gRPC. For details, see [ScalarDB Cluster SQL gRPC API Guide](../scalardb-cluster/scalardb-cluster-sql-grpc-api-guide.mdx). ### Modify SQL statements You may need to change the SQL statements in your application due to the differences in SQL grammar. Typical examples are as follows. For more details, see [ScalarDB SQL Grammar](./grammar.mdx). - `JOIN` queries - ScalarDB supports only `JOIN` queries in the style of writing the table to be joined and the condition in the `FROM` clause. - The `JOIN` condition and filtering also have a few limitations. - You may need to rewrite the queries based on the above. You can choose application-level modification if your SQL queries are not compliant with the ScalarDB specifications. - `WHERE` clause - In ScalarDB, predicates must be an OR-wise of `AND` predicate lists (known as disjunctive normal form or DNF) or an AND-wise of `OR` predicate lists (known as conjunctive normal form or CNF). Thus, you may have to change the `WHERE` clause, but note that an arbitrary form of predicates can be changed to either DNF or CNF. - Similarly, if you use `IN` clauses, you will need to change them to either DNF or CNF. For `IN` clauses with sub-queries, see [Modify application logic](#modify-application-logic). - ScalarDB adopts a specification similar to that of the `LIKE` operator and the escape sequence of PostgreSQL and MySQL. If your database is neither PostgreSQL nor MySQL, you may need to change predicates with the `LIKE` operator. ### Modify application logic Although ScalarDB SQL does not provide some functionalities, such as aggregate queries and sub-queries, those queries can be modified to application-level implementations. Typical modification techniques are as follows: - Aggregate queries - For simple aggregate queries such as `count()` and `sum()` without the `GROUP BY` clause, you can use `SELECT` for the target records and then count the number of records or calculate the sum by using the results. - For `GROUP BY` aggregate queries, first use `SELECT` for all target records without the `GROUP BY` clause. Then, put result records into a multi-map data structure while categorizing them based on the columns specified in the `GROUP BY` clause, which should be used as keys of the multi-map. Finally, aggregate records for each key in the multi-map. For the multi-map, you can use libraries such as [Guava](https://github.com/google/guava). - Sub-queries - For sub-queries in the `IN` clause, first use `SELECT` for the records specified in the sub-queries, then add result values as `OR` predicates in the `WHERE` clause. - For other sub-queries, basically, you need to use `SELECT` for the records for each query, then join or filter results records in your application. - Read-modify-write by using a single update query - `UPDATE` queries may often have an expression like an increment or a decrement, for example, `UPDATE table SET a = a + 1 WHERE ...`. In ScalarDB, you need to use `SELECT` for a target record and then set the incremented value as a constant in a single transaction, just like `UPDATE table SET a = 5 WHERE ...`. ## Limitations Due to the difference in data types, ScalarDB will throw an error when writing data larger than the maximum size of the column in the underlying database, even if the size is acceptable for the ScalarDB data type. Conversely, in a few types, the data in the underlying database may be larger than the maximum size in ScalarDB. For details, see [Data-type mapping from JDBC databases to ScalarDB](../schema-loader-import.mdx#data-type-mapping-from-jdbc-databases-to-scalardb). ## References - [Supported Databases](../requirements.mdx#databases) - [ScalarDB SQL API Guide](./sql-api-guide.mdx) - [ScalarDB JDBC Guide](./jdbc-guide.mdx) - [ScalarDB SQL Grammar](./grammar.mdx) - [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx) ================================================ FILE: docs/scalardb-sql/scalardb-sql-status-codes.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB SQL Error Codes This page provides a list of error codes in ScalarDB SQL. ## Error code classes and descriptions | Class | Description | |:---------------|:-----------------------------------| | `DB-SQL-1xxxx` | Errors for the user error category | ## `DB-SQL-1xxxx` status codes The following are status codes and messages for the user error category. ### `DB-SQL-10000` **Message** ```markdown The namespace does not exist. Namespace: %s ``` ### `DB-SQL-10001` **Message** ```markdown The table does not exist. Table: %s ``` ### `DB-SQL-10002` **Message** ```markdown The column %s does not exist ``` ### `DB-SQL-10003` **Message** ```markdown The column does not exist. Table: %s; Column: %s ``` ### `DB-SQL-10004` **Message** ```markdown The column index is out of bounds. Index: %d; Size: %d ``` ### `DB-SQL-10005` **Message** ```markdown A positional bind marker is not allowed when binding named values ``` ### `DB-SQL-10006` **Message** ```markdown A named bind marker is not allowed when binding positional values ``` ### `DB-SQL-10007` **Message** ```markdown Cannot convert BLOB values to SQL. Please use a bind marker for a BLOB value and bind it separately ``` ### `DB-SQL-10008` **Message** ```markdown No namespace name has been specified. Set a default namespace name, or explicitly specify a namespace name ``` ### `DB-SQL-10009` **Message** ```markdown Zero bytes may not occur in string parameters ``` ### `DB-SQL-10010` **Message** ```markdown Mixing positional values and named values is not allowed ``` ### `DB-SQL-10011` **Message** ```markdown Preparing a transaction is supported only in two-phase commit transaction mode ``` ### `DB-SQL-10012` **Message** ```markdown Validating a transaction is supported only in two-phase commit transaction mode ``` ### `DB-SQL-10013` **Message** ```markdown The previous transaction is still in progress. Commit, roll back, or suspend the previous transaction first ``` ### `DB-SQL-10014` **Message** ```markdown This SQL session has already been closed ``` ### `DB-SQL-10015` **Message** ```markdown A transaction has not begun ``` ### `DB-SQL-10016` **Message** ```markdown The type %s is not supported ``` ### `DB-SQL-10017` **Message** ```markdown No connection mode implementations are found. Please add a connection mode implementation to the classpath ``` ### `DB-SQL-10018` **Message** ```markdown The connection mode is not specified, but multiple connection mode implementations are found. Specify one of the following connection modes: %s ``` ### `DB-SQL-10019` **Message** ```markdown The connection mode '%s' is not found. Specify one of the following connection modes: %s ``` ### `DB-SQL-10020` **Message** ```markdown Access denied: You need the %s privilege on the namespace %s to execute this operation ``` ### `DB-SQL-10021` **Message** ```markdown Access denied: You need the %s privilege on the table %s to execute this operation ``` ### `DB-SQL-10022` **Message** ```markdown Access denied: You can't grant the %s privilege because you don't have the same privilege on the table %s ``` ### `DB-SQL-10023` **Message** ```markdown Access denied: You can't grant the %s privilege because you don't have the same privilege on the namespace %s ``` ### `DB-SQL-10024` **Message** ```markdown Access denied: You can't revoke the %s privilege because you don't have the same privilege on the table %s ``` ### `DB-SQL-10025` **Message** ```markdown Access denied: You can't revoke the %s privilege because you don't have the same privilege on the namespace %s ``` ### `DB-SQL-10026` **Message** ```markdown Syntax error. Line %d:%d %s ``` ### `DB-SQL-10027` **Message** ```markdown Syntax error. Multiple PRIMARY KEYs specified (exactly one required) ``` ### `DB-SQL-10028` **Message** ```markdown Cannot grant the INSERT privilege if the user doesn't have the UPDATE privilege ``` ### `DB-SQL-10029` **Message** ```markdown Cannot grant the UPDATE privilege if the user doesn't have the INSERT privilege ``` ### `DB-SQL-10030` **Message** ```markdown Cannot grant the UPDATE privilege if the user doesn't have the SELECT privilege ``` ### `DB-SQL-10031` **Message** ```markdown Cannot grant the DELETE privilege if the user doesn't have the SELECT privilege ``` ### `DB-SQL-10032` **Message** ```markdown Cannot revoke the SELECT privilege if the user has the UPDATE privilege ``` ### `DB-SQL-10033` **Message** ```markdown Cannot revoke the SELECT privilege if the user has the DELETE privilege ``` ### `DB-SQL-10034` **Message** ```markdown Cannot revoke the INSERT privilege if the user has the UPDATE privilege ``` ### `DB-SQL-10035` **Message** ```markdown Cannot revoke the UPDATE privilege if the user has the INSERT privilege ``` ### `DB-SQL-10036` **Message** ```markdown A non-clustering-key column is specified in the CLUSTERING ORDER directive. Column: %s ``` ### `DB-SQL-10037` **Message** ```markdown The order of the columns in the CLUSTERING ORDER directive must match the order for the clustering key (%s must appear before %s) ``` ### `DB-SQL-10038` **Message** ```markdown The CLUSTERING ORDER is missing for the column %s ``` ### `DB-SQL-10039` **Message** ```markdown Empty SQL is specified ``` ### `DB-SQL-10040` **Message** ```markdown Multiple SQLs are not allowed ``` ### `DB-SQL-10041` **Message** ```markdown The column %s is ambiguous ``` ### `DB-SQL-10042` **Message** ```markdown The column %s cannot be specified in the %s clause. Only the columns in the table %s can be specified in the %s clause ``` ### `DB-SQL-10043` **Message** ```markdown An unbound bind marker is still in the escape character of the LIKE predicate for the column %s ``` ### `DB-SQL-10044` **Message** ```markdown The escape character must be a TEXT value for the LIKE predicate for the column %s ``` ### `DB-SQL-10045` **Message** ```markdown The value of the predicate must not be null unless the operator is 'IS NULL' or 'IS NOT NULL'. Predicate: %s ``` ### `DB-SQL-10046` **Message** ```markdown An unbound bind marker is still in the LIMIT clause ``` ### `DB-SQL-10047` **Message** ```markdown The LIMIT must be an INT value ``` ### `DB-SQL-10048` **Message** ```markdown Unmatched column names or values ``` ### `DB-SQL-10049` **Message** ```markdown The column %s is specified twice ``` ### `DB-SQL-10050` **Message** ```markdown All primary key columns must be specified in the INSERT or UPSERT statement ``` ### `DB-SQL-10051` **Message** ```markdown An unbound bind marker is still in the value of the column %s ``` ### `DB-SQL-10052` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a boolean value (BOOLEAN) is specified ``` ### `DB-SQL-10053` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a decimal number (INT or BIGINT) is specified ``` ### `DB-SQL-10054` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a floating point number (FLOAT or DOUBLE) is specified ``` ### `DB-SQL-10055` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a string (TEXT, DATE, TIME, TIMESTAMP, or TIMESTAMPTZ) is specified ``` ### `DB-SQL-10056` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a BOOLEAN value is specified ``` ### `DB-SQL-10057` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but an INT value is specified ``` ### `DB-SQL-10058` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a BIGINT value is specified ``` ### `DB-SQL-10059` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a FLOAT value is specified ``` ### `DB-SQL-10060` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a DOUBLE value is specified ``` ### `DB-SQL-10061` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a TEXT value is specified ``` ### `DB-SQL-10062` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a BLOB value is specified ``` ### `DB-SQL-10063` **Message** ```markdown RIGHT OUTER JOIN can only be specified as the first join ``` ### `DB-SQL-10064` **Message** ```markdown The JOIN predicate is not specified properly. Predicate: %s ``` ### `DB-SQL-10065` **Message** ```markdown The data types of the columns in the JOIN predicate do not match. Predicate: %s ``` ### `DB-SQL-10066` **Message** ```markdown The column %s is specified twice in the JOIN predicates. Predicates: %s ``` ### `DB-SQL-10067` **Message** ```markdown Either all primary key columns or an indexed column for the table %s must be specified in the JOIN predicates. Predicates: %s ``` ### `DB-SQL-10068` **Message** ```markdown Cannot issue mutation DML SQLs such as INSERT, UPDATE or DELETE with executeQuery() ``` ### `DB-SQL-10069` **Message** ```markdown Cannot issue SELECT SQLs with executeUpdate() ``` ### `DB-SQL-10070` **Message** ```markdown The TWO_PHASE_COMMIT_TRANSACTION mode is not supported in the current transaction manager ``` ### `DB-SQL-10071` **Message** ```markdown The encrypted column %s is not allowed in the %s clause ``` ### `DB-SQL-10072` **Message** ```markdown The user %s does not exist ``` ### `DB-SQL-10073` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a DATE value is specified ``` ### `DB-SQL-10074` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a TIME value is specified ``` ### `DB-SQL-10075` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a TIMESTAMP value is specified ``` ### `DB-SQL-10076` **Message** ```markdown Unmatched column type. The type of the column %s should be %s, but a TIMESTAMPTZ value is specified ``` ### `DB-SQL-10077` **Message** ```markdown The policy %s does not exist ``` ### `DB-SQL-10078` **Message** ```markdown Beginning a transaction in read-only mode is not supported in two-phase commit transaction mode ``` ### `DB-SQL-10079` **Message** ```markdown Starting a transaction in read-only mode is not supported in two-phase commit transaction mode ``` ### `DB-SQL-10080` **Message** ```markdown Cannot change read-only mode while a transaction is in progress ``` ### `DB-SQL-10081` **Message** ```markdown Wrong number of arguments for the %s function. Min: %d; max: %d ``` ### `DB-SQL-10082` **Message** ```markdown The column %s must appear in the GROUP BY clause ``` ### `DB-SQL-10083` **Message** ```markdown Unknown function: %s ``` ### `DB-SQL-10084` **Message** ```markdown The ordering %s must appear in the projections ``` ### `DB-SQL-10085` **Message** ```markdown The %s function does not support the data type: %s ``` ### `DB-SQL-10086` **Message** ```markdown Numeric overflow in the %s function for the column: %s ``` ### `DB-SQL-10087` **Message** ```markdown The %s function does not support the argument type: %s ``` ### `DB-SQL-10088` **Message** ```markdown The %s in the HAVING clause must appear in the SELECT projections ``` ### `DB-SQL-10089` **Message** ```markdown Invalid LIKE pattern: '%s'. Detail: %s ``` ================================================ FILE: docs/scalardb-sql/spring-data-guide.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # Guide of Spring Data JDBC for ScalarDB Directly using the ScalarDB API may be difficult because you need to write a lot of code and consider how and when to call the APIs (e.g., `rollback()` and `commit()`) for transactions. Since we assume most ScalarDB users develop their applications in Java, you can take advantage of the Spring Framework, which is one of the most popular application frameworks for developing in Java. By using Spring Data JDBC for ScalarDB, you can streamline development by using a familiar framework. ![Rough overall architecture of Spring Data JDBC for ScalarDB](images/spring_data_ingegration_overall_arch.png) The usage of Spring Data JDBC for ScalarDB basically follows [Spring Data JDBC - Reference Documentation](https://docs.spring.io/spring-data/jdbc/docs/3.0.x/reference/html/). This guide describes several important topics to use Spring Data JDBC for ScalarDB and its limitations. :::warning Spring Data JDBC for ScalarDB extends Spring Data JDBC, but full compatibility is not guaranteed. Only the features listed on this page are officially tested and supported. ::: ## Add Spring Data JDBC for ScalarDB to your project To add the dependencies on Spring Data JDBC for ScalarDB by using Gradle, use the following, replacing `` with the versions of Spring Data JDBC for ScalarDB and the related library, respectively, that you are using: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-sql-spring-data:' implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:' } ``` To add the dependencies by using Maven, use the following, replacing `...` with the version of Spring Data JDBC for ScalarDB that you are using: ```xml com.scalar-labs scalardb-sql-spring-data ... com.scalar-labs scalardb-cluster-java-client-sdk ... ``` ## Configurations Spring Data JDBC for ScalarDB is supposed to be used as part of a Spring application. Set the following properties. ### spring.datasource.driver-class-name This needs to be set to the fixed value `com.scalar.db.sql.jdbc.SqlJdbcDriver`. Always required. ```console spring.datasource.driver-class-name=com.scalar.db.sql.jdbc.SqlJdbcDriver ``` ### spring.datasource.url This value follows the ScalarDB JDBC connection URL configuration. For more information, see [ScalarDB JDBC Guide](jdbc-guide.mdx) and [ScalarDB Cluster SQL client configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations). Always required. ```console spring.datasource.url=jdbc:scalardb:\ ?scalar.db.sql.connection_mode=direct\ &scalar.db.contact_points=jdbc:mysql://localhost:3306/my_app_ns\ &scalar.db.username=root\ &scalar.db.password=mysql\ &scalar.db.storage=jdbc\ &scalar.db.consensus_commit.isolation_level=SERIALIZABLE\ &scalar.db.sql.default_namespace_name.existence_check.enabled=false ``` ### scalar.db.sql.default_namespace_name.existence_check.enabled (in `spring.datasource.url`) This configuration is recommended, and it is required when `scalar.db.sql.default_namespace_name` is set. In that case, leaving it at the default (`true`) makes Spring Data JDBC fail to start, so set it to `false`. ## Annotations The `@EnableScalarDbRepositories` annotation is needed on the JVM application to use Spring Data JDBC for ScalarDB as follows: ```java @SpringBootApplication @EnableScalarDbRepositories public class MyApplication { // These repositories are described in the next section in details @Autowired private GroupRepository groupRepository; @Autowired private UserRepository userRepository; ``` The annotation supports the following parameters: | Parameter | Default | Description | |---|---|---| | `basePackages` (or `value`) | Package of the annotated class | Base packages to scan for repository interfaces. Required when using multiple `@EnableScalarDbRepositories` annotations to avoid scan conflicts. | | `transactionManagerRef` | `scalarDbTransactionManager` | Bean name of the transaction manager. Set to `scalarDbSuspendableTransactionManager` for [two-phase commit transactions](#two-phase-commit-transaction). | | `jdbcOperationsRef` | `scalarDbNamedParameterJdbcTemplate` | Bean name of the `NamedParameterJdbcOperations` to use. | | `dataAccessStrategyRef` | `""` (auto-detected) | Bean name of the `DataAccessStrategy`. Needed in multi-datasource setups. | In the simplest case (a single ScalarDB datasource), you don't need to set any of these parameters because the defaults point to the auto-configured beans described in the next section. ## Auto-configured beans Spring Data JDBC for ScalarDB automatically registers the following beans through `ScalarDbJdbcConfiguration`: | Bean name | Type | Description | |---|---|---| | `jdbcTemplate` | `JdbcTemplate` | Default JDBC template (`@Primary`). Defined to prevent bean conflicts with the ScalarDB-specific beans of the same type. | | `namedParameterJdbcTemplate` | `NamedParameterJdbcTemplate` | Default named-parameter JDBC template (`@Primary`). Defined to prevent bean conflicts with the ScalarDB-specific beans of the same type. | | `transactionManager` | `JdbcTransactionManager` | Default transaction manager (`@Primary`). Defined to prevent bean conflicts with the ScalarDB-specific beans of the same type. | | `scalarDbJdbcTemplate` | `ScalarDbJdbcTemplate` | ScalarDB-aware JDBC template with custom exception translation. | | `scalarDbNamedParameterJdbcTemplate` | `ScalarDbNamedParameterJdbcTemplate` | ScalarDB-aware named-parameter template. Used by `@EnableScalarDbRepositories` by default. | | `scalarDbTransactionManager` | `ScalarDbTransactionManager` | ScalarDB transaction manager. Used by `@EnableScalarDbRepositories` by default. | | `scalarDbSuspendableTransactionManager` | `ScalarDbSuspendableTransactionManager` | ScalarDB transaction manager for two-phase commit transactions. | ## Persistent entity model The users of Spring Data JDBC for ScalarDB needs to write classes for object mapping to ScalarDB tables. How to write those classes are written in [Persisting Entities](https://docs.spring.io/spring-data/jdbc/docs/3.0.x/reference/html/#jdbc.entity-persistence), so this section describes some limitations on the integration. These are example model classes: ### domain/model/User ```java // This model class corresponds to the following table schema: // // create table my_app_ns.user (id bigint, group_id bigint, name text, primary key (id)); // // -- UserRepository can use `name` column as a condition in SELECT statement // -- as the column is a ScalarDB secondary index. // create index on my_app_ns.user (name); // Set `schema` parameter in @Table annotation if you don't use `scalar.db.sql.default_namespace_name` property. // // Spring Data automatically decides the target table name based on a model class name. // You can also specify a table name by setting `value` parameter. // // @Table(schema = "my_app_ns", value = "user") @Table public class User { @Id public final Long id; public final Long groupId; // Spring Data automatically decides the target column name based on an instance variable name. // You can also specify a column name by setting `value` parameter in @Column annotation. // @Column("name") public final String name; public User(Long id, Long groupId, String name) { this.id = id; this.groupId = groupId; this.name = name; } } ``` ### domain/model/Group ```java // This model class corresponds to the following table schema: // // create table my_app_ns.group (account_id int, group_type int, balance int, primary key (account_id, group_type)); @Table public class Group { // This column `account_id` is a part of PRIMARY KEY in ScalarDB SQL // // Spring Data JDBC always requires a single @Id annotation while it doesn't allow multiple @Id annotations. // The corresponding ScalarDB SQL table `group` has a primary key consisting of multiple columns. // So, Spring Data @Id annotation can't be used in this case, but @Id annotation must be put on any instance variable // (@Id annotation can be put on `balance` as well.) @Id public final Integer accountId; // This column `group_type` is also a part of PRIMARY KEY in ScalarDB SQL public final Integer groupType; public final Integer balance; public Group(Integer accountId, Integer groupType, Integer balance) { this.accountId = accountId; this.groupType = groupType; this.balance = balance; } } ``` [This sample implementation](https://github.com/scalar-labs/scalardb-samples/tree/main/spring-data-sample/src/main/java/sample/domain/model) can be used as a reference as well. ### domain/repository/UserRepository ```java @Transactional @Repository public interface UserRepository extends ScalarDbRepository { // `insert()` and `update()` are automatically enabled with `ScalarDbRepository` (or `ScalarDbTwoPcRepository`). // Many APIs of `CrudRepository` and `PagingAndSortingRepository` are automatically enabled. // https://docs.spring.io/spring-data/commons/docs/3.0.x/api/org/springframework/data/repository/CrudRepository.html // https://docs.spring.io/spring-data/commons/docs/3.0.x/api/org/springframework/data/repository/PagingAndSortingRepository.html // Also, you can prepare complicated APIs with the combination of the method naming conventions. // https://docs.spring.io/spring-data/jdbc/docs/3.0.x/reference/html/#repositories.definition-tuning // These APIs use the ScalarDB secondary index List findByName(String name); List findTop2ByName(String name); // Current ScalarDB SQL doesn't support range scan or order using secondary indexes // List findByNameBetween(String name); // List findByGroupIdOrderByName(long groupId); default void reverseName(long id) { Optional model = findById(id); if (model.isPresent()) { User existing = model.get(); User updated = new User( existing.id, existing.groupId, existing.name.reverse()); update(updated); } } default void deleteAfterSelect(long id) { Optional existing = findById(id); existing.ifPresent(this::delete); } } ``` ### domain/repository/GroupRepository ```java @Transactional @Repository public interface GroupRepository extends ScalarDbRepository { // @Id annotation is put only on Group.accountId, but ScalarDB SQL expects the combination of // `account_id` and `group_type` columns as the table uses them as a primary key. So `findById()` can't be used. Optional findFirstByAccountIdAndGroupType(int accountId, int groupType); List findByAccountIdAndGroupTypeBetweenOrderByGroupTypeDesc( int accountId, int groupTypeFrom, int groupTypeTo); List findTop2ByAccountIdAndGroupTypeBetween( int accountId, int groupTypeFrom, int groupTypeTo); // `update()` method also depends on @Id annotation as well as `findById()`, // so users need to write ScalarDB SQL in @Query annotation. @Modifying @Query( "UPDATE \"my_app_ns\".\"group\" SET \"balance\" = :balance \n" + " WHERE \"my_app_ns\".\"group\".\"account_id\" = :accountId \n" + " AND \"my_app_ns\".\"group\".\"group_type\" = :groupType \n") int updateWithAttributes( @Param("accountId") int accountId, @Param("groupType") int groupType, @Param("balance") int balance); default void incrementBalance(int accountId, int groupType, int value) { Optional model = findFirstByAccountIdAndGroupType(accountId, groupType); model.ifPresent( found -> updateWithAttributes( found.accountId, found.groupType, found.balance + value)); } default void transfer( int accountIdFrom, int groupTypeFrom, int accountIdTo, int groupTypeTo, int value) { incrementBalance(accountIdFrom, groupTypeFrom, -value); incrementBalance(accountIdTo, groupTypeTo, value); } // This method name and signature results in issuing an unexpected SELECT statement and // results in query failure. It looks a bug of Spring Data... // // void deleteByAccountIdAndGroupType(int accountId, int groupType); @Modifying @Query( "DELETE FROM \"my_app_ns\".\"group\" \n" + " WHERE \"my_app_ns\".\"group\".\"account_id\" = :accountId \n" + " AND \"my_app_ns\".\"group\".\"group_type\" = :groupType \n") int deleteByAccountIdAndGroupType( @Param("accountId") int accountId, @Param("groupType") int groupType); default void deleteByAccountIdAndGroupTypeAfterSelect(int accountId, int groupType) { Optional entity = findFirstByAccountIdAndGroupType(accountId, groupType); entity.ifPresent(found -> deleteByAccountIdAndGroupType(accountId, groupType)); } } ``` [This sample implementation](https://github.com/scalar-labs/scalardb-samples/tree/main/spring-data-sample/src/main/java/sample/domain/repository) can be used as a reference as well. ## Error handling Spring Data JDBC for ScalarDB can throw the following exceptions. - com.scalar.db.sql.springdata.exception.ScalarDbTransientException - This is thrown when a transaction fails due to a transient error - The transaction should be retried - This is a subclass of `org.springframework.dao.TransientDataAccessException` and catching the superclass is safer to handle other type of transient errors thrown from Spring Data - com.scalar.db.sql.springdata.exception.ScalarDbNonTransientException - This is thrown when a transaction fails due to a non-transient error - The transaction should not be retried - This is a subclass of `org.springframework.dao.NonTransientDataAccessException` and catching the superclass is safer to handle other type of non-transient errors thrown from Spring Data - com.scalar.db.sql.springdata.exception.ScalarDbUnknownTransactionStateException - This is a subclass of `ScalarDbNonTransientException` and the transaction should not be retried as well - This is thrown when a transaction commit fails and the final state is unknown - Whether the transaction is actually committed or not needs to be decided by the application side (e.g. check if the target record is expectedly updated) These exceptions include the transaction ID, which can be useful for troubleshooting purposes. ## Limitations ### Multiple column PRIMARY KEY As you see in the above example, Spring Data JDBC's `@Id` annotation doesn't support multiple columns. So, if a table has a primary key consisting of multiple columns, users can't use the following APIs and may need to write Scalar SQL DB query in `@Query` annotation. - `findById()` - `existsById()` - `update(T entity)` - `delete(T entity)` - `deleteById(ID id)` - `deleteAllById(Iterable ids)` ### One-to-many relationships between two entities Spring Data JDBC supports one-to-many relationships. But it implicitly deletes and re-creates all the associated child records even if only the parent's attributes are changed. This behavior would result in a performance penalty. Considering this concern, it's not recommended to use the feature in Spring Data JDBC for ScalarDB. For instance, assuming a Bank record contains many Account records, calling `BankRepository#update()` on the parent entity causes Spring Data JDBC to implicitly delete and re-insert all child Account records, even if only the parent's attributes are changed. ```java @Autowired BankRepository bankRepository; ... bankRepository.insert(new Bank(42, "My bank", ImmutableSet.of( new Account(1, "Alice"), new Account(2, "Bob"), new Account(3, "Carol") ))); Bank bank = bankRepository.findById(42).get(); System.out.printf("Bank: " + bank); // `DELETE FROM "account" WHERE "account"."bank_id" = ?` is implicitly issued by Spring Data JDBC // followed by re-inserting all child records, causing unnecessary performance overhead bankRepository.update(new Bank(bank.bankId, bank.name + " 2", bank.accounts)); ``` ## Advanced features ### Multi-storage transaction ScalarDB supports [Multi-storage Transaction](../multi-storage-transactions.mdx), and users can use the feature via Spring Data JDBC for ScalarDB. The following needs to be configured to use the feature. #### spring.datasource.url Here is a sample datasource URL assuming there are two namespaces "north" and "south" that manage data with MySQL and PostgreSQL respectively. ``` spring.datasource.url=jdbc:scalardb:\ ?scalar.db.sql.connection_mode=direct\ &scalar.db.storage=multi-storage\ &scalar.db.multi_storage.storages=mysql,postgresql\ &scalar.db.multi_storage.namespace_mapping=north:mysql,south:postgresql\ &scalar.db.multi_storage.default_storage=postgresql\ &scalar.db.multi_storage.storages.mysql.storage=jdbc\ &... ``` #### @Table annotation on model classes - `schema` parameter: multi-storage mapping key name (`scalar.db.multi_storage.namespace_mapping`) - `value` parameter: actual table name ```java @Table(schema = "north", value = "account") public class NorthAccount { ``` ### Retry #### Retry with Spring Retry Spring Data JDBC for ScalarDB could throw exceptions when concurrent transactions conflict. Users need to take care of those exceptions by retrying the operations. [Spring Retry](https://github.com/spring-projects/spring-retry) provides some functionalities for retry. Also in Spring Data JDBC for ScalarDB, Spring Retry would be helpful to make retry handling simpler and easier. This section introduces how to use Spring Retry. ##### Dependencies The following dependencies need to be added to your project. ```gradle dependencies { implementation "org.springframework.boot:spring-boot-starter:${springBootVersion}" implementation "org.springframework.boot:spring-boot-starter-aop:${springBootVersion}" implementation "org.springframework.retry:spring-retry:${springRetryVersion}" } ``` ##### Annotation `@EnableRetry` annotation needs to be added in the JVM application. ```java @SpringBootApplication @EnableScalarDbRepositories @EnableRetry public class MyApp { ``` `@Retryable` annotation makes Spring Data repository class or method automatically retry a failed operation. Spring Data JDBC for ScalarDB can throw a transient error exception, so it's highly recommended to specify `org.springframework.dao.TransientDataAccessException` as a target class in the annotation. Also, backoff and max attempts can be configured in the annotation like this: ```java @Transactional @Retryable( include = TransientDataAccessException.class, maxAttempts = 4, backoff = @Backoff(delay = 500, maxDelay = 2000, multiplier = 2)) default void insertWithRetry(Player player) { insert(player); } ``` With `@Recover` annotation, retry-exhausted failure will be automatically recovered by a specified method. ```java @Transactional @Retryable(include = TransientDataAccessException.class, recover = "recoverInsert") default void insertWithRetryAndRecover(Player player) { insert(player); } @Transactional @Recover default void recoverInsert(Throwable throwable, Player player) throws Throwable { Optional existing = findById(player.id); if (!existing.isPresent()) { throw throwable; } logger.info( "Found an existing record {}. Updating it with a new record {}", existing.get(), player); update(player); } ``` #### Retry with other retry library There are other options available for retrying transactions, such as Spring Retry's RetryTemplate or Resilience4j. Feel free to choose and use your preferred retry library. ### Custom type converters If you need to convert between Java types and database column types that are not natively supported by ScalarDB SQL (for example, `UUID`), you can register custom converters by extending `AbstractJdbcConfiguration`: ```java @Configuration public class MyJdbcConfiguration extends AbstractJdbcConfiguration { @WritingConverter public static class UuidToStringConverter implements Converter { @Override public String convert(UUID source) { return source.toString(); } } @ReadingConverter public static class StringToUuidConverter implements Converter { @Override public UUID convert(String source) { return UUID.fromString(source); } } @Override protected List userConverters() { return Arrays.asList(new UuidToStringConverter(), new StringToUuidConverter()); } } ``` This is a standard Spring Data JDBC pattern. For details, see [Custom Conversions](https://docs.spring.io/spring-data/jdbc/docs/3.0.x/reference/html/#jdbc.custom-converters) in the Spring Data JDBC reference. ### Two-phase commit transaction ScalarDB supports [Two-phase commit transaction](../two-phase-commit-transactions.mdx), and users can use the feature via Spring Data JDBC for ScalarDB. The following configurations are needed. #### spring.datasource.url - `scalar.db.sql.default_transaction_mode` property: `two_phase_commit_transaction` ```console spring.datasource.url=jdbc:scalardb:\ ?scalar.db.sql.connection_mode=direct\ &scalar.db.contact_points=jdbc:mysql://localhost:3306/my_app_ns\ &...\ &scalar.db.sql.default_transaction_mode=two_phase_commit_transaction ``` #### Configuration of Spring Data transaction manager Spring Data JDBC for ScalarDB provides a custom Spring Data transaction manager to achieve 2PC transactions. You need to configure either of the following annotations to enable the custom transaction manager. - Set `transactionManager` parameter of all the `@Transactional` to `scalarDbSuspendableTransactionManager` - Set `transactionManagerRef` parameter of the `@EnableScalarDbRepositories` to `scalarDbSuspendableTransactionManager` #### Repository classes ##### APIs Spring Data JDBC for ScalarDB supports 2 types of APIs for 2PC transaction. One is primitive APIs and the other is high level API. ###### Primitive 2PC APIs `ScalarDbTwoPcRepository` is an extension of `ScalarDbRepository` and it has the following APIs that correspond to the same name methods in ScalarDB and users can use them to build custom repository methods for 2PC transaction. - begin() - returns an auto-generated transaction ID - prepare() - validate() - suspend() - commit() - join(`transactionId`) - resume(`transactionId`) All in-flight operations are rolled back when any exception is thrown from Spring Data repository method. See [How to execute Two-phase Commit Transactions](../two-phase-commit-transactions.mdx#how-to-execute-two-phase-commit-transactions) for details. ```java @Transactional(transactionManager = "scalarDbSuspendableTransactionManager") @Repository public interface TwoPcPlayerRepository extends ScalarDbTwoPcRepository { Logger logger = LoggerFactory.getLogger(TwoPcPlayerRepository.class); // Either of twoPcJoinAndInsert() or twoPcBeginAndInsert() can be used to start a transaction default void twoPcJoinAndInsert(String txId, Player player) throws SQLException { join(txId); insert(player); suspend(); } default String twoPcBeginAndInsert(String id, Player player) throws SQLException { String txId = begin(); insert(player); suspend(); return txId; } default void twoPcPrepare(String txId) throws SQLException { resume(txId); prepare(); suspend(); } default void twoPcValidate(String txId) throws SQLException { resume(txId); validate(); suspend(); } default void twoPcCommit(String txId) throws SQLException { resume(txId); commit(); } ``` ###### High level 2PC API The above primitive APIs are powerful and make it possible to explicitly control 2PC transaction operations in flexible and fine-grained ways. On the other hand, users need to consider which APIs to call in a proper order when using the APIs. Especially coordinator side operations for local state and remote service calls would be easily complicated. `ScalarDbTwoPcRepository` also provides some user-friendly APIs called high-level APIs to cover common use cases. With these APIs, you can develop your microservice applications more easily and securely. For the development of coordinator service in a microservice, `ScalarDbTwoPcRepository` provides `executeTwoPcTransaction` API that implicitly executes 2PC related operations in the following order. By using the API, you don’t need to think about how and when to execute transactional operations. - Start a local transaction with a global transaction ID - Execution phase: Local and remote CRUD operations (*) - Prepare phase: Local and remote prepare operations (**) in parallel - Validation phase: Local and remote validation operations (**) in parallel - This is needed only if `scalar.db.consensus_commit.isolation_level` is `SERIALIZABLE` and `scalar.db.consensus_commit.serializable_strategy` is `EXTRA_READ` - Commit phase: Local commit operation is first executed. Remote commit operations are executed (**) in parallel after the local commit operation succeeded - (If any operation except for remote commit operation fails) rollback phase: Local and remote rollback operations (**) in parallel (* This implementation of local and remote operation callbacks is injected by users)\ (** This implementation of remote operation callbacks is injected by users) Rollback operations for local and remote participants will be executed when an exception is thrown from any operation. As for the error handling of `executeTwoPcTransaction()`, - The following exceptions can be thrown from the API - `ScalarDbTransientException` - Users should retry the 2PC transaction operations from the beginning when this exception is thrown - `ScalarDbNonTransientException` - `ScalarDbUnknownTransactionStateException` - Whether the 2PC transaction is actually committed or not needs to be decided by the application side - The exceptions contain the 2PC global transaction ID. It should be useful for trouble shootings As for the implementations of Execution phase operations (in local and remote participants) and remote operations of Prepare/Validation/Commit/Rollback phases that are passed by users, those callbacks need to throw either of the exceptions when it fails: - `ScalarDbTransientException` when any transient issue happens including network disconnection and database transaction conflict - `ScalarDbNonTransientException` when any non-transient issue happens including authentication error and permission error - `ScalarDbUnknownTransactionStateException` when any exception that contains `UnknownTransactionStatusException` as a cause - Other exceptions thrown from the callbacks are treated as `ScalarDbTransientException` For the development of participant service in a microservice, `ScalarDbTwoPcRepository` provides the following APIs. By using the API, you don’t need to think about how and when to join, resume and suspend a transaction in details. - `joinTransactionOnParticipant` - Join the transaction, execute the CRUD operations and suspend the transaction on the participant service. This API should be called first, and then `prepareTransactionOnParticipant` and following APIs are supposed to be called. - `resumeTransactionOnParticipant` - Resume the transaction, execute the CRUD operations and suspend the transaction on the participant service. This API can be called after `joinTransactionOnParticipant` before `prepareTransactionOnParticipant` if needed. - `prepareTransactionOnParticipant` - Prepare the transaction and suspend the transaction on the participant service. This API should be called after `joinTransactionOnParticipant`, and then `validateTransactionOnParticipant` and following APIs are supposed to be called. - `validateTransactionOnParticipant` - Validate the transaction and suspend the transaction on the participant service. This API should be called after `prepareTransactionOnParticipant`, and then `commitTransactionOnParticipant` or `rollbackTransactionOnParticipant` is supposed to be called. - This is needed only if `scalar.db.consensus_commit.isolation_level` is `SERIALIZABLE` and `scalar.db.consensus_commit.serializable_strategy` is `EXTRA_READ` - `commitTransactionOnParticipant` - Commit the transaction on the participant service. This API should be called after `prepareTransactionOnParticipant` or `validateTransactionOnParticipant, depending on the transaction manager configurations. - `rollbackTransactionOnParticipant` - Rollback the transaction on the participant service. This API should be called after `prepareTransactionOnParticipant` or `validateTransactionOnParticipant, depending on the transaction manager configurations. With the high-level 2PC APIs of Spring Data JDBC for ScalarDB, you can focus on the business logic by hiding complicated transaction operations inside the APIs as follows: **Coordinator service** ```java @Autowired private AccountRepository accountRepository; private final StockService stockService = ...; private final NotificationService notificationService = ...; private final List remotePrepareCommitOpsList = Arrays.asList( RemotePrepareCommitPhaseOperations.createSerializable( stockService::prepareTransaction, stockService::validateTransaction, stockService::commitTransaction, stockService::rollbackTransaction), RemotePrepareCommitPhaseOperations.createSerializable( notificationService::prepareTxn, notificationService::validateTxn, notificationService::commitTxn, notificationService::rollbackTxn)); ``` ```java private Result> executeTwoPcTransactionUsingHighLevelApi( Account account, String itemName, int itemPrice, String notificationEventName) { return accountRepository.executeTwoPcTransaction( // CRUD operations for local and remote participants in execution phase. txId -> { // [local] Read the account's balance Optional stored = accountRepository.findById(account.id); if (!stored.isPresent()) { // Cancel the transaction if the account doesn't exist. // No need to retry. throw new ScalarDbNonTransientException( "The local state doesn't meet the condition. Aborting this transaction"); } // [remote] Start a transaction with the transaction ID, // read the item information and decrement the count Optional price = stockService.purchaseItem(txId, account.id, itemName); // [remote] Start a transaction with the transaction ID, // read the notification and remove it Optional notification = notificationService.getNotification(txId, account.id, notificationEventName); if (price.isPresent() && notification.isPresent()) { int currentBalance = stored.get().balance - price.get(); if (currentBalance < 0) { // Cancel the transaction if the global state doesn't meet the condition. // No need to retry. throw new ScalarDbNonTransientException( "The state of local and remote participants doesn't meet the condition. Aborting this transaction"); } // [local] Decrease the account's balance for the item accountRepository.update(new Account(account.id, currentBalance)); return Pair.of(currentBalance, notification.get()); } // Cancel the transaction if the global state doesn't meet the condition. // No need to retry. throw new ScalarDbNonTransientException( "The remote state doesn't meet the condition. Aborting this transaction"); }, // Remote operations for Prepare/Validate/Commit/Rollback remotePrepareCommitOpsList); } ``` ```java RetryTemplate retryTemplate = new RetryTemplateBuilder() .retryOn(TransientDataAccessException.class) .exponentialBackoff(500, 2.0, 8000) .maxAttempts(8) .withListener( new RetryListenerSupport() { @Override public void onError(RetryContext context, RetryCallback callback, Throwable throwable) { if (throwable instanceof ScalarDbUnknownTransactionStateException) { // Report an exception occurred that requires special treatments reportToDevelopers( String.format("Failed to process a 2PC transaction (%s). The final transaction status is unknown. Please check current application status", ((ScalarDbUnknownTransactionStateException) throwable).getTransactionId()), throwable); } }}) .build(); Result> result = retryTemplate.execute(context -> executeTwoPcTransactionUsingHighLevelApi(account, itemName, itemPrice, notificationEventName)); ``` [This sample implementation](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java) can be used as a reference as well. **Participant service** ```java @RestController public class StockController { @Autowired private StockRepository stockRepository; @PostMapping("/purchaseItem") public Optional purchaseItem( @RequestParam("transactionId") String transactionId, @RequestParam("accountId") int accountId, @RequestParam("itemName") String itemName) { return stockRepository.joinTransactionOnParticipant(txId, () -> { Optional item = stockRepository.findById(itemName); ... return Optional.of(item.price); }); } @PostMapping("/prepareTransaction") public void prepareTransaction(@RequestParam("transactionId") String transactionId) { return stockRepository.prepareTransactionOnParticipant(txId); } @PostMapping("/validateTransaction") public void validateTransaction(@RequestParam("transactionId") String transactionId) { return stockRepository.validateTransactionOnParticipant(txId); } @PostMapping("/commitTransaction") public void commitTransaction(@RequestParam("transactionId") String transactionId) { return stockRepository.commitTransactionOnParticipant(txId); } @PostMapping("/rollbackTransaction") public void rollbackTransaction(@RequestParam("transactionId") String transactionId) { return stockRepository.rollbackTransactionOnParticipant(txId); } } ``` [This sample implementation](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java) uses gRPC not REST API, but it can be used as a reference as well. #### Limitations ##### `@Transactional` methods don't implicitly call `commit()` In microservice applications with ScalarDB, commits must be explicitly invoked by a coordinator service, not be locally triggered by the Spring Data transaction framework when exiting `@Transactional` methods. The `@Transactional(transactionManager = "scalarDbSuspendableTransactionManager")` annotation prevents such local commits. This extended behavior may confuse developers who expect `@Transactional` methods to implicitly commit transactions. For instance, assuming you want to use the `@Transactional` annotation on methods of a service class, the following code works in the **normal** transaction mode. ```java @Service public class SampleService { ... // For the normal transaction mode @Transactional // For the 2PC transaction mode // @Transactional(transactionManager = "scalarDbSuspendableTransactionManager") public void repayment(int customerId, int amount) { Customer customer = customerRepository.getById(customerId); int updatedCreditTotal = customer.creditTotal - amount; // Check if over repayment or not if (updatedCreditTotal < 0) { throw new RuntimeException( String.format( "Over repayment. creditTotal:%d, payment:%d", customer.creditTotal, amount)); } // Reduce credit_total for the customer customerRepository.update(customer.withCreditTotal(updatedCreditTotal)); } } ``` However, that code doesn't work in the 2PC transaction mode even with `transactionManager = "scalarDbSuspendableTransactionManager"`. Instead, use `ScalarDbTwoPcRepository.executeOneshotOperations()` as follows. ```java @Service public class SampleService { ... public void repayment(int customerId, int amount) { customerRepository.executeOneshotOperations(() -> { Customer customer = customerRepository.getById(customerId); int updatedCreditTotal = customer.creditTotal - amount; // Check if over repayment or not if (updatedCreditTotal < 0) { throw new RuntimeException( String.format( "Over repayment. creditTotal:%d, payment:%d", customer.creditTotal, amount)); } // Reduce credit_total for the customer customerRepository.update(customer.withCreditTotal(updatedCreditTotal)); return null; }); } } ``` ### Multi-datasource setup with Spring Data JDBC You can use ScalarDB repositories alongside standard Spring Data JDBC repositories in the same application, each backed by a different datasource. This is useful when your application manages some tables through ScalarDB and other tables through a standard JDBC connection. #### Properties ```console # Non-ScalarDB Spring Data JDBC datasource spring.datasource.default.url=jdbc:postgresql://localhost/mydb?user=postgres&password=postgres # ScalarDB datasource spring.datasource.scalardb.driver-class-name=com.scalar.db.sql.jdbc.SqlJdbcDriver spring.datasource.scalardb.url=jdbc:scalardb:\ ?scalar.db.sql.connection_mode=direct\ &scalar.db.contact_points=jdbc:postgresql://localhost/\ &... ``` #### Configuration classes Create two `@Configuration` classes. The ScalarDB side uses `@EnableScalarDbRepositories` and the non-ScalarDB side uses Spring Data JDBC's `@EnableJdbcRepositories`. Use `basePackages` to direct each annotation to the correct set of repository interfaces. **ScalarDB configuration:** ```java @Configuration @EnableScalarDbRepositories( basePackages = "com.example.scalardb.domain", transactionManagerRef = "myScalarDbTransactionManager", dataAccessStrategyRef = "myScalarDbDataAccessStrategy") public class ScalarDbConfiguration { @Bean @ConfigurationProperties("spring.datasource.scalardb") DataSourceProperties myScalarDbDataSourceProperties() { return new DataSourceProperties(); } @Bean DataSource myScalarDbDataSource() { return myScalarDbDataSourceProperties().initializeDataSourceBuilder().build(); } @Bean JdbcTemplate myScalarDbJdbcTemplate() { return new ScalarDbJdbcTemplate(myScalarDbDataSource()); } @Bean DataAccessStrategy myScalarDbDataAccessStrategy( RelationalMappingContext context, JdbcConverter converter) { // Create a DataAccessStrategy using ScalarDbDialect // and the beans from the ScalarDB datasource ... } @Bean PlatformTransactionManager myScalarDbTransactionManager() { return new ScalarDbTransactionManager(myScalarDbDataSource()); } } ``` **Non-ScalarDB Spring Data JDBC configuration:** ```java @Configuration @EnableJdbcRepositories( basePackages = "com.example.myapp.domain", transactionManagerRef = "defaultTransactionManager", dataAccessStrategyRef = "defaultDataAccessStrategy") public class DefaultConfiguration { @Bean @ConfigurationProperties("spring.datasource.default") DataSourceProperties defaultDataSourceProperties() { return new DataSourceProperties(); } @Bean @Primary DataSource defaultDataSource() { return defaultDataSourceProperties().initializeDataSourceBuilder().build(); } @Bean JdbcTemplate defaultJdbcTemplate() { return new JdbcTemplate(defaultDataSource()); } @Bean @Primary DataAccessStrategy defaultDataAccessStrategy( RelationalMappingContext context, JdbcConverter converter) { // Create a DataAccessStrategy using DialectResolver // and the beans from the default datasource ... } @Bean PlatformTransactionManager defaultTransactionManager() { return new JdbcTransactionManager(defaultDataSource()); } } ``` :::note The non-ScalarDB side uses standard `JdbcTemplate` and `JdbcTransactionManager`, while the ScalarDB side uses `ScalarDbJdbcTemplate` and `ScalarDbTransactionManager`. ::: #### Injecting beans with `@Qualifier` When you need to inject specific `JdbcTemplate` beans in application code, use `@Qualifier` to select the correct one: ```java @Qualifier("defaultJdbcTemplate") @Autowired JdbcTemplate template; @Qualifier("myScalarDbJdbcTemplate") @Autowired JdbcTemplate scalarDbTemplate; ``` ## Spring Boot version compatibility Spring Data JDBC for ScalarDB is tested with Spring Boot 2, 3, and 4. ## Troubleshooting This section describes methods to troubleshoot errors that may occur when using Spring Data JDBC. ### `A constructor parameter name must not be null to be used with Spring Data JDBC` runtime error The runtime error `A constructor parameter name must not be null to be used with Spring Data JDBC` may occur when using Spring Boot 3. To work around this issue, you can pass the `-parameters` option to `javac` as follows: ```gradle compileJava { options.compilerArgs << '-parameters' } ``` ## Sample application You can see the following sample applications that use Spring Data JDBC for ScalarDB. It only serves as a reference and does not necessarily meet production code standards. - [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](../scalardb-cluster/getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) - [Sample application of Spring Data JDBC for ScalarDB with Multi-storage Transactions](../scalardb-samples/spring-data-multi-storage-transaction-sample/README.mdx) - [Sample application of Spring Data JDBC for ScalarDB with Microservice Transactions](../scalardb-samples/spring-data-microservice-transaction-sample/README.mdx) ## How it works In order to use Spring Data JDBC for ScalarDB, the following features are implemented in the integration - Map `jdbc:scalardb` protocol in JDBC Connection URL to a Spring Data JDBC dialect class for ScalarDB SQL - This feature is handled by ScalarDbDialect and ScalarDbDialectProvider - Prevent users from using some APIs of Spring Data Repository classes (CrudRepository and PagingAndSortingRepository) unsupported in ScalarDB SQL - This feature is handled by ScalarDbJdbcAggregateTemplate which is a bit lower layer Spring Data JDBC component used by Repository classes - Make Spring Data Repository classes implicitly use the custom JdbcAggregateTemplate (ScalarDbJdbcAggregateTemplate) - This feature is handled by ScalarDbJdbcRepositoryFactory and ScalarDbJdbcRepositoryFactoryBean - Add explicit `insert()` and `update()` APIs to Spring Data Repository classes instead of bundled `save()` which depends on autoincrement ID feature in underlying databases while ScalarDB SQL doesn't support it - This feature is handled by ScalarDbRepository (or ScalarDbTwoPcRepository) and ScalarDbRepositoryImpl - Enable all the above features in Spring framework manner - This configuration is handled by - some Java classes in `com.scalar.db.sql.springdata` - `@EnableScalarDbRepositories` annotation - `resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports` - `resources/META-INF/spring.factories` ## References - [Spring Data JDBC - Reference Documentation](https://docs.spring.io/spring-data/jdbc/docs/3.0.x/reference/html/) - [ScalarDB JDBC Guide](jdbc-guide.mdx) - [Javadoc for Spring Data JDBC for ScalarDB](https://javadoc.io/doc/com.scalar-labs/scalardb-sql-spring-data/3.18.0/index.html) ================================================ FILE: docs/scalardb-sql/sql-api-guide.mdx ================================================ --- tags: - Enterprise Premium displayed_sidebar: docsEnglish --- # ScalarDB SQL API Guide import JavadocLink from '/src/theme/JavadocLink.js'; This guide describes how to use ScalarDB SQL API. ## Add ScalarDB SQL API to your project To add the dependencies on ScalarDB SQL API by using Gradle, use the following, replacing `` with the versions of ScalarDB SQL API and the related library, respectively, that you are using: ```gradle dependencies { implementation 'com.scalar-labs:scalardb-sql:' implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:' } ``` To add the dependencies by using Maven, use the following, replacing `...` with the version of ScalarDB SQL API that you are using: ```xml com.scalar-labs scalardb-sql ... com.scalar-labs scalardb-cluster-java-client-sdk ... ``` ## SqlSessionFactory In ScalarDB SQL API, you execute all operations through a `SqlSession` instance, which is instantiated with `SqlSessionFactory`. This section explains how to use them. Before explaining `SqlSessionFactory`, we start with the explanation for Connection mode and Transaction mode. ### Transaction mode Also, ScalarDB SQL offers two transaction modes: *Transaction* mode and *Two-phase Commit Transaction* mode. Transaction mode exposes only `commit` interface to users and runs two-phase commit behind the scene, while Two-phase Commit Transaction mode exposes two-phase commit style interfaces (`prepare` and `commit`) to users. You can specify the default transaction mode in your configuration file or when you build `SqlSessionFactory`. And you also can change it with the `setTransactionMode()` method of `SqlSession`. ### Build SqlSessionFactory You can build `SqlSessionFactory` with a properties file as follows: ```java SqlSessionFactory sqlSessionFactory = SqlSessionFactory.builder() .withPropertiesFile("") // If you need to set custom properties, you can specify them with withProperty() or withProperties() .withProperty("", "") .build(); ``` Please see [ScalarDB Cluster SQL client configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations) for the details of the configurations. ### Get a SqlSession instance You can get a `SqlSession` instance with `SqlSessionFactory` as follows: ```java SqlSession sqlSession = sqlSessionFactory.createSqlSession(); ``` Note that `SqlSession` is not thread-safe. Please don't use it from multiple threads at the same time. #### Close a SqlSession instance Once all operations are done with a `SqlSession` instance, you should close the SqlSession instance: ```java sqlSession.close(); ``` ### Close a SqlSessionFactory instance `sqlSessionFactory` should also be closed once it's no longer needed: ```java sqlSessionFactory.close(); ``` ## Execute SQLs You can execute a SQL with `SqlSession` as follows: ```java ResultSet resultSet = sqlSession.execute(""); ``` You can also execute a `Statement` object with `SqlSession` as follows: ```java // Build a statement Statement statement = StatementBuilder....; // Execute the statement ResultSet resultSet = sqlSession.execute(statement); ``` For mutation statements (`INSERT`, `UPSERT`, `UPDATE`, or `DELETE`), the returned `ResultSet` contains a single `Record` with one `INT` column named `updateCount`, which reports the number of rows affected. See [Execute transactions](#execute-transactions) for how DML, DDL, and DCL statements interact with transactions. `Statement` objects can be built by `StatementBuilder` that has factory methods for corresponding SQLs. For more details, see the page in the Javadoc and [ScalarDB SQL Grammar](grammar.mdx). ### Handle ResultSet objects As the result of the SQL execution, `SqlSession` returns a `ResultSet` object. Here, we describe how to handle `ResultSet` objects. If you want to get results one by one from the `ResultSet` object, you can use the `one()` method as follows: ```java Optional record = resultSet.one(); ``` Or, if you want to get results all at once as a `List`, you can use the `all()` method as follows: ```java List records = resultSet.all(); ``` Also, as `ResultSet` implements `Iterable`, you can use it in a for-each loop as follows: ```java for (Record record : resultSet) { ... } ``` If you want to get the metadata of the `ResultSet` object, you can use the `getColumnDefinitions()` method as follows: ```java ColumnDefinitions columnDefinitions = resultSet.getColumnDefinitions(); ``` For more details, see the page in the Javadoc. ### Handle Record objects As mentioned, a `ResultSet` object returns `Record` objects that represent records of the database. You can get a column value of a result with `getXXX("")` or `getXXX()` methods (XXX is a type name) as follows: ```java // Get a BOOLEAN value of a column boolean booleanValueGottenByName = record.getBoolean(""); boolean booleanValueGottenByIndex = record.getBoolean(); // Get an INT value of a column int intValueGottenByName = record.getInt(""); int intValueGottenByIndex = record.getInt(); // Get a BIGINT value of a column long bigIntValueGottenByName = record.getBigInt(""); long bigIntValueGottenByIndex = record.getBigInt(); // Get a FLOAT value of a column float floatValueGottenByName = record.getFloat(""); float floatValueGottenByIndex = record.getFloat(); // Get a DOUBLE value of a column double doubleValueGottenByName = record.getDouble(""); double doubleValueGottenByIndex = record.getDouble(); // Get a TEXT value of a column String textValueGottenByName = record.getText(""); String textValueGottenByIndex = record.getText(); // Get a BLOB value of a column (as a ByteBuffer) ByteBuffer blobValueGottenByName = record.getBlob(""); ByteBuffer blobValueGottenByIndex = record.getBlob(); // Get a BLOB value of a column as a byte array byte[] blobValueAsBytesGottenByName = record.getBlobAsBytes(""); byte[] blobValueAsBytesGottenByIndex = record.getBlobAsBytes(); // Get a DATE value of a column as a LocalDate LocalDate dateValueGottenByName = record.getDate(""); LocalDate dateValueGottenByName = record.getDate(); // Get a TIME value of a column as a LocalTime LocalTime timeValueGottenByName = record.getTime(""); LocalTime timeValueGottenByName = record.getTime(); // Get a TIMESTAMP value of a column as a LocalDateTime LocalDateTime timestampValueGottenByName = record.getTimestamp(""); LocalDateTime timestampValueGottenByName = record.getTimestamp(); // Get a TIMESTAMPTZ value of a column as an Instant Instant timestampTZValueGottenByName = record.getTimestampTZ(""); Instant timestampTZValueGottenByName = record.getTimestampTZ(); ``` And if you need to check if a value of a column is null, you can use the `isNull("")` or `isNull()` method. ``` java // Check if a value of a column is null boolean isNullGottenByName = record.isNull(""); boolean isNullGottenByIndex = record.isNull(); ``` For more details, see the page of the Javadoc. ### Prepared Statements You can use `PreparedStatement` for queries that are executed multiple times in your application: ```java PreparedStatement preparedStatement = sqlSession.prepareStatement(""); ResultSet result = sqlSession.execute(preparedStatement.bind()); ``` If you execute the same query a second time or later, the cached pre-parsed statement object is used. Thus, you can gain a performance advantage with `PreparedStatement` when you execute the query multiple times. If you execute a query only once, a prepared statement is inefficient because it requires extra processing. Consider using the `sqlSession.execute()` method instead in that case. Calling `bind(...)` on a `PreparedStatement` returns a `BoundStatement` that holds the parameter values. You then pass the `BoundStatement` to `sqlSession.execute()`. Parameters can be either positional or named: ```java // Positional parameters PreparedStatement preparedStatement1 = sqlSession.prepareStatement("INSERT INTO tbl (c1, c2) VALUES (?, ?)"); // Named parameters PreparedStatement preparedStatement2 = sqlSession.prepareStatement("INSERT INTO tbl (c1, c2) VALUES (:a, :b)"); ``` You can bind values in one call by passing them to `bind(...)`: ```java // Positional values sqlSession.execute(preparedStatement1.bind(Value.ofInt(10), Value.ofText("value"))); // Named values Map namedValues = new HashMap<>(); namedValues.put("a", Value.ofInt(10)); namedValues.put("b", Value.ofText("value")); sqlSession.execute(preparedStatement2.bind(namedValues)); ``` Or you can use fluent setters on the `BoundStatement` returned by `bind()`: ```java // Positional setters sqlSession.execute( preparedStatement1.bind() .setInt(0, 10) .setText(1, "value")); // Named setters sqlSession.execute( preparedStatement2.bind() .setInt("a", 10) .setText("b", "value")); ``` To wrap a plain SQL string as a `Statement`, for example when constructing a `BatchedStatements`, use `SimpleStatement.of("")`. For more details, see the , , and pages of the Javadoc. ### Execute batch statements When you need to execute multiple statements in a single round trip, use `sqlSession.executeBatch(...)`. This is useful for bulk inserts, updates, or mixed mutations against one or more namespaces. Use `BatchedStatements` to build a batch where each statement can carry its own default namespace: ```java BatchedStatements batchedStatements = BatchedStatements.builder() .add(boundStatement1, "namespace1") .add(boundStatement2, "namespace2") .add(SimpleStatement.of("INSERT INTO tbl (c1, c2) VALUES (1, 'a')")) .build(); List results = sqlSession.executeBatch(batchedStatements); ``` If every statement uses the session's default namespace, you can pass a `List` instead: ```java List results = sqlSession.executeBatch(Arrays.asList(boundStatement1, boundStatement2)); ``` `executeBatch(...)` returns a `List` containing one `ResultSet` per input statement, in the same order as the input. Batch execution accepts DML, DDL, and DCL statements. Command statements (`BEGIN`, `COMMIT`, `ROLLBACK`, and other transaction-control statements) will be rejected, so use the corresponding `SqlSession` methods instead. See [Execute transactions](#execute-transactions) for how DML, DDL, and DCL statements interact with transactions. For more details, see the and pages of the Javadoc. ## Execute transactions In ScalarDB SQL, DML statements (`SELECT`, `INSERT`, `UPSERT`, `UPDATE`, and `DELETE`) always run within a transaction. The transaction can be one you started explicitly with `sqlSession.begin()` or one that ScalarDB manages automatically: - If a transaction is active (after `sqlSession.begin()`), both `sqlSession.execute(...)` and `sqlSession.executeBatch(...)` run their DML statements in that transaction. - If no transaction is active, each `sqlSession.execute(...)` call runs its DML statement in its own auto-managed transaction, and each `sqlSession.executeBatch(...)` call runs all DML statements in the batch atomically in a single auto-managed transaction. In addition to `begin()`, `SqlSession` provides other variants, such as `beginReadOnly()` for starting a read-only transaction and overloads that accept transaction attributes. For details, see the page of the Javadoc. To make multiple DML statements from separate `execute(...)` calls atomic, you must begin an explicit transaction before executing them. `executeBatch(...)` provides atomicity across the DML statements in its batch without requiring an explicit transaction. DDL and DCL statements are never transactional. Even when called within an explicit transaction or as part of a batch, they are executed immediately and do not participate in any transaction. This section describes how to execute an explicit transaction for each transaction mode: Transaction mode and Two-phase Commit Transaction mode. ### Transaction Mode An example code for Transaction mode is as follows: ```java try { // Begin a transaction sqlSession.begin(); // Execute statements (SELECT/INSERT/UPDATE/DELETE) in the transaction ... // Commit the transaction sqlSession.commit(); } catch (UnknownTransactionStatusException e) { // If you catch `UnknownTransactionStatusException`, it indicates that the status of the // transaction, whether it has succeeded or not, is unknown. In such a case, you need to check if // the transaction is committed successfully or not and retry it if it failed. How to identify a // transaction status is delegated to users } catch (SqlException e) { // For other exceptions, you can try retrying the transaction // Rollback the transaction sqlSession.rollback(); // For `TransactionRetryableException`, you can basically retry the transaction. However, for // the other exceptions, the transaction may still fail if the cause of the exception is // nontransient. For such a case, you need to limit the number of retries and give up retrying } ``` If you catch `UnknownTransactionStatusException`, it indicates that the status of the transaction, whether it has succeeded or not, is unknown. In such a case, you need to check if the transaction is committed successfully or not and retry it if it fails. How to identify a transaction status is delegated to users. You may want to create a transaction status table and update it transactionally with other application data so that you can get the status of a transaction from the status table. If you catch another exception, you can try retrying the transaction. For `TransactionRetryableException`, you can basically retry the transaction. However, for the other exceptions, the transaction may still fail if the cause of the exception is nontransient. For such a case, you need to limit the number of retries and give up retrying. ### Two-phase Commit Transaction Mode Before reading this, please read [this document](../two-phase-commit-transactions.mdx) to learn the concept of Two-phase commit transactions. To begin a transaction for a coordinator, you can do as follows: ```java sqlSession.begin(); ``` To retrieve the transaction ID that the coordinator hands to participants, use `getTransactionId()`: ```java String transactionId = sqlSession .getTransactionId() .orElseThrow(() -> new IllegalStateException("transaction ID is not available")); ``` And to join a transaction for participants, you can do as follows: ```java sqlSession.join(transactionId); ``` An example code of Two-phase Commit Transaction mode is as follows: ```java try { // Begin a transaction sqlSession.begin(); // Execute statements (SELECT/INSERT/UPDATE/DELETE) in the transaction ... // Prepare the transaction sqlSession.prepare(); // Validate the transaction sqlSession.validate(); // Commit the transaction sqlSession.commit(); } catch (UnknownTransactionStatusException e) { // If you catch `UnknownTransactionStatusException` when committing the transaction, it // indicates that the status of the transaction, whether it has succeeded or not, is unknown. // In such a case, you need to check if the transaction is committed successfully or not and // retry it if it failed. How to identify a transaction status is delegated to users } catch (SqlException e) { // For other exceptions, you can try retrying the transaction // Rollback the transaction sqlSession.rollback(); // For `TransactionRetryableException`, you can basically retry the transaction. However, for // the other exceptions, the transaction may still fail if the cause of the exception is // nontransient. For that case, you need to limit the number of retries and give up retrying } ``` The exception handling is the same as Transaction mode. ## Get Metadata You can get metadata with the `SqlSession.getMetadata()` method as follows: ```java Metadata metadata = sqlSession.getMetadata(); ``` For more details, see the page of the Javadoc. ## References - [ScalarDB SQL Grammar](grammar.mdx) - [Two-phase Commit Transactions](../two-phase-commit-transactions.mdx) - [Javadoc for ScalarDB SQL](https://javadoc.io/doc/com.scalar-labs/scalardb-sql/3.18.0/index.html) ================================================ FILE: src/components/en-us/_certificate-management.mdx ================================================ You have several options for certificate management: 1. Management of private key and certificate files 1. Manage your private key and certificate files automatically by using [cert-manager](https://cert-manager.io/docs/). - This method can reduce maintenance or operation costs. For example, cert-manager automatically renews certificates before they expire and Scalar Helm Chart automatically mounts private key and certificate files on the Scalar product pods. - You cannot use a CA that cert-manager does not support. You can see the supported issuers in the [cert-manager documentation](https://cert-manager.io/docs/configuration/issuers/). 1. Manage your private key and certificate files manually. - You can issue and manage your private key and certificate files on your own by using your preferred method. - You can use any certificate even if cert-manager does not support it. - You must update secret resources when certificates expire. 1. Kinds of certificates 1. Use a trusted CA (signed certificate by third party). - You can use trusted certificates from a third-party certificate issuer. - You can encrypt packets. - You must pay costs to issue trusted certificates. 1. Use self-signed certificates. - You can reduce costs to issue certificates. - Reliability of certificates is lower than a trusted CA, but you can encrypt packets. In other words, you have the following four options: 1. Use a self-signed CA with automatic management. 1. Use a trusted CA with automatic management. 1. Use a self-signed CA with manual management. 1. Use a trusted CA with manual management. You should consider which method to use based on your security requirements. For guidance and related documentation for each method, refer to the following decision tree: ```mermaid flowchart TD A[Do you want to use
cert-manager to manage your
private key and certificate
files automatically?] A -->|Yes, I want to manage my
certificates automatically.| B A -->|No, I want to manage my
certificates manually by myself.| C B[Do you want to use a
self-signed CA or a trusted CA?] C[Do you want to use a
self-signed CA or a trusted CA?] B -->|I want to use a
self-signed CA.| D B -->|I want to use a
trusted CA.| E C -->|I want to use a
self-signed CA.| F C -->|I want to use a
trusted CA.| G D[See the Use a self-signed
CA with cert-manager to
manage your private key and
certificate files section.] E[See the Use a trusted
CA with cert-manager to
manage private key and
certificate files section.] F[See the Use your private
key and certificate files
section, and use the self-signed
certificate you generated.] G[See the Use your private key
and certificate files section,
and use the trusted certificate
generated by the third party.] ``` ================================================ FILE: src/components/en-us/_helm-command-usage.mdx ================================================ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock';

Prerequisites

1. Add the Scalar Helm Chart repository and update it to the latest version by using the `helm repo add` command and `helm repo update` command as follows: helm repo add scalar-labs https://scalar-labs.github.io/helm-charts helm repo update 1. Decide on the version of the ScalarDB product (strictly, the corresponding chart version) that you will deploy or upgrade. You can check the version by running the following command: helm search repo scalar-labs/<CHART_NAME> -l :::tip In this document (when you deploy the {props.productName}), run the following command: helm search repo scalar-labs/{props.helmChartName} -l ::: For example, you should see a similar output as below: NAME CHART VERSION APP VERSION scalar-labs/<CHART_NAME> 1.9.0 3.16.1 scalar-labs/<CHART_NAME> 1.8.1 3.16.1 scalar-labs/<CHART_NAME> 1.8.0 3.16.0 scalar-labs/<CHART_NAME> 1.7.6 3.15.5 scalar-labs/<CHART_NAME> 1.7.5 3.15.5 scalar-labs/<CHART_NAME> 1.7.4 3.15.5 scalar-labs/<CHART_NAME> 1.7.3 3.15.4 scalar-labs/<CHART_NAME> 1.7.2 3.15.3 scalar-labs/<CHART_NAME> 1.7.1 3.15.2 scalar-labs/<CHART_NAME> 1.7.0 3.15.1 scalar-labs/<CHART_NAME> 1.6.4 3.14.4 scalar-labs/<CHART_NAME> 1.6.3 3.14.3 scalar-labs/<CHART_NAME> 1.6.2 3.14.2 scalar-labs/<CHART_NAME> 1.6.1 3.14.1 scalar-labs/<CHART_NAME> 1.6.0 3.14.0 :::note - `APP VERSION` means the version of the ScalarDB product itself. First, check this version to decide which version of the ScalarDB product you will deploy or upgrade. - After checking the version under `APP VERSION` and deciding which version of the ScalarDB product you will deploy or upgrade, note the corresponding version under `CHART VERSION`. - If there are several of the same versions under `APP VERSION`, note the latest version under `CHART VERSION`. - For example: - If you want to deploy the ScalarDB product 3.16.1, note `1.9.0` as `CHART VERSION`. - If you want to deploy the ScalarDB product 3.15.5, note `1.7.6` as `CHART VERSION`. - If you want to deploy the ScalarDB product 3.14.4, note `1.6.4` as `CHART VERSION`. :::

Deploy, upgrade, or uninstall

Deploy the {props.productName} by using the `helm install` command as follows: helm install <RELEASE_NAME> scalar-labs/{props.helmChartName} -f {props.helmChartName}.yaml --namespace <KUBERNETES_NAMESPACE> --version <CHART_VERSION> :::note - Change `` to the arbitrary (unique) name of your deployment. - For the `--namespace` option, change `` to the name of the Kubernetes namespace that you want to deploy the {props.productName} to. - For the `--version` option, change `` to the version that you noted in the previous step. ::: Upgrade the existing {props.productName} deployment by using the `helm upgrade` command as follows: helm upgrade <RELEASE_NAME> scalar-labs/{props.helmChartName} -f {props.helmChartName}.yaml --namespace <KUBERNETES_NAMESPACE> --version <CHART_VERSION> :::note - Change `` to the arbitrary (unique) name of the deployment that you want to upgrade. - For the `--namespace` option, change `` to the name of the Kubernetes namespace that you want to upgrade the {props.productName} of. - For the `--version` option, change `` to the version that you noted in the previous step. ::: :::important Downgrading the version of the {props.productName} is not supported. When specifying a version number, you can do only the following: - Specify the same version as the existing deployment. For example, you might do this when updating configurations. - Specify a version that is greater than the existing deployment. For example, you might do this when upgrading the version of the {props.productName}. ::: Uninstall the existing {props.productName} deployment by using the `helm uninstall` command as follows: helm uninstall <RELEASE_NAME> --namespace <KUBERNETES_NAMESPACE> :::note - Change `` to the arbitrary (unique) name of the deployment that you want to uninstall. - For the `--namespace` option, change `` to the name of the Kubernetes namespace that you want to uninstall the {props.productName} of. ::: ================================================ FILE: src/components/en-us/_prerequisites-jdk-versions.mdx ================================================ - **[Oracle JDK](https://www.oracle.com/java/):** {props.versionNumbers} (LTS versions) - **OpenJDK distribution ([Eclipse Temurin](https://adoptium.net/temurin/), [Amazon Corretto](https://aws.amazon.com/corretto/), or [Microsoft Build of OpenJDK](https://learn.microsoft.com/en-us/java/openjdk/)):** {props.versionNumbers} (LTS versions) ================================================ FILE: src/components/en-us/_warning-license-key-contact.mdx ================================================ :::warning You need to have a license key (trial license or commercial license) to use {props.product}. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact-us). :::