Deploy ScalarDB Analytics in Public Cloud Environments

This guide explains how to deploy ScalarDB Analytics in a public cloud environment. ScalarDB Analytics consists of two main components: a ScalarDB Analytics server and Apache Spark. In this guide, you can choose Amazon EMR, Databricks, or Azure Synapse Analytics for the Spark environment.

For details about ScalarDB Analytics, refer to ScalarDB Analytics Design.

Deploy ScalarDB Analytics server

ScalarDB Analytics requires a catalog server to manage metadata and data source connections. The catalog server should be deployed by using Helm Charts on a Kubernetes cluster.

For detailed deployment instructions, see How to install Scalar products through AWS Marketplace.

After deploying the catalog server, note the following information for Spark configuration:

• Catalog server host address
• Catalog port (default: 11051)
• Metering port (default: 11052)

Deploy Spark with ScalarDB Analytics

After deploying the catalog server, you can configure and deploy Spark with ScalarDB Analytics by using managed Spark services.

Supported managed Spark services and their application types

ScalarDB Analytics supports the following managed Spark services and application types.

Public Cloud Service	Spark Driver	Spark Connect	JDBC
Amazon EMR (EMR on EC2)	✅	✅	❌
Databricks	✅	❌	✅
Azure Synapse Analytics	✅	❌	❌

Configure and deploy

Select your public cloud environment, and follow the instructions to set up and deploy Spark with ScalarDB Analytics.

Amazon EMR

Use Amazon EMR

You can use Amazon EMR (EMR on EC2) to run analytical queries through ScalarDB Analytics. For the basics to launch an EMR cluster, please refer to the AWS EMR on EC2 documentation.

ScalarDB Analytics configuration

To enable ScalarDB Analytics, you need to add the following configuration to the Software setting when you launch an EMR cluster. Be sure to replace the content in the angle brackets:

[
  {
    "Classification": "spark-defaults",
    "Properties": {
      "spark.jars.packages": "com.scalar-labs:scalardb-analytics-spark-all-<SPARK_VERSION>_<SCALA_VERSION>:<SCALARDB_ANALYTICS_VERSION>",
      "spark.extraListeners": "com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener",
      "spark.sql.catalog.<CATALOG_NAME>": "com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog",
      "spark.sql.catalog.<CATALOG_NAME>.server.host": "<CATALOG_SERVER_HOST>",
      "spark.sql.catalog.<CATALOG_NAME>.server.catalog.port": "11051",
      "spark.sql.catalog.<CATALOG_NAME>.server.metering.port": "11052"
    }
  }
]

The following describes what you should change the content in the angle brackets to:

• <SPARK_VERSION>: The version of Spark (3.5 or 3.4).
• <SCALA_VERSION>: The version of Scala used to build Spark (2.13 or 2.12).
• <SCALARDB_ANALYTICS_VERSION>: The version of ScalarDB Analytics.
• <CATALOG_NAME>: The name of the catalog. This must match a catalog created on the ScalarDB Analytics server.
• <CATALOG_SERVER_HOST>: The host address of your ScalarDB Analytics server.

For more details, refer to Set up ScalarDB Analytics in the Spark configuration.

Run analytical queries via the Spark driver

After the EMR Spark cluster has launched, you can use ssh to connect to the primary node of the EMR cluster and run your Spark application. For details on how to create a Spark driver application, refer to Spark driver application.

Run analytical queries via Spark Connect

You can use Spark Connect to run your Spark application remotely by using the EMR cluster that you launched.

You first need to configure the Software setting in the same way as the Spark driver application. You also need to set the following configuration to enable Spark Connect.

Allow inbound traffic for a Spark Connect server

1. Create a security group to allow inbound traffic for a Spark Connect server. (Port 15001 is the default).
2. Allow the role of "Amazon EMR service role" to attach the security group to the primary node of the EMR cluster.
3. Add the security group to the primary node of the EMR cluster as "Additional security groups" when you launch the EMR cluster.

Launch the Spark Connect server via a bootstrap action

1. Create a script file to launch the Spark Connect server as follows:

   #!/usr/bin/env bash
   
   set -eu -o pipefail
   cd /var/lib/spark
   sudo -u spark /usr/lib/spark/sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_<SCALA_VERSION>:<SPARK_FULL_VERSION>,com.scalar-labs:scalardb-analytics-spark-all-<SPARK_VERSION>_<SCALA_VERSION>:<SCALARDB_ANALYTICS_VERSION>

   The following describes what you should change the content in the angle brackets to:

   • <SCALA_VERSION>: The major and minor version of Scala that matches your Spark installation (2.12 or 2.13).
   • <SPARK_FULL_VERSION>: The full version of Spark you are using (such as 3.5.3).
   • <SPARK_VERSION>: The major and minor version of Spark you are using (3.4 or 3.5).
   • <SCALARDB_ANALYTICS_VERSION>: The version of ScalarDB Analytics.

2. Upload the script file to S3.

3. Allow the role of "EC2 instance profile for Amazon EMR" to access the uploaded script file in S3.

4. Add the uploaded script file to "Bootstrap actions" when you launch the EMR cluster.

Run analytical queries

You can run your Spark application via Spark Connect from anywhere by using the remote URL of the Spark Connect server, which is sc://<PRIMARY_NODE_PUBLIC_HOSTNAME>:15001.

For details on how to create a Spark application by using Spark Connect, refer to Spark Connect application.

Databricks

Use Databricks

You can use Databricks to run analytical queries through ScalarDB Analytics.

note

Note that Databricks provides a modified version of Apache Spark, which works differently from the original Apache Spark.

Prepare an init script for loading the ScalarDB Analytics library JAR

1. Download the ScalarDB Analytics library JAR file from the Maven repository. Choose the appropriate JAR file based on your Spark, Scala, and ScalarDB versions:

   • scalardb-analytics-spark-all-3.4_2.13 (Spark v3.4, Scala v2.13)
   • scalardb-analytics-spark-all-3.5_2.13 (Spark v3.5, Scala v2.13)
   • scalardb-analytics-spark-all-3.4_2.12 (Spark v3.4, Scala v2.12)
   • scalardb-analytics-spark-all-3.5_2.12 (Spark v3.5, Scala v2.12)

2. Upload the JAR file to the Databricks workspace.

3. Create an init script as follows, replacing <PATH_TO_YOUR_JAR_FILE_IN_WORKSPACE> with the path to your JAR file in the Databricks workspace:

   #!/bin/bash
   
   # Target directories
   TARGET_DIRECTORIES=("/databricks/jars" "/databricks/hive_metastore_jars")
   JAR_PATH="<PATH_TO_YOUR_JAR_FILE_IN_WORKSPACE>"
   
   # Copy the JAR file to the target directories
   for TARGET_DIR in "${TARGET_DIRECTORIES[@]}"; do
    mkdir -p "$TARGET_DIR"
    cp "$JAR_PATH" "$TARGET_DIR/"
   done

4. Upload the init script to the Databricks workspace.

Launch Databricks compute

ScalarDB Analytics works with all-purpose compute on Databricks. When you launch compute, you need to configure the compute to enable ScalarDB Analytics as follows:

1. Select Create compute in the Compute menu.

2. Select Unrestricted from the Policy dropdown menu.

3. Select an appropriate Databricks runtime version that supports Spark 3.4 or 3.5.

4. Go to the Advanced section. In the Access mode tab, select Manual as the access mode, and choose No isolation shared.

5. In the Advanced section, select the Spark tab, and enter the following configurations in Spark config:

   spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener
   spark.sql.catalog.<CATALOG_NAME> com.scalar.db.analytics.spark.ScalarDBAnalyticsCatalog
   spark.sql.catalog.<CATALOG_NAME>.server.host <CATALOG_SERVER_HOST>
   spark.sql.catalog.<CATALOG_NAME>.server.catalog.port 11051
   spark.sql.catalog.<CATALOG_NAME>.server.metering.port 11052

   Replace the placeholders:

   • <CATALOG_NAME>: The name of the catalog. This must match a catalog created on the ScalarDB Analytics server.
   • <CATALOG_SERVER_HOST>: The host address of your ScalarDB Analytics server.

6. In the Advanced section, select the init scripts tab, and specify the path to the init script in the workspace you uploaded.

7. Select Create.

Run analytical queries via the Spark driver

You can run your Spark application on the properly configured Databricks compute with Databricks Notebook or Databricks Jobs to access the tables in ScalarDB Analytics. To run the Spark application, you can migrate your Pyspark, Scala, or Spark SQL application to Databricks Notebook, or use Databricks Jobs to run your Spark application. ScalarDB Analytics works with task types for Notebook, Python, JAR, and SQL.

For more details on how to use Databricks Jobs, refer to the Databricks Jobs documentation.

Run analytical queries via the JDBC driver

Databricks supports JDBC to run SQL jobs on compute. After compute is launched, you can get the JDBC URL of the compute in the Advanced > JDBC/ODBC tab. To connect to the compute by using JDBC, you need to add the Databricks JDBC driver to your application dependencies. For example, if you are using Gradle, you can add the following dependency to your build.gradle file after replacing <DRIVER_VERSION> with the version of the Databricks JDBC driver you want to use:

implementation("com.databricks:databricks-jdbc:<DRIVER_VERSION>")

Then, you can connect to the compute by using JDBC with the JDBC URL (<YOUR_COMPUTE_JDBC_URL>), as is common with JDBC applications.

Class.forName("com.databricks.client.jdbc.Driver");
String url = "<YOUR_COMPUTE_JDBC_URL>";
Connection conn = DriverManager.getConnection(url);

For more details on how to use JDBC with Databricks, refer to the Databricks JDBC Driver documentation.

Azure Synapse Analytics

Use Azure Synapse Analytics

You can use Azure Synapse Analytics to run analytical queries through ScalarDB Analytics. For the basics of Azure Synapse Analytics, please refer to the Azure Synapse Analytics documentation.

note

Azure Synapse Analytics uses serverless Apache Spark pools to run Spark workloads. For more details about Spark pools, see Apache Spark in Azure Synapse Analytics.

Prerequisites

Before configuring ScalarDB Analytics on Azure Synapse, ensure the following requirements are met.

Synapse workspace

A Synapse workspace is a collaboration environment for analytics in Azure. You must create a Synapse workspace before you run Spark workloads. For more details about creating a Synapse workspace, see Quickstart: Create a Synapse workspace.

note

This guide assumes that you have created a Synapse workspace with a Managed Virtual Network enabled. A Managed Virtual Network is recommended for network isolation and security. For more details, see Azure Synapse Analytics Managed Virtual Network.

Storage permissions

If you plan to run a Spark driver application, both the Synapse workspace managed identity and the user account that submits the job require the Storage Blob Data Contributor role on the storage account. This role must be assigned by a subscription owner or user access administrator.

ScalarDB Analytics configuration

To enable ScalarDB Analytics, add the following configuration to the Spark pool configuration in Synapse Studio.

1. Go to Synapse Studio, select manage Manage, and choose Apache Spark pools.

2. Select your Spark pool, and then select Apache Spark configuration.

3. Create a new configuration or edit an existing one with the following properties:

   spark.jars.packages com.scalar-labs:scalardb-analytics-spark-all-<SPARK_VERSION>_<SCALA_VERSION>:<SCALARDB_ANALYTICS_VERSION>
   spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener
   spark.sql.catalog.<CATALOG_NAME> com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog
   spark.sql.catalog.<CATALOG_NAME>.server.host <CATALOG_SERVER_IP>
   spark.sql.catalog.<CATALOG_NAME>.server.catalog.port 11051
   spark.sql.catalog.<CATALOG_NAME>.server.metering.port 11052

The following describes what you should change the content in the angle brackets to:

• <SPARK_VERSION>: The version of Spark (3.4 or 3.5).
• <SCALA_VERSION>: The version of Scala used to build Spark (2.12 or 2.13).
• <SCALARDB_ANALYTICS_VERSION>: The version of ScalarDB Analytics.
• <CATALOG_NAME>: The name of the catalog. This must match a catalog created on the ScalarDB Analytics server.
• <CATALOG_SERVER_IP>: The IP address of your Managed Private Endpoint for the ScalarDB Analytics server.

Set up private connectivity

ScalarDB Analytics requires network connectivity from the Managed Virtual Network of the Synapse workspace to both the ScalarDB Analytics server and data sources. To establish this connectivity, you must use a Private Link Service and Managed Private Endpoints as follows:

1. Connectivity between the Synapse workspace and the ScalarDB Analytics server: Create a Private Link Service for the Internal Load Balancer in Azure Kubernetes Service (AKS), then create a Managed Private Endpoint in the Managed Virtual Network of the Synapse workspace.
2. Connectivity between the Synapse workspace and data sources: Create a Managed Private Endpoint in the Managed Virtual Network of the Synapse workspace directly to each data source (for example, Azure Database for PostgreSQL).

caution

If you are using Azure Database for PostgreSQL Flexible Server, it must be created with Public access mode. VNet integration mode does not support adding private endpoints after creation. For more details, see the Networking section of the Azure Database for PostgreSQL documentation.

Create a Private Link Service for the ScalarDB Analytics server

To create an Internal Load Balancer for the ScalarDB Analytics server, set scalarDbAnalyticsServer.service.type to LoadBalancer in your custom values file for the Helm chart.

note

The Helm chart includes the service.beta.kubernetes.io/azure-load-balancer-internal: "true" annotation by default, so the Load Balancer will be created as an Internal Load Balancer.

After deploying with this configuration, proceed with the following steps.

1. Verify that an internal IP address is assigned to the ScalarDB Analytics server service:

   kubectl get svc -n <NAMESPACE> <SCALARDB_ANALYTICS_SERVICE_NAME>

   Replace <NAMESPACE> with the namespace where ScalarDB Analytics server is deployed and <SCALARDB_ANALYTICS_SERVICE_NAME> with the name of the service (for example, scalardb-analytics-server). Confirm that the service has a private IP address (for example, 10.x.x.x) assigned in the EXTERNAL-IP column. For Internal Load Balancer services, the EXTERNAL-IP shows a private IP address instead of a public IP.

2. Create a Private Link Service in Azure Portal:

   1. Go to Private Link, select Private Link Services, and choose Create.
   2. Select the Internal Load Balancer for the ScalarDB Analytics server service.
   3. Configure the Source NAT subnet and access security.

3. Create a Managed Private Endpoint in Synapse Studio:

   1. Go to Manage, select Managed private endpoints, and choose New.
   2. Select Private Link Service, and then choose the Private Link Service you created.
   3. After creation, approve the connection in the Private Link Service settings.

4. Note the IP address of the Managed Private Endpoint for use in Spark configuration.

Create a Managed Private Endpoint for the data source

1. In Synapse Studio, go to Manage, select Managed private endpoints, and choose New.
2. Select your database type (for example, Azure Database for PostgreSQL Flexible Server).
3. Select your database server, and then create the endpoint.
4. Approve the connection in the database server's networking settings:
   1. Go to Azure Portal, select Your database server, choose Networking, and select Private access.
   2. Select the pending connection, and then approve it.

Run analytical queries via the Spark driver

To run Spark driver applications:

1. Upload your Spark driver application JAR to Azure Data Lake Storage Gen2.
2. Go to Develop, and select New Spark job definition.
3. Configure the job:
   • Language: Spark (Scala/Java)
   • Main class: Your application's main class
   • Main definition file: Path to your JAR file (for example, abfss://<CONTAINER>@<STORAGE>.dfs.core.windows.net/path/to/app.jar)
     • <CONTAINER>: Your Azure Data Lake Storage Gen2 container name
     • <STORAGE>: Your Azure storage account name
   • Spark pool: Your configured Spark pool
4. Submit the job.

note

Make sure the required permissions are set as described in the Prerequisites section.

Run interactive queries using Synapse notebooks

After configuring the Spark pool, you can run interactive queries by using Synapse notebooks.

1. Go to Synapse Studio, select Develop, and choose New notebook.

2. Attach the configured Spark pool.

3. Run SQL queries by using the %%sql magic command:

   %%sql
   -- List available catalogs
   SHOW CATALOGS

   %%sql
   -- List databases in ScalarDB catalog
   SHOW DATABASES IN <CATALOG_NAME>

   %%sql
   -- Query data
   SELECT * FROM <CATALOG_NAME>.<DATA_SOURCE_NAME>.<NAMESPACE_NAME>.<TABLE_NAME> LIMIT 10

   Replace the above angle brackets with your actual catalog, data source, namespace, and table names configured in the ScalarDB Analytics server.