ScalarDB Analytics Configurations

This page provides a comprehensive reference for configuring all components of ScalarDB Analytics.

Overview

ScalarDB Analytics consists of three main components that require configuration:

1. ScalarDB Analytics server - The server that hosts the catalog information and metering services
2. CLI client - The command-line interface for managing catalogs and data sources
3. Spark integration - Configuration for using ScalarDB Analytics with Apache Spark

ScalarDB Analytics server configuration

The server is configured using a standard Java properties file (for example, scalardb-analytics-server.properties) that defines database connections, network settings, licensing, and optional features.

Metadata database configurations

Configure the metadata database that stores catalog information.

db.url

• Field: scalar.db.analytics.server.db.url
• Description: The JDBC URL for the metadata database used by ScalarDB Analytics.

db.username

• Field: scalar.db.analytics.server.db.username
• Description: The username for connecting to the metadata database.

db.password

• Field: scalar.db.analytics.server.db.password
• Description: The password for the metadata database user.

Server network configuration

Configure network settings including service ports and TLS/SSL encryption.

catalog.port

• Field: scalar.db.analytics.server.catalog.port
• Description: Port for the catalog service.
• Default value: 11051

metering.port

• Field: scalar.db.analytics.server.metering.port
• Description: Port for the metering service.
• Default value: 11052

tls.enabled

• Field: scalar.db.analytics.server.tls.enabled
• Description: Enable TLS/SSL for secure communication.
• Default value: false

tls.cert_chain_path

• Field: scalar.db.analytics.server.tls.cert_chain_path
• Description: Path to the server certificate chain file. Required when tls.enabled is true.

tls.private_key_path

• Field: scalar.db.analytics.server.tls.private_key_path
• Description: Path to the server private key file. Required when tls.enabled is true.

License configuration

Configure your ScalarDB Analytics license.

licensing.license_key

• Field: scalar.db.analytics.server.licensing.license_key
• Description: Your ScalarDB Analytics license key.

licensing.license_check_cert_pem

• Field: scalar.db.analytics.server.licensing.license_check_cert_pem
• Description: License verification certificate as PEM string. Either this or license_check_cert_path must be specified.

licensing.license_check_cert_path

• Field: scalar.db.analytics.server.licensing.license_check_cert_path
• Description: Path to license verification certificate file. Either this or license_check_cert_pem must be specified.

Metering storage configuration

Configure storage for metering data.

metering.storage.provider

• Field: scalar.db.analytics.server.metering.storage.provider
• Description: Storage provider for metering data (filesystem, aws-s3, azureblob, google-cloud-storage).

metering.storage.containerName

• Field: scalar.db.analytics.server.metering.storage.containerName
• Description: Container/bucket name for cloud storage.
• Default value: metering

metering.storage.path

• Field: scalar.db.analytics.server.metering.storage.path
• Description: Local directory path. Required when provider is filesystem.

metering.storage.accessKeyId

• Field: scalar.db.analytics.server.metering.storage.accessKeyId
• Description: Access key ID for cloud storage providers. Required for aws-s3, azureblob, and google-cloud-storage.

metering.storage.secretAccessKey

• Field: scalar.db.analytics.server.metering.storage.secretAccessKey
• Description: Secret access key for cloud storage providers. Required for aws-s3, azureblob, and google-cloud-storage.

metering.storage.prefix

• Field: scalar.db.analytics.server.metering.storage.prefix
• Description: Optional prefix for all storage paths.

CLI client configuration

The CLI client requires connection settings to communicate with the ScalarDB Analytics server using a Java properties file (for example, client.properties).

Configuration properties

This section describes the configuration properties.

Server connection configuration

The following is a list of configurations for connecting to the server.

server.host

• Field: scalar.db.analytics.client.server.host
• Description: Hostname or IP address of the ScalarDB Analytics server.

server.catalog.port

• Field: scalar.db.analytics.client.server.catalog.port
• Description: Port number for the catalog service.
• Default value: 11051

server.metering.port

• Field: scalar.db.analytics.client.server.metering.port
• Description: Port number for the metering service.
• Default value: 11052

TLS configuration

The following is a list of configurations for TLS.

server.tls.enabled

• Field: scalar.db.analytics.client.server.tls.enabled
• Description: Enable TLS/SSL for server connections.
• Default value: false

server.tls.ca_root_cert_path

• Field: scalar.db.analytics.client.server.tls.ca_root_cert_path
• Description: Path to the CA certificate file for verifying server certificates. Required when tls.enabled is true.

server.tls.override_authority

• Field: scalar.db.analytics.client.server.tls.override_authority
• Description: Override the server authority for TLS verification (useful for testing).

Spark integration configuration

To use ScalarDB Analytics with Apache Spark, configure your Spark application by adding the necessary settings to your Spark configuration file (spark-defaults.conf).

Spark Core configuration

The following is a list of configurations for Spark Core.

spark.jars.packages

• Field: spark.jars.packages
• Description: Maven coordinates for ScalarDB Analytics dependencies.

spark.extraListeners

• Field: spark.extraListeners
• Description: Register the ScalarDB Analytics metering listener.

Catalog configuration

The following is a list of configurations for the catalog.

spark.sql.catalog.<catalog-name>

• Field: spark.sql.catalog.<catalog-name>
• Description: Register the ScalarDB Analytics catalog implementation. Replace <catalog-name> with the exact name of the catalog created in ScalarDB Analytics server. Use com.scalar.db.analytics.spark.catalog.ScalarDBAnalyticsCatalog as the value.

important

The <catalog-name> must match the catalog name created in ScalarDB Analytics server using the CLI. For example, if you created a catalog named production in the server, use spark.sql.catalog.production.

Server connection configurations

The following is a list of configurations for the server connection.

spark.sql.catalog.<catalog-name>.server.host

• Field: spark.sql.catalog.<catalog-name>.server.host
• Description: Hostname or IP address of the ScalarDB Analytics server.

spark.sql.catalog.<catalog-name>.server.catalog.port

• Field: spark.sql.catalog.<catalog-name>.server.catalog.port
• Description: Port number for the catalog service.
• Default value: 11051

spark.sql.catalog.<catalog-name>.server.metering.port

• Field: spark.sql.catalog.<catalog-name>.server.metering.port
• Description: Port number for the metering service.
• Default value: 11052

TLS/SSL configurations

The following is a list of configurations for TLS/SSL.

spark.sql.catalog.<catalog-name>.server.tls.enabled

• Field: spark.sql.catalog.<catalog-name>.server.tls.enabled
• Description: Enable TLS/SSL for server connections.
• Default value: false

spark.sql.catalog.<catalog-name>.server.tls.ca_root_cert_path

• Field: spark.sql.catalog.<catalog-name>.server.tls.ca_root_cert_path
• Description: Path to the CA certificate file for verifying server certificates. Required when tls.enabled is true.

spark.sql.catalog.<catalog-name>.server.tls.override_authority

• Field: spark.sql.catalog.<catalog-name>.server.tls.override_authority
• Description: Override the server authority for TLS verification.

Replace <catalog-name> with your chosen catalog name (for example, analytics).

Configuration examples

This section provides some configuration examples.

Basic development configuration

The following are examples of configurations for the server, CLI client, and Spark.

Server configuration (scalardb-analytics-server.properties)

# Metadata database
scalar.db.analytics.server.db.url=jdbc:postgresql://localhost:5432/scalardb_analytics
scalar.db.analytics.server.db.username=dev_user
scalar.db.analytics.server.db.password=dev_password

# License
scalar.db.analytics.server.licensing.license_key=YOUR_DEV_LICENSE_KEY
scalar.db.analytics.server.licensing.license_check_cert_path=/path/to/license_cert.pem

# Metering storage (filesystem for development)
scalar.db.analytics.server.metering.storage.provider=filesystem
scalar.db.analytics.server.metering.storage.path=/tmp/scalardb-analytics-metering

CLI client configuration (client.properties)

scalar.db.analytics.client.server.host=localhost

Spark configuration (spark-defaults.conf)

spark.jars.packages                         com.scalar-labs:scalardb-analytics-spark-all-3.5_2.12:3.16.2
spark.extraListeners                        com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener
spark.sql.catalog.analytics                 com.scalar.db.analytics.spark.catalog.ScalarDBAnalyticsCatalog
spark.sql.catalog.analytics.server.host     localhost

Production configuration with TLS

The following are examples of configurations for TLS, CLI client, and Spark in production environments.

Server configuration (scalardb-analytics-server.properties)

# Metadata database
scalar.db.analytics.server.db.url=jdbc:postgresql://db.internal:5432/scalardb_analytics_prod
scalar.db.analytics.server.db.username=analytics_prod
scalar.db.analytics.server.db.password=your_secure_password

# gRPC ports
scalar.db.analytics.server.catalog.port=11051
scalar.db.analytics.server.metering.port=11052

# TLS
scalar.db.analytics.server.tls.enabled=true
scalar.db.analytics.server.tls.cert_chain_path=/path/to/server.crt
scalar.db.analytics.server.tls.private_key_path=/path/to/server.key

# License
scalar.db.analytics.server.licensing.license_key=YOUR_LICENSE_KEY
scalar.db.analytics.server.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIID...certificate content...\n-----END CERTIFICATE-----

# Metering storage (S3)
scalar.db.analytics.server.metering.storage.provider=aws-s3
scalar.db.analytics.server.metering.storage.containerName=analytics-metering
scalar.db.analytics.server.metering.storage.accessKeyId=AKIAIOSFODNN7EXAMPLE
scalar.db.analytics.server.metering.storage.secretAccessKey=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
scalar.db.analytics.server.metering.storage.prefix=prod/

CLI Client configuration (client.properties)

scalar.db.analytics.client.server.host=analytics.example.com
scalar.db.analytics.client.server.tls.enabled=true
scalar.db.analytics.client.server.tls.ca_root_cert_path=/path/to/cert.pem

Spark configuration (spark-defaults.conf)

spark.jars.packages                                     com.scalar-labs:scalardb-analytics-spark-all-3.5_2.12:3.16.2
spark.extraListeners                                    com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener
spark.sql.catalog.analytics                             com.scalar.db.analytics.spark.catalog.ScalarDBAnalyticsCatalog
spark.sql.catalog.analytics.server.host                 analytics.example.com
spark.sql.catalog.analytics.server.tls.enabled          true
spark.sql.catalog.analytics.server.tls.ca_root_cert_path /path/to/cert.pem

Next steps

• Create a ScalarDB Analytics Catalog - Learn how to create catalogs and add data sources
• Run Analytical Queries Through ScalarDB Analytics - Start running queries with your configuration
• Deploy ScalarDB Analytics in Public Cloud Environments - Deploy ScalarDB Analytics in production