Skip to main content
Version: 3.16

ScalarDB Analytics Configurations

This page provides a comprehensive reference for configuring all components of ScalarDB Analytics.

Overview​

ScalarDB Analytics consists of three main components that require configuration:

  1. ScalarDB Analytics server - The server that hosts the catalog information and metering services
  2. CLI client - The command-line interface for managing catalogs and data sources
  3. Spark integration - Configuration for using ScalarDB Analytics with Apache Spark

ScalarDB Analytics server configuration​

The server is configured using a standard Java properties file (for example, scalardb-analytics-server.properties) that defines database connections, network settings, licensing, and optional features.

Metadata database configurations​

Configure the metadata database that stores catalog information.

db.url​

  • Field: scalar.db.analytics.server.db.url
  • Description: The JDBC URL for the metadata database used by ScalarDB Analytics.

db.username​

  • Field: scalar.db.analytics.server.db.username
  • Description: The username for connecting to the metadata database.

db.password​

  • Field: scalar.db.analytics.server.db.password
  • Description: The password for the metadata database user.

Server network configuration​

Configure network settings including service ports and TLS/SSL encryption.

catalog.port​

  • Field: scalar.db.analytics.server.catalog.port
  • Description: Port for the catalog service.
  • Default value: 11051

metering.port​

  • Field: scalar.db.analytics.server.metering.port
  • Description: Port for the metering service.
  • Default value: 11052

tls.enabled​

  • Field: scalar.db.analytics.server.tls.enabled
  • Description: Enable TLS/SSL for secure communication.
  • Default value: false

tls.cert_chain_path​

  • Field: scalar.db.analytics.server.tls.cert_chain_path
  • Description: Path to the server certificate chain file. Required when tls.enabled is true.

tls.private_key_path​

  • Field: scalar.db.analytics.server.tls.private_key_path
  • Description: Path to the server private key file. Required when tls.enabled is true.

License configuration​

Configure your ScalarDB Analytics license.

licensing.license_key​

  • Field: scalar.db.analytics.server.licensing.license_key
  • Description: Your ScalarDB Analytics license key.

licensing.license_check_cert_pem​

  • Field: scalar.db.analytics.server.licensing.license_check_cert_pem
  • Description: License verification certificate as PEM string. Either this or license_check_cert_path must be specified.

licensing.license_check_cert_path​

  • Field: scalar.db.analytics.server.licensing.license_check_cert_path
  • Description: Path to license verification certificate file. Either this or license_check_cert_pem must be specified.

Metering storage configuration​

Configure storage for metering data.

metering.storage.provider​

  • Field: scalar.db.analytics.server.metering.storage.provider
  • Description: Storage provider for metering data (filesystem, aws-s3, azureblob, google-cloud-storage).

metering.storage.containerName​

  • Field: scalar.db.analytics.server.metering.storage.containerName
  • Description: Container/bucket name for cloud storage.
  • Default value: metering

metering.storage.path​

  • Field: scalar.db.analytics.server.metering.storage.path
  • Description: Local directory path. Required when provider is filesystem.

metering.storage.accessKeyId​

  • Field: scalar.db.analytics.server.metering.storage.accessKeyId
  • Description: Access key ID for cloud storage providers. Required for aws-s3, azureblob, and google-cloud-storage.

metering.storage.secretAccessKey​

  • Field: scalar.db.analytics.server.metering.storage.secretAccessKey
  • Description: Secret access key for cloud storage providers. Required for aws-s3, azureblob, and google-cloud-storage.

metering.storage.prefix​

  • Field: scalar.db.analytics.server.metering.storage.prefix
  • Description: Optional prefix for all storage paths.

CLI client configuration​

The CLI client requires connection settings to communicate with the ScalarDB Analytics server using a Java properties file (for example, client.properties).

Configuration properties​

This section describes the configuration properties.

Server connection configuration​

The following is a list of configurations for connecting to the server.

server.host​

  • Field: scalar.db.analytics.client.server.host
  • Description: Hostname or IP address of the ScalarDB Analytics server.

server.catalog.port​

  • Field: scalar.db.analytics.client.server.catalog.port
  • Description: Port number for the catalog service.
  • Default value: 11051

server.metering.port​

  • Field: scalar.db.analytics.client.server.metering.port
  • Description: Port number for the metering service.
  • Default value: 11052

TLS configuration​

The following is a list of configurations for TLS.

server.tls.enabled​

  • Field: scalar.db.analytics.client.server.tls.enabled
  • Description: Enable TLS/SSL for server connections.
  • Default value: false

server.tls.ca_root_cert_path​

  • Field: scalar.db.analytics.client.server.tls.ca_root_cert_path
  • Description: Path to the CA certificate file for verifying server certificates. Required when tls.enabled is true.

server.tls.override_authority​

  • Field: scalar.db.analytics.client.server.tls.override_authority
  • Description: Override the server authority for TLS verification (useful for testing).

Spark integration configuration​

To use ScalarDB Analytics with Apache Spark, configure your Spark application by adding the necessary settings to your Spark configuration file (spark-defaults.conf).

Spark Core configuration​

The following is a list of configurations for Spark Core.

spark.jars.packages​

  • Field: spark.jars.packages
  • Description: Maven coordinates for ScalarDB Analytics dependencies.

spark.extraListeners​

  • Field: spark.extraListeners
  • Description: Register the ScalarDB Analytics metering listener.

Catalog configuration​

The following is a list of configurations for the catalog.

spark.sql.catalog.<catalog-name>​

  • Field: spark.sql.catalog.<catalog-name>
  • Description: Register the ScalarDB Analytics catalog implementation. Replace <catalog-name> with the exact name of the catalog created in ScalarDB Analytics server. Use com.scalar.db.analytics.spark.catalog.ScalarDBAnalyticsCatalog as the value.
important

The <catalog-name> must match the catalog name created in ScalarDB Analytics server using the CLI. For example, if you created a catalog named production in the server, use spark.sql.catalog.production.

Server connection configurations​

The following is a list of configurations for the server connection.

spark.sql.catalog.<catalog-name>.server.host​

  • Field: spark.sql.catalog.<catalog-name>.server.host
  • Description: Hostname or IP address of the ScalarDB Analytics server.

spark.sql.catalog.<catalog-name>.server.catalog.port​

  • Field: spark.sql.catalog.<catalog-name>.server.catalog.port
  • Description: Port number for the catalog service.
  • Default value: 11051

spark.sql.catalog.<catalog-name>.server.metering.port​

  • Field: spark.sql.catalog.<catalog-name>.server.metering.port
  • Description: Port number for the metering service.
  • Default value: 11052

TLS/SSL configurations​

The following is a list of configurations for TLS/SSL.

spark.sql.catalog.<catalog-name>.server.tls.enabled​

  • Field: spark.sql.catalog.<catalog-name>.server.tls.enabled
  • Description: Enable TLS/SSL for server connections.
  • Default value: false

spark.sql.catalog.<catalog-name>.server.tls.ca_root_cert_path​

  • Field: spark.sql.catalog.<catalog-name>.server.tls.ca_root_cert_path
  • Description: Path to the CA certificate file for verifying server certificates. Required when tls.enabled is true.

spark.sql.catalog.<catalog-name>.server.tls.override_authority​

  • Field: spark.sql.catalog.<catalog-name>.server.tls.override_authority
  • Description: Override the server authority for TLS verification.

Replace <catalog-name> with your chosen catalog name (for example, analytics).

Configuration examples​

This section provides some configuration examples.

Basic development configuration​

The following are examples of configurations for the server, CLI client, and Spark.

Server configuration (scalardb-analytics-server.properties)​

# Metadata database
scalar.db.analytics.server.db.url=jdbc:postgresql://localhost:5432/scalardb_analytics
scalar.db.analytics.server.db.username=dev_user
scalar.db.analytics.server.db.password=dev_password

# License
scalar.db.analytics.server.licensing.license_key=YOUR_DEV_LICENSE_KEY
scalar.db.analytics.server.licensing.license_check_cert_path=/path/to/license_cert.pem

# Metering storage (filesystem for development)
scalar.db.analytics.server.metering.storage.provider=filesystem
scalar.db.analytics.server.metering.storage.path=/tmp/scalardb-analytics-metering

CLI client configuration (client.properties)​

scalar.db.analytics.client.server.host=localhost

Spark configuration (spark-defaults.conf)​

spark.jars.packages                         com.scalar-labs:scalardb-analytics-spark-all-3.5_2.12:3.16.2
spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener
spark.sql.catalog.analytics com.scalar.db.analytics.spark.catalog.ScalarDBAnalyticsCatalog
spark.sql.catalog.analytics.server.host localhost

Production configuration with TLS​

The following are examples of configurations for TLS, CLI client, and Spark in production environments.

Server configuration (scalardb-analytics-server.properties)​

# Metadata database
scalar.db.analytics.server.db.url=jdbc:postgresql://db.internal:5432/scalardb_analytics_prod
scalar.db.analytics.server.db.username=analytics_prod
scalar.db.analytics.server.db.password=your_secure_password

# gRPC ports
scalar.db.analytics.server.catalog.port=11051
scalar.db.analytics.server.metering.port=11052

# TLS
scalar.db.analytics.server.tls.enabled=true
scalar.db.analytics.server.tls.cert_chain_path=/path/to/server.crt
scalar.db.analytics.server.tls.private_key_path=/path/to/server.key

# License
scalar.db.analytics.server.licensing.license_key=YOUR_LICENSE_KEY
scalar.db.analytics.server.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIID...certificate content...\n-----END CERTIFICATE-----

# Metering storage (S3)
scalar.db.analytics.server.metering.storage.provider=aws-s3
scalar.db.analytics.server.metering.storage.containerName=analytics-metering
scalar.db.analytics.server.metering.storage.accessKeyId=AKIAIOSFODNN7EXAMPLE
scalar.db.analytics.server.metering.storage.secretAccessKey=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
scalar.db.analytics.server.metering.storage.prefix=prod/

CLI Client configuration (client.properties)​

scalar.db.analytics.client.server.host=analytics.example.com
scalar.db.analytics.client.server.tls.enabled=true
scalar.db.analytics.client.server.tls.ca_root_cert_path=/path/to/cert.pem

Spark configuration (spark-defaults.conf)​

spark.jars.packages                                     com.scalar-labs:scalardb-analytics-spark-all-3.5_2.12:3.16.2
spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener
spark.sql.catalog.analytics com.scalar.db.analytics.spark.catalog.ScalarDBAnalyticsCatalog
spark.sql.catalog.analytics.server.host analytics.example.com
spark.sql.catalog.analytics.server.tls.enabled true
spark.sql.catalog.analytics.server.tls.ca_root_cert_path /path/to/cert.pem

Next steps​