Skip to main content
Version: 3.13

Getting Started with ScalarDB Analytics with Spark

This guide explains how to get started with ScalarDB Analytics with Spark.

Prerequisites​

Before you can run queries with ScalarDB Analytics with Spark, you'll need to set up ScalarDB tables and install Apache Spark.

Set up ScalarDB tables​

To use ScalarDB Analytics with Spark, you need at least one underlying database in ScalarDB to run analytical queries on. If you have your own underlying database set up in ScalarDB, you can skip this section and use your database instead.

If you don't have your own database set up yet, you can set up ScalarDB with a sample underlying database by following the instructions in Run Analytical Queries on Sample Data by Using ScalarDB Analytics with Spark.

Install Apache Spark​

You also need a packaged release of Apache Spark. If you already have Spark installed, you can skip this section.

If you need Spark, you can download it from the Spark website. After downloading the compressed Spark file, you'll need to uncompress the file by running the following command, replacing X.X.X with the version of Spark that you downloaded:

$ tar xf spark-X.X.X-bin-hadoop3.tgz

Then, enter the directory by running the following command, again replacing X.X.X with the version of Spark that you downloaded:

$ cd spark-X.X.X-bin-hadoop3

Configure the Spark shell​

The following explains how to perform interactive analysis by using the Spark shell.

Since ScalarDB Analytics with Spark is available on the Maven Central Repository, you can use it to enable ScalarDB Analytics with Spark in the Spark shell by using the --packages option, replacing <SPARK_VERSION>_<SCALA_VERSION>:<SCALARDB_ANALYTICS_WITH_SPARK_VERSION> with the versions that you're using.

$ ./bin/spark-shell --packages com.scalar-labs:scalardb-analytics-spark-<SPARK_VERSION>_<SCALA_VERSION>:<SCALARDB_ANALYTICS_WITH_SPARK_VERSION>
warning

ScalarDB Analytics with Spark offers different artifacts for various Spark and Scala versions, provided in the format scalardb-analytics-spark-<SPARK_VERSION>_<SCALA_VERSION>. Make sure that you select the artifact matching the Spark and Scala versions you're using.

For reference, see Version Compatibility of ScalarDB Analytics with Spark.

Next, you'll need to configure the ScalarDB Analytics with Spark environment in the shell. ScalarDB Analytics with Spark provides a helper method for this purpose, which get everything set up to run analytical queries for you.

spark-shell> import com.scalar.db.analytics.spark.implicits._
spark-shell> spark.setupScalarDbAnalytics(
           | // ScalarDB config file
           | configPath = "/<PATH_TO_YOUR_SCALARDB_PROPERTIES>/config.properties", 
           | // Namespaces in ScalarDB to import
           | namespaces = Set("<YOUR_NAMESPACE_NAME_1>", "<YOUR_NAMESPACE_NAME_2>"),
           | // License information
           | license = License.certPath("""{"your":"license", "key":"in", "json":"format"}""", "/<PATH_TO_YOUR_LICENSE>/cert.pem")
           | )

Now, you can read data from the tables in the underlying databases of ScalarDB and run any arbitrary analytical queries through the Spark Dataset API. For example:

spark-shell> spark.sql("select * from <YOUR_NAMESPACE_NAME_1>.<YOUR_TABLE_NAME>").show()

Implement and submit a Spark application​

This section explains how to implement a Spark application with ScalarDB Analytics with Spark and submit it to the Spark cluster.

You can integrate ScalarDB Analytics with Spark into your application by using build tools like SBT, Gradle, or Maven.

For Gradle projects, add the following to your build.gradle.kts file, replacing <SPARK_VERSION>_<SCALA_VERSION>:<SCALARDB_ANALYTICS_WITH_SPARK_VERSION> with the versions that you're using:

implementation("com.scalar-labs:scalardb-analytics-spark-<SPARK_VERSION>_<SCALA_VERSION>:<SCALARDB_ANALYTICS_WITH_SPARK_VERSION>")

After integrating ScalarDB Analytics with Spark into your application, you can use the same helper method explained above to configure ScalarDB Analytics with Spark in your Spark application.

The following is a sample application that uses Scala:

import com.scalar.db.analytics.spark.implicits._

object YourApp {
  def main(args: Array[String]): Unit = {
    // Initialize SparkSession as usual
    val spark = SparkSession.builder.appName("<YOUR_APPLICATION_NAME>").getOrCreate()
    // Setup ScalarDB Analytics with Spark via helper method
    spark.setupScalarDbAnalytics(
      // ScalarDB config file
      configPath = "/<PATH_TO_YOUR_SCALARDB_PROPERTIES>/config.properties", 
      // Namespaces in ScalarDB to import
      namespaces = Set("<YOUR_NAMESPACE_NAME_1>", "<YOUR_NAMESPACE_NAME_2>"),
      // License information
      license = License.certPath("""{"your":"license", "key":"in", "json":"format"}""", "/<PATH_TO_YOUR_LICENSE>/cert.pem")
    )
    // Run arbitrary queries
    spark.sql("select * from <YOUR_NAMESPACE_NAME_1>.<YOUR_TABLE_NAME>").show()
    // Stop SparkSession
    spark.stop()
  }
}

Then, you need to build a .jar file by using your preferred build tool, like sbt package or ./gradlew assemble.

After building the .jar file, you can submit that .jar file to your Spark cluster by using spark-submit, using the --packages option to enable the ScalarDB Analytics libraries on your cluster by running the following command, replacing <SPARK_VERSION>_<SCALA_VERSION>:<SCALARDB_ANALYTICS_WITH_SPARK_VERSION> with the versions that you're using:

$ ./bin/spark-submit \
      --class "YourApp" \
      --packages com.scalar-labs:scalardb-analytics-spark-<SPARK_VERSION>_<SCALA_VERSION>:<SCALARDB_ANALYTICS_WITH_SPARK_VERSION> \
      <YOUR_APP_NAME>.jar

For more information about general Spark application development, see the Apache Spark documentation.