Warning

This version of ScalarDB is no longer supported. We recommend that you update to the latest version of ScalarDB.

ScalarDB Benchmarks

This repository contains benchmark programs for ScalarDB.

Available workloads

• TPC-C

Prerequisites

• Java (OpenJDK 8 or higher)
• Gradle
• Kelpie

The benchmark uses Kelpie, which is a simple yet general framework for performing end-to-end testing such as system benchmarking and verification. Get the latest version of Kelpie from here and unzip the archive.

Usage

Set up an environment

This benchmark requires the followings:

• A client to execute this benchmark
• A target database (See here for supported databases)

Build

./gradlew shadowJar

Create tables

Before loading initial data, the tables must be defined using ScalarDB Schema Loader. Get the latest schema loader here and execute it with the workload-specific schema file. For setting ScalarDB properties, see also the related documents here.

For example, execute the following command with tpcc-schema.json to create tables for TPC-C benchmark.

java -jar scalardb-schema-loader-<version>.jar --config /path/to/scalardb.properties -f tpcc-schema.json --coordinator

Load and run

1. Prepare a configuration file
   • A configuration file requires at least the locations of workload modules to run and the database configuration. The following example shows the case for running TPC-C benchmark. The database configuration should be matched with the benchmark environment set up above. You can use the ScalarDB property file instead of specifying each configuration item. If the config_file is specified, all other configuration items will be ignored.

     [modules]
     [modules.preprocessor]
     name = "com.scalar.db.benchmarks.tpcc.TpccLoader"
     path = "./build/libs/scalardb-benchmarks-all.jar"
     [modules.processor]
     name = "com.scalar.db.benchmarks.tpcc.TpccBench"
     path = "./build/libs/scalardb-benchmarks-all.jar"
     [modules.postprocessor]
     name = "com.scalar.db.benchmarks.tpcc.TpccReporter"
     path = "./build/libs/scalardb-benchmarks-all.jar"
      
     [database_config]
     contact_points = "localhost"
     contact_port = 9042
     username = "cassandra"
     password = "cassandra"
     storage = "cassandra"
     #config_file = "/path/to/scalardb.properties"
     

   • You can define static parameters to pass to modules in the file. For details, see example configuration files such as tpcc-benchmark-config.toml and available parameters in the following section.
2. Run a benchmark

   ${kelpie}/bin/kelpie --config your_config.toml
   

   • ${kelpie} is a Kelpie directory, which is extracted from the archive you downloaded above.
   • There are other options such as --only-pre (i.e., loading data) and --only-process (i.e., running benchmark), which run only the specified process. --except-pre and --except-process run a job without the specified process.

Common parameters

name	description	default
concurrency	Number of threads for benchmarking.	1
run_for_sec	Duration of benchmark (in seconds).	60
ramp_for_sec	Duration of ramp up time before benchmark (in seconds).	0

Workload-specific parameters

TPC-C

name	description	default
num_warehouses	Number of warehouses (scale factor) for benchmarking.	1
load_concurrency	Number of threads for loading.	1
load_start_warehouse	Start ID of loading warehouse. This option can be useful with --skip-item-load when loading large scale data with multiple clients or adding additional warehouses.	1
load_end_warehouse	End ID of loading warehouse. You can use either --num-warehouses or --end-warehouse to specify the number of loading warehouses.	1
skip_item_load	Whether or not to skip loading item table	false
use_table_index	Whether or not to use a generic table-based secondary index instead of ScalarDB’s secondary index.	false
np_only	Run benchmark with only new-order and payment transactions (50% each).	false
rate_new_order	Percentage of new-order transaction. Specify all the rate parameters to use your own mix.	N/A
rate_payment	Percentage of payment transaction. Specify all the rate parameters to use your own mix.	N/A
rate_order_status	Percentage of order-status transaction. Specify all the rate parameters to use your own mix.	N/A
rate_delivery	Percentage of delivery transaction. Specify all the rate parameters to use your own mix.	N/A
rate_stock_level	Percentage of stock-level transaction. Specify all the rate parameters to use your own mix.	N/A
backoff	Sleep time in milliseconds inserted after a transaction is aborted due to a conflict.	0