Skip to content

Commit

Permalink
update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
Alif-2001 committed Dec 4, 2024
1 parent b93e7cb commit 003a0c9
Showing 1 changed file with 6 additions and 116 deletions.
122 changes: 6 additions & 116 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,140 +4,30 @@ BenchBase (formerly [OLTPBench](https://github.com/oltpbenchmark/oltpbench/)) is

**Table of Contents**

- [Before you begin](#before-you-begin)
- [Quickstart](#quickstart)
- [Description](#description)
- [Usage Guide](#usage-guide)

---

## Before you begin

### Uderstanding Benchbase: A Benchmarking Framework

[Benchbase](https://github.com/cmu-db/benchbase) is an open-source benchmarking tool designed to assess the performance of relational and NoSQL databases under different workloads. It was originally developed by the Carnegie Mellon Database Group and is widely used to evaluate database systems, both in academic and industry settings. The tool simplifies the process of setting up and running benchmarks by providing a standard interface and a range of workloads, such as TPC-C, TPC-H, etc.

### Install Java and Maven

The benchbase package we will be using requires [Java 21](https://docs.aws.amazon.com/corretto/latest/corretto-21-ug/downloads-list.html) and [Maven](https://maven.apache.org/install.html).

### Have a DSQL Cluster ready

If you haven’t done it already, go ahead and create a cluster to use for the TPC-C benchmarking following this [guide](https://docs.aws.amazon.com/aurora-dsql/latest/userguide/programming-with-cli-crud.html) or using the [AWS console](https://console.aws.amazon.com/dsql).


## Quickstart

### Step 1: Setting up Benchbase

1. We will be using a forked version of the original Benchbase repository that can be pulled from the amazon-contributing github repo using the following command:

To clone and build BenchBase using the auroradsql profile,
```bash
git clone --depth 1 https://github.com/amazon-contributing/aurora-dsql-benchbase-benchmarking.git
cd aurora-dsql-benchbase-benchmarking
./mvnw clean package -P auroradsql
```

2. After cloning and building the package:

This produces artifacts in the target folder, which can be extracted,
```bash
cd target
tar xvzf benchbase-auroradsql.tgz
cd benchbase-auroradsql
```

### Step 2: Loading TPC-C data

We have added sample configuration and ddl files to the repository, allowing you to run TPC-C benchmarks against an Aurora DSQL cluster.

1. Edit the `config/auroradsql/sample_tpcc_config.xml file`:

This file contains various settings that can be adjusted based on how users want to run the benchmarking test. Before loading any data into the table, replace `localhost` in the `<url></url>` tag with your Aurora DSQL cluster endpoint.

Next, set the username and the password token inside the `<username></username>` and `<password></password>` tags. If you don’t know how to generate a password token, follow this [guide](https://docs.aws.amazon.com/aurora-dsql/latest/userguide/SECTION_authentication-token.html).

We have also added automatic password/token generation using IAM authentication in our custom Benchbase implementation. To use it, simply leave the `<password></password>` field empty. To understand where the credentials and region information are fetched from, checkout these libraries:
- [DefaultCredentialsProvider](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html)
- [DefaultAwsRegionProviderChain](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html)

Set the `<scalefactor></scalefactor>` to the number of TPC-C warehouses you would like to create and run the benchmark on.

Finally, set the workload parameters. These settings will be used by the workload in the next step. Change the number of `<terminals></terminals>` to determine how many concurrent connections to make. These connections are evenly spread over the number of warehouses. Note that more terminals mean more memory usage.

2. Save the config file and run the following command to create tables and load data:
```shell
java -jar benchbase.jar -b tpcc -c config/auroradsql/sample_tpcc_config.xml --create=true --load=true --execute=false
```

We separated the loading and test execution into two steps. You can set the `--execute` flag to `true` to have the benchmarking run immediately after the data is loaded.


Step 3: Running TPC-C and Interpreting Results

1. Run the benchmark using the following command:

```shell
java -jar benchbase.jar -b tpcc -c config/auroradsql/sample_tpcc_config.xml --create=false --load=false --execute=true
```

Once the workload has finished running, Benchbase will output a histogram in the terminal, several `.csv` files in the results folder and a `summary.json` file in the same folder. For a 1-warehouse run with 10 terminals, the histogram might look like this:

```shell
[INFO ] 2024-11-21 19:10:23,042 [main] com.oltpbenchmark.DBWorkload runWorkload - ======================================================================
[INFO ] 2024-11-21 19:10:23,043 [main] com.oltpbenchmark.DBWorkload runWorkload - Rate limited reqs/s: Results(state=EXIT, nanoSeconds=60999908412, measuredRequests=3168) = 51.93450420618642 requests/sec (throughput), 52.24598008368564 requests/sec (goodput)
[INFO ] 2024-11-21 19:10:23,049 [main] com.oltpbenchmark.DBWorkload writeOutputs - Output Raw data into file: tpcc_2024-11-21_19-10-23.raw.csv
[INFO ] 2024-11-21 19:10:23,081 [main] com.oltpbenchmark.DBWorkload writeOutputs - Output samples into file: tpcc_2024-11-21_19-10-23.samples.csv
[INFO ] 2024-11-21 19:10:23,088 [main] com.oltpbenchmark.DBWorkload writeOutputs - Output summary data into file: tpcc_2024-11-21_19-10-23.summary.json
[INFO ] 2024-11-21 19:10:23,096 [main] com.oltpbenchmark.DBWorkload writeOutputs - Output DBMS parameters into file: tpcc_2024-11-21_19-10-23.params.json
[INFO ] 2024-11-21 19:10:23,096 [main] com.oltpbenchmark.DBWorkload writeOutputs - Output benchmark config into file: tpcc_2024-11-21_19-10-23.config.xml
[INFO ] 2024-11-21 19:10:23,131 [main] com.oltpbenchmark.DBWorkload writeOutputs - Output results into file: tpcc_2024-11-21_19-10-23.results.csv with window size 5
[INFO ] 2024-11-21 19:10:23,149 [main] com.oltpbenchmark.DBWorkload writeHistograms - ======================================================================
[INFO ] 2024-11-21 19:10:23,149 [main] com.oltpbenchmark.DBWorkload writeHistograms - Workload Histograms:

Completed Transactions:
com.oltpbenchmark.benchmarks.tpcc.procedures.NewOrder/01 [1473]
com.oltpbenchmark.benchmarks.tpcc.procedures.Payment/02 [1330] *
com.oltpbenchmark.benchmarks.tpcc.procedures.OrderStatus/03 [ 136] *
com.oltpbenchmark.benchmarks.tpcc.procedures.Delivery/04 [ 130] *
com.oltpbenchmark.benchmarks.tpcc.procedures.StockLevel/05 [ 118] *

Aborted Transactions:
com.oltpbenchmark.benchmarks.tpcc.procedures.NewOrder/01 [ 15]

Rejected Transactions (Server Retry):
com.oltpbenchmark.benchmarks.tpcc.procedures.NewOrder/01 [ 91] *
com.oltpbenchmark.benchmarks.tpcc.procedures.Payment/02 [3468]
com.oltpbenchmark.benchmarks.tpcc.procedures.Delivery/04 [ 173] **

Rejected Transactions (Retry Different):
<EMPTY>

Unexpected SQL Errors:
<EMPTY>

Unknown Status Transactions:
<EMPTY>


[INFO ] 2024-11-21 19:10:23,149 [main] com.oltpbenchmark.DBWorkload writeHistograms - ======================================================================
```

This result indicates that the test ran for `60 seconds` and processed `3168 transactions`, of which `1473` were New Order transactions. The number of New Order transactions is your `tpmC`. If the test runs for longer than 60 seconds, you can calculate the tpmC by dividing the number of completed `New Order Transactions / the test duration in minutes`.

(Checkout the TPC-C documentation to understand how the Tpmc is calculated: https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c_v5.11.0.pdf)

### Step 4: Clean Up

When you are done testing your DSQL cluster.

1. Run the following commands to clean up the package:
```shell
cd ../..
./mvnw clean
Inside this folder, run BenchBase by executing the tpcc benchmark,
> To learn more about the config file and the benchmarking results, checkout this wiki [link](https://github.com/amazon-contributing/aurora-dsql-benchbase-benchmarking/wiki#loading-data-and-running-tpc-c-against-an-aurora-dsql-cluster).
```bash
java -jar benchbase.jar -b tpcc -c config/auroradsql/sample_tpcc_config.xml --create=true --load=true --execute=true
```

2. Delete your cluster by following this [guide](https://docs.aws.amazon.com/aurora-dsql/latest/userguide/programming-with-cli-crud.html#programming-with-cli-crud-delete) or using the [AWS console](https://console.aws.amazon.com/dsql).

---

## Description
Expand Down

0 comments on commit 003a0c9

Please sign in to comment.