Skip to content

Commit

Permalink
[doc, Q1'25-CQH] synthetic benchmarks documentation updates
Browse files Browse the repository at this point in the history
Clarifies usage and some gotchas I encountered while getting acquainted with the tool.
  • Loading branch information
ssavenko-near committed Jan 28, 2025
1 parent 8097202 commit 038ab83
Showing 1 changed file with 23 additions and 6 deletions.
29 changes: 23 additions & 6 deletions docs/practices/workflows/benchmarking_synthetic_workloads.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The main drawbacks of synthetic benchmarks are:
- Drawing conclusions is limited as real world traffic is not homogeneous.
- Calibrating traffic generation parameters can be cumbersome.

The tooling for synthetic benchmarks is available in [`benchmarks/bm-synth`](../../../benchmarks/bm-synth).
The tooling for synthetic benchmarks is available in [`benchmarks/synth-bm`](../../../benchmarks/synth-bm).

## Common parameters

Expand All @@ -38,6 +38,18 @@ Before an RPC request is sent, the tooling awaits capacity on a buffered channel

The tooling's [`justfile`](../../../benchmarks/synth-bm/justfile) contains recipes for the most relevant workflows.

A typical workflow benchmarking the native token transfers using the above `justfile` would be something along the:
```command
[t1]$ just init_localnet
[t1]$ just create_subaccounts

Check warning on line 44 in docs/practices/workflows/benchmarking_synthetic_workloads.md

View workflow job for this annotation

GitHub Actions / spellcheck

Unknown word (subaccounts)
# apply the desired config modifications (see below)
[t2]$ just run_localnet
# set the desired tx rate and total volume in the justfile (see above)
[t1]$ just benchmark_native_transfers
```
Watch out! Running the `neard` via `just run_localnet` is going to pick whatever `release` executable currently compiled.
So comparing different binaries one needs to be careful to (re-)compile the `neard` as needed.

### Create sub accounts

Creating the state for synthetic benchmarks usually starts with creating accounts. We create sub accounts for the account specified by `--signer-key-path`. This avoids dealing with the registrar, which would be required for creating top level accounts. To view all options, run:
Expand All @@ -53,12 +65,15 @@ Generates a native token transfer workload involving the accounts provided in `-
```command
cargo run --release -- benchmark-native-transfers --help
```

Automatic calculation of transactions per second (TPS) when RPC requests are sent with `wait_until: NONE` is coming up shortly. In the meantime, they can be calculated manually by querying the `near_transaction_processed_successfully_total` metric, e.g. with:

```command
http localhost:3030/metrics | grep transaction_processed
For the native transfer benchmark transactions are sent with `wait_until: None`, meaning the responses the `near_synth_bm` tool receives are basically just an ACK by the RPC confirming it received the transaction.
Thus the numbers reported by the tool as if in
```
[2025-01-27T14:05:12Z INFO near_synth_bm::native_transfer] Sent 200000 txs in 6.50 seconds
[2025-01-27T14:05:12Z INFO near_synth_bm::rpc] Received 200000 tx responses in 6.49 seconds
```
are not directly indicative of the runtime performance.
The number of transactions actually processed may be obtained by querying the `near_transaction_processed_successfully_total` metric, e.g. with: `http://localhost:3030/metrics | grep transaction_processed`.
Automatic calculation of transactions per second (TPS) when RPC requests are sent with `wait_until: NONE` is coming up shortly.

### Benchmark calls to the `sign` method of an MPC contract

Expand Down Expand Up @@ -98,6 +113,8 @@ The configuration generated by the above command does not enable memtrie. Howeve

Following these steps so far creates a config that will throttle throughput due to various factors related to state witness size, gas/compute limits, and congestion control. In case you want to benchmark a node that fully utilizes its hardware, you can do the following modifications to effectively run with unlimited configuration:

Config modifications need to be applied before running the `neard` (as in `just run_localnet`) for the first time. Otherwise one likely to run into initialization issues.

```
# Modifications in .near/genesis.json
Expand Down

0 comments on commit 038ab83

Please sign in to comment.