From 038ab832e9ee6e50e6c19e33105f4748b0c6ac3e Mon Sep 17 00:00:00 2001 From: slava Date: Tue, 28 Jan 2025 09:44:34 +0100 Subject: [PATCH] [doc, Q1'25-CQH] synthetic benchmarks documentation updates Clarifies usage and some gotchas I encountered while getting acquainted with the tool. --- .../benchmarking_synthetic_workloads.md | 29 +++++++++++++++---- 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/docs/practices/workflows/benchmarking_synthetic_workloads.md b/docs/practices/workflows/benchmarking_synthetic_workloads.md index 121dc01b549..ec3a41f4469 100644 --- a/docs/practices/workflows/benchmarking_synthetic_workloads.md +++ b/docs/practices/workflows/benchmarking_synthetic_workloads.md @@ -14,7 +14,7 @@ The main drawbacks of synthetic benchmarks are: - Drawing conclusions is limited as real world traffic is not homogeneous. - Calibrating traffic generation parameters can be cumbersome. -The tooling for synthetic benchmarks is available in [`benchmarks/bm-synth`](../../../benchmarks/bm-synth). +The tooling for synthetic benchmarks is available in [`benchmarks/synth-bm`](../../../benchmarks/synth-bm). ## Common parameters @@ -38,6 +38,18 @@ Before an RPC request is sent, the tooling awaits capacity on a buffered channel The tooling's [`justfile`](../../../benchmarks/synth-bm/justfile) contains recipes for the most relevant workflows. +A typical workflow benchmarking the native token transfers using the above `justfile` would be something along the: +```command +[t1]$ just init_localnet +[t1]$ just create_subaccounts +# apply the desired config modifications (see below) +[t2]$ just run_localnet +# set the desired tx rate and total volume in the justfile (see above) +[t1]$ just benchmark_native_transfers +``` +Watch out! Running the `neard` via `just run_localnet` is going to pick whatever `release` executable currently compiled. +So comparing different binaries one needs to be careful to (re-)compile the `neard` as needed. + ### Create sub accounts Creating the state for synthetic benchmarks usually starts with creating accounts. We create sub accounts for the account specified by `--signer-key-path`. This avoids dealing with the registrar, which would be required for creating top level accounts. To view all options, run: @@ -53,12 +65,15 @@ Generates a native token transfer workload involving the accounts provided in `- ```command cargo run --release -- benchmark-native-transfers --help ``` - -Automatic calculation of transactions per second (TPS) when RPC requests are sent with `wait_until: NONE` is coming up shortly. In the meantime, they can be calculated manually by querying the `near_transaction_processed_successfully_total` metric, e.g. with: - -```command -http localhost:3030/metrics | grep transaction_processed +For the native transfer benchmark transactions are sent with `wait_until: None`, meaning the responses the `near_synth_bm` tool receives are basically just an ACK by the RPC confirming it received the transaction. +Thus the numbers reported by the tool as if in ``` +[2025-01-27T14:05:12Z INFO near_synth_bm::native_transfer] Sent 200000 txs in 6.50 seconds +[2025-01-27T14:05:12Z INFO near_synth_bm::rpc] Received 200000 tx responses in 6.49 seconds +``` +are not directly indicative of the runtime performance. +The number of transactions actually processed may be obtained by querying the `near_transaction_processed_successfully_total` metric, e.g. with: `http://localhost:3030/metrics | grep transaction_processed`. +Automatic calculation of transactions per second (TPS) when RPC requests are sent with `wait_until: NONE` is coming up shortly. ### Benchmark calls to the `sign` method of an MPC contract @@ -98,6 +113,8 @@ The configuration generated by the above command does not enable memtrie. Howeve Following these steps so far creates a config that will throttle throughput due to various factors related to state witness size, gas/compute limits, and congestion control. In case you want to benchmark a node that fully utilizes its hardware, you can do the following modifications to effectively run with unlimited configuration: +Config modifications need to be applied before running the `neard` (as in `just run_localnet`) for the first time. Otherwise one likely to run into initialization issues. + ``` # Modifications in .near/genesis.json