VM benchmark framework

[Voxelot]

• Setup a benchmark script in “fuel-core/fuel-tests/benches” using Criterion for each opcode in isolation, running against a full node on RocksDB

  • Criterion Harness using a sync executor (Add fuel-benches #616)
  • Opcode Benchmarks
    • For each opcode being tested, repeat inside the script a large number of times to ensure that the vm is being exercised a relatively large amount of time compared to the rest of the node
    • For instructions that can’t be measured in isolation (for example CALL), exclude the benchmarked times of the dependent opcodes used for setup.
    • For opcodes with variable execution time depending on the arguments (e.g. cost of MCP is based on the number of bytes being copied), we should parameterize a range of values to establish a linear fee model (constant ADD, linear MCP/MCL, logarithmic via merklized storage access)

• Framework to adjust opcode gas cost based on relative performance differences between opcodes. #667

  • Use NOOP as the base unit of cost (1)
  • All other opcodes are priced based on how many times slower they are compared to NOOP
  • Should be a rerunnable script that automatically processes the output from the benchmark harness

• Target some kind of low-medium cpu ec2 instance type (i.e. the minimum requirements for a validator)

  • Set block size based on target TPS after opcodes are priced

Future Work

• Consider disabling VM instruction inlining to enable real-time prod monitoring of opcodes using perf
• Investigate how much these relative opcode costs change based on different machine hardware

[ControlCplusControlV]

I am assuming we would use cargo bench for this, do we also want to modify configs or added a cpu limiting programmer for capped performance on machines to adjust for background usage when benchmarking? (I remember a Go optimization talk on Slack stickers noticeably impacting benchmarking).

We could also clone the sway-applications repo to run that repo's tests against fuel-core to get a real world benchmark, then have a seperate set of tests dedicated to trying to find DOS vectors (like 17000 NOPs to see if runtime is mostly in line with gas costs)

[Voxelot]

cargo bench is typically for micro-benchmarking use-cases. I'm thinking we may need to setup a custom harness with it's own CLI for this 🤔