[3.2] benchmarking framework and its use in benchmarking crypto primitives

[linh2931]

Resolve #786 and partially resolve #780.

Implement a benchmarking framework to help writing benchmarking code by reusing common measuring functions and unifying reports. After the PR is approved, the framework can be used to benchmark other functions.

For end users

Below are a few examples

Help

./benchmark -h
benchmark command line options:
  -m [ --module ] arg       the module to be benchmarked; if this option is not
                            present, all modules are benchmarked.
  -l [ --list ]             list of supported modules
  -r [ --runs ] arg (=1000) the number of runs per function
  -h [ --help ]             execute the benchmarked functions for the specified
                            number of times, and report average, minimum, and
                            maximum time in nanoseconds

Benchmarking All Modules (functions)

./benchmark
function                      runs               avg              min              max

alt_bn_128:
alt_bn128_add                 1,000         6,467 ns         3,727 ns       113,625 ns
alt_bn128_mul                 1,000       251,105 ns       235,337 ns       395,856 ns
alt_bn128_pair (1 pair)       1,000     6,405,913 ns     6,354,466 ns     8,265,015 ns
alt_bn128_pair (10 pairs)     1,000    25,379,638 ns    25,236,388 ns    36,013,377 ns

modexp:
1024-bit width                1,000       389,252 ns       388,298 ns       396,263 ns
2048-bit width                1,000     2,914,494 ns     2,910,697 ns     2,999,909 ns

Benchmarking Particular Module with Specific Number of Runs

/benchmark -m modexp -r 500
function                      runs               avg              min              max

modexp:
1024-bit width                500         441,057 ns       388,091 ns       818,064 ns
2048-bit width                500       2,914,402 ns     2,911,318 ns     3,001,094 ns

For developers

Please see alt_bn_128.cpp and modexp.cpp for examples.

The general approach to adding code to benchmark a set of functions is

1. Implement a function where for each function to be benchmarked, prepare input, define a lamda to call the function, and call common benchmarking.
2. Add an entry to modules in benchmark.cpp.

[heifner]

Should we consider using https://github.com/google/benchmark which has built in support for preventing the compiler from optimizing out what you are trying to benchmark as well as other useful features for micro-benchmarking.

[linh2931]

The Google microbenchmarking library looks great. It has lots of features and more code. Just wondering if it is brought in, would it increase dependencies and build time?

[stephenpdeos]

We will not move forward with Google benchmarking framework at this time, because it doesn't seem to have enough value add compared to current code used.

[spoonincode]

This will need to be wrapped in quotes to handle spaces. You may also be able to just put a . here since paths are relative to CMAKE_CURRENT_SOURCE_DIR anyways. Or just forego it all together and include "benchmark.hpp" but that's typically not the pattern we use.

[linh2931]

Thanks for the suggestions. I will make changes.

[spoonincode]

I wonder if there should be a machine-readable output (like JSON) to make it easier to ingest in to some reporting framework.

[spoonincode]

It feels like high_resolution_clock is what we would want when making tiny measurements like this, but it's not a monotonic clock on libstdc++ 😕

[larryk85]

I don't think it matters for this use case if the clock is not monotonic. It's only a relative number that is just needed for orders of magnitudes and comparisons. So, I agree that this should be high_resolution_clock.

[linh2931]

Thanks. Will change to use high resolution clock.

[swatanabe]

Only needing relative times is exactly the use for monotonic clocks. It is probably okay to use a non-monotonic clock, but that's because the small chance of producing wild results if the system clock happens to change while running the benchmark is a manageable problem.

[linh2931]

I will add a -j option for JSON format output.

[larryk85]

I wonder if there should be a machine-readable output (like JSON) to make it easier to ingest in to some reporting framework.

That would be nice to have, but maybe we can list that as a phase 2 item for after this is merged and done.

[linh2931]

Close this as it is being ported over to Leap as AntelopeIO/leap#17.