feat: Cl/ci3.3 (#10946)

[CI3 introduction.](https://hackmd.io/bTnKHtTHT8mAdTtD0t7JvA?view)

This is a majority step towards the vision of CI3, still namely missing
merge queue.

New features:
- Grinding flakes in master. We run all tests on 5 separate runners to
report on flakes at the source.
- External contributors can now have CI run just by approving their PR. 
- Ability to debug CI entirely from commandline from any machine. Get
dropped into a productive shell right after the CI failure by doing
`./ci.sh ec2` while your PR is a draft (note: do not do this if pushing
to a non-draft PR).
- Add tests to CI by adding tests to bootstrap. Target a rich
environment with no differences from running inside the dev container.
- Releases that are fully dry-runnable and deployable from a single
command. See above hackmd for details.
- Recovery from spot eviction (finally implemented correctly).

Some remaining items are tracked here.
AztecProtocol/aztec-packages#10775

---------

Co-authored-by: ludamad <domuradical@gmail.com>
Co-authored-by: ludamad <adam.domurad@gmail.com>
Co-authored-by: thunkar <gregojquiros@gmail.com>

README.md

@@ -191,9 +191,7 @@ CMake can be passed various build options on its command line:
 - `-DBENCHMARK=ON | OFF`: Enable/disable building of benchmarks.
 - `-DFUZZING=ON | OFF`: Enable building various fuzzers.
 
-If you are cross-compiling, you can use a preconfigured toolchain file:
-
-- `-DCMAKE_TOOLCHAIN_FILE=<filename in ./cmake/toolchains>`: Use one of the preconfigured toolchains.
+Various presets are defined in CMakePresets.json for scenarios such as instrumentation, cross-compiling and targets such as WASM.
 
 #### WASM build
 

acir_tests/.gitignore

@@ -1,4 +1,5 @@
 acir_tests
 crs
 **/.yarn/*
-!**/.yarn/releases
+!**/.yarn/releases
+*.jsonl

acir_tests/Earthfile

@@ -8,7 +8,7 @@ test:
 benches-bb:
   FROM ../../+bootstrap-noir-bb
   WORKDIR /usr/src/barretenberg/acir_tests
-  RUN ./bootstrap.sh prepare-benches
+  RUN ./bootstrap.sh ci
 
 bench:
   # This target is used for debugging the benchmarking target.

acir_tests/README.md

@@ -7,31 +7,50 @@ The aim is to verify acir tests verify through a given backend binary. "Backend
 - bb.js-dev (symlink in your PATH that runs the typescript CLI via ts-node)
 - bb.js.browser (script in `headless-test` that runs a test through bb.js in a browser instance via playwright)
 
-To run:
+## Building the tests.
+
+To build all the tests:
+
+```
+./bootstrap.sh
+```
+
+This will clone the acir test vectors from the noir repo, removing any that are not relevent.
+It will then compile them all using local repo versions of nargo and bb (used for generating recursive inputs).
+
+## Running the tests.
 
 ```
-$ ./run_acir_tests.sh
+./bootstrap.sh test
 ```
 
-This will clone the acir test vectors from the noir repo, and will iterate over each one, running it through the
-`../cpp/build/bin/bb` binary (by default) `prove_and_verify` command.
+This will run all the tests as returned by `./bootstrap.sh test_cmds`.
+
+To run a single test you can:
+
+```
+./run_test.sh <test name>
+```
+
+By default this will use the native binary `../cpp/build/bin/bb` and the `prove_and_verify` flow.
 
 You can substitute the backend binary using the `BIN` environment variable.
 You can turn on logging with `VERBOSE` environment variable.
-You can specify a specific test to run.
+You can specify which proving system to use with the `SYS` variable (ultra_honk, ultra_rollup_honk, mega_honk).
+If not specified it defaults to plonk (TODO: Make explicit).
 
 ```
-$ BIN=bb.js VERBOSE=1 ./run_acir_tests.sh 1_mul
+$ SYS=ultra_honk BIN=bb.js VERBOSE=1 ./run_test.sh 1_mul
 ```
 
 You can use a relative path to an executable. e.g. if bb.js-dev is not symlinked into your PATH:
 
 ```
-$ BIN=../ts/bb.js-dev VERBOSE=1 ./run_acir_tests.sh 1_mul
+$ BIN=../ts/bb.js-dev VERBOSE=1 ./run_test.sh 1_mul
 ```
 
 ```
-$ BIN=./headless-test/bb.js.browser VERBOSE=1 ./run_acir_tests.sh 1_mul
+$ BIN=./headless-test/bb.js.browser VERBOSE=1 ./run_test.sh 1_mul
 ```
 
 You can specify a different testing "flow" with `FLOW` environment variable. Flows are in the `flows` dir.
@@ -43,20 +62,5 @@ The `all_cmds` flow tests all the supported commands on the binary. Slower, but
 $ FLOW=all_cmds ./run_acir_tests.sh 1_mul
 ```
 
-## Regenerating witness for `double_verify_proof` and `double_verify_nested_proof`
-
-`double_verify_proof` has inputs that are proof system specific such as the circuit verification key and the proofs themselves which are being recursively verified. Certain proof system changes can sometimes lead to the key or inner proofs now being invalid.
-
-This means we have to generate the proof specific inputs using our backend and pass it back into `double_verify_proof` to regenerate the accurate witness. The following is a temporary solution to manually regenerate the inputs for `double_verify_proof` on a specific Noir branch.
-
-First find `acir_tests/gen_inner_proof_inputs.sh`. Change the $BRANCH env var to your working branch and $PROOF_NAME to your first input you want to recursively verify. The script is going to generate the proof system specific verification key output and proof for the `assert_statement` test.
-
-To run:
-```
-./gen_inner_proof_inputs.sh
-```
-To generate a new input you can run the script again. To generate a new file under `assert_statement/proofs/` be sure to change the $PROOF_NAME inside of the script.
-
-You can then copy these inputs over to your working branch in Noir and regenerate the witness for `double_verify_proof`. You can then change the branch in `run_acir_tests.sh` to this Noir working branch as well and `double_verify_proof` should pass.
-
-The same process should then be repeated, but now `double_verify_proof_recursive` will be the circuit for which we will be generating recursive inputs using `gen_inner_proof_inputs.sh`. The recursive artifacts should then be supplied as inputs to `double_verify_nested_proof`.
+We currently have to use a separate flow script to run client_ivc scheme as opposed to just setting `SYS` due to
+how cli commands are handled non-uniformly.

acir_tests/bench_acir_tests.sh

@@ -3,7 +3,7 @@ set -e
 
 cd "$(dirname "$0")"
 
-USE_CACHE=1 ./bootstrap.sh full
+./bootstrap.sh full
 
 TEST_NAMES=("$@")
 THREADS=(1 4 16 32 64)

acir_tests/bootstrap.sh

@@ -4,84 +4,70 @@ source $(git rev-parse --show-toplevel)/ci3/source_bootstrap
 cmd=${1:-}
 export CRS_PATH=$HOME/.bb-crs
 
-function prepare_tests {
-  set -eu
-
-  github_group "acir_tests copy tests"
-
-  rm -rf acir_tests
-  cp -R ../../noir/noir-repo/test_programs/execution_success acir_tests
-  # Running these requires extra gluecode so they're skipped.
-  rm -rf acir_tests/{diamond_deps_0,workspace,workspace_default_member}
-  # TODO(https://github.com/AztecProtocol/barretenberg/issues/1108): problem regardless the proof system used
-  rm -rf acir_tests/regression_5045
-
-  # Regenerate verify_honk_proof and verify_rollup_honk_proof recursive input.
-  echo "Regenerating verify_honk_proof and verify_rollup_honk_proof recursive inputs."
-  COMPILE=2 ./run_test.sh assert_statement
-  local bb=$(realpath ../cpp/build/bin/bb)
-  (cd ./acir_tests/assert_statement && \
-    $bb write_recursion_inputs_ultra_honk -b ./target/program.json -o ../verify_honk_proof --recursive && \
-    $bb write_recursion_inputs_rollup_honk -b ./target/program.json -o ../verify_rollup_honk_proof --recursive)
-
-  # COMPILE=2 only compiles the test.
-  denoise "parallel --joblog joblog.txt --line-buffered 'COMPILE=2 ./run_test.sh \$(basename {})' ::: ./acir_tests/*"
-
-  github_endgroup
-}
-
-function build_tests {
-  set -eu
-
-  github_group "acir_tests build"
-
-  prepare_tests
+tests_tar=barretenberg-acir-tests-$(cache_content_hash \
+    ../../noir/.rebuild_patterns \
+    ../../noir/.rebuild_patterns_tests \
+    ../cpp/.rebuild_patterns \
+    ).tar.gz
 
-  # Update yarn.lock so it can be committed.
+tests_hash=$(cache_content_hash \
+    ^barretenberg/acir_tests/ \
+    ../../noir/.rebuild_patterns \
+    ../../noir/.rebuild_patterns_tests \
+    ../cpp/.rebuild_patterns \
+    ../ts/.rebuild_patterns)
+
+function build {
+  echo_header "acir_tests build"
+
+  if ! cache_download $tests_tar; then
+    rm -rf acir_tests
+    denoise "cd ../../noir/noir-repo/test_programs/execution_success && git clean -fdx"
+    cp -R ../../noir/noir-repo/test_programs/execution_success acir_tests
+    # Running these requires extra gluecode so they're skipped.
+    rm -rf acir_tests/{diamond_deps_0,workspace,workspace_default_member}
+    # TODO(https://github.com/AztecProtocol/barretenberg/issues/1108): problem regardless the proof system used
+    # TODO: Check if resolved. Move to .test_skip_patterns if not.
+    rm -rf acir_tests/regression_5045
+
+    # COMPILE=2 only compiles the test.
+    denoise "parallel --joblog joblog.txt --line-buffered 'COMPILE=2 ./run_test.sh \$(basename {})' ::: ./acir_tests/*"
+
+    echo "Regenerating verify_honk_proof and verify_rollup_honk_proof recursive inputs."
+    local bb=$(realpath ../cpp/build/bin/bb)
+    (cd ./acir_tests/assert_statement && \
+      $bb write_recursion_inputs_ultra_honk -b ./target/program.json -o ../verify_honk_proof --recursive && \
+      $bb write_recursion_inputs_rollup_honk -b ./target/program.json -o ../verify_rollup_honk_proof --recursive)
+
+    cache_upload $tests_tar acir_tests
+  fi
+
+  # TODO: Revisit. Update yarn.lock so it can be committed.
   # Be lenient about bb.js hash changing, even if we try to minimize the occurrences.
-  denoise "cd browser-test-app && yarn add --dev @aztec/bb.js@../../ts && yarn"
+  denoise "cd browser-test-app && yarn add --dev @aztec/bb.js@portal:../../ts && yarn"
   denoise "cd headless-test && yarn"
   denoise "cd sol-test && yarn"
-  # The md5sum of everything is the same after each yarn call.
+  # TODO: Revist. The md5sum of everything is the same after each yarn call.
   # Yet seemingly yarn's content hash will churn unless we reset timestamps
   find {headless-test,browser-test-app} -exec touch -t 197001010000 {} + 2>/dev/null || true
 
   denoise "cd browser-test-app && yarn build"
-
-  github_endgroup
-}
-
-function hash {
-  cache_content_hash \
-    ../../noir/.rebuild_patterns \
-    ../../noir/.rebuild_patterns_tests \
-    ../../barretenberg/cpp/.rebuild_patterns \
-    ../../barretenberg/ts/.rebuild_patterns \
-    ../../barretenberg/acir_tests/.rebuild_patterns
 }
 
 function test {
-  set -eu
-
-  local hash=$(hash)
-  test_should_run barretenberg-acir-tests-$hash || return 0
-
-  github_group "acir_tests testing"
-
-  # TODO: These are some magic numbers that fit our dev/ci environments. They ultimately need to work on lower hardware.
-  export HARDWARE_CONCURRENCY=${HARDWARE_CONCURRENCY:-8}
-  # local jobs=$(($(nproc) / HARDWARE_CONCURRENCY))
-  local jobs=64
-
-  test_cmds | (cd $root; parallel --memsuspend 64G -j$jobs --tag --line-buffered --joblog joblog.txt)
+  echo_header "acir_tests testing"
+  # TODO: 64 is bit of a magic number for CI/mainframe. Needs to work on lower hardware.
+  test_cmds | filter_test_cmds | parallelise 64
+}
 
-  cache_upload_flag barretenberg-acir-tests-$hash
-  github_endgroup
+function test_cmds {
+  # Prefix the test hash on each command.
+  test_cmds_internal | awk "{ print \"$tests_hash \" \$0 }"
 }
 
 # Prints to stdout, one per line, the command to execute each individual test.
 # Paths are all relative to the repository root.
-function test_cmds {
+function test_cmds_internal {
   local plonk_tests=$(find ./acir_tests -maxdepth 1 -mindepth 1 -type d | \
     grep -vE 'verify_honk_proof|double_verify_honk_proof|verify_rollup_honk_proof')
   local honk_tests=$(find ./acir_tests -maxdepth 1 -mindepth 1 -type d | \
@@ -106,10 +92,10 @@ function test_cmds {
 
   # barretenberg-acir-tests-bb.js:
   # Browser tests.
-  echo BROWSER=chrome THREAD_MODEL=mt PORT=8080 $run_test_browser verify_honk_proof
-  echo BROWSER=chrome THREAD_MODEL=st PORT=8081 $run_test_browser 1_mul
-  echo BROWSER=webkit THREAD_MODEL=mt PORT=8082 $run_test_browser verify_honk_proof
-  echo BROWSER=webkit THREAD_MODEL=st PORT=8083 $run_test_browser 1_mul
+  echo BROWSER=chrome THREAD_MODEL=mt $run_test_browser verify_honk_proof
+  echo BROWSER=chrome THREAD_MODEL=st $run_test_browser 1_mul
+  echo BROWSER=webkit THREAD_MODEL=mt $run_test_browser verify_honk_proof
+  echo BROWSER=webkit THREAD_MODEL=st $run_test_browser 1_mul
   # echo ecdsa_secp256r1_3x through bb.js on node to check 256k support.
   echo BIN=$bbjs_bin FLOW=prove_then_verify $run_test ecdsa_secp256r1_3x
   # echo the prove then verify flow for UltraHonk. This makes sure we have the same circuit for different witness inputs.
@@ -149,33 +135,27 @@ function test_cmds {
   echo FLOW=prove_then_verify_client_ivc $run_test databus_two_calldata
 }
 
-export -f build_tests test
+function bench {
+  # TODO: Move to scripts dir along with run_test.sh.
+  LOG_FILE=bench-acir.jsonl ./bench_acir_tests.sh
+}
 
 case "$cmd" in
   "clean")
     git clean -fdx
-    (cd ../../noir/noir-repo/test_programs/execution_success && git clean -fdx)
-    ;;
-  ""|"fast"|"full")
-    ;;
-  "build-tests")
-    build_tests
     ;;
   "ci")
-    build_tests
+    build
     test
     ;;
-  "hash")
-    hash
-    ;;
-  "test")
-    test
+  ""|"fast"|"full")
+    build
     ;;
-  "prepare-benches")
-    prepare_tests
+  "hash")
+    echo $tests_hash
     ;;
-  "test-cmds")
-    test_cmds
+  test|test_cmds|bench)
+    $cmd
     ;;
   *)
     echo "Unknown command: $cmd"

acir_tests/browser-test-app/package.json

@@ -10,7 +10,7 @@
     "serve:dest:mt": "serve -n -L -p ${PORT:-8080} -c ../serve.json dest"
   },
   "devDependencies": {
-    "@aztec/bb.js": "../../ts",
+    "@aztec/bb.js": "portal:../../ts",
     "@types/debug": "^4.1.12",
     "@types/pako": "^2.0.3",
     "debug": "^4.3.4",

acir_tests/browser-test-app/src/index.ts

@@ -20,7 +20,7 @@ async function runTest(
   await backend.destroy();
 
   debug(`verifying...`);
-  const verifier = new BarretenbergVerifier({ threads });
+  const verifier = new BarretenbergVerifier();
   const verified = await verifier.verifyUltraHonkProof(proof, verificationKey);
   debug(`verified: ${verified}`);