Merge branch 'ZcashFoundation:main' into main

oxarbitrage · Feb 27, 2024 · f5b5cdf · f5b5cdf
2 parents 467b50a + 58bfe97
commit f5b5cdf
Show file tree

Hide file tree

Showing 402 changed files with 17,109 additions and 4,234 deletions.
diff --git a/.cargo/config.toml b/.cargo/config.toml
@@ -56,6 +56,9 @@ rustflags = [
     "-Wmissing_docs",
 
     # TODOs:
+    # Fix this lint eventually.
+    "-Aclippy::result_large_err",
+
     # `cargo fix` might help do these fixes,
     # or add a config.toml to sub-directories which should allow these lints,
     # or try allowing the lint in the specific module (lib.rs doesn't seem to work in some cases)
@@ -80,7 +83,7 @@ rustflags = [
 [build]
 rustdocflags = [
     # The -A and -W settings must be the same as the `RUSTDOCFLAGS` in:
-    # https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/docs.yml#L61
+    # https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/docs-deploy-firebase.yml#L68
 
     # Links in public docs can point to private items.
     "-Arustdoc::private_intra_doc_links",

diff --git a/.codespellrc b/.codespellrc
@@ -1,3 +1,3 @@
 [codespell]
-ignore-words-list=crate,Sur,inout
+ignore-words-list=crate,Sur,inout,Groth,groth,re-use,
 exclude-file=book/mermaid.min.js
diff --git a/.github/ISSUE_TEMPLATE/release.md b/.github/ISSUE_TEMPLATE/release.md
@@ -16,7 +16,7 @@ They can be skipped for urgent releases.
 
 To check consensus correctness, we want to test that the state format is valid after a full sync. (Format upgrades are tested in CI on each PR.)
 
-- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/continous-integration-docker.yml?query=event%3Aschedule) since the last state change, or
+- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=event%3Aschedule) since the last state change, or
 - [ ] Start a manual workflow run with a Zebra and `lightwalletd` full sync.
 
 State format changes can be made in `zebra-state` or `zebra-chain`. The state format can be changed by data that is sent to the state, data created within the state using `zebra-chain`, or serialization formats in `zebra-state` or `zebra-chain`. 

diff --git a/.github/PULL_REQUEST_TEMPLATE/release-checklist.md b/.github/PULL_REQUEST_TEMPLATE/release-checklist.md
@@ -9,7 +9,7 @@ assignees: ''
 
 # Prepare for the Release
 
-- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/continous-integration-docker.yml?query=event%3Aschedule) since the last state change, or start a manual full sync.
+- [ ] Make sure there has been [at least one successful full sync test](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=event%3Aschedule) since the last state change, or start a manual full sync.
 - [ ] Make sure the PRs with the new checkpoint hashes and missed dependencies are already merged.
       (See the release ticket checklist for details)
 
@@ -83,13 +83,22 @@ Check that the release will work:
 - [ ] Update crate versions, commit the changes to the release branch, and do a release dry-run:
 
 ```sh
-cargo release version --verbose --execute --allow-branch '*' --workspace --exclude zebrad beta
+# Update everything except for alpha crates and zebrad:
+cargo release version --verbose --execute --allow-branch '*' --workspace --exclude zebrad --exclude zebra-scan --exclude zebra-grpc beta
+# Due to a bug in cargo-release, we need to pass exact versions for alpha crates:
+cargo release version --verbose --execute --allow-branch '*' --package zebra-scan 0.1.0-alpha.4
+cargo release version --verbose --execute --allow-branch '*' --package zebra-grpc 0.1.0-alpha.2
+# Update zebrad:
 cargo release version --verbose --execute --allow-branch '*' --package zebrad patch # [ major | minor | patch ]
+# Continue with the release process:
 cargo release replace --verbose --execute --allow-branch '*' --package zebrad
 cargo release commit --verbose --execute --allow-branch '*'
 ```
 
-Crate publishing is [automatically checked in CI](https://github.com/ZcashFoundation/zebra/actions/workflows/release-crates-io.yml) using "dry run" mode.
+Crate publishing is [automatically checked in CI](https://github.com/ZcashFoundation/zebra/actions/workflows/release-crates-io.yml) using "dry run" mode, however due to a bug in `cargo-release` we need to pass exact versions to the alpha crates:
+
+- [ ] Update `zebra-scan` and `zebra-grpc` alpha crates in the [release-crates-dry-run workflow script](https://github.com/ZcashFoundation/zebra/blob/main/.github/workflows/scripts/release-crates-dry-run.sh)
+- [ ] Push the above version changes to the release branch.
 
 ## Update End of Support
 
@@ -131,8 +140,10 @@ The end of support height is calculated from the current blockchain height:
 
 ## Test the Pre-Release
 
-- [ ] Wait until the [Docker binaries have been built on `main`](https://github.com/ZcashFoundation/zebra/actions/workflows/continous-integration-docker.yml), and the quick tests have passed.
-- [ ] Wait until the [pre-release deployment machines have successfully launched](https://github.com/ZcashFoundation/zebra/actions/workflows/continous-delivery.yml)
+- [ ] Wait until the Docker binaries have been built on `main`, and the quick tests have passed:
+    - [ ] [ci-unit-tests-docker.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-unit-tests-docker.yml?query=branch%3Amain)
+    - [ ] [ci-integration-tests-gcp.yml](https://github.com/ZcashFoundation/zebra/actions/workflows/ci-integration-tests-gcp.yml?query=branch%3Amain)
+- [ ] Wait until the [pre-release deployment machines have successfully launched](https://github.com/ZcashFoundation/zebra/actions/workflows/cd-deploy-nodes-gcp.yml?query=event%3Arelease)
 
 ## Publish Release
 
@@ -148,7 +159,7 @@ The end of support height is calculated from the current blockchain height:
       and put the output in a comment on the PR.
 
 ## Publish Docker Images
-- [ ] Wait for the [the Docker images to be published successfully](https://github.com/ZcashFoundation/zebra/actions/workflows/release-binaries.yml).
+- [ ] Wait for the [the Docker images to be published successfully](https://github.com/ZcashFoundation/zebra/actions/workflows/release-binaries.yml?query=event%3Arelease).
 - [ ] Un-freeze the [`batched` queue](https://dashboard.mergify.com/github/ZcashFoundation/repo/zebra/queues) using Mergify.
 - [ ] Remove `do-not-merge` from the PRs you added it to
 

diff --git a/.github/PULL_REQUEST_TEMPLATE/rfc.md b/.github/PULL_REQUEST_TEMPLATE/rfc.md
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
@@ -1,5 +1,6 @@
 version: 2
 updates:
+  # Rust section
   - package-ecosystem: cargo
     directory: '/'
     # serde, clap, and other dependencies sometimes have multiple updates in a week
@@ -18,110 +19,14 @@ updates:
         ecc:
           patterns:
             # deliberately include zcash_script (even though it is maintained by ZF)
-            - "zcash_*"
-            - "orchard"
-            - "halo2*"
-            - "incrementalmerkletree"
-            - "bridgetree"
-            - "equihash"
-            # addresses
-            - "bs58"
-            - "ripemd"
-        # groups are limited to 10 items
-        crypto:
-          patterns:
-            - "bellman"
-            # reddsa, redjubjub
-            - "red*"
-            - "jubjub"
-            - "group"
-            - "bls12_381"
-            - "blake*"
-            - "secp256k1"
-            - "sha2"
-            - "*25519*"
-            - "rand*"
-        async:
-          patterns:
-            - "tokio*"
-            - "console-subscriber"
-            - "tower*"
-            - "hyper*"
-            - "h2"
-            - "reqwest"
-            - "futures*"
-            - "pin-project*"
-        log-time:
-          patterns:
-            - "tracing*"
-            - "log"
-            - "*eyre*"
-            - "thiserror"
-            # displaydoc, spandoc
-            - "*doc"
-            - "owo-colors"
-            - "sentry*"
-            - "metrics*"
-            # time, humantime
-            - "*time*"
-            - "chrono*"
-        concurrency:
-          patterns:
-            - "once_cell"
-            - "lazy_static"
-            - "rayon*"
-            - "crossbeam*"
-            - "num_cpus"
-        progress-bar:
-          patterns:
-            - "indicatif"
-            - "howudoin"
-        app:
-          patterns:
-            - "abscissa*"
-            - "structopt*"
-            - "clap*"
-            - "atty*"
-            - "semver*"
-            # dirs, directories, directories-next
-            - "dir*"
-            - "vergen"
-            - "*git*"
-            - "toml*"
-            - "rlimit"
-        formats:
-          patterns:
-            - "serde*"
-            - "jsonrpc*"
-            - "hex*"
-            - "regex"
-            - "byteorder"
-            - "bytes"
-            - "bincode"    
-        data-structures:
-          patterns:
-            - "bitflags*"
-            - "bitvec"
-            - "indexmap"
-            - "num-integer"
-            - "primitive-types"
-            - "uint"
-            - "tinyvec"
-            - "itertools"
-            - "ordered-map"
-            - "mset"
-        test:
-          patterns:
-            - "proptest*"
-            - "insta"
-            - "prost*"
-            - "tonic*"
-            # depends on tonic and prost
-            - "console-subscriber"
-            - "tempfile"
-            - "static_assertions"
-            - "criterion"
-            - "inferno"
+            - "zcash_*|orchard|halo2*|incrementalmerkletree|bridgetree|equihash"
+        prod:
+          dependency-type: "production"
+          exclude-patterns:
+            - "zcash_*|orchard|halo2*|incrementalmerkletree|bridgetree|equihash"
+        dev:
+          dependency-type: "development"
+  # Devops section
   - package-ecosystem: github-actions
     directory: '/'
     schedule:
@@ -135,3 +40,7 @@ updates:
       - 'A-devops'
       - 'A-dependencies'
       - 'P-Low :snowflake:'
+    groups:
+      devops:
+        patterns:
+          - "*"
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -1,49 +1,72 @@
 ## Motivation
-
 <!--
 Thank you for your Pull Request.
-How does this change improve Zebra?
+Does it close any issues?
 -->
+_What are the most important goals of the ticket or PR?_
+
+
+### PR Author Checklist
+
+#### Check before marking the PR as ready for review:
+  - [ ] Will the PR name make sense to users?
+  - [ ] Does the PR have a priority label?
+  - [ ] Have you added or updated tests?
+  - [ ] Is the documentation up to date?
+
+##### For significant changes:
+  - [ ] Is there a summary in the CHANGELOG?
+  - [ ] Can these changes be split into multiple PRs?
+
+_If a checkbox isn't relevant to the PR, mark it as done._
 
 ### Specifications
 
 <!--
 If this PR changes consensus rules, quote them, and link to the Zcash spec or ZIP:
 https://zips.z.cash/#nu5-zips
-If this PR changes network behaviour, quote and link to the Bitcoin network reference:
-https://developer.bitcoin.org/reference/p2p_networking.html
 -->
 
+
 ### Complex Code or Requirements
 
 <!--
 Does this PR change concurrency, unsafe code, or complex consensus rules?
-If it does, ask for multiple reviewers on this PR.
+If it does, label this PR with `extra-reviews`.
 -->
 
+
 ## Solution
 
 <!--
 Summarize the changes in this PR.
-Does it close any issues?
-If this is a large change, list commits of key functional changes here.
 -->
 
+
+### Testing
+
+<!--
+Which tests were changed or added in this PR? Were there manual tests?
+-->
+
+
 ## Review
 
 <!--
 Is this PR blocking any other work?
 If you want specific reviewers for this PR, tag them here.
 -->
 
+
 ### Reviewer Checklist
 
-  - [ ] Will the PR name make sense to users?
-    - [ ] Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
-  - [ ] Are the PR labels correct?
-  - [ ] Does the code do what the ticket and PR says?
-    - [ ] Does it change concurrent code, unsafe code, or consensus rules?
-  - [ ] How do you know it works? Does it have tests?
+Check before approving the PR:
+  - [ ] Does the PR scope match the ticket?
+  - [ ] Are there enough tests to make sure it works? Do the tests cover the PR motivation?
+  - [ ] Are all the PR blockers dealt with?
+        PR blockers can be dealt with in new tickets or PRs.
+
+_And check the PR Author checklist is complete._
 
 ## Follow Up Work
 

diff --git a/.github/workflows/cd-deploy-nodes-gcp.patch-external.yml b/.github/workflows/cd-deploy-nodes-gcp.patch-external.yml
@@ -0,0 +1,38 @@
+# Workflow patches for skipping Google Cloud CD deployments on PRs from external repositories.
+name: Deploy Nodes to GCP
+
+# Run on PRs from external repositories, let them pass, and then Mergify will check them.
+# GitHub doesn't support filtering workflows by source branch names, so we have to do it for each
+# job.
+on:
+  pull_request:
+
+# IMPORTANT
+#
+# The job names in `cd-deploy-nodes-gcp.yml`, `cd-deploy-nodes-gcp.patch.yml` and
+# `cd-deploy-nodes-gcp.patch-external.yml` must be kept in sync.
+jobs:
+  # We don't patch the testnet job, because testnet isn't required to merge (it's too unstable)
+  build:
+    name: Build CD Docker / Build images
+    # Only run on PRs from external repositories, skipping ZF branches and tags.
+    if: ${{ startsWith(github.event_name, 'pull') && github.event.pull_request.head.repo.fork }}
+    runs-on: ubuntu-latest
+    steps:
+      - run: 'echo "Skipping job on fork"'
+
+  test-configuration-file:
+    name: Test CD default Docker config file / Test default-conf in Docker
+    # This dependency allows all these jobs to depend on a single condition, making it easier to
+    # change.
+    needs: build
+    runs-on: ubuntu-latest
+    steps:
+      - run: 'echo "Skipping job on fork"'
+
+  test-zebra-conf-path:
+    name: Test CD custom Docker config file / Test custom-conf in Docker
+    needs: build
+    runs-on: ubuntu-latest
+    steps:
+      - run: 'echo "Skipping job on fork"'