Download Zcash Sapling parameters and load them from cached files

[teor2345]

Motivation

We want to avoid linking half a gigabyte of Zcash parameters into the zebrad executable, tests, and build products.

This might also save us some redundant copies of the parameters crates, if they are duplicated for dev and release builds.

This is the Sapling refactor part of ticket #3037.

Solution

Zebra:

• Load Sapling parameters from the downloaded files, rather than crates
• When Zebra is initialised, spawn a task to download parameters
• Mention zebrad download and the cached parameters directory in download error messages

CI:

• Download Zcash parameters in Docker using a new zebrad download command
• Download and cache Zcash parameters in GitHub CI using zebra-consensus examples, like zcash_proofs does

TODO in the next PR:

• Implement Sprout parameter downloads in zcash_proofs
• Load Sprout parameters from the downloaded files (already implemented in zcash_proofs, just needs glue)

Review

@jvff is reviewing this PR.

Reviewer Checklist

• Code implements Specs and Designs
• Existing tests pass
• CI changes pass reliably

Follow Up Work

• download and load Sprout parameters using zcash_proofs - the rest of Download JoinSplit Groth16 verification parameters and load them from disk #3037
• Validate JoinSplit proofs #322 using the sprout parameters

[str4d]

ICYMI, we have a download_params function in the zcash_proofs crate that emulates fetch-params.sh. Limitations:

• It currently only fetches Sapling parameters (as that is all our Rust crates supported making transactions with), not Sprout (as they were too large to just fetch inline all the time).
• It doesn't include the IPFS fetching logic (pulling from there instead of HTTPS if ipfs is installed locally).
• It hardcodes minreq as the fetcher (as we just needed a simple self-contained HTTP stack), whereas good probably want an async-compatible function that can use whatever HTTP stack you're already using elsewhere.

I'd be interested in collaborating on extending that Rust function with additional functionality, or building out a separate dependency for it.

[teor2345]

ICYMI, we have a download_params function in the zcash_proofs crate that emulates fetch-params.sh. Limitations:

Thanks, I had missed download_parameters:
https://github.com/zcash/librustzcash/blob/85780f994d51abb965cbf4b981f8730ff0554607/zcash_proofs/src/lib.rs#L68

It looks like our code is an uncredited copy of that method 😞

• It currently only fetches Sapling parameters (as that is all our Rust crates supported making transactions with), not Sprout (as they were too large to just fetch inline all the time).

We need to fetch Sprout to close ticket #3037, so I'm happy to submit a patch with those changes.

• It doesn't include the IPFS fetching logic (pulling from there instead of HTTPS if ipfs is installed locally).

IPFS is out of scope for Zebra for NU5 mainnet activation.

• It hardcodes minreq as the fetcher (as we just needed a simple self-contained HTTP stack), whereas good probably want an async-compatible function that can use whatever HTTP stack you're already using elsewhere.

We're trying to minimise our scope before Zebra for NU5 mainnet activation. So I don't really want to rewrite working code.

We can just spawn a blocking async task for these downloads, and the executor will put it on its own thread. Then we can open a ticket for an async rewrite that uses reqwest.

(We haven't explicitly picked a HTTP stack yet. But some of Zebra's dependencies use reqwest.)

I'd be interested in collaborating on extending that Rust function with additional functionality, or building out a separate dependency for it.

We don't really have time to split dependencies right now. But I'd be happy to work on a Sprout download.

[teor2345]

it seems like building Zebra is too slow for the time limits on these tests.

That's weird, because all the built crates should just get re-used. Maybe there is some minor difference between cargo run and cargo test that's causing a rebuilt.

I might have to make a "test" that just downloads the parameter files. That should be much smaller than all of zebrad. And it should have exactly the same build settings.

[str4d]

You might find this logic interesting, which we use in our CI to cache the parameters in GH Actions workers: https://github.com/zcash/librustzcash/blob/master/.github/workflows/ci.yml#L20-L33

[teor2345]

You might find this logic interesting, which we use in our CI to cache the parameters in GH Actions workers: https://github.com/zcash/librustzcash/blob/master/.github/workflows/ci.yml#L20-L33

Thanks, this was really helpful, I've used similar logic in our CI.

[jvff]

Looks good, I just added some optional suggestions.

[jvff]

Looks great!