-
Notifications
You must be signed in to change notification settings - Fork 248
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Chia proof of space re-implementation #1448
Conversation
… implementation (code doesn't compile in this commit)
57f7be8
to
0b8ab04
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Disclaimer for visibility/audits: The new implementation almost exactly matches the Chia's reference implementation (chiapos) results. It finds all the proofs chiapos finds for the same seed & challenge. When there's just one proof for a challenge, our results match chiapos perfectly. However, there are a couple of differences:
-
We find 0.02% more proofs in cases where chiapos misses some due to compression in phase 4. Those proofs are present in chiapos after phase 1, but aren't recoverable after phase 4 (dropped here).
-
When multiple proofs exist for the same challenge, we pick the "first" proof, but it might not be the same as chiapos' first proof because of the reordering it does in later phases. Our proof still matches one of the chiapos proofs for that challenge, but not always the first one.
-
Regarding quality values, the two values chiapos finds are present in our proof. Still, it's difficult to choose their positions in the same way as chiapos due to the reordering it does in later phases.
At this stage, I believe it's best not to spend more time trying to match Chia tests exactly, as our goal isn't to create a Chia network plotter. As a generic PoS implementation, I take the responsibility of declaring it, perfectly good to be integrated with the rest of the protocol.
0b8ab04
to
4d19fe6
Compare
Updated test UPD: Pushed again with additional comments. |
4d19fe6
to
8199410
Compare
Merging, feel free to leave review even after merge though |
This is a re-implementation of Chia proof of space in Rust (just the first phase) and fixes #1400
Specification: https://www.notion.so/subspacelabs/PoS-Specification-WIP-08339021db504ce9aa0be6cc2f52a949
Research forum discussion: https://forum.subspace.network/t/performance-of-chia-proof-of-space-re-implementation-in-rust/1478
Here are comparison number, where
chia2
is the new implementation:Here is an implementation guide:
chia
feature, module and enum variant were renamed tochia-legacy
chia
and used by defaultchiapos
module is where Chia proof of space implementation lives, we still use it as a black box just like the C++ versiontable
is implementation of Chia tables (1st to 7th)tables
is a higher level abstraction generic overK
that allows creation of tables, querying qualities and proofs and has static method for verifying proofschiapos
module exposes tables for specificK
supported by current platform (largerK
are not supported on 32-bit platforms), also limitsK
to only those that make the most sense (allK
can be enabled withall-chia-k
feature flag and increases compilation time significantly)You may notice a bunch of experimental and even incomplete (
generic_const_exprs
) Rust features used. The idea behind architecture is to make implementation generic overK
while keeping data structures compact and efficient. This is possible with nightly Rust, though awkward at times and generic bounds are quite large (I had a revision where bounds were over 100 lines long, but thankfully was able to get away without that).While new implementation is faster, it is still not as fast as could be. Once we settle with specific
K
or very narrow range ofK
we can optimize it further, there are still compute-bound aspects of the algorithm. C++ chiapos while is slower overall, still manages to derive the first table faster than Rust (is uses much more memory though). It is still beneficial to have generic implementation first such that we can compare it against reference implementation.table
tests are derived fromchiapos
and are passing successfully. At the same timetables
results are not matching C++ and with @dariolina we were not yet able to figure out why. They do a few strange things in the implementation comparing to the paper (seeto_chia_seed
in tests for instance). But we do have tests that ensure new implementation is self-consistent: qualities and proofs match, valid proofs verify successfully, invalid proofs fail to verify. Overall we decided this is good enough for now to land, but it is a breaking change unfortunately.Earlier revisions used
bitvec
create, which was nice, but slow for our use case, very slow. I then writecopy_bits
function that required a lot of mental gymnastics and still hasbitvec
sanity check in debug builds to ensure it works properly (it is very tricky, there might still be bugs). There are still improvements that can be done tocopy_bits
, but quick attempts didn't show massive improvement and given the fact that we'll doK
-specific bit manipulation in the future rather than generic bit copying anyway didn't seem worth it. I may refactorcopy_bits
into a generic standalone library, it seems to be generally useful for Rust ecosystem.Overall with these changes I was able to get sub-0.5s proving time on my machine:
I tried to leave comments around tricky parts of the implementation, but let me know if something is still not clear. I'm happy to do another implementation walk-though and some diagrams.
Code contributor checklist: