Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel chiapos #1453

Merged
merged 4 commits into from
May 13, 2023
Merged

Parallel chiapos #1453

merged 4 commits into from
May 13, 2023

Conversation

nazar-pc
Copy link
Member

@nazar-pc nazar-pc commented May 12, 2023

This PR introduces parallelism in chiapos that further massively reduces table construction time.

chia/table/parallel     time:   [25.994 ms 26.368 ms 27.029 ms]

Proving before:

proving/memory          time:   [344.57 ms 361.17 ms 379.25 ms]

Proving after:

proving/memory          time:   [150.26 ms 154.47 ms 158.15 ms]
                        thrpt:  [6.3231  elem/s 6.4737  elem/s 6.6550  elem/s]

Note that this is less efficient than single-threaded version in terms of CPU utilization and memory usage, we only use it for proving where latency is critical.

This has a breaking change for the farmer since order of qualities/proofs wasn't canonical before, now it is and can be different from what was expected in the past (and no migration code was written for this). As far as consensus verification is concerned, nothing has changed.

Code contributor checklist:

@nazar-pc nazar-pc requested review from dariolina and rahulksnv May 12, 2023 19:59
@nazar-pc nazar-pc added the breaking-farmer This PR introduces breaking changes to the farmer implementation label May 12, 2023
liuchengxu
liuchengxu previously approved these changes May 13, 2023
crates/subspace-proof-of-space/src/chiapos/table.rs Outdated Show resolved Hide resolved
@@ -23,14 +25,29 @@ fn pos_bench<PosTable>(
) where
PosTable: Table,
{
ThreadPoolBuilder::new()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're missing a #[cfg(feature = "parallel")] here?
It fails to compile subspace-proof-of-space separately:

error[E0433]: failed to resolve: use of undeclared type `ThreadPoolBuilder`
  --> crates/subspace-proof-of-space/benches/pos.rs:28:5
   |
28 |     ThreadPoolBuilder::new()
   |     ^^^^^^^^^^^^^^^^^ use of undeclared type `ThreadPoolBuilder`

For more information about this error, try `rustc --explain E0433`.
error: could not compile `subspace-proof-of-space` (bench "pos") due to previous error

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And if I bench anyway (cargo bench --features chia --bench pos) it panics

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: GlobalPoolAlreadyInitialized }', crates/subspace-proof-of-space/benches/pos.rs:32:10

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, though I'm not even sure if it should be limited to 4 cores by default (I wanted to have more representative results, using all cores would be faster).

@nazar-pc nazar-pc requested a review from dariolina May 13, 2023 12:39
Copy link
Member

@dariolina dariolina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is so cool! Both proving from memory and disk improved drastically for me.

proving/disk         time:   [5.5822 s 5.8284 s 6.0547 s]
                              thrpt:  [1.6516  elem/s 1.7157  elem/s 1.7914  elem/s]
proving/memory          time:   [197.67 ms 199.90 ms 203.87 ms]
                                      thrpt:  [4.9051  elem/s 5.0025  elem/s 5.0590  elem/s]
                                      

Copy link
Contributor

@rahulksnv rahulksnv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-farmer This PR introduces breaking changes to the farmer implementation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants