feat: add ETL to Hashing Stages
#7030
Conversation
| @@ -356,91 +310,6 @@ mod tests { | |||
| assert!(runner.validate_execution(input, result.ok()).is_ok(), "execution validation"); | |||
| } | |||
|
|
|||
| #[tokio::test] | |||
| async fn execute_clean_account_hashing_with_commit_threshold() { | |||
There was a problem hiding this comment.
clean hashing execution no longer relies on commit_threshold
| @@ -310,156 +263,21 @@ mod tests { | |||
| } | |||
| } | |||
|
|
|||
| #[tokio::test] | |||
| async fn execute_clean_storage_hashing_with_commit_threshold() { | |||
There was a problem hiding this comment.
clean hashing execution no longer relies on commit_threshold
| // Spawn the hashing task onto the global rayon pool | ||
| rayon::spawn(move || { |
There was a problem hiding this comment.
this has the same issue as sender recovery stage,
what we could do instead is add a pool that has < available_parallelism() threads
this would also solve the sender recovery issue I believe,
ref
reth/crates/rpc/rpc-builder/src/constants.rs
Lines 30 to 31 in 273f3c7
There was a problem hiding this comment.
should take care of it
| let mut channels = Vec::with_capacity(10_000); | ||
|
|
||
| // itertools chunks doesn't support enumerate, so we use a counter to know when to flush | ||
| // hashes from channels to disk. | ||
| let mut flush_counter = 1; | ||
|
|
||
| // channels used to return result of account hashing | ||
| for chunk in &accounts_cursor.walk(start)?.chunks(100) { |
There was a problem hiding this comment.
where are these numbers coming from?
There was a problem hiding this comment.
Made a few changes, but they're arbitrary. I tried this in some benchmarks, and didn't have apparent performance degradation. We might want to tweak them in the future though.
Before we were just pushing all channels into vec, which would grow the memory usage immensily. So batching and flushing seems a more reasonable approach.
ref #6909
When taking the code path of hashing every account/storage (aka clean hashing), it now uses ETL without comitting inbetween.
Mainnet
AccountsHashing
StorageHashing
Holesky @ 1M
AccountsHashing
StorageHashing
349 seconds