perf(persistence): reuse cursor for updating history indices #13622
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hai-rise
commented
Jan 3, 2025
| where | ||
| T: Table<Value = BlockNumberList>, | ||
| { | ||
| let mut cursor = self.tx.cursor_read::<T>()?; |
There was a problem hiding this comment.
This created a new cursor per account or storage slot change.
| // delete old shard so new one can be inserted. | ||
| self.tx.delete::<T>(shard_key, None)?; |
There was a problem hiding this comment.
This created another cursor and re-seeked even when cursor is already on the needed key.
| while let Some(list) = chunks.next() { | ||
| let highest_block_number = if chunks.peek().is_some() { | ||
| *list.last().expect("`chunks` does not return empty list") | ||
| } else { | ||
| // Insert last list with `u64::MAX`. | ||
| u64::MAX | ||
| }; | ||
| self.tx.put::<T>( |
There was a problem hiding this comment.
This created another cursor per chunk.
mattsse
approved these changes
Jan 3, 2025
Comment on lines
+842
to
+840
| let mut cursor = self.tx.cursor_write::<T>()?; | ||
| for (partial_key, indices) in index_updates { | ||
| let mut last_shard = | ||
| self.take_shard::<T>(sharded_key_factory(partial_key, u64::MAX))?; | ||
| self.take_shard::<T>(&mut cursor, sharded_key_factory(partial_key, u64::MAX))?; |
There was a problem hiding this comment.
I think this makes sense @rkrasiuk
rkrasiuk
approved these changes
Jan 3, 2025
| while let Some(list) = chunks.next() { | ||
| let highest_block_number = if chunks.peek().is_some() { | ||
| *list.last().expect("`chunks` does not return empty list") | ||
| } else { | ||
| // Insert last list with `u64::MAX`. | ||
| u64::MAX | ||
| }; | ||
| self.tx.put::<T>( | ||
| cursor.insert( |
There was a problem hiding this comment.
@shekhirin @joshieDo any difference between tx.put and cursor.insert?
There was a problem hiding this comment.
when inserting multiple values, generally yes: #1130 (comment)
joshieDo
approved these changes
Jan 8, 2025
|
@hai-rise mind rebasing this pr? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In our latest test load,
save_blocksspends ~2.5% of its time creating and dropping new cursors intake_shardalone. Another ~15% is spent on relatedtx.deleteandtx.putthat internally also create a new cursor per call.This PR minimizes the number of cursors per
append_history_indexfrom the number of changes (accounts or storage slots) * (2 + number of block shards) to only 1 cursor per table.P/s: this is untested code but I hope it improves
save_blocksby 3~5% for us.