refactor: Clean flat state via range #9109
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pugachAG
requested changes
May 25, 2023
core/store/src/flat/manager.rs
Outdated
| shard_uid: ShardUId, | ||
| shard_layout: ShardLayout, | ||
| ) -> Result<(), StorageError> { | ||
| pub fn remove_flat_storage_for_shard(&self, shard_uid: ShardUId) -> Result<(), StorageError> { | ||
| let mut flat_storages = self.0.flat_storages.lock().expect(POISONED_LOCK_ERR); | ||
|
|
||
| match flat_storages.remove(&shard_uid) { |
There was a problem hiding this comment.
let refactor this to avoid nasty None => {} case:
if let Some(flat_store) = flat_storages.remove(&shard_uid) {
...
}
| @@ -307,24 +307,20 @@ impl FlatStorage { | |||
| // TODO (#7327): call it just after we stopped tracking a shard. | |||
| // TODO (#7327): remove FlatStateChanges. Consider custom serialization of keys to remove them by | |||
| // prefix. | |||
| // TODO (#7327): support range deletions which are much faster than naive deletions. For that, we | |||
There was a problem hiding this comment.
also remove the following comments above, that is no longer relevant:
// Removes all items belonging to the shard one by one.
// Note that it does not work for resharding.
// TODO (#7327): remove FlatStateChanges.
core/store/src/flat/storage.rs
Outdated
| guard.shard_uid, | ||
| &[DBCol::FlatState], | ||
| ); | ||
| let removed_items = guard.store.iter(DBCol::FlatState).count(); |
There was a problem hiding this comment.
this is potentially expensive operation and the result is only used for logging, let's drop that part
core/store/src/flat/store_helper.rs
Outdated
| @@ -57,11 +57,16 @@ pub fn remove_delta(store_update: &mut StoreUpdate, shard_uid: ShardUId, block_h | |||
| store_update.delete(DBCol::FlatStateDeltaMetadata, &key); | |||
| } | |||
|
|
|||
| pub fn remove_all_deltas(store_update: &mut StoreUpdate, shard_uid: ShardUId) { | |||
| pub fn remove_range_by_shard_uid( | |||
There was a problem hiding this comment.
this is potentially dangerous public API since it accepts any DBCol while not all of those are prefixed by shard_uid.
I suggest to keep that as a private helper function and expose 2 public functions:
pub(crate) fn remove_all_flat_state_values(store_update: &mut StoreUpdate, shard_uid: ShardUId) -> Result<..> {
remove_range_by_shard_uid(store_update, shard_uid, &DBCol::FlatState)
}
pub(crate) fn remove_all_deltas(store_update: &mut StoreUpdate, shard_uid: ShardUId) ...
also I suggest changing parameter cols: &[DBCol] to col: &DBCol and just calling it twice in remove_all_deltas for changes and metadata.
nikurt
approved these changes
May 26, 2023
pugachAG
approved these changes
May 26, 2023
nikurt
pushed a commit
that referenced
this pull request
May 31, 2023
Use range deletion in `clean_state` when removing values from `FlatState` column. Addresses #8332
nikurt
pushed a commit
that referenced
this pull request
Jun 13, 2023
Use range deletion in `clean_state` when removing values from `FlatState` column. Addresses #8332
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use range deletion in
clean_statewhen removing values fromFlatStatecolumn. Addresses #8332