feat: flat state value inlining migration #9037
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
df80d27 to
68ff47f
Compare
68ff47f to
5f23a97
Compare
jbajic
reviewed
May 16, 2023
Longarithm
approved these changes
May 17, 2023
5f23a97 to
452bb05
Compare
jbajic
approved these changes
May 19, 2023
452bb05 to
f6fc151
Compare
near-bulldozer bot
pushed a commit
that referenced
this pull request
May 25, 2023
…9093) Part of #8243. This PR enables the migration added in #9037 to be executed in the background on the running node. It supports graceful stop when the node is shut down. The implementation is heavily inspired by state sync background dumping to S3. This PR also introduces a new column `DBCol::Misc`. For now it only stores the status of the migration, but it can hold any small pieces of data, similar to `DBCol::BlockMisc`. `FlatStorageManager` is exposed as part of `RuntimeAdapter` in this PR. This is the first step in cleaning `RuntimeAdapter` from all other flat storage related methods, as the manager can be directly used instead. Tested by manually running a node and checking metrics and log messages. After that flat storage was checked with `flat-storage verify` cmd.
nikurt
pushed a commit
that referenced
this pull request
May 31, 2023
…9093) Part of #8243. This PR enables the migration added in #9037 to be executed in the background on the running node. It supports graceful stop when the node is shut down. The implementation is heavily inspired by state sync background dumping to S3. This PR also introduces a new column `DBCol::Misc`. For now it only stores the status of the migration, but it can hold any small pieces of data, similar to `DBCol::BlockMisc`. `FlatStorageManager` is exposed as part of `RuntimeAdapter` in this PR. This is the first step in cleaning `RuntimeAdapter` from all other flat storage related methods, as the manager can be directly used instead. Tested by manually running a node and checking metrics and log messages. After that flat storage was checked with `flat-storage verify` cmd.
nikurt
pushed a commit
that referenced
this pull request
Jun 13, 2023
…9093) Part of #8243. This PR enables the migration added in #9037 to be executed in the background on the running node. It supports graceful stop when the node is shut down. The implementation is heavily inspired by state sync background dumping to S3. This PR also introduces a new column `DBCol::Misc`. For now it only stores the status of the migration, but it can hold any small pieces of data, similar to `DBCol::BlockMisc`. `FlatStorageManager` is exposed as part of `RuntimeAdapter` in this PR. This is the first step in cleaning `RuntimeAdapter` from all other flat storage related methods, as the manager can be directly used instead. Tested by manually running a node and checking metrics and log messages. After that flat storage was checked with `flat-storage verify` cmd.
github-merge-queue bot
pushed a commit
that referenced
this pull request
Dec 15, 2024
…pshots (#12589) `test_resharding_v3_shard_shuffling_slower_post_processing_tasks` exposes a bug that can be triggered if child flat storages are not split after a resharding by the time we want to take a state snapshot. Then the state snapshot code will fail because the flat storage is not ready, but will not retry. To fix it, we add a `want_snapshot` field that will be set when we decide to take a state snapshot. We also add a `split_in_progress` field to the `FlatStorageManager` that will be set to `true` when a resharding is started, and back to false when it's finished and the catchup code has progressed to a height close to the desired snapshot height. The state snapshot code will wait until `split_in_progress` is false to proceed, and the flat storage catchup code will wait until `want_snapshot` is cleared if it has already advanced to the desired snapshot hash, so that we don't advance past the point that was wanted by the state snapshot. The first one is the one actually causing the test failure, but the second one is also required. We implement this waiting by rescheduling the message sends in the future. A Condvar would be a very natural choice, but it unfortunately doesn't seem to work in testloop, since actors that are normally running on different threads are put on the same thread, and a blocker on a Condvar won't be woken up. Here we are making a change to the behavior of the old `set_flat_state_updates_mode()`, which used to refuse to proceed if the update mode was already set to the same value. This seems to be an artifact of the fact that when state snapshots were implemented in #9090, this extra logic was added because there was another user of this function (`inline_flat_state_values()` added in #9037), but that function has since been deleted, so the state snapshot code is now the only user of `set_flat_state_updates_mode()`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #8243.
This PR implements migration process for inlining
FlatStatevalues. For more details see the second approach in this comment.Migration is not currently executed on the running node, that will be implemented in a separate PR. Instead
migrate-value-inliningsub-command is added as part offlat-storagecommand.Can be executed via
cargo run --release -p neard -- --verbose store flat-storage migrate-value-inlining. Progress log example: