You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The idea is to support updating the trie with given storage directly into the underlying database instead of collecting the db_updates via PrefixedMemoryDB and then applied into the database. Such that we can avoid the potential OOM issue when dealing with a large state import.
Once this feature is implemented, #4 can be resolved easily by forwarding the storage key/values to the db whenever a piece of the downloaded state is received from the peer instead of keeping them in the memory until the completion of entire state download.
Implementation plan
Create TrieCommitter by adapting Ephermal, specifically, replace overlay which is PrefixedMemoryDB with the underlying database. And then the new keys will be directly written to the DB.
@bkchr Please validate the implementation plan above and I'd appreciate if some expert could help look into the IncompleteDatabase error, I have no idea why merely inserting new keys would result in this error. Just pull https://github.com/liuchengxu/polkadot-sdk/tree/trie-committer and starting a local fast sync test should reproduce it.
The text was updated successfully, but these errors were encountered:
The idea is to support updating the trie with given storage directly into the underlying database instead of collecting the
db_updates
viaPrefixedMemoryDB
and then applied into the database. Such that we can avoid the potential OOM issue when dealing with a large state import.Once this feature is implemented, #4 can be resolved easily by forwarding the storage key/values to the db whenever a piece of the downloaded state is received from the peer instead of keeping them in the memory until the completion of entire state download.
Implementation plan
TrieCommitter
by adaptingEphermal
, specifically, replaceoverlay
which isPrefixedMemoryDB
with the underlying database. And then the new keys will be directly written to the DB.polkadot-sdk/substrate/primitives/state-machine/src/trie_backend_essence.rs
Lines 717 to 720 in df12fd3
Add new API
commit_trie_changes()
inBackend
traitReplace
reset_storage
with the new APIcommit_trie_changes()
.polkadot-sdk/substrate/client/service/src/client/client.rs
Line 688 in df12fd3
I have made a PoC based on the above flow, you can find it here. I added a basic test for the new API
commit_trie_changes()
https://github.com/liuchengxu/polkadot-sdk/blob/09c2b4d6f35ebd9f42917813357d57075123f222/substrate/client/db/src/lib.rs#L4768, it passes, which means applying the state changes using new API seems to work, but simplying replacingreset_storage
withcommit_trie_changes()
runs into the errorIncompleteDatabase
when conducting a fast sync test locally.@bkchr Please validate the implementation plan above and I'd appreciate if some expert could help look into the
IncompleteDatabase
error, I have no idea why merely inserting new keys would result in this error. Just pull https://github.com/liuchengxu/polkadot-sdk/tree/trie-committer and starting a local fast sync test should reproduce it.The text was updated successfully, but these errors were encountered: