Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IF: Support Leap 5 Fork Database disk format and new Savanna disk format. #2306

Merged
merged 14 commits into from
Mar 15, 2024

Conversation

greg7mdp
Copy link
Contributor

@greg7mdp greg7mdp commented Mar 12, 2024

Resolves #2285.

This adds support for a new fork_database persistent file version (v2), which can store two versions of the fork_database (the legacy version storing block_state_legacy pointers, and the Savanna version storing block_state pointers.

In addition, the in_use variable indicating which version is currently in use is stored as well.

Old version (v1) which stores only the legacy version of fork_database, can still be read.

Re-enabled ship_streamer_if_test which now passes.

@greg7mdp greg7mdp marked this pull request as draft March 12, 2024 21:10
@greg7mdp greg7mdp requested review from heifner and linh2931 March 13, 2024 18:46
@greg7mdp greg7mdp marked this pull request as ready for review March 13, 2024 18:46
libraries/chain/fork_database.cpp Outdated Show resolved Hide resolved
tests/CMakeLists.txt Outdated Show resolved Hide resolved
libraries/chain/include/eosio/chain/fork_database.hpp Outdated Show resolved Hide resolved
libraries/chain/controller.cpp Outdated Show resolved Hide resolved
libraries/chain/include/eosio/chain/fork_database.hpp Outdated Show resolved Hide resolved
libraries/chain/include/eosio/chain/fork_database.hpp Outdated Show resolved Hide resolved
libraries/chain/include/eosio/chain/fork_database.hpp Outdated Show resolved Hide resolved
@@ -157,16 +161,18 @@ namespace eosio::chain {
// see fork_database_t::fetch_branch(forkdb->head()->id())
block_branch_t fetch_branch_from_head() const;

void reset_root(const block_handle::block_handle_variant_t& v);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really dislike that this exposes block_handle::block_handle_variant_t. I would prefer that type to not escape controller.cpp

Copy link
Contributor Author

@greg7mdp greg7mdp Mar 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I defined block_state_variant_t in block_state.hpp.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my mind this is even worse. I would say we either go all the way and provide every method that takes a variant or we continue to use apply/etc in controller. I say we keep with using apply in controller.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, I don't think it is worse.

I get that block_handle is its own class providing a unified view of a block_state and that we don't want other things to depend on its internals.

However, it seems that there are some cases where we need the new block_state_variant_t type, like when we want to reset the root of our external fork_database which contains two fork_db pointers. I don't see why it is bad to define this type?

My intent in doing this was to try to help with the transition case where we may have two simultaneous fork_dbs. It may not be the best solution for your actual needs. If you know at this point what the interface should be, I'm happy to update this PR.

Did you mean in your comment that I should replace all instances of block_state_variant_t with std::variant<block_state_legacy_ptr, block_state_ptr>?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean in your comment that I should replace all instances of block_state_variant_t with std::variant<block_state_legacy_ptr, block_state_ptr>?

No. We use apply for this type of thing elsewhere, just seem to me to be consisted to keep using apply. In my PR the creation of the new_head is in controller.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can leave as is, it will go away with my PR anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, I'll leave as-is, feel free to update as you see fit.
I don't think we can use controller's apply always because what we want to do may depend of in_use inside fork_db.

libraries/chain/include/eosio/chain/fork_database.hpp Outdated Show resolved Hide resolved
libraries/chain/include/eosio/chain/fork_database.hpp Outdated Show resolved Hide resolved
libraries/chain/fork_database.cpp Outdated Show resolved Hide resolved
libraries/chain/include/eosio/chain/block_handle.hpp Outdated Show resolved Hide resolved
return;
}

std::ofstream out( fork_db_file.generic_string().c_str(), std::ios::out | std::ios::binary | std::ofstream::trunc );
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems a bit odd to use cfile to read and std::ofstream to write. Might be better to use cfile.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was that way before my change. But yes I agree it is a little bit inconsistent. I can change it if you think it is worthwhile.

libraries/chain/fork_database.cpp Outdated Show resolved Hide resolved
libraries/chain/fork_database.cpp Outdated Show resolved Hide resolved
libraries/chain/fork_database.cpp Show resolved Hide resolved
libraries/chain/include/eosio/chain/fork_database.hpp Outdated Show resolved Hide resolved
@@ -1694,12 +1689,12 @@ struct controller_impl {
// If we start at a block during or after the IF transition, we need to provide this information
// at startup.
// ---------------------------------------------------------------------------------------------
if (fork_db.fork_db_if_present()) {
if (auto in_use = fork_db.version_in_use(); in_use == fork_database::in_use_t::both || in_use == fork_database::in_use_t::savanna) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you can have a method version_is_savanna() and version_is_legacy() to avoid this long expression. But no need to change now.

@@ -3025,7 +3020,7 @@ struct controller_impl {

chain_head = block_handle{std::move(new_head)};
if (s != controller::block_status::irreversible) {
fork_db.switch_from_legacy(chain_head);
fork_db.switch_from_legacy(chain_head.internal());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not your PR. This internal() implies details exposed.

@greg7mdp greg7mdp merged commit 36ebf7a into hotstuff_integration Mar 15, 2024
30 checks passed
@greg7mdp greg7mdp deleted the gh_2285 branch March 15, 2024 17:38
@ericpassmore
Copy link
Contributor

Note:start
group: IF
category: INTERNALS
summary: Adds support for a new fork_database persistent file version (v2), which can store two versions of the fork_database. Needed for consensus transition.
Note:end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

IF: Support Leap 5 Fork Database disk format and new Savanna disk format.
4 participants