Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FlatDbStrategy #5901

Merged
merged 5 commits into from
Sep 22, 2023
Merged

Conversation

garyschulte
Copy link
Contributor

@garyschulte garyschulte commented Sep 19, 2023

PR description

This is a prerequisite PR for #5865 and #5887.

It:

  • expands the concept of FlatDbReaderStrategy to FlatDbStrategy, including write operations.
  • adds getNearestTo implementation to SegmentedKeyValueStorage and its implementations
  • adds getNearestTo test coverage and expands KeyValueStorage tests to SegmentedKeyValueStorage implementations

This PR will allow for parallel progress on SnapServer server and Bonsai Archive, and mitigate merge conflicts that would otherwise arise

Fixed Issue(s)

related to #3165 and #5864

@github-actions
Copy link

  • I thought about documentation and added the doc-change-required label to this PR if updates are required.
  • I thought about the changelog and included a changelog update if required.
  • If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

@garyschulte garyschulte requested a review from matkt September 19, 2023 23:25
@garyschulte garyschulte marked this pull request as ready for review September 19, 2023 23:26
@garyschulte garyschulte force-pushed the feature/flat-db-strategy branch from 13da7d2 to 4eb1ed6 Compare September 19, 2023 23:26
// Only ourNearest is present
return ourNearest;
} else {
// Either both are not present, or only parentNearest is present
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Either both are not present, or only parentNearest is present

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it took me longer to parse this comment than the line of code

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this end up being Optional.empty if neither are present? What happens then?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, if neither are present there was nothing at or before the requested key and we return empty. Callers should expect to handle the optional empty return

Copy link
Contributor Author

@garyschulte garyschulte Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this end up being Optional.empty if neither are present? What happens then?

This question made me add another case that checks we correctly return Optional.empty(). Though the rocksdb nearestTo would behave that way, the In-memory comparators are not. I added a test for this and a filter to the in-memory streaming implementation. 👍

final Hash accountHash,
final Hash slotHash) {
transaction.remove(
ACCOUNT_STORAGE_STORAGE, Bytes.concatenate(accountHash, slotHash).toArrayUnsafe());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it called ACCOUNT_STORAGE_STORAGE ? there's no ACCOUNT_STORAGE, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, not my favorite name, but it is what the segment enum calls it. It is the underlying storage for account storage, but makes for a weird name. I am all for renaming the flat segments to ACCOUNT_INFO_FLAT, ACCOUNT_STORAGE_FLAT, and CODE_STORAGE_FLAT or something else that is more meaningful. I'd rather do that in another PR though since it would touch a lot of places

@garyschulte garyschulte force-pushed the feature/flat-db-strategy branch 3 times, most recently from 21a3368 to dd4e2b4 Compare September 22, 2023 05:23
Copy link
Contributor

@matkt matkt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
…es the same

Signed-off-by: garyschulte <garyschulte@gmail.com>
@garyschulte garyschulte force-pushed the feature/flat-db-strategy branch from dd4e2b4 to 850c4ca Compare September 22, 2023 15:57
@garyschulte garyschulte enabled auto-merge (squash) September 22, 2023 15:58
@garyschulte garyschulte merged commit d81e1f3 into hyperledger:main Sep 22, 2023
@garyschulte garyschulte deleted the feature/flat-db-strategy branch September 22, 2023 19:40
jflo pushed a commit to jflo/besu that referenced this pull request Nov 10, 2023
* move FlatDbReader to FlatDbStrategy (including writes), add getNearestTo

Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: Justin Florentine <justin+github@florentine.us>
NickSneo pushed a commit to NickSneo/besu that referenced this pull request Nov 12, 2023
* move FlatDbReader to FlatDbStrategy (including writes), add getNearestTo

Signed-off-by: garyschulte <garyschulte@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants