Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Trie log pruning #6000

Closed
wants to merge 16 commits into from
Closed

[WIP] Trie log pruning #6000

wants to merge 16 commits into from

Conversation

siladu
Copy link
Contributor

@siladu siladu commented Oct 9, 2023 •
edited
Loading

Added some subcommands for debugging...

(based on https://github.com/ahamlat/RocksdDB-Column-Families-Size)

Usage: besu operator x-rocksdb [-hV] [COMMAND]
Print RocksDB information
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
Commands:
  usage  Prints disk usage

Usage: besu operator x-trie-log [-hV] [--block[=<LONG>]] [--from[=<LONG>]] [--to
                                [=<LONG>]] [COMMAND]
Manipulate trie logs
      --block[=<LONG>]   The block
      --from, --from-block-number[=<LONG>]
                         Start of the block number range
  -h, --help             Show this help message and exit.
      --to, --to-block-number[=<LONG>]
                         End of the block number range
  -V, --version          Print version information and exit.
Commands:
  count   This command counts all the trie logs
  list    This command lists all the trie logs
  delete  Deletes the trie logs stored under the specified hash or block number
            range.
  prune   This command prunes all trie log layers below the specified block
            number, including orphaned trie logs.

Steps to install on a box:

cd /opt/besu
sudo wget -O 6000.tar.gz https://output.circle-artifacts.com/output/job/4d3d3d9c-25d5-41f1-ba04-d888dc1e4413/artifacts/0/distributions/besu-23.10.1-SNAPSHOT.tar.gz
sudo tar zxf 6000.tar.gz --transform s/besu-23.10.1-SNAPSHOT/6000/
time sudo /opt/besu/6000/bin/besu --config-file=/etc/besu/config.toml operator x-rocksdb usage

# offline only command:
time sudo /opt/besu/6000/bin/besu --config-file=/etc/besu/config.toml operator x-trie-log list

siladu added 7 commits October 9, 2023 06:45
@siladu
WIP - Add TrieLogSubCommand to get and delete trie log using TrieLogM…
6298a6a 
…anager

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
WIP - list trie logs
4ecc1f2 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
Add RocksDbSubCommand to print disk usage and count trie logs
dad1b64 
Copied from https://github.com/ahamlat/RocksdDB-Column-Families-Size

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
Tidy up printouts
c978279 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
List and Delete by block number range
e3e162f 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
Reorder methods
3c18d76 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
Add prune subcommand to prune below a specified block number
8142eb6 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@github-actions
Copy link

github-actions bot commented Oct 9, 2023

  • I thought about documentation and added the doc-change-required label to this PR if updates are required.
  • I thought about the changelog and included a changelog update if required.
  • If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

siladu added 2 commits October 10, 2023 15:06
@siladu
Add WIP comments
e197c5c 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
Print TRIE_LOG_STORAGE usage before and after prune
fba6eeb 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 10, 2023
View reviewed changes
besu/src/main/java/org/hyperledger/besu/cli/subcommands/operator/RocksDbUsageHelper.java
String size = rocksdb.getProperty(cfHandle, "rocksdb.estimate-live-data-size");
boolean emptyColumnFamily = false;
if (!size.isEmpty() && !size.isBlank()) {
long sizeLong = Long.parseLong(size);

Check notice

Code scanning / CodeQL

Missing catch of NumberFormatException

Potential uncaught 'java.lang.NumberFormatException'.
besu/src/main/java/org/hyperledger/besu/cli/subcommands/operator/RocksDbUsageHelper.java
String totolSstFilesSize = rocksdb.getProperty(cfHandle, "rocksdb.total-sst-files-size");
if (!totolSstFilesSize.isEmpty() && !totolSstFilesSize.isBlank()) {
out.println(
"Total size of SST Files : " + formatOutputSize(Long.parseLong(totolSstFilesSize)));

Check notice

Code scanning / CodeQL

Missing catch of NumberFormatException

Potential uncaught 'java.lang.NumberFormatException'.
besu/src/main/java/org/hyperledger/besu/cli/subcommands/operator/RocksDbUsageHelper.java
String liveSstFilesSize = rocksdb.getProperty(cfHandle, "rocksdb.live-sst-files-size");
if (!liveSstFilesSize.isEmpty() && !liveSstFilesSize.isBlank()) {
out.println(
"Size of live SST Filess : " + formatOutputSize(Long.parseLong(liveSstFilesSize)));

Check notice

Code scanning / CodeQL

Missing catch of NumberFormatException

Potential uncaught 'java.lang.NumberFormatException'.
siladu added 4 commits October 10, 2023 20:49
@siladu
Logging
8722162 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
More logging
a20375c 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
Revert "Logging"
797e55d 
This reverts commit 8722162.

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
Layers to retain rather than below block number
9a15a2a 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
Logging and +1 error
dfdf0f1 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
github-advanced-security[bot]
github-advanced-security bot found potential problems Oct 11, 2023
View reviewed changes
besu/src/main/java/org/hyperledger/besu/cli/subcommands/operator/TrieLogSubCommand.java

out.printf("Current head block number: %d\n", blockchain.getChainHead().getHeight());
out.printf("Trie log layers to retain: %d\n", layersToRetain);
final long deleteBelowHere = blockchain.getChainHead().getHeight() - layersToRetain + 1;

Check warning

Code scanning / CodeQL

Dereferenced variable may be null

Variable [layersToRetain](1) may be null at this access as suggested by [this](2) null guard.
siladu added 2 commits October 11, 2023 10:29
@siladu
Append counts to list subcommand
7212761 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu
Remove per trie log layer logging to test timings
6acc998 
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu siladu mentioned this pull request Oct 26, 2023
Merged
17 tasks
@siladu siladu mentioned this pull request Nov 17, 2023
Closed
21 tasks
@siladu
Copy link
Contributor Author

siladu commented Nov 21, 2023

Closing in favour on #6185 and #6188

@siladu siladu closed this Nov 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@siladu