Generate Daily Benchmark Results

[jdjaustin]

Summary of changes
Changes introduced in this pull request:
Generate the following daily benchmark results:

1. Time to load a mainnet snapshot.
2. Validation time in tipsets per second.

Reference issue to close (if applicable)

Closes #2208

Other information and links

This PR is just for the script. The script will be added to forest-iac under a separate PR.

[jdjaustin]

@elmattic I mentioned I would tag you once I had a draft PR; if you have some guidance on how to handle the case where I need to have the node running and run commands in a separate window (my plan was to open a new terminal with system 'gnome-terminal' but if there's a better strategy, I'm open to ideas), that would be helpful! I'm not finding a lot of documentation on how to implement that task...

[elmattic]

@jdjaustin What is the purpose of opening the gnome terminal?

Generaly speaking I would more first perform all commands related to Forest, then Lotus (or vice versa). Otherwise you will have to keep both db in-between and this will require to double the size.

[elmattic]

You can find inspiration in my db script, if you need to run a command while the node is running, ie this one I'm using a thread to monitor RSS of the process and sample the value 2 times per second.

[jdjaustin]

@elmattic I've uploaded my calcs for Forest import time and validation time if you want to take a look and see if you have any suggestions. I've started the task of replicating this for Lotus, then I'll work on the logic to move all these calcs into the bench.rb file. rss doesn't seem to be updating for memory usage during the import step, so I need to troubleshoot that as well.

[CLAassistant]

All committers have signed the CLA.

[jdjaustin]

Tested all the latest changes with mainnet on a droplet running Fedora 37.

[LesnyRumcajs]

1. Lingering files in the repository. Should they be git-ignored?

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        scripts/benchmark_db/.bundle/
        scripts/benchmark_db/result_1684246301.md
        scripts/benchmark_db/vendor/

2. Ruby gem missing when following instructions. Please try the instructions from README from a brand-new, clean droplet.

[LesnyRumcajs]

How about we put at least a dry run in the CI?

[LesnyRumcajs]

In general, it looks good, it doesn't affect Forest so I'm good with merging it as is. It'd be great to go through all of the instructions and possible flags on a clean VPS, just to be sure the README is up-to-date.

I'd also add it in the CI in at least some form unless we want to migrate it soon to forest-iac.

[jdjaustin]

2. Ruby gem missing when following instructions. Please try the instructions from README from a brand-new, clean droplet.

I was able to reproduce this issue, but I'm not sure how to fix it yet. bundle install says it is successful, but then the script says cannot load such file for the offending gems (concurrent-ruby, deep_merge, tomlrb, and toml-rb). Interestingly, a manual gem install of these seems to work. @elmattic did you encounter this when you were testing on Fedora 37? I'm tempted to just add a gem install step to the instructions, unless y'all have any suggestions for a better fix.

[elmattic]

How about we put at least a dry run in the CI?

This is a good idea. However I would like a true dry run flag for forest-cli and forest binary, so we can validate inputs. (edit: added to them a --dry-run flag).

[elmattic]

2. Ruby gem missing when following instructions. Please try the instructions from README from a brand-new, clean droplet.

I was able to reproduce this issue, but I'm not sure how to fix it yet. bundle install says it is successful, but then the script says cannot load such file for the offending gems (concurrent-ruby, deep_merge, tomlrb, and toml-rb). Interestingly, a manual gem install of these seems to work. @elmattic did you encounter this when you were testing on Fedora 37? I'm tempted to just add a gem install step to the instructions, unless y'all have any suggestions for a better fix.

I let you know, will try the script on a fresh Fedora 37.

[elmattic]

I've tested on Ubuntu 22.04 (LTS) x64 and did manage to reproduce the issue.

$ bundle config set --local path 'vendor/bundle'
$ bundle install
Bundler 2.3.5 is running, but your lockfile was generated with 2.4.6. Installing Bundler 2.4.6 and restarting using that version.
Fetching gem metadata from https://rubygems.org/.
Fetching bundler 2.4.6
Installing bundler 2.4.6
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Using bundler 2.4.6
Fetching concurrent-ruby 1.2.2
Fetching deep_merge 1.2.2
Fetching citrus 3.0.2
Fetching csv 3.2.6
Installing deep_merge 1.2.2
Installing citrus 3.0.2
Installing csv 3.2.6
Fetching fileutils 1.7.0
Installing fileutils 1.7.0
Fetching logger 1.5.3
Installing concurrent-ruby 1.2.2
Fetching open3 0.1.2
Fetching optparse 0.3.1
Installing open3 0.1.2
Installing logger 1.5.3
Installing optparse 0.3.1
Fetching set 1.0.3
Fetching tomlrb 2.0.3
Installing set 1.0.3
Installing tomlrb 2.0.3
Fetching tmpdir 0.1.3
Fetching toml-rb 2.2.0
Installing tmpdir 0.1.3
Installing toml-rb 2.2.0
Bundle complete! 11 Gemfile dependencies, 13 gems now installed.
Bundled gems are installed into `./vendor/bundle`
$ cd ..
$ ./scripts/benchmark_db/bench.rb --daily --chain calibnet --tempdir ../benchdir
<internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- concurrent-ruby (LoadError)
	from <internal:/usr/lib/ruby/vendor_ruby/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from ./scripts/benchmark_db/bench.rb:7:in `<main>'

[elmattic]

I think the error is due to improperly setting the path where GEMs are installed,vendor/bundle (check using bundler env).
A cleaner way to install and run the script could be:

• cd to scripts/benchmark_db
• bundle config set --local path 'vendor/bundle'
• bundle config set --local deployment true
• bundle install
• bundle exec bench.rb --daily --chain calibnet ...