Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Isolate the runtime instance of each benchmark #9285

Closed
kianenigma opened this issue Jul 6, 2021 · 3 comments · Fixed by #9373
Closed

Isolate the runtime instance of each benchmark #9285

kianenigma opened this issue Jul 6, 2021 · 3 comments · Fixed by #9373
Labels
J0-enhancement An additional feature request.

Comments

@kianenigma
Copy link
Contributor

kianenigma commented Jul 6, 2021

Currently, we create a single runtime instance, call into it once via the Benchmark_dispatch_benchmark, and all the benchmarks of a pallet is executed in one executor instance, with a single heap.

This makes verifying memory usage and safety pretty hard, since a single poor heap is being used (and constantly growing) for all benchmarks.

First, if you have a benchmark that is designed to just check a certain operation can succeed within a number of heap pages, you can't really test this.

Furthermore, if a pallet has a very large number of benchmarks, with relatively high number of components, you might get unexpected failures that one of the benchmarks ran out of heap, while if you run that benchmark in isolation it will succeed.

This will probably make everything very slow, but potentially we can move all the for loops outside of the runtime, and make one new runtime call per-benchmark-component, clearing the heap in between.

Alternatively, we could leave the current benchmarks as they are now, and build a secondary type of benchmarks that does what I just explained.

@kianenigma kianenigma added the J0-enhancement An additional feature request. label Jul 6, 2021
@kianenigma
Copy link
Contributor Author

cc @shawntabrizi

@bkchr
Copy link
Member

bkchr commented Jul 6, 2021

@shawntabrizi
Copy link
Member

yes, I have been meaning to do this as a part of the refactor i am working on

This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
J0-enhancement An additional feature request.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants