Evaluating solver performance in BENCHMARK with different reordering algorithms

[stanisic]

I would like to benchmark the impact of different reordering algorithms (most importantly RCM) already implemented in Ginkgo on the total time to solve some matrices. Ideally, this would be a BENCHMARK option similar to PRECONDS, but I understand that implementing and documenting this properly might take some time. Until then, I would highly appreciate any quick, dirty and temporary patch/suggestion :)

BTW I already tried the code implemented in PR #1354, and it indeed helps evaluating performance of reordering algorithms themselves. However, I am not confident that I know what the correct way would be to integrate something similar into solver benchmark.

[upsj]

I am not sure how much we should be extending the run_all_benchmarks.sh script, I don't think we're using it anywhere outside CI benchmarking jobs.

[stanisic]

run_all_benchmarks.sh script is very convenient for benchmarking Ginkgo's performance in order to find hotspots, compare it to the other solvers... Therefore, in our team we used this script extensively. I do not know if there are many other external users like us.

That being said, I followed the discussion in #1408 and I understand now why adding this as a cross-product to already existing precond-solver combination is probably not a good solution. I guess solutions proposed in #1408 should be OK, especially if combined with description/examples on BENCHMARKING documentation page about how to benchmark different reordering algorithms.