Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store official GH pipeline benchmarks separately #146

Open
jordandsullivan opened this issue Dec 16, 2024 · 1 comment
Open

Store official GH pipeline benchmarks separately #146

jordandsullivan opened this issue Dec 16, 2024 · 1 comment
Labels
feature New feature or request infrastructure Non-quantum things to improve the robustness of our package, e.g. CI/CD

Comments

@jordandsullivan
Copy link
Collaborator

jordandsullivan commented Dec 16, 2024

Within the run_benchmarks.sh script, we just save everything in the ucc/benchmarks/results folder. It would be nice if we had a way to separate these official runs (on the GH action) from local testing runs we might do, or indeed the previous local runs from before we implemented the pipeline.

One way to do this is to modify the benchmark script something like

#!/bin/bash

if [ "$GITHUB_ACTIONS" == "true" ]; then
    # Define the results folder path
    RESULTS_FOLDER="$SCRIPT_DIR/../results/pipeline"
else
    # Define the results folder path
    RESULTS_FOLDER="$SCRIPT_DIR/../results"
fi

This would also solve the problem of file conflicts when writing to datafiles created on the same day (initially this is something I did to not end up with a bunch of individual datafiles when running multiple compiler x benchmark combinations in parallel.

In #133 I change the ucc/benchmarks/scripts/save function to only append to the same file if they were run in the same hour on the same day, which seems like a reasonable compromise to me.

@jordandsullivan jordandsullivan added feature New feature or request infrastructure Non-quantum things to improve the robustness of our package, e.g. CI/CD labels Dec 16, 2024
@bachase
Copy link
Collaborator

bachase commented Feb 12, 2025

An alternative would be to have two styles of automated runs

  1. Per release, do the existing style run and store the results back to the repo. I believe Update benchmark plot resolution to monthly #200 will be doing this.
  2. Add a separate action for any PR, that runs the benchmarks and adds a comment back to that PR showing the delta from the current baseline run. (Something like codecov does)This results would never be store in the repo, but github would have enough info to tell us what (if any changed) from that specific PR. This would also avoid needing to cleanup/delete commits from any manual runs. We might have to add some special tags, or limited which users can trigger a run, to avoid abuse of the action.

For any local testing, we could either not have it save the benchmark results at all, or save to a separate directory that .gitignore prevents from sharing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request infrastructure Non-quantum things to improve the robustness of our package, e.g. CI/CD
Projects
None yet
Development

No branches or pull requests

2 participants