You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Compile ratio and time are changing despite no changes to the benchmarking circuits and methods, and no version changes in any of the compilers (including UCC).
Expected behavior
Performance should be the same until UCC makes a change, a new compiler version is released, or the benchmarks are updated.
Screenshots
Additional context
Re-running filled in missing data in a different plot, but didn't solve the entire issue.
The text was updated successfully, but these errors were encountered:
I believe the root cause of this issue is that run-benchmarks attempt # 1 on commit c135493 began close to midnight UTC, so some data were saved on Feb 3 at 23h and and some on Feb 4 at 00h, resulting in separate files and separate points on the plot over time.
The benchmarking logs indicate results that were saved to ucc/benchmarks/scripts/../results/gates_2025-02-03_23.csv and the rest of the results saved to ucc/benchmarks/scripts/../results/gates_2025-02-04_00.csv.
We need a way of unifying the results of a benchmarking run in a single file, even when the hour (day, month, or even year) changes.
Describe the bug
Compile ratio and time are changing despite no changes to the benchmarking circuits and methods, and no version changes in any of the compilers (including UCC).
Expected behavior
Performance should be the same until UCC makes a change, a new compiler version is released, or the benchmarks are updated.
Screenshots

Additional context
Re-running filled in missing data in a different plot, but didn't solve the entire issue.
The text was updated successfully, but these errors were encountered: