Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected spikes in benchmark plots #206

Closed
Misty-W opened this issue Feb 4, 2025 · 2 comments · Fixed by #207
Closed

Unexpected spikes in benchmark plots #206

Misty-W opened this issue Feb 4, 2025 · 2 comments · Fixed by #207
Assignees
Labels
bug Something isn't working

Comments

@Misty-W
Copy link
Collaborator

Misty-W commented Feb 4, 2025

Describe the bug
Compile ratio and time are changing despite no changes to the benchmarking circuits and methods, and no version changes in any of the compilers (including UCC).

Expected behavior
Performance should be the same until UCC makes a change, a new compiler version is released, or the benchmarks are updated.

Screenshots
Image

Additional context
Re-running filled in missing data in a different plot, but didn't solve the entire issue.

@Misty-W Misty-W added the bug Something isn't working label Feb 4, 2025
@Misty-W Misty-W self-assigned this Feb 4, 2025
@Misty-W
Copy link
Collaborator Author

Misty-W commented Feb 4, 2025

Some ideas so far:

  1. Was the later flow on 2/4 used?
  2. Incomplete data needs to be scrubbed from the "over time" plots

@Misty-W
Copy link
Collaborator Author

Misty-W commented Feb 5, 2025

I believe the root cause of this issue is that run-benchmarks attempt # 1 on commit c135493 began close to midnight UTC, so some data were saved on Feb 3 at 23h and and some on Feb 4 at 00h, resulting in separate files and separate points on the plot over time.

The benchmarking logs indicate results that were saved to ucc/benchmarks/scripts/../results/gates_2025-02-03_23.csv and the rest of the results saved to ucc/benchmarks/scripts/../results/gates_2025-02-04_00.csv.

We need a way of unifying the results of a benchmarking run in a single file, even when the hour (day, month, or even year) changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant