Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plot expectation value benchmarks #168

Merged
merged 37 commits into from
Feb 13, 2025
Merged

Plot expectation value benchmarks #168

Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
9f62ec5
remove unused QASM generation functions
jordandsullivan Jan 13, 2025
47632ad
Bug fixes for save
jordandsullivan Jan 13, 2025
c7dcd18
Add N=10 qubits QCNN file
jordandsullivan Jan 13, 2025
1ed386c
refactor expval benchmark to run single iteration
natestemen Jan 14, 2025
fc9a3e5
Merge branch 'plot-expectation-value' of https://github.com/unitaryfu…
jordandsullivan Jan 14, 2025
019a232
Copy benchpress QASM 2 files for N=10 and N=9 qubits
jordandsullivan Jan 14, 2025
c32ee7c
Rename qasm file to include basis
jordandsullivan Jan 14, 2025
dcb4322
Add filenames to expal run
jordandsullivan Jan 14, 2025
1c28fd3
Fix filenames
jordandsullivan Jan 14, 2025
22a35a7
ensure observable matches circuit size
natestemen Jan 15, 2025
5353c4b
add qisit aer to requirements
natestemen Jan 15, 2025
683310f
remove old expval data
natestemen Jan 15, 2025
1fb692f
first results
natestemen Jan 15, 2025
39f4ed2
first plot
natestemen Jan 15, 2025
8471ab3
more plots
natestemen Jan 15, 2025
782c920
uncomment first commands to run
natestemen Jan 17, 2025
011b185
s/relative/absolute for errors
natestemen Jan 17, 2025
e334d0c
add new data and remove relative errors
natestemen Jan 17, 2025
f6b6417
remove old plot
natestemen Jan 17, 2025
496731a
Merge branch 'main' into plot-expectation-value
jordandsullivan Jan 22, 2025
d4b4eed
Merge branch 'main' into plot-expectation-value
natestemen Feb 9, 2025
176e8b7
ensure noise is added to all target gates
natestemen Feb 9, 2025
4869cda
fix dependency name
natestemen Feb 11, 2025
12e40d6
use transpile as translate for consistency
natestemen Feb 11, 2025
6afbc51
Merge branch 'main' into plot-expectation-value
natestemen Feb 11, 2025
c4f5134
ignore python compiled files
natestemen Feb 11, 2025
2b6542b
new lockfile
natestemen Feb 11, 2025
ee4cc04
remove pycache from repo
natestemen Feb 11, 2025
914ea08
fix botched imports
natestemen Feb 11, 2025
f6d634e
use `glob` for consistency
natestemen Feb 12, 2025
8cb27ea
remove average from y axis
natestemen Feb 12, 2025
b9cbea5
fix typo
natestemen Feb 12, 2025
3bb18dd
describe violin plot of expectation values
natestemen Feb 12, 2025
a334c7a
use poetry to run expval benchmarks
natestemen Feb 12, 2025
cbeaa5b
refactor expval script
natestemen Feb 12, 2025
22945b6
formatting
natestemen Feb 12, 2025
6ec0268
Merge branch 'main' into plot-expectation-value
natestemen Feb 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ucc-benchmarks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ jobs:
poetry run ./benchmarks/scripts/run_benchmarks.sh 8 && \
poetry run python ./benchmarks/scripts/plot_avg_benchmarks_over_time.py && \
poetry run python ./benchmarks/scripts/plot_latest_benchmarks.py
poetry run python ./benchmarks/scripts/plot_latest_expval_benchmarks.py
"

# Commit and push benchmark results
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
*.ipynb_checkpoints
*.jupyter_cache
_build
*__pycache__/
__pycache__/
docs/build/
*.egg-info/
dist/
Expand Down
5 changes: 5 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,11 @@ def test_cirq_compile():

We run benchmarks regularly to compare against the most recent versions of the most popular quantum compiler frameworks for a range of circuits. Here's the latest:
![alt text](benchmarks/latest_compiler_benchmarks_by_circuit.png)

In addition to raw compilation stats, we simulate the compiled circuits with a noisy density matrix simulation to see how each compiler impacts performance:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth a sentence or two on what the absolute error definition is here? Or maybe better in the docs?

Copy link
Collaborator

@bachase bachase Feb 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related+for my own understanding.

We have two circuits, the original O and the compiled C = compile(O). For some measurement/metric M, we then generate

  • M(O)
  • M(noisy_simulation(C)) = M(noisy_simulation(compile(O))

My questions are

  1. Is the relative error we plot the ratio of the above two measurements?
  2. Are we assuming M(0) and M(C) are equal? In principle yes, but are all circuit transformations lossless, especially if floating point precision comes into play?
  3. Or asking from another angle, do we care about a difference between M(noisy_simulation(O)) and M(noisy_simulation(C)) compared to M(O) to know how much more noise-tolerant the compiled circuit is than the original circuit?

cc @Misty-W as this also related to the benchmark specific measurement work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. currently we are only plotting absolute error due to numerical blow-ups when trying to plot relative errors when the ideal value is 0, or 1e-20. But, to answer your question, I was using the following definition of relative error which is recorded in the results object.

$$ \frac{|\text{idea} - \text{noisy}|}{\text{ideal}}$$

  1. We do make the assumption that M(O) == M(C), and that's a good point about approximate compilation. I'll punt that question to @Misty-W since I'm not super familiar with the exactness of current passes.
  2. Yeah I think that would be a great future addition. Usually approximate compilation techniques are quantified by a trace/operator norm on the unitaries, but I'm sure it would be interesting to look at how the unitaries give rise to different expectation values as well.

Is it worth a sentence or two on what the absolute error definition is here?

I'll add a note in the README which is also part of the built documentation.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do have approximate compilation passes in UCC, they are:

  • Optimize1qGates: Approximates single-qubit unitaries up to numerical precision.
  • UnitarySynthesis: Approximates arbitrary unitaries with a decomposition based on a given synthesis method.
  • Collect2qBlocks: Can group two-qubit operations in ways that lead to approximate synthesis (in some cases).

Agreed, benchmarking circuits for noise resilience is a great follow-up.


![Violin plot showing absolute error of each compiler across a variety of circuits](benchmarks/latest_expval_benchmark_by_compiler.png)

And here you can see progress over time, with new package versions labeled for each compiler:
![alt text](benchmarks/avg_compiler_benchmarks_over_time.png)
Note that the compile times before 2024-12-10 may have been run on different classical compute instances, so their runtime is not reported here, but you can find this data in benchmarks/results.
Expand Down
Binary file added benchmarks/average_absolute_error_over_time.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added benchmarks/latest_absolute_errors_by_circuit.png
View file
Open in desktop
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little suspicious how the QAOA, QV, and QCNN circuit results are all basically identical. We should make sure this is real and not an artifact of how we perform simulation (or something else)!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you plot the expectation values themselves and standard deviations as suggested above?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's compare the compiled gate counts between the different compilers in case they are returning approximately the same circuits.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added benchmarks/latest_expval_benchmark_by_compiler.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
197 changes: 197 additions & 0 deletions ...arks/qasm_circuits/qasm2/benchpress/qaoa_barabasi_albert_N10_3reps_basis_rz_rx_ry_cx.qasm
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
OPENQASM 2.0;
include "qelib1.inc";
qreg q[10];
ry(pi/2) q[0];
rx(pi) q[0];
ry(pi/2) q[1];
rx(pi) q[1];
cx q[0],q[1];
rz(1.5846238651156088) q[1];
cx q[0],q[1];
ry(pi/2) q[2];
rx(pi) q[2];
cx q[0],q[2];
rz(1.5846238651156088) q[2];
cx q[0],q[2];
rx(1.9993980494552113) q[2];
ry(pi/2) q[3];
rx(pi) q[3];
cx q[0],q[3];
rz(1.5846238651156088) q[3];
cx q[0],q[3];
rx(1.9993980494552113) q[0];
cx q[1],q[3];
rz(1.5846238651156088) q[3];
cx q[1],q[3];
ry(pi/2) q[4];
rx(pi) q[4];
cx q[1],q[4];
rz(1.5846238651156088) q[4];
cx q[1],q[4];
cx q[3],q[4];
rz(1.5846238651156088) q[4];
cx q[3],q[4];
ry(pi/2) q[5];
rx(pi) q[5];
cx q[1],q[5];
rz(1.5846238651156088) q[5];
cx q[1],q[5];
cx q[3],q[5];
rz(1.5846238651156088) q[5];
cx q[3],q[5];
ry(pi/2) q[6];
rx(pi) q[6];
cx q[4],q[6];
rz(1.5846238651156088) q[6];
cx q[4],q[6];
cx q[5],q[6];
rz(1.5846238651156088) q[6];
cx q[5],q[6];
rx(1.9993980494552113) q[6];
ry(pi/2) q[7];
rx(pi) q[7];
cx q[3],q[7];
rz(1.5846238651156088) q[7];
cx q[3],q[7];
cx q[4],q[7];
rz(1.5846238651156088) q[7];
cx q[4],q[7];
rx(1.9993980494552113) q[4];
rx(1.9993980494552113) q[7];
ry(pi/2) q[8];
rx(pi) q[8];
cx q[3],q[8];
rz(1.5846238651156088) q[8];
cx q[3],q[8];
cx q[5],q[8];
rz(1.5846238651156088) q[8];
cx q[5],q[8];
rx(1.9993980494552113) q[5];
rx(1.9993980494552113) q[8];
ry(pi/2) q[9];
rx(pi) q[9];
cx q[1],q[9];
rz(1.5846238651156088) q[9];
cx q[1],q[9];
rx(1.9993980494552113) q[1];
cx q[0],q[1];
rz(0.8199726992929931) q[1];
cx q[0],q[1];
cx q[0],q[2];
rz(0.8199726992929931) q[2];
cx q[0],q[2];
rx(9.808607416083087) q[2];
cx q[3],q[9];
rz(1.5846238651156088) q[9];
cx q[3],q[9];
rx(1.9993980494552113) q[3];
cx q[0],q[3];
rz(0.8199726992929931) q[3];
cx q[0],q[3];
rx(9.808607416083087) q[0];
cx q[1],q[3];
rz(0.8199726992929931) q[3];
cx q[1],q[3];
cx q[1],q[4];
rz(0.8199726992929931) q[4];
cx q[1],q[4];
cx q[1],q[5];
cx q[3],q[4];
rz(0.8199726992929931) q[4];
cx q[3],q[4];
cx q[4],q[6];
rz(0.8199726992929931) q[5];
cx q[1],q[5];
cx q[3],q[5];
rz(0.8199726992929931) q[5];
cx q[3],q[5];
cx q[3],q[7];
rz(0.8199726992929931) q[6];
cx q[4],q[6];
cx q[5],q[6];
rz(0.8199726992929931) q[6];
cx q[5],q[6];
rx(9.808607416083087) q[6];
rz(0.8199726992929931) q[7];
cx q[3],q[7];
cx q[3],q[8];
cx q[4],q[7];
rz(0.8199726992929931) q[7];
cx q[4],q[7];
rx(9.808607416083087) q[4];
rx(9.808607416083087) q[7];
rz(0.8199726992929931) q[8];
cx q[3],q[8];
cx q[5],q[8];
rz(0.8199726992929931) q[8];
cx q[5],q[8];
rx(9.808607416083087) q[5];
rx(9.808607416083087) q[8];
rx(1.9993980494552113) q[9];
cx q[1],q[9];
rz(0.8199726992929931) q[9];
cx q[1],q[9];
rx(9.808607416083087) q[1];
cx q[0],q[1];
rz(1.2724239654332838) q[1];
cx q[0],q[1];
cx q[0],q[2];
rz(1.2724239654332838) q[2];
cx q[0],q[2];
rx(4.304211127616218) q[2];
cx q[3],q[9];
rz(0.8199726992929931) q[9];
cx q[3],q[9];
rx(9.808607416083087) q[3];
cx q[0],q[3];
rz(1.2724239654332838) q[3];
cx q[0],q[3];
rx(4.304211127616218) q[0];
cx q[1],q[3];
rz(1.2724239654332838) q[3];
cx q[1],q[3];
cx q[1],q[4];
rz(1.2724239654332838) q[4];
cx q[1],q[4];
cx q[1],q[5];
cx q[3],q[4];
rz(1.2724239654332838) q[4];
cx q[3],q[4];
cx q[4],q[6];
rz(1.2724239654332838) q[5];
cx q[1],q[5];
cx q[3],q[5];
rz(1.2724239654332838) q[5];
cx q[3],q[5];
cx q[3],q[7];
rz(1.2724239654332838) q[6];
cx q[4],q[6];
cx q[5],q[6];
rz(1.2724239654332838) q[6];
cx q[5],q[6];
rx(4.304211127616218) q[6];
rz(1.2724239654332838) q[7];
cx q[3],q[7];
cx q[3],q[8];
cx q[4],q[7];
rz(1.2724239654332838) q[7];
cx q[4],q[7];
rx(4.304211127616218) q[4];
rx(4.304211127616218) q[7];
rz(1.2724239654332838) q[8];
cx q[3],q[8];
cx q[5],q[8];
rz(1.2724239654332838) q[8];
cx q[5],q[8];
rx(4.304211127616218) q[5];
rx(4.304211127616218) q[8];
rx(9.808607416083087) q[9];
cx q[1],q[9];
rz(1.2724239654332838) q[9];
cx q[1],q[9];
rx(4.304211127616218) q[1];
cx q[3],q[9];
rz(1.2724239654332838) q[9];
cx q[3],q[9];
rx(4.304211127616218) q[3];
rx(4.304211127616218) q[9];
Loading