Plot expectation value benchmarks #168
Conversation
the new data isn't the same format
I think it's because the ideal values are coming out so close to 0 that the relative errors are blowing up. You can find all the most recent results in ucc/benchmarks/results/expval_2025-01-14_20.csv Lines 11 to 14 in 8471ab3 Not sure what the best course of action here is. |
|
Okay just as a sanity check can you plot the simulated and ideal expectation values and standard deviation, similar to what I did here for #58 (where I was running on real hardware)? This reminds me, we can also simply measure an array of observables in addition to ZZZZZZ like I did there. Maybe just adding in some that measure like XIIIIIII or XXXXXZZZZZ, etc. |
|
@natestemen #220 (and f6a53bd) in particular is my attempt to fix the cirq issue. Let me know if this helps with your noise model approach (or I can try testing that myself too) |
There are some new additions (circuits of sizes that are reasonable to simulate). These were added so we could have something to test one. If you see any QASM files that were modified let me know, because that was definitely not the intention. |
There was a problem hiding this comment.
Was it intentional to delete this file?
| ax.set_xlabel("Date") | ||
| ax.set_ylabel("Average Absolute Error") | ||
| ax.grid(True) | ||
| ax.legend(title="Compiler") |
There was a problem hiding this comment.
Do we want to use the same version labels shown in the other plots?
There was a problem hiding this comment.
Hmmm I think probably yes, but I'll leave that for a subsequent PR.
|
|
||
| simulator = AerSimulator( | ||
| method="density_matrix", noise_model=depolarizing_noise | ||
| def parse_arguments() -> tuple[str, str, str, bool]: |
There was a problem hiding this comment.
Another style nit for the future -- we should consider [argparse](https://docs.python.org/3/library/argparse.html) or [click](https://click.palletsprojects.com/en/stable/) for managing command line parsing. Likely some other libraries out there these days too.
| for qasm_file in "${QASM_EXPVAL_FILES[@]}"; do | ||
| for compiler in "${COMPILERS[@]}"; do | ||
| # Combine the common folder path with the QASM file | ||
| full_qasm_file="${QASM_FOLDER}${qasm_file}" | ||
|
|
||
| # Build the command, passing the results folder as an argument | ||
| command="python3 $(dirname "$0")/expval_benchmark.py \"$full_qasm_file\" \"$compiler\" \"$RESULTS_FOLDER\" $LOG_EXPVAL" | ||
| command="poetry run python $(dirname "$0")/expval_benchmark.py \"$full_qasm_file\" \"$compiler\" \"$RESULTS_FOLDER\" $LOG_EXPVAL" |
There was a problem hiding this comment.
One thing to be careful about here is if this script itself is run via poetry, which is how I think the prior benchmark runs were done in the github actions (e.g. they were run via poetry run ./benchmarks/scripts .....
update Suprisingly, this seems to work!
Description
This PR refactors the expectation value benchmarking script to ensure it works with the
run_benchmarks.shscript. It also introduces new circuits to broaden the test of the expectation value testing. Scripts to visualize relative and absolute errors across different compilers over time are added, with one plot added to the README.@jordandsullivan we have the option to plot relative or absolute error, but relative is much higher for
ucc, which I don't understand the reason for.