You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the solution you'd like
As part of benchmarking, our goal is to have reasonable comparable settings across different compilers. One assumption today is that all compilers are targeting the same set of gates. In #201 we learned this was not the case. Although we are addressing in #224 , we should explicitly check any assumptions about the generated circuits during the benchmarking run to avoid us mistakenly having an unfair or at least unintended comparison.
Rather than having to write checks using each compiler's native capabilities, we would ideally convert to a common format (qasm?), and perform the checks there.
Suggested initial checks include
target gateset
qubit topology/layout
Other Considerations
It will be important to decide what to do with a benchmark run if a check fails - does the whole suite of benchmarks also fail? Or let successful ones complete and mark the run for that specific compiler + circuit as failed?
The text was updated successfully, but these errors were encountered:
Describe the solution you'd like
As part of benchmarking, our goal is to have reasonable comparable settings across different compilers. One assumption today is that all compilers are targeting the same set of gates. In #201 we learned this was not the case. Although we are addressing in #224 , we should explicitly check any assumptions about the generated circuits during the benchmarking run to avoid us mistakenly having an unfair or at least unintended comparison.
Rather than having to write checks using each compiler's native capabilities, we would ideally convert to a common format (qasm?), and perform the checks there.
Suggested initial checks include
Other Considerations
It will be important to decide what to do with a benchmark run if a check fails - does the whole suite of benchmarks also fail? Or let successful ones complete and mark the run for that specific compiler + circuit as failed?
The text was updated successfully, but these errors were encountered: