123 save compiler versions with data (#134)

* Fix date label in load_data notebook

* Create script to generate new benchmarks plot

* Delete load_benchmark notebook, no longer needed.

* Add generate new plot to benchmarks action

* Remove possibly duplication execution of shell script

* Complete merge using new naming convention for results

* Correct filepath

* Add version info for compilers in header of save_results

* Temporary importable versioning for UCC

* Updated example results file

* Remove seaborn requirement

* Get abolute filepath

* Commit all contents of benchmark including .png

* Increase parallelism

* Point to 8 core Runner

* Fix filepath concatonation

* Checkout correct branch of the repo

* Update benchmark results

* Remove unnecessary print statement

* Push results to same branch

* Update to 16 core and 16 parallelism CL arg

* Update benchmark results

* Reducing paralellism to 8 since 16 did not significantly speed up.

* Try to pull the correct branch

* Add small test shell script

* Test on smaller benchmarks for debugging

* Move test script to scripts directory

* Update benchmark results

* Update benchmark results

* Return to full benchmarks

* Update benchmark results

* Add print statements for debugging

* Return to small test

* Update benchmark results

* Test adding .png explicitly

* Debugging why .png is not being committed

* Add manual commit for .png

* Try force adding .png and removing existing cached versions

* Try git add --all

* Update benchmark results

* Add debugging for git ignore

* Remove gitmodules check

* Fix yml issues from merge

* Update benchmark results

* Touch new png to acknowleddge as new plot.

* Update benchmark results

* Rotate xlabels

* Update benchmark results

* Try mounting entire GH workspace in the docker volume

* Remove accidental comment within docker command

* Fix syntax error

* Simplifying to use built in GH Actions API

* Remove syntax error comment

* Refactor GitHub Actions workflow to improve benchmark result handling and ensure proper branch context for commits

* Syntax error

* Add git config to Action

* Update benchmark results

* Add individual plot scripts for circuits and split into latest and avg over time

* Add new plotting scripts to GH Action workflow

* Delete inaccurate data and add new plots

* Update benchmark results

* Delete test data

* Return to full benchmark shell script.

* Update benchmark results

* Complete merge

* Store the hour in the filename to differentiate runs on the same day but keep the number of files lower (when running benchmarks in parallel)

* Generalize small_test shell script

---------

Co-authored-by: GitHub Actions <actions@github.com>

.github/workflows/ucc-benchmarks.yml

@@ -14,14 +14,7 @@ jobs:
       - name: Checkout code
         uses: actions/checkout@v4
         with:
-          fetch-depth: 0 # Ensure the full history is fetched for branch operations
-
-      # Checkout the correct branch
-      - name: Checkout the branch
-        run: |
-          BRANCH_NAME="${{ github.head_ref || github.ref_name }}"
-          echo "Checking out branch: $BRANCH_NAME"
-          git checkout $BRANCH_NAME
+          fetch-depth: 0 # Ensure the full history is fetched
 
       # Build the Docker image
       - name: Build Docker image
@@ -31,16 +24,26 @@ jobs:
       - name: Run benchmarks
         run: |
           docker run --rm \
-            -v ${{ github.workspace }}/benchmarks/results:/ucc/benchmarks/results \
-            ucc-benchmark bash -c "source /venv/bin/activate && ./benchmarks/scripts/run_benchmarks.sh 8"
+            -v "/home/runner/work/ucc/ucc:/ucc" \
+            ucc-benchmark bash -c "
+              source /venv/bin/activate && \
+              ./benchmarks/scripts/run_benchmarks.sh 8 && \
+              python ./benchmarks/scripts/plot_avg_benchmarks_over_time.py && \
+              python ./benchmarks/scripts/plot_latest_benchmarks.py
+            "
+
+      # Commit and push benchmark results
+      - name: Configure Git for commit
+        run: |
+          git config --global user.email "actions@github.com"
+          git config --global user.name "GitHub Actions"
 
-      
-      # Commit and push benchmark results using a dedicated action
       - name: Commit and push results
-        uses: EndBug/add-and-commit@v9
-        with:
-          author_name: GitHub Actions
-          author_email: actions@github.com
-          message: "Update benchmark results"
-          add: "."
-          push: true
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          git switch ${{ github.head_ref || github.ref_name }}
+          git add benchmarks/*
+          git status
+          git commit -m "Update benchmark results" || echo "No changes to commit"
+          git push origin HEAD:${{ github.head_ref || github.ref_name }}

Dockerfile

@@ -16,8 +16,3 @@ RUN python3 -m venv /venv
 RUN /venv/bin/pip install --no-cache-dir -r /ucc/requirements.txt
 RUN /venv/bin/pip install -e . && /venv/bin/pip show ucc  
 
-# Make sure the shell script is executable
-RUN chmod +x /ucc/benchmarks/scripts/run_benchmarks.sh
-
-# Set the default command to run your benchmark script with the virtual environment activated
-CMD ["/bin/bash", "-c", "source /venv/bin/activate && cd /ucc/benchmarks/scripts && ./run_benchmarks.sh"]

benchmarks/results/gates_2024-12-10.csv

@@ -1,3 +1,4 @@
+# Compiler versions: qiskit=1.2.0, cirq=1.4.1, pytket=1.35.0, ucc=0.1.1
 compiler,circuit_name,raw_multiq_gates,compile_time,compiled_multiq_gates
 pytket,qaoa_barabasi_albert,1176,0.02424311637878418,1176
 ucc,qaoa_barabasi_albert,1176,0.7855277061462402,1176

benchmarks/results/gates_2024-12-16.csv

@@ -1,49 +1,25 @@
 compiler,circuit_name,raw_multiq_gates,compile_time,compiled_multiq_gates
-qiskit,qaoa_barabasi_albert,1176,0.172929048538208,1176
-ucc,qaoa_barabasi_albert,1176,0.6496548652648926,1176
-pytket,qaoa_barabasi_albert,1176,0.026659250259399414,1176
-cirq,qaoa_barabasi_albert,1176,7.629104852676392,1176
-qiskit,qft,10050,0.5776305198669434,3244
-ucc,qft,10050,2.9913699626922607,2740
-pytket,qft,10050,0.3402554988861084,5890
-ucc,square_heisenberg,2160,0.33558034896850586,540
-qiskit,square_heisenberg,2160,0.3129239082336426,540
-pytket,square_heisenberg,2160,0.1243584156036377,2160
-cirq,square_heisenberg,2160,9.479390382766724,540
-qiskit,prep_select,9744,1.4973485469818115,9708
-ucc,prep_select,9744,7.07340669631958,9702
-ucc,qcnn,388,0.17171573638916016,388
-qiskit,qcnn,388,0.10727858543395996,388
-cirq,qft,10050,50.487868309020996,4648
-pytket,qcnn,388,0.024190902709960938,388
-pytket,prep_select,9744,0.3228616714477539,9712
-cirq,qcnn,388,2.4818356037139893,388
-cirq,prep_select,9744,74.03872561454773,9712
-pytket,qv,15000,4.559576034545898,15000
-ucc,qv,15000,4.738056898117065,14856
-qiskit,qv,15000,5.317723989486694,14856
-cirq,qv,15000,186.38663792610168,14856
-ucc,qaoa_barabasi_albert,1176,0.6466026306152344,1176
-pytket,qaoa_barabasi_albert,1176,0.02384018898010254,1176
-qiskit,qaoa_barabasi_albert,1176,0.17044854164123535,1176
-cirq,qaoa_barabasi_albert,1176,7.612349510192871,1176
-qiskit,qft,10050,0.5786969661712646,3244
-ucc,qft,10050,2.9486944675445557,2740
-pytket,qft,10050,0.2715871334075928,5890
-ucc,square_heisenberg,2160,0.3280293941497803,540
-qiskit,square_heisenberg,2160,0.35325193405151367,540
-pytket,square_heisenberg,2160,0.19542360305786133,2160
-cirq,square_heisenberg,2160,10.833353996276855,540
-qiskit,prep_select,9744,1.5344769954681396,9708
-ucc,prep_select,9744,6.940417051315308,9702
-ucc,qcnn,388,0.1656789779663086,388
-qiskit,qcnn,388,0.11114120483398438,388
-pytket,qcnn,388,0.024122238159179688,388
-cirq,qft,10050,51.793689012527466,4648
-pytket,prep_select,9744,0.25701141357421875,9712
-cirq,qcnn,388,3.374324083328247,388
-ucc,qv,15000,4.737441778182983,14856
-cirq,prep_select,9744,89.24731159210205,9712
-qiskit,qv,15000,5.160390138626099,14856
-pytket,qv,15000,4.397194862365723,15000
-cirq,qv,15000,185.35534381866455,14856
+qiskit,qaoa_barabasi_albert,1176,0.167680025100708,1176
+pytket,qaoa_barabasi_albert,1176,0.02307295799255371,1176
+ucc,qaoa_barabasi_albert,1176,0.704157829284668,1176
+cirq,qaoa_barabasi_albert,1176,7.5352256298065186,1176
+qiskit,qft,10050,0.5787725448608398,3244
+ucc,qft,10050,2.949899911880493,2740
+pytket,qft,10050,0.2687070369720459,5890
+ucc,square_heisenberg,2160,0.3382143974304199,540
+qiskit,square_heisenberg,2160,0.32480406761169434,540
+pytket,square_heisenberg,2160,0.12444162368774414,2160
+cirq,square_heisenberg,2160,9.622225999832153,540
+qiskit,prep_select,9744,1.5326259136199951,9708
+ucc,prep_select,9744,6.6899003982543945,9702
+ucc,qcnn,388,0.15262794494628906,388
+qiskit,qcnn,388,0.10137224197387695,388
+cirq,qft,10050,50.238999128341675,4648
+pytket,qcnn,388,0.024954795837402344,388
+pytket,prep_select,9744,0.30251026153564453,9712
+cirq,qcnn,388,3.7540109157562256,388
+cirq,prep_select,9744,73.64335703849792,9712
+qiskit,qv,15000,5.770041465759277,14856
+ucc,qv,15000,5.550801992416382,14856
+pytket,qv,15000,4.363432884216309,15000
+cirq,qv,15000,189.20356154441833,14856

benchmarks/scripts/common.py

@@ -13,6 +13,11 @@
 from pytket.predicates import CompilationUnit
 from qiskit import transpile as qiskit_transpile
 from qbraid.transpiler import transpile as translate
+from qiskit import __version__ as qiskit_version
+from cirq import __version__ as cirq_version
+from pytket import __version__ as pytket_version
+from ucc import __version__ as ucc_version
+
 import sys  # Add sys to accept command line arguments
 import os
 from ucc import compile as ucc_compile
@@ -124,36 +129,68 @@ def count_multi_qubit_gates(circuit, compiler_alias):
         case _:
             return "Unknown compiler alias."
 
+def get_header(df):
+    # Get version information for the compilers
+    compiler_versions = {
+        "qiskit": qiskit_version,
+        "cirq": cirq_version,
+        "pytket": pytket_version,
+        "ucc": ucc_version,
+    }
+
+    # Create version header as a string formatted for CSV
+    version_header = "# Compiler versions: " + ", ".join(
+        f"{key}={value}" for key, value in compiler_versions.items()
+    )
 
+    return version_header
 
 def save_results(results_log, benchmark_name="gates", folder="../results", append=False):
-    """Save the results of the benchmarking to a CSV file.
+    """Save the results of the benchmarking to a CSV file with compiler versions as a header.
+    
     Parameters:
         results_log: Benchmark results. Type can be any accepted by pd.DataFrame.
         benchmark_name: Name of the benchmark to be stored as prefix to the filename. Default is "gates".
         folder: Folder where the results will be stored. Default is "../results".
         append: Whether to append to an existing file created on the same date (if True) or overwrite (if False). Default is False.
     """
     df = pd.DataFrame(results_log)
-    current_date = datetime.now().strftime("%Y-%m-%d")
+    # This will store results run during the same date and hour in the same file
+    current_date = datetime.now().strftime("%Y-%m-%d_%H")
 
     # Ensure the folder exists
     os.makedirs(folder, exist_ok=True)
 
     # Create the filename based on the current date
     file_name = f"{benchmark_name}_{current_date}.csv"
     file_path = os.path.join(folder, file_name)
-    # Check if the file exists and append if needed
-    if append:
-        # If the file exists and the date matches, append data
+
+    header = get_header(df)
+
+    def prepend_header_if_missing(file_path, header):
+        """Check if the file has a header, and prepend it if not."""
         if os.path.exists(file_path):
-            df.to_csv(file_path, mode='a', header=False, index=False)
+            with open(file_path, "r+") as f:
+                content = f.read()
+                if not content.startswith("# Compiler versions:"):
+                    # Prepend the header and write back
+                    f.seek(0)
+                    f.write(f"{header}\n{content}")
         else:
-            # If the file doesn't exist, create a new one with headers
-            df.to_csv(file_path, mode='w', header=True, index=False)
+            # Create a new file with the header
+            with open(file_path, "w") as f:
+                f.write(f"{header}\n")
+
+    # Ensure the header is present in the file
+    prepend_header_if_missing(file_path, header)
+
+    # Write results to the file
+    if append:
+        # Append results without adding column headers again
+        df.to_csv(file_path, mode="a", header=False, index=False)
     else:
-        # If append is False, overwrite the file (or create a new one if it doesn't exist)
-        df.to_csv(file_path, mode='w', header=True, index=False)
+        # Overwrite or create the file
+        df.to_csv(file_path, mode="w", header=True, index=False)
 
     print(f"Results saved to {file_path}")