scripts : fix missing key in compare-llama-bench.py

[ggerganov]

cont #10319

I get the following error on master:

Traceback (most recent call last):
  File "./scripts/compare-llama-bench.py", line 307, in <module>
    gpu_blas = bool(rows_full[0][KEY_PROPERTIES.index("gpu_blas")])
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: 'gpu_blas' is not in list

Copilot reviewed 1 out of 1 changed files in this pull request and generated no suggestions.

[JohannesGaessler]

This happens when you don't explicitly tell the script which columns to show, which is a case I didn't check.

[ggerganov]

Strangely, the compare-llama-bench.py script stopped working on my Mac Studio with this error:

$ ▶ ./scripts/compare-commits.sh master gg/metal-enable-mv -m models/llama-3.2-1b-instruct/ggml-model-q4_0.gguf -p 1,2,3,4,5,6,7,8
+ ./scripts/compare-llama-bench.py --check
+ bench_args='-m models/llama-3.2-1b-instruct/ggml-model-q4_0.gguf -p 1,2,3,4,5,6,7,8'
+ rm -f llama-bench.sqlite
+ git checkout master
Switched to branch 'master'
+ make clean
++ nproc
+ make -j24 llama-bench
+ ./llama-bench -o sql -oe md -m models/llama-3.2-1b-instruct/ggml-model-q4_0.gguf -p 1,2,3,4,5,6,7,8
+ sqlite3 llama-bench.sqlite
| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp1 |        309.22 ± 5.13 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp2 |        140.87 ± 0.52 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp3 |        209.10 ± 0.48 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp4 |        280.78 ± 1.56 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp5 |        348.03 ± 1.30 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp6 |        413.57 ± 1.32 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp7 |        482.86 ± 1.97 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp8 |        550.37 ± 1.47 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |         tg128 |        310.18 ± 0.36 |

build: f6d12e7df (4162)
+ git checkout gg/metal-enable-mv
Switched to branch 'gg/metal-enable-mv'
+ make clean
++ nproc
+ make -j24 llama-bench
+ ./llama-bench -o sql -oe md -m models/llama-3.2-1b-instruct/ggml-model-q4_0.gguf -p 1,2,3,4,5,6,7,8
+ sqlite3 llama-bench.sqlite
| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp1 |        310.12 ± 2.93 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp2 |       529.01 ± 12.88 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp3 |       678.46 ± 13.34 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp4 |        826.56 ± 6.58 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp5 |        347.55 ± 0.92 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp6 |        415.98 ± 1.37 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp7 |        485.46 ± 0.59 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |           pp8 |        553.17 ± 1.59 |
| llama 1B Q4_0                  | 727.75 MiB |     1.24 B | Metal,BLAS |      16 |         tg128 |        310.30 ± 1.27 |

build: 7670486c8 (4163)
+ ./scripts/compare-llama-bench.py -b master -c gg/metal-enable-mv
Traceback (most recent call last):
  File "/Users/ggerganov/development/github/llama.cpp/./scripts/compare-llama-bench.py", line 307, in <module>
    ngl = int(rows_full[0][KEY_PROPERTIES.index("n_gpu_layers")])
              ~~~~~~~~~^^^
IndexError: list index out of range

The weird thing is the same command runs successfully on my MacBook. Can't figure out what is the cause. @JohannesGaessler do you have any ideas?

[JohannesGaessler]

I think this is caused by the program for some reason not finding the correct commits to compare or not finding any data for the commits in question. rows_full ends up being an empty list so you get an error. It could have to do with implicit state through the contents of llama-bench.sqlite or different master branches.

Someone messaged me asking for useful things to do for the project and I told them (among other things) that better handling of failure cases in compare-llama-bench.py would be useful. I haven't heard anything back so I guess I may need to do it myself after all.