llama-cpp: add support for Vulkan backend

[mschwaig]

Description of changes

This adds support for the newly added Vulkan backend for llama-cpp, implemented in the same way as ggerganov/llama.cpp#5173.

The one thing that is tricky about this is the vulkanSupport flag that I added to the function arguments alongside the existing arguments cudaSupport, rocmSupoort, openclSupport, blasSupport and metalSupport, which raises two questions for me.

Does using vulkanSupport directly work for all users?

I am not sure if we would expect people to set cudaSupport or rocmSupport at the same time as vulkanSupport and if the result would/should be sane if it's passed to llama-cpp like that or if the package should do some prioritization and compile with the faster vendor-specific API only when it is set alongside of Vulkan. The upstream flake has separate outputs for each backend.

Will using vulkanSupport directly still make sense if even more backends are added?

A number of other backends have been added to the upstream at the same time with the one from this PR (see ggerganov/llama.cpp#5138).
There is a second Vulkan backend that goes through Kompute and the authors of those two Vulkan backends want to merge them eventually. There is also a SYCL backend for Intel GPUs.
If we would add support for those backends as well I am not sure how users should access those in a way that is idiomatic for nixpkgs:

• Kompute also targets Vulkan,
• one of the Vulkan backends will go away again if the two are actually merged, and
• would there be a useSYCLflag?

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[philiptaron]

I'd wait to land this until ggerganov/llama.cpp#5311 lands, so that there aren't the awkward darwin checks.

More broadly speaking, Vulkan support is aiming to be the new "default" llama.cpp backend, superseding the opencl backend. (At least, that's what I understand from @0cc4m's PRs and comments.) If that comes to be, I expect the OpenCL backend to diminish and get relegated to a non-default state, and the Vulkan backend be promoted to default.

That moment hasn't arrived, but I'm looking forward to when it does. llama.cpp's defaults should give users the fastest local AI possible, and the results from Vulkan are better than those from OpenCL.

[mschwaig]

I'd wait to land this until ggerganov/llama.cpp#5311 lands, so that there aren't the awkward darwin checks.

I don´t mind either way.

The one thing that's not properly marked as broken right now are Darwin builds on ARM with .override = { useVulkan = true; } .

Darwin on Intel is marked as broken in any case, but someone could also check if that actually starts working after Mac support is merged upstream and we change the package to make vulkanSupport=true the default in that case already.

I will not do either of those things myself since I don´t have any Macs at home.

[happysalada]

Im integrating upstream flake changes in #289513
Would you mind if we went for my PR ?

[mschwaig]

Im integrating upstream flake changes in #289513 Would you mind if we went for my PR ?

In general I don't mind, but I have some concerns about the current state of your PR which I will mention in a comment there.

[philiptaron]

I agree that pursuing #289513 is the better path.

[mschwaig]

Ok, I will close this then.