vulkan: select only one device for single gpu with multiple drivers

[Adriankhl]

An issue which an user encountered using my library: Adriankhl/godot-llm#14

Sometimes, you have multiple drivers installed for the same GPU, e.g.

• https://www.reddit.com/r/vulkan/comments/j94ct2/multiple_physical_devices_found/
• https://www.reddit.com/r/vulkan/comments/jts8mz/detect_if_physical_devices_refer_to_the_same_gpu/

And in vulkan, the GPU will be listed as separated physical devices. It causes an issue when llama.cpp attempts to split the layer.

One workaround is to set split mode to "none", which didn't work before but I fixed it here: #7552

This PR fixes the issue by checking the deviceID, and if there are two devices with the same ID, pick the one with a larger heap size.

I don't have a discrete GPU myself so this needs to be tested by someone else with a discrete GPU and multiple drivers for the same GPU.

The device selection criteria is also a bit naive, if anyone has a better suggestion please let me know.

[0cc4m]

I don't think picking by heap size will do anything, since that's a hardware feature any driver should report identical numbers there.

Currently you can handle this case manually by setting the devices to be used with GGML_VK_VISIBLE_DEVICES or by keeping only a single driver installed (which I would recommend, generally)

I know only of Linux where AMD has 3 available drivers (RADV, AMDVLK and whatever the pro Vulkan driver is called), and Nvidia has their proprietary driver and the new NVK one that is being developed. And now there's Windows (which is the one reported on your repo), with their Vulkan to DX12 thing. I don't know if that can be disabled.

I would write the logic like this: In the AMD case, you should pick RADV. In the Nvidia case, you should pick the proprietary driver. In any other case, you could pick the first available one.

What do you think?

[Adriankhl]

@0cc4m Make sense. I get a list of driver names here: https://vulkan.gpuinfo.org/displaycoreproperty.php?core=1.2&name=driverName&platform=all

And this is my current priorities, smaller numbers are prioritized

                    std::map<std::string, int> driver_priorities {};
                    driver_priorities["NVIDIA"] = 1;
                    driver_priorities["nvk"] = 2;
                    driver_priorities["radv"] = 3;
                    driver_priorities["AMD open-source driver"] = 4;
                    driver_priorities["AMD proprietary driver"] = 5;
                    driver_priorities["Intel open-source Mesa driver"] = 6;
                    driver_priorities["Intel Corporation"] = 7;

[0cc4m]

That's better, but I would prefer if we avoided the string comparison here. I propose to use the data in VkPhysicalDeviceDriverProperties. We can use the device vendor ID similar to here and afterwards select by the driver ID.

That should simplify the code.

Also, please don't write to stdout. stderr is sometimes okay, but I think it's not necessary for this feature to output anything.

[Adriankhl]

@0cc4m Ok, changed it.

I still keep the std::cerr output when GGML_VULKAN_DEBUG is enabled, I think it is good to let the user know which driver is selected.

[0cc4m]

@0cc4m Ok, changed it.

I still keep the std::cerr output when GGML_VULKAN_DEBUG is enabled, I think it is good to let the user know which driver is selected.

I think informing the user which driver is in use is helpful. But GGML_VULKAN_DEBUG is a very verbose debugging mode for developers, so that's not gonna help with that. Your debug outputs should stay like that I think, but if you want to inform the user you could add this:

diff --git a/ggml-vulkan.cpp b/ggml-vulkan.cpp
index 7a82f9f9..0d8c1cd7 100644
--- a/ggml-vulkan.cpp
+++ b/ggml-vulkan.cpp
@@ -1550,8 +1550,10 @@ static void ggml_vk_print_gpu_info(size_t idx) {
     vk::PhysicalDeviceProperties2 props2;
     vk::PhysicalDeviceMaintenance3Properties props3;
     vk::PhysicalDeviceSubgroupProperties subgroup_props;
+    vk::PhysicalDeviceDriverProperties driver_props;
     props2.pNext = &props3;
     props3.pNext = &subgroup_props;
+    subgroup_props.pNext = &driver_props;
     physical_device.getProperties2(&props2);

     const size_t subgroup_size = subgroup_props.subgroupSize;
@@ -1595,7 +1597,7 @@ static void ggml_vk_print_gpu_info(size_t idx) {
     fp16 = fp16 && vk12_features.shaderFloat16;

     std::string device_name = props2.properties.deviceName.data();
-    std::cerr << GGML_VK_NAME << idx << ": " << device_name << " | uma: " << uma << " | fp16: " << fp16 << " | warp size: " << subgroup_size << std::endl;
+    std::cerr << GGML_VK_NAME << idx << ": " << device_name << " (" << driver_props.driverName << ") | uma: " << uma << " | fp16: " << fp16 << " | warp size: " << subgroup_size << std::endl;

     if (props2.properties.deviceType == vk::PhysicalDeviceType::eCpu) {
         std::cerr << "ggml_vulkan: Warning: Device type is CPU. This is probably not the device you want." << std::endl;

That way the driver name is added to the device info.

I was able to test the code using AMD's drivers RADV and AMDVLK on Linux and it works fine. Thank you for your contribution.

[Adriankhl]

Now also added the driver information to ggml_vk_print_gpu_info, ready to merge

[Adriankhl]

Wait, NVK doesn't exist in old vulkan sdk, let me see how to fix the CI

[0cc4m]

I can't track your changes if you force push every time. It's okay to have multiple commits, they'll get squashed on merge. Are you still working on it?

[Adriankhl]

@0cc4m Sorry, I didn't know it will get squashed. Now the problem is fixed by checking the VK_HEADER_VERSION macro, also I found vk::DriverId enum is there for cpp binding, here is the diff

 static void ggml_vk_print_gpu_info(size_t idx) {
@@ -1550,8 +1567,10 @@ static void ggml_vk_print_gpu_info(size_t idx) {
     vk::PhysicalDeviceProperties2 props2;
     vk::PhysicalDeviceMaintenance3Properties props3;
     vk::PhysicalDeviceSubgroupProperties subgroup_props;
+    vk::PhysicalDeviceDriverProperties driver_props;
     props2.pNext = &props3;
     props3.pNext = &subgroup_props;
+    subgroup_props.pNext = &driver_props;
     physical_device.getProperties2(&props2);

     const size_t subgroup_size = subgroup_props.subgroupSize;
@@ -1595,7 +1614,7 @@ static void ggml_vk_print_gpu_info(size_t idx) {
     fp16 = fp16 && vk12_features.shaderFloat16;

     std::string device_name = props2.properties.deviceName.data();
-    std::cerr << GGML_VK_NAME << idx << ": " << device_name << " | uma: " << uma << " | fp16: " << fp16 << " | warp size: " << subgroup_size << std::endl;
+    std::cerr << GGML_VK_NAME << idx << ": " << device_name << " (" << driver_props.driverName << ") | uma: " << uma << " | fp16: " << fp16 << " | warp size: " << subgroup_size << std::endl;

@@ -1724,17 +1743,19 @@ void ggml_vk_instance_init() {
                     // Smaller number -> higher priority
                     switch (old_prop.properties.vendorID) {
                         case VK_VENDOR_ID_AMD:
-                            driver_priorities[static_cast<vk::DriverId>(VkDriverId::VK_DRIVER_ID_MESA_RADV)] = 1;
-                            driver_priorities[static_cast<vk::DriverId>(VkDriverId::VK_DRIVER_ID_AMD_OPEN_SOURCE)] = 2;
-                            driver_priorities[static_cast<vk::DriverId>(VkDriverId::VK_DRIVER_ID_AMD_PROPRIETARY)] = 3;
+                            driver_priorities[vk::DriverId::eMesaRadv] = 1;
+                            driver_priorities[vk::DriverId::eAmdOpenSource] = 2;
+                            driver_priorities[vk::DriverId::eAmdProprietary] = 3;
                             break;
                         case VK_VENDOR_ID_INTEL:
-                            driver_priorities[static_cast<vk::DriverId>(VkDriverId::VK_DRIVER_ID_INTEL_OPEN_SOURCE_MESA)] = 1;
-                            driver_priorities[static_cast<vk::DriverId>(VkDriverId::VK_DRIVER_ID_INTEL_PROPRIETARY_WINDOWS)] = 2;
+                            driver_priorities[vk::DriverId::eIntelOpenSourceMESA] = 1;
+                            driver_priorities[vk::DriverId::eIntelProprietaryWindows] = 2;
                             break;
                         case VK_VENDOR_ID_NVIDIA:
-                            driver_priorities[static_cast<vk::DriverId>(VkDriverId::VK_DRIVER_ID_NVIDIA_PROPRIETARY)] = 1;
-                            driver_priorities[static_cast<vk::DriverId>(VkDriverId::VK_DRIVER_ID_MESA_NVK)] = 2;
+                            driver_priorities[vk::DriverId::eNvidiaProprietary] = 1;
+#if defined(VK_API_VERSION_1_3) && VK_HEADER_VERSION >= 235
+                            driver_priorities[vk::DriverId::eMesaNvk] = 2;
+#endif
                             break;
                     }

[richardanaya]

I believe this PR broke multiple graphics cards for users of vulkan. I have two 7900xtx (exact same card), and now it will not detect two cards. Bug: #7997