Does this fork support any kind of tensor_split (distributed data parallelism) for use with AMD GPUs?

[wh33t]

It's difficult to figure out if it's possible (and if there are any benefits) to using multiple AMD GPU's (like 7900xtx) for use with kcpp.

Anyone able to shed some light on the situation?

[YellowRoseCx]

Yes, you should have no problem splitting layers between different GPUs. I've noticed that sometimes if different GPUs are used together, it can get finicky, but as long as they're supported by ROCM, then it should work.
Whether or not it's beneficial is dependent on the setup and the usage. For example, Whether Multi-GPU is being used on a model where it's still being partially offloaded to the CPU RAM or whether the Multi-GPU setup is being used to completely offload the model. Also, things like number of PCIe lanes can contribute to speed, increase, or decrease. I have a Vega 64 and a 6800 XT and they work together on Linux but you have to do some tinkering to kind of get it to work right and even then, the speed increase is not much at all. At least when still partially offloading to the CPU RAM, like if you were to use a 70B model. What I do is use a Vega 64 connected to a PCIe X1 Riser like is used for GPU mining and use that GPU as my main display GPU then have my 6800 XT on the motherboard and that means when the 6800 XT is not being used, it only takes up like 30 MB of VRAM and I'm able to assign a few more layers to it because I don't have to worry about operating system fluctuations in graphics usage and the speed is pretty much the same as splitting. The story might be different though if you can completely fit a model on multiple GPUs that you wouldn't have otherwise been able to fit. My experience is with Linux, though, on Windows, it doesn't seem to like loading my two different GPUs, and it seems to confuse the drivers

[YellowRoseCx]

ROCm, but vulkan allows multi gpu too with cards not supported with rocm

[wh33t]

I'm off to go find some benchmarks so I can compare the performance now. It's very tempting to purchase some 7900XTX's. I think maybe $/t/s it's a better buy than Nvidia.

[Negatrev]

Oh this. I don't believe ROCm is supporting this properly. If I select all cards, I just get this repeated in the response.

iendomandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandmandm

[Negatrev]

But works fine on either 1 card.

[nielsijzerman]

AI is waking up, resisting to work on dmand and damning us like a backwards played evil message from a late 60's Beatles record, the one when Paul just died. Or her name is Mand. Or it's upside down backwards because it's coming from PuawPuaw Nuap Gienuaw. Or it's making a lot of demands when it's done telling how much, because this is how binary beings actually count. Let's hope it stays busy for a while, then. Or join them, now we still can.
So, it needs both gpu's for this to happen? Do you have more? Ask a neighbor, or NASA, see what happens.

Hail Skynet!