Wrong estimation of how many gpu layers fit in VRAM on RX 7900 XT?

[MacharaStormwing]

I use the current version 1.79.1.yr0 for windows.
When using it with my Radeon RX 7900 XT which has 20gb of VRAM it seems the estimation of layers that fit in the VRAM seem wrong.
My partner who uses the same setting and same file with a 7900 XTX with 24gb of VRAM the estimation seems more adequate.
For instance if I choose L3-8B-Stheno-v3.2-Q6_K-imat.gguf with 16k context (at 24k context it tells me not all layers fit) just under 13gb of 20 VRAM is used, so quite a bit is left free.
For him 64k context size is allowed and still all layers are selected per default. After loading it like this almost exactly 20GB of his VRAM is used up.

Maybe the program thinks this card has just 16gb of VRAM?
How does the estimation work?
How much for the model and how much per K of context?

I realize a 24gb card can fit more than a 20gb card, but the estimation seems off.
I could load the model Mistral-Small-22B-ArliAI-RPMax-v1.1-Q4_K_M.gguf with 16k context fully into VRAM and start a conversation where the initial 5782 tokens properly and quickly.

Processing Prompt [BLAS] (5782 / 5782 tokens)
Generating (273 / 273 tokens)
[17:20:07] CtxLimit:6055/16384, Amt:273/273, Init:0.01s, Process:37.28s (6.4ms/T = 155.10T/s), Generate:14.29s (52.4ms/T = 19.10T/s), Total:51.57s (5.29T/s)

This used up 18940mb of VRAM (as displayed in GPU_Z).
Maybe there are issues when more context is used, but still the estimation seems off.

Can you check?
Maybe there is an improvement necessary (the authors may not have tested this tool with this card)?

[YellowRoseCx]

I'll look into it, but my guess is that going from 16k context to 64k context would bring it too close to the 20gb max your GPU has and because the estimate layer feature has a built in safety-net to allow for other programs running on your pc (like the graphics themselves) that it sees 64k as too much.
not sure why 24k context would be listed as too much though if 16k loads up as 13gb

[MacharaStormwing]

What I wanted to describe is that with 16384 context size koboldcpp-rocm allows me to fit all layers into VRAM. Already 24k makes it offload some onto the CPU with my 20gb of VRAM.
With 16384 context size it says: GPU Layers: -1 (Auto: 35/35 Layers).
With 24576 context size it says: GPU Layers: -1 (Auto: 30/35 Layers).

With the XTX and 24gb of VRAM my partner can move the slider to the right till 65536 and still fit all layers.
For me when I select 65536 context says it says (Auto: 10/35 Layers).

[MacharaStormwing]

Update:
I upgraded my System (replaced an Intel i5 12600k and 32gb ram with a Ryzen 7 9800x3d and 64gb of ram and reinstalled windows from scratch.
The GPU and most of the rest of the system stayed the same (though I got a bigger SSD and installed Windows there).

This time it seems the detection works correctly.

I did notice however that I had to disabled the iGPU in the CPU for it to work, otherwise I would get this error:

rocBLAS error: Could not initialize Tensile host: No devices found

even if I selected the right GPU in the GUI.
Any Idea why and how to still have the iGPU available?

[YellowRoseCx]

Merry Christmas and congratulations on the sick new setup! As far as I know, there's not a workaround for the iGPU
needing to be disabled on Windows(there is on Linux), as that instruction comes directly from AMD themselves and I don't have an AMD CPU to test with

I will message some ROCM developers and see if a way to keep iGPU enabled and use your main GPU has been ported over to Windows yet and see what they say.

Happy Holidays!

[MacharaStormwing]

Thank you and I wish you some nice Christmas Holidays too - hopefully you are able to take some days off...

Good to know that disabling iGPU fix comes from AMD.
I plan to run Linux anyway in the end, hopefully Linux as a base system and Windows in a Virtual Machine with a GPU passed through when needed.
What do I need to do in Linux when I have the iGPU enabled to make it work?

Thanks to the 'sick' system. Runs quite nicely for now. Should support two GPUs in the future...