CUDA: fix partial offloading for ne0 % 256 != 0

[JohannesGaessler]

Fixes #8558 .

The MMVQ and MMQ kernels handle out-of-bounds accesses for ne00/ne10 by padding the last row of src0 and every column of src1 to a multiple of MATRIX_ROW_PADDING. The padding is memset to 0 so that it has no influence on the result. Crucially both paddings have to be memset to 0 because if one of the values were to randomly encode NaN or infinity then the result of a floating point multiplication with 0 is NaN.

MMQ currently works on 256 ne00/ne10 values per iteration. GLM 4 has tensors with ne00 % 256 == 128. So for this model the content of the padding is relevant. On master, when using partial offloading the padding is not being zeroed when the data is copied from RAM to VRAM. Therefore, depending on the input parameters, the result can become NaN. This PR adds the missing calls.

There is also another, different issue with --split-mode row but so far I have not been able to track it down.

[slaren]

I am not sure that this is completely correct. The padding is already cleared in init_tensor. For a tensor allocated in the compute buffer, this should happen before the graph computation, but the problem is that the padding may be overwritten by the previous operations before the tensor is actually used. Clearing the padding in the set_tensor function will not work on every case, because the tensor may have been initialized in a different way (eg. from a ggml_cpy operation). I think this would work more reliably:

• Add GGML_BACKEND_BUFFER_USAGE_COMPUTE to ggml_backend_buffer_usage
• Call ggml_backend_buffer_set_usage to this type in ggml-alloc so that compute buffers can be reliably identified
• Before computing the mul_mat operation in the CUDA backend, check if src0->buffer->usage == GGML_BACKEND_BUFFER_USAGE_COMPUTE, and if so, clear the padding there.

[JohannesGaessler]

Thanks for the pointers. I wasn't sure about the correct way to edit ggml-alloc.c, I'm setting the property immediately after allocation. Also I am zeroing the memory in ggml_cuda_op_mul_mat instead because for ne2*ne3 > 1 only the last ne0 x ne1 matrix should be padded.

[slaren]

ggml_backend_cuda_buffer_init_tensor could also be updated to skip the cudaMemset if the tensor is on a compute buffer, it might save a bit of time.

[slaren]

ggml_backend_cuda_buffer_init_tensor could also be updated to skip the cudaMemset if the tensor is on a compute buffer, it might save a bit of time.

I see that you removed the cudaMemset entirely, but I think this still needs to be done for weights & KV, it can only be skipped if the tensor is on a compute buffer.

[JohannesGaessler]

Thanks for all the help.

[forworldm]

got illegal argument with float32, i think ggml_is_quantized(src0->type) should be added