whisper : try to fix the parallel whisper_state functionality

[ggerganov]

ref #1472

./main -m ./models/ggml-base.en.bin -f ./samples/gb0.wav -p 2

• CPU
• Metal
• CUDA

[slaren]

I will change ggml-backend to decouple the buffers from backend instances, but that is still going to require changing the metal backend such that the list of buffers is available to all the instances of the metal backend. In the long term, I think it would be better to get rid of the list of buffers entirely by making the ggml_backend_buffer of metal more an abstraction of the metal buffer object (I think it is MTLBuffer). That may also help fix issues that people with intel macs with discrete GPUs have with the metal backend, since it wouldn't assume unified memory.

[ggerganov]

In the long term, I think it would be better to get rid of the list of buffers entirely by making the ggml_backend_buffer of metal more an abstraction of the metal buffer object (I think it is MTLBuffer).

Yes, will probably reimplement this when I start working on ggerganov/llama.cpp#3229. Planning to do various improvements in the Metal backend in addition to that.

For now, I've implemented a workaround where in ggml_metal_get_buffer we check if the tensor belongs to a backend buffer and if it does, we search the list of buffers for that backend. Otherwise, we fallback to the old logic. This seems to resolve the issue for now.