CUDA support

[Macoron]

Whisper.cpp now supports CUDA. While it's really annoying to setup CUDA and its drivers, it should give a nice performance boost for Windows and Linux PCs with Nvidia cards.

I want to make it optional. Probably best way to do that is to compile one .dll with CUDA support and one without. I would need to figure out how to switch between them in runtime.

Also it will probably be smart to add fallback to CPU when no CUDA available.

[atx-barnes]

This would be awesome. Any updates on this enhancement? What would be the use case for switching during runtime.

[Macoron]

Any updates on this enhancement?

I don't have PC with CUDA supported GPU. But in theory, you would need to just recompile dlls with this flag and replace it in plugin folder.

What would be the use case for switching during runtime.

I want to make CUDA optional, because this is a very niche use case. Most people will use CPU inference. If someone without installed CUDA will try to run CUDA compiled dlls - it won't fallback to CPU inference and may crash (I guess?).

Therefore I need to figure out how to ship plugin with CPU inference and optional CUDA inference. This is what I mean by switch in runtime, because Unity usually load all dlls at start.

[atx-barnes]

Got it that makes sense. Yeah I might try and recompile the dlls with the -j flag and see what happens in Unity.

[Macoron]

Done, though no fallback to CPU yet. Hope whisper.cpp gang will add it in future.

[ehossai2]

I have pytorch installed, do I need to install CUDA separately?

On the other note, I am little bit struggling to enable CUDA. I read your guidance, unfortunately, the instructions are not clear to me. Can you enlighten me more?

[Macoron]

I have pytorch installed, do I need to install CUDA separately?

You would need to install CUDA Toolkit 12.2.0 (download it here). Select your platform and follow instructions. Pytorch isn't relevant.

On the other note, I am little bit struggling to enable CUDA. I read your guidance, unfortunately, the instructions are not clear to me. Can you enlighten me more?

It should be very straightforward. Open your Unity project and find Project Settings window. Next select Whisper category and click enable CUDA.