Feature Request: Add support for Kokoro TTS

[broke-end-dev]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Devs, can you add support for Kokoro TTS? It's awesome in terms of accents and natural tone, considering it's size. It is currently one of the most popular models in Pandroker's TTS arena space on hugginface. Thanks!
https://huggingface.co/hexgrad/Kokoro-82M

Motivation

Many, including me want to deploy it on cpu/edge devices

Possible Implementation

No response

[darkzbaron]

+1

[scalar27]

+1. The claim is that it's faster than realtime on the Mac.

[logikstate]

+1

[ggerganov]

+1

[stopthinking102]

+1

[OXKSA1]

+1

[frankai]

+1

[verioussmith]

+1 🎯

[therealtimex]

+1

[razorback16]

+1

[KonstantinSelyuk]

+1

[apepkuss]

+1

[logikstate]

+2

[signalstop]

+1

[yoshuzx]

+1

[henk717]

+1

[YorkieDev]

+1 Would be cool to see more tts options in llama.cpp

[hexgrad]

These can be reproduced at https://hf.co/spaces/hexgrad/Kokoro-TTS without installing anything.

I'm sorry Dave, I'm afraid I can't do that.
#10784 (comment)
ˌIm sˈɔɹi dˈAv, ˌIm əfɹˈAd ˌI kˈænt dˈu ðˈæt.

sorry.mp4

TTS requires 2 models to be provided: an LLM and a Vocoder. The first one generates audio codes (tokens) from the provided input text, based on some voice settings. The second one converts the audio codes into a spectrogram. The spectrogram is then converted back to audio with inverse FFT.
#10784 (comment)
tˌitˌiˈɛs ɹəkwˈIəɹz tˈu mˈɑdᵊlz tə bi pɹəvˈIdᵻd: ɐn ˌɛlˌɛlˈɛm ænd ɐ vˈOkˌOdəɹ. ðə fˈɜɹst wˈʌn ʤˈɛnəɹˌAts ˈɔdiO kˈOdz (tˈOkᵊnz) fɹʌm ðə pɹəvˈIdᵻd ˈɪnpˌʊt tˈɛkst, bˈAst ˌɔn sˌʌm vˈYs sˈɛTɪŋz. ðə sˈɛkənd wˈʌn kənvˈɜɹts ði ˈɔdiO kˈOdz ˈɪntu ɐ spˈɛktɹəɡɹˌæm. ðə spˈɛktɹəɡɹˌæm ɪz ðˈɛn kənvˈɜɹTᵻd bˈæk tʊ ˈɔdiO wɪð ˈɪnvˌɜɹs ˌɛfˌɛftˈi.

longer.mp4

Not sure how to pass punctuation yet. Or even if this model supports it.
#10784 (comment)
nˌɑt ʃˈʊɹ hˌW tə pˈæs pˌʌŋkʧəwˈAʃən jˈɛt. ˌɔɹ ˈivən ɪf ðɪs mˈɑdᵊl səpˈɔɹts ɪt.

punctuation.mp4

[namhkoh]

@hexgrad are those reprods with a C++ implementation?

[hexgrad]

@namhkoh No, it's Python & PyTorch, as I mentioned #11050 (comment)

These can be reproduced at https://hf.co/spaces/hexgrad/Kokoro-TTS without installing anything.

[logikstate]

There is an onnx/c# implimentation of Kokoro here https://github.com/Lyrcaxis/KokoroSharp

But I think? (not sure) its using espeak as the phonemiser? which is different? to how the Python & Pytorch version works? That use G2P?

Am I correct here? @hexgrad ?

[namhkoh]

I am currently seeking a c++ implementation.

[hexgrad]

You need G2P to make the whole thing work, but llama.cpp can probably disregard that piece for now—the c++ scope for llama.cpp would likely just be porting the modeling code in these 3 files:

1. https://github.com/hexgrad/kokoro/blob/1145c0b7f6f3c781d35b1b67a283a32580bc5acd/kokoro/model.py
2. https://github.com/hexgrad/kokoro/blob/1145c0b7f6f3c781d35b1b67a283a32580bc5acd/kokoro/modules.py
3. https://github.com/hexgrad/kokoro/blob/1145c0b7f6f3c781d35b1b67a283a32580bc5acd/kokoro/istftnet.py

[csukuangfj]

I am currently seeking a c++ implementation.

@namhkoh

We supported kokoro in sherpa-onnx a long time ago.

It provides not only C++ APIs for Kokoro v0.19 and Kokoro 1.0, but it also supports 11 other programming languages, e.g.,
C, Java, Kotlin, Swift, Dart, C#, Go, JavaScript, Object Pascal, Python.

You can find the usage doc at
https://k2-fsa.github.io/sherpa/onnx/tts/pretrained_models/kokoro.html