Experimental speech streaming for LMNT (useChat/useCompletion React)

[lgrammel]

Summary

Adds speech streaming to useChat and useCompletion with streamData.

• useCompletion & useChat (for React) provide a experimental_speechUrl that can be used html audio elements
• Integration functions for lmnt speech streams through experimental_forwardLmntSpeechStream
• streamData.experimental_appendSpeech: add speech stream chunks to data stream (used automatically through forward functions)
• Example: examples/next-lmnt: LMNT completion & chat speech streaming
• Docs: LMNT provider docs, API docs for experimental_forwardLmntSpeechStream

Notes

• The LMNT SDK does not work in the edge environment (as of v1.1.2)

[untilhamza]

This is exciting

[llermaly]

This is awesome!

[llermaly]

@lgrammel Hi Lars, I tried to test this one locally with no luck, it is showing this error:

 ⚠ ./app/api/chat-speech-elevenlabs/route.ts
Attempted import error: 'forwardModelFusionSpeechStream' is not exported from 'ai' (imported as 'forwardModelFusionSpeechStream').

I go to node_modules/ai and I see the function there, not sure if I need to do anything else. (I cloned the fork, checkout to the branch and run the example)

It is ready to test?

Thanks!

[lgrammel]

@lgrammel Hi Lars, I tried to test this one locally with no luck, it is showing this error:

 ⚠ ./app/api/chat-speech-elevenlabs/route.ts
Attempted import error: 'forwardModelFusionSpeechStream' is not exported from 'ai' (imported as 'forwardModelFusionSpeechStream').

I go to node_modules/ai and I see the function there, not sure if I need to do anything else. (I cloned the fork, checkout to the branch and run the example)

It is ready to test?

Thanks!

Have you rebuilt the ai package? The easiest way is to just rebuild the whole repository (pnpm i, pnpm build) and then try out the example.

[llermaly]

@lgrammel Hi Lars, I tried to test this one locally with no luck, it is showing this error:

 ⚠ ./app/api/chat-speech-elevenlabs/route.ts
Attempted import error: 'forwardModelFusionSpeechStream' is not exported from 'ai' (imported as 'forwardModelFusionSpeechStream').

I go to node_modules/ai and I see the function there, not sure if I need to do anything else. (I cloned the fork, checkout to the branch and run the example)
It is ready to test?
Thanks!

Have you rebuilt the ai package? The easiest way is to just rebuild the whole repository (pnpm i, pnpm build) and then try out the example.

That did the trick thank you!. I was doing npm run dev , I did pnpm build , npm start and it worked.

It works really, really fast. I hope we can get this merged very soon.

[llermaly]

Hi @MaxLeiter! did you have a chance to take a look?

[tgonzales]

Hello @lgrammel I saw that you changed from eleven labs to LMNT, there is a technical reason for this, eleven labs supports multi languages, LMNT still has no plans to launch this, wouldn't it be interesting to keep both options?

Thank you and congratulations for the excellent work

[lgrammel]

Hello @lgrammel I saw that you changed from eleven labs to LMNT, there is a technical reason for this, eleven labs supports multi languages, LMNT still has no plans to launch this, wouldn't it be interesting to keep both options?

Thank you and congratulations for the excellent work

Thanks. We want to use the official elevenlabs node SDK, but it does not support duplex streaming yet: elevenlabs/elevenlabs-js#4

In the meantime, you could use modelfusion elevenlabs with the adapter that I had in an earlier version of this PR.

[Iven2132]

@lgrammel Hi! I can't find the example app for speech streaming in the Vercel AI SDK repo. where it's gone?

[lgrammel]

@lgrammel Hi! I can't find the example app for speech streaming in the Vercel AI SDK repo. where it's gone?

this feature has not been merged yet

[Iven2132]

Hi @MaxLeiter Can you merge this?

[Iven2132]

Hi @MaxLeiter Can you please approve this?

[llermaly]

bump

[pixelcatgg]

we could really use this as well 🙏 thank you so much for the work on this

[shaper]

Hi @lgrammel FYI @kaikato just merged a short README in lmnt-node describing how we got this working -- if you see an even better way let us know, but with the one change to the next.config.js file it should work with edge. lmnt-com/lmnt-node#32

[shaper]

I thought to add that when we were hacking on vercel/ai-chatbot#151 I did see issues that looked like a challenge re: websockets staying alive on edge and so for deployment I switched to nodejs and didn't look further at the time. You can see the deployment focused work I did atop that PR here: https://github.com/shaper/lmnt-ai-chatbot/commits/main/

[Iven2132]

@lgrammel @MaxLeiter When TTS will come?

[allenchuang]

Any reason why this is closed? TTS is a great feature to have

[solanacryptodev]

Would be cool to see TTS added with the addition of gpt-4o

[alokwhitewolf]

@lgrammel / @MaxLeiter
Any follow up plans on adding TTS to vercel AI ?

[alicercedigital]

we are very excited about this!