How to create chat endpoint with express?

[spy16]

I am trying to setup Vercel AI SDK with an express endpoint which I will then use with useChat() hook.

After all my custom logic I have a StreamingTextResponse type value. How do I stream this to the client?

This just has a pipeTo method but I am not able to find a way to get a Writable from the express-js Response type.

[MaxLeiter]

You can use streamToResponse: https://sdk.vercel.ai/docs/api-reference/stream-to-response#example-nodejs-http-server

[spy16]

Oh i just found an open issue on this

#199

[MaxLeiter]

Ah apologies, I didn't know you were using remix. can we close this and track there / upstream?

[spy16]

Yea, i didn't think the remix part was necessary since i was making an express handler, so didn't mention that initially. It's clear now that it is related to remix .

I think this is already marked as closed. I will follow the already open issues. Thanks! 🙂

[spy16]

I just figured this out. Just leaving the idea here for anyone else stumbling on this discussion looking for answers:

1. Copy-paste the file from this comment: StreamingTextResponse not compatible with Remix action function #199 (comment)

2. Take the response from openai.createChatCompletion() and re-create it as:

    new Response(
      StreamCompat.toReadable(resp.body),
      {
          status: resp.status,
          headers: resp.headers,
          statusText: resp.statusText,
      }
    )

3. Then use the new StreamingTextResponse(OpenAiStream(resp)) to create the stream that can be returned from remix-run action.

[lgrammel]

Not sure if this helps, but the AI SDK uses the Node standard fetch.

Have you tried setting the following in vite.config.js?

installGlobals({ nativeFetch: true })

More info: https://remix.run/docs/en/main/guides/single-fetch

[johnnyoshika]

I just used streamProtocol text on the client (instead of the default data) so that the server can send the text directly without SSE framing (or any other type of framing that useChat expects).

On the client:

  const { messages, input, handleInputChange, handleSubmit } =
    useChat({
      // Use streamProtocol 'text' so the server can sent text deltas directly without SSE framing
      streamProtocol: 'text',
    });

On the server:

app.post('/api/chat', async (req, res) => {
  const { messages } = req.body as { messages: CoreMessage[] };

  if (!messages) {
    res.status(400).send('messages field in body is required');
    return;
  }

  // Set headers for streaming plain text (not SSE)
  res.setHeader('Content-Type', 'text/plain;charset=utf-8');
  res.setHeader('Cache-Control', 'no-cache, no-transform');
  res.setHeader('Connection', 'keep-alive');
  res.flushHeaders(); // Ensure headers are sent immediately

  try {
    const { textStream } = await streamText({
      model: openai('gpt-4o'),
      messages,
      onFinish: event => {
        console.log('onFinish', event.text);
        // TODO: save to database
      },
    });

    for await (const delta of textStream) {
      // Client's useChat is configured to use streamProtocol 'text', so we can send text deltas directly
      // without SSE framing.
      res.write(delta);
    }

    res.end();
  } catch (error) {
    console.error('Error processing request:', error);
    res.status(500).send('Internal server error');
  }
});

[lgrammel]

https://sdk.vercel.ai/examples/api-servers/express

[johnnyoshika]

https://sdk.vercel.ai/examples/api-servers/express

That's definitely the way to do it!