[Ray Serve] need support for streaming outputs

[jiaanguo]

Description

I am building a chatbot with ray serves my model, currently my model supports streaming outputs like generating word tokens one after another. However, I could not get ray to send token back immediately in streaming manner. This would be a feature greatly required!

@serve.deployment(route_prefix="/", ray_actor_options={"num_gpus": 1})
@serve.ingress(app)
class ChatbotModelDeployment:

    def _infer(...):
        for output in self._model.stream_chat(...):
            yield output+'\n'

    @app.post("/")
    async def query(self, request: Request):
        data = await request.json()
        query = data.get("query", "")

        output = self._infer(query, ...)
        return StreamingResponse(output, media_type="text/plain")

  
chatbot_model_deployment = ChatbotModelDeployment.bind()

Something like above that allows StreamingResponse from FastAPI, currently even I tried using Streaming Response, it still responses the whole paragraph once all tokens have been generated.

Use case

Especially in nlp model deployment, need this feature to send token back directly rather than wait for the whole paragraph being generated.

[merionum]

Hello guys! Any estimates on when this can be included in ray serve? This feature would be crucial for LLM work happening globally at the moment

[edoakes]

@merionum we are currently working on adding this and hoping to have an experimental version ready for use in Ray 2.5

[edoakes]

This is now available as an experimental feature for 2.5. Planning to have it on by default in 2.6:
https://docs.ray.io/en/latest/serve/http-guide.html#streaming-responses