"Continue generation" request to API is missing a parameter

[Originalimoc]

When you click stop generation, there will be an additional continue generation button which won't appear if you let the generation finish itself. That button should trigger a generation request contains the specific parameter in another project's issue closing comment or the HF endpoint will assume it's a new round and model will not behave as what continue generation intended. It's called "Prefilled" generation request. Another use case of this feature is model steering. To prevent this, add add_generation_prompt to false in the request to prevent adding an extra turn:

Relavant:
"https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html?ref=blog.mozilla.ai#:~:text=role.%22)%2C%0A%20%20%20%20)-,add_generation_prompt,-%3A%20bool%20%3D"
&
"huggingface/transformers#33198"

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

add_generation_prompt is not an OpenAI compatible parameter, so it's not missing. There's no way of knowing if your endpoint is Huggingface-based or not.

[Originalimoc]

@danny-avila Given this is a project that people are interested in accessing locally deployed model through it. Plus vLLM and tabbyAPI and textgenwebui OpenAI compatible endpoint supports it. Should we at least add an option to include this? Prefill is also properly supported by Claude and such.