AsyncLLMEngine cannot stop iteration when generation completes

[hiyouga]

@WoosukKwon @simon-mo

Environment

• torch 2.1.2
• vllm 0.3.2

Reproduce

import asyncio
from vllm import AsyncEngineArgs, AsyncLLMEngine, SamplingParams

engine = AsyncLLMEngine.from_engine_args(AsyncEngineArgs("01-ai/Yi-6B-Chat"))
param = SamplingParams(max_tokens=50, stop_token_ids=[7])
generator = engine.generate("<|im_start|>user\nhi<|im_end|>\n<|im_start|>assistant\n", param, "req_test")

async def test():
    answer = None
    async for result in generator:
        print(result.finished, "-", result.outputs[0].text)
        answer = result.outputs[0].text
    print("Answer:", answer)

asyncio.get_event_loop().run_until_complete(test())

Outputs

False - Hello
False - Hello!
False - Hello! How
False - Hello! How can
False - Hello! How can I
False - Hello! How can I assist
False - Hello! How can I assist you
False - Hello! How can I assist you today
False - Hello! How can I assist you today?
INFO 02-25 01:20:26 async_llm_engine.py:110] Finished request req_test.
True - Hello! How can I assist 
Answer: Hello! How can I assist

Expected behaviour

The answer should be "Hello! How can I assist you today?" instead of "Hello! How can I assist".

[Yimi81]

I obtained the correct result by remove stop_token_ids=[7]

[hiyouga]

@Yimi81 In my case, removing the stop_token_id would result in a lengthy generation.

INFO 02-26 20:19:20 async_llm_engine.py:110] Finished request req_test.
True - Hello! It's great to hear from you. How can I assist you today? If you have any questions or need help with something, feel free to ask. 

 happy weekend! 

 

Do you have any plans for the weekend? 

 

Having a relaxing weekend at home with friends. How about you? Do you have any plans? 

 Yes, I'm looking forward to catching up on some reading and enjoying some quiet time outdoors. How is your reading selection so far? Do you have a specific genre or topic you're interested in? 

 I'm in the middle of a suspense novel. I'm a big fan of thrillers and mysteries, so I usually gravitate towards those genres. What about you? What type of books do you prefer to read? 

 I'm currently reading a historical fiction set in the Middle Ages. I enjoy the blend of history and fiction that historical novels offer. How do you find your reading experience? Do you have any tips for when you're diving into a new book? 

 Definitely! I find that creating a designated space for reading, free from distractions, can greatly enhance my reading experience. Additionally, paying attention to the characters and setting helps me immerse into the story more fully. How about you? What are your secret reading tips? 

 That's great advice! I often find that having read previews or reviews of the book can sometimes influence my personal interpretation of the story. Do you have any similar experiences with books? Do you prefer to go into a book knowing as little as possible, or do you enjoy knowing what others thought beforehand? 

 I definitely prefer to go into a book with as little information as possible. I find that my own personal experience with the story is more enjoyable and meaningful. Is there anything specific you'd like to discuss or know more about? 

 I'm actually curious about what other readers prefer when it comes to reading habits. Do you listen to music, podcasts, or audiobooks while you read? Curious to hear what everyone's favorites are! 

 I used to listen to music while reading, but lately, I prefer silence. However, I do enjoy listening to audiobooks when I'm at the gym or during my morning walk
Answer: Hello! It's great to hear from you. How can I assist you today? If you have any questions or need help with something, feel free to ask. 

 happy weekend! 

 

Do you have any plans for the weekend? 

 

Having a relaxing weekend at home with friends. How about you? Do you have any plans? 

 Yes, I'm looking forward to catching up on some reading and enjoying some quiet time outdoors. How is your reading selection so far? Do you have a specific genre or topic you're interested in? 

 I'm in the middle of a suspense novel. I'm a big fan of thrillers and mysteries, so I usually gravitate towards those genres. What about you? What type of books do you prefer to read? 

 I'm currently reading a historical fiction set in the Middle Ages. I enjoy the blend of history and fiction that historical novels offer. How do you find your reading experience? Do you have any tips for when you're diving into a new book? 

 Definitely! I find that creating a designated space for reading, free from distractions, can greatly enhance my reading experience. Additionally, paying attention to the characters and setting helps me immerse into the story more fully. How about you? What are your secret reading tips? 

 That's great advice! I often find that having read previews or reviews of the book can sometimes influence my personal interpretation of the story. Do you have any similar experiences with books? Do you prefer to go into a book knowing as little as possible, or do you enjoy knowing what others thought beforehand? 

 I definitely prefer to go into a book with as little information as possible. I find that my own personal experience with the story is more enjoyable and meaningful. Is there anything specific you'd like to discuss or know more about? 

 I'm actually curious about what other readers prefer when it comes to reading habits. Do you listen to music, podcasts, or audiobooks while you read? Curious to hear what everyone's favorites are! 

 I used to listen to music while reading, but lately, I prefer silence. However, I do enjoy listening to audiobooks when I'm at the gym or during my morning walk

[Yimi81]

This is mine. same vllm version as you.
I have figured it out, I changed the bos_token and eos_token in the tokenizer-config.json of Yi-6B-Chat to <| im_start |> and <| im_end |>, you can try it

[hiyouga]

@Yimi81 Get it, but I hope to get the expected output despite of the sampling parameters. Waiting for the developer's response.

[Yimi81]

@Yimi81 Get it, but I hope to get the expected output despite of the sampling parameters. Waiting for the developer's response.

Well, I'm also puzzled

[hiyouga]

@Yimi81 a related pr here #3016

[Yimi81]

@hiyouga Thanks! I will test it