max_num_batched_tokens and max_num_seqs values

[isRambler]

Hello, because I am new to vllm, I want to know how to set the max_num_batched_tokens and max_num_seqs values in order to achieve maximum inference performance. What is the relationship between max_num_batched_tokens and max_num_seqs? Why do the output tokens appear when I set different max_num_batched_tokens and max_num_seqs? The totals may be inconsistent

[JasonZhu1313]

max_num_batched_tokens and max_num_seqs essentially determines the batch size at prefill stage - the first time when the model performs inference to predict the next token in a sequence. vLLM utilizes continuous batching to achieve high throughput. The underlying scheduler determines, at a certain step, the batched samples for inference based on the state of GPU memory and the availability of KV Cache blocks. The scheduler separates queries into three categories: Waiting Queue (sequences at prefill stage), Running Queue (sequences at decoding stage), Swapping Queue (if cpu / gpu swapping is enabled).

I have drawn a diagram to better illustrate the scheduling workflow and you can see max_num_batched_tokens and max_num_seqs determines the batch size of waiting queue:

If you set the max_num_batched_tokens or max_num_seqs with low value then the prefill batch size will be small (e.g., 1) which might not hurt performance, there is no one size fit all suggestion I guess, I think you can tweak the prefill batch size through these two knobs and use benchmark_serving.py in vLLM to determine what size leads to best performance.

[isRambler]

max_num_batched_tokens and max_num_seqs essentially determines the batch size at prefill stage - the first time when the model performs inference to predict the next token in a sequence. vLLM utilizes continuous batching to achieve high throughput. The underlying scheduler determines, at a certain step, the batched samples for inference based on the state of GPU memory and the availability of KV Cache blocks. The scheduler separates queries into three categories: Waiting Queue (sequences at prefill stage), Running Queue (sequences at decoding stage), Swapping Queue (if cpu / gpu swapping is enabled).

I have drawn a diagram to better illustrate the scheduling workflow and you can see max_num_batched_tokens and max_num_seqs determines the batch size of waiting queue:

If you set the max_num_batched_tokens or max_num_seqs with low value then the prefill batch size will be small (e.g., 1) which might not hurt performance, there is no one size fit all suggestion I guess, I think you can tweak the prefill batch size through these two knobs and use benchmark_serving.py in vLLM to determine what size leads to best performance.

OK, thanks for the explanation

[banyip]

Hello, because I am new to vllm, I want to know how to set the max_num_batched_tokens and max_num_seqs values in order to achieve maximum inference performance. What is the relationship between max_num_batched_tokens and max_num_seqs? Why do the output tokens appear when I set different max_num_batched_tokens and max_num_seqs? The totals may be inconsistent

Is there a final answer to set this variable?

[realJaydenCheng]

Hello, because I am new to vllm, I want to know how to set the max_num_batched_tokens and max_num_seqs values in order to achieve maximum inference performance. What is the relationship between max_num_batched_tokens and max_num_seqs? Why do the output tokens appear when I set different max_num_batched_tokens and max_num_seqs? The totals may be inconsistent

Is there a final answer to set this variable?

You can set these variables when initializing the LLM object:

from vllm import LLM
llm = LLM(
    model=LLM_PATH,
    max_num_batched_tokens=512*50,
    max_model_len=512*50,
    gpu_memory_utilization=0.3,
)

[ehartford]

can we have some kind of guidance or rule of thumb? How do we decide, practically speaking, what values to set in order to maximize performance?