[Bugfix][Misc]: fix graph capture for decoder (vllm-project#9549)

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
neuralmagic · Nov 23, 2024 · d4097b5 · d4097b5
1 parent 96ec0c7
commit d4097b5
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/vllm/worker/model_runner.py b/vllm/worker/model_runner.py
@@ -828,7 +828,7 @@ def build(self) -> ModelInputForGPU:
 
         cuda_graph_pad_size = self._get_cuda_graph_pad_size(
             num_seqs=len(seq_lens),
-            max_decode_seq_len=max_encoder_seq_len,
+            max_decode_seq_len=max_decode_seq_len,
             max_encoder_seq_len=max_encoder_seq_len)
 
         batch_size = len(input_tokens)