GPTBigCodeForCausalLM, TP >= 2, output is bloken. Is this BUG?

[s-natsubori]

I updated vLLM version from 0.2.1.post1 to 0.2.7.
Model Generation is Broken when tensor_parallel_size >=2.
(tensor_parallel_size=1 is NOT bloken）

First I found it with my Starchatβ+awq model,
and it repro with non awq model HuggingFaceH4/starchat-beta
and bigcode/starcoderbase-1b too.

Base Env

• docker image based nvcr.io/nvidia/pytorch:23.08-py3
• Nvidia Driver Version: 525.85.12
• CUDA Version: 12.0
• GPU Tesla T4
• Model bigcode/starcoderbase-1b

Old Env

• transformers==4.35.0
• vllm==0.2.1.post1
• xformers==0.0.22

Engine args

INFO 01-11 07:12:45 llm_engine.py:72] Initializing an LLM engine with config: model='/usr/local/model/llm', tokenizer='/usr/local/model/llm', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=2, quantization=None, seed=0)

Input

import numpy as np
import scipy as sp

def hello_world():

generated

    print('hello world')

def show_me_a_table(table):
    for row in table:
        print(row)

New Env

• transformers==4.36.0
• autoawq==0.1.7
• vllm==0.2.7
• xformers==0.0.23.post1

Engine args

INFO 01-11 07:14:24 llm_engine.py:73] Initializing an LLM engine with config: model='/usr/local/model/llm', tokenizer='/usr/local/model/llm', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=2, quantization=None, enforce_eager=False, seed=0)

Input

import numpy as np
import scipy as sp

def hello_world():

generated

return "Hello world"

def Hello_world():
    return "Hello"
    def Hello":
    return "Hello"
    def():
        def
        def(def):
            def
        def def():
            def
        def
        def def: def def defDEFdefDEFDEFDEFDEFDEFdefDEFDEFDEFdefdefdefdefDEFDEFdefdefdefdefdefdefdefdefdefdefdefdefdefdefDEFDEFdefdefdefdefdefdefdefdefdefDefDefdefdef

or starchat-beta generate

What
¿

Cu

What
¿

¿

With old env , only update vllm==0.2.2 , generation is broken.
So I guess it is Bug with tensor_parallel.

[esmeetu]

Hi, @s-natsubori. Could you try the latest main branch? I think #2379 can solve this problem.

[s-natsubori]

@esmeetu
Thanks following.
It works perfect. amazing!!