[Tokenizer][bug] LLAVA 1.6 tokenizer problem

[lanking520]

System Info

Any OS system that can run transformers

related issue: vllm-project/vllm#6224

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

transformers (4.42.3) and got the issue

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("llava-hf/llava-v1.6-34b-hf")
print(tokenizer.encode("<image>"))

print(tokenizer.vocab_size)

output

[64003]
64000

Can confirm transformers (4.40.1) does not have this issue

[64000]
64000

Expected behavior

It is supposed to be the same?

[64000]
64000

[DarkLight1337]

After trying this code on a few different versions, it looks like this got changed between 4.41.2 and 4.42.0.

[DarkLight1337]

Also, this issue only occurs for llava-hf/llava-v1.6-34b-hf. It works fine for llava-hf/llava-v1.6-mistral-7b-hf and llava-hf/llava-v1.6-vicuna-7b-hf.

Update: I think this is the same issue as #31713.

[zucchini-nlp]

Answered in #31713 (comment)

[zucchini-nlp]

Closing because #31902 was merged