"Qwen2Tokenizer" cannot encode some words correctly

[Line290]

Below is the BUG:
transformers==4.37.1

t = AutoTokenizer.from_pretrained("Qwen1.5-72B-Chat", trust_remote_code=True, use_fast=False)
t_fast = AutoTokenizer.from_pretrained("Qwen1.5-72B-Chat", trust_remote_code=True, use_fast=True)

t.encode("#######")
# Out: [2, 2, 2, 2, 2, 2, 2]

t_fast.encode("#######")
# Out: [97864]

There are 120+ words in the vocabulary with the same BUG.

[jklj077]

For the issues related to "#" in the slow tokenizer (Qwen2Tokenizer), please try this branch and see if the problem could be fixed. There should be 121 tokens affected by this.

[Line290]

It was fixed after I followed your branch and modified the code at here. Thank you.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.