Code

Fix Qwen2Tokenizer#29929

Code

Merged

ArthurZucker merged 1 commit intohuggingface:mainhuggingface/transformers:mainfrom jklj077:fix-qwen2tokenizerjklj077/transformers:fix-qwen2tokenizerCopy head branch name to clipboardApr 3, 2024

+23-4

Commits

Commits on Mar 28, 2024

• qwen2: fixed tokens starting with # in slow tokenizer; add tests

  jklj077

  committed
  deef05a

  Copy full SHA for deef05a