Code

Fix Llama 3 TikToken conversion#33538

Code

Merged

pcuenca merged 2 commits intomainhuggingface/transformers:mainfrom fix-llama-tokenizer-conversionhuggingface/transformers:fix-llama-tokenizer-conversionCopy head branch name to clipboardSep 19, 2024

+1-2

Commits

Commits on Sep 17, 2024

• Fix Llama 3 TikToken conversion

  pcuenca

  committed
  01cc9cd

  Copy full SHA for 01cc9cd

Commits on Sep 18, 2024

• No need to add tokens again

  pcuenca

  committed
  1a0506b

  Copy full SHA for 1a0506b