Code

#32184 save total_vocab_size#32240

Code

Merged

itazap merged 3 commits intomainhuggingface/transformers:mainfrom vocab_size_32184huggingface/transformers:vocab_size_32184Copy head branch name to clipboardAug 5, 2024

+24-3

Commits

Commits on Jul 26, 2024

• save total_vocab_size = vocab_size + user added tokens to speed up operation

  itazap

  committed
  24f2abb

  Copy full SHA for 24f2abb

Commits on Jul 31, 2024

• updating length when added_tokens_decoder is set

  itazap

  committed
  cf00ef7

  Copy full SHA for cf00ef7

Commits on Aug 2, 2024

• add test len(tokenizer)

  itazap

  committed
  00eef14

  Copy full SHA for 00eef14