Code

Refactor tokenization interface#65

Code

Merged

Saibo-creator merged 3 commits intoepfl-dlab:mainepfl-dlab/transformers-CFG:mainfrom Saibo-creator:refactor_TokenRecognizer_add_NonIncremental_versionSaibo-creator/transformers-CFG:refactor_TokenRecognizer_add_NonIncremental_versionCopy head branch name to clipboardJul 5, 2024

+540-530

Commits

Commits on Jul 5, 2024

• feat: rename accept_state to parsing_state; rename all *consume* methods to *update_state_with* methods; rename all *token_acceptance* to *next_token_acceptance*; add class NonIncrementalTokenRecog…

  Show description for 6bfff39
  Saibo-creator

  committed
  6bfff39

  Copy full SHA for 6bfff39

• unify tokenizer interface and add unicode support for llama and mistral

  Saibo-creator

  committed
  8f19142

  Copy full SHA for 8f19142

• add support for t5 tokenizer

  Saibo-creator

  committed
  bbd1349

  Copy full SHA for bbd1349