Code

Tokenizer fixes#8379

Code

Draft

jaime-m-p wants to merge 9 commits intoggerganov:masterggerganov/llama.cpp:masterfrom jaime-m-p:tokenizer-fixesjaime-m-p/llama.cpp:tokenizer-fixesCopy head branch name to clipboard

+136-40

Commits

Commits on Jul 8, 2024

• Fix pyparse problems: gcc inline functions

  

  jaime-m-p

  committed
  e8b3955

  Copy full SHA for e8b3955

• Test l/r-strip for more than 4 spaces

  

  jaime-m-p

  committed
  9307c3f

  Copy full SHA for 9307c3f

• Improve mismatch range localization

  

  jaime-m-p

  committed
  a943b42

  Copy full SHA for a943b42

• Compare vocabs

  

  jaime-m-p

  committed
  dec64ef

  Copy full SHA for dec64ef

• Options to mange token text decoding errors:

  Show description for c184db7
  

  jaime-m-p

  committed
  c184db7

  Copy full SHA for c184db7

Commits on Jul 9, 2024

• Skip literal UNUSED token checks

  

  jaime-m-p

  committed
  3eb1900

  Copy full SHA for 3eb1900

Commits on Jul 11, 2024

• update test: fix special and added token lists

  

  jaime-m-p

  committed
  c4956e4

  Copy full SHA for c4956e4

• Merge commit 'f4444d99' into tokenizer-fixes

  

  jaime-m-p

  committed
  9b8e05b

  Copy full SHA for 9b8e05b

Commits on Jul 19, 2024

• Merge branch 'master' into tokenizer-fixes

  

  jaime-m-p

  committed
  3db5058

  Copy full SHA for 3db5058