Code

Per token attributes#7685

Code

Merged

jaime-m-p merged 10 commits intoggerganov:masterggerganov/llama.cpp:masterfrom jaime-m-p:per-token-attribsjaime-m-p/llama.cpp:per-token-attribsCopy head branch name to clipboardJun 4, 2024

+155-62

Commits

Commits on Jun 1, 2024

• Add per token attrib enum

  

  jaime-m-p

  committed
  cec6a3b

  Copy full SHA for cec6a3b

• Using phi-3 for testing 'rstrip'

  

  jaime-m-p

  committed
  3ead1b9

  Copy full SHA for 3ead1b9

• bugfix: assertions, wrong special token list

  

  jaime-m-p

  committed
  33de247

  Copy full SHA for 33de247

• Implement 'rstrip' properly

  

  jaime-m-p

  committed
  ada961c

  Copy full SHA for ada961c

• Refactor + add 'jina-v2' for testing 'lstrip'

  

  jaime-m-p

  committed
  01c9229

  Copy full SHA for 01c9229

Commits on Jun 2, 2024

• Update phi-3 GGUF file (obsolete since 917dc8c)

  

  jaime-m-p

  committed
  8564c19

  Copy full SHA for 8564c19

• Update brute force test: testing 'lstrip' and 'rstrip'

  

  jaime-m-p

  committed
  83cac0e

  Copy full SHA for 83cac0e

• Fix previous commit

  

  jaime-m-p

  committed
  54e9f23

  Copy full SHA for 54e9f23

• Replace llama_token_type with llama_token_attribs

  

  jaime-m-p

  committed
  ac40ff0

  Copy full SHA for ac40ff0

Commits on Jun 3, 2024

• Rename token attributes

  

  jaime-m-p

  committed
  18f5fc7

  Copy full SHA for 18f5fc7