Code

HF Tokenizer: get charspans #1584

Code

Merged

frankfliu merged 6 commits intodeepjavalibrary:masterdeepjavalibrary/djl:masterfrom andreabrduque:tokenizer/charspansandreabrduque/djl:tokenizer/charspansCopy head branch name to clipboardApr 20, 2022

+150-6

Commits

Commits on Apr 19, 2022

• Add getTokenCharSpans to rust lib

  

  andreaduque

  committed
  1c9b450

  Copy full SHA for 1c9b450

• Implement Java char spans

  

  andreaduque

  committed
  d95e1f3

  Copy full SHA for d95e1f3

• fix char span path

  

  andreaduque

  committed
  4334688

  Copy full SHA for 4334688

• Create char spans tests

  

  andreaduque

  committed
  8a190ad

  Copy full SHA for 8a190ad

• Add javadocs and copyright

  

  andreaduque

  committed
  7be2d60

  Copy full SHA for 7be2d60

Commits on Apr 20, 2022

• Add javadoc

  Show description for 9d035f5
  frankfliu

  committed
  9d035f5

  Copy full SHA for 9d035f5