BOS/EOS/PAD options in tokens cli; speed up tokenization by segment…
#428
| Job | Run time |
|---|---|
| 5m 13s | |
| 8m 44s | |
| 1s | |
| 7m 43s | |
| 5m 33s | |
| 3m 42s | |
| 8m 14s | |
| 5m 41s | |
| 5m 10s | |
| 13s | |
| 4m 9s | |
| 8s | |
| 12s | |
| 9s | |
| 13s | |
| 1m 49s | |
| 0s | |
| 56m 54s |