Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update test to use new tokenizer type API #2692

Merged
merged 22 commits into from
Aug 21, 2023
Merged

Update test to use new tokenizer type API #2692

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
bb89266
Merge tokenizer fixes into the gguf branch.
goerch Aug 8, 2023
98f5b1f
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
goerch Aug 8, 2023
4566533
Add test vocabularies
goerch Aug 8, 2023
712b614
Merge branch 'gguf' into gguf
goerch Aug 14, 2023
fb591e1
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
goerch Aug 14, 2023
7108448
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
goerch Aug 14, 2023
cfb0e6f
Adapt convert-new.py (and fix a clang-cl compiler error on windows)
goerch Aug 14, 2023
c9c3b87
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
goerch Aug 14, 2023
99e0e90
Improved tokenizer test
goerch Aug 15, 2023
c545d85
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
goerch Aug 15, 2023
d864596
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
goerch Aug 17, 2023
c16ea8e
Merge branch 'ggerganov:gguf' into gguf
goerch Aug 19, 2023
21d8864
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
goerch Aug 19, 2023
370a95f
Improve token type support
goerch Aug 19, 2023
dc65fb3
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
goerch Aug 19, 2023
da83740
Exclude platform dependent tests
goerch Aug 19, 2023
aea173f
More sentencepiece compatibility by eliminating magic numbers
goerch Aug 19, 2023
dea1e4c
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
goerch Aug 20, 2023
6586487
Restored accidentally removed comment
goerch Aug 20, 2023
44dd9ed
Improve commentary
goerch Aug 20, 2023
a856685
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
goerch Aug 21, 2023
11e3806
Use token type API in test-tokenizer-1.cpp
goerch Aug 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions convert.py
100755 → 100644
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -741,6 +741,8 @@ def add_meta_vocab(self, vocab: Vocab) -> None:
tokens = []
scores = []
toktypes = []
# NOTE: `all_tokens` returns the the base vocabulary and added tokens
# TODO: add special tokens?
for text, score, toktype in vocab.all_tokens():
tokens.append(text)
scores.append(score)
Expand All @@ -751,8 +753,6 @@ def add_meta_vocab(self, vocab: Vocab) -> None:
self.gguf.add_token_scores(scores)
self.gguf.add_token_types(toktypes)

# TODO: added / special tokens

def add_tensor_info(self, name: str, tensor: LazyTensor) -> None:
n_elements = 1
for dim in tensor.shape:
Expand Down
4 changes: 2 additions & 2 deletions tests/test-tokenizer-1.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ int main(int argc, char **argv) {
return 2;
}
} else {
// TODO: needs access to token types
if (0 <= i && i < 259) {
llama_token_type type = llama_token_get_type(ctx, i);
if (type == LLAMA_TOKEN_TYPE_UNKNOWN || type == LLAMA_TOKEN_TYPE_CONTROL || type == LLAMA_TOKEN_TYPE_BYTE) {
fprintf(stderr, "%s : info: token %d is string %s and bpe returns tokens %s\n",
__func__, i, llama_token_to_str(ctx, i).c_str(), unescape_whitespace(ctx, tokens).c_str());
} else {
Expand Down