Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add vocab_size to Tokenizer.train #69

Merged
merged 3 commits into from
Mar 31, 2023
Merged

Add vocab_size to Tokenizer.train #69

merged 3 commits into from
Mar 31, 2023

Conversation

lantiga
Copy link
Collaborator

@lantiga lantiga commented Mar 31, 2023

Addresses #68

awaelchli
awaelchli reviewed Mar 31, 2023
View reviewed changes
@lantiga lantiga requested a review from awaelchli March 31, 2023 11:58
@lantiga lantiga merged commit f7aa8b9 into main Mar 31, 2023
@lantiga lantiga deleted the tokenizer-vocab-size branch March 31, 2023 12:07
timothylimyl referenced this pull request in timothylimyl/lit-llama-qa May 21, 2023
@lantiga
Add vocab_size to Tokenizer.train (#69)
72aeb85
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@awaelchli awaelchli awaelchli approved these changes

@carmocca carmocca Awaiting requested review from carmocca

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lantiga @awaelchli