Add torch.no_grad, fix greedy_until bug #161
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gituser768
approved these changes
Apr 19, 2024
| primary_until = None | ||
| for tokenized_until in tokenizer(untils)["input_ids"]: | ||
| primary_until = tokenizer.eos_token_id | ||
| for tokenized_until in tokenizer(untils, add_special_tokens=False)["input_ids"]: |
There was a problem hiding this comment.
The add_special_tokeens=False prevents the tokenizer from adding a <bos> token of sorts at the start of the string, which some tokenizers do, and would mess up the test that there's only one token (unfortunately, other tokenizers, like Llama/Mistral, also adds a "space" token when you tokenize "\n", so this is still not effective for those - will be improved in next iteration of model handling, more centrally)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This was an unnecessary oversight spotted by @jjyang77, that we're not wrapping model calls with
torch.no_grad(). Added here for thelanguage_model.pymodel type. In a couple of experiments it doesn't seem to speed evaluation up by much, but does improve memory usage (was able to run Llama-3-8B on single GPU rather than needing two).Also fixed a bug in the
greedy_untilmethod which causes crashes for some models (like Mistral) whenprimary_untilis set asNone, as noted by @dmh43.