Skip to content

Commit

Permalink
Careful with that cleanup!
Browse files Browse the repository at this point in the history
  • Loading branch information
Rocketknight1 committed Feb 19, 2024
1 parent 069739e commit cb74b51
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/transformers/generation/stopping_criteria.py
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ def _get_matching_positions(self) -> Tuple[Dict[str, Dict[str, List[int]]], Dict
valid positions, as well as a dictionary mapping tokens to a list of possible overlap lengths at the
end of the stop string."""
tok_list = list(self.vocab.keys())
reversed_filtered_tok_list = [self._cleanup_token(token[::-1]) for token in tok_list]
reversed_filtered_tok_list = [self._cleanup_token(token)[::-1] for token in tok_list]
token_valid_positions = {}
token_end_overlaps = {}
for stop_string in self.stop_strings:
Expand Down

0 comments on commit cb74b51

Please sign in to comment.