Skip to content

Commit

Permalink
Apply suggestions from code review
Browse files Browse the repository at this point in the history
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
  • Loading branch information
jaime-m-p and ggerganov authored May 19, 2024
1 parent a46dfcf commit 0ae2860
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions llama.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -12820,13 +12820,13 @@ static std::vector<llama_vocab::id> llama_tokenize_internal(const llama_vocab &
while (isspace(raw_text[num_whitespaces])) {
num_whitespaces++;
}
if(num_whitespaces == raw_text.size()) {
if (num_whitespaces == raw_text.size()) {
continue; // skip if all whitespaces
}
raw_text = raw_text.substr(num_whitespaces);
}

if(vocab.add_space_prefix) {
if (vocab.add_space_prefix) {
if (!output.size() || is_prev_special) { // prefix with space if first token
raw_text = " " + raw_text;
}
Expand Down

0 comments on commit 0ae2860

Please sign in to comment.