Skip to content

Commit

Permalink
server : fix the disappearance of the end of the text (ggml-org#9867)
Browse files Browse the repository at this point in the history
* server: fix the disappearance of the end of the text when streaming with stop strings

* simplify "send text" checks
  • Loading branch information
z80maniac authored and dsx1986 committed Oct 29, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
1 parent 859b462 commit c0c80ff
Showing 1 changed file with 4 additions and 5 deletions.
9 changes: 4 additions & 5 deletions examples/server/server.cpp
Original file line number Diff line number Diff line change
@@ -1090,22 +1090,21 @@ struct server_context {
size_t pos = std::min(slot.n_sent_text, slot.generated_text.size());

const std::string str_test = slot.generated_text.substr(pos);
bool is_stop_full = false;
bool send_text = true;

size_t stop_pos = slot.find_stopping_strings(str_test, token_str.size(), STOP_TYPE_FULL);
if (stop_pos != std::string::npos) {
is_stop_full = true;
slot.generated_text.erase(
slot.generated_text.begin() + pos + stop_pos,
slot.generated_text.end());
pos = std::min(slot.n_sent_text, slot.generated_text.size());
} else {
is_stop_full = false;
} else if (slot.has_next_token) {
stop_pos = slot.find_stopping_strings(str_test, token_str.size(), STOP_TYPE_PARTIAL);
send_text = stop_pos == std::string::npos;
}

// check if there is any token to predict
if (stop_pos == std::string::npos || (!slot.has_next_token && !is_stop_full && stop_pos > 0)) {
if (send_text) {
// no send the stop word in the response
result.text_to_send = slot.generated_text.substr(pos, std::string::npos);
slot.n_sent_text += result.text_to_send.size();

0 comments on commit c0c80ff

Please sign in to comment.