Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Misc. bug: missing chat-template support in llama-run #11178

Open
engelmi opened this issue Jan 10, 2025 · 1 comment
Open

Misc. bug: missing chat-template support in llama-run #11178

engelmi opened this issue Jan 10, 2025 · 1 comment
Labels
bug-unconfirmed stale

Comments

@engelmi
Copy link
Contributor

engelmi commented Jan 10, 2025

Name and Version

$ llama-cli --version
version: 4460 (ba8a1f9)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

Other (Please specify in the next section)

Command line

llama-run granite-code

Problem description & steps to reproduce

The llama-run CLI currently doesn't take the chat template of a model into account. Thus executing llama-run on a model requiring such a template will fail.

First Bad Commit

No response

Relevant log output

failed to apply the chat template
@engelmi
Copy link
Contributor Author

engelmi commented Jan 10, 2025 •
edited
Loading

One possible solution could be to extend llama-run so that it implicitly searches for a <model-name>.ollama.template and applies it via common_chat_apply_template - or directly downloads and saves it together with the model with the mentioned name pattern (if build with LLAMA_USE_CURL). In addition, another CLI option such as --chat-template could be added to not rely on an implicit naming convention (and template location).

Although it might take me a while, I'd be up for implementing this - if it makes sense, of course.

@engelmi engelmi changed the title Misc. bug: Misc. bug: missing chat-template support in llama-run Jan 11, 2025
engelmi added a commit to engelmi/llama.cpp that referenced this issue Jan 13, 2025
@engelmi
Added chat template support to llama-run
8dbb9e5 
Fixes: ggml-org#11178

The llama-run CLI currently doesn't take the chat template of
a model into account. Thus executing llama-run on a model
requiring a chat template will fail.
In order to solve this, the chat template is being downloaded
from ollama or huggingface as well and applied during the chat.

Signed-off-by: Michael Engel <mengel@redhat.com>
@engelmi engelmi mentioned this issue Jan 13, 2025
Closed
engelmi added a commit to engelmi/llama.cpp that referenced this issue Jan 13, 2025
@engelmi
Added chat template support to llama-run
c0b1b49 
Fixes: ggml-org#11178

The llama-run CLI currently doesn't take the chat template of
a model into account. Thus executing llama-run on a model
requiring a chat template will fail.
In order to solve this, the chat template is being downloaded
from ollama or huggingface as well and applied during the chat.

Signed-off-by: Michael Engel <mengel@redhat.com>
engelmi added a commit to engelmi/llama.cpp that referenced this issue Jan 13, 2025
@engelmi
Added chat template support to llama-run
baf2844 
Fixes: ggml-org#11178

The llama-run CLI currently doesn't take the chat template of
a model into account. Thus executing llama-run on a model
requiring a chat template will fail.
In order to solve this, the chat template is being downloaded
from ollama or huggingface as well and applied during the chat.

Signed-off-by: Michael Engel <mengel@redhat.com>
engelmi added a commit to engelmi/llama.cpp that referenced this issue Jan 14, 2025
@engelmi
Added chat template support to llama-run
a899673 
Fixes: ggml-org#11178

The llama-run CLI currently doesn't take the chat template of
a model into account. Thus executing llama-run on a model
requiring a chat template will fail.
In order to solve this, the chat template is being downloaded
from ollama or huggingface as well and applied during the chat.

Signed-off-by: Michael Engel <mengel@redhat.com>
@github-actions github-actions bot added the stale label Feb 11, 2025
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 17, 2025
@engelmi
Added --chat-template-file to llama-run
36a0c4f 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
@engelmi engelmi mentioned this issue Feb 17, 2025
Closed
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 19, 2025
@engelmi
Added --chat-template-file to llama-run
2ec2107 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 19, 2025
@engelmi
Added --chat-template-file to llama-run
6e50443 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 19, 2025
@engelmi
Added --chat-template-file to llama-run
f01a139 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 19, 2025
@engelmi
Added --chat-template-file to llama-run
86f68a8 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
@engelmi engelmi mentioned this issue Feb 19, 2025
Merged
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 19, 2025
@engelmi
Added --chat-template-file to llama-run
a154293 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 19, 2025
@engelmi
Added --chat-template-file to llama-run
22fc626 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_init.

Signed-off-by: Michael Engel <mengel@redhat.com>
@engelmi engelmi mentioned this issue Feb 19, 2025
Closed
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 19, 2025
@engelmi
Added --chat-template-file to llama-run
193eb87 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 19, 2025
@engelmi
Added --chat-template-file to llama-run
d1b57cf 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
engelmi added a commit to engelmi/llama.cpp that referenced this issue Feb 19, 2025
@engelmi
Added --chat-template-file to llama-run
530ba31 
Relates to: ggml-org#11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
ggerganov pushed a commit that referenced this issue Feb 20, 2025
@engelmi
run : add --chat-template-file (#11961)
0d55958 
Relates to: #11178

Added --chat-template-file CLI option to llama-run. If specified, the file
will be read and the content passed for overwriting the chat template of
the model to common_chat_templates_from_model.

Signed-off-by: Michael Engel <mengel@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug-unconfirmed stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Added chat template support to llama-run engelmi/llama.cpp
1 participant
@engelmi