Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

main: use jinja chat template system prompt by default #12118

Merged
merged 5 commits into from
Mar 2, 2025

Conversation

CISC
Copy link
Contributor

@CISC CISC commented Feb 28, 2025

Using the new --jinja option would override the builtin system prompt with You are a helpful assistant if in conversation mode and/or no prompt was defined.

Fixed this behaviour so that -p option now either defines system prompt (in conversation mode) or user prompt, and does not fill in a default so that the chat template's default is used instead.

Adds some extra safety checks to be able to process an interactive user prompt with no defined system prompt.

@ochafik PTAL


No system prompt:

./build/bin/llama-cli -m phi-4-IQ2_M.gguf

 - Not using system message. To change it, set a different value via -sys PROMPT

> hi
Hello! How can I assist you today?

Incorrect use -p

./build/bin/llama-cli -m phi-4-IQ2_M.gguf -p test

*** User-specified prompt in conversation mode will be ignored, did you mean to set --system-prompt (-sys) instead?

 - Not using system message. To change it, set a different value via -sys PROMPT
...

Correctly using system prompt:

./build/bin/llama-cli -m phi-4-IQ2_M.gguf -sys "Your name is Sydney"

systemYour name is Sydney


> what is your name
My name is Sydney. How can I assist you today?

@ngxson
Copy link
Collaborator

ngxson commented Mar 1, 2025

now either defines system prompt (in conversation mode) or user prompt

From UX perspective, this is very confusing for end-user. I would suggest that we either:

  • Simply remove default system prompt if jinja is set
  • Or, add a dedicated argument for system prompt, for example -sp, --system-prompt

@CISC
Copy link
Contributor Author

CISC commented Mar 1, 2025

now either defines system prompt (in conversation mode) or user prompt
From UX perspective, this is very confusing for end-user. I would suggest that we either:

Perhaps, but it is already doing this, just wrongly. :)

* Or, add a dedicated argument for system prompt, for example `-sp, --system-prompt`

Makes sense, but maybe in a different PR?

@ngxson
Copy link
Collaborator

ngxson commented Mar 1, 2025

Makes sense, but maybe in a different PR?

Yes and it would be better if you we can merge that PR (adding -sp, --system-prompt) before merging this one.

@CISC
Copy link
Contributor Author

CISC commented Mar 1, 2025

Makes sense, but maybe in a different PR?
Yes and it would be better if you we can merge that PR (adding -sp, --system-prompt) before merging this one.

Sure, I'll take a look.

@CISC
Copy link
Contributor Author

CISC commented Mar 1, 2025

@ngxson See #12131 (-sp was already taken).

@CISC CISC changed the title Use jinja chat template system prompt by default main: use jinja chat template system prompt by default Mar 1, 2025
@CISC CISC requested a review from ngxson March 1, 2025 13:35
@CISC
Copy link
Contributor Author

CISC commented Mar 1, 2025

Failing tests seems to just be server outage.

@ngxson
Copy link
Collaborator

ngxson commented Mar 1, 2025

I'll test this with Phi-4 a bit later

@ngxson
Copy link
Collaborator

ngxson commented Mar 2, 2025

Ok that works, here is my test command + result (leaving it here for visibility):

No system prompt:

./build/bin/llama-cli -m phi-4-IQ2_M.gguf

 - Not using system message. To change it, set a different value via -sys PROMPT

> hi
Hello! How can I assist you today?

Incorrect use -p

./build/bin/llama-cli -m phi-4-IQ2_M.gguf -p test

*** User-specified prompt in conversation mode will be ignored, did you mean to set --system-prompt (-sys) instead?

 - Not using system message. To change it, set a different value via -sys PROMPT
...

Correctly using system prompt:

./build/bin/llama-cli -m phi-4-IQ2_M.gguf -sys "Your name is Sydney"

systemYour name is Sydney


> what is your name
My name is Sydney. How can I assist you today?

@CISC
Copy link
Contributor Author

CISC commented Mar 2, 2025

@ngxson I'll make a followup PR to add template formatting in non-conversation mode, and combined use of -sys and -p together.

@ngxson ngxson merged commit 14dec0c into ggml-org:master Mar 2, 2025
47 checks passed
@CISC CISC deleted the jinja-system-prompt branch March 2, 2025 14:12
mglambda pushed a commit to mglambda/llama.cpp that referenced this pull request Mar 8, 2025
* Use jinja chat template system prompt by default

* faster conditional order

* remove nested ternary

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants