Automatically apply chat template in non-chat scenarios

[sbalandi]

CVS-157276

[AlexKoff88]

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

[ilya-lavrenov]

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline?
do they apply chat_template by default?

@eaidova

[AlexKoff88]

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?

@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

[ilya-lavrenov]

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

[AlexKoff88]

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

Double-checked, and it seems like HF changed the behaviour at some point for text-generation pipeline. Details. But the input should be formatted appropreatelly to trigger chat template usage. If the user just passes a string data, no chat template is applied.

[ilya-lavrenov]

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

Double-checked, and it seems like HF changed the behaviour at some point for text-generation pipeline. Details. But the input should be formatted appropreatelly to trigger chat template usage. If the user just passes a string data, no chat template is applied.

do you think it's better to add explicit flag, then?

pipe.generate(prompt, apply_chat_template=True, max_new_tokens=40)

[AlexKoff88]

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

Double-checked, and it seems like HF changed the behaviour at some point for text-generation pipeline. Details. But the input should be formatted appropreatelly to trigger chat template usage. If the user just passes a string data, no chat template is applied.

do you think it's better to add explicit flag, then?

pipe.generate(prompt, apply_chat_template=True, max_new_tokens=40)

This option looks good to me but for drop-in replacement of HF API to OV GenAI it is better to follow HF approach with message format. Anyway, they should have more experience and user's feedback.

[sbalandi]

If using a chat template is supposed to be a default behavior for .generate() method it is not aligned with HF Transformers lib. We should turn it off in the tools at least (both WWB and LLM-Bench).

what's about HF e2e pipeline? do they apply chat_template by default?
@eaidova

text2text-generation pipeline does not use a chat template by default from what I know.

what if it's instruction model?

Double-checked, and it seems like HF changed the behaviour at some point for text-generation pipeline. Details. But the input should be formatted appropreatelly to trigger chat template usage. If the user just passes a string data, no chat template is applied.

do you think it's better to add explicit flag, then?

pipe.generate(prompt, apply_chat_template=True, max_new_tokens=40)

This option looks good to me but for drop-in replacement of HF API to OV GenAI it is better to follow HF approach with message format. Anyway, they should have more experience and user's feedback.

Should both ways be added - possibility to put messages to the generate() function, apply chat_template if input value is messages and leave as is if it's string and add apply_chat_template as input parameter for generate() ?

[sbalandi]

to discuss:

1. Test tests/python_tests/test_sampling.py::test_greedy with use_cb=True , model_id katuni4ka/tiny-random-phi3 , prompt="I have an interview about product speccing with the company Weekend Health. Give me an example of a question they might ask with regards about a new feature" and max_new_tokens=300 fails on Linux, so was added apply_chat_template=False
2. Test tests/python_tests/test_llm_pipeline.py::test_string_inputs with beam search args and prompt 'Alan Turing was a' fails on MacOS , prompt was changed to Why is the Sun yellow?

[sbalandi]

to discuss:

1. Test tests/python_tests/test_sampling.py::test_greedy with use_cb=True , model_id katuni4ka/tiny-random-phi3 , prompt="I have an interview about product speccing with the company Weekend Health. Give me an example of a question they might ask with regards about a new feature" and max_new_tokens=300 fails on Linux, so was added apply_chat_template=False
2. Test tests/python_tests/test_llm_pipeline.py::test_string_inputs with beam search args and prompt 'Alan Turing was a' fails on MacOS , prompt was changed to Why is the Sun yellow?

Created bugs: CVS161498 , CVS161499