Add image-text-to-text task guide #31777
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
| processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b") | ||
| ``` | ||
|
|
||
| This model has a [chat template](./chat_templating) format that's required for the input. Moreover, the model can also accept multiple images as input in a single conversation or message. We will now prepare the inputs. |
There was a problem hiding this comment.
This isn't quite right, we don't need to use a chat template for the model inputs. It's just useful to correctly format the prompt in the case of message-style inputs
There was a problem hiding this comment.
You likely know better, I thought when fine-tuning these chat templates are included in fine-tuning data, thus it is required to use chat templates no? e.g. Mistral one has <INST> </INST>
There was a problem hiding this comment.
It really depends - technically the data could be already formatted as the chat string. It just happens that the message format is commonly used. There's no reason I can't pass a string directly to the tokenizer and model directly.
There was a problem hiding this comment.
I think I confused this with prompt templates
|
|
||
| acc_text = "" | ||
| for text_token in streamer: | ||
| time.sleep(0.04) |
There was a problem hiding this comment.
Why do we need to add this?
There was a problem hiding this comment.
otherwise the text flows super fast which is essentially against streaming (and also from my experience it was crashing too)
There was a problem hiding this comment.
otherwise the text flows super fast which is essentially against streaming
I'm a bit confused - don't we want our models to generate text as fast as possible? My understanding of streaming is just that we don't wait for completion before returning the result
There was a problem hiding this comment.
@amyeroberts the streaming feature enables one to see the tokens flow and stop them from flowing if the generation is going to a bad place, as in https://huggingface.co/docs/text-generation-inference/en/conceptual/streaming so we'd like it to wait a bit in between tokens
| quantized_model = Idefics2ForConditionalGeneration.from_pretrained(model_id, device_map="cuda", quantization_config=quantization_config) | ||
| ``` | ||
|
|
||
| And that's it, we can use the model the same way with no changes. |
There was a problem hiding this comment.
It would be good here to note what kind of change this makes e.g. x% reduction in memory footprint
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
|
@amyeroberts I have addressed all your comments, can you merge if you approve? |
| processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b") | ||
| ``` | ||
|
|
||
| This model has a [chat template](./chat_templating) format that's required for the input. Moreover, the model can also accept multiple images as input in a single conversation or message. We will now prepare the inputs. |
There was a problem hiding this comment.
It really depends - technically the data could be already formatted as the chat string. It just happens that the message format is commonly used. There's no reason I can't pass a string directly to the tokenizer and model directly.
|
|
||
| acc_text = "" | ||
| for text_token in streamer: | ||
| time.sleep(0.04) |
There was a problem hiding this comment.
otherwise the text flows super fast which is essentially against streaming
I'm a bit confused - don't we want our models to generate text as fast as possible? My understanding of streaming is just that we don't wait for completion before returning the result
|
@merveenoyan Not really sure why the CI runs are consistently failing here, but since this PR is just a doc page so shouldn't affect hub etc. I'm going to merge |
Added shortly
image-text-to-texttask guide that includes streaming and more