[Llava] Add Llava to transformers

[younesbelkada]

What does this PR do?

Adds Llava - a multimodal model in transformers library.
Llava is a multi-modal model that claims to have competitive performance than GPT-4 for multi-modal tasks. There are currently 3 main variants of this architecture:

• Llava - Llama
• Llava - MPT
• Llava - Mistral (known as Bakllava): https://github.com/SkunkworksAI/BakLLaVA

This implementation leverages AutoModelForCausalLM , similarly as Blip2 to load the correct language model. The goal of this PR is to make it agnostic across all language models architectures.

Closes #25789
Closes #27221

Original llava author: https://github.com/haotian-liu/LLaVA @haotian-liu
Original PR author: @shauray8

import requests
from PIL import Image

import torch
from transformers import AutoProcessor, LlavaForVisionText2Text

model_id = "llava-hf/llava-1.5-7b-hf"
processor = AutoProcessor.from_pretrained(model_id)

prompt = "<image>\n"
prompt += "USER: What are the things I should be cautious about when I visit this place?\nASSISTANT:"
image_file = "https://llava-vl.github.io/static/images/view.jpg"

model = LlavaForVisionText2Text.from_pretrained(model_id, torch_dtype=torch.float16, low_cpu_mem_usage=True).to(0)

raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)

output = model.generate(**inputs, max_new_tokens=200)
print(processor.decode(output[0][2:], skip_special_tokens=True))
>>> USER: What are the things I should be cautious about when I visit this place?
ASSISTANT: When visiting this place, which appears to be a dock or pier extending into a body of water, you should be cautious about several factors. First, be aware of the water depth and any potential hazards, such as submerged rocks or debris, that could pose a risk to your safety. Second, be mindful of the weather conditions, as sudden changes in weather can make the dock or pier unsafe to use. Third, be cautious of the surrounding environment, as there may be wildlife or other natural elements that could pose a threat. Finally, be aware of any local regulations or guidelines for using the dock or pier, as these may include rules about fishing, swimming, or other activities. By being cautious and following any applicable guidelines, you can ensure a safe and enjoyable experience at this location.

[NielsRogge]

cc @ydshieh as can be seen, any multimodal processor has a different signature here, which will make it very difficult to support them on the image-to-text pipeline for instance. Hence we need additional tests that check whether they all comply to the same API. In this case I'd advocate for the images keyword argument followed by text as that's what most processors use.

[ydshieh]

I would suggest here to use text as the argument name to LlavaProcessor.

[NielsRogge]

Yes, but also reversed order right?

[ydshieh]

Hmm, when I worked on Kosmos-2, I kinda copied from Blip2/InstructBlip, where the order is images, text.
I see Fuyu and CLIP processors are text, images.

I don't know if there is a clear criteria to determine this. In this case, we can probably say images is the main input, so should be before text.

What's your opinion on this order thing?

[NielsRogge]

Ok so that's another reason why we need tests for this 😓 perhaps we could extend test_forward_signature for both models + processors (currently it's still implemented for text-only models). Personally I would go for images and then text (and ideally the latter should have been called texts to also be plural).

[ydshieh]

The current text is likely because it is the argument name in PreTrainedTokenizerBase (so every tokenizer classes)

text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[molbap]

That's very smart! Maybe add a comment there, like

# Calculate new positions for text tokens in merged image-text sequence.
# `image_token_mask` identifies image tokens. Each image token will be replaced by `nb_text_tokens_per_images - 1` text tokens.
# `torch.cumsum` computes how each image token shifts subsequent text token positions.
# - 1 to adjust for zero-based indexing, as `cumsum` inherently increases indices by one.

or make it part of a step-by-step explanation in the docstrings

[molbap]

Nit, maybe add type hints there, and maybe explicit args to tokenizer?

def __call__(
        self,
        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,
        images: ImageInput = None,
        padding: Union[bool, str, PaddingStrategy] = False,
        truncation: Union[bool, str, TruncationStrategy] = None,
        max_length: Optional[int] = None,
        return_tensors: Optional[Union[str, TensorType]] = None,
        **kwargs,
    ) -> BatchFeature:
    ```
    
    also, would either pass **kwargs to tokenizer (or pass explicitly needed arguments), I'm thinking about `pad_to_multiple_of` and `verbose` for instance

[NielsRogge]

Just a question here, are we ok with not having a LlavaModel? I remember for BLIP-2 people were asking for it afterwards, and as we're using AutoModelForCausalLM in the head class, we cannot use AutoModel in the base LlavaModel class, as it would not be compatible with the weights. Hence if we were to add a LlavaModel, we would need to use AutoModelForCausalLM and remove the head, I assume.

[ArthurZucker]

I'd rather we leave it as is without extra layer of complexity! And jiust add a new model output class maybe with the hidden states before the head

[ArthurZucker]

(also don't want to have issue with base model prefixes)

[molbap]

very nice trick

[asetsuna]

cool trick

[NielsRogge]

This made me realize the current way of getting features out of our AutoBackbone classes is very inefficient as well (they also use output_hidden_states=True as done here for instance).

Will update this for all backbones

[NielsRogge]

Will create a Github issue for this. Maybe we can leverage that as well here?

[ArthurZucker]

Yep that would be nice I think

[NielsRogge]

See #27873

[Hambaobao]

Hello, can you tell me how to use LlavaForConditionalGeneration in transformers to train from scratch? I want to use the weights of Vicuna and Clip Vision Transformer for training, just like the original author did.