Optimize Qwen2VL vision model by precomputing cos/sin embeds before ViT blocks

[li-plus]

What does this PR do?

The current implementation of Qwen2VL recomputes cos/sin embeddings and repeats them on each layer of the ViT model. This is computational inefficient. This PR attempts to precompute the rotary embeddings at the beginning of the ViT forward function to save computation and memory.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @amyeroberts @qubvel

[qubvel]

cc @zucchini-nlp for vlms

[zucchini-nlp]

Makes total sense, thanks! One tiny nit, we'd need to keep backwards compatibility for the vision attention classes and add a small deprecation warning

As an example of how it was in llama:

transformers/src/transformers/models/llama/modeling_llama.py

Lines 347 to 356 in c2820c9

	if position_embeddings is None:
	logger.warning_once(
	"The attention layers in this model are transitioning from computing the RoPE embeddings internally "
	"through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed "
	"`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be "
	"removed and `position_embeddings` will be mandatory."
	)
	cos, sin = self.rotary_emb(value_states, position_ids)
	else:
	cos, sin = position_embeddings

[li-plus]

I just rebased and the CI is failing on this check. Is it related to this PR?

...
No differences found for 
src/transformers/models/moonshine/modeling_moonshine.py.
Traceback (most recent call last):
  File "/root/transformers/utils/check_modular_conversion.py", line 80, in <module>
    raise ValueError("Some diff and their modeling code did not match.")
ValueError: Some diff and their modeling code did not match.

Exited with code exit status 1

[zucchini-nlp]

@li-plus Right, you need to add the changes in a "modular_model.py" file and then running make fix-copies will copy everything to the actual "modeling_model.py". So you make CI green by adding the same changes to the modular_qwen2vl.py and then with make fix-copies

[zucchini-nlp]

Thanks for iterating. Left a small comment and we need to make CI green. Approving and after changes feel free to @ ArthurZucker , the core maintainer

For the core maintainer: not sure if we need to keep the signature in functions (apply_rotary_pos_emb_vision) for BC

[li-plus]

I see. It seems that this change also affects the implementation of Qwen2.5VL since it reuses the modules in Qwen2VL. I need to get a Qwen2.5VL model to ensure it's bug-free.

[zucchini-nlp]

@li-plus running make- fix-copies should fix it and copy content from Qwen2VL to 2.5VL

[li-plus]

Running make fix-copies ports changes to Qwen2.5VL incompatible with other modules in it. Fortunately just now Qwen2.5VL has been released so I have tested and fixed it. Now this PR is compatible for both Qwen2VL & Qwen2.5VL.

[li-plus]

@ArthurZucker Hi, would you take a look at this PR? It's been approved and the CI turns green.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Super super late review! Yeah LGTM and it is welcome!

[ArthurZucker]

Suggested change

	q = q.float()
	k = k.float()
	cos = cos.unsqueeze(-2)
	sin = sin.unsqueeze(-2)
	q_embed = (q * cos) + (rotate_half(q) * sin)
	k_embed = (k * cos) + (rotate_half(k) * sin)
	q_embed = q_embed.to(orig_q_dtype)
	k_embed = k_embed.to(orig_k_dtype)
	q, k = q.float(), k.float()
	cos, sin = cos.unsqueeze(-2), sin.unsqueeze(-2)
	q_embed = (q * cos) + (rotate_half(q) * sin)
	k_embed = (k * cos) + (rotate_half(k) * sin)
	q_embed = q_embed.to(orig_q_dtype)
	k_embed = k_embed.to(orig_k_dtype)

would just like it to be a tad less verbose!

[li-plus]

Thanks! Already fixed.

[ArthurZucker]

why do we need chunk here?

[li-plus]

Because they use apply_rotary_emb provided by flash-attn when attn impl is FA2. It only needs half of the cos/sin embeddings, i.e. [seq_len, head_dim//2], while the naive apply_rotary_pos_emb_vision for eager/sdpa requires full embeds ([seq_len, head_dim]). Conventionally we pass full embeddings to this API so I have to slice it for FA2. Do you have an elegant and efficient implementation for this?

[ArthurZucker]

nope thanks for explaining!

[ArthurZucker]

Could you update to fix the conflicts? 🤗

[li-plus]

Conflicts are just solved.

[ArthurZucker]

Thanks!

[ArthurZucker]

nope thanks for explaining!