Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Correct check for SDPA in Vision Language Models #30565

Closed
12 tasks
zucchini-nlp opened this issue Apr 30, 2024 · 2 comments · Fixed by #32238
Closed
12 tasks

Correct check for SDPA in Vision Language Models #30565

zucchini-nlp opened this issue Apr 30, 2024 · 2 comments · Fixed by #32238
Labels
Should Fix This has been identified as a bug and should be fixed. Vision WIP Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress

Comments

@zucchini-nlp
Copy link
Member

zucchini-nlp commented Apr 30, 2024 •
edited
Loading

System Info

In current implementation of VLMs, the "_supports_sdpa" attribute checks and activates SDPA attention only for the language model. For example in Llava

It should also check and if available use SDPA attention for vision tower.

We can raise a warning for composite models if only one part support sdpa, but other does not, and activate SDPA for the supported part. That waythe user knows what is happening in the background.

Verified models

  • BLIP-2
  • InstructBLIP
  • InstructBLIPVideo
  • KOSMOS-2
  • LLaVa
  • LLaVa-NeXT
  • LLaVa-NeXT-Video
  • VipLLaVa
  • Video-LLaVa
  • Idefics
  • Idefics2
  • PaliGemma
@zucchini-nlp zucchini-nlp added Should Fix This has been identified as a bug and should be fixed. Vision labels Apr 30, 2024
@NielsRogge
Copy link
Contributor

Edited your issue to include a list of models to check ;) feel free to expand

@zucchini-nlp zucchini-nlp mentioned this issue Jun 13, 2024
Closed
@lucasjinreal
Copy link

Please consider add for specific Vsion Encoders, such as Siglip CLIP InternViT etc.

@zucchini-nlp zucchini-nlp added the WIP Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress label Jul 8, 2024
@huggingface huggingface deleted a comment from github-actions bot Jul 8, 2024
This was referenced Jul 25, 2024
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Should Fix This has been identified as a bug and should be fixed. Vision WIP Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Attn implementation for composite models zucchini-nlp/transformers
3 participants
@lucasjinreal @NielsRogge @zucchini-nlp