Sdpa dino v2

[avishaiElmakies]

adds SDPA to dinov2

copy from VIT

address #28005

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@amyeroberts @fxmarty

notes

• seems to fail some tests about output_attentions=True. but sdpa doesn't support it. when i added to OPT, if output_attentions==True then you send to eager mode. but ViT didn't have it. so i didn't add it here. can add if needed

[amyeroberts]

Thanks for adding!

Before merge we'll need to run the slow model tests. Could you push an empty commit with the message [run-slow] dinov2?

The currently failing tests will also need to be resolved

[avishaiElmakies]

@amyeroberts how should i resolve the failing tests?

[amyeroberts]

@avishaiElmakies You'll need to look at the CI logs and debug to figure out the problem based on the error message. First step is to make sure you can replicate the errors locally. You can also inspect other PRs which added SDPA e.g. #30555 to see if there's a common pattern of fixes which need to be applied

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[avishaiElmakies]

@amyeroberts thanks for the reply
I would love some guidance if possible.

from what i understand, those are the tests that are failing:

FAILED tests/models/dinov2/test_modeling_dinov2.py::Dinov2ModelTest::test_attention_outputs - AttributeError: 'NoneType' object has no attribute 'shape'
FAILED tests/models/dinov2/test_modeling_dinov2.py::Dinov2ModelTest::test_retain_grad_hidden_states_attentions - AttributeError: 'NoneType' object has no attribute 'retain_grad'
FAILED tests/models/dinov2/test_modeling_dinov2.py::Dinov2ModelTest::test_equivalence_flax_to_pt - AssertionError: False is not true : outputs.attentions_0: `pt_outputs` should a tensor when `fx_outputs` is
FAILED tests/models/dinov2/test_modeling_dinov2.py::Dinov2ModelTest::test_equivalence_pt_to_flax - AssertionError: False is not true : outputs.attentions_0: `pt_outputs` should a tensor when `fx_outputs` is

from what I understand, those are related to the fact that sdpa doesn't return output_attentions. so i would love to know how to handle those. if output_attentions == True should i send them to eager?

FAILED tests/models/dinov2/test_modeling_dinov2.py::Dinov2ModelTest::test_eager_matches_sdpa_inference_0_float16 - AttributeError: 'Dinov2ModelTester' object has no attribute 'num_masks'
FAILED tests/models/dinov2/test_modeling_dinov2.py::Dinov2ModelTest::test_eager_matches_sdpa_inference_1_bfloat16 - AttributeError: 'Dinov2ModelTester' object has no attribute 'num_masks'
FAILED tests/models/dinov2/test_modeling_dinov2.py::Dinov2ModelTest::test_eager_matches_sdpa_inference_2_float32 - AttributeError: 'Dinov2ModelTester' object has no attribute 'num_masks'
FAILED tests/models/dinov2/test_modeling_dinov2.py::Dinov2ModelIntegrationTest::test_inference_no_head - AssertionError: False is not true

Those are very weird to me. I am able to replicate them, but something is weird. It seems someone changed the Dinov2Embeddings class, and it is different from the main branch. When I use the DinoV2Embedding from the main branch, I pass the tests. I can't really find the culprit or the relevant pull request to understand. so I would love some guidance as well

[avishaiElmakies]

@amyeroberts
would love some guidance here.
about the first 4 tests, I couldn't really find any PR that talks about this situation. What i have seen is that models (e.g. llama) send to eager version if output_attentions == True
Would also love some guidance about the other 4 tests.

[amyeroberts]

For the first four tests - refer to the linked PR and the adaptations to the tests made there. You'll see that "eager" is indeed selected. The diffs will show how this should be done.

For the other tests -- re the comment "It seems someone changed the Dinov2Embeddings class, and it is different from the main branch.". The version of DinoV2 on this branch will depend on which commit from main you branch off from. You can see all the changes to the dino v2 model here: https://github.com/huggingface/transformers/commits/ce62a41880b5b70a304d068eb58f55894a5a7af8/src/transformers/models/dinov2/modeling_dinov2.py.

Any PR should be in sync and up-to-date with the main branch. To make sure this is the case, you'll need to do frequent merges from main into this branch or rebases of this branch onto main.

If the branches are up-to-date and there still remains a differences between tests passing on main and this branch then it means the changes in this PR are causing the tests to fail and you'll need to debug to investigate this. Here's a handy guide on debugging with pytest: https://docs.pytest.org/en/stable/how-to/failures.html

[avishaiElmakies]

@amyeroberts fixed the 7 tests (at least on my machine). test_inference_no_head fails in the main branch as well

[amyeroberts]

@avishaiElmakies Thanks for running the tests and adding this feature! Can confirm tests/models/dinov2/test_modeling_dinov2.py::Dinov2ModelIntegrationTest::test_inference_no_head fails on main so good to merge!

[lezhang7]

@avishaiElmakies Thanks for you good contribution, this new features help us a lot! BTW, could you share an example or update doc on how to use this new features?

[avishaiElmakies]

@amyeroberts great to hear, thanks for the guidance!

@lezhang7 I don't understand, this is sdpa in dinov2

[NielsRogge]

@lezhang7 see the docs here: https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#pytorch-scaled-dot-product-attention

[lezhang7]

Thank you! @NielsRogge , I use following for loading the model:

processor = AutoImageProcessor.from_pretrained('facebook/dinov2-large')
model = AutoModel.from_pretrained('facebook/dinov2-large',attn_implementation="sdpa")

However, I didn't find accelerated inference speed compared to original code without explicitly flag spda, is there anything wrong with my code?

[avishaiElmakies]

I believe that after adding sdpa implementation, the default implementation is sdpa(at least for dinov2). So you don't need to use the flag to get sdpa implementation. You should compare

attn_implementation="eager" vs attn_implementation="sdpa"