Fix vision encoder decoder attention implementation pick

[fxmarty]

Noticed this bug while making the CI pass for #30138.

Essentially, VisionEncoderDecoderModel.from_pretrained(xxx) does not correctly load with SDPA (e.g. with bert, deit) while it should be. The reason is that VisionEncoderDecoderModel does not have a class attribute _supports_sdpa, and we can not generally set _supports_sdpa = True (as multiple archs are supported). Thus, in from_pretrained logic to pick the attention implementation, SDPA is never picked.

We need to rely on the encoder/decoder classes to rightfully pick the attention implementation.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

This highlights a wider issue with how attention is selected for models through configs. Having things autoset breaks a lot of the "don't do things by magic" convention that we have in transformers and this has already caused quite a few issues / surpising behaviour within the library.

Instead, I think we should completely re-think how attention is passed through and selected.

[fxmarty]

agreed

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.