Fix quality Olmo + SDPA

[fxmarty]

Byproduct of me not merging/rebasing after #29890, conflicting with #30070

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[fxmarty]

For the record some olmo slows tests were already failign before this PR (running on A100). Otherwise all good.

FAILED tests/models/olmo/test_modeling_olmo.py::OlmoIntegrationTest::test_model_1b_logits - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index ...
FAILED tests/models/olmo/test_modeling_olmo.py::OlmoIntegrationTest::test_model_7b_greedy_generation - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index ...
FAILED tests/models/olmo/test_modeling_olmo.py::OlmoIntegrationTest::test_model_7b_logits - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index ...
FAILED tests/models/olmo/test_modeling_olmo.py::OlmoIntegrationTest::test_model_7b_twin_2t_logits - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index ...