Workaround for Transformers 4.49.0 component loading dtype issue

[dsocek]

What does this PR do?

Fixes issue with Transformers 4.49.0 not loading assigned dtype in some diffusers components

Transformers release 4.49.0 has an issue that it fails to properly load assigned component dtype for some components.

This bug was created here:
https://github.com/huggingface/transformers/blob/v4.49.0/src/transformers/modeling_utils.py#L1461

torch_dtype = kwargs.pop("torch_dtype", config.torch_dtype)

Which changed from older transformers:
https://github.com/huggingface/transformers/blob/v4.46.3/src/transformers/modeling_utils.py#L1508

torch_dtype = kwargs.pop("torch_dtype", torch.get_default_dtype())

This bug has been fixed since in 4.50.0dev here:
huggingface/transformers#36335

@regisss If we will still update Optimum-Habana transformers to release 4.49.0, then we need also fix from this PR which enforces assigned dtype in a from_pretrained wrapper for diffusers pipelines

Fix Test

Easiest to test is to check text_encoder dtype which sometimes fails to get assigned dtype:

#!/usr/bin/env python
import torch
from optimum.habana.diffusers import (
   GaudiStableDiffusionPipeline,
   GaudiStableDiffusionXLPipeline,
   GaudiFluxPipeline,
   GaudiStableDiffusion3Pipeline,
)
from optimum.habana.diffusers.pipelines.stable_diffusion_xl.pipeline_stable_diffusion_xl_mlperf import (
   StableDiffusionXLPipeline_HPU,
)

model_name_or_path = "stabilityai/stable-diffusion-xl-base-1.0"
sd_pipe = GaudiStableDiffusionXLPipeline.from_pretrained(
    model_name_or_path,
    torch_dtype=torch.bfloat16,
)
print(sd_pipe.text_encoder.dtype)
print(sd_pipe.text_encoder_2.dtype)

model_name_or_path = "CompVis/stable-diffusion-v1-4"
sd_pipe = GaudiStableDiffusionPipeline.from_pretrained(
    model_name_or_path,
    torch_dtype=torch.bfloat16,
)
print(sd_pipe.text_encoder.dtype)

model_name_or_path = "black-forest-labs/FLUX.1-dev"
sd_pipe = GaudiFluxPipeline.from_pretrained(
    model_name_or_path,
    torch_dtype=torch.bfloat16,
)
print(sd_pipe.text_encoder.dtype)
print(sd_pipe.text_encoder_2.dtype)

model_name_or_path = "stabilityai/stable-diffusion-3.5-large"
sd_pipe = GaudiStableDiffusion3Pipeline.from_pretrained(
    model_name_or_path,
    torch_dtype=torch.bfloat16,
)
print(sd_pipe.text_encoder.dtype)
print(sd_pipe.text_encoder_2.dtype)
print(sd_pipe.text_encoder_3.dtype)

model_name_or_path = "stabilityai/stable-diffusion-xl-base-1.0"
sd_pipe = StableDiffusionXLPipeline_HPU.from_pretrained(
    model_name_or_path,
    torch_dtype=torch.bfloat16,
)
print(sd_pipe.text_encoder.dtype)
print(sd_pipe.text_encoder_2.dtype)

Before fix

torch.bfloat16
torch.float16

torch.bfloat16

torch.bfloat16
torch.bfloat16

torch.float16
torch.float16
torch.bfloat16

torch.bfloat16
torch.float16

Note incorrect types (FP16) for SDXL 2nd text encoder, SD3 1st and 2nd text encoder, and mlperf SDXL 2nd text encoder.

After fix

torch.bfloat16
torch.bfloat16

torch.bfloat16

torch.bfloat16
torch.bfloat16

torch.bfloat16
torch.bfloat16
torch.bfloat16

torch.bfloat16
torch.bfloat16

All types are BF16 as expected

[12010486]

LGTM as well, I've re-run python -m pytest tests/test_diffusers.py -v -s and none failed this time

[regisss]

LGTM!