Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

7529 do not disable autocast for cuda devices #7530

Merged
merged 15 commits into from
Apr 2, 2024
Merged

7529 do not disable autocast for cuda devices #7530

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
688051c
7529 do not disable autocast for cuda devices
Mar 30, 2024
14a72b8
Merge branch 'main' into bugfix/7529-autocast-failure-cuda
bghira Mar 30, 2024
20a7c8f
Remove typecasting error check for non-mps platforms, as a correct au…
Mar 31, 2024
ad3eb80
add autocast fix to other training examples
Mar 31, 2024
381f73d
disable native_amp for dreambooth (sdxl)
Apr 1, 2024
59b9b41
disable native_amp for pix2pix (sdxl)
Apr 1, 2024
2f9e7c6
Merge branch 'main' into bugfix/7529-autocast-failure-cuda
sayakpaul Apr 1, 2024
35cdecd
remove tests from remaining files
Apr 1, 2024
561bf2d
disable native_amp on huggingface accelerator for every training exam…
Apr 1, 2024
55912ed
convert more usages of autocast to nullcontext, make style fixes
Apr 1, 2024
455bb6f
make style fixes
Apr 1, 2024
0ce09ec
Merge branch 'main' into bugfix/7529-autocast-failure-cuda
sayakpaul Apr 2, 2024
b1b8bc9
Merge branch 'main' into bugfix/7529-autocast-failure-cuda
bghira Apr 2, 2024
1483581
style.
sayakpaul Apr 2, 2024
d084ad5
Empty-Commit
sayakpaul Apr 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions examples/dreambooth/train_dreambooth_lora_sdxl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,11 +208,7 @@ def log_validation(
# Currently the context determination is a bit hand-wavy. We can improve it in the future if there's a better
# way to condition it. Reference: https://github.com/huggingface/diffusers/pull/7126#issuecomment-1968523051
enable_autocast = True
if torch.backends.mps.is_available() or (
accelerator.mixed_precision == "fp16" or accelerator.mixed_precision == "bf16"
):
enable_autocast = False
if "playground" in args.pretrained_model_name_or_path:
if torch.backends.mps.is_available() or "playground" in args.pretrained_model_name_or_path:
enable_autocast = False

with torch.autocast(
Expand Down
4 changes: 1 addition & 3 deletions examples/instruct_pix2pix/train_instruct_pix2pix_sdxl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -985,9 +985,7 @@ def collate_fn(examples):

# Some configurations require autocast to be disabled.
enable_autocast = True
if torch.backends.mps.is_available() or (
accelerator.mixed_precision == "fp16" or accelerator.mixed_precision == "bf16"
):
if torch.backends.mps.is_available():
enable_autocast = False

# Train!
Expand Down
4 changes: 1 addition & 3 deletions examples/text_to_image/train_text_to_image_lora_sdxl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -981,9 +981,7 @@ def collate_fn(examples):

# Some configurations require autocast to be disabled.
enable_autocast = True
if torch.backends.mps.is_available() or (
accelerator.mixed_precision == "fp16" or accelerator.mixed_precision == "bf16"
):
if torch.backends.mps.is_available():
enable_autocast = False

# Train!
Expand Down
4 changes: 1 addition & 3 deletions examples/text_to_image/train_text_to_image_sdxl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -988,9 +988,7 @@ def unwrap_model(model):

# Some configurations require autocast to be disabled.
enable_autocast = True
if torch.backends.mps.is_available() or (
accelerator.mixed_precision == "fp16" or accelerator.mixed_precision == "bf16"
):
if torch.backends.mps.is_available():
enable_autocast = False

# Train!
Expand Down
4 changes: 0 additions & 4 deletions src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1241,10 +1241,6 @@ def __call__(
if torch.backends.mps.is_available():
# some platforms (eg. apple mps) misbehave due to a pytorch bug: https://github.com/pytorch/pytorch/pull/99272
self.vae = self.vae.to(latents.dtype)
else:
raise ValueError(
"For the given accelerator, there seems to be an unexpected problem in type-casting. Please file an issue on the PyTorch GitHub repository. See also: https://github.com/huggingface/diffusers/pull/7446/."
)

# unscale/denormalize the latents
# denormalize with the mean and std if available and not None
Expand Down
4 changes: 0 additions & 4 deletions src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1418,10 +1418,6 @@ def denoising_value_valid(dnv):
if torch.backends.mps.is_available():
# some platforms (eg. apple mps) misbehave due to a pytorch bug: https://github.com/pytorch/pytorch/pull/99272
self.vae = self.vae.to(latents.dtype)
else:
raise ValueError(
"For the given accelerator, there seems to be an unexpected problem in type-casting. Please file an issue on the PyTorch GitHub repository. See also: https://github.com/huggingface/diffusers/pull/7446/."
)

# unscale/denormalize the latents
# denormalize with the mean and std if available and not None
Expand Down
4 changes: 0 additions & 4 deletions src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1785,10 +1785,6 @@ def denoising_value_valid(dnv):
if torch.backends.mps.is_available():
# some platforms (eg. apple mps) misbehave due to a pytorch bug: https://github.com/pytorch/pytorch/pull/99272
self.vae = self.vae.to(latents.dtype)
else:
raise ValueError(
"For the given accelerator, there seems to be an unexpected problem in type-casting. Please file an issue on the PyTorch GitHub repository. See also: https://github.com/huggingface/diffusers/pull/7446/."
)

# unscale/denormalize the latents
# denormalize with the mean and std if available and not None
Expand Down
4 changes: 0 additions & 4 deletions src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_instruct_pix2pix.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -950,10 +950,6 @@ def __call__(
if torch.backends.mps.is_available():
# some platforms (eg. apple mps) misbehave due to a pytorch bug: https://github.com/pytorch/pytorch/pull/99272
self.vae = self.vae.to(latents.dtype)
else:
raise ValueError(
"For the given accelerator, there seems to be an unexpected problem in type-casting. Please file an issue on the PyTorch GitHub repository. See also: https://github.com/huggingface/diffusers/pull/7446/."
)

# unscale/denormalize the latents
# denormalize with the mean and std if available and not None
Expand Down
Loading