Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EulerDiscreteScheduler add rescale_betas_zero_snr #6024

Merged
merged 5 commits into from
Dec 7, 2023

Conversation

Beinsezii
Copy link
Contributor

@Beinsezii Beinsezii commented Dec 1, 2023

Overview

Adds support for the rescale_betas_zero_snr config option to EulerDiscreteScheduler. Currently it works equivelently to DDIM with the concession that the final alpha_cumprod is patched to 2 ** -24 to resolve the inf issue, similar to what ComfyUI does.

This does not follow the 'curve' so to speak, and a value closer to 0 would be more appropriate, however lower values such as 2 ** -24 run into precision issues on fp16/bf16 inference. A more proper fix would be finding exactly where the precision issues lie in the pipeline code and resolving them (upcast?). don't think I am math enough for this; 2 ** -16 is satisfactory from my benchmark images on the model "ptx0/terminus-xl-gamma-training".

I have fixed the precision issues by upcasting the sample for the duration of the step() function, then re-casting back to the model's dtype before return. There appears to be no performance loss.

Examples

Unpatched EulerDiscrete
unpatched

DDIM as reference
ddim

Patched EulerDiscrete
00012

Uncertainties

  • The function rescale_zero_terminal_snr could reasonably be moved to scheduling_utils.py and shared between EulerDiscrete and DDIM, however with convention shown via betas_for_alpha_bar I've opted to duplicate it. If I or someone eventually adds ZSNR support to other samplers, this may be desirable.

Adds support for the `rescale_betas_zero_snr` config option to
EulerDiscreteScheduler. Currently it works equivelently to DDIM with
the concession that the final alpha_cumprod is patched to 2 ** -16 to
resolve the `inf` issue, similar to what ComfyUI does.

This does not follow the 'curve' so to speak, and a value closer to 0
would be more appropriate, however lower values such as 2 ** -24 run
into precision issues on fp16/bf16 inference.

A more proper fix would be finding exactly where the precision issues
lie in the pipeline code and resolving them (upcast?).

I don't think I am math enough for this; 2 ** -16 is satisfactory from
my benchmark images on the model "ptx0/terminus-xl-gamma-training".
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@Beinsezii Beinsezii marked this pull request as draft December 2, 2023 00:38
Fixes the ZSNR precision issues on fp16/bf16 with no measureable
performance loss. Now using the full 2 ** -24, the results are
effectively equivalent to DDIM's ZSNR rescaling
@Beinsezii Beinsezii marked this pull request as ready for review December 2, 2023 01:25
Beinsezii added a commit to Beinsezii/diffusers that referenced this pull request Dec 2, 2023
@sayakpaul sayakpaul requested a review from yiyixuxu December 4, 2023 04:30
Copy link
Contributor

@patrickvonplaten patrickvonplaten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok for me! Could we also add a test to

class EulerDiscreteSchedulerTest(SchedulerCommonTest):

similar to

def test_rescale_betas_zero_snr(self):

cc @yiyixuxu wdyt?

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! left a comment about #Copies from statement
and +1 on adding a test

@@ -92,6 +92,42 @@ def alpha_bar_fn(t):
return torch.tensor(betas, dtype=torch.float32)


def rescale_zero_terminal_snr(betas):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this copied from ddim? if so let's add a # Copy from statement

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is copied from DDIM which itself is copied 1:1 (and renamed) from Page 3 Alrogithm 1 of the ZSNR paper.

@Beinsezii
Copy link
Contributor Author

Added the Copy comment and rescale test. pytest passed it locally so I think that'll be it once the CI checks finish.

@bghira
Copy link
Contributor

bghira commented Dec 6, 2023

thanks. this should fix the soupy background details and splotchy skies in XL models, which Euler never suffered from. there's a slight display of that in SD 2.x models as well. it's less noticeable, which is likely why no one tried to fix it.

that said, DDIM still exhibits problems, likely something to resolve in a follow-up PR (see the sky's aberrations):
image

@Beinsezii
Copy link
Contributor Author

Beinsezii commented Dec 6, 2023

Yea you can see it in my ddim reference image too. Compared to the patched euler, it has a lot of residual noise. That's one of my main motivations for porting the beta rescale.

@patrickvonplaten
Copy link
Contributor

@yiyixuxu feel free to merge whenever

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@yiyixuxu yiyixuxu merged commit 6bf1ca2 into huggingface:main Dec 7, 2023
21 checks passed
sayakpaul pushed a commit that referenced this pull request Dec 7, 2023
* EulerDiscreteScheduler add `rescale_betas_zero_snr`
sayakpaul added a commit that referenced this pull request Dec 7, 2023
sayakpaul added a commit that referenced this pull request Dec 12, 2023
…5839)

* add poc for benchmarking workflow.

* import

* fix argument

* fix: argument

* fix: path

* fix

* fix

* path

* output csv files.

* workflow cleanup

* append token

* add utility to push to hf dataset

* fix: kw arg

* better reporting

* fix: headers

* better formatting of the numbers.

* better type annotation

* fix: formatting

* moentarily disable check

* push results.

* remove disable check

* introduce base classes.

* img2img class

* add inpainting pipeline

* intoduce base benchmark class.

* add img2img and inpainting

* feat: utility to compare changes

* fix

* fix import

* add args

* basepath

* better exception handling

* better path handling

* fix

* fix

* remove

* ifx

* fix

* add: support for controlnet.

* image_url -> url

* move images to huggingface hub

* correct urls.

* root_ckpt

* flush before benchmarking

* don't install accelerate from source

* add runner

* simplify Diffusers Benchmarking step

* change runner

* fix: subprocess call.

* filter percentage values

* fix controlnet benchmark

* add t2i adapters.

* fix filter columns

* fix t2i adapter benchmark

* fix init.

* fix

* remove safetensors flag

* fix args print

* fix

* feat: run_command

* add adapter resolution mapping

* benchmark t2i adapter fix.

* fix adapter input

* fix

* convert to L.

* add flush() add appropriate places

* better filtering

* okay

* get env for torch

* convert to float

* fix

* filter out nans.

* better coment

* sdxl

* sdxl for other benchmarks.

* fix: condition

* fix: condition for inpainting

* fix: mapping for resolution

* fix

* include kandinsky and wuerstchen

* fix: Wuerstchen

* Empty-Commit

* [Community] AnimateDiff + Controlnet Pipeline (#5928)

* begin work on animatediff + controlnet pipeline

* complete todos, uncomment multicontrolnet, input checks

Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com>

* update

Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com>

* add example

* update community README

* Update examples/community/README.md

---------

Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* EulerDiscreteScheduler add `rescale_betas_zero_snr` (#6024)

* EulerDiscreteScheduler add `rescale_betas_zero_snr`

* Revert "[Community] AnimateDiff + Controlnet Pipeline (#5928)"

This reverts commit 821726d.

* Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (#6024)"

This reverts commit 3dc2362.

* add SDXL turbo

* add lcm lora to the mix as well.

* fix

* increase steps to 2 when running turbo i2i

* debug

* debug

* debug

* fix for good

* fix and isolate better

* fuse lora so that torch compile works with peft

* fix: LCMLoRA

* better identification for LCM

* change to cron job

---------

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: Aryan V S <contact.aryanvs@gmail.com>
Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com>
Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com>
donhardman pushed a commit to donhardman/diffusers that referenced this pull request Dec 18, 2023
* EulerDiscreteScheduler add `rescale_betas_zero_snr`
donhardman pushed a commit to donhardman/diffusers that referenced this pull request Dec 18, 2023
…uggingface#5839)

* add poc for benchmarking workflow.

* import

* fix argument

* fix: argument

* fix: path

* fix

* fix

* path

* output csv files.

* workflow cleanup

* append token

* add utility to push to hf dataset

* fix: kw arg

* better reporting

* fix: headers

* better formatting of the numbers.

* better type annotation

* fix: formatting

* moentarily disable check

* push results.

* remove disable check

* introduce base classes.

* img2img class

* add inpainting pipeline

* intoduce base benchmark class.

* add img2img and inpainting

* feat: utility to compare changes

* fix

* fix import

* add args

* basepath

* better exception handling

* better path handling

* fix

* fix

* remove

* ifx

* fix

* add: support for controlnet.

* image_url -> url

* move images to huggingface hub

* correct urls.

* root_ckpt

* flush before benchmarking

* don't install accelerate from source

* add runner

* simplify Diffusers Benchmarking step

* change runner

* fix: subprocess call.

* filter percentage values

* fix controlnet benchmark

* add t2i adapters.

* fix filter columns

* fix t2i adapter benchmark

* fix init.

* fix

* remove safetensors flag

* fix args print

* fix

* feat: run_command

* add adapter resolution mapping

* benchmark t2i adapter fix.

* fix adapter input

* fix

* convert to L.

* add flush() add appropriate places

* better filtering

* okay

* get env for torch

* convert to float

* fix

* filter out nans.

* better coment

* sdxl

* sdxl for other benchmarks.

* fix: condition

* fix: condition for inpainting

* fix: mapping for resolution

* fix

* include kandinsky and wuerstchen

* fix: Wuerstchen

* Empty-Commit

* [Community] AnimateDiff + Controlnet Pipeline (huggingface#5928)

* begin work on animatediff + controlnet pipeline

* complete todos, uncomment multicontrolnet, input checks

Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com>

* update

Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com>

* add example

* update community README

* Update examples/community/README.md

---------

Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024)

* EulerDiscreteScheduler add `rescale_betas_zero_snr`

* Revert "[Community] AnimateDiff + Controlnet Pipeline (huggingface#5928)"

This reverts commit 821726d.

* Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024)"

This reverts commit 3dc2362.

* add SDXL turbo

* add lcm lora to the mix as well.

* fix

* increase steps to 2 when running turbo i2i

* debug

* debug

* debug

* fix for good

* fix and isolate better

* fuse lora so that torch compile works with peft

* fix: LCMLoRA

* better identification for LCM

* change to cron job

---------

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: Aryan V S <contact.aryanvs@gmail.com>
Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com>
Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com>
yoonseokjin pushed a commit to yoonseokjin/diffusers that referenced this pull request Dec 25, 2023
* EulerDiscreteScheduler add `rescale_betas_zero_snr`
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
* EulerDiscreteScheduler add `rescale_betas_zero_snr`
AmericanPresidentJimmyCarter pushed a commit to AmericanPresidentJimmyCarter/diffusers that referenced this pull request Apr 26, 2024
…uggingface#5839)

* add poc for benchmarking workflow.

* import

* fix argument

* fix: argument

* fix: path

* fix

* fix

* path

* output csv files.

* workflow cleanup

* append token

* add utility to push to hf dataset

* fix: kw arg

* better reporting

* fix: headers

* better formatting of the numbers.

* better type annotation

* fix: formatting

* moentarily disable check

* push results.

* remove disable check

* introduce base classes.

* img2img class

* add inpainting pipeline

* intoduce base benchmark class.

* add img2img and inpainting

* feat: utility to compare changes

* fix

* fix import

* add args

* basepath

* better exception handling

* better path handling

* fix

* fix

* remove

* ifx

* fix

* add: support for controlnet.

* image_url -> url

* move images to huggingface hub

* correct urls.

* root_ckpt

* flush before benchmarking

* don't install accelerate from source

* add runner

* simplify Diffusers Benchmarking step

* change runner

* fix: subprocess call.

* filter percentage values

* fix controlnet benchmark

* add t2i adapters.

* fix filter columns

* fix t2i adapter benchmark

* fix init.

* fix

* remove safetensors flag

* fix args print

* fix

* feat: run_command

* add adapter resolution mapping

* benchmark t2i adapter fix.

* fix adapter input

* fix

* convert to L.

* add flush() add appropriate places

* better filtering

* okay

* get env for torch

* convert to float

* fix

* filter out nans.

* better coment

* sdxl

* sdxl for other benchmarks.

* fix: condition

* fix: condition for inpainting

* fix: mapping for resolution

* fix

* include kandinsky and wuerstchen

* fix: Wuerstchen

* Empty-Commit

* [Community] AnimateDiff + Controlnet Pipeline (huggingface#5928)

* begin work on animatediff + controlnet pipeline

* complete todos, uncomment multicontrolnet, input checks

Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com>

* update

Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com>

* add example

* update community README

* Update examples/community/README.md

---------

Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024)

* EulerDiscreteScheduler add `rescale_betas_zero_snr`

* Revert "[Community] AnimateDiff + Controlnet Pipeline (huggingface#5928)"

This reverts commit 821726d.

* Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024)"

This reverts commit 3dc2362.

* add SDXL turbo

* add lcm lora to the mix as well.

* fix

* increase steps to 2 when running turbo i2i

* debug

* debug

* debug

* fix for good

* fix and isolate better

* fuse lora so that torch compile works with peft

* fix: LCMLoRA

* better identification for LCM

* change to cron job

---------

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: Aryan V S <contact.aryanvs@gmail.com>
Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com>
Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants