-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
EulerDiscreteScheduler add rescale_betas_zero_snr
#6024
Conversation
Adds support for the `rescale_betas_zero_snr` config option to EulerDiscreteScheduler. Currently it works equivelently to DDIM with the concession that the final alpha_cumprod is patched to 2 ** -16 to resolve the `inf` issue, similar to what ComfyUI does. This does not follow the 'curve' so to speak, and a value closer to 0 would be more appropriate, however lower values such as 2 ** -24 run into precision issues on fp16/bf16 inference. A more proper fix would be finding exactly where the precision issues lie in the pipeline code and resolving them (upcast?). I don't think I am math enough for this; 2 ** -16 is satisfactory from my benchmark images on the model "ptx0/terminus-xl-gamma-training".
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
Fixes the ZSNR precision issues on fp16/bf16 with no measureable performance loss. Now using the full 2 ** -24, the results are effectively equivalent to DDIM's ZSNR rescaling
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok for me! Could we also add a test to
class EulerDiscreteSchedulerTest(SchedulerCommonTest): |
similar to
def test_rescale_betas_zero_snr(self): |
cc @yiyixuxu wdyt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks! left a comment about #Copies from
statement
and +1 on adding a test
@@ -92,6 +92,42 @@ def alpha_bar_fn(t): | |||
return torch.tensor(betas, dtype=torch.float32) | |||
|
|||
|
|||
def rescale_zero_terminal_snr(betas): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this copied from ddim? if so let's add a # Copy from statement
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is copied from DDIM which itself is copied 1:1 (and renamed) from Page 3 Alrogithm 1 of the ZSNR paper.
Added the |
Yea you can see it in my ddim reference image too. Compared to the patched euler, it has a lot of residual noise. That's one of my main motivations for porting the beta rescale. |
@yiyixuxu feel free to merge whenever |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks!
* EulerDiscreteScheduler add `rescale_betas_zero_snr`
…5839) * add poc for benchmarking workflow. * import * fix argument * fix: argument * fix: path * fix * fix * path * output csv files. * workflow cleanup * append token * add utility to push to hf dataset * fix: kw arg * better reporting * fix: headers * better formatting of the numbers. * better type annotation * fix: formatting * moentarily disable check * push results. * remove disable check * introduce base classes. * img2img class * add inpainting pipeline * intoduce base benchmark class. * add img2img and inpainting * feat: utility to compare changes * fix * fix import * add args * basepath * better exception handling * better path handling * fix * fix * remove * ifx * fix * add: support for controlnet. * image_url -> url * move images to huggingface hub * correct urls. * root_ckpt * flush before benchmarking * don't install accelerate from source * add runner * simplify Diffusers Benchmarking step * change runner * fix: subprocess call. * filter percentage values * fix controlnet benchmark * add t2i adapters. * fix filter columns * fix t2i adapter benchmark * fix init. * fix * remove safetensors flag * fix args print * fix * feat: run_command * add adapter resolution mapping * benchmark t2i adapter fix. * fix adapter input * fix * convert to L. * add flush() add appropriate places * better filtering * okay * get env for torch * convert to float * fix * filter out nans. * better coment * sdxl * sdxl for other benchmarks. * fix: condition * fix: condition for inpainting * fix: mapping for resolution * fix * include kandinsky and wuerstchen * fix: Wuerstchen * Empty-Commit * [Community] AnimateDiff + Controlnet Pipeline (#5928) * begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * update Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * EulerDiscreteScheduler add `rescale_betas_zero_snr` (#6024) * EulerDiscreteScheduler add `rescale_betas_zero_snr` * Revert "[Community] AnimateDiff + Controlnet Pipeline (#5928)" This reverts commit 821726d. * Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (#6024)" This reverts commit 3dc2362. * add SDXL turbo * add lcm lora to the mix as well. * fix * increase steps to 2 when running turbo i2i * debug * debug * debug * fix for good * fix and isolate better * fuse lora so that torch compile works with peft * fix: LCMLoRA * better identification for LCM * change to cron job --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com>
* EulerDiscreteScheduler add `rescale_betas_zero_snr`
…uggingface#5839) * add poc for benchmarking workflow. * import * fix argument * fix: argument * fix: path * fix * fix * path * output csv files. * workflow cleanup * append token * add utility to push to hf dataset * fix: kw arg * better reporting * fix: headers * better formatting of the numbers. * better type annotation * fix: formatting * moentarily disable check * push results. * remove disable check * introduce base classes. * img2img class * add inpainting pipeline * intoduce base benchmark class. * add img2img and inpainting * feat: utility to compare changes * fix * fix import * add args * basepath * better exception handling * better path handling * fix * fix * remove * ifx * fix * add: support for controlnet. * image_url -> url * move images to huggingface hub * correct urls. * root_ckpt * flush before benchmarking * don't install accelerate from source * add runner * simplify Diffusers Benchmarking step * change runner * fix: subprocess call. * filter percentage values * fix controlnet benchmark * add t2i adapters. * fix filter columns * fix t2i adapter benchmark * fix init. * fix * remove safetensors flag * fix args print * fix * feat: run_command * add adapter resolution mapping * benchmark t2i adapter fix. * fix adapter input * fix * convert to L. * add flush() add appropriate places * better filtering * okay * get env for torch * convert to float * fix * filter out nans. * better coment * sdxl * sdxl for other benchmarks. * fix: condition * fix: condition for inpainting * fix: mapping for resolution * fix * include kandinsky and wuerstchen * fix: Wuerstchen * Empty-Commit * [Community] AnimateDiff + Controlnet Pipeline (huggingface#5928) * begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * update Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024) * EulerDiscreteScheduler add `rescale_betas_zero_snr` * Revert "[Community] AnimateDiff + Controlnet Pipeline (huggingface#5928)" This reverts commit 821726d. * Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024)" This reverts commit 3dc2362. * add SDXL turbo * add lcm lora to the mix as well. * fix * increase steps to 2 when running turbo i2i * debug * debug * debug * fix for good * fix and isolate better * fuse lora so that torch compile works with peft * fix: LCMLoRA * better identification for LCM * change to cron job --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com>
* EulerDiscreteScheduler add `rescale_betas_zero_snr`
* EulerDiscreteScheduler add `rescale_betas_zero_snr`
…uggingface#5839) * add poc for benchmarking workflow. * import * fix argument * fix: argument * fix: path * fix * fix * path * output csv files. * workflow cleanup * append token * add utility to push to hf dataset * fix: kw arg * better reporting * fix: headers * better formatting of the numbers. * better type annotation * fix: formatting * moentarily disable check * push results. * remove disable check * introduce base classes. * img2img class * add inpainting pipeline * intoduce base benchmark class. * add img2img and inpainting * feat: utility to compare changes * fix * fix import * add args * basepath * better exception handling * better path handling * fix * fix * remove * ifx * fix * add: support for controlnet. * image_url -> url * move images to huggingface hub * correct urls. * root_ckpt * flush before benchmarking * don't install accelerate from source * add runner * simplify Diffusers Benchmarking step * change runner * fix: subprocess call. * filter percentage values * fix controlnet benchmark * add t2i adapters. * fix filter columns * fix t2i adapter benchmark * fix init. * fix * remove safetensors flag * fix args print * fix * feat: run_command * add adapter resolution mapping * benchmark t2i adapter fix. * fix adapter input * fix * convert to L. * add flush() add appropriate places * better filtering * okay * get env for torch * convert to float * fix * filter out nans. * better coment * sdxl * sdxl for other benchmarks. * fix: condition * fix: condition for inpainting * fix: mapping for resolution * fix * include kandinsky and wuerstchen * fix: Wuerstchen * Empty-Commit * [Community] AnimateDiff + Controlnet Pipeline (huggingface#5928) * begin work on animatediff + controlnet pipeline * complete todos, uncomment multicontrolnet, input checks Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * update Co-Authored-By: EdoardoBotta <botta.edoardo@gmail.com> * add example * update community README * Update examples/community/README.md --------- Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024) * EulerDiscreteScheduler add `rescale_betas_zero_snr` * Revert "[Community] AnimateDiff + Controlnet Pipeline (huggingface#5928)" This reverts commit 821726d. * Revert "EulerDiscreteScheduler add `rescale_betas_zero_snr` (huggingface#6024)" This reverts commit 3dc2362. * add SDXL turbo * add lcm lora to the mix as well. * fix * increase steps to 2 when running turbo i2i * debug * debug * debug * fix for good * fix and isolate better * fuse lora so that torch compile works with peft * fix: LCMLoRA * better identification for LCM * change to cron job --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: EdoardoBotta <botta.edoardo@gmail.com> Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com>
Overview
Adds support for the
rescale_betas_zero_snr
config option to EulerDiscreteScheduler. Currently it works equivelently to DDIM with the concession that the final alpha_cumprod is patched to 2 ** -24 to resolve theinf
issue, similar to what ComfyUI does.This does not follow the 'curve' so to speak, and a value closer to 0 would be more appropriate, however lower values such as 2 ** -24 run into precision issues on fp16/bf16 inference. A more proper fix would be finding exactly where the precision issues lie in the pipeline code and resolving them (upcast?). don't think I am math enough for this; 2 ** -16 is satisfactory from my benchmark images on the model "ptx0/terminus-xl-gamma-training".I have fixed the precision issues by upcasting the sample for the duration of the step() function, then re-casting back to the model's dtype before return. There appears to be no performance loss.
Examples
Unpatched EulerDiscrete
![unpatched](https://private-user-images.githubusercontent.com/39478211/287391508-816bc7ee-114e-4088-a7c5-71f2cef96293.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzk2MTQxODEsIm5iZiI6MTczOTYxMzg4MSwicGF0aCI6Ii8zOTQ3ODIxMS8yODczOTE1MDgtODE2YmM3ZWUtMTE0ZS00MDg4LWE3YzUtNzFmMmNlZjk2MjkzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjE1VDEwMDQ0MVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY2ODI2MDdmNmRiMTliMWNiZDY0ZjAyYzljYWJhMTRkM2RjMWM4NzNiMWQ3YTJkMTgzODMxZTdkYzg4N2NkNDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.DVZcsYGz0JcMnDtWzX2-L0YINuxaf1mJRpEgJ-u4lxE)
DDIM as reference
![ddim](https://private-user-images.githubusercontent.com/39478211/287391478-661bbb1f-4667-4fc2-9721-3e6755895da4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzk2MTQxODEsIm5iZiI6MTczOTYxMzg4MSwicGF0aCI6Ii8zOTQ3ODIxMS8yODczOTE0NzgtNjYxYmJiMWYtNDY2Ny00ZmMyLTk3MjEtM2U2NzU1ODk1ZGE0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjE1VDEwMDQ0MVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTQ2MDFkZjIwNDFmMmE0NDAyOTFhYzYyNjVjOTI3MDY2NmRiMGM2NTYxOTMzMmQ0ZDQ1ODY2ZWEzMDVmMjczMmUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.KGuiu8WpNZc79c0HcRodpyK1V33bYC4AZvTSHYFERYo)
Patched EulerDiscrete
![00012](https://private-user-images.githubusercontent.com/39478211/287410909-30b560d1-82dd-4f3c-9c47-7e626188ab4a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzk2MTQxODEsIm5iZiI6MTczOTYxMzg4MSwicGF0aCI6Ii8zOTQ3ODIxMS8yODc0MTA5MDktMzBiNTYwZDEtODJkZC00ZjNjLTljNDctN2U2MjYxODhhYjRhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjE1VDEwMDQ0MVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJkYmExZWQyZGVlODhiMDE4YmIzNzMyOGJmN2VjNDc5NWNiYmVlM2QzODQwNmE1MWEzOTg5NGY4NDhmM2M5ZTUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.cszIcgpaXF8Maejn8pUP6m0wDrUMzF1xsTjAz_6mZgM)
Uncertainties
rescale_zero_terminal_snr
could reasonably be moved toscheduling_utils.py
and shared between EulerDiscrete and DDIM, however with convention shown viabetas_for_alpha_bar
I've opted to duplicate it. If I or someone eventually adds ZSNR support to other samplers, this may be desirable.