instruct pix2pix pipeline: remove sigma scaling when computing classifier free guidance

[erliding]

Fixes (#6981)

instruct pix2pix pipeline scale model prediction to sigma space before computing classifier free guidance and scale back when done for schedulers operating in sigma space, which seems unnecessary, especially when rescale_betas_zero_snr is enabled for those schedulers, this scaling can result in fp16 overflow with high rates and generating black images

for example (noise_pred - latents) below can goes beyond -65504 and result in -inf for fp16 (for this specific case reorganize rhs of the assignment to latents / sigma - noise_pred / sigma should also work)

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py

Line 439 in 3a7e481

noise_pred = (noise_pred - latents) / (-sigma)

@yiyixuxu @sayakpaul

[sayakpaul]

@erliding do you have some results for us to verify things here?

Cc: @patil-suraj here as well.

[yiyixuxu]

can you maybe run the docstring example with fixed seed from main branch and this branch and see if they are identical?
(I think math is the same)

[erliding]

@erliding do you have some results for us to verify things here?

Cc: @patil-suraj here as well.

yeah, so i tested with EulerDiscreteScheduler and EulerAncestralDiscreteScheduler for both eps and v_prediction with StableDiffusionInstructPix2PixPipeline, removing sigma scaling doesn't seem to have noticeable impact on generated images, and setting rescale_betas_zero_snr=True won't cause generating black images for fp16 inference anymore

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[erliding]

i ran the example below on main branch and this branch, it use EulerAncestralDiscreteScheduler
https://github.com/huggingface/diffusers/tree/main/examples/instruct_pix2pix#inference
attached the edited images, first from main, second from this branch

[sayakpaul]

Alright, thanks. But I varely see any qualitative difference though.

[erliding]

Alright, thanks. But I varely see any qualitative difference though.

it shouldn't, it is to show sigma scaling can be removed without introducing difference

[sayakpaul]

Alright, yes, that makes sense!

[sayakpaul]

I am okay with the changes but @patil-suraj needs to approve this.

[yiyixuxu]

gentle pin @patil-suraj

[patil-suraj]

Great find! This was done to mimic the original code but actually shouldn't make difference here. Thanks a lot for the PR.