add TCD Scheduler

[mhh0318]

What does this PR do?

Add a new scheduler for trajectory consistency distillation: Paper and Official Implement.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[yiyixuxu]

hi!
thanks for your PR :)
does this scheduler with SD and SDXL? can we see some examples?

[mhh0318]

Yes, it suits with SD and XL. Currently we only release SDXL LoRA. Here is a demo Space: https://huggingface.co/spaces/h1t/TCD ,
And one example can be:

import torch
from diffusers import StableDiffusionXLPipeline

+ from diffusers import TCDScheduler 

device = "cuda"
base_model_id = "stabilityai/stable-diffusion-xl-base-1.0"
tcd_lora_id = "h1t/TCD-SDXL-LoRA"

pipe = StableDiffusionXLPipeline.from_pretrained(base_model_id, torch_dtype=torch.float16, variant="fp16").to(device)
pipe.scheduler = TCDScheduler.from_config(pipe.scheduler.config)

pipe.load_lora_weights(tcd_lora_id)
pipe.fuse_lora()

prompt = "Beautiful woman, bubblegum pink, lemon yellow, minty blue, futuristic, high-detail, epic composition, watercolor."

image = pipe(
    prompt=prompt,
    num_inference_steps=4,
    guidance_scale=0,
    # Eta (referred to as `gamma` in the paper) is used to control the stochasticity in every step.
    # A value of 0.3 often yields good results.
    # We recommend using a higher eta when increasing the number of inference steps.
    eta=0.3, 
    generator=torch.Generator(device=device).manual_seed(0),
).images[0]

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

is this ready for a review?

[mhh0318]

Yup. at least looks good for me now.

[yiyixuxu]

hey thank you for the PR!
looks great and I think we can merge this soon

I left a few nits,
additionally, we introduced begin_index in PR very recently adcbe67#diff-98c53466d792a8faefb0045d29513a1c4bcd6b8144b8c06f66b697d4117039d7 - can you apply the same changes here? this will ensure your scheduler working properly with img2img pipelines

[yiyixuxu]

it seems like we are handling the case when original_steps = None

but this code here will default original_steps to config.original_inference_steps when user pass the None value - is this intended?

[mhh0318]

This is an option for setting the range of t from a) ~ [0, 999] or b)~[9, 19, ... 999] whenconfig.original_inference_steps=50.

[yiyixuxu]

ok, got it
so I think there might be an error in the below code then, if user pass original_inference_steps = None, orignal_steps will be set to config.original_inference_steps , so orignal_steps won't be None and the else part won't get evaluated

        if original_steps is not None:
            if original_steps > self.config.num_train_timesteps:
                raise ValueError(
                    f"`original_steps`: {original_steps} cannot be larger than `self.config.train_timesteps`:"
                    f" {self.config.num_train_timesteps} as the unet model trained with this scheduler can only handle"
                    f" maximal {self.config.num_train_timesteps} timesteps."
                )
            # TCD Timesteps Setting
            # The skipping step parameter k from the paper.
            k = self.config.num_train_timesteps // original_steps
            # TCD Training/Distillation Steps Schedule
            tcd_origin_timesteps = np.asarray(list(range(1, int(original_steps * strength) + 1))) * k - 1
        else:
            tcd_origin_timesteps = np.asarray(list(range(0, int(self.config.num_train_timesteps * strength))))

[yiyixuxu]

left one question
let me know, if it is not an error

PR looks ready to merge for me. thanks for the great work!

[yiyixuxu]

Suggested change

	if original_steps is not None:
	if original_inference_steps is not None:

maybe this? not sure..
see https://github.com/huggingface/diffusers/pull/7174/files#r1511499565

[mhh0318]

else should only be evaluated with customised training steps or some special situation. I think following would be better:

# default option
if original_inference_steps is None: 
  tcd_origin_timesteps align with discrete inference steps
# customised 
else:
  tcd_origin_timesteps truly randomised

[sayakpaul]

Nice!

I think there also needs to a guide in our docs for the users to know how to best use TCD just like we have https://huggingface.co/docs/diffusers/main/en/using-diffusers/inference_with_lcm_lora.

@yiyixuxu WDYT?

[yiyixuxu]

I think there also needs to a guide in our docs for the users to know how to best use TCD just like we have https://huggingface.co/docs/diffusers/main/en/using-diffusers/inference_with_lcm_lora.

sure!

[yiyixuxu]

let us know when it's ready to merge

I think there also needs to a guide in our docs for the users to know how to best use TCD just like we have https://huggingface.co/docs/diffusers/main/en/using-diffusers/inference_with_lcm_lora.

what @sayakpaul suggested here is a great way to help promote this scheduler, feel free to add a guide either in this PR or a separate one

[mhh0318]

let us know when it's ready to merge

I think there also needs to a guide in our docs for the users to know how to best use TCD just like we have https://huggingface.co/docs/diffusers/main/en/using-diffusers/inference_with_lcm_lora.

what @sayakpaul suggested here is a great way to help promote this scheduler, feel free to add a guide either in this PR or a separate one

I think it is ready to merge for TCD scheduler now. I would open a new PR for the guideline.