Add Intro page of TCD #7259

mhh0318 · 2024-03-08T16:56:44Z

What does this PR do?

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sayakpaul

Hi Sayak, I've updated an intro page for TCD. Please have a review.

sayakpaul · 2024-03-09T06:18:38Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+> ***Better than Teacher:*** TCD maintains superior generative quality at both low NFEs and high NFEs, even exceeding the performance of DPM-Solver++(2S) with origin SDXL. It is worth noting that there is no additional discriminator or LPIPS supervision included during training.
+
+> ***Flexible NFEs:*** The NFEs for TCD sampling can be varied at will without adversely affecting the quality of the results.
+
+> ***Freely Change the Detailing:*** During inference, the level of detail in the image can be simply modified by adjusing one hyper-parameter gamma. This option does not require the introduction of any additional parameters.


(nit): Could we maybe make these bullet points?

sayakpaul · 2024-03-09T06:20:10Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+
+From the [Official Project Page](https://mhh0318.github.io/tcd/), the major merit of TCD can be outlined as follows:
+
+> ***Better than Teacher:*** TCD maintains superior generative quality at both low NFEs and high NFEs, even exceeding the performance of DPM-Solver++(2S) with origin SDXL. It is worth noting that there is no additional discriminator or LPIPS supervision included during training.


The reader doesn't yet know what NFE is. Also, it's worth hyperlinking DPM-Solver++(2S).

sayakpaul · 2024-03-09T06:20:42Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+
+From the [Official Project Page](https://mhh0318.github.io/tcd/), the major merit of TCD can be outlined as follows:
+
+> ***Better than Teacher:*** TCD maintains superior generative quality at both low NFEs and high NFEs, even exceeding the performance of DPM-Solver++(2S) with origin SDXL. It is worth noting that there is no additional discriminator or LPIPS supervision included during training.


Suggested change

> ***Better than Teacher:*** TCD maintains superior generative quality at both low NFEs and high NFEs, even exceeding the performance of DPM-Solver++(2S) with origin SDXL. It is worth noting that there is no additional discriminator or LPIPS supervision included during training.

> ***Better than Teacher:*** TCD maintains superior generative quality at both low NFEs and high NFEs, even exceeding the performance of DPM-Solver++(2S) with Stable Diffusion XL (SDXL). It is worth noting that no additional discriminator or LPIPS supervision is included during training.

sayakpaul · 2024-03-09T06:21:59Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+
+For more technical details of TCD, please refer to [the paper](https://arxiv.org/abs/2402.19159).
+
+Trajectory consistency distillation can directly place on top of a pre-trained diffusion model as a LoRA module. Such LoRA can be identified as a versatile acceleration module applicable to different fine-tuned models or LoRAs sharing the same base model without the need for additional training.


Suggested change

Trajectory consistency distillation can directly place on top of a pre-trained diffusion model as a LoRA module. Such LoRA can be identified as a versatile acceleration module applicable to different fine-tuned models or LoRAs sharing the same base model without the need for additional training.

Trajectory consistency distillation can be directly placed on top of a pre-trained diffusion model as a [LoRA](https://huggingface.co/docs/diffusers/main/en/training/lora) module. Such a LoRA can be identified as a versatile acceleration module applicable to different fine-tuned models or LoRAs sharing the same base model without the need for additional training.

sayakpaul · 2024-03-09T06:22:37Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+
+TCD-LoRAs are available for [stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5), [stable-diffusion-2-1-base](https://huggingface.co/stabilityai/stable-diffusion-2-1-base), and [stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0). 
+
+The corresponding checkpoints can be found at [TCD-SD15](https://huggingface.co/h1t/TCD-SD15-LoRA), [TCD-SD21-base](https://huggingface.co/h1t/TCD-SD21-base-LoRA) and [TCD-SDXL](https://huggingface.co/h1t/TCD-SDXL-LoRA), separately.


Suggested change

The corresponding checkpoints can be found at [TCD-SD15](https://huggingface.co/h1t/TCD-SD15-LoRA), [TCD-SD21-base](https://huggingface.co/h1t/TCD-SD21-base-LoRA) and [TCD-SDXL](https://huggingface.co/h1t/TCD-SDXL-LoRA), separately.

The corresponding checkpoints can be found at [TCD-SD15](https://huggingface.co/h1t/TCD-SD15-LoRA), [TCD-SD21-base](https://huggingface.co/h1t/TCD-SD21-base-LoRA), and [TCD-SDXL](https://huggingface.co/h1t/TCD-SDXL-LoRA), respectively.

sayakpaul · 2024-03-09T06:23:09Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+- IP-Adapter
+- AnimateDiff
+
+TCD-LoRA can be considered an advanced method compared with [LCM-LoRA](https://latent-consistency-models.github.io/). The guide of TCD-LoRA workflow is:


Suggested change

TCD-LoRA can be considered an advanced method compared with [LCM-LoRA](https://latent-consistency-models.github.io/). The guide of TCD-LoRA workflow is:

TCD-LoRA can be considered an advanced method compared with [LCM-LoRA](https://huggingface.co/docs/diffusers/main/en/using-diffusers/inference_with_lcm_lora). The main parts of the TCD-LoRA workflow are as follows:

sayakpaul · 2024-03-09T06:26:11Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+<Tip>
+Eta (referred to as `gamma` in the paper) is used to control the stochasticity in every step.
+A value of 0.3 often yields good results, where eta = 0 means determinstic and eta = 1 is identity to Multi-step Consistency Sampler (as well as LCMScheduler).
+We recommend using a higher eta when increasing the number of inference steps.
+</Tip>


Suggested change

<Tip>

Eta (referred to as `gamma` in the paper) is used to control the stochasticity in every step.

A value of 0.3 often yields good results, where eta = 0 means determinstic and eta = 1 is identity to Multi-step Consistency Sampler (as well as LCMScheduler).

We recommend using a higher eta when increasing the number of inference steps.

</Tip>

<Tip>

Eta (referred to as `gamma` in the paper) is used to control the stochasticity in every step.

A value of 0.3 often yields good results, where eta = 0 means determinstic and eta = 1 is identity to Multi-step Consistency Sampler (as well as LCMScheduler).

We recommend using a higher eta when increasing the number of inference steps.

</Tip>

sayakpaul · 2024-03-09T06:27:33Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+pipe.load_lora_weights(tcd_lora_id)
+pipe.fuse_lora()
+
+prompt = "Beautiful woman, bubblegum pink, lemon yellow, minty blue, futuristic, high-detail, epic composition, watercolor."


Could we try to make use of non-human characters? Since it's the official documentation, I am slightly concerned about the reception of this.

WDYT? @yiyixuxu @stevhliu

Let's replace it with a cute cat.

Yeah works for me. Could you generate the results for those as well?

For animagine-xl-3.0 and IP-Adapter, the human characters is performing better. Do we need to forcefully replace these?

I will let @stevhliu comment further here.

Should be ok for this example 👍

sayakpaul · 2024-03-09T06:28:34Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+
+## TCD-LoRA is Versatile for Community Models
+
+As mentioned above, the TCD-LoRA is versatile for community models and plugins. We initially demonstrate the results with  a community fine-tuned base model [animagine-xl-3.0](https://huggingface.co/cagliostrolab/animagine-xl-3.0).


Suggested change

As mentioned above, the TCD-LoRA is versatile for community models and plugins. We initially demonstrate the results with a community fine-tuned base model [animagine-xl-3.0](https://huggingface.co/cagliostrolab/animagine-xl-3.0).

As mentioned above, the TCD-LoRA is versatile for community models and plugins. To test-drive this, load a community fine-tuned base model [animagine-xl-3.0](https://huggingface.co/cagliostrolab/animagine-xl-3.0).

sayakpaul · 2024-03-09T06:29:32Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+
+![](https://github.com/jabir-zheng/TCD/raw/main/assets/animagine_xl.png)
+
+Furthermore, TCD-LoRA also support other style LoRA. Here is an example with [Papercut](https://huggingface.co/TheLastBen/Papercut_SDXL). To learn more about how to combine LoRAs, refer to [this guide](https://huggingface.co/docs/diffusers/tutorials/using_peft_for_inference#combine-multiple-adapters).


Suggested change

Furthermore, TCD-LoRA also support other style LoRA. Here is an example with [Papercut](https://huggingface.co/TheLastBen/Papercut_SDXL). To learn more about how to combine LoRAs, refer to [this guide](https://huggingface.co/docs/diffusers/tutorials/using_peft_for_inference#combine-multiple-adapters).

Furthermore, TCD-LoRA also supports LoRAs corresponding to other styles. Below is an example with [Papercut](https://huggingface.co/TheLastBen/Papercut_SDXL). To learn more about how to combine LoRAs, refer to [this guide](https://huggingface.co/docs/diffusers/tutorials/using_peft_for_inference#combine-multiple-adapters).

sayakpaul · 2024-03-09T06:30:03Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+
+## Compatibility with ControlNet
+
+For this example, we'll keep using the SDXL model and the TCD-LoRA for SDXL with depth and canny ControlNet.


Suggested change

For this example, we'll keep using the SDXL model and the TCD-LoRA for SDXL with depth and canny ControlNet.

For this example, you'll keep using the SDXL model and the TCD-LoRA for SDXL with depth and canny ControlNets.

sayakpaul

Very useful guide. Thanks for writing it in such details!

Let's wait for @stevhliu's reviews before applying suggestions.

HuggingFaceDocBuilderDev · 2024-03-09T06:38:47Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

stevhliu

Super impressive results! My main comments are about how to organize and structure the guide so that it flows better without feeling too repetitive. Great job again! 👍

docs/source/en/using-diffusers/inference_with_tcd_lora.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

stevhliu

Just need to move the inpainting section and then we should be ready to merge :)

stevhliu · 2024-03-12T17:59:44Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+).images[0]
+```
+
+![](https://github.com/jabir-zheng/TCD/raw/main/assets/demo_image.png)


Suggested change

![](https://github.com/jabir-zheng/TCD/raw/main/assets/demo_image.png)

![](https://github.com/jabir-zheng/TCD/raw/main/assets/demo_image.png)

</hfoption>

<hfoption id="inpainting">

move inpainting content here

</hfoption>

</hfoptions>

stevhliu · 2024-03-12T18:00:04Z

docs/source/en/using-diffusers/inference_with_tcd_lora.md

+![](https://github.com/jabir-zheng/TCD/raw/main/assets/styled_lora.png)
+
+
+## Inpainting with TCD 


Looks like this section hasn't been moved yet! Take a look at my suggestion above :)

sayakpaul

Thanks for your great contributions! @stevhliu could you review once and merge?

stevhliu

LGTM, thanks again for your awesome contribution! 🤗

mhh0318 · 2024-03-13T16:32:22Z

Thanks for your great efforts and assistance, Steven and Sayak!

add tcd intro

28a9967

yiyixuxu requested review from sayakpaul and stevhliu March 8, 2024 22:38

sayakpaul reviewed Mar 9, 2024

View reviewed changes

Merge branch 'main' into tcd_intro

e0d4eac

resolve repos

50c8944

stevhliu reviewed Mar 11, 2024

View reviewed changes

mhh0318 and others added 3 commits March 12, 2024 11:19

Apply suggestions from code review

f3c8090

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

revise NFEs related

3298376

Merge branch 'main' into tcd_intro

432e611

stevhliu reviewed Mar 12, 2024

View reviewed changes

mhh0318 and others added 3 commits March 13, 2024 03:50

change inpainting location

46450f6

Merge branch 'main' into tcd_intro

e5e9e7d

Merge branch 'main' into tcd_intro

ca40084

sayakpaul approved these changes Mar 13, 2024

View reviewed changes

stevhliu approved these changes Mar 13, 2024

View reviewed changes

stevhliu merged commit b300517 into huggingface:main Mar 13, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Intro page of TCD #7259

Add Intro page of TCD #7259

mhh0318 commented Mar 8, 2024 •

edited

Loading

sayakpaul Mar 9, 2024

sayakpaul Mar 9, 2024

sayakpaul Mar 9, 2024

sayakpaul Mar 9, 2024

sayakpaul Mar 9, 2024

sayakpaul Mar 9, 2024 •

edited

Loading

sayakpaul Mar 9, 2024

sayakpaul Mar 9, 2024

mhh0318 Mar 9, 2024

sayakpaul Mar 9, 2024

mhh0318 Mar 12, 2024

sayakpaul Mar 12, 2024

stevhliu Mar 12, 2024

sayakpaul Mar 9, 2024

sayakpaul Mar 9, 2024

sayakpaul Mar 9, 2024

sayakpaul left a comment

HuggingFaceDocBuilderDev commented Mar 9, 2024

stevhliu left a comment

stevhliu left a comment

stevhliu Mar 12, 2024

stevhliu Mar 12, 2024

sayakpaul left a comment

stevhliu left a comment

mhh0318 commented Mar 13, 2024


		From the [Official Project Page](https://mhh0318.github.io/tcd/), the major merit of TCD can be outlined as follows:

		> *Better than Teacher:* TCD maintains superior generative quality at both low NFEs and high NFEs, even exceeding the performance of DPM-Solver++(2S) with origin SDXL. It is worth noting that there is no additional discriminator or LPIPS supervision included during training.


		For more technical details of TCD, please refer to [the paper](https://arxiv.org/abs/2402.19159).

		Trajectory consistency distillation can directly place on top of a pre-trained diffusion model as a LoRA module. Such LoRA can be identified as a versatile acceleration module applicable to different fine-tuned models or LoRAs sharing the same base model without the need for additional training.


		TCD-LoRAs are available for [stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5), [stable-diffusion-2-1-base](https://huggingface.co/stabilityai/stable-diffusion-2-1-base), and [stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0).

		The corresponding checkpoints can be found at [TCD-SD15](https://huggingface.co/h1t/TCD-SD15-LoRA), [TCD-SD21-base](https://huggingface.co/h1t/TCD-SD21-base-LoRA) and [TCD-SDXL](https://huggingface.co/h1t/TCD-SDXL-LoRA), separately.

	TCD-LoRA can be considered an advanced method compared with [LCM-LoRA](https://latent-consistency-models.github.io/). The guide of TCD-LoRA workflow is:
	TCD-LoRA can be considered an advanced method compared with [LCM-LoRA](https://huggingface.co/docs/diffusers/main/en/using-diffusers/inference_with_lcm_lora). The main parts of the TCD-LoRA workflow are as follows:


		## TCD-LoRA is Versatile for Community Models

		As mentioned above, the TCD-LoRA is versatile for community models and plugins. We initially demonstrate the results with a community fine-tuned base model [animagine-xl-3.0](https://huggingface.co/cagliostrolab/animagine-xl-3.0).


		![](https://github.com/jabir-zheng/TCD/raw/main/assets/animagine_xl.png)

		Furthermore, TCD-LoRA also support other style LoRA. Here is an example with [Papercut](https://huggingface.co/TheLastBen/Papercut_SDXL). To learn more about how to combine LoRAs, refer to [this guide](https://huggingface.co/docs/diffusers/tutorials/using_peft_for_inference#combine-multiple-adapters).

	Furthermore, TCD-LoRA also support other style LoRA. Here is an example with [Papercut](https://huggingface.co/TheLastBen/Papercut_SDXL). To learn more about how to combine LoRAs, refer to [this guide](https://huggingface.co/docs/diffusers/tutorials/using_peft_for_inference#combine-multiple-adapters).
	Furthermore, TCD-LoRA also supports LoRAs corresponding to other styles. Below is an example with [Papercut](https://huggingface.co/TheLastBen/Papercut_SDXL). To learn more about how to combine LoRAs, refer to [this guide](https://huggingface.co/docs/diffusers/tutorials/using_peft_for_inference#combine-multiple-adapters).


		## Compatibility with ControlNet

		For this example, we'll keep using the SDXL model and the TCD-LoRA for SDXL with depth and canny ControlNet.

	For this example, we'll keep using the SDXL model and the TCD-LoRA for SDXL with depth and canny ControlNet.
	For this example, you'll keep using the SDXL model and the TCD-LoRA for SDXL with depth and canny ControlNets.

-![](https://github.com/jabir-zheng/TCD/raw/main/assets/demo_image.png)
+![](https://github.com/jabir-zheng/TCD/raw/main/assets/demo_image.png)
+</hfoption>
+<hfoption id="inpainting">
+move inpainting content here
+</hfoption>
+</hfoptions>

		![](https://github.com/jabir-zheng/TCD/raw/main/assets/styled_lora.png)


		## Inpainting with TCD

Add Intro page of TCD #7259

Add Intro page of TCD #7259

Conversation

mhh0318 commented Mar 8, 2024 • edited Loading

What does this PR do?

Before submitting

Who can review?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul Mar 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Mar 9, 2024

stevhliu left a comment

Choose a reason for hiding this comment

stevhliu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

stevhliu left a comment

Choose a reason for hiding this comment

mhh0318 commented Mar 13, 2024

mhh0318 commented Mar 8, 2024 •

edited

Loading

sayakpaul Mar 9, 2024 •

edited

Loading