Add ControlNet Pipeline #585

nngokhale · 2023-12-06T13:44:54Z

What does this PR do?

Adds ControlNet pipeline.
The port is based on GaudiStableDiffusionPipeline.

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

examples/controlnet/example_canny2image.py

optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py

libinta

can you rebase to latest?

examples/controlnet/example_canny2image.py

nngokhale · 2023-12-16T06:14:58Z

can you rebase to latest?

Done

examples/controlnet/README.md

libinta · 2023-12-21T23:25:50Z

optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py

+from diffusers.utils.torch_utils import is_compiled_module
+from ....utils import speed_metrics, HabanaProfile
+from ..pipeline_utils import GaudiDiffusionPipeline
+


please use #611 as guideline to port this file. also move the pipeline_controlnet.py under pipelines

Yes, there should be things that are not necessary anymore. I'm going to try to point them out below.

examples/controlnet/text_to_image_generation_canny.py

libinta · 2023-12-21T23:35:16Z

optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py

+
+        # Workaround for Synapse 1.11 for full bf16
+        if bf16_full_eval:
+            unet.conv_in.float()


remove line141-142

@libinta We don't need this anymore?

optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py

libinta

also, please check https://github.com/huggingface/optimum-habana/blob/main/tests/test_diffusers.py to add test case for regression protection

HuggingFaceDocBuilderDev · 2024-01-03T10:27:38Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

regisss

There are many comments related to #611 that should be quick to address.

Can you also run the following to make sure the code format is right please?

pip install --upgrade ruff
make style

examples/controlnet/text_to_image_generation_canny.py

examples/controlnet/README.md

regisss · 2024-01-03T10:26:12Z

optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py

+from diffusers.utils.torch_utils import is_compiled_module
+from ....utils import speed_metrics, HabanaProfile
+from ..pipeline_utils import GaudiDiffusionPipeline
+


Yes, there should be things that are not necessary anymore. I'm going to try to point them out below.

optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py

regisss · 2024-01-15T20:26:23Z

@nngokhale Can you integrate the new ControlNet example into the existing Stable Diffusion example please? It will be much easier to maintain as there is a lot of duplicated code.

nngokhale · 2024-01-16T12:41:48Z

@nngokhale Can you integrate the new ControlNet example into the existing Stable Diffusion example please? It will be much easier to maintain as there is a lot of duplicated code.
Yes, working on it and all other comments. Sorry for the delay, lost access to gaudi systems for couple of weeks.

nngokhale · 2024-01-16T20:01:14Z

There are many comments related to #611 that should be quick to address.

Can you also run the following to make sure the code format is right please?
pip install --upgrade ruff
make style

Style fixed

nngokhale · 2024-01-18T19:43:47Z

also, please check https://github.com/huggingface/optimum-habana/blob/main/tests/test_diffusers.py to add test case for regression protection

Added tests for ControlNet pipeline.

nngokhale · 2024-01-19T18:52:38Z

All comments / issues have been resolved, please review.

Fix scheduler.step call to latest

Add hpu graph for controlnet Fix Code style

regisss

I left a few comments.

Can you also run make style please?

Do you have any throughput number to share?

optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py

tests/test_diffusers.py

examples/stable-diffusion/text_to_image_generation.py

nngokhale · 2024-01-24T17:15:01Z

I left a few comments.

Can you also run make style please?

Do you have any throughput number to share?

Resolved latest comments, ran 'make style'.

Here are my throughput numbers:

For 512x512, 1 prompt, 50 inf steps, 20 generations per prompt, 4 samples per batch, BF16 + HPU Graphs,

on Gaudi2 HL-SMI: hl-1.13.0-rc-fw-47.0.0.0, Habana Driver: 1.13.0-ee32e42, Installed Habana Torch plug-in: 1.13.0.463, Installed Optimum-Habana: 1.10.0.dev0

GaudiStableDiffusionControlNetPipeline with lllyasviel/sd-controlnet-canny and runwayml/stable-diffusion-v1-5
Speed metrics: {'generation_runtime': 112.9972, 'generation_samples_per_second': 0.693, 'generation_steps_per_second': 0.289}

For comparison
GaudiStableDiffusionPipeline with runwayml/stable-diffusion-v1-5
Speed metrics: {'generation_runtime': 83.9506, 'generation_samples_per_second': 0.975, 'generation_steps_per_second': 0.406}

regisss

LGTM!

nngokhale requested a review from regisss as a code owner December 6, 2023 13:44

libinta reviewed Dec 15, 2023

View reviewed changes

examples/controlnet/example_canny2image.py Outdated Show resolved Hide resolved

libinta reviewed Dec 15, 2023

View reviewed changes

examples/controlnet/example_canny2image.py Outdated Show resolved Hide resolved

libinta reviewed Dec 15, 2023

View reviewed changes

examples/controlnet/example_canny2image.py Outdated Show resolved Hide resolved

libinta reviewed Dec 15, 2023

View reviewed changes

optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated Show resolved Hide resolved

libinta reviewed Dec 15, 2023

View reviewed changes

optimum/habana/diffusers/pipelines/controlnet/pipeline_controlnet.py Outdated Show resolved Hide resolved

libinta reviewed Dec 15, 2023

View reviewed changes

vidyasiv reviewed Dec 15, 2023

View reviewed changes

examples/controlnet/example_canny2image.py Outdated Show resolved Hide resolved

nngokhale force-pushed the controlnet branch 2 times, most recently from e22fb1d to 56da112 Compare December 16, 2023 06:06

nngokhale force-pushed the controlnet branch 6 times, most recently from cf5d5a4 to e87c4ed Compare December 20, 2023 07:22

nngokhale requested a review from libinta December 21, 2023 05:59

libinta reviewed Dec 21, 2023

View reviewed changes

regisss reviewed Jan 3, 2024

View reviewed changes

nngokhale force-pushed the controlnet branch 2 times, most recently from 200c908 to e0dab26 Compare January 12, 2024 14:52

nngokhale force-pushed the controlnet branch from c0d82fb to 3f77934 Compare January 17, 2024 15:21

nngokhale force-pushed the controlnet branch from 1d65025 to cbaab7e Compare January 19, 2024 18:40

nngokhale requested a review from regisss January 19, 2024 18:51

vidyasiv mentioned this pull request Jan 19, 2024

Controlnet training #650

Merged

3 tasks

nngokhale added 10 commits January 22, 2024 23:23

Add ControlNet

6708521

Add Profiler and requirements.txt

d64b2ec

Add t2i sample

b23d6f8

Fix bf16 support

2754715

Fix scheduler.step call to latest

Add inheritance in controlnet pipeline

ebfa04b

Add hpu graph for controlnet Fix Code style

Move controlnet example and readme to stable diffusion

92a5c4d

Add tests for ControlNet

54c5ff3

Remove duplicate code, reuse from GaudiStableDiffusionPipeline

21b1a41

Add more ControlNet examples and MultiControlNet tests

13b99ac

Add preprocessing control to example

256c7bb

nngokhale force-pushed the controlnet branch from f39d2da to 256c7bb Compare January 22, 2024 18:04

nngokhale requested a review from libinta January 22, 2024 18:08

regisss reviewed Jan 23, 2024

View reviewed changes

Fix seed, guess mode and style

b3b5f3b

Add SD2 example, conditional import cv2

e45d550

nngokhale requested a review from regisss January 25, 2024 10:53

regisss added the run-test Run CI for PRs from external contributors label Jan 30, 2024

regisss approved these changes Jan 30, 2024

View reviewed changes

regisss merged commit 937a411 into huggingface:main Jan 30, 2024
11 of 12 checks passed

regisss pushed a commit that referenced this pull request Feb 12, 2024

Add ControlNet Pipeline (#585)

58c4ea1

jychen21 pushed a commit to jychen21/optimum-habana that referenced this pull request Feb 27, 2024

Add ControlNet Pipeline (huggingface#585)

7dcfab9

dudilester pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Feb 29, 2024

Add ControlNet Pipeline (huggingface#585)

6ebb46e

dudilester pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Feb 29, 2024

Add ControlNet Pipeline (huggingface#585)

829a2d8

HolyFalafel pushed a commit to HabanaAI/optimum-habana-fork that referenced this pull request Mar 11, 2024

Add ControlNet Pipeline (huggingface#585)

23da729

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ControlNet Pipeline #585

Add ControlNet Pipeline #585

nngokhale commented Dec 6, 2023 •

edited

Loading

libinta left a comment

nngokhale commented Dec 16, 2023

libinta Dec 21, 2023

regisss Jan 3, 2024

libinta Dec 21, 2023

regisss Jan 3, 2024

libinta left a comment

HuggingFaceDocBuilderDev commented Jan 3, 2024

regisss left a comment

regisss Jan 3, 2024

regisss commented Jan 15, 2024

nngokhale commented Jan 16, 2024

nngokhale commented Jan 16, 2024

nngokhale commented Jan 18, 2024

nngokhale commented Jan 19, 2024

regisss left a comment

nngokhale commented Jan 24, 2024

regisss left a comment

Add ControlNet Pipeline #585

Add ControlNet Pipeline #585

Conversation

nngokhale commented Dec 6, 2023 • edited Loading

What does this PR do?

Before submitting

libinta left a comment

Choose a reason for hiding this comment

nngokhale commented Dec 16, 2023

libinta Dec 21, 2023

Choose a reason for hiding this comment

regisss Jan 3, 2024

Choose a reason for hiding this comment

libinta Dec 21, 2023

Choose a reason for hiding this comment

regisss Jan 3, 2024

Choose a reason for hiding this comment

libinta left a comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Jan 3, 2024

regisss left a comment

Choose a reason for hiding this comment

regisss Jan 3, 2024

Choose a reason for hiding this comment

regisss commented Jan 15, 2024

nngokhale commented Jan 16, 2024

nngokhale commented Jan 16, 2024

nngokhale commented Jan 18, 2024

nngokhale commented Jan 19, 2024

regisss left a comment

Choose a reason for hiding this comment

nngokhale commented Jan 24, 2024

regisss left a comment

Choose a reason for hiding this comment

nngokhale commented Dec 6, 2023 •

edited

Loading