-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support of ip-adapter to the StableDiffusionControlNetInpaintPipeline #5887
Support of ip-adapter to the StableDiffusionControlNetInpaintPipeline #5887
Conversation
…ges are similar to those in pipeline_controlnet
…ng image_encoder: None
The documentation is not available anymore as the PR was closed or merged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thank you!
src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py
Outdated
Show resolved
Hide resolved
Co-authored-by: YiYi Xu <yixu310@gmail.com>
@@ -342,6 +344,7 @@ def init_weights(m): | |||
"tokenizer": tokenizer, | |||
"safety_checker": None, | |||
"feature_extractor": None, | |||
"image_encoder": None, | |||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's also add one test related to IP Adapters here :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yiyixuxu wdyt?
Hi @juancopi81 @yiyixuxu - Can we expect this PR to support StableDiffusionXLControlNetInpaintPipeline? |
Hi @wwirving, not really, this PR is only for the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks! looks great :)
can we add a test like @patrickvonplaten requested?
Sure @patrickvonplaten and @yiyixuxu I'll work on that. Just to be sure, you are expecting something like this: class MultiControlNetInpaintPipelineFastTests(
PipelineTesterMixin, PipelineKarrasSchedulerTesterMixin, unittest.TestCase
):
pipeline_class = StableDiffusionControlNetInpaintPipeline
params = TEXT_GUIDED_IMAGE_INPAINTING_PARAMS
batch_params = TEXT_GUIDED_IMAGE_INPAINTING_BATCH_PARAMS
def get_dummy_components(self):
torch.manual_seed(0)
unet = ...
### NEW CODE ####
torch.manual_seed(0)
image_encoder_config = CLIPVisionConfig(
image_size=64,
num_channels=3,
hidden_size=32,
projection_dim=32,
num_hidden_layers=2,
num_attention_heads=4,
intermediate_size=37,
dropout=0.1,
attention_dropout=0.1,
initializer_range=0.02,
scope=None,
)
image_encoder = CLIPVisionModelWithProjection(config=image_encoder_config)
torch.manual_seed(0)
feature_extractor_config = {
"image_size": 64,
"num_channels": 3,
"do_resize": True,
"size": {"shortest_edge": 20},
"do_center_crop": True,
"crop_size": {"height": 64, "width": 64},
"do_normalize": True,
"image_mean": [0.48145466, 0.4578275, 0.40821073],
"image_std": [0.26862954, 0.26130258, 0.27577711],
"do_convert_rgb": True
}
feature_extractor = CLIPImageProcessor(**feature_extractor_config)
components = {
"unet": unet,
"controlnet": controlnet,
"scheduler": scheduler,
"vae": vae,
"text_encoder": text_encoder,
"tokenizer": tokenizer,
"safety_checker": None,
"feature_extractor": feature_extractor,
"image_encoder": image_encoder,
}
return components And something else? Maybe a specific test of the image_adapter? like: def test_with_ip_adapter_image_encoder(self):
components = self.get_dummy_components()
pipe = self.pipeline_class(**components)
pipe.to(torch_device)
inputs = self.get_dummy_inputs(torch_device)
pipe.set_ip_adapter_scale(1.0)
output_1 = pipe(**inputs)[0]
inputs = self.get_dummy_inputs(torch_device)
pipe.set_ip_adapter_scale(0.0)
output_2 = pipe(**inputs)[0]
# make sure that all outputs are different
assert np.sum(np.abs(output_1 - output_2)) > 1e-3
self.assertEqual(output_2.shape, (1, 64, 64, 3)) Right now this test:
fails 😢, but I could take a look into that if that is what you expect, so I wanted to ask first. EDIT: I changed the code so the feature_extractor gets also initialized in the get_dummy_components function. |
Actually, I think he should be good here without adapting all the controlnet tests. We also didn't add it here: #5713 |
…huggingface#5887) * Change pipeline_controlnet_inpaint.py to add ip-adapter support. Changes are similar to those in pipeline_controlnet * Change tests for the StableDiffusionControlNetInpaintPipeline by adding image_encoder: None * Update src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py Co-authored-by: YiYi Xu <yixu310@gmail.com> --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>
…huggingface#5887) * Change pipeline_controlnet_inpaint.py to add ip-adapter support. Changes are similar to those in pipeline_controlnet * Change tests for the StableDiffusionControlNetInpaintPipeline by adding image_encoder: None * Update src/diffusers/pipelines/controlnet/pipeline_controlnet_inpaint.py Co-authored-by: YiYi Xu <yixu310@gmail.com> --------- Co-authored-by: YiYi Xu <yixu310@gmail.com>
What does this PR do?
This PR adds support of the ip-adapter to the StableDiffusionControlNetInpaintPipeline. The ip-adapter was added in #5713 and help was asked in #5884 I think it is very cool to have this support (see example below).
This PR refers to #5884
This is how it works:
Initial image | Mask | ip-adapter image | canny image | output
![stablediffusioncontrolnetinpaintpipeline](https://private-user-images.githubusercontent.com/4313860/284987357-29bd4bbc-fefd-4d75-a704-005f9d060525.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkyNjIxMzYsIm5iZiI6MTczOTI2MTgzNiwicGF0aCI6Ii80MzEzODYwLzI4NDk4NzM1Ny0yOWJkNGJiYy1mZWZkLTRkNzUtYTcwNC0wMDVmOWQwNjA1MjUucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDIxMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTAyMTFUMDgxNzE2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NGNkMGEwY2UzNTNlZWZkYTRhZDZiYzc0ZTIxZTFiYmJiN2I1ZjQzMjQwOTQ1MDYyYjNjM2M1NTdlNmI0YmRkOCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.2W2qQf8OzKcg6jhBrBIupy52XnQq1NX03mU4ex8IAw4)
Final output:
![output_final](https://private-user-images.githubusercontent.com/4313860/284987569-7238115a-71a7-44bc-801d-bbe9994f54e0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzkyNjIxMzYsIm5iZiI6MTczOTI2MTgzNiwicGF0aCI6Ii80MzEzODYwLzI4NDk4NzU2OS03MjM4MTE1YS03MWE3LTQ0YmMtODAxZC1iYmU5OTk0ZjU0ZTAucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI1MDIxMSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNTAyMTFUMDgxNzE2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MzlmYWI4NTk2YzJkNWM1Y2IzNTViMDNjMWJmZTQ1OWE5NGVkYmM3YWNjNzE3ZDYxOWM4ZDhlMmJhMmU4NjU2NSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.en2JXZpe8K8BCnDwlYtv6DBCAiDND4hFHaCnBbNYV-M)
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
FAILED tests/pipelines/controlnet/test_controlnet_inpaint.py::ControlNetInpaintPipelineFastTests::test_save_load_local - AssertionError: 0.0016172826 not less than 0.0005
FAILED tests/pipelines/controlnet/test_controlnet_inpaint.py::ControlNetSimpleInpaintPipelineFastTests::test_save_load_local - AssertionError: 0.00053209066 not less than 0.0005
FAILED tests/pipelines/controlnet/test_controlnet_inpaint.py::MultiControlNetInpaintPipelineFastTests::test_save_load_local - AssertionError: 0.00051498413 not less than 0.0005
Who can review?
@yiyixuxu Hi @yiyixuxu please let me know any comments 😃 I hope it is ok, I am happy to change anything you need.