Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IP-Adapter support for StableDiffusionXLControlNetInpaintPipeline #6941

Merged
merged 3 commits into from
Feb 19, 2024

Conversation

rootonchair
Copy link
Contributor

What does this PR do?

Fixes #6866

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu
Copy link
Collaborator

thanks for the PR!
I think we need to make updates based on #6868

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@rootonchair
Copy link
Contributor Author

@yiyixuxu I have updated accordingly

Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@yiyixuxu yiyixuxu requested a review from sayakpaul February 15, 2024 08:37
Copy link
Member

@sayakpaul sayakpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovely! Do we have any example code snippet that shows the results with this modification?

@nkaewam
Copy link
Contributor

nkaewam commented Feb 19, 2024

Hi, I would like to request to extend IPAdapterMixin to the StableDiffusionXLControlNetInpaintPipeline class so it could inherit load_ip_adapter property

class StableDiffusionXLControlNetInpaintPipeline(
    DiffusionPipeline, 
    StableDiffusionXLLoraLoaderMixin, 
    FromSingleFileMixin, 
+    IPAdapterMixin
):

r"""
    Pipeline for text-to-image generation using Stable Diffusion XL.

    This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the
    library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

    The pipeline also inherits the following loading methods:
        - [`~loaders.StableDiffusionXLLoraLoaderMixin.load_lora_weights`] for loading LoRA weights
        - [`~loaders.StableDiffusionXLLoraLoaderMixin.save_lora_weights`] for saving LoRA weights
        - [`~loaders.FromSingleFileMixin.from_single_file`] for loading `.ckpt` files
+        - [`~loaders.IPAdapterMixin.load_ip_adapter`] for loading IP Adapters
...
"""

For code snippet that shows the result as @sayakpaul requested, it looks kinda like this after the modification.

image_encoder = CLIPVisionModelWithProjection.from_pretrained(
            "h94/IP-Adapter",
            subfolder="models/image_encoder",
            torch_dtype=torch.float16,
            cache_dir=CACHE_DIR
).to(device)


pipe = StableDiffusionXLControlNetInpaintPipeline.from_pretrained(
            weight_path,
            cache_dir=CACHE_DIR,
            vae=sd_vae,
            controlnet=controlnet, 
            image_encoder=image_encoder,
            torch_dtype=torch.float16
).to(device)
        
pipe.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name="ip-adapter-plus_sdxl_vit-h.safetensors", cache_dir=CACHE_DIR)

images = pipe(
            image=room_image,
            ip_adapter_image=style_image, 
            control_image=control_image, 
            mask_image=mask_image,
            ...other-params
).images

@yiyixuxu
Copy link
Collaborator

oh thanks!! @tontan2545
I read your comment too fast and I thought it was a code example provided by the OP

thanks for opening a PR to fix this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

IP-Adapter support for StableDiffusionXLControlNetInpaintPipeline
5 participants