[Community] PromptDiffusion Pipeline #6752
Conversation
Added PromptDiffusionControlNetModel
Added PromptDiffusionControlNetModel
Added Prompt Diffusion pipeline.
Added download_promptdiffusion_from_original_ckpt, convert_promptdiffusion_checkpoint
src/diffusers/__init__.py
Outdated
| @@ -80,6 +80,7 @@ | |||
| "AutoencoderTiny", | |||
| "ConsistencyDecoderVAE", | |||
| "ControlNetModel", | |||
| "PromptDiffusionControlNetModel", | |||
There was a problem hiding this comment.
Can we maybe move this pipeline to the research_projects as it has a new model component too?
| @@ -0,0 +1,401 @@ | |||
| # Copyright 2023 The HuggingFace Team. All rights reserved. | |||
There was a problem hiding this comment.
This should probably go into the community folder
|
|
||
| # load prompt diffusion control net and prompt diffusion | ||
|
|
||
| controlnet = PromptDiffusionControlNetModel.from_pretrained("path-to-promptdiffusion-controlnet", torch_dtype=torch.float16) |
There was a problem hiding this comment.
I think it makes sense to first talk about how this checkpoint was obtained i.e., an example command of running the conversion.
There was a problem hiding this comment.
Could you provide the checkpoint path from the Hub directly? I.e., it could make sense to directly load it from the Hub? A few options:
- Host it under your HF profile while clearly providing the citations to the original model.
- Reach out to the author to see if they'd be willing to have the converted checkpoint under their HF profile.
Let me know what you think.
There was a problem hiding this comment.
Zhendong-Wang/Prompt-Diffusion#12 No official release date, so mine has to suffice.
|
|
||
| controlnet = PromptDiffusionControlNetModel.from_pretrained("path-to-promptdiffusion-controlnet", torch_dtype=torch.float16) | ||
| model_id = "path-to-model" | ||
| pipe = DiffusionPipeline.from_pretrained(model_id, controlnet=controlnet, torch_dtype=torch.float16, variant="fp16", custom_pipeline="path-to-pipeline_prompt_diffusion") |
There was a problem hiding this comment.
Shouldn't this be PromptDiffusionPipeline?
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
This PR looks solid to me.
I just left one comment: https://github.com/huggingface/diffusers/pull/6752/files#r1483236580.
|
nice work! @iczaw we can merge this once this is addressed :) |
|
Thanks for your contributions. |
What does this PR do?
Adds the SD implementation of Prompt Diffusion
Project page: https://zhendong-wang.github.io/prompt-diffusion.github.io/
Paper: https://arxiv.org/abs/2305.01115
Code: https://github.com/Zhendong-Wang/Prompt-Diffusion
Models: https://huggingface.co/zhendongw/prompt-diffusion
Fixes #6214
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@sayakpaul @patrickvonplaten