Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Composable diffusion to community pipeline examples #951

Merged
merged 9 commits into from
Oct 25, 2022
49 changes: 49 additions & 0 deletions examples/community/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ If a community doesn't work as expected, please open an issue and ping the autho
| Long Prompt Weighting Stable Diffusion | **One** Stable Diffusion Pipeline without tokens length limit, and support parsing weighting in prompt. | [Long Prompt Weighting Stable Diffusion](#long-prompt-weighting-stable-diffusion) | - | [SkyTNT](https://github.com/SkyTNT) |
| Speech to Image | Using automatic-speech-recognition to transcribe text and Stable Diffusion to generate images | [Speech to Image](#speech-to-image) | - | [Mikail Duzenli](https://github.com/MikailINTech)
| Wild Card Stable Diffusion | Stable Diffusion Pipeline that supports prompts that contain wildcard terms (indicated by surrounding double underscores), with values instantiated randomly from a corresponding txt file or a dictionary of possible values | [Wildcard Stable Diffusion](#wildcard-stable-diffusion) | - | [Shyam Sudhakaran](https://github.com/shyamsn97) |
| Composable Stable Diffusion| Stable Diffusion Pipeline that supports prompts that contain "|" in prompts (as an AND condition) and weights (separated by "|" as well) to positively / negatively weight prompts. | [Composable Stable Diffusion](#composable-stable-diffusion) | - | [Mark Rich](https://github.com/MarkRich) |


To load a custom pipeline you just need to pass the `custom_pipeline` argument to `DiffusionPipeline`, as one of the files in `diffusers/examples/community`. Feel free to send a PR with your own pipelines, we will merge them quickly.
```py
Expand Down Expand Up @@ -322,3 +324,50 @@ out = pipe(
wildcard_files=["object.txt", "animal.txt"],
num_prompt_samples=1
)
```


### Composable Stable diffusion

```python
import torch as th
import numpy as np
import torchvision.utils as tvu
from diffusers import DiffusionPipeline

has_cuda = th.cuda.is_available()
device = th.device('cpu' if not has_cuda else 'cuda')

pipe = DiffusionPipeline.from_pretrained(
"CompVis/stable-diffusion-v1-4",
use_auth_token=True,
custom_pipeline="composable_stable_diffusion",
).to(device)


def dummy(images, **kwargs):
return images, False

pipe.safety_checker = dummy

images = []
generator = th.Generator("cuda").manual_seed(0)

seed = 0
prompt = "a forest | a camel"
weights = " 1 | 1" # Equal weight to each prompt. Cna be negative

images = []
for i in range(4):
res = pipe(
prompt,
guidance_scale=7.5,
num_inference_steps=50,
weights=weights,
generator=generator)
image = res.images[0]
images.append(image)

for i, img in enumerate(images):
img.save(f"./composable_diffusion/image_{i}.png")
```
Loading