Wildcard stable diffusion pipeline

[shyamsn97]

Wildcard Stable Diffusion pipeline

Part of #872 and bigger story #841

Following the great examples from https://github.com/jtkelm2/stable-diffusion-webui-1/blob/master/scripts/wildcards.py and https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts#wildcards, here's a minimal implementation that allows for users to add "wildcards", denoted by __wildcard__ to prompts that are used as placeholders for randomly sampled values given by either a dictionary or a .txt file. For example:

Say we have a prompt:

prompt = "__animal__ sitting on a __object__ wearing a __clothing__"

We can then define possible values to be sampled for animal, object, and clothing. These can either be from a .txt with the same name as the category, ex animal.txt, with contents like:

dog
cat
mouse

The possible values can also be defined / combined by using a dictionary like: {"animal":["dog", "cat", mouse"]}.

The actual pipeline works just like StableDiffusionPipeline, except the __call__ method takes in:

wildcard_files: list of file paths for wild card replacement
wildcard_option_dict: dict with key as wildcard and values as a list of possible replacements
num_prompt_samples: number of prompts to sample, uniformly sampling wildcards

A full example:

pipe = WildcardStableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
    revision="fp16",
    torch_dtype=torch.float16,
)
prompt = "__animal__ sitting on a __object__ wearing a __clothing__"
out = pipe(
    prompt,
    wildcard_option_dict={
        "clothing":["hat", "shirt", "scarf", "beret"]
    },
    wildcard_files=["object.txt", "animal.txt"],
    num_prompt_samples=1
)

Will need to add some additional documentation, but adding this as a start. Hopefully the implementation is straightforward and easy to understand, and let me know your thoughts on how I can improve this 😄

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

maybe delete this code? :-)

[patrickvonplaten]

Hey @shyamsn97,

The PR looks great :-)
One final thing then I'd be more than happy to merge.

To increase the easy-of-use and discover-ability of your pipeline, could you add one line to this table: https://github.com/huggingface/diffusers/tree/main/examples/community#community-examples in your PR

and also add one code example in a new markdown section here: https://github.com/huggingface/diffusers/tree/main/examples/community#example-usages

Also happy to do it for you if you could quickly post one usage example in this PR :-)

[GreenLandisaLie]

I'm no dev so maybe I'm misunderstanding something here but if I'm not - why make a pipeline for this?
I'm pretty sure this can easily be achieved through a simple single function that loops through every .txt(?) file inside a 'wildtags' folder and create a dict where the keys are the filenames and the values are a list of strings - each being a line of that file. Then all it would take would be to pass on the prompt to that function and replace every 'filename' with a random string from their respective list, no?
I mean I've literally been doing this for myself because its such an obvious, easy and useful thing to do plus I've even added the ability to force include/exclude certain words based on a given string. For instance, lets say you want to replace artist with a random artist but you dont want 'Greg Rutkowski' (which is on the list), so I just do something like:

prompt = replaceWildtags(prompt, "-rutkowski") - and because the first char is '-' the result should be anything that does not have "rutkowski" unless no such string exists. This ofc would affect all wildtags - not just artists - but there are easy solutions for that too.

Feel free to ignore this - I'm probably the one misunderstanding here.

[patrickvonplaten]

Hey @GreenLandisaLie,

No, this makes total sense! If you prefer just adding a function yourself before the pipeline that's great :-) For people that would like to have everything in one place such a community pipeline could still be useful.
In general, the idea of community pipeline is to let the community share very freely a pipeline example with everybody else by adding it here, so we don't have hard requirements on how the community pipelines should look like

[GreenLandisaLie]

In general, the idea of community pipeline is to let the community share very freely a pipeline example with everybody else by adding it here, so we don't have hard requirements on how the community pipelines should look like

Makes sense actually. I did notice a bunch of pipelines (through your link) that I wasn't even aware of and came to that realization soon after posting that comment. Decided to leave it there because the ability to force include/exclude certain strings might be a useful addition to this.

[shyamsn97]

Fixed the README but may have done some weird merge shenanigans accidentally, let me know if this pr still works @patrickvonplaten. Otherwise I can just close this and upload a new one :)

[patrickvonplaten]

Fixed it ;-)

Pulling the main GitHub branch with git pull upstream main often helps in such cases.

[patrickvonplaten]

Thanks a lot for the contribution - looks very nice!