ProCreate Diffusion

Paper | Project Page | FSCG-8 Dataset

Public code release for the paper "ProCreate, Don’t Reproduce! Propulsive Energy Diffusion for Creative Generation" (ECCV 2024).

Setup

Download and set up the repo:

git clone https://github.com/Agentic-Learning-AI-Lab/procreate-diffusion-public.git
cd procreate-diffusion-public

Install requirements:

pip install -r requirements.txt

Dataset FSCG-8

The FSCG-8 dataset is locally stored under few-shot-creative-generation-8. FSCG-8 can also be loaded from Huggingface as Jacklu0831/few-shot-creative-generation-8.

Training

The following command fine-tunes a Stable Diffusion checkpoint on Amedeo Modigliani paintings. To fine-tune on other FSCG-8 categories, see commands in commands_train.sh. You can also collect your own few-shot caption-image pairs and put them under dataset_dir.

python src/train.py \
    --output_dir temp \
    --dataset_dir few-shot-creative-generation-8/amedeo_modigliani \
    --enable_xformers_memory_efficient_attention \
    --allow_tf32

Sampling

The following command runs ProCreate sampling for the prompt "a Amedeo Modigliani painting of a boy in a suit and hat" from a Stable Diffusion checkpoint that is fine-tuned on Amedeo Modigliani paintings. src/inference.py automatically downloads the model weights from Huggingface.

python src/inference.py \
    --dataset_dir few-shot-creative-generation-8/amedeo_modigliani \
    --unet_ckpt_dir Jacklu0831/procreate-diffusion-amedeo-modigliani \
    --prompt "a Amedeo Modigliani painting of a boy in a suit and hat" \
    --dreamsim_w 250 \
    --max_grad_norm 0.5 \
    --enable_xformers_memory_efficient_attention \
    --allow_tf32

Sample outputs:

We also provide a trained checkpoint for each other category in FSCG-8. We use commands in commands_inference.sh to display ProCreate samples from each checkpoint.

Apple checkpoint with prompt "an Apple VR headset":

Burberry checkpoint with prompt "a Burberry stuffed bear":

Frank Gehry checkpoint with prompt "a twisting tall apartment building, designed by Frank Gehry":

Nouns checkpoint with prompt "a mountain Nouns character":

One Piece checkpoint with prompt "a One Piece man in a purple robe":

Pokemon checkpoint with prompt "a jellyfish Pokemon":

Rococo checkpoint with prompt "a Rococo style bed":

Acknowledgments

The code is adapted from Huggingface Diffusers.