Fix PixArt 256px inference #6789

lawrence-cj · 2024-01-31T11:17:42Z

This PR

Removed the interpolation_scale>=1 checking. Instead, we change it into config file(config.json). Besides, we add 256bin for 256px generation.
Change the definition of T5 max token length into pipeline config file(model_index.json). Talked about here
fix bug in convert weight file

Fixes #6783 too

lawrence-cj · 2024-02-01T04:34:42Z

from diffusers import PixArtAlphaPipeline, Transformer2DModel
import torch

transformer = Transformer2DModel.from_pretrained("PixArt-alpha/PixArt-XL-2-256x256", subfolder="transformer", torch_dtype=torch.float16)
pipe = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-1024-MS", transformer=transformer, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0]  
    
image.save("astronaut_rides_horse.png")

The test code is here for reference.

scripts/convert_pixart_alpha_to_diffusers.py

sayakpaul · 2024-02-03T14:26:27Z

scripts/convert_pixart_alpha_to_diffusers.py

-        "--orig_ckpt_path", default=None, type=str, required=False, help="Path to the checkpoint to convert."
-    )
+    # set multi_scale_train=True if using PixArtMS structure during training else set it to False
+    parser.add_argument("--multi_scale_train", default=True, type=str, required=True, help="If use Multi-Scale PixArtMS structure during training.")


The type is str and we're defaulting to a bool. This needs to be fixed.

Also, do we have any other 1024x1024 checkpoints that are affected by this? If not, do we really need this flag?

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py

sayakpaul

Thanks for the changes. Left a couple of comments.

HuggingFaceDocBuilderDev · 2024-02-03T14:32:41Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

yiyixuxu

thanks!

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py

yiyixuxu · 2024-02-08T20:51:49Z

can we fix the tests here too?

Co-authored-by: YiYi Xu <yixu310@gmail.com>

lawrence-cj · 2024-02-12T11:39:45Z

Is there any test failing here?

sayakpaul · 2024-02-13T04:12:32Z

src/diffusers/models/transformers/transformer_2d.py

@@ -97,6 +97,7 @@ def __init__(
        norm_eps: float = 1e-5,
        attention_type: str = "default",
        caption_channels: int = None,
+        interpolation_scale: float = None,


We are not leveraging this anywhere no? Let's remove it?

sayakpaul · 2024-02-13T04:13:31Z

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py

@@ -228,6 +264,7 @@ def __init__(
        vae: AutoencoderKL,
        transformer: Transformer2DModel,
        scheduler: DPMSolverMultistepScheduler,
+        model_token_max_length: int = 120,


@yiyixuxu why did we decide to make this as a config variable instead of a pipeline call arg?

@sayakpaul @lawrence-cj
oh I'm not sure
Is this something we need to change each generation? e.g., if we need to adjust this value based on the prompt, I think it makes sense to add it to pipeline call arg; Otherwise, we can add it as a config, no?

The maximum sequence length bit should be something a user wants to experiment with. I don't think it needs to be a configuration variable.

sayakpaul

Looks very nice to me! Thank you ❤️

It'd be very nice to also include an example of using the 256x256 checkpoint in the PixArt-Alpha doc: https://huggingface.co/docs/diffusers/main/en/api/pipelines/pixart. WDYT?

sayakpaul · 2024-02-13T04:20:10Z

Let's fix the quality tests :)

lawrence-cj · 2024-02-13T04:29:15Z

Looks very nice to me! Thank you ❤️

It'd be very nice to also include an example of using the 256x256 checkpoint in the PixArt-Alpha doc: https://huggingface.co/docs/diffusers/main/en/api/pipelines/pixart. WDYT?

import torch
from diffusers import PixArtAlphaPipeline

# You can replace the checkpoint id with "PixArt-alpha/PixArt-XL-2-512x512" or "PixArt-alpha/PixArt-XL-2-256x256" too.
pipe = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-1024-MS", torch_dtype=torch.float16)
# Enable memory optimizations.
pipe.enable_model_cpu_offload()

prompt = "A small cactus with a happy face in the Sahara desert."
image = pipe(prompt).images[0]

How about this one? The usage of 256px is totally the same as 512px or 1024px for simplicity and efficiency.

sayakpaul · 2024-02-13T05:15:40Z

Yeah that should work. But we still need to fix the tests here.

lawrence-cj · 2024-02-13T05:24:14Z

Sure. Most of the failures are cauesed by the model_token_max_length. Maybe you guys should decide how to arrange it first and I may help to commit a new version.

sayakpaul · 2024-02-18T07:01:49Z

@lawrence-cj let's keep max_sequence_length as a pipeline call argument and default it at 120. WDYT? @yiyixuxu do we have your go here?

sayakpaul · 2024-02-18T09:19:25Z

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py

@@ -688,6 +725,7 @@ def __call__(
        callback_steps: int = 1,
        clean_caption: bool = True,
        use_resolution_binning: bool = True,
+        model_token_max_length: int = 120,


Suggested change

model_token_max_length: int = 120,

max_sequence_length: int = 120,

Let's use this variable name throughout?

Can we also add this argument to the call docstrings?

sayakpaul

Looking very nice. Just one comment and then I think we can merge it!

lawrence-cj · 2024-02-18T09:36:21Z

Done on my side. Thanks so much. @sayakpaul @yiyixuxu

sayakpaul · 2024-02-18T09:39:09Z

Everything looks good. We just need to add max_sequence_length to the pipeline docstrings here:

diffusers/src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py

Line 756 in 8974c50

use_resolution_binning (`bool` defaults to `True`):

sayakpaul · 2024-03-03T05:01:35Z

Thanks a lot @lawrence-cj for your contributions here!

lawrence-cj added 3 commits January 31, 2024 18:15

feat 256px diffusers inference bug

824b1c5

change the max_length of T5 to pipeline config file

7d03ce5

fix bug in convert_pixart_alpha_to_diffusers.py

a57a89a

This was referenced Jan 31, 2024

PixArt-XL-2-256x256 generations are messed up #6783

Closed

Fix/pixart weight convert bug #6424

Closed

Feat/pixart lora #6199

Closed

sayakpaul requested a review from yiyixuxu January 31, 2024 11:26

sayakpaul reviewed Feb 3, 2024

View reviewed changes

scripts/convert_pixart_alpha_to_diffusers.py Outdated Show resolved Hide resolved

sayakpaul reviewed Feb 3, 2024

View reviewed changes

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py Outdated Show resolved Hide resolved

sayakpaul reviewed Feb 3, 2024

View reviewed changes

lawrence-cj and others added 4 commits February 3, 2024 08:36

Update scripts/convert_pixart_alpha_to_diffusers.py

ec391fe

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

Merge branch 'huggingface:main' into main

6bcaa8e

remove multi_scale_train parser

186b662

Merge branch 'main' into main

22264e5

yiyixuxu reviewed Feb 8, 2024

View reviewed changes

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py Outdated Show resolved Hide resolved

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py Outdated Show resolved Hide resolved

lawrence-cj and others added 3 commits February 11, 2024 13:07

Update src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py

64e7c49

Co-authored-by: YiYi Xu <yixu310@gmail.com>

Update src/diffusers/pipelines/pixart_alpha/pipeline_pixart_alpha.py

55002c1

Co-authored-by: YiYi Xu <yixu310@gmail.com>

Merge branch 'main' into main

109db96

sayakpaul reviewed Feb 13, 2024

View reviewed changes

Merge branch 'main' into main

2c475a5

sayakpaul added 2 commits February 18, 2024 12:29

Merge branch 'main' into main

d4850cc

styling

56e31c1

change model_token_max_length to call argument.

597a510

lawrence-cj requested a review from sayakpaul February 18, 2024 07:37

sayakpaul reviewed Feb 18, 2024

View reviewed changes

sayakpaul approved these changes Feb 18, 2024

View reviewed changes

sayakpaul and others added 2 commits February 18, 2024 14:50

Merge branch 'main' into main

bb97a45

Refactoring

46513c2

sayakpaul added 2 commits March 3, 2024 09:46

Merge branch 'main' into main

12d416f

add: max_sequence_length to the docstring.

0199bcb

sayakpaul merged commit f55873b into huggingface:main Mar 3, 2024
15 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix PixArt 256px inference #6789

Fix PixArt 256px inference #6789

lawrence-cj commented Jan 31, 2024 •

edited by sayakpaul

Loading

lawrence-cj commented Feb 1, 2024

sayakpaul Feb 3, 2024

sayakpaul Feb 3, 2024

sayakpaul left a comment

HuggingFaceDocBuilderDev commented Feb 3, 2024

yiyixuxu left a comment

yiyixuxu commented Feb 8, 2024

lawrence-cj commented Feb 12, 2024

sayakpaul Feb 13, 2024

sayakpaul Feb 13, 2024

yiyixuxu Feb 18, 2024

sayakpaul Feb 18, 2024

yiyixuxu Feb 18, 2024

sayakpaul left a comment

sayakpaul commented Feb 13, 2024

lawrence-cj commented Feb 13, 2024 •

edited

Loading

sayakpaul commented Feb 13, 2024

lawrence-cj commented Feb 13, 2024

sayakpaul commented Feb 18, 2024

sayakpaul Feb 18, 2024

sayakpaul left a comment

lawrence-cj commented Feb 18, 2024

sayakpaul commented Feb 18, 2024

sayakpaul commented Mar 3, 2024

	model_token_max_length: int = 120,
	max_sequence_length: int = 120,

Fix PixArt 256px inference #6789

Fix PixArt 256px inference #6789

Conversation

lawrence-cj commented Jan 31, 2024 • edited by sayakpaul Loading

lawrence-cj commented Feb 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Feb 3, 2024

yiyixuxu left a comment

Choose a reason for hiding this comment

yiyixuxu commented Feb 8, 2024

lawrence-cj commented Feb 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

sayakpaul commented Feb 13, 2024

lawrence-cj commented Feb 13, 2024 • edited Loading

sayakpaul commented Feb 13, 2024

lawrence-cj commented Feb 13, 2024

sayakpaul commented Feb 18, 2024

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

lawrence-cj commented Feb 18, 2024

sayakpaul commented Feb 18, 2024

sayakpaul commented Mar 3, 2024

lawrence-cj commented Jan 31, 2024 •

edited by sayakpaul

Loading

lawrence-cj commented Feb 13, 2024 •

edited

Loading