UnetMotionModel ignores important parameters

[jon-chuang]

Describe the bug

{'center_input_sample': False, 'flip_sin_to_cos': True, 'freq_shift': 0, 'mid_block_type': 'UNetMidBlock2DCrossAttn', 'only_cross_attention': False, 'dropout': 0.0, 'transformer_layers_per_block': 1, 'attention_head_dim': 8, 'dual_cross_attention': False, 'class_embed_type': None, 'addition_embed_type': None, 'addition_time_embed_dim': None, 'num_class_embeds': None, 'upcast_attention': None, 'resnet_time_scale_shift': 'default', 'resnet_skip_time_act': False, 'resnet_out_scale_factor': 1.0, 'time_embedding_type': 'positional', 'time_embedding_dim': None, 'time_embedding_act_fn': None, 'timestep_post_act': None, 'time_cond_proj_dim': 256, 'conv_in_kernel': 3, 'conv_out_kernel': 3, 'projection_class_embeddings_input_dim': None, 'attention_type': 'default', 'class_embeddings_concat': False, 'mid_block_only_cross_attention': None, 'cross_attention_norm': None, 'addition_embed_type_num_heads': 64}

In particular, it does not initialize time_embedding with 'cond_proj_dim': 256

Reproduction

DreamshaperV7 + MotionAdapter in AnimateDiff pipeline

Logs

No response

System Info

• diffusers version: 0.24.0
• Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
• Python version: 3.10.12
• PyTorch version (GPU?): 2.2.0a0+git177a1a6 (True)
• Huggingface_hub version: 0.16.4
• Transformers version: 4.35.2
• Accelerate version: 0.25.0
• xFormers version: 0.0.22.post7
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@DN6 @sayakpaul @patrickvonplaten

[DN6]

Hi @jon-chuang I don't think the original implementation used cond_proj_dim in its UNet. Hence it was omitted. Would you be able to open a PR to add it to the current motion UNet?

[jon-chuang]

Sure, this is used for example in some non-standard models like DreamshaperV7

[DN6]

Hmm actually. Would you be able to provide a snippet to reproduce? I think the UNet is being initialised with the right weights, but the warning is being raised because of how the config is loaded.

[jon-chuang]

Well, long story short, it throws an error, saying the parameter in question is unexpected. That's because the TimeEmbedding state dict has not been initialized from the JSON config properly, so it does not have a cond.proj field/attr.

By contrast, the Dreamshaper unet timeembedding has a cond.proj field.

[jon-chuang]

Repro:

import torch
from diffusers import MotionAdapter, AnimateDiffPipeline

adapter = MotionAdapter.from_pretrained("guoyww/animatediff-motion-adapter-v1-5-2").to('cuda', dtype=torch.float16)
pipe = AnimateDiffPipeline.from_pretrained(
    "SimianLuo/LCM_Dreamshaper_v7", use_safetensors=True, motion_adapter=adapter
).to('cuda', dtype=torch.float16)

[DN6]

Ah yes. You're right. Would you mind opening a PR with a fix?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

@DN6 feel free to tackle it if the author of the issue doesn't answer

[stephen-iezzi]

I worked on an issue with the UNetMotionModel yesterday, I'd be happy to go for this one as well!

[sayakpaul]

Feel free to submit your PR :)

[DN6]

@Stepheni12 You can add both changes to your PR: #7035

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

I guess we can close this now.