[Peft] fix saving / loading when unet is not "unet" #6046

kashif · 2023-12-04T15:37:44Z

What does this PR do?

Fix lora load and save when the unet is not the "unet" I believe this is a regression from #5991

HuggingFaceDocBuilderDev · 2023-12-04T15:44:29Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

src/diffusers/loaders/lora.py

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

patrickvonplaten · 2023-12-06T23:01:57Z

@kashif can you add a code snippet that shows the failure case? What exactly is failing because of #5991 ?

kashif · 2023-12-07T07:28:44Z

so the error happens when I load lora weights for the Wuerschten lora trainer that if you recall has no unet but rather a self.prior unet:

  File "/mnt/scratch/kashif/diffusers/examples/wuerstchen/text_to_image/train_text_to_image_lora_prior.py", line 894, in main
    pipeline.prior_pipe.load_lora_weights(args.output_dir, weight_name="pytorch_lora_weights.safetensors")
  File "/mnt/scratch/kashif/diffusers/src/diffusers/loaders/lora.py", line 114, in load_lora_weights
    self.load_lora_into_unet(
  File "/mnt/scratch/kashif/diffusers/src/diffusers/loaders/lora.py", line 449, in load_lora_into_unet
    inject_adapter_in_model(lora_config, unet, adapter_name=adapter_name)
  File "/mnt/scratch/kashif/peft/src/peft/mapping.py", line 163, in inject_adapter_in_model
    peft_model = tuner_cls(model, peft_config, adapter_name=adapter_name)
  File "/mnt/scratch/kashif/peft/src/peft/tuners/lora/model.py", line 119, in __init__
    super().__init__(model, config, adapter_name)
  File "/mnt/scratch/kashif/peft/src/peft/tuners/tuners_utils.py", line 95, in __init__
    self.inject_adapter(self.model, adapter_name)
  File "/mnt/scratch/kashif/peft/src/peft/tuners/tuners_utils.py", line 255, in inject_adapter
    raise ValueError(
ValueError: Target modules {'unet.blocks.2.attention.to_k', 'unet.blocks.50.attention.to_v', 'unet.blocks.26.attention.to_k', 'unet.blocks.68.attention.to_v', 'unet.blocks.44.attention.to_k', 'unet.blocks.38.attention.to_q', 'unet.blocks.56.attention.to_k', 'unet.blocks.35.attention.to_v', 'unet.blocks.89.attention.to_k', 'unet.blocks.14.attention.to_v', 'unet.blocks.23.attention.to_q', 'unet.blocks.53.attention.to_out.0', 'unet.blocks.35.attention.to_out.0', 'unet.blocks.2.attention.to_q', 'unet.blocks.44.attention.to_q', 'unet.blocks.74.attention.to_out.0', 'unet.blocks.17.attention.to_out.0', 'unet.blocks.83.attention.to_k', 'unet.blocks.89.attention.to_q', 'unet.blocks.32.attention.to_k', 'unet.blocks.47.attention.to_k', 'unet.blocks.26.attention.to_v', 'unet.blocks.8.attention.to_k', 'unet.blocks.92.attention.to_k', 'unet.blocks.20.attention.to_out.0', 'unet.blocks.56.attention.to_q', 'unet.blocks.83.attention.to_q', 'unet.blocks.41.attention.to_k', 'unet.blocks.20.attention.to_k', 'unet.blocks.86.attention.to_out.0', 'unet.blocks.71.attention.to_out.0', 'unet.blocks.77.attention.to_v', 'unet.blocks.59.attention.to_q', 'unet.blocks.47.attention.to_q', 'unet.blocks.5.attention.to_out.0', 'unet.blocks.35.attention.to_k', 'unet.blocks.71.attention.to_v', 'unet.blocks.65.attention.to_k', 'unet.blocks.29.attention.to_out.0', 'unet.blocks.62.attention.to_out.0', 'unet.blocks.11.attention.to_q', 'unet.blocks.23.attention.to_out.0', 'unet.blocks.74.attention.to_q', 'unet.blocks.50.attention.to_q', 'unet.blocks.29.attention.to_k', 'unet.blocks.17.attention.to_v', 'unet.blocks.26.attention.to_q', 'unet.blocks.53.attention.to_v', 'unet.blocks.77.attention.to_k', 'unet.blocks.23.attention.to_k', 'unet.blocks.44.attention.to_v', 'unet.blocks.29.attention.to_q', 'unet.blocks.59.attention.to_out.0', 'unet.blocks.23.attention.to_v', 'unet.blocks.35.attention.to_q', 'unet.blocks.14.attention.to_out.0', 'unet.blocks.80.attention.to_out.0', 'unet.blocks.11.attention.to_v', 'unet.blocks.89.attention.to_out.0', 'unet.blocks.26.attention.to_out.0', 'unet.blocks.47.attention.to_v', 'unet.blocks.92.attention.to_v', 'unet.blocks.17.attention.to_q', 'unet.blocks.74.attention.to_v', 'unet.blocks.59.attention.to_k', 'unet.blocks.11.attention.to_out.0', 'unet.blocks.44.attention.to_out.0', 'unet.blocks.80.attention.to_q', 'unet.blocks.68.attention.to_q', 'unet.blocks.83.attention.to_v', 'unet.blocks.71.attention.to_q', 'unet.blocks.32.attention.to_q', 'unet.blocks.5.attention.to_v', 'unet.blocks.8.attention.to_out.0', 'unet.blocks.56.attention.to_out.0', 'unet.blocks.38.attention.to_out.0', 'unet.blocks.53.attention.to_q', 'unet.blocks.5.attention.to_k', 'unet.blocks.92.attention.to_out.0', 'unet.blocks.29.attention.to_v', 'unet.blocks.68.attention.to_out.0', 'unet.blocks.74.attention.to_k', 'unet.blocks.65.attention.to_v', 'unet.blocks.41.attention.to_q', 'unet.blocks.68.attention.to_k', 'unet.blocks.41.attention.to_v', 'unet.blocks.95.attention.to_out.0', 'unet.blocks.14.attention.to_q', 'unet.blocks.62.attention.to_v', 'unet.blocks.2.attention.to_out.0', 'unet.blocks.17.attention.to_k', 'unet.blocks.83.attention.to_out.0', 'unet.blocks.80.attention.to_v', 'unet.blocks.65.attention.to_q', 'unet.blocks.86.attention.to_v', 'unet.blocks.32.attention.to_out.0', 'unet.blocks.80.attention.to_k', 'unet.blocks.62.attention.to_k', 'unet.blocks.38.attention.to_k', 'unet.blocks.50.attention.to_k', 'unet.blocks.59.attention.to_v', 'unet.blocks.8.attention.to_v', 'unet.blocks.50.attention.to_out.0', 'unet.blocks.11.attention.to_k', 'unet.blocks.86.attention.to_q', 'unet.blocks.86.attention.to_k', 'unet.blocks.77.attention.to_q', 'unet.blocks.77.attention.to_out.0', 'unet.blocks.32.attention.to_v', 'unet.blocks.89.attention.to_v', 'unet.blocks.20.attention.to_v', 'unet.blocks.62.attention.to_q', 'unet.blocks.92.attention.to_q', 'unet.blocks.71.attention.to_k', 'unet.blocks.95.attention.to_v', 'unet.blocks.53.attention.to_k', 'unet.blocks.95.attention.to_q', 'unet.blocks.41.attention.to_out.0', 'unet.blocks.5.attention.to_q', 'unet.blocks.65.attention.to_out.0', 'unet.blocks.38.attention.to_v', 'unet.blocks.20.attention.to_q', 'unet.blocks.47.attention.to_out.0', 'unet.blocks.14.attention.to_k', 'unet.blocks.8.attention.to_q', 'unet.blocks.2.attention.to_v', 'unet.blocks.56.attention.to_v', 'unet.blocks.95.attention.to_k'} not found in the base model. Please check the target modules and try again.

sayakpaul

Thank you!

Could you also run the PEFT LoRA slow tests just to make sure things are not being broken here?

younesbelkada

Looks great! Can you also run the slow tests as suggested by sayak?

kashif · 2023-12-14T17:16:04Z

sure i'll test with slow tests

younesbelkada · 2023-12-14T17:18:50Z

note you might want to run them on a V100

sayakpaul · 2023-12-20T04:32:09Z

@kashif a gentle ping.

* [Peft] fix saving / loading when unet is not "unet" * Update src/diffusers/loaders/lora.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * undo stablediffusion-xl changes * use unet_name to get unet for lora helpers * use unet_name --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

This reverts commit 4c7e983.

* add: script to train lcm lora for sdxl with 🤗 datasets * suit up the args. * remove comments. * fix num_update_steps * fix batch unmarshalling * fix num_update_steps_per_epoch * fix; dataloading. * fix microconditions. * unconditional predictions debug * fix batch size. * no need to use use_auth_token * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> * make vae encoding batch size an arg * final serialization in kohya * style * state dict rejigging * feat: no separate teacher unet. * debug * fix state dict serialization * debug * debug * debug * remove prints. * remove kohya utility and make style * fix serialization * fix * add test * add peft dependency. * add: peft * remove peft * autocast device determination from accelerator * autocast * reduce lora rank. * remove unneeded space * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> * style * remove prompt dropout. * also save in native diffusers ckpt format. * debug * debug * debug * better formation of the null embeddings. * remove space. * autocast fixes. * autocast fix. * hacky * remove lora_sayak * Apply suggestions from code review Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * style * make log validation leaner. * move back enabled in. * fix: log_validation call. * add: checkpointing tests * taking my chances to see if disabling autocasting has any effect? * start debugging * name * name * name * more debug * more debug * index * remove index. * print length * print length * print length * move unet.train() after add_adapter() * disable some prints. * enable_adapters() manually. * remove prints. * some changes. * fix params_to_optimize * more fixes * debug * debug * remove print * disable grad for certain contexts. * Add support for IPAdapterFull (#5911) * Add support for IPAdapterFull Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Fix a bug in `add_noise` function (#6085) * fix * copies --------- Co-authored-by: yiyixuxu <yixu310@gmail,com> * [Advanced Diffusion Script] Add Widget default text (#6100) add widget * [Advanced Training Script] Fix pipe example (#6106) * IP-Adapter for StableDiffusionControlNetImg2ImgPipeline (#5901) * adapter for StableDiffusionControlNetImg2ImgPipeline * fix-copies * fix-copies --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * IP adapter support for most pipelines (#5900) * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_upscale.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_attend_and_excite.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py * update tests * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_panorama.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_sag.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion_safe/pipeline_stable_diffusion_safe.py * support ip-adapter in src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py * support ip-adapter in src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py * revert changes to sd_attend_and_excite and sd_upscale * make style * fix broken tests * update ip-adapter implementation to latest * apply suggestions from review --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * fix: lora_alpha * make vae casting conditional/ * param upcasting * propagate comments from #6145 Co-authored-by: dg845 <dgu8957@gmail.com> * [Peft] fix saving / loading when unet is not "unet" (#6046) * [Peft] fix saving / loading when unet is not "unet" * Update src/diffusers/loaders/lora.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * undo stablediffusion-xl changes * use unet_name to get unet for lora helpers * use unet_name --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * [Wuerstchen] fix fp16 training and correct lora args (#6245) fix fp16 training Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * [docs] fix: animatediff docs (#6339) fix: animatediff docs * add: note about the new script in readme_sdxl. * Revert "[Peft] fix saving / loading when unet is not "unet" (#6046)" This reverts commit 4c7e983. * Revert "[Wuerstchen] fix fp16 training and correct lora args (#6245)" This reverts commit 0bb9cf0. * Revert "[docs] fix: animatediff docs (#6339)" This reverts commit 11659a6. * remove tokenize_prompt(). * assistive comments around enable_adapters() and diable_adapters(). --------- Co-authored-by: Suraj Patil <surajp815@gmail.com> Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Fabio Rigano <57982783+fabiorigano@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: yiyixuxu <yixu310@gmail,com> Co-authored-by: apolinário <joaopaulo.passos@gmail.com> Co-authored-by: Charchit Sharma <charchitsharma11@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: dg845 <dgu8957@gmail.com> Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

* [Peft] fix saving / loading when unet is not "unet" * Update src/diffusers/loaders/lora.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * undo stablediffusion-xl changes * use unet_name to get unet for lora helpers * use unet_name --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* add: script to train lcm lora for sdxl with 🤗 datasets * suit up the args. * remove comments. * fix num_update_steps * fix batch unmarshalling * fix num_update_steps_per_epoch * fix; dataloading. * fix microconditions. * unconditional predictions debug * fix batch size. * no need to use use_auth_token * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> * make vae encoding batch size an arg * final serialization in kohya * style * state dict rejigging * feat: no separate teacher unet. * debug * fix state dict serialization * debug * debug * debug * remove prints. * remove kohya utility and make style * fix serialization * fix * add test * add peft dependency. * add: peft * remove peft * autocast device determination from accelerator * autocast * reduce lora rank. * remove unneeded space * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> * style * remove prompt dropout. * also save in native diffusers ckpt format. * debug * debug * debug * better formation of the null embeddings. * remove space. * autocast fixes. * autocast fix. * hacky * remove lora_sayak * Apply suggestions from code review Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * style * make log validation leaner. * move back enabled in. * fix: log_validation call. * add: checkpointing tests * taking my chances to see if disabling autocasting has any effect? * start debugging * name * name * name * more debug * more debug * index * remove index. * print length * print length * print length * move unet.train() after add_adapter() * disable some prints. * enable_adapters() manually. * remove prints. * some changes. * fix params_to_optimize * more fixes * debug * debug * remove print * disable grad for certain contexts. * Add support for IPAdapterFull (huggingface#5911) * Add support for IPAdapterFull Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Fix a bug in `add_noise` function (huggingface#6085) * fix * copies --------- Co-authored-by: yiyixuxu <yixu310@gmail,com> * [Advanced Diffusion Script] Add Widget default text (huggingface#6100) add widget * [Advanced Training Script] Fix pipe example (huggingface#6106) * IP-Adapter for StableDiffusionControlNetImg2ImgPipeline (huggingface#5901) * adapter for StableDiffusionControlNetImg2ImgPipeline * fix-copies * fix-copies --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * IP adapter support for most pipelines (huggingface#5900) * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_upscale.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_attend_and_excite.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py * update tests * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_panorama.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_sag.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion_safe/pipeline_stable_diffusion_safe.py * support ip-adapter in src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py * support ip-adapter in src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py * revert changes to sd_attend_and_excite and sd_upscale * make style * fix broken tests * update ip-adapter implementation to latest * apply suggestions from review --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * fix: lora_alpha * make vae casting conditional/ * param upcasting * propagate comments from huggingface#6145 Co-authored-by: dg845 <dgu8957@gmail.com> * [Peft] fix saving / loading when unet is not "unet" (huggingface#6046) * [Peft] fix saving / loading when unet is not "unet" * Update src/diffusers/loaders/lora.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * undo stablediffusion-xl changes * use unet_name to get unet for lora helpers * use unet_name --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * [Wuerstchen] fix fp16 training and correct lora args (huggingface#6245) fix fp16 training Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * [docs] fix: animatediff docs (huggingface#6339) fix: animatediff docs * add: note about the new script in readme_sdxl. * Revert "[Peft] fix saving / loading when unet is not "unet" (huggingface#6046)" This reverts commit 4c7e983. * Revert "[Wuerstchen] fix fp16 training and correct lora args (huggingface#6245)" This reverts commit 0bb9cf0. * Revert "[docs] fix: animatediff docs (huggingface#6339)" This reverts commit 11659a6. * remove tokenize_prompt(). * assistive comments around enable_adapters() and diable_adapters(). --------- Co-authored-by: Suraj Patil <surajp815@gmail.com> Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Fabio Rigano <57982783+fabiorigano@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: yiyixuxu <yixu310@gmail,com> Co-authored-by: apolinário <joaopaulo.passos@gmail.com> Co-authored-by: Charchit Sharma <charchitsharma11@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: dg845 <dgu8957@gmail.com> Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

* [Peft] fix saving / loading when unet is not "unet" * Update src/diffusers/loaders/lora.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * undo stablediffusion-xl changes * use unet_name to get unet for lora helpers * use unet_name --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* add: script to train lcm lora for sdxl with 🤗 datasets * suit up the args. * remove comments. * fix num_update_steps * fix batch unmarshalling * fix num_update_steps_per_epoch * fix; dataloading. * fix microconditions. * unconditional predictions debug * fix batch size. * no need to use use_auth_token * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> * make vae encoding batch size an arg * final serialization in kohya * style * state dict rejigging * feat: no separate teacher unet. * debug * fix state dict serialization * debug * debug * debug * remove prints. * remove kohya utility and make style * fix serialization * fix * add test * add peft dependency. * add: peft * remove peft * autocast device determination from accelerator * autocast * reduce lora rank. * remove unneeded space * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> * style * remove prompt dropout. * also save in native diffusers ckpt format. * debug * debug * debug * better formation of the null embeddings. * remove space. * autocast fixes. * autocast fix. * hacky * remove lora_sayak * Apply suggestions from code review Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * style * make log validation leaner. * move back enabled in. * fix: log_validation call. * add: checkpointing tests * taking my chances to see if disabling autocasting has any effect? * start debugging * name * name * name * more debug * more debug * index * remove index. * print length * print length * print length * move unet.train() after add_adapter() * disable some prints. * enable_adapters() manually. * remove prints. * some changes. * fix params_to_optimize * more fixes * debug * debug * remove print * disable grad for certain contexts. * Add support for IPAdapterFull (huggingface#5911) * Add support for IPAdapterFull Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Fix a bug in `add_noise` function (huggingface#6085) * fix * copies --------- Co-authored-by: yiyixuxu <yixu310@gmail,com> * [Advanced Diffusion Script] Add Widget default text (huggingface#6100) add widget * [Advanced Training Script] Fix pipe example (huggingface#6106) * IP-Adapter for StableDiffusionControlNetImg2ImgPipeline (huggingface#5901) * adapter for StableDiffusionControlNetImg2ImgPipeline * fix-copies * fix-copies --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * IP adapter support for most pipelines (huggingface#5900) * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_upscale.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_attend_and_excite.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py * update tests * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_panorama.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_sag.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion_safe/pipeline_stable_diffusion_safe.py * support ip-adapter in src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py * support ip-adapter in src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py * revert changes to sd_attend_and_excite and sd_upscale * make style * fix broken tests * update ip-adapter implementation to latest * apply suggestions from review --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * fix: lora_alpha * make vae casting conditional/ * param upcasting * propagate comments from huggingface#6145 Co-authored-by: dg845 <dgu8957@gmail.com> * [Peft] fix saving / loading when unet is not "unet" (huggingface#6046) * [Peft] fix saving / loading when unet is not "unet" * Update src/diffusers/loaders/lora.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * undo stablediffusion-xl changes * use unet_name to get unet for lora helpers * use unet_name --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * [Wuerstchen] fix fp16 training and correct lora args (huggingface#6245) fix fp16 training Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * [docs] fix: animatediff docs (huggingface#6339) fix: animatediff docs * add: note about the new script in readme_sdxl. * Revert "[Peft] fix saving / loading when unet is not "unet" (huggingface#6046)" This reverts commit 4c7e983. * Revert "[Wuerstchen] fix fp16 training and correct lora args (huggingface#6245)" This reverts commit 0bb9cf0. * Revert "[docs] fix: animatediff docs (huggingface#6339)" This reverts commit 11659a6. * remove tokenize_prompt(). * assistive comments around enable_adapters() and diable_adapters(). --------- Co-authored-by: Suraj Patil <surajp815@gmail.com> Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Fabio Rigano <57982783+fabiorigano@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: yiyixuxu <yixu310@gmail,com> Co-authored-by: apolinário <joaopaulo.passos@gmail.com> Co-authored-by: Charchit Sharma <charchitsharma11@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: dg845 <dgu8957@gmail.com> Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

* [Peft] fix saving / loading when unet is not "unet" * Update src/diffusers/loaders/lora.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * undo stablediffusion-xl changes * use unet_name to get unet for lora helpers * use unet_name --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* add: script to train lcm lora for sdxl with 🤗 datasets * suit up the args. * remove comments. * fix num_update_steps * fix batch unmarshalling * fix num_update_steps_per_epoch * fix; dataloading. * fix microconditions. * unconditional predictions debug * fix batch size. * no need to use use_auth_token * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> * make vae encoding batch size an arg * final serialization in kohya * style * state dict rejigging * feat: no separate teacher unet. * debug * fix state dict serialization * debug * debug * debug * remove prints. * remove kohya utility and make style * fix serialization * fix * add test * add peft dependency. * add: peft * remove peft * autocast device determination from accelerator * autocast * reduce lora rank. * remove unneeded space * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> * style * remove prompt dropout. * also save in native diffusers ckpt format. * debug * debug * debug * better formation of the null embeddings. * remove space. * autocast fixes. * autocast fix. * hacky * remove lora_sayak * Apply suggestions from code review Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * style * make log validation leaner. * move back enabled in. * fix: log_validation call. * add: checkpointing tests * taking my chances to see if disabling autocasting has any effect? * start debugging * name * name * name * more debug * more debug * index * remove index. * print length * print length * print length * move unet.train() after add_adapter() * disable some prints. * enable_adapters() manually. * remove prints. * some changes. * fix params_to_optimize * more fixes * debug * debug * remove print * disable grad for certain contexts. * Add support for IPAdapterFull (huggingface#5911) * Add support for IPAdapterFull Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Fix a bug in `add_noise` function (huggingface#6085) * fix * copies --------- Co-authored-by: yiyixuxu <yixu310@gmail,com> * [Advanced Diffusion Script] Add Widget default text (huggingface#6100) add widget * [Advanced Training Script] Fix pipe example (huggingface#6106) * IP-Adapter for StableDiffusionControlNetImg2ImgPipeline (huggingface#5901) * adapter for StableDiffusionControlNetImg2ImgPipeline * fix-copies * fix-copies --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * IP adapter support for most pipelines (huggingface#5900) * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_upscale.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_attend_and_excite.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py * update tests * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_panorama.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_sag.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion_safe/pipeline_stable_diffusion_safe.py * support ip-adapter in src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py * support ip-adapter in src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py * support ip-adapter in src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py * revert changes to sd_attend_and_excite and sd_upscale * make style * fix broken tests * update ip-adapter implementation to latest * apply suggestions from review --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * fix: lora_alpha * make vae casting conditional/ * param upcasting * propagate comments from huggingface#6145 Co-authored-by: dg845 <dgu8957@gmail.com> * [Peft] fix saving / loading when unet is not "unet" (huggingface#6046) * [Peft] fix saving / loading when unet is not "unet" * Update src/diffusers/loaders/lora.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * undo stablediffusion-xl changes * use unet_name to get unet for lora helpers * use unet_name --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * [Wuerstchen] fix fp16 training and correct lora args (huggingface#6245) fix fp16 training Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * [docs] fix: animatediff docs (huggingface#6339) fix: animatediff docs * add: note about the new script in readme_sdxl. * Revert "[Peft] fix saving / loading when unet is not "unet" (huggingface#6046)" This reverts commit 4c7e983. * Revert "[Wuerstchen] fix fp16 training and correct lora args (huggingface#6245)" This reverts commit 0bb9cf0. * Revert "[docs] fix: animatediff docs (huggingface#6339)" This reverts commit 11659a6. * remove tokenize_prompt(). * assistive comments around enable_adapters() and diable_adapters(). --------- Co-authored-by: Suraj Patil <surajp815@gmail.com> Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Fabio Rigano <57982783+fabiorigano@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: yiyixuxu <yixu310@gmail,com> Co-authored-by: apolinário <joaopaulo.passos@gmail.com> Co-authored-by: Charchit Sharma <charchitsharma11@gmail.com> Co-authored-by: Aryan V S <contact.aryanvs@gmail.com> Co-authored-by: dg845 <dgu8957@gmail.com> Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

[Peft] fix saving / loading when unet is not "unet"

99da518

kashif requested a review from BenjaminBossan December 4, 2023 15:50

sayakpaul reviewed Dec 5, 2023

View reviewed changes

src/diffusers/loaders/lora.py Outdated Show resolved Hide resolved

kashif and others added 2 commits December 5, 2023 13:43

Update src/diffusers/loaders/lora.py

4045a1a

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

Merge branch 'main' into kashif-patch-1

7a66d44

kashif added 5 commits December 7, 2023 08:32

Merge branch 'main' into kashif-patch-1

3fa067b

undo stablediffusion-xl changes

597d1fd

use unet_name to get unet for lora helpers

9c386f8

Merge branch 'main' into kashif-patch-1

b59dda6

Merge remote-tracking branch 'upstream/main' into kashif-patch-1

06c0e05

kashif requested review from sayakpaul and removed request for BenjaminBossan December 11, 2023 13:11

sayakpaul approved these changes Dec 14, 2023

View reviewed changes

sayakpaul requested a review from younesbelkada December 14, 2023 16:42

younesbelkada approved these changes Dec 14, 2023

View reviewed changes

kashif added 2 commits December 14, 2023 19:03

Merge branch 'main' into kashif-patch-1

ba1c0d7

Merge branch 'main' into kashif-patch-1

3af6abc

kashif mentioned this pull request Dec 20, 2023

[Wuerstchen] fix fp16 training and correct lora args #6245

Merged

kashif added 2 commits December 20, 2023 10:24

Merge branch 'main' into kashif-patch-1

6f4ae39

use unet_name

0e276cd

kashif mentioned this pull request Dec 20, 2023

[Wuerstchen] dreambooth lora #6247

Closed

Merge branch 'main' into kashif-patch-1

76b7256

kashif merged commit e0d8c91 into main Dec 26, 2023
16 checks passed

kashif deleted the kashif-patch-1 branch December 26, 2023 10:39

sayakpaul added a commit that referenced this pull request Dec 26, 2023

Revert "[Peft] fix saving / loading when unet is not "unet" (#6046)"

fd64acf

This reverts commit 4c7e983.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Peft] fix saving / loading when unet is not "unet" #6046

[Peft] fix saving / loading when unet is not "unet" #6046

kashif commented Dec 4, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 4, 2023

patrickvonplaten commented Dec 6, 2023

kashif commented Dec 7, 2023

sayakpaul left a comment

younesbelkada left a comment

kashif commented Dec 14, 2023

younesbelkada commented Dec 14, 2023

sayakpaul commented Dec 20, 2023

[Peft] fix saving / loading when unet is not "unet" #6046

[Peft] fix saving / loading when unet is not "unet" #6046

Conversation

kashif commented Dec 4, 2023 • edited Loading

What does this PR do?

HuggingFaceDocBuilderDev commented Dec 4, 2023

patrickvonplaten commented Dec 6, 2023

kashif commented Dec 7, 2023

sayakpaul left a comment

Choose a reason for hiding this comment

younesbelkada left a comment

Choose a reason for hiding this comment

kashif commented Dec 14, 2023

younesbelkada commented Dec 14, 2023

sayakpaul commented Dec 20, 2023

kashif commented Dec 4, 2023 •

edited

Loading