Separate text_encoder learning rate for LoRA scripts

[jmaccall316]

This is one I notice with LoRA you can train with a very high unet learning rate of 1e-4, but the text encoder being a separate entity I feel this lr is way too high, would be cool to have a bit more control over both.

[patrickvonplaten]

There is an open PR here: #5508

[rmac85]

There is an open PR here: #5508

Thanks, I know some of you guys are weary about bulking up the scripts too much and making them harder to use for beginners, but the text_encoder_lr at least for me is working well with SDXL LoRA. I'm still testing combos, but have achieved much better results already with a text_encoder_lr as low as 8e-7 compared to 1e-4.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.