Auto mask uniform background base on PR #589 mask loss #1114
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
implement mask loading from mask folder
# Conflicts: # fine_tune.py # sdxl_train.py # train_db.py # train_network.py # train_textual_inversion.py # train_textual_inversion_XTI.py
# Conflicts: # fine_tune.py # sdxl_train.py
|
Thank you for this PR. As I wrote before in #589 , I've implemented ControlNetDataset which has conditioning data recently. I believe the ControlNetDataset can have not only canny or pose control images but also mask images. I think it might be an idea to add a capability to handle ControlNetDataset for each training script. However, it will take some time to implement. So, please let me carefully consider how to handle this PR. I hope you will understand. I also think |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Base on PR #589. Thanks recris.
(1) Change the mechanism for caching_latent_to_disk by directly storing the mask in the NPZ file. Actually, due to trim_and_resize, it was impossible to match the required size, made cache_latent_to_disk unusable.
(2) Modify the way of color augmentation to preserve the original alpha channel unchanged, which should allow for more accurate handling of the mask.
(3) Add
--mask_simple_background. Enable auto-masking of latent loss based on the dominant edge color if it occupies more than 30% of the image edges. This helps in focusing the model on the main content by ignoring simple or uniform background colors such as solid white or black.I think this will help improve the quality of datasets with a high proportion of white backgrounds, transparent backgrounds, and simple backgrounds in anime images.
This is a simpler feature implementation. It might be possible to add automatic masking for faces (for clothing training), characters, etc. However, because automatically processed masks are harder to inspect and would introduce a significant amount of additional requirements, it might be better to use the script I wrote or the webui's rembg to pre-generate masks for manual inspection.