promote Mixup and Cutmix from prototype to transforms v2 #7731

pmeier · 2023-07-10T19:38:58Z

TL;DR: this PR promotes Mixup and Cutmix from torchvision.prototype.transforms to torchvision.transforms.v2 by reusing the labels_getter functionality that we have for SanitizeBoundingBoxes.

To achieve this, the following this are implemented here:

Factor out the static "labels_getter" methods from SanitizeBoundingBoxes. While doing that we also change the functionality slightly to make the handling a little easier:
- Remove the functionality of passing a string to labels_getter, since that is just slightly more convenient than passing a callable directly, while making the handling harder for us.
- Add the functionality to return labels if we find a tensor as second element of a tuple. We need this for Mixup and Cutmix.

Remove the p parameter. We have this parameter in our references, since we have based them on a research implementation:

vision/references/classification/transforms.py

Line 22 in 08c9938

    
           def __init__(self, num_classes: int, p: float = 0.5, alpha: float = 1.0, inplace: bool = False) -> None:

However, by design, a research implementation will have more knobs than a stable library. In fact, we are hardcoding the parameter in our references:

vision/references/classification/train.py

Lines 224 to 227 in 08c9938

    
           if args.mixup_alpha > 0.0: 
        
               mixup_transforms.append(transforms.RandomMixup(num_classes, p=1.0, alpha=args.mixup_alpha)) 
        
           if args.cutmix_alpha > 0.0: 
        
               mixup_transforms.append(transforms.RandomCutmix(num_classes, p=1.0, alpha=args.cutmix_alpha))

By removing the p parameter in this PR, we get the same behavior that we currently have in our references as well. If the user need the more flexible behavior back, they can always wrap the transform like RandomApply(Mixup(...), p=...).

Since the existence of the p parameter was the reason to prefix "Random" before the "canonical" names Mixup and Cutmix, I've dropped the prefix here as well.

Follow-up to Add --use-v2 support to classification references #7724. The implementation of this PR will be available in the classification references with --use-v2. I also refactored the training script to just use the transform inside the training loop rather than putting it inside the collate_fn. This makes it more clear that this transform needs batching, but is otherwise independent of the data loader.

ToDo

Fix old tests for SanitizeBoundingBoxes
Write new tests for Mixup and cutmix
Remove prototype transforms RandomMixup and RandomCutmix

cc @vfdev-5

pytorch-bot · 2023-07-10T19:39:01Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7731

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures

As of commit 993f693:

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

NicolasHug

Thanks Philip, only made minor comments for now, overall this looks great. Will give a more thorough look once we have the tests

NicolasHug · 2023-07-11T09:12:20Z

references/classification/train.py

+        if batch_transform:
+            image, target = batch_transform(image, target)


I failed to notice this when we discussed it offline, but we should keep those transforms as collate_fn: calling them after the dataloder like done here means we can't leverage multi-processing.

NicolasHug · 2023-07-11T09:34:09Z

references/classification/train.py

 from torchvision.transforms.functional import InterpolationMode
+from transforms import get_batch_transform


Sooooo to avoid bikeshedding on how we should call those (batch transforms vs pairwise transforms vs something else), maybe we should just rename that to get_cutmix_mixup?

NicolasHug · 2023-07-11T09:36:49Z

torchvision/transforms/v2/_augment.py

+            msg = "Couldn't find a label in the inputs."
+            if self.labels_getter == "default":
+                msg = f"{msg} To overwrite the default find behavior, pass a callable for labels_getter."


Maybe we can write that entire message regardless of whether "default" was passed. It would simplify the logic a bit and avoid storing self.labels_getter.

NicolasHug · 2023-07-11T09:37:13Z

torchvision/transforms/v2/_augment.py

+            msg = "Couldn't find a label in the inputs."
+            if self.labels_getter == "default":
+                msg = f"{msg} To overwrite the default find behavior, pass a callable for labels_getter."
+            raise RuntimeError(msg)


Technically this could qualify as a ValueError as well?

torchvision/transforms/v2/_augment.py

NicolasHug · 2023-07-11T09:44:07Z

torchvision/transforms/v2/_augment.py

+
+        # By default, the labels will be False inside needs_transform_list, since they are a torch.Tensor, but coming
+        # after an image or video. However, since we want to handle them in _transform, we
+        needs_transform_list[next(idx for idx, inpt in enumerate(flat_inputs) if inpt is labels)] = True


We used a different strategy in SanitizeBoundingBox where we called _transform() on all inputs and just handled that filtering logic within _transform(). I don't have a pref right now (haven't thought about it much). But maybe we should align both transforms to follow the same strat? (we could do it in another PR)

We can't use the same strategy here. SanitizeBoundingBox does not affect images or videos, so we don't care about needs_transform_list there. However, here we transform images. Meaning, we need to use needs_transform_list to make use of the heuristic about what image to transform. This cannot be done in _transform since in there we have no concept if an image should be transformed or not.

Shouldn't we transform all images (each image is collated as (N, C, H, W)) ?

I think I understand what you mean: we'd need to re-implement the "tensor pass-through heuristic" in _transform() if we were to do something like in SanitizeBoundingBox(), and we don't want to do that. I feel like we could use the same strategy used here in SanitizeBoudingBox() though. But that's OK.

Shouldn't we transform all images (each image is collated as (N, C, H, W)) ?

We are transforming all images yes

I feel like we could use the same strategy used here in SanitizeBoudingBox() though. But that's OK.

Yes, we can certainly also use needs_transform_list there. I'm ok with that. Up to you.

torchvision/transforms/v2/_augment.py

vfdev-5 · 2023-07-11T10:25:14Z

torchvision/transforms/v2/_utils.py

@@ -93,3 +96,61 @@ def _check_padding_arg(padding: Union[int, Sequence[int]]) -> None:
 def _check_padding_mode_arg(padding_mode: Literal["constant", "edge", "reflect", "symmetric"]) -> None:
    if padding_mode not in ["constant", "edge", "reflect", "symmetric"]:
        raise ValueError("Padding mode should be either constant, edge, reflect or symmetric")
+
+
+def _find_labels_default_heuristic(inputs: Any) -> torch.Tensor:


I wonder if this method can be a class that could finetune itself after the first iteration on the input type and skip checking for tuple if batch is a dict and set up the label key ?

Another idea could be to provide predefined labels_getter for these two situations...

Good point' it'd be interesting to figure out whether this makes things faster. Might be best to leave this out as future improvement though, to keep this PR simpler.
One thing to note: doing this would tie the transform instance to a specific dataset [format]. IDK whether this is a problem in practice, but worth keeping in mind

torchvision/transforms/v2/_augment.py

torchvision/transforms/v2/_misc.py

torchvision/transforms/v2/_utils.py

test/test_transforms_v2_refactored.py

…ixup

NicolasHug · 2023-07-25T13:23:54Z

Quick update: after chatting with @pmeier we decided to remove support for 2D labels. There isn't a strong need for it considering CutMix and MixUp should probably never be called consecutively - it's either one of the other.

Should users request this to be supported for whatever reason (maybe Compose(CutMix(), CutMix()) make sense??) then we can add this back.

support was removed in 9f4a9e6

torchvision/transforms/v2/_misc.py

pmeier · 2023-07-27T09:16:13Z

torchvision/transforms/v2/_utils.py

+    contains no "label-like" key.
+
+    """
+    # TODO: Document list and why


Sooo I've decided not to document it in the code because this will probably just add more confusion. But the reason we need to add support for list is because this is what the DataLoader actually returns (in its most default setting):

for x in DataLoader(...): # x is a list [img_batch, labels_batch] and we want to support CutMix(x)

torchvision/transforms/v2/_utils.py

NicolasHug · 2023-07-27T10:16:29Z

torchvision/transforms/v2/_misc.py

-            the key whose value corresponds to the labels. It can also be a callable that takes the same input
-            as the transform, and returns the labels.
-            By default, this will try to find a "labels" key in the input, if
+            By default, this will try to find a "labels" key in the input (case-insensitive), if


I chose not to document the "labels" key matching in the finer details. We can revisit (or point users to check the code?)

NicolasHug · 2023-07-27T11:29:22Z

gallery/plot_cutmix_mixup.py

+How to use Cutmix and Mixup
+===========================
+
+TODO


NicolasHug · 2023-07-27T11:30:23Z

references/classification/train.py

 import utils
 from sampler import RASampler
 from torch import nn
 from torch.utils.data.dataloader import default_collate
 from torchvision.transforms.functional import InterpolationMode
+from transforms import get_mixup_cutmix


I'll validate the changes made to the references once this is merged.

NicolasHug · 2023-07-27T11:31:29Z

test/test_transforms_v2.py

        transforms.SanitizeBoundingBox(labels_getter=12)

    with pytest.raises(ValueError, match="Could not infer where the labels are"):
        bad_labels_key = {"bbox": good_bbox, "BAD_KEY": torch.arange(good_bbox.shape[0])}
        transforms.SanitizeBoundingBox()(bad_labels_key)

-    with pytest.raises(ValueError, match="If labels_getter is a str or 'default'"):


Note that I had to delete this. I feel like it was a valid error to raise. We could put it back if we were to have a different labels_getter logic for SanitizeBBox and the Cutmix/Mixup ones (which is probably going to be needed eventually anyway).

This is OK for now.

NicolasHug

I think this LGTM. Since I made a bunch of changes, this should get another round of reviews from @pmeier @vfdev-5 before merging

vfdev-5 · 2023-07-27T22:25:13Z

There are conflicts to resolve

NicolasHug · 2023-07-28T05:22:50Z

They're trivial, I'll address at the next (and hopefully last) review

pmeier

LGTM, thanks Nicolas for finishing this. I can't approve though, since it is technically my PR.

vfdev-5 · 2023-07-28T07:38:22Z

docs/source/transforms.rst

@@ -261,6 +261,22 @@ The new transform can be used standalone or mixed-and-matched with existing tran
    AugMix
    v2.AugMix

+Cutmix - Mixup


Nit: technically, paper names of these techniques are CutMix and MixUp.

Sorry, saw this after I merged. I'll address via #7766

github-actions · 2023-07-28T07:41:28Z

Hey @NicolasHug!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py

) Reviewed By: matteobettini Differential Revision: D48642303 fbshipit-source-id: e1e379d7dc99fee094fb5a3f7f97e0cd1eb93028 Co-authored-by: Nicolas Hug <nicolashug@meta.com> Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

pmeier added 2 commits July 10, 2023 21:05

promote Mixup and Cutmix from prototype

6e9eb90

add v2 Mixup and Cutmix to references

de92eb6

pmeier added the module: transforms label Jul 10, 2023

pmeier requested a review from NicolasHug July 10, 2023 19:38

facebook-github-bot added the cla signed label Jul 10, 2023

NicolasHug reviewed Jul 11, 2023

View reviewed changes

go back to using collate_fn

c160ae7

pmeier requested a review from vfdev-5 July 11, 2023 09:57

vfdev-5 reviewed Jul 11, 2023

View reviewed changes

torchvision/transforms/v2/_augment.py Show resolved Hide resolved

vfdev-5 requested review from NicolasHug and vfdev-5 July 11, 2023 10:45

NicolasHug reviewed Jul 13, 2023

View reviewed changes

torchvision/transforms/v2/_misc.py Show resolved Hide resolved

NicolasHug reviewed Jul 13, 2023

View reviewed changes

torchvision/transforms/v2/_utils.py Outdated Show resolved Hide resolved

pmeier and others added 5 commits July 17, 2023 10:27

Merge branch 'main' into cutmix-mixup

1cd7c7a

Address minor comments and typos

7934566

Added some tests, slightly changed error messages

d5bb664

More tests

b4e6d43

some more tests

fa97d52

NicolasHug reviewed Jul 24, 2023

View reviewed changes

test/test_transforms_v2_refactored.py Outdated Show resolved Hide resolved

NicolasHug reviewed Jul 24, 2023

View reviewed changes

test/test_transforms_v2_refactored.py Outdated Show resolved Hide resolved

Update test/test_transforms_v2_refactored.py

f3708be

pmeier commented Jul 25, 2023

View reviewed changes

NicolasHug added 6 commits July 25, 2023 10:10

Address comments

50fa4d2

Merge branch 'cutmix-mixup' of github.com:pmeier/vision into cutmix-m…

26f55de

…ixup

Got rid of FakeData, doesn't work tho

e91e879

Use bigger images

45bf28c

Remove support for 2d labels

9f4a9e6

num_categories -> num_classes

0505f24

pmeier commented Jul 27, 2023

View reviewed changes

update docs

4d5890d

NicolasHug reviewed Jul 27, 2023

View reviewed changes

NicolasHug added 2 commits July 27, 2023 11:28

Fix some SanitizeBBox tests

4538c10

Add docs, add default for alpha

6542fd0

NicolasHug mentioned this pull request Jul 27, 2023

Add example gallery for Mixup and Cutmix #7766

Closed

NicolasHug reviewed Jul 27, 2023

View reviewed changes

gallery/plot_cutmix_mixup.py

How to use Cutmix and Mixup

===========================

TODO

Copy link

Member

NicolasHug Jul 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#7766

NicolasHug reviewed Jul 27, 2023

View reviewed changes

Merge branch 'main' of github.com:pytorch/vision into cutmix-mixup

0c3b932

NicolasHug marked this pull request as ready for review July 27, 2023 11:34

NicolasHug approved these changes Jul 27, 2023

View reviewed changes

NicolasHug added 2 commits July 27, 2023 12:44

separate doc section

acc7a98

Hopefully fix cuda test

5e02675

pmeier commented Jul 28, 2023

View reviewed changes

Merge branch 'main' of github.com:pytorch/vision into cutmix-mixup

993f693

vfdev-5 reviewed Jul 28, 2023

View reviewed changes

NicolasHug merged commit 3591371 into pytorch:main Jul 28, 2023

NicolasHug added the new feature label Jul 28, 2023

pmeier deleted the cutmix-mixup branch July 28, 2023 09:45

NicolasHug mentioned this pull request Jul 31, 2023

Add gallery example for MixUp and CutMix #7772

Merged

NicolasHug mentioned this pull request May 28, 2024

cutmix alpha argument in references/classification/transforms.py #8440

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

promote Mixup and Cutmix from prototype to transforms v2 #7731

promote Mixup and Cutmix from prototype to transforms v2 #7731

pmeier commented Jul 10, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 10, 2023 •

edited

Loading

NicolasHug left a comment

NicolasHug Jul 11, 2023

NicolasHug Jul 11, 2023

NicolasHug Jul 11, 2023

NicolasHug Jul 11, 2023

NicolasHug Jul 11, 2023

pmeier Jul 11, 2023

vfdev-5 Jul 11, 2023 •

edited

Loading

NicolasHug Jul 17, 2023

pmeier Jul 28, 2023

vfdev-5 Jul 11, 2023 •

edited

Loading

NicolasHug Jul 17, 2023

NicolasHug commented Jul 25, 2023

pmeier Jul 27, 2023

NicolasHug Jul 27, 2023

NicolasHug Jul 27, 2023

NicolasHug Jul 27, 2023

NicolasHug Jul 27, 2023

NicolasHug Jul 27, 2023

NicolasHug left a comment

vfdev-5 commented Jul 27, 2023

NicolasHug commented Jul 28, 2023

pmeier left a comment

vfdev-5 Jul 28, 2023

NicolasHug Jul 28, 2023

github-actions bot commented Jul 28, 2023

	if args.mixup_alpha > 0.0:
	mixup_transforms.append(transforms.RandomMixup(num_classes, p=1.0, alpha=args.mixup_alpha))
	if args.cutmix_alpha > 0.0:
	mixup_transforms.append(transforms.RandomCutmix(num_classes, p=1.0, alpha=args.cutmix_alpha))

		if batch_transform:
		image, target = batch_transform(image, target)

		from torchvision.transforms.functional import InterpolationMode
		from transforms import get_batch_transform

promote Mixup and Cutmix from prototype to transforms v2 #7731

promote Mixup and Cutmix from prototype to transforms v2 #7731

Conversation

pmeier commented Jul 10, 2023 • edited by pytorch-bot bot Loading

ToDo

pytorch-bot bot commented Jul 10, 2023 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7731

❌ 8 New Failures

NicolasHug left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vfdev-5 Jul 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vfdev-5 Jul 11, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NicolasHug commented Jul 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NicolasHug left a comment

Choose a reason for hiding this comment

vfdev-5 commented Jul 27, 2023

NicolasHug commented Jul 28, 2023

pmeier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 28, 2023

pmeier commented Jul 10, 2023 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 10, 2023 •

edited

Loading

vfdev-5 Jul 11, 2023 •

edited

Loading

vfdev-5 Jul 11, 2023 •

edited

Loading