Add mmsegmentation DeepLabv3(+)

[Landanjs]

Motivation

We were using torchvision's DeepLabv3 model for segmentation results, but most papers use DeepLabv3+ as a baseline for semantic segmentation tasks.

Implementation

This PR uses the DeepLabv3+ model from mmsegmentation which is a commonly used segmentation library (used in at least ConvNext, Swin, and SegFormer). One caveat is that mmsegmentation makes adjustments to the original DeepLabv3(+) model, but maybe this can be justified since ADE20k should be more difficult than the datasets DeepLabv3 was originally proposed for.

Instead of completely removing DeepLabv3, I've added a flag to toggle between the two implementations in mmsegmentation.

Somewhat unrelated, I added an argument to specify the url to download the backbone weights from to make some future experiments easier.

Side notes:

• In the future, we should have a more general backbone-head interface when we have more backbones and heads.
• One alternative to mmsegmentation is to use detectron2, but this seems less friendly. Please let me know if this would be a better alternative

Results

mmsegmentation one-run results: 45 mIoU and 45.47 mIoU for DeepLabv3 and DeepLabv3+, respectively. There are several differences that make it difficult to compare our numbers to theirs:

1. Auxiliary loss
2. Different pre-trained weights
3. Different ResNet stem (v1c)

Model	mIoU	TTT
Old Deeplabv3 Unoptimized	44.21 +/- 0.30	6.8 hours
DeepLabv3 Unoptimized	44.82 +/- 0.19	7.8 hours
DeepLabv3+ Unoptimized	44.69 +/- 0.34	7.3 hours
Old DeepLabv3 Optimized	45.62 +/- 0.16	4.9 hours
DeepLabv3 Optimized	45.95 +/- 0.20	6 hours
DeepLabv3+ Optimized	45.75 +/- 0.3	TBD

[A-Jacobson]

Somewhat unrelated, I added an argument to specify the url to download the backbone weights from to make some future experiments easier.

I like the thought but is this necessary right now? does mmseg support something like this or are the urls aliased?

Side notes:

• In the future, we should have a more general backbone-head interface when we have more backbones and heads.

Agreed, I'm thinking directories can specify whole model families sort of like our resnets. We should discuss this elsewhere

• One alternative to mmsegmentation is to use detectron2, but this seems less friendly. Please let me know if this would be a better alternative

all the openmm lab stuff is more actively developed. additionally, it looks like mmseg/mmdet are supersets of detectron with similar api's so if we're going to pick one i think mmseg is a good bet.

[Landanjs]

Somewhat unrelated, I added an argument to specify the url to download the backbone weights from to make some future experiments easier.

I like the thought but is this necessary right now? does mmseg support something like this or are the urls aliased?

As of now, we are only using the heads from mmsegmentation, but still using torchvision backbones. The url argument functionality is to specify a specific torchvision url weight i.e. swap between weights from the old and new torchvision training recipes. This isn't necessary, but I foresee running experiments that swap between different pretrained weights.

[Landanjs]

@A-Jacobson Updated numbers in first table based off of recent runs. DeepLabv3 looks really good while DeepLabv3+ may be falling a bit short. One seed reaches the target mIoU the epoch before, not sure if that is sufficient 😬

[A-Jacobson]

@A-Jacobson Updated numbers in first table based off of recent runs. DeepLabv3 looks really good while DeepLabv3+ may be falling a bit short. One seed reaches the target mIoU the epoch before, not sure if that is sufficient 😬

I'm good with this as the baseline wasn't significantly different!

[Landanjs]

@ravi-mosaicml just double checking... Does this good to you?

[ExtReMLapin]

As you quoted detectron, do you have any quick solution to easily integrate this in detectron2 and possibly speed up training ?

[dblalock]

@Landanjs would be the right person to answer that, but he's on vacation this week. Maybe @florescl can comment?

[A-Jacobson]

Hi @ExtReMLapin! Thanks for checking out composer. We don't yet have any convenience wrappers for detectron2 like we do for timm, but you could still train a detectron model using composer methods. I can see two ways to accomplish this right now.

1. choose methods from our functional interface and drop them directly into your training loop. You may have to copy and modify the training loop from detectron if that's what you plan to use.

2. create a ComposerModel from your detectron model and use our trainer. This may be a bit tricky as detectron models can behave differently during training and inference + they calculated loss internally. Some of the huggingface models and SSD models in composer.models may serve as good examples for dealing with this.

Regarding which methods to use, SAM and SWA are likely to play nicely with detection/instance segmentation. Augmentation based methods and mixup style methods likely wont. Yet.

[ExtReMLapin]

Thank you for your answer, i'll give a try as soon as I'm confident enough with all the pytorch/detectron2 ecosystem