🔴 🔴 🔴 Added `segmentation maps` support for DPT image processor #34345

simonreise · 2024-10-23T12:21:25Z

Added `segmentation maps` support for DPT image processor

Most of image processors for vision models that support semantic segmentation task accept images and segmentation_maps as inputs, but for some reason DPT image processor does not process segmentation maps, only images. This PR can make code that one uses for training or evaluation of semantic segmentation models more reusable, as now DPT image processor can process segmentation maps as most of other image processors do.

I also added do_reduce_labels arg because other image processors that support segmentation masks use it.

I added two new tests: one that tests segmentation_masks support and one that tests if do_reduce_labels work as expected.

Most of the code is adapted from BEIT image processor.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@amyeroberts, @qubvel

LysandreJik · 2024-10-24T12:35:34Z

cc @molbap as well in case bandwidth permits

molbap

LGTM - just a small refactor of the method to be more aligned with existing models!

molbap · 2024-10-24T14:15:29Z

tests/models/dpt/test_image_processing_dpt.py

+
+    def test_call_segmentation_maps(self):
+        # Initialize image_processing
+        image_processing = self.image_processing_class(**self.image_processor_dict)


nit, image_processor would be better

Renamed image_processing to image_processor. Should I also rename it in the other tests?

molbap · 2024-10-24T14:20:55Z

src/transformers/models/dpt/image_processing_dpt.py

+        if segmentation_maps is not None:
+            segmentation_maps = [to_numpy_array(segmentation_map) for segmentation_map in segmentation_maps]
+
+            # Add channel dimension if missing - needed for certain transformations
+            if segmentation_maps[0].ndim == 2:
+                added_channel_dim = True
+                segmentation_maps = [segmentation_map[None, ...] for segmentation_map in segmentation_maps]
+                input_data_format = ChannelDimension.FIRST
+            else:
+                added_channel_dim = False
+                if input_data_format is None:
+                    input_data_format = infer_channel_dimension_format(segmentation_maps[0], num_channels=1)
+
+            if do_reduce_labels:
+                segmentation_maps = [self.reduce_label(segmentation_map) for segmentation_map in segmentation_maps]
+
+            if do_resize:
+                segmentation_maps = [
+                    self.resize(
+                        image=segmentation_map,
+                        size=size,
+                        resample=resample,
+                        keep_aspect_ratio=keep_aspect_ratio,
+                        ensure_multiple_of=ensure_multiple_of,
+                        input_data_format=input_data_format,
+                    )
+                    for segmentation_map in segmentation_maps
+                ]
+
+            if do_pad:
+                segmentation_maps = [
+                    self.pad_image(
+                        image=segmentation_map, size_divisor=size_divisor, input_data_format=input_data_format
+                    )
+                    for segmentation_map in segmentation_maps
+                ]
+
+            # Remove extra channel dimension if added for processing
+            if added_channel_dim:
+                segmentation_maps = [segmentation_map.squeeze(0) for segmentation_map in segmentation_maps]
+            segmentation_maps = [segmentation_map.astype(np.int64) for segmentation_map in segmentation_maps]
+
+            data["labels"] = segmentation_maps


Perfect - if there isn't any difference with Beit, can this be wrapped in a _preprocess_segmentation_map() method in a loop, that can be flagged as # Copied from ... the beit image processor?

Wrapped segmentation map preprocessing code to _preprocess_segmentation_map(), and also moved image preprocessing to separate _preprocess_image() function and general preprocessing functionality to _preprocess() function.

simonreise · 2024-11-11T15:16:49Z

Could you please re-review the pull request? In the last commit I made all the changes you asked for: wrapped segmentation map preprocessing code to separate functions, added comments and renamed a variable in tests. Do I need to make any other changes to the code?

molbap · 2024-11-18T16:30:46Z

hey @simonreise , will review in a moment, we were all at a team gathering last week hence the inactivity. On my radar!

ArthurZucker · 2024-11-19T11:43:19Z

@molbap you are fobidden to work for this week 🤣 go and rest, @yonigozlan will have a look! 🤗

qubvel · 2024-12-17T12:37:04Z

friendly ping @yonigozlan

yonigozlan

Thanks for working on this!
Looks good to me, and something that would be useful to have.
Just left some comments about adding # Copied from statements where needed, and about breaking backward compatibility (mainly intended for a core maintainer)

yonigozlan · 2024-12-18T04:03:38Z

src/transformers/models/dpt/image_processing_dpt.py

+    def reduce_label(self, label: ImageInput) -> np.ndarray:
+        label = to_numpy_array(label)
+        # Avoid using underflow conversion
+        label[label == 0] = 255
+        label = label - 1
+        label[label == 254] = 255
+        return label


This seems to be fully copied from beit image processor, you should add a # Copied from statement above if that's the case :)

yonigozlan · 2024-12-18T04:07:17Z

src/transformers/models/dpt/image_processing_dpt.py

+    def __call__(self, images, segmentation_maps=None, **kwargs):
+        # Overrides the `__call__` method of the `Preprocessor` class such that the images and segmentation maps can both
+        # be passed in as positional arguments.
+        return super().__call__(images, segmentation_maps=segmentation_maps, **kwargs)


Same here for adding a # Copied from, and same for all the other methods copied from beit as well.

yonigozlan · 2024-12-18T04:13:53Z

src/transformers/models/dpt/image_processing_dpt.py

    @filter_out_non_signature_kwargs()
    def preprocess(
        self,
        images: ImageInput,
+        segmentation_maps: Optional[ImageInput] = None,


This is a bit tricky as it could be a breaking change, if some users use do_resize etc. as args and not kwargs. However this would not be good practice, and I don't see any way of adding segmentation_maps processing without breaking BC. I'll let a core maintainer give the green light on this or not.

yonigozlan · 2024-12-18T04:17:41Z

tests/models/dpt/test_image_processing_dpt.py

+
+    def test_call_segmentation_maps(self):


Same as before about the # Copied from, if this and the following tests are indeed fully copied from beit

yonigozlan · 2024-12-19T17:01:58Z

Thanks, but the # Copied from statement must be placed above the function definition. You can refer to other parts of the library to see how it's done.
This is not just for information purposes; it enables the make fix-copies CLI command to propagate any modifications in the original function to its copied versions.

After making the required changes, you can ensure everything is in order by running the make fixup command.

yonigozlan · 2024-12-20T19:32:14Z

Thanks for iterating! you just have to rebase on main and check that the tests are still passing, then LGTM!

…ithub.com/simonreise/transformers into segmentation-maps-for-dpt-image-processor

yonigozlan · 2025-01-03T13:39:38Z

Everything looks good to me, pinging @ArthurZucker for a final review.
There's a potential breaking change @ArthurZucker, see this comment

simonreise · 2025-01-10T10:39:19Z

There was a merge conflict that appeared after #35439 was merged into main. So I also changed the order of do_rescale and is_scaled_image in the code

ArthurZucker

Don't mind breaking this, but we need to add a 🔴 on the PR name to sort it out when releasing! 🤗

…ingface#34345) * Added `segmentation_maps` support for DPT image processor * Added tests for dpt image processor * Moved preprocessing into separate functions * Added # Copied from statements * Fixed # Copied from statements * Added `segmentation_maps` support for DPT image processor * Added tests for dpt image processor * Moved preprocessing into separate functions * Added # Copied from statements * Fixed # Copied from statements

simonreise added 2 commits October 23, 2024 14:45

Added segmentation_maps support for DPT image processor

03cfd0a

Added tests for dpt image processor

0d97965

molbap reviewed Oct 24, 2024

View reviewed changes

Moved preprocessing into separate functions

c900bd9

simonreise requested a review from molbap November 11, 2024 15:16

qubvel added Vision Processing labels Nov 18, 2024

ArthurZucker requested review from yonigozlan and removed request for molbap November 19, 2024 11:42

yonigozlan reviewed Dec 18, 2024

View reviewed changes

Added # Copied from statements

2a134f6

Fixed # Copied from statements

d94832b

simonreise and others added 8 commits December 21, 2024 13:53

Added segmentation_maps support for DPT image processor

0bff533

Added tests for dpt image processor

dd1a4e7

Moved preprocessing into separate functions

d4c2857

Added # Copied from statements

6f4e61e

Fixed # Copied from statements

b65774e

Merge branch 'segmentation-maps-for-dpt-image-processor' of https://g…

a30323c

…ithub.com/simonreise/transformers into segmentation-maps-for-dpt-image-processor

Merge branch 'main' into segmentation-maps-for-dpt-image-processor

696e89a

Merge branch 'main' into segmentation-maps-for-dpt-image-processor

47ab04c

qubvel requested a review from ArthurZucker January 7, 2025 11:04

Merge branch 'main' into segmentation-maps-for-dpt-image-processor

a43d2e4

simonreise requested review from qubvel and Rocketknight1 as code owners January 10, 2025 10:29

ArthurZucker approved these changes Jan 27, 2025

View reviewed changes

ArthurZucker changed the title ~~Added segmentation maps support for DPT image processor~~ 🔴 🔴 🔴 Added segmentation maps support for DPT image processor Jan 27, 2025

ArthurZucker merged commit 5450e7c into huggingface:main Jan 27, 2025
9 checks passed

simonreise deleted the segmentation-maps-for-dpt-image-processor branch January 30, 2025 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🔴 🔴 🔴 Added `segmentation maps` support for DPT image processor #34345

🔴 🔴 🔴 Added `segmentation maps` support for DPT image processor #34345

simonreise commented Oct 23, 2024 •

edited

Loading

LysandreJik commented Oct 24, 2024

molbap left a comment

molbap Oct 24, 2024

simonreise Oct 29, 2024

molbap Oct 24, 2024

simonreise Oct 29, 2024

simonreise commented Nov 11, 2024

molbap commented Nov 18, 2024

ArthurZucker commented Nov 19, 2024

qubvel commented Dec 17, 2024

yonigozlan left a comment

yonigozlan Dec 18, 2024

simonreise Dec 19, 2024

yonigozlan Dec 18, 2024

yonigozlan Dec 18, 2024

yonigozlan Dec 18, 2024

yonigozlan commented Dec 19, 2024

yonigozlan commented Dec 20, 2024

yonigozlan commented Jan 3, 2025

simonreise commented Jan 10, 2025

ArthurZucker left a comment

🔴 🔴 🔴 Added segmentation maps support for DPT image processor #34345

🔴 🔴 🔴 Added segmentation maps support for DPT image processor #34345

Conversation

simonreise commented Oct 23, 2024 • edited Loading

Added segmentation maps support for DPT image processor

Before submitting

Who can review?

LysandreJik commented Oct 24, 2024

molbap left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonreise commented Nov 11, 2024

molbap commented Nov 18, 2024

ArthurZucker commented Nov 19, 2024

qubvel commented Dec 17, 2024

yonigozlan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yonigozlan commented Dec 19, 2024

yonigozlan commented Dec 20, 2024

yonigozlan commented Jan 3, 2025

simonreise commented Jan 10, 2025

ArthurZucker left a comment

Choose a reason for hiding this comment

🔴 🔴 🔴 Added `segmentation maps` support for DPT image processor #34345

🔴 🔴 🔴 Added `segmentation maps` support for DPT image processor #34345

simonreise commented Oct 23, 2024 •

edited

Loading

Added `segmentation maps` support for DPT image processor