Add Cityscapes format

CHANGELOG.md

@@ -11,11 +11,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Support for escaping in attribiute values in LabelMe format (<https://github.com/openvinotoolkit/datumaro/issues/49>)
 - Support for Segmentation Splitting (<https://github.com/openvinotoolkit/datumaro/pull/223>)
 - Support for CIFAR-10/100 dataset format (<https://github.com/openvinotoolkit/datumaro/pull/225>, <https://github.com/openvinotoolkit/datumaro/pull/243>)
-- Support COCO panoptic and stuff format (<https://github.com/openvinotoolkit/datumaro/pull/210>)
+- Support for COCO panoptic and stuff format (<https://github.com/openvinotoolkit/datumaro/pull/210>)
 - Documentation file and integration tests for Pascal VOC format (<https://github.com/openvinotoolkit/datumaro/pull/228>)
 - Support for MNIST and MNIST in CSV dataset formats (<https://github.com/openvinotoolkit/datumaro/pull/234>)
 - Documentation file for COCO format (<https://github.com/openvinotoolkit/datumaro/pull/241>)
 - Documentation file and integration tests for YOLO format (<https://github.com/openvinotoolkit/datumaro/pull/246>)
+- Support for Cityscapes dataset format (<https://github.com/openvinotoolkit/datumaro/pull/249>)
 
 ### Changed
 - LabelMe format saves dataset items with their relative paths by subsets without changing names (<https://github.com/openvinotoolkit/datumaro/pull/200>)

README.md

@@ -137,6 +137,7 @@ CVAT annotations                             ---> Publication, statistics etc.
   - [MNIST](http://yann.lecun.com/exdb/mnist/) (`classification`)
   - [MNIST in CSV](https://pjreddie.com/projects/mnist-in-csv/) (`classification`)
   - [CamVid](http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/)
+  - [Cityscapes](https://www.cityscapes-dataset.com/)
   - [CVAT](https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/xml_format.md)
   - [LabelMe](http://labelme.csail.mit.edu/Release3.0)
   - [ICDAR13/15](https://rrc.cvc.uab.es/?ch=2) (`word_recognition`, `text_localization`, `text_segmentation`)

datumaro/components/extractor.py

@@ -7,6 +7,7 @@
 from glob import iglob
 from typing import Iterable, List, Dict, Optional
 import numpy as np
+import os
 import os.path as osp
 
 import attr
@@ -236,7 +237,7 @@ def __eq__(self, other):
 class CompiledMask:
     @staticmethod
     def from_instance_masks(instance_masks,
-            instance_ids=None, instance_labels=None):
+            instance_ids=None, instance_labels=None, dtype=None):
         from datumaro.util.mask_tools import make_index_mask
 
         if instance_ids is not None:
@@ -266,7 +267,7 @@ def from_instance_masks(instance_masks,
         m, idx, instance_id, class_id = next(it)
         if not class_id:
             idx = 0
-        index_mask = make_index_mask(m, idx)
+        index_mask = make_index_mask(m, idx, dtype=dtype)
         instance_map.append(instance_id)
         class_map.append(class_id)
 
@@ -282,8 +283,8 @@ def from_instance_masks(instance_masks,
         else:
             merged_instance_mask = np.array(instance_map,
                 dtype=np.min_scalar_type(instance_map))[index_mask]
-        merged_class_mask = np.array(class_map,
-            dtype=np.min_scalar_type(class_map))[index_mask]
+        dtype_mask = dtype if dtype else np.min_scalar_type(class_map)
+        merged_class_mask = np.array(class_map, dtype=dtype_mask)[index_mask]
 
         return __class__(class_mask=merged_class_mask,
             instance_mask=merged_instance_mask)
@@ -673,7 +674,11 @@ def __call__(self, path, **extra_params):
     @classmethod
     def _find_sources_recursive(cls, path, ext, extractor_name,
             filename='*', dirname='', file_filter=None, max_depth=3):
-        if path.endswith(ext) and osp.isfile(path):
+
+        if (path.endswith(ext) and osp.isfile(path)) or \
+                (not ext and osp.isdir(path) and dirname and \
+                os.sep + osp.normpath(dirname) + os.sep in \
+                    osp.abspath(path) + os.sep):
             sources = [{'url': path, 'format': extractor_name}]
         else:
             sources = []