draft creating datasets for sentinelbench

[wangyi111]

This is a draft of PRs to add datasets from SentinelBench. This specific PR starts from the SenBench-Cloud-S3 dataset for cloud segmentation. It also initializes a wrapper to dynamically load a dataset from SentinelBench with the dataset name. The latter is just a rough idea, not sure if it's good or not.

[wangyi111]

@adamjstewart

[adamjstewart]

My first thoughts on seeing these:

• Each dataset itself is a lot of code, so it makes sense to have each one in a separate file
• We may want some kind of base class containing shared code (e.g., download, extract, len)

So as to avoid torchgeo/datasets/ becoming too cluttered, I would propose a torchgeo/datasets/senbench/ directory. This could look like:

torchgeo/datasets/
    __init__.py: from senbench import *
    senbench/
        __init__.py: from * import *, wrapper dataset
        base.py: shared base class if necessary
        cloud_s3.py: class SenBenchCloudS3
        lulc_s3.py: class SenBenchLC100SegS3
        ...

Here, we wouldn't actually import import *, we would import each individual class from each file.

This will be our first big meta dataset/benchmarking suite. Since it's the first, we have more flexibility, but also more decisions to make. Let's mull this over for a bit and I'll consult the other TorchGeo devs.

[adamjstewart]

Could also organize subfiles by level or sensor. Need to look at the full table again to think about how to organize.

[wangyi111]

My first thoughts on seeing these:

• Each dataset itself is a lot of code, so it makes sense to have each one in a separate file
• We may want some kind of base class containing shared code (e.g., download, extract, len)

So as to avoid torchgeo/datasets/ becoming too cluttered, I would propose a torchgeo/datasets/senbench/ directory. This could look like:

torchgeo/datasets/
    __init__.py: from senbench import *
    senbench/
        __init__.py: from * import *, wrapper dataset
        base.py: shared base class if necessary
        cloud_s3.py: class SenBenchCloudS3
        lulc_s3.py: class SenBenchLC100SegS3
        ...

Here, we wouldn't actually import import *, we would import each individual class from each file.

This will be our first big meta dataset/benchmarking suite. Since it's the first, we have more flexibility, but also more decisions to make. Let's mull this over for a bit and I'll consult the other TorchGeo devs.

this sounds a good idea!