diff --git a/docs/tutorials/contribute_non_geodataset.ipynb b/docs/tutorials/contribute_non_geodataset.ipynb new file mode 100644 index 00000000000..3bae386e407 --- /dev/null +++ b/docs/tutorials/contribute_non_geodataset.ipynb @@ -0,0 +1,472 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Contribute a new Non-Geospatial Dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Open-source datasets have significantly accelerated Machine Learning Research. Within the area of Geospatial Machine Learning, the dataset can be singnificantly more complex to handle and load than more standard RGB-based Vision datasets for example. To spare the community from having to repeatly implement the logic over and over, TorchGeo supports dozens of datasets such that they can be downloaded and ready for use in a PyTorch framework within a single line of code. This tutorial will show how you can add a Non-Geospatial Dataset to this growing collection. As a reminder, TorchGeo differentiates between two dataset types: Geospatial and Non-Geospatial datasets. The difference is that Non-Geospatial datasets are integer index based datasets like the datasets one might be familar with from torchvision, while Geospatial datasets are indexed via geospatial coordinate bounding boxes. Non-geospatial datasets can still return geospatial and other metadata and should be specific to the remote-sensing domain. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Where to start\n", + "\n", + "If you are interested of an overview of existing geospatial datasets see for example the [Satellite-Image-Deep-Learning](https://github.com/satellite-image-deep-learning/datasets) page that contains a list of datasets and links to other dataset lists. \n", + "\n", + "Two aspects that will make it a lot easier to add the dataset are whether or not the dataset can be easily downloaded and whether or the dataset comes with a Github repository and publication that outlines how the authors intend the dataset to be used. These are not necessariy criteria, and sometimes it might be even more worth to add a dataset without an existing code base, precisely because the marginal contribution to the community might be greater because a use of the dataset does not necessitate writing the loading implementation from scratch." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Adding the dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once you have identified a dataset that you would like to add to TorchGeo, you could identify in what application category it might roughly fall in. For example, a segmentation dataset based on a collection of *.png* files, versus a classification dataset based on pre-defined image chips in *.tif* files. In the later case, if you find that the dataset contains *.tif* files that have very large pixel sizes, such that loading a single file might be costly, consider adding the dataset as a `Geospatial` dataset for easier indexing. Once, you have identified the \"task\" such as segmentation vs classification and the dataset format, see whether a dataset of the same or similar category exists in TorchGeo already. All datasets inherit from a `BaseClass` that provides an outline for the implementation logic as well as additional utility functions that should be reused. This reduces code duplication and makes it easier to unit test datasets.\n", + "\n", + "Adding a dataset to TorchGeo consists of roughly four parts:\n", + "- a `dataset_name.py` file itself that implements the logic of the dataset\n", + "- a `data.py` file that creates dummy data in the same structure and format as the original dataset for unit tests\n", + "- a `test_dataset_name.py` file that implements unit tests for the dataset\n", + "- an entry to the documentation page files: `non_geo_datasets.csv` and `datasets.rst`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The `dataset_name.py` file" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This file implements the logic to load a sample from the dataset as well as downloading the dataset automatically if possible. The new dataset inherits from a base class and the documentation string of the class should contain:\n", + "\n", + "- a short summary of the dataset\n", + "- outline features, such as the task \n", + "- outline the format the dataset comes in, i.e. file types, pixel dimensions etc.\n", + "- a proper reference to the dataset such as a link to the paper so users can adequatly cite the dataset when using it\n", + "- if required, a note about additional dependencies that are not part of TorchGeo's dependencies\n", + "\n", + "The dataset implementation itself should contain:\n", + "\n", + "- method to create an index structure the dataset can iterate over to load samples. This index structure also defines the length (`__len__`) of the dataset, i.e. how many individual samples can be loaded from the dataset\n", + "- a `__getitem__` method that takes an integer index argument, loads a sample of the dataset, and returns its components in a dictionary\n", + "- a `_verify` method that checks whether the dataset can be found on the filesystem, has already been downloaded and only needs to be extracted, or downloads and extracts the dataset from the web\n", + "- a `plot` method that can visually display a single sample of the dataset\n", + "\n", + "The code below attempts to roughly outline the parts required for a new `NonGeoDataset`. Specfics are of course very dependent on the type of dataset you want to add, but this template and other existing datasets should give you a decent starting point." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "from typing import Any\n", + "from torchgeo.datasets import NonGeoDataset\n", + "from pathlib import Path\n", + "from matplotlib.pyplot import Figure\n", + "\n", + "class MyNewDataset(NonGeoDataset):\n", + " \"\"\"MyNewDatasets\n", + " \n", + " Small Sumamry of the dataset\n", + "\n", + " Dataset features:\n", + " * number of classes, different sensors, area covered etc.\n", + "\n", + " Dataste format:\n", + " * what file format and shape the input data comes in\n", + " * what file format and shape the target data comes in\n", + " * possible metadata files\n", + "\n", + " Mention publication\n", + " \n", + " .. versionadded: tag number here\n", + " \"\"\"\n", + " # in this part of the code you can define class attributes such as list of class names, color maps,\n", + " # url and checksums for data download, and other attributes that one might require repeatedly in the\n", + " # subsequent logic\n", + " \n", + " def __init__(self, root: Path, download: bool=False, transform: bool=None) -> None:\n", + " \"\"\"Initialize the dataset.\n", + "\n", + " The init arguments can include additional arguments, for example a dedicated split of the data,\n", + " being able to subselect specific modalities or even individual bands, or other arguments that give\n", + " dedicated control over the dataset to run experiments that might come from publication itself or be\n", + " helpful to the community. They should be reasonable defaults.\n", + "\n", + " Args:\n", + " root: root directory where the dataset is stored\n", + " download: whether to download the dataset if it is not found in the root directory. Defaults to False.\n", + " transform: transformation to apply to the dataset. Defaults to None.\n", + " \"\"\"\n", + " \n", + " def _load_files(self) -> Any:\n", + " \"\"\"This method should create a data structure from which one can retrieve individual samples\n", + " and from which the total number of dataset samples can be derived. You can name the method as you like\n", + " but sticking with naming schemes that are common across other implemented datasets is recommended.\n", + " \"\"\"\n", + "\n", + " def __len__(self) -> int:\n", + " \"\"\"Define the length of the dataset, so the total number of samples can be retrieved for the specific setting of arguments\n", + " specified in the __init__ method.\"\"\"\n", + "\n", + "\n", + " def __getitem__(self, index: int) -> dict[str, Any]:\n", + " \"\"\"Based on an index, return a dictionary with the data and target information for the dataset.\n", + "\n", + " This might involve separate class methods to load the data and target information etc, but the __getitem__\n", + " decides \"how to assemble\" a dataset sample.\n", + " \"\"\"\n", + " \n", + " def plot(self) -> Figure:\n", + " \"\"\"Plot a sample of the dataset for visualization purposes.\n", + " \n", + " This might involve subselecting RGB bands that can be displayed, displaying class labels in segmentation tasks etc.\n", + " Implemented datasets have already covered a wide range of visualization techniques, so it should be helpful to check\n", + " for similar datasets.\n", + " \"\"\"\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The `data.py` file" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `data.py` file is placed under `tests/data/dataset_name/` directory and creates a smaller dummy dataset that replicates the features and formats of the actual full datasets for unit tests. The script should therefore:\n", + "\n", + "- replicate the directory structure\n", + "- replicate the naming scheme of directories and individual files\n", + "- replicate roughly the value ranges found in the dataset, for example contain the same number of classes\n", + "- use the same compression scheme to simulate downloading the dataset and its checksum for verification\n", + "\n", + "This is usually highly dependent on the dataset format and structure the new dataset comes in. However, again below is an outline of the usual building blocks of a `data.py` script, for example an image segmentation dataset with 10 classes. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```python\n", + "\n", + "import hashlib\n", + "import os\n", + "import shutil\n", + "\n", + "import numpy as np\n", + "from PIL import Image\n", + "\n", + "# Define the root directory and subdirectories\n", + "root_dir = 'my_new_dataset'\n", + "sub_dirs = ['sub_dir_1', 'sub_dir_2', 'sub_dir_3']\n", + "splits = ['train', 'val', 'test']\n", + "\n", + "image_file_names = [\n", + " 'sample_1.png',\n", + " 'sample_2.png',\n", + " 'sample_3.png',\n", + "]\n", + "\n", + "IMG_SIZE = 32\n", + "\n", + "# Function to create dummy input images\n", + "def create_input_image(path: str, shape: tuple[int], pixel_values: list[int]) -> None:\n", + " data = np.random.choice(pixel_values, size=shape, replace=True).astype(np.uint8)\n", + " img = Image.fromarray(data)\n", + " img.save(path)\n", + "\n", + "# function to create dummy targets\n", + "def create_target_images(split: str, filename: str) -> None:\n", + " target_pixel_values = range(10)\n", + " path = os.path.join(root_dir, 'target', split, filename)\n", + " create_dummy_image(path, (IMG_SIZE, IMG_SIZE), target_pixel_values)\n", + "\n", + "# create a new clean version when rerunning script\n", + "if os.path.exists(root_dir):\n", + " shutil.rmtree(root_dir)\n", + "\n", + "# Create the directory structure\n", + "for sub_dir in sub_dirs:\n", + " for split in splits:\n", + " os.makedirs(os.path.join(root_dir, sub_dir, split), exist_ok=True)\n", + "\n", + "# Create dummy data for all splits and filenames\n", + "for split in splits:\n", + " for filename in zone_file_names:\n", + " create_input_image(split, filename)\n", + " create_target_images(split, filename.replace('_', '_target_'))\n", + "\n", + "# zip directory \n", + "shutil.make_archive(root_dir, 'zip', '.', root_dir)\n", + "\n", + "# compute checksum\n", + "def md5(fname: str) -> str:\n", + " hash_md5 = hashlib.md5()\n", + " with open(fname, 'rb') as f:\n", + " for chunk in iter(lambda: f.read(4096), b''):\n", + " hash_md5.update(chunk)\n", + " return hash_md5.hexdigest()\n", + "\n", + "\n", + "md5sum = md5('dummy_data.zip')\n", + "print(f'MD5 checksum: {md5sum}')\n", + "# add this checksum to the test_dataset_name.py file to mock dataset download if applicable\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## The `test_dataset_name.py` file" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `test_dataset_name.py` file is placed under the `tests/datasets/` directory. This file implements the unit tests for the dataset, such that every line of code in `dataset_name.py` is tested. The logic of the individual test cases will likely be very similar to existing test files so you can look at those to to see how you can test the individual parts of the dataset logic." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "```python\n", + "\n", + "from _pytest.fixtures import SubRequest\n", + "from pytest import MonkeyPatch\n", + "\n", + "import torchgeo.datasets.utils\n", + "from torchgeo.datasets import MyNewDataset\n", + "\n", + "\n", + "def download_url(url: str, root: str | Path, *args: str, **kwargs: str) -> None:\n", + " shutil.copy(url, root)\n", + "\n", + "\n", + "class TestMyNewDataset:\n", + " # pytest fixtures can be used to define variables to test different argument\n", + " # configurations to test, for example the different splits of the dataset\n", + " # or subselection of modalities/bands\n", + " @pytest.fixture(\n", + " params=product(['train', 'val', 'test'])\n", + " )\n", + " def dataset(\n", + " self, monkeypatch: MonkeyPatch, tmp_path: Path, request: SubRequest\n", + " ) -> MyNewDataset:\n", + " split: str = request.param\n", + " # monkeypatch can overwrite the class attributes defined above the __init__ method\n", + " # and use the specific unit tests settings to mock behavior such as downloading\n", + " monkeypatch.setattr(torchgeo.datasets.my_new_dataset, 'download_url', download_url)\n", + "\n", + " # for downloads you can also add the specific checksums that the data.py script yields\n", + " # for the dummy data\n", + "\n", + " root = tmp_path\n", + " transforms = nn.Identity()\n", + " return MyNewDataset(\n", + " root=root, split=split, transforms=transforms, download=True, checksum=True\n", + " )\n", + "\n", + " def test_getitem(self, dataset: MyNewDataset) -> None:\n", + " # retrieve a sample and check some of the desired properties\n", + " x = dataset[0]\n", + " assert isinstance(x, dict)\n", + " assert isinstance(x['image'], torch.Tensor)\n", + " assert isinstance(x['label'], torch.Tensor)\n", + "\n", + " # for all the additional class arguments, check what happens if you define invalid parameters\n", + " def test_invalid_split(self) -> None:\n", + " with pytest.raises(AssertionError):\n", + " MyNewDataset(split='foo')\n", + " # for example if you have a list of bands, check what happens if you define invalid bands\n", + " def test_invalid_bands(self) -> None:\n", + " with pytest.raises(ValueError):\n", + " MyNewDataset(bands=('OK', 'BK'))\n", + "\n", + " # test the length of the dataset, this should coincide with the dummy data created in data.py\n", + " def test_len(self, dataset: MyNewDataset) -> None:\n", + " assert len(dataset) == 2\n", + "\n", + " # test the logic when the dataset is already downloaded\n", + " def test_already_downloaded(self, dataset: MyNewDataset, tmp_path: Path) -> None:\n", + " MyNewDataset(root=tmp_path, download=True)\n", + "\n", + " # test the logic when the dataset is already downloaded but not extracted\n", + " def test_already_downloaded_not_extracted(\n", + " self, dataset: MyNewDataset, tmp_path: Path\n", + " ) -> None:\n", + " shutil.rmtree(dataset.root)\n", + " download_url(dataset.url, root=tmp_path)\n", + " MyNewDataset(root=tmp_path, download=False)\n", + "\n", + " # mock the download function to test the logic when the dataset is not downloaded\n", + " def test_not_downloaded(self, tmp_path: Path) -> None:\n", + " with pytest.raises(DatasetNotFoundError, match='Dataset not found'):\n", + " MyNewDataset(tmp_path)\n", + "\n", + " # test the plotting method through something like the following\n", + " def test_plot(self, dataset: MyNewDataset) -> None:\n", + " x = dataset[0].copy()\n", + " dataset.plot(x, suptitle='Test')\n", + " plt.close()\n", + " dataset.plot(x, show_titles=False)\n", + " plt.close()\n", + " x['prediction'] = x['label'].clone()\n", + " dataset.plot(x)\n", + " plt.close()\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Documentation Entries" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The entry point for new and experienced users of domain libraries is often the dedicated documentation page that accompanies a Github repository. TorchGeo uses the popular `sphinx` framework to build its documentation. To display the documentation strings you have written in `dataset_name.py` on the actual documentation page, you need to create an entry in `docs/api/datastes.rst` in alphabetical order:\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "```rst\n", + "Dataset Name\n", + "^^^^^^^^^^^^\n", + "\n", + ".. autoclass:: Dataset Class Name\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Additionally, add a row in the `non_geo_datasets.csv` file under `docs/api/datasets` to include the dataset in the overview table." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Linters\n", + "\n", + "See the [linter docs](https://torchgeo.readthedocs.io/en/stable/user/contributing.html#linters) for an overview of linters that TorchGeo employs and how to apply them during commits for example. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Test Coverage" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "TorchGeo maintains a test coverage of 100%. This means, that every line of code written within the torchgeo directory is being called by some unit test. The [testing docs](https://torchgeo.readthedocs.io/en/stable/user/contributing.html#tests) provide instructions how you can test the coverage locally for the `dataset_new.py` file that you are adding." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Final Checklist" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This final checklist might provide a useful overview of the individual parts discussed in this tutorial. You definitely do not need to check all boxes, before submitting a PR. If you have any questions feel free to ask in the Slack channel or open a PR already such that maintainers or other community members can answer specific questions or give pointers. If you want to run your PR as a work of progress, such that the CI tests are run against your code while you work on ticking more boxes you can also convert the PR to a draft on the right side." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- Dataset implementation in `dataset_name.py`\n", + " - Class doc string containining:\n", + " - Summary intro\n", + " - Dataset features\n", + " - Dataset format\n", + " - Link to publication\n", + " - `versionadded` tag\n", + " - if applicable a note on additional dependencies\n", + " - all class methods have docstrings\n", + " - all class methods have argument and return type hints, mypy (the tool that checks type hints) can be confusing at the beginning so don't hesitate to ask for help\n", + " - if dataset is on Huggingface, url link should contain the commit hash\n", + " - checksum added\n", + " - plot method that can display a single sample from the dataset (you can add the resulting figure in your PR description)\n", + " - add the dataset to `torchgeo/datastes/__init__`\n", + " - microsoft copy right at top of the file\n", + "- Dummy data script `data.py`\n", + " - replicate directory structure\n", + " - replicate naming of directory and files\n", + " - for image based datasets use not the actual pixelsize but smaller extend, commonly we use 32x32\n", + "- Unit tests `test_dataset_name.py`\n", + " - 100% test coverage \n", + "- Documentation with `non_geo_datasets.csv` and `datasets.rst`\n", + " - entry in `datasets.rst`\n", + " - entry in `non_geo_datasets.csv`\n", + " - documentation displays properly, this can be checked locally or via the GitHub CI tests under `docs/readthedocs.org:torchgeo`" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "torchEnv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.13" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}