GSOC idea: Add Spatial Data Support to the Data Retriever

[ShivamNegi]

I would like to know more about this idea. Could you help me with insight of the project goals and implementation requirements. Thanks.

[henrykironde]

@ShivamNegi, The Data retriever handles tabular data from fetching it to cleaning and installing this data into the respective database systems like Postgres, Mysql, sqlite, csv and etc. The Data retriever also handles spacial data but to a limited support. The tool can fetch spacial data to your local system, however no installation of this data into any database management systems is supported.

Spatial data can be inform of points, lines, polygons and based on the data producers, different data structures are used to store the data. A few GIS file formats are

Raster formats
Esri grid – ASCII raster formats by Esri
GeoTIFF – TIFF with GIS relevant metadata
IMG – ERDAS image file format
JPEG2000 – Open-source raster format
netCDF-CF – netCDF file format with CF medata conventions for earth science data
Hierarchical Data Format (HDF5 /HDF4) – The HDF Group

Vector formats
Shapefile
Esri TIN - proprietary binary format for triangulated irregular network data used by Esri

Grid formats:
GeoTIFF – TIFF variant enriched with GIS relevant metadata

examples of libraries
GDAL/OGR, a library for reading and writing most of the formats

The project focus on adding support for this type of data. A good example could be Postgres using Postgis, data can be stored in to the database

[ShivamNegi]

So If I understood right. Python scripts will have to be made for installing above mentioned GIS formats in Postgis. These scripts will be added under ./engines. Retriever already has support for non GIS data formats. For adding support for GIS file formats scripts can be divided into three phases -

1. Adding Postgis in the build.
2. Downloading the different file formats like Esri grid, GeoTIFF etc.
3. Adding installation support of all of the different GIS file formats using GDAL.

[henrykironde]

Pretty much. So for Postgis we can add the scripts to the postgres engine in ./engine.

[ethanwhite]

Hi @ShivamNegi - glad to hear your interested in this one. Spatial data is becoming a such a common format it would be great to actively support importing it into databases.

In addition to PostgreSQL + PostGIS we'd also like to support SQLite + SpatiaLite.

[ethanwhite]

Closing for the moment, but feel free to reopen if you have more questions as you work on your proposal.