Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

63 Adding metadata and data source classification to produced netCDF files #68

Conversation

rosepearson
Copy link
Owner

@rosepearson rosepearson commented Apr 19, 2022

Write out netCDF files with metdata linking it to a particular GeoFabrics version and instruction set. Also add a layer of information classifying the source of each pixel (i.e. LiDAR, ocean bathymetry, river bathymetry, reference DEM, interpolated, etc):

  • Make code change
  • Update tests
    • Update / create new tests
    • Ensure these tests have the expected behavour
    • Test locally and ensure tests are passing
  • Ensure tests passing on GH Actions
  • Update documentation
  • Update package version (if needed)

…sor_local_files_after fails it's test now as xarray or rioxarray seems to ignore the fact I have created a dataset not a dataArray
…a update the package init to reference version.py for the version information. update the processor self.instructions as default values are used.
… how these are read in to ensure rasterio doesn't add a 'band' coordinate. Also added some explicit garbage collection as the freeing of memory associated with netCDF files is not being done otherwise.
… destructor. Also replaced explicit loads of netCDF files with the destructors. Tidied up the two updated tests slightly too.
@rosepearson rosepearson linked an issue Apr 20, 2022 that may be closed by this pull request
@rosepearson rosepearson changed the title 54 adding mean and median interpolation options in dem generation process 54 Adding metadata and data source classification to produced netCDF files Apr 20, 2022
@rosepearson rosepearson marked this pull request as ready for review April 20, 2022 21:06
@rosepearson
Copy link
Owner Author

rosepearson commented Apr 20, 2022

Kia ora @CyprienBosserelle - I've created this PR for moving to a more standard form of netCDF files, which hopefully is closer to both CF conventions and as well as the form we will ultimately want to achieve.

Could you have a look at one or two of the new benchmark files and make sure you're happy with them (at least for now)!

The current data source classifications are:

  • -1 - No data
  • 0 - Interpolated
  • 1 - LiDAR
  • 2 - Ocean bathymetry
  • 3 - River bathymetry
  • 4 - Reference DEM

The netCDF files now have a selection of standard and custom global attributes (source, title, description, geofabrics_instructions) and each variable also has a selection of standard and custom attributes (_FillValue, units, long_name, vertical_CRS (only z), classifications (only source_class)).

I've not yet moved to writing grouped netCDF files with different resolutions (a trivial change - but will need to decide on a system for group names). I might save that till next time you or Alice need a high resolution patch. What do you think?

image

rosepearson and others added 2 commits April 21, 2022 12:07
* Added support for computing the mean, median, min, max, std, or pt count in addition to the iwd for a set of lidar points in a pixel.
* Updated package version
* updated test for adding lidar averaging options
Copy link
Collaborator

@CyprienBosserelle CyprienBosserelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the demo test case... and looks great, the additional info is useful and I like the well describe attributes for each variables.

Good stuff!

@rosepearson rosepearson changed the title 54 Adding metadata and data source classification to produced netCDF files 63 Adding metadata and data source classification to produced netCDF files Apr 27, 2022
@rosepearson rosepearson merged commit 9fbb041 into main Apr 27, 2022
@rosepearson rosepearson deleted the 54-adding-mean-and-median-interpolation-options-in-dem-generation-process branch April 27, 2022 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

NetCDF preparation for BGFLOOD
2 participants