Update omps edr reader and hdf5_utils to handle OMPS SO2 data from FMI #1126

TAlonglong · 2020-03-31T11:57:49Z

Update OMPS edr reader and hdf5_utils to handle OMPS SO2 DATA from FMI SAMPO.

The OMPS SO2 data from FMI SAMPO in hdf5 format contains references that need special handling. This update does this.

Also added dataset names special to the data from FMI SAMPO

Closes Converting hdf5 attributes to string containing h5py.Reference of size 1 causes a AttributeError #630, closes satpy resample call -> numby.ndarray deepcopy error #883
Tests added
Tests passed
Passes flake8 satpy
Fully documented

…I SAMPO

ghost · 2020-03-31T11:58:05Z

Congratulations 🎉. DeepCode analyzed your code in 1.109 seconds and we found no issues. Enjoy a moment of no bugs ☀️.

👉 View analysis in DeepCode’s Dashboard

TAlonglong · 2020-03-31T12:01:37Z

Added updates after major help from @mraspaud, @pnuu and Timo

djhoese · 2020-03-31T12:15:01Z

This is related to #883 and #630 I think. @TAlonglong thanks for working on this. Any idea if this fixes things for these issues?

TAlonglong · 2020-03-31T12:17:37Z

Yes @djhoese both of them. One of them is mine. Forgot I issued that :-)

djhoese

Thanks again for this. I had a small suggestion on the name of the lon/lat datasets. Could you also add a test to the tests for the HDF5 utils module?

satpy/etc/readers/omps_edr.yaml

mraspaud

LGTM, just some consistency casing remarks

satpy/etc/readers/omps_edr.yaml

mraspaud · 2020-03-31T12:10:16Z

satpy/etc/readers/omps_edr.yaml

+    resolution: 50000
+    coordinates: [Longitude, Latitude]
+    file_type: omps_sampo
+    file_key: SCIENCE_DATA/CloudFraction


Also about consistency, should the dataset have same case and wording as the ones from the other files ?

coveralls · 2020-03-31T12:31:32Z

Coverage increased (+0.08%) to 89.605% when pulling 4820890 on TAlonglong:feature-omps-reader-update-for-sampo into 103d380 on pytroll:master.

codecov · 2020-03-31T12:31:37Z

Codecov Report

Merging #1126 into master will increase coverage by 0.08%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1126      +/-   ##
==========================================
+ Coverage   89.52%   89.60%   +0.08%     
==========================================
  Files         200      200              
  Lines       29379    29484     +105     
==========================================
+ Hits        26301    26420     +119     
+ Misses       3078     3064      -14

Impacted Files	Coverage Δ
satpy/readers/hdf5_utils.py	`92.42% <100.00%> (+0.62%)`	⬆️
satpy/readers/omps_edr.py	`100.00% <100.00%> (+3.84%)`	⬆️
satpy/tests/reader_tests/test_omps_edr.py	`98.98% <100.00%> (+0.66%)`	⬆️
satpy/readers/fci_l1c_fdhsi.py	`96.15% <0.00%> (-0.03%)`	⬇️
satpy/tests/writer_tests/test_cf.py	`98.53% <0.00%> (ø)`
satpy/tests/reader_tests/test_utils.py	`100.00% <0.00%> (ø)`
satpy/tests/reader_tests/test_ahi_hsd.py	`100.00% <0.00%> (ø)`
satpy/tests/reader_tests/test_geos_area.py	`100.00% <0.00%> (ø)`
satpy/tests/reader_tests/test_fci_l1c_fdhsi.py	`100.00% <0.00%> (ø)`
satpy/tests/reader_tests/test_seviri_l1b_hrit.py	`100.00% <0.00%> (ø)`
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 103d380...4820890. Read the comment docs.

TAlonglong · 2020-04-01T08:02:09Z

I have renamed the datsets to match with the other dataset and marked them with sampo to reflect the data source.

for the test on the hdf5_utils it looks to me this is already covered.

mraspaud · 2020-04-01T08:54:41Z

I'm good with the changes, looks good to me. One last thing: the reader seems to be missing from the index.rst documentation table, can you update it @TAlonglong ?

TAlonglong · 2020-04-01T11:02:17Z

OK. Updated.

But I see that the data DIMENSION_LIST stuff we added is not tested. I tried to add test to handle this, but what ever I try the DIMENSION_LIST attribute I add never get to where this is used in the reader.

mraspaud · 2020-04-01T11:14:19Z

If you want to test the dimension list, it should ideally be a list of references. Actually in the files it looks like this:

variable.attrs['DIMENSION_LIST'] = np.array([np.array([ref1]), np.array([ref2])])

TAlonglong · 2020-04-01T12:22:01Z

OK, I tried with this: https://github.com/pytroll/satpy/pull/1126/files#diff-c736dc353b90f5ae010c5b8b42896df5R114 but the attribute I set here is not handled. So it never get into the data.attrs. And I dont understand why this attr I set just disapear.

mraspaud · 2020-04-01T12:34:58Z

I can try to have a look later

mraspaud

LGTM

Update omps edr reader and hdf5_utils to handle OMPS SO2 data from FM…

0dcf940

…I SAMPO

TAlonglong requested review from djhoese and mraspaud as code owners March 31, 2020 11:57

mraspaud assigned TAlonglong Mar 31, 2020

mraspaud added component:readers enhancement code enhancements, features, improvements labels Mar 31, 2020

djhoese requested changes Mar 31, 2020

View reviewed changes

satpy/etc/readers/omps_edr.yaml Outdated Show resolved Hide resolved

mraspaud reviewed Mar 31, 2020

View reviewed changes

Trygve Aspenes added 3 commits April 1, 2020 09:57

Renamed dataset to match the other available datasates

8718363

Move reference handling to the end

0e55d7d

Add test to check new type of datasets

38a8466

Added reader to index.rst. Added test.

67de2c8

mraspaud and others added 2 commits April 3, 2020 11:28

Fix omps metadata retrieval

0acdb00

Add test for DIMENSION_LIST, and some other small missing parts

4820890

mraspaud approved these changes Apr 3, 2020

View reviewed changes

djhoese approved these changes Apr 3, 2020

View reviewed changes

djhoese merged commit c35e8c0 into pytroll:master Apr 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update omps edr reader and hdf5_utils to handle OMPS SO2 data from FMI #1126

Update omps edr reader and hdf5_utils to handle OMPS SO2 data from FMI #1126

TAlonglong commented Mar 31, 2020 •

edited by mraspaud

Loading

ghost commented Mar 31, 2020 •

edited by ghost

Loading

TAlonglong commented Mar 31, 2020

djhoese commented Mar 31, 2020

TAlonglong commented Mar 31, 2020

djhoese left a comment

mraspaud left a comment

mraspaud Mar 31, 2020

coveralls commented Mar 31, 2020 •

edited

Loading

codecov bot commented Mar 31, 2020 •

edited

Loading

TAlonglong commented Apr 1, 2020

mraspaud commented Apr 1, 2020

TAlonglong commented Apr 1, 2020

mraspaud commented Apr 1, 2020

TAlonglong commented Apr 1, 2020

mraspaud commented Apr 1, 2020

mraspaud left a comment

Update omps edr reader and hdf5_utils to handle OMPS SO2 data from FMI #1126

Update omps edr reader and hdf5_utils to handle OMPS SO2 data from FMI #1126

Conversation

TAlonglong commented Mar 31, 2020 • edited by mraspaud Loading

ghost commented Mar 31, 2020 • edited by ghost Loading

👉 View analysis in DeepCode’s Dashboard

TAlonglong commented Mar 31, 2020

djhoese commented Mar 31, 2020

TAlonglong commented Mar 31, 2020

djhoese left a comment

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

mraspaud Mar 31, 2020

Choose a reason for hiding this comment

coveralls commented Mar 31, 2020 • edited Loading

codecov bot commented Mar 31, 2020 • edited Loading

Codecov Report

TAlonglong commented Apr 1, 2020

mraspaud commented Apr 1, 2020

TAlonglong commented Apr 1, 2020

mraspaud commented Apr 1, 2020

TAlonglong commented Apr 1, 2020

mraspaud commented Apr 1, 2020

mraspaud left a comment

Choose a reason for hiding this comment

TAlonglong commented Mar 31, 2020 •

edited by mraspaud

Loading

ghost commented Mar 31, 2020 •

edited by ghost

Loading

coveralls commented Mar 31, 2020 •

edited

Loading

codecov bot commented Mar 31, 2020 •

edited

Loading