Add reader for MWS onboard EPS-SG-A #2120

adybbroe · 2022-06-02T10:18:00Z

This is a first draft reader for the EPS-SG MWS level-1b netCDF file format.

Closes #xxxx
Tests added
Fully documented
Add your name to AUTHORS.md if not there already

This is an early draft so far, and several things needs to be addressed:

Rearrange the FrquencyRange and related classes defined in the AAPP-MHS reader so they can be reused in several readers
Add unittests for the reader
Decide how the channel comparisons should behave!
Add handling of auxiliary data sets, see FCI and VII readers

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

adybbroe · 2022-06-02T10:19:55Z

Here a sketch of some parts of the MWS temperature sounding channels, to illustrate the quadruple side bands (as for instance also seen on AMSU-A):

codecov · 2022-06-02T10:37:34Z

Codecov Report

Merging #2120 (40e8461) into main (acd0745) will increase coverage by 0.10%.
The diff coverage is 98.93%.

@@            Coverage Diff             @@
##             main    #2120      +/-   ##
==========================================
+ Coverage   94.03%   94.14%   +0.10%     
==========================================
  Files         289      293       +4     
  Lines       44564    45091     +527     
==========================================
+ Hits        41904    42449     +545     
+ Misses       2660     2642      -18

Flag	Coverage Δ
behaviourtests	`4.68% <0.00%> (-0.07%)`	⬇️
unittests	`94.79% <98.93%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
satpy/readers/yaml_reader.py	`97.50% <ø> (+0.22%)`	⬆️
satpy/readers/pmw_channels_definitions.py	`97.60% <97.60%> (ø)`
satpy/readers/mws_l1b.py	`98.51% <98.51%> (ø)`
satpy/readers/aapp_mhs_amsub_l1c.py	`95.89% <100.00%> (+2.76%)`	⬆️
satpy/tests/reader_tests/test_mws_l1b_nc.py	`100.00% <100.00%> (ø)`
satpy/tests/test_dataset.py	`100.00% <100.00%> (ø)`
satpy/tests/test_yaml_reader.py	`99.60% <100.00%> (ø)`
satpy/tests/reader_tests/test_hrit_base.py	`96.55% <0.00%> (-1.67%)`	⬇️
satpy/readers/utils.py	`91.95% <0.00%> (-0.39%)`	⬇️
satpy/utils.py	`24.69% <0.00%> (-0.31%)`	⬇️
... and 23 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

coveralls · 2022-06-02T10:56:02Z

Coverage increased (+0.1%) to 94.741% when pulling 40e8461 on adybbroe:add-epssg-mws-level1b-reader into acd0745 on pytroll:main.

adybbroe · 2022-06-08T14:24:29Z

I have a hard time deciding how the channel/band inter-comparison should behave here! :-|

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

adybbroe · 2022-06-15T11:14:54Z

Here, a quick example usage:

In [1]: from satpy import Scene
In [2]: AREAID = 'nsper_swe'
In [3]: FILENAMES = ['/path-to-my-epssg-mws-testdata/W_XX-EUMETSAT-Darmstadt,SAT,SGA1-MWS-1B-RAD_C_EUMT_20210609095009_G_D_20070912084321_20070912102225_T_N____.nc']
In [4]: scn = Scene(filenames=FILENAMES, reader='mws_l1b_nc')
In [5]: from satpy.dataset import DataQuery
In [6]: frq_24 = DataQuery(frequency_range=23.8)
In [7]: scn.load([frq_24])
In [8]: local = scn.resample(AREAID)
In [9]: local.show(frq_24)

It is also possible to load the data simply by specifying the channel number, like: scn.load(['1'])

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

gerritholl

Lots of new content in a part of Satpy I'm not too familiar with, so do with my questions and remarks what you wish. There seems to be some code duplication, though.

satpy/readers/mws_l1b.py

gerritholl · 2022-07-29T09:02:12Z

satpy/readers/pmw_channels_definitions.py

+        return False
+
+    def distance(self, value):
+        """Get the distance from value."""


Here too could be some info on how this distance is defined.

Added more explanation.

satpy/readers/pmw_channels_definitions.py

gerritholl · 2022-07-29T09:07:46Z

satpy/readers/pmw_channels_definitions.py

+    def __lt__(self, other):
+        """Compare to another frequency."""
+        if other is None:
+            return False
+        return super().__lt__(other)
+
+    def __gt__(self, other):
+        """Compare to another frequency."""
+        if other is None:
+            return True
+        return super().__gt__(other)
+
+    def __hash__(self):
+        """Hash this tuple."""
+        return tuple.__hash__(self)


Seems to repeat lines 94–108. Could this be refactored out?

That is correct. I also discussed with @mraspaud that one could probably define the quadruple side band from the double side band, and then use inheritance to get rid of this redundancy. However, I am not sure we will ever find use with all the arithmetics on these bands. For instance I have chosen not to try to be too smart when calculating distances etc between them. Thus I would rather leave such refactoring till we really see a need for this.
I also don't have much time/resources for now to elaborate much further on this.

I think the refactoring can still be done using a mixin class though, even if the link between double- and quadrupleband isn't implemented.

Good point! I did this just now.

satpy/readers/pmw_channels_definitions.py

# Conflicts: # doc/source/index.rst

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

satpy/readers/mws_l1b.py

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

…alue Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

…ency types Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

mraspaud

Still some comments, sorry...

mraspaud · 2022-09-01T08:45:11Z

satpy/readers/mws_l1b.py

+MWS_CHANNEL_NAMES_TO_NUMBER = {'1': 1, '2': 2, '3': 3, '4': 4,
+                               '5': 5, '6': 6, '7': 7, '8': 8,
+                               '9': 9, '10': 10, '11': 11, '12': 12,
+                               '13': 13, '14': 14, '15': 15, '16': 16,
+                               '17': 17, '18': 18, '19': 19, '20': 20,
+                               '21': 21, '22': 22, '23': 23, '24': 24}


wouldn't be simpler to do int(channel_name) than using a dict like this?

I was not sure this would actually be the final channel/band naming in the file

satpy/readers/mws_l1b.py

mraspaud · 2022-09-01T08:48:54Z

satpy/readers/mws_l1b.py

+        if 'scale_factor' in variable.attrs and 'add_offset' in variable.attrs:
+            missing_value = variable.attrs['missing_value']
+            variable.data = da.where(variable.data == missing_value, np.nan,


Is this tested?

Probably not. I think this code should never be used as all datasets will have a scale and ofsset in the netCDF file

Forget what I said above! The code is now tested, thx!

mraspaud · 2022-09-01T08:53:40Z

satpy/readers/pmw_channels_definitions.py

+                left_side_dist = abs(value.central - value.side - value.sideside - left_left)
+                right_side_dist = abs(value.central + value.side + value.sideside - right_right)
+            except AttributeError:
+                if isinstance(value, (tuple, list)):
+                    msg = 'Distance to a quadruple side band frequency not supported for this type'
+                    raise NotImplementedError(msg)


Is this tested?

You got me! Right, it wasn't. It is now. But the NotImplementedError would never happen as it is implemented right now. So, I removed the check for tuple/list.

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Simplify - reuse already created list Co-authored-by: Martin Raspaud <martin.raspaud@smhi.se>

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

…tpy into add-epssg-mws-level1b-reader

mraspaud

LGTM. Regarding the title, is this still a first draft?

adybbroe · 2022-09-05T13:59:17Z

LGTM. Regarding the title, is this still a first draft?

I changed the title. It is adding a reader. Then as this has been developed on test data and we cannot start testing and use it in real applications I suppose there might be things we would like to change and improve. But we don't know yet.

Add first draft reader for MWS onboard EPS-SG-A

58ddaa0

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

mraspaud mentioned this pull request Jun 8, 2022

Add a reader for EPS-SG Ice Cloud Imager #2118

Merged

4 tasks

Adam.Dybbroe added 2 commits June 14, 2022 11:13

Refactor frequency handling

e7b4e63

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Refactor frequency handling

05afe37

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

adybbroe self-assigned this Jun 14, 2022

adybbroe added enhancement code enhancements, features, improvements component:readers labels Jun 14, 2022

Fix module import

da2d843

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

adybbroe marked this pull request as ready for review June 15, 2022 11:15

adybbroe requested review from djhoese and mraspaud as code owners June 15, 2022 11:15

adybbroe requested review from gerritholl and pnuu June 15, 2022 11:18

Adam.Dybbroe added 2 commits June 23, 2022 13:10

Fix resolution and add support for navigation and other aux data

c0d4b34

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Update list of supported readers

f1b4cf1

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

gerritholl reviewed Jul 29, 2022

View reviewed changes

BengtRydberg mentioned this pull request Aug 19, 2022

adding a reader for ATMS level1b data #2187

Merged

4 tasks

Adam.Dybbroe added 4 commits August 23, 2022 15:16

Merge branch 'main' into add-epssg-mws-level1b-reader

67d5bb8

# Conflicts: # doc/source/index.rst

Adding tests for the reader

af49fc8

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Remove line never used - module not meant to be run as script

27c5008

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Improve and bugfix documentation, and follow feedback from reviewer

4c0a81f

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

mraspaud reviewed Aug 24, 2022

View reviewed changes

satpy/readers/mws_l1b.py Outdated Show resolved Hide resolved

Adam.Dybbroe added 3 commits August 24, 2022 14:37

Refactor, add tests and change string representations of bands

123c8fb

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Improve test coverage and enhance attribute handling

6b447da

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Refactor

f3463a6

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Adam.Dybbroe added 10 commits August 26, 2022 16:13

Refactoring and adding test

3b7bd70

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Improve test coverage

8e12963

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Remove redundant code, all datasets in the file will have a missing_v…

f739ca6

…alue Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Improve documentation

10e5d60

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Increase test coverage

ea6155f

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Bugfix

6d94e72

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Bugfix - Throw exception if expected dataset is not present in file

c1f71ff

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Increase test coverage

953fb80

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Refactor and use a mixin class with common operations among all Frequ…

cba9a24

…ency types Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Move back the hash'es to the child classes

b54129f

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

mraspaud reviewed Sep 1, 2022

View reviewed changes

Adam.Dybbroe and others added 4 commits September 2, 2022 11:07

Remove redundant code and add test cases and coverage

afd2358

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Update satpy/readers/mws_l1b.py

6183e1a

Simplify - reuse already created list Co-authored-by: Martin Raspaud <martin.raspaud@smhi.se>

Add test coverage and test scaling the data upon reading

d03fc0a

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Merge branch 'add-epssg-mws-level1b-reader' of github.com:adybbroe/sa…

40e8461

…tpy into add-epssg-mws-level1b-reader

mraspaud approved these changes Sep 5, 2022

View reviewed changes

adybbroe changed the title ~~Add first draft reader for MWS onboard EPS-SG-A~~ Add reader for MWS onboard EPS-SG-A Sep 5, 2022

mraspaud merged commit a4417c6 into pytroll:main Sep 9, 2022

adybbroe mentioned this pull request Oct 5, 2022

Fix ici after mws merge #2221

Merged

4 tasks

ameraner mentioned this pull request Dec 5, 2022

Fix amsub_l1c_aapp.yaml frequency handling pointers #2309

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add reader for MWS onboard EPS-SG-A #2120

Add reader for MWS onboard EPS-SG-A #2120

adybbroe commented Jun 2, 2022 •

edited

Loading

adybbroe commented Jun 2, 2022

codecov bot commented Jun 2, 2022 •

edited

Loading

coveralls commented Jun 2, 2022 •

edited

Loading

adybbroe commented Jun 8, 2022

adybbroe commented Jun 15, 2022 •

edited

Loading

gerritholl left a comment

gerritholl Jul 29, 2022

adybbroe Aug 29, 2022

gerritholl Jul 29, 2022

adybbroe Aug 29, 2022

mraspaud Aug 29, 2022

adybbroe Aug 31, 2022

mraspaud left a comment

mraspaud Sep 1, 2022

adybbroe Sep 2, 2022

mraspaud Sep 1, 2022

adybbroe Sep 2, 2022

adybbroe Sep 5, 2022

mraspaud Sep 1, 2022

adybbroe Sep 2, 2022

mraspaud left a comment

adybbroe commented Sep 5, 2022

Add reader for MWS onboard EPS-SG-A #2120

Add reader for MWS onboard EPS-SG-A #2120

Conversation

adybbroe commented Jun 2, 2022 • edited Loading

adybbroe commented Jun 2, 2022

codecov bot commented Jun 2, 2022 • edited Loading

Codecov Report

coveralls commented Jun 2, 2022 • edited Loading

adybbroe commented Jun 8, 2022

adybbroe commented Jun 15, 2022 • edited Loading

gerritholl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

adybbroe commented Sep 5, 2022

adybbroe commented Jun 2, 2022 •

edited

Loading

codecov bot commented Jun 2, 2022 •

edited

Loading

coveralls commented Jun 2, 2022 •

edited

Loading

adybbroe commented Jun 15, 2022 •

edited

Loading