Add scanline acquisition times to hrit_jma #1185

sfinkens · 2020-05-06T10:28:27Z

Add scanline acquisition times to the hrit_jma reader.

Example:

In [1]: import satpy                                                                                                                                                                                                                         
In [2]: import glob                                                                                                                                                                                                                                                                                                                                                                                                                                                        
In [3]: scene = satpy.Scene(filenames=glob.glob('/data/georing/ahi_hrit/*2018011109*'), 
   ...:                     reader='ahi_hrit')                                                                                                                                                                                               
In [4]: scene.load(['B14'])                                                                                                                                                                                                                  
In [5]: scene['B14']['acq_time']                                                                                                                                                                                                             
Out[5]: 
<xarray.DataArray 'acq_time' (y: 5500)>
array(['2018-01-11T09:00:20.995200000', '2018-01-11T09:00:21.055948000',
       '2018-01-11T09:00:21.116695000', ...,
       '2018-01-11T09:09:40.226990000', '2018-01-11T09:09:40.287895000',
       '2018-01-11T09:09:40.348800000'], dtype='datetime64[ns]')
Coordinates:
    acq_time  (y) datetime64[ns] 2018-01-11T09:00:20.995200 ... 2018-01-11T09:09:40.348800
    crs       object +proj=geos +lon_0=140.7 +h=35785831 +x_0=0 +y_0=0 +a=6378169 +b=6356583.8 +units=m +no_defs +type=crs
  * y         (y) float64 5.5e+06 5.498e+06 5.496e+06 ... -5.496e+06 -5.498e+06
Attributes:
    long_name:  Scanline acquisition time

Tested with JAMI (non-segmented), MTSAT-2 Imager (both segmented and non-segmented) and AHI (segmented).

Tests added
Tests passed
Passes flake8 satpy
Fully documented

sfinkens · 2020-05-06T10:34:33Z

This also fixes the problem, that datasets loaded with the jma_hrit readers currently have ds.attrs['end_time'] = ds.attrs['start_time'] + 15 minutes, because this is hard coded in hrit_base: https://github.com/pytroll/satpy/blob/master/satpy/readers/hrit_base.py#L173

sfinkens · 2020-05-06T10:45:13Z

satpy/readers/hrit_jma.py

+        lines_sparse = [int(s.split(':=')[1]) for s in splits[0::2]]
+        times_sparse = [float(s.split(':=')[1]) for s in splits[1::2]]
+
+        if self.platform == HIMAWARI8:


In case you wonder why only Himawari-8 is checked: The projection name in the header defines the platform, but at the moment it is identical for Himawari-8 and 9 (GEOS(140.70)). So there is no way to distinguish them.

Great move, JMA...

Isn't that because the design of the two platforms is that they aren't meant to be used (in operations) at the same time?

That might be an explanation!

coveralls · 2020-05-06T11:33:03Z

Coverage increased (+0.02%) to 89.635% when pulling 89988ea on sfinkens:feature-hrit-jma-acq-time into ef24f29 on pytroll:master.

coveralls · 2020-05-06T11:33:03Z

Coverage increased (+0.01%) to 89.629% when pulling a7fffb8 on sfinkens:feature-hrit-jma-acq-time into ef24f29 on pytroll:master.

mraspaud

LGTM, just a couple documentation comments and a question

satpy/readers/hrit_jma.py

satpy/tests/reader_tests/test_ahi_hrit.py

mraspaud · 2020-05-06T11:51:53Z

satpy/readers/hrit_jma.py

+        lines_sparse = [int(s.split(':=')[1]) for s in splits[0::2]]
+        times_sparse = [float(s.split(':=')[1]) for s in splits[1::2]]
+
+        if self.platform == HIMAWARI8:


Great move, JMA...

int becomes np.int_ which is platform dependent. Use np.int64 instead.

codecov · 2020-05-11T10:43:32Z

Codecov Report

Merging #1185 into master will increase coverage by 0.02%.
The diff coverage is 96.15%.

@@            Coverage Diff             @@
##           master    #1185      +/-   ##
==========================================
+ Coverage   89.61%   89.63%   +0.02%     
==========================================
  Files         200      200              
  Lines       29504    29600      +96     
==========================================
+ Hits        26439    26532      +93     
- Misses       3065     3068       +3

Impacted Files	Coverage Δ
satpy/readers/hrit_jma.py	`97.94% <92.59%> (-1.22%)`	⬇️
satpy/readers/seviri_l1b_hrit.py	`92.18% <100.00%> (ø)`
satpy/tests/reader_tests/test_ahi_hrit.py	`100.00% <100.00%> (ø)`
satpy/readers/yaml_reader.py	`95.30% <0.00%> (-0.13%)`	⬇️
satpy/writers/geotiff.py	`90.74% <0.00%> (ø)`
satpy/readers/seviri_l1b_native.py	`71.96% <0.00%> (ø)`
satpy/tests/reader_tests/test_fci_l1c_fdhsi.py	`100.00% <0.00%> (ø)`
satpy/tests/reader_tests/test_seviri_l1b_native.py	`95.16% <0.00%> (ø)`
satpy/tests/test_scene.py	`99.72% <0.00%> (+<0.01%)`	⬆️
satpy/readers/__init__.py	`95.10% <0.00%> (+0.01%)`	⬆️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a82e4a6...89988ea. Read the comment docs.

sfinkens added 2 commits May 6, 2020 12:08

Add scanline acquisition time to hrit_jma

904b8cb

Update docstring

9cc0941

sfinkens requested review from djhoese and mraspaud as code owners May 6, 2020 10:28

sfinkens requested a review from pnuu May 6, 2020 10:28

sfinkens changed the title ~~Feature hrit jma acq time~~ Add scanline acquisition times to hrit_jma May 6, 2020

sfinkens self-assigned this May 6, 2020

sfinkens added component:readers enhancement code enhancements, features, improvements labels May 6, 2020

sfinkens commented May 6, 2020

View reviewed changes

sfinkens added 2 commits May 6, 2020 12:50

Fix tests

0cd0779

Make stickler happy

a7fffb8

mraspaud reviewed May 6, 2020

View reviewed changes

sfinkens added 3 commits May 6, 2020 13:59

Fix docstring

667ebc2

Explicitly add acq_time as a coordinate

d1420a8

Add documentation for JMA HRIT reader

5a6af93

sfinkens requested a review from sjoro as a code owner May 7, 2020 09:58

sfinkens added 2 commits May 7, 2020 12:00

Make stickler happy

4fe9499

Use numpy dtype when casting datetime64 to integer

89988ea

int becomes np.int_ which is platform dependent. Use np.int64 instead.

mraspaud merged commit 10d5e11 into pytroll:master May 13, 2020

sfinkens deleted the feature-hrit-jma-acq-time branch October 18, 2024 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add scanline acquisition times to hrit_jma #1185

Add scanline acquisition times to hrit_jma #1185

sfinkens commented May 6, 2020 •

edited by mraspaud

Loading

sfinkens commented May 6, 2020

sfinkens May 6, 2020

mraspaud May 6, 2020

djhoese May 6, 2020

sfinkens May 7, 2020

coveralls commented May 6, 2020 •

edited

Loading

coveralls commented May 6, 2020

mraspaud left a comment

mraspaud May 6, 2020

codecov bot commented May 11, 2020 •

edited

Loading

Add scanline acquisition times to hrit_jma #1185

Add scanline acquisition times to hrit_jma #1185

Conversation

sfinkens commented May 6, 2020 • edited by mraspaud Loading

sfinkens commented May 6, 2020

sfinkens May 6, 2020

Choose a reason for hiding this comment

mraspaud May 6, 2020

Choose a reason for hiding this comment

djhoese May 6, 2020

Choose a reason for hiding this comment

sfinkens May 7, 2020

Choose a reason for hiding this comment

coveralls commented May 6, 2020 • edited Loading

coveralls commented May 6, 2020

mraspaud left a comment

Choose a reason for hiding this comment

mraspaud May 6, 2020

Choose a reason for hiding this comment

codecov bot commented May 11, 2020 • edited Loading

Codecov Report

sfinkens commented May 6, 2020 •

edited by mraspaud

Loading

coveralls commented May 6, 2020 •

edited

Loading

codecov bot commented May 11, 2020 •

edited

Loading