Remove offset on GOES obs converter #732

braczka · 2024-09-09T18:45:48Z

Description:

I removed the satellite channel offset within GOES ABI observation converter such that the wavelength in the obs_seq.out match that
within the RTTOV coefficient file.

I also improved the documentation both within the GOES obs converter and the RTTOV module to alert the user to potential
mismatch issues.

Fixes issue

#731

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.

Tests

I have performed offline checks with the GOES obs converter, then used those obs_seq.out files to run filter
to confirm functionality.

Checklist for merging

Updated changelog entry
Documentation updated
Update conf.py

Checklist for release

Merge into main
Create release from the main branch with appropriate tag
Delete feature-branch

Testing Datasets

Dataset needed for testing available upon request
Dataset download instructions included
No dataset needed

hkershaw-brown · 2024-09-11T13:26:31Z

Hi Brett, can you provide 2 GOES ABI observation files for testing since we also have this issue outstanding: #579
Thanks!

braczka · 2024-09-11T14:52:07Z

My work directory for testing the GOES obs converter (convert_goes_ABI_L1b) is located here: /glade/work/bmraczka/DART/observations/obs_converters/GOES/work/

An example of a GOES radiance raw data file is located there : OR_ABI-L1b-RadF-M6C08_G16_s20211841720210_e20211841729518_c20211841729575.nc. I can download another if needed.

FYI -- my folder for testing the RTTOV channel offset bug is here: /glade/work/bmraczka/DART/models/wrf/Akeem_bug/

hkershaw-brown · 2024-09-11T15:07:04Z

two would be great since the other bug is "bug: GOES converter does not handle multiple input files"
Cheers,
Helen

braczka · 2024-09-11T16:38:13Z

two would be great since the other bug is "bug: GOES converter does not handle multiple input files" Cheers, Helen

Done

hkershaw-brown · 2024-09-11T17:14:02Z

If you build the converter with rttov13, then run, it errors out on the namelist.
I think this all the rrtov12 options in the input.nml &obs_def_rttov_nm

hkershaw@dec2158:/glade/derecho/scratch/hkershaw/DART/pull_reqeuests/pull_732/observations/obs_converters/GOES/work(GOES_obs_converter)$ ./convert_goes_ABI_L1b 
 
 --------------------------------------
 Starting ... at YYYY MM DD HH MM SS = 
                 2024  9 11  9 57 16
 Program convert_goes_ABI_L1b
 --------------------------------------
 
  set_nml_output Echo NML values to log file only
  goes_load_abi_map  Now loading L1C data from file OR_ABI-L1b-RadF-M6C08_G16_s2
 0211841720210_e20211841729518_c20211841729575.nc
 ERROR FROM:
  source : utilities_mod.f90
  routine: check_namelist_read
  message:  INVALID NAMELIST ENTRY:    cfrac_data             = .true. in nameli
 st obs_def_rttov_nml

braczka · 2024-09-11T17:53:23Z

Sorry -- the input.nml file got reverted to the default. The input.nml I used for the GOES conversion is here:
/glade/work/bmraczka/DART/observations/obs_converters/GOES/work/input.nml.GOES_converter

hkershaw-brown

see default input.nml comment

braczka · 2024-10-02T16:19:51Z

see default input.nml comment

Changes made to make input.nml compatible to run the converter. Should be ready for approval.

hkershaw-brown · 2024-10-02T16:33:32Z

see default input.nml comment

Changes made to make input.nml compatible to run the converter. Should be ready for approval.

ok I haven't actually looked at this pull request since trying to run it. I'll update to main and take a look.

hkershaw-brown

Hi Brett,

A few things
There's a broken link in to the GEOS converter, typo $ vs ~

convert_goes_ABI_L1b is missing from DART/.gitignore

Some of the documentation on rttov database vs. rttov coeffiencts is not that clear (e.g. the database is a DART file)
The vertical location for radiance obs I think would benefit from mentioning the namelist option to set a vertical location.

ps. I'm waiting for a Derecho job to actually run the code.

Cheers,
Helen

hkershaw-brown · 2024-10-02T16:39:13Z

observations/forward_operators/obs_def_rttov_mod.rst

+   HIMAWARI_9_AHI	31	9	56	ir	rtcoef_himawari_9_ahi.dat
+
+The coefficent file (.dat) is included with the RTTOV installation and can be found at the
+path  ``~{RTTOV_install}/rtcoef_rttov13/rttov9pred54L/rtcoef_himawari_9_ahi.dat``. This file


$path to the rttov install, rather than the home directory of ~rttov_install

Suggested change

path ``~{RTTOV_install}/rtcoef_rttov13/rttov9pred54L/rtcoef_himawari_9_ahi.dat``. This file

path ``${RTTOV_install}/rtcoef_rttov13/rttov9pred54L/rtcoef_himawari_9_ahi.dat``. This file

hkershaw-brown · 2024-10-02T16:43:05Z

observations/forward_operators/obs_def_rttov_mod.rst

+     288.370817896852
+     288.456378689407


Looks like two observations copies in this example. Do you want two copies? I think for an obs_seq.out it would normally be obs & qc:

288.456378689407
0.000000000000000E+000

hkershaw-brown · 2024-10-02T16:50:55Z

observations/forward_operators/obs_def_rttov_mod.rst

+provides a 4 integer description (31/9/56/8) of the platform/satellite/sensor/channel
+combination specific to this satellite observation.  For more information on this
+metadata refer to this GOES observation converter example here: 
+:doc:`../observations/obs_converters/GOES/README.rst`


This link is not correct and so does not get built:

/Users/hkershaw/DART/pull_requests/pull_732/observations/forward_operators/obs_def_rttov_mod.rst:105: WARNING: unknown document: '../observations/obs_converters/GOES/README.rst'

The relative path is incorrect and you have .rst on the file

Suggested change

:doc:`../observations/obs_converters/GOES/README.rst`

:doc:`../../observations/obs_converters/GOES/README`

or add an anchor to the GEOS/README.rst file and link to that anchor.

hkershaw-brown · 2024-10-02T16:58:01Z

observations/forward_operators/obs_def_rttov_mod.rst

+as undefined (VERTISUNDEF, integer = -2), however, this limits the ability to vertically
+localize the impact of the observation on the model state.
+
+The RTTOV specific metadata is located after the ``visir`` line.  This includes the


This example if for visir observations, the microwave observations have
'mw ' instead of visir'

hkershaw-brown · 2024-10-02T17:00:34Z

observations/forward_operators/obs_def_rttov_mod.rst

+  the file contains all wavelengths versus only IR wavelengths is **extremely important** because
+  it will shift the value of the channel number. Recommended practice is to choose a coefficient file
+  with all channels included.  If, on the other hand, you subset your coefficent file to only include
+  IR channels, you will also have to adjust your channel number in the obs_seq.out file.


For this, are you suggesting people edit the obs_seq.out file or run the converter with the correct channels? It is probably better to tell people to change the converter if they are using a subset of channels.

hkershaw-brown · 2024-10-02T17:14:15Z

observations/obs_converters/GOES/README.rst

+the vertical location undefined (i.e. VERTISUNDEF) within the ``obs_seq.out``
+file. This approach, however, limits the application of vertical localization
+during the assimilation step.
+
+Alternatively, for some applications it may be appropriate to assign 
+a vertical location to the radiance observation. This is an ongoing area
+of observation-space localization research, and is the standard
+workaround pioneered by Lili Lei and Jeff Whittaker.


Is this section "Specifying a vertical location" talking about the namelist option vloc_pres_hPa?

I think it would be better to explain the vertical choice in terms of the namelist option for the GEOS converter.
The default is to have VERTISUNDEFINED. If you want to set a vertical location (science blah blah), set vloc_pres_hPa to the vertical location of the observation in hPa.

hkershaw-brown · 2024-10-02T17:25:03Z

observations/obs_converters/GOES/goes_ABI_L1b_mod.f90

-      goes_channel = map%channel-6
+      ! Defines the GOES ABI wavelength channels (band_id)
+      ! Channels range from 1-16 covering UV/VIS/NIR/IR spectrum
+      goes_channel = map%channel


note in release notes, this will affect people using subsetted rttov files.

hkershaw-brown · 2024-10-02T17:30:44Z

observations/forward_operators/obs_def_rttov_mod.rst

+.. Important ::
+
+  **It is imperative that the user confirms the satellite integer metadata matches the
+  appropriate RTTOV coefficient (parameter) file.  See next section for more information.**


It is a little hard to tell which file is which from the description.

hkershaw-brown · 2024-10-02T17:45:35Z

observations/forward_operators/obs_def_rttov_mod.rst

+radiance observation. The RTTOV database file (``rttov_sensor_db.csv``) refers to the coefficent
+file.  For the ``HIMAWARI_9_AHI_RADIANCE`` observation type, for example, the following information


The rttov_sensor_db.csv a DART file, correct? I would be good to make that clear. At the moment it reads like the RTTOV database file is something you should be getting from RTTOV.

The namelist option:

rttov_sensor_db_file character(len=512) The location of the RTTOV sensor database. The format for the database is a comma-separated file. The columns of the database are the DART observation-kind, the platform/satellite/sensor ID, the observation type, the coefficient file, and a comma-separated list of RTTOV channels to use for this observation type.

looking at the code the channel list is optional:

DART/observations/forward_operators/obs_def_rttov13_mod.f90

Line 1059 in 464aa57

! channel_list is an optional list of channels (can be zero-length, meaning all available channels should be used)

there are no channels in the rttov_sensor_db.csv sensor file in the repo:

NOAA_1_VTPR1,1,1,7,ir,rtcoef_noaa_1_vtpr1.dat NOAA_2_VTPR1,1,2,7,ir,rtcoef_noaa_2_vtpr1.dat NOAA_3_VTPR1,1,3,7,ir,rtcoef_noaa_3_vtpr1.dat ...

are users supposed to edit rttov_sensor_db.csv or leave it as is?

The rttov_sensor_db.csv a DART file, correct? I would be good to make that clear. At the moment it reads like the RTTOV database file is something you should be getting from RTTOV.

The namelist option:

rttov_sensor_db_file character(len=512) The location of the RTTOV sensor database. The format for the database is a comma-separated file. The columns of the database are the DART observation-kind, the platform/satellite/sensor ID, the observation type, the coefficient file, and a comma-separated list of RTTOV channels to use for this observation type.
looking at the code the channel list is optional:

DART/observations/forward_operators/obs_def_rttov13_mod.f90

Line 1059 in 464aa57

! channel_list is an optional list of channels (can be zero-length, meaning all available channels should be used)

there are no channels in the rttov_sensor_db.csv sensor file in the repo:

NOAA_1_VTPR1,1,1,7,ir,rtcoef_noaa_1_vtpr1.dat NOAA_2_VTPR1,1,2,7,ir,rtcoef_noaa_2_vtpr1.dat NOAA_3_VTPR1,1,3,7,ir,rtcoef_noaa_3_vtpr1.dat ...

are users supposed to edit rttov_sensor_db.csv or leave it as is?

The recommendation is to leave as is, and allow for all channels to be available, and the correct channel parameters are specified through the obs_seq.out channel metadata. Not clear to me if there would be any application where excluding certain channels through the DART database file would be appropriate.

@hkershaw-brown, I think I addressed all your review suggestions. Check through these docs changes, and make sure the converter code doesn't give you any issues.

…obs_converter

hkershaw-brown

looks great, approved.

todo: note in release notes, this will affect people using subsetted rttov files.

braczka added 3 commits September 9, 2024 11:07

Remove GOES converter channel offset

d5bd801

Enhance GOES converter docs

89932c4

Enhance obs_def_RTTOV docs for coefficent files

f2f2e8d

braczka added Documentation Improvements or additions to documentation wrf Weather Research & Forecasting Model rttov13 Radiative Transfer for TOVS labels Sep 9, 2024

braczka added 2 commits September 9, 2024 14:07

syntax corrections

7b00844

Remove unnecessary line of docs

e470a4a

braczka requested a review from hkershaw-brown September 9, 2024 20:20

braczka marked this pull request as ready for review September 9, 2024 20:20

hkershaw-brown requested changes Sep 13, 2024

View reviewed changes

Make input.nml compatible with RTTOV13

4f969a4

Merge branch 'main' into GOES_obs_converter

899e7b4

hkershaw-brown requested changes Oct 2, 2024

View reviewed changes

braczka added 5 commits October 2, 2024 17:45

Review edits for obs_def_rttov docs

02c92ad

Review edits for channel metadata

775e1bd

Merge remote-tracking branch 'upstream/GOES_obs_converter' into GOES_…

4d94575

…obs_converter

More docs on radiance channel settings

ddfec02

Ignore GOES obs converter exec

c9a330b

hkershaw-brown approved these changes Oct 21, 2024

View reviewed changes

hkershaw-brown added the release! bundle with next release label Oct 21, 2024

bump version and changelog for release

effe720

Merge branch 'main' into GOES_obs_converter

a58694a

hkershaw-brown merged commit e218864 into main Oct 22, 2024
4 checks passed

hkershaw-brown deleted the GOES_obs_converter branch October 22, 2024 18:40

hkershaw-brown mentioned this pull request Oct 23, 2024

bug: GOES ABI obs converter gives incorrect wavelength channel #731

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove offset on GOES obs converter #732

Remove offset on GOES obs converter #732

braczka commented Sep 9, 2024

hkershaw-brown commented Sep 11, 2024

braczka commented Sep 11, 2024

hkershaw-brown commented Sep 11, 2024

braczka commented Sep 11, 2024

hkershaw-brown commented Sep 11, 2024

braczka commented Sep 11, 2024

hkershaw-brown left a comment

braczka commented Oct 2, 2024

hkershaw-brown commented Oct 2, 2024

hkershaw-brown left a comment

hkershaw-brown Oct 2, 2024

hkershaw-brown Oct 2, 2024

hkershaw-brown Oct 2, 2024

hkershaw-brown Oct 2, 2024

hkershaw-brown Oct 2, 2024

hkershaw-brown Oct 2, 2024

hkershaw-brown Oct 2, 2024

hkershaw-brown Oct 2, 2024

hkershaw-brown Oct 2, 2024

braczka Oct 3, 2024

braczka Oct 3, 2024

hkershaw-brown left a comment

	path ``~{RTTOV_install}/rtcoef_rttov13/rttov9pred54L/rtcoef_himawari_9_ahi.dat``. This file
	path ``${RTTOV_install}/rtcoef_rttov13/rttov9pred54L/rtcoef_himawari_9_ahi.dat``. This file

	:doc:`../observations/obs_converters/GOES/README.rst`
	:doc:`../../observations/obs_converters/GOES/README`

		radiance observation. The RTTOV database file (``rttov_sensor_db.csv``) refers to the coefficent
		file. For the ``HIMAWARI_9_AHI_RADIANCE`` observation type, for example, the following information

Remove offset on GOES obs converter #732

Remove offset on GOES obs converter #732

Conversation

braczka commented Sep 9, 2024

Description:

Fixes issue

Types of changes

Documentation changes needed?

Tests

Checklist for merging

Checklist for release

Testing Datasets

hkershaw-brown commented Sep 11, 2024

braczka commented Sep 11, 2024

hkershaw-brown commented Sep 11, 2024

braczka commented Sep 11, 2024

hkershaw-brown commented Sep 11, 2024

braczka commented Sep 11, 2024

hkershaw-brown left a comment

Choose a reason for hiding this comment

braczka commented Oct 2, 2024

hkershaw-brown commented Oct 2, 2024

hkershaw-brown left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hkershaw-brown left a comment

Choose a reason for hiding this comment