Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove offset on GOES obs converter #732

Merged
merged 14 commits into from
Oct 22, 2024
Merged

Remove offset on GOES obs converter #732

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
d5bd801
Remove GOES converter channel offset
braczka Sep 9, 2024
89932c4
Enhance GOES converter docs
braczka Sep 9, 2024
f2f2e8d
Enhance obs_def_RTTOV docs for coefficent files
braczka Sep 9, 2024
7b00844
syntax corrections
braczka Sep 9, 2024
e470a4a
Remove unnecessary line of docs
braczka Sep 9, 2024
4f969a4
Make input.nml compatible with RTTOV13
braczka Oct 2, 2024
899e7b4
Merge branch 'main' into GOES_obs_converter
hkershaw-brown Oct 2, 2024
02c92ad
Review edits for obs_def_rttov docs
braczka Oct 2, 2024
775e1bd
Review edits for channel metadata
braczka Oct 3, 2024
4d94575
Merge remote-tracking branch 'upstream/GOES_obs_converter' into GOES_…
braczka Oct 3, 2024
ddfec02
More docs on radiance channel settings
braczka Oct 3, 2024
c9a330b
Ignore GOES obs converter exec
braczka Oct 3, 2024
effe720
bump version and changelog for release
hkershaw-brown Oct 22, 2024
a58694a
Merge branch 'main' into GOES_obs_converter
hkershaw-brown Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 100 additions & 5 deletions observations/forward_operators/obs_def_rttov_mod.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,19 +46,114 @@ Although a model may not have the necessary inputs by itself,
the defaults in RTTOV based on climatology can be used.
The impact on the quality of the results should be investigated.

The quanities for each observation type are defined in obs_def_rttov{13}_mod.f90, like so:
The quanities for each observation type are defined in obs_def_rttov{13}_mod.f90, for example:

.. code::

! HIMAWARI_8_AHI_RADIANCE, QTY_RADIANCE
! HIMAWARI_9_AHI_RADIANCE, QTY_RADIANCE

If you want to change the quantity associated with an observation, for example, if you want
to assimilate HIMAWARI_8_AHI_RADIANCE as QTY_BRIGHTNESS_TEMPERATURE, edit the QTY
in obs_def_rttov{13}_mod.f90 and rerun quickbuild.sh.
to assimilate HIMAWARI_9_AHI_RADIANCE as QTY_BRIGHTNESS_TEMPERATURE, edit the QTY
in obs_def_rttov{13}_mod.f90 and rerun quickbuild.sh. Although both spectral radiance
(mW/cm/m^2/sr) and brightness temperature (Kelvin) quantify the same emitted/reflected
radiance from the atmosphere, the tendency for brightness temperatures to adhere closer
to a gaussian distribution may improve the quality of the assimilation if using
a DART filter type that depends on Gaussian assumptions (e.g. EAKF). This is
an ongoing area of research.


Known issues:

RTTOV Metadata
---------------

The RTTOV module ingests metadata from the ``obs_seq.out`` file in order to calculate the
expected observed radiance. For example, a single ``HIMAWARI_9_AHI_RADIANCE``
synthetic observation in units of brightness temperature (Kelvin) looks like the following:

.. code::


OBS 1
288.370817896852
288.456378689407
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like two observations copies in this example. Do you want two copies? I think for an obs_seq.out it would normally be obs & qc:

288.456378689407
0.000000000000000E+000

0.000000000000000E+000
-1 2 -1
obdef
loc3d
1.766273140907288 0.1535889655351639 34000.00000000000 2
kind
304
visir
100.500000000000 46.6700000000000 -888888.000000000
-888888.000000000
31 9 56 8
-888888.000000000
1
0 154166
1.00000000000000


Please note, that in this example the radiance observation was assigned a vertical level (34000 Pa)
with the ``VERTISPRESSURE`` (integer = 2) vertical coordinate.
Although radiance/BT observations are technically representative of the entire atmospheric
column and not a single vertical level, in some applications
this approximation improves the skill of the assimilation forecast. This is an ongoing
area of research. As an alternative, it is also common to leave the vertical level
as undefined (VERTISUNDEF, integer = -2), however, this limits the ability to vertically
localize the impact of the observation on the model state.

The RTTOV specific metadata is located after the ``visir`` line. This includes the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example if for visir observations, the microwave observations have
'mw ' instead of visir'

azimuth and elevation angle of the satellite and the sun respectively. Note, in this
instance the sun azimuth/elevation are given missing values (-888888) because solar
reflectance has no impact on an IR radiance observation. Also note, the observation
provides a 4 integer description (31/9/56/8) of the platform/satellite/sensor/channel
combination specific to this satellite observation. For more information on this
metadata refer to this GOES observation converter example here:
:doc:`../observations/obs_converters/GOES/README.rst`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This link is not correct and so does not get built:

/Users/hkershaw/DART/pull_requests/pull_732/observations/forward_operators/obs_def_rttov_mod.rst:105: WARNING: unknown document: '../observations/obs_converters/GOES/README.rst'

The relative path is incorrect and you have .rst on the file

Suggested change
:doc:`../observations/obs_converters/GOES/README.rst`
:doc:`../../observations/obs_converters/GOES/README`

or add an anchor to the GEOS/README.rst file and link to that anchor.


.. Important ::

**It is imperative that the user confirms the satellite integer metadata matches the
appropriate RTTOV coefficient (parameter) file. See next section for more information.**
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a little hard to tell which file is which from the description.


RTTOV coefficient files
-----------------------

The RTTOV coefficent file contains the appropriate parameter values for a specific satellite
radiance observation. The RTTOV database file (``rttov_sensor_db.csv``) refers to the coefficent
file. For the ``HIMAWARI_9_AHI_RADIANCE`` observation type, for example, the following information
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rttov_sensor_db.csv a DART file, correct? I would be good to make that clear. At the moment it reads like the RTTOV database file is something you should be getting from RTTOV.

The namelist option:

rttov_sensor_db_file character(len=512) The location of the RTTOV sensor database. The format for the database is a comma-separated file. The columns of the database are the DART observation-kind, the platform/satellite/sensor ID, the observation type, the coefficient file, and a comma-separated list of RTTOV channels to use for this observation type.

looking at the code the channel list is optional:

DART/observations/forward_operators/obs_def_rttov13_mod.f90

Line 1059 in 464aa57

! channel_list is an optional list of channels (can be zero-length, meaning all available channels should be used)

there are no channels in the rttov_sensor_db.csv sensor file in the repo:

NOAA_1_VTPR1,1,1,7,ir,rtcoef_noaa_1_vtpr1.dat
NOAA_2_VTPR1,1,2,7,ir,rtcoef_noaa_2_vtpr1.dat
NOAA_3_VTPR1,1,3,7,ir,rtcoef_noaa_3_vtpr1.dat
...

are users supposed to edit rttov_sensor_db.csv or leave it as is?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rttov_sensor_db.csv a DART file, correct? I would be good to make that clear. At the moment it reads like the RTTOV database file is something you should be getting from RTTOV.

The namelist option:

rttov_sensor_db_file character(len=512) The location of the RTTOV sensor database. The format for the database is a comma-separated file. The columns of the database are the DART observation-kind, the platform/satellite/sensor ID, the observation type, the coefficient file, and a comma-separated list of RTTOV channels to use for this observation type.
looking at the code the channel list is optional:

DART/observations/forward_operators/obs_def_rttov13_mod.f90

Line 1059 in 464aa57

! channel_list is an optional list of channels (can be zero-length, meaning all available channels should be used)

there are no channels in the rttov_sensor_db.csv sensor file in the repo:

NOAA_1_VTPR1,1,1,7,ir,rtcoef_noaa_1_vtpr1.dat
NOAA_2_VTPR1,1,2,7,ir,rtcoef_noaa_2_vtpr1.dat
NOAA_3_VTPR1,1,3,7,ir,rtcoef_noaa_3_vtpr1.dat
...

are users supposed to edit rttov_sensor_db.csv or leave it as is?

The recommendation is to leave as is, and allow for all channels to be available, and the correct channel parameters are specified through the obs_seq.out channel metadata. Not clear to me if there would be any application where excluding certain channels through the DART database file would be appropriate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hkershaw-brown, I think I addressed all your review suggestions. Check through these docs changes, and make sure the converter code doesn't give you any issues.

is provided:

.. code::

HIMAWARI_9_AHI 31 9 56 ir rtcoef_himawari_9_ahi.dat

The coefficent file (.dat) is included with the RTTOV installation and can be found at the
path ``~{RTTOV_install}/rtcoef_rttov13/rttov9pred54L/rtcoef_himawari_9_ahi.dat``. This file
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$path to the rttov install, rather than the home directory of ~rttov_install

Suggested change
path ``~{RTTOV_install}/rtcoef_rttov13/rttov9pred54L/rtcoef_himawari_9_ahi.dat``. This file
path ``${RTTOV_install}/rtcoef_rttov13/rttov9pred54L/rtcoef_himawari_9_ahi.dat``. This file

should be included in your run folder at runtime. Additional coefficent files for a given
satellite sensor may be required.

It is good practice to always view your coefficent file (.dat) to confirm that the
channels listed in the file match the channel from the ``obs_seq.out`` file. The coefficent
file will include a list of channels (wavebands) with the associated wavelength (microns).


.. Important ::

The RTTOV package includes multiple coefficent files (e.g. all wavelengths, IR only, etc.) that
contain the appropriate parameter data for each satellite/sensor/channel combination. Whether
the file contains all wavelengths versus only IR wavelengths is **extremely important** because
it will shift the value of the channel number. Recommended practice is to choose a coefficient file
with all channels included. If, on the other hand, you subset your coefficent file to only include
IR channels, you will also have to adjust your channel number in the obs_seq.out file.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this, are you suggesting people edit the obs_seq.out file or run the converter with the correct channels? It is probably better to tell people to change the converter if they are using a subset of channels.

If RTTOV always returns expected observations of radiance = 0, or if the prior expected radiance
is unusually biased from your prior, this could be a sign there is a mismatch between the
obs_seq.out channel and the coefficient file channel.



Known issues:
-------------
- DART does not yet provide any type of bias correction
- Cross-channel error correlations are not yet supported. A principal component approach has been discussed. For now,
the best bet is to use a subset of channels that are nearly independent of one another.
Expand Down
193 changes: 126 additions & 67 deletions observations/obs_converters/GOES/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,70 +35,129 @@ observation type).
Specifying a vertical location
------------------------------

Jeff Steward added (PR 48) the capability to specify a vertical location
if desired. This allows for localization in the vertical.

It’s sometimes helpful, even though definitely wrong from a theoretical
standpoint, to give a vertical location to satellite observations
(which are integrated quantities). This has been an issue with
observation-space localization for some time, and this is the standard
workaround pioneered by Lili Lei and Jeff Whittaker.

A short description of the namelist options
-------------------------------------------

This table is meant to familiarize you with some of the options
available. Until we fully implement automatic documentation generation,
you would be well advised to familiarize yourself with the code. This is
not the full list of namelist variables …

+-------------------------+---------------------+---------------------+
| variable | default | meaning |
+=========================+=====================+=====================+
| x_thin | 0 | Skip this many per |
| | | X scan. |
+-------------------------+---------------------+---------------------+
| y_thin | 0 | Skip this many per |
| | | Y scan. |
+-------------------------+---------------------+---------------------+
| goes_num | 16 | GOES Satellite |
| | | number. |
+-------------------------+---------------------+---------------------+
| reject_dqf_1 | .true. | Bad scan rejection |
| | | criteria. If .true. |
| | | and DQF /= 0, the |
| | | scan is rejected. |
| | | If .false. any DQF |
| | | > 1 rejects the |
| | | scan. |
+-------------------------+---------------------+---------------------+
| verbose | .false. | Run-time output |
| | | verbosity |
+-------------------------+---------------------+---------------------+
| obs_err | MISSING_R8 | The observation |
| | | error standard |
| | | deviation (std dev, |
| | | in radiance units) |
| | | TODO: make this |
| | | more sophisticated. |
| | | You must supply a |
| | | value other than |
| | | MISSING_R8. Be |
| | | aware that the |
| | | observation |
| | | sequence files |
| | | convert this to a |
| | | variance. |
+-------------------------+---------------------+---------------------+
| vloc_pres_hPa | -1.0 | The vertical |
| | | location of this |
| | | observation (hPa). |
| | | A negative means |
| | | there is no |
| | | vertical location |
| | | (which is typical |
| | | for a |
| | | ve |
| | | rtically-integrated |
| | | quantity). |
+-------------------------+---------------------+---------------------+
Top of the atmosphere radiance observations are sensitive to the
atmospheric constituents (e.g. water vapor) that reside in the vertical
profile of the atmosphere. Given this is an integrated quantity and does
not depend on a single vertical location, it may be appropriate to leave
the vertical location undefined (i.e. VERTISUNDEF) within the ``obs_seq.out``
file. This approach, however, limits the application of vertical localization
during the assimilation step.

Alternatively, for some applications it may be appropriate to assign
a vertical location to the radiance observation. This is an ongoing area
of observation-space localization research, and is the standard
workaround pioneered by Lili Lei and Jeff Whittaker.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this section "Specifying a vertical location" talking about the namelist option vloc_pres_hPa?

I think it would be better to explain the vertical choice in terms of the namelist option for the GEOS converter.
The default is to have VERTISUNDEFINED. If you want to set a vertical location (science blah blah), set vloc_pres_hPa to the vertical location of the observation in hPa.


Radiance versus Brightness Temperature
--------------------------------------

This converter assigns the observation type as ``GOES[16-19]_ABI_RADIANCE``.
The default setup in DART is that this radiance observation type is assigned
the quantity ``QTY_RADIANCE``. Radiance observations are commonly expressed
in spectrally resolved units (mW/cm/m^2/sr).

Alternatively, radiances can also be expressed as brightness temperatures
(units: Kelvin) and the DART code also supports observation quantities of
``QTY_BRIGHTNESS_TEMPERATURE``. Both the spectral and temperature units
quantify the same physical properties of the atmosphere
(emitted and reflected radiation), however, in some applications it may
be advantageous to use brightness temperatures given their Gaussian
distribution. This is an ongoing area of research.


An overview of the namelist options
-----------------------------------

A description of the most important namelist options. Note that supplying
an observation error value is mandatory. This is not the full list of namelist
variables.

+-------------------------+------------+-----------------------------+
| Variable | Default | Description |
+=========================+============+=============================+
| x_thin | 0 | Skip this many observations |
| | | per X scan |
+-------------------------+------------+-----------------------------+
| y_thin | 0 | Skip this many observations |
| | | per Y scan |
+-------------------------+------------+-----------------------------+
| goes_num | 16 | GOES Satellite number |
+-------------------------+------------+-----------------------------+
| reject_dqf_1 | .true. | Bad scan rejection critera. |
| | | If .true. and DQF /= 0, the |
| | | scan is rejected. If |
| | | .false. any DQF > 1 |
| | | rejects the scan. |
+-------------------------+------------+-----------------------------+
| verbose | .false. | Run-time output verbosity |
+-------------------------+------------+-----------------------------+
| obs_err | MISSING_R8 | The observation error |
| | | standard deviation in units |
| | | of radiance. IMPORTANT: |
| | | the user must supply a |
| | | value other than MISSING_R8.|
| | | Be aware that the |
| | | observation sequence files |
| | | convert this to a variance. |
+-------------------------+------------+-----------------------------+
| vloc_pres_hPa | -1.0 | If a positive value, the |
| | | vertical location of the |
| | | observation (hPa) is |
| | | assigned with a vertical |
| | | coordinate of |
| | | VERTISPRESSURE. If negative |
| | | value there is no vertical |
| | | location and the coordinate |
| | | is VERTISUNDEFINED. |
+-------------------------+------------+-----------------------------+

Radiance metadata supplied to obs_seq.out file
----------------------------------------------

This converter is designed to supply metadata to the radiative transfer
model (RTTOV) :doc:`../../../observations/forward_operators/obs_def_rttov_mod`
that supports the calculation of the expected radiance
observation during the assimilation step. Below is a description
of this metadata information.

The integer values that describe the platform/satellite/sensor/channel
combination for the GOES satellite are required by the RTTOV radiance
model to assign the appropriate coefficent file during the radiance
calculation. For more details refer to the
`RTTOV user guide. <https://www.nwpsaf.eu/site/software/rttov/documentation/>`__


+-------------------------+------------+-----------------------------+
| Variable | Value | Description |
+=========================+============+=============================+
| sat az/el | Supplied | GOES satellite azimuth and |
| | by data | elevation angle |
| | file | |
+-------------------------+------------+-----------------------------+
| sun az/el | -888888 | Sun azimuth and elevation |
| | | angle. The default is for |
| | | missing values. For IR |
| | | channels reflected solar |
| | | has no impact. For NIR/ |
| | | VIS/UV providing sun az/el |
| | | may be desirable but not |
| | | required by RTTOV. |
+-------------------------+------------+-----------------------------+
| platform | 4 | Platform number supplied |
| | | by converter code. |
+-------------------------+------------+-----------------------------+
| sat_id | 16 | Satellite ID number |
| | | supplied by converter code. |
+-------------------------+------------+-----------------------------+
| instrument | 44 | Sensor number supplied by |
| | | converter code |
+-------------------------+------------+-----------------------------+
| channel | Supplied | The wavelength channel |
| | by data | (1-16) assigned from band_id|
| | file | variable in data file. |
+-------------------------+------------+-----------------------------+





5 changes: 3 additions & 2 deletions observations/obs_converters/GOES/goes_ABI_L1b_mod.f90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,8 +462,9 @@ subroutine make_obs_sequence (seq, map, lon1, lon2, lat1, lat2, &
specularity = MISSING_R8
end if

! the RTTOV ABI coefficients start from channel 7
goes_channel = map%channel-6
! Defines the GOES ABI wavelength channels (band_id)
! Channels range from 1-16 covering UV/VIS/NIR/IR spectrum
goes_channel = map%channel
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note in release notes, this will affect people using subsetted rttov files.


! add additional metadata for this obs type. returns key to use in create call
call set_visir_metadata(key, sat_az, sat_ze, sun_az, sun_ze, platform_id, sat_id, sensor_id, &
Expand Down
Loading