Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update IASI YAML files for end-to-end testing #769

Merged
merged 6 commits into from
Dec 5, 2023
Merged

Update IASI YAML files for end-to-end testing #769

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
39882e6
Add/update IASI YAML files for testing and config; Add bufr to ioda p…
emilyhcliu Nov 26, 2023
4546af4
Uncomment UseFlag Check for config; Update passed benchmark for testing.
emilyhcliu Nov 27, 2023
adfa72a
Merge branch 'feature/gdas-validation' into feature/gdas-validation-i…
emilyhcliu Nov 28, 2023
943a904
Add linear obs operator
emilyhcliu Nov 30, 2023
aeacece
Comment out the Diagnostic Flags to save memory use.
emilyhcliu Dec 1, 2023
d5c6553
Turn off DiagnosticFlags to save memory
emilyhcliu Dec 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
756 changes: 756 additions & 0 deletions parm/atm/obs/config/iasi_metop-a.yaml
View file
Open in desktop

Large diffs are not rendered by default.

756 changes: 756 additions & 0 deletions parm/atm/obs/config/iasi_metop-b.yaml
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion parm/atm/obs/lists/gdas_prototype_3d.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ observers:
##- !INC ${OBS_YAML_DIR}/ompstc8_npp.yaml
##- !INC ${OBS_YAML_DIR}/cris-fsr_n20.yaml
##- !INC ${OBS_YAML_DIR}/cris-fsr_npp.yaml
##- !INC ${OBS_YAML_DIR}/iasi_metop-a.yaml
##- !INC ${OBS_YAML_DIR}/iasi_metop-b.yaml
##- !INC ${OBS_YAML_DIR}/sfc.yaml
##- !INC ${OBS_YAML_DIR}/sfcship.yaml
##- !INC ${OBS_YAML_DIR}/gnssro.yaml
##- !INC ${OBS_YAML_DIR}/gnssro.yaml
22 changes: 15 additions & 7 deletions parm/atm/obs/testing/iasi_metop-a.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ obs prior filters:

obs post filters:
# Wavenumber Check
- filter: BlackList
- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: 7024, 7027, 7029, 7032, 7038, 7043, 7046, 7049, 7069, 7072,
Expand All @@ -176,7 +176,8 @@ obs post filters:
minvalue: 1.0e-12
action:
name: reject
- filter: BlackList

- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: *iasi_metop-a_channels
Expand All @@ -187,6 +188,7 @@ obs post filters:
channels: *iasi_metop-a_channels
options:
channels: *iasi_metop-a_channels

# Observation Range Sanity Check
- filter: Bounds Check
filter variables:
Expand All @@ -197,8 +199,9 @@ obs post filters:
# maxvalue: 100.00000
action:
name: reject

# Topography Check
- filter: BlackList
- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: *iasi_metop-a_channels
Expand All @@ -210,8 +213,9 @@ obs post filters:
options:
channels: *iasi_metop-a_channels
sensor: iasi_metop-a

# Transmittance Top Check
- filter: BlackList
- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: *iasi_metop-a_channels
Expand All @@ -222,6 +226,7 @@ obs post filters:
channels: *iasi_metop-a_channels
options:
channels: *iasi_metop-a_channels

# Cloud Detection Check
- filter: Bounds Check
filter variables:
Expand Down Expand Up @@ -361,6 +366,7 @@ obs post filters:
maxvalue: 1.0e-12
action:
name: reject

# NSST Retrieval Check
- filter: Bounds Check
filter variables:
Expand Down Expand Up @@ -438,8 +444,9 @@ obs post filters:
maxvalue: 1.0e-12
action:
name: reject

# Surface Jacobians Check
- filter: BlackList
- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: *iasi_metop-a_channels
Expand All @@ -453,6 +460,7 @@ obs post filters:
obserr_demisf: [0.01, 0.02, 0.03, 0.02, 0.03]
obserr_dtempf: [0.50, 2.00, 4.00, 2.00, 4.00]
sensor: iasi_metop-a

# Gross check
- filter: Background Check
filter variables:
Expand Down Expand Up @@ -537,6 +545,7 @@ obs post filters:
error parameter vector: *iasi_metop-a_oberr
action:
name: reject

# Useflag Check
- filter: Bounds Check
filter variables:
Expand Down Expand Up @@ -612,5 +621,4 @@ obs post filters:
minvalue: 1.0e-12
action:
name: reject
passedBenchmark: 127348

passedBenchmark: 144704 #127348
158 changes: 158 additions & 0 deletions parm/atm/obs/testing/iasi_metop-a_noqc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
obs space:
name: iasi_metop-a
obsdatain:
engine:
type: H5File
obsfile: iasi_metop-a_obs_${CDATE}_subset.nc4
_source: gdas
obsdataout:
engine:
type: H5File
obsfile: iasi_metop-a_diag_${CDATE}.nc4
simulated variables: [brightnessTemperature]
channels: &iasi_metop-a_channels 16, 29, 32, 35, 38, 41, 44, 47, 49, 50, 51, 53,
55, 56, 57, 59, 61, 62, 63, 66, 68, 70, 72, 74, 76, 78, 79, 81, 82, 83,
84, 85, 86, 87, 89, 92, 93, 95, 97, 99, 101, 103, 104, 106, 109, 110,
111, 113, 116, 119, 122, 125, 128, 131, 133, 135, 138, 141, 144, 146,
148, 150, 151, 154, 157, 159, 160, 161, 163, 167, 170, 173, 176, 179,
180, 185, 187, 191, 193, 197, 199, 200, 202, 203, 205, 207, 210, 212,
213, 214, 217, 218, 219, 222, 224, 225, 226, 228, 230, 231, 232, 236,
237, 239, 243, 246, 249, 252, 254, 259, 260, 262, 265, 267, 269, 275,
279, 282, 285, 294, 296, 299, 300, 303, 306, 309, 313, 320, 323, 326,
327, 329, 332, 335, 345, 347, 350, 354, 356, 360, 363, 366, 371, 372,
373, 375, 377, 379, 381, 383, 386, 389, 398, 401, 404, 405, 407, 408,
410, 411, 414, 416, 418, 423, 426, 428, 432, 433, 434, 439, 442, 445,
450, 457, 459, 472, 477, 483, 509, 515, 546, 552, 559, 566, 571, 573,
578, 584, 594, 625, 646, 662, 668, 705, 739, 756, 797, 867, 906, 921,
1027, 1046, 1090, 1098, 1121, 1133, 1173, 1191, 1194, 1222, 1271, 1283,
1338, 1409, 1414, 1420, 1424, 1427, 1430, 1434, 1440, 1442, 1445, 1450,
1454, 1460, 1463, 1469, 1474, 1479, 1483, 1487, 1494, 1496, 1502, 1505,
1509, 1510, 1513, 1518, 1521, 1526, 1529, 1532, 1536, 1537, 1541, 1545,
1548, 1553, 1560, 1568, 1574, 1579, 1583, 1585, 1587, 1606, 1626, 1639,
1643, 1652, 1658, 1659, 1666, 1671, 1675, 1681, 1694, 1697, 1710, 1786,
1791, 1805, 1839, 1884, 1913, 1946, 1947, 1991, 2019, 2094, 2119, 2213,
2239, 2271, 2289, 2321, 2333, 2346, 2349, 2352, 2359, 2367, 2374, 2398,
2426, 2562, 2701, 2741, 2745, 2760, 2819, 2889, 2907, 2910, 2919, 2921,
2939, 2944, 2945, 2948, 2951, 2958, 2971, 2977, 2985, 2988, 2990, 2991,
2993, 3002, 3008, 3014, 3027, 3029, 3030, 3036, 3047, 3049, 3052, 3053,
3055, 3058, 3064, 3069, 3087, 3093, 3098, 3105, 3107, 3110, 3116, 3127,
3129, 3136, 3146, 3151, 3160, 3165, 3168, 3175, 3178, 3189, 3207, 3228,
3244, 3248, 3252, 3256, 3263, 3281, 3295, 3303, 3309, 3312, 3322, 3326,
3354, 3366, 3375, 3378, 3411, 3416, 3432, 3438, 3440, 3442, 3444, 3446,
3448, 3450, 3452, 3454, 3458, 3467, 3476, 3484, 3491, 3497, 3499, 3504,
3506, 3509, 3518, 3527, 3555, 3575, 3577, 3580, 3582, 3586, 3589, 3599,
3610, 3626, 3638, 3646, 3653, 3658, 3661, 3673, 3689, 3700, 3710, 3726,
3763, 3814, 3841, 3888, 4032, 4059, 4068, 4082, 4095, 4160, 4234, 4257,
4411, 4498, 4520, 4552, 4567, 4608, 4646, 4698, 4808, 4849, 4920, 4939,
4947, 4967, 4991, 4996, 5015, 5028, 5056, 5128, 5130, 5144, 5170, 5178,
5183, 5188, 5191, 5368, 5371, 5379, 5381, 5383, 5397, 5399, 5401, 5403,
5405, 5446, 5455, 5472, 5480, 5483, 5485, 5492, 5497, 5502, 5507, 5509,
5517, 5528, 5558, 5697, 5714, 5749, 5766, 5785, 5798, 5799, 5801, 5817,
5833, 5834, 5836, 5849, 5851, 5852, 5865, 5869, 5881, 5884, 5897, 5900,
5916, 5932, 5948, 5963, 5968, 5978, 5988, 5992, 5994, 5997, 6003, 6008,
6023, 6026, 6039, 6053, 6056, 6067, 6071, 6082, 6085, 6098, 6112, 6126,
6135, 6140, 6149, 6154, 6158, 6161, 6168, 6174, 6182, 6187, 6205, 6209,
6213, 6317, 6339, 6342, 6366, 6381, 6391, 6489, 6962, 6966, 6970, 6975,
6977, 6982, 6985, 6987, 6989, 6991, 6993, 6995, 6997, 6999, 7000, 7004,
7008, 7013, 7016, 7021, 7024, 7027, 7029, 7032, 7038, 7043, 7046, 7049,
7069, 7072, 7076, 7081, 7084, 7089, 7099, 7209, 7222, 7231, 7235, 7247,
7267, 7269, 7284, 7389, 7419, 7423, 7424, 7426, 7428, 7431, 7436, 7444,
7475, 7549, 7584, 7665, 7666, 7831, 7836, 7853, 7865, 7885, 7888, 7912,
7950, 7972, 7980, 7995, 8007, 8015, 8055, 8078
geovals:
filename: iasi_metop-a_geoval_${CDATE}_subset.nc4
obs operator:
name: CRTM
Absorbers: [H2O,O3,CO2]
obs options:
Sensor_ID: iasi_metop-a
EndianType: little_endian
CoefficientPath: crtm/
obs bias:
input file: iasi_metop-a_satbias_${GDATE}.nc4
variational bc:
predictors:
- name: constant
- name: lapse_rate
order: 2
tlapse: &iasi_metop-a_tlap iasi_metop-a_tlapmean_${GDATE}.txt
- name: lapse_rate
tlapse: *iasi_metop-a_tlap
- name: emissivity
- name: scan_angle
order: 4
- name: scan_angle
order: 3
- name: scan_angle
order: 2
- name: scan_angle
# covariance:
# minimal required obs number: 20
# variance range: [1.0e-6, 10.0]
# step size: 1.0e-4
# largest analysis variance: 10000.0
# prior:
# inflation:
# ratio: 1.1
# ratio for small dataset: 2.0

obs prior filters:
- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: *iasi_metop-a_channels
action:
name: assign error
error parameter vector: &iasi_metop-a_oberr [ 1.38, 0.81, 0.75, 0.79, 0.72, 0.74, 0.68, 0.72, 0.65, 0.65,
0.65, 0.69, 0.64, 0.64, 0.65, 0.67, 0.62, 0.61, 0.62, 0.64, 0.59, 0.76,
1.22, 0.78, 0.64, 0.62, 0.61, 0.69, 0.65, 0.59, 0.61, 0.59, 0.68, 0.62,
0.68, 4.38, 3.05, 2.31, 1.56, 1.33, 1.58, 0.93, 1.67, 0.72, 0.57, 0.58,
0.55, 0.68, 0.59, 0.68, 0.59, 0.65, 0.58, 0.62, 0.64, 0.58, 0.64, 0.55,
0.64, 0.5, 0.82, 0.59, 0.62, 0.51, 0.64, 0.52, 0.51, 0.51, 0.76, 0.52,
0.57, 0.55, 0.69, 0.58, 0.65, 0.61, 0.59, 0.64, 0.76, 0.72, 1.05, 0.75,
0.51, 0.65, 1.3, 0.69, 0.93, 1.49, 1.12, 0.68, 0.66, 0.67, 0.59, 0.59,
0.69, 0.67, 0.64, 0.62, 0.72, 0.69, 0.66, 0.79, 0.78, 0.74, 0.88, 0.77,
0.88, 0.86, 1, 0.87, 0.85, 0.88, 0.84, 0.84, 0.84, 0.8, 0.8, 0.87,
0.98, 0.52, 0.65, 0.69, 0.61, 0.6, 0.67, 0.79, 0.62, 0.66, 0.7, 0.65,
0.62, 0.61, 0.62, 0.53, 0.6, 0.68, 0.95, 0.63, 0.97, 0.65, 0.98, 0.58,
0.73, 0.65, 0.85, 0.99, 0.76, 0.85, 0.97, 0.77, 0.62, 0.63, 1.21, 1.41,
1.55, 1.78, 1.35, 1.14, 1.69, 1.79, 1.46, 1.63, 1.94, 2.01, 1.24, 1.76,
1.26, 1.47, 1.9, 1.66, 2.13, 1.49, 1.52, 1.55, 1.96, 2.31, 2.33, 2.32,
2.31, 2.33, 2.23, 2.33, 1.84, 2.29, 2.28, 2.28, 2.28, 2.26, 2.26, 2.26,
2.27, 2.24, 2.23, 2.24, 2.26, 2.28, 2.28, 2.3, 2.15, 2.31, 2.37, 2.27,
2.29, 2.29, 2.23, 2.28, 2.32, 2.32, 2.31, 2.32, 2.32, 2.31, 2.31, 2.28,
2.29, 2.28, 2.26, 2.29, 2.27, 2.26, 2.25, 2.27, 2.24, 2.21, 2.24, 2.17,
2.18, 2.17, 2.21, 1.99, 2.16, 2.2, 2.13, 2.12, 2.13, 2.1, 2.12, 2.11,
2.09, 2.09, 2.08, 2.09, 2.04, 2.04, 2.1, 2.01, 2.05, 2.03, 2.06, 1.98,
1.95, 1.94, 1.91, 1.7, 1.76, 1.77, 1.83, 2.04, 1.91, 1.99, 1.99, 2.07,
2.02, 2.04, 2.1, 2.06, 2.18, 2.21, 2.24, 2.23, 2.23, 1.98, 2.2, 2.18,
2.18, 2.21, 2.23, 2.24, 2.24, 2.25, 1.8, 2.24, 1.73, 1.73, 2.27, 1.67,
2.21, 1.72, 2.23, 2.23, 2.23, 2.24, 2.23, 2.12, 2.17, 1.74, 2.02, 1.88,
1.67, 1.73, 1.83, 1.82, 1.73, 1.83, 2.19, 1.84, 1.89, 1.6, 1.71, 1.86,
1.85, 1.84, 1.87, 1.91, 1.52, 1.95, 1.87, 1.89, 1.91, 1.91, 1.93, 1.9,
1.91, 1.9, 1.89, 1.89, 1.91, 1.9, 1.91, 1.91, 1.91, 1.93, 1.94, 1.91,
1.92, 1.77, 1.91, 1.95, 1.19, 1.96, 1.98, 1.94, 1.55, 1.91, 1.92, 1.92,
1.97, 1.93, 1.99, 1.86, 1.12, 1.93, 1.92, 1.95, 1.85, 1.84, 1.91, 1.12,
1.82, 1.82, 1.95, 1.24, 1.94, 1.96, 1.21, 1.83, 1.96, 1.36, 1.96, 1.82,
1.92, 1.68, 1.93, 1.23, 1.96, 1.93, 1.86, 1.41, 1.16, 1.6, 1.25, 1.2,
1.65, 1.66, 1.87, 1.94, 1.96, 1.91, 1.25, 1.93, 1.91, 1.7, 0.99, 1.81,
1.92, 1.95, 1.5, 1.47, 1.15, 1.58, 1.18, 1.82, 1.13, 1.83, 1.91, 1.26,
1.27, 1.91, 1.45, 1.6, 1.29, 1.94, 1.94, 1.23, 1.95, 1.21, 1.94, 1.86,
1.9, 1.33, 1.75, 2.02, 1.98, 2.03, 1.83, 1.5, 2.04, 2.02, 1.9, 2, 2.02,
1.95, 1.93, 1.95, 1.95, 1.99, 2, 1.94, 1.96, 1.86, 1.92, 1.88, 1.86,
1.84, 1.87, 1.77, 1.89, 1.89, 1.88, 1.94, 1.82, 1.79, 1.86, 2.06, 2.33,
1.88, 1.86, 1.81, 1.8, 1.8, 1.86, 1.9, 2, 2.06, 2.1, 2.2, 2, 2.16,
1.98, 1.8, 1.8, 1.85, 1.75, 2.04, 2.19, 2.14, 2.19, 1.86, 2.1, 2.11,
2.18, 2.03, 2.28, 2.19, 2.26, 2.26, 2.21, 2.21, 2.26, 2.33, 2.27, 2.21,
2.12, 2.23, 2.26, 2.25, 1.88, 2.26, 2.24, 2.36, 2.29, 2.35, 2.3, 2.27,
2.08, 2.05, 2.27, 2.28, 2.27, 2.28, 1.97, 2.25, 2.25, 2.25, 2.31, 2.28,
2.27, 2.13, 2.24, 2.28, 2.28, 2.41, 2.34, 9.32, 2.28, 2.38, 2.27, 2.27,
2.39, 2.11, 2.09, 2.1, 2.06, 2.12, 2.08, 2, 1.93, 2.02, 2.55, 1.54,
1.64, 1.51, 1.55, 2.82, 2.92, 2.55, 2.37, 1.85, 1.6, 1.72, 1.74, 1.79,
1.9, 1.94, 2, 2.04, 2.08, 2.12, 2.13, 2.16, 2.18, 2.18, 2.2, 2.2, 2.41,
2.39, 2.38, 2.4, 2.42, 2.41, 2.43, 2.45, 2.43, 2.45, 2.43, 2.4, 2.44,
2.4, 2.42, 2.43, 2.45, 2.45, 2.45, 2.46, 2.45, 2.45, 2.43, 2.51, 2.48,
2.48, 2.53, 2.46, 2.49, 2.5, 2.5, 2.5, 2.52, 2.52, 2.54, 2.5, 2.48,
2.5, 2.55, 2.5, 2.48, 2.5, 2.5, 2.52, 2.52, 2.48, 2.5, 2.5, 2.52, 2.46,
2.53, 9 ]
passedBenchmark: 801416
22 changes: 15 additions & 7 deletions parm/atm/obs/testing/iasi_metop-b.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ obs prior filters:

obs post filters:
# Wavenumber Check
- filter: BlackList
- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: 7024, 7027, 7029, 7032, 7038, 7043, 7046, 7049, 7069, 7072,
Expand All @@ -176,7 +176,8 @@ obs post filters:
minvalue: 1.0e-12
action:
name: reject
- filter: BlackList

- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: *iasi_metop-b_channels
Expand All @@ -187,6 +188,7 @@ obs post filters:
channels: *iasi_metop-b_channels
options:
channels: *iasi_metop-b_channels

# Observation Range Sanity Check
- filter: Bounds Check
filter variables:
Expand All @@ -197,8 +199,9 @@ obs post filters:
# maxvalue: 100.00000
action:
name: reject

# Topography Check
- filter: BlackList
- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: *iasi_metop-b_channels
Expand All @@ -210,8 +213,9 @@ obs post filters:
options:
channels: *iasi_metop-b_channels
sensor: iasi_metop-b

# Transmittance Top Check
- filter: BlackList
- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: *iasi_metop-b_channels
Expand All @@ -222,6 +226,7 @@ obs post filters:
channels: *iasi_metop-b_channels
options:
channels: *iasi_metop-b_channels

# Cloud Detection Check
- filter: Bounds Check
filter variables:
Expand Down Expand Up @@ -361,6 +366,7 @@ obs post filters:
maxvalue: 1.0e-12
action:
name: reject

# NSST Retrieval Check
- filter: Bounds Check
filter variables:
Expand Down Expand Up @@ -438,8 +444,9 @@ obs post filters:
maxvalue: 1.0e-12
action:
name: reject

# Surface Jacobians Check
- filter: BlackList
- filter: Perform Action
filter variables:
- name: brightnessTemperature
channels: *iasi_metop-b_channels
Expand All @@ -453,6 +460,7 @@ obs post filters:
obserr_demisf: [0.01, 0.02, 0.03, 0.02, 0.03]
obserr_dtempf: [0.50, 2.00, 4.00, 2.00, 4.00]
sensor: iasi_metop-b

# Gross check
- filter: Background Check
filter variables:
Expand Down Expand Up @@ -537,6 +545,7 @@ obs post filters:
error parameter vector: *iasi_metop-b_oberr
action:
name: reject

# Useflag Check
- filter: Bounds Check
filter variables:
Expand Down Expand Up @@ -612,5 +621,4 @@ obs post filters:
minvalue: 1.0e-12
action:
name: reject
passedBenchmark: 124731

passedBenchmark: 142616 # 124731
Loading