Update IASI YAML files for end-to-end testing #769
Conversation
…ython script for iasil Update data list for 3dvar
|
Add 8 files changed in this PR to a working copy of @RussTreadon-NOAA You acted fast!! I just added the information in this PR that it comes with a paring UFO PR. Please see the description part. You will need to check out a UFO branch: feature/satrad in gdas-validation. |
|
GDAS Validation Results (End-to-End) --- IASI Window Channel (921) Comparison Thinned Data between JEDI and GSI Thinned and Data from the Edge of the Scan Removed (GSI) Comparison Data passed QC between JEDI and GSI Data passed QC (GSI) Notes:
|
|
|
||
| ioda: | ||
| backend: netcdf | ||
| obsdataout: "{{ COM_OBS }}/{{ RUN }}.t{{ cyc }}z.mtiasi_$(splitvar).tm00.nc" |
There was a problem hiding this comment.
any reason why this needs to be mtiasi and not iasi? I ask because our convention has been lately $sensor_$platform so it would match the CRTM coefficients. But if there are different input streams, then it makes sense to distinguish.
There was a problem hiding this comment.
@cory For eacg IASI platform has three bufr sources:
gdas.t00z.mtiasi_metop-a.tm00.bufr_d --- normal feed
gdas.t00z.esiasi_metop-a.tm00.bufr_d --- EARS feed
gdas.t00z.iasidb_metop-a.tm00.bufr_d--- direct broadcast feed
The one I am working on in this PR is the normal feed. So, I use mtiasi in the filename to note that this file is converted from the normal feed.
In the future, I will need to merge these three files into one, which will be called iasi_metop-a for the combined IODA file.
|
Install UFO branch Notice that the The log file with Is this simply a printout error or indicative of a bug in the code? If we rerun without See log file |
@RussTreadon-NOAA I am investigating........ |
|
Does |
No, it does not. I re-ran iasi -metop-b case using The misspelled brightnessTemper?ture marked as 'user error' may be cuased by OOM. /work2/noaa/da/eliu/gdas-validation/comrot/gdas_eval_iasi_JEDI/logs/2021080100/ gdasatmanlrun.log I re-ran ATMS case in the global-workflow which ran successfully before. The ATMS run got killed without an obvious reason. Here is the message in the log: /work2/noaa/da/eliu/gdas-validation/comrot/gdas_eval_atms_JEDI/logs/2021080100/ gdasatmanlrun.log |
|
I would expect |
After my issue with global-workflow setup step was resolved, I set up a new experiment for ATMS which should run to the end without a problem. I still got workflow exception message (see below) |
|
Orion tests Successfully run Caveat: I did not use the original yaml created but atmanlinit. Two modifications have been made
|
|
IASI bias correction minimization error First JEDI library reference is ufo. Is this a useful clue? Have the JEDI core team or JEDI partners assimilated IASI with variational bias correction active? |
|
@CoryMartin-NOAA suggested retaining The increments are small but the identity B is used for a single iteration. The strange order 1e-20 cloud ice and cloud water increments remain. We need to figure out where these increments come from. We should see 0.000000e+00 increments for these variables. An attempt to run with both metop-a and metop-b iasi died with an OOM kill prior to entering the minimization. ppn=8 is too dense for two iasi instruments. I'll try ppn=6 or 4 later for both instruments.
|
@RussTreadon-NOAA This error message comes from the UFO ObsBiasCovariance::linearized code. It seems to me that the ObsBiasCovariance code does not handle the dimension correctly when both Thanks for your help in fixing the memory issue in the global-workflow and pointing out that the problem likely comes from the ObsBias part. We won't see the problem unless the memory issue gets fixed first. |
|
Thank you, @emilyhcliu , for agreeing that we are likely dealing with a code bug and identifying the possible cause. Hopefully the fix is not too difficult.
My runs to date use an {8,8} layout. The two iasi run was repeated with a {10,10} layout. This required 150 nodes with ppn=4, 1 thread. Per task memory usage ranged from 18.26 to 18.39 Gb with The initial (pre-minimization) observation statistics differ between the {8,8} and {10,10} runs. {10,10} layout {8,8} layout This seems wrong. Changing the layout shouldn't alter QC (and analysis) results. The GSI generates the same analysis independent of task and thread count. Is this above Tagging @DavidNew-NOAA since you are exploring memory, wall time, and reproducibility issues in JEDI applications. This is an important study as we incrementally move towards operational implementation of JEDI apps. |
|
@RussTreadon-NOAA The IASI (with ObsBias included) test can run to completion with this fix. I also commented out the DiagnosticFlags in the YAML to save memory use. The HofX and data filtering results are comparable with GSI. There are non-zero increments for clouds (they should be zeros). |
|
I think 1e-20 for a max value is close enough to zero for the clouds. |
|
Wait, why are all the increments so small. Temperature increments on the order of 1e-14??? |
|
Here is the result from @RussTreadon-NOAA without ObsBias included. The increments are small as well |
|
Thanks @emilyhcliu oh it's the identity B, ok, then it's probably fine. Thanks for the fix! |
There was a problem hiding this comment.
Added change from UFO PR #3131 to working copy of gdas-validation ufo. Recompile and rerun 2021080100. atmanlrun completed without error with VarBC active for iasi metop-a and metop-b.
Approve pending merger of required UFO PRs into authoritative develop.
|
@RussTreadon-NOAA The PRs related to IASI are: The other PRs are for MW: |
|
Thank you, @emilyhcliu , for the list of UFO PRs related to the GDAS-validation sprint. My question, however, is different. You mention four UFO PRs in the description of this PR. Currently three of the four have been merged into jcsda-internal UFO
The UFO PR #3122 will only affect all-sky MW in the end-to-end test. |
|
@RussTreadon-NOAA I think the validation sprint branch. But that begs the larger question, when should the sprint branch be merged into develop? I'm not sure it's required (in my mind) that 100% reproducibility exists to go into |
|
Agreed. I rebuilt gdas-validation on Orion. My snapshot of UFO is the authoritative The fact that UFO PR #3122 remains a draft does not produce a failure when the changes in GDASApp PR #769. are exercised in gdas-validation. Given this, I think this PR can be merged into feature/gdas-validation. That said, I'm not on the developer side of the gdas-validation PRs. I defer to the developers as to when it's the appropriate time to merge gdas-validation sprint PRs into develop. Hence this conversation with @emilyhcliu to see if it's OK to go ahead and merge this PR into gdas-validation. |
|
Absent objections for anyone I will merge this PR into |
This PR includes the following updates:
data thinninganddiagnostic flagsincludedThis PR has a paring UFO PR #3121 to fix the dimension for
sensorCentralWavenumberTo test this PR, please check out the UFO branch: feature/satrad from JCSDA-internal. This branch consolidates all proposed code changes to UFO for gdas validation:
UFO PR #3122 --- add cloud seeding (for all-sky)
UFO PR #3121 --- generalize handling for sensor wavenumber
UFO PR #3094 --- fix ObsError bug and minor fix for Hydrometeor check
UFO PR #3122 --- fix ObsBiasCovariance::linear bug
The test results in global-workflow encountered Out-Of-Memory (OOM) issue.
Therefore, the test of the update was performed outside of the global-workflow using the fv3 nomodel executable. The data filtering results are comparable between GSI and JEDI (check validation results here)