GFS v16 GSI minimization issues #154
Comments
|
GSI tests for normal minimization cases Compile Curves labeled cntrl on the plots below pertain to The GSI was run for 4 cycles using a stand-alone run script. These runs are not cycled. Each run is independent. Input for each run was taken from the cycled GFS v16.1 real-time parallel. The cycles run were 2021042912 to 2021043006. None of these cycles were by flagged by MinMon as having anomalous minimizations. Adding the
Again, these four cycles were not cycled. The GSI was individually run for each cycle. Below is a collection over all four cycles of the (observation - analysis) fits for sonde temperature, winds, and humidity. The similarity between the fits is not surprising given the similarity in the convergence curves. This test indicates that the |
|
The |
|
2021041718 gfs case Rerun global_gsi.x for 2021041718 gfs case with analysis fits added to gsistat. Do this for global_gsi.x built from master (cntrl) and global_gsi.x build from master with
None of the plots included thus far in this issue are from cycled DA runs. All plots are based on independent executations of the global_gsi.x for a separate cases. It would be interesting to see the effect of the |
|
2021050418 gfs case The minimization for the operational 2021050418 gfs analysis exhibited a saw-tooth pattern on the second outer loop as shown in the figure below NOAA-EMC/GSI PR #149 includes the qsat change which improved minimization in the 2021041718 gfs case. The PR #149 global_gsi.x was run for the 2021050418 gfs case. The qmin change did not improve minimization (see figure below). Through trial and error it was found that turning off correlated error over sea surfaces yielded a smoother minimization for this case Correlated error is applied to both IASI and CrIS. Additional runs of the PR #149 global_gsi.x show a smoother minimization curve when only turning off correlated error over sea points for CrIS than for only turning off correlated error over sea points for IASI
The two runs indicate, though, that turning off correlated error over IASI sea points has a greater "smoothing" effect on the gnorm curve in the first outer loop than turning off correlated error over CrIS sea points. Other cases, gfs or gdas, should be examined. It's possible minimizations for other cases are not sensitive to correlated error. A horizontal plot of the IASI and/or CrIS sea points for the 2021050418 case might prove enlightening. This issue only reports a sensitivity. Much work remains to understand the nature of this sensitivity. |
|
Kristen suggested increasing 2021051000 gfs Below is the gnorm curve for this case with Discussion between Kristen and John led to suggesting a run with
|
|
@jderber-NOAA , @KristenBathmann-NOAA , @dtkleist Additional runs and cases have been made based on today's email exchange. Thank you for the Files to run the 2021051000 gfs case are being rsync'd to Hera. Look in /scratch2/NCEPDEV/stmp1/Russ.Treadon/gfs/prod for enkfgdas, gdas, and gfs directories and files. |
|
2021051100 gfs Toggling correlated error parameters 2021051012 gfs Toggling correlated error parameters do not reduce the choppiness of the gnorm curve (not shown). Even removing all IASI and CrIS data (and therefore no correlated error) does not alter the choppiness. There's more to learn about cost function minimization in GFS v16. FYI, both the 2021051012 and 2021051100 cases were run using global_gsi.x built from PR #149. |
|
2021051012 gfs Sensitivity tests found that placing ASCAT winds in monitor mode yields a smooth gnorm curve. Placing METOP-2(A) ASCAT (report type 290, subtype 4) in monitor mode yields a smoother gnorm curve than placing METOP-1(B) ASCAT (report type 290, subtype 3) in monitor mode. One of the proposed changes for GFS v16.x is to turn on thinning for ASCAT winds. 100 km thinning was active in GFS v15. GFS v16.0 turned off this thinning. The 2021051012 gfs case was rerun with METOP-2(A) and METOP-1(B) ASCAT assimilated with two thinning grids: 50 and 100 km. The resulting gnorm curves are shown below |
|
Add Emily @emilyhcliu |
|
Russ, I am looking into a different issue with the case you put on Hera. I noted a call to reset_predictors_var within the outer loop in setuprhsall. This changes the background error covariance within the outer loop and thus the relationship between xhatsave and yhatsave will no longer be consistent. I suppose this has been included to account for the change in numbers of channels and aircraft that pass the quality control in setuprhs, but I am afraid it is leading to issues. Also, what is done in reset_predictors_var is inconsistent with what is done in set_predictors_var. The first tests I have done (just removing the call to reset_predictors_var) showed a much smoother second outer iteration. Also, the final value for the gradient went from 41 to 10. I don't think just removing the call to reset_predictors_var is the necessarily the best solution (I am trying some other things now), but I think it shows that this may be an issue. Also, it may explain why we see the sawtooth pattern mostly in the second outer iteration. |
|
Thanks, John, for the update. This sounds like a fruitful path to investigate. Perhaps making |
|
OK - I was wrong for this case. It turns out the background error does not change because the number of observations do not change enough to hit the criterion in reset_predictors_var from the first to second outer iteration. However, the routine does change the values specified in set_predictors_var and the improved convergence appears to be coming from using those original values in set_predictors_var. I think the inconsistency between xhatsave and yhatsave could occur, just not in this case. What is done in reset_predictors_var is very crude and does not seem to be as good as that in set_predictors_var. Wish I knew why this routine was included. Will look at set_predictors_var to see if we can make an improvement in these values so that reset_predictors_var is not needed. |
|
Russ, I was finally able to get things where I was happy. The output is in /scratch2/NCEPDEV/stmp1/John.Derber/tmp766/ pr149_gfs_b6.2021051000, I don't know what my changes to the background error to the radiances and aircraft temperatures, but the convergence looks better to me. Code changes are in /scratch1/NCEPDEV/da/John.Derber/jderber/src/gsi in routines berror, aircraftinfo, radinfo, and setuprhsall. John |
|
Thanks, John. I am cloning your corr branch on WCOSS_D and will try it in the cases of poor minimization. |
|
Thanks Russ. Hope it helps!JohnSent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: RussTreadon-NOAA ***@***.***> Date: 5/14/21 2:36 PM (GMT-05:00) To: NOAA-EMC/GSI ***@***.***> Cc: jderber-NOAA ***@***.***>, Mention ***@***.***> Subject: Re: [NOAA-EMC/GSI] GFS v16 GSI minimization issues (#154)
Thanks, John. I am cloning your corr branch on WCOSS_D and will try it in the cases of poor minimization.
—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
2021051000 gfs The minimization is improved. Earlier sensitivity tests altering It would be interesting to test the |
|
2021051318 gfs The operational MinMon package flagged the operational 2021051318 gfs analysis with the following message: Below is the MinMon gnorm curve for this case Through a series of sensitivity tests it was found that placing npp cris-fsr channel 1008 in monitor mode results in a much smoother gnorm curve (right). The choppy gnorm curve (left) was generated by running PR #149 CrIS fsr channel 1008 is a water vapor channel. It would be good to dig deeper into this case to better understand why assimilation of npp cris-fsr channel 1008 has this impact on the minimization. |
|
Summary (thus far) The following cases have been examined and sensitivities identified.
Note: gfs (early dump) cases have been examined thus far due to quicker turn around for stand alone gsi run script. The gfs gsi has 50 fewer iterations on the second outer loop than the gdas (late dump) gsi. |
|
Russ, I have been trying various values of kreq on the case I have on Hera. I think .5 is too large and I was trying some smaller values. I was hoping that .1 (a small value that does not change the variance too much), but I am not seeing much improvement. In fact it looks a little worse than 0.. However, somewhat smaller values (.15 and .25) seem to be a bit better. I think .25 might still be a bit too large. I really think kmult = 2. is way too large, eventhough it produces smoother results. kmult will increase the errors assigned to the radiances by a lot and I am afraid this will change things too much. John |
|
What about testing the sensitivity to a group of channels, for example only applying kreq or kmult to the water vapor channels? The GSI code doesn't allow this, but I can do it offline. |
|
2021051318 gfs update John pointed out the error in the above test. By commenting out |
|
Russ, Very nice. My test is still waiting on Hera. I hope you replaced the calls with "ptr3dges = ptr3dges + ptr3dinc" for 3d fields ("ptr2dges = ptr2dges + ptr2dinc" for 2d). Otherwise the fields are not updated. This update is not necessary for the jacobian calculations and I cannot see any issues with replacing the calls. We might want to add something that limits the fields before writing out so that the model does not get negative values. John |
Touché, you caught my mistake. I simply commented out the call to The gnorm curve remains choppy on the second outer loop. Back to the drawing board ... at least for this case. |
|
When I removed the check on minimum values in both update_guess and compute derived I am getting a much smoother reduction in gradient in the second outer iteration for the case I have on Hera. This case was not too bad in the second outer iteration, so I cannot say how much it will help in other cases. The first outer iteration was identical, indicating the that qmin was being applied earlier in the code as well. This looks very promising. |
|
The third case Russ put on Hera (2021052006) does not appear to be moisture related. By changing preconditioning I have been able to get a significant change in the convergence properties, but still resulted in a magnitude of the gradient that was a bit too large at the end of the second outer iteration. I am looking into some changes that might help. |
|
This sounds very promising. The operational 2021052318 gfs has a choppy gnorm curve on both outer loops. We may want to test your preconditioning changes on this case, too. |
|
Not much progress through the preconditioning route. Some improvement on the smoothness, but gradient still ends too high. Appears to be a problem related to the ensembles, but getting weird results (i.e., input parameters are changed and after read in they are set correctly, but then they change back to the original values.) |
|
John opened issue #170 to document changes to bias correction variances. These changes improve convergences in certain cases and overall improve consistency between outer loops. Please see this issue for details regarding his changes. |
|
2021060806 gfs The operational MinMon package flagged the operational 2021060806 gfs analysis with the following message: Below is the MinMon gnorm curve for this case Through trial and error it was found that placing Metop-A and Metop-B IASI channels 2701 to 3310 (8 channels in all) in monitor mode results in a much smoother gnorm curve. Tests placing eight IASI channels from 2701 to 3310 in monitor mode from a single satellite show Metop-A contributes more to the choppy gnorm curve than Metop-B for this case. The above subset of monitored channels can be narrowed to {2993, 3002, 3049} and still yield a smoother gnorm curve than the original operational run. What atmospheric constituents are these IASI channels sensitive to? What parts of the atmospheric column are these channels sensitive to? It would be interesting to generate horizontal maps to see where large innovations occur. |
|
2021061918 gfs While MinMon did not flag the operational gfs analysis for 2021061918, the gnorm curve is choppy for this cycle. The NOAA-EMC/GSI master at a67b816 was built and used to run the 2021061918 case. The gnorm curve remains choppy with the master global_gsi.x. Three tests were subsequently run:
The resulting gnorm curves are shown below.
The improvement in convergence from small changes (e.g. placing 7 channels in monitor mode) is considerable. These results and those for other cycles suggest that we should take a closer look at the assimilation of CrIS and IASI water vapor channels in v16. This is not meant to say that there is a problem with assimilation of these water vapor channels. The sensitivity of the gnorm curve to these channels may lead other aspects of the analysis (e.g., ensembles, correlated error, increased vertical resolution, ...). |
|
Russ,
Your results would indicate to me that we should be using larger
observation errors and/or not using these channels for the cloudy radiance
assimilation.
John
…On Tue, Jun 22, 2021 at 1:18 PM RussTreadon-NOAA ***@***.***> wrote:
*2021061918 gfs*
While MinMon did not flag the operational gfs analysis for 2021061918, the
gnorm curve is choppy for this cycle.
[image: GFS_gfs 2021061918 gnorms]
<https://user-images.githubusercontent.com/26926959/122967772-cf186f80-d358-11eb-80a8-6d7489d8484e.png>
The NOAA-EMC/GSI master at a67b816
<a67b816>
was built and used to run the 2021061918 case. The gnorm curve remains
choppy with the master global_gsi.x.
[image: gnorm_master all_data_2021061918]
<https://user-images.githubusercontent.com/26926959/122969004-279c3c80-d35a-11eb-9125-0cb8f78b1f69.png>
Three tests were subsequently run:
1. thin ASCAT winds to 75 km
2. monitor Metop-A and Metop-B IASI water vapor channels 2889 to 5480
(10 channels for each satellite)
3. monitor CrIS-FSR N20 water vapor channels 882 to 1058 (7 channels).
All CrIS-FSR NPP channels are monitored due to an instrument anomaly.
The resulting gnorm curves are shown below.
[image: gnorm_master thin_ascat75_2021061918]
<https://user-images.githubusercontent.com/26926959/122969019-2c60f080-d35a-11eb-9dc0-cc7aa6d50456.png>
Thinning ASCAT data to 75 km does not yield a smoother gnorm curve for
this case.
[image: gnorm_master iasi_mon2889_5480_2021061918]
<https://user-images.githubusercontent.com/26926959/122969086-426eb100-d35a-11eb-8068-76cf3a67d891.png>
Monitoring Metop-A/B IASI water vapor channels does not impact the gnorm
curve on the first outer loop but the gnorm curve on the second outer loop
is much smoother.
[image: gnorm_master crisn20_mon882_1058_2021061918]
<https://user-images.githubusercontent.com/26926959/122969217-63370680-d35a-11eb-9c6c-e994632a8085.png>
Monitoring CrIS-FSR N20 water vapor channels impacts both the first and
second outer loops.
The improvement in convergence from small changes (e.g. placing 7 channels
in monitor mode) is considerable. These results and those for other cycles
suggest that we should take a closer look at the assimilation of CrIS and
IASI water vapor channels in v16. This is not meant to say that there is a
problem with assimilation of these water vapor channels. The sensitivity of
the gnorm curve to these channels may lead other aspects of the analysis
(e.g., ensembles, correlated error, increased vertical resolution, ...).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#154 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ASD2M5TWDQOGKFZEBQRNTN3TUDAUTANCNFSM434IKOQQ>
.
|
|
Reran 2021061918 gfs case with satinfo Rerain 2021061918 gfs case with satinfo
Rerun with incrementally smaller The second outer loop gnorm curve is progressively smoother as One final run was made with For previous cases we experimented with adjustments to correlated error parameters Earlier on in this issue Kristen mentioned the following channel specific option
Kristen, do we have the necessary code modifications in hand to run channel specific |
|
We use correlated error for CrIS over sea surfaces only, so non-sea surfaces continue to use the satinfo errors. Running channel specific tests with kreq and kmult is very instrument-dependent, so I can either supply separate covariance files for each option you want to test, or make some inelegant code changes to the GSI. |
|
Rather than generate a bunch of Rcov files for cris-fsr_n20, I'd opt for
the inelegant GSI code changes for exploratory testing. Which option do
you prefer?
…On Thu, Jun 24, 2021 at 8:15 AM Kristen Bathmann ***@***.***> wrote:
We use correlated error for CrIS over sea surfaces only, so non-sea
surfaces continue to use the satinfo errors. Running channel specific tests
with kreq and kmult is very instrument-dependent, so I can either supply
separate covariance files for each option you want to test, or make some
inelegant code changes to the GSI.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#154 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGNN634RO25HFLFXVDTE6LTTUMOWRANCNFSM434IKOQQ>
.
|
|
Changing the GSI code would probably be the easiest option for testing. Are these tests are based on the GSI master, or on John's corr branch? |
|
… On Thu, Jun 24, 2021 at 10:42 AM Kristen Bathmann ***@***.***> wrote:
Changing the GSI code would probably be the easiest option for testing.
Are these tests are based on the GSI master, or on John's corr branch?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#154 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AGNN634KPERP5AYBTBHD2XDTUM74BANCNFSM434IKOQQ>
.
|
|
I created a branch called kreqtune to allow for separate values of kreq for surface, water vapor and other channels. This is an additive inflation value kreq_surf and kreq_wv are the inflations for the surface and water vapor channels. I also updated the Rcov_cris* covariance files to the versions I plan to use when IASI-C is implemented. |
|
Rerun with correlated error turned off for CrIS-FSR N20. Runs using John's biasvar and corr branches do not yield smoother gnorm curves. CrIS-FSR N20 water vapor channels seem to be the key for this case. Rather than adjusting correlated error would it suffice to adjust satinfo parameter |
|
It might be useful to look at the jacobains in the diag files, but this requires re-running with certain options turned on. I have a code somewhere that can plot jacobians. It would be sufficient to just look at the control file, and examine observations that have 3(O-G)>0.4. |
|
Install Kristen's kreqtune on Venus and run 2021061918 case. Below is the resulting gnorm curve. This run used updated Rcov files in the kreqtune branch. CrIS-FSR N20 Repeat this test with Repeat this test with |
|
Here are spaghetti plots of the temperature, humidity and ozone jacobians for CrIS channel 1058, from the operational gfs diag file at 2021061918:
The jacobians are definitely larger with errmax>0.4. The ozone jacobians look a bit odd. |
|
To clarify on my last comment, plots with errmax>0.4 show jacobians of observations that would pass quality control if errmax is greater than 0.4. Plots with errmax<0.4 show jacobians of observations that would pass quality control if errmax is less than 0.4. |
|
2021070106 gfs MinMon flagged the operational GFS 2021070106 gfs cycle with the following alert Below is the gnorm curve for this case Replicated this gnorm behavior using NOAA-GSI/EMC master at 17cddf0. Rerun this case using stand-alone GSI script testing various configurations. Find gnorm sensitivities to ASCAT thinning and correlated error tuning for CrIS and IASI water vapor channels. Use Kristen's kreqtune branch at 7a1b66a to explore gnorm sensitivity to CrIS and IASI water vapor error tuning. Best gnorm behavior obtained with the following configuration
The resulting gnorm curve for this configuration is shown below |
|
2021070106 gfs (update)
Turning off assimilation of CrIS and IASI does not yield a smooth minimization. Kristen's kreqtune branch with kreq_wv = 0.2 for CrIS (sea) and IASI (sea and land) yields the following gnorm curve with ensembles (ie, 4denvar) used. |
|
2021072312 gfs The operational MinMon package flagged 2021072312 gfs with the following alert
This is a case of poor convergence. Various sensitivity tests have been run by Kristen. In all tests correlated error was turned off.
Additional sensitivity tests have been run on Mars. The best convergence thus far has been obtained by not assimilating IASI and CrIS along with thinning ASCAT winds to 75 km. Additional tests should be run. |
|
Some changes in minimization technique. In the right direction, but more improvement necessary
|
|
Both the gfs and gdas cycles for 2021073018 were flagged by MinMon for minimization resets (4 gfs, 3 gdas). Below are the gnorm curves for each Various sensitivity tests were run for the gfs case using a standalone GSI run script and the master at 3656d50. Thinning ASCAT winds to 75 km by itself did not noticeably improve the minimization. This agrees with a check of the The best convergence for 4denvar was obtained by thinning ASCAT winds to 75 km and setting the ensemble contribution to the q background error to zero. Attempts to reduce the ensemble q perturbations above the tropopause did not improve convergence. Sensitivity to q perturbations may be in the lower troposphere where the L127 model has extra layers compared with L64. |
|
2021073018 gfs Additional tests were run in which the ensemble q perturbations were set to zero on specified ranges of model layers. The greatest impact on convergence (in a positive sense) was found when zeroing the ensemble q perturbations on layers 42 to 55. A check of the atmanl.nc shows layers 42 to 55 approximately correspond to 669 to 455 hPa. |
|
2021081906_gfs MinMon flagged the 2021081906 gfs cycle with the following warning
Below is the gnorm curve. No resets occurred in the minimization. This is a case of poor convergence.
Examination of Previous cases have shown sensitivity to CrIS and/or IASI water vapor channels. The master Three more runs were made, each placing water vapor channels from only one sensor in monitor mode. The goal here was to see if the smoothness of the above gnorm curve correlates with a specific satellite. Placing N20 CrIS or Metop-B IASI water vapor channels in monitor mode yields gnorm curves with cyclic behavior on the second outer loop. Placing Metop-A IASI water vapor channels in monitor mode yields the smoothest gnorm curve. Additional tests could be run to quantify the impact of specific Metop-A IASI water vapor channels on the gnorm curve. |
|
20211118 06 & 18Z gfs MinMon flagged the operational gfs analyses for insufficient convergence on 2021111818 in the 06, and 18Z cycles. Sensitivity tests using the master at f442020 were run for each cycle. The resulting gnorm curves are available as links in the table below. All rows except the first, gfs.v16.1.x, ran the master global_gsi.x with the indicated configuration. Configurations are explained below the table.
The following configurations were tested
For both cases placing IASI and CrIS water vapor channels in monitor mode and turning off correlated error for the remaining assimilated IASI and CrIS channels yielded much smoother gnorm curves. Adding to this the thinning of ASCAT winds and increased ASCAT wind observation error yielded additional improvements. Gnorm curves for sensitivity tests not mentioned above are found in |
|
20211118 06 & 18Z gfs follow up Run an additional tests toggling on/off correlated error for entries in global_anavinfo.l127.txt. Gnorm curves for various tests are tabulated below
Examination of the above plots shows that turning off correlated error for Metop-C IASI over sea has the greatest impact in terms of yielding a smoother gnorm curve.
Turning off correlated error for N20 CrIS does not yield as great an improvement in convergence as does turning off correlated error over sea for IASI. Some interconnected questions:
Other ideas or thoughts? |
|
(Try testing with the kreqtune branch and remember to add columns to the anavinfo. Use negative values for kreq* for CrIS NPP.) |
|
2021112118 gdas and 2021112200 gfs MinMon flagged the operational 2021112118_gdas and 2021112200_gfs with minimization resets. Given findings from the 20211118 gfs cases, the f442020 master global_gsi.x was run with the following configuration
The resulting 2021112118_gdas and 2021112200_gfs gnorm curves are much smoother with significantly improved reduction in the gradient norm. Additional tests for the 2021112200_gfs case show that the combination of monitoring IASI and CrIS water vapor channels along with turning off correlated error over sea points contributes the most to smoothing the gnorm curve and improving convergence. Thinning ASCAT to 75 km and increasing the observation error improves convergence, but this is a secondary effect for this case. Click here to view gnorm curves for other tests run for 2021112200_gfs. |
|
2021112506 gfs MinMon flagged the operational 2021112506 gfs cycle for insufficient reduction.
Various sensitivity tests were run
For this case the combination of thinning ASCAT winds to 75 km with an increased observation error along with monitoring IASI and CrIS water vapor channels (test 3) yields the greatest improvement in convergence with respect to operations. Turning off correlated error over all land or sea points in test 3 improves convergence a bit. The overall best gnorm performance is obtained by combining test 3 with turning off correlated error off over sea points. Gnorm curves for all tests run for this case are in https://www.emc.ncep.noaa.gov/gmb/wd20rt/plots/gnorm/gfs_2021112506/. |
|
2021112906 gfs and gdas and 2021112912 gfs MinMon flagged the 2021112906 gfs, 2021112906 gdas, and 2021112912 gfs cycles for minimization resets and insufficient convergence. Sensitivity tests were run for the two gfs cycles. The greatest gnorm reduction was obtained with the following combinations:
|
|
@jderber-NOAA has extensively investigated GSI minimizations issues. He has developed a set of changes which significantly improve the minimization. NOAA-EMC/GSI issue #375 has been opened to document these changes. |
The operational MinMon package occasionally detects and reports problematic GFS v16 GSI minimizations. One such case was the operational 2021041718 gfs cycle (prod curve in figure below). Investigation of this case found that placing a lower bound of 1.e-07 on the saturation specific humidity computed in
genqsat.f90yielded a much smoother minimization (master_qmin in figure below).Constant
qminis defined inconstants.f90asreal(r_kind),parameter:: qmin = 1.e-07_r_kind ! lower bound on ges_qThe corresponding change in
genqsat.f90isqsat(i,j,k) = max(qmin,qsat(i,j,k))This issue is opened to document further evaluation of this change. Pending evaluation results this change will be proposed for merger into the master.
The text was updated successfully, but these errors were encountered: