Changes to updating of bias correction variances

[jderber-NOAA]

While looking at issue 154 (convergence issues), it was noted that in certain cases the convergence was much worse in the second outer iteration than the first. There is no good reason for this unless there was a large change in the observations used in the outer iteration. We do not see a large change in the number of observations so there must be another reason. Looking at the code from one outer iteration to another it was noted that there was a new routine added "reset_predictor_vars". By resetting the variances an inconsistency between the definition of xhatsave and yhatsave was introduced. It was also noted that the updating of the variances was confused.

Modifications were made to the code to remove the call to the reset_predictors_var routine and other changes were made to the updating of the variances. The code was simplified to:

1. initialize the variances for new observations (new tail numbers, new instruments ) to 10000. This is basically saying that there is no information in the guess.
2. Update the variances at the beginning of the analysis with consistent factors. The code updated the variances differently dependent on the number of observations available. This does not really make sense since the update to the variances is being done because of the uncertainty of the variances from one analysis time to the next. The number of observations enter into the next step.
3. The number of observations and other values in rstats (and rstats_t) are used to update the variances in pcinfo. This routine is called each outer iteration and accounts for changes from one outer iteration to the next for the change in the number of observations used.
4. Default values for the variances are set to 10000. rather than zero. A maximum on the variance is also set to 10000. This says that when insufficient data or zero data are available, the variance is a large number rather than zero.

[jderber-NOAA]

Changes are put into branch biasvar.

In occasional cases (where the convergence is bad in the second outer iteration), these changes appear to eliminate convergence issues. In other cases, there appears to be some improvement in the convergence, but not completely eliminate the issue. It appears to be a step in the right direction. The improvements are not as dramatic as those resulting from thinning ascat.

[jderber-NOAA]

It was also noted that the use of a minimum value in compute_derived and update_guess makes xhatsave and yhatsave inconsistent. The minimum value for qmin was removed from compute_derived. In update_guess, the minimum values (qmin and tgmin) were only applied in the last outer iteration (jiter = jiterend). This change also has an impact on the convergence after the first outer iteration.

[MichaelLueken]

Hi @jderber-NOAA. Looking through the updates made to the authoritative repository, it looks like changes associated with this issue were merged at d13b3f8 from PR #242. Is it safe to close this issue?

[jderber-NOAA]

Mike, Yes! You are correct this issue can be closed. John

…

On Wed, Aug 24, 2022 at 8:39 AM MichaelLueken-NOAA ***@***.***> wrote: Hi @jderber-NOAA <https://github.com/jderber-NOAA>. Looking through the updates made to the authoritative repository, it looks like changes associated with this issue were merged at d13b3f8 <d13b3f8> from PR #242 <#242>. Is it safe to close this issue? — Reply to this email directly, view it on GitHub <#170 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASD2M5UWT532VIIH4LJ3QALV2YJWZANCNFSM455CZQGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[MichaelLueken]

@jderber-NOAA Thanks for the confirmation! Will close this issue now.