Invalid values when submesoscale variables are masked

[cbegeman]

Fill value masking of the submesoscale variable normalMLEVelocity resulted in bad values in vertMLEBolusVelocityTop (Inf) that affected timeMonthly_avg_mocStreamvalLatAndDepth. The issue arises when (a) the masking array for normalMLEVelocity is specified in the Registry and (b) the normalMLEVelocity variable is included in a timeSeriesStats stream (regardless of whether masking is turned on via useMissingValMask). In MPAS-Ocean standalone, this bug manifests as state validation errors.

[cbegeman]

The underlying issue may be due to inadequate initialization of normalMLEVelocity. However, when I attempted to initialize the array to zeros, the QU240 simulation crashed at runtime.

[xylar]

We are seeing -Infinity values in both timeMonthly_avg_vertMLEBolusVelocityTop and timeMonthly_avg_mocStreamvalLatAndDepth in monthly output (already visible in the first month):

$ pwd
/lcrc/group/acme/ac.jwolfe/scratch/anvil/230131.PR5254base.anvil/run
(mpas_dev) [ac.xasay-davis@chrlogin1 run]$ ncdump -v timeMonthly_avg_mocStreamvalLatAndDepth 230131.PR5254base.anvil.mpaso.hist.am.timeSeriesStatsMonthly.0010-12-01.nc | tail
    Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, 
    Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, 
    Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, 
    Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, 
    Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, 
    Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, 
    Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, 
    Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, Infinityf, 
    Infinityf ;
}

and

$ ncdump -v timeMonthly_avg_vertMLEBolusVelocityTop 230131.PR5254base.anvil.mpaso.hist.am.timeSeriesStatsMonthly.0010-12-01.nc | tail
  -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, 
    -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, 
    -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, 
    -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, 
    -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, 
    -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, 
    -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, 
    -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, -Infinityf, 
    -Infinityf, -Infinityf, -Infinityf, -Infinityf, 0, 0, 0, 0, 0, 0, 0, 0, 0 ;
}

We are seeing -0 values in timeMonthly_avg_normalMLEvelocity:

$ ncdump -v timeMonthly_avg_normalMLEvelocity 230131.PR5254base.anvil.mpaso.hist.am.timeSeriesStatsMonthly.0010-12-01.nc | tail
  -0.009481622, 0.001539794, 0.006025193, 0.001916643, -0, -0, -0, -0, -0, 
    -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, 
    -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, 
    -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, 0,
  0.006610637, 0.001642826, -0.0005252801, -0.002020832, -0.002636817, 
    -0.002759336, -0.000311133, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, 
    -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, 
    -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, 0, _, _, 
    _, _, _, _, _ ;
}

As a result, the MOC cannot be plotted, and it can also not be computed in post-processing.

@jonbob has traced this to #5325.

[xylar]

It appears this went untested because these fields are not part of any E3SM or compass tests at this point. We will attempt to remedy this and report back here.