Reproducibility issue: hafs_regional_specified_moving_1nest_atm on orion and wcoss2 (intel)

[jkbk2004]

Description

Baseline of hafs_regional_specified_moving_1nest_atm does not reproduce only on orion and wcoss2. The case was introduced with PR #1362 and the the issue was captured in PR #1399. Baseline is created ok but comparison doesn't pass.

To Reproduce:

What compilers/machines are you seeing this with?
Give explicit steps to reproduce the behavior.

1. rt.sh -c -e and rt.sh -m -e
2. comparison log doesn't pass: Comparing HURPRS.GrbF06.nest02 .........NOT OK

Additional context

absv (absolute vorticity) fields are not identical in the grib2 file comparison. No issue is identified on other dynamical fields.

-- GRIB #12 -- shortName=absv paramId=3041 stepRange=6 levelType=pl level=2 packingType=grid_complex_spatial_differencing gridType=rotated_ll --
long [totalLength]: [174022] != [170619]
long [numberOfGroupsOfDataValues]: [11265] != [11111]
long [section7Length]: [142432] != [139029]
double [codedValues]: 1172 out of 154988 different
max absolute diff. = 3.3320000000000008e-04, relative diff. = 1.18156
max diff. element 56383: -2.82000000000000078295e-04 5.12000000000000112356e-05
tolerance=0.0000000000000000e+00
values max= [0.0003898] [0.0003898] min= [-0.000282] [-0.000282]
-- GRIB #12 -- shortName=absv paramId=3041 stepRange=6 levelType=pl level=2 packingType=grid_complex_spatial_differencing gridType=rotated_ll --
long [totalLength]: [174022] != [170619]
long [numberOfGroupsOfDataValues]: [11265] != [11111]
long [section7Length]: [142432] != [139029]
double [codedValues]: 1172 out of 154988 different
max absolute diff. = 3.3320000000000008e-04, relative diff. = 1.18156
max diff. element 56383: -2.82000000000000078295e-04 5.12000000000000112356e-05
tolerance=0.0000000000000000e+00
values max= [0.0003898] [0.0003898] min= [-0.000282] [-0.000282]

Output

Screenshots
If applicable, drag and drop screenshots to help explain your problem.

output logs
If applicable, include relevant output logs.
Either drag and drop the entire log file here (if a long log) or

paste the code here (if a short section of log)

[jkbk2004]

GRIB2-file-from-python.pptx

[DusanJovic-NOAA]

I just ran hafs_regional_specified_moving_1nest_atm in develop branch on Orion, and it reproduced the current baseline:

$ cat log_orion.intel/rt_001_hafs_regional_specified_moving_1nest_atm.log 

baseline dir = /work/noaa/nems/emc.nemspara/RT/NEMSfv3gfs/develop-20220830/INTEL/hafs_regional_specified_moving_1nest_atm
working dir  = /work/noaa/stmp/djovic/stmp/djovic/FV3_RT/rt_24260/hafs_regional_specified_moving_1nest_atm
Checking test 001 hafs_regional_specified_moving_1nest_atm results ....
 Comparing atmf006.nc .........OK
 Comparing sfcf006.nc .........OK
 Comparing atm.nest02.f006.nc .........OK
 Comparing sfc.nest02.f006.nc .........OK
 Comparing HURPRS.GrbF06 .........OK
 Comparing HURPRS.GrbF06.nest02 .........OK

  0: The total amount of wall time                        = 262.586697
  0: The maximum resident set size (KB)                   = 539176

Test 001 hafs_regional_specified_moving_1nest_atm PASS