Add valid time to the 6 tile cubed sphere grid history files

[DusanJovic-NOAA]

PR Checklist

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.

• This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR

• An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
  are specified below.

• Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.

• New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Description

Add 'time_iso' variable to cubed sphere grid output when ESMF write routines are used. This variable has already been added to the parallel version of the netcdf write routine.
This PR includes changes from NOAA-EMC/fv3atm#515

New regression test has been added that outputs history files with all 6 cubed sphere tiles in a single file.

Issue(s) addressed

Link the issues to be closed with this PR, whether in this repository, or in another repository.
(Remember, issues must always be created before starting work on a PR branch!)

• fixes Add valid time to the 6 tile cubed sphere grid history files #1252

Testing

How were these changes tested? Yes.
What compilers / HPCs was it tested with? Intel. Hera
Are the changes covered by regression tests? Yes.
Have the ufs-weather-model regression test been run? On what platform?
Will the code updates change regression test baseline? Yes. New netcdf variable has been added to the history files.

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss2.intel
• wcoss_cray
• wcoss_dell_p3
• opnReqTest for newly added/changed feature
• CI

Dependencies

If testing this branch requires non-default branches in other repositories, list them. Those branches should have matching names (ideally).

Do PRs in upstream repositories need to be merged first? Yes.
If so add the "waiting for other repos" label and list the upstream PRs

• waiting on Add valid time to the 6 tile cubed sphere grid history files NOAA-EMC/fv3atm#550

[junwang-noaa]

Thanks for updating the code, the changes look good to me.

[DusanJovic-NOAA]

I had to revert commit that changes the calendar type from 'julian' to 'gregorian' because some restart (warm start) tests have calendar type set to julian in the input data:

$ find . -name coupler.res
./FV3_input_data384/INPUT_L127_GDAS/coupler.res
./FV3_input_data_conus13km/INPUT/coupler.res
./FV3_input_data_L149/INPUT/coupler.res

$ cat ./FV3_input_data384/INPUT_L127_GDAS/coupler.res
     2        (Calendar: no_calendar=0, thirty_day_months=1, julian=2, gregorian=3, noleap=4)
  2021  03  21  18     0     0        Model start time:   year, month, day, hour, minute, second
  2021  03  21  21     0     0        Current model time: year, month, day, hour, minute, second

$ cat ./FV3_input_data_conus13km/INPUT/coupler.res
     2        (Calendar: no_calendar=0, thirty_day_months=1, julian=2, gregorian=3, noleap=4)
  2021     5    12    16     0     0        Model start time:   year, month, day, hour, minute, second
  2021     5    12    16     0     0        Current model time: year, month, day, hour, minute, second

$ cat ./FV3_input_data_L149/INPUT/coupler.res
     2        (Calendar: no_calendar=0, thirty_day_months=1, julian=2, gregorian=3, noleap=4)
  2021     3    22     6     0     0        Model start time:   year, month, day, hour, minute, second
  2021     3    23     0     0     0        Current model time: year, month, day, hour, minute, second

Changing calendar will require new input data directory.

[jkbk2004]

@DusanJovic-NOAA Brian wants a bit more time with AQM PR. Can we start working on this PR to create new BL?

[BrianCurtis-NOAA]

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: BL
[BL] Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/963206649/20220623180021/ufs-weather-model
[BL] Error: Test control_csawmg_debug 051 failed in run_test failed
[BL] Error: Test control_thompson_debug 056 failed in run_test failed
[BL] Error: Test regional_debug 060 failed in run_test failed
[BL] Error: Test hafs_regional_atm_ocn_wav 078 failed in run_test failed
[BL] Error: Test datm_cdeps_iau_gefs 084 failed in run_test failed
[BL] Error: Test datm_cdeps_stochy_gefs 085 failed in run_test failed
[BL] Error: Test datm_cdeps_bulk_gefs 087 failed in run_test failed
[BL] Error: Test datm_cdeps_mx025_cfsr 088 failed in run_test failed
[BL] Error: Test datm_cdeps_3072x1536_cfsr 090 failed in run_test failed
[BL] Error: Test datm_cdeps_gfs 091 failed in run_test failed
[BL] Error: Test compile_006 failed in run_compile failed
[BL] Error: Test compile_017 failed in run_compile failed
Please make changes and add the following label back: jet-intel-BL

[BrianCurtis-NOAA]

WCOSS_Dell_P3 on baseline creation: + echo 'Test cpld_debug_noaero_p8 007 failed in run_test failed '
Test cpld_debug_noaero_p8 007 failed in run_test failed

+ echo 'Test cpld_debug_noaero_p8 007 failed in run_test failed '
+ read -r failed_test_name
+ echo 'Test control_stochy_debug 051 failed in run_test failed '
Test control_stochy_debug 051 failed in run_test failed
+ echo 'Test control_stochy_debug 051 failed in run_test failed '
+ read -r failed_test_name
+ echo 'Test rap_rrtmgp_debug 071 failed in run_test failed '
Test rap_rrtmgp_debug 071 failed in run_test failed
+ echo 'Test rap_rrtmgp_debug 071 failed in run_test failed '

[BrianCurtis-NOAA]

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: RT
[RT] Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/963206649/20220624130021/ufs-weather-model
[RT] Error: Test rrfs_v1beta 058 failed in check_result failed
[RT] Error: Test rrfs_v1beta 058 failed in run_test failed
[RT] Error: Test control_ras 067 failed in run_test failed
Please make changes and add the following label back: jet-intel-RT

[jkbk2004]

@BrianCurtis-NOAA we need to copy over two BL files on jet. @DeniseWorthen all test passed. You can leave final comments and approve the PR.

[jkbk2004]

@DusanJovic-NOAA we are ready to merge fv3atm PR.

[DeniseWorthen]

@jkbk2004 What did you mean when you told Brian "we need to copy over two BL files on jet"?

[jkbk2004]

@jkbk2004 What did you mean when you told Brian "we need to copy over two BL files on jet"?

I recreated two BL files on jet (control_ras rrfs_v1beta) and passed ok now.

[jkbk2004]

@jkbk2004 What did you mean when you told Brian "we need to copy over two BL files on jet"?

I recreated two BL files on jet (control_ras rrfs_v1beta) and passed ok now.

Everything ready to merge the fv3atm PR and update pointers.

[DeniseWorthen]

@jkbk2004 So the initial baselines for those two tests were not reproducible when we first ran the verification RT? Do we know why?

[jkbk2004]

/lfs4/HFIP/h-nems/emc.nemspara/autort/pr/963206649/20220624130021/ufs-weather-model

fv_core.res.tile4.nc ............ALT CHECK......NOT OK ?

[jkbk2004]

/lfs4/HFIP/h-nems/emc.nemspara/autort/pr/963206649/20220624130021/ufs-weather-model

fv_core.res.tile4.nc ............ALT CHECK......NOT OK ?

nccmp tells some of variables do not exist in those initial BL files. Somehow jet got trouble to correctly finish up writing out files.
Jet system issue.

[jkbk2004]

@DeniseWorthen can we merge in Dusan's fv3atm PR? we need one more approval there.

[DusanJovic-NOAA]

Ready for merge.