Atmosphere cycling with a Coupled model (NOAA-EMC#1274)

This PR enables:
- cycling the atmosphere with GSI using the coupled model (S2S) intended for WCDA development
- allows staging of cycled IC's for coupled/atm-only model (in warm start and cold start)
- updates dependencies in the XML to account for coupled model tasks (currently disable in cycled mode due to inadequacies in `ocnpost` jobs)

This PR also:
- disables `GLDAS` by default as it is expected to be deprecated.  Enthusiastic developers may turn it `ON`
- allows a tiny flexibility for DA developers to turn ON 3Dvar automatically if `nens = 0`.

Also in this PR:
- `diag_table_da` was updated to include instantaneous ocean fields.  When running the model in `APP=ATM`, these entries from the `diag_table_da` are ignored. 
- `diag_table` was also updated to include coupled fields from `diag_table_cpl`.  This is a step towards unifying the `diag_table`.  There will be more work done in this area in the near future.
- `MOM_input_template_500` is added for the 5 degree ocean configuration.

This PR was built on initial work from @guillaumevernieres and @NeilBarton-NOAA 
@guillaumevernieres provided initial conditions
@NeilBarton-NOAA and @guillaumevernieres both provided updates to the scripts that were used in the creation of this PR.
The following commands are used to setup and configure the cycled experiment with S2S model configuration.
@guillaumevernieres @NeilBarton-NOAA @DeniseWorthen and @junwang-noaa provided valuable assistance in debugging the coupled model failures.

To setup and run a coupled model, cycled with atmosphere test at C48 atmosphere 5 degree ocean/ice resolution:
```
# Setup experiment and COMROT directories.  Copy initial conditions to COMROT.
./setup_expt.py cycled --expdir <path_to_expdir> --comrot <path_to_comrot> --idate 2021032312 --edate 2021032400 --resdet 48 --nens 0 --gfs_cyc 0 --icsdir /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/ICSDIR/C48O500 --start warm --pslot <pslot> --app S2S

# cd into EXPDIR and disable IAU, METP in config.base

# Generate XML
./setup_xml.py <path_to_expdir>
```

The test was concluded successfully w/ no failures.
The test result can be viewed at ` /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/EXPDIR/prcycs2s` on Hera.

The test only performed a 3DVar with the GSI in the atmosphere and no IAU.  Including IAU will require additional work and flexibility both in the model code as well as in the workflow.

A test with C384 and 0.25 degree ocean/ice model was also configured by obtaining files from @NeilBarton-NOAA and @guillaumevernieres .  The forecast model crashed after about 3 hours of integration with an "Out of memory" fault.   Additional compute resources might be needed to get that configuration up and running.  

Controls for "restart", "restart interval", "history" and "history frequencies" are controlled in multiple places for multiple components.  A discussion with @junwang-noaa and @DeniseWorthen is ongoing to develop a solution for higher level control of clock related properties in the coupled model.

Adopting the current infrastructure built for forecast only prototypes for cycling was very enlightening.
Several enhancements could be made to make this system more flexible for the needs of both model development as well as DA development.  I have left verbose comments and an extensive array of `TODO` (to be transformed into GH issues), for mapping out future work.

.gitignore

@@ -7,6 +7,7 @@ __pycache__
 *.sw[a-p]
 ._*
 .DS_Store
+#nohup.out - some users do not want this to be a part of .gitignore.  TODO: review against best practices
 .idea/
 .vscode/
 

parm/config/config.base.emc.dyn

@@ -165,6 +165,23 @@ export DOBNDPNT_WAVE="NO"
 export cplwav2atm=".false."
 export FRAC_GRID=".true."
 
+# Set operational resolution
+export OPS_RES="C768" # Do not change
+
+# Resolution specific parameters
+export LEVS=128
+export CASE="@CASECTL@"
+export CASE_ENKF="@CASEENS@"
+case "$CASE" in
+    "C48") export OCNRES=500;;
+    "C96") export OCNRES=100;;
+    "C192") export OCNRES=050;;
+    "C384") export OCNRES=025;;
+    "C768") export OCNRES=025;;
+    *) export OCNRES=025;;
+esac
+export ICERES=$OCNRES
+
 case "${APP}" in
   ATM)
     export confignamevarfornems="atm"
@@ -212,23 +229,6 @@ case "${APP}" in
     ;;
 esac
 
-# Set operational resolution
-export OPS_RES="C768" # Do not change
-
-# Resolution specific parameters
-export LEVS=128
-export CASE="@CASECTL@"
-export CASE_ENKF="@CASEENS@"
-case "${CASE}" in
-    "C48") export OCNRES=500;;
-    "C96") export OCNRES=100;;
-    "C192") export OCNRES=050;;
-    "C384") export OCNRES=025;;
-    "C768") export OCNRES=025;;
-    *) export OCNRES=025;;
-esac
-export ICERES=${OCNRES}
-
 # Surface cycle update frequency
 if [[ "${CDUMP}" == "gdas" ]] ; then
    export FHCYC=1
@@ -289,7 +289,7 @@ export IAU_DELTHRS_ENKF=6
 export lobsdiag_forenkf=".true."
 
 # run GLDAS to spin up land ICs
-export DO_GLDAS="YES"
+export DO_GLDAS="NO"
 export gldas_cyc=00
 
 # Exception handling that when DO_GLDAS is set, the FHOUT must be 1
@@ -312,7 +312,7 @@ export DO_JEDIENS="NO"
 export DO_JEDIOCNVAR="NO"
 
 # Hybrid related
-export DOHYBVAR="YES"
+export DOHYBVAR="@DOHYBVAR@"
 export NMEM_ENKF=@NMEM_ENKF@
 export NMEM_EFCS=30
 export SMOOTH_ENKF="NO"
@@ -363,7 +363,7 @@ export MAKE_ACFTBUFR="@MAKE_ACFTBUFR@"
 # Analysis increments to zero in CALCINCEXEC
 export INCREMENTS_TO_ZERO="'liq_wat_inc','icmr_inc'"
 
-# Write analysis files for early cycle EnKF 
+# Write analysis files for early cycle EnKF
 export DO_CALC_INCREMENT_ENKF_GFS="YES"
 
 # Stratospheric increments to zero

parm/config/config.defaults.s2sw

@@ -3,7 +3,6 @@
 # Empty variables must include a space otherwise they will be overwritten
 
 # config.base
-# CASE=C384
 FHMAX_GFS_00=48
 FHMAX_GFS_06=48
 FHMAX_GFS_12=48
@@ -15,21 +14,27 @@ FHOUT_HF_GFS=-1
 min_seaice="1.0e-6"
 use_cice_alb=".true."
 
-# config.fv3
-DELTIM=300
-layout_x_gfs=8
-layout_y_gfs=8
-WRITE_GROUP_GFS=1
-WRTTASK_PER_GROUP_GFS=24
-#The settings below will result in S2SWA running 35 days under 8 hours wallclock on hera
-#layout_x_gfs=24
-#layout_y_gfs=16
-#WRTTASK_PER_GROUP_GFS=86
-WRTIOBUF="32M"
-MEDPETS=300
-
+# config.fv3  # TODO: This is hard-wired for P8 and needs to be refactored.  For now, use case C384
+case "${CASE}" in
+  "C384")
+    DELTIM=300
+    layout_x_gfs=8
+    layout_y_gfs=8
+    WRITE_GROUP_GFS=1
+    WRTTASK_PER_GROUP_GFS=24
+    #The settings below will result in S2SWA running 35 days under 8 hours wallclock on hera
+    #layout_x_gfs=24
+    #layout_y_gfs=16
+    #WRTTASK_PER_GROUP_GFS=86
+    WRTIOBUF="32M"  # TODO: This value is for P8 w/ C384.  Why not update/set this as default in config.fv3 C384?
+    MEDPETS=300  # TODO: P8 wants to use 300 instead of ATMPETS = layout_x * layout_y * 6
+    ;;
+esac
+
+# config.ice
+
+# TODO: These also are likely P8 hard-wired configurations.  Perhaps this file should be config.P8.defaults
 # config.wave
-
 waveGRD='gwes_30m'
 waveinterpGRD=' '
 waveuoutpGRD='gwes_30m'

parm/config/config.fcst

@@ -93,7 +93,7 @@ export h2o_phys=".true."
 # Options of stratosphere O3 physics reaction coefficients
 export new_o3forc="YES"
 
-export gwd_opt=2  
+export gwd_opt=2
 
 # --GFS.v16 uGWD.v0, used for suite FV3_GFS_v16 and UFS p6 etc
 # do_ugwp=T: use unified CGWD and OGWD, and turbulent orographic form drag (TOFD)
@@ -121,8 +121,8 @@ if [[ "$gwd_opt" -eq 2 ]]; then
     #export  do_gsl_drag_ss=".true."
     #export  do_gsl_drag_tofd=".true."
     #export  do_ugwp_v1_orog_only=".false."
-  
-    #--used for UFS p8 
+
+    #--used for UFS p8
     export knob_ugwp_version=0
     export do_ugwp=".false."
     export do_tofd=".false."
@@ -272,7 +272,7 @@ export FSICS="0"
 export ideflate=1
 export nbits=14
 export ishuffle=0
-# compression for RESTART files written by FMS 
+# compression for RESTART files written by FMS
 export shuffle=1
 export deflate_level=1
 
@@ -283,20 +283,16 @@ export USE_COUPLER_RES="NO"
 if [[ "$CDUMP" == "gdas" ]] ; then # GDAS cycle specific parameters
 
     # Variables used in DA cycling
-    if [[ "$QUILTING" = ".true." && "$OUTPUT_GRID" = "gaussian_grid" ]]; then
-        export DIAG_TABLE="$HOMEgfs/parm/parm_fv3diag/diag_table_da"
-    else
-        export DIAG_TABLE="$HOMEgfs/parm/parm_fv3diag/diag_table_da_orig"
-    fi
+    export DIAG_TABLE="$HOMEgfs/parm/parm_fv3diag/diag_table_da"
 
-    # Write restart files, where $number is current model start time. 
+    # Write restart files, where $number is current model start time.
     # restart_interval:        $number
-    #    number=0, writes out restart files at the end of forecast. 
+    #    number=0, writes out restart files at the end of forecast.
     #    number>0, writes out restart files at the frequency of $number and at the end of forecast.
     # restart_interval:        "$number -1"
     #    writes out restart files only once at $number forecast hour.
     # restart_interval:        "$number1 $number2 $number3 ..."
-    #    writes out restart file at the specified forecast hours 
+    #    writes out restart file at the specified forecast hours
     export restart_interval=${restart_interval:-6}
 
     # For IAU, write restarts at beginning of window also
@@ -313,11 +309,7 @@ if [[ "$CDUMP" == "gdas" ]] ; then # GDAS cycle specific parameters
 elif [[ "$CDUMP" == "gfs" ]] ; then # GFS cycle specific parameters
 
     # Write more variables to output
-    if [[ "$QUILTING" = ".true." && $OUTPUT_GRID = "gaussian_grid" ]]; then
-        export DIAG_TABLE="$HOMEgfs/parm/parm_fv3diag/diag_table"
-    else
-        export DIAG_TABLE="$HOMEgfs/parm/parm_fv3diag/diag_table_orig"
-    fi
+    export DIAG_TABLE="$HOMEgfs/parm/parm_fv3diag/diag_table"
 
     # Write gfs restart files to rerun fcst from any break point
     export restart_interval_gfs=${restart_interval_gfs:-0}
@@ -359,10 +351,6 @@ elif [[ "$CDUMP" == "gfs" ]] ; then # GFS cycle specific parameters
 
 fi
 
-if [[ "$DO_COUPLED" = "YES" ]] ; then # coupled model
-    export DIAG_TABLE="$HOMEgfs/parm/parm_fv3diag/diag_table_cpl"
-fi
-
 if [[ "$DO_AERO" = "YES" ]]; then # temporary settings for aerosol coupling
     export AERO_DIAG_TABLE="${AERO_DIAG_TABLE:-$HOMEgfs/parm/parm_fv3diag/diag_table.aero}"
     export AERO_FIELD_TABLE="${AERO_FIELD_TABLE:-$HOMEgfs/parm/parm_fv3diag/field_table.aero}"

parm/config/config.fv3

@@ -45,18 +45,18 @@ fi
 # (Standard) Model resolution dependent variables
 case ${case_in} in
     "C48")
-        export DELTIM=450
-        export layout_x=3
-        export layout_y=2
-        export layout_x_gfs=3
-        export layout_y_gfs=2
+        export DELTIM=1200
+        export layout_x=1
+        export layout_y=1
+        export layout_x_gfs=1
+        export layout_y_gfs=1
         export nth_fv3=1
         export nth_fv3_gfs=1
         export cdmbgwd="0.071,2.1,1.0,1.0"  # mountain blocking, ogwd, cgwd, cgwd src scaling
         export WRITE_GROUP=1
-        export WRTTASK_PER_GROUP=64
+        export WRTTASK_PER_GROUP=6
         export WRITE_GROUP_GFS=1
-        export WRTTASK_PER_GROUP_GFS=64
+        export WRTTASK_PER_GROUP_GFS=6
         export WRTIOBUF="1M"
         ;;
     "C96")

parm/config/config.ice

@@ -1,4 +1,17 @@
 #! /usr/bin/env bash
 
-export NX_GLB="1440"
-export NY_GLB="1080"
+case "${ICERES}" in
+  "025")
+    export NX_GLB="1440"
+    export NY_GLB="1080"
+    ;;
+  "500")
+    export NX_GLB="72"
+    export NY_GLB="35"
+    export cice_processor_shape="slenderX1"
+    ;;
+  *)
+    echo "FATAL ERROR: Unsupported ICERES = ${ICERES}, ABORT!"
+    exit 1
+    ;;
+esac

parm/config/config.ocn

@@ -1,11 +1,7 @@
 #! /usr/bin/env bash
 
 # OCNRES is currently being set in config.base
-# case "$CASE" in
-#     "C48") export OCNRES=500;;
-#     "C96") export OCNRES=100;;
-#     "C192") export OCNRES=050;;
-#     "C384") export OCNRES=025;;
-#     "C768") export OCNRES=025;;
-#     *) export OCNRES=025;;
-# esac
+ case "$CASE" in
+     "C48") export OCNTIM=3600;;
+     *) export OCNTIM=1800;;
+ esac

parm/config/config.resources

@@ -391,7 +391,11 @@ elif [ ${step} = "gldas" ]; then
 
 elif [ ${step} = "fcst" ]; then
 
-    export wtime_fcst="00:40:00"
+    if [ ${CASE} = "C48" ]; then
+        export wtime_fcst="00:30:00"
+    else
+        export wtime_fcst="00:40:00"
+    fi
     if [ ${CASE} = "C768" ]; then
       export wtime_fcst_gfs="06:00:00"
     elif [ ${CASE} = "C384" ]; then
@@ -458,6 +462,7 @@ elif [ ${step} = "fcst" ]; then
         if [[ ${DO_OCN} == "YES" ]]; then
             case ${OCNRES} in
                 # Except for 025, these are guesses for now
+                500) export OCNPETS=8 ;;
                 100) export OCNPETS=20 ;;
                 050) export OCNPETS=60 ;;
                 025) export OCNPETS=220 ;;
@@ -472,6 +477,7 @@ elif [ ${step} = "fcst" ]; then
         if [[ ${DO_ICE} == "YES" ]]; then
             case ${ICERES} in
                 # Except for 025, these are guesses for now
+                500) export ICEPETS=4 ;;
                 100) export ICEPETS=10 ;;
                 050) export ICEPETS=30 ;;
                 025) export ICEPETS=120 ;;