Port the global workflow to Jet (#1301)

Provides initial cycled and free-forecast support for the global workflow on Jet.  References #357.  Not included in this port is support for GLDAS, verif-global, and TC_tracker.

Jet will eventually support the following, with the currently tested options bolded, on xjet and kjet partitions
- uncoupled, cycled and free-forecast experiments at C48, C96, C192, C384, and C768 resolutions
- coupled, free-forecast experiments (ATMA, ATMW, S2S, and S2SW) at C384
~Note that currently coupled ICs are not available on Jet, so only ATM-only experiments can performed.~

Fixes #357

docs/note_fixfield.txt

@@ -4,6 +4,8 @@ They are saved locally on all platforms
 
 Hera: /scratch1/NCEPDEV/global/glopara/fix
 Orion: /work/noaa/global/glopara/fix
+Jet: /mnt/lfs4/HFIP/hfv3gfs/glopara/git/fv3gfs/fix
+S4: /data/prod/glopara/fix
 
 ------------------------------------------------------------------------------
 09/28/2018

docs/source/components.rst

@@ -60,6 +60,7 @@ Observation data, also known as dump data, is prepared in production and then ar
 
 * Hera: /scratch1/NCEPDEV/global/glopara/dump
 * Orion: /work/noaa/rstprod/dump
+* Jet: /mnt/lfs4/HFIP/hfv3gfs/glopara/dump
 * WCOSS2: /lfs/h2/emc/global/noscrub/emc.global/dump
 * S4: /data/prod/glopara/dump
 

docs/source/hpc.rst

@@ -22,6 +22,7 @@ HPC helpdesks
 * HPSS: rdhpcs.hpss.help@noaa.gov
 * Gaea: oar.gfdl.help@noaa.gov
 * S4: david.huber@noaa.gov
+* Jet: rdhpcs.jet.help@noaa.gov
 
 ======================
 Restricted data access
@@ -76,6 +77,8 @@ It is advised to use Git v2+ when available. At the time of writing this documen
 +---------+----------+---------------------------------------+
 | Orion   | v1.8.3.1 | **module load git/2.28.0**            |
 +---------+----------+---------------------------------------+
+| Jet     | v2.18.0  | default                               |
++---------+----------+---------------------------------------+
 | WCOSS2  | v2.26.2  | default or **module load git/2.29.0** |
 +---------+----------+---------------------------------------+
 | S4      | v1.8.3.1 | **module load git/2.30.0**            |
@@ -96,9 +99,9 @@ For the manage_externals utility functioning::
    Error: fatal: ssh variant 'simple' does not support setting port
    Fix: git config --global ssh.variant ssh
 
-===================================
-Stacksize on R&Ds (Hera, Orion, S4)
-===================================
+========================================
+Stacksize on R&Ds (Hera, Orion, Jet, S4)
+========================================
 
 Some GFS components, like the UPP, need an unlimited stacksize. Add the following setting into your appropriate .*rc file to support these components:
 

docs/source/init.rst

@@ -190,6 +190,7 @@ Forecast-only P8 prototype initial conditions are made available to users on sup
     WCOSS2: /lfs/h2/emc/global/noscrub/emc.global/IC/COUPLED
     HERA: /scratch1/NCEPDEV/climate/role.ufscpara/IC
     ORION: /work/noaa/global/glopara/data/ICSDIR/prototype_ICs
+    JET: /mnt/lfs4/HFIP/hfv3gfs/glopara/data/ICSDIR/prototype_ICs
     S4: /data/prod/glopara/coupled_ICs
 
 These locations are known within the workflow via paths set in ``parm/config/config.coupled_ic``.
@@ -217,7 +218,7 @@ Not yet supported. See :ref:`Manual Generation<manual-generation>` section below
 ---------------------
 Forecast-only coupled
 ---------------------
-Coupled initial conditions are currently only generated offline and copied prior to the forecast run. Prototype initial conditions will automatically be used when setting up an experiment as an S2SW app, there is no need to do anything additional. Copies of initial conditions from the prototype runs are currently maintained on Hera, Orion, and WCOSS2. The locations used are determined by ``parm/config/config.coupled_ic``. If you need prototype ICs on another machine, please contact Walter (Walter.Kolczynski@noaa.gov).
+Coupled initial conditions are currently only generated offline and copied prior to the forecast run. Prototype initial conditions will automatically be used when setting up an experiment as an S2SW app, there is no need to do anything additional. Copies of initial conditions from the prototype runs are currently maintained on Hera, Orion, Jet, and WCOSS2. The locations used are determined by ``parm/config/config.coupled_ic``. If you need prototype ICs on another machine, please contact Walter (Walter.Kolczynski@noaa.gov).
 
 .. _forecastonly-atmonly:
 

docs/source/setup.rst

@@ -28,6 +28,13 @@ Experiment Setup
 
            module load miniconda/3.8-s4
 
+   * - Jet
+     - ::
+
+           module use /mnt/lfs4/HFIP/hfv3gfs/role.epic/miniconda3/modulefiles
+           module load miniconda3/4.12.0
+           conda activate ufswm
+
 If running with Rocoto make sure to have a Rocoto module loaded before running setup scripts:
 
 .. list-table:: ROCOTO Module Load Commands
@@ -54,6 +61,10 @@ If running with Rocoto make sure to have a Rocoto module loaded before running s
      - ::
 
            module load rocoto/1.3.4
+   * - Jet
+     - ::
+
+           module load rocoto/1.3.3
 
 ^^^^^^^^^^^^^^^^^^^^^^^^
 Forecast-only experiment

env/JET.env

@@ -18,11 +18,13 @@ if [[ "${PARTITION_BATCH}" = "xjet" ]]; then
   export npe_node_max=24
 elif [[ "${PARTITION_BATCH}" = "vjet" ]]; then
   export npe_node_max=16
+elif [[ "${PARTITION_BATCH}" = "kjet" ]]; then
+  export npe_node_max=40
 fi
-export launcher="srun -l --export=ALL"
+export launcher="srun -l --epilog=/apps/local/bin/report-mem --export=ALL"
 export mpmd_opt="--multi-prog --output=${step}.%J.%t.out"
 
-# Configure STACK
+# Configure MPI environment
 export OMP_STACKSIZE=2048000
 export NTHSTACK=1024000000
 
@@ -34,46 +36,85 @@ if [[ "${step}" = "prep" ]] || [[ "${step}" = "prepbufr" ]]; then
     nth_max=$((npe_node_max / npe_node_prep))
 
     export POE="NO"
-    export BACK=${BACK:-"NO"}
+    export BACK="NO"
     export sys_tp="JET"
+    export launcher_PREP="srun"
 
 elif [[ "${step}" = "waveinit" ]] || [[ "${step}" = "waveprep" ]] || [[ "${step}" = "wavepostsbs" ]] || [[ "${step}" = "wavepostbndpnt" ]] || [[ "${step}" = "wavepostbndpntbll" ]] || [[ "${step}" = "wavepostpnt" ]]; then
 
+    export CFP_MP="YES"
     if [[ "${step}" = "waveprep" ]]; then export MP_PULSE=0 ; fi
     export wavempexec=${launcher}
     export wave_mpmd=${mpmd_opt}
 
 elif [[ "${step}" = "atmanalrun" ]]; then
 
+    export CFP_MP=${CFP_MP:-"YES"}
+    export USE_CFP=${USE_CFP:-"YES"}
+    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"
+
     nth_max=$((npe_node_max / npe_node_atmanalrun))
 
     export NTHREADS_ATMANAL=${nth_atmanalrun:-${nth_max}}
     [[ ${NTHREADS_ATMANAL} -gt ${nth_max} ]] && export NTHREADS_ATMANAL=${nth_max}
-    export APRUN_ATMANAL="${launcher} ${npe_atmanalrun}"
+    export APRUN_ATMANAL="${launcher} -n ${npe_atmanalrun}"
 
 elif [[ "${step}" = "atmensanalrun" ]]; then
 
+    export CFP_MP=${CFP_MP:-"YES"}
+    export USE_CFP=${USE_CFP:-"YES"}
+    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"
+
     nth_max=$((npe_node_max / npe_node_atmensanalrun))
 
     export NTHREADS_ATMENSANAL=${nth_atmensanalrun:-${nth_max}}
     [[ ${NTHREADS_ATMENSANAL} -gt ${nth_max} ]] && export NTHREADS_ATMENSANAL=${nth_max}
-    export APRUN_ATMENSANAL="${launcher} ${npe_atmensanalrun}"
+    export APRUN_ATMENSANAL="${launcher} -n ${npe_atmensanalrun}"
 
 elif [[ "${step}" = "aeroanlrun" ]]; then
 
+    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"
+
     nth_max=$((npe_node_max / npe_node_aeroanlrun))
 
     export NTHREADS_AEROANL=${nth_aeroanlrun:-${nth_max}}
     [[ ${NTHREADS_AEROANL} -gt ${nth_max} ]] && export NTHREADS_AEROANL=${nth_max}
-    export APRUN_AEROANL="${launcher} ${npe_aeroanlrun}"
+    export APRUN_AEROANL="${launcher} -n ${npe_aeroanlrun}"
+
+elif [[ "${step}" = "ocnanalbmat" ]]; then
+
+    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"
+
+    nth_max=$((npe_node_max / npe_node_ocnanalbmat))
+
+    export NTHREADS_OCNANAL=${nth_ocnanalbmat:-${nth_max}}
+    [[ ${NTHREADS_OCNANAL} -gt ${nth_max} ]] && export NTHREADS_OCNANAL=${nth_max}
+    export APRUN_OCNANAL="${launcher} -n ${npe_ocnanalbmat}"
+
+elif [[ "${step}" = "ocnanalrun" ]]; then
+
+    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"
+
+    nth_max=$((npe_node_max / npe_node_ocnanalrun))
+
+    export NTHREADS_OCNANAL=${nth_ocnanalrun:-${nth_max}}
+    [[ ${NTHREADS_OCNANAL} -gt ${nth_max} ]] && export NTHREADS_OCNANAL=${nth_max}
+    export APRUN_OCNANAL="${launcher} -n ${npe_ocnanalrun}"
 
-elif [[ "${step}" = "anal" ]]; then
+elif [[ "${step}" = "anal" ]] || [[ "${step}" = "analcalc" ]]; then
+
+    export MKL_NUM_THREADS=4
+    export MKL_CBWR=AUTO
+
+    export CFP_MP=${CFP_MP:-"YES"}
+    export USE_CFP=${USE_CFP:-"YES"}
+    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"
 
     nth_max=$((npe_node_max / npe_node_anal))
 
-    export NTHREADS_GSI=${nth_gsi:-${nth_max}}
+    export NTHREADS_GSI=${nth_anal:-${nth_max}}
     [[ ${NTHREADS_GSI} -gt ${nth_max} ]] && export NTHREADS_GSI=${nth_max}
-    export APRUN_GSI="${launcher} ${npe_gsi:-${npe_anal:-${PBS_NP}}}"
+    export APRUN_GSI="${launcher} -n ${npe_gsi:-${npe_anal}}"
 
     export NTHREADS_CALCINC=${nth_calcinc:-1}
     [[ ${NTHREADS_CALCINC} -gt ${nth_max} ]] && export NTHREADS_CALCINC=${nth_max}
@@ -82,47 +123,50 @@ elif [[ "${step}" = "anal" ]]; then
     export NTHREADS_CYCLE=${nth_cycle:-12}
     [[ ${NTHREADS_CYCLE} -gt ${npe_node_max} ]] && export NTHREADS_CYCLE=${npe_node_max}
     npe_cycle=${ntiles:-6}
-    export APRUN_CYCLE="${launcher} ${npe_cycle}"
+    export APRUN_CYCLE="${launcher} -n ${npe_cycle}"
 
     export NTHREADS_GAUSFCANL=1
     npe_gausfcanl=${npe_gausfcanl:-1}
-    export APRUN_GAUSFCANL="${launcher} ${npe_gausfcanl}"
+    export APRUN_GAUSFCANL="${launcher} -n ${npe_gausfcanl}"
 
 elif [[ "${step}" = "sfcanl" ]]; then
     nth_max=$((npe_node_max / npe_node_sfcanl))
 
     export NTHREADS_CYCLE=${nth_sfcanl:-14}
     [[ ${NTHREADS_CYCLE} -gt ${npe_node_max} ]] && export NTHREADS_CYCLE=${npe_node_max}
     npe_sfcanl=${ntiles:-6}
-    export APRUN_CYCLE="${launcher} ${npe_sfcanl}"
+    export APRUN_CYCLE="${launcher} -n ${npe_sfcanl}"
 
 elif [[ "${step}" = "gldas" ]]; then
 
-    nth_max=$((npe_node_max / npe_node_gldas))
-
-    export NTHREADS_GLDAS=${nth_gldas:-${nth_max}}
-    [[ ${NTHREADS_GLDAS} -gt ${nth_max} ]] && export NTHREADS_GLDAS=${nth_max}
-    export APRUN_GLDAS="${launcher} ${npe_gldas}"
-
-    export NTHREADS_GAUSSIAN=${nth_gaussian:-1}
-    [[ ${NTHREADS_GAUSSIAN} -gt ${nth_max} ]] && export NTHREADS_GAUSSIAN=${nth_max}
-    export APRUN_GAUSSIAN="${launcher} ${npe_gaussian}"
+    echo "WARNING: ${step} is not enabled on ${machine}!"
 
 elif [[ "${step}" = "eobs" ]]; then
 
+    export MKL_NUM_THREADS=4
+    export MKL_CBWR=AUTO
+
     nth_max=$((npe_node_max / npe_node_eobs))
 
-    export NTHREADS_GSI=${nth_gsi:-${nth_max}}
+    export NTHREADS_GSI=${nth_eobs:-${nth_max}}
     [[ ${NTHREADS_GSI} -gt ${nth_max} ]] && export NTHREADS_GSI=${nth_max}
-    export APRUN_GSI="${launcher} ${npe_gsi:-${npe_eobs:-${PBS_NP}}}"
+    export APRUN_GSI="${launcher} -n ${npe_gsi:-${npe_eobs}}"
+
+    export CFP_MP=${CFP_MP:-"YES"}
+    export USE_CFP=${USE_CFP:-"YES"}
+    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"
 
 elif [[ "${step}" = "eupd" ]]; then
 
     nth_max=$((npe_node_max / npe_node_eupd))
 
-    export NTHREADS_ENKF=${nth_enkf:-${nth_max}}
+    export NTHREADS_ENKF=${nth_eupd:-${nth_max}}
     [[ ${NTHREADS_ENKF} -gt ${nth_max} ]] && export NTHREADS_ENKF=${nth_max}
-    export APRUN_ENKF="${launcher} ${npe_enkf:-${npe_eupd:-${PBS_NP}}}"
+    export APRUN_ENKF="${launcher} -n ${npe_enkf:-${npe_eupd}}"
+
+    export CFP_MP=${CFP_MP:-"YES"}
+    export USE_CFP=${USE_CFP:-"YES"}
+    export APRUNCFP="${launcher} -n \$ncmd ${mpmd_opt}"
 
 elif [[ "${step}" = "fcst" ]] || [[ "${step}" = "efcs" ]]; then
 
@@ -145,64 +189,65 @@ elif [[ "${step}" = "post" ]]; then
 
     export NTHREADS_NP=${nth_np:-1}
     [[ ${NTHREADS_NP} -gt ${nth_max} ]] && export NTHREADS_NP=${nth_max}
-    export APRUN_NP="${launcher}"
+    export APRUN_NP="${launcher} -n ${npe_post}"
 
     export NTHREADS_DWN=${nth_dwn:-1}
     [[ ${NTHREADS_DWN} -gt ${nth_max} ]] && export NTHREADS_DWN=${nth_max}
-    export APRUN_NP="${launcher}"
+    export APRUN_DWN="${launcher} -n ${npe_dwn}"
 
 elif [[ "${step}" = "ecen" ]]; then
 
     nth_max=$((npe_node_max / npe_node_ecen))
 
     export NTHREADS_ECEN=${nth_ecen:-${nth_max}}
     [[ ${NTHREADS_ECEN} -gt ${nth_max} ]] && export NTHREADS_ECEN=${nth_max}
-    export APRUN_ECEN="${launcher} ${npe_ecen:-${PBS_NP}}"
+    export APRUN_ECEN="${launcher} -n ${npe_ecen}"
 
     export NTHREADS_CHGRES=${nth_chgres:-12}
     [[ ${NTHREADS_CHGRES} -gt ${npe_node_max} ]] && export NTHREADS_CHGRES=${npe_node_max}
     export APRUN_CHGRES="time"
 
     export NTHREADS_CALCINC=${nth_calcinc:-1}
     [[ ${NTHREADS_CALCINC} -gt ${nth_max} ]] && export NTHREADS_CALCINC=${nth_max}
-    export APRUN_CALCINC="${launcher} ${npe_ecen:-${PBS_NP}}"
+    export APRUN_CALCINC="${launcher} -n ${npe_ecen}"
 
 elif [[ "${step}" = "esfc" ]]; then
 
     nth_max=$((npe_node_max / npe_node_esfc))
 
     export NTHREADS_ESFC=${nth_esfc:-${nth_max}}
     [[ ${NTHREADS_ESFC} -gt ${nth_max} ]] && export NTHREADS_ESFC=${nth_max}
-    export APRUN_ESFC="${launcher} ${npe_esfc:-${PBS_NP}}"
+    export APRUN_ESFC="${launcher} -n ${npe_esfc}"
 
-    export NTHREADS_CYCLE=${nth_cycle:-12}
+    export NTHREADS_CYCLE=${nth_cycle:-14}
     [[ ${NTHREADS_CYCLE} -gt ${npe_node_max} ]] && export NTHREADS_CYCLE=${npe_node_max}
-    export APRUN_CYCLE="${launcher} ${npe_esfc}"
-
+    export APRUN_CYCLE="${launcher} -n ${npe_esfc}"
 
 elif [[ "${step}" = "epos" ]]; then
 
     nth_max=$((npe_node_max / npe_node_epos))
 
     export NTHREADS_EPOS=${nth_epos:-${nth_max}}
     [[ ${NTHREADS_EPOS} -gt ${nth_max} ]] && export NTHREADS_EPOS=${nth_max}
-    export APRUN_EPOS="${launcher} ${npe_epos:-${PBS_NP}}"
+    export APRUN_EPOS="${launcher} -n ${npe_epos}"
 
 elif [[ "${step}" = "init" ]]; then
 
-    export APRUN="${launcher}"
+    export APRUN="${launcher} -n ${npe_init}"
 
 elif [[ "${step}" = "postsnd" ]]; then
 
+    export CFP_MP="YES"
+
     nth_max=$((npe_node_max / npe_node_postsnd))
 
     export NTHREADS_POSTSND=${nth_postsnd:-1}
     [[ ${NTHREADS_POSTSND} -gt ${nth_max} ]] && export NTHREADS_POSTSND=${nth_max}
-    export APRUN_POSTSND="${launcher} ${npe_postsnd}"
+    export APRUN_POSTSND="${launcher} -n ${npe_postsnd}"
 
     export NTHREADS_POSTSNDCFP=${nth_postsndcfp:-1}
     [[ ${NTHREADS_POSTSNDCFP} -gt ${nth_max} ]] && export NTHREADS_POSTSNDCFP=${nth_max}
-    export APRUN_POSTSNDCFP="${launcher} ${npe_postsndcfp}"
+    export APRUN_POSTSNDCFP="${launcher} -n ${npe_postsndcfp} ${mpmd_opt}"
 
 elif [[ "${step}" = "awips" ]]; then