Add preamble, convert to bash, and remove env

This is the first in a wave of commits to improve and standardize the
scripts within global workflow. In this commit, all scripts run during
execution are converted to bash and a preamble is added to every script
that is not sourced by another script.

Every script executed during a forecast cycle is converted to bash. This
was mostly straightforward, though there were a couple Korne-shell
conventions (primarily using `typeset` to format strings) that had to be
replaced with bash-compatable alternatives like `printf`. This in turn
required a few modification to prevent zero-padded numbers from being
treated as octals (other may have been pre-existing bugs).

The preamble contains a number of feature to standardize code and
improve script flow and debugging.

	First, it uses two variables, `$STRICT` and `$TRACE` to control the
	behavior of `set`. When `$STRICT` is `"YES"`, error on undefined
	variables (`set -u`) and exit on non-zero return (`set -e`) are turned
	on. When `$TRACE` is `"YES"`, command trace (`set -x`) is turned on and
	a useful string is set to `$PS4` that gives the name and line number of
	the script. Both `$STRICT` and `$TRACE` default to `"YES"`. They also
	set up commands, `$ERR_EXIT_ON` and `$TRACE_ON`, that will restore the
	setting of each in the event a script needs to temporarily turn them
	off.

	Second, the preamble sets up primative timing of scripts using Posix
	`date`.

	Third, it echos the script is beginning and at what time.

	Finally, it also establishes a postamble function and sets it to run as
	a trap of EXIT. The postamble will use the end time to calculate the
	run time of the script, then print that the script has ended at what
	time, how long has elapsed, and the exit code. By setting this up as a
	trap instead of just calling it at the end of the script, it ensures
	the postamble is called even if the script exits early because there is
	an error.

	In response to this standardization, parts of scripts that performed
	these preamble functions (announcing start/end, `set -x`, etc) have
	been deleted. For some scripts where temporarily turning off `-x` or
	`-e` is needed, they now use `$ERR_EXIT_ON` and `$TRACE_ON` to return
	to the correct state afterwards, instead of blindly turning the setting
	back on.

	Additionally, some modifications were needed to comply with `set -eu`.
	Mostly taking care of undefined variables, but also a couple instances
	where a non-zero return code had to be dealt with.

If users wish to use their own preamble script instead, the default
script can be overriden by setting `$PREAMBLE_SCRIPT` before the run
begins.

Instance where scripts would print the full list of environment
variables have been removed. These can be spot added back in to debug as
necessary. Alternatively, a future PR will add them back in in a
standardized way.

`rstprod.sh` is added to the link list from gsi_monitor.fd, as it is
needed for the radmon scripts.

The placeholders for AWIPS and GEMPAK in the Hera and Orion environment
scripts were replaced with the correct definitions. There were also
other modifications to AWIPS and GEMPAK scripts to get it working for
development (AWIPS still isn't and will be fixed in the future).

GSI scripts that were brought in recently had all of their backticks
replaced with `$(  )` as was done with all other script previously.

Refs: NOAA-EMC#397

.gitignore

@@ -188,6 +188,7 @@ ush/radmon_verf_angle.sh
 ush/radmon_verf_bcoef.sh
 ush/radmon_verf_bcor.sh
 ush/radmon_verf_time.sh
+ush/rstprod.sh
 ush/wafs_blending.sh
 ush/wafs_grib2.regrid.sh
 ush/wafs_intdsk.sh

env/HERA.env

@@ -1,4 +1,4 @@
-#!/bin/ksh -x
+#! /usr/bin/env bash
 
 if [ $# -ne 1 ]; then
 
@@ -236,9 +236,17 @@ elif [ $step = "postsnd" ]; then
 
 elif [ $step = "awips" ]; then
 
-     echo "WARNING: $step is not enabled on $machine!"
+    nth_max=$(($npe_node_max / $npe_node_awips))
+
+    export NTHREADS_AWIPS=${nth_awips:-2}
+    [[ $NTHREADS_AWIPS -gt $nth_max ]] && export NTHREADS_AWIPS=$nth_max
+    export APRUN_AWIPSCFP="$launcher -n $npe_awips --multi-prog"
 
 elif [ $step = "gempak" ]; then
 
-     echo "WARNING: $step is not enabled on $machine!"
+    nth_max=$(($npe_node_max / $npe_node_gempak))
+
+    export NTHREADS_GEMPAK=${nth_gempak:-1}
+    [[ $NTHREADS_GEMPAK -gt $nth_max ]] && export NTHREADS_GEMPAK=$nth_max
+    export APRUN="$launcher -n $npe_gempak --multi-prog"
 fi

env/ORION.env

@@ -1,4 +1,4 @@
-#!/bin/ksh -x
+#! /usr/bin/env bash
 
 if [ $# -ne 1 ]; then
 
@@ -234,9 +234,17 @@ elif [ $step = "postsnd" ]; then
 
 elif [ $step = "awips" ]; then
 
-     echo "WARNING: $step is not enabled on $machine!"
+    nth_max=$(($npe_node_max / $npe_node_awips))
+
+    export NTHREADS_AWIPS=${nth_awips:-2}
+    [[ $NTHREADS_AWIPS -gt $nth_max ]] && export NTHREADS_AWIPS=$nth_max
+    export APRUN_AWIPSCFP="$launcher -n $npe_awips --multi-prog"
 
 elif [ $step = "gempak" ]; then
 
-     echo "WARNING: $step is not enabled on $machine!"
+    nth_max=$(($npe_node_max / $npe_node_gempak))
+
+    export NTHREADS_GEMPAK=${nth_gempak:-1}
+    [[ $NTHREADS_GEMPAK -gt $nth_max ]] && export NTHREADS_GEMPAK=$nth_max
+    export APRUN="$launcher -n $npe_gempak --multi-prog"
 fi

jobs/JGDAS_ATMOS_ANALYSIS_DIAG

@@ -1,10 +1,11 @@
-#!/bin/bash
-set -x
+#! /usr/bin/env bash
 
-export RUN_ENVIR=${RUN_ENVIR:-"nco"}
-export PS4='$SECONDS + '
-date
+PREAMBLE_SCRIPT="${PREAMBLE_SCRIPT:-$HOMEgfs/ush/preamble.sh}"
+if [ -f "${PREAMBLE_SCRIPT}" ]; then
+  source $PREAMBLE_SCRIPT
+fi
 
+export RUN_ENVIR=${RUN_ENVIR:-"nco"}
 
 #############################
 # Source relevant config files
@@ -112,11 +113,6 @@ fi
 
 ###############################################################
 # Run relevant script
-env
-echo "HAS BEGUN on $(hostname)"
-$LOGSCRIPT
-
-
 ${ANALDIAGSH:-$SCRgfs/exglobal_diag.sh}
 status=$?
 [[ $status -ne 0 ]] && exit $status
@@ -133,15 +129,11 @@ if [ -e "$pgmout" ] ; then
   cat $pgmout
 fi
 
-
-echo "ENDED NORMALLY."
-
-
 ##########################################
 # Remove the Temporary working directory
 ##########################################
 cd $DATAROOT
 [[ $KEEPDATA = "NO" ]] && rm -rf $DATA
 
-date
+
 exit 0

jobs/JGDAS_ATMOS_CHGRES_FORENKF

@@ -1,10 +1,11 @@
-#!/bin/bash
-set -x
+#! /usr/bin/env bash
 
-export RUN_ENVIR=${RUN_ENVIR:-"nco"}
-export PS4='$SECONDS + '
-date
+PREAMBLE_SCRIPT="${PREAMBLE_SCRIPT:-$HOMEgfs/ush/preamble.sh}"
+if [ -f "${PREAMBLE_SCRIPT}" ]; then
+  source $PREAMBLE_SCRIPT
+fi
 
+export RUN_ENVIR=${RUN_ENVIR:-"nco"}
 
 #############################
 # Source relevant config files
@@ -97,11 +98,6 @@ export COMIN_GES_ENS="$ROTDIR/enkfgdas.$gPDY/$gcyc/$COMPONENT"
 
 ###############################################################
 # Run relevant script
-env
-echo "HAS BEGUN on $(hostname)"
-$LOGSCRIPT
-
-
 ${CHGRESFCSTSH:-$SCRgfs/exgdas_atmos_chgres_forenkf.sh}
 status=$?
 [[ $status -ne 0 ]] && exit $status
@@ -118,15 +114,11 @@ if [ -e "$pgmout" ] ; then
   cat $pgmout
 fi
 
-
-echo "ENDED NORMALLY."
-
-
 ##########################################
 # Remove the Temporary working directory
 ##########################################
 cd $DATAROOT
 [[ $KEEPDATA = "NO" ]] && rm -rf $DATA
 
-date
+
 exit 0

jobs/JGDAS_ATMOS_GEMPAK

@@ -1,8 +1,9 @@
-#!/bin/sh
+#! /usr/bin/env bash
 
-set -xa
-export PS4='$SECONDS + '
-date
+PREAMBLE_SCRIPT="${PREAMBLE_SCRIPT:-$HOMEgfs/ush/preamble.sh}"
+if [ -f "${PREAMBLE_SCRIPT}" ]; then
+  source $PREAMBLE_SCRIPT
+fi
 
 ############################################
 # GDAS GEMPAK PRODUCT GENERATION
@@ -73,7 +74,6 @@ fi
 
 export pgmout=OUTPUT.$$
 
-env
 
 if [ -f $DATA/poescrip ]; then
    rm $DATA/poescript
@@ -107,11 +107,6 @@ APRUNCFP=$(eval echo $APRUN_GEMPAKCFP)
 $APRUNCFP $DATA/poescript
 export err=$?; err_chk
 
-########################################################
-
-msg="JOB $job HAS COMPLETED NORMALLY!"
-postmsg $jlogfile "$msg"
-
 ############################################
 # print exec I/O output
 ############################################
@@ -126,4 +121,3 @@ if [ "$KEEPDATA" != "YES" ] ; then
   rm -rf $DATA
 fi
 
-date

jobs/JGDAS_ATMOS_GEMPAK_META_NCDC

@@ -1,8 +1,9 @@
-#!/bin/sh
+#! /usr/bin/env bash
 
-set -xa
-export PS4='$SECONDS + '
-date
+PREAMBLE_SCRIPT="${PREAMBLE_SCRIPT:-$HOMEgfs/ush/preamble.sh}"
+if [ -f "${PREAMBLE_SCRIPT}" ]; then
+  source $PREAMBLE_SCRIPT
+fi
 
 ############################################
 # GDAS GEMPAK META NCDC PRODUCT GENERATION
@@ -86,7 +87,6 @@ fi
 
 export pgmout=OUTPUT.$$
 
-env
 
 ########################################################
 # Execute the script.
@@ -107,9 +107,6 @@ $SRCgfs/exgdas_atmos_gempak_gif_ncdc.sh
 export err=$?; err_chk
 ########################################################
 
-msg="JOB $job HAS COMPLETED NORMALLY!"
-postmsg $jlogfile "$msg"
-
 ############################################
 # print exec I/O output
 ############################################
@@ -124,4 +121,3 @@ if [ "$KEEPDATA" != "YES" ] ; then
   rm -rf $DATA
 fi
 
-date

jobs/JGDAS_EFSOI

@@ -1,10 +1,11 @@
-#!/bin/ksh
-set -x
+#! /usr/bin/env bash
 
-export RUN_ENVIR=${RUN_ENVIR:-"nco"}
-export PS4='$SECONDS + '
-date
+PREAMBLE_SCRIPT="${PREAMBLE_SCRIPT:-$HOMEgfs/ush/preamble.sh}"
+if [ -f "${PREAMBLE_SCRIPT}" ]; then
+  source $PREAMBLE_SCRIPT
+fi
 
+export RUN_ENVIR=${RUN_ENVIR:-"nco"}
 
 #############################
 # Source relevant config files
@@ -97,10 +98,6 @@ mkdir -p $OSENSE_SAVE_DIR
 
 ###############################################################
 # Run relevant exglobal script
-env
-msg="HAS BEGUN on `hostname`"
-postmsg "$jlogfile" "$msg"
-$LOGSCRIPT
 
 ${EFSOIUPDSH:-$SCRgfs/exgdas_efsoi.sh}
 status=$?
@@ -117,17 +114,12 @@ if [ -e "$pgmout" ] ; then
   cat $pgmout
 fi
 
-
-msg="ENDED NORMALLY."
-postmsg "$jlogfile" "$msg"
-
-
 ##########################################
 # Remove the Temporary working directory
 ##########################################
 cd $DATAROOT
 
 [[ $KEEPDATA = "NO" ]] && rm -rf $DATA
 
-date
+
 exit 0

jobs/JGDAS_EFSOI_ECEN

@@ -1,10 +1,11 @@
-#!/bin/ksh
-set -x
+#! /usr/bin/env bash
 
-export RUN_ENVIR=${RUN_ENVIR:-"nco"}
-export PS4='$SECONDS + '
-date
+PREAMBLE_SCRIPT="${PREAMBLE_SCRIPT:-$HOMEgfs/ush/preamble.sh}"
+if [ -f "${PREAMBLE_SCRIPT}" ]; then
+  source $PREAMBLE_SCRIPT
+fi
 
+export RUN_ENVIR=${RUN_ENVIR:-"nco"}
 
 #############################
 # Source relevant config files
@@ -106,11 +107,6 @@ export COMIN_GES_ENS="$ROTDIR/enkf$CDUMP.$gPDY/$gcyc/$COMPONENT"
 
 ###############################################################
 # Run relevant script
-env
-msg="HAS BEGUN on `hostname`"
-postmsg "$jlogfile" "$msg"
-$LOGSCRIPT
-
 
 ${ENKFRECENSH:-$SCRgfs/exgdas_enkf_ecen.sh}
 status=$?
@@ -129,15 +125,11 @@ if [ -e "$pgmout" ] ; then
 fi
 
 
-msg="ENDED NORMALLY."
-postmsg "$jlogfile" "$msg"
-
-
 ##########################################
 # Remove the Temporary working directory
 ##########################################
 cd $DATAROOT
 [[ $KEEPDATA = "NO" ]] && rm -rf $DATA
 
-date
+
 exit 0

jobs/JGDAS_EFSOI_FCST

@@ -1,10 +1,11 @@
-#!/bin/ksh
-set -x
+#! /usr/bin/env bash
 
-export RUN_ENVIR=${RUN_ENVIR:-"nco"}
-export PS4='$SECONDS + '
-date
+PREAMBLE_SCRIPT="${PREAMBLE_SCRIPT:-$HOMEgfs/ush/preamble.sh}"
+if [ -f "${PREAMBLE_SCRIPT}" ]; then
+  source $PREAMBLE_SCRIPT
+fi
 
+export RUN_ENVIR=${RUN_ENVIR:-"nco"}
 
 #############################
 # Source relevant config files
@@ -93,11 +94,6 @@ export ENSBEG=$((ENSEND - NMEM_EFCSGRP + 1))
 
 ###############################################################
 # Run relevant script
-env
-msg="HAS BEGUN on `hostname`"
-postmsg "$jlogfile" "$msg"
-$LOGSCRIPT
-
 
 ${ENKFFCSTSH:-$SCRgfs/exgdas_enkf_fcst.sh}
 status=$?
@@ -129,16 +125,11 @@ if [ -e "$pgmout" ] ; then
   cat $pgmout
 fi
 
-
-msg="ENDED NORMALLY."
-postmsg "$jlogfile" "$msg"
-
-
 ##########################################
 # Remove the Temporary working directory
 ##########################################
 cd $DATAROOT
 [[ $KEEPDATA = "NO" ]] && rm -rf $DATA
 
-date
+
 exit 0