Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GDASapp (first wave of JEDI changes) #871

Merged
merged 67 commits into from
Jul 22, 2022
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
b3e8156
Issue #529: add ufsda to checkout, build, and link
RussTreadon-NOAA Dec 15, 2021
8ccdd24
Issue #529: merge branch develop at 033b1d8 into feature/ufsda
RussTreadon-NOAA Dec 20, 2021
8663666
atmanal rocoto job scripts
CoryMartin-NOAA Dec 20, 2021
b583b44
Start of env and parm/config changes for atmanal
CoryMartin-NOAA Dec 21, 2021
40eeda1
Changes to parm/config files
CoryMartin-NOAA Dec 22, 2021
6948c0f
Make rocoto jobs all use same j-job for atmanal tasks
CoryMartin-NOAA Dec 22, 2021
9caf5cc
start of atmanal config and added to resources
CoryMartin-NOAA Dec 22, 2021
5994545
Rename static B template YAML
CoryMartin-NOAA Dec 22, 2021
b706b8a
Add APRUN_ATMANAL to ORION env
CoryMartin-NOAA Dec 28, 2021
2b54e97
Additional config changes for atmanal
CoryMartin-NOAA Dec 29, 2021
9cca1e4
Merge branch 'feature/ufsda-rocoto-jobs' into feature/ufsda-parm
CoryMartin-NOAA Dec 29, 2021
a31b44b
typo/bugfixes
CoryMartin-NOAA Dec 29, 2021
a0a0cbd
Merge branch 'feature/ufsda-rocoto-jobs' into feature/ufsda-parm
CoryMartin-NOAA Dec 29, 2021
aeb1765
reduce wallclock for atmanalpost to 1 hour
CoryMartin-NOAA Dec 29, 2021
da78b18
change r2d2 vars
CoryMartin-NOAA Jan 3, 2022
6078e73
make r2d2 exp oper_gdas
CoryMartin-NOAA Jan 3, 2022
bb86101
Fix for orion env file and add link to j-job
CoryMartin-NOAA Jan 3, 2022
9594e5a
Change rocoto job variables
CoryMartin-NOAA Jan 4, 2022
7836e13
Typo in config.atmanal
CoryMartin-NOAA Jan 4, 2022
94acdde
config.resources bug
CoryMartin-NOAA Jan 4, 2022
cc62d14
Add link to ufsda directory
RussTreadon-NOAA Jan 4, 2022
e3e30cf
Merge branch 'develop' into feature/ufsda
CoryMartin-NOAA Feb 7, 2022
2b4077c
Issue #521: merge branch 'develop' at 97ebc4d into feature/ufsda-roc…
RussTreadon-NOAA Feb 7, 2022
4c47f6c
Issue #532: merge branch 'develop' at 97ebc4d into feature/ufsda-parm
RussTreadon-NOAA Feb 7, 2022
74f6f11
Merge branch 'feature/ufsda-rocoto-jobs' into feature/ufsda-parm
RussTreadon-NOAA Feb 7, 2022
dd9c79d
Issue #529: merge branch 'feature/ufsda-parm' at 74f6f11 into featur…
RussTreadon-NOAA Feb 7, 2022
bf02216
replace atmanal with atmanalrun in ORION.env
RussTreadon-NOAA Feb 7, 2022
29c9ce2
Merge branch 'develop' at f8867d3 into feature/ufsda
RussTreadon-NOAA Feb 14, 2022
134fbe9
Issue #521: extend rocoto workflow xml generator to include UFS DA
RussTreadon-NOAA Feb 15, 2022
dd31c92
Issue #521 add missing task_prep to setup_workflow.py
RussTreadon-NOAA Feb 18, 2022
eeab532
Merge branch develop at cb8b5ad into feature/ufsda
RussTreadon-NOAA Feb 18, 2022
8cb2442
Merge branch develop at e3d64ab into feature/ufsda
RussTreadon-NOAA Feb 25, 2022
c39e0ac
Merge branch 'develop' at d758e8b into feature/ufsda
RussTreadon-NOAA Mar 18, 2022
52e4d12
Change UFS-DA to GDASApp
CoryMartin-NOAA Mar 18, 2022
a720c3f
Updated build for GDAS
CoryMartin-NOAA Apr 14, 2022
078b96d
Changes to rocoto scripts
CoryMartin-NOAA Apr 14, 2022
e34e82f
some changes to config.atmanal
CoryMartin-NOAA Apr 14, 2022
431028d
Changes to link_workflow and temp change to feature branch for gdas
CoryMartin-NOAA Apr 18, 2022
cdf86c0
Changes to config files
CoryMartin-NOAA Apr 18, 2022
9406bca
Updates to cycle GDASApp
RussTreadon-NOAA Apr 26, 2022
2782c07
Update gsi build and link
RussTreadon-NOAA Jun 9, 2022
ce04b51
Extend prototype JEDI-GDAS cycling to include JEDI hybvar and lgetkf
RussTreadon-NOAA Jun 17, 2022
b3d0898
Merge branch develop at b41a36a into feature/ufsda-gdasapp
RussTreadon-NOAA Jun 17, 2022
dfbc6c7
Add jobs and configs for JEDI_VAR and JEDI_ENS
RussTreadon-NOAA Jun 17, 2022
828e952
Remove jobs/JGDAS_GLOBAL_ATMOS_ENSANAL (erroneously committed)
RussTreadon-NOAA Jun 21, 2022
b448260
Add external jobs/JGDAS_GLOBAL_ATMOS_ENSANAL to .gitignore
RussTreadon-NOAA Jun 21, 2022
444f6a9
Update sfcanl in env and setup_workflow.py
RussTreadon-NOAA Jun 24, 2022
eab4bc4
Incoporate review feedback for JEDI, postanl, and sfcanl
RussTreadon-NOAA Jun 29, 2022
26901d6
Remove DOGCYCLE from sfcanl script, correct typo in config.postanl
RussTreadon-NOAA Jun 29, 2022
0441b70
Merge branch 'develop' at f0f1025 into feature/ufsda-gdasapp
RussTreadon-NOAA Jun 29, 2022
6674720
Refactor JEDI-VAR and JEDI-ENS jobs, remove postanl (#871)
RussTreadon-NOAA Jul 1, 2022
fa42b0b
add JEDI-VAR and JEDI-ENS j-jobs to .gitignore
RussTreadon-NOAA Jul 1, 2022
24bf705
Merge branch 'develop' into feature/ufsda-gdasapp
RussTreadon-NOAA Jul 5, 2022
c39a498
Merge branch 'develop' into feature/ufsda-gdasapp
RussTreadon-NOAA Jul 6, 2022
85df2df
Merge branch 'develop' at 13385d9c into feature/ufsda-gdasapp
RussTreadon-NOAA Jul 18, 2022
ff1ac51
Merge branch 'develop' at e8361cc into feature/ufsda-gdasapp
RussTreadon-NOAA Jul 18, 2022
66597db
Merge branch 'develop' at 5a58fa8 into feature/ufsda-gdasapp
RussTreadon-NOAA Jul 18, 2022
e2c2322
Merge branch 'develop' into feature/ufsda-gdasapp
RussTreadon-NOAA Jul 18, 2022
cbb3f6d
Merge branch 'develop' into feature/ufsda-gdasapp
RussTreadon-NOAA Jul 20, 2022
fbbc582
Update xml generation to include JEDI_VAR and JEDI_ENS
RussTreadon-NOAA Jul 21, 2022
c0ed4eb
Merge branch 'develop' into feature/ufsda-gdasapp
RussTreadon-NOAA Jul 22, 2022
abb15ab
Update JEDI_VAR dependencies for sfcanl and analcalc
RussTreadon-NOAA Jul 22, 2022
b163190
Update JEDI_ENS dependencies for ecen and esfc
RussTreadon-NOAA Jul 22, 2022
594f68d
Update workflow python scripts as recommended by reviewer
RussTreadon-NOAA Jul 22, 2022
39fe881
Update atmanalrun dependency as suggested by reviewer
RussTreadon-NOAA Jul 22, 2022
866467e
Update initialization of gdas_tasks as suggested by reviewers
RussTreadon-NOAA Jul 22, 2022
b8f639c
Add flag for JEDI_ENS to config.base.nco.static
RussTreadon-NOAA Jul 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ sorc/gfs_post.fd
sorc/gfs_wafs.fd
sorc/gldas.fd
sorc/gsi.fd
sorc/gdas.cd
sorc/ufs_utils.fd
sorc/verif-global.fd

Expand Down Expand Up @@ -139,6 +140,8 @@ jobs/JGDAS_ENKF_POST
jobs/JGDAS_ENKF_SELECT_OBS
jobs/JGDAS_ENKF_SFC
jobs/JGDAS_ENKF_UPDATE
jobs/JGDAS_GLOBAL_ATMOS_ANALYSIS
jobs/JGDAS_GLOBAL_ATMOS_ENSANAL
jobs/JGFS_ATMOS_VMINMON
jobs/JGFS_ATMOS_WAFS
jobs/JGFS_ATMOS_WAFS_BLENDING
Expand Down Expand Up @@ -205,6 +208,7 @@ ush/radmon_verf_angle.sh
ush/radmon_verf_bcoef.sh
ush/radmon_verf_bcor.sh
ush/radmon_verf_time.sh
ush/ufsda
ush/wafs_blending.sh
ush/wafs_grib2.regrid.sh
ush/wafs_intdsk.sh
49 changes: 39 additions & 10 deletions env/HERA.env
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ if [ $# -ne 1 ]; then

echo "Must specify an input argument to set runtime environment variables!"
echo "argument can be any one of the following:"
echo "anal fcst post vrfy metp"
echo "atmanalrun atmensanalrun"
echo "anal sfcanl fcst post vrfy metp"
echo "eobs eupd ecen efcs epos"
echo "postsnd awips gempak"
exit 1
Expand Down Expand Up @@ -50,6 +51,30 @@ elif [ $step = "waveinit" -o $step = "waveprep" -o $step = "wavepostsbs" -o $ste
export wavempexec=${launcher}
export wave_mpmd=${mpmd}

elif [ $step = "atmanalrun" ]; then

export CFP_MP=${CFP_MP:-"YES"}
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="$launcher -n \$ncmd --multi-prog"

nth_max=$(($npe_node_max / $npe_node_atmanalrun))

export NTHREADS_ATMANAL=${nth_atmanalrun:-$nth_max}
[[ $NTHREADS_ATMANAL -gt $nth_max ]] && export NTHREADS_ATMANAL=$nth_max
export APRUN_ATMANAL="$launcher -n $npe_atmanalrun"

elif [ $step = "atmensanalrun" ]; then

export CFP_MP=${CFP_MP:-"YES"}
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="$launcher -n \$ncmd --multi-prog"

nth_max=$(($npe_node_max / $npe_node_atmensanalrun))

export NTHREADS_ATMENSANAL=${nth_atmensanalrun:-$nth_max}
[[ $NTHREADS_ATMENSANAL -gt $nth_max ]] && export NTHREADS_ATMENSANAL=$nth_max
export APRUN_ATMENSANAL="$launcher -n $npe_atmensanalrun"

elif [ $step = "anal" ]; then

export MKL_NUM_THREADS=4
Expand Down Expand Up @@ -79,9 +104,13 @@ elif [ $step = "anal" ]; then
npe_gausfcanl=${npe_gausfcanl:-1}
export APRUN_GAUSFCANL="$launcher -n $npe_gausfcanl"

export NTHREADS_CHGRES=${nth_echgres:-1}
[[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max
export APRUN_CHGRES=""
elif [ $step = "sfcanl" ]; then
nth_max=$(($npe_node_max / $npe_node_sfcanl))

export NTHREADS_CYCLE=${nth_sfcanl:-14}
[[ $NTHREADS_CYCLE -gt $npe_node_max ]] && export NTHREADS_CYCLE=$npe_node_max
npe_sfcanl=${ntiles:-6}
export APRUN_CYCLE="$launcher -n $npe_sfcanl"

elif [ $step = "gldas" ]; then

Expand All @@ -106,28 +135,28 @@ elif [ $step = "eobs" ]; then
export MKL_NUM_THREADS=4
export MKL_CBWR=AUTO

export CFP_MP=${CFP_MP:-"YES"}
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="$launcher -n \$ncmd --multi-prog"

nth_max=$(($npe_node_max / $npe_node_eobs))

export NTHREADS_GSI=${nth_eobs:-$nth_max}
[[ $NTHREADS_GSI -gt $nth_max ]] && export NTHREADS_GSI=$nth_max
export APRUN_GSI="$launcher"

elif [ $step = "eupd" ]; then

export CFP_MP=${CFP_MP:-"YES"}
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="$launcher -n \$ncmd --multi-prog"

elif [ $step = "eupd" ]; then

nth_max=$(($npe_node_max / $npe_node_eupd))

export NTHREADS_ENKF=${nth_eupd:-$nth_max}
[[ $NTHREADS_ENKF -gt $nth_max ]] && export NTHREADS_ENKF=$nth_max
export APRUN_ENKF="$launcher"

export CFP_MP=${CFP_MP:-"YES"}
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="$launcher -n \$ncmd --multi-prog"

elif [ $step = "fcst" ]; then

#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
Expand Down
29 changes: 25 additions & 4 deletions env/JET.env
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ if [ $# -ne 1 ]; then

echo "Must specify an input argument to set runtime environment variables!"
echo "argument can be any one of the following:"
echo "anal fcst post vrfy metp"
echo "atmanalrun atmensanalrun"
echo "anal sfcanl fcst post vrfy metp"
echo "eobs eupd ecen efcs epos"
echo "postsnd awips gempak"
exit 1
Expand Down Expand Up @@ -44,6 +45,22 @@ elif [ $step = "waveinit" -o $step = "waveprep" -o $step = "wavepostsbs" -o $ste
export wavempexec=${launcher}
export wave_mpmd=${mpmd}

elif [ $step = "atmanalrun" ]; then

nth_max=$(($npe_node_max / $npe_node_atmanalrun))

export NTHREADS_ATMANAL=${nth_atmanalrun:-$nth_max}
[[ $NTHREADS_ATMANAL -gt $nth_max ]] && export NTHREADS_ATMANAL=$nth_max
export APRUN_ATMANAL="$launcher $npe_atmanalrun"

elif [ $step = "atmensanalrun" ]; then

nth_max=$(($npe_node_max / $npe_node_atmensanalrun))

export NTHREADS_ATMENSANAL=${nth_atmensanalrun:-$nth_max}
[[ $NTHREADS_ATMENSANAL -gt $nth_max ]] && export NTHREADS_ATMENSANAL=$nth_max
export APRUN_ATMENSANAL="$launcher $npe_atmensanalrun"

elif [ $step = "anal" ]; then

nth_max=$(($npe_node_max / $npe_node_anal))
Expand All @@ -65,9 +82,13 @@ elif [ $step = "anal" ]; then
npe_gausfcanl=${npe_gausfcanl:-1}
export APRUN_GAUSFCANL="$launcher $npe_gausfcanl"

export NTHREADS_CHGRES=${nth_echgres:-1}
[[ $NTHREADS_CHGRES -gt $npe_node_max ]] && export NTHREADS_CHGRES=$npe_node_max
export APRUN_CHGRES=""
elif [ $step = "sfcanl" ]; then
nth_max=$(($npe_node_max / $npe_node_sfcanl))

export NTHREADS_CYCLE=${nth_sfcanl:-14}
[[ $NTHREADS_CYCLE -gt $npe_node_max ]] && export NTHREADS_CYCLE=$npe_node_max
npe_sfcanl=${ntiles:-6}
export APRUN_CYCLE="$launcher $npe_sfcanl"

elif [ $step = "gldas" ]; then

Expand Down
38 changes: 35 additions & 3 deletions env/ORION.env
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ if [ $# -ne 1 ]; then

echo "Must specify an input argument to set runtime environment variables!"
echo "argument can be any one of the following:"
echo "anal fcst post vrfy metp"
echo "atmanalrun atmensanalrun"
echo "anal sfcanl fcst post vrfy metp"
echo "eobs eupd ecen efcs epos"
echo "postsnd awips gempak"
exit 1
Expand Down Expand Up @@ -41,14 +42,38 @@ if [ $step = "prep" -o $step = "prepbufr" ]; then
export BACK=${BACK:-"YES"}
export sys_tp="ORION"

elif [ $step = "waveinit" -o $step = "waveprep" -o $step = "wavepostsbs" -o $step = "wavepostbndpnt" -o $step = "wavepostpnt" ]; then
elif [ $step = "waveinit" -o $step = "waveprep" -o $step = "wavepostsbs" -o $step = "wavepostbndpnt" -o $step = "wavepostbndpntbll" -o $step = "wavepostpnt" ]; then

export mpmd="--multi-prog"
export CFP_MP="YES"
if [ $step = "waveprep" ]; then export MP_PULSE=0 ; fi
export wavempexec=${launcher}
export wave_mpmd=${mpmd}

elif [ $step = "atmanalrun" ]; then

export CFP_MP=${CFP_MP:-"YES"}
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="$launcher -n \$ncmd --multi-prog"

nth_max=$(($npe_node_max / $npe_node_atmanalrun))

export NTHREADS_ATMANAL=${nth_atmanalrun:-$nth_max}
[[ $NTHREADS_ATMANAL -gt $nth_max ]] && export NTHREADS_ATMANAL=$nth_max
export APRUN_ATMANAL="$launcher -n $npe_atmanalrun"

elif [ $step = "atmensanalrun" ]; then

export CFP_MP=${CFP_MP:-"YES"}
export USE_CFP=${USE_CFP:-"YES"}
export APRUNCFP="$launcher -n \$ncmd --multi-prog"

nth_max=$(($npe_node_max / $npe_node_atmensanalrun))

export NTHREADS_ATMENSANAL=${nth_atmensanalrun:-$nth_max}
[[ $NTHREADS_ATMENSANAL -gt $nth_max ]] && export NTHREADS_ATMENSANAL=$nth_max
export APRUN_ATMENSANAL="$launcher -n $npe_atmensanalrun"

elif [ $step = "anal" ]; then

export MKL_NUM_THREADS=4
Expand All @@ -75,9 +100,16 @@ elif [ $step = "anal" ]; then

export NTHREADS_GAUSFCANL=1
npe_gausfcanl=${npe_gausfcanl:-1}

export APRUN_GAUSFCANL="$launcher -n $npe_gausfcanl"

elif [ $step = "sfcanl" ]; then
nth_max=$(($npe_node_max / $npe_node_sfcanl))

export NTHREADS_CYCLE=${nth_sfcanl:-14}
[[ $NTHREADS_CYCLE -gt $npe_node_max ]] && export NTHREADS_CYCLE=$npe_node_max
npe_sfcanl=${ntiles:-6}
export APRUN_CYCLE="$launcher -n $npe_sfcanl"

elif [ $step = "gldas" ]; then

nth_max=$(($npe_node_max / $npe_node_gldas))
Expand Down
Loading