Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add checkpoint restarts for ufs-cpld #342

Merged
merged 19 commits into from
Dec 31, 2020
Merged

Add checkpoint restarts for ufs-cpld #342

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
72ef6b7
changes for checkpoint restarts
DeniseWorthen Dec 21, 2020
2faa781
update tests
DeniseWorthen Dec 21, 2020
fe05576
test of 3 restart tests on hera; skip-ci
DeniseWorthen Dec 21, 2020
14a0452
Merge branch 'develop' into feature/cpld_checkpoint_restarts
DeniseWorthen Dec 22, 2020
980f9c7
fix instances of srf_data.*; skip-ci
DeniseWorthen Dec 23, 2020
2aef386
update bmark restart test to run from hour 12; skip-ci
DeniseWorthen Dec 23, 2020
2dc4f86
Merge branch 'develop' into feature/cpld_checkpoint_restarts
DeniseWorthen Dec 28, 2020
7656418
Merge branch 'develop' into feature/cpld_checkpoint_restarts
DeniseWorthen Dec 30, 2020
454bf17
fix spacing in rt.conf; skip-ci
DeniseWorthen Dec 30, 2020
86b698d
Merge branch 'develop' into feature/cpld_checkpoint_restarts
DeniseWorthen Dec 30, 2020
42b2978
update rt.conf for cpld and datm
DeniseWorthen Dec 30, 2020
13f76b7
add cheyenne.intel logs; skip-ci
DeniseWorthen Dec 30, 2020
a91e7fb
add orion.intel log; skip-ci
DeniseWorthen Dec 30, 2020
d674aa5
add cheyenne.gnu log; skip-ci
DeniseWorthen Dec 30, 2020
7002281
add hera.intel log; skip-ci
DeniseWorthen Dec 30, 2020
4612323
add dell-p3 log; skip-ci
DeniseWorthen Dec 31, 2020
6cbb15c
add wcoss-cray log; skip-ci
DeniseWorthen Dec 31, 2020
0336894
add gaea.intel and jet.intel logs; skip-ci
DeniseWorthen Dec 31, 2020
d1112b0
add hera.gnu log
DeniseWorthen Dec 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions tests/default_vars.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,10 @@ export FRAZIL_FWSALT='.true.'
# default to write CICE average history files
export CICE_HIST_AVG='.true.'

# checkpoint restarts
export RESTART_FILE_PREFIX=''
export RESTART_FILE_SUFFIX_HRS=''
export RESTART_FILE_SUFFIX_SECS=''
export RT35D=''
}
export_35d_run ()
Expand Down
46 changes: 40 additions & 6 deletions tests/fv3_conf/cpld_bmark_run.IN
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,43 @@ if [[ $CPLWAV == .T. && $CPLWAV2ATM == .T. ]]; then
fi
fi

# ICs
cp @[INPUTDATA_ROOT]/${FV3_IC}/sfc_data*.nc ./INPUT
cp @[INPUTDATA_ROOT]/${FV3_IC}/gfs_data*.nc ./INPUT
cp @[INPUTDATA_ROOT]/${MOM6_IC}/MOM*.nc ./INPUT
# the BM ICs are still named cice5 and need to remain so until P5.0 is completed
cp @[INPUTDATA_ROOT]/${CICE_IC}/cice5_model_@[ICERES].*.nc ./cice_model.res.nc
# No restart
if [ $WARM_START = .F. ]; then
# ICs
cp @[INPUTDATA_ROOT]/${FV3_IC}/sfc_data*.nc ./INPUT
cp @[INPUTDATA_ROOT]/${FV3_IC}/gfs_data*.nc ./INPUT
cp @[INPUTDATA_ROOT]/${MOM6_IC}/MOM*.nc ./INPUT
# the BM ICs are still named cice5 and need to remain so until P5.0 is completed
cp @[INPUTDATA_ROOT]/${CICE_IC}/cice5_model_@[ICERES].*.nc ./cice_model.res.nc
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe in the future we can consider changing "cice5_model" to "cice_model" to avoid confusion as we are running cice6?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we do need to do this. We can change it in the RT since the files are no longer sym-linked to the BM input directory that the workflow uses. But there is potential confusion if the workflow continues to use the name cice5 and the RT uses cice6.

else
# NOTE: bmark-wave model is not currently tested for restart
# Restart files
cp -r ../${DEP_RUN}${RT_SUFFIX}/RESTART/${RESTART_FILE_PREFIX}.* ./INPUT
rm -f INPUT/fv_core.res.*
rm -f INPUT/fv_srf_wnd.res.*
rm -f INPUT/fv_tracer.res.*
rm -f INPUT/phy_data.*c
rm -f INPUT/srf_data.*
for RFILE in ../${DEP_RUN}${RT_SUFFIX}/RESTART/${RESTART_FILE_PREFIX}.*; do
[ -e $RFILE ] || exit 1
RFILE_OLD=$(basename $RFILE)
RFILE_NEW="${RFILE_OLD//${RESTART_FILE_PREFIX}./}"
cp $RFILE INPUT/${RFILE_NEW}
done

# Restart files MOM6 mx025
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM.res.${RESTART_FILE_SUFFIX_HRS}-00-00.nc ./INPUT/MOM.res.nc
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM.res.${RESTART_FILE_SUFFIX_HRS}-00-00_1.nc ./INPUT/MOM.res_1.nc
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM.res.${RESTART_FILE_SUFFIX_HRS}-00-00_2.nc ./INPUT/MOM.res_2.nc
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM.res.${RESTART_FILE_SUFFIX_HRS}-00-00_3.nc ./INPUT/MOM.res_3.nc

# CMEPS restart and pointer files
RFILE="ufs.cpld.cpl.r.${RESTART_FILE_SUFFIX_SECS}.nc"
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/${RFILE} .
ls -1 ${RFILE}>rpointer.cpl

# CICE restart and pointer files
RFILE="iced.${RESTART_FILE_SUFFIX_SECS}.nc"
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/${RFILE} ./INPUT
ls -1 "./INPUT/"${RFILE}>ice.restart_file
fi
35 changes: 26 additions & 9 deletions tests/fv3_conf/cpld_control_run.IN
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,19 +55,36 @@ if [ $WARM_START = .F. ]; then
# Restart
else
# Restart files
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/coupler.res ./INPUT
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/fv_*.nc ./INPUT
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/sfc_data*.nc ./INPUT
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/phy_data*.nc ./INPUT
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM*.nc ./INPUT
cp -r ../${DEP_RUN}${RT_SUFFIX}/RESTART/${RESTART_FILE_PREFIX}.* ./INPUT
rm -f INPUT/fv_core.res.*
rm -f INPUT/fv_srf_wnd.res.*
rm -f INPUT/fv_tracer.res.*
rm -f INPUT/phy_data.*c
rm -f INPUT/srf_data.*
for RFILE in ../${DEP_RUN}${RT_SUFFIX}/RESTART/${RESTART_FILE_PREFIX}.*; do
[ -e $RFILE ] || exit 1
RFILE_OLD=$(basename $RFILE)
RFILE_NEW="${RFILE_OLD//${RESTART_FILE_PREFIX}./}"
cp $RFILE INPUT/${RFILE_NEW}
done

#if not mx025, then mom6 restart is a single file
if [[ $OCNRES == '025' ]]; then
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM.res.${RESTART_FILE_SUFFIX_HRS}-00-00.nc ./INPUT/MOM.res.nc
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM.res.${RESTART_FILE_SUFFIX_HRS}-00-00_1.nc ./INPUT/MOM.res_1.nc
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM.res.${RESTART_FILE_SUFFIX_HRS}-00-00_2.nc ./INPUT/MOM.res_2.nc
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM.res.${RESTART_FILE_SUFFIX_HRS}-00-00_3.nc ./INPUT/MOM.res_3.nc
else
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/MOM.res.${RESTART_FILE_SUFFIX_HRS}-00-00.nc ./INPUT/MOM.res.nc
fi

# CMEPS restart and pointer files
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/ufs.cpld.cpl.r.2016-10-03-43200.nc .
RFILE="ufs.cpld.cpl.r.2016-10-03-43200.nc"
RFILE="ufs.cpld.cpl.r.${RESTART_FILE_SUFFIX_SECS}.nc"
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/${RFILE} .
ls -1 ${RFILE}>rpointer.cpl

# CICE restart and pointer files
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/iced.2016-10-03-43200.nc ./INPUT
RFILE="iced.2016-10-03-43200.nc"
RFILE="iced.${RESTART_FILE_SUFFIX_SECS}.nc"
cp ../${DEP_RUN}${RT_SUFFIX}/RESTART/${RFILE} ./INPUT
ls -1 "./INPUT/"${RFILE}>ice.restart_file
fi
17 changes: 6 additions & 11 deletions tests/rt.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,35 +209,30 @@ RUN | fv3_ccpp_esg_HAFS_v0_hwrf_thompson_debug
COMPILE | CCPP=Y SUITES=FV3_GFS_2017_coupled,FV3_GFS_2017_satmedmf_coupled,FV3_GFS_v15p2_coupled S2S=Y | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
#COMPILE | CCPP=Y SUITES=FV3_GFS_2017_coupled,FV3_GFS_2017_satmedmf_coupled,FV3_GFS_v15p2_coupled S2S=Y | standard | cheyenne.intel | fv3 |
RUN | cpld_control | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
# following two tests are not run on any platform
#RUN | cpld_control_12h | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
#RUN | cpld_restart | standard | hera.intel orion.intel wcoss_dell_p3 | | cpld_control_12h
RUN | cpld_restart | standard | hera.intel orion.intel wcoss_dell_p3 | | cpld_control
RUN | cpld_2threads | standard | hera.intel orion.intel wcoss_dell_p3 | |
RUN | cpld_decomp | standard | hera.intel orion.intel wcoss_dell_p3 | |
RUN | cpld_satmedmf | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
RUN | cpld_ca | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
#RUN | cpld_control | standard | cheyenne.intel | fv3 |
#RUN | cpld_restart | standard | cheyenne.intel | | cpld_control
#RUN | cpld_2threads | standard | cheyenne.intel | |
#RUN | cpld_decomp | standard | cheyenne.intel | |
#RUN | cpld_satmedmf | standard | cheyenne.intel | fv3 |
#RUN | cpld_ca | standard | cheyenne.intel | fv3 |

# restart test at c96mx025
RUN | cpld_control_mx025 | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
RUN | cpld_control_mx025_12h | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
RUN | cpld_restart_mx025 | standard | hera.intel orion.intel wcoss_dell_p3 | | cpld_control_mx025_12h
#RUN | cpld_control_mx025 | standard | cheyenne.intel | fv3 |
#RUN | cpld_control_mx025_12h | standard | cheyenne.intel | fv3 |
#RUN | cpld_restart_mx025 | standard | cheyenne.intel | | cpld_control_mx025_12h

RUN | cpld_control_c192 | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
RUN | cpld_restart_c192 | standard | hera.intel orion.intel wcoss_dell_p3 | | cpld_control_c192
RUN | cpld_control_c384 | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
RUN | cpld_controlfrac_c384 | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
RUN | cpld_bmark | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
RUN | cpld_restart_bmark | standard | hera.intel orion.intel wcoss_dell_p3 | | cpld_bmark
#RUN | cpld_control_c192 | standard | cheyenne.intel | fv3 |
#RUN | cpld_restart_c192 | standard | cheyenne.intel | | cpld_control_c192
#RUN | cpld_control_c384 | standard | cheyenne.intel | fv3 |
#RUN | cpld_controlfrac_c384 | standard | cheyenne.intel | fv3 |
#RUN | cpld_bmark | standard | cheyenne.intel | fv3 |
#RUN | cpld_restart_bmark | standard | cheyenne.intel | | cpld_bmark

COMPILE | CCPP=Y SUITES=FV3_GFS_2017_coupled,FV3_GFS_2017_satmedmf_coupled,FV3_GFS_v15p2_coupled S2S=Y WW3=Y | standard | hera.intel orion.intel wcoss_dell_p3 | fv3 |
#COMPILE | CCPP=Y SUITES=FV3_GFS_2017_coupled,FV3_GFS_2017_satmedmf_coupled,FV3_GFS_v15p2_coupled S2S=Y WW3=Y | standard | cheyenne.intel | fv3 |
Expand Down
5 changes: 5 additions & 0 deletions tests/tests/cpld_bmark
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#
# cpld_bmark test
#

export TEST_DESCR="Fully coupled FV3-CCPP-MOM6-CICE-CMEPS system - C384 MX025 - Benchmark test"

export CNTL_DIR="cpld_bmark"
Expand Down Expand Up @@ -59,6 +60,9 @@ export LIST_FILES="phyf024.tile1.nc \
export_fv3
export_cpl

export RESTART_INTERVAL="6"
export RESTART_N=${RESTART_INTERVAL}

export SYEAR="2013"
export SMONTH="04"
export SDAY="01"
Expand Down Expand Up @@ -101,6 +105,7 @@ export CPL_FAST=${DT_ATMOS}
export coupling_interval_slow_sec=${CPL_SLOW}
export coupling_interval_fast_sec=${CPL_FAST}

#TODO: Remove when BM Frac grid input is added
export FRAC_GRID_INPUT='.F.'

# resolution dependent files
Expand Down
2 changes: 2 additions & 0 deletions tests/tests/cpld_bmark_35d
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ export CPL_FAST=${DT_ATMOS}
export coupling_interval_slow_sec=${CPL_SLOW}
export coupling_interval_fast_sec=${CPL_FAST}

#TODO: Remove when BM Frac grid input is added
export FRAC_GRID_INPUT='.F.'

# resolution dependent files
Expand All @@ -74,6 +75,7 @@ export OZ_PHYS_NEW=".T."

export MOM6_RIVER_RUNOFF='True'
export MOM6_RESTART_SETTING="r"
#TODO: must remove for restart repro
export MOM6_REPRO_LA='True'

export RUNID="cpcice"
Expand Down
2 changes: 2 additions & 0 deletions tests/tests/cpld_bmark_wave
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ export NEMS_CONFIGURE="nems.configure.cpld_wave.IN"
export coupling_interval_slow_sec=${CPL_SLOW}
export coupling_interval_fast_sec=${CPL_FAST}

#TODO: Remove when BM Frac grid input is added
export FRAC_GRID_INPUT='.F.'

export CPLWAV='.T.'
Expand All @@ -132,6 +133,7 @@ export OZ_PHYS_NEW=".T."
export MOM6_USE_WAVES='True'
export MOM6_RIVER_RUNOFF='True'
export MOM6_RESTART_SETTING="r"
#TODO: must remove for restart repro
export MOM6_REPRO_LA='True'

export RUNID="cpcice"
Expand Down
2 changes: 2 additions & 0 deletions tests/tests/cpld_bmark_wave_35d
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ export NEMS_CONFIGURE="nems.configure.cpld_wave.IN"
export coupling_interval_slow_sec=${CPL_SLOW}
export coupling_interval_fast_sec=${CPL_FAST}

#TODO: Remove when BM Frac grid input is added
export FRAC_GRID_INPUT='.F.'

export CPLWAV='.T.'
Expand All @@ -93,6 +94,7 @@ export OZ_PHYS_NEW=".T."
export MOM6_USE_WAVES='True'
export MOM6_RIVER_RUNOFF='True'
export MOM6_RESTART_SETTING="r"
#TODO: must remove for restart repro
export MOM6_REPRO_LA='True'

export RUNID="cpcice"
Expand Down
3 changes: 3 additions & 0 deletions tests/tests/cpld_control
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,4 +57,7 @@ export LIST_FILES="phyf024.tile1.nc \
export_fv3
export_cpl

export RESTART_INTERVAL="12"
export RESTART_N=${RESTART_INTERVAL}

export FV3_RUN=cpld_control_run.IN
64 changes: 0 additions & 64 deletions tests/tests/cpld_control_12h
View file
Open in desktop

This file was deleted.

3 changes: 3 additions & 0 deletions tests/tests/cpld_control_c192
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ export LIST_FILES="phyf024.tile1.nc \
export_fv3
export_cpl

export RESTART_INTERVAL="6"
export RESTART_N=${RESTART_INTERVAL}

export TASKS=$TASKS_cpl_c192
export TPN=$TPN_cpl_c192
export INPES=$INPES_cpl_c192
Expand Down
2 changes: 2 additions & 0 deletions tests/tests/cpld_control_c384
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ export CPL_FAST=${DT_ATMOS}
export coupling_interval_slow_sec=${CPL_SLOW}
export coupling_interval_fast_sec=${CPL_FAST}

#TODO: Should remove
export FRAC_GRID_INPUT='.F.'

# resolution dependent files
Expand All @@ -115,6 +116,7 @@ export FNSMCC="'global_soilmgldas.t766.1536.768.grb',"
export FNABSC="'global_mxsnoalb.uariz.t766.1536.768.rg.grb',"

export MOM6_RIVER_RUNOFF='True'
#TODO: must remove for restart repro
export MOM6_REPRO_LA='True'

export FV3_RUN=cpld_control_run.IN
Loading