Correct ocean/ice output settings and surface forcing #1823

mark-petersen · 2017-10-07T16:39:20Z

This PR updates output settings for MPAS-Ocean and MPAS-sea ice. They are not extremely time critical.

Changes in MPAS and ACME for G case salinity restoring which is bit-restartable and bfb across different partitions.
Change truncate to overwrite on most ocean and ice streams. Otherwise, we delete high frequency and global stats files with mid-year restarts.
Stop writing sea ice time series stats restart files (see Stop writing ocean time series stats restart files #1777)
Change high frequency streams variables (for movies). We currently record many unneeded variables for debugging that are also recorded in the monthly history files.
White space clean up in buildnml
MPAS timer integer division was changed from I4 to I8 so that clock can run past 34 years (counted in seconds).

[NML]
[BFB]

mark-petersen · 2017-10-07T17:04:17Z

@vanroekel this is still a PR in progress, but you can set up and test the high rez G case on here. The MPAS-ocean submodule points to the head of your commit for G case salinity restoring.
@stephenprice and @xylar my testing process for this includes low and mid rez land ice cases on theta.

Upgrade build system. Make incompatible flags to case.build mutually exclusive Add new flag for building individual components Never run buildnml when building individual libs/components Test suite: by-hand, scripts_regression_tests Test baseline: Test namelist changes: Test status: bit for bit Fixes #1741 User interface changes?: Yes, new -b flag to case.build. Update gh-pages html (Y/N)?: N Code review: jedwards, sacks

mark-petersen · 2017-10-16T19:01:09Z

@jonbob and @vanroekel This PR now has everything from last Friday that made the BFB restarts with G case salinity restoring work. It passed a quick test:

PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel -p m2833 -m cori-knl

without salinity restoring. @jonbob could you test ERS with salinity restoring on?

- framework updates - mpas-o surface restoring still a work in progress.

Previously, the sea ice would write monthly time-averaged files, and also monthly restart files for the time average. This serves no purpose, and wastes i/o time and disk space. Restart files are only needed for time-averaged files if the restart interval is shorter than the output interval (strictly speaking, if the restart interval is not a multiple of the output interval). The MPAS-sea ice submodule can now use config_am_timeSeriesStatsMonthly_restart_stream = 'none' and similar flags. In addition, flags are changed so that the default behavior in ACME is to not write time-averaged restart files for both Monthly and Daily sea ice averages. (Daily averages are default off anyway). Note this does not remove any capability. If you wanted restart files written for monthly time averages for, say, a high-resolution run that proceeds 10 days at a time, you would just change input_interval and output_interval in the streams file. I think that will be a small minority of cases. see ocean version here: #1777

The files in ACME streams.ocean have clobber_mode="truncate" and when I run it with truncate in stand alone, for files with more than one time like highFrequencyOutput, it will delete the whole file upon restart. It is safer to just use clobber_mode="overwrite" for everything. That does not delete the file, but just overwrites a record with the same time. It only writes a warning to the processor zero log file, so does not create a log file for every single processor. For simulations with yearly restarts in ACME everything is fine. For monthly or other restarts, it will delete some analysis data. All those lines are originally from an old commit: c79031a (Robert Jacob 2016-07-25 00:03:30 -0500 569) print $stream_file ' clobber_mode="truncate"' . "\n";

Makes changes for ocean forcing to work with new framework

Ocean submodule also has salinity restoring changes. Remove config_forcing_restart_file

mark-petersen · 2017-10-18T13:20:42Z

Passes:

PET_Ln9.ne30_oECv3_ICG.A_WCYCL1850S.cori-knl_gnu
PEM_Ln9.ne30_oECv3_ICG.A_WCYCL1850S.cori-knl_gnu
PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel
PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_gnu

Overwrite was causing a pio error that we don't yet understand.

This is because we changed clobber mode back to truncate. If we use yearly file intervals but restart within the year, we will delete the previous part of the file.

mark-petersen · 2017-10-18T20:54:19Z

We were having some runs hang on PIO writes at random times. We suspected the change from truncate to overwrite, so went back. Changed some analysis output to monthly so mid-year restarts don't delete files. Latest commit passes:

PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_gnu

jonbob · 2017-10-19T14:53:38Z

With salinity restoring on, it passes:

ERS_P480.T62_oEC60to30v3.GMPAS-NYF.edison_gnu
PET_Ln9_P240.T62_oEC60to30v3.GMPAS-NYF.edison_intel
PEM_Ln9_P480.T62_oEC60to30v3.GMPAS-NYF.edison_intel

mark-petersen · 2017-10-19T15:15:19Z

Also tested cases with land ice cavities. These run successfully:

./create_newcase -case $CASE_ROOT/$ACME_CASE -compiler intel -mach theta -project OceanClimate -compset GMPAS-IAF -res T62_oEC60to30v3wLI
./create_newcase -case $CASE_ROOT/$ACME_CASE -compiler intel -mach theta -project OceanClimate -compset A_WCYCL1850 -res ne30_oECv3wLI

@JeremyFyke note the second one is a B case. The only thing I had to do was add

tail -n 1 user_nl_clm
finidat=''

jonbob · 2017-10-19T18:38:51Z

I updated the mpasli namelist scripts to match the new codebase and added some fields @akturner had requested to mpascice output. It passed SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.edison_gnu with new changes

jonbob · 2017-10-19T19:36:06Z

Salinity restoring is on for G-cases by default now

mark-petersen · 2017-10-19T21:47:20Z

So far, last commit passes:

PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel
ERS_Ld3.T62_oQU240.GMPAS-IAF.cori-knl_gnu

Also checked that salinity restoring is on for:

PET_Ln9.T62_oEC60to30v3.GMPAS-IAF.cori-knl_gnu

using:

cori01:run$ grep config_use_surface_salinity_monthly_restoring mpas-o_in
 config_use_surface_salinity_monthly_restoring = .true.
cori01:run$ grep config_use_activetracers_surface_restoring mpas-o_in
 config_use_activetracers_surface_restoring = .true.

Correct ocean/ice output settings and surface forcing This PR updates the MPAS framework and corrects the use of mpas_forcing in all components. It also updates output settings for all MPAS components. * Changes in MPAS and ACME for G case salinity restoring which is now bit-restartable and bfb across different partitions. It is set on by default for G-cases with resolutions that have the required restoring data files. * Stop writing sea ice time series stats restart files (see 1777) * Change high frequency streams variables (for movies). We currently record many unneeded variables for debugging that are also recorded in the monthly history files. * White space clean up in buildnml * MPAS timer integer division was changed from I4 to I8 so that clock can run past 34 years (counted in seconds). Tested with: * PET_Ln9.ne30_oECv3_ICG.A_WCYCL1850S.cori-knl_gnu * PEM_Ln9.ne30_oECv3_ICG.A_WCYCL1850S.cori-knl_gnu * PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel * PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_gnu * ERS_P480.T62_oEC60to30v3.GMPAS-NYF.edison_gnu (with restoring on) * PET_Ln9_P240.T62_oEC60to30v3.GMPAS-NYF.edison_intel (with restoring on) * PEM_Ln9_P480.T62_oEC60to30v3.GMPAS-NYF.edison_intel (with restoring on) [NML] [BFB]

jonbob · 2017-10-19T22:44:02Z

merged to next

mark-petersen · 2017-10-20T14:07:16Z

Last commit passes

PET_Ln9.ne30_oECv3_ICG.A_WCYCL1850S.cori-knl_gnu
PET_Ln9.T62_oEC60to30v3.GMPAS-IAF.cori-knl_gnu (salinity restoring on by default)

Also ran several cases successfully with ice cavities.

Correct ocean/ice output settings and surface forcing This PR updates the MPAS framework and corrects the use of mpas_forcing in all components. It also updates output settings for all MPAS components. * Changes in MPAS and ACME for G case salinity restoring which is now bit-restartable and bfb across different partitions. It is set on by default for G-cases with resolutions that have the required restoring data files. * Stop writing sea ice time series stats restart files (see 1777) * Change high frequency streams variables (for movies). We currently record many unneeded variables for debugging that are also recorded in the monthly history files. * White space clean up in buildnml * MPAS timer integer division was changed from I4 to I8 so that clock can run past 34 years (counted in seconds). Tested with: * PET_Ln9.ne30_oECv3_ICG.A_WCYCL1850S.cori-knl_gnu * PEM_Ln9.ne30_oECv3_ICG.A_WCYCL1850S.cori-knl_gnu * PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel * PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_gnu * ERS_P480.T62_oEC60to30v3.GMPAS-NYF.edison_gnu (with restoring on) * PET_Ln9_P240.T62_oEC60to30v3.GMPAS-NYF.edison_intel (with restoring on) * PEM_Ln9_P480.T62_oEC60to30v3.GMPAS-NYF.edison_intel (with restoring on) [NML] [BFB]

jonbob · 2017-10-20T15:04:34Z

merged to master

mark-petersen · 2017-10-20T15:36:22Z

Woo-hoo! I did a few last tests on this branch - restart tests on G case with salinity restoring. I confirmed that salinity restoring is on, and they all pass. @vanroekel: good job and thank you for your dedication to a thorny issue. Note I tested both intel and gnu, due on recent issues.

ERS_Ld3.T62_oEC60to30v3.GMPAS-IAF.cori-knl_intel
ERS_Ld3.T62_oEC60to30v3.GMPAS-IAF.cori-knl_gnu

vanroekel · 2017-10-20T15:56:54Z

Thanks @mark-petersen and thanks to @jonbob for his help on this.

Minor clean-up namelists to match new versions of mpas components This PR pulls together some minor script clean-ups that go along with the recent mpas update in PR #1823. There are some minor but necessary namelist and streams improvements. Tested with: * PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_gnu * PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel * ERS_Ld3.T62_oQU240.GMPAS-IAF.cori-knl_gnu * PET_Ln9.T62_oQU240.GMPAS-IAF.theta_intel [BFB] [NML]

Correct ocean/ice output settings and surface forcing This PR updates the MPAS framework and corrects the use of mpas_forcing in all components. It also updates output settings for all MPAS components. * Changes in MPAS and ACME for G case salinity restoring which is now bit-restartable and bfb across different partitions. It is set on by default for G-cases with resolutions that have the required restoring data files. * Stop writing sea ice time series stats restart files (see 1777) * Change high frequency streams variables (for movies). We currently record many unneeded variables for debugging that are also recorded in the monthly history files. * White space clean up in buildnml * MPAS timer integer division was changed from I4 to I8 so that clock can run past 34 years (counted in seconds). Tested with: * PET_Ln9.ne30_oECv3_ICG.A_WCYCL1850S.cori-knl_gnu * PEM_Ln9.ne30_oECv3_ICG.A_WCYCL1850S.cori-knl_gnu * PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_intel * PET_Ln9.T62_oQU240.GMPAS-IAF.cori-knl_gnu * ERS_P480.T62_oEC60to30v3.GMPAS-NYF.edison_gnu (with restoring on) * PET_Ln9_P240.T62_oEC60to30v3.GMPAS-NYF.edison_intel (with restoring on) * PEM_Ln9_P480.T62_oEC60to30v3.GMPAS-NYF.edison_intel (with restoring on) [NML] [BFB]

mark-petersen added BFB PR leaves answers BFB NML mpas-ocean mpas-seaice labels Oct 7, 2017

rljacob assigned jonbob Oct 7, 2017

mark-petersen force-pushed the mark-petersen/update_mpas_171006 branch 2 times, most recently from 78f8e99 to e438153 Compare October 11, 2017 12:59

mark-petersen force-pushed the mark-petersen/update_mpas_171006 branch 3 times, most recently from 53976b3 to 1fc3a2b Compare October 12, 2017 16:00

vanroekel force-pushed the mark-petersen/update_mpas_171006 branch from 7b6d531 to f3fe44a Compare October 12, 2017 18:26

mark-petersen changed the title ~~Correct output settings for ocean and sea ice~~ Correct ocean/ice output settings and surface forcing Oct 12, 2017

mark-petersen mentioned this pull request Oct 12, 2017

SMS_D.T62_oQU120_ais20.MPAS_LISIO_TEST failing with Intel #1829

Closed

mark-petersen force-pushed the mark-petersen/update_mpas_171006 branch 2 times, most recently from 34f0d4a to 962369c Compare October 16, 2017 18:56

mark-petersen and others added 12 commits October 17, 2017 12:50

Update MPAS submodules.

b852d54

- framework updates - mpas-o surface restoring still a work in progress.

Update I8 kind for internal division.

022c867

Change high frequency streams variables.

6cb99ec

White space clean up.

7b291a8

A few more high freq stats changes.

2b2b48b

Changes in ocean core for new forcing framework

98d6da3

Makes changes for ocean forcing to work with new framework

Update MPAS submodules with new framework.

fc123a6

Ocean submodule also has salinity restoring changes. Remove config_forcing_restart_file

Add reference time to all streams. Add high freq fields.

b56a2e6

remove cice namelist variable config_forcing_restart_file

c5917ff

remove ocean namelist variable config_forcing_restart_file

22f0e1e

mark-petersen added 2 commits October 18, 2017 11:17

Change overwrite back to truncate

3332f89

Overwrite was causing a pio error that we don't yet understand.

Change filename_interval to monthly on all streams.

7778cc7

This is because we changed clobber mode back to truncate. If we use yearly file intervals but restart within the year, we will delete the previous part of the file.

Update build-namelist scripts to reflect new mpasli codebase

5c303d3

Turn salinity restoring on by default for G-cases

2b88e31

jonbob merged commit 2b88e31 into master Oct 20, 2017

jonbob deleted the mark-petersen/update_mpas_171006 branch October 20, 2017 15:04

mark-petersen mentioned this pull request Oct 20, 2017

merge develop, update surface restoring MPAS-Dev/MPAS#1427

Merged

jonbob mentioned this pull request Oct 23, 2017

G-cases fail exact restart tests with salinity restoring on #1524

Closed

This was referenced Oct 23, 2017

Data race condition in threaded halo exchange in MPAS framework #1722

Closed

MPAS analysis streams don't like restarting anywhere but month boundaries #1294

Closed

This was referenced Oct 23, 2017

MPAS CICE failure on bebop #1711

Closed

Minor clean-up namelists to match new versions of mpas components #1862

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Correct ocean/ice output settings and surface forcing #1823

Correct ocean/ice output settings and surface forcing #1823

mark-petersen commented Oct 7, 2017 •

edited

Loading

mark-petersen commented Oct 7, 2017 •

edited

Loading

mark-petersen commented Oct 16, 2017

mark-petersen commented Oct 18, 2017

mark-petersen commented Oct 18, 2017

jonbob commented Oct 19, 2017

mark-petersen commented Oct 19, 2017

jonbob commented Oct 19, 2017

jonbob commented Oct 19, 2017

mark-petersen commented Oct 19, 2017 •

edited

Loading

jonbob commented Oct 19, 2017

mark-petersen commented Oct 20, 2017

jonbob commented Oct 20, 2017

mark-petersen commented Oct 20, 2017

vanroekel commented Oct 20, 2017

Correct ocean/ice output settings and surface forcing #1823

Correct ocean/ice output settings and surface forcing #1823

Conversation

mark-petersen commented Oct 7, 2017 • edited Loading

mark-petersen commented Oct 7, 2017 • edited Loading

mark-petersen commented Oct 16, 2017

mark-petersen commented Oct 18, 2017

mark-petersen commented Oct 18, 2017

jonbob commented Oct 19, 2017

mark-petersen commented Oct 19, 2017

jonbob commented Oct 19, 2017

jonbob commented Oct 19, 2017

mark-petersen commented Oct 19, 2017 • edited Loading

jonbob commented Oct 19, 2017

mark-petersen commented Oct 20, 2017

jonbob commented Oct 20, 2017

mark-petersen commented Oct 20, 2017

vanroekel commented Oct 20, 2017

mark-petersen commented Oct 7, 2017 •

edited

Loading

mark-petersen commented Oct 7, 2017 •

edited

Loading

mark-petersen commented Oct 19, 2017 •

edited

Loading