Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change all ocean output files to single precision #3360

Merged
merged 1 commit into from
Mar 2, 2020

Conversation

mark-petersen
Copy link
Contributor

@mark-petersen mark-petersen commented Dec 11, 2019

Currently, all MPAS-Ocean output files are double precision. This is simply not needed, and doubles file space and write time. I added the single line to specify single precision to all output streams with 2D and 3D fields. For standard simulations, this only alters:

  • monthly time averages, used for MPAS-Analysis
  • high frequency stats, used for animations

Restart files remain double precision, of course. The time average is computed in double precision, and the final value is then written in single precision.

This will be bit-for-bit for comparison of restart and coupler files, but not for ocean output files.

[BFB]

@mark-petersen
Copy link
Contributor Author

@vanroekel and I had a conversation about this last month, and we couldn't think of any reason to save any output in double precision. Can anyone else?

One possible accommodation is to save all density fields as (density-1000) to gain a few digits. This would require adding a few variables.

@xylar does MPAS-Analysis make any assumption about double precision fields in the monthly time averages?

@xylar
Copy link
Contributor

xylar commented Dec 12, 2019

@mark-petersen, all I can say is that I ran into some pretty serious plotting artifacts when I worked with 4-byte-float salinity output from POP back in the day, though I'm not sure why. Density does seem like a case where we would be more likely to run into trouble, and writing out density - 1000 seems like a pain (plus a pain to parse again). But we could try just writing out densities with 4-byte floats and run the analysis to see how it goes.

Can we do a test G-case run of a year or so and run MPAS-Analysis on it to make sure we're not going to have problems?

@mark-petersen mark-petersen force-pushed the mark-petersen/ocean/single_precision_output branch from 1cb8973 to b8bca6c Compare December 18, 2019 14:20
@mark-petersen
Copy link
Contributor Author

Ran one year of EC60to30 G case on cori-knl, available here:

/global/cscratch1/sd/mpeterse/acme_scratch/cori-knl/191218_oEC60to30v3_single_precision_output_1

and confirmed that these streams are all writing 4-byte reals:

grep 'precision="single"' streams.ocean -B 2|grep stream
<stream name="eddyProductVariablesOutput"
<stream name="highFrequencyOutput"
<stream name="mixedLayerDepthsOutput"
<stream name="timeSeriesStatsMonthlyOutput"
<stream name="timeSeriesStatsMonthlyMaxOutput"
<stream name="timeSeriesStatsMonthlyMinOutput"

e.g.

ncdump mpaso.hist.am.timeSeriesStatsMonthly.0001-12-01.nc|grep float|head
	float timeMonthly_avg_ssh(Time, nCells) ;
...

where float is 4-byte and double is 8-byte. Next, test with MPAS-Analysis...

@xylar
Copy link
Contributor

xylar commented Dec 23, 2019

@mark-petersen, I've got a job in to run the full analysis on the output. I'll post the results here as soon as they're up.

@xylar
Copy link
Contributor

xylar commented Dec 23, 2019

The analysis is here:
https://portal.nersc.gov/project/m3412/mpas_analysis_output/test/191218_oEC60to30v3_single_precision_output_1/yrs001-001/

I don't see any troubling artifacts of the single precision, even in potential density, so I think we're good to go.

@mark-petersen
Copy link
Contributor Author

@xylar thanks for the quick analysis! That is very reassuring!

@rljacob
Copy link
Member

rljacob commented Jan 30, 2020

Waiting on further analysis from @mark-petersen

@mark-petersen
Copy link
Contributor Author

@milenaveneziani @vanroekel @maltrud Does anyone want to take a look at the analysis Xylar posted and comment or approve? I really think we need to change our output from double to single precision.

Copy link

@sterlingbaldwin sterlingbaldwin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This wont affect any of my workflow tools.

@vanroekel
Copy link
Contributor

@mark-petersen I'm not sure what to look for in the analysis. With only a one year run, it's unclear what a problem would look like. I'm fine approving this based on @xylar's pass through the plots

@milenaveneziani
Copy link
Contributor

What about tendency fields? Could we keep outputting those in DB?

@milenaveneziani
Copy link
Contributor

I just spoke with Tarun, who's been calculating freshwater budgets for the CESM LENS model outputs: he says that all terms were saved as single precision in that case.
So, I think (hope) we should be fine making the switch, even for T and S budget terms.

Copy link
Contributor

@vanroekel vanroekel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr looks good, approving based on Xylar's comments/analysis figures

@mark-petersen mark-petersen removed the request for review from maltrud February 13, 2020 22:13
@mark-petersen
Copy link
Contributor Author

@jonbob, we are still testing the Redi PR, so it makes sense to merge this now. Please proceed when you have the chance.

@rljacob
Copy link
Member

rljacob commented Feb 20, 2020

NOTE: If any of our test cases output ocean history files, I think cprnc will complain that the single precision are not BFB with the double precision ones.

jonbob added a commit that referenced this pull request Feb 25, 2020
…(PR #3360)

Change all ocean output files to single precision

Currently, all MPAS-Ocean output files are double precision. This is not
necessary and doubles file space and write time. This PR adds the single
line to specify single precision to all output streams with 2D and 3D fields.
For standard simulations, this only alters:
* monthly time averages, used for MPAS-Analysis
* high frequency stats, used for animations
Restart files remain double precision. The time average is computed in double
precision, and the final value is then written in single precision.

This will be bit-for-bit for comparison of restart and coupler files,
but not for ocean output files.

[BFB]
@jonbob
Copy link
Contributor

jonbob commented Feb 25, 2020

merged to next

1 similar comment
@jonbob
Copy link
Contributor

jonbob commented Feb 25, 2020

merged to next

@E3SM-Project E3SM-Project deleted a comment from jonbob Feb 25, 2020
@jonbob jonbob merged commit 621e360 into master Mar 2, 2020
@jonbob
Copy link
Contributor

jonbob commented Mar 2, 2020

merged to master

@jonbob jonbob deleted the mark-petersen/ocean/single_precision_output branch March 2, 2020 20:49
jonbob added a commit that referenced this pull request Mar 27, 2020
#3513)

Update maint1.2: add 3D varying GM

This update brings in a new mpas-source submodule with the following PRs from
MPAS ocean/develop:
* Adds options for 3d varying GM bolus and 2d varying phase speed
  (MPAS-Dev/MPAS-Model#288)
* Add GM bolus eddy stats (MPAS-Dev/MPAS-Model#339)
* Fix threading issue in MPAS-O GM routine (MPAS-Dev/MPAS-Model#376)
* Compute landIceMask using geometric_features (MPAS-Dev/MPAS-Model#447)
* Bug fixes for the nonlocal source term in KPP (MPAS-Dev/MPAS-Model#305)
and from E3SM master:
* Change all ocean output files to single precision (E3SM #3360)

[non-BFB]
mark-petersen added a commit to MPAS-Dev/MPAS-Model that referenced this pull request Mar 3, 2021
Change output to single precision in stand-alone default streams files #414

This adds the flag

precision="single"

to the default streams file for output files. It does not alter restart
streams. Bit-for-bit testing in the nightly regression remains 8-byte
reals because that is specified in compass.

The purpose of adding single precision to the default streams is to
remind users that single precision is sufficient and would typically be
used for output. This does not affect E3SM, as the streams files are
created from a script there. We are changing to single precision
analysis output there as well, see E3SM-Project/E3SM#3360
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFB PR leaves answers BFB mpas-ocean Performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants