Add CF-compliant Time coordinate #19

xylar · 2022-05-18T03:32:40Z

This variable has CF-compliant units and calendar. To accommodate the CF-compliant units, a new config option has been added to supply the reference time, 0001-01-01 by default.

xylar · 2022-05-18T03:37:26Z

Here are the results so far (using the QU240/PHC/performance test in compass):

$ ncdump -h output.nc
...
	double Time(Time) ;
		Time:long_name = "time" ;
		Time:units = "days since 0001-01-01 00:00:00" ;
		Time:calendar = "gregorian_noleap" ;
...

$ cfchecks output.nc 
CHECKING NetCDF FILE: output.nc
=====================
Using CF Checker Version 4.1.0
Checking against CF Version CF-1.8
Using Standard Name Table Version 79 (2022-03-19T15:25:54Z)
Using Area Type Table Version 10 (23 June 2020)
Using Standardized Region Name Table Version 4 (18 December 2018)

ERROR: (2.6.1): This netCDF file does not appear to contain CF Convention data.

------------------
Checking variable: Time
------------------
...

Still to do:

fix timeDaily_avg_Time --> Time in time_series_stats analysis member
add units and calendar to Time in time_series_stats analysis member
add bnds attribute and Time_bnds coordinate in time_series_stats analysis member, perhaps following @mark-petersen's work: https://github.com/mark-petersen/MPAS-Model/blob/cf_compliant_time_using_io_streams/src/core_ocean/analysis_members/mpas_ocn_time_series_stats.F
add Time and Time_bnds to streams that have xtime or similar (xtime_startMonthly, etc.)
add standard_name
...

xylar · 2022-05-18T04:03:14Z

This needs work (in QU240/PHC/daily_output_test):

$ ncdump -h forward/analysis_members/mpaso.hist.am.timeSeriesStatsDaily.0001-01-01.nc
...
	double timeDaily_avg_Time(Time) ;
		timeDaily_avg_Time:long_name = "time" ;
...

xylar · 2022-05-25T06:00:16Z

Getting there:

	double timeDaily_avg_Time(Time) ;
		timeDaily_avg_Time:long_name = "time" ;
		timeDaily_avg_Time:units = "days since 0001-01-01 00:00:00" ;
		timeDaily_avg_Time:calendar = "gregorian_noleap" ;
	double timeDaily_avg_Time_bnds(Time, bnds) ;
		timeDaily_avg_Time_bnds:long_name = "time bounds" ;

But the names in output aren't Time and Time_bnds, as expected.

cbegeman

I ran the QU240/PHC daily_output test case on chrysalis with intel (mpi) compiler and the Time and Time_bounds variables look as expected in the daily stats analysis member and in the output stream, when they are added as variables to that stream.

Thanks, @xylar!

xylar · 2022-08-29T18:27:01Z

Thanks @cbegeman. I ran into trouble with the daily_output_test test case on my laptop. xarray isn't happy with the existing gregorian_noleap calendar. I'm hoping to solve that in E3SM-Project#5162 and then rebase this.

xylar · 2022-08-29T18:27:47Z

The detailed error was:

compass calling: compass.ocean.tests.global_ocean.daily_output_test.DailyOutputTest.validate()
  in /home/xylar/code/compass/compass/compass/ocean/tests/global_ocean/daily_output_test/__init__.py

Exception raised in validate()
Traceback (most recent call last):
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/coding/times.py", line 261, in decode_cf_datetime
    dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar)
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/coding/times.py", line 201, in _decode_datetime_with_pandas
    raise OutOfBoundsDatetime(
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Cannot decode times from a non-standard calendar, 'gregorian_noleap', using pandas.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/coding/times.py", line 174, in _decode_cf_datetime_dtype
    result = decode_cf_datetime(example_value, units, calendar, use_cftime)
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/coding/times.py", line 263, in decode_cf_datetime
    dates = _decode_datetime_with_cftime(
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/coding/times.py", line 195, in _decode_datetime_with_cftime
    cftime.num2date(num_dates, units, calendar, only_use_cftime_datetimes=True)
  File "src/cftime/_cftime.pyx", line 549, in cftime._cftime.num2date
  File "src/cftime/_cftime.pyx", line 117, in cftime._cftime._dateparse
  File "src/cftime/_cftime.pyx", line 1153, in cftime._cftime.datetime.__init__
ValueError: calendar must be one of ['standard', 'gregorian', 'proleptic_gregorian', 'noleap', 'julian', 'all_leap', '365_day', '366_day', '360_day'], got 'gregorian_noleap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xylar/code/compass/compass/compass/run/serial.py", line 144, in run_tests
    test_case.validate()
  File "/home/xylar/code/compass/compass/compass/ocean/tests/global_ocean/daily_output_test/__init__.py", line 61, in validate
    compare_variables(
  File "/home/xylar/code/compass/compass/compass/validate.py", line 140, in compare_variables
    result = _compare_variables(
  File "/home/xylar/code/compass/compass/compass/validate.py", line 214, in _compare_variables
    ds1 = xarray.open_dataset(filename1)
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/backends/api.py", line 495, in open_dataset
    backend_ds = backend.open_dataset(
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 567, in open_dataset
    ds = store_entrypoint.open_dataset(
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/backends/store.py", line 27, in open_dataset
    vars, attrs, coord_names = conventions.decode_cf_variables(
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/conventions.py", line 516, in decode_cf_variables
    new_vars[k] = decode_cf_variable(
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/conventions.py", line 364, in decode_cf_variable
    var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/coding/times.py", line 673, in decode
    dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
  File "/home/xylar/miniconda3/envs/dev_compass_1.2.0-alpha.1_mpich/lib/python3.9/site-packages/xarray/coding/times.py", line 184, in _decode_cf_datetime_dtype
    raise ValueError(msg)
ValueError: unable to decode time units 'days since 0001-01-01 00:00:00' with "calendar 'gregorian_noleap'". Try opening your dataset with decode_times=False or installing cftime if it is not installed.

Rename gregorian_noleap to just noleap calendar in mpas-framework The noleap is the CF-compliant name for what MPAS currently calls the gregorian_noleap calendar. This merge changes gregorian_noleap to noleap in the MPAS framework and in all 3 MPAS components. The context of this change is that the config_calendar_type namelist option is being used for the calendar attribute of the Time coordinate in E3SM-Ocean-Discussion#19. However, the result is not making post-processing tools (notably xarray) happy because the gregorian_noleap calendar is not CF-compliant. [NML] [BFB]

This variable has CF-compliant units and calendar. To accommodate the CF-compliant units, a new config option has been added to supply the reference time, 0001-01-01 by default.

xylar · 2022-09-25T05:00:21Z

Closed in favor of E3SM-Project#5202

activate snicar in icepack, check values

xylar changed the title ~~Add Time coordinate~~ Add CF-compliant Time coordinate May 18, 2022

xylar force-pushed the ocn/add-cf-time branch from 83d0c89 to 04d0f5b Compare May 25, 2022 05:57

xylar marked this pull request as draft May 29, 2022 14:53

xylar mentioned this pull request Aug 6, 2022

[FRAMEWORK] Add a new name_in_output registry attribute to variables #23

Closed

xylar force-pushed the ocn/add-cf-time branch from 04d0f5b to 1dbc11a Compare August 9, 2022 18:59

xylar force-pushed the ocn/add-cf-time branch from 1dbc11a to 357c92c Compare August 29, 2022 09:18

xylar changed the base branch from master to alternate August 29, 2022 09:21

xylar changed the base branch from alternate to master August 29, 2022 09:22

xylar marked this pull request as ready for review August 29, 2022 09:22

xylar requested review from mark-petersen and cbegeman August 29, 2022 09:33

xylar force-pushed the ocn/add-cf-time branch from 357c92c to 4d7845a Compare August 29, 2022 09:38

xylar mentioned this pull request Aug 29, 2022

[MPAS-Framework] Rename gregorian_noleap to just noleap calendar E3SM-Project/E3SM#5162

Merged

cbegeman approved these changes Aug 29, 2022

View reviewed changes

xylar added 6 commits September 24, 2022 22:24

Add Time coordinate

a4f641c

This variable has CF-compliant units and calendar. To accommodate the CF-compliant units, a new config option has been added to supply the reference time, 0001-01-01 by default.

Add Time_bnds variable

cc931c2

Fix typos and white-space in time series stats

c178ffa

Clean up input and output field names

d3b80ae

Always add Time and Time_bnds to time series stats

31ffa23

Compute Time and Time_bnds in time series stats

4733d97

xylar force-pushed the ocn/add-cf-time branch from 4d7845a to 4733d97 Compare September 25, 2022 04:24

xylar changed the base branch from master to alternate September 25, 2022 04:27

xylar changed the base branch from alternate to master September 25, 2022 04:28

Add standard name

1523108

xylar mentioned this pull request Sep 25, 2022

Add CF-compliant Time coordinate E3SM-Project/E3SM#5202

Merged

xylar closed this Sep 25, 2022

xylar deleted the ocn/add-cf-time branch October 7, 2022 15:58

xylar pushed a commit that referenced this pull request Nov 8, 2023

Merge pull request #19 from eclare108213/snicar_active

d25dc66

activate snicar in icepack, check values

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CF-compliant Time coordinate #19

Add CF-compliant Time coordinate #19

xylar commented May 18, 2022

xylar commented May 18, 2022 •

edited

Loading

xylar commented May 18, 2022

xylar commented May 25, 2022

cbegeman left a comment

xylar commented Aug 29, 2022

xylar commented Aug 29, 2022

xylar commented Sep 25, 2022

Add CF-compliant Time coordinate #19

Add CF-compliant Time coordinate #19

Conversation

xylar commented May 18, 2022

xylar commented May 18, 2022 • edited Loading

xylar commented May 18, 2022

xylar commented May 25, 2022

cbegeman left a comment

Choose a reason for hiding this comment

xylar commented Aug 29, 2022

xylar commented Aug 29, 2022

xylar commented Sep 25, 2022

xylar commented May 18, 2022 •

edited

Loading