Handling of time ranges (mostly in datafinder)

[zklaus]

There are a couple of issues with the current handling of time ranges in the data finder:

• Make datafinder understand dates in the middle of the file name #311 deals with an emac requirement to have time ranges in the middle of the filename
• as we start dealing with sub-monthly frequencies we might need higher resolution than the current start_year end_year pair permits
• fixed frequency data (fx variables) don't have a time range
• piControl aligned with Historical data #342 shows a case where ideally the time range for some datasets would be computed from the metadata of other datasets

The intuitive choice seems to be to specify time ranges as a single string item per dataset, and adopting the CMIP convention for the format (that has stayed consistent from CMIP5 to CMIP6) that is also used for obs4MIPs these days.

This could also open up the possibility of having special values like corresponding to determine the correct range from other datasets and would allow the explicit use of the -clim suffix that appears in climatological variables such as those in the monC table.

Challenges might include older obs4MIPs datasets that separate the two parts of their timerange with _ instead of - and non-standard datasets.

So to sum up the idea is:

• remove start_year and end_year in favor of a timerange item in the dataset specs
• adopt the CMIP convention for specifying time ranges (p 14-16) including -clim suffix
• from this information build a {timerange} tag for use in the config-developer.yml file

What do you guys think?

[bouweandela]

I think this would mostly be applicable to the way we specify times in the recipe. What do you think about this, @mattiarighi ?

[mattiarighi]

Sounds convenient.
How would the new dataset key look like?

timerange: 1996-2001

?

How many formats are we going to support?
I can think about the following ones:

timerange: 1996-2001
timerange: 199603-200110
timerange: 19960105-20010509

[zklaus]

Ideally, I would like to go completely with CMIP, ie quoting from the abovementioned document:

The <time_range> is a string generated consistent with the following:
If frequency = “fx” then
<time_range>=””
else
<time_range> = N1-N2[suffix] where N1 and N2 are integers of the form ‘yyyy[MM[dd[hh[mm[ss]]]]]’ (expressed as a string, where where ‘yyyy’, ‘MM’, ‘dd’, ‘hh’ ‘mm’ and ‘ss’ are integer year, month, day, hour, minute, and second, respectively)
endif

where suffix is defined as follows:
if the variable identified by variable_id has a time dimension with a “climatology” attribute then
suffix = “-clim”
else
suffix = “”
endif

and where the precision of the time_range strings is determined by the “frequency” global attribute as specified in Table 2.

and Table 2 is

Frequency	Precision of time label	Notes
yr, dec, yrPt	“yyyy”	Label with the years recorded in the  first and last coordinate values.
mon, monC	“yyyyMM”	For “mon”, label with the months recorded in the  first and last coordinate values; for “monC” label with the first and last months contributing to the climatology.
day	“yyyyMMdd”	Label with the days recorded in the  first and last coordinate values.
6hr, 3hr, 1hr, 1hrCM, 6hrPt, 3hrPt, 1hrPt	“yyyyMMddhhmm”	Label 1hrCM files with the beginning of the first hour and the end of the last hour contributing to climatology (rounded to the nearest minute); for other frequencies in this category, label with the first and last time-coordinate values (rounded to the nearest minute).
subhrPt	“yyyyMMddhhmmss”	Label with the first and last time-coordinate values (rounded to the nearest second)
fx	Omit time label	This frequency applies to variables that are independent of time (“fixed”).

[mattiarighi]

Makes sense.

It might be quite some work to adapt all recipes and test them after this change, but I support it.
Other opinions? @ESMValGroup/esmvaltool-coreteam

[jvegreg]

It's great news for me. I would need something like this for the decadal predictions support

[BenMGeo]

I fully support this. In C3S_511 we have to analyze data "as is". I currently filled the noncomplete years "by hand" with missing values.

Would make my life easier ;)

[bouweandela]

It sounds like everyone would be really happy to have this feature. @zklaus Would you be able to implement it?

[zklaus]

I'll have a look in the new year, that is after 2020-01-13.

[Peter9192]

We're addressing at least part of this issue in #358

[zklaus]

I think this has been addressed in #1133