Use ISO standard date-time formats

[cylc]

This is a completely general notation for specifying date/times and time ranges. It would presumably allow us to replace all the current cycling modules (daily, monthly, etc.) with a single one that could handle any cycle time sequence. And cycling for paleo-climate simulations?

MPI is developing an ISO time library for Python.

[cylc]

Also: http://pypi.python.org/pypi/isodate

[cylc]

http://en.wikipedia.org/wiki/ISO_8601

[matthewrmshin]

Would this eventual change be quite disruptive? Perhaps it would be desirable for cylc to write cycle time in the YYYYmmddTHH format sooner rather than later?

[hjoliver]

Matt, unfortunately I haven't had a chance to think about this yet, or even read the ISO standard references. Is the notation you've suggested sufficiently general to handle everything we could conceivably want to do with cylc - including cycling for paleo-climate simulations (how are pre-historic dates represented? And, we would need to include minutes too for future NWP suites.

Luis @m214089 - do you have any advice on this: i.e. can we quickly convert to a general cycle time notation that will make the future transition to ISO date-time representation easy on users?

[m214089]

Hi,

we currently change our time control in the models to an ISO based scheme. By the way a date looks like Matt wrote as one possible way. Recurrences/event are denoted RyyyyyYmmMddDT...... where everything not used could be missing.

I have a shorter than the standard description of the whole thing, but I can provide it to you only after the 15th, because I'm leaving for vacation later today ...

If that's fine with you just answer, and I'll provide you with the information by then.

Cheerio,
Luis

[matthewrmshin]

I think the ISO 8601 format should be sufficient to handle every date time representation we'll need, as it is designed to do so. If it does not do so, we can put the blame on the standard format ;-) In particular, historical dates and 9999+ years can both be represented.

(A second thought. I guess this change does not have to be disruptive at all. When the ISO functionality is introduced in Cylc, we can provide a classic mode where YYYYmmddHH will be used, and an ISO mode where the (basic) ISO 8601 format will be used.)

[m214089]

Hi Matt,
you're right. I do have some regex for parsing ISO 8601:2004, but can supply them only after the 15th. I hope that is
still fine.
Luis

[hjoliver]

Luis, that's fine - have a great vacation!

On 7/10/2012, at 7:51, m214089 notifications@github.com wrote:

Hi Matt,
you're right. I do have some regex for parsing ISO 8601:2004, but can supply them only after the 15th. I hope that is
still fine.
Luis

—
Reply to this email directly or view it on GitHub.

[hjoliver]

@matthewrmshin, I know the ISO standard is meant to cover all bases, I was just asking if the notation you suggested (YYYYmmddTHH) is general (i.e. can that particular notation express all that the standard covers?) It's not obvious that addition of a "T" between date and time helps any (how does that handle pre-historic dates?). Anyhow, that's just to explain better what I was asking above - I should read the standard myself, of course!

I agree this need not be a disruptive change, but at the same time if we could easily change to a new ISO-compatible cycle time notation before promised backward compatibility kicks in, maybe we could avoid having to support a distinct "classic mode" in the future.

[matthewrmshin]

@hjoliver The basic notation of the date would match this Pythonic regular expression (?P<year>-?\d*\d\d\d\d)(?P<month>\d\d)(?P<day?\d\d). All years will be at least 4 digits, left padded with zeros. E.g. 1BC is represented as +0000, 2BC is -0001, etc. The T helps because without it you are only dealing with a date. With it you are dealing with a date and time. (Hence, a digit-only string is always a date.)

[m214089]

The original standard can be found at:

http://dotat.at/tmp/ISO_8601-2004_E.pdf

Useful ISO 6801 features from my point of view:

Designators

P is time-interval designator
R is recurring time-interval designator
T indicates start of Time elements
Z is UTC designator

Duration designators

Y, M, D, H, M, S may be used
M can be minute, month, or both

Calendar date

year, month, day number within month

Maybe ordinal date

 year, day number within year

Representing dates

Date Basic Format Extended Format
Calendar date YYYYMMDD YYYY-MM-DD
19850412 1985-04-12
Expanded ±YYYYYYMMDD ±YYYYYY-MM-DD
±0019850412 ±001985-04-12
Maybe ordinal date YYYYDDD YYYY-DDD
1985102 1985-102

Representing times

Time Basic Format Extended Format
Local time hhmmss hh:mm:ss
232050 23:20:50
Milli seconds hhmmss.sss hh:mm:ss.sss
232050.500 23:20:50.500

For universal time (UT) a Z should be required at the end of the time string. Are timezones required?
Combined date and time

T indicates start of time.

Calendar dates + UT time: YYYY-MM-DDThh:mmZ
Maybe ordinal dates + UT time: YYYY-DDDThh:mmZ

Representing time-intervals

P indicates a duration (period). Intervals can come in four different varieties:

• start and end time: YYYY-MM-DDThh:mm:ss/YYYY-MM-DDThh:mm:ss
• duration: PnYnMnDTnHnMnS - NOTE: PnYnMnD and PTnHnMns are possible shortcuts. If n of a designator is 0,

it can be ommited, if it is trailing.

• start time and duration: YYYY-MM-DDThh:mm:ss/PnYnMnDTnHnMnS
• duration and end time: PnYnMnDTnHnMnS/YYYY-MM-DDThh:mm:ss

Recurring time-intervals

R is used to indicate recurrence. Recurring time intervals are expressed by:

• number of recurrences and start and end times: Rn/YYYY-MM-DDThh:mm:ss/YYYY-MM-DDThh:mm:ss
• number of recurrences and a duration: Rn/PnYnMnDTnHnMnS
• number of recurrences and start time and a duration: Rn/YYYY-MM-DDThh:mm:ss/PnYnMnDTnHnMnS
• number of recurrences and duration and end time: Rn/PnYnMnDTnHnMnS/YYYY-MM-DDThh:mm:ss

If number of recurrences is not provided, then the number is unbounded. For simplicity the implementation is only supporting the second expression for recurring time intervals.

Regex definitions/collection

Regex for date and time so far constructed:

^([\+-]?\d{4}(?!\d{2}\b))((-?)((0[1-9]|1[0-2])(\3([12]\d|0[1-9]|3[01]))?|W([0-4]\d|5[0-2])(-?[1-7])?|(00[1-9]|0[1-9]\d|[12]\d{2}|3([0-5]\d|6[1-6])))([T\s]((([01]\d|2[0-3])((:?)[0-5]\d)?|24\:?00)([\.,]\d+(?!:))?)?(\17[0-5]\d([\.,]\d+)?)?([zZ]|([\+-])([01]\d|2[0-3]):?([0-5]\d)?)?)?)?$

Regex for duration/time intervall so far constructed:

^P(?:\d+Y|Y)?(?:\d+M|M)?(?:\d+D|D)?(?:T(?:\d+H|H)?(?:\d+M|M)?(?:\d+(?:\.\d{1,2})?S|S)?)?$

or

^P(\d+Y)?(\d+M)?(\d+D)?(T(((\d+H)(\d+M)?(\d+(\.\d{1,2})?S)?)|((\d+M)(\d+(\.\d{1,2})?S)?)|((\d+(\.\d{1,2})?S))))?$

Not sure which one to use yet. The regex are for the full standard including business week, which we do not need.
This needs to be cleaned up.

As well I have a set of routines in C for calculating the Julian date to proleptic Gregorian in a pure integer implementation including negative Julian days. The same for pure 360 day years and 365 day years. We would need to convert those to Python. Unfortunately none of the Python date utils is sufficient clean to be used.

I would be happy if you could have a look into the regex. They are not trivial ...

[benfitzpatrick]

We've riffed on this a bit here - please have a look:

https://github.com/cylc/cylc/wiki/ISO-8601

[hjoliver]

Nice, well done!! I'll print it out and read it on the plane tomorrow.

[hjoliver]

[This discussion temporarily went to #887]

[hjoliver]

Some progress:

[hjoliver]

10-minute cycling #68:

[benfitzpatrick]

Very cool

[benfitzpatrick]

Could we use the no-seconds, no-delimiters kind of dump format for the date/times?

[hjoliver]

Could we use the no-seconds, no-delimiters ...

Yes. What I've done so far is pretty rough, there will no doubt be some detail to attend to...

[hjoliver]

I'm closing this as our master branch now does "use ISO standard date-time formats".