-
Notifications
You must be signed in to change notification settings - Fork 1
Time hell
Python has a bazillion ways to deal with time. That is the problem.
Because some day you will want to do this, and you will want it in datetime so that you can use, say hvplot. You will also need to have your times straight to catch subtle time issues.
For the CF file, decode_times+decode_cf are not helpful. Details are here: https://gist.github.com/mmartini-usgs/6d6ebd5dab437928b20b25b0b143c0cf
puv = xr.open_dataset(data_path+puv_file, decode_times=False)
print('puv time units {}'.format(puv['time'].units))
buf = puv['time'].units.split()
t0 = datetime.datetime.strptime(buf[2]+' '+buf[3], '%Y-%m-%d %H:%M:%S.%f')
dt = pd.to_timedelta(puv['time'][:].values,unit='s')
puv_t = t0+dt
print(puv_t[0])
print(type(puv_t[0]))
For the EPIC file, with time, time2, where epic_to_datetime can be found here: https://github.com/dnowacki-usgs/stglib
wmn = xr.open_dataset(data_path+wavesmon_file, decode_times=False)
print('wmn time units {}'.format(wmn['time'].units))
print(type(wmn['time'][0].values))
print(wmn['time'][0].values)
wmn_t = epic_to_datetime(wmn['time'][:].values,wmn['time2'][:].values)
print(wmn_t[0])
print(type(wmn_t[0]))
Use case #2 Converting time strings and other time formats I commonly use (and forget, because there are so many flavors):
import netCDF4 as nc
ds = nc.Dataset(datapath+infileroot,'r')
cft = nc.num2date(ds['time'][:],ds['time'].units)
print(f'first time stamp in file {cft[0]}')
print(type(cft[0])) # returns datetime.datetime
This is done all the time and there's information all over the internet about it.
datetime.datetime.strptime(burst_date,'%Y-%m-%d %H:%M:%S')
The make up of the format string, and all the specifications, though, are not so easy to find. Here is one place that has a good listing: https://www.tutorialspoint.com/python/time_strptime.htm
From use case #1, we have out time series time stamps in datetime.datetime objects, the thing to remember here is that idx_date is a tuple of ndarrays. That drove me nuts for a while.
import datetime
time_to_find = datetime.datetime.strptime(burst_date,'%Y-%m-%d %H:%M:%S')
idx_date = np.where(cft[:] >= time_to_find)
print(f'{cft[idx_date[0][0]]} is at {idx_date[0][0]}')
burst_number = idx_date[0][0]
This was a recent run-around on pangeo and my laptop. Open a very simple netCDF file as follows: ds = xr.open_dataset(ncfile.open(), decode_cf=True, use_cftime=True) and try to plot with matplot lib and get float argument must be a string or a number, not 'cftime._cftime.DatetimeProlepticGregorian' error.
Full code is in this gist: https://nbviewer.jupyter.org/gist/rsignell-usgs/dc4dc10a7a0ef0dcb911c19c04a7bffd.
It seemed to all come down to a difference of version of libnetcdf, laptop is 4.6.2, pangeo is 4.7.4 and/or the fact that nc-time-axis is required in the later version of libnetcdf. Installing nc-time-axis in my pangeo setup fixed the problem.
- Some data manipulation examples from netCDF4: https://www.programcreek.com/python/example/89490/netCDF4.date2num