ocgis2.0 MFTime requires that the time variable in all files have a calendar attribute

[nilshempelmann]

@bekozi
@koldunovn
@cehbrecht

Here is a test snippet:
https://github.com/bird-house/flyingpigeon/blob/next/scripts/ocgis_indices_simple.py

running with a single file but not with mulitple ones.

$ python ocgis_indices_simple.py
/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/driver/nc.py:129: UserWarning: endian-ness of dtype and endian kwarg do not match, using endian kwarg
ncvar = dataset.createVariable(var.name, dtype, dimensions=dimensions, fill_value=fill_value, **kwargs)
/home/nils/birdhouse/flyingpigeon/scripts/single_file/single_file.nc
Traceback (most recent call last):
File "ocgis_indices_simple.py", line 26, in
output_format='nc').execute()
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/ops/core.py", line 312, in execute
return interp.execute()
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/ops/interpreter.py", line 135, in execute
ret = conv.write()
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/conv/base.py", line 170, in write
for coll in self:
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/conv/base.py", line 120, in iter
for coll in self.colls:
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/ops/engine.py", line 58, in iter
for coll in self.iter_collections():
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/ops/engine.py", line 137, in iter_collections
coll = self.cengine.execute(coll, file_only=self.ops.file_only, tgds=tgds)
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/calc/engine.py", line 73, in execute
tgds_to_use[field.name] = field.time.get_grouping(self.grouping)
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/variable/temporal.py", line 249, in get_grouping
new_bounds, date_parts, repr_dt, dgroups = self.get_grouping_other(grouping)
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/variable/temporal.py", line 478, in get_grouping_other
value_datetime_bounds = self.bounds.value_datetime
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/variable/temporal.py", line 120, in value_datetime
if get_datetime_conversion_state(self.get_value().flatten()[0]):
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/variable/base.py", line 1224, in get_value
self._value = self.get_value()
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/variable/base.py", line 1459, in get_value
self._request_dataset.driver.init_variable_value(self)
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/driver/base.py", line 428, in init_variable_value
value = self.get_variable_value(variable)
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/driver/nc.py", line 50, in get_variable_value
return get_value_from_request_dataset(variable)
File "/home/nils/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis-2.0.0-py2.7.egg/ocgis/driver/nc.py", line 490, in get_value_from_request_dataset
ncvar = MFTime(ncvar)
File "netCDF4/_netCDF4.pyx", line 6104, in netCDF4._netCDF4.MFTime.init (netCDF4/_netCDF4.c:78273)
ValueError: MFTime requires that the time variable in all files have a calendar attribute

[cehbrecht]

i've run the test script with latest from next branch and a fresh install:

$ git clone 
$ cd flyingpigeon
$ make clean install
$ source activate flyingpigeon
$ vim scripts/ocgis_indices_simple.py   # edit path to dataset
$  bin/python scripts/ocgis_indices_simple.py 
/home/Shared/pingu/sandbox/birdhouse/flyingpigeon/single_file/single_file.nc
/home/Shared/pingu/sandbox/birdhouse/flyingpigeon/multi_file/multi_file.nc
/home/pingu/.conda/envs/flyingpigeon/lib/python2.7/site-packages/ocgis/interface/base/crs.py:764: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future.
Check the NumPy 1.11 release notes for more information.
  new_value[0, :, :] = new_row
['/home/Shared/pingu/sandbox/birdhouse/flyingpigeon/TG_AFR-44_MPI-M-MPI-ESM-LR_historical_r1i1p1_MPI-CSC-REMO2009_v1_yr_20010101-20051231.nc']

Looks like it produced something:

$ cdo sinfo /home/Shared/pingu/sandbox/birdhouse/flyingpigeon/TG_AFR-44_MPI-M-MPI-ESM-LR_historical_r1i1p1_MPI-CSC-REMO2009_v1_yr_20010101-20051231.nc
   File format : NetCDF4 classic
    -1 : Institut Source   Steptype Levels Num    Points Num Dtype : Parameter ID
     1 : unknown  unknown  instant       1   1       425   1  F32  : -1            
   Grid coordinates :
     1 : projection               : points=425 (17x25)
                          mapping : rotated_latitude_longitude
                             rlon : 8.8 to 15.84 by 0.44 degrees
                             rlat : 2.2 to 12.76 by 0.44 degrees
   Vertical coordinates :
     1 : surface                  : levels=1
   Time coordinate :  5 steps
     RefTime =  1949-12-01 00:00:00  Units = days  Calendar = proleptic_gregorian  Bounds = true
  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss
  2001-07-01 00:00:00  2002-07-01 00:00:00  2003-07-01 00:00:00  2004-07-01 00:00:00
  2005-07-01 00:00:00
cdo sinfo: Processed 1 variable over 5 timesteps ( 0.00s )

[nilshempelmann]

ocgis is currently set back to ocgis-next v1.4 in flyingpigeon
ocgis2.0 is causing the error

[bekozi]

@cehbrecht Are you okay pulling from master? I'm interested in phasing out the next branch. If it will be complicated, let me know, and we'll keep next alive!

@nilshempelmann This is an MFDataset and MFTime limitation I think. Was this a process that worked in a previous ocgis version/snapshot?

[nilshempelmann]

Running fine with last ocgis-next version.
recieved the error with ocgis=2.0

[bekozi]

Thanks, @nilshempelmann. This sounds similar to NCPP/ocgis#435. The issue results from inconsistent calendar-units on the time bounds variables. Older ocgis versions did not use MFTime on bounds variables so would have missed this.

Is it easy for you to try and ignore the bounds with these datasets? I think a better solution is needed, but this is the first step.

import ocgis

rd = ocgis.RequestDataset(paths)
rd.dimension_map.set_bounds(ocgis.constants.DimensionMapKey.TIME, None)
ops = ocgis.OcgOperations(dataset=rd, ...)

[nilshempelmann]

OK I ll give it a try

[nilshempelmann]

Fixed in next branch with:

import ocgis

rd = ocgis.RequestDataset(paths)
rd.dimension_map.set_bounds(ocgis.constants.DimensionMapKey.TIME, None)
ops = ocgis.OcgOperations(dataset=rd, ...)