ensemble robustness fails for ocgis clipped files

[nilshempelmann]

@bekozi
Ocgis skipps metadata, and following processes are failing

[bekozi]

Was there supposed to be some additional information here? 😄

[nilshempelmann]

still investigating the issue. But didn't fully understand the issue so far
there are some attributes missing after ocgis performance.
here as well an issue:
#156

[nilshempelmann]

@bekozi
Missing information after ocgis clipping:
original rawdata:

tcdf tas_AFR-44_MPI-M-MPI-ESM-LR_historical_r1i1p1_MPI-CSC-REMO2009_v1_sem_200012-200511 {
dimensions:
time = UNLIMITED ; // (20 currently)
rlat = 201 ;
rlon = 194 ;
bnds = 2 ;
vertices = 4 ;
variables:
double time(time) ;
time:bounds = "time_bnds" ;
time:units = "days since 1949-12-01T00:00:00Z" ;
time:calendar = "proleptic_gregorian" ;
time:axis = "T" ;
time:long_name = "time" ;
time:standard_name = "time" ;
double time_bnds(time, bnds) ;
double rlat(rlat) ;
rlat:units = "degrees" ;
rlat:axis = "Y" ;
rlat:long_name = "latitude in rotated pole grid" ;
rlat:standard_name = "grid_latitude" ;
double rlon(rlon) ;
rlon:units = "degrees" ;
rlon:axis = "X" ;
rlon:long_name = "longitude in rotated pole grid" ;
rlon:standard_name = "grid_longitude" ;
int rotated_latitude_longitude ;
rotated_latitude_longitude:grid_mapping_name = "rotated_latitude_longitude" ;
rotated_latitude_longitude:grid_north_pole_latitude = 90. ;
rotated_latitude_longitude:grid_north_pole_longitude = 180. ;
rotated_latitude_longitude:north_pole_grid_longitude = 0. ;
float lat(rlat, rlon) ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude coordinate" ;
lat:units = "degrees_north" ;
lat:bounds = "lat_vertices" ;
float lon(rlat, rlon) ;
lon:standard_name = "longitude" ;
lon:long_name = "longitude coordinate" ;
lon:units = "degrees_east" ;
lon:bounds = "lon_vertices" ;
float lat_vertices(rlat, rlon, vertices) ;
lat_vertices:units = "degrees_north" ;
float lon_vertices(rlat, rlon, vertices) ;
lon_vertices:units = "degrees_east" ;
double height ;
height:units = "m" ;
height:axis = "Z" ;
height:positive = "up" ;
height:long_name = "height" ;
height:standard_name = "height" ;
float tas(time, rlat, rlon) ;
tas:standard_name = "air_temperature" ;
tas:long_name = "Near-Surface Air Temperature" ;
tas:comment = "daily-mean near-surface (usually, 2 meter) air temperature." ;
tas:units = "K" ;
tas:cell_methods = "time: mean" ;
tas:history = "2015-07-16T14:41:08Z altered by CMOR: Treated scalar dimension: 'height'. 2015-07-16T14:41:08Z altered by CMOR: replaced missing value flag (-1e+20) with standard missing value (1e+20)." ;
tas:coordinates = "height lat lon" ;
tas:missing_value = 1.e+20f ;
tas:_FillValue = 1.e+20f ;
tas:associated_files = "gridspecFile: gridspec_atmos_fx_MPI-CSC-REMO2009_historical_r0i0p0.nc" ;
tas:grid_mapping = "rotated_latitude_longitude" ;

and after clipping:

  time = UNLIMITED ; // (120 currently)
    bnds = 2 ;
    rlat = 60 ;
    rlon = 51 ;

variables:
double time(time) ;
time:bounds = "time_bnds" ;
time:axis = "T" ;
time:long_name = "time" ;
time:standard_name = "time" ;
time:calendar = "proleptic_gregorian" ;
time:units = "days since 1949-12-01T00:00:00Z" ;
double time_bnds(time, bnds) ;
time_bnds:calendar = "proleptic_gregorian" ;
time_bnds:units = "days since 1949-12-01T00:00:00Z" ;
double rlat(rlat) ;
rlat:units = "degrees" ;
rlat:axis = "Y" ;
rlat:long_name = "latitude in rotated pole grid" ;
rlat:standard_name = "grid_latitude" ;
double rlon(rlon) ;
rlon:units = "degrees" ;
rlon:axis = "X" ;
rlon:long_name = "longitude in rotated pole grid" ;
rlon:standard_name = "grid_longitude" ;
char rotated_latitude_longitude ;
rotated_latitude_longitude:grid_mapping_name = "rotated_latitude_longitude" ;
rotated_latitude_longitude:north_pole_grid_longitude = 0. ;
rotated_latitude_longitude:grid_north_pole_longitude = 180. ;
rotated_latitude_longitude:grid_north_pole_latitude = 90. ;
rotated_latitude_longitude:proj4 = "" ;
rotated_latitude_longitude:proj4_transform = "+proj=ob_tran +o_proj=latlon +o_lon_p=180.0 +o_lat_p=90.0 +lon_0=180 +ellps=sphere" ;
float tas(time, rlat, rlon) ;
tas:_FillValue = -1.202529e+27f ;
tas:standard_name = "air_temperature" ;
tas:long_name = "Near-Surface Air Temperature" ;
tas:comment = "daily-mean near-surface (usually, 2 meter) air temperature." ;
tas:cell_methods = "time: mean" ;
tas:history = "2015-07-16T14:41:05Z altered by CMOR: Treated scalar dimension: 'height'. 2015-07-16T14:41:05Z altered by CMOR: replaced missing value flag (-1e+20) with standard missing value (1e+20)." ;
tas:coordinates = "height lat lon" ;
tas:missing_value = 1.e+20f ;
tas:associated_files = "gridspecFile: gridspec_atmos_fx_MPI-CSC-REMO2009_historical_r0i0p0.nc" ;
tas:grid_mapping = "rotated_latitude_longitude" ;
tas:units = "K" ;
double lat(rlat, rlon) ;
lat:standard_name = "latitude" ;
lat:long_name = "latitude coordinate" ;
lat:units = "degrees_north" ;
double lon(rlat, rlon) ;
lon:standard_name = "longitude" ;
lon:long_name = "longitude coordinate" ;
lon:units = "degrees_east" ;

the variables lat/lon is allready 'manually added' by flyingpigoen workaround:
https://github.com/bird-house/flyingpigeon/blob/master/flyingpigeon/ocgis_module.py#L240
(should be done within ocgis)

ensemble robustness can still not perform an ensemble mean:
lyingpigeon) nils@dove:~/data/robustness$ cdo ensmean tas_AFR-44_*_CMRCOD.nc ensmean.nc
Warning (cdfScanVarAttributes) : NetCDF: Variable not found - height
Warning (define_all_grids) : Inconsistent grid structure for variable tas!
Warning (cdfScanVarAttributes) : NetCDF: Variable not found - height
Warning (define_all_grids) : Inconsistent grid structure for variable tas!
Warning (cdfScanVarAttributes) : NetCDF: Variable not found - height
Warning (define_all_grids) : Inconsistent grid structure for variable tas!
Warning (cdfScanVarAttributes) : NetCDF: Variable not found - height
Warning (define_all_grids) : Inconsistent grid structure for variable tas!
Warning (cdfScanVarAttributes) : NetCDF: Variable not found - height
Warning (define_all_grids) : Inconsistent grid structure for variable tas!
Warning (cdfScanVarAttributes) : NetCDF: Variable not found - height
Warning (define_all_grids) : Inconsistent grid structure for variable tas!
Warning (cdfScanVarAttributes) : NetCDF: Variable not found - height
Warning (define_all_grids) : Inconsistent grid structure for variable tas!
cdo ensmean: Processed 2570400 values from 7 variables over 840 timesteps ( 0.24s )

Any suggestions?

[nilshempelmann]

@bekozi
The ensemble calculations are performed with python-cdo.
Would it be possible to translate that into ocgis to be conform in the language end reduce the dependency?
here is the cdo containing code:
https://github.com/bird-house/flyingpigeon/blob/master/flyingpigeon/robustness.py

[bekozi]

ensemble robustness can still not perform an ensemble mean:
lyingpigeon) nils@dove:~/data/robustness$ cdo ensmean tas_AFR-44_*_CMRCOD.nc ensmean.nc
Warning (cdfScanVarAttributes) : NetCDF: Variable not found - height

I see the height scalar dimension is getting dropped by OCGIS. Let me look into a fix for this. Will work in the next version, but this is a bug and should be addressed in the current release.

The ensemble calculations are performed with python-cdo. Would it be possible to translate that into ocgis to be conform in the language end reduce the dependency?

The short answer is yes. It looks moderately complex so it would take a bit to incorporate. It sounds like this works provided the variable structure is maintained. Will make a ticket.

[bekozi]

@nilshempelmann: Did you see this? NCPP/ocgis#439. Should be fixed in ocgis.