Issue with new indexing tool

[dwayne-hart]

Tried to index the RIOPS forecast 2D latlon data from SeDOO using a carefully crafted --file-list.

(navigator) buildadm@u1604-on-production-b:~/ocean-nav/db$ ls /data/eccc_forecasts_netcdf/riops_forecast/2D/LL/ | grep -P "2019.*(00|06|12|18)_00[0-5]_2D_latlon0.1x0.1.nc" | awk '{print "/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/"$1}' > RIOPS-FC-2D-LL.txt

Next I then ran the new version of the indexing tool

(navigator) buildadm@u1604-on-production-b:~/ocean-nav/db$ ./nc-timestamp-mapper -n RIOPS-FC-2D-LL -i /data/eccc_forecasts_netcdf/riops_forecast/2D/LL -o /home/buildadm/ocean-nav/db -h --file-list RIOPS-FC-2D-LL.txt

I then configured the datasetconfig.json file to include;

"riops_forecast": {
    "url": "/home/buildadm/ocean-nav/db/RIOPS-FC-2D-LL.sqlite3",
    "name": "RIOPS Forecast",
    "quantum": "hour",
    "type": "forecast",
    "enabled": true,
    "time_dim_units": "seconds since 1950-01-01 00:00:00",
    "climatology": "http://10.0.3.56:8080/thredds/dodsC/climatology/Levitus98_PHC21/aggregated.ncml",
    "attribution": "RIOPS Forecast from CONCEPTS",
    "cache": 1,
    "help": "Regional Ice Ocean Prediction System�        <ul>�            <li>North Atlantic and Arctic</li>�            <li>Tri-polar ORCA grid 1/12° resolution</li>�            <li>50 vertical z-levels</li>�            <li>Available as daily values on 3 hr intervals from 00:00 (May 28, 2016 to present)</li>�            <li>Variables Available:�                <ul>�                    <li>Depth-Integrated Sea Ice Strength</li>�                    <li>Depth-Integrated Internal Sea Ice Pressure</li>�                    <li>Ocean Mixed Layer Depth Based on Density Criterion</li>�                    <li>Sea Ice Area Fraction</li>�                    <li>Sea Ice Volume Per Unit Grid Cell Area</li>�                    <li>Sea Ice Velocity</li>�                    <li>Sea Ice East Velocity</li>�                    <li>Sea Ice North Velocity</li>�                    <li>Sea Ice X Velocity</li>�                      <ul>                                                   �                          <li>ice velocity along model x grid lines</li>�                      </ul>�                    <li>Sea Ice Y Velocity</li>�                      <ul>�                          <li>ice velocity along model y grid lines</li>�                      </ul>�                    <li>Sea Surface Height Above Geoid</li>�                    <li>Sea Water Potential Temperature</li>�                    <li>Sea Water Salinity</li>�                    <li>Sea Water Velocity</li>�                    <li>Sea Water East Velocity</li>�                    <li>Sea Water North Velocity</li>�                    <li>Sea Water X Velocity</li>�                      <ul>�                         <li>water velocity along model x grid lines</li>�                      </ul>�                    <li>Sea Water Y Velocity</li>�                      <ul>�                         <li>water velocity along model y grid lines</li>�                      </ul>�                    <li>Surface Temperature Of Snow Over Sea Ice Or Bare Sea Ice</li>�                    <li>Turbocline Depth</li>�                    <li>Volume Of Snow On Sea Ice Per Unit Grid Cell Area</li>�                </ul>�            </li>�        </ul>",

        "variables": {
            "vozocrtx": { "name": "Water X Velocity", "unit": "m/s", "scale": [-3, 3], "zero_centered": "true" },
            "vomecrty": { "name": "Water Y Velocity", "unit": "m/s", "scale": [-3, 3], "zero_centered": "true" },
            "vozocrtx,vomecrty": { "name": "Water Velocity", "unit": "m/s", "scale": [0, 3] },
            "votemper": { "name": "Temperature", "unit": "Celsius", "scale": [-5, 30], "equation": "votemper - 273.15", "dims": ["time", "depth", "yc", "xc"] },
            "vosaline": { "name": "Salinity", "unit": "PSU", "scale": [30, 40] },
            "sossheig": { "name": "Sea Surface Height", "unit": "m", "scale": [-3, 3], "zero_centered": "true" },
            "iiceconc": { "name": "Ice Concentration", "unit": "fraction", "scale": [0, 1] },
            "iicevol": { "name": "Ice Volume", "unit": "m", "scale": [0, 10] },
            "isnowvol": { "name": "Snow Volume", "unit": "cm", "scale": [0, 10] },
            "iicesurftemp": { "name": "Ice/Snow Surface Temperature", "unit": "Celsius", "scale": [-5, 10], "equation": "iicesurftemp - 273.15", "dims": ["time", "depth", "yc", "xc"] },
            "itzocrtx": { "name": "Sea Ice X Velocity", "unit": "m/s", "scale": [-1, 1], "zero_centered": "true" },
            "itmecrty": { "name": "Sea Ice Y Velocity", "unit": "m/s", "scale": [-1, 1], "zero_centered": "true" },
            "itzocrtx,itmecrty": { "name": "Sea Ice Velocity", "unit": "m/s", "scale": [0, 1] },
            "iicepressure": { "name": "Internal Sea Ice Pressure", "unit": "N/m", "scale": [0, 200000] },
            "iicestrength": { "name": "Compressive Sea Ice Strength", "unit": "N/m", "scale": [0, 60000] },
            "somixhgt": { "name": "Turbocline Depth", "unit": "m", "scale": [0, 4000] },
            "sokaraml": { "name": "Ocean Mixed Layer Depth", "unit": "m", "scale": [0, 4000] }
        }
},

I then restarted the runserver.sh script and when I selected the RIOPS Forecast the following error was found...

Traceback (most recent call last):

File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
self.handle_request(listener, req, client, addr)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 191, in handle_request
six.reraise(*sys.exc_info())
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/gunicorn/six.py", line 625, in reraise
raise value
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1997, in call
return self.wsgi_app(environ, start_response)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/compat.py", line 33, in reraise
raise value
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/compat.py", line 33, in reraise
raise value
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/buildadm/ocean-nav/Ocean-Data-Map-Project/routes/api_v1_0.py", line 410, in tile_v1_0
return routes.routes_impl.tile_impl(projection, interp, radius, neighbours, dataset, variable, time, depth, scale, zoom, x, y)
File "/home/buildadm/ocean-nav/Ocean-Data-Map-Project/routes/routes_impl.py", line 566, in tile_impl
'scale': scale,
File "/home/buildadm/ocean-nav/Ocean-Data-Map-Project/plotting/tile.py", line 176, in plot
with open_dataset(config, variable=variable, timestamp=time) as dataset:
File "/home/buildadm/ocean-nav/Ocean-Data-Map-Project/data/mercator.py", line 31, in enter
super(Mercator, self).enter()
File "/home/buildadm/ocean-nav/Ocean-Data-Map-Project/data/netcdf_data.py", line 63, in enter
self.url, decode_times=decode_times)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/xarray/backends/api.py", line 321, in open_dataset
filename_or_obj, group=group, lock=lock, **backend_kwargs)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/xarray/backends/netCDF4.py", line 355, in open
return cls(manager, lock=lock, autoclose=autoclose)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/xarray/backends/netCDF4.py", line 314, in init
self.format = self.ds.data_model
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/xarray/backends/netCDF4_.py", line 359, in ds
return self._manager.acquire().value
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/xarray/backends/file_manager.py", line 172, in acquire
file = self._opener(*self.args, **kwargs)
File "/home/sdfo000/ocean-nav/miniconda/3/envs/navigator/lib/python3.6/site-packages/xarray/backends/netCDF4.py", line 247, in _open_netcdf4_group
ds = nc4.Dataset(filename, mode=mode, **kwargs)
File "netCDF4/_netCDF4.pyx", line 2291, in netCDF4._netCDF4.Dataset.init
File "netCDF4/_netCDF4.pyx", line 1855, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -51] NetCDF: Unknown file format: b'/home/buildadm/ocean-nav/db/RIOPS-FC-2D-LL.sqlite3'

The ON code is based on your release after James had provided the review.

[dwayne-hart]

(navigator) buildadm@u1604-on-production-b:~/ocean-nav/db$ file /home/buildadm/ocean-nav/db/RIOPS-FC-2D-LL.sqlite3
/home/buildadm/ocean-nav/db/RIOPS-FC-2D-LL.sqlite3: SQLite 3.x database
(navigator) buildadm@u1604-on-production-b:~/ocean-nav/db$ ls -l /home/buildadm/ocean-nav/db/RIOPS-FC-2D-LL.sqlite3
-rw-r--r-- 1 buildadm buildadm 1107968 Nov 21 22:27 /home/buildadm/ocean-nav/db/RIOPS-FC-2D-LL.sqlite3
(navigator) buildadm@u1604-on-production-b:~/ocean-nav/db$ stat /home/buildadm/ocean-nav/db/RIOPS-FC-2D-LL.sqlite3
  File: '/home/buildadm/ocean-nav/db/RIOPS-FC-2D-LL.sqlite3'
  Size: 1107968         Blocks: 2168       IO Block: 4096   regular file
Device: fc02h/64514d    Inode: 8659119677  Links: 1
Access: (0644/-rw-r--r--)  Uid: (  501/buildadm)   Gid: (  501/buildadm)
Access: 2019-11-21 22:30:03.885855863 -0400
Modify: 2019-11-21 22:27:31.888116202 -0400
Change: 2019-11-21 22:27:31.888116202 -0400
 Birth: -

The file list that I had created had the following sample layout...

(navigator) buildadm@u1604-on-production-b:~/ocean-nav/db$ head -10  RIOPS-FC-2D-LL.txt
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100700_000_2D_latlon0.1x0.1.nc
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100700_001_2D_latlon0.1x0.1.nc
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100700_002_2D_latlon0.1x0.1.nc
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100700_003_2D_latlon0.1x0.1.nc
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100700_004_2D_latlon0.1x0.1.nc
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100700_005_2D_latlon0.1x0.1.nc
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100706_000_2D_latlon0.1x0.1.nc
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100706_001_2D_latlon0.1x0.1.nc
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100706_002_2D_latlon0.1x0.1.nc
/data/eccc_forecasts_netcdf/riops_forecast/2D/LL/2019100706_003_2D_latlon0.1x0.1.nc

[nsoontie]

@dwayne-hart Are you sure these files have a depth dimension? In the datasetconfig.json you have votemper, iicesurftemp with "dims": ["time", "depth", "yc", "xc"]. Could that be causing the problem given that these are 2D files?

ncdump -h on the files will tell you the names of the dimensions.