Issue with reading n5 array

[unidesigner]

• pyn5 version: 1.0.0
• Python version: 3.7.3
• Operating System: Linux

Description

I get an exception when reading a n5 file that I just created with pyn5

What I Did

First create an n5

import pyn5
import numpy as np

root = "test.n5"
dataset = "test"
dtype = "UINT8"
dataset_size = [10, 10, 10]
block_size = [2, 2, 2]

pyn5.create_dataset(root, dataset, dataset_size, block_size, dtype)
n5 = pyn5.open(root, dataset, dtype, False)

pyn5.write(
    n5,
    (np.array([0, 0, 0]), np.array(dataset_size)),
    np.ones(dataset_size),
)

Then, in a new Python session, read the file

import pyn5
import numpy as np
n5 = pyn5.open("test.n5", 'test', "UINT8", True)
arr = pyn5.read(n5, ( np.array([0,0,0]), np.array([5,5,5]) ), dtype=np.uint8 )

I get the traceback

thread '<unnamed>' panicked at 'An error occurred while initializing class SliceBox', /home/pattonw/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.7.0/src/type_object.rs:260:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
fatal runtime error: failed to initiate panic, error 5
Aborted (core dumped)

[unidesigner]

I get the same exception when running the example from https://rust-pyn5.readthedocs.io/en/latest/usage.html#differences-from-h5py

[clbarnes]

I'll take a look at this today. We made a big change to the way wheels are built and deployed to pypi for this release so there might be some issues surrounding that which CI didn't pick up.

[pattonw]

I managed to replicate the behavior pretty easily, and it only seems to be a problem with the version on pypi.
Running our pyn5 tests on the source code works fine. I then tested just building the wheels with maturin build and ran the tests on that which worked. There shouldn't be a difference between the wheels uploaded when using maturin publish and the wheels built with maturin build, so I'm not sure what could be going wrong here.

[clbarnes]

I was going to suggest an install from sdist as a workaround, but I can't do that either... There seems to be a dependency on setuptools, which I guess isn't included by default in the temporary env.

EDIT: Nope that's just the pypa not having their shit together. The entire point of PEP517 is to allow people to install from sdists where the sdist describes how it wants to be built. The entire point of pip install --no-binary is to build stuff from sdists. But pip install --no-binary ignores PEP517 build systems.

[clbarnes]

I then tested just building the wheels with maturin build and ran the tests on that which worked.

I tried this and it failed - for me, maturin develop (which builds the .so from rust and puts it in the package directory) passes the tests fine; maturin build --release (which builds optimised binaries from rust and packages that with the python code to make a wheel), then installing that wheel and trying to run the example, does not.

[clbarnes]

The problem seems to be in the optimised binaries. maturin develop and installing wheels from maturin build are both fine, but if you add --release to either it'll fail. Seems to be a pyo3 issue, so maybe that's fixed in 0.8.0. There's some API breakage since 0.7.0; I'll see if I can fix that.

[pattonw]

Ah, yeah I didn't use the --release tag when I ran maturin build, and I it makes sense that publish uses the optimized version.

[clbarnes]

Bumping pyo3 and numpy(rust) didn't help.

Seems to be related to PyO3/rust-numpy#97 , which was resolved by a new nightly toolchain. The toolchain they specified isn't compatible with pyo3 any more, but nightly-2019-07-19-x86_64-unknown-linux-gnu (the oldest pyo3-compatible compiler) is - there must be a later compiler which breaks it again. That toolchain successfully builds the project, and installing that optimised wheel passes the tests and example code.

If only we could have anticipated the difficulties of relying on a nightly toolchain.

I have little interest in bisecting compiler releases to find the top range so I'll just stick that working version in the docs and travis and call it done. I've kept the bumped pyo3 and numpy(rust) too. Hopefully pyo3 will be able to use a stable compiler soon, but it doesn't look likely (PyO3/pyo3#5 , rust-lang/rust#31844).

[clbarnes]

Fix merged but not released yet

[pattonw]

I released v1.0.1 that should have fixed this.

[unidesigner]

Thanks for the quick fix! This works now for me too.