Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

h5py=3.1 upgrade causes test_FormatHDF5EigerNearlyNexusSPring8 failure #267

Closed
rjgildea opened this issue Dec 14, 2020 · 6 comments
Closed

Comments

@rjgildea
Copy link
Contributor

With the previous h5py=2.10.0 this test passes as expected:

$ dials.python -c "import h5py; print(h5py.__version__)"
2.10.0
$ pytest tests/format/test_FormatHDF5EigerNearlyNexusSPring8.py --regression -vs
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.8.6, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /dls/science/users/hko55533/software/cctbx/build/../conda_base/bin/python3.8
cachedir: .pytest_cache
rootdir: /dls/science/users/hko55533/software/cctbx/modules/dxtbx, configfile: pytest.ini
plugins: dials-data-2.1.30, forked-1.2.0, mock-1.13.0, xdist-2.1.0, cov-2.10.1
collected 1 item                                                                                                                                                                                                  

tests/format/test_FormatHDF5EigerNearlyNexusSPring8.py::test_spring8_ccp4_2018_zenodo_1443110_data03 PASSED

After upgrading h5py to 3.1:

$ conda install h5py=3.1
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /dls/science/users/hko55533/software/cctbx/conda_base

  added / updated specs:
    - h5py=3.1


The following packages will be UPDATED:

  h5py                        2.10.0-nompi_py38h7442b35_105 --> 3.1.0-nompi_py38hafa665b_100

this test now fails with the following error:


    def test_spring8_ccp4_2018_zenodo_1443110_data03():
        # https://zenodo.org/record/1443110#.XD8bD5ynzmE
        master_h5 = "/dls/science/groups/scisoft/DIALS/zenodo/1443110/ccp4school2018_bl41xu/05/data03/data03_master.h5"
        assert FormatHDF5EigerNearlyNexusSPring8.understand(master_h5)
    
>       expts = ExperimentListFactory.from_filenames([master_h5])

tests/format/test_FormatHDF5EigerNearlyNexusSPring8.py:27: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
model/experiment_list.py:525: in from_filenames
    for db in DataBlockFactory.from_filenames(
datablock.py:1489: in from_filenames
    importer = DataBlockFilenameImporter(
datablock.py:934: in __init__
    imageset = self._create_single_file_imageset(
datablock.py:1153: in _create_single_file_imageset
    return format_class.get_imageset(abspath(filename), format_kwargs=format_kwargs)
format/FormatMultiImage.py:147: in get_imageset
    format_instance = cls(filenames[0], **format_kwargs)
format/FormatHDF5.py:15: in __init__
    Format.__init__(self, image_file, **kwargs)
format/Format.py:137: in __init__
    self.setup()
format/Format.py:147: in setup
    self._start()
format/FormatHDF5EigerNearlyNexusSPring8.py:26: in _start
    super(FormatHDF5EigerNearlyNexusSPring8, self)._start()
format/FormatHDF5EigerNearlyNexus.py:366: in _start
    self._goniometer_model = GoniometerFactory(sample).model
format/nexus.py:1118: in __init__
    axes, angles, axis_names, scan_axis = construct_axes(
format/nexus.py:339: in construct_axes
    visitor.visit(nx_file, item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <dxtbx.format.nexus.construct_axes.<locals>.Visitor object at 0x7f0b8cebfeb0>, nx_file = <HDF5 file "tmp_master_0ab6ed0a3dea11eb82d5989096dc1b01.nxs" (mode r+)>
depends_on = '/entry/sample/transformations/omega'

    def visit(self, nx_file, depends_on):
        item = nx_file[depends_on]
        value = item[()]
        units = item.attrs["units"]
        ttype = item.attrs["transformation_type"]
        vector = [float(v) for v in item.attrs["vector"]]
        if ttype == numpy.string_("translation"):
            return
        elif ttype == numpy.string_("rotation"):
            if hasattr(value, "__iter__") and len(value):
                value = value[0]
            if units == numpy.string_("rad"):
                value *= 180 / math.pi
            elif units not in [
                numpy.string_("deg"),
                numpy.string_("degree"),
                numpy.string_("degrees"),
            ]:
                raise RuntimeError("Invalid units: %s" % units)
    
            # is the axis moving? Check the values for this axis
            v = item[()]
            if min(v) < max(v):
                is_scan_axis = True
            else:
                is_scan_axis = False
    
            # Is different coordinate system called mcstas
            # Rotate 180 about up if memory serves
            axis_name = item.name.split("/")[-1]
            self._axes.append(vector)
            self._angles.append(float(value))
            self._axis_names.append(str(axis_name))
            self._is_scan_axis.append(is_scan_axis)
    
        else:
>           raise RuntimeError("Unknown transformation_type: %s" % ttype)
E           RuntimeError: Unknown transformation_type: rotation

format/nexus.py:305: RuntimeError
@graeme-winter
Copy link
Collaborator

Does #263 fix it?

@graeme-winter
Copy link
Collaborator

Also humbly & respectfully note that this was my fault, should have run more comprehensive tests before putting the 3.1 PR change in.

@rjgildea
Copy link
Contributor Author

Checking out the branch in #263 results in a different error:

    def test_spring8_ccp4_2018_zenodo_1443110_data03():
        # https://zenodo.org/record/1443110#.XD8bD5ynzmE
        master_h5 = "/dls/science/groups/scisoft/DIALS/zenodo/1443110/ccp4school2018_bl41xu/05/data03/data03_master.h5"
        assert FormatHDF5EigerNearlyNexusSPring8.understand(master_h5)
    
>       expts = ExperimentListFactory.from_filenames([master_h5])

tests/format/test_FormatHDF5EigerNearlyNexusSPring8.py:27: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
model/experiment_list.py:525: in from_filenames
    for db in DataBlockFactory.from_filenames(
datablock.py:1489: in from_filenames
    importer = DataBlockFilenameImporter(
datablock.py:934: in __init__
    imageset = self._create_single_file_imageset(
datablock.py:1153: in _create_single_file_imageset
    return format_class.get_imageset(abspath(filename), format_kwargs=format_kwargs)
format/FormatMultiImage.py:147: in get_imageset
    format_instance = cls(filenames[0], **format_kwargs)
format/FormatHDF5.py:15: in __init__
    Format.__init__(self, image_file, **kwargs)
format/Format.py:137: in __init__
    self.setup()
format/Format.py:147: in setup
    self._start()
format/FormatHDF5EigerNearlyNexusSPring8.py:26: in _start
    super(FormatHDF5EigerNearlyNexusSPring8, self)._start()
format/FormatHDF5EigerNearlyNexus.py:366: in _start
    self._goniometer_model = GoniometerFactory(sample).model
format/nexus.py:1118: in __init__
    axes, angles, axis_names, scan_axis = construct_axes(
format/nexus.py:339: in construct_axes
    visitor.visit(nx_file, item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <dxtbx.format.nexus.construct_axes.<locals>.Visitor object at 0x7f7d4dc3c3a0>, nx_file = <HDF5 file "tmp_master_3dddf71e3def11ebba47989096dc1b01.nxs" (mode r+)>
depends_on = '/entry/sample/transformations/omega'

    def visit(self, nx_file, depends_on):
        item = nx_file[depends_on]
        value = item[()]
        units = item.attrs["units"]
        ttype = item.attrs["transformation_type"]
        vector = [float(v) for v in item.attrs["vector"]]
        if numpy.string_(ttype) == numpy.string_("translation"):
            return
        elif numpy.string_(ttype) == numpy.string_("rotation"):
            if hasattr(value, "__iter__") and len(value):
                value = value[0]
            if numpy.string_(units) == numpy.string_("rad"):
                value *= 180 / math.pi
            elif units not in [
                numpy.string_("deg"),
                numpy.string_("degree"),
                numpy.string_("degrees"),
            ]:
>               raise RuntimeError("Invalid units: %s" % units)
E               RuntimeError: Invalid units: degree

format/nexus.py:287: RuntimeError

@ndevenish
Copy link
Collaborator

Also humbly & respectfully note that this was my fault, should have run more comprehensive tests before putting the 3.1 PR change in.

No, I think what you did was reasonable. This test only works at diamond

def test_spring8_ccp4_2018_zenodo_1443110_data03():
# https://zenodo.org/record/1443110#.XD8bD5ynzmE
master_h5 = "/dls/science/groups/scisoft/DIALS/zenodo/1443110/ccp4school2018_bl41xu/05/data03/data03_master.h5"

@rjgildea
Copy link
Contributor Author

I think this test should be able to use dials.data now:
dials/data#11
dials/data#192

@ndevenish
Copy link
Collaborator

Okay, so #263 fixes this error now, but not yet dials/dials#1523

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants