Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

importlib_resources/tests/test_functional.py test failures on big endian architectures #312

Closed
mgorny opened this issue Jul 10, 2024 · 1 comment
Closed

importlib_resources/tests/test_functional.py test failures on big endian architectures #312

mgorny opened this issue Jul 10, 2024 · 1 comment

Comments

@mgorny
Copy link

mgorny commented Jul 10, 2024

When running the tests on big-endian architectures (sparc64, ppc64), we're getting the followin test failures:

$ tox -e py312 
.pkg-cpython312: recreate env because dependencies removed: setuptools>=56
.pkg-cpython312: remove tox env folder /home/mgorny/importlib_resources/.tox/.pkg-cpython312
.pkg-cpython312: install_requires> python -I -m pip install 'setuptools>=61.2' 'setuptools_scm[toml]>=3.4.1'
.pkg-cpython312: _optional_hooks> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg-cpython312: get_requires_for_build_editable> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg-cpython312: build_editable> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py312: recreate env because dependencies removed: pytest-mypy; platform_python_implementation != "PyPy", pytest>=6
py312: remove tox env folder /home/mgorny/importlib_resources/.tox/py312
py312: install_package_deps> python -I -m pip install 'jaraco.test>=5.4' 'pytest!=8.1.*,>=6' 'pytest-checkdocs>=2.4' pytest-cov 'pytest-enabler>=2.2' pytest-mypy 'pytest-ruff>=0.2.1' 'zipp>=3.1.0; python_version < "3.10"' 'zipp>=3.17'
py312: install_package> python -I -m pip install --force-reinstall --no-deps /home/mgorny/importlib_resources/.tox/.tmp/package/10/importlib_resources-6.4.1.dev23+gbe5a270-0.editable-py3-none-any.whl
py312: commands[0]> pytest
========================================================= test session starts =========================================================
platform linux -- Python 3.12.3, pytest-8.2.2, pluggy-1.5.0
cachedir: .tox/py312/.pytest_cache
rootdir: /home/mgorny/importlib_resources
configfile: pytest.ini
plugins: enabler-3.1.1, mypy-0.10.3, checkdocs-2.13.0, cov-5.0.0, jaraco.test-5.4.0, ruff-0.4.0, typeguard-4.3.0
collected 313 items                                                                                                                   

docs/conf.py ....                                                                                                               [  1%]
importlib_resources/__init__.py ...                                                                                             [  2%]
importlib_resources/_adapters.py ...                                                                                            [  3%]
importlib_resources/_common.py ....                                                                                             [  4%]
importlib_resources/_functional.py ...                                                                                          [  5%]
importlib_resources/_itertools.py ....                                                                                          [  6%]
importlib_resources/abc.py ...                                                                                                  [  7%]
importlib_resources/compat/__init__.py ...                                                                                      [  8%]
importlib_resources/compat/py38.py ...                                                                                          [  9%]
importlib_resources/compat/py39.py ...                                                                                          [ 10%]
importlib_resources/future/__init__.py ...                                                                                      [ 11%]
importlib_resources/future/adapters.py ...                                                                                      [ 12%]
importlib_resources/readers.py ...                                                                                              [ 13%]
importlib_resources/simple.py ...                                                                                               [ 14%]
importlib_resources/tests/__init__.py ...                                                                                       [ 15%]
importlib_resources/tests/_path.py ....                                                                                         [ 16%]
importlib_resources/tests/compat/__init__.py ...                                                                                [ 17%]
importlib_resources/tests/compat/py312.py ...                                                                                   [ 18%]
importlib_resources/tests/compat/py39.py ...                                                                                    [ 19%]
importlib_resources/tests/data01/__init__.py ...                                                                                [ 20%]
importlib_resources/tests/data01/subdirectory/__init__.py ...                                                                   [ 21%]
importlib_resources/tests/data02/__init__.py ...                                                                                [ 22%]
importlib_resources/tests/data02/one/__init__.py ...                                                                            [ 23%]
importlib_resources/tests/data02/two/__init__.py ...                                                                            [ 24%]
importlib_resources/tests/test_compatibilty_files.py ...................                                                        [ 30%]
importlib_resources/tests/test_contents.py ......                                                                               [ 32%]
importlib_resources/tests/test_custom.py ....                                                                                   [ 33%]
importlib_resources/tests/test_files.py .........................                                                               [ 41%]
importlib_resources/tests/test_functional.py .......F..F.....F..F.                                                              [ 48%]
importlib_resources/tests/test_open.py ...........................................                                              [ 61%]
importlib_resources/tests/test_path.py ................                                                                         [ 67%]
importlib_resources/tests/test_read.py .......................................                                                  [ 79%]
importlib_resources/tests/test_reader.py ..................                                                                     [ 85%]
importlib_resources/tests/test_resource.py ....................s..................                                              [ 97%]
importlib_resources/tests/util.py ...                                                                                           [ 98%]
importlib_resources/tests/zip.py ...                                                                                            [ 99%]
. .                                                                                                                             [100%]

============================================================== FAILURES ===============================================================
____________________________________________ FunctionalAPITest_StringAnchor.test_open_text ____________________________________________

self = <importlib_resources.tests.test_functional.FunctionalAPITest_StringAnchor testMethod=test_open_text>

    def test_open_text(self):
        with resources.open_text(self.anchor01, 'utf-8.file') as f:
            self.assertEqual(f.read(), 'Hello, UTF-8 world!\n')
        for path_parts in self._gen_resourcetxt_path_parts():
            with resources.open_text(
                self.anchor02,
                *path_parts,
                encoding='utf-8',
            ) as f:
                self.assertEqual(f.read(), 'a resource')
        # Use generic OSError, since e.g. attempting to read a directory can
        # fail with PermissionError rather than IsADirectoryError
        with self.assertRaises(OSError):
            resources.open_text(self.anchor01)
        with self.assertRaises(OSError):
            resources.open_text(self.anchor01, 'no-such-file')
        with resources.open_text(self.anchor01, 'utf-16.file') as f:
            with self.assertRaises(UnicodeDecodeError):
                f.read()
        with resources.open_text(
            self.anchor01,
            'binary.file',
            encoding='latin1',
        ) as f:
            self.assertEqual(f.read(), '\x00\x01\x02\x03')
        with resources.open_text(
            self.anchor01,
            'utf-16.file',
            errors='backslashreplace',
        ) as f:
>           self.assertEqual(
                f.read(),
                'Hello, UTF-16 world!\n'.encode('utf-16').decode(
                    errors='backslashreplace',
                ),
            )
E           AssertionError: '\\xff\\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00[67 chars]\x00' != '\\xfe\\xff\x00H\x00e\x00l\x00l\x00o\x00,\x00 [67 chars]00\n'
E           - \xff\xfeHello, UTF-16 world!
E           ?     ^^^^
E           + \xfe\xffHello, UTF-16 world!
E           ? ++++    ^
E           - 
E           +

importlib_resources/tests/test_functional.py:131: AssertionError
____________________________________________ FunctionalAPITest_StringAnchor.test_read_text ____________________________________________

self = <importlib_resources.tests.test_functional.FunctionalAPITest_StringAnchor testMethod=test_read_text>

    def test_read_text(self):
        self.assertEqual(
            resources.read_text(self.anchor01, 'utf-8.file'),
            'Hello, UTF-8 world!\n',
        )
        self.assertEqual(
            resources.read_text(
                self.anchor02,
                'subdirectory',
                'subsubdir',
                'resource.txt',
                encoding='utf-8',
            ),
            'a resource',
        )
        for path_parts in self._gen_resourcetxt_path_parts():
            self.assertEqual(
                resources.read_text(
                    self.anchor02,
                    *path_parts,
                    encoding='utf-8',
                ),
                'a resource',
            )
        # Use generic OSError, since e.g. attempting to read a directory can
        # fail with PermissionError rather than IsADirectoryError
        with self.assertRaises(OSError):
            resources.read_text(self.anchor01)
        with self.assertRaises(OSError):
            resources.read_text(self.anchor01, 'no-such-file')
        with self.assertRaises(UnicodeDecodeError):
            resources.read_text(self.anchor01, 'utf-16.file')
        self.assertEqual(
            resources.read_text(
                self.anchor01,
                'binary.file',
                encoding='latin1',
            ),
            '\x00\x01\x02\x03',
        )
>       self.assertEqual(
            resources.read_text(
                self.anchor01,
                'utf-16.file',
                errors='backslashreplace',
            ),
            'Hello, UTF-16 world!\n'.encode('utf-16').decode(
                errors='backslashreplace',
            ),
        )
E       AssertionError: '\\xff\\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00[67 chars]\x00' != '\\xfe\\xff\x00H\x00e\x00l\x00l\x00o\x00,\x00 [67 chars]00\n'
E       - \xff\xfeHello, UTF-16 world!
E       ?     ^^^^
E       + \xfe\xffHello, UTF-16 world!
E       ? ++++    ^
E       - 
E       +

importlib_resources/tests/test_functional.py:79: AssertionError
____________________________________________ FunctionalAPITest_ModuleAnchor.test_open_text ____________________________________________

self = <importlib_resources.tests.test_functional.FunctionalAPITest_ModuleAnchor testMethod=test_open_text>

    def test_open_text(self):
        with resources.open_text(self.anchor01, 'utf-8.file') as f:
            self.assertEqual(f.read(), 'Hello, UTF-8 world!\n')
        for path_parts in self._gen_resourcetxt_path_parts():
            with resources.open_text(
                self.anchor02,
                *path_parts,
                encoding='utf-8',
            ) as f:
                self.assertEqual(f.read(), 'a resource')
        # Use generic OSError, since e.g. attempting to read a directory can
        # fail with PermissionError rather than IsADirectoryError
        with self.assertRaises(OSError):
            resources.open_text(self.anchor01)
        with self.assertRaises(OSError):
            resources.open_text(self.anchor01, 'no-such-file')
        with resources.open_text(self.anchor01, 'utf-16.file') as f:
            with self.assertRaises(UnicodeDecodeError):
                f.read()
        with resources.open_text(
            self.anchor01,
            'binary.file',
            encoding='latin1',
        ) as f:
            self.assertEqual(f.read(), '\x00\x01\x02\x03')
        with resources.open_text(
            self.anchor01,
            'utf-16.file',
            errors='backslashreplace',
        ) as f:
>           self.assertEqual(
                f.read(),
                'Hello, UTF-16 world!\n'.encode('utf-16').decode(
                    errors='backslashreplace',
                ),
            )
E           AssertionError: '\\xff\\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00[67 chars]\x00' != '\\xfe\\xff\x00H\x00e\x00l\x00l\x00o\x00,\x00 [67 chars]00\n'
E           - \xff\xfeHello, UTF-16 world!
E           ?     ^^^^
E           + \xfe\xffHello, UTF-16 world!
E           ? ++++    ^
E           - 
E           +

importlib_resources/tests/test_functional.py:131: AssertionError
____________________________________________ FunctionalAPITest_ModuleAnchor.test_read_text ____________________________________________

self = <importlib_resources.tests.test_functional.FunctionalAPITest_ModuleAnchor testMethod=test_read_text>

    def test_read_text(self):
        self.assertEqual(
            resources.read_text(self.anchor01, 'utf-8.file'),
            'Hello, UTF-8 world!\n',
        )
        self.assertEqual(
            resources.read_text(
                self.anchor02,
                'subdirectory',
                'subsubdir',
                'resource.txt',
                encoding='utf-8',
            ),
            'a resource',
        )
        for path_parts in self._gen_resourcetxt_path_parts():
            self.assertEqual(
                resources.read_text(
                    self.anchor02,
                    *path_parts,
                    encoding='utf-8',
                ),
                'a resource',
            )
        # Use generic OSError, since e.g. attempting to read a directory can
        # fail with PermissionError rather than IsADirectoryError
        with self.assertRaises(OSError):
            resources.read_text(self.anchor01)
        with self.assertRaises(OSError):
            resources.read_text(self.anchor01, 'no-such-file')
        with self.assertRaises(UnicodeDecodeError):
            resources.read_text(self.anchor01, 'utf-16.file')
        self.assertEqual(
            resources.read_text(
                self.anchor01,
                'binary.file',
                encoding='latin1',
            ),
            '\x00\x01\x02\x03',
        )
>       self.assertEqual(
            resources.read_text(
                self.anchor01,
                'utf-16.file',
                errors='backslashreplace',
            ),
            'Hello, UTF-16 world!\n'.encode('utf-16').decode(
                errors='backslashreplace',
            ),
        )
E       AssertionError: '\\xff\\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00[67 chars]\x00' != '\\xfe\\xff\x00H\x00e\x00l\x00l\x00o\x00,\x00 [67 chars]00\n'
E       - \xff\xfeHello, UTF-16 world!
E       ?     ^^^^
E       + \xfe\xffHello, UTF-16 world!
E       ? ++++    ^
E       - 
E       +

importlib_resources/tests/test_functional.py:79: AssertionError
========================================================== warnings summary ===========================================================
docs/conf.py::mypy
  /home/mgorny/importlib_resources/.tox/py312/lib/python3.12/site-packages/mypy/config_parser.py:271: EncodingWarning: 'encoding' argument not specified
    config_parser.read(config_file)

docs/conf.py::mypy
  /home/mgorny/importlib_resources/.tox/py312/lib/python3.12/site-packages/mypy/modulefinder.py:851: EncodingWarning: 'encoding' argument not specified
    with open(versions_path) as f:

docs/conf.py::mypy
docs/conf.py::mypy
docs/conf.py::mypy
docs/conf.py::mypy
  /home/mgorny/importlib_resources/.tox/py312/lib/python3.12/site-packages/mypy/metastore.py:95: EncodingWarning: 'encoding' argument not specified
    with open(os.path.join(self.cache_dir_prefix, name)) as f:

docs/conf.py: 13 warnings
  /home/mgorny/importlib_resources/.tox/py312/lib/python3.12/site-packages/mypy/metastore.py:108: EncodingWarning: 'encoding' argument not specified
    with open(tmp_filename, "w") as f:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

---------- coverage: platform linux, python 3.12.3-final-0 -----------
Name                                                        Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------------------------
docs/conf.py                                                   12      0   100%
importlib_resources/__init__.py                                 4      0   100%
importlib_resources/_adapters.py                               90      2    98%   58, 157
importlib_resources/_common.py                                101      5    95%   68-72
importlib_resources/_functional.py                             32      0   100%
importlib_resources/abc.py                                     66      8    88%   27, 40, 48, 53, 162, 165, 168, 171
importlib_resources/compat/__init__.py                          0      0   100%
importlib_resources/compat/py38.py                              6      1    83%   11
importlib_resources/compat/py39.py                              5      1    80%   10
importlib_resources/future/__init__.py                          0      0   100%
importlib_resources/future/adapters.py                         42      3    93%   37, 83-84
importlib_resources/readers.py                                106      9    92%   29, 42-45, 52-53, 188-194
importlib_resources/tests/__init__.py                           0      0   100%
importlib_resources/tests/compat/__init__.py                    0      0   100%
importlib_resources/tests/compat/py39.py                        3      0   100%
importlib_resources/tests/compat/py312.py                       9      4    56%   11-15
importlib_resources/tests/data01/__init__.py                    0      0   100%
importlib_resources/tests/data01/subdirectory/__init__.py       0      0   100%
importlib_resources/tests/data02/__init__.py                    0      0   100%
importlib_resources/tests/data02/one/__init__.py                0      0   100%
importlib_resources/tests/data02/two/__init__.py                0      0   100%
importlib_resources/tests/test_compatibilty_files.py           67      0   100%
importlib_resources/tests/test_contents.py                     19      0   100%
importlib_resources/tests/test_custom.py                       29      0   100%
importlib_resources/tests/test_files.py                        67      1    99%   117
importlib_resources/tests/test_functional.py                  109      2    98%   7-9
importlib_resources/tests/test_open.py                         58      1    98%   89
importlib_resources/tests/test_path.py                         37      1    97%   65
importlib_resources/tests/test_read.py                         51      1    98%   97
importlib_resources/tests/test_reader.py                       94      1    99%   145
importlib_resources/tests/test_resource.py                    128      1    99%   241
importlib_resources/tests/util.py                              91      5    95%   32-35, 51
importlib_resources/tests/zip.py                               20      0   100%
-----------------------------------------------------------------------------------------
TOTAL                                                        1246     46    96%

================================================================ mypy =================================================================
Success: no issues found in 36 source files
======================================================= short test summary info =======================================================
FAILED importlib_resources/tests/test_functional.py::FunctionalAPITest_StringAnchor::test_open_text - AssertionError: '\\xff\\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00[67 chars]\x00' != '\\xfe\\xff\x00H\x00e\x00l\x00l\x00o\x00,\x00 [67...
FAILED importlib_resources/tests/test_functional.py::FunctionalAPITest_StringAnchor::test_read_text - AssertionError: '\\xff\\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00[67 chars]\x00' != '\\xfe\\xff\x00H\x00e\x00l\x00l\x00o\x00,\x00 [67...
FAILED importlib_resources/tests/test_functional.py::FunctionalAPITest_ModuleAnchor::test_open_text - AssertionError: '\\xff\\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00[67 chars]\x00' != '\\xfe\\xff\x00H\x00e\x00l\x00l\x00o\x00,\x00 [67...
FAILED importlib_resources/tests/test_functional.py::FunctionalAPITest_ModuleAnchor::test_read_text - AssertionError: '\\xff\\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00[67 chars]\x00' != '\\xfe\\xff\x00H\x00e\x00l\x00l\x00o\x00,\x00 [67...
================================== 4 failed, 308 passed, 1 skipped, 19 warnings in 64.12s (0:01:04) ===================================
py312: exit 1 (67.31 seconds) /home/mgorny/importlib_resources> pytest pid=1199072
  py312: FAIL code 1 (135.61=setup[68.30]+cmd[67.31] seconds)
  evaluation failed :( (136.06 seconds)

Reproduced with be5a270. I'll submit a PR shortly.
mgorny added a commit to mgorny/importlib_resources that referenced this issue Jul 10, 2024
@mgorny
Fix functional API tests to be endian-agnostic
3167e4b 
Fix the "backslashreplace" tests for the functional API to be
endian-agnostic.  The tests used to rely on `.encode("utf-16")`
producing the same data as found in the test file.  However, on big
endian platforms it would produce a big endian encoding, while the test
file is little endian.  To avoid the problem, explicitly specify
`utf-16-le` encoding.  Since this meant that the BOM is no longer
produced, explicitly include it in input.

Fixes python#312
@mgorny mgorny mentioned this issue Jul 10, 2024
Closed
@jaraco jaraco mentioned this issue Aug 14, 2024
Merged
@jaraco
Copy link
Member

jaraco commented Aug 14, 2024

Thanks for the report. It looks like this issue was also reported downstream at python/cpython#117569. Since there's already a fix there, I'm going to apply that one here to keep things in sync.

@jaraco jaraco closed this as completed in ca5f0e1 Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix functional API tests to be endian-agnostic mgorny/importlib_resources
2 participants
@mgorny @jaraco