Cannot create a path from a zipped resource with Python 3.6

[jaraco]

In GitLab by @Dan737 on Jul 3, 2018, 12:13

I'm using Python 3.6.4. I'm trying to access a resource in a whl file. Using the following code:

import sys, importlib_resources
sys.path.append('path/to/my/wheel/demo_package.whl')
import demo_package
with importlib_resources.path('demo_package', 'somefile.txt') as p:
    print(str(p))

This leads to a FileNotFoundError: [Errno 2] No such file or directory error.

I tried a couple of things and I figured out that the imported package has a loader=<zipimporter object>. But this object does not have any get_resource_reader method.

I noticed that in Python 3.7 this method is now defined on zipimporter class. So my question is basically: can we use this backport for zipped files with Python less than 3.7?

[jaraco]

In GitLab by @Dan737 on Jul 4, 2018, 03:06

changed the description

[jaraco]

In GitLab by @Dan737 on Jul 4, 2018, 03:07

changed the description

[jaraco]

In GitLab by @pauleveritt on Jul 12, 2018, 08:49

I also have a project that wants to use importlib targeting Python 3.6, with wheels.

[jaraco]

In GitLab by @warsaw on Jul 12, 2018, 14:58

This should definitely work. importlib_resources has a fallback for zip importers in Pythons < 3.7, but perhaps there's something funky about wheels that's breaking the assumptions in the code. This is kind of a nasty part of the backport, and it's had bugs in the past, so I wouldn't be surprised. We have a lot of tests for .zip files in general, but none specifically for .whl files.

[jaraco]

In GitLab by @Dan737 on Jul 13, 2018, 04:50

Thanks for taking the time to look into this. Here are the minimal steps I could find to reproduce the problem.

Create a directory with the following structure:

/dummy_pkg
    /dist
        |- get_resource.py
    /dummy_pkg
        |- __init__.py
        |- readme.txt
    setup.py

setup.py

from setuptools import setup, find_packages

pkg_data = {'': ['*.txt']}

setup(
    name='dummy_pkg',
    version='0.0.1',
    description='Dummy package',
    long_description='Dummy package',
    author='Daniel Gillet',
    author_email='whatever@gmail.com',
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'Topic :: Software Development',
        'License :: OSI Approved :: BSD License',
        'Programming Language :: Python :: 3.6',
    ],
    keywords='dummy',
    packages=find_packages(),
    package_data=pkg_data,
)

dist/get_resource.py

import sys
import importlib_resources
sys.path.append('dummy_pkg-0.0.1-py3-none-any.whl')

import dummy_pkg
with importlib_resources.path('dummy_pkg', 'readme.txt') as p:
    print(str(p))

Initially, run python setup.py bdist_wheel to build the wheel in the dist folder.
Then cd into the dist folder and run python get_resource.py.

On Python 3.6, I get the following Traceback:

(resource3.6) C:\OfflineFiles\Python\dummy_pkg\dist>python get_resource.py
Traceback (most recent call last):
  File "C:\OfflineFiles\Python\venvs\resource3.6\lib\site-packages\importlib_res
ources\_py3.py", line 85, in open_binary
    return open(full_path, mode='rb')
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\OfflineFiles\\Pytho
n\\dummy_pkg\\dist\\dummy_pkg-0.0.1-py3-none-any.whl\\dummy_pkg\\readme.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "get_resource.py", line 6, in <module>
    with importlib_resources.path('dummy_pkg', 'readme.txt') as p:
  File "C:\Users\DANIELG\AppData\Local\Programs\Python\Python36\lib\contextlib.p
y", line 81, in __enter__
    return next(self.gen)
  File "C:\OfflineFiles\Python\venvs\resource3.6\lib\site-packages\importlib_res
ources\_py3.py", line 188, in path
    with open_binary(package, resource) as fp:
  File "C:\OfflineFiles\Python\venvs\resource3.6\lib\site-packages\importlib_res
ources\_py3.py", line 99, in open_binary
    raise FileNotFoundError(message)
FileNotFoundError: 'readme.txt' resource not found in 'dummy_pkg'

On Python 3.7 it works with the following output:

(resource3.7) C:\OfflineFiles\Python\dummy_pkg\dist>python get_resource.py
C:\Users\DANIELG\AppData\Local\Temp\tmpih78yy0f

[jaraco]

In GitLab by @warsaw on Aug 16, 2018, 02:39

Actually, the problem is in Python 3.6 and we cannot fix it here. Nor can we fix it in 3.6 without breaking backward compatibility.

Here's the problem: In 3.6, if you add the zip/whl to sys.path with a relative (i.e. not absolute) path name, then the zipimporter's loader's get_data() can't resolve package relative file names. We worked around this in 3.7 because it doesn't use this bit of loader logic, and we were able to do the right thing in the ResourceReader implementation.

If you change the code in get_resource.py to use sys.path.append(os.path.abspath('dummy_pkg-0.0.1-py3-none-any.whl')) everything will work.

I'll keep this bug open but change the labels. We should document this peculiarity.

[jaraco]

In GitLab by @warsaw on Sep 7, 2018, 18:48

mentioned in commit 6f880d6

[jaraco]

In GitLab by @warsaw on Sep 7, 2018, 18:50

mentioned in merge request !65

[jaraco]

In GitLab by @warsaw on Sep 7, 2018, 18:56

closed via merge request !65

[jaraco]

In GitLab by @warsaw on Sep 7, 2018, 18:56

closed via commit 6f880d6

[jaraco]

In GitLab by @warsaw on Sep 7, 2018, 18:56

mentioned in commit e6e4e46