Package resources with pkgutil, pkg_resources and importlib.resources

[thijsmie]

I've tested several mechanisms for package resources with in-memory PyOxidizer builds and the results are inconsistent. I made a little demo repository where I try out different mechanisms: https://github.com/thijsmie/pyoxidizer-resource-tests I hope this can serve as a reproducer while debugging this.

A short summary would be that pkgutil never works, importlib.resources works in some cases and pkg_resources always works in my tests so far. This is probably related to #529.

[kambe-mikb]

I got importlib.resources to work, at least for my use case (loading translation files), but I found that it's broken on anything less than Python 3.10. The package does not work as advertised on 3.8 or 3.9 - it can only handle resources in a file system.

[thijsmie]

Indeed, I've also verified that on 3.9 it is not working at all. Sorry for tagging you @indygreg but can you comment on this? I would be happy to contribute a fix but I need a place to start...

[kambe-mikb]

I was probably too terse in my earlier comment. The problem, at least in the case of importlib.resources is not with PyOxidizer, or any of its moving parts. Before Python 3.10, the code in the importlib.resources consisted of stubs which just handled resource requests as calls to the file system, and didn't utilize the modern importlib machinery. This means that the machinery PyOxidizer installs to handle resource requests in in-memory modules is simply ignored. Python 3.10 and later now access the module spec to locate and open resources, which means resources in an in-memory package can be found.

My code uses importlib.resources.is_resource() and importlib.resources.open_binary(), and only runs properly with at least 3.10.

Because it's a Python problem, and the fix is "upgrade to current Python", I would suggest this can be closed.