--target doesn't work for namespace packages

[webmaven]

When doing a local install of a namespace package, the .pth file doesn't get picked up for some reason:

MAC-C08182:projects c08182$ mkdir targettest
MAC-C08182:projects c08182$ cd targettest/
MAC-C08182:targettest c08182$ pip install repoze.lru -t lib/
Downloading/unpacking repoze.lru
  Downloading repoze.lru-0.6.tar.gz
  Running setup.py (path:/private/var/folders/9w/_wt1czb57k9c5b0hgg6dfl2x7jbl85/T/pip_build_c08182/repoze.lru/setup.py) egg_info for package repoze.lru

Installing collected packages: repoze.lru
  Running setup.py install for repoze.lru

    Skipping installation of /var/folders/9w/_wt1czb57k9c5b0hgg6dfl2x7jbl85/T/tmp7uY1G8/lib/python/repoze/__init__.py (namespace package)
    Installing /var/folders/9w/_wt1czb57k9c5b0hgg6dfl2x7jbl85/T/tmp7uY1G8/lib/python/repoze.lru-0.6-py2.7-nspkg.pth
Successfully installed repoze.lru
Cleaning up...
MAC-C08182:targettest c08182$ PYTHONPATH=$(pwd)/lib
MAC-C08182:targettest c08182$ python
Python 2.7.7 |Anaconda 2.0.0 (x86_64)| (default, Jun  2 2014, 12:48:16) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> import repoze.lru
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named repoze.lru
>>> 
MAC-C08182:targettest c08182$ 

However, all necessary files are in fact installed in lib/:

MAC-C08182:targettest c08182$ cd lib/
MAC-C08182:lib c08182$ ls
repoze              repoze.lru-0.6-py2.7.egg-info
repoze.lru-0.6-py2.7-nspkg.pth
MAC-C08182:lib c08182$ 

And of course the whole procedure works just fine for non-namespace packages.

Thanks to @mmerickel for setting me straight on the source of the issue.

[mmerickel]

The Python docs say only the site-packages folders relative to sys.prefix and sys.exec_prefix are searched for .pth files. I guess the question is how to make the -t option work?

[webmaven]

@mmerickel I am having difficulty locating the relevant section of the docs. Link, please?

[mmerickel]

https://docs.python.org/2/library/site.html

[webmaven]

A path configuration file is a file whose name has the form name.pth and exists in one of the four directories mentioned above;

Darn, not a lot of ambiguity there.

Any ideas on how I can work around this limitation? App Engine inserts the lib subdirectory into sys.path (eg. https://github.com/GoogleCloudPlatform/appengine-python-flask-skeleton/blob/master/appengine_config.py#L6), but that obviously doesn't enable the .pth file.

[Ivoz]

You could manually move the namespace'd package that you want to the actual folder where it would be located if it was simply part of the overall package, in a build / deploy step.

Alternately, call addsitedir explicitly in your own sitecustomize.py file.

[webmaven]

@Ivoz site.addsitedir('lib') was exactly what I needed. Thanks!

[webmaven]

Unfortunately, site.addsitedir(path) appends the directory to the path, rather than inserting it at the beginning. This causes problems if one of the packages you're trying to use in the target directory is a newer version than one that exists earlier in the path.

I have a workaround which I am trying to simplify:

import os
import site
import sys

dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)

sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)

But that seems rather ugly to me. What would help a lot would be if a site.insertsitedir(index, path) was available (or something similar) as detailed in this issue that was brought to my attention by @gotgenes: http://bugs.python.org/issue7744

But in the absence of such a method, can the above code be improved or is there another, simpler workaround?

[zyv]

Sadly, this not only affects --target, but also --prefix... for now settled with the suggestion of @Ivoz, but this is ugly as hell:

for i in $(find lib -name site-packages -print); do echo 'import site, os.path; site.addsitedir(os.path.dirname(__file__))' > $i/sitecustomize.py; done

pip really should do something about it.

[jaraco]

This issue is implicated in jaraco/pip-run#5.

[jaraco]

Here's another thought - since Python 3.3+ supports namespace packages natively, why not on those Pythons drop the use of .pth files altogether and let Python handle the importing?

[jaraco]

I was going to explore the possibility of implementing my suggestion, but searching the pip codebase, I see no reference to nspkg.pth.

In seaching the web, I stumble across this distutils sig thread.

[jaraco]

Other interesting details in this thread.

[jaraco]

I've found that the nspkg.pth files are generated in setuptools, in the install_egg_info command.

[jaraco]

In that latest thread, PJE says:

Setuptools has essentially always
built non-egg, non-exe binary distributions in a PEP 420-compatible
way (i.e., without __init__.py). And pkg_resources already builds a
namespace path by asking importers if they can import a package at
that location. So if PEP 420 importers say "yes" when asked to
find_module('somepkg') in the case of an __init__-less subdirectory
named 'somepkg', then pkg_resources already supports mixed-mode
installations under PEP 420, and doesn't even need to be updated!

I haven't checked whether that's the case, but if it is, then the only
thing that setuptools neds to change is its .pth generation magic, to
not do the magic if it's on a PEP 420 platform at runtime, and to stop
including __init__.py's for namespace packages.

So checking if it's the case:

$ mkdir foo
$ touch foo/text.txt
$ python
Python 3.6.0b1+ (3.6:b0c56ee58478, Sep 24 2016, 09:12:37) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pvd = pkg_resources.get_provider('foo')
>>> pvd
<pkg_resources.NullProvider object at 0x10c9ec470>
>>> pvd.loader
<_frozen_importlib_external._NamespaceLoader object at 0x109fab978>

So it seems that at least nominally pkg_resources is able to provide resources for the trivial package. Although,

>>> pvd.get_resource_string('foo', 'text.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1457, in get_resource_string
    return self._get(self._fn(self.module_path, resource_name))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1532, in _get
    "Can't perform this operation for loaders without 'get_data()'"
NotImplementedError: Can't perform this operation for loaders without 'get_data()'

I suspect that if the package had some metadata (dist-info or egg_info), it would also work. So I want to say PJE's supposition is tenatively confirmed and focus on

the only
thing that setuptools [needs] to change is its .pth generation magic, to
not do the magic if it's on a PEP 420 platform at runtime

[sebcagnon]

Hello

I had this issue when using Google Cloud APIs with the command

./env/bin/pip install -r requirements.txt --target my-project/lib

My workaround is to add the following lines in my setup file...

cp resources/__init__.py my-project/lib/google
cp resources/__init__.py my-project/lib/google/cloud
cp resources/__init__.py my-project/lib/google/cloud/speech

etc...

Do you have anything better? I work with Python 2.7
I see you tried many things, but not sure which one would be official workaround.

[webmaven]

What with all the issues that reference this one (all closed), I'm a bit confused as to whether this issue remains relevant. @jaraco, can you comment?

[jaraco]

I don't believe there's much else to do here in pip. It's a limitation in python that the namespace package support required the -nspkg.pth hook to function. As long as the hook is needed, namespace packages won't load unless installed to a site dir. With PEP 420, these packages do run and with the latest updates to the hook, can coexist with hook-facilitated packages.

The pip-run (formerly rwt) package makes use of --target and works with namespace packages by installing a sitecustomize module. Perhaps that technique could be employed in other environments that rely on --target to get better namespace support in older Pythons.

[zyv]

The rwt package makes use of --target and works with namespace packages by installing a sitecustomize module. Perhaps that technique could be employed in other environments that rely on --target to get better namespace support in older Pythons.

That's how I solved the problem at my last employer. Would be awesome if it worked out of the box...

[dstufft]

I'm going to close this as there isn't much pip can do here, this is just how Python's sys.path works.

[JensTimmerman]

So if I get this correct pip will not try to have a working namespace package when installed with --target? Could in this case some info (warning?) with some useful information be printed out to the user? e.g. pip doesn't support this, you might want to run code for workaround with site package

[jaraco]

@JensTimmerman That's an interesting idea. I agree, it's a bit hostile to install packages into a --target and then they not be viable without some additional steps.

I'm not sure a warning is quite appropriate - as it may be the case that a tool invoking pip will be correcting for the condition as rwt does.

More importantly, though, this issue largely goes away with Python 3.3, so perhaps the best solution is just to work around it until you can port everything to Python 3.

[cas--]

For reference the missing __init__.py namespace files is a setuptools issue, which is a problem if using pkg_resources namespaces: pypa/setuptools#1097

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.