PYTHONPATH can break editable packages #1005
Labels
Needs Triage
Issues that need to be evaluated for severity and status.
Comments
|
You've demonstrated the unexpected behavior in the upstream ticket, but let's see if we can demonstrate the issue in a more distilled fashion using setuptools only. In my initial attempt, I was unsuccessful: Even when I use pip on Flask, I don't have the issue: So on first blush, I'm unable to replicate your issue. But more importantly, as reported, I'm not even sure this is an issue we wish to tackle. It doesn't strike me as unreasonable that if the installer is in an environment where the package is already on sys.path that it would choose not to add the redundant So to proceed, we need two things: first, an example that can be readily replicated on late versions of Python and Setuptools (without pip); and second, a justification for why the use-case is valid (i.e. an answer to why would anyone want to set PYTHONPATH and setup.py develop the package?). |
|
Rationale :
A user set PYTHONPATH in a shell environment.
He can also start other shells ( with his PYTHONPATH set a different way ).
Having the installed files content depending on PYTHONPATH makes the
install quite brittle : the installed environment will work as expected
with that PYTHONPATH value only, and not with a different one.
Use case :
I am using Python with ROS (http://www.ros.org) , which entails a lot of
different shells, potentially with different setups...
ROS uses PYTHONPATH heavily to build an environment from multiple install
prefix. So manipulating PYTHONPATH is common place.
Also ROS still relies on distutils, and custom "__init__.py" interpreting
files in other places to achieve the "editable install" effect.
I am currently working to bring ROS usage of python up to date, as much as
I can, by relying on setuptools and the current way to do things in Python.
In that context a user creates an editable install, and then run it
multiple times, from different shells with potentially different
configurations.
I will post here again when I can reproduce only with setuptools. It s
likely also that the version that we are using impacts the behavior as ROS
is still using python 2.7 and I was testing on ubuntu trusty where the
system python setup is quite old.
…On Apr 8, 2017 10:57 PM, "Jason R. Coombs" ***@***.***> wrote:
You've demonstrated the unexpected behavior in the upstream ticket, but
let's see if we can demonstrate the issue in a more distilled fashion using
setuptools only.
In my initial attempt, I was unsuccessful:
$ mkdir issue-1005
$ cd issue-1005
$ python -m venv .env
$ . .env/bin/activate
(.env) $ cat > setup.py
import setuptools
setuptools.setup(name='foo', version='1.0', packages=['foo_pkg'])
(.env) $ mkdir foo_pkg
(.env) $ touch foo_pkg/__init__.py
(.env) $ PYTHONPATH=$(pwd) python setup.py develop -q
running develop
running egg_info
creating foo.egg-info
writing foo.egg-info/PKG-INFO
writing dependency_links to foo.egg-info/dependency_links.txt
writing top-level names to foo.egg-info/top_level.txt
writing manifest file 'foo.egg-info/SOURCES.txt'
reading manifest file 'foo.egg-info/SOURCES.txt'
writing manifest file 'foo.egg-info/SOURCES.txt'
running build_ext
Creating /Users/jaraco/issue-1005/.env/lib/python3.6/site-packages/foo.egg-link (link to .)
Adding foo 1.0 to easy-install.pth file
Installed /Users/jaraco/issue-1005
Processing dependencies for foo==1.0
Finished processing dependencies for foo==1.0
(.env) $ cat .env/lib/python3.6/site-packages/easy-install.pth
/Users/jaraco/issue-1005
Even when I use pip on Flask, I don't have the issue:
(.env) $ git clone -q https://github.com/mitsuhiko/flask.git
(.env) $ cat .env/lib/python3.6/site-packages/easy-install.pth
(.env) $ PYTHONPATH=$(pwd)/flask pip install -v --no-deps -e flask
Obtaining file:///Users/jaraco/issue-1005/flask
Running setup.py (path:/Users/jaraco/issue-1005/flask/setup.py) egg_info for package from file:///Users/jaraco/issue-1005/flask
Running command python setup.py egg_info
running egg_info
writing Flask.egg-info/PKG-INFO
writing dependency_links to Flask.egg-info/dependency_links.txt
writing entry points to Flask.egg-info/entry_points.txt
writing requirements to Flask.egg-info/requires.txt
writing top-level names to Flask.egg-info/top_level.txt
/Users/jaraco/issue-1005/.env/lib/python3.6/site-packages/setuptools/dist.py:331: UserWarning: Normalizing '0.13-dev' to '0.13.dev0'
normalized_version,
warning: manifest_maker: standard file '-c' not found
reading manifest file 'Flask.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes'
writing manifest file 'Flask.egg-info/SOURCES.txt'
Source in ./flask has version 0.13.dev0, which satisfies requirement Flask==0.13.dev0 from file:///Users/jaraco/issue-1005/flask
Installing collected packages: Flask
Found existing installation: Flask 0.13.dev0
Not uninstalling flask at /Users/jaraco/issue-1005/flask, outside environment /Users/jaraco/issue-1005/.env
Running setup.py develop for Flask
Running command /Users/jaraco/issue-1005/.env/bin/python3 -c "import setuptools, tokenize;__file__='/Users/jaraco/issue-1005/flask/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps
running develop
running egg_info
writing Flask.egg-info/PKG-INFO
writing dependency_links to Flask.egg-info/dependency_links.txt
writing entry points to Flask.egg-info/entry_points.txt
writing requirements to Flask.egg-info/requires.txt
writing top-level names to Flask.egg-info/top_level.txt
/Users/jaraco/issue-1005/.env/lib/python3.6/site-packages/setuptools/dist.py:331: UserWarning: Normalizing '0.13-dev' to '0.13.dev0'
normalized_version,
warning: manifest_maker: standard file '-c' not found
reading manifest file 'Flask.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes'
writing manifest file 'Flask.egg-info/SOURCES.txt'
running build_ext
Creating /Users/jaraco/issue-1005/.env/lib/python3.6/site-packages/Flask.egg-link (link to .)
Adding Flask 0.13.dev0 to easy-install.pth file
Installing flask script to /Users/jaraco/issue-1005/.env/bin
Installed /Users/jaraco/issue-1005/flask
Successfully installed Flask
Cleaning up...
(.env) $ cat .env/lib/python3.6/site-packages/easy-install.pth
/Users/jaraco/issue-1005/flask
So on first blush, I'm unable to replicate your issue.
But more importantly, as reported, I'm not even sure this is an issue we
wish to tackle. It doesn't strike me as unreasonable that if the installer
is in an environment where the package is already on sys.path that it would
choose not to add the redundant .pth entry.
So to proceed, we need two things: first, an example that can be readily
replicated on late versions of Python and Setuptools (without pip); and
second, a justification for why the use-case is valid (i.e. an answer to
why would anyone want to set PYTHONPATH and setup.py develop the package?).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1005 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AANgSGoDKoPqRVP0mZ5wVU18zi4C9LE4ks5rt5JKgaJpZM4MykCB>
.
|
|
Firstly, I was able to reproduce the behavior you had, but with python2.7 : But this is already the case in the upstream issue. The problem is that if, just after, I do : Flask is now gone... In your setuptools example, you have only one directory with one package. So here we can see the path |
|
Aha. That helps. So it's the I don't know when I'll have time to dig deeper, so if you have the time and drive, the next step will be to trace the code during the install of this second package and see if you can detect where the foo_repo is getting removed from the path. |
|
I m on my first adventure inside setuptools code, so I m not sure I understand much yet... I tried to debug with PyCharm, but Pycharm also modify the PYTHONPATH with the current projects path, depending what you launch and the order of the projects in the UI, so it s a bit tricky to find... Maybe they were attempting to work around the problem we are having... In case it is useful I started https://github.com/asmodehn/setuptools-1005
I am not sure what is the intent behind removing, from the |
|
Hello! I ran into this as well. As others have pointed out, the cleanup function here is a little over-eager. It's easy to repro as well. Do you agree that this is a bug / would you accept a PR fixing this? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I haven't traced the issue down to the setuptools details yet but here is the symptom :
pypa/pip#4261
TL;DR if a path is present in PYTHONPATH when installing (--editable -> setup.py develop ) a package , that path is not added to
easy-install.pth.We probably need a more detailed (and setuptools focused) report.
The text was updated successfully, but these errors were encountered: