Migrate locations.py from distutils to sysconfig

[uranusjr]

Hi there! If you're here from a message that's something like:

WARNING: Value for scheme.scripts does not match. Please report this to <https://github.com/pypa/pip/issues/9617

Please upgrade to the newest pip release (21.1.1 at the time of writing).

- @pradyunsg (on 3-May-2021)

---

First off, this specific class of warnings can be safely ignored. They were added to help pip's maintainers identify incorrectly patched/configured Python distributions provided by redistributors (eg: with your OS, or as part of $corp's VM/Docker images etc). You can read more about actionable steps for end-users, in this StackOverflow answer.

Secondly, if you came here to report the message: thank you! This issue has been locked now, since this warning has been significantly more widespread than we'd expected -- in other words there are lots of Python installations that are subtly wrong in a way that hasn't mattered till today. This warning has helped surface that and enables us to plan for a smoother transition in the future.

Thirdly, we've identified certain cases where the warning is overzealous, and we'll fix that in a bugfix release soon.

Finally, I encourage you to subscribe to this issue if you want to stay up to date with this issue. We'll be discussing how we're going to handle the various edge cases reported by the users, and also announce it here once there's a release containing those fixes.

- @pradyunsg (on 26-Apr-2021)

---

Assuming normal CPython deprecation and release cycles, distutils will be deprecated in 3.10 and removed in 3.12. This gives us a bit more than two years to make the transition happen. Unfortunately there are a lot of wanky downstreams we need to help transition correctly in the two years.

Fortunately, logic ins location.py is relatively cheap, so we can probably try to detect issues by calculating the locations from both sources, and warn when they don’t match. This will hopefully prompt users to find issues for us that we can forward to downstreams.

So the plan is:

1. Build a locations.py implementation that’s based entirely on sysconfig. Maintain the two implementations in parallel.
2. When a value is requested (e.g. locations.distutils_scheme()), check whether the two implementations produce the same value. Emit a warning if they don’t (and tell the user to report this). Return the value from the old implementation.
3. After a while, switch to return value to the new implementation. Keep the old value accessible via --use-deprecated. Keep warning when the two values don’t match.
4. Drop the old implementation (and the warning) when distutils is formally dropped in CPython master.

[KOLANICH]

Value for scheme.<something> does not match. Please report this to <https://github.com/pypa/pip/issues/9617>

https://gitlab.com/KOLANICH-subgroups/docker-images/fixed_python/-/jobs/1081913528

[uranusjr]

Is this Debian/Ubuntu’s apt-installed Python?

[uranusjr]

Update: Seems like it is. Link to Dockerfile, which is based on debian:unstable-slim.

So it seems like Debian patches distutils to use the /usr/local prefix (I’m not familiar with how they do it), but does not do the same with sysconfig. We’ll need some Debian involvement here.

cc @stefanor

[KOLANICH]

Is this Debian/Ubuntu’s apt-installed Python?

It is (as you see from the Dockerfile). Sorry for non-providing the direct link to the first Docker image. I also can add that the same issue is seen on Ubuntu 20.10 and on some old version of Anaconda on Windows.

[pradyunsg]

/cc @doko42 @kitterma as well.

[doko42]

The problem that I'm trying to address: Users of a Linux distro (users, not necessarily python developers) can break their system by calling

sudo python3 setup.py install
sudo pip install

overriding packages installed with the distro package manager. The implemented solution (at least for Debian) is to have all modules installed by the distro package manager into a directory /usr/lib/python3/dist-packages, and to re-direct installations of "sudo python3 setup.py install" to /usr/local/lib/python3.9/dist-packages.

Looking at the docker run cited above:
WARNING: Value for scheme.platlib does not match. Please report this to #9617
distutils: /usr/local/lib/python3.9/dist-packages
sysconfig: /usr/lib/python3.9/site-packages

distutils/setuptools have a local patch to install into /usr/local/lib/python3.9/dist-packages by default, and an extra option to select the install layout, like python3 setup.py install --install-layout=posix|unix_local|posix_local|deb_system, with the default to not install into /usr/lib/python3/dist-packages, and potentially breaking the user's Linux installation.

The docker run above warns:
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

however it goes on potentially breaking the Linux installation.
https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe also sees the default behavior of "sudo pip" as problematic.

Now there might be use cases like doing these kind of installations in docker images where you either just want to use the python interpreter provided by a Linux distro, or even might want to mix system installed python modules and pip installed modules.

I'm open to suggestions how to make the docker experience better, otoh I see the need to provide work arounds for a default upstream behavior potentially breaking a Linux installation. Compare that with the somehow safe design of autoconf based software, doing

./configure && make && make install

installing into /usr/local by default, not breaking a users system.

[doko42]

pulling in setuptools as well, @jaraco

[uranusjr]

I agree that installing into /usr/local by default is the correct thing to do for Debian. The problem though is that Debian only patches distutils, but not sysconfig, which is designed to replace distutils usage, especially since distutils is scheduled to be removed from stdlib. So I think the solution would be for Debian to also patch sysconfig in a way to return the same value as distutils currently does.

[doko42]

agreed, maybe the patch should be extended.

Note that just patching Debian isn't good enough. E.g. users using an upstream pip on Debian's Python3 still shouldn't be able to b0rk their system.

[uranusjr]

Yes, pip will need some logic to choose the correct scheme on a platform. We already do a similar thing for PyPy (so installing system-wide PyPy-specific packages won’t break the system CPython installation), and we can do something similar for an APT-installed CPython.

Longer term, we’ll need a built-in way to query a CPython installation how they want site-packages to be installed. I’ve opened an entry on b.p.o. on this, but it seems like not have drawn much interest. Please also join the discussion there so we can push a change into CPython and have enough time to migrate from distutils before its removal.

[pradyunsg]

IIUC, based on the discussion we had for the deprecation of distutils PEP, the plan is to essentially have redistributors patch both until we transition over to just sysconfig. In the 3.10 release, we'd basically have them have the same source of truth, but older versions may start warning/failing, if users use newer pip versions on them.

I'm going 100% off memory though, and this could be wrong.

[doko42]

I have a patch which I never applied:
https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/patches/sysconfig-debian-schemes.diff

This would make the schemes known, and decide on some context/magic which one should be the default. I think the reason I never applied that, is that there's no control when 'deb_system' is choosen.

When 'deb_system' is requested, you probably could also check the env for a set SUDO_USER and USER=="root", and a YES_INSTALL_INTO_INTO_PYTHON_SYSTEM_LOCATION, and then, if requirements are not met, raise an exception?

[jaraco]

• @FFY00 as he’s helping with the distutils adoption in Setuptools.

[uranusjr]

This would make the schemes known, and decide on some context/magic which one should be the default. I think the reason I never applied that, is that there's no control when 'deb_system' is choosen.

Yeah, that’s the main purpose behind my b.p.o. filing, so Debian’s Python can also patch get_default_scheme() and get_preferred_schemes() to return 'posix_local' when appropriate for tools to choose.

Before that happens, pip can carry special logic to choose 'posix_local' when pip detects itself is running under a Debian Python instead of e.g. self-compiled (I’m assuming there’s a way to do this). But we’ll need Debian to provide 'posix_local' for pip first.

[doko42]

can we agree on the name 'posix_local', having 'dist-packages' in the paths, or should that be 'debian_local', or something else?

[uranusjr]

You can choose anything you want. debian_local is probably a bit better if you ask me (to make it clearer it’s added by Debian), but it’s up to you.

[pradyunsg]

Yea, I also prefer debian_local, especially if we're adding Debian-specific checks here.

I do want to double check though: these checks would only be for the transition from distutils.sysconfig to sysconfig right? If the intention is to keep them indefinitely, that's probably a conversation we should have on a dedicated issue, because my understanding is that we've generally avoided doing something like that (i.e. hold distro specific checks upstream).

[pfmoore]

these checks would only be for the transition from distutils.sysconfig to sysconfig right

My big concern here is that we'd end up having to retain these "temporary" checks for a long time, because we'll be expected to support users who remain on a system that hasn't completed the transition, long after newer versions of Debian are available. Ideally, I'd like a clearly agreed, relatively strict timescale for how long we're going to need to maintain these Debian-specific checks, and a very clear definition of which distro versions the checks are for (so we can advise users "you need to upgrade from X to Y as X is no longer supported by us").

[doko42]

@pfmoore, I don't understand "completed the transition".

@pradyunsg, do we have a common understanding that "sudo pip install" into a system location is harmful, not Debian specific, and that we should find a solution that prevents doing that?

[pfmoore]

@doko42 See @uranusjr's comment:

Before that happens, pip can carry special logic to choose 'posix_local' when pip detects itself is running under a Debian Python instead of e.g. self-compiled (I’m assuming there’s a way to do this). But we’ll need Debian to provide 'posix_local' for pip first.

It's that "special logic" which we're talking about carrying within pip, "before that happens". What I'm saying is when will "that" (whatever "that" is...) happen, and hence when will pip be able to remove that special code. My instinct says that we won't be able to pinpoint a precise time when "that" has happened, because we'll get told "yes, current versions of Debian have fixed this, but you can't drop support for Debian users who haven't upgraded to that version yet". And so we'll be carrying that supposedly-temporary code for much longer than comments like "Debian will have transitioned by XX date" would suggest 🙁

do we have a common understanding that "sudo pip install" into a system location is harmful, not Debian specific, and that we should find a solution that prevents doing that?

In principle, yes. But sudo rm -rf / is also harmful, and nobody expects rm to detect and prevent that. So I think in terms of trade-offs here, not absolutes.

But if you mean by this that the locations returned by sysconfig should be:

1. Safe to install to with non-system tools, and
2. Accessible to tools without needing distro-specific logic in those tools

then yes, I agree 100% (why wouldn't I, it means that pip has to do nothing except use the Python stdlib interfaces in the way they are designed to be used 🙂)

[doko42]

@doko42 See @uranusjr's comment:

Before that happens, pip can carry special logic to choose 'posix_local' when pip detects itself is running under a Debian Python instead of e.g. self-compiled (I’m assuming there’s a way to do this). But we’ll need Debian to provide 'posix_local' for pip first.

It's that "special logic" which we're talking about carrying within pip, "before that happens". What I'm saying is when will "that" (whatever "that" is...) happen, and hence when will pip be able to remove that special code. My instinct says that we won't be able to pinpoint a precise time when "that" has happened, because we'll get told "yes, current versions of Debian have fixed this, but you can't drop support for Debian users who haven't upgraded to that version yet". And so we'll be carrying that supposedly-temporary code for much longer than comments like "Debian will have transitioned by XX date" would suggest

ok, Debian is currently in freeze to prepare for it's next release. I'll still try get the 'deb_local' scheme included.

do we have a common understanding that "sudo pip install" into a system location is harmful, not Debian specific, and that we should find a solution that prevents doing that?

In principle, yes. But sudo rm -rf / is also harmful, and nobody expects rm to detect and prevent that. So I think in terms of trade-offs here, not absolutes.

sure, but package managers should be a bit more intelligent ;) Trying to do a

$ sudo apt remove bash
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
bash gdm3 ubuntu-desktop ubuntu-desktop-minimal
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
bash
0 upgraded, 0 newly installed, 4 to remove and 3 not upgraded.
After this operation, 3,959 kB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
?]

so, yes, I can continue, but I get this extra warning, and decide if I want to continue. This is what I would like to see for pip as well.

But if you mean by this that the locations returned by sysconfig should be:

1. Safe to install to with non-system tools, and
2. Accessible to tools without needing distro-specific logic in those tools

then yes, I agree 100% (why wouldn't I, it means that pip has to do nothing except use the Python stdlib interfaces in the way they are designed to be used )

yes, but that means, once we add the 'deb_system' scheme as well, we are in the same situation as before, allowing a

sudo pip install --scheme=deb_system

to potentially break the users system. That's why I think such kind of installation should be guarded
#9617 (comment)

[pfmoore]

yes, but that means, once we add the 'deb_system' scheme as well, we are in the same situation as before

Agreed, and I have no problem with pip including an "are you sure?" type of warning, just like apt does. And I agree with what I think you were saying above, this shouldn't be specific to one distro. But IMO that means it's an issue that should be raised with the Python core devs, rather than with individual packaging tools.

Maybe what's needed here is a way for distros to designate a scheme as "protected", so that installers can know when to say "are you sure?" That could be added to sysconfig for Python 3.10 with a backport of the functionality available on PyPI, which pip could vendor for pre-3.10 Python versions.

To put it another way, I think the ultimate solution should be agreed between distros and core Python. Tools like pip can provide temporary stopgaps (because the pace of change in the Python core is slow) but we should be very clear that solving the problem in pip doesn't actually fix the problem, it's just reintroducing the sort of implementation-defined behaviour that we've been trying so hard to stop including in pip for years now.

(A half-way measure might be to define a packaging standard that documents how tools should handle installation schemes. But I don't really see much advantage in that over a core Python change - both would involve the same sort of PEP process, after all...)

[FFY00]

A few years ago played with the idea of having sort of a posix_local scheme in arch to prevent conflicts between pip and the package manager, but I now see that is wrong. Having a posix_local would conflict with a user-installed Python (prefix=/usr/local), and I also do not think this is correct according to FHS. The prefix shouldn't be used as a way to prevent unrelated conflicts, system Python packages should go to /usr and (system-wide) user Python packages should go to /usr/local.

---

I had a chat with @jaraco and the solution that kind of resulted was to simply let Python providers (like Debian, Fedora, etc.) register new schemes in sysconfig.INSTALL_SCHEMES to prevent conflicts. These new schemes would be included in the site import path.

Python distributors shouldn't do any other modification, letting pip, distutils, etc. always install to the same place in every system and behave consistently.
Then can install their own packages to their scheme, like discussed here.

sudo pip install --scheme=deb_system

@doko42 as far as Debian is concerned, is there anything you want to note here? Any requirements we did not take into account?

[doko42]

A few years ago played with the idea of having sort of a posix_local scheme in arch to prevent conflicts between pip and the package manager, but I now see that is wrong. Having a posix_local would conflict with a user-installed Python (prefix=/usr/local), and I also do not think this is correct according to FHS. The prefix shouldn't be used as a way to prevent unrelated conflicts, system Python packages should go to /usr and (system-wide) user Python packages should go to /usr/local.

yes, that' why the 'deb_local' scheme uses 'dist-packages' in the paths. That was once suggested by Barry Warsaw.

Python distributors shouldn't do any other modification, letting pip, distutils, etc. always install to the same place in every system and behave consistently.
Then can install their own packages to their scheme, like discussed here.

sudo pip install --scheme=deb_system

@doko42 as far as Debian is concerned, is there anything you want to note here? Any requirements we did not take into account?

sorry, could you point to the requirements you're speaking about?

[doko42]

I think at one of the Cleveland PyCons I spoke either with @jaraco or @pradyunsg about co-managing Python module installations with pip and the distro manager. But I didn't follow-up on that topic either.

[sbidoul]

For folks landing here after installing pip 21.1.1, please note it is normal that the message appears when pip 21.1 upgrades itself to 21.1.1. After that the message is only visible with -v, and in less common situations.

[speer-kinjo]

Error while installing on M1 mac matplotlib_venn : Scheme.headers does not match

% pip3 install matplotlib_venn
WARNING: Value for scheme.headers does not match. Please report this to #9617
distutils: /opt/homebrew/Caskroom/miniforge/base/include/python3.9/UNKNOWN
sysconfig: /opt/homebrew/Caskroom/miniforge/base/include/python3.9
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Requirement already satisfied: matplotlib_venn in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (0.11.6)
Requirement already satisfied: matplotlib in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib_venn) (3.4.2)
Requirement already satisfied: scipy in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib_venn) (1.6.3)
Requirement already satisfied: numpy in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib_venn) (1.20.2)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (2.4.7)
Requirement already satisfied: python-dateutil>=2.7 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (2.8.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (1.3.1)
Requirement already satisfied: pillow>=6.2.0 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (8.2.0)
Requirement already satisfied: cycler>=0.10 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (0.10.0)
Requirement already satisfied: six in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from cycler>=0.10->matplotlib->matplotlib_venn) (1.15.0)
WARNING: Value for scheme.headers does not match. Please report this to #9617
distutils: /opt/homebrew/Caskroom/miniforge/base/include/python3.9/UNKNOWN
sysconfig: /opt/homebrew/Caskroom/miniforge/base/include/python3.9
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
(base) speer@speer ~ % >....
Requirement already satisfied: pyparsing>=2.2.1 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (2.4.7)
Requirement already satisfied: python-dateutil>=2.7 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (2.8.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (1.3.1)
Requirement already satisfied: pillow>=6.2.0 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (8.2.0)
Requirement already satisfied: cycler>=0.10 in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from matplotlib->matplotlib_venn) (0.10.0)
Requirement already satisfied: six in /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages (from cycler>=0.10->matplotlib->matplotlib_venn) (1.15.0)
WARNING: Value for scheme.headers does not match. Please report this to #9617
distutils: /opt/homebrew/Caskroom/miniforge/base/include/python3.9/UNKNOWN
sysconfig: /opt/homebrew/Caskroom/miniforge/base/include/python3.9
WARNING: Additional context:
user = False
home = None
root = None
prefix = None

[gothicVI]

@speer-kinjo what's your pip version? Did you update it?

[speer-kinjo]

@speer-kinjo what's your pip version? Did you update it?

Yeah I had updated to version 21.1.1, I then downgraded it to 21.0.1 and the error is gone

[Zaryob]

Error while using conda environment with jupyter-notebook

WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: /home/linaro/mambaforge/envs/jupyter_notebook/include/python3.7m/UNKNOWN
sysconfig: /home/linaro/mambaforge/envs/jupyter_notebook/include/python3.7m
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Collecting jupyter-c-kernel
  Downloading jupyter_c_kernel-1.2.2.tar.gz (4.4 kB)
Building wheels for collected packages: jupyter-c-kernel
  Building wheel for jupyter-c-kernel (setup.py) ... done
  Created wheel for jupyter-c-kernel: filename=jupyter_c_kernel-1.2.2-py3-none-any.whl size=6620 sha256=95cb82143262c5f7bec430bc803695ab07b2dbf9fee5230c7cb8859e3ef6d0e2
  Stored in directory: /home/linaro/.cache/pip/wheels/76/ae/15/6053344767c25a22a50438761b14b939fee321b1f39d089c4e
Successfully built jupyter-c-kernel
Installing collected packages: jupyter-c-kernel
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: /home/linaro/mambaforge/envs/jupyter_notebook/include/python3.7m/UNKNOWN
sysconfig: /home/linaro/mambaforge/envs/jupyter_notebook/include/python3.7m
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Successfully installed jupyter-c-kernel-1.2.2

[speer-kinjo]

@speer-kinjo what's your pip version? Did you update it?

pip3 --version
pip 21.1.1 from /opt/homebrew/lib/python3.9/site-packages/pip (python 3.9)

[DiddiLeija]

Previously I had the same issue, the same warning. But I just updated pip 21.1 to 21.1.1. After that, the problem hasn't appeared again. I am using Python 3.8.2 on Windows 7 (only using the command prompt cmd.exe). But I'm not so sure if the problem can happen again. How can I help for that?

[uranusjr]

The issue is always there, we just hid the messages from you. But we already know what to do with your issue as well, so you shouldn’t need to do anything. The Windows one has been fixed (or has a standing PR that will fix it? I forget), and we’re working with Homebrew for the Mac one.

[DiddiLeija]

@uranusjr Thank you for your time.

The Windows one has been fixed (or has a standing PR that will fix it? I forget),

If there is a standing PR for that, I would like to help. Thank you very much.

[bjtucker]

I see that the fix for this is noted as being temporary (titled Remove the location warnings for now), but I'd like to state a preference for messages like this to always go to debug. It was occurring in one of my environments (apparently) but causing no trouble for us. The pipeline ran fine until pip 21.1.0 and then started to balk because something unrecognized was written to stderr.

[uranusjr]

I would like to point out that if your environment saw this warning, it’s going to outright break some point in the future (not just a warning message, the environment won’t work). The intention of the error message is to signal the future breakage to users so you could report it back for us to do something. But if you prefer to let it break without giving us a chance to do anything, that’s fine for me.

[gothicVI]

@KrisUz take a look at the first post and the couple of posts above yours.

[pradyunsg]

Locking this, to prevent superfluous notifications for everyone.

Please read the first post.