Incorrect vesion displayed with pip list when using --target and PYTHONPATH

[enoekki]

Description

when installing the package using --target and PYTHONPATH, an incorrect version is displayed when running pip list.

case 1. No --upgrade, This does not install a newer version. The expected result should be pip 24.1 installed

$ pip3 install pip==24.1.0 --target=/path/to/pip-upgrade-issue/PYPATH

...

Successfully installed pip-24.1

[notice] A new release of pip is available: 24.1 -> 24.2

[notice] To update, run: pip install --upgrade pip

$ pip3 install pip==24.1.2 --target=/path/to/pip-upgrade-issue/PYPATH

...

Successfully installed pip-24.1.2

WARNING: Target directory <PATH>/PYPATH/pip already exists. Specify --upgrade to force replacement.

WARNING: Target directory <PATH>/PYPATH/bin already exists. Specify --upgrade to force replacement.

$ pip list | grep pip

pip                24.1.2

$ python3 -c "import pip; print(pip.__version__)"

24.1

case 2. with --upgrade, This does a fresh install to a directory, then installs a newer version, The expected result should be pip==24.2 installed/displayed

$ pip3 install pip==24.1.2 --target=/path/to/pip-upgrade-issue/PYPATH

...

Successfully installed pip-24.1.2

$ pip3 install pip==24.2 --upgrade --target=/path/to/pip-upgrade-issue/PYPATH

...

Successfully installed pip-24.2

$ pip list | grep pip

pip                24.1.2

$ python3 -c "import pip; print(pip.__version__)"

24.2

I've investigated the issues and identified a possible cause for the instability of pip list when used with --target and PYTHONPATH.

How pip list works

1. pip uses os.listdir() to retrieve the names of the entries in the directory given by sys.path

2. pip sorts out the entries and evaluates them in order(However, os.listdir does not have any guaranteed ordering in its results) they appear in the loop, saving the package name and information directory from the first entry it encounters.
   https://github.com/pypa/pip/blob/24.1.2/src/pip/_internal/metadata/importlib/_envs.py#L52
   e.g) in case 2 below, pip list shows version 24.1.2 because pip-24.1.2-dist.info appears before pip-24.2.dist-info. when the entry for 24.2 comes into the loop, pip will simply continue, because it already has the package in _found_names.

The issue occurs because pip-target, by default, does not replace existing files/folders in the directory, resulting in multiple dist-info files being left in the directory. However, when the user specifies --upgrade, it should replace existing packages in the directory with new versions and remove the old dist-info file.

As you can see from the below code, it won't replace dist-info file.
https://github.com/pypa/pip/blob/24.1.2/src/pip/_internal/commands/install.py#L520

I have a fix for this issue, if the community confirms this as a bug, I can submit the PR.
Fix - enoekki@54dfb89

Expected behavior

No response

pip version

24.2 - probably all versions?

Python version

python3.11 - 3.12

OS

Mac & z/OS

How to Reproduce

rm -rf PYPATH*

mkdir PYPATH

export PYTHONPATH=$(pwd)/PYPATH

python3 -c "import sys; print(sys.path)"

# 1 - This does a fresh install to a given directory. This works fine.

export PIP_TARGET=$PYTHONPATH

pip3 install pip==24.1.2 &> PYPATH.1.INSTALL

pip3 list &> PYPATH.1.LIST

mv PYPATH PYPATH.1

mkdir PYPATH

# 2 - This does not install a newer version. This does not work. The expected result should be pip 24.1 displayed/installed

pip3 install pip==24.1.0 &> PYPATH.2.INSTALL

pip3 install pip==24.1.2 >> PYPATH.2.INSTALL 2>&1

pip3 list &> PYPATH.2.LIST

mv PYPATH PYPATH.2

# 3 - This does a fresh install to a directory, then installs a newer version, This does not work. The expected result should be pip 24.2 displayed/installed

pip3 install pip==24.1.2 &> PYPATH.3.INSTALL

pip3 install --upgrade pip==24.2 >> PYPATH.3.INSTALL 2>&1

pip3 list &> PYPATH.3.LIST

mv PYPATH PYPATH.3

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

As a general principle, --target is only really supported when being used to install into an empty target directory. We don't formally state that other uses aren't valid, but if they do work, to be honest it's more or less by accident.

Personally, I see little value in trying to fix issues like this on a case by case basis. Instead, we should either formally desupport everything except the case where the target directory is empty (and check for that, and error if it's not true), or rewrite --target support to work like a proper install scheme (see #11366 for some background on this, although very little work has actually been done on it). In spite of the fact that it was me who raised #11366, I'm neutral on which of these two options we should take.

[enoekki]

@pfmoore I think it would be best to go with the first option and formally announce it so that users don't get confused.
Although rewriting the --target to follow the proper install scheme might be the best approach, it could take a lot of time.
for now, I think we should go with the first option to reduce user confusion until the target can be properly supported again.
If you agree, I will implement the first option and submit a new PR.

[pfmoore]

I'm pretty sure that actually blocking --target on a non-empty directory would break a lot of existing code. It may be flaky, but experience shows that people are likely to use it anyway (avoiding the cases where it doesn't work) and we can't break those users without (a) a deprecation period, and (b) some form of migration path.

A documentation PR stating that supplying a non-empty directory to --target is "use at your own risk" and may change behaviour or break in some (unspecified) future version of pip is probably the best that we can reasonably do in the short term.