pipx doesn't respect manually selected point releases on upgrade

[tobwen]

Describe the bug

In the following example, the system's headers and library of gdal are in v3.2.2. Latest gdal is on v3.3.2, so is pygdal looking for this version, can't find it and breaks. Even when installed with pygdal==3.2.2.*, pipx upgrade doesn't respect this limitation and the upgrade fails, since the installer can't find latest gdal:

_main__.GDALConfigError: Version mismatch 3.2.2 != 3.3.2

How to reproduce

$ apt-install libgdal-dev gdal-bin gdal-data # should be 3.2.2 on Debian
$ pipx install --include-deps pygdal==3.2.2.* # works
$ pipx upgrade pygdal # fails

It gets worse when pygdal has been injected into another package. This completely fails then:

$ pipx upgrade --include-injected --force mapproxy

Expected behavior

Read package_or_url from pipx_metadata.json and limit it to the version stated in there.

Workarounds

pipx upgrade --pip-args 'install pygdal==3.2.2.*' pygdal
pipx upgrade --include-injected --pip-args 'install pygdal==3.2.2.*' mapproxy

~/.local/pipx/venvs/pygdal/pipx_metadata.json

{
    "injected_packages": {},
    "main_package": {
        "app_paths": [],
        "app_paths_of_dependencies": {
            "numpy": [
                {
                    "__Path__": "/home/tobwen/.local/pipx/venvs/pygdal/bin/f2py",
                    "__type__": "Path"
                },
                {
                    "__Path__": "/home/tobwen/.local/pipx/venvs/pygdal/bin/f2py3",
                    "__type__": "Path"
                },
                {
                    "__Path__": "/home/tobwen/.local/pipx/venvs/pygdal/bin/f2py3.9",
                    "__type__": "Path"
                }
            ]
        },
        "apps": [],
        "apps_of_dependencies": [
            "f2py",
            "f2py3",
            "f2py3.9"
        ],
        "include_apps": true,
        "include_dependencies": true,
        "package": "pygdal",
        "package_or_url": "pygdal==3.2.2.*",
        "package_version": "3.2.2.10",
        "pip_args": [],
        "suffix": ""
    },
    "pipx_metadata_version": "0.2",
    "python_version": "Python 3.9.2",
    "venv_args": []
}

[dukecat0]

Currently pipx upgrade will ignore any version specifiers and upgrade the package to the newest version.

Maybe we can introduce a new option (probably --include-version-spec?) for users to upgrade their packages with version specifiers? i.e. pip install -U package==20.0.* instead of pip install -U package.

[uranusjr]

Do we save the original user request in metadata? (I don’t quite remember) If we do, we should make pipx upgrade respect that original input, and introduce a new flag (say pip upgrade --spec) to ignore existing constraints.

[dukecat0]

Do we save the original user request in metadata?

package_or_url is the original user request, and pipx upgrade actually respects it. However, packages specifiers and markers are removed by this function:

pipx/src/pipx/package_specifier.py

Lines 176 to 188 in 71fe102

	def parse_specifier_for_upgrade(package_spec: str) -> str:
	"""Return package_or_url suitable for pip upgrade
	Specifically:
	* Strip any version specifiers (e.g. package == 1.5.4)
	* Strip any markers (e.g. python_version > 3.4)
	* Convert local paths to absolute paths
	"""
	parsed_package = _parse_specifier(package_spec)
	package_or_url = _parsed_package_to_package_or_url(
	parsed_package, remove_version_specifiers=True
	)
	return package_or_url

and introduce a new flag (say pip upgrade --spec) to ignore existing constraints.

EDIT: Or we should simply introduce --include-version-spec? Since pipx upgrade also has --include-injected option, and it won't work if injected packages contain version specifiers like package==1.0.0. Probably we can allow --include-version-spec if * exists in the version specifier?

[jaklan]

I would really love to have any solution for that issue - I used to run pipx upgrade-all automatically, but recently I wanted to maintain two Poetry versions: 1.1.x and 1.2.x. I've installed the previous one with pipx install "poetry<1.2" --suffix 1.1, but unfortunately when I run the upgrade comment - the constraint is not respected and it's being updated to 1.2.x, so each time I have to reinstall it.

[jaklan]

Hah, it seems I can help myself here - there's some workaround:
pipx install poetry --pip-args 'install poetry<1.2' --suffix 1.1
This way the constraint is preserved (so actually that's almost the same like the workaround from the first comment).

Anyway, I believe pipx install "poetry<1.2" --suffix 1.1 workflow is much more intuitive.

[dukecat0]

Maybe #891 can help with this case. Run pipx pin poetry1.1 and then pipx upgrade-all will not upgrade it.

(But I am still implementing pipx pin and pipx unpin, so I guess it will be available in the next release.)

[jaklan]

(But I am still implementing pipx pin and pipx unpin, so I guess it will be available in the next release.)

@dukecat0 what's the status of that? 😉

[dukecat0]

Hi @jaklan, I've implemented the logic of it before, but seems like the changes were dismissed accidentally, so I will try to work on it again.

[jaklan]

Great to hear, that would be a huge improvement! Btw, for pipx upgrade-all I believe the logic is quite simple - upgrade all apps except the pinned ones, but what is the plan for pipx upgrade behaviour? Would it fail and suggest to use pipx upgrade --force, fail and suggest pipx unpin && pipx upgrade, or some other approach?

[dukecat0]

fail and suggest pipx unpin && pipx upgrade

IIRC I used this approach in my previous implementation.

[gaborbernat]

PR weclome.