Updating Python from 3.9.1_4 to 3.9.1_8 causes No such file or directory

[jaraco]

Yesterday, I ran brew upgrade. After that, my shell (xonsh), as installed by pipx, stopped running. It would fail with:

login: /Users/jaraco/.local/bin/xonsh: No such file or directory

So I load up bash and inspect:

$ ls -la ~/.local/pipx/venvs/xonsh/bin/python3.9
lrwxr-xr-x   1 jaraco  primarygroup    69 Jan  5 11:39 python3.9 -> /Users/jaraco/.local/homebrew/Cellar/python@3.9/3.9.1_4/bin/python3.9

It seems python3.9 no longer points to a valid Python, and thus all pipx commands are now broken.

I was able to reinstall-all, which worked, but also emitted an error:

$ pipx reinstall-all
Failed to upgrade shared libraries
Traceback (most recent call last):
  File "/Users/jaraco/.local/homebrew/lib/python3.9/site-packages/pipx/shared_libs.py", line 92, in upgrade
    upgrade_process = run_subprocess(
  File "/Users/jaraco/.local/homebrew/lib/python3.9/site-packages/pipx/util.py", line 123, in run_subprocess
    completed_process = subprocess.run(
  File "/Users/jaraco/.local/homebrew/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/jaraco/.local/homebrew/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 947, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/jaraco/.local/homebrew/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1819, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/jaraco/.local/pipx/shared/bin/python'
uninstalled oathtool! ✨ 🌟 ✨
...
done! ✨ 🌟 ✨
$ ls -la ~/.local/pipx/venvs/xonsh/bin/python3.9
lrwxr-xr-x  1 jaraco  primarygroup  69 Feb 19 09:25 /Users/jaraco/.local/pipx/venvs/xonsh/bin/python3.9 -> /Users/jaraco/.local/homebrew/Cellar/python@3.9/3.9.1_8/bin/python3.9

I have pipx 0.16.0.0.

Is it the case that a reinstall-all is expected to be needed between patch updates to Python? I haven't experienced this issue on my other mac where I have Homebrew installed in the canonical location. Is there anything that one can do to avoid this failure when running brew upgrade?

[uranusjr]

Is your pipx also installed from Homebrew? If so, this is basically unfixable on pipx’s side due to how it swaps Python installations from under a Python application whenever the python@3 formula is upgraded. The only “fix” is to not use the Python interpreter from Homebrew (set pipx install --python to something else).

[itsayellow]

Because the symlink in the venv points to the full path for python3, and because Homebrew changes the path with each patch, unfortunately yes, you will need to run pipx reinstall-all for Homebrew updates.

[uranusjr]

It seems like the shared lib venv is broken as well though, and reinstall-all doesn't fix that (iiuc). Maybe pipx should automatically rebuild the shared lib venv if it finds the underlying interpreter is gone?

[itsayellow]

We now do upgrade the shared libraries during reinstall-all: #554

[itsayellow]

Oh I see I missed that error message on reinstall-all. Hmm that is weird.

I haven't seen that error before and I have to occasionally reinstall-all due to Homebrew python upgrades.

From the stacktrace though, yes it does look like upgrading the shared libraries is not working right.

OK, sorry now I'm up to speed @uranusjr . We probably need to replace shared_libs.upgrade with shared_libs.create in the code for reinstall-all.

[itsayellow]

One thing we could consider is to not resolve the symlinks all the way to the source for the python executable. This is how we get to the specific
/usr/local/Cellar/python@3.9/3.9.1_8/bin/python3
instead of just
/usr/local/bin/python3

I haven't figured out in the code exactly how this happens, but I do know that if I manually make a venv using python3 -m venv venv it links my python3 to /usr/local/bin/python3, not /usr/local/Cellar/python@3.9/3.9.1_8/bin/python3 which /usr/local/bin/python3 points to.

[uranusjr]

I believe full symlink resolution is a hard requirement of PEP 405 and cannot change.

[itsayellow]

I don't quite understand why we get to the path that we do. On my system, running /usr/local/bin/python3 yields a sys.executable value of
'/usr/local/opt/python@3.9/bin/python3.9'
If I use that to create a venv, I do in fact get a venv that is linked to
/usr/local/Cellar/python@3.9/3.9.1_8/bin/python3.9
but I don't know why. It isn't the ultimate resolution of the /usr/local/opt/python@3.9/bin/python3.9 path

[uranusjr]

PEP 405 virtual environments finds the “original” interpreter by the home value in pyvenv.cfg. That value must points to a valid Python installation prefix, and is obtained with sys.base_prefix. The location of the executable is irrelavant.`

[itsayellow]

pyvenv.cfg is created in the new venv. My question is how is the python executable is determined in the first place when creating the venv.

If I create a venv with

python3 -m venv myvenv

then I end up in the myvenv/bin with a python3 link to /usr/local/bin/python3

If I create a venv with

/usr/local/opt/python@3.9/bin/python3 -m venv myvenv

then I end up in the myvenv/bin with a python3 link to /usr/local/Cellar/python@3.9/3.9.1_8/bin/python3

How does that happen? Especially because /usr/local/opt/python@3.9/bin/python3 doesn't eventually link to /usr/local/Cellar/python@3.9/3.9.1_8/bin/python3?

[uranusjr]

Huh, that’s interesting. venv just uses sys._base_executable (which is like sys.executable but resolves nested virtual environments), so my best guess is there’s some Homebrew trickery going on.

[jaraco]

Is your pipx also installed from Homebrew?

No, but it is pip installed into that Homebrew Python system site-packages (i.e. ~/.local/homebrew/bin/python -m pip install pipx).

I did observe that my Python links seem to have been lost in the latest homebrew update as well. In particular, ~/.local/homebrew/bin/python3.9 was missing and I had to run brew link python to get that link back. I'm pretty sure it was there before the brew upgrade, which I did not babysit. This also means that ~/.local/homebrew/bin/python was pointing to a non-existent file. These concerns seem possibly implicated, as without them, python resolved to the system Python 2 and python3 resolved to the system Python 3.8 install, and certainly could have affected how pipx or venv resolved a prefix.

[itsayellow]

Huh, that’s interesting. venv just uses sys._base_executable (which is like sys.executable but resolves nested virtual environments), so my best guess is there’s some Homebrew trickery going on.

Ah yes, thanks, it is the difference between sys.executable and sys._base_executable. So the difference in resolving those gives us a different answer (seemingly not necessarily dependent on where the system python3 symlinks resolve to?)

I'm wondering if it's possible for us to get a link to a python in our venvs that's more upgrade-independent, e.g. for example not linking to the exact Homebrew python3.9 update.

> python3
Python 3.9.1 (default, Feb  3 2021, 07:38:02) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/usr/local/opt/python@3.9/bin/python3.9'
>>> sys._base_executable
'/usr/local/bin/python3'
>>> exit()
> /usr/local/opt/python@3.9/bin/python3
Python 3.9.1 (default, Feb  3 2021, 07:38:02) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/usr/local/opt/python@3.9/bin/python3.9'
>>> sys._base_executable
'/usr/local/Cellar/python@3.9/3.9.1_8/bin/python3'
>>> 

[itsayellow]

@jaraco one workaround would be to set PIPX_DEFAULT_PYTHON to something that won't change for Homebrew upgrades, possibly something like /usr/local/bin/python3.

> PIPX_DEFAULT_PYTHON=/usr/local/bin/python3 pipx install pylint  
  installed package pylint 2.6.2, Python 3.9.2
  These apps are now globally available
    - epylint
    - pylint
    - pyreverse
    - symilar
done! ✨ 🌟 ✨
> ll ~/.local/pipx/venvs/pylint/bin
total 96
-rw-r--r--  1 mclapp  staff  8834 Feb 20 22:03 Activate.ps1
-rw-r--r--  1 mclapp  staff  1915 Feb 20 22:03 activate
-rw-r--r--  1 mclapp  staff   864 Feb 20 22:03 activate.csh
-rw-r--r--  1 mclapp  staff  2004 Feb 20 22:03 activate.fish
-rwxr-xr-x  1 mclapp  staff   252 Feb 20 22:03 epylint*
-rwxr-xr-x  1 mclapp  staff   242 Feb 20 22:03 isort*
-rwxr-xr-x  1 mclapp  staff   276 Feb 20 22:03 isort-identify-imports*
-rwxr-xr-x  1 mclapp  staff   250 Feb 20 22:03 pylint*
-rwxr-xr-x  1 mclapp  staff   256 Feb 20 22:03 pyreverse*
lrwxr-xr-x  1 mclapp  staff     7 Feb 20 22:03 python@ -> python3
lrwxr-xr-x  1 mclapp  staff    22 Feb 20 22:03 python3@ -> /usr/local/bin/python3
lrwxr-xr-x  1 mclapp  staff     7 Feb 20 22:03 python3.9@ -> python3
-rwxr-xr-x  1 mclapp  staff   252 Feb 20 22:03 symilar*

[uranusjr]

Would that work? Even though this would give you a working python symlink, the venv's home would still point to a non-existent location and break when Homebrew removes the previous installation. So you'd probably end up with a python that runs but can't find its stdlib.

[itsayellow]

I think it should be fine, this is what pyvenv.cfg is if you use /usr/local/bin/python3:

home = /usr/local/bin
include-system-site-packages = false
version = 3.9.2

There's something much different that happens when using /usr/local/bin/python3 as the python to setup a venv, instead of what pipx gets from sys.executable which is /usr/local/opt/python@3.9/bin/python3.9 to setup a venv.

[jaraco]

Thanks for the quick response on this. I’ll keep an eye out next time I do an update in this environment.

[itsayellow]

There's something much different that happens when using /usr/local/bin/python3 as the python to setup a venv, instead of what pipx gets from sys.executable which is /usr/local/opt/python@3.9/bin/python3.9 to setup a venv.

I've been doing experiments and am still slightly puzzled where sys._base_executable comes from, although I have a suspicion it's somehow related to the use on macOS Framework builds of __PYENV_LAUNCHER__.

[jaraco]

I'm still experiencing this issue. I've added the following to by shell init config:

# workaround for https://github.com/pypa/pipx/issues/633
export PIPX_DEFAULT_PYTHON=$(which python3)

And run pipx reinstall-all. Hopefully that will bypass the issue going forward.