pyodide xbuildenv install won't work if installed in uv-managed virtual environments

[agriyakhetarpal]

Description

I stumbled into this issue where installing pyodide-build in a uv-managed venv and then running pyodide xbuildenv install <...> to set up a cross-build environment fails with the following logs, because we hardcode the pip install <...> command here:

pyodide-build/pyodide_build/xbuildenv.py

Lines 297 to 312 in f78d587

	host_site_packages = self._host_site_packages_dir(xbuildenv_pyodide_root)
	host_site_packages.mkdir(exist_ok=True, parents=True)
	result = subprocess.run(
	[
	"pip",
	"install",
	"--no-user",
	"-t",
	str(host_site_packages),
	"-r",
	str(xbuildenv_root / "requirements.txt"),
	],
	capture_output=True,
	encoding="utf8",
	)

and uv removes the pip executable wrapper from its environments when activated so that a system pip isn't used in favour of uv pip:

Tap to show logs

❯ pyodide xbuildenv install 0.27.0a2
Downloading Pyodide cross-build environment from https://github.com/pyodide/pyodide/releases/download/0.27.0a2/xbuildenv-0.27.0a2.tar.bz2                    
Installing Pyodide cross-build environment                                                                                                                   
╭──────────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────────╮
│ /Users/agriyakhetarpal/Desktop/numcodecs/venv/lib/python3.12/site-packages/pyodide_build/cli/xbuildenv.py:61 in _install                                  │
│                                                                                                                                                           │
│    58 │   if url:                                                                                                                                         │
│    59 │   │   manager.install(url=url, force_install=force_install)                                                                                       │
│    60 │   else:                                                                                                                                           │
│ ❱  61 │   │   manager.install(version=version, force_install=force_install)                                                                               │
│    62 │                                                                                                                                                   │
│    63 │   typer.echo(f"Pyodide cross-build environment installed at {path.resolve()}")                                                                    │
│    64                                                                                                                                                     │
│                                                                                                                                                           │
│ /Users/agriyakhetarpal/Desktop/numcodecs/venv/lib/python3.12/site-packages/pyodide_build/xbuildenv.py:221 in install                                      │
│                                                                                                                                                           │
│   218 │   │   except Exception as e:                                                                                                                      │
│   219 │   │   │   # if the installation failed, remove the downloaded directory                                                                           │
│   220 │   │   │   shutil.rmtree(download_path)                                                                                                            │
│ ❱ 221 │   │   │   raise e                                                                                                                                 │
│   222 │   │                                                                                                                                               │
│   223 │   │   return xbuildenv_pyodide_root                                                                                                               │
│   224                                                                                                                                                     │
│                                                                                                                                                           │
│ /Users/agriyakhetarpal/Desktop/numcodecs/venv/lib/python3.12/site-packages/pyodide_build/xbuildenv.py:208 in install                                      │
│                                                                                                                                                           │
│   205 │   │   │   │   logger.info("Installing Pyodide cross-build environment")                                                                           │
│   206 │   │   │   │                                                                                                                                       │
│   207 │   │   │   │   if not skip_install_cross_build_packages:                                                                                           │
│ ❱ 208 │   │   │   │   │   self._install_cross_build_packages(                                                                                             │
│   209 │   │   │   │   │   │   xbuildenv_root, xbuildenv_pyodide_root                                                                                      │
│   210 │   │   │   │   │   )                                                                                                                               │
│   211                                                                                                                                                     │
│                                                                                                                                                           │
│ /Users/agriyakhetarpal/Desktop/numcodecs/venv/lib/python3.12/site-packages/pyodide_build/xbuildenv.py:299 in _install_cross_build_packages                │
│                                                                                                                                                           │
│   296 │   │   """                                                                                                                                         │
│   297 │   │   host_site_packages = self._host_site_packages_dir(xbuildenv_pyodide_root)                                                                   │
│   298 │   │   host_site_packages.mkdir(exist_ok=True, parents=True)                                                                                       │
│ ❱ 299 │   │   result = subprocess.run(                                                                                                                    │
│   300 │   │   │   [                                                                                                                                       │
│   301 │   │   │   │   "pip",                                                                                                                              │
│   302 │   │   │   │   "install",                                                                                                                          │
│                                                                                                                                                           │
│ /opt/homebrew/Caskroom/miniforge/base/lib/python3.12/subprocess.py:548 in run                                                                             │
│                                                                                                                                                           │
│    545 │   │   kwargs['stdout'] = PIPE                                                                                                                    │
│    546 │   │   kwargs['stderr'] = PIPE                                                                                                                    │
│    547 │                                                                                                                                                  │
│ ❱  548 │   with Popen(*popenargs, **kwargs) as process:                                                                                                   │
│    549 │   │   try:                                                                                                                                       │
│    550 │   │   │   stdout, stderr = process.communicate(input, timeout=timeout)                                                                           │
│    551 │   │   except TimeoutExpired as exc:                                                                                                              │
│                                                                                                                                                           │
│ /opt/homebrew/Caskroom/miniforge/base/lib/python3.12/subprocess.py:1026 in __init__                                                                       │
│                                                                                                                                                           │
│   1023 │   │   │   │   │   self.stderr = io.TextIOWrapper(self.stderr,                                                                                    │
│   1024 │   │   │   │   │   │   │   encoding=encoding, errors=errors)                                                                                      │
│   1025 │   │   │                                                                                                                                          │
│ ❱ 1026 │   │   │   self._execute_child(args, executable, preexec_fn, close_fds,                                                                           │
│   1027 │   │   │   │   │   │   │   │   pass_fds, cwd, env,                                                                                                │
│   1028 │   │   │   │   │   │   │   │   startupinfo, creationflags, shell,                                                                                 │
│   1029 │   │   │   │   │   │   │   │   p2cread, p2cwrite,                                                                                                 │
│                                                                                                                                                           │
│ /opt/homebrew/Caskroom/miniforge/base/lib/python3.12/subprocess.py:1955 in _execute_child                                                                 │
│                                                                                                                                                           │
│   1952 │   │   │   │   │   if errno_num != 0:                                                                                                             │
│   1953 │   │   │   │   │   │   err_msg = os.strerror(errno_num)                                                                                           │
│   1954 │   │   │   │   │   if err_filename is not None:                                                                                                   │
│ ❱ 1955 │   │   │   │   │   │   raise child_exception_type(errno_num, err_msg, err_filename)                                                               │
│   1956 │   │   │   │   │   else:                                                                                                                          │
│   1957 │   │   │   │   │   │   raise child_exception_type(errno_num, err_msg)                                                                             │
│   1958 │   │   │   │   raise child_exception_type(err_msg)                                                                                                │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: 'pip'

Steps to reproduce

uv venv venv --python 3.X
source venv/bin/activate
uv pip install pyodide-build
pyodide xbuildenv install 0.27.0a2

Potential fix

A simple fix is to add a small try-catch block that runs another uv pip install command in a subprocess in case the pip one fails – I implemented this locally, and it seems to work well. We could go further and also check what is available beforehand by using shutil.which().

Additional comments

I'm not sure if this is a valid issue (please feel free to close if it is), but since uv is gaining popularity in the Python ecosystem, might it be worth exploring the short (albeit hacky) fix I suggested above? I can work with `virtualenv'- managed environments for some time – though, I'm sure someone is going to come across this in the same way as I did.

The other thing I'd say is that it's always better to be explicit about the Python interpreter being used, so we should at least change the use of this command (and uses elsewhere), to use f"{sys.executable}", "-m", "pip" to avoid the chance of breaking external environments.

[hoodmane]

If [sys.executable, "-m", "pip", ...] works, that sounds good to me. Testing ahead of time with shutil.which() would also be okay with me. I don't like catching an error in subprocess and retrying.

[ryanking13]

[sys.executable, "-m", "pip", ...] sounds good to me too.