Failing to compile numpy from source against a uv-managed interpreter (Cannot compile Python.h.)
#10558
Comments
|
The header file is available, e.g., on my machine at I wonder how their configure script seaches for it? |
|
I've just run into what I assume is the same issue trying to build python-flint with a uv-installed Python. I have made a simple demonstration repo for a project that has meson as build system and builds a Cython extension module: https://github.com/oscarbenjamin/meson_simple It builds fine with a system Python or pyenv Python but not with a uv python. Steps: I don't know if this is a new issue because I have not previously tried to build anything like this with a uv-installed Python. This is the meson configure output: $ meson setup build --wipe
The Meson build system
Version: 1.6.1
Source dir: /stuff/current/active/mesontest
Build dir: /stuff/current/active/mesontest/build
Build type: native build
Project name: meson-test
Project version: undefined
C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
Cython compiler for the host machine: cython (cython 3.0.11)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/stuff/current/active/mesontest/.venv/bin/python3)
Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
Run-time dependency python found: YES 3.13
Build targets in project: 1
Found ninja-1.10.1 at /usr/bin/ninja
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /usr/bin/ninja -C /stuff/current/active/mesontest/buildThe compile step fails with: $ meson compile -C build
ninja: Entering directory `/stuff/current/active/mesontest/build'
[2/3] Compiling C object meson_test/_meson_test.c....p/meson-generated_meson_test__meson_test.pyx.c.o
FAILED: meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o
cc -Imeson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p -Imeson_test -I../meson_test -I/install/include/python3.13 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c99 -O2 -g -fPIC -MD -MQ meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o -MF meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o.d -o meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o -c meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson_test/_meson_test.pyx.c
meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson_test/_meson_test.pyx.c:16:10: fatal error: Python.h: No such file or directory
16 | #include "Python.h"
| ^~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.The specific compiler command wrapped is: Running with a venv off of a system Python I get this command instead: Separating only the include paths it is: # uv python
-Imeson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p
-Imeson_test
-I../meson_test
-I/install/include/python3.13
# system python
-Imeson_test/_meson_test.cpython-310-x86_64-linux-gnu.so.p
-Imeson_test
-I../meson_test
-I/usr/include/python3.10
-I/usr/include/x86_64-linux-gnu/python3.10Somehow it seems that meson has been unable to locate the include directories I guess. |
|
Possibly @eli-schwartz could help with understanding how meson identifies the include directories. |
|
Thanks for the additional details. Looks like this comes from https://github.com/mesonbuild/meson/blob/dfb449d099d6f3066ca646af197bc7af85059a86/mesonbuild/dependencies/python.py#L369-L370 I don't really see a difference between the variables referenced Can you see the |
|
@oscarbenjamin your logs show |
|
Oh, actually after updating uv and reinstalling Python it works for the demo: |
|
Great to hear! Yeah the pkg-config file for these look similar too |
|
@neutrinoceros can you use |
|
I've confirmed I can build python-flint and run the whole test suite. The problems I was seeing were fixed by updating uv, using the updated uv to reinstall Python and then rebuilding the venv. Thanks @zanieb ! |
|
Looks like you were able to figure out the critical references we use, and it turns out everything works with current |
|
Thanks @eli-schwartz.
Yes in my testing but it would be good for @neutrinoceros to confirm. I just tried installing numpy from git as described in the OP and it seems to work okay: $ uv pip install git+https://github.com/numpy/numpy
Updated https://github.com/numpy/numpy (bbf4836a8)
Resolved 1 package in 34.85s
Built numpy @ git+https://github.com/numpy/numpy@bbf4836a82d255c2a2161d277abb4b6f5455ca6d
Prepared 1 package in 5m 54s
Installed 1 package in 19ms
+ numpy==2.3.0.dev0 (from git+https://github.com/numpy/numpy@bbf4836a82d255c2a2161d277abb4b6f5455ca6d)
$ python -c 'import numpy; print(numpy.__version__)'
2.3.0.dev0+git20250110.bbf4836If anyone wants to try that one at home then they would first need to install openblas e.g. something like |
Sorry I should have mentioned: I tried this with uv 0.5.14, which if I'm following correctly has the most recent updates in this area, so I didn't think it was necessary to try again with 0.5.18. I just tried now: (throwing |
|
What are the values for you that @zanieb described in #10558 (comment) ? |
|
|
I just realized that Running them again from a directory without a |
|
A bit random, but is the problem that there's a space in that include directory path? I've seen that elsewhere. If you delete |
|
Seeing the log file at /private/tmp/numpy/.mesonpy-xqqxtm9n/meson-logs/meson-log.txt would help to determine whether this is the issue. (I know that pip deletes the log file along with the entire build tree on build failures which is frustrating, but I have no idea what uv might do here.) |
I assume that this works with |
|
The logfile is created inside a subdirectory of the extracted sdist, and in at least some cases / versions, for building The compilation artifacts need to be available, at any rate. |
|
meson's logfile seems to indeed be removed before I get a chance to read it. I know that |
That's it ! |
When you disable build isolation you need to install the dependencies before building e.g. |
|
I'm aware. But "mesonpy" isn't the name of a PyPI package so I don't know where I'm supposed to get it from. |
|
The package is called meson-python. You can see it in the simpler demo repo I linked above https://github.com/oscarbenjamin/meson_simple/blob/53edd241263528b8551393bbb8a8237b5365880d/pyproject.toml#L1-L3 or in numpy's pyproject.toml: https://github.com/numpy/numpy/blob/bbf4836a82d255c2a2161d277abb4b6f5455ca6d/pyproject.toml#L1-L6 I'm not sure in the case of numpy specifically though because I think numpy at least used to vendor its own fork of meson-python. For local development with numpy I think it is probably now recommended to use |
|
Thanks for these pointers ! |
|
Great thanks everyone! |
I'm not able to build numpy from source against a uv-managed Python:
this seems similar to #10185, which was transferred from https://github.com/astral-sh/python-build-standalone, so I'm opening this ticket directly in uv's repo. This issue was already present with uv 0.5.13 and appears to reproduce with 0.5.18
setup:
then
A shorter path to the same error
The text was updated successfully, but these errors were encountered: