Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support python3-dll-a free-threaded generation #4749

Merged
merged 1 commit into from
Dec 2, 2024

Conversation

davidhewitt
Copy link
Member

Companion to PyO3/python3-dll-a#82, which fixes the generate-import-lib feature to correctly generate a library with the free-threaded symbols.

Without this fix, a build attempting to generate import libs for the free-threaded interpreter will fail with link errors:

Example build failure
$ /Users/david/.virtualenvs/pyo3-3.13t/bin/maturin build --target x86_64-pc-windows-msvc -i 3.13t
🔗 Found pyo3 bindings
🐍 Found CPython 3.13t
📡 Using build options features from pyproject.toml
  Manifest [0s] ============================================================ 11.39 MiB/11.39 MiB 📥 downloaded
  CRT.headers [0s] =========================================================== 17.50 MiB/17.50 MiB 📦 splatted
  CRT.libs.x86_64.desktop [0s] ============================================= 183.89 MiB/183.89 MiB 📦 splatted
  CRT.libs.x86_64.store [0s] ================================================= 65.38 MiB/65.38 MiB 📦 splatted
  CRT.libs.aarch64.desktop [0s] ============================================ 211.59 MiB/211.59 MiB 📦 splatted
  CRT.libs.aarch64.store [0s] ============================================== 114.20 MiB/114.20 MiB 📦 splatted
  SDK.headers.all.none [0s] ================================================== 89.79 MiB/89.79 MiB 📦 splatted
  SDK.headers.all.store [0s] =============================================== 219.85 MiB/219.85 MiB 📦 splatted
  SDK.headers.x86_64.none [0s] =============================================== 45.17 KiB/45.17 KiB 📦 splatted
  SDK.headers.aarch64.none [0s] ============================================ 103.02 KiB/103.02 KiB 📦 splatted
  SDK.libs.x86_64 [0s] ======================================================= 71.15 MiB/71.15 MiB 📦 splatted
  SDK.libs.aarch64 [0s] ==================================================== 131.90 MiB/131.90 MiB 📦 splatted
  SDK.libs.store.all [0s] ================================================== 116.26 MiB/116.26 MiB 📦 splatted
  SDK.ucrt.all [0s] ======================================================== 241.38 MiB/241.38 MiB 📦 splatted   Compiling target-lexicon v0.12.16
   Compiling python3-dll-a v0.2.10
   Compiling once_cell v1.20.2
   Compiling proc-macro2 v1.0.87
   Compiling unicode-ident v1.0.13
   Compiling libc v0.2.159
   Compiling autocfg v1.4.0
   Compiling heck v0.5.0
   Compiling cfg-if v1.0.0
   Compiling indoc v2.0.5
   Compiling unindent v0.2.3
   Compiling memoffset v0.9.1
   Compiling quote v1.0.37
   Compiling syn v2.0.79
   Compiling pyo3-build-config v0.23.2 (/Users/david/Dev/pyo3/pyo3-build-config)
   Compiling pyo3-macros-backend v0.23.2 (/Users/david/Dev/pyo3/pyo3-macros-backend)
   Compiling pyo3-ffi v0.23.2 (/Users/david/Dev/pyo3/pyo3-ffi)
   Compiling pyo3 v0.23.2 (/Users/david/Dev/pyo3)
   Compiling pyo3-macros v0.23.2 (/Users/david/Dev/pyo3/pyo3-macros)
   Compiling pyo3-scratch v0.1.0 (/Users/david/Dev/pyo3-scratch)
error: linking with `lld-link` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/david/.virtualenvs/pyo3-3.13t/bin:/Users/david/.vscode/extensions/ms-python.python-2024.20.0-darwin-arm64/python_files/deactivate/zsh:/Users/david/.virtualenvs/pyo3-3.12/bin:/Users/david/.local/state/fnm_multishells/18789_1732889105618/bin:/Users/david/.pyenv/shims:/Users/david/.vscode/extensions/ms-python.python-2024.20.0-darwin-arm64/python_files/deactivate/zsh:/Users/david/.virtualenvs/pyo3-3.12/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/Library/Frameworks/Python.framework/Versions/3.11/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/david/.vscode/extensions/ms-python.python-2024.20.0-darwin-arm64/python_files/deactivate/zsh:/Users/david/.virtualenvs/pyo3-3.12/bin:/Users/david/.local/state/fnm_multishells/48683_1732139667691/bin:/Users/david/Library/Python/3.10/bin:/Users/david/.fly/bin:/Users/david/.pyenv/bin:/Users/david/.local/bin:/Users/david/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/Library/Frameworks/Python.framework/Versions/3.11/bin:/Users/david/.cargo/bin:/Users/david/Library/Caches/cargo-xwin:/opt/homebrew/opt/llvm/bin" VSLANG="1033" "lld-link" "-flavor" "link" "/DEF:/var/folders/z2/3t02d9cn6gxg11n5yd9yk3wm0000gn/T/rustc2DywIY/lib.def" "/NOLOGO" "/var/folders/z2/3t02d9cn6gxg11n5yd9yk3wm0000gn/T/rustc2DywIY/symbols.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.02et0owkctrmvfz0dmcic5d7g.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.05m0kjpg46g339e1vjyhtr1a3.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.0gv6ppfiwqjb2k0owwnr9qk7s.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.0v3hqzjbwzyqpvhr328mo1ww1.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.3d1hkur3kmdx2wpiadwn9t2gf.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.5i0wevsvp9z3fruoy54j5eqrv.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.6n0i2ikz0wi3wlqz77bvv2uqs.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.b9t4aemh1lhqsbu27uorkwlq7.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.bd7r31mqecl3v83f2twep3mnj.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.bwtn1htiiqu10r8o4swzsxgbb.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.cxrj19bft77hw7o67tyxh3qbd.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.e1zvwjwr5yhusuqg5wbsniftz.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.e7oc6yuhe5qnfvjd0q6q4hhmf.rcgu.o" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/libpyo3-b289257478b5385f.rlib" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/libcfg_if-d0ce6dd80f2f48e8.rlib" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/libmemoffset-3a6bc25fb0ac8bff.rlib" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/libonce_cell-e27dcc30812c3bcf.rlib" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/libpyo3_ffi-65c802c964a36bd7.rlib" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/liblibc-597c4c4e65d94977.rlib" "/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/libunindent-1891a7bfdae04cc5.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/libstd-2df1f22abef96888.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/libpanic_unwind-7fa781213a0698f8.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/libwindows_targets-2440cb72ce7deb9b.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/librustc_demangle-f04b9120076f20fa.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/libstd_detect-b521ee511095af2f.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/libhashbrown-124aa6c4c6ef4b4c.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/librustc_std_workspace_alloc-c86a42f7194744c8.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/libunwind-a416069596473508.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/libcfg_if-e246a9218bd1ed0e.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/liballoc-8f9b5fcbcd27c22e.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/librustc_std_workspace_core-65178e86c6c71ba8.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/libcore-fbeb171b69c59b37.rlib" "/Users/david/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/x86_64-pc-windows-msvc/lib/libcompiler_builtins-e3a3e7896142045d.rlib" "python313.lib" "legacy_stdio_definitions.lib" "kernel32.lib" "kernel32.lib" "advapi32.lib" "ntdll.lib" "userenv.lib" "ws2_32.lib" "dbghelp.lib" "/defaultlib:msvcrt" "/NXCOMPAT" "/LIBPATH:/Users/david/Library/Caches/cargo-xwin/xwin/crt/lib/x86_64" "/LIBPATH:/Users/david/Library/Caches/cargo-xwin/xwin/sdk/lib/um/x86_64" "/LIBPATH:/Users/david/Library/Caches/cargo-xwin/xwin/sdk/lib/ucrt/x86_64" "/LIBPATH:/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/build/pyo3-ffi-5a070067b51aacc2/out/lib" "/OUT:/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.dll" "/OPT:REF,NOICF" "/DLL" "/IMPLIB:/Users/david/Dev/pyo3-scratch/target/x86_64-pc-windows-msvc/debug/deps/pyo3_scratch.dll.lib" "/DEBUG" "/PDBALTPATH:%_PDB%"
  = note: lld-link: error: undefined symbol: PyCriticalSection_End
          >>> referenced by /Users/david/Dev/pyo3/src/sync.rs:468
          >>>               libpyo3-b289257478b5385f.rlib(pyo3-b289257478b5385f.7q0hxo5wi51vmvn3nkkwjadpy.rcgu.o):(_$LT$pyo3..sync..with_critical_section..Guard$u20$as$u20$core..ops..drop..Drop$GT$::drop::he26faa4e81b86f55)
          
          lld-link: error: undefined symbol: PyUnstable_Module_SetGIL
          >>> referenced by /Users/david/Dev/pyo3/src/types/module.rs:557
          >>>               libpyo3-b289257478b5385f.rlib(pyo3-b289257478b5385f.5bo3y7fokwn52hwe7ta23uzl9.rcgu.o):(_$LT$pyo3..instance..Bound$LT$pyo3..types..module..PyModule$GT$$u20$as$u20$pyo3..types..module..PyModuleMethods$GT$::gil_used::hce71703d33b4e5ef)
          >>> referenced by /Users/david/Dev/pyo3/src/impl_/pymodule.rs:153
          >>>               libpyo3-b289257478b5385f.rlib(pyo3-b289257478b5385f.7x4tmlb1l4x543cf0qix0t7v5.rcgu.o):(pyo3::impl_::pymodule::ModuleDef::make_module::_$u7b$$u7b$closure$u7d$$u7d$::h5229d7969c833544)
          

error: could not compile `pyo3-scratch` (lib) due to 1 previous error
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `env -u CARGO AR_x86_64_pc_windows_msvc="llvm-lib" BINDGEN_EXTRA_CLANG_ARGS_x86_64_pc_windows_msvc="-I/Users/david/Library/Caches/cargo-xwin/xwin/crt/include -I/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/ucrt -I/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/um -I/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/shared" CARGO_ENCODED_RUSTFLAGS="-C\u{1f}linker-flavor=lld-link\u{1f}-Lnative=/Users/david/Library/Caches/cargo-xwin/xwin/crt/lib/x86_64\u{1f}-Lnative=/Users/david/Library/Caches/cargo-xwin/xwin/sdk/lib/um/x86_64\u{1f}-Lnative=/Users/david/Library/Caches/cargo-xwin/xwin/sdk/lib/ucrt/x86_64" CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER="lld-link" CC_x86_64_pc_windows_msvc="clang-cl" CFLAGS_x86_64_pc_windows_msvc="--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/crt/include /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/ucrt /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/um /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/shared  " CL_FLAGS="--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/crt/include /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/ucrt /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/um /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/shared " CMAKE_GENERATOR="Ninja" CMAKE_SYSTEM_NAME="Windows" CMAKE_TOOLCHAIN_FILE_x86_64_pc_windows_msvc="/Users/david/Library/Caches/cargo-xwin/cmake/x86_64-pc-windows-msvc-toolchain.cmake" CXXFLAGS_x86_64_pc_windows_msvc="--target=x86_64-pc-windows-msvc -Wno-unused-command-line-argument -fuse-ld=lld-link /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/crt/include /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/ucrt /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/um /imsvc/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/shared  " CXX_x86_64_pc_windows_msvc="clang-cl" PATH="/Users/david/.virtualenvs/pyo3-3.13t/bin:/Users/david/.vscode/extensions/ms-python.python-2024.20.0-darwin-arm64/python_files/deactivate/zsh:/Users/david/.virtualenvs/pyo3-3.12/bin:/Users/david/.local/state/fnm_multishells/18789_1732889105618/bin:/Users/david/.pyenv/shims:/Users/david/.vscode/extensions/ms-python.python-2024.20.0-darwin-arm64/python_files/deactivate/zsh:/Users/david/.virtualenvs/pyo3-3.12/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/Library/Frameworks/Python.framework/Versions/3.11/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Users/david/.vscode/extensions/ms-python.python-2024.20.0-darwin-arm64/python_files/deactivate/zsh:/Users/david/.virtualenvs/pyo3-3.12/bin:/Users/david/.local/state/fnm_multishells/48683_1732139667691/bin:/Users/david/Library/Python/3.10/bin:/Users/david/.fly/bin:/Users/david/.pyenv/bin:/Users/david/.local/bin:/Users/david/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/Library/Frameworks/Python.framework/Versions/3.11/bin:/Users/david/.cargo/bin:/Users/david/Library/Caches/cargo-xwin:/opt/homebrew/opt/llvm/bin" PYO3_CONFIG_FILE="/Users/david/Dev/pyo3-scratch/target/maturin/pyo3-config-x86_64-pc-windows-msvc-3.13.txt" RCFLAGS="-I/Users/david/Library/Caches/cargo-xwin/xwin/crt/include -I/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/ucrt -I/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/um -I/Users/david/Library/Caches/cargo-xwin/xwin/sdk/include/shared" TARGET_AR="llvm-lib" TARGET_CC="clang-cl" TARGET_CXX="clang-cl" "cargo" "rustc" "--features" "pyo3/extension-module" "--target" "x86_64-pc-windows-msvc" "--message-format" "json-render-diagnostics" "--manifest-path" "/Users/david/Dev/pyo3-scratch/Cargo.toml" "--lib"`

@davidhewitt davidhewitt force-pushed the python3-dll-a-freethreaded branch from 2d524fe to d4edae7 Compare November 30, 2024 14:15
@davidhewitt davidhewitt marked this pull request as ready for review November 30, 2024 14:16
@messense messense added this pull request to the merge queue Dec 2, 2024
Merged via the queue into PyO3:main with commit 2ed3bb4 Dec 2, 2024
46 checks passed
@davidhewitt davidhewitt deleted the python3-dll-a-freethreaded branch December 3, 2024 09:01
@davidhewitt davidhewitt changed the title support python3-dll-a free-threaded generatiton support python3-dll-a free-threaded generation Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants