Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(build): Python wheels workflow and build backend (#4428)
### Summary This PR is the spiritual successor to #4011. It implements a `scikit-build-core`-based python build-backend, making it possible to use `pip install .` to build from source; and it adds a Github workflow for building with `cibuildwheel` and publishing to pypi.org binary distributions (bdists) of the Python module / extensions / CLI tools for cpython 3.8-3.13, across major operating systems and architectures. When you `pip install OpenImageIO`, pip attempts to retrieve an OpenImageIO bdist from pypi.org for the host's platform / architecture / Python interpreter. If it can't find something appropriate, pip will attempt to build locally from the OpenImageIO source distribution (sdist), downloading and temporarily installing cmake and ninja if necessary. ### PEP-Compliant Packaging: `pyproject.toml` The `pyproject.toml` file is organized in three parts: 1. **Package metadata**: standard attributes identifying and describing the Python package, its run-time and build-time requirements, entry-points to executable scripts, and so forth. 2. **scikit-build-core options**: governs how `pip install ...` interacts with `cmake`. 3. **cibuildwheel options**: additional steps and considerations for building, packaging, and testing relocatable wheel build artifacts in isolated environments. ### Additions to `__ init __.py` Previously, we were using a custom OpenImageIO/__ init__.py file to help Python-3.8+ on Windows load the shared libraries linked by the Python module (i.e., the .dll files that live alongside oiiotool.exe under $PATH). This PR adds an additional method for loading the DLL path, necessitated by differences between pip-based and traditional CMake-based installs. It also adds a mechanism for invoking binary executables found in the .../site-packages/OpenImageIO/bin directory. This provides a means for exposing Python script "shims" for each CLI tool, installed to platform-specific locations under $PATH, while keeping the actual binaries in a static location relative to the dynamic libraries. Upshot is, in `pyproject.toml`, each item under `[project.scripts]` is turned into a Python script upon installation that behaves indistinguishably to the end user to the CLI binary executable of the same name. ### Relocatable Binary Distributions with `cibuildwheel` + `repairwheel` [cibuildwheel](https://github.com/pypa/cibuildwheel) is a widely-used tool for drastically streamlining the process of building, repairing, and testing Python wheels across platforms, architectures, interpreters, and interpreter versions. Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to "MinSizeRel" to optimize for size (instead of speed) -- this seems to shave ~1.5MB off each .whl's size, compared to "Release" ### "Wheels" Github workflow I straight-up copied `.github/workflows/wheel.yml` from OpenColorIO and made a few OIIO-specific modifications. When pushing a commit tagged v3*, the workflow will invoke a platform-agnostic "build sdist" (source distribution) task, followed by a series of tasks for building OIIO wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new libstdc++), and MacOS (x86_64 + arm64) and persisting build artifacts; followed finally by a task for publishing the build artifacts to pypi.org Note: For the sake of simplicity and troubleshooting, I've made as few changes to OpenColorIO's wheel.yml as I could get away with; but in the future, we can also build wheels for the PyPy interpreter, and possibly pyodide. Note: A "trusted publisher" must be set up on pypi.org. See https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/ ### Other Changes I made some minor adjustments to `pythonutils.cmake` and `fancy_add_executable.cmake` that only affect scikit-build-core-based installs: - -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property to point to the relative path to the dynamic libraries, for the Python module and CLI tools, respectively. This helps ensure that pip-based builds and installs from source (as opposed to installs from repaired, pre-built wheels) yield relocatable, importable packages, without needing to mess with $LD_LIBRARY_PATH etc. ## Tests `cibuildwheel` tests if `oiiotool --buildinfo` runs. If that command elicits code zero, it means the "oiiotool" Python script installed by the wheel is able to `import OpenImageIO`; that the actual binary executable `oiiotool` is properly packaged and exists in the expected location (e.g., at `.../site-packages/OpenImageIO/bin`); and that all runtime dependencies are found. ## Inspiration, Credit, Prior Art - @aclark4life's and @JeanChristopheMorinPerso's efforts + direction + discussion + advice. See [#3249](#3249), and [#4011](#4011), as well as JCM's [python_wheels](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels) and [python_wheels_windows](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels_windows) branches. This PR is an attempt to leverage OIIO-2.6+ self-building-dependency features with # 4011's minimalist and modern approach to packaging. - OpenColorIO -- I tried to copy as much as I could from @remia et al's fantastic work with all things wheels-related. The __init __.py modifications, the way we're wrapping the CLI tools, and the github Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about reinventing the wheel here. - @joaovbs96's help and patience with testing stuff on Windows. --------- Signed-off-by: Zach Lewis <zachcanbereached@gmail.com> Signed-off-by: Larry Gritz <lg@larrygritz.com> Signed-off-by: Anton Dukhovnikov <antond@wetafx.co.nz> Signed-off-by: Basile Fraboni <basile.fraboni@gmail.com> Signed-off-by: Vlad (Kuzmin) Erium <libalias@gmail.com> Signed-off-by: Joseph Goldstone <joseph.goldstone@mac.com> Signed-off-by: Darby Johnston <darbyjohnston@yahoo.com> Signed-off-by: Jeremy Retailleau <jeremy.retailleau@gmail.com> Signed-off-by: zachlewis <zachlewis@users.noreply.github.com> Co-authored-by: Larry Gritz <lg@larrygritz.com> Co-authored-by: Anton Dukhovnikov <antond@wetafx.co.nz> Co-authored-by: Basile Fraboni <basile.fraboni@gmail.com> Co-authored-by: Vlad (Kuzmin) Erium <libalias@gmail.com> Co-authored-by: Joseph Goldstone <joseph.goldstone@mac.com> Co-authored-by: Darby Johnston <darbyjohnston@yahoo.com> Co-authored-by: Jeremy Retailleau <buddly27@users.noreply.github.com> Co-authored-by: Jean-Christophe Morin <38703886+JeanChristopheMorinPerso@users.noreply.github.com>
- Loading branch information