Building and installing nanobind library with CMake

[yurybura]

I would like to implement a vcpkg port for the nanobind library. This package manager uses CMake internally and supports many target platforms (Windows, Linux, MacOS, etc.). Using a package manager, the nanobind library can be built once and used in many projects without rebuilding the dependencies.

Unfortunately nanobind doesn't support building and installing using CMake (Pip or Conda only). This is a big limitation because the C/C++ package manager doesn't manage Python packages.

I would suggest doing an alternative way of building a static or shared library (depending on the options requested), installing the binaries and headers, generating CMake configuration files using the helpers.

Related to microsoft/vcpkg#33401.
Related to #55, #291, #339.

[wjakob]

A few quick thoughts: nanobind is designed to be compiled on the fly for each project, and it's small enough that this takes literally a second. Distributing pre-compiled versions of nanobind is not something that I would endorse. (It would also be complicated, because you would have to do this for different platforms, Python versions, while also handling the various parameters that can be passed to nanobind during compilation).

In this way, nanobind is different from, say, libpng or other more standard libraries that are supported via headers and pre-compiled libraries. The CMake build system is part of the distribution, which is why adopting other build tools is tricky.

Now that said, nanobind can be imported through CMake. In fact, this is how the pip distribution mechanism works. The PyPI file contains the CMake infrastructure and the C++ headers and source files so that other packages can run find_package and then have access to the build system. If that is supoprted by vcpkg (don't know enough about it), then it's probably not too difficult to integrate.

[ekilmer]

I have a refactor-cmake branch that I've been working on (hoping to integrate upstream, if interested/acceptable) that refactors the CMake of this project to support packaging with vcpkg and other package managers.

Fair warning: the changes in that branch aren't exactly minimal (nor all necessary), but I'm happy to break it up into smaller pieces. I think opening a PR with that branch and then have people comment, break out, document, or clean up as needed into other smaller PRs would be a decent idea? What do you think @wjakob?

Importantly, I've kept support for compiling nanobind on the fly, i.e., my refactor installs the source files of nanobind and CMake to support finding/building, like how it's done now. While I think vcpkg should be fine with this, I also agree that it makes more sense to distribute nanobind this way for the reasons @wjakob mentions.

Again, I know there are quite a few changes, and I'm happy to break some of it up (or explain it) for incremental merging/review, if there's a chance for integrating and supporting this. I also tried to update the documentation (let me know if I can add anything else), but for now, the CI (and ccmake) should serve as documentation.

I've been testing for compatibility with three branches on the nanobind_example repo:

1. backward-compat branch to make sure we're backward compatible with the current code.
2. add_subdirectory-compat branch to ensure nanobind can still be used with CMake's add_subdirectory as a submodule.
3. vcpkg-support to show an example of how to integrate nanobind as a vcpkg port, along with some other native dependencies. I also added support for building (with vcpkg) on all cibuildwheel platforms and architectures (a bit difficult and messier than expected, but not the fault of nanobind, and there is almost certainly room for improvement/simplification here)

Thank you for the great project!!

[wjakob]

I was curious to see if a vcpkg package for nanobind now exists and couldn't find one. Are there remaining issues to be addressed?

[yurybura]

The PR is exist but haven't done yet. It should be tested with a new nanobind release.

[ekilmer]

I also just created a PR microsoft/vcpkg#35488. I need to investigate the failing Windows builds. Seems to only be when Python is built dynamically. It's probably my fault.

[ekilmer]

The vcpkg PR (microsoft/vcpkg#35488) CI is now passing, after adding the Py_DEBUG compilation flag to the debug builds for the CI test example package. I also added an allowed failure for Windows arm64 cross-compilation because there were issues with finding/running the Python executable.

I'll request review in vcpkg after a new nanobind release is made. I don't believe any new large changes will be needed, and if there are, then they can be incorporated as a patch in vcpkg and upstreamed into a new release at a later date.

[wjakob]

It will likely take a long while for the next release. Is there a particular reason to not proceed with the current one?