Interoperate between pybind11 and nanobind?

[qnzhou]

Is it possible to have python extensions built with pybind11 and extensions built with nanobind to interop?

For example, library A may expose A::Shape class via pybind11, and library B may have a function B::process_shape(A::Shape) which is bound via nanobind. Based on my limited testing, although both library's python binding compiles fine, there is no way of sending a A::Shape object to B::process_shape in python because of different binding libraries are used. Is this a hard limitation? I am asking because sometimes we do not have control of library A's binding choice, but we would love to use nanobind for library B.

As always, thanks for the great work on both pybind11 and nanobind!

[wjakob]

It's a hard limitation. nanobind can access a pybind11 object as generic Python object (in the sense of duck typing) and vice versa, but you won't be able to safely unpack it and do C++ calls. Having a tighter coupling would require sharing internal data structures between libraries, and that would ruin the point of nanobind (which is to avoid the complexity of pybind11's internal data structures).

I will also point out that interop is impossible even between various pybind11 and compiler versions, which illustrates how hard/messy this is.

[wbthomason]

As a follow-up to this question: is it possible to do interop only with return types? In other words, I have a library A that uses pybind11 to bind a type class A::Foo. In my library, I want to use nanobind to bind a function A::Foo make_a_foo(...), where I don't care about taking any instances of A::Foo in as arguments. I know (or at least believe) that this is possible if I'm also using pybind11 (with matched versions and compilers), but I'm not sure if it'll work across the two libraries.

[qnzhou]

I don't quite understand the desired use case here. Do you mean the A::Foo object created from make_a_foo() method will never be passed back to the C++ side? If so, I feel there is no inter-op happening here, the object can simply remain as an opaque object in python...

[wbthomason]

@qnzhou Sorry, I may be misunderstanding something. I'll try to clarify:

Library A has existing pybind11 bindings. I want to be able to return an instance of the same PyObject type those bindings create from my library, by linking against the Python extension library for A, without having to create my own bindings for A::Foo. I believe that this is possible if I use pybind11 for my library, but I'm wondering if it's possible if I'm using nanobind instead - I suspect not?

Thanks! I hope this helps explain what I'm trying to figure out.

[wjakob]

Whether you return values or take them as inputs does not matter much. If you expect nanobind to make some kind of association between a C++ type and a Python binding, and when that binding has been created in pybind11, it will not work.

What you can do is to have nanobind code receive a generic nb::object and pass that around or return it. But it does not sound like that's what you want to do.

In any case, pybind11 and nanobind are now sufficiently on par that you can probably just recompile your existing project with superficial changes to have nanobind-based bindings, and then that whole question will disappear.

[wbthomason]

Ok, thank you! Unfortunately, the other project I want to use isn't under my control, so I can't switch it to nanobind, but this is good to confirm my belief that I need to use pybind11 if I want easy interop. Thanks again, as well as for the excellent libraries!

[eirrgang]

I'm sorry to stray off-topic from nano bind, but, Will, I'm not sure Wenzel was promising that different compiled modules would necessarily be able to share C++ <-> Python mappings just because they _are_ both pybind11 modules. As of pybind11 ~2.1 I asked about this on pybind11 forums and someone clarified that the type mapping system is not necessarily compatible across pybind11 versions, and so it is (was?) hidden in an ABI-versioned C++ namespace. You may find that C++/Python type mappings are only shared with the same pybind11 version. I believe there is some discussion of this sort of thing in the pybind docs, though, and you should be able to get quick answers on the pybind11 Gitter channel. To bring this back to nano bind, though: Since nanobind embeds its extra stuff in the PyObject itself, separately compiled (and distributed) modules able to share type mappings when exchanging nb::objects? That's super cool!

[wjakob]

To bring this back to nano bind, though: Since nanobind embeds its extra stuff in the PyObject itself, separately compiled (and distributed) modules able to share type mappings when exchanging nb::objects? That's super cool!

Yes, that works. But, but, but -- the same caveats as in pybind11 apply. A newer nanobind release might change the data layout of some of the information that is stored within the PyObject (which is not just the C++ object but also extra internal stuff added by nanobind). Well, then it's no longer safe to access it from separate modules compiled with different nanobind versions, and nanobind must isolate them from each other to avoid undefined behavior. So the overall policy is quite similar to pybind11. This is called the ABI/internals version number, which is not the same as the pybind11 version number.

Since this project is still very young, the ABI version has been changing rather frequently(https://github.com/wjakob/nanobind/blob/master/src/nb_internals.cpp#L20). Almost at every release, so we're now at v6. It is anticipated that this project, too, will stabilize more over time, and then the ABI version will allow longer-term interoperability.

[qnzhou]

Yet another related question: Is it (supporting modules built using different versions of the binding library) also the reason that both nanobind and pybind11 hide symbols by default (as discussed here). If we force-unhide those symbols, we may run into symbol collision at load time when importing modules using a different version of nanobind.

[oremanj]

I have implemented mostly-complete nanobind/pybind interop in a way that doesn't require each library to understand the other's internals. (This was a prerequisite for embarking on a port of a large set of interdependent extension modules from pybind to nanobind at my employer, too large to do in one fell swoop.) It seems to work well and I'm wondering if there would be appetite for upstreaming it. The basic approach is to extend each library with the ability to "export" a

struct foreign_caster_hooks {
    // Context pointer to be passed in calls to from_python/to_python;
    // in pybind this is a detail::type_info pointer
    void* context;

    // Pointer to the Python type object that this C++ type maps to.
    // This is used mainly to allow determining the Python type name.
    PyTypeObject* pytype;

    // Extract a C++ object of this type from `pyobj`. Return a
    // pointer to it, or null if not possible. If `convert` is true,
    // be more willing to perform conversions. `from_python` may make
    // calls to `decref_later` to request that some Python objects
    // remain referenced until the returned C++ object is no longer
    // needed.
    void* (*from_python)(void* context, PyObject* pyobj, bool convert,
                         void (*decref_later)(PyObject*)) noexcept;

    // Wrap `cppobj` into a Python object using the given return value
    // policy. `parent` is relevant only if rvp == reference_internal.
    // Returns null if this is not possible, or a new reference
    // otherwise.
    PyObject* (*to_python)(void* context, void* cppobj, uint8_t rvp,
                           PyObject* parent) noexcept;
};

for its own types, and "import" one for a foreign (other-library-bound) type. These go in a new internals map (type_map<foreign_caster_hooks> foreign_types;) which is checked if there is no "native" type available to do a conversion. There is a little bit of API shear between pybind/nanobind but they're similar enough that it's totally possible to write the glue
functions. This approach doesn't support polymorphic downcasting or holder/smartptr casting yet, but seems amenable to being extended with such.

If you'd be interested in supporting an interface like this, let me know; I'll clean up what I have and submit a PR.

[mikelui]

@oremanj I'm in a similar predicament and would interested in such a solution. Without such interop we are stuck on pybind11

[wjakob]

Sorry for the late reply, I just saw this now. Are you able to implement this on top of nanobind, or does it require changes in the library? If it is sufficiently self-contained, then the nanobind documentation could link to your project to advertise it.

[oremanj]

The approach I implemented requires changes (including an ABI version bump) in both pybind11 and nanobind, but does not create a dependency of one on the other. The new codepath executes only when a conversion would otherwise fail, so I don't think it would adversely impact performance.