Is there some way to create proxy for pybind11-exported class ?

[edimetia3d]

Hi,

What we want

We want to create proxy class of pybind11-exported class at python level, so we can enhance these exported class with python code. Because we want to code in python as much as possible, to leverage python's simplicity and power.

What we have done.

We have exported a class Animal to python, and we injected some more methods/ attributes to the class at runtime, like the code below.

import _wrap_module

def _Animal_print_me(self, extra_info):
    self._print_me() # exported by pybind11
    print(extra_info)

_wrap_module.Animal.print_me = _Animal_print_me

By doing this, we also injected this method to all subclass transparently, e.g., A c++ function Animal * GetFirstChild(Animal * animal) may return a python _wrap_module.Dog at runtime by auto down-casting, this Dog object will also find the print_me method by python attribute lookup system.

Question

As you see, the solution we used is not fancy, so I am wondering if there is some better solution.

Also, we had used an inheriting solution like below, but abandoned it soon,

import _wrap_module

class Animal(_wrap_module.Animal):

    def __init__(): ...

    def  print_me(): ....

This introduced some new problems, the first one is what we care

1. when we return an Animal * from a pybind11 exported function, like Animal * GetFirstChild(Animal * animal), it is returned as the _wrap_module.Animal (or _wrap_module.Dog by auto downcast), not the wrapper class we created. We had to wrap the returned value again at the call site, to retrieve the lost print_me method, this is tedious and err-prone, for we have hundreds of functions that will do things like this.
2. if we created a new wrapper class class Dog(_wrap_module.Dog):..., the wrapper class Dog is not a subclass of Animal at python side, thus can not find the print_me method, we had to add some tricky code to work around this.

Sincerely

[jiwaszki]

@edimetia3d that is a very good question! I was recently working on stuff and came up with this example. Before that I was using similar technique as you mentioned, "monkey-patching" all of the classes methods to the extend of patching classes with classes (🤯).

Here you have cpp header and source, very simple ones:

#include <iostream>
#include <string>

namespace base
{
    class Animal
    {
    public:
        Animal() = default;

        Animal(std::string name);

        void print();

    protected:
        std::string m_name;
    };

    class Dog : public Animal
    {
    public:
        Dog(std::string name, size_t age);

        size_t get_age();

    private:
        size_t m_age;
    };
} // namespace base

#include "code.hpp"

#include <pybind11/pybind11.h>

namespace base
{
  Animal::Animal(std::string name) : m_name(name){};

  void Animal::print()
  {
    std::cout << "I am " << Animal::m_name << std::endl;
  }

  Dog::Dog(std::string name, size_t age) : Animal(name), m_age(age){};

  size_t Dog::get_age()
  {
    return m_age;
  }
}

namespace py = pybind11;

PYBIND11_MODULE(mymodule, module)
{
  py::class_<base::Animal> animal(module, "Animal");

  animal.def(py::init<std::string>());
  animal.def("print", &base::Animal::print);

  py::class_<base::Dog, base::Animal> dog(module, "Dog");

  dog.def(py::init<std::string, size_t>());
  dog.def("get_age", &base::Dog::get_age);
}

And corresponding python code:

import mymodule

class Dog(mymodule.Dog):

    def print(self):
        super().print()  # mymodule.Dog.print(self)
        print("I am a pug.")

    def woof(self):
        self.print()
        print("I have {} years, woof!".format(self.get_age()))

animal = mymodule.Animal("Capybara")
dog = Dog("Lebron", 6)

animal.print()
print("=== Just print:")
dog.print()
print("=== Look at it:")
dog.woof()

Results from terminal:

I am Capybara
=== Just print:
I am Lebron
I am a pug.
=== Look at it:
I am Lebron
I am a pug.
I have 6 years, woof!

As you see it is actually possible to preserve parent's class methods and override them with extra code. However in more complex code bases it will be difficult to adjust C++ code to your actual needs, keep that in mind.

Inheritance of C++ classes is done according to documentation, please take a look at it: https://pybind11.readthedocs.io/en/stable/classes.html#inheritance-and-automatic-downcasting

The "cherry-on-top" is using super() from Python's world. This way you are not patching anything at all, clean solution.

However I would advise to use different naming for the classes wrapped in pybind and the ones created on python side. I know that module name will probably be different and so on. But it can create confusion in project. I would advise to use mymodule.PyDog instead of mymodule.Dog as well for other classes, just to mark the differences between them. Create one rule and follow it.

Hope this helps you! 🚀

PS: calling isinstance(dog, mymodule.Dog) and isinstance(dog, mymodule.Animal) are returning True as well. I didn't have time to check up on auto-downcast and pointer-return cases that you mentioned. If it is not working or you extend a solution, let me know here - I am very curious about it.

[jiwaszki]

Adding to solution if return type needs to be a wrapper, one of the solution is adding new constructor to bindings:

  dog.def(py::init([](base::Dog &dog) { return dog;} ));
  dog.def("new_child", [](base::Dog &self)
          { return new base::Dog("Boobie", 8); });

And wrapping returning type with it:

class Dog(mymodule.Dog):

    def new_child(self):
        return Dog(super().new_child())

[edimetia3d]

Thank you for your suggestion. But I think it might not solve my problem.

Your solution is basically same as what I mentioned, and we abandoned it. For example, in your code , you added a method woof to class Dog(mymodule.Dog), suppose we have a more specific "Dog", which is class Husky(mymodule.Husky) , this class Husky(mymodule.Husky) will not found the woof directly.

And, as I said, call wrapper function at call site is also unacceptable in our project.

However, we had indeed found a much better solution that leverage the pybind11's builtin type_caster, which let us to mimic the actual Proxy Pattern, It is much more clean. I will post it here later, and may open a issue to talk with developers to find if we could contribute it to upstream.

[edimetia3d]

Here, we had found a solution.

I want to first show what we had achieved.

Given the _zoo is a simple pybind11 module that has Animal and Dog class.

We created two proxy classes at python side, which will hold their own wrapped object by using the has-a logical, not is-a.

These proxy classes will follow the Proxy pattern, they could be used in any pybind11 exported function that requires an raw object.

And more, any pybind11 exported function will return the proxy object instead of raw object.

By using this, we could completely hide the pybind11-exported types, and extend these pybind11-exported types with pure python code in a more structural way.

import _zoo


def real_type(RealT):
    def deco(cls):
        cls.RawType = RealT
        return cls

    return deco


@real_type(_zoo.Animal)
class Animal:

    def __init__(self):
        self._raw_obj = self.RawType()

    def animal_new_method(self):
        print(f"animal new method with {self._raw_obj}")


@real_type(_zoo.Dog)
class Dog(Animal):
    def dog_new_method(self):
        print(f"dog new method with {self._raw_obj}")


# case 1, Create Python object directly

animal = Animal()
animal.animal_new_method()
_zoo.AccessAnimal(animal)  # wrapper object could be used with pybind11 exported method directly
_zoo.AccessAnimal(animal._raw_obj)  # same as before

dog = Dog()
dog.animal_new_method()  # inherit works
dog.dog_new_method()
_zoo.AccessAnimal(dog)
_zoo.AccessAnimal(dog._raw_obj)  # same as before
_zoo.AccessDog(dog)  # wrapper object could be used with pybind11 exported method directly
_zoo.AccessDog(dog._raw_obj)  # same as before
# case 2, Return Python object from C++

new_animal = _zoo.CreateAnimal()
assert isinstance(new_animal, Animal)
assert isinstance(new_animal._raw_obj, _zoo.Animal)
new_dog = _zoo.CreateDog()
assert isinstance(new_dog, Dog)
assert isinstance(new_dog._raw_obj, _zoo.Dog)

And the c++ pybind11 code is just based on the bulit-in type_caster

The c++ code behind the python wrapper is like this.
To make things clear, there is some hard coding, when you get the idea, you could extend it to a much more complicated/powerful one.

#include <pybind11/pybind11.h>

namespace py = pybind11;

/////////////////// Utils Begin /////////////////////////////////////////
struct WrapInfo {
  static constexpr const char *wrap_attr_name = "_raw_obj";
  static constexpr const char *module_name = "__main__";
  static constexpr const char *wrapper_cls_name = nullptr;
};
template <typename WrapInfo> py::handle Wrap(py::handle obj) {
  auto module = py::module::import(WrapInfo::module_name);
  py::object wrapper_cls = module.attr(WrapInfo::wrapper_cls_name);
  py::object wrapper = wrapper_cls.attr("__new__")(wrapper_cls);
  setattr(wrapper, WrapInfo::wrap_attr_name, obj);
  return wrapper.release();
}
template <typename WrapInfo> py::handle Unwrap(py::handle obj) {
  if (py::hasattr(obj, WrapInfo::wrap_attr_name)) {
    return getattr(obj, WrapInfo::wrap_attr_name);
  }
  return obj;
}

namespace pybind11 {
namespace detail {
template <class RawType, handle WRAP_HOOK(handle), handle UNWRAP_HOOK(handle)>
struct proxy_caster : public type_caster_base<RawType> {
  using base = type_caster_base<RawType>;

public:
  // use template to make this function ODR-use
  template <class = void> bool load(handle src, bool convert) {
    return base::load(UNWRAP_HOOK(src), convert);
  }

  template <class T>
  static handle cast(T &&src, return_value_policy policy, handle parent) {
    handle raw_obj = base::cast(std::forward<T>(src), policy, parent);
    auto ret = WRAP_HOOK(raw_obj);
    raw_obj.dec_ref();
    return ret;
  }
};
} // namespace detail
} // namespace pybind11
/////////////////// Utils End ///////////////////////////////////////////

class Animal {
public:
  virtual ~Animal() = default;
};

class Dog : public Animal {
public:
  ~Dog() override = default;
};

namespace pybind11 {
namespace detail {
struct AnimalWrapInfo : public WrapInfo {
  static constexpr const char *wrapper_cls_name = "Animal";
};
template <>
struct type_caster<Animal> : public proxy_caster<Animal, ::Wrap<AnimalWrapInfo>,
                                                 ::Unwrap<AnimalWrapInfo>> {};
struct DogWrapInfo : public WrapInfo {
  static constexpr const char *wrapper_cls_name = "Dog";
};
template <>
struct type_caster<Dog>
    : public proxy_caster<Dog, ::Wrap<DogWrapInfo>, ::Unwrap<DogWrapInfo>> {};
} // namespace detail
} // namespace pybind11

PYBIND11_MODULE(_zoo, m) {
  py::class_<Animal>(m, "Animal").def(py::init<>());
  py::class_<Dog, Animal>(m, "Dog").def(py::init<>());
  m.def("CreateAnimal", []() { return new Animal(); });
  m.def("CreateDog", []() { return new Dog(); });
  m.def("AccessAnimal",
        [](Animal *a) { printf("Raw animal %p accessed.\n", a); });
  m.def("AccessDog", [](Dog *d) { printf("Raw dog %p accessed.\n", d); });
}

[edimetia3d]

also, a __getattr__(self,item) may be added to the base class Animal , to forward unfound attribute to the self._raw_obj