Introduce `return_value_policy_pack` (will need follow-on adjustments) #30011

rwgk · 2023-03-16T15:00:55Z

Description

The new py::return_value_policy_pack is a generalization of py::return_value_policy, to enable easy and/or automatic wrapping of container types like std::map<std::string, std::string> where the key type is bound to Python str and the value type to Python bytes. This can also be achieved with lambda functions, but becomes very cumbersome (noisy and difficult to maintain) for nested container types (e.g. std::map<std::string, std::pair<std::string, std::string>>).

The idea for return_value_policy_pack came from: https://github.com/google/clif/blob/a8815b2972cdf2e81516d0bded28e36564a120f8/clif/python/postconv.h#L49-L78

This PR will need follow-on adjustments, to replace return_value_policy policy with const return_value_policy_pack &rvpp in more places, but the current state of introducing the new feature is a useful resting point (was already used repeatedly in Google-internal global testing of PyCLIF-pybind11 developments).

This PR was transferred from pybind/pybind11#4499.

See also: discussions under pybind/pybind11#3838

Suggested changelog entry:

…google#4539) * Move try_as_void_ptr_capsule out from modified_type_caster_generic_load_impl. * Try fixing clangtidy

Currently only for string_caster, tuple_caster, map_caster

…_policy_pack).

…onal.h

…policies> args_policies;`

Core diff: ``` diff --git a/include/pybind11/attr.h b/include/pybind11/attr.h - bool Zonvert : 1; ///< True if the argument is allowed to convert when loading - bool Mone : 1; ///< True if None is allowed when loading + from_python_policies policies; ```

…onstexpr - value = func_wrapper(func_handle(std::move(func)), fpp.convert); + value = func_wrapper(func_handle(std::move(func)), fpp.rvpp); Passes: scons -j 24 selected_test_cpp=test_exceptions.cpp,test_return_value_policy_pack.cpp && /usr/bin/python3 $HOME/clone/pybind11_scons/run_tests.py ../pybind11 test_return_value_policy_pack -s -vv But many other tests broken because arg is non-constexpr and therefore this had to be commented out in cast.h: constexpr arg operator"" _a(const char *name, size_t) { return arg(name); }

…tional, variant). Also fix oversight: missing Python-side test for variant.

…idy auto fix.

….9, 5): ``` test_return_value_policy_pack.cpp:51:70: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] ```

… emitted by CUDA and ICC: CUDA 11.7 Ubuntu 22.04: ``` cast.h(1318): error google#186-D: pointless comparison of unsigned integer with zero ``` ICC latest x64 (Intel 2021.8.0.20221119): ``` cast.h(1318): error google#186: pointless comparison of unsigned integer with zero ```

…i.yml logs). ``` 2023-02-17T09:47:45.7114041Z D:\a\pybind11\pybind11\tests\test_factory_constructors.cpp:381:30: error: 'void operator delete(void*)' called on pointer returned from a mismatched allocation function [-Werror=mismatched-new-delete] 2023-02-17T09:47:45.7114692Z 381 | ::operator delete(p); 2023-02-17T09:47:45.7115055Z | ~~~~~~~~~~~~~~~~~^~~ 2023-02-17T09:47:45.7115444Z In lambda function, 2023-02-17T09:47:45.7116873Z inlined from 'pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>' at D:/a/pybind11/pybind11/include/pybind11/detail/init.h:376:33, 2023-02-17T09:47:45.7119232Z inlined from 'Return pybind11::detail::argument_loader<Args>::call_impl(Func&&, pybind11::detail::index_sequence<Is ...>, Guard&&) && [with Return = void; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&; long long unsigned int ...Is = {0, 1, 2}; Guard = pybind11::detail::void_type; Args = {pybind11::detail::value_and_holder&, int, int}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1563:37, 2023-02-17T09:47:45.7122014Z inlined from 'pybind11::detail::enable_if_t<std::is_void<_Dummy>::value, pybind11::detail::void_type> pybind11::detail::argument_loader<Args>::call(Func&&) && [with Return = void; Guard = pybind11::detail::void_type; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&; Args = {pybind11::detail::value_and_holder&, int, int}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1537:65, 2023-02-17T09:47:45.7125679Z inlined from 'pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>, void, pybind11::detail::value_and_holder&, int, int, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&&, void (*)(pybind11::detail::value_and_holder&, int, int), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:249:71, 2023-02-17T09:47:45.7130204Z inlined from 'static pybind11::handle pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>, void, pybind11::detail::value_and_holder&, int, int, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, int)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, int), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, int)>&&, void (*)(pybind11::detail::value_and_holder&, int, int), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>::_FUN(pybind11::detail::function_call&)' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:224:21: 2023-02-17T09:47:45.7132694Z D:\a\pybind11\pybind11\tests\test_factory_constructors.cpp:399:71: note: returned from 'static void* test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc::operator new(size_t)' 2023-02-17T09:47:45.7133385Z 399 | pyNoisyAlloc.def(py::init([](int i, int) { return new NoisyAlloc(i); })); 2023-02-17T09:47:45.7133874Z | ^ 2023-02-17T09:47:45.7134610Z In static member function 'static void test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc::operator delete(void*, size_t)', 2023-02-17T09:47:45.7135502Z inlined from 'test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>' at D:\a\pybind11\pybind11\tests\test_factory_constructors.cpp:408:74, 2023-02-17T09:47:45.7139172Z inlined from 'pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>' at D:/a/pybind11/pybind11/include/pybind11/detail/init.h:376:33, 2023-02-17T09:47:45.7141646Z inlined from 'Return pybind11::detail::argument_loader<Args>::call_impl(Func&&, pybind11::detail::index_sequence<Is ...>, Guard&&) && [with Return = void; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&; long long unsigned int ...Is = {0, 1, 2}; Guard = pybind11::detail::void_type; Args = {pybind11::detail::value_and_holder&, int, double}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1563:37, 2023-02-17T09:47:45.7144252Z inlined from 'pybind11::detail::enable_if_t<std::is_void<_Dummy>::value, pybind11::detail::void_type> pybind11::detail::argument_loader<Args>::call(Func&&) && [with Return = void; Guard = pybind11::detail::void_type; Func = pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&; Args = {pybind11::detail::value_and_holder&, int, double}]' at D:/a/pybind11/pybind11/include/pybind11/cast.h:1537:65, 2023-02-17T09:47:45.7147880Z inlined from 'pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>, void, pybind11::detail::value_and_holder&, int, double, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&&, void (*)(pybind11::detail::value_and_holder&, int, double), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:249:71, 2023-02-17T09:47:45.7358442Z inlined from 'static pybind11::handle pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>, void, pybind11::detail::value_and_holder&, int, double, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor>(pybind11::detail::initimpl::factory<test_submodule_factory_constructors(pybind11::module_&)::<lambda(int, double)>, pybind11::detail::void_type (*)(), test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc*(int, double), pybind11::detail::void_type()>::execute<pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc> >(pybind11::class_<test_submodule_factory_constructors(pybind11::module_&)::NoisyAlloc>&) &&::<lambda(pybind11::detail::value_and_holder&, int, double)>&&, void (*)(pybind11::detail::value_and_holder&, int, double), const pybind11::name&, const pybind11::is_method&, const pybind11::sibling&, const pybind11::detail::is_new_style_constructor&)::<lambda(pybind11::detail::function_call&)>::_FUN(pybind11::detail::function_call&)' at D:/a/pybind11/pybind11/include/pybind11/pybind11.h:224:21: ```

wangxf123456 and others added 30 commits February 28, 2023 17:35

[smart_holder] Implement try_as_void_ptr_capsule as a free function (…

114c0f2

…google#4539) * Move try_as_void_ptr_capsule out from modified_type_caster_generic_load_impl. * Try fixing clangtidy

Introduce return_value_policy_pack

3480ce8

Currently only for string_caster, tuple_caster, map_caster

Add return_value_policy_pack::override_policy() helpers.

a7ef2da

PYBIND11_TYPE_CASTER_RVPP return_value_policy_pack

3b885c6

Systematically use return_value_policy_pack in stl.h

f668edd

clang-tidy auto fix

2fd26d6

Fix MSVC warning C4458: declaration of 'policy' hides class member

75d801b

Fix oversight (when renaming return_value_policy_opts to return_value…

c943407

…_policy_pack).

Cover all changed casters in stl.h

f2414c2

WIP callbacks

f2dcfc6

Explicitly specify return_value_policy::automatic_reference in functi…

cd0fb22

…onal.h

WIP proof of concept manipulating the return value policy for callbacks.

4138bc9

Introduce from_python_policies as generalization of load bool convert

0031428

Change std::vector<bool> args_convert; to `std::vector<from_python_…

499da47

…policies> args_policies;`

clang-tidy auto fix

dc15431

WIP: tests pass

3aae97c

Split out detail::arg_literal to make ALL tests work again.

fb8edf8

clang-format auto fixes

e7486dd

Quick workaround for MSVC private/friend issue.

a99c7e7

Add test_nested_callbacks_rtn_string

0541ab5

WIP: Add collect_arguments_rvpp() functions (unused).

09cdbe5

Use collect_arguments_rvpp() from object_api<Derived>::operator()

7f0836b

clang-format auto fixes

b4cbcab

Make test_return_value_policy_pack.cpp compatible with C++11 (skip op…

9570e45

…tional, variant). Also fix oversight: missing Python-side test for variant.

Add test_return_value_policy_pack to tests/CMakeLists.txt and clang-t…

5a5d5d4

…idy auto fix.

Resolve -Wmissing-braces emitted by old clang versions (3.6, 3.7, 3…

dfef68f

….9, 5): ``` test_return_value_policy_pack.cpp:51:70: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] ```

object_api<Derived>::call_with_policies()

53ec32b

rwgk added 14 commits March 15, 2023 23:23

Use object_api<Derived>::call_with_policies() in functional.h

763d254

test_call_callback_pass_pair_string

d1c0963

Systematically exercise nested callbacks to level 4.

3319e6e

make_tuple -> make_tuple_rvpp

a52921a

simple_collector -> simple_collector_rvpp

b7e9e23

unpacking_collector -> unpacking_collector_rvpp

c56698c

PYBIND11_TYPE_CASTER_IMPL

3dfae01

Universally pass return_value_policy_pack via const &

885325c

Rename arg_literal to arg_base

d086e80

Fix git rebase accident.

0ff74b4

Fix copy-paste mishap (thanks @lalaland for catching this).

cb51567

ruff auto-fixes

e891d0f

Merge branch 'google_pywrapcc_main' into rvpp_pywrapcc

7346abf

rwgk mentioned this pull request Mar 16, 2023

[smart_holder] WIP Introduce return_value_policy_pack pybind/pybind11#4499

Closed

rwgk added 2 commits March 16, 2023 12:14

Merge branch 'google_pywrapcc_main' into rvpp_pywrapcc

c50f6ed

Undo temporary change to test_gil_scoped.py

984b0e7

rwgk marked this pull request as ready for review March 16, 2023 20:48

rwgk requested a review from wangxf123456 March 16, 2023 20:48

rwgk added the gha_wf_python312dev_enable Enable python312dev.yml label Mar 16, 2023

rwgk closed this Mar 16, 2023

rwgk reopened this Mar 16, 2023

wangxf123456 approved these changes Mar 16, 2023

View reviewed changes

rwgk merged commit c5bb69e into google:main Mar 16, 2023

rwgk deleted the rvpp_pywrapcc branch March 16, 2023 21:13

github-actions bot added the needs changelog label Mar 16, 2023

rwgk mentioned this pull request Mar 19, 2023

Extend return_value_policy_pack support to py::cast() and trampolines #30015

Merged

rwgk removed the needs changelog label Nov 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `return_value_policy_pack` (will need follow-on adjustments) #30011

Introduce `return_value_policy_pack` (will need follow-on adjustments) #30011

rwgk commented Mar 16, 2023 •

edited

Loading

Introduce return_value_policy_pack (will need follow-on adjustments) #30011

Introduce return_value_policy_pack (will need follow-on adjustments) #30011

Conversation

rwgk commented Mar 16, 2023 • edited Loading

Description

Suggested changelog entry:

Introduce `return_value_policy_pack` (will need follow-on adjustments) #30011

Introduce `return_value_policy_pack` (will need follow-on adjustments) #30011

rwgk commented Mar 16, 2023 •

edited

Loading