Simplify C++ extension code by using pybind11

[aloisklink]

In PR #2, we added some initial code that allows us to write a Python module in C++, using the Python C API. However, the current C Python API is very verbose, as it's designed for use with C code. However, since we're using C++, we can use something a C++ API, such as pybind11, to simplify our code.

As you can see, it worked wonders! From src/nqm/irimager/irimager.cpp, we deleted 135 lines of code, and only added 9. More importantly, pybind11 also has some NumPy API helpers, so that we can easily return numpy arrays from our C++ code.

Edit: Sorry, forgot to mention this, but one issue with pybind11 is that it adds a pybind11 metaclass to all objects. Since we can't represent this in the typing stubs, stubtest always prints an error about this, so we need to ignore it in CI.

[AshleySetter]

Looks much nicer using pybind11, so much boilerplate code reduction!

I get the following error when I attempt to run pytest:

(base) ash@ash-XPS-13-9305:~/nqm-irimager$ poetry run pytest
================================================================================= test session starts =================================================================================
platform linux -- Python 3.9.7, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/ash/nqm-irimager, configfile: pyproject.toml
collected 0 items / 1 error                                                                                                                                                           

======================================================================================= ERRORS ========================================================================================
_______________________________________________________________________ ERROR collecting tests/test_irimager.py _______________________________________________________________________
ImportError while importing test module '/home/ash/nqm-irimager/tests/test_irimager.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_irimager.py:2: in <module>
    from nqm.irimager import IRImager
E   ImportError: /home/ash/nqm-irimager/src/nqm/irimager.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr10_M_releaseEv
=============================================================================== short test summary info ===============================================================================
ERROR tests/test_irimager.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================== 1 error in 0.05s ===================================================================================

This _ZNSt15__exception_ptr13exception_ptr10_M_releaseEv demangles to std::__exception_ptr::exception_ptr::_M_release() it seems from a cursory google that it's related to which compiler is being used; I've got gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1) and cmake version 3.22.1 installed. We may need some way to specify the compiler (or compiler version)?

[AshleySetter]

This seems relevant: pybind/pybind11#3623

[AshleySetter]

Seems like a toolchain compatibility issue according to this: pybind/pybind11#3623 (comment)

[aloisklink]

This _ZNSt15__exception_ptr13exception_ptr10_M_releaseEv demangles to std::__exception_ptr::exception_ptr::_M_release()

My gut feeling is that this is due to a dynamic library mismatch, e.g. you're compiling with g++ 11.3.0, but you're running with a different C++ standard library.

Can you show me the output of:

me@pc:~/nqm-irimager (nqm-irimager-2)$ ldd build/lib.linux-x86_64-cpython-310/nqm/irimager.cpython-310-x86_64-linux-gnu.so 
	linux-vdso.so.1 (0x00007fff315d7000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f49dce87000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f49dce67000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f49dcc3f000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f49dcb58000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f49dd0ec000)

You can also run echo $LD_LIBRARY_PATH and see if that prints anything.

I've also got g++ 11.3.0 and it works for me (same in a new Ubuntu 20.04 container).

[AshleySetter]

I don't have a build directory in the repo directory but:

(base) ash@ash-XPS-13-9305:~/nqm-irimager$ ldd src/nqm/irimager.cpython-39-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffd885b1000)
        libstdc++.so.6 => /home/ash/anaconda3/lib/libstdc++.so.6 (0x00007f3bf4c6e000)
        libgcc_s.so.1 => /home/ash/anaconda3/lib/libgcc_s.so.1 (0x00007f3bf4c5a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3bf4a00000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3bf4919000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3bf4e08000)

echo $LD_LIBRARY_PATH prints nothing

[aloisklink]

libstdc++.so.6 => /home/ash/anaconda3/lib/libstdc++.so.6

Ahhhh, this looks a bit suspicious. I suspect that you may be compiling with /lib/x86_64-linux-gnu/libstdc++.so.6, and if it's newer than /home/ash/anaconda3/lib/libstdc++.so.6, you're going to have problems.

Does LD_LIBRARY_PATH=/lib/x86_64-linux-gnu ldd src/nqm/irimager.cpython-39-x86_64-linux-gnu.so do anything different? If so, you can try running LD_LIBRARY_PATH=/lib/x86_64-linux-gnu poetry run pytest and see if that helps.

Edit: If you're not using Anaconda, you could also try uninstalling it and see if that helps.

[AshleySetter]

It was me using poetry from anaconda 🤦