Numpy 2.0 support

[ericpre]

Describe the bug
Support numpy 2.0.

To Reproduce

import quaternion

Give the following error with numpy 2.0.0rc2:

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0rc2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "C:\Users\M0041User\mambaforge\envs\np2\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\spyder_kernels\console\__main__.py", line 24, in <module>
    start.main()
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\spyder_kernels\console\start.py", line 331, in main
    kernel.start()
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\ipykernel\kernelapp.py", line 739, in start
    self.io_loop.start()
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\tornado\platform\asyncio.py", line 205, in start
    self.asyncio_loop.run_forever()
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\asyncio\base_events.py", line 603, in run_forever
    self._run_once()
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\asyncio\base_events.py", line 1909, in _run_once
    handle._run()
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\ipykernel\kernelbase.py", line 545, in dispatch_queue
    await self.process_one()
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\ipykernel\kernelbase.py", line 534, in process_one
    await dispatch(*args)
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\ipykernel\kernelbase.py", line 437, in dispatch_shell
    await result
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\ipykernel\ipkernel.py", line 362, in execute_request
    await super().execute_request(stream, ident, parent)
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\ipykernel\kernelbase.py", line 778, in execute_request
    reply_content = await reply_content
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\ipykernel\ipkernel.py", line 449, in do_execute
    res = shell.run_cell(
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\ipykernel\zmqshell.py", line 549, in run_cell
    return super().run_cell(*args, **kwargs)
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\IPython\core\interactiveshell.py", line 3075, in run_cell
    result = self._run_cell(
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\IPython\core\interactiveshell.py", line 3130, in _run_cell
    result = runner(coro)
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\IPython\core\async_helpers.py", line 129, in _pseudo_sync_runner
    coro.send(None)
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\IPython\core\interactiveshell.py", line 3334, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\IPython\core\interactiveshell.py", line 3517, in run_ast_nodes
    if await self.run_code(code, result, async_=asy):
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\IPython\core\interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "C:\Users\M0041User\AppData\Local\Temp\ipykernel_37580\3305195879.py", line 1, in <module>
    import quaternion
  File "C:\Users\M0041User\mambaforge\envs\np2\lib\site-packages\quaternion\__init__.py", line 27, in <module>
    from .numpy_quaternion import (
AttributeError: _ARRAY_API not found

Traceback (most recent call last):

  Cell In[3], line 1
    import quaternion

  File ~\mambaforge\envs\np2\lib\site-packages\quaternion\__init__.py:27
    from .numpy_quaternion import (

ImportError: numpy.core.multiarray failed to import

Expected behavior
A clear and concise description of what you expected to happen and why.

Environment (please complete the following information):

• OS, including version: Windows 10
• Installation method (conda, pip, or compiled from source): pip
• Numpy version (use np.version.version): 2.0.0rc2
• Quaternion version (use quaternion.__version__): 2023.0.3

Additional context
Add any other context about the problem here.

[moble]

I started an effort in this direction, but it's very difficult. Will probably need help from some numpy devs.

[moble]

See also quaternionic for a pure-python package I wrote that doesn't depend on numpy's C API.

[2sn]

Now that Numpy 2+ is the official current release, finding a fix becomes urgent. I hope someone is able to help out. I have never done any programming with Numpy API, only use f2py.

[johann-petrak]

Until there is a solution for numpy 2.0.0 or higher, maybe add <2.0.0 to the requirements.txt file?

[2sn]

For me, installing chainconsumer, which does require numpy < 2 does the trick.

But, yes, I think also adding it to numpy-quaternion directly would be useful.

[moble]

Unfortunately, because of how pip versioning works, I believe that only means that anyone with numpy 2 trying to install this package will just get the last version of this package that didn't include the restriction to numpy<2. But FWIW, I've added it now.

[2sn]

You may be correct about that. Not sure how this can be fixed, there must be a way to modify past versions on PyPI - this would not be the first packages affected by incompatible changes of a dependent package. Rather, that must be the common case.

[hawkinsp]

In case it helps you, this diff was enough to get quaternion building and tests passing on NumPy 1 and 2:

diff --git a/__init__.py b/__init__.py
--- a/__init__.py
+++ b/__init__.py
@@ -780,8 +780,8 @@ def isclose(a, b, rtol=4*np.finfo(float)
             result = np.less_equal(abs(x-y), atol + rtol * abs(y))
         return result[()]
 
-    x = np.array(a, copy=False, subok=True, ndmin=1)
-    y = np.array(b, copy=False, subok=True, ndmin=1)
+    x = np.array(a, subok=True, ndmin=1)
+    y = np.array(b, subok=True, ndmin=1)
 
     # Make sure y is an inexact type to avoid bad behavior on abs(MIN_INT).
     # This will cause casting of x later. Also, make sure to allow subclasses
@@ -790,7 +790,7 @@ def isclose(a, b, rtol=4*np.finfo(float)
         dt = np.result_type(y, 1.)
     except TypeError:
         dt = np.dtype(np.quaternion)
-    y = np.array(y, dtype=dt, copy=False, subok=True)
+    y = np.array(y, dtype=dt, subok=True)
 
     xfin = np.isfinite(x)
     yfin = np.isfinite(y)
diff --git a/numpy_quaternion.c b/numpy_quaternion.c
--- a/numpy_quaternion.c
+++ b/numpy_quaternion.c
@@ -22,6 +22,12 @@ typedef npy_intp NPY_INTP_CONST;
 typedef npy_intp const NPY_INTP_CONST;
 #endif
 
+#if NPY_ABI_VERSION < 0x02000000
+#define PyArray_DescrProto PyArray_Descr
+#define PyDataType_ELSIZE(d) ((d)->elsize)
+#define PyDataType_GetArrFuncs(d) ((d)->f)
+#endif
+
 // The following definitions, along with `#define NPY_PY3K 1`, can
 // also be found in the header <numpy/npy_3kcompat.h>.
 #if PY_MAJOR_VERSION >= 3
@@ -59,6 +65,7 @@ static PyTypeObject PyQuaternion_Type;
 // built-in numpy data type.  We will describe its features below.
 PyArray_Descr* quaternion_descr;
 
+PyArray_DescrProto quaternion_proto = {PyObject_HEAD_INIT(NULL)};
 
 static NPY_INLINE int
 PyQuaternion_Check(PyObject* object) {
@@ -260,7 +267,7 @@ pyquaternion_##fake_name##_array_operato
   }                                                                     \
   iternext = NpyIter_GetIterNext(iter, NULL);                           \
   innerstride = NpyIter_GetInnerStrideArray(iter)[0];                   \
-  itemsize = NpyIter_GetDescrArray(iter)[1]->elsize;                    \
+  itemsize = PyDataType_ELSIZE(NpyIter_GetDescrArray(iter)[1]);         \
   innersizeptr = NpyIter_GetInnerLoopSizePtr(iter);                     \
   dataptrarray = NpyIter_GetDataPtrArray(iter);                         \
   if(PyArray_EquivTypes(PyArray_DESCR((PyArrayObject*) b), quaternion_descr)) { \
@@ -927,10 +934,10 @@ QUATERNION_nonzero (char *ip, PyArrayObj
   else {
     PyArray_Descr *descr;
     descr = PyArray_DescrFromType(NPY_DOUBLE);
-    descr->f->copyswap(&q.w, ip, !PyArray_ISNOTSWAPPED(ap), NULL);
-    descr->f->copyswap(&q.x, ip+8, !PyArray_ISNOTSWAPPED(ap), NULL);
-    descr->f->copyswap(&q.y, ip+16, !PyArray_ISNOTSWAPPED(ap), NULL);
-    descr->f->copyswap(&q.z, ip+24, !PyArray_ISNOTSWAPPED(ap), NULL);
+    PyDataType_GetArrFuncs(descr)->copyswap(&q.w, ip, !PyArray_ISNOTSWAPPED(ap), NULL);
+    PyDataType_GetArrFuncs(descr)->copyswap(&q.x, ip+8, !PyArray_ISNOTSWAPPED(ap), NULL);
+    PyDataType_GetArrFuncs(descr)->copyswap(&q.y, ip+16, !PyArray_ISNOTSWAPPED(ap), NULL);
+    PyDataType_GetArrFuncs(descr)->copyswap(&q.z, ip+24, !PyArray_ISNOTSWAPPED(ap), NULL);
     Py_DECREF(descr);
   }
   return (npy_bool) !quaternion_equal(q, zero);
@@ -942,7 +949,7 @@ QUATERNION_copyswap(quaternion *dst, qua
 {
   PyArray_Descr *descr;
   descr = PyArray_DescrFromType(NPY_DOUBLE);
-  descr->f->copyswapn(dst, sizeof(double), src, sizeof(double), 4, swap, NULL);
+  PyDataType_GetArrFuncs(descr)->copyswapn(dst, sizeof(double), src, sizeof(double), 4, swap, NULL);
   Py_DECREF(descr);
 }
 
@@ -953,10 +960,10 @@ QUATERNION_copyswapn(quaternion *dst, np
 {
   PyArray_Descr *descr;
   descr = PyArray_DescrFromType(NPY_DOUBLE);
-  descr->f->copyswapn(&dst->w, dstride, &src->w, sstride, n, swap, NULL);
-  descr->f->copyswapn(&dst->x, dstride, &src->x, sstride, n, swap, NULL);
-  descr->f->copyswapn(&dst->y, dstride, &src->y, sstride, n, swap, NULL);
-  descr->f->copyswapn(&dst->z, dstride, &src->z, sstride, n, swap, NULL);
+  PyDataType_GetArrFuncs(descr)->copyswapn(&dst->w, dstride, &src->w, sstride, n, swap, NULL);
+  PyDataType_GetArrFuncs(descr)->copyswapn(&dst->x, dstride, &src->x, sstride, n, swap, NULL);
+  PyDataType_GetArrFuncs(descr)->copyswapn(&dst->y, dstride, &src->y, sstride, n, swap, NULL);
+  PyDataType_GetArrFuncs(descr)->copyswapn(&dst->z, dstride, &src->z, sstride, n, swap, NULL);
   Py_DECREF(descr);
 }
 
@@ -1481,28 +1488,29 @@ PyMODINIT_FUNC initnumpy_quaternion(void
   _PyQuaternion_ArrFuncs.fillwithscalar = (PyArray_FillWithScalarFunc*)QUATERNION_fillwithscalar;
 
   // The quaternion array descr
-  quaternion_descr = PyObject_New(PyArray_Descr, &PyArrayDescr_Type);
-  quaternion_descr->typeobj = &PyQuaternion_Type;
-  quaternion_descr->kind = 'V';
-  quaternion_descr->type = 'q';
-  quaternion_descr->byteorder = '=';
-  quaternion_descr->flags = NPY_NEEDS_PYAPI | NPY_USE_GETITEM | NPY_USE_SETITEM;
-  quaternion_descr->type_num = 0; // assigned at registration
-  quaternion_descr->elsize = quaternion_elsize;
-  quaternion_descr->alignment = quaternion_alignment;
-  quaternion_descr->subarray = NULL;
-  quaternion_descr->fields = NULL;
-  quaternion_descr->names = NULL;
-  quaternion_descr->f = &_PyQuaternion_ArrFuncs;
-  quaternion_descr->metadata = NULL;
-  quaternion_descr->c_metadata = NULL;
+  Py_SET_TYPE(&quaternion_proto, &PyArrayDescr_Type);
+  quaternion_proto.typeobj = &PyQuaternion_Type;
+  quaternion_proto.kind = 'V';
+  quaternion_proto.type = 'q';
+  quaternion_proto.byteorder = '=';
+  quaternion_proto.flags = NPY_NEEDS_PYAPI | NPY_USE_GETITEM | NPY_USE_SETITEM;
+  quaternion_proto.type_num = 0;  // assigned at registration
+  quaternion_proto.elsize = quaternion_elsize;
+  quaternion_proto.alignment = quaternion_alignment;
+  quaternion_proto.subarray = NULL;
+  quaternion_proto.fields = NULL;
+  quaternion_proto.names = NULL;
+  quaternion_proto.f = &_PyQuaternion_ArrFuncs;
+  quaternion_proto.metadata = NULL;
+  quaternion_proto.c_metadata = NULL;
 
   Py_INCREF(&PyQuaternion_Type);
-  quaternionNum = PyArray_RegisterDataType(quaternion_descr);
+  quaternionNum = PyArray_RegisterDataType(&quaternion_proto);
 
   if (quaternionNum < 0) {
     INITERROR;
   }
+  quaternion_descr = PyArray_DescrFromType(quaternionNum);
 
   register_cast_function(NPY_BOOL, quaternionNum, (PyArray_VectorUnaryFunc*)BOOL_to_quaternion);
   register_cast_function(NPY_BYTE, quaternionNum, (PyArray_VectorUnaryFunc*)BYTE_to_quaternion);

(This doesn't include any of the setup.py changes, but I think you have them on your numpy2 branch.)

[moble]

Oh, thank you very much! I won't get a chance for another week or so, but I'll take a look at this ASAP!

[moble]

Finally finished in 2024.0.2, with some crucial help from @hawkinsp. Thanks!