adios2.so not built. Docs not up-to-date in terms of python bindings from source?

[lsawade]

Describe the bug
Docs not up to date in terms of Parallel Python bindings? adios2.so not built.

I'm trying to install parallel ADIOS2 bindings for Python on Frontier and everything in the CMake log looks alright. It finds the right python executable

To Reproduce

Loaded modules

Currently Loaded Modules:
  1) cce/15.0.0           5) craype-network-ofi      9) hsi/default        13) craype-accel-amd-gfx90a
  2) craype/2.7.19        6) cray-libsci/22.12.1.1  10) DefApps/default
  3) cray-dsmml/0.2.2     7) PrgEnv-cray/8.3.3      11) amd-mixed/5.3.0
  4) libfabric/1.15.2.0   8) darshan-runtime/3.4.0  12) cray-mpich/8.1.23

This is where it becomes slightly tricky because you will have to install your own miniconda/mamba. Which I will not delineate here. Then, create clean environment for the installation.

conda create --name adios2-env 'python==3.11'
conda activate adios2-env
conda install numpy
MPICC="cc -shared" pip install --no-cache-dir --no-binary=mpi4py mpi4py==3.1.4

Finally, clone adios2 from github and use cmake 3.20.4 to configure and install adios2

git clone git@github.com:ornladios/ADIOS2.git
mkdir adios2_build
cd adios2_build
CC=$(which mpicc) CXX=$(which mpicxx) MPICC=$(which mpicc) cmake -DCMAKE_INSTALL_PREFIX=../install -DADIOS2_USE_MPI=ON -DADIOS2_USE_FORTRAN=ON -DADIOS2_USE_HDF5=OFF -DPython_EXECUTABLE=$(which python3) -DADIOS2_USE_Python=ON ../ADIOS2
make -j 10
make install

The configuration, compilation and installation all seem to be successful. No errors, but a missing adios2.so.

Expected behavior

I expect the ../install/lib/python3.11/{site-packages/,site-packages/adios2} to contain a adios2.so file, as described in the docs, but that's not happening. I also tried adding the ../install/lib/python3.11/site-packages/adios2/bindings to that PYTHONPATH, but the bindings have completely different names compared to what's described in the docs. Am I missing something?

This used to work flawlessly on Summit with cmake and then
conda develop ../install/lib/python3.11/site-packages/ but conda develop is also being deprecated.

I feel like I'm overcomplicating things, and the installation docs are not overly descriptive for the python portion from source. Is there something that I can do to simplify this? E.g. define the python site-packages for my environment as installation location at CMake configuration time?

Desktop:

• OS/Platform: Frontier
• Build: cmake 3.20.4, Currently Loaded Modules:

  1) cce/15.0.0           5) craype-network-ofi      9) hsi/default        13) craype-accel-amd-gfx90a
  2) craype/2.7.19        6) cray-libsci/22.12.1.1  10) DefApps/default
  3) cray-dsmml/0.2.2     7) PrgEnv-cray/8.3.3      11) amd-mixed/5.3.0
  4) libfabric/1.15.2.0   8) darshan-runtime/3.4.0  12) cray-mpich/8.1.23

[lsawade]

I just found #3813. It made me look at the newer examples in the docs, Doc Examples, which differ substantially from the examples in the examples directory: Repo Examples

Note that the doc examples also have errors in them fh -> s, but wasn't updated for the rest of the scripts.

Which I assume is due to the update from low-level to high-level API in #3813 .

[pnorbert]

The python interface is being redesigned and there is no adios2.so anymore there. You should find plenty of .py files under lib/python3/dist-packages/adios2 and a folder bindings, which has a .so file adios2_bindings.XXXX.so

This is in the master and release_210 branch. Your existing python codes for processing adios files need to be revised. See e.g. examples/simulations/gray-scott/plot/gsplot.py

[pnorbert]

Oops, did not see your last comment. Exactly, as you say, the examples haven't been worked on yet.

[lsawade]

Thanks for confirming, @pnorbert!
Alright, for now, I'll downgrade to 2.9, I output some chunky files that need to processed. I'll move to 2.10 at a later stage.

Since I realize, it is a work in progress, I think this can be closed for now. Unless you want to keep it open for later closing.

[pnorbert]

Okay. I will keep open to track the outdated examples and documentatio

[pnorbert]

Just a note: examples are up to date with the new API now.
Documentation needs to be checked/updated before the release. Keeping this issue open

[abhishek1297]

I am working on a cluster. So, before cmake building. I do,

module load conda openmpi/4.1.5_gcc-10.4.0
conda create -n adios2 python=3.10
conda activate adios2
conda install numpy
conda install -c conda-forge mpi4py openmpi

As a workaround while working with conda envs, I set the prefix to the conda env's prefix

cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX  ..

Then, conda will load all the shared lib and bin that are required from its prefix. Maybe turn off bindings other than Python ones

>>> import adios2
>>> adios2
<module 'adios2' from '/home/apurandare/.conda/envs/adios2/lib/python3.10/site-packages/adios2/__init__.py'>
>>> adios2.is_built_with_mpi
True

[pnorbert]

Closing this issue. The python documentation should be up to date now.