New high level Python API for ADIOS2

[vicentebolea]

The adios2 high level python API is a great tool for users who want to interact with the adios2 library without having to learn the C++ API. However, the high level API is incomplete and there is not a possible way to connect the high level API with the low level API (Python bindings). The low level API is discourage to be used since it is a simply 1to1 implementation of the C++ bindings API.

Proposal:

I propose that we refactor/improve/update the adios2 high level python API to address the above issues. Specifically, I propose the following changes:

For the High level API I do not only propose to bring the low level API classes as it is but also made them more pythonic in a few ways. For example, the use of keyword arguments could be more consistent, and the API could use more pythonic data structures, such as lists and dictionaries, instead of a simply translation of our C++ API.

Benefits:

The proposed changes would make the adios2 high level python API more pythonic and easier to use. They would also make the Python API more stable as it does directly depends on the C++ API.

Implementation:

We want to implement the High level API purely in Python using the ADIOS2 python binding as a backend. The implementation will take the following steps:

• Translate current High level API from C++ to Python.
• Add intensive unit testing to for the current High level API
• Add rest of the ADIOS2 classes to the High level API And integrate with the existing API

Timeline:

We aim to finish this by the end of 2023

[eisenhauer]

One thing that came up in earlier discussions was that it would be nice if there were a transition path from high-level API to lower-level API (because not everything is/was possible in the high-level API). Might be a good thing to think about in the rework.

[vicentebolea]

@scottwittenburg

[williamfgc]

The proposed changes would make the adios2 high level python API more pythonic and easier to use

An example would help illustrate this point. The goal was always to keep the high level API simple. Also, minimizing Python code was a goal to make the bindings as lightweight as possible. Checking with current bindings users to get their feedback would also help.

[vicentebolea]

@williamfgc the rational for this task was:

• To provide a way to connect the high level types to the low level types.
• Ease the use (for python users) of low-level types.
• Simplify bindings

The proposed idea solves it by:

• Create a high level representation of the bindings that conform to python current API standard.
• Move File implementation (now split into Stream and FileReader classes) to Python so that it can connect to the other high level representation of the bindings. You can now for instance access the engine associated to a "Stream" using getters/setters, you can also access to all variable, attribute member functions from the Stream (high level API).
• This was a milestone of a future plan to automatize the creation of bindings for different programming languages, such approach requires high level APIs to fine tune those bindings.

[vicentebolea]

resolved at #3974

[williamfgc]

@vicentebolea thanks for the great work! I was just curious on how it will impact existing users. We spent a great deal of time looking at different options for bindings, even deprecating high-level APIs in low-level C and Fortran. Glad this is done.