From 54c2e800986266b5a5e5cae6469a5f6945d98297 Mon Sep 17 00:00:00 2001 From: Axel Huebl Date: Fri, 18 Aug 2023 22:14:51 -0400 Subject: [PATCH] Release 0.15.2 (#1507) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Fix gcc9 warning the implicitly-defined constructor does not initialize 'openPMD::Datatype openPMD::detail::BufferedUniquePtrPut::dtype' * More careful documentation of streaming API * Doc: Fix Bib Authors Make sure the bib authors match the quoted openPMD-standard authors. * Update .readthedocs.yml Update to newer Ubuntu, shipping a newer OpenSSL * Fix deprecated storeChunk APIs in first read/write examples * CI: macOS-11 Update The older macOS image is now removed. The latest points already to macOS-12. macoS-13 runners are marked experimental. https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources * Docs: Linking to C++ Projects Move a section only written in the README to our developer section on readthedocs. * Use lazy imports for dask and pandas * Better error message when loading to a buffer with mismatched type * Remove schema 2021 from documentation It will be removed and should no longer be advertised. * Document ROMIO/HDF5/Chunking issue https://github.com/open-mpi/ompi/issues/7795 * Use a "minor" RST link Co-authored-by: Axel Huebl * HDF5: HDF5_DO_MPI_FILE_SYNC Document a new work-around option for MPI-parallel HDF5 for filesystems that are super limited for paralle I/O features relevant in HPC. * Fix typo Co-authored-by: Franz Pöschel * openpmd-pipe: set correct install permissions * Fix headers in workflow.rst * Fix documentation for preferred_flush_target in ADIOS2 * HDF5: Fix Char Type Matching In HDF5, there are only the signed and unsigned char type. The third `char` type is an alias to one or the other, different to the C/C++ fundamental types. This tries to fix the type matching order to be platform independent between ppc64le/aarch64/x86-64. * Add failing HDF5 test * loadChunk(): consider equivalent char types for casting * Doc strings * newline & comment * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Doc: Sphinx Copybutton and Design Add the Sphinx extensions `copybutton` (copy code blocks with a single click) and `design` (for boxes, tabs, dropdowns, etc.). * CI: Doxygen 1.9.7 Broken Markdown support for main file broken in 1.9.7. Go back to previous patch release. Upstream already fixed. * Docs: Analysis Add a data analysis & visualization section. This is meant to show entry points and workflows to work with openPMD data in larger frameworks and compatible ecosystems. * [Draft] DASK, Pandas, ... * Doc: DASK * Pandas * RAPIDS * Typos * Don't require unitSI when reading patch record component * CI: oneAPI 2023.2.0 Update CI to breaking `apt` changes in the latest oneAPI release. * Update __repr__ method of major objects in openPMD hierarchy * Deal with trailing slashes in file paths Happens in bash completion on BP files since they are folders. * Throw better error messages when selecting a bad backend * Adapt CoreTest to new error message * Follow-up to #1470 Changes there left the PatchRecordComponent dirty after parsing * HDF5IOHandler: Support for float128 data types with 80bit precision on ARM64/PPC64 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleanup Unify little/big endian, use double80 also in other places Long double size 8 -> 16 Use malloc to avoid alignment issues Same treatment for complex long double Add this for readAttribute too Avoid non-native datatypes in writing * Make this fix little-endian only * Add comment * Suggestions from review * Use new instead of malloc everywhere Also, add a more explanative comment * CI fix: type comparison in Python No idea why this check is triggered by this PR * ADIOS2: Ensure that a step is always active at write time Even when not using steps, dump everything into a single large step. BP5 will fail otherwise. * CI: macOS 11.0+ We do not have the runners anymore to deploy and test macOS 10.15, so we bump our tests to 11.0+, too. Technically, we could build & deploy a bit longer on GH actions `macos-11` for `10.15` (Catalina), but since it is unmaintained by Apple/end-of-life already and macOS-11 is the oldest still maintained OS by Apple, we do also stop support for it. At the time of writing, old and maintained are: - macOS 11, Big Sur - macOS 12, Monterey latest is: macOS 13, Ventura - and in preview is macOS 14, Sonoma. This also unifies our arm64/aarch64 (M1/M2) requirements, which are macOS 11.0+ as well. * HDF5: Throw ReadError at dataset open * Try parent types when a dataset datatype is unknown * Update Sample Download Scripts * Add test for old HDF5-plugin written openPMD dataset from PIConGPU * Remove debugging comments * Warn on BP5+Blosc in ADIOS2 v2.9 up to patch level 1 * Fix unused parameter * Update Warning String * Throw error when steps are needed but not supported * Fix variableBasedSingleIteration test * Test that the error is correctly thrown * Introduce Series::parseBase alias for readIterations(), fix workflow * Have only one instance of SeriesIterator * Break memory cycle * Use simple API in test again * Snapshot attribute in file-based encoding Snapshot attribute must be written in Iteration::endStep() in file-based encoding * Optional debugging output for AbstractIOHandlerImpl::flush() * Revert "Optional debugging output for AbstractIOHandlerImpl::flush()" This reverts commit ee8de45bc8f62160d8a982f51037a62959447ac0. * Post-rebase fixes 1) Don't write snapshot attributes during initialization of an iteration 2) Catch unsuccessful flush run also in beginStep() * Ensure that m_lastFlushSuccessful is always called * Pandas DataFrames: Add Row Column Name By default, the row index (!= particle index) in a pandas dataframe has no name. This can be a bit cumbersome for exports, e.g., to CSV - where this header field would just be empty. This PR names the index now "row", because it is not a (macro) particle id property. * Python: 3.8+ Python 3.7 went EOL last month. This bumps our supported versions to 3.8+. * Optional debugging output for AbstractIOHandlerImpl::flush() * Add an environment variable for this * Version 0.15.2 + Changelog * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Python: Fix Series Compile Fix compile issue introduced during backporting. * Unmerge 0.16 content from variableBasedSeries test * Reintroduce backend selection in variableBasedSeries test Still needed due to ADIOS1 BP env variable * Add src/Dataset.cpp to ADIOS1 source * Replace openPMD_Datatypes global with function * Windows CI: Bump Version to 0.15.2 * OPENPMD_BP_BACKEND=="ADIOS1": Skip BP5 Tests * Streaming examples: Set WAN as default transport * Exclude ADIOS1 from variableBasedSeries test * Changelog: Bump Date * replace extent in weighting and displacement store extent value in n_particles * Examples: Fix Types of Constants & Attributes Backports from #1316 and #1510 * Changelog: Streaming Example Note * CMake: Warn and Continue on Empty HDF5_VERSION Seen on Conda-Forge for arm64 on macOS for HDF5 1.14.1 --------- Co-authored-by: Franz Pöschel Co-authored-by: Dave Grote Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ulrik Guenther Co-authored-by: Kara-Mostefa Ilian --- .github/workflows/dependencies/install_icc | 11 +- .github/workflows/dependencies/install_icx | 13 +- .github/workflows/macos.yml | 20 +- .github/workflows/source.yml | 2 +- .github/workflows/windows.yml | 2 +- .readthedocs.yml | 3 + CHANGELOG.rst | 69 +++ CITATION.cff | 2 +- CMakeLists.txt | 23 +- Dockerfile | 28 +- NEWS.rst | 7 + README.md | 10 +- conda.yml | 2 +- docs/requirements.txt | 2 + docs/source/analysis/contrib.rst | 34 ++ docs/source/analysis/dask.rst | 50 +++ docs/source/analysis/pandas.rst | 101 +++++ docs/source/analysis/paraview.rst | 55 +++ docs/source/analysis/rapids.rst | 99 +++++ docs/source/analysis/viewer.rst | 68 +++ docs/source/backends/adios2.rst | 32 +- docs/source/backends/hdf5.rst | 24 +- docs/source/citation.rst | 1 + docs/source/conf.py | 6 +- docs/source/dev/dependencies.rst | 2 +- docs/source/dev/linking.rst | 88 ++++ docs/source/index.rst | 18 +- docs/source/usage/firstread.rst | 5 +- docs/source/usage/firstwrite.rst | 15 +- docs/source/usage/workflow.rst | 14 +- examples/10_streaming_read.cpp | 16 +- examples/10_streaming_read.py | 16 +- examples/10_streaming_write.cpp | 16 +- examples/10_streaming_write.py | 17 +- examples/12_span_write.cpp | 5 + examples/12_span_write.py | 5 + examples/13_write_dynamic_configuration.cpp | 5 + examples/13_write_dynamic_configuration.py | 14 +- examples/1_structure.cpp | 8 +- examples/3_write_serial.cpp | 5 + examples/3_write_serial.py | 5 + examples/3a_write_thetaMode_serial.cpp | 5 + examples/3a_write_thetaMode_serial.py | 5 + examples/3b_write_resizable_particles.cpp | 5 + examples/3b_write_resizable_particles.py | 2 +- examples/5_write_parallel.cpp | 10 +- examples/5_write_parallel.py | 6 + examples/7_extended_write_serial.cpp | 5 +- examples/7_extended_write_serial.py | 4 +- examples/8a_benchmark_write_parallel.cpp | 10 + examples/8b_benchmark_read_parallel.cpp | 5 + examples/9_particle_write_serial.py | 18 +- include/openPMD/Datatype.hpp | 76 +++- include/openPMD/Datatype.tpp | 356 ++++++++++++++++ include/openPMD/DatatypeHelpers.hpp | 281 +----------- include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp | 8 +- include/openPMD/IO/AbstractIOHandler.hpp | 8 + .../openPMD/IO/AbstractIOHandlerHelper.hpp | 8 +- include/openPMD/IO/AbstractIOHandlerImpl.hpp | 206 +-------- include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp | 10 + include/openPMD/IO/IOTask.hpp | 1 + include/openPMD/ReadIterations.hpp | 22 +- include/openPMD/RecordComponent.tpp | 3 +- include/openPMD/Series.hpp | 67 ++- include/openPMD/WriteIterations.hpp | 13 + include/openPMD/backend/Attributable.hpp | 1 + include/openPMD/version.hpp | 2 +- setup.py | 5 +- share/openPMD/download_samples.ps1 | 5 + share/openPMD/download_samples.sh | 5 +- src/Datatype.cpp | 83 ++-- src/IO/ADIOS/ADIOS2IOHandler.cpp | 45 +- src/IO/AbstractIOHandler.cpp | 13 +- src/IO/AbstractIOHandlerHelper.cpp | 22 +- src/IO/AbstractIOHandlerImpl.cpp | 351 +++++++++++++++ src/IO/HDF5/HDF5Auxiliary.cpp | 7 +- src/IO/HDF5/HDF5IOHandler.cpp | 399 +++++++++++++++--- src/Iteration.cpp | 26 +- src/ReadIterations.cpp | 75 ++-- src/Series.cpp | 65 ++- src/WriteIterations.cpp | 16 +- src/backend/Attributable.cpp | 1 + src/backend/PatchRecordComponent.cpp | 38 +- src/binding/python/Attributable.cpp | 11 +- src/binding/python/Container.cpp | 18 + src/binding/python/Dataset.cpp | 20 +- src/binding/python/Iteration.cpp | 4 +- src/binding/python/Mesh.cpp | 3 +- src/binding/python/MeshRecordComponent.cpp | 22 +- src/binding/python/ParticlePatches.cpp | 7 +- src/binding/python/ParticleSpecies.cpp | 9 +- src/binding/python/PatchRecordComponent.cpp | 23 + src/binding/python/Record.cpp | 8 +- src/binding/python/RecordComponent.cpp | 36 +- src/binding/python/Series.cpp | 153 ++++++- src/binding/python/openpmd_api/DaskArray.py | 13 +- .../python/openpmd_api/DaskDataFrame.py | 32 +- src/binding/python/openpmd_api/DataFrame.py | 21 +- .../python/openpmd_api/pipe/__main__.py | 4 +- test/CoreTest.cpp | 37 +- test/ParallelIOTest.cpp | 11 +- test/SerialIOTest.cpp | 229 +++++++++- 102 files changed, 2958 insertions(+), 914 deletions(-) create mode 100644 docs/source/analysis/contrib.rst create mode 100644 docs/source/analysis/dask.rst create mode 100644 docs/source/analysis/pandas.rst create mode 100644 docs/source/analysis/paraview.rst create mode 100644 docs/source/analysis/rapids.rst create mode 100644 docs/source/analysis/viewer.rst create mode 100644 docs/source/dev/linking.rst create mode 100644 include/openPMD/Datatype.tpp diff --git a/.github/workflows/dependencies/install_icc b/.github/workflows/dependencies/install_icc index 479a0a7870..b990aa1f56 100755 --- a/.github/workflows/dependencies/install_icc +++ b/.github/workflows/dependencies/install_icc @@ -7,9 +7,14 @@ sudo apt-get -qqq update sudo apt-get install g++ # libopenmpi-dev sudo apt-get install -y wget build-essential pkg-config cmake ca-certificates gnupg -sudo wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB -sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB -echo "deb https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list + +# download the key to system keyring +wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \ +| gpg --dearmor | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null + +# add signed entry to apt sources and configure the APT client to use Intel repository +echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list + sudo apt-get update sudo apt-get install -y intel-oneapi-compiler-dpcpp-cpp-and-cpp-classic # intel-oneapi-python diff --git a/.github/workflows/dependencies/install_icx b/.github/workflows/dependencies/install_icx index f15ff48bc9..b2ce244235 100755 --- a/.github/workflows/dependencies/install_icx +++ b/.github/workflows/dependencies/install_icx @@ -5,17 +5,20 @@ set -eu -o pipefail # Ref.: https://github.com/rscohn2/oneapi-ci # intel-basekit intel-hpckit are too large in size -wget -q -O - https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB \ - | sudo apt-key add - -echo "deb https://apt.repos.intel.com/oneapi all main" \ - | sudo tee /etc/apt/sources.list.d/oneAPI.list + +# download the key to system keyring +wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \ +| gpg --dearmor | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null + +# add signed entry to apt sources and configure the APT client to use Intel repository +echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list sudo apt-get update sudo apt-get install -y --no-install-recommends \ build-essential \ cmake \ - intel-oneapi-dpcpp-cpp-compiler intel-oneapi-mkl-devel \ + intel-oneapi-compiler-dpcpp-cpp intel-oneapi-mkl-devel \ g++ gfortran # libopenmpi-dev # openmpi-bin diff --git a/.github/workflows/macos.yml b/.github/workflows/macos.yml index 9f6616fa9c..5893a3b513 100644 --- a/.github/workflows/macos.yml +++ b/.github/workflows/macos.yml @@ -28,10 +28,11 @@ jobs: python3 -m pip install -U mpi4py numpy pandas set -e - name: Build - env: {CXXFLAGS: -Werror, MACOSX_DEPLOYMENT_TARGET: 10.15} + env: {CXXFLAGS: -Werror, MACOSX_DEPLOYMENT_TARGET: 11.0} # 10.14+ due to std::visit # 10.15+ due to std::filesystem in toml11 # https://cibuildwheel.readthedocs.io/en/stable/cpp_standards/#macos-and-deployment-target-versions + # 11.0+ for arm64/aarch64 (M1/M2) builds run: | share/openPMD/download_samples.sh build cmake -S . -B build \ @@ -43,14 +44,13 @@ jobs: cmake --build build --parallel 2 ctest --test-dir build --verbose - appleclang12_py_ad1: - runs-on: macos-10.15 - # next: macOS-11 + appleclang13_py: + runs-on: macos-11 if: github.event.pull_request.draft == false steps: - uses: actions/checkout@v3 - name: Install - env: {MACOSX_DEPLOYMENT_TARGET: 10.14} + env: {MACOSX_DEPLOYMENT_TARGET: 11.0} run: | set +e python3 -m pip install -U numpy pandas @@ -63,10 +63,11 @@ jobs: make install set -e - name: Build - env: {CXXFLAGS: -Werror -DTOML11_DISABLE_STD_FILESYSTEM, MACOSX_DEPLOYMENT_TARGET: 10.14} + env: {CXXFLAGS: -Werror, MACOSX_DEPLOYMENT_TARGET: 11.0} # 10.14+ due to std::visit - # std::filesystem in toml11 needs macOS 10.15 + # 10.15+ due to std::filesystem in toml11 # https://cibuildwheel.readthedocs.io/en/stable/cpp_standards/#macos-and-deployment-target-versions + # 11.0+ for arm64/aarch64 (M1/M2) builds run: | share/openPMD/download_samples.sh build cmake -S . -B build \ @@ -75,8 +76,9 @@ jobs: -DopenPMD_USE_HDF5=OFF \ -DopenPMD_USE_ADIOS1=ON \ -DopenPMD_USE_ADIOS2=OFF \ - -DopenPMD_USE_INVASIVE_TESTS=ON - cmake --build build --parallel 2 + -DopenPMD_USE_INVASIVE_TESTS=ON \ + -DPython_EXECUTABLE=$(which python3) + cmake --build build --parallel 3 ctest --test-dir build --verbose # TODO: apple_conda_ompi_all (similar to conda_ompi_all on Linux) diff --git a/.github/workflows/source.yml b/.github/workflows/source.yml index 4a76960656..7f8d0bc272 100644 --- a/.github/workflows/source.yml +++ b/.github/workflows/source.yml @@ -41,7 +41,7 @@ jobs: update-conda: true conda-channels: conda-forge - name: Install - run: conda install -c conda-forge doxygen + run: conda install -c conda-forge doxygen=1.9.6 - name: Doxygen run: .github/workflows/source/buildDoxygen diff --git a/.github/workflows/windows.yml b/.github/workflows/windows.yml index 7b1cb5d9a9..0b7d4ec17e 100644 --- a/.github/workflows/windows.yml +++ b/.github/workflows/windows.yml @@ -42,7 +42,7 @@ jobs: python3.exe -m pip wheel . if(!$?) { Exit $LASTEXITCODE } - python3.exe -m pip install openPMD_api-0.15.1-cp39-cp39-win_amd64.whl + python3.exe -m pip install openPMD_api-0.15.2-cp39-cp39-win_amd64.whl if(!$?) { Exit $LASTEXITCODE } python3.exe -c "import openpmd_api as api; print(api.variants)" diff --git a/.readthedocs.yml b/.readthedocs.yml index 54a296c29d..2fc118e82d 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -10,5 +10,8 @@ formats: - epub build: + os: ubuntu-22.04 + tools: + python: "3.11" apt_packages: - librsvg2-bin diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 8d03c5bb98..04ac98dfbe 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -3,6 +3,75 @@ Changelog ========= +0.15.2 +------ +**Date:** 2023-08-18 + +Python, ADIOS2 and HDF5 Fixes + +This release fixed regressions in the Python frontend as well as the ADIOS2 and HDF5 backends. +Supported macOS versions are now 11.0+ and Python versions are 3.8+. + +Changes to "0.15.1" +^^^^^^^^^^^^^^^^^^^ + +Bug Fixes +""""""""" + +- Don't require unitSI when reading a patch record component #1470 +- Examples: + + - Streaming examples: Set WAN as default transport #1511 + - Fix types of particle constant records #1316 #1510 +- Python: + + - DataFrame to ASCII: Header of First Column in CSV bug documentation third party #1480 #1501 + - Update ``__repr__`` method of major objects in openPMD hierarchy #1476 + - openpmd-pipe: set correct install permissions #1459 + - Better error message when loading to a buffer with mismatched type #1452 + - Use lazy imports for dask and pandas #1442 +- ADIOS2: + + - Fixes for variable-based encoding in backends without step support #1484 #1481 + - Warn on BP5+Blosc in ADIOS2 v2.9 up to patch level 1 #1497 + - Ensure that a step is always active at write time #1492 + - Fix gcc9 warning #1429 +- HDF5: + + - Handle unknown datatypes in datasets #1469 + - Support for float128 on ARM64/PPC64 #1364 + - Fix Char Type Matching #1433 #1431 + - Install: Warn and Continue on Empty ``HDF5_VERSION`` in CMake #1512 +- CI: + + - type comparison in openpmd-pipe #1490 + +Other +""""" + +- Better handling for file extensions #1473 #1471 +- Optional debugging output for ``AbstractIOHandlerImpl::flush()`` #1495 +- Python: 3.8+ #1502 +- CI: + + - macOS 11.0+ #1486 #1446 + - oneAPI 2023.2.0 #1478 + - Doxygen 1.9.7 Broken #1464 +- Docs: + + - Analysis with third party data science frameworks #1444 + - Sphinx Copybutton and Design #1461 + - Fix small documentation issues after 0.15 release #1440 + - ``HDF5_DO_MPI_FILE_SYNC`` #1427 + - OpenMPI-ROMIO/HDF5/Chunking issue #1441 + - Remove ADIOS2 schema 2021 #1451 + - Linking to C++ Projects #1445 + - Fix deprecated APIs in first read/write examples #1435 + - Update ``.readthedocs.yml`` #1438 + - Fix Bib Authors #1434 + - More careful documentation of streaming API #1430 + + 0.15.1 ------ **Date:** 2023-04-02 diff --git a/CITATION.cff b/CITATION.cff index df49273bf9..e2d58c43a4 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -25,7 +25,7 @@ contact: orcid: https://orcid.org/0000-0003-1943-7141 email: axelhuebl@lbl.gov title: "openPMD-api: C++ & Python API for Scientific I/O with openPMD" -version: 0.15.1 +version: 0.15.2 repository-code: https://github.com/openPMD/openPMD-api doi: 10.14278/rodare.27 license: LGPL-3.0-or-later diff --git a/CMakeLists.txt b/CMakeLists.txt index 244794c9da..4eea591f5b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2,7 +2,7 @@ # cmake_minimum_required(VERSION 3.15.0) -project(openPMD VERSION 0.15.1) # LANGUAGES CXX +project(openPMD VERSION 0.15.2) # LANGUAGES CXX # the openPMD "markup"/"schema" standard version set(openPMD_STANDARD_VERSION 1.1.0) @@ -334,10 +334,16 @@ endif() # HDF5 checks string(CONCAT openPMD_HDF5_STATUS "") # version: lower limit -if(openPMD_HAVE_HDF5 AND HDF5_VERSION VERSION_LESS 1.8.13) - string(CONCAT openPMD_HDF5_STATUS - "Found HDF5 version ${HDF5_VERSION} is too old. At least " - "version 1.8.13 is required.\n") +if(openPMD_HAVE_HDF5) + if(HDF5_VERSION STREQUAL "") + message(WARNING "HDF5_VERSION is empty. Now assuming it is 1.8.13 or newer.") + else() + if(HDF5_VERSION VERSION_LESS 1.8.13) + string(CONCAT openPMD_HDF5_STATUS + "Found HDF5 version ${HDF5_VERSION} is too old. At least " + "version 1.8.13 is required.\n") + endif() + endif() endif() # we imply support for parallel I/O if MPI variant is ON if(openPMD_HAVE_MPI AND openPMD_HAVE_HDF5 @@ -458,7 +464,7 @@ if(CMAKE_VERSION VERSION_LESS 3.18.0) set(_PY_DEV_MODULE Development) endif() if(openPMD_USE_PYTHON STREQUAL AUTO) - find_package(Python 3.7.0 COMPONENTS Interpreter ${_PY_DEV_MODULE}) + find_package(Python 3.8.0 COMPONENTS Interpreter ${_PY_DEV_MODULE}) if(Python_FOUND) if(openPMD_USE_INTERNAL_PYBIND11) add_subdirectory("${openPMD_SOURCE_DIR}/share/openPMD/thirdParty/pybind11") @@ -546,6 +552,7 @@ set(IO_ADIOS1_SEQUENTIAL_SOURCE src/auxiliary/JSON.cpp src/IO/AbstractIOHandlerImpl.cpp src/ChunkInfo.cpp + src/Datatype.cpp src/Error.cpp src/IO/IOTask.cpp src/IO/ADIOS/CommonADIOS1IOHandler.cpp @@ -556,6 +563,7 @@ set(IO_ADIOS1_SOURCE src/auxiliary/JSON.cpp src/IO/AbstractIOHandlerImpl.cpp src/ChunkInfo.cpp + src/Datatype.cpp src/Error.cpp src/IO/IOTask.cpp src/IO/ADIOS/CommonADIOS1IOHandler.cpp @@ -1323,10 +1331,9 @@ if(openPMD_INSTALL) if(openPMD_BUILD_CLI_TOOLS) foreach(toolname ${openPMD_PYTHON_CLI_TOOL_NAMES}) install( - FILES ${openPMD_SOURCE_DIR}/src/cli/${toolname}.py + PROGRAMS ${openPMD_SOURCE_DIR}/src/cli/${toolname}.py DESTINATION ${openPMD_INSTALL_BINDIR} RENAME openpmd-${toolname} - PERMISSIONS OWNER_EXECUTE OWNER_WRITE OWNER_READ ) endforeach() endif() diff --git a/Dockerfile b/Dockerfile index a7830ecbb7..f08ddb1dfc 100644 --- a/Dockerfile +++ b/Dockerfile @@ -5,8 +5,8 @@ FROM quay.io/pypa/manylinux2010_x86_64 as build-env # FROM quay.io/pypa/manylinux1_x86_64 as build-env ENV DEBIAN_FRONTEND noninteractive -# Python 3.7-3.11 via "37m 38 39 311" -ARG PY_VERSIONS="37m 38 39 310 311" +# Python 3.8-3.11 via "38 39 311" +ARG PY_VERSIONS="38 39 310 311" # static libs need relocatable symbols for linking to shared python lib ENV CFLAGS="-fPIC ${CFLAGS}" @@ -122,30 +122,6 @@ RUN for whl in /opt/src/dist/*.whl; do \ && du -hs /opt/src/dist/* \ && du -hs /wheelhouse/* -# test in fresh env: Debian:Buster + Python 3.7 -FROM debian:buster -ENV DEBIAN_FRONTEND noninteractive -COPY --from=build-env /wheelhouse/openPMD_api-*-cp37-cp37m-manylinux2010_x86_64.whl . -RUN apt-get update \ - && apt-get install -y --no-install-recommends python3 python3-pip \ - && rm -rf /var/lib/apt/lists/* - # binutils -RUN python3 --version \ - && python3 -m pip install -U pip \ - && python3 -m pip install openPMD_api-*-cp37-cp37m-manylinux2010_x86_64.whl -RUN find / -name "openpmd*" -RUN ls -hal /usr/local/lib/python3.7/dist-packages/ -RUN ls -hal /usr/local/lib/python3.7/dist-packages/openpmd_api/ -# RUN ls -hal /usr/local/lib/python3.7/dist-packages/.libsopenpmd_api -# RUN objdump -x /usr/local/lib/python3.7/dist-packages/openpmd_api.cpython-37m-x86_64-linux-gnu.so | grep RPATH -RUN ldd /usr/local/lib/python3.7/dist-packages/openpmd_api/openpmd_api_cxx.cpython-37m-x86_64-linux-gnu.so -RUN python3 -c "import openpmd_api as io; print(io.__version__); print(io.variants)" -RUN python3 -m openpmd_api.ls --help -RUN openpmd-ls --help -#RUN echo "* soft core 100000" >> /etc/security/limits.conf && \ -# python3 -c "import openpmd_api as io"; \ -# gdb -ex bt -c core - # test in fresh env: Debian:Sid + Python 3.8 FROM debian:sid ENV DEBIAN_FRONTEND noninteractive diff --git a/NEWS.rst b/NEWS.rst index 40246be635..fbcb590753 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -3,6 +3,13 @@ Upgrade Guide ============= +0.15.2 +------ + +Python 3.7 is removed, please use 3.8 or newer. +macOS 10.15 is removed, please use 11.0 or newer. + + 0.15.0 ------ diff --git a/README.md b/README.md index b422379d44..cee89c54ec 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ while those can be built either with or without: Optional language bindings: * Python: - * Python 3.7 - 3.11 + * Python 3.8 - 3.11 * pybind11 2.10.1+ * numpy 1.15+ * mpi4py 2.1+ (optional, for MPI) @@ -296,7 +296,7 @@ The install will contain header files and libraries in the path set with `-DCMAK ### CMake -If your project is using CMake for its build, one can conveniently use our provided `openPMDConfig.cmake` package which is installed alongside the library. +If your project is using CMake for its build, one can conveniently use our provided `openPMDConfig.cmake` package, which is installed alongside the library. First set the following environment hint if openPMD-api was *not* installed in a system path: @@ -308,7 +308,7 @@ export CMAKE_PREFIX_PATH=$HOME/somepath:$CMAKE_PREFIX_PATH Use the following lines in your project's `CMakeLists.txt`: ```cmake # supports: COMPONENTS MPI NOMPI HDF5 ADIOS1 ADIOS2 -find_package(openPMD 0.9.0 CONFIG) +find_package(openPMD 0.15.0 CONFIG) if(openPMD_FOUND) target_link_libraries(YourTarget PRIVATE openPMD::openPMD) @@ -336,13 +336,13 @@ set(openPMD_INSTALL OFF) # or instead use: set(openPMD_USE_PYTHON OFF) FetchContent_Declare(openPMD GIT_REPOSITORY "https://github.com/openPMD/openPMD-api.git" - GIT_TAG "dev") + GIT_TAG "0.15.0") FetchContent_MakeAvailable(openPMD) ``` ### Manually -If your (Linux/OSX) project is build by calling the compiler directly or uses a manually written `Makefile`, consider using our `openPMD.pc` helper file for `pkg-config` which are installed alongside the library. +If your (Linux/OSX) project is build by calling the compiler directly or uses a manually written `Makefile`, consider using our `openPMD.pc` helper file for `pkg-config`, which are installed alongside the library. First set the following environment hint if openPMD-api was *not* installed in a system path: diff --git a/conda.yml b/conda.yml index 2bb1713efd..e82567b2ae 100644 --- a/conda.yml +++ b/conda.yml @@ -36,7 +36,7 @@ dependencies: - pre-commit - pyarrow # for dask # - pybind11 # shipped internally - - python>=3.7 + - python>=3.8 # just a note for later hackery, we could install pip packages inside the env, too: # - pip: diff --git a/docs/requirements.txt b/docs/requirements.txt index 05ac0aa486..62f525b92c 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -7,6 +7,8 @@ pygments recommonmark scipy sphinx>=5.3 +sphinx-copybutton +sphinx-design sphinx_rtd_theme>=1.1.1 sphinxcontrib-svg2pdfconverter sphinxcontrib.programoutput diff --git a/docs/source/analysis/contrib.rst b/docs/source/analysis/contrib.rst new file mode 100644 index 0000000000..f1ccf9df67 --- /dev/null +++ b/docs/source/analysis/contrib.rst @@ -0,0 +1,34 @@ +.. _analysis-contrib: + +Contributed +=========== + +This page contains contributed projects and third party integrations to analyze openPMD data. +See the `openPMD-projects `__ catalog for more community integrations. + + +.. _analysis-contrib-visualpic: + +3D Visualization: VisualPIC +--------------------------- + +openPMD data can be visualized with the domain-specific VisualPIC renderer. +Please see `the WarpX page for details `__. + + +.. _analysis-contrib-visit: + +3D Visualization: VisIt +----------------------- + +openPMD **HDF5** data can be visualized with VisIt 3.1.0+. +VisIt supports openPMD HDF5 files and requires to rename the files from ``.h5`` to ``.opmd`` to be automatically detected. + + +.. _analysis-contrib-yt: + +yt-project +---------- + +openPMD **HDF5** data can be visualized with `yt-project `__. +Please see the `yt documentation `__ for details. diff --git a/docs/source/analysis/dask.rst b/docs/source/analysis/dask.rst new file mode 100644 index 0000000000..e00dce0e98 --- /dev/null +++ b/docs/source/analysis/dask.rst @@ -0,0 +1,50 @@ +.. _analysis-dask: + +DASK +==== + +The Python bindings of openPMD-api provide direct methods to load data into the parallel, `DASK data analysis ecosystem `__. + + +How to Install +-------------- + +Among many package managers, `PyPI `__ ships the latest packages of DASK: + +.. code-block:: python + + python3 -m pip install -U dask + python3 -m pip install -U pyarrow + + +How to Use +---------- + +The central Python API calls to convert to DASK datatypes are the ``ParticleSpecies.to_dask`` and ``Record_Component.to_dask_array`` methods. + +.. code-block:: python + + s = io.Series("samples/git-sample/data%T.h5", io.Access.read_only) + electrons = s.iterations[400].particles["electrons"] + + # the default schedulers are local/threaded. We can also use local + # "processes" or for multi-node "distributed", among others. + dask.config.set(scheduler='processes') + + df = electrons.to_dask() + type(df) # ... + + E = s.iterations[400].meshes["E"] + E_x = E["x"] + darr_x = E_x.to_dask_array() + type(darr_x) # ... + + # note: no series.flush() needed + + +Example +------- + +A detailed example script for particle and field analysis is documented under as ``11_particle_dataframe.py`` in our :ref:`examples `. + +See a video of openPMD on DASK in action in `pull request #963 `__ (part of openPMD-api v0.14.0 and later). diff --git a/docs/source/analysis/pandas.rst b/docs/source/analysis/pandas.rst new file mode 100644 index 0000000000..dcfe97aae2 --- /dev/null +++ b/docs/source/analysis/pandas.rst @@ -0,0 +1,101 @@ +.. _analysis-pandas: + +Pandas +====== + +The Python bindings of openPMD-api provide direct methods to load data into the `Pandas data analysis ecosystem `__. + +Pandas computes on the CPU, for GPU-accelerated data analysis see :ref:`RAPIDS `. + + +.. _analysis-pandas-install: + +How to Install +-------------- + +Among many package managers, `PyPI `__ ships the latest packages of pandas: + +.. code-block:: python + + python3 -m pip install -U pandas + + +.. _analysis-pandas-df: + +Dataframes +---------- + +The central Python API call to convert to openPMD particles to a Pandas dataframe is the ``ParticleSpecies.to_df`` method. + +.. code-block:: python + + import openpmd_api as io + + s = io.Series("samples/git-sample/data%T.h5", io.Access.read_only) + electrons = s.iterations[400].particles["electrons"] + + df = electrons.to_df() + + type(df) # pd.DataFrame + print(df) + + # note: no series.flush() needed + +One can also combine all iterations in a single dataframe like this: + +.. code-block:: python + + import pandas as pd + + df = pd.concat( + ( + s.iterations[i].particles["electrons"].to_df().assign(iteration=i) + for i in s.iterations + ), + axis=0, + ignore_index=True, + ) + + # like before but with a new column "iteration" and all particles + print(df) + + +.. _analysis-pandas-ascii: + +openPMD to ASCII +---------------- + +Once converted to a Pandas dataframe, export of openPMD data to text is very simple. +We generally do not recommend this because ASCII processing is slower, uses significantly more space on disk and has less precision than the binary data usually stored in openPMD data series. +Nonetheless, in some cases and especially for small, human-readable data sets this can be helpful. + +The central Pandas call for this is `DataFrame.to_csv `__. + +.. code-block:: python + + # creates a electrons.csv file + df.to_csv("electrons.csv", sep=",", header=True) + + +.. _analysis-pandas-sql: + +openPMD as SQL Database +----------------------- + +Once converted to a Pandas dataframe, one can query and process openPMD data also with `SQL syntax `__ as provided by many databases. + +A project that provides such syntax is for instance `pandasql `__. + +.. code-block:: python + + python3 -m pip install -U pandasql + +or one can `export into an SQL database `__. + + +.. _analysis-pandas-example: + +Example +------- + +A detailed example script for particle and field analysis is documented under as ``11_particle_dataframe.py`` in our :ref:`examples `. diff --git a/docs/source/analysis/paraview.rst b/docs/source/analysis/paraview.rst new file mode 100644 index 0000000000..696f08bbb2 --- /dev/null +++ b/docs/source/analysis/paraview.rst @@ -0,0 +1,55 @@ +.. _analysis-paraview: + +3D Visualization: ParaView +========================== + +openPMD data can be visualized by ParaView, an open source visualization and analysis software. +ParaView can be downloaded and installed from httpshttps://www.paraview.org. +Use the latest version for best results. + +Tutorials +--------- + +ParaView is a powerful, general parallel rendering program. +If this is your first time using ParaView, consider starting with a tutorial. + +* https://www.paraview.org/Wiki/The_ParaView_Tutorial +* https://www.youtube.com/results?search_query=paraview+introduction +* https://www.youtube.com/results?search_query=paraview+tutorial + + +openPMD +------- + +openPMD files can be visualized with ParaView 5.9+, using 5.11+ is recommended. +ParaView supports ADIOS1, ADIOS2 and HDF5 files, as it implements against the Python bindings of openPMD-api. + +For openPMD output to be recognized, create a small textfile with ``.pmd`` ending per data series, which can be opened with ParaView: + +.. code-block:: console + + $ cat paraview.pmd + openpmd_%06T.bp + +The file contains the same string as one would put in an openPMD ``Series("....")`` object. + +.. tip:: + + When you first open ParaView, adjust its global ``Settings`` (Linux: under menu item ``Edit``). + ``General`` -> ``Advanced`` -> Search for ``data`` -> ``Data Processing Options``. + Check the box ``Auto Convert Properties``. + + This will simplify application of filters, e.g., contouring of components of vector fields, without first adding a calculator that extracts a single component or magnitude. + +.. warning:: + + As of ParaView 5.11 and older, the axisLabel is not yet read for fields. + See, e.g., `WarpX issue 21162 `__. + Please apply rotation of, e.g., ``0 -90 0`` to mesh data where needed. + +.. warning:: + + `ParaView issue 21837 `__: + In order to visualize particle traces with the ``Temporal Particles To Pathlines``, you need to apply the ``Merge Blocks`` filter first. + + If you have multiple species, you may have to extract the species you want with ``Extract Block`` before applying ``Merge Blocks``. diff --git a/docs/source/analysis/rapids.rst b/docs/source/analysis/rapids.rst new file mode 100644 index 0000000000..41acc55308 --- /dev/null +++ b/docs/source/analysis/rapids.rst @@ -0,0 +1,99 @@ +.. _analysis-rapids: + +RAPIDS +====== + +The Python bindings of openPMD-api enable easy loading into the GPU-accelerated `RAPIDS.ai datascience & AI/ML ecosystem `__. + + +.. _analysis-rapids-install: + +How to Install +-------------- + +Follow the `official documentation `__ to install RAPIDS. + +.. code-block:: python + + # preparation + conda update -n base conda + conda install -n base conda-libmamba-solver + conda config --set solver libmamba + + # install + conda create -n rapids -c rapidsai -c conda-forge -c nvidia rapids python cudatoolkit openpmd-api pandas + conda activate rapids + + +.. _analysis-rapids-cudf: + +Dataframes +---------- + +The central Python API call to convert to openPMD particles to a cuDF dataframe is the ``ParticleSpecies.to_df`` method. + +.. code-block:: python + + import openpmd_api as io + import cudf + + s = io.Series("samples/git-sample/data%T.h5", io.Access.read_only) + electrons = s.iterations[400].particles["electrons"] + + cdf = cudf.from_pandas(electrons.to_df()) + + type(cdf) # cudf.DataFrame + print(cdf) + + # note: no series.flush() needed + +One can also combine all iterations in a single dataframe like this: + +.. code-block:: python + + cdf = cudf.concat( + ( + cudf.from_pandas(s.iterations[i].particles["electrons"].to_df().assign(iteration=i)) + for i in s.iterations + ), + axis=0, + ignore_index=True, + ) + + # like before but with a new column "iteration" and all particles + print(cdf) + + +.. _analysis-rapids-sql: + +openPMD as SQL Database +----------------------- + +Once converted to a dataframe, one can query and process openPMD data also with `SQL syntax `__ as provided by many databases. + +A project that provides such syntax is for instance `BlazingSQL `__ (see the `BlazingSQL install documentation `__). + +.. code-block:: python + + import openpmd_api as io + from blazingsql import BlazingContext + + s = io.Series("samples/git-sample/data%T.h5", io.Access.read_only) + electrons = s.iterations[400].particles["electrons"] + + bc = BlazingContext(enable_progress_bar=True) + bc.create_table('electrons', electrons.to_df()) + + # all properties for electrons > 3e11 kg*m/s + bc.sql('SELECT * FROM electrons WHERE momentum_z > 3e11') + + # selected properties + bc.sql('SELECT momentum_x, momentum_y, momentum_z, weighting FROM electrons WHERE momentum_z > 3e11') + + +.. _analysis-rapids-example: + +Example +------- + +A detailed example script for particle and field analysis is documented under as ``11_particle_dataframe.py`` in our :ref:`examples `. diff --git a/docs/source/analysis/viewer.rst b/docs/source/analysis/viewer.rst new file mode 100644 index 0000000000..acfbd08a9e --- /dev/null +++ b/docs/source/analysis/viewer.rst @@ -0,0 +1,68 @@ +.. _analysis-viewer: + +openPMD-viewer +============== + +`openPMD-viewer `__ (`documentation `__) is a Python package to access openPMD data. + +It allows to: + +* Quickly browse through the data, with a GUI-type interface in the Jupyter notebook +* Have access to the data numpy array, for more detailed analysis + +Installation +------------ + +openPMD-viewer can be installed via ``conda`` or ``pip``: + +.. code-block:: bash + + conda install -c conda-forge openpmd-viewer openpmd-api + +.. code-block:: bash + + python3 -m pip install openPMD-viewer openPMD-api + +Usage +----- + +openPMD-viewer can be used either in simple Python scripts or in `Jupyter `__. +For interactive plots in Jupyter lab, add this `"cell magic" `__ to the first line of your notebook: + +.. code-block:: python + + %matplotlib widget + +and for Jupyter notebook use this instead: + +.. code-block:: python + + %matplotlib notebook + +If none of those work, e.g. because `ipympl `__ is not properly installed, you can as a last resort always try ``%matplotlib inline`` for non-interactive plots. + +In both interactive and scripted usage, you can import openPMD-viewer, and load the data with the following commands: + +.. code-block:: python + + from openpmd_viewer import OpenPMDTimeSeries + ts = OpenPMDTimeSeries('path/to/data/series/') + +.. note:: + + If you are using the Jupyter notebook, then you can start a pre-filled + notebook, which already contains the above lines, by typing in a terminal: + + :: + + openPMD_notebook + +When using the Jupyter notebook, you can quickly browse through the data +by using the command: + +:: + + ts.slider() + +You can also access the particle and field data as numpy arrays with the methods ``ts.get_field`` and ``ts.get_particle``. +See the openPMD-viewer tutorials `on read-the-docs `_ for more info. diff --git a/docs/source/backends/adios2.rst b/docs/source/backends/adios2.rst index 807538c2f6..65d10fbdfa 100644 --- a/docs/source/backends/adios2.rst +++ b/docs/source/backends/adios2.rst @@ -141,33 +141,11 @@ Be aware that extreme-scale I/O is a research topic after all. Experimental new ADIOS2 schema ------------------------------ -We are experimenting with a breaking change to our layout of openPMD datasets in ADIOS2. -It is likely that we will in future use ADIOS attributes only for a handful of internal flags. -Actual openPMD attributes will be modeled by ADIOS variables of the same name. -In order to distinguish datasets from attributes, datasets will be suffixed by ``/__data__``. +The experimental new ADIOS2 schema is deprecated and **will be removed soon**. It used to be activated via the JSON parameter ``adios2.schema = 20210209`` or via the environment variable ``export OPENPMD2_ADIOS2_SCHEMA=20210209``. -We hope that this will bring several advantages: +**Do no longer use these options**, the created datasets will no longer be read by the openPMD-api. -* Unlike ADIOS attributes, ADIOS variables are mutable. -* ADIOS variables are more closely related to the concept of ADIOS steps. - An ADIOS variable that is not written to in one step is not seen by the reader. - This will bring more manageable amounts of metadata for readers to parse through. - -The new layout may be activated **for experimental purposes** in two ways: - -* Via the JSON parameter ``adios2.schema = 20210209``. -* Via the environment variable ``export OPENPMD2_ADIOS2_SCHEMA=20210209``. - -The ADIOS2 backend will automatically recognize the layout that has been used by a writer when reading a dataset. - -.. tip:: - - This schema does not use ADIOS2 attributes anymore, thus ``bpls -a`` and ``bpls -A`` attribute switches do not show openPMD attributes. - Their functionality can be emulated via regexes: - - * Print datasets and attributes: Default behavior - * Print datasets only: ``bpls -e '.*/__data__$'`` - * Print attributes only: ``bpls -e '^(.(?!/__data__$))*$'`` +An alternative data layout with less intrusive changes and similar features is `currently in development `__. Memory usage ------------ @@ -264,7 +242,7 @@ In the openPMD-api, this can be done by specifying backend-specific parameters t .. code:: cpp - series.flush(R"({"adios2": {"preferred_flush_target": "disk"}})") + series.flush(R"({"adios2": {"engine": {"preferred_flush_target": "disk"}}})") The memory consumption of this approach shows that the 2GB buffer is first drained and then recreated after each ``flush()``: @@ -285,7 +263,7 @@ Note that this involves data copies that can be avoided by either flushing direc .. code:: cpp - series.flush(R"({"adios2": {"preferred_flush_target": "buffer"}})") + series.flush(R"({"adios2": {"engine": {"preferred_flush_target": "buffer"}}})") With this strategy, the BP5 engine will slowly build up its buffer until ending the step. Rather than by reallocation as in BP4, this is done by appending a new chunk, leading to a clearly more acceptable memory profile: diff --git a/docs/source/backends/hdf5.rst b/docs/source/backends/hdf5.rst index 4387b5dcf9..4786f7fa48 100644 --- a/docs/source/backends/hdf5.rst +++ b/docs/source/backends/hdf5.rst @@ -33,8 +33,9 @@ Environment variable Default Description ``OPENPMD_HDF5_PAGED_ALLOCATION_SIZE`` ``33554432`` Size of the page, in bytes, if HDF5 paged allocation optimization is enabled. ``OPENPMD_HDF5_DEFER_METADATA`` ``ON`` Tuning parameter for parallel I/O in HDF5 to enable deferred HDF5 metadata operations. ``OPENPMD_HDF5_DEFER_METADATA_SIZE`` ``ON`` Size of the buffer, in bytes, if HDF5 deferred metadata optimization is enabled. -``H5_COLL_API_SANITY_CHECK`` unset Debug: Set to ``1`` to perform an ``MPI_Barrier`` inside each meta-data operation. ``HDF5_USE_FILE_LOCKING`` ``TRUE`` Work-around: Set to ``FALSE`` in case you are on an HPC or network file system that hang in open for reads. +``HDF5_DO_MPI_FILE_SYNC`` driver-dep. Work-around: Set to ``FALSE`` to overcome MPI-I/O synchronization issues on some filesystems, e.g., NFS. +``H5_COLL_API_SANITY_CHECK`` unset Debug: Set to ``1`` to perform an ``MPI_Barrier`` inside each meta-data operation. ``OMPI_MCA_io`` unset Work-around: Disable OpenMPI's I/O implementation for older releases by setting this to ``^ompio``. ======================================== ============ =========================================================================================================== @@ -72,10 +73,6 @@ The metadata buffer size can be controlled by the ``OPENPMD_HDF5_DEFER_METADATA_ ``OPENPMD_HDF5_DEFER_METADATA_SIZE``: this option configures the size of the buffer if ``OPENPMD_HDF5_DEFER_METADATA`` optimization is enabled via `H5Pset_mdc_config `__. Values are expressed in bytes. Default is set to 32MB. -``H5_COLL_API_SANITY_CHECK``: this is a HDF5 control option for debugging parallel I/O logic (API calls). -Debugging a parallel program with that option enabled can help to spot bugs such as collective MPI-calls that are not called by all participating MPI ranks. -Do not use in production, this will slow parallel I/O operations down. - ``HDF5_USE_FILE_LOCKING``: this is a HDF5 1.10.1+ control option that disables HDF5 internal file locking operations (see `HDF5 1.10.1 release notes `__). This mechanism is mainly used to ensure that a file that is still being written to cannot (yet) be opened by either a reader or another writer. On some HPC and Jupyter systems, parallel/network file systems like GPFS are mounted in a way that interferes with this internal, HDF5 access consistency check. @@ -83,6 +80,17 @@ As a result, read-only operations like ``h5ls some_file.h5`` or openPMD ``Series If you are sure that the file was written completely and is closed by the writer, e.g., because a simulation finished that created HDF5 outputs, then you can set this environment variable to ``FALSE`` to work-around the problem. You should also report this problem to your system support, so they can fix the file system mount options or disable locking by default in the provided HDF5 installation. +``HDF5_DO_MPI_FILE_SYNC``: this is an MPI-parallel HDF5 1.14+ control option that adds an ``MPI_File_sync()`` call `after every collective write operation `__. +This is sometimes needed by the underlying parallel MPI-I/O driver if the filesystem has very limited parallel features. +Examples are NFS and UnifyFS, where this can be used to overcome synchronization issues/crashes. +The default value for this is *MPI-IO driver-dependent* and defaults to ``TRUE`` for these filesystems in newer HDF5 versions. +Setting the value back to ``FALSE`` has been shown to overcome `issues on NFS with parallel HDF5 `__. +Note that excessive sync calls can severely reduce parallel write performance, so ``TRUE`` should only be used when truly needed for correctness/stability. + +``H5_COLL_API_SANITY_CHECK``: this is a HDF5 control option for debugging parallel I/O logic (API calls). +Debugging a parallel program with that option enabled can help to spot bugs such as collective MPI-calls that are not called by all participating MPI ranks. +Do not use in production, this will slow parallel I/O operations down. + ``OMPI_MCA_io``: this is an OpenMPI control variable. OpenMPI implements its own MPI-I/O implementation backend *OMPIO*, starting with `OpenMPI 2.x `__ . This backend is known to cause problems in older releases that might still be in use on some systems. @@ -100,6 +108,12 @@ Known Issues Collective HDF5 metadata reads (``OPENPMD_HDF5_COLLECTIVE_METADATA=ON``) broke in 1.10.5, falling back to individual metadata operations. HDF5 releases 1.10.4 and earlier are not affected; versions 1.10.9+, 1.12.2+ and 1.13.1+ fixed the issue. +.. warning:: + + The ROMIO backend of OpenMPI has `a bug `__ that leads to segmentation faults in combination with parallel HDF5 I/O with chunking enabled. + This bug usually does not occur when using default configurations as OpenMPI `uses the OMPIO component by default `__. + The bug affects at least the entire OpenMPI 4.* release range and is currently set to be fixed for release 5.0 (release candidate available at the time of writing this). + Selected References ------------------- diff --git a/docs/source/citation.rst b/docs/source/citation.rst index 858f67bf1b..2291d8a3a9 100644 --- a/docs/source/citation.rst +++ b/docs/source/citation.rst @@ -30,6 +30,7 @@ The equivalent BibTeX code is: Sbalzarini, Ivo and Kuschel, Stephan and Sagan, David and + Mayes, Christopher and P{\'e}rez, Fr{\'e}d{\'e}ric and Koller, Fabian and Bussmann, Michael}, diff --git a/docs/source/conf.py b/docs/source/conf.py index 2096024ad2..95720591e5 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -38,6 +38,8 @@ # ones. extensions = ['sphinx.ext.mathjax', 'breathe', + 'sphinx_copybutton', + 'sphinx_design', 'sphinxcontrib.programoutput', 'sphinxcontrib.rsvgconverter', 'matplotlib.sphinxext.plot_directive'] @@ -85,9 +87,9 @@ # built documents. # # The short X.Y version. -version = u'0.15.1' +version = u'0.15.2' # The full version, including alpha/beta/rc tags. -release = u'0.15.1' +release = u'0.15.2' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/docs/source/dev/dependencies.rst b/docs/source/dev/dependencies.rst index d8c441ea07..7fccba587d 100644 --- a/docs/source/dev/dependencies.rst +++ b/docs/source/dev/dependencies.rst @@ -39,7 +39,7 @@ Optional: language bindings * Python: - * Python 3.7 - 3.11 + * Python 3.8 - 3.11 * pybind11 2.10.1+ * numpy 1.15+ * mpi4py 2.1+ (optional, for MPI) diff --git a/docs/source/dev/linking.rst b/docs/source/dev/linking.rst new file mode 100644 index 0000000000..64858cb25f --- /dev/null +++ b/docs/source/dev/linking.rst @@ -0,0 +1,88 @@ +.. _development-linking: + +Linking to C++ +============== + +The install will contain header files and libraries in the path set with the ``-DCMAKE_INSTALL_PREFIX`` option :ref:`from the previous section `. + + +CMake +----- + +If your project is using CMake for its build, one can conveniently use our provided ``openPMDConfig.cmake`` package, which is installed alongside the library. + +First set the following environment hint if openPMD-api was *not* installed in a system path: + +.. code-block:: bash + + # optional: only needed if installed outside of system paths + export CMAKE_PREFIX_PATH=$HOME/somepath:$CMAKE_PREFIX_PATH + +Use the following lines in your project's ``CMakeLists.txt``: + +.. code-block:: cmake + + # supports: COMPONENTS MPI NOMPI HDF5 ADIOS2 + find_package(openPMD 0.15.0 CONFIG) + + if(openPMD_FOUND) + target_link_libraries(YourTarget PRIVATE openPMD::openPMD) + endif() + +*Alternatively*, add the openPMD-api repository source directly to your project and use it via: + +.. code-block:: cmake + + add_subdirectory("path/to/source/of/openPMD-api") + + target_link_libraries(YourTarget PRIVATE openPMD::openPMD) + +For development workflows, you can even automatically download and build openPMD-api from within a depending CMake project. +Just replace the ``add_subdirectory`` call with: + +.. code-block:: cmake + + include(FetchContent) + set(CMAKE_POLICY_DEFAULT_CMP0077 NEW) + set(openPMD_BUILD_CLI_TOOLS OFF) + set(openPMD_BUILD_EXAMPLES OFF) + set(openPMD_BUILD_TESTING OFF) + set(openPMD_BUILD_SHARED_LIBS OFF) # precedence over BUILD_SHARED_LIBS if needed + set(openPMD_INSTALL OFF) # or instead use: + # set(openPMD_INSTALL ${BUILD_SHARED_LIBS}) # only install if used as a shared library + set(openPMD_USE_PYTHON OFF) + FetchContent_Declare(openPMD + GIT_REPOSITORY "https://github.com/openPMD/openPMD-api.git" + GIT_TAG "0.15.0") + FetchContent_MakeAvailable(openPMD) + + +Manually +-------- + +If your (Linux/OSX) project is build by calling the compiler directly or uses a manually written ``Makefile``, consider using our ``openPMD.pc`` helper file for ``pkg-config``, which are installed alongside the library. + +First set the following environment hint if openPMD-api was *not* installed in a system path: + +.. code-block:: bash + + # optional: only needed if installed outside of system paths + export PKG_CONFIG_PATH=$HOME/somepath/lib/pkgconfig:$PKG_CONFIG_PATH + +Additional linker and compiler flags for your project are available via: + +.. code-block:: bash + + # switch to check if openPMD-api was build as static library + # (via BUILD_SHARED_LIBS=OFF) or as shared library (default) + if [ "$(pkg-config --variable=static openPMD)" == "true" ] + then + pkg-config --libs --static openPMD + # -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lopenPMD -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.so /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so /usr/lib/x86_64-linux-gnu/libsz.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/libdl.so /usr/lib/x86_64-linux-gnu/libm.so -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.so + else + pkg-config --libs openPMD + # -L${HOME}/somepath/lib -lopenPMD + fi + + pkg-config --cflags openPMD + # -I${HOME}/somepath/include diff --git a/docs/source/index.rst b/docs/source/index.rst index 37b1e4c630..9df323d876 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -21,6 +21,7 @@ Writing & reading through those backends and their associated files is supported section#api-details, section#utilities, section#backends, + section#data-analysis, section#development, section#maintenance { display:none; @@ -43,7 +44,7 @@ openPMD-api version supported openPMD standard versions ======================== =================================== ``2.0.0+`` ``2.0.0+`` (not released yet) ``1.0.0+`` ``1.0.1-1.1.0`` (not released yet) -``0.13.1-0.15.1`` (beta) ``1.0.0-1.1.0`` +``0.13.1-0.15.2`` (beta) ``1.0.0-1.1.0`` ``0.1.0-0.12.0`` (alpha) ``1.0.0-1.1.0`` ======================== =================================== @@ -128,6 +129,20 @@ Backends backends/adios2 backends/hdf5 +Data Analysis +------------- +.. toctree:: + :caption: DATA ANALYSIS + :maxdepth: 1 + :hidden: + + analysis/viewer + analysis/paraview + analysis/pandas + analysis/dask + analysis/rapids + analysis/contrib + Development ----------- .. toctree:: @@ -141,6 +156,7 @@ Development dev/backend dev/dependencies dev/buildoptions + dev/linking dev/sphinx Maintenance diff --git a/docs/source/usage/firstread.rst b/docs/source/usage/firstread.rst index f3edf391f3..9a01f4b0fa 100644 --- a/docs/source/usage/firstread.rst +++ b/docs/source/usage/firstread.rst @@ -347,12 +347,11 @@ C++17 .. code-block:: cpp - // destruct series object, - // e.g. when out-of-scope + series.close() Python ^^^^^^ .. code-block:: python3 - del series + series.close() diff --git a/docs/source/usage/firstwrite.rst b/docs/source/usage/firstwrite.rst index 193530f09f..a42652d7d9 100644 --- a/docs/source/usage/firstwrite.rst +++ b/docs/source/usage/firstwrite.rst @@ -297,11 +297,9 @@ C++17 .. code-block:: cpp B_x.storeChunk( - io::shareRaw(x_data), - {0, 0}, {150, 300}); + x_data, {0, 0}, {150, 300}); B_z.storeChunk( - io::shareRaw(z_data), - {0, 0}, {150, 300}); + z_data, {0, 0}, {150, 300}); B_y.makeConstant(y_data); @@ -310,10 +308,10 @@ Python .. code-block:: python3 - B_x.store_chunk(x_data) + B_x[:, :] = x_data - B_z.store_chunk(z_data) + B_z[:, :] = z_data @@ -354,12 +352,11 @@ C++17 .. code-block:: cpp - // destruct series object, - // e.g. when out-of-scope + series.close() Python ^^^^^^ .. code-block:: python3 - del series + series.close() diff --git a/docs/source/usage/workflow.rst b/docs/source/usage/workflow.rst index 64194629ba..f4171a1415 100644 --- a/docs/source/usage/workflow.rst +++ b/docs/source/usage/workflow.rst @@ -1,7 +1,10 @@ .. _workflow: +Workflow +======== + Access modes -============ +------------ The openPMD-api distinguishes between a number of different access modes: @@ -61,9 +64,6 @@ The openPMD-api distinguishes between a number of different access modes: It is a user's responsibility to ensure that the appended dataset and the appended-to dataset are compatible with each other. The results of using incompatible backend configurations are undefined. -Workflow -======== - Deferred Data API Contract -------------------------- @@ -99,3 +99,9 @@ Attributes are (currently) unaffected by this: Some backends (e.g. the BP5 engine of ADIOS2) have multiple implementations for the openPMD-api-level guarantees of flush points. For user-guided selection of such implementations, ``Series::flush`` and ``Attributable::seriesFlush()`` take an optional JSON/TOML string as a parameter. See the section on :ref:`backend-specific configuration ` for details. + +Deferred Data API Contract +-------------------------- + +A verbose debug log can optionally be printed to the standard error output by specifying the environment variable ``OPENPMD_VERBOSE=1``. +Note that this functionality is at the current time still relatively basic. diff --git a/examples/10_streaming_read.cpp b/examples/10_streaming_read.cpp index eae79dd28a..6e6aba1fd9 100644 --- a/examples/10_streaming_read.cpp +++ b/examples/10_streaming_read.cpp @@ -19,8 +19,22 @@ int main() return 0; } - Series series = Series("electrons.sst", Access::READ_LINEAR); + Series series = Series("electrons.sst", Access::READ_LINEAR, R"( +{ + "adios2": { + "engine": { + "parameters": { + "DataTransport": "WAN" + } + } + } +})"); + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. + // `Series::iterations` can be directly accessed in random-access workflows. for (IndexedIteration iteration : series.readIterations()) { std::cout << "Current iteration: " << iteration.iterationIndex diff --git a/examples/10_streaming_read.py b/examples/10_streaming_read.py index 5d0f688b94..f33d778842 100755 --- a/examples/10_streaming_read.py +++ b/examples/10_streaming_read.py @@ -7,7 +7,8 @@ # pass-through for ADIOS2 engine parameters # https://adios2.readthedocs.io/en/latest/engines/engines.html config = {'adios2': {'engine': {}, 'dataset': {}}} -config['adios2']['engine'] = {'parameters': {'Threads': '4'}} +config['adios2']['engine'] = {'parameters': + {'Threads': '4', 'DataTransport': 'WAN'}} config['adios2']['dataset'] = {'operators': [{'type': 'bzip2'}]} if __name__ == "__main__": @@ -21,12 +22,13 @@ json.dumps(config)) # Read all available iterations and print electron position data. - # Use `series.read_iterations()` instead of `series.iterations` - # for streaming support (while still retaining file-reading support). - # Direct access to `series.iterations` is only necessary for random-access - # of iterations. By using `series.read_iterations()`, the openPMD-api will - # step through the iterations one by one, and going back to an iteration is - # not possible once it has been closed. + # Direct access to iterations is possible via `series.iterations`. + # For streaming support, `series.read_iterations()` needs to be used + # instead of `series.iterations`. + # `Series.write_iterations()` and `Series.read_iterations()` are + # intentionally restricted APIs that ensure a workflow which also works + # in streaming setups, e.g. an iteration cannot be opened again once + # it has been closed. for iteration in series.read_iterations(): print("Current iteration {}".format(iteration.iteration_index)) electronPositions = iteration.particles["e"]["position"] diff --git a/examples/10_streaming_write.cpp b/examples/10_streaming_write.cpp index 57bbcb6287..2eb825ae4a 100644 --- a/examples/10_streaming_write.cpp +++ b/examples/10_streaming_write.cpp @@ -20,7 +20,16 @@ int main() } // open file for writing - Series series = Series("electrons.sst", Access::CREATE); + Series series = Series("electrons.sst", Access::CREATE, R"( +{ + "adios2": { + "engine": { + "parameters": { + "DataTransport": "WAN" + } + } + } +})"); Datatype datatype = determineDatatype(); constexpr unsigned long length = 10ul; @@ -29,6 +38,11 @@ int main() std::shared_ptr local_data( new position_t[length], [](position_t const *ptr) { delete[] ptr; }); + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. + // `Series::iterations` can be directly accessed in random-access workflows. WriteIterations iterations = series.writeIterations(); for (size_t i = 0; i < 100; ++i) { diff --git a/examples/10_streaming_write.py b/examples/10_streaming_write.py index 956b683b05..e0b830bd35 100755 --- a/examples/10_streaming_write.py +++ b/examples/10_streaming_write.py @@ -8,7 +8,8 @@ # pass-through for ADIOS2 engine parameters # https://adios2.readthedocs.io/en/latest/engines/engines.html config = {'adios2': {'engine': {}, 'dataset': {}}} -config['adios2']['engine'] = {'parameters': {'Threads': '4'}} +config['adios2']['engine'] = {'parameters': + {'Threads': '4', 'DataTransport': 'WAN'}} config['adios2']['dataset'] = {'operators': [{'type': 'bzip2'}]} if __name__ == "__main__": @@ -27,13 +28,13 @@ # now, write a number of iterations (or: snapshots, time steps) for i in range(10): - # Use `series.write_iterations()` instead of `series.iterations` - # for streaming support (while still retaining file-writing support). - # Direct access to `series.iterations` is only necessary for - # random-access of iterations. By using `series.write_iterations()`, - # the openPMD-api will adhere to streaming semantics while writing. - # In particular, this means that only one iteration can be written at a - # time and an iteration can no longer be modified after closing it. + # Direct access to iterations is possible via `series.iterations`. + # For streaming support, `series.write_iterations()` needs to be used + # instead of `series.iterations`. + # `Series.write_iterations()` and `Series.read_iterations()` are + # intentionally restricted APIs that ensure a workflow which also works + # in streaming setups, e.g. an iteration cannot be opened again once + # it has been closed. iteration = series.write_iterations()[i] ####################### diff --git a/examples/12_span_write.cpp b/examples/12_span_write.cpp index d53181cea0..089ceddff3 100644 --- a/examples/12_span_write.cpp +++ b/examples/12_span_write.cpp @@ -19,6 +19,11 @@ void span_write(std::string const &filename) std::vector fallbackBuffer; + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. + // `Series::iterations` can be directly accessed in random-access workflows. WriteIterations iterations = series.writeIterations(); for (size_t i = 0; i < 10; ++i) { diff --git a/examples/12_span_write.py b/examples/12_span_write.py index bfe0f69784..c985192383 100644 --- a/examples/12_span_write.py +++ b/examples/12_span_write.py @@ -10,6 +10,11 @@ def span_write(filename): extent = [length] dataset = io.Dataset(datatype, extent) + # `Series.write_iterations()` and `Series.read_iterations()` are + # intentionally restricted APIs that ensure a workflow which also works + # in streaming setups, e.g. an iteration cannot be opened again once + # it has been closed. + # `Series.iterations` can be directly accessed in random-access workflows. iterations = series.write_iterations() for i in range(12): iteration = iterations[i] diff --git a/examples/13_write_dynamic_configuration.cpp b/examples/13_write_dynamic_configuration.cpp index a398eccf27..b6e7f3694c 100644 --- a/examples/13_write_dynamic_configuration.cpp +++ b/examples/13_write_dynamic_configuration.cpp @@ -75,6 +75,11 @@ chunks = "auto" std::shared_ptr local_data( new position_t[length], [](position_t const *ptr) { delete[] ptr; }); + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. + // `Series::iterations` can be directly accessed in random-access workflows. WriteIterations iterations = series.writeIterations(); for (size_t i = 0; i < 100; ++i) { diff --git a/examples/13_write_dynamic_configuration.py b/examples/13_write_dynamic_configuration.py index 8670961592..bb7e81ce4a 100644 --- a/examples/13_write_dynamic_configuration.py +++ b/examples/13_write_dynamic_configuration.py @@ -63,13 +63,13 @@ def main(): # now, write a number of iterations (or: snapshots, time steps) for i in range(10): - # Use `series.write_iterations()` instead of `series.iterations` - # for streaming support (while still retaining file-writing support). - # Direct access to `series.iterations` is only necessary for - # random-access of iterations. By using `series.write_iterations()`, - # the openPMD-api will adhere to streaming semantics while writing. - # In particular, this means that only one iteration can be written at a - # time and an iteration can no longer be modified after closing it. + # Direct access to iterations is possible via `series.iterations`. + # For streaming support, `series.write_iterations()` needs to be used + # instead of `series.iterations`. + # `Series.write_iterations()` and `Series.read_iterations()` are + # intentionally restricted APIs that ensure a workflow which also works + # in streaming setups, e.g. an iteration cannot be opened again once + # it has been closed. iteration = series.write_iterations()[i] ####################### diff --git a/examples/1_structure.cpp b/examples/1_structure.cpp index fe4381884f..6e595c56ba 100644 --- a/examples/1_structure.cpp +++ b/examples/1_structure.cpp @@ -38,7 +38,13 @@ int main() /* Access to individual positions inside happens hierarchically, according * to the openPMD standard. Creation of new elements happens on access * inside the tree-like structure. Required attributes are initialized to - * reasonable defaults for every object. */ + * reasonable defaults for every object. + * `Series::writeIterations()` and `Series::readIterations()` are + * intentionally restricted APIs that ensure a workflow which also works + * in streaming setups, e.g. an iteration cannot be opened again once + * it has been closed. + * `Series::iterations` can be directly accessed in random-access workflows. + */ ParticleSpecies electrons = series.writeIterations()[1].particles["electrons"]; diff --git a/examples/3_write_serial.cpp b/examples/3_write_serial.cpp index a66db6c080..aeb62aef6c 100644 --- a/examples/3_write_serial.cpp +++ b/examples/3_write_serial.cpp @@ -44,6 +44,11 @@ int main(int argc, char *argv[]) Series series = Series("../samples/3_write_serial.h5", Access::CREATE); cout << "Created an empty " << series.iterationEncoding() << " Series\n"; + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. + // `Series::iterations` can be directly accessed in random-access workflows. MeshRecordComponent rho = series.writeIterations()[1].meshes["rho"][MeshRecordComponent::SCALAR]; cout << "Created a scalar mesh Record with all required openPMD " diff --git a/examples/3_write_serial.py b/examples/3_write_serial.py index 8e136f9512..b1bdd2d063 100644 --- a/examples/3_write_serial.py +++ b/examples/3_write_serial.py @@ -28,6 +28,11 @@ print("Created an empty {0} Series".format(series.iteration_encoding)) print(len(series.iterations)) + # `Series.write_iterations()` and `Series.read_iterations()` are + # intentionally restricted APIs that ensure a workflow which also works + # in streaming setups, e.g. an iteration cannot be opened again once + # it has been closed. + # `Series.iterations` can be directly accessed in random-access workflows. rho = series.write_iterations()[1]. \ meshes["rho"][io.Mesh_Record_Component.SCALAR] diff --git a/examples/3a_write_thetaMode_serial.cpp b/examples/3a_write_thetaMode_serial.cpp index 9367e43f70..1e5086303f 100644 --- a/examples/3a_write_thetaMode_serial.cpp +++ b/examples/3a_write_thetaMode_serial.cpp @@ -51,6 +51,11 @@ int main() geos << "m=" << num_modes << ";imag=+"; std::string const geometryParameters = geos.str(); + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. + // `Series::iterations` can be directly accessed in random-access workflows. Mesh E = series.writeIterations()[0].meshes["E"]; E.setGeometry(Mesh::Geometry::thetaMode); E.setGeometryParameters(geometryParameters); diff --git a/examples/3a_write_thetaMode_serial.py b/examples/3a_write_thetaMode_serial.py index ec81435558..c9aee2c9d7 100644 --- a/examples/3a_write_thetaMode_serial.py +++ b/examples/3a_write_thetaMode_serial.py @@ -30,6 +30,11 @@ geometry_parameters = "m={0};imag=+".format(num_modes) + # `Series.write_iterations()` and `Series.read_iterations()` are + # intentionally restricted APIs that ensure a workflow which also works + # in streaming setups, e.g. an iteration cannot be opened again once + # it has been closed. + # `Series.iterations` can be directly accessed in random-access workflows. E = series.write_iterations()[0].meshes["E"] E.geometry = io.Geometry.thetaMode E.geometry_parameters = geometry_parameters diff --git a/examples/3b_write_resizable_particles.cpp b/examples/3b_write_resizable_particles.cpp index d4be87a0fc..91face7f2b 100644 --- a/examples/3b_write_resizable_particles.cpp +++ b/examples/3b_write_resizable_particles.cpp @@ -32,6 +32,11 @@ int main() Series series = Series("../samples/3b_write_resizable_particles.h5", Access::CREATE); + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. + // `Series::iterations` can be directly accessed in random-access workflows. ParticleSpecies electrons = series.writeIterations()[0].particles["electrons"]; diff --git a/examples/3b_write_resizable_particles.py b/examples/3b_write_resizable_particles.py index 440fac7de6..f188559816 100644 --- a/examples/3b_write_resizable_particles.py +++ b/examples/3b_write_resizable_particles.py @@ -64,7 +64,7 @@ # The iteration's content will be flushed automatically. # An iteration once closed cannot (yet) be reopened. # after this call, the provided data buffers can be used again or deleted - series.write_iterations()[0].close() + series.iterations[0].close() # rinse and repeat as needed :) diff --git a/examples/5_write_parallel.cpp b/examples/5_write_parallel.cpp index bfe737d9be..2b70c775cb 100644 --- a/examples/5_write_parallel.cpp +++ b/examples/5_write_parallel.cpp @@ -55,8 +55,14 @@ int main(int argc, char *argv[]) << " MPI ranks\n"; // In parallel contexts, it's important to explicitly open iterations. - // This is done automatically when using `Series::writeIterations()`, - // or in read mode `Series::readIterations()`. + // You can either explicitly access Series::iterations and use + // Iteration::open() afterwards, or use `Series::writeIterations()`, + // or in read mode `Series::readIterations()` where iterations are opened + // automatically. + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. series.iterations[1].open(); MeshRecordComponent mymesh = series.iterations[1].meshes["mymesh"][MeshRecordComponent::SCALAR]; diff --git a/examples/5_write_parallel.py b/examples/5_write_parallel.py index c956b6eed1..27aa01f94c 100644 --- a/examples/5_write_parallel.py +++ b/examples/5_write_parallel.py @@ -40,6 +40,12 @@ # In parallel contexts, it's important to explicitly open iterations. # This is done automatically when using `Series.write_iterations()`, # or in read mode `Series.read_iterations()`. + # + # `Series.write_iterations()` and `Series.read_iterations()` are + # intentionally restricted APIs that ensure a workflow which also works + # in streaming setups, e.g. an iteration cannot be opened again once + # it has been closed. + # `Series.iterations` can be directly accessed in random-access workflows. series.iterations[1].open() mymesh = series.iterations[1]. \ meshes["mymesh"][io.Mesh_Record_Component.SCALAR] diff --git a/examples/7_extended_write_serial.cpp b/examples/7_extended_write_serial.cpp index bfb64e1fff..580894a34f 100644 --- a/examples/7_extended_write_serial.cpp +++ b/examples/7_extended_write_serial.cpp @@ -83,8 +83,9 @@ int main() {{io::UnitDimension::M, 1}}); electrons["displacement"]["x"].setUnitSI(1e-6); electrons.erase("displacement"); - electrons["weighting"][io::RecordComponent::SCALAR].makeConstant( - 1.e-5); + electrons["weighting"][io::RecordComponent::SCALAR] + .resetDataset({io::Datatype::FLOAT, {1}}) + .makeConstant(1.e-5); } io::Mesh mesh = cur_it.meshes["lowRez_2D_field"]; diff --git a/examples/7_extended_write_serial.py b/examples/7_extended_write_serial.py index 84ca5002db..e16b4b993a 100755 --- a/examples/7_extended_write_serial.py +++ b/examples/7_extended_write_serial.py @@ -90,7 +90,9 @@ electrons["displacement"].unit_dimension = {Unit_Dimension.M: 1} electrons["displacement"]["x"].unit_SI = 1.e-6 del electrons["displacement"] - electrons["weighting"][SCALAR].make_constant(1.e-5) + electrons["weighting"][SCALAR] \ + .reset_dataset(Dataset(np.dtype("float32"), extent=[1])) \ + .make_constant(1.e-5) mesh = cur_it.meshes["lowRez_2D_field"] mesh.axis_labels = ["x", "y"] diff --git a/examples/8a_benchmark_write_parallel.cpp b/examples/8a_benchmark_write_parallel.cpp index 82c32ce73a..c509c879ff 100644 --- a/examples/8a_benchmark_write_parallel.cpp +++ b/examples/8a_benchmark_write_parallel.cpp @@ -748,6 +748,11 @@ void AbstractPattern::store(Series &series, int step) std::string scalar = openPMD::MeshRecordComponent::SCALAR; storeMesh(series, step, field_rho, scalar); + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. + // `Series::iterations` can be directly accessed in random-access workflows. ParticleSpecies &currSpecies = series.writeIterations()[step].particles["ion"]; storeParticles(currSpecies, step); @@ -770,6 +775,11 @@ void AbstractPattern::storeMesh( const std::string &fieldName, const std::string &compName) { + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also works + // in streaming setups, e.g. an iteration cannot be opened again once + // it has been closed. + // `Series::iterations` can be directly accessed in random-access workflows. MeshRecordComponent compA = series.writeIterations()[step].meshes[fieldName][compName]; Datatype datatype = determineDatatype(); diff --git a/examples/8b_benchmark_read_parallel.cpp b/examples/8b_benchmark_read_parallel.cpp index 3809707a72..98cd81add2 100644 --- a/examples/8b_benchmark_read_parallel.cpp +++ b/examples/8b_benchmark_read_parallel.cpp @@ -274,6 +274,11 @@ class TestInput << std::endl; } + // `Series::writeIterations()` and `Series::readIterations()` are + // intentionally restricted APIs that ensure a workflow which also + // works in streaming setups, e.g. an iteration cannot be opened + // again once it has been closed. `Series::iterations` can be + // directly accessed in random-access workflows. { int counter = 1; for (auto i : series.readIterations()) diff --git a/examples/9_particle_write_serial.py b/examples/9_particle_write_serial.py index aebd266528..5af8796d49 100644 --- a/examples/9_particle_write_serial.py +++ b/examples/9_particle_write_serial.py @@ -16,7 +16,7 @@ if __name__ == "__main__": # open file for writing f = Series( - "../samples/7_particle_write_serial_py.h5", + "../samples/9_particle_write_serial_py.h5", Access.create ) @@ -35,25 +35,29 @@ "Electrons... the necessary evil for ion acceleration! ", "Just kidding.") + n_particles = 234 + # let's set a weird user-defined record this time electrons["displacement"].unit_dimension = {Unit_Dimension.M: 1} electrons["displacement"][SCALAR].unit_SI = 1.e-6 - dset = Dataset(np.dtype("float64"), extent=[2]) + dset = Dataset(np.dtype("float64"), extent=[n_particles]) electrons["displacement"][SCALAR].reset_dataset(dset) electrons["displacement"][SCALAR].make_constant(42.43) # don't like it anymore? remove it with: # del electrons["displacement"] - electrons["weighting"][SCALAR].make_constant(1.e-5) + electrons["weighting"][SCALAR] \ + .reset_dataset(Dataset(np.dtype("float32"), extent=[n_particles])) \ + .make_constant(1.e-5) - particlePos_x = np.random.rand(234).astype(np.float32) - particlePos_y = np.random.rand(234).astype(np.float32) + particlePos_x = np.random.rand(n_particles).astype(np.float32) + particlePos_y = np.random.rand(n_particles).astype(np.float32) d = Dataset(particlePos_x.dtype, extent=particlePos_x.shape) electrons["position"]["x"].reset_dataset(d) electrons["position"]["y"].reset_dataset(d) - particleOff_x = np.arange(234, dtype=np.uint) - particleOff_y = np.arange(234, dtype=np.uint) + particleOff_x = np.arange(n_particles, dtype=np.uint) + particleOff_y = np.arange(n_particles, dtype=np.uint) d = Dataset(particleOff_x.dtype, particleOff_x.shape) electrons["positionOffset"]["x"].reset_dataset(d) electrons["positionOffset"]["y"].reset_dataset(d) diff --git a/include/openPMD/Datatype.hpp b/include/openPMD/Datatype.hpp index f9661b60e8..05d0ddefbb 100644 --- a/include/openPMD/Datatype.hpp +++ b/include/openPMD/Datatype.hpp @@ -92,7 +92,7 @@ enum class Datatype : int * listed in order in a vector. * */ -extern std::vector openPMD_Datatypes; +std::vector openPMD_Datatypes(); /** @brief Fundamental equivalence check for two given types T and U. * @@ -651,6 +651,41 @@ inline bool isSameInteger(Datatype d) return false; } +/** + * Determines if d represents a char type. + * + * @param d An openPMD datatype. + * @return true If d is a scalar char, signed char or unsigned char. + * @return false Otherwise. + */ +constexpr bool isChar(Datatype d) +{ + switch (d) + { + case Datatype::CHAR: + case Datatype::SCHAR: + case Datatype::UCHAR: + return true; + default: + return false; + } +} + +/** + * Determines if d and T_Char are char types of same representation. + * + * Same representation means that on platforms with signed `char` type, `char` + * and `signed char` are considered to be eqivalent, similarly on platforms + * with unsigned `char` type. + * + * @tparam T_Char A type, as template parameter. + * @param d A type, as openPMD datatype. + * @return true If both types are chars of the same representation. + * @return false Otherwise. + */ +template +constexpr bool isSameChar(Datatype d); + /** Comparison for two Datatypes * * Besides returning true for the same types, identical implementations on @@ -710,6 +745,43 @@ void warnWrongDtype(std::string const &key, Datatype store, Datatype request); std::ostream &operator<<(std::ostream &, openPMD::Datatype const &); +/** + * Generalizes switching over an openPMD datatype. + * + * Will call the function template found at Action::call< T >(), instantiating T + * with the C++ internal datatype corresponding to the openPMD datatype. + * + * @tparam ReturnType The function template's return type. + * @tparam Action The struct containing the function template. + * @tparam Args The function template's argument types. + * @param dt The openPMD datatype. + * @param args The function template's arguments. + * @return Passes on the result of invoking the function template with the given + * arguments and with the template parameter specified by dt. + */ +template +constexpr auto switchType(Datatype dt, Args &&...args) + -> decltype(Action::template call(std::forward(args)...)); + +/** + * Generalizes switching over an openPMD datatype. + * + * Will call the function template found at Action::call< T >(), instantiating T + * with the C++ internal datatype corresponding to the openPMD datatype. + * Ignores vector and array types. + * + * @tparam ReturnType The function template's return type. + * @tparam Action The struct containing the function template. + * @tparam Args The function template's argument types. + * @param dt The openPMD datatype. + * @param args The function template's arguments. + * @return Passes on the result of invoking the function template with the given + * arguments and with the template parameter specified by dt. + */ +template +constexpr auto switchNonVectorType(Datatype dt, Args &&...args) + -> decltype(Action::template call(std::forward(args)...)); + } // namespace openPMD #if !defined(_MSC_VER) @@ -737,3 +809,5 @@ inline bool operator!=(openPMD::Datatype d, openPMD::Datatype e) /** @} */ #endif + +#include "openPMD/Datatype.tpp" diff --git a/include/openPMD/Datatype.tpp b/include/openPMD/Datatype.tpp new file mode 100644 index 0000000000..64d6d30d5b --- /dev/null +++ b/include/openPMD/Datatype.tpp @@ -0,0 +1,356 @@ +/* Copyright 2017-2023 Fabian Koller, Franz Poeschel, Axel Huebl + * + * This file is part of openPMD-api. + * + * openPMD-api is free software: you can redistribute it and/or modify + * it under the terms of of either the GNU General Public License or + * the GNU Lesser General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * openPMD-api is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License and the GNU Lesser General Public License + * for more details. + * + * You should have received a copy of the GNU General Public License + * and the GNU Lesser General Public License along with openPMD-api. + * If not, see . + */ +#pragma once + +#include "openPMD/Datatype.hpp" + +#include +#include // std::void_t +#include // std::forward + +namespace openPMD +{ +namespace detail +{ + // std::void_t is C++17 + template + using void_t = void; + + /* + * Check whether class T has a member "errorMsg" convertible + * to type std::string. + * Used to give helpful compile-time error messages with static_assert + * down in CallUndefinedDatatype. + */ + template + struct HasErrorMessageMember + { + static constexpr bool value = false; + }; + + template + struct HasErrorMessageMember> + { + static constexpr bool value = true; + }; + + /** + * Purpose of this struct is to detect at compile time whether + * Action::template operator()\<0\>() exists. If yes, call + * Action::template operator()\() with the passed arguments. + * If not, throw an error. + * + * @tparam n As in switchType(). + * @tparam ReturnType As in switchType(). + * @tparam Action As in switchType(). + * @tparam Args As in switchType(). + */ + template + struct CallUndefinedDatatype + { + static ReturnType call(Args &&...args) + { + if constexpr (HasErrorMessageMember::value) + { + throw std::runtime_error( + "[" + std::string(Action::errorMsg) + + "] Unknown Datatype."); + } + else + { + return Action::template call(std::forward(args)...); + } + throw std::runtime_error("Unreachable!"); + } + }; +} // namespace detail + +/** + * Generalizes switching over an openPMD datatype. + * + * Will call the function template found at Action::call< T >(), instantiating T + * with the C++ internal datatype corresponding to the openPMD datatype. + * + * @tparam ReturnType The function template's return type. + * @tparam Action The struct containing the function template. + * @tparam Args The function template's argument types. + * @param dt The openPMD datatype. + * @param args The function template's arguments. + * @return Passes on the result of invoking the function template with the given + * arguments and with the template parameter specified by dt. + */ +template +constexpr auto switchType(Datatype dt, Args &&...args) + -> decltype(Action::template call(std::forward(args)...)) +{ + using ReturnType = + decltype(Action::template call(std::forward(args)...)); + switch (dt) + { + case Datatype::CHAR: + return Action::template call(std::forward(args)...); + case Datatype::UCHAR: + return Action::template call( + std::forward(args)...); + case Datatype::SCHAR: + return Action::template call(std::forward(args)...); + case Datatype::SHORT: + return Action::template call(std::forward(args)...); + case Datatype::INT: + return Action::template call(std::forward(args)...); + case Datatype::LONG: + return Action::template call(std::forward(args)...); + case Datatype::LONGLONG: + return Action::template call(std::forward(args)...); + case Datatype::USHORT: + return Action::template call( + std::forward(args)...); + case Datatype::UINT: + return Action::template call(std::forward(args)...); + case Datatype::ULONG: + return Action::template call( + std::forward(args)...); + case Datatype::ULONGLONG: + return Action::template call( + std::forward(args)...); + case Datatype::FLOAT: + return Action::template call(std::forward(args)...); + case Datatype::DOUBLE: + return Action::template call(std::forward(args)...); + case Datatype::LONG_DOUBLE: + return Action::template call(std::forward(args)...); + case Datatype::CFLOAT: + return Action::template call>( + std::forward(args)...); + case Datatype::CDOUBLE: + return Action::template call>( + std::forward(args)...); + case Datatype::CLONG_DOUBLE: + return Action::template call>( + std::forward(args)...); + case Datatype::STRING: + return Action::template call(std::forward(args)...); + case Datatype::VEC_CHAR: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_SHORT: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_INT: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_LONG: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_LONGLONG: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_UCHAR: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_SCHAR: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_USHORT: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_UINT: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_ULONG: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_ULONGLONG: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_FLOAT: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_DOUBLE: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_LONG_DOUBLE: + return Action::template call>( + std::forward(args)...); + case Datatype::VEC_CFLOAT: + return Action::template call>>( + std::forward(args)...); + case Datatype::VEC_CDOUBLE: + return Action::template call>>( + std::forward(args)...); + case Datatype::VEC_CLONG_DOUBLE: + return Action::template call>>( + std::forward(args)...); + case Datatype::VEC_STRING: + return Action::template call>( + std::forward(args)...); + case Datatype::ARR_DBL_7: + return Action::template call>( + std::forward(args)...); + case Datatype::BOOL: + return Action::template call(std::forward(args)...); + case Datatype::UNDEFINED: + return detail:: + CallUndefinedDatatype<0, ReturnType, Action, Args &&...>::call( + std::forward(args)...); + default: + throw std::runtime_error( + "Internal error: Encountered unknown datatype (switchType) ->" + + std::to_string(static_cast(dt))); + } +} + +/** + * Generalizes switching over an openPMD datatype. + * + * Will call the function template found at Action::call< T >(), instantiating T + * with the C++ internal datatype corresponding to the openPMD datatype. + * Ignores vector and array types. + * + * @tparam ReturnType The function template's return type. + * @tparam Action The struct containing the function template. + * @tparam Args The function template's argument types. + * @param dt The openPMD datatype. + * @param args The function template's arguments. + * @return Passes on the result of invoking the function template with the given + * arguments and with the template parameter specified by dt. + */ +template +constexpr auto switchNonVectorType(Datatype dt, Args &&...args) + -> decltype(Action::template call(std::forward(args)...)) +{ + using ReturnType = + decltype(Action::template call(std::forward(args)...)); + switch (dt) + { + case Datatype::CHAR: + return Action::template call(std::forward(args)...); + case Datatype::UCHAR: + return Action::template call( + std::forward(args)...); + case Datatype::SCHAR: + return Action::template call(std::forward(args)...); + case Datatype::SHORT: + return Action::template call(std::forward(args)...); + case Datatype::INT: + return Action::template call(std::forward(args)...); + case Datatype::LONG: + return Action::template call(std::forward(args)...); + case Datatype::LONGLONG: + return Action::template call(std::forward(args)...); + case Datatype::USHORT: + return Action::template call( + std::forward(args)...); + case Datatype::UINT: + return Action::template call(std::forward(args)...); + case Datatype::ULONG: + return Action::template call( + std::forward(args)...); + case Datatype::ULONGLONG: + return Action::template call( + std::forward(args)...); + case Datatype::FLOAT: + return Action::template call(std::forward(args)...); + case Datatype::DOUBLE: + return Action::template call(std::forward(args)...); + case Datatype::LONG_DOUBLE: + return Action::template call(std::forward(args)...); + case Datatype::CFLOAT: + return Action::template call>( + std::forward(args)...); + case Datatype::CDOUBLE: + return Action::template call>( + std::forward(args)...); + case Datatype::CLONG_DOUBLE: + return Action::template call>( + std::forward(args)...); + case Datatype::STRING: + return Action::template call(std::forward(args)...); + case Datatype::BOOL: + return Action::template call(std::forward(args)...); + case Datatype::UNDEFINED: + return detail:: + CallUndefinedDatatype<0, ReturnType, Action, Args &&...>::call( + std::forward(args)...); + default: + throw std::runtime_error( + "Internal error: Encountered unknown datatype (switchType) ->" + + std::to_string(static_cast(dt))); + } +} + +namespace detail +{ + template + struct is_char + { + static constexpr bool value = false; + }; + template <> + struct is_char + { + static constexpr bool value = true; + }; + template <> + struct is_char + { + static constexpr bool value = true; + }; + template <> + struct is_char + { + static constexpr bool value = true; + }; + template + constexpr bool is_char_v = is_char::value; + + template + inline bool isSameChar() + { + return + // both must be char types + is_char_v && is_char_v && + // both must have equivalent sign + std::is_signed_v == std::is_signed_v && + // both must have equivalent size + sizeof(T_Char1) == sizeof(T_Char2); + } + + template + struct IsSameChar + { + template + static bool call() + { + return isSameChar(); + } + + static constexpr char const *errorMsg = "IsSameChar"; + }; + +} // namespace detail + +template +constexpr inline bool isSameChar(Datatype d) +{ + return switchType>(d); +} +} // namespace openPMD diff --git a/include/openPMD/DatatypeHelpers.hpp b/include/openPMD/DatatypeHelpers.hpp index 07f3c7fc37..b2e9171754 100644 --- a/include/openPMD/DatatypeHelpers.hpp +++ b/include/openPMD/DatatypeHelpers.hpp @@ -1,4 +1,4 @@ -/* Copyright 2017-2021 Fabian Koller, Franz Poeschel, Axel Huebl +/* Copyright 2023 Franz Poeschel * * This file is part of openPMD-api. * @@ -18,282 +18,11 @@ * and the GNU Lesser General Public License along with openPMD-api. * If not, see . */ -#pragma once - -#include "openPMD/Datatype.hpp" - -#include -#include -#include // std::forward - -namespace openPMD -{ -namespace detail -{ - // std::void_t is C++17 - template - using void_t = void; - - /* - * Check whether class T has a member "errorMsg" convertible - * to type std::string. - * Used to give helpful compile-time error messages with static_assert - * down in CallUndefinedDatatype. - */ - template - struct HasErrorMessageMember - { - static constexpr bool value = false; - }; - - template - struct HasErrorMessageMember > - { - static constexpr bool value = true; - }; - /** - * Purpose of this struct is to detect at compile time whether - * Action::template operator()\<0\>() exists. If yes, call - * Action::template operator()\() with the passed arguments. - * If not, throw an error. - * - * @tparam n As in switchType(). - * @tparam ReturnType As in switchType(). - * @tparam Action As in switchType(). - * @tparam Args As in switchType(). - */ - template - struct CallUndefinedDatatype - { - static ReturnType call(Args &&...args) - { - if constexpr (HasErrorMessageMember::value) - { - throw std::runtime_error( - "[" + std::string(Action::errorMsg) + - "] Unknown Datatype."); - } - else - { - return Action::template call(std::forward(args)...); - } - throw std::runtime_error("Unreachable!"); - } - }; -} // namespace detail +#pragma once -/** - * Generalizes switching over an openPMD datatype. - * - * Will call the function template found at Action::call< T >(), instantiating T - * with the C++ internal datatype corresponding to the openPMD datatype. - * - * @tparam ReturnType The function template's return type. - * @tparam Action The struct containing the function template. - * @tparam Args The function template's argument types. - * @param dt The openPMD datatype. - * @param args The function template's arguments. - * @return Passes on the result of invoking the function template with the given - * arguments and with the template parameter specified by dt. +/* + * Legacy header, its functions are now included directly in Datatype.hpp. */ -template -auto switchType(Datatype dt, Args &&...args) - -> decltype(Action::template call(std::forward(args)...)) -{ - using ReturnType = - decltype(Action::template call(std::forward(args)...)); - switch (dt) - { - case Datatype::CHAR: - return Action::template call(std::forward(args)...); - case Datatype::UCHAR: - return Action::template call( - std::forward(args)...); - case Datatype::SCHAR: - return Action::template call(std::forward(args)...); - case Datatype::SHORT: - return Action::template call(std::forward(args)...); - case Datatype::INT: - return Action::template call(std::forward(args)...); - case Datatype::LONG: - return Action::template call(std::forward(args)...); - case Datatype::LONGLONG: - return Action::template call(std::forward(args)...); - case Datatype::USHORT: - return Action::template call( - std::forward(args)...); - case Datatype::UINT: - return Action::template call(std::forward(args)...); - case Datatype::ULONG: - return Action::template call( - std::forward(args)...); - case Datatype::ULONGLONG: - return Action::template call( - std::forward(args)...); - case Datatype::FLOAT: - return Action::template call(std::forward(args)...); - case Datatype::DOUBLE: - return Action::template call(std::forward(args)...); - case Datatype::LONG_DOUBLE: - return Action::template call(std::forward(args)...); - case Datatype::CFLOAT: - return Action::template call >( - std::forward(args)...); - case Datatype::CDOUBLE: - return Action::template call >( - std::forward(args)...); - case Datatype::CLONG_DOUBLE: - return Action::template call >( - std::forward(args)...); - case Datatype::STRING: - return Action::template call(std::forward(args)...); - case Datatype::VEC_CHAR: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_SHORT: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_INT: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_LONG: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_LONGLONG: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_UCHAR: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_SCHAR: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_USHORT: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_UINT: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_ULONG: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_ULONGLONG: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_FLOAT: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_DOUBLE: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_LONG_DOUBLE: - return Action::template call >( - std::forward(args)...); - case Datatype::VEC_CFLOAT: - return Action::template call > >( - std::forward(args)...); - case Datatype::VEC_CDOUBLE: - return Action::template call > >( - std::forward(args)...); - case Datatype::VEC_CLONG_DOUBLE: - return Action::template call > >( - std::forward(args)...); - case Datatype::VEC_STRING: - return Action::template call >( - std::forward(args)...); - case Datatype::ARR_DBL_7: - return Action::template call >( - std::forward(args)...); - case Datatype::BOOL: - return Action::template call(std::forward(args)...); - case Datatype::UNDEFINED: - return detail:: - CallUndefinedDatatype<0, ReturnType, Action, Args &&...>::call( - std::forward(args)...); - default: - throw std::runtime_error( - "Internal error: Encountered unknown datatype (switchType) ->" + - std::to_string(static_cast(dt))); - } -} -/** - * Generalizes switching over an openPMD datatype. - * - * Will call the function template found at Action::call< T >(), instantiating T - * with the C++ internal datatype corresponding to the openPMD datatype. - * Ignores vector and array types. - * - * @tparam ReturnType The function template's return type. - * @tparam Action The struct containing the function template. - * @tparam Args The function template's argument types. - * @param dt The openPMD datatype. - * @param args The function template's arguments. - * @return Passes on the result of invoking the function template with the given - * arguments and with the template parameter specified by dt. - */ -template -auto switchNonVectorType(Datatype dt, Args &&...args) - -> decltype(Action::template call(std::forward(args)...)) -{ - using ReturnType = - decltype(Action::template call(std::forward(args)...)); - switch (dt) - { - case Datatype::CHAR: - return Action::template call(std::forward(args)...); - case Datatype::UCHAR: - return Action::template call( - std::forward(args)...); - case Datatype::SCHAR: - return Action::template call(std::forward(args)...); - case Datatype::SHORT: - return Action::template call(std::forward(args)...); - case Datatype::INT: - return Action::template call(std::forward(args)...); - case Datatype::LONG: - return Action::template call(std::forward(args)...); - case Datatype::LONGLONG: - return Action::template call(std::forward(args)...); - case Datatype::USHORT: - return Action::template call( - std::forward(args)...); - case Datatype::UINT: - return Action::template call(std::forward(args)...); - case Datatype::ULONG: - return Action::template call( - std::forward(args)...); - case Datatype::ULONGLONG: - return Action::template call( - std::forward(args)...); - case Datatype::FLOAT: - return Action::template call(std::forward(args)...); - case Datatype::DOUBLE: - return Action::template call(std::forward(args)...); - case Datatype::LONG_DOUBLE: - return Action::template call(std::forward(args)...); - case Datatype::CFLOAT: - return Action::template call >( - std::forward(args)...); - case Datatype::CDOUBLE: - return Action::template call >( - std::forward(args)...); - case Datatype::CLONG_DOUBLE: - return Action::template call >( - std::forward(args)...); - case Datatype::STRING: - return Action::template call(std::forward(args)...); - case Datatype::BOOL: - return Action::template call(std::forward(args)...); - case Datatype::UNDEFINED: - return detail:: - CallUndefinedDatatype<0, ReturnType, Action, Args &&...>::call( - std::forward(args)...); - default: - throw std::runtime_error( - "Internal error: Encountered unknown datatype (switchType) ->" + - std::to_string(static_cast(dt))); - } -} -} // namespace openPMD +#include "openPMD/Datatype.hpp" diff --git a/include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp b/include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp index 6804d60ed7..40656a8606 100644 --- a/include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp +++ b/include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp @@ -440,6 +440,12 @@ class ADIOS2IOHandlerImpl Extent const &extent, adios2::IO &IO, std::string const &var); + + struct + { + bool noGroupBased = false; + bool blosc2bp5 = false; + } printedWarningsAlready; }; // ADIOS2IOHandlerImpl /* @@ -920,7 +926,7 @@ namespace detail Offset offset; Extent extent; UniquePtrWithLambda data; - Datatype dtype; + Datatype dtype = Datatype::UNDEFINED; void run(BufferedActions &); }; diff --git a/include/openPMD/IO/AbstractIOHandler.hpp b/include/openPMD/IO/AbstractIOHandler.hpp index f6316dbe97..1106f78f16 100644 --- a/include/openPMD/IO/AbstractIOHandler.hpp +++ b/include/openPMD/IO/AbstractIOHandler.hpp @@ -253,6 +253,14 @@ class AbstractIOHandler Access const m_frontendAccess; internal::SeriesStatus m_seriesStatus = internal::SeriesStatus::Default; std::queue m_work; + /** + * This is to avoid that the destructor tries flushing again if an error + * happened. Otherwise, this would lead to confusing error messages. + * Initialized as false, set to true after successful construction. + * If flushing results in an error, set this back to false. + * The destructor will only attempt flushing again if this is true. + */ + bool m_lastFlushSuccessful = false; }; // AbstractIOHandler } // namespace openPMD diff --git a/include/openPMD/IO/AbstractIOHandlerHelper.hpp b/include/openPMD/IO/AbstractIOHandlerHelper.hpp index 95d2d5537e..a5ce7a39be 100644 --- a/include/openPMD/IO/AbstractIOHandlerHelper.hpp +++ b/include/openPMD/IO/AbstractIOHandlerHelper.hpp @@ -40,6 +40,7 @@ namespace openPMD * @param comm MPI communicator used for IO. * @param options JSON-formatted option string, to be interpreted by * the backend. + * @param pathAsItWasSpecifiedInTheConstructor For error messages. * @tparam JSON Substitute for nlohmann::json. Templated to avoid including nlohmann::json in a .hpp file. * @return Smart pointer to created IOHandler. @@ -51,7 +52,8 @@ std::unique_ptr createIOHandler( Format format, std::string originalExtension, MPI_Comm comm, - JSON options); + JSON options, + std::string const &pathAsItWasSpecifiedInTheConstructor); #endif /** Construct an appropriate specific IOHandler for the desired IO mode. @@ -65,6 +67,7 @@ std::unique_ptr createIOHandler( * specified by the user. * @param options JSON-formatted option string, to be interpreted by * the backend. + * @param pathAsItWasSpecifiedInTheConstructor For error messages. * @tparam JSON Substitute for nlohmann::json. Templated to avoid including nlohmann::json in a .hpp file. * @return Smart pointer to created IOHandler. @@ -75,7 +78,8 @@ std::unique_ptr createIOHandler( Access access, Format format, std::string originalExtension, - JSON options = JSON()); + JSON options, + std::string const &pathAsItWasSpecifiedInTheConstructor); // version without configuration to use in AuxiliaryTest std::unique_ptr createIOHandler( diff --git a/include/openPMD/IO/AbstractIOHandlerImpl.hpp b/include/openPMD/IO/AbstractIOHandlerImpl.hpp index f382a73258..8d13f3feb8 100644 --- a/include/openPMD/IO/AbstractIOHandlerImpl.hpp +++ b/include/openPMD/IO/AbstractIOHandlerImpl.hpp @@ -20,12 +20,12 @@ */ #pragma once +#include "openPMD/Error.hpp" #include "openPMD/IO/AbstractIOHandler.hpp" #include "openPMD/IO/IOTask.hpp" #include "openPMD/auxiliary/DerefDynamicCast.hpp" #include -#include namespace openPMD { @@ -35,198 +35,11 @@ class Writable; class AbstractIOHandlerImpl { public: - AbstractIOHandlerImpl(AbstractIOHandler *handler) : m_handler{handler} - {} + AbstractIOHandlerImpl(AbstractIOHandler *handler); virtual ~AbstractIOHandlerImpl() = default; - std::future flush() - { - using namespace auxiliary; - - while (!(*m_handler).m_work.empty()) - { - IOTask &i = (*m_handler).m_work.front(); - try - { - switch (i.operation) - { - using O = Operation; - case O::CREATE_FILE: - createFile( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::CHECK_FILE: - checkFile( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::CREATE_PATH: - createPath( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::CREATE_DATASET: - createDataset( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::EXTEND_DATASET: - extendDataset( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::OPEN_FILE: - openFile( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::CLOSE_FILE: - closeFile( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::OPEN_PATH: - openPath( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::CLOSE_PATH: - closePath( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::OPEN_DATASET: - openDataset( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::DELETE_FILE: - deleteFile( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::DELETE_PATH: - deletePath( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::DELETE_DATASET: - deleteDataset( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::DELETE_ATT: - deleteAttribute( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::WRITE_DATASET: - writeDataset( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::WRITE_ATT: - writeAttribute( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::READ_DATASET: - readDataset( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::GET_BUFFER_VIEW: - getBufferView( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::READ_ATT: - readAttribute( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::LIST_PATHS: - listPaths( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::LIST_DATASETS: - listDatasets( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::LIST_ATTS: - listAttributes( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::ADVANCE: - advance( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::AVAILABLE_CHUNKS: - availableChunks( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::KEEP_SYNCHRONOUS: - keepSynchronous( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - case O::DEREGISTER: - deregister( - i.writable, - deref_dynamic_cast >( - i.parameter.get())); - break; - } - } - catch (...) - { - std::cerr << "[AbstractIOHandlerImpl] IO Task " - << internal::operationAsString(i.operation) - << " failed with exception. Clearing IO queue and " - "passing on the exception." - << std::endl; - while (!m_handler->m_work.empty()) - { - m_handler->m_work.pop(); - } - throw; - } - (*m_handler).m_work.pop(); - } - return std::future(); - } + std::future flush(); /** * Close the file corresponding with the writable and release file handles. @@ -263,6 +76,14 @@ class AbstractIOHandlerImpl */ virtual void advance(Writable *, Parameter ¶meters) { + if (parameters.isThisStepMandatory) + { + throw error::OperationUnsupportedInBackend( + m_handler->backendName(), + "Variable-based encoding requires backend support for IO steps " + "in order to store more than one iteration (only supported in " + "ADIOS2 backend)."); + } *parameters.status = AdvanceStatus::RANDOMACCESS; } @@ -583,5 +404,10 @@ class AbstractIOHandlerImpl deregister(Writable *, Parameter const ¶m) = 0; AbstractIOHandler *m_handler; + bool m_verboseIOTasks = false; + + // Args will be forwarded to std::cerr if m_verboseIOTasks is true + template + void writeToStderr(Args &&...) const; }; // AbstractIOHandlerImpl } // namespace openPMD diff --git a/include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp b/include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp index 77f9cadb60..300b3f19f3 100644 --- a/include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp +++ b/include/openPMD/IO/HDF5/HDF5IOHandlerImpl.hpp @@ -94,6 +94,16 @@ class HDF5IOHandlerImpl : public AbstractIOHandlerImpl hid_t m_H5T_CFLOAT; hid_t m_H5T_CDOUBLE; hid_t m_H5T_CLONG_DOUBLE; + /* See https://github.com/openPMD/openPMD-api/issues/1363 + * long double values written on AMD64 architectures cannot be read on + * ARM64/PPC64 architectures without doing some special tricks. + * We generally don't implement custom conversions, but instead pass-through + * the cross-platform support of HDF5. + * But this case is common and important enough to warrant a custom + * workaround. + */ + hid_t m_H5T_LONG_DOUBLE_80_LE; + hid_t m_H5T_CLONG_DOUBLE_80_LE; protected: #if openPMD_HAVE_MPI diff --git a/include/openPMD/IO/IOTask.hpp b/include/openPMD/IO/IOTask.hpp index 0ee6cd66d7..1ad4acd5a6 100644 --- a/include/openPMD/IO/IOTask.hpp +++ b/include/openPMD/IO/IOTask.hpp @@ -625,6 +625,7 @@ struct OPENPMDAPI_EXPORT Parameter //! input parameter AdvanceMode mode; + bool isThisStepMandatory = false; //! output parameter std::shared_ptr status = std::make_shared(AdvanceStatus::OK); diff --git a/include/openPMD/ReadIterations.hpp b/include/openPMD/ReadIterations.hpp index c6a1e4fc36..e9256c71ff 100644 --- a/include/openPMD/ReadIterations.hpp +++ b/include/openPMD/ReadIterations.hpp @@ -77,7 +77,20 @@ class SeriesIterator std::set ignoreIterations; }; - std::shared_ptr m_data; + /* + * The shared data is never empty, emptiness is indicated by std::optional + */ + std::shared_ptr> m_data = + std::make_shared>(std::nullopt); + + SharedData &get() + { + return m_data->value(); + } + SharedData const &get() const + { + return m_data->value(); + } public: //! construct the end() iterator @@ -99,7 +112,7 @@ class SeriesIterator private: inline bool setCurrentIteration() { - auto &data = *m_data; + auto &data = get(); if (data.iterationsInCurrentStep.empty()) { std::cerr << "[ReadIterations] Encountered a step without " @@ -114,7 +127,7 @@ class SeriesIterator inline std::optional peekCurrentIteration() { - auto &data = *m_data; + auto &data = get(); if (data.iterationsInCurrentStep.empty()) { return std::nullopt; @@ -144,6 +157,8 @@ class SeriesIterator void deactivateDeadIteration(iteration_index_t); void initSeriesInLinearReadMode(); + + void close(); }; /** @@ -171,7 +186,6 @@ class ReadIterations using iterator_t = SeriesIterator; Series m_series; - std::optional alreadyOpened; std::optional m_parsePreference; ReadIterations( diff --git a/include/openPMD/RecordComponent.tpp b/include/openPMD/RecordComponent.tpp index 4256fc6fa8..d29635989e 100644 --- a/include/openPMD/RecordComponent.tpp +++ b/include/openPMD/RecordComponent.tpp @@ -100,7 +100,8 @@ RecordComponent::loadChunk(std::shared_ptr data, Offset o, Extent e) if (dtype != getDatatype()) if (!isSameInteger(getDatatype()) && !isSameFloatingPoint(getDatatype()) && - !isSameComplexFloatingPoint(getDatatype())) + !isSameComplexFloatingPoint(getDatatype()) && + !isSameChar(getDatatype())) { std::string const data_type_str = datatypeToString(getDatatype()); std::string const requ_type_str = diff --git a/include/openPMD/Series.hpp b/include/openPMD/Series.hpp index 7b85986992..1d99b54a84 100644 --- a/include/openPMD/Series.hpp +++ b/include/openPMD/Series.hpp @@ -53,6 +53,7 @@ namespace openPMD { class ReadIterations; +class SeriesIterator; class Series; class Series; @@ -88,6 +89,23 @@ namespace internal * the same instance. */ std::optional m_writeIterations; + + /** + * Series::readIterations() returns an iterator type that modifies the + * state of the Series (by proceeding through IO steps). + * Hence, we need to make sure that there is only one of them, otherwise + * they will both make modifications to the Series that the other + * iterator is not aware of. + * + * Plan: At some point, we should add a second iterator type that does + * not change the state. Series::readIterations() should then return + * either this or that iterator depending on read mode (linear or + * random-access) and backend capabilities. + * + * Due to include order, this member needs to be a pointer instead of + * an optional. + */ + std::unique_ptr m_sharedStatefulIterator; /** * For writing: Remember which iterations have been written in the * currently active output step. Use this later when writing the @@ -151,14 +169,15 @@ namespace internal * True if a user opts into lazy parsing. */ bool m_parseLazily = false; + /** - * This is to avoid that the destructor tries flushing again if an error - * happened. Otherwise, this would lead to confusing error messages. - * Initialized as false, set to true after successful construction. - * If flushing results in an error, set this back to false. - * The destructor will only attempt flushing again if this is true. + * In variable-based encoding, all backends except ADIOS2 can only write + * one single iteration. So, we remember if we already had a step, + * and if yes, Parameter::isThisStepMandatory is + * set as true in variable-based encoding. + * The backend will then throw if it has no support for steps. */ - bool m_lastFlushSuccessful = false; + bool m_wroteAtLeastOneIOStep = false; /** * Remember the preference that the backend specified for parsing. @@ -188,6 +207,7 @@ class Series : public Attributable friend class Attributable; friend class Iteration; friend class Writable; + friend class ReadIterations; friend class SeriesIterator; friend class internal::SeriesData; friend class WriteIterations; @@ -491,20 +511,46 @@ class Series : public Attributable * Creates and returns an instance of the ReadIterations class which can * be used for iterating over the openPMD iterations in a C++11-style for * loop. + * `Series::readIterations()` is an intentionally restricted API that + * ensures a workflow which also works in streaming setups, e.g. an + * iteration cannot be opened again once it has been closed. + * For a less restrictive API in non-streaming situations, + * `Series::iterations` can be accessed directly. * Look for the ReadIterations class for further documentation. * * @return ReadIterations */ ReadIterations readIterations(); + /** + * @brief Parse the Series. + * + * Only necessary in linear read mode. + * In linear read mode, the Series constructor does not do any IO accesses. + * This call effectively triggers the side effects of + * Series::readIterations(), for use cases where data needs to be accessed + * before iterating through the iterations. + * + * The reason for introducing this restricted alias to + * Series::readIterations() is that the name "readIterations" is misleading + * for that use case: When using IO steps, this call only ensures that the + * first step is parsed. + */ + void parseBase(); + /** * @brief Entry point to the writing end of the streaming API. * - * Creates and returns an instance of the WriteIterations class which is a - * restricted container of iterations which takes care of - * streaming semantics. + * Creates and returns an instance of the WriteIterations class which is an + * intentionally restricted container of iterations that takes care of + * streaming semantics, e.g. ensuring that an iteration cannot be reopened + * once closed. + * For a less restrictive API in non-streaming situations, + * `Series::iterations` can be accessed directly. * The created object is stored as member of the Series object, hence this * method may be called as many times as a user wishes. + * There is only one shared iterator state per Series, even when calling + * this method twice. * Look for the WriteIterations class for further documentation. * * @return WriteIterations @@ -686,7 +732,8 @@ OPENPMD_private /** * @brief Called at the end of an IO step to store the iterations defined - * in the IO step to the snapshot attribute. + * in the IO step to the snapshot attribute and to store that at + * least one step was written. * * @param doFlush If true, flush the IO handler. */ diff --git a/include/openPMD/WriteIterations.hpp b/include/openPMD/WriteIterations.hpp index 3099af7025..63dc6e1c86 100644 --- a/include/openPMD/WriteIterations.hpp +++ b/include/openPMD/WriteIterations.hpp @@ -50,6 +50,19 @@ namespace internal class SeriesData; } +/** + * @brief Writing side of the streaming API. + * + * Create instance via Series::writeIterations(). + * Restricted Container of Iterations, designed to allow reading any kind + * of Series, streaming and non-streaming alike. + * Calling Iteration::close() manually before opening the next iteration is + * encouraged and will implicitly flush all deferred IO actions. + * Otherwise, Iteration::close() will be implicitly called upon + * opening the next iteration or upon destruction. + * Since this is designed for streaming mode, reopening an iteration is + * not possible once it has been closed. + */ class WriteIterations { friend class Series; diff --git a/include/openPMD/backend/Attributable.hpp b/include/openPMD/backend/Attributable.hpp index 90da6655bf..db7aec37de 100644 --- a/include/openPMD/backend/Attributable.hpp +++ b/include/openPMD/backend/Attributable.hpp @@ -223,6 +223,7 @@ class Attributable * Notice that RecordComponent::SCALAR is included in this list, too. */ std::vector group; + Access access; /** Reconstructs a path that can be passed to a Series constructor */ std::string filePath() const; diff --git a/include/openPMD/version.hpp b/include/openPMD/version.hpp index 29a4f7d5ba..a0b345a215 100644 --- a/include/openPMD/version.hpp +++ b/include/openPMD/version.hpp @@ -29,7 +29,7 @@ */ #define OPENPMDAPI_VERSION_MAJOR 0 #define OPENPMDAPI_VERSION_MINOR 15 -#define OPENPMDAPI_VERSION_PATCH 0 +#define OPENPMDAPI_VERSION_PATCH 2 #define OPENPMDAPI_VERSION_LABEL "" /** @} */ diff --git a/setup.py b/setup.py index 09cad27c19..2eacf715d0 100644 --- a/setup.py +++ b/setup.py @@ -170,7 +170,7 @@ def build_extension(self, ext): setup( name='openPMD-api', # note PEP-440 syntax: x.y.zaN but x.y.z.devN - version='0.15.1', + version='0.15.2', author='Axel Huebl, Franz Poeschel, Fabian Koller, Junmin Gu', author_email='axelhuebl@lbl.gov, f.poeschel@hzdr.de', maintainer='Axel Huebl', @@ -192,7 +192,7 @@ def build_extension(self, ext): cmdclass=dict(build_ext=CMakeBuild), # scripts=['openpmd-ls'], zip_safe=False, - python_requires='>=3.7', + python_requires='>=3.8', # tests_require=['pytest'], install_requires=install_requires, # see: src/bindings/python/cli @@ -221,7 +221,6 @@ def build_extension(self, ext): 'Topic :: Database :: Front-Ends', 'Programming Language :: C++', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Programming Language :: Python :: 3.10', diff --git a/share/openPMD/download_samples.ps1 b/share/openPMD/download_samples.ps1 index 5fc8987c4d..9880aa891f 100755 --- a/share/openPMD/download_samples.ps1 +++ b/share/openPMD/download_samples.ps1 @@ -15,18 +15,23 @@ New-item -ItemType directory -Name samples\git-sample\3d-bp4\ Invoke-WebRequest https://github.com/openPMD/openPMD-example-datasets/raw/f3b73e43511db96217a153dc3ab3cb2e8f81f7db/example-3d.tar.gz -OutFile example-3d.tar.gz Invoke-WebRequest https://github.com/openPMD/openPMD-example-datasets/raw/f3b73e43511db96217a153dc3ab3cb2e8f81f7db/example-thetaMode.tar.gz -OutFile example-thetaMode.tar.gz Invoke-WebRequest https://github.com/openPMD/openPMD-example-datasets/raw/f3b73e43511db96217a153dc3ab3cb2e8f81f7db/example-3d-bp4.tar.gz -OutFile example-3d-bp4.tar.gz +Invoke-WebRequest https://github.com/openPMD/openPMD-example-datasets/raw/566b356030df38f56049484941baacafef331163/legacy_datasets.tar.gz -OutFile legacy_datasets.tar.gz 7z.exe x -r example-3d.tar.gz 7z.exe x -r example-3d.tar 7z.exe x -r example-thetaMode.tar.gz 7z.exe x -r example-thetaMode.tar 7z.exe x -r example-3d-bp4.tar.gz 7z.exe x -r example-3d-bp4.tar +7z.exe x -r legacy_datasets.tar.gz +7z.exe x -r legacy_datasets.tar Move-Item -Path example-3d\hdf5\* samples\git-sample\ Move-Item -Path example-thetaMode\hdf5\* samples\git-sample\thetaMode\ Move-Item -Path example-3d-bp4\* samples\git-sample\3d-bp4\ +Move-Item -Path legacy_datasets\* samples\git-sample\legacy\ Remove-Item -Recurse -Force example-3d* Remove-Item -Recurse -Force example-thetaMode* Remove-Item -Recurse -Force example-3d-bp4* +Remove-Item -Recurse -Force legacy_datasets* # Ref.: https://github.com/yt-project/yt/pull/1645 New-item -ItemType directory -Name samples\issue-sample\ diff --git a/share/openPMD/download_samples.sh b/share/openPMD/download_samples.sh index aef491ca1f..8691ce47a5 100755 --- a/share/openPMD/download_samples.sh +++ b/share/openPMD/download_samples.sh @@ -15,14 +15,17 @@ mkdir -p samples/git-sample/3d-bp4 curl -sOL https://github.com/openPMD/openPMD-example-datasets/raw/f3b73e43511db96217a153dc3ab3cb2e8f81f7db/example-3d.tar.gz curl -sOL https://github.com/openPMD/openPMD-example-datasets/raw/f3b73e43511db96217a153dc3ab3cb2e8f81f7db/example-thetaMode.tar.gz curl -sOL https://github.com/openPMD/openPMD-example-datasets/raw/f3b73e43511db96217a153dc3ab3cb2e8f81f7db/example-3d-bp4.tar.gz +curl -sOL https://github.com/openPMD/openPMD-example-datasets/raw/566b356030df38f56049484941baacafef331163/legacy_datasets.tar.gz tar -xzf example-3d.tar.gz tar -xzf example-thetaMode.tar.gz tar -xzf example-3d-bp4.tar.gz +tar -xzf legacy_datasets.tar.gz mv example-3d/hdf5/* samples/git-sample/ mv example-thetaMode/hdf5/* samples/git-sample/thetaMode/ mv example-3d-bp4/* samples/git-sample/3d-bp4 +mv legacy_datasets/* samples/git-sample/legacy chmod 777 samples/ -rm -rf example-3d.* example-3d example-thetaMode.* example-thetaMode example-3d-bp4 example-3d-bp4.* +rm -rf example-3d.* example-3d example-thetaMode.* example-thetaMode example-3d-bp4 example-3d-bp4.* legacy_datasets legacy_datasets.* # Ref.: https://github.com/yt-project/yt/pull/1645 mkdir -p samples/issue-sample/ diff --git a/src/Datatype.cpp b/src/Datatype.cpp index 71565b3d52..f0f26f7ae9 100644 --- a/src/Datatype.cpp +++ b/src/Datatype.cpp @@ -224,46 +224,49 @@ std::string datatypeToString(openPMD::Datatype dt) return buf.str(); } -std::vector openPMD_Datatypes{ - Datatype::CHAR, - Datatype::UCHAR, - Datatype::SCHAR, - Datatype::SHORT, - Datatype::INT, - Datatype::LONG, - Datatype::LONGLONG, - Datatype::USHORT, - Datatype::UINT, - Datatype::ULONG, - Datatype::ULONGLONG, - Datatype::FLOAT, - Datatype::DOUBLE, - Datatype::LONG_DOUBLE, - Datatype::CFLOAT, - Datatype::CDOUBLE, - Datatype::CLONG_DOUBLE, - Datatype::STRING, - Datatype::VEC_CHAR, - Datatype::VEC_SHORT, - Datatype::VEC_INT, - Datatype::VEC_LONG, - Datatype::VEC_LONGLONG, - Datatype::VEC_UCHAR, - Datatype::VEC_USHORT, - Datatype::VEC_UINT, - Datatype::VEC_ULONG, - Datatype::VEC_ULONGLONG, - Datatype::VEC_FLOAT, - Datatype::VEC_DOUBLE, - Datatype::VEC_LONG_DOUBLE, - Datatype::VEC_CFLOAT, - Datatype::VEC_CDOUBLE, - Datatype::VEC_CLONG_DOUBLE, - Datatype::VEC_SCHAR, - Datatype::VEC_STRING, - Datatype::ARR_DBL_7, - Datatype::BOOL, - Datatype::UNDEFINED}; +std::vector openPMD_Datatypes() +{ + return { + Datatype::CHAR, + Datatype::UCHAR, + Datatype::SCHAR, + Datatype::SHORT, + Datatype::INT, + Datatype::LONG, + Datatype::LONGLONG, + Datatype::USHORT, + Datatype::UINT, + Datatype::ULONG, + Datatype::ULONGLONG, + Datatype::FLOAT, + Datatype::DOUBLE, + Datatype::LONG_DOUBLE, + Datatype::CFLOAT, + Datatype::CDOUBLE, + Datatype::CLONG_DOUBLE, + Datatype::STRING, + Datatype::VEC_CHAR, + Datatype::VEC_SHORT, + Datatype::VEC_INT, + Datatype::VEC_LONG, + Datatype::VEC_LONGLONG, + Datatype::VEC_UCHAR, + Datatype::VEC_USHORT, + Datatype::VEC_UINT, + Datatype::VEC_ULONG, + Datatype::VEC_ULONGLONG, + Datatype::VEC_FLOAT, + Datatype::VEC_DOUBLE, + Datatype::VEC_LONG_DOUBLE, + Datatype::VEC_CFLOAT, + Datatype::VEC_CDOUBLE, + Datatype::VEC_CLONG_DOUBLE, + Datatype::VEC_SCHAR, + Datatype::VEC_STRING, + Datatype::ARR_DBL_7, + Datatype::BOOL, + Datatype::UNDEFINED}; +} namespace { diff --git a/src/IO/ADIOS/ADIOS2IOHandler.cpp b/src/IO/ADIOS/ADIOS2IOHandler.cpp index 266063b722..51171f5907 100644 --- a/src/IO/ADIOS/ADIOS2IOHandler.cpp +++ b/src/IO/ADIOS/ADIOS2IOHandler.cpp @@ -645,6 +645,42 @@ void ADIOS2IOHandlerImpl::createDataset( parameters.extent.begin(), parameters.extent.end()); auto &fileData = getFileData(file, IfFileNotOpen::ThrowError); + +#define HAS_BP5_BLOSC2_BUG \ + (ADIOS2_VERSION_MAJOR * 100 + ADIOS2_VERSION_MINOR == 209 && \ + ADIOS2_VERSION_PATCH <= 1) +#if HAS_BP5_BLOSC2_BUG + std::string engineType = fileData.getEngine().Type(); + std::transform( + engineType.begin(), + engineType.end(), + engineType.begin(), + [](unsigned char c) { return std::tolower(c); }); + if (!printedWarningsAlready.blosc2bp5 && engineType == "bp5writer") + { + for (auto const &op : operators) + { + std::string operatorType = op.op.Type(); + std::transform( + operatorType.begin(), + operatorType.end(), + operatorType.begin(), + [](unsigned char c) { return std::tolower(c); }); + if (operatorType == "blosc") + { + std::cerr << &R"( +[Warning] Use BP5+Blosc with care in ADIOS2 v2.9.0 and v2.9.1. +Unreadable data might be created, to mitigate either deactivate Blosc or use BP4+Blosc. +For further details see +https://github.com/ornladios/ADIOS2/issues/3504. + )"[1] << std::endl; + printedWarningsAlready.blosc2bp5 = true; + } + } + } +#endif +#undef HAS_BP5_BLOSC2_BUG + switchAdios2VariableType( parameters.dtype, fileData.m_IO, varName, operators, shape); fileData.invalidateVariablesMap(); @@ -2560,7 +2596,9 @@ namespace detail // might have been closed previously if (engine) { - if (streamStatus == StreamStatus::DuringStep) + if (streamStatus == StreamStatus::DuringStep || + (streamStatus == StreamStatus::NoStream && + m_mode == adios2::Mode::Write)) { engine.EndStep(); } @@ -3097,6 +3135,11 @@ namespace detail // the streaming API was used. m_engine = std::make_optional( adios2::Engine(m_IO.Open(m_file, tempMode))); + if (streamStatus == StreamStatus::NoStream) + { + // Write everything into one big step + m_engine->BeginStep(); + } break; } #if HAS_ADIOS_2_8 diff --git a/src/IO/AbstractIOHandler.cpp b/src/IO/AbstractIOHandler.cpp index 25adae581e..440b663286 100644 --- a/src/IO/AbstractIOHandler.cpp +++ b/src/IO/AbstractIOHandler.cpp @@ -28,7 +28,18 @@ namespace openPMD std::future AbstractIOHandler::flush(internal::FlushParams const ¶ms) { internal::ParsedFlushParams parsedParams{params}; - auto future = this->flush(parsedParams); + auto future = [this, &parsedParams]() { + try + { + return this->flush(parsedParams); + } + catch (...) + { + m_lastFlushSuccessful = false; + throw; + } + }(); + m_lastFlushSuccessful = true; json::warnGlobalUnusedOptions(parsedParams.backendConfig); return future; } diff --git a/src/IO/AbstractIOHandlerHelper.cpp b/src/IO/AbstractIOHandlerHelper.cpp index 27efe722c1..d16e7f4ca0 100644 --- a/src/IO/AbstractIOHandlerHelper.cpp +++ b/src/IO/AbstractIOHandlerHelper.cpp @@ -89,7 +89,8 @@ std::unique_ptr createIOHandler( std::string originalExtension, MPI_Comm comm, - json::TracingJSON options) + json::TracingJSON options, + std::string const &pathAsItWasSpecifiedInTheConstructor) { (void)options; switch (format) @@ -154,9 +155,14 @@ std::unique_ptr createIOHandler( std::move(options), "ssc", std::move(originalExtension)); + case Format::JSON: + throw error::WrongAPIUsage( + "JSON backend not available in parallel openPMD."); default: - throw std::runtime_error( - "Unknown file format! Did you specify a file ending?"); + throw error::WrongAPIUsage( + "Unknown file format! Did you specify a file ending? Specified " + "file name was '" + + pathAsItWasSpecifiedInTheConstructor + "'."); } } #endif @@ -167,7 +173,8 @@ std::unique_ptr createIOHandler( Access access, Format format, std::string originalExtension, - json::TracingJSON options) + json::TracingJSON options, + std::string const &pathAsItWasSpecifiedInTheConstructor) { (void)options; switch (format) @@ -227,7 +234,9 @@ std::unique_ptr createIOHandler( "JSON", path, access); default: throw std::runtime_error( - "Unknown file format! Did you specify a file ending?"); + "Unknown file format! Did you specify a file ending? Specified " + "file name was '" + + pathAsItWasSpecifiedInTheConstructor + "'."); } } @@ -242,6 +251,7 @@ std::unique_ptr createIOHandler( access, format, std::move(originalExtension), - json::TracingJSON(json::ParsedConfig{})); + json::TracingJSON(json::ParsedConfig{}), + ""); } } // namespace openPMD diff --git a/src/IO/AbstractIOHandlerImpl.cpp b/src/IO/AbstractIOHandlerImpl.cpp index d762df6a1f..af827704a1 100644 --- a/src/IO/AbstractIOHandlerImpl.cpp +++ b/src/IO/AbstractIOHandlerImpl.cpp @@ -20,14 +20,365 @@ */ #include "openPMD/IO/AbstractIOHandlerImpl.hpp" + +#include "openPMD/auxiliary/Environment.hpp" #include "openPMD/backend/Writable.hpp" +#include + namespace openPMD { +AbstractIOHandlerImpl::AbstractIOHandlerImpl(AbstractIOHandler *handler) + : m_handler{handler} +{ + if (auxiliary::getEnvNum("OPENPMD_VERBOSE", 0) != 0) + { + m_verboseIOTasks = true; + } +} + void AbstractIOHandlerImpl::keepSynchronous( Writable *writable, Parameter param) { writable->abstractFilePosition = param.otherWritable->abstractFilePosition; writable->written = true; } + +template +void AbstractIOHandlerImpl::writeToStderr([[maybe_unused]] Args &&...args) const +{ + if (m_verboseIOTasks) + { + (std::cerr << ... << args) << std::endl; + } +} + +std::future AbstractIOHandlerImpl::flush() +{ + using namespace auxiliary; + + while (!(*m_handler).m_work.empty()) + { + IOTask &i = (*m_handler).m_work.front(); + try + { + switch (i.operation) + { + using O = Operation; + case O::CREATE_FILE: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] CREATE_FILE: ", + parameter.name); + createFile(i.writable, parameter); + break; + } + case O::CHECK_FILE: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] CHECK_FILE: ", + parameter.name); + checkFile(i.writable, parameter); + break; + } + case O::CREATE_PATH: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] CREATE_PATH: ", + parameter.path); + createPath(i.writable, parameter); + break; + } + case O::CREATE_DATASET: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] CREATE_DATASET: ", + parameter.name); + createDataset(i.writable, parameter); + break; + } + case O::EXTEND_DATASET: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] EXTEND_DATASET"); + extendDataset(i.writable, parameter); + break; + } + case O::OPEN_FILE: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] OPEN_FILE: ", + parameter.name); + openFile(i.writable, parameter); + break; + } + case O::CLOSE_FILE: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", i.writable->parent, "->", i.writable, "] CLOSE_FILE"); + closeFile(i.writable, parameter); + break; + } + case O::OPEN_PATH: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] OPEN_PATH: ", + parameter.path); + openPath(i.writable, parameter); + break; + } + case O::CLOSE_PATH: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", i.writable->parent, "->", i.writable, "] CLOSE_PATH"); + closePath(i.writable, parameter); + break; + } + case O::OPEN_DATASET: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] OPEN_DATASET: ", + parameter.name); + openDataset(i.writable, parameter); + break; + } + case O::DELETE_FILE: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", i.writable->parent, "->", i.writable, "] DELETE_FILE"); + deleteFile(i.writable, parameter); + break; + } + case O::DELETE_PATH: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", i.writable->parent, "->", i.writable, "] DELETE_PATH"); + deletePath(i.writable, parameter); + break; + } + case O::DELETE_DATASET: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] DELETE_DATASET"); + deleteDataset(i.writable, parameter); + break; + } + case O::DELETE_ATT: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", i.writable->parent, "->", i.writable, "] DELETE_ATT"); + deleteAttribute(i.writable, parameter); + break; + } + case O::WRITE_DATASET: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] WRITE_DATASET"); + writeDataset(i.writable, parameter); + break; + } + case O::WRITE_ATT: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] WRITE_ATT: (", + parameter.dtype, + ") ", + parameter.name); + writeAttribute(i.writable, parameter); + break; + } + case O::READ_DATASET: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] READ_DATASET"); + readDataset(i.writable, parameter); + break; + } + case O::GET_BUFFER_VIEW: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] GET_BUFFER_VIEW"); + getBufferView(i.writable, parameter); + break; + } + case O::READ_ATT: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] READ_ATT: ", + parameter.name); + readAttribute(i.writable, parameter); + break; + } + case O::LIST_PATHS: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", i.writable->parent, "->", i.writable, "] LIST_PATHS"); + listPaths(i.writable, parameter); + break; + } + case O::LIST_DATASETS: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] LIST_DATASETS"); + listDatasets(i.writable, parameter); + break; + } + case O::LIST_ATTS: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", i.writable->parent, "->", i.writable, "] LIST_ATTS"); + listAttributes(i.writable, parameter); + break; + } + case O::ADVANCE: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", i.writable->parent, "->", i.writable, "] ADVANCE"); + advance(i.writable, parameter); + break; + } + case O::AVAILABLE_CHUNKS: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] AVAILABLE_CHUNKS"); + availableChunks(i.writable, parameter); + break; + } + case O::KEEP_SYNCHRONOUS: { + auto ¶meter = + deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", + i.writable->parent, + "->", + i.writable, + "] KEEP_SYNCHRONOUS"); + keepSynchronous(i.writable, parameter); + break; + } + case O::DEREGISTER: { + auto ¶meter = deref_dynamic_cast>( + i.parameter.get()); + writeToStderr( + "[", i.writable->parent, "->", i.writable, "] DEREGISTER"); + deregister(i.writable, parameter); + break; + } + } + } + catch (...) + { + std::cerr << "[AbstractIOHandlerImpl] IO Task " + << internal::operationAsString(i.operation) + << " failed with exception. Clearing IO queue and " + "passing on the exception." + << std::endl; + while (!m_handler->m_work.empty()) + { + m_handler->m_work.pop(); + } + throw; + } + (*m_handler).m_work.pop(); + } + return std::future(); +} } // namespace openPMD diff --git a/src/IO/HDF5/HDF5Auxiliary.cpp b/src/IO/HDF5/HDF5Auxiliary.cpp index 3a6c9e7cd4..53fb1fb390 100644 --- a/src/IO/HDF5/HDF5Auxiliary.cpp +++ b/src/IO/HDF5/HDF5Auxiliary.cpp @@ -56,15 +56,16 @@ hid_t openPMD::GetH5DataType::operator()(Attribute const &att) using DT = Datatype; switch (att.dtype) { - case DT::CHAR: - case DT::VEC_CHAR: - return H5Tcopy(H5T_NATIVE_CHAR); case DT::UCHAR: case DT::VEC_UCHAR: return H5Tcopy(H5T_NATIVE_UCHAR); case DT::SCHAR: case DT::VEC_SCHAR: return H5Tcopy(H5T_NATIVE_SCHAR); + // NOTE: in HDF5, CHAR is actually either UCHAR or SCHAR. + case DT::CHAR: + case DT::VEC_CHAR: + return H5Tcopy(H5T_NATIVE_CHAR); case DT::SHORT: case DT::VEC_SHORT: return H5Tcopy(H5T_NATIVE_SHORT); diff --git a/src/IO/HDF5/HDF5IOHandler.cpp b/src/IO/HDF5/HDF5IOHandler.cpp index 00f2f0245e..c502188398 100644 --- a/src/IO/HDF5/HDF5IOHandler.cpp +++ b/src/IO/HDF5/HDF5IOHandler.cpp @@ -31,6 +31,7 @@ #include "openPMD/auxiliary/Filesystem.hpp" #include "openPMD/auxiliary/Mpi.hpp" #include "openPMD/auxiliary/StringManip.hpp" +#include "openPMD/auxiliary/TypeTraits.hpp" #include "openPMD/backend/Attribute.hpp" #include @@ -73,6 +74,8 @@ HDF5IOHandlerImpl::HDF5IOHandlerImpl( , m_H5T_CFLOAT{H5Tcreate(H5T_COMPOUND, sizeof(float) * 2)} , m_H5T_CDOUBLE{H5Tcreate(H5T_COMPOUND, sizeof(double) * 2)} , m_H5T_CLONG_DOUBLE{H5Tcreate(H5T_COMPOUND, sizeof(long double) * 2)} + , m_H5T_LONG_DOUBLE_80_LE{H5Tcopy(H5T_IEEE_F64BE)} + , m_H5T_CLONG_DOUBLE_80_LE{H5Tcreate(H5T_COMPOUND, 16 * 2)} { // create a h5py compatible bool type VERIFY( @@ -107,6 +110,28 @@ HDF5IOHandlerImpl::HDF5IOHandlerImpl( H5Tinsert(m_H5T_CLONG_DOUBLE, "r", 0, H5T_NATIVE_LDOUBLE); H5Tinsert(m_H5T_CLONG_DOUBLE, "i", sizeof(long double), H5T_NATIVE_LDOUBLE); + // Create a type that understands 128bit floats with 80 bits of precision + // even on those platforms that do not have it (ARM64, PPC64). + // Otherwise, files created on e.g. AMD64 platforms might not be readable + // on such platforms. + H5Tset_size(m_H5T_LONG_DOUBLE_80_LE, 16); + H5Tset_order(m_H5T_LONG_DOUBLE_80_LE, H5T_ORDER_LE); + H5Tset_precision(m_H5T_LONG_DOUBLE_80_LE, 80); + H5Tset_fields(m_H5T_LONG_DOUBLE_80_LE, 79, 64, 15, 0, 64); + H5Tset_ebias(m_H5T_LONG_DOUBLE_80_LE, 16383); + H5Tset_norm(m_H5T_LONG_DOUBLE_80_LE, H5T_NORM_NONE); + + VERIFY( + m_H5T_LONG_DOUBLE_80_LE >= 0, + "[HDF5] Internal error: Failed to create 128-bit long double"); + + H5Tinsert(m_H5T_CLONG_DOUBLE_80_LE, "r", 0, m_H5T_LONG_DOUBLE_80_LE); + H5Tinsert(m_H5T_CLONG_DOUBLE_80_LE, "i", 16, m_H5T_LONG_DOUBLE_80_LE); + + VERIFY( + m_H5T_LONG_DOUBLE_80_LE >= 0, + "[HDF5] Internal error: Failed to create 128-bit complex long double"); + m_chunks = auxiliary::getEnvString("OPENPMD_HDF5_CHUNKS", "auto"); // JSON option can overwrite env option: if (config.json().contains("hdf5")) @@ -188,6 +213,14 @@ HDF5IOHandlerImpl::~HDF5IOHandlerImpl() std::cerr << "[HDF5] Internal error: Failed to close complex double type\n"; status = H5Tclose(m_H5T_CLONG_DOUBLE); + if (status < 0) + std::cerr << "[HDF5] Internal error: Failed to close complex long " + "double type\n"; + status = H5Tclose(m_H5T_LONG_DOUBLE_80_LE); + if (status < 0) + std::cerr + << "[HDF5] Internal error: Failed to close long double type\n"; + status = H5Tclose(m_H5T_CLONG_DOUBLE_80_LE); if (status < 0) std::cerr << "[HDF5] Internal error: Failed to close complex long " "double type\n"; @@ -958,10 +991,15 @@ void HDF5IOHandlerImpl::openDataset( node_id = H5Gopen( file.id, concrete_h5_file_position(writable->parent).c_str(), gapl); - VERIFY( - node_id >= 0, - "[HDF5] Internal error: Failed to open HDF5 group during dataset " - "opening"); + if (node_id < 0) + { + throw error::ReadError( + error::AffectedObject::Dataset, + error::Reason::NotFound, + "HDF5", + "Internal error: Failed to open HDF5 group during dataset " + "opening"); + } /* Sanitize name */ std::string name = parameters.name; @@ -971,10 +1009,15 @@ void HDF5IOHandlerImpl::openDataset( name += '/'; dataset_id = H5Dopen(node_id, name.c_str(), H5P_DEFAULT); - VERIFY( - dataset_id >= 0, - "[HDF5] Internal error: Failed to open HDF5 dataset during dataset " - "opening"); + if (dataset_id < 0) + { + throw error::ReadError( + error::AffectedObject::Dataset, + error::Reason::NotFound, + "HDF5", + "Internal error: Failed to open HDF5 dataset during dataset " + "opening"); + } hid_t dataset_type, dataset_space; dataset_type = H5Dget_type(dataset_id); @@ -987,47 +1030,114 @@ void HDF5IOHandlerImpl::openDataset( if (dataset_class == H5S_SIMPLE || dataset_class == H5S_SCALAR || dataset_class == H5S_NULL) { - if (H5Tequal(dataset_type, H5T_NATIVE_CHAR)) - d = DT::CHAR; - else if (H5Tequal(dataset_type, H5T_NATIVE_UCHAR)) - d = DT::UCHAR; - else if (H5Tequal(dataset_type, H5T_NATIVE_SCHAR)) - d = DT::SCHAR; - else if (H5Tequal(dataset_type, H5T_NATIVE_SHORT)) - d = DT::SHORT; - else if (H5Tequal(dataset_type, H5T_NATIVE_INT)) - d = DT::INT; - else if (H5Tequal(dataset_type, H5T_NATIVE_LONG)) - d = DT::LONG; - else if (H5Tequal(dataset_type, H5T_NATIVE_LLONG)) - d = DT::LONGLONG; - else if (H5Tequal(dataset_type, H5T_NATIVE_FLOAT)) - d = DT::FLOAT; - else if (H5Tequal(dataset_type, H5T_NATIVE_DOUBLE)) - d = DT::DOUBLE; - else if (H5Tequal(dataset_type, H5T_NATIVE_LDOUBLE)) - d = DT::LONG_DOUBLE; - else if (H5Tequal(dataset_type, m_H5T_CFLOAT)) - d = DT::CFLOAT; - else if (H5Tequal(dataset_type, m_H5T_CDOUBLE)) - d = DT::CDOUBLE; - else if (H5Tequal(dataset_type, m_H5T_CLONG_DOUBLE)) - d = DT::CLONG_DOUBLE; - else if (H5Tequal(dataset_type, H5T_NATIVE_USHORT)) - d = DT::USHORT; - else if (H5Tequal(dataset_type, H5T_NATIVE_UINT)) - d = DT::UINT; - else if (H5Tequal(dataset_type, H5T_NATIVE_ULONG)) - d = DT::ULONG; - else if (H5Tequal(dataset_type, H5T_NATIVE_ULLONG)) - d = DT::ULONGLONG; - else if (H5Tget_class(dataset_type) == H5T_STRING) - d = DT::STRING; - else - throw std::runtime_error("[HDF5] Unknown dataset type"); + constexpr size_t max_retries = 10; + /* + * It happens that an HDF5 file has a type that is not equal to any of + * the native types, but can still be read as its parent type. + * For example an enum (which some applications use to emulate bools) + * can still be read as its parent type, a char. + * Upon not matching any native type, don't give up yet, but check the + * parent type. + * Normally, this procedure should stop at the point where + * H5Tget_super() returns H5I_INVALID_HID, but this is putting a bit + * too much trust in an external library to be the loop's exit + * condition. So, we restrict the loop to a maximum of 10 iterations + * before manually canceling it. + */ + size_t remaining_tries = max_retries; + bool repeat = false; + do + { + repeat = false; + if (H5Tequal(dataset_type, H5T_NATIVE_UCHAR)) + d = DT::UCHAR; + else if (H5Tequal(dataset_type, H5T_NATIVE_SCHAR)) + d = DT::SCHAR; + // NOTE: in HDF5, CHAR is actually either UCHAR or SCHAR. + else if (H5Tequal(dataset_type, H5T_NATIVE_CHAR)) + d = DT::CHAR; + else if (H5Tequal(dataset_type, H5T_NATIVE_SHORT)) + d = DT::SHORT; + else if (H5Tequal(dataset_type, H5T_NATIVE_INT)) + d = DT::INT; + else if (H5Tequal(dataset_type, H5T_NATIVE_LONG)) + d = DT::LONG; + else if (H5Tequal(dataset_type, H5T_NATIVE_LLONG)) + d = DT::LONGLONG; + else if (H5Tequal(dataset_type, H5T_NATIVE_FLOAT)) + d = DT::FLOAT; + else if (H5Tequal(dataset_type, H5T_NATIVE_DOUBLE)) + d = DT::DOUBLE; + else if ( + H5Tequal(dataset_type, H5T_NATIVE_LDOUBLE) || + H5Tequal(dataset_type, m_H5T_LONG_DOUBLE_80_LE)) + d = DT::LONG_DOUBLE; + else if (H5Tequal(dataset_type, m_H5T_CFLOAT)) + d = DT::CFLOAT; + else if (H5Tequal(dataset_type, m_H5T_CDOUBLE)) + d = DT::CDOUBLE; + else if ( + H5Tequal(dataset_type, m_H5T_CLONG_DOUBLE) || + H5Tequal(dataset_type, m_H5T_CLONG_DOUBLE_80_LE)) + d = DT::CLONG_DOUBLE; + else if (H5Tequal(dataset_type, H5T_NATIVE_USHORT)) + d = DT::USHORT; + else if (H5Tequal(dataset_type, H5T_NATIVE_UINT)) + d = DT::UINT; + else if (H5Tequal(dataset_type, H5T_NATIVE_ULONG)) + d = DT::ULONG; + else if (H5Tequal(dataset_type, H5T_NATIVE_ULLONG)) + d = DT::ULONGLONG; + else if (H5Tget_class(dataset_type) == H5T_STRING) + d = DT::STRING; + else + { + auto throw_error = []() { + throw error::ReadError( + error::AffectedObject::Dataset, + error::Reason::UnexpectedContent, + "HDF5", + "Unknown dataset type"); + }; + if (remaining_tries == 0) + { + throw_error(); + } + hid_t next_type = H5Tget_super(dataset_type); + if (next_type == H5I_INVALID_HID) + { + throw_error(); + } + else if (H5Tequal(dataset_type, next_type)) + { + H5Tclose(next_type); + throw_error(); + } + else + { + if (H5Tclose(dataset_type) != 0) + { + throw error::ReadError( + error::AffectedObject::Group, + error::Reason::Other, + "HDF5", + "Internal error: Failed to close HDF5 dataset type " + "during " + "dataset opening"); + } + dataset_type = next_type; + --remaining_tries; + repeat = true; + } + } + } while (repeat); } else - throw std::runtime_error("[HDF5] Unsupported dataset class"); + throw error::ReadError( + error::AffectedObject::Dataset, + error::Reason::UnexpectedContent, + "HDF5", + "Unknown dataset class"); auto dtype = parameters.dtype; *dtype = d; @@ -1045,30 +1155,55 @@ void HDF5IOHandlerImpl::openDataset( herr_t status; status = H5Sclose(dataset_space); - VERIFY( - status == 0, - "[HDF5] Internal error: Failed to close HDF5 dataset space during " - "dataset opening"); + if (status != 0) + { + throw error::ReadError( + error::AffectedObject::Group, + error::Reason::Other, + "HDF5", + "Internal error: Failed to close HDF5 dataset space during " + "dataset opening"); + } status = H5Tclose(dataset_type); - VERIFY( - status == 0, - "[HDF5] Internal error: Failed to close HDF5 dataset type during " - "dataset opening"); + if (status != 0) + { + throw error::ReadError( + error::AffectedObject::Group, + error::Reason::Other, + "HDF5", + "Internal error: Failed to close HDF5 dataset type during " + "dataset opening"); + } status = H5Dclose(dataset_id); - VERIFY( - status == 0, - "[HDF5] Internal error: Failed to close HDF5 dataset during dataset " - "opening"); + if (status != 0) + { + throw error::ReadError( + error::AffectedObject::Group, + error::Reason::Other, + "HDF5", + "Internal error: Failed to close HDF5 dataset during dataset " + "opening"); + } status = H5Gclose(node_id); - VERIFY( - status == 0, - "[HDF5] Internal error: Failed to close HDF5 group during dataset " - "opening"); + if (status != 0) + { + throw error::ReadError( + error::AffectedObject::Group, + error::Reason::Other, + "HDF5", + "Internal error: Failed to close HDF5 group during dataset " + "opening"); + } status = H5Pclose(gapl); - VERIFY( - status == 0, - "[HDF5] Internal error: Failed to close HDF5 property during dataset " - "opening"); + if (status != 0) + { + throw error::ReadError( + error::AffectedObject::Group, + error::Reason::Other, + "HDF5", + "Internal error: Failed to close HDF5 property during dataset " + "opening"); + } writable->written = true; writable->abstractFilePosition = std::make_shared(name); @@ -1760,6 +1895,38 @@ void HDF5IOHandlerImpl::readDataset( {typeid(std::complex).name(), m_H5T_CLONG_DOUBLE}, }); hid_t dataType = getH5DataType(a); + if (H5Tequal(dataType, H5T_NATIVE_LDOUBLE)) + { + // We have previously determined in openDataset() that this dataset is + // of type long double. + // We cannot know if that actually was H5T_NATIVE_LDOUBLE or if it was + // the worked-around m_H5T_LONG_DOUBLE_80_LE. + // Check this. + hid_t checkDatasetTypeAgain = H5Dget_type(dataset_id); + if (!H5Tequal(checkDatasetTypeAgain, H5T_NATIVE_LDOUBLE)) + { + dataType = m_H5T_LONG_DOUBLE_80_LE; + } + status = H5Tclose(checkDatasetTypeAgain); + VERIFY( + status == 0, + "[HDF5] Internal error: Failed to close HDF5 dataset type during " + "dataset reading"); + } + else if (H5Tequal(dataType, m_H5T_CLONG_DOUBLE)) + { + // Same deal for m_H5T_CLONG_DOUBLE + hid_t checkDatasetTypeAgain = H5Dget_type(dataset_id); + if (!H5Tequal(checkDatasetTypeAgain, m_H5T_CLONG_DOUBLE)) + { + dataType = m_H5T_CLONG_DOUBLE_80_LE; + } + status = H5Tclose(checkDatasetTypeAgain); + VERIFY( + status == 0, + "[HDF5] Internal error: Failed to close HDF5 dataset type during " + "dataset reading"); + } VERIFY( dataType >= 0, "[HDF5] Internal error: Failed to get HDF5 datatype during dataset " @@ -1951,6 +2118,14 @@ void HDF5IOHandlerImpl::readAttribute( status = H5Aread(attr_id, attr_type, &l); a = Attribute(l); } + else if (H5Tequal(attr_type, m_H5T_LONG_DOUBLE_80_LE)) + { + char bfr[16]; + status = H5Aread(attr_id, attr_type, bfr); + H5Tconvert( + attr_type, H5T_NATIVE_LDOUBLE, 1, bfr, nullptr, H5P_DEFAULT); + a = Attribute(reinterpret_cast(bfr)[0]); + } else if (H5Tget_class(attr_type) == H5T_STRING) { if (H5Tis_variable_str(attr_type)) @@ -2069,6 +2244,20 @@ void HDF5IOHandlerImpl::readAttribute( status = H5Aread(attr_id, attr_type, &cld); a = Attribute(cld); } + else if (complexSize == 16) + { + char bfr[2 * 16]; + status = H5Aread(attr_id, attr_type, bfr); + H5Tconvert( + attr_type, + m_H5T_CLONG_DOUBLE, + 1, + bfr, + nullptr, + H5P_DEFAULT); + a = Attribute( + reinterpret_cast *>(bfr)[0]); + } else throw error::ReadError( error::AffectedObject::Attribute, @@ -2088,7 +2277,8 @@ void HDF5IOHandlerImpl::readAttribute( error::AffectedObject::Attribute, error::Reason::UnexpectedContent, "HDF5", - "[HDF5] Unsupported scalar attribute type"); + "[HDF5] Unsupported scalar attribute type for '" + attr_name + + "'."); } else if (attr_class == H5S_SIMPLE) { @@ -2210,6 +2400,49 @@ void HDF5IOHandlerImpl::readAttribute( status = H5Aread(attr_id, attr_type, vcld.data()); a = Attribute(vcld); } + else if (H5Tequal(attr_type, m_H5T_CLONG_DOUBLE_80_LE)) + { + // worst case: + // sizeof(long double) is only 8, but the dataset on disk has + // 16-byte long doubles + // --> do NOT use `new long double[]` as the buffer would be too + // small + auto *tmpBuffer = + reinterpret_cast(new char[16lu * 2lu * dims[0]]); + status = H5Aread(attr_id, attr_type, tmpBuffer); + H5Tconvert( + attr_type, + m_H5T_CLONG_DOUBLE, + dims[0], + tmpBuffer, + nullptr, + H5P_DEFAULT); + std::vector > vcld{ + tmpBuffer, tmpBuffer + dims[0]}; + delete[] tmpBuffer; + a = Attribute(std::move(vcld)); + } + else if (H5Tequal(attr_type, m_H5T_LONG_DOUBLE_80_LE)) + { + // worst case: + // sizeof(long double) is only 8, but the dataset on disk has + // 16-byte long doubles + // --> do NOT use `new long double[]` as the buffer would be too + // small + auto *tmpBuffer = + reinterpret_cast(new char[16lu * dims[0]]); + status = H5Aread(attr_id, attr_type, tmpBuffer); + H5Tconvert( + attr_type, + H5T_NATIVE_LDOUBLE, + dims[0], + tmpBuffer, + nullptr, + H5P_DEFAULT); + std::vector vld80{tmpBuffer, tmpBuffer + dims[0]}; + delete[] tmpBuffer; + a = Attribute(std::move(vld80)); + } else if (H5Tget_class(attr_type) == H5T_STRING) { std::vector vs; @@ -2244,11 +2477,39 @@ void HDF5IOHandlerImpl::readAttribute( a = Attribute(vs); } else + { + auto order = H5Tget_order(attr_type); + auto prec = H5Tget_precision(attr_type); + auto ebias = H5Tget_ebias(attr_type); + size_t spos, epos, esize, mpos, msize; + H5Tget_fields(attr_type, &spos, &epos, &esize, &mpos, &msize); + + auto norm = H5Tget_norm(attr_type); + auto cset = H5Tget_cset(attr_type); + auto sign = H5Tget_sign(attr_type); + + std::stringstream detailed_info; + detailed_info << "order " << std::to_string(order) << std::endl + << "prec " << std::to_string(prec) << std::endl + << "ebias " << std::to_string(ebias) << std::endl + << "fields " << std::to_string(spos) << " " + << std::to_string(epos) << " " + << std::to_string(esize) << " " + << std::to_string(mpos) << " " + << std::to_string(msize) << "norm " + << std::to_string(norm) << std::endl + << "cset " << std::to_string(cset) << std::endl + << "sign " << std::to_string(sign) << std::endl + << std::endl; + throw error::ReadError( error::AffectedObject::Attribute, error::Reason::UnexpectedContent, "HDF5", - "[HDF5] Unsupported simple attribute type"); + "[HDF5] Unsupported simple attribute type " + + std::to_string(attr_type) + " for " + attr_name + + ".\n(Info for debugging: " + detailed_info.str() + ")"); + } } else throw std::runtime_error("[HDF5] Unsupported attribute class"); diff --git a/src/Iteration.cpp b/src/Iteration.cpp index 26ab93940e..9fd8a835ec 100644 --- a/src/Iteration.cpp +++ b/src/Iteration.cpp @@ -282,26 +282,34 @@ void Iteration::flushVariableBased( Parameter pOpen; pOpen.path = ""; IOHandler()->enqueue(IOTask(this, pOpen)); - /* - * In v-based encoding, the snapshot attribute must always be written, - * so don't set the `changesOverSteps` flag of the IOTask here. - * Reason: Even in backends that don't support changing attributes, - * variable-based iteration encoding can be used to write one single - * iteration. Then, this attribute determines which iteration it is. - */ - this->setAttribute("snapshot", i); } switch (flushParams.flushLevel) { case FlushLevel::CreateOrOpenFiles: - break; + return; case FlushLevel::SkeletonOnly: case FlushLevel::InternalFlush: case FlushLevel::UserFlush: flush(flushParams); break; } + + if (!written()) + { + /* create iteration path */ + Parameter pOpen; + pOpen.path = ""; + IOHandler()->enqueue(IOTask(this, pOpen)); + /* + * In v-based encoding, the snapshot attribute must always be written, + * so don't set the `changesOverSteps` flag of the IOTask here. + * Reason: Even in backends that don't support changing attributes, + * variable-based iteration encoding can be used to write one single + * iteration. Then, this attribute determines which iteration it is. + */ + this->setAttribute("snapshot", i); + } } void Iteration::flush(internal::FlushParams const &flushParams) diff --git a/src/ReadIterations.cpp b/src/ReadIterations.cpp index fdc64c7845..e92ede964f 100644 --- a/src/ReadIterations.cpp +++ b/src/ReadIterations.cpp @@ -25,6 +25,7 @@ #include "openPMD/Series.hpp" #include +#include namespace openPMD { @@ -58,7 +59,7 @@ SeriesIterator::SeriesIterator() = default; void SeriesIterator::initSeriesInLinearReadMode() { - auto &data = *m_data; + auto &data = get(); auto &series = *data.series; series.IOHandler()->m_seriesStatus = internal::SeriesStatus::Parsing; try @@ -103,13 +104,27 @@ void SeriesIterator::initSeriesInLinearReadMode() series.IOHandler()->m_seriesStatus = internal::SeriesStatus::Default; } +void SeriesIterator::close() +{ + *m_data = std::nullopt; // turn this into end iterator +} + SeriesIterator::SeriesIterator( Series series_in, std::optional parsePreference) - : m_data{std::make_shared()} + : m_data{std::make_shared>(std::in_place)} { - auto &data = *m_data; + auto &data = get(); data.parsePreference = std::move(parsePreference); - data.series = std::move(series_in); + /* + * Since the iterator is stored in + * internal::SeriesData::m_sharedStatefulIterator, + * we need to use a non-owning Series instance here for tie-breaking + * purposes. + * This is ok due to the usual C++ iterator invalidation workflows + * (deleting the original container invalidates the iterator). + */ + data.series = Series(std::shared_ptr( + series_in.m_series.get(), [](auto const *) {})); auto &series = data.series.value(); if (series.IOHandler()->m_frontendAccess == Access::READ_LINEAR && series.iterations.empty()) @@ -120,7 +135,7 @@ SeriesIterator::SeriesIterator( auto it = series.get().iterations.begin(); if (it == series.get().iterations.end()) { - *this = end(); + this->close(); return; } else if ( @@ -212,12 +227,12 @@ SeriesIterator::SeriesIterator( if (status == AdvanceStatus::OVER) { - *this = end(); + this->close(); return; } if (!setCurrentIteration()) { - *this = end(); + this->close(); return; } it->second.setStepStatus(StepStatus::DuringStep); @@ -226,7 +241,7 @@ SeriesIterator::SeriesIterator( std::optional SeriesIterator::nextIterationInStep() { - auto &data = *m_data; + auto &data = get(); using ret_t = std::optional; if (data.iterationsInCurrentStep.empty()) @@ -298,7 +313,7 @@ std::optional SeriesIterator::nextIterationInStep() std::optional SeriesIterator::nextStep(size_t recursion_depth) { - auto &data = *m_data; + auto &data = get(); // since we are in group-based iteration layout, it does not // matter which iteration we begin a step upon AdvanceStatus status{}; @@ -339,7 +354,7 @@ std::optional SeriesIterator::nextStep(size_t recursion_depth) if (status == AdvanceStatus::RANDOMACCESS || status == AdvanceStatus::OVER) { - *this = end(); + this->close(); return {this}; } else @@ -366,7 +381,7 @@ std::optional SeriesIterator::nextStep(size_t recursion_depth) if (status == AdvanceStatus::RANDOMACCESS || status == AdvanceStatus::OVER) { - *this = end(); + this->close(); return {this}; } else @@ -390,7 +405,7 @@ std::optional SeriesIterator::nextStep(size_t recursion_depth) if (status == AdvanceStatus::OVER) { - *this = end(); + this->close(); return {this}; } @@ -399,7 +414,7 @@ std::optional SeriesIterator::nextStep(size_t recursion_depth) std::optional SeriesIterator::loopBody() { - auto &data = *m_data; + auto &data = get(); Series &series = data.series.value(); auto &iterations = series.iterations; @@ -485,7 +500,7 @@ std::optional SeriesIterator::loopBody() if (series.iterationEncoding() == IterationEncoding::fileBased) { // this one is handled above, stream is over once it proceeds to here - *this = end(); + this->close(); return {this}; } @@ -495,7 +510,7 @@ std::optional SeriesIterator::loopBody() void SeriesIterator::deactivateDeadIteration(iteration_index_t index) { - auto &data = *m_data; + auto &data = get(); switch (data.series->iterationEncoding()) { case IterationEncoding::fileBased: { @@ -520,10 +535,10 @@ void SeriesIterator::deactivateDeadIteration(iteration_index_t index) SeriesIterator &SeriesIterator::operator++() { - auto &data = *m_data; + auto &data = get(); if (!data.series.has_value()) { - *this = end(); + this->close(); return *this; } auto oldIterationIndex = data.currentIteration; @@ -570,7 +585,7 @@ SeriesIterator &SeriesIterator::operator++() IndexedIteration SeriesIterator::operator*() { - auto &data = *m_data; + auto &data = get(); return IndexedIteration( data.series.value().iterations[data.currentIteration], data.currentIteration); @@ -578,10 +593,12 @@ IndexedIteration SeriesIterator::operator*() bool SeriesIterator::operator==(SeriesIterator const &other) const { - return (this->m_data.operator bool() && other.m_data.operator bool() && - (this->m_data->currentIteration == - other.m_data->currentIteration)) || - (!this->m_data.operator bool() && !other.m_data.operator bool()); + return + // either both iterators are filled + (this->m_data->has_value() && other.m_data->has_value() && + (this->get().currentIteration == other.get().currentIteration)) || + // or both are empty + (!this->m_data->has_value() && !other.m_data->has_value()); } bool SeriesIterator::operator!=(SeriesIterator const &other) const @@ -600,20 +617,24 @@ ReadIterations::ReadIterations( std::optional parsePreference) : m_series(std::move(series)), m_parsePreference(std::move(parsePreference)) { - if (access == Access::READ_LINEAR) + auto &data = m_series.get(); + if (access == Access::READ_LINEAR && !data.m_sharedStatefulIterator) { // Open the iterator now already, so that metadata may already be read - alreadyOpened = iterator_t{m_series, m_parsePreference}; + data.m_sharedStatefulIterator = + std::make_unique(m_series, m_parsePreference); } } ReadIterations::iterator_t ReadIterations::begin() { - if (!alreadyOpened.has_value()) + auto &series = m_series.get(); + if (!series.m_sharedStatefulIterator) { - alreadyOpened = iterator_t{m_series, m_parsePreference}; + series.m_sharedStatefulIterator = + std::make_unique(m_series, m_parsePreference); } - return alreadyOpened.value(); + return *series.m_sharedStatefulIterator; } ReadIterations::iterator_t ReadIterations::end() diff --git a/src/Series.cpp b/src/Series.cpp index 779ba97906..cd360aa58c 100644 --- a/src/Series.cpp +++ b/src/Series.cpp @@ -23,6 +23,7 @@ #include "openPMD/IO/AbstractIOHandler.hpp" #include "openPMD/IO/AbstractIOHandlerHelper.hpp" #include "openPMD/IO/Format.hpp" +#include "openPMD/IterationEncoding.hpp" #include "openPMD/ReadIterations.hpp" #include "openPMD/auxiliary/Date.hpp" #include "openPMD/auxiliary/Filesystem.hpp" @@ -424,6 +425,11 @@ std::unique_ptr Series::parseInput(std::string filepath) filepath = auxiliary::replace_all(filepath, "\\", "/"); } #endif + if (auxiliary::ends_with(filepath, auxiliary::directory_separator)) + { + filepath = auxiliary::replace_last( + filepath, std::string(&auxiliary::directory_separator, 1), ""); + } auto const pos = filepath.find_last_of(auxiliary::directory_separator); if (std::string::npos == pos) { @@ -660,7 +666,7 @@ Given file pattern: ')END" break; } } - series.m_lastFlushSuccessful = true; + IOHandler()->m_lastFlushSuccessful = true; } void Series::initDefaults(IterationEncoding ie, bool initAll) @@ -704,8 +710,7 @@ std::future Series::flush_impl( internal::FlushParams flushParams, bool flushIOHandler) { - auto &series = get(); - series.m_lastFlushSuccessful = true; + IOHandler()->m_lastFlushSuccessful = true; try { switch (iterationEncoding()) @@ -721,16 +726,18 @@ std::future Series::flush_impl( } if (flushIOHandler) { + IOHandler()->m_lastFlushSuccessful = true; return IOHandler()->flush(flushParams); } else { + IOHandler()->m_lastFlushSuccessful = true; return {}; } } catch (...) { - series.m_lastFlushSuccessful = false; + IOHandler()->m_lastFlushSuccessful = false; throw; } } @@ -1842,6 +1849,13 @@ AdvanceStatus Series::advance( else { param.mode = mode; + if (iterationEncoding() == IterationEncoding::variableBased && + access::write(IOHandler()->m_frontendAccess) && + mode == AdvanceMode::BEGINSTEP && series.m_wroteAtLeastOneIOStep) + { + // If the backend does not support steps, we cannot continue here + param.isThisStepMandatory = true; + } IOTask task(&file.m_writable, param); IOHandler()->enqueue(task); } @@ -1932,6 +1946,13 @@ AdvanceStatus Series::advance(AdvanceMode mode) Parameter param; param.mode = mode; + if (iterationEncoding() == IterationEncoding::variableBased && + access::write(IOHandler()->m_frontendAccess) && + mode == AdvanceMode::BEGINSTEP && series.m_wroteAtLeastOneIOStep) + { + // If the backend does not support steps, we cannot continue here + param.isThisStepMandatory = true; + } IOTask task(&series.m_writable, param); IOHandler()->enqueue(task); @@ -1947,7 +1968,7 @@ void Series::flushStep(bool doFlush) { auto &series = get(); if (!series.m_currentlyActiveIterations.empty() && - IOHandler()->m_frontendAccess != Access::READ_ONLY) + access::write(IOHandler()->m_frontendAccess)) { /* * Warning: changing attribute extents over time (probably) unsupported @@ -1961,6 +1982,7 @@ void Series::flushStep(bool doFlush) wAttr.resource = std::vector{ series.m_currentlyActiveIterations.begin(), series.m_currentlyActiveIterations.end()}; + series.m_currentlyActiveIterations.clear(); wAttr.dtype = Datatype::VEC_ULONGLONG; IOHandler()->enqueue(IOTask(&series.iterations, wAttr)); if (doFlush) @@ -1968,6 +1990,7 @@ void Series::flushStep(bool doFlush) IOHandler()->flush(internal::defaultFlushParams); } } + series.m_wroteAtLeastOneIOStep = true; } auto Series::openIterationIfDirty(IterationIndex_t index, Iteration iteration) @@ -2262,12 +2285,21 @@ namespace internal * `Series` is needlessly flushed a second time. Otherwise, error * messages can get very confusing. */ - if (this->m_lastFlushSuccessful && m_writable.IOHandler && - m_writable.IOHandler->has_value()) + Series impl{{this, [](auto const *) {}}}; + if (auto IOHandler = impl.IOHandler(); + IOHandler && IOHandler->m_lastFlushSuccessful) { - Series impl{{this, [](auto const *) {}}}; impl.flush(); - impl.flushStep(/* doFlush = */ true); + /* + * In file-based iteration encoding, this must be triggered by + * Iteration::endStep() since the "snapshot" attribute is different + * for each file. + * Also, at this point the files might have already been closed. + */ + if (impl.iterationEncoding() != IterationEncoding::fileBased) + { + impl.flushStep(/* doFlush = */ true); + } } // Not strictly necessary, but clear the map of iterations // This releases the openPMD hierarchy @@ -2309,7 +2341,8 @@ Series::Series( input->format, input->filenameExtension, comm, - optionsJson); + optionsJson, + filepath); init(std::move(handler), std::move(input)); json::warnGlobalUnusedOptions(optionsJson); } @@ -2326,7 +2359,12 @@ Series::Series( auto input = parseInput(filepath); parseJsonOptions(optionsJson, *input); auto handler = createIOHandler( - input->path, at, input->format, input->filenameExtension, optionsJson); + input->path, + at, + input->format, + input->filenameExtension, + optionsJson, + filepath); init(std::move(handler), std::move(input)); json::warnGlobalUnusedOptions(optionsJson); } @@ -2344,6 +2382,11 @@ ReadIterations Series::readIterations() this->m_series, IOHandler()->m_frontendAccess, get().m_parsePreference}; } +void Series::parseBase() +{ + readIterations(); +} + WriteIterations Series::writeIterations() { auto &series = get(); diff --git a/src/WriteIterations.cpp b/src/WriteIterations.cpp index 2bc34f0416..c8cf9b06b0 100644 --- a/src/WriteIterations.cpp +++ b/src/WriteIterations.cpp @@ -33,8 +33,8 @@ WriteIterations::SharedResources::SharedResources( WriteIterations::SharedResources::~SharedResources() { - if (currentlyOpen.has_value() && - iterations.retrieveSeries().get().m_lastFlushSuccessful) + if (auto IOHandler = iterations.IOHandler(); currentlyOpen.has_value() && + IOHandler && IOHandler->m_lastFlushSuccessful) { auto lastIterationIndex = currentlyOpen.value(); auto &lastIteration = iterations.at(lastIterationIndex); @@ -82,7 +82,17 @@ WriteIterations::mapped_type &WriteIterations::operator[](key_type &&key) auto &res = s.iterations[std::move(key)]; if (res.getStepStatus() == StepStatus::NoStep) { - res.beginStep(/* reread = */ false); + try + { + res.beginStep(/* reread = */ false); + } + catch (error::OperationUnsupportedInBackend const &) + { + s.iterations.retrieveSeries() + .get() + .m_currentlyActiveIterations.clear(); + throw; + } res.setStepStatus(StepStatus::DuringStep); } return res; diff --git a/src/backend/Attributable.cpp b/src/backend/Attributable.cpp index eb93888718..2d97dd0ff3 100644 --- a/src/backend/Attributable.cpp +++ b/src/backend/Attributable.cpp @@ -203,6 +203,7 @@ auto Attributable::myPath() const -> MyPath res.seriesName = series.name(); res.seriesExtension = suffix(seriesData.m_format); res.directory = IOHandler()->directory; + res.access = IOHandler()->m_backendAccess; return res; } diff --git a/src/backend/PatchRecordComponent.cpp b/src/backend/PatchRecordComponent.cpp index e1477ef7bd..891125d249 100644 --- a/src/backend/PatchRecordComponent.cpp +++ b/src/backend/PatchRecordComponent.cpp @@ -116,24 +116,28 @@ void PatchRecordComponent::flush( void PatchRecordComponent::read() { - Parameter aRead; - - aRead.name = "unitSI"; - IOHandler()->enqueue(IOTask(this, aRead)); - IOHandler()->flush(internal::defaultFlushParams); - if (auto val = Attribute(*aRead.resource).getOptional(); - val.has_value()) - setUnitSI(val.value()); - else - throw error::ReadError( - error::AffectedObject::Attribute, - error::Reason::UnexpectedContent, - {}, - "Unexpected Attribute datatype for 'unitSI' (expected double, " - "found " + - datatypeToString(Attribute(*aRead.resource).dtype) + ")"); - readAttributes(ReadMode::FullyReread); // this will set dirty() = false + + if (containsAttribute("unitSI")) + { + /* + * No need to call setUnitSI + * If it's in the attributes map, then it's already set + * Just verify that it has the right type (getOptional<>() does + * conversions if possible, so this check is non-intrusive) + */ + if (auto val = getAttribute("unitSI").getOptional(); + !val.has_value()) + { + throw error::ReadError( + error::AffectedObject::Attribute, + error::Reason::UnexpectedContent, + {}, + "Unexpected Attribute datatype for 'unitSI' (expected double, " + "found " + + datatypeToString(getAttribute("unitSI").dtype) + ")"); + } + } } bool PatchRecordComponent::dirtyRecursive() const diff --git a/src/binding/python/Attributable.cpp b/src/binding/python/Attributable.cpp index 61f1376b94..d89fa86510 100644 --- a/src/binding/python/Attributable.cpp +++ b/src/binding/python/Attributable.cpp @@ -362,6 +362,15 @@ bool setAttributeFromObject( void init_Attributable(py::module &m) { + py::class_(m, "AttributablePath") + .def_readonly("directory", &Attributable::MyPath::directory) + .def_readonly("series_name", &Attributable::MyPath::seriesName) + .def_readonly( + "series_extension", &Attributable::MyPath::seriesExtension) + .def_readonly("group", &Attributable::MyPath::group) + .def_readonly("access", &Attributable::MyPath::access) + .def_property_readonly("file_path", &Attributable::MyPath::filePath); + py::class_(m, "Attributable") .def(py::init()) @@ -369,7 +378,7 @@ void init_Attributable(py::module &m) "__repr__", [](Attributable const &attr) { return ""; + std::to_string(attr.numAttributes()) + "' attribute(s)>"; }) .def( "series_flush", diff --git a/src/binding/python/Container.cpp b/src/binding/python/Container.cpp index 28bda651ff..8adebc570d 100644 --- a/src/binding/python/Container.cpp +++ b/src/binding/python/Container.cpp @@ -41,7 +41,9 @@ #include "openPMD/backend/PatchRecord.hpp" #include "openPMD/backend/PatchRecordComponent.hpp" +#include #include +#include #include #include @@ -100,6 +102,22 @@ bind_container(py::handle scope, std::string const &name, Args &&...args) // keep container alive while iterator exists py::keep_alive<0, 1>()); + cl.def("__repr__", [name](Map const &m) { + std::stringstream stream; + stream << ""; + return stream.str(); + }); + cl.def( "items", [](Map &m) { return py::make_iterator(m.begin(), m.end()); }, diff --git a/src/binding/python/Dataset.cpp b/src/binding/python/Dataset.cpp index e24d3b52ba..bdd35db956 100644 --- a/src/binding/python/Dataset.cpp +++ b/src/binding/python/Dataset.cpp @@ -59,8 +59,24 @@ void init_Dataset(py::module &m) .def( "__repr__", [](const Dataset &d) { - return ""; + std::stringstream stream; + stream << ""; + } + else + { + auto begin = d.extent.begin(); + stream << '[' << *begin++; + for (; begin != d.extent.end(); ++begin) + { + stream << ", " << *begin; + } + stream << "]>"; + } + return stream.str(); }) .def_readonly("extent", &Dataset::extent) diff --git a/src/binding/python/Iteration.cpp b/src/binding/python/Iteration.cpp index 0ac290f7ff..de9dedf65f 100644 --- a/src/binding/python/Iteration.cpp +++ b/src/binding/python/Iteration.cpp @@ -40,7 +40,9 @@ void init_Iteration(py::module &m) [](Iteration const &it) { std::stringstream ss; ss << ""; + << it.template time() * it.timeUnitSI() + << " s' with " << std::to_string(it.numAttributes()) + << " attributes>"; return ss.str(); }) diff --git a/src/binding/python/Mesh.cpp b/src/binding/python/Mesh.cpp index 744712b05e..8b6da30e64 100644 --- a/src/binding/python/Mesh.cpp +++ b/src/binding/python/Mesh.cpp @@ -42,7 +42,8 @@ void init_Mesh(py::module &m) "__repr__", [](Mesh const &mesh) { return ""; + std::to_string(mesh.size()) + "' record component(s) and " + + std::to_string(mesh.numAttributes()) + " attributes>"; }) .def_property( diff --git a/src/binding/python/MeshRecordComponent.cpp b/src/binding/python/MeshRecordComponent.cpp index ff702b53d5..98e602bbcb 100644 --- a/src/binding/python/MeshRecordComponent.cpp +++ b/src/binding/python/MeshRecordComponent.cpp @@ -38,9 +38,25 @@ void init_MeshRecordComponent(py::module &m) m, "Mesh_Record_Component"); cl.def( "__repr__", - [](MeshRecordComponent const &rc) { - return ""; + [](RecordComponent const &rc) { + std::stringstream stream; + stream << ""; + } + else + { + auto begin = extent.begin(); + stream << '[' << *begin++; + for (; begin != extent.end(); ++begin) + { + stream << ", " << *begin; + } + stream << "]>"; + } + return stream.str(); }) .def_property( diff --git a/src/binding/python/ParticlePatches.cpp b/src/binding/python/ParticlePatches.cpp index 7c52652717..28deeab1b5 100644 --- a/src/binding/python/ParticlePatches.cpp +++ b/src/binding/python/ParticlePatches.cpp @@ -36,8 +36,11 @@ void init_ParticlePatches(py::module &m) .def( "__repr__", [](ParticlePatches const &pp) { - return ""; + std::stringstream stream; + stream << ""; + return stream.str(); }) .def_property_readonly("num_patches", &ParticlePatches::numPatches); diff --git a/src/binding/python/ParticleSpecies.cpp b/src/binding/python/ParticleSpecies.cpp index b6929ad66f..349081ea8e 100644 --- a/src/binding/python/ParticleSpecies.cpp +++ b/src/binding/python/ParticleSpecies.cpp @@ -27,6 +27,7 @@ #include "openPMD/backend/Container.hpp" #include "openPMD/binding/python/Pickle.hpp" +#include #include #include @@ -38,7 +39,13 @@ void init_ParticleSpecies(py::module &m) py::class_ > cl(m, "ParticleSpecies"); cl.def( "__repr__", - [](ParticleSpecies const &) { return ""; }) + [](ParticleSpecies const &p) { + std::stringstream stream; + stream << ""; + return stream.str(); + }) .def_readwrite("particle_patches", &ParticleSpecies::particlePatches); add_pickle( diff --git a/src/binding/python/PatchRecordComponent.cpp b/src/binding/python/PatchRecordComponent.cpp index f2fe0aa753..bf6a687b04 100644 --- a/src/binding/python/PatchRecordComponent.cpp +++ b/src/binding/python/PatchRecordComponent.cpp @@ -52,6 +52,29 @@ void init_PatchRecordComponent(py::module &m) &BaseRecordComponent::unitSI, &PatchRecordComponent::setUnitSI) + .def( + "__repr__", + [](PatchRecordComponent const &rc) { + std::stringstream stream; + stream << ""; + } + else + { + auto begin = extent.begin(); + stream << '[' << *begin++; + for (; begin != extent.end(); ++begin) + { + stream << ", " << *begin; + } + stream << "]>"; + } + return stream.str(); + }) + .def("reset_dataset", &PatchRecordComponent::resetDataset) .def_property_readonly( "ndims", &PatchRecordComponent::getDimensionality) diff --git a/src/binding/python/Record.cpp b/src/binding/python/Record.cpp index 39ad120c6e..ee96e304e6 100644 --- a/src/binding/python/Record.cpp +++ b/src/binding/python/Record.cpp @@ -38,7 +38,13 @@ void init_Record(py::module &m) py::class_ > cl(m, "Record"); cl.def(py::init()) - .def("__repr__", [](Record const &) { return ""; }) + .def( + "__repr__", + [](Record const &r) { + return ""; + }) .def_property( "unit_dimension", diff --git a/src/binding/python/RecordComponent.cpp b/src/binding/python/RecordComponent.cpp index 1b6c4ebee5..32e1e0fb0d 100644 --- a/src/binding/python/RecordComponent.cpp +++ b/src/binding/python/RecordComponent.cpp @@ -23,6 +23,7 @@ #include #include "openPMD/DatatypeHelpers.hpp" +#include "openPMD/Error.hpp" #include "openPMD/RecordComponent.hpp" #include "openPMD/Series.hpp" #include "openPMD/backend/BaseRecordComponent.hpp" @@ -35,6 +36,7 @@ #include #include #include +#include #include #include #include @@ -519,18 +521,28 @@ void load_chunk( // whether we have that if (strides.size() == 0) { - throw std::runtime_error( + throw error::WrongAPIUsage( "[Record_Component::load_chunk()] Empty buffer passed."); } { py::ssize_t accumulator = toBytes(r.getDatatype()); + if (buffer_info.itemsize != accumulator) + { + std::stringstream errorMsg; + errorMsg << "[Record_Component::load_chunk()] Loading from a " + "record component of type " + << r.getDatatype() << " with item size " << accumulator + << ", but Python buffer has item size " + << buffer_info.itemsize << "."; + throw error::WrongAPIUsage(errorMsg.str()); + } size_t dim = strides.size(); while (dim > 0) { --dim; if (strides[dim] != accumulator) { - throw std::runtime_error( + throw error::WrongAPIUsage( "[Record_Component::load_chunk()] Requires contiguous slab" " of memory."); } @@ -754,8 +766,24 @@ void init_RecordComponent(py::module &m) cl.def( "__repr__", [](RecordComponent const &rc) { - return ""; + std::stringstream stream; + stream << ""; + } + else + { + auto begin = extent.begin(); + stream << '[' << *begin++; + for (; begin != extent.end(); ++begin) + { + stream << ", " << *begin; + } + stream << "]>"; + } + return stream.str(); }) .def_property( diff --git a/src/binding/python/Series.cpp b/src/binding/python/Series.cpp index cdff83fd43..0a6cec6547 100644 --- a/src/binding/python/Series.cpp +++ b/src/binding/python/Series.cpp @@ -19,9 +19,12 @@ * If not, see . */ +#include #include #include +#include "openPMD/IO/Access.hpp" +#include "openPMD/IterationEncoding.hpp" #include "openPMD/Series.hpp" #include "openPMD/auxiliary/JSON.hpp" #include "openPMD/config.hpp" @@ -32,6 +35,7 @@ #include #endif +#include #include namespace py = pybind11; @@ -53,9 +57,35 @@ struct openPMD_PyMPICommObject using openPMD_PyMPIIntracommObject = openPMD_PyMPICommObject; #endif +struct SeriesIteratorPythonAdaptor : SeriesIterator +{ + SeriesIteratorPythonAdaptor(SeriesIterator it) + : SeriesIterator(std::move(it)) + {} + + /* + * Python iterators are weird and call `__next__()` already for getting the + * first element. + * In that case, no `operator++()` must be called... + */ + bool first_iteration = true; +}; + void init_Series(py::module &m) { - py::class_(m, "WriteIterations") + py::class_(m, "WriteIterations", R"END( +Writing side of the streaming API. + +Create instance via Series.writeIterations(). +Restricted Container of Iterations, designed to allow reading any kind +of Series, streaming and non-streaming alike. +Calling Iteration.close() manually before opening the next iteration is +encouraged and will implicitly flush all deferred IO actions. +Otherwise, Iteration.close() will be implicitly called upon +opening the next iteration or upon destruction. +Since this is designed for streaming mode, reopening an iteration is +not possible once it has been closed. + )END") .def( "__getitem__", [](WriteIterations writeIterations, Series::IterationIndex_t key) { @@ -65,7 +95,55 @@ void init_Series(py::module &m) py::return_value_policy::copy); py::class_(m, "IndexedIteration") .def_readonly("iteration_index", &IndexedIteration::iterationIndex); - py::class_(m, "ReadIterations") + + py::class_(m, "SeriesIterator") + .def( + "__next__", + [](SeriesIteratorPythonAdaptor &iterator) { + if (iterator == SeriesIterator::end()) + { + throw py::stop_iteration(); + } + /* + * Closing the iteration must happen under the GIL lock since + * Python buffers might be accessed + */ + if (!iterator.first_iteration) + { + if (!(*iterator).closed()) + { + (*iterator).close(); + } + py::gil_scoped_release release; + ++iterator; + } + iterator.first_iteration = false; + if (iterator == SeriesIterator::end()) + { + throw py::stop_iteration(); + } + else + { + return *iterator; + } + } + + ); + + py::class_(m, "ReadIterations", R"END( +Reading side of the streaming API. + +Create instance via Series.readIterations(). +For use in a foreach loop over iterations. +Designed to allow reading any kind of Series, streaming and non-streaming alike. +Calling Iteration.close() manually before opening the next iteration is +encouraged and will implicitly flush all deferred IO actions. +Otherwise, Iteration.close() will be implicitly called upon +SeriesIterator.__next__(), i.e. upon going to the next iteration in +the foreach loop. +Since this is designed for streaming mode, reopening an iteration is +not possible once it has been closed. + )END") .def( "__iter__", [](ReadIterations &readIterations) { @@ -153,6 +231,20 @@ void init_Series(py::module &m) py::arg("options") = "{}") #endif .def("__bool__", &Series::operator bool) + .def( + "__repr__", + [](Series const &s) { + std::stringstream stream; + auto myPath = s.myPath(); + stream << ""; + return stream.str(); + }) .def("close", &Series::close, R"( Closes the Series and release the data storage/transport backends. @@ -232,11 +324,64 @@ this method. py::return_value_policy::reference, // garbage collection: return value must be freed before Series py::keep_alive<1, 0>()) - .def("read_iterations", &Series::readIterations, py::keep_alive<0, 1>()) + .def( + "read_iterations", + [](Series &s) { + py::gil_scoped_release release; + return s.readIterations(); + }, + py::keep_alive<0, 1>(), + R"END( +Entry point to the reading end of the streaming API. + +Creates and returns an instance of the ReadIterations class which can +be used for iterating over the openPMD iterations in a C++11-style for +loop. +`Series.read_iterations()` is an intentionally restricted API that +ensures a workflow which also works in streaming setups, e.g. an +iteration cannot be opened again once it has been closed. +For a less restrictive API in non-streaming situations, +`Series.iterations` can be accessed directly. +Look for the ReadIterations class for further documentation. + )END") + .def( + "parse_base", + [](Series &s) { + py::gil_scoped_release release; + s.parseBase(); + }, + &R"END( +Parse the Series. + +Only necessary in linear read mode. +In linear read mode, the Series constructor does not do any IO accesses. +This call effectively triggers the side effects of +Series::readIterations(), for use cases where data needs to be accessed +before iterating through the iterations. + +The reason for introducing this restricted alias to +Series.read_iterations() is that the name "read_iterations" is misleading +for that use case: When using IO steps, this call only ensures that the +first step is parsed.)END"[1]) .def( "write_iterations", &Series::writeIterations, - py::keep_alive<0, 1>()); + py::keep_alive<0, 1>(), + R"END( +Entry point to the writing end of the streaming API. + +Creates and returns an instance of the WriteIterations class which is an +intentionally restricted container of iterations that takes care of +streaming semantics, e.g. ensuring that an iteration cannot be reopened +once closed. +For a less restrictive API in non-streaming situations, +`Series.iterations` can be accessed directly. +The created object is stored as member of the Series object, hence this +method may be called as many times as a user wishes. +There is only one shared iterator state per Series, even when calling +this method twice. +Look for the WriteIterations class for further documentation. + )END"); m.def( "merge_json", diff --git a/src/binding/python/openpmd_api/DaskArray.py b/src/binding/python/openpmd_api/DaskArray.py index 0d2a1ec4ad..8c6fc3001b 100644 --- a/src/binding/python/openpmd_api/DaskArray.py +++ b/src/binding/python/openpmd_api/DaskArray.py @@ -9,12 +9,6 @@ import numpy as np -try: - from dask.array import from_array - found_dask = True -except ImportError: - found_dask = False - class DaskRecordComponent: # shape, .ndim, .dtype and support numpy-style slicing @@ -80,6 +74,13 @@ def record_component_to_daskarray(record_component): are used internally to parallelize reading dask.array : the (potentially distributed) array object created here """ + # Import dask here for a lazy import + try: + from dask.array import from_array + found_dask = True + except ImportError: + found_dask = False + if not found_dask: raise ImportError("dask NOT found. Install dask for Dask DataFrame " "support.") diff --git a/src/binding/python/openpmd_api/DaskDataFrame.py b/src/binding/python/openpmd_api/DaskDataFrame.py index fa18f7c076..926879fd66 100644 --- a/src/binding/python/openpmd_api/DaskDataFrame.py +++ b/src/binding/python/openpmd_api/DaskDataFrame.py @@ -7,18 +7,6 @@ """ import numpy as np -try: - import dask.dataframe as dd - from dask.delayed import delayed - found_dask = True -except ImportError: - found_dask = False -try: - import pandas # noqa - found_pandas = True -except ImportError: - found_pandas = False - def read_chunk_to_df(species, chunk): stride = np.s_[chunk.offset[0]:chunk.offset[0]+chunk.extent[0]] @@ -51,6 +39,19 @@ def particles_to_daskdataframe(particle_species): are used internally to parallelize particle processing dask.dataframe : the central dataframe object created here """ + # import here for lazy imports + try: + import dask.dataframe as dd + from dask.delayed import delayed + found_dask = True + except ImportError: + found_dask = False + try: + import pandas # noqa + found_pandas = True + except ImportError: + found_pandas = False + if not found_dask: raise ImportError("dask NOT found. Install dask for Dask DataFrame " "support.") @@ -86,4 +87,11 @@ def particles_to_daskdataframe(particle_species): ] df = dd.from_delayed(dfs) + # set a header for the first column (row index) + # note: this is NOT the particle id + # TODO both these do not work: + # https://github.com/dask/dask/issues/10440 + # df.index.name = "row" + # df.index = df.index.rename("row") + return df diff --git a/src/binding/python/openpmd_api/DataFrame.py b/src/binding/python/openpmd_api/DataFrame.py index d0e01acab8..1248136a5a 100644 --- a/src/binding/python/openpmd_api/DataFrame.py +++ b/src/binding/python/openpmd_api/DataFrame.py @@ -9,12 +9,6 @@ import numpy as np -try: - import pandas as pd - found_pandas = True -except ImportError: - found_pandas = False - def particles_to_dataframe(particle_species, slice=None): """ @@ -46,6 +40,13 @@ def particles_to_dataframe(particle_species, slice=None): are optimal arguments for the slice parameter pandas.DataFrame : the central dataframe object created here """ + # import pandas here for a lazy import + try: + import pandas as pd + found_pandas = True + except ImportError: + found_pandas = False + if not found_pandas: raise ImportError("pandas NOT found. Install pandas for DataFrame " "support.") @@ -66,4 +67,10 @@ def particles_to_dataframe(particle_species, slice=None): columns[column_name] = np.multiply( columns[column_name], rc.unit_SI) - return pd.DataFrame(columns) + df = pd.DataFrame(columns) + + # set a header for the first column (row index) + # note: this is NOT the particle id + df.index.name = "row" + + return df diff --git a/src/binding/python/openpmd_api/pipe/__main__.py b/src/binding/python/openpmd_api/pipe/__main__.py index c5312d71b5..b3bcc6f068 100644 --- a/src/binding/python/openpmd_api/pipe/__main__.py +++ b/src/binding/python/openpmd_api/pipe/__main__.py @@ -225,7 +225,7 @@ def run(self): sys.stdout.flush() # In Linear read mode, global attributes are only present after calling # this method to access the first iteration - inseries.read_iterations() + inseries.parse_base() self.__copy(inseries, outseries) def __copy(self, src, dest, current_path="/data/"): @@ -234,7 +234,7 @@ def __copy(self, src, dest, current_path="/data/"): Copies data from src to dest. May represent any point in the openPMD hierarchy, but src and dest must both represent the same layer. """ - if (type(src) != type(dest) + if (type(src) is not type(dest) and not isinstance(src, io.IndexedIteration) and not isinstance(dest, io.Iteration)): raise RuntimeError( diff --git a/test/CoreTest.cpp b/test/CoreTest.cpp index 496cb0a36f..1da1f39f2f 100644 --- a/test/CoreTest.cpp +++ b/test/CoreTest.cpp @@ -25,6 +25,8 @@ using namespace openPMD; +Dataset globalDataset(Datatype::CHAR, {1}); + TEST_CASE("versions_test", "[core]") { auto const apiVersion = getVersion(); @@ -439,11 +441,11 @@ TEST_CASE("record_constructor_test", "[core]") ps["position"][RecordComponent::SCALAR].resetDataset(dset); ps["positionOffset"][RecordComponent::SCALAR].resetDataset(dset); - REQUIRE(r["x"].unitSI() == 1); + REQUIRE(r["x"].resetDataset(dset).unitSI() == 1); REQUIRE(r["x"].numAttributes() == 1); /* unitSI */ - REQUIRE(r["y"].unitSI() == 1); + REQUIRE(r["y"].resetDataset(dset).unitSI() == 1); REQUIRE(r["y"].numAttributes() == 1); /* unitSI */ - REQUIRE(r["z"].unitSI() == 1); + REQUIRE(r["z"].resetDataset(dset).unitSI() == 1); REQUIRE(r["z"].numAttributes() == 1); /* unitSI */ std::array zeros{{0., 0., 0., 0., 0., 0., 0.}}; REQUIRE(r.unitDimension() == zeros); @@ -488,13 +490,15 @@ TEST_CASE("recordComponent_modification_test", "[core]") r["x"].setUnitSI(2.55999e-7); r["y"].setUnitSI(4.42999e-8); - REQUIRE(r["x"].unitSI() == static_cast(2.55999e-7)); + REQUIRE( + r["x"].resetDataset(dset).unitSI() == static_cast(2.55999e-7)); REQUIRE(r["x"].numAttributes() == 1); /* unitSI */ - REQUIRE(r["y"].unitSI() == static_cast(4.42999e-8)); + REQUIRE( + r["y"].resetDataset(dset).unitSI() == static_cast(4.42999e-8)); REQUIRE(r["y"].numAttributes() == 1); /* unitSI */ r["z"].setUnitSI(1); - REQUIRE(r["z"].unitSI() == static_cast(1)); + REQUIRE(r["z"].resetDataset(dset).unitSI() == static_cast(1)); REQUIRE(r["z"].numAttributes() == 1); /* unitSI */ } @@ -505,13 +509,13 @@ TEST_CASE("mesh_constructor_test", "[core]") Mesh &m = o.iterations[42].meshes["E"]; std::vector pos{0}; - REQUIRE(m["x"].unitSI() == 1); + REQUIRE(m["x"].resetDataset(globalDataset).unitSI() == 1); REQUIRE(m["x"].numAttributes() == 2); /* unitSI, position */ REQUIRE(m["x"].position() == pos); - REQUIRE(m["y"].unitSI() == 1); + REQUIRE(m["y"].resetDataset(globalDataset).unitSI() == 1); REQUIRE(m["y"].numAttributes() == 2); /* unitSI, position */ REQUIRE(m["y"].position() == pos); - REQUIRE(m["z"].unitSI() == 1); + REQUIRE(m["z"].resetDataset(globalDataset).unitSI() == 1); REQUIRE(m["z"].numAttributes() == 2); /* unitSI, position */ REQUIRE(m["z"].position() == pos); REQUIRE(m.geometry() == Mesh::Geometry::cartesian); @@ -534,9 +538,9 @@ TEST_CASE("mesh_modification_test", "[core]") Series o = Series("./MyOutput_%T.json", Access::CREATE); Mesh &m = o.iterations[42].meshes["E"]; - m["x"]; - m["y"]; - m["z"]; + m["x"].resetDataset(globalDataset); + m["y"].resetDataset(globalDataset); + m["z"].resetDataset(globalDataset); m.setGeometry(Mesh::Geometry::spherical); REQUIRE(m.geometry() == Mesh::Geometry::spherical); @@ -1031,13 +1035,16 @@ TEST_CASE("no_file_ending", "[core]") { REQUIRE_THROWS_WITH( Series("./new_openpmd_output", Access::CREATE), - Catch::Equals("Unknown file format! Did you specify a file ending?")); + Catch::Equals("Unknown file format! Did you specify a file ending? " + "Specified file name was './new_openpmd_output'.")); REQUIRE_THROWS_WITH( Series("./new_openpmd_output_%T", Access::CREATE), - Catch::Equals("Unknown file format! Did you specify a file ending?")); + Catch::Equals("Unknown file format! Did you specify a file ending? " + "Specified file name was './new_openpmd_output_%T'.")); REQUIRE_THROWS_WITH( Series("./new_openpmd_output_%05T", Access::CREATE), - Catch::Equals("Unknown file format! Did you specify a file ending?")); + Catch::Equals("Unknown file format! Did you specify a file ending? " + "Specified file name was './new_openpmd_output_%05T'.")); { Series( "../samples/no_extension_specified", diff --git a/test/ParallelIOTest.cpp b/test/ParallelIOTest.cpp index 5ace4a2cd4..038426e047 100644 --- a/test/ParallelIOTest.cpp +++ b/test/ParallelIOTest.cpp @@ -1606,8 +1606,9 @@ void append_mode( ++counter; } REQUIRE(counter == 8); - // Cannot do listSeries here because the Series is already drained - REQUIRE_THROWS_AS(helper::listSeries(read), error::WrongAPIUsage); + // listSeries will not see any iterations since they have already + // been read + helper::listSeries(read); } break; case ParseMode::AheadOfTimeWithoutSnapshot: { @@ -1747,9 +1748,9 @@ void append_mode( ++counter; } REQUIRE(counter == 8); - // Cannot do listSeries here because the Series is already - // drained - REQUIRE_THROWS_AS(helper::listSeries(read), error::WrongAPIUsage); + // listSeries will not see any iterations since they have already + // been read + helper::listSeries(read); } } #endif diff --git a/test/SerialIOTest.cpp b/test/SerialIOTest.cpp index 78316dd0cf..1631bd24aa 100644 --- a/test/SerialIOTest.cpp +++ b/test/SerialIOTest.cpp @@ -1,4 +1,6 @@ // expose private and protected members for invasive testing +#include "openPMD/Datatype.hpp" +#include "openPMD/IO/Access.hpp" #if openPMD_USE_INVASIVE_TESTS #define OPENPMD_private public: #define OPENPMD_protected public: @@ -81,10 +83,24 @@ std::vector testedFileExtensions() { auto allExtensions = getFileExtensions(); auto newEnd = std::remove_if( - allExtensions.begin(), allExtensions.end(), [](std::string const &ext) { + allExtensions.begin(), + allExtensions.end(), + []([[maybe_unused]] std::string const &ext) { +#if openPMD_HAVE_ADIOS2 +#define HAS_ADIOS_2_9 (ADIOS2_VERSION_MAJOR * 100 + ADIOS2_VERSION_MINOR >= 209) +#if HAS_ADIOS_2_9 + // sst and ssc need a receiver for testing + // bp5 is already tested via bp + return ext == "sst" || ext == "ssc" || ext == "bp5"; +#else // sst and ssc need a receiver for testing // bp4 is already tested via bp return ext == "sst" || ext == "ssc" || ext == "bp4"; +#endif +#undef HAS_ADIOS_2_9 +#else + return false; +#endif }); return {allExtensions.begin(), newEnd}; } @@ -215,6 +231,54 @@ TEST_CASE("adios2_char_portability", "[serial][adios2]") } #endif +namespace detail +{ +template +void writeChar(Series &series, std::string const &component_name) +{ + auto component = series.iterations[0].meshes["E"][component_name]; + std::vector data(10); + component.resetDataset({determineDatatype(), {10}}); + component.storeChunk(data, {0}, {10}); + series.flush(); +} +template +void readChar(Series &series, std::string const &component_name) +{ + auto component = series.iterations[0].meshes["E"][component_name]; + std::vector data(10); + auto chunk = component.loadChunk(); + series.flush(); + for (size_t i = 0; i < 10; ++i) + { + REQUIRE(data[i] == chunk.get()[i]); + } +} +} // namespace detail + +void char_roundtrip(std::string const &extension) +{ + Series write("../samples/char_rountrip." + extension, Access::CREATE); + ::detail::writeChar(write, "char"); + ::detail::writeChar(write, "uchar"); + ::detail::writeChar(write, "schar"); + write.close(); + + Series read("../samples/char_rountrip." + extension, Access::READ_ONLY); + ::detail::readChar(read, "char"); + ::detail::readChar(read, "uchar"); + ::detail::readChar(read, "schar"); + read.close(); +} + +TEST_CASE("char_roundtrip", "[serial]") +{ + for (auto const &t : testedFileExtensions()) + { + char_roundtrip(t); + } +} + void write_and_read_many_iterations( std::string const &ext, bool intermittentFlushes) { @@ -2800,6 +2864,85 @@ TEST_CASE("git_hdf5_sample_structure_test", "[serial][hdf5]") #endif } +namespace +{ +struct LoadDataset +{ + template + static void call(RecordComponent &rc) + { + auto chunk = rc.loadChunk(); + rc.seriesFlush(); + } + + static constexpr char const *errorMsg = "LoadDataset"; +}; +} // namespace + +TEST_CASE("git_hdf5_legacy_picongpu", "[serial][hdf5]") +{ + try + { + Series o = Series( + "../samples/git-sample/legacy/simData_%T.h5", Access::READ_ONLY); + + /* + * That dataset was written directly via HDF5 (not the openPMD-api) + * and had two issues: + * + * 1) No unitSI defined for numParticles and numParticlesOffset. + * unitSI does not really make sense there, but the openPMD-standard + * is not quite clear if it is required, so the API writes it and + * also required it. We will keep writing it, but we don't require + * it any longer. + * 2) A custom enum was used for writing a boolean dataset. + * At the least, the dataset should be skipped in parsing instead + * of failing the entire procedure. Ideally, the custom datatype + * should be upcasted to char type and treated as such. + */ + + auto radiationMask = + o.iterations[200] + .particles["e"]["radiationMask"][RecordComponent::SCALAR]; + switchNonVectorType( + radiationMask.getDatatype(), radiationMask); + + auto particlePatches = o.iterations[200].particles["e"].particlePatches; + REQUIRE(particlePatches.size() == 4); + for (auto key : {"extent", "offset"}) + { + REQUIRE(particlePatches.contains(key)); + REQUIRE(particlePatches.at(key).size() == 3); + for (auto subkey : {"x", "y", "z"}) + { + REQUIRE(particlePatches.at(key).contains(subkey)); + // unitSI is present in those records + particlePatches.at(key).at(subkey).unitSI(); + } + } + for (auto key : {"numParticles", "numParticlesOffset"}) + { + REQUIRE(particlePatches.contains(key)); + REQUIRE(particlePatches.at(key).contains(RecordComponent::SCALAR)); + // unitSI is not present in those records + REQUIRE_THROWS_AS( + particlePatches.at(key).at(RecordComponent::SCALAR).unitSI(), + no_such_attribute_error); + } + + helper::listSeries(o, true, std::cout); + } + catch (error::ReadError &e) + { + if (e.reason == error::Reason::Inaccessible) + { + std::cerr << "git sample not accessible. (" << e.what() << ")\n"; + return; + } + throw; + } +} + TEST_CASE("git_hdf5_sample_attribute_test", "[serial][hdf5]") { try @@ -4296,11 +4439,15 @@ void adios2_bp5_flush(std::string const &cfg, FlushDuringStep flushDuringStep) TEST_CASE("adios2_bp5_flush", "[serial][adios2]") { + if (auxiliary::getEnvString("OPENPMD_BP_BACKEND", "") == "ADIOS1") + return; + std::string cfg1 = R"( [adios2] [adios2.engine] -usesteps = true +# Check that BP5 can also be used without steps +usesteps = false type = "bp5" preferred_flush_target = "disk" @@ -5118,7 +5265,7 @@ void variableBasedSingleIteration(std::string const &file) writeSeries.iterationEncoding() == IterationEncoding::variableBased); auto iterations = writeSeries.writeIterations(); - auto iteration = writeSeries.iterations[0]; + auto iteration = iterations[0]; auto E_x = iteration.meshes["E"]["x"]; E_x.resetDataset(openPMD::Dataset(openPMD::Datatype::INT, {1000})); std::vector data(1000, 0); @@ -5128,7 +5275,8 @@ void variableBasedSingleIteration(std::string const &file) } { - Series readSeries(file, Access::READ_ONLY); + Series readSeries(file, Access::READ_LINEAR); + readSeries.parseBase(); auto E_x = readSeries.iterations[0].meshes["E"]["x"]; REQUIRE(E_x.getDimensionality() == 1); @@ -5450,20 +5598,27 @@ TEST_CASE("git_adios2_sample_test", "[serial][adios2]") } } -void variableBasedSeries(std::string const &file) +void variableBasedSeries( + std::string const &file, std::string const &backendSelection) { - std::string selectADIOS2 = R"({"backend": "adios2"})"; constexpr Extent::value_type extent = 1000; { - Series writeSeries(file, Access::CREATE, selectADIOS2); + Series writeSeries(file, Access::CREATE, backendSelection); writeSeries.setAttribute("some_global", "attribute"); writeSeries.setIterationEncoding(IterationEncoding::variableBased); REQUIRE( writeSeries.iterationEncoding() == IterationEncoding::variableBased); auto iterations = writeSeries.writeIterations(); + bool is_not_adios2 = writeSeries.backend() != "ADIOS2"; for (size_t i = 0; i < 10; ++i) { + if (i > 0 && is_not_adios2) + { + REQUIRE_THROWS_AS( + iterations[i], error::OperationUnsupportedInBackend); + return; + } auto iteration = iterations[i]; auto E_x = iteration.meshes["E"]["x"]; E_x.resetDataset({openPMD::Datatype::INT, {1000}}); @@ -5501,20 +5656,25 @@ void variableBasedSeries(std::string const &file) } } - REQUIRE(auxiliary::directory_exists(file)); + REQUIRE( + (auxiliary::directory_exists(file) || auxiliary::file_exists(file))); - auto testRead = [&file, &extent, &selectADIOS2]( + auto testRead = [&file, &extent, &backendSelection]( std::string const &jsonConfig) { /* * Need linear read mode to access more than a single iteration in * variable-based iteration encoding. */ Series readSeries( - file, Access::READ_LINEAR, json::merge(selectADIOS2, jsonConfig)); + file, + Access::READ_LINEAR, + json::merge(backendSelection, jsonConfig)); + + bool is_adios2 = readSeries.backend() == "ADIOS2"; size_t last_iteration_index = 0; REQUIRE(!readSeries.containsAttribute("some_global")); - readSeries.readIterations(); + readSeries.parseBase(); REQUIRE( readSeries.getAttribute("some_global").get() == "attribute"); @@ -5575,7 +5735,7 @@ void variableBasedSeries(std::string const &file) last_iteration_index = iteration.iterationIndex; } - REQUIRE(last_iteration_index == 9); + REQUIRE(last_iteration_index == (is_adios2 ? 9 : 0)); }; testRead("{\"defer_iteration_parsing\": true}"); @@ -5585,7 +5745,16 @@ void variableBasedSeries(std::string const &file) #if openPMD_HAVE_ADIOS2 TEST_CASE("variableBasedSeries", "[serial][adios2]") { - variableBasedSeries("../samples/variableBasedSeries.bp"); + for (auto const &[backend_name, t] : testedBackends()) + { + if (backend_name == "adios1") + { + continue; + } + variableBasedSeries( + "../samples/variableBasedSeries." + t, + R"({"backend": ")" + backend_name + R"("})"); + } } #endif @@ -5665,6 +5834,9 @@ void variableBasedParticleData() TEST_CASE("variableBasedParticleData", "[serial][adios2]") { + if (auxiliary::getEnvString("OPENPMD_BP_BACKEND", "") == "ADIOS1") + return; + variableBasedParticleData(); } #endif @@ -5673,6 +5845,9 @@ TEST_CASE("variableBasedParticleData", "[serial][adios2]") #ifdef ADIOS2_HAVE_BZIP2 TEST_CASE("automatically_deactivate_span", "[serial][adios2]") { + if (auxiliary::getEnvString("OPENPMD_BP_BACKEND", "") == "ADIOS1") + return; + // automatically (de)activate span-based storeChunking { Series write("../samples/span_based.bp", Access::CREATE); @@ -6031,7 +6206,7 @@ void adios2_bp5_no_steps(bool usesteps) IO.DefineAttribute("/openPMD", std::string("1.1.0")); IO.DefineAttribute("/openPMDextension", uint32_t(0)); IO.DefineAttribute("/software", std::string("openPMD-api")); - IO.DefineAttribute("/softwareVersion", std::string("0.15.1-dev")); + IO.DefineAttribute("/softwareVersion", std::string("0.15.2")); IO.DefineAttribute("/data/0/dt", double(1)); IO.DefineAttribute( @@ -6088,6 +6263,9 @@ void adios2_bp5_no_steps(bool usesteps) TEST_CASE("adios2_bp5_no_steps", "[serial][adios2]") { + if (auxiliary::getEnvString("OPENPMD_BP_BACKEND", "") == "ADIOS1") + return; + adios2_bp5_no_steps(/* usesteps = */ false); adios2_bp5_no_steps(/* usesteps = */ true); } @@ -6469,11 +6647,19 @@ void unfinished_iteration_test( */ it5.setAttribute("__openPMD_internal_fail", "asking for trouble"); auto it10 = write.writeIterations()[10]; + Dataset ds(Datatype::INT, {10}); auto E_x = it10.meshes["E"]["x"]; auto e_density = it10.meshes["e_density"][RecordComponent::SCALAR]; auto electron_x = it10.particles["e"]["position"]["x"]; auto electron_mass = it10.particles["e"]["mass"][RecordComponent::SCALAR]; + + RecordComponent *resetThese[] = { + &E_x, &e_density, &electron_x, &electron_mass}; + for (RecordComponent *rc : resetThese) + { + rc->resetDataset(ds); + } } auto tryReading = [&config, file, encoding]( Access access, @@ -6869,8 +7055,9 @@ void append_mode( ++counter; } REQUIRE(counter == 8); - // Cannot do listSeries here because the Series is already drained - REQUIRE_THROWS_AS(helper::listSeries(read), error::WrongAPIUsage); + // listSeries will not see any iterations since they have already + // been read + helper::listSeries(read); } break; case ParseMode::AheadOfTimeWithoutSnapshot: { @@ -6905,7 +7092,9 @@ void append_mode( * should see both instances when reading. * Final goal: Read only the last instance. */ - REQUIRE_THROWS_AS(helper::listSeries(read), error::WrongAPIUsage); + // listSeries will not see any iterations since they have already + // been read + helper::listSeries(read); } break; } @@ -7005,9 +7194,9 @@ void append_mode( ++counter; } REQUIRE(counter == 8); - // Cannot do listSeries here because the Series is already - // drained - REQUIRE_THROWS_AS(helper::listSeries(read), error::WrongAPIUsage); + // listSeries will not see any iterations since they have already + // been read + helper::listSeries(read); } } #endif