segfault with HDF5 chunked file #7795
Comments
|
Just to add some info, I found that the segfault happens when the file is located on the Lustre file system. The same code works fine if the file is stored on a local disk, or on a BeeGFS share. Can this have something to do with Lustre integration / version / etc? Does anyone have suggestions on how to debug this? |
|
I'm afraid I don't know much about HDF. @edgargabriel any insight into this? Does the same problem happen with OMPIO? |
|
@jsquyres Perfect! thanks, |
|
ROMIO is an import of MPI-IO functionality from MPICH. Originally, ROMIO was a standalone MPI-IO library written at Argonne (back in the early days of MPI-2 when MPI-IO was new). It eventually got slurped up into MPICH itself. But ever since it was created, ROMIO was slurped up into other MPI implementations too -- such as Open MPI. We've continued to import newer versions of ROMIO from MPICH over the years. I don't remember offhand which version of ROMIO we have, but perhaps it's got a bug in this case. OMPIO is our own, native MPI-IO implementation -- wholly separate from ROMIO. It was spearheaded by Dr. Edgar Gabriel at U. Houston (i.e., @edgargabriel). OMPIO is Open MPI's default MPI-IO these days, except in a few cases (I don't remember which cases offhand, sorry!). Put simply: OMPI vs. ROMIO is just another run-time plugin/component decision in Open MPI, just like all the others. 😄 We tend to prefer OMPIO 😉, but we keep ROMIO because of its age, maturity, and simply because some people/apps have a preference and/or established/verified compatibility with it. |
|
@jsquyres Thanks a lot, that's good to know! I run 4.0.3, which seems to use I guess I will simply change that in |
|
@jsquyres and some more info: I checked OpenMPI 3.1.4 with |
|
I am not sure I have much to contribute to this discussion, I haven't seen this bug yet with romio321. Generally speaking, romio is used by default on Lustre file system (that's why it has a higher priority in this case), and ompio basically everywhere else. That being said, ompio does have support for Lustre as well, and we are working on some interesting features that if they work out the way we hope, we can also switch on Lustre to ompio. |
|
didn't get fixed in 4.0.4 |
|
This sounds like a bug we fixed in ROMIO at some point in the last four years, but I haven't waded through the history to find what might be the fix. I would love to see a |
|
For reference: I just encountered the same error on a system with GPFS filesystem as well, with OpenMPI 4.1.0-rc1. |
|
Ran into what seems like this bug when trying to build hdf5 1.12.0 on a system with OpenMPI 4.0.5. System has Lustre (2.12.4.1_cray_139_g0763d21) and the backtrace is exactly like above from H5PB_write on. This happens during the test finishes successfully. Unfortunately I lack information to make a debug build of OpenMPI on that system that would exactly match the system version but I'll try to get more information out of the person who installed that package. |
|
@tjahns not sure whether it is relevant for your work or not, but note that ompio is now the default even on Lustre file systems starting from the 4.1.x release. The romio component in Open MPI will also be updated to resolve the issues, but I am not 100% sure on what the status of this effort is. |
|
While OMPIO resolves this issue, we haven't finished the ROMIO update yet because we're waiting for some fixes from upstream. See #8371. |
|
Just one more item to resolve: pmodels/mpich#5101 thanks for your patience |
|
Not going to update ROMIO in v4.0.x. Removing label. |
|
Moved the milestone to 5.0 and removed the 4.1.x label; given that OMPIO works for this use case, we're not going to backport significant ROMIO changes into 4.1.x at this point. |
* Document ROMIO/HDF5/Chunking issue open-mpi/ompi#7795
* Fix gcc9 warning the implicitly-defined constructor does not initialize 'openPMD::Datatype openPMD::detail::BufferedUniquePtrPut::dtype' * More careful documentation of streaming API * Doc: Fix Bib Authors Make sure the bib authors match the quoted openPMD-standard authors. * Update .readthedocs.yml Update to newer Ubuntu, shipping a newer OpenSSL * Fix deprecated storeChunk APIs in first read/write examples * CI: macOS-11 Update The older macOS image is now removed. The latest points already to macOS-12. macoS-13 runners are marked experimental. https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources * Docs: Linking to C++ Projects Move a section only written in the README to our developer section on readthedocs. * Use lazy imports for dask and pandas * Better error message when loading to a buffer with mismatched type * Remove schema 2021 from documentation It will be removed and should no longer be advertised. * Document ROMIO/HDF5/Chunking issue open-mpi/ompi#7795 * Use a "minor" RST link Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * HDF5: HDF5_DO_MPI_FILE_SYNC Document a new work-around option for MPI-parallel HDF5 for filesystems that are super limited for paralle I/O features relevant in HPC. * Fix typo Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com> * openpmd-pipe: set correct install permissions * Fix headers in workflow.rst * Fix documentation for preferred_flush_target in ADIOS2 * HDF5: Fix Char Type Matching In HDF5, there are only the signed and unsigned char type. The third `char` type is an alias to one or the other, different to the C/C++ fundamental types. This tries to fix the type matching order to be platform independent between ppc64le/aarch64/x86-64. * Add failing HDF5 test * loadChunk(): consider equivalent char types for casting * Doc strings * newline & comment * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Doc: Sphinx Copybutton and Design Add the Sphinx extensions `copybutton` (copy code blocks with a single click) and `design` (for boxes, tabs, dropdowns, etc.). * CI: Doxygen 1.9.7 Broken Markdown support for main file broken in 1.9.7. Go back to previous patch release. Upstream already fixed. * Docs: Analysis Add a data analysis & visualization section. This is meant to show entry points and workflows to work with openPMD data in larger frameworks and compatible ecosystems. * [Draft] DASK, Pandas, ... * Doc: DASK * Pandas * RAPIDS * Typos * Don't require unitSI when reading patch record component * CI: oneAPI 2023.2.0 Update CI to breaking `apt` changes in the latest oneAPI release. * Update __repr__ method of major objects in openPMD hierarchy * Deal with trailing slashes in file paths Happens in bash completion on BP files since they are folders. * Throw better error messages when selecting a bad backend * Adapt CoreTest to new error message * Follow-up to #1470 Changes there left the PatchRecordComponent dirty after parsing * HDF5IOHandler: Support for float128 data types with 80bit precision on ARM64/PPC64 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Cleanup Unify little/big endian, use double80 also in other places Long double size 8 -> 16 Use malloc to avoid alignment issues Same treatment for complex long double Add this for readAttribute too Avoid non-native datatypes in writing * Make this fix little-endian only * Add comment * Suggestions from review * Use new instead of malloc everywhere Also, add a more explanative comment * CI fix: type comparison in Python No idea why this check is triggered by this PR * ADIOS2: Ensure that a step is always active at write time Even when not using steps, dump everything into a single large step. BP5 will fail otherwise. * CI: macOS 11.0+ We do not have the runners anymore to deploy and test macOS 10.15, so we bump our tests to 11.0+, too. Technically, we could build & deploy a bit longer on GH actions `macos-11` for `10.15` (Catalina), but since it is unmaintained by Apple/end-of-life already and macOS-11 is the oldest still maintained OS by Apple, we do also stop support for it. At the time of writing, old and maintained are: - macOS 11, Big Sur - macOS 12, Monterey latest is: macOS 13, Ventura - and in preview is macOS 14, Sonoma. This also unifies our arm64/aarch64 (M1/M2) requirements, which are macOS 11.0+ as well. * HDF5: Throw ReadError at dataset open * Try parent types when a dataset datatype is unknown * Update Sample Download Scripts * Add test for old HDF5-plugin written openPMD dataset from PIConGPU * Remove debugging comments * Warn on BP5+Blosc in ADIOS2 v2.9 up to patch level 1 * Fix unused parameter * Update Warning String * Throw error when steps are needed but not supported * Fix variableBasedSingleIteration test * Test that the error is correctly thrown * Introduce Series::parseBase alias for readIterations(), fix workflow * Have only one instance of SeriesIterator * Break memory cycle * Use simple API in test again * Snapshot attribute in file-based encoding Snapshot attribute must be written in Iteration::endStep() in file-based encoding * Optional debugging output for AbstractIOHandlerImpl::flush() * Revert "Optional debugging output for AbstractIOHandlerImpl::flush()" This reverts commit ee8de45. * Post-rebase fixes 1) Don't write snapshot attributes during initialization of an iteration 2) Catch unsuccessful flush run also in beginStep() * Ensure that m_lastFlushSuccessful is always called * Pandas DataFrames: Add Row Column Name By default, the row index (!= particle index) in a pandas dataframe has no name. This can be a bit cumbersome for exports, e.g., to CSV - where this header field would just be empty. This PR names the index now "row", because it is not a (macro) particle id property. * Python: 3.8+ Python 3.7 went EOL last month. This bumps our supported versions to 3.8+. * Optional debugging output for AbstractIOHandlerImpl::flush() * Add an environment variable for this * Version 0.15.2 + Changelog * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Python: Fix Series Compile Fix compile issue introduced during backporting. * Unmerge 0.16 content from variableBasedSeries test * Reintroduce backend selection in variableBasedSeries test Still needed due to ADIOS1 BP env variable * Add src/Dataset.cpp to ADIOS1 source * Replace openPMD_Datatypes global with function * Windows CI: Bump Version to 0.15.2 * OPENPMD_BP_BACKEND=="ADIOS1": Skip BP5 Tests * Streaming examples: Set WAN as default transport * Exclude ADIOS1 from variableBasedSeries test * Changelog: Bump Date * replace extent in weighting and displacement store extent value in n_particles * Examples: Fix Types of Constants & Attributes Backports from #1316 and #1510 * Changelog: Streaming Example Note * CMake: Warn and Continue on Empty HDF5_VERSION Seen on Conda-Forge for arm64 on macOS for HDF5 1.14.1 --------- Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com> Co-authored-by: Dave Grote <grote1@llnl.gov> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ulrik Guenther <hello@ulrik.is> Co-authored-by: Kara-Mostefa Ilian <ilian.kara-mostefa@student-cs.fr>
I am looking at OpenMPI 4.0.3 and HDF5 1.10.6 compiled against it. A user reported segfault in
ADIOI_Flatten()when using a chunked dataset, i.e., when the following line is executed:A simple FORTRAN reproducer is attached (compile with
h5pfc ioerror.F90, run withmpirun -np2 ./a.out). The same code works with Intel MPI. Here is the stack:Could that be an OpenMPI problem, or do you think it is HDF5 that's causing it?
I'd appreciate any help! thanks!
ioerror.zip
The text was updated successfully, but these errors were encountered: