Lassen: Fix Chunked HDF5 with MPI #2863
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Try to work-around segfaults with HDF5 when running on more than one node.
EZoni
approved these changes
Feb 17, 2022
|
Orthogonal to this, we see the following IBM MPI error now: Backtrace: |
|
For the last issue, we could try to use export OMPI_MCA_coll_ibm_skip_barrier=trueto mitigate collectives as we do on Summit, tests pending. On Summit, the problem was with barriers, here it's an AllGather it seams. Maybe there is a separate flag for that one. |
|
@joshualudwig8 can you please try with the latest WarpX if setting this environment variable helps with the HDF5 crashes on Lassen? |
|
Actually, I found a guide: https://www.ibm.com/docs/en/SSZTET_EOS/eos/guide_101.pdf Let's try this: export OMPI_MCA_coll_ibm_skip_allgatherv=true |
|
Awesome, will open another PR to document this! Buggy IBM implementations everywhere, hah. |
|
Documentation update incoming via #2874 |
roelof-groenewald
added a commit
to ModernElectron/WarpX
that referenced
this pull request
Feb 19, 2022
* NCIGodfreyFilter: Fix Int Division (ECP-WarpX#2837) * NCIGodfreyFilter: Fix Int Division `m_cdtodz` is between 0 and 1, and we interpolate a set of coefficients from a table. * reset benchmarks Co-authored-by: Tools <warpx@lbl.gov> * Adding documentation for lxplus (ECP-WarpX#2756) * Adding documentation for lxplus * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Adding the new documentation * Apply suggestions from code review Some suggestions from the code review. Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Removing suggestion about miniconda * Switched to using anonymous environment * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixing the architecture * Apply suggestions from code review Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update Tools/machines/lxplus-cern/spack.yaml Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Updating env to use pre-installed openmpi * Moving a file * Docs: Now using GCC 11.2.0 * Add OpenMPI Version in Spec * Add CPU target architecture note * One more GCC 9.2.0->11.2.0 Update * Finalize Spack Stack Setup Notes * Move AFS Spack Config to Environment File * Comment on Variations of the Spack Env * Improve git clone and spack activation * Apply suggestions from code review Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Fixing ncurses Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Fix: Spack Concretization (CUDA/Python) Somehow, this concretizes the variants not properly otherwise. * Not using the preinstalled openmpi anymore * updated lxplus.rst * added lxplus_warpx.profile.example * Apply suggestions from code review Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Apply suggestions from code review Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Updated documentation * Added a precisation about warpx.profile * Apply suggestions from code review Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Add missing empty newline Co-authored-by: lgiacome <lorenzo.giacome@cern.ch> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * BackTransformParticleFunctor: Unused Counter (ECP-WarpX#2840) Remove an unused counter in BTD particle filtering. Seen first with a HIP diagnostics. * ABLASTR: particle weights `const` (ECP-WarpX#2838) * ABLASTR: particle weights `const` We can declare the particle weights `const` because we don't change values in them during deposition. * DepositCharge: `const`-ify usage * Rename ngE as ngEB (used for E,B) (ECP-WarpX#2841) * Fix some offsets with the gather buffers (ECP-WarpX#2847) * Add amrex REPO and BRANCH flags for python builds (ECP-WarpX#2845) * Add WarpX_amrex_repo and _branch options to Python In python setup, environment variables WARPX_AMREX_REPO and WARPX_AMREX_BRANCH will now set these variables * Update documentation with new compile envvars * User-defined integer and real particle attributes (ECP-WarpX#2735) * define user attributes, parse them, initialize with respective parsers * fix warning by using static_cast for int attribute as parser returns real * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * clean-up from self-review * adding dimensionless velocity, gamma*v/c and time to parser argument * add documentation * typo in comment * unused var * device vector for kernels * particle attribute in developer doc * data ptr for device vector * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ignore_unused * Docs: Describe all particle attributes including pre-defined ones :) * Docs: Fix formatting (user params) * Add: 1D and RZ Support * Docs: Fix Typo in Function Declaration * Laser-Ion Example: User-Defined Attrib. Add two user-defined attributes to the laser-ion acceleration example. This is a 2D test. Documents the name in the table of commonly used, user-defined attribute names. The attribute added is the original position of particles, which I like to plot in "potential" plots that correlate original position in the target with final energy. * changing user-interface API with .attribute. and no need for separate 1D 2D 3D RZ code for parser. pos.x/y/z returns the right values * Adding 1D, 3D, and rz tests * attribute in inputs * at(i) * refinining names for inputs for laser ion and acceleration tests * typo in input * reset benchmarks for test-cases that included attributes Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Don't cut all particles in a Gaussian beam when x_rms=0 (ECP-WarpX#2844) * [pre-commit.ci] pre-commit autoupdate (ECP-WarpX#2851) updates: - [github.com/Lucas-C/pre-commit-hooks: v1.1.11 → v1.1.12](Lucas-C/pre-commit-hooks@v1.1.11...v1.1.12) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Use parser to read laser spatio-temporal couplings direction (ECP-WarpX#2843) * Only set modified k to 0 for even number of points (ECP-WarpX#2852) * Only set modified k to 0 for even number of points * Update Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp * Allow flux injection in the out-of-plane direction for RZ/2D geometry (ECP-WarpX#2788) * Implement injection orthogal to plane * Generalize momentum distribution for flux injection * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" This reverts commit b0cd189. * Revert "Generalize momentum distribution for flux injection" This reverts commit 0a22b1d. * Rotate momentum initialization * Correct flux number when the direction is normal to plane * Update distribution of particles within a cell * Clean-up injection code * Add more documentation * Add more comments * Handle 1D case * Only do the rotation for Gaussian flux profile * Fix compilation error * Correct compilation for GPU * Start adding automated test * Correct sign of velocity * Update to add continuous injection * Finalize test * Correct processing of flux_normal_axis * Add checksum * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix bug * Update script * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update checksum Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * AMReX/PICSAR: Weekly Update (ECP-WarpX#2849) * AMReX: Update latest commit * AMReX: Update to 76d08651adb987e3fba6b232e806c5e7c365a8d9 * update CI to use ascent 0.8.0 release container (ECP-WarpX#2858) * use ascent 0.8.0 release container * try again * restore * use new install loc * Correct typo in the relativistic Poisson solver (ECP-WarpX#2853) * Correct typo in the relativistic Poisson solver * Fix unused variable * Update benchmark * Gaussian particle beam: add error message when using y_rms = 0 in 2D (ECP-WarpX#2862) * ParticleBuffer: Generalize & Move (ECP-WarpX#2860) * ParticleBuffer: Generalize & Move - move the `ParticleBuffer` to ABLASTR - generalize the API - remove `amr_core` argument - use more semantic naming - add docs * Use `amrex::ParticleContainer::make_alike` * Update AMREX to include AMReX-Codes/amrex#2630 * ABLASTR: Refactor `deposit_charge` API (ECP-WarpX#2856) Simplified and re-ordered interface for `ablastr::particles::deposit_charge`. * Provide `t_min` and `t_max` for flux injection (ECP-WarpX#2842) * Implement injection orthogal to plane * Generalize momentum distribution for flux injection * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks" This reverts commit b0cd189. * Revert "Generalize momentum distribution for flux injection" This reverts commit 0a22b1d. * Rotate momentum initialization * Correct flux number when the direction is normal to plane * Update distribution of particles within a cell * Clean-up injection code * Add more documentation * Add more comments * Handle 1D case * Only do the rotation for Gaussian flux profile * Fix compilation error * Correct compilation for GPU * Start adding automated test * Correct sign of velocity * Update to add continuous injection * Finalize test * Correct processing of flux_normal_axis * Add checksum * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix bug * Update script * Implement maximum injection time * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add parameter tmin * Make parameter optional ; update documentation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Lassen: Fix Chunked HDF5 with MPI (ECP-WarpX#2863) Try to work-around segfaults with HDF5 when running on more than one node. * Refactor Current Correction Functions (ECP-WarpX#2839) * Refactor Current Correction Functions * Clean Up, Reset Benchmark * Rotate momentum for RZ flux injection (ECP-WarpX#2867) * Add warning to FieldProbe re: Boosted Frame (ECP-WarpX#2868) * Add warning to FieldProbe re: Boosted Frames Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update parsing of FieldProbe in 2D and 1D (ECP-WarpX#2818) * Update parsing of FieldProbe in 2D and 1D * Fix unused variables * Allow plane probe in 2D and line probe in 1D * doc update * Do Not Fill PML Guard Cells w/ Inverse FFTs (ECP-WarpX#2854) * Fix number of guard cell for coarse patch (ECP-WarpX#2869) * openPMD: Add ADIOS2 Engine Parameter Control (ECP-WarpX#2872) * Adds support for ADIOS engines ECP-WarpX#2866 Input file can now have lines like diag1.adios2_engine.parameters.NumAggregators=2 * Docs for ADIOS engine type and parameters ECP-WarpX#2866 * Aesthetic edit in adios2 engine documentation ECP-WarpX#2866 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removed debug print statement ECP-WarpX#2866 * Style Updates Co-authored-by: Mehta, Kshitij V <kshitij-v-mehta@github.com> Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Add `PHistDiag` for scraping (#153) * refactor of surface flux diagnostic handling before implementing PHistDiag * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * initial commit of `ParticleHistDiag` * updated changelog and version number * added plotting functionality specifically for ZPlane assemblies * save histogram binning details to file as well * code cleanup * changes requested during PR review * add comment about 2d plotting * remove debugging print statement * further code changes from PR review * Fix typo Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com> * fix another typo Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com> * Add initialization of pairwise Coulomb collisions (#155) * refactor of surface flux diagnostic handling before implementing PHistDiag * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * initial commit of `ParticleHistDiag` * updated changelog and version number * added plotting functionality specifically for ZPlane assemblies * save histogram binning details to file as well * code cleanup * changes requested during PR review * add comment about 2d plotting * added Coulomb collision installation to picmi.py * added pairwise Coulomb collision initialization * remove debugging print statement * further code changes from PR review * Fix typo Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com> * fix docstring * cleaned up the logging message for Coulomb scattering Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com> Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Co-authored-by: Tools <warpx@lbl.gov> Co-authored-by: Lorenzo Giacomel <47607756+lgiacome@users.noreply.github.com> Co-authored-by: lgiacome <lorenzo.giacome@cern.ch> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Co-authored-by: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com> Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com> Co-authored-by: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> Co-authored-by: Remi Lehe <remi.lehe@normalesup.org> Co-authored-by: Cyrus Harrison <cyrush@llnl.gov> Co-authored-by: Tiberius Rheaume <35204125+TiberiusRheaume@users.noreply.github.com> Co-authored-by: Kshitij Mehta <kshitij-v-mehta@users.noreply.github.com> Co-authored-by: Mehta, Kshitij V <kshitij-v-mehta@github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Try to work-around segfaults with HDF5 when running on more than one node.
I think that is caused by: open-mpi/ompi#7795