Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MergeBuffersForPlotfile: Barrier #2608

Merged
merged 1 commit into from
Dec 1, 2021
Merged

MergeBuffersForPlotfile: Barrier #2608

merged 1 commit into from
Dec 1, 2021

Conversation

ax3l
Copy link
Member

@ax3l ax3l commented Nov 30, 2021 •
edited
Loading

Make sure that all MPI ranks are in sync, i.e., have closed the files that they wrote, before trying to merge them.

Fix #2382

@ax3l ax3l added bug Something isn't working bug: affects latest release Bug also exists in latest release version component: diagnostics all types of outputs labels Nov 30, 2021
@ax3l ax3l force-pushed the fix-MergeBuffersForPlotfileBarrier branch from f929776 to ab702a8 Compare November 30, 2021 18:44
ax3l
ax3l commented Nov 30, 2021
View reviewed changes
Source/Diagnostics/BTDiagnostics.cpp
// Note: additionally, since a Barrier does not guarantee a FS sync
// on a parallel FS, we might need to add timeouts and re-tries
// to the open calls below when running at scale.
amrex::ParallelDescriptor::Barrier();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RevathiJambunathan are we guaranteeing that all MPI ranks reach this point? :)
Otherwise we could call a barrier on a sub-communicator, if needed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the I/O for BTD is done within amrex:ParallelContext::IOProcessorSub() for the cases when we have only some processors dedicated to I/O

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also amrex::ParallelContext::BarrierSub() for that.

Copy link
Member Author

@ax3l ax3l Nov 30, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, cool!
So that means the current PR proposal should work (sync all that potentially wrote a plotfile).

Btw, is there a specific reason for the barrier at the end of BTDiagnostics::MergeBuffersForPlotfile?

@ax3l
MergeBuffersForPlotfile: Barrier
e5d6236 
Make sure that all MPI ranks are in sync, i.e., have closed the
files that they wrote, before trying to merge them.
@ax3l ax3l force-pushed the fix-MergeBuffersForPlotfileBarrier branch from ab702a8 to e5d6236 Compare November 30, 2021 18:46
@ax3l ax3l enabled auto-merge (squash) December 1, 2021 05:52
RevathiJambunathan
RevathiJambunathan approved these changes Dec 1, 2021
View reviewed changes
Copy link
Member

@RevathiJambunathan RevathiJambunathan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Axel for this PR!

@ax3l ax3l merged commit 882e19b into ECP-WarpX:development Dec 1, 2021
@ax3l ax3l deleted the fix-MergeBuffersForPlotfileBarrier branch December 1, 2021 06:35
@ax3l ax3l mentioned this pull request Dec 1, 2021
Closed
roelof-groenewald added a commit to ModernElectron/WarpX that referenced this pull request Dec 3, 2021
@roelof-groenewald @ax3l
@EZoni @RemiLehe @lgiacome @KZhu-ME @peterscherpelz @dpgrote @PhilMiller @s9105947 @tropf @prkkumar
Update memaster (#110)
033b51e 
* Fix Init of Vector Members (ECP-WarpX#2595)

Fix default init of `Vector` member variables. The old construct
is not valid C++.

https://stackoverflow.com/a/11491003/2719194

* C++17: Work-Around NVCC gatherParticles (ECP-WarpX#2596)

The `noexcept` lambda does not compile in C++17 mode due to an NVCC compiler bug, at least in NVCC 11.3.109. Compiles in C++14 mode with the same compiler.

* requirements.txt - PICMI development version (ECP-WarpX#2588)

Document in `requirements.txt` on how to install a pre-release
version of PICMI.

* CONTRIBUTING: Update/Modernize (ECP-WarpX#2600)

* CONTRIBUTING: Update/Modernize

- Add GitHub account setup
- Add local git setup
- Modernize for CMake workflows

* Apply suggestions by Edoardo

Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>

* Replaced duplicated current deposition documentation (ECP-WarpX#2604)

* Throwing a warning if particle_shape>1 with EB (ECP-WarpX#2592)

* Aborting if particle_shape!=1 with EB

* Throw warning instead of aborting

* Checking at runtime if EB is initialized

* Added missing preprocessor directive

* Ignoring an unused variable

* Fix typo

* Improve style

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Fix segfault when importing _libwarpx without initializing WarpX (ECP-WarpX#2580)

* Added check for if warpx was initialized when calling finalize

* Renamed to be warpx_initialized

* Fixed reference to global variable

Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com>

* Changed global variable to member of libwarpx

* Fixed syntax errors

* Remove custom arg from argv to avoid parmparse error

Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com>

* Added parallel pragma to ApplyBoundaryConditions (ECP-WarpX#2612)

* Note that CCache 4.2 introduced large CUDA improvements (ECP-WarpX#2606)

* Dimensionality Docs: Default (ECP-WarpX#2609)

Just adds the note that 3D is the default geometry.

* AMReX: Weekly Update (ECP-WarpX#2613)

* MergeBuffersForPlotfile: Barrier (ECP-WarpX#2608)

Make sure that all MPI ranks are in sync, i.e., have closed the
files that they wrote, before trying to merge them.

* Fix installation location for libraries (ECP-WarpX#2583)

During configuration the installation location for libraries is given by
dumping the cmake variable `CMAKE_INSTALL_LIBDIR`.
This commit adjusts the installation of WarpX libraries (WarpX_LIB=ON)
to respect this setting.

Co-authored-by: Rolf Pfeffertal <tropf@users.noreply.github.com>

* Release 21.12 (ECP-WarpX#2614)

* AMReX: 21.12

* PICSAR: 21.12

* WarpX: 21.12

* div(E,B) Cleaning Options for PSATD (ECP-WarpX#2403)

* Implement div(E)/div(B) Cleaning with Standard PSATD

* Cleaning

* Update Benchmark

* Add Nodal Synchronization of F,G

* OneStep_multiJ: Nodal Syncs, Damp PML

* OneStep_multiJ: Push PSATD Fields in PML

* div Cleaning Defaults (Domain v. PML)

* Include Fix of ECP-WarpX#2429 until Merged

* Reset Benchmark of Langmuir_multi_psatd_div_cleaning

* Multi-J: Remove PML Support

* Include Fix of ECP-WarpX#2474 Until Merged

* Exchange All Guard Cells for F,G

* Fix Defaults

* Update Test, Reset Benchmark

* Fix Defaults

* Cleaning

* Default update_with_rho=1 if do_dive_cleaning=1

* Update CI Test pml_psatd_dive_divb_cleaning

* Replace Warning with Abort

* Add 2D Langmuir Test w/ MR & PSATD (ECP-WarpX#2605)

* Add 2D Langmuir Test w/ MR & PSATD

* Add Missing Compile String

* Fix out-of-bound in Inverse FFT of F,G (ECP-WarpX#2619)

* Mention that the potentail should be constant inside EB (ECP-WarpX#2618)

* Mention that the potentail should be constant inside EB

* Update text

* Replace AMREX_SPACEDIM: Boundary & Parallelization (ECP-WarpX#2620)

* AMREX_SPACEDIM : Boundary Conditions
* AMREX_SPACEDIM : Parallelization
* Fix compilation
* Update Source/Parallelization/WarpXComm_K.H

* Fix out-of-bound in the initialization of EB (ECP-WarpX#2607)

* Call FillBoundary when initializing EB

* Avoid out-of-bound

* Bug fix

* Apply suggestions from code review

* update version number

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>
Co-authored-by: Remi Lehe <remi.lehe@normalesup.org>
Co-authored-by: Lorenzo Giacomel <47607756+lgiacome@users.noreply.github.com>
Co-authored-by: Kevin Z. Zhu <86268612+KZhu-ME@users.noreply.github.com>
Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com>
Co-authored-by: David Grote <grote1@llnl.gov>
Co-authored-by: Phil Miller <phil@intensecomputing.com>
Co-authored-by: s9105947 <80697868+s9105947@users.noreply.github.com>
Co-authored-by: Rolf Pfeffertal <tropf@users.noreply.github.com>
Co-authored-by: Prabhat Kumar <89051199+prkkumar@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RevathiJambunathan RevathiJambunathan RevathiJambunathan approved these changes

@atmyers atmyers Awaiting requested review from atmyers

Assignees
No one assigned
Labels
bug: affects latest release Bug also exists in latest release version bug Something isn't working component: diagnostics all types of outputs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

cartesian3d: BTD_ReducedSliceDiag Test Unstable
3 participants
@ax3l @atmyers @RevathiJambunathan