Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

openPMD: Add ADIOS2 Engine Parameter Control #2872

Merged
merged 6 commits into from
Feb 18, 2022
Merged

openPMD: Add ADIOS2 Engine Parameter Control #2872

merged 6 commits into from
Feb 18, 2022

Conversation

kshitij-v-mehta
Copy link
Contributor

Add support for setting ADIOS engine types and their parameters for openPMD. Similar to how ADIOS operators are setup.
Addresses #2866

@ax3l ax3l requested review from ax3l and guj February 18, 2022 00:25
@ax3l ax3l added component: diagnostics all types of outputs component: openPMD openPMD I/O enhancement New feature or request labels Feb 18, 2022
ax3l
ax3l reviewed Feb 18, 2022
View reviewed changes
Copy link
Member

@ax3l ax3l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks a lot! :)

Docs/source/usage/parameters.rst Outdated Show resolved Hide resolved
Docs/source/usage/parameters.rst Outdated Show resolved Hide resolved
Docs/source/usage/parameters.rst Outdated Show resolved Hide resolved
Docs/source/usage/parameters.rst Outdated Show resolved Hide resolved
Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp Outdated Show resolved Hide resolved
Source/Diagnostics/FlushFormats/FlushFormatOpenPMD.cpp Show resolved Hide resolved
@ax3l ax3l self-assigned this Feb 18, 2022
@ax3l
Copy link
Member

ax3l commented Feb 18, 2022

@kshitij-v-mehta did you have a chance to test this already? Let me know what you find in terms of reduced total runtime of the application compared to synchronous I/O.

@guj knows our usual benchmarks. You could start with full GPUs (about <=256^3 cells per GPU on Summit for homogeneous plasma cases) and dumps every 1000, 100 or 10 time steps, to cover a few differing scenarios of interest.

@ax3l ax3l changed the title Adios engine support openPMD: Add ADIOS2 Engine Parameter Control Feb 18, 2022
@kshitij-v-mehta
Copy link
Contributor Author

did you have a chance to test this already? Let me know what you find in terms of reduced total runtime of the application compared to synchronous I/O.

I have tested these changes only for correctness yet, and not for performance improvements. I am currently looking at how to effectively use the node-local NVM on machines such as Summit. I will follow up with you on that.

@ax3l
Copy link
Member

ax3l commented Feb 18, 2022

Cool, thanks a lot! :)

@ax3l ax3l merged commit 8e1517a into ECP-WarpX:development Feb 18, 2022
@ax3l ax3l mentioned this pull request Feb 18, 2022
Closed
roelof-groenewald added a commit to ModernElectron/WarpX that referenced this pull request Feb 19, 2022
@roelof-groenewald @ax3l
@lgiacome @pre-commit-ci @EZoni @NeilZaim @peterscherpelz @RevathiJambunathan @RemiLehe @cyrush @ElisaRheaume @kshitij-v-mehta
Update AMReX version number - fixes restart bug (#154)
6f459f5 
* NCIGodfreyFilter: Fix Int Division (ECP-WarpX#2837)

* NCIGodfreyFilter: Fix Int Division

`m_cdtodz` is between 0 and 1, and we interpolate a set of
coefficients from a table.

* reset benchmarks

Co-authored-by: Tools <warpx@lbl.gov>

* Adding documentation for lxplus (ECP-WarpX#2756)

* Adding documentation for lxplus

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Adding the new documentation

* Apply suggestions from code review

Some suggestions from the code review.

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Removing suggestion about miniconda

* Switched to using anonymous environment

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fixing the architecture

* Apply suggestions from code review

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Update Tools/machines/lxplus-cern/spack.yaml

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Updating env to use pre-installed openmpi

* Moving a file

* Docs: Now using GCC 11.2.0

* Add OpenMPI Version in Spec

* Add CPU target architecture note

* One more GCC 9.2.0->11.2.0 Update

* Finalize Spack Stack Setup Notes

* Move AFS Spack Config to Environment File

* Comment on Variations of the Spack Env

* Improve git clone and spack activation

* Apply suggestions from code review

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Fixing ncurses

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Fix: Spack Concretization (CUDA/Python)

Somehow, this concretizes the variants not properly otherwise.

* Not using the preinstalled openmpi anymore

* updated lxplus.rst

* added lxplus_warpx.profile.example

* Apply suggestions from code review

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Apply suggestions from code review

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Updated documentation

* Added a precisation about warpx.profile

* Apply suggestions from code review

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Add missing empty newline

Co-authored-by: lgiacome <lorenzo.giacome@cern.ch>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* BackTransformParticleFunctor: Unused Counter (ECP-WarpX#2840)

Remove an unused counter in BTD particle filtering.

Seen first with a HIP diagnostics.

* ABLASTR: particle weights `const` (ECP-WarpX#2838)

* ABLASTR: particle weights `const`

We can declare the particle weights `const` because we don't
change values in them during deposition.

* DepositCharge: `const`-ify usage

* Rename ngE as ngEB (used for E,B) (ECP-WarpX#2841)

* Fix some offsets with the gather buffers (ECP-WarpX#2847)

* Add amrex REPO and BRANCH flags for python builds (ECP-WarpX#2845)

* Add WarpX_amrex_repo and _branch options to Python

In python setup, environment variables WARPX_AMREX_REPO and
WARPX_AMREX_BRANCH will now set these variables

* Update documentation with new compile envvars

* User-defined integer and real particle attributes (ECP-WarpX#2735)

* define user attributes, parse them, initialize with respective parsers

* fix warning by using static_cast for int attribute as parser returns real

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* clean-up from self-review

* adding dimensionless velocity, gamma*v/c and time to parser argument

* add documentation

* typo in comment

* unused var

* device vector for kernels

* particle attribute in developer doc

* data ptr for device vector

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* ignore_unused

* Docs: Describe all particle attributes

including pre-defined ones :)

* Docs: Fix formatting (user params)

* Add: 1D and RZ Support

* Docs: Fix Typo in Function Declaration

* Laser-Ion Example: User-Defined Attrib.

Add two user-defined attributes to the laser-ion acceleration
example. This is a 2D test.

Documents the name in the table of commonly used, user-defined
attribute names. The attribute added is the original position
of particles, which I like to plot in "potential" plots that
correlate original position in the target with final energy.

* changing user-interface API with .attribute. and no need for separate 1D 2D 3D RZ code for parser. pos.x/y/z returns the right values

* Adding 1D, 3D, and rz tests

* attribute in inputs

* at(i)

* refinining names for inputs for laser ion and acceleration tests

* typo in input

* reset benchmarks for test-cases that included attributes

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Don't cut all particles in a Gaussian beam when x_rms=0 (ECP-WarpX#2844)

* [pre-commit.ci] pre-commit autoupdate (ECP-WarpX#2851)

updates:
- [github.com/Lucas-C/pre-commit-hooks: v1.1.11 → v1.1.12](Lucas-C/pre-commit-hooks@v1.1.11...v1.1.12)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Use parser to read laser spatio-temporal couplings direction (ECP-WarpX#2843)

* Only set modified k to 0 for even number of points (ECP-WarpX#2852)

* Only set modified k to 0 for even number of points

* Update Source/FieldSolver/SpectralSolver/SpectralKSpace.cpp

* Allow flux injection in the out-of-plane direction for RZ/2D geometry (ECP-WarpX#2788)

* Implement injection orthogal to plane

* Generalize momentum distribution for flux injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"

This reverts commit b0cd189.

* Revert "Generalize momentum distribution for flux injection"

This reverts commit 0a22b1d.

* Rotate momentum initialization

* Correct flux number when the direction is normal to plane

* Update distribution of particles within a cell

* Clean-up injection code

* Add more documentation

* Add more comments

* Handle 1D case

* Only do the rotation for Gaussian flux profile

* Fix compilation error

* Correct compilation for GPU

* Start adding automated test

* Correct sign of velocity

* Update to add continuous injection

* Finalize test

* Correct processing of flux_normal_axis

* Add checksum

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

* Update script

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update checksum

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* AMReX/PICSAR: Weekly Update (ECP-WarpX#2849)

* AMReX: Update latest commit

* AMReX: Update to 76d08651adb987e3fba6b232e806c5e7c365a8d9

* update CI to use ascent 0.8.0 release container (ECP-WarpX#2858)

* use ascent 0.8.0 release container

* try again

* restore

* use new install loc

* Correct typo in the relativistic Poisson solver (ECP-WarpX#2853)

* Correct typo in the relativistic Poisson solver

* Fix unused variable

* Update benchmark

* Gaussian particle beam: add error message when using y_rms = 0 in 2D (ECP-WarpX#2862)

* ParticleBuffer: Generalize & Move (ECP-WarpX#2860)

* ParticleBuffer: Generalize & Move

- move the `ParticleBuffer` to ABLASTR
- generalize the API
- remove `amr_core` argument
- use more semantic naming
- add docs

* Use `amrex::ParticleContainer::make_alike`

* Update AMREX

to include AMReX-Codes/amrex#2630

* ABLASTR: Refactor `deposit_charge` API (ECP-WarpX#2856)

Simplified and re-ordered interface for
`ablastr::particles::deposit_charge`.

* Provide `t_min` and `t_max` for flux injection (ECP-WarpX#2842)

* Implement injection orthogal to plane

* Generalize momentum distribution for flux injection

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Revert "[pre-commit.ci] auto fixes from pre-commit.com hooks"

This reverts commit b0cd189.

* Revert "Generalize momentum distribution for flux injection"

This reverts commit 0a22b1d.

* Rotate momentum initialization

* Correct flux number when the direction is normal to plane

* Update distribution of particles within a cell

* Clean-up injection code

* Add more documentation

* Add more comments

* Handle 1D case

* Only do the rotation for Gaussian flux profile

* Fix compilation error

* Correct compilation for GPU

* Start adding automated test

* Correct sign of velocity

* Update to add continuous injection

* Finalize test

* Correct processing of flux_normal_axis

* Add checksum

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug

* Update script

* Implement maximum injection time

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add parameter tmin

* Make parameter optional ; update documentation

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Lassen: Fix Chunked HDF5 with MPI (ECP-WarpX#2863)

Try to work-around segfaults with HDF5 when running on more than one node.

* Refactor Current Correction Functions (ECP-WarpX#2839)

* Refactor Current Correction Functions

* Clean Up, Reset Benchmark

* Rotate momentum for RZ flux injection (ECP-WarpX#2867)

* Add warning to FieldProbe re: Boosted Frame (ECP-WarpX#2868)

* Add warning to FieldProbe re: Boosted Frames

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Update parsing of FieldProbe in 2D and 1D (ECP-WarpX#2818)

* Update parsing of FieldProbe in 2D and 1D

* Fix unused variables

* Allow plane probe in 2D and line probe in 1D

* doc update

* Do Not Fill PML Guard Cells w/ Inverse FFTs (ECP-WarpX#2854)

* Fix number of guard cell for coarse patch (ECP-WarpX#2869)

* openPMD: Add ADIOS2 Engine Parameter Control (ECP-WarpX#2872)

* Adds support for ADIOS engines ECP-WarpX#2866

Input file can now have lines like
diag1.adios2_engine.parameters.NumAggregators=2

* Docs for ADIOS engine type and parameters ECP-WarpX#2866

* Aesthetic edit in adios2 engine documentation ECP-WarpX#2866

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Removed debug print statement ECP-WarpX#2866

* Style Updates

Co-authored-by: Mehta, Kshitij V <kshitij-v-mehta@github.com>
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Add `PHistDiag` for scraping (#153)

* refactor of surface flux diagnostic handling before implementing PHistDiag

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* initial commit of `ParticleHistDiag`

* updated changelog and version number

* added plotting functionality specifically for ZPlane assemblies

* save histogram binning details to file as well

* code cleanup

* changes requested during PR review

* add comment about 2d plotting

* remove debugging print statement

* further code changes from PR review

* Fix typo

Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com>

* fix another typo

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com>

* Add initialization of pairwise Coulomb collisions (#155)

* refactor of surface flux diagnostic handling before implementing PHistDiag

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* initial commit of `ParticleHistDiag`

* updated changelog and version number

* added plotting functionality specifically for ZPlane assemblies

* save histogram binning details to file as well

* code cleanup

* changes requested during PR review

* add comment about 2d plotting

* added Coulomb collision installation to picmi.py

* added pairwise Coulomb collision initialization

* remove debugging print statement

* further code changes from PR review

* Fix typo

Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com>

* fix docstring

* cleaned up the logging message for Coulomb scattering

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com>

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
Co-authored-by: Tools <warpx@lbl.gov>
Co-authored-by: Lorenzo Giacomel <47607756+lgiacome@users.noreply.github.com>
Co-authored-by: lgiacome <lorenzo.giacome@cern.ch>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com>
Co-authored-by: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com>
Co-authored-by: Peter Scherpelz <31747262+peterscherpelz@users.noreply.github.com>
Co-authored-by: Revathi  Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com>
Co-authored-by: Remi Lehe <remi.lehe@normalesup.org>
Co-authored-by: Cyrus Harrison <cyrush@llnl.gov>
Co-authored-by: Tiberius Rheaume <35204125+TiberiusRheaume@users.noreply.github.com>
Co-authored-by: Kshitij Mehta <kshitij-v-mehta@users.noreply.github.com>
Co-authored-by: Mehta, Kshitij V <kshitij-v-mehta@github.com>
@ax3l ax3l mentioned this pull request Mar 28, 2022
Merged
ax3l
ax3l reviewed Mar 28, 2022
View reviewed changes
Docs/source/usage/parameters.rst

.. code-block:: text

<diag_name>.adios2_engine.parameter.NumAggregators = 2048
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Little typo fixed in #3002

@ax3l
Copy link
Member

ax3l commented Oct 10, 2022

Hi @kshitij-v-mehta, I just tried <diag_name>.adios2_engine.parameters.NumAggregators = 2 with a 2-rank MPI run again, and I think it does not work...

Have you observed the same?

ax3l
ax3l reviewed Oct 10, 2022
View reviewed changes
Source/Diagnostics/WarpXOpenPMD.cpp
Comment on lines +182 to +188
if (!engine_type.empty())
op_block += ",";

} // end operator string block

// add the engine string block
if (!engine_type.empty()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, these blocks are only added if a user explicitly specifies an engine. Maybe we should add a default here?

@ax3l ax3l mentioned this pull request Oct 10, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ax3l ax3l ax3l approved these changes

@guj guj Awaiting requested review from guj

Assignees

@ax3l ax3l

Labels
component: diagnostics all types of outputs component: openPMD openPMD I/O enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kshitij-v-mehta @ax3l