Improve rho diagnostic #1099

EZoni · 2020-06-17T01:28:01Z

This PR fixes two issues mentioned in #1037, namely:

the charge deposition diagnostics contained rho from the previous time step;
rho was computed at every time step whenever requested in the diagnostics.

The last improvement mentioned in #1037, namely making the new functor compute per-species charge deposition, can be addressed in another PR.

The variable WarpX::plot_rho has been removed from the WarpX class. A new functor RhoFunctor is defined and implemented in Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.H/.cpp, following the implementation of similar functors, such as, for example, DivEFunctor.

I had to reset the benchmarks for the following tests: Langmuir_multi, Langmuir_multi_nodal, Langmuir_multi_single_precision, and Python_LaserAccelerationMR. These tests use all the FDTD solver, hence the rho values in the respective benchmarks were computed from the second-to-last iteration, because of the time staggering issue that this PR is fixing.

This PR should allow #1048 to be unblocked and merged, as the possible performance regression introduced there is now avoided by not depositing rho at every time step when requested in the diagnostics.

- new functor RhoFunctor to compute rho; - fix FDTD benchmarks (previously staggered in time by one time step); - fix one PSATD benchmark for single precision test; - update documentation and remove old variable rho_plot from class WarpX; - TODO: check test Python_LaserAccelerationMR on Travis CI; - TODO: check correctness of implementation in RZ geometry.

dpgrote · 2020-06-18T00:03:09Z

Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.cpp

+    rho_ptr = std::unique_ptr<amrex::MultiFab>
+              (new amrex::MultiFab( ba, warpx.DistributionMap(m_lev), 2*warpx.n_rz_azimuthal_modes-1, ng ));
+    rho_ptr = mypc.GetChargeDensity( m_lev );
+    amrex::MultiFab& rho = *rho_ptr;


I'll jump in. All that is needed here (instead of lines 23 to 35) is
std::unique_ptr<amrex::MultiFab> rho = mypc.GetChargeDensity( m_lev );
Then in the code below, use pointer dereferencing, e.g. rho->boxArray() and *rho.

dpgrote · 2020-06-18T00:12:03Z

Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.cpp

+    // mf_src should not be used, let's make sure it is null
+    AMREX_ALWAYS_ASSERT(mf_src == nullptr);
+    // Write only in one output component
+    AMREX_ALWAYS_ASSERT(ncomp == 1);


This assert should not be done since it is valid to request writing out all modes. See my suggested changes in FullDiagnostics.cpp.

dpgrote · 2020-06-18T00:43:50Z

Source/Diagnostics/FullDiagnostics.cpp

@@ -6,6 +6,7 @@
 #include "ComputeDiagFunctors/PartPerGridFunctor.H"
 #include "ComputeDiagFunctors/DivBFunctor.H"
 #include "ComputeDiagFunctors/DivEFunctor.H"
+#include "ComputeDiagFunctors/RhoFunctor.H"
 #include "FlushFormats/FlushFormat.H"
 #include "FlushFormats/FlushFormatPlotfile.H"
 #include "FlushFormats/FlushFormatCheckpoint.H"


Some code needs to be added to the routine FullDiagnostics::AddRZModesToDiags to add the writing of the rho components. Duplicate the code that is there for divE, but writing rho. Note that the check for rho_requested can be simplified to by
bool rho_requested = WarpXUtilStr::is_in(m_varnames, "rho");
instead of the explicit loop over m_varnames.

👍 looks addressed below now.

dpgrote · 2020-06-18T00:46:16Z

@EZoni Thanks for the PR! See my suggestions.

dpgrote · 2020-06-18T17:00:29Z

Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.cpp

+    // Guard cell is set to 1 for generality. However, for a cell-centered
+    // output Multifab the guard-cell data is not needed, especially considering
+    // the operations performed in CoarsenIO::Coarsen
+    constexpr int ng = 1;


ng is not used here so this can be deleted.

dpgrote · 2020-06-18T17:02:53Z

Source/Diagnostics/FullDiagnostics.cpp

+                              m_crse_ratio, false, ncomp_multimodefab);
+        icomp += 1;
+        AddRZModesToOutputNames(std::string("rho"), ncomp_multimodefab);
+    }


Thanks for adding this!

dpgrote · 2020-06-18T17:08:37Z

Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.cpp

+      m_convertRZmodes2cartesian(convertRZmodes2cartesian)
+{
+    // mf_src should not be used, let's make sure it is null
+    AMREX_ALWAYS_ASSERT(mf_src == nullptr);


You can remove the mf_src argument since it isn't used for anything. This will be cleaner.

dpgrote · 2020-06-18T17:13:20Z

Regression/Checksum/benchmarks_json/Langmuir_multi_psatd_single_precision.json

@@ -9,7 +9,7 @@
    "particle_weight": 127999991808.0
  },
  "lev=0": {
-    "Bx": 16.283409055462556,
+    "Bx": 16.302038904962248,


I would be careful about this change and make sure you understand why this update is needed. If this is due to a change in a different PR, then the change should have been part of that PR.

dpgrote · 2020-06-18T17:14:44Z

@EZoni Thanks for the fixes. I have some more comments.

EZoni · 2020-06-18T21:31:39Z

@dpgrote Thank you for all the suggestions. I think I implemented all of them. I also reverted the Bx benchmark for the single precision PSATD test to its old value. I do not understand why this particular checksum test fails at all, given that I changed only the rho diagnostic, but I will try to look into this.

EZoni · 2020-06-18T22:26:49Z

Despite reverting the Bx benchmark value, the checksum test did not fail, contrary to my expectations. So everything seems OK about that now. I see the test fail locally on my machine, which is why I reset that benchmark in the first place, but it could be something wrong on my side.

dpgrote

This looks good. Thanks for the PR!

dpgrote · 2020-06-19T17:00:38Z

@EZoni Is this ready for merge or do you want to wait for a review from someone else?

EZoni · 2020-06-19T17:10:50Z

@dpgrote Thank you very much for your review!

Before merging, I would like to know @RevathiJambunathan's opinion about the question I asked in the PR description: do we have to be careful about back-transformed diagnostics and treat them as a special case, as suggested in @RemiLehe's comment to #1048?

I am not familiar with back-transformed diagnostics, so I don't know if this PR is introducing a bug somewhere where back-transformed diagnostics are used. The related tests pass, but they do not perform a checksum test, as far as I understand.

ax3l · 2020-06-20T21:59:36Z

Source/Diagnostics/ComputeDiagFunctors/RhoFunctor.H

+     * \param[in] lev level of MultiFab
+     * \param[in] crse_ratio coarsening ratio for interpolation of field values
+     *                       from simulation MultiFabs to the output MultiFab mf_dst
+     * \param[in] convertRZmodes2cartesian


this params lacks a description :)

MaxThevenet · 2020-06-22T07:10:54Z

Thanks for this PR!!
Indeed, we probably need a specific handling of back-transformed diagnostics as they also use rho, see WarpX::GetCellCenteredData, which contains

        const std::unique_ptr<MultiFab>& charge_density = mypc->GetChargeDensity(lev);
        AverageAndPackScalarField( *cc[lev], *charge_density, dmap[lev], dcomp, ng );

A few options:

Keep the old behavior of plot_rho just for BTD
remove the capability to plot rho in the BTD, at least temporarily.

I would be in favor of 1., because even if @RevathiJambunathan is reformatting BTD, the final implementation would have to do sth like plot_rho.
I will put the WIP tag back in the title of this PR.

RevathiJambunathan · 2020-06-22T14:22:40Z

I agree. For now, we can have the old functionality just for BTD. I have an open PR and another follow-up with new BTD reformatting. I will include this functionality from RhoFunctor.cpp, in the new BTD.

EZoni · 2020-06-22T18:36:08Z

@MaxThevenet @RevathiJambunathan

Thank you for your comments and suggestions. I restored the old behavior of plot_rho for back-transformed diagnostics. In particular, the relevant changes are:

Source/Diagnostics/Diagnostics.cpp: warpx.setplot_rho(true) (previously removed) is called again, but only when back-transformed diagnostics are used:

// If user requests rho with back-transformed diagnostics, we set plot_rho=true
// and compute rho at each iteration
if (WarpXUtilStr::is_in(m_varnames, "rho") && WarpX::do_back_transformed_diagnostics) {
     warpx.setplot_rho(true);
}

Source/Diagnostics/FullDiagnostics.cpp: the old functor CellCenterFunctor (previously removed) is called again, but only when back-transformed diagnostics are used:

else if ( m_varnames[comp] == "rho" ){
    if ( WarpX::do_back_transformed_diagnostics ) {
#ifdef WARPX_USE_PSATD
        // rho_new is stored in component 1 of rho_fp when using PSATD
        amrex::MultiFab* rho_new = new amrex::MultiFab(*warpx.get_pointer_rho_fp(lev), amrex::make_alias, 1, 1);
        m_all_field_functors[lev][comp] = std::make_unique<CellCenterFunctor>(rho_new, lev, m_crse_ratio);
#else
        m_all_field_functors[lev][comp] = std::make_unique<CellCenterFunctor>(warpx.get_pointer_rho_fp(lev), lev, m_crse_ratio);
#endif
    }
    else {
        m_all_field_functors[lev][comp] = std::make_unique<RhoFunctor>(lev, m_crse_ratio);
    }
}

I thought that restoring the old functor in this case is necessary in order to avoid depositing the charge twice at every iteration. The new functor RhoFunctor is used in any other case, when back-transformed diagnostics are not used.

Source/WarpX.cpp: the MultiFabs rho_fp and rho_cp are now allocated when both plot_rho and do_back_transformed_diagnostics are true.

I will re-request your review, so that you can crosscheck this fix and provide feedback. Thank you.

RevathiJambunathan · 2020-06-23T22:35:33Z

Thank you for this PR and for the changes @EZoni ! Looks good to me!

Docs/source/visualization/advanced.rst

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

ax3l

Thank you Edoardo! ✨
Also thx to Reva and Dave for reviews!

@ax3l

[Mini] Z Offset to loaded openPMD particles (ECP-WarpX#1070) * Added description of singleparticle particle injection style * Revert "Added description of singleparticle particle injection style" This reverts commit f02d842. * Added use_q_shift and q_shift variables * Removed use_q_shift * Added description of q_shift to the Docs * Replaced q_shift by z_shift as suggested by @ax3l * Added recommendation to PWFA example * z_shift should be optional (used query instead of get) * Fix compile problem Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Stop mailing errors from Battra (ECP-WarpX#1079) * stop mailing errors from Battra * remove Axel from Battra mails * Battra mails: remove Diana & Edoardo Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> update doc on ionization (ECP-WarpX#1068) Docs: Summit Restructure (ECP-WarpX#1090) * Docs: Summit Restructure Restructures the instructions for Summit (OLCF). This loads all software that might be needed to reduce the "if then else" logic a user has to understand during install. Everything is now required or optional, but everything is compatible or simply ignored. Uses new conventions for downloaded source and manually built software to structure things in a simpler, scalable manner. * Summit Docs: Fix Typos Add CMake Scripts (ECP-WarpX#759) * [skip travis] CMake: Add CMake Scripts This add CMake scripts for the CMake build system generator. * CUDA Support Achieved Fastest compile via: cmake .. -DENABLE_CUDA=ON -DCUDA_ARCH=6.0 * CMake: openPMD & QED * CMake: Dimension Support WarpX_DIMS=2,3,RZ * CCache & Out-of-Source Enable CCache when found and prevent in-root builds (in-source okay-ish). * CMake: CUDA C++ Language Minimum * Add new Files * CI: Migrate macOS to CMake * Fortran: Enable Because Transitive ... for now :) * CI: Migrate Linux ICC to CMake * CI: Verbose Make * CMake: Add Missing PSATD Sources * CMake: PSATD, Compute, Precision, Clean remove amrex namespace from diag functor cpp files and use amrex:: prefic (ECP-WarpX#1093) Docs: Update Ascent (ECP-WarpX#1094) - GNUmake: Unify the controlling variable to `ASCENT_DIR` as in AMReX. - Update input parameter reference - General docs overhaul - Note current (work-in-progress) Ascent path on Summit New BTD class layout and member functions - Part 1 (ECP-WarpX#1085) * fixing bug to initialize CellCenterFunctor for Bx * create new BTB class and move FullDiagnostics functions from Diagnostics class * [skip ci] Adding BTDiag class members and allocating them * output_mf vector of buffers for both Full and BTD * [skip ci] a few comments * added more functions in BTDiagnostics for initialization * [skip-ci] a small comment abour FlushRawData * [skip-ci] Add new functor class for BTD * Included BTDiagnostics class and skeleton of functions to be populated * fix eol and make destructor of base Diag class virtual * remove redundant query file_prefix in FullDiag class * Apply suggestions from code review adding PR suggestions Co-authored-by: MaxThevenet <mthevenet@lbl.gov> * cleaning function calls and definitions as suggested in review * uncomment the assert. It is valid for FullDiag and BTDiag * Update Source/Diagnostics/Diagnostics.H adding PR suggestion including comments Co-authored-by: MaxThevenet <mthevenet@lbl.gov> * moved InitData to Diagnostics.cpp and call relevant functions from there * moving FilterComputeAndPack to Diagnostics * removing commented out code * remove gpu vector containing map of fields from cc_mf to dst_mf. New diag machinery ensures cc_mf.ncomp == dst_mf.ncomp * fix eol whitespace * DefineCellCenteredMultiFab should be called only once for all buffers * Apply suggestions from code review add pr suggestions Co-authored-by: MaxThevenet <mthevenet@lbl.gov> * clarifying pure virtual and virtual functions * removing namespace amrex and adding corresponding amrex:: prefix * the number of levels to be output could be different from nlev, nmax_lev * add missing semi-colon! * add amrex:: prefix to MultiFab to comile psatd * add new diag files to CMakeLists.txt * Apply suggestions from code review PR suggestions from @ax3l. Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * fix EOL * including sensei and ascent in doxygen docs for format variable Co-authored-by: MaxThevenet <mthevenet@lbl.gov> Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Travis CI: Wait Longer for Make (ECP-WarpX#1098) It looks like "make" run from our python test scripts does sometimes take longer than 10 minutes and does not produce output on stdout during that time (probably because we hide it). This adds a 20min grace time for this (default: 10min) before ending the job with a timeout. Generally, tests on Travis are close to the 50min mark of total job time, this patch does not change this :) Doc: CMake Basics (ECP-WarpX#1101) Adds a first documentation on how to build WarpX with CMake. CMake: Ascent (ECP-WarpX#1100) Adds CMake support for Ascent in situ vis. Adds experimental instructions for Summit builds. Renamining: development (ECP-WarpX#1102) In the process of renaming our central development branch to `development` the following updates are needed. reimplementation of interpolation and removal of Fortran from gnu make (ECP-WarpX#1103) * remove obsolete functions in FieldIO * * Remimplement interpolation of coarse patch onto fine grids without div cleaning in C++. * No longer need to have a Fortran compiler. CMake: No Fortran (ECP-WarpX#1108) Save compile time and complexity. Fix: Extra Semicolons & In-Line Doxygen (ECP-WarpX#1109) Fix superfluous semicolons and two in-line doxygen strings. Style Check: Eval-uable (ECP-WarpX#1107) This adjusts the output of `hasEOLwhiteSpace` and `hasTabs` so that one can run them like this in a bash shell: ```bash eval "$(.github/workflows/source/hasEOLwhiteSpace)" eval "$(.github/workflows/source/hasTabs)" ``` Fix Race condition in GetParticleSlice for BTD (ECP-WarpX#1105) * define temp array outside the ParIter loop for GetParticleSlice * Moving definition of temp arrays closer to the ParIter loop where used with comments [Mini] Docs info for blocking_factor (ECP-WarpX#1096) * Added description of singleparticle particle injection style * Revert "Added description of singleparticle particle injection style" This reverts commit f02d842. * More info on blocking factor * Update Docs/source/running_cpp/parallelization.rst Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Update Docs/source/running_cpp/parameters.rst Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> AddPlasmaFromFile fix for periodic boundaries (ECP-WarpX#1089) * Added description of singleparticle particle injection style * Revert "Added description of singleparticle particle injection style" This reverts commit f02d842. * Tried to inject particles only in simulation Box * Added Todo for multiple levels of mesh refinement * Explain todo better * Added x and y_shifts and replaced cascading ifs by bools and continue * Replaced cascading ifs by bools and continue * Replaced TODO by FIXME as reviewer suggested * Made the insidebounds function use box limits as defaults * Fixed identation issue * Fixed identation issue * Fixed identation issue * Added to Docs the default for xmin,xmax and other dirs * Fix missing y and z dirs * Update Source/Initialization/PlasmaInjector.cpp * Fixed PlasmaInjector limits with periodic condition * Removed check if 3D because now ylims are +/-inf * Restored if to check for y velocity also in 2D/RZ * Fixed Docs to recent changes * Update Source/Initialization/PlasmaInjector.cpp * Fix extra bracket * Tiny bool cleanup Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Remove persistent E+B (ECP-WarpX#1050) * add functor for doing the tmp particles copy for the back-transformed diagnosti * merge the particle push options into one kernel * EOL * fix assertion * add a FieldGatherandPushPX method to PhysicalParticleContainer * handle offset in copyAttribs * allow this functor to be constructed even it we aren't doing the back transformed diagnostics * EOL * update the overloads of PushPX for the Photon and RigidInjected ParticleContainers * function for dispatching the right field gather * init this val to 0.0 * fix some typos * handle scaling the fields for rigid injection * EOL * don't need to get pointers to E and B arrays in PushPX any more. * actually I can't remove these yet * EOL * variable order bug * move the QED stuff to the proper place * EOL * make sure we don't build these functors unless the runtime options are toggled * EOL * perform the field gather prior to the photon particle push * remove E and B components and FieldGather methods. Reimplement PushP for rigid injected and physical particles * update ionization to do field gather inline * remove E and B from the particle diagnostics * don't write E or B in these tests for particles * add missing files * remove EB from the Regtest ini file too * no need to do this twice * important typo * also do the gather inline for the QED processes that need to * move these sources inside ifdef for QED * fix bug in RZ * remove some fields from the Python tests. * remove all particle E and B comps from json benchmarks * don't assert that Ey is the langmuir output * remove uy from this output * update test * restore the mesh fields I turned off by mistake * turn off field IO for a few python tests I missed * fix typo * reset Langmuir_multi benchmark * update Langmuir_multi_nodal benchmark * update single precision langmuir bench * update psatd single precision languir one too * also do ionization_lab * finally, ionizaiton_boost * update benchmarks_json/Langmuir_multi_psatd.json * update benchmarks_json/Langmuir_multi_psatd_current_correction.json * update benchmarks_json/Langmuir_multi_psatd_momentum_conserving.json * update benchmarks_json/Langmuir_multi_psatd_nodal.json * remove the particle E and B from the choices in the docs * fix offset bug * also add the Gather subdirectory * Update Source/WarpX.H Co-authored-by: MaxThevenet <mthevenet@lbl.gov> * add docstring for LowerCornerWithGalilean * add new source files to CMakeLists.txt * also need to update the GPU regression tests * update the name of the output file for this python test * remove field gather call from FieldDiagnostics * fix typo in docstring * init fields to 0 * add docstring to the CopyParticleAttribs constructor * some explicit amrex::namepace Co-authored-by: MaxThevenet <mthevenet@lbl.gov> Added check of gridding for RZ spectral solver (ECP-WarpX#1005) * Added CheckGriddingForRZSpectral * Bug fix in check gridding for RZ Spectral, fixing handling of refinement ratio * Change numprocs=2 for python reg tests and reset benchmarks (ECP-WarpX#1075) * changing nprocs to 2 for Python_PlasmaAcceleration test * Resetting benchmark for Python_PlasmaAcceleration test case due to change in nprocs=2 in reg test * resetting benchmarks for Python test-cases except Python_Langmuir due to change in numprocs in WarpX-tests.ini * changing numprocs=2 for Python tests * resetting python reg test benchmarks after updating pywarpx Co-authored-by: Revathi Jambunathan <revanathan@pop-os.localdomain> Co-authored-by: Tools <warpx.lbl.gov> Co-authored-by: Tools <warpx@lbl.gov> * Updates to picmi, mostly related to fields, and add psatd (ECP-WarpX#1077) * Fixed bad arguments in PICMI_inputs_laser_acceleration.py (ECP-WarpX#1074) Note that polarization_angle was changed to 0 since that was the value being used because the polarization_argument was bad. * Fix typo for pml_ncell in parameters doc (ECP-WarpX#1071) * remove #line wp_parser.l because it causes issues with AMReX make system (ECP-WarpX#1078) * remove amrex namespace from diag functor cpp files and use amrex:: prefic (ECP-WarpX#1093) * For RZ spectral, updated documentation regarding blocking factor and max grid size Co-authored-by: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> Co-authored-by: Revathi Jambunathan <revanathan@pop-os.localdomain> Co-authored-by: Tools <warpx@lbl.gov> Co-authored-by: NeilZaim <49716072+NeilZaim@users.noreply.github.com> Co-authored-by: WeiqunZhang <WeiqunZhang@lbl.gov> Co-authored-by: Remi Lehe <remi.lehe@normalesup.org> Add automated test for divE cleaning (ECP-WarpX#949) * Added automated test for dive cleaning * Add analysis script * Include test in automated suite * Add 3D test * Modify script so that it could run in 3d * Reduce tolerance for 3D test * Set benchmarks for divE cleaning tests * Remove unused parameter Co-authored-by: Tools <warpx@lbl.gov> Use new github picsar repo (ECP-WarpX#1111) * Use new picsar repo on github * Add new repo in PICSAR.cmake New BTD class layout and member functions - Part 2 (ECP-WarpX#1092) * fixing bug to initialize CellCenterFunctor for Bx * create new BTB class and move FullDiagnostics functions from Diagnostics class * [skip ci] Adding BTDiag class members and allocating them * output_mf vector of buffers for both Full and BTD * [skip ci] a few comments * added more functions in BTDiagnostics for initialization * [skip-ci] a small comment abour FlushRawData * [skip-ci] Add new functor class for BTD * Included BTDiagnostics class and skeleton of functions to be populated * fix eol and make destructor of base Diag class virtual * remove redundant query file_prefix in FullDiag class * Apply suggestions from code review adding PR suggestions Co-authored-by: MaxThevenet <mthevenet@lbl.gov> * cleaning function calls and definitions as suggested in review * uncomment the assert. It is valid for FullDiag and BTDiag * Update Source/Diagnostics/Diagnostics.H adding PR suggestion including comments Co-authored-by: MaxThevenet <mthevenet@lbl.gov> * moved InitData to Diagnostics.cpp and call relevant functions from there * moving FilterComputeAndPack to Diagnostics * removing commented out code * remove gpu vector containing map of fields from cc_mf to dst_mf. New diag machinery ensures cc_mf.ncomp == dst_mf.ncomp * fix eol whitespace * populating Init functions in BTDiagnostics * DefineCellCenteredMultiFab should be called only once for all buffers * add Init functions and constructing BackTransform functor * call BackTransform operator which currently does nothing * EOL * average-down cc data to coarsest level * loop over level used to allocate output buffer * some comment about avg-down functions to be moved * EOL * Apply suggestions from code review add pr suggestions Co-authored-by: MaxThevenet <mthevenet@lbl.gov> * clarifying pure virtual and virtual functions * removing namespace amrex and adding corresponding amrex:: prefix * this should be nlev -- number of levels in the sim, not max allocated levels * the number of levels to be output could be different from nlev, nmax_lev * add missing semi-colon! * add amrex:: prefix to MultiFab to comile psatd * add new diag files to CMakeLists.txt * fix compiler errors. * add amrex:: prefix to fix psatd compilation error * some more conflicts * minor cleaning * EOL * flatten cc data using finestLevel() and use moving window dir * EOL * Apply suggestions from code review PR suggestions Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * making it readable and adding comments * add missing semicolon Co-authored-by: MaxThevenet <mthevenet@lbl.gov> Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Summit: no `-n` in jsrun (ECP-WarpX#1114) * Summit: no `-n` in jsrun The calculation of `-n` does not seem to work anymore. Luckily, the parameter seems to be automatically taken by some LFS setting in the job. * Summit Runs: OMP, GPU-Aware MPI, Latency Refs.: - https://jsrunvisualizer.olcf.ornl.gov/?s4f0o11n6c7g1r11d1b1l0= - https://docs.olcf.ornl.gov/systems/summit_user_guide.html#cuda-aware-mpi USE_HDF5 is now supported by AMReX make system (ECP-WarpX#1118) Fixed minor typos and added pytest requirement (ECP-WarpX#1119) For RZ, added assert that periodicity is not set radially (ECP-WarpX#1115) CMake: MPI, Asserts and Advanced AMReX (ECP-WarpX#1120) * CMake: WarpX_MPI Wrap the last user-facing AMReX option into a WarpX option. This reduces confusion and improves consistency further. * CMake: Assertations Enable AMReX' `ENABLE_ASSERTIONS` with build type `Debug` and allow to overwrite on the command line with `-DENABLE_ASSERTIONS=ON` for manual activation with other build types. * CMake: Advanced AMReX Options Declutter user-facing options further. * CMake: Toggle for CMAKE_BUILD_TYPE Add allowed options for toggling the `CMAKE_BUILD_TYPE`. * CMake: Floating Point Exceptions Enable AMReX' `ENABLE_FPE` with build type `Debug` and allow to overwrite on the command line with `-DENABLE_FPE=ON` for manual activation with other build types. Moved doxygen to header file for consistency (ECP-WarpX#1116) [small] Inject Particles from openPMD in RZ (ECP-WarpX#1113) * Added description of singleparticle particle injection style * Revert "Added description of singleparticle particle injection style" This reverts commit f02d842. * Tried to inject particles only in simulation Box * Added Todo for multiple levels of mesh refinement * Explain todo better * Added x and y_shifts and replaced cascading ifs by bools and continue * Replaced cascading ifs by bools and continue * Replaced TODO by FIXME as reviewer suggested * Made the insidebounds function use box limits as defaults * Fixed identation issue * Fixed identation issue * Fixed identation issue * Added to Docs the default for xmin,xmax and other dirs * Fix missing y and z dirs * Update Source/Initialization/PlasmaInjector.cpp * Fixed PlasmaInjector limits with periodic condition * Removed check if 3D because now ylims are +/-inf * Restored if to check for y velocity also in 2D/RZ * Fixed Docs to recent changes * Update Source/Initialization/PlasmaInjector.cpp * Fix extra bracket * Added information about RZ geometry for openPMD external file * Fix to issue without periodic boundaries * Added variable rz_take_absolute, to allow to inject x<0 particles in RZ * Fix to declaration of addplasmafromfile rz_take_absolute * Fix to declaration of addplasmafromfile rz_take_absolute * Fixed documentation to include rz_take_absolute option * Fix merging conflict * Final fix to xmax=inf in RZ * In RZ particles are injected in x,y,z * Fixed remaining rz_take_absolute * Converted (x,y,z) to (r,0,z) * Fixed bounds limits (r,z) but injection is in 3D * Re-fix to insideBounds() * AMReX assert of periodic in RZ * Removed Assert -> went to PR 1115 by @dpgrote * Fixed issues pointed out by reviewer @dpgrote * Undo changes for future PR * Undo changes for future PR Include guard cells for init fields (ECP-WarpX#1123) * add guard cells for initializing EB fields using parser * add bool type for init_guard_cells * fix bugs in initializing Box * removing init_guard_cell flag Removed references to particle E and B from Python (ECP-WarpX#1122) CMake: AMReX Advanced Options (ECP-WarpX#1124) Avoid setting advanced AMReX options with `INTERNAL` as this implies `FORCE` and we cannot overwrite them from the CMake GUIs. fix typo (ECP-WarpX#1130) Add plasma refactor (ECP-WarpX#830) * add 'overlapsWith' methods to InjectorPosition and PlasmaInjector * add helper routine for computing positions within a cell * use new function in AddPlasma * use the XDim3 directly * refactor add plasma to only add particles in cells that could overlap with the plasma region * handle refined injection * account for lorentz tranform in first pass * can't capture statics in device lambda like that * eol * fix logic error * fix RZ compilation * eol * a few docstrings * missed a spot * include the bulk momentum in the first pass * reuse applyBallisticCorrection function where we can * simplify the applyBallisticCorrection function * eol * fix equation for ballistic correction in the gamma_boost > 1 case * need a sync here now * fix typo in docstring * use _rt * add _rt * add some _rt * update the benchmarks because the particle id / cpu numbers (and occassionally the momenta, when that is random) are different now typo in libEnsemble scripts (ECP-WarpX#1129) Improve rho diagnostic (ECP-WarpX#1099) * Fix computation of rho in diagnostics: - new functor RhoFunctor to compute rho; - fix FDTD benchmarks (previously staggered in time by one time step); - fix one PSATD benchmark for single precision test; - update documentation and remove old variable rho_plot from class WarpX; - TODO: check test Python_LaserAccelerationMR on Travis CI; - TODO: check correctness of implementation in RZ geometry. * Fix benchmark for Python_LaserAccelerationMR * Small clean-up * Simplify use of unique pointer to rho * Remove ASSERT: it is valid to write out all modes * Add missing code to write all rho modes in RZ geometry * Clean up following up reviewer's suggestions * Revert Bx benchmark fix for PSATD (needs bug fix?) * Improve docs and comments * Restore old behavior of plot_rho for back-transformed diagnostics * Update Docs/source/visualization/advanced.rst Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Docs: Ascent & Python Summit Updates (ECP-WarpX#1127) * Docs: Ascent & Python Summit Updates Updates the instructions on how to use Python with WarpX, e.g. for libEnsemble runs on Summit. Also updates Ascent location (compiles already with CMake). * libEnsemble: backlink to Summit * Summit libEnsemble: Missing steps Document missing steps to be performed - thx Diana & Maxence! * Summit FC: gfortran * Summit: mpi4py Not yet used with libEnsemble as we use it, but useful for yt post-processing and future workflows. Use PICSAR new development branch (ECP-WarpX#1134) clarify Visualization notebook and fix path in the doc (ECP-WarpX#1137) CMake: DPC++, OMP Def., Binary Names (ECP-WarpX#1125) With regards to the `WarpX_COMPUTE` options the following changes are introduced: - default: `OMP` (as in GNUmake) - rename: `NONE` -> `NOACC` (no accelerated computing backend) - adds: `DPCPP` (for SYCL/Intel DPC++ beta7 or newer) Furthermore, we encode relevant, user-facing options in our binary name. The namings are intentionally a different and skip few options that are verbose and unlikely to appear even in development at the same time. Furthermore, this builds a `warpx` symlink in `bin/` to the latest build executable (when swtiching options), so that repeated compile-and-test workflows do not need to take care of the executable name anymore. Doc: how to compile and run on Juwels (ECP-WarpX#1133) * Doc how to compile and run on Juwels * add a bunch of AMReX-specific options * Update juwels.rst * Update Docs/source/building/juwels.rst Co-authored-by: MaxThevenet <mthevenet@lbl.gov> * Juwels: Filesystem Note * Jewels Docs: Missing newline Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> update release tag in all files on the repo (ECP-WarpX#1138) Tests tests PSATD: current correction works only with global FFTs (ECP-WarpX#1131) * Clean up implementation of current correction: - abort when current correction runs with psatd.periodic_single_box_fft=0 - rename input parameter from psatd.do_current_correction to psatd.current_correction - add nodal tests in 2D and 3D (using direct current deposition) - change 2D test: use Esirkepov deposition, instead of direct, as in 3D test - add/update relevant checksum benchmarks * Small clean-up Moved ShiftGalileanBoundary() call to precede back transformed diagnostics (ECP-WarpX#1126) CMake: Guard CMake 3.17 Feature (ECP-WarpX#1143) Add a feature guard for a CMake 3.17+ feature (CUDA C++ std). assert the particles are positive to check for overflow (ECP-WarpX#1142) Docs: OMP new CMake default (ECP-WarpX#1145) Update the CMake documentation to reflect that OpenMP is enabled by default (as in GNUmake). PSATD: add option to update E without using rho (ECP-WarpX#1128) * Introduce option to update E with/without rho * Clean up * Include equations in docs * Fix EOL whitespaces error * Small clean-up * Clean up Plasma profile regression test EOL Parabolic channel regression test amrex math sqrt input diags file prefix diag --> diag1 Update WarpX-tests.ini step is -1 for diags->ComputeAndFlush at initialization (ECP-WarpX#1148) Raise the stack limit when running with cuda (ECP-WarpX#1154) * raise the stack limit when running with cuda * fix comment + only do this in debug mode Averaged Galilean PSATD (ECP-WarpX#869) * Read Galilean velocity * Prepare structures for Galilean solver * Started implementing Galilean equations * Analytical limits for X1, X2, X3, X4 coefficients added * Slight changes added * Added Galilean position pusher * Scale galilean velocity * Remove unneeded Abort * Fix Galilean pusher * Allocate Theta2 array * Fix definition of coefficients * Increase guard cells for Galilean * Add guard cell in particle exchange * Added modifications for PICSAR galilean branch * Averaged coefficients added * Type corrected * v_gal added to warpx_current_deposition * v_gal added to WarpXParticleContainer.H * Bug fixed - update particle x-position over one time step * Fix issues with merge from dev * Averaged fileds allocated on fine patch (Ex) * Preparation for merging dev into galilean. * remove TABs by hand * Removed a tab. * Redeclared v_galilean as a vector & related changed * Added an automated test * Moved v_galilean inside WarpX constructor * Added analysis script for the automated test * Changed name of the automated test to galilean_psatd * Added InitializeSpectralCoefficients method * Removed temporary comments * Averaged fields added to FiledGather * Added infinite order k[i] * Setting kx_mod( kz_mod) = 0 for index = nx/2(-nz/2) * Adding galilean shift * Implemented galilean shift * Changed method's name from GalileanShift to ShiftGalileanBoundary * Added doxygen string for ShiftGalileanBoundary * Removed never used method LowerCornerWithCentering * Removed temporary comments * Removed dt as a variable from DepositCharge method and its dependencies * Changing type of v_galilean from amrex::Vector to amrex::Array * Changed back ng_fft to be equal nox_fft * Changed v_galilean's type from amrex::vector to amrex::array * Removed type * Removed temporary comments * Added flag 'do_time_averaging' to swith from unaveraged to averaged Galilean PSATD * Updated doxygen * Small fix on using 'do_time_averaging' flag * Remove some commented Print statements * [skip ci] Further cleanup * Fix compilation * Guard cells update of the averaged E,B fields * Corrected included header file accordingly * Removed EOL * Removed EOL * Corrected path of the included header file * Updated choice of the spectral solver. * Cleanup. * use amrex::exp instead of std::exp * no backward FFT for avg fields if avg is off * Need to shift avg fields in MoveWindow * Further cleaning * Added 2D automated test for averaged PSATD * Added automated 2D and 3D tests for averaged PSATD * Removed comments * Added specifications for averaged Galilean PSATD tests. * Bug fixed - do FillBoundary of the averaged fields only if averaged is activated * Do shiftMF of the averaged fields only if fft_do_time_averaging=true * Add checksum json benchmarks for averaged Galilean PSATD * Add missing indentation * Add missing indentation * Updated automated analysis script * Updated diags period * Fixed bug: no backtransform of the averaged fields when WARPX_DIM_RZ is set * Fixed an uninitialized variable * Clean-up * Changed permissions for analysis_avg_*d.py * Compactified variables initialization via conditional assignment * Initialized averaged E_avg, B_avg fields * Updated automated test for 2D averaged galilean PSATD * Updated benchmark for 2D averaged galilean PSATD * Updated automated 2D averaged PSATD test & benchmark * Updated automated 3D averaged PSATD test & benchmark * Fixed typo * Updated benchmark averaged_galilean_3d_psatd * Initialize the averaged fields only if the averaged Galilean PSATD algorithm is enabled * Encreased time step for automated 3D averaged Galilean PSATD (to be equal c*dt=dz while dz/dx=3) * Updated 3D automated test: specified standard deviations `.ux_th, .uy_th, .uz_th` along each direction * Typo from the previous commit: enabled the averaged algorithm * Trying out another input parameters for 3D averaged automated test with v_gal = v_plasma * Update benchmark for previously added 3D averaged automated test * Update 2D averaged automated test and corresponding benchmark * Revert "Updated diags period" This reverts commit 7334729. * Cleanup: removed unwanted comment. * Added descriptions of the 2D and 3D automated tests for the averaged Galilean PSATD. * Updated value calculated via standard Galilean PSATD. * Encreased relative error tolerance for 3D automated test for the averaged Galilean PSATD. * Removed to avoid duplication since it already specified in the corresponding 2D/3D input scripts. * Removed unwanted empty lines * Added spaces after function's names * Removed unwanted empty line * Removed * Fixed indentation * Cleanup: removed #include <math.h> * Cleanup: removed empty lines and fixed indentation * Added 'AvgGalileanAlgorithm.cpp' to 'CMakeLists.txt' Co-authored-by: Remi Lehe <remi.lehe@normalesup.org> Co-authored-by: MaxThevenet <mthevenet@lbl.gov> openPMD: Directory Cleanup, 6 Digits (ECP-WarpX#1149) This reorders the naming of diagnostics to be a bit more readable. Currently it was: ```bash $ ls Bin/diags/ diag1bp/<N>.bp diag1h5/<N>.h5 ... diag100000 diag100100 <N> ... ``` Now it proposes for openPMD output ```bash $ ls Bin/diags/ diag1/openpmd_<N>.bp diag1/openpmd_<N>.h5 ``` This is also a bit easier for some tooling (i.e. yt and openPMD-viewer that expect full consistent dirs atm.) and keeps things nicely together. Also bumps the zero-padding to use consistently 6 digits for iterations in output files (<1M iterations), since sims (ions) with >100k steps are not uncommon. This does not yet change plot files, since this is still requiring to make the regression testing scripts more flexible. Fix bug and test with LaserAccelerationRZ (ECP-WarpX#1150) Fix compilation of AvgGalilean for GPU (ECP-WarpX#1159) Summit Power9 CPUs documentation (ECP-WarpX#1162) * Power9 CPU docs * Update Tools/BatchScripts/batch_summit_power9.sh Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> * Subsection convention Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

ax3l self-requested a review June 17, 2020 01:40

ax3l self-assigned this Jun 17, 2020

ax3l added bug Something isn't working bug: affects latest release Bug also exists in latest release version component: diagnostics all types of outputs labels Jun 17, 2020

ax3l removed their request for review June 17, 2020 01:43

EZoni added 3 commits June 17, 2020 09:58

Fix benchmark for Python_LaserAccelerationMR

c3889df

Merge branch master into rho-diagnostics

a23633d

Small clean-up

6fbd0cf

EZoni changed the title ~~[WIP] Improve rho diagnostic~~ Improve rho diagnostic Jun 17, 2020

EZoni requested review from MaxThevenet, RevathiJambunathan, ax3l and dpgrote June 17, 2020 22:18

EZoni marked this pull request as ready for review June 17, 2020 22:39

dpgrote reviewed Jun 18, 2020

View reviewed changes

EZoni added 3 commits June 18, 2020 08:32

Simplify use of unique pointer to rho

0b1b54a

Remove ASSERT: it is valid to write out all modes

a169cdd

Add missing code to write all rho modes in RZ geometry

702ca97

dpgrote reviewed Jun 18, 2020

View reviewed changes

ax3l assigned dpgrote Jun 18, 2020

ax3l changed the base branch from master to development June 18, 2020 17:24

EZoni added 3 commits June 18, 2020 14:08

Clean up following up reviewer's suggestions

789c9e9

Merge branch development into rho-diagnostics

8836a80

Revert Bx benchmark fix for PSATD (needs bug fix?)

beaaa1f

dpgrote approved these changes Jun 19, 2020

View reviewed changes

ax3l reviewed Jun 20, 2020

View reviewed changes

MaxThevenet changed the title ~~Improve rho diagnostic~~ [WIP] Improve rho diagnostic Jun 22, 2020

EZoni added 3 commits June 22, 2020 10:03

Improve docs and comments

09295a4

Merge branch development into rho-diagnostics

1c3ba57

Restore old behavior of plot_rho for back-transformed diagnostics

6e337dc

EZoni changed the title ~~[WIP] Improve rho diagnostic~~ Improve rho diagnostic Jun 23, 2020

RevathiJambunathan approved these changes Jun 23, 2020

View reviewed changes

ax3l reviewed Jun 26, 2020

View reviewed changes

Docs/source/visualization/advanced.rst Outdated Show resolved Hide resolved

EZoni and others added 2 commits June 26, 2020 08:37

Update Docs/source/visualization/advanced.rst

d1666b2

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

Merge branch development into rho-diagnostics

40666d2

EZoni requested a review from ax3l June 26, 2020 17:06

ax3l approved these changes Jun 30, 2020

View reviewed changes

ax3l merged commit 0107878 into ECP-WarpX:development Jun 30, 2020

EZoni deleted the rho-diagnostics branch July 1, 2020 21:11

ax3l mentioned this pull request Jul 4, 2020

LWFA: Diag add Charge Density #1048

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve rho diagnostic #1099

Improve rho diagnostic #1099

EZoni commented Jun 17, 2020 •

edited

Loading

dpgrote Jun 18, 2020

dpgrote Jun 18, 2020

dpgrote Jun 18, 2020

ax3l Jun 30, 2020

dpgrote commented Jun 18, 2020

dpgrote Jun 18, 2020

dpgrote Jun 18, 2020

dpgrote Jun 18, 2020

dpgrote Jun 18, 2020

dpgrote commented Jun 18, 2020

EZoni commented Jun 18, 2020 •

edited

Loading

EZoni commented Jun 18, 2020

dpgrote left a comment

dpgrote commented Jun 19, 2020

EZoni commented Jun 19, 2020

ax3l Jun 20, 2020

MaxThevenet commented Jun 22, 2020

RevathiJambunathan commented Jun 22, 2020

EZoni commented Jun 22, 2020

RevathiJambunathan commented Jun 23, 2020

ax3l left a comment

Improve rho diagnostic #1099

Improve rho diagnostic #1099

Conversation

EZoni commented Jun 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpgrote commented Jun 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpgrote commented Jun 18, 2020

EZoni commented Jun 18, 2020 • edited Loading

EZoni commented Jun 18, 2020

dpgrote left a comment

Choose a reason for hiding this comment

dpgrote commented Jun 19, 2020

EZoni commented Jun 19, 2020

Choose a reason for hiding this comment

MaxThevenet commented Jun 22, 2020

RevathiJambunathan commented Jun 22, 2020

EZoni commented Jun 22, 2020

RevathiJambunathan commented Jun 23, 2020

ax3l left a comment

Choose a reason for hiding this comment

EZoni commented Jun 17, 2020 •

edited

Loading

EZoni commented Jun 18, 2020 •

edited

Loading