ModernElectron · roelof-groenewald · Feb 19, 2022 · Feb 8, 2022 · Feb 9, 2022 · Feb 10, 2022
diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml
@@ -78,7 +78,7 @@ jobs:
         which nvcc || echo "nvcc not in PATH!"
 
         git clone https://github.com/AMReX-Codes/amrex.git ../amrex
-        cd amrex && git checkout --detach 212ffa61c1651f041393f46efbfccaabd29be3e8 && cd -
+        cd amrex && git checkout --detach d618810d677d485f48e8de2ae12cb3e9c79ba3c1 && cd -
         make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE -j 2
 
   build_nvhpc21-9-nvcc:

diff --git a/.github/workflows/insitu.yml b/.github/workflows/insitu.yml
@@ -38,9 +38,9 @@ jobs:
     env:
       CXX: g++
       CC: gcc
-      CMAKE_PREFIX_PATH: /ascent/install-debug/lib/cmake/
+      CMAKE_PREFIX_PATH: /ascent/install/lib/cmake/
     container:
-      image: alpinedav/ascent:ascent-ubuntu-18_2021-11-02-sha458045
+      image: alpinedav/ascent:0.8.0
     steps:
     - uses: actions/checkout@v2
     - name: Configure

diff --git a/Docs/source/developers/particles.rst b/Docs/source/developers/particles.rst
@@ -107,19 +107,49 @@ To reduce numerical artifacts at the boundary of a mesh-refinement patch, WarpX
 
    Buffers are complex!
 
-Runtime particle attributes
----------------------------
-
-WarpX allows extra runtime attributes to be added to particle containers (through ``AddRealComp("attrname")`` or ``AddIntComp("attrname")``). The attribute name can then be used to access the values of that attribute. For example, using a particle iterator, ``pti``, to loop over the particles the command ``pti.GetAttribs(particle_comps["attrname"]).dataPtr();`` will return the values of the ``"attrname"`` attribute.
-
-Commonly used runtime attributes are described in the table below.
+Particle attributes
+-------------------
 
-==================  ================  =================================  =============
+WarpX adds the following particle attributes by default to WarpX particles.
+These attributes are either stored in an Array-of-Struct (AoS) or Struct-of-Array (SoA) location of the AMReX particle containers.
+The data structures for those are either pre-described at compile-time (CT) or runtime (RT).
+
+====================  ================  ==================================  ===== ==== =====================
+Attribute name        ``int``/``real``  Description                         Where When Notes
+====================  ================  ==================================  ===== ==== =====================
+``position_x/y/z``    ``real``          Particle position.                  AoS   CT
+``cpu``               ``int``           CPU index where the particle        AoS   CT
+                                        was created.
+``id``                ``int``           CPU-local particle index            AoS   CT
+                                        where the particle was created.
+``ionization_level``  ``int``           Ion ionization level                SoA   RT   Added when ionization
+                                                                                       physics is used.
+``opticalDepthQSR``   ``real``          QED: optical depth of the Quantum-  SoA   RT   Added when PICSAR QED
+                                        Synchrotron process                            physics is used.
+``opticalDepthBW``    ``real``          QED: optical depth of the Breit-    SoA   RT   Added when PICSAR QED
+                                        Wheeler process                                physics is used.
+====================  ================  ==================================  ===== ==== =====================
+
+WarpX allows extra runtime attributes to be added to particle containers (through ``AddRealComp("attrname")`` or ``AddIntComp("attrname")``).
+The attribute name can then be used to access the values of that attribute.
+For example, using a particle iterator, ``pti``, to loop over the particles the command ``pti.GetAttribs(particle_comps["attrname"]).dataPtr();`` will return the values of the ``"attrname"`` attribute.
+
+User-defined integer or real attributes are initialized when particles are generated in ``AddPlasma()``.
+The attribute is initialized with a required user-defined parser function.
+Please see the :ref:`input options <running-cpp-parameters-particle>` ``addIntegerAttributes`` and ``addRealAttributes`` for a user-facing documentation.
+
+Commonly used runtime attributes are described in the table below and are all part of SoA particle storage:
+
+==================  ================  =================================  ==============
 Attribute name      ``int``/``real``  Description                        Default value
-==================  ================  =================================  =============
-``prev_x/y/z``      ``real``          The coordinates of the particles
+==================  ================  =================================  ==============
+``prev_x/y/z``      ``real``          The coordinates of the particles   *user-defined*
                                       at the previous timestep.
-==================  ================  =================================  =============
+``orig_x/y/z``      ``real``          The coordinates of the particles   *user-defined*
+                                      when they were created.
+==================  ================  =================================  ==============
+
+A Python example that adds runtime options can be found in :download:`Examples/Tests/ParticleDataPython <../../../Examples/Tests/ParticleDataPython/PICMI_inputs_prev_pos_2d.py>`
 
 .. note::
 

diff --git a/Docs/source/install/cmake.rst b/Docs/source/install/cmake.rst
@@ -269,6 +269,8 @@ Environment Variable          Default & Values                             Descr
 ``HDF5_USE_STATIC_LIBRARIES`` ON/**OFF**                                   Prefer static libraries for HDF5 dependency (openPMD)
 ``ADIOS_USE_STATIC_LIBS``     ON/**OFF**                                   Prefer static libraries for ADIOS1 dependency (openPMD)
 ``WARPX_AMREX_SRC``           *None*                                       Absolute path to AMReX source directory (preferred if set)
+``WARPX_AMREX_REPO``          *None (uses cmake default)*                  Repository URI to pull and build AMReX from
+``WARPX_AMREX_BRANCH``        *None (uses cmake default)*                  Repository branch for ``WARPX_AMREX_REPO``
 ``WARPX_AMREX_INTERNAL``      **ON**/OFF                                   Needs a pre-installed AMReX library if set to ``OFF``
 ``WARPX_OPENPMD_SRC``         *None*                                       Absolute path to openPMD-api source directory (preferred if set)
 ``WARPX_OPENPMD_INTERNAL``    **ON**/OFF                                   Needs a pre-installed openPMD-api library if set to ``OFF``

diff --git a/Docs/source/install/hpc.rst b/Docs/source/install/hpc.rst
@@ -34,6 +34,7 @@ HPC Systems
    hpc/quartz
    hpc/lawrencium
    hpc/ookami
+   hpc/lxplus
 
 .. tip::
 

diff --git a/Docs/source/install/hpc/lxplus.rst b/Docs/source/install/hpc/lxplus.rst
@@ -0,0 +1,155 @@
+.. _building-lxplus:
+
+LXPLUS (CERN)
+=============
+
+The LXPLUS cluster is located at CERN.
+
+* `Lxplus documentation <https://lxplusdoc.web.cern.ch>`__
+* Batch system: `HTCondor <https://batchdocs.web.cern.ch/index.html>`__
+* Filesystem locations:
+    * User folder: ``/afs/cern.ch/user/<a>/<account>`` (10GByte)
+    * Work folder: ``/afs/cern.ch/work/<a>/<account>`` (100GByte)
+    * Eos storage: ``/eos/home-<a>/<account>`` (1T)
+
+Through LXPLUS we have access to CPU and GPU nodes (the latter equipped with NVIDIA V100 and T4 GPUs).
+
+Installation
+------------
+Only very little software is pre-installed on LXPLUS so we show how to install from scratch all the dependencies using `Spack <https://spack.io>`__.
+
+For size reasons it is not advisable to install WarpX in the ``$HOME`` directory, while it should be installed in the "work directory". For this purpose we set an environment variable with the path to the "work directory"
+
+.. code-block:: bash
+
+    export WORK=/afs/cern.ch/work/${USER:0:1}/$USER/
+
+We clone WarpX in ``$WORK``:
+
+.. code-block:: bash
+
+    cd $WORK
+    git clone https://github.com/ECP-WarpX/WarpX.git warpx
+
+Installation profile file
+^^^^^^^^^^^^^^^^^^^^^^^^^
+The easiest way to install the dependencies is to use the pre-prepared ``warpx.profile`` as follows:
+
+.. code-block:: bash
+
+    cp $WORK/warpx/WarpX/Tools/machines/lxplus-cern/lxplus_warpx.profile.example $WORK/lxplus_warpx.profile
+    source $WORK/lxplus_warpx.profile
+
+When doing this one can directly skip to the :ref:`Building WarpX <building-lxplus-warpx>` section.
+
+To have the environment activated at every login it is then possible to add the following lines to the ``.bashrc``
+
+.. code-block:: bash
+
+    export WORK=/afs/cern.ch/work/${USER:0:1}/$USER/
+    source $WORK/lxplus_warpx.profile
+
+GCC
+^^^
+The pre-installed GNU compiler is outdated so we need a more recent compiler. Here we use the gcc 11.2.0 from the LCG project, but other options are possible.
+
+We activate it by doing
+
+.. code-block:: bash
+
+    source /cvmfs/sft.cern.ch/lcg/releases/gcc/11.2.0-ad950/x86_64-centos7/setup.sh
+
+In order to avoid using different compilers this line could be added directly into the ``$HOME/.bashrc`` file.
+
+Spack
+^^^^^
+We download and activate Spack in ``$WORK``:
+
+.. code-block:: bash
+
+    cd $WORK
+    git clone -c feature.manyFiles=true https://github.com/spack/spack.git
+    source spack/share/spack/setup-env.sh
+
+Now we add our gcc 11.2.0 compiler to spack:
+
+.. code-block:: bash
+
+    spack compiler find /cvmfs/sft.cern.ch/lcg/releases/gcc/11.2.0-ad950/x86_64-centos7/bin
+
+Installing the Dependencies
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To install the dependencies we create a virtual environment, which we call ``warpx-lxplus``:
+
+.. code-block:: bash
+
+    spack env create warpx-lxplus $WORK/WarpX/Tools/machines/lxplus-cern/spack.yaml
+    spack env activate warpx-lxplus
+    spack install
+
+If the GPU support or the Python bindings are not needed, it's possible to skip the installation by respectively setting
+the following environment variables export ``SPACK_STACK_USE_PYTHON=0`` and ``export SPACK_STACK_USE_CUDA = 0`` before
+running the previous commands.
+
+After the installation is done once, all we need to do in future sessions is just ``activate`` the environment again:
+
+.. code-block:: bash
+
+    spack env activate warpx-lxplus
+
+The environment ``warpx-lxplus`` (or ``-cuda`` or ``-cuda-py``) must be reactivated everytime that we log in so it could
+be a good idea to add the following lines to the ``.bashrc``:
+
+.. code-block:: bash
+
+    source $WORK/spack/share/spack/setup-env.sh
+    spack env activate -d warpx-lxplus
+    cd $HOME
+
+.. _building-lxplus-warpx:
+
+Building WarpX
+^^^^^^^^^^^^^^
+
+We prepare and load the Spack software environment as above.
+Then we build WarpX:
+
+.. code-block:: bash
+
+    cmake -S . -B build
+    cmake --build build -j 6
+
+Or if we need to compile with CUDA:
+
+.. code-block:: bash
+
+    cmake -S . -B build -DWarpX_COMPUTE=CUDA
+    cmake --build build -j 6
+
+Python Bindings
+^^^^^^^^^^^^^^^
+
+Here we assume that a Python interpreter has been set up as explained previously.
+
+Now, ensure Python tooling is up-to-date:
+
+.. code-block:: bash
+
+   python3 -m pip install -U pip setuptools wheel
+
+Then we compile WarpX as in the previous section (with or without CUDA) adding ``-DWarpX_LIB=ON`` and then we install it into our Python:
+
+.. code-block:: bash
+
+   cmake -S . -B build -DWarpX_COMPUTE=CUDA -DWarpX_LIB=ON
+   cmake --build build --target pip_install -j 6
+
+This builds WarpX for 3D geometry.
+
+Alternatively, if you like to build WarpX for all geometries at once, use:
+
+.. code-block:: bash
+
+   BUILD_PARALLEL=6 python3 -m pip wheel .
+   python3 -m pip install pywarpx-*whl
diff --git a/Docs/source/usage/examples.rst b/Docs/source/usage/examples.rst
@@ -100,7 +100,7 @@ Test cases
 
 PICMI (Python) test cases included that can be used as a reference:
 
-* :download:`Gaussian beam <../../../Examples//Modules/gaussian_beam/PICMI_inputs_gaussian_beam.py>`
+* :download:`Gaussian beam <../../../Examples/Modules/gaussian_beam/PICMI_inputs_gaussian_beam.py>`
 * :download:`Langmuir plasma wave test in 3d <../../../Examples//Tests/Langmuir/PICMI_inputs_langmuir_rt.py>`
 * :download:`Langmuir plasma wave test in RZ <../../../Examples//Tests/Langmuir/PICMI_inputs_langmuir_rz_multimode_analyze.py>`
 * :download:`Langmuir plasma wave test in 2D <../../../Examples//Tests/Langmuir/PICMI_inputs_langmuir2d.py>`

diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst
@@ -610,9 +610,11 @@ Particle initialization
       The density specified by the density profile is interpreted to have the units of #/m^2/s.
       This requires the additional parameters:
       ``<species_name>.surface_flux_pos`` (`double`, location of the injection plane [meter])
-      ``<species_name>.flux_normal_axis`` (`x`, `y`, or `z` for 3D, `x` or `z` for 2D, or `r` or `z` for RZ)
+      ``<species_name>.flux_normal_axis`` (`x`, `y`, or `z` for 3D, `x` or `z` for 2D, or `r`, `t`, or `z` for RZ. When `flux_normal_axis` is `r` or `t`, the `x` and `y` components of the user-specified momentum distribution are interpreted as the `r` and `t` components respectively)
       ``<species_name>.flux_direction`` (`-1` or `+1`, direction of flux relative to the plane)
       ``<species_name>.num_particles_per_cell`` (`double`)
+      ``<species_name>.flux_tmin`` (`double`, Optional time at which the flux will be turned on. Ignored when negative.)
+      ``<species_name>.flux_tmax`` (`double`, Optional time at which the flux will be turned off. Ignored when negative.)
 
     * ``none``: Do not inject macro-particles (for example, in a simulation that starts with neutral, ionizable atoms, one may want to create the electrons species -- where ionized electrons can be stored later on -- without injecting electron macro-particles).
 
@@ -857,6 +859,39 @@ Particle initialization
     If `1` is given, this species will not be pushed
     by any pusher during the simulation.
 
+* ``<species_name>.addIntegerAttributes`` (list of `string`)
+    User-defined integer particle attribute for species, ``species_name``.
+    These integer attributes will be initialized with user-defined functions
+    when the particles are generated.
+    If the user-defined integer attribute is ``<int_attrib_name>`` then the
+    following required parameter must be specified to initialize the attribute.
+    * ``<species_name>.attribute.<int_attrib_name>(x,y,z,ux,uy,uz,t)`` (`string`)
+    ``t`` represents the physical time in seconds during the simulation.
+    ``x``, ``y``, ``z`` represent particle positions in the unit of meter.
+    ``ux``, ``uy``, ``uz`` represent the particle velocities in the unit of
+    :math:`\gamma v/c`, where
+    :math:`\gamma` is the Lorentz factor,
+    :math:`v/c` is the particle velocity normalized by the speed of light.
+    E.g. If ``electrons.addIntegerAttributes = upstream``
+    and ``electrons.upstream(x,y,z,ux,uy,uz,t) = (x>0.0)*1`` is provided
+    then, an integer attribute ``upstream`` is added to all electron particles
+    and when these particles are generated, the particles with position less than ``0``
+    are assigned a value of ``1``.
+
+* ``<species_name>.addRealAttributes`` (list of `string`)
+    User-defined real particle attribute for species, ``species_name``.
+    These real attributes will be initialized with user-defined functions
+    when the particles are generated.
+    If the user-defined real attribute is ``<real_attrib_name>`` then the
+    following required parameter must be specified to initialize the attribute.
+    * ``<species_name>.attribute.<real_attrib_name>(x,y,z,ux,uy,uz,t)`` (`string`)
+    ``t`` represents the physical time in seconds during the simulation.
+    ``x`, ``y``, ``z` represent particle positions in the unit of meter.
+    ``ux``, ``uy``, ``uz`` represent the particle velocities in the unit of
+    :math:`\gamma v/c`, where
+    :math:`\gamma` is the Lorentz factor,
+    :math:`v/c` is the particle velocity normalized by the speed of light.
+
 * ``<species>.save_particles_at_xlo/ylo/zlo``,  ``<species>.save_particles_at_xhi/yhi/zhi`` and ``<species>.save_particles_at_eb`` (`0` or `1` optional, default `0`)
     If `1` particles of this species will be copied to the scraped particle
     buffer for the specified boundary if they leave the simulation domain in
@@ -1857,6 +1892,21 @@ In-situ capabilities can be used by turning on Sensei or Ascent (provided they a
         <diag_name>.adios2_operator.type = zfp
         <diag_name>.adios2_operator.parameters.precision = 3
 
+* ``<diag_name>.adios2_engine.type`` (``bp4``, ``sst``, ``ssc``, ``dataman``) optional,
+    `ADIOS2 Engine type <https://openpmd-api.readthedocs.io/en/0.14.0/details/backendconfig.html#adios2>`__ for `openPMD <https://www.openPMD.org>`_ data dumps.
+    See full list of engines at `ADIOS2 readthedocs <https://adios2.readthedocs.io/en/latest/engines/engines.html>`__
+
+* ``<diag_name>.adios2_engine.parameters.*`` optional,
+    `ADIOS2 Engine parameters <https://openpmd-api.readthedocs.io/en/0.14.0/details/backendconfig.html#adios2>`__ for `openPMD <https://www.openPMD.org>`_ data dumps.
+
+    An example for parameters for the BP engine are setting the number of writers (``NumAggregators``), transparently redirecting data to burst buffers etc.
+    A detailed list of engine-specific parameters are available at the official `ADIOS2 documentation <https://adios2.readthedocs.io/en/latest/engines/engines.html>`__
+
+    .. code-block:: text
+
+        <diag_name>.adios2_engine.parameter.NumAggregators = 2048
+        <diag_name>.adios2_engine.parameters.BurstBufferPath="/mnt/bb/username"
+
 * ``<diag_name>.fields_to_plot`` (list of `strings`, optional)
     Fields written to output.
     Possible values: ``Ex`` ``Ey`` ``Ez`` ``Bx`` ``By`` ``Bz`` ``jx`` ``jy`` ``jz`` ``part_per_cell`` ``rho`` ``phi`` ``F`` ``part_per_grid`` ``divE`` ``divB`` and ``rho_<species_name>``, where ``<species_name>`` must match the name of one of the available particle species. Note that ``phi`` will only be written out when do_electrostatic==labframe.
@@ -2163,6 +2213,7 @@ Reduced Diagnostics
         **Plane**: probe a 2 dimensional plane of points to create a square plane detector.
         Initial input parameters ``x_probe``, ``y_probe``, and ``z_probe`` designate the center of the detector.
         The detector plane is normal to a vector specified by ``<reduced_diags_name>.target_normal_x``, ``<reduced_diags_name>.target_normal_y``, and ``<reduced_diags_name>.target_normal_z``.
+        Note that it is not necessary to specify the ``target_normal`` vector in a 2D simulation (the only supported normal is in ``y``).
         The top of the plane is perpendicular to an "up" vector denoted by ``<reduced_diags_name>.target_up_x``, ``<reduced_diags_name>.target_up_y``, and ``<reduced_diags_name>.target_up_z``.
         The detector has a square radius to be determined by ``<reduced_diags_name>.detector_radius``.
         Similarly to the line detector, the plane detector requires a resolution ``<reduced_diags_name>.resolution``, which denotes the number of detector particles along each side of the square detector.
@@ -2187,6 +2238,11 @@ Reduced Diagnostics
         Integrated electric and magnetic field components can instead be obtained by specifying
         ``<reduced_diags_name>.integrate == true``.
 
+        .. warning::
+
+           The FieldProbe reduced diagnostic does not yet add a Lorentz back transformation for boosted frame simulations.
+           Thus, it records field data in the boosted frame, not (yet) in the lab frame.
+
     * ``RhoMaximum``
         This type computes the maximum and minimum values of the total charge density as well as
         the maximum absolute value of the charge density of each charged species.

diff --git a/Examples/Physics_applications/laser_acceleration/inputs_1d b/Examples/Physics_applications/laser_acceleration/inputs_1d
@@ -45,6 +45,10 @@ electrons.profile = constant
 electrons.density = 2.e23  # number of electrons per m^3
 electrons.momentum_distribution_type = "at_rest"
 electrons.do_continuous_injection = 1
+electrons.addRealAttributes = orig_z
+electrons.attribute.orig_z(x,y,z,ux,uy,uz,t) = "z"
+electrons.addIntegerAttributes = regionofinterest
+electrons.attribute.regionofinterest(x,y,z,ux,uy,uz,t) = " (z>12.0e-6) * (z<13.0e-6)"
 
 #################################
 ############ PLASMA #############