Catalyst2 support (ECP-WarpX#5155)

* Added Catalyst support

- Created a parent FlushFormat for Catalyst and Ascent

- Added Catalyst documentation
	- Fixed documentation mistake

- Added mesh (electrostatic sphere) and particle (ioniaztion) examples

- Fixed Catalyst particle striding error

* FlushFormatCatalyst merge fixes

* FlushFormatCatalyst: Support compiling without conduit

* FlushFormatInSitu fix warnings

* cmake: Add WarpX_CATALYST option

Created based on the logic of WarpX_ASCENT

* ci: Add catalyst job to test WarpX integration

* ci: fix style job

- Update wrongFileNameInExamples script to support catalyst scripts to satisfy "Proper file names in Examples" task
- move catalyst-specific options to insitu.yaml instead of separate input file to satisfy "Examples are tested" task

* ci: constrain OMP threads in insitu

Avoids oversubscription in GH runners see ECP-WarpX#5155 (comment)

* docs: Fix spacing in documentation

* catalyst: update compatibility version

* ci: disable ruff F403 warning for catalyst scripts

Using paraview.simple as such in a catalyst script is a common practice

* docs: use anonymous links

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Docs: Catalyst in TOC

Include in TOC to ensure docs get rendered.

---------

Co-authored-by: Andrew Combs <andrew.combs@kitware.com>
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

.github/workflows/insitu.yml

@@ -17,6 +17,7 @@ jobs:
       CXXFLAGS: "-Werror -Wshadow -Woverloaded-virtual -Wunreachable-code -Wno-error=pass-failed"
       CMAKE_GENERATOR: Ninja
       CMAKE_PREFIX_PATH: /root/install/sensei/v4.0.0/lib64/cmake
+      OMP_NUM_THREADS: 1
     container:
       image: senseiinsitu/ci:fedora35-amrex-20220613
     steps:
@@ -38,6 +39,7 @@ jobs:
       CXX: g++
       CC: gcc
       CMAKE_PREFIX_PATH: /ascent/install/lib/cmake/
+      OMP_NUM_THREADS: 1
     container:
       image: alpinedav/ascent:0.9.2
     steps:
@@ -68,3 +70,56 @@ jobs:
           *.png
           conduit_*
         if-no-files-found: error
+
+  catalyst:
+    name: Catalyst
+    runs-on: ubuntu-20.04
+    if: github.event.pull_request.draft == false
+    env:
+      CXX: g++
+      CC: gcc
+      CMAKE_PREFIX_PATH: "/opt/conduit:/opt/catalyst"
+      CATALYST_DEBUG: 1
+      CATALYST_IMPLEMENTATION_PATHS: /opt/paraview/lib/catalyst
+      OMP_NUM_THREADS: 1
+
+    container:
+      image: kitware/paraview:ci-catalyst-amrex-warpx-20240701
+    steps:
+    - uses: actions/checkout@v4
+    - name: Configure
+      run: |
+        cmake -S . -B build   \
+            -DWarpX_DIMS="2;3" \
+            -DWarpX_CATALYST=ON
+    - name: Build
+      run: |
+        cmake --build build -j 10
+    - name: 2D Test
+      run: |
+        cp Examples/Tests/ionization/inputs_2d_rt .
+        cp Examples/Tests/ionization/catalyst_pipeline.py .
+        mpiexec -n 2 ./build/bin/warpx.2d \
+          inputs_2d_rt \
+            catalyst.script_paths = catalyst_pipeline.py\
+            catalyst.implementation = paraview\
+            diag1.intervals = 16\
+            diag1.species = "electrons ions"\
+            diag1.format = catalyst
+    - name: 3D Test
+      run: |
+        cp Examples/Tests/electrostatic_sphere/inputs_3d .
+        cp Examples/Tests/electrostatic_sphere/catalyst_pipeline.py .
+        mpiexec -n 2 ./build/bin/warpx.3d \
+          inputs_3d \
+          catalyst.script_paths = catalyst_pipeline.py \
+          catalyst.implementation = paraview \
+          diagnostics.diags_names = diag1 \
+          diag1.format = catalyst\
+          diag1.intervals = 3
+    - uses: actions/upload-artifact@v4
+      with:
+        name: catalyst-test-artifacts
+        path: |
+          datasets/*.png
+        if-no-files-found: error

.github/workflows/source/wrongFileNameInExamples

@@ -17,6 +17,7 @@ do
     if [[ ${file:0:6 } != inputs       ]] &&
        [[ ${file:0:12} != PICMI_inputs ]] &&
        [[ ${file:0:8 } != analysis     ]] &&
+       [[ ${file:0:8 } != catalyst     ]] &&
        [[ ${file:  -4} != yaml         ]] &&
        [[ ${file:0:4 } != plot         ]] &&
        [[ ${file:0:6 } != README       ]]

CMakeLists.txt

@@ -74,6 +74,7 @@ set_default_install_dirs()
 include(CMakeDependentOption)
 option(WarpX_APP           "Build the WarpX executable application"     ON)
 option(WarpX_ASCENT        "Ascent in situ diagnostics"                 OFF)
+option(WarpX_CATALYST      "Catalyst in situ diagnostics"               OFF)
 option(WarpX_EB            "Embedded boundary support"                  OFF)
 option(WarpX_LIB           "Build WarpX as a library"                   OFF)
 option(WarpX_MPI           "Multi-node support (message-passing)"       ON)

Docs/source/dataanalysis/catalyst.rst

@@ -0,0 +1,167 @@
+.. _visualization-catalyst:
+
+In situ Visualization with Catalyst 2
+=====================================
+Catalyst 2 (further referred to as just Catalyst) is a lightweight in-situ visualization and analysis framework API developed for simulations and other scientific data producers. It has a lightweight implementation
+(or **stub**) and an SDK to develop custom implementations of Catalyst. ParaView comes with its own implementation (known as **ParaView Catalyst**) for leveraging ParaView's
+visualization and analysis capabilities, which is what this document will focus on.
+
+
+Enabling Catalyst
+-----------------
+In order to use Catalyst with WarpX, you must `build Catalyst 2 <https://catalyst-in-situ.readthedocs.io/en/latest/build_and_install.html>`_ and `build <https://github.com/Kitware/ParaView/blob/master/Documentation/dev/build.md>`__ or `install <https://www.paraview.org/download/>`__ ParaView 5.9+. Afterward, AMReX must be built with ``AMReX_CONDUIT=TRUE``,
+``AMReX_CATALYST=TRUE``, ``Conduit_DIR=/path/to/conduit``, and ``Catalyst_DIR=/path/to/catalyst`` (``/path/to/catalyst`` should be the directory containing ``catalyst-config.cmake``, not the path to the implementation).
+
+Once AMReX is appropriately built, WarpX can be built with the following options:
+
+.. code-block:: cmake
+
+    WarpX_amrex_internal=FALSE
+    AMReX_DIR="/path/to/amrex/build"
+
+If they cannot be found, ``Conduit_DIR`` and ``Catalyst_DIR`` will have to be set again. Ensure that AMReX is built with all required options, some common ones being:
+
+.. code-block:: cmake
+
+    AMReX_MPI=TRUE
+    AMReX_MPI_THREAD_MULTIPLE=TRUE
+    AMReX_LINEAR_SOLVERS=TRUE
+    AMReX_PARTICLES=TRUE
+    AMReX_PARTICLES_PRECISION=DOUBLE
+    AMReX_PIC=TRUE
+    AMReX_TINY_PROFILE=TRUE
+
+
+Inputs File Configuration
+-------------------------
+Once WarpX has been compiled with Catalyst support, it will need to be enabled and configured at runtime.
+This is done using our usual inputs file (read with ``amrex::ParmParse``).
+The supported parameters are part of the :ref:`FullDiagnostics <running-cpp-parameters-diagnostics>` with ``<diag_name>.format`` parameter set to ``catalyst``.
+
+In addition to configuring the diagnostics, the following parameters must be included:
+    * ``catalyst.script_paths``: The locations of the pipeline scripts, separated by either a colon or semicolon (e.g. ``/path/to/script1.py;/path/to/script2.py``).
+    * ``catalyst.implementation`` (default ``paraview``): The name of the implementation being used (case sensitive).
+    * ``catalyst.implementation_search_paths``: The locations to search for the given implementation. The specific file being searched for will be ``catalyst_{implementation}.so``.
+
+Because the scripts and implementations are global, Catalyst does not benefit from nor differentiate between multiple diagnostics.
+
+
+Visualization/Analysis Pipeline Configuration
+---------------------------------------------
+Catalyst uses the files specified in ``catalyst.script_paths`` to run all analysis.
+
+The following script, :code:`simple_catalyst_pipeline.py`, automatically detects the type of data for both the mesh and particles, then creates an extractor for them. In most
+cases, these will be saved as ``.VTPC`` files which can be read with the ``XML Partitioned Dataset Collection Reader``.
+
+.. code-block:: python
+
+    from paraview.simple import *
+    from paraview import catalyst
+
+    # Helper function
+    def create_extractor(data_node, filename="Dataset"):
+        VTK_TYPES = ["vtkImageData", "vtkRectilinearGrid", "vtkStructuredGrid", "vtkPolyData", "vtkUnstructuredGrid", "vtkUniformGridAMR", "vtkMultiBlockDataSet", "vtkPartitionedDataSet", "vtkPartitionedDataSetCollection", "vtkHyperTreeGrid"]
+        FILE_ASSOCIATIONS = ["VTI", "VTR", "VTS", "VTP", "VTU", "VTH", "VTM", "VTPD", "VTPC", "HTG"]
+        clientside_data = data_node.GetClientSideObject().GetOutputDataObject(0) # Gets the dataobject from the default output port
+
+        # Loop is required because .IsA() detects valid classes that inherit from the VTK_TYPES
+        for i, vtk_type in enumerate(VTK_TYPES):
+            if (clientside_data.IsA(vtk_type)):
+                filetype = FILE_ASSOCIATIONS[i]
+                extractor = CreateExtractor(filetype, data_node, registrationName=f"_{filetype}")
+                extractor.Writer.FileName = filename + "_{timestep:}" + f".{filetype}"
+                return extractor
+
+        raise RuntimeError(f"Unsupported data type: {clientside_data.GetClassName()}")
+
+    # Camera settings
+    paraview.simple._DisableFirstRenderCameraReset()  # Prevents the camera from being shown
+
+    # Options
+    options = catalyst.Options()
+
+    options.CatalystLiveTrigger = "TimeStep"  # "Python", "TimeStep", "TimeValue"
+    options.EnableCatalystLive = 0  # 0 (disabled), 1 (enabled)
+    if (options.EnableCatalystLive == 1):
+        options.CatalystLiveURL = "localhost:22222"  # localhost:22222 is default
+
+    options.ExtractsOutputDirectory = "datasets"  # Base for where all files are saved
+    options.GenerateCinemaSpecification = 0 # 0 (disabled), 1 (enabled), generates additional descriptor files for cinema exports
+    options.GlobalTrigger = "TimeStep"  # "Python", "TimeStep", "TimeValue"
+
+    meshSource = PVTrivialProducer(registrationName="mesh")  # "mesh" is the node where the mesh data is stored
+    create_extractor(meshSource, filename="meshdata")
+    particleSource = PVTrivialProducer(registrationName="particles")  # "particles" is the node where particle data is stored
+    create_extractor(particleSource, filename="particledata")
+
+    # Called on catalyst initialize (after Cxx side initialize)
+    def catalyst_initialize():
+        return
+
+    # Called on catalyst execute (after Cxx side update)
+    def catalyst_execute(info):
+        print(f"Time: {info.time}, Timestep: {info.timestep}, Cycle: {info.cycle}")
+        return
+
+    # Callback if global trigger is set to "Python"
+    def is_activated(controller):
+        return True
+
+    # Called on catalyst finalize (after Cxx side finalize)
+    def catalyst_finalize():
+        return
+
+    if __name__ == '__main__':
+        paraview.simple.SaveExtractsUsingCatalystOptions(options)
+
+
+For the case of ParaView Catalyst, pipelines are run with ParaView's included ``pvbatch`` executable and use the ``paraview`` library to modify the data. While pipeline scripts
+could be written manually, this is not advised for anything beyond the script above. It is much more practical to use ParaView's built in ``Save Catalyst State`` button.
+
+The process for creating a pipeline is as follows:
+    1. Run at least one step of simulation and save the data in a ParaView compatible format, then open it in ParaView.
+    2. Set up the desired scene, including filters, camera and views, and extractors.
+    3. Press ``Save Catalyst State``, or the multicolored flask icon in the top left corner, and save it to a desired location.
+    4. Open the script and replace the used producer with ``PVTrivialProducer``, setting the ``registrationName`` to either ``mesh`` or ``particles`` based on what data is used.
+
+As an example for step four, here are a few lines from a script directly exported from ParaView:
+
+.. code-block:: python
+
+    # create a new 'XML Image Data Reader'
+    meshdatavti = XMLImageDataReader(registrationName='meshdata.vti', FileName=['/path/to/meshdata.vti'])
+    meshdatavti.CellArrayStatus = ['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez']
+    meshdatavti.TimeArray = 'None'
+
+    # Calculator sample filter
+    calculator1 = Calculator(registrationName='Calculator1', Input=meshdatavti)
+    calculator1.AttributeType = 'Cell Data'
+    calculator1.ResultArrayName = 'BField'
+    calculator1.Function = 'sqrt(Bx^2 + By^2 + Bz^2)'
+
+In order to use it with the mesh data coming from the simulation, the above code would be changed to:
+
+.. code-block:: python
+
+    # create the producer
+    meshdata = PVTrivialProducer(registrationName='mesh')
+    meshdata.CellArrayStatus = ['Bx', 'By', 'Bz', 'Ex', 'Ey', 'Ez']
+    meshdata.TimeArray = 'None'
+
+    # Calculator sample filter
+    calculator1 = Calculator(registrationName='Calculator1', Input=meshdata)
+    calculator1.AttributeType = 'Cell Data'
+    calculator1.ResultArrayName = 'BField'
+    calculator1.Function = 'sqrt(Bx^2 + By^2 + Bz^2)'
+
+Steps one is advised so that proper scaling and framing can be done, however in certain cases it may not be possible. If this is the case, a dummy object can be used instead
+(such as a wavelet or geometric shape scaled appropriately) and the rest of the steps can be followed as usual.
+
+Replay
+------
+
+Catalyst 2 supports replay capabilities, which can be read about `here <https://catalyst-in-situ.readthedocs.io/en/latest/catalyst_replay.html>`_.
+
+.. note::
+
+   * TODO: Add more extensive documentation on replay

Docs/source/dataanalysis/formats.rst

@@ -41,8 +41,9 @@ files to disk).
 .. toctree::
    :maxdepth: 1
 
-   sensei
    ascent
+   catalyst
+   sensei
 
 If you like the 3D rendering of laser wakefield acceleration
 on the WarpX documentation front page (which is

Docs/source/dataanalysis/paraview.rst

@@ -54,3 +54,9 @@ Plotfiles (AMReX)
 
 ParaView also supports visualizing AMReX plotfiles.
 Please see `the AMReX documentation <https://amrex-codes.github.io/amrex/docs_html/Visualization.html#paraview>`__ for more details.
+
+
+In Situ Analysis with Catalyst 2
+--------------------------------
+
+Continue reading :ref:`here <visualization-catalyst>`.

Docs/source/install/cmake.rst

@@ -85,6 +85,7 @@ CMake Option                  Default & Values                             Descr
 ``CMAKE_VERBOSE_MAKEFILE``    ON/**OFF**                                   `Print all compiler commands to the terminal during build <https://cmake.org/cmake/help/latest/variable/CMAKE_VERBOSE_MAKEFILE.html>`__
 ``WarpX_APP``                 **ON**/OFF                                   Build the WarpX executable application
 ``WarpX_ASCENT``              ON/**OFF**                                   Ascent in situ visualization
+``WarpX_CATALYST``            ON/**OFF**                                   Catalyst in situ visualization
 ``WarpX_COMPUTE``             NOACC/**OMP**/CUDA/SYCL/HIP                  On-node, accelerated computing backend
 ``WarpX_DIMS``                **3**/2/1/RZ                                 Simulation dimensionality. Use ``"1;2;RZ;3"`` for all.
 ``WarpX_EB``                  ON/**OFF**                                   Embedded boundary support (not supported in RZ yet)