Skip to content

Commit

Permalink
Summit Power9 CPUs documentation (#1162)
Browse files Browse the repository at this point in the history
* Power9 CPU docs

* Update Tools/BatchScripts/batch_summit_power9.sh

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>

* Subsection convention

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
  • Loading branch information
mrowan137 and ax3l authored Jul 11, 2020
1 parent e53b3a3 commit 8af11ca
Show file tree
Hide file tree
Showing 2 changed files with 58 additions and 1 deletion.
37 changes: 36 additions & 1 deletion Docs/source/running_cpp/platforms.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,14 @@ regime), the following set of parameters provided good performance:
Running on Summit at OLCF
-------------------------

.. _running-cpp-summit-V100-GPUs:

V100 GPUs
^^^^^^^^^

The batch script below can be used to run a WarpX simulation on 2 nodes on
the supercomputer Summit at OLCF. Replace descriptions between chevrons ``<>``
by relevalt values, for instance ``<input file>`` could be
by relevant values, for instance ``<input file>`` could be
``plasma_mirror_inputs``. Note that the only option so far is to run with one
MPI rank per GPU.

Expand Down Expand Up @@ -74,3 +79,33 @@ regime), the following set of parameters provided good performance:

A batch script with more options regarding profiling on Summit can be found at
:download:`Summit batch script<../../../Tools/BatchScripts/script_profiling_summit.sh>`

.. _running-cpp-summit-Power9-CPUs:

Power9 CPUs
^^^^^^^^^^^

Similar to above, the batch script below can be used to run a WarpX simulation on
1 node on the supercomputer Summit at OLCF, on Power9 CPUs (i.e., the GPUs are
ignored).

.. literalinclude:: ../../../Tools/BatchScripts/batch_summit_power9.sh
:language: bash

For a 3D simulation with a few (1-4) particles per cell using FDTD Maxwell
solver on Summit for a well load-balanced problem, the following set of
parameters provided good performance:

* ``amr.max_grid_size=64`` and ``amr.blocking_factor=64``

* **Two MPI ranks per node** (i.e. 2 resource sets per node; equivalently, 1
resource set per socket)

* **21 physical CPU cores per MPI rank**

* **21 OpenMP threads per MPI rank** (i.e. 1 OpenMP thread per physical core)

* **SMT 1 (Simultaneous Multithreading level 1)**

* **Sixteen `64x64x64` grids per MPI rank** (with default tiling in WarpX, this
results in ~49 tiles per OpenMP thread)
22 changes: 22 additions & 0 deletions Tools/BatchScripts/batch_summit_power9.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/bin/bash

# Copyright 2019-2020 Maxence Thevenet, Axel Huebl, Michael Rowan
#
# This file is part of WarpX.
#
# License: BSD-3-Clause-LBNL
#
# Refs.:
# https://jsrunvisualizer.olcf.ornl.gov/?s1f0o121n2c21g0r11d1b1l0=

#BSUB -P <allocation ID>
#BSUB -W 00:10
#BSUB -nnodes 1
#BSUB -alloc_flags "smt1"
#BSUB -J WarpX
#BSUB -o WarpXo.%J
#BSUB -e WarpXe.%J


export OMP_NUM_THREADS=21
jsrun -n 2 -a 1 -c 21 -r 2 -l CPU-CPU -d packed -b rs <path/to/executable> <input file> > output.txt

0 comments on commit 8af11ca

Please sign in to comment.