Forest Of Octrees

CHANGELOG.md

@@ -26,6 +26,7 @@
 - [[PR978]](https://github.com/parthenon-hpc-lab/parthenon/pull/978) remove erroneous sparse check
 
 ### Infrastructure (changes irrelevant to downstream codes)
+- [[PR 1009]](https://github.com/parthenon-hpc-lab/parthenon/pull/1009) Move from a single octree to a forest of octrees
 - [[PR 1017]](https://github.com/parthenon-hpc-lab/parthenon/pull/1017) Make regression tests more verbose on failure
 - [[PR 1007]](https://github.com/parthenon-hpc-lab/parthenon/pull/1007) Split template instantiations for HDF5 Read/Write attributes to speed up compile times
 - [[PR 990]](https://github.com/parthenon-hpc-lab/parthenon/pull/990) Partial refactor of HDF5 I/O code for readability/extendability

doc/sphinx/src/boundary_communication.rst

@@ -155,17 +155,7 @@ In practice, we denote each channel by a unique key
 so that the ``Mesh`` can contain a map from these keys to communication
 channels. Then, at each remesh, sending blocks and blocks that are
 receiving from blocks on a different rank can create new communication
-channels and register them in this map. *Implementation detail:* To
-build these keys, we currently rely on the
-``MeshBlock::std::unique_ptr<BoundaryValues> pbval`` object to get
-information about the neighboring blocks and build the channels and
-keys. ``BoundaryValues`` has its own communication methods defined, but
-none of these are used for the sparse communication. We really only rely
-on the information stored in ``BoundaryBase`` (which contains general
-information about all of the neighboring blocks on the mesh), which
-``BoundaryValues`` inherits from. Eventually, I think ``pbval`` should
-be turned into a ``BoundaryBase`` object and ``BoundaryValues`` should
-be removed from the code base.
+channels and register them in this map. 
 
 MPI Communication IDs
 ~~~~~~~~~~~~~~~~~~~~~

doc/sphinx/src/interface/boundary.rst

@@ -12,53 +12,3 @@ BoundaryCommunication
 
 Pure abstract base class, defines interfaces for managing
 ``BoundaryStatus`` flags and MPI requests
-
-BoundaryBuffer
---------------
-
-Pure abstract base class, defines interfaces for managing MPI
-send/receive and loading/storing data from communication buffers.
-
-BoundaryVariable
-----------------
-
-**Derived from**: ``BoundaryCommunication`` and ``BoundaryBuffer``
-
-**Contains**: ``BoundaryData`` for variable and flux correction.
-
-**Knows about**: ``MeshBlock`` and ``Mesh``
-
-Still abstract base class, but implements some methods for sending and
-receiving buffers.
-
-BoundaryBase
-------------
-
-**Contains**: ``NeighborIndexes`` and ``NeighborBlock`` PER NEIGHBOR,
-number of neighbors, neighbor levels
-
-**Knows about**: ``MeshBlock``
-
-Implements ``SearchAndSetNeighbors``
-
-BoundaryValues
---------------
-
-**Derived from**: ``BoundaryBase`` and ``BoundaryCommunication``
-
-**Knows about**: ``MeshBlock``, all the ``BoundaryVariable`` connected
-to variables of this block
-
-Central class to interact with individual variable boundary data. Owned
-by ``MeshBlock``.
-
-CellCenteredBoundaryVariable
-----------------------------
-
-**Derived from**: ``BoundaryVariable``
-
-**Contains**: Shallow copies of variable data, coarse buffer, and fluxes
-(owned by ``Variable``)
-
-Owned by ``Variable``, implements loading and setting boundary data,
-sending and receiving flux corrections, and more.

example/particles/particles.cpp

@@ -529,8 +529,8 @@ TaskStatus StopCommunicationMesh(const BlockList_t &blocks) {
 #ifdef MPI_PARALLEL
   for (auto &block : blocks) {
     auto swarm = block->swarm_data.Get()->Get("my_particles");
-    for (int n = 0; n < block->pbval->nneighbor; n++) {
-      NeighborBlock &nb = block->pbval->neighbor[n];
+    for (int n = 0; n < block->neighbors.size(); n++) {
+      NeighborBlock &nb = block->neighbors[n];
       // TODO(BRR) May want logic like this if we have non-blocking TaskRegions
       // if (nb.snb.rank != Globals::my_rank) {
       //  if (swarm->vbswarm->bd_var_.flag[nb.bufid] != BoundaryStatus::completed) {
@@ -563,8 +563,8 @@ TaskStatus StopCommunicationMesh(const BlockList_t &blocks) {
     auto &pmb = block;
     auto sc = pmb->swarm_data.Get();
     auto swarm = sc->Get("my_particles");
-    for (int n = 0; n < swarm->vbswarm->bd_var_.nbmax; n++) {
-      auto &nb = pmb->pbval->neighbor[n];
+    for (int n = 0; n < pmb->neighbors.size(); n++) {
+      auto &nb = block->neighbors[n];
       swarm->vbswarm->bd_var_.flag[nb.bufid] = BoundaryStatus::waiting;
     }
   }

example/sparse_advection/sparse_advection_driver.cpp

@@ -122,12 +122,12 @@ TaskCollection SparseAdvectionDriver::MakeTaskCollection(BlockList_t &blocks,
                              mdudt.get(), beta * dt, mc1.get());
 
     // do boundary exchange
-    auto restrict =
+    auto boundary =
         parthenon::AddBoundaryExchangeTasks(update, tl, mc1, pmesh->multilevel);
 
     // if this is the last stage, check if we can deallocate any sparse variables
     if (stage == integrator->nstages) {
-      tl.AddTask(restrict, SparseDealloc, mc1.get());
+      tl.AddTask(boundary, SparseDealloc, mc1.get());
     }
   }
 

scripts/python/packages/parthenon_performance_app/parthenon_performance_app/githubapp.py

@@ -76,7 +76,6 @@ def printTree(self):
 
 
 class GitHubApp:
-
     """
     GitHubApp Class
 

src/CMakeLists.txt

@@ -108,14 +108,11 @@ add_library(parthenon
   bvals/boundary_conditions_generic.hpp
   bvals/boundary_conditions.cpp
   bvals/boundary_conditions.hpp
-
   bvals/bvals.cpp
   bvals/bvals.hpp
-  bvals/bvals_base.cpp
-  bvals/bvals_interfaces.hpp
+  bvals/neighbor_block.cpp
+  bvals/neighbor_block.hpp
   bvals/boundary_flag.cpp
-  bvals/bvals_var.cpp
-  bvals/bvals_swarm.cpp
 
   coordinates/coordinates.hpp
   coordinates/uniform_cartesian.hpp
@@ -161,6 +158,13 @@ add_library(parthenon
 
   mesh/amr_loadbalance.cpp
   mesh/domain.hpp
+  mesh/forest/cell_center_offsets.hpp
+  mesh/forest/forest.cpp
+  mesh/forest/forest.hpp
+  mesh/forest/relative_orientation.hpp
+  mesh/forest/relative_orientation.cpp
+  mesh/forest/tree.hpp
+  mesh/forest/tree.cpp
   mesh/logical_location.cpp
   mesh/logical_location.hpp
   mesh/mesh_refinement.cpp
@@ -170,8 +174,6 @@ add_library(parthenon
   mesh/mesh.hpp
   mesh/meshblock.hpp
   mesh/meshblock_pack.hpp
-  mesh/meshblock_tree.cpp
-  mesh/meshblock_tree.hpp
   mesh/meshblock.cpp
 
   outputs/ascent.cpp

src/bvals/boundary_conditions.cpp

@@ -16,7 +16,7 @@
 
 #include "bvals/boundary_conditions.hpp"
 #include "bvals/boundary_conditions_generic.hpp"
-#include "bvals/bvals_interfaces.hpp"
+#include "bvals/neighbor_block.hpp"
 #include "defs.hpp"
 #include "interface/meshblock_data.hpp"
 #include "mesh/domain.hpp"

src/bvals/boundary_flag.cpp

@@ -90,9 +90,8 @@ std::string GetBoundaryString(BoundaryFlag input_flag) {
 //! \fn CheckBoundaryFlag(BoundaryFlag block_flag, CoordinateDirection dir)
 //  \brief Called in each MeshBlock's BoundaryValues() constructor. Mesh() ctor only
 //  checks the validity of user's input mesh/ixn_bc, oxn_bc string values corresponding to
-//  a BoundaryFlag enumerator before passing it to a MeshBlock and then BoundaryBase
-//  object. However, not all BoundaryFlag enumerators can be used in all directions as a
-//  valid MeshBlock boundary.
+//  a BoundaryFlag enumerator before passing it to a MeshBlock. However, not all
+//  BoundaryFlag enumerators can be used in all directions as a valid MeshBlock boundary.
 
 void CheckBoundaryFlag(BoundaryFlag block_flag, CoordinateDirection dir) {
   std::stringstream msg;