Merge pull request #991 from parthenon-hpc-lab/lroberts36/add-fine-va…

…riables

Add fine field option

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Current develop
 
 ### Added (new features/APIs/variables/...)
+- [[PR 991]](https://github.com/parthenon-hpc-lab/parthenon/pull/991) Add fine fields.
 - [[PR 1106]](https://github.com/parthenon-hpc-lab/parthenon/pull/1106) Add CMake options for turning on ASAN and HWASAN
 - [[PR 1100]](https://github.com/parthenon-hpc-lab/parthenon/pull/1100) Custom refinement ops propagated to fluxes
 - [[PR 1090]](https://github.com/parthenon-hpc-lab/parthenon/pull/1090) SMR with swarms

doc/sphinx/src/interface/metadata.rst

@@ -55,6 +55,13 @@ mutually exclusive:
    cell-corners. The variable might be volume-averaged, or defined
    pointwise.
 
+For any type of topology, the flag ``Metadata::Fine`` can be set which 
+causes the field to have twice the resolution of normal fields. These 
+fields should be able to be specified for output, undergo ghost exchange,
+etc. but much of this has not been carefully tested. **As a result, ``Fine`` 
+fields should be handled with care in downstream codes and carefully checked 
+to make sure they are behaving as expected.**
+
 Variable Behaviors
 ------------------
 

example/CMakeLists.txt

@@ -13,6 +13,7 @@
 
 add_subdirectory(stochastic_subgrid)
 add_subdirectory(advection)
+add_subdirectory(fine_advection)
 add_subdirectory(calculate_pi)
 add_subdirectory(kokkos_pi)
 add_subdirectory(particles)

example/fine_advection/CMakeLists.txt

@@ -0,0 +1,28 @@
+#=========================================================================================
+# (C) (or copyright) 2024. Triad National Security, LLC. All rights reserved.
+#
+# This program was produced under U.S. Government contract 89233218CNA000001 for Los
+# Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
+# for the U.S. Department of Energy/National Nuclear Security Administration. All rights
+# in the program are reserved by Triad National Security, LLC, and the U.S. Department
+# of Energy/National Nuclear Security Administration. The Government is granted for
+# itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide
+# license in this material to reproduce, prepare derivative works, distribute copies to
+# the public, perform publicly and display publicly, and to permit others to do so.
+#=========================================================================================
+
+get_property(DRIVER_LIST GLOBAL PROPERTY DRIVERS_USED_IN_TESTS)
+if( "fine_advection-example" IN_LIST DRIVER_LIST OR NOT PARTHENON_DISABLE_EXAMPLES)
+  add_executable(
+      fine_advection-example
+          advection_driver.cpp
+          advection_driver.hpp
+          advection_package.cpp
+          advection_package.hpp
+          main.cpp
+          parthenon_app_inputs.cpp
+          stokes.hpp
+  )
+  target_link_libraries(fine_advection-example PRIVATE Parthenon::parthenon)
+  lint_target(fine_advection-example)
+endif()

example/fine_advection/README.md

@@ -0,0 +1,5 @@
+This example implements upwind advection of a cell-centered scalar variable defined 
+on the regular grid and for another cell-centered variable on the fine grid (which
+is twice the resolution and is selected using Metadata::Fine). The newer type-based
+`SparsePack`s are used throughout and machinery for doing a generalized Stoke's 
+theorem based update is included.

example/fine_advection/advection_driver.cpp

@@ -0,0 +1,169 @@
+//========================================================================================
+// (C) (or copyright) 2020-2024. Triad National Security, LLC. All rights reserved.
+//
+// This program was produced under U.S. Government contract 89233218CNA000001 for Los
+// Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
+// for the U.S. Department of Energy/National Nuclear Security Administration. All rights
+// in the program are reserved by Triad National Security, LLC, and the U.S. Department
+// of Energy/National Nuclear Security Administration. The Government is granted for
+// itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide
+// license in this material to reproduce, prepare derivative works, distribute copies to
+// the public, perform publicly and display publicly, and to permit others to do so.
+//========================================================================================
+
+#include <memory>
+#include <string>
+#include <vector>
+
+// Local Includes
+#include "advection_driver.hpp"
+#include "advection_package.hpp"
+#include "amr_criteria/refinement_package.hpp"
+#include "bvals/comms/bvals_in_one.hpp"
+#include "interface/metadata.hpp"
+#include "interface/update.hpp"
+#include "mesh/meshblock_pack.hpp"
+#include "parthenon/driver.hpp"
+#include "prolong_restrict/prolong_restrict.hpp"
+#include "stokes.hpp"
+
+using namespace parthenon::driver::prelude;
+
+namespace advection_example {
+
+// *************************************************//
+// define the application driver. in this case,    *//
+// that mostly means defining the MakeTaskList     *//
+// function.                                       *//
+// *************************************************//
+AdvectionDriver::AdvectionDriver(ParameterInput *pin, ApplicationInput *app_in, Mesh *pm)
+    : MultiStageDriver(pin, app_in, pm) {
+  // fail if these are not specified in the input file
+  pin->CheckRequired("parthenon/mesh", "ix1_bc");
+  pin->CheckRequired("parthenon/mesh", "ox1_bc");
+  pin->CheckRequired("parthenon/mesh", "ix2_bc");
+  pin->CheckRequired("parthenon/mesh", "ox2_bc");
+
+  // warn if these fields aren't specified in the input file
+  pin->CheckDesired("parthenon/mesh", "refinement");
+  pin->CheckDesired("parthenon/mesh", "numlevel");
+  pin->CheckDesired("Advection", "cfl");
+  pin->CheckDesired("Advection", "vx");
+  pin->CheckDesired("Advection", "refine_tol");
+  pin->CheckDesired("Advection", "derefine_tol");
+}
+
+// See the advection.hpp declaration for a description of how this function gets called.
+TaskCollection AdvectionDriver::MakeTaskCollection(BlockList_t &blocks, const int stage) {
+  using namespace parthenon::Update;
+  TaskCollection tc;
+  TaskID none(0);
+
+  // Build MeshBlockData containers that will be included in MeshData containers. It is
+  // gross that this has to be done by hand.
+  const auto &stage_name = integrator->stage_name;
+  if (stage == 1) {
+    for (int i = 0; i < blocks.size(); i++) {
+      auto &pmb = blocks[i];
+      // first make other useful containers
+      auto &base = pmb->meshblock_data.Get();
+      pmb->meshblock_data.Add("dUdt", base);
+      for (int s = 1; s < integrator->nstages; s++)
+        pmb->meshblock_data.Add(stage_name[s], base);
+    }
+  }
+
+  const Real beta = integrator->beta[stage - 1];
+  const Real dt = integrator->dt;
+
+  const int num_partitions = pmesh->DefaultNumPartitions();
+  TaskRegion &single_tasklist_per_pack_region = tc.AddRegion(num_partitions);
+
+  for (int i = 0; i < num_partitions; i++) {
+    auto &tl = single_tasklist_per_pack_region[i];
+    auto &mbase = pmesh->mesh_data.GetOrAdd("base", i);
+    auto &mc0 = pmesh->mesh_data.GetOrAdd(stage_name[stage - 1], i);
+    auto &mc1 = pmesh->mesh_data.GetOrAdd(stage_name[stage], i);
+    auto &mdudt = pmesh->mesh_data.GetOrAdd("dUdt", i);
+
+    auto start_send = tl.AddTask(none, parthenon::StartReceiveBoundaryBuffers, mc1);
+    auto start_flxcor = tl.AddTask(none, parthenon::StartReceiveFluxCorrections, mc0);
+
+    // Make a sparse variable pack descriptors that can be used to build packs
+    // including some subset of the fields in this example. This will be passed
+    // to the Stokes update routines, so that they can internally create variable
+    // packs that operate on only the desired set of variables.
+    static auto desc =
+        parthenon::MakePackDescriptor<advection_package::Conserved::scalar>(
+            pmesh->resolved_packages.get(), {parthenon::Metadata::WithFluxes},
+            {parthenon::PDOpt::WithFluxes});
+    using pack_desc_t = decltype(desc);
+
+    static auto desc_fine =
+        parthenon::MakePackDescriptor<advection_package::Conserved::scalar_fine>(
+            pmesh->resolved_packages.get(), {parthenon::Metadata::WithFluxes},
+            {parthenon::PDOpt::WithFluxes});
+    using pack_desc_fine_t = decltype(desc_fine);
+
+    using TE = parthenon::TopologicalElement;
+    std::vector<TE> faces{TE::F1};
+    if (pmesh->ndim > 1) faces.push_back(TE::F2);
+    if (pmesh->ndim > 2) faces.push_back(TE::F3);
+    auto flx = none;
+    auto flx_fine = none;
+    for (auto face : faces) {
+      flx = flx | tl.AddTask(none, advection_package::CalculateFluxes<pack_desc_t>, desc,
+                             face, parthenon::CellLevel::same, mc0.get());
+      flx_fine = flx_fine |
+                 tl.AddTask(none, advection_package::CalculateFluxes<pack_desc_fine_t>,
+                            desc_fine, face, parthenon::CellLevel::fine, mc0.get());
+    }
+
+    auto set_flx = parthenon::AddFluxCorrectionTasks(start_flxcor | flx | flx_fine, tl,
+                                                     mc0, pmesh->multilevel);
+
+    auto flux_div = tl.AddTask(set_flx, Stokes<pack_desc_t>, parthenon::CellLevel::same,
+                               parthenon::TopologicalType::Cell, desc, pmesh->ndim,
+                               mc0.get(), mdudt.get());
+
+    auto flux_div_fine = tl.AddTask(
+        set_flx, Stokes<pack_desc_fine_t>, parthenon::CellLevel::fine,
+        parthenon::TopologicalType::Cell, desc_fine, pmesh->ndim, mc0.get(), mdudt.get());
+
+    auto avg_data =
+        tl.AddTask(flux_div, WeightedSumData<pack_desc_t>, parthenon::CellLevel::same,
+                   parthenon::TopologicalElement::CC, desc, mc0.get(), mbase.get(), beta,
+                   1.0 - beta, mc0.get());
+    auto avg_data_fine =
+        tl.AddTask(flux_div_fine, WeightedSumData<pack_desc_fine_t>,
+                   parthenon::CellLevel::fine, parthenon::TopologicalElement::CC,
+                   desc_fine, mc0.get(), mbase.get(), beta, 1.0 - beta, mc0.get());
+
+    auto update =
+        tl.AddTask(avg_data, WeightedSumData<pack_desc_t>, parthenon::CellLevel::same,
+                   parthenon::TopologicalElement::CC, desc, mc0.get(), mdudt.get(), 1.0,
+                   beta * dt, mc1.get());
+    auto update_fine =
+        tl.AddTask(avg_data_fine, WeightedSumData<pack_desc_fine_t>,
+                   parthenon::CellLevel::fine, parthenon::TopologicalElement::CC,
+                   desc_fine, mc0.get(), mdudt.get(), 1.0, beta * dt, mc1.get());
+
+    auto boundaries = parthenon::AddBoundaryExchangeTasks(
+        update | update_fine | start_send, tl, mc1, pmesh->multilevel);
+
+    auto fill_derived =
+        tl.AddTask(boundaries, parthenon::Update::FillDerived<MeshData<Real>>, mc1.get());
+
+    if (stage == integrator->nstages) {
+      auto new_dt = tl.AddTask(fill_derived, EstimateTimestep<MeshData<Real>>, mc1.get());
+      if (pmesh->adaptive) {
+        auto tag_refine =
+            tl.AddTask(new_dt, parthenon::Refinement::Tag<MeshData<Real>>, mc1.get());
+      }
+    }
+  }
+
+  return tc;
+}
+
+} // namespace advection_example

example/fine_advection/advection_driver.hpp

@@ -0,0 +1,45 @@
+//========================================================================================
+// (C) (or copyright) 2020-2021. Triad National Security, LLC. All rights reserved.
+//
+// This program was produced under U.S. Government contract 89233218CNA000001 for Los
+// Alamos National Laboratory (LANL), which is operated by Triad National Security, LLC
+// for the U.S. Department of Energy/National Nuclear Security Administration. All rights
+// in the program are reserved by Triad National Security, LLC, and the U.S. Department
+// of Energy/National Nuclear Security Administration. The Government is granted for
+// itself and others acting on its behalf a nonexclusive, paid-up, irrevocable worldwide
+// license in this material to reproduce, prepare derivative works, distribute copies to
+// the public, perform publicly and display publicly, and to permit others to do so.
+//========================================================================================
+
+#ifndef EXAMPLE_FINE_ADVECTION_ADVECTION_DRIVER_HPP_
+#define EXAMPLE_FINE_ADVECTION_ADVECTION_DRIVER_HPP_
+
+#include <memory>
+#include <vector>
+
+#include <parthenon/driver.hpp>
+#include <parthenon/package.hpp>
+
+namespace advection_example {
+using namespace parthenon::driver::prelude;
+
+class AdvectionDriver : public MultiStageDriver {
+ public:
+  AdvectionDriver(ParameterInput *pin, ApplicationInput *app_in, Mesh *pm);
+  // This next function essentially defines the driver.
+  // Call graph looks like
+  // main()
+  //   EvolutionDriver::Execute (driver.cpp)
+  //     MultiStageBlockTaskDriver::Step (multistage.cpp)
+  //       DriverUtils::ConstructAndExecuteTaskLists (driver.hpp)
+  //         AdvectionDriver::MakeTaskCollection (advection_driver.cpp)
+  TaskCollection MakeTaskCollection(BlockList_t &blocks, int stage);
+};
+
+void ProblemGenerator(MeshBlock *pmb, parthenon::ParameterInput *pin);
+
+parthenon::Packages_t ProcessPackages(std::unique_ptr<parthenon::ParameterInput> &pin);
+
+} // namespace advection_example
+
+#endif // EXAMPLE_FINE_ADVECTION_ADVECTION_DRIVER_HPP_