Add command-line args to the Step class and a ModelStep base class

[xylar]

For task parallelism, it is necessary for steps to use a consistent number of resources. A compass step can no longer have a python piece followed by an MPI-parallel piece.

Currently, combination python and MPI steps are common in compass. Many such steps involve setting up a model run in some way in python before running the model with MPI. More complex steps may run the model multiple times, performing python operations between each model run. Other such steps include those that generate mapping files (using ESMF_RegridWeightGen run with MPI).

To support task parallelism, it will be necessary to break these steps into smaller steps, each either running python code or MPI code. This is necessary because each step will be called with srun and the specified resources (e.g. ntasks, cpus_per_task, or sometimes both). The python code must be run with ntasks = 1, while the MPI code typically is run with ntasks > 1. @altheaden and I took a good deal of time to investigate breaking steps in to "substeps" to support different numbers of resources. Ultimately, the complexity of both the concepts and the code itself was worse than if we just break current steps into multiple steps with unique resources.

As a significant step toward this goal, this PR:

1. Adds an args attribute to the Step class. If args is defined, the step's run() method isn't called (and shouldn't be overridden). Instead, the contents of args should be a list of command-line arguments to be called with srun (or mpirun on a laptop) and an appropriate set of resources. Running the model is an obvious example of this, as we discuss next.
2. Adds a ModelStep class for reducing the amount of redundancy involved in making steps that run the MPAS model. This class descends from Step and takes care of work like creating graph and partition files, updating PIO namelist options, etc. that are needed before running the model. It also automatically creates the args attribute used to run the model itself.
3. Migrates many (but by no means all) of the ocean test groups to using ModelStep

Migrated test groups:

• baroclinic_channel
• drying_slope
• global_convergence
• gotm
• hurricane
• internal_waves
• planar_convergence
• soma
• sphere_transport
• spherical_harmonic_transform
• ziso

Not migrated (typically because of the complexity of SSH adjustment for ice-shelf cavities):

• global_ocean
• ice_shelf_2d
• isomip_plus

[pep8speaks]

Hello @xylar! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-08-30 13:54:23 UTC

[xylar]

Testing

The pr suite is bit-for-bit with master using this branch on Anvil with Intel and Intel-MPI.

Further testing is needed to make sure the following test groups that are not in pr are still working:
Migrated test groups:

• drying_slope
• gotm
• hurricane
• internal_waves
• planar_convergence
• sphere_transport
• spherical_harmonic_transform

[xylar]

I'm going to close this PR and move this work to Polaris once we have a repo for it.