Merge branch 'origin/whannah/atm/fix-physgrid-aqua-restart' into next…

… (PR #3204)

This fixes an issues where aqua planet cases were crashing on restart due to
not correctly setting the topography to zero. Instead the realistic topography
was added on restart, which caused the run to crash. This PR resolves this issue
by explicitly resetting the topography to zero when restarting a physgrid run.
This also fixes an issue running certain physgrid configurations, such as ne4pg2,
because the default NTASK was set higher than the number of physics columns.
To get the physgrid specification to match correctly in config_pesall.xml an underscore
was to be added to other grid query strings. This should not affect non-physgrid cases.
The underscore is used because it always separates the atmosphere grid from the other grids.
For example, A typical ne4 grid string is
a%ne4np4_l%ne4np4_oi%ne4np4_r%null_g%null_w%null_z%null_m%oQU240
But the string for ne4pg2 is
a%ne4np4.pg2_l%ne4np4_oi%ne4np4_r%null_g%null_w%null_z%null_m%oQU240
so in order to differentiate the grids we use "ne4np4_" and "ne4np4.pg2" to get
different default PE layouts. Using "ne4np4" without the underscore leads to an error
when running physgrid cases.
After some discussion we decided to also include a change to the default physgrid mapping algorithm
to use the high-order option. This will make all pg2 tests non-BFB.

[BFB] except for SMS_Ln5.ne4pg2_ne4pg2.FC5AV1C-L tests

cime/config/e3sm/allactive/config_pesall.xml

@@ -662,6 +662,43 @@
       </pes>
     </mach>
   </grid>
+  <grid name="a%ne4np4.pg2">
+    <mach name="melvin">
+      <pes compset="any" pesize="any">
+        <comment>ne4np4pg2 compset on melvin</comment>
+        <ntasks>
+          <ntasks_atm>48</ntasks_atm>
+          <ntasks_lnd>48</ntasks_lnd>
+          <ntasks_rof>48</ntasks_rof>
+          <ntasks_ice>48</ntasks_ice>
+          <ntasks_ocn>48</ntasks_ocn>
+          <ntasks_glc>48</ntasks_glc>
+          <ntasks_wav>48</ntasks_wav>
+          <ntasks_cpl>48</ntasks_cpl>
+        </ntasks>
+        <nthrds>
+          <nthrds_atm>1</nthrds_atm>
+          <nthrds_lnd>1</nthrds_lnd>
+          <nthrds_rof>1</nthrds_rof>
+          <nthrds_ice>1</nthrds_ice>
+          <nthrds_ocn>1</nthrds_ocn>
+          <nthrds_glc>1</nthrds_glc>
+          <nthrds_wav>1</nthrds_wav>
+          <nthrds_cpl>1</nthrds_cpl>
+        </nthrds>
+        <rootpe>
+          <rootpe_atm>0</rootpe_atm>
+          <rootpe_lnd>0</rootpe_lnd>
+          <rootpe_rof>0</rootpe_rof>
+          <rootpe_ice>0</rootpe_ice>
+          <rootpe_ocn>0</rootpe_ocn>
+          <rootpe_glc>0</rootpe_glc>
+          <rootpe_wav>0</rootpe_wav>
+          <rootpe_cpl>0</rootpe_cpl>
+        </rootpe>
+      </pes>
+    </mach>
+  </grid>
   <grid name="a%ne30np4_">
     <mach name="edison">
       <pes compset="any" pesize="any">