Merge pull request #1413 from billsacks/billsacks/glc_tavg_v2

Pass annual-average fields to GLC

These changes are from Mariana Vertenstein (primarily) and myself

Until now, the CESM coupler has been sending CISM daily-average fields,
and then CISM has been averaging these fields to annual-averages in
order to do its mass balance time step using annual-average SMB (and
temperature). This has a number of problems:

(1) We could not restart mid-year, because CISM doesn't write the
    necessary partial accumulation information to its restart files.

(2) With Bill Lipscomb's new remapping (bilinear + conservation
    correction; coming soon), there were too-large corrections needed
    when doing the remapping from CLM to CISM on daily fields.

(3) This led to a discrepancy between TG and BG/IG compsets: In BG/IG,
    CISM received the annual average of remapped daily fields; in TG,
    CISM received the remapped annual average fields. The remapping is
    not a linear operation, so there would typically be differences in
    these two runs.

This tag, together with cism2_1_30, allows the coupler to make annual
averages before remapping fields to CISM. This was a bit tricky: CISM
still needs to be called once per day in order to support mid-year
restarts (CESM can only restart at a frequency when all components are
called). So now the GLC calling frequency has been separated from the
GLC averaging frequency. Then GLC/CISM is passed a flag saying whether
the inputs are currently valid; CISM only accumulates averages of the
inputs when they are valid. (In the standard case, this means that CISM
accumulates a single value each year, leaving all of the averaging up to
the coupler. But this supports the ability to still do the averaging in
CISM if this is desired - however, note that mid-year restarts won't
work in that case [CISM now aborts if you try to do a mid-year restart
in that configuration].)

This is implemented in such a way that ACME runs with MPAS-LI should be
unaffected: the default GLC_AVG_PERIOD for ACME is
'glc_coupling_period', which makes things collapse to the old behavior.

There are some mostly-unrelated changes in
src/drivers/mct/cime_config/buildnml and
src/drivers/mct/cime_config/namelist_definition_drv.xml. These use
add_default rather than set_value for the various *_cpl_dt entries,
because in general we should not be calling set_value directly. And,
since we're using add_default, we needed to add
skip_default_entry="true" to the relevant xml variables.

In CESM, this change must be coordinated with cism2_1_30 and
CESM-Development/cime_config#6

Test suite: scripts_regression_tests on yellowstone
   Also, aux_glc test suite with cism2_1_30:
   - out-of-the-box: tests pass, and bit-for-bit except for IG tests
     without test_coupling
   - with GLC_AVG_PERIOD = 'glc_coupling_period': tests pass and
     bit-for-bit, except for expected failures in tests that restart
     mid-year
Test baseline: For aux_glc tests, baseline was cism2_1_29
Test namelist changes: YES - adds new glc_avg_period
Test status: greater than roundoff-level changes for multi-year runs
   with CISM, other than TG

Fixes none

User interface changes?: new xml variable, GLC_AVG_PERIOD

Code review: Tony Craig, Jim Edwards, Rob Jacob, Matt Hoffman

src/components/data_comps/dice/dice_comp_mod.F90

@@ -621,7 +621,6 @@ subroutine dice_comp_run( EClock, cdata,  x2i, i2x)
    integer(IN)   :: nl                ! ice frac index
    integer(IN)   :: lsize           ! size of attr vect
    integer(IN)   :: shrlogunit, shrloglev ! original log unit and level
-   logical       :: glcrun_alarm      ! is glc going to run now
    logical       :: newdata           ! has newdata been read
    logical       :: mssrmlf           ! remove old data
    integer(IN)   :: idt               ! integer timestep

src/components/data_comps/docn/docn_comp_mod.F90

@@ -539,7 +539,6 @@ subroutine docn_comp_run( EClock, cdata,  x2o, o2x)
    integer(IN)   :: nl                ! ocn frac index
    integer(IN)   :: lsize           ! size of attr vect
    integer(IN)   :: shrlogunit, shrloglev ! original log unit and level
-   logical       :: glcrun_alarm      ! is glc going to run now
    logical       :: newdata           ! has newdata been read
    logical       :: mssrmlf           ! remove old data
    integer(IN)   :: idt               ! integer timestep

src/drivers/mct/cime_config/buildnml

@@ -101,7 +101,7 @@ def _create_drv_namelists(case, infile, confdir, nmlgen, files):
             totaldt = cpl_dt * int(ncpl)
             if totaldt != basedt:
                 expect(False, " %s ncpl doesn't divide base dt evenly" %comp)
-            nmlgen.set_value(comp.lower() + '_cpl_dt', value=cpl_dt)
+            nmlgen.add_default(comp.lower() + '_cpl_dt', value=cpl_dt)
             mindt = min(mindt, cpl_dt)
 #        elif comp.lower() is not 'cpl':
 

src/drivers/mct/cime_config/config_component.xml

@@ -303,7 +303,7 @@
     <group>run_begin_stop_restart</group>
     <file>env_run.xml</file>
     <desc>
-      Sets the run length along with STOP_N and STOP_DATE (must be nyear(s) for _GLC compsets for restarts to work properly).
+      Sets the run length along with STOP_N and STOP_DATE
     </desc>
   </entry>
 
@@ -335,8 +335,7 @@
     <group>run_begin_stop_restart</group>
     <file>env_run.xml</file>
     <desc>
-      sets frequency of model restart writes (same options as STOP_OPTION) (must be nyear(s) for _GLC compsets)
-      (must be nyear(s) for _GLC compsets)
+      sets frequency of model restart writes (same options as STOP_OPTION)
     </desc>
   </entry>
 
@@ -368,7 +367,7 @@
     <group>run_begin_stop_restart</group>
     <file>env_run.xml</file>
     <desc>
-      Sets the pause frequency along with PAUSE_N (must be nyear(s) for _GLC compsets for restarts to resume properly).
+      Sets the pause frequency along with PAUSE_N
     </desc>
   </entry>
 

src/drivers/mct/cime_config/config_component_acme.xml

@@ -365,6 +365,24 @@
     <desc>Number of glc coupling intervals per NCPL_BASE_PERIOD.</desc>
   </entry>
 
+  <entry id="GLC_AVG_PERIOD">
+    <type>char</type>
+    <valid_values>glc_coupling_period,yearly</valid_values>
+    <default_value>glc_coupling_period</default_value>
+    <group>run_coupling</group>
+    <file>env_run.xml</file>
+    <desc>Period at which coupler averages fields sent to GLC.
+    This supports doing the averaging to GLC less frequently than GLC is called
+    (i.e., separating the averaging frequency from the calling frequency).
+    This is useful because there are benefits to only averaging the GLC inputs
+    as frequently as they are really needed (yearly for CISM), but GLC needs to
+    still be called more frequently than that in order to support mid-year restarts.
+
+    Setting GLC_AVG_PERIOD to 'glc_coupling_period' means that the averaging is
+    done exactly when the GLC is called (governed by GLC_NCPL).
+    </desc>
+  </entry>
+
   <entry id="ROF_NCPL">
     <type>integer</type>
     <default_value>8</default_value>

src/drivers/mct/cime_config/config_component_cesm.xml

@@ -200,6 +200,27 @@
     <desc>Number of glc coupling intervals per NCPL_BASE_PERIOD.</desc>
   </entry>
 
+  <entry id="GLC_AVG_PERIOD">
+    <type>char</type>
+    <valid_values>glc_coupling_period,yearly</valid_values>
+    <default_value>yearly</default_value>
+    <group>run_coupling</group>
+    <file>env_run.xml</file>
+    <desc>Period at which coupler averages fields sent to GLC.
+    This supports doing the averaging to GLC less frequently than GLC is called
+    (i.e., separating the averaging frequency from the calling frequency).
+    This is useful because there are benefits to only averaging the GLC inputs
+    as frequently as they are really needed (yearly for CISM), but GLC needs to
+    still be called more frequently than that in order to support mid-year restarts.
+
+    Setting GLC_AVG_PERIOD to 'glc_coupling_period' means that the averaging is
+    done exactly when the GLC is called (governed by GLC_NCPL).
+
+    IMPORTANT: In order to restart mid-year when running with CISM, you MUST specify GLC_AVG_PERIOD = 'yearly'.
+    If using GLC_AVG_PERIOD = 'glc_coupling_period' with CISM, you can only restart on year boundaries.
+    </desc>
+  </entry>
+
   <entry id="ROF_NCPL">
     <type>integer</type>
     <default_value>8</default_value>

src/drivers/mct/cime_config/namelist_definition_drv.xml

@@ -1387,7 +1387,7 @@
   <!-- group seq_timemgr_inparm    -->
   <!-- =========================== -->
 
-  <entry id="atm_cpl_dt" modify_via_xml="ATM_NCPL">
+  <entry id="atm_cpl_dt" modify_via_xml="ATM_NCPL" skip_default_entry="true">
     <type>integer</type>
     <category>time</category>
     <group>seq_timemgr_inparm</group>
@@ -1398,12 +1398,9 @@
       NCPL_BASE_PERIOD is also set in env_run.xml and is the base period
       associated with NCPL coupling frequency, and has valid values: hour,day,year,decade
     </desc>
-    <values>
-      <value>-999</value>
-    </values>
   </entry>
 
-  <entry id="lnd_cpl_dt" modify_via_xml="LND_NCPL">
+  <entry id="lnd_cpl_dt" modify_via_xml="LND_NCPL" skip_default_entry="true">
     <type>integer</type>
     <category>time</category>
     <group>seq_timemgr_inparm</group>
@@ -1414,12 +1411,9 @@
       NCPL_BASE_PERIOD is also set in env_run.xml and is the base period
       associated with NCPL coupling frequency, nad has valid values: hour,day,year,decade
     </desc>
-    <values>
-      <value>-999</value>
-    </values>
   </entry>
 
-  <entry id="rof_cpl_dt" modify_via_xml="ROF_NCPL">
+  <entry id="rof_cpl_dt" modify_via_xml="ROF_NCPL" skip_default_entry="true">
     <type>integer</type>
     <category>time</category>
     <group>seq_timemgr_inparm</group>
@@ -1428,12 +1422,9 @@
       currently set by default to 10800 seconds.
       default: 10800
     </desc>
-    <values>
-      <value>10800</value>
-    </values>
   </entry>
 
-  <entry id="ice_cpl_dt" modify_via_xml="ICE_NCPL">
+  <entry id="ice_cpl_dt" modify_via_xml="ICE_NCPL" skip_default_entry="true">
     <type>integer</type>
     <category>time</category>
     <group>seq_timemgr_inparm</group>
@@ -1444,12 +1435,9 @@
       NCPL_BASE_PERIOD is also set in env_run.xml and is the base period
       associated with NCPL coupling frequency, nad has valid values: hour,day,year,decade
     </desc>
-    <values>
-      <value>-999</value>
-    </values>
   </entry>
 
-  <entry id="ocn_cpl_dt" modify_via_xml="OCN_NCPL">
+  <entry id="ocn_cpl_dt" modify_via_xml="OCN_NCPL" skip_default_entry="true">
     <type>integer</type>
     <category>time</category>
     <group>seq_timemgr_inparm</group>
@@ -1460,12 +1448,9 @@
       NCPL_BASE_PERIOD is also set in env_run.xml and is the base period
       associated with NCPL coupling frequency, nad has valid values: hour,day,year,decade
     </desc>
-    <values>
-      <value>-999</value>
-    </values>
   </entry>
 
-  <entry id="glc_cpl_dt" modify_via_xml="GLC_NCPL">
+  <entry id="glc_cpl_dt" modify_via_xml="GLC_NCPL" skip_default_entry="true">
     <type>integer</type>
     <category>time</category>
     <group>seq_timemgr_inparm</group>
@@ -1476,12 +1461,30 @@
       NCPL_BASE_PERIOD is also set in env_run.xml and is the base period
       associated with NCPL coupling frequency, nad has valid values: hour,day,year,decade
     </desc>
+  </entry>
+
+  <entry id="glc_avg_period" modify_via_xml="GLC_AVG_PERIOD">
+    <type>char</type>
+    <category>time</category>
+    <group>seq_timemgr_inparm</group>
+    <valid_values>glc_coupling_period,yearly</valid_values>
     <values>
-      <value>-999</value>
+      <value>$GLC_AVG_PERIOD</value>
     </values>
+    <desc>
+      Period at which coupler averages fields sent to GLC.
+      This supports doing the averaging to GLC less frequently than GLC is called
+      (i.e., separating the averaging frequency from the calling frequency).
+      This is useful because there are benefits to only averaging the GLC inputs
+      as frequently as they are really needed (yearly for CISM), but GLC needs to
+      still be called more frequently than that in order to support mid-year restarts.
+
+      Setting glc_avg_period to 'glc_coupling_period' means that the averaging is
+      done exactly when the GLC is called (governed by GLC_NCPL).
+    </desc>
   </entry>
 
-  <entry id="wav_cpl_dt" modify_via_xml="WAV_NCPL">
+  <entry id="wav_cpl_dt" modify_via_xml="WAV_NCPL" skip_default_entry="true">
     <type>integer</type>
     <category>time</category>
     <group>seq_timemgr_inparm</group>
@@ -1492,9 +1495,6 @@
       NCPL_BASE_PERIOD is also set in env_run.xml and is the base period
       associated with NCPL coupling frequency, nad has valid values: hour,day,year,decade
     </desc>
-    <values>
-      <value>-999</value>
-    </values>
   </entry>
 
   <entry id="esp_cpl_dt" skip_default_entry="true">

src/drivers/mct/main/cesm_comp_mod.F90

@@ -91,6 +91,7 @@ module cesm_comp_mod
    use seq_timemgr_mod, only: seq_timemgr_alarm_ocnrun
    use seq_timemgr_mod, only: seq_timemgr_alarm_icerun
    use seq_timemgr_mod, only: seq_timemgr_alarm_glcrun
+   use seq_timemgr_mod, only: seq_timemgr_alarm_glcrun_avg
    use seq_timemgr_mod, only: seq_timemgr_alarm_ocnnext
    use seq_timemgr_mod, only: seq_timemgr_alarm_tprof
    use seq_timemgr_mod, only: seq_timemgr_alarm_histavg
@@ -254,6 +255,7 @@ module cesm_comp_mod
    logical  :: ocnrun_alarm           ! ocn run alarm
    logical  :: ocnnext_alarm          ! ocn run alarm on next timestep
    logical  :: glcrun_alarm           ! glc run alarm
+   logical  :: glcrun_avg_alarm       ! glc run averaging alarm
    logical  :: rofrun_alarm           ! rof run alarm
    logical  :: wavrun_alarm           ! wav run alarm
    logical  :: esprun_alarm           ! esp run alarm
@@ -2131,6 +2133,11 @@ subroutine cesm_run()
       !  (this is time that models should have before they return
       !  to the driver).  Write timestamp and run alarm status
       !----------------------------------------------------------
+      ! Note that the glcrun_avg_alarm just controls what is passed to glc in terms
+      ! of averaged fields - it does NOT control when glc is called currently -
+      ! glc will be called on the glcrun_alarm setting - but it might not be passed relevant 
+      ! info if the time averaging period to accumulate information passed to glc is greater 
+      ! than the glcrun interval
 
       call seq_timemgr_clockAdvance( seq_SyncClock, force_stop, force_stop_ymd, force_stop_tod)
       call seq_timemgr_EClockGetData( EClock_d, curr_ymd=ymd, curr_tod=tod )
@@ -2141,6 +2148,7 @@ subroutine cesm_run()
       rofrun_alarm  = seq_timemgr_alarmIsOn(EClock_d,seq_timemgr_alarm_rofrun)
       icerun_alarm  = seq_timemgr_alarmIsOn(EClock_d,seq_timemgr_alarm_icerun)
       glcrun_alarm  = seq_timemgr_alarmIsOn(EClock_d,seq_timemgr_alarm_glcrun)
+      glcrun_avg_alarm = seq_timemgr_alarmIsOn(EClock_d,seq_timemgr_alarm_glcrun_avg)
       wavrun_alarm  = seq_timemgr_alarmIsOn(EClock_d,seq_timemgr_alarm_wavrun)
       esprun_alarm  = seq_timemgr_alarmIsOn(EClock_d,seq_timemgr_alarm_esprun)
       ocnrun_alarm  = seq_timemgr_alarmIsOn(EClock_d,seq_timemgr_alarm_ocnrun)
@@ -2154,6 +2162,17 @@ subroutine cesm_run()
 
       ! Does the driver need to pause?
       drv_pause = pause_alarm .and. seq_timemgr_pause_component_active(drv_index)
+
+      ! Check alarm consistency
+      if (glcrun_avg_alarm .and. .not. glcrun_alarm) then
+         write(logunit,*) 'ERROR: glcrun_avg_alarm is true, but glcrun_alarm is false'
+         write(logunit,*) 'Make sure that NCPL_BASE_PERIOD, GLC_NCPL and GLC_AVG_PERIOD'
+         write(logunit,*) 'are set so that glc averaging only happens at glc coupling times.'
+         write(logunit,*) '(It is allowable for glc coupling to be more frequent than glc averaging,'
+         write(logunit,*) 'but not for glc averaging to be more frequent than glc coupling.)'
+         call shr_sys_abort(subname//' glcrun_avg_alarm is true, but glcrun_alarm is false')
+      end if
+
 
       ! this probably belongs in seq_timemgr somewhere using proper clocks
       t1hr_alarm = .false.
@@ -2171,8 +2190,6 @@ subroutine cesm_run()
       if (tod            == 0) t24hr_alarm = .true.
       if (month==1 .and. day==1 .and. tod==0) t1yr_alarm = .true.
 
-      call seq_infodata_putData(infodata, glcrun_alarm=glcrun_alarm)
-
       if (seq_timemgr_alarmIsOn(EClock_d,seq_timemgr_alarm_datestop)) then
          if (iamroot_CPLID) then
             write(logunit,*) ' '
@@ -2900,20 +2917,36 @@ subroutine cesm_run()
             if (drv_threading) call seq_comm_setnthreads(nthreads_CPLID)
 
             if (lnd_c2_glc) then
-               call prep_glc_accum_avg(timer='CPL:glcprep_avg')
+               ! NOTE - only create appropriate input to glc if the avg_alarm is on
+               if (glcrun_avg_alarm) then
+                  call prep_glc_accum_avg(timer='CPL:glcprep_avg')
 
-               ! Note that l2x_gx is obtained from mapping the module variable l2gacc_lx
-               call prep_glc_calc_l2x_gx(fractions_lx, timer='CPL:glcprep_lnd2glc')
+                  ! Note that l2x_gx is obtained from mapping the module variable l2gacc_lx
+                  call prep_glc_calc_l2x_gx(fractions_lx, timer='CPL:glcprep_lnd2glc')
 
-               call prep_glc_mrg(infodata, fractions_gx, timer_mrg='CPL:glcprep_mrgx2g')
+                  call prep_glc_mrg(infodata, fractions_gx, timer_mrg='CPL:glcprep_mrgx2g')
 
-               call component_diag(infodata, glc, flow='x2c', comment='send glc', &
-                    info_debug=info_debug, timer_diag='CPL:glcprep_diagav')
-            endif
+                  call component_diag(infodata, glc, flow='x2c', comment='send glc', &
+                       info_debug=info_debug, timer_diag='CPL:glcprep_diagav')
+
+               else
+                  call prep_glc_zero_fields()
+               end if  ! glcrun_avg_alarm
+            end if  ! lnd_c2_glc
 
             if (drv_threading) call seq_comm_setnthreads(nthreads_GLOID)
             call t_drvstopf  ('CPL:GLCPREP',cplrun=.true.)
-         endif
+
+         end if  ! iamin_CPLID .and. glc_prognostic
+
+         ! Set the infodata field on all tasks (not just those with iamin_CPLID).
+         if (glc_prognostic) then
+            if (glcrun_avg_alarm) then
+               call seq_infodata_PutData(infodata, glc_valid_input=.true.)
+            else
+               call seq_infodata_PutData(infodata, glc_valid_input=.false.)
+            end if
+         end if
 
          !----------------------------------------------------
          !| cpl -> glc

src/drivers/mct/main/prep_glc_mod.F90

@@ -32,6 +32,8 @@ module prep_glc_mod
 
   public :: prep_glc_calc_l2x_gx
 
+  public :: prep_glc_zero_fields
+
   public :: prep_glc_get_l2x_gx
   public :: prep_glc_get_l2gacc_lx
   public :: prep_glc_get_l2gacc_lx_cnt
@@ -444,6 +446,31 @@ end subroutine prep_glc_map_one_field_lnd2glc
 
   !================================================================================================
 
+  subroutine prep_glc_zero_fields()
+
+    !---------------------------------------------------------------
+    ! Description
+    ! Set glc inputs to zero
+    !
+    ! This is appropriate during time intervals when we're not sending valid data to glc.
+    ! In principle we shouldn't need to zero the fields at these times (instead, glc
+    ! should just ignore the fields at these times). However, some tests (like an ERS or
+    ! ERI test that stops the final run segment mid-year) can fail if we don't explicitly
+    ! zero the fields, because these x2g fields can then differ upon restart.
+
+    ! Local Variables
+    integer :: egi
+    type(mct_avect), pointer :: x2g_gx
+    !---------------------------------------------------------------
+
+    do egi = 1,num_inst_glc
+       x2g_gx => component_get_x2c_cx(glc(egi))
+       call mct_aVect_zero(x2g_gx)
+    end do
+  end subroutine prep_glc_zero_fields
+
+  !================================================================================================
+
   function prep_glc_get_l2x_gx()
     type(mct_aVect), pointer :: prep_glc_get_l2x_gx(:)
     prep_glc_get_l2x_gx => l2x_gx(:)