Add ocean wetting-and-drying ramp feature

[cbegeman]

This PR enhances the existing wetting-and-drying algorithm using an approach from O'Dea et al. 2020 (doi:10.1016/j.ocemod.2020.101708). Instead of zeroing out normalVelocity and normalVelocity tendencies in cells where the projected layerThickness drops below the user-defined minimum, this method ramps down the normalVelocity and its tendencies between a range of layerThicknesses.

Wetting-and-drying is currently only used in standalone ocean runs.

[BFB]

[sbrus89]

@cbegeman, while testing this in debug mode on compy, I ran into an out-of-bounds error for the dam-break case. The error was from line 677:

E3SM/components/mpas-ocean/src/mode_forward/mpas_ocn_time_integration_rk4.F

Lines 671 to 682 in c7f67cb

	if (config_disable_tr_all_tend) then
	num_tracers = size(activeTracersNew, dim=2)
	do iTracer = 1, num_tracers
	!$omp parallel
	!$omp do schedule(runtime)
	do iCell = 1, nCells
	activeTracersNew(iTracer, :, iCell) = activeTracersCur(iTracer, :, iCell)
	end do
	!$omp end do
	!$omp end parallel
	end do
	end if

I think num_tracers should be calculated as num_tracers = size(activeTracersNew, dim=1).

[cbegeman]

@sbrus89 Thanks for catching that! Totally a typo. That bug was introduced with #5519 and is now fixed by fc4375d

[sbrus89]

Thanks for fixing that @cbegeman! Also, testing in debug I see an error in the single layer drying slope case:

forrtl: error (72): floating overflow
Image              PC                Routine            Line        Source    
libpnetcdf.so.4    00002B8A5B1C3B1C  for__signal_handl     Unknown  Unknown
libpthread-2.17.s  00002B8A5D1715D0  Unknown               Unknown  Unknown
ocean_model        000000000367CBBC  mpas_io_mp_mpas_i        2496  mpas_io.F
ocean_model        00000000036A6657  mpas_io_mp_mpas_i        3052  mpas_io.F
ocean_model        000000000378571A  mpas_io_streams_m        4097  mpas_io_streams.F
ocean_model        00000000038188BC  mpas_stream_manag        3382  mpas_stream_manager.F
ocean_model        0000000003813BE7  mpas_stream_manag        2819  mpas_stream_manager.F
ocean_model        0000000001A14D6A  ocn_forward_mode_         750  mpas_ocn_forward_mode.F
ocean_model        00000000019D5E30  ocn_core_mp_ocn_c         111  mpas_ocn_core.F
ocean_model        00000000004179B5  mpas_subdriver_mp         358  mpas_subdriver.F
ocean_model        00000000004126D4  MAIN__                     20  mpas.F
ocean_model        0000000000412612  Unknown               Unknown  Unknown
libc-2.17.so       00002B8A5DA923D5  __libc_start_main     Unknown  Unknown
ocean_model        0000000000412529  Unknown               Unknown  Unknown

I'm having a little more trouble tracking the source of this one down. Similar to the last issue, I don't think this has anything to do with this PR.

[cbegeman]

@sbrus89 I can look into that as well. I haven't run these tests on compy recently. Do you know if that error is emerging when config_disable_tr_all_tend is true, false or both?

[sbrus89]

The above error was for config_disable_tr_all_tend=.true. for config_disable_tr_all_tend=.false. I get

forrtl: severe (408): fort: (2): Subscript #1 of the array TRACERCUR has value 2 which is greater than the upper bound of 1

Image              PC                Routine            Line        Source    
libpnetcdf.so.4    00002B53DB25D6B2  for_emit_diagnost     Unknown  Unknown
ocean_model        0000000002EAF9E6  ocn_tracer_advect         700  mpas_ocn_tracer_advection_mono.F
libiomp5.so        00002B53DD840713  __kmp_invoke_micr     Unknown  Unknown
libiomp5.so        00002B53DD7CF6A3  __kmp_fork_call       Unknown  Unknown
libiomp5.so        00002B53DD79663D  __kmpc_fork_call      Unknown  Unknown
ocean_model        0000000002E85F69  ocn_tracer_advect         691  mpas_ocn_tracer_advection_mono.F
ocean_model        000000000267C43F  ocn_tracer_advect         102  mpas_ocn_tracer_advection.F
ocean_model        0000000002691DD0  ocn_tendency_mp_o        1097  mpas_ocn_tendency.F
ocean_model        00000000027D5F2E  ocn_time_integrat        1075  mpas_ocn_time_integration_rk4.F
ocean_model        00000000027BDEF1  ocn_time_integrat         523  mpas_ocn_time_integration_rk4.F
ocean_model        0000000001AB65DD  ocn_time_integrat         131  mpas_ocn_time_integration.F
ocean_model        0000000001A1493C  ocn_forward_mode_         728  mpas_ocn_forward_mode.F
ocean_model        00000000019D5E30  ocn_core_mp_ocn_c         111  mpas_ocn_core.F
ocean_model        00000000004179B5  mpas_subdriver_mp         358  mpas_subdriver.F
ocean_model        00000000004126D4  MAIN__                     20  mpas.F
ocean_model        0000000000412612  Unknown               Unknown  Unknown
libc-2.17.so       00002B53DDB303D5  __libc_start_main     Unknown  Unknown
ocean_model        0000000000412529  Unknown               Unknown  Unknown

[cbegeman]

@sbrus89 Thanks for looking into this.

I will plan to debug the config_disable_tr_all_tend=.true. case

There seems to be an assumption in ocn_tracer_advection_mono_tend that nVertLevels > 1 that leads to the above error, so I don't think we'll want to run with config_disable_tr_all_tend=.false. for these single layer test cases.

[cbegeman]

@sbrus89 I finally made the time to figure this out and it comes down to the fact that the RK4 changes I made in #5519 allow activeTracers to be supported (but disabled) but not debugTracers. To prevent this from happening in the future I will:

• open a compass PR to remove debugTracers from single_layer cases: Fixup namelists for single layer RK4 ocean tests MPAS-Dev/compass#619
• add a commit here that terminates the run when it is a single layer case and config_use_debugTracers = .true. in RK4 init

[cbegeman]

@sbrus89 Ready for your re-review with MPAS-Dev/compass#619

[sbrus89]

@cbegeman - thanks I'll test it out.

[sbrus89]

@cbegeman, this passes the drying slope and dam break cases with debug flags on compy. Thanks for putting this feature in!

[cbegeman]

@sbrus89 Thanks for your review and testing!

[jonbob]

passes:

• SMS_D_Ld3.T62_oQU120.CMPASO-IAF.chrysalis_intel
• SMS_D_Ld1.ne30pg2_EC30to60E2r2.WCYCL1850.chrysalis_intel.allactive-wcprod
• ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel

merged to next

[jonbob]

merged to master

[cbegeman]

@jonbob Thank you for your testing! Happy to have this merged!