Imperfect restarts in Bering domain

[kshedstrom]

By the end of the routine set_BBL_TKE in MOM_set_diffusivity.F90, some values of visc%DKE_BBL don't match between the restart and starting from the beginning. There are mismatches between v2_bbl coming from mismatches in visc%bbl_thick_v, but I haven't figured out where they came from yet. The latter are sometimes only visible in hexadecimal mode in totalview. Surprisingly, bbl_thick_[uv] doesn't report differences in the debug output.

[kshedstrom]

u and v differ coming into step_MOM right from the get go at t=3600.

[adcroft]

To clarify, check sums for u,v after restart match but by the end of the first step they don't?

[kshedstrom]

Sorry, u and v do match at time 3600. What I learned is that to do this in debug mode, you have to make your restart file in debug mode too. bbl_thick_v now matches as well, but still not TKE_BBL right at j=4 (southern edge). v2_bbl at j=4 matches for some values of i, not others.

[kshedstrom]

This could be due to that h outside issue we were talking about. What is the best fix for setting h to zero outside on restarts? Or do we fix it by adding OBC code to not use h outside? Both?

[kshedstrom]

Honestly, h outside got set to sensible values in the non-restart case, but was still 1e-10 in the restart case.

[Hallberg-NOAA]

Hi Kate, We are doing a pull onto dev/gfdl this week of a bunch of changes to parameter defaults (and subsequently updates to our MOM6-examples test cases) that will allow us to set massless layers to actually have 0 thickness and give reasonable answers. This should help with at least with the non-reproducing answers with the specific symptoms you are describing. Most of these changes are already available by setting DEFAULT_2018_ANSWERS = False. These changes also include a number of changes that make things like our linear algebra solvers more robust. If we are very lucky, these might even address some of the instabilities you have been encountering. - Bob Hallberg Dr. Robert Hallberg Oceanographer NOAA Geophysical Fluid Dynamics Laboratory NOAA GFDL || Phone: (609) 452-6508 Princeton University Forrestal Campus || Cell: (732) 599-0459 201 Forrestal Road || Fax: (609) 987-5063 Princeton, New Jersey 08540-6649 || Email: Robert.Hallberg@noaa.gov

…

On Thu, Jun 11, 2020 at 9:31 PM Kate Hedstrom ***@***.***> wrote: Honestly, h outside got set to sensible values in the non-restart case, but was still 1e-10 in the restart case. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/NOAA-GFDL/MOM6/issues/1130#issuecomment-643015057>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABR2BFJ66MUAROT5AMQVXJDRWGAM5ANCNFSM4N2BTNTA> .

[kshedstrom]

Hi Bob and Alistair,

I did have DEFAULT_2018_ANSWERS already set to False.
Fixing the instabilities would be great.

The use of outside h starting at line 1711 of MOM_set_diffusivity.F90 needs fixing. I don't know if that is coming in your pull request or if I should fuss with it tomorrow. I was surprised to find h values of order 10, 20 outside the domain because I thought we had agreed not to set those things away from 1.e-10 or zero or what have you.

[kshedstrom]

PR #1136 pushes worrisome changes down to:

  u-point: mean=  -6.0560599528976952E-02 min=  -1.0000000000000000E+00 max=   1.0000000000000000E+00 u calc_Visbeck_coeffs slope_[xy]
  u-point: c= 112070200 sw= 112099677 se= 111941954 nw= 111986777 ne= 111832860 u calc_Visbeck_coeffs slope_[xy]
  v-point: mean=  -6.9146776804563954E-03 min=  -1.0000000000000000E+00 max=   1.0000000000000000E+00 v calc_Visbeck_coeffs slope_[xy]
  v-point: c= 112102125 sw= 112125928 se= 112059106 nw= 111914938 ne= 111852046 v calc_Visbeck_coeffs slope_[xy]
  u-point: mean=   2.1285611217717305E-05 min=  -1.4786217361362181E-04 max=   6.3638164433869110E-03 u calc_Visbeck_coeffs N2_u, N2_v
  u-point: c=  71412479 u calc_Visbeck_coeffs N2_u, N2_v
  v-point: mean=   2.1196614671105384E-05 min=  -1.4781651039840000E-04 max=   5.8967828118764573E-03 v calc_Visbeck_coeffs N2_u, N2_v
  v-point: c=  71295206 v calc_Visbeck_coeffs N2_u, N2_v
  u-point: mean=   4.7432796016329614E-06 min=   0.0000000000000000E+00 max=   1.9925280330821771E-03 u calc_Visbeck_coeffs SN_[uv]
  u-point: c=   1479060 u calc_Visbeck_coeffs SN_[uv]
  v-point: mean=   4.9393701975289042E-06 min=   0.0000000000000000E+00 max=   1.6428482245135264E-02 v calc_Visbeck_coeffs SN_[uv]

Back to the debugger.

[Hallberg-NOAA]

Based on this new output, I think that the problem is that calc_isoneutral_slopes is not aware of the faces that have open boundary conditions. For lack anything better to suggest, I think that calc_isoneutral_slopes should return a slope of 0 at OBC velocity points. (For simplicity, I would set the values as is currently done, but then zero them out at the end of the routine.) N2_u and N2_v are also set to be the stratification extrapolated from tracer to velocity points in calc_isoneutral_slopes, so these might need to be dealt with as well, but if the slopes are set to zero at OBC points, it might not matter what the stratification is, since it is the product of the two that are actually being used in the parameterizations.

There may also be a similar issue with calc_slope_functions_using_just_e (in MOM_lateral_mixing_coeffs.F90), and then we probably also want to set CS%SN_[uv] to 0 at OBC points in calc_Visbeck_coeffs.

I suspect that this knot of changes is going to require passing an OBC type into calc_slope_functions, and then on to the other 3 subroutines.

Given that you have the tests that are detecting these issues, @kshedstrom, how would you like to proceed?

[kshedstrom]

I had already started on this and will get back to you.

Speaking of h outside, do you know why T and S are being set outside as well? They too don't match on restart, but I'm not sure they are being used.

[Hallberg-NOAA]

I believe that the temperatures and salinity values outside of the domain are being used in calculations that are then masked by multiplication by a land-mask field or (in the case of OBCs) should be replaced. This works provided that there are no NaNs outside of the domain, because unfortunately 0*NaN = NaN.

[kshedstrom]

The values of N2_u and N2_v don't match yet. Are they being used? How about the values of slope_x, slope_y one grid outside the domain? Those don't match either. Something leads to differences.

[kshedstrom]

All we need now are halo updates of OBC%tres_x and/or OBC%tres_y. Adding both at once as a vector update causes trouble for dumbbell because it only has one.

[Hallberg-NOAA]

There are two viable near-term solutions that I could see for the problems you are encountering when doing the halo updates on the open boundary condition tracer reservoirs and similar fields. The first and simplest solution would be to add both of these arrays to do the vector-pair halo update, even if there are only open boundary conditions on X- or Y- faces. This is a little wasteful, but it has the advantage of working even if we move to grids with topologies that mix up the two components, like a cubed-sphere. Also, it can be implemented now in the MOM_open_boundary.F90 without any changes anywhere else.

The second near-term solution would be to extend the code in pass_var to work with variables that are staggered on the cell faces, instead of just the cell centers or corners. The underlying FMS infrastructure supports this, so it should be a minor extension to the MOM6 framework interfaces. We could then do halo updates separately on the two faces if the other does not exist, or together if both are there. I will put this benign extension of the MOM6 framework code on my to-do list, unless someone else steps up and does it first.

A third, longer term option would be to revisit the need to use global arrays to do halo updates on open boundary segment data, but that would be a very involved exercise and would go deeply into the communication code in the FMS infrastructure, and I do not see this happening any time soon.

[kshedstrom]

Closing this now with PR #1159