Coupling at T+0

[ukmo-juan-castillo]

The wave model currently does not allow coupling at T+0, and the first coupling exchange occurs at T+coupling time step. This means that the wave model in coupled mode can only run with a coupling lag equal or larger than the coupling time step. Although this is acceptable for high resolution configurations with very small coupling time steps, it is not for configurations with coupling time steps in the range of hours, such as low resolution or climate configurations.

Including coupling at T+0 by default will not affect the currently implemented coupling exchanges using a coupling lag. The only downside is that the coupling fields in WWIII that are sent to other models must be written in the restart file, so that they are available at T+0 before any model integration is performed.

I have already tested the approach of coupling at T+0 in a local copy of the code successfully and I would like to include it in the code. On the other hand, I would like to have some advice on the approach regarding the inclusion of the coupling fields in the restart file (e.g., adding them by default, or only when a compilation switch or namelist variable is set?)

[ukmo-juan-castillo]

Furthermore, the variable ID_OASIS_TIME has an INTEGER type, but it is often assigned a REAL number (such as the function DSEC21). To prevent possible rounding errors, I would propose wrapping this type of assignments with a NINT function using this same issue, as the value of ID_OASIS_TIME will be updated in a different way or location if the T+0 coupling is implemented.

[JessicaMeixner-NOAA]

Hi @ukmo-juan-castillo I do not use OASIS coupling and this appears to be an oasis-only coupling issue, correct?

[ukmo-juan-castillo]

Also an issue regarding the addition of coupling variables to the restart file. I believe you were working on something similar, and I added you to count with your opinion.

[JessicaMeixner-NOAA]

With ESMF we can send fields to other components at t=0/initialization. Currently fields are calculated at t=0 so that they can be sent. However, for check-point restarting we likely will need these to be in the restart file in the future.

[ukmo-juan-castillo]

I will start development now. I came with a solution that works, but in the output the fields that are received via coupling will be shifted by one hour (e.g. the current in the output file at t=3 hours will in fact be the current at t= 2 hours). This can be fixed, but as the changes required would be quite large and complex, I would propose to do it in a separate ticket if you believe this must be done.

I will also add the coupling fields to the restart in a different ticket, as again it is a large change in terms of the code that has to be added, although not in complexity.

[ukmo-ansaulter]

Thanks @ukmo-juan-castillo . Sorry to be a pain but can you just clarify why the output fields change and whether that has a potential knock on effect for non-coupled models?

Unless @JessicaMeixner-NOAA and @mickaelaccensi disagree, I'd also suggest that the restart work can be part of this ticket as it is an essential part of getting the T+0 coupling to work properly? Otherwise I'd suggest that this is part of #206 ?

[JessicaMeixner-NOAA]

I'm generally for separating work in small chunks. Also, depending on what exactly needs to be done for the restart, it will end up being a larger effort that might make sense to separate it out and do things incrementally even if both things will be required to obtain the final goal.

[aliabdolali]

@ukmo-juan-castillo Does this shift happen to current and wind together? what about water level if you have it?

[ukmo-juan-castillo]

You are right, it is better to clarify what I mean here:

With OASIS coupling the model output does not change when allowing coupling at T+0, but the logic in the code is a bit problematic when coupling without a coupling lag and when allowing coupling at T+0. To avoid deadlocks, it is advisable to first send and then receive coupling fields when coupling with oasis, and this is how the code is working right now. The order in which operations are done in a cycle is in summary:

0. Read input fields (or receive them via coupling) at time T
1. Advance time to T+dT
2. Calculate source terms at T+dT
3. Write the output grid file at T+dT
4. Send coupling fields at T+dT

This means that when writing the output grid file in step 3 for time T+dT in coupled mode, the fields that are received via coupling are only updated up to time T, as the update to T+dT is done only in step 0 at the next cycle. When looking at the output grid, all the fields received via coupling will be time shifted. This does not happen for fields received via file, as the information for both T and T+dT are read in step 0 to allow interpolation.

I found a solution that looks a bit complicated, but it consists in moving code around in only in the case of OASIS coupling, so that in this case we do:

0. Send and receive input fields (including those read by file) only if it is the first time step, do nothing otherwise
1. Advance time to T+dT
2. Calculate source terms at T+dT
3. Send and receive input fields (including those read by file) at T+dT if it is not the last time step
4. Write the output grid file at T+dT

I believe this will solve the problem in coupling without changing any of the code for uncoupled runs, and it will be easy to test.

[ukmo-juan-castillo]

While writing the changes to test this approach, I believe I have found a bug when reading the ice concentration as an homogeneous field, which can be easily be fixed.

[ukmo-ansaulter]

@ukmo-juan-castillo that makes perfect sense, thankyou.

My main question, for everyone tagged on this issue is whether this is simply an OASIS problem or something that needs to be thought about more widely for WW3 coupling? There is a fundamental question here about whether the forcing data being output by the wave model should be a) the values used to generate the wave field at time T+dt (i.e. coupled fields passed at time T), or b) the contemporary values at time T+dt.

From a wave modelling point of view, I'd probably want a) for diagnostic purposes; from a user's point of view I'd probably want b), but I guess in a lot of systems that need would be negated as I'd be using the outputs from the other component models?

So, and really sorry, a couple more questions:

• how easy would it be for the outputs time-lagging (or otherwise) to be an option?
• should this be on a generic coupling switch rather than OASIS specific?

[ukmo-juan-castillo]

I believe this is not an OASIS problem, but rather a problem in the design of the WWIII code structure.

The code in 'forced' mode works without problems because the part of the code that reads the input, reads information for time T and time T+dT at the same time for interpolation purposes. This is not possible in coupled mode when there is not a coupling lag, simply because the information for time T+dT does not exist at that point, or because if it exists and we try to read it, it will result in a deadlock.

The case is that I can make coupling without a lag work easily in WWIII, only that in the output the fields that are received via coupling will be written delayed in time. Solving this issue is more complicated but I think I found a satisfactory solution, that I would like to implement for you to review.

We definitely want to have coupling at T+0 and no lag capabilities, otherwise the information sent and received by the wave model will be delayed by the coupling frequency, which in our particular (Met Office) models is one hour. This means for example that the ocean will receive the significant weight height with one hour delay, which can be an important delay for example when trying to predict storm surges. With a coupling lag our results obtained in a forced mode will probably be of the same or better quality than in coupled mode, and if this is the case, why bother with coupling then?

[JessicaMeixner-NOAA]

I think there are some common issues in all coupling implementations, but also situations or solutions that will be unique to one coupler.

For instance, with NUOPC/ESMF coupling, we calculate the variables that are needed at t=0 and then send them to other models at initialization. So NUOPC/ESMF coupling with WW3 does not have an issue with not being able to coupled at t=0. This strategy likely needs improving for check-point restart capabilities, which might require either updates of those calculations or additional fields stored in restarts to be restart reproducible for some cases which haven't been fully tested for restart reproducibility yet.

Also, with NUOPC/ESMF coupling we have the option of different coupling sequences, either running the wave model in sync in time with the other models or with a leap-frog scheme (possibly what @ukmo-juan-castillo is referring to as 'lag'). So this also seems to be an issue with OASIS although depending on the coupling sequence the values of WXN/WYN at TWN will look different (ie. if no wind value at TWN is known the values of WX0/WY0 from TW0 are used). For restart reproducibility for this scenario in the future with NUOPC/ESMF at EMC we will be utilizing a mediator as the wave model alone will not be sufficient for storing all information in restarts due to some of the issues of coupling sequence.

[ukmo-ansaulter]

While writing the changes to test this approach, I believe I have found a bug when reading the ice concentration as an homogeneous field, which can be easily be fixed.

To be raised as a separate bugfix issue

[ukmo-ansaulter]

Thanks @JessicaMeixner-NOAA and @ukmo-juan-castillo for your inputs. Juan and I have also discussed this further offline.

In this instance the main concern is a leap-frog scheme in which the wave model lags the atmosphere model by one coupling timestep; i.e. the wave model parameters developed at T will have used winds originating at T-dtc, where dtc is the coupling timestep.

Following discussions, our proposal is that it is appropriate (in wave modelling terms) that the wind field stored at wave model time T is, in fact, the wind from T-dtc, because this is the condition from which wave parameters are model time T are derived. This only affects the coupled system, in which case a user should be accessing data appropriately from each of the component models, i.e time T waves from the wave model, time T winds from the atmosphere model, time T ocean from the ocean model.

This is the current behaviour of WW3 so, on reflection, no significant coding changes to the model output data storage is required.

However, it is suggested that we look more closely at the manual entry to ensure users are aware of output diagnostic behaviour in different coupling scenarios.

Hope that is OK with everyone?

[ukmo-juan-castillo]

In order to allow proper restart of coupled configurations, it will be possible to write in the restart the 'sent' fields that could be used in coupling both in coupled and uncoupled configurations. This will give the flexibility of starting a coupled configuration using a restart generated with an uncoupled configuration. The fields that will be written in the restart will be specified by a namelist parameter, but in the case of a coupled run all the sent fields will need to be written by default.

In order to reduce the size of the restart file, the extra fields will be written just on demand and without a write POS parameter. The disadvantage of this approach is restart files will not be reusable between configurations writing or writing a different set of coupled fields to the restart.

[JessicaMeixner-NOAA]

The lack of usability of WW3 restarts between configurations/versions is true with WW3 in general, not even considering coupled or uncoupled.