New interface to the Aether lat-lon model

[kdraeder]

Description:

Add a new interface to enable assimilation with the lat-lon formulation of the Aether space weather model.

Fixes issue

Fixes #559

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Documentation changes needed?

• My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • documentation external to the interface has been updated

Tests

We compiled and built state conversion programs aether_to_dart and dart_to_aether
and tested them on the latest Aether restart file sets.
Model_mod_check passes tests 1-5 and 7, except for expected failure of interpolation at the poles.
Filter is able to assimilate several observations of AIRS temperature
and GPS profiles of electron density (even though that's not available in the restart files).

Checklist for merging

• Confirm that we will not develop cycling scripting at this time (or develop it).
• Updated changelog entry
• Documentation updated
• Update conf.py

Checklist for release

• Merge into main
• Create release from the main branch with appropriate tag
• Delete feature-branch

Testing Datasets

• Dataset needed for testing available upon request
• Dataset download instructions included
• No dataset needed

[johnsonbk]

Thanks for everyone's work on this pull request. I ran into two problems following the directions in readme.rst.

First problem: model_mod_check test number 2 fails

Starting with a fresh clone of DART

cd <abs path to local DART installation>
git clone https://github.com/NCAR/DART.git DART
cd DART/
git fetch origin
git switch aether
cd build_templates
mv <suitable mkmf.template> mkmf.template
cd ../models/aether_lat-lon/work
sftp <user>@data-access.ucar.edu
get /glade/work/raeder/Exp/Aether/Ens_renamed_derecho.tgz
exit
tar -zxvf Ens_renamed_derecho.tgz
cp Ens_renamed_derecho/filter_input_0001.nc ./
./quickbuild.sh
# The executables compile
./model_mod_check
...
***************** RUNNING    TEST 2    ***********************
 -- Read and write restart file
**************************************************************
--------------------------------------------------------------
 Reading File : filter_input_0001.nc
--------------------------------------------------------------
 ERROR FROM:
  source : dart_time_io_mod.f90
  routine: read_model_time:
  message:  inconsistent calendar types between DART program and input file.
  message: ...  DART initialized with: NO_CALENDAR File uses: GREGORIAN
  message: ...  You may need to supply a model-specific "read_model_time()" to read the time.

Perhaps this failure is related to commit 3c7d8d?

Second problem: aether_to_dart doesn't use the aether_restart_dirname namelist entry in open_block_file

While trying to run aether_to_dart:

pwd 
<abs path to local DART installation>/DART/models/aether_lat-lon/work
vim input.nml
# Edit input.nml to set
aether_restart_dirname = 
      '<abs path to local DART installation>/DART/models/aether_lat-lon/work/Ens_renamed_derecho/'

# Continuing with the directions in readme.rst:
cd <aether_restart_dirname>
mkdir Orig
cp *m0000* Orig/

The following command to change directory back to the work directory is not in the directions in readme.rst but that's where aether_to_dart is compiled.

cd <abs path to local DART installation>/DART/models/aether_lat-lon/work
./aether_to_dart  0
...
ERROR FROM:
  source : aether_lat-lon/transform_state_mod.f90
  routine: open_block_file
  message: cannot open file grid_g0000.nc 
                                                                                                                   for read

If aether_to_dart and input.nml are in the aether work directory and the aether netcdf files are in a different directory, <aether_restart_dirname> this error is thrown.

Attempt number 1 to fix this error involves copying the grid files to the work directory

cp <aether_restart_dirname>/grid_g000?.nc ./
./aether_to_dart 0
...
ERROR FROM:
  source : aether_lat-lon/transform_state_mod.f90
  routine: open_block_file
  message: cannot open file neutrals_m0000_g0000.nc                                                                                                                                                                                                                                          for read

Attempt number 2 to fix this error involves copying the aether_to_dart executable and input.nml to <aether_restart_dirname>.

cp aether_to_dart <aether_restart_dirname>
cp input.nml <aether_restart_dirname>
cd <aether_restart_dirname>
./aether_to_dart
...
aether_to_dart Successfully converted the Aether restart files to 'filter_input_0001.nc'
 
 --------------------------------------
 Finished ... at YYYY MM DD HH MM SS = 
                 2024  2 14 11 23 29
 --------------------------------------

The cause of this unexpected behavior seems to be that the function open_block_file is not passed a filename including the path of <aether_restart_dirname>. The executable only works if it is copied to <aether_restart_dirname> or if <aether_restart_dirname> is the same directory as the aether work directory. The reason this behavior is unexpected is that aether_restart_dirname is set in transform_state_nml so a user would expect that aether_to_dart would actually use that namelist entry as the directory in which the restart files are stored.

[hkershaw-brown]

...
***************** RUNNING TEST 2 ***********************
-- Read and write restart file

---

---

Reading File : filter_input_0001.nc

ERROR FROM:
source : dart_time_io_mod.f90
routine: read_model_time:
message: inconsistent calendar types between DART program and input file.
message: ... DART initialized with: NO_CALENDAR File uses: GREGORIAN
message: ... You may need to supply a model-specific "read_model_time()" to read the time.

Perhaps this failure is related to commit 3c7d8d?

This is my bad, I meant force the calendar to be Gregorian rather than having a variable calder (implying a user choice). I'll go ahead and fix this.

[kdraeder]

@johnsonbk Thanks for working through the file transformation instructions.
I'll work on that, unless you already started fixing the issues.

The cause of this unexpected behavior seems to be that the function open_block_file is not passed a filename including the path of <aether_restart_dirname>. The executable only works if it is copied to <aether_restart_dirname> or if <aether_restart_dirname> is the same directory as the aether work directory. The reason this behavior is unexpected is that aether_restart_dirname is set in transform_state_nml so a user would expect that aether_to_dart would actually use that namelist entry as the directory in which the restart files are stored.

[johnsonbk]

I haven't begun to start fixing the transform_state bug, @kdraeder. Please proceed.

[kdraeder]

Second problem: aether_to_dart doesn't use the aether_restart_dirname namelist entry in open_block_file

This brings up a strategy question, which I thought about at the beginning, but lost track of.
We've decided not to develop full cycling scripting at this time,
so we don't have that context to define the use of aether_to_dart, filter, and dart_to_aether.
The experiment structure I'm most familiar with has the following:

1. All of the DART executables are copied to an exec_dir.
2. All of the ensemble model output is written to a run_dir and has a full date in the filenames. (The Aether restart file names do not have the date in them, so the scripting will need to prevent overwriting of files which need to be kept.)
3. The assimilation script runs the executables in the run_dir using full pathnames of the executables.
4. All of the DART output is written to the run_dir.
5. The updated state variables are written into the existing model restart files.

An advantage of this is that the run_dir does not need to be specified in the DART executables
because the scripting changes to that directory before running the executables.
They only need to know file names. So far I don't see any disadvantages.

Is this the structure we want here?
Or do we want the executables to be able to find the restart files wherever they are?
And write their output to an output location, which may be different from the input files location?

[nancycollins]

the wrf folks seem to prefer putting each ensemble member's input and output in a separate subdir, probably because that makes the model advances easier to run in parallel by letting each member run in their own subdirectory without interfering with each other.

as long as it doesn't complicate the code too much, enabling flexible options for how someone wants to run both the assimilation and the model advances would be good. i know in cesm that the model advances can run in parallel using cesm options, but that may not be true for aether.

also, you probably don't want to assume that the scripting is always going to do a model advance before an assimilation, like cesm does. it's very useful for exploring parameter/namelist settings or debugging to run filter alone from a simple batch script without any other scripting requirements, as long as all the files are in the correct place with the correct names.

having said all that, it sounds like getting something running is the highest priority, so whatever works quickly might be the strategic thing here.

[kdraeder]

I opted to make aether_to_dart and dart_to_aether use the same directory for the Aether and filter restart files,
but allow those programs to executed from any directory.
I tested and pushed those commits.

[hkershaw-brown]

say this is c shell

[mgharamti]

I guess whoever runs this would need to write a loop to generate all filter restarts one by one. Any reason why not generate all members internally? Is there a scenario where a user doesn't need to create all members?

[kdraeder]

We needed working code as soon as possible and a single member program was simpler to write.
It's possible to run multiple aether_to_darts in parallel, so it shouldn't be much slower than
having aether_to_dart handle all members at once.
There are testing and development scenarios which only need 1 member, and perfect_model_obs.

[mgharamti]

I tried to ncview individual members are running aether_to_dart but the fields don't show up as I expect. I was hoping to see a global field and then change through the levels. However, I could only change through the lon dimension which is not very useful. This is because the dimensions of the 3D variables are ordered as lon, lat, alt. I ran a simple 'ncpdq -a alt,lon filter_input_0001.nc filter_input_0001_swap.nc' on the file to permute the dimensions and then I was able to see the entire globe at different levels. I suggest modifying aether_to_dart to adjust the dimension order.

[kdraeder]

The coordinate order is annoying. Aether won't change, and we decided to use it in order to make the conversion programs simpler to code and run faster (no transposing).
Ncview can show a lat-lon slice by changing the axes. But the lat and lon are transposed, so it's not perfect. Other common viewing tools can transpose it.

[mgharamti]

If I run aether_to_dart with an argument that is larger than the ensemble size, it fails. Yet, it creates a filter_input netcdf file including only dimension variables. I think it should fail before creating any file

[kdraeder]

Good catch! I'll try to fix that.

[kdraeder]

The fix is to read the model time from the actual member instead of member 0.
It will fail before the filter_input.nc file is created.

[mgharamti]

Do we need any clamping? maybe for O+?

[kdraeder]

We could certainly make 0.0 be the lower limit for the ion densities and temperatures, since we expect that inflation will sometimes make those be negative. Or leave that as an exercize for the user.
Whether it's needed or not depends on whether Aether and the assimilation setup work well together. One advantage of not clamping (especially the max values) is that it helps users see when an assimilation is failing, instead of silently correcting itself.

[mgharamti]

How about the cutoff, is 0.2 the default that we want users to use?

[kdraeder]

My guess is that this is a science question the users will need to answer.
0.2 has been reasonable in other global models, but if @johnsonbk has any insight, we can set a different default value.

[mgharamti]

in pmo: the input_state_files is set to 'wrfinput_d01'. We probably need to change that

[kdraeder]

That will need to be changed, but pmo is not in the scope of this stage of development,
so we haven't looked at that. I'll replace wrfinput_d01 with "pmo not ready for use".

[mgharamti]

add outlier_threshold = 3

[kdraeder]

I added that, and 'input_qc_threshold = 3.0'. This input.nml hasn't been used directly for assimilation, so it may be missing other variables, or have suboptimal values.
@johnsonbk may also have an input.nml that's been updated to do an assimilation.

[mgharamti]

set inf_flavor to 5. Also, change inf_sd_initial and inf_sd_lower_bound to 0.6

[kdraeder]

Done.

[nancycollins]

i just left a comment on the change to the dart version of the netcdf utilities cover functions. it's not important for getting aether running, but sometime in the future someone should go back and remove all the NF90 constants and make better named functions that either return those values or do the code that uses the constants.

[nancycollins]

i'm not sure when the NF90 constants got added to this fie, but the intent was never to replicate the actual netcdf utilities mod. adding more constants to this is just going down the wrong path. these functions were intended to be more user-friendly ways to access netcdf without having to get into the weeds of the netcdf interface. if you need these constants in your code, 'use netcdf' or make a cover function that has the name of what you're doing and uses this constant inside the function. e.g. from the name i'm guessing it fills an array. make a
nc_fill_array_blank(array) that doesn't require netcdf constants in the calling code.

i know this isn't an important fix, but it's a long-term bad idea. maybe after the crush of getting this working is done, someone can go back and make this better?

[kdraeder]

Yes, I didn't do much work to figure out the existing strategy for using netcdf parameters.
The template I was working from used some, so I assumed it was not terrible to do that.

[nancycollins]

which is the only reason i brought it up. once the camel's nose is in the tent...
it's not important for this pull request and a later change to your code will be easy. i will open an issue on this because i'm not requesting a change to this pull request, so this is the wrong place to address this.

[mgharamti]

@kdraeder I tested your changes. I am still getting a restart file even though the argument is larger than the ensemble size. I was expecting it to fail before creating a new netcdf file. And btw, we need the same logic applied to dart_to_aether.
This, however, is not a deal breaker and I'm OK to move forward with the PR in its current form.

[mgharamti]

Great, thanks Kevin. This works in both directions now!

[hkershaw-brown]

Moha and Kevin in person approved!