Update WRF-DART interface to support WRF4

[braczka]

Description:

Make WRF-DART scripting compatible with WRF4 as outlined in Issue #661. Also provides updates to
documentation in WRF tutorial and main model page, advising of this change, and describing limits
to backward compatibility to WRFv3.9 and earlier.

Fixes issue

Fixes #661 fixes #672 fixes #680

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Documentation changes needed?

• My change requires a change to the documentation.
• I have updated the documentation accordingly.

Tests

Performed testing against WRF-DART Tutorial example as outlined in PR #650

Checklist for merging

• Updated changelog entry
• Documentation updated
• Update conf.py

Checklist for release

• Merge into main
• Create release from the main branch with appropriate tag
• Delete feature-branch

Testing Datasets

• Dataset needed for testing available upon request
• Dataset download instructions included
• No dataset needed

[hkershaw-brown]

nice stuff @braczka

[braczka]

Just remembered -- I still need to make minor updates to the tutorial tar.gz file, and documentation:

cd $BASE_DIR
wget http://www.image.ucar.edu/wrfdart/tutorial/wrf_dart_tutorial_23May2018_v3.tar.gz
tar -xzvf wrf_dart_tutorial_23May2018_v3.tar.gz

The perturbations, surface file and initial conditions were generated from a pre WRF4 version, but we
can still use the same data to run the tutorial example. I just need to add a few more lines of documentation so the user is not confused. Will also update README files in the tar.gz file.

[braczka]

Just to document our conversation from DART standup today ... my testing of these changes have been limited to the tutorial test case example as outlined in PR #650, where the results looked similar to the original WRFv3.9 test case. A thorough evaluation of the impact of these changes upon forward operators (i.e. radio occultation, radar etc) has not been done. We may need to rely on our user-base to test these impacts....

[hkershaw-brown]

Hi Brett, this looks great.

One definite change is the default table for state variables should be THM rather than T.

I've put a couple of questions on the add_bank_perts.ncl. This is more for my own understanding of ncl. I don't think anything is incorrect, just at first glance it looks like some unnecessary (duplicate) writes. But this also could be me mis-reading the .ncl.

I checked the changes against the notes in issue #661. Checked all the links on the docs worked. Let me know if there is anything else that needs looking at.

Edit: looked at the .tar.gz download, README looks good in that.

Cheers,
Helen

[hkershaw-brown]

I might not be following the ncl here, but this looks like you set
temp_w = wrf_in->$pert_fields(n)$ which is T
then immediately overwrite it with
temp_w = wrf_in->$wrf_fields(n)$ which is THM

[braczka]

Good catch -- I think I just forgot to remove the original line that I edited. Not very smart on my part.

[hkershaw-brown]

ok one last comment I promise, do you want to fix #672 in this pull request? (update the wrf/work/input.nml)

[braczka]

ok one last comment I promise, do you want to fix #672 in this pull request? (update the wrf/work/input.nml)

Yes, I should include #672, and probably #660 as well, to clean up those two issues.

[braczka]

I seem to be getting a new error since the last shutdown of Derecho. I have been getting the following error when executing step ./driver.csh 2017042706 param.csh >& run.out &. The error is related to the adding of perturbations to the wrfinput file as:

host is  dec2323
assim_advance.csh is running in /glade/derecho/scratch/bmraczka/WRFv4.5_Tutorial/rundir
new_advance_model.csh is running in /glade/derecho/scratch/bmraczka/WRFv4.5_Tutorial/rundir
/glade/derecho/scratch/bmraczka/WRFv4.5_Tutorial/rundir
use wrfvar set
stuff var  U
wrf.info is read
1
/glade/derecho/scratch/bmraczka/WRFv4.5_Tutorial/rundir/WRF/wrfbdy_d01_152057_43200_mean
Error! Non-zero status returned from add_bank_perts.ncl. Check /glade/derecho/scratch/bmraczka/WRFv4.5_Tutorial/rundir/advance_temp1/add_perts.err.
warning:_NclOpenFile: Can not open file <wrfvar_output>; file format not supported or file is corrupted
^Mfatal:file (wrf_in) isn't defined
^Mfatal:["Execute.c":8637]:Execute: Error occurred at or near line 55 in file /glade/derecho/scratch/bmraczka/WRFv4.5_Tutorial/rundir/advance_temp1/add_bank_perts.ncl

^Mduration = 13

The source of the issue seems to be that two calls are made to assim_advance.csh for ensemble member 1, diagnosed by two log files that are created:

-rw-r--r-- 1 bmraczka ncar 858 May  9 16:22 assim_advance_1.o4413480
-rw-r--r-- 1 bmraczka ncar 858 May  9 16:22 assim_advance_1.o4413479

Scripts operating simultaneously seem to compete for access to wrfvar_output file leading to error. Solution seems to be adding some sleep commands within driver.csh script, so script can detect that the assim_advance.csh has been started before generating a second. Will include this commit within this PR.

[mgharamti]

Brett, could this be due to the CPU binding issue we experienced in WRF-Hydro?
If you would like to test our fix, here is the environment command:
export PALS_CPU_BIND=none (or setenv in csh)

You can add this in your submission script right after the PBS preamble.

[braczka]

Brett, could this be due to the CPU binding issue we experienced in WRF-Hydro? If you would like to test our fix, here is the environment command: export PALS_CPU_BIND=none (or setenv in csh)

You can add this in your submission script right after the PBS preamble.

Hmmm -- not sure, but I will test that too !

[braczka]

@hkershaw-brown I think I have addressed all of your concerns. Some additional changes we discussed at standup are not optimal (adding scripting pauses to csh scripts). A more robust fix likely would require more substantial refactor. I will meet with WRF users from EOL and RAL to better address need for refactor. However, this PR fix for hybrid coordinate system and T to THM switch are important to get to community.

[braczka]

Brett, could this be due to the CPU binding issue we experienced in WRF-Hydro? If you would like to test our fix, here is the environment command: export PALS_CPU_BIND=none (or setenv in csh)
You can add this in your submission script right after the PBS preamble.

Hmmm -- not sure, but I will test that too !

@mgharamti This PALS_CPU_BIND variable did not seem to influence the WRF simulation. I was also going to test this for slow performance with CLM-DART, but that appears to be related to compression of large data files related to campaign migration.

[hkershaw-brown]

Approved!