Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable different coupled options (NSST on/off) #230

Closed
XuLi-NOAA opened this issue Dec 30, 2020 · 49 comments · Fixed by #359
Closed

Enable different coupled options (NSST on/off) #230

XuLi-NOAA opened this issue Dec 30, 2020 · 49 comments · Fixed by #359
Assignees
@KateFriedman-NOAA
Milestone
Prototype 7

Comments

@XuLi-NOAA
Copy link
Contributor

XuLi-NOAA commented Dec 30, 2020 •
edited by WalterKolczynski-NOAA
Loading

In order to turn the NSST models, including the diurnal warming and skin-layer cooling model, on in the coupled model, some modifications of the global-workflow are necessary, since the NSST model hasn't been tested systematically in the coupled frame.

In principle, the NSST model is ready to run since it has been built in fv3atm and it is a component of the operational GFS (uncoupled). There is a switch to turn on the NSST model through a name list array.

However, there is a fundamental issue to deal with when turning on the NSST model in the coupled system, i.e., how to combine the NSST T-Profile, particularly the diurnal warming profile, with the oceanic model layer T-Profile to determine the foundation temperature, and then obtain the required underlying surface temperatures for the air-sea interface fluxes calculation for the coupled model and a radiative transfer model such as CRTM for the data assimilation.
There is a scheme to determine the foundation temperature and the required surface temperatures, referred to Scheme-A, already built in the current fv3atm, specifically in the ccpp/physics submodule.
The goal of this task is to make sure the coupled system work well the the NSST model turned on, including the necessary modifications in the workflow, the test of the Scheme-A and the development of a more effective scheme, Scheme-B, to incorporate the NSST T-Profile into oceanic model layer T-Profile to provide a more realistic surface temperatures in fluxes calculation and the radiance simulation.
@XuLi-NOAA XuLi-NOAA changed the title branch feature/ufs_nst_1.0: global-workflow for the coupled model with the NSST model on Branch feature/ufs_nst_1.0: global-workflow for the coupled model with the NSST model on Dec 30, 2020
@XuLi-NOAA
Copy link
Contributor Author

XuLi-NOAA commented Dec 30, 2020 •
edited by WalterKolczynski-NOAA
Loading

Based on the branch workflow-crow, and ufs-s2s prototype 5, with Jiande's help, a coupled model configuration with the NSST model on has been established. See https://vlab.ncep.noaa.gov/redmine/issues/54074 (enable the nsst in fv3-mom6-cice5).
Two 35-day runs, CTL (NSSTM off) & NST (NSSTM on), have been done with the initial conditions from the current CFS and GODAS at 2011101011. The results analysis (oceanic side here, Joshua will do more on the atmospheric side, and then the coupled system later on) have shown: (1) The NSST model works in the coupled model but with an issue, the dominant feature is that the skin-layer cooling effect is doubled unexpectedly; (2) The temporal (diurnal) variability of the derived foundation temperature is too large (the monthly mean diurnal variability is up to a few tenth degree); (3) The verifications against the drifting buoys and two DYNAMO moored buoys show that the diurnal cycle can be simulated by the NSST diurnal warming model and MOM6 reasonably well, but with errors.
A Scheme-A.1 has been proposed to replace Scheme-A, and the test run has been done and it shows some improvement.

@XuLi-NOAA
Copy link
Contributor Author

A new version of the coupled model has been adopted with Scheme-A.1 and a bug fix in ccpp/physics, see NCAR/ccpp-physics#537.

The two test run, ufs_nst_ctl (w/o NSST model) and ufs_nst_nst (with NSST model), have been done. The preliminary results analysis have shown that the "too much cooling" issue has been fixed, but the too large diurnal variability of the foundation temperature is still there, which means that a better scheme is needed to replace Scheme-A.1.

@WalterKolczynski-NOAA
Copy link
Contributor

@XuLi-NOAA @joshuafu Is this a duplicate of #196?

@JoshuaFu-NOAA
Copy link

JoshuaFu-NOAA commented Feb 23, 2021 •
edited
Loading

Two groups worked on the same project.

@JoshuaFu-NOAA
Copy link

Thanks for merging these two issues together!

@WalterKolczynski-NOAA WalterKolczynski-NOAA added duplicate This issue or pull request already exists and removed duplicate This issue or pull request already exists labels Feb 23, 2021
@WalterKolczynski-NOAA
Copy link
Contributor

Since this is the one with activity, I'm going to close the other one

Closed
@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title Branch feature/ufs_nst_1.0: global-workflow for the coupled model with the NSST model on Enable different coupled options (NSST on/off) and output frequencies in coupled models Feb 23, 2021
@JessicaMeixner-NOAA
Copy link
Contributor

To enable NSST or not, all code changes are already in ufs-weather-model as of today. An example of how it's used can be found in the p7b branch https://github.com/ufs-community/ufs-weather-model/tree/release/P7b in the 35 day tests.

@KateFriedman-NOAA KateFriedman-NOAA self-assigned this Jun 16, 2021
@WalterKolczynski-NOAA
Copy link
Contributor

Split the output frequency part, which was inherited from #196 when it was marked as a duplicate, off into its own issue as it is separate from NSST updates.

@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title Enable different coupled options (NSST on/off) and output frequencies in coupled models Enable different coupled options (NSST on/off) Jun 22, 2021
@WalterKolczynski-NOAA
Copy link
Contributor

To enable NSST or not, all code changes are already in ufs-weather-model as of today. An example of how it's used can be found in the p7b branch https://github.com/ufs-community/ufs-weather-model/tree/release/P7b in the 35 day tests.

It would be useful to know what the settings are, or at least exactly which file to look for. Is it actually more than setting NST_MODEL to 2?

@JessicaMeixner-NOAA
Copy link
Contributor

To enable NSST or not, all code changes are already in ufs-weather-model as of today. An example of how it's used can be found in the p7b branch https://github.com/ufs-community/ufs-weather-model/tree/release/P7b in the 35 day tests.

It would be useful to know what the settings are, or at least exactly which file to look for. Is it actually more than setting NST_MODEL to 2?

Comparing the test with NSST (https://github.com/ufs-community/ufs-weather-model/blob/release/P7b/tests/tests/cpld_bmark_wave_v16_noahmp_nsst) and without NSST (https://github.com/ufs-community/ufs-weather-model/blob/release/P7b/tests/tests/cpld_bmark_wave_v16_noahmp_nsst) should tell you everything you need.

@XuLi-NOAA can give more specific guidance in particulars, but the main this is that you use the correct CPP suite file with NSST in the build and then when running. Then there are some input.nml changes such as https://github.com/ufs-community/ufs-weather-model/blob/release/P7b/tests/tests/cpld_bmark_wave_v16_noahmp_nsst#L111

@WalterKolczynski-NOAA WalterKolczynski-NOAA added this to the Prototype 7 milestone Jun 22, 2021
@XuLi-NOAA
Copy link
Contributor Author

XuLi-NOAA commented Jun 22, 2021 •
edited by KateFriedman-NOAA
Loading

In order to turn on the NSST model in the coupled model, the following are required:

  1. Modify workflow/config/nsst.yaml to handle nst_spinup parameter setting (by Jian Kuang)
    See the example of this file at /work/noaa/da/xli/git/coupled-workflow/workflow/config/nsst.yaml

#This file is used to generate config.nsst, which would be used to
#control NSST settings. Specifically, it controls the nstf_name
#option in the gfs physics namelist (&gfs_physics_nml)

config_nsst:
  filename: config.nsst
  content: !expand |
    #!/bin/ksh -x

    # This file is automatically generated from the YAML-based system
    # in ecf/ecfutils/.  Any changes will be overwritten if
    # setup_case.sh is rerun.

    ########## config.nsst ##########
    # NSST specific

    echo "BEGIN: config.nsst"

    # NSST parameters contained within nstf_name

    # nstf_name(1) : NST_MODEL (NSST Model) : 0 = OFF, 1 = ON but uncoupled, 2 = ON and coupled
    export NST_MODEL={doc.nsst.NST_MODEL}

    # NSST Spinup
    export NST_SPINUP={doc.nsst.NST_SPINUP}

    # nstf_name(3) : NST_RESV (Reserved, NSST Analysis) : 0 = OFF, 1 = ON
    export NST_RESV={doc.nsst.NST_RESV}

    # nstf_name(4,5) : ZSEA1, ZSEA2 the two depths to apply vertical average (bias correction)
    export ZSEA1={doc.nsst.ZSEA1}
    export ZSEA2={doc.nsst.ZSEA2}

    export NST_GSI={doc.nsst.NST_GSI}          # default 0: No NST info at all;
                              #         1: Input NST info but not used in GSI;
                              #         2: Input NST info, used in CRTM simulation, no Tr analysis
                              #         3: Input NST info, used in both CRTM simulation and Tr analysis
    export NSTINFO={doc.nsst.NSTINFO}          # number of elements added in obs. data array (default = 0)
    if [ $NST_GSI -gt 0 ]; then export NSTINFO=4; fi

    echo "END: config.nsst"

  1. Modify ush/parsing_namelists_FV3.sh to be able to use the suite with NSST

    There is a line in this file such as:

elif [ $CCPP_SUITE = "FV3_GFS_v16_coupled" -o $CCPP_SUITE = "FV3_GFS_v16_couplednsst" ]; then

we can see (in that version, P6),

FV3_GFS_v16_coupled is the suite to use sfc_ocean (no NSST)

FV3_GFS_v16_couplednsst is the suite to use NSST

Now, for P7, the suite is different, but there are still two suites for NSST model on or off respectively.

  1. Set the namelist variables to turn on the NSST model
    See the example (P6),

/work/noaa/da/xli/git/coupled-workflow/workflow/cases/prototype6_nst.yaml
/work/noaa/da/xli/git/coupled-workflow/workflow/cases/prototype6_ctl.yaml

The diff of these two yaml files shows:

 diff prototype6_nst.yaml prototype6_ctl.yaml
23,27c23
<     NST_MODEL: 2
<     NST_SPINUP: 1
<     NST_RESV: 0
<     ZSEA1: 0
<     ZSEA2: 0
---
>     NST_MODEL: 0
44c40
<     CCPP_SUITE: FV3_GFS_v16_couplednsst
---
>     CCPP_SUITE: FV3_GFS_v16_coupled

@XuLi-NOAA
Copy link
Contributor Author

@KateFriedman-NOAA @WalterKolczynski-NOAA @JessicaMeixner-NOAA

See the above updates

@KateFriedman-NOAA
Copy link
Member

KateFriedman-NOAA commented Jun 24, 2021 •
edited
Loading

I added "FV3_GFS_v16_couplednsst" to the CCPP_SUITES list in build_ufs_coupled.sh:

diff --git a/sorc/build_ufs_coupled.sh b/sorc/build_ufs_coupled.sh
index e18e6fd3..c80c642d 100755
--- a/sorc/build_ufs_coupled.sh
+++ b/sorc/build_ufs_coupled.sh
@@ -23,5 +23,5 @@ if [[ -d build ]]; then rm -Rf build; fi
 if [[ -d GOCART ]]; then
   ./build.sh
 else
-  CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_v16" ./build.sh
+  CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_v16_couplednsst,FV3_GFS_v16" ./build.sh
 fi

...and reran build_ufs_coupled.sh (ran it previously before adding "FV3_GFS_v16_couplednsst"). The build got a failure, please see this log on Hera:

/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/sorc/logs/build_ufs_coupled_2.log

I ran sh checkout.sh -c and sh build_all.sh -c when setting up this global-workflow clone of feature/coupled-crow. Did I miss a step? Thanks!

@JessicaMeixner-NOAA
Copy link
Contributor

JessicaMeixner-NOAA commented Jun 24, 2021 •
edited
Loading

@KateFriedman-NOAA it does not look the the suite file you are requesting FV3_GFS_v16_couplednsst exists.

$ls /scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/sorc/ufs_coupled.fd/FV3/ccpp/suites/FV3_GFS_v16_couplednsst
ls: cannot access /scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/sorc/ufs_coupled.fd/FV3/ccpp/suites/FV3_GFS_v16_couplednsst: No such file or directory

Two options to move forward, temporarily use the 'FV3_GFS_2017_couplednsst' suite file or build from @WalterKolczynski-NOAA work on updating the model and the suite file FV3_GFS_v16_couplednsst exists. Note eventually for p7 we'll likely be using "FV3_GFS_v16_coupled_nsstNoahmp" but any of the suite files that activate NSST should work for testing the functionality of turning it on/off

@KateFriedman-NOAA
Copy link
Member

@JessicaMeixner-NOAA Thanks, it built with the 'FV3_GFS_2017_couplednsst' suite. I will test with that to start.

@XuLi-NOAA Other than testing with NST_MODEL=2 & NST_SPINUP=1 what other NST settings should I try? Are there other settings for NST_RESV, ZSEA1, and/or ZSEA2 that you'd like me to test? Thanks!

@KateFriedman-NOAA
Copy link
Member

All, I am starting to run feature/coupled-crow for this task on Hera. I ran with the 'coupled_free_forecast' case first to make sure I could run the branch (no issues). I then tried the following prototype tests:

  1. cfftest2 - prototype6 case (as is - no changes)
  2. cfftest2b - prototype6 case with CCPP_SUITE: FV3_GFS_2017_couplednsst and NSST off (NSST settings as is)
  3. cfftest3 - prototype6 case with CCPP_SUITE: FV3_GFS_2017_couplednsst and NSST variables added and turned on

For the cfftest3 I initially created a new prototype6_nst.yaml file (following Xu's example):

-bash-4.2$ pwd
/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/cases
-bash-4.2$ diff prototype6.yaml prototype6_nst.yaml
25c25,29
<     NST_MODEL: 0
---
>     NST_MODEL: 2
>     NST_SPINUP: 1
>     NST_RESV: 0
>     ZSEA1: 0
>     ZSEA2: 0
42c46
<     CCPP_SUITE: FV3_GFS_v16_coupled
---
>     CCPP_SUITE: FV3_GFS_v16_couplednsst

...but since the 'FV3_GFS_v16_couplednsst' suite file isn't available in feature/coupled-crow (when I branched off it) I set up a similar case file with the suggested 'FV3_GFS_2017_couplednsst' suite to test NSST:

-bash-4.2$ diff prototype6.yaml prototype6_nsttest.yaml
25c25,29
<     NST_MODEL: 0
---
>     NST_MODEL: 2
>     NST_SPINUP: 1
>     NST_RESV: 0
>     ZSEA1: 0
>     ZSEA2: 0
42c46
<     CCPP_SUITE: FV3_GFS_v16_coupled
---
>     CCPP_SUITE: FV3_GFS_2017_couplednsst

For the cfftest2b test I only change the suite in a test case file:

-bash-4.2$ diff prototype6.yaml prototype6_nsttest0.yaml
42c42
<     CCPP_SUITE: FV3_GFS_v16_coupled
---
>     CCPP_SUITE: FV3_GFS_2017_couplednsst

I rebuilt my coupled FV3 code with 'FV3_GFS_2017_couplednsst' for cfftest2[b][3]:

-bash-4.2$ pwd
/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/sorc
-bash-4.2$ git diff build_ufs_coupled.sh
diff --git a/sorc/build_ufs_coupled.sh b/sorc/build_ufs_coupled.sh
index e18e6fd3..b3aab68c 100755
--- a/sorc/build_ufs_coupled.sh
+++ b/sorc/build_ufs_coupled.sh
@@ -23,5 +23,6 @@ if [[ -d build ]]; then rm -Rf build; fi
 if [[ -d GOCART ]]; then
   ./build.sh
 else
-  CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_v16" ./build.sh
+  #CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_v16_couplednsst,FV3_GFS_v16" ./build.sh
+  CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_2017_couplednsst,FV3_GFS_v16" ./build.sh
 fi

My cfftest2 (prototype case as is) is running ok so far (fcst/post jobs running now):
EXPDIR: /scratch1/NCEPDEV/global/Kate.Friedman/expdir/cfftest2
COMROT: /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest2

Both my cfftest2b and cfftest3 tests with the 'FV3_GFS_2017_couplednsst' suite failed with the same error, see error and log below from cfftest3:

FATAL from PE 1054: atmosphere_dynamics: error in ccpp_physics_init for group fast_physics: Group fast_physics not found

See line 17648 in this log on Hera: /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest3/logs/2013040100/gfs.forecast.highres.log

Here is the resulting input.nml diffs between the three prototype tests:

-bash-4.2$ pwd
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot
-bash-4.2$ diff cfftest2/input.nml cfftest2b/input.nml
15c15
<   ccpp_suite = FV3_GFS_v16_coupled
---
>   ccpp_suite = FV3_GFS_2017_couplednsst
156,170c156,157
<   iovr         = 3
<   ltaerosol    = .false.
<   lradar       = .false.
<   ttendlim     = 0.005
<   oz_phys      = .false.
<   oz_phys_2015 = .true.
<   lsoil_lsm    = 4
<   do_mynnedmf  = .false.
<   do_mynnsfclay = .false.
<   icloud_bl    = 1
<   bl_mynn_edmf = 1
<   bl_mynn_tkeadvect = .true.
<   bl_mynn_edmf_mom = 1
<   min_lakeice  = 0.15
<   min_seaice   = 1e-11
---
>   iovr_lw      = 3
>   iovr_sw      = 3
-bash-4.2$ diff cfftest2b/input.nml cfftest3/input.nml
204c204
<   nstf_name    = 0,0,0,0,0
---
>   nstf_name    = 2,1,0,0,0

build from @WalterKolczynski-NOAA work on updating the model and the suite file FV3_GFS_v16_couplednsst exists.

Should I work off of a branch where the 'FV3_GFS_v16_couplednsst' suite file exists? Thanks!

Additionally, I see the following errors in the logs for both cfftest2 and cfftest3 (are these expected?):

/bin/cp: cannot stat '/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest3/gfs.20130401/00/ufs.cpld*.nc': No such file or directory
32 + /bin/cp -p /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest3/gfs.20130401/00/rpointer.cpl /scratch1/NCEPDEV/stmp2/Kate.Friedman/RUNDIRS/cfftest3/2013040100/gfs/fcst.38549/
/bin/cp: cannot stat '/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest3/gfs.20130401/00/rpointer.cpl': No such file or directory

line 24673 in failed fcst job log ($FHRGRP isn't known):

61 + '[' -eq 0 ']'
24674 /scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/scripts/../ush/forecast_postdet.sh: line 681: [: -eq: unary operator expected

@XuLi-NOAA
Copy link
Contributor Author

XuLi-NOAA commented Jun 28, 2021 via email

@XuLi-NOAA
Copy link
Contributor Author

@JessicaMeixner-NOAA Thanks, it built with the 'FV3_GFS_2017_couplednsst' suite. I will test with that to start.

@XuLi-NOAA Other than testing with NST_MODEL=2 & NST_SPINUP=1 what other NST settings should I try? Are there other settings for NST_RESV, ZSEA1, and/or ZSEA2 that you'd like me to test? Thanks!

@KateFriedman-NOAA : NST_MODEL=2 & NST_SPINUP=1, NST_RESV, ZSEA1, and/or ZSEA2 are zeros, which corresponds to nstf_name=(2,1,0,0,0), is what needs to be tested.

@XuLi-NOAA
Copy link
Contributor Author

All, I am starting to run feature/coupled-crow for this task on Hera. I ran with the 'coupled_free_forecast' case first to make sure I could run the branch (no issues). I then tried the following prototype tests:

  1. cfftest2 - prototype6 case (as is - no changes)
  2. cfftest2b - prototype6 case with CCPP_SUITE: FV3_GFS_2017_couplednsst and NSST off (NSST settings as is)
  3. cfftest3 - prototype6 case with CCPP_SUITE: FV3_GFS_2017_couplednsst and NSST variables added and turned on

For the cfftest3 I initially created a new prototype6_nst.yaml file (following Xu's example):

-bash-4.2$ pwd
/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/cases
-bash-4.2$ diff prototype6.yaml prototype6_nst.yaml
25c25,29
<     NST_MODEL: 0
---
>     NST_MODEL: 2
>     NST_SPINUP: 1
>     NST_RESV: 0
>     ZSEA1: 0
>     ZSEA2: 0
42c46
<     CCPP_SUITE: FV3_GFS_v16_coupled
---
>     CCPP_SUITE: FV3_GFS_v16_couplednsst

...but since the 'FV3_GFS_v16_couplednsst' suite file isn't available in feature/coupled-crow (when I branched off it) I set up a similar case file with the suggested 'FV3_GFS_2017_couplednsst' suite to test NSST:

-bash-4.2$ diff prototype6.yaml prototype6_nsttest.yaml
25c25,29
<     NST_MODEL: 0
---
>     NST_MODEL: 2
>     NST_SPINUP: 1
>     NST_RESV: 0
>     ZSEA1: 0
>     ZSEA2: 0
42c46
<     CCPP_SUITE: FV3_GFS_v16_coupled
---
>     CCPP_SUITE: FV3_GFS_2017_couplednsst

For the cfftest2b test I only change the suite in a test case file:

-bash-4.2$ diff prototype6.yaml prototype6_nsttest0.yaml
42c42
<     CCPP_SUITE: FV3_GFS_v16_coupled
---
>     CCPP_SUITE: FV3_GFS_2017_couplednsst

I rebuilt my coupled FV3 code with 'FV3_GFS_2017_couplednsst' for cfftest2[b][3]:

-bash-4.2$ pwd
/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/sorc
-bash-4.2$ git diff build_ufs_coupled.sh
diff --git a/sorc/build_ufs_coupled.sh b/sorc/build_ufs_coupled.sh
index e18e6fd3..b3aab68c 100755
--- a/sorc/build_ufs_coupled.sh
+++ b/sorc/build_ufs_coupled.sh
@@ -23,5 +23,6 @@ if [[ -d build ]]; then rm -Rf build; fi
 if [[ -d GOCART ]]; then
   ./build.sh
 else
-  CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_v16" ./build.sh
+  #CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_v16_couplednsst,FV3_GFS_v16" ./build.sh
+  CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_2017_couplednsst,FV3_GFS_v16" ./build.sh
 fi

My cfftest2 (prototype case as is) is running ok so far (fcst/post jobs running now):
EXPDIR: /scratch1/NCEPDEV/global/Kate.Friedman/expdir/cfftest2
COMROT: /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest2

Both my cfftest2b and cfftest3 tests with the 'FV3_GFS_2017_couplednsst' suite failed with the same error, see error and log below from cfftest3:

FATAL from PE 1054: atmosphere_dynamics: error in ccpp_physics_init for group fast_physics: Group fast_physics not found

See line 17648 in this log on Hera: /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest3/logs/2013040100/gfs.forecast.highres.log

Here is the resulting input.nml diffs between the three prototype tests:

-bash-4.2$ pwd
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot
-bash-4.2$ diff cfftest2/input.nml cfftest2b/input.nml
15c15
<   ccpp_suite = FV3_GFS_v16_coupled
---
>   ccpp_suite = FV3_GFS_2017_couplednsst
156,170c156,157
<   iovr         = 3
<   ltaerosol    = .false.
<   lradar       = .false.
<   ttendlim     = 0.005
<   oz_phys      = .false.
<   oz_phys_2015 = .true.
<   lsoil_lsm    = 4
<   do_mynnedmf  = .false.
<   do_mynnsfclay = .false.
<   icloud_bl    = 1
<   bl_mynn_edmf = 1
<   bl_mynn_tkeadvect = .true.
<   bl_mynn_edmf_mom = 1
<   min_lakeice  = 0.15
<   min_seaice   = 1e-11
---
>   iovr_lw      = 3
>   iovr_sw      = 3
-bash-4.2$ diff cfftest2b/input.nml cfftest3/input.nml
204c204
<   nstf_name    = 0,0,0,0,0
---
>   nstf_name    = 2,1,0,0,0

build from @WalterKolczynski-NOAA work on updating the model and the suite file FV3_GFS_v16_couplednsst exists.

Should I work off of a branch where the 'FV3_GFS_v16_couplednsst' suite file exists? Thanks!

Additionally, I see the following errors in the logs for both cfftest2 and cfftest3 (are these expected?):

/bin/cp: cannot stat '/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest3/gfs.20130401/00/ufs.cpld*.nc': No such file or directory
32 + /bin/cp -p /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest3/gfs.20130401/00/rpointer.cpl /scratch1/NCEPDEV/stmp2/Kate.Friedman/RUNDIRS/cfftest3/2013040100/gfs/fcst.38549/
/bin/cp: cannot stat '/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest3/gfs.20130401/00/rpointer.cpl': No such file or directory

line 24673 in failed fcst job log ($FHRGRP isn't known):

61 + '[' -eq 0 ']'
24674 /scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/scripts/../ush/forecast_postdet.sh: line 681: [: -eq: unary operator expected

@KateFriedman-NOAA : Actually, you have added the ccpp/suite and done the compile again. So, just make sure the right suite is used (for P7 exactly, as Jiande has used already, I think, but it's better to confirm with him or Jun).
Here is another remind, see the item 2 comment I made 6 days ago:

  1. Modify ush/parsing_namelists_FV3.sh to be able to use the suite with NSST

    There is a line in this file such as:

elif [ $CCPP_SUITE = "FV3_GFS_v16_coupled" -o $CCPP_SUITE = "FV3_GFS_v16_couplednsst" ]; then

we can see (in that version, P6),

FV3_GFS_v16_coupled is the suite to use sfc_ocean (no NSST)

FV3_GFS_v16_couplednsst is the suite to use NSST

Now, for P7, the suite is different, but there are still two suites for NSST model on or off respectively.

===

That means you need to modify ush/parsing_namelists_FV3.sh to use the suite with nsst as well.

@KateFriedman-NOAA
Copy link
Member

NST_MODEL=2 & NST_SPINUP=1, NST_RESV, ZSEA1, and/or ZSEA2 are zeros, which corresponds to nstf_name=(2,1,0,0,0), is what needs to be tested.

Ok, thanks! I think the NSST variables are making it through ok then. Just need to get a successful fcst job run.

You need modify build_ufs_coupled.sh to add the new ccpp/suite, and then recompile the coupled model

@XuLi-NOAA I modified build_ufs_coupled.sh already; see my prior comment in this issue: #230 (comment)

I tried with 'FV3_GFS_v16_couplednsst' in the CCPP_SUITES list:

CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_v16_couplednsst,FV3_GFS_v16" ./build.sh

...but the build failed because the suite file isn't available (see this log: /scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/sorc/logs/build_ufs_coupled_2.log). Jessica recommended I try with one of the other existing NSST suite files so I reran the build with 'FV3_GFS_2017_couplednsst' (build completed without issue):

  CMAKE_FLAGS="-DS2S=ON -DWW3=ON" CCPP_SUITES="FV3_GFS_v15p2_coupled,FV3_GFS_v16_coupled,FV3_GFS_2017_couplednsst,FV3_GFS_v16" ./build.sh

I then set the CCPP suite to also be 'FV3_GFS_2017_couplednsst' in my case file. Here is the resulting setting in my config.base:

export CCPP_SUITE="FV3_GFS_2017_couplednsst"

If the 'FV3_GFS_2017_couplednsst' suite doesn't work or isn't the proper one to test with then please point me to a copy of the 'FV3_GFS_v16_couplednsst' suite file so I can pull it into my copy of the FV3 or point me to a different FV3 hash that has the file. Thanks!

@XuLi-NOAA
Copy link
Contributor Author

I can see FV3_GFS_v16_coupled_nsstNoahmpUGWPv1 is the one Jiande has used. You should use the same one. But better confirm with him or Jun.

@WalterKolczynski-NOAA
Copy link
Contributor

This looks like a WW3 version mismatch that I thought we fixed.

@JessicaMeixner-NOAA
Copy link
Contributor

It's a WW3 error, which case file are you using @KateFriedman-NOAA

@WalterKolczynski-NOAA
Copy link
Contributor

Update CPL_WAVIC to GEFSwave20210623 in the case file.

@KateFriedman-NOAA
Copy link
Member

Chatted with @JessicaMeixner-NOAA offline, I started with the wrong case file (prototype6.yaml), I need to add NST variables to the coupled_free_forecast_wave.yaml case file. Doing that and retrying. Thanks!

@JessicaMeixner-NOAA
Copy link
Contributor

At this point we've gone far enough away from prototype6, that if someone wants to run that, they should probably just use the exact commit hash and we could delete the p6 yaml case files.

@KateFriedman-NOAA
Copy link
Member

Alrighty...got myself using the right starting case file (coupled_free_forecast_wave.yaml). I created a copy of it for testing (coupled_free_forecast_wave_nst.yaml). Diffs:

-bash-4.2$ diff coupled_free_forecast_wave.yaml coupled_free_forecast_wave_nst.yaml
25c25,29
<     NST_MODEL: 0
---
>     NST_MODEL: 2
>     NST_SPINUP: 1
>     NST_RESV: 0
>     ZSEA1: 0
>     ZSEA2: 0
42c46
<     CCPP_SUITE: FV3_GFS_v16_coupled
---
>     CCPP_SUITE: FV3_GFS_v16_couplednsst

I set up two tests:

  1. cfftest0 - coupled_free_forecast_wave.yaml as is, baseline
  2. cfftest6 - coupled_free_forecast_wave_nst.yaml

Both failed with the same error, see logs:

/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest6/logs/2013040100/gfs.forecast.highres.log
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest0/logs/2013040100/gfs.forecast.highres.log

Update CPL_WAVIC to GEFSwave20210623 in the case file.

Per an earlier suggestion from Walter I changed the case file to use 'GEFSwave20210623` for CPL_WAVIC:

-bash-4.2$ grep CPL_WAVIC coupled_free_forecast_wave*yaml
coupled_free_forecast_wave.yaml:    CPL_WAVIC: GEFSwave20210623
coupled_free_forecast_wave_nst.yaml:    CPL_WAVIC: GEFSwave20210623

...but when I tried to set up a retest of the baseline (cfftest0b) it complained about CPL_WAVIC:

-bash-4.2$ ./setup_case.sh -p HERA coupled_free_forecast_wave cfftest0b
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/CROW/utils/worktools.py", line 745, in setup_case
    doc=from_dir(EXPDIR,validation_stage='setup')
  File "/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/CROW/crow/config/__init__.py", line 106, in from_dir
    validation_stage=validation_stage)
  File "/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/CROW/crow/config/__init__.py", line 50, in from_string
    multi_document=multi_document)
  File "/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/CROW/crow/config/from_yaml.py", line 275, in convert
    self.result,validation_stage)
  File "/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/CROW/crow/config/eval_tools.py", line 533, in recursively_validate
    recursively_validate(subobj,stage,validation_memo,inheritence_memo)
  File "/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/CROW/crow/config/eval_tools.py", line 529, in recursively_validate
    obj._validate(stage)
  File "/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/CROW/crow/config/eval_tools.py", line 349, in _validate
    tmpl._check_scope(self,stage,memo)
  File "/scratch1/NCEPDEV/global/Kate.Friedman/git/feature-coupled-crow-nsst/workflow/CROW/crow/config/template.py", line 196, in _check_scope
    if errors: raise TemplateErrors(errors)
crow.config.exceptions.TemplateErrors: Scope did not validate:
doc.wave_settings[0].CPL_WAVIC='GEFSwave20210623': not an allowed value ('CFSR', 'CFSRwave20200925')

@WalterKolczynski-NOAA
Copy link
Contributor

workflow/schema/wave.yaml needs to be updated to allow GEFSwave20210623

This change is made in the PR, are you sure you started from a clean copy?

@KateFriedman-NOAA
Copy link
Member

@XuLi-NOAA Jessica/Walter helped me get the feature/coupled-crow branch running (both a control and one with NSST on). Both forecasts ran ok but you'll notice an error at the end of both because of a known issue (unrelated to my testing). Please see the following logs and output below on Hera. Please let me know if you see any issues with the NSST on run. The NSST workflow updates are shown below as well; I will be sending them to Walter tomorrow. Thanks!

1. control (no NSST, system as is):

log: /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest0b/logs/2013040100/gfs.forecast.highres.log
output: /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest0b/gfs.20130401/00/

-bash-4.2$ grep nstf_name /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest0b/logs/2013040100/gfs.forecast.highres.log
0 + nstf_name=0,0,0,0,0
0 + nstf_name=0,0,0,0,0
  nstf_name    = 0,0,0,0,0
  nstf_name    = 0,0,0,0,0
  nstf_name    = 0,0,0,0,0
  nstf_name    = 0,0,0,0,0
  nstf_name         :            0           0           0           0

2. test (NSST updates in system and NSST turned on including a spinup):

log: /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest7/logs/2013040100/gfs.forecast.highres.log.0
output: /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest7/gfs.20130401/00/

-bash-4.2$ grep nstf_name /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest7/logs/2013040100/gfs.forecast.highres.log
1 + nstf_name=2,1,0,0,0
1 + nstf_name=2,1,0,0,0
  nstf_name    = 2,1,0,0,0
  nstf_name    = 2,1,0,0,0
  nstf_name    = 2,1,0,0,0
  nstf_name    = 2,1,0,0,0
  nstf_name(1)=           2
  nstf_name(2)=           1
  nstf_name(3)=           0
  nstf_name(4)=           0
  nstf_name(5)=           0
  nstf_name         :            2           1           0           0

Workflow updates for NSST:

diff --git a/sorc/build_ufs_coupled.sh b/sorc/build_ufs_coupled.sh
index e2176967..9d368fff 100755
--- a/sorc/build_ufs_coupled.sh
+++ b/sorc/build_ufs_coupled.sh
@@ -25,5 +25,5 @@ if [[ -d GOCART ]]; then
   CMAKE_FLAGS="-DAPP=ATMAERO" CCPP_SUITES="FV3_GFS_v16" ./build.sh
   ./build.sh
 else
-  CMAKE_FLAGS="-DAPP=S2SW" CCPP_SUITES="FV3_GFS_v16_coupled,FV3_GFS_v16" ./build.sh
+  CMAKE_FLAGS="-DAPP=S2SW" CCPP_SUITES="FV3_GFS_v16_coupled,FV3_GFS_v16_couplednsst,FV3_GFS_v16" ./build.sh
 fi

diff --git a/ush/parsing_namelists_FV3.sh b/ush/parsing_namelists_FV3.sh
index ddcca3a2..b14eec96 100755
--- a/ush/parsing_namelists_FV3.sh
+++ b/ush/parsing_namelists_FV3.sh
@@ -215,7 +215,7 @@ elif [ $CCPP_SUITE = "FV3_GSD_v0" ]; then
   min_lakeice  = ${min_lakeice:-"0.15"}
   min_seaice   = ${min_seaice:-"0.15"}
 EOF
-elif [ $CCPP_SUITE = "FV3_GFS_v16_coupled" ]; then
+elif [ $CCPP_SUITE = "FV3_GFS_v16_coupled" -o $CCPP_SUITE = "FV3_GFS_v16_couplednsst" ]; then
   cat >> input.nml << EOF
   iovr         = ${iovr:-"3"}
   ltaerosol    = ${ltaerosol:-".false."}

diff --git a/workflow/config/nsst.yaml b/workflow/config/nsst.yaml
index 6a188a03..bc965adf 100644
--- a/workflow/config/nsst.yaml
+++ b/workflow/config/nsst.yaml
@@ -4,14 +4,6 @@

 config_nsst:
   filename: config.nsst
-  nst_spinup_logic: !FirstTrue
-    - when: !calc doc.nsst.get("NST_SPINUP","")
-      do: !expand "export NST_SPINUP={doc.nsst.NST_SPINUP}"
-    - otherwise: |
-        export NST_SPINUP=0
-        #if [[ "$CDATE" = $SDATE ]]; then
-        #    export NST_SPINUP=1
-        #fi
   content: !expand |
     #!/bin/ksh -x

@@ -30,7 +22,7 @@ config_nsst:
     export NST_MODEL={doc.nsst.NST_MODEL}

     # nstf_name(2) : NST_SPINUP : 0 = OFF, 1 = ON,
-    {nst_spinup_logic}
+    export NST_SPINUP={doc.nsst.NST_SPINUP}

     # nstf_name(3) : NST_RESV (Reserved, NSST Analysis) : 0 = OFF, 1 = ON
     export NST_RESV={doc.nsst.NST_RESV}

@KateFriedman-NOAA
Copy link
Member

@XuLi-NOAA @WalterKolczynski-NOAA Question on the yaml case files and NSST...should I add the NSST settings to the coupled_free_forecast_wave.yaml or add a new case file (e.g. coupled_free_forecast_wave_nst.yaml). I created a new one for testing but can merge the changes into the existing case file; see diffs between existing and new testing yaml case files:

-bash-4.2$ diff coupled_free_forecast_wave.yaml coupled_free_forecast_wave_nst.yaml
25c25,29
<     NST_MODEL: 0
---
>     NST_MODEL: 2
>     NST_SPINUP: 1
>     NST_RESV: 0
>     ZSEA1: 0
>     ZSEA2: 0
42c46
<     CCPP_SUITE: FV3_GFS_v16_coupled
---
>     CCPP_SUITE: FV3_GFS_v16_couplednsst

@XuLi-NOAA
Copy link
Contributor Author

XuLi-NOAA commented Jul 8, 2021 via email

@XuLi-NOAA
Copy link
Contributor Author

XuLi-NOAA commented Jul 8, 2021 via email

@WalterKolczynski-NOAA
Copy link
Contributor

WalterKolczynski-NOAA commented Jul 8, 2021 •
edited
Loading

@XuLi-NOAA @WalterKolczynski-NOAA Question on the yaml case files and NSST...should I add the NSST settings to the coupled_free_forecast_wave.yaml or add a new case file (e.g. coupled_free_forecast_wave_nst.yaml). I created a new one for testing but can merge the changes into the existing case file; see diffs between existing and new testing yaml case files:

-bash-4.2$ diff coupled_free_forecast_wave.yaml coupled_free_forecast_wave_nst.yaml
25c25,29
<     NST_MODEL: 0
---
>     NST_MODEL: 2
>     NST_SPINUP: 1
>     NST_RESV: 0
>     ZSEA1: 0
>     ZSEA2: 0
42c46
<     CCPP_SUITE: FV3_GFS_v16_coupled
---
>     CCPP_SUITE: FV3_GFS_v16_couplednsst

Make a new case for now. In the future we'll update the main case files.

@JessicaMeixner-NOAA
Copy link
Contributor

Updating the main case file even now seems fine as we're turning on NSST.

@JessicaMeixner-NOAA
Copy link
Contributor

JessicaMeixner-NOAA commented Jul 9, 2021 •
edited
Loading

@LydiaStefanova-NOAA can you and/or @SulagnaRay-NOAA look at the actual output of Kate's run (ignoring known issues with ice) and make sure it looks like NSST is correctly turned on in the output:
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest7/gfs.20130401/00/
compared to the run without NSST:
/scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest0b/gfs.20130401/00/
(Note not all the settings are the same as the mini-prototypes yet, so this is just to make sure it seems that NSST is in fact turned on correctly).

I can double check the inputs, but @KateFriedman-NOAA has already done that as well.

KateFriedman-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this issue Jul 9, 2021
@KateFriedman-NOAA
Add spaces to indent FV3 build section for better readability
a51d4ef 
Refs: NOAA-EMC#230
KateFriedman-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this issue Jul 9, 2021
@KateFriedman-NOAA
Add FV3_GFS_v16_couplednsst CCPP suite option to UFS coupled build an…
5bec2a7 
…d FV3 namelist parser

Refs: NOAA-EMC#230
KateFriedman-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this issue Jul 9, 2021
@KateFriedman-NOAA
Replace CROW yaml nst_spinup_logic in nsst.yaml with NST_SPINUP variable
bbcad18 
- removed CROW FirstTrue check when building config.nsst from nsst.yaml
- replace nst_spinup_logic with NST_SPINUP variable that is parsed from variable dictionary
- nst_spinup_logic not needed currently

Refs: NOAA-EMC#230
KateFriedman-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this issue Jul 9, 2021
@KateFriedman-NOAA
Add NSST settings to the coupled_free_forecast_wave.yaml case file
cec6188 
- set NST_MODEL=2 (turns it on)
- add additional NSST namelist variables: NST_SPINUP, NST_RESV, ZSEA1, ZSEA2
- set NST_SPINUP=1 (NSST will spinup)
- change CCPP_SUITE from FV3_GFS_v16_coupled to FV3_GFS_v16_couplednsst

Refs: NOAA-EMC#230
@LydiaStefanova-NOAA
Copy link

I confirm that the results are consistent with NSST being turned on, based on seeing that the surface temperature in the experiment (cfftest7) is cooler than the control (cfftest0b)) over ocean, ~40S to 40N, as illustrated below.

image

@JessicaMeixner-NOAA
Copy link
Contributor

Thanks @LydiaStefanova-NOAA

@LydiaStefanova-NOAA
Copy link

PS: I do notice some warmer surface temps for lakes ~(60E,45N) and ~(50E,Eq) but don't have any experience with whether that's ok or not

@KateFriedman-NOAA
Copy link
Member

I confirm that the results are consistent with NSST being turned on

Awesome, thank you for checking @LydiaStefanova-NOAA !

I have pushed the needed NSST changes up to @WalterKolczynski-NOAA 's feature/model_update branch in PR #359. Walter, perhaps that PR can now close issue #230 as well? Thanks!

@XuLi-NOAA
Copy link
Contributor Author

XuLi-NOAA commented Jul 9, 2021 via email

@JessicaMeixner-NOAA
Copy link
Contributor

@KateFriedman-NOAA I confirmed that everything looks like I believe we want it in the input.nml:

diff /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest0b/gfs.20130401/00/atmos/input.nml /scratch1/NCEPDEV/stmp4/Kate.Friedman/comrot/cfftest7/gfs.20130401/00/atmos/input.nml 
15c15
<   ccpp_suite = FV3_GFS_v16_coupled
---
>   ccpp_suite = FV3_GFS_v16_couplednsst
217c217
<   nstf_name    = 0,0,0,0,0
---
>   nstf_name    = 2,1,0,0,0

Thanks!

@KateFriedman-NOAA
Copy link
Member

I confirmed that everything looks like I believe we want it in the input.nml

Thanks @JessicaMeixner-NOAA !

@XuLi-NOAA
Copy link
Contributor Author

XuLi-NOAA commented Jul 9, 2021 via email

@WalterKolczynski-NOAA WalterKolczynski-NOAA linked a pull request Jul 10, 2021 that will close this issue
Update coupled to latest UFS version #359
Merged
Merged
lgannoaa pushed a commit to lgannoaa/global-workflow that referenced this issue Feb 9, 2022
@gthompsnJCSDA @gthompsnWRF
freshly updated code to handle Thompson 2-moment cloud water and chan…
1b703a3 
…ged graupel Y-intercept calculation (NOAA-EMC#230)

Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KateFriedman-NOAA KateFriedman-NOAA

Labels
None yet
Projects
None yet
Milestone
Prototype 7
Development

Successfully merging a pull request may close this issue.

Update coupled to latest UFS version WalterKolczynski-NOAA/global-workflow
7 participants
@JessicaMeixner-NOAA @JoshuaFu-NOAA @WalterKolczynski-NOAA @KateFriedman-NOAA @junwang-noaa @LydiaStefanova-NOAA @XuLi-NOAA