Enable CCPP host model under CMEPS and updates for UFS exchange grid capability

[uturuncoglu]

Description of changes

This PR aims to bring exchange grid capability to UFS weather model. For this purpose, CMEPS is extended to act as a CCPP host model to calculate atmosphere-ocean fluxes by running CCPP suite files. This feature is currently only available for UFS model.

Specific notes

Contributors other than yourself, if any:
None

CMEPS Issues Fixed (include github issue #):

Are changes expected to change answers? (specify if bfb, different at roundoff, more substantial)
No

Any User Interface Changes (namelist or namelist defaults changes)?

• Two new coupling mode is included for UFS;
  • coupling_mode = nems_frac_aoflux, It allows to pass mediator calculated atmosphere-ocean fluxes to FV3
  • coupling_mode = nems_frac_aoflux_sbs, This mode is just for side-by-side comparisons of fluxes calculated on FV3/CCPP and CMEPS/CCPP. In this mode, the fluxes are calculated under mediator but not sent to FV3. So, the results of fully coupled model has no answer change since the fluxes does not received by FV3 but the mediator history files will include also fluxes calculated under mediator.
• Following new configuration options are available under MED_attributes:: group.
  • Atmosphere-ocean flux calculation specific:
    • aoflux_code = 'ccpp', It allows to select desired atmosphere-ocean flux scheme. The available options are cesm (default) and ccpp. The ccpp option is only available for UFS since it requires to find FV3 sub-directory for CCPP/physics and CCPP/framework.
    • aoflux_ccpp_suite = 'FV3_sfc_ocean', The name of the CCPP suite file that will be used to calculate atmosphere-ocean fluxes.
  • CCPP/Physics specific options: Some of those options is not used by atmosphere-ocean flux calculation but required to satisfy running simplified CCPP suite (*). Maybe there is no need to make all of them configurable through the use of nems.configure and default values works but it might be required in the near future when external land component brings in.
    • ccpp_phy_lseaspray: Include sea-spray effect to atmosphere-ocean flux calculation. The default value is true.
    • ccpp_phy_ivegsrc (*): Flag for vegetation dataset. The default value is 1.
    • ccpp_phy_redrag: Flag for reduced drag coeff. over sea. The default value is true.
    • ccpp_phy_lsm (*): Flag for land surface model. The default value is 2.
    • ccpp_phy_frac_grid: Flag for fractional grid. The default value is true.
    • ccpp_phy_restart: flag whether this is a coldstart (false) or a warmstart/restart (true). The default value is true.
    • ccpp_phy_cplice (*): Flag for activating ice coupling under CCPP. The default value is true.
    • ccpp_phy_cplflx: Flag for activating ocean coupling under CCPP. The default value is true.
    • ccpp_phy_lheatstrg (*): Flag for canopy heat storage parameterization. The default value is true.

Testing performed

Testing performed if application target is CESM:

• (recommended) CIME_DRIVER=nuopc scripts_regression_tests.py
  • machines:
  • details (e.g. failed tests):
• (recommended) CESM testlist_drv.xml
  • machines and compilers: Cheyenne, Intel and MPT
  • details (e.g. failed tests): details is in Enable CCPP host model under CMEPS and updates for UFS exchange grid capability #282 (comment)
• (optional) CESM prealpha test
  • machines and compilers
  • details (e.g. failed tests):
• (other) please described in detail
  • machines and compilers
  • details (e.g. failed tests):

Testing performed if application target is UFS-coupled:

• (recommended) UFS-coupled testing
  • description: All UFS RTs are passed on Cheyenne
  • details (e.g. failed tests):

Testing performed if application target is UFS-HAFS:

• (recommended) UFS-HAFS testing
  • description: All UFS RTs are passed on Cheyenne
  • details (e.g. failed tests):

Hashes used for testing:

• CESM:
  • repository to check out: https://github.com/ESCOMP/CESM.git
  • branch/hash: cesm2_3_alpha09a
• UFS-coupled, then umbrella repostiory to check out and associated hash:
  • repository to check out: https://github.com/uturuncoglu/ufs-weather-model
  • branch/hash: feature/xgrid_dev_host
• UFS-HAFS, then umbrella repostiory to check out and associated hash:
  • repository to check out: https://github.com/uturuncoglu/ufs-weather-model
  • branch/hash: feature/xgrid_dev_host

[climbfuji]

An impressive amount of work, thanks also for the bug fixes in ccpp-framework. I only skimmed through the CCPP-related parts of this PR. Will need to take a closer look again when it is time to commit, but from what I saw so far it's all looking good.

May want to ask @grantfirl @mkavulich to take a look as well.

[uturuncoglu]

@climbfuji the OpnReqTests threading test is failing with newly added RT. I am not sure about the source of it but answer changes when threading activated. I'll try to borrow down the issue but I just wonder if I need to be careful about CCPP host under CMEPS with threading. As I know the CCPP/physics has capability for threading. So, let me know what do you think?

[uturuncoglu]

@climbfuji I have also issue with restart test but this is more complicated then threading and might require splitting FV3 physics to call the CMEPS aoflux phase between them to allow calling aoflux phase in the same execution order with FV3/CCPP sfc_ocean. Anyway, let me know if you have also suggestion about it.

[uturuncoglu]

@jedwards4b i am getting error if I try to test cesm2_3_beta08 with updated CMEPS and CIME. Maybe it is not possible to test beta08 under this configuration.

/glade/scratch/turuncu/CESM_282/components/cmeps/cesm/nuopc_cap_share/nuopc_shr_methods.F90(135): error #6580: Name in only-list does not exist or is not accessible.   [SHR_PIO_LOG_COMP_SETTINGS]
    use shr_pio_mod, only : shr_pio_log_comp_settings
----------------------------^
compilation aborted for /glade/scratch/turuncu/CESM_282/components/cmeps/cesm/nuopc_cap_share/nuopc_shr_methods.F90 (code 1)

[jedwards4b]

Yes I'm sorry to have wasted your time. I will discuss with @mvertens at 4 today.

[uturuncoglu]

@jedwards4b No worries. In any case, i am still debugging something with threading and it could take little bit time. So, I don't have rush for this PR at this point.

[uturuncoglu]

BTW, we have another CESM baseline for testing at least.

[uturuncoglu]

@jedwards4b please do not merge this until you get confirmation from me. I am still working on fixing restart and threading ORT tests. The last commit dfdb479 seems fixed the restart issue but I just tested it manually. So, I'll update you about it.

[uturuncoglu]

@climbfuji @grantfirl As we discussed in the today's exchange grid call, I would like to update you about the current progress of ORT tests (restart and threading) that are failing. I fixed the restart issue and I have only issue with I/O at this point for xgrid case which we are looking with @jedwards4b. The agrid could be restarted without any issue.

But, I need your guidance about the threading issue. As you knows. tried to set thread number and block size explicitly to 1 with d307cd5 but this seems not working. At this point, I need your suggestion. As you already know that cdata is 2d array in FV3 implementation but it is scalar in here. So, that would be difference at this point. Anyway, let me know if you have any suggestion.

[uturuncoglu]

@jedwards4b this PR is ready but needs to be coordinated with top level UFS PR. In the mean time, if you want me to do any test, just let me know. Most of the work that I did was in UFS part and I don't think it will affect CESM.

[uturuncoglu]

@jedwards4b i am planing to merge this if you have no any additional review or testing. Then, I will create another PR in NOAA EMC side to update CMEPS overs there.