Add ensemble forcing functionality

[weiyuan-jiang]

Addresses #343 -- option to use ensemble of surface met forcing (typically from ADAS ensemble when LDAS is running within LADAS).

[gmao-rreichle]

Thanks, @weiyuan-jiang, for putting this together, looks great!
I added on commit with minimal edits to comments.
I did not look into why the GNU build test failed.
I guess the next step is to add the ENSEMBLE_FORCING parameter to the resource file and ask Sara to test the new functionality.

cc: @saraqzhang

[weiyuan-jiang]

@saraqzhang , each member will be initialized independently and call LDAS_GetForcing() independently too. The "path" and "tag" are passed into the subroutine LDAS_Getforcing. I would like your help to locate the files with the path and tag. I assume the tag you were (or will be) using is new.

[saraqzhang]

@weiyuan-jiang Weiyuan, I am setting up an experiment for using ens forcing, and wanted to confirm the following:
(1) the ensemble forcing path for each ensemble member is expected to be $path with 3 character ( from id_string) attached . For example, if we define MET_PATH : ../input/met_forcing/forcens in LDAS.rc , then in ../input/met_forcing/ we should setup forcens001 -> $adashome/atmens/ensddiag/mem001
...
forcens031 -> $adashome/atmens/ensdiag/mem031
(2) I need to add ENSEMBLE_FORCING: YES to LDAS.rc

[weiyuan-jiang]

@saraqzhang , I don't think we can run this branch now. We need to add codes to locate the forcing files. I think "ENSEMBLE_FORCING" is helpful for the script to figure out how to locate the forcing files. What are the tag and fields are you using?

[saraqzhang]

@weiyuan-jiang
I looked at the following code in GEOS_MetforceGridComp.F90:

if (ensemble_forcing .and. NUM_ENSEMBLE > 1) then
id_string = ""
call MAPL_GetResource ( MAPL, ens_id_width, Label="ENS_ID_WIDTH:", DEFAULT=0, RC=STATUS)
k = len(trim(comp_name))
id_string = comp_name(k-ens_id_width+1:k)
k = len(trim(id_string))
! hard coded 3 character for forcing
call ESMF_CFIOStrTemplate(ens_forcing_path, trim(adjustl(mf%Path)),'GRADS', xid = trim(id_string(k-2:k)), stat=status)
mf%Path = ens_forcing_path
endif
! Put MetForcing in Ldas' pvt internal state
internal%mf = mf

---

if we have:
in LDAS.rc : MET_TAG: x529_C180__Nx+-
MET_PATH: ../input/met_forcing/forcens
in met_forcing/ we set up forcens001 ( -> link to adas ens001/ where the forcing is ) , forcens002 ,.. forcens024.
Does ESMF_CFIOStrTemplate() make a path string unique for each member (each igc) : ../input/met_forcing/forcensxxx , which will be in pvt mf%path ?

[weiyuan-jiang]

@saraqzhang , Do you have codes that implement the tag "x529_C180__Nx+-" ? Basically, I will need to know the fiedds ( or variables ) that need to be read in.

[saraqzhang]

@weiyuan-jiang the forcing fields for the ensemble case are same as that of one-set forcing case. You can find the fields specification in /discover/nobackup/projects/c612/user/qzhang/x529_C180/run/atmens/HISTAENS.rc.tmpl the collections of "inst1_2d_lfo_Nx+-" and "tavg1_2d_lfo_Nx+-" .

[weiyuan-jiang]

It seems the "inst" and "avg" have different fields. I am still looking for your codes to read those fields since you have already done some calculations.

[saraqzhang]

@weiyuan-jiang the forcing fields in tavg and inst are different, as they have always been for ldas.

[weiyuan-jiang]

Does ADAS forcing member start from 1 ? GEOSldas by default starts form 0. @saraqzhang

[saraqzhang]

@weiyuan-jiang adas ens index starts from 1 ( mem001/ ,... , mem/032) .

[saraqzhang]

@weiyuan-jiang @gmao-rreichle
so far in LADAS setup we have not used LDAS configuration FIRST_ENS_ID=1. In the current ADAS it is assumed the ldas ens index starts at 0 ( in the script to connect ensemble ldas_incr to atm ensemble member ) .
We certainly can require setup with FIRST_ENS_ID=1 when come to configure future LADAS experiments , and put in modification correspondingly in ADAS script ( atmos_eldas.csh) ,

[gmao-rreichle]

so far in LADAS setup we have not used LDAS configuration FIRST_ENS_ID=1.

I suppose we didn't have a need because there was no connection between individual ens members in the LDAS and the ADAS. Going forward, though, I think we should have the ADAS setup script ensure FIRST_ENS_ID=1 in the LDAS setup so that the ens IDs match across the ADAS and LDAS. That is, assume that there is no offset between the ADAS and LDAS ens IDs.

[weiyuan-jiang]

That is a good point. I will take advantage of the flexibility that GEOSldas provides

[weiyuan-jiang]

@saraqzhang If you add "ENSEMBLE_FORCING: YES" and MET_PATH: /.../mem without id numbers, you should be able to run ldas setup ( FIRST_ENS_ID should start from 1 too). If it cannot be setup, please leave your exeinput file to, I will debug it.

[saraqzhang]

@weiyuan-jiang I will give it a try and let you know.

[saraqzhang]

@weiyuan-jiang I tried the setup with ENSEMBLE_FORCING: YES, MET_PATH: /.../mem , FIRST_ENS_ID : 1,
and it is for LDAS_COUPLING : 2 .
The link in met_forcing/ is not going to work properly : input/met_forcing/mem001 -> ../input/met_forcing/mem001
It should be something like : met_forcing/mem001 -> /discover/nobackup//qzhang/LA5294tag/atmens/ensdiag/mem001
where "/discover/nobackup//qzhang/LA5294tag" is the ADAS_EXPDIR in coupled ldas setup.

[saraqzhang]

@weiyuan-jiang is there more input parameter for forcing ( such as explicit forcing path for linking ) that I did not have in my expinp ?

[weiyuan-jiang]

In your case , did you set "MET_PATH: /discover/nobackup//qzhang/LA5294tag/atmens/ensdiag/mem" in you exeinp?

[saraqzhang]

@weiyuan-jiang no I have not used the whole path in MET_PATH, I will try this again.

[saraqzhang]

@weiyuan-jiang with MET_PATH: /discover/nobackup//qzhang/LA5294tag/atmens/ensdiag/mem in setup, now they are linked to the correct adas directories. Is it ready for a test run?

[weiyuan-jiang]

Yes, please try to run it. If anything wrong, please let me know. @saraqzhang

[saraqzhang]

@weiyuan-jiang I ran the test and compared the results ( fcst, incr) with the case of using ens-averaged forcing, the ensemble frocing connection and application work correctly.
A couple items to consider : (1) if ENSEMBLE_FORCING: YES & LDAS_COUPLING : 2 , the block that carries out ensemble forcing averaging in lenkf.j should be skipped. (2) if FIRST_ENS_ID : 1 , RESTART: 1 should consider the possibility that restart files could from a previous run with first-ens-id 0.

[weiyuan-jiang]

@weiyuan-jiang I ran the test and compared the results ( fcst, incr) with the case of using ens-averaged forcing, the ensemble frocing connection and application work correctly. A couple items to consider : (1) if ENSEMBLE_FORCING: YES & LDAS_COUPLING : 2 , the block that carries out ensemble forcing averaging in lenkf.j should be skipped. (2) if FIRST_ENS_ID : 1 , RESTART: 1 should consider the possibility that restart files could from a previous run with first-ens-id 0.

Sara, 1) Can we give a value 3 to LDAS_COUPLING when ENSEMBLE_FORCING is Yes? 2) I have talked to Rolf about this. We can either generate the ensembles from scratch or change the 0th member to the last one to match ADAS.

[gmao-rreichle]

@saraqzhang:

(2) if FIRST_ENS_ID : 1 , RESTART: 1 should consider the possibility that restart files could from a previous run with first-ens-id 0.

Sara, [..] 2) I have talked to Rolf about this. We can either generate the ensembles from scratch or change the 0th member to the last one to match ADAS.

I think it would be too confusing to have a translation from FIRST_ENS_ID=0 (in the restart) to FIRST_ENS_ID=1 (in the simulation) built into the code for RESTART=1. Most of the time, the restart files should already be produced with FIRST_ENS_ID=1 (ie, have matching numbering). If we ever need to restart from a run that has FIRST_ENS_ID=0, we could manually create a link (or copy) from member 0 to member NENS.

[saraqzhang]

@weiyuan-jiang @gmao-rreichle
for 1) Can we give a value 3 to LDAS_COUPLING when ENSEMBLE_FORCING is Yes?
we can make if ( LDAS_COUPLING =2 & ENSEMBLE_FORCING No ) then into the block.
for 2) I agree what Rolf suggests. ( keep first-ens-id =1 ) .

[weiyuan-jiang]

Probably it is simpler to remove parameter ENSEMBLE_FORCING and overload the meaning LDAS_COUPLING=3

[saraqzhang]

@weiyuan-jiang it is fine with overloading the meaning LDAS_COUPLING=3, if there will not be a case in the future that ldas would run offline ( not coupling) with ensemble forcing.

[gmao-rreichle]

it is fine with overloading the meaning LDAS_COUPLING=3, if there will not be a case in the future that ldas would run offline ( not coupling) with ensemble forcing.

Sara has a good point. We'll probably want to run GEOSldas offline with an ensemble of forcing fields, so ENSEMBLE_FORCING is probably needed as a separate resource variable. But Weiyuan also has a good point. The evolution of the coupling in the LADAS means that we need to define a suite of LDAS_COUPLING options, some of which would trigger ENSEMBLE_FORCING=YES (although probably only in the coarse-resolution LDAS that is coupled to the ensemble, and not in the fine-resolution LDAS that is coupled to the deterministic ADAS). We'll find out more as we keep developing.

[weiyuan-jiang]

@saraqzhang , I have added "ENSEMBLE_FORCING" to lenk.j according to your suggestion. Would you please reviee and test it? After that we can ask @gmao-rreichle to review and merge it to our develop branch.

[saraqzhang]

@weiyuan-jiang I pulled the addition , tested setup and run. all passed.