Modify the J-job and Ex-scripts to meet NCO implementation standards for AQMv7

[JianpingHuang-NOAA]

The J-job and Ex-scripts do not meet NCO implementation standards for AQMv7

Here is the list of suggestions

1. The ush scripts should not be called in the beginning,
2. It should do all the variable and paths definition before any script call, also, it is better to call ex-scripts, then in ex-scripts call ush script
3. Need to define and mkdir $DATA before everything else, then turn on the "set -x", call setpdy, etc...
4. Please follow the other production job scripts.

Examples for AMQv6
/lfs/h1/ops/prod/packages/aqm.v6.1.7/jobs
/lfs/h1/ops/prod/packages/aqm.v6.1.7/scripts

[JianpingHuang-NOAA]

Here is the link of the implementation standards from NCO

https://www.nco.ncep.noaa.gov/idsb/implementation_standards/ImplementationStandards.v11.0.0.pdf

[chan-hoo]

@JianpingHuang-NOAA, all the above descriptions are incorrect:
(1) Compared to the old system, the 'ush' directory contains many utilities in the current UFS SRW App. The app just reads (activates) these utilities like function calls when the scripts start running because these utility functions are used in J-job and ex-scripts.
(2) I think Lin misunderstands the structure of the UFS SRW App. In the current UFS SRW App, The variables are set when the workflow is generated (python3 generate_FV3LAM_wflow.py) and then they are loaded at the very beginning of the J-job script.

. $USHdir/source_util_funcs.sh
source_config_for_task "cpl_aqm_parm|task_nexus_gfs_sfc" ${GLOBAL_VAR_DEFNS_FP}
. $USHdir/job_preamble.sh "TRUE"

You talked about the diagram in pp.4 of the NCO standards in the meeting (job card => J-job => ex-script). In the current system, the `job card' is the rocoto XML file like:

  <task name="&TN_NEXUS_GFS_SFC;" cycledefs="forecast" maxtries="2">
    &RSRV_DEFAULT;
    <command>&LOAD_MODULES_RUN_TASK_FP; "&TN_NEXUS_GFS_SFC;" "&JOBSdir;/JREGIONAL_NEXUS_GFS_SFC"</command>
    <nodes>1:ppn=1</nodes>
    <walltime>00:30:00</walltime>

As you can see, in the job card (xml file), it loads modules and then runs the J-job (JREGIONAL_NEUXS_GFS_SFC). The ex-script runs in the J-job script. All the python scripts and utility scripts, which are called in the ex-scripts, are located in the ush directory. This means that the current UFS SRW App DOES meet the NCO standards.

(3) The $DATA dir is defined in 'job_preamble.sh' at the very beginning in the 'J-job' script:

. $USHdir/source_util_funcs.sh
source_config_for_task "cpl_aqm_parm|task_nexus_gfs_sfc" ${GLOBAL_VAR_DEFNS_FP}
. $USHdir/job_preamble.sh "TRUE"

Another call for $DATA can be found in the J-job scripts. However, it is only for the community mode (NOT nco mode).

Regarding the set -x issue, the UFS SRW App provides more powerful debugging option (DEBUG). I talked to Matt about this. If NCO doesn't approve this option, I'll add "set -x" to the scripts later.

(4) As I mentioned above, this is a totally new system. Following the old ones is to degrade the capabilities of the current system. Moreover, in my opinion, the current UFS SRW App follows the NCO standards mostly all.

[HaixiaLiu-NOAA]

@rmontuoro here is the github issue opened to address meeting the NCO script standards.

[arunchawla-NOAA]

We need to know

1. Which jobs/scripts do not meet standards
2. Why they do not meet standards

[JianpingHuang-NOAA]

1. Standard Environment Variables for coming-in model, like GFS: Where is the ${COMINgfs} defined for the J-jobs, JREGIONAL_GET_EXTRN_MDL_FILES or JREGIONAL_MAKE_ICS or JREGIONAL_MAKE_LBCS ?
2. $COMIN and $COMOUT for bias correction training data: for the ex-scripts, exregional_bias_correction_o3.sh and exregional_bias_correction_pm25.sh, We use "${AQM_AIRNOW_HIST_DIR}" as the path of bias-correction training data files. We need to define it as $COMINbicor when we use existing data files, and define it as "COMOUTbicor" when we save the training data for future use. Please the operational J-job scripts at /lfs/h1/ops/prod/packages/aqm.v6.1.7/jobs
3. The path for real-time fire emission data files. $AQM_FIRE_DIR is used by exregional_fire_emission.sh, we should use $DCOMINfire, something like that.
4. compath.py utility: It is only seen in the JREGIONAL_AQM_LBCS and JREGIONAL_NEXUS_GFS_SFC, but not for other jobs for defining $COMIN and $COMOUT.

[chan-hoo]

@arunchawla-NOAA, The four items on the above new list are not the structural issues but minor change requests. I'll address them one by one.

[arunchawla-NOAA]

OK thank you. Can we open individual issues for each of them and close this one? I am thinking since these are connected to workflow then the issues need to be opened in the SRW app

[chan-hoo]

@JianpingHuang-NOAA, please open a separate issue for each to the UFS SRW App as Arun asked for. To avoid any confusion by other reviewers/developer of the 'develop' branch, please add the prefix [production/AQM.v7] to the title of each issue.

[JianpingHuang-NOAA]

@chan-hoo @arunchawla-NOAA I will do that after I test the updated SRW workflow with the BC code/scripts. Let me know when the updated SRW rocoto WK is ready for test.

[chan-hoo]

@JianpingHuang-NOAA, @KaiWang-NOAA, regarding Issue No.3, what is the full path of "$DCOMINfire" on wcoss2? Where can I find the fire emission data for NRT on wcoss2?

[JianpingHuang-NOAA]

@chan-hoo
The fire emission files (e.g., GBBEPx) are available at :/lfs/h1/ops/prod/dcom/$yyyymmdd/firewx to support operational model, AQMv6.

We expect that dataflow will put the RAVE data files at similar location like /lfs/h1/ops/prod/dcom/$yyyymmdd/fire_rave (or rave).

[JianpingHuang-NOAA]

@chan-hoo For the current rocoto workflow, we use ${yyyymmdd} rather than a random number as as job_id for the forecast job. This has caused some troubles for me when I run multiple scenario jobs for the same day and same cycle. Can you switch back to using a random number as a job_ID?

Thanks,

[JianpingHuang-NOAA]

@chan-hoo

Please see the issues opened with UFS-SRW Apps

ufs-community/ufs-srweather-app#747
ufs-community/ufs-srweather-app#748
ufs-community/ufs-srweather-app#749
ufs-community/ufs-srweather-app#750

[chan-hoo]

@chan-hoo For the current rocoto workflow, we use ${yyyymmdd} rather than a random number as as job_id for the forecast job. This has caused some troubles for me when I run multiple scenario jobs for the same day and same cycle. Can you switch back to using a random number as a job_ID?

Thanks,

@JianpingHuang-NOAA, I've added "{workflow_id}" to the forecast job_id: ufs-community/ufs-srweather-app@d648bf9

[bbakernoaa]

is this finished?

[JianpingHuang-NOAA]

Yes, thanks !