Can not setup experiment through JEDI-T2 feature/gdas-validation branch

[emilyhcliu]

@CoryMartin-NOAA and @RussTreadon-NOAA
I tried to clone, build, and setup experiment from the JEDI-T2O feature/gdas-validation branch and ran into issue in the setup step. The experiment xml file can not be generated.

We ran into the same failure before ( .... can not find magic string) and then it was resolved before the code sprint. I have been setting up several new experiments using the following command without any problem.

Looks like something is broken between the ./setup_workflow.sh and the global-workflow/workflow/wxflow/.....
I updated the global-workflow (git pull) from the develop so it is the current develop.
(/work2/noaa/da/eliu/gdas-validation/global-workflow)

./setup_workflow.sh -s

The error message is:

sourcing config.prepatmiodaobs
sourcing config.atmanlinit
sourcing config.atmanlrun
sourcing config.atmanlfinal
sourcing config.sfcanl
sourcing config.analcalc
sourcing config.fcst
sourcing config.post
sourcing config.arch
sourcing config.cleanup
sourcing config.fit2obs
sourcing config.verfozn
sourcing config.verfrad
sourcing config.vminmon
sourcing config.tracker
Traceback (most recent call last):
  File "./setup_xml.py", line 79, in <module>
    main()
  File "./setup_xml.py", line 70, in main
    app_config = app_config_factory.create(f'{net}_{mode}', cfg)
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/workflow/wxflow/factory.py", line 72, in create
    return self._builders[key](*args, **kwargs)
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/workflow/applications/applications.py", line 22, in __call__
    obj._init_finalize(*args, **kwargs)
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/workflow/applications/applications.py", line 86, in _init_finalize
    self.configs = self._source_configs(conf)
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/workflow/applications/applications.py", line 153, in _source_configs
    configs[config] = conf.parse_config(files)
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/workflow/wxflow/configuration.py", line 82, in parse_config
    return cast_strdict_as_dtypedict(self._get_script_env(files))
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/workflow/wxflow/configuration.py", line 98, in _get_script_env
    and_script_env = cls._get_shell_env(scripts)
  File "/work2/noaa/da/eliu/gdas-validation/global-workflow/workflow/wxflow/configuration.py", line 118, in _get_shell_env
    'at least one script failed: ' + repr(out))
wxflow.configuration.ShellScriptException: Cannot find magic string; at least one script failed: 'BEGIN: config.base\nBEGIN: config.com\nEND: config.base\nBEGIN: config.tracker\nBEGIN: config.resources\nInvalid step = tracker, ABORT!\n': error processing/work2/noaa/da/eliu/gdas-validation/expdir/gdas_eval_satwind_JEDI/config.base /work2/noaa/da/eliu/gdas-validation/expdir/gdas_eval_satwind_JEDI/config.tracker
sed: can't read /work2/noaa/da/eliu/gdas-validation/expdir/gdas_eval_satwind_GSI/gdas_eval_satwind_GSI.xml: No such file or directory
sed: can't read /work2/noaa/da/eliu/gdas-validation/expdir/gdas_eval_satwind_JEDI/gdas_eval_satwind_JEDI.xml: No such file or directory

[RussTreadon-NOAA]

@emilyhcliu , as the error message indicates

`wxflow.configuration.ShellScriptException: Cannot find magic string; at least one script failed:
BEGIN: config.base\n
BEGIN: config.com\n
END: config.base\n
BEGIN: config.tracker\n
BEGIN: config.resources\n
Invalid step = tracker, ABORT!\n
': error processing/work2/noaa/da/eliu/gdas-validation/expdir/gdas_eval_satwind_JEDI/config.base /work2/noaa/da/eliu/gdas-validation/expdir/gdas_eval_satwind_JEDI/config.tracker`

file config.resources is outdated. It does not contain resource settings for the tracker job.

The g-w team has been separating vrfy into separate jobs. g-w PR #2058 moved the tracker outside of vrfy. We need to update config.resources in JEDI-T2O feature/gdas-validation to keep in step with recent g-w changes.

An updated config.resources has been committed to feature/gdas-validation. Done at 082dc4f

[emilyhcliu]

@emilyhcliu , as the error message indicates

`wxflow.configuration.ShellScriptException: Cannot find magic string; at least one script failed:
BEGIN: config.base\n
BEGIN: config.com\n
END: config.base\n
BEGIN: config.tracker\n
BEGIN: config.resources\n
Invalid step = tracker, ABORT!\n
': error processing/work2/noaa/da/eliu/gdas-validation/expdir/gdas_eval_satwind_JEDI/config.base /work2/noaa/da/eliu/gdas-validation/expdir/gdas_eval_satwind_JEDI/config.tracker`

file config.resources is outdated. It does not contain resource settings for the tracker job.

The g-w team has been separating vrfy into separate jobs. g-w PR #2058 moved the tracker outside of vrfy. We need to update config.resources in JEDI-T2O feature/gdas-validation to keep in step with recent g-w changes.

An updated config.resources has been committed to feature/gdas-validation. Done at 082dc4f

@RussTreadon-NOAA Thanks for your help! Now the workflow setup step works well.