Use invariant.nc as input to replace init.nc, when generating lbc files

[chunhuazhou]

When generating MPAS lbc files, the default input file is init.nc. However if the lbc tasks are run at different cycles from the ic task cycles, we don't want to add an unnecessary ic tasks. To skip this unnecessary ic task, we can use invariant.nc as input when generating the lbc files. The only change is to the streams.init_atmosphere:
from

<streams>
<immutable_stream name="input"
                  type="input"
                  filename_template="init.nc"
                  input_interval="initial_only" />

to

<streams>
<immutable_stream name="input"
                  type="input"
                  filename_template="invariant.nc"
                  input_interval="initial_only" />

I compared the lbc files generated using init.nc vs invariant.nc and they are identical.

Chunhua.Zhou@fe6 rrfsdet_lbc_g01_12.invariant$ for files in lbc.2025-02-1* ; do nccmp -ldfsS $files ../rrfsdet_lbc_g01_12/$files ; done
Files "lbc.2025-02-18_12.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_12.00.00.nc" are identical.
Files "lbc.2025-02-18_13.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_13.00.00.nc" are identical.
Files "lbc.2025-02-18_14.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_14.00.00.nc" are identical.
Files "lbc.2025-02-18_15.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_15.00.00.nc" are identical.
Files "lbc.2025-02-18_16.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_16.00.00.nc" are identical.
Files "lbc.2025-02-18_17.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_17.00.00.nc" are identical.
Files "lbc.2025-02-18_18.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_18.00.00.nc" are identical.
Files "lbc.2025-02-18_19.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_19.00.00.nc" are identical.
Files "lbc.2025-02-18_20.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_20.00.00.nc" are identical.
Files "lbc.2025-02-18_21.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_21.00.00.nc" are identical.
Files "lbc.2025-02-18_22.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_22.00.00.nc" are identical.
Files "lbc.2025-02-18_23.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-18_23.00.00.nc" are identical.
Files "lbc.2025-02-19_00.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-19_00.00.00.nc" are identical.
Files "lbc.2025-02-19_01.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-19_01.00.00.nc" are identical.
Files "lbc.2025-02-19_02.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-19_02.00.00.nc" are identical.
Files "lbc.2025-02-19_03.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-19_03.00.00.nc" are identical.
Files "lbc.2025-02-19_04.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-19_04.00.00.nc" are identical.
Files "lbc.2025-02-19_05.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-19_05.00.00.nc" are identical.
Files "lbc.2025-02-19_06.00.00.nc" and "../rrfsdet_lbc_g01_12/lbc.2025-02-19_06.00.00.nc" are identical.

[guoqing-noaa]

@chunhuazhou
Thanks a lot for the test! This is great! It will save us lots of extra work.

[chunhuazhou]

To add more information here, I also compared the lbc metadata from the two methods:

$ nccmp -lmgfsS lbc.2025-02-18_13.00.00.nc ../rrfsdet_lbc_g01_12/lbc.2025-02-18_13.00.00.nc 
DIFFER : VALUES OF GLOBAL ATTRIBUTE : parent_id : a2wx4ma3h6
oz6cdkzg6g
 <> tbbqzcjezs
oz6cdkzg6g

DIFFER : VALUES OF GLOBAL ATTRIBUTE : file_id : 16b8glan4r <> lh0m558dyk

[guoqing-noaa]

addressed by PR #647

[chunhuazhou]

Re-open this issue.
The above tests are based on RAP data, that produced identical lbc files when using init.nc vs invariant.nc as input.
When generating the lbc files from GFS data, the lbc files are different when using init.nc vs invariant.nc in the rrfs-workflow that was generated in the RAP initialized run. And this caused the model to crash immediately after forecast hour 0.

[chunhuazhou]

To confirm the above hypothesis that the model data source for the lbc files and the invariant.nc should be consistent, another test for 00Z 05/27/2024, over conus-12km, was conducted using the latest rrfs-workflow (hash 7e106bd). For this case, the the lbc files and its input stream invariant.nc are both generated using GFS as data source. And the MPAS model completed the 12-hour forecast.
Tests have also been conducted for conus-3km, using RAP data. Same conclusion as the conus-12km tests. When the lbc files and its input stream invariant.nc are both using RAP data, the cold start forecast succeeded.

[chunhuazhou]

According to the tests, we might need to add a one-time task to generate invariant.nc within the workflow run, to ensure consistency between the invariant.nc and the external model source.

[chunhuazhou]

A new test done for conus-3km using GFS data. The default run, using the default invariant.nc from RAP, failed right after forecast hour 0.
In this new test, invariant.nc was generated within the workflow, then lbc files generated using the new invariant.nc, and then the forecast job ran fine using the updated lbc files.