Add member ratminc files to glopara ICSDIR for C96C48_ufs_hybatmDA CI

[RussTreadon-NOAA]

What is wrong?

While testing g-w PR #2534 is was discovered that the recentered analysis increment files for mem001 and mem002 are missing from the g-w maintained ICSDIR for C96C48_ufs_hybatmDA.

What should have happened?

Directory $ICSDIR/C96C48/enkfgdas.20240223/ should contain 18/mem00*/analysis/atmos/enkfgdas.t18z.ratminc.nc for mem001 and mem002.

What machines are impacted?

All or N/A

Steps to reproduce

1. clone g-w develop in user specified HOMEgfs
2. set machine specific ICSDIR_ROOT based config files in $HOMEgfs/ci/platforms
3. $ICSDIR_ROOT/C96C48/enkfgdas.20240223

Only directory 12 will be listed. We also need the 18 directory with the mem001 and mem002 ratminc.nc files.

Additional information

The ratminc.nc files are available on Hera in /scratch1/NCEPDEV/stmp2/Russ.Treadon/ICS/C96C48/enkfgdas.20240223/18

Do you have a proposed solution?

Do the following to get the ratminc.nc files into the g-w ICSDIR_ROOT

1. log into Hera role.glopara
2. set ICSDIR_ROOT=/scratch1/NCEPDEV/global/glopara/data/ICSDIR
3. cd /scratch1/NCEPDEV/stmp2/Russ.Treadon/ICS/C96C48/enkfgdas.20240223
4. rsync -av --progress 18 $ICSDIR_ROOT/C96C48/enkfgdas.20240223/
5. rsync Hera ICSDIR_ROOT/C96C48/enkfgdas.20240223/18 to other supported machines

[WalterKolczynski-NOAA]

Copied for Hera and MSU. @KateFriedman-NOAA would you please do WCOSS, I seem to be having password issues.

[RussTreadon-NOAA]

Thank you @WalterKolczynski-NOAA for populating the enkfgdas.20240223/18 directory on Hera and MSU.

Unfortunately, setup_expt.py did not copy the enkfgdas.20240223/18 directory to COMROOT. A check of workflow/setup_expt.py shows that it only links increments for the warm start deterministic forecast.

    # First 1/2 cycle also needs a atmos increment if doing warm start
    if inputs.start in ['warm']:
        for ftype in ['atmi003.nc', 'atminc.nc', 'atmi009.nc']:
            fname = f'{inputs.cdump}.t{idatestr[8:]}z.{ftype}'
            src_file = os.path.join(src_dir, fname)
            if os.path.exists(src_file):
                os.symlink(src_file, os.path.join(dst_dir, fname))

    return

Shouldn't the warm start block include if inputs.nens > 0 and link ensemble increments when setup_expt.py specifies ensemble members?

[RussTreadon-NOAA]

A modified copy of setup_expt.py has been created on Hera in /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/workflow. The revised script copies the 18Z enkfgdas ratminc files to the appropriate COMROOT directory.

--- a/workflow/setup_expt.py
+++ b/workflow/setup_expt.py
@@ -231,7 +231,19 @@ def fill_ROTDIR_cycled(host, inputs):
             src_file = os.path.join(src_dir, fname)
             if os.path.exists(src_file):
                 os.symlink(src_file, os.path.join(dst_dir, fname))
-
+        if inputs.nens > 0:
+            current_cycle_dir = f'enkf{inputs.cdump}.{idatestr[:8]}/{idatestr[8:]}'
+            for ii in range(1, inputs.nens + 1):
+                memdir = f'mem{ii:03d}'
+                src_dir = os.path.join(inputs.icsdir, current_cycle_dir, memdir, src_atm_anl_dir)
+                dst_dir = os.path.join(rotdir, current_cycle_dir, memdir, dst_atm_anl_dir)
+                makedirs_if_missing(dst_dir)
+                for ftype in ['ratmi003.nc', 'ratminc.nc', 'ratmi009.nc']:
+                    fname = f'enkf{inputs.cdump}.t{idatestr[8:]}z.{ftype}'
+                    src_file = os.path.join(src_dir, fname)
+                    if os.path.exists(src_file):
+                        os.symlink(src_file, os.path.join(dst_dir, fname))
+
     return

[RussTreadon-NOAA]

@KateFriedman-NOAA , @WalterKolczynski-NOAA encountered problems copying the enkfgdas.20240223/18 directory to the WCOSS2 ICSDIR.

I rsync'd /scratch1/NCEPDEV/global/glopara/data/ICSDIR/C96C48/enkfgdas.20240223/18 to Cactus /lfs/h2/emc/ptmp/russ.treadon.

Would you please

1. log into Cactus as emc.global
2. cd /lfs/h2/emc/ptmp/russ.treadon
3. rsync -av --progress 18 /lfs/h2/emc/global/noscrub/emc.global/data/ICSDIR/C96C48/enkfgdas.20240223/
4. rsync the Cactus /lfs/h2/emc/global/noscrub/emc.global/data/ICSDIR/C96C48/enkfgdas.20240223/18 to Dogwood.

[RussTreadon-NOAA]

@KateFriedman-NOAA and @WalterKolczynski-NOAA : I logged onto Cactus and see that I can still sudo into emc.global. If neither of you mind, I can rsync the 18/ directory into the g-w ICSDIR. Let me know if this is OK with you.

[WalterKolczynski-NOAA]

@RussTreadon-NOAA Go ahead

[RussTreadon-NOAA]

Thanks @WalterKolczynski-NOAA . Directory 18/ copied to Cactus and Dogwood /lfs/h2/emc/global/noscrub/emc.global/data/ICSDIR/C96C48/enkfgdas.20240223.

[RussTreadon-NOAA]

@WalterKolczynski-NOAA: Any other g-w supported machines on which we need the ratminc files? I only use Hera, Orion (Hercules), and WCOSS2.

If not, feel free to close this issue.

[KateFriedman-NOAA]

Thanks for copying those ICs to WCOSS2 @RussTreadon-NOAA ! Sorry I didn't reply, I was out on sick leave. Appreciate you doing the copy.

[RussTreadon-NOAA]

Hope you are feeling better @KateFriedman-NOAA . We can close this issue unless there are other machines to which we need to rsync the ratminc files.

[KateFriedman-NOAA]

Thanks @RussTreadon-NOAA , I am feeling better thankfully! :)

I have also synced the update to the ICSDIR set on Jet and AWS. I will now close this as complete.