Fixes for C12 MOM6 #285

mathomp4 · 2022-01-04T16:27:46Z

This PR updates the setup scripts for the C12 MOM6 setup that @sanAkel and @yvikhlya have developed. It includes:

Fixes some csh logic for making the tile names
Remove setenv I_MPI_DAPL_UD enable. Intel MPI doesn't even recognize this flag anymore:
```
I_MPI_DAPL_UD variable has been removed from the product, its value is ignored
```
Remove the ROMIO environment variable for Intel MPI
Default C12 MOM6 as 1x6 atmos/3x2 ocean. Note that C12 data ocean will still be 2x12 by default

Removes the symlink and binarytile.x runs for tile_hist.data. Tests by @bena-nasa and myself seem to show that MAPL_Tripolar.nc coupled with the usual History grid_label:

geosgcm_ocn2d.template:  '%y4%m2%d2_%h2%n2z.nc4',
geosgcm_ocn2d.archive:   '%c/Y%y4',
geosgcm_ocn2d.mode:      'time-averaged',
geosgcm_ocn2d.format: 'CFIO',
geosgcm_ocn2d.frequency:  060000,
geosgcm_ocn2d.grid_label: PC360x181-DC,
geosgcm_ocn2d.fields:    'UW'    , 'MOM6', 'US'
                 'VW'    , 'MOM6', 'VS'
                 'TW'    , 'MOM6', 'TS'
                 'SW'    , 'MOM6', 'SS'
                    ::

works. And it seems to make usable data:

NOTE1: Keeping this draft until I make sure I don't break C90 MOM6 as well as make sure all works on my laptop

NOTE2: At this point I think you can only run C12 MOM6 with my TinyBCs. I believe @yvikhlya needs to copy some files to a new /discover/nobackup/yvikhlia/coupled/Forcings/MOM6/CF0012x6C_TM0072xTM0036 directory that I can provide.

sanAkel · 2022-02-02T02:50:22Z

NOTE2: At this point I think you can only run C12 MOM6 with my TinyBCs. I believe @yvikhlya needs to copy some files to a new /discover/nobackup/yvikhlia/coupled/Forcings/MOM6/CF0012x6C_TM0072xTM0036 directory that I can provide.

I think this has happened.
@mathomp4 would you please check if ALL still works at your end?

mathomp4 · 2022-02-02T17:41:50Z

@sanAkel I still do not see a CF0012x6C_TM0072xTM0036 directory in:

/discover/nobackup/yvikhlia/coupled/Forcings/MOM6/

I think until that's done, C12 MOM6 will only work with TinyBCs.

yvikhlya · 2022-02-02T18:01:21Z

@sanAkel I still do not see a CF0012x6C_TM0072xTM0036 directory in:
/discover/nobackup/yvikhlia/coupled/Forcings/MOM6/
I think until that's done, C12 MOM6 will only work with TinyBCs.

All coupled bc files are moved to /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs. I keep the old files at their old locations for now because people are running old experiments with them.

sanAkel · 2022-02-02T18:12:56Z

@sanAkel I still do not see a CF0012x6C_TM0072xTM0036 directory in:
/discover/nobackup/yvikhlia/coupled/Forcings/MOM6/
I think until that's done, C12 MOM6 will only work with TinyBCs.

@mathomp4 I am not sure when and how @yvikhlya is handling all these ... let's please proceed with TinyBCs can @sdrabenh work with TinyBCs ?

sdrabenh · 2022-02-02T20:25:15Z

can @sdrabenh work with TinyBCs ?

I am not familiar with running anything at C12. Whatever you guys decide is fine.

sanAkel · 2022-02-02T20:35:19Z

can @sdrabenh work with TinyBCs ?

I am not familiar with running anything at C12. Whatever you guys decide is fine.

@sdrabenh you always ease the pain! 😄

mathomp4 · 2022-02-02T20:50:30Z

@sanAkel I still do not see a CF0012x6C_TM0072xTM0036 directory in:
/discover/nobackup/yvikhlia/coupled/Forcings/MOM6/
I think until that's done, C12 MOM6 will only work with TinyBCs.
All coupled bc files are moved to /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs. I keep the old files at their old locations for now because people are running old experiments with them.

@yvikhlya The problem is that I don't think:

/discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM6/CF0012x6C_TM0072xTM0036

has all the needed files. For example, when I try to run with that as the GRIDDIR I have five broken symlinks in the scratch/ directory:

lrwxrwxrwx 1 mathomp4 g0620  126 Feb  2 15:45 SEAWIFS_KPAR_mon_clim.data -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM6/CF0012x6C_TM0072xTM0036/SEAWIFS_KPAR_mon_clim.72x36
lrwxrwxrwx 1 mathomp4 g0620  115 Feb  2 15:45 MAPL_Tripolar.nc -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM6/CF0012x6C_TM0072xTM0036/MAPL_Tripolar.nc
lrwxrwxrwx 1 mathomp4 g0620  112 Feb  2 15:45 vgrid.ascii -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM6/CF0012x6C_TM0072xTM0036/vgrid50.ascii
...
lrwxrwxrwx 1 mathomp4 g0620  116 Feb  2 15:45 kmt_cice.bin -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM6/CF0012x6C_TM0072xTM0036/cice/kmt_cice.bin
lrwxrwxrwx 1 mathomp4 g0620  117 Feb  2 15:45 grid_cice.bin -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus-NLv3/MOM6/CF0012x6C_TM0072xTM0036/cice/grid_cice.bin

These are in the TinyBCs set that @sanAkel provided me.

I then based the TinyBCs directory structure on that of:

/discover/nobackup/yvikhlia/coupled/Forcings/MOM6/CF0090x6C_TM0360xTM0210/

where there is a cice/ directory and the other three files.

mathomp4 · 2022-02-02T20:51:58Z

Note that this PR is still safe as long as anyone that wants to run C12 Newland MOM6, runs with my TinyBCs. It will still work fine for C90 OldLand MOM6.

sanAkel · 2022-02-02T21:00:30Z

Note that this PR is still safe as long as anyone that wants to run C12 Newland MOM6, runs with my TinyBCs. It will still work fine for C90 OldLand MOM6.

Thanks @mathomp4 Let me think about this a bit ...

yvikhlya · 2022-02-02T21:06:53Z

@mathomp4 I split ocean and agcm bcs, since ocean bcs do not depend on agcm resolution and land model version, there is no need to copy them under every configuration. There is no GRIDDIR anymore, new locations are

/discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs
/discover/nobackup/projects/gmao/ssd/aogcm/ocean_bcs

I updated gcm_run.j and gcm_setup to work with new locations on the feature/aogcm branch, but this is not merged with develop yet.

mathomp4 · 2022-02-02T21:10:50Z

@mathomp4 I split ocean and agcm bcs, since ocean bcs do not depend on agcm resolution and land model version, there is no need to copy them under every configuration. There is no GRIDDIR anymore, new locations are
/discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs
/discover/nobackup/projects/gmao/ssd/aogcm/ocean_bcs
I updated gcm_run.j and gcm_setup to work with new locations on the feature/aogcm branch, but this is not merged with develop yet.

@yvikhlya Ahh. Okay. If you don't mind, I can take a look at that branch and then merge the GRIDDIR to new dirs changes into my PR here?

ETA: Ooh. I also need to work with @bena-nasa to make sure we have all that data in our location at NAS!

yvikhlya · 2022-02-02T21:48:32Z

@yvikhlya Ahh. Okay. If you don't mind, I can take a look at that branch and then merge the GRIDDIR to new dirs changes into my PR here?

Oh yes, you are welcomed to do this. Instead of GRIDDIR, there are OBCS and ABCS variables now. Not sure if I did it the best way, so if you feel something needs to be tweaked, feel free to do this.

mathomp4 · 2022-02-03T21:40:01Z

I'm blocking this until I can get in the updates from @yvikhlya. Hopefully unblocking tomorrow

yvikhlya · 2022-02-03T22:27:08Z

I'm blocking this until I can get in the updates from @yvikhlya. Hopefully unblocking tomorrow

What updates exactly?

mathomp4 · 2022-02-04T13:12:14Z

I'm blocking this until I can get in the updates from @yvikhlya. Hopefully unblocking tomorrow

What updates exactly?

The replacement for GRIDDIR. I need to make sure I get the merge of changes over correctly. (since feature/aogcm still uses the >>><<< style instead of the @-style for options preferred by Ricardo).

yvikhlya · 2022-02-04T13:39:49Z

The replacement for GRIDDIR.

It is split into

/discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs
/discover/nobackup/projects/gmao/ssd/aogcm/ocean_bcs

Am I getting wrong something?

EDIT: Note, that these directories are not copied to NAS currently.
EDIT2: Matt, it looks like you get everything right in your commit. What I could be missing?

mathomp4 · 2022-02-04T15:40:14Z

The replacement for GRIDDIR.

It is split into
/discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs
/discover/nobackup/projects/gmao/ssd/aogcm/ocean_bcs
Am I getting wrong something?

EDIT: Note, that these directories are not copied to NAS currently. EDIT2: Matt, it looks like you get everything right in your commit. What I could be missing?

Yury,

I found some files are missing at NCCS. I pinged on Teams, but for history's sake:

Some updates are needed to your atmosphere_bcs directory for Icarus Old Land C90 TM360. The model is expecting files like visdf_90x540.dat but they aren't there. These are all broken symlinks:

visdf.dat -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus/MOM6/CF0090x6C_TM0360xTM0210/visdf_90x540.dat
nirdf.dat -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus/MOM6/CF0090x6C_TM0360xTM0210/nirdf_90x540.dat
vegdyn.data -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus/MOM6/CF0090x6C_TM0360xTM0210/vegdyn_90x540.dat
lai.data -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus/MOM6/CF0090x6C_TM0360xTM0210/lai_clim_90x540.data
green.data -> /discover/nobackup/projects/gmao/ssd/aogcm/atmosphere_bcs/Icarus/MOM6/CF0090x6C_TM0360xTM0210/green_clim_90x540.data

I think your directory only has them without the "90x540"

I think the NLv3 C12 ocean_bcs has some missing files. These two files are in my TinyBCs from Santha, but not in your directory:

/gpfsm/dnb44/mathomp4/TinyBCs-GitV10/ocean_bcs/MOM6/72x36/INPUT/grid_spec.nc
/gpfsm/dnb44/mathomp4/TinyBCs-GitV10/ocean_bcs/MOM6/72x36/INPUT/ocean_mosaic.nc

yvikhlya · 2022-02-04T16:31:57Z

@mathomp4 Could you check again? Let me know if anything is still missing.

mathomp4 · 2022-02-04T18:29:24Z

@mathomp4 Could you check again? Let me know if anything is still missing.

@yvikhlya I think we got it. It seems to work for me. I also fixed up TinyBCs to, I think, work with the new ABCSDIR and OBCSDIR. Though testing will find that out...

mathomp4 · 2022-02-04T18:35:11Z

Sigh. Or not. Dang it. I messed up something...

mathomp4 · 2022-02-04T18:38:25Z

@yvikhlya I think one more directory still missing. When I try to run C90 360x210 (which is my "usual" MOM6 run) I see these broken symlinks:

kmt_cice.bin -> /discover/nobackup/projects/gmao/ssd/aogcm/ocean_bcs/MOM6/360x210/cice/kmt_cice.bin
grid_cice.bin -> /discover/nobackup/projects/gmao/ssd/aogcm/ocean_bcs/MOM6/360x210/cice/grid_cice.bin

Indeed, the 360x210 case isn't in ocean_bcs:

$ ls -l /discover/nobackup/projects/gmao/ssd/aogcm/ocean_bcs/MOM6/
total 4
drwxr-xr-x 6 yvikhlia g0609 4096 Sep 16 12:12 1440x1080/
drwxr-xr-x 6 yvikhlia g0609 4096 Jan 31 13:01 1440x1080_newtopo/
drwxr-xr-x 6 yvikhlia g0609 4096 Oct 13 12:19 72x36/
-rw-r--r-- 1 yvikhlia g0609 3160 Jan 20 12:01 generate_topo.md

yvikhlya · 2022-02-04T19:44:17Z

@mathomp4 This is because MOM6 c90x360x210 does not work with new land BCs and I thought we are going to abandon it. But I can copy it to new location to use with old land BCs, hold on.

EDIT: Done.

mathomp4 · 2022-02-04T20:00:59Z

@mathomp4 This is because MOM6 c90x360x210 does not work with new land BCs and I thought we are going to abandon it. But I can copy it to new location to use with old land BCs, hold on.

EDIT: Done.

@yvikhlya Thanks. I'll retest as soon as I get nodes.

And, yes, it is sort of "abandoned", but I can only really run two cases now. C90 Oldland and C12 Newland. Probably good to keep the old girl going until I can transition to another "mid-resolution" case that you or @sanAkel can provide me. 😄

sanAkel · 2022-02-04T21:16:56Z

@mathomp4 This is because MOM6 c90x360x210 does not work with new land BCs and I thought we are going to abandon it. But I can copy it to new location to use with old land BCs, hold on.
EDIT: Done.

@yvikhlya Thanks. I'll retest as soon as I get nodes.

And, yes, it is sort of "abandoned", but I can only really run two cases now. C90 Oldland and C12 Newland. Probably good to keep the old girl going until I can transition to another "mid-resolution" case that you or @sanAkel can provide me. 😄

@mathomp4 There are a few things already queued/piled up. Building about a 1-deg ocean case is one of them, but not at the top right now. However, the 0.25-deg case is, which is (from a non oceanographer perspective) at the "high" end of resolution spectrum! Meanwhile, enjoy the C12 x 5-deg!

mathomp4 · 2022-02-07T17:22:08Z

Removing the blocker. It looks like all is well with this. C12 runs. @bena-nasa and I need to do some BCs work at NAS, but I think we are good.

mathomp4 · 2022-02-10T15:37:50Z

Closing in favor of #292

mathomp4 added 2 commits January 4, 2022 08:52

Fix up IMO/JMO for Coupled

342bc78

Use 1x6 for C12 MOM6

640af9f

mathomp4 added the 0 diff The changes in this pull request have verified to be zero-diff with the target branch. label Jan 4, 2022

mathomp4 self-assigned this Jan 4, 2022

mathomp4 added 2 commits January 4, 2022 11:32

Update the other setup scripts

b91fd15

Don't use tile_hist for MOM6

49c3014

mathomp4 marked this pull request as ready for review January 4, 2022 21:31

mathomp4 requested a review from a team as a code owner January 4, 2022 21:31

Remove unneeded I_MPI_DAPL_UD

d8b50eb

mathomp4 added the Contingent - DNA Do Not Approve (DNA). These changes are contingent on other PRs label Feb 3, 2022

Update based on feature/aogcm

c9d26fa

mathomp4 marked this pull request as draft February 3, 2022 21:47

yvikhlya previously approved these changes Feb 3, 2022

View reviewed changes

Fix missed chevron

218ebc1

mathomp4 dismissed yvikhlya’s stale review via 218ebc1 February 4, 2022 15:41

mathomp4 marked this pull request as ready for review February 4, 2022 18:29

mathomp4 removed the Contingent - DNA Do Not Approve (DNA). These changes are contingent on other PRs label Feb 4, 2022

mathomp4 requested a review from yvikhlya February 4, 2022 18:34

mathomp4 added the Contingent - DNA Do Not Approve (DNA). These changes are contingent on other PRs label Feb 4, 2022

mathomp4 removed the Contingent - DNA Do Not Approve (DNA). These changes are contingent on other PRs label Feb 7, 2022

yvikhlya approved these changes Feb 7, 2022

View reviewed changes

mathomp4 mentioned this pull request Feb 10, 2022

Omnibus Ocean and NAS Updates to setup #292

Merged

mathomp4 closed this Feb 10, 2022

mathomp4 deleted the feature/mathomp4/fix-app-for-c12-mom6 branch February 10, 2022 15:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes for C12 MOM6 #285

Fixes for C12 MOM6 #285

mathomp4 commented Jan 4, 2022 •

edited

Loading

sanAkel commented Feb 2, 2022

mathomp4 commented Feb 2, 2022

yvikhlya commented Feb 2, 2022

sanAkel commented Feb 2, 2022

sdrabenh commented Feb 2, 2022

sanAkel commented Feb 2, 2022

mathomp4 commented Feb 2, 2022

mathomp4 commented Feb 2, 2022

sanAkel commented Feb 2, 2022

yvikhlya commented Feb 2, 2022

mathomp4 commented Feb 2, 2022 •

edited

Loading

yvikhlya commented Feb 2, 2022

mathomp4 commented Feb 3, 2022

yvikhlya commented Feb 3, 2022

mathomp4 commented Feb 4, 2022

yvikhlya commented Feb 4, 2022 •

edited

Loading

mathomp4 commented Feb 4, 2022 •

edited

Loading

yvikhlya commented Feb 4, 2022

mathomp4 commented Feb 4, 2022

mathomp4 commented Feb 4, 2022

mathomp4 commented Feb 4, 2022

yvikhlya commented Feb 4, 2022 •

edited

Loading

mathomp4 commented Feb 4, 2022

sanAkel commented Feb 4, 2022

mathomp4 commented Feb 7, 2022

mathomp4 commented Feb 10, 2022

Fixes for C12 MOM6 #285

Fixes for C12 MOM6 #285

Conversation

mathomp4 commented Jan 4, 2022 • edited Loading

sanAkel commented Feb 2, 2022

mathomp4 commented Feb 2, 2022

yvikhlya commented Feb 2, 2022

sanAkel commented Feb 2, 2022

sdrabenh commented Feb 2, 2022

sanAkel commented Feb 2, 2022

mathomp4 commented Feb 2, 2022

mathomp4 commented Feb 2, 2022

sanAkel commented Feb 2, 2022

yvikhlya commented Feb 2, 2022

mathomp4 commented Feb 2, 2022 • edited Loading

yvikhlya commented Feb 2, 2022

mathomp4 commented Feb 3, 2022

yvikhlya commented Feb 3, 2022

mathomp4 commented Feb 4, 2022

yvikhlya commented Feb 4, 2022 • edited Loading

mathomp4 commented Feb 4, 2022 • edited Loading

yvikhlya commented Feb 4, 2022

mathomp4 commented Feb 4, 2022

mathomp4 commented Feb 4, 2022

mathomp4 commented Feb 4, 2022

yvikhlya commented Feb 4, 2022 • edited Loading

mathomp4 commented Feb 4, 2022

sanAkel commented Feb 4, 2022

mathomp4 commented Feb 7, 2022

mathomp4 commented Feb 10, 2022

mathomp4 commented Jan 4, 2022 •

edited

Loading

mathomp4 commented Feb 2, 2022 •

edited

Loading

yvikhlya commented Feb 4, 2022 •

edited

Loading

mathomp4 commented Feb 4, 2022 •

edited

Loading

yvikhlya commented Feb 4, 2022 •

edited

Loading