Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix physgrid issues with PE layout and aqua restarts #3204

Merged
merged 18 commits into from
Nov 11, 2019
Merged

Fix physgrid issues with PE layout and aqua restarts #3204

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
720b327
Fix restart bug with physgrid on aquaplanet
whannah1 Sep 23, 2019
d7af49d
Fix more stuff
whannah1 Sep 23, 2019
d31d268
change phys_tmp to phis_tmp
whannah1 Sep 24, 2019
3d29819
add fv_physgrid
whannah1 Sep 24, 2019
d592db3
Add new pes defaults for ne4 + pg1,2,3
whannah1 Sep 24, 2019
f8e1e2c
simplify aqua topo bug fix
whannah1 Sep 24, 2019
b504569
Fix PE-layout grid matching issue for physgrid
whannah1 Sep 25, 2019
90d5a1b
Fix topo mapping
whannah1 Sep 25, 2019
4f39178
Cosmetic update for consistency
whannah1 Sep 30, 2019
b3655b3
Add check to not set phis_tmp when not allocated
whannah1 Oct 1, 2019
66a5c3e
Another cosmetic update
whannah1 Oct 3, 2019
2b22458
Change se_fv_phys_remap_alg default to 1
whannah1 Oct 23, 2019
a0661d4
Add allocate and deallocate for fv_physgrid%topo
whannah1 Oct 31, 2019
30ae2b9
Merge commit '2b22458eb3984ed483690a1055c92a2497bb417a' into whannah/…
whannah1 Nov 1, 2019
807e994
ange physgrid option in homme control_mod
whannah1 Nov 1, 2019
1155037
Fix default setting for se_fv_phys_remap_alg
whannah1 Nov 1, 2019
aab7b44
Add deallocate statement for phis_tmp
whannah1 Nov 6, 2019
01ee915
adding melving settings for ne4np4pg2
oksanaguba Nov 8, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 113 additions & 32 deletions cime/config/e3sm/allactive/config_pesall.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne16np4">
<grid name="a%ne16np4_">
<mach name="any">
<pes compset="any" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -222,7 +222,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne240np4">
<grid name="a%ne240np4_">
<mach name="any">
<pes compset="any" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -259,7 +259,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne240np4">
<grid name="a%ne240np4_">
<mach name="any">
<pes compset="CAM.+CLM.+DOCN." pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -296,7 +296,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne120np4">
<grid name="a%ne120np4_">
<mach name="titan|stampede|bluewaters|edison|eos|cori-haswell">
<pes compset="any" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -440,7 +440,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne120np4">
<grid name="a%ne120np4_">
<mach name="any">
<pes compset="CAM.+CLM.+DOCN." pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -477,7 +477,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne120np4">
<grid name="a%ne120np4_">
<mach name="titan|stampede|bluewaters|edison|eos|cori-haswell">
<pes compset="CAM.+CLM.+DOCN." pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -514,7 +514,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne120np4">
<grid name="a%ne120np4_">
<mach name="titan|stampede|bluewaters">
<pes compset="CAM.+CLM.+CICE.+POP" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -551,7 +551,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="any">
<pes compset="any" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -588,7 +588,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="any">
<pes compset="CAM.+CLM.+DOCN." pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -625,7 +625,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="melvin">
<pes compset="any" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -662,7 +662,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="edison">
<pes compset="any" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -699,7 +699,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="cori-haswell">
<pes compset="any" pesize="L">
<comment>169 nodes, 19 sypd</comment>
Expand Down Expand Up @@ -802,7 +802,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="cori-knl">
<pes compset="any" pesize="L">
<comment>cori-knl, generic ne30, 85 nodes, 64x1</comment>
Expand Down Expand Up @@ -906,7 +906,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="eos">
<pes compset="any" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -943,7 +943,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="blues">
<pes compset="XATM|DATM.+CLM" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -980,7 +980,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="titan|stampede|bluewaters">
<pes compset="any" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -1017,7 +1017,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="titan|stampede|bluewaters">
<pes compset="CAM.+CLM.+DOCN." pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -2541,7 +2541,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne16np4">
<grid name="a%ne16np4_">
<mach name="edison|cori-haswell">
<pes compset="any" pesize="any">
<comment>none</comment>
Expand Down Expand Up @@ -2578,7 +2578,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne16np4">
<grid name="a%ne16np4_">
<mach name="cori-knl">
<pes compset="any" pesize="any">
<comment>cori-knl, 6 nodes, 64x4, sypd=2.93 (for F-compset)</comment>
Expand Down Expand Up @@ -2607,7 +2607,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne11np4">
<grid name="a%ne11np4_">
<mach name="cori-knl">
<pes compset="any" pesize="any">
<comment>6 nodes, 64x2, sypd=11.1 (for F-compset)</comment>
Expand Down Expand Up @@ -4552,7 +4552,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4_l%ne30np4_oi%ne30np4">
<grid name="a%ne30np4_l%ne30np4_oi%ne30np4_">
<mach name="anvil|bebop">
<pes compset="any" pesize="any">
<comment>ne30_ne30 grid on 40 nodes 36 ppn pure-MPI</comment>
Expand Down Expand Up @@ -4810,7 +4810,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="mira|cetus">
<pes compset="any" pesize="any">
<comment>ne30 grid on 128x16x4 PEs</comment>
Expand Down Expand Up @@ -6109,7 +6109,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="edison">
<pes compset="CAM.+CLM.+DOCN." pesize="any">
<comment>"113-node 12x4 F-compset sypd=10.8"</comment>
Expand Down Expand Up @@ -6148,7 +6148,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="edison">
<pes compset="CAM5.+CLM45.+MPASSI.+MPASO.+MOSART.+SGLC.+SWAV" pesize="any">
<comment>"133 node version gets 6 SYPD. This will be the default and M size"</comment>
Expand Down Expand Up @@ -6185,7 +6185,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="edison">
<pes compset="CAM5.+CLM45.+MPASSI.+MPASO.+MOSART.+SGLC.+SWAV" pesize="S">
<comment>"39 node version gets 2.1 SYPD."</comment>
Expand Down Expand Up @@ -6222,7 +6222,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="edison">
<pes compset="CAM5.+CLM45.+MPASSI.+MPASO.+MOSART.+SGLC.+SWAV" pesize="L">
<comment>"285 node version gets 11.5 SYPD"</comment>
Expand Down Expand Up @@ -6259,7 +6259,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="cori-haswell">
<pes compset="CAM5.+CLM45.+MPASSI.+MPASO.+MOSART.+SGLC.+SWAV" pesize="any">
<comment>"185 nodes, 32x1, ~5sypd (wmod185)"</comment>
Expand Down Expand Up @@ -6329,7 +6329,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne30np4">
<grid name="a%ne30np4_">
<mach name="cori-knl">
<pes compset=".*CAM5.+CLM45.+MPASSI.+MPASO.+MOSART.*" pesize="L">
<comment>"cori-knl ne30 coupled compest on 120 nodes, 64x1 (2 threads CPL/OCN/ICE), (kmod125) sypd=4.1"</comment>
Expand Down Expand Up @@ -6758,7 +6758,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne120np4">
<grid name="a%ne120np4_">
<mach name="mira">
<pes compset=".*CAM5.+CLM45.+MPASSI.+MPASO.+MOSART.*" pesize="S">
<comment>ne120 coupled-compset on 1024 nodes</comment>
Expand Down Expand Up @@ -8210,7 +8210,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne4np4">
<grid name="a%ne4np4_">
<mach name="cetus">
<pes compset="any" pesize="any">
<comment>any compset on ne4 grid</comment>
Expand Down Expand Up @@ -8247,7 +8247,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne4np4">
<grid name="a%ne4np4_">
<mach name="cori-haswell">
<pes compset="any" pesize="any">
<comment>3 nodes, any compset on ne4 grid</comment>
Expand All @@ -8274,7 +8274,7 @@
</pes>
</mach>
</grid>
<grid name="a%ne4np4">
<grid name="a%ne4np4_">
<mach name="cori-knl">
<pes compset=".*CAM5.+CLM45.+MPASSI.+MPASO.+MOSART.*" pesize="any">
<comment>"cori-knl ne4 coupled compest on 6 nodes, sypd=22.9"</comment>
Expand Down Expand Up @@ -8421,6 +8421,87 @@
</pes>
</mach>
</grid>
<grid name="a%ne4np4.pg1">
<mach name="any">
<pes compset="any" pesize="any">
<comment>any compset on ne4np4.pg1 grid</comment>
<ntasks>
<ntasks_atm>96</ntasks_atm>
<ntasks_lnd>96</ntasks_lnd>
<ntasks_rof>96</ntasks_rof>
<ntasks_ice>96</ntasks_ice>
<ntasks_ocn>96</ntasks_ocn>
<ntasks_cpl>96</ntasks_cpl>
<ntasks_glc>1</ntasks_glc>
<ntasks_wav>1</ntasks_wav>
</ntasks>
<nthrds>
<nthrds_atm>1</nthrds_atm>
<nthrds_lnd>1</nthrds_lnd>
<nthrds_rof>1</nthrds_rof>
<nthrds_ice>1</nthrds_ice>
<nthrds_ocn>1</nthrds_ocn>
<nthrds_glc>1</nthrds_glc>
<nthrds_wav>1</nthrds_wav>
<nthrds_cpl>1</nthrds_cpl>
</nthrds>
</pes>
</mach>
</grid>
<grid name="a%ne4np4.pg2">
<mach name="any">
<pes compset="any" pesize="any">
<comment>any compset on ne4np4.pg2 grid</comment>
<ntasks>
<ntasks_atm>96</ntasks_atm>
<ntasks_lnd>96</ntasks_lnd>
<ntasks_rof>96</ntasks_rof>
<ntasks_ice>96</ntasks_ice>
<ntasks_ocn>96</ntasks_ocn>
<ntasks_cpl>96</ntasks_cpl>
<ntasks_glc>1</ntasks_glc>
<ntasks_wav>1</ntasks_wav>
</ntasks>
<nthrds>
<nthrds_atm>1</nthrds_atm>
<nthrds_lnd>1</nthrds_lnd>
<nthrds_rof>1</nthrds_rof>
<nthrds_ice>1</nthrds_ice>
<nthrds_ocn>1</nthrds_ocn>
<nthrds_glc>1</nthrds_glc>
<nthrds_wav>1</nthrds_wav>
<nthrds_cpl>1</nthrds_cpl>
</nthrds>
</pes>
</mach>
</grid>
<grid name="a%ne4np4.pg3">
<mach name="any">
<pes compset="any" pesize="any">
<comment>any compset on ne4np4.pg3 grid</comment>
<ntasks>
<ntasks_atm>96</ntasks_atm>
<ntasks_lnd>96</ntasks_lnd>
<ntasks_rof>96</ntasks_rof>
<ntasks_ice>96</ntasks_ice>
<ntasks_ocn>96</ntasks_ocn>
<ntasks_cpl>96</ntasks_cpl>
<ntasks_glc>1</ntasks_glc>
<ntasks_wav>1</ntasks_wav>
</ntasks>
<nthrds>
<nthrds_atm>1</nthrds_atm>
<nthrds_lnd>1</nthrds_lnd>
<nthrds_rof>1</nthrds_rof>
<nthrds_ice>1</nthrds_ice>
<nthrds_ocn>1</nthrds_ocn>
<nthrds_glc>1</nthrds_glc>
<nthrds_wav>1</nthrds_wav>
<nthrds_cpl>1</nthrds_cpl>
</nthrds>
</pes>
</mach>
</grid>
<grid name=".*oi%oRRS30to10.*">
<mach name="theta">
<pes compset=".*MPASSI.+MPASO.+" pesize="any">
Expand Down
1 change: 1 addition & 0 deletions components/cam/src/dynamics/se/dyn_grid.F90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ module dyn_grid
real(kind=r8), allocatable :: lat(:,:) ! latitude
real(kind=r8), allocatable :: corner_lon(:,:,:) ! longitude
real(kind=r8), allocatable :: corner_lat(:,:,:) ! latitude
real(kind=r8), allocatable :: topo(:,:) ! topography data
end type fv_physgrid_struct
type(fv_physgrid_struct), public, pointer :: fv_physgrid(:) => null()

Expand Down
Loading