Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update XML defaults for theta-l dycore (V2) #3368

Merged
merged 8 commits into from
Jan 9, 2020

Conversation

mt5555
Copy link
Contributor

@mt5555 mt5555 commented Dec 29, 2019

Add XML defaults for theta-l dycore
Remove unused/obsolete dycore variables from XML defaults
Add code to allow namelist defaults to be set based on value of cam_target
Removed se_ftype=2 from all use cases (and made this a default)
Support new timestep interface (se_tstep, dt_tracer_factor, dt_remap_factor) for theta-l dycore
Update all theta "testmods" test cases to preserve original settings

[BFB] with nlfails.

@mt5555
Copy link
Contributor Author

mt5555 commented Dec 29, 2019

this touches a lot of code, so I will run the full acme_integration on sandiatoss3 to verify BFB.

@oksanaguba , can you configure an ne30 and ne30pg2 F compset and see if the dycore settings are what you would use for the upcoming v2 watercycle configuration?

<nu_top dyn_target="theta-l" hgrid="ne240np4"> 4.3e4 </nu_top>
<nu_top dyn_target="theta-l" hgrid="ne256np4"> 4e4 </nu_top>
<nu_top dyn_target="theta-l" hgrid="ne512np4"> 2e4 </nu_top>
<nu_top dyn_target="theta-l" hgrid="ne1024np4"> 1e4 </nu_top>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is if fair to say these nu_top coefs are still wip?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, these are the ones that work with hsv_tom=1. since we dont know the correct value for high-res, i thought we should start with the most efficient.

<hypervis_subcycle hgrid="ne0np4_svalbard_x8v1_lowcon" dyn_target="preqx" > 8 </hypervis_subcycle>
<hypervis_subcycle hgrid="ne0np4_sooberingoa_x4x8v1_lowcon" dyn_target="preqx" > 8 </hypervis_subcycle>
<hypervis_subcycle hgrid="ne0np4_enax4v1" dyn_target="preqx">7</hypervis_subcycle>
<hypervis_subcycle hgrid="ne0np4_twpx4v1" dyn_target="preqx"> 7 </hypervis_subcycle>
Copy link
Contributor

@oksanaguba oksanaguba Dec 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a lot of changes for resolutions that are not tested nightly, like ne120 and RRM with preqx dycore. How would we know settings for them are not modified?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for all grid specific settings, I added a dyn_target="preqx" clause, so they will all be preserved for the preqx dycore. The the grid independent settings, the integration suite should be a good test of if any preqx settings are changed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, some changes can be read like that, that preqx dycore was added, but some other changes are harder to parse.


<entry id="nu_s" type="real" category="se"
group="ctl_nl" valid_values="" >
Hyperviscosity applied to dynamics scalars [m^4/s].
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... applied to temperature/pot. temperature? dp is a scalar, too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this need to be changed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, thanks

@oksanaguba
Copy link
Contributor

oksanaguba commented Dec 30, 2019

Ran atm_integration against today's master on anvil. All is ok, except for 1 test, SMS_D_Ln5.ne4_ne4.FC5AV1C-L.anvil_intel.cam-cosplite, stack is below:

(base) [onguba@blueslogin2 as]$ ./cs.status.20191230_143327_y16ero | grep '(Overall:' 
  ERP_Ln9.ne4_ne4.F-EAMv1-AQP1.anvil_intel (Overall: NLFAIL) details:
  ERS_Ld5.ne4_ne4.FC5AV1C-L.anvil_intel.cam-gust_param (Overall: NLFAIL) details:
  ERS_Ld5.ne4_ne4.FC5AV1C-L.anvil_intel.cam-rrtmgp (Overall: NLFAIL) details:
  ERS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetahy_ftype2 (Overall: NLFAIL) details:
  ERS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetanh_ftype2 (Overall: NLFAIL) details:
  PEM_Ln5.ne4_ne4.FC5AV1C-L.anvil_intel (Overall: NLFAIL) details:
  PET_Ln5.ne4_ne4.FC5AV1C-L.anvil_intel.allactive-mach-pet (Overall: NLFAIL) details:
  REP_Ln5.ne4_ne4.FC5AV1C-L.anvil_intel (Overall: NLFAIL) details:
  SMS_D_Ln5.ne4_ne4.FC5AV1C-L.anvil_intel.cam-cosplite_nhtfrq5 (Overall: FAIL) details:
  SMS_Ld1.ne4_ne4.F-EAMv1-AQP1.anvil_intel.cam-clubb_only (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-preqx_ftype0 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-preqx_ftype1 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-preqx_ftype4 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetahy_ftype0 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetahy_ftype1 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetahy_ftype2 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetahy_ftype4 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetahy_sl (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetanh_ftype0 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetanh_ftype1 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetanh_ftype2 (Overall: NLFAIL) details:
  SMS.ne4_ne4.FC5AV1C-L.anvil_intel.cam-thetanh_ftype4 (Overall: NLFAIL) details:

Log for failed test:

 96:  reading ctl namelist...
 84: forrtl: severe (194): Run-Time Check Failure. The variable 'namelist_mod_mp_readnl_$SE_PARTMETHOD' is being used in '/blues/gpfs/home/onguba/acme-fork-mt/components/homme/src/share/namelist_mod.F90(617,5)' without being defined
 84: Image              PC                Routine            Line        Source
 84: e3sm.exe           00000000043F6B21  namelist_mod_mp_r         617  namelist_mod.F90
 84: e3sm.exe           000000000421C27C  dyn_comp_mp_dyn_i         135  dyn_comp.F90
 84: e3sm.exe           0000000002DA8FA1  inital_mp_cam_ini          31  inital.F90
 84: e3sm.exe           00000000008556A1  cam_comp_mp_cam_i         159  cam_comp.F90
 84: e3sm.exe           0000000000820BB0  atm_comp_mct_mp_a         312  atm_comp_mct.F90
 84: e3sm.exe           000000000046276C  component_mod_mp_         257  component_mod.F90
 84: e3sm.exe           0000000000427799  cime_comp_mod_mp_        1338  cime_comp_mod.F90
 84: e3sm.exe           00000000004594F5  MAIN__                    122  cime_driver.F90

@oksanaguba
Copy link
Contributor

Default run for ne30np4 fails with

346: seq_domain_check_grid - n: 71 d1:   52.654817 d2:   52.656098 diff:  0.00128053861823 eps:  0.00000000000100
346: seq_domain_check_grid - n: 72 d1:   53.702956 d2:   53.705905 diff:  0.00294888806849 eps:  0.00000000000100
 44:  (seq_domain_check_grid) ERROR: incompatible domain grid coordinates
 44:  ERROR: (seq_domain_check_grid)  incompatible domain grid coordinates
117:  (seq_domain_check_grid) ERROR: incompatible domain grid coordinates

This is because of _map=2 in default theta-l. I always run with _map=0 to avoid setting tolerances. What should I do here?

@oksanaguba
Copy link
Contributor

This is what I did before:

contrun-sl.sh:#./xmlchange EPS_FRAC=1.0
contrun-sl.sh:#./xmlchange EPS_AAREA=1.0
contrun-sl.sh:#./xmlchange EPS_AMASK=1.0
contrun-sl.sh:#./xmlchange EPS_AGRID=1.0
contrun-sl.sh:#./xmlchange EPS_OAREA=1.0
contrun-sl.sh:#./xmlchange EPS_OMASK=1.0
contrun-sl.sh:#./xmlchange EPS_OGRID=1.0

@oksanaguba
Copy link
Contributor

For ne30np4 run I did not have another run with all these defaults, so, comparing parameters by eye. All looks ok, now default rsplit is 2 (all our ne30 runs were probably done with 3 or 1), ftype is 2, not 4. Map is 2, all newer runs were done with 0.

dt_remap: (0=disabled)      600.00
dt_tracer (SE), per RK stage:   1800.00   900.00
dt_dyn:                     300.00
dt_dyn (viscosity):         300.00
dt_tracer (viscosity):      300.00
dt_vis_TOM:     300.00
CAM dtime (dt_phys):           1800.00

 Reference element projection: cubed_sphere_map=           2

 readnl: theta_hydrostatic_mode =  T
 readnl: transport_alg   =           12
 readnl: semi_lagrange_cdr_alg   =           21
 readnl: semi_lagrange_cdr_check   =  F
 readnl: semi_lagrange_hv_q   =            1
 readnl: semi_lagrange_nearest_point_lev   =          256
 readnl: tstep_type    =            5
 readnl: theta_advect_form =            1
 readnl: vert_remap_q_alg  =           10
 readnl: se_nsplit         =           -1
 readnl: se_tstep         =    300.000000000000
 readnl: se_ftype          =            2
 readnl: se_limiter_option =            9
 readnl: qsplit (deprecated) =            6
 readnl: vertical remap frequency rsplit (0=disabled) (deprecated):            1
 readnl: dt_tracer_factor =            6
 readnl: vertical remap frequency dt_remap_factor (0=disabled):            2
 readnl: runtype       =            0
 readnl: se_fv_phys_remap_alg =            1
 Constant (hyper)viscosity used.
 hypervis_subcycle     =            1
 hypervis_subcycle_tom =            1
 hypervis_subcycle_q   =            6
viscosity:  nu (vor/div) =  0.10E+16 0.10E+16
viscosity:  nu_s      =  0.10E+16
viscosity:  nu_q      =  0.10E+16
viscosity:  nu_p      =  0.10E+16
viscosity:  nu_top      =  0.25E+06

 COMPOSE> use HV; nu_q, all:  1.000000000000000E+015 

@mt5555
Copy link
Contributor Author

mt5555 commented Dec 31, 2019

cubed_sphere_map issues should be fixed now
np4 grids (non-RRM): 0
RRM grids: 2
pg2 grids: 2

there will be an issue for new np4 grids made with TR mapping files, issue #3369

@mt5555
Copy link
Contributor Author

mt5555 commented Jan 1, 2020

testing with e3sm_integration found two more issues:

need to set se_ftype=0 default with SCAM=1 (single column model)

Other erros were cause by some confusion since namelist defaults are based on cfg_ref->dyn_target = "preqx" or "theta-l", while the build options are based on CAM_TARGET (stored in opts{'dyn_target'}) can have many options (preqx, preqx_acc, preqx_kokkos, theta-l)

fix SCAM=1 default value of se_ftype
make all preqx_* targets use the 'preqx' defaults
dont access uniitalized variables on the non-root MPI task
@oksanaguba
Copy link
Contributor

So error 84: forrtl: severe (194): Run-Time Check Failure. The variable 'namelist_mod_mp_readnl_$SE_PARTMETHOD' is being used in '/blues/gpfs/home/onguba/acme-fork-mt/components/homme/src/share/namelist_mod.F90(617,5)' without being defined
was fixed by the last commit?

@mt5555
Copy link
Contributor Author

mt5555 commented Jan 1, 2020

yes, should be fixed by afc60ae

@mt5555
Copy link
Contributor Author

mt5555 commented Jan 1, 2020

with afc60ae, all the e3sm_integration tests pass on sandiatoss3 except for the theta dycore ones that have not yet been reblessed, and the ICG test which has been failing for a while.

@oksanaguba
Copy link
Contributor

Ran e3sm_integration suite against master of Dec 20: A lot of NL fails, also,

(base) [onguba@blueslogin3 as]$ ./cs.status.20200102_101722_oohijy | grep 'Overall: FAIL'
  HOMMEBFB_P24.f19_g16_rx1.A.anvil_intel (Overall: FAIL) details:
  PET.f45_g37_rx1.A.anvil_intel.allactive-mach-pet (Overall: FAIL) details:
  SMS_D_Ld1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.anvil_intel.allactive-v1cmip6 (Overall: FAIL) details:
  SMS.f09_g16_a.MALI.anvil_intel (Overall: FAIL) details:

For hommebfb it is a build fail, for PET is was failing with the same sigterm error 15 as before, for SMS_D_Ld1 test the error looks the same as on dashboard ( 806: e3sm.exe 000000000D36E86F m_matattrvectmul_ 267 m_MatAttrVectMul.F90), for SMS.f09_..MALI it is a build fail as on cdash.

@oksanaguba
Copy link
Contributor

Configuring ne30np4 (which ran) atm log has these values (I see nothing alarming)"

/home/onguba/as/theta-ne30-hy-mtdefaults-dec30/run
 tstep_type =            5
dt_remap: (0=disabled)      600.00
dt_tracer (SE), per RK stage:   1800.00   900.00
dt_dyn:                     300.00
dt_dyn (viscosity):         300.00
dt_tracer (viscosity):      300.00
dt_vis_TOM:     300.00
CAM dtime (dt_phys):           1800.00

 Reference element projection: cubed_sphere_map=           0

 readnl: theta_hydrostatic_mode =  T
 readnl: transport_alg   =           12
 readnl: semi_lagrange_cdr_alg   =           21
 readnl: semi_lagrange_cdr_check   =  F
 readnl: semi_lagrange_hv_q   =            1
 readnl: semi_lagrange_nearest_point_lev   =          256
 readnl: tstep_type    =            5
 readnl: theta_advect_form =            1
 readnl: vert_remap_q_alg  =           10
 readnl: se_nsplit         =           -1
 readnl: se_tstep         =    300.000000000000
 readnl: se_ftype          =            2
 readnl: se_limiter_option =            9
 readnl: qsplit (deprecated) =            6
 readnl: vertical remap frequency rsplit (0=disabled) (deprecated):            1
 readnl: dt_tracer_factor =            6
 readnl: vertical remap frequency dt_remap_factor (0=disabled):            2
 readnl: runtype       =            0
 readnl: se_fv_phys_remap_alg =            1
 Constant (hyper)viscosity used.
 hypervis_subcycle     =            1
 hypervis_subcycle_tom =            1
 hypervis_subcycle_q   =            6
viscosity:  nu (vor/div) =  0.10E+16 0.10E+16
viscosity:  nu_s      =  0.10E+16
viscosity:  nu_q      =  0.10E+16
viscosity:  nu_p      =  0.10E+16
viscosity:  nu_top      =  0.25E+06

@oksanaguba
Copy link
Contributor

Configuring ne30pg2 (which ran) I see nothing alarming:

/home/onguba/as/theta-ne30pg2-hy-mtdefaults-dec30/run
dt_remap: (0=disabled)      600.00
dt_tracer (SE), per RK stage:   1800.00   900.00
dt_dyn:                     300.00
dt_dyn (viscosity):         300.00
dt_tracer (viscosity):      300.00
dt_vis_TOM:     300.00
CAM dtime (dt_phys):           1800.00

 Reference element projection: cubed_sphere_map=           2

 readnl: theta_hydrostatic_mode =  T
 readnl: transport_alg   =           12
 readnl: semi_lagrange_cdr_alg   =           21
 readnl: semi_lagrange_cdr_check   =  F
 readnl: semi_lagrange_hv_q   =            1
 readnl: semi_lagrange_nearest_point_lev   =          256
 readnl: tstep_type    =            5
 readnl: theta_advect_form =            1
 readnl: vert_remap_q_alg  =           10
 readnl: se_nsplit         =           -1
 readnl: se_tstep         =    300.000000000000
 readnl: se_ftype          =            2
 readnl: se_limiter_option =            9
 readnl: qsplit (deprecated) =            6
 readnl: vertical remap frequency rsplit (0=disabled) (deprecated):            1
 readnl: dt_tracer_factor =            6
 readnl: vertical remap frequency dt_remap_factor (0=disabled):            2
 readnl: runtype       =            0
 readnl: se_fv_phys_remap_alg =            1
 Constant (hyper)viscosity used.
 hypervis_subcycle     =            1
 hypervis_subcycle_tom =            1
 hypervis_subcycle_q   =            6
viscosity:  nu (vor/div) =  0.10E+16 0.10E+16
viscosity:  nu_s      =  0.10E+16
viscosity:  nu_q      =  0.10E+16
viscosity:  nu_p      =  0.10E+16
viscosity:  nu_top      =  0.25E+06

@oksanaguba oksanaguba added the BFB PR leaves answers BFB label Jan 7, 2020
@oksanaguba
Copy link
Contributor

This is ready.

oksanaguba added a commit that referenced this pull request Jan 8, 2020
…3368)

Add XML defaults for theta-l dycore
Remove unused/obsolete dycore variables from XML defaults
Add code to allow namelist defaults to be set based on value of cam_target
Removed se_ftype=2 from all use cases (and made this a default)
Support new timestep interface (se_tstep, dt_tracer_factor, dt_remap_factor) for theta-l dycore
Update all theta "testmods" test cases to preserve original settings

[BFB] with nlfails.
@oksanaguba
Copy link
Contributor

merged to next

oksanaguba added a commit that referenced this pull request Jan 9, 2020
…3368)

Add XML defaults for theta-l dycore
Remove unused/obsolete dycore variables from XML defaults
Add code to allow namelist defaults to be set based on value of cam_target
Removed se_ftype=2 from all use cases (and made this a default)
Support new timestep interface (se_tstep, dt_tracer_factor, dt_remap_factor) for theta-l dycore
Update all theta "testmods" test cases to preserve original settings

[BFB] with nlfails.
@oksanaguba oksanaguba merged commit afc60ae into E3SM-Project:master Jan 9, 2020
@mt5555 mt5555 deleted the theta-xml-defaults2 branch January 22, 2020 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Atmosphere BFB PR leaves answers BFB HOMME
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants