Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added nonhydrostatic capabilities to MPAS-O #21

Open
wants to merge 44 commits into
base: master
Choose a base branch
from
Open

added nonhydrostatic capabilities to MPAS-O #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
88fb291
added nonhydro to v2
scalandr Feb 8, 2022
4c0eeaa
internal wave and overflow run fine
scalandr Feb 9, 2022
28d6588
added seven modules for turbulence
scalandr Feb 14, 2022
e84e755
modified files in shared folder for les turbulence
scalandr Feb 15, 2022
91dff6b
added les turbulence changes in the time-stepping schemes, the code r…
scalandr Feb 16, 2022
dba0784
correction for RK4
scalandr Feb 17, 2022
7355644
removed a couple of print statements
scalandr Mar 24, 2022
5ca480c
changes in mpas_ocn_core_interface.F
scalandr May 12, 2022
a5fd43f
compiles with OpenMP
scalandr May 18, 2022
95295fa
got rid of two equation model (for now)
scalandr May 26, 2022
046ce71
addressed comments 1-5
scalandr May 26, 2022
650c6dd
changed name to nonhydrostaticPKG and added variables to package
scalandr Jun 10, 2022
c2b2412
changed name from verticalVelocity to vertVelocityNonhydro
scalandr Jun 28, 2022
3eacc9e
cleaned nonhydrostatic_pressure_solve.F and vel_hadv_coriolis.F
scalandr Jun 29, 2022
c9cf963
cleaned subroutine ocn_vert_transport_velocity_top
scalandr Jul 1, 2022
e81723c
cleaned mpas_ocn_vel_pressure_grad.F
scalandr Jul 1, 2022
2c5bcbf
now it compiles, fixed src/shared/Makefile
scalandr Jul 1, 2022
2f4fa64
fixed indentation in src/Registry.xml
scalandr Jul 5, 2022
c947a8e
added vertVelocityNonhydro to nonhydrostaticPKG
scalandr Jul 5, 2022
59bb822
added nonhydrostaticPressure to nonhydrostaticPKG
scalandr Jul 6, 2022
2f46b53
added blank lines in rk4 and fixed indentation
scalandr Jul 6, 2022
bc8290b
added tendVertVelocityNonhydro to nonhydrostaticPKG
scalandr Jul 6, 2022
fdb1d58
cleaned mpas_ocn_tendency.F and mpas_ocn_diagnostics.F
scalandr Jul 6, 2022
d55b293
now it compiles, fixed issues in split-implicit
scalandr Jul 7, 2022
1577b35
got rid of velocity*Tend variables
scalandr Jul 8, 2022
58ce136
got rid of unused flags
scalandr Jul 12, 2022
4fc4447
changed nonhydrostaticPressure name in time-steppers
scalandr Jul 12, 2022
5fc334c
cleaned split explicit and vel hmix del2
scalandr Jul 12, 2022
6b2320a
cleaned mpas_ocn_vel_pressure_grad.F
scalandr Jul 13, 2022
a111c41
inlined implicit vert mix in vert momentum
scalandr Jul 14, 2022
18a8b6e
fixes in ocean.cmake, mpas_ocn_nonhydrostatic_pressure_solve.F and mp…
scalandr Jan 20, 2023
e209e96
corrected ocn_vertVel_vmix_tend_implicit in mpas_ocn_vmix.F
scalandr Mar 10, 2023
1934c38
cleaned code
scalandr Mar 10, 2023
3e9778b
added nonhydro vars to restart file and corrected vvel coriolis
scalandr Mar 22, 2023
bc9cabc
changes to hmix for vertical momentum
scalandr Apr 6, 2023
9b31e2b
added flags in bld folder and corrected ocn_vel_coriolis_nonhydro_ten…
scalandr Apr 7, 2023
b94a4c0
added lines to build the e3sm way on Anvil
scalandr May 1, 2023
f9a76a1
added del4 in vertical momentum
scalandr May 1, 2023
2dec6bd
merged with most recent version of e3sm
scalandr May 2, 2023
707cc46
fixed building issue
scalandr May 2, 2023
a882fd9
Merge remote-tracking branch 'E3SM-Project/E3SM/master' into ocean/no…
scalandr May 19, 2023
3759f61
possible fix for e3sm tests
scalandr May 24, 2023
85718b4
possible fix for problem with ocn_vvel_hmix_del4
scalandr May 24, 2023
9052bc6
fixed omp pragma in mpas_ocn_vmix.F
scalandr May 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file modified .circleci/config.yml
100644 → 100755
View file
Open in desktop
Empty file.
Empty file modified .gitignore
100644 → 100755
View file
Open in desktop
Empty file.
Empty file modified .gitmodules
100644 → 100755
View file
Open in desktop
Empty file.
2 changes: 2 additions & 0 deletions cime_config/machines/cmake_macros/gnu_chicoma-cpu.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,5 @@ set(MPIFC "ftn")
set(SCC "gcc")
set(SCXX "g++")
set(SFC "gfortran")
set(PETSC_PATH "$ENV{PETSC_PATH}")
string(APPEND SLIBS " -L$ENV{PETSC_PATH}/lib -lpetsc")
2 changes: 2 additions & 0 deletions cime_config/machines/cmake_macros/intel_anvil.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,5 @@ endif()
set(NETCDF_C_PATH "$ENV{NETCDF_C_PATH}")
set(NETCDF_FORTRAN_PATH "$ENV{NETCDF_FORTRAN_PATH}")
set(PNETCDF_PATH "$ENV{PNETCDF_PATH}")
set(PETSC_PATH "$ENV{PETSC_PATH}")
string(APPEND SLIBS " -L$ENV{PETSC_PATH}/lib -lpetsc")
6 changes: 6 additions & 0 deletions cime_config/machines/config_machines.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2335,6 +2335,7 @@
<command name="load">netcdf-cxx/4.2-gkqc6fq</command>
<command name="load">netcdf-fortran/4.4.4-eanrh5t</command>
<command name="load">parallel-netcdf/1.11.0-y3nmmej</command>
<command name="load">petsc-3.16.1-intel-20.0.4-3vcputc</command>
</modules>
<modules compiler="intel" mpilib="openmpi">
<command name="load">openmpi/4.1.1-v3b3npd</command>
Expand Down Expand Up @@ -2377,6 +2378,7 @@
<env name="NETCDF_C_PATH">$SHELL{dirname $(dirname $(which nc-config))}</env>
<env name="NETCDF_FORTRAN_PATH">$SHELL{dirname $(dirname $(which nf-config))}</env>
<env name="PNETCDF_PATH">$SHELL{dirname $(dirname $(which pnetcdf_version))}</env>
<env name="PETSC_PATH">/lcrc/soft/climate/compass/anvil/spack/spack_for_mache_1.4.1/var/spack/environments/compass_1_1_0_intel_impi_netlib_lapack_petsc/.spack-env/view</env>
<env name="PATH">/lcrc/group/e3sm/soft/perl/5.26.0/bin:$ENV{PATH}</env>
</environment_variables>
<environment_variables mpilib="mvapich">
Expand Down Expand Up @@ -4065,6 +4067,10 @@
<env name="FI_CXI_RX_MATCH_MODE">software</env>
<env name="MPICH_COLL_SYNC">MPI_Bcast</env>
</environment_variables>
<environment_variables compiler="gnu" mpilib="mpich">
<env name="PETSC_PATH">/usr/projects/climate/SHARED_CLIMATE/software/chicoma-cpu/petsc/gcc-12.1.0/cray-mpich-8.1.21</env>
<env name="LD_LIBRARY_PATH">/usr/projects/climate/SHARED_CLIMATE/software/chicoma-cpu/petsc/gcc-12.1.0/cray-mpich-8.1.21/lib:$ENV{LD_LIBRARY_PATH}</env>
</environment_variables>
<resource_limits>
<resource name="RLIMIT_STACK">-1</resource>
</resource_limits>
Expand Down
23 changes: 21 additions & 2 deletions components/mpas-framework/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -508,7 +508,26 @@ CPPINCLUDES =
FCINCLUDES =
LIBS =

#
# Add links to PETSC links if requested
ifeq "$(USE_PETSC)" "true"
ifndef PETSC
$(error PETSC is not set. Please set PETSC to the PETSC install directory when USE_PETSC=true)
endif
ifneq (, $(shell ls $(PETSC)/lib/libpetsc.*))
LIBS += -L$(PETSC)/lib
CPPINCLUDES += -I$(PETSC)/include
FCINCLUDES += -I$(PETSC)/include
else ifneq (, $(shell ls $(PETSC)/libpetsc.*))
LIBS += -L$(PETSC)
CPPINCLUDES += -I$(PETSC)
FCINCLUDES += -I$(PETSC)
else
$(error libpetsc.* does NOT exist in $(PETSC) or $(PETSC)/lib)
endif
LIBS += -lpetsc
override CPPFLAGS += -DUSE_PETSC
endif

# If user has indicated a PIO2 library, define USE_PIO2 pre-processor macro
#
ifeq "$(USE_PIO2)" "true"
Expand All @@ -524,7 +543,7 @@ ifneq ($(wildcard $(PIO)/lib), )
else
PIO_LIB = $(PIO)
endif
LIBS = -L$(PIO_LIB)
LIBS += -L$(PIO_LIB)

#
# Regardless of PIO library version, look for an include subdirectory of PIO path
Expand Down
19 changes: 19 additions & 0 deletions components/mpas-ocean/bld/build-namelist
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -526,6 +526,8 @@ add_default($nl, 'config_use_mom_del2');
add_default($nl, 'config_mom_del2');
add_default($nl, 'config_use_tracer_del2');
add_default($nl, 'config_tracer_del2');
add_default($nl, 'config_use_vertMom_del2');
add_default($nl, 'config_vertMom_del2');

#############################
# Namelist group: hmix_del4 #
Expand Down Expand Up @@ -836,6 +838,19 @@ add_default($nl, 'config_vegetation_drag_coefficient');

add_default($nl, 'config_density0');

###########################################
# Namelist group: nonhydrostatic_pressure #
###########################################

add_default($nl, 'config_enable_nonhydrostatic_mode');
add_default($nl, 'config_nonhydrostatic_solve_surface_boundary_condition');
add_default($nl, 'config_nonhydrostatic_remove_rhs_mean');
add_default($nl, 'config_nonhydrostatic_preconditioner');
add_default($nl, 'config_nonhydrostatic_solver_type');
add_default($nl, 'config_petsc_rtol');
add_default($nl, 'config_petsc_atol');
add_default($nl, 'config_petsc_maxit');

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

were these added by the automated scripts? Or hand entered?

#####################################
# Namelist group: pressure_gradient #
#####################################
Expand Down Expand Up @@ -956,6 +971,9 @@ add_default($nl, 'config_disable_vel_topographic_wave_drag');
add_default($nl, 'config_disable_vel_explicit_bottom_drag');
add_default($nl, 'config_disable_vel_vmix');
add_default($nl, 'config_disable_vel_vadv');
add_default($nl, 'config_disable_vvel_all_tend');
add_default($nl, 'config_disable_vvel_coriolis');
add_default($nl, 'config_disable_vvel_adv');
add_default($nl, 'config_disable_tr_all_tend');
add_default($nl, 'config_disable_tr_adv');
add_default($nl, 'config_disable_tr_hmix');
Expand Down Expand Up @@ -1744,6 +1762,7 @@ my @groups = qw(run_modes
rayleigh_damping
vegetation_drag
ocean_constants
nonhydrostatic_pressure
pressure_gradient
eos
eos_linear
Expand Down
1 change: 1 addition & 0 deletions components/mpas-ocean/bld/build-namelist-group-list
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ my @groups = qw(run_modes
rayleigh_damping
vegetation_drag
ocean_constants
nonhydrostatic_pressure
pressure_gradient
eos
eos_linear
Expand Down
18 changes: 18 additions & 0 deletions components/mpas-ocean/bld/build-namelist-section
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ add_default($nl, 'config_use_mom_del2');
add_default($nl, 'config_mom_del2');
add_default($nl, 'config_use_tracer_del2');
add_default($nl, 'config_tracer_del2');
add_default($nl, 'config_use_vertMom_del2');
add_default($nl, 'config_vertMom_del2');

#############################
# Namelist group: hmix_del4 #
Expand Down Expand Up @@ -355,6 +357,19 @@ add_default($nl, 'config_vegetation_drag_coefficient');

add_default($nl, 'config_density0');

###########################################
# Namelist group: nonhydrostatic_pressure #
###########################################

add_default($nl, 'config_enable_nonhydrostatic_mode');
add_default($nl, 'config_nonhydrostatic_solve_surface_boundary_condition');
add_default($nl, 'config_nonhydrostatic_remove_rhs_mean');
add_default($nl, 'config_nonhydrostatic_preconditioner');
add_default($nl, 'config_nonhydrostatic_solver_type');
add_default($nl, 'config_petsc_rtol');
add_default($nl, 'config_petsc_atol');
add_default($nl, 'config_petsc_maxit');

#####################################
# Namelist group: pressure_gradient #
#####################################
Expand Down Expand Up @@ -475,6 +490,9 @@ add_default($nl, 'config_disable_vel_topographic_wave_drag');
add_default($nl, 'config_disable_vel_explicit_bottom_drag');
add_default($nl, 'config_disable_vel_vmix');
add_default($nl, 'config_disable_vel_vadv');
add_default($nl, 'config_disable_vvel_all_tend');
add_default($nl, 'config_disable_vvel_coriolis');
add_default($nl, 'config_disable_vvel_adv');
add_default($nl, 'config_disable_tr_all_tend');
add_default($nl, 'config_disable_tr_adv');
add_default($nl, 'config_disable_tr_hmix');
Expand Down
15 changes: 15 additions & 0 deletions components/mpas-ocean/bld/namelist_files/namelist_defaults_mpaso.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,8 @@
<config_mom_del2 ocn_grid="ECwISC30to60E2r1">1000.0</config_mom_del2>
<config_use_tracer_del2>.false.</config_use_tracer_del2>
<config_tracer_del2>10.0</config_tracer_del2>
<config_use_vertMom_del2>.false.</config_use_vertMom_del2>
<config_vertMom_del2>10.0</config_vertMom_del2>

<!-- hmix_del4 -->
<config_use_mom_del4>.true.</config_use_mom_del4>
Expand Down Expand Up @@ -395,6 +397,16 @@
<!-- ocean_constants -->
<config_density0>1026.0</config_density0>

<!-- nonhydrostatic_pressure -->
<config_enable_nonhydrostatic_mode>.false.</config_enable_nonhydrostatic_mode>
<config_nonhydrostatic_solve_surface_boundary_condition>'noGradient'</config_nonhydrostatic_solve_surface_boundary_condition>
<config_nonhydrostatic_remove_rhs_mean>.true.</config_nonhydrostatic_remove_rhs_mean>
<config_nonhydrostatic_preconditioner>'jacobi'</config_nonhydrostatic_preconditioner>
<config_nonhydrostatic_solver_type>'pipecgrr'</config_nonhydrostatic_solver_type>
<config_petsc_rtol>1.0e-10</config_petsc_rtol>
<config_petsc_atol>1.0E-20</config_petsc_atol>
<config_petsc_maxit>10000</config_petsc_maxit>

<!-- pressure_gradient -->
<config_pressure_gradient_type>'Jacobian_from_TS'</config_pressure_gradient_type>
<config_common_level_weight>0.5</config_common_level_weight>
Expand Down Expand Up @@ -509,6 +521,9 @@
<config_disable_vel_explicit_bottom_drag>.false.</config_disable_vel_explicit_bottom_drag>
<config_disable_vel_vmix>.false.</config_disable_vel_vmix>
<config_disable_vel_vadv>.false.</config_disable_vel_vadv>
<config_disable_vvel_all_tend>.false.</config_disable_vvel_all_tend>
<config_disable_vvel_coriolis>.false.</config_disable_vvel_coriolis>
<config_disable_vvel_adv>.false.</config_disable_vvel_adv>
<config_disable_tr_all_tend>.false.</config_disable_tr_all_tend>
<config_disable_tr_adv>.false.</config_disable_tr_adv>
<config_disable_tr_hmix>.false.</config_disable_tr_hmix>
Expand Down
105 changes: 105 additions & 0 deletions components/mpas-ocean/bld/namelist_files/namelist_definition_mpaso.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,22 @@ Valid values: any positive real
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_use_vertMom_del2" type="logical"
category="hmix_del2" group="hmix_del2">
If true, Laplacian horizontal mixing is used on the vertical momentum equation.

Valid values: .true. or .false.
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_vertMom_del2" type="real"
category="hmix_del2" group="hmix_del2">
Horizontal diffusion.

Valid values: any positive real
Default: Defined in namelist_defaults.xml
</entry>


<!-- hmix_del4 -->

Expand Down Expand Up @@ -1821,6 +1837,71 @@ Valid values: any positive real, but typically 1000-1035
Default: Defined in namelist_defaults.xml
</entry>

<!-- nonhydrostatic_pressure -->

<entry id="config_enable_nonhydrostatic_mode" type="logical"
category="nonhydrostatic_pressure" group="nonhydrostatic_pressure">
Flag to enable the nonhydrostatic pressure calculation

Valid values: .true. or .false.
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_nonhydrostatic_solve_surface_boundary_condition" type="char*1024"
category="nonhydrostatic_pressure" group="nonhydrostatic_pressure">
setting for surface boundary condition for the pressure correction solve

Valid values: noGradient or no pressure
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_nonhydrostatic_remove_rhs_mean" type="logical"
category="nonhydrostatic_pressure" group="nonhydrostatic_pressure">
flag to enabling mean from the right hand side of the nonhydro solve

Valid values: .true. or .false.
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_nonhydrostatic_preconditioner" type="char*1024"
category="nonhydrostatic_pressure" group="nonhydrostatic_pressure">
String describing preconditioner to use, recommended options -- jacobi, bjacobi, ilu

Valid values: jacobi, bjacobi, ilu
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_nonhydrostatic_solver_type" type="char*1024"
category="nonhydrostatic_pressure" group="nonhydrostatic_pressure">
String describing the solver context, recommended types -- pipecgrr, pipecg, cg, bcgs, gmres

Valid values: pipecgrr, pipecg, cg, bcgs, gmres
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_petsc_rtol" type="real"
category="nonhydrostatic_pressure" group="nonhydrostatic_pressure">
relative tolerance for the norm in petsc convergence

Valid values: very small positive reals
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_petsc_atol" type="real"
category="nonhydrostatic_pressure" group="nonhydrostatic_pressure">
absolute tolerance for the norm in petsc convergence

Valid values: very small positive reals
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_petsc_maxit" type="integer"
category="nonhydrostatic_pressure" group="nonhydrostatic_pressure">
maximum number of iterations in petsc solve

Valid values: positive integers
Default: Defined in namelist_defaults.xml
</entry>

<!-- pressure_gradient -->

Expand Down Expand Up @@ -2419,6 +2500,30 @@ Valid values: .true. or .false.
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_disable_vvel_all_tend" type="logical"
category="debug" group="debug">
Disables all tendencies on vertical momentum field.

Valid values: .true. or .false.
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_disable_vvel_coriolis" type="logical"
category="debug" group="debug">
Disables tendencies on the vetical velocity field from the Coriolis force.

Valid values: .true. or .false.
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_disable_vvel_adv" type="logical"
category="debug" group="debug">
Disables tendencies on the vertical velocity field from vertical and horizontal advection.

Valid values: .true. or .false.
Default: Defined in namelist_defaults.xml
</entry>

<entry id="config_disable_tr_all_tend" type="logical"
category="debug" group="debug">
Disables all tendencies on tracer fields.
Expand Down
Loading