Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MPAS standalone: Add gnu >=10 support on Cori #4854

Merged
merged 1 commit into from
Apr 1, 2022

Conversation

xylar
Copy link
Contributor

@xylar xylar commented Mar 25, 2022

This merge adds the -fallow-argument-mismatch flag to the gnu-nersc target for MPAS standalone builds. This got missed in #4821.

[BFB]

@xylar xylar added BFB PR leaves answers BFB mpas-framework labels Mar 25, 2022
@xylar xylar requested a review from akturner March 25, 2022 08:09
@xylar xylar removed their assignment Mar 25, 2022
This merge adds the `-fallow-argument-mismatch` flag to the
gnu-nersc target for MPAS standalone builds.  This got missed
in a previous update.
@xylar xylar force-pushed the mpas-framework/fix-gnu-nersc branch from 56e8b36 to ef2a1d9 Compare March 25, 2022 08:12
@xylar
Copy link
Contributor Author

xylar commented Mar 25, 2022

Testing

I compiled the code on Cori-Haswell with this branch and the updated Gnu modules (#4839), and ran the compass pr test suite.

The ocean/global_ocean/QU240/PHC/files_for_e3sm test failed on one of the calls to ESMF_RegridWeightGen, so that deservest some follow-up in compass (MPAS-Dev/compass#335). But other tests were successful.

@xylar
Copy link
Contributor Author

xylar commented Mar 25, 2022

@mark-petersen, @matthewhoffman, @trhille and @akturner, this is ready for you to review. (@trhille, I can't add you as a reviewer so please just comment below).

@xylar xylar mentioned this pull request Mar 25, 2022
85 tasks
@trhille
Copy link
Contributor

trhille commented Mar 25, 2022

This enabled me to build MALI in COMPASS on Cori using @xylar's Spack instance from /global/cfs/cdirs/e3sm/software/compass/cori-haswell/test_spack

I cherry-picked commit ef2a1d9 onto MALI-Dev/develop and built using:
make clean gnu-nersc ALBANY=true DEBUG=true

I still get warnings about rank mismatch, but they are no longer errors. I get a dozen or so warnings like this one from mpas_global_sum_mod.F:

Warning: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)
mpas_global_sum_mod.F:1470:25:

 1470 |       call MPI_ALLREDUCE(localSum, globalSum, 1, MPI_INTEGER8, &
      |                         1
......
 6407 |       call MPI_ALLREDUCE(localSum, globalSum, nFields, &
      |                         2

As well as another dozen from mpas_dmpar.F:

Warning: Rank mismatch between actual argument at (1) and actual argument at (2) (scalar and rank-2)
mpas_dmpar.F:941:28:

  941 |          call MPI_Allreduce(sendbuf, recvbuf, 1, MPI_2INTEGERKIND, MPI_MINLOC, dminfo % comm, mpi_ierr)
      |                            1
......
 1733 |         call MPI_Allreduce(nUniqueNeededList, totalSize, 1, MPI_INTEGERKIND, MPI_MAX, dminfo % comm, mpi_ierr)
      |                           2 

I ran the full_integration landice test suite, which failed during test execution for landice_hydro_radial_restart_test
and landice_hydro_radial_decomposition_test and during validation for landice_circular_shelf_decomposition_test and landice_thwaites_restart_test.

@xylar
Copy link
Contributor Author

xylar commented Mar 25, 2022

Thanks @trhille!

I ran the full_integration landice test suite, which failed during test execution for landice_hydro_radial_restart_test
and landice_hydro_radial_decomposition_test and during validation for landice_circular_shelf_decomposition_test and landice_thwaites_restart_test.

I just want to note that, at least as far as I understand it, we don't have any reason to think these errors have anything to do with the changes in this PR.

@xylar
Copy link
Contributor Author

xylar commented Mar 25, 2022

I still get warnings about rank mismatch, but they are no longer errors.

I believe that's the nature of the flag I added. I also always see the warnings. Presumably, it's something we'll want to address later on if we can.

Copy link
Contributor

@mark-petersen mark-petersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I compiled ocean, sea ice, and mali with gnu 10 on cori and this works. Specifically, with today's compass/master 127b6d08:

./conda/configure_compass_env.py --conda ~/miconda3 --compiler gnu
source load_dev_compass_1.0.0_cori-haswell_gnu_mpt.sh
module list
  17) gcc/10.3.0
cd /global/homes/m/mpeterse/repos/E3SM/pr/components/mpas-ocean
make gnu-nersc USE_PIO2=true OPENMP=false DEBUG=true GEN_F90=true

Also for mpas-seaice and mpas-albany-landice.

@mark-petersen
Copy link
Contributor

@matthewhoffman and @akturner if a successful compile of your cores with gnu10 is sufficient testing, you can approve based on that.

@xylar
Copy link
Contributor Author

xylar commented Mar 31, 2022

Thanks @mark-petersen, @matthewhoffman and @akturner!

jonbob added a commit that referenced this pull request Mar 31, 2022
MPAS standalone: Add gnu >=10 support on Cori

This merge adds the -fallow-argument-mismatch flag to the gnu-nersc
target for MPAS standalone builds. This got missed in #4821.

[BFB]
@jonbob
Copy link
Contributor

jonbob commented Mar 31, 2022

passes sanity testing -- merged to next

@xylar
Copy link
Contributor Author

xylar commented Mar 31, 2022

Thank you, @jonbob!

@jonbob jonbob merged commit c41fd42 into E3SM-Project:master Apr 1, 2022
@jonbob
Copy link
Contributor

jonbob commented Apr 1, 2022

merged to master

@xylar xylar deleted the mpas-framework/fix-gnu-nersc branch April 2, 2022 07:18
@xylar
Copy link
Contributor Author

xylar commented Apr 2, 2022

@jonbob, thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFB PR leaves answers BFB mpas-framework
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants