Adding ADIOS install paths #5492

jayeshkrishna · 2023-03-02T03:21:42Z

Adding ADIOS install paths and library dependencies for the
different machines+compilers

[BFB]

jayeshkrishna · 2023-03-02T03:36:04Z

To use ADIOS to write E3SM outputs the following changes are needed,

PR Adding support for ADIOS in MPAS #5490 (for MPAS components)
PR Adding ADIOS install paths #5492
PR Support for ADIOS type in drivers #5494
Merging https://github.com/ESMCI/cime/tree/jayeshkrishna/adios2_support to ESMCI CIME Master and updating it in E3SM (PR Adding ADIOS install paths #5492 is a pre-req for merging this branch in CIME)

dqwu · 2023-03-02T15:13:04Z

@jayeshkrishna
In this PR, ADIOS2 libs are added to cmake macros for Cori and Theta, but corresponding ADIOS install paths are not added for
Cori and Theta in config_machines.xml. Since Cori will retire soon and we will probably never need to run E3SM on Theta, we can create a future PR to fix this minor issue.

dqwu · 2023-03-02T15:31:30Z

@jayeshkrishna
ADIOS install path is also missing for anvil. We can create a future PR to add it (together with Cori and Theta).

dqwu · 2023-03-02T15:44:20Z

@jayeshkrishna
When testing ADIOS type on compy and bebop, there are some minor issues encountered, but they are unrelated to ADIOS type.

On compy, there is a confirmed setup issue with --mpilib=mvapich2 (case.setup failure on compy with --mpilib=mvapich2 #5487).
On bebop, there is a confirmed build issue with GNU compiler (buildlib.pio failed with HDF5 linker errors on bebop using GNU compilier #5486). This will be fixed in a separate PR.

dqwu · 2023-03-02T15:56:04Z

@jayeshkrishna
When testing on bebop, --machine=bebop must be explicitly specified, see issue #5485

dqwu · 2023-03-02T16:06:53Z

@jayeshkrishna
On Summit, there are build errors for ADIOS type with "--compiler=ibmgpu"
This is because we actually do not have ibmgpu_summit.cmake (we do have ibmgpu_ascent.cmake) to add ADIOS library dependencies.

Shall we fix this issue in a future PR?

jayeshkrishna · 2023-03-02T17:55:46Z

@jayeshkrishna On Summit, there are build errors for ADIOS type with "--compiler=ibmgpu" This is because we actually do not have ibmgpu_summit.cmake (we do have ibmgpu_ascent.cmake) to add ADIOS library dependencies.

Shall we fix this issue in a future PR?

Let us fix that in a future PR since its not related to ADIOS support (please create an issue for it, if needed)

dqwu · 2023-03-02T18:04:33Z

@jayeshkrishna On Summit, there are build errors for ADIOS type with "--compiler=ibmgpu" This is because we actually do not have ibmgpu_summit.cmake (we do have ibmgpu_ascent.cmake) to add ADIOS library dependencies.
Shall we fix this issue in a future PR?

Let us fix that in a future PR since its not related to ADIOS support (please create an issue for it, if needed)

Actually ibmgpu_summit.cmake is not required for PnetCDF type so far. We can create an issue for ADIOS type after this PR is merged.

jayeshkrishna · 2023-03-02T18:31:59Z

Not sure why ibmgpu_summit.cmake is not required for the PnetCDF type (and required for ADIOS), but let us use a separate issue to discuss it further (maybe a detailed issue description, for the new issue, would help).

jayeshkrishna · 2023-03-02T19:58:54Z

@dqwu in his testing found that ADIOS paths were missing for Anvil (but the ADIOS libs were listed as dependencies) and this caused build failures on Anvil. This issue has now been fixed and the branch was rebased.

Also, as part of the rebase process the ADIOS settings for Anvil/Cori/Theta were restored (these settings can be deleted later, if needed, along with the machine configs) & a fix for PR #5486 was also added

The fix for PR #5486 was to add the paths to the HDF5 library in the config machines and cmake macro files for gnu on bebop.

Adding ADIOS install paths for several machines.

Adding ADIOS library dependencies to CMake macro files for compiler+machines where ADIOS builds are currently available/tested

The build script for SCORPIO in CIME is being moved to a separate script, buildlib.spio (from buildlib.pio). Adding the location to the new script.

dqwu · 2023-03-03T17:47:23Z

@jayeshkrishna
FYI, I have rebased this branch again to upgrade installed ADIOS2 libs from 2.7.0 to 2.8.3 for anvil, bebop and compy. The old 2.7.0 libs were not configured to support latest BP5 format used by SCORPIO.

So far, ADIOS 2.7.0 is only used by anlworkstation (already retired), cori-haswell (to be retired), and theta (we will probably never need to run E3SM on it).

dqwu · 2023-03-04T00:04:57Z

@jayeshkrishna
There are some known issues with IBM XL compiler on Summit, and users can use GNU or PGI compilers to test ADIOS type so far.

jayeshkrishna · 2023-03-04T00:22:42Z

@ndkeen & @grnydawn please review this PR when you get a chance (Note: As discussed above you need couple of other PRs to start using ADIOS out of the box. You might just want to review that the changes in this PR does not break anything for you.).

ndkeen

If I'm reading this correctly, it looks like we are adding quite a few libs to the link line -- even if a case is not using ADIOS. Is that right?

jayeshkrishna · 2023-03-08T02:43:26Z

Yeah, the ADIOS support is added during the SCORPIO build (yeah, and ADIOS adds a number of dep libs) and the user then has the choice to use ADIOS at runtime (PIO_TYPENAME in env_run.xml).

ndkeen · 2023-03-08T03:19:41Z

It doesn't look like it's adding much to the executable, but it does seem like it makes sense to only link with libraries that are actually needed. But, as you note, it's the norm to have XML settings that can toggle options such as this.

grnydawn · 2023-03-08T14:15:13Z

@jayeshkrishna adios2 setting is missing for Crusher. Also we may need a future PR for Frontier support

In config_machines.xml, "ADIOS2_DIR" is missing for crusher machine
In several *_crusher.cmake files in cmake_macros folder, adios2 library setting is missing.

On Crusher( and on Frontier) adios2 is installed through Spack and can be loaded using module.

jayeshkrishna · 2023-03-08T15:25:54Z

Yeah, this PR does not include support for ADIOS on all the machines. @dqwu is working on adding support for ADIOS on the remaining E3SM machines.
(PS: We cannot use ADIOS modules on machines because we need custom fixes that will only be available in the upcoming ADIOS releases)

… next (PR #5492) Adding ADIOS install paths and library dependencies for the different machines+compilers Fixes #5486 [BFB]

ndkeen · 2023-03-13T23:14:51Z

The cori-knl tests failed with:

/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -ladios2_c_mpi
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -ladios2_c
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -ladios2_core_mpi
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -ladios2_core
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -ladios2_evpath
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -ladios2_ffs
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -ladios2_dill
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -ladios2_atl
/usr/lib64/gcc/x86_64-suse-linux/7/../../../../x86_64-suse-linux/bin/ld: cannot find -ladios2_enet

jayeshkrishna · 2023-03-14T02:56:39Z

@dqwu has fixed the permissions for the ADIOS install directories on Cori (that caused the missing adios* libs issue above)

jayeshkrishna added Machine Files BFB PR leaves answers BFB SCORPIO The E3SM I/O library (derived from PIO) labels Mar 2, 2023

jayeshkrishna requested a review from dqwu March 2, 2023 03:21

jayeshkrishna self-assigned this Mar 2, 2023

jayeshkrishna mentioned this pull request Mar 2, 2023

Support for ADIOS type in drivers #5494

Merged

rljacob requested review from ndkeen and grnydawn March 2, 2023 04:03

jayeshkrishna force-pushed the jayeshkrishna/machinefiles/add_adios_install_path branch from e06aca9 to 0f6abb5 Compare March 2, 2023 19:48

jayeshkrishna requested review from grnydawn and ndkeen and removed request for ndkeen and grnydawn March 2, 2023 19:53

dqwu and others added 3 commits March 3, 2023 10:56

Adding ADIOS lib install paths

41a4287

Adding ADIOS install paths for several machines.

Adding ADIOS2 libs to cmake macros files

bb5e62e

Adding ADIOS library dependencies to CMake macro files for compiler+machines where ADIOS builds are currently available/tested

Adding path to CIME SCORPIO build script

aa5b46e

The build script for SCORPIO in CIME is being moved to a separate script, buildlib.spio (from buildlib.pio). Adding the location to the new script.

dqwu force-pushed the jayeshkrishna/machinefiles/add_adios_install_path branch from 0f6abb5 to aa5b46e Compare March 3, 2023 17:22

dqwu approved these changes Mar 4, 2023

View reviewed changes

ndkeen reviewed Mar 8, 2023

View reviewed changes

ndkeen self-requested a review March 8, 2023 03:19

ndkeen approved these changes Mar 8, 2023

View reviewed changes

grnydawn approved these changes Mar 8, 2023

View reviewed changes

jayeshkrishna added a commit that referenced this pull request Mar 9, 2023

Merge branch 'jayeshkrishna/machinefiles/add_adios_install_path' into…

58a1b53

… next (PR #5492) Adding ADIOS install paths and library dependencies for the different machines+compilers Fixes #5486 [BFB]

jayeshkrishna merged commit 6b10de2 into master Mar 11, 2023

jayeshkrishna deleted the jayeshkrishna/machinefiles/add_adios_install_path branch March 11, 2023 03:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding ADIOS install paths #5492

Adding ADIOS install paths #5492

jayeshkrishna commented Mar 2, 2023 •

edited

Loading

jayeshkrishna commented Mar 2, 2023

dqwu commented Mar 2, 2023

dqwu commented Mar 2, 2023

dqwu commented Mar 2, 2023

dqwu commented Mar 2, 2023

dqwu commented Mar 2, 2023

jayeshkrishna commented Mar 2, 2023

dqwu commented Mar 2, 2023

jayeshkrishna commented Mar 2, 2023

jayeshkrishna commented Mar 2, 2023

dqwu commented Mar 3, 2023

dqwu commented Mar 4, 2023

jayeshkrishna commented Mar 4, 2023

ndkeen left a comment

jayeshkrishna commented Mar 8, 2023

ndkeen commented Mar 8, 2023

grnydawn commented Mar 8, 2023

jayeshkrishna commented Mar 8, 2023 •

edited

Loading

ndkeen commented Mar 13, 2023

jayeshkrishna commented Mar 14, 2023

Adding ADIOS install paths #5492

Adding ADIOS install paths #5492

Conversation

jayeshkrishna commented Mar 2, 2023 • edited Loading

jayeshkrishna commented Mar 2, 2023

dqwu commented Mar 2, 2023

dqwu commented Mar 2, 2023

dqwu commented Mar 2, 2023

dqwu commented Mar 2, 2023

dqwu commented Mar 2, 2023

jayeshkrishna commented Mar 2, 2023

dqwu commented Mar 2, 2023

jayeshkrishna commented Mar 2, 2023

jayeshkrishna commented Mar 2, 2023

dqwu commented Mar 3, 2023

dqwu commented Mar 4, 2023

jayeshkrishna commented Mar 4, 2023

ndkeen left a comment

Choose a reason for hiding this comment

jayeshkrishna commented Mar 8, 2023

ndkeen commented Mar 8, 2023

grnydawn commented Mar 8, 2023

jayeshkrishna commented Mar 8, 2023 • edited Loading

ndkeen commented Mar 13, 2023

jayeshkrishna commented Mar 14, 2023

jayeshkrishna commented Mar 2, 2023 •

edited

Loading

jayeshkrishna commented Mar 8, 2023 •

edited

Loading