Update config_machines.xml for Feb 18 software update on Frontier. #6977
Conversation
|
@jgfouca , Just to check, do you think |
|
@grnydawn my guess, looking at the module changes, is no. |
|
Have you verified that tests are BFB? and similar performance? Including ne256 and ne1024? |
The chart above shows that the various configurations have similar performance for a short NE30 run using 8 nodes. If you have NE256 and NE1024 runs to suggest, then I'm happy to try them, though they may be too big to test I compared BFB using
|
|
I might suggest at least trying a day or two of ne1024 (with any recent production script) to ensure no issues before merging. |
|
I would like us to settle the machine/compiler definition issue (#6773) first and then apply these changes to the unified entry. |
|
Would this also be a good chance to set the missing |
|
I'm aware but 11 days seems like enough time to settle the machine/compiler name issue. |
|
Okay, the |
I am trying an NE1024 test from @whannah1, and I got a seg fault for the new compiler setting. I'm investigating with NE30 and NE256 versions of the same test. To be continued... |
|
Testing of NE1024 revealed an issue with Then I found that performance takes a slight hit moving to
I just pushed a commit that uses the last line. We can use this for now until we resolve the performance loss or get new versions. |
|
To summarize, this pull request makes the following changes to
|
Fixes an issue encountered during testing of PR #6977, where the tags we used went above the max tag allowed by the MPI distribution [BFB]
|
I will replace this with two separate pull requests after the changes of #6990: one for the minimal |
|
Replaced partly by #7021 |
Various changes to
config_machines.xmlin preparation for system software updates on Frontier on 2025-02-18. Within thefrontier-scream-gpumachine:lmodpaths from/usr/shareto/opt/cray/pebecause the/usr/sharesoftware is not maintained and is not available on internal OLCF test computers.craygnuamdgputo latest version ofcpeavailable on Frontier,cpe/24.11.craygnuamdgpu. I used to think this wasn't needed, sincemodule load cpe/24.11should set all the appropriate defaults. I recently discovered that this setting of defaults only works if themodule load cpe/24.11is a separate command before the othermodule loads. Cime uses a singlemodule loadwith a list of modules, which does not update the defaults. I selected the explicit versions based on thecpe/24.11defaults, including the default forrocm,rocm/6.2.4.libfabricmodule fromcraygnuamdgpu. On Feb 18, the defaultlibfabricwill change fromlibfabric/1.20.1tolibfabric/1.22.0, but the latter is not yet available on Frontier. Only the default version on a given computer is officially supported by HPE. I removed thelibfabricmodule fromcraygnuamdgpuso that it will stay with the default when it changes.libfabric/1.15.2.0tolibfabric/1.20.1forcrayclang-scream. On Feb 18,libfabric/1.15.2.0will disappear from Frontier. And the new default,libfabric/1.22.0, does not support Cray MPI versions beforecray-mpich/8.1.28. Becausecrayclang-screamis stuck back atcpe/22.12andcray-mpich/8.1.26, we have to uselibfabric/1.20.1. We previously switched fromlibfabric/1.20.1tolibfabric/1.15.2.0because the former had a serious performance regression. That is supposed to be fixed. And, yes, this does mean thatcrayclang-screamwill be using a version oflibfabricthat isn't officially supported with thelibfabric/1.22.0drivers, mais c'est la vie.I ran an 8-node test of NE30 on Frontier and on our internal system (for
libfabric/1.22.0) to confirm that the changes work and probably don't seriously impact performance. Here are timing results as reported ine3sm_timing.t.*.cperocmlibfabricCPM:ATM_RUNcrayclang-screamcraygnuamdgpuIt would probably be good to confirm with NE1024 runs.