EAMxx: avoid MPI tags in GridImportExport gather/scatter methods #7007
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…MPI tags * Tags were 1-1 with grid GIDs, which at large resolutions can exceed the TAG limit for certain MPI distributions * Use a pack/unpack approach to send a single msg per pid
trey-ornl
reviewed
Feb 13, 2025
|
I ran an ne30 test with output remapped to ne4, and cmp against what I get with master, and the two are BFB. This PR is ready to integrate upon review. |
|
I was finally able to verify that this fix allows a ne1024 case to run one day using Amazing work to find/fix this so quickly! |
tcclevenger
approved these changes
Feb 17, 2025
tcclevenger
approved these changes
Feb 17, 2025
|
The only CI fails are unrelated, and are currently being dealt with. Merging. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes an issue encountered during testing of PR #6977 (see this comment), where the tags we used went above the max tag allowed by the MPI distribution
[BFB]
Instead of sending one msg per grid point, we follow the common pack/unpack paradigm, and send/recv a single message per PID, allowing us to just always use 0 as a tag. It's worth reminding that the gather/scatter routines are used only during initialization, to set up remappers weights (we read map files "evenly" across ranks, and then ship the CRS matrix triplets to the ranks that need them).
@trey-ornl This PR should fix the tags issue. The unit tests for this struct pass on my laptop, but I won't claim it works until I run some tests where coarsening remappers are used (our CI runs ne4 only, so no coarsened output). But if you want to give it a shot, you could try to integrate it on the branch of the Frontier software update, and give it a try on Frontier...
@tcclevenger I don't know if you've ever taken a look at this part of the code. Let me know if you're not familiar with it, and you'd rather walk through it together.