progression discussion [work in progress] #3640
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We have a progresion issue in openib when we're out of resources and a completion function wants to send another message. Currently the completion function blocks (spins calling progression) expecting its send to happen eventually. But in the out-of-resources case the send is queued and the needed resources won't be freed until we return from our completion function which will allow processing to continue through the rest of the completed work list. I can only see three categories of fix: 1. don't let completion functions block like that 2. allow recursive handle_wc() while already inside a completion function 3. have some reserved resources for completion functions only so they can always complete This is an example implementation of open-mpi#2. This includes a conditional recursion that gets less frequent the deeper the call stack becomes. The biggest risk I see for open-mpi#2 is that as written there is no longer a strict ordering of completion functions. They would be initiated in the same order they are seen in ibv_poll_cq() but you can't be sure WC1.cbfunc has finished before WC2.cbfunc starts. I'm not sure if that's a requirement for correctness or not. Signed-off-by: Mark Allen <markalle@us.ibm.com>
|
The issue report around this PR including a testcase and some discussion is here: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have a progresion issue in openib when we're out of resources and
a completion function wants to send another message.
Currently the completion function blocks (spins calling progression)
expecting its send to happen eventually. But in the out-of-resources case
the send is queued and the needed resources won't be freed until we
return from our completion function which will allow processing to
continue through the rest of the completed work list.
I can only see three categories of fix:
This is an example implementation of #2. This includes a conditional
recursion that gets less frequent the deeper the call stack becomes.
The biggest risk I see for #2 is that as written there is no longer a
strict ordering of completion functions. They would be initiated in the
same order they are seen in ibv_poll_cq() but you can't be sure WC1.cbfunc
has finished before WC2.cbfunc starts. I'm not sure if that's a requirement
for correctness or not.
Signed-off-by: Mark Allen markalle@us.ibm.com