task proxy: reduce unnecessary rtconfig deepcopy on submit #1810
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change reduces the time taken to submit a minimal task by around 40% in a modified 'busy' suite (would be more for larger tasks?) on a fast filesystem. (When the filesystem is slow, more of the time will be taken in writing the job script file).
For extremely active suites (100s of 1 second tasks active at once) the pdeepcopy and poverride here can be one of the dominant costs. I find it amazing how much time it takes just to move or re-organise data structures!
The rtconfig is chucked away in the
set_from_rtconfig
method that follows, so I don't think this saves memory in the long run.The speedup won't apply if there are broadcast overrides, so some of our larger suites that use broadcasts of reservation ids for each task, etc, won't benefit.
@arjclark, @matthewrmshin please review.