Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

task proxy: reduce unnecessary rtconfig deepcopy on submit #1810

Merged

Conversation

benfitzpatrick
Copy link
Contributor

This change reduces the time taken to submit a minimal task by around 40% in a modified 'busy' suite (would be more for larger tasks?) on a fast filesystem. (When the filesystem is slow, more of the time will be taken in writing the job script file).

For extremely active suites (100s of 1 second tasks active at once) the pdeepcopy and poverride here can be one of the dominant costs. I find it amazing how much time it takes just to move or re-organise data structures!

The rtconfig is chucked away in the set_from_rtconfig method that follows, so I don't think this saves memory in the long run.

The speedup won't apply if there are broadcast overrides, so some of our larger suites that use broadcasts of reservation ids for each task, etc, won't benefit.

@arjclark, @matthewrmshin please review.

@benfitzpatrick benfitzpatrick added this to the next-release milestone Apr 26, 2016
@arjclark
Copy link
Contributor

Looks OK to me, test battery passing in my environment. @matthewrmshin can you double check this isn't going to break something further on in the task proxy code?

@arjclark arjclark assigned matthewrmshin and unassigned arjclark Apr 26, 2016
@matthewrmshin matthewrmshin merged commit 2464bc0 into cylc:master Apr 26, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants