-
Notifications
You must be signed in to change notification settings - Fork 6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[tune/internal] Move signal handling into separate method #31004
[tune/internal] Move signal handling into separate method #31004
Conversation
Signed-off-by: Kai Fricke <kai@anyscale.com>
python/ray/tune/tune.py
Outdated
@@ -127,6 +127,39 @@ def _report_progress( | |||
reporter.report(trials, done, sched_debug_str, executor_debug_str) | |||
|
|||
|
|||
def _setup_signal_catching() -> threading.Event: | |||
original_handler = signal.getsignal(signal.SIGINT) | |||
stop_event = threading.Event() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
more specific name? tune_loop_interrupted_event
or something alike?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you're ok with it I'd like to keep this, as we set up general signal catching here and not just the event. Let me know if you disagree!
Meanwhile I'll kick of tune cloud release tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh I mean the event name can be more specific? The method name _setup_signal_catching
sounds fine to me!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah I see! Yes, thanks, will updated!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Thanks!
Signed-off-by: Kai Fricke <kai@anyscale.com>
…t#31004) This PR improves the tune.py code structure: - Signal handling is set up in a separate method - Signal handling now uses a threading event instead of a global dict - The GPU check is moved to before a trial runner is instantiated (and thus restored) in order to fail faster if fail conditions are met - Trial executor creation is moved closer to where the trial executor is passed Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
This PR improves the tune.py code structure: - Signal handling is set up in a separate method - Signal handling now uses a threading event instead of a global dict - The GPU check is moved to before a trial runner is instantiated (and thus restored) in order to fail faster if fail conditions are met - Trial executor creation is moved closer to where the trial executor is passed Signed-off-by: Kai Fricke <kai@anyscale.com>
…t#31004) This PR improves the tune.py code structure: - Signal handling is set up in a separate method - Signal handling now uses a threading event instead of a global dict - The GPU check is moved to before a trial runner is instantiated (and thus restored) in order to fail faster if fail conditions are met - Trial executor creation is moved closer to where the trial executor is passed Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: tmynn <hovhannes.tamoyan@gmail.com>
Signed-off-by: Kai Fricke kai@anyscale.com
Why are these changes needed?
This PR improves the tune.py code structure:
Related issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.