Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tune] [PBT] [Doc] Add example PBT notebook #28519

Merged
merged 23 commits into from
Oct 4, 2022
Merged
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
d4ffcde
Add better PBT logging for exploit, explore
justinvyu Sep 3, 2022
d227adc
Simplify perturb logic
justinvyu Sep 6, 2022
dfef665
Add operation tracking and logging for PBT perturbs
justinvyu Sep 7, 2022
de064ab
[Debug] Temporary fix for PBT checkpoint setting and loading
justinvyu Sep 3, 2022
b07f4aa
Add forced checkpoint logic for PBT
justinvyu Sep 7, 2022
aff2bb4
Add example notebook walking through paper toy example
justinvyu Sep 8, 2022
ea54f45
Add animation, 4 trial expeirment, and more explanations to notebok
justinvyu Sep 9, 2022
421dd03
Add 4 trial gif, separate make_animation function
justinvyu Sep 9, 2022
6077337
Add open in colab button
justinvyu Sep 9, 2022
8521074
Add __init__.py to pbt_visualization doc module
justinvyu Sep 13, 2022
f08ccfe
Merge branch 'master' of https://github.com/ray-project/ray into pbt_…
justinvyu Sep 14, 2022
21eac42
Rerun 2 trial PBT to generate a better visual
justinvyu Sep 14, 2022
0bdd8c8
Clean-up tune examples TOC into sub-sections + add PBT notebook into TOC
justinvyu Sep 14, 2022
9726837
Add test for PBT mutations logging + fix for empty `hyperparam_mutati…
justinvyu Sep 27, 2022
550fd44
Fix some wording in the notebook
justinvyu Sep 27, 2022
1aa30c3
Merge branch 'master' of https://github.com/ray-project/ray into pbt_…
justinvyu Sep 27, 2022
d5f6357
Revert commits related to forced checkpoint
justinvyu Sep 27, 2022
b9763aa
Fix failing `_exploit` tests
justinvyu Sep 28, 2022
03e6d17
Improve pbt example notebook explanations (mention async behavior), a…
justinvyu Sep 29, 2022
f965de7
Clean up references to the pbt example notebook
justinvyu Sep 29, 2022
df6940b
Remove missing reference to utils file
justinvyu Sep 29, 2022
65ecf6d
Improve documentation and typing hints + fix shift noop case
justinvyu Sep 30, 2022
755bebf
Add assertion for matching keys in summarize_hyperparam_changes
justinvyu Sep 30, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[Debug] Temporary fix for PBT checkpoint setting and loading
Signed-off-by: Justin Yu <justinvyu@berkeley.edu>
justinvyu committed Sep 8, 2022
commit de064ab68753f9eb1a04ac0a147fdabad1ef56a9
16 changes: 11 additions & 5 deletions python/ray/tune/execution/checkpoint_manager.py
Original file line number Diff line number Diff line change
@@ -100,11 +100,17 @@ def newest_persistent_checkpoint(self):
@property
def newest_checkpoint(self):
"""Returns the newest checkpoint (based on training iteration)."""
newest_checkpoint = max(
[self.newest_persistent_checkpoint, self.newest_memory_checkpoint],
key=lambda c: c.id,
)
return newest_checkpoint
# NOTE: For PBT Debugging purposes only.
# TODO(justinvyu): Remove this.
# print("[DEBUGGING] newest_memory_checkpoint.id = ", self.newest_memory_checkpoint.id)
# print("[DEBUGGING] newest_persistent_checkpoint.id = ", self.newest_persistent_checkpoint.id)

# newest_checkpoint = max(
# [self.newest_memory_checkpoint, self.newest_persistent_checkpoint],
# key=lambda c: c.id,
# )
# return newest_checkpoint
return self.newest_memory_checkpoint

@property
def newest_memory_checkpoint(self):