[Train] LightningTrainer always resumes from the latest AIR checkpoint during restoration. #35617
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
|
@woshiyyya Please make sure to also create the cherry pick PR yourself after this is merged! |
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
…ng_trainer_restoration_ckpt_path
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
gjoliver
reviewed
May 24, 2023
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
…ng_trainer_restoration_ckpt_path
matthewdeng
reviewed
May 25, 2023
| @@ -541,7 +541,15 @@ def _lightning_train_loop_per_worker(config): | |||
|
|
|||
| # Restore from a previously failed run | |||
| checkpoint = session.get_checkpoint() | |||
| if checkpoint and "ckpt_path" not in trainer_fit_params: | |||
| if checkpoint: | |||
| logger.info("Restoring LightningTrainer from a failed run. ") | |||
There was a problem hiding this comment.
Actually is this always true? What happens if the user passes in an AIR checkpoint?
LightningTrainer(..., resume_from_checkpoint=some_checkpoint)
There was a problem hiding this comment.
It might not be a failed run. Updated the log message.
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
matthewdeng
reviewed
May 25, 2023
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
matthewdeng
approved these changes
May 25, 2023
woshiyyya
added a commit
to woshiyyya/ray
that referenced
this pull request
May 25, 2023
…t during restoration. (ray-project#35617) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
8 tasks
scv119
pushed a commit
to scv119/ray
that referenced
this pull request
Jun 16, 2023
…t during restoration. (ray-project#35617) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
…t during restoration. (ray-project#35617) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
If users provide a
ckpt_pathin LightningConfigBuilder.fit_param(), when resuming training from a failed run, LightningTrainer will always resume training fromckpt_pathagain, instead of the latest AIR checkpoint. In that case, the training won't proceed after restored from failures.Code:
ray/python/ray/train/lightning/lightning_trainer.py
Lines 543 to 549 in 0fd06ad
We should always restore from the latest checkpoint from
session.get_checkpoint(), otherwise, the training re-start from the beginning after restoration.Details in #35613.
Related issue number
Close #35613
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.