-
Notifications
You must be signed in to change notification settings - Fork 467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change Pytorch materializer to support remote storage #525
Conversation
…using fileio context manager
…ch-materializer-remote-storage
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you have tested it works on remote backends then I'm happy to approve. Would be cool if you could test on GCP and AWS at least and Azure would be a bonus :-)
@htahir1 I have added an other save in pytorch materializer to save entire_model and checkpoint. using the entire model cause error in inference (something related to torch.load) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great changes. I left a tiny comment, but other than that everything looks good to me!
) as f: | ||
torch.save(model.state_dict(), f) | ||
|
||
else: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we don't need to do anything here, could we perhaps remove the last two lines?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes sure, it's removed!
Describe changes
change PyTorch materializer input & output handler to support remote storage using zenml.io fileio context manager.
this will allow saving model artifacts directly to any storage system.
I have added another save for the model checkpoint which is essential for the inference phase.
Pre-requisites
Please ensure you have done the following:
Types of changes