feat: Upload model after finishing training #826
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #826 +/- ##
=======================================
Coverage 86.40% 86.40%
=======================================
Files 194 196 +2
Lines 15752 15848 +96
=======================================
+ Hits 13610 13693 +83
- Misses 2142 2155 +13 ☔ View full report in Codecov by Sentry. |
dataquality/utils/upload_model.py
Outdated
| return response.status_code, response.text | ||
|
|
||
|
|
||
| def upload_model_to_dq() -> None: |
There was a problem hiding this comment.
i think this fn should take in a model
also you are not upoading the model to dq you are uploading it to object store
There was a problem hiding this comment.
done, model is Any for now since huggingface can have many different model classes
|
I agree with auto. We can just make sure the dq.auto on sagemaker always uploads so you can later use it for inference |
|
@elboy3 Sir, please reconsider (I have added all your requests) |
6a9b93b to
8446fcd
Compare
| def get_uploaded_model_info(self, project_id: UUID4, run_id: UUID4) -> Any: | ||
| """ | ||
| Returns information about the model for a given run. | ||
| Will also update the status to complete. |
There was a problem hiding this comment.
why does it update the status to complete?
also what does it return, the model or a presigned url to download it?
There was a problem hiding this comment.
since we don't know on the backend when a minio upload is completed we update it every time we get the model and if the filename is not saved.
There was a problem hiding this comment.
so what status is it updating, the job status?
There was a problem hiding this comment.
just the upload is completed. at first it's the put link, once it's put we will pull the download link from minio and add it to the entry
pyproject.toml
Outdated
| @@ -13,7 +13,7 @@ readme = "README.md" | |||
| license = {text = 'See LICENSE'} | |||
| requires-python = ">=3.8" | |||
| dependencies = [ | |||
| "pydantic>=2.0.0", | |||
There was a problem hiding this comment.
i don't think you need any of these.... i fixed the tests and all the formatting issues earlier today just merge the latest from main and keep the pyproject.toml the same
tests/loggers/test_seq2seq.py
Outdated
| @@ -647,6 +647,7 @@ def test_create_data_embs_df_custom_column( | |||
|
|
|||
| # Check that no exception is thrown and that data embs are created | |||
| assert "text" not in df.get_column_names() | |||
|
|
|||
|
Added test and reverted pyproject, ready to merge |
This allows inference and auto training
Part of:
https://app.shortcut.com/galileo/story/10988/jpmc-allow-simple-upload-of-csv-to-train-a-model-and-get-dataquality-insights