Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge path_utils into fileio & refactor what was left #168

Merged
merged 104 commits into from
Nov 11, 2021
Merged

Merge path_utils into fileio & refactor what was left #168

merged 104 commits into from
Nov 11, 2021

Conversation

alex-zenml
Copy link
Contributor

Pre-requisites

Please ensure you have done the following:

  • I have read the CONTRIBUTING.md document.
  • If my change requires a change to the documentation, I have updated the documentation accordingly.
  • I have added tests to cover my changes.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Describe changes

  • Merged all functions from path_utils.py over into io/fileio.py.
  • Fixed a reference to our own GCS filesystem plugin which was pointing to the old dir structure.

I wasn't able to find the place where we're importing tensorflow. By my investigation, that used to be happening whenever something imported tensorflow_gfile, but we have our own version of that in integrations now which doesn't import `tensorflow.

strickvl and others added 30 commits November 3, 2021 18:33
@strickvl
Merge remote-tracking branch 'origin/main' into alex/ENG-61-pathutils…
2070c7f 
…-refactor
@schustmi
Add caching option to step decorator
6874f54
@strickvl
add paperspace env check
e04ffd3
@strickvl
add metadata annotation
e65d3b8
@schustmi
Merge branch 'main' into michael/ENG-24-caching
85ac0af
@schustmi
Improve step parameter handling
4676b6c
@schustmi
Fix error type in base step parameter handling
d2cbf79
@schustmi
Implement cache disabling on a per step basis
eb9a6d0
@htahir1
Merge pull request #156 from zenml-io/alex/add-paperspace-check
d88737b 
Add Paperspace environment check
@schustmi
Fix existing tests and add some more for step initialization
f35b5be
@htahir1
Merge pull request #157 from zenml-io/michael/ENG-24-caching
4b68e54 
Step caching enabled
@htahir1
Fixed import
008dd3f
@strickvl
refactor out the walk function
45928d4
@strickvl
Merge remote-tracking branch 'origin/main' into alex/ENG-61-pathutils…
d2cdd44 
…-refactor
@strickvl
refactor out the is_root function
72ddc7d
@strickvl
refactor out the isdir function
f7f36bc
@strickvl
add todo for future refactor
f6c41d1
@strickvl
refactor out the find_files function
0f03355
@strickvl
refactor out the is_remote function
4e598ce
@strickvl
add comment about naming convention
119838c
@strickvl
refactor out the is_gcs_path function
ee8ada6
@strickvl
refactor out the list_dir function
e818eaf
@strickvl
refactor out duplicated listdir function
1ceced1
@strickvl
refactor out create_file_if_not_exists
f62479f
@strickvl
refactor out append_file
acaf5bf
@strickvl
refactor out create_dir_if_not_exists
e78d908
@strickvl
refactor out create_dir_recursive_if_not_exists
f64ab35
@strickvl
refactor out resolve_relative_path
e4d7f06
@strickvl
refactor out file_exists
4a1de9b
@strickvl
remove exists function from fileio
ee4a5b8
strickvl and others added 21 commits November 9, 2021 12:09
@strickvl
move below contributing
5e14f2b
@alex-zenml @htahir1
add extra line break
c09555e 
Co-authored-by: Hamza Tahir <htahir111@gmail.com>
@htahir1
Merge pull request #163 from zenml-io/alex/repobeats-location-shift
c3759f7 
Move location of repobeats image
@schustmi
Allow args when creating a pipeline, improve error handling
741cae3
@schustmi
Add tests for pipeline initialization
832b88e
@schustmi
Add missing type annotation
913ca87
@schustmi
Update error message to include BaseStep subclasses as allowed inputs…
e75b250 
… to a pipeline
@htahir1 @gitbook-bot
GitBook: [#548] Added Sam
8ddc30a
@schustmi
Add expected steps/artifacts to error message
3ad9bcb
@htahir1
Added Sam
118ad73
@htahir1
Added Sam
9036f46
@htahir1
Added sam images
f36b910
@htahir1
switched the div and style to tables as github sanitizes css
49f30c1
@schustmi
Extract pipeline and step creation into fixtures, separate pipeline i…
282e118 
…nitialization tests
@htahir1
Merge pull request #165 from zenml-io/hamza/add_sam
e37b461 
Hamza/add sam
@schustmi
Merge pull request #164 from zenml-io/michael/pipeline-with-args
d742be6 
Pipeline initialization with *args
@schustmi
Improve detection of third party modules during class resolving
3fbcf6a
@schustmi
Add test for third party module detection
0e040bc
@schustmi
Merge pull request #167 from zenml-io/michael/third-party-source-dete…
77fc06f 
…ction

Improve detection of third party modules during class resolving
@strickvl
fix class import path
33569a0
@strickvl
Merge remote-tracking branch 'origin/main' into alex/ENG-61-pathutils…
b9babf8 
…-refactor
@alex-zenml alex-zenml added the internal To filter out internal PRs and issues label Nov 11, 2021
@alex-zenml
Copy link
Contributor Author

@htahir1 I'm running into a bunch of issues with the linting that are because Baris' branch isn't finished yet. Maybe we want to have him review this? Not sure how to handle it.

@alex-zenml alex-zenml requested a review from bcdurak November 11, 2021 10:23
htahir1
htahir1 approved these changes Nov 11, 2021
View reviewed changes
Copy link
Contributor

@htahir1 htahir1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing job!

@alex-zenml alex-zenml merged commit fff4986 into baris/ENG-37-standard-import-check Nov 11, 2021
@alex-zenml alex-zenml deleted the alex/ENG-61-pathutils-refactor branch November 11, 2021 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@htahir1 htahir1 htahir1 approved these changes

@schustmi schustmi Awaiting requested review from schustmi

@bcdurak bcdurak Awaiting requested review from bcdurak

Assignees
No one assigned
Labels
internal To filter out internal PRs and issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@alex-zenml @htahir1 @strickvl @schustmi