Add Kubeflow tensorboard viz and fix tensorflow file IO for cloud back-ends #447

stefannica · 2022-03-07T08:30:10Z

Describe changes

Enable the tensorboard visualization for Kubeflow and fixes the tensorflow file IO support when shared cloud storage like AWS S3 is used.

fix the kubeflow container entrypoint to consider ZenML ModelArtifacts in addition to TFX ModelRun types when enabling the Tensorboard view
when tensorflow is installed, TFX activates the tensorflow file IO plugin with a priority higher than the ZenML IO plugin. Use a higher priority for the ZenML file IO plugin
the 2.6 tensorflow release moves the IO support into a separate python package tensorflow_io that must be installed and imported explicitly for the S3/GCP/Azure FS filesystem schemes to be supported
fix the Kubeflow Metadata Store implementation
suppress printing locals in rich trace logs if not in debug mode to reduce verbosity
implement a TensorboardService to track and manage locally running Tensorboard daemons
implement a step TensorboardVisualizer that starts a local Tensorboard daemon to visualize the entire history of a model logged by a step
change the kubeflow example to use Tensorflow instead of scikit-learn and make use of Tensorboard visualizations

Pre-requisites

Please ensure you have done the following:

I have read the CONTRIBUTING.md document.
If my change requires a change to docs, I have updated the documentation accordingly.
If I have added an integration, I have updated the integrations table.
I have added tests to cover my changes.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Other (add details above)

* use fixed tensorflow version * reorganize code structure

* when tensorflow is installed, TFX activates its file IO plugin with a priority higher than the ZenML IO plugin * the 2.6 tensorflow release moves the IO support into a separate python package (tensorflow_io) that must be installed and imported explicitly for the S3/GCP/Azure FS schemes to be supported

htahir1

Thank you for taking this over Stefan! Really like it!

I would have some suggestions:

Potentially adding this example as a file in the kubeflow example to showcase the tensorboard integration. I dont believe this deserves an extra example unless we have a native tensorboard integration that works in a magical way for different configurations
Adding a picture of the 'Launch Tensorboard' button on the Kubeflow UI for users to understand the true value is when this is done on Kubeflow. This can be put on the examples/kubeflow/README.md in a subheading where we explain the kubeflow-tensorboard integration we just enabled
Adding in the limitations of Kubeflow (e.g. with S3) as you decribed in the JIRA ticket
Potentially adding a 'TensorboardVisualizer' and showcasing launching tensorboard in a post-execution workflow. This should be trivial but exemplrary

I can also implement some of the above if you give me the go-ahead. Otherwise, its its clear I would be glad to review these changes! Thanks again for doing the deep dive!

* use Kubeflow volume metadata support [1] to mount the local artifact store as a hostpath volume in the Kubeflow UI and Tensorboard pods [1] https://github.com/kubeflow/pipelines/blob/master/docs/config/volume-support.md

stefannica · 2022-03-08T21:18:05Z

Thanks @htahir1 , I hope I've addressed all your suggestions with the last commit (see inline).

Thank you for taking this over Stefan! Really like it!

I would have some suggestions:

Potentially adding this example as a file in the kubeflow example to showcase the tensorboard integration. I dont believe this deserves an extra example unless we have a native tensorboard integration that works in a magical way for different configurations

I merged the Tensorboard and Kubeflow examples. This basically meant switching the existing Kubeflow example from sklearn to tensorflow and adding some post-execution visualizations.

Adding a picture of the 'Launch Tensorboard' button on the Kubeflow UI for users to understand the true value is when this is done on Kubeflow. This can be put on the examples/kubeflow/README.md in a subheading where we explain the kubeflow-tensorboard integration we just enabled

Done. There are new pics and new information regarding Tensorboard in that README.md file now.

Adding in the limitations of Kubeflow (e.g. with S3) as you decribed in the JIRA ticket

No need. I fixed all the limitations :)

Potentially adding a 'TensorboardVisualizer' and showcasing launching tensorboard in a post-execution workflow. This should be trivial but exemplrary

You're going to love this: I implemented a TensorboardService to wrap around the Tensorboard local server, and a visualizer that makes it easy to start/stop Tensorboard services. This goes along with the idea that Services are for more than just model serving. The visualizer also works with Jupyter notebooks, which reminds me: I also updated the notebook in the kubeflow example to reflect this.

…rd examples * Tensorboard service to track Tensorboard daemons running locally * Tensorboard visualizer to start Tensorboard service for a pipeline step * switched Kubeflow example to use Tensorflow instead of sklearn * add more pics and info about the Tensorboard service and the Kubeflow Tensorboard UI

src/zenml/integrations/kubeflow/orchestrators/local_deployment_utils.py

schustmi

Nice! Just found a few typos and a tiny amount of duplicated code, other than that good to go

src/zenml/integrations/kubeflow/orchestrators/local_deployment_utils.py

src/zenml/integrations/tensorflow/visualizers/tensorboard_visualizer.py

* removed duplicated code in Tensorboard visualizer

…_utils.py Co-authored-by: Michael Schuster <schustmi@users.noreply.github.com>

htahir1

Are you kidding here!!!! Whatttt the heck just happened. AMAZING WORK! I absolutely love the new Tensorboard service and can only be astonished at the effort put in here. Kudos to you and LGTM!!!!

htahir1

Ah wait I just noticed one TINY problem. No updates to the docs. Would really love to have some of this stuff on the docs, including the service you implemented

…d-tensorboard

htahir1 · 2022-03-10T14:19:45Z

I have made the following changes to this branch.

Added docs
Added the following error to make sure kubeflow orchestrator does not fail non gracefully

@stefannica and @schustmi please do take a look

stefannica

Nice job with the docs, I like it.

examples/kubeflow/README.md

schustmi · 2022-03-11T08:22:44Z

@htahir1 The only thing i'd change is to not use an AssertionError. Maybe a RuntimeError instead?

htahir1 and others added 7 commits March 7, 2022 13:58

Added tensorboard example

3a71887

Container entrypoint updated to use tensorboard if right model type

e038ecb

Fix tensorboard example

0f471eb

* use fixed tensorflow version * reorganize code structure

Don't print locals with rich traces unless in debug mode

3d9fe5c

Fix linter errors

2bbe2bf

Updated tensorboard example

2806dc5

stefannica force-pushed the feature/ENG-496-add-tensorboard branch from 5cbcf49 to a2ff7cf Compare March 7, 2022 13:00

stefannica marked this pull request as ready for review March 7, 2022 13:02

stefannica requested review from schustmi and htahir1 March 7, 2022 13:02

stefannica added the internal To filter out internal PRs and issues label Mar 7, 2022

Add tensorboard example to integration tests

dfebffa

stefannica force-pushed the feature/ENG-496-add-tensorboard branch from a2ff7cf to dfebffa Compare March 7, 2022 13:05

htahir1 requested changes Mar 7, 2022

View reviewed changes

stefannica added 2 commits March 8, 2022 11:54

Fix Kubeflow metadata store implementation

7c20e0d

Fix Kubeflow Tensorboard visualizer with local artifact

2261c61

* use Kubeflow volume metadata support [1] to mount the local artifact store as a hostpath volume in the Kubeflow UI and Tensorboard pods [1] https://github.com/kubeflow/pipelines/blob/master/docs/config/volume-support.md

stefannica force-pushed the feature/ENG-496-add-tensorboard branch from 402cd86 to 86653c0 Compare March 8, 2022 21:26

stefannica force-pushed the feature/ENG-496-add-tensorboard branch from 86653c0 to 7010889 Compare March 8, 2022 22:14

stefannica added 2 commits March 9, 2022 09:26

Fix tensorboard visualizer on windows

3d2bc58

Fix local artifact store detection in Kubeflow

e2b0feb

stefannica force-pushed the feature/ENG-496-add-tensorboard branch from 61fbeec to 03f04f1 Compare March 9, 2022 10:20

schustmi reviewed Mar 9, 2022

View reviewed changes

src/zenml/integrations/kubeflow/orchestrators/local_deployment_utils.py Outdated Show resolved Hide resolved

schustmi requested changes Mar 9, 2022

View reviewed changes

stefannica added 2 commits March 9, 2022 14:56

Fix windows integration test cases

db2c7cc

Addressed code review

2e772cb

* removed duplicated code in Tensorboard visualizer

stefannica force-pushed the feature/ENG-496-add-tensorboard branch from 03f04f1 to 2e772cb Compare March 9, 2022 14:05

Update src/zenml/integrations/kubeflow/orchestrators/local_deployment…

5c93f68

…_utils.py Co-authored-by: Michael Schuster <schustmi@users.noreply.github.com>

Update src/zenml/integrations/kubeflow/orchestrators/local_deployment…

467a59a

…_utils.py Co-authored-by: Michael Schuster <schustmi@users.noreply.github.com>

schustmi approved these changes Mar 10, 2022

View reviewed changes

htahir1 approved these changes Mar 10, 2022

View reviewed changes

htahir1 reviewed Mar 10, 2022

View reviewed changes

htahir1 added 4 commits March 10, 2022 14:31

Added kubeflow notebook error message

f133278

Made small changes to example README

412e9e4

Merge remote-tracking branch 'origin/develop' into feature/ENG-496-ad…

0320de5

…d-tensorboard

added docs

18c39fb

stefannica commented Mar 10, 2022

View reviewed changes

examples/kubeflow/README.md Show resolved Hide resolved

fixed a nav bar thing [ci skip]

be188a3

Changed to RuntimeError [ci skip]

8769f25

htahir1 merged commit 74a0bc8 into develop Mar 11, 2022

htahir1 deleted the feature/ENG-496-add-tensorboard branch March 11, 2022 09:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Kubeflow tensorboard viz and fix tensorflow file IO for cloud back-ends #447

Add Kubeflow tensorboard viz and fix tensorflow file IO for cloud back-ends #447

stefannica commented Mar 7, 2022 •

edited

Loading

htahir1 left a comment

stefannica commented Mar 8, 2022

schustmi left a comment

htahir1 left a comment

htahir1 left a comment

htahir1 commented Mar 10, 2022

stefannica left a comment

schustmi commented Mar 11, 2022

Add Kubeflow tensorboard viz and fix tensorflow file IO for cloud back-ends #447

Add Kubeflow tensorboard viz and fix tensorflow file IO for cloud back-ends #447

Conversation

stefannica commented Mar 7, 2022 • edited Loading

Describe changes

Pre-requisites

Types of changes

htahir1 left a comment

Choose a reason for hiding this comment

stefannica commented Mar 8, 2022

schustmi left a comment

Choose a reason for hiding this comment

htahir1 left a comment

Choose a reason for hiding this comment

htahir1 left a comment

Choose a reason for hiding this comment

htahir1 commented Mar 10, 2022

stefannica left a comment

Choose a reason for hiding this comment

schustmi commented Mar 11, 2022

stefannica commented Mar 7, 2022 •

edited

Loading