Support for Kubernetes (with Argo)

[nlaille]

Another implementation of #16

This idea is to provide metaflow with native kubernetes implementation using Argo (https://github.com/argoproj/argo) for the workflow part.

[valayDave]

@nlaille : I was recently exploring building plugin support for Argo but the DAG templating seemed a little complicated to fit into Metaflow within a short amount of time.

So meanwhile I have built a plugin to support Kubernetes with isolated self packaged deployments of the runtime itself to the cluster. It will require a service-based metadata provider. It will deploy the entire workflow's runtime as a job that will orchestrate different containerized jobs. So deploying your entire flow with its container orchestration on Kubernetes becomes one command: python myflow.py --with kube:cpu=2,memory=2000,image=python:3.7 kube-deploy run --custom_param_1 hello --custom_param_2 100

It works using the @kube decorator. Supports the same features as the batch plugin. You can even use the --with kube:cpu=2,memory=2000,image=tensorflow/tensorflow:latest-py3.

https://github.com/valayDave/metaflow/tree/kube_cpu_stable

Documentation is in the links within that repo. The plugin is under constant development so there will be more features soon and hopefully even support Argo. But it is currently tied to S3 as its datastore and doesn't support GPUs.

Hope this helps!

[repocho]

Is there a plan to include it in Metaflow official repository?

[alexec]

Argo team would be happy to help make this a reality.

[savingoyal]

@alexec That would be wonderful! Let me email you to set up some time.

[alexec]

You can book 30m: https://calendly.com/argoproj/30m

[repocho]

Ohhh my good. Thanks for this collaboration @alexec and @savingoyal !!!

[terrytangyuan]

Is there any update on this? BTW, the URL for Argo Workflows has changed and the old URL does not redirect correctly. Here's the new location: https://github.com/argoproj/argo-workflows

[alexec]

This is an extremely popular issue. It'd be great to see it implemented.

[savingoyal]

If you would like to try out and give feedback on our Kubernetes integration, please reach out at http://slack.outerbounds.co

[terrytangyuan]

@savingoyal Looks like the current support for K8s is added via K8s Python API and noticed the following the doc:

Scheduling Metaflow flows on Argo
This functionality is in-flight!

How's the integration with Argo Workflows coming along?

[terrytangyuan]

FYI we now have a low-level Python SDK officially maintained by the Argo Workflows team. It should be much easier to use it to integrate Metaflow with Argo Workflows. https://github.com/argoproj/argo-workflows/tree/master/sdks/python

[savingoyal]

#992 is now available for review

[savingoyal]

#992 has been merged. The next Metaflow release will address this issue.

[alexec]

Yay!

[terrytangyuan]

Amazing work! Looking forward to the new release!

[repocho]

Thank you for making this possible !! 🥇

[savingoyal]

Support for Kubernetes and Argo Workflows is now available in Metaflow 2.6.0 - https://blog.argoproj.io/human-centric-data-science-on-kubernetes-with-metaflow-7f60aad34cba