Enable local test runner for Kubeflow Pipelines

[neuromage]

Today, there is no way to run a KFP pipeline on a local machine (outside of running a mini Kubernetes cluster locally). This makes testing pipelines for user difficult as they always have to go through the steps of uploading to a cluster to run even small tests on the correctness of the pipeline.

To solve this problem, it would be nice if KFP offers a way to run pipelines locally. Such a mode does not have to have full fidelity with what's possible in a cluster run with Argo to begin with. A simple approach of using docker run to run each step sequentially, with a mounted local volume for passing parameters, will go a long way towards satisfying most local-run use-cases.

As a suggestion, we can modify the pipeline decorator to enable this behaviour when the user adds a keyword like test = True, i.e.

@dsl.Pipeline(
   name = "..",
   test = True,
   ...

[issue-label-bot]

Issue-Label Bot is automatically applying the label feature_request to this issue, with a confidence of 0.98. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: dashboard, app homepage and code for this bot.

[vicaire]

Some ideal requirements to consider:

• The local executor should support all the same features as execution within a GKE cluster.
• The local executor should not duplicate the code of the controllers used to execute the pipeline.
• The local executor should execute the yaml, not Python, so that both yaml and Python can be tested.

[Ark-kun]

The local executor should support all the same features as execution within a GKE cluster.

I guess only the proper local kubernetes cluster can satisfy this requirement. The test environment needs Minikube installed.

[Ark-kun]

As a suggestion, we can modify the pipeline decorator to enable this behaviour when the user adds a keyword like test = True, i.e.

I think it's better have a function to perform this instead of overloading the decorator:

kfp.run_pipeline_locally(my_pipeline, arguments={...})

If we want to differentiate, we can have

• run_pipeline_locally
• run_pipeline_using_docker
• run_pipeline_on_kubernetes
• run_pipeline_on_kfp

[vicaire]

BTW, what about providing users with a VM image that has all the tools installed (minikube and etc.) to make local execution easy?

[ucdmkt]

Thank you so much for tracking this issue.

Another thing to consider is that we may want to have a way to inject a stub to a component inside the pipeline, if a component is making call to foreign services such as Dataflow.

[kevinbache]

related: #1104

[neuromage]

Closing as infeasible/obsolete for now.

[neuromage]

@Ark-kun not sure if you intentionally meant to re-open this issue? Are you working on local runner for KFP?

[Ark-kun]

@Ark-kun not sure if you intentionally meant to re-open this issue? Are you working on local runner for KFP?

I think that your idea is still pretty valid and useful. It would be nice to have that feature for component testing and experimentation. I've recently started creating components in my free time and testing them was not that easy without having a Kubernetes cluster.
Our SDK already runs some tests in a local environment, but it would be less hacky to just use docker.
So I just wanted to keep this feature request in mind.
The priority of this is not P0 or P1 of course.

What do you think?

[rmgogogo]

FYI, I'm thinking on bring a lightweight local runner in a new design. It may rely on TFX SDK's docker launcher. It's still early investigation phase, together with IR work.

[numerology]

FYI, for testing TFX has this tensorflow/tfx#1986 WIP.

That said it's pretty different than local dev.

[rmgogogo]

(in IR based impl, we may handle it with Docker runner)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

/lifecycle frozen
we'd still want this open

[NikeNano]

Are we interested in adding instructions on how to deploy locally to Minikube based upon the local code for development @Bobgy? Will not be a big change but would require the user to build all the images and add an overlay that uses these images instead of the official released images.

[Bobgy]

@NikeNano I think that's a different topic, this issue is about creating a local runner that helps experiment with KFP. Rather than developing KFP.

[lynnmatrix]

@Bobgy @Ark-kun @numerology I have create a PR(#4983) trying to provide a local runner which will run Kubeflow pipeline on docker or locally.
As @Ark-kun suggested, kfp.run_pipeline_func_locally(my_pipeline, arguments={...}) is picked.