KFP docker executor doesn't support Kubernetes 1.19 or above

[zijianjoy]

UPDATE 2021-8-16

you can use emissary executor instead starting from Kubeflow Pipelines 1.7, this is the only executor that works, if you don't have Docker runtime in your cluster (Docker runtime is not the default installation option since GKE 1.18).
Documentation: https://www.kubeflow.org/docs/components/pipelines/installation/choose-executor/

issue reference

#5663

Error message:

1. If you deploy KFP on Google Cloud Marketplace, you will see the following message if you select create a cluster

Basic authentication was removed for GKE cluster versions >= 1.19. 
The cluster cannot be created with basic authentication enabled. 
Instructions for choosing an alternative authentication method can be found at: https://cloud.google.com/kubernetes-engine/docs/how-to/api-server-authentication.

2. If you deploy KFP on an existing K8s cluster with version v1.19 or above, you will see following error if you run any pipeline:

This step is in Error state with this message: failed to save outputs: Error response from daemon: No such container: XXXXXX

Workaround

Kubeflow Pipelines docker executor doesn't support containerd nodes which come by default with Kubernetes 1.19 or above.

To deploy KFP, you need to create a cluster satisfying one of the following conditions:

• a K8s cluster with version 1.18 or earlier
• Explicitly using docker nodepools (Container-Optimized OS with Docker (cos)) https://cloud.google.com/kubernetes-engine/docs/concepts/node-images

Example command to create kubernetes cluster:

# The following parameters can be customized based on your needs.

CLUSTER_NAME="kubeflow-pipelines-standalone"
ZONE="us-central1-a"
MACHINE_TYPE="e2-standard-2" 
SCOPES="cloud-platform" # This scope is needed for running some pipeline samples. Read the warning below for its security implication

gcloud container clusters create $CLUSTER_NAME \
     --zone $ZONE \
     --machine-type $MACHINE_TYPE \
     --scopes $SCOPES \
     --cluster-version 1.18

Warning: Using SCOPES="cloud-platform" grants all GCP permissions to the cluster. For a more secure cluster setup, refer to Authenticating Pipelines to GCP.

Install KFP on Google Cloud:
If you are on Google Cloud, install KFP by following this documentation: https://cloud.google.com/ai-platform/pipelines/docs/setting-up#existing-cluster

Install KFP standalone:
If you are on other platform, follow the documentation here to install standalone KFP: https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/#deploying-kubeflow-pipelines

Root cause

Kubeflow Pipelines docker executor doesn't support GKE 1.19 or above, because new GKE 1.19 clusters have containerd nodes by default.

Kubeflow Pipelines need to migrate to new argo version and use emissary executor. tracking issue: #1654

---

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[Bobgy]

/assign

[juliusvonkohout]

I have created an extended description of everything at #5718 . What you describe here is just one single out of many reasons to upgrade argo and use the emissary executor.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zijianjoy]

Kubeflow Pipelines backend v1.8.1 has released. We have switched to Emissary Executor by default. Therefore you can use KFP on K8s 1.19 or above without further configuraiton.