Pull kubeflow pipelines manifest from upstream using kpt

hack/pull_kfp_upstream.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+
+export PIPELINES_SRC_REPO=https://github.com/kubeflow/pipelines.git
+export PIPELINES_VERSION=1.0.0-rc.1
+# Pulling for the first time
+# kpt pkg get $PIPELINES_SRC_REPO/manifests/kustomize@$PIPELINES_VERSION pipeline/upstream
+
+# Updates
+kpt pkg update pipeline/upstream/@$PIPELINES_VERSION --strategy force-delete-replace

pipeline/upstream/Kptfile

@@ -0,0 +1,11 @@
+apiVersion: kpt.dev/v1alpha1
+kind: Kptfile
+metadata:
+    name: upstream
+upstream:
+    type: git
+    git:
+        commit: 2384d8f1c6083b4ec5c144a2fb6e247bcbe33b05
+        repo: https://github.com/kubeflow/pipelines
+        directory: /manifests/kustomize
+        ref: 1.0.0-rc.1

pipeline/upstream/OWNERS

@@ -0,0 +1,8 @@
+approvers:
+  - Bobgy
+  - IronPan
+  - rmgogogo
+reviewers:
+  - Bobgy
+  - IronPan
+  - rmgogogo

pipeline/upstream/README.md

@@ -0,0 +1,100 @@
+# Install Kubeflow Pipelines
+
+This folder contains Kubeflow Pipelines Kustomize manifests for a light weight
+deployment. You can follow the instruction and deploy Kubeflow Pipelines in an
+existing cluster.
+
+To install Kubeflow Pipelines, you have several options.
+- Via [GCP AI Platform UI](http://console.cloud.google.com/ai-platform/pipelines).
+- Via an upcoming commandline tool.
+- Via Kubectl with Kustomize, it's detailed here.
+  - Community maintains a repo [here](https://github.com/e2fyi/kubeflow-aws/tree/master/pipelines) for AWS.
+
+## Install via Kustomize
+
+Deploy latest version of Kubeflow Pipelines.
+
+It uses following default settings.
+- image: latest released images
+- namespace: kubeflow
+- application name: pipeline
+
+### Option-1 Install it to any K8s cluster
+It's based on in-cluster PersistentVolumeClaim storage.
+
+```
+kubectl apply -k cluster-scoped-resources/
+kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s
+kubectl apply -k env/platform-agnostic/
+kubectl wait applications/pipeline -n kubeflow --for condition=Ready --timeout=1800s
+kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80
+```
+Now you can access it via localhost:8080
+
+### Option-2 Install it to GCP with in-cluster PersistentVolumeClaim storage
+It's based on in-cluster PersistentVolumeClaim storage.
+Additionally, it introduced a proxy in GCP to allow user easily access KFP safely.
+
+```
+kubectl apply -k cluster-scoped-resources/
+kubectl wait crd/applications.app.k8s.io --for condition=established --timeout=60s
+
+kubectl apply -k env/dev/
+kubectl wait applications/pipeline -n kubeflow --for condition=Ready --timeout=1800s
+
+# Or visit http://console.cloud.google.com/ai-platform/pipelines
+kubectl describe configmap inverse-proxy-config -n kubeflow | grep googleusercontent.com
+```
+
+### Option-3 Install it to GCP with CloudSQL & GCS-Minio managed storage
+Its storage is based on CloudSQL & GCS. It's better than others for production usage.
+
+Please following [sample](sample/README.md) for a customized installation.
+
+## Uninstall
+
+If the installation is based on CloudSQL/GCS, after the uninstall, the data is still there,
+reinstall a newer version can reuse the data.
+
+```
+### 1. namespace scoped
+# Depends on how you installed it:
+kubectl kustomize env/platform-agnostic | kubectl delete -f -
+# or
+kubectl kustomize env/dev | kubectl delete -f -
+# or
+kubectl kustomize env/gcp | kubectl delete -f -
+# or
+kubectl delete applications/pipeline -n kubeflow
+
+### 2. cluster scoped
+kubectl delete -k cluster-scoped-resources/
+```
+
+## Troubleshooting
+
+### Permission error installing Kubeflow Pipelines to a cluster
+
+Run
+
+```
+kubectl create clusterrolebinding your-binding --clusterrole=cluster-admin --user=[your-user-name]
+```
+
+### Samples requires "user-gcp-sa" secret
+
+If sample code requires a "user-gcp-sa" secret, you could create one by
+
+-   First download the GCE VM service account token
+    [Document](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys)
+
+```
+gcloud iam service-accounts keys create application_default_credentials.json \
+  --iam-account [SA-NAME]@[PROJECT-ID].iam.gserviceaccount.com
+```
+
+-   Run
+
+```
+kubectl create secret -n [your-namespace] generic user-gcp-sa --from-file=user-gcp-sa.json=application_default_credentials.json`
+```

pipeline/upstream/STRUCTURE.md

@@ -0,0 +1,49 @@
+# Kubeflow Pipelines Kustomize Manifest Folder Structure
+
+Folder structure:
+```
+kustomize
+├── cluster-scoped-resources
+│   ├── README.md (explains this folder structure)
+│   └── kustomization.yaml (lists all cluster-scoped folders in base/)
+├── base
+│   ├── cache-deployer
+│   │   ├── cluster-scoped (not included in cache-deployer's kustomization.yaml)
+│   │   │   ├── clusteroles
+│   │   │   └── clusterrolebindings
+|   |   └── ... (namespace scoped)
+│   ├── argo
+│   │   ├── cluster-scoped
+│   │   │   └── workflow-crd.yaml
+|   |   └── ... (namespace scoped)
+│   ├── application
+│   │   ├── cluster-scoped
+│   │   │   ├── application-crd.yaml
+│   │   │   └── ...
+|   |   └── ... (namespace scoped)
+│   ├── pipeline
+│   │   ├── cluster-scoped
+│   │   │   ├── viewer-crd.yaml
+│   │   │   ├── scheduledworkflow-crd.yaml
+│   │   │   └── ...
+|   |   └── ... (namespace scoped)
+│   └── ...
+└── env
+    ├── platform-agnostic
+    │   └── kustomization.yaml (based on "base")
+    ├── dev
+    │   └── kustomization.yaml (based on "env/platform-agnostic")
+    └── gcp
+        └── kustomization.yaml (based on "base")
+```
+
+* User facing manifest entrypoint is `cluster-scoped-resources` package and `env/<env-name>` package.
+    * `cluster-scoped-resources` should collect all cluster-scoped resources.
+    * `env/<env-name>` should collect env specific namespace scoped resources.
+* Universal components live in `base/<component-name>` folders.
+    * If a component requires cluster-scoped resources, it should have a folder inside named `cluster-scoped` with related resources, but note that `base/<component-name>/kustomization.yaml` shouldn't include the `cluster-scoped` folder. `cluster-scoped` folders should be collected by top level `cluster-scoped-resources` folder.
+* Env specific overlays live in `env/<env-name>` folders.
+
+Constraints we need to comply with (that drove above structure):
+* CRDs must be applied separately, because if we apply CRs in the same `kubectl apply` command, the CRD may not have been accepted by k8s api server (e.g. Application CRD).
+* [A Kubeflow 1.0 constraint](https://github.com/kubeflow/pipelines/issues/2884#issuecomment-577158715) is that we should separate cluster scoped resources from namespace scoped resources, because sometimes different roles are required to deploy them. Cluster scoped resources usually need a cluster admin role, while namespaced resources can be deployed by individual teams managing a namespace.

pipeline/upstream/base/application/application-controller-deployment.yaml

@@ -0,0 +1,38 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: controller-manager
+  labels:
+    control-plane: controller-manager
+    controller-tools.k8s.io: "1.0"
+spec:
+  selector:
+    matchLabels:
+      control-plane: controller-manager
+      controller-tools.k8s.io: "1.0"
+  template:
+    metadata:
+      labels:
+        control-plane: controller-manager
+        controller-tools.k8s.io: "1.0"
+    spec:
+      containers:
+        - command:
+            - /root/manager
+          # A customized image with https://github.com/kubernetes-sigs/application/pull/127
+          image: gcr.io/ml-pipeline/application-crd-controller:1.0-beta-non-cluster-role
+          imagePullPolicy: IfNotPresent
+          name: manager
+          env:
+            - name: NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+          resources:
+            limits:
+              cpu: 100m
+              memory: 30Mi
+            requests:
+              cpu: 100m
+              memory: 20Mi
+      serviceAccountName: application

pipeline/upstream/base/application/application-controller-role.yaml

@@ -0,0 +1,21 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: application-manager-role
+rules:
+  - apiGroups:
+      - '*'
+    resources:
+      - '*'
+    verbs:
+      - get
+      - list
+      - update
+      - patch
+      - watch
+  - apiGroups:
+      - app.k8s.io
+    resources:
+      - '*'
+    verbs:
+      - '*'

pipeline/upstream/base/application/application-controller-rolebinding.yaml

@@ -0,0 +1,11 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: application-manager-rolebinding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: application-manager-role
+subjects:
+  - kind: ServiceAccount
+    name: application

pipeline/upstream/base/application/application-controller-sa.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: application

pipeline/upstream/base/application/application-controller-service.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: controller-manager-service
+  labels:
+    control-plane: controller-manager
+    controller-tools.k8s.io: "1.0"
+spec:
+  selector:
+    control-plane: controller-manager
+    controller-tools.k8s.io: "1.0"
+  ports:
+    - port: 443