Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MKP/doc] Update doc for changes of service account credential. #2309

Merged
merged 4 commits into from
Oct 5, 2019
Merged

[MKP/doc] Update doc for changes of service account credential. #2309

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b64aca2
Update README for MKP development. Remove managed SQL part and update…
Oct 4, 2019
d3faa6f
Update guide for MKP deployment.
Oct 4, 2019
9ee75f5
Small fix.
Oct 4, 2019
016b3ab
fix one missing thing
Oct 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 1 addition & 12 deletions manifests/gcp_marketplace/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,12 +115,6 @@ Creat the namespace
kubectl create namespace $NAMESPACE
```

Download token for your service account which you want to use for calling GCP APIs from the pipelines.
```shell
gcloud iam service-accounts keys create application_default_credentials.json --iam-account [your-service-account]
export SERVICE_ACCOUNT_TOKEN="$(cat application_default_credentials.json | base64 -w 0)"
```

Follow the [instruction](https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/master/docs/tool-prerequisites.md#tool-prerequisites) and install mpdev
TODO: The official mpdev won't work because it doesn't have permission to deploy CRD. The latest unofficial build will have right permission. Remove following instruction when change is in prod.
```
Expand All @@ -134,15 +128,10 @@ export MARKETPLACE_TOOLS_IMAGE=gcr.io/cloud-marketplace-staging/marketplace-k8s-
Run the install script

```shell
mpdev scripts/install --deployer=gcr.io/ml-pipeline/google/pipelines/deployer:0.2 --parameters='{"name": "'$APP_INSTANCE_NAME'", "namespace": "'$NAMESPACE'", "serviceAccountCredential": "'$SERVICE_ACCOUNT_TOKEN'"}'
mpdev scripts/install --deployer=gcr.io/ml-pipeline/google/pipelines/deployer:0.1 --parameters='{"name": "'$APP_INSTANCE_NAME'", "namespace": "'$NAMESPACE'"}'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the image tag getting decreased to 0.1 expected?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently for alpha launch, we'll pin the version to 0.1, so yes.


```

Or if using CloudSQL and GCS,
```
mpdev scripts/install --deployer=gcr.io/ml-pipeline/google/pipelines/deployer:0.2 --parameters='{"name": "'$APP_INSTANCE_NAME'", "namespace": "'$NAMESPACE'", "serviceAccountCredential": "'$SERVICE_ACCOUNT_TOKEN'", "managedstorage.enabled": true, "managedstorage.cloudsqlInstanceConnectionName": "[your-name]", "managedstorage.dbPassword": "[your-pwd]"}'
```

Watch the deployment come up with

```shell
Expand Down
41 changes: 29 additions & 12 deletions manifests/gcp_marketplace/guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,26 +17,43 @@ Specify a [Kubenetes namespace](https://kubernetes.io/docs/concepts/overview/wor
## App instance name
Specify an app instance name to help you identify this instance.

## Deploy
Click `Deploy` to start deploying Kubeflow Pipelines into the cluster you specified.
Deployment might take few minutes, so please be patient. After deployment is complete, go to the [Pipelines Console](http://pantheon.corp.google.com/ai-platform/pipelines) to access the Kubeflow Pipelines instance.

## GCP Service Account credentials
This deployment requires a [GCP service account](https://cloud.google.com/iam/docs/service-accounts) to use for authentication when calling other GCP services. This includes Cloud Storage and Cloud SQL if you are using managed storage, as well as other services your pipeline might need, for example Dataflow. Specify the base64-encoded credentials for the service account you want to use.
After deployment, you can grant KFP proper permission by specifying its service account and binding
proper role to it.

You can get these credentials by running the following command in a terminal window. This command will create a new key under the service account. Please note that a single service account can only have 10 keys.
Usually a functional KFP pipeline requires a [GCP service account](https://cloud.google.com/iam/docs/service-accounts) to use for
authentication when calling other GCP services. This includes Cloud Storage as well as other services your pipeline might need,
for example Dataflow, Dataproc. Specify the base64-encoded credentials for the service account you want to use.

This can be done through command line using `kubectl`.
```
$ gcloud iam service-accounts keys create application_default_credentials.json --iam-account [your-service-account] && cat application_default_credentials.json | base64
export CLUSTER=<cluster-where-kfp-was-installed>
export ZONE=<zone-where-kfp-was-installed>
# Configure kubectl to connect with the cluster
gcloud container clusters get-credentials "$CLUSTER" --zone "$ZONE"
```

Existing key also can be used.

Then you can create and inject service account credential.
```
cat existing_credentials.json | base64
export PROJECT=<my-project>
export SA_NAME=<my-account>
# Create service account
gcloud iam service-accounts create $SA_NAME --display-name $SA_NAME
gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:my-account@$PROJECT.iam.gserviceaccount.com --role=roles/storage.admin
# Also do this binding for other roles you need. For example, dataproc.admin and dataflow.admin
gcloud iam service-accounts keys create application_default_credentials.json --iam-account $SA_NAME@$PROJECT.iam.gserviceaccount.com
export SERVICE_ACCOUNT_TOKEN="$(cat application_default_credentials.json | base64 -w 0)"
echo -e "apiVersion: v1\nkind: Secret\nmetadata:\n name: \"user-gcp-sa\"\n namespace: \"${NAMESPACE}\"\n labels:\n app: gcp-sa\n app.kubernetes.io/name: \"${APP_INSTANCE_NAME}\"\ntype: Opaque\ndata:\n application_default_credentials.json: ${SERVICE_ACCOUNT_TOKEN}\n user-gcp-sa.json: $SERVICE_ACCOUNT_TOKEN" > secret.yaml
kubectl apply -f secret.yaml
# Remove secret files
rm application_default_credentials.json test.yaml
```

If you are running this command on Linux, please use `base64 -w 0` to disable line wrapping.

## Deploy
Click `Deploy` to start deploying Kubeflow Pipelines into the cluster you specified.
Deployment might take few minutes, so please be patient. After deployment is complete, go to the [Pipelines Console](http://pantheon.corp.google.com/ai-platform/pipelines) to access the Kubeflow Pipelines instance.
Note that the above commands use `base64 -w 0` to disable line wrapping, this could be slightly different
across platforms.

## Tips

Expand Down