From e392f5e901d2028e80205c45d312c4dc71ceea8d Mon Sep 17 00:00:00 2001 From: Rostam Dinyari Date: Tue, 12 Feb 2019 22:09:34 -0800 Subject: [PATCH 1/3] Kubeflow pipelines quickstart notebooks added. --- samples/notebooks/quickstart.ipynb | 572 ++++++++++++++++++++++++++ samples/notebooks/quickstart_iris.csv | 150 +++++++ 2 files changed, 722 insertions(+) create mode 100644 samples/notebooks/quickstart.ipynb create mode 100644 samples/notebooks/quickstart_iris.csv diff --git a/samples/notebooks/quickstart.ipynb b/samples/notebooks/quickstart.ipynb new file mode 100644 index 00000000000..0ae637efd21 --- /dev/null +++ b/samples/notebooks/quickstart.ipynb @@ -0,0 +1,572 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Copyright 2019 Google Inc. All Rights Reserved.\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# http://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Part 1\n", + "# Multiple ways to author a component to list blobs in a GCS bucket\n", + "A pipeline is composed of one or more components. In this section, you will build a single component that that lists the blobs in a GCS bucket. Then you buid a pipeline that consists of this component." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Create a 'lightweight python component' from a Python function." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.1 Define component function\n", + "The requirements for the component function:\n", + "* The function must be stand-alone.\n", + "* The function can only import packages that are available in the base image.\n", + "* If the function operates on numbers, the parameters must have type hints. Supported types are `int`, `float`, `bool`. Everything else is passed as `str`, that is, string.\n", + "* To build a component with multiple output values, use Python’s `typing.NamedTuple` type hint syntax." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def list_blobs(bucket_name: str) -> str:\n", + " '''Lists all the blobs in the bucket.'''\n", + " import subprocess\n", + "\n", + " subprocess.call(['pip', 'install', '--upgrade', 'google-cloud-storage'])\n", + " from google.cloud import storage\n", + " storage_client = storage.Client()\n", + " bucket = storage_client.get_bucket(bucket_name)\n", + " list_blobs_response = bucket.list_blobs()\n", + " blobs = ','.join([blob.name for blob in list_blobs_response])\n", + " print(blobs)\n", + " return blobs" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.2 Create a lightweight Python component" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import kfp.components as comp\n", + "\n", + "# Converts the function to a lightweight Python component.\n", + "list_blobs_op = comp.func_to_container_op(list_blobs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.3 Define pipeline" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import kfp.dsl as dsl\n", + "\n", + "# Defines the pipeline.\n", + "@dsl.pipeline(name='List GCS blobs', description='Lists GCS blobs.')\n", + "def pipeline_func(bucket_name=dsl.PipelineParam('bucket')):\n", + " list_blobs_task = list_blobs_op(bucket_name)\n", + "\n", + "# Compile the pipeline to a file.\n", + "import kfp.compiler as compiler\n", + "compiler.Compiler().compile(pipeline_func, 'list_blobs.pipeline.tar.gz')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Build a new Docker container image from a Python function.\n", + "Refer [here](https://runnable.com/docker/python/dockerize-your-python-application#alternatives) for more information." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Wrap an existing Docker container image using `ContainerOp`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.1 Create a Docker container\n", + "Create your own container image that includes your program. If your component creates some outputs to be fed as inputs to the downstream components, each separate output must be written as a string to a separate local text file by the container image. For example, if a trainer component needs to output the trained model path, it can write the path to a local file `/output.txt`. The string written to an output file cannot be too big. If it is too big (> 500kb), you can save the output to an external persistent storage and pass the storage path to the next component.\n", + "\n", + "Start by entering the value of your Google Cloud Platform Project ID." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# GCP Project ID\n", + "PROJECT_ID='PROJECT_ID'\n", + "\n", + "assert(PROJECT_ID is not 'PROJECT_ID')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The following cell creates a file `app.py` that contains a Python script. The script takes a GCS bucket name as an input argument, gets the lists of blobs in that bucket, prints the list of blobs and also writes them to an output file." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "\n", + "# Create folders if they don't exist.\n", + "mkdir -p tmp/components/list-gcs-blobs\n", + "\n", + "# Create the Python file that lists GCS blobs.\n", + "cat > ./tmp/components/list-gcs-blobs/app.py < ./tmp/components/list-gcs-blobs/Dockerfile < ./tmp/components/list-gcs-blobs/build_image.sh < ./tmp/components/view-input/app.py < ./tmp/components/view-input/Dockerfile < ./tmp/components/view-input/build_image.sh < Date: Sun, 24 Mar 2019 00:46:56 -0700 Subject: [PATCH 2/3] Incorporated comments. --- samples/notebooks/quickstart.ipynb | 118 ++++++++++++++--------------- 1 file changed, 56 insertions(+), 62 deletions(-) diff --git a/samples/notebooks/quickstart.ipynb b/samples/notebooks/quickstart.ipynb index 0ae637efd21..27f1359a88c 100644 --- a/samples/notebooks/quickstart.ipynb +++ b/samples/notebooks/quickstart.ipynb @@ -26,15 +26,15 @@ "metadata": {}, "source": [ "# Part 1\n", - "# Multiple ways to author a component to list blobs in a GCS bucket\n", - "A pipeline is composed of one or more components. In this section, you will build a single component that that lists the blobs in a GCS bucket. Then you buid a pipeline that consists of this component." + "# Two ways to author a component to list blobs in a GCS bucket\n", + "A pipeline is composed of one or more components. In this section, you will build a single component that that lists the blobs in a GCS bucket. Then you buid a pipeline that consists of this component. There are two ways to author a component. In the following sections we will go through each of them." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## 1. Create a 'lightweight python component' from a Python function." + "## 1. Create a lightweight python component from a Python function." ] }, { @@ -117,39 +117,19 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## 2. Build a new Docker container image from a Python function.\n", - "Refer [here](https://runnable.com/docker/python/dockerize-your-python-application#alternatives) for more information." + "## 2. Wrap an existing Docker container image using `ContainerOp`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## 3. Wrap an existing Docker container image using `ContainerOp`" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 3.1 Create a Docker container\n", - "Create your own container image that includes your program. If your component creates some outputs to be fed as inputs to the downstream components, each separate output must be written as a string to a separate local text file by the container image. For example, if a trainer component needs to output the trained model path, it can write the path to a local file `/output.txt`. The string written to an output file cannot be too big. If it is too big (> 500kb), you can save the output to an external persistent storage and pass the storage path to the next component.\n", + "### 2.1 Create a Docker container\n", + "Create your own container image that includes your program. If your component creates some outputs to be fed as inputs to the downstream components, each separate output must be written as a string to a separate local text file by the container image. For example, if a trainer component needs to output the trained model path, it can write the path to a local file `/output.txt`. The string written to an output file cannot be too big. If it is too big (>> 100 kB), save the output to an external persistent storage and pass the storage path to the next component.\n", "\n", "Start by entering the value of your Google Cloud Platform Project ID." ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# GCP Project ID\n", - "PROJECT_ID='PROJECT_ID'\n", - "\n", - "assert(PROJECT_ID is not 'PROJECT_ID')" - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -174,7 +154,8 @@ "from google.cloud import storage\n", "# Parse agruments.\n", "parser = argparse.ArgumentParser()\n", - "parser.add_argument('--bucket', type=str, required=True, help='GCS bucket name.')\n", + "parser.add_argument(\n", + " '--bucket', type=str, required=True, help='GCS bucket name.')\n", "args = parser.parse_args()\n", "# Create a client.\n", "storage_client = storage.Client()\n", @@ -216,7 +197,26 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now create a Shell script that builds a container image and stores it in the Google Container Registry (GCR)." + "Now that we have created our Dockerfile we can create our Docker image. Then we need to push the image to a registry to host the image. Here, we will use Google Container Registry, but any other accessible registry works as well. In the following cell set your project ID that will be used to to push your image to Google Container Registry." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# GCP Project ID\n", + "PROJECT_ID='PROJECT_ID'\n", + "\n", + "assert(PROJECT_ID is not 'PROJECT_ID')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now create a Shell script that builds a container image and stores it in the Google Container Registry." ] }, { @@ -268,8 +268,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### 3.2 Create a Python class for each component\n", - "Define a Python class that describes the interactions with the Docker container image created in the previous step. The Python class specifies the component name, the image to use, the command to run after the container starts, the input arguments, and the file outputs. Each component needs to inherit from `kfp.dsl.ContainerOp`." + "### 2.2 Define each component\n", + "Define a component by creating an instance of `kfp.dsl.ContainerOp` that describes the interactions with the Docker container image created in the previous step. You need to specify the component name, the image to use, the command to run after the container starts, the input arguments, and the file outputs. ." ] }, { @@ -280,22 +280,21 @@ "source": [ "import kfp.dsl\n", "\n", - "class ListGcsBlobsOp(kfp.dsl.ContainerOp):\n", - " def __init__(self, name, bucket):\n", - " super(ListGcsBlobsOp, self).__init__(\n", + "def list_gcs_blobs_op(name, bucket):\n", + " return kfp.dsl.ContainerOp(\n", " name=name,\n", " image='gcr.io/{}/listgcsblobs:latest'.format(PROJECT_ID),\n", " command=['python', '/app/app.py'],\n", " file_outputs={'blobs': '/blobs.txt'},\n", " arguments=['--bucket', bucket]\n", - " )" + " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### 3.3 Create your workflow as a Python function\n", + "### 2.3 Create your workflow as a Python function\n", "Start by creating a folder to store the pipeline file." ] }, @@ -330,11 +329,13 @@ " name='List GCS Blobs',\n", " description='Takes a GCS bucket name as input and lists the blobs.'\n", ")\n", - "def pipeline_func(bucket=kfp.dsl.PipelineParam('bucket', value='Enter your bucket name here.')):\n", - " list_blobs_task = ListGcsBlobsOp('List', bucket)\n", + "def pipeline_func(\n", + " bucket=kfp.dsl.PipelineParam('bucket', value='Enter your bucket name here.')):\n", + " list_blobs_task = list_gcs_blobs_op('List', bucket)\n", "\n", "# Compile the pipeline to a file.\n", - "filename = 'tmp/pipelines/list_blobs{dt:%Y%m%d_%H%M%S}.pipeline.tar.gz'.format(dt=datetime.datetime.now())\n", + "filename = 'tmp/pipelines/list_blobs{dt:%Y%m%d_%H%M%S}.pipeline.tar.gz'.format(\n", + " dt=datetime.datetime.now())\n", "compiler.Compiler().compile(pipeline_func, filename)" ] }, @@ -342,7 +343,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Go to your Cloud Shell, run `conn.sh`, click on the provided link. In the new tab that opens, click on \"Pipelines Dashboard\" and go to Kubeflow pipelines UI. Upload the created pipeline and run it.\n", + "Follow the [instructions](https://www.kubeflow.org/docs/other-guides/accessing-uis/) on kubeflow.org to access Kubeflow UIs. Upload the created pipeline and run it.\n", "\n", "**Warning:** When the pipeline is run, it pulls the image from the repository to the Kubernetes cluster to create a container. Kubernetes caches pulled images. One solution is to use the image digest instead of the tag in your component dsl, for example, `s/v1/sha256:9509182e27dcba6d6903fccf444dc6188709cc094a018d5dd4211573597485c9/g`. Alternatively, if you don't want to update the digest every time, you can try `:latest` tag, which will force the k8s to always pull the latest image.." ] @@ -460,8 +461,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## 2 Create a Python class for each component\n", - "The Python classes describe the interactions with the Docker container image created in step one. Each component needs to inherit from `kfp.dsl.ContainerOp`." + "## 2 Define each component\n", + "Define each of your components by using `kfp.dsl.ContainerOp`. Decribe the interactions with the Docker container image created in the previous step by specifying the component name, the image to use, the command to run after the container starts, the input arguments, and the file outputs." ] }, { @@ -472,24 +473,22 @@ "source": [ "import kfp.dsl\n", "\n", - "class ListGcsBlobsOp(kfp.dsl.ContainerOp):\n", - " def __init__(self, name, bucket):\n", - " super(ListGcsBlobsOp, self).__init__(\n", + "def list_gcs_blobs_op(name, bucket):\n", + " return kfp.dsl.ContainerOp(\n", " name=name,\n", " image='gcr.io/{}/listgcsblobs:latest'.format(PROJECT_ID),\n", " command=['python', '/app/app.py'],\n", " file_outputs={'blobs': '/blobs.txt'},\n", " arguments=['--bucket', bucket]\n", - " )\n", + " )\n", "\n", - "class ViewInputOp(kfp.dsl.ContainerOp):\n", - " def __init__(self, name, blobs):\n", - " super(ViewInputOp, self).__init__(\n", + "def view_input_op(name, blobs):\n", + " return kfp.dsl.ContainerOp(\n", " name=name,\n", " image='gcr.io/{}/viewinput:latest'.format(PROJECT_ID),\n", " command=['python', '/app/app.py'],\n", " arguments=['--blobs', blobs]\n", - " )" + " )" ] }, { @@ -497,7 +496,7 @@ "metadata": {}, "source": [ "## 3 Create your workflow as a Python function\n", - "Define your pipeline as a Python function. ` @kfp.dsl.pipeline` is a required decoration including `name` and `description` properties. `pipeline_func` defines the pipeline. `bucket=kfp.dsl.PipelineParam(...)` specifies that the pipeline takes an input parameter `bucket`. Later when you load the pipeline, `kfp.dsl.PipelineParam('bucket', value='Enter your bucket name here.')` will create an input box in the UI with the initial value `Enter your bucket name here.`. You can change the initial value with your bucket name at runtime. `ListGcsBlobsOp('List', bucket)` will create a component named `List` that lists the blobs. `ViewInputOp('View', list_blobs_task.outputs['blobs'])` will create a component named `View` that views a CSV. `list_blobs_task.outputs['blobs']` tells the pipeline to take the output of the first component stored as string in `blobs.txt` as an input for the second component." + "Define your pipeline as a Python function. ` @kfp.dsl.pipeline` is a required decoration including `name` and `description` properties. `pipeline_func` defines the pipeline. `bucket=kfp.dsl.PipelineParam(...)` specifies that the pipeline takes an input parameter `bucket`. Later when you load the pipeline, `kfp.dsl.PipelineParam('bucket', value='Enter your bucket name here.')` will create an input box in the UI with the initial value `Enter your bucket name here.`. You can change the initial value with your bucket name at runtime. `list_gcs_blobs_op('List', bucket)` will create a component named `List` that lists the blobs. `view_input_op('View', list_blobs_task.outputs['blobs'])` will create a component named `View` that views a CSV. `list_blobs_task.outputs['blobs']` tells the pipeline to take the output of the first component stored as string in `blobs.txt` as an input for the second component." ] }, { @@ -524,12 +523,14 @@ " name='Quickstart pipeline',\n", " description='Takes a GCS bucket name views a CSV input file in the bucket.'\n", ")\n", - "def pipeline_func(bucket=kfp.dsl.PipelineParam('bucket', value='Enter your bucket name here.')):\n", - " list_blobs_task = ListGcsBlobsOp('List', bucket)\n", - " view_input_task = ViewInputOp('View', list_blobs_task.outputs['blobs'])\n", + "def pipeline_func(bucket=kfp.dsl.PipelineParam(\n", + " 'bucket', value='Enter your bucket name here.')):\n", + " list_blobs_task = list_gcs_blobs_op('List', bucket)\n", + " view_input_task = view_input_op('View', list_blobs_task.outputs['blobs'])\n", "\n", "# Compile the pipeline to a file.\n", - "filename = 'tmp/pipelines/quickstart_pipeline{dt:%Y%m%d_%H%M%S}.pipeline.tar.gz'.format(dt=datetime.datetime.now())\n", + "filename = 'tmp/pipelines/quickstart_pipeline{dt:%Y%m%d_%H%M%S}.pipeline.tar.gz'.format(\n", + " dt=datetime.datetime.now())\n", "compiler.Compiler().compile(pipeline_func, filename)" ] }, @@ -537,15 +538,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Go to your Cloud Shell and run `conn.sh`, and click on the provided link. In the new tab that opens, click on \"Pipelines Dashboard\". Upload the created pipeline and run it." + "Follow the [instructions](https://www.kubeflow.org/docs/other-guides/accessing-uis/) on kubeflow.org to access Kubeflow UIs. Upload the created pipeline and run it." ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { From 50e73f06b528cdeabcceeedbe996255b20564b79 Mon Sep 17 00:00:00 2001 From: Rostam Dinyari Date: Tue, 2 Apr 2019 13:23:07 -0700 Subject: [PATCH 3/3] Incorporated comments. --- samples/notebooks/quickstart.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/notebooks/quickstart.ipynb b/samples/notebooks/quickstart.ipynb index 27f1359a88c..e880a118d11 100644 --- a/samples/notebooks/quickstart.ipynb +++ b/samples/notebooks/quickstart.ipynb @@ -27,7 +27,7 @@ "source": [ "# Part 1\n", "# Two ways to author a component to list blobs in a GCS bucket\n", - "A pipeline is composed of one or more components. In this section, you will build a single component that that lists the blobs in a GCS bucket. Then you buid a pipeline that consists of this component. There are two ways to author a component. In the following sections we will go through each of them." + "A pipeline is composed of one or more components. In this section, you will build a single component that lists the blobs in a GCS bucket. Then you buid a pipeline that consists of this component. There are two ways to author a component. In the following sections we will go through each of them." ] }, {