Add cloud specific deployment guide + refactoring

docs/book/features/cloud-pipelines.md → docs/book/features/cloud-pipelines/cloud-pipelines.md

@@ -59,7 +59,7 @@ configuration to it - it just works out of the box.
 ### MySQL
 
 Using MySQL as a Metadata Store is where ZenML can become really powerful, especially in highly dynamic environments 
-(read: running experiments locally, in the Cloud, and across team members). Some ZenML integrations even require a 
+(read: running experiments locally, in the cloud, and across team members). Some ZenML integrations even require a 
 dedicated MySQL-based Metadata Store to unfold their true potential.
 
 The Metadata Store can be simply configured to use any MySQL server (=>5.6):
@@ -83,8 +83,10 @@ Closely related to the [Metadata Store](https://github.com/zenml-io/zenml/blob/1
 You have the following options to configure the Artifact Store:
 
 * Local (Default)
-* Remote (Google Cloud Storage)
-  * **Soon**: AWS S3-compatible stacks
+* Remote
+  * GCS Bucket
+  * AWS S3 Bucket
+  * Azure Blob Storage
 
 ### Local (Default)
 
@@ -94,7 +96,7 @@ and inputs will be persisted there.
 Using the default Artifact Store can be a limitation to the integrations you might want to use. Please check the 
 documentation of the individual integrations to make sure they are compatible.
 
-### Remote (GCS/S3)
+### Remote (GCS/S3/Azure)
 
 Many experiments and many ZenML integrations require a remote Artifact Store to reliable retrieve and persist pipeline 
 step artifacts. Especially dynamic scenarios with heterogeneous environments will be only possible when using a remote 
@@ -122,9 +124,9 @@ This is the default orchestrator for ZenML pipelines. It runs pipelines sequenti
 local environment. You can use this orchestrator for quick experimentation and work on smaller datasets in your 
 local environment.
 
-#### Airflow
+#### Kubeflow
 
-Coming Soon!
+The Kubeflow orchestrator supports both local and remote execution. In the local case, a `k3d` cluster is spun up and Kubeflow Pipelines is used to run your pipelines. In the remote case, ZenML can work with a Kubeflow installation on your remote Kubernetes cluster. Check this [page](../../guides/functional-api/deploy-to-production.md) for more details. 
 
 #### GCP Orchestrator
 
@@ -150,9 +152,10 @@ Coming Soon!
 
 Coming Soon!
 
-#### Kubeflow
+#### Airflow
+
+Coming Soon!
 
-Coming soon!
 
 ### Creating a custom orchestrator
 

docs/book/features/cloud-pipelines/guide-aws-gcp-azure.md

@@ -0,0 +1,182 @@
+---
+description: Deploy pipelines to public cloud
+---
+
+# Guide for cloud-specific deployments
+
+This guide will show how you can run the a pipeline in Kubeflow Pipelines deployed to a public cloud cluster. We will start with some pre-requisites and then move on to show the integration of your cloud provider's components with your ZenML stack.
+
+
+## Pre-requisites
+
+### Orchestrator
+
+{% tabs %}
+
+{% tab title="AWS" %}
+
+- Have an existing AWS [EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) set up.
+- Download and [install](https://kubernetes.io/docs/tasks/tools/) `kubectl` and [configure](https://aws.amazon.com/premiumsupport/knowledge-center/eks-cluster-connection/) it to talk to your EKS cluster using the following command. Make sure you have the `aws` cli set up first.
+
+    ```powershell
+    aws eks --region REGION update-kubeconfig --name CLUSTER_NAME
+    ```
+
+- [Install](https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/#deploying-kubeflow-pipelines) Kubeflow Pipelines onto your cluster.
+
+    {% hint style="info" %}
+     If one or more of the deployments are not in the `Running` state, try increasing the number of nodes in your cluster.
+    {% endhint %}
+
+    {% hint style="warning" %}
+    If you are doing a manual install of the Kubeflow Pipelines, make sure that the service name for the ML pipeline is exactly `ml-pipeline` . This will ensure that ZenML can talk to your Kubeflow deployment.
+    {% endhint %}
+{% endtab %}
+
+
+{% tab title="GCP" %} 
+
+- Have an existing GCP [GKE cluster](https://cloud.google.com/kubernetes-engine/docs/quickstart) set up.
+- Download and [install](https://kubernetes.io/docs/tasks/tools/) `kubectl` and [configure](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl) it to talk to your GKE cluster using the following command. Make sure you have the Google Cloud CLI set up first.
+
+    ```powershell
+    gcloud container clusters get-credentials CLUSTER_NAME
+    ```
+
+- [Install](https://www.kubeflow.org/docs/distributions/gke/deploy/overview/) Kubeflow Pipelines onto your cluster.
+
+    {% hint style="info" %}
+     If one or more of the deployments are not in the \`Running\` state, try increasing the number of nodes in your cluster.
+    {% endhint %}
+
+    {% hint style="warning" %}
+    If you are doing a manual install of the Kubeflow Pipelines, make sure that the service name for the ML pipeline is exactly \`ml-pipeline\` . This will ensure that ZenML can talk to your Kubeflow deployment.
+    {% endhint %}
+{% endtab %}
+
+
+{% tab title="Azure" %} 
+
+Coming soon!
+
+{% endtab %} 
+
+
+{% endtabs %}
+
+
+
+### Container Registry
+
+{% tabs %}
+
+{% tab title="AWS" %} 
+
+- [Set up](https://docs.aws.amazon.com/AmazonECR/latest/userguide/get-set-up-for-amazon-ecr.html) an Elastic Container Registry (ECR) and create a repository (either public or private) with the name `zenml-kubeflow` . This is the repository to which ZenML will push your pipeline images to.
+- Authenticate your local `docker` cli with your ECR registry using the following command.
+
+    ```powershell
+    aws ecr get-login-password --region REGION | docker login --username AWS --password-stdin ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com
+    ```
+
+{% endtab %} 
+
+{% tab title="GCP" %}
+
+- Set up a [GCP Container Registry](https://cloud.google.com/container-registry/docs).
+- [Authenticate](https://cloud.google.com/container-registry/docs/advanced-authentication) your local `docker` cli with your GCP container registry.
+
+
+{% endtab %}
+
+
+{% tab title="Azure" %} 
+
+Coming soon!
+
+{% endtab %} 
+
+{% endtabs %}
+
+
+
+### Artifact Store
+
+{% tabs %}
+
+{% tab title="AWS" %} 
+
+- [Create](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) an Amazon S3 bucket in a region of your choice.
+- Make sure that your EKS cluster is authorized to access the S3 bucket. This can be done in one of the following ways:
+    - A simple way is to add an [`AmazonS3FullAccess`](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AmazonS3FullAccess) policy to your cluster’s node group IAM role.
+    - A complex way would be to create `ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` environment variables in your EKS cluster. This can be done by extending the `zenmldocker/zenml` image and [adding](https://docs.docker.com/engine/reference/builder/#env) these variables in a Dockerfile. 
+    This image can then be used by modifying the `custom_docker_base_image_name` field inside your orchestrator’s YAML configuration at `.zen/orchestrators/*.yaml` .
+- The path for your bucket should be in this format `s3:\\your-bucket`
+
+{% endtab %} 
+
+{% tab title="GCP" %}
+
+- [Create](https://cloud.google.com/storage/docs/creating-buckets) a GCP Cloud Storage Bucket in a region of your choice.
+- Make sure that your GKE cluster is authorized to access the GCS bucket.
+- The path for your bucket should be in this format `gs://your-bucket`
+
+{% endtab %}
+
+
+{% tab title="Azure" %} 
+
+- [Create](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-overview) an Azure Storage Account and [add](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal) a *container* to hold your *blobs.*
+- Open your storage account page on the Azure portal and follow these steps:
+    - Keep the name of the storage account handy - it will be required in a later step.
+    - From the navigation panel on the left, select **Access Keys** under **Security + networking**. Note any of the keys available there for future use.
+- Add your storage `Account Name` and the `Account Key` you just created as environment variables in your Docker image. This can be done by extending the `zenmldocker/zenml` image and [adding](https://docs.docker.com/engine/reference/builder/#env) these variables in a Dockerfile. 
+This image can then be used by modifying the `custom_docker_base_image_name` field inside your orchestrator’s YAML configuration at `.zen/orchestrators/*.yaml` .
+- The path for your bucket should be in this format `az://<CONTAINER-NAME>`
+
+{% endtab %} 
+
+{% endtabs %}
+
+
+
+
+## Integrating with ZenML
+
+To run our pipeline on Kubeflow Pipelines deployed to AWS, we will create a new stack with these components that you have just created. 
+
+1. Install the cloud provider and the `kubeflow` plugin
+
+    ```powershell
+    zenml integration install <aws/gcp/azure>
+    zenml integration install kubeflow
+    ```
+
+2. Register the stack components
+
+    ```powershell
+    zenml container-registry register cloud-registry --type=default --uri=$PATH_TO_YOUR_CONTAINER_REGISTRY
+    zenml orchestrator register cloud_orchestrator --type=kubeflow
+    zenml metadata-store register kubeflow_metadata_store --type=kubeflow
+    zenml artifact-store register cloud_artifact_store --type=<s3/gcp/azure> --path=$PATH_TO_YOUR_BUCKET
+
+    # Register the cloud stack
+    zenml stack register cloud_kubeflow_stack -m kubeflow_metadata_store -a cloud_artifact_store -o cloud_orchestrator -c cloud_registry
+    ```
+
+3. Activate the newly created stack.
+
+    ```powershell
+    zenml stack set cloud_kubeflow_stack
+    ```
+
+4. Do a pipeline run and check your Kubeflow UI to see it running there! 🚀
+
+{% hint style="info" %}
+* The **metadata store** stores metadata inside the Kubeflow Pipelines internal MySQL database.
+* You can choose any name for your stack components apart from the ones used in the script above.
+{% endhint %}
+
+{% hint style="warning" %}
+&#x20;Make sure to replace `$PATH_TO_YOUR_BUCKET`and `$PATH_TO_YOUR_CONTAINER_REGISTRY` with the actual URI's of your bucket and container registry.
+{% endhint %}

docs/book/guides/functional-api/deploy-to-production.md

@@ -130,47 +130,10 @@ Once you're done experimenting, you can delete the local Kubernetes cluster and
 zenml stack down
 ```
 
-### Run the same pipeline on Kubeflow Pipelines deployed to GCP
+### Run the same pipeline on Kubeflow Pipelines deployed to the cloud
 
-We will now run the same pipeline in Kubeflow Pipelines deployed to a Google Kubernetes Engine cluster. As you can see from the long list of additional pre-requisites, this requires lots of external setup steps at the moment. In future releases ZenML will be able to automate most of these steps for you, so make sure to revisit this guide if this is something you're interested in!
-
-#### Additional pre-requisites
-
-* An existing [GCP container registry](https://cloud.google.com/container-registry/docs).
-* An existing [GCP bucket](https://cloud.google.com/storage/docs/creating-buckets).
-* [Kubeflow Pipelines](https://www.kubeflow.org/docs/distributions/gke/deploy/overview/) deployed to a Google Kubernetes Engine cluster.
-* The local docker client has to be [authorized](https://cloud.google.com/container-registry/docs/advanced-authentication) to access the GCP container registry.
-* Kubectl can [access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl) your GCP Kubernetes cluster.
-* The [current context](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-context-and-configuration) configured in Kubectl points to your GCP cluster.
-
-#### Create a GCP Kubeflow Pipelines stack
-
-To run our pipeline on Kubeflow Pipelines deployed to GCP, we will create a new stack with these components:
-
-* The **artifact store** stores step outputs in a GCP Bucket.
-* The **metadata store** stores metadata inside the Kubeflow Pipelines internal MySQL database.
-* The docker images that are created to run your pipeline are stored in GCP **container registry**.
-* The **Kubeflow orchestrator** is the same as in the local Kubeflow Pipelines example.
-
-When running the upcoming commands, make sure to replace `$PATH_TO_YOUR_CONTAINER_REGISTRY` and `$PATH_TO_YOUR_GCP_BUCKET` with the actual URI's of your container registry and bucket.
-
-```bash
-# In order to create the GCP artifact store, we need to install one additional ZenML integration:
-zenml integration install gcp
-
-# Create the stack and its components
-zenml container-registry register gcp_registry  --type=default --uri=$PATH_TO_YOUR_CONTAINER_REGISTRY
-zenml metadata-store register kubeflow_metadata_store --type=kubeflow
-zenml artifact-store register gcp_artifact_store --type=gcp --path=$PATH_TO_YOUR_GCP_BUCKET
-zenml stack register gcp_kubeflow_stack \
-    -m kubeflow_metadata_store \
-    -a gcp_artifact_store \
-    -o kubeflow_orchestrator \
-    -c gcp_registry
-
-# Activate the newly created stack
-zenml stack set gcp_kubeflow_stack
-```
+We will now run the same pipeline in Kubeflow Pipelines deployed to a cluster on the cloud. 
+Refer to the Cloud Pipelines Deployment Guide [here](../../features/cloud-pipelines/cloud-pipelines.md) to know more and follow along!
 
 ## Conclusion