From ad3e0ae116a9427a568e05665f289faaa3a23906 Mon Sep 17 00:00:00 2001 From: ryandawsonuk Date: Mon, 22 Mar 2021 10:54:35 +0000 Subject: [PATCH 1/4] seldon explainer quickstart --- .../seldon-deploy-canary-quickstart.yaml | 13 +++--- .../seldon-deploy-explainer-quickstart.yaml | 44 +++++++++++++++++++ 2 files changed, 52 insertions(+), 5 deletions(-) create mode 100644 data/quickstarts/seldon-deploy-explainer-quickstart.yaml diff --git a/data/quickstarts/seldon-deploy-canary-quickstart.yaml b/data/quickstarts/seldon-deploy-canary-quickstart.yaml index 90781654b5..b67063e42e 100644 --- a/data/quickstarts/seldon-deploy-canary-quickstart.yaml +++ b/data/quickstarts/seldon-deploy-canary-quickstart.yaml @@ -10,21 +10,24 @@ spec: introduction: |- ### This quick start shows you how to launch a SKLearn model and update model by canarying. Seldon Deploy is a specialist set of tools designed to simplify and accelerate the process of deploying and managing your machine learning models. + + This quick start is a short version of https://deploy.seldon.io/docs/demos/seldon-core/canary/ tasks: - title: Deploy a pretrained sklearn iris model description: |- ### Deploy a pretrained sklearn iris model 1. Open Seldon Console and click create. Deployment creation wizard appears. - 2. Put the location of the model url. For example: gs://seldon-models/sklearn/iris + 2. Choose type SeldonDeployment and click next. + 3. Choose toolkit scikit-learn and enter the model url gs://seldon-models/sklearn/iris + 4. Click through wizard to deploy the model summary: success: You have deployed a pretrained sklearn iris model - failed: Try the steps again. + failed: Try the steps again or see https://deploy.seldon.io/docs/demos/seldon-core/canary/ - title: Start Load Test description: |- ### Complete the load test wizard: - 1. Use the request.json file in this folder: - 2 {"data": {"names": ["Sepal length","Sepal width","Petal length", "Petal Width"], "ndarray": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}} - 3. When running you should see metrics on dashboard. Enter the request logs screen to view request payloads. + 1. Use this request payload to make a prediction {"data": {"names": ["Sepal length","Sepal width","Petal length","Petal Width"], "ndarray": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}} + 2. When running you should see metrics on dashboard. Enter the request logs screen to view request payloads. summary: success: You successfully performed a load test against the pre-trained iris model. failed: Try the steps again. diff --git a/data/quickstarts/seldon-deploy-explainer-quickstart.yaml b/data/quickstarts/seldon-deploy-explainer-quickstart.yaml new file mode 100644 index 0000000000..5e27861b17 --- /dev/null +++ b/data/quickstarts/seldon-deploy-explainer-quickstart.yaml @@ -0,0 +1,44 @@ +kind: ConsoleQuickStart +metadata: + name: seldon-deploy-model-explainer +spec: + displayName: See predictions and explanations for a deployed SKLearn model + durationMinutes: 10 + icon: 'images/seldon.svg' + description: See predictions and explanations for deployed income classifier model + introduction: |- + ### This quick start shows you how to launch an income classifier model and see explanations. + Seldon Deploy is a specialist set of tools designed to simplify and accelerate the process of deploying and managing your machine learning models. + + This quick start is a short version of https://deploy.seldon.io/docs/demos/seldon-core/explainer-anchortabular/ + tasks: + - title: Deploy a pretrained income classifier model + description: |- + ### Deploy a pretrained income classifier model + 1. Open Seldon Console and click create. Deployment creation wizard appears. + 2. Choose type SeldonDeployment and click next. + 3. Choose toolkit scikit-learn and enter the model url gs://seldon-models/sklearn/income/model-0.23.2 + 4. Click through wizard to deploy the model + summary: + success: You have deployed a pretrained sklearn iris model + failed: Try the steps again or see https://deploy.seldon.io/docs/demos/seldon-core/explainer-anchortabular/ + - title: Add an explainer + description: |- + ### Deploy a pretrained explainer model for the income classifier + 1. From the income classifier model's page, go to the alibi explainer wizard + 2. Enter the model url gs://seldon-models/sklearn/income/explainer-py36-0.5.2 for an Anchor Tabular explainer and use the defaults + 3. Click through wizard to deploy the explainer + summary: + success: The explainer shows as loaded + failed: Try the steps again or see https://deploy.seldon.io/docs/demos/seldon-core/explainer-anchortabular/ + - title: Make a prediction and see explanations + description: |- + ### Make a prediction: + 1. Use this request payload to make a prediction {"data": {"names": ["Age","Workclass","Education","Marital Status","Occupation","Relationship","Race","Sex","Capital Gain","Capital Loss","Hours per week","Country"], "ndarray": [[53,4,0,2,8,4,2,0,0,0,60,9]]}} + 2. You should get a successful result back from the prediction. + 3. Click explain under the prediction to understand why this result was given by seeing anchors of most influential features + summary: + success: You see anchor-style explanations for income classifier model. + failed: Try the steps again or see https://deploy.seldon.io/docs/demos/seldon-core/explainer-anchortabular/ + conclusion: You are now able to understand the predictoins of the model. + nextQuickStart: [] From be82719829888b05da550b2bb1d50748cc0a19fa Mon Sep 17 00:00:00 2001 From: ryandawsonuk Date: Mon, 22 Mar 2021 13:46:20 +0000 Subject: [PATCH 2/4] outlier example --- .../seldon-deploy-canary-quickstart.yaml | 2 +- .../seldon-deploy-explainer-quickstart.yaml | 2 +- .../seldon-deploy-outlier-quickstart.yaml | 53 +++++++++++++++++++ 3 files changed, 55 insertions(+), 2 deletions(-) create mode 100644 data/quickstarts/seldon-deploy-outlier-quickstart.yaml diff --git a/data/quickstarts/seldon-deploy-canary-quickstart.yaml b/data/quickstarts/seldon-deploy-canary-quickstart.yaml index b67063e42e..d13692e3cb 100644 --- a/data/quickstarts/seldon-deploy-canary-quickstart.yaml +++ b/data/quickstarts/seldon-deploy-canary-quickstart.yaml @@ -18,7 +18,7 @@ spec: ### Deploy a pretrained sklearn iris model 1. Open Seldon Console and click create. Deployment creation wizard appears. 2. Choose type SeldonDeployment and click next. - 3. Choose toolkit scikit-learn and enter the model url gs://seldon-models/sklearn/iris + 3. Choose runtime scikit-learn and enter the model url gs://seldon-models/sklearn/iris 4. Click through wizard to deploy the model summary: success: You have deployed a pretrained sklearn iris model diff --git a/data/quickstarts/seldon-deploy-explainer-quickstart.yaml b/data/quickstarts/seldon-deploy-explainer-quickstart.yaml index 5e27861b17..8aa8103293 100644 --- a/data/quickstarts/seldon-deploy-explainer-quickstart.yaml +++ b/data/quickstarts/seldon-deploy-explainer-quickstart.yaml @@ -17,7 +17,7 @@ spec: ### Deploy a pretrained income classifier model 1. Open Seldon Console and click create. Deployment creation wizard appears. 2. Choose type SeldonDeployment and click next. - 3. Choose toolkit scikit-learn and enter the model url gs://seldon-models/sklearn/income/model-0.23.2 + 3. Choose runtime scikit-learn and enter the model url gs://seldon-models/sklearn/income/model-0.23.2 4. Click through wizard to deploy the model summary: success: You have deployed a pretrained sklearn iris model diff --git a/data/quickstarts/seldon-deploy-outlier-quickstart.yaml b/data/quickstarts/seldon-deploy-outlier-quickstart.yaml new file mode 100644 index 0000000000..7942d033cb --- /dev/null +++ b/data/quickstarts/seldon-deploy-outlier-quickstart.yaml @@ -0,0 +1,53 @@ +kind: ConsoleQuickStart +metadata: + name: seldon-deploy-model-outlier +spec: + displayName: See outlier scores for predictions to deployed model + durationMinutes: 10 + icon: 'images/seldon.svg' + description: See outlier scores for predictions to a deployed image classifier model + introduction: |- + ### This quick start shows you how to launch a tensorflow image classifier model and detect requests which are outliers. + Seldon Deploy is a specialist set of tools designed to simplify and accelerate the process of deploying and managing your machine learning models. + + This quick start is a short version of https://deploy.seldon.io/docs/demos/seldon-core/outlier/ + tasks: + - title: Deploy a pretrained image classifier model + description: |- + ### Deploy a pretrained image classifier model + 1. Open Seldon Console and click create. Deployment creation wizard appears. + 2. Choose type SeldonDeployment and protocol tensorflow and click next. + 3. Choose runtime tensorflow and enter the model url gs://seldon-models/tfserving/cifar10/resnet32 + 4. Click through wizard to deploy the model + summary: + success: You have deployed a pretrained tensorflow image classifier + failed: Try the steps again or see https://deploy.seldon.io/docs/demos/seldon-core/outlier/ + - title: Add an outlier detector + description: |- + ### Deploy a pretrained outlier detector for the image classifier + 1. From the image classifier model's page, go to the outlier detector wizard + 2. Enter the model name cifar10 and url gs://seldon-models/alibi-detect/od/OutlierVAE/cifar10 for an outlier detector and use the defaults + 3. Click through wizard to deploy the outlier detector + summary: + success: The outlier detector shows as loaded + failed: Try the steps again or see https://deploy.seldon.io/docs/demos/seldon-core/outlier/ + - title: Make predictions + description: |- + ### Make predictions: + 1. Go to the make predictions screen + 2. Run a single prediction using the tensorflow payload format of an image truck - file is https://deploy.seldon.io/docs/demos/seldon-core/outlier/cifar10_image.json + 3. Run a prediction payload format of an outlier/perturbed truck - file is https://deploy.seldon.io/docs/demos/seldon-core/outlier/outlier_image.json + summary: + success: You see ok responses for each of the predictions. + failed: Try the steps again or see https://deploy.seldon.io/docs/demos/seldon-core/outlier/ + - title: Review Outliers + description: |- + ### Review Predictions/Outliers: + 1. Go to the requests screen (request log) + 2. Observe the outlier value on each instance. + 3. Optionally highlight outliers based on this score and also use the filter to see only the outliers as needed. + summary: + success: You see outlier scores for each of the predictions. + failed: Try the steps again or see https://deploy.seldon.io/docs/demos/seldon-core/outlier/ + conclusion: You are now able to identify outliers in requests sent to the model. + nextQuickStart: [] From 0c0777faa15f582749d92c4164bf324a90ccf107 Mon Sep 17 00:00:00 2001 From: ryandawsonuk Date: Tue, 23 Mar 2021 11:05:56 +0000 Subject: [PATCH 3/4] explain seldon core and seldon deploy --- data/getting-started/seldon-deploy.md | 30 +++++++++++++++++-- .../seldon-deploy-explainer-quickstart.yaml | 2 +- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/data/getting-started/seldon-deploy.md b/data/getting-started/seldon-deploy.md index 6241f684d3..55da6611db 100644 --- a/data/getting-started/seldon-deploy.md +++ b/data/getting-started/seldon-deploy.md @@ -1,4 +1,30 @@ -## Subscribe to the operator on Market place +# Seldon Deploy + +[Seldon Deploy](https://deploy.seldon.io) provides oversight and governance for machine learning deployments. + +Seldon Deploy [builds on top of Seldon Core](https://deploy.seldon.io/docs/about/), an open source platform for deploying machine learning models on a Kubernetes cluster. + +Deployments in Seldon Deploy can be SeldonDeployments, which are [Seldon Core resources](https://docs.seldon.io/en/latest/). + +Seldon Core is an open source operator and custom resource (the SeldonDeployment) for deploying models. Models can be built in different toolkits, packaged by either serializing or dockerising and then deployed to kubernetes in SeldonDeployments. + +Features include: + +* Integrations to gateways for progressive rollout strategies such as canary and A/B test. +* Graphs for pre-process and post-process steps +* Integration to elasticsearch for audit trails of requests +* Integration to prometheus for monitoring +* Ways to deploy explainers, outlier detectors and drift detectors for advanced monitoring. + +Seldon deploy adds a user interface for easily deploying, monitoring and managing models. It adds enterprise functionality such as user roles and permissions, overall resource usage and integrations such as gitops. + +## Installing Seldon Deploy + +Seldon Deploy has a Red Hat marketplace listing. That install includes Red Hat versions of required sub-components such as Seldon Core. + +Documentation on the Seldon Deploy Red Hat marketplace install is at [https://deploy-master.seldon.io/docs/getting-started/redhat-installation/](https://deploy-master.seldon.io/docs/getting-started/redhat-installation/) + +### Subscribe to the operator on Market place - [https://marketplace.redhat.com/en-us/products/seldon-deploy](https://marketplace.redhat.com/en-us/products/seldon-deploy) -## Install the operator and validate +### Install the operator and validate - [https://marketplace.redhat.com/en-us/documentation/operators](https://marketplace.redhat.com/en-us/documentation/operators) diff --git a/data/quickstarts/seldon-deploy-explainer-quickstart.yaml b/data/quickstarts/seldon-deploy-explainer-quickstart.yaml index 8aa8103293..454d8884f9 100644 --- a/data/quickstarts/seldon-deploy-explainer-quickstart.yaml +++ b/data/quickstarts/seldon-deploy-explainer-quickstart.yaml @@ -40,5 +40,5 @@ spec: summary: success: You see anchor-style explanations for income classifier model. failed: Try the steps again or see https://deploy.seldon.io/docs/demos/seldon-core/explainer-anchortabular/ - conclusion: You are now able to understand the predictoins of the model. + conclusion: You are now able to understand the predictions of the model. nextQuickStart: [] From 91c1a5b9903e8368119f4dbefc4a7895dff441f6 Mon Sep 17 00:00:00 2001 From: ryandawsonuk Date: Tue, 23 Mar 2021 11:25:37 +0000 Subject: [PATCH 4/4] drift demo --- .../seldon-deploy-drift-quickstart.yaml | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 data/quickstarts/seldon-deploy-drift-quickstart.yaml diff --git a/data/quickstarts/seldon-deploy-drift-quickstart.yaml b/data/quickstarts/seldon-deploy-drift-quickstart.yaml new file mode 100644 index 0000000000..f3c06e2b15 --- /dev/null +++ b/data/quickstarts/seldon-deploy-drift-quickstart.yaml @@ -0,0 +1,53 @@ +kind: ConsoleQuickStart +metadata: + name: seldon-deploy-model-drift +spec: + displayName: Monitor drift for deployed model + durationMinutes: 10 + icon: 'images/seldon.svg' + description: Monitor drift for deployed image classifier model + introduction: |- + ### This quick start shows you how to launch a tensorflow image classifier model and detect when requests to the model are drifting. + Seldon Deploy is a specialist set of tools designed to simplify and accelerate the process of deploying and managing your machine learning models. + + This quick start is a short version of https://deploy-master.seldon.io/docs/demos/seldon-core/drift/ + tasks: + - title: Deploy a pretrained image classifier model + description: |- + ### Deploy a pretrained image classifier model + 1. Open Seldon Console and click create. Deployment creation wizard appears. + 2. Choose type SeldonDeployment and protocol tensorflow and click next. + 3. Choose runtime tensorflow and enter the model url gs://seldon-models/tfserving/cifar10/resnet32 + 4. Click through wizard to deploy the model + summary: + success: You have deployed a pretrained tensorflow image classifier + failed: Try the steps again or see https://deploy-master.seldon.io/docs/demos/seldon-core/drift/ + - title: Add a drift detector + description: |- + ### Deploy a drift detector for the image classifier + 1. From the image classifier model's page, go to the drift detector wizard + 2. Enter the model name cifar10 and url gs://seldon-models/alibi-detect/cd/ks/cifar10-0_4_3 for a drift detector. + 3. Set the batch size to 2, protocol to tensorflow and port to 8080 + 3. Click through wizard to deploy the drift detector + summary: + success: The drift detector shows as loaded + failed: Try the steps again or see https://deploy-master.seldon.io/docs/demos/seldon-core/drift/ + - title: Make predictions + description: |- + ### Make predictions: + 1. Go to the make predictions screen + 2. Run a single prediction using the tensorflow payload format of an image truck - file is https://deploy.seldon.io/docs/demos/seldon-core/outlier/cifar10_image.json + 3. Run a prediction payload format of an outlier/perturbed truck - file is https://deploy.seldon.io/docs/demos/seldon-core/outlier/outlier_image.json + summary: + success: You see ok responses for each of the predictions. + failed: Try the steps again or see https://deploy-master.seldon.io/docs/demos/seldon-core/drift/ + - title: Monitor Drift + description: |- + ### Monitor Drift: + 1. Go to the monitor section of the deployment + 2. Under the ‘Monitor’ section of your deployment, you can see a timeline revealing drift of requests + summary: + success: You see graphs revealing drift of predictions. + failed: Try the steps again or see https://deploy-master.seldon.io/docs/demos/seldon-core/drift/ + conclusion: You are now able to identify drift in requests sent to the model. + nextQuickStart: []