lab/00_Warmup/01_BasicModel_Part1_TrainDeployTest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SageMaker Warmup Exercises\n",
    "\n",
    "This is a series of exercises for you to get familiar with the most important features SageMaker offers. It is divided in four parts:  \n",
    "\n",
    "- **Part 1**: This exercise. Let's explore a toy dataset (iris), train, deploy and test a ML model, using XGBoost\n",
    "- **Part 2**: After training your first ML model using SageMaker, now it's time to optimize it using Automatic Model Tuning\n",
    "- **Part 3**: Ah, you don't need to do real-time predictions, no problem. Let's do Batch Predictions\n",
    "- **Part 4**: Finally, let's use SageMaker Model Monitoring to get some info from the real-time endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1/4 - Train/Deploy/Test a multiclass model using SageMaker Built-in XGBoost\n",
    "\n",
    "This exercise is about executing all the steps of the Machine Learning development pipeline, using some features SageMaker offers. We'll use here a public dataset called iris. Iris is a toy dataset and this is a very simple example. The idea here is to focus on the SageMaker features and not on a complex scenario. Let's see how SageMaker can accelerate your work and avoid wasting your time with tasks that aren't related to your business. \n",
    "\n",
    "SageMaker library 2.0+ is required!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's  start by importing the dataset and visualize it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn import datasets\n",
    "sns.set(color_codes=True)\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "\n",
    "X=iris.data\n",
    "y=iris.target\n",
    "\n",
    "dataset = np.insert(iris.data, 0, iris.target,axis=1)\n",
    "\n",
    "df = pd.DataFrame(data=dataset, columns=['iris_id'] + iris.feature_names)\n",
    "df['species'] = df['iris_id'].map(lambda x: 'setosa' if x == 0 else 'versicolor' if x == 1 else 'virginica')\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking the class distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = df.groupby(df['species'])['species'].count().plot(kind='bar')\n",
    "x_offset = -0.05\n",
    "y_offset = 0\n",
    "for p in ax.patches:\n",
    "    b = p.get_bbox()\n",
    "    val = \"{}\".format(int(b.y1 + b.y0))\n",
    "    ax.annotate(val, ((b.x0 + b.x1)/2 + x_offset, b.y1 + y_offset))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr = df.corr()\n",
    "\n",
    "f, ax = plt.subplots(figsize=(15, 8))\n",
    "sns.heatmap(corr, annot=True, fmt=\"f\",\n",
    "            xticklabels=corr.columns.values,\n",
    "            yticklabels=corr.columns.values,\n",
    "            ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pairplots & histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(df.drop(['iris_id'], axis=1), hue='species', height=2.5, diag_kind=\"kde\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now with linear regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(df.drop(['iris_id'], axis=1), kind=\"reg\", hue='species', height=2.5,diag_kind=\"kde\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit a plot a kernel density estimate.\n",
    "We can see in this dimension an overlaping between **versicolor** and **virginica**. This is a better representation of what we identified above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_df = df[(df.iris_id==0.0)]\n",
    "sns.kdeplot(tmp_df['petal width (cm)'], tmp_df['petal length (cm)'], bw='silverman', cmap=\"Blues\", shade=False, shade_lowest=False)\n",
    "\n",
    "tmp_df = df[(df.iris_id==1.0)]\n",
    "sns.kdeplot(tmp_df['petal width (cm)'], tmp_df['petal length (cm)'], bw='silverman', cmap=\"Greens\", shade=False, shade_lowest=False)\n",
    "\n",
    "tmp_df = df[(df.iris_id==2.0)]\n",
    "sns.kdeplot(tmp_df['petal width (cm)'], tmp_df['petal length (cm)'], bw='silverman', cmap=\"Reds\", shade=False, shade_lowest=False)\n",
    "\n",
    "plt.xlabel('species')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. Petal length and petal width have the highest linear correlation with our label. Also, sepal width seems to be useless, considering the linear correlation with our label.\n",
    "\n",
    "Since versicolor and virginica cannot be split linearly, we need a more versatile algorithm to create a better classifier. In this case, we'll use XGBoost, a tree ensemble that can give us a good model for predicting the flower."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ok, now let's split the dataset into training and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42, stratify=y)\n",
    "yX_train = np.column_stack((y_train, X_train))\n",
    "yX_test = np.column_stack((y_test, X_test))\n",
    "np.savetxt(\"iris_train.csv\", yX_train, delimiter=\",\", fmt='%0.3f')\n",
    "np.savetxt(\"iris_test.csv\", yX_test, delimiter=\",\", fmt='%0.3f')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now it's time to train our model with the builtin algorithm XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "\n",
    "from sagemaker import get_execution_role\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "prefix='mlops/iris'\n",
    "# Retrieve the default bucket\n",
    "sagemaker_session = sagemaker.Session()\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "assert(sagemaker.__version__ >= \"2.0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ok. Let's continue, upload the dataset and train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload the dataset to an S3 bucket\n",
    "input_train = sagemaker_session.upload_data(path='iris_train.csv', key_prefix='%s/data' % prefix)\n",
    "input_test = sagemaker_session.upload_data(path='iris_test.csv', key_prefix='%s/data' % prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = sagemaker.inputs.TrainingInput(s3_data=input_train,content_type=\"csv\")\n",
    "test_data = sagemaker.inputs.TrainingInput(s3_data=input_test,content_type=\"csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the URI for new container\n",
    "container_uri = sagemaker.image_uris.retrieve('xgboost', boto3.Session().region_name, version='1.0-1')\n",
    "\n",
    "# Create the estimator\n",
    "xgb = sagemaker.estimator.Estimator(container_uri,\n",
    "                                    role, \n",
    "                                    instance_count=1, \n",
    "                                    instance_type='ml.m4.xlarge',\n",
    "                                    output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
    "                                    sagemaker_session=sagemaker_session)\n",
    "# Set the hyperparameters\n",
    "xgb.set_hyperparameters(eta=0.1,\n",
    "                        max_depth=10,\n",
    "                        gamma=4,\n",
    "                        num_class=len(np.unique(y)),\n",
    "                        alpha=10,\n",
    "                        min_child_weight=6,\n",
    "                        silent=0,\n",
    "                        objective='multi:softmax',\n",
    "                        num_round=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# takes around 3min 11s\n",
    "xgb.fit({'train': train_data, 'validation': test_data, })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the model and create an endpoint for it\n",
    "The following action will:\n",
    " * get the assets from the job we just ran and then create an input in the Models Catalog\n",
    " * create a endpoint configuration (a metadata for our final endpoint)\n",
    " * create an endpoint, which is our model wrapped in a format of a WebService\n",
    " \n",
    "After that we'll be able to call our deployed endpoint for doing predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# Enable log capturing in the endpoint\n",
    "data_capture_configuration = sagemaker.model_monitor.data_capture_config.DataCaptureConfig(\n",
    "    enable_capture=True, \n",
    "    sampling_percentage=100, \n",
    "    destination_s3_uri='s3://{}/{}/monitoring'.format(bucket, prefix), \n",
    "    sagemaker_session=sagemaker_session\n",
    ")\n",
    "xgb_predictor = xgb.deploy(\n",
    "    initial_instance_count=1, \n",
    "    instance_type='ml.m4.xlarge',\n",
    "    data_capture_config=data_capture_configuration\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alright, now that we have deployed the endpoint, with data capturing enabled, it's time to setup the monitor\n",
    "Let's start by configuring our predictor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.serializers import CSVSerializer\n",
    "from sklearn.metrics import f1_score\n",
    "csv_serializer = CSVSerializer()\n",
    "\n",
    "endpoint_name = xgb_predictor.endpoint_name\n",
    "model_name = boto3.client('sagemaker').describe_endpoint_config(\n",
    "    EndpointConfigName=endpoint_name\n",
    ")['ProductionVariants'][0]['ModelName']\n",
    "!echo $model_name > model_name.txt\n",
    "!echo $endpoint_name > endpoint_name.txt\n",
    "xgb_predictor.serializer = csv_serializer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now, let's do a basic test with the deployed endpoint\n",
    "In this test, we'll use a helper object called predictor. This object is always returned from a **Deploy** call. The predictor is just for testing purposes and we'll not use it inside our real application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_test = [ float(xgb_predictor.predict(x).decode('utf-8')) for x in X_test] \n",
    "score = f1_score(y_test,predictions_test,labels=[0.0,1.0,2.0],average='micro')\n",
    "\n",
    "print('F1 Score(micro): %.1f' % (score * 100.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Then, let's  test the API for our trained model\n",
    "This is how your application will call the endpoint. Using boto3 for getting a sagemaker runtime client and then we'll call invoke_endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.serializers import CSVSerializer\n",
    "csv_serializer = CSVSerializer()\n",
    "\n",
    "sm = boto3.client('sagemaker-runtime')\n",
    "resp = sm.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType='text/csv',\n",
    "    Body=csv_serializer.serialize(X_test[0])\n",
    ")\n",
    "prediction = float(resp['Body'].read().decode('utf-8'))\n",
    "print('Predicted class: %.1f for [%s]' % (prediction, csv_serializer.serialize(X_test[0])) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alright, now that you know how to train/deploy/test a model let's optimize it\n",
    "\n",
    "Click [here to start the Part 2/4](02_BasicModel_Part2_HyperparameterOptimization.ipynb) of this warmup: Hyperparameter Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning up (Attention! Read the message before deleting the Endpoint)\n",
    "Only run the next cell if you will **NOT** continue running the next part of the WarmUp. If you decide to continue, please, click on the link above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The end"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}