2. model_performance.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "%matplotlib inline \n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "=================================================================================================================\n",
    "# Lecture Notes: Model Performance\n",
    "\n",
    "\n",
    "##### D.Vidotto, Data Mining: JBI030 2019/2020\n",
    "\n",
    "\n",
    "=================================================================================================================\n",
    "\n",
    "In this lecture, we will explore several metrics for the evaluation of Machine Learning/Data Mining algorithms, which are tailored specifically for *Regression* and *Classification* tasks. It is important to be aware that evaluation metrics can be (and in general are) different from the *cost* (or *loss*) functions used when estimating the model (by means of optimization techniques). In this notebooks, we will deal with evaluation of model perfomance; in future lectures, we will see the algorithm-specific cost functions (which is assessed *after* the model is trained).    \n",
    "\n",
    "The topics covered in this lecture are the following ones: \n",
    " 1. performance measures in regression \n",
    "   * (mean) squared error\n",
    "   * (mean) absolute error\n",
    " 1. performance measures in classification\n",
    "   * accuracy\n",
    "   * confusion matrix \n",
    "   * Sensitivity, Specificity, Precision, Recall\n",
    "   * ROC and AUC\n",
    "   * Precision/Recall Curve\n",
    "   * $F_1$ score\n",
    "   * multiclass classification\n",
    " \n",
    " \n",
    "In this lecture (and the next ones), we are going to use the following notation. A rectangular data frame and its variables/observations will be denoted as follows: \n",
    "\n",
    "\n",
    "\n",
    "|Instance| $X_1$ | $X_2$ | ... |$X_j$ | $\\cdots$  | $X_p$ |  $Y$ | \n",
    "|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n",
    "|1| $x_{11}$ | $x_{12}$ | $\\cdots$  |$x_{1j}$ | $\\cdots$  | $x_{1p}$ | $y_{1}$ |\n",
    "|2| $x_{21}$ | $x_{22}$ | $\\cdots$  |$x_{2j}$ | $\\cdots$  | $x_{2p}$ | $y_{2}$ |\n",
    "|$\\vdots$ |  $\\vdots$    |  $\\vdots$ |$\\vdots$ | $\\vdots$ |  $\\vdots$ |  $\\vdots$ |  $\\vdots$  |\n",
    "|i| $x_{i1}$ | $x_{i2}$ | $\\cdots$  |$x_{ij}$ | $\\cdots$  | $x_{ip}$ | $y_{i}$ |\n",
    "|$\\vdots$ |  $\\vdots$    |  $\\vdots$ |$\\vdots$ | $\\vdots$ |  $\\vdots$ |  $\\vdots$ |  $\\vdots$  |\n",
    "|n| $x_{n1}$ | $x_{n2}$ | $\\cdots$  |$x_{nj}$ | $\\cdots$  | $x_{np}$ | $y_{n}$ |\n",
    "\n",
    "where $n$ is the number of observations, $p$ is the number of features, $i$ indexes the instances (the examples), $j$ indexes the features, and $Y$ is the output variable (we are in a *supervised learning* settings). Remember: in regression, the label $y_i \\in \\mathbb{R}$, while in classification, each label $y_i \\in \\{1,\\cdots,C\\}$ with $C \\in \\mathbb{N}$. The $p$-dimensional vector of observations for the $i$-th unit will be denoted by $\\mathbf{x}_i$. \n",
    "\n",
    "**Note**: features can also be referred to as predictors, input variables, attributes. Observations are also called units, data points, examples, or instances. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Performance Measures in Regression\n",
    "\n",
    "As observed during the introductory class, when we feed data into a Machine Learning model, the algorithm is trying to learn a function $f$; for example, in regression, we try to model the output $y$ with  \n",
    "\n",
    "$$ y_i = f(\\mathbf{x}_i) + e_i, $$\n",
    "\n",
    "that is, a map from the input space $X$ to the output space $Y$. Because it is in general impossible to learn the $y_i$'s exactly (as all ML algorithms make assumptions on the data), we allow for the presence of an error term $e_i$. Such term is define as the *discrepancy* between the observed output and the model prediction, i.e.   \n",
    "\n",
    "$$ e_i = y_i-\\hat{y}_i $$\n",
    "\n",
    "where $\\hat{y}_i = \\hat{f}(\\mathbf{x}_i)$. Error terms tell us how much the model is \"off\" the truth for each example: the smaller (i.e., close to 0) the errors, the better the model can predict it. \n",
    "<br> \n",
    "\n",
    "<img src=\"./img/model_performance/regression_residuals.png\" width=\"350\" height=\"50\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we are considering the full dataset (all observations from 1 to $n$) we use the sum of the errors for the global evaluation of the model. However, because negative and positive errors will tend to balance each other, such sum will be generally equal to 0. To obviate this, we take the sum of the *squared* errors. The function that measures the sum of the squared differences between the true observations and model predictions are called *sum of squares* (or **squared error**); formally, this is: \n",
    "\n",
    "$$SS(\\hat{y},y) = \\sum_{i=1}^{n} (y_i-\\hat{y}_i)^2$$\n",
    "\n",
    "Because what we want to do is to obtain SS that is as low as possible, what is typically done in machine learning is to minimize this value, w.r.t. a specific shape of the function $f(X)$ (for example, a linear model). Furthermore, $SS$ can be used to evaluate the model once the estimation step has been performed. Since $SS$ depends on the sample size, it is common to take the mean, to obtain what is known as the *mean squared error* (**MSE**). If, as a further step, we square-root of the MSE, we obtain the *root mean squared error* (**RMSE**): \n",
    "\n",
    "$$RMSE(\\hat{y},y) = \\sqrt{\\frac{1}{n}\\sum_{i=1}^{n} (y_i-\\hat{y}_i)^2}$$\n",
    "\n",
    "The RMSE can be nicely interpreted as the \"average error made by the model for each unit\", as it is in the same scale of $y$. An advantage of RMSE is that it is twice differentiable everywhere, and for some models (such as linear models) we can find the solutions for $\\hat{y}_i$ in close-form. \n",
    "<br> \n",
    "A disadvantage of RMSE is that it is very sensible to the presence of outliers, as the operation of squaring strongly inflates larger (positive and negative) values. An alternative to it is represented by the *mean absolute error* (**MAE**), which is an average of the *absolute values* of the error made by the model: \n",
    "\n",
    "$$ MAE(\\hat{y},y) = \\frac{1}{n}\\sum_{i=1}^{n} |{y_i-\\hat{y}_i}|  $$\n",
    "\n",
    "The issue with MAE is that it is not differentiable at 0, and thus performing optimization with such loss function requires more involved techniques. Nevertheless, MAE can be used as a performance metrics under a regression setting.\n",
    "<br>\n",
    "\n",
    "In what follows, you can see why MSE and MAE lead to different error sizes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATIAAAE9CAYAAACIkO4QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deVxVVfc/8M8GUVScQuvJSEBzBAQVp3KgNGfNMc00faxMe3JosOzbr0yfehosK20wrSzLIXMozUQFxSFxANJUnBMVtRQUBxQVWL8/1gURGS7cYZ9z73q/XvelwLnnLIa77t777L22IiIIIYSZeegOQAghbCWJTAhhepLIhBCmJ4lMCGF6ksiEEKYniUwIYXplHHHS6tWrU0BAgCNOLYRwY/Hx8SlEVCP/5x2SyAICAhAXF+eIUwsh3JhS6lhBn5eupRDC9CSRCSFMTxKZEML0HDJGJuzvxo0bSE5ORkZGhu5QRDG8vb3h5+cHLy8v3aG4DUlkJpGcnIxKlSohICAASind4YhCEBFSU1ORnJyMwMBA3eG4DelamkRGRgZ8fX0liRmcUgq+vr7ScnYySWQmIknMHOT35HySyITV3n77bQQFBaFx48YICwvDtm3bdIcEAEhKSkJwcHCBny9fvjzCwsJyH3PnztUQoXA0GSMTVomNjcWvv/6KhIQElCtXDikpKbh+/bpDr5mVlQVPT0+bzlGnTh3s3LmzRNex9rqZmZkoU0ZeQkagvUW2ezcwc6buKERxTp8+jerVq6NcuXIAgOrVq6NmzZoAgMjISDRo0ABNmzbF2LFj0aNHDwDAm2++iQ8++CD3HMHBwUhKSgIA9O7dG82aNUNQUBBmzZqVe4yPjw9efPFFhIaGIjY2FvHx8Wjfvj2aNWuGzp074/Tp0wCA+Ph4hIaGIjQ0FJ999lmJv5/81wkICMArr7yCpk2b4qeffsLOnTvRqlUrNG7cGH369MH58+cBABERERg/fjzCw8PxySeflPwHKQAA8+cDv/9uxxMSkd0fzZo1I2u99RYRQJSSYvVT3FJiYqLW61+6dIlCQ0Opbt26NHr0aIqJiSEioqtXr5Kfnx8dPHiQsrOzacCAAdS9e3ciIpo0aRJNnTo19xxBQUF09OhRIiJKTU0lIqIrV65QUFAQpVj+AADQjz/+SERE169fp9atW9OZM2eIiGjhwoX073//m4iIQkJCaMOGDURE9NJLL1FQUNBtMR89epS8vb0pNDQ097Fx48bbrkNE5O/vT++9917uxyEhIbnf4+uvv07jxo0jIqL27dvT6NGji/156f59GVlWFlH16kRPPFHy5wKIowJyjvZ28UMP8b8xMUC/flpDMY3x44FiekslFhYGfPxx4V/38fFBfHw8Nm3ahPXr12PgwIF49913ERYWhsDAQNStWxcAMGTIkFtaWIWZPn06li1bBgA4ceIEDh06BF9fX3h6eqKf5Q/hwIED2LNnDx5++GEA3OW7++67kZaWhrS0NLRr1w4AMHToUKxatarA6xTWtcx7nRwDBw4EAFy4cAFpaWlo3749AGDYsGEYMGDAbceJ0tmzB0hJufnatwftiSw8HPDxAaKjJZEZnaenJyIiIhAREYGQkBB89913CAsLK/T4MmXKIDs7O/fjnCkJMTExiIqKQmxsLCpUqICIiIjcr3l7e+eOTxERgoKCEBsbe8t509LSbP5e8l4nR8WKFa16rrXHiYJFR/O/LpXIvLyA9u2Bdet0R2IeRbWcHOXAgQPw8PDIbXnt3LkT/v7+aNCgAZKSknDkyBHUqVMHCxYsyH1OQEAAfv31VwBAQkICjh49CoBbPNWqVUOFChWwf/9+bN26tcBr1q9fH2fPnkVsbCxat26NGzdu4ODBgwgKCkLVqlWxefNmtGnTBvPmzbPr91qlShVUq1YNmzZtQtu2bfH999/nts6E7datA+rWBe69137n1J7IAM7MK1cCJ08C99yjOxpRkMuXL2PMmDFIS0tDmTJlcN9992HWrFnw9vbGrFmz0L17d1SoUAFt27bFpUuXAAD9+vXD3LlzERQUhJYtW6JevXoAgC5dumDmzJlo2LAh6tevj1atWhV4zbJly2Lx4sUYO3YsLly4gMzMTIwfPx5BQUGYM2cORowYAaUUOnXqVGjcR44cuaXVOGLECIwdO7bY7/e7777DqFGjcOXKFdSuXRtz5swpyY9LFCIzE9iwARg82M4nLmjgzNZHSQb7iYj++IMH/OfOLfngn7swy+Dx+vXrcwf73ZlZfl/OFhvLr/U891lKBIUM9muffgEAjRsDvr7SvRTC1eW8xh980L7nNUTX0sODv7HoaIAIkBUe5pVzM0CIgkRHc8Olxm3Fqm1jiBYZwONkJ04AR47ojkQI4QgZGTwJ1p53K3MYJpF16MD/SvdSCNcUGwtcu3bztW5PhklkdevyHcuoKN2RCCEcIToa8PQE2ra1/7kNk8iUAjp25BZZnjmUQggXERUFtGgBVKli/3MbJpEBnMhSU4Fdu3RHIgrz888/QymF/fv3534uJiYmd6G4LYYPH47FixcXeUxMTAy2bNlSovPGxMSgSpUqt5TziZKmv1OlpQE7dvBr3BEMlchyBgHlb8y4FixYgDZt2twyg9+ZSpPIAKBt27bYuXNn7qNjvlcUEd2ynArgtZ3WsPY4dxYTwz0tt0hkNWsCjRrdXIsljOXy5cvYvHkzvv76ayxcuPCWr128eBHdu3dH/fr1MWrUKGRnZyMrKwvDhw9HcHAwQkJC8NFHHwFAoSVy8goICEBKSgoAIC4uDhEREUhKSsLMmTPx0UcfISwsDJs2bcLZs2fRr18/NG/eHM2bN8fvJagNk5SUhPr16+OJJ55AcHAwTpw4cVt5n+joaDRp0gQhISEYMWIErl27lhtf3rI/omjR0UCFCkAhizhsZoh5ZHl17AjMns13Nyylr4RB/PLLL+jSpQvq1asHX19fxMfHo1mzZgCA7du3IzExEf7+/ujSpQuWLl2KwMBAnDx5Env27AFwc7H3E088gRkzZqB9+/Z44403MHnyZHxsxQLSgIAAjBo1Cj4+PnjppZcAAIMHD8bzzz+PNm3a4Pjx4+jcuTP27dt323M3bdp0y1KlJUuWwNPTE4cOHcJ3332Xu0wqPT0dLVu2xIcffoiMjAzUrVsX0dHRqFevHp544gl88cUXGD9+PADA19cXCQkJNvxE3UdUFNCuHVC2rGPOb8hENn0636qVeZWF0FHHB9ytHDduHABg0KBBWLBgQW4ia9GiBWrXrg0AeOyxx7B582Z06NABf/31F8aMGYPu3bujU6dOxZbIKamoqCgkJibmfnzx4kVcvnwZPj4+txzXtm3b3AXsOZKSkuDv73/LWs/8ZYQCAwNz14gOGzYMn332WW4ik3I+1klOBvbvB556ynHXMFwia9+eb9FGRUkiM5Jz585h3bp12L17N5RSyMrKglIKU6dOBXD7hhtKKVSrVg27du3C6tWrMXPmTCxatCi3e1mcvCWAitqRKDs7G1u3boW3t3epvq/8JXkKKu9j7XNFwXKGihw1PgYYMJFVrsy3aKOjgbfe0h2NQWmo47N48WIMHToUX375Ze7n2rdvj02bNgHgruXRo0fh7++PH3/8ESNHjkRKSgrKli2Lfv36oX79+hgyZIjVJXICAgIQHx+Prl27YsmSJbmfr1SpEi5evJj7cadOnTBjxgxMmDABAI+/FVUjrSTq16+PpKQkHD58GPfdd5+U8yml6GhekhQS4rhrGGqwP0fHjsD27XzLVhjDggUL0KdPn1s+169fv9y7l82bN8dzzz2Hhg0bIjAwEH369MHJkycRERGBsLAwDBkyBO+88w4ALpEzYcIENG7cGDt37sQbb7xx2/UmTZqEcePGITw8/JYWUs+ePbFs2bLcwf7p06cjLi4OjRs3RqNGjTCzkA0gcsbIch7FTfMAuHU2Z84cDBgwACEhIfDw8MCoUaOs/pkJXjsdFcWz+T0cmG0UV8awr/DwcIqLiyv18zdu5C7msmVA7952DMzE9u3bh4YNG+oOQ1hJfl9s714gOBj46ivgySdtP59SKp6IwvN/3pAtslatgIoVgTVrdEcihLBFzmvYsu2CwxgykZUtywP9a9fqjkQIYYu1a4F69YBatRx7HUMmMoAz+OHDgGUbRCGEyVy7xmWtHd0aA6xMZEqp55VSe5VSe5RSC5RSpbvXXQI537y0ym5yxHimsD/5PbHYWODKFYMkMqXUPQDGAggnomAAngAGOTqwhg25rI+MkzFvb2+kpqbKi8TgiAipqamlntfmStau5Tmh9i5rXRBr55GVAVBeKXUDQAUApxwXElOKM/kvvwBZWfwDcWd+fn5ITk7G2bNndYciiuHt7Q0/Pz/dYWi3di3fuKtc2fHXKjaREdFJpdQHAI4DuApgDRE5pZ308MPAt98CCQlA8+bOuKJxeXl5ITAwUHcYQljl3DkgLg6YNMk517Oma1kNwCMAAgHUBFBRKTWkgONGKqXilFJx9mo15CxpkHEyIcxl3TqeDOuM8THAusH+jgCOEtFZIroBYCmA+/MfRESziCiciMJr2GmLlDvv5LXMMk4mhLmsWXNzuaEzWJPIjgNopZSqoHhlcAcAt9dJcZBOnYAtWwDL5tVCCIMjAlav5mVJZZy0mrvYREZE2wAsBpAAYLflObMcHFeuzp2BGze4wqQQwvgOHgSOH+fXrrNYNY+MiCYRUQMiCiaioUR0zdGB5XjgAa4suXq1s64ohLBFzmu1UyfnXdOwM/tzlCvHy5VknEwIc1i9mrd3dOZNdsMnMoCbqIcOAUeP6o5ECFGUa9d4GMiZ3UrAJIksp4kq3UshjO3333lZkjO7lYBJEln9+rx6XrqXQhjb6tWAl5dzliXlZYpEphQ3VaOj+Q6mEMKYVq/mG3T59n5xOFMkMoCbqhcvAtu26Y5ECFGQv/8Gdu1yfrcSMFEi69iRF45HRuqORAhRkJyhny5dnH9t0ySyqlV5Jb0kMiGMadUq4K67gNBQ51/bNIkMALp2BeLjgTNndEcihMgrK4tbZF26OHa3pMKYKpHlNFllGoYQxhIXx6V7dHQrAZMlsiZNeKNP6V4KYSyRkTeLoepgqkTm4cHTMFav5qasEMIYIiO5ZI+vr57rmyqRATxOlprKVWOFEPqlpvK0qK5d9cVgukT28MPchF21SnckQgiAKzgT6RsfA0yYyGrUAMLDJZEJYRSRkcAdd/DrUhfTJTKAm7DbtgEpKbojEcK9ZWdzo6JzZ707nZkykXXrxk1ZWUQuhF4JCTyvs1s3vXGYMpGFhwPVq0v3UgjdfvvtZlEHnUyZyDw9eWAxMlKmYQih02+/8bQLO22cVmqmTGQAN2VTUnhGsRDC+c6eBbZv19+tBEycyDp14gmyv/2mOxIh3NOaNTxWLYnMBr6+XA1DEpkQevz2G2+i3bSp7khMnMgAfieIiwP++Ud3JEK4l6wsHqPWVe0iPwOEUHo5TVpZRC6Ec23bxtUujNCtBEyeyMLCgJo1gZUrdUcihHtZuZJnD+iedpHD1IlMKaB7d66Gcf267miEcB+//gq0acOVm43A1IkM4ER28SKwebPuSIRwDydOAH/+ya89ozB9IuvQAShXTrqXQjhLzmutRw+9ceRl+kTm4wNERHBTVwjheCtXAoGBQIMGuiO5yfSJDOB3hoMHgUOHdEcihGu7cgWIiuLXnFK6o7nJJRJZTl9dupdCONb69UBGhrG6lYCLJLLAQKBRI+leCuFoK1cCFSsC7dvrjuRWLpHIAH6H2LABuHBBdyRCuCYibix07Mg32IzEZRJZz55AZqbseSmEo+zaxVMvevXSHcntXCaRtW7NC8lXrNAdiRCuafnym5PQjcZlEpmnJ/+AV67klpkQwr5WrABatgTuukt3JLdzmUQGcPfy/HlgyxbdkQjhWk6d4kozRuxWAi6WyDp3BsqWle6lEPaWMyOgZ0+9cRTGpRJZpUo8y3/5ct2RCOFali/naU5BQbojKZhLJTKA3zEOHgQOHNAdiRCuIT0diI7m15aRZvPn5ZKJDJDupRD2EhXFs/mN2q0EXDCR+fsDoaHAL7/ojkQI1/DLL0CVKsabzZ+XyyUyAHjkEb5zeeaM7kiEMLesLB7o79YN8PLSHU3hXDKR9e4NZGfL2kshbBUby/tX9u6tO5KiuWQiCwsDatWS7qUQtvr5Z26JdemiO5KiuWQiU4on7q1dy/WThBAlR8SNgYceAipX1h1N0axKZEqpqkqpxUqp/UqpfUqp1o4OzFa9ewNXr/JuyEKIktu3Dzh82PjdSsD6FtknACKJqAGAUAD7HBeSfbRrxzu8SPdSiNLJee0YdVlSXmWKO0ApVQVAOwDDAYCIrgMw/OZrXl58p2XFCl5EXqbY71QIkdcvvwDNm/PesUZnTYssEMBZAHOUUn8opb5SSlV0cFx20bs3kJoK/P677kiEMJeTJ3k3cTN0KwHrElkZAE0BfEFETQCkA5iY/yCl1EilVJxSKu7s2bN2DrN0unblSpbLlumORAhzyelW9umjNw5rWZPIkgEkE9E2y8eLwYntFkQ0i4jCiSi8Ro0a9oyx1Hx8gE6dOJER6Y5GCPNYuhSoXx9o2FB3JNYpNpER0d8ATiil6ls+1QFAokOjsqM+fYDjx4GEBN2RCGEO584BMTFA3766I7GetXctxwCYp5T6E0AYgP85LiT76tkT8PCQ7qUQ1lqxgpcmmaVbCViZyIhop6Xb2JiIehPReUcHZi/Vq/NiV0lkQlhn2TLAzw8ID9cdifVccmZ/fn36AImJUqNMiOKkp/NOZL17G7f2WEHcIpHl3EKWVpkQRYuM5NpjZhofA9wkkd17L0/sW7pUdyRCGNvSpbytYtu2uiMpGbdIZADQvz+wYwdw7JjuSIQwpmvXeKC/d2/zrYRxm0TWrx//K60yIQq2di1w6dLN14qZuE0iq1OHS2AvXqw7EiGMackSLmndoYPuSErObRIZwN3LLVt4s1EhxE03bvCypF69eG9Ys3GrRJbTZJa7l0Lcav164Px5frM3I7dKZA0b8kO6l0LcasmSm2uTzcitEhnA7zgbN8oOS0LkyMri2vw9egDe3rqjKR23TGTZ2dK9FCJHzhu7Ge9W5nC7RBYSAtSrByxapDsSIYxh0SKgQgWuqGxWbpfIlAIefZTLlEj3Uri7zEweH+vZk5OZWbldIgOAAQO4eymTY4W727iRN+AdMEB3JLZxy0QWEsLVL3/6SXckQui1aBFQsSKXhTczt0xkSvE7kHQvhTvLzOReSY8e5u5WAm6ayAAeJ5PupXBnGzZwt/LRR3VHYju3TWTBwdy9/PFH3ZEIoYerdCsBN05kSgEDB/K70unTuqMRwrlu3OAVLr16AeXL647Gdm6byABOZESyZEm4n+ho3i1p0CDdkdiHWyeyRo2Axo2BhQt1RyKEcy1cyCV7OnfWHYl9uHUiA/gdacsWqRwr3EdGBi/R69sXKFdOdzT24faJbOBA/leWLAl3ERkJXLzoOt1KQBIZatcGWrSQ7qVwHwsX8n6vDz2kOxL7cftEBvA7U0ICcPCg7kiEcKz0dN5gpH9/820wUhRJZOAJgUpJq0y4vuXLgStXXKtbCUgiAwDccw8QEQHMm8fTMYRwVfPnA35+5tu3sjiSyCwGD+auZUKC7kiEcIyUFB7of+wxwMPFXvku9u2UXr9+vHvMvHm6IxHCMRYv5oXijz+uOxL7k0RmUa0aV8hcuJBrmAvhaubPvzkJ3NVIIstj8GBed7lhg+5IhLCvY8eATZv4b1wp3dHYnySyPHr04C2xpHspXE3OHfnBg/XG4SiSyPIoX56XbSxezMs4hHAFRMAPPwCtWwOBgbqjcQxJZPkMHcrLN1as0B2JEPaxaxewZw//bbsqSWT5PPggULMm8P33uiMRwj6+/x7w8nKNSrCFkUSWj6cn355etYrLAAthZpmZfLeyWzfA11d3NI4jiawAQ4fyH4CUwRZmFx0N/P23a3crAUlkBQoJAUJDpXspzO/774GqVfmOvCuTRFaIoUOB7duBAwd0RyJE6Vy+zAUUH33UdQooFkYSWSEGD+b1aNIqE2a1dClXunD1biUgiaxQd98NdOrEiSw7W3c0QpTct99y4dAHHtAdieNJIivCsGHA8eO8I7kQZnLsGLB+PTB8uGsuScpPElkRHnmEd5r59lvdkQhRMnPn8r/u0K0EJJEVqXx53pxkyRLg0iXd0QhhHSLgu+94cndAgO5onEMSWTGGD+cBU9nEV5jF778DR47w3667kERWjFatgHr1pHspzOPbb4GKFbkAgruwOpEppTyVUn8opX51ZEBGoxQP+m/cCBw+rDsaIYqWns57tA4YwCWp3EVJWmTjAOxzVCBGNmwYzymTVpkwusWLeTx3xAjdkTiXVYlMKeUHoDuArxwbjjHdcw/QpQsnMimDLYzs66+BunWBNm10R+Jc1rbIPgbwMgC3nRr65JPAyZPAmjW6IxGiYAcPcjnrESPcY+5YXsUmMqVUDwBniCi+mONGKqXilFJxZ12w/k2PHkCNGvyOJ4QRzZnDZaiGDdMdifNZ0yJ7AEAvpVQSgIUAHlJK/ZD/ICKaRUThRBReo0YNO4epX9myPLlw+XKpUyaMJzOT54517crL69xNsYmMiF4lIj8iCgAwCMA6Ihri8MgMaMQI4MYNrn8uhJFERvIOYE8+qTsSPWQeWQkEBfG8stmzefa0EEYxezZw551A9+66I9GjRImMiGKIyMVLtBXt6aeBffuALVt0RyIEO3kSWLmSewxeXrqj0UNaZCU0cCBQqRIwa5buSIRgc+bwtKCnntIdiT6SyEqoYkXenGTRIuD8ed3RCHeXnQ189RXQoQNQp47uaPSRRFYKI0fyBr6yI7nQbe1arj02cqTuSPSSRFYKTZoAzZpx91IG/YVOs2YB1atz7Tx3JomslEaOBHbvBrZu1R2JcFenT/O8xmHDXH9zkeJIIiulwYN50P+LL3RHItzVV1/xRNhnntEdiX6SyErJx4dn+i9aBKSm6o5GuJvMTO5WPvwwLxJ3d5LIbDB6NHDtGt/+FsKZVq4EkpP5b1BIIrNJcDCXS5k5U7aME871xRdcXqpnT92RGIMkMhuNHs310aOjdUci3MVffwGrV/MqkzJldEdjDJLIbNSvH5f3+fxz3ZEIdzFzJpfrceeZ/PlJIrNRuXL8B7V8OU9MFMKRrlzhu5V9+nDXUjBJZHYwahT/O3Om3jiE61u4kJfGPfec7kiMRRKZHdSqBfTuzaVUrl7VHY1wVUTAjBl8k6ldO93RGIskMjt57jmeT/bjj7ojEa5qyxZg505gzBj3q8lfHElkdhIRwYUXZ8yQ9ZfCMT79FKhShauviFtJIrMTpbhVlpAAxMbqjka4mlOneM/KESO4lJS4lSQyOxo6FKhaFfjkE92RCFfz+edcPPE//9EdiTFJIrOjihW5KsaSJcDx47qjEa7i6lW+I/7II+5dPLEoksjsLOcd89NP9cYhXMcPP/CNpPHjdUdiXJLI7KxWLZ7tP3s2cPmy7miE2REBH38MhIXJlIui6E9k69bx4FJKiu5I7Gb8eCAtDZg7V3ckwuyiooDERP6bcpkpF1euABMmAF9/bbdT6k9k+/fzdOVGjXgSlgvMXWjVCmjRgt9JpSqGsMVHH/F+lYMG6Y7ETmJigMaNgQ8+4Ne+nehPZM8+y3MW/P35t9W7N2/UZ2JKAS++CBw6BKxYoTsaYVZ79gCrVvEEWNOXsr5wgUvZPvggf7xuHTB1qt1Orz+RAUBICE+++uAD3hYmKIhXxpq4dda3LxAQwN+SEKUxbRpQvrwLFE/89debr+kXXwT+/PNmQrMTYyQygAsr5XyTTZpwsaUOHbjYlwmVKQM8/zywebNsUCJK7vRpvls5YgTg66s7mlI6e5Y3t+jZE6hW7WZjpUIFu1/KOIksx333cZXCL78E4uO5tTZtGs8GNJkRI3iCrLTKRElNn851+Z9/XnckpUAELFjA496LFwOTJ/NruUULh13SeIkMADw8eGbp3r3cKnvxReD++3nQwER8fLhbsHQpcPiw7miEWVy+zBNg+/Y14QTY5GSgVy9uidWuzePfb7wBlC3r0MsaM5Hl8PPjioULFnB936ZNObtfv647MquNGQN4eQEffqg7EmEWs2fz9J0JE3RHUgLZ2bytU1AQ96imTeNyHcHBzrk+Edn90axZM7K7s2eJBg8mAoiCg4m2b7f/NRxk5EiicuWITp/WHYkwuowMonvuIYqI0B1JCRw6xAEDRA8+SHT4sMMuBSCOCsg5xm6R5VW9OjBvHrfQzp/nyVovvcST6wxuwgTgxg2eVyZEUebN49lHEyfqjsQKmZk8ABwSwl3I2bO5NaajP1xQdrP14ZAWWV5paUTPPMPvAHXqEK1f79jr2cHAgUSVK3PoQhQkM5OoXj2iJk2IsrN1R1OMP/8kat6cX4O9ehElJzvlsjB9iyyvKlV4NHT9ev74wQd5st2FC3rjKsIrrwAXL8puS6JwP/8MHDzIrTHDLke6dg2YNInHq5OSeFXOzz/r3wmloOxm68PhLbK80tOJXnqJyMODBxeWL3fetUuoSxeiGjU4ZCHyys4mataM6L77uGVmSFu3EgUFcStsyBAet3YyuFSLLK8KFXipw9atwB133Lz1e/as7shu83//x2HNnq07EmE0q1bxVKuJE3nPSkNJTwdeeAFo3Zp7PStXAt9/z+PWRlFQdrP14dQWWV7XrhFNnkzk5UXk60s0b57hBhvatyeqWZPo6lXdkQijyM4matWKqFYt/hM2lKgoosBAboWNHk104YLWcOCyLbK8ypblyXd//MErBB5/nFtoycm6I8v1+utcf33OHN2RCKOIjuYOxcSJDp83ar20NF4m2LEjr7fbsIEHeCtX1h1ZwQrKbrY+tLXI8srMJJo2jahCBaJKlYhmziTKytIdFWVnE7VubdB3X6FFu3YGa6X//DPR3XfzuPPLLxNduaI7olxwixZZXp6evFBt925e4zVqFPDQQ1xbRyOluFV2/LgUXhTAxo38ePllwNtbczBnztwspVWjBrBtGxO8W10AABfYSURBVPDee1yCw+gKym62PgzRIssrO5to9myiKlWIvL2Jpk4lunFDazjNmxP5+0urzN1FRBD961+a72RnZxP98AOPK5ctSzRlimH/MOF2LbK8lAKeeoprBnfuzFPtW7fmkkGawpkyBTh2TMbK3Nn69Vww9dVXHVLZxjonTgA9egBDhgB16/L48uuvG2iwzjrukchy1KwJLFvGJbWPHQOaNeObA9euOT2Uzp05l771lpbLC82I+E/vnnu40IvTZWcDX3zBi7xjYnj93ObNXHrHhNwrkQHcHHr0UW6dDRwI/Pe/PEvZydUPc1plyclcOFO4l6gozhuvvaZhbOzgQV4N8+yzQMuWXB5r3DgDTmCznvslshzVq3MJzpUree3Q/ffzzYH0dKeF0KED0LYt8L//8Saswj3ktMZq1eLim06TmQm8/z4QGgrs2sW7GK1ZAwQGOjEIx3DfRJajWzcu4DhqFDevQ0J4Yo8TKMUNwlOngM8+c8olhQGsWMEdgNdfd+KmIrt2cevrlVeALl24RzJihIEXdZaMJDKAJ/l9/jlP+itThicBPv00Twp0sPbt+e/qnXcMveZd2ElWFi9Vq1cPGD7cCRe8do0zZng4j2P89BOXLK5Z0wkXd55iE5lS6l6l1HqlVKJSaq9SapwzAtOiXTt+53r5ZeCbb3jg85dfHH7Z//0POHdOavu7g/nzuQPw1lv8nulQsbG8kc9bb/H648REoH9/l2mF3aKgORl5HwDuBtDU8v9KAA4CaFTUcww3j6w0duwgatyY15gNHEj0zz8OvdyjjxJVrEj0998OvYzQKCODKCCAqGlTBy8yuXSJaNw4IqV4CcmqVQ68mHOhtPPIiOg0ESVY/n8JwD4AmosPOUF4OBAXx+9my5YBDRvyzQFyzF6b//0vkJHBlxOu6csvuYTXO+/w/joOsXYtj/N+8gnvfLNnD49duLqCslthDwABAI4DqFzUcS7RIssrMZEXSAJEXbsSHTvmkMs88wxRmTJEBw445PRCo/PneeL8Qw85qCDLuXNE//43/43Wq0e0caMDLqIfCmmRlSSJ+QCIB9C3kK+PBBAHIK5WrVrO/e6cITOT6JNPeBG6jw/R55/bvX9w+jSfum9fu55WGMArr3BPLyHBASdfupTXOXl6Er36qoFWn9ufTYkMgBeA1QBesOZ4l2uR5fXXX0QPP8w/urZt7d58mjKFT71pk11PKzRKSuJdtIYOtfOJ//6bqH9//oMJCyOKj7fzBYynsERmzV1LBeBrAPuIaJp9O7YmFBgIrF7NdzV37+bJhe+/z5MN7eCFF/jO+IQJDhuOE072//4f3yi02/gnEZdOadiQJ6W9/TawfTuvUHFXBWU3urU11gYAAfgTwE7Lo1tRz3HpFllep04R9enD74hNmxLt3GmX037zDZ9y/ny7nE5otH07/y4nTrTTCZOSePMHgOj++4n27bPTic0Bto6RleThNoksx08/Ed11F4/Uv/aazWMUmZmcF/38ZKMSM8spYX3XXUQXL9p4sqwsok8/5UHUihWJpk83RKFQZysskcnMfnvo358nGz7+ODfzmzTh7eJLydOTV0slJ3OvVZjT/Pm8FOmdd4BKlWw40YEDvATkued4TfCePcCYMQ6cw2E+8pOwlzvuAL79FoiM5N3P27ThigKXL5fqdG3bcnGO99/narLCXNLTeVljs2bAsGGlPMmNG8C77/I47N69N/++AgLsGKlrkERmb5078zvms88C06fz5MS1a0t1qvff53HdCRPsHKNwuHffBU6e5HmppWo4/fEHL/J+9VWgZ09u8Q8b5prLi+xAEpkjVKoEfPopsGkTlzfo1IkrDZw/X6LT1KrFf8eLFjmtIIewg0OH+E1o8GDggQdK+OSMDF5V3rw5l0VZsoQXev/rXw6J1WUUNHBm68PtBvuLcvUqT1L09ORJi0uXlvjpdeoQ1a/Pa/WEsWVnE3XqRFS5Mt/ULpHNm/kXDRANH86z9cUtIIP9mnh7c3mLHTv4XbVvX2DAAODvv61++owZPN770UcOjlXYbOlSrlU4ZQpw991WPunSJR68b9uWW2SrV/NmDtWqOTRWl1JQdrP1IS2yQly/TvS///E072rViL791uqFd3368OqopCQHxyhK7dIlnjITGlqCTboiI7lChVJEY8fySUShIC0yA/Dy4kGvnTu51tnw4UDXrrwRSjE+/pjHef/zH5nxb1Svv85TZj7/3IpaY+fO8eB9ly5AxYpcwP+TTwAfH6fE6mokkenQoAHvyjpjBv8BBwXxzYHs7EKfUqsWL3FZuZIH/4Wx7NjBN6lHj+apXkVasoTfyObP5/VLf/xhxZNEkQpqptn6kK5lCSQlEXXuzAO8DzxQ5JKTzEyi8HCiO+8kSk11YoyiSNevc3eyZk2itLQiDjx9mkub5Cxp++MPp8XoKiBdS4Py9wdWrQK++47nCoWF8VTwGzduO9TTE5g9G0hNlbllRjJtGldI//RToEqVAg4g4smsDRtyk/rdd4Ft2/h3LeyjoOxm60NaZKWUvyxLIcWrXnmFD4mMdHJ84jaJiXzvptAackeP3iz71KaNVM20EaRFZgJ33cWTH5cs4ekZzZvzzYGMjFsOe/NNfnN/6inZeUmnzEy+X+PjwwP8t8jK4jHQ4GDeBOSzz3iXrnr1dITq8iSRGVHfvtzNfOKJm2vtNm/O/bK3N/dUTp0CXnxRX5ju7sMPuQzYp5/ye1Cufft4R66xY3lu2N69vGRNFnk7jPxkjapaNS7euHo1703Yrh1Pmrx0CQDQogWPk339NQ+xCefau5d3C+/blxf3A+Bxzbff5rGv/fu5+OFvv/EtZ+FYBfU3bX3IGJmdXbrEkyVztveyDI5dvUoUFMQrn86c0RyjG8nI4LuUNWrk2b4vLo4/CRANGCD7+jkIZIzMxHx8eLLk5s1AhQo8iXLYMHhfOYf583lu5dNPy0RZZ3n9db5L+c03wF2VrwITJ3KlijNneOvARYvy9TWFo0kiM5P77+fJk6+9xpMpGzZE44OL8c47vCH611/rDtD1rV/PO8KPGgX0qLyRxy/fe49H/RMTgd69dYfoliSRmY23N0/x37ED8PMDBgzA85v6YkCb0xg3jseZhWOkpABDhwJhtS9ietZ/uGprZiYQFQV89RVQtaruEN2WJDKzCgvjSZXvvgsVuQoLdzfCk55z8OgAwpUruoNzPdnZvDSy6T+rsDU9GF5ffQGMH887aXXooDs8tyeJzMzKlOF6yrt2waNxCKZfGoEP93bGlH8f1R2Zy/lscgoG/TYUyzO7oewdlXhPho8+4gXfQjtJZK6gXj0gJgb4/HO0LxuL1xcFY8eQT3hSprANEfZPWYSBUxrhMbUQ9PobQEIC0KqV7shEHpLIXIWHBzB6NDz37cXuau3RfN54pDdtK4Nmtjh1Cle79UWDSQPxdzl/XN0UDzVlMpcvF4YiiczFlKldC4GJKzG22ve4vucAKCyMbw4UsAhdFIII+OYbUKNG8FgTide83ofH1lhUeqCx7shEISSRuaC7/qXw+KohCPHchw3V+vDEp/BwID5ed2jGd/Qo8PDDwJNP4lDFMARl70boDxMQHFZcpUShkyQyF9WyJTD5izvx4D8LMbv7z8DZs/zJV14Brl7VHZ7xZGVxGd7gYGD7dsQMmokGp9ah38v34dFHdQcnilXQdH9bH7JEyTief55Xzcyeep7oySf5g7p1iTZs0B2acezdS9SqFf9sunendXNPkIcHUa9eXMxSGAdkiZJ7mjqV93cdNbEqIvt/xZM3MzN5MuezzwIXL+oOUZ/r14H//pfn5B06BPzwA/a8swKP/McPISHAvHlczFIYnyQyF+fpyauZgoOB/v2BuCodeBLn+PHAzJn8BXcsn7FjB48bvvEG0K8fsG8fjrd9HF26Kvj4ACtWyD4gZiKJzA34+HA1mRo1eNOmgycr8mTOLVt4V/Ru3XjtTWqq7lAd78oV4OWXeR5YaiovUl2wAKkeNdC5M1dJiowE7r1Xd6CiJCSRuYmaNbm0GQB07gycPAl+MSckcKtk4UIuO7tokeuW0YiJ4UXeU6dyed3ERKBXL1y+DPTowTcsV6wAGsssC9ORROZG6tXjlllKCvDQQ5bNzsuVAyZP5qkZ/v5cJbBPHy4/6youXOByFQ8+yEl63Trgyy+BKlWQng507849zQULuH6lMB9JZG6meXMeEjt5kpPZmTOWLzRuzLXlp07lplujRlzRweyts5Ured/Q2bOBF14A/vyTExp4FkqvXlzm7YcfOH8Lc5JE5obatAF+/RVISuJkdvq05QtlygAvvcQ3A8LCuFpjx47AX3/pDLd0zp4FHn+c+4zVqnGS/vBDLkwJID2dk9j69bz/waBBesMVtpFE5qYiIm4ms7ZtgWPH8nzxvvu4+zVzJve5QkL45oAZFqET8Xhfo0a8I9WkSdxtbtEi95C0NKBTJ/4Wv/2W73MIkytocpmtD5kQax6xsURVqxL5+fEejbc5cYKoRw+eLNqyJdGePU6P0WrJyUQ9e3KszZsT7d592yF//03UpAmRlxfR4sUaYhQ2gUyIFQVp1Ypv5t24wZW0Y2LyHeDnByxfzpPRjhwBmjQBpkzhyaRGQcRjYI0a8YTfDz/krmRw8C2HJSby93vgAH9L/fppilfYnSQygdBQYOtW4O67ucs1d26+A5QCHnuMM0H//txdCw/nbqduR45whdaRI4FmzXh874UXbpuSv24dJ+qMDGDjRt6/RbgOSWQCABAQwPNj27blks7PP19A5Z8aNbhltnw5b93UqhVvrqmjtnZWFjBtGo/fxccDs2YB0dFAnTq3HEbEw3udOnHjcutWznfCxRTU37T1IWNk5nX9OtG4cTzM1K4d0enThRyYlkb09NN84H33EcXEOC/I3buJWrTga/fsyWNjBbh0iWjgQD6sb1+iCxecF6JwDBQyRiaJTBRo3jyi8uWJqlcnWr68iAPXrSOqU4f/lJ55hhOco1y7RjRpEo/UV69OtGABUXZ2gYdu3cpheXgQvftuoYcJk5FEJkosMZEoLIz/SkaOLKJFk55O9OKLnDXuuYdoxQr7B7NtG2+rDhA9/jjR2bMFHnbtGtGbbxJ5ehL5+xNt3Gj/UIQ+kshEqWRkEE2YQKQUUc2aREuXFnHwtm1EwcH8ZzV4MNGZM7YHkJ5O9MILnCT9/Ih+/bXQQzdvJmrU6Gauc2TjUOghiUzYZNs2otBQ/ovp2rWI6WQ5TaKc7t/8+aXv161bR1S7Nl901KhCm4QnThANG8aH1apVZK4TJieJTNjs+nWiDz4gqlKFG0hPPUX011+FHGzlgHyBrLyRcOYM0cSJRN7eROXKEb38Mg/wC9cliUzYTUoK0fjx3Ojy9ORu3B9/FHBgZibRtGl816ByZaIvvyTKyir65MuXcx/Ww4P7tFeu3HbIkSNEY8bwaZXi6ycl2ed7E8ZmUyID0AXAAQCHAUws7nhJZO4hOZmHrypW5L+k8HDOVSkp+Q48fJjooYf4oIgIokOHbj/ZP/8QDRrEx4SEEO3YccuXL13im5QdOvAhZcoQDR9OtG+f474/YTylTmQAPAEcAVAbQFkAuwA0Kuo5ksjcy7lzRJ98cnOc39OT89UHH/DY2vXrxONks2Zxy6x8ef7isWNEx48Tff89ka8vN/EmTya6do2ysrh3+vnnvNSzXDk+t78/0ZQpJeupCtdRWCJT/LXCKaVaA3iTiDpbPn7VMpH2ncKeEx4eTnFxcaWcoivMiogLzv78M7BsGbB3L3++QgUuCdagAdDkzpPos2Y0AnavuOW5/wS2xC89v8b29CDs38/PTUvjr/n7A71786NdO95UXbgnpVQ8EYXf9nkrEll/AF2I6CnLx0MBtCSi5/IdNxLASACoVatWs2O31IUR7ujUKS5a+PvvvExz/34gORkACB0RhVo4DgA4j2r4BY8gG56oUYMrbjdsCLRuzUumAgN5uacQhSUyu22fTESzAMwCuEVmr/MK86pZE3j0Udyywe2NG0BqqkJKysO5BTQ8PYEZ1QFfX8DbW0+swtysSWQnAeTdU8bP8jkhSszLC/jXv/ghhL1YM9qwA0BdpVSgUqosgEEAljs2LCGEsF6xLTIiylRKPQdgNfgO5jdEtNfhkQkhhJWsGiMjot8A/ObgWIQQolTkRrYQwvQkkQkhTE8SmRDC9CSRCSFMTxKZEML0JJEJIUxPEpkQwvSKXTReqpMqdRZASVaNVweQYvdAHMMssUqc9mWWOAHzxFqaOP2JqEb+TzokkZWUUiquoBXtRmSWWCVO+zJLnIB5YrVnnNK1FEKYniQyIYTpGSWRzdIdQAmYJVaJ077MEidgnljtFqchxsiEEMIWRmmRCSFEqRkmkSml/quU+lMptVMptUYpVVN3TAVRSk1VSu23xLpMKVVVd0yFUUoNUErtVUplK6UMdxdLKdVFKXVAKXVYKTVRdzwFUUp9o5Q6o5TaozuWoiil7lVKrVdKJVp+5+N0x1QYpZS3Umq7UmqXJdbJNp/TKF1LpVRlIrpo+f9Y8JZzozSHdRulVCcA6ywFJ98DACJ6RXNYBVJKNQSQDeBLAC8RkWG2tlJKeQI4COBhAMngSsSPEVGi1sDyUUq1A3AZwFwiCtYdT2GUUncDuJuIEpRSlQDEA+httJ8nACilFICKRHRZKeUFYDOAcUS0tbTnNEyLLCeJWVQEYIwMmw8RrSGiTMuHW8F7GBgSEe0jogO64yhECwCHiegvIroOYCGARzTHdBsi2gjgnO44ikNEp4kowfL/SwD2AbhHb1QFs2xRednyoZflYdPr3TCJDACUUm8rpU4AeBzAG7rjscIIAKt0B2FS9wA4kefjZBj0hWc2SqkAAE0AbNMbSeGUUp5KqZ0AzgBYS0Q2xerURKaUilJK7Sng8QgAENFrRHQvgHkAniv6bPritBzzGoBMS6zaWBOrcB9KKR8ASwCMz9fLMRQiyiKiMHCPpoVSyqZuu932tbQGEXW08tB54D0CJjkwnEIVF6dSajiAHgA6kOZBxhL8TI1Gthm0M8t40xIA84hoqe54rEFEaUqp9QC6ACj1DRXDdC2VUnXzfPgIgP26YimKUqoLgJcB9CKiK7rjMTHZZtCOLAPoXwPYR0TTdMdTFKVUjZy7/Uqp8uAbPja93o1013IJgPrgu2zHAIwiIsO9QyulDgMoByDV8qmtRry7CgBKqT4AZgCoASANwE4i6qw3qpuUUt0AfIyb2wy+rTmk2yilFgCIAFdq+AfAJCL6WmtQBVBKtQGwCcBu8GsIAP7PsgOaoSilGgP4Dvx79wCwiIim2HROoyQyIYQoLcN0LYUQorQkkQkhTE8SmRDC9CSRCSFMTxKZEML0JJGJYimlsixVSXIeDqtUoZSqoJSap5TabVmhsNkyWx1KqS2Ouq4wN6fO7BemddWynKRQSilPIsrK83GZPIvri3pe/uPGAfiHiEIsX68P4AYAENH9pYpeuDxpkYlSU0olKaXeU0olABiglIpRSn2slIoDME4pFaCUWmep3RatlKpled63SqmZSqltAN7Pd9q7kWepEhEdIKJrluddtvw7JU/r8KRSao7l80Msda52KqW+tJQKEm5AEpmwRvl8XcuBeb6WSkRNiWih5eOyRBRORB+CVxV8R0SNwetnp+d5nh+A+4nohXzX+gbAK0qpWKXUW/mWrgEAiOgNSwsxAlxi51NL7bWBAB6wfC0LXEVFuAHpWgprFNW1/LGIj1sD6Gv5//e4tfX1U96uaA4i2qmUqg2gE4COAHYopVoT0b68x1nWFv4AYBoRxSulngPQzHI8AJQHl4gRbkASmbBVejEfW/u8XJaie0sBLFVKZQPoBi4UmNebAJKJaI7lYwVu/b1q5fWFC5GupXCkLeCqFgB38zYV9wSl1ANKqWqW/5cF0AhcRCDvMT3BrbWxeT4dDaC/UupOyzF3KKX8bf4OhClIi0xYo7ylmmeOSCKyZgrGGABzlFITAJwF8G8rnlMHwBeWrqMHgJXgGlt5vQCuJrvd0o1cTkRvKKX+H4A1SikP8J3O/yBfEhSuSapfCCFMT7qWQgjTk0QmhDA9SWRCCNOTRCaEMD1JZEII05NEJoQwPUlkQgjTk0QmhDC9/w9K4Dafnl0KrAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5,5))\n",
    "plt.plot(np.linspace(-3,3,100), np.square(np.linspace(-3,3,100)), \"b-\", label=\"Squared Error\")\n",
    "plt.plot(np.linspace(-3,3,100), np.absolute(np.linspace(-3,3,100)), \"r-\", label=\"Absolute Error\")\n",
    "plt.xlabel(\"Error Size\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another common measure of fit in regression is the $R^2$, a.k.a. Coefficient of determination. It can be described as the percentage of variance in the output variable that is explained by the model, or - equivalently - as the square of the Pearson correlation between the observed $y$ and the predicted $\\hat{y}$:\n",
    "\n",
    "$$R^2 = corr(y, \\hat{y})^2$$\n",
    "\n",
    "$R^2$ ranges between 0 and 1, where values close to 1 indicate a better fit. Thus, unlike MAE and RMSE, the goal is to *maximize*, rather than minimize, $R^2$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now going to evaluate a multiple linear regression model on a real-data example. We *train* (which really means *fitting*) the model on the *Boston house-price* dataset, which is included in *scikit-learn*. For the training, we will use the *training set* created with `train_test_split` (already encountered during the Data Preprocessing lecture); the models will then be evaluated on the *test set*. The goal is to predict new house prices given the input features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAW4UlEQVR4nO3de5RlZX3m8e8jIKCg3Erk1rYKXjDGxmkRvAXxElBH0OWY0cRggtOakURnmSgysxLMUgdnjKhLo9MGhIyIEhUloFEEDGIQ0kiLIDgogtI2dHNpATUY8Dd/7LfkdFmX01WnqtjU97PWWXX25ez92+855zlvvXufqlQVkqT+edBiFyBJmh0DXJJ6ygCXpJ4ywCWppwxwSeopA1ySesoAF0meneR7C7zP1yd5/4i3eVWSQ0a97lwkWZ6kkmw9D9veNsk1ScZGvN3jkvzdKLep+WGAb6Ek1yd5/oR5r01y0WLVNFGr594kdyW5I8naJC+Zav2q+npVPX4B63sw8D+A/92mRxJyVfWkqvraqNddKEkOSXLjsOtX1d3AycCxW7CP45N8YpL5lWTftt13V9XrhtjW15LMuJ7mjwH+wHVxVe0A7AScBJyRZOeJK81Hz3AIRwDXVNW6YR+wSHX2wSeBo5Jsu9iFjJLP93AM8HmQ5Imtd7Kp/ar+0oFlm/VaBnvv6ZyYZEPrOX8nyW+1ZdsmeW+SHyW5OclHk2w/Uy1V9Su6Xtr2wGPHe3lJ3pbkJuDjE3t+SfZJ8rkkG5PcmuRDA8v+OMnVSW5P8uUkj5qp9kkcDvzzwPSF7eem9lvDwa1dvtG2eStwfJLHJjm/1XRLktOS7DRQ269/O2o9zTOS/H2SO9vzsHKW6z41yeVt2T8k+XSSd052YEm2as/TLUmuA148Yfkftfa7M8l1SV7f5j8U+BKwZ2uDu5LsmeTAJBe319L6JB9qv8GMP783ArcDB03R1ltssJeeZLskn2htvinJvybZPcm7gGcDH2q1fqit/4y2zk/bz2cMbPfRSS5sx/7VJB8e2M/4b2FHJ/kRcH6b/w9JbmrbuzDJkwa2d0qSv03ypVbDN5I8Msn72+vzmiQHjKpd7o8M8BFLsg3wj8BXgEcAfwqclmSYIYoXAs8BHgc8HHglcGtbdkKbvwLYF9gL+Msh6tkaeB1wF3Btm/1IYBfgUcCqCetvBZwN3AAsb/v5VFt2BHAc8HJgDPg6cPoQtU/0ZGBwzP057edOVbVDVV3cpp8OXAfsDrwLCPA/gT2BJwL7AMdPc/gvbbXvBJwFfGhL121heSZwCl2bnQ68bJrt/BfgJcABwErgFROWb2jLHwb8EXBikqdW1c/oPth+0tpgh6r6CXAv8N+A3YCDgecB/3XCNq8GntLqXdaCdtk0NW6Jo+iez32AXYE3AL+oqv9O9/wf02o9JskuwDnAB9u67wPOSbJr29YngUvbsuOB10yyv9+he25/t01/CdiP7r30LeC0Ceu/km44bjfgbuDitt5uwGdaDQ9cVeVtC27A9XRhuGng9nPgorb82cBNwIMGHnM6cHy7/zXgdQPLXjvw2EOB/0fXmxp8fICfAY8dmHcw8MMpanwtcE+r7Rbgm8Dz27JDgF8C2w2sfwhw48B2NwJbT7LdLwFHD0w/qB37o6aqfYr6rgUOG5heDtTgPtsx/GiG7RwJXD7huRk/zuOBrw4s258ueLZoXboPl3VABpZfBLxziprOB94wMP3Cicc2Yf3PA2+a+DxMc8xvBs6cMO804C+HfP0e357/TRNuBew7sM4n2v0/Bv4F+O1JtvU1Nn8tvwa4dMI6F7fncll7TT5kYNknBvYz/hp4zDS179TWeXibPgX42MDyPwWuHph+MrBpNu/zvtzsgc/OkVW10/iNzXtEewI/rm7oYtwNdD3ZaVXV+XQ9vw8DG5KsTvIwut7uQ4DLWu9qE/BPbf5Uvtnq262qDqqqrw4s21hV/zbF4/YBbqiqeyZZ9ijgAwM13Eb34bLXNLVP5nZgx2lqH/fjwYn2q/unkqxLcgddAOw2zeNvGrj/c2C7TD22OtW6ewLrqiXCZHVNsOeE5TdMOIbDk3wzyW2tDV803TEkeVySs9swwh3AuydZf0e6EB7WGYOv3/Yansr/Bb4MfCrJT5L8r/Zb5mT2ZMLxct9rf0/gtqr6+cCyydrx1/PacNQJSX7Qjv36tmjw+G8euP+LSaZ3mKLWBwQDfPR+AuyTZLBtl9H14qDrST9kYNkjBx9cVR+sqv9A1wt8HPAXdL3oXwBPGnjTPby6k5SzMd2foPwxsGyKoPsx8PoJb/7tq+pfpql9Mle05TPVM3H+u9u8J1fVw4A/oPsAmU/rgb2SDO5nnxnWH1z+66GMdCcaPwu8F9i9BecXue8YJmuHjwDXAPu1Yz6O3zzmJwLfnvlQtlxV/XtVvaOq9geeQTf884dT1PsTug/5QeOv/fXALkkGX/uTtePgNl9Nd8L7+XTDOMvb/Pl+znvDAB+9S+h6cG9Nsk26a43/I20cGVgLvDzJQ9JdtnX0+AOTPC3J01sP52fAvwG/ar35j9GNlz6irbtXkt9l9C6le7OdkOSh7STWM9uyjwJvHz+RlOThSf7TdLVPsY8v0o11jtvY1n3MDLXtSDd89dMkezH1B8QoXUw3Dn1Mkq3beYADp1n/DODPkuyd7qqfwUv8HgxsS3e89yQ5nG6IZdzNwK5JHj4wb0fgDuCuJE8A/mRwZ60ddqEbJhu5JM9N8uR2buQO4N+573m9mc2fsy8Cj0vy6tZWv0f3YX52Vd0ArKE7Gf3gJAfTvS+msyPduPatdJ2ed4/swB4gDPARq6pf0r0wD6frOf8t8IdVdU1b5US6McibgVPZ/KTMw+iC+na6Xz1vpV0rDbwN+D7wzfbr5FeBkV+7XVX3tvr3BX4E3Aj8Xlt2JvAeul+n7wCubMc5U+0T/SPwhCR7tu3+nO4k5Tfa8MxUV1S8A3gq8FO6k2Wfm/2RDqc9ny+n+6DdRNfrP5suWCbzMbohh2/TnUz7dY1VdSfwZ3QhfztdD/OsgeXX0J0vua61w57An7f17mzb/vSE/b0aOLW6a8LHT2LeNcKTmI+kOxl4B93J0n+mG1YB+ADwinbFxwer6la6Hvpb6J7/twIvqapb2vq/T3eO5Vbgne1YpmpHgL+ney2tA77LPH1I9Vk2H9qTFkaSVcD+VfXmxa5lSyW5BPhoVX18kevYlu6D4jlVtWExa5mNJJ+m+z7AXy12LX1lgEszSPI7dJc93kLXi/wo3dUS6xe1sJ5J8jS6E98/pBs6+jxwcFVdvqiF9ZjfdpJm9ni6YY+H0l2X/grDe1YeSTektCvd0NyfGN5zYw9cknrKk5iS1FMLOoSy22671fLlyxdyl5LUe5dddtktVfUbX9xb0ABfvnw5a9asWchdSlLvJZn4DVfAIRRJ6i0DXJJ6ygCXpJ4ywCWppwxwSeopA1ySesoAl6SeMsAlqacMcEnqKf8aoX7D8mPPWbR9X3/Cixdt31Lf2AOXpJ4ywCWppwxwSeopA1ySesoAl6SeMsAlqacMcEnqqRkDPMl2SS5N8u0kVyV5R5t/SpIfJlnbbivmv1xJ0rhhvshzN3BoVd2VZBvgoiRfasv+oqo+M3/lSZKmMmOAV1UBd7XJbdqt5rMoSdLMhhoDT7JVkrXABuDcqrqkLXpXkiuSnJhk2ykeuyrJmiRrNm7cOKKyJUlDBXhV3VtVK4C9gQOT/BbwduAJwNOAXYC3TfHY1VW1sqpWjo2NjahsSdIWXYVSVZuAC4DDqmp9de4GPg4cOB8FSpImN8xVKGNJdmr3twdeAFyTZI82L8CRwJXzWagkaXPDXIWyB3Bqkq3oAv+Mqjo7yflJxoAAa4E3zGOdkqQJhrkK5QrggEnmHzovFUmShuI3MSWppwxwSeopA1ySesoAl6SeMsAlqacMcEnqKQNcknrKAJeknjLAJamnDHBJ6ikDXJJ6ygCXpJ4ywCWppwxwSeopA1ySesoAl6SeMsAlqacMcEnqKQNcknpqmP9Kv12SS5N8O8lVSd7R5j86ySVJvp/k00kePP/lSpLGDdMDvxs4tKqeAqwADktyEPAe4MSq2he4HTh6/sqUJE00Y4BX5642uU27FXAo8Jk2/1TgyHmpUJI0qaHGwJNslWQtsAE4F/gBsKmq7mmr3AjsNcVjVyVZk2TNxo0bR1GzJIkhA7yq7q2qFcDewIHAE4bdQVWtrqqVVbVybGxslmVKkibaoqtQqmoTcAFwMLBTkq3bor2BdSOuTZI0jWGuQhlLslO7vz3wAuBquiB/RVvtKOAL81WkJOk3bT3zKuwBnJpkK7rAP6Oqzk7yXeBTSd4JXA6cNI91SpImmDHAq+oK4IBJ5l9HNx4uSVoEfhNTknrKAJeknjLAJamnDHBJ6ikDXJJ6ygCXpJ4ywCWppwxwSeopA1ySesoAl6SeMsAlqacMcEnqKQNcknrKAJeknjLAJamnDHBJ6ikDXJJ6ygCXpJ4ywCWpp4b5r/T7JLkgyXeTXJXkTW3+8UnWJVnbbi+a/3IlSeOG+a/09wBvqapvJdkRuCzJuW3ZiVX13vkrT5I0lWH+K/16YH27f2eSq4G95rswSdL0tmgMPMly4ADgkjbrmCRXJDk5yc5TPGZVkjVJ1mzcuHFOxUqS7jN0gCfZAfgs8OaqugP4CPBYYAVdD/1vJntcVa2uqpVVtXJsbGwEJUuSYMgAT7INXXifVlWfA6iqm6vq3qr6FfAx4MD5K1OSNNEwV6EEOAm4uqreNzB/j4HVXgZcOfryJElTGeYqlGcCrwG+k2Rtm3cc8KokK4ACrgdePy8VSpImNcxVKBcBmWTRF0dfjiRpWH4TU5J6ygCXpJ4ywCWppwxwSeopA1ySesoAl6SeMsAlqacMcEnqKQNcknrKAJeknjLAJamnDHBJ6ikDXJJ6ygCXpJ4ywCWppwxwSeopA1ySesoAl6SeMsAlqaeG+a/0+yS5IMl3k1yV5E1t/i5Jzk1ybfu58/yXK0kaN0wP/B7gLVW1P3AQ8MYk+wPHAudV1X7AeW1akrRAZgzwqlpfVd9q9+8Ergb2Ao4ATm2rnQocOV9FSpJ+0xaNgSdZDhwAXALsXlXr26KbgN2neMyqJGuSrNm4ceMcSpUkDRo6wJPsAHwWeHNV3TG4rKoKqMkeV1Wrq2plVa0cGxubU7GSpPsMFeBJtqEL79Oq6nNt9s1J9mjL9wA2zE+JkqTJDHMVSoCTgKur6n0Di84Cjmr3jwK+MPryJElT2XqIdZ4JvAb4TpK1bd5xwAnAGUmOBm4AXjk/JUqSJjNjgFfVRUCmWPy80ZajQcuPPWexS5B0P+Y3MSWppwxwSeopA1ySesoAl6SeMsAlqacMcEnqKQNcknrKAJeknjLAJamnDHBJ6ikDXJJ6ygCXpJ4ywCWppwxwSeopA1ySesoAl6SeMsAlqacMcEnqKQNcknpqmP9Kf3KSDUmuHJh3fJJ1Sda224vmt0xJ0kTD9MBPAQ6bZP6JVbWi3b442rIkSTOZMcCr6kLgtgWoRZK0BeYyBn5MkivaEMvOU62UZFWSNUnWbNy4cQ67kyQNmm2AfwR4LLACWA/8zVQrVtXqqlpZVSvHxsZmuTtJ0kSzCvCqurmq7q2qXwEfAw4cbVmSpJnMKsCT7DEw+TLgyqnWlSTNj61nWiHJ6cAhwG5JbgT+CjgkyQqggOuB189jjZKkScwY4FX1qklmnzQPtUiStoDfxJSknjLAJamnDHBJ6ikDXJJ6ygCXpJ4ywCWppwxwSeopA1ySesoAl6SeMsAlqacMcEnqKQNcknrKAJeknjLAJamnDHBJ6ikDXJJ6asZ/6CAtpOXHnrMo+73+hBcvyn6lubAHLkk9ZYBLUk/NGOBJTk6yIcmVA/N2SXJukmvbz53nt0xJ0kTD9MBPAQ6bMO9Y4Lyq2g84r01LkhbQjAFeVRcCt02YfQRwart/KnDkiOuSJM1gtleh7F5V69v9m4Ddp1oxySpgFcCyZctmubvFtVhXRkjSdOZ8ErOqCqhplq+uqpVVtXJsbGyuu5MkNbMN8JuT7AHQfm4YXUmSpGHMNsDPAo5q948CvjCaciRJwxrmMsLTgYuBxye5McnRwAnAC5JcCzy/TUuSFtCMJzGr6lVTLHreiGuRJG0Bv4kpST1lgEtSTxngktRTBrgk9ZQBLkk9ZYBLUk8Z4JLUUwa4JPWUAS5JPWWAS1JPGeCS1FMGuCT1lAEuST1lgEtSTxngktRTBrgk9dRs/yu9pBFZfuw5i7Lf60948aLsV6NjD1ySesoAl6SemtMQSpLrgTuBe4F7qmrlKIqSJM1sFGPgz62qW0awHUnSFnAIRZJ6aq498AK+kqSA/1NVqyeukGQVsApg2bJls97RYp2pl/TAsZg5Mh9X/cy1B/6sqnoqcDjwxiTPmbhCVa2uqpVVtXJsbGyOu5MkjZtTgFfVuvZzA3AmcOAoipIkzWzWAZ7koUl2HL8PvBC4clSFSZKmN5cx8N2BM5OMb+eTVfVPI6lKkjSjWQd4VV0HPGWEtUiLxpPk6iMvI5SknjLAJamnDHBJ6ikDXJJ6ygCXpJ7yHzpIS9QD7WvlS5E9cEnqKQNcknrKAJeknjLAJamnDHBJ6ikDXJJ6ygCXpJ4ywCWppwxwSeopA1ySesqv0ktacP4DjdGwBy5JPWWAS1JPzSnAkxyW5HtJvp/k2FEVJUma2awDPMlWwIeBw4H9gVcl2X9UhUmSpjeXHviBwPer6rqq+iXwKeCI0ZQlSZrJXK5C2Qv48cD0jcDTJ66UZBWwqk3eleR7c9jn/cFuwC2LXcT9iO1xH9tic7bHgLxnTu3xqMlmzvtlhFW1Glg93/tZKEnWVNXKxa7j/sL2uI9tsTnbY3Pz0R5zGUJZB+wzML13mydJWgBzCfB/BfZL8ugkDwb+M3DWaMqSJM1k1kMoVXVPkmOALwNbASdX1VUjq+z+6wEzHDQitsd9bIvN2R6bG3l7pKpGvU1J0gLwm5iS1FMGuCT1lAE+jSQnJ9mQ5MqBebskOTfJte3nzotZ40JJsk+SC5J8N8lVSd7U5i/V9tguyaVJvt3a4x1t/qOTXNL+vMSn2wn+JSHJVkkuT3J2m17KbXF9ku8kWZtkTZs38veKAT69U4DDJsw7FjivqvYDzmvTS8E9wFuqan/gIOCN7U8nLNX2uBs4tKqeAqwADktyEPAe4MSq2he4HTh6EWtcaG8Crh6YXsptAfDcqloxcO33yN8rBvg0qupC4LYJs48ATm33TwWOXNCiFklVra+qb7X7d9K9Ufdi6bZHVdVdbXKbdivgUOAzbf6SaY8kewMvBv6uTYcl2hbTGPl7xQDfcrtX1fp2/yZg98UsZjEkWQ4cAFzCEm6PNmSwFtgAnAv8ANhUVfe0VW6k+5BbCt4PvBX4VZvelaXbFtB9mH8lyWXtz4nAPLxX/I88c1BVlWRJXYeZZAfgs8Cbq+qOrqPVWWrtUVX3AiuS7AScCTxhkUtaFEleAmyoqsuSHLLY9dxPPKuq1iV5BHBukmsGF47qvWIPfMvdnGQPgPZzwyLXs2CSbEMX3qdV1efa7CXbHuOqahNwAXAwsFOS8Y7RUvnzEs8EXprkerq/Snoo8AGWZlsAUFXr2s8NdB/uBzIP7xUDfMudBRzV7h8FfGERa1kwbUzzJODqqnrfwKKl2h5jredNku2BF9CdF7gAeEVbbUm0R1W9var2rqrldH9S4/yq+n2WYFsAJHlokh3H7wMvBK5kHt4rfhNzGklOBw6h+7OYNwN/BXweOANYBtwAvLKqJp7ofMBJ8izg68B3uG+c8zi6cfCl2B6/TXciaiu6jtAZVfXXSR5D1wvdBbgc+IOqunvxKl1YbQjlz6vqJUu1Ldpxn9kmtwY+WVXvSrIrI36vGOCS1FMOoUhSTxngktRTBrgk9ZQBLkk9ZYBLUk8Z4JLUUwa4JPXU/wdZZ19klR+ChwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.metrics import r2_score\n",
    "from sklearn.datasets import load_boston\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LinearRegression\n",
    "X, y = load_boston(return_X_y=True)\n",
    "# Split the dataset retaining 80% of the data for the training\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.8, random_state=1)\n",
    "# Show the histogram of house prices\n",
    "pd.Series(y_train).hist(grid=False)\n",
    "plt.title(\"House Prices (training data): Histogram\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now fit the model (`scikit-learn` trains the linear regression model by minimizing the sum-of-squares):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: here we are training a linear regression model without performing any data preprocessing \n",
    "# In real world applications, data should be preprocessed first!\n",
    "# Remember the step of a sklearn model: 1.initialize; 2.train; 3.predict\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(X_train,y_train)\n",
    "y_pred = lin_reg.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the RMSE/MAE/$R^2$ metrics to evaluate the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'RMSE': 4.814941555935472,\n",
       " 'MAE': 3.4962308560549102,\n",
       " 'R2:': 0.7219648298245811}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rmse = np.sqrt(mean_squared_error(y_test,y_pred))\n",
    "mae = mean_absolute_error(y_test,y_pred)\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "{\"RMSE\":rmse, \"MAE\":mae, \"R2:\":r2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`scikit-learn` models also come automatically with a `score()` method, which usually returns the $R^2$ coefficient:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7219648298245811"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lin_reg.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: while we are using here RMSE, MAE and $R^2$ together for introducing the three metrics, in real world applications you should choose beforehand one metric and stick with it all along the analysis; that is, different models must be compared using always the same metric. You can learn about more regression metrics in the [scikit-learn documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Performance Measures in Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike regression, in classification we are interested in predicting *labels* rather than numbers. For example, we might interested in predicting wether a person is with/without heart disease, if an e-mail can be considered ham/spam, and so on. In this case, we can *map* each label to a number, but the value of the number in itself is meaningless: we could choose, for example, to set *presence of heart disease* to 0 and *absence of heart disease* to 1; equivalently, we could choose *presence of heart diseae* = 103, and *absence of hear disease* = -54. What matters are the meaning of the labels, rather than the numbers. For classification, we will use the convention that $y_i \\in \\{1,...,C\\}$ with $C \\in \\mathbb{N}$ when we need to predict an output with $C$ possible labels. In the special case where $C=2$, we will refer (without loss of generality) to *positive class* and *negative class* for the two classes, where the positive one is considered the class of interest.  <br>\n",
    "In classification, we are interested in detecting **decision regions** in the feature space. That is, given a point $\\mathbf{x}$ in the feature space, we are interested in knowing to what label $\\{1,...,C\\}$ the classifier must assign the point. We will explore different classifiers during the course, and each classifier produces predictions in a different way; how labels are predicted is the subject of the coming lectures. The different regions are separated by the **decision boundary**. The decision boundary are all those points in the feature space that have the same probability to belong to two (or more) classes. In what follows, you will see a simulated example of *decision regions* and *decision boundary* for a binary (two-class) case, with two features, $x_1$ and $x_2$. The decision regions are highlighted by different colors in the background, and represent the model predictions. \n",
    "\n",
    "**Note**: in this example, classification is performed with a method called *k-nearest-neighbors* (knn). We will explore such model later on in the course. For the moment, you just need to know roughly how this algorithm works: given a point $\\mathbf{x}$ in the feature space: knn (1) takes the $k$ closest data points (\"neighbors\") to $\\mathbf{x}$, (2) evaluates their labels, and (3) assigns to $\\mathbf{x}$ the most frequent label across such points. In knn, $k$ is a *hyperparameter* that can vary and, in general, must be tuned. In this example, we will use $k=5$ (the default in `scikit-learn`). You can read more about the implementation of knn in the [scikit-learn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from matplotlib.colors import ListedColormap\n",
    "\n",
    "# We randomly generate a two-class, two-features dataset with 500 instances \n",
    "X, y = make_classification(n_samples=500, n_features=2, n_redundant=0, n_informative=1,\n",
    "                             n_clusters_per_class=1, random_state=1)\n",
    "# Retain 80% of the samples in the training dataset \n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)\n",
    "# Fit knn (default: k=5)\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train, y_train)\n",
    "y_pred = knn.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This algorithm produces the following decision boundary and regions: \n",
    "\n",
    "<img src=\"./img/model_performance/knn_binary.png\" width=\"2000\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: in what follows, and until section 2.7 (`Multiclass classification`), we will assume we are in the two-class case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is reasonable, in classification, to consider \"errors\" all those units which are miss-classified by the model. That is, those units which are in the 'wrong side' of the decision boundary; in the example above, we can detect the miss-classified instances by observing which data points have one color, but are located in a region of opposite color. These units are the 'incorrectly classified instances' by the model (knn in this case); they can be mathematically denoted as all those $i$ for which $y_i \\neq \\hat{y}_i$. Conversely, correctly classified instances can be indicated as all those units $i$ for which $y_i = \\hat{y}_i$. \n",
    "<br> \n",
    "When evaluating a classifier, a typical performance measure is given by the **miss-classification rate**:  \n",
    "\n",
    "$$ \\frac{1}{n}\\sum_{i=1}^{n} \\mathcal{I}(y_i \\neq \\hat{y}_i) $$\n",
    "\n",
    "which ranges between 0 (no misclassifications at all, or perfect classifications), and 1 (all units are miss-classified). Here, $\\mathcal{I}(condition)$ is the indicator function:\n",
    "\n",
    "$$\\mathcal{I}(condition) = \n",
    "\\left\\{\n",
    "\\begin{array}{rl}\n",
    "1 & if\\ condition\\ is\\ True \\\\\n",
    "0 & otherwise\n",
    "\\end{array}\n",
    "\\right.\n",
    "$$\n",
    "\n",
    "\n",
    "Equivalently, a classifier can be evaluated with the **accuracy** measure, which is symmetric to the miss-classification rate: \n",
    "\n",
    "$$Accuracy = \\frac{1}{n}\\sum_{i=1}^{n} \\mathcal{I}(y_i = \\hat{y}_i).  $$\n",
    "\n",
    "Accuracy also varies between $0$ and $1$, but of course the interpretation of its values must be flipped w.r.t. the miss-classification rate. Clearly, the goal is to maximize accuracy (that is, achieving as many accurate predictions as possible), or equivalently to minimize the miss-classification rate (that is, achieving as few errors as possible).  \n",
    "\n",
    "`scikit-learn` allows evaluting the accuracy of a classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.89"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that 89% of the (test) data points were correctly classified by knn. Classifiers in scikit-learn also have a built-in `score()` method (see the scikit-learn documentation for knn linked above), which reports the accuracy: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.89"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`scikit-learn` also allows picking different evaluation functions for classification. You can find more about it in the [documentation](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics). In what follows, we are going to explore some of them. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Confusion matrix \n",
    "\n",
    "While accuracy is a good way to obtain a single measure of model performance, it is also not very informative. In cases of imbalanced datasets (as we have seen during the Data Pre-Processing lecures), resorting to accuracy can be misleading: if there was a class observed 99% of the time, a classifier that always predicts that class (without using any information from the predictors) would always get 99% accuracy. Furthermore, accuracy doesn't tell us in *what class* the miss-classifications occur more frequently. Is it the positive labels that are mostly classified as negative, or the other way around?  For instance, we had 11% miss-classified instances by knn in the example above: which one is the most mis-classified class?\n",
    "\n",
    "Knowing this information can help us to better tailor our classifiers, and correct them towards one or the other direction. Possibly, such information can also help us to revise our data preprocessing strategy in a new stage of analysis refining. \n",
    "\n",
    "Fortunately, there is a simple tool that allows us retrieving such information: the confusion matrix! It is a $C \\times C$  (in our example: 2 $\\times$ 2) matrix which displays the true class in the rows, and the predicted class in the columns. In the cells of the matrix we find a count of how many units observed within each class (rows) are predicted to be in a specific class by the model (columns). Therefore, the diagonal elements of the matrix display the number of points correctly classified by the model, and the off-diagonal tell us how many units observed in the class of row $i$ are classified in the column of the row $j$. For a two-class case, the confusion matrix looks as follows: \n",
    "\n",
    "|            | Predicted Positive | Predicted Negative  |\n",
    "|:---:|:---:|:---:|\n",
    "|Observed Positive      | TP | FN |\n",
    "|Observed Negative     | FP  | TN |\n",
    "\n",
    "where: \n",
    " * TP = number of *true positives* \n",
    " * FN = number of *false negatives* \n",
    " * FP = number of *false positives* \n",
    " * TN = number of *true negatives*\n",
    " \n",
    "For example, the false negatives are the points that are classified in the negative class, but actually belong to the positive one. The sum of the rows gives information about how many instances belong to each class; the sum of the columns gives information about how many instances have been classified to each class. \n",
    "\n",
    "Clearly, a classifier with 100% accuracy has the off-diagonal elements of the confusion matrix all equals to 0; maximizing the accuracy then corresponds to maximizing the elements in the diagonal of the confusion matrix. In scikit-learn we can visualize the confusion matrix for our example in this way: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[42,  5],\n",
       "       [ 6, 47]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should be aware of the fact that `scikit-learn` order the rows and columns numerically; therefore, the first row of the matrix correspond to the units having label '0', and the second row to units having label '1', and similarly for the columns. Therefore, if we take class '1' as being the positive class, we must swap the order of rows and columns of the confusion matrix seen above; for example, in this case knn classifies 42 true negatives, and 6 false negatives. \n",
    "\n",
    "We can also obtain a 'normalized' version of the confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.42, 0.05],\n",
       "       [0.06, 0.47]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_test, y_pred, normalize='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which tells us that 5% of the test units are false positives (once again, considering class 1 as the 'positive' class). We can also visualize the confusion matrix graphically:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATwAAAEGCAYAAAD45CnNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3debxXVb3/8df7HAQVZBJUBhFELHBCOELmzTRFUQvUrNAyvVaGya1ug+nVnwPmzfR2b92in3Jv/ErLeYoSI80cckgOiAMoCiiTJGNoDsA5fH5/fDeHL8fD+e7Nmb7f830/fewHe1h7r/UV/TzW2muvtRQRmJmVg4q2LoCZWWtxwDOzsuGAZ2ZlwwHPzMqGA56ZlY0ObV2AfOrUJbTbnm1dDMvg8MG927oIlsGSJa+zZs0aNeUZlV33i6h5L1XaeG/1zIgY25T8mlNxBbzd9qTTMZe2dTEsgyfuOr+ti2AZHDW6qsnPiJr36PShz6ZK+/7cKb2anGEzKqqAZ2alQKDSfBvmgGdm2QioqGzrUuwUBzwzy05Neg3YZhzwzCwjN2nNrJy4hmdmZUGUbA2vNEttZm1IuRpemq3Qk6SxkhZIWijp4kbSfVpSSKpKjgdKek/S3GS7IU3JXcMzs+yaoZdWUiUwBRgDLAdmSZoeEfPrpdsD+Abw13qPWBQRw7Pk6RqemWWUdFqk2Ro3ClgYEYsjYhNwGzC+gXRXAz8E3m9qyR3wzCwbkaVJ20tSdd6WPzSnH7As73h5cm5bVtIIYN+IuL+BkgyS9KykRyV9LE3R3aQ1s+zSd1qsiYidGs8mqQL4T+DcBi6vBAZExFpJI4H7JB0UEW819kzX8Mwso2Zr0q4A9s077p+c22oP4GDgEUmvAx8BpkuqioiNEbEWICJmA4uAAwtl6BqemWUjoLJZhpbNAoZIGkQu0E0Aztp6MSI2AHWTD0h6BPhORFRL6g2si4haSfsDQ4DFhTJ0wDOz7Jrhw+OIqJE0CZgJVALTImKepMlAdURMb+T2o4HJkjYDW4CJEbGuUJ4OeGaWUfMNLYuIGcCMeucu30HaY/L27wbuzpqfA56ZZeehZWZWNkp0aJkDnpllk3LYWDFywDOz7DwBqJmVB8+HZ2blxE1aMysLJTwfngOemWXkJq2ZlRN3WphZ2fA7PDMrC3KT1szKiWt4ZlYu5IBnZuUgN8O7A56ZlQMJVTjgmVmZKNUaXml2tZhZm5KUakvxnJ1aiDs5d0ly3wJJJ6Ypt2t4ZpZZc9TwmrIQt6Rh5NbAOAjoCzwk6cCIqG0sT9fwzCwbZdga15SFuMcDtyWrl70GLEye1ygHPDPLRKRrzqaoBTZlIe6C9zbETVozy6yiInVdqZek6rzjqRExNc2NBRbi3ikOeGaWWYZ3eGsiomoH17IsxA2wD7mFuMeluLdBbtKaWTbN9w6vbiFuSR3JdULUrUUbERsioldEDIyIgcDTwLiIqE7STZDUKVnIewjwTKEMXcMzs8yao5e2KQtxJ+nuAOYDNcCFhXpowQHPzDLa2mnRHHZ2Ie7k+Brgmiz5OeCZWWYeWmZm5UGlO7TMAc/MMnPAM7Oy4YBnZmWhOTstWpsDnpllV5rxzgHPzDJSpqFlRcUBz8wyc5PWzMpHacY7B7ymOG54f37wzx+lskLc/KeX+fF9zzWY7lOjB3HTd8dw7PfuYe6iNRxzaD+u+PwoOnaoZFNNLZff/Fcef/GNVi59eXroyflc8qO7qN2yhbPHf5R/PfeE7a5v3LSZC664mbkvL6Vnt85M+/fzGNB3T5a+sZbRn/0+BwzYC4CqQwbyX5ec2RY/oSi4htcASWOBn5AbJ/e/EXFtS+bXmioqxPVf/idOm3w/b6x7h4evPY0HqpewYPnft0vXZdddmHjKwcx65c26c2vffp8zr53J39a/y9B9e3DXZSdz0Fd/09o/oezU1m7hu9fdwb0/m0TfvbvziXOu56SjD+HD+/epS3Pzb5+iW9fdmHPvldz9x2qu/OlvmfaD8wAY2K8Xj99ySVsVv2iknb69GLXYm8e86ZtPAoYBZybTMrcLIw/ozeK/bWDJqrfZXLOFe55YxMlHDPxAun+bUMVP7pvLxs3bxjW/8Npa/rb+XQBeWrae3TpW0rFDab4ELiWz573O/vv2YmD/XnTcpQOnjxnBjEef3y7NA489z5mnjAZg/CcO59FZC4iItihuUWuuNS1aW0v+X5Z2+uaS1KdnZ1aseafu+I2179CnZ+ft0hw6aE/69erCH+csq397nXEfGcRzr61hU82WFiur5axcvYF+e/eoO+67dw9Wrt6wXZo3Vm1L06FDJV277Ma6Dbm/56VvrOXoz1/LKef/mCefXdh6BS9CqlCqrdi0ZJO2oSmYR9dPJOl84HwAduvZgsVpXRJcc+6RfO1nj+wwzYf79+DKL4zm9Kvrz15txWbvXl154XeT6dm9C3NfWsrnvzOVp26/lK5ddmvrorWJYqy9pdHm7aiImBoRVRFRpY5d2ro4qa1c9w79em2r0fXdszMr122r8e2x2y4M3bcnv7/qUzz38zOpGrIXt3zvRIYP7pVL37MzN180hgt++mdef/PtVi9/OerTuxsr3lxfd/zGm+vp07vbdmn67rUtTU1NLW/94z16dutMp4670LN77r/P4UMHMKh/LxYtXdV6hS8mcpO2ITs1BXOpmLNwNYP7dGPAXnuwS4cKTj9qMA/MWlJ3/a13N3PAeTdx2Ndu5bCv3Ur1q6s464czmbtoDV1378jt/zaWq37zDH9d8GYjuVhzGjFsPxYtXc2SFWvYtLmGex6cw0lHH7pdmrEfO4Rb78+tBvjbh5/l6CMORBJr1r9NbW3utcPry9eweNlqBvbr1eq/oRiIXAsmzVZsWrJJWzd9M7lANwE4qwXza1W1W4KL/vcJ7r7sJCorKvjNwwt4efl6LvncSOYuWsMD1Ut2eO9XTjqIQft05aIzRnDRGSMAOP3qGax56/0d3mNN16FDJddd9Fk+/fUp1NYGnx/3EYYO7sO/3/B7hg8dwMkfP5Szx3+UiVfcxIjTrqRH18784pp/BuDJZxfygxvup0OHSioqxI8unkCPbp0L5NheFWftLQ21ZA+UpJOBH7Nt+uZGZyet6L5fdDrm0hYrjzW/9Xed39ZFsAyOGl3F7NnVTYpWu+5zYOx3zk9TpX3lurGzG1nEp+Cna5ImAhcCtcA/gPMjYr6kgcBLwIIk6dMRMbFQeVr0O7yGpm82sxLXTM3VvE/XxpDr1JwlaXpEzM9LdktE3JCkH0du2caxybVFETE8S54eaWFmmYjch/fNoO7TNQBJWz9dqwt4EfFWXvrOQJOapA54ZpZZhhpeYwtxp/107ULgW0BH4BN5lwZJehZ4C7gsIh4vVBgHPDPLrJkW4k4lIqYAUySdBVwGnAOsBAZExFpJI4H7JB1Ur0b4AW3+HZ6ZlZiUn6SkiIlZP127DTgVICI2RsTaZH82sAg4sFCGDnhmlokQFRUVqbYC6j5dk9SR3Kdr2y2+LWlI3uEpwKvJ+d5JpweS9geGAIsLZegmrZll1hy9tBFRI2kSMJNtn67NkzQZqI6I6cAkSccDm4H15JqzAEcDkyVtBrYAEyNiXaE8HfDMLLPm+vC4oU/XIuLyvP1v7OC+u4G7s+bngGdm2RTpsLE0HPDMLJPcWNrSjHgOeGaWWYnGOwc8M8uumUZatDoHPDPLRm7SmlmZ2DofXilywDOzjEp3PjwHPDPLrETjnQOemWUkd1qYWZnwd3hmVlYc8MysbJRovHPAM7PsXMMzs/LgyQPMrFzkJgAtzYjngGdmmVWUaBXPAc/MMivReOc1LcwsGyWTB6TZCj9LYyUtkLRQ0sUNXJ8o6QVJcyX9RdKwvGuXJPctkHRimrLvsIYnqWtjNxZaDs3M2q/meIWXLMIzBRhDbk3aWZKmR8T8vGS3RMQNSfpxwH8CY5PANwE4COgLPCTpwIiobSzPxpq088it8p3/07YeBzAgy48zs/ajmTotRgELI2IxgKTbgPFAXcCrV7HqTC72kKS7LSI2Aq9JWpg876nGMtxhwIuIfXd0zczKl8j11KbUS1J13vHUiJia7PcDluVdWw6M/kB+0oXAt4COwCfy7n263r39ChUmVaeFpAnA/hHx75L6A3sni9+aWRnKUMFbExFVTckrIqYAUySdBVzGtqUaMyvYaSHpZ8CxwNnJqXeBG3Y2QzMrcSk7LFJ0WqwA8luS/ZNzO3IbcOpO3guk66X9aER8FXgfIFnstmOK+8ysnZLSbQXMAoZIGiSpI7lOiOnb56MheYenAK8m+9OBCZI6SRoEDAGeKZRhmibtZkkVJC8LJe1JbqVvMytDonk+PI6IGkmTgJlAJTAtIuZJmgxUR8R0YJKk44HNwHqS5myS7g5yHRw1wIWFemghXcCbQm6F796SrgI+C1yV/eeZWXvRXEPLImIGMKPeucvz9r/RyL3XANdkya9gwIuImyTNBo5PTn0mIl7MkomZtR8pm6tFKe3QskpyVcrAozPMyl6pjqVN00t7KXArua+Z+wO3SLqkpQtmZsVLKbdik6aG90Xg8Ih4F0DSNcCzwA9asmBmVrza8wSgK+ul65CcM7MylOulbetS7JzGJg/4L3Lv7NYB8yTNTI5PIPf9jJmVI7XPCUC39sTOA+7PO/90A2nNrIy0uyZtRPyiNQtiZqWhXTZpt5I0mNzHfcOAXbeej4gDW7BcZlbESrWGl+abul8C/49cYD8JuAO4vQXLZGZFrlQ/S0kT8HaPiJkAEbEoIi4jF/jMrAxJUFmhVFuxSfNZysZk8oBFkiaSm4Jlj5YtlpkVs1Jt0qYJeP9Kbmrlr5N7l9cNOK8lC2Vmxa1E412qyQP+muy+zbZJQM2sTAmV7Fjaxj48vpdtC2Z8QESc3iIlMrPi1k5nS/lZq5UiMXxwbx697cutna01QY8jJrV1ESyDjQuWNstz2t07vIj4U2sWxMxKg4DKEg14ntvOzDKrULqtEEljJS2QtFDSxQ1c/5ak+ZKel/QnSfvlXauVNDfZpte/tyFpJwA1M6vTHJ/YSaokt4TEGHLrys6SND0i5uclexaoioh3JV0AXAd8Lrn2XkQMz1TuDIXrlOXBZtY+5aZ4b5ZlGkcBCyNicURsIrcM4/j8BBHx561zcZKbuKR/U8qeZsbjUZJeIFkeTdJhkn7alEzNrLRlaNL2klSdt52f95h+wLK84+XJuR35EvBA3vGuyTOflnTqjm7Kl6ZJ+9/AJ4H7ACLiOUnHpnm4mbVPGfos1kREVdPz0xeAKuDjeaf3i4gVkvYHHpb0QkQsauw5aQJeRUQsqVc9Lbj+o5m1TwI6NE8v7Qpg37zj/sm57fPLrUt7KfDxiNi49XxErEj+XCzpEeBwoNGAl+Yd3jJJo4CQVCnpm8ArKe4zs3Zq61KNhbYCZgFDJA2S1BGYAGzX2yrpcOBGYFxErMo732Nrv4KkXsBR5BblblSaGt4F5Jq1A4A3gYeSc2ZWhqTmGVoWETWSJgEzyS0FOy0i5kmaDFRHxHTgeqALcGfSylwaEeOAocCNkraQq7hdW693t0FpxtKuIhd5zcyA5htaFhEzgBn1zl2et3/8Du57Ejgka35pZjz+HxoYUxsR5zeQ3MzKQBFOdZdKmibtQ3n7uwKnsX1XspmVEUFRTu6ZRpom7XbTuUu6GfhLi5XIzIpbymFjxWhnhpYNAvZu7oKYWelQUa5YUViad3jr2fYOr4LcwtwfGORrZuWh3S7TqFw/8GFs+xhwS0TscFJQMysPpRrwGv3wOAluMyKiNtkc7MysuSYPaHVpRlrMTb52NjNLlmlMtxWbxta06BARNeTGp82StAh4h1wTPiJiRCuV0cyKTLtbxAd4BhgBjGulsphZCWivnRYCKDTdipmVnxKt4DUa8HpL+taOLkbEf7ZAecys6ImKdvgdXiW5WQpK85eZWYsQ7bOGtzIiJrdaScysNAg6lOhLvILv8MzM8rXXGt5xrVYKMysp7e6zlIhY15oFMbPSUaLxLv26tGZmkHyHl3Ir+CxprKQFkhZK+sCkJJK+JWm+pOcl/UnSfnnXzpH0arKdk6bsDnhmlo1yTdo0W6OPkSqBKcBJwDDgTEnD6iV7FqiKiEOBu4Drknt7AlcAo8kt6H2FpB6Fiu6AZ2aZ5EZaND3gkQtUCyNicURsAm4DxucniIg/R8S7yeHT5JZyBDgReDAi1kXEeuBBYGyhDB3wzCwzpdyAXpKq87b8tXD6sf1yEcuTczvyJeCBnbwX2LkZj82szGXotFgTEVVNz09fAKqAjzflOa7hmVlG6ebCSzEf3gpg37zj/mybbHhbbtLxwKXkFuPemOXe+hzwzCyTZuylnQUMkTRIUkdy619P3y6v3FycN5ILdqvyLs0ETpDUI+msOCE51yg3ac0ss+b48DgiaiRNIheoKoFpETFP0mSgOiKmA9eTG9N/Z1JjXBoR4yJinaSryQVNgMlpvh12wDOzbESzTd8eETOAGfXOXZ63f3wj904DpmXJzwHPzDLZ2qQtRQ54ZpZZMS7Qk4YDnpllVprhzgHPzDISUOkanpmVixKNdw54ZpaVUIk2ah3wzCwz1/DMrCzkPkspzYjngGdm2cg1PDMrI+1uTQszs4bkJgBt61LsHAc8M8vMvbRmVjZKtEXrgNcUDz81n0t/fA+1tVv4wrgj+foXx2x3feOmzUya/Guee3kZPbt1Zur3z2VAnz0BmLdwBd/54e384533qZCYOe077Nppl7b4GWXluCOH8oNvn0FlRQU3//ZJfvyrBxtM96ljh3PTdV/m2C9ex9yXlvKZsVX8y9nbJu446IC+fPzsH/LiKwXnnGyXXMOrR9I04JPAqog4uKXyaSu1tVv43o/u5M6fXEjfvbpzwnn/wYkfO5gPDepTl+Y3v3uabnvszjN3Xc69D87m6inT+Z/v/zM1NbV87cqbmXLF2Rw8pB/rNrzDLh0q2/DXlIeKCnH9RZ/ltEk/4403/87Dv/ouDzz2Agte+9t26brs3omJE45h1guv1Z278w/V3PmHagCGDe7Lr//jK2Uc7Er3HV5LzvLyS1KsIlSq5sxfwqD+vRnYrxcdd+nAaceP4A+PvbBdmj88/gKfO3kUkKsxPF79ChHBI8+8zLAD+nLwkNyaIz27daayslQn3CkdIw8ayOJla1iyYi2ba2q558E5nPzxQz+Q7t8mfpKf3PQgGzfVNPicT584knv+OKeli1u8Uq5YVow9uS32f1lEPAYUnIG0VP1t9d/pt1f3uuM+e3Vn5eoN9dJsoN/euTQdOlSyR5ddWbfhHRYtXYUEn/3mzznunOv46a8fatWyl6s+vbux4s31dcdvvLmePr27bZfm0A/1p9/ePfjjE/N2+JzTxozg7j9Wt1g5S0GGVcuKSptXKySdv3UJtzWrV7d1cVpFTe0WnnluMf/3yi/yuxu/yYxHn+exWQvaulhlTxLX/OunuezH9+wwzciD9uO99zfz0qKVrViy4tKM69IiaaykBZIWSrq4getHS5ojqUbSGfWu1Uqam2zT69/bkDYPeBExNSKqIqKqV+/ebV2c1Pbp3Z0Vq/5ed7xy1d8/UFvYp3c3VryZS1NTU8vb/3ifnt0603ev7nxk+AHs2b0Lu+/akeOPHMbzC5a3avnL0crVG+i397bF6fvu3WO7Wvkeu3di6OA+/P6Gb/Dcb6+i6uCB3PKjrzJ86IC6NKefMJK7Z5Z37Q6ap4YnqRKYApwEDAPOlDSsXrKlwLnALQ084r2IGJ5s49KUu80DXqk6fOgAFi9bzZI31rJpcw33PjSHEz92yHZpTvyng7l9xjMA/O7Pc/mnkUOQxLGjh/LSojd49/1N1NTU8uSzC/nQoH3a4meUlTnzlzB4QG8G9N2TXTpUcvqYETzw2PN11996530OGHMxh42/gsPGX0H1i69z1rdvZO5LS4FcDfDU40dw94Oz2+onFI/madOOAhZGxOKI2ATcBozPTxARr0fE88CW5ii2P0vZSR06VHLtt8/gc9/8ObVbtnDWJz/Ch/fvw7VT72f40AGM/dghfP5TR3LhVTcz6ozJ9Oi6OzdefS4A3bvuzsQzj+XE8/4DSRx35DDGHHVQm/6eclBbu4WLrruDu//7QiorxW+mP83Li//GJV89hbkvLeWBep1O9X308ANY8eZ6lqxY20olLl4ZOiR6ScqvEk+NiKnJfj9gWd615cDoDMXYNXl2DXBtRNxX6AZFRIbnpyfpVuAYoBfwJnBFRPyisXtGjKyKR594pkXKYy1jryO/3tZFsAw2LriDLe+ualJ/wtBDDo+bfvtIqrSjBnefHRFVDV1L3smNjYgvJ8dnA6MjYlIDaX8J/D4i7so71y8iVkjaH3gYOC4iFjVWnhar4UXEmS31bDNrY83TBbsC2DfvuH9yLpWIWJH8uVjSI8DhQKMBz+/wzCyT3Ou5dP8UMAsYImmQpI7ABCBVb6ukHpI6Jfu9gKOA+YXuc8Azs2yS+fDSbI2JiBpgEjATeAm4IyLmSZosaRyApCMkLQc+A9woaesHkkOBaknPAX8m9w6vYMBzp4WZZdZcHxVHxAxgRr1zl+ftzyLX1K1/35PAIfXPF+KAZ2YZyQtxm1n5KNF454BnZtkU6zjZNBzwzCy7Eo14DnhmlpknADWzsuF3eGZWHrwurZmVEzdpzawsCNfwzKyMlGi8c8Azs51QohHPAc/MMivGFcnScMAzs8xKM9w54JnZzijRiOeAZ2aZbJ0AtBQ54JlZNv7w2MzKSYnGO0/xbmZZ5SYATbMVfJI0VtICSQslXdzA9aMlzZFUk6xyln/tHEmvJts5aUruGp6ZZdYcTVpJlcAUYAy5NWlnSZpeb22KpcC5wHfq3dsTuAKoAgKYndy7vrE8XcMzs0yUYStgFLAwIhZHxCbgNmB8foKIeD0inge21Lv3RODBiFiXBLkHgbGFMnTAM7Ps0ke8XpKq87bz857SD1iWd7w8OZfGTt3rJq2ZZZbhs5Q1EVHVkmXJwjU8M8usOdalBVYA++Yd90/OpbFT9zrgmVk2goqUWwGzgCGSBknqCEwApqcsxUzgBEk9JPUATkjONcoBz8x2QtO7LSKiBphELlC9BNwREfMkTZY0DkDSEZKWA58BbpQ0L7l3HXA1uaA5C5icnGuU3+GZWSbNOQFoRMwAZtQ7d3ne/ixyzdWG7p0GTMuSnwOemWVWqiMtHPDMLDOPpTWzspFm2FgxcsAzs8xKM9w54JlZRim/sStKDnhmlpknADWz8lGa8c4Bz8yyK9F454BnZlnJyzSaWXlozpEWrc1jac2sbLiGZ2aZlWoNzwHPzDLzZylmVh784bGZlYtS7rRwwDOzzNykNbOyUao1PH+WYmaZNdO6tEgaK2mBpIWSLm7geidJtyfX/yppYHJ+oKT3JM1NthvSlNs1PDPLrhlqeJIqgSnAGHLrys6SND0i5ucl+xKwPiIOkDQB+CHwueTaoogYniVP1/DMLBMBFVKqrYBRwMKIWBwRm4DbgPH10owHfpXs3wUcpybMPlpUNbxn58xe03W3yiVtXY4W0AtY09aFsEza69/Zfk19wJw5s2futot6pUy+q6TqvOOpETE12e8HLMu7thwYXe/+ujQRUSNpA7Bncm2QpGeBt4DLIuLxQoUpqoAXEb3bugwtQVJ1Ma2+boX572zHImJsW5cBWAkMiIi1kkYC90k6KCLeauwmN2nNrK2sAPbNO+6fnGswjaQOQDdgbURsjIi1ABExG1gEHFgoQwc8M2srs4AhkgZJ6ghMAKbXSzMdOCfZPwN4OCJCUu+k0wNJ+wNDgMWFMiyqJm07NrVwEisy/jtrYck7uUnATKASmBYR8yRNBqojYjrwC+BmSQuBdeSCIsDRwGRJm4EtwMSIWFcoT0VES/wWM7Oi4yatmZUNBzwzKxsOeC2o0LAZKz6SpklaJenFti6LNT8HvBaSN2zmJGAYcKakYW1bKkvhl0AxfGdmLcABr+WkGTZjRSYiHiPXG2jtkANey2lo2Ey/NiqLmeGAZ2ZlxAGv5aQZNmNmrcgBr+WkGTZjZq3IAa+FREQNsHXYzEvAHRExr21LZYVIuhV4CviQpOWSvtTWZbLm46FlZlY2XMMzs7LhgGdmZcMBz8zKhgOemZUNBzwzKxsOeCVEUm2y6PCLku6UtHsTnnWMpN8n++Mam81FUndJX9uJPK6U9J205+ul+aWkMzLkNdAznFghDnil5b2IGB4RBwObgIn5F5WT+e80IqZHxLWNJOkOZA54ZsXGAa90PQ4ckNRsFki6CXgR2FfSCZKekjQnqQl2gbr5+V6WNAc4feuDJJ0r6WfJ/t6S7pX0XLJ9FLgWGJzULq9P0n1X0ixJz0u6Ku9Zl0p6RdJfgA8V+hGSvpI85zlJd9ertR4vqTp53ieT9JWSrs/L+6tN/Rdp5cMBrwQly9WdBLyQnBoC/DwiDgLeAS4Djo+IEUA18C1JuwL/A3wKGAnss4PH/zfwaEQcBowA5gEXA4uS2uV3JZ2Q5DkKGA6MlHR0sj7ohOTcycARKX7OPRFxRJLfS0D+yIaBSR6nADckv+FLwIaIOCJ5/lckDUqRj5lXLSsxu0mam+w/Tm5Fp77Akoh4Ojn/EXITjj4hCaAjuaFSHwZei4hXAST9Gji/gTw+AXwRICJqgQ2SetRLc0KyPZscdyEXAPcA7o2Id5M80owdPljS98k1m7uQG4q31R0RsQV4VdLi5DecABya936vW5L3KynysjLngFda3ouI4fknkqD2Tv4p4MGIOLNeuu3uayIBP4iIG+vl8c2deNYvgVMj4jlJ5wLH5F2rP+4xkrz/JSLyAyOSBu5E3lZm3KRtf54GjpJ0AICkzpIOBF4GBkoanKQ7cwf3/wm4ILm3UlI34G1ytbetZgLn5b0b7CdpL+Ax4FRJu0nag1zzuZA9gJWSdgE+X+/aZyRVJGXeH1iQ5H1Bkh5JB0rqnCIfM9fw2puIWJ3UlG6V1Ck5fVlEvCLpfOB+Se+SaxLv0cAjvgFMTWYJqQUuiIinJD2RfPbxQPIebyjwVFLD/AfwhYiYI+l24DlgFbkpsgr5P5/7iigAAABUSURBVMBfgdXJn/llWgo8A3Qlt9Dy+5L+l9y7vTnKZb4aODXdvx0rd54txczKhpu0ZlY2HPDMrGw44JlZ2XDAM7Oy4YBnZmXDAc/MyoYDnpmVjf8PWYl58GIAKrIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import plot_confusion_matrix\n",
    "plot_cm = plot_confusion_matrix(knn, X_test, y_test, cmap=plt.cm.Blues,\n",
    "                                 normalize='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, finally, we can extract the number of true positives, true negative, false positives and false negatives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "47"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()\n",
    "# Show the true positives: \n",
    "tp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Sensitivity, Specificity, Precision, Recall\n",
    "**Precision and Recall**. From the confusion matrix there are other quantities of interest that we can calculate. The **precision** of a (binary) classifier measures how many units, among the ones classified as positives, really belong to the positive class: \n",
    "\n",
    "$$Precision = \\frac{TP}{TP + FP}.$$\n",
    "\n",
    "If the classifier classifies only one single positive unit as positive, we would have 100% precision, but the classifier wouldn't be a very useful classifier, as we would not consider all other positive instances. Thus, Precision is usually evaluated along with the **recall**, which measures how many units, among the ones that are truly positive, have been classified correctly: \n",
    "\n",
    "$$Recall = \\frac{TP}{TP + FN}$$\n",
    "\n",
    "While it is desirable to have both high precision and recall (close to 1, the maximum), this can be difficult to achieve: increasing the recall usually corresponds to decreasing precision, and vice versa. This is the *precision/recall tradeoff*. \n",
    "\n",
    "The following image [taken from Wikipedia](https://en.wikipedia.org/wiki/Precision_and_recall) can help to get a clearer picture:\n",
    "\n",
    "<img src=\"./img/model_performance/precision_recall_wikipedia.png\" width=\"300\"/>\n",
    "\n",
    "We can calculate the precision and recall of our knn classifier in scikit-learn as follows (once again, pay attention to what class is considered positive by `scikit-learn`!): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9038461538461539, 0.8867924528301887)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score\n",
    "precision, recall = (precision_score(y_test, y_pred), recall_score(y_test, y_pred))   \n",
    "precision, recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Sensitivity and Specificiy**. Other two quantities that are usually considered together are the **sensitivity** of a classifier, and its **specificity**. The sensitivity is equivalent to the Recall, i.e., proportion of positives units among the ones classified as positives. Sensitivity and Recall are also known as True Positive Rate (TPR). The specificity, instead, is also known as True Negative Rate, and is calculated as the proportion of instances correctly classified as negatives among all the truly negative instances: \n",
    "\n",
    "$$Specificity = TNR = \\frac{TN}{TN+FP}$$\n",
    "\n",
    "This can be viewed as the ability of the classifier to correctly identify negative instances. As for precision and recall, there exists a tradeoff between sensitivity and specificity. As before, [Wikipedia](https://en.wikipedia.org/wiki/Sensitivity_and_specificity) can help to obtain a better graphical overview of TPR and TNR:\n",
    "\n",
    "<img src=\"./img/model_performance/sensitivity_specificity_wikipedia.png\" width=\"300\"/>\n",
    "\n",
    "Classifiers can be evaluated and compared by plotting two curves that are widely used: the *Precision/Recall curve*, and the *ROC curve*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "============================================================================================\n",
    "####  Digression: how classifiers classify\n",
    "Before understanding Precision/Recall and ROC curves, it is important to understand how a classifier performs predictions. In fact, most (but not all!) classifiers do not predict a class directly; instead, for each instance $i$ they first produce (directly or indirectly) a real number by means of a *decision function* $f(\\mathbf{x}_i)$, which is in turn transfomed into a probability (included between $0$ and $1$). The latter tells us how much we are confident about the probability for each unit to belong to the positive class, and can be interpreted as the probability for a unit to be predicted in the positive class by the model. For instance, a probability close to $0$ indicates that the classifier is very confident that the unit belongs to the negative class; and a probability close to $1$ that it belongs to the positive class. We will see how these numbers are yielded by each method in the coming lectures. Keep in mind that some classifiers only calculate the decision function in order to make a prediction, while others calculate the class probability directly (without decision function). (In `scikit-learn`, the decision function values can be retrieved with the `decision_function` method of a classifier; the transformed probabilities can be retrieved with `predict_proba`. Not all classifiers support the `decision_function` method (for example, `KNearestNeighbors` doesn't); and not all classifiers have a `predict_proba` method (for instance, `SVC` doesn't). Some classifiers, like logisic regression, have both methods).\n",
    "\n",
    "To perform predictions, classifiers assigns the unit to a class based on a certain *threshold*; in general, it is sensible to use a threshold of 0.5 (when working with probabilities), which means that units are assigned to the positive class only when we are more than 50% certain of such decision. Although reasonable, this choice is purely arbitrary: in reality, we can potentially modify such threshold as we prefer; this, of course, depends on the application, data, and context with which we are performing the analysis. \n",
    "\n",
    "Let's take an example; suppose we have 10 test units (4 from the P class and 6 from the N class), and an already trained classifier has estimated the following probabilities (units are sorted from the least to the most likely): \n",
    "\n",
    "|**Observed Class**|N|N|N|N|P|N|P|P|N|P|\n",
    "|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n",
    "|**Predicted Probability**|0.1|0.2|0.25|0.27|0.4|0.6|0.8|0.9|0.95|0.99|\n",
    " \n",
    "With a threshold at 0.5, we would obtain the following predictions (highlighted are the miss-classifications):\n",
    "\n",
    "|**Observed Class**|N|N|N|N|P|N|P|P|N|P|\n",
    "|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n",
    "|**Predicted Probability**|0.1|0.2|0.25|0.27|0.4|0.6|0.8|0.9|0.95|0.99|\n",
    "|**Predicted Class (thr. 0.5)**|N|N|N|N|**N**|**P**|P|P|**P**|P|\n",
    " \n",
    " Which yields an accuracy of 70%, recall (sensitivity) of 3/4 = 0.75, precision of 3/5 = 0.6, and specificity of 4/6 = 0.66. \n",
    " \n",
    " \n",
    " Now, it appears clear that moving the threshold will modify such values; let's see how they change when the threshold to classify an instance to the positive class is increased to 0.85: \n",
    " \n",
    "|**Observed Class**|N|N|N|N|P|N|P|P|N|P|\n",
    "|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n",
    "|**Predicted Probability**|0.1|0.2|0.25|0.27|0.4|0.6|0.8|0.9|0.95|0.99|\n",
    "|**Predicted Class (thr. 0.85)**|N|N|N|N|**N**|N|**N**|P|**P**|P| \n",
    " \n",
    " The accuracy is (by coincidence) still the same, 70%. The recall is now 2/4=0.5, the precision is 2/3=0.66, and the specificity is 5/6=0.83. \n",
    " \n",
    "Lastly, let's see what happens when we decrease the threshold to 0.15: \n",
    " \n",
    "|**Observed Class**|N|N|N|N|P|N|P|P|N|P|\n",
    "|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n",
    "|**Predicted Probability**|0.1|0.2|0.25|0.27|0.4|0.6|0.8|0.9|0.95|0.99|\n",
    "|**Predicted Class (thr. 0.15)**|N|**P**|**P**|**P**|P|**P**|P|P|**P**|P|  \n",
    "\n",
    "The accuracy has dropped to 50%; the recall now is at its maximum value, 4/4=1 (as all the positive instances are correctly classified), but the precision has dropped to 4/9=0.44, and the specificity is even lower, to 1/6=0.16. \n",
    "\n",
    "\n",
    "With this example, we have seen the Precision/Recall tradeoff in action: by modifying the decision threshold, we can increase the recall while decreasing the precision, and vice versa: it is not possible to increase both simultaneously. The same effect is observed for sensitivity (=recall) vs. specificity.\n",
    "<br>\n",
    "\n",
    "============================================================================================"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 ROC and AUC\n",
    "\n",
    "Because the threshold used to classify units is arbitrary and can vary from setting to setting, the *receiver operating characteristic* (ROC) curve allows visualizing how the performance of the classifier vary under  different thresholds. \n",
    "\n",
    "In particular, the ROC curve plots the TPR (Sensitivity) in the y-axis, and the *false positive rate* (FPR) in the x-axis. FPR is defined as \n",
    "\n",
    "$$FPR = 1-Specificity,$$ \n",
    "\n",
    "and denotes the ratio of negative examples that are incorrectly classified as positive. We are now going to plot the ROC curve for the `knn` classifier. \n",
    "\n",
    "**Note**: the `roc_curve` function below takes as second argument the `predict_proba()` method of `knn`, which gives the probabilities estimated by the model to perform the predictions (in particular, column 1 -the second column- selects the probabilities to belong to the positive class). Some other classifiers (but not `knn`) also have a a `decision_function()` method that can be equally used for the input of `roc_curve`. We are going to explore how such functions (and class probabilities) are calculated in future lectures.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUwAAAFNCAYAAABrMlb6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXgUVfbw8e9JQoIQQFYVEBAFFRREkddlJEDYZFBE0QEBRZnBXUcZR8Edf47bjM64jTKiKIILDiKrKEtwRWAEXEB2VESQnQTIft4/qgJNzFJJulLdnfN5njx0V1dXn8pyuHXvrXNFVTHGGFO6uKADMMaYaGEJ0xhjPLKEaYwxHlnCNMYYjyxhGmOMR5YwjTHGI0uYxhjjkSVM4ysR2SQiB0UkQ0S2ish4EUkutM95IjJfRNJFZK+ITBeRNoX2qS0i/xSRH91jrXefNyjmc0VEbhWRb0Vkv4hsFpHJInK6n+drYpslTFMZLlLVZOAMoAMwquAFETkX+BB4H2gMnACsAD4TkZbuPonAPKAt0BuoDZwL7AQ6FfOZ/wJuA24F6gGtganA78savIgklPU9Jkapqn3Zl29fwCage8jzJ4CZIc8/AV4o4n2zgdfdx38EtgHJHj+zFZAHdCphnzTgjyHPhwGfhjxX4CZgLbAR+Dfw90LHeB+4w33cGPgvsN3d/9agv/f2Ff4va2GaSiMiTYELgXXu8xrAecDkInZ/B+jhPu4OfKCqGR4/KhXYrKqLKxYxlwD/D2gDvAn8QUQEQETqAj2Bt0QkDpiO0zJu4n7+n0WkVwU/30QYS5imMkwVkXTgJ+BX4AF3ez2c38FfinjPL0BB/2T9YvYpTln3L86jqrpLVQ/itIQVuMB9bQDwhapuAc4GGqrqGFXNVtUNwH+AgWGIwUQQS5imMlyiqrWALsApHE6Eu4F84Lgi3nMcsMN9vLOYfYpT1v2L81PBA3Wuu98CBrmbrgQmuo+bA41FZE/BFzAaOCYMMZgIYgnTVBpVXQiMB/7uPt8PfAFcXsTuV+AM9ADMBXqJSE2PHzUPaCoiHUvYZz9QI+T5sUWFXOj5m8AAEWmOc6n+X3f7T8BGVT065KuWqvbxGK+JEpYwTWX7J9BDRNq7z+8GrnanANUSkboi8n84o+APuftMwElK/xWRU0QkTkTqi8hoEflNUlLVtcALwJsi0kVEEkWkuogMFJG73d2WA5eKSA0ROQkYXlrgqroMp9X7MjBHVfe4Ly0G0kXkLhE5SkTiReQ0ETm7PN8gE7ksYZpKparbgdeB+93nnwK9gEtx+h1/wJl69Ds38aGqWTgDP98DHwH7cJJUA+DLYj7qVuA54HlgD7Ae6I8zOAPwNJCNM/r+Gocvr0szyY1lUsg55QF9caZNbeRwUq3j8ZgmSojTNWOMMaY01sI0xhiPLGEaY4xHljCNMcYjS5jGGOORJUxjjPEo6qqwNGjQQFu0aBF0GMaYGPO///1vh6o2LGkf3xKmiLyCMzftV1U9rYjXBacEVx/gADBMVb8q7bgtWrRg6dKl4Q7XGFPFicgPpe3j5yX5eJzahcW5EKcMVytgBE75LGOMiVi+JUxV/RjYVcIu/XDqHaqqLgKOFpFwFEwwxhhfBNmH2YSQajDAZndbOMpyGWNiWFZuHgey8sjIymV/di77s/LYn5XLgexcMtzHzvbDr+3PzmXfwWz27NzJOW1bcm/fNqV/UCFRMegjIiNwLttp1qxZwNEYY8pCVcnMySfjUELL5UC2m+yycg8nvqxc9me7yS00EYYmPvdxTl75bunO3raBHdOfpOYDr+LUhS6bIBPmz8DxIc+butt+Q1XHAmMBOnbsaDe/G+Oj/HzlQE5I4nITWkGy25+VF/I4NMkV37LLD/NfbUKcUDMpgeSkBGomxVMj8fDjmokJ1ExKoEZSPMkFjxPjSK5ejZpJZ7Nz6Dmcefqp5fvc8J5GmUwDbhaRt3BqC+5VVbscN6aMcvPyC7XMnMcZhS5RD2Q5j4tOdocfH8jOC3uMSQlxJLtJrCChOQnvyGR3+HECNRPjD+1X84j3xZOUEO/5s7Ozs7nyyiu55JJL6DtkCJzcqNzn4ee0ojdxKmw3EJHNOMsSVANQ1ReBWThTitbhTCu6xq9YjIkkWbl5xbbGQi9XD2QXbt0VcbmalUtWbn7YY6xRkKxCk5b7ODkpwU1s8W5LrnDiO/J9NRLjqRYfzD0ymZmZDBgwgJkzZ5KSklLh4/mWMFV1UCmvF6zKZ0zECu1/K5yofpPssp3+uMKJb3+hJFje/rfixAlHtL6chFX4+ZHJrmZSfEjiC9kvKYEa1eKJi5OwxhiEAwcO0L9/fz788ENefPFFrrvuugofMyoGfYzxKj9f2Z99ZLI6MsEdmewKLlF/07rzsf+tWryEJLWi+99+k+SSQhPekcmuerU43MUsjSsnJ4e+ffuSlpbGK6+8wjXXhOcC1hKmCVRuXr5z2Zld0MdWKNllHzn4cLgld+SoaUHiq4z+t+SQy9DQllxp/W8FxyhL/5spn2rVqpGSksK1117LkCFDwnZcS5jGM1UlKzefA9lFtNKySh5IKEiEoS29jKxcsn3sfytojXnpfytqIKLgfQkB9b+ZstuzZw+bN2/mtNNO44EHHij9DWVkCTOGqSoHcw5fYoYmrd8MJBQxFWR/1m8TX26Yr0+99L+FXobWLOKSNBb730zZ7dq1i169evHzzz+zbt06atSoUfqbysgSZpTLzctn0879rN6awept6azeuo812zLYnp7F/uxcwr1kU+H+t6Ifl97/VrCv9b+ZcNixYwc9evRg5cqV/Pe///UlWYIlzKihqmzefZA129JZvS2dNVvT+X5rOhu27yc7r/jL2oL+tyNaZkX0v4UmuyMuSa3/zUS4bdu2kZqayvr165k+fTo9e/b07bMsYUagHRlZrN6azuqt6UckyP3FDGgcX+8oTj6mFq2PqcXJxzr/Nj76KOt/M1XCww8/zMaNG5k5cybdunXz9bOibpndjh07aqzUw0zPzGHNtgwnKYYkyJ37s4vcv0FyEqccW5AYk2ntJsmaSfb/nqm6MjMzWblyJWeeeWaFjiMi/1PVjiXtY39plSAzJ4/12wsSY8ahfsaf9xwscv9aSQm0PtZpLRa0HFsfk0z95KRKjtyYyLRp0yZGjhzJyy+/TN26dSucLL2yhOmz17/YxMMzVhZ5d0diQhytGiVzcsGltJsgj6tT3QZCjCnGunXr6NatGxkZGfz444/UrVu30j7bEqaPPl+/gwenfUe+wokNax7qXyxIkM3r1yTepsAY49nq1avp1q0bWVlZzJ8/n/bt21fq51vC9MnWvZnc+uYy8hVu7noSf+l1ctAhGRPVVq5cSbdu3VBV0tLSOO203ywV5jsbQvVBTl4+N0/6ih0Z2Zx/Un1u79E66JCMiXrJycmceOKJLFy4MJBkCdbC9MVjs79n6Q+7ObZ2dZ4Z2MEuu42pgHXr1nHCCSfQrFkzPv3000D7962FGWazvvmFcZ9uJCFOeH7wmTaybUwFLFq0iLPOOov7778fIPDBUEuYYbR+ewZ3Tl4BwL2/P5Wzmlfe6J0xsebTTz+lR48eNGzYMCy1LMPBEmaYHMjO5YY3/sf+7Dwuat+Yq89rEXRIxkSttLQ0evXqRZMmTVi4cGHELH5oCTMMVJVRU75hzbYMTmqUzGOXnh74pYMx0So9PZ0BAwbQokUL0tLSaNKkSdAhHWKDPmHwxqIfeH/5FmokxvPikDPtVkVjKqBWrVpMmTKFU089lYYNGwYdzhHsL7uClv+0hzEzVgLw2GXtOKlRrYAjMiY6vf/+++zYsYPhw4fTuXPnoMMpkl2SV8Cu/dnc+Mb/yMlThp3XgovbNw46JGOi0uTJkxkwYADjxo0jNzc36HCKZQmznPLyldveWsaWvZl0aHY0o/uUb2F4Y6q6SZMmMXDgQM455xw++OADEhIi98LXEmY5PTNvLZ+s3UG9mom8MPhMEhPsW2lMWY0fP54hQ4aQkpLC7NmzqV27dtAhlcj+ysshbfWvPDN/LSLwzMAOHFfnqKBDMiYqbdu2je7duzNjxgySk5ODDqdUVkC4DJb/tIfxn21k5je/kJOnjOzRmltSWwUSizHRbPv27YdGwHNzcyPiMtxLAWFrYZYiJy+faSu20P+Fz7jk+c+YunwLefnKoE7NuKnrSUGHZ0zUeeqpp2jdujXff/89QEQkS6+iJ9JKtjMjizcX/8iERT+wbV8WALWrJzCoUzOGntucpnX9WZXOmFj26KOPMnr0aC6//HJOPPHEoMMpM0uYhazcso9XP9vI+yu2kJ3rrMbYqlEyw85vQf8OTaiRaN8yY8pKVRkzZgwPPvgggwcPZvz48VHVsiwQfRH75MPvtjLu0418uXEXACKQekojrjn/BM4/qb7d6mhMBbz55ps8+OCDDBs2jJdffpn4+OhcrtkSJk5JthsnfgVAclICl3dsytXntqBFg5oBR2ZMbBgwYAB79+7luuuuIy4ueodOLGHitC4BhpzTjLt6n0Kt6tUCjsiY6Jefn88TTzzB8OHDadiwITfccEPQIVVY9Kb6MFr6w24AruzU3JKlMWGQn5/P9ddfz6hRo5g4cWLQ4YRNlW9hbt2byebdB6mVlMDJx1rhDGMqKi8vj+HDh/Paa68xevRobrvttqBDCpsqnzCX/uAM8pzR7Ghbe8eYCsrNzeXqq69m0qRJPPTQQ9x3330xNWBqCXOTcznesXm9gCMxJvrt3r2bJUuW8Oijj3L33XcHHU7YVfmE+T+3//LsFrb+jjHllZ2dTVxcHA0bNmTZsmXUrBmbM0yq9KDP/qxcVv6yj/g44YxmRwcdjjFRKTMzk/79+3PNNdegqjGbLKGKJ8zlP+0hL19pc1xtu4PHmHI4cOAAF198MbNnz+aCCy6Iqf7KolTpLFHQf2nL4RpTdhkZGVx00UUsXLiQV155hWHDhgUdku+qdsJ0R8jPbmEDPsaU1RVXXMHHH3/MhAkTGDx4cNDhVIoqe0mel68s+3EPAB1twMeYMrvrrrt46623qkyyhCrcwvx+6z4ysnJpWvcojqldPehwjIkKu3btYs6cOQwaNIiUlJSgw6l0VbaFeXg6kV2OG+PF9u3b6dq1K9deey2bN28OOpxAVNkWpg34GOPd1q1bSU1NZePGjUybNo2mTZsGHVIgfG1hikhvEVktIutE5DfT/kWkmYgsEJFlIvK1iPTxM55QSzc5Az7Wf2lMyX7++We6dOnCDz/8wKxZs+jRo0fQIQXGt4QpIvHA88CFQBtgkIi0KbTbvcA7qtoBGAi84Fc8obbsOciWvZnUqp5A60ZWcMOYknz44Yds2bKFDz74gC5dugQdTqD8bGF2Atap6gZVzQbeAvoV2keBgoWI6wBbfIznkIJybmc1r0ucFdwwpkh5eXkAXHPNNaxZs4bf/e53AUcUPD8TZhPgp5Dnm91toR4EhojIZmAWcEtRBxKRESKyVESWbt++vcKB/a/gctz6L40p0tq1azn99NP5/PPPATj22GMDjigyBD1KPggYr6pNgT7ABBH5TUyqOlZVO6pqx4K1jCtiyaEBHxshN6awVatWkZKSwvbt22P6vvDy8DNh/gwcH/K8qbst1HDgHQBV/QKoDjTwMSYysnL5fus+EuKEM463ghvGhPr222/p0qUL+fn5pKWl0b59+6BDiih+JswlQCsROUFEEnEGdaYV2udHIBVARE7FSZgVv+YuwbIfd5Ov0LZJHY5KjM6V64zxw/r16+nSpQsJCQksXLiQtm3bBh1SxCkxYYrI2SLyLxH5SkR+EZENIjJNRK4TkRKHl1U1F7gZmAOswhkN/05ExojIxe5uI4E/icgK4E1gmKpqxU+reIcLBlv/pTGhmjdvzqBBg1i4cCEnn3xy0OFEpGInrovITGAH8D7wD+BXnBZga6ArMFNEnlDVGcUdQ1Vn4QzmhG67P+TxSuD8ipxAWRUU3LCEaYxjyZIlNG3alOOOO45nn3026HAiWkl3+lyrqtsKbcsEFrtfj4tII98i80FuXv6hghtn2YR1Y/jkk0/o06cP3bp14/333w86nIhX7CV5QbIUkRtEpE4x+/zqV2B++Gn3QQ5k59G4TnUa1bKCG6Zqmz9/Pr1796Zp06b8+9//DjqcqOBl0Kc58JWITBKR7n4H5Kd9B3MAqJ+cFHAkxgRrzpw5/P73v6dly5akpaXRuHHjoEOKCqUmTFW9G2gFTASuF5G17sBNC59jC7v0zFwAalWvsjVHjCE/P5/Ro0dzyimnsGDBAo455pigQ4oanjKHquaLyCZgE3A6cBzwvojMUtVR/oUXXhlZTgszOckSpqm64uLimDlzJomJidSrZzdvlEWpLUwRuUlEFgP/Av4HtFPVPwEdgD/4HF9Y7XNbmMnWwjRV0DvvvMPAgQPJzc3l2GOPtWRZDl76MBsDg1S1u6q+qapZ4LQ6gYtLfmtkyXATZu3q1QKOxJjK9cYbbzBo0CC2bNlCZmZm0OFELS8Js4mqrg/dICLjAVT1Wz+C8ktBH6Zdkpuq5JVXXuGqq66iS5cuzJ49m+Tk5KBDilpeEma70CducYyz/QnHXwV9mDboY6qKcePGMXz4cHr27MmMGTOsmEYFFZswReQuEdkNtBORXe7Xbpy7f2YV975Ilm59mKaKadu2LYMGDWLq1KkcddRRQYcT9UpqYT4BNASedv9tCDRQ1XqqemdlBBdu6VkF04qsD9PEtiVLlgBwzjnnMGnSJKpXtxs1wqGkhHmSW0BjAtC24EtE2olIuxLeF7EOzcO0PkwTwx555BE6derEzJkzgw4l5pSUOe7GqVf5fBGvKdDZl4h8lJFpfZgmdqkqDz74IGPGjGHo0KH06tUr6JBiTrGZQ1WHu/9eUHnh+Csjy/owTWxSVUaNGsXjjz/Otddey9ixY4mPt3qv4eZl4vpXInKniDSvjID8dPjWSOvDNLFl0aJFPP7441x//fX85z//sWTpEy9Nrctx7uiZJiIHgLdxigFXygqP4ZRh8zBNjDr33HNZuHAhF1xwASK2EqpfvBTfWK+qf1PV9sC1wJk4S0tElfx8JSPbEqaJHfn5+dx6662kpaUB0LlzZ0uWPvOUOUSkKXAFTkszAbjHz6D8sD87F1WomRhPvK1FbqJcXl4ew4cP57XXXqNBgwZ06dIl6JCqhFITpoh8DiQDk4EhqrrW96h8YJPWTazIzc3l6quvZtKkSYwZM4b77rsv6JCqDC/Z40+q+p3vkfgswyatmxiQk5PDlVdeybvvvstjjz3GXXfdFXRIVUpJi6ANUtU3gVQRSS38uqo+42tkYZaeabUwTfSLi4ujevXqPPXUU9x+++1Bh1PllJQ9ClYJa1jEa74uhesHq7ZuollmZia7du2icePGvP766za4E5CSJq6/4D6cqaqLQl8TkXN8jcoHljBNtDpw4AD9+vXjxx9/5OuvvyYpydakCoqX8m4vFLGtqNslI9qhPswk68M00SMjI4M+ffowf/58Ro8ebckyYCX1YXYCzgUaisitIS/VBqIu62TYKLmJMvv27aNPnz4sWrToUMV0E6ySskdNoIG7T2g/ZjrO3T9RJd0Kb5goc/vtt/Pll1/y9ttvc9lllwUdjqHkPswFwAIReVVVN1RiTL4oqIVpo+QmWjz++OP84Q9/oGfPnkGHYlwlVVz/h/vwHyIypfBXJcUXNum2AJqJAr/++it33HEH2dnZNGjQwJJlhCmpufW2++9zlRGI36wP00S6X375hdTUVDZt2sTgwYM566yzgg7JFFLSJfli9995BdtEpA7OKpIrKyG2sErPsonrJnJt3ryZbt26sWXLFmbPnm3JMkJ5qYc5T0Rqi0hdYDkwQUSe9D+08MqweZgmQv3www+kpKSwdetW5syZQ0pKStAhmWJ4mYdZT1X3AZcCb6jqWUDU1b63iesmUu3atYv8/Hzmzp3L+eefH3Q4pgReskeCiDTEmUp0v8/x+MZWjDSRZteuXdSrV48OHTqwZs0aqlWz381I56WF+QiwEPhRVReLSEtgo79hhZ8V3zCRZNWqVbRt25Z//etfAJYso0Sp2UNV3wLeCnm+AejnZ1DhlpOXT2ZOPnECNRJtrRMTrG+++YbU1FTi4+Pp0aNH0OGYMvBSQLgBztIULUL3V9UR/oUVXvtDJq1blRcTpGXLltGjRw+qV6/O/Pnzad26ddAhmTLwcn36PrAI+BTI8zccf9hqkSYS7Nmzh+7du5OcnMz8+fM58cQTgw7JlJGXhFlTVUf6HomPbITcRIKjjz6a5557jvPOO4/mzaN+1eoqycugz2wRier7s6zwhgnSxx9/zIcffgjAoEGDLFlGMS8Z5HrgLndN8mxAAFXVer5GFkYZVnjDBGTevHlcdNFFnHrqqXTv3p24OC9tFBOpvGSQBr5H4bPDK0ZaH6apPB988AH9+/enVatWzJ4925JlDCj1J6iqeTiT1u9yHx8HnOF3YOF0eNK6tTBN5Zg+fTr9+vXj1FNPZcGCBTRq1CjokEwYeLmX/DmgKzDU3XQAeNHPoMLtUB+mXZKbSjJ79mzat2/PvHnzqF+/ftDhmDDxco1wnqpeB2QCqOouINHLwUWkt4isFpF1InJ3MftcISIrReQ7EZnkOfIysMIbprJkZmYC8NxzzzF//nzq1q1byjtMNPGSMHNEJA53aV0RqQ/kl/YmEYnHWSztQqANMEhE2hTapxUwCjhfVdsCfy5b+N4c6sO0Fqbx0euvv07btm3ZvHkzcXFxJCcnBx2SCTMvCfN54L84i6E9hDOB/XEP7+sErFPVDaqajXN7ZeFbKv8EPK+quwFU9VfPkZdBhhXeMD4bN24cw4YNo0WLFtaqjGFe7iV/XUT+B3R3N12uqt96OHYT4KeQ55uB/1don9YAIvIZEA88qKofeDh2maRbtXXjoxdeeIGbbrqJ3r17M2XKFI466qigQzI+KWlNn+ruZTWq+h0wE+dSvGUYPz8BaAV0AQYB/xGRo4uIZYSILBWRpdu3by/zh9jEdeOXN998k5tuuomLLrqIqVOnWrKMcSVdks8BTgQQkROBxTh9kXeIyCMejv0zcHzI86butlCbgWmqmqOqG4E1OAn0CKo6VlU7qmrHhg0bFn65VIcuyZPsktyE14UXXsioUaN49913SUpKCjoc47OSEmY9VV3jPr4aeEtVb8Cptn6xh2MvAVqJyAkikggMBKYV2mcqTuuyoCpSayDsS/raveQm3CZOnMjBgwc5+uij+dvf/kZioqeJIybKlZQwNeRxN+AjAFXNwsMouarmAjfjtFRXAe+o6nciMkZEChLuHGCniKwEFgB3qurOsp9GyQ7dGmkJ01SQqnLfffcxZMgQXnrppaDDMZWspAzynYg8hnMZ3Rr4EA6tHOmpqKSqzgJmFdp2f8hjBe5wv3yhqlZt3YSFqnL33XfzxBNPMHz4cG655ZagQzKVrKQW5h+BDOAUoLeq7ne3nwY85Xdg4ZKVm09OnpIYH0f1alZt3ZSPqnL77bfzxBNPcMMNNzB27Fji4+33qaopaV3y/cD/FbH9M+AzP4MKJ5tSZMLh559/5o033uC2227j6aeftsr9VVSxWUREpgIvAR+5/ZGhrzXHGQjarKqv+BtixWRY4Q1TAfn5+YgITZs2Zfny5TRp0sSSZRVW0iX5TUAPYI2IfCEi00TkQxFZB7wKfBfpyRJstUhTfrm5uQwbNowHHngAgKZNm1qyrOKKTZiq+rOq3qGqLXEqFT0JjAY6qGo3Vf1vZQVZEVZ4w5RHTk4OQ4cOZcKECTZlyBziKYuo6jpgnc+x+GLfocIbNmndeJOdnc2gQYOYMmUKTzzxBHfeeWfQIZkIEfPNroI+zNrWwjQeqCoDBw7kvffe45///Ce33XZb0CGZCBLzWSSjoA/TEqbxQES49NJL6dGjBzfccEPQ4ZgI4ymLuLc2NnMvzaOK3RZpvNi/fz9fffUVF1xwAUOGDAk6HBOhvCxR8XvgG9xbI0XkDBF5z+/AwuXwipHWh2mKlp6ezoUXXkivXr3YunVr0OGYCOalgPAYnDqWewBUdTlwkp9BhdM+a2GaEuzdu5devXrx+eef8+qrr3LssccGHZKJYF6ySI6q7ik0/0yL2znS2MR1U5zdu3fTq1cvli9fzjvvvMOll14adEgmwnnJIqtE5AogTkROAG4FFvkbVvjYxHVTnHHjxrFixQqmTJlC3759gw7HRAEvl+Q3A2fhlHSbAmQBUTPX4vDEdevDNEcaOXIkS5cutWRpPPOSMHup6l2q2sH9uhtnJcioYCtGmlC//PILPXv2ZP369YgIp59+etAhmSjiJWHeW8S2e8IdiF+sD9MU2Lx5MykpKXzxxRf88ssvQYdjolBJ1Yp6Ab2BJiISWv+yNh4qrkeKfbYAmgE2bdpEt27d2LlzJ3PmzOG8884LOiQThUrKIr8C3wKZwHch29OBu/0MKlxUNWQepiXMqmrTpk2kpKSwb98+5s6dy9lnnx10SCZKlVRAeBmwTEQmqmpmJcYUNgey81CFo6rFkxDvpffBxKL69evTrl07xowZQ4cOHYIOx0QxL82uJu6yum2A6gUbVbW1b1GFid0WWbWtXbuWxo0bU6tWLaZPnx50OCYGeGl2jccpGCw4o+PvAG/7GFPYZGRZ4Y2q6uuvv+b888/nT3/6U9ChmBjiJWHWUNU5AKq6XlXvJUqmFe2zOZhV0ldffUXXrl1JTEzkwQcfDDocE0O8NL2yRCQOWC8i1+Msu1vL37DC49CkdRvwqTIWL15Mr169qFOnDvPnz6dly5ZBh2RiiJdMcjtQE+eWyEeAOsC1fgYVLjZpvWrJzc1l6NCh1KtXj/nz59O8efOgQzIxptRMoqpfug/Tcdb2QUSa+BlUuBT0YdqgT9WQkJDAe++9R506dWjSJCp+RU2UKbEPU0TOFpFLRKSB+7ytiLwOfFnS+yKFrUleNcydO5d77rkHVaVNmzaWLI1vik2YIvIoMBEYDHwgIg8CC4AVQMRPKYLQaUU26BOrZs2aRd++fZk+fToZGRlBh2NiXElNr35Ae1U9KCL1gJ+A01V1Q+WEVp3eUkEAAB4uSURBVHHpNugT095//30uv/xyTj/9dD788ENq1YqKsUgTxUq6JM9U1YMAqroLWBNNyRKsDzOWvfvuuwwYMIAOHTowb9486tevH3RIpgooKZO0FJEp7mMBTgh5jqpGfHnqQ/eRW8KMOSLC+eefz7Rp06hdu3bQ4ZgqoqRMclmh58/5GYgfrA8z9vz0008cf/zxXHbZZVx66aUUWjrFGF+VVHxjXmUG4gebhxlb/vOf/3DzzTfz0Ucf0blzZ0uWptLFdAmfgvV8atsledR7/vnnGTFiBKmpqXTq1CnocEwVFdMJ0/owY8PTTz/NzTffTL9+/XjvvfeoXr166W8yxgeeE6aIJPkZiB/skjz6LViwgDvuuIMBAwYwefJkkpKi7tfQxJBSE6aIdBKRb4C17vP2IvKs75FVUF6+ciA7DxGomWgJM1p16dKF119/nTfffJNq1WzwzgTLSwvzGaAvsBNAVVcAXf0MKhwKKhUlJyYQF2eDA9FEVXnsscf4/vvvERGGDh1KQoL9p2eC5yVhxqnqD4W25fkRTDil26T1qKSq/PWvf2XUqFG89tprQYdjzBG8ZJOfRKQToCISD9wCrPE3rIqzwhvRR1X585//zDPPPMNNN93EI488EnRIxhzBSwvzBuAOoBmwDTjH3RbRDq9Hbv1e0SA/P58bb7yRZ555httvv51nn32WuLiYnsRhopCX5leuqg70PZIwy7AR8qiSlZXFqlWruOuuu3j00UdtUrqJSF6yyRIRWY2z8NkUVU33Oaaw2JdpfZjRIDc3l8zMTJKTk5kzZw6JiYmWLE3EKvWaR1VPBP4POAv4RkSmikjEtzgPX5JbwoxUOTk5XHnllVx44YXk5OSQlJRkydJENE+dRKr6uareCpwJ7MMpLFwqEektIqtFZJ2I3F3CfpeJiIpIR09Re2CFNyJbdnY2V1xxBZMnT6Zfv342x9JEBS8T15NFZLCITAcWA9uB8zy8Lx54HmdJ3jbAIBFpU8R+tYDbCPOyF9aHGbkyMzO59NJLmTp1Kv/617/4y1/+EnRIxnjiJZt8C0wHnlDVT8pw7E7AuoKiwyLyFk4V95WF9nsYeBy4swzHLlW69WFGrBtvvJGZM2fy4osvct111wUdjjGeeckmLVU1vxzHboKzrEWBzcD/C91BRM4EjlfVmSIS3oSZZS3MSDVq1Ci6devGkCFDgg7FmDIpNpuIyD9UdSTwXxHRwq9XtOK6iMQBTwHDPOw7AhgB0KxZM0/HP9yHaQkzEqSnp/Pqq69yyy230KpVK1q1ahV0SMaUWUnZ5G333/JWWv8ZOD7keVN3W4FawGlAmjsyeiwwTUQuVtWloQdS1bHAWICOHTv+JnkXJcMGfSLG3r17ufDCC1m8eDEXXHABHTp0CDokY8qlpIrri92Hp6rqEUlTRG4GSqvIvgRoJSIn4CTKgcCVIcffCzQIOWYa8JfCybK8Cu4lt0vyYO3atYtevXqxYsUKJk+ebMnSRDUv04quLWLb8NLepKq5wM3AHGAV8I6qficiY0Tk4rKFWXYZdkkeuB07dpCamsrXX3/NlClT6N+/f9AhGVMhJfVh/gGnVXjEapE4l9J7vBxcVWcBswptu7+Yfbt4OaZXVm09eMuXL2fDhg1Mnz6dnj17Bh2OMRVWUjZZjFMDsynOfMoC6cAyP4MKh31uC7O29WFWuuzsbBITE+nevTubNm2ibt26QYdkTFgUe0muqhtVda6qnq2q80K+FqtqTmUGWVZZuXlk5+aTECckJVjFm8r0008/0a5dO9555x0AS5YmppR0Sb5QVVNEZDcQOjItgKpqPd+jK6fQ/ku7N7nybNq0iW7durFz506OP/740t9gTJQp6ZK8YBmKBiXsE5Gs/7LyrVu3jm7dupGRkcG8efPo2DFsZQGMiRglXZIX3N1zPBCvqnnAucB1QM1KiK3cDk1aT7L+y8qwY8cOUlJSOHjwIPPnz7dkaWKWlw6+qTjLU5wIvAq0Aib5GlUF2fIUlat+/frcfPPNLFiwgDPOOCPocIzxjZeMkq+qOSJyKfCsqj4jIhE9Sn6o8IZNWvfVihUrEBHatWvHqFGjgg7HGN95WqJCRC4HhgKXuNsi+lrXigf7b+nSpfTs2ZOWLVuyZMkSG1wzVYLXO3264pR32+De6vimv2FVjF2S+2vRokWkpqZSp04dJk+ebMnSVBlelqj4FrgVWCoipwA/qWpEr39qK0b655NPPqFHjx40bNiQhQsXcsIJJwQdkjGVptQmmIhcAEzAKaAhwLEiMlRVP/M7uPJKt2rrvnn66adp0qQJ8+bNo0mTJkGHY0yl8pJRngb6qOpKABE5FSeBRuzckYJBn9p2SR42qoqI8MYbb5Cens4xxxwTdEjGVDovfZiJBckSQFVXAYn+hVRxNnE9vGbOnElKSgp79+6lRo0alixNleUlYX4lIi+KyO/cr38T4cU3bOJ6+EydOpX+/ftz4MAB8vLygg7HmEB5SZjXAxuAv7pfG3Du9olYGTZKHhaTJ0/m8ssv58wzz2Tu3LnUqxex5QOMqRQlZhQROR04EXhPVZ+onJAqbp+tGFlhU6ZMYeDAgZx33nnMnDmT2rVrBx2SMYErtoUpIqNxboscDHwkIkVVXo9Ih6YV2SV5uZ155plceeWVzJ4925KlMa6SLskHA+1U9XLgbOCGygmp4mzievmlpaWRn59PixYtmDBhAsnJyUGHZEzEKClhZqnqfgBV3V7KvhFDVQ+Pkts8zDJ57rnn6Nq1Ky+99FLQoRgTkUrKKC1D1vIR4MTQtX0qui65Xw7m5JGXryQlxJFo1dY9e+qppxg5ciSXXHIJw4eXusadMVVSSQnzskLPy7s+eaWy9cjL7tFHH2X06NFcfvnlTJw4kWrV7HtnTFFKWpe8tHXHI1K6VSoqkw0bNvDQQw8xePBgxo8fT0KCfd+MKU7M/XWk23rkZdKyZUu++OIL2rVrR3x8fNDhGBPRYq6TL8MKb5RKVfnrX//Ka6+9BkCHDh0sWRrjgeeEKSJJfgYSLuk2ab1Eqsptt93Gk08+yVdffRV0OMZElVITpoh0EpFvgLXu8/Yi8qzvkZVT+qEpRTZwUVh+fj7XX389zz77LCNHjuSf//xn0CEZE1W8tDCfAfoCOwFUdQWHl+CNONaHWTRV5Y9//CNjx45l9OjRPPnkk1Yp3Zgy8pIw41T1h0LbIrZsTYYlzCKJCC1atODBBx/k//7v/yxZGlMOXrLKTyLSCWep3XjgFmCNv2GVn/VhHiknJ4eNGzfSunVr7r///qDDMSaqeWlh3gDcATQDtgHnEMH3lWdYH+YhWVlZXHHFFZx77rns2LEj6HCMiXqlNsNU9VdgYCXEEhZWeMORmZnJZZddxqxZs3j22Wdp0KBB0CEZE/W8LIL2H0ALb1fVEb5EVEF2pw8cOHCASy65hLlz5/LSSy8xYkRE/qiMiTpessrckMfVgf7AT/6EU3GH+jCr8MT1xx9/nLlz5/LKK68wbNiwoMMxJmZ4uSR/O/S5iEwAPvUtogqy4hswevRoLrjgArp37x50KMbElPLcGnkCELHLBlbVFSP37NnD8OHD2bVrF0lJSZYsjfGBlzt9dovILvdrD/ARMMr/0MqnKk5c37VrF927d2fChAl2u6MxPiptETQB2gM/u5vyVfU3A0CRIj//cLX1molVI2Fu376d7t27s3r1aqZOnWotS2N8VGIL002Os1Q1z/2K2GQJkJF9uFJRfFzs38mydetWunTpwtq1a5k+fTp9+vQJOiRjYpqXPszlItLB90jCoKqVdsvNzSUhIYFZs2bRo0ePoMMxJuYVm1lEJEFVc4EOwBIRWQ/sx1nfR1X1zEqK0bOq0n+5bds2GjRoQNOmTVm2bBlxcTFX1tSYiFRSZlkMnAlcXEmxVFhGljMHM5ZHyDdu3EjXrl25+OKLeeaZZyxZGlOJSsosAqCq6ysplgrbF+OX5GvXrqVbt27s37+fq6++OuhwjKlySsosDUXkjuJeVNWnfIinQgr6MGvH4KT1VatWkZqaSk5ODgsWLKB9+/ZBh2RMlVNSwowHknFbmtEgPUZbmDk5Ofz+978nPz+ftLQ02rZtG3RIxlRJJWWWX1R1TEUOLiK9gX/hJN+XVfWxQq/fAfwRyAW2A9cWUazYs4I+zFgb9KlWrRrjxo3juOOO45RTTgk6HGOqrJJGDCrUsnSLDT8PXAi0AQaJSJtCuy0DOqpqO+Bd4ImKfGZGjJV2W7p0KePGjQOga9euliyNCVhJCTO1gsfuBKxT1Q2qmg28BfQL3UFVF6jqAffpIqBpRT5wXwwV3vjiiy9ITU3lb3/7GwcOHCj9DcYY3xWbMFV1VwWP3YQjy8BtdrcVZzgwu6gXRGSEiCwVkaXbt28v9gAFt0VGe2m3Tz75hJ49e9KoUSPS0tKoUaNG0CEZYyhftaKwE5EhQEfgyaJeV9WxqtpRVTs2bNiw2OPEwno+8+fPp3fv3jRt2pSFCxdy/PHHBx2SMcblZ2b5GQj9a2/K4SIeh4hId+AeIEVVsyrygbFQ2m3FihW0bNmSuXPncswxEVtFz5gqyc8W5hKglYicICKJOOsCTQvdwb1H/SXgYnftoApJj+I+zPT0dABuv/12Fi9ebMnSmAjkW8J070O/GZgDrALeUdXvRGSMiBTcbvkkzlzPySKyXESmFXM4T6K1+MZ7771Hy5YtWb58OQBHHXVUwBEZY4ria2ZR1VnArELb7g95HNbijfuisPjG22+/zeDBgzn77LM54YQTgg7HGFOCiBj0CZdom7j+xhtvcOWVV3Leeefx4YcfUqdOnaBDMsaUIGYSZk5ePpk5+cTHCUdViw86nFLNmzePq666ii5dujB79mxq1aoVdEjGmFLETMIM7b90VtaIbJ07d+bhhx9mxowZ1KxZM+hwjDEexE7CzIqOAZ8JEybw66+/Uq1aNe655x4b4DEmisRMwtwXBZPW//73v3PVVVfx97//PehQjDHlEDMJMyPCR8gfeeQR7rzzTv7whz/wyCOPBB2OMaYcYiZhRuqkdVXlgQce4N5772Xo0KG88cYbVKsWWTEaY7yJmYQZqX2YGRkZvP3221x77bW8+uqrJCREVnzGGO9i5q830gpvqCp5eXnUqlWLzz77jLp169qCZcZEuZj5C06PoMIb+fn53HLLLQwdOpS8vDzq169vydKYGBAzf8XpEbIAWn5+Ptdddx3PP/88xx9/vCVKY2JIzPw1R0Lhjby8PK699lpefvll7r33Xh5//PGomERvjPEm+OvXMCnowwwyYd5444289tprjBkzhvvuuy+wOIwx/oiZhHloeYoA+zCHDh1K69atGTlyZGAxGGP8EzOX5OkBrRiZlZXF+++/D8Dvfvc7S5bGxLCYS5iVOehz8OBB+vfvT//+/fnuu+8q7XONMcGIuUvyyurDPHDgAP369WPevHmMHTuWtm3bVsrnGmOCEzMJszInrmdkZNC3b18++eQTxo8fz1VXXeX7ZxpjghcTCVNVK3XFyA8++IBPP/2UN954g0GDBvn+ecaYyBATCTMrN5+cPCUxIY6kBP+qrasqIsKAAQP4/vvvOemkk3z7LGNM5ImJQZ9DlYp87L/cuXMnKSkpfPrppwCWLI2pgmKihel3/+Wvv/5K9+7dWbNmzaH1w40xVU9MJEw/+y9/+eUXUlNT2bRpEzNmzKB797CuDGyMiSIxkTDTfbqPfPv27aSkpLBlyxZmz55NSkpKWI9vjIkusdWHGeZJ63Xr1qVz587MmTPHkqUxJjZamIfuIw9TC3PDhg1Ur16dxo0b8/LLL4flmMaY6BcjLczwDfqsXbuWzp07c8UVV6CqFT6eMSZ2xETCzAhT4Y1Vq1bRuXNnsrOzeeGFF6yWpTHmCDGRMNOzKt6H+c033xzqp0xLS6Ndu3Zhic0YEztiog8zHKPkt912G4mJicyfP5/WrVuHKzRjTAyJkYRZ8T7Mt956i/T0dE488cRwhWWMiTExcUle3mrrn3/+OVdddRU5OTk0atTIkqUxpkQx0sIsex/mxx9/TJ8+fWjcuDG7du3imGOO8Ss8Y0yMiI0WZhn7MOfNm0fv3r1p1qwZCxcutGRpjPEkJhJmWVaMnDNnDn379uWkk04iLS2N4447zu/wjDExIjYSZpb39Xzq1q3LOeecw4IFC2jUqJHfoRljYkjUJ8zQaus1k4ovHrxmzRoAOnXqxPz586lfv36lxGeMiR1RnzD3Z+ehCjUS40mIL/p03nrrLdq0acOkSZMA7A4eY0y5RH3CLG3A5/XXX2fw4MGcf/75XHTRRZUZmjEmxkR9wixp0vq4ceMYNmwYXbp0YdasWdSqVauywzPGxJDoT5iHqq0fOeCzevVqRowYQa9evZgxYwY1a9YMIjxjTAyJ+onrBZPWaxdqYZ588slMnz6d1NRUkpKSggjNGBNjor6FWbgP86mnnmL+/PkA9OnTx5KlMSZsfE2YItJbRFaLyDoRubuI15NE5G339S9FpEVZPyO0D/Phhx9m5MiRTJw4scKxG2NMYb5dkotIPPA80APYDCwRkWmqujJkt+HAblU9SUQGAo8DfyjL52Rk5aKqfDn533z8zksMHTqUsWPHhus0jDHmED9bmJ2Adaq6QVWzgbeAfoX26Qe85j5+F0iVMk6S3Hswhz0Lx/PxOy8xfPhwXn31VeLji5/Abowx5eVnwmwC/BTyfLO7rch9VDUX2AuU6Rac9IM55O3fQ5dLBjN27FhLlsYY30TFKLmIjABGADRr1uyI1845sT5/fugf9GnXhLi4qB/DMsZEMD8T5s/A8SHPm7rbitpns4gkAHWAnYUPpKpjgbEAHTt2PGIpx96nHUfv06zikDHGf342yZYArUTkBBFJBAYC0wrtMw242n08AJivtratMSZC+dbCVNVcEbkZmAPEA6+o6nciMgZYqqrTgHHABBFZB+zCSarGGBORfO3DVNVZwKxC2+4PeZwJXO5nDMYYEy4SbVfAIrId+KHQ5gbAjgDC8ZudV/SIxXOCqnVezVW1YUlvirqEWRQRWaqqHYOOI9zsvKJHLJ4T2HkVZvNwjDHGI0uYxhjjUawkzFi9edzOK3rE4jmBndcRYqIP0xhjKkOstDCNMcZ3UZUwK6O+ZhA8nNcdIrJSRL4WkXki0jyIOMuitHMK2e8yEVERiYqRWC/nJSJXuD+v70RkUmXHWB4efgebicgCEVnm/h72CSLOshCRV0TkVxH5tpjXRUSecc/5axE5s9SDqmpUfOHcLbQeaAkkAiuANoX2uRF40X08EHg76LjDdF5dgRru4xsi/by8nJO7Xy3gY2AR0DHouMP0s2oFLAPqus8bBR13mM5rLHCD+7gNsCnouD2cV2fgTODbYl7vA8wGBDgH+LK0Y0ZTC7NS6msGoNTzUtUFqnrAfboIp5BJJPPyswJ4GKdodGZlBlcBXs7rT8DzqrobQFV/reQYy8PLeSlQ231cB9hSifGVi6p+jHPLdXH6Aa+rYxFwtIiUWMknmhJmpdTXDICX8wo1HOd/xUhW6jm5lz/Hq+rMygysgrz8rFoDrUXkMxFZJCK9Ky268vNyXg8CQ0RkM87tzrdUTmi+KuvfXnTUwzQOERkCdARSgo6lIkQkDngKGBZwKH5IwLks74JzJfCxiJyuqnsCjariBgHjVfUfInIuTtGc01Q1P+jAKlM0tTDLUl+TkuprRhgv54WIdAfuAS5W1axKiq28SjunWsBpQJqIbMLpP5oWBQM/Xn5Wm4FpqpqjqhuBNTgJNJJ5Oa/hwDsAqvoFUB3nfuxo5ulvL1Q0JcxYra9Z6nmJSAfgJZxkGQ19YiWek6ruVdUGqtpCVVvg9MterKpLgwnXMy+/g1NxWpeISAOcS/QNlRlkOXg5rx+BVAARORUnYW6v1CjDbxpwlTtafg6wV1V/KfEdQY9klXHUqw/O/9jrgXvcbWNw/tjA+SFOBtYBi4GWQcccpvOaC2wDlrtf04KOuaLnVGjfNKJglNzjz0pwuhtWAt8AA4OOOUzn1Qb4DGcEfTnQM+iYPZzTm8AvQA5Oy384cD1wfcjP6nn3nL/x8jtod/oYY4xH0XRJbowxgbKEaYwxHlnCNMYYjyxhGmOMR5YwjTHGI0uYEUhE8kRkechXixL2bVFcNZYyfmaaW61mhXtb38nlOMb1InKV+3iYiDQOee1lEWkT5jiXiMgZHt7zZxGpUY7P+qeIdHYf3+xWtVF3fmVx7+nrVvRZ4VYsuq6sn1tKTGPcmxgQkQvcikjLRaSJiLxbynsP/QxEZLSHz2ooIh+EJ/IYEfRcKfsqcv5YRhn2bUEx1VjK+JlpuPPQgBFUcK4nPs2tLBTnNcBHHt6zCWhQxs+pDywKed7B/V4XeyygGk5Riqbu8yTgZB9/T14Ehvj5Owa8Cpzv1zlE25e1MKOE25L8RES+cr/OK2KftiKy2G1xfC0irdztQ0K2vyQi8aV83MfASe57U90W0zdufcEkd/tjcrhG59/dbQ+KyF9EZADOPe8T3c88ym0ZdnRboU+GxDxMRJ4rZ5xfEFIsQUT+LSJL3VbXQ+62W4HGwAIRWeBu6ykiX7jfx8kiklzEsS8DDrWuVHWZqm4qJZ5aOPeS73Tfk6Wqq93PHC8iL7rxrRGRvu72eBF50m0tfx3aIhWRu9zv+woReSzkOANE5I/AFcDDIjIx9ErDPebfReRb95i3uNsLfgaPAUe53+eJbqv1zyGf+4iI3OY+nQoMLuW8q46gM7Z9/fYLyOPwXT3vudtqANXdx62Ape7jFrgtTOBZYLD7OBE4CjgVmA5Uc7e/AFxVxGemcbjldifwNs6dUz8Brd3trwN/xml9rebwEidHu/8+CPyl8PFCnwMNcUqJFWyfDfyunHH+GfhbyGv13H/j3f3auc834bYKce5//hio6T6/C7i/iM95DbioiO2HjlXMz+5l4Fecu0wGA3Hu9vE4CTjO/fltdr+/I4B73X2SgKXACcCFwOccroNaL+Q4A4p4HPp7cANOecOEQu8N/d5lhMTcAvjKfRyHc+dLffd5E+CboP8mIuXLqhVFpoOqWrhvrhrwnNtnl4dzj3JhXwD3iEhTYIqqrhWRVOAsYIk4pUGPwvmDLspEETmIkxRuAU4GNqrqGvf114CbgOdwaliOE5EZwAyvJ6aq20Vkgzj37q4FTsG55e6mMsaZCCQDod+nK0RkBE4r7zic2/m+LvTec9ztn7mfk4jzfSvsOMpxr7Sq/lFETge6A38BenC4KtM76lT3WSsiG3DOvSfQzm2Vg1MwppX7/lfVrYOqqiXVdSysO04h7Vwv71XVTSKyU5yaBccAy1S1oGjNrzgtdIOVd4smt+PcT94epxXwm6K7qjpJRL4Efg/Mci/vBHhNVUd5+IzBGlIAQ0TqFbWTquaKSCecYgwDgJuBbmU4l7dwLie/x2lBqzjZy3OcwP+AJ3Fa1ZeKyAk4CepsVd0tIuNxWnCFCU6/56BSPuNgMe8/8mAic3CSzFJV/SOAqn4DfCMiE4CNHE6Yhe9DVjeeW1R1TqHj9irts8PsZZw4jwVeCdleHed7YbBR8mhSB/jFbaEMxbnsPIKItAQ2qOozwPtAO2AeMEBEGrn71BPvawKtBlqIyEnu86HAQrfPr46qzsJJ5O2LeG86Tp9eUd7DqXY9CCd5UtY41blevA84R0ROwakGvh/YKyLH4FzSFhXLIuD8gnMSkZoiUlRrfRVuP25JVLWXqp7htiyTRaRLyMtnAD+EPL9cROJE5ESc5SBWA3OAG0SkmhtPaxGpCXwEXCPu6H5x/3kV4yPgOnFKHBb33pyCz3S9B/QGznZjKtAaqPAsjFhhCTN6vABcLSIrcC7l9hexzxXAtyKyHKfe5OuquhK4F/hQRL7G+WMqsQx/AVXNxBmJniwi3wD5OCOztYAZ7vE+Be4o4u3jgRcLBn0KHXc3TkJqrqqL3W1ljlNVDwL/AO5U1RU4a+l8D0zCucwvMBb4QEQWqOp2nJbUm+7nfIHz/SxsJm6ZNnAGj8SpNt4U+FpEXi7iPQL8VZxpT8uBhziySPKPOFW0ZuNUzMnEadmtBL5yB21ewul7/ACn/NhS91h/Kel7UcjL7md97f6+XFnEPmPd1ycCqLM0xQKcboO8kP26ut8Lg61LbkyxRORToK+GoVq620UwQ1VLnCsZFHGq4H8FXK6qa0O2fwz0c/+Tq/KshWlM8UYCzYIOwm/iTGZfB8wrlCwbAk9ZsjzMWpjGGOORtTCNMcYjS5jGGOORJUxjjPHIEqYxxnhkCdMYYzyyhGmMMR79f3BrvZLdLudPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "knn_probs = knn.predict_proba(X_test)[:,1]\n",
    "fpr, tpr, thresholds = roc_curve(y_test, knn_probs)\n",
    "def plot_roc(fpr, tpr, label=None):\n",
    "    plt.plot(fpr, tpr, linewidth=2, label=label)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.axis([0, 1, 0, 1])\n",
    "    plt.xlabel('False Positive Rate (1-Specificity)')\n",
    "    plt.ylabel('True Positive Rate (Sensitivity)')\n",
    "    plt.xlim((-0.02,1.02))\n",
    "    plt.ylim((-0.02,1.02))\n",
    "    plt.title('ROC Curve')\n",
    "plt.figure(figsize=(5,5))\n",
    "plot_roc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also for FPR and TPR there is the tradeoff similar to the one observed previously: the higher the rate of true positives, the lower the rate of false positives, and vice versa. The diagonal line in the middle represents the ROC curve of a random classifier, i.e., of a classifier that performs classifications without using any information. The more a classifier is far from this line (towards the top-left corner), the better it performs.   \n",
    "\n",
    "Let's now compare the ROC curves of the knn with the one of another classifier, in this case *knn* using $k=2$ neighbors (the plot, however, can be used also to classify different models, such as trees vs. logistic regression vs. NaiveBayes, not just knn models!):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUwAAAFNCAYAAABrMlb6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3hUZfbA8e9JDySBUEQBaQprRUFkFaVIEURFQFAQUYQVu67ycxV2VRbXupa1rrJiA1HERQRpSrWyiIJdioCCiAKhpJAymfP7Y2bCpE+SmdyZyfk8T57M3Llz5wzC8W33vKKqGGOMqVyM0wEYY0yksIRpjDEBsoRpjDEBsoRpjDEBsoRpjDEBsoRpjDEBsoRpjDEBsoRpQkpEtonIIRHJEpFdIvKyiKSUOKebiCwXkUwROSAi80XkhBLnpInIv0TkZ++1fvQ+b1LO54qI3Cwi34hItojsEJHZInJyKL+viW6WME1tuFBVU4BTgU7ARN8LInIm8B7wDtAcaAt8CXwsIu285yQAy4ATgQFAGnAmsBfoWs5nPgHcAtwMNAI6AHOB86savIjEVfU9Jkqpqv3YT8h+gG1AX7/nDwML/J5/CDxbxvsWAa96H/8J+A1ICfAz2wOFQNcKzlkJ/Mnv+RjgI7/nCtwAbAK2Av8GHilxjXeA27yPmwP/BXZ7z7/Z6T97+wn+j7UwTa0RkZbAecBm7/N6QDdgdhmnvwn08z7uCyxW1awAP6oPsENV19QsYgYDfwROAF4HLhURARCRdOBc4A0RiQHm42kZt/B+/p9FpH8NP9+EGUuYpjbMFZFMYDvwO3CP93gjPH8Hfy3jPb8CvvHJxuWcU56qnl+eB1Q1Q1UP4WkJK9Dd+9ow4FNV3QmcDjRV1Smqmq+qW4D/ACOCEIMJI5YwTW0YrKqpQC/gOA4nwn2AGziqjPccBezxPt5bzjnlqer55dnue6CefvcbwEjvocuA17yPWwPNRWS/7weYBDQLQgwmjFjCNLVGVVcBLwOPeJ9nA58Cw8s4/RI8Ez0AS4H+IlI/wI9aBrQUkS4VnJMN1PN7fmRZIZd4/jowTERa4+mq/9d7fDuwVVUb+v2kqurAAOM1EcISpqlt/wL6icgp3ud3Ald6lwCliki6iPwDzyz4373nTMeTlP4rIseJSIyINBaRSSJSKimp6ibgWeB1EeklIgkikiQiI0TkTu9p64GhIlJPRI4FxlUWuKquw9PqfQFYoqr7vS+tATJF5A4RSRaRWBE5SUROr84fkAlfljBNrVLV3cCrwN3e5x8B/YGheMYdf8Kz9Ohsb+JDVfPwTPz8ALwPHMSTpJoA/yvno24GngaeAfYDPwJD8EzOADwO5OOZfX+Fw93rysz0xjLT7zsVAhfgWTa1lcNJtUGA1zQRQjxDM8YYYypjLUxjjAmQJUxjjAmQJUxjjAmQJUxjjAmQJUxjjAlQxFVhadKkibZp08bpMIwxUebzzz/fo6pNKzonZAlTRF7Eszbtd1U9qYzXBU8JroFADjBGVb+o7Lpt2rRh7dq1wQ7XGFPHichPlZ0Tyi75y3hqF5bnPDxluNoD4/GUzzLGmLAVsoSpqh8AGRWcchGeeoeqqquBhiISjIIJxhgTEk6OYbbArxoMsMN7LBhluYwxUSzPVUhOXiFZeS6y813kZGeTn7kXV3YGBdn7cOfsg9z9kLuf2NwDxOYfIKHgILF5BzmUuZe4Vqdz5nXPVflzI2LSR0TG4+m206pVK4ejMcZUhaqSW+AmK89FTr7L+9ub7PJcRYkvNycL96H9qDfZxeTtJzbvIPEFnmSX6MokuTCT+u4sUsmiAdk0kGyakk2SFAQUy/pdhVzz30NMuz65Wt/FyYT5C3C03/OW3mOlqOpUYCpAly5d7OZ3Y0LI7VZyCgrJ9ia0bG9C8yW77LxCv8cusvO95+a6KMjNRvL2E5t3gLj8g95Ed5A0b3Lz/fYlu5Z+zxMDTHplDSQWEEdOTAqHYlPJjUslPy6NgoQ0ChMa4E5qiDsxjZh6jYitn86jvXNo0emP1fqzcTJhzgNuFJE38NQWPKCq1h03popche7DScsvgR1OcoXelpzncZnJruhxAZqfU5TQin57H6d5fzeWbNqVOCeNbBLFVTw4AeID/B4ST35cGvneRFeY2ABNaoAkpxNTryGx9dKJT2lEYkoj4us3guSGkNQQkhsSH1+PBiJllofKz8/nsssuY/DgwVw+aBQ12TY0lMuKXsdTYbuJiOzAsy1BPICqPgcsxLOkaDOeZUVXhSoWY8JJnquQbG8Sy84/3Io7nOQOt+iKt+78k9vh9+S53CU+QalHXomkl1WU1Jr7J0JKJEPJJiGpsNrfzR2TQGGip1UnSQ0Rb6KLSU4vluDK+h0Xn0ycSLGqzjWVm5vLsGHDWLBgAT179qzx9UKWMFV1ZCWv+3blMyZs+Y+/lUxUpZJdvmc8rmTiyy6RBAsKAxlVUuqTW6qF16ysZJeQTbrk0FA8SS9Fs4nHVflHlPfJcUlIBYmtot8x8clhc/tgTk4OQ4YM4b333uO5557jmmuuqfE1I2LSx5hAud3qmTX1S1bFE1zxZOfropZq3fmd5672qHnxpHe0N8k1isumSXwujWNyaBTrSXQNySaVLOprFvULs0hyZxKr1Wjp+WKNS65WwiOpIRKfVN0vHDYKCgq44IILWLlyJS+++CJXXRWcDqwlTOMoV6Hb0+3M942xlUh2+cUnHw635LzPSyS+nPzqdyfLpjSKy+OohDyOSDhE09hDNI07RHpsDo1ismlADmlkkaJZ1HNnUa8wk6TCTBILDhJXkEmMltPSUzw7p1cUbny9aiU8khtCXGKQ/xwiS3x8PD179mTs2LFcfvnlQbuuJUwTMFUlz+UmJ7+MVlpeRRMJhxOhf0svK89Ffqnxt5qrlxBL/cQ4UhLjPI8TYmkSn0eTuEM0ic2hoeSQHuMZz0shixT/ROfKJKHAM8Mbk3cAyT2AaKFnb8vcagQTXw+S06uR+BrU+aRXHfv372fHjh2cdNJJ3HPPPZW/oYosYUYxVeVQweEupn/SKjWRkF9B99Uv8bmq3z8tU4xA/YQ46ifGUT/Rk+hKPk9JjKNevNAoNo+G3kSXRhb13Z6f5MJMklwHSXBlEpd3APEuWOaQ92ffAdAaJOb4+tXu3hKXELw/LFOhjIwM+vfvzy+//MLmzZupVy+Y00celjAjnKvQzba92WzYlcWG3zLZsOsgG3/LYndmHtn5LoK9ZVN8rPgltdhyHsdR37+V50t8CTGkkE2aZhd1XxMKDiK5vx9OcP6/D/g9zztYs6SXkFK9hJfUwJJeBNizZw/9+vXju+++47///W9IkiVYwowYqsqOfYfY+FsmG37LZOOuTH7YlcmW3dnkF5afSBLjYkhJ9CSxegmx3gQWR0pibLktu5TEWOr5v5ZwOPElxgjkHfC23ry3nx36rXTCyywjAeYepPRW31WQkFq621oq0ZXV/W0AsQEuBjQR57fffqNPnz78+OOPzJ8/n3PPPTdkn2UJMwztycpjw65MNuzKLJYgs8uZ0Di6UTJ/aJZKh2ap/OFIz+/mDZOpnxBLXGwZizzchZB7oOxWXc5+yCjjeLCSXmKaN5E1CKCFl168pRdrf11Naffeey9bt25lwYIF9O7dO6SfFXHb7Hbp0kWjpR5mZm4BG3/L8iRFvwS5Nzu/zPObpCRynDchHtcsmeMaKsemuqjnziw/wZX6fcDTQqyJxAYBJrwSLb/ENEt6Juhyc3P57rvv6Ny5c42uIyKfq2qXis6xv721ILegkB93+xJjVtE44679WaXure1GNkck5tIutYCjk/M5MiGXxrE5pGk28QUHIPMA7N4Pnx+sQUQCSWnVH9OLiQ3an40x1bFt2zYmTJjACy+8QHp6eo2TZaAsYYbYq59u4953vyt2d0c8Lv4V/zTnJ60p/41Z3p9ySTljeAH8TkyzpGci1ubNm+nduzdZWVn8/PPPpKen19pnW8IMoU9+3MPked/iVjimaX3P+OIRKQzd+Qittq5BEaSqSc83qZGYBjHhchOaMbVjw4YN9O7dm7y8PJYvX84pp5xSq59vCTNEdh3I5ebX1+FWuPGcY/m//n/wvPC/qfDxmxCXhIxZCC1PczZQYyLEd999R+/evVFVVq5cyUknldoqLOSsiRICBYVubpz5BXuy8jnr2Mbc2q+D54UfV8DiOz2PBz1tydKYKkhJSeGYY45h1apVjiRLsBZmSDy46AfW/rSPI9OSeHJEJ2JjBPb+CLOvBC2E7hOg43CnwzQmImzevJm2bdvSqlUrPvroIzwbzjrDWphBtvDrX5n20VbiYoRnRnWmcUqiZ0nPzEs9ax//cD6c8zenwzQmIqxevZrTTjuNu+++G8DRZAmWMIPqx91Z3D77SwD+dv7xnNY6HQpd8NZY2LsJjjgRhk61yRpjAvDRRx/Rr18/mjZtGpRalsFg/3KDJCffxXUzPic7v5ALT2nOld3aeF54/274cRnUawwjX4fEFEfjNCYSrFy5kv79+9OiRQtWrVoVNpsfWsIMAlVl4pyv2fhbFscekcKDQ0/2dB2+eBVWPwMx8XDpDEhv7XSoxoS9zMxMhg0bRps2bVi5ciUtWrRwOqQiNukTBDNW/8Q763dSLyGW5y7vTP3EOPjpU3j3Ns8JFzwGrbs5G6QxESI1NZU5c+Zw/PHH07RpU6fDKcYSZg2t376fKe9+B8CDF3fk2CNSYd9PMOtycBfAGddD5yscjtKY8PfOO++wZ88exo0bR48ePZwOp0zWJa+BjOx8rp/xOQWFyphubRh0SnPIy4LXR0LOHjimD/S71+kwjQl7s2fPZtiwYUybNg2Xq/obuIWaJcxqKnQrt7yxjp0HcunUqiGTBh4PbjfMGQ+/fwuNj4VhL1p1HmMqMXPmTEaMGMEZZ5zB4sWLiYsL338zljCr6cllm/hw0x4a1U/g2VGdSYiLgRX3wYYFnqIYI2d57v02xpTr5Zdf5vLLL6dnz54sWrSItLQ0p0OqkCXMali54XeeXL4JEXhyRCeOapAMX82GDx8BiYXhL0OTY50O05iw99tvv9G3b1/effddUlLCf8mdFRCugvXb9/Pyx1tZ8PWvFBQqE/p14KY+7eGXz+GlgeDKhfMehj+GxyJbY8LV7t27i2bAXS5XWHTDAykgbC3MShQUupn35U6GPPsxg5/5mLnrd1LoVkZ2bcUN5xwLB3fC65d5kmXnK6HreKdDNiasPfbYY3To0IEffvgBICySZaAiJ9Jatjcrj9fX/Mz01T/x28E8ANKS4hjZtRWjz2xNy/R6UHAI3rgMsnZB67Ng4CPg8L2uxoSzBx54gEmTJjF8+HCOOeYYp8OpMkuYJXy38yAvfbyVd77cSb7Lsxtj+yNSGHNWG4Z0akG9BO8fmSq8cwPsXAcNW8El0207VmPKoapMmTKFyZMnM2rUKF5++eWIaln6RF7EIfLet7uY9tFW/rc1A/A0FPscdwRXndWWs45tXLpKyoePwjf/9ex3PXIW1G/sQNTGRIbXX3+dyZMnM2bMGF544QViYyNzixRLmHhKsl3/2hcApCTGMbxLS648sw1tmtQv+w3fz4fl9wICQ/8DzU6ovWCNiUDDhg3jwIEDXHPNNcREcLUuS5h4WpcAl5/RijsGHEdqUnz5J+/6BuZ4Z8H73gPHDayFCI2JPG63m4cffphx48bRtGlTrrvuOqdDqrHITfVBtPanfQBc1rV1xckya7fntseCbOh4KZz151qK0JjI4na7ufbaa5k4cSKvvfaa0+EETZ1vYe46kMuOfYdITYzjD0emln+iKw/eHA0HfoYWp8GFT9qMuDFlKCwsZNy4cbzyyitMmjSJW265xemQgqbOJ8y1P3kmeU5t1dCz905ZVGHBbfDzp5DaHEbMhPikWozSmMjgcrm48sormTlzJn//+9+56667HN9WIpgsYW7zdMe7tG5U/kmr/w3rZkBcMoycCalH1lJ0xkSWffv28dlnn/HAAw9w5513Oh1O0NX5hPm5d/zy9DbpZZ+waSm891fP48HPQvNOtRSZMZEjPz+fmJgYmjZtyrp166hfv5wVJhGuTk/6ZOe5+O7Xg8TGCKe2KqOy0O6N8NZVoG7o8Rc4aWjtB2lMmMvNzWXIkCFcddVVqGrUJkuo4wlz/fb9FLqVE45KO3wHj09OBrx+KeQdhOMvhF4TnQnSmDCWk5PDoEGDWLRoEd27d4+q8cqy1OkuuW/88rTWJbrjhQUwewxkbIFmJ8OQ521rXGNKyMrK4sILL2TVqlW8+OKLjBkzxumQQq5uJ0zvDPnpbUpM+CyZBFtXQf2mnq1xE6K3i2FMdV1yySV88MEHTJ8+nVGjRjkdTq2os82mQrey7uf9AHTxn/BZ+yKsmQqxCXDpa9DwaIciNCa83XHHHbzxxht1JllCHW5h/rDrIFl5LlqmJ9MszbumcuuHsPB2z+ML/gWt/uhcgMaEoYyMDJYsWcLIkSPp2bOn0+HUujrbwjy8nMjbHc/Y6rmTx+2CM2+ETnXn/5rGBGL37t2cc845jB07lh07djgdjiPqbAuz2IRPXqbnHvFD++DYftBvisPRGRNedu3aRZ8+fdi6dSvz5s2jZcuWTofkiJC2MEVkgIhsEJHNIlJq2b+ItBKRFSKyTkS+EpFaK/2zdptnwqdLm3T46k3Y/T006QDDpkFMZNbqMyYUfvnlF3r16sVPP/3EwoUL6devn9MhOSZkCVNEYoFngPOAE4CRIlKycOTfgDdVtRMwAng2VPH427n/EDsP5JKaFEeHI1Lh4C+eF04e7tki1xhT5L333mPnzp0sXryYXr16OR2Oo0LZwuwKbFbVLaqaD7wBXFTiHAV8GxE3AHaGMJ4ivnJup7VOJyZGIHuP54V6VjXdGJ/CwkIArrrqKjZu3MjZZ5/tcETOC2XCbAFs93u+w3vM32TgchHZASwEbirrQiIyXkTWisja3bt31ziwz33dcd+C9Zy9nt+WMI0BYNOmTZx88sl88sknABx5pBWcAednyUcCL6tqS2AgMF1ESsWkqlNVtYuqdvHtZVwTnxVN+HhnyH0Js36TGl/bmEj3/fff07NnT3bv3h3V94VXRygT5i+A/6rvlt5j/sYBbwKo6qdAEhDSrJWV5+KHXQeJixFOPdpbcKOoS24J09Rt33zzDb169cLtdrNy5UpOOeUUp0MKK6FMmJ8B7UWkrYgk4JnUmVfinJ+BPgAicjyehFnzPncF1v28D7fCiS0akJzgnQ3PsTFMY3788Ud69epFXFwcq1at4sQTT3Q6pLBTYcIUkdNF5AkR+UJEfhWRLSIyT0SuEZEK9nMAVXUBNwJLgO/xzIZ/KyJTRGSQ97QJwNUi8iXwOjBGVbXmX6t8hwsGe8cvC11waD8gUK+CIsLGRLnWrVszcuRIVq1axR/+8AenwwlL5S5cF5EFwB7gHeBR4Hc8LcAOwDnAAhF5WFXfLe8aqroQz2SO/7G7/R5/B5xVky9QVb6CG0UJ89A+QCG5ka2/NHXSZ599RsuWLTnqqKN46qmnnA4nrFV0p89YVf2txLFcYI335yEROSJkkYWAq9BdVHDjNF/BDeuOmzrsww8/ZODAgfTu3Zt33nnH6XDCXrldcl+yFJHrRKTM1dyq+nuoAguF7fsOkZNfSPMGSRyR6i24YTPkpo5avnw5AwYMoGXLlvz73/92OpyIEMikT2vgCxGZKSJ9Qx1QKB08VABA45TEwwdt0bqpg5YsWcL5559Pu3btWLlyJc2bN3c6pIhQacJU1TuB9sBrwLUissk7cdMmxLEFXWauC4DUJL+RCOuSmzrG7XYzadIkjjvuOFasWEGzZs2cDiliBFStSFXdIrIN2AacDBwFvCMiC1U1Yja7ycrztDBTEv0TpmcSyLrkpq6IiYlhwYIFJCQk0KiRrQypikpbmCJyg4isAZ4APgc6qurVQCfg0hDHF1QHvS3MFP8Wpi1aN3XEm2++yYgRI3C5XBx55JGWLKshkDHM5sBIVe2rqq+rah54Wp3AoIrfGl6yvAkzLSn+8EHrkps6YMaMGYwcOZKdO3eSm5vrdDgRK5CE2UJVf/Q/ICIvA6jqN6EIKlR8Y5jFu+S+WXJLmCY6vfjii1xxxRX06tWLRYsWkZKS4nRIESuQhNnR/4m3OMbpoQkntHxjmMUmfbJ9lYqsS26iz7Rp0xg3bhznnnsu7777rhXTqKFyE6aI3CEi+4COIpLh/dmH5+6fheW9L5xlljWGaV1yE8VOPPFERo4cydy5c0lOTnY6nIhXUQvzYaAp8Lj3d1Ogiao2UtXbayO4YMvM8y0r8o5hqh6e9LFZchNFPvvsMwDOOOMMZs6cSVJSksMRRYeKEuax3gIa04ETfT8i0lFEOlbwvrBVtA7TN4aZlwnuAoivD/H2f18THe677z66du3KggULnA4l6lS0DvNOPPUqnynjNQV6hCSiEMrKLTGGad1xE0VUlcmTJzNlyhRGjx5N//79nQ4p6pSbMFV1nPd399oLJ7Sy8kqMYWbbDLmJDqrKxIkTeeihhxg7dixTp04lNtaqbwVbIAvXvxCR20WkdW0EFEqHb430jmHm2Ay5iQ6rV6/moYce4tprr+U///mPJcsQCeTWyOF47uiZJyI5wCw8xYBrZYfHYMoquQ7TuuQmSpx55pmsWrWK7t27IyJOhxO1Aim+8aOq3q+qpwBjgc54tpaIKG63kpVfImHaDLmJYG63m5tvvpmVK1cC0KNHD0uWIRZQ8Q0RaQlcgqelGQf8NZRBhUJ2vgtVqJ8QS2yM9y+Vba9rIlRhYSHjxo3jlVdeoUmTJvTq1cvpkOqEShOmiHwCpACzgctVdVPIowqBshetW8I0kcflcnHllVcyc+ZMpkyZwl133eV0SHVGIC3Mq1X125BHEmJZJRetg3XJTcQpKCjgsssu46233uLBBx/kjjvucDqkOqWiTdBGqurrQB8R6VPydVV9MqSRBVlmblm1MG2W3ESWmJgYkpKSeOyxx7j11ludDqfOqaiF6d0ljKZlvBbSrXBDwaqtm0iWm5tLRkYGzZs359VXX7XJHYdUtHD9We/DBaq62v81ETkjpFGFQJkJ0xaumwiQk5PDRRddxM8//8xXX31FYmJi5W8yIRFIebdnyzhW1u2SYa1oDDPRO4bpyoP8TJBYSGroYGTGlC8rK4uBAweyfPlyJk2aZMnSYRWNYXYFzgSaisjNfi+lAfFlvyt8ZZWcJfefIbfujQlDBw8eZODAgaxevbqoYrpxVkVjmPWBJt5z/McxM/Hc/RNRMksW3rAZchPmbr31Vv73v/8xa9YsLr74YqfDMVQ8hrkCWCEiL6nqllqMKSR8tTAP3xZpazBNeHvooYe49NJLOffcc50OxXhVVHH9Ue/DR0VkTsmfWoovaDJLboBWtJePtTBN+Pj999+57bbbyM/Pp0mTJpYsw0xFXfJZ3t9P10YgoVZqDDPblhSZ8PLrr7/Sp08ftm3bxqhRozjttNOcDsmUUFGXfI339zLfMRFpgGcXye9qIbagyswrsXDdFq2bMLJjxw569+7Nzp07WbRokSXLMBVIPcxlIpImIunAemC6iPwz9KEFV1bJdZg5NuljwsNPP/1Ez5492bVrF0uWLKFnz55Oh2TKEcg6zEaqehAYCsxQ1dOAiKt9X2rhelGXvJFDERnjkZGRgdvtZunSpZx11llOh2MqEEjxjTgRaYpnKdHdIY4nZErtGJmT4fltXXLjkIyMDBo1akSnTp3YuHEj8fERt7y5zgmkhXkfsAr4WVXXiEg7YGtowwq+UsU3rEtuHPT9999z4okn8sQTTwBYsowQlbYwVfUN4A2/51uAi0IZVLAVFLrJLXATI1AvwbvXic2SG4d8/fXX9OnTh9jYWPr16+d0OKYKAikg3ATP1hRt/M9X1fGhCyu4sv0WrYsIuN1wyNclt4Rpas+6devo168fSUlJLF++nA4dOjgdkqmCQMYw3wFWAx8BhaENJzRK7RaZux/UDUkNINa6QqZ27N+/n759+5KSksLy5cs55phjnA7JVFEgCbO+qk4IeSQhVP4MubUuTe1p2LAhTz/9NN26daN164jftbpOCmTSZ5GIRPT9WaUKb9iidVOLPvjgA9577z0ARo4cackyggXSwrwWuMO7J3k+IICqasQsYMwqVXjDZshN7Vi2bBkXXnghxx9/PH379iUmJpA2iglXgSTMiM8qh3eM9I5X2qJ1UwsWL17MkCFDaN++PYsWLbJkGQUq/S+oqoV4Fq3f4X18FHBqqAMLpsOL1q1LbmrH/Pnzueiiizj++ONZsWIFRxxxhNMhmSAI5F7yp4FzgNHeQznAc6EMKtiKxjBLFt6wLrkJkUWLFnHKKaewbNkyGje2ycVoEUgfoZuqXgPkAqhqBpAQyMVFZICIbBCRzSJyZznnXCIi34nItyIyM+DIq6BU4Q2bJTchkpubC8DTTz/N8uXLSU9Pr+QdJpIEkjALRCQG79a6ItIYcFf2JhGJxbNZ2nnACcBIETmhxDntgYnAWap6IvDnqoUfmKIxTCvtZkLo1Vdf5cQTT2THjh3ExMSQkpLidEgmyAJJmM8A/8WzGdrf8SxgfyiA93UFNqvqFlXNx3N7ZclbKq8GnlHVfQCq+nvAkVdBVqnCG75ZcmthmuCYNm0aY8aMoU2bNtaqjGKB3Ev+qoh8DvT1Hhquqt8EcO0WwHa/5zuAP5Y4pwOAiHwMxAKTVXVxANeuksxS1dZtPx8TPM8++yw33HADAwYMYM6cOSQnJzsdkgmRivb0SfJ2q1HVb4EFeLri7YL4+XFAe6AXMBL4j4iU2iRcRMaLyFoRWbt79+4qf4gtXDeh8vrrr3PDDTdw4YUXMnfuXEuWUa6iLvkS4BgAETkGWINnLPI2EbkvgGv/Ahzt97yl95i/HcA8VS1Q1a3ARjwJtBhVnaqqXVS1S9OmTUu+XKmiLnliPORng+sQxCVBQv0qX8sYf+eddx4TJ07krbfeIjEx0elwTIhVlDAbqepG7+MrgTdU9To81dYHBXDtz4D2ItJWRBKAEcC8EufMxdO69FVF6gAEfUvfYveS+8+Qi5BUL+kAACAASURBVAT7o0wd8dprr3Ho0CEaNmzI/fffT0JCQAtHTISrKGGq3+PewPsAqppHALPkquoCbsTTUv0eeFNVvxWRKSLiS7hLgL0i8h2wArhdVfdW/WtUrOjWyKQ424/c1Iiqctddd3H55Zfz/PPPOx2OqWUVTfp8KyIP4ulGdwDeg6KdIwNqmqnqQmBhiWN3+z1W4DbvT0ioavFq67Zo3VSTqnLnnXfy8MMPM27cOG666SanQzK1rKIW5p+ALOA4YICqZnuPnwQ8FurAgiXP5aagUEmIjSEpPtYWrZtqUVVuvfVWHn74Ya677jqmTp1KbGys02GZWlbRvuTZwD/KOP4x8HEogwqmUkuKbIbcVMMvv/zCjBkzuOWWW3j88cc9lftNnVNuwhSRucDzwPve8Uj/11rjmQjaoaovhjbEmskqVXjDFq2bwLndbkSEli1bsn79elq0aGHJsg6rqEt+A9AP2Cgin4rIPBF5T0Q2Ay8B34Z7soQydou0LrkJkMvlYsyYMdxzzz0AtGzZ0pJlHVduwlTVX1T1NlVth6dS0T+BSUAnVe2tqv+trSBrolThDduP3ASgoKCA0aNHM336dFsyZIoEUkAYVd0MbA5xLCFxsKjwRsn7yC1hmrLl5+czcuRI5syZw8MPP8ztt9/udEgmTASUMCOZbwwzzUq7mQCoKiNGjODtt9/mX//6F7fccovTIZkwEv0J0zeGabPkJgAiwtChQ+nXrx/XXXed0+GYMBNQwvTe2tjK2zWPKMVuiyws8OxJLjGQXKrGh6nDsrOz+eKLL+jevTuXX3650+GYMBXIFhXnA1/jvTVSRE4VkbdDHViwHN4xMv7whE9yOsTYomPjkZmZyXnnnUf//v3ZtWuX0+GYMBZIAeEpeOpY7gdQ1fXAsaEMKpgO+rcwfRM+1h03XgcOHKB///588sknvPTSSxx55JFOh2TCWCBd8gJV3V9i/ZmWd3K4KbZw3e4jN3727dtH//79Wb9+PW+++SZDhw51OiQT5gJJmN+LyCVAjIi0BW4GVoc2rOAptnDdZsiNn2nTpvHll18yZ84cLrjgAqfDMREgkC75jcBpeEq6zQHygIhZa3F44Xq8lXYzxUyYMIG1a9dasjQBCyRh9lfVO1S1k/fnTjw7QUaEYjtGWpe8zvv1118599xz+fHHHxERTj75ZKdDMhEkkIT5tzKO/TXYgYRKsTHMbJv0qct27NhBz549+fTTT/n111+dDsdEoIqqFfUHBgAtRMS//mUaAVRcDxcH/TdAy7ExzLpq27Zt9O7dm71797JkyRK6devmdEgmAlU06fM78A2QC3zrdzwTuDOUQQWLqvqtw/TvklvCrEu2bdtGz549OXjwIEuXLuX00093OiQToSoqILwOWCcir6lqbi3GFDQ5+YWoQnJ8LHGxMX77kVuXvC5p3LgxHTt2ZMqUKXTq1MnpcEwEC2RZUQvvtronAEm+g6raIWRRBUlmqdJu1iWvSzZt2kTz5s1JTU1l/vz5TodjokAgkz4v4ykYLHhmx98EZoUwpqDJyvMrvKFqs+R1yFdffcVZZ53F1Vdf7XQoJooEkjDrqeoSAFX9UVX/RoQsKzrovwYz9wC4XZCQCnGJDkdmQumLL77gnHPOISEhgcmTJzsdjokigXTJ80QkBvhRRK7Fs+1uamjDCo6iRev+Ez71GjkYkQm1NWvW0L9/fxo0aMDy5ctp166d0yGZKBJIwrwVqI/nlsj7gAbA2FAGFSy2aL1ucblcjB49mkaNGrF8+XJat27tdEgmylSaMFX1f96HmXj29kFEWoQyqGDxjWF6Fq3/5jloM+RRKy4ujrfffpsGDRrQokVE/BU1EabCMUwROV1EBotIE+/zE0XkVeB/Fb0vXBTbk9xmyKPW0qVL+etf/4qqcsIJJ1iyNCFTbsIUkQeA14BRwGIRmQysAL4Ewn5JEfgvK4q3RetRauHChVxwwQXMnz+frKwsp8MxUa6iLvlFwCmqekhEGgHbgZNVdUvthFZzmf6TPnYfedR55513GD58OCeffDLvvfceqakRMRdpIlhFXfJcVT0EoKoZwMZISpZQYgzTSrtFlbfeeothw4bRqVMnli1bRuPG9t/VhF5FLcx2IjLH+1iAtn7PUdWwL09ddB+5VVuPOiLCWWedxbx580hLS3M6HFNHVJQwLy7x/OlQBhIKxcYwrUseFbZv387RRx/NxRdfzNChQymxdYoxIVVR8Y1ltRlIKBRfh+lLmLZwPVL95z//4cYbb+T999+nR48elixNrQvk1siI5dvPJy0p7vAWu9Ylj0jPPPMM48ePp0+fPnTt2tXpcEwdFdUJs2gMM84F+VkQEw+JNt4VaR5//HFuvPFGLrroIt5++22SkpIqf5MxIRBwwhSRiKtYUTSGWXjAc6BeY7BuXERZsWIFt912G8OGDWP27NkkJkbcX0MTRSpNmCLSVUS+BjZ5n58iIk+FPLIaKnQrOfmFiEC9gv2eg9Ydjzi9evXi1Vdf5fXXXyc+Pt7pcEwdF0gL80ngAmAvgKp+CZwTyqCCwVepKCUhjphDtgYzkqgqDz74ID/88AMiwujRo4mLC6ROjDGhFUjCjFHVn0ocKwxFMMGUaYvWI5Kq8pe//IWJEyfyyiuvOB2OMcUE8r/t7SLSFVARiQVuAjaGNqyaK154wxatRwJV5c9//jNPPvkkN9xwA/fdd5/TIRlTTCAtzOuA24BWwG/AGd5jYe3wfuS2aD0SuN1urr/+ep588kluvfVWnnrqKWJionoRh4lAgbQwXao6IuSRBFmWLVqPKHl5eXz//ffccccdPPDAA7Yo3YSlQBLmZyKyAc/GZ3NUNTPEMQXFwdwyxjCtSx52XC4Xubm5pKSksGTJEhISEixZmrBVaZ9HVY8B/gGcBnwtInNFJOxbnIe75HG2H3mYKigo4LLLLuO8886joKCAxMRES5YmrAU0SKSqn6jqzUBn4CCewsKVEpEBIrJBRDaLyJ0VnHexiKiIdAko6gAULx5s1dbDTX5+PpdccgmzZ8/moosusjWWJiIEsnA9RURGich8YA2wG+gWwPtigWfwbMl7AjBSRE4o47xU4BaCvO1Flm2AFrZyc3MZOnQoc+fO5YknnuD//u//nA7JmIAEMob5DTAfeFhVP6zCtbsCm31Fh0XkDTxV3L8rcd69wEPA7VW4dqWKCm8kyuHCG8k26RMOrr/+ehYsWMBzzz3HNddc43Q4xgQskITZTlXd1bh2CzzbWvjsAP7of4KIdAaOVtUFIhLchOkdw0yXbEAhqSHE2t0i4WDixIn07t2byy+/3OlQjKmScjOIiDyqqhOA/4qIlny9phXXRSQGeAwYE8C544HxAK1atQro+r4xzEbindS37rijMjMzeemll7jpppto37497du3dzokY6qsoibXLO/v6lZa/wU42u95S+8xn1TgJGCld2b0SGCeiAxS1bX+F1LVqcBUgC5dupRK3mXxjWE21IOeAzZD7pgDBw5w3nnnsWbNGrp3706nTp2cDsmYaqmo4voa78PjVbVY0hSRG4HKKrJ/BrQXkbZ4EuUI4DK/6x8AirKYiKwE/q9ksqyuonvJC61SkZMyMjLo378/X375JbNnz7ZkaSJaIMuKxpZxbFxlb1JVF3AjsAT4HnhTVb8VkSkiMqhqYVadr4VZv6gWpk341LY9e/bQp08fvvrqK+bMmcOQIUOcDsmYGqloDPNSPK3CYrtF4ulK7w/k4qq6EFhY4tjd5ZzbK5BrBsq3cD3Ztc9zwLrktW79+vVs2bKF+fPnc+655zodjjE1VtEY5ho8NTBb4llP6ZMJrAtlUMFw0NvCTMr3Jkzrktea/Px8EhIS6Nu3L9u2bSM9Pd3pkIwJinK75Kq6VVWXqurpqrrM72eNqhbUZpBVlecqJN/lJi5GiM31tTDtLp/asH37djp27Mibb74JYMnSRJWKuuSrVLWniOwD/GemBVBVDdtBwazcw/eRi5V2qzXbtm2jd+/e7N27l6OPPrryNxgTYSrqkvu2oYi4TFO0W2SSX2m3+tbCDKXNmzfTu3dvsrKyWLZsGV26BK0sgDFho6Iuue/unqOBWFUtBM4ErgHq10Js1VZUeCMx/vBtkdYlD5k9e/bQs2dPDh06xPLlyy1ZmqgVyLKiuXi2pzgGeAloD8wMaVQ1VLQ9RWKsVVuvBY0bN+bGG29kxYoVnHrqqU6HY0zIBHJztVtVC0RkKPCUqj4pImE9S+4rvNE0oQAK8yC+HiTUcziq6PPll18iInTs2JGJEyc6HY4xIRfQFhUiMhwYDQz2Hgvr4oW+McxmcdmeA9YdD7q1a9dy7rnn0q5dOz777DMr/GvqhEDv9DkHT3m3Ld5bHV8PbVg14+uSN431Ft6whBlUq1evpk+fPjRo0IDZs2dbsjR1RiBbVHwD3AysFZHjgO2qGtb7n/pamE1irFJRsH344Yf069ePpk2bsmrVKtq2bet0SMbUmkq75CLSHZiOp4CGAEeKyGhV/TjUwVVXUWk3fJWKrIUZLI8//jgtWrRg2bJltGjRwulwjKlVgYxhPg4MVNXvAETkeDwJNGzXjvgmfRr6Nri0GfIaU1VEhBkzZpCZmUmzZs2cDsmYWhfIGGaCL1kCqOr3QELoQqq5oh0j3b7SbtbCrIkFCxbQs2dPDhw4QL169SxZmjorkIT5hYg8JyJne3/+TZgX3yhah+nylXazhFldc+fOZciQIeTk5FBYWOh0OMY4KpCEeS2wBfiL92cLnrt9wpbvXvIkl7eFaV3yapk9ezbDhw+nc+fOLF26lEaNwrZ8gDG1osIxTBE5GTgGeFtVH66dkGruoHcM00q7Vd+cOXMYMWIE3bp1Y8GCBaSlpTkdkjGOK7eFKSKT8NwWOQp4X0TKqrwelnxjmPF5dh95dXXu3JnLLruMRYsWWbI0xquiLvkooKOqDgdOB66rnZBqzjeGGZtrCbOqVq5cidvtpk2bNkyfPp2UlBSnQzImbFSUMPNUNRtAVXdXcm7YUFWy8lzE4yIm7yBIrGdPclOpp59+mnPOOYfnn3/e6VCMCUsVjWG289vLR4Bj/Pf2qem+5KFyqKCQQrfSPC7Lc6BeI4iJiFzvqMcee4wJEyYwePBgxo2rdI87Y+qkihLmxSWeV3d/8lrlmyFvkXgICrEZ8gA88MADTJo0ieHDh/Paa68RHx/WtVWMcUxF+5JXtu94WMr0Tvi0iM/2JEybIa/Qli1b+Pvf/86oUaN4+eWXiYsL5OYvY+qmqPvX4ZvwOTI+C3Kx/cgr0a5dOz799FM6duxIbGys0+EYE9aibnDP1yVvGuMbw7QWZkmqyl/+8hdeeeUVADp16mTJ0pgABJwwRSQxlIEEi6/whpV2K5uqcsstt/DPf/6TL774wulwjIkolSZMEekqIl8Dm7zPTxGRp0IeWTX5xjDTseLBJbndbq699lqeeuopJkyYwL/+9S+nQzImogTSwnwSuADYC6CqX3J4C96w4xvDbKhWC9OfqvKnP/2JqVOnMmnSJP75z39apXRjqiiQhBmjqj+VOBa2ZWt8Y5ipbm+lIuuSAyAitGnThsmTJ/OPf/zDkqUx1RDILPl2EemKZ6vdWOAmYGNow6o+3xhmSqGvUlHdbmEWFBSwdetWOnTowN133+10OMZEtEBamNcBtwGtgN+AMwjj+8p9hTeSC6y0W15eHpdccglnnnkme/bscTocYyJepS1MVf0dGFELsQRFZq4LwU1iQd1uYebm5nLxxRezcOFCnnrqKZo0qbv/4zAmWALZBO0/gJY8rqrjQxJRDWXmuUgjhxgthMQ0iAvr3TRCIicnh8GDB7N06VKef/55xo8Py/9UxkScQMYwl/o9TgKGANtDE07NZeYW0Ejq9pKihx56iKVLl/Liiy8yZswYp8MxJmoE0iWf5f9cRKYDH4UsohrKynUd3l63js6QT5o0ie7du9O3b1+nQzEmqlTn1si2QNhuG5iV56Kx+NZg1p2EuX//fsaNG0dGRgaJiYmWLI0JgUDu9NknIhnen/3A+8DE0IdWPZm5LtLFdx953eiSZ2Rk0LdvX6ZPn263OxoTQpVtgibAKcAv3kNuVS01ARQu3G5PtfXGsb4uefQnzN27d9O3b182bNjA3LlzrWVpTAhV2ML0JseFqlro/QnbZAmQle9Zg9ksrm5UKtq1axe9evVi06ZNzJ8/n4EDBzodkjFRLZAxzPUi0inkkQRBUWm32LrRJXe5XMTFxbFw4UL69evndDjGRL1yu+QiEqeqLqAT8JmI/Ahk49nfR1W1cy3FGDBf4Y0mEt2l3X777TeaNGlCy5YtWbduHTG2Z5ExtaKiMcw1QGdgUC3FUmNZeZ77yBtF8Sz51q1bOeeccxg0aBBPPvmkJUtjalFFCVMAVPXHWoqlxg56W5gN1LdwPbq2p9i0aRO9e/cmOzubK6+80ulwjKlzKkqYTUXktvJeVNXHQhBPjfjGMNOisLTb999/T58+fSgoKGDFihWccsopTodkTJ1TUcKMBVLwtjQjQWauiyTySNRciE2EhBSnQwqKgoICzj//fNxuNytXruTEE090OiRj6qSKEuavqjqlJhcXkQHAE3iS7wuq+mCJ128D/gS4gN3A2DKKFQcsK6+ARv5bU0RJkdz4+HimTZvGUUcdxXHHHed0OMbUWRXNGNQo23iLDT8DnAecAIwUkRNKnLYO6KKqHYG3gIdr8plZua7DEz5RsGh97dq1TJs2DYBzzjnHkqUxDqsoYfap4bW7AptVdYuq5gNvABf5n6CqK1Q1x/t0NdCyJh94MNdF46JKRZE9fvnpp5/Sp08f7r//fnJycip/gzEm5MpNmKqaUcNrt6B4Gbgd3mPlGQcsKusFERkvImtFZO3u3bvLvUBWnisqdov88MMPOffcczniiCNYuXIl9erVczokYwzVq1YUdCJyOdAF+GdZr6vqVFXtoqpdmjZtWu51MnMLDlcqitAZ8uXLlzNgwABatmzJqlWrOProo50OyRjjFUgB4er6BfD/196Sw0U8iohIX+CvQE9VzavJB2bluSJ+0fqXX35Ju3btWLp0Kc2ahW0VPWPqpFC2MD8D2otIWxFJwLMv0Dz/E7z3qD8PDPLuHVQjmbn+XfLIWrSememJ+9Zbb2XNmjWWLI0JQyFLmN770G8ElgDfA2+q6rciMkVEfLdb/hPPWs/ZIrJeROaVc7mAZPlP+kRQl/ztt9+mXbt2rF+/HoDk5GSHIzLGlCWUXXJUdSGwsMSxu/0eB7V448HcyOuSz5o1i1GjRnH66afTtm1bp8MxxlQgLCZ9gqXUwvUwN2PGDC677DK6devGe++9R4MGDZwOyRhTgahJmAWFbnIL3Id3jAzzLvmyZcu44oor6NWrF4sWLSI1NdXpkIwxlYiahJmV6yKWQu9+PgLJ6U6HVKEePXpw77338u6771K/fn2nwzHGBCB6Emaei4Z4K60np0NMrLMBlWP69On8/vvvxMfH89e//tUmeIyJIFGTMA/mFoR9d/yRRx7hiiuu4JFHHnE6FGNMNURNwvQsKQrfGfL77ruP22+/nUsvvZT77rvP6XCMMdUQNQkzXBetqyr33HMPf/vb3xg9ejQzZswgPj7e6bCMMdUQNQkzK88VlveRZ2VlMWvWLMaOHctLL71EXFxIl74aY0Ioav71Zub6r8F0PmGqKoWFhaSmpvLxxx+Tnp5uG5YZE+Gi5l9wZrHCG84uWne73dx0002MHj2awsJCGjdubMnSmCgQNf+KM3NdYTFL7na7ueaaa3jmmWc4+uijLVEaE0Wi5l9zVq6LRjjbwiwsLGTs2LG88MIL/O1vf+Ohhx5ComRfIWNMlI1hHt6ewpmEef311/PKK68wZcoU7rrrLkdiMMaETtQkzKw8F+kOd8lHjx5Nhw4dmDBhgiOfb4wJrahJmJmHChzpkufl5bF48WIuuugizj77bM4+++xa+2wTnQoKCtixYwe5ublOhxKVkpKSaNmyZbXWQ0dNwizMPUiCFFIYV4/Y+Nq5P/vQoUNcfPHFLF68mK+//poTTzyxVj7XRLcdO3aQmppKmzZtbAw8yFSVvXv3smPHjmrVn42aSZ+4XM8ml+7k2mld5uTkMGjQIBYvXszUqVMtWZqgyc3NpXHjxpYsQ0BEaNy4cbVb71HTwkzI8yRMqYXxy6ysLC644AI+/PBDXn75Za644oqQf6apWyxZhk5N/myjooWpqiQX7ANAUkKfMBcvXsxHH33EjBkzLFmaqJSSklL0eOHChXTo0IGffvqJyZMnU69ePX7//fcyzxWRYpOejzzyCJMnTy7zM+bOncuUKVMAGDNmDG+99VaV41y5ciUNGjTg1FNP5dRTTy26Xn5+Pj169MDlclX5mhWJioSZ53KTpp4Jn9gQtjBVFYBhw4bxww8/MHLkyJB9ljHhYNmyZdx8880sWrSI1q1bA9CkSRMeffTRMs9PTExkzpw57Nmzp9JrP/zww1x//fU1jrF79+6sX7+e9evXc/fdni3DEhIS6NOnD7Nmzarx9f1FRcLMzHWFfC+fvXv30rNnTz766CMAjj322JB8jjHh4oMPPuDqq6/m3Xff5Zhjjik6PnbsWGbNmkVGRkap98TFxTF+/Hgef/zxCq+9ceNGEhMTadKkdAPnrrvuYsyYMRQWFtYo/sGDB/Paa6/V6BolRcUYZmZuweH7yEPQwvz999/p27cvGzduLNo/3Jja0ObOBSG57rYHz6/w9by8PAYPHszKlSs57rjjir2WkpLC2LFjeeKJJ/j73/9e6r033HADHTt25C9/+Uu51//444/p3LlzqeO33347mZmZvPTSS4gIt956KytWrCh13ogRI7jzzjsB+PTTTznllFNo3rw5jzzySNEE7EknncRnn31W4fesqqhImFl5oWth/vrrr/Tp04dt27bx7rvv0rdvUHcGNiYsxcfH061bN6ZNm8YTTzxR6vWbb76ZU089lf/7v/8r9VpaWhpXXHEFTz75ZLlbsPz66680bdq02LF7772XP/7xj0ydOrXoWGUt1c6dO/PTTz+RkpLCwoULGTx4MJs2bQIgNjaWhIQEMjMzg7bJYFQkzGKFN4JY2m337t307NmTnTt3smjRInr27Bm0axsTiMpagqESExPDm2++SZ8+fbj//vuZNGlSsdcbNmzIZZddxjPPPFPm+//85z/TuXNnrrrqqjJfT05O5sCBA8WOnX766Xz++edkZGTQqJGnCHhlLcy0tLSiYwMHDuT6669nz549RV39vLw8kpKSAv/ilYiahHlkCLrk6enp9OjRg6uuuoqzzjoraNc1JhLUq1ePBQsW0L17d5o1a8a4ceOKvX7bbbdx+umnlzkT3ahRIy655BKmTZvG2LFjS71+/PHHM2PGjGLHBgwYQP/+/Tn//PN57733SE1NrbSFuWvXLpo1a4aIsGbNGtxuN40be3qZe/fupUmTJkHd4SAqEmZWnv/2FDXvkm/ZsoWkpCSaN2/OCy+8UOPrGROpGjVqxOLFi+nRo0epLnSTJk0YMmRIuUltwoQJPP3002W+1qNHDyZMmICqFlsXOXz4cDIzMxk0aBALFy6sdFfVt956i3//+9/ExcWRnJzMG2+8UXS9FStWcP75QW6hq2pE/Zx22mla0osfbdGDdzdTvSdNNWdfqderYuPGjdqiRQs966yz1O121+haxlTHd99953QIteLmm2/W999/P2TXHzJkiG7YsKHM18r6MwbWaiX5JyqWFeXk5JAqhyiUWEhqUO3rfP/99/To0YP8/HyeffZZu9vCmBCaNGkSOTk5Ibl2fn4+gwcPpkOHDkG9blQkzMJszyLZ3Ph0qGaS+/rrr4smdVauXEnHjh2DFp8xprRmzZoxaNCgkFw7ISEhJHfhRUXCJHsvAPkJ6dW+xC233EJCQgKrVq3ihBNOCFZkxpgoEhWTPjE5nhamK6n6+5G/8cYbZGZmFrujwRhj/EVFCzM2z1farWoJ85NPPuGKK66goKCAI444wpKlMaZCUdHCTMjzVCqqyhrMDz74gIEDB9K8eXMyMjJo1qxZiKIzxkSLqGhhJuV7EmZM/aaVnOmxbNkyBgwYQKtWrVi1apUlS2NMQKIiYSYX7AcgLoBamEuWLOGCCy7g2GOPZeXKlRx11FGhDs+YiBMp9TBfe+01OnbsyMknn0y3bt348ssvAauHWaEUtydhJjY4otJz09PTOeOMM1ixYgVHHFH5+cbUZeFeD7Nt27asWrWKr7/+mrvuuovx48cDoauHGfFjmKpKA/cBiKk4YW7cuJEOHTrQtWtXli9fbovSTWSYXP0bMSq+7oFKT/HVw1y4cGGpepgvv/wyd9xxR1GRDB//epj33XdfudeurB7m9u3bmTZtGrGxsRXG2K1bt6LHZ5xxBjt27Ch6PnjwYCZOnMioUaMq/a6BivgWZnZ+YdF95LHldMnfeOMNTjjhBGbOnAnYfinGVMZXD3Pu3LkV1sMsyw033MBrr71WqhqRv4rqYe7evZuXXnqJ2NhYbr311qLtJ/x/HnzwwVLvnTZtGuedd17Rc6uHWYasSkq7vfrqq1x11VWcffbZXHjhhbUcnTE1FEBLMBQipR6mz4oVK5g2bVrRjggQmnqYEd/CzDyU51epqHj3YNq0aYwZM4ZevXqxcOHCoP2hGRPtfPUw16xZw/3331/q9UDqYU6bNo3s7OwyX09OTi611a1/PUyfQFqYX331FX/605945513ikq7+Vg9zBJyDu4lVpQsSSEl9nDduw0bNjB+/Hj69+/PnDlzKi0TZYwpLhLqYf78888MHTqU6dOnlyq0EYp6mBHfwsw74FnekBlbfHD8D3/4A/Pnz2fu3LmWLI2pJl89zH/84x/Mmzev2Gu+eph5eXllvnfChAnlzpb36NGDdevWFe3E6jN8+HCuvvpqBg0axKFDvvQiCgAAC9FJREFUhyqNb8qUKezdu5frr7+eU089lS5duhS9ZvUwy6iH+fGyear3pOnmB85UVdVHH31Uly1bVlGZPGPCmtXDDI6Iq4cpIgNEZIOIbBaRO8t4PVFEZnlf/5+ItKnqZxRm+Uq7NeTee+9lwoQJQd9a0xgTfJFYDzNkY5giEgs8A/QDdgCficg8Vf3O77RxwD5VPVZERgAPAZdW6YNy9qCqPP7+dqYtWcTo0aOLzbIZY8KT1cMsriuwWVW3qGo+8AZwUYlzLgJe8T5+C+gjVV0kmb2HO5fmMW3JN4wbN65o/ZYxkUxLjO2Z4KnJn20oE2YLYLvf8x3eY2Weo6ou4ABQpV3MYg9l8Fu2MrRPF6ZOnWrJ0kS8pKQk9u7da0kzBFSVvXv3VnupUUQsKxKR8cB4gFatWhV7Lbn92Ywfm0/qaZcSExPxk/7G0LJlS3bs2MHu3budDiUqJSUl0bJly2q9N5QJ8xfgaL/nLb3Hyjpnh4jEAQ2AvSUvpKpTgakAXbp0Kfa/3U79r4T+VwYxbGOcFR8fT9u2bZ0Ow5QhlE2yz4D2ItJWRBKAEcC8EufMA3zZbhiwXK0fYowJUyFrYaqqS0RuBJYAscCLqvqtiEzBs95pHjANmC4im4EMPEnVGGPCUkjHMFV1IbCwxLG7/R7nAsNDGYMxxgSLRFoPWER2Az+VONwEqLxiaeSx7xU5ovE7Qd36Xq1VtcJ9biIuYZZFRNaqapfKz4ws9r0iRzR+J7DvVZKtwzHGmABZwjTGmABFS8KM1pvH7XtFjmj8TmDfq5ioGMM0xpjaEC0tTGOMCbmISpi1UV/TCQF8r9tE5DsR+UpElolIayfirIrKvpPfeReLiIpIRMzEBvK9ROQS73+vb0VkZm3HWB0B/B1sJSIrRGSd9+/hQCfirAoReVFEfheRb8p5XUTkSe93/kpESm9jWVJlFYbD5QfP3UI/Au2ABOBL4IQS51wPPOd9PAKY5XTcQfpe5wD1vI+vC/fvFch38p6XCnwArAa6OB13kP5btQfWAene50c4HXeQvtdU4Drv4xOAbU7HHcD36gF0Br4p5/WBwCJAgDOA/1V2zUhqYdZOfc3aV+n3UtUVquorTb0aTyGTcBbIfyuAe/EUjc4t47VwFMj3uhp4RlX3Aajq77UcY3UE8r0USPM+bgDsrMX4qkVVP8Bzy3V5LgJeVY/VQEMROaqia0ZSwqyV+poOCOR7+RuH5/+K4azS7+Tt/hytqgtqM7AaCuS/VQegg4h8LCKrRWRArUVXfYF8r8nA5SKyA8/tzjfVTmghVdV/e5FRD9N4iMjlQBegp9Ox1ISIxACPAWMcDiUU4vB0y3vh6Ql8ICInq+p+R6OquZHAy6r6qIiciadozkmq6nY6sNoUSS3MqtTXpKL6mmEmkO+FiPQF/goMUtWy9zUNH5V9p1TgJGCliGzDM340LwImfgL5b7UDmKeqBaq6FdiIJ4GGs0C+1zjgTQBV/RRIwnM/diQL6N+ev0hKmNFaX7PS7yUinYDn8STLSBgTq/A7qeoBVW2iqm1UtQ2ecdlBqrrWmXADFsjfwbl4WpeISBM8XfQttRlkNQTyvX4G+gCIyPF4Emakl4SfB1zhnS0/Azigqr9W+A6nZ7KqOOv1/+2de4xdVRWHv19rW0pLMMWKQKPDoy1iKDWKIaJGBYtijYkMjViesSklUK1aJAoawGiaFI3RCoWM0mIGlBrro0JrIS0VKJSxzINUSrHUR0KEEGKEFAN1+cdat3N6OXfm3OlMbgfWl5zcc/bdj7X3OXfddR77d87F/7H/ClwbaTfiPzbwnbgGeBrYBpzQapuHqV/3Af8CumP5XattPtg+1eXdzCi4S15xXwm/3LAD6AM+32qbh6lfpwAP4XfQu4E5rba5Qp/uAp4FXsUj/y8Ci4BFhX31k+hzX5VjMGf6JEmSVGQ0nZInSZK0lHSYSZIkFUmHmSRJUpF0mEmSJBVJh5kkSVKRdJiHIJL2SeouLG0D5G1rpMbSZJubQ62mJ6b1zRxCHYskXRzrl0o6tvBdh6RThtnOxyTNrlBmiaTDh9DWDyV9JNavClUbi+crG5WZG4o+PaFYdHmz7Q5i040xiQFJHw5FpG5Jx0n61SBl9+8DSd+s0NZUSeuHx/I3CK1+ViqX0ufHXmoibxsN1FiabHMz8RwasJCDfNaTEXq2ss7Oy4CNFcrsAd7WZDtHAY8Utt8bY92wLmAcLkoxLbYnADNH8DhZCVw4kscYcDtw5kj1YbQtGWGOEiKS/JOk7bF8sCTPeyRti4ijV9L0SL+wkH6rpLGDNLcFOCnKnhURU1/oC06I9GXq1+i8KdKul7RUUjs+570z2pwYkeH7IwpdXrD5UkkrhmjnVgpiCZJukdQVUdcNkfYl4Fhgk6RNkTZH0tYYxzWSJpfUfR6wP7oys8fNbM8g9hyBzyV/Icr818x2RpurJK0M+56SNDfSx0paHtFybzEilXRNjHuPpGWFetolLQDmAd+R1Fk804g6b5L0RNS5ONJr+2AZMDHGuTOi1iWFdr8r6cux+Rtg/iD9fvPQao+dy+sXYB/9s3rWRtrhwGGxPh3oivU2IsIEfgzMj/XxwETg3cDvgXGRfjNwcUmbm+mP3K4GfonPnPoHMCPS7wCW4NHXTvpfcfLW+LweWFpfX3EbmIpLidXS7wU+NEQ7lwDfK3w3JT7HRr5Zsb2HiArx+c9bgEmxfQ3w7ZJ2VgOfKUnfX1eDfdcBPIfPMpkPjIn0VbgDHhP7758xvguB6yLPBKALOB74FPAw/TqoUwr1tJesF4+DK3B5w7fUlS2O3UsFm9uA7bE+Bp/5clRsHwf0tfo3cagsqVZ0aLLXzOqvzY0DVsQ1u334HOV6tgLXSpoG/NrMdkk6C3gf8JhcGnQi/oMuo1PSXtwpLAZmAs+Y2VPx/WrgSmAFrmH5U0nrgHVVO2Zmz0vaLZ+7uws4GZ9yd2WTdo4HJgPFcZonaSEe5R2DT+frrSt7RqQ/FO2Mx8etnmMYwlxpM1sg6VTgbGAp8An6VZnuNlf32SVpN973OcCsiMrBBWOmR/nbLXRQzWwgXcd6zsaFtF+rUtbM9kh6Qa5ZcDTwuJnVRGuewyP0hJR3G018BZ9PfhoeBbxOdNfM7pT0KPBp4J44vROw2sy+UaGN+VYQwJA0pSyTmb0m6QO4GEM7cBXw8Sb68gv8dPJJPII2ufeqbCfwZ2A5HlV/TtLxuIM63cxelLQKj+DqEX7d84JB2tjboPyBlUkbcCfTZWYLAMysD+iT9HPgGfodZv08ZAt7FpvZhrp6zxms7WGmA7fzHcDPCumH4WORkHfJRxNHAs9GhHIRftp5AJJOAHab2Y+A3wKzgPuBdklvjzxTVP2dQDuBNkknxfZFwANxze9IM7sHd+SnlZT9D35Nr4y1uNr1BbjzpFk7zc8XvwWcIelkXA38ZeDfko7GT2nLbHkEOLPWJ0mTJJVF638hruMOhJmdY2azI7KcLOmjha9nA38rbJ8vaYykE/HXQewENgBXSBoX9syQNAnYCFymuLvf6M+rARuBy+USh43KvlprM1gLfBI4PWyqMQM46Kcw3iikwxw93AxcIqkHP5V7uSTPPOAJSd243uQdZrYDuA74o6Re/Mc0oAx/DTN7Bb8TvUZSH/A//M7sEcC6qO9B4KslxVcBK2s3ferqfRF3SO8ys22R1rSdZrYX+D5wtZn14O/SeRK4Ez/Nr3EbsF7SJjN7Ho+k7op2tuLjWc8fCJk28JtHcrXxaUCvpI6SMgK+Ln/sqRu4gQNFkv+Oq2jdiyvmvIJHdjuA7XHT5lb82uN6XH6sK+paOtBY1NERbfXG8fKFkjy3xfedAOavptiEXzbYV8j3sRiLhHwveZI0RNKDwFwbBrX0uESwzswGfFayVchV8LcD55vZrkL6FuCz8Sf3picjzCRpzNeAd7baiJFG/jD708D9dc5yKvCDdJb9ZISZJElSkYwwkyRJKpIOM0mSpCLpMJMkSSqSDjNJkqQi6TCTJEkqkg4zSZKkIv8HaDYYeP8WFQ0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "knn2 = KNeighborsClassifier(n_neighbors=2)\n",
    "knn2.fit(X_train, y_train)\n",
    "knn2_probs = knn2.predict_proba(X_test)[:,1]\n",
    "knn2_fpr, knn2_tpr, knn2_thresholds = roc_curve(y_test, knn2_probs)\n",
    "# Plot knn ROC curves\n",
    "plt.figure(figsize=(5,5))\n",
    "plt.plot(fpr, tpr, linewidth=2, label=\"KNN (k=5)\")\n",
    "plot_roc(knn2_fpr, knn2_tpr, label=\"KNN (k=2)\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plot, knn with $k=5$ clearly performs better than knn with $k=2$. A measure that can summarize the information contained in the ROC curve is the *Area under ROC* (AUC). A perfect classifier would have an AUC equal to 1, while a completely random classifier has AUC=0.5.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9421918908069049"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "knn_auc = roc_auc_score(y_test, knn_probs)\n",
    "knn_auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.925531914893617"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn2_auc = roc_auc_score(y_test, knn2_probs)\n",
    "knn2_auc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, as the graph suggested, knn with k=5 (default option in scikit-learn) has a better AUC than knn with k=2 for this dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's suppose that we are interested, in our application, to maximize the TPR while keeping the FPR as low as possible. This can happen, for instance, when we want to have as little false positives as possible, while having a good level of true positives (e.g., when recognizing ham/spam e-mail, we want to mark as little ham e-mail as possible as spam, but at the same time try to detect as much spam as we can). After some search in the ROC curve of our original knn classifier (with $k$=5), we observe that the value we are searching for leads to a tpr of about 0.88. As a consequence, we can detect the thresholds needed to classify our examples as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_threshold = thresholds[np.max(np.where(tpr <= 0.88))]\n",
    "new_threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUwAAAFNCAYAAABrMlb6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3hUZfbA8e9JQhIggDQLBAQUVJCmwKIovYkoFlCQIguuZcW+NtaCuK71Z8GOIs2OooKAsFQrIgrYkI4CIoSeAOnn98e9CUNImSQzuTOT83meeZi5c+fecyfJ4W33fUVVMcYYU7QorwMwxphwYQnTGGP8ZAnTGGP8ZAnTGGP8ZAnTGGP8ZAnTGGP8ZAnTGGP8ZAnTBJWIbBaRwyKSIiJ/icgkEUnIs8+5IrJQRJJFZL+IzBSRpnn2qSoiz4rIH+6xNrivaxVwXhGRm0XkZxE5KCJbRWSaiDQP5vWayGYJ05SFi1Q1AWgFtAbuzXlDRM4B5gGfAHWAhsAq4CsRaeTuEwssAJoBvYGqwDnAbqBdAed8DrgFuBmoATQBPgYuLG7wIhJT3M+YCKWq9rBH0B7AZqC7z+sngFk+r78AXsrnc3OAKe7za4AdQIKf52wMZAHtCtlnMXCNz+vhwJc+rxW4EVgHbAJeBp7Kc4xPgNvd53WAD4Ekd/+bvf7u7RH4h5UwTZkRkUTgAmC9+7oScC4wLZ/d3wd6uM+7A5+paoqfp+oGbFXVZaWLmEuAvwFNgXeAK0VEAESkOtATeFdEooCZOCXjuu75bxWRXqU8vwkxljBNWfhYRJKBLcBO4EF3ew2c38Ht+XxmO5DTPlmzgH0KUtz9C/Koqu5R1cM4JWEFznff6w98o6p/Am2B2qo6VlXTVXUj8BowMAAxmBBiCdOUhUtUtQrQGTidI4lwL5ANnJTPZ04CdrnPdxewT0GKu39BtuQ8Uafe/S4wyN10FfCW+/xkoI6I7Mt5AKOBEwIQgwkhljBNmVHVJcAk4Cn39UHgG2BAPrtfgdPRAzAf6CUilf081QIgUUTaFLLPQaCSz+sT8ws5z+t3gP4icjJOVf1Dd/sWYJOqHufzqKKqffyM14QJS5imrD0L9BCRlu7re4Cr3SFAVUSkuoj8B6cX/CF3n6k4SelDETldRKJEpKaIjBaRY5KSqq4DXgLeEZHOIhIrIvEiMlBE7nF3WwlcJiKVRORUYGRRgavqCpxS7+vAXFXd5761DEgWkbtFpKKIRIvImSLStiRfkAldljBNmVLVJGAK8ID7+kugF3AZTrvj7zhDj85zEx+qmobT8fMb8D/gAE6SqgV8W8CpbgZeAF4E9gEbgEtxOmcAngHScXrfJ3Okel2Ut91Y3va5piygL86wqU0cSarV/DymCRPiNM0YY4wpipUwjTHGT5YwjTHGT5YwjTHGT5YwjTHGT5YwjTHGT2E3C0utWrW0QYMGXodhjIkw33///S5VrV3YPkFLmCLyBs7YtJ2qemY+7wvOFFx9gEPAcFX9oajjNmjQgOXLlwc6XGNMOScivxe1TzCr5JNw5i4syAU403A1Bq7FmT7LGGNCVtASpqp+DuwpZJd+OPMdqqouBY4TkUBMmGCMMUHhZRtmXXxmgwG2utsCMS2XMSaCpWVmcSgti5S0TA6mZ3IwLYuDaZkcSs8kxX3ubD/y3sH0TA4cTmff7t20b9aI+/o2LfpEeYRFp4+IXItTbad+/foeR2OMKQ5VJTUjm5TchJbJoXQ32aVlHkl8aZkcTHeTm28i9E187vOMrJLd0p2+YyO7Zj5J5Qcn4swLXTxeJsxtQD2f14nutmOo6nhgPECbNm3s5ndjgig7WzmU4ZO43ISWk+wOpmX5PPdNcgWX7LID/FcbEyVUjoshIS6GynHRVIo98rxybAyV42KoFBdNQs7z2CgS4itQOa4tu4e256zmZ5TsvIG9jGKZAYwSkXdx5hbcr6pWHTemmDKzsvOUzJznKXmqqIfSnOf5J7sjzw+lZwU8xriYKBLcJJaT0JyEd3SyO/I8hsqx0bn7VT7qc9HExUT7fe709HSuuuoqLrnkEvoOGQKnHV/i6wjmsKJ3cGbYriUiW3GWJagAoKqvALNxhhStxxlW9PdgxWJMKEnLzCqwNOZbXT2Unrd0l091NS2TtMzsgMdYKSdZ+SYt93lCXIyb2KLdklzexHf05yrFRlMh2pt7ZFJTU+nfvz+zZs2iU6dOpT5e0BKmqg4q4v2cVfmMCVm+7W95E9UxyS7daY/Lm/gO5kmCJW1/K0iUcFTpy0lYeV8fnewqx0X7JD6f/eJiqFQhmqgoCWiMXjh06BCXXnop8+bN45VXXuG6664r9THDotPHGH9lZysH049OVkcnuKOTXU4V9ZjSXRDb3ypEi09Sy7/97ZgkF+eb8I5OdvEVonAXszSujIwM+vbty+LFi3njjTf4+98DU4G1hGk8lZmV7VQ703Pa2PIku/SjOx+OlOSO7jXNSXxl0f6W4FMN9S3JFdX+lnOM4rS/mZKpUKECnTp1YsSIEQwZMiRgx7WEafymqqRlZnMoPZ9SWlrhHQk5idC3pJeSlkl6ENvfckpj/rS/5dcRkfO5GI/a30zx7du3j61bt3LmmWfy4IMPFv2BYrKEGcFUlcMZR6qYvknrmI6EfIaCHEw7NvFlBrh+6k/7m281tHI+VdJIbH8zxbdnzx569erFtm3bWL9+PZUqVSr6Q8VkCTPMZWZls3n3Qdb8lcKaHcms+esAa3ekkJScxsH0TAK9ZFPe9rf8nxfd/pazr7W/mUDYtWsXPXr04Ndff+XDDz8MSrIES5hhQ1XZuvcwa3cks2ZHMmv/Sua3v5LZmHSQ9KyCq7U57W9HlczyaX/zTXZHVUmt/c2EuB07dtCtWzc2bNjAzJkz6dmzZ9DOZQkzBO1KSWPNX8ms+Sv5qAR5sIAOjXo1KnLaCVVockIVTjvR+bfOcRWt/c2UCw8//DCbNm1i1qxZdO3aNajnCrtldtu0aaORMh9mcmoGa3ekOEnRJ0HuPpie7/61EuI4/cScxJhAEzdJVo6z//dM+ZWamsqvv/7KWWedVarjiMj3qtqmsH3sL60MpGZksSEpJzGm5LYzbtt3ON/9q8TF0OREp7SYU3JsckICNRPiyjhyY0LT5s2bueOOO3j99depXr16qZOlvyxhBtmUbzbz8Ke/5nt3R2xMFI2PT+C0nKq0myBPqhZvHSHGFGD9+vV07dqVlJQU/vjjD6pXr15m57aEGURfb9jFmBm/kK1wSu3Kue2LOQny5JqVibYhMMb4bc2aNXTt2pW0tDQWLlxIy5Yty/T8ljCD5K/9qdz8zgqyFUZ1OZV/9TrN65CMCWu//vorXbt2RVVZvHgxZ555zFJhQWddqEGQkZXNqLd/YFdKOh1OrcltPZp4HZIxYS8hIYFTTjmFJUuWeJIswUqYQfHYnN9Y/vteTqwaz7iBra3abUwprF+/noYNG1K/fn2+/PJLT9v3rYQZYLN/2s6ELzcREyW8OPgs69k2phSWLl3K2WefzQMPPADgeWeoJcwA2pCUwp3TVgFw34VncPbJZdd7Z0yk+fLLL+nRowe1a9cOyFyWgWAJM0AOpWdyw5vfczA9i4ta1uHqcxt4HZIxYWvx4sX06tWLunXrsmTJkpBZ/NASZgCoKvdO/4m1O1I49fgEHrusuedVB2PCVXJyMv3796dBgwYsXryYunXreh1SLuv0CYA3l/7OJyv/pFJsNK8MOctuVTSmFKpUqcL06dM544wzqF27ttfhHMX+sktp5ZZ9jP30VwAeu7wFpx5fxeOIjAlPn3zyCbt27WLkyJF07NjR63DyZVXyUthzMJ1/vvk9GVnK8HMbcHHLOl6HZExYmjZtGv3792fChAlkZmZ6HU6BLGGWUFa2csu7K/hzfyqt6x/H6D4lWxjemPLu7bffZuDAgbRv357PPvuMmJjQrfhawiyhcQvW8cW6XdSoHMtLg88iNsa+SmOKa9KkSQwZMoROnToxZ84cqlat6nVIhbK/8hJYvGYn4xauQwTGDWzNSdUqeh2SMWFpx44ddO/enU8//ZSEhASvwymSTSBcDCu37GPSV5uY9dN2MrKUO3o04aZujT2JxZhwlpSUlNsDnpmZGRLVcJtA2B+H9sAfS2HTEkhLhrgq0LAT1G8PlWqQkZXNnJ//YuJXm1jxxz7AWelwULv63NjlVI+DNyb8PP300zz88MN88803nH766SGRLP0VPpEGw+4N8PXzkJUBlWpC1WqQmQq/zeLwr5/xQVw/Xlil7DiQBkDV+BgGtavP0HNOJrF6cFalMyaSPfroo4wePZoBAwZwyimneB1OsZXfhHloj5MsYypCwgm5m5NSo1ixLYZtO3ZQIftFUjOvpPHxJzK8QwMubV2XSrHl9yszpqRUlbFjxzJmzBgGDx7MpEmTwqpkmSP8Ig6UP5Y6JUs3WW5ISuGHP/b6rLNTkbOrZfN2G2jauaPd6mhMKbzzzjuMGTOG4cOH8/rrrxMdHZ7LNZffhLlpiVMNB9btTGbWT9sBiI2OommdqrRKPI7jKmRB2gqQK72M1Jiw179/f/bv3891111HVFT4Ds4J38hLKy0ZYuIB2LAzBYAWdasx8ryGdG5yPMdVioWYOEhN9jJKY8JWdnY2jz32GElJScTGxnLDDTeEdbKE8pww46o4HTzAn/udf5vXPY64GJ+qQmYaxNu94cYUV3Z2Ntdffz333nsvb731ltfhBEz5TZgNO8Gh3aSkZXIgNYPY6ChqJsQevc+hXc5+xhi/ZWVlMWLECF577TVGjx7NLbfc4nVIAVN+E2b99hBdgZ1JOwE4sVo8Ub4dO6kHIDoW6v3NowCNCT+ZmZkMGzaMyZMn89BDD/Gf//wnojpMy2/CrFQDzr2J3fsOcBK7qZcgoNmQcRj2b4HMw3DuTc5+xhi/7N27l++++45HH32UBx54IKKSJZTnXnKAmqfwdPZAKmYv5z9Vd8OBP502yzMuckqWliyN8Ut6ejpRUVHUrl2bFStWULlyZa9DCopynTAPpmWybIcCZ3PcZT3BBqUbU2ypqalcfvnl1KhRgylTpkRssoTyXCXHmUwjK1tpelJVu4PHmBI4dOgQF198MXPmzOH888+PuCp4XuU6SyzfvBfAlsM1pgRSUlK46KKLWLJkCW+88QbDhw/3OqSgK98J8/c9ALRtYG2VxhTXFVdcweeff87UqVMZPHiw1+GUiXJbJc/K1tzp2to0sBKmMcV199138+6775abZAnluIT5218HSEnLJLF6RU6oGu91OMaEhT179jB37lwGDRpEp07l76aOclvC/P53p/3SquPG+CcpKYkuXbowYsQItm7d6nU4nii3JUzr8DHGf3/99RfdunVj06ZNzJgxg8TERK9D8kRQS5gi0ltE1ojIehG5J5/364vIIhFZISI/ikifYMbja/lmp8PH2i+NKdy2bdvo3Lkzv//+O7Nnz6ZHjx5eh+SZoCVMEYkGXgQuAJoCg0SkaZ7d7gPeV9XWwEDgpWDF4+vPfYf5c38qVeJjaHK8zUZkTGHmzZvHn3/+yWeffUbnzp29DsdTwSxhtgPWq+pGVU0H3gX65dlHgZyFiKsBfwYxnlzLfz9SHY+KiuyBtsaUVFZWFgB///vfWbt2Leedd57HEXkvmAmzLrDF5/VWd5uvMcAQEdkKzAZuyu9AInKtiCwXkeVJSUmlDuz7nOq4tV8ak69169bRvHlzvv76awBOPPFEjyMKDV73kg8CJqlqItAHmCoix8SkquNVtY2qtslZy7g0vsvt8LEecmPyWr16NZ06dSIpKSmi7wsviWAmzG1APZ/Xie42XyOB9wFU9RsgHqgVxJhIScvkt78OEBMltKp3XDBPZUzY+fnnn+ncuTPZ2dksXryYli1beh1SSAlmwvwOaCwiDUUkFqdTZ0aeff4AugGIyBk4CbP0de5CrPhjL9kKzepWo2JseK5cZ0wwbNiwgc6dOxMTE8OSJUto1qyZ1yGFnEITpoi0FZHnROQHEdkuIhtFZIaIXCcihXYvq2omMAqYC6zG6Q3/RUTGisjF7m53AP8QkVXAO8BwVdXSX1bBcsZfWvulMUc7+eSTGTRoEEuWLOG0007zOpyQVODAdRGZBewCPgH+D9iJUwJsAnQBZonIE6r6aUHHUNXZOJ05vtse8Hn+K9ChNBdQXDkTbljCNMbx3XffkZiYyEknncTzzz/vdTghrbA7fUao6o4821KBZe7jcRE5PmiRBUFmVnbuhBtn24B1Y/jiiy/o06cPXbt25ZNPPvE6nJBXYJU8J1mKyA0iUq2AfXYGK7Bg2LL3MIfSs6hTLZ7jq9iEG6Z8W7hwIb179yYxMZGXX37Z63DCgj+dPicDP4jI2yLSPdgBBdOBwxkA1EyI8zgSY7w1d+5cLrzwQho1asTixYupU6eO1yGFhSITpqreAzQG3gKuF5F1bsdNgyDHFnDJqZkAVIkvt3OOGEN2djajR4/m9NNPZ9GiRZxwwglehxQ2/MocqpotIpuBzUBz4CTgExGZrar3Bi+8wEpJc0qYCXGWME35FRUVxaxZs4iNjaVGDbt5oziKLGGKyI0isgx4DvgeaKGq/wBaA1cGOb6AOuCWMBOshGnKoffff5+BAweSmZnJiSeeaMmyBPxpw6wDDFLV7qr6jqqmgVPqBC4u/KOhJcVNmFXjK3gciTFl680332TQoEH8+eefpKameh1O2PInYdZV1Q2+G0RkEoCq/hyMoIIlpw3TquSmPHnjjTcYNmwYnTt3Zs6cOSQkJHgdUtjyJ2G28H3hTo7RNjjhBFdOG6Z1+pjyYsKECYwcOZKePXvy6aef2mQapVRgwhSRu0VkL9BCRPa4j704d//MLuhzoSzZ2jBNOdOsWTMGDRrExx9/TMWKFb0OJ+wVVsJ8AqgNPOP+Wxuopao1VPXOsggu0JLTcoYVWRumiWzfffcdAO3bt+ftt98mPt5u1AiEwhLmqe4EGlOBZjkPEWkhIi0K+VzIyh2HaW2YJoI98sgjtGvXjlmzZnkdSsQpLHPcgzNf5Yv5vKdAx6BEFEQpqdaGaSKXqjJmzBjGjh3L0KFD6dWrl9chRZwCM4eqjnT/Pb/swgmulDRrwzSRSVW59957efzxxxkxYgTjx48nOtrmew00fwau/yAid4rIyWURUDAduTXS2jBNZFm6dCmPP/44119/Pa+99polyyDxp6g1AOeOnhkicgh4D2cy4DJZ4TGQUmwcpolQ55xzDkuWLOH8889HxFZCDRZ/Jt/YoKr/VdWWwAjgLJylJcJKdraSkm4J00SO7Oxsbr75ZhYvXgxAx44dLVkGmV+ZQ0QSgStwSpoxwL+DGVQwHEzPRBUqx0YTbWuRmzCXlZXFyJEjmTx5MrVq1aJz585eh1QuFJkwReRrIAGYBgxR1XVBjyoIbNC6iRSZmZlcffXVvP3224wdO5b777/f65DKDX+yxz9U9ZegRxJkKTZo3USAjIwMrrrqKj744AMee+wx7r77bq9DKlcKWwRtkKq+A3QTkW5531fVcUGNLMCSU20uTBP+oqKiiI+P5+mnn+a2227zOpxyp7DskbNKWO183gvqUrjBYLOtm3CWmprKnj17qFOnDlOmTLHOHY8UNnD9JffpLFVd6vueiLQPalRBYAnThKtDhw7Rr18//vjjD3788Ufi4mxNKq/4M73bS/lsy+92yZCW24YZZ22YJnykpKTQp08fFi5cyOjRoy1ZeqywNsx2wDlAbRG52eetqkDYZZ0U6yU3YebAgQP06dOHpUuX5s6YbrxVWPaoDNRy9/Ftx0zGufsnrCTbxBsmzNx22218++23vPfee1x++eVeh2MovA1zEbBIRCaq6sYyjCkocubCtF5yEy4ef/xxrrzySnr27Ol1KMZV2Izr/+c+/T8RmZ73UUbxBUyyLYBmwsDOnTu5/fbbSU9Pp1atWpYsQ0xhxa333H9fKItAgs3aME2o2759O926dWPz5s0MHjyYs88+2+uQTB6FVcmXuf8uyNkmItVwVpH8tQxiC6jkNBu4bkLX1q1b6dq1K3/++Sdz5syxZBmi/JkPc4GIVBWR6sBKYKqIPBn80AIrxcZhmhD1+++/06lTJ/766y/mzp1Lp06dvA7JFMCfcZg1VPUAcBnwpqqeDYTd3Pc2cN2Eqj179pCdnc38+fPp0KGD1+GYQviTPWJEpDbOUKIHghxP0NiKkSbU7Nmzhxo1atC6dWvWrl1LhQr2uxnq/ClhPgIsAf5Q1WUi0gjYFNywAs8m3zChZPXq1TRr1oznnnsOwJJlmCgye6jqu8C7Pq83Av2CGVSgZWRlk5qRTZRApVhb68R466effqJbt25ER0fTo0cPr8MxxeDPBMK1cJamaOC7v6peG7ywAuugz6B1m+XFeGnFihX06NGD+Ph4Fi5cSJMmTbwOyRSDP/XTT4ClwJdAVnDDCQ5bLdKEgn379tG9e3cSEhJYuHAhp5xyitchmWLyJ2FWVtU7gh5JEFkPuQkFxx13HC+88ALnnnsuJ58c9qtWl0v+dPrMEZGwvj/LJt4wXvr888+ZN28eAIMGDbJkGcb8ySDXA3e7a5KnAwKoqtYIamQBlGITbxiPLFiwgIsuuogzzjiD7t27ExXlTxnFhCp/MkitoEcRZEdWjLQ2TFN2PvvsMy699FIaN27MnDlzLFlGgCJ/gqqahTNo/W73+UlAq2AHFkhHBq1bCdOUjZkzZ9KvXz/OOOMMFi1axPHHH+91SCYA/LmX/AWgCzDU3XQIeCWYQQVabhumVclNGZkzZw4tW7ZkwYIF1KxZ0+twTID4U0c4V1WvA1IBVHUPEOvPwUWkt4isEZH1InJPAftcISK/isgvIvK235EXg028YcpKamoqAC+88AILFy6kevXqRXzChBN/EmaGiEThLq0rIjWB7KI+JCLROIulXQA0BQaJSNM8+zQG7gU6qGoz4Nbihe+f3DZMK2GaIJoyZQrNmjVj69atREVFkZCQ4HVIJsD8SZgvAh/iLIb2EM4A9sf9+Fw7YL2qblTVdJzbK/PeUvkP4EVV3Qugqjv9jrwYUmziDRNkEyZMYPjw4TRo0MBKlRHMn3vJp4jI90B3d9MAVf3Zj2PXBbb4vN4K/C3PPk0AROQrIBoYo6qf+XHsYkm22dZNEL300kvceOON9O7dm+nTp1OxYkWvQzJBUtiaPvFutRpV/QWYhVMVbxTA88cAjYHOwCDgNRE5Lp9YrhWR5SKyPCkpqdgnsYHrJljeeecdbrzxRi666CI+/vhjS5YRrrAq+VzgFAAROQVYhtMWebuIPOLHsbcB9XxeJ7rbfG0FZqhqhqpuAtbiJNCjqOp4VW2jqm1q166d9+0i5VbJ46xKbgLrggsu4N577+WDDz4gLi7O63BMkBWWMGuo6lr3+dXAu6p6A85s6xf7cezvgMYi0lBEYoGBwIw8+3yMU7rMmRWpCRDwJX3tXnITaG+99RaHDx/muOOO47///S+xsX4NHDFhrrCEqT7PuwL/A1DVNPzoJVfVTGAUTkl1NfC+qv4iImNFJCfhzgV2i8ivwCLgTlXdXfzLKFzurZGWME0pqSr3338/Q4YM4dVXX/U6HFPGCssgv4jIYzjV6CbAPMhdOdKvSSVVdTYwO8+2B3yeK3C7+wgKVbXZ1k1AqCr33HMPTzzxBCNHjuSmm27yOiRTxgorYV4DpACnA71V9aC7/Uzg6WAHFihpmdlkZCmx0VHEV7DZ1k3JqCq33XYbTzzxBDfccAPjx48nOtp+n8qbwtYlPwj8J5/tXwFfBTOoQLIhRSYQtm3bxptvvsktt9zCM888YzP3l1MFZhER+Rh4Ffif2x7p+97JOB1BW1X1jeCGWDopNvGGKYXs7GxEhMTERFauXEndunUtWZZjhVXJbwR6AGtF5BsRmSEi80RkPTAR+CXUkyXYapGm5DIzMxk+fDgPPvggAImJiZYsy7kCE6aqblPV21W1Ec5MRU8Co4HWqtpVVT8sqyBLwybeMCWRkZHB0KFDmTp1qg0ZMrn8yiKquh5YH+RYguJA7sQbNmjd+Cc9PZ1BgwYxffp0nnjiCe68806vQzIhIuKLXTltmFWthGn8oKoMHDiQjz76iGeffZZbbrnF65BMCIn4LJKS04ZpCdP4QUS47LLL6NGjBzfccIPX4ZgQ41cWcW9trO9WzcOK3RZp/HHw4EF++OEHzj//fIYMGeJ1OCZE+bNExYXAT7i3RopIKxH5KNiBBcqRFSOtDdPkLzk5mQsuuIBevXrx119/eR2OCWH+TCA8Fmcey30AqroSODWYQQXSASthmkLs37+fXr168fXXXzNx4kROPPFEr0MyIcyfLJKhqvvyjD/TgnYONTZw3RRk79699OrVi5UrV/L+++9z2WWXeR2SCXH+ZJHVInIFECUiDYGbgaXBDStwbOC6KciECRNYtWoV06dPp2/fvl6HY8KAP1XyUcDZOFO6TQfSgLAZa3Fk4Lq1YZqj3XHHHSxfvtySpfGbPwmzl6reraqt3cc9OCtBhgVbMdL42r59Oz179mTDhg2ICM2bN/c6JBNG/EmY9+Wz7d+BDiRYrA3T5Ni6dSudOnXim2++Yfv27V6HY8JQYbMV9QJ6A3VFxHf+y6r4MeN6qDhgC6AZYPPmzXTt2pXdu3czd+5czj33XK9DMmGosCyyE/gZSAV+8dmeDNwTzKACRVV9xmFawiyvNm/eTKdOnThw4ADz58+nbdu2XodkwlRhEwivAFaIyFuqmlqGMQXMofQsVKFihWhiov1pfTCRqGbNmrRo0YKxY8fSunVrr8MxYcyfYlddd1ndpkB8zkZVbRK0qALEboss39atW0edOnWoUqUKM2fO9DocEwH8KXZNwpkwWHB6x98H3gtiTAGTkmYTb5RXP/74Ix06dOAf//iH16GYCOJPwqykqnMBVHWDqt5HmAwrOmBjMMulH374gS5duhAbG8uYMWO8DsdEEH+KXmkiEgVsEJHrcZbdrRLcsAIjd9C6dfiUG8uWLaNXr15Uq1aNhQsX0qhRI69DMhHEn0xyG1AZ55bIR4BqwIhgBhUoNmi9fMnMzGTo0KHUqFGDhQsXcmxZrjUAACAASURBVPLJJ3sdkokwRWYSVf3WfZqMs7YPIlI3mEEFSk4bpnX6lA8xMTF89NFHVKtWjbp1w+JX1ISZQtswRaStiFwiIrXc181EZArwbWGfCxW2Jnn5MH/+fP7973+jqjRt2tSSpQmaAhOmiDwKvAUMBj4TkTHAImAVEPJDisB3WJF1+kSq2bNn07dvX2bOnElKSorX4ZgIV1jRqx/QUlUPi0gNYAvQXFU3lk1opZdsnT4R7ZNPPmHAgAE0b96cefPmUaVKWPRFmjBWWJU8VVUPA6jqHmBtOCVLsDbMSPbBBx/Qv39/WrduzYIFC6hZs6bXIZlyoLBM0khEprvPBWjo8xpVDfnpqXPvI7eEGXFEhA4dOjBjxgyqVq3qdTimnCgsk1ye5/ULwQwkGKwNM/Js2bKFevXqcfnll3PZZZeRZ+kUY4KqsMk3FpRlIMFg4zAjy2uvvcaoUaP43//+R8eOHS1ZmjIX0VP45KznU9Wq5GHvxRdf5Nprr6Vbt260a9fO63BMORXRCdPaMCPDM888w6hRo+jXrx8fffQR8fHxRX/ImCDwO2GKSFwwAwkGq5KHv0WLFnH77bfTv39/pk2bRlxc2P0amghSZMIUkXYi8hOwzn3dUkSeD3pkpZSVrRxKz0IEKsdawgxXnTt3ZsqUKbzzzjtUqGCdd8Zb/pQwxwF9gd0AqroK6BLMoAIhZ6aihNgYoqKscyCcqCqPPfYYv/32GyLC0KFDiYmx//SM9/xJmFGq+nuebVnBCCaQkm3QelhSVe666y7uvfdeJk+e7HU4xhzFn2yyRUTaASoi0cBNwNrghlV6NvFG+FFVbr31VsaNG8eNN97II4884nVIxhzFnxLmDcDtQH1gB9De3RbSjqxHbu1e4SA7O5t//vOfjBs3jttuu43nn3+eqKiIHsRhwpA/xa9MVR0Y9EgCLMV6yMNKWloaq1ev5u677+bRRx+1QekmJPmTTb4TkTU4C59NV9XkIMcUEAdSrQ0zHGRmZpKamkpCQgJz584lNjbWkqUJWUXWeVT1FOA/wNnATyLysYiEfInzSJXcEmaoysjI4KqrruKCCy4gIyODuLg4S5YmpPnVSKSqX6vqzcBZwAGciYWLJCK9RWSNiKwXkXsK2e9yEVERaeNX1H6wiTdCW3p6OldccQXTpk2jX79+NsbShAV/Bq4niMhgEZkJLAOSgHP9+Fw08CLOkrxNgUEi0jSf/aoAtxDgZS+sDTN0paamctlll/Hxxx/z3HPP8a9//cvrkIzxiz/Z5GdgJvCEqn5RjGO3A9bnTDosIu/izOL+a579HgYeB+4sxrGLlGxtmCHrn//8J7NmzeKVV17huuuu8zocY/zmTzZppKrZJTh2XZxlLXJsBf7mu4OInAXUU9VZIhLYhJlmJcxQde+999K1a1eGDBnidSjGFEuB2URE/k9V7wA+FBHN+35pZ1wXkSjgaWC4H/teC1wLUL9+fb+Of6QN0xJmKEhOTmbixIncdNNNNG7cmMaNG3sdkjHFVlg2ec/9t6QzrW8D6vm8TnS35agCnAksdntGTwRmiMjFqrrc90CqOh4YD9CmTZtjknd+UqzTJ2Ts37+fCy64gGXLlnH++efTunVrr0MypkQKm3F9mfv0DFU9KmmKyCigqBnZvwMai0hDnEQ5ELjK5/j7gVo+x1wM/CtvsiypnHvJrUrurT179tCrVy9WrVrFtGnTLFmasObPsKIR+WwbWdSHVDUTGAXMBVYD76vqLyIyVkQuLl6YxZdiVXLP7dq1i27duvHjjz8yffp0Lr30Uq9DMqZUCmvDvBKnVHjUapE4Vel9/hxcVWcDs/Nse6CAfTv7c0x/2Wzr3lu5ciUbN25k5syZ9OzZ0+twjCm1wrLJMpw5MBNxxlPmSAZWBDOoQDjgljCrWhtmmUtPTyc2Npbu3buzefNmqlev7nVIxgREgVVyVd2kqvNVta2qLvB5LFPVjLIMsrjSMrNIz8wmJkqIi7EZb8rSli1baNGiBe+//z6AJUsTUQqrki9R1U4ishfw7ZkWQFW1RtCjKyHf9ku7N7nsbN68ma5du7J7927q1atX9AeMCTOFVclzlqGoVcg+IcnaL8ve+vXr6dq1KykpKSxYsIA2bQI2LYAxIaOwKnnO3T31gGhVzQLOAa4DKpdBbCWWO2g9ztovy8KuXbvo1KkThw8fZuHChZYsTcTyp4HvY5zlKU4BJgKNgbeDGlUp2fIUZatmzZqMGjWKRYsW0apVK6/DMSZo/Mko2aqaISKXAc+r6jgRCele8tyJN2zQelCtWrUKEaFFixbce++9XodjTND5tUSFiAwAhgKXuNtCuq5rkwcH3/Lly+nZsyeNGjXiu+++s841Uy74e6dPF5zp3Ta6tzq+E9ywSseq5MG1dOlSunXrRrVq1Zg2bZolS1Nu+LNExc/AzcByETkd2KKqIb3+qa0YGTxffPEFPXr0oHbt2ixZsoSGDRt6HZIxZabIIpiInA9MxZlAQ4ATRWSoqn4V7OBKKtlmWw+aZ555hrp167JgwQLq1q3rdTjGlCl/MsozQB9V/RVARM7ASaAhO3Ykp9OnqlXJA0ZVERHefPNNkpOTOeGEE7wOyZgy508bZmxOsgRQ1dVAbPBCKj0buB5Ys2bNolOnTuzfv59KlSpZsjTllj8J8wcReUVEznMfLxPik2/YwPXA+fjjj7n00ks5dOgQWVlZXodjjKf8SZjXAxuBu9zHRpy7fUJWivWSB8S0adMYMGAAZ511FvPnz6dGjZCdPsCYMlFoRhGR5sApwEeq+kTZhFR6B2zFyFKbPn06AwcO5Nxzz2XWrFlUrVrV65CM8VyBJUwRGY1zW+Rg4H8ikt/M6yEpd1iRVclL7KyzzuKqq65izpw5liyNcRVWJR8MtFDVAUBb4IayCan0bOB6yS1evJjs7GwaNGjA1KlTSUhI8DokY0JGYQkzTVUPAqhqUhH7hgxVPdJLbuMwi+WFF16gS5cuvPrqq16HYkxIKiyjNPJZy0eAU3zX9intuuTBcjgji6xsJS4milibbd1vTz/9NHfccQeXXHIJI0cWucadMeVSYQnz8jyvS7o+eZmy9ciL79FHH2X06NEMGDCAt956iwoV7LszJj+FrUte1LrjISnZZioqlo0bN/LQQw8xePBgJk2aREyMfW/GFCTi/jqSbT3yYmnUqBHffPMNLVq0IDo62utwjAlpEdfIl2ITbxRJVbnrrruYPHkyAK1bt7ZkaYwf/E6YIhIXzEACJdkGrRdKVbnlllt48skn+eGHH7wOx5iwUmTCFJF2IvITsM593VJEng96ZCWUnDukyDou8srOzub666/n+eef54477uDZZ5/1OiRjwoo/JcxxQF9gN4CqruLIErwhx9ow86eqXHPNNYwfP57Ro0fz5JNP2kzpxhSTPwkzSlV/z7MtZKetSbGEmS8RoUGDBowZM4b//Oc/liyNKQF/ssoWEWmHs9RuNHATsDa4YZWctWEeLSMjg02bNtGkSRMeeOABr8MxJqz5U8K8AbgdqA/sANoTwveVp1gbZq60tDSuuOIKzjnnHHbt2uV1OMaEvSKLYaq6ExhYBrEEhE284UhNTeXyyy9n9uzZPP/889SqVcvrkIwJe/4sgvYaoHm3q+q1QYmolOxOHzh06BCXXHIJ8+fP59VXX+Xaa0PyR2VM2PEnq8z3eR4PXApsCU44pZfbhlmOB64//vjjzJ8/nzfeeIPhw4d7HY4xEcOfKvl7vq9FZCrwZdAiKiWbfANGjx7N+eefT/fu3b0OxZiIUpJbIxsCIbtsYHldMXLfvn2MHDmSPXv2EBcXZ8nSmCDw506fvSKyx33sA/4H3Bv80EqmPA5c37NnD927d2fq1Kl2u6MxQVTUImgCtAS2uZuyVfWYDqBQkZ19ZLb1yrHlI2EmJSXRvXt31qxZw8cff2wlS2OCqNASppscZ6tqlvsI2WQJkJJ+ZKai6KjIv5Plr7/+onPnzqxbt46ZM2fSp08fr0MyJqL504a5UkRaBz2SAChvU7tlZmYSExPD7Nmz6dGjh9fhGBPxCswsIhKjqplAa+A7EdkAHMRZ30dV9awyitFv5aX9cseOHdSqVYvExERWrFhBVFTETWtqTEgqLLMsA84CLi6jWEotJc0ZgxnJPeSbNm2iS5cuXHzxxYwbN86SpTFlqLDMIgCquqGMYim1AxFeJV+3bh1du3bl4MGDXH311V6HY0y5U1hmqS0itxf0pqo+HYR4SiWnDbNqBA5aX716Nd26dSMjI4NFixbRsmVLr0MyptwpLGFGAwm4Jc1wkByhJcyMjAwuvPBCsrOzWbx4Mc2aNfM6JGPKpcIyy3ZVHVuag4tIb+A5nOT7uqo+luf924FrgEwgCRiRz2TFfstpw4y0Tp8KFSowYcIETjrpJE4//XSvwzGm3Cqsx6BUJUt3suEXgQuApsAgEWmaZ7cVQBtVbQF8ADxRmnOmRNjUbsuXL2fChAkAdOnSxZKlMR4rLGF2K+Wx2wHrVXWjqqYD7wL9fHdQ1UWqesh9uRRILM0JD0TQxBvffPMN3bp147///S+HDh0q+gPGmKArMGGq6p5SHrsuR08Dt9XdVpCRwJz83hCRa0VkuYgsT0pKKvAAObdFhvvUbl988QU9e/bk+OOPZ/HixVSqVMnrkIwxlGy2ooATkSFAG+DJ/N5X1fGq2kZV29SuXbvA40TCej4LFy6kd+/eJCYmsmTJEurVq+d1SMYYVzAzyzbA9689kSOTeOQSke7Av4FOqppWmhNGwtRuq1atolGjRsyfP58TTgjZWfSMKZeCWcL8DmgsIg1FJBZnXaAZvju496i/Clzsrh1UKslh3IaZnJwMwG233cayZcssWRoTgoKWMN370EcBc4HVwPuq+ouIjBWRnNstn8QZ6zlNRFaKyIwCDueXcJ1846OPPqJRo0asXLkSgIoVK3ockTEmP0HNLKo6G5idZ9sDPs8DOnnjgTCcfOO9995j8ODBtG3bloYNG3odjjGmECHR6RMo4TZw/c033+Sqq67i3HPPZd68eVSrVs3rkIwxhYiYhJmRlU1qRjbRUULFCtFeh1OkBQsWMGzYMDp37sycOXOoUqWK1yEZY4oQMQnTt/3SWVkjtHXs2JGHH36YTz/9lMqVK3sdjjHGD5GTMNPCo8Nn6tSp7Ny5kwoVKvDvf//bOniMCSMRkzAPhMGg9aeeeophw4bx1FNPeR2KMaYEIiZhpoR4D/kjjzzCnXfeyZVXXskjjzzidTjGmBKImIQZqoPWVZUHH3yQ++67j6FDh/Lmm29SoUJoxWiM8U/EJMxQbcNMSUnhvffeY8SIEUycOJGYmNCKzxjjv4j56w21iTdUlaysLKpUqcJXX31F9erVbcEyY8JcxPwFJ4fQxBvZ2dncdNNNDB06lKysLGrWrGnJ0pgIEDF/xckhsgBadnY21113HS+++CL16tWzRGlMBImYv+ZQmHgjKyuLESNG8Prrr3Pffffx+OOPh8UgemOMf7yvvwZIThumlwnzn//8J5MnT2bs2LHcf//9nsVhjAmOiEmYuctTeNiGOXToUJo0acIdd9zhWQzGmOCJmCp5skcrRqalpfHJJ58AcN5551myNCaCRVzCLMtOn8OHD3PppZdy6aWX8ssvv5TZeY0x3oi4KnlZtWEeOnSIfv36sWDBAsaPH0+zZs3K5LzGGO9ETMIsy4HrKSkp9O3bly+++IJJkyYxbNiwoJ/TGOO9iEiYqlqmK0Z+9tlnfPnll7z55psMGjQo6OczxoSGiEiYaZnZZGQpsTFRxMUEb7Z1VUVE6N+/P7/99hunnnpq0M5ljAk9EdHpkztTURDbL3fv3k2nTp348ssvASxZGlMORUQJM9jtlzt37qR79+6sXbs2d/1wY0z5ExEJM5jtl9u3b6dbt25s3ryZTz/9lO7dA7oysDEmjEREwkwO0n3kSUlJdOrUiT///JM5c+bQqVOngB7fGBNeIqsNM8CD1qtXr07Hjh2ZO3euJUtjTGSUMHPvIw9QCXPjxo3Ex8dTp04dXn/99YAc0xgT/iKkhBm4Tp9169bRsWNHrrjiClS11MczxkSOiEiYKQGaeGP16tV07NiR9PR0XnrpJZvL0hhzlIhImMlppW/D/Omnn3LbKRcvXkyLFi0CEpsxJnJERBtmIHrJb7nlFmJjY1m4cCFNmjQJVGgmwmVkZLB161ZSU1O9DsX4KT4+nsTExBItdx0hCbP0bZjvvvsuycnJnHLKKYEKy5QDW7dupUqVKjRo0MCacMKAqrJ79262bt1Kw4YNi/35iKiSl3S29a+//pphw4aRkZHB8ccfb8nSFFtqaio1a9a0ZBkmRISaNWuWuEYQISXM4rdhfv755/Tp04c6deqwZ88eTjjhhGCFZyKc38ny0B74YylsWgJpyRBXBRp2gvrtoVKN4AZpcpXmP7fIKGEWsw1zwYIF9O7dm/r167NkyRJLlib4dm+ABWPht1kQHQdVE51/f5vlbN+9oWSH3b2bVq1a0apVK0488UTq1q1Lq1atOO6442jatGmAL8LpEO3bt2+xPtO5c2eWL19+zPZJkyYxatQov48zefJkGjduTOPGjZk8eXK++6xcuZL27dvTqlUr2rRpw7Jly4oVa1EiImEWZ8XIuXPn0rdvX0499VQWL17MSSedFOzwTHl3aA98/TzEVIRqiVChIog4/1ZLdLZ//byzXzHVrFmTlStXsnLlSq6//npuu+223NdRUUX/eWdmZpbkisrcnj17eOihh/j2229ZtmwZDz30EHv37j1mv7vuuosHH3yQlStXMnbsWO66666AxhEZCTPN//V8qlevTvv27Vm0aBHHH398sEMzxqmGZ2VAfNX834+vClnpsOXbgJ42KyuLf/zjHzRr1oyePXty+PBhwCnx3XrrrbRp04bnnnuOpKQkLr/8ctq2bUvbtm356quvAFiyZElu6bV169a5M3WlpKTQv39/Tj/9dAYPHpx7g8eCBQto3bo1zZs3Z8SIEaSlpR0T08SJE2nSpAnt2rXLPY8/5s6dS48ePahRowbVq1enR48efPbZZ8fsJyIcOHAAgP3791OnTp3ifWlFCPs2TN/Z1ivHFTx58Nq1a3N/UAsXLrRGehN46+ZDyo5jt/80DaKi4XAhJcisdPh2/LGlzIQToHHJZshat24d77zzDq+99hpXXHEFH374IUOGDAEgPT09t5p81VVXcdttt3Heeefxxx9/0KtXL1avXs1TTz3Fiy++SIcOHUhJSSE+Ph6AFStW8Msvv1CnTh06dOjAV199RZs2bRg+fDgLFiygSZMmDBs2jJdffplbb701N57t27fz4IMP8v3331OtWjW6dOlC69atAXjrrbd48sknj7mGU089lQ8++IBt27ZRr1693O2JiYls27btmP2fffZZevXqxb/+9S+ys7P5+uuvS/TdFSTsS5gH07NQhUqx0cRE53857777Lk2bNuXtt98GStfoa0yxZRyGqCJqP1EVnP0CqGHDhrRq1QqAs88+m82bN+e+d+WVV+Y+nz9/PqNGjaJVq1ZcfPHFHDhwgJSUFDp06MDtt9/OuHHj2LdvHzExTvmqXbt2JCYmEhUVRatWrdi8eTNr1qyhYcOGuWOYr776aj7//POj4vn222/p3LkztWvXJjY29qgYBg8enNuU4Pv44IMPinXNL7/8Ms888wxbtmzhmWeeYeTIkcX6fFHCvoRZVIfPlClT+Pvf/855553HRRddVJahmfKmoJLgXz86HTwVKhb82YzDkJ0OrQcHLJy4uLjc59HR0blVcoDKlSvnPs/Ozmbp0qW5Jcgc99xzDxdeeCGzZ8+mQ4cOzJ07N9/jBqIdtKgSZt26dVm8eHHu9q1bt9K5c+dj9p88eTLPPfccAAMGDOCaa64pdWy+wr6EWdig9QkTJjB8+HA6d+7M7NmzqVKlSlmHZ4wzdOjQ7sL3ObTL2c8DPXv25Pnnn899vXLlSgA2bNhA8+bNufvuu2nbti2//fZbgcc47bTT2Lx5M+vXrwdg6tSpx0yJ+Le//Y0lS5awe/duMjIymDZtWu57RZUwe/Xqxbx589i7dy979+5l3rx59OrV65g46tSpw5IlSwBYuHAhjRs3LuG3kr/wT5i5s60fXeVZs2YN1157Lb169eLTTz896n9UY8pU/fYQXQFSD+T/fuoBiI6Fen8r27hc48aNY/ny5bRo0YKmTZvyyiuvAE574JlnnkmLFi2oUKECF1xwQYHHiI+PZ+LEiQwYMIDmzZsTFRXF9ddff9Q+J510EmPGjOGcc86hQ4cOnHHGGX7HWKNGDe6///7cjqkHHniAGjWcsavXXHNNbnvsa6+9xh133EHLli0ZPXo048ePL+7XUSgJtynM2rRpo75jupasTeLqN5ZxfuNaTB159C/c7Nmz6dat21FVCGMCafXq1f794e/e4AwdykqHSrUgJg4y05ySZXQsnHsT1LQ7zcpKfj83EfleVdsU9rmwL2HmbcN8+umnWbhwIQB9+vSxZGlCQ81ToNsDcMZFTlvlgT+df8+4yNluyTIsBLXTR0R6A88B0cDrqvpYnvfjgCnA2cBu4EpV3Vycc/i2YT788MM88MADjBgxgq5duwbgCowJoEo14LQLnIcJS0FLmCISDbwI9AC2At+JyAxV/dVnt5HAXlU9VUQGAo8DVx57tIKlpGWiqnw77WU+f/9Vhg4dGvB2C2OMgeBWydsB61V1o6qmA+8C/fLs0w/IuSn0A6CbFHOQ5P7DGexbMonP33+VkSNHMnHiRKKjCx7AbkyghVs/QHlXmp9XMBNmXWCLz+ut7rZ891HVTGA/ULM4J0k+nEHWwX10vmQw48ePt2RpylR8fDy7d++2pBkmcubDzDvm1F9hMXBdRK4FrgWoX7/+Ue+1P6Umtz70f/RpUdevyQaMCaTExES2bt1KUlKS16EYP+XMuF4SwUyY24B6Pq8T3W357bNVRGKAajidP0dR1fHAeHCGFfm+1/vMk+h9ps04ZLxRoUKFEs3cbcJTMItk3wGNRaShiMQCA4EZefaZAVztPu8PLFSr2xhjQlTQSpiqmikio4C5OMOK3lDVX0RkLLBcVWcAE4CpIrIe2IOTVI0xJiQFtQ1TVWcDs/Nse8DneSowIJgxGGNMoITdrZEikgT8nmdzLWCXB+EEm11X+IjEa4LydV0nq2rtwj4UdgkzPyKyvKh7QMORXVf4iMRrAruuvGwcjjHG+MkSpjHG+ClSEmak3jxu1xU+IvGawK7rKBHRhmmMMWUhUkqYxhgTdGGVMEWkt4isEZH1InJPPu/Hich77vvfikiDso+y+Py4rttF5FcR+VFEFojIyV7EWRxFXZPPfpeLiIpIWPTE+nNdInKF+/P6RUTeLusYS8KP38H6IrJIRFa4v4d9vIizOETkDRHZKSI/F/C+iMg495p/FJGzijyoqobFA+duoQ1AIyAWWAU0zbPPP4FX3OcDgfe8jjtA19UFqOQ+vyHUr8ufa3L3qwJ8DiwF2ngdd4B+Vo2BFUB19/XxXscdoOsaD9zgPm8KbPY6bj+uqyNwFvBzAe/3AeYAArQHvi3qmOFUwiyT+TU9UOR1qeoiVT3kvlyKM5FJKPPnZwXwMM6k0allGVwp+HNd/wBeVNW9AKq6s4xjLAl/rkuBqu7zasCfZRhfiajq5zi3XBekHzBFHUuB40Sk0Jl8wilhlsn8mh7w57p8jcT5XzGUFXlNbvWnnqrOKsvASsmfn1UToImIfCUiS91lWkKdP9c1BhgiIltxbne+qWxCC6ri/u2Fx3yYxiEiQ4A2gDcLWAeIiEQBTwPDPQ4lGGJwquWdcWoCn4tIc1Xd52lUpTcImKSq/yci5+BMmnOmqmZ7HVhZCqcSZnHm16Sw+TVDjD/XhYh0B/4NXKyqaWUUW0kVdU1VgDOBxSKyGaf9aEYYdPz487PaCsxQ1QxV3QSsxUmgocyf6xoJvA+gqt8A8Tj3Y4czv/72fIVTwozU+TWLvC4RaQ28ipMsw6FNrNBrUtX9qlpLVRuoagOcdtmLVXV5/ocLGf78Dn6MU7pERGrhVNE3lmWQJeDPdf0BdAMQkTNwEma4TzM/Axjm9pa3B/ar6vZCP+F1T1Yxe7364PyPvQH4t7ttLM4fGzg/xGnAemAZ0MjrmAN0XfOBHcBK9zHD65hLe0159l1MGPSS+/mzEpzmhl+Bn4CBXsccoOtqCnyF04O+Eujpdcx+XNM7wHYgA6fkPxK4Hrje52f1onvNP/nzO2h3+hhjjJ/CqUpujDGesoRpjDF+soRpjDF+soRpjDF+soRpjDF+soQZgkQkS0RW+jwaFLJvg4JmYynmORe7s9Wscm/rO60Ex7heRIa5z4eLSB2f914XkaYBjvM7EWnlx2duFZFKJTjXsyLS0X0+yp3VRt3xlQV9pq87o88qd8ai64p73iJiGuvexICInO/OiLRSROqKyAdFfDb3ZyAio/04V20R+SwwkUcIr8dK2SPf8WMpxdi3AQXMxlLMcy7GHYcGXEspx3oSpLGVeeL8O/A/Pz6zGahVzPPUBJb6vG7tftcFHguogDMpRaL7Og44LYi/J68AQ4L5OwZMBDoE6xrC7WElzDDhliS/EJEf3Me5+ezTTESWuSWOH0Wksbt9iM/2V0UkuojTfQ6c6n62m1ti+smdXzDO3f6YHJmj8yl32xgR+ZeI9Me55/0t95wV3ZJhG7cU+qRPzMNF5IUSxvkNPpMliMjLIrLcLXU95G67GagDLBKRRe62niLyjfs9ThORhHyOfTmQW7pS1RWqurmIeKrg3Eu+2/1Mmqqucc85SUReceNbKyJ93e3RIvKkW1r+0bdEKiJ3u9/7KhF5zOc4/UXkGuAK4GERecu3puEe8ykR+dk95k3u9pyfwWNARfd7fssttd7qc95HROQW9+XHwOAi0xclDwAABLBJREFUrrv88Dpj2+PYB5DFkbt6PnK3VQLi3eeNgeXu8wa4JUzgeWCw+zwWqAicAcwEKrjbXwKG5XPOxRwpud0JvIdz59QWoIm7fQpwK07paw1Hljg5zv13DPCvvMfzfQ3UxplKLGf7HOC8EsZ5K/Bfn/dquP9Gu/u1cF9vxi0V4tz//DlQ2X19N/BAPueZDFyUz/bcYxXws3sd2Ilzl8lgIMrdPgknAUe5P7+t7vd7LXCfu08csBxoCFwAfM2ReVBr+Bynfz7PfX8PbsCZ3jAmz2d9v7sUn5gbAD+4z6Nw7nyp6b6uC/zk9d9EqDxstqLQdFhV87bNVQBecNvssnDuUc7rG+DfIpIITFfVdSLSDTgb+E6cqUEr4vxB5+ctETmMkxRuAk4DNqnqWvf9ycCNwAs4c1hOEJFPgU/9vTBVTRKRjeLcu7sOOB3nlrsbixlnLJAA+H5PV4jItTilvJNwbuf7Mc9n27vbv3LPE4vzveV1EiW4V1pVrxGR5kB34F9AD47MyvS+OrP7rBORjTjX3hNo4ZbKwZkwprH7+YnqzoOqqoXN65hXd5yJtDP9+ayqbhaR3eLMWXACsEJVcyat2YlTQjfY9G7h5Dac+8lb4pQCjpl0V1XfFpFvgQuB2W71ToDJqnqvH+cYrD4TYIhIjfx2UtVMEWmHMxlDf2AU0LUY1/IuTnXyN5wStIqTvfyOE/geeBKnVH2ZiDTESVBtVXWviEzCKcHlJTjtnoOKOMfhAj5/9MFE5uIkmeWqeg2Aqv4E/CQiU4FNHEmYee9DVjeem1R1bp7j9irq3AH2Ok6cJwJv+GyPx/kuDNZLHk6qAdvdEspQnGrnUUSkEbBRVccBnwAtgAVAfxE53t2nhvi/JtAaoIGInOq+Hgoscdv8qqnqbJxE3jKfzybjtOnl5yOc2a4H4SRPihunOvXF+4H2InI6zmzgB4H9InICTpU2v1iWAh1yrklEKotIfqX11bjtuIVR1V6q2sotWSaISGeft1sBv/u8HiAiUSJyCs5yEGuAucANIlLBjaeJiFQG/gf8Xdze/YL+8yrA/4DrxJnisKDPZuSc0/UR0Bto68aUowlQ6lEYkcISZvh4CbhaRFbhVOUO5rPPFcDPIrISZ77JKar6K3AfME9EfsT5Yyp0Gv4cqpqK0xM9TUR+ArJxemarAJ+6x/sSuD2fj08CXsnp9Mlz3L04CelkVV3mbit2nKp6GPg/4E5VXYWzls5vwNs41fwc44HPRGSRqibhlKTecc/zDc73mdcs3GnawOk8Eme28UTgRxF5PZ/PCHCXOMOeVgIPcfQkyX/8f3t3jENAEAVg+J9DaN1FL1FpXINz6InWDSQ0IhGlKBQaDqKRUbxJiCBTEIX/K3cnu5vJ5u3L28kboovWnOiYcyYyuwOwKz9tRkTtcUG0H9uWaw3ezcWDSbnXvrwvvSdjxuX8FCDH1hQromxwuRvXKnMh3JdceimltAHa+QPd0kuJYJZzfrtW8ldSdMHfAd2c8/Hu+BrolI/c3zPDlF7rA81fP8S3pVjMfgKWD8GyAQwNljdmmJJUyQxTkioZMCWpkgFTkioZMCWpkgFTkioZMCWp0hWg5iawTuzWWgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5,5))\n",
    "plot_roc(fpr, tpr)\n",
    "plt.plot(fpr[np.max(np.where(tpr <= 0.88))], tpr[np.max(np.where(tpr <= 0.88))], marker=\"o\", markersize=10, \n",
    "         alpha=0.5, label=\"Threshold=0.8\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[44,  3],\n",
       "       [ 7, 46]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predictions and confusion matrix with the new threshold\n",
    "y_pred_new_threshold = knn_probs >= new_threshold\n",
    "y_pred_new_threshold\n",
    "confusion_matrix(y_test, y_pred_new_threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.8679245283018868, 0.06382978723404255)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_tpr, new_fpr = 46/(46+7), 3/(3+44)\n",
    "new_tpr, new_fpr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.5 Precision/Recall Curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly to the ROC curve, the P/R (Precision/Recall) curve is plotted with the goal of evaluating classifiers when different thresholds are used to label the instances. In the P/R curve, Precision is plotted in the y-axis, and Recall is in the x-axis. Let's now visualize the P/R curve of the two classifiers we have used so far, `knn` with $k=5$ and `knn` with $k=2$. (Similar to the `roc_curve` function, also `precision_recall_curve` needs either class probabilities or decision function values as second argument). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUwAAAFNCAYAAABrMlb6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de3wddZ3/8dcn15M0N3KhQO8tlJZbK5arPyouKLe1VHZhwWVZLtJdUUHBC7orVlRWXdTFta6gVZRFAVkXu1rBBVtYFJYWuUm7QCm0TWlpbk1zv35+f8wkPUlPkmmSk+ScvJ+Pxzw6Z2bOd76TtO9+Z74z3zF3R0REhpYx3hUQEUkVCkwRkYgUmCIiESkwRUQiUmCKiESkwBQRiUiBKSISkQJTRsTMXjazM4fYZqaZNZpZ5hhVK6nM7Ewzq4z7/KaZnT2edZKxocBMU+E/4pYwqN42s7vNrGC09+Pux7r7+iG22e7uBe7eNRr7NLMjegKr33HuTtZxjoSZnWxma81sr5nVmtkzZnbVeNdLDp4CM729390LgBOBJcA/9t/AAqn29+B84OG4zz3HuRh4B/DZcalVAmZ2GvA74HHgSKAM+DBw3jDLS4tWeqpKtX8oMgzuvhP4DXAcgJmtN7OvmNnvgWZgrpkVm9lqM9tlZjvN7Mvx/zjN7Foz22xmDWa2ycxODJf3no6GLamNZrYvbNV+M1w+28zczLLCz0eY2ZqwtbXFzK6N289KM3vAzH4S7utlM1vS75DOB9YmOM7dwCMEwdlTXq6Z3W5m28M6fc/M8uLWX2hmz4d1ft3Mzg2XXxV3vFvN7O+G+eP/Z+DH7v41d6/2wLPufkm4nyvN7Mn4L4Q/qyPD+bvN7N/CFmoT8MmwJR3/u/mAmb0YzmeY2c3hsdSEP8vSYdZd+lFgTgJmNoMgZJ6LW/w3wAqgENgG3A10ErSC3gG8D/hQ+P2LgZXAFUARsAyoSbCrO4A73L0ImAc8MECV7gMqgSOAvwRuM7M/i1u/LNymBFgDfCfuWLKBpcB/JzjO6QQtty1xi78KzCcI0SOBacAt4fYnAz8BPhXuaynwZvi9PcCfh8d7FfCtnv8kojKzfOA04MGD+V4CHwS+QvC7ugNoAv6s3/qfhvMfA5YD7yb4+dYBq0a4f+nh7prScCL4h98I7CUIxO8CeeG69cCtcdtOBdp61ofLLgPWhfOPADcMsp+zw/kngC8C5f22mQ04kAXMALqAwrj1/wTcHc6vBB6NW3cM0BL3+SzgsQTH2RDu4zGgJFxnBOEyL27704A3wvk7gW9F/Hk+1PMzAM4EKhP9DPp9Z1pYpwWDlHsl8GS/ZQ4cGc7fDfyk3/ovAz8M5wvDY5wVft4MnBW37eFAB5A13n8n02FSCzO9LXf3Enef5e7XuXtL3LodcfOzgGxgV9gxsZcgTA4N188AXo+wv2sIWnP/Z2YbzOzPE2xzBFDr7g1xy7YRhEuP3XHzzUCs53SexKfjy929kCDIFgDl4fIKIB94Nu64Hg6XD3pcZnaemT0dXjbYG+63PNG2g6gDuglCayR29Pv8U+AiM8sFLgL+6O7bwnWzgP+MO97NBP9BTR1hHQSdkk9m8eP67SBoYZaHAVvi7kXufmzc+nlDFuj+mrtfRhC0XwMeNLMp/TZ7Cyg1s8K4ZTOBnRHrnfD6Zbj/xwlaZLeHi6qBFuDYuOMq9qCDCAY4rjCI/iMsZ6q7l4T7tIh17KlPM/AU8BeDbNZEEOo9+z4sUVH9yt1E8J/MefQ9HYfgmM6LO94Sd495cB1bRkiBKbj7LuC3wDfMrCjsOJhnZu8ON/kBQWfDO8Ne9SPNbFb/cszscjOrcPdugksBELSw4ve1A/gD8E9mFjOzEwhapv8+VD3NbA6Q6+6bB9nsX4D3mtmisB7fJ7j+eGhYxjQzOyfcdjVwlZmdFR7zNDNbAOQAuUAV0Glm5xFc0x2OTwNXmtmnzKwsrMMiM7svXP8CcKyZLTazGMEliSh+CtxAcN3153HLvwd8pef3Y2YVZnbhMOsu/SgwpccVBEGxieBU8kHCU0l3/zlBp8NPCa4VPgQk6nk9F3jZzBoJOicu7XcZoMdlBNc13wL+E/iCuz8aoY4XMEDrsoe7VxF05NwSLvoMQSfQ02a2D3gUODrc9hnCDh2gnuDWn1nh5YLrCTqt6ghacWsi1C9Rff5A0EHzZ8BWM6sF7uo5Dnd/Fbg1rNdrwJMDFNXfzwg6dn7n7tVxy+8I6/pbM2sAngZOGU7d5UAWXhgWmfDMbC3wHXcfNDRFkkUtTEkl64F1410JmbzUwhQRiUgtTBGRiBSYIiIRZQ29ycRSXl7us2fPHu9qiEiaefbZZ6vdvWKwbZIWmGb2Q4Jncfe4+3EJ1hvBLRDnEzzNcaW7/3GocmfPns3GjRtHu7oiMsmZ2bahtknmKfndBPflDeQ84KhwWgH8WxLrIiIyYkkLTHd/AqgdZJMLCQYVcHd/Gigxs5E+cysikjTjeQ1zGn0HFagMl+06mEKe/tltlLy+hpbsYlqzS2jPLaUrdgieV4rll5NRUEZ2YQU5RRXkF5VSlJdLYSyLgtwssjLV5yUi0aVEp4+ZrSA4bWfmzJl911W/yoLOzcFIjokewovT6RnUUcAeL+QVCqm3Ipoyi2nOKqEtp4T23EPoipVCXilMKSczv4zYlCIK83IojGVRGMumMJZFUfhnYUyhKzKZjGdg7iQYXqvHdAYYscbd7yJ4/pYlS5b0udN+5vLPs/mtD9LRUEVXYw3eXIO11JDVWkt2Wx2x9jryOusp6KpnCk1UsI8K27e/gG6gPZwaD9x3m2dRRyF1XkitF/ImwZ914Z+NmUHLtiP3ELpiZXh+KXl5U3oDtbA3XPeHbFG/ZdkKXZGUMJ6BuQb4aDhqyylAfThqzkE5fMY8Dp8x5Mhjga4OaK6F5hq6m6pp21dF274qOhqq6W6sDsO2dn/Yduwlt7uFw6jjMKsbpFyCfv5moBaaPLc3UOu8kNowcN+MC9r49S3ZReTF8npDtKgnbHMPDNuBQjg3S696EUm2ZN5W9DOCAV3LLXjD3xcIBqnF3b9HMFrL+QQjyTQTjBqTXJnZUDgVCqeSAeSF06A6WnpDlubquPkavKmGzqYgbIkL2yndbUyhjelWPVTpvfa151PbVkhdfRCktXFBuzOuhdsTtPVMIX7855ysjDBo48I1QeD2b93Gz8eyFboig0m5Z8mXLFniE/o+THdob+wN1fiA7TsFyz38bN49dNlxusiggQJqKaSmu6A3ZBO1bHv+bCCPwcbAzcnMSNCK7duiLRriUkNuVgbBLbYiqcXMnnX3/i/c6yMlOn1SihnkFgbTIbOH3hyguxta9w4ZrvFTZuteSthHCfuYG/ESaJdl0ZhZTENGEfUU9Ybtnq4C3u6cQlVXAXUthdQ2F1LjhbxGIa3kHtThZ2faoK3cogHCdn8YZxPLVujKxKTAnAgyMiC/NJg4Mtp3ujqhpW6IgK3usyyzvZHizhqKqWF6//Iyw6n/bjJjtGaX0JJdEoStFbHXiqjzAqq7C6nqKmB3Zz5vtU+hsi2fPV1TqG1yapvah/3jyMqwAVu5RQO0fPuHcV52pkJXRp0CM1VlZkFBRTBF1dEKLf1bq4O0apuqyexqZUrXbqa07h76DWDZwdSdU0hXrJT23ENoyy6hOauYpswS9mUUspcian1/y3Z3Zz672/Oob3MaWjvY19pJe2c3dc0d1DV3DP/H0xu6fVu5g11SiG/lFsayyM9R6EpfCszJJDsG2UdA0RHRtneHjua+Qdo09CWDjPYGMtobyGYbU0j8Lou+DPJKoLgMDi+jK6+UjpxDaM05JAzbYvZZEXW2P2yrO2I0tHXS0NozddDQ2sm+cL6ts5u9zR3sbe5gyBt0B5CZYRTk9m/hDnx9N9ElhykK3bSiwJSBmUHOlGAqmTn09hCEbGt9pA6v3qmlbv9Us6X36kAMKBloPxlZwQMG+WXBVBE3n19GZ+wQmrNKaMwoZl9GEXspZG9nTsKQbWjrG7Y961o7uqlv6aC+Zfihm2GEoZsoZKN1qBXkZJGRodCdCBSYMrosbC3mlUBZ1PtjO8NOr2gdXjTXQts+aNoTTAlkAUXh1NuezszdH6pT9ocr5T3zPaE7FfLL6MgtoaEzK6712nFg2MYt25dgWUtHF/vCMB7Jj7QgZ/CwHapDrTBXoTsaFJgy/jKzYEp5MEXV2Z7gemxcwDZV91tWDZ2t0PBWMEWQDZTmFFCa37f12huu4SWE4HN58GfeIcHxhDq6umkcMFB7WridvddvE61vbu8KtmnrhPrWg/zh7rf/8kLfYJ1amMvcigLmVkxhbvkUKgpzdRlhAApMSU1ZOVB4WDBF1d4cofXab3l7YzDt3R59P7GS3mDNzi/jkPwyDukfuhVx87GS4E6JAXR2ddPY1j90DwzW/pcU4rdrau+isa2TxrZOdtUPXv3C3CzmhOG5P0gLmFM+hbycyf1wgwJTJo+c/GAqmTH0thBcj21rGORWrUShWxtcXmjdC7WvR9uPZfS9HtsvXLOmlFOSX0ZJz/KyMsgpDc7VI+rqdhoTtXLbOnhrbyuvVzXyRnUTW6uaqG/p4MXKel6sPDBZp5XkMad8Sm9rtCdQjyjOmxSn/ApMkYGYQawomErnRPtOd1dcp1fEa7Kt9WEQR3+UlsycAQM20fLM/DKK8/Mozs8etFj34B7ardVNbK1qZGtVE69XNbG1upHtNc3s3NvCzr0tPLmlb11j2RnMLpvCvIqC/YEahmlRbPB9phI9Giky3uIGhYnU4dVcAx1NB7+f7Py4IC0fMFz7LMvcH3adXd3sqGvpDdKt1Y1BmFY1Ud3YNuBuywtyOWVOKbd94PghA3s86dFIkVQQNyhMZH0GhUkUsNUHLutohvpmqN8xdPk9cot7wzRr6jHM+X83MmfhHM5a2Hezfa0dvBGG6NYwRHtO86sb2/j1S7uoKMxl5bJjo+97AlILU2QyOMhBYXqn/oPCZGTDkqtg6aeg4NAhd9vd7Ty3o46Lv/cUGWb89hNLmVtRkKSDHJkoLUwFpogk1t0NbfVBiDbugefuged/CjhkT4HTPwqnfywYaGYIn3nwRe7fuINzjp3KnX8zaCaNmyiBqaG+RSSxjIzgvtKyeTDrNFj+XfjwH2D+ecE11Me/Bncshqe/B50DX8MEuPF988nLzuSRl9/mmTcGezfixKbAFJHoph4DH7wPrnoYZpwSXCt9+DPwnZPgxQeCVmmirxXFWLF0LgBf+fUmurtT68y2hwJTRA7erNPg6kfg0p9BxQLYuw1+cS3cuRRe++/gmmk/K5bOpaIwlxcq6/nVSwf9NpoJQYEpIsNjBgvOD07TL1wFRdPh7Zfg3r+EH78fKvv2NUzJzeKm984H4OsP/x9tnV3jUesRUWCKyMhkZMI7LoePPQvv+3LwqOeb/wM/OAvuvxyqX+vd9OIlMzh6aiGVdS385A/bxrHSw6PAFJHRkR0Les1veAH+342QlQeb/wtWnQJrrod9b5GZYXz2/AUA/OvvXuO57YO8jXUCUmCKyOjKK4GzvwDXPwfvvDJY9scfw7dPhEdX8u6Z2SydX8G+1k4+8N0/cMmdT/HY5rdToiNI92GKSHJVvwa/+xJs+mXwOVZCy6k3sKrpz/jxhrdpCMcKPerQAq5dOpcLFx9BbtbYj4qkG9dFZOKofBYe/UJwfROgaBot7/o097aczuqndrArHOtzalEuV71rDh88ZeaYDtyhwBSRicUdXn8MHl0Ju18KlpUfTcd7Ps9/tS7mzife4JW3G4BgwOMPnjKTq941m8OL85JeNQWmiExM3d3w8i/gsVuDezgBpp+Mn72Sx9uO4s7Ht/LU1hogeNf9skXTWLF0LkcfNvRjmMOlwBSRia2zHZ69O3jMsmc80KPOgbO/wIsd07jzia385qVd9PQHvefoClYsncepc0tH/TUaCkwRSQ1tDfDUKvjDvwajKmGw6FJ4z+fY3lXOD57cygMbd9DaETx6uWh6Md+9/J1MKxm9U3UFpoiklsYqeOKfYeMPobsjGFn+pGvhjJuopZB7ntrGj596k9qmdi5ZMp2v/+WiUdu1RisSkdRSUAHnfx0+ugGOvyQYjf7pVfDtxZRuvIMblh7BvR86BYD1r1Qx1g0+BaaITDylc+Avvg9/9wQceXbwHvp1X4Zvv4MFOx5gWmEWexra2LRr35hWS4EpIhPX4SfA5f8Bf/srmPZOaHwbW3sTv7QbmWFvs/6VqjGtjgJTRCa+OWfAhx6DS+6B0nmUt1fyocy1PK7AFBFJwAyOWRZc4wQWZuzg2e111Ld0jFkVFJgikloODd48uTBzJ13d3Tz52kG8z32EFJgikloKD4NYCYXewKHsZf0re8Zs1wpMEUktZjA1aGUuyNjO+lfH7vYiBaaIpJ5DjwFgSd4uqhraePmtsbm9SIEpIqlnahCY7yoMTscff3VsessVmCKSesKOnyPZDjBm1zEVmCKSeg4N3gtU1LiVnAzn2W111Dcn//YiBaaIpJ5YMRTPwDpbuWB6M90O/7Ml+aflCkwRSU1hx895FcGbJ8fiMUkFpoikprDj58TYW0DQ8ZPsN08qMEUkNYUdP2XNr3N4cYyqMRi9KKmBaWbnmtkrZrbFzG5OsH6mma0zs+fM7EUzOz+Z9RGRNHLoQgDs7U2ceXQFkPze8qQFppllAquA84BjgMvM7Jh+m/0j8IC7vwO4FPhusuojImmmfD5kZEHtVk6dHryqYvOuhqTuMpktzJOBLe6+1d3bgfuAC/tt40BROF8MvJXE+ohIOsnKgbKjAOforCA63qxpSuoukxmY04AdcZ8rw2XxVgKXm1klsBb4WKKCzGyFmW00s41VVWM7/p2ITGBhx8/0tjcA2F7TnNTnyse70+cy4G53nw6cD9xjZgfUyd3vcvcl7r6koqJizCspIhNUeB1zSv2rFOZm0dDWSV0Sb2BPZmDuBGbEfZ4eLot3DfAAgLs/BcSA8iTWSUTSSdhTbns2MbMsH0juaXkyA3MDcJSZzTGzHIJOnTX9ttkOnAVgZgsJAlPn3CISTXhKzp5NzC6bAgSn5cmStMB0907go8AjwGaC3vCXzexWM1sWbnYTcK2ZvQD8DLjSU+1F6SIyfopnQk4BNL7N0UXtQHJbmFlJKxlw97UEnTnxy26Jm98EvCuZdRCRNJaREVzHrNzA8VmVQF5qtjBFRMZE2PEzuzsY6i1Vr2GKiCRf2PEzteV1ALbXqoUpIpJY2PGTV/cKOVkZVDe209jWmZRdKTBFJLX13FpUtZnZh8QA2Jak03IFpoiktillUDAV2hs5sTh4lnxbkjp+FJgikvrCjp/FubsABaaIyMDC0/L5GcHwFTolFxEZSNjxM61tK6AWpojIwML3+xzSuAVQC1NEZGAVCwAje+/rxDK62LWvlbbOrlHfjQJTRFJfTj6UzsG6Ozm1qAZ32FHbMuq7UWCKSHoIT8tPzt8NJOe0XIEpIulhatBTfmxWJZCcjh8Fpoikh7CFOadrG6AWpojIwMIWZkU4CMe2JAzCocAUkfRwyBzIzCWv+S0KaNYpuYjIgDKzoOJoAOZbJZV1zXR2dY/qLhSYIpI+wtPyk/N30dHl7KpvHdXiFZgikj7Cjp/FuW8Bo99TrsAUkfQRBuZRJOd1FQpMEUkfPYNwtL8J+Ki/rkKBKSLpo/BwiJUQ66znUPbyZrVamCIiiZn1dvwsyNiuFqaIyKDC65hH2w621TTj7qNWtAJTRNJL+LqKE7J30tLRRVVD26gVrcAUkfQSnpIvzNoJjO4jkgpMEUkvYQtzZtd2Muge1Y4fBaaIpJdYMRTPIMfbmW27R7XjR4EpIuknruPnzVF82keBKSLpJzwtX5Cxg+2j+LSPAlNE0k/Y8TNfLUwRkSGEp+QLMyqpb+lgb3P7qBSrwBSR9FM+HzKymGm7idE2aqMWKTBFJP1k5UDZUWTgHGU7R+1eTAWmiKSn3o6f7WwbpXsxFZgikp7Cod7mW6VamCIigzo06CkPBuFQC1NEZGBhC3NBxg51+oiIDKp4Jp5TwKG2l46GKprbO0dcpAJTRNJTRgZWsQCAozMqR+WZcgWmiKSvqX0HEx4pBaaIpK/ejp/to9Lxo8AUkfQ1yh0/SQ1MMzvXzF4xsy1mdvMA21xiZpvM7GUz+2ky6yMik8yh++/F3F7dOOLiskZcwgDMLBNYBbwXqAQ2mNkad98Ut81RwGeBd7l7nZkdmqz6iMgkNKWczrwKClqqaK3ZBpw2ouKS2cI8Gdji7lvdvR24D7iw3zbXAqvcvQ7A3fcksT4iMgllHBa0MosbXqW9s3tkZY1GhQYwDdgR97kyXBZvPjDfzH5vZk+b2blJrI+ITEIZU48DYD472Lm3ZWRljUaFRiALOAo4E7gM+L6ZlfTfyMxWmNlGM9tYVVU1xlUUkZQW1/Hz5gh7ypMZmDuBGXGfp4fL4lUCa9y9w93fAF4lCNA+3P0ud1/i7ksqKiqSVmERSUPhqEVH2w62j7CnPJmBuQE4yszmmFkOcCmwpt82DxG0LjGzcoJT9K1JrJOITDYVC3GMubaL7VV7R1RU0gLT3TuBjwKPAJuBB9z9ZTO71cyWhZs9AtSY2SZgHfApd69JVp1EZBLKyae5YCbZ1kXH26+MqKik3VYE4O5rgbX9lt0SN+/AjeEkIpIUXeULoXEbubX/N6JyxrvTR0Qk6WLTjgegvPl1urt92OVEbmGa2TRgVvx33P2JYe9ZRGSM5BwRPFN+pG9n975WjijJG1Y5kQLTzL4G/BWwCegKFzugwBSRia9nEI7w1qKkBiawHDja3duGtRcRkfFUOpcOy2E61Ty9ew/MKx9WMVGvYW4Fsoe1BxGR8ZaZRV3+HABadv5p2MVEbWE2A8+b2WNAbyvT3a8f9p5FRMZQa+kCaHqFzOpNQ288gKiBuYYDbzoXEUkZmYcdAzt+SWH9q8MuI1JguvuPw6d15oeLXnH3jmHvVURkjBXNWgQb4LDWN3B3zOygy4h0DdPMzgReIxjf8rvAq2a29KD3JiIyTgpmnADAkWynpnF4/ddRO32+AbzP3d/t7kuBc4BvDWuPIiLjwIqOoMEKOMQaeavyzWGVETUws9299yFMd38V9ZqLSCoxY3dsLgD7tj0/rCKiBuZGM/uBmZ0ZTt8HNg5rjyIi46SxKBg9snv3y8P6ftTA/DDBUz7Xh9OmcJmISMrorggGE47VDm/Uoqi95G3AN8NJRCQlxaYfD3+Cgta3hvX9QQPTzB5w90vM7CWCZ8f7cPcThrVXEZFxMG/RGVTP+CMLD58zrO8P1cK8Ifzzz4dVuojIBBLLyyc2bd6wvz/oNUx33xXOVgM73H0bkAssAobXphURSVFRO32eAGLhmJi/Bf4GuDtZlRIRmYiiBqa5ezNwEfBdd78YODZ51RIRmXgiB6aZnQb8NfDrcFlmcqokIjIxRQ3MjwOfBf4zfPPjXIK3PIqITBpR78N8HHg87vNWghvYRUQmjaHuw/wXd/+4mf0Xie/DXJbgayIiaWmoFuY94Z+3J7siIiIT3aCB6e7PhrMbgRZ37wYws0yC+zFFRCaNqJ0+jwH5cZ/zgEdHvzoiIhNX1MCMuXtjz4dwPn+Q7UVE0k7UwGwysxN7PpjZO4GW5FRJRGRiivrWyI8DPzeztwADDgP+Kmm1EhGZgKLeh7nBzBYAR4eL9NZIEZl0or41Mh/4DHCDu/8JmG1mGvJNRCaVqNcwfwS0A6eFn3cCX05KjUREJqiogTnP3b8OdACEIxcd/FvQRURSWNTAbDezPMLHI81sHjC8N6GLiKSoqL3kXwAeBmaY2b3Au4Ark1UpEZGJaMjANDMD/o9g8OBTCU7Fb3D36iTXTURkQhkyMN3dzWytux/P/sGDRUQmnajXMP9oZicltSYiIhNc1GuYpwCXm9mbQBPBabnrveQiMplEDcxzkloLEZEUMNSI6zHg74EjgZeA1e7eORYVExGZaIa6hvljYAlBWJ4HfCPpNRIRmaCGOiU/Juwdx8xWA88kv0oiIhPTUC3M3hGJhnMqbmbnmtkrZrbFzG4eZLu/MDM3syUHuw8RkbEyVAtzkZntC+cNyAs/9/SSFw30xfC9P6uA9wKVwAYzW+Pum/ptVwjcAPzvMI9BRGRMDNrCdPdMdy8Kp0J3z4qbHzAsQycDW9x9q7u3A/cBFybY7kvA14DWYR2BiMgYiXrj+nBMA3bEfa4Ml/UKX3sxw931BJGITHjJDMxBmVkG8E3gpgjbrjCzjWa2saqqKvmVExFJIJmBuROYEfd5erisRyFwHLA+fILoVGBNoo4fd7/L3Ze4+5KKiookVllEZGDJDMwNwFFmNsfMcoBLgTU9K9293t3L3X22u88GngaWufvGJNZJRGTYkhaY4W1IHwUeATYDD7j7y2Z2q5ktS9Z+RUSSJeqz5MPi7muBtf2W3TLAtmcmsy4iIiM1bp0+IiKpRoEpIhKRAlNEJCIFpohIRApMEZGIFJgiIhEpMEVEIlJgiohEpMAUEYlIgSkiEpECU0QkIgWmiEhECkwRkYgUmCIiESkwRUQiUmCKiESkwBQRiUiBKSISkQJTRCQiBaaISEQKTBGRiBSYIiIRKTBFRCJSYIqIRKTAFBGJSIEpIhKRAlNEJCIFpohIRApMEZGIFJgiIhEpMEVEIlJgiohEpMAUEYlIgSkiEpECU0QkIgWmiEhECkwRkYgUmCIiESkwRUQiUmCKiESkwBQRiUiBKSISkQJTRCQiBaaISERJDUwzO9fMXjGzLWZ2c4L1N5rZJjN70cweM7NZyayPiMhIJC0wzSwTWAWcBxwDXGZmx/Tb7DlgibufADwIfD1Z9RERGalktjBPBra4+1Z3bwfuAy6M38Dd17l7c/jxaWB6EusjIjIiyQzMacCOuM+V4bKBXAP8JtEKM1thZhvNbGNVVdUoVlFEJLoJ0eljZpcDS4B/TrTe3e9y9yXuvqSiomJsKyciEspKYtk7gRlxn6eHy/ows7OBfwDe7e5tSayPiMiIJLOFuQE4yszmmFkOcCmwJn4DM3sHcCewzN33JLEuIiIjlrTAdMDogMwAAAzCSURBVPdO4KPAI8Bm4AF3f9nMbjWzZeFm/wwUAD83s+fNbM0AxYmIjLtknpLj7muBtf2W3RI3f3Yy9y8iMpomRKePiEgqUGCKiESkwBQRiUiBKSISkQJTRCQiBaaISEQKTBGRiBSYIiIRKTBFRCJSYIqIRKTAFBGJSIEpIhKRAlNEJCIFpohIREkd3m2sdHR0UFlZSWtr63hXJS3FYjGmT59Odnb2eFdFZFylRWBWVlZSWFjI7NmzMbPxrk5acXdqamqorKxkzpw5410dkXGVFqfkra2tlJWVKSyTwMwoKytT612ENAlMQGGZRPrZigTSJjDHW0FBQe/82rVrmT9/Ptu2bWPlypXk5+ezZ8+ehNuaGTfddFPv59tvv52VK1cm3MdDDz3ErbfeCsCVV17Jgw8+eND1XL9+PcXFxSxevJjFixf3ltfe3s7SpUvp7Ow86DJFJgsF5ih77LHHuP766/nNb37DrFmzACgvL+cb3/hGwu1zc3P5xS9+QXV19ZBlf/3rX+e6664bcR3POOMMnn/+eZ5//nluuSV4xVJOTg5nnXUW999//4jLF0lXCsxR9MQTT3Dttdfyq1/9innz5vUuv/rqq7n//vupra094DtZWVmsWLGCb33rW4OW/eqrr5Kbm0t5efkB6z7/+c9z5ZVX0tXVNaL6L1++nHvvvXdEZYiks7ToJY83++ZfJ6XcN796waDr29raWL58OevXr2fBggV91hUUFHD11Vdzxx138MUvfvGA737kIx/hhBNO4NOf/vSA5f/+97/nxBNPPGD5pz71KRoaGvjRj36EmfGJT3yCdevWHbDdpZdeys033wzAU089xaJFizjiiCO4/fbbOfbYYwE47rjj2LBhw6DHKTKZpV1gjpfs7GxOP/10Vq9ezR133HHA+uuvv57FixfzyU9+8oB1RUVFXHHFFXz7298mLy8vYfm7du2ioqKiz7IvfelLnHLKKdx11129y4ZqqZ544ols27aNgoIC1q5dy/Lly3nttdcAyMzMJCcnh4aGBgoLC4c8ZpHJJu0Cc6iWYLJkZGTwwAMPcNZZZ3Hbbbfxuc99rs/6kpISPvjBD7Jq1aqE3//4xz/OiSeeyFVXXZVwfV5eHvX19X2WnXTSSTz77LPU1tZSWloKMGQLs6ioqHfZ+eefz3XXXUd1dXXvqX5bWxuxWCz6gYtMImkXmOMpPz+fX//615xxxhlMnTqVa665ps/6G2+8kZNOOilhT3RpaSmXXHIJq1ev5uqrrz5g/cKFC/n3f//3PsvOPfdczjnnHC644AJ++9vfUlhYOGQLc/fu3UydOhUz45lnnqG7u5uysjIAampqKC8v1xM9IgNQp88oKy0t5eGHH+bLX/4ya9as6bOuvLycD3zgA7S1tSX87k033TRgb/nSpUt57rnncPc+yy+++GKuvfZali1bRktLy5D1e/DBBznuuONYtGgR119/Pffdd1/vfZbr1q3jggvGp4Uukgqs/z/AiW7JkiW+cePGPss2b97MwoULx6lGY+eGG27g/e9/P2effXZSyr/ooov46le/yvz58w9YN1l+xjJ5mdmz7r5ksG3Uwkwhn/vc52hubk5K2e3t7SxfvjxhWIpIQIGZQqZOncqyZcuSUnZOTg5XXHFFUsoWSRcKTBGRiBSYIiIRKTBFRCJSYIqIRKTAFBGJSIE5SlJlPMx7772XE044geOPP57TTz+dF154AdB4mCJRKDBH2UQfD3POnDk8/vjjvPTSS3z+859nxYoVgMbDFIki/Z4lX1mcpHLrh9ykZzzMtWvXHjAe5t13381nPvOZ3kEyesSPh/mVr3xlwLKHGg9zx44drF69mszMzEHrePrpp/fOn3rqqVRWVvZ+Xr58OZ/97Gf567/+6yGPVWQyUgtzlPSMh/nQQw8NOh5mIh/5yEe49957DxiNKN5g42FWVVXxox/9iMzMTD7xiU/0vn4ifvrqV796wHdXr17Neeed1/tZ42GKDC4NW5hDtwSTIVXGw+yxbt06Vq9ezZNPPtm7TONhigxOLcxR0jMe5jPPPMNtt912wPoo42GuXr2apqamhOvz8vIOeNVt/HiYPaK0MF988UU+9KEP8ctf/rJ3aLceGg9TZGDp18IcR6kwHub27du56KKLuOeeew4YaEPjYYoMTi3MUTbRx8O89dZbqamp4brrrmPx4sUsWbJ/NCuNhykyOI2HmUI0HqZI8oz7eJhmdq6ZvWJmW8zs5gTrc83s/nD9/5rZ7GTWJ9VpPEyR8ZW0wDSzTGAVcB5wDHCZmR3Tb7NrgDp3PxL4FvC1ZNUnHWg8TJHxlcwW5snAFnff6u7twH3Ahf22uRD4cTj/IHCW9bxg5iCl2qWFVKKfrUggmYE5DdgR97kyXJZwG3fvBOqBMg5SLBajpqZG/7CTwN2pqanRrUYipMhtRWa2AlgBMHPmzAPWT58+ncrKSqqqqsa6apNCLBZj+vTp410NkXGXzMDcCcyI+zw9XJZom0ozywKKgZr+Bbn7XcBdEPSS91+fnZ3NnDlzRqnaIiKJJfOUfANwlJnNMbMc4FJgTb9t1gB/G87/JfA713m1iExQSWthununmX0UeATIBH7o7i+b2a3ARndfA6wG7jGzLUAtQaiKiExISb2G6e5rgbX9lt0SN98KXJzMOoiIjJaUe9LHzKqAbf0WlwNDj8CbenRcqSMdjwkm13HNcveKRBv3SLnATMTMNg71SFMq0nGljnQ8JtBx9afBN0REIlJgiohElC6BedfQm6QkHVfqSMdjAh1XH2lxDVNEZCykSwtTRCTpUiow03V8zQjHdaOZbTKzF83sMTObNR71PBhDHVPcdn9hZm5mKdETG+W4zOyS8Pf1spn9dKzrOBwR/g7ONLN1ZvZc+Pfw/PGo58Ewsx+a2R4z+9MA683Mvh0e84tmduBrWftz95SYCJ4Weh2YC+QALwDH9NvmOuB74fylwP3jXe9ROq73APnh/Icn+nFFOaZwu0LgCeBpYMl413uUfldHAc8Bh4SfDx3veo/Scd0FfDicPwZ4c7zrHeG4lgInAn8aYP35wG8AA04F/neoMlOphTmm42uOoSGPy93XuXvPUOtPEwxkMpFF+V0BfIlg0OjWBOsmoijHdS2wyt3rANx9zxjXcTiiHJcDReF8MfDWGNZvWNz9CYJHrgdyIfATDzwNlJjZ4YOVmUqBOWbja46xKMcV7xqC/xUnsiGPKTz9meHuvx7Lio1QlN/VfGC+mf3ezJ42s3PHrHbDF+W4VgKXm1klwePOHxubqiXVwf7bS43xMCVgZpcDS4B3j3ddRsLMMoBvAleOc1WSIYvgtPxMgjOBJ8zseHffO661GrnLgLvd/RtmdhrBoDnHuXv3eFdsLKVSC/NgxtdksPE1J5gox4WZnQ38A7DM3RO/p3fiGOqYCoHjgPVm9ibB9aM1KdDxE+V3VQmscfcOd38DeJUgQCeyKMd1DfAAgLs/BcQInsdOZZH+7cVLpcBM1/E1hzwuM3sHcCdBWKbCNbFBj8nd69293N1nu/tsguuyy9x9Y+LiJowofwcfImhdYmblBKfoW8eyksMQ5bi2A2cBmNlCgsBM9VccrAGuCHvLTwXq3X3XoN8Y756sg+z1Op/gf+zXgX8Il91K8I8Ngl/iz4EtwDPA3PGu8ygd16PA28Dz4bRmvOs80mPqt+16UqCXPOLvygguN2wCXgIuHe86j9JxHQP8nqAH/XngfeNd5wjH9DNgF9BB0PK/Bvh74O/jflerwmN+KcrfQT3pIyISUSqdkouIjCsFpohIRApMEZGIFJgiIhEpMEVEIlJgyoRnZl1m9ryZ/cnM/svMSka5/CvN7Dvh/Eoz++Roli/pQ4EpqaDF3Re7+3EEgyl8ZLwrJJOTAlNSzVPEDZBgZp8ysw3heIZfjFt+RbjsBTO7J1z2/nCc1OfM7FEzmzoO9ZcUpsE3JGWYWSbB43mrw8/vI3hO+2SCpzbWmNlSgvED/hE43d2rzaw0LOJJ4FR3dzP7EPBp4KYxPgxJYQpMSQV5ZvY8QctyM/Df4fL3hdNz4ecCggBdBPzc3asB3L1nTMTpwP3hmIc5wBtjU31JFzoll1TQ4u6LgVkELcmea5gG/FN4fXOxux/p7qsHKedfge+4+/HA3xGMPSASmQJTUoYHo85fD9wUDt/3CHC1mRUAmNk0MzsU+B1wsZmVhct7TsmL2T98198icpB0Si4pxd2fM7MXgcvc/Z5wqLGnwjeRNAKXu/vLZvYV4HEz6yI4Zb+SYNTwn5tZHUGozhmPY5DUpdGKREQi0im5iEhECkwRkYgUmCIiESkwRUQiUmCKiESkwBQRiUiBKSISkQJTRCSi/w/82n2Svw04CgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "precisions, recalls, thresholds = precision_recall_curve(y_test, knn_probs)\n",
    "knn2_precisions, knn2_recalls, knn2_thresholds = precision_recall_curve(y_test, \n",
    "                                                                        knn2_probs)\n",
    "\n",
    "def plot_pr(recall, precision, label=None):\n",
    "    plt.plot(recall, precision, linewidth=2, label=label)\n",
    "    plt.xlabel('Recall')\n",
    "    plt.ylabel('Precision')\n",
    "    plt.xlim((-0.02,1.02))\n",
    "    plt.ylim((-0.02,1.02))\n",
    "    plt.title('Precision/Recall Curve')\n",
    "    \n",
    "plt.figure(figsize=(5,5))\n",
    "plt.plot(recalls, precisions, linewidth=2, label=\"KNN (k=5)\")\n",
    "plot_pr(knn2_recalls, knn2_precisions, label=\"KNN (k=2)\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In contrast to the ROC curve, models that stay as close as possible to the top-right corner of the P/R curve perform better (=high precision, high recall). A measure for the area under the P/R curve is given by [the average precision score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.928148107878731"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import average_precision_score\n",
    "average_precision_score(y_test, knn_probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9022666968402592"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_precision_score(y_test, knn2_probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The P/R curve considers, for this data, `knn` with $k=5$ a better classifier. Similar to what we have seen for the ROC curve, we can select an optimal threshold (based on our application, of course) also with the P/R curve. Let's say we decide to have no less than 90% precision: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_threshold = thresholds[np.min(np.where(precisions >= 0.9))]\n",
    "new_threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9038461538461539"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precisions[np.min(np.where(precisions >= 0.9))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This corresponds to a recall of about 0.89 which, depending on the application, can be considered a reasonable level (otherwise the level of precision must be adjusted accordingly):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8867924528301887"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recalls[np.min(np.where(precisions >= 0.9))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUwAAAFNCAYAAABrMlb6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXwX1b3/8dcne0ICmARR2VVwQ0ENuFCVXndtkdq6VX8WsXpba9WqbbntrUW7t3axlmptrVpr69beXu4VlysFbK2ooaKtooiIEMSaBARCyP75/TGT8E3IMiSZJN9v3s/HIw/nOzPfM+cQeHtmzswZc3dERKRraf1dARGRZKHAFBGJSIEpIhKRAlNEJCIFpohIRApMEZGIFJgiIhEpMKVHzOxVM5vZxT5jzazKzNL7qFqxMrOZZlaW8HmdmZ3Sn3WSvqHATFHhP+KdYVD9y8zuNbP83j6Oux/m7ku72Ge9u+e7e2NvHNPM9msOrDbtfC+udvaEmU03s0Vm9oGZbTazF8zssv6ul+w5BWZq+6i75wNHASXAf7bdwQLJ9vfgLOCJhM/N7ZwKHAn8R7/Uqh1mdhzwZ2AZcCBQBHwWOLOb5aVELz1ZJds/FOkGd98IPA5MBjCzpWb2LTN7FqgG9jezYWZ2t5ltMrONZvbNxH+cZnaFma0ys+1m9pqZHRWubzkdDXtSpWa2LezV/ihcP97M3Mwyws/7mdnCsLe1xsyuSDjOfDN72Mx+Ex7rVTMradOks4BF7bTzPeBJguBsLi/bzG41s/Vhne40s9yE7eeY2cqwzm+Z2Rnh+ssS2rvWzP69m3/8PwDuc/fvuXuFB1a4+/nhceaY2V8TvxD+WR0YLt9rZneEPdQdwI1hTzrxd/MxM3slXE4zs3lhWyrDP8vCbtZd2lBgDgJmNoYgZF5KWP3/gCuBAuAd4F6ggaAXdCRwGvDp8PvnAfOBS4GhwCygsp1D3Qbc5u5DgQOAhzuo0oNAGbAf8Ang22b2bwnbZ4X7DAcWAj9LaEsmcCLwf+20czRBz21NwurvApMIQvRAYBRwU7j/dOA3wBfDY50IrAu/9z7wkbC9lwE/bv6fRFRmlgccBzy6J99rxyeBbxH8rm4DdgD/1mb778LlzwOzgZMI/ny3AAt6eHxp5u76ScEfgn/4VcAHBIH4cyA33LYUuCVh35FAbfP2cN1FwJJw+Ung2k6Oc0q4/AxwM1DcZp/xgAMZwBigEShI2P4d4N5weT7wdMK2Q4GdCZ9PBha3087t4TEWA8PDbUYQLgck7H8c8Ha4/AvgxxH/PP/U/GcAzATK2vszaPOdUWGdDu6k3DnAX9usc+DAcPle4Ddttn8T+HW4XBC2cVz4eRVwcsK++wL1QEZ//51MhR/1MFPbbHcf7u7j3P0qd9+ZsG1DwvI4IBPYFA5MfEAQJnuH28cAb0U43uUEvbnXzexFM/tIO/vsB2x29+0J694hCJdm7yUsVwM5zafztH86PtvdCwiC7GCgOFw/AsgDViS064lwfaftMrMzzWx5eNngg/C4xe3t24ktQBNBaPXEhjaffweca2bZwLnA3939nXDbOOC/Etq7iuB/UCN7WAdBp+SDWeK8fhsIepjFYcAOd/eh7n5YwvYDuizQ/U13v4ggaL8HPGpmQ9rs9i5QaGYFCevGAhsj1rvd65fh8ZcR9MhuDVdVADuBwxLaNcyDASLooF1hEP0hLGekuw8Pj2kR69hcn2rgOeDjney2gyDUm4+9T3tFtSn3NYL/yZxJ69NxCNp0ZkJ7h7t7jgfXsaWHFJiCu28CngJ+aGZDw4GDA8zspHCXXxEMNhwdjqofaGbj2pZjZpeY2Qh3byK4FABBDyvxWBuAvwHfMbMcMzuCoGf6267qaWYTgGx3X9XJbj8BTjWzKWE9fklw/XHvsIxRZnZ6uO/dwGVmdnLY5lFmdjCQBWQD5UCDmZ1JcE23O74EzDGzL5pZUViHKWb2YLj9ZeAwM5tqZjkElySi+B1wLcF110cS1t8JfKv592NmI8zsnG7WXdpQYEqzSwmC4jWCU8lHCU8l3f0RgkGH3xFcK/wT0N7I6xnAq2ZWRTA4cWGbywDNLiK4rvku8F/A19396Qh1PJsOepfN3L2cYCDnpnDVlwkGgZab2TbgaeCgcN8XCAd0gK0Et/6MCy8XXEMwaLWFoBe3MEL92qvP3wgGaP4NWGtmm4G7mtvh7quBW8J6vQn8tYOi2vo9wcDOn929ImH9bWFdnzKz7cBy4Jju1F12Z+GFYZEBz8wWAT9z905DUyQu6mFKMlkKLOnvSsjgpR6miEhE6mGKiESkwBQRiSij610GluLiYh8/fnx/V0NEUsyKFSsq3H1EZ/vEFphm9muCZ3Hfd/fJ7Ww3glsgziJ4mmOOu/+9q3LHjx9PaWlpb1dXRAY5M3unq33iPCW/l+C+vI6cCUwMf64E7oixLiIiPRZbYLr7M8DmTnY5h2BSAXf35cBwM+vpM7ciIrHpz2uYo2g9qUBZuG7TnhRyz7Nv8z8vv0tBTiYFORkU5GQyNCejZbn1fzMYGi7nZ2eQka4xLxGJLikGfczsSoLTdsaOHdtq29ryHfx9/Qftfa1LeVnp7Qbr0Obl7PaDd2hCACt0RQaP/gzMjQTTazUbTQcz1rj7XQTP31JSUtLqTvvPzDyAWVP3Y3tNPdtrGthW09CyvOu/rZe31dRTVdtAdV0j1XWN/GtbbbcbkZuZ3hKe7QVqZ73c5nWZCl2RpNCfgbkQuDqcteUYYGs4a84eGTU8l1HDc7vesY2mJmdHXcNugbqt3ZBtHbbN66pqG9hZ38jO+kbe39790M3JTGv/kkL27mHbUQhnZ+hVLyJxi/O2ot8TTOhabMEb/r5OMEkt7n4nwWwtZxHMJFNNMGtMn0lLszBwMrtdhruzo66xVbBu6yRsE7dX1e5aX1PfRE19LeU9CN2sjLRdlxI6Cdy2vdvE5ZxMha5IZ2ILzHAi2c62O/C5uI7fF8yM/OxgAGnfYd0rw93ZWd8YKXC3dXKpoa6hiYqqOiqq6rrdnqz0tHZ6sa17tF0NqGVnpBHcYiuSepJi0CeVmRl5WRnkZWUwcmhOt8pwd2rqmxICt/3LCruFcW3r/eoam6jcUUflju6Hbma6ddrLHdpB2O4K40xyMhW6MjApMFOAmZGblU5uVjp7D+1eGe5ObUNTp9dwowyo1TU2sXlHHZt7ELoZadZhL7ezAbXEMM7NTFfoSq9TYAoQhG5OZjo5mensXdD1/h2pSbi8EGlArXb3a7x1DU1sqa5nS3V9t+uR3hK6rXu5nV1SSOzlFuRkkJel0JXWFJjSq5pDd0RBdrfLqG1obLeX29WAWmIvuLahiQ+q6/mgup7gPWh7Lj0tuEbd8SWFznu5BTmZDFHophQFpgw42RnpZOenU5zf/dCta2jq4A6Fjnu5bS851NQ3sXVnPVt3dj9004wwdNsL2WgDavlZGaSlKXQHAgWmpKSsjDSK8rMp6kHo1jc2dfse3eZ1O+sb2RaGcXeZQX5W52Hb1YBaQbZCtzcoMEU6kJmeRuGQLAqHZHW7jPrGJqo6DNTmHm7nlxyq6xqDfWobYGtNt+uy6/JC62AdWZDN/iPy2X/EEPYvHsKIgmxdRuiAAlMkRpnpaew1JIu9ehC6DY1NVNW2Dd09u4thR10jVbUNVNU2sGlr58cryM5gQhieu4I0nwnFQ8jNGtwPNygwRQa4jPQ0hudlMTyv+6Hb2ORUtdfLra3n3Q9qeKu8ircrdrC2fAdbd9bzStlWXinbPVlHDc9lQvGQlt5oc6DuNyx3UJzyKzBFBoH0NGNYXibD8jp/FNjd2byjjrUVO1hbXsXa8h28Vb6DtRVVrK+sZuMHO9n4wU7+uqai1fdyMtMYXzSEA0bk7wrUMEyH9uDx44FGgSkiLcysZbBs2vjCVtsaGpvYsGVnS5CuragKwrR8BxVVtbz+3nZef2/7bmUW52dzzIRCvv2xw7sM7IFOgSkikWSkpzGheAgTiodw8iGtt22rqeftMETXhiHafJpfUVXLY//YxIiCbObPOqx/Kt9LFJgi0mNDczKZMmY4U8YMb7W+qcl5acMWzrvzOX67/B0uPW4c+4/I76da9pxmrhWR2KSlGUePK+S8o8fQ0OR874nX+7tKPaLAFJHYXX/aJHIz03ny1X/xwtudvRtxYNMpuYjEbuTQHK48cX/uXfwST/3x15QcUk5a3XbILoAJJ8HYYyGvsOuC+pl6mCLSJ/59snNT7iMcuOUZ3tpSD0NHQ3o2vP4YLL4FKt/q7yp2SYEpIvGr3kxe6R1MHrsP71HEX96uosEdMnNh2GjIyIW/3Q7VA/t0XYEpIvFbvxwa65k4bj+KhmSzraaelzckPEmUMxQa62DD8/1XxwgUmCISv7eXQV4RaWacMLEYgBfWVbJpa8K0eXnFwX4DmAJTROJXux0ygndWjSvKY1zhEGobmniodAOPrNjA2ooqPD0LanZ/Umgg0Si5iMQvuwAaaiAzF8M48/B9KF23hVc2ftDyfPo+uc6Ro/MZ39BIdsbAnBVJPUwRid+Ek6C6suVjTkY6HzqwmMtnTOCEA0eQn52B7dzM994YyYnfX8Kdy95iW0333+kUFwWmiMRv7LGQngk121qtzs5I5+hxe3HZ0UVMO2Ak24qO4l/bavnu469z/Hf+zLcXrWp9nbOfmbv3dx32SElJiZeWlvZ3NURkT1W+Fdw61FgXDPBkZENDLVRXQHoWHP95vHB/lq0u5xfL1vLc2qBHmpluzJoyiitP3J+D9unBK027YGYr3L2k030UmCLSZ6o3B7cOvb0sGODJCZ/0GXPMbk/6vFL2Ab94Zi2P/2MTTWFMffigEVx54gEcu39hr79GQ4EpIklvfWU1v/rrWh4u3UBNfRMAU0YP4+eXHM2o4bm9dpwogalrmCIyoI0tyuOWcybzt3kn84VTJlE4JIuXy7Zy29Or+7wuCkwRSQqFQ7K49pSJPPDpYwBY+kY5fX2GrMAUkaRy8D4F7DM0h/e31/Lapm1df6EXKTBFJKmYGSdNGgEEvcy+pMAUkaTz4YODwFymwBQR6dyMA4vJSDNWrN/C1p1990SQAlNEkk5BTiZHj9uLxibnr29WdP2FXqLAFJGkNPOgvQFY+sb7fXZMBaaIJKXm65hLV/fd7UUKTBFJSgeNDG4vKt9ey6vv9s3tRQpMEUlKZsbMg8LR8tV9M1quwBSRpNUcmH11HVOBKSJJq+X2one2sLU6/tuLFJgikrQKcjIpGb8XTQ5/WRP/abkCU0SS2q7bixSYIiKdShz4aWqK9/YiBaaIJLWDRhaw77Dg9qK4Zy+KNTDN7Awze8PM1pjZvHa2jzWzJWb2kpm9YmZnxVkfEUk9ibcXxT1aHltgmlk6sAA4EzgUuMjMDm2z238CD7v7kcCFwM/jqo+IpK5p44P3Aa3atD3W48TZw5wOrHH3te5eBzwInNNmHweGhsvDgHdjrI+IpKjxxUMAWFe5I9bjxBmYo4ANCZ/LwnWJ5gOXmFkZsAj4fHsFmdmVZlZqZqXl5X07/52IDHzjCvOA4IVpcT5X3t+DPhcB97r7aOAs4H4z261O7n6Xu5e4e8mIESP6vJIiMrAVDsmiIDuD7bUNbInxBvY4A3MjMCbh8+hwXaLLgYcB3P05IAcojrFOIpKCzIyxRUEvM87T8jgD80VgoplNMLMsgkGdhW32WQ+cDGBmhxAEps65RWSPjS8KrmOur6yO7RixBaa7NwBXA08CqwhGw181s1vMbFa42w3AFWb2MvB7YI739XszRSQl9EUPMyO2kgF3X0QwmJO47qaE5deAGXHWQUQGh/FFuwZ+4tLfgz4iIr1ibGH8txYpMEUkJYwvDnuYm9XDFBHp1MiCHLIy0qioqqOqtiGWYygwRSQlpKVZyw3s78R0Wq7AFJGUMa6oOTDjOS1XYIpIymge+FFgioh0oXngR6fkIiJdGFuoU3IRkUiaH49UD1NEpAuj9solPc3YtK2G2obGXi9fgSkiKSMzPY1Rw3Nxhw2bd/Z6+QpMEUkpu24t6v3TcgWmiKSUOO/FVGCKSEoZVxjfwI8CU0RSSksPM4ZJOBSYIpJSxhXF97SPAlNEUkrzzetlW6ppaGzq1bIVmCKSUnKz0hk5NJv6RmfT1ppeLVuBKSIpJ67TcgWmiKSc5nkxe/t1FQpMEUk544vDV+728ki5AlNEUk7zwM+6CvUwRUQ61TxrkXqYIiJdGJvweKS791q5CkwRSTnDcjMZnpfJzvpGyrfX9lq5CkwRSUkttxb14mm5AlNEUtK4GAZ+FJgikpLGh9cxe3PgR4EpIilpbHhKvq4Xn/ZRYIpISmrpYfbi0z4KTBFJSc23FqmHKSLShRH52eRlpbN1Zz0fVNf1SpkKTBFJSWbW8ohkb81apMAUkZQ1vpfvxVRgikjKanm/Ty/di6nAFJGU1dtP+ygwRSRl7XpHuXqYIiKdGlekQR8RkUj2HZZLZrrx/vZaqusaelyeAlNEUlZ6mjGmsPeeKVdgikhKG9eL92IqMEUkpe165W7PB34UmCKS0npz4CfWwDSzM8zsDTNbY2bzOtjnfDN7zcxeNbPfxVkfERl8ejMwM3pcQgfMLB1YAJwKlAEvmtlCd38tYZ+JwH8AM9x9i5ntHVd9RGRw2nXz+sA+JZ8OrHH3te5eBzwInNNmnyuABe6+BcDd34+xPiIyCI3eKxcz2LhlJ3UNTT0qK87AHAVsSPhcFq5LNAmYZGbPmtlyMzsjxvqIyCCUnZHOfsNyaXLY+MHOHpXV34M+GcBEYCZwEfBLMxvediczu9LMSs2stLy8vI+rKCLJblzLZMI9Oy2PMzA3AmMSPo8O1yUqAxa6e727vw2sJgjQVtz9LncvcfeSESNGxFZhEUlNzdcx1/dw4CfOwHwRmGhmE8wsC7gQWNhmnz8R9C4xs2KCU/S1MdZJRAahAd/DdPcG4GrgSWAV8LC7v2pmt5jZrHC3J4FKM3sNWAJ80d0r46qTiAxOu16I1rMeZmy3FQG4+yJgUZt1NyUsO3B9+CMiEouxhc2v3B2gPUwRkYGi+ZR8w5adNDV5t8uJ3MM0s1HAuMTvuPsz3T6yiEgfGZKdQXF+NhVVtby3rYb9hud2q5xIgWlm3wMuAF4DGsPVDigwRSQpjC/Ko6KqlnWVO+INTGA2cJC713brKCIi/WxsUR6l72xhfWU1xx/QvTKiXsNcC2R27xAiIv2v+ZW763owUh61h1kNrDSzxUBLL9Pdr+n2kUVE+lDzwM/6HkzCETUwF7L7TeciIkmj+WmfdRUx9zDd/b7waZ1J4ao33L2+20cVEelj4xLe7ePumNkelxHpGqaZzQTeJJjf8ufAajM7cY+PJiLST4bnZVKQk0FVbQOVO+q6VUbUQZ8fAqe5+0nufiJwOvDjbh1RRKQfmFnLwE93Z1+PGpiZ7v5G8wd3X41GzUUkyYxteV1F9wZ+og76lJrZr4Dfhp8vBkq7dUQRkX4yvofv94kamJ8FPgc030b0F4JrmSIiSWNcYc9euRt1lLwW+FH4IyKSlJrvxXx3a023vt9pYJrZw+5+vpn9g+DZ8Vbc/YhuHVVEpB9MGTOcF75yMiMKsrv1/a56mNeG//1It0oXERlAcjLTyclM7/b3Ox0ld/dN4WIFsMHd3wGygSnAu90+qohIEop6W9EzQE44J+ZTwP8D7o2rUiIiA1HUwDR3rwbOBX7u7ucBh8VXLRGRgSdyYJrZcQT3Xz4Wruv+hQARkSQUNTCvA/4D+K/wzY/7E7zlUURk0Ih6H+YyYFnC57XsuoldRGRQ6Oo+zJ+4+3Vm9j+0fx/mrHa+JiKSkrrqYd4f/vfWuCsiIjLQdRqY7r4iXCwFdrp7E4CZpRPcjykiMmhEHfRZDOQlfM4Fnu796oiIDFxRAzPH3auaP4TLeZ3sLyKScqIG5g4zO6r5g5kdDeyMp0oiIgNT1PkwrwMeMbN3AQP2AS6IrVYiIgNQ1PswXzSzg4GDwlV6a6SIDDpR3xqZB3wZuNbd/wmMNzNN+SYig0rUa5j3AHXAceHnjcA3Y6mRiMgAFTUwD3D37wP1AOHMRXv+FnQRkSQWNTDrzCyX8PFIMzsAqI2tViIiA1DUUfKvA08AY8zsAWAGMCeuSomIDERdBqaZGfA6weTBxxKcil/r7hUx101EZEDpMjDd3c1skbsfzq7Jg0VEBp2o1zD/bmbTYq2JiMgAF/Ua5jHAJWa2DthBcFruei+5iAwmUQPz9FhrISKSBLqacT0H+AxwIPAP4G53b+iLiomIDDRdXcO8DyghCMszgR/GXiMRkQGqq1PyQ8PRcczsbuCF+KskIjIwddXDbJmRqDun4mZ2hpm9YWZrzGxeJ/t93MzczEr29BgiIn2lqx7mFDPbFi4bkBt+bh4lH9rRF8P3/iwATgXKgBfNbKG7v9ZmvwLgWuD5brZBRKRPdNrDdPd0dx8a/hS4e0bCcodhGZoOrHH3te5eBzwInNPOft8AvgfUdKsFIiJ9JOqN690xCtiQ8LksXNcifO3FGHfXE0QiMuDFGZidMrM04EfADRH2vdLMSs2stLy8PP7KiYi0I87A3AiMSfg8OlzXrACYDCwNnyA6FljY3sCPu9/l7iXuXjJixIgYqywi0rE4A/NFYKKZTTCzLOBCYGHzRnff6u7F7j7e3ccDy4FZ7l4aY51ERLottsAMb0O6GngSWAU87O6vmtktZjYrruOKiMQl6rPk3eLui4BFbdbd1MG+M+Osi4hIT/XboI+ISLJRYIqIRKTAFBGJSIEpIhKRAlNEJCIFpohIRApMEZGIFJgiIhEpMEVEIlJgiohEpMAUEYlIgSkiEpECU0QkIgWmiEhECkwRkYgUmCIiESkwRUQiUmCKiESkwBQRiUiBKSISkQJTRCQiBaaISEQKTBGRiBSYIiIRKTBFRCJSYIqIRKTAFBGJSIEpIhKRAlNEJCIFpohIRApMEZGIFJgiIhEpMEVEIlJgiohEpMAUEYlIgSkiEpECU0QkIgWmiEhECkwRkYgUmCIiESkwRUQiUmCKiESkwBQRiUiBKSISUayBaWZnmNkbZrbGzOa1s/16M3vNzF4xs8VmNi7O+oiI9ERsgWlm6cAC4EzgUOAiMzu0zW4vASXufgTwKPD9uOojItJTcfYwpwNr3H2tu9cBDwLnJO7g7kvcvTr8uBwYHWN9RER6JM7AHAVsSPhcFq7ryOXA4+1tMLMrzazUzErLy8t7sYoiItENiEEfM7sEKAF+0N52d7/L3UvcvWTEiBF9WzkRkVBGjGVvBMYkfB4drmvFzE4Bvgqc5O61MdZHRKRH4uxhvghMNLMJZpYFXAgsTNzBzI4EfgHMcvf3Y6yLiEiPxRaY7t4AXA08CawCHnb3V83sFjObFe72AyAfeMTMVprZwg6KExHpd3GekuPui4BFbdbdlLB8SpzHFxHpTQNi0EdEJBkoMEVEIlJgiohEpMAUEYlIgSkiEpECU0QkIgWmiEhECkwRkYgUmCIiESkwRUQiUmCKiESkwBQRiUiBKSISkQJTRCSiWKd36yv19fWUlZVRU1PT31WRiHJychg9ejSZmZn9XRWRyFIiMMvKyigoKGD8+PGYWX9XR7rg7lRWVlJWVsaECRP6uzoikaXEKXlNTQ1FRUUKyyRhZhQVFemMQJJOSgQmoLBMMvp9STJKmcDsb/n5+S3LixYtYtKkSbzzzjvMnz+fvLw83n///Xb3NTNuuOGGls+33nor8+fPb/cYf/rTn7jlllsAmDNnDo8++uge13Pp0qUMGzaMqVOnMnXq1Jby6urqOPHEE2loaNjjMkUGCwVmL1u8eDHXXHMNjz/+OOPGjQOguLiYH/7wh+3un52dzR//+EcqKiq6LPv73/8+V111VY/reMIJJ7By5UpWrlzJTTcFr1jKysri5JNP5qGHHupx+SKpSoHZi5555hmuuOIK/vd//5cDDjigZf3cuXN56KGH2Lx5827fycjI4Morr+THP/5xp2WvXr2a7OxsiouLd9v2ta99jTlz5tDY2Nij+s+ePZsHHnigR2WIpLKUGCVPNH7eY7GUu+67Z3e6vba2ltmzZ7N06VIOPvjgVtvy8/OZO3cut912GzfffPNu3/3c5z7HEUccwZe+9KUOy3/22Wc56qijdlv/xS9+ke3bt3PPPfdgZnzhC19gyZIlu+134YUXMm/ePACee+45pkyZwn777cett97KYYcdBsDkyZN58cUXO22nyGCWcoHZXzIzMzn++OO5++67ue2223bbfs011zB16lRuvPHG3bYNHTqUSy+9lJ/+9Kfk5ua2W/6mTZsYMWJEq3Xf+MY3OOaYY7jrrrta1nXVUz3qqKN45513yM/PZ9GiRcyePZs333wTgPT0dLKysti+fTsFBQVdtllksEm5wOyqJxiXtLQ0Hn74YU4++WS+/e1v85WvfKXV9uHDh/PJT36SBQsWtPv96667jqOOOorLLrus3e25ubls3bq11bpp06axYsUKNm/eTGFhIUCXPcyhQ4e2rDvrrLO46qqrqKioaDnVr62tJScnJ3rDRQaRlAvM/pSXl8djjz3GCSecwMiRI7n88stbbb/++uuZNm1auyPRhYWFnH/++dx9993MnTt3t+2HHHIIv/3tb1utO+OMMzj99NM5++yzeeqppygoKOiyh/nee+8xcuRIzIwXXniBpqYmioqKAKisrKS4uFhP34h0QIM+vaywsJAnnniCb37zmyxcuLDVtuLiYj72sY9RW1vb7ndvuOGGDkfLTzzxRF566SXcvdX68847jyuuuIJZs2axc+fOLuv36KOPMnnyZKZMmcI111zDgw8+2HJP5JIlSzj77P7poYskA2v7D3CgKykp8dLS0lbrVq1axSGHHNJPNeo71157LR/96Ec55ZRTYin/3HPP5bvf/S6TJk2Kpfy2BsvvTZKDma1w95LO9lEPM4l85Stfobq6Opay6+rqmD17dp+FpUgyUmAmkZEjRzJr1qxYys7KyuLSSy+NpWyRVKHAFBGJSIEpIhKRAlNEJKLBdx9m9WZYvxzeXga12yG7ACacBGOPhbzC/q6diAxggwJF588AAAuaSURBVKuHWfkWLL4FXn8M0rNh6Ojgv68/FqyvfKt7xVZWtkyXts8++zBq1CimTp3K8OHDOfTQQ3u5EcEUbR/5yEf26DszZ86k7e1YAPfeey9XX3115HLuu+8+Jk6cyMSJE7nvvvs63O/222/n4IMP5rDDDuv0GXmRZDJ4epjVm+Fvt0NGLuSP3LU+MxeGjYaabcH2k2/a455mUVERK1euBGD+/Pnk5+dz4403sm7dukjB1tDQQEbGwP9VbN68mZtvvpnS0lLMjKOPPppZs2ax1157tdpvyZIl/Pd//zcvv/wy2dnZreYCFUlmg6eHuX45NNZDztD2t+cMhcY62PB8rx62sbGRK664gsMOO4zTTjut5WmcmTNnct1111FSUsJtt91GeXk5H//4x5k2bRrTpk3j2WefBWDZsmUtvdcjjzyS7du3A1BVVcUnPvEJDj74YC6++OKWJ4AWL17MkUceyeGHH87cuXPbfaronnvuYdKkSUyfPr3lOFE8+eSTnHrqqRQWFrLXXntx6qmn8sQTT+y23x133MG8efPIzs4GYO+9996zPzSRAWrgd2v21JtPQ9W/dl//j0cgLR127j4nZYvGOnj+rqA3mih/JEzs3tM1b775Jr///e/55S9/yfnnn88f/vAHLrnkEiC4Wbz5NPmTn/wkX/jCF/jQhz7E+vXrOf3001m1ahW33norCxYsYMaMGVRVVbVMjPHSSy/x6quvst9++zFjxgyeffZZSkpKmDNnDosXL2bSpElceuml3HHHHVx33XUt9dm0aRNf//rXWbFiBcOGDePDH/4wRx55JAAPPPAAP/jBD3Zrw4EHHsijjz7Kxo0bGTNmTMv60aNHs3Hjxt32X716NX/5y1/46le/Sk5ODrfeeivTpk3r1p+fyECSeoHZkfqdkLtX5/ukZULtB7162AkTJjB16lQAjj76aNatW9ey7YILLmhZfvrpp3nttddaPm/bto2qqipmzJjB9ddfz8UXX8y5557L6NGjAZg+fXrL8tSpU1m3bh0FBQVMmDCh5WmdT33qUyxYsKBVYD7//PPMnDmzZaq4Cy64gNWrVwNw8cUXc/HFF/e4zQ0NDWzevJnly5fz4osvcv7557N27Vq9x0eSXuoFZkc9wfdeCQZ4MtufbxIIQrWpDo7seWg0az4thWC+ycQJMoYMGdKy3NTUxPLly3ebWm3evHmcffbZLFq0iBkzZvDkk0+2W25vvIunqx7mqFGjWLp0acv6srIyZs6cudv+o0eP5txzz8XMmD59OmlpaVRUVOw2n6dIshk81zAnnATVlZ3vU10R7NcPTjvtNG6//faWz82DSG+99RaHH344X/7yl5k2bRqvv/56h2UcdNBBrFu3jjVr1gBw//33c9JJrdtzzDHHsGzZMiorK6mvr+eRRx5p2XbxxRe3vOsn8af5ZWunn346Tz31FFu2bGHLli089dRTnH766bvVY/bs2S1zcq5evZq6urp2X60hkmwGT2COPRbSM4PR8PbUbIP0LBhzTN/WK/TTn/6U0tJSjjjiCA499FDuvPNOAH7yk58wefJkjjjiCDIzMznzzDM7LCMnJ4d77rmH8847j8MPP5y0tDQ+85nPtNpn3333Zf78+Rx33HHMmDFjj2YLKiws5Gtf+1rLwNRNN93UMnHxpz/96ZbrsXPnzmXt2rVMnjyZCy+8kPvuu0+n45ISBtf0bpVvBbcONdZBXjFkZENDbdCzTM+C4z8PRQd0XY70Ck3vJgNJlOndUu8aZmeKDgjus9zwfPCkz7ZKyCmAQz4a9Cz1pI+IdCLWwDSzM4DbgHTgV+7+3Tbbs4HfAEcDlcAF7r4uzjqRVwgHnRn8iIjsgdiuYZpZOrAAOBM4FLjIzNo+J3g5sMXdDwR+DHwvrvqIiPRUnIM+04E17r7W3euAB4Fz2uxzDtD8QPKjwMnWzdGBZLsWO9jp9yXJKM7AHAVsSPhcFq5rdx93bwC2AkV7eqCcnBwqKyv1jzBJuDuVlZV6na8knaQY9DGzK4ErAcaOHbvb9tGjR1NWVkZ5eXlfV026KScnp+VJJZFkEWdgbgTGJHweHa5rb58yM8sAhhEM/rTi7ncBd0FwW1Hb7ZmZmUyYMKGXqi0i0r44T8lfBCaa2QQzywIuBBa22Wch8Klw+RPAn13n1SIyQMXWw3T3BjO7GniS4LaiX7v7q2Z2C1Dq7guBu4H7zWwNsJkgVEVEBqRYr2G6+yJgUZt1NyUs1wDnxVkHEZHeknSPRppZOfBOm9XFQEU/VCdualfySMU2weBq1zh373RKraQLzPaYWWlXz4AmI7UreaRim0DtamvwzFYkItJDCkwRkYhSJTDv6u8KxETtSh6p2CZQu1pJiWuYIiJ9IVV6mCIisUuqwDSzM8zsDTNbY2bz2tmebWYPhdufN7PxfV/LPRehXdeb2Wtm9oqZLTazcf1Rzz3RVZsS9vu4mbmZJcVIbJR2mdn54e/rVTP7XV/XsTsi/B0ca2ZLzOyl8O/hWf1Rzz1hZr82s/fN7J8dbDcz+2nY5lfM7KguC3X3pPgheFroLWB/IAt4GTi0zT5XAXeGyxcCD/V3vXupXR8G8sLlzw70dkVpU7hfAfAMsBwo6e9699LvaiLwErBX+Hnv/q53L7XrLuCz4fKhwLr+rneEdp0IHAX8s4PtZwGPAwYcCzzfVZnJ1MPs0/k1+1CX7XL3Je5eHX5cTjCRyUAW5XcF8A2CSaNr+rJyPRClXVcAC9x9C4C7v9/HdeyOKO1yYGi4PAx4tw/r1y3u/gzBI9cdOQf4jQeWA8PNbN/OykymwOyz+TX7WJR2Jbqc4P+KA1mXbQpPf8a4+2N9WbEeivK7mgRMMrNnzWx5+JqWgS5Ku+YDl5hZGcHjzp/vm6rFak//7SXHfJgSMLNLgBKgf16e3kvMLA34ETCnn6sShwyC0/KZBGcCz5jZ4e7+Qb/WqucuAu519x+a2XEEk+ZMdvem/q5YX0qmHuaezK9JZ/NrDjBR2oWZnQJ8FZjl7rV9VLfu6qpNBcBkYKmZrSO4frQwCQZ+ovyuyoCF7l7v7m8DqwkCdCCL0q7LgYcB3P05IIfgeexkFunfXqJkCsxUnV+zy3aZ2ZHALwjCMhmuiXXaJnff6u7F7j7e3ccTXJed5e6l7Rc3YET5O/gngt4lZlZMcIq+ti8r2Q1R2rUeOBnAzA4hCMxkf8XBQuDScLT8WGCru2/q9Bv9PZK1h6NeZxH8H/st4KvhulsI/rFB8Et8BFgDvADs39917qV2PQ38C1gZ/izs7zr3tE1t9l1KEoySR/xdGcHlhteAfwAX9nede6ldhwLPEoygrwRO6+86R2jT74FNQD1Bz/9y4DPAZxJ+VwvCNv8jyt9BPekjIhJRMp2Si4j0KwWmiEhECkwRkYgUmCIiESkwRUQiUmDKgGdmjWa20sz+aWb/Y2bDe7n8OWb2s3B5vpnd2JvlS+pQYEoy2OnuU919MsFkCp/r7wrJ4KTAlGTzHAkTJJjZF83sxXA+w5sT1l8arnvZzO4P1300nCf1JTN72sxG9kP9JYlp8g1JGmaWTvB43t3h59MIntOeTvDUxkIzO5Fg/oD/BI539wozKwyL+CtwrLu7mX0a+BJwQx83Q5KYAlOSQa6ZrSToWa4C/i9cf1r481L4OZ8gQKcAj7h7BYC7N8+JOBp4KJzzMAt4u2+qL6lCp+SSDHa6+1RgHEFPsvkapgHfCa9vTnX3A9397k7KuR34mbsfDvw7wdwDIpEpMCVpeDDr/DXADeH0fU8Cc80sH8DMRpnZ3sCfgfPMrChc33xKPoxd03d9CpE9pFNySSru/pKZvQJc5O73h1ONPRe+iaQKuMTdXzWzbwHLzKyR4JR9DsGs4Y+Y2RaCUJ3QH22Q5KXZikREItIpuYhIRApMEZGIFJgiIhEpMEVEIlJgiohEpMAUEYlIgSkiEpECU0Qkov8PC0gPCUQoD7kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5,5))\n",
    "plot_pr(recalls, precisions, label=\"KNN (k=5)\")\n",
    "plt.plot(recalls[np.min(np.where(precisions >= 0.9))], precisions[np.min(np.where(precisions >= 0.9))], \n",
    "         marker=\"o\", markersize=10, label=\"Threshold=0.6\", alpha=0.5)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can use this new threshold for our new predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[42,  5],\n",
       "       [ 6, 47]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_new_threshold = knn_probs >= new_threshold\n",
    "y_pred_new_threshold\n",
    "confusion_matrix(y_test, y_pred_new_threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9038461538461539, 0.8867924528301887)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_precision, new_recall = 47/(47+5), 47/(47+6) \n",
    "new_precision, new_recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: while here we have chosen the best classifier based on the AUC and Average Precision Score, it is not always the case that, for our applications, we must choose the classifier based on such scores. Usually, different classifiers work best in different parts of the curve (at different operating points). For a required Precision/Recall point, for example, it is possible that the classifier with worse Average Precision Score is performing better than the one with best score (and a similar reasoning can be done with the ROC curve). It is up to us, and to our business goal/research question, to understand which algorithm is the most appropriate.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision/Recall vs. ROC curve\n",
    "ROC and P/R curves convey similar information. Therefore, it is not always clear when one curve should be used in place of the other. In general, this really depends on the application domain and goals; in medical research, for instance, the ROC curve is widely used. \n",
    "\n",
    "However, there is an important difference that should be kept in mind. Unlike the ROC curve, the P/R curve is more sensitive to imbalanced datasets. *Imbalanced data* occur (as we have observed in the data-preprocessing lecture) when the positive class is much rarer than the other. The ROC curve, in fact, compares TPR vs. TNR, which are both probabilities conditioned on the observed classes (for instance, we can regard the TPR as $\\Pr(\\hat{y}=P|y=P)$). Therefore, they are normalized by the amount of observations we observe within each class; this means that arbitrarily changing the size of each class does not change TPR and TNR and, as a consequence, the ROC curve. On the other hand, the P/R curve compares Recall vs. Precision; while the Recall is equal to the TPR, the Precision is not a quantity conditioned on the observed class. The Precision of a classifier is the probability that an instance is actually positive, conditioned on the fact that it has been classified as positive: $\\Pr(y=P|\\hat{y}=P)$. In this case the class size matters, as a rare positive class affects both numerator and denominator of such quantity. In this case, modifying the proportions of the classes will strongly affect the P/R curve.\n",
    "\n",
    "In light of this fact, the P/R curve can be used when it is of interest the result of a classifier *with respect to class proportions*, i.e., keeping them into account. The ROC curve can then be used when the interest is on performance *irrespective of class proportions*. This can be seen as a rule of thumb. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.6 $F_1$ score\n",
    "The $f_1$ score is a measure that summarizes the Precision and Recall levels of a classifiers. It consists of a particalar form of mean between Precision and Recall, the *harmonic mean*. This type of mean is used instead of the regular mean, because the regular mean gives the same weight to all values, while the harmonic mean gives more weight to lower values. Therefore, a classifier that has either low Recall or low Precision (or both) are more strongly penalized.\n",
    "\n",
    "$$f_1 = \\frac{2}{\\frac{1}{precision}+\\frac{1}{recall}} = 2\\frac{precision \\times recall}{precision + recall}$$\n",
    "      \n",
    "The higher the $f_1$ score, the better the classifier. $f_1$ favors, in general, classifiers with both high precision and high recall (which, as we have seen above, might not always be the desired choice). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8952380952380953"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "# f1 score for knn with k=5:\n",
    "f1_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`scikit-learn` comes with a `classification_report` function that can help us to summarize the results: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.89      0.88        47\n",
      "           1       0.90      0.89      0.90        53\n",
      "\n",
      "    accuracy                           0.89       100\n",
      "   macro avg       0.89      0.89      0.89       100\n",
      "weighted avg       0.89      0.89      0.89       100\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(y_test,y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the report we can find: \n",
    "\n",
    "* the precision, recall, $f_1$ score, and support (= number of examples in the test set) observed for each class (that is, when each class is considered the positive one)\n",
    "* test-set accuracy \n",
    "* the normal (\"macro\") average of precision, recall, and $f_1$ score (preferable when you want to treat classes equally)\n",
    "* the average of precision, recall, and $f_1$ score, weighted by class sizes (preferable in imbalanced cases)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.7 Multiclass Classification\n",
    "When dealing with more than two labels, we can use three different approaches: \n",
    " * learn a classifier that directly works with multiple classes (not supported by all classifiers; an example of classifier that supports multiple classes is the Classification Tree algorithm)\n",
    " * train as many binary classifiers as the number of classes, treating each time a class as the positive one and all the others as the negative; use the classifier with the highest score to predict the class (as returned by the `decision_function` or `predict_proba` methods of the `scikit-learn` classifiers). This method is known as One-versus-All (OvA) or **One-versus-Rest** (OvR), and it is supported by the [OneVsRest classifier in scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsRestClassifier.html) \n",
    " * train a binary classifier for each pair of classes: Class 1 vs. Class 2, Class 1 vs. Class 3, Class 2 vs. Class 3, and so on; with $C$ classes, this will train $C(C-1)/2$ classifiers. The class that wins \"most rounds\" will be the predicted one. While, with a large number of classes, the number of models to train can quickly grow, these classifiers are also faster to train, as we only used a subset of the original data (more precisely, the subset of data coming from the classes that are trained). This is known as *One-versus-One* (OvO) approach, and can be implemented in scikit-learn with the [OneVsOne classifier](https://scikit-learn.org/stable/modules/generated/sklearn.multiclass.OneVsOneClassifier.html#sklearn.multiclass.OneVsOneClassifier)\n",
    " \n",
    "Nevertheless, `scikit-learn` is able to detect automatically when we are performing multiclass classification, and calls internally the OvR approach for all classifiers whenever there are several classes in the dataset (with exception of a classifier called Support Vector Machine, which calls the OvO strategy). This behaviour is the default, and can be changed by means of the `multi_class` parameter of the model (you should check this option in the scikit-learn documentation of the algorithm functions). As mentioned above some models can handle directly multiple classes and do not have this parameter. One of such classifiers is K-nearest-Neighbors. Once again, let's see how it behaves in a multiclass case. We will start by creating a fictitious 3-class dataset, and then perform prediction with the [KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "                     metric_params=None, n_jobs=None, n_neighbors=5, p=2,\n",
       "                     weights='uniform')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We randomly generate a three-class, two-features dataset with 500 examples\n",
    "X, y = make_classification(n_samples=500, n_features=2, n_redundant=0, n_informative=2,\n",
    "                             n_clusters_per_class=1, n_classes=3, random_state=100, class_sep=0.95)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)\n",
    "# Fit knn (default: k=5)\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, knn yields the following boundaries: \n",
    "\n",
    "<img src=\"./img/model_performance/knn_multi_class.png\" width=\"2000\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to implement a classifier with an OvR approach; for instance, we can use the Logistic Regression classifier. This classifier learns *linear* decision boundaries (as we will see in the lecture dedicated to logistic regression). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='ovr', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr_mod = LogisticRegression(multi_class='ovr')\n",
    "lr_mod.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, thw following decision boundaries where generated by logistic regression: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./img/model_performance/multi_log_reg.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also observe the probabilities for each unit to belong to each class. Let's do it for the first five units with the Logistic Regression model just trained: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4.24450270e-01, 4.72993096e-02, 5.28250420e-01],\n",
       "       [8.88100918e-01, 7.63133870e-04, 1.11135949e-01],\n",
       "       [6.75039163e-02, 9.05994686e-01, 2.65013982e-02],\n",
       "       [6.00605090e-04, 8.68077199e-03, 9.90718623e-01],\n",
       "       [9.66644081e-01, 1.34824451e-02, 1.98734738e-02]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_mod.predict_proba(X_test)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, each column represents a class, and each row corresponds to a unit. We can retrieve the predicted class for all units by seeing what is the class that maximizes the probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What is the predicted class for the first unit?\n",
    "np.argmax(lr_mod.predict_proba(X_test)[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's check if it is what the classifier actually predicts: \n",
    "lr_mod.predict(X_test[0].reshape(1,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can visualize how the classes are encoded with the `classes_` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 2])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_mod.classes_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, in this case both original and encoded classes corresponds to numbers and have the same values. The `classes_` attribute will turn out useful when dealing with classes that are expressed through their labels (e.g., \"Ham/Spam\" emails). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike knn, models like logistic regression also calculate a *decision function* for class assignment and class probability estimation (how this function is computed will be explained in the coming lectures). As before, let's see this for the first 5 units:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.11619105, -3.56592797, -0.81392854],\n",
       "       [ 4.68106686, -7.06778186, -1.95518028],\n",
       "       [-2.97916364,  0.61584737, -3.9445625 ],\n",
       "       [-7.40986544, -4.73076131,  6.10810038],\n",
       "       [ 3.299234  , -4.29514594, -3.90066442]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_mod.decision_function(X_test)[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the `intercept_` and `coef_` attributes, we can see the *parameter* learned by the model in order to calculate the decision functions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model coefficients:[[-3.36873203  3.11913395]\n",
      " [ 0.72131269 -4.710679  ]\n",
      " [ 2.16625716  2.62029574]]\n"
     ]
    }
   ],
   "source": [
    "# Model coefficients: one set for each classifier\n",
    "print(\"Model coefficients:{0}\".format(lr_mod.coef_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model intercepts:[-3.00954582 -0.55883224 -2.57343571]\n"
     ]
    }
   ],
   "source": [
    "# Intercepts: one for each classifier\n",
    "print(\"Model intercepts:{0}\".format(lr_mod.intercept_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now compare the class-specific decision boundaries of each OvR model, and compare them with model predictions: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./img/model_performance/multi_logistic_boundaries.png\" width=\"5000\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The region in the middle is assigned to the class with the largest values of the decision function $f(\\mathbf{x})$ (or class probability). Of course, attributes like `intercept_` and `coef_` are available also for the binary case. We can evaluate the accuracy of the multi-class classifier, and explore its confusion matrix. Let's do it for the logistic regression classifier:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.92"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_lr = lr_mod.predict(X_test)\n",
    "accuracy_score(y_test, y_pred_lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[31,  0,  4],\n",
       "       [ 0, 33,  1],\n",
       "       [ 2,  1, 28]])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conf_mat = confusion_matrix(y_test, y_pred_lr)\n",
    "conf_mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATwAAAEGCAYAAAD45CnNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXwUVdbw8d9Jh02WEAwTIOwICIisougjKiiCKCiCg6LiPog87jq4gjiiA+MyCj4jo7iNDi7gDK/CgIojbigEQXYElCUECKuAQEjnvH90ETshpKuS7nR3+nz51IdabtU9FePh1nJviapijDGJICnaARhjTHmxhGeMSRiW8IwxCcMSnjEmYVjCM8YkjORoBxBMKldXqVYn2mHErE4npUc7hJiXl29vHZRk08YN7Nq5Q8pyDF+tJqp5B12V1YM5s1W1T1nqC6fYSnjV6lCl+93RDiNmffXhXdEOIebtPpAb7RBiWp9zu5f5GJp3kCqtr3BV9tDiSWllrjCMYirhGWPigYDE590wS3jGGG8ESPJFO4pSsYRnjPFOynQbMGos4RljPLJLWmNMIrEWnjEmIQjWwjPGJAqxFp4xJoHYU1pjTGKwhxbGmEQh2CWtMSaBWAvPGJMY7JLWGJMoBPDZQwtjTKKwe3jGmMRgl7TGmEQSpy28+EzTxpjokiR3U6jDiPQRkdUislZERhWzfbiILBWRxSLypYi0Ddr2gLPfahG50E3YlvCMMd6IuJ9KPIz4gElAX6AtcGVwQnO8rartVbUjMB54xtm3LTAEaAf0AV50jlciS3jGGO+SfO6mknUD1qrqelXNBaYCA4ILqOovQYvVgaMfLRkATFXVw6r6E7DWOV6J7B6eMcYjTw8t0kRkYdDyZFWd7MxnAJuCtm0GTj+mNpHbgLuBykDPoH3nF9k3I1QwlvCMMd65f2ixQ1W7lqUqVZ0ETBKRq4CHgWGlPZYlPGOMN+EbDy8LaBS03NBZdzxTgf8r5b6A3cMzxngm4XpKuwBoKSLNRKQygYcQMwrVJNIyaLEf8KMzPwMYIiJVRKQZ0BL4LlSF1sIzxngXhvHwVDVPREYCswEfMEVVl4vIWGChqs4ARorI+cARYDfO5axT7l1gBZAH3Kaq/lB1WsIzxngXphePVXUmMLPIukeD5u8oYd8ngCe81GcJzxjjjVjXMmNMIonTrmWW8IwxnoklPGNMIgiM8G4JzxiTCESQJEt4catX5yY8ecu5+JKSeHPOMp57f0Gh7df3PZWb+nXAn5/PgYNHuHPiJ6zetCtK0caGT75ewQNPv48/P59rBpzJXdf1jnZIEff5tysZO/Ff5PvzuaLfGdw6tFeh7Ydz87j3ybdZtnoTtVOq88Kj19Kwfh2+WLiaCZM/IvdIHpUrJTNq+CWc2TnwetmMTxfx4j8+QURIP7EWzzw0lDq1a0Tj9DyJ1xZeRB+1hBr6JRYkJQkTbu3J4NH/4owRr3P5Oa1p3ahOoTLv/3cVZ418kx63v8Xz0xbyp5vOiVK0scHvz+e+8e/y3l9HMP/dh5k2J5NV67OjHVZE+f35jP7rdF798y3Mfv2P/L+5i/jx562Fyrw781tq1ajGZ28/xA2DzuHPkz8EoE5Kdf4+7kb+8+r9TBh1JfeMewuAvDw/j7/wL95+dgSzptzHyS0a8MYHX5b7uZWGiLiaYk3EEp7LoV+irkureqzP3sOGbXs5kpfP9HmrueiMFoXK7DuYWzB/QtVKoFr0MAklc/nPNG+URtOGaVSulMzACzoz8/Mfoh1WRC1ZtZEmGWk0bnAilSslc3HPTnz81bJCZT75ahmX9zkNgL7nnMrXmT+iqrRr2ZD0tBQAWjWrx6HDRzicm4cCqsrBQ7moKvsOHCooF+viNeFF8pK2YOgXABE5OvTLigjW6Vn9E2uQlbOvYHnLjv10aV3vmHI39evAiEs7UznZR/+H3i/PEGNOds5eMtJTC5YbpKeSuezn6AVUDrbm7KV+3doFy/Xr1mbxig2FymwLKpOc7KNmjars3nug0CXqrM9/oF3LhlSpHPhfb+xdg+h7wwSqVa1M04Z1GXvn5eVwNmUkzhSHInlJW9zQLyGHb4lVL3+0hM43v8qY177g3t8fM4KNMSGt+Wkr4yd/yBP3DAbgSJ6ft2Z8zf/7+z3MnzaGk5vX5//e+jTKUYYmuGvdxWILL+qvS4vILSKyUEQWau6Bcq8/e+d+MurWLFhukFaD7J37j1t+2rzV9CtyyZto6tdNIWvb7oLlLdt2U79ufFyKlVa9uilk5+wpWM7O2UN6kXNODyqTl+dn3/5DpKZUD5Tfvofhj7zKXx64iiYZaQCsWBsY3KNJRhoiQr/zOpK5/KfyOJ0yS0pKcjXFmkhG5Gr4FlWdrKpdVbWrVK4ewXCKt2jNVlo0SKVxei0qJScxsEdrZn27vlCZ5g1+u5S58LTmrNuyp+hhEkrntk1YtzGHDVk7yD2Sx/SPF9G3x6nRDiuiTm3diJ8357Apeye5R/L4cO73nH/mKYXK9DqzHdP+E3jCP+vzH+je+SREhF/2HeTGB/7O/bf0o2v7ZgXl66WlsPbnrezcE/gH9suFazipSXr5nVQZxGsLL5L38AqGfiGQ6IYAV0WwvlLx5yv3/20u08YOxJckvPXxclZt3MkDQ7uz+MdtzPpuPTdf3JFzOjQmz+9nz/7DjHh2drTDjqrkZB/j77+Cy2+fhN+vDO1/Bm1a1I92WBGVnOxjzB0DGXbfZPLz8xnctxutmtXj2SmzaN+6EeefdQq/v+h07h73Nudd9QQptU7g+UevBeCND75kQ9ZOXnh9Di+8PgeA1//yB9LTUrh92IUMuX0iyck+MtJTmTDqymiepjtxfA9PNIJPHEXkIuA5fhv6pcSRDZJSGmmV7ndHLJ54t/vDu6IdQszbfSA3dKEE1ufc7iz5PrNM6So5rbnWvnicq7I7X78ys6wjHodTRF88Lm7oF2NMfDv60CIeWU8LY4xn1rXMGJMYJH67llnCM8Z4ZgnPGJMwLOEZYxKCPbQwxiSW+Mx3lvCMMR4JMdltzA1LeMYYz+yS1hiTOOIz30V/tBRjTPwJ1+ABoUZFF5G7RWSFiPwgIp+KSJOgbX4RWexMM9zEbS08Y4wn4RoJJWhU9AsIjJe5QERmqGrwIMHfA11V9VcRuRUYD/ze2XZQVTt6qdNaeMYYz8LUwisYFV1Vc4Gjo6IXUNXPVPVXZ3E+gWHmSs0SnjHGM0kSVxOQdnSAX2e6JegwXkdFvxGYFbRc1TnmfBG51E3cdklrjPHMwyXtjnAMDyUiVwNdgeBPBjZR1SwRaQ7MFZGlqrqupONYwjPGeBO+wQNcjYouIucDDwHnqOrho+tVNcv5e72I/BfoBJSY8OyS1hjjiQAi7qYQCkZFF5HKBEZFL/S0VUQ6AS8B/VV1e9D6VBGp4synAWfh4ouI1sIzxngUnqe0qponIiOB2fw2KvpyERkLLFTVGcAEoAbwnlPnRlXtD7QBXhKRfAINt6eKPN0tliU8Y4xnSWEaALS4UdFV9dGg+fOPs9/XQHuv9VnCM8Z44+5yNSZZwjPGeCKEr4VX3izhGWM8sxaeMSZh2GgpxpjEYPfwjDGJQhAbANQYkzishWeMSRh2D88YkxjsHp4xJlEE+tLGZ8azhGeM8SxO850lPGOMd9bTwhiTGMI3Hl65i6mE1+mkdL768K5ohxGzUk8bGe0QYt6u716IdggxzReGltnR8fDiUUwlPGNMPAjPeHjRYAnPGONZnOY7S3jGGI/EHloYYxKEvYdnjEkolvCMMQkjTvOdJTxjjHfWwjPGJAYbPMAYkygCA4DGZ8azhGeM8SwpTpt48TlOszEmqkTcTaGPI31EZLWIrBWRUcVsv1tEVojIDyLyqYg0Cdo2TER+dKZhbuK2hGeM8UScwQPcTCUfR3zAJKAv0Ba4UkTaFin2PdBVVU8F3gfGO/vWAUYDpwPdgNEikhoq9uMmPBGpVdIU6sDGmIorSdxNIXQD1qrqelXNBaYCA4ILqOpnqvqrszgfaOjMXwh8rKq7VHU38DHQJ1SFJd3DWw4ogRerC+p3lhVoHPJ0jDEVkoeHFmkisjBoebKqTnbmM4BNQds2E2ixHc+NwKwS9s0IFcxxE56qNgq1szEm8QiBJ7Uu7VDVrmWuU+RqoCtwTlmO4+oenogMEZEHnfmGItKlLJUaY+JbmC5ps4DghlVDZ10hInI+8BDQX1UPe9n3mLhDFRCRicB5wDXOql+Bv4XazxhTQbl8YOGiN8YCoKWINBORysAQYEbhqqQT8BKBZLc9aNNsoLeIpDoPK3o760rk5j28M1W1s4h8D6Cqu5zgjDEJKhyv4alqnoiMJJCofMAUVV0uImOBhao6A5gA1ADecxLoRlXt7+ShxwkkTYCxqrorVJ1uEt4REUki8KACETkRyPd6csaYikEI34vHqjoTmFlk3aNB8+eXsO8UYIqX+twkvEnANKCuiDwGXAE85qUSY0zFUmG7lqnqGyKSCRzNtINVdVlkwzLGxCq3vShikdu+tD7gCIHLWuudYUyCq7B9aUXkIeCfQAMCj37fFpEHIh2YMSZ2icsp1rhp4V0LdDravUNEniDQv+3JSAZmjIldFXkA0Owi5ZKddcaYBBR4ShvtKErnuAlPRJ4lcM9uF7BcRGY7y7357d0XY0yikYo5AOjRJ7HLgY+C1s+PXDjGmHhQ4S5pVfWV8gzEGBMfKuQl7VEi0gJ4gsAAfVWPrlfVVhGMyxgTw+K1hefmnbrXgFcJJPa+wLvAOxGMyRgT4+L1tRQ3Ce8EVZ0NoKrrVPVhAonPGJOARMCXJK6mWOPmtZTDzuAB60RkOIExp2pGNqzo+uTrFTzw9Pv48/O5ZsCZ3HVd72iHVO56dW/Dk/cMwpeUxJv//prnXv+40PbrB/4PNw3ugT8/nwO/HubOcf9k9U9b6dy2Cc89dCUQ+Bf+qb/P5KP//hCFMwi/T75ZwYNPT3N+L7pz57DCvxeHc49w65g3WbJqE6kp1ZnyxPU0bnAiAM++Nod/zPgGX1IST94ziF7d27B5225GjHmT7bv2IcCwy85i+JBzy//ESiFeL2ndJLy7gOrA7QTu5aUAN4TaSUSmABcD21X1lLIEWZ78/nzuG/8uH0wcSYP02vQcNoG+PdpzcvP60Q6t3CQlCRPuv4LLRk5ky7Y9zH39PmbNW8rqn7YWlHl/9kJenf4lAH17tOdPdw1k8O0vsnLdFs67djx+fz7pJ9bii7cf4D9fLMPvj+8Bdvz+fO4f/x7TJ95Gg9/VptewCfQ5u/DvxT9mfEPtmieQOX000+ZkMmbiv5ky7gZWrc9m+pxMvp76IFtz9nLZyEkseP8Rkn1JPH7HZXQ4uRH7Dhyi57XjObdb67j4XYvTfBf6klZVv1XVfaq6UVWvccai+srFsV/DxUc1Yk3m8p9p3iiNpg3TqFwpmYEXdGbm5xWjheJWl3ZNWb9pBxuydnIkz8/0jxdx0TmnFiqz78ChgvkTqlYGVQAOHj5SkNyqVKmEOuvjXebyDTRrmEbTDOf3oncXZs1bWqjMzM+XMqRf4JMMA3p2ZN6CNagqs+YtZWDvLlSpXIkmGWk0a5hG5vIN1EtLocPJgUF7a1avSqtm9cjO2Vvu5+aVICSJuynWlPTi8Qc4Y+AVR1UHlnRgVZ0nIk1LHVmUZOfsJSP9t6+9NUhPJXPZz9ELKArq100ha9vuguUt23bT5ZSmx5S7aXAPRlx1HpUrJdP/1ucL1ndp14QXHr2aRvXqMHz063HfugPIztlT+Pfid7XJXP5zkTJ7yUivDUByso9aNaqxa+8BsnP20PWUZoX2zc7ZU2jfjVt28sPqzXRp14SYV0FHS5lYHgGIyC3ALQCNGtuH0OLJy+/N4+X35jHowq7ce0MfRjz2JhBoDZ35+ydo1TSdF8dcwydfr+Bwbl6Uo41d+389zLBRrzDu7oHUqlEt2uG4Eq/38I57Sauqn5Y0hSsAVZ2sql1VtWvdtLrhOmypFde6qV83JYoRlb/iWrklXWpNm5NJv3NPPWb9mp+3ceDXw7Rp0SAicZan+nVrF/692L6H+nVrFymTQta2QMstL8/PL/sPUieleon7HsnzM+yPLzPowq5ccl7HcjiTshPAJ+JqijU2tl0Rnds2Yd3GHDZk7SD3SB7TP15E3x7H/s9ckS1asYEWjevSuMGJVEr2MfCCzsyaV/g+ZvNGv/3jdOH/tGPdxhwAGjc4EZ8v8GvVqF4qLZvWY+OWneUXfIR0btuY9ZuCfi/mZNLn7PaFyvTt0Z6pH30LwL/nLubsrq0QEfqc3Z7pczI5nHuEDVk7WL8phy7tmqCq3P74W7RqVo/bhvaMxmmVWpi+Wlbu3A4AmjCSk32Mv/8KLr99En6/MrT/GbRpEftPzcIp8ETyXaY9fxs+n/DWjPmsWr+VB/7Qj8UrNzJr3lJuvqIH53Q7mbw8P3t++ZURj70BQPcOzbnjut7k5fnJz1fu/fM77Np7IMpnVHbJyT7G3zeYQbe/iD9fGXpJ4Pdi3Esf0alNY/r2aM/V/bszfPQbdBn4GKm1TuDlJ64HoE2L+lx6fme6/34cyb4kxt8/GJ8vifmL1/HOrAW0PakBPYY+BcAjIy7hgrPaRfNUXYnFZOaGuH2KJiJVgr4J6ab8P4FzgTRgGzA6VP/cLl266lffLiypSEJLPW1ktEOIebu+eyHaIcS0s844jUWZC8uUruq1PEWHPjPNVdln+p+cGY4PcYeLm7603YBXCLx/11hEOgA3qer/lrSfql4ZnhCNMbEmXlt4bu7hPU/gBeKdAKq6hMCHuY0xCeroh3xCTbHGTcJLUtUNRdb5IxGMMSb2CZAs4moKeSyRPiKyWkTWisioYrb3EJFFIpInIoOKbPOLyGJnmuEmdjcPLTY5l7UqIj7gf4E1bg5ujKmYwtF6c/LJJOACYDOwQERmqOqKoGIbgeuAe4s5xEFV9fQuj5uEdyuBy9rGBB4+fOKsM8YkIAlft7FuwFpVXe8cdyowAChIeKr6s7MtLN113HyIezswJByVGWMqhjDdn8sANgUtbwZO97B/VRFZCOQBT6nqv0Lt4OYp7d8ppk+tqt7iITBjTAXi4SltmpOUjpqsqpPDFEYTVc0SkebAXBFZqqrrStrBzSXtJ0HzVYHLKJyVjTEJRMDL4J47SngPLwtoFLTc0FnniqpmOX+vF5H/Ap2AsiU8VS00nLuIvAl86TYoY0wFE75uYwuAliLSjECiGwJc5SoEkVTgV1U9LCJpwFnA+FD7laYvbTMgvRT7GWMqCHH5pySqmgeMBGYDK4F3VXW5iIwVkf4AInKaiGwGBgMvichyZ/c2wEIRWQJ8RuAe3opjaynMzT283fx2Dy+JwIe5j3lfxhiTGML5mUZVnQnMLLLu0aD5BQQudYvu9zXQvuj6UEpMeBIY9KoDv11X52tFGcLWGFNqFbJrmZPcZqqq35ks2RljEBFXU6xxcw9vsYh0ingkxpi4EPhMo7sp1pT0TYtk56ZiJwJdPtYBBwhcwquqdi6nGI0xMSYWP9DjRkn38L4DOgP9yykWY0wcCOdDi/JWUsITgFBvLhtjEk+cNvBKTHh1ReTu421U1WciEI8xJuYJSSHesYtVJSU8H1AD4vTMjDERIVTMFl62qo4tt0iMMfFBIDlOb+KFvIdnjDHBKmoLr1e5RWGMiSsV7rUUVd1VnoEYY+JHnOY7+xC3McYboXTDLMUCS3jGGG+kAl7SGmNMcQI9LSzhGWMSRHymO0t4xphSiNMGniU8Y4xXsTnWnRuW8IwxnthTWmNMQrGHFmGQl6/s3J8b7TBi1q7vXoh2CDGvTr+nox1CTDv847ayH0SwS1pjTGKwS1pjTEKxFp4xJmHEZ7qL35apMSZKBPCJuJpCHkukj4isFpG1IjKqmO09RGSRiOSJyKAi24aJyI/ONMxN7NbCM8Z4Fo4rWhHxAZOAC4DNBL6OOENVVwQV2whcB9xbZN86wGigK6BAprPv7pLqtBaeMcYjcf0nhG7AWlVdr6q5wFRgQHABVf1ZVX8A8ovseyHwsarucpLcx0CfUBVawjPGeCbibgLSRGRh0HRL0GEygE1By5uddW6Ual+7pDXGeBJ4LcX1Ne0OVe0awXA8sRaeMcYbl607F/f5soBGQcsNnXVulGpfS3jGGM+SRFxNISwAWopIMxGpDAwBZrgMYTbQW0RSRSQV6O2sKzlulwc3xhjg6ACg7qaSqGoeMJJAoloJvKuqy0VkrIj0BxCR00RkMzAYeElEljv77gIeJ5A0FwBj3XyHx+7hGWM8c/EE1hVVnQnMLLLu0aD5BQQuV4vbdwowxUt9lvCMMZ7Fac8yS3jGGO/C1cIrb5bwjDGeHL2HF48s4RljvHH3BDYmWcIzxngWn+nOEp4xxiP7Lq0xJqHEZ7qzhGeMKY04zXiW8IwxntklrTEmYcRnurOEZ4wpjTjNeJbwjDGeCNbTwhiTKNyNdReTLOEZYzyL03xnCc8Y45XYh7iNMYkjTvOdJTxjjDeCXdIaYxJJnGY8S3jGGM/stZQ48vl3K/nTxH/h9+dzRb8zGH5Vr0LbD+fmcd+Tb7NszSZSa1Xnr6OvpWG9OixZuYGHn34PAFXl9usupPfZp0bjFCLik29W8ODT0/Dn53PNgO7cOax3oe2Hc49w65g3WbJqE6kp1ZnyxPU0bnAiAM++Nod/zPgGX1IST94ziF7d27B5225GjHmT7bv2IcCwy85i+JBzy//EykGvLk158g898SUJb85eynPvfVdo+/UXdeCmizvi9ysHDuVy5/Mfs3rTzihFW3Z2D68IEWkEvAGkAwpMVtW/Rqo+t/z+fMb8dTqvTxhOvbopDBz+LL3ObEfLpvUKyrw381tSalZj7lsP8eHc7xn/0oc8P/paWjWrzwcv3UWyz8f2nb9w8U1/oeeZ7Uj2+aJ4RuHh9+dz//j3mD7xNhr8rja9hk2gz9ntObl5/YIy/5jxDbVrnkDm9NFMm5PJmIn/Zsq4G1i1PpvpczL5euqDbM3Zy2UjJ7Hg/UdI9iXx+B2X0eHkRuw7cIie147n3G6tCx2zIkhKEiaMOJ/LHnqPLTv2Mfe5q5k1f12hhPb+Zyt5deYSAPqe3oI/3Xwugx+dFq2QyyaO38OL5Gca84B7VLUtcAZwm4i0jWB9rixZtZEmDdJo3OBEKldKpl/PTnzy1bJCZT75ahmXXXgaAH3OOZVvFv2IqlKtauWC5HY490jc/kcvTubyDTRrmEbTjDQqV0pmYO8uzJq3tFCZmZ8vZUi/0wEY0LMj8xasQVWZNW8pA3t3oUrlSjTJSKNZwzQyl2+gXloKHU4OfCu5ZvWqtGpWj+ycveV+bpHWpVU91m/ZzYatezmSl8/0eau4qHuLQmX2HcwtmD+haqXyDjHsxOWfWBOxFp6qZgPZzvw+EVkJZAArIlWnG9t27KX+72oXLNerW5slKzcct0yyz0eNGlXZ/csB6qTUYPGKDYwaP5Ut23bzlwevqhCtO4DsnD1kpKcWLDf4XW0yl/9cpMxeMtKdn0uyj1o1qrFr7wGyc/bQ9ZRmhfbNztlTaN+NW3byw+rNdGnXJHInESX1T6xJ1o59BctbduynS+tjW7E3XdyREZd1pXJyEv0feLc8QwwrwVp4JRKRpkAn4NvyqC+SOrZtwn9e+yPT/3YXf3v7Uw7nHol2SDFv/6+HGTbqFcbdPZBaNapFO5yoefnDxXS+8WXGvDqPe4d0j3Y4ZSIup1gT8YQnIjWAacCdqvpLMdtvEZGFIrJw544dkQ6H9LQUsrf/1vrYmrOH9LSU45bJ8/vZv/8QqbWqFypzUpN0TqhWhTU/bY14zOWhft3aZG3bXbC8Zfse6tetXaRMClnbnJ9Lnp9f9h+kTkr1Evc9kudn2B9fZtCFXbnkvI7lcCblL3vnPjLSahYsN0irQfbOfcctP+3zVfTrflJ5hBY5Ycp4ItJHRFaLyFoRGVXM9ioi8o6z/Vun8YSINBWRgyKy2Jn+5ibsiCY8EalEINm9parTiyujqpNVtauqdj0xLS2S4QBw6smN2JCVw6bsneQeyeOjud/T68xTCpXpdWY7Ppi9AID/fP4DZ3Q6CRFhU/ZO8vx+ALK27mL9xu1k1Es9po541LltY9ZvymFD1g5yj+QxfU4mfc5uX6hM3x7tmfpRoJH+77mLObtrK0SEPme3Z/qcTA7nHmFD1g7Wb8qhS7smgSfZj79Fq2b1uG1oz2icVrlYtGYrLRqk0jg9hUrJSQzscTKz5q8rVKZ5g9/+8bjwtOas27K76GHiSpLz5bJQU0lExAdMAvoCbYEri7nPfyOwW1VPAp4F/hy0bZ2qdnSm4W7ijuRTWgFeAVaq6jORqserZJ+P0bcP5Pr7J+PPz2dw3260alaP56bM4pTWjTj/rFO4ot/p3DPubXoOfYLatU7guUeuBWDh0p946e1PqZTsQ5KEx+68nDopNaJ8RuGRnOxj/H2DGXT7i/jzlaGXnEGbFvUZ99JHdGrTmL492nN1/+4MH/0GXQY+RmqtE3j5iesBaNOiPpee35nuvx9Hsi+J8fcPxudLYv7idbwzawFtT2pAj6FPAfDIiEu44Kx20TzVsPPnK/f/36dM+9Pl+JKSeGvOUlZt3MkDV5/F4h+3Muvbddx8SSfO6diEvLx89uw/xIinZ0U77DIJ0+VqN2Ctqq4HEJGpwAAK3+cfAIxx5t8HJkoZOvKKqpZ235IPLPI/wBfAUiDfWf2gqs483j4dOnXRmZ99E5F4KoI61eP/6V6k1en3dLRDiGmH5/+V/F82lSlfndKhs06f86Wrsq3rVd8ABN+rmqyqkwFEZBDQR1VvcpavAU5X1ZFHC4vIMqfMZmd5HXA6UANYDqwBfgEeVtUvQsUTyae0XxKb9y2NMWXgcQDQHaraNQJhZAONVXWniHQB/iUi7Yp7ThCsXJ7SGmMqEOfFYzdTCFlAo6Dlhs66YsuISDKQAuxU1cOquhNAVTOBdUCrUBVawjPGeBamh7QLgJYi0kxEKgNDgBlFyswAhjnzg4C5qqoiUtd56IGINAdaAutDVZiQfWmNMWURngFAVWmkNfwAAAbrSURBVDVPREYCswEfMEVVl4vIWGChqs4g8ODzTRFZC+wikBQBegBjReQIgWcEw1V1V6g6LeEZYzwLV08L5yHmzCLrHg2aPwQMLma/aQReefPEEp4xxpNY7UXhhiU8Y4x3cZrxLOEZYzyLxZFQ3LCEZ4zxLF5HS7GEZ4zxRiDJEp4xJnHEZ8azhGeM8SSeBwC1hGeM8SxO850lPGOMd9bCM8YkjHB0LYsGS3jGGM/iM91ZwjPGeORy6KeYZAnPGOOZ9bQwxiSO+Mx3lvCMMd7Fab6zhGeM8Sr0JxhjlSU8Y4wn8dzTwr5pYYxJGNbCM8Z4Fq8tPEt4xhjP7LUUY0xisBePjTGJIp4fWljCM8Z4Zpe0xpiEEa8tPHstxRjjmbicQh5HpI+IrBaRtSIyqpjtVUTkHWf7tyLSNGjbA8761SJyoZu4LeEZY7wLQ8YTER8wCegLtAWuFJG2RYrdCOxW1ZOAZ4E/O/u2BYYA7YA+wIvO8UpkCc8Y44kASSKuphC6AWtVdb2q5gJTgQFFygwAXnfm3wd6SWD00QHAVFU9rKo/AWud45Uopu7h/bB40Y6GqVU2RDuOIGnAjmgHEcPs5xNarP2MmpT1AIsWZc6uVknSXBavKiILg5Ynq+pkZz4D2BS0bTNwepH9C8qoap6I7AVOdNbPL7JvRqhgYirhqWrdaMcQTEQWqmrXaMcRq+znE1pF/Bmpap9ox1BadklrjImWLKBR0HJDZ12xZUQkGUgBdrrc9xiW8Iwx0bIAaCkizUSkMoGHEDOKlJkBDHPmBwFzVVWd9UOcp7jNgJbAd6EqjKlL2hg0OXSRhGY/n9DsZ3Qczj25kcBswAdMUdXlIjIWWKiqM4BXgDdFZC2wi0BSxCn3LrACyANuU1V/qDolkCyNMabis0taY0zCsIRnjEkYlvCKEaq7S6ITkSkisl1ElkU7llgkIo1E5DMRWSEiy0XkjmjHZALsHl4RTveUNcAFBF5mXABcqaorohpYDBGRHsB+4A1VPSXa8cQaEakP1FfVRSJSE8gELrXfoeizFt6x3HR3SWiqOo/AEzNTDFXNVtVFzvw+YCUuegGYyLOEd6ziurvYL6spFWd0j07At9GNxIAlPGMiRkRqANOAO1X1l2jHYyzhFadUXVaMCSYilQgku7dUdXq04zEBlvCO5aa7izHH5Qxf9AqwUlWfiXY85jeW8IpQ1TzgaHeXlcC7qro8ulHFFhH5J/AN0FpENovIjdGOKcacBVwD9BSRxc50UbSDMvZaijEmgVgLzxiTMCzhGWMShiU8Y0zCsIRnjEkYlvCMMQnDEl4cERG/84rDMhF5T0ROKMOxzhWRD535/iWNCiMitUVkRCnqGCMi97pdX6TMayIyyENdTW30FhOKJbz4clBVOzojlOQCw4M3SoDn/6aqOkNVnyqhSG3Ac8IzJtZYwotfXwAnOS2b1SLyBrAMaCQivUXkGxFZ5LQEa0DBOH+rRGQRMPDogUTkOhGZ6Myni8gHIrLEmc4EngJaOK3LCU65+0RkgYj8ICKPBR3rIRFZIyJfAq1DnYSI3OwcZ4mITCvSaj1fRBY6x7vYKe8TkQlBdf+hrD9Ikzgs4cUh53N1fYGlzqqWwIuq2g44ADwMnK+qnYGFwN0iUhX4O3AJ0AWod5zDPw98rqodgM7AcmAUsM5pXd4nIr2dOrsBHYEuItJDRLoQ6IrXEbgIOM3F6UxX1dOc+lYCwb02mjp19AP+5pzDjcBeVT3NOf7NzlerjAnJvloWX6qJyGJn/gsC/TUbABtU9ehX2M8A2gJfBbp0UplAN7CTgZ9U9UcAEfkHcEsxdfQErgVwvgK1V0RSi5Tp7UzfO8s1CCTAmsAHqvqrU4ebPsiniMifCFw21yDQpe+od1U1H/hRRNY759AbODXo/l6KU/caF3WZBGcJL74cVNWOwSucpHYgeBXwsapeWaRcof3KSIAnVfWlInXcWYpjvUZgNOAlInIdcG7QtqL9HtWp+39VNTgxHh13zpgS2SVtxTMfOEtETgIQkeoi0gpYBTQVkRZOuSuPs/+nwK3Ovj4RSQH2EWi9HTUbuCHo3mCGiPwOmAdcKiLVnKHNL3ERb00g2xlOaWiRbYNFJMmJuTmw2qn7Vqc8ItJKRKq7qMcYa+FVNKqa47SU/ikiVZzVD6vqGhG5BfhIRH4lcElcs5hD3AFMdkZA8QO3quo3IvKV89rHLOc+XhvgG6eFuR+42vmGwzvAEmA7gaG2QnmEwGjAOc7fwTFtJPA1+VrAcFU9JCIvE7i3t8gZhikHuNTdT8ckOhstxRiTMOyS1hiTMCzhGWMShiU8Y0zCsIRnjEkYlvCMMQnDEp4xJmFYwjPGJIz/D4wz5lofK9MPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_cm = plot_confusion_matrix(lr_mod, X, y, cmap=plt.cm.Blues,\n",
    "                                 normalize='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also perform *error analysis*, by visualizing for what class the model makes the largest number of errors. In order to do this, we can normalize the matrix by each class total (the `true` counts), and subsequently setting the diagonal elements to 0 (because the diagonal does not represent errors). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAECCAYAAADXWsr9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAHPUlEQVR4nO3dQYhVBRyF8XOayYh0IcwsQqfRhQjSRphqUbQQAmtjtMqFbYRZCQpt3Ld310ZIIopCyIWLIAqCKEKcJhfpYIhRTQiOGGgkycC/xczCauDd8t65773z/WBg3vVxPdzx476ZN6CrSgDG2yN9DwDQPUIHAhA6EIDQgQCEDgQgdCDA2Idu+6Dtq7av2T7Z955hZfuM7Zu2v+97yzCzPWP7C9tXbF+2fbzvTU14nN9Htz0h6QdJL0lalnRR0uGqutLrsCFk+0VJv0t6r6qe7nvPsLL9pKQnq2rR9jZJ30p6ddj/TY37Hf1ZSdeq6npV3Zf0kaRDPW8aSlX1paTbfe8YdlV1o6oW1z+/K2lJ0o5+Vw027qHvkPTLA4+XNQJfFIwG27sk7Zd0od8lg4176EAnbG+V9LGkE1V1p+89g4x76L9Kmnng8c71Y8D/ZvtRrUX+QVWd63tPE+Me+kVJe2zvtr1F0uuSzve8CSPMtiW9I2mpqk71vaepsQ69qlYlHZP0qdZ+aHK2qi73u2o42f5Q0jeS9tpetn20701D6nlJRyQdsH1p/eOVvkcNMtZvrwFYM9Z3dABrCB0IQOhAAEIHAhA6ECAmdNvzfW8YBVyn5kbpWsWELmlkvig94zo1NzLXKil0IFYnvzAzNTVVs7O7Wj/vw1i5taLpqem+Zwy9Yb1O3y393PeEf6nVe/Lk433P+Ju6f1e1es//PD7ZxV82O7tLX19Y6OLUCLX9mWN9TxgJf149u+FxXroDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EaBS67YO2r9q+Zvtk16MAtGtg6LYnJL0t6WVJ+yQdtr2v62EA2tPkjv6spGtVdb2q7kv6SNKhbmcBaFOT0HdI+uWBx8vrxwCMiNZ+GGd73vaC7YWVWyttnRZAC5qE/qukmQce71w/9jdVdbqq5qpqbnpquq19AFrQJPSLkvbY3m17i6TXJZ3vdhaANk0OekJVrdo+JulTSROSzlTV5c6XAWjNwNAlqao+kfRJx1sAdITfjAMCEDoQgNCBAIQOBCB0IAChAwEIHQhA6EAAQgcCEDoQgNCBAIQOBCB0IAChAwEIHQhA6EAAQgcCEDoQgNCBAIQOBCB0IAChAwEIHQhA6EAAQgcCEDoQgNCBAIQOBCB0IAChAwEIHQhA6ECAyb4HAE18fvatvieMhKOvfb3hce7oQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQICBods+Y/um7e83YxCA9jW5o78r6WDHOwB0aGDoVfWlpNubsAVAR/geHQjQWui2520v2F5YubXS1mkBtKC10KvqdFXNVdXc9NR0W6cF0AJeugMBmry99qGkbyTttb1s+2j3swC0aXLQE6rq8GYMAdAdXroDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EGPj/o/8fP97+Q2+8v9jFqcfK/HNP9T1hZLywZ6rvCSPhiS0TGx7njg4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EIHQgAKEDAQgdCEDoQABCBwIQOhCA0IEAhA4EGBi67RnbX9i+Yvuy7eObMQxAeyYbPGdV0ptVtWh7m6RvbX9WVVc63gagJQPv6FV1o6oW1z+/K2lJ0o6uhwFoz3/6Ht32Lkn7JV3oYgyAbjQO3fZWSR9LOlFVdzb483nbC7YX/rz7W5sbATykRqHbflRrkX9QVec2ek5Vna6quaqae2zb9jY3AnhITX7qbknvSFqqqlPdTwLQtiZ39OclHZF0wPal9Y9XOt4FoEUD316rqq8keRO2AOgIvxkHBCB0IAChAwEIHQhA6EAAQgcCEDoQgNCBAIQOBCB0IAChAwEIHQhA6EAAQgcCEDoQgNCBAIQOBCB0IAChAwEIHQhA6EAAQgcCEDoQgNCBAIQOBCB0IAChAwEIHQhA6EAAQgcCEDoQgNCBAK6q9k9qr0j6qfUTP5wpSbf6HjECuE7NDeO1mq2q6X8e7CT0YWR7oarm+t4x7LhOzY3SteKlOxCA0IEASaGf7nvAiOA6NTcy1yrme3QgWdIdHYhF6EAAQgcCEDoQgNCBAH8BjAg+R6Tz2S8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "norm_conf_mat = confusion_matrix(y_test, y_pred_lr, normalize='true')\n",
    "np.fill_diagonal(norm_conf_mat, 0)\n",
    "plt.matshow(norm_conf_mat, cmap=plt.cm.Blues)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this plot, it results clear that the classifier has more difficulties at classifying instances of class 0, which are most often predicted in class 2. \n",
    "<br>\n",
    "While there is no direct way to calculate the $f_1$ score in the multi-class case, we can exploit the One-vs-Rest approach used by `sklearn`. That is, it is possible to consider the models computed using each class as the positive one, and averaging the resulting $f_1$ scores with the methods seen in the binary case (macro and weighted):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.89      0.91        35\n",
      "           1       0.97      0.97      0.97        34\n",
      "           2       0.85      0.90      0.88        31\n",
      "\n",
      "    accuracy                           0.92       100\n",
      "   macro avg       0.92      0.92      0.92       100\n",
      "weighted avg       0.92      0.92      0.92       100\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, y_pred_lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we can query the weighted $f_1$ score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9203676470588235"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f1_score(y_test, y_pred_lr, average='weighted')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}