Solutions to Chapter 8 Support Vector Machine.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add lib input sys.path\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import sklearn\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.optimize import minimize\n",
    "import math\n",
    "from sklearn.preprocessing import normalize\n",
    "from functools import partial\n",
    "import h5py\n",
    "from scipy.spatial import distance\n",
    "\n",
    "nb_dir = os.path.split(os.getcwd())[0]\n",
    "if nb_dir not in sys.path:\n",
    "    sys.path.append(nb_dir)\n",
    "\n",
    "from matplotlib.colors import ListedColormap\n",
    "import libs.linear_models as lm\n",
    "import libs.data_util as data\n",
    "import libs.nn as nn\n",
    "import libs.plot as myplot\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 8.1\n",
    "\n",
    "* (a) If there's such a hyperplane that can tolerate noise radius greater than $\\frac{1}{2}|x_{+}-x_{-}|$, we draw a line between two points, for $(x_{+}, +1)$, we can pick a point on the line that just pass the middle point between $x_{+}$ and $x_{-}$ and still within the radius (which is greater than $\\frac{1}{2}|x_{+}-x_{-}|$) of $x_{+}$. This point will have label $+1$. However, it's obviously also in the radius of $x_{-}$, so it shall have a label of $-1$ as well. It is impossible to classify such point. This thus contradicts the fact that a hyperplane exists to tolerate such a noise radius. Our assumption is wrong, there's no such hyperplane that can tolerate noise radius greater than $\\frac{1}{2}|x_{+}-x_{-}|$.\n",
    "\n",
    "* (b) We can choose the hyperplane that perpendicular to the line between $x_{+}$ and $x_{-}$ and passes through the middle point. The two balls with radius of $\\frac{1}{2}|x_{+}-x_{-}|$ and centered at $x_{+}, x_{-}$ are separated by this hyperplane totally. The projection of any point in the ball of $x_{+}$ on the norm of the hyperplane is positive.\n",
    "\n",
    "Thus the hyperplane can tolerate such a noise radius.\n",
    "\n",
    "#### Exercise 8.2\n",
    "\n",
    "* (a) \n",
    "\n",
    "\\begin{align*}\n",
    "Y\\otimes (Xw + b) &= Y \\otimes ( \\begin{bmatrix}0 \\\\ -4 \\\\ 2.4\\end{bmatrix}-0.5)\\\\\n",
    "&= \\begin{bmatrix}-1 \\\\ -1 \\\\ +1\\end{bmatrix} \\otimes \\begin{bmatrix}-0.5 \\\\ -4.5 \\\\ 1.9\\end{bmatrix}\\\\\n",
    "&= \\begin{bmatrix}0.5\\\\ 4.5 \\\\ 1.9\\end{bmatrix}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "So $\\rho = \\min y_n(w^Tx_n +b) = 0.5$\n",
    "\n",
    "* (b) So we have the new weights $w' = \\frac{1}{\\rho}(b,w) = (-1, \\begin{bmatrix}2.4 \\\\ -6.4 \\end{bmatrix})$\n",
    "\n",
    "We compute\n",
    "\n",
    "\\begin{align*}\n",
    "Y\\otimes (Xw' + b) &= Y \\otimes ( \\begin{bmatrix}0 \\\\ -8 \\\\ 4.8\\end{bmatrix}-1)\\\\\n",
    "&= \\begin{bmatrix}-1 \\\\ -1 \\\\ +1\\end{bmatrix} \\otimes \\begin{bmatrix}-1 \\\\ -9 \\\\ 3.8\\end{bmatrix}\\\\\n",
    "&= \\begin{bmatrix}1\\\\ 9 \\\\ 3.8\\end{bmatrix}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "The new minimum is $\\rho' = 1$ which satisfies equation (8.2)\n",
    "\n",
    "* (c) It's easy to see that the two hyperplanes are the same in 2-Dimensional space.\n",
    "\n",
    "\n",
    "#### Exercise 8.3\n",
    "\n",
    "If $h$ is the optimal hyperplane, its $w_h$ minimizes $w^Tw$. Also, for a separating hyperplane, the data point that is nearest to the hyperplance has distance $\\frac{1}{|w_h|}$. Assume $\\rho_{+} > \\rho_{-}$, so we should have $\\rho_{-} = \\frac{1}{|w_h|}$. \n",
    "\n",
    "Let $l=\\frac{\\rho_{+}-\\rho_{-}}{2}$, and $\\rho'_{+} = \\rho_{+} - l = \\frac{\\rho_{+}+\\rho_{-}}{2} > \\frac{1}{|w_h|}$, $\\rho'_{-} = \\rho_{-} + l = \\frac{\\rho_{+}+\\rho_{-}}{2} > \\frac{1}{|w_h|}$.\n",
    "\n",
    "If we let $w' = \\rho'_{+} = \\rho'_{-}$, then we have $|w'| < |w_h|$, which contradicts the fact that $w_h$ minimizes $w^Tw$.\n",
    "\n",
    "\n",
    "#### Exercise 8.4\n",
    "\n",
    "Suppose $Y=\\begin{bmatrix}y^1 & y^2 & \\dots & y^N\\end{bmatrix}$ and $X=\\begin{bmatrix}1 & x^T_1 \\\\ \\dots & \\dots \\\\ 1 & x^T_N\\end{bmatrix}$, where the columns of $Y$ is $y^n = \\begin{bmatrix}0 \\\\ \\dots \\\\ y_n \\\\ \\dots \\end{bmatrix}$, the only non-zero entry is at the $n$-th row with value of $y_n$.\n",
    "\n",
    "We use column times row in matrix multiplication, so \n",
    "\n",
    "\\begin{align*}\n",
    "A &=YX \\\\\n",
    "&= \\sum^N_{n=1} \\begin{bmatrix}0 \\\\ \\dots \\\\ y_n \\\\ \\dots \\end{bmatrix} \\begin{bmatrix}1 & x^T_n\\end{bmatrix}\\\\\n",
    "&= \\sum^N_{n=1} \\begin{bmatrix}0 & 0^T_d \\\\ \\dots & \\dots \\\\ y_n & y_nx^T_n \\\\ \\dots & \\dots \\\\ 0 & 0^T_d \\end{bmatrix}\\\\\n",
    "&= \\begin{bmatrix}y_1 & y_1x^T_1 \\\\ \\dots & \\dots \\\\ y_n & y_nx^T_n \\\\ \\dots & \\dots \\\\ y_N & y_Nx^T_N \\end{bmatrix}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "This is exactly the formula we are seeking for.\n",
    "\n",
    "#### Exercise 8.5\n",
    "\n",
    "It's clear that the eigenvalues of $Q$ are eigher 0 or 1, so we know $Q$ is positive semi-definite.\n",
    "\n",
    "#### Exercise 8.6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_data_84(N, x1_lb=0, x1_ub=1, x2_lb=-1, x2_ub=1):\n",
    "    dim = 1\n",
    "    max_v = 10000\n",
    "    x1 = data.generate_random_numbers(N, dim, max_v, x1_lb, x1_ub)\n",
    "    x2 = data.generate_random_numbers(N, dim, max_v, x2_lb, x2_ub)\n",
    "    df = pd.DataFrame({'x1':x1.flatten(), 'x2':x2.flatten()})\n",
    "    df['y'] = np.sign(df['x2'])\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- Number of positive points: 13\n",
      "-- In-Sample Error: 0.0\n",
      "-- SVM Margin: 0.127515635876689\n",
      "-- Out-of-Sample Error: 0.04\n"
     ]
    }
   ],
   "source": [
    "N = 20\n",
    "N_t = 100\n",
    "df = generate_data_84(N)\n",
    "print(f\"-- Number of positive points: {df.loc[df['y']>0]['y'].count()}\")\n",
    "X = df[['x1', 'x2']].values\n",
    "y = df['y'].values\n",
    "my_svm = lm.SVM()\n",
    "my_svm.fit(X, y)\n",
    "print(f\"-- In-Sample Error: {my_svm.calc_error(X, y)}\")\n",
    "print(f\"-- SVM Margin: {my_svm.margin()}\")      \n",
    "df_t = generate_data_84(N_t)\n",
    "X_t = df_t[['x1', 'x2']].values\n",
    "y_t = df_t['y'].values\n",
    "print(f\"-- Out-of-Sample Error: {my_svm.calc_error(X_t, y_t)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEICAYAAABbOlNNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHpBJREFUeJzt3XucHWWd5/HPN4kksknMPUIuECCMCmjQBoZlQQWiICPgDgOBQYNGk3EFdxfHAYYd18HLwozCrMprJBBuwoAXBojKRa7bEommeRkuiQOdRIQ2SEJIJE24GPq3f1Q1nDqc091J1elz6e/79Tqvc6rqqaqnKyfne56qp56jiMDMzKzXsHpXwMzMGouDwczMMhwMZmaW4WAwM7MMB4OZmWU4GMzMLMPBYC1N0l9L+lm967EjJE2W9LikUQMo+25JvxiMetnQ4WCwmpH0pKSXJHWXPL4zmHWIiOsj4kNFb1fSWZJ+K+kFSR2S/ks/5edJ+o2kFyWtlXR4H8XPBa6KiJf7q0dEPAJskfTRPva9n6SfSdosaYukhyR9RNI0Sdsl7V1hnZslfSN9HZKelTSiZPkISRsk+UaoFuRgsFr7aESMLnmcWeTGSz+sBoukQ4ALgZOAtwFLgJslDa9Sfi5wEfBJYAxwBLCuStmRwHzguh2o0vXAoj6W/xi4C5gKTAE+D7wQEb8H7gE+XlaHCcBHgGtKZm8Bji2Z/giweQfqaE3EwWB1IelfJf2oZPoiSfdIUjr9F5JWpt9wfyHp3SVln5R0jqRHgBfTb68zJP27pI2SNvW2TCSdIemB9LUkXZJ+0/2jpEck7Z8uGynpG5KeSr8df1fSW6tUf09gVUQ8FMnQAdcCk0g+dCv5R+CCiFgeET0R8fv0Q7mSQ4AtEdFV8vd+Mm1tbJW0TlJ5CNwPHJWGSvlxngTMAi6PiFfTx7KIeCAtcg1lwQDMS/++R0vmfQ/4RMn0J9K/21qQg8Hq5QvAu9MP7sOBBcD8iAhJ7wWuJPkWPBG4DFha9sF3KnAcMA4I4CfA70g+tKcBN1bY54dIvq3vm653CrApXXZROn8OsE+6jS9VqfvtwHBJh6SthE8BK4E/lBdMl7cBkyWtkdQl6Tt9hM4BwONl8zYAfwGMJWl1XJIeIwDSkPkT8GcVtrcJWANcJ+lESVPLlt8MTCo7FfZx3vyhfwtwhKRxksYBhwO3VvkbrNlFhB9+1OQBPAl0k5yG6H18pmT5wcDzJB/op5bM/1fgK2Xbehx4f8l2P1Wy7FBgIzCiQh3OAB5IXx8JPAH8OTCspIyAF4G9y7b52yp/l4C/J/kw3g48BxxUpezuJMHVAexG0rJYBnytSvnzgRv7Oa63AP+9bN7vgSOqlJ8OfAdYC/QA7cDskuVXAIvT17OBV4EpJcuDJCyvIAnrvwEuT+dFvd9nfhT/cIvBau3EiBhX8ri8d0FE/IrkXLuAH5SsswfwhfQ00hZJW4AZJB+yvZ4ueT0D+F1EbO+rIhFxL8kH5KXAs5IWSxoLTAZ2BR4q2d8d6fxKPk3SStgP2AU4HfiJpN0rlH0pff52RDwTEc8BF5Oco69kM8l1iNdJOlbScknPp3X7CEnAlBpDEryV/u6uiDgzIvYmObYvkm0RXAOcnPaC+jhwR0RsqLCpa0lOIfk0UotzMFjdSPocMBJYD/xdyaKnSb5RlwbKrhFxQ0mZKCs/cyAXoiPiWxHxPpIP9X2BL5J8438J2K9kf2+LiNFVNvMe4McR8UQk1wzuAJ4B/nOF/W0Gusrq25dH0noBr1+Mvgn4BjA1IsYBt5GEaW+Z3UkCqvwU1JtExNMkwbh/ybyfk5xyOoEk5Kp96P+cpNUzFXigShlrAQ4GqwtJ+wJfJfkg+jjwd5LmpIsvB/4mPYcvSf9J0nGSxlTZ3K9IPpgvTMuOknRYhX0elG7zLSTfml8GXouInnSfl0iakpadJunDVfa3AjhO0l5p/eaSfJg/VqX8VcBZkqZIGg/8D5JrItX+lnGSpqXTu5CE50Zgu6RjSa6VlPoAcG9EvFLhbx4v6R8l7SNpWHox+lPA8rKi15JcZxlH0ovpTSIigI8Cx6evrUU5GKzWfqzsfQw3p9/srwMuioiHI6KT5Jz99ySNjIgO4DMkp302k1w8PaPaDiLiNZIPrH2Ap0i+oZ9SoehYkgDYTHJdYxPJN3GAc9L9LJf0AnA3lS/mQvIheiNJb6AXgG8BiyLiPwAk/b2k20vKf4UkTJ4AfgP8Gvhalb/lVeBqksAkIraSdC/9QVrv04ClZav9NfDdKnV9leSC/N1pXR8DXuHNx/NaYCbw/UoBU1K/VRGxqtpyaw1y8Js1FkmTSU7bHBgRL/VT9gCSC8eHDkrlbEhwMJiZWYZPJZmZWYaDwczMMhwMZmaWMegDkBVh0ujRsefEifWuhplZU3noqaeei4hqN26+rimDYc+JE+k4//x6V8PMrKlo0aLfDaScTyWZmVmGg8HMzDIcDGZmltGU1xjMzGrhTyNG0LXXXry86671rkouo7ZtY/q6dbxle58DDlflYDAzS3XttRdjZsxgzzFjSH9MsOlEBJu2bqULmPXEEzu1DZ9KMjNLvbzrrkxs4lAAkMTEMWNytXocDGZmJZo5FHrl/RscDGZmluFgMDNrEj9cupT9Dj+cYVOn0rFyZc3242AwM2tA9y9bxhlnnZWZt/873sG/X3UVRxxa25/fcK8kM7Mm8c599+2/UAHcYjAzy6OnBzZsgBb60TO3GMzMdlZPD3zsY7BiBRx0ENx8MwzL9337kGOO4ZVXXqH7xRd5fssW5nzwgwBc9A//wIePPLKIWvfLwWBmtrOeey4Jhe3bk+fnnoMpU3Jt8pd33AEk1xiuvvFGrv72t4uo6Q7xqSQzs501eXLSUhgxInme3O9PHTQFB4OZ2c6SktNHK1fCLbck0zV0809/yvT3vIcHOzo47rTT+PDJJ9dkPz6VZGaWx7BhuU8fVfKBww7jA4cdlpn3seOO42PHHVf4vsq5xWBmZhkOBjMzy3AwmJlZRiHBIOkYSY9LWiPp3ArLL5G0Mn08IWlLybLXSpYtLaI+Zg2lpwdeeKGlboCy1pb74rOk4cClwFygC1ghaWlErO4tExH/s6T8WcCBJZt4KSLm5K2HWUPq6YGLL4a1a2HvveHss3PfAGVWa0W8Qw8G1kTEuoh4FbgROKGP8qcCNxSwX7PG192dhEJPT/Lc3V3vGpn1q4hgmAY8XTLdlc57E0l7ALOAe0tmj5LUIWm5pBMLqI9Z4xgzJmkpDBuWPI8ZU+8aWZP4j85ODj32WEZOn843Lr10UPddxH0Mle7oqHYydR7wo4h4rWTezIhYL2kv4F5Jj0bE2jftRFoILASYOWFC3jqbDQ4pOX3U3Z2EQgv8OpgNjgnjxvGtr3+dW26/fdD3XUSLoQuYUTI9HVhfpew8yk4jRcT69HkdcD/Z6w+l5RZHRFtEtE0ePTpvnc0Gz7BhMHasQ8F2yJTJkznowAN5y4jBvw+5iGBYAcyWNEvSLiQf/m/qXSTpz4DxwIMl88ZLGpm+ngQcBqwuX9fMzAZP7iiKiO2SzgTuBIYDV0bEKkkXAB0R0RsSpwI3RmT67L0TuExSD0lIXVjam8nMrNFpSjJwXmzYWOeaFKeQNkpE3AbcVjbvS2XTX66w3i+AA4qog5kNIT09LXnd5tIlS7j8uusAuO2GG9j97W+vSz08iJ6ZNZcGuzekyJbC5xYs4HMLFhS2vZ3lYDCz5lLp3pCxY+tdq8L94dlnafvQh3hh61aGDRvGvyxezOoHHmDsIHR5djCYWXPpvTekt8XQoveGvH3qVLoefrgu+3YwmFlz8b0hNedgMLPm03tviNWER/Mys+ZW8Oi10QKj4Ob9G9xiMGtmLdptc8AK7qE0ats2Nm3dysQxY1CTHs+IYNPWrYzatm2nt+FgMGtWDdZtsy4K7qE0fd06uoCNu+5aXB3rYNS2bUxft26n13cwmDWrIdJts08F91B6y/btzHriiYIq17wcDGbNaoh02+yTeyjVhIPBrFn5QzHhHkqFczCYNTN/KFoNDLErVWZm1h8Hg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMgoJBknHSHpc0hpJ51ZYfoakjZJWpo9PlyybL6kzfcwvoj5mZrbzct/gJmk4cCkwF+gCVkhaGhGry4p+PyLOLFt3AvC/gTYggIfSdTfnrZeZme2cIloMBwNrImJdRLwK3AicMMB1PwzcFRHPp2FwF3BMAXUysyIU/FsH1hyKCIZpwNMl013pvHJ/KekRST+SNGMH10XSQkkdkjo2dncXUG0z61PvsN7nnAPf/GYybUNCEcFQaeSu8q8XPwb2jIh3A3cD1+zAusnMiMUR0RYRbZNHj97pyprZAFUa1tuGhCKCoQuYUTI9HVhfWiAiNkXEK+nk5cD7BrqumdVJ77Dew4YN3WG9h6giRlddAcyWNAv4PTAPOK20gKTdIuKZdPJ44Dfp6zuBr0san05/CDivgDqZWV4e1nvIyh0MEbFd0pkkH/LDgSsjYpWkC4COiFgKfF7S8cB24HngjHTd5yV9hSRcAC6IiOfz1snMCuJhvYekQn6PISJuA24rm/elktfnUaUlEBFXAlcWUQ8zM8vPdz6bmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMgoJBknHSHpc0hpJ51ZYfrak1ZIekXSPpD1Klr0maWX6WFpEfZpeTw+88AJE1LsmZjYE5f7NZ0nDgUuBuUAXsELS0ohYXVLs10BbRGyT9Fngn4BT0mUvRcScvPVoGT09cPHFsHYt7L03nH128oPsZmaDpIhPnIOBNRGxLiJeBW4ETigtEBH3RcS2dHI5ML2A/bam7u4kFHp6kufu7nrXyMyGmCKCYRrwdMl0VzqvmgXA7SXToyR1SFou6cQC6tPcxoxJWgrDhiXPY8bUu0ZmNsTkPpUEqMK8iifHJZ0OtAHvL5k9MyLWS9oLuFfSoxGxtsK6C4GFADMnTMhf60YlJaePuruTUFClw2tmVjtFtBi6gBkl09OB9eWFJB0NnA8cHxGv9M6PiPXp8zrgfuDASjuJiMUR0RYRbZNHjy6g2g1s2DAYO9ahYGZ1UUQwrABmS5olaRdgHpDpXSTpQOAyklDYUDJ/vKSR6etJwGFA6UVrMzMbZLlPJUXEdklnAncCw4ErI2KVpAuAjohYCvwzMBr4oZJvwU9FxPHAO4HLJPWQhNSFZb2ZzKwoPT2NcYqyUephVRVxjYGIuA24rWzel0peH11lvV8ABxRRBzPrQ6N0g26UelifWutfxDeG2VAz0Pd8o3SDbpR6WJ8KaTE0BH8TsaFmR97zvd2ge8vWqxt0o9TD+tQ6wVDpm8jYsfWulVnt7Mh7vlG6QTdKPaxPrfOV2jeG2VCzo+/5RukG3Sj1sKpap8XgbyI21Pg9bzXSOsEAb3wTMRsq/J63GmidU0lmrcq97WyQtVaLwazVuLed1YHfYWaNzP3+rQ4cDGaNzL3trA58KsmskbnnkdWBWwxmjc79/m0gCuyk4BaDmVmzK7iTglsMZmbNruBOCg4GM7NmV3AnheY8lfTcc7BkSbHbnD0bjjii2G2amQ2GgjspNGUwdG8fxQMbZhe2valbO5m6dhljly2DKVMcEmbWfAocHqUpg+HpEbP4wpTritvgFNiwAS7eeDqzN3YyedUyppaGRNEcOtaK/JOdLaMpg2HSJFiwoPjt3tR+HZ2dSUicunUxp21cwsi1nYXvZ+odyxg7BrdOrHV46I6WUkgwSDoG+L/AcOCKiLiwbPlI4FrgfcAm4JSIeDJddh6wAHgN+HxE3FlEnXbGEUe88Rnd3r6QBZ0LC99Hb+gctLGd/X0Ky1qFfyirpeQOBknDgUuBuUAXsELS0ohYXVJsAbA5IvaRNA+4CDhF0ruAecB+wO7A3ZL2jYjX8tYrr9KQKN5CbmpfyIWdcNTabEhwx7Ja7bSisXs7kKwA/snOlqLIeZecpEOBL0fEh9Pp8wAi4v+UlLkzLfOgpBHAH4DJwLmlZUvL9bXPPfZoi/PP78hV70bS3s7rp7AG06lbF3M47bz35WW8dZRDwnLyNYaGp0WLHoqItv7KFXEqaRrwdMl0F3BItTIRsV3SH4GJ6fzlZetOq7QTSQuBhQATJswsoNqNo7atk+ra2xfy5c6Fr5/eOnxtO/us7XzjGkjRfLqstflHg1pGEcFQ6atBeTOkWpmBrJvMjFgMLIakxbAjFbTKyq+plIYEW4vd10EvtzN7YyczfU3FrOEVEQxdwIyS6enA+iplutJTSW8Dnh/gujYIsq2W4i+639S+8E09vjLdgmvBwWO2U4oIhhXAbEmzgN+TXEw+razMUmA+8CBwEnBvRISkpcC/SbqY5OLzbOBXBdTJGkylHl+9IVH0Was5W9vZB/f4MttZuYMhvWZwJnAnSXfVKyNilaQLgI6IWAosAb4naQ1JS2Feuu4qST8AVgPbgc81Qo8kq63ykCjaKpLWSXmPr7G1bJ1U40CyJpS7V1I9tFqvJKud3h5fR61dzEEvtzNq1OD2pJy90T2+rHEMZq8ks4b1Rutk4evXOQbTBqr0+DrssNrs0MFjBXCLwazGSu9T6b13ZB9qMNQKGzzUivXJLQazBlHe4+vLS4q/rgJ99PhySNgOcjCYDbJaDADZq7zHV29IvLUGQ624ddK6HAxmLaRat+CieSDI1uZrDGa2U8p7fM2mk5mjNtRmOJW+OJAGzNcYhqBFi5Lnyy6rbz1saKjU4+v1IVUG0eFb23lverrs9R5fDolcHAxmlluth1Sppr2d7ECQW9vZ5w4PBJmXTyWZWctYsuSNVkstbmT86MYlTGYDUyfTlCHhU0lmNuS80eNrIe3txW//lio9vsbu3VoDQToYzKwl1eLztNpAkIevbS+8hTK1jj2+fCrJzCyH3pZJLYZbqdjjqzckdoKuv96nkszMaq33S3xtvsxne3xdvPF0Zm/sZOTa2g765WAwM2tgpT2+bmq/Ll/LZNPAfovbwWBm1iTy/j78rwb4M2jDdn4XZmbWihwMZmaW4WAwM7OMXMEgaYKkuyR1ps/jK5SZI+lBSaskPSLplJJlV0v6raSV6WNOnvqYmVl+eVsM5wL3RMRs4J50utw24BMRsR9wDPAvksaVLP9iRMxJHytz1sfMzHLKGwwnANekr68BTiwvEBFPRERn+no9yc/gTs65XzMzq5G8wTA1Ip4BSJ/7HDBE0sHALsDaktlfS08xXSJpZB/rLpTUIamju3tjzmqbmVk1/QaDpLslPVbhccKO7EjSbsD3gE9GRE86+zzgHcBBwATgnGrrR8TiiGiLiLbRo93gMDOrlX5vcIuIo6stk/SspN0i4pn0g7/ijwhKGgv8FPhfEbG8ZNvPpC9fkXQV8Lc7VHszMytc3lNJS4H56ev5wK3lBSTtAtwMXBsRPyxbtlv6LJLrE4/lrI+ZmeWUNxguBOZK6gTmptNIapN0RVrmZOAI4IwK3VKvl/Qo8CgwCfhqzvqYmVlOucZKiohNwFEV5ncAn05fXwdcV2X9I/Ps38zMiuc7n83MLMPBYGZmGQ4GMzPLcDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8vIFQySJki6S1Jn+jy+SrnXJK1MH0tL5s+S9Mt0/e9L2iVPfczMLL+8LYZzgXsiYjZwTzpdyUsRMSd9HF8y/yLgknT9zcCCnPUxM7Oc8gbDCcA16etrgBMHuqIkAUcCP9qZ9c2sdSxalDysMeQNhqkR8QxA+jylSrlRkjokLZfU++E/EdgSEdvT6S5gWrUdSVqYbqOju3tjzmqbmVk1I/orIOlu4O0VFp2/A/uZGRHrJe0F3CvpUeCFCuWi2gYiYjGwGGCPPdqqljOz5nPZZfWugZXqNxgi4uhqyyQ9K2m3iHhG0m7AhirbWJ8+r5N0P3AgcBMwTtKItNUwHVi/E3+DmZkVKO+ppKXA/PT1fODW8gKSxksamb6eBBwGrI6IAO4DTuprfTMzG1x5g+FCYK6kTmBuOo2kNklXpGXeCXRIepgkCC6MiNXpsnOAsyWtIbnmsCRnfczMLKd+TyX1JSI2AUdVmN8BfDp9/QvggCrrrwMOzlMHMzMrlu98NjOzDAeDmZllOBjMzCzDwWBmZhkOBjMzy3AwmJlZhoPBzMwyHAxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZlYDzfyrdA4GMzPLyDW6qpmZVdbMv0rnFoOZmWU4GMzMLMPBYGZmGQ4GMzPLyBUMkiZIuktSZ/o8vkKZD0paWfJ4WdKJ6bKrJf22ZNmcPPUxM7P88rYYzgXuiYjZwD3pdEZE3BcRcyJiDnAksA34WUmRL/Yuj4iVOetjZmY55Q2GE4Br0tfXACf2U/4k4PaI2JZzv2ZmViN5g2FqRDwDkD5P6af8POCGsnlfk/SIpEskjay2oqSFkjokdXR3b8xXa7MCNfMdrmaV9BsMku6W9FiFxwk7siNJuwEHAHeWzD4PeAdwEDABOKfa+hGxOCLaIqJt9OjJO7JrMzPbAf3e+RwRR1dbJulZSbtFxDPpB/+GPjZ1MnBzRPypZNvPpC9fkXQV8LcDrLdZw2jmO1zNKsl7KmkpMD99PR+4tY+yp1J2GikNEySJ5PrEYznrY2ZmOeUNhguBuZI6gbnpNJLaJF3RW0jSnsAM4P+VrX+9pEeBR4FJwFdz1sfMzHLKNYheRGwCjqowvwP4dMn0k8C0CuWOzLN/MzMrnu98NjOzDAeDmZllOBjMzCzDwWDWgnzTneXhYDAzswz/tKdZC/JNd5aHWwxmZpbhYDAzswwHg5mZZTgYzMwsw8FgZmYZDgYzM8twMJiZWYaDwczMMhwMOXjYATNrRQ4GMzPL8JAYOXjYATNrRW4xWMvzKT+zHeNgMDOzjFzBIOmvJK2S1COprY9yx0h6XNIaSeeWzJ8l6ZeSOiV9X9IueepjVslll/m0n9mOyNtieAz4r0B7tQKShgOXAscC7wJOlfSudPFFwCURMRvYDCzIWR8zM8spVzBExG8i4vF+ih0MrImIdRHxKnAjcIIkAUcCP0rLXQOcmKc+ZmaW32D0SpoGPF0y3QUcAkwEtkTE9pL506ptRNJCYGE6+cqiRXqsBnVtVpOA5+pdiQbi4/FmPiZZQ/V47DGQQv0Gg6S7gbdXWHR+RNw6gH2owrzoY35FEbEYWJzWqSMiql7TGGp8PLJ8PN7MxyTLx6Nv/QZDRBydcx9dwIyS6enAepK0HidpRNpq6J1vZmZ1NBjdVVcAs9MeSLsA84ClERHAfcBJabn5wEBaIGZmVkN5u6t+TFIXcCjwU0l3pvN3l3QbQNoaOBO4E/gN8IOIWJVu4hzgbElrSK45LBngrhfnqXcL8vHI8vF4Mx+TLB+PPij54m5mZpbwnc9mZpbhYDAzs4yGDoZqQ2mULB+ZDqWxJh1aY8/Br+XgGcDxOFvSakmPSLpH0oD6LDer/o5HSbmTJEVfw7a0goEcD0knp++RVZL+bbDrONgG8H9mpqT7JP06/X/zkXrUs+FEREM+gOHAWmAvYBfgYeBdZWX+G/Dd9PU84Pv1rnedj8cHgV3T158d6scjLTeGZMiW5UBbvetd5/fHbODXwPh0ekq9690Ax2Qx8Nn09buAJ+td70Z4NHKLoeJQGmVlTiAZSgOSoTWOSofaaEX9Ho+IuC8itqWTy0nuDWlVA3l/AHwF+Cfg5cGsXB0M5Hh8Brg0IjYDRMSGQa7jYBvIMQlgbPr6bfheKqCxTyVVGkqjfMiM18tE0i32jyTdXlvRQI5HqQXA7TWtUX31ezwkHQjMiIifDGbF6mQg7499gX0lLZO0XNIxg1a7+hjIMfkycHra7f424KzBqVpja+RfcBvIkBk7NKxGkxvw3yrpdKANeH9Na1RffR4PScOAS4AzBqtCdTaQ98cIktNJHyBpTf5c0v4RsaXGdauXgRyTU4GrI+Kbkg4Fvpcek57aV69xNXKLodpQGhXLSBpB0hR8flBqN/gGcjyQdDRwPnB8RLwySHWrh/6Oxxhgf+B+SU8Cfw4sbeEL0AP9/3JrRPwpIn4LPE4SFK1qIMdkAfADgIh4EBhFMsDekNbIwVBxKI2yMktJhtKAZGiNeyO9itSC+j0e6amTy0hCodXPH/d5PCLijxExKSL2jIg9Sa65HB8RHfWpbs0N5P/LLSQdFJA0ieTU0rpBreXgGsgxeQo4CkDSO0mCYeOg1rIBNWwwRJWhNCRdIOn4tNgSYGI6pMbZQNUui81ugMfjn4HRwA8lrZRU/p+gZQzweAwZAzwedwKbJK0mGafsixGxqT41rr0BHpMvAJ+R9DBwA3BGC3+5HDAPiWFmZhkN22IwM7P6cDCYmVmGg8HMzDIcDGZmluFgMDOzDAeDmZllOBjMzCzj/wMSgcr8SIoClQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xsp1 = df.loc[df['y']==1]['x1'].values\n",
    "ysp1 = df.loc[df['y']==1]['x2'].values\n",
    "xsm1 = df.loc[df['y']==-1]['x1'].values\n",
    "ysm1 = df.loc[df['y']==-1]['x2'].values\n",
    "\n",
    "#plt.tight_layout()\n",
    "X_train = df[['x1', 'x2']].values\n",
    "y_train = df['y'].values\n",
    "cls = my_svm\n",
    "x1_min, x1_max = 0, 1\n",
    "x2_min, x2_max = -1, 1\n",
    "xx1, xx2 = myplot.get_grid(x1_min, x1_max, x2_min, x2_max, step=0.02)\n",
    "myplot.plot_decision_boundaries(xx1, xx2, 2, cls)\n",
    "\n",
    "myplot.plt_plot([xsp1, xsm1], [ysp1, ysm1], 'scatter', \n",
    "               colors = ['r', 'b'], markers = ['o', '+'], labels = ['+1', '-1'], \n",
    "               title = \"Exercise 8.6 (a) SVM\", yscale = None, ylb = -1, yub = 1,\n",
    "               xlb = 0, xub = 1, xlabel = None, ylabel = None,\n",
    "               legends = ['+1', '-1'], legendx = None, legendy = None, marker_sizes=[5, 5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 8.6 (b)\n",
    "\n",
    "Picked one positive point on the left and one negative on the right lower corner of the plot, so their perpendicular bisector separating plane is a bad separator. \n",
    "\n",
    "We need put them at the end of the points so the PLA algo will have no chance to adjust the separating line using other points. \n",
    "\n",
    "In this case, it took 16 iterations to make PLA converge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = pd.DataFrame({'x0':np.ones(N)})\n",
    "df1 = df.copy()\n",
    "df1.insert(0,'x0',df2['x0'])\n",
    "\n",
    "indexes = np.arange(N)\n",
    "indexes[18] = 0\n",
    "indexes[0] = 18\n",
    "indexes[19] = 17\n",
    "indexes[17] = 19\n",
    "df1 = df1.reindex(indexes)\n",
    "#df1.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of positive points:  13\n",
      "Number of negatives points:  7\n",
      "Final correctness:  20 . Total iteration:  16\n",
      "Final w: [0.         1.50015002 3.00990099]\n",
      "-- Out-of-Sample Error: 0.13\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf0AAAEVCAYAAADq2zxnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPXZ///XlYR9j4R9CSCI7FvxtrLVAmrdi1q3ije41H5729alar1/yk3rWpfau3pXalVQq1atShFlE3C3BpFVEAj7JgpCZAtJrt8f50ycJBMIS2Ymmffz8ZhH5pzzOWeuSZTrs53PMXdHREREqr+0RAcgIiIi8aGkLyIikiKU9EVERFKEkr6IiEiKUNIXERFJEUr6IiIiKUJJP0HMLNvMdpjZnPA1q5I+549mlnUU599mZp+Y2b/N7IZyylxhZrPMbLaZXRrjeLqZvW5mGeWc383M/ljOsdXhdd8Pf09nHSLeYWbWqyLf7XCEf69zKli2hZk9eKxjiPE5dcxshpm9Z2YfmdkZUcd+G/7O3jaz7MqORUSqhpj/CEvczHP34cfiQmaW7u6Fpfe7+6+O4poNgDFAV8CApWb2uLvvjirTHRgODPfyF304D5jr7gWxDrr7UjPrZGZZ7r6t1OFCd/9B+FmtgKlmtsLdl5fzWcOAlcDCin3LCssGzgEmH6qgu28BbjzGnx9LAXC1u68xs6bA+8CbZtYVONXdTzGzIcC9wMVxiEdEkpxa+knGzP4vbDmnmdk0Mzsp3H+Pmc01sw8jrV0zG2dmT5vZZOAiM/tBVIv44bDMHDNrY2bdw3Nnm9mb4bG2ZvZG2Bp8I0aPwF5gE1AnfO0FDpQqcwGwG5huZq+aWZsYX+siYGr4mTXN7EkzezeMpXdYZiZw7sF+N+6+CXg0/EzC38+csBfiZDPLBK4Ebg/3p5vZxPD9p7Fa6mb2azP7OIzll+G+C8P43jOzO8KiNwBnhtfqH3W+mdnfo77PkLBXYGZ4/KXwnPfN7Otw39DwbznHzP5iZnaw732Q38cBd18Tbu4DisL3w4A3wjLvAL3LnCwiKUkt/cTqb2Zzwvcb3f0y4NfA28ApwCx3/9jMTgeauPtQM6sLfGhmb4Tn7Xf3c8LE8Tkw1N23mll6qc86DXjK3SeYWaSy9wfgd+7+kZmdC9wC3BQ5wd0LzGwqsJyggvh7d88vdd1WQCYwEjgTeICyrcruwKrw/VXAFncfA0EPRbj/C+D0Q//KWA/0Dd//2N13m9mJwKPufqqZPQ2sdPdnw+v/PCxzHDCXsi31y4AfuHteWNFqQtBKH+zuB8KKTE/gIeByd7+q1PmZQHtgkLt7+LttFzno7heGcTwNPBb+nf4IDHP3nWHl7ExgSuQcM+sAPBXju09w97+X83t5BLg/KqZNUcdK/7cgIilKST+xynTvu/s+M3uK4B/wluHunsDQqApCLeC48P0H4c8s4Gt33xpep3RX/1MELeDnCLq+7wuve2/Y0Mwg6BYvZmZdgFFAR4KkP9fMXnP3jVHFtgOfhAlvWnjd0qJbsj2AV6O+b2FUmYqsCd0W2GhmdYBHzOwEoBBoXeZDgwR8p5l9n6ArvH2M6/0K+JMF8w0eB/aH5WaEv5fG4fa3sYJx96/N7K/AM2a2BxgfI457gKXu/lzYm5INvB5evz5BpSr6mqsJWusVYmb/H7DD3SMVhe1h3BFlhn1EJDUp6ScZM2sJjAV+D9xN0K28BJju7pHu55runh8mjcg/6NuATAvHxc0szd2Loi69391vCs+fGbbglwD3uPv8yHVLhwPkufv+8Ph+giQVbQ5wPvAE0J/vWvTRFgOdgGXh+2HAjPCakTg7h/Ec7HfTAvg5cAlBr0Chuw82s25814LP57v/rnsDvYDBBJWkWLF96u7vhcMSrxP0WKwkmKNQEFYcDDiJGP+/mFkN4Fl3f9rMLifoqfnfqOM/B+q5+23hrq+AXOAsd/826hrR16xwS9/MfkHwuxsdtXsuQW/CH8MKz4IY1xKRFKSkn1jR3fsQTBR7CvhV2OX+gpmd6e5vhGPWcwhawxuAn0ZfKGxp/z9gcpic5xMkoIhLzOzK8PwtBK3LG4FHzSySyJ8Eno265vJwvPwjgsQ3O9zXArjZ3W8EpgOnh7GlAdfE+J4vAT8iSPpPAI+b2XsECfoG4DNgBMGkwdLSw2vXJJhPMM7dl5nZN8Bt4dj5+1HlZxAku7OAK4AaBBWTz4BvYlz/GQsmwdUmGCL42oI7Cd42s8LwM68AFgGdzOxl4H/cfVF4fjPghbBsTeD6Utf/E/BB+B2+dfezLLgLYnLY1V9E8HcqnnhY0Za+mTUj6Nb/EJgdVgJ/6O6fh/MR3if4HY891LVEJDWYnrInlS0ct3+VYAy+zAz+sKV+baQnQ0REKoeSvoiISIrQLXsiIiIpQklfREQkRSjpi4iIpAglfRERkRSRkrfsNW3a1LOzsxMdhkjKmTdv3lfufsQPgBKRo5OSST87O5ucnJxEhyGScsxsbaJjEEll6t4XERFJEUr6IiIiKUJJX0REJEUo6YuIiKQIJX0REZEUoaQvIiKSIpT0pUoaNix4iYhIxSnpi4iIpIikWJzHzE4HHgHSgSfc/d5Sxx8GfhBu1gWauXvj8FghsCg8ts7dz4lP1JIIkdb93Lklt+fMSUAwIiJVTMKTvpmlA48CI4ANwCdmNtndl0bKuPuvo8r/F9A36hJ73b1PvOIVERGpqhKe9IGBwEp3zwUwsxeAc4Gl5ZS/BLgzTrFJkom06NXCFxE5fMkwpt8aWB+1vSHcV4aZtQc6AG9H7a5tZjlm9pGZnVfeh5jZNWG5nG3bth2LuEVERKqUZGjpW4x9Xk7Zi4GX3b0wal87d99kZh2Bt81skbuvKnNB9wnABIABAwaUd32pItTCFxE5fMnQ0t8AtI3abgNsKqfsxcDz0TvcfVP4MxeYQ8nxfhEREQklQ9L/BOhsZh3MrCZBYp9cupCZnQA0AT6M2tfEzGqF75sCp1D+XACRpKV1B0QkHhLeve/uBWb2C2AawS17T7r7EjMbD+S4e6QCcAnwgrtHd82fCDxuZkUEFZh7o2f9i4iIyHesZA5NDQMGDPCcnJxEhyFSZt2BoUODn9V1zoKZzXP3AYmOQyRVJUP3voiIiMRBwrv3RVKZ1h0QkXhSS19ERCRFqKUvkgTU4heReFBLX0REJEWopS8pJxlb03p6oIjEg1r6IiIiKUItfUkZydya1pi+iMSDWvoiIiIpQi19SRlVoTWdjDGJSPWhlr6IiEiKUEtfUo5a0yKSqtTSFxERSRFK+iIiIilCSV9ERCRFKOmLiIikCCV9ERGRFKGkLyIikiKU9EVERFJEUiR9MzvdzJab2UozuzXG8SvNbJuZfRa+roo6NtrMVoSv0fGNXEREpOpI+OI8ZpYOPAqMADYAn5jZZHdfWqroi+7+i1LnZgJ3AgMAB+aF5+6IQ+iSYpJ5+V4RkYpIhpb+QGClu+e6ez7wAnBuBc89DZjh7tvDRD8DOL2S4hQREanSEt7SB1oD66O2NwAnxSg3ysyGAF8Av3b39eWc2zrWh5jZNcA1AO3atTsGYUuqSOZH8oqIHI5kaOlbjH1eavtfQLa79wJmAhMP49xgp/sEdx/g7gOysrKOOFgREZGqKhla+huAtlHbbYBN0QXc/euozb8C90WdO6zUuXOOeYSS0qrCI3lFRCoiGVr6nwCdzayDmdUELgYmRxcws5ZRm+cAn4fvpwEjzayJmTUBRob7REREpJSEt/TdvcDMfkGQrNOBJ919iZmNB3LcfTJwvZmdAxQA24Erw3O3m9nvCCoOAOPdfXvcv4SkBLXwRaSqM/eYQ+DV2oABAzwnJyfRYYikHDOb5+4DEh2HSKpKhu59ERERiQMlfRERkRShpC8iIpIilPRFRERShJK+iIhIilDSFxERSRFK+iIiIilCSV9ERCRFKOmLiIikCCV9ERGRFKGkLyIikiKU9EVERFKEkr6IiEiKUNIXERFJEUr6IiIiKUJJX0REJEUo6YuIiKQIJX0REZEUoaSf5IYNC14iIiJHKymSvpmdbmbLzWylmd0a4/gNZrbUzBaa2Swzax91rNDMPgtfk+MbuYiISNWRkegAzCwdeBQYAWwAPjGzye6+NKrYfGCAu+8xs+uA+4GfhMf2unufuAYdB5HW/dy5JbfnzElAMCIiUi0kQ0t/ILDS3XPdPR94ATg3uoC7z3b3PeHmR0CbOMcoIiJS5SW8pQ+0BtZHbW8ATjpI+bHAm1Hbtc0sBygA7nX31459iPEXadGrhS8iIsdKMiR9i7HPYxY0uxwYAAyN2t3O3TeZWUfgbTNb5O6rYpx7DXANQLt27Y4+ahERkSomGZL+BqBt1HYbYFPpQmY2HLgdGOru+yP73X1T+DPXzOYAfYEySd/dJwATAAYMGBCzUpGM1MIXEZFjJRnG9D8BOptZBzOrCVwMlJiFb2Z9gceBc9z9y6j9TcysVvi+KXAKED0BUEREREIJb+m7e4GZ/QKYBqQDT7r7EjMbD+S4+2TgD0B94CUzA1jn7ucAJwKPm1kRQQXm3lKz/kUqrDLmT2hOhogkk4QnfQB3nwpMLbXvjqj3w8s57wOgZ+VGF9A/3tWH/pYikqqSIumLJFJlrImgdRZEJBkp6R+C/vGuPsr7W4qIpAolfUl5lbEmgtZZEDk88+bNa5aRkfEE0IPkmGReFRUBiwsKCq7q37//l7EKKOkfgv7xrj70txRJXhkZGU+0aNHixKysrB1paWlV5rbqZFJUVGTbtm3rtmXLlieAc2KVUdIXCVVGJUAVC5EK66GEf3TS0tI8Kytr55YtW3qUV0ZJv4L0j3f1ob+lSFJKU8I/euHvsNzhEY2bSLU3bJgm7YnIoaWnp/fv2rVrt86dO3c/44wzOubl5aUB1K1bt29555xwwgndzj777A7xi/LoKOmLiIgAtWrVKlq2bNnSFStWLKlRo4Y/+OCDWQcr/+mnn9Z2dz7++OMGu3btqhL5tEoEKXIkIi38uXODl1r8IlJRgwYN+nblypW1DlZm4sSJmRdddNHXQ4YM2fX88883jldsR0Nj+iIiklTGvD6m7eIvF9c9ltfs0azHnifPfXL9oUvCgQMHmDZtWsORI0fuOli5119/PXP69OlfLF68eO+f//znZtdee+32YxNt5VHSl2pLt+iJyOHYv39/WteuXbsBnHTSSXm//OUvvyqv7Ny5c+tmZmYWdOnSJb9jx4751113Xfa2bdvSs7KyCuMX8eFT0heRClMFSuKhoi3yYy0ypl+Rss8880xmbm5u7datW/cE2L17d/ozzzzT5IYbbii3opAMNKYv1d6cOUpSInLsFBYWMmXKlMz58+cv2bhx46KNGzcuev7551e+9NJLmYmO7VDU0heRQ9IzKCSV7du3L6158+a9ItvXXXfd1ubNm+d36NDhQGTfGWeckTdmzJgOa9eurdG+ffsDsa+UeEr6IiIiwJ49e+bH2l9UVDSv9L5x48Ztjd7OyMhg27ZtCysrtmNFSb88ixdDq1bQpAmYJToakYTSpEiR6kFJP5aiIujfH/LzoUED6NABsrODV+R9hw7Bq2HDBAcrIiJSMUr6sRQVwQsvwJo1sHp18DM3F2bNgt27S5Zt0qRsRSBSQcjOhnr14h29JECqtICr+/cTqe6U9GPJyIDzzy+73x2+/vq7ykCkQrB6NSxdClOnwr59Jc/JyopdKejQAdq1g9q1K//7iIiIoKR/eMygadPgNWBA2eNFRfDll2UrBWvWwKefwquvwoFSkzpbtSo5bBBdOWjbFmrUqOxvJUdBs9pFpCpJiqRvZqcDjwDpwBPufm+p47WASUB/4GvgJ+6+Jjx2GzAWKASud/dpcQy9pLQ0aNEieP3Hf5Q9XlgImzeX7CGI/Hz//WBIobCw5PXatIndS5CdDa1bQ3p6XL6aiIhUfQlP+maWDjwKjAA2AJ+Y2WR3j14VaSyww92PN7OLgfuAn5hZN+BioDvQCphpZl3cPTmXQUxPD5J4mzYweHDZ4wUFsGFDycpA5P3MmbBpUzDEEJGREQwRxJpgmJ0dVD7StP5SZdKsdpHq45ZbbmnxyiuvHJeWluZpaWk0a9bsQM+ePfc8+uijGyNlPvjggzqXX355x9zc3CWtW7fu2aJFi/x58+Ytjxzv2rVrt8LCQluxYsWSxHyLg0t40gcGAivdPRfAzF4AzgWik/65wLjw/cvAn83Mwv0vuPt+YLWZrQyv9+HRBOTunPHcGXRt2pWRnUYytP1Q6tWMw4S8jIzvuvhj2b8f1q37bsggulLwxhuwZUvJ8rVqlZxUWHoIIStLtyOKiAAzZ86sN23atMaLFi1aWqdOHd+8eXPGp59+Wvvaa6/tEJ30n3322cxRo0YVP1hn9+7d6StXrqxx/PHHH/j000+TfpJWMiT91kD0OssbgJPKK+PuBWa2Ezgu3P9RqXNbx/oQM7sGuAagXbt2Bw1o94Fghv7j8x7nkY8foWZ6TU5pewojO41kZKeR9GnRhzRLQAu6Vi3o3Dl4xbJnT1ApiFQGoucU5OQEkxCj1a1bfi9Bhw7QuLEqBRWkFr5I1bZx48YamZmZBXXq1HGAli1bFpx55pnfNmzYsODtt9+ud+qpp+4GmDx5cuabb775ReS88847b/ukSZMyx48fv3XSpEmZo0aN2v6Pf/zjuER9j0NJhqQfK6t4BctU5Nxgp/sEYALAgAEDYpaJqF+zPm9d/hb7Cvbx3rr3mL5qOtNXTee2Wbdx26zbaFq3KcM7Dmdkx5GM6DSCNg3bHOxy8VO3LnTtGrxiycsr2UsQ3Vvw7ruwq9RTJBs2LFspiK4cNGhQmd9GRFLVmDFtWXxsH61Ljx57eLL8B/mcd955u+65555W2dnZPQYNGrTrkksu2X7mmWd+O2rUqO3PPfdc5qmnnrp71qxZ9Ro3blzQs2fP/ZHzLr300h2jR4/uMH78+K3Tpk1r/Oyzz+Yq6R/cBqBt1HYbYFM5ZTaYWQbQCNhewXOPWO2M2gzvOJzhHYdz/4j72frtVmbmzmR6blAJeGHxCwCc2PTE4l6AuA0FHIkGDaBnz+AVy44dJXsHIpWCVauCOQWl1yjIzIxdIcjOhvbttUaBiFQZjRo1Klq8ePHSt956q8GsWbMajB49utMdd9yxYfTo0dsHDRp0YmFh4frnnnsu84ILLtgefV5WVlZho0aNCiZMmNDk+OOP31u/fv2iRH2HikiGpP8J0NnMOgAbCSbmXVqqzGRgNMFY/QXA2+7uZjYZ+LuZPUQwka8z8O/KCrR5/eZc1usyLut1Ge7O4i8XB70AudOTbyjgSDRpErz69St7zB2++qpsL8GaNcGSxVOmBHMOojVrVn4vQbt2wXCFiEhpB2mRV6aMjAzOOuusvLPOOiuvV69ee5955pnjrr/++q9bt269f+rUqQ2mTp3a5P333/+89HkXXHDBjt/85jftH3vssdWJiPtwJDzph2P0vwCmEdyy96S7LzGz8UCOu08G/gY8E07U205QMSAs9w+CSX8FwP+L18x9M6Nn8570bN6TG79/I3sP7C0eCpiRO6PEUMCIjiMY2WkkIzqOoHXDmFMOkp9ZMPEvKwu+972yx4uKYOvW2JWCnBx45ZXg7oTo60XWKIjVU6A1CkQkjhYsWFArLS2NSNf9/Pnz67Rp0yYf4MILL9x+8803t23Xrt3+Tp06lXmC3mWXXbZj8+bNNX784x/vWrt2bVL/w5XwpA/g7lOBqaX23RH1fh9wYTnn3gXcVakBVkCdGnUY0WkEIzqNAGDLt1uYsWoGM3JnMH3VdJ5f/DwA3bK6MbJj0AswpP2Q5B0KOFxpadCyZfA6+eSyxwsLg1sOS08wXLMG3nsPnn8+qDhEX69t29gVAq1RICLH2K5du9Kvv/76drt27UpPT0/37Ozs/RMnTlwLcMUVV+z47//+77Z33313zB6IJk2aFN11111bYh1LNuZ+0DltJQub1XX3PeH749z960Odk4wGDBjgOTk5cfu8yFDAtFXTmJE7g3fWvsO+gn3UTK/JoHaDiisBvVv0rjpDAcfagQPfrVEQXSmIbG8qNVUjskZBrEWLOnQI1ijQnQdJx8zmuXuM5Swl1S1YsGBN7969v0p0HNXBggULmvbu3Ts71rEKJ30z+1+CSXNL3f23ZvaYu//82IUZP/FO+qVFhgIilYCFW4NHMGfVzQp6CzqOqNpDAZUhskZBrNUM16wJhhai1a4dTCYs7+6Dpk1VKUgAJX0pj5L+sXOwpH843fuN3P08M/uRmd1x6OJSntJDAZvzNhffFTBj1Qz+vujvAHTP6l48F6BaDQUciYqsURBr0aLy1iioV6/8CkF2djChUUSkmjmcpJ8Pwfi7mTUDrgKqZEs/2bRs0JKf9v4pP+39U9ydRV8uYvqq6UxbNY3HPnmMhz96WEMBh1K3LnTrFrxi2bUL1q6NvXDRO++UXaOgUaOyD0CKrhRojQIRqYIO2b1vZn3c/TMzO8Xd34/af6a7v1HpEVaCRHfvH469B/by7rp3ixcIWvTlIuC7oYDIAkGtGrRKcKRVmHuwRkGshYsiP/fsKXnOcceVXdY48rN9+6ASImWoe1/Ko+79Y+dou/dfNLMrSyX8oVU14Vc1dWrUKb7nH0oOBUxfNb14KKBHsx7FtwYOaT+EujWUdCrMLFhoKDPz4GsURPcQRN4vXAj/+lfZNQqaNy+/l0BrFIhIglSkpd8ZeA34JcEKePcDx7t7Of2oya8qtfQPpsiLWLR1UfFtge+sfYf9hfupmV6Twe0GF1cCNBRQySJrFJSuEES2160rf42CWM89aNs2uDuhGlJLX8qjlv6xc9Sz982sD/Au8C1wJ/C3pH18bQVUl6QPwLhx8D//A8DeDHinPcy4/GSmN/9WQwHJorAQNm6MfSvimjXBrYrRaxSkp5ddoyD6fatWVXaNAiV9KU9VSPpPPvlkk7vvvrtVbm5u7Tlz5nw+ZMiQPaXLLF++vOZZZ53V+WCP1p0yZUqDBx98sPns2bNXVvSzBw4ceMIDDzywPtZnlnZU3fvhTP2fETzz/kfAtqqc8KudceOCF1AHOC18AWzK21S8QNCM3BklhgIiEwIHtx+soYDKlp4edOm3awdDh5Y9fuAArF9ftjKwejW89RZs3lyyfI0aB1+joHlz3Y4ochSmTJnS4KmnnjrulVdeWRO9v0+fPntfeeWVlVdffXV2YiI7ehXpQ2wPDHD3TeEa91PNrJG7P125ocnRatWgFaP7jGZ0n9EUeRELty4sXib40U8e5aGPHqJWeq3groBw3kCv5r00FBBvNWpAx47BK5Z9+4IhglhPR3ztNdi2rWT52rW/6x2INYRw3HGqFEi1MHAgJwD8+98sj8fn9evXb9/hlF++fHnNSy+9tMPevXvTAB555JF1I0aM2A2Ql5eXPmLEiE65ubm1TzrppLxnnnlmXXp6Ov/85z8bjh8/vlV+fr61b99+/wsvvLCmUaNGxV2BBQUF/OQnP8leuHBhPTPzyy677Ks777zzy4rGdMik7+5jo95/aWbDgX8BTx/Gd5cES7M0+rToQ58WffjNKb9hz4E9vLv23eJKwC0zb+GWmbfQrF6zEs8KaNmgZaJDl9q1oUuX4BXL7t0lb0eMHkL4+OPgzoRo9evHvh0xst24cWV+G5GU0apVq4J33333i7p16/qiRYtqXXLJJR0XL178OcCiRYvqzZ8/f3GXLl3yhwwZ0nnSpElNTj/99Ly777675TvvvPNFw4YNi26//fYWv/vd75o/8MADxd19H374Yd3NmzfXiAwffPXVV4c11nfYs4Xc/RszG3m450lyqVujLqcdfxqnHR8MBmzK28TM3JlMWzWN6aum89yi5wDo2axncS/A4HaDqVOjTiLDlljq1Tv0GgWxhg7WrIG5cyEvr2T5Ro3KfzpidnZQaRBJoEgL/5NPqB+9fbQt/l69enXNz89P27NnT9rOnTszunbt2g3grrvu2jBq1Khdhzq/tPz8fBs7dmz7pUuX1klLS2Pt2rXFt+307Nlzd7du3fIBLrroou3vvvtu/dq1axetWrWq9sCBA7sCHDhwwPr37/9t9DW7du26f/369bVGjx7d9uyzz955/vnnH1ZcRzRF2N33Hsl5krxaNWjFFb2v4IreV5QYCpi2ahr/++//5cEPH6RWei0Gtx9cPB+gZ/OeGgqoCho2hF69gldp7rB9e8kHIEUqBF98AdOmwd5S/7s3bVp2kuEPfgAnnljJX0Skci1cuHAZlD+mf7juuuuu5s2aNTvwyiuvrC4qKqJOnTr9I8es1BCbmeHuDBo0aNe//vWvch/Rm5WVVbh48eKlr776asPHHnus2Ysvvpj50ksvVTjO6nlfkByVgw0FTFs1jd/M/A2/mfkbDQVUB2bBGP9xx0H//mWPu8OXX8a+62DBAnj9dcjPhz//WUlf4ibSoo/3mP7h2rlzZ3qbNm3y09PT+fOf/3xcYeF3c+AXLVpUb9myZTU7d+6c//LLL2deddVV24YNG7b7xhtvbLd48eJaPXr02J+Xl5e2evXqGr169SpeCGTz5s0ZtWrVKrryyiu/6dKly/4xY8Z0OJyYlPTlkKKHAh7kweK7AiILBGkooBozC+4GaN4cTjqp7PGiItiyBeroby3V36RJkxrffPPN7Xbs2JFx/vnndz7xxBP3vPfeeyvKK/+rX/3qy1GjRnV67bXXmgwaNCivTp06xRPy+vTp8+2NN97YZtmyZXVOOumkvJ/+9KffpKen8/jjj6+5+OKLO+bn5xvAnXfeuTE66a9Zs6bG2LFjs4uKigxg/PjxGw7nOxzWo3Wri2p1n36CFXkRn235rPjWwHfXvUt+YT610msxpP2Q4p6AXs17lenOktSj+/SlPFXhPv2q4lg9ZU+kjDRLo1/LfvRr2Y9bBt3CngN7mLtmbvHaAJGhgOb1mhcvEDS843ANBYiIJICSvhxTdWvU5YzOZ3BG5zMA2LhrY3EFYNrKaTy78Fngu6GA0zqdxqB2gzQUICISB0r6UqlaN2zNlX2u5Mo+V1LkRSzYsoBpq6YxI3dG8V0BtTNO+0l7AAAUE0lEQVRqM7jd4OL5AD2b9dRQgIhIJUho0jezTOBFIBtYA1zk7jtKlekD/B/QECgE7nL3F8NjTwNDgZ1h8Svd/bN4xC6HL83S6NuyL31b9uXWQbeyO38376x9p3iBoJtn3MzNM26mRf0WjOg4Inh1GkGL+i0SHbqISLWQ6Jb+rcAsd7/XzG4Nt28pVWYPcIW7rzCzVsA8M5vm7t+Ex29295fjGLMcI/Vq1os5FDB91XTeXPkmzyx8BoBezXsVrw2goQARkSOX6KR/LjAsfD8RmEOppO/uX0S932RmXwJZwDdItVJ6KOCzLZ8Vrw3wyMeP8MCHD1A7ozZD2g8prgT0aNZDQwEiIhWU6OXUmrv7ZoDwZ7ODFTazgUBNYFXU7rvMbKGZPWxmtco5FTO7xsxyzCxnW+kHlEjSidwVcOugW5k9ejY7btnB1Eun8rP+P2PDrg3cNOMmev2lF60easUVr17BswufZeu3WxMdtohUE/Pnz6/dp0+frjVr1ux3xx13NC+vXOvWrXtu3rz5oA3ounXr9j2cz77hhhtaHewzj0alt/TNbCYQa1D29sO8TkvgGWC0u0cWOLgN2EJQEZhA0EswPtb57j4hLMOAAQNSb3GCKq70UMCGXRuK1waIHgro3bx38QqBGgoQkSPVrFmzgkceeWTdyy+/3CTRsRxLld7Sd/fh7t4jxut1YGuYzCNJPebjAc2sIfAG8N/u/lHUtTd7YD/wFDCwsr+PJIc2Ddvwn33/k7+P+jtbb9pKztU53PPDe8isk8kfP/ojI58dSeb9mZz27Gk8+MGDLNq6iFRciEpEjkzr1q0Lhg4duqdGjRoV/odj+PDhnbp3737i8ccf3/2BBx5oGn3s6quvbtOtW7cTTz755C6bNm3KAFiyZEmtwYMHd+7evfuJ/fv3P2H+/Pm1S1/z97//fbNOnTp179KlS7ezzjqrnOdvV1yix/QnA6OBe8Ofr5cuYGY1gVeBSe7+UqljLd19swWDuucBiys/ZEk2aZZG/1b96d+qP7cOupVv878tXiBo+qrp3DTjJpgBLeu3LLFAUPP6ldJ7JiIp6rnnnlvTvHnzwm+//db69u3b7fLLL9/RokWLwr1796b169dvz1//+tcNN910U8tbb7211aRJk9ZdddVV7SdMmLC2Z8+e+99+++161113XbuPPvroi+hr/ulPf2qxdu3aRXXq1PHDfYxuLIlO+vcC/zCzscA64EIAMxsA/MzdrwIuAoYAx5nZleF5kVvznjOzLMCAz4CfxTl+SUL1a9bnzC5ncmaXM4FgKCByW+AbX7zBpAWTgO+GAiJ3BdTOKFPJFpFkdcMNrXj44ZJLe/7615t56KFNCYqI++67r/kbb7zRGGDLli01lixZUrtFixa709LSuOqqq7YDjBkz5usf//jHx+/cuTNt/vz59S+88MJOkfMj6+1HO+GEE/aef/75Hc4555xvLrvssqOewJ7QpO/uXwM/jLE/B7gqfP8s8Gw5559aqQFKtdCmYRvG9B3DmL5jKPIi5m+eX7xA0B8/+iN/+OAP1M6ozdD2Q4srAd2zuuuuAJFk9tBDm45lgr/nnnuyJk6cmAXw1ltvrcjOzj5wOOdPmTKlwdy5cxvk5OQsa9CgQdHAgQNP2Lt3b8whdDOjsLCQBg0aFCxbtmzpwa47e/bsFW+++WaD1157rfH999/fasWKFYtr1KhxOKGVkOjZ+yJxFRkK+O3g3zJ79Gy237KdKZdM4Zp+17Bu5zpunH4jPf+vJ60fas2Vr13Jcwuf010BIingtttu27Zs2bKly5YtW3q4CR/gm2++SW/UqFFhgwYNiubPn197wYIF9SLHioqKeOqpp5oAPP3008cNHDgwLzMzs6hNmzb5Tz75ZJNImQ8//LDEzOPCwkJWrVpV8+yzz8577LHHNuTl5aXv3LnzqLr4E929L5JQpYcC1u9cXzwXYMoXU5i4YCIAfVr0KV4b4JR2p2goQKSaW7duXcb3vve9brt37043M3/88cebf/7554szMzOLYpUfNWrUzgkTJmR16dKlW6dOnfb17t17d+RYnTp1ipYsWVKne/fuLRo0aFD4z3/+Mxfg+eefz7366qvb33fffS0LCgrs/PPP337yySfvjZxXUFBgl156aYe8vLx0d7drr712a9OmTQuP5nvp0boi5YgMBUxfNZ3pudN5f937HCg6QJ2MOsECQRoKOGx6tK6UR4/WPXb0aF2RIxB9V8Btg28rvisgUgm4cfqNQHBXQKQCMLzjcJrVO+gaUyIiCaOkL1JBpYcC1u1cV7xA0L+++FfxUEDfFn2LFwjSUICIJBMlfZEj1K5RO8b2G8vYfmMpLCrk082fFs8HePDDB7nv/fuok1GHodlDi+cDdMvqpqEAEUkYJX2RYyA9LZ3vtf4e32v9PX47+Lfk7c9jzpo5xZWAG6bfAECrBq2CoYBwgaCselkJjlwkaRQVFRVZWlpa6k00O4aKiooMiDnZEJT0RSpFg1oNOPuEszn7hLOBYCggskDQ5OWTefqzpwHo17IfIzqOCO4KaHsKtTLKfWaUSHW3eNu2bd2ysrJ2KvEfmaKiItu2bVsjDrI6rWbvi8RZZCggUgl4f/37FBQVULdGXYa2H1pcCaiOQwGavS/lmTdvXrOMjIwngB5oDZkjVQQsLigouKp///6xn2WjpC+SWHn785i7di7TVgarBC7/ejlQPYcClPRFEktJXyTJrP1mbfFcgJm5M9mxbwcQDAVEJgR+v+33q+RQgJK+SGIp6YskscKiQuZtnseMVTOYnjudD9Z/UGIoILI+wIlNT6wSQwFK+iKJpaQvUoVE7gqILBD0xdfBUzhbN2hd4rHByToUoKQvklhK+iJV2Jpv1hQvEFQVhgKU9EUSS0lfpJooLCokZ1NO8XyADzd8WDwUMCx7WHEloGvTrgkbClDSF0ksJX2Raipvfx6z18wung8QGQpo07BNcQXghx1/SNO6TeMWk5K+SGIp6YukiDXfrCleG2Bm7ky+2fcNhtG/Vf/itQG+3/b71EyvWWkxKOmLJJaSvkgKigwFRCoBkaGAejXqBUMB4QODjvVQgJK+SGIp6YsIu/bv+u6ugFXTWbF9BXDshwKU9EUSK6FJ38wygReBbGANcJG774hRrhBYFG6uc/dzwv0dgBeATOBT4Kfunn+oz1XSFzm41TtWF08InLV6VomhgEgl4OS2Jx/2UICSvkhiJTrp3w9sd/d7zexWoIm73xKj3LfuXj/G/n8A/3T3F8zsL8ACd/+/Q32ukr5IxRUUFTBv07zitQE+XP8hhV5YYijgnBPOIbtx9iGvpaQvkliJTvrLgWHuvtnMWgJz3P2EGOXKJH0LBhq3AS3cvcDMTgbGuftph/pcJX2RI7dr/y5mr55dPB9gxfYVPHL6I1x/0vWHPFdJXySxEv0ko+buvhkg/NmsnHK1zSzHzD4ys/PCfccB37h7Qbi9AWhdueGKHKFx48Cs5GvcuERHdUQa1mrIuV3P5dEzH+WL//qC3OtzuaznZYkOS0QqIKOyP8DMZgItYhy6/TAu087dN5lZR+BtM1sE7IpRrtxuCzO7BrgGoF27dofx0SLHwLhxVTbJH0qHJh0SHYKIVFClJ313H17eMTPbamYto7r3Yz7/1903hT9zzWwO0Bd4BWhsZhlha78NsOkgcUwAJkDQvX+k30dERKSqSnT3/mRgdPh+NPB66QJm1sTMaoXvmwKnAEs9mIwwG7jgYOeLiIhIINFJ/15ghJmtAEaE25jZADN7IixzIpBjZgsIkvy97r40PHYLcIOZrSQY4/9bXKMXERGpQrQ4j4jEjWbviyRWolv6IiIiEidK+iIiIilCSV9ERCRFKOmLiIikCCV9ERGRFKGkLyIikiKU9EVERFKEkr6IiEiKUNIXERFJEUr6IiIiKUJJX0REJEUo6YuIiKQIJX0REZEUoaQvkgrGjQOzkq9x4xIdlYjEmR6tKyJxo0friiSWWvoiIiIpQklfREQkRSjpi4iIpAglfRERkRShpC8iIpIiEpr0zSzTzGaY2YrwZ5MYZX5gZp9FvfaZ2XnhsafNbHXUsT7x/xYiIiJVQ6Jb+rcCs9y9MzAr3C7B3We7ex937wOcCuwBpkcVuTly3N0/i0vUIiIiVVCik/65wMTw/UTgvEOUvwB40933VGpUIiIi1VCik35zd98MEP5sdojyFwPPl9p3l5ktNLOHzaxWeSea2TVmlmNmOdu2bTu6qCW+tJqciMgxUekr8pnZTKBFjEO3AxPdvXFU2R3uXmZcPzzWElgItHL3A1H7tgA1gQnAKncff6iYtCKfSGJoRT6RxMqo7A9w9+HlHTOzrWbW0t03hwn8y4Nc6iLg1UjCD6+9OXy738yeAm46JkGLiIhUQ4nu3p8MjA7fjwZeP0jZSyjVtR9WFDAzI5gPsLgSYhQREakWEp307wVGmNkKYES4jZkNMLMnIoXMLBtoC8wtdf5zZrYIWAQ0BX4fh5hFRESqpErv3j8Yd/8a+GGM/TnAVVHba4DWMcqdWpnxiYiIVCeJbumLiIhInCjpi4iIpAglfRERkRShpC+pS4v+iEiKqfTFeZKRFucRSQwtziOSWGrpi4iIpAglfRERkRShpC8iIpIilPRFRERShJK+iIhIilDSFxERSRFK+iIiIilCSV9ERCRFKOmLiIikCCV9ERGRFKGkHy9a511ERBJMa++LSNxo7X2RxFJLX+JDPR0iIgmnlr6IxI1a+iKJldCWvpldaGZLzKzIzMr9h8DMTjez5Wa20sxujdrfwcw+NrMVZvaimdWMT+QiIiJVT6K79xcDPwbeKa+AmaUDjwJnAN2AS8ysW3j4PuBhd+8M7ADGVm64IiIiVVdCk767f+7uyw9RbCCw0t1z3T0feAE418wMOBV4OSw3ETiv8qIVERGp2hLd0q+I1sD6qO0N4b7jgG/cvaDU/pjM7BozyzGznG3btlVasCIiIskqo7I/wMxmAi1iHLrd3V+vyCVi7POD7I/J3ScAEyCYyFeBzxUREalWKj3pu/vwo7zEBqBt1HYbYBPwFdDYzDLC1n5kv4iIiMRQFbr3PwE6hzP1awIXA5M9uNdwNnBBWG40UJGeAxERkZSU6Fv2zjezDcDJwBtmNi3c38rMpgKErfhfANOAz4F/uPuS8BK3ADeY2UqCMf6/xfs7iIiIVBVanEdE4kaL84gkVkomfTPbBqytQNGmBHMHkllViBGqRpyK8dgpL8727p4V72BEJJCSSb+izCwn2VslVSFGqBpxKsZjp6rEKZJqqsJEPhERETkGlPRFRERShJL+wU1IdAAVUBVihKoRp2I8dqpKnCIpRWP6IiIiKUItfRERkRShpC8iIpIilPQBMzvdzJab2UozuzXG8Vpm9mJ4/GMzy07CGG8ws6VmttDMZplZ+2SLMarcBWbmZpaQW7oqEqeZXRT+PpeY2d+TLUYza2dms81sfvg3/1ECYnzSzL40s8XlHDcz+1P4HRaaWb94xygipbh7Sr+AdGAV0BGoCSwAupUq83PgL+H7i4EXkzDGHwB1w/fXJWOMYbkGwDvAR8CAJP17dwbmA03C7WZJGOME4LrwfTdgTQJ+l0OAfsDico7/CHiT4ImY/wF8HO8Y9dJLr5IvtfRhILDS3XPdPR94ATi3VJlzgYnh+5eBH5pZrEf7JixGd5/t7nvCzY8InjoYTxX5PQL8Drgf2BfP4KJUJM6rgUfdfQeAu3+ZhDE60DB834gEPGHS3d8Bth+kyLnAJA98RPBUzJbxiU5EYlHSh9bA+qjtDeG+mGU8eADQToIH/MRLRWKMNpaghRVPh4zRzPoCbd19SjwDK6Uiv8suQBcze9/MPjKz0+MWXaAiMY4DLg8fWDUV+K/4hHZYDve/WxGpZBmJDiAJxGqxl76PsSJlKlOFP9/MLgcGAEMrNaIYHx1jX3GMZpYGPAxcGa+AylGR32UGQRf/MIIek3fNrIe7f1PJsUVUJMZLgKfd/UEzOxl4JoyxqPLDq7BE/38jIqWopR+0PtpGbbehbFdpcRkzyyDoTj1Yt+axVpEYMbPhwO3AOe6+P06xRRwqxgZAD2COma0hGOOdnIDJfBX9e7/u7gfcfTWwnKASEC8ViXEs8A8Ad/8QqE3wkJtkUqH/bkUkfpT04ROgs5l1MLOaBBP1JpcqMxkYHb6/AHjb3ePZYjlkjGHX+eMECT/eY9CHjNHdd7p7U3fPdvdsgnkH57h7vJ9xXJG/92sEEyMxs6YE3f25SRbjOuCHYYwnEiT9bXGMsSImA1eEs/j/A9jp7psTHZRIKkv57n13LzCzXwDTCGZNP+nuS8xsPJDj7pOBvxF0n64kaOFfnIQx/gGoD7wUzjFc5+7nJFmMCVfBOKcBI81sKVAI3OzuXydZjDcCfzWzXxN0mV8Z54ooZvY8wRBI03BuwZ1AjfA7/IVgrsGPgJXAHuA/4xmfiJSlZXhFRERShLr3RUREUoSSvoiISIpQ0hcREUkRSvoiIiIpQklfREQkRSjpi4iIpAglfUlq4eNb7wzfdzazL/SIVhGRI6P79CWpmVlrYB5wBvAccI27v5fYqEREqia19CWpuftGgscavwv82t3fM7NGYQ/A+kOcLiIiUZT0JamZWTOCpVx3893jjXe6+xiCB+GIiEgFpfza+5K8zKwx8CbBs+OzgPuAsxMZk4hIVaaWviQlM6sLTAEec/dXgCcInjz3g8RGJiJSdWkin1RJZvYocC5BxeC+8Ln3IiJyEEr6IiIiKULd+yIiIilCSV9ERCRFKOmLiIikCCV9ERGRFKGkLyIikiKU9EVERFKEkr6IiEiKUNIXERFJEUr6IiIiKeL/B8ndy0gKRSQZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#### Exercise 8.6 (b)\n",
    "\n",
    "\n",
    "eta = 1\n",
    "use_adaline=False\n",
    "maxit = 1000\n",
    "dim = 2\n",
    "\n",
    "positives = df1.loc[df1['y']==1]\n",
    "negatives = df1.loc[df1['y']==-1]\n",
    "\n",
    "figsize = plt.figaspect(1)\n",
    "f, ax = plt.subplots(1, 1, figsize=figsize)\n",
    "\n",
    "ps = ax.scatter(positives[['x1']].values, positives[['x2']].values, marker='+', c= 'b', label='+1 labels')\n",
    "ns = ax.scatter(negatives[['x1']].values, negatives[['x2']].values, marker=r'$-$', c= 'r', label='-1 labels')\n",
    "print('Number of positive points: ', len(positives))\n",
    "print('Number of negatives points: ', len(negatives))\n",
    "\n",
    "norm_g, num_its, _ = lm.perceptron(df1.values, dim, maxit, use_adaline, eta, \n",
    "                                   randomize=False, print_out = True)    \n",
    "\n",
    "\n",
    "x1 = np.arange(0, 1, 0.02)\n",
    "norm_g = norm_g/norm_g[-1]\n",
    "\n",
    "c = np.ones((N_t, 1))\n",
    "XX_t = np.hstack((c, X_t))        \n",
    "print(f\"-- Out-of-Sample Error: {lm.calc_error(norm_g, XX_t, y_t)}\")\n",
    "\n",
    "# ------- Ploting ------\n",
    "hypothesis = ax.plot(x1, -(norm_g[0]+norm_g[1]*x1), c = 'g', label='PLA')\n",
    "svm = ax.plot(x1, -(my_svm.w[0]*x1+my_svm.b)/my_svm.w[1], c = 'r', label='SVM')\n",
    "\n",
    "ax.set_ylabel(r\"$x_2$\", fontsize=11)\n",
    "ax.set_xlabel(r\"$x_1$\", fontsize=11)\n",
    "ax.set_title('Exercise 8.6 (c) Data set size = %s'%N, fontsize=9)\n",
    "ax.axis('tight')\n",
    "legend_x = 2.0\n",
    "legend_y = 0.5\n",
    "ax.legend(['PLA', 'SVM',\n",
    "           '+1 labels', '-1 labels', ], \n",
    "          loc='center right', bbox_to_anchor=(legend_x, legend_y))\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 8.6 (c)\n",
    "It's clear that PLA can be greatly affected by the ordering of data points, while SVM is stable with respect to the point orders. \n",
    "\n",
    "The out-of-sample error is much larger from PLA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 8.7\n",
    "\n",
    "* (a) Assume we have $\\rho_1 < \\rho_2$, consider the corresponding hypothesis sets $\\mathcal{H}_{\\rho_1}$ and $\\mathcal{H}_{\\rho_2}$. For a given data set $(x_1,y_1),\\dots, (x_N, y_N)$, for any given dichotomy on the data set, if the dichotomy can be implemented by a hypothesis $h_2 \\in \\mathcal{H}_{\\rho_2}$, then there must be a hypothesis $h_1 \\in \\mathcal{H}_{\\rho_1}$ that can implement the dichotomy as well. \n",
    "\n",
    "This is because we can choose the $(b_2,w_2)$ from $h_2$, since $\\rho_2 = \\frac{1}{|w_2|}$, if we set $(b_1, w_1) =  \\frac{\\rho_2}{\\rho_1}(b_2, w_2)$ such that $|w_1| = \\frac{1}{\\rho_1}$, so $h_1 = (b_1, w_1)$ belongs to $\\mathcal{H}_{\\rho_1}$ and has a margin of $\\rho_1$.\n",
    "\n",
    "As the margin of $h_1$ is smaller than the margin of $h_2$ ($\\rho_1 < \\rho_2$), if $h_2$ can implement the dichotomy, $h_1$ certainly can do that too. The reverse is not necessarily true since if we add points (Assuming adding the proper signs on the two sides of $h_1$) that are in the margin of $h_2$ but outside the margin of $h_1$, $h_2$ won't be able to implement the new dichotomy while $h_1$ can still do. \n",
    "\n",
    "So we see that $d_{VC}(\\rho)$ is non-increasing in $\\rho$.\n",
    "\n",
    "* (b) We first prove that for any 3 points in the unit disc, there must be two that are within distance $\\sqrt{3}$ of each other. If this is not ture, then there exists 3 points in the unit disc, such that the distances between any two points are larger than $\\sqrt{3}$. \n",
    "\n",
    "The center of the disc can't be one of such 3 points, because the distance from center to any point is less than or equal to 1. \n",
    "Now we draw lines from the center to 3 points and extend them to meet with the boundary of the disc. We check the lengths of the segments (formed by connecting the 3 points on the circle), they must be all larger than $\\sqrt{3}$.\n",
    "\n",
    "We can compute the angle that each segment corresponds to, e.g. $\\cos(\\theta) = \\frac{a^2+b^2-c^2}{2ab}$, where $a=b=1$ and $c$ is the segment, $\\theta$ is the corresponding angle. We have $\\cos(\\theta) = 1 - \\frac{c^2}{2} < -0.5$, we conclude that $\\theta > \\frac{2\\pi}{3}$. This is true for all three angles, so we have a total angle of larger than $3\\times \\frac{2\\pi}{3} = 2\\pi$, which contradicts. \n",
    "\n",
    "So for any 3 points in the unit disc, there must be two that are within distance of $\\sqrt{3}$ of each other.\n",
    "\n",
    "Suppose that when $\\rho >\\frac{\\sqrt{3}}{2}$, $d_{VC}(\\rho) \\ge 3$, that is the $\\mathcal{H}_{\\rho}$ can shatter at least 3 points. \n",
    "\n",
    "Now consider any 3 points in the unit disc, from what we have proved, there must be two that are within distance of $\\sqrt{3}$ of each other. Consider the dichotomy when the two points is one positive, one negative, any hypothesis hyperplane (a line) that can implement this dichotomy needs to cross with the line between the two points. We see that of the two distances from the two points to the hyperplane, there is at least one that is in the margin, i.e. less than $\\rho$. The distances are equal when the line is passing through the middle and perpendicular to the line between the two points. And the distances there are both $\\frac{\\sqrt{3}}{2}$, which is smaller than $\\rho$, meaning at least one point will always stay in the margin of any hyperplane that can implement the dichotomy. \n",
    "\n",
    "This contradicts with the assumption that the hypothesis set $\\mathcal{H}_{\\rho}$ can shatter any 3 points. Thus $d_{VC}(\\rho) < 3$ when $\\rho >\\frac{\\sqrt{3}}{2}$\n",
    "\n",
    "#### Exercise 8.8\n",
    "\n",
    "* (a) In Figure 8.5, there are total 23 points with 5 support vectors, so the bound is $E_{CV}(SVM) \\le \\frac{5}{23}$\n",
    "\n",
    "* (b) If one of the four support vectors in a gray box are removed, it doesn't change the classifier. The classifier still has the maximum margin among all candidates.\n",
    "\n",
    "* (c) The bound is now $\\frac{4}{23}$\n",
    "\n",
    "#### Exercise 8.9\n",
    "\n",
    "* (a) Since $u_0$ is optimal for (8.10), so we have $c-a^Tu_0 \\le 0$, the maximum value that can be achieved with $\\alpha \\ge 0$ is thus $\\max_{\\alpha \\ge 0} \\alpha (c-a^Tu_0) = 0$\n",
    "\n",
    "* (b) We now prove that $u_1$ is feasible for (8.10). Assume the contrary that $c-a^Tu_1 > 0$, then $\\max_{\\alpha \\ge 0} \\alpha (c-a^Tu_0) = \\infty$ and the objective in (8.11) goes to infinite. However, when $u=u_0$, we have a finite objective in (8.11) as $\\max_{\\alpha \\ge 0} \\alpha (c-a^Tu_0) = 0$. This contradicts the assumption that $u_1$ is optimal for (8.11). So we have $c-a^Tu_1 \\le 0$ and $u_1$ is feasible for (8.10).\n",
    "\n",
    "* (c) For the objective in (8.11), we know from problem (a),(b) that for both $u_0, u_1$, we have $c-a^Tu_0 \\le 0$ and $c-a^Tu_1 \\le 0$. Then we have $\\max_{\\alpha \\ge 0} \\alpha (c-a^Tu_1) = \\max_{\\alpha \\ge 0} \\alpha (c-a^Tu_1) = 0$. \n",
    "\n",
    "Thus we see that $u_1$ actually minimizes $\\min_{u\\in R^L}\\frac{1}{2}u^TQu + p^Tu$, on the other hand, by definition, $u_0$ also minimize the objective in (8.10), which is the same objective here. So we must have $\\frac{1}{2}u^T_0Qu_0 + p^Tu_0 = \\frac{1}{2}u^T_1Qu_1 + p^Tu_1$\n",
    "\n",
    "* (d) Let $u^*$ be any optimal solution for (8.11), then by problem (b) and (a), we have $\\max_{\\alpha \\ge 0} \\alpha (c-a^Tu^*) = 0$, if the maximum is attained at $\\alpha^*$, then $\\alpha^* (c-a^Tu^*) = 0$, so we either have $c-a^Tu^*= 0$ or $\\alpha^* = 0$.\n",
    "\n",
    "#### Exercise 8.10\n",
    "\n",
    "\\begin{align*}\n",
    "\\frac{\\partial{\\mathcal{L}}}{\\partial{u}_1} &= 2u_1 - \\alpha_1 -\\alpha_2 = 0 \\\\\n",
    "\\frac{\\partial{\\mathcal{L}}}{\\partial{u}_2} &= 2u_2 - 2\\alpha_1 -\\alpha_3 = 0 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Let the derivatives equal to 0, we have $u_1 = \\frac{\\alpha_1 + \\alpha_2}{2}, u_2 = \\frac{2\\alpha_1 + \\alpha_3}{2}$.\n",
    "\n",
    "\\begin{align*}\n",
    "\\mathcal{L}(u,\\alpha) &= u^2_1 + u^2_2 + \\alpha_1(2-u_1-2u_2) -\\alpha_2u_1 -\\alpha_3u_2 \\\\\n",
    "&=  \\frac{(\\alpha_1 + \\alpha_2)^2}{4} + \\frac{(2\\alpha_1 + \\alpha_3)^2}{4} + \\alpha_1 ( 2- \\frac{\\alpha_1 + \\alpha_2}{2} - (2\\alpha_1 + \\alpha_3)) - \\alpha_2 \\frac{\\alpha_1 + \\alpha_2}{2} - \\alpha_3 \\frac{2\\alpha_1 + \\alpha_3}{2} \\\\\n",
    "&= \\frac{1}{4}\\alpha^2_1 + \\frac{1}{2}\\alpha_1\\alpha_2 + \\frac{1}{4}\\alpha^2_2 + \\alpha^2_1 + \\alpha_1\\alpha_3 +  \\frac{1}{4}\\alpha^2_3 + 2\\alpha_1 -  \\frac{1}{2}\\alpha^2_1 -  \\frac{1}{2}\\alpha_1\\alpha_2 - 2\\alpha^2_1 - \\alpha_1\\alpha_3 -  \\frac{1}{2}\\alpha_1\\alpha_2 -  \\frac{1}{2}\\alpha^2_2 - \\alpha_1\\alpha_3 - \\frac{1}{2}\\alpha^2_3\\\\\n",
    "&= -\\frac{5}{4}\\alpha^2_1 - \\frac{1}{4}\\alpha^2_2 \\frac{1}{4}\\alpha^2_3 - \\frac{1}{2}\\alpha_1\\alpha_2 - \\alpha_1\\alpha_3 + 2\\alpha_1\\\\\n",
    "\\end{align*}\n",
    "\n",
    "\n",
    "#### Exercise 8.11\n",
    "\n",
    "Change $\\sum^N_{n=1}y_n\\alpha_n = 0$ into $\\sum^N_{n=1}y_n\\alpha_n \\ge 0$ and $\\sum^N_{n=1}y_n\\alpha_n \\le 0$, i.e. $-\\sum^N_{n=1}y_n\\alpha_n \\ge 0$, combine these two conditions with $\\alpha_n \\ge 0$, we have\n",
    "\n",
    "\\begin{align*}\n",
    "A_D &= \\begin{bmatrix}y^T \\\\ -y^T \\\\ I_{N\\times N}\\end{bmatrix}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "We set $M=\\begin{bmatrix}y_1x_1 & y_2x_2 & \\dots & y_Nx_N\\end{bmatrix}$, where $x_n, n=1,\\dots, N$ are the column vectors with dimension $d\\times 1$, so $M$ has a dimension of $d \\times N$. \n",
    "\n",
    "We also have column vectors of $y=\\begin{bmatrix}y_1 \\\\ y_2 \\\\ \\vdots \\\\ y_N\\end{bmatrix}$ and $\\alpha = \\begin{bmatrix}\\alpha_1 \\\\ \\alpha_2 \\\\ \\vdots \\\\ \\alpha_N\\end{bmatrix}$\n",
    "\n",
    "Let $Q_D= M^TM = \\begin{bmatrix}y_1y_1x^T_1x_1 & y_1y_2x^T_1x_2 & \\dots & y_1y_Nx^T_1x_N\\\\ y_ny_1x^T_nx_1 & y_ny_2x^T_nx_2 & \\dots & y_ny_Nx^T_nx_N\\\\ \\dots & \\dots & \\dots & \\dots\\\\ y_Ny_1x^T_Nx_1 & y_Ny_2x^T_Nx_2 & \\dots & y_Ny_Nx^T_Nx_N\\end{bmatrix}$. $M$ has a dimension of $N\\times N$.\n",
    "\n",
    "\n",
    "\n",
    "So we can write \n",
    "\n",
    "\\begin{align*}\n",
    "\\frac{1}{2}\\sum^N_{m=1}\\sum^N_{n=1} y_ny_m\\alpha_n \\alpha_m x^T_nx_m &= \\frac{1}{2}\\sum^N_{m=1}y_m\\alpha_m\\sum^N_{n=1} y_n\\alpha_n  x^T_nx_m\\\\\n",
    "&= \\frac{1}{2}\\sum^N_{m=1}\\alpha_m \\sum^N_{n=1}\\alpha_n y_nx^T_nx_my_m\\\\\n",
    "&= \\frac{1}{2}\\sum^N_{m=1}\\alpha_m \\sum^N_{n=1}\\alpha_n [Q_D]_{n,m}\\\\\n",
    "&= \\frac{1}{2}\\alpha^T Q_D \\alpha\\\\\n",
    "\\end{align*}\n",
    "\n",
    "* (b) This part is proved in problem (a) when we construct matrix $Q_D$. So $X_s = M^T$, and $Q_D = X_sX^T_s$\n",
    "\n",
    "For any vector $\\alpha \\ne 0$, we have \n",
    "\n",
    "\\begin{align*}\n",
    "\\alpha^TQ_D \\alpha & = \\alpha^T X_sX^T_s \\alpha \\\\\n",
    "&= (\\alpha^TX_s)(\\alpha^TX_s)^T\\\\\n",
    "&= |\\alpha^TX_s|^2 \\\\\n",
    "&\\ge 0\\\\\n",
    "\\end{align*}\n",
    "\n",
    "So $Q_D$ is positive semi-definite.\n",
    "\n",
    "#### Exercise 8.12\n",
    "\n",
    "If all the data is from one class, then $\\alpha^*_n = 0$ for $n=1,\\dots, N$.\n",
    "\n",
    "* (a) $w^* = \\sum^N_{n=1}y_n\\alpha^*_n x_n = 0$\n",
    "\n",
    "* (b) Since all $\\alpha^*_n = 0$, we can apply KKT conditions here to obtain $b^*$. To satisfy the constraints on the primal optimization problem, we need to have $b \\ge 1$ if all points are positive.\n",
    "\n",
    "#### Exercise 8.13\n",
    "\n",
    "Consider a data set with two positive examples at $x_1 = (0,0)$ and $x_2 = (1,0)$, and one negative example at $x_3 = (0,1)$. We look for hyperplane (line) that separate the negative example with the positive examples. As there's only 1 negative example, it has to be the support vector, either one of the two positive examples or both of them can be the support vectors. It's not hard by trial and error to find out that the optimal fat-hyperplane is $-2[x]_2+1 = 0$, i.e. with $(b,w) = (1, [0, -2])$\n",
    "\n",
    "The optimal solution $\\alpha^*$ has to satisfy $w^* = \\sum^N_{n=1}y_n\\alpha^*_nx_n$,\n",
    "\n",
    "\\begin{align*}\n",
    "w^* &= \\sum^N_{n=1}y_n\\alpha^*_nx_n \\\\\n",
    "&= y_1\\alpha_1x_1 + y_2\\alpha_2x_2 + y_3\\alpha_3x_3 \\\\\n",
    "&= \\alpha_1\\begin{bmatrix}0 \\\\ 0\\end{bmatrix} + \\alpha_2\\begin{bmatrix}1 \\\\ 0\\end{bmatrix} - \\alpha_3\\begin{bmatrix}0 \\\\ 1\\end{bmatrix} \\\\\n",
    "&= \\begin{bmatrix}\\alpha_1 \\\\ -\\alpha_3\\end{bmatrix}\n",
    "\\end{align*}\n",
    "\n",
    "since $w^* = \\begin{bmatrix}0 \\\\ -2\\end{bmatrix}$, we have $\\alpha_1 = 0$.\n",
    "\n",
    "On the other hand, for this hyperplane, all three points are support vectors. \n",
    "It's easy to check that for $n=1,2,3$, we have $y_n(w^*Tx_n + b^*) = 1$. \n",
    "\n",
    "So if a point $(x_n, y_n)$ is on the boundary satisfying $y_n((w^*)^Tx_n + b^*)=1$, it's possible that $\\alpha^*_n= 0$ as the $\\alpha^*_1 = 0$ here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 8.14\n",
    "\n",
    "If we remove a data point $(x_n, y_n)$ with $\\alpha^*_n = 0$, suppose the previous optimal solution is $\\mathbf{\\alpha}^*$.\n",
    "\n",
    "* (a) Since $\\mathbf{\\alpha}^*$ is the optimal solution for the previous dual problem, It satisfies the constraints in (8.21). i.e. $\\alpha_i \\ge 0$ for $i=1,\\dots, N$. And\n",
    "\n",
    "$\\sum^N_{i=1} y_i\\alpha_i = \\sum^N_{i\\ne n} y_i\\alpha_i = 0$ since $\\alpha^*_n = 0$. The second part is exactly the new constraint for the problem with $(x_n, y_n)$ removed.\n",
    "\n",
    "So the solution $\\mathbf{\\alpha}^*$ (after removing $\\alpha^*_n$) is feasible for the new dual problem.\n",
    "\n",
    "* (b) If there's another feasible solution ($\\alpha'$) for the new dual and it has a lower objective value than $\\mathbf{\\alpha}^*$. We construct a new solution for previous dual problem by adding $\\alpha^*_n = 0$ into $\\alpha'$, i.e. $\\alpha^c$. It's clear that $\\alpha^c$ is a feasible solution for the previous dual problem.\n",
    "\n",
    "From (8.21), the objective value of $\\alpha^c$ for previous dual problem is thus:\n",
    "\n",
    "\\begin{align*}\n",
    "V(\\alpha^c) &= \\frac{1}{2}\\sum^N_{i=1}\\sum^N_{j=1}y_iy_j\\alpha^c_i\\alpha^c_j x^T_ix_j - \\sum^N_{i=1}\\alpha^c_i \\\\\n",
    "&= \\frac{1}{2}\\sum^N_{i\\ne n}\\sum^N_{j\\ne n}y_iy_j\\alpha^c_i\\alpha^c_j x^T_ix_j - \\sum^N_{i\\ne n}\\alpha^c_i\\\\\n",
    "&< \\frac{1}{2}\\sum^N_{i\\ne n}\\sum^N_{j\\ne n}y_iy_j\\alpha^*_i\\alpha^*_j x^T_ix_j - \\sum^N_{i\\ne n}\\alpha^*_i\\\\\n",
    "&= V(\\alpha^*) \\\\\n",
    "\\end{align*}\n",
    "\n",
    "This contradicts the fact that $\\alpha^*$ is the optimal solution for the previous problem. So we conclude there's no other feasible solution for the new dual problem that has a lower objective value than $\\mathbf{\\alpha}^*$.\n",
    "\n",
    "* (c) Hence we showed that $\\mathbf{\\alpha}^*$ (minus $\\alpha^*_n$) is optimal for the new dual problem.\n",
    "\n",
    "* (d) Since $w^* = \\sum^N_{i=1}y_i\\alpha^*_ix_i = \\sum^N_{i\\ne n}y_i\\alpha^*_ix_i$. $w^*$ is the same as previous problem. \n",
    "Also $b^*$ is computed using a point where $\\alpha^*_s > 0$, it's not affected by $\\alpha^*_n$ as well. So we conclude that the optimal hyperplane doesn't change. \n",
    "\n",
    "* (e) As the final hypothesis is not changed when we throw out any data point with $\\alpha^*_n=0$, after we throw out all such points, we are left with data points that have $\\alpha^*_n > 0$, thus we shows that $E_{CV} = \\frac{1}{N}\\sum^N_{n=1}e_n \\le \\frac{\\text{number of }\\alpha^*_n > 0}{N}$\n",
    "\n",
    "#### Exercise 8.15\n",
    "\n",
    "* (a) \n",
    "\n",
    "\\begin{align*}\n",
    "\\Phi^T(x)\\Phi(x) &= \\begin{bmatrix}\\Phi^T_1 & \\Phi^T_2\\end{bmatrix}\\begin{bmatrix}\\Phi_1 \\\\ \\Phi_2\\end{bmatrix}\\\\\n",
    "&= \\Phi^T_1 \\Phi_1 + \\Phi^T_2\\Phi_2 \\\\\n",
    "&= K_1 + K_2 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "* (b) Let $\\Phi_1 = \\begin{bmatrix} a_1 \\\\ \\dots \\\\ a_N\\end{bmatrix}$, and $\\Phi_2 = \\begin{bmatrix} b_1 \\\\ \\dots \\\\ b_M\\end{bmatrix}$, so we have $K_1 = \\Phi_1^T \\Phi_1 = \\sum a^2_i $ and $K_2 = \\Phi_2^T \\Phi_2 = \\sum b^2_i $\n",
    "\n",
    "$\\Phi = \\Phi_1 \\Phi_2^T = \\begin{bmatrix}v_1 \\\\ \\dots \\\\ v_N\\end{bmatrix}$ where $v_n = \\begin{bmatrix}a_nb_1 \\\\ \\dots \\\\ a_nb_M\\end{bmatrix}$\n",
    "\n",
    "So we have \n",
    "\n",
    "$K=\\Phi^T\\Phi = \\sum v_i^Tv_i = \\sum_{n=1} \\sum_{m=1} a_n^2b_m^2 = \\sum_{n=1}a^2_n \\sum_{m=1}b^2_m = K_1K_2$\n",
    "* (c) It shows that $K_1$ and $K_2$ are kernels, then so are $K_1 + K_2$ and $K_1K_2$. \n",
    "\n",
    "#### Exercise 8.16\n",
    "\n",
    "* (a) (8.30) minimizes w.r.t. $w, b, \\xi$, so the optimization variable is $u = \\begin{bmatrix} b \\\\ w \\\\ \\xi \\end{bmatrix}$\n",
    "\n",
    "* (b) There are $N$ constraints for $y_n(w^Tx_n +b) \\ge 1 - \\xi_n$, and $N$ constraints for $\\xi \\ge 0$. It's easy to see that RHS has $c=\\begin{bmatrix}1_N \\\\ 0_N \\end{bmatrix}$. \n",
    "\n",
    "On the LHS, $A=\\begin{bmatrix}YX & I_N \\\\ 0_{N\\times (d+1)} & I_N \\end{bmatrix}$ where $YX = \\begin{bmatrix}y_1 & y_1x^T_1 \\\\ \\dots & \\dots \\\\ y_n & y_nx^T_n \\\\ \\dots & \\dots \\\\ y_N & y_Nx^T_N \\end{bmatrix}$ as in exercise 8.4\n",
    "\n",
    "The $p^Tu$ term is $C\\sum\\xi_n$, so we have $p = \\begin{bmatrix}0_{(d+1)\\times 1} \\\\ C_{N\\times 1}\\end{bmatrix}$\n",
    "\n",
    "And finally the $Q = \\begin{bmatrix}0 & 0^T_d & 0^T_N \\\\ 0_d & I_d & 0_{d\\times N} \\\\ 0_N & 0_{N\\times d} & 0_{N\\times N}\\end{bmatrix}$ with some work. \n",
    "\n",
    "* (c) Once we have $u^*$, then $b^* = u^*_0$, $w^* = u^*_{1:d+1}$, $\\xi^* = u^*_{d+2:d+1+N}$\n",
    "\n",
    "* (d) Any point with $\\xi_n > 0$ violates the margin. The points with $y_n(w^Tx_n +b) = 1$ are on the margin, and the points with $y_n(w^Tx_n +b) > 1$ are correctly separated and outside the margin.\n",
    "\n",
    "#### Exercise 8.17\n",
    "\n",
    "With classification 0/1 error, $E_{in} = \\frac{1}{N}\\sum^N_{n=1}[sign(w^Tx_n+b)\\ne y_n]$ and $E_{SVM}(b,w) = \\frac{1}{N}\\sum^N_{n=1} max(1-y_n(w^Tx_n+b), 0)$.\n",
    "\n",
    "For any $n$, \n",
    "* If the point is correctly classified with no violation, then $y_n(w^Tx_n+b) \\ge 1$ and $max(1-y_n(w^Tx_n+b), 0) = 0$, so $e_{in} = e_{SVM}$\n",
    "* If the point is correctly classified with violation, then $ 1 > y_n(w^Tx_n+b) \\ge 0$, so $max(1-y_n(w^Tx_n+b), 0) = 1-y_n(w^Tx_n+b)$, so we have $0 = e_{in} < e_{SVM}$ \n",
    "* If the point is wrongly classified, then we have $y_n(w^Tx_n+b) < 0$ and $max(1-y_n(w^Tx_n+b), 0) = 1 - y_n(w^Tx_n+b) > 1$, so we have $1 = e_{in} < e_{SVM}$ \n",
    "\n",
    "We conclude that $E_{in} \\le E_{SVM}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.1\n",
    "\n",
    "let the two points be $(x_{+},y_{+})$ and $(x_{-}, y_{-})$, and $y_{+} = 1, y_{-} = -1$. From (8.4), we have to minimize\n",
    "\n",
    "$E = \\frac{1}{2}w^Tw $ with two constraints: $y_{+}(w^Tx_{+} + b) \\ge 1$ and $y_{-}(w^Tx_{-} + b) \\ge 1$. That is\n",
    "\n",
    "\\begin{align*}\n",
    "w^Tx_{+} + b &\\ge 1 \\\\\n",
    "w^Tx_{-} + b &\\le -1 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Subtract the two inequations, we have $w^T(x_{+} - x_{-}) \\ge 2$, so we have $|w^T||x_{+} - x_{-}| \\ge |w^T(x_{+} - x_{-})| \\ge 2$, then $|w| \\ge \\frac{2}{|x_{+} - x_{-}|}$. The minimal objective value is thus \n",
    "\n",
    "$E=\\frac{1}{2} \\frac{4}{|x_{+} - x_{-}|^2} = \\frac{2}{|x_{+} - x_{-}|^2}$\n",
    "\n",
    "The optimal hyperplane then have $w^* = \\frac{2}{x_{+} - x_{-}}$ and $b^* = 1 - (w^*)^Tx_{+} = \\frac{x_{+}+x_{-}}{x_{-} - x_{+}}$. \n",
    "\n",
    "The margin is thus $\\frac{1}{|w^*|} = \\frac{|x_{+} - x_{-}|}{2}$\n",
    "\n",
    "This agrees with the results from exercise 8.1\n",
    "\n",
    "#### Problem 8.2\n",
    "\n",
    "For this data, we let $w=\\begin{bmatrix}w_1 \\\\ w_2 \\end{bmatrix}$, we have the objective value\n",
    "\n",
    "\\begin{align*}\n",
    "E &= \\frac{1}{2}w^Tw \\\\\n",
    "&= \\frac{1}{2}(w^2_1 + w^2_2) \\\\\n",
    "\\end{align*}\n",
    "\n",
    "The constraints are \n",
    "\n",
    "\\begin{align*}\n",
    "y_1(w_1x_{11} + w_2x_{12} + b) &= -b \\ge 1 \\\\\n",
    "y_2(w_1x_{21} + w_2x_{22} + b) &= w_2 - b \\ge 1 \\\\\n",
    "y_3(w_1x_{31} + w_2x_{32} + b) &= -2w_1 + b \\ge 1 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Combine the first and third inequalities, we have $w_1 \\le -1$. Combine the first and the second, we have $w_2 \\ge 0$. \n",
    "So the objective achieves minimal at $w_1 = -1, w_2 = 0$, where $E=\\frac{1}{2}$.\n",
    "\n",
    "The optimal $b= -1$.\n",
    "\n",
    "The margin is thus $\\frac{1}{|w|} = \\frac{1}{\\sqrt{w^2_1+w^2_2}} = 1$\n",
    "\n",
    "\n",
    "#### Problem 8.3\n",
    "\n",
    "* (a) \n",
    "Let $\\alpha = \\begin{bmatrix}\\alpha_1 \\\\ \\alpha_2 \\\\\\alpha_3 \\\\\\alpha_4 \\end{bmatrix}$. \n",
    "\n",
    "With $Q_D=\\begin{bmatrix}0 & 0 & 0 & 0 \\\\ 0 & 8 & -4 & -6 \\\\ 0 & -4 & 4 & 6 \\\\ 0 & -6 & 6 & 9\\end{bmatrix}$, and $A_D = \\begin{bmatrix}-1 & -1 & 1 & 1 \\\\ 1 & 1 & -1 & -1 \\\\ 1 & 0 & 0 & 0 \\\\ 0 & 1 & 0 & 0 \\\\ 0 & 0 & 1 & 0 \\\\ 0 & 0 & 0 & 1\\end{bmatrix}$,  we have\n",
    "\n",
    "\\begin{align*}\n",
    "\\mathcal{L}(\\alpha) &=\\frac{1}{2}\\alpha^TQ_D\\alpha - 1^T_N\\alpha \\\\\n",
    "&=\\frac{1}{2}( 8\\alpha^2_2 -4\\alpha_2\\alpha_3 -6\\alpha_2\\alpha_4 -4\\alpha_2\\alpha_3 + 4\\alpha^2_3 + 6\\alpha_3\\alpha_4 -6\\alpha_2\\alpha_4 + 6\\alpha_3\\alpha_4+9\\alpha^2_4) - (\\alpha_1+\\alpha_2+\\alpha_3+\\alpha_4)\\\\\n",
    "&= 4\\alpha^2_2+ 2\\alpha^2_3+\\frac{9}{2}\\alpha^2_4 -4\\alpha_2\\alpha_3 -6\\alpha_2\\alpha_4  + 6\\alpha_3\\alpha_4 - \\alpha_1-\\alpha_2-\\alpha_3-\\alpha_4\\\\\n",
    "\\end{align*}\n",
    "\n",
    "The constraints are: $A_D\\alpha \\ge 0$, i.e.\n",
    "\n",
    "\\begin{align*}\n",
    "-\\alpha_1-\\alpha_2 + \\alpha_3 + \\alpha_4 &\\ge 0 \\\\\n",
    "\\alpha_1+\\alpha_2 - \\alpha_3 - \\alpha_4 &\\ge 0 \\\\\n",
    "\\alpha_1 &\\ge 0 \\\\\n",
    "\\alpha_2 &\\ge 0 \\\\\n",
    "\\alpha_3 &\\ge 0 \\\\\n",
    "\\alpha_4 &\\ge 0 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "The first two inequalities combine into $\\alpha_1+\\alpha_2 = \\alpha_3 + \\alpha_4$\n",
    "\n",
    "* (b) \n",
    "\n",
    "From the equality constraint, we have $\\alpha_1 = \\alpha_3+\\alpha_4 - \\alpha_2$, replace $\\alpha_1$ in the $\\mathcal{L}(\\alpha)$, we have \n",
    "\n",
    "\\begin{align*}\n",
    "\\mathcal{L}(\\alpha) &= 4\\alpha^2_2+ 2\\alpha^2_3+\\frac{9}{2}\\alpha^2_4 -4\\alpha_2\\alpha_3 -6\\alpha_2\\alpha_4  + 6\\alpha_3\\alpha_4 - (\\alpha_3+\\alpha_4 - \\alpha_2)-\\alpha_2-\\alpha_3-\\alpha_4\\\\\n",
    "&= 4\\alpha^2_2+ 2\\alpha^2_3+\\frac{9}{2}\\alpha^2_4 -4\\alpha_2\\alpha_3 -6\\alpha_2\\alpha_4  + 6\\alpha_3\\alpha_4 -2\\alpha_3-2\\alpha_4\\\\\n",
    "\\end{align*}\n",
    "\n",
    "* (c) Fix $\\alpha_3, \\alpha_4 \\ge 0$, take derivative of $\\mathcal{L}(\\alpha)$ w.r.t. $\\alpha_2$ and set it to zero, we have\n",
    "\n",
    "\\begin{align*}\n",
    "\\frac{\\partial{\\mathcal{L}(\\alpha)}}{\\partial{\\alpha_2}} &= 8\\alpha_2 - 4\\alpha_3 - 6\\alpha_4 \\\\\n",
    "\\alpha_2 &= \\frac{1}{2}\\alpha_3 + \\frac{3}{4}\\alpha_4\\\\\n",
    "\\end{align*}\n",
    "\n",
    "So $\\alpha_1 = \\alpha_3 + \\alpha_4 - \\alpha_2 = \\frac{1}{2}\\alpha_3 + \\frac{1}{4}\\alpha_4$\n",
    "\n",
    "Note that both $\\alpha_1 \\ge 0$ and $\\alpha_3 \\ge 0$ if $\\alpha_3,\\alpha_4\\ge0$.\n",
    "\n",
    "* (d) Now we compute $\\mathcal{L}(\\alpha)$\n",
    "\n",
    "\\begin{align*}\n",
    "\\mathcal{L}(\\alpha) &= 4\\alpha^2_2+ 2\\alpha^2_3+\\frac{9}{2}\\alpha^2_4 -4\\alpha_2\\alpha_3 -6\\alpha_2\\alpha_4  + 6\\alpha_3\\alpha_4 -2\\alpha_3-2\\alpha_4\\\\\n",
    "&= 4(\\frac{1}{2}\\alpha_3 + \\frac{3}{4}\\alpha_4)^2 + 2\\alpha^2_3+\\frac{9}{2}\\alpha^2_4 -4(\\frac{1}{2}\\alpha_3 + \\frac{3}{4}\\alpha_4)\\alpha_3-6(\\frac{1}{2}\\alpha_3 + \\frac{3}{4}\\alpha_4)\\alpha_4 +6\\alpha_3\\alpha_4-2\\alpha_3  -2\\alpha_4\\\\\n",
    "&=\\alpha^2_3 + 3\\alpha_3\\alpha_4 + \\frac{9}{4}\\alpha^2_4 + 2\\alpha^2_3 + \\frac{9}{2}\\alpha^2_4 - 2\\alpha_3^2 - 3\\alpha_3\\alpha_4 - 3\\alpha_3\\alpha_4 - \\frac{9}{2}\\alpha_4^2 + 6\\alpha_3\\alpha_4 -2\\alpha_3-2\\alpha_4\\\\\n",
    "&= \\alpha_3^2 + \\frac{9}{4}\\alpha^2_4 + 3\\alpha_3\\alpha_4 -2\\alpha_3 - 2\\alpha_4\\\\\n",
    "&= \\left(\\alpha_3 + \\frac{3\\alpha_4 -2}{2}\\right)^2 + \\alpha_4 -1 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "It's clear that $\\mathcal{L}(\\alpha)$ achieves minimal value of $-1$, when $\\alpha_4 = 0$ and $\\alpha_3 = 1$\n",
    "\n",
    "Then we have $\\alpha_2 = \\frac{1}{2}\\alpha_3 + \\frac{3}{4}\\alpha_4 = \\frac{1}{2}$ and $\\alpha_1 = \\alpha_3+\\alpha_4 - \\alpha_2 = \\frac{1}{2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.4\n",
    "\n",
    "Using data from exercise 8.2, we have $N=3$, let $\\alpha = \\begin{bmatrix}\\alpha_1 \\\\ \\alpha_2 \\\\ \\alpha_3 \\end{bmatrix}$, then we have constraints:\n",
    "\n",
    "$\\alpha_1 \\ge 0, \\alpha_2 \\ge 0, \\alpha_3 \\ge 0$ and $\\sum^N y_n\\alpha_n = -\\alpha_1 - \\alpha_2 + \\alpha_3 = 0$\n",
    "\n",
    "Let $X=\\begin{bmatrix}0 & 0 \\\\ 2 & 2 \\\\ 2 & 0 \\end{bmatrix} = \\begin{bmatrix}x^T_1 \\\\ x^T_2 \\\\ x^T_3 \\end{bmatrix}$, then we have \n",
    "\n",
    "$M = XX^T = \\begin{bmatrix}0 & 0 & 0 \\\\ 0 & 8 & 4 \\\\ 0 & 4 & 4\\end{bmatrix}$, which has $M_{ij} = x^T_ix_j$\n",
    "\n",
    "The objective value is \n",
    "\n",
    "\\begin{align*}\n",
    "\\mathcal{L}(\\alpha) &= \\frac{1}{2} \\sum^N_{m=1}\\sum^N_{n=1} y_ny_m\\alpha_n\\alpha_m x^T_nx_m - \\sum^N \\alpha_n \\\\\n",
    "&= \\frac{1}{2} \\sum^N_{m=1}y_m\\alpha_m\\sum^N_{n=1} y_n\\alpha_n x^T_nx_m - (\\alpha_1+\\alpha_2+\\alpha_3) \\\\\n",
    "&= \\frac{1}{2} \\sum^N_{m=2}y_m\\alpha_m\\sum^N_{n=2} y_n\\alpha_n x^T_nx_m - (\\alpha_1+\\alpha_2+\\alpha_3) \\\\\n",
    "&= \\frac{1}{2} (-\\alpha_2(-8\\alpha_2 + 4\\alpha_3) + \\alpha_3(-4\\alpha_2 + 4\\alpha_3)) - (\\alpha_1+\\alpha_2+\\alpha_3) \\\\\n",
    "&= 4\\alpha_2^2 - 2\\alpha_2\\alpha_3 -2\\alpha_2\\alpha_3 + 2\\alpha_3^2 - \\alpha_1 - \\alpha_2 - \\alpha_3\\\\\n",
    "&= 4\\alpha_2^2 - 4\\alpha_2\\alpha_3 + 2\\alpha_3^2 - \\alpha_1 - \\alpha_2 - \\alpha_3\\\\\n",
    "\\end{align*}\n",
    "\n",
    "Take $\\alpha_1 = \\alpha_3 - \\alpha_2$ into the formula, we have\n",
    "\n",
    "$\\mathcal{L}(\\alpha) = 4\\alpha_2^2 - 4\\alpha_2\\alpha_3 + 2\\alpha_3^2 - 2\\alpha_3 = (2\\alpha_2 - \\alpha_3)^2 + (\\alpha_3 -1)^2 -1$\n",
    "\n",
    "It's clear that at the minimal, $\\alpha_3 = 1, \\alpha_2 = \\frac{1}{2}$ so $\\alpha_1 = \\frac{1}{2}$.\n",
    "\n",
    "We compute the optimal weights $w^* = \\sum^N y_n\\alpha_nx_n = \\begin{bmatrix}1 \\\\ -1 \\end{bmatrix}$, since all $\\alpha > 0$, pick one, e.g. $\\alpha_1 = \\frac{1}{2}$, we should have $y_1(w^Tx_1 + b) = 1$, i.e. $-b= 1$, so $b=-1$.\n",
    "\n",
    "This is consistent with the results from example 8.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_random_separator_85(lb=-1, ub=1):\n",
    "    dim = 1\n",
    "    max_v = 10000\n",
    "    N = 1\n",
    "    a = data.generate_random_numbers(N, dim, max_v, lb, ub)\n",
    "    return a.flatten()[0]\n",
    "\n",
    "def get_max_margin_separator_85(df):\n",
    "    min_pos_x2 = np.min(df.loc[df['y']==1]['x2'])\n",
    "    max_neg_x2 = np.max(df.loc[df['y']==-1]['x2'])\n",
    "    margin = (min_pos_x2 - max_neg_x2)/2\n",
    "    sep = min_pos_x2 - margin\n",
    "    return sep, margin\n",
    "\n",
    "def gen_data_85(N):\n",
    "    dfp = generate_data_84(int(N/2), x1_lb = -1, x1_ub=1, x2_lb = 0, x2_ub = 1)\n",
    "    dfn = generate_data_84(int(N/2), x1_lb = -1, x1_ub=1, x2_lb = -1, x2_ub = 0)\n",
    "    df = pd.concat([dfp, dfn])\n",
    "    return df\n",
    "\n",
    "def run_one_experiment_85(N, do_print = False):\n",
    "    df = gen_data_85(N)\n",
    "    if do_print:\n",
    "        print(f\"-- Number of positive points: {df.loc[df['y']>0]['y'].count()}\")\n",
    "\n",
    "    rand_sep = get_random_separator_85()    \n",
    "    svm_sep, svm_margin = get_max_margin_separator_85(df)\n",
    "    if do_print:\n",
    "        print(f\"-- The separator selected randomly: x_2 = {rand_sep}\")  \n",
    "        print(f\"-- The separator selected by SVM: x_2 = {svm_sep}\")       \n",
    "    return df, rand_sep, svm_sep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.5 (a)-(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- Number of positive points: 3\n",
      "-- The separator selected randomly: x_2 = -0.8821882188218821\n",
      "-- The separator selected by SVM: x_2 = -0.029652965296529655\n"
     ]
    }
   ],
   "source": [
    "N=6\n",
    "df, rand_sep, svm_sep = run_one_experiment_85(N, do_print = True)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHblJREFUeJzt3X2UVPWd5/H3B2wesspzoxiEJhEjAtpo+8BkFRIBYxIEZ1Qg0QGGxHFkkugcPTFxzkzWGaMkbuKqnM2QkICGbXyKkWTMKiI4a4KuTSSKooIOaguBFgIJKyjid/+o252iqX6iqqu6+35e59Spe3/3V/f3rVvVn7p163aVIgIzM0uXbqUuwMzMis/hb2aWQg5/M7MUcvibmaWQw9/MLIUc/mZmKeTwty5D0rck/bSZ5VskTSpmTe1B0hRJP8+aD0knNtH3IknLi1eddRYOfyupJJD3Sdorabukn0g6utR15UMZ/yrpbUl7JK2RNLqZ/tnbYK+kx1oY4tvAra2pJSJWAGMkndqGu2Ap4PC3jmBqRBwNnA6cCfxj4w5JoHaW5+ulwN8A5wIDgLXAPS3cZmpEHJ1cpjTVSdKZQN+IeLoN9VQDV7ahv6VAZ/ljshSIiLeBXwFjAJI95psl/Rp4F/iYpOMlrZC0S9JmSV9utJpeku6V9CdJv5V0Wq6xJHWTdIOk1yTtlHSfpAHJsorkUMpcSW9J+oOkqySdKel5Sbsl3dXMXRkBPBURr0fEQeCnwCl5bp56FwJP5mj/rKTXJb0j6buNXijXAJ8r0PjWRTj8rcOQdALwWeC5rOYryOy1HgO8QWYvthY4HrgE+Lak87P6TwPuJ7PH/b+An0sqyzHcV4HpwIRkXX8AFjbqczYwEpgB3A7cCEwCRgOXSZrQxF1ZDpwo6aRk7NnA/27h7i+TVCfpsaZesBJjgVdytF8MVJF59zSNzDuPehuBCkl9WqjBUsThbx3BzyXtBp4is1f77axlSyLixYj4ADgO+K/A1yNif0SsB35E5gWi3rqIeCAiDgDfA3oB5+QY82+BGyOiNiLeA74FXCLpqKw+/5KM8xjw/4DqiNiRvEP5P8C4Ju7PtmT5K8A+MoeBrm3m/n8RqACGA6uBRyX1a6JvP+BPOdoXRMSuiHiTzAvVrKxl9f2bWqelkMPfOoLpEdEvIoZHxNURsS9r2VtZ08cDuyIiO/zeAD6aq39EfMif3yU0Nhx4KDmEs5vM3vFB4NisPtuzpvflmG/qg+l/JvPZxQlkXnz+G/CEpI/k6hwRv46IfRHxbkTcAuwm83lBLn8g8y6osezt9AaH3uf6/rubWKelkMPfOrrsr53dCgyQlB1+w4C3s+ZPqJ9IjnsPTW7X2FvAhcmLTv2lV7JXn6/TgHuTdxUfRMQSoD+tP+4fgJpY9jxwUo72E7Kmh3HofR4FbImIP7ZyfEsBh791GhHxFvAb4BZJvZLTF+cBy7K6nSHpL5PDN9cA7wG5zoz5AXCzpOEAksolTStQqc8Cl0o6Nvlg+QqgDNjcuKOkYZI+KalHcp+uBwYBv25i3Y+Q+Zyisesl9U8+N/kacG/WsglkPkg3a3BUy13MOpRZZIJ7K5lDIP8cESuzlj9M5gPapWTC9i+T4/+N/Q8ye9ePSToe2EEmMB8uQI0LgMHAeuC/JHX8VUTsBpD0A4CIuIrMIZn/CXwc2J/c5sKI2JlrxRHx2+R/B86OiGeyFj0MrAP6AkuAxVnLZgGXF+B+WRci/5iLWeciaQpwdURMb0XfqcAVEXFZ+1dmnYnD38wshXzM38wshRz+ZmYp5PA3M0uhDnu2z6BBg6KioqLUZZiZdSrr1q17JyLKW+rXYcO/oqKCmpqaUpdhZtapSHqjNf182MfMLIUc/mZmKeTwNzNLIYe/mVkKOfzNzFLI4W9mlkIOfzOzFHL4F1tdHTz7bObazKxEHP7FVF0Nw4fD5MmZ6+rqUldkZinl8C+WujqYNw/27YM9ezLX8+b5HYCZlYTDv1i2bIEePQ5tKyvLtJuZFZnDv1gqKuD99w9tO3Ag025mVmQO/2IpL4fFi6F3b+jTJ3O9eHGm3cysyDrst3p2SbNmwaRJmUM9FRUOfjMrGYd/sZWXO/TNrOR82MfMLIUc/mZmKeTwNzNLIYe/mVkKOfzNzFLI4W9mlkIOfzOzFHL4m5mlkMPfzCyFHP5mZilUkPCX9BlJr0jaLOmGHMv/QdJLkp6XtErS8EKMa2ZmRybv8JfUHVgIXAicAsySdEqjbs8BVRFxKvAA8J18xzUzsyNXiD3/s4DNEfF6RLwPLAemZXeIiNUR8W4y+zQwtADjmpnZESpE+H8UeCtrvjZpa8o84FcFGNfMzI5QIb7SWTnaImdH6XKgCpjQxPIrgSsBhg0bVoDSzMwsl0Ls+dcCJ2TNDwW2Nu4kaRJwI3BRRLyXa0URsSgiqiKiqtzfeW9m1m4KEf7PAiMljZDUA5gJrMjuIGkc8G9kgn9HAcY0M7M85B3+EfEB8PfAo8BG4L6IeFHSTZIuSrp9FzgauF/SekkrmlidmZkVQUF+xjEiHgEeadT2T1nTkwoxjpmZFYb/w9fMLIUc/mZmKeTwNzNLIYe/mVkKOfzNzFLI4W9mlkIOfzOzFHL4m5mlkMPfzCyFHP5mZink8DczSyGHv5lZCjn8S6WuDp59NnNtZlZkDv9SqK6G4cNh8uTMdXV1qSsys5Rx+BdbXR3Mmwf79sGePZnrefP8DsDMisrhX2xbtkCPHoe2lZVl2s3MisThX2wVFfD++4e2HTiQaTczKxKHf7GVl8PixdC7N/Tpk7levDjTbmZWJAX5GUdro1mzYNKkzKGeigoHv5kVncO/VMrLHfpmVjLpOezj8+rNzBqkI/x9Xr2Z2SG6fvj7vHozs8N0/fD3efVmZocpSPhL+oykVyRtlnRDjuU9Jd2bLH9GUkUhxm0Vn1dvZnaYvMNfUndgIXAhcAowS9IpjbrNA/4QEScC3wcW5Dtuq/m8ejOzwxTiVM+zgM0R8TqApOXANOClrD7TgG8l0w8Ad0lSREQBxm+Zz6s3MztEIcL/o8BbWfO1wNlN9YmIDyTtAQYC72R3knQlcCXAsGHDClBaFp9Xb2bWoBDH/JWjrfEefWv6EBGLIqIqIqrKHdRmZu2mEOFfC5yQNT8U2NpUH0lHAX2BXQUY28zMjkAhwv9ZYKSkEZJ6ADOBFY36rABmJ9OXAE8U7Xi/mZkdJu9j/skx/L8HHgW6Az+OiBcl3QTURMQKYDFwj6TNZPb4Z+Y7bnOuuQbWr2/PEczMCq+yEm6/vThjFeSL3SLiEeCRRm3/lDW9H7i0EGOZmVn+uuS3ehbrldPMrLPq+l/vYGZmh3H4m5mlkMPfzCyFHP5mZink8DczSyGHv3UKEydmLmZWGA5/6/AmToQnn8xc/AJgVhgOfzOzFOpU/+R14MABamtr2b9/f6lL6TR69erF0KFDKSsrK3UpR2zNmj/v8a9ZU8JCzLqQThX+tbW1HHPMMVRUVCDl+pZoyxYR7Ny5k9raWkaMGFHqcvLi0DcrrE512Gf//v0MHDjQwd9Kkhg4cKDfKZnZYTpV+AMO/jby9jKzXDpd+Jda9+7dqaysZMyYMUydOpXdu3cXZL1btmxhzJgxBVmXmVlLHP5t1Lt3b9avX8+GDRsYMGAACxcuLHVJZmZt5vDPw/jx43n77bcB2Lt3L+effz6nn346Y8eO5eGHHwYye/SjRo3iy1/+MqNHj2bKlCns27cPgHXr1nHaaacxfvz4Q15E9u/fz9y5cxk7dizjxo1j9erVACxZsoTp06czdepURowYwV133cX3vvc9xo0bxznnnMOuXf5lTDNrnU51ts8h2uPnutrwMzoHDx5k1apVzJs3D8icUvnQQw/Rp08f3nnnHc455xwuuugiADZt2kR1dTU//OEPueyyy3jwwQe5/PLLmTt3LnfeeScTJkzg+uuvb1h3/QvBCy+8wMsvv8yUKVN49dVXAdiwYQPPPfcc+/fv58QTT2TBggU899xzXHvttdx9991cc801hdwiZtZFec+/jfbt20dlZSUDBw5k165dTJ48GcicVvnNb36TU089lUmTJvH222+zfft2AEaMGEFlZSUAZ5xxBlu2bGHPnj3s3r2bCRMmAHDFFVc0jPHUU081zJ988skMHz68Ifw/9alPccwxx1BeXk7fvn2ZOnUqAGPHjmXLli1F2QZm1vl13j3/Ev1cV/0x/z179vD5z3+ehQsX8tWvfpVly5ZRV1fHunXrKCsro6KiouEUy549ezbcvnv37uzbt4+IaPJMnOZ+2z57Xd26dWuY79atGx988EEh7qKZpYD3/I9Q3759ueOOO7jttts4cOAAe/bsYfDgwZSVlbF69WreeOONZm/fr18/+vbty1NPPQXAsmXLGpadd955DfOvvvoqb775Jp/4xCfa786YWeo4/PMwbtw4TjvtNJYvX84Xv/hFampqqKqqYtmyZZx88skt3v4nP/kJ8+fPZ/z48fTu3buh/eqrr+bgwYOMHTuWGTNmsGTJkkP2+M3M8qXmDjGUUlVVVdTU1BzStnHjRkaNGlWiijovbzez9JC0LiKqWurnPX8zsxRy+JuZpVBe4S9pgKSVkjYl1/1z9KmUtFbSi5KelzQjnzHNzCx/+e753wCsioiRwKpkvrF3gb+OiNHAZ4DbJfXLc1wzM8tDvuE/DViaTC8FpjfuEBGvRsSmZHorsAMoz3NcMzPLQ77hf2xEbANIrgc311nSWUAP4LUmll8pqUZSTV1dXZ6lmZlZU1oMf0mPS9qQ4zKtLQNJGgLcA8yNiA9z9YmIRRFRFRFV5eUd983BzTffzOjRozn11FOprKzkwgsv5Bvf+MYhfdavX99wemVFRQXnnnvuIcvrvxbazKwUWvx6h4iY1NQySdslDYmIbUm472iiXx/g34F/jIinj7jaDmDt2rX88pe/5Le//S09e/bknXfe4cUXX2Tu3LnccsstDf2WL1/OF77whYb5P/3pT7z11luccMIJbNy4sRSlm5k1yPewzwpgdjI9G3i4cQdJPYCHgLsj4v48xyu5bdu2MWjQoIb/uB00aBATJkygX79+PPPMMw397rvvPmbOnNkwf9lll3HvvfcCUF1dzaxZs4pbuJlZlny/2O1W4D5J84A3gUsBJFUBV0XEl4DLgPOAgZLmJLebExF5fR9zqb7RecqUKdx0002cdNJJTJo0iRkzZjBhwgRmzZrF8uXLOfvss3n66acZOHAgI0eObLjdJZdcwpw5c7juuuv4xS9+wbJly7jnnnsKewfMzFoprz3/iNgZEedHxMjkelfSXpMEPxHx04goi4jKrEuBY7t4jj76aNatW8eiRYsoLy9v+O6dmTNn8sADD/Dhhx+yfPnyw/bsBwwYQP/+/Vm+fDmjRo3iIx/5SInugZlZJ/5K5xJ9ozOQ+VrmiRMnMnHiRMaOHcvSpUuZM2cOFRUVPPnkkzz44IOsXbv2sNvNmDGD+fPns2TJkuIXbWaWpdOGf6m88sordOvWreGQzvr16xk+fDgAs2bN4tprr+XjH/84Q4cOPey2F198Mdu2beOCCy5g69atRa3bzCybw7+N9u7dy1e+8hV2797NUUcdxYknnsiiRYsAuPTSS/na177GnXfemfO2xxxzDF//+teLWa6ZWU4O/zY644wz+M1vfpNzWXl5OQcOHDisPdfPK1ZUVLBhw4ZCl2dm1ir+Vk8zsxRy+JuZpZDD38wshRz+ZmYp5PA3M0shh7+ZWQo5/Avo/vvvZ/To0XTr1o2amppSl2NmncjEiZlLsTj8j9CaNWuYM2fOIW1jxozhZz/7Geedd15pijKzTmniRHjyycylWC8AXf+fvOrqYMsWqKiAdv6BmPofbzEz6+i69p5/dTUMHw6TJ2euq6tLXZGZ2WHWrIEJEzKXNWuKM2bX3fOvq4N582DfvswFMvOTJuX1DuDss8/mvffeY+/evezatYvKykoAFixYwAUXXFCIys0shYoV+vW6bvhv2QI9evw5+AHKyjLteYR//a91rVmzhiVLlvjrmc2sU+q6h30qKuD99w9tO3Ag025mlnJdN/zLy2HxYujdG/r0yVwvXtyuH/o+9NBDDB06lLVr1/K5z33Oh4HMrMNSRJS6hpyqqqqi8bnyGzdubPsZNUU826ejOqLtZmadkqR1EVHVUr+ue8y/Xnl5akPfzKwpXfewj5mZNcnhb2aWQg5/M7MUcvibmaVQXuEvaYCklZI2Jdf9m+nbR9Lbku7KZ0wzM8tfvnv+NwCrImIksCqZb8q/AE/mOV6H8vLLLzN+/Hh69uzJbbfdVupyzMxaLd/wnwYsTaaXAtNzdZJ0BnAs8Fie43UoAwYM4I477uC6664rdSlmZm2Sb/gfGxHbAJLrwY07SOoG/Hfg+pZWJulKSTWSaurq6vIsrf0NHjyYM888k7KyslKXYmbWJi3+k5ekx4Hjciy6sZVjXA08EhFvSWq2Y0QsAhZB5j98W7n+ZtX/MEKxvzHPzKwjazH8I2JSU8skbZc0JCK2SRoC7MjRbTxwrqSrgaOBHpL2RkRznw8URP2v49RP+wXAzCwj38M+K4DZyfRs4OHGHSLiixExLCIqgOuAu4sR/O1l4cKFVFZWUllZydatW0tdjpnZEcn3u31uBe6TNA94E7gUQFIVcFVEfCnP9edlzZrCH/aZP38+8+fPL8zKLPV8WNJKJa/wj4idwPk52muAw4I/IpYAS/IZs63a84/q97//PVVVVfzxj3+kW7du3H777bz00kv06dOn/Qa1LsOHJa2Uuv63eraj4447jtra2lKXYWbWZg5/sxJpj8OSZq3l8DcrIYe+lYq/2M3MLIU6Xfh31J+d7Ki8vcwsl04V/r169WLnzp0OtFaKCHbu3EmvXr1KXYqZdTCd6pj/0KFDqa2tpTN8709H0atXL4YOHVrqMsysg+lU4V9WVsaIESNKXYaZWafXqQ77mJlZYTj8zcxSyOFvZpZCDn8zsxRy+JuZpZDD38wshRz+ZmYp5PA3M0shh7+ZWQo5/M3MUsjhb2aWQg5/M7MUcvibmaWQw9/MLIUc/mZmKeTwNzNLobzCX9IASSslbUqu+zfRb5ikxyRtlPSSpIp8xjUzs/zku+d/A7AqIkYCq5L5XO4GvhsRo4CzgB15jmtmZnnIN/ynAUuT6aXA9MYdJJ0CHBURKwEiYm9EvJvnuGZmlod8w//YiNgGkFwPztHnJGC3pJ9Jek7SdyV1z7UySVdKqpFU4x9pNzNrPy3+gLukx4Hjciy6sQ1jnAuMA94E7gXmAIsbd4yIRcAigKqqqmjl+s3MrI1aDP+ImNTUMknbJQ2JiG2ShpD7WH4t8FxEvJ7c5ufAOeQIfzMzK458D/usAGYn07OBh3P0eRboL6k8mf808FKe45qZWR7yDf9bgcmSNgGTk3kkVUn6EUBEHASuA1ZJegEQ8MM8xzUzszy0eNinORGxEzg/R3sN8KWs+ZXAqfmMZWZmheP/8DUzSyGHv5lZCjn8zcxSyOFvZpZCDn8zsxRy+JuZpZDD38wshRz+ZmYp5PA3M0shh7+ZWQo5/M3MUsjhb2aWQg5/M7MUcvibmaWQw9/MLIUc/mZmKeTwNzNLIYe/mVkKOfzNzFLI4W9mlkIOfzOzFHL4m5mlkMPfzCyF8gp/SQMkrZS0Kbnu30S/70h6UdJGSXdIUj7jmplZfvLd878BWBURI4FVyfwhJP0F8EngVGAMcCYwIc9xzcwsD/mG/zRgaTK9FJieo08AvYAeQE+gDNie57hmZpaHfMP/2IjYBpBcD27cISLWAquBbcnl0YjYmGtlkq6UVCOppq6uLs/SzMysKUe11EHS48BxORbd2JoBJJ0IjAKGJk0rJZ0XEf/RuG9ELAIWAVRVVUVr1m9mZm3XYvhHxKSmlknaLmlIRGyTNATYkaPbxcDTEbE3uc2vgHOAw8LfzMyKI9/DPiuA2cn0bODhHH3eBCZIOkpSGZkPe3Me9jEzs+LIN/xvBSZL2gRMTuaRVCXpR0mfB4DXgBeA3wG/i4hf5DmumZnlocXDPs2JiJ3A+Tnaa4AvJdMHgb/NZxwzMyss/4evmVkKOfzNzFLI4W9mlkIOfzOzFHL4m5mlkMPfzCyFHP5mZimU13n+HdY118D69aWuwsysbSor4fbbizKU9/zNzFKoa+75F+mV08yss/Kev5lZCjn8zcxSyOFvZpZCDn8zsxRy+JuZpZDD38wshRz+ZmYp5PA3M0shRUSpa8hJUh3wRh6rGAS8U6ByCsl1tY3raruOWpvrapsjrWt4RJS31KnDhn++JNVERFWp62jMdbWN62q7jlqb62qb9q7Lh33MzFLI4W9mlkJdOfwXlbqAJriutnFdbddRa3NdbdOudXXZY/5mZta0rrznb2ZmTXD4m5mlUKcOf0mXSnpR0oeSmjwlStJnJL0iabOkG7LaR0h6RtImSfdK6lGgugZIWpmsd6Wk/jn6fErS+qzLfknTk2VLJP1n1rLKYtWV9DuYNfaKrPZSbq9KSWuTx/t5STOylhV0ezX1fMla3jO5/5uT7VGRtewbSfsrki7Ip44jqOsfJL2UbJ9VkoZnLcv5mBaprjmS6rLG/1LWstnJ475J0uwi1/X9rJpelbQ7a1l7bq8fS9ohaUMTyyXpjqTu5yWdnrWscNsrIjrtBRgFfAJYA1Q10ac78BrwMaAH8DvglGTZfcDMZPoHwN8VqK7vADck0zcAC1roPwDYBXwkmV8CXNIO26tVdQF7m2gv2fYCTgJGJtPHA9uAfoXeXs09X7L6XA38IJmeCdybTJ+S9O8JjEjW072IdX0q6zn0d/V1NfeYFqmuOcBdOW47AHg9ue6fTPcvVl2N+n8F+HF7b69k3ecBpwMbmlj+WeBXgIBzgGfaY3t16j3/iNgYEa+00O0sYHNEvB4R7wPLgWmSBHwaeCDptxSYXqDSpiXra+16LwF+FRHvFmj8prS1rgal3l4R8WpEbEqmtwI7gBb/i/EI5Hy+NFPvA8D5yfaZBiyPiPci4j+Bzcn6ilJXRKzOeg49DQwt0Nh51dWMC4CVEbErIv4ArAQ+U6K6ZgHVBRq7WRHxH2R29poyDbg7Mp4G+kkaQoG3V6cO/1b6KPBW1nxt0jYQ2B0RHzRqL4RjI2IbQHI9uIX+Mzn8iXdz8pbv+5J6FrmuXpJqJD1dfyiKDrS9JJ1FZm/utazmQm2vpp4vOfsk22MPme3Tmtu2Z13Z5pHZe6yX6zEtZl1/lTw+D0g6oY23bc+6SA6PjQCeyGpur+3VGk3VXtDt1eF/wF3S48BxORbdGBEPt2YVOdqimfa862rtOpL1DAHGAo9mNX8D+D2ZgFsEfB24qYh1DYuIrZI+Bjwh6QXgjzn6lWp73QPMjogPk+Yj3l65hsjR1vh+tstzqgWtXreky4EqYEJW82GPaUS8luv27VDXL4DqiHhP0lVk3jV9upW3bc+66s0EHoiIg1lt7bW9WqMoz68OH/4RMSnPVdQCJ2TNDwW2kvnCpH6Sjkr23urb865L0nZJQyJiWxJWO5pZ1WXAQxFxIGvd25LJ9yT9BLiumHUlh1WIiNclrQHGAQ9S4u0lqQ/w78A/Jm+H69d9xNsrh6aeL7n61Eo6CuhL5m18a27bnnUhaRKZF9QJEfFefXsTj2khwqzFuiJiZ9bsD4EFWbed2Oi2awpQU6vqyjITmJ/d0I7bqzWaqr2g2ysNh32eBUYqc6ZKDzIP9IrIfIKymszxdoDZQGveSbTGimR9rVnvYccakwCsP84+Hch5VkB71CWpf/1hE0mDgE8CL5V6eyWP3UNkjoXe32hZIbdXzudLM/VeAjyRbJ8VwExlzgYaAYwE/m8etbSpLknjgH8DLoqIHVntOR/TItY1JGv2ImBjMv0oMCWprz8whUPfAbdrXUltnyDz4enarLb23F6tsQL46+Ssn3OAPckOTmG3V3t9ol2MC3AxmVfD94DtwKNJ+/HAI1n9Pgu8SuaV+8as9o+R+ePcDNwP9CxQXQOBVcCm5HpA0l4F/CirXwXwNtCt0e2fAF4gE2I/BY4uVl3AXyRj/y65ntcRthdwOXAAWJ91qWyP7ZXr+ULmMNJFyXSv5P5vTrbHx7Jue2Nyu1eACwv8fG+prseTv4P67bOipce0SHXdAryYjL8aODnrtn+TbMfNwNxi1pXMfwu4tdHt2nt7VZM5W+0AmfyaB1wFXJUsF7AwqfsFss5kLOT28tc7mJmlUBoO+5iZWSMOfzOzFHL4m5mlkMPfzCyFHP5mZink8DczSyGHv5lZCv1/hpSxRJZd2EIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xsp1 = df.loc[df['y']==1]['x1'].values\n",
    "ysp1 = df.loc[df['y']==1]['x2'].values\n",
    "xsm1 = df.loc[df['y']==-1]['x1'].values\n",
    "ysm1 = df.loc[df['y']==-1]['x2'].values\n",
    "\n",
    "x1_min, x1_max = 0, 1\n",
    "x2_min, x2_max = -1, 1\n",
    "xx1, xx2 = myplot.get_grid(x1_min, x1_max, x2_min, x2_max, step=0.02)\n",
    "\n",
    "\n",
    "x1 = np.arange(-1, 1, 0.02)\n",
    "rand_y = rand_sep * np.ones(x1.shape)\n",
    "svm_y = svm_sep * np.ones(x1.shape)\n",
    "\n",
    "myplot.plt_plot([x1, x1], [rand_y, svm_y], 'plot', \n",
    "               colors = ['r', 'b', 'g'], markers = [None, None], labels = ['Random', 'SVM'], \n",
    "               title = \"Problem 8.5 (b)\", yscale = None, ylb = -1, yub = 1,\n",
    "               xlb = -1, xub = 1, xlabel = None, ylabel = None,\n",
    "               legends = ['Random', 'SVM'], legendx = None, legendy = None, marker_sizes=[5, 5])\n",
    "\n",
    "myplot.plt_plot([xsp1, xsm1], [ysp1, ysm1], 'scatter', \n",
    "               colors = ['r', 'b'], markers = ['o', '+'], labels = ['+1', '-1'], \n",
    "               title = \"Problem 8.5 (b)\", yscale = None, ylb = -1, yub = 1,\n",
    "               xlb = -1, xub = 1, xlabel = None, ylabel = None,\n",
    "               legends = ['+1', '-1'], legendx = None, legendy = None, marker_sizes=[20, 20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.5 (c)-(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random algo: bias = 6.003535985229288e-07, variance = 0.33376439549984216\n",
      "SVM algo: bias = 1.1452071413084401e-08, variance = 0.018813320545832343\n"
     ]
    }
   ],
   "source": [
    "def calc_average_hypothesis(seps):\n",
    "    return np.mean(seps)\n",
    "\n",
    "def calc_bias(seps, target_sep):\n",
    "    avg_sep = calc_average_hypothesis(seps)\n",
    "    return (avg_sep - target_sep)**2\n",
    "\n",
    "def calc_variance(seps):\n",
    "    return np.var(seps)\n",
    "\n",
    "tot_exps = 1000000\n",
    "N = 6\n",
    "target_sep = 0 # target function is x_2 = 0\n",
    "\n",
    "rand_seps = []\n",
    "svm_seps = []\n",
    "for t in np.arange(tot_exps):\n",
    "    df, rand_sep, svm_sep = run_one_experiment_85(N, do_print = False)\n",
    "    rand_seps.append(rand_sep)\n",
    "    svm_seps.append(svm_sep)\n",
    "    \n",
    "rand_bias = calc_bias(rand_seps, target_sep)\n",
    "svm_bias = calc_bias(svm_seps, target_sep)\n",
    "\n",
    "rand_var = calc_variance(rand_seps)\n",
    "svm_var = calc_variance(svm_seps)\n",
    "print(f\"Random algo: bias = {rand_bias}, variance = {rand_var}\")\n",
    "print(f\"SVM algo: bias = {svm_bias}, variance = {svm_var}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([49649., 50531., 50066., 49792., 49963., 49749., 49870., 49811.,\n",
       "        50261., 49820., 49760., 49696., 50161., 50296., 49946., 50323.,\n",
       "        49789., 49926., 50441., 50150.]),\n",
       " array([-1. , -0.9, -0.8, -0.7, -0.6, -0.5, -0.4, -0.3, -0.2, -0.1,  0. ,\n",
       "         0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ]),\n",
       " <a list of 20 Patch objects>)"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD8CAYAAACcjGjIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAE8tJREFUeJzt3X+s3fV93/HnK3Yh2boEExzmYlIT1etCM5UQi1iLtCYQgaFTTDXYjNbhZkxWGJk6bdMCyyS2pGjJ/hgTWprWLR6QrSGULsJrzTyXH6omAcFZCAQY8YVkwbOHnRpooiikkPf+OJ8bnflzru+5P88FPx/S0fl+39/P93ve53uu7+ue7/d7jlNVSJI07E2TbkCStPIYDpKkjuEgSeoYDpKkjuEgSeoYDpKkjuEgSeoYDpKkjuEgSeqsnnQD83XGGWfUhg0bJt2GJL1ufPWrX/1uVa0dZ+zrNhw2bNjA/v37J92GJL1uJPnf4471sJIkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqTNWOCT5dpInkjyWZH+rnZ5kX5ID7X5NqyfJLUmmkjye5Pyh7Wxv4w8k2T5Uf1/b/lRbN4v9RCVJ45vLO4cPVdV5VbWpzV8P3FdVG4H72jzApcDGdtsBfB4GYQLcCLwfuAC4cTpQ2pgdQ+ttmfczkiQt2EI+Ib0V+GCbvh14EPhEq99RVQU8nOS0JOva2H1VdQwgyT5gS5IHgbdW1UOtfgdwOXDvAnpbMhuu/6N5r/vtz/zyInYiaZLe6L8Lxg2HAv57kgJ+u6p2AmdW1WGAqjqc5B1t7FnA80PrHmy1E9UPjqhLC/JG/8d7vIU834VayP462V4neH0853HD4QNVdagFwL4k/+sEY0edL6h51PsNJzsYHH7ine9854k7XoEm+QMxqcd+Pfwj0MJNMpgW4vXa93IYKxyq6lC7P5LkywzOGbyQZF1717AOONKGHwTOHlp9PXCo1T94XP3BVl8/YvyoPnYCOwE2bdo0MkDG8Xr8gZhkz5N67JPxOWtufJ2WzqwnpJP8xSR/aXoauBj4BrAbmL7iaDtwT5veDVzdrlraDLzcDj/tBS5OsqadiL4Y2NuWfS/J5naV0tVD25IkTcA47xzOBL7cri5dDfxeVf23JI8CdyW5BvgOcGUbvwe4DJgCfgB8FKCqjiX5NPBoG/ep6ZPTwLXAbcBbGJyIXpEnoyXpZDFrOFTVc8Avjqj/KXDRiHoB182wrV3ArhH1/cB7xuhXkrQM/IS0JKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOmOHQ5JVSb6W5A/b/DlJHklyIMmXkpzS6qe2+am2fMPQNm5o9WeSXDJU39JqU0muX7ynJ0maj7m8c/h14Omh+c8CN1fVRuBF4JpWvwZ4sap+Dri5jSPJucA24BeALcBvtsBZBXwOuBQ4F7iqjZUkTchY4ZBkPfDLwO+2+QAXAne3IbcDl7fprW2etvyiNn4rcGdVvVJV3wKmgAvabaqqnquqHwF3trGSpAkZ953Dvwf+OfDjNv924KWqerXNHwTOatNnAc8DtOUvt/E/qR+3zkx1SdKEzBoOSf4mcKSqvjpcHjG0Zlk21/qoXnYk2Z9k/9GjR0/QtSRpIcZ55/AB4CNJvs3gkM+FDN5JnJZkdRuzHjjUpg8CZwO05W8Djg3Xj1tnpnqnqnZW1aaq2rR27doxWpckzces4VBVN1TV+qrawOCE8v1V9XeBB4Ar2rDtwD1tenebpy2/v6qq1be1q5nOATYCXwEeBTa2q59OaY+xe1GenSRpXlbPPmRGnwDuTPIbwNeAW1v9VuALSaYYvGPYBlBVTya5C3gKeBW4rqpeA0jycWAvsArYVVVPLqAvSdICzSkcqupB4ME2/RyDK42OH/ND4MoZ1r8JuGlEfQ+wZy69SJKWjp+QliR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUsdwkCR1DAdJUmfWcEjy5iRfSfL1JE8m+detfk6SR5IcSPKlJKe0+qltfqot3zC0rRta/ZkklwzVt7TaVJLrF/9pSpLmYpx3Dq8AF1bVLwLnAVuSbAY+C9xcVRuBF4Fr2vhrgBer6ueAm9s4kpwLbAN+AdgC/GaSVUlWAZ8DLgXOBa5qYyVJEzJrONTA99vsT7VbARcCd7f67cDlbXprm6ctvyhJWv3Oqnqlqr4FTAEXtNtUVT1XVT8C7mxjJUkTMtY5h/YX/mPAEWAf8CzwUlW92oYcBM5q02cBzwO05S8Dbx+uH7fOTHVJ0oSMFQ5V9VpVnQesZ/CX/rtHDWv3mWHZXOudJDuS7E+y/+jRo7M3LkmalzldrVRVLwEPApuB05KsbovWA4fa9EHgbIC2/G3AseH6cevMVB/1+DuralNVbVq7du1cWpckzcE4VyutTXJam34L8GHgaeAB4Io2bDtwT5ve3eZpy++vqmr1be1qpnOAjcBXgEeBje3qp1MYnLTevRhPTpI0P6tnH8I64PZ2VdGbgLuq6g+TPAXcmeQ3gK8Bt7bxtwJfSDLF4B3DNoCqejLJXcBTwKvAdVX1GkCSjwN7gVXArqp6ctGeoSRpzmYNh6p6HHjviPpzDM4/HF//IXDlDNu6CbhpRH0PsGeMfiVJy8BPSEuSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKljOEiSOoaDJKkzazgkOTvJA0meTvJkkl9v9dOT7EtyoN2vafUkuSXJVJLHk5w/tK3tbfyBJNuH6u9L8kRb55YkWYonK0kazzjvHF4F/mlVvRvYDFyX5FzgeuC+qtoI3NfmAS4FNrbbDuDzMAgT4Ebg/cAFwI3TgdLG7Bhab8vCn5okab5mDYeqOlxV/7NNfw94GjgL2Arc3obdDlzeprcCd9TAw8BpSdYBlwD7qupYVb0I7AO2tGVvraqHqqqAO4a2JUmagDmdc0iyAXgv8AhwZlUdhkGAAO9ow84Cnh9a7WCrnah+cERdkjQhY4dDkp8G/gD4x1X1ZycaOqJW86iP6mFHkv1J9h89enS2liVJ8zRWOCT5KQbB8J+r6r+08gvtkBDt/kirHwTOHlp9PXBolvr6EfVOVe2sqk1VtWnt2rXjtC5JmodxrlYKcCvwdFX9u6FFu4HpK462A/cM1a9uVy1tBl5uh532AhcnWdNORF8M7G3Lvpdkc3usq4e2JUmagNVjjPkA8PeAJ5I81mr/AvgMcFeSa4DvAFe2ZXuAy4Ap4AfARwGq6liSTwOPtnGfqqpjbfpa4DbgLcC97SZJmpBZw6Gq/gejzwsAXDRifAHXzbCtXcCuEfX9wHtm60WStDz8hLQkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6hoMkqWM4SJI6s4ZDkl1JjiT5xlDt9CT7khxo92taPUluSTKV5PEk5w+ts72NP5Bk+1D9fUmeaOvckiSL/SQlSXMzzjuH24Atx9WuB+6rqo3AfW0e4FJgY7vtAD4PgzABbgTeD1wA3DgdKG3MjqH1jn8sSdIymzUcqupPgGPHlbcCt7fp24HLh+p31MDDwGlJ1gGXAPuq6lhVvQjsA7a0ZW+tqoeqqoA7hrYlSZqQ+Z5zOLOqDgO0+3e0+lnA80PjDrbaieoHR9QlSRO02CekR50vqHnUR2882ZFkf5L9R48enWeLkqTZzDccXmiHhGj3R1r9IHD20Lj1wKFZ6utH1Eeqqp1VtamqNq1du3aerUuSZjPfcNgNTF9xtB24Z6h+dbtqaTPwcjvstBe4OMmadiL6YmBvW/a9JJvbVUpXD21LkjQhq2cbkOSLwAeBM5IcZHDV0WeAu5JcA3wHuLIN3wNcBkwBPwA+ClBVx5J8Gni0jftUVU2f5L6WwRVRbwHubTdJ0gTNGg5VddUMiy4aMbaA62bYzi5g14j6fuA9s/UhSVo+fkJaktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJHcNBktQxHCRJnRUTDkm2JHkmyVSS6yfdjySdzFZEOCRZBXwOuBQ4F7gqybmT7UqSTl4rIhyAC4Cpqnquqn4E3AlsnXBPknTSWinhcBbw/ND8wVaTJE3A6kk30GRErbpByQ5gR5v9fpJn5vl4ZwDfnee6S8m+5sa+5sa+5mZF9pXPLqivnx134EoJh4PA2UPz64FDxw+qqp3AzoU+WJL9VbVpodtZbPY1N/Y1N/Y1Nyd7XyvlsNKjwMYk5yQ5BdgG7J5wT5J00loR7xyq6tUkHwf2AquAXVX15ITbkqST1ooIB4Cq2gPsWaaHW/ChqSViX3NjX3NjX3NzUveVqu68ryTpJLdSzjlIklaQN2w4JLkyyZNJfpxkxjP7M31tRzs5/kiSA0m+1E6UL0ZfpyfZ17a7L8maEWM+lOSxodsPk1zelt2W5FtDy85brr7auNeGHnv3UH2S++u8JA+11/vxJH9naNmi7q/ZvuYlyant+U+1/bFhaNkNrf5MkksW0sc8+vonSZ5q++e+JD87tGzka7pMff1akqNDj/8PhpZtb6/7gSTbl7mvm4d6+maSl4aWLcn+SrIryZEk35hheZLc0np+PMn5Q8sWf19V1RvyBrwb+HngQWDTDGNWAc8C7wJOAb4OnNuW3QVsa9O/BVy7SH39W+D6Nn098NlZxp8OHAP+Qpu/DbhiCfbXWH0B35+hPrH9BfwVYGOb/hngMHDaYu+vE/28DI35h8BvteltwJfa9Llt/KnAOW07q5axrw8N/QxdO93XiV7TZerr14D/MGLd04Hn2v2aNr1mufo6bvw/YnCRzFLvr78BnA98Y4bllwH3Mvhc2GbgkaXcV2/Ydw5V9XRVzfYhuZFf25EkwIXA3W3c7cDli9Ta1ra9cbd7BXBvVf1gkR5/JnPt6ycmvb+q6ptVdaBNHwKOAGsX6fGHjfM1L8P93g1c1PbPVuDOqnqlqr4FTLXtLUtfVfXA0M/Qwww+S7TUFvK1OJcA+6rqWFW9COwDtkyor6uALy7SY8+oqv6EwR+CM9kK3FEDDwOnJVnHEu2rN2w4jGmmr+14O/BSVb16XH0xnFlVhwHa/TtmGb+N/gfzpva28uYkpy5zX29Osj/Jw9OHulhB+yvJBQz+Gnx2qLxY+2ucr3n5yZi2P15msH+W8iti5rrtaxj8BTpt1Gu6nH39rfb63J1k+sOwK2J/tcNv5wD3D5WXan/NZqa+l2RfrZhLWecjyR8Df3nEok9W1T3jbGJErU5QX3Bf426jbWcd8NcYfP5j2g3A/2XwC3An8AngU8vY1zur6lCSdwH3J3kC+LMR4ya1v74AbK+qH7fyvPfXqIcYUTv+eS7Jz9Qsxt52kl8FNgG/NFTuXtOqenbU+kvQ138FvlhVryT5GIN3XReOue5S9jVtG3B3Vb02VFuq/TWbZf3Zel2HQ1V9eIGbmOlrO77L4C3b6vbX38iv85hPX0leSLKuqg63X2ZHTrCpvw18uar+fGjbh9vkK0n+I/DPlrOvdtiGqnouyYPAe4E/YML7K8lbgT8C/mV7yz297XnvrxHG+ZqX6TEHk6wG3sbgUMFYXxGzhH2R5MMMAveXquqV6foMr+li/LKbta+q+tOh2d8BPju07gePW/fBRehprL6GbAOuGy4s4f6azUx9L8m+OtkPK4382o4anOV5gMHxfoDtwDjvRMaxu21vnO12xzrbL8jp4/yXAyOvbFiKvpKsmT4sk+QM4APAU5PeX+21+zKD47G/f9yyxdxf43zNy3C/VwD3t/2zG9iWwdVM5wAbga8soJc59ZXkvcBvAx+pqiND9ZGv6TL2tW5o9iPA0216L3Bx628NcDH//zvoJe2r9fbzDE7wPjRUW8r9NZvdwNXtqqXNwMvtj5+l2VdLcdZ9JdyAX2GQqK8ALwB7W/1ngD1D4y4Dvskg+T85VH8Xg3+8U8DvA6cuUl9vB+4DDrT701t9E/C7Q+M2AP8HeNNx698PPMHgl9x/An56ufoC/np77K+3+2tWwv4CfhX4c+Cxodt5S7G/Rv28MDhM9ZE2/eb2/Kfa/njX0LqfbOs9A1y6yD/vs/X1x+3fwfT+2T3ba7pMff0b4Mn2+A8Af3Vo3b/f9uMU8NHl7KvN/yvgM8ett2T7i8Efgofbz/JBBueGPgZ8rC0Pg/8U7dn22JuG1l30feUnpCVJnZP9sJIkaQTDQZLUMRwkSR3DQZLUMRwkSR3DQZLUMRwkSR3DQZLU+X8ZVtmgTwecbAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(rand_seps, bins = 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([   254.,   1827.,   5708.,  13042.,  24334.,  40422.,  61350.,\n",
       "         87105., 118854., 155067., 152396., 115131.,  84326.,  59266.,\n",
       "         38915.,  22977.,  12077.,   5187.,   1562.,    200.]),\n",
       " array([-0.48054805, -0.43224822, -0.38394839, -0.33564856, -0.28734873,\n",
       "        -0.2390489 , -0.19074907, -0.14244924, -0.09414941, -0.04584958,\n",
       "         0.00245025,  0.05075008,  0.0990499 ,  0.14734973,  0.19564956,\n",
       "         0.24394939,  0.29224922,  0.34054905,  0.38884888,  0.43714871,\n",
       "         0.48544854]),\n",
       " <a list of 20 Patch objects>)"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAD8CAYAAACLrvgBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAF+RJREFUeJzt3X+wXOV93/H3J1IhdmLML4GJJCpSK2kx48bmBkgzSRzLBmE8iD8gFpMG2dWMpjbEadPUFnVmmLHNjJy0JWZi06pGRXg8lgl1iiaIKgqYeDoDGGEc24I4usEUriGWbMnUKWOo7G//2Oc2m8vq3sPu1V0JvV8zO3vO9zzn7POI4X72OefsbqoKSZK6+LFxd0CSdOwwNCRJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjpbPFeDJFuAdwL7quq8vvpvAtcBh4C7q+oDrX49sB74IfD+qtrZ6quBjwOLgE9V1aZWPwfYBpwKfBn4jap6McmJwO3A+cB3gXdV1ZNz9ff000+vFStWdBq8JKnnkUce+U5VLZmr3ZyhAdwG/CG9P+AAJPlVYA3wxqp6IckZrX4usBZ4A/BTwJ8l+Zm22yeAtwNTwMNJtlfVY8DHgJuqaluS/0QvcG5pzwer6vVJ1rZ275qrsytWrGD37t0dhiVJmpbkf3VpN+fpqar6InBgRvm9wKaqeqG12dfqa4BtVfVCVX0TmAQuaI/Jqnqiql6kN7NYkyTAW4E72/5bgSv6jrW1Ld8JrGrtJUljMuw1jZ8BfinJQ0n+PMnPt/pS4Om+dlOtdrj6acD3qurQjPrfO1bb/lxrL0kaky6npw633ynARcDPA3ck+Wlg0EygGBxONUt75tj29yTZAGwAOPvss2ftuCRpeMPONKaAz1fPl4AfAae3+vK+dsuAZ2apfwc4OcniGXX692nbX8tLT5MBUFWbq2qiqiaWLJnzOo4kaUjDhsZ/p3ctgnah+wR6AbAdWJvkxHZX1ErgS8DDwMok5yQ5gd7F8u3V+zGPLwBXtuOuA+5qy9vbOm37feWPf0jSWHW55fazwFuA05NMATcAW4AtSb4OvAisa3/Q9yS5A3iM3q2411bVD9txrgN20rvldktV7Wkv8UFgW5KPAo8Ct7b6rcCnk0zSm2GsnYfxSpJGkFfam/eJiYnylltJenmSPFJVE3O18xPhkqTODA1JUmfD3nIrHZdWbLx76H2f3HTZPPZEGg9nGpKkzgwNSVJnnp6SFsgop7bA01s6OjjTkCR1ZmhIkjozNCRJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjozNCRJnRkakqTO5gyNJFuS7Gu/Bz5z2+8kqSSnt/UkuTnJZJKvJnlzX9t1Sfa2x7q++vlJvtb2uTlJWv3UJLta+11JTpmfIUuShtVlpnEbsHpmMcly4O3AU33lS4GV7bEBuKW1PRW4AbgQuAC4oS8Ebmltp/ebfq2NwL1VtRK4t61LksZoztCoqi8CBwZsugn4AFB9tTXA7dXzIHBykrOAS4BdVXWgqg4Cu4DVbdtJVfVAVRVwO3BF37G2tuWtfXVJ0pgMdU0jyeXAt6rqL2ZsWgo83bc+1Wqz1acG1AHOrKpnAdrzGcP0VZI0f172jzAleTXwIeDiQZsH1GqI+svt0wZ6p7g4++yzX+7ukqSOhplp/CPgHOAvkjwJLAO+nOR19GYKy/vaLgOemaO+bEAd4Nvt9BXted/hOlRVm6tqoqomlixZMsSQJEldvOzQqKqvVdUZVbWiqlbQ+8P/5qr6G2A7cE27i+oi4Ll2amkncHGSU9oF8IuBnW3b95Nc1O6auga4q73UdmD6Lqt1fXVJ0ph0ueX2s8ADwM8mmUqyfpbmO4AngEngvwDvA6iqA8BHgIfb48OtBvBe4FNtn78G7mn1TcDbk+yld5fWppc3NEnSfJvzmkZVXT3H9hV9ywVce5h2W4AtA+q7gfMG1L8LrJqrf5KkheMnwiVJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjozNCRJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjozNCRJnRkakqTOuvxG+JYk+5J8va/2+0n+MslXk/xxkpP7tl2fZDLJN5Jc0ldf3WqTSTb21c9J8lCSvUk+l+SEVj+xrU+27Svma9CSpOF0mWncBqyeUdsFnFdVbwT+CrgeIMm5wFrgDW2fTyZZlGQR8AngUuBc4OrWFuBjwE1VtRI4CKxv9fXAwap6PXBTaydJGqM5Q6OqvggcmFH706o61FYfBJa15TXAtqp6oaq+CUwCF7THZFU9UVUvAtuANUkCvBW4s+2/Fbii71hb2/KdwKrWXpI0JvNxTeNfAPe05aXA033bplrtcPXTgO/1BdB0/e8dq21/rrWXJI3JSKGR5EPAIeAz06UBzWqI+mzHGtSPDUl2J9m9f//+2TstSRra4mF3TLIOeCewqqqm/5hPAcv7mi0DnmnLg+rfAU5OsrjNJvrbTx9rKsli4LXMOE02rao2A5sBJiYmBgaLNG3FxrvH3QXpmDXUTCPJauCDwOVV9Xzfpu3A2nbn0znASuBLwMPAynan1An0LpZvb2HzBeDKtv864K6+Y61ry1cC9/WFkyRpDOacaST5LPAW4PQkU8AN9O6WOhHY1a5NP1hV/7Kq9iS5A3iM3mmra6vqh+041wE7gUXAlqra017ig8C2JB8FHgVubfVbgU8nmaQ3w1g7D+OVJI1gztCoqqsHlG8dUJtufyNw44D6DmDHgPoT9O6umln/AXDVXP2TJC0cPxEuSeps6AvhkhbWKBfwn9x02Tz2RMczZxqSpM4MDUlSZ4aGJKkzQ0OS1JmhIUnqzNCQJHVmaEiSOjM0JEmdGRqSpM4MDUlSZ4aGJKkzQ0OS1JmhIUnqzNCQJHVmaEiSOjM0JEmdzRkaSbYk2Zfk6321U5PsSrK3PZ/S6klyc5LJJF9N8ua+fda19nuTrOurn5/ka22fm9N+dPxwryFJGp8uM43bgNUzahuBe6tqJXBvWwe4FFjZHhuAW6AXAMANwIX0fg/8hr4QuKW1nd5v9RyvIUkakzlDo6q+CByYUV4DbG3LW4Er+uq3V8+DwMlJzgIuAXZV1YGqOgjsAla3bSdV1QNVVcDtM4416DUkSWMy7DWNM6vqWYD2fEarLwWe7ms31Wqz1acG1Gd7jZdIsiHJ7iS79+/fP+SQJElzme8L4RlQqyHqL0tVba6qiaqaWLJkycvdXZLU0bCh8e12aon2vK/Vp4Dlfe2WAc/MUV82oD7ba0iSxmTY0NgOTN8BtQ64q69+TbuL6iLguXZqaSdwcZJT2gXwi4Gdbdv3k1zU7pq6ZsaxBr2GJGlMFs/VIMlngbcApyeZoncX1CbgjiTrgaeAq1rzHcA7gEngeeA9AFV1IMlHgIdbuw9X1fTF9ffSu0PrVcA97cEsryFJGpM5Q6Oqrj7MplUD2hZw7WGOswXYMqC+GzhvQP27g15DkjQ+fiJcktSZoSFJ6szQkCR1ZmhIkjozNCRJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1Nud3T0lHoxUb7x53F6TjkjMNSVJnhoYkqTNDQ5LUmdc0pOPAKNeAntx02Tz2RMc6ZxqSpM5GCo0k/zrJniRfT/LZJD+e5JwkDyXZm+RzSU5obU9s65Nt+4q+41zf6t9IcklffXWrTSbZOEpfJUmjGzo0kiwF3g9MVNV5wCJgLfAx4KaqWgkcBNa3XdYDB6vq9cBNrR1Jzm37vQFYDXwyyaIki4BPAJcC5wJXt7aSpDEZ9fTUYuBVSRYDrwaeBd4K3Nm2bwWuaMtr2jpt+6okafVtVfVCVX0TmAQuaI/Jqnqiql4EtrW2kqQxGTo0qupbwL8HnqIXFs8BjwDfq6pDrdkUsLQtLwWebvseau1P66/P2OdwdUnSmIxyeuoUeu/8zwF+CvgJeqeSZqrpXQ6z7eXWB/VlQ5LdSXbv379/rq5LkoY0yumptwHfrKr9VfV/gc8D/ww4uZ2uAlgGPNOWp4DlAG37a4ED/fUZ+xyu/hJVtbmqJqpqYsmSJSMMSZI0m1FC4yngoiSvbtcmVgGPAV8Armxt1gF3teXtbZ22/b6qqlZf2+6uOgdYCXwJeBhY2e7GOoHexfLtI/RXkjSioT/cV1UPJbkT+DJwCHgU2AzcDWxL8tFWu7Xtcivw6SST9GYYa9tx9iS5g17gHAKuraofAiS5DthJ786sLVW1Z9j+SpJGN9InwqvqBuCGGeUn6N35NLPtD4CrDnOcG4EbB9R3ADtG6aMkaf74iXBJUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjozNCRJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjozNCRJnRkakqTODA1JUmeGhiSps5FCI8nJSe5M8pdJHk/yC0lOTbIryd72fEprmyQ3J5lM8tUkb+47zrrWfm+SdX3185N8re1zc5KM0l9J0mhGnWl8HPgfVfWPgX8KPA5sBO6tqpXAvW0d4FJgZXtsAG4BSHIqvd8Zv5Deb4vfMB00rc2Gvv1Wj9hfSdIIFg+7Y5KTgF8G3g1QVS8CLyZZA7ylNdsK3A98EFgD3F5VBTzYZilntba7qupAO+4uYHWS+4GTquqBVr8duAK4Z9g+6+ixYuPd4+6CpCGMMtP4aWA/8F+TPJrkU0l+Ajizqp4FaM9ntPZLgaf79p9qtdnqUwPqkqQxGSU0FgNvBm6pqjcB/4e/OxU1yKDrETVE/aUHTjYk2Z1k9/79+2fvtSRpaEOfnqL3zn+qqh5q63fSC41vJzmrqp5tp5/29bVf3rf/MuCZVn/LjPr9rb5sQPuXqKrNwGaAiYmJgcEiaTijnEp8ctNl89gTHQ2GnmlU1d8ATyf52VZaBTwGbAem74BaB9zVlrcD17S7qC4Cnmunr3YCFyc5pV0AvxjY2bZ9P8lF7a6pa/qOJUkag1FmGgC/CXwmyQnAE8B76AXRHUnWA08BV7W2O4B3AJPA860tVXUgyUeAh1u7D09fFAfeC9wGvIreBXAvgkvSGI0UGlX1FWBiwKZVA9oWcO1hjrMF2DKgvhs4b5Q+SpLmj58IlyR1ZmhIkjozNCRJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjozNCRJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjobOTSSLEryaJI/aevnJHkoyd4kn2u/H06SE9v6ZNu+ou8Y17f6N5Jc0ldf3WqTSTaO2ldJ0mjmY6bxW8DjfesfA26qqpXAQWB9q68HDlbV64GbWjuSnAusBd4ArAY+2YJoEfAJ4FLgXODq1laSNCYjhUaSZcBlwKfaeoC3Ane2JluBK9rymrZO276qtV8DbKuqF6rqm8AkcEF7TFbVE1X1IrCttZUkjcniEff/A+ADwGva+mnA96rqUFufApa25aXA0wBVdSjJc639UuDBvmP27/P0jPqFgzqRZAOwAeDss88eYTh6OVZsvHvcXZC0wIaeaSR5J7Cvqh7pLw9oWnNse7n1lxarNlfVRFVNLFmyZJZeS5JGMcpM4xeBy5O8A/hx4CR6M4+Tkyxus41lwDOt/RSwHJhKshh4LXCgrz6tf5/D1SUdA0aZjT656bJ57Inmy9Azjaq6vqqWVdUKehey76uqXwe+AFzZmq0D7mrL29s6bft9VVWtvrbdXXUOsBL4EvAwsLLdjXVCe43tw/ZXkjS6Ua9pDPJBYFuSjwKPAre2+q3Ap5NM0pthrAWoqj1J7gAeAw4B11bVDwGSXAfsBBYBW6pqzxHorySpo3kJjaq6H7i/LT9B786nmW1+AFx1mP1vBG4cUN8B7JiPPkqSRucnwiVJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjozNCRJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1diR+hEnHkFF+jlPS8ceZhiSps6FnGkmWA7cDrwN+BGyuqo8nORX4HLACeBL4tao6mCTAx4F3AM8D766qL7djrQN+tx36o1W1tdXPB24DXkXvF/x+q/2uuKRXuFFnwU9uumyeeqJ+o8w0DgH/pqr+CXARcG2Sc4GNwL1VtRK4t60DXAqsbI8NwC0ALWRuAC6k9zOxNyQ5pe1zS2s7vd/qEforSRrR0KFRVc9OzxSq6vvA48BSYA2wtTXbClzRltcAt1fPg8DJSc4CLgF2VdWBqjoI7AJWt20nVdUDbXZxe9+xJEljMC/XNJKsAN4EPAScWVXPQi9YgDNas6XA0327TbXabPWpAXVJ0piMHBpJfhL4b8C/qqr/PVvTAbUaoj6oDxuS7E6ye//+/XN1WZI0pJFCI8k/oBcYn6mqz7fyt9upJdrzvlafApb37b4MeGaO+rIB9Zeoqs1VNVFVE0uWLBllSJKkWQwdGu1uqFuBx6vqP/Zt2g6sa8vrgLv66tek5yLguXb6aidwcZJT2gXwi4Gdbdv3k1zUXuuavmNJksZglA/3/SLwG8DXknyl1f4dsAm4I8l64CngqrZtB73bbSfp3XL7HoCqOpDkI8DDrd2Hq+pAW34vf3fL7T3tIUkak6FDo6r+J4OvOwCsGtC+gGsPc6wtwJYB9d3AecP2UZI0v/xEuCSpM0NDktSZoSFJ6szQkCR15lejvwL49eaSFoozDUlSZ840JL0ijTID92vVD8+ZhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjozNCRJnRkakqTO/JzGUcJPdUs6FhgakjSDHww8PE9PSZI6MzQkSZ0d9aGRZHWSbySZTLJx3P2RpOPZUX1NI8ki4BPA24Ep4OEk26vqsfH27KW8kC3peHBUhwZwATBZVU8AJNkGrAGOutCQJHjlX0Q/2kNjKfB03/oUcOGRejFnC5I0u6M9NDKgVi9plGwANrTVv03yjSPaq9GdDnxn3J0YE8d+fHLsHeRjR7gns/uHXRod7aExBSzvW18GPDOzUVVtBjYvVKdGlWR3VU2Mux/j4Ngd+/HmlTb2o/3uqYeBlUnOSXICsBbYPuY+SdJx66ieaVTVoSTXATuBRcCWqtoz5m5J0nHrqA4NgKraAewYdz/m2TFzKu0IcOzHJ8f+CpGql1xXliRpoKP9moYk6ShiaCyAJKcm2ZVkb3s+ZZa2JyX5VpI/XMg+Hildxp7k55I8kGRPkq8medc4+jof5vramyQnJvlc2/5QkhUL38sjo8PYfzvJY+2/8b1JOt3ieSzo+nVHSa5MUkmO2bupDI2FsRG4t6pWAve29cP5CPDnC9KrhdFl7M8D11TVG4DVwB8kOXkB+zgv+r725lLgXODqJOfOaLYeOFhVrwduAsZ7Z/486Tj2R4GJqnojcCfwewvbyyOj49hJ8hrg/cBDC9vD+WVoLIw1wNa2vBW4YlCjJOcDZwJ/ukD9Wghzjr2q/qqq9rblZ4B9wJIF6+H8+f9fe1NVLwLTX3vTr//f405gVZJBH2I91sw59qr6QlU931YfpPe5q1eCLv/dofeG8PeAHyxk5+abobEwzqyqZwHa8xkzGyT5MeA/AP92gft2pM059n5JLgBOAP56Afo23wZ97c3Sw7WpqkPAc8BpC9K7I6vL2PutB+45oj1aOHOOPcmbgOVV9ScL2bEj4ai/5fZYkeTPgNcN2PShjod4H7Cjqp4+1t54zsPYp49zFvBpYF1V/Wg++rbAunztTaevxjkGdR5Xkn8OTAC/ckR7tHBmHXt7Q3gT8O6F6tCRZGjMk6p62+G2Jfl2krOq6tn2h3HfgGa/APxSkvcBPwmckORvq+qo/w2ReRg7SU4C7gZ+t6oePEJdPdK6fO3NdJupJIuB1wIHFqZ7R1Snr/xJ8jZ6byZ+papeWKC+HWlzjf01wHnA/e0N4euA7Ukur6rdC9bLeeLpqYWxHVjXltcBd81sUFW/XlVnV9UK4HeA24+FwOhgzrG3r4j5Y3pj/qMF7Nt86/K1N/3/HlcC99Ur48NSc469naL5z8DlVTXwzcMxataxV9VzVXV6Va1o/38/SO/f4JgLDDA0Fsom4O1J9tL7QalNAEkmknxqrD078rqM/deAXwbeneQr7fFz4+nu8No1iumvvXkcuKOq9iT5cJLLW7NbgdOSTAK/zex30h0zOo799+nNov+o/Td+RXyPXMexv2L4iXBJUmfONCRJnRkakqTODA1JUmeGhiSpM0NDktSZoSFJ6szQkCR1ZmhIkjr7f3a2p6rLEZN0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(svm_seps, bins = 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.5 (d)\n",
    "\n",
    "* The histogram of the random separators is really flat. That's because they are chosen uniformly between $[-1, 1]$\n",
    "\n",
    "* The histogram of the SVM separators concentrates around 0, which is the target function. That's because we pick the maximum margin between the positive and negative points. \n",
    "\n",
    "#### Problem 8.5 (e)\n",
    "\n",
    "* The bias of random and SVM separators are close to each other. On average, their separators are close to the target separator. \n",
    "* However the variance of the SVM separator is much smaller than the ones from random separator. It indicates that the SVM is much better at generalization "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.6\n",
    "\n",
    "\\begin{align*}\n",
    "L &= \\sum^N_{n=1}|x_n-\\mu|^2 \\\\\n",
    "&= \\sum^N_{n=1}(x_n-\\mu)^T(x_n-\\mu) \\\\\n",
    "&= \\sum^N_{n=1}x_n^Tx_n-x_n^T\\mu + \\mu^Tx_n+\\mu^T\\mu \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Take derivative w.r.t. $\\mu$ and let it equal to 0, we have\n",
    "\n",
    "\\begin{align*}\n",
    "\\frac{\\partial{L}}{\\partial{\\mu}} &= \\sum^N_{n=1} 2x_n - 2\\mu \\\\\n",
    "&= 2 \\sum^N_{n=1} x_n - 2N \\\\\n",
    "&= 0 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "So we have $\\mu =  \\frac{1}{N}\\sum^N_{n=1} x_n $\n",
    "\n",
    "#### Problem 8.7\n",
    "\n",
    "Let $X=\\begin{bmatrix}x_1 & \\dots & x_N \\end{bmatrix}$, where $x_i$, $i=1,\\dots, N$ are the column vectors. Let $y=\\begin{bmatrix}y_1 \\\\ \\dots \\\\ y_N\\end{bmatrix}$.\n",
    "\n",
    "* (a) \n",
    "\n",
    "\\begin{align*}\n",
    "|\\sum^N_{n=1}y_nx_n|^2 &= |Xy|^2 \\\\\n",
    "&= (Xy)^T(Xy) \\\\\n",
    "&= y^TX^TXy \\\\\n",
    "&= y^TMy \\\\\n",
    "&= y^T\\begin{bmatrix}\\sum^N_{m=1}M_{1m}y_m \\\\ \\dots \\\\ \\sum^N_{m=1}M_{Nm}y_m\\end{bmatrix} \\\\\n",
    "&= \\sum^N_{n=1} y_n\\sum^N_{m=1}M_{nm}y_m \\\\\n",
    "&= \\sum^N_{n=1} \\sum^N_{m=1}y_ny_mM_{nm} \\\\\n",
    "&= \\sum^N_{n=1} \\sum^N_{m=1}y_ny_mx^T_nx_m \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Where $M=X^TX$ and $M_{nm} = x^T_nx_m$.\n",
    "\n",
    "* (b) When $n=m$, we have $y_ny_m = y_n^2 = 1$.\n",
    "When $n\\ne m$, we have $P(y_n=1) = \\frac{1}{2}$ because there are $\\frac{N}{2}$ positive points and $\\frac{N}{2}$ negative points. We also have $P(y_n = 1|y_m=1) = \\frac{\\frac{N}{2}-1}{N-1}$ because if we have $y_m=1$, then we need choose another positive point (there are $\\frac{N}{2}-1$ of them) from the remaining $N-1$ points.\n",
    "\n",
    "\\begin{align*}\n",
    "P(y_ny_m = 1) &= P(y_ny_m = 1|y_m=1)P(y_m=1) + P(y_ny_m = 1|y_m=-1)P(y_m=-1)\\\\\n",
    "&= P(y_n = 1|y_m=1)P(y_m=1) + P(y_n = -1|y_m=-1)P(y_m=-1)\\\\\n",
    "&= P(y_n = 1|y_m=1)\\frac{1}{2} + P(y_n = -1|y_m=-1)\\frac{1}{2}\\\\\n",
    "&= \\frac{\\frac{N}{2}-1}{N-1}\\frac{1}{2} + \\frac{\\frac{N}{2}-1}{N-1}\\frac{1}{2}\\\\\n",
    "&= \\frac{\\frac{N}{2}-1}{N-1}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "We thus have $E[y_ny_m] = 1$ when $m=n$ because the when $n=m$, the probability of $P[y_ny_m=1] = 1$. \n",
    "\n",
    "When $n\\ne m$, \n",
    "\n",
    "\\begin{align*}\n",
    "E[y_ny_m] &= 1P(y_ny_m =1 \\cap n\\ne m) - 1P(y_ny_m =-1 \\cap n\\ne m)\\\\\n",
    "&= \\frac{\\frac{N}{2}-1}{N-1} - \\frac{\\frac{N}{2}}{N-1}\\\\\n",
    "&= -\\frac{1}{N-1}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "* (c) \n",
    "\n",
    "\\begin{align*}\n",
    "E\\left[\\|\\sum^N_{n=1}y_nx_n\\|^2\\right] &= E\\left[\\sum^N_{n=1} \\sum^N_{m=1}y_ny_mx^T_nx_m\\right]\\\\\n",
    "&= \\sum^N_{n=1} \\sum^N_{m=1}E[y_ny_m]x^T_nx_m\\\\\n",
    "&= \\sum^N_{n=1}\\left[ \\sum^N_{m\\ne n}E[y_ny_m]x^T_nx_m + E[y_ny_n]x^T_nx_n\\right]\\\\\n",
    "&= \\sum^N_{n=1}\\left[x^T_nx_n - \\sum^N_{m\\ne n}\\frac{1}{N-1}x^T_nx_m\\right]\\\\\n",
    "&= \\sum^N_{n=1}\\left[x^T_nx_n -\\frac{1}{N-1} \\sum^N_{m\\ne n}x^T_nx_m\\right]\\\\\n",
    "&= \\frac{N}{N-1}\\sum^N_{n=1}\\left[\\frac{N-1}{N}x^T_nx_n - \\frac{1}{N}\\sum^N_{m\\ne n}x^T_nx_m\\right]\\\\\n",
    "&= \\frac{N}{N-1}\\sum^N_{n=1}\\left[x^T_nx_n - \\frac{1}{N}\\sum^N_{m=1}x^T_nx_m\\right]\\\\\n",
    "&= \\frac{N}{N-1}\\sum^N_{n=1}\\left[x^T_nx_n - x^T_n\\frac{1}{N}\\sum^N_{m=1}x_m\\right]\\\\\n",
    "&= \\frac{N}{N-1}\\sum^N_{n=1}\\left[x^T_nx_n - x^T_n\\bar{x}\\right]\\\\\n",
    "&= \\frac{N}{N-1}\\sum^N_{n=1}\\left[x^T_nx_n - x^T_n\\bar{x}-(\\bar{x}^Tx_n - \\bar{x}^T\\bar{x})\\right]\\\\\n",
    "&= \\frac{N}{N-1}\\sum^N_{n=1}(x_n - \\bar{x})^T(x_n - \\bar{x})\\\\\n",
    "&= \\frac{N}{N-1}\\sum^N_{n=1}\\|x_n - \\bar{x}\\|^2\\\\\n",
    "\\end{align*}\n",
    "\n",
    "Where in the third to the last equation we  have used the fact that $\\sum^N_{n=1}\\bar{x}^Tx_n = \\sum^N_{n=1}\\bar{x}^T\\bar{x}$\n",
    "\n",
    "* (d) From problem 8.6, we see that $\\sum^N_{n=1}\\|x_n - \\mu\\|^2 $ achieves minimal value when $\\mu = \\frac{1}{N}\\sum^N_{n=1}x_n$, since $\\bar{x}=\\frac{1}{N}\\sum^N_{n=1}x_n$, let $\\mu=0$, we have \n",
    "\n",
    "$\\sum^N_{n=1}\\|x_n - \\bar{x}\\|^2 \\le \\sum^N_{n=1}\\|x_n\\|^2 \\le NR^2$ by the assumption.\n",
    "\n",
    "* (e) Take the result of problem (d) into the equation of (c), we have\n",
    "\n",
    "\\begin{align*}\n",
    "E\\left[\\|\\sum^N_{n=1}y_nx_n\\|^2\\right] &\\le \\frac{N}{N-1}\\sum^N_{n=1} \\|x_n\\|^2 \\\\\n",
    "&\\le \\frac{N}{N-1}NR^2 \\\\ \n",
    "&= \\frac{N^2R^2}{N-1}\\\\ \n",
    "\\end{align*}\n",
    "\n",
    "Let $Z=\\|\\sum^N_{n=1}y_nx_n\\|^2$, and $t=\\frac{N^2R^2}{N-1}$, we claim that $P[\\sqrt{Z} \\le \\sqrt{t}]  = P[\\|\\sum^N_{n=1}y_nx_n\\| \\le \\frac{N^2R^2}{\\sqrt{N-1}}] > 0$, otherwise, if $P[\\sqrt{Z} \\le \\sqrt{t}]  = 0$, then   $P[\\sqrt{Z} > \\sqrt{t}]  = 1$, we have (since $Z\\le 0$, and $t>0$)\n",
    "\n",
    "\\begin{align*}\n",
    "E[Z] &= \\int_0^{\\infty}ZP(dZ)\\\\\n",
    "&= \\int_t^{\\infty}ZP(dZ)\\\\\n",
    "&> t\\int_t^{\\infty}P(dZ)\\\\\n",
    "&= t\\\\\n",
    "\\end{align*}\n",
    "\n",
    "This contradicts with our conclusion that $E[Z] \\le t$\n",
    "\n",
    "So $P[\\|\\sum^N_{n=1}y_nx_n\\| \\le \\frac{NR}{\\sqrt{N-1}}] > 0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.8\n",
    "\n",
    "* (a) Summing (8.31) on the $k$ positive and $k+1$ negative points respectively, we have\n",
    "\n",
    "\\begin{align*}\n",
    "k\\rho|w| &\\le \\sum_{y_n=1} y_n(w^Tx_n+b) \\\\\n",
    "\\rho|w| &\\le \\sum_{y_n=1} l_ny_n(w^Tx_n+b) \\\\\n",
    "\\end{align*}\n",
    "\n",
    "where $l_n = \\frac{1}{k}$ when $y_n = 1$\n",
    "and\n",
    "\n",
    "\\begin{align*}\n",
    "(k+1)\\rho|w| &\\le \\sum_{y_n=-1} y_n(w^Tx_n+b) \\\\\n",
    "\\rho|w| &\\le \\sum_{y_n=-1} l_ny_n(w^Tx_n+b) \\\\\n",
    "\\end{align*}\n",
    "\n",
    "where $l_n = \\frac{1}{k+1}$ when $y_n = -1$. Note, there are $k$ of $l_n=\\frac{1}{k}$ and $k+1$ of $l_n=\\frac{1}{k+1}$.\n",
    "\n",
    "Add the two inequalities, we have\n",
    "\n",
    "\\begin{align*}\n",
    "2\\rho|w| &\\le \\sum^N_{n=1}l_ny_n(w^Tx_n+b)\\\\\n",
    "&= \\sum^N_{n=1}l_ny_nw^Tx_n+b\\sum^N_{n=1}l_ny_n\\\\\n",
    "&= w^T\\sum^N_{n=1}l_ny_nx_n+b(\\sum_{y_n=1}\\frac{1}{k}-\\sum_{y_n=-1}\\frac{1}{k+1})\\\\\n",
    "&= w^T\\sum^N_{n=1}l_ny_nx_n\\\\\n",
    "&\\le |w||\\sum^N_{n=1}l_ny_nx_n| \\\\\n",
    "\\end{align*}\n",
    "\n",
    "So we obtain $2\\rho \\le |\\sum^N_{n=1}l_ny_nx_n|$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* (b) \n",
    "  * (i) This is similar to problem 8.7 (a), just replace the $y_n$ with $l_ny_n$.\n",
    "  * (ii) For $m=n$, we have $y_ny_m = 1$, $E[l_nl_my_ny_m] = E[l_nl_m] = \\frac{1}{k(k+1)}$\n",
    "  * (iii) For $m\\ne n$, we have \n",
    "  \n",
    "  $P[l_nl_my_ny_m = \\frac{1}{k^2}] =P[l_nl_my_ny_m = \\frac{1}{k^2}|y_n=1]P(y_n=1) + P[l_nl_my_ny_m = \\frac{1}{k^2}|y_n=-1]P(y_n=-1) = \\frac{k-1}{N-1}\\frac{k}{N} + 0 = \\frac{k(k-1)}{N(N-1)}$\n",
    "  \n",
    "  $P[l_nl_my_ny_m = \\frac{1}{(k+1)^2}] =P[l_nl_my_ny_m = \\frac{1}{(k+1)^2}|y_n=1]P(y_n=1) + P[l_nl_my_ny_m = \\frac{1}{(k+1)^2}|y_n=-1]P(y_n=-1) = 0 + \\frac{k}{N-1}\\frac{k+1}{N} = \\frac{k(k+1)}{N(N-1)}$\n",
    "  \n",
    "  $P[l_nl_my_ny_m = -\\frac{1}{k(k+1)}] =P[l_nl_my_ny_m = -\\frac{1}{k(k+1)}|y_n=1]P(y_n=1) + P[l_nl_my_ny_m = -\\frac{1}{k(k+1)}|y_n=-1]P(y_n=-1) = \\frac{k+1}{N-1}\\frac{k}{N} + \\frac{k}{N-1}\\frac{k+1}{N} = \\frac{2k(k+1)}{N(N-1)}$\n",
    "  \n",
    "  So the expectation\n",
    "  \n",
    "  $E[l_nl_my_ny_m] = \\frac{1}{k^2}P[l_nl_my_ny_m = \\frac{1}{k^2}] + \\frac{1}{(k+1)^2}P[l_nl_my_ny_m = \\frac{1}{(k+1)^2}] - \\frac{1}{k(k+1)}P[l_nl_my_ny_m = -\\frac{1}{k(k+1)}] = \\frac{1}{k^2}\\frac{k(k-1)}{N(N-1)} + \\frac{1}{(k+1)^2}\\frac{k(k+1)}{N(N-1)} -\\frac{1}{k(k+1)}\\frac{2k(k+1)}{N(N-1)} = -\\frac{1}{(N-1)k(k+1)}$\n",
    "  \n",
    "  * (iv) \n",
    "  \n",
    "\\begin{align*}\n",
    "E\\left[\\|\\sum^N_{n=1}l_ny_nx_n\\|^2\\right] &= E\\left[\\sum^N_{n=1}\\sum^N_{m=1}l_nl_my_ny_mx^T_nx_m\\right] \\\\\n",
    "&= \\sum^N_{n=1}\\sum^N_{m=1}E[l_nl_my_ny_m]x^T_nx_m \\\\\n",
    "&= \\sum^N_{n=1}\\left[E[l_nl_ny_ny_n]x^T_nx_n + \\sum^N_{m\\ne n}E[l_nl_my_ny_m]x^T_nx_m\\right] \\\\\n",
    "&= \\sum^N_{n=1}\\left[\\frac{1}{k(k+1)}x^T_nx_n - \\sum^N_{m\\ne n}\\frac{1}{(N-1)k(k+1)}x^T_nx_m\\right] \\\\\n",
    "&= \\frac{N}{(N-1)k(k+1)}\\sum^N_{n=1}\\left[\\frac{(N-1)}{N}x^T_nx_n - \\frac{1}{N}\\sum^N_{m\\ne n}x^T_nx_m\\right] \\\\\n",
    "&= \\frac{N}{(N-1)k(k+1)}\\sum^N_{n=1}\\left[x^T_nx_n - \\frac{1}{N}\\sum^N_{m=1}x^T_nx_m\\right] \\\\\n",
    "&= \\frac{N}{(N-1)k(k+1)}\\sum^N_{n=1}\\left[x^T_nx_n - x^T_n\\frac{1}{N}\\sum^N_{m=1}x_m\\right] \\\\\n",
    "&= \\frac{N}{(N-1)k(k+1)}\\sum^N_{n=1}\\left[x^T_nx_n - x^T_n\\bar{x}\\right] \\\\\n",
    "&= \\frac{N}{(N-1)k(k+1)}\\left[\\sum^N_{n=1}[x^T_nx_n - x^T_n\\bar{x}] - \\sum^N_{n=1}(\\bar{x}^Tx_n - \\bar{x}^T\\bar{x})\\right] \\\\\n",
    "&= \\frac{N}{(N-1)k(k+1)}\\left[\\sum^N_{n=1}[x^T_nx_n - x^T_n\\bar{x}- \\bar{x}^Tx_n + \\bar{x}^T\\bar{x}\\right] \\\\\n",
    "&= \\frac{N}{(N-1)k(k+1)}\\sum^N_{n=1}[(x_n - \\bar{x})^T(x_n - \\bar{x}) \\\\\n",
    "&= \\frac{N}{(N-1)k(k+1)}\\sum^N_{n=1}[\\|x_n - \\bar{x}\\|^2 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "  * (v) $E\\left[\\|\\sum^N_{n=1}l_ny_nx_n\\|^2\\right] = \\frac{N}{(N-1)k(k+1)}\\sum^N_{n=1}[\\|x_n - \\bar{x}\\|^2 \\le \\frac{N}{(N-1)k(k+1)}\\sum^N_{n=1}[\\|x_n\\|^2 \\le  \\frac{N^2R^2}{(N-1)k(k+1)}$\n",
    "  Similarly to problem 8.7 (d)-(e) , we can prove that $P[\\|\\sum^N_{n=1}l_ny_nx_n\\| \\le \\sqrt{\\frac{N^2R^2}{(N-1)k(k+1)}}] > 0$, so we say $\\|\\sum^N_{n=1}l_ny_nx_n\\| \\le \\sqrt{\\frac{N^2R^2}{(N-1)k(k+1)}} = \\frac{2NR}{(N-1)\\sqrt{N+1}}$\n",
    "  \n",
    "* (c) Combine (a) and (b) we have $2\\rho \\le \\|\\sum^N_{n=1}l_ny_nx_n\\| \\le \\frac{2NR}{(N-1)\\sqrt{N+1}}$\n",
    "So $\\frac{R}{\\rho} \\ge \\frac{(N-1)\\sqrt{N+1}}{N}$, square both sides, we have\n",
    "\n",
    "$\\frac{R^2}{\\rho^2} \\ge N -1 - \\frac{1}{N} + \\frac{1}{N^2}$, i.e. $N \\le \\frac{R^2}{\\rho^2} + \\frac{1}{N} + 1$ by ignoring terms in $O(\\frac{1}{N^2})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.9\n",
    "\n",
    "* (a) Since $g$ is a separator for $\\mathcal{D}$, if we remove a point from $\\mathcal{D}$ to get $\\mathcal{D}^{-1}$, $g$ still separates the rest of the data. \n",
    "\n",
    "* (b) If there exists another maximum margin classifier $g_1$, the we have $|w_1| = |w|$ since $g_1$ and $g$ have the same maximum margin. We construct another hyperplane, $w_m = \\frac{1}{2}(w+w_1)$. Then we have (by triangle inequality)\n",
    "\n",
    "$|w_m| = |\\frac{1}{2}(w+w_1)| = \\frac{1}{2}|w+w_1| \\le \\frac{1}{2}(|w|+|w_1|) = |w|$,\n",
    "\n",
    "If $w$ is not a nonnegative multiple of $w_1$, then we have $|w_m| < |w|$. this indicates that the new hyperplane has a larger margin ($\\frac{1}{|w_m|}$) than the one of $w$, i.e. $\\frac{1}{|w|}$,  which is supposed to have the largest margin. This contradicts. \n",
    "\n",
    "So we must have $w=w_1$. That is, the maximum margin classifier is unique.\n",
    "\n",
    "* (c) TODO\n",
    "\n",
    "If $g^{-}$ has a larger margin than $g$ on $\\mathcal{D}^{-1}$, \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.10\n",
    "\n",
    "Set $u = \\begin{bmatrix}b \\\\ w\\end{bmatrix}$. From problem 8.9, we see that there's no effect of the SVM classifier, we focus on the support vectors $x_1,\\dots, x_p$. They should all satisfy the equation $y_i(w^Tx_i + b) = 1$ for $i=1,\\dots, p$.\n",
    "\n",
    "Write in matrix format, we have \n",
    "\n",
    "$A_{p\\times (d+1)}u_{(d+1)\\times 1}=1_{p}$, where $A=\\begin{bmatrix}y_1 & y_1x^T_1\\\\ y_2 & y_2x^T_2 \\\\ \\vdots \\\\ y_p & y_px^T_p\\end{bmatrix}$.\n",
    "\n",
    "Consider the augmented matrix $A^* = \\begin{bmatrix} A & 1 \\end{bmatrix}$\n",
    "\n",
    "Since the data is separable, we must have at least one solution to the system of equations $Au=b$. So we have $rank(A) = rank(A')$.\n",
    "\n",
    "The $rank(A)\\le d+1$, so only at most $d+1$ rows are independent, and thus can determine the solution $u$. The other rows are linear combinations of these independent rows. These independent rows are the essential support vectors. \n",
    "\n",
    "So we have $E_{CV} \\le \\frac{d+1}{N}$.\n",
    "\n",
    "#### Problem 8.11\n",
    "\n",
    "* (a) From problem 1.3, if $\\rho' = \\min_{1\\le n \\le N}y_n((w^*)^Tx_n)$, we have the number of iterations when PLA converges is $t \\le \\frac{R^2|w^*|^2}{\\rho'^2}$. In SVM, we have the margin $\\rho = \\frac{\\rho'}{|w^*|}$, so we have $t \\le \\frac{R^2|w^*|^2}{\\rho'^2} = \\frac{R^2}{\\rho^2}$\n",
    "\n",
    "In each iteration, PLA updates if the point is misclassified. This is thus the upper bound of the number of updates $T$ that PLA makes, i.e. $T \\le \\frac{R^2}{\\rho^2}$\n",
    "\n",
    "* (b) Since in each iteration, if the point is correctly classified, PLA skips it, if it's misclassified, PLA updates with the point (hence 'visit'). So PLA 'visits' at most $\\frac{R^2}{\\rho^2}$ different points.\n",
    "\n",
    "* (c) If leaving out any point $(x_n,y_n)$ that is not 'visited', which means PLA properly classifies it during iterations. Such points don't cause any update of PLA, so PLA will return the same classifier.\n",
    "\n",
    "* (d) The leave-one-out error $e_n$ for these points that were not visited is zero, because they are always correctly classified. The leave-one-out error $e_n$ for the points that were visited is 1, since there are at most $\\frac{R^2}{\\rho^2}$ 'visited' points, the error $E_{CV}(PLA) = \\frac{1}{N}\\sum e_n = \\frac{1}{N} \\frac{R^2}{\\rho^2} = \\frac{R^2}{N\\rho^2}$\n",
    "\n",
    "#### Problem 8.12\n",
    "\n",
    "With $C\\to \\infty$, to minimize the objective value in (8.30), we need to have $\\sum^N_{n=1}\\xi_n \\to 0$, otherwise the minimal objective value will be infinity. Since we constrain $\\xi_n \\ge 0$ for all $n=1,\\dots, N$, The minimal value of $\\sum^N_{n=1}\\xi_n$ is 0. So $\\sum^N_{n=1}\\xi_n \\to 0$ is equivalent to minimize $\\sum^N_{n=1}\\xi_n$, which is the objective value of the optimization problem in problem 3.6 (c).\n",
    "\n",
    "The other constraints are all the same. So the two optimization problems are the same when $C\\to \\infty$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.13"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X90XOV95/H313akiT3IEyCuwMaIRGYT8BJB3NhJWTokEMsJgaRJzibEBZ2SdV1Ki+i2XafhEEFo65z+8JAlxIUkZ0hMfv9oCMXyFujU69PKiSGDCyGAYIUQ4JhAxqpMJCH87B/3jjySZ6SRZzT3ztzP65w50v2hex9djb7z3O/z3Ocx5xwiIhItC4IugIiI1J6Cv4hIBCn4i4hEkIK/iEgEKfiLiESQgr+ISAQp+IuIRJCCv9QVM/uEmf2foMtRTWb2+2aWKnPfvzezzfNdJml8Cv5SNjMbMLNfm9lIwevWWpbBOXeXc+691T6umf2Rmf0/Mxs2s31mdv4M+17j7zNmZulp287yt/3Kf91nZmfNcKwm4Hrgb8os6t8An/Z/rtQxm83sr81s0P97PWlmf2pmVuY5Co/1t/7P/6eZ/dzMrpjrMSScFPxlrj7gnIsXvK6p5sHNbFE1j1fmOdcCW4GPAEuBLwM/MLOFJX7keeBm4Csltn0EOBE4Gbgb+OYMp78M+Llz7rlyyuqcewH4OXDpDLt9B3gP8D7gBOB3gd8H/q6cc0xzGPgA3nW5ErjFzN51HMeRkFHwl6owsy+a2XcLlj9nZvfna5tmdomZZc0sZ2b/ZmbnFOw7YGb/y8z2A4fNbJGZnWZm3zezF83spfwdhpl1mdke/3szs21mdtDMDpnZfjNb7W9r9mutg2b2CzPbbmavL1H8NuBR59yDzhvv5Kt4gXtZsZ2dc993zv0j8FKRbTnn3IB/HANeA9pnuHQbgH+ddi2/Y2YH/N9pt5mdPe1nMsD7ix3MzN4DvBf4sHPuEefchHOuD9gIXGtmb5qhLMdwzn3GOfdz59wR59xe4P8C75zLMSScFPylWv4ncI4fnP8bcBVwpXPOmdl5eLXk3wdOAv4BuNvMmgt+/uN4AS0BOOAe4Bm8wLyc4rXn9wIXAGf6P/ffORqQP+ev78ALvsuBG0qUfSew0MzW+rX93wOywIE5XoNJZpYDRoH/DfzVDLv+V+DxIuVZhffh8xBw17TtjwFvK3G8i4G9zrlnC1f6gXsI744AM7vN/yAu9tpf4nd6PfCbwKMz/D5SJ2p+iy117x/NbKJg+c+cc3c4514xs41AL/CfwB8554b8ff4H8A9+AAK408z+AljH0Vrv5/MBy8zeCZzqHzt/rj1FyvIqXlrjLcCPnXOP+T9v/jnPcc697K/7K+DrwKeKHOc/ge/55zAgB2xwFYx66JxLmNkSvFTJMzPsmvDPX/izk+kkM+sBfmVmS51zhwrKmyhxvJOBF0psewF4o3+Oq4GrZ/k1ptsOPAzsmuPPSQgp+MtcfdA5d1+xDc65H5vZ03g11m8XbDoduNLM/qhgXRNegM8rrKmeBjxTEPiLcs494KeDvgCsNLMfAH8KxIDFwIMFbZwGlMrhfxKvtn820I93R3GPmZ3rnHt+pjLMUr7DZrYdeNHM3uqcO1hkt1/hfYB5hfTuPP4S+CheoD7ibzoZyAf/E/A+oIr5Jd5dQzGnAC/O6Zc4Wq6/AVYDF1byoSjhobSPVI2Z/SHQjNfo+ecFm54F/tI5lyh4LXbOfaNgHzdt/5XlNP465z7vnHs7XuA+E/gzvAD4a+DsgvMtdc7FSxzmbcCPnHNP+LntXrxacjUaNhfgfRAtL7F9v1/uvMvxGoEvwmtkbfPXF/bUeSteDbyY+4C1ZnZa4UozewewEtjtL2+f1mur8PXotJ+9Ea9t4r3OueEZf1upGwr+UhVmdiZeD5iNeL1L/tzMOvzNdwCb/Zy6mdkSM3u/mZ1Q4nA/xgu+W/19Y2b2W0XO+Zv+MV+H1ytlFHjNOXfEP+c2M1vm77vczNaXON9PgPeb2Zv88l2MF5AfKfG7LjKzGN6dxEK/fIv8bReb2blmttDMWoC/x6vdP1bi3PcCv12wfAIwhtd2sZji7QW/jdcucAz/rux+4HtmdrZfjnV47QZfdc497u+3eVqvrcLXZAOzmX0K7wPpYufcMQ3cUsecc3rpVdYLGMCrUY8UvH6Alz78MbClYN8/AP4DaPaXO/GCbA4vsH8HOKHguBdNO9dKIN+j5pd4bQIAXcAe//v34NWcR/x97gLi/rYYXuB8GhjGC75/XOL3MuAmYBAvn/4Y8LsF2/8C2Fmw3IN3p1L46vG3fRSvK+YIXorlXry2h1LX9HX+eU/1l+PAD/1yPANc4R+/3d9+Cl7DbdMMx4zhNXg/i9cu4vAanpuP42/u8D6MCv/mfxH0e1Gvyl/m/4FFJCBmtgk4yznXXca+fwc85Zy7bQ7HvxMv7fQ+59z48ZdUGomCv0iD89NifwL8q/P6/Iso+IuIRJEafEVEIii0/fwXLz7ZJRJtQRdDRKSuvPDCg790zr1xtv1CG/wTiTY2bdoXdDFEROrKjTfaTE+UT1LaR0QkghT8RUQiSMFfRCSCQpvzL6ap6VXOO2+IpUtHgy5KRQ4divHQQysYH39d0EURkYiqq+B/3nlDvOlNJ7BkSRs29xnpQsE5x+HDLwFD9PWdEXRxRCSi6irts3TpKEuWnFS3gR/AzFiy5KS6v3sRkfpWV8EfqOvAn9cIv4OI1Le6C/4iIlI5Bf8q+tGPvsOFF57NihULePhhPaAmIuGl4H+c/u3fMnR3d01Z95a3rOaOO77PunUXBFEkEZGy1VVvn7l67TV44AF45BFYvRre/W5YWGoW1ypYteqt83dwEZEqatjg/9prcPnl8NOfwiuvwOLFcO658PWvz+8HgIhIPahK8DezrwCXAAedc6uLbDfgFuB9wCtAl3PuoWqcu5QHHvAC/+HD3vLhw/DQQ976iy8+/uNecslaxsbGeOWVEXK5l7n4Ym+a2k9/+nMkk6WmiBURCZdq1fzTwK3AV0ts3wCs8l9rgS/6X+fNI494Nf5Cv/41PPpoZcH/nnv2Al7O/9vfTpNKpY//YCIiAalKg69zbjfw8gy7XAZ81Xn6gISZnVKNc5eyerWX6in0+tfD2WfP51lFROpDrXr7LAeeLVge8tdNYWabzGyfme175ZUXKzrhu9/t5fgXLwYz7+t553nr58vOnT/g7W9fwYMP/jtXXPF+Lr9caSARCadaNfgWe6T1mMmDnXO3A7cDnHrqmoomF1640GvcfeABL9Vz9tnV7e3zrnclede7klPWbdjwITZs+FB1TiAiMo9qFfyHgNMKllcAz8/3SRcu9PL7leT4RUQaUa3SPncDV5hnHXDIOfdCjc4tIiLTVCX4m9k3gH8H/ouZDZnZVWa22cw2+7vcCzwN9AN3AFdX47wiEg7t/b3g/Eytc96yhFpV0j7OuY/Pst0Bf1iNc4lIuLT397Lxrg30rb2W3vXb6Nx1Hev23sKOT+ykv70z6OJJCQ37hK+I1Eb/m9fTt/Za1u29hXV7bwGgb+219L9Zvd3CTAO7iUhlzOhdv23Kqt7127w+1hJaCv4V6O//OR/4wDs544xmtm//27J+pnn0kHKj0lico3PXdVNWde667uj7XEJJaZ8KJBIn8tnPfp7e3n8sa//m0UOc9PKTjCxZBqDcaIhkMtDZ1xN0MeZke2sPXV1BlwLan9rFur23HJPz72/v1Ps6xBo6+Df/Sy9jyfXe7adzNGd2MXZh9d6MJ5+8jJNPXsb99/9TWfuPNbcwsmQZ8cMHSRz6JW/1/2GUG51fmczM27NZuGckSQdZ4vGaFKliIyNw+WCac1IDdHQcuz2ZrF1Z+ts7vQrMm9dPpoAU+MOvYYN/87/0ctLGDYxcdS3DN26j5TPXEf/yLby0Y2dVPwDmxIzhltOIHz44uUq50fmTTsOOoSSds7zLzxrP0tQEsXUdtY2aFYgD8VSKweEEP+s7NvpP7IE9i5L0ruupya80JdCbKfDXgYYN/mPJ9YxcdS3xL99C/MteD4SRq6717gSC4hwtw89OWdW56zp9AFSgVK2+s6+Hh8dSNDdDrDUxy1EShCJ/Mlfd3bSk06xj4NhtuRznDGc5f0+GjQMZ2tqKH6JOPutkHjRs8MeM4Ru3TQZ+gOEbKw+y6fQXuOuuOwD42tfupbX11LJ/tnlsmPjhg4wsWUbu1SWT3eN0izx36TRsPtDDjol00e0JcsRPr9OgPhcz/H4tmQxr+rLsH0wwPHTsB+CRI3BOdoDu7nksn4RW4wZ/52j5zNQeCC2fua7iD4Curj+kq+v4nlcbiy3lpRNXMdbcArmfKzc6B+n00e9zOdg/3MaJC3LEVyRQtbaEZJJYMkksk6GlyObRviyDwwlu29rN9taeyfVtbbp0UdCwwb85s4v4l285Juc/dmFn1XL+Bw8eYMOGNYyMDLNgwQLuuCNFJvMzTjih2L+aZyy29OiCcqOzyvfC2T2epqnJWxefyBFvAVVZy1QikseSSWLpNNcMpeg6kJ5c3z/UxsaBTMPfNFVTe3/vZIM3ztH+1K7Q/283bPAfu7DTa9z1e/sM37itqoEfYNmyVh58cKhqxxOvhp/LHV2erOGvnFbDV9W0Orq6vMbjgsaTRF+Wh59J8MWt3WyN9Uyu76if9vCaqtfhLRo2+ANTA71ZcL18ZFb5Gn6+kXZSnfXCqVsF1zd/R9B9IEX3aAqA8XHYv6dDdwRF1OvwFg0d/KOk3m478w22eTdPZLx+9ktROicMurqIFSzG/Mbj/B1B3tZYj/5c/rMN+cAP9dGFu+6Cv3MOC/lFnY2r8mPv9XTbWZjDX9aUI1YYYZRXCK9843EqxRZSk6uvHk6x/+YONq6I8B1BieEtwv4BUFfB/9ChGIcPv8SSJSfV7QeAc47Dh1/i0KHY7DuXKcy3ncVq+OccydKiBtv6NO1v1pJO0zGUZfdg2+TTxlH7/K7X4S2s2rXQajn11DVu06Z9U9Y1Nb3KeecNsXTpaEClqo5Dh2I89NAKxsdfV72DOkfPTUfH6eu54UjgtY5UqqDBtnDYhChGiEaXSjE8DPsXRPMuIExp1xtvtAedc2tm26+ugr+U4N92FuYc87WQID4A8sMqqIYfMZkMo31ZDo4n+HpTV82GlpCpyg3+dZX2keLCctuZyXiDpO0fbvPy+a21fcL2tSPGzv5V/PSFVs495QAb2p9k4YJwVm4akt8usDKV4urhVNGhJfRhEB6q+TeIoG87g67tv3bEWL/jd9n73HIOj7+OJU2vsnb5c+za+DV9AATBvwsYH4fhBd7QEkeOwDktGk5ivqnmHzG1HlWxcLiFzQcKBlELqE/+zv5V7H1uOSPj3kMCI+PN7H1uOTv7V3HJmU/UvDyRV2xoiaw3nETkeweFhIK/zEm+hv8nR7IsKJgHLuj++T99oZXD0xrQD4+/juyBVgX/IBVWBJJJWtJp1hzwegd9favaBYKk4C9lKcznTw63EKKq27mnHGBJ06uTNX+AJU2v0tF6IMBSyTH8h8emtwuE6K0UGZrDV2aVTsPNe5IMDidY2ZIjfn13qAI/wIb2J1m7/DniTWMYR4g3jbF2+XNsaH8y6KJJMd3dtFzQwZpF3lPDsa09s864JtWlBl8pKf807uX5p3FDPsZOvrdP9kArHa3q7VM3UilGRiCL2gKqQf38pSJB996RiNEzAlWj3j5yXKaPrhn22r40iIJnBKo38InMRMFfJqVScM9IUqNrSqDWjWZIDQRdisan4C+Ttf1nVNuXoCUSrI74QHG1ouAfcVMGX4vChOcSbvmZxVIp74EwTSAzb9TVM6IyGYht7WFwOMGyphzx8zsU+CU8CrqC7h5sU1fQeaCafwRNqe2H7GGtKNKAdCWUGChOb9fqUPCPkMKhGVpa0Lj6IaAB6crQ3U3LtGkk1RW0ckr7BKy9vxfyz1o45y3Pg1QKdg+20UGWlpUJryeP/nsCVzggnWPBlAHppEAySWxLN0uXQmcsw8BA0AWqfwr+AcrPvdu567rJCVk23rWhqh8A6TRcdHO4h2aIspkGpJMiEomgS9AwlPYJ0HzOvRv0xCpSHg1IN3ftI1l2D3tdQfUoyvFTzT9IZt5UiwWqMfViOu3128/X9mNbVNsPKw1IN0ddXcSv72ZlS47BYW9AuMK5JaR8VQn+ZtZpZo+bWb+ZbSmyvcvMXjSzrP/6ZDXOW/f8VE+hfAqoUp2xzNHcvoTWwgWOXRu/xjc+/D1uujDDNz78PTX2lqO7m5YWaFV27LhVHPzNbCHwBWADcBbwcTM7q8iu33LOdfivL1V63kZQOPduzw1HJlNA7U/tqui4uRy0jg5Up5Ay7xYucFxy5hNcf8FuLjnzCQX+Oeg8kCaXC7oU9akaNf93AP3Ouaedc+PAN4HLqnDchtff3smOT+ycTPX0rt/Gjk/sPO4pGNPpow9utcZyTJk5W6TRdHSQ4Gj6Rw+BzU01GnyXA88WLA8Ba4vs92EzuwB4ArjOOffs9B3MbBOwCWDp0pVVKFr4VWvuXT24JZGTnyc4neaaoRT0QS/q/1+uatT8i7VOTr9v/RHQ5pw7B7gPuLPYgZxztzvn1jjn1ixe/MYqFK3xqSunRF5XF/E4xDQW9JxUI/gPAacVLK8Ani/cwTn3knNuzF+8A3h7Fc4beZmMl99vXzSgxl2JvN8bTtHZp/RPuaoR/H8CrDKzM8ysCfgYcHfhDmZ2SsHipcBjVTiv4D3zEouh/L5Em9/755qJFDfvSar7ZxkqDv7OuQngGmAXXlD/tnPuUTO7ycwu9Xf7YzN71MweBv4Y6Kr0vFGXH4N//2CCBOruIEJ3N/HzO1gdH1APoDJU5Qlf59y9wL3T1t1Q8P2ngE9V41yixl2R2SQSXgVJjb+l6QnfOqIx+EVmkUySIMf+wYTSP7PQ2D51Ij8ccwdZ1fZFZtLdTUs6zZoDR6eDVF+IY6nmX0ficYivUOAXmVVXF7F1Hd68FVKUav51IJOBzQd6aJ/IakjbgGnWLWkUCv4hNyXdE6fmtX4Fu6M065Y0EgX/EMsH/jWLssTW1X7KRQW7qQpn3QKmzLp1yZlPBFw6kblRzj/k4nG8iVgC6LOmKQan0qxb0kgU/EMqX+tvH8kGVgYFu6nys24V0qxb4RYbzbF/uI1UKuiShI+Cf8hkMkcnW1+zKBto7x4Fu6mOd9at144Y9zxxJp/91wu454kzee1IZTO1SZmSSWLrOljWpFm/ilHOP4R66GFlSy7wgdrywW56zj+qUwzmZ93a2b+K7IFWOlpnbwBXu0nACoZ97sxl2B50eUJEwV9KOp5g1+jys26V28CrRmIJKwX/ECkcs4cV4ejPP9dgJ1PN1G6iaypBUvAPgfwInY9MpFnRkoOO2nfrlPmRbzfJ1/wh2u0mEh4K/iERi8EKgs/zS3Wp3UTCSsFfZB6p3UTCSsFfZJ6p3UTCSP38QyCbhY+MpIMuhohEiIJ/gNJpuOjmJIPDCVbEle8XkdpR8A9QLgftiwZoWZlQ4BeZZ+0jWXYMaXavPAX/gMViQFtb0MUQaWxdXcTP72DNoiwPP+MN9ZDJBF2oYCn4i0g0JJPEtnSz9PQEnbEMAwNBFyhYCv4iIhGk4C8iEkHq5x+QTAa2jPYQIxd0UUQkghT8A1A4gFtsRTCzdIlItCn411gqBb2jSZY15QKZl1cqp0ntpREo+AegtRViqMZfjzQ5izQKNfiKzIEmtZdGoeAvMgea1F4ahYK/yBxoUntpFAr+InOQn5wl3jSGcYR405gmZ5G6pAZfkTnQ5CzSKBT8A5DIDUA45meX46DJWepcLsfoaNCFCJ6Cf42k07D5QA+D4ylaWoC2jqCLJBItmQyjfVnGxiDTnIz8YLoK/jWQTsOOoSQdZImvTEBXV9BFEomWdJqRoRxZOth4eoauLkgGXaaAKfjXSDwO8YQCv0hQ+uMdbExk9C/oU28fEZEIqkrwN7NOM3vczPrNbEuR7c1m9i1/+14za6vGeUVE5PhUHPzNbCHwBWADcBbwcTM7a9puVwG/cs61A9uAz1V6XhEROX7VqPm/A+h3zj3tnBsHvglcNm2fy4A7/e+/C7zHzKwK5xYRkeNQjeC/HHi2YHnIX1d0H+fcBHAIOGn6gcxsk5ntM7N9r7zyYhWKJiIixVQj+BerwU9/3LGcfXDO3e6cW+OcW7N48RurUDQRESmmGsF/CDitYHkF8HypfcxsEbAUeLkK5xYRkeNQjeD/E2CVmZ1hZk3Ax4C7p+1zN3Cl//1HgAeccxoMRUQkIBU/5OWcmzCza4BdwELgK865R83sJmCfc+5u4MvA18ysH6/G/7FKzysiMquCIR16NaTDFFV5wtc5dy9w77R1NxR8Pwp8tBrnEhEpix/4901oSIdiNLyDhIImRZf5MBpLcAkZuruCLkn4KPhL4DQpukjtaWwfCZwmRRepPQV/CZwmRZeq8/P9w8NBFyS8lPaRwOUnRR8Zb55cp0nR5bilUgwPw/4FHWxcqXx/Kar518jICAwP5iCVCroooaNJ0aVqMhlyJPhKSzfXn6+x+2eimn8NdHXB9ZkMnX09XD6cZtnWFLF1HZBMBl20UNCk6FJtqUQPbUEXIuQU/GskmYTRZA/npHrIJpK0DQwEXaRQqZdJ0dUlVRqFgr9ImdQltX7kcrqxno1y/iJlUpfUkEulGN6d5ZGRNhKJoAsTfqr5i5Rppi6pYU9XNbxUiqGRBKtbBujogK5k0AUKP9X8RcqU75JaSF1Sw+O78S461I+ibAr+ImVSl1RpJEr7iJRJXVKlkSj4B2BoCFoX5Yil0+gplPpSL11SI8V/one0KeiC1BelfWqsuxs2rsiwb6JDT/yKVCKdZuTmFIPDCVa25Bjd0qN8/xwo+Aegqwvuuz7DypYcORKQyQRdJJH6k8vx3RXdXLBygO7uoAtTfxT8RUQiSDl/EakvBfPyHhiFtnVBF6g+KfiLhIzGD5pBOs3IUI4smpe3Ugr+IiGi8YNm1x/vYGNCwzVXSjn/gI2OAtls0MWQkND4QTPIZBg9kPP+Z6RiCv4B6uiANF0cOgSjW1Pq9SOa0rKUdJqRPVkOjifojKnWXw0K/gFKJmF0Sw9vO93LYeoOQDR+0DSZDKNbUxx6Jseti7q54gJ166wWBf8QaGuDvlgy6GJICGj8oAKpFCN7suyb6OD0pXqIq9rU4BsSo6MwNJHgZE3xGGkaP8iXyTA6Crcu6qZ3XQ/dyaAL1HhU8w+BfPpndXyAg+MJRvZkIZ0OulgSkPz4QddfsJtLznwieoHfNxpLsDWm2v58UfAPke5uuOKCAb4U7/bmoROJIr9xd3g46II0NqV9RCQc/Cd3x8fhtqZutrf20N0VdKEal2r+IdRDD4PDCUZuTin9I9FQ0JUzP0KnunPOLwX/kEkmvfTPBSsHyNLByFBOHwDSuNSVMzAK/iHV1eWN+98f7wi6KCLzw6/tqytnMBT8Q25kBEYP5PT0rzQWf4C2Wxd1c/35GdX2A6DgH2L52v++iQ6Gd2c165c0lP54B9tbVdsPioJ/yHV1wfXnZ7itqZuREZT/l/rn1/pHRoIuSLQp+NeBZBK2t/Yo/y/1LZPxJlsf9NI9G1dogLYgKfjXkZER1PtH6pPfuFs42boCf7AqCv5mdqKZ/bOZPel/fUOJ/V4zs6z/uruSc0ZVPv1z66JuhgdzGgJa6oPflTNf21dXzvCotOa/BbjfObcKuN9fLubXzrkO/3VpheeMrPwYQCtbchoDSMKvyINbatwNj0qD/2XAnf73dwIfrPB4Uob8GEC3Lurm0DO6C5CQ0YNbdaHS4P8bzrkXAPyvy0rsFzOzfWbWZ2YlPyDMbJO/375XXnmxwqI1tsKJYPZNdHh3AeoKKkHTGPx1Y9bgb2b3mdkjRV6XzeE8K51za4DLgZSZvbnYTs65251za5xzaxYvfuMcDh9dU9oChtEHgAQjX9s/hB7cqhOzjurpnLuo1DYz+4WZneKce8HMTgEOljjG8/7Xp80sA5wLPHV8RZbpkknopYdYFrpR8JcaS6UYHob9CzrYeLrXfTMZdJlkVpWmfe4GrvS/vxL44fQdzOwNZtbsf38y8FvAzyo8rxQxOooeBJPa8Wv7g8MJbmvyavvqvlk/Kh3PfyvwbTO7ChgEPgpgZmuAzc65TwJvBf7BzI7gfdhsdc4p+FdZMgnbB3rgAFw9mKJJ00HKfCqs7a9Ubb8emXPhnCLu1FPXuE2b9gVdjLqUSsH+4TZOXJAjviKBqmNSNf7QDC8fSXBOywAdql+Ezo032oN+G+uM9IRvA1JXUJkX/tAMWTq4YKXXfVOBv35pGscGlUzCaLKH01M93DOSpGNPlvjAgO4CpDyZDGSzk4sjI0zW9ru7oSuwgkm1KPg3uO5uuD6TobOvhz94JkWz2gJkNgX5/D2Lkt66RWhO3Qaj4B8B+buAt6V72DGU5JzdWZr6jtbqYq1qFxAgnWb0gDd0SLF8fldQ5ZJ5oZx/hOQfCFvZkqM15r1SqF1AmMzn75tQPj8qVPOPmGRy6j/1KF67wP7hNk5Uu0A0ZDKM9mWZmDi6Svn86FHNX9Q7KEpSKYZ3e2PvrF6Rm3zla/sSHar5C3Bsu0DHnizxbBZFhAZRkM//elMXvet66EoGXSgJkoK/TNHVVdA76JDfO6g1cXSHtjYlg+tB4RAfuZyexpVjKPjLMabfBSw64K1fMTFAYkjtAqHm5/PHx+FnTd6czxMTcElLRvl8mULBP+Ta+3vpf/N6MAPnaH9qF/3tnTU5d/4uoNCU5wViBRuUHgpGOg253ORiYQ2/re3obt3J2hYryPetlEfBP8Ta+3vZeNcG+tZeS+/6bXTuuo51e29hxyd21uwfaXqGJ//U8JbRHhj11l0+nubEm1MaR6iW/Br+2Bh8sbngg7eJwPP5YXjfyuw0sFuYOTf5j5OX/4fCLMCCTZVOw+YDPVw9nqKpCd0RVMu0IRYKjYxAlg4uiYdw0pQ6ed82qnIHdlPwDzvn6LnpaI/cnhuOhPYfKJXCuyPwXT6e1siix6vYEAvT9K6cnzj0AAAHC0lEQVQL8RSJdfS+bTTlBn+lfcLMr0EV6tx1XWhrUN3d3kNjeRfkh5MYzNK0tfgMY5EdWsKv1Y+OFt9caoiFQiVWB6/O3rdRpeAfYu1P7WLd3luOyZ32t3fWRe60qwvuI8MlJWaW3DIa0cHmpo2JX0yilbodRK3e37dRobRPyDV6r4n8xDNNTRCfyM2476JFhPtDwg/qs8mRYMeirnCnbSrU6O/bMFPOX+pGJgMDAzPvk8tNm52ssB9jCOT71t/W1M321p5Z949ipktqQzl/qRvl1n6vyAzQ2dfD5YNpFgwNzGeR5uyIBkYLnO425kbBX+pG/snjKzI9QRelqFo/SCVH6dmCuVPwl7rTqHlyOX79b15P39prWbf3lsnnC/rWXuvdCUhRGtJZROqfmdeVtIC6ls5MwV9E6l+JZwsIaYeWMFDwD5n2/t6jb1jnvGURmVHhswU9NxyZTAG1P7Ur6KKFloJ/iOQbrfI1ls5d17Hxrg36ABCZRX97Jzs+sXMy1dO7fpsae2ehBt8QUaOVyPGbEujNFPhnoZp/mKjRSkRqRME/TNRoJSI1ouAfItVotFKDsYiUQ8E/RCpttFKDsYiUSw2+IVNJo5UajEWkXKr5NxI1GItImRT8KxSqHLsajEWkTAr+FQhbjl1POYpIuZTzr0DYcuz5BuP8mOa967dp6jwRKUo1/0qEMMfe39559Px6ylFESqgo+JvZR83sUTM7YmYlpw0zs04ze9zM+s1sSyXnDBXl2EWkTlVa838E+B1gd6kdzGwh8AVgA3AW8HEzO6vC84aCcuwiUq8qyvk75x4DsJnTHO8A+p1zT/v7fhO4DPhZJecOA+XYRaRe1SLnvxx4tmB5yF/XEJRjF5F6NGvN38zuA1qLbPq0c+6HZZyj2G1B0aS4mW0CNgEsXbqyjEOLiMjxmDX4O+cuqvAcQ8BpBcsrgOdLnOt24HaAU09do1ZTEZF5Uou0z0+AVWZ2hpk1AR8D7q7BeUVEpIRKu3p+yMyGgHcC/2Rmu/z1p5rZvQDOuQngGmAX8Bjwbefco5UVW0REKlFpb58fAD8osv554H0Fy/cC91ZyLhERqR494SsiEkEK/iIiEaTgLyISQQr+IiIRpOAvIhJBCv4iIhGk4C8iEkEK/iIiEaTgLyISQQr+IiIRpOAvIhJBCv4iIhGk4C8iEkHmXDjnTDGzF4Fnanzak4Ff1vic9UDXpThdl9J0bYqrxXU53Tn3xtl2Cm3wD4KZ7XPOrQm6HGGj61KcrktpujbFhem6KO0jIhJBCv4iIhGk4D/V7UEXIKR0XYrTdSlN16a40FwX5fxFRCJINX8RkQhS8BcRiaBIB38z+6iZPWpmR8ysZPcrM+s0s8fNrN/MttSyjEEwsxPN7J/N7En/6xtK7PeamWX91921LmetzPb3N7NmM/uWv32vmbXVvpS1V8Z16TKzFwveI58Mopy1ZmZfMbODZvZIie1mZp/3r9t+Mzuv1mWEiAd/4BHgd4DdpXYws4XAF4ANwFnAx83srNoULzBbgPudc6uA+/3lYn7tnOvwX5fWrni1U+bf/yrgV865dmAb8LnalrL25vB/8a2C98iXalrI4KSBzhm2bwBW+a9NwBdrUKZjRDr4O+cec849Pstu7wD6nXNPO+fGgW8Cl81/6QJ1GXCn//2dwAcDLEvQyvn7F16v7wLvMTOrYRmDEMX/i7I453YDL8+wy2XAV52nD0iY2Sm1Kd1RkQ7+ZVoOPFuwPOSva2S/4Zx7AcD/uqzEfjEz22dmfWbWqB8Q5fz9J/dxzk0Ah4CTalK64JT7f/FhP7XxXTM7rTZFC71QxJRFtT5hrZnZfUBrkU2fds79sJxDFFlX9/1jZ7ouczjMSufc82b2JuABM/sP59xT1SlhaJTz92/I98gsyvmdfwR8wzk3Zmab8e6O3j3vJQu/ULxfGj74O+cuqvAQQ0BhjWUF8HyFxwzcTNfFzH5hZqc4517wb0cPljjG8/7Xp80sA5wLNFrwL+fvn99nyMwWAUuZ+ba/Ecx6XZxzLxUs3kEE2kLKFIqYorTP7H4CrDKzM8ysCfgY0LA9W3x3A1f6318JHHOHZGZvMLNm//uTgd8CflazEtZOOX//wuv1EeAB1/hPT856XablsS8FHqth+cLsbuAKv9fPOuBQPs1aU865yL6AD+F9Co8BvwB2+etPBe4t2O99wBN4tdpPB13uGlyXk/B6+Tzpfz3RX78G+JL//buA/wAe9r9eFXS55/F6HPP3B24CLvW/jwHfAfqBHwNvCrrMIbkufw086r9H/gV4S9BlrtF1+QbwAvCqH1+uAjYDm/3thtdT6in/f2dNEOXU8A4iIhGktI+ISAQp+IuIRJCCv4hIBCn4i4hEkIK/iEgEKfiLiESQgr+ISAT9fyDyprIwsvjFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_813(df, cls, tit):\n",
    "    xsp1 = df.loc[df['y']==1]['x1'].values\n",
    "    ysp1 = df.loc[df['y']==1]['x2'].values\n",
    "    xsm1 = df.loc[df['y']==-1]['x1'].values\n",
    "    ysm1 = df.loc[df['y']==-1]['x2'].values    \n",
    "    \n",
    "    x1_min, x1_max = -1.2, 1.2\n",
    "    x2_min, x2_max = -1.2, 1.2\n",
    "    xx1, xx2 = myplot.get_grid(x1_min, x1_max, x2_min, x2_max, step=0.02)\n",
    "    myplot.plot_decision_boundaries(xx1, xx2, 2, cls, alpha=0.5)\n",
    "\n",
    "    myplot.plt_plot([xsp1, xsm1], [ysp1, ysm1], 'scatter', \n",
    "                   colors = ['b', 'r'], markers = ['o', 'x'], labels = ['+1', '-1'], \n",
    "                   title = tit, yscale = None, ylb = -1, yub = 1,\n",
    "                   xlb = 0, xub = 1, xlabel = None, ylabel = None,\n",
    "                   legends = ['+1', '-1'], legendx = None, legendy = None, marker_sizes=[25, 25])\n",
    "\n",
    "def gen_data_813():\n",
    "    datap = np.array([[-0.494, 0.363], [-0.311, -0.101], [-0.0064, 0.374], \n",
    "                      [-0.0089, -0.173], [0.0014, 0.138], [-0.189, 0.718],\n",
    "                     [0.085, 0.32208], [0.171, -0.302], [0.142, 0.568]])\n",
    "    datan = np.array([[0.491, 0.920], [-0.892, -0.946], [-0.721, -0.710],\n",
    "                     [0.519, -0.715], [-0.775, 0.551], [-0.646, 0.773],\n",
    "                     [-0.803, 0.878], [0.944, 0.801], [0.724, -0.795],\n",
    "                     [-0.748, -0.853], [-0.635, -0.905]])\n",
    "    p_df = pd.DataFrame(datap, columns=['x1', 'x2'])\n",
    "    p_df['y'] = 1\n",
    "    n_df = pd.DataFrame(datan, columns=['x1', 'x2'])\n",
    "    n_df['y'] = -1\n",
    "    df = pd.concat([p_df, n_df])\n",
    "    return df\n",
    "\n",
    "df = gen_data_813()\n",
    "\n",
    "X_train = df[['x1', 'x2']].values\n",
    "y_train = df['y'].values\n",
    "\n",
    "reg_type = None\n",
    "lambda_t = 0\n",
    "Q = 2\n",
    "lr2 = lm.LinearRegression('pesudo_inv', reg_type, reg_param = lambda_t, \n",
    "                          poly_degree = Q, to_classify=True)\n",
    "lr2.fit(X_train, y_train)\n",
    "\n",
    "Q = 3\n",
    "lr3 = lm.LinearRegression('pesudo_inv', reg_type, reg_param = lambda_t, \n",
    "                          poly_degree = Q, to_classify=True)\n",
    "lr3.fit(X_train, y_train)\n",
    "\n",
    "plot_813(df, lr2, \"Exercise 8.13 (a) Q=2 \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X90XOV95/H31zbSBAtJAeIaWxhBZDYF1gjijV3K0mkCtZzQkLTJ2YS4RGfpOiShZehPt3BSOXVbZ9OGcZYECknOQIH8bNM6BNtbIFMfn1ZODJm4EAIIVjYCKxAcSZFBErKf/ePekUbyjDTSjObemft5nTNHmnvv3PvM1eg7z+/HnHOIiEi0LAo6ASIiUnkK/iIiEaTgLyISQQr+IiIRpOAvIhJBCv4iIhGk4C8iEkEK/lJVzOwjZvZ/g05HOZnZx8wsWeSxnzOzGxY6TVL7FPylaGbWa2avm9lwzuP2SqbBOXe/c+43yn1eM/s9M/t/ZjZkZgfM7PIZjr3RP2bUzFLT9l3g7/u5/3jYzC6Y4Vx1wK3AZ4tM6meBW/zXFTpnvZn9jZkd9v9ez5rZH5mZFXmN3HP9bzN7wb8vh8zslrmeQ8JJwV/m6jedcw05jxvLeXIzW1LO8xV5zXXAduADQBPwZeDbZra4wEteArYBXymw7wPA6cCZwE7gazNc/hrgJ865F4tJq3PuCPAT4L0zHPZN4F3Au4HTgN8BPgb8XTHXmObLwNucc43AZcC1ZvZb8ziPhIyCv5SFmd1hZt/Kef4ZM3skm9s0s6vNLGNmA2b272a2JufYXjP7UzM7CBwzsyVmdraZ/ZOZvWJmr2ZLGGbWaWb7/N/NzG4zs5fNbNDMDprZRf6+ejP7Wz/3+1Mzu9PM3lQg+a3Ak865x5w338m9eIF7Wb6DnXP/5Jz7Z+DVPPsGnHO9/nkMOA60zXDrNgL/Nu1eftPM+v33tNfMLpz2mjTwnnwnM7N3Ab8B/LZz7gnn3LhzrhvYBNxkZufNkJaTOOeeds4dy9l0Ypb3I1VCwV/K5Q+BNX5w/u/A9cBHnXPOzC7FyyV/DDgD+Htgp5nV57z+w3gBrRlwwIPAIbzAvJL8ueffAK4Azvdf9z+YDMif8be34wWrlcCnCqR9F7DYzNb5uf3/CWSA/jnegwlmNgCMAP8H+OsZDv2vwNN50rMa78vnceD+afufAi4ucL6rgP3OuRdyNzrn9gN9eCUCzOyL/hdxvsfBae9li5kN+69fCjwww/uRKlHxIrZUvX82s/Gc53/snLvbOfeamW0CdgO/AH7POdfnH/O/gL/3AxDAPWb258B6JnO9n88GLDP7FWCFf+7stfblScsbeNUabwO+75x7yn+9+ddc45w76m/7a7yg9Wd5zvML4B/9axgwAGx0Jcx66JxrNrOlwEfxvsQKafavn/vaieokM+sCfm5mTc65wZz0Nhc435nAkQL7jgBv8a/xCeATs7yNbHq2m9ln8L5I3wcMzvISqQLK+ctcvc8515zzuDu7wzn3feB5vAD6jZzXnAP8YW7uEjgbL8Bn5eZUzwYO5QT+vJxzjwK3A18Afmpmd5lZI16AOxV4LOd6u/3t+fwuXm7/QqAOr4rkQTNbUeD4ovjVJXcC95pZ3iok4Od4X2AAmNliM9tuZs+Z2RDQ6+86M+c1p+F9QeXzM+CsAvvOAl4pMvlTOM8PgdeBrfM5h4SLgr+UjZl9EqjHa/T8k5xdLwB/Ne1L41Tn3FdzjnHTjl9VTOOvc+7zzrm34wXu84E/xguArwMX5lyvyTnXUOA0FwPfcc4945w74ZzbjZdLvqy4dz6jRXhfRCsL7D/opzvrWrxG4CvxGp9b/e25PXV+GfhRgfM9DKwzs7NzN5rZO4BVwF7/+Z02tddW7uPJGd7PEuCtM+yXKqHgL2VhZufj9YDZhNe75E/MrN3ffTdwg1+nbma21MzeY2anFTjd9/GC73b/2JiZ/Wqea/43/5ynAMfw6tiPO+dO+Ne8LZvjNrOVZrahwPV+ALzHzM7z03cVXkB+osB7XWJmMWAxXltBLPtFZWZXmdklfg6+EfgcXu7+qQLXfgj4tZznpwGjeG0Xp5K/veDX8NoFTuKcexh4BPhHM7vQT8d6vHaDe51zT/vH3TCt11bu40L/vSwybwzCm/378g7gk/75pdo55/TQo6gHXhXE68BwzuPbeLnB7wNbco79OPCfQL3/vAMvyA7gBfZvAqflnPfKaddaBWR71PwMr00AoBPY5//+Lryc87B/zP1Ag78vhhc4nweG8ILv7xd4XwZ8GjiMV5/+FPA7Ofv/HNiV87wLr6SS++jy930QryvmMF4Vy0N4bQ+F7ukp/nVX+M8bgH/x03EIuM4/f5u//yy8hte6Gc4Zw2vwfgGvXcThNTzXz/HvvQivuuyo/36e8e+FBf1Z1KP0h/l/ZBEJiJltBi5wziWKOPbvgOecc1+cw/nvwat2erdzbmz+KZVaouAvUuP8arE/AP7NeX3+RRT8RUSiSA2+IiIRFNpBXqeeeqZrbm4NOhmyQH7xCxibVvt8xhnBpEWklhw58tjPnHOFxrRMCG3wb25uZfPmA0EnQxZAKgW7Xo+z5kSGRX7Z8/hxuONogt3ru4jHg0ydSHXbutVmGlE+IbTBX6pbssDs9FtGuvjRaJL6eoitb2ci0qdS3NiX5Nq9KdZkevO+trkZOjsXIrUi0aPgL2WVTkNHdxeHx5LUFZhxPtYEJKb1auzspCGdpiGToX/k5Glrxsfh6FAza5K9tOd8Z4jI/Cj4S9kkk3BwqJXTFw3QsGoe2fR4HOJxYgV2NySTHB5q5uC+dvZ1x/Meo2ojkeJUVfCvq3uDSy/to6lpJOiklGRwMMbjj7cwNnZK0Ekpi1QK7uuL8wcnMjQ2woJlzRMJGtNpLs9kuJzMSbuHh+HyfWmuzqRPKliIyFRVFfwvvbSP8847jaVLW7G5r0gXCs45jh17Feiju/vcoJNTspJz+3Pllw7yaUinWdud4dBgM3dsV+OxyEyqKvg3NY1UdeAHMDOWLj2DpqZ5zawbGtm6/SfGU7Q0Dpxchx+EeJxYPE5sWuNxe/uUQ0SEKgv+QFUH/qxaeA8AsRi0NAOt7bMeW1F+4/GS7gyHh5r5cbeXvvFx2NcdV4lAhCoM/hIOmQzcN5wCBsKZnc4pBazProcyMED7cIbL96XZ1Juecri6kErUKPiX0Xe+800+97kunn32Kb773e9z8cVrg05S2WWrew75ffWn1KmE0bSonm0XeKKvmeElXpfSsTHUhVQiR3P7zNO//3uaRKJzyra3ve0i7r77n1i//oogklQRmQx0kqLpnGZiWxLVFy3jcWJbEjRc3s7y9a0sX9/KqsYBDg81s21fnFQq6ASKVEZN5/yPH4dHH4UnnoCLLoJ3vhMWL164661e/csLd/IQicWA1tagk1Ga3C+teJxGv0Two0PqKSTRULPB//hxuPZa+OEP4bXX4NRT4ZJL4IEHFvYLoFadVN1Ta7JtBMkknxhKTowXyNLUElJryhL8zewrwNXAy865i/LsN2AH8G7gNaDTOfd4Oa5dyKOPeoH/2DHv+bFj8Pjj3varrpr/ea++eh2jo6O89towAwNHueoqr877lls+QzxeaInY6pYdxLXmRIbGfFMz1BJ/INna7slpJjS1hNSicuX8U8DtwL0F9m8EVvuPdcAd/s8F88QTXo4/1+uvw5NPlhb8H3xwP+DV+X/jGymSydT8T1YlBgagpQUaiUj21y8F5MqdWmJTbzoSt0FqW1kafJ1ze/EWeS7kGuBe5+kGms3srHJcu5CLLvKqenK96U1w4YULedXa1TzQG3QSgpVI0HhFO2uXZNh7uJXY9i7S6aATJTJ/larzXwm8kPO8z992JPcgfyHrzQBNTatKuuA73+nV8T/+uJfjf9Ob4NJLve0LZdeub3Prrb/H0aOvcN117+HCC9t54IE9C3fBBXbSnD2dNVzdUwy/RLAqp12AfZO7N7WoRCDVo1LBP9+Q1pMWD3bO3QXcBbBixdqSFhdevNhr3H30Ua+q58ILy9vb57LL4lx2WXzKto0b38/Gje8vzwUClg387WQqM2dPNclOMNfbO7FpuG+AvYdbWZPsrekmEakdlQr+fcDZOc9bgJcW+qKLF3v1+6XU8UdZQwM0qJtLftPbBMhpF9jWrlKAhF6lBnntBK4zz3pg0Dl3ZLYXiVSVRILGVc1T2gVSKSYeImFSluBvZl8F/gP4L2bWZ2bXm9kNZnaDf8hDwPNAD3A38IlyXFckdDo7iW1JsKpxgBvHk3T3t9Ld38q+vlau3Fa7I4jbenaD82tqnfOeS6iVpdrHOffhWfY74JPluJYsvGx9fxsZb3STzF0i4S1LmX3e20tzX6Ym2wXaenaz6f6NdK+7id0bbqNjz82s37+D+z6yi562jqCTJwXU7AhfmbvsKN69YymW1Q0QW676/pJEpF2g560b6F53E+v372D9/h0AdK+7iZ631uagx1qh4C9TxGKwKhaSxVlqUSJBYyrF2v7JUsC03dXHjN0bbpsI/AC7N9wGNbJuRa3SrJ4l6On5Cb/5m7/CuefWc+edf1vUa+pHBkNbN9rbq1qeishpF+gfaZ54HB5qnmgkrirO0bHn5imbOvbcPPk5l1BS8C9Bc/Pp/OVffp6PfeyPijq+fmSQM44+S+OQN96tY8/NbLp/Y6i+AKSCEgliWyYfjauauXE8yd7DrSSTQSeueG3P7WH9/h10r7uJrk+dmKgCanuuegc4RkFNV/vUf283o/ENXvHTOerTexj99fI1QJ155jLOPHMZjzzy3aKOH61vZHjpMhqOvUzz4M/4Zf8fRnWjAnjLT3Jyu0B29uywTijX09bhNe6+dcNEFVBPW4cae0OuZnP+9d/bzRmbNtL4F17xs/EvbuaMTRup/16AuWwzhhrPnrJJdaNyEn+8QDsZ9vW1sr07zvbuOFdui4e2RNDT1jH5OTZT4K8CNRv8R+MbGL7+Jhq+vIMVLYto+PIOhq+/ySsJBMW5iSqfrDDVjQ4MQEd/KuhkCHilgFsTtLTA+uW9rF/ey9olmeptF5DQqdngjxlDW2+bsmloa+m57FTqC1x1VTtXXdVOf//cZqioHx2i4djLDC9dxkDTOaGpG02l4MptcQ4PNbM8NhD+dXmjpLNz4lHN7QISPrVb5+9X9eRq/IubS/4C6Oz8JJ2d8xuvNhpr4tXTVzNa3wgDPwlF3Wg6DTf0d2kCt2pRoF0gq7U1vG0DEi41m/OvT++ZqOp5qe/ERBVQfbp8ueyXX+7n7W9v4a67PseOHdt4+9tb+MUvhmZ8zWisKXR1o7GYN4mbAn8VyWkXyE4h0d3fyr17VSIIQjVOb1GzOf/RX+/g1ft2TfT2Gdp6G6O/3lHW3j7Lli3nscf6ynY+kTnJlgJyVpUZ6fbaBb64PcH2WNfEdi0/uXCqdXqLmg3+wNRAb1bWwC8SGjlRPRaPE0ulSPQnSYx4RYDxccho+ckFU63TW9R08I+Stp7dE/2scY625/aEOtchC6izk1juc39B+h8dauaO7ZPzR2yPdalEUA5VOr1F1dX5u5B0iyxFud9DttiZ7TaqkcMyRTxObEuCpnOa2RJLTjwODzWzbV/tTjNdMVU6vUVV5fwHB2McO/YqS5eegYX8W7UQ5xzHjr3K4GBs9oOLNN9iZzIJDw57SzXSokl9at60Op/GGUoEVTnBXEByp7fIrfMPuiffbCysOekVK9a6zZsPTNlWV/cGl17aR1PTSECpKo/BwRiPP97C2Ngp5Tupc3R9erIg1/WpEzMWO5NJODjU6k3dvF5l/0ib1j1oaAgOLpo6tQToIzKTMFW7bt1qjznn1s52XFXl/MfGTqG7+9ygkxE+BYqds9U7NjZCTJW+Mi2b35hK0d7nTS3R198KeI3GV2fSKhEUMCXQh6QL92yqKvhLftVa7Cy34yeMXT2r+eGR5VxyVj8b255l8aJwlmxDLduFNJWihV4ARvoHJrqQ3rm8SyWCGqDgXwM0q6IX+Dfc9zvsf3Elx8ZOYWndG6xb+SJ7Nv2DvgDmK6eNIAbEUilu7EuyqS8F/vCWXlpVIqhSCv41ohqLneW0q2c1+19cyfBYPQDDY/Xsf3Elu3pWc/X5zwScuhqRZ1DZmd0ZDg16DcZ3Lu8q9DJZQOm0txDTXCn4S0344ZHlHJvWgH5s7BQy/csV/Mstz6CyG/uS3Nh38rwSJ05QU+sVh00qBff1xVlzIsMiv7/HaUW+VsFfasIlZ/WztO6NiZw/wNK6N2hf3h9gqiLCLxHk5Tce51uvOJ9mzS0IeEF9YGDmY7aMdPGj0ST19Uztsbd1a1HXUPCXmrCx7VnWrXzxpDr/jW3PBp20aMuZhbR/ZOaxJOPjcHSomTXJ3si2IaTT0NHdxd6xFMvqZon+QKyJk3prFUvBP2JSKW8K58NjSerqgk5N+Sxe5Niz6R/Y1bOaTP9y2pert0+oJBIUM6wxd6rqfUviRZ8+TAPTkkkvVz4f28bTrDmRobGReQf1YlXVIC8pTXZg1+mLBmhoUflaQqqYOo8cIyMwOgp31CfYvb4rsK6n2Vz7x7NVMfMdxF/i2BvburX2BnlJ6RoboUEDuyTM5pgpye2Geu3eFA90z+315XLvWMrLWJ1THRkrBX8RqX6dnTSk0zRkMmwhoNVsYlTVwgkK/iJSG+Lxqgm8YVB1UzqLiEjpFPxFRCJI1T4iAdOEdBIEBX+RAGlCOgmKqn0C1taze3K5N+e09GLE5E5I51g0ZUI6kYWk4B8grb0rM01IJ7KQVO0ToPmuvTtX2ZGHh/yRhxIempBOgqLgHyR/4ZVs4AdmXXpxrrJTvraTqZqRh1GiCekkKGUJ/mbWAewAFgNfcs5tn7a/E/gs8KK/6Xbn3JfKce2qNs+1d+eqoQEaNFduKGlCOglKycHfzBYDXwCuwlvc7QdmttM59+Nph37dOXdjqderJVp7V8D7Arj6/Ge06IxUVDly/u8AepxzzwOY2deAa4DpwV+m0dq7IhKUcvT2WQm8kPO8z9823W+b2UEz+5aZnZ3vRGa22cwOmNmB1157pQxJC7+eto7JKp4Irr0rIsEoR/DPVzk9vcLyO0Crc24N8DBwT74TOefucs6tdc6tPfXUt5QhaSIikk85gn8fkJuTbwFeyj3AOfeqc27Uf3o38PYyXFdEROapHMH/B8BqMzvXzOqADwE7cw8ws7Nynr4XeKoM1xURkXkqucHXOTduZjcCe/C6en7FOfekmX0aOOCc2wn8vpm9FxgHjgKdpV5XRETmryz9/J1zDwEPTdv2qZzf/wz4s3JcS0RESqe5fUREIkjBv8YNDMDykd6gkyEiIaPgX6NSKYht7+LwUDPNDEBra9BJEpEQ0cRuNWjKZG6rNKdPOWnVLakVCv41qlyTuSnYTdKqW1JLFPylIAW7qXJX3QKmrLqlSdmk2qjOXwrSEoNTadUtqSUK/lKQgt1U2VW3cmnVLalWCv5SkILdVNlVtxrqRjFO0FA3WtSqW8dPGA8+cz5/+W9X8OAz53P8RPkW6hGZL9X5S0FaYnCq+ay6pXYTCSsFfylISwyebK6rbqmRWMJKwV9mpCUGSzNTu4nuqQRJdf4iC0jtJhJWCv4iC2i+jcQiC03VPiILSO0mElYK/iILTO0mEkaq9hERiSAFfxGRCFLwrzHZ6ZzbhjNBJ0VEQkx1/jUinYZMBg4OtbKsboDYcs3jLyKFKfjXkC66WNU4AIlE0EkRkZBTtY+ISAQp+IuIRJCCv4hIBCn4i4hEkBp8ReZIi9pLLVDwF5kDLc4itULVPiJzoEXtpVYo+IvMgRa1l1qh4C8yB1qcRWqFgr/IHGhxFqkVavAVmQMtziK1QsFfZI60OIvUAlX7iIhEkIK/iEgEKfiLiESQgr+ISASVJfibWYeZPW1mPWa2Jc/+ejP7ur9/v5m1luO6IiIyPyUHfzNbDHwB2AhcAHzYzC6Ydtj1wM+dc23AbcBnSr2uiIjMXzly/u8AepxzzzvnxoCvAddMO+Ya4B7/928B7zIzK8O1RURkHsoR/FcCL+Q87/O35T3GOTcODAJnTD+RmW02swNmduC1114pQ9JERCSfcgT/fDn46cMdizkG59xdzrm1zrm1p576ljIkTURE8ilH8O8Dzs553gK8VOgYM1sCNAFHy3BtERGZh3IE/x8Aq83sXDOrAz4E7Jx2zE7go/7vHwAedc5pMhQRkYCUPLePc27czG4E9gCLga845540s08DB5xzO4EvA/9gZj14Of4PlXpdERGZv7JM7Oacewh4aNq2T+X8PgJ8sBzXEhGR0mlWTwkFLYouUlkK/hI4LYouUnma20cCp0XRRSpPwV8Cp0XRRSpPwV8Cp0XRRSpPwV8Cp0XRRSpPDb4SOC2KLlJ5Cv4SCtWyKLq6pEqtUPAXKZK6pEotUZ2/SJHUJVVqiYK/SJHUJVVqiYK/SJHUJVVqiYK/SJHUJVVqiRp8RYqkLqlSSxT8a8jICAyNQWMyCYlE0MmpSdXSJVVkNqr2qRHxOIxs6WJV4wCHh5oZ3paEVCroZIlISCn415hEAq5Y1UtPQ3vQSRGREFPwFxGJIAV/EZEIUoOvSMho/iCpBAV/kRDR/EFSKar2EQkRzR8klaLgLxIimj9IKkXBv0YND8Nw34D6+lcZzR8klaLgX4M6O2FTS5rblyQYOjwAySSk00EnS4qg+YOkUtTgW6M6O2GELlYlu/jJSCvLe3uDTpIUQfMHSaUo+Ne45mboH2hlOb1BJ0WKpPmDpBJU7RMBw8NMVv+IiKDgX/M6O+HWy9MTE76NbFf9v4go+EdCPO5N+LamsZeRWHPQyRGREFDwj5iREUCNvyKRp+AfIe3tkKKTwUMDqv4RiTgF/wjJLvhy8TkDHBhvZ6Q7oy8AkYhS8I+gzk64uiGt+n+RCFPwj7DhYSCTCToZIhIABf+ISiTgooZerfcrElEK/hGWXe83OweQGoFFoqOk4G9mp5vZv5rZs/7PNxc47riZZfzHzlKuKeXV2ek1Aq9qHODlsWaG92VUChCJgFLn9tkCPOKc225mW/znf5rnuNedc+0lXksWUCIB16V76eju4uOHktRvn5wKIhbzDxCRmlFqtc81wD3+7/cA7yvxfBKgbFfQc5oGWB6bfKhdQKT2lJrz/yXn3BEA59wRM1tW4LiYmR0AxoHtzrl/zneQmW0GNgM0Na0qMWkyX9Mz+Vekermhv4tPHE6yaNtkiaChIc/BUZNOM9KdYXw86IQUZ8kSiC1v9ur7JNLMuZnnCTezh4F8a8jdAtzjnGvOOfbnzrmT6v3NbIVz7iUzOw94FHiXc+65ma67YsVat3nzgWLeg1RIbsZ/YAAODrWyrG7ACyatrUElKziZDENDcHBRO5ta0kGnpig39Hfx8dEk9fUQW19CTWw8XrY0SXnZ1q2POefWznbcrDl/59yVBS9i9lMzO8vP9Z8FvFzgHC/5P583szRwCTBj8JfwmZ5ZvCLVy319cVr7elnS3xtEkgI1NtbMA3Wd7F7fRWc86NQUZ4Quzkl2cXColbru3nmdo2F8gIZMRqW+Kjdrzn/GF5t9Fng1p8H3dOfcn0w75s3Aa865UTM7E/gP4Brn3I9nOrdy/tUjyk0B1Vp7kk7Pf36/KaWH5UWMEq/Wm1Slis35lxr8zwC+AawCDgMfdM4dNbO1wA3Oud81s8uAvwdO4DUwJ51zX57t3Ar+IuGVSsF9fXGWzFJ3cMFYhro6v4pJVUUVUZHgv5AU/EXCrZjxgL293pfEmhMZGhvx1hXNR6WDsilbnb+ISD7FZuRvTafp6O7i8uE0DJ+8f82JDI3JpNoQKkzBX0QWVDwOI/EuHi6w/+qk13Ps9G1Jr/twVruqihaSgr+IBCqRmBxLwoi37fLxNGv2ZmhUr6IFo+AvIoHr7PS6oWbdmvZmGz841MqynKlGQI3H5aJZPUUkdOLxyVlnc6caOTDe7k0+mEzOeg6ZmXL+IhJa0zsBZRuPPz6YZHHOVCNLlqhEMFcK/iJSNbKNxxenuia2ZacaOX1fhobe3qlTjejLoCAFfxGpOtNLBNnpyK89nKLOn2pE01DMTMFfRKpetkRwXbprYtqKiRLBtiQNLTmDy1pbVSJAwV9Easj0mJ7tQhrvT09su+CwupCCgn/otfXspuetG8AMnKPtuT30tHUEnSyRGYXlc5vtQrolPbkt24U06oPK1NUzxNp6drPp/o107LkZnKNjz81sun8jbT27g06aSEFh/NzG45OPbBfS25ck2D7iPfYNtzO0N1pdSDWxW5j5/zjr9++Y2NS97iZ2b7jNy1GJhFEVfm7TabwupLkL3VRpKaDYid2U8w8zM+8fJkeY/4FEgKr83GbXr774HG8gWRRKAarzDzM/B5WrY8/Nof9Hkoir4s9tZ6c3kCwKU0so5x9ibc/tYf3+HXSvu4muT52ge91NrN+/g7bn9gSdNJGCqv1zG5WpJVTnH3Jh6TUhMhe1+LmtlnYBreQlIrIAsktYTqxOFrLxAmrwFZFIaevZDdnMrHML1rW0sxNuvTzNF+sSHB5qZmR70vtGqDIK/iJS9So9tiDbO+iKVb1e76DDA15bQCo1+Qg5VfuISPULeGxBMgkPDsdZ4vefvGAsQ11dMO0CWsBdRKLDH1uQG/wr2bU0kfC6iGb19vrtAiFeilLVPiJS/QqMLaCCNRu5U0jkbRfI+XIIAwX/kKlUo5VILQnj2IKT2gVCNmpYwT9Ewjghlkg16Gnr4L6P7Jqo6tm94Tbu+8iuUIwtyJYCVjUOcHiomeFt4egdpAbfMKnCCbFEpHi5YwTq6ia3x2KUrV1A/fyrURVOiCUixevshIdv9UoB2WkjkiQYHKTi7QIK/mESgkYrEVl4icTkI6jZRBX8Q6QcjVZqMBapPtPbBSoxaljBP0RKbbRSg7FI9cqdTXTKqOEFogbfWqIGY5GakUz6aw0vGqChpdkrHhRBDb5RpAZjkZqRLQVkyCkFlLFBWMG/RKGqY1eDsUhNye0ddHio2VtMpkxtAQr+JQhbHXsYRzmKSOkSCbjuil5uX5Jg6PBAWbqFqs6/FCFv9zuCAAAFT0lEQVSsY6/FFZREZNJsbQGq86+EENax97R1TF7fTIFfpMbklgIGD82/FFBS8DezD5rZk2Z2wswKftOYWYeZPW1mPWa2pZRrhorq2EUkANlJ485pmlxYfmR70vsiKFKpOf8ngN8C9hY6wMwWA18ANgIXAB82swtKvG4oqI5dRIKUSHiDw1oaJqeLKFZJi7k4554CsJmrOd4B9DjnnveP/RpwDfDjUq4dBtlBWdk69t0bbqOnrUNVLSJSMdk1BLK2bi3udZWo818JvJDzvM/fVhNUxy4i1WjWnL+ZPQwsz7PrFufcvxRxjXzFgryV4ma2GdgM0NS0qohTi4jIfMwa/J1zV5Z4jT7g7JznLcBLBa51F3AXeF09S7yuiIgUUIlqnx8Aq83sXDOrAz4E7KzAdUVEpIBSu3q+38z6gF8Bvmtme/ztK8zsIQDn3DhwI7AHeAr4hnPuydKSLSIipSi1t8+3gW/n2f4S8O6c5w8BD5VyLRERKR+N8BURiSAFfxGRCFLwFxGJIAV/EZEIUvAXEYkgBX8RkQhS8BcRiSAFfxGRCFLwFxGJIAV/EZEIUvAXEYkgBX8RkQhS8BcRiSBzLpxrppjZK8ChCl/2TOBnFb5mNdB9yU/3pTDdm/wqcV/Occ69ZbaDQhv8g2BmB5xza4NOR9jovuSn+1KY7k1+YbovqvYREYkgBX8RkQhS8J/qrqATEFK6L/npvhSme5NfaO6L6vxFRCJIOX8RkQhS8BcRiaBIB38z+6CZPWlmJ8ysYPcrM+sws6fNrMfMtlQyjUEws9PN7F/N7Fn/55sLHHfczDL+Y2el01kps/39zazezL7u799vZq2VT2XlFXFfOs3slZzPyO8Gkc5KM7OvmNnLZvZEgf1mZp/379tBM7u00mmEiAd/4Angt4C9hQ4ws8XAF4CNwAXAh83sgsokLzBbgEecc6uBR/zn+bzunGv3H++tXPIqp8i///XAz51zbcBtwGcqm8rKm8P/xddzPiNfqmgig5MCOmbYvxFY7T82A3dUIE0niXTwd8495Zx7epbD3gH0OOeed86NAV8Drln41AXqGuAe//d7gPcFmJagFfP3z71f3wLeZWZWwTQGIYr/F0Vxzu0Fjs5wyDXAvc7TDTSb2VmVSd2kSAf/Iq0EXsh53udvq2W/5Jw7AuD/XFbguJiZHTCzbjOr1S+IYv7+E8c458aBQeCMiqQuOMX+X/y2X7XxLTM7uzJJC71QxJQllb5gpZnZw8DyPLtucc79SzGnyLOt6vvHznRf5nCaVc65l8zsPOBRM/tP59xz5UlhaBTz96/Jz8gsinnP3wG+6pwbNbMb8EpH71zwlIVfKD4vNR/8nXNXlniKPiA3x9ICvFTiOQM3030xs5+a2VnOuSN+cfTlAud4yf/5vJmlgUuAWgv+xfz9s8f0mdkSoImZi/21YNb74px7Nefp3USgLaRIoYgpqvaZ3Q+A1WZ2rpnVAR8CarZni28n8FH/948CJ5WQzOzNZlbv/34m8KvAjyuWwsop5u+fe78+ADzqan/05Kz3ZVo99nuBpyqYvjDbCVzn9/pZDwxmq1kryjkX2Qfwfrxv4VHgp8Aef/sK4KGc494NPIOXq70l6HRX4L6cgdfL51n/5+n+9rXAl/zfLwP+E/iR//P6oNO9gPfjpL8/8Gngvf7vMeCbQA/wfeC8oNMckvvyN8CT/mfke8Dbgk5zhe7LV4EjwBt+fLkeuAG4wd9veD2lnvP/d9YGkU5N7yAiEkGq9hERiSAFfxGRCFLwFxGJIAV/EZEIUvAXEYkgBX8RkQhS8BcRiaD/DyFDvUYcHX21AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_813(df, lr3, \"Exercise 8.13 (a) Q=3 \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.13 (b) \n",
    "\n",
    "The fit with $Q=3$ seems to have overfitted. It's more complicated and seems to fit the on the far right too much. \n",
    "\n",
    "#### Problem 8.13 (c)\n",
    "\n",
    "After adding regularization, we find that the $Q=3$ fit becomes much closer to $Q=2$ fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xt8W/V9//HXJ0ltlbiOCjQzSQguOKwFlhqaNVnHMtFC47SstKPdWmDUKyzNumyYdlvTll9rWramu0V05TIYPMStt/XCKCPORqnIj1/rtIEKFu6GOcFAGi5V/HPAMU6+++McObIi2XJ0OUfS+/l46GHpnKNzvj46+uh7vldzziEiIo1lVtAJEBGR6lPwFxFpQAr+IiINSMFfRKQBKfiLiDQgBX8RkQak4C8i0oAU/OucmZ1vZv8ZdDrKycw+YWbxIrb7JzNbW400Vcp0n5+ZxcxsaAb7S5rZxeVJ3aT9tpuZM7M5RW4/o3RL+Sn4V5CZDZrZq2Y2kvX4ejXT4Jy7zTn3nnLv18z+3Mz+x8yGzWybmZ0+xbbr/G32mVkiZ91J/rpf+Y+7zeykKfbVBFwG/H0Ryfx74PP+ewrtr9nMvmJmO/3P6kkz+0szsyL2n7uvf/Df///N7DEzu3Cm+8iV+/n5Abaj1P02IjM7w8x+bGZ7zGww6PQETcG/8n7POdeS9VhXzp0Xm9Mq8zGXAxuADwHzgBuAH5jZ7AJveQ64ArixwLoPAUcCRwN3AN+a4vDnAI85556dLp3OueeBx4D3T7HZvwHvBt4LvAH4I+ATwD9Ot/889gK/h3dOPgZcaWbvPIz9SGXsxbsG/yrohISBgn9AzOwaM/tu1uuvmtmPMjlOMzvbzFJmljazn5jZ0qxtB83sM2b2ELDXzOaY2bFm9n0ze8HMXsrcYZhZt5nd5z83M9toZrv93M9DZnaKv67Zz7nuNLNfmtm1Zvb6AslvBx52zt3vvPFBbsYL3PPzbeyc+75z7nbgpTzr0s65QX8/BuwHpsrZrgbuzTmXp/vnKG1mz5hZd9bqJPC+fDsys3cD7wHOdc5td86NO+f6gQuAS8zs+CnScQjn3Bedc4855w4457YC/xf4rQLHvtfMzs1KvzOz9/qvzzSzlP88+/Pb4r/9Qf8u8g+z9vdp/3N93sz+uJj0mtkJZnaPf728aGa3mVk0a/2gmf2Vf53sNbMbzOzXzGyTf3dzt5m9MWe3Hzez5/x0fDprX683s4R/d/cI8Js5aVlvZk/5+33EzD5YzP8wE865nznnbgGeLve+a5GCf3A+DSz1v9y/A1wEfMw558zsNLwcyieAo4B/Ae4ws+as938UL6hFAQfcCezAC8wLyZ97fg+wEjjRf98fcjAgf9Vf3okXfBcCXyiQ9k3AbDNb7uf2Pw6kgF0zPAcTzCwNjAL/DPztFJv+BvB41vsW++n5Z+BNfvpTWds/CrytwL7OArY6557JXugH7iG8OwLM7Gr/hyXf46EC/8/r8QLcwwWOfS8Q85+vxAtIv5v1+t7cNzjnVvpP3+bfRX7bf92Gd7exEO86uipPUM6bTOArwALgrcCxQG/ONufinacT8e5qNgGfw/uxnwX8Rc72ZwBL8K619WZ2pr/8i8AJ/mMV3p1RtqeA3/H/j8uBW83smLyJNjtvis8j7V8TMh3nnB4VegCDwAiQznr8Sdb6dwAv4wXtj2Ytvwb4cs6+Hgd+N2u/H89a91vAC8CcPGnoBu7zn78LeAJYAczK2sbwbolPyNnn/xT4vwwvALwGjAMvAr9ZxPm4AkhMsX4u8EngfVNs8yTQlfX6s8APptj+LODpAuv+FfhWgXX9wOdK+OxvAvoAK7D+3cBD/vM+4GKg3399L/D7uZ+f/9oBHVmvY8Cr2Z89sBtYUeC4SeDiAus+APwi5/o9P+v194Brsl7/OXC7/7zdT9tbstb/HXCD//zpnM9tDTA0xflLAecc7vmf5rM5ExisxL5r6aGcf+V9wDkXzXpcn1nhnPsZ3pfCgO9kvec44NPZuRm8XNmCrG2yc6vHAjucc+NTJcQ5dw/wdeAq4Jdmdp2ZteLlmI8A7s86Xp+/PJ+L8XL7JwNNeMUkd5rZggLbF8U5txe4FrjZzPIWIQG/wiubzzgWL9dYyBvwfnTzeRHIm7v0l78wxX4LMrO/B04B/sD50SaPnwInmtmv4d2t3Awca2ZH42UKthR4Xz4v5Xz2rwAtRaRzvpl9y8yeNbNh4Fa8HH22X2Y9fzXP69zjZF+XOzh4zS7Isy47LRdmFXOm8c5fblqkjBT8A2RmfwY041V6/nXWqmeAv8n50TjCOffNrG1czvaLrYjKX+fc15xzb8cL3CfiVX69iPdFPjnrePOcc4UCyNuAHzrnnnBe+XYf8DxQjsrNWXg/RAsLrH/IT3fGM3hFCYW8FXiwwLq7geVmdmz2QjN7B7AYPwD79R8jBR4P57z3crx6ifc454YLJco59wpwP3AJsN05Nwb8BPgU8JRz7sUp/qdy+QredbTUOdeK9yM+41ZOObLP5WK8axu86yN3HQBmdhxwPbAOOMo5FwW2F0qLec1fC30eIyr2KY6Cf0DM7ES8YpAL8FqY/LWZdfqrrwfW+mXqZmZzzex9ZvaGArv7Gd6Xa4O/bcTMfjvPMX/T3+fr8Ip5RoH9zrkD/jE3ZnLcZrbQzFYVON7PgfeZ2fF++jJlwtsL/K9zzCwCzMarK4hkfqjM7CwzO9XMZvt3If+El7t/tMCx7+Jg2TjAbcCZZvYH/nGOyjqP+Ntuyrcj59zdwI+A75nZyX4aVvj7vNk597i/3Vo3ucVW9uPkrP/zs8B5wFnOuUMqt/O4Fy/gZcr3kzmv8/klMKOK6Cm8Ab9Y0swWUp5WMP/HzI4ws5OBPwYy9RLfAT5rZm80s0V4RUYZc/F+hF4AMK/C+pRCB3Be89dCn0eLc25nvveZ2Sz/Onyd99IiNkUz4Hqn4F95P8zJlfzAD3y3Al91zj3onHsSrwz9FjNrds5tA/4Er4jmV8AAXtlvXs65/XiVcR3ATrzKyj/Ms2krXpD/Fd5t90vAP/jrPuMfp98vArgb+PUCh7wZr0I5CQwDXwM+4Zx7DMDMPmdm2QH3Mrw7i/V4P3av+svAq3j+JrAHr/imA69seLTAsX8IvCVTxOR/0d+LV4H+Ml5Z8dv8dBwDnATcXmBf4FVo/hivmGsUrzimD69Meqb+Fi9H+2TW5/25Kba/Fy8AbynwOp9e4Ca/eOQPDiON2S4HTsM79/8BfL/E/YH3Pwzg/aj+g3Mu00Htcrxr7n+A/wRuybzBOfcIXtPan+L9uP0G8P/KkJZcK/GuvbvwPqdX/bQ0JCtcJCkSTma2BjjJOdczzXb/iFeEcvUM9n0TXpHTe/2iGJG6pOAvksUvEvsUcK/z2vyL1CUFfxGRBqQyfxGRBlT1cWGKdcQRR7totD3oZIiI1JTnn7//RedcoT46E0Ib/KPRdtas2RZ0MkREasrll9uO6bdSsY+ISENS8BcRaUAK/iIiDSi0Zf75NDW9xmmnDTFvXqHOn7Vhz54IDzywiLGx1wWdFBFpUDUV/E87bYjjj38Dc+e2YzOfZS8UnHPs3fsSMER//5uDTo6INKiaKvaZN2+UuXOPqtnAD2BmzJ17VM3fvYhIbaup4A/UdODPqIf/QURqW80FfxERKZ2Cfxn98If/xhlnnMyiRbN48EF1UBOR8FLwP0w/+UmSnp7uScve8pZTuP7677Nixcp8bxERCY2aau0zU/v3wz33wPbtcMop8K53wezZlTvekiVvrdzORUTKqG6D//79cN558ItfwCuvwBFHwKmnwje+UdkfABGRWlCW4G9mNwJnA7udc4fMvWle85Yr8abbewXods49UI5jF3LPPV7g37vXe713LzzwgLf8rLMOf79nn72cffv28corI6TTL3PWWd50sZ///FeJxQpNeSsiEi7lyvkn8OabvbnA+tXAEv+xHLjG/1sx27d7Of5sr74KDz9cWvC/886tgFfm/53vJIjHE4e/MxGRgJSlwtc5twVv8uxCzgFudp5+IOpPrl0xp5ziFfVke/3r4eSTK3lUEZHaUK3WPguBZ7JeD/nLJjGzNWa2zcy2vfLKCyUd8F3v8sr4jzgCzLy/p53mLa+UTZt+wNvfvoj77/8pF174Ps47T8VAIhJO1arwzdel9ZDJg51z1wHXASxYsKykyYVnz/Yqd++5xyvqOfnk8rb2eec7Y7zznbFJy1av/iCrV3+wPAcQEamgagX/IeDYrNeLgOcqfdDZs73y/VLK+EVE6lG1in3uAC40zwpgj3Pu+SodW0REcpQl+JvZN4GfAr9uZkNmdpGZrTWztf4mdwFPAwPA9cAny3FcEQmHjoE+cH5JrXPeawm1shT7OOc+Os16B/xZOY4lUivi8aBTUDk9PQefdwz0ccFtq+lffgl9qzbStflSVmy9klvP38RAR1dwiZQp1W0PX5GgJBJw61CMTx1I0dQUdGrKb2wMrt7QQ9+KXmIxGDhhFf3LL2HF1itZsfVKAPqXX8LACWrtFmYK/iJFSiQgnZ5+u4eG25nflCbSFoXu7konq+oiiQTrhuKctyXB0tQgYMTdRtJcObFN16Mb4TGvkV9nJ8RigSRVpqDgX4KBgce49NI/Zvv2B/jMZ/6GtWv/ctr3NI/uYV9zq/fCOTqe2qxb45BLJiGV8oL6kbPSzJnmWxNpZXK5SL3p7qYlmaQllWLXaBScY87+yTPTvbg3wvjsCOP7jZe3RFmaGqzrU1KLFPxLEI0eyZe//DX6+m4vavvm0T0c9fKTjMydD6Cy0RBJJqGrvzfvuivGkyw9kKK13oP6TMRiEIsRARgYgNtug+XLYdUq2LyZOVu3MucjH4ahIVpSKXYOR7l6w+Rzlyk2kmDUdfBv/nEf+2KrvC6+ztGc3My+M8oXZI8+ej5HHz2fH/3oP4rafl9zKyNz59OydzfRPS/y1q1Xqmy0ypLJQ5elUnDnSIxOUrS0FHijyi4K6+iA88+HE07wvmurVnnLMo9YjNZEgvXpgzXgIyNw+n1Jzk4l6ew8dJc61ZVXt8G/+cd9HHXBakYuuoThyzfS+sVLabnhSl66dVNZfwBmxIzh1mNp2bt7YlHfqo3eF0YqKh73AnxXniv+pDGvYjayQgH+sHV0HHxuNvk1HFL30ZJMsqzfuyN4pP/Q6D9+H5zdktSNVgXVbfDfF1vFyEWX0HLDlbTc4FVEjVx0iXcnEBTnaB1+ZtKirs2X6gegzLJz94ODB1vetLYC0Wied9RnxWyoxWJEYjEiiQQrGDx0fTrNzuEoD13RyQWLkrS3T3qrlEHdBn/MGL5840TgBxi+vPQgm0hcxW23XQ/ALbfcRVvbgqLf27xvmJa9uxmZO5/0a3MnmscNdHSpzL9MMjn89qyAEiVNy2IF+FCa4jNpTSToHEpx31C7N0AMcOAAqjwuk/oN/s7R+sVLJy1q/eKlJf8AdHf/Gd3dh9dfbV9kHi8ducRr7ZN+jL5VGxX4S5BMejn7jLW7etmxL05zs1+Ek6GsYm3q7qYFr4goY7T/YOWxKoxLU7fBvzm5mZYbrjykzH/fGV1lK/PfvXsXq1cvY2RkmFmzZnH99XGSyUd4wxtaC75nX2TewRdmCvyHKdORqp3BiaaXLeNpWo5TDr/uZEX4TFFRdj+DMNwFdAz0eQ03/MYltdCEu26D/74zurzKXb+1z/DlG8sa+AHmz2/j/vuHyrY/yS/Tzj5j/WgvDyqH37iy+hlk6gXObklOrK52w6xaHd6iboM/MDnQmwXXykcOWyaHvzRnqISIcviNze9n0JpIsGyX39kMGB+n6p3KanV4i7oO/o2kFm8784nHvZx9xpaxhDdUgpphSj7d3V5Hs4xkkjn9KXbsiXJNteoFzOhbtXEi8ENtNOGu1nj+ZeNcSRN8hUK5/4fMbWfX5kvBObo2X8oFt62uqWF1Ewk484oYO4ejrI/EJx6LW9NE1vco8EtxYjEi63uYd1yUdeNxbt7SXvnRVf3vXLbMdzHMairnv2dPhL17X2Lu3KOwkP+qFuKcY+/el9izJzL9xkWqxdvOfDn8I2epSaaUSU69wNUberi2rbcil1bHU5tZ4ffWzy7zD3tLPgtrTnrBgmVuzZptk5Y1Nb3GaacNMW/eaIF31YY9eyI88MAixsZeV76dOkfvlw7eyPV+4UAobzszY+h8cizu9arN/g0MQ7MNqT+JBCNDaV4+EGVpa2XqAsJU7Hr55Xa/c27ZdNvVVM5/bOx19Pe/OehkhE+B286wlTvG4wdHxlQOX6om018gHp/Ua7icl9+kQF8jTbhrKvhLfmG87ZxyiIUKtcXbf8DYNLCEXzzfxqnH7GJ1x5PMnhXOO1sJQE/PRK/hLTvbWRofbOjx+mqq2EcKC9NtZyaHPyurOUGUNC2LKpfb33/AWHXrH7H12YXsHXsdc5teY/nCZ9l8wS36AZBDxeMMD8NDs8p/FxC0Yot9aq61j+Q30NF1sIgnoNvOZBIiG3rZsSfK/KY0i05vn3i0XNZT0WKeTQNL2PrsQkbGmnHMYmSsma3PLmTTwJKKHVNqWE8PrSs7WTbHuwuIbOjNO9x3PVOxj5TFpPL8ADpg/eL5NvbmVKDvHXsdqV1tnH3iE1VNi9QIf2TRxfE4nxyOc/p9SS4YrK+7gKko+MuM5RtuYedYvKLl+dM59ZhdzG16jZGx5ollc5teo7NtV9XTIjWmp4dWf36BB3dUsXNYwBT8ZUYyQyZ3kpo0l20k4NY7qzueZPnCZw8p81/d8WRgaZIakplfIB5n3Uh8Ypaxem59rOAvRcm0z580ZHKIskazZzk2X3ALmwaWkNrVRmebWvvIYejpmZhlrKpDRARArX1kWpPK8yvYYkckVKrQOawS1NpHSpZIeK13dg57rXdaTu9U4JfG0d1Ny+mdzG/yppSstxZBKvaRvNQbV4RJcw2vG6qvFkEK/jJJZvz8id64tXKvK1JJ3d20JBK0pINOSPmo2EcmxOOwZWc7naRoXRxV4BfJ0TGSYu2u+ij+UfCXSWPpL25NV7w3rkhN6u6mpQXWjce54r4YiUTQCSqNgn8DywzHsGVnO8vmKLcflP0HjDufOJEv37uSO584kf0HwjMSq+To6aHldG9YiFq/A1CZf4PKnhtXZfvB0YB0NSgWI5JKUb7pmIKhnH/AOgb6Dk735lzFp17M5PYf3BH1cvsrOxX4A6QB6WrXx4dru/hHwT9A1Z57N5GAK+6LsW48zrx5aG7cEJhqQDoJsZ4eWhdH6cQbFTQep+aKgFTsE6Bqzb2bGZrhwZAOzdDINCBdDcuaIayXXm6nN+gUzYiCf5DM6Fu1cSLwA2WfejHooZZlahqQToJSlmIfM+sys8fNbMDM1udZ321mL5hZyn9cXI7j1rwCc+9ShvGWcidW0dAM4ZQZkO6b536PL52R5Jvnfk+VvbUmGuXikTg3b/GKf2pFyTl/M5sNXAWcBQwBPzezO5xzj+Rs+m3n3LpSj1dPKjX3rnL7tWX2LMfZJz6hSWdqVXc3Lckkc/pTbB9p55R4bQwCV45in3cAA865pwHM7FvAOUBu8JccAx1d3Hr+pom5d/tWbSwp8CcSsHZX8BOriDScWIwI0JIaDDolRStHsc9C4Jms10P+slznmtlDZvZdMzs2347MbI2ZbTOzba+88kIZkhZ+5Zp7N5n0Av+68fjBzloK/CJVFRlNc+dIbTT/LEfwz1c7mVtg+UOg3Tm3FLgbuCnfjpxz1znnljnnlh1xxJvKkLTGEolASwsq5hEJQixGZIXX+/fBHeEfArocwX8IyM7JLwKey97AOfeSc26f//J64O1lOK74Mk05Pz5cQ7VNIvUoFiOyvod586Arkpw013XYlKPM/+fAEjN7M/As8BHgvOwNzOwY59zz/sv3A4+W4bjC5Dl1Ne6+SEh0dtIe8vL/koO/c27czNYBm4HZwI3OuYfN7EvANufcHcBfmNn7gXHgZaC71OM2urDPqSsi4VaWTl7OubuAu3KWfSHr+WeBz5bjWKKmnCJSOvXwrSGZ3P7OsThNTcrti8jhU/CvEZkhmFW2LyLloOBfQ1paoCWqwC8ipdOQzjUg04GrYyTE7cYahGbdkmJFRtOsHw1vW3/l/ENuUnFPAB249h8wNg0s4RfPt3HqMbtY3fFkww46plm3pGixGJHBQdYNxTlvS4KlqfCN96PgH2KZwL9sTiqQyl0Fu8myZ90CJs26pUHZ5BBZA77dORLjgkQyVCW2KvYJuZYWiLRFA2nVoykGJ9OsWzJjsRiRtqh31x4yCv4hlcn1B1nOr2A3WWbWrWyadUtqlYp9QiaZhFTK68Q1vynt5foDulfUFIOTHe6sW6o3kTBS8A+RTG5/6YGUNx5/wDVEmmJwssysW5sGlpDa1UZn2/SBXPUmElYK/iGQ6bm7ZSzh5fZD0nP3cIJdvZvprFuqJJawUvAPiUgEFkfSgef2c2mKwdJMVW+icypBUoVvCAwOBp0CqRRVEktYKecfsOwROlkUDTo5UmaqN5GwUvAPUDwO20faWdQavuIeKQ/Vm0hYKfgHrKUF6OwMOhlSQao3kTBSmb+ISANS8BcRaUAq9glIpqI30pQOOiki0oAU/Kss04v3UyHpxSsijUnBv8rSaVi0CFrRjFwiEhyV+YuINCAFfxGRBqTgX0XJJKwf7SWaHgw6KSJSLek0o6NBJ+JQKvOvkknDNUeAblX0itS1RILRXWl2j0Xpak3S2R50giZT8K+CoOfilfLS5CwyrWSS0V1pto13csHiJD3dQSfoUAr+VdLSApFoMHPxSvlochYp1mgkytmEM/CDyvxFZkST2ku9UPAXmQFNai/1QsFfZAY0OYvUCwV/kRnITM7S0rQP4wAtTfs0OYvUJFX4Vtikmbqimqmr1mlyFplWPM7ICGynk+iioBNTmIJ/hSST0NXfy86xuAZwqzOanEXySiYZ7U+xbx9c09xD34peumNBJ6owBf8KGRyEtjZoTaPAL9IgRiNR2iOD9PRALOjETENl/iIiDUjBX0SkASn4i4iUQyrFyEjQiSheWYK/mXWZ2eNmNmBm6/Osbzazb/vrt5pZezmOKyISuGSS0Q1xdg5HuXVON52dQSeoOCUHfzObDVwFrAZOAj5qZiflbHYR8CvnXAewEfhqqccVEQmFVIoUnaxcPMjo+t6aGb6rHDn/dwADzrmnnXNjwLeAc3K2OQe4yX/+XeDdZmZlOLaISOD6IzHa24NOxcyUI/gvBJ7Jej3kL8u7jXNuHNgDHJW7IzNbY2bbzGzbK6+8UIakiYhIPuUI/vly8LndHYvZBufcdc65Zc65ZUcc8aYyJE1ERPIpRyevIeDYrNeLgOcKbDNkZnOAecDLZTi2iEgwEglGhtK8fCBKb2svNVLPO6Ecwf/nwBIzezPwLPAR4Lycbe4APgb8FPgQcI9zToOhiEhtiscZHoaHZoV3pq7plBz8nXPjZrYO2AzMBm50zj1sZl8Ctjnn7gBuAG4xswG8HP9HSj2uiEiQbmzt4fbOcI/fM5WyjO3jnLsLuCtn2Reyno8CHy7HsUREApVMMjoKo0Gno0Qa2E1CQZOiS03IKu65dlHt5vpBwV9CQJOiS01IJBgZgaubwj9cczE0to8ETpOiS60YaOnk2rba6cU7FQV/CZwmRZfQ85t11tLAbdNRsY8ELjMp+shY88QyTYouoeDPzjU2Vj/FPRnK+UvgNCm6hNbgIGmiLG5N19SgbcVQzl8Cp0nRJcx2RdqJRoNORfkp+Eso1Mqk6GqS2kD8Zp0jswAFf5HGpSapDSJrzJ6lrd5k7N1Bp6kCVOYvUiQ1SW0A8TjDO9MTk7P09ASdoMpR8K+gXbtgdBRIJoNOipSBmqTWsWQS4nGGRrzK3bsvS9LdHXSiKkvBv0K6u+Hatl62jXcyvCUF8XjQSZISZZqkZlOT1DoSjfLdltqZg7dUCv4V1N0Nl52e5OqmHq9zSCIRcIqkFGqSWsdSKYaG/Dv1BqEK3wqLxeDawV660kk6GQw6OVICNUmtQ34nrt1jUb7R1E3fivpqyz8VBX+RGaiVJqlShJwJWbq7IRZ0mqpIwV9EGktOU87OTupmyIaZUPAXkcZRB9MvlouCv4jUvwbpuDUTau1TBe3t0DcaY3hnmtENcbX7FwnAQEv9d9yaCQX/KojFYHR9L4tb0+weizLan1KzTylo/wHjzidO5Mv3ruTOJ05k/wELOkm1L51uqGacxVCxTxX19MDS+CCpaIx2NfuUPDR+UJn5TTn37YNkc4z29qATFB7K+Qdg1y4gnQ46GRJCGj+ojOJxhrek2DbeyduOq7/x+EulnH+V9fRAVzzJQ8PtHHlFnJZFUep+EBEp2lTjB6lvQZESCUZ3eUWsjdyUczrK+QegpwdWLh4kRSeju9KqAJYJGj+oRP6onNvGD1buKrefn3L+AenuhrPjSQZpJxJ0YiQ0MuMH5Zb5a/ygaagp54wp+AdseBgi/SnvB0BZlIan8YNmKJmEVEodtw6DORfOi2rBgmVuzZptQSej4hIJuHUoxtIDKVpbQY2QRYqUJ7cvcPnldr9zbtl026nMP2CZYZ9vbNWVK1KUZJLRDV7Z/tfn9HDhSgX+w6HgHyKa9UtkGokEI/d5QzAvblXzzVIo+IdALAa99LJ7LMrIfer9K3IIP7e/Z4dy++Wi4B8SPT1w4cpBvj6nhz07NAaQyAQ/t79tvJPj5im3Xy4K/iGSGQPouHleO2XdBUhDy8ntX3Z6Urn9MlJTzxDq6YHLkkm6+ntZNxSnJZFQL2Cpf8kkDA56z9Ppg803j2u8WbaqQcE/pDT3rzQMf/C1sTF4pKkTgPFxuK8pRt+KXg3NUCEK/iE3MgIjI2nl/qU+5ZlSMSMWU26/khT8Q6y7Gy5IJFm7q5dP7ozTGo9DZ6d6Akvtyxpq+ZrmHvpW9NITCzpRjUUVviHX3X1wIpidw2oKKnUgHp9ovaOhloNTUvA3syPN7L/M7En/7xsLbLffzFL+445SjtmospuCajpIqUmZ1jt7mGi9o5LM4JQ0to+Z/R3wsnNug5mtB97onPtMnu1GnHMtM9nJUTflAAAKpUlEQVR3o4ztczjicbz5AGalNR+AhJc/rn7G2JjfemeRgn4lVWtsn3OAm/znNwEfKHF/UgR1CJNQy2qfH6eHtkiatkiaxa1p5fZDpNScf9o5F816/Svn3CFFP2Y2DqSAcWCDc+72AvtbA6wBmDdv8dt7enYcdtoaRWZU0E5StLSgUUElWPE4IyOQopOzW9QpKwjF5vynDf5mdjfQlmfV54Gbigz+C5xzz5nZ8cA9wLudc09NdVwV+xQvmYSu/l7+dF+c5maIrFCLIKmCnLvN3NY7ugSDUWzwn7app3PuzELrzOyXZnaMc+55MzsG2F1gH8/5f582syRwKjBl8JfixWIwGuvlbYleb26ALSlaUyndBUjl+O3zR+Z4eb/xcRhEvXFrSant/O8APgZs8P/+e+4GfgugV5xz+8zsaOC3gb8r8biSR3f3wWEhzhtOMH9DnEhb9OAG7e26I5DDk9W8eHRXeiKHf21b78Ty9nZNlF5LSg3+G4DvmNlFwE7gwwBmtgxY65y7GHgr8C9mdgCvgnmDc+6REo8rBWTuAlb6dwFzsub9PmUoRcvgoFoHSfH8HH6aKENz2gEvl3/2vKTmya1xmsaxjuU2AFK9gEwruwOhP7ja1U1eDr+9/eAqXTrhVbYyf6lduV9Q1QtIQX6b/N1jUXbOap9YfHarcvj1SsG/wWTXC/zpnjjNG+JEIlkbaOygxhCPT3o5MXzy4snt8JU1qF8K/g0ou3XQ2l29MOotP28swZH3qV6gruUMqJaxobVXOfwGozJ/mZDbXyAjEkF3BLVKQyw0HJX5y4xl3xGkD8YLbxwh3RHUlpwc/oZIr7c84v2Oq0mmKPjLIXLj+4XJQbr6vTkFZl1xsKxYw0kEzM/Vj4/nX53K6nSlT0lyKfjLtDJ3BEsTvRPL0mn/juCKuDeyaHY7wNw3y+ErNGBfKnWwknZR/m3U6UqmouAvRcu9I1iZGOTWoRjtQ4PM2TV4yPYt42kVFR0uv9hmfJyJIRSyjY150x6qklYOl4K/HLbubribZMGJxdbu6uVPd/jNSdsODWATO2kkySQMDk67Wfa8ttE8p669HU17KCVR8JeSFYrfo/RyXLyXO0cmDzORsWh8kKM3xBunt3GeoRLyGR+H++bENK+tVJSCf8h1DPQxcMIqMAPn6HhqMwMdXUEnq2g9PV6nsnwGB5nc27hU0QrMapZMQhnSNjrKpOGOpxOL1fbImLV+3TYCBf8Q6xjo44LbVtO//BL6Vm2ka/OlrNh6Jbeev6mmvkhTZeovSyZJpWD9aG/Jx5koYirXnYSfU3/5QJRvNHWXvLsN87yOVLGS9xRu9XLd1jt18goz5ya+OBmZLxRmASYsnDKzmi09kKK1tbR9jY56HaKubtLEJDOm6zZQZZvJKygK/j7n6P3SwamWe79wQF+gKWRKacpxJ7Eh0qtuDIdL121g1MO3Hvg5qGxdmy9VDmoKsZjfL4HekveluH+YdN3WhFnTbyJB6XhqMyu2Xkn/8kvo/cIB+pdfwoqtV9Lx1OagkyZSkK7b2qBin5BTqwmpRbpug6Ninzox6Qtjpi+Q1ARdt+GnYh8RqQsdA32QKclwznstBSn4i0jNy/Qt6Np86USF8wW3rdYPwBRU7CMiNW/ghFUTFcuZ/gX9yy/x6h0kL+X8RaT2mXlNSbOoaenUFPxFpPYV6FtASFszhoGCf8io0kpk5tS3YOYU/ENElVYih2ego4tbz980UdTTt2qjBpKbhip8Q0SVViKHT30LZkY5/zBRpZWIVImCf5io0kpEqkTBP0TKUWmlCmMRKYaCf4iUWmmlCmMRKZYqfEOmlEorVRiLSLGU868nqjAWkSIp+JcoVGXsqjAWkSIp+JcgbGXs6uUoIsVSmX8JwlbGnqkwzsyg1LdqIwMdXersIiKHUM6/FCEsYx/o6Dp4fPVyFJECSgr+ZvZhM3vYzA6YWcE5I82sy8weN7MBM1tfyjFDRWXsIlKjSs35bwd+H9hSaAMzmw1cBawGTgI+amYnlXjcUFAZu4jUqpLK/J1zjwLY1MUc7wAGnHNP+9t+CzgHeKSUY4eBythFpFZVo8x/IfBM1ushf1ldUBm7iNSiaXP+ZnY30JZn1eedc/9exDHy3RbkLRQ3szXAGoB58xYXsWsRETkc0wZ/59yZJR5jCDg26/Ui4LkCx7oOuA5gwYJlqjUVEamQahT7/BxYYmZvNrMm4CPAHVU4roiIFFBqU88PmtkQ8FvAf5jZZn/5AjO7C8A5Nw6sAzYDjwLfcc49XFqyRUSkFKW29vkB8IM8y58D3pv1+i7grlKOJSIi5aMeviIiDUjBX0SkASn4i4g0IAV/EZEGpOAvItKAFPxFRBqQgr+ISANS8BcRaUAK/iIiDUjBX0SkASn4i4g0IAV/EZEGpOAvItKAzLlwzpliZi8AO6p82KOBF6t8zFqg85KfzkthOjf5VeO8HOece9N0G4U2+AfBzLY555YFnY6w0XnJT+elMJ2b/MJ0XlTsIyLSgBT8RUQakIL/ZNcFnYCQ0nnJT+elMJ2b/EJzXlTmLyLSgJTzFxFpQAr+IiINqKGDv5l92MweNrMDZlaw+ZWZdZnZ42Y2YGbrq5nGIJjZkWb2X2b2pP/3jQW2229mKf9xR7XTWS3Tff5m1mxm3/bXbzWz9uqnsvqKOC/dZvZC1jVycRDprDYzu9HMdpvZ9gLrzcy+5p+3h8zstGqnERo8+APbgd8HthTawMxmA1cBq4GTgI+a2UnVSV5g1gM/cs4tAX7kv87nVedcp/94f/WSVz1Ffv4XAb9yznUAG4GvVjeV1TeD78W3s66Rf61qIoOTALqmWL8aWOI/1gDXVCFNh2jo4O+ce9Q59/g0m70DGHDOPe2cGwO+BZxT+dQF6hzgJv/5TcAHAkxL0Ir5/LPP13eBd5uZVTGNQWjE70VRnHNbgJen2OQc4Gbn6QeiZnZMdVJ3UEMH/yItBJ7Jej3kL6tnv+acex7A/zu/wHYRM9tmZv1mVq8/EMV8/hPbOOfGgT3AUVVJXXCK/V6c6xdtfNfMjq1O0kIvFDFlTrUPWG1mdjfQlmfV551z/17MLvIsq/n2sVOdlxnsZrFz7jkzOx64x8z+2zn3VHlSGBrFfP51eY1Mo5j/+YfAN51z+8xsLd7d0bsqnrLwC8X1UvfB3zl3Zom7GAKycyyLgOdK3GfgpjovZvZLMzvGOfe8fzu6u8A+nvP/Pm1mSeBUoN6CfzGff2abITObA8xj6tv+ejDteXHOvZT18noaoC6kSKGIKSr2md7PgSVm9mYzawI+AtRtyxbfHcDH/OcfAw65QzKzN5pZs//8aOC3gUeqlsLqKebzzz5fHwLucfXfe3La85JTjv1+4NEqpi/M7gAu9Fv9rAD2ZIpZq8o517AP4IN4v8L7gF8Cm/3lC4C7srZ7L/AEXq7280Gnuwrn5Si8Vj5P+n+P9JcvA/7Vf/5O4L+BB/2/FwWd7gqej0M+f+BLwPv95xHg34AB4GfA8UGnOSTn5SvAw/418mPgLUGnuUrn5ZvA88Brfny5CFgLrPXXG15Lqaf8786yINKp4R1ERBqQin1ERBqQgr+ISANS8BcRaUAK/iIiDUjBX0SkASn4i4g0IAV/EZEG9L93B/S3FIlr6AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "reg_type = None\n",
    "lambda_t = 1\n",
    "Q = 2\n",
    "lr2 = lm.LinearRegression('pesudo_inv', reg_type, reg_param = lambda_t, \n",
    "                          poly_degree = Q, to_classify=True)\n",
    "lr2.fit(X_train, y_train)\n",
    "\n",
    "Q = 3\n",
    "lr3 = lm.LinearRegression('pesudo_inv', reg_type, reg_param = lambda_t, \n",
    "                          poly_degree = Q, to_classify=True)\n",
    "lr3.fit(X_train, y_train)\n",
    "\n",
    "plot_813(df, lr2, \"Exercise 8.13 (c) Q=2 with lambda = 1 \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XucXHV9//HXJ4m7I1k2I2AakhBW2VAFGhdMTWppOig0G6Wioq1CiluhMdW0LNrWWPjpovRntJcMVoFC8THcFK23IiWbFnHIg59uNOBAAwgsdBMWiOHiZLuBzbLJ9/fHObOZnczszu5czpmZ9/PxmMfOucw53z0z85nv+V7NOYeIiDSWWUEnQEREqk/BX0SkASn4i4g0IAV/EZEGpOAvItKAFPxFRBqQgr+ISANS8K9zZnahmf1n0OkoJzP7mJnFi9jvn8xsfTXSVClTvX9mFjOzwWkcL2lml5QndROO22ZmzszmFLn/tNIt5afgX0FmNmBmr5jZcNbjq9VMg3PuNufcH5T7uGb2F2b2P2Y2ZGY7zOzMSfbd4O9zwMwSOdtO8bf92n/cbWanTHKsJuAK4O+LSObfA5f7ryl0vGYz+6KZ7fbfqyfM7K/MzIo4fu6xvmxmT/vXZJeZXT7dY+TKff/8ANte6nEbkZmdZWY/NrN9ZjYQdHqCpuBfeX/onGvJemwo58GLzWmV+ZwrgE3AB4B5wI3A981sdoGXPAtcBXy9wLYPAMcAxwF3ALdPcvrzgF86556ZKp3OueeAXwLvmWS3fwPeCbwLOBr4E+BjwD9Odfw8bgTe5JxrBd4OXGBm75/BcaQy9uN9Bv866ISEgYJ/QMzsWjP7Ttbyl8zsR5kcp5mda2YpM0ub2U/MbFnWvgNm9mkzewjYb2ZzzOwEM/uemT1vZi9m7jDMrMvM7vOfm5ltNrO9fu7nITM7zd/WbGb/4OeAf2Vm15nZawskvw142Dl3v/PGB7kZL3DPz7ezc+57zrkfAC/m2ZZ2zg34xzHgIDBZznYNcG/OtTzTv0ZpP+fdlbU5Cbw734HM7J3AHwDnO+d2OufGnHN9wFrgUjN74yTpOIJz7jHn3P6sVYcK/S9mdq+ZnZ+Vfmdm7/KXzzazlP88+/3b5r/8Qf8u8o+zjvcp/319zsz+tJj0mtlJZnaP/3l5wcxuM7No1vYBM/tr/3Oy38xuNLPfMLMtZva//l3a63IO+1Eze9ZPx6eyjvVaM0v4d3ePAL+dk5aNZvakf9xHzOx9xfwP0+Gc+5lz7hbgqXIfuxYp+AfnU8Ay/8v9e8DFwEecc87MzsDLoXwMOBb4F+AOM2vOev2H8YJaFHDAncAuvMC8iPy55z8AVgEn+6/7Yw4H5C/56zvwAtYi4LMF0r4FmG1mK/zc/keBFLBnmtdgnJmlgRHgn4H/O8muvwU8lvW6JX56/hl4vZ/+VNb+jwJvKXCsc4Dtzrmns1c657YDg3h3BJjZNf4PS77HQzn/x0YzG/ZfPxf4RoFz3wvE/Oer8ALS72ct35v7AufcKv/pW/y7yG/5ywvw7sAW4X2OvpYnKOdjwBeBhcCbgROAnpx9zse7TicDf4h3rf8W78d+FvCXOfufBSzF+6xtNLOz/fWfA07yH6uBj+S87kng9/z/40rgVjM7Pm+izS6Y5P1I+58JmYpzTo8KPYABYBhIZz3+LGv724CX8IL2h7PWXwt8IedYjwG/n3Xcj2Zt+x3geWBOnjR0Aff5z98BPA6sBGZl7WN4t8Qn5Rzzfwr8X4YXAF4FxoAXgN8u4npcBSQm2T4X+Djw7kn2eQLozFr+DPD9SfY/B3iqwLZ/BW4vsK0P+NsZvu8GnI4XxI4usM87gYf8573AJUCfv3wv8P7c989fdkB71nIMeCX7vQf2AisLnDcJXFJg23uBX+R8fi/MWv4ucG3W8l8AP/Cft/lpe1PW9i8DN/rPn8p539YBg5NcwxRw3kyufxHvz9nAQCWOXUsP5fwr773OuWjW44bMBufcz/C+FAZ8O+s1JwKfys7N4OXKFmbtk51bPQHY5Zwbmywhzrl7gK8CXwN+ZWbXm1krXo75KOD+rPP1+uvzuQQvt38q0IRXTHKnmS0ssH9RnFdkch1ws5nlLUICfo1XNp9xAl6usZCj8X5083kByJu79Nc/P8lxC3KeX+AF5SsL7PZT4GQz+w28u5WbgRPM7Di8TMG2Aq/L58Wc9/5loGWqF5nZfDO73cyeMbMh4Fa8HH22X2U9fyXPcu55sj+Xuzj8mV2YZ1t2Wi7KKuZMA6flSYuUkYJ/gMzsE0AzXqXn32Rtehr4u5wfjaOcc9/M2sfl7L/Eiqj8dc59xTn3VrzAfTJe5dcLeF/kU7PON885VyiAvAX4oXPucefcIedcL/AcXiVnqWbh/RAtKrD9IT/dGU/jFSUU8mbgwQLb7gZWmNkJ2SvN7G3AEvwA7Nd/DBd4PDzJuecUSptz7mXgfuBSYKdzbhT4CfBJ4Enn3AuTHLdcvoj3OVrmvErqtXgZkVJkX8sleJ9t8D4fudsAMLMTgRuADcCxzrkosLNQWsxr/lro/RhWsU9xFPwDYmYn4xWDrMVrYfI3Ztbhb74BWO+XqZuZzTWzd5vZ0QUO9zO8L9cmf9+Imf1unnP+tn/M1+AV84wAB51zh/xzbs7kuM1skZmtLnC+nwPvNrM3+unLlAnvLPC/zjGzCDAbr64gkvmhMrNzzOx0M5vt34X8E17u/tEC576Lw2XjALcBZ5vZH/nnOTbrOuLvuyXfgZxzdwM/Ar5rZqf6aVjpH/Nm59xj/n7r3cQWW9mPU/3/Y5Z5/Q9e51+TtwGf8I9fyL14AS9Tvp/MWc7nV8C0KqIncTR+saSZLaI8rWD+j5kdZWanAn8KZOolvg18xr8+i/GKjDLm4v0IPQ9gXoX1aYVO4Lzmr4Xejxbn3O58r/PfowjwGm/RIjZJM+B6p+BfeT/MyZV83w98twJfcs496Jx7Aq8M/RYza3bO7QD+DK+I5tdAP17Zb17OuYN4lXHtwG68ysY/zrNrK16Q/zXebfeLwD/42z7tn6fPLwK4G/jNAqe8Ga9COQkMAV8BPuac+yWAmf2tmWUH3Cvw7iw24v3YveKvA6/i+ZvAPrzim3a8suGRAuf+IfCmTBGT/0V/F14F+kt4ZcVv8dNxPHAK8IMCxwKvQvPHeMVcI3jFMb14ZdLT9T7/f/hfvPf3n/1HIffiBeBtBZbz6QFu8otH/mgGacx2JXAG3rX/D+B7JR4PvP+hH+9H7x+cc5kOalfifeb+B/hP4JbMC5xzj+A1rf0p3o/bbwH/rwxpybUK77N3F96dxyt+WhqS+RUgIjXDzNYBpzjnuqfY7x/xilCumcaxb8IrcnqXXxQjUpcU/EWy+EVinwTudV6bf5G6pOAvItKAVOYvItKAqj4uTLGOOuo4F422BZ0MEZGa8txz97/gnCvUR2dcaIN/NNrGunU7gk6GiEhNufJK2zX1Xir2ERFpSAr+IiINSMFfRKQBhbbMP5+mplc544xB5s0r1PmzNuzbF+GBBxYzOvqaoJMiIg2qpoL/GWcM8sY3Hs3cuW3Y9GfZCwXnHPv3vwgM0tf3hqCTIyINqqaKfebNG2Hu3GNrNvADmBlz5x5b83cvIlLbair4AzUd+DPq4X8QkdpWc8FfRERKp+BfRj/84b9x1lmnsnjxLB58UB3URCS8FPxn6Cc/SdLd3TVh3ZvedBo33PA9Vq5cle8lIiKhUVOtfabr4EG45x7YuRNOOw3e8Q6YPbty51u69M2VO7iISBnVbfA/eBAuuAB+8Qt4+WU46ig4/XT4xjcq+wMgIlILyhL8zezrwLnAXufcEXNvmte85Wq86fZeBrqccw+U49yF3HOPF/j37/eW9++HBx7w1p9zzsyPe+65Kzhw4AAvvzxMOv0S55zjTRd7+eVfIhYrNOWtiEi4lCvnn8Cbb/bmAtvXAEv9xwrgWv9vxezc6eX4s73yCjz8cGnB/847twNemf+3v50gHk/M/GAiIgEpS4Wvc24b3uTZhZwH3Ow8fUDUn1y7Yk47zSvqyfba18Kpp1byrCIitaFarX0WAU9nLQ/66yYws3VmtsPMdrz88vMlnfAd7/DK+I86Csy8v2ec4a2vlC1bvs9b37qY++//KRdd9G4uuEDFQCISTtWq8M3XpfWIyYOdc9cD1wMsXLi8pMmFZ8/2Knfvuccr6jn11PK29nn722O8/e2xCevWrHkfa9a8rzwnEBGpoGoF/0HghKzlxcCzlT7p7Nle+X4pZfwiIvWoWsU+dwAXmWclsM8591yVzi0iIjnKEvzN7JvAT4HfNLNBM7vYzNab2Xp/l7uAp4B+4Abg4+U4r4iEQ3t/Lzi/pNY5b1lCrSzFPs65D0+x3QGfKMe5RGpFPB50Ciqnu/vw8/b+Xtbetoa+FZfSu3oznVsvY+X2q7n1wi30t3cGl0iZVN328BUJSiIBtw7G+OShFE1NQaem/EZH4ZpN3fSu7CEWg/6TVtO34lJWbr+alduvBqBvxaX0n6TWbmGm4C9SpEQC0ump93toqI35TWkiC6LQ1VXpZFVdJJFgw2CcC7YlWJYaAIy420yaq8f36Xx0M/zSa+TX0QGxWCBJlUko+Jegv/+XXHbZn7Jz5wN8+tN/x/r1fzXla5pH9nGgudVbcI72J7fq1jjkkklIpbygfsysNHOm+NZEWplYLlJvurpoSSZpSaXYMxIF55hzcOLMdC/sjzA2O8LYQeOlbVGWpQbq+pLUIgX/EkSjx/CFL3yF3t4fFLV/88g+jn3pCYbnzgdQ2WiIJJPQ2deTd9tVY0mWHUrRWu9BfTpiMYjFiAD098Ntt8GKFbB6NWzdypzt25nzoQ/C4CAtqRS7h6Jcs2nitcsUG0kw6jr4N/+4lwOx1V4XX+doTm7lwFnlC7LHHTef446bz49+9B9F7X+guZXhufNp2b+X6L4XePP2q1U2WmXJ5JHrUim4czhGBylaWgq8UGUXhbW3w4UXwkkned+11au9dZlHLEZrIsHG9OEa8OFhOPO+JOemknR0HHlIXerKq9vg3/zjXo5du4bhiy9l6MrNtH7uMlpuvJoXb91S1h+AaTFjqPUEWvbvHV/Vu3qz94WRiorHvQDfmecTf8qoVzEbWakAP2Pt7Yefm01chiPqPlqSSZb3eXcEj/QdGf3H7oNzW5K60aqgug3+B2KrGb74UlpuvJqWG72KqOGLL/XuBILiHK1DT09Y1bn1Mv0AlFl27n5g4HDLm9ZWIBrN84r6rJgNtViMSCxGJJFgJQNHbk+n2T0U5aGrOli7OElb24SXShnUbfDHjKErN48HfoChK0sPsonE17jtthsAuOWWu1iwYGHRr20+METL/r0Mz51P+tW5483j+ts7VeZfJpkcfltWQImSpmWJAnwoTfKetCYSdAymuG+wzRsgBjh0CFUel0n9Bn/naP3cZRNWtX7uspJ/ALq6PkFX18z6qx2IzOPFY5Z6rX3Sv6R39WYF/hIkk17OPmP9nh52HYjT3OwX4WQoq1iburpowSsiyhjpO1x5rArj0tRt8G9ObqXlxquPKPM/cFZn2cr89+7dw5o1yxkeHmLWrFnccEOcZPIRjj66teBrDkTmHV4wU+CfoUxHqjYGxptetoylaTlROfy6kxXhM0VF2f0MwnAX0N7f6zXc8BuX1EIT7roN/gfO6vQqd/3WPkNXbi5r4AeYP38B998/WLbjSX6ZdvYZG0d6eFA5/MaV1c8gUy9wbktyfHO1G2bV6vAWdRv8gYmB3iy4Vj4yY5kc/rKcoRIiyuE3Nr+fQWsiwfI9fmczYGyMqncqq9XhLeo6+DeSWrztzCce93L2GdtGE95QCWqGKfl0dXkdzTKSSeb0pdi1L8q11aoXMKN39ebxwA+10YS7WuP5l41zJU3wFQrl/h8yt52dWy8D5+jcehlrb1tTU8PqJhJw9lUxdg9F2RiJjz+WtKaJbOxW4JfixGJENnYz78QoG8bi3LytrfKjq/rfuWyZ72KY1VTOf9++CPv3v8jcucdiIf9VLcQ5x/79L7JvX2TqnYtUi7ed+XL4x8xSk0wpk5x6gWs2dXPdgp6KfLTan9zKSr+3fnaZf9hb8llYc9ILFy5369btmLCuqelVzjhjkHnzRgq8qjbs2xfhgQcWMzr6mvId1Dl6Pn/4Rq7ns4dCeduZGUPn46Nxr1dt9m9gGJptSP1JJBgeTPPSoSjLWitTFxCmYtcrr7T7nXPLp9qvpnL+o6Ovoa/vDUEnI3wK3HaGrdwxHj88MqZy+FI1mf4C8fiEXsPl/PhNCPQ10oS7poK/5BfG285Jh1ioUFu8g4eMLf1L+cVzCzj9+D2saX+C2bPCeWcrAejuHu81vG13G8viAw09Xl9NFftIYWG67czk8GdlNSeIkqZlceVy+wcPGatv/RO2P7OI/aOvYW7Tq6xY9Axb196iHwA5UjzO0BA8NKv8dwFBK7bYp+Za+0h+/e2dh4t4ArrtTCYhsqmHXfuizG9Ks/jMtvFHyxXdFS3m2dK/lO3PLGJ4tBnHLIZHm9n+zCK29C+t2DmlhnV307qqg+VzvLuAyKaevMN91zMV+0hZTCjPD6AD1i+eW8D+nAr0/aOvIbVnAeee/HhV0yI1wh9ZdEk8zseH4px5X5K1A/V1FzAZBX+ZtnzDLewejVe0PH8qpx+/h7lNrzI82jy+bm7Tq3Qs2FP1tEiN6e6m1Z9f4MFdVewcFjAFf5mWzJDJHaQmzGUbCbj1zpr2J1ix6JkjyvzXtD8RWJqkhmTmF4jH2TAcH59lrJ5bHyv4S1Ey7fMnDJkcoqzR7FmOrWtvYUv/UlJ7FtCxQK19ZAa6u8dnGavqEBEBUGsfmdKE8vwKttgRCZUqdA6rBLX2kZIlEl7rnd1DXuudljM7FPilcXR10XJmB/ObvCkl661FkIp9JC/1xhVhwlzDGwbrq0WQgr9MkBk/f7w3bq3c64pUUlcXLYkELemgE1I+KvaRcfE4bNvdRgcpWpdEFfhFcrQPp1i/pz6KfxT8ZcJY+kta0xXvjStSk7q6aGmBDWNxrrovRiIRdIJKo+DfwDLDMWzb3cbyOcrtB+XgIePOx0/mC/eu4s7HT+bgofCMxCo5urtpOdMbFqLW7wBU5t+gsufGVdl+cDQgXQ2KxYikUpRvOqZgKOcfsPb+3sPTvTlX8akXM7n9B3dFvdz+qg4F/gBpQLra9dGh2i7+UfAPULXn3k0k4Kr7YmwYizNvHpobNwQmG5BOQqy7m9YlUTrwRgWNx6m5IiAV+wSoWnPvZoZmeDCkQzM0Mg1IV8OyZgjroYcf0BN0iqZFwT9IZvSu3jwe+IGyT70Y9FDLMjkNSCdBKUuxj5l1mtljZtZvZhvzbO8ys+fNLOU/LinHeWtegbl3KcN4S7kTq2hohnDKDEj3zfO/y+fPSvLN87+ryt5aE41yyXCcm7d5xT+1ouScv5nNBr4GnAMMAj83szucc4/k7Pot59yGUs9XTyo1965y+7Vl9izHuSc/rklnalVXFy3JJHP6UuwcbuO0eG0MAleOYp+3Af3OuacAzOx24DwgN/hLjv72Tm69cMv43Lu9qzeXFPgTCVi/J/iJVUQaTixGBGhJDQSdkqKVo9hnEfB01vKgvy7X+Wb2kJl9x8xOyHcgM1tnZjvMbMfLLz9fhqSFX7nm3k0mvcC/YSx+uLOWAr9IVUVG0tw5XBvNP8sR/PPVTuYWWP4QaHPOLQPuBm7KdyDn3PXOueXOueVHHfX6MiStsUQi0NKCinlEghCLEVnp9f59cFf4h4AuR/AfBLJz8ouBZ7N3cM696Jw74C/eALy1DOcVX6Yp50eHaqi2SaQexWJENnYzbx50RpIT5roOm3KU+f8cWGpmbwCeAT4EXJC9g5kd75x7zl98D/BoGc4rTJxTV+Pui4RERwdtIS//Lzn4O+fGzGwDsBWYDXzdOfewmX0e2OGcuwP4SzN7DzAGvAR0lXreRhf2OXVFJNzK0snLOXcXcFfOus9mPf8M8JlynEvUlFNESqcevjUkk9vfPRqnqUm5fRGZOQX/GpEZglll+yJSDgr+NaSlBVqiCvwiUjoN6VwDMh242odD3G6sQWjWLSlWZCTNxpHwtvVXzj/kJhT3BNCB6+AhY0v/Un7x3AJOP34Pa9qfaNhBxzTrlhQtFiMyMMCGwTgXbEuwLBW+8X4U/EMsE/iXz0kFUrmrYDdR9qxbwIRZtzQomxwha8C3O4djrE0kQ1Viq2KfkGtpgciCaCCtejTF4ESadUumLRYjsiDq3bWHjIJ/SGVy/UGW8yvYTZSZdSubZt2SWqVin5BJJiGV8jpxzW9Ke7n+gO4VNcXgRDOddUv1JhJGCv4hksntLzuU8sbjD7iGSFMMTpSZdWtL/1JSexbQsWDqQK56EwkrBf8QyPTc3Taa8HL7Iem5O5NgV++mO+uWKoklrBT8QyISgSWRdOC5/VyaYrA0k9Wb6JpKkFThGwIDA0GnQCpFlcQSVsr5Byx7hE4WR4NOjpSZ6k0krBT8AxSPw87hNha3hq+4R8pD9SYSVgr+AWtpATo6gk6GVJDqTSSMVOYvItKAFPxFRBqQin0CkqnojTSlg06KiDQgBf8qy/Ti/WRIevGKSGNS8K+ydBoWL4ZWNCOXiARHZf4iIg1IwV9EpAEp+FdRMgkbR3qIpgeCToqIVEs6zchI0Ik4ksr8q2TCcM0RoEsVvSJ1LZFgZE+avaNROluTdLQFnaCJFPyrIOi5eKW8NDmLTCmZZGRPmh1jHaxdkqS7K+gEHUnBv0paWiASDWYuXikfTc4ixRqJRDmXcAZ+UJm/yLRoUnupFwr+ItOgSe2lXij4i0yDJmeReqHgLzINmclZWpoOYByipemAJmeRmqQK3wqbMFNXVDN11TpNziJTiscZHoaddBBdHHRiClPwr5BkEjr7etg9GtcAbnVGk7NIXskkI30pDhyAa5u76V3ZQ1cs6EQVpuBfIQMDsGABtKZR4BdpECORKG2RAbq7IRZ0YqagMn8RkQak4C8i0oAU/EVEyiGVYng46EQUryzB38w6zewxM+s3s415tjeb2bf87dvNrK0c5xURCVwyycimOLuHotw6p4uOjqATVJySg7+ZzQa+BqwBTgE+bGan5Ox2MfBr51w7sBn4UqnnFREJhVSKFB2sWjLAyMaemhm+qxw5/7cB/c65p5xzo8DtwHk5+5wH3OQ//w7wTjOzMpxbRCRwfZEYbW1Bp2J6yhH8FwFPZy0P+uvy7uOcGwP2AcfmHsjM1pnZDjPb8fLLz5chaSIikk85gn++HHxud8di9sE5d71zbrlzbvlRR72+DEkTEZF8ytHJaxA4IWt5MfBsgX0GzWwOMA94qQznFhEJRiLB8GCalw5F6WntoUbqeceVI/j/HFhqZm8AngE+BFyQs88dwEeAnwIfAO5xzmkwFBGpTfE4Q0Pw0KzwztQ1lZKDv3NuzMw2AFuB2cDXnXMPm9nngR3OuTuAG4FbzKwfL8f/oVLPKyISpK+3dvODjnCP3zOZsozt45y7C7grZ91ns56PAB8sx7lERAKVTDIyAiNBp6NEGthNQkGToktNyCruuW5x7eb6QcFfQkCToktNSCQYHoZrmsI/XHMxNLaPBE6Tokut6G/p4LoFtdOLdzIK/hI4TYouoec366ylgdumomIfCVxmUvTh0ebxdZoUXULBn51rdLR+insylPOXwGlSdAmtgQHSRFnSmq6pQduKoZy/BE6TokuY7Ym0EY0GnYryU/CXUKiVSdHVJLWB+M06h2cBCv4ijUtNUhtE1pg9y1q9ydi7gk5TBajMX6RIapLaAOJxhnanxydn6e4OOkGVo+BfQXv2wMgIkEwGnRQpAzVJrWPJJMTjDA57lbt3X5GkqyvoRFWWgn+FdHXBdQt62DHWwdC2FMTjQSdJSpRpkppNTVLrSDTKd1pqZw7eUin4V1BXF1xxZpJrmrq9ziGJRMApklKoSWodS6UYHPTv1BuEKnwrLBaD6wZ66Ewn6WAg6ORICdQktQ75nbj2jkb5RlMXvSvrqy3/ZBT8RaahVpqkShFyJmTp6oJY0GmqIgV/EWksOU05OzqomyEbpkPBX0QaRx1Mv1guCv4iUv8apOPWdKi1TxW0tUHvSIyh3WlGNsXV7l8kAP0t9d9xazoU/KsgFoORjT0saU2zdzTKSF9KzT6loIOHjDsfP5kv3LuKOx8/mYOHLOgk1b50uqGacRZDxT5V1N0Ny+IDpKIx2tTsU/LQ+EFl5jflPHAAks0x2tqCTlB4KOcfgD17gHQ66GRICGn8oDKKxxnalmLHWAdvObH+xuMvlXL+VdbdDZ3xJA8NtXHMVXFaFkep+0FEpGiTjR+kvgVFSiQY2eMVsTZyU86pKOcfgO5uWLVkgBQdjOxJqwJYxmn8oBL5o3LuGDtcuavcfn7K+QekqwvOjScZoI1I0ImR0MiMH5Rb5q/xg6agppzTpuAfsKEhiPSlvB8AZVEansYPmqZkElIpddyaAXMunB+qhQuXu3XrdgSdjIpLJODWwRjLDqVobQU1QhYpUp7cvsCVV9r9zrnlU+2nMv+AZYZ9/nqrPrkiRUkmGdnkle1/dU43F61S4J8JBf8Q0axfIlNIJBi+zxuCeUmrmm+WQsE/BGIx6KGHvaNRhu9T71+RI/i5/X27lNsvFwX/kOjuhotWDfDVOd3s26UxgETG+bn9HWMdnDhPuf1yUfAPkcwYQCfO89op6y5AGlpObv+KM5PK7ZeRmnqGUHc3XJFM0tnXw4bBOC2JhHoBS/1LJmFgwHueTh9uvnli482yVQ0K/iGluX+lYfiDr42OwiNNHQCMjcF9TTF6V/ZoaIYKUfAPueFhGB5OK/cv9SnPlIoZsZhy+5Wk4B9iXV2wNpFk/Z4ePr47Tms8Dh0d6gkstS9rqOVrm7vpXdlDdyzoRDUWVfiGXFfX4Ylgdg+pKajUgXh8vPWOhloOTknB38yOMbP/MrMn/L+vK7DfQTNL+Y87Sjlno8puCqrpIKUmZVrv7GO89Y5KMoNT0tg+ZvZl4CXn3CYz2wi8zjn36Tz7DTvnWqZz7EYZ22cm4nHnyO5EAAAKnUlEQVS8+QBmpTUfgISXP65+xuio33pnsYJ+JVVrbJ/zgJv85zcB7y3xeFIEdQiTUMtqnx+nmwWRNAsiaZa0ppXbD5FSc/5p51w0a/nXzrkjin7MbAxIAWPAJufcDwocbx2wDmDevCVv7e7eNeO0NYrMqKAdpGhpQaOCSrDicYaHIUUH57aoU1YQis35Txn8zexuYEGeTZcDNxUZ/Bc65541szcC9wDvdM49Odl5VexTvGQSOvt6+PMDcZqbIbJSLYKkCnLuNnNb7+gjGIxig/+UTT2dc2cX2mZmvzKz451zz5nZ8cDeAsd41v/7lJklgdOBSYO/FC8Wg5FYD29J9HhzA2xL0ZpK6S5AKsdvnz88x8v7jY3BAOqNW0tKbed/B/ARYJP/999zd/BbAL3snDtgZscBvwt8ucTzSh5dXYeHhbhgKMH8TXEiC6KHd2hr0x2BzExW8+KRPenxHP51C3rG17e1aaL0WlJq8N8EfNvMLgZ2Ax8EMLPlwHrn3CXAm4F/MbNDeBXMm5xzj5R4Xikgcxewyr8LmJM17/dpgylaBgbUOkiK5+fw00QZnNMGeLn8c+clNU9ujdM0jnUstwGQ6gVkStkdCP3B1a5p8nL4bW2HN+mjE15lK/OX2pX7BVW9gBTkt8nfOxpl96y28dXntiqHX68U/BtMdr3An++L07wpTiSStYPGDmoM8fiExfHhk5dMbIevrEH9UvBvQNmtg9bv6YERb/0FowmOuU/1AnUtZ0C1jE2tPcrhNxiV+cu43P4CGZEIuiOoVRpioeGozF+mLfuOIH04XnjjCOmOoLbk5PA3RXq89RHvd1xNMkXBX46QG98vSg7Q2efNKTDrqsNlxRpOImB+rn5sLP/mVFanK71LkkvBX6aUuSNYlugZX5dO+3cEV8W9kUWz2wHmvlhmrtCAfanU4Uraxfn3UacrmYyCvxQt945gVWKAWwdjtA0OMGfPwBH7t4ylVVQ0U36xzdgY40MoZBsd9aY9VCWtzJSCv8xYVxfcTbLgxGLr9/Tw57v85qQLjgxg4wdpJMkkDAxMuVv2vLbRPJeurQ1NeyglUfCXkhWK3yP0cGK8hzuHJw4zkbF4bIDjNsUbp7dxnqES8hkbg/vmxDSvrVSUgn/Itff30n/SajAD52h/civ97Z1BJ6to3d1ep7J8BgaY2Nu4VNEKzGqWTEIZ0jYywoThjqcSi9X2yJi1/rltBAr+Idbe38va29bQt+JSeldvpnPrZazcfjW3Xrilpr5Ik2Xqr0gmSaVg40hPyecZL2Iq152En1N/6VCUbzR1lXy4TfO8jlSxko8UbvXyua136uQVZs6Nf3EyMl8ozAJMWDhlZjVbdihFa2tpxxoZ8TpEXdOkiUmmTZ/bQJVtJq+gKPj7nKPn84enWu757CF9gSaRKaUpx53EpkiPujHMlD63gVEP33rg56CydW69TDmoScRifr8Eeko+luL+DOlzWxNmTb2LBKX9ya2s3H41fSsupeezh+hbcSkrt19N+5Nbg06aSEH63NYGFfuEnFpNSC3S5zY4KvapExO+MGb6AklN0Oc2/FTsIyJ1ob2/FzIlGc55y1KQgr+I1LxM34LOrZeNVzivvW2NfgAmoWIfEal5/SetHq9YzvQv6FtxqVfvIHkp5y8itc/Ma0qaRU1LJ6fgLyK1r0DfAkLamjEMFPxDRpVWItOnvgXTp+AfIqq0EpmZ/vZObr1wy3hRT+/qzRpIbgqq8A0RVVqJzJz6FkyPcv5hokorEakSBf8wUaWViFSJgn+IlKPSShXGIlIMBf8QKbXSShXGIlIsVfiGTCmVVqowFpFiKedfT1RhLCJFUvAvUajK2FVhLCJFUvAvQdjK2NXLUUSKpTL/EoStjD1TYZyZQal39Wb62zvV2UVEjqCcfylCWMbe3955+Pzq5SgiBZQU/M3sg2b2sJkdMrOCc0aaWaeZPWZm/Wa2sZRzhorK2EWkRpWa898JvB/YVmgHM5sNfA1YA5wCfNjMTinxvKGgMnYRqVUllfk75x4FsMmLOd4G9DvnnvL3vR04D3iklHOHgcrYRaRWVaPMfxHwdNbyoL+uLqiMXURq0ZQ5fzO7G1iQZ9Plzrl/L+Ic+W4L8haKm9k6YB3AvHlLiji0iIjMxJTB3zl3donnGAROyFpeDDxb4FzXA9cDLFy4XLWmIiIVUo1in58DS83sDWbWBHwIuKMK5xURkQJKber5PjMbBH4H+A8z2+qvX2hmdwE458aADcBW4FHg2865h0tLtoiIlKLU1j7fB76fZ/2zwLuylu8C7irlXCIiUj7q4Ssi0oAU/EVEGpCCv4hIA1LwFxFpQAr+IiINSMFfRKQBKfiLiDQgBX8RkQak4C8i0oAU/EVEGpCCv4hIA1LwFxFpQAr+IiINyJwL55wpZvY8sKvKpz0OeKHK56wFui756boUpmuTXzWuy4nOuddPtVNog38QzGyHc2550OkIG12X/HRdCtO1yS9M10XFPiIiDUjBX0SkASn4T3R90AkIKV2X/HRdCtO1yS8010Vl/iIiDUg5fxGRBqTgLyLSgBo6+JvZB83sYTM7ZGYFm1+ZWaeZPWZm/Wa2sZppDIKZHWNm/2VmT/h/X1dgv4NmlvIfd1Q7ndUy1ftvZs1m9i1/+3Yza6t+KquviOvSZWbPZ31GLgkindVmZl83s71mtrPAdjOzr/jX7SEzO6PaaYQGD/7ATuD9wLZCO5jZbOBrwBrgFODDZnZKdZIXmI3Aj5xzS4Ef+cv5vOKc6/Af76le8qqnyPf/YuDXzrl2YDPwpeqmsvqm8b34VtZn5F+rmsjgJIDOSbavAZb6j3XAtVVI0xEaOvg75x51zj02xW5vA/qdc08550aB24HzKp+6QJ0H3OQ/vwl4b4BpCVox73/29foO8E4zsyqmMQiN+L0oinNuG/DSJLucB9zsPH1A1MyOr07qDmvo4F+kRcDTWcuD/rp69hvOuecA/L/zC+wXMbMdZtZnZvX6A1HM+z++j3NuDNgHHFuV1AWn2O/F+X7RxnfM7ITqJC30QhFT5lT7hNVmZncDC/Jsutw59+/FHCLPuppvHzvZdZnGYZY45541szcC95jZfzvnnixPCkOjmPe/Lj8jUyjmf/4h8E3n3AEzW493d/SOiqcs/ELxean74O+cO7vEQwwC2TmWxcCzJR4zcJNdFzP7lZkd75x7zr8d3VvgGM/6f58ysyRwOlBvwb+Y9z+zz6CZzQHmMfltfz2Y8ro4517MWryBBqgLKVIoYoqKfab2c2Cpmb3BzJqADwF127LFdwfwEf/5R4Aj7pDM7HVm1uw/Pw74XeCRqqWweop5/7Ov1weAe1z9956c8rrklGO/B3i0iukLszuAi/xWPyuBfZli1qpyzjXsA3gf3q/wAeBXwFZ//ULgrqz93gU8jpervTzodFfhuhyL18rnCf/vMf765cC/+s/fDvw38KD/9+Kg013B63HE+w98HniP/zwC/BvQD/wMeGPQaQ7Jdfki8LD/Gfkx8Kag01yl6/JN4DngVT++XAysB9b72w2vpdST/ndneRDp1PAOIiINSMU+IiINSMFfRKQBKfiLiDQgBX8RkQak4C8i0oAU/EVEGpCCv4hIA/r/nfHqkO8zMpQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_813(df, lr2, \"Exercise 8.13 (c) Q=3 with lambda = 1 \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.14\n",
    "\n",
    "* (a) Let $z_n = \\begin{bmatrix}\\Phi_{n1} \\\\ \\dots \\\\ \\Phi_{n\\tilde{d}}\\end{bmatrix}$, where $\\tilde{d}$ is the dimension in feature space $\\mathcal{Z}$. So we have $\\tilde{w}^* = \\sum^N_{n=1}\\beta^*_nz_n = Z^T\\beta*$ using representor theorem. \n",
    "\n",
    "Where $Z=\\begin{bmatrix}z^T_1 \\\\ \\dots \\\\ z^T_N\\end{bmatrix}$, and $K=ZZ^T$, where $K_{ij}=K(x_i,x_j) = \\Phi^T(x_i)\\Phi(x_j) = z^T_iz_j$.\n",
    "\n",
    "Let  $\\tilde{w} = Z^T\\beta$, we now have: \n",
    "\n",
    "\\begin{align*}\n",
    "E_{aug}(\\tilde{w}) &= \\|Z\\tilde{w} - y\\|^2 + \\lambda \\tilde{w}^T\\tilde{w}\\\\\n",
    "&= \\|ZZ^T\\beta - y\\|^2 + \\lambda (Z^T\\beta)^T(Z^T\\beta)\\\\\n",
    "&= \\|K\\beta - y\\|^2 + \\lambda \\beta^TK\\beta\\\\\n",
    "\\end{align*}\n",
    "\n",
    "So $\\beta^*$ minimizes $E(\\beta) = \\|K\\beta - y\\|^2 + \\lambda \\beta^TK\\beta$\n",
    "\n",
    "* (b) $K=ZZ^T$, it's clear that it's symmetric. Or we have $K(x_i,x_j) = \\Phi^T(x_i)\\Phi(x_j) = \\Phi^T(x_j)\\Phi(x_i) = K(x_j, x_i)$\n",
    "\n",
    "* (c) Take derivative of $E(\\beta)$ w.r.t. $\\beta$ and set it to zero, we have\n",
    "\n",
    "\\begin{align*}\n",
    "\\frac{\\partial{E(\\beta)}}{\\partial{\\beta}} &= 2K^TK\\beta - 2K^Ty + \\lambda (K+K^T)\\beta\\\\\n",
    "&= 2(K^TK+\\lambda K^T)\\beta - 2K^Ty\\\\\n",
    "&= 2K^T\\left((K+\\lambda I)\\beta - y\\right)\\\\\n",
    "&= 0 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "The solution is $\\beta^* = (K + \\lambda I)^{-1}y$\n",
    "\n",
    "* (d) The final hypothesis is thus\n",
    "\n",
    "\\begin{align*}\n",
    "g(x) &= \\text{sign}\\left((\\tilde{w}^*)^Tz\\right)\\\\\n",
    "&= \\text{sign}\\left((Z^T\\beta^*)^Tz\\right)\\\\\n",
    "&= \\text{sign}\\left((\\beta^*)^TZz\\right)\\\\\n",
    "&= \\text{sign}\\left((\\beta^*)^T\\begin{bmatrix}z^T_1 \\\\ \\dots \\\\ z^T_N\\end{bmatrix}z\\right)\\\\\n",
    "&= \\text{sign}\\left((\\beta^*)^T\\begin{bmatrix}z^T_1z \\\\ \\dots \\\\ z^T_Nz\\end{bmatrix}\\right)\\\\\n",
    "&= \\text{sign}\\left((\\beta^*)^T\\begin{bmatrix}K(x_1,x) \\\\ \\dots \\\\ K(x_N,x)\\end{bmatrix}\\right)\\\\\n",
    "&= \\text{sign}\\left(\\sum^N_{n=1}\\beta^*_nK(x_n,x)\\right)\\\\\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.15\n",
    "\n",
    "* (a) This is similar to problem 5.2 (a), we copy the answer here:\n",
    "\n",
    "$E_{in}(g_i)$ has to be non-increasing in $i$ because for $p \\lt q$, $g_p \\in \\mathcal{H}_p$ and $g_p \\in \\mathcal{H}_q$ because $\\mathcal{H}_p \\subset \\mathcal{H}_q$. So $E_{in}(g_q) \\le E_{in}(g_p)$, otherwise we can always let $g_q = g_p$ and achieve the same performance as $E_{in}(g_p)$. \n",
    "\n",
    "As we increase $m$, the hypotheses are increasing as well, so the complexity penalty $\\Omega(\\mathcal{H}_m)$ shall be non-decreasing.\n",
    "\n",
    "As we increase $m$, the number of hypotheses increases, so the growth function $m_{\\mathcal{H}_i}$ is increasing as well. \n",
    "The VC bound is $E_{out}(g) \\le E_{in}(g) + \\sqrt{\\frac{8}{N}\\ln\\frac{4m_{\\mathcal{H}_i}(2N)}{\\delta}}$\n",
    "\n",
    "For a given number $N$ points, the first term is non-increasing, the second term is non-decreasing, so the VC bound can be a mixed situation.\n",
    "\n",
    "* (b) The target function is more complex, the larger hypothesis sets will have larger probability. \n",
    " \n",
    "* (c)  According to VC generalization bound theorem 2.5, we have  $E_{out}(g) \\le E_{in}(g) + \\sqrt{\\frac{8}{N}\\ln \\frac{4m_{\\mathcal{H}}(2N)}{\\delta}}$ with probability of at least $1-\\delta$, thus with at most a probability of $\\delta$, we have $P\\left[|E_{out}(g)-E_{in}(g)| \\gt \\epsilon \\right] \\le \\delta$, where $\\epsilon = \\sqrt{\\frac{8}{N}\\ln \\frac{4m_{\\mathcal{H}}(2N)}{\\delta}}$, solve for $\\delta$ we have $P\\left[|E_{out}(g)-E_{in}(g)| \\gt \\epsilon \\right] \\le 4m_{\\mathcal{H}}(2N)e^{-\\frac{\\epsilon^2N}{8}}$, this applies to all $g$ in SRM, let $\\mathcal{H}_i$ be the set where we have maximum error, i.e. $|E_{out}(g_i)-E_{in}(g_i)| \\ge |E_{out}(g_k)-E_{in}(g_k)|$ for $k\\ne i$. so we have $P\\left[\\max_{g\\in \\mathcal{H}_i}|E_{out}(g)-E_{in}(g)| \\gt \\epsilon \\right] \\le 4m_{\\mathcal{H}_i}(2N)e^{-\\frac{\\epsilon^2N}{8}}$\n",
    "\n",
    "Now: \n",
    "\n",
    "\\begin{align*}\n",
    "P\\left[|E_{in}(g_i) - E_{out}(g_i)| \\gt \\epsilon | g^* = g_m\\right] &= \\frac{P\\left[|E_{in}(g_i) - E_{out}(g_i)| \\gt \\epsilon \\cap  g^* = g_m\\right]}{P(g^* = g_m)}\\\\\n",
    "&= \\frac{P[g^* = g_m| \\;|E_{in}(g_i) - E_{out}(g_i)| \\gt \\epsilon]P\\left[|E_{in}(g_i) - E_{out}(g_i)| \\gt \\epsilon\\right]}{p_m}\\\\\n",
    "&\\le \\frac{P\\left[|E_{in}(g_i) - E_{out}(g_i)| \\gt \\epsilon\\right]}{p_m}\\\\\n",
    "&= \\frac{4m_{\\mathcal{H}_i}(2N)e^{-\\frac{\\epsilon^2N}{8}}}{p_m}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "Since $\\mathcal{H}_i$ has the largest error, set $i=m$, we have\n",
    "\n",
    "$P\\left[|E_{in}(g_m) - E_{out}(g_m)| \\gt \\epsilon | g^* = g_m\\right] \\le  \\frac{4m_{\\mathcal{H}_m}(2N)e^{-\\frac{\\epsilon^2N}{8}}}{p_m}$\n",
    "\n",
    "If we use SRM and end up with $g_m$ then the generalization bound is a factor $\\frac{1}{p_m}$ worse than the bound we would have gotten if we simply started with $\\mathcal{H}_m$.\n",
    "\n",
    "\n",
    "#### Problem 8.16\n",
    "\n",
    "We can pose the soft order constraints $\\{\\mathcal{H}_C\\}_{C>0}$ within the SRM framework.\n",
    "In the soft order constraint, we are optimizing the problem:\n",
    "\n",
    "$\\min_{w}E_{in}(w)$ subject to $w^Tw \\le C$, this is equivalent to minimizing $E_{in}$ over $\\mathcal{H}(C) = \\{h\\in\\mathcal{H}, w^Tw \\le C\\}$.\n",
    "\n",
    "If $C_1 < C_2$, then we have $\\mathcal{H}(C_1) \\subset \\mathcal{H}(C_2)$\n",
    "\n",
    "On the other hand, if we use regularization parameters, we are optimizing:\n",
    "\n",
    "$\\min_{w}E_{in}(w) + \\lambda w^Tw $, which can't be posed as SRM framework.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 8.17\n",
    "\n",
    "* (a) VC dimension is the largest value of $N$ for which $m_{\\mathcal{H}}(N) = 2^N$. If the number of points $N> d_{VC}(\\mathcal{H}_{m+1})$, then it's possible that $E_{in}(\\mathcal{H}_m) < E_{in}(\\mathcal{H}_{m+1})$ since the hypotheses in the two sets can be totally different yet still satisfy the constraint on the VC dimension. If the target function comes from $\\mathcal{H}_m$, it's possible that it has a smaller in-sample error.\n",
    "\n",
    "* (b) From problem 8.15, suppose that $\\mathcal{H}_i$ is the set with maximum error, and $g^* = g_m$, for any set $k$, we have \n",
    "\n",
    "\\begin{align*}\n",
    "P\\left[|E_{in}(g_k) - E_{out}(g_k)| \\gt \\epsilon | g^* = g_m\\right] &\\le P\\left[|E_{in}(g_i) - E_{out}(g_i)| \\gt \\epsilon | g^* = g_m\\right] \\\\\n",
    "&\\le \\frac{4m_{\\mathcal{H}_i}(2N)e^{-\\frac{\\epsilon^2N}{8}}}{p_m}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "So \n",
    "\n",
    "\\begin{align*}\n",
    "P[|E_{in}(g_k) - E_{out}(g_k)| \\gt \\epsilon] &= \\sum_n P[g^* = g_n]P\\left[|E_{in}(g_k) - E_{out}(g_k)| \\gt \\epsilon | g^* = g_n\\right] \\\\\n",
    "&\\le \\sum_n 4m_{\\mathcal{H}_i}(2N)e^{-\\frac{\\epsilon^2N}{8}}\\\\\n",
    "&= 4Mm_{\\mathcal{H}_i}(2N)e^{-\\frac{\\epsilon^2N}{8}}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "and apply $m_{\\mathcal{H}}(N) \\le N^{d_{VC}}+1$, let $\\delta = 4Mm_{\\mathcal{H}_i}(2N)e^{-\\frac{\\epsilon^2N}{8}}$, we have $\\epsilon = \\sqrt{\\frac{8}{N}\\ln \\frac{4Mm_{\\mathcal{H}_i}(2N)}{\\delta}}$\n",
    "\n",
    "So we have the generalization error bounded by\n",
    "\n",
    "\\begin{align*}\n",
    "E_{out}(g_k) &\\le E_{in}(g_k) + \\epsilon \\\\\n",
    "&= E_{in}(g_k) + \\sqrt{\\frac{8}{N}\\ln \\frac{4Mm_{\\mathcal{H}_i}(2N)}{\\delta}}\\\\\n",
    "&= E_{in}(g_k) + \\sqrt{\\frac{8}{N}\\ln \\frac{4M\\left((2N)^{d_{VC}(\\mathcal{H}_i)}+1\\right)}{\\delta}}\\\\\n",
    "&< E_{in}(g_k) + \\sqrt{\\frac{8}{N}\\ln \\frac{4M\\left((2N)^{d_{VC}(\\mathcal{H}_M)}+1\\right)}{\\delta}}\\\\\n",
    "\\end{align*}\n",
    "\n",
    "#### Problem 8.18\n",
    "\n",
    "* (a) We can't use $m$ on the VC-bound, because before we run the learning algorithm, we don't know when will we stop, i.e. what is the $m$ value. So we don't have just $m$ hypotheses when we start.\n",
    "\n",
    "* (b) for $m=1,\\dots$, we have $\\mathcal{H}_m \\subset \\mathcal{H}_{m+1}$, so we can have $\\mathcal{H}_1, \\dots, \\mathcal{H}_m, \\dots$, where $\\mathcal{H}_m = \\{h_1, h_2, \\dots, h_m\\}$ as in SRM structure.\n",
    "\n",
    "As we move along $h_1,h_2,\\dots, $, we are effectively computing the $E_{in}$ for each $\\mathcal{H}_m$. For example, \n",
    "\n",
    "$g_1 = \\text{argmin}_{h\\in \\mathcal{H}_1}E_{in}(h) = h_1$\n",
    "\n",
    "$g_2 = \\text{argmin}_{h\\in \\mathcal{H}_2}E_{in}(h) = \\text{argmin}_h (E_{in}(h_1), E_{in}(h_2))$\n",
    "\n",
    "$g_m = \\text{argmin}_{h\\in \\mathcal{H}_m}E_{in}(h) = \\text{argmin}_h (E_{in}(h_1), \\dots, E_{in}(h_m))$\n",
    "\n",
    "If we stop at $h_m$, then we know that $g_m = h_m$ and $g^* = \\text{argmin}_{g_m}E_{in}(g_m) = h_m$.\n",
    "\n",
    "This is the SRM process.\n",
    "\n",
    "* (c) The SRM bound we derived in previous problems 8.15 is applicable regardless of the number of hypotheses sets. So we can use the bound derived in problem 8.15 (c) here.  However, since $d_{VC}(\\mathcal{H}) = \\infty$, the bound is only useful when the VC dimension $d_{VC}(\\mathcal{H}_m) < \\infty$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}