10c. neural_networks_in_practice_scikit_learn.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "=================================================================================================================\n",
    "# Neural Networks in Practice: scikit-learn\n",
    "\n",
    "\n",
    "##### D.Vidotto, Data Mining: JBI030 2019/2020\n",
    "\n",
    "\n",
    "=================================================================================================================\n",
    "\n",
    "\n",
    "In this notebook, we are going to see how to setup and train Feedforward Neural Networks with scikit-learn. For this purpose, we are going to explore the usage of two functions from the `neural_network` module: [MLPRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html) and [MLPClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html).\n",
    "\n",
    "In this notebook we will go through the same examples of the notebook dedicated to Keras. Since we are already familiar with scikit-learn, we will have simply have a quick overview of their Neural Network implementations. \n",
    "\n",
    "Here, we will go through the following examples:\n",
    "\n",
    "1. Regression \n",
    "1. Classification\n",
    "  * Binary Classification\n",
    "  * Multiclass Classification\n",
    "1. cross-validation with the Heart Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Regression\n",
    "For the regression example, we will use the [Boston Houses dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html). The data contains contains information collected by the U.S Census Service concerning housing in Boston; the sample size is 506, and there are 13 features. The target variable is the median value of houses in different areas of Boston. More information about the dataset can be found [here](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html).\n",
    "\n",
    "Let's load the data: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target', 'feature_names', 'DESCR', 'filename'])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_boston\n",
    "bunch_data = load_boston()\n",
    "bunch_data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',\n",
       "       'TAX', 'PTRATIO', 'B', 'LSTAT'], dtype='<U7')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Feature names \n",
    "bunch_data.feature_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(506, 13)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = bunch_data.data, bunch_data.target\n",
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Some preprocessing**..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train-test split (separate 10% of the data for the test set - we won't use a separate validation set here)\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.0290e-01, 3.0000e+01, 4.9300e+00, 0.0000e+00, 4.2800e-01,\n",
       "        6.3580e+00, 5.2900e+01, 7.0355e+00, 6.0000e+00, 3.0000e+02,\n",
       "        1.6600e+01, 3.7275e+02, 1.1220e+01],\n",
       "       [2.9850e-02, 0.0000e+00, 2.1800e+00, 0.0000e+00, 4.5800e-01,\n",
       "        6.4300e+00, 5.8700e+01, 6.0622e+00, 3.0000e+00, 2.2200e+02,\n",
       "        1.8700e+01, 3.9412e+02, 5.2100e+00],\n",
       "       [2.3004e+00, 0.0000e+00, 1.9580e+01, 0.0000e+00, 6.0500e-01,\n",
       "        6.3190e+00, 9.6100e+01, 2.1000e+00, 5.0000e+00, 4.0300e+02,\n",
       "        1.4700e+01, 2.9709e+02, 1.1100e+01],\n",
       "       [2.0746e-01, 0.0000e+00, 2.7740e+01, 0.0000e+00, 6.0900e-01,\n",
       "        5.0930e+00, 9.8000e+01, 1.8226e+00, 4.0000e+00, 7.1100e+02,\n",
       "        2.0100e+01, 3.1843e+02, 2.9680e+01],\n",
       "       [1.1027e-01, 2.5000e+01, 5.1300e+00, 0.0000e+00, 4.5300e-01,\n",
       "        6.4560e+00, 6.7800e+01, 7.2255e+00, 8.0000e+00, 2.8400e+02,\n",
       "        1.9700e+01, 3.9690e+02, 6.7300e+00]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT8AAAEvCAYAAAAzcMYwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAPfUlEQVR4nO3df6zdd13H8eeLFgYDCZu9nXXtvMM0yCD8ys2czpjJGFS3rPtnSZdgGl3SmEwdBoId/LFosmRGg/iHmDRsrglzS8MP2wAiTYGgiWze/cB162obVrfS0l4kiD+S4cbbP+538Vhvae/5fs9uez7PR9J8z/fz/Z57Pp9/nvmec+/3NFWFJLXmFSs9AUlaCcZPUpOMn6QmGT9JTTJ+kppk/CQ1afVKTwBgzZo1NTs7u9LTkDRlHnnkke9W1cxSx86J+M3OzjI/P7/S05A0ZZL8y+mO+bZXUpOMn6QmGT9JTTJ+kppk/CQ1yfhJapLxk9Qk4yepScZPUpPOGL8k9yY5mWT/Esc+lKSSrBkZuyPJ4SQHk7xv6AlL0hDO5srvPmDTqYNJNgDXAc+OjF0BbAHe0j3nE0lWDTJTSRrQGe/traqvJ5ld4tCfAh8Gdo+MbQYerKrngWeSHAauBP6h/1TVx+z2L0z8NY7cff3EX0Maylif+SW5Efh2VX3zlEOXAs+N7B/txiTpnLLsb3VJciHwUeC9Sx1eYmzJ/x4uyTZgG8Bll1223GlIUi/jXPn9LHA58M0kR4D1wKNJforFK70NI+euB44t9UOqakdVzVXV3MzMkl+3JUkTs+z4VdUTVbW2qmarapbF4L2rqr4D7AG2JLkgyeXARuDhQWcsSQM4mz91eYDFX1i8KcnRJLee7tyqehLYBTwFfAm4rapeHGqykjSUs/lt7y1nOD57yv5dwF39piVJk+UdHpKaZPwkNcn4SWqS8ZPUJOMnqUnGT1KTjJ+kJhk/SU0yfpKaZPwkNcn4SWqS8ZPUJOMnqUnGT1KTjJ+kJhk/SU0yfpKaZPwkNcn4SWqS8ZPUJOMnqUnGT1KTjJ+kJhk/SU0yfpKaZPwkNcn4SWqS8ZPUpDPGL8m9SU4m2T8y9sdJnk7yT0k+l+QNI8fuSHI4ycEk75vUxCWpj7O58rsP2HTK2F7grVX1NuCfgTsAklwBbAHe0j3nE0lWDTZbSRrIGeNXVV8HvnfK2Jer6oVu9xvA+u7xZuDBqnq+qp4BDgNXDjhfSRrEEJ/5/SbwN93jS4HnRo4d7cb+nyTbkswnmV9YWBhgGpJ09nrFL8lHgReA+18aWuK0Wuq5VbWjquaqam5mZqbPNCRp2VaP+8QkW4EbgGur6qXAHQU2jJy2Hjg2/vQkaTLGuvJLsgn4feDGqvqvkUN7gC1JLkhyObAReLj/NCVpWGe88kvyAHANsCbJUeBOFn+7ewGwNwnAN6rqt6rqySS7gKdYfDt8W1W9OKnJS9K4zhi/qrplieF7fsz5dwF39ZmUJE2ad3hIapLxk9Qk4yepScZPUpOMn6QmGT9JTTJ+kppk/CQ1yfhJapLxk9Qk4yepScZPUpOMn6QmGT9JTTJ+kppk/CQ1yfhJapLxk9Qk4yepScZPUpOMn6QmGT9JTTJ+kppk/CQ1yfhJapLxk9Qk4yepSWeMX5J7k5xMsn9k7OIke5Mc6rYXjRy7I8nhJAeTvG9SE5ekPs7myu8+YNMpY9uBfVW1EdjX7ZPkCmAL8JbuOZ9Ismqw2UrSQM4Yv6r6OvC9U4Y3Azu7xzuBm0bGH6yq56vqGeAwcOVAc5WkwYz7md8lVXUcoNuu7cYvBZ4bOe9oNyZJ55Shf+GRJcZqyROTbUnmk8wvLCwMPA1J+vHGjd+JJOsAuu3JbvwosGHkvPXAsaV+QFXtqKq5qpqbmZkZcxqSNJ5x47cH2No93grsHhnfkuSCJJcDG4GH+01Rkoa3+kwnJHkAuAZYk+QocCdwN7Arya3As8DNAFX1ZJJdwFPAC8BtVfXihOYuSWM7Y/yq6pbTHLr2NOffBdzVZ1KSNGne4SGpScZPUpOMn6QmGT9JTTJ+kppk/CQ1yfhJapLxk9Qk4yepScZPUpOMn6QmGT9JTTJ+kppk/CQ1yfhJapLxk9Qk4yepScZPUpOMn6QmGT9JTTJ+kppk/CQ1yfhJapLxk9Qk4yepScZPUpOMn6QmGT9JTeoVvyS/l+TJJPuTPJDk1UkuTrI3yaFue9FQk5WkoYwdvySXAr8LzFXVW4FVwBZgO7CvqjYC+7p9STqn9H3buxp4TZLVwIXAMWAzsLM7vhO4qedrSNLgxo5fVX0b+BPgWeA48G9V9WXgkqo63p1zHFg7xEQlaUh93vZexOJV3uXATwOvTfL+ZTx/W5L5JPMLCwvjTkOSxtLnbe97gGeqaqGq/hv4LPCLwIkk6wC67cmlnlxVO6pqrqrmZmZmekxDkpavT/yeBa5KcmGSANcCB4A9wNbunK3A7n5TlKThrR73iVX1UJJPA48CLwCPATuA1wG7ktzKYiBvHmKikjSkseMHUFV3AneeMvw8i1eBknTO8g4PSU0yfpKaZPwkNcn4SWqS8ZPUJOMnqUm9/tRFw5jd/oWVnsIgXq51HLn7+pfldTTdvPKT1CTjJ6lJxk9Sk4yfpCYZP0lNMn6SmmT8JDXJ+ElqkvGT1CTjJ6lJxk9Sk4yfpCYZP0lNMn6SmmT8JDXJ+ElqkvGT1CTjJ6lJxk9Sk4yfpCYZP0lN6hW/JG9I8ukkTyc5kOQXklycZG+SQ932oqEmK0lD6Xvl92fAl6rq54C3AweA7cC+qtoI7Ov2JemcMnb8krwe+GXgHoCq+mFVfR/YDOzsTtsJ3NR3kpI0tD5Xfm8EFoC/TPJYkk8meS1wSVUdB+i2aweYpyQNqk/8VgPvAv6iqt4J/CfLeIubZFuS+STzCwsLPaYhScvXJ35HgaNV9VC3/2kWY3giyTqAbntyqSdX1Y6qmququZmZmR7TkKTlGzt+VfUd4Lkkb+qGrgWeAvYAW7uxrcDuXjOUpAlY3fP5vwPcn+RVwLeA32AxqLuS3Ao8C9zc8zUkaXC94ldVjwNzSxy6ts/PlaRJ8w4PSU0yfpKaZPwkNcn4SWqS8ZPUJOMnqUnGT1KTjJ+kJhk/SU0yfpKaZPwkNcn4SWqS8ZPUJOMnqUnGT1KTjJ+kJhk/SU3q+zX2U212+xdWegqSJsQrP0lNMn6SmmT8JDXJ+ElqkvGT1CTjJ6lJxk9Sk4yfpCYZP0lNMn6SmtQ7fklWJXksyee7/YuT7E1yqNte1H+akjSsIa78bgcOjOxvB/ZV1UZgX7cvSeeUXvFLsh64HvjkyPBmYGf3eCdwU5/XkKRJ6Hvl93Hgw8CPRsYuqarjAN127VJPTLItyXyS+YWFhZ7TkKTlGTt+SW4ATlbVI+M8v6p2VNVcVc3NzMyMOw1JGkuf7/O7Grgxya8BrwZen+RTwIkk66rqeJJ1wMkhJipJQxr7yq+q7qiq9VU1C2wBvlJV7wf2AFu707YCu3vPUpIGNom/87sbuC7JIeC6bl+SzimDfI19VX0N+Fr3+F+Ba4f4uZI0Kd7hIalJxk9Sk4yfpCYZP0lN8v/t1Xnn5fj/lI/cff3EX0Mryys/SU0yfpKaZPwkNcn4SWqS8ZPUJOMnqUnGT1KTjJ+kJhk/SU0yfpKa5O1t0hJejlvowNvoVpJXfpKaZPwkNcn4SWqS8ZPUJOMnqUnGT1KTjJ+kJhk/SU0yfpKaZPwkNcn4SWrS2PFLsiHJV5McSPJkktu78YuT7E1yqNteNNx0JWkYfa78XgA+WFVvBq4CbktyBbAd2FdVG4F93b4knVPGjl9VHa+qR7vH/w4cAC4FNgM7u9N2Ajf1naQkDW2Qz/ySzALvBB4CLqmq47AYSGDtEK8hSUPqHb8krwM+A3ygqn6wjOdtSzKfZH5hYaHvNCRpWXrFL8krWQzf/VX12W74RJJ13fF1wMmlnltVO6pqrqrmZmZm+kxDkpZt7G9yThLgHuBAVX1s5NAeYCtwd7fd3WuGp/FyfdOupOnU52vsrwZ+HXgiyePd2EdYjN6uJLcCzwI395uiJA1v7PhV1d8DOc3ha8f9uZLOP+fj/3niHR6SmmT8JDXJ+ElqkvGT1CTjJ6lJxk9Sk4yfpCYZP0lNMn6SmmT8JDXJ+ElqkvGT1CTjJ6lJxk9Sk4yfpCb1+TJTSecBv/V8aV75SWqSV37SCvKqbOV45SepScZPUpOMn6QmGT9JTTJ+kppk/CQ1yfhJapLxk9Qk4yepScZPUpMmFr8km5IcTHI4yfZJvY4kjWMi8UuyCvhz4FeBK4BbklwxideSpHFM6srvSuBwVX2rqn4IPAhsntBrSdKyTSp+lwLPjewf7cYk6Zwwqa+0yhJj9X9OSLYB27rd/0hycEJzGcoa4LsrPYkJm/Y1ur7zXP5o2Wv8mdMdmFT8jgIbRvbXA8dGT6iqHcCOCb3+4JLMV9XcSs9jkqZ9ja7v/DfkGif1tvcfgY1JLk/yKmALsGdCryVJyzaRK7+qeiHJbwN/C6wC7q2qJyfxWpI0jol9jX1VfRH44qR+/go4b96i9zDta3R957/B1piqOvNZkjRlvL1NUpOM3xKS3JvkZJL9I2MXJ9mb5FC3vWgl59hHkg1JvprkQJInk9zejU/FGpO8OsnDSb7Zre8PuvGpWN9LkqxK8liSz3f707a+I0meSPJ4kvlubLA1Gr+l3QdsOmVsO7CvqjYC+7r989ULwAer6s3AVcBt3e2H07LG54F3V9XbgXcAm5JcxfSs7yW3AwdG9qdtfQC/UlXvGPnzluHWWFX+W+IfMAvsH9k/CKzrHq8DDq70HAdc627gumlcI3Ah8Cjw89O0Phb/dnYf8G7g893Y1KyvW8MRYM0pY4Ot0Su/s3dJVR0H6LZrV3g+g0gyC7wTeIgpWmP3lvBx4CSwt6qman3Ax4EPAz8aGZum9cHiXWFfTvJId0cYDLjGif2pi859SV4HfAb4QFX9IFnqrsTzU1W9CLwjyRuAzyV560rPaShJbgBOVtUjSa5Z6flM0NVVdSzJWmBvkqeH/OFe+Z29E0nWAXTbkys8n16SvJLF8N1fVZ/thqdqjQBV9X3gayx+hjst67sauDHJERa/MendST7F9KwPgKo61m1PAp9j8duiBluj8Tt7e4Ct3eOtLH5Odl7K4iXePcCBqvrYyKGpWGOSme6KjySvAd4DPM2UrK+q7qiq9VU1y+Kto1+pqvczJesDSPLaJD/x0mPgvcB+Blyjf+S8hCQPANew+C0ZJ4A7gb8GdgGXAc8CN1fV91Zqjn0k+SXg74An+N/PjD7C4ud+5/0ak7wN2MnirZWvAHZV1R8m+UmmYH2jure9H6qqG6ZpfUneyOLVHix+PPdXVXXXkGs0fpKa5NteSU0yfpKaZPwkNcn4SWqS8ZPUJOMnqUnGT1KTjJ+kJv0PE0pIqcOYPhkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Histogram of the target\n",
    "plt.figure(figsize=(5,5))\n",
    "plt.hist(y_train)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rescale data with min-max scaler\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "mms = MinMaxScaler()\n",
    "X_train_scaled = mms.fit_transform(X_train)\n",
    "X_test_scaled =mms.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Setting the Regression FNN**. We are now going to use Keras to set up a Regression NN. Let's review the components of a Regression network: \n",
    "* the input layer contains as many neurons as the number of features\n",
    "* as we are predicting just one continuous output, we can set up the network with one output layer\n",
    "* the loss function is the mean squared error \n",
    "* no activation function is used for the output layer\n",
    "\n",
    "Here, we are going to build a Neural Network with the following characteristics: \n",
    "* 5 neurons in the first hidden layer (`hidden_layer_sizes` which takes a tuple with the number of neurons as input)\n",
    "* ReLU activation function for the hidden layer (`activation='relu'`)\n",
    "* early stopping (`early_stopping=True`)\n",
    "    * training will be stopped if there is no improvement on the validation set for 10 iterations  (`n_iter_no_change=10`)\n",
    "* a maximum number of 1000 epochs (`max_iter=1000`)\n",
    "* SGD (`solver='sgd'`) with momentum optimizer with:\n",
    "    * batch size = 32 (`batch_size=32`) \n",
    "    * constant learning rate = 0.001 (`learning_rate_init=0.001`)\n",
    "    * momentum = 0.9 (`momentum=0.9`)\n",
    "* $l_2$ with penalty = 0.001 (`alpha=0.001`)\n",
    "\n",
    "Furthrermore, we will also set `validation_fraction=0.1` to evaluate the early stopping, and `random_state=1` for reproducibility. Last, note that MLPRegressor uses already the squared error as a loss function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPRegressor\n",
    "model = MLPRegressor(hidden_layer_sizes=(5, ), \n",
    "                     activation='relu', \n",
    "                     solver='sgd', \n",
    "                     alpha=0.001, \n",
    "                     batch_size=32, \n",
    "                     learning_rate='constant', \n",
    "                     learning_rate_init=0.001, \n",
    "                     max_iter=1000, \n",
    "                     random_state=1, \n",
    "                     tol=0.0001,   # the tolerance criterion for the optimization algorithm\n",
    "                     verbose=False, \n",
    "                     momentum=0.9, \n",
    "                     early_stopping=True, \n",
    "                     validation_fraction=0.1, \n",
    "                     n_iter_no_change=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will only train networks with one layer in this example; if you want to add layers with a specific number of neurons you can just insert other values in the ` hidden_layer_size` argument; for example, to have two hidden layers with 5 and 4 neurons respectively, you can set: \n",
    "\n",
    "```python\n",
    "hidden_layer_size = (5, 4, )\n",
    "```\n",
    "\n",
    "Let's go back to our single-layer model, and train it: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPRegressor(activation='relu', alpha=0.001, batch_size=32, beta_1=0.9,\n",
       "             beta_2=0.999, early_stopping=True, epsilon=1e-08,\n",
       "             hidden_layer_sizes=(5,), learning_rate='constant',\n",
       "             learning_rate_init=0.001, max_fun=15000, max_iter=1000,\n",
       "             momentum=0.9, n_iter_no_change=10, nesterovs_momentum=True,\n",
       "             power_t=0.5, random_state=1, shuffle=True, solver='sgd',\n",
       "             tol=0.0001, validation_fraction=0.1, verbose=False,\n",
       "             warm_start=False)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the model estimates first. The weights of the first hidden layers are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.09603048,  0.22989646, -0.57706882, -1.58376335, -0.39085469],\n",
       "       [-0.47060502, -0.39892824, -0.17828496,  0.56280347,  0.0050342 ],\n",
       "       [-0.09726905, -0.04002049, -0.34118083,  0.37832569, -0.65892536],\n",
       "       [ 0.1906156 , -0.12020428,  0.06775164,  0.56555095, -0.35850449],\n",
       "       [ 0.34297502,  0.304535  , -0.21538345, -1.12516912,  0.47638559],\n",
       "       [ 0.44932352, -0.7900406 , -0.53211595,  3.13195256,  2.54304055],\n",
       "       [-0.46875472, -0.51633007,  0.52858846, -0.0676868 , -0.08141338],\n",
       "       [-0.21340524,  0.09357604,  0.38629245, -2.04612217, -0.4292811 ],\n",
       "       [ 0.5581062 ,  0.02982997, -0.25345583,  1.00644459, -0.34184823],\n",
       "       [-0.06583269,  0.19399651, -0.23825218, -0.8985927 , -0.84688964],\n",
       "       [-0.55987902, -0.17907106, -0.332897  , -1.10194343, -1.01758611],\n",
       "       [-0.52116395, -0.45212288, -0.40781714,  0.88484004,  0.07991964],\n",
       "       [-0.45965152, -0.27862171,  0.22441587, -3.59041253, -1.53614447]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.coefs_[0]    # shape = n. inputs x n. neurons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the weights of the output layers: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.80623153],\n",
       "       [-0.32857698],\n",
       "       [-0.7212606 ],\n",
       "       [ 5.63707697],\n",
       "       [ 3.17262482]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.coefs_[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The intercepts of the hidden and output layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([ 0.03527594, -0.42069774,  0.01719247,  3.41236003, -0.02724533]),\n",
       " array([2.64637129])]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.intercepts_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can assess the convergence of the loss as follows: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT8AAADCCAYAAAA/3cXYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAVkklEQVR4nO3da2xc533n8e/vzPCiC6VIFiXr5khO5cRyLnKqVRV4L07c1K5RxAmwBeRFA78I4LxwgAQIUNhdYNt9YSAvtunuAk0Kt/bGQIO4bpI2RjZo46rebdzdxKEc2bEsK1It2WIkS5RlR1deZs6/L84hPZJJkTxD8vCQvw9AzMwzZ+b8ORz+5nmec84cRQRmZotNUnYBZmZlcPiZ2aLk8DOzRcnhZ2aLksPPzBYlh5+ZLUr1sgsAWLNmTWzZsqXsMsxsgdm3b9+ZiOgd7755EX5btmyhr6+v7DLMbIGR9NpE93nYa2aLksPPzBYlh5+ZLUoOPzNblCoZfn/2f/+FP33mSNllmFmFVTL8fnR4gH985XTZZZhZhVUy/LrqNYYazbLLMLMKq2j4JQyNpGWXYWYVNmn4Sdos6RlJByUdkPTFvP2PJP1S0v785+6Wxzwk6YikQ5LunOmiu+oJQw2Hn5kVN5UjPBrAlyPieUk9wD5JT+f3/UlE/LfWhSVtB/YAtwAbgH+QdFNEzNg41cNeM2vXpD2/iDgZEc/n188DB4GN13jIPcATETEUEUeBI8CumSh2VFeHe35m1p5pzflJ2gLcCvwkb/qCpBclPSZpVd62ETje8rB+xglLSfdL6pPUNzAwMK2iPednZu2acvhJWg58B/hSRJwDvg68D9gBnAT+eHTRcR7+rrMkRcQjEbEzInb29o77pQsTGh32+uRLZlbUlMJPUgdZ8H0zIr4LEBGnIqIZESnw57wztO0HNrc8fBNwYuZKznp+aUAjdfiZWTFT2dor4FHgYER8taV9fctinwFeyq8/BeyR1CVpK7ANeG7mSs7m/ADP+5lZYVPZ2nsb8Fng55L2521/ANwraQfZkPYY8HmAiDgg6UngZbItxQ/M5JZeyIa9AEMjTZZ3zYuvJDSzipk0OSLiWcafx/vBNR7zMPBwG3VdU1fdPT8za081j/DwsNfM2lTN8Bsd9npHZzMrqKLhl/f8vK+fmRVU0fAb7fk5/MysmGqG39icn4e9ZlZMNcPPw14za1NFw8/DXjNrT0XDz8NeM2tPNcPP+/mZWZuqGX4th7eZmRVR0fBzz8/M2uPwM7NFqZLhV68l1BJ5g4eZFVbJ8AN/lb2Ztafa4edhr5kVVOHw8+krzay46oafT19pZm2obvh5zs/M2lDh8POw18yKq3D4edhrZsVN5dSVmyU9I+mgpAOSvpi3r5b0tKTD+eWqlsc8JOmIpEOS7pyNwj3nZ2btmErPrwF8OSJuBnYDD0jaDjwI7I2IbcDe/Db5fXuAW4C7gK9Jqs104R72mlk7Jg2/iDgZEc/n188DB4GNwD3A4/lijwOfzq/fAzwREUMRcRQ4Auya6cK9wcPM2jGtOT9JW4BbgZ8A6yLiJGQBCazNF9sIHG95WH/eNqM852dm7Zhy+ElaDnwH+FJEnLvWouO0xTjPd7+kPkl9AwMDUy1jjIe9ZtaOKYWfpA6y4PtmRHw3bz4laX1+/3rgdN7eD2xuefgm4MTVzxkRj0TEzojY2dvbO+3CvcHDzNoxla29Ah4FDkbEV1vuegq4L79+H/C9lvY9krokbQW2Ac/NXMkZz/mZWTvqU1jmNuCzwM8l7c/b/gD4CvCkpM8BrwO/CxARByQ9CbxMtqX4gYiY8fHp6LA3Isjy2cxs6iYNv4h4lvHn8QDumOAxDwMPt1HXpLrqCWlAIw06ag4/M5ue6h7h4ZMYmVkbqht+PomRmbWhwuHnnp+ZFVfd8POw18zaUN3wGx32ekdnMyugwuGX9/y8r5+ZFVDh8Bvt+Tn8zGz6qht+Y3N+Hvaa2fRVN/w87DWzNlQ4/DzsNbPiKhx+HvaaWXHVDT/v52dmbahu+PnwNjNrQ4XDzz0/MyvO4Wdmi1Jlw69eS6gl8gYPMyuksuEH/ip7Myuu+uHnYa+ZFVDx8PPpK82smGqHn09faWYFVTv8POdnZgVN5by9j0k6LemllrY/kvRLSfvzn7tb7ntI0hFJhyTdOVuFg4e9ZlbcVHp+3wDuGqf9TyJiR/7zAwBJ24E9wC35Y74mqTZTxV7NGzzMrKhJwy8i/gk4O8Xnuwd4IiKGIuIocATY1UZ91+Q5PzMrqp05vy9IejEfFq/K2zYCx1uW6c/b3kXS/ZL6JPUNDAwUKsDDXjMrqmj4fR14H7ADOAn8cd6ucZaN8Z4gIh6JiJ0RsbO3t7dQEd7gYWZFFQq/iDgVEc2ISIE/552hbT+wuWXRTcCJ9kqcmOf8zKyoQuEnaX3Lzc8Ao1uCnwL2SOqStBXYBjzXXokT87DXzIqqT7aApG8BtwNrJPUDfwjcLmkH2ZD2GPB5gIg4IOlJ4GWgATwQEbOWTt7gYWZFTRp+EXHvOM2PXmP5h4GH2ylqqjznZ2ZFVfwIj2zYGzHuNhUzswlVPPwS0oBG6vAzs+mpdvj5JEZmVlC1w88nMTKzgioefu75mVkx1Q4/D3vNrKBqh9/osNc7OpvZNFU8/PKen/f1M7Npqnj4jfb8HH5mNj3VDr+xOT8Pe81seqodfh72mllBFQ8/D3vNrJiKh5+HvWZWTLXDz/v5mVlB1Q4/H95mZgVVPPzc8zOzYhx+ZrYoVTr86rWEWiJv8DCzaat0+IG/yt7MilkY4edhr5lN0wIIP5++0symb9Lwk/SYpNOSXmppWy3paUmH88tVLfc9JOmIpEOS7pytwkf59JVmVsRUen7fAO66qu1BYG9EbAP25reRtB3YA9ySP+ZrkmozVu04POdnZkVMGn4R8U/A2aua7wEez68/Dny6pf2JiBiKiKPAEWDXDNU6Lg97zayIonN+6yLiJEB+uTZv3wgcb1muP297F0n3S+qT1DcwMFCwDG/wMLNiZnqDh8ZpG/ekuhHxSETsjIidvb29hVfoOT8zK6Jo+J2StB4gvzydt/cDm1uW2wScKF7e5DzsNbMiiobfU8B9+fX7gO+1tO+R1CVpK7ANeK69Eq/NGzzMrIj6ZAtI+hZwO7BGUj/wh8BXgCclfQ54HfhdgIg4IOlJ4GWgATwQEbPaLfOcn5kVMWn4RcS9E9x1xwTLPww83E5R0+Fhr5kVUf0jPLzBw8wKqH74ec7PzApYAOGXDXsjxt2jxsxsXAsg/BLSgEbq8DOzqat++PkkRmZWQPXDzycxMrMCFkD4uednZtNX/fDzsNfMCqh++I0Oe72js5lNwwIIv7zn5339zGwaKh9+K5d0APDmxaGSKzGzKql8+L3/+h4AXj5xruRKzKxKKh9+Pd0dvPe6pRxw+JnZNFQ+/ABu2bDC4Wdm07JAwm8lr5+9xLnBkbJLMbOKWBDht33DCsDzfmY2dQsi/G7Jw89DXzObqgURfmt7uunt6eLAiV+VXYqZVcSCCD/Ien8e9prZVC2o8Dty+oIPczOzKWkr/CQdk/RzSfsl9eVtqyU9LelwfrlqZkq9tu3rV9JIg1+8cWEuVmdmFTcTPb+PR8SOiNiZ334Q2BsR24C9+e1Z985GD8/7mdnkZmPYew/weH79ceDTs7COd7lh9VKWd9W9xdfMpqTd8Avgh5L2Sbo/b1sXEScB8su1ba5jSpJEbF+/wj0/M5uSSU9aPonbIuKEpLXA05JemeoD87C8H+CGG25os4zM9g0r+KufHqeZBrVEM/KcZrYwtdXzi4gT+eVp4G+AXcApSesB8svTEzz2kYjYGRE7e3t72yljzC0bVnB5pMnRMxdn5PnMbOEqHH6SlknqGb0O/BbwEvAUcF++2H3A99otcqpu2bAS8EYPM5tcOz2/dcCzkl4AngP+d0T8HfAV4JOSDgOfzG/PiW3rltNZS7yzs5lNqvCcX0S8CnxknPY3gTvaKaqojlrCzRtW8KPDZ3gwAsnzfmY2vgVzhMeoPf9mMy+fPMePXz1bdilmNo8tuPD7zK0bWb2sk0effbXsUsxsHltw4dfdUeP3dr+Xfzh4mlcHfKibmY1vwYUfwGd3v5fOesKjzx4tuxQzm6cWZPj19nTxmR0b+c7z/Zy9OFx2OWY2Dy3I8AP43L/byuBIyjd//FrZpZjZPLRgw++mdT38h5t6efz/v+YTG5nZuyzY8AN44OO/xluXhvmd//ksLxx/u+xyzGweWdDht2vrap78/G6aafAf/+z/8Rc/epWIKLssM5sHNB/CYOfOndHX1zdrz//2pWF+/9sv8sOXT3Fj7zJue98aPva+69h943WsXtY5a+s1s3JJ2tfyRctX3rcYwg8gIvj2vn6+/+JJfnrsLJeGmySC235tDZ/6yAbu/OD1rOjumNUazGxuOfyuMtJMebH/bZ55ZYCnXjjB62cv0VlP2H3jdey+cTW/sfU6PrxpJR21BT0rYLbgOfyuISL42fG3eWr/Cf75yBkOn86OCumqJ9y0rocPXN/DB9av4MObVvKhjSvp7qiVUqeZTd+1wq/db3KuPEl89IZVfPSG7CRzZy4M8dzRszz/2lscOnWeZw4N8Nf7+gHorCV8cOMKbr1hFR+4vof3X9/DtrU9LOl0IJpVzaIPv6utWd7F3R9az90fWj/WNnB+iJ+9/hb7Xn+Lfcfe4i9//BpDjRQACVYv7WTN8i7W9HSyrqebrWuWsbV3GVvXLGPDyiW8Z2mHv17LbJ5Z9MPeIppp8PrZSxx64zy/OHWeN84Ncub8EG9eHObk25c58avBK5bvqIne5V30ruhmbU8X61Z0sbanm5VLOujprrO8q87y7jpLOmp0d9RY0lFjSWf+01Hz3KNZQR72zrBaoqx3t2YZd33w+nfdf3m4ybE3L3L0zEXe+NUgAxeGOH1uiNPnBzl+9hJ9x87y1qWpH3XSWUtYnofksq46redmqtcSerrqLOuqsbSzzkgzZbiRMtxMqSdiSWedZXmQdtQSaomoJ6KeJNRr2fXRtloikkR01bPQXdpZo6uewOj64oqLMRIIEQQREJG1ddUTujuy5+ioJXTUEzry9bQ+h4BEIpGo1UR3PaHeEvjDjZTLw02Gm+m4r48E9bzurnpCMgMnrxrtFLjHvnA5/GbBks4aN69fwc3rV0y4zHAj5fzgCOcHG5wfbHBhqMFgo8ngcJNLw00GG00uDzcZHGlyYajJhaERLgw2uDDUpDV+hhopF4caDJwf4uJwg45aQlc9obOe0EyDS8NNLg41uDzcpJEGjTRlpFl+b38yWZglDDVSGun06q0nojWzEonOejIWwpAFdCspaxtqNLk80mSokZJILO2o0Z33wOv5h0NNIo1gpJm9lhExtsySjhqJ3vkgGF0/giT/kBhd39V11Guio5bQWUtIEmg0gzRi7GyEHbVk7AMsjSCNLKSl7Fmllg+RJF/vOL9r6++cLSLSNBhJUxrNoBlBd73Gks5k7Pdpptn60oixD8rRH0kkY+tm7DUKslFSmgYBef0a+4CqJxqrcXj0Q7uR0owYt2YJbr9pLds3TPx/NR0Ov5J01hOuW97Fdcu7Sll/Mw/CRjNo5G/QZmSXQ42US8NNLg03GBxJrwiS0asa+8fK3thpBEn+T5gkGnueoUbK4EgzD4osLJpp5L3FTABp/s810kwZHEkZbDQZGknp6kjynmudzprg6p5Y/l8y0gyGmylDIylDjea7ftehvDc80jJXOxpErUHV3VGjuyPrsaaRfXiMfgg10ivDaLT3LDT2wXV5pEkagUiuCLg0gjQdW9tYe2uvebCRB2oj+1u0hkz2d8pev0aaUstDDmUvYNryd0hTxuocrUFc+bqN9dLzOmoJ1JMsnJJEDI2kXB7Jfvc0st+3lq8vTbP3TDN/Pab52dSW1Us7HX7WnuyfqkaX3wE2AyLiit7haPAmynp48M4H1HAjHVummSdnZz3r8XbWk7Fzbl89nQKMPddM8FvfzNomiXrt2sFUr8ES5s9uYbO2GVHSXZIOSToi6cHZWo+ZWRGzEn6SasCfAr8NbAfulbR9NtZlZlbEbPX8dgFHIuLViBgGngDumaV1mZlN22yF30bgeMvt/rxtjKT7JfVJ6hsYGJilMszMxjdb4TfezOcVG8Qj4pGI2BkRO3t7e2epDDOz8c3W1t5+YHPL7U3AiYkW3rdv3xlJ0z3T0BrgTIHaZsN8qWW+1AGuZSLzpZb5UgfMbi3vneiOWTm2V1Id+AVwB/BL4KfAf4qIAzO4jr6Jjtmba/OllvlSB7iWicyXWuZLHVBeLbPS84uIhqQvAH8P1IDHZjL4zMzaNWs7OUfED4AfzNbzm5m1o8rflfRI2QW0mC+1zJc6wLVMZL7UMl/qgJJqmRff52dmNteq3PMzMyuscuFX9jHDkh6TdFrSSy1tqyU9LelwfrlqDurYLOkZSQclHZD0xRJr6Zb0nKQX8lr+a1m15OutSfqZpO+XXMcxST+XtF9SX8m1vEfStyW9kr9nPlbSe+X9+esx+nNO0pfKqKVS4TdPjhn+BnDXVW0PAnsjYhuwN7892xrAlyPiZmA38ED+WpRRyxDwiYj4CLADuEvS7pJqAfgicLDldll1AHw8Ina07MpRVi3/A/i7iPgA8BGy12fOa4mIQ/nrsQP4deAS8Ddl1JJ9GWVFfoCPAX/fcvsh4KES6tgCvNRy+xCwPr++HjhUQk3fAz5Zdi3AUuB54DfKqIVsh/q9wCeA75f59wGOAWuuaivjNVkBHCWf4y+zlqvW/1vAP5dVS6V6fkzhmOGSrIuIkwD55dq5XLmkLcCtwE/KqiUfau4HTgNPR0RZtfx34PeB1hN+lPX3CeCHkvZJur/EWm4EBoD/lU8H/IWkZSXV0moP8K38+pzXUrXwm/SY4cVG0nLgO8CXIuJcWXVERDOyocwmYJekD851DZJ+BzgdEfvmet0TuC0iPko2TfOApH9fUh114KPA1yPiVuAiczv0fxdJncCngL8uq4aqhd+0jhmeQ6ckrQfIL0/PxUoldZAF3zcj4rtl1jIqIt4G/g/ZvOhc13Ib8ClJx8i+Ru0Tkv6yhDoAiIgT+eVpsnmtXSXV0g/0571xgG+ThWGZ75XfBp6PiFP57TmvpWrh91Ngm6St+SfHHuCpkmuCrIb78uv3kc2/zSpJAh4FDkbEV0uupVfSe/LrS4DfBF6Z61oi4qGI2BQRW8jeG/8YEb8313UASFomqWf0Otn81ktl1BIRbwDHJb0/b7oDeLmMWlrcyztDXkqpZS4nOGdokvRusi9N+BfgP5ew/m8BJ4ERsk/UzwHXkU2yH84vV89BHf+WbMj/IrA//7m7pFo+DPwsr+Ul4L/k7XNeS0tNt/POBo8yXpMbgRfynwOj79WyXhOyrfB9+d/ob4FVJdayFHgTWNnSNue1+AgPM1uUqjbsNTObEQ4/M1uUHH5mtig5/MxsUXL4mdmi5PAzs0XJ4Wdmi5LDz8wWpX8FuJ6dll1JujkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5,3))\n",
    "plt.plot(model.loss_curve_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `n_iter` you can check if and when early stopping terminated the computations: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "74"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.n_iter_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `loss_` we can obtain the value of the loss function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.246038492810406"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.loss_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model seems to have converged. Let's now assess model predictions and performance. Let's predict the value of the first five test units: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([31.41147816, 26.80401856, 16.73545148, 21.00468812, 18.65367359])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = model.predict(X_test_scaled[:5])\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, last, let's check the MSE on the test set: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15.045212757099515"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "mean_squared_error(model.predict(X_test_scaled), y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Classification\n",
    "### 2.1 Binary Classification\n",
    "For the binary classification case, we use the \"circles\" dataset already encountered with kernel SVM's. Let's load it and view it again: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAAE9CAYAAABZd+pgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO2df5QeVZnnv0+/6d7YAY7yIhrA7hDkuCS4EQwRZoDDqsti9Cw6YxidHswRJEOzOHKOc87A5qzjzpjZ4+jRgyMy0+MgYPfirsy4w1GEEXaYnBEROjHhV0wMoRsyoIboIENAEvrZP+p9k+rqulX3Vt1bdW/V8zmnTvdbb71V91bV/d7nPs/9QcwMQRAEoTgDdSdAEAQhdERIBUEQSiJCKgiCUBIRUkEQhJKIkAqCIJREhFQQBKEki+pOgAuOO+44XrZsWd3JEAShYWzZsuU5Zn59cn8jhXTZsmWYnp6uOxmCIDQMIppN2y9Ne0EQhJKIkAqCIJREhFQQBKEkIqSCIAglqU1IiehNRPSPRLSDiB4jok+kHENE9CUi2k1EDxPRmXWkVRAEIYs6o/aHAHySmbcS0dEAthDR95j58dgx7wFwam97B4Abe38FQRC8oTaLlJmfZeatvf9fALADwImJwy4GcCtHPADgtUS0tOKkCoIgZOKFj5SIlgE4A8APE1+dCODp2Oe9WCi2guCMqSlg2TJgYCD6OzVVd4oEH6m9Qz4RHQXgbwFcw8y/Sn6d8pPUmaiJaAOADQAwMjJiNY1CO5maAjZsAA4ciD7PzkafAWBsrL50Cf5Rq0VKRIOIRHSKmf8u5ZC9AN4U+3wSgGfSzsXME8y8mplXv/71C0ZwCYIxGzceEdE+Bw5E+wUhTp1RewLwNwB2MPMXFIfdAeAjvej92QCeZ+ZnK0uk0Gqeespsv9Be6mza/yaASwE8QkTbevv+G4ARAGDmvwRwJ4C1AHYDOADgozWkU2gpIyNRcz5tvyDEqU1Imfmfke4DjR/DAP5rNSkShPls2jTfRwoAw8PRfkGI40XUXhB8ZGwMmJgARkcBoujvxIQEmoSFiJAKQgZjY8DMDDA3F/0tK6JXXQUsWhQJ86JF0WchfGrv/iQIbeGqq4Abbzzy+dVXj3z+ylfqSZNgB7FIBaEiJibM9gvhIEIqWENGAWXz6qtm+4VwECEVrNAfBTQ7CzAfGQXUNjHNqkw6nfTfqPYL4SBCKljB5SigUCzdvMqkP7w0iWq/EBDM3Ljt7W9/OwvVQsQcycf8jSj9+MlJ5tHR6PvR0eiz6rjh4fnnHB5WH18no6Pp96DbPZLXJUuYBwai/Z0O8/h43akWTAAwzSmaIxapYAXVaJ+0/WmW2+/9HnDccQutzSrHu+tavqrjVENH9+8/ktcXXwQWLwYmJ4FDh+xF60Ox2htLmrqGvolFWj0mlqPKckv7jaml6zr9Wcdl5Su5jY5Wn3ahPFBYpLWLnotNhLQedJvrKnFMExmVOBURoqz06V4n67jxcX0htVkR2LxHQjYipII35FlucZHRsbZ0BDzvPFnpiZNlIZtapHErttOZv9+Eqqx2QYRUKIiulWl6ziyrNGlJZaVBt1mbZ7X1hSy5dTrzr686rv+9jogOD0fWazLdRZvlYpFWhwhpyykiiC59b1lC40JE8qy2POHL+z7LRxqP2ictUd3KJAvxkVaHCGmLKVrQXFo6WaJjIvq6zdq8vKi+V1mg8e/7XZhM7nOe9ZrVLE+7P7ZaDi5aIE1ChLTFFBVEG743VcFUiU5akzdL9HXzlidyqu+zxE51Hh0hKmqRurQ+xbLNR4S0xRQVxLIWqY546TZ5bQhLnsiZpKeslZ6WbpsVRxHE15qPCGmL0SkgquZiGQulSMEsIvoum6NZglfUSk+mG9CP2ruM0Ev0Px8R0hZTtFlb1vdWpGD6aBXpRu2roA6LtNMRn2kfEdKWY6MzuilFzuu7n67u9JmMwDINSOlY30X8wU1ChFRQ4qpJl1Yw+9fqdqNN1TfU1wI6ORmlu5+ffi+DqtNgKohDQ8yDg2YCnGV9112h1IUIaWBUKSYm/R9NifsBs7r8hFAIQxEPW2P+i4zianpgSoQ0IKousLoWDFHxad90CrfvhTAU8dAdYdV/pqpKOyu/bQ1MeSmkAG4C8HMAjyq+vwDA8wC29bZP6Zw3dCGto8AmC1O8+ZpW8EzRKdy+F8JQxMPEIu12swONpjNd+Vap2MZXIT0fwJk5Qvpt0/OGLqQ+FFiTsfA6iEVaHSY+UlWF2c+T6YAK39wctvFSSKN0YZkI6XzqKrA6gYaigp4XEQ6hEFYpHmV95LpR+zKVts9BQVeELKT7AWwH8F0AK3XOGbqQ1lHb63Y8LyPo8YLXj9oD5aaQq5oqxKPK5x+Kle0LoQrpMQCO6v2/FsBPMs6zAcA0gOmRkRHrN7Bqqq7tdf1qRX2kabS1eZiHLXHTeYfkGZgRpJCmHDsD4Li840K3SOvAJNJrC7GG0rE1WYytOQiEI6iE1OvF74jojUREvf/XIFo+en+9qWomqsXrkoyO2rumarE41f62YLKQoAqTRQPHxoCZGWBuLvo7NqZ/HSGiViElotsA/ADAW4hoLxFdTkRXEtGVvUM+COBRItoO4EsAPtSrFQTLbNoEDA9nHzM8HB1nCxuC0UTSnoXpvZdKqmLSzNTQN2naFyPZxBsfd9vkE/+cGpPmto3pCAU94KuP1MUmQlqeqvxm4p8rh6oyMp0gW9BDJaQUfdcsVq9ezdPT03UnI1impoANG+b72IaHgYkJ8Z/5xrJlwOzswv2jo5ErYOPGqDk/MhJ9ludXDiLawsyrk/u9Dja1gampqDAMDER/p6bqTpFZoEKolyxfqASRqkOEtEb6lt/sbNT4mp2NPmeJaRXCK4GKcHAVsMt6z3ys/Gsnrb0f+haKj9Tl+kRVpkuoD9vvRHK+1eQ52x4ghASb/MO043VVAtf2whIatgJ2ecOER0fdzl0bAiohlWBTjWQFCmZmFu4fGIhe2yREkR/MJlNTEqhoG6r3sU80NCb9HUzS1OCkBJs8xLTjdZUd2CVQ0T7yfOAjI/rvWtuCkyKkNTI2FtXao6NRbT86ml2L2xjxIggqskSy/57pjIDr06bgpAhpzZhYfqbCKwgmqESy2z3ynqW9g91u+vnaNNRXhDQwpMktuCJNJCcngeeem/+eJd/B669331LyvcvVoroTIAiCP/StTtPfAO6Ck8mRdv3+1vFr141YpBXje80qCEVw2VIKYaSdCGmF6IxkEqEVhPmEMNJOhLRC8mrWIkNGG4fUJEKCEOatFSGtkLyaNYQmjFOkJhFSCKHbnwipJXQMqbyatYomjNcGX+trEiGNILr9pY0bDX2reqy97tj0vONcj6X3fgy9jVXfBMEhCHHxu1DQNaTyalbXTRjvDb4QnGGA52a9UAtp6hr6VrVFatOQcrn0hvcGn/cmM4eRRksUfRer/l2VQKbRc0co83cGkU7fS1MQN7E8ReuLqn9XNSKkDgnlJQglnV7jvVlvh6L1RdW/qxqVkIqP1AJBRBURTjq9JhQ/bkmK9iCp+ne+UKuQEtFNRPRzInpU8T0R0ZeIaDcRPUxEZ1adRl1CmUwklHR6SwidGi1QtL6o+ne+ULdFejOAizK+fw+AU3vbBgA3VpAmQVDTErO+aH2xaRMwNDR/39CQ3u9Crp9qFVJm3gzgFxmHXAzg1p574gEAryWipdWkzh7SW6YAtm9amfMlfwu4Nes9eGHK1BfM2Z/zrgcAnc6RrnlBlJc0x2mVG4BlAB5VfPdtAOfGPt8LYHXeOX1a/E4CPBokI/Xj4/aXxix6vqofYOAvTF7QKK9Thu/Zh69R+xwh/U6KkL5dcewGANMApkdGRqzfwKLoRCN97/HjhH6m+xHvZATcZgi3aEh4cpK507GbFldp9YSsTg06Iul79kMV0r8C8OHY550Aluad0yeLNK+3jO81sBPy1v3N2rJujKpGKtJlKS+Nrro7edy9SqfCzxJCHZH0OPvMHK6QvhfAdwEQgLMBPKhzTp+ENO/l8b0GdoIq0zqbqpbJqpGK3OS8NLbMIrUxn4SOSHqa/cN4KaQAbgPwLICDAPYCuBzAlQCu7H1PAG4A8ASAR3T8o+yZkOa9gKqXq//yNLK5n5VpnS2tVGWVwCJmf1YaW+gjNRE4leWq6+byMPuH8VJIXW0+CSlzdpNI9XIly/HgIHO32xBhLWORJk2YPjo+FJOaqdtVn8/14PHx8SO+2U4n+lwzNprcKpEcH18Ya/TViBAh9ZS0l0vHYPOpljYmK9Ojo2oRK2qRmqYtS+i7Xb1zqPLQ7eb7eT00yWzfXlcdNFwjQuoxyZdL1zDzxW9UiCxrLSvQU8RHapKmvCBYngmmc46sdHnqJLSt73n1la/vtghpQOiKqS+RTCfES1q/mZvXzivymzg6Nz6rhGd1l0punU56JeJx2NpWN70i9ZUvXQRFSD1A92XQ7R3kZa1d9xtfxnTK86lknadMl64ql0nwANP6yidvhwhpzZi+DHE96naZh4b8eJEy8eGNLyNEeW3NrHyUDaDFh/4MDs7/bnDQw4ddHNP6yqe6RYS0Zsq+DHUbelr48MaXaRq7tGbztnjvgmStOTTk6QMvhml95ZO3Q4S0Znx6GZxhO5NFao+6aixbFqkPlZFjTOsrVQcIsUhbKKQtKB92em3Hv3exZoVN0z7P/6JrpeqM0GhUjZv+GFT7krcVqM/bIUJaM6qukx70tbZHmXGE8X6kRYd1xs+fJpY2fbhpvsyBgfmjJtI6SaYJZPwlaEWNuxDVo8nqjlsHIqQ1MznJfNRR6eUorh/Bo2Px5TWDXU0WYlOkdEt4/H7ozCTlQ8CuBkw9I3UZ6CKkNaLbM6bx5SWvF3Z8czF9nc2JDbLSbnp9XztNVsTkpJmI1mmgi5DWSGtGK6nIGjJpYpmWrWl0JzbQuU7eQzSZWKHTabxYqsgzMrpdvwx0EdIaMekZ07CYQvGO6nFfqS3LzGRig7waTbdiSAa6TIe+2sJTKzfLyOjfEp+SLkJaIzYtUp9eKi2KdAtyPU2dzsQGOmPq08LJeQ81axipq+aISsDzJlCpgCwjw8d3W4S0Rmz5SIOMQ+SZ42nLjFTZlcFm7wBdUTbt4lS29tQx+2oitE4KIqQ1o9tvLovQXjpmzhcYFxkyudkmtVPeA9N9QKb9bcvWnnmVWbdbWzPHpXHgovUmQppCaM1kJ321XXRQB440X9M6qvf361pwJmlOK5mDg9mTFejcA50Sr6sK4+P6AS4btaepe6ViK9VFOXQl0CKkCUJsJlu3SG13UFf5L1TT+xfJUF6aXXWR0E1rkRFbWe4MV1PT27w3HuKq9SZCmiDEZrJ18bd5E/IELG3+zSIZykuzqy4StpoDpvc8q8uUiQln2gUt8O4jrkbaipAmCHVIs9VmkM2bYCJgZca+56W5bou06H3KCjSVmXE/7Xzx++3TjCAWEYu0IiEN0SK1TpUWqa0brbpOP2DSF6X4d3k+Uh1sNQey7lOWf9ZkmKkJIfq4NBAfaUVCWuRG67i/QgpelX7b4hlWBZVUW5mp9XSCSclJDGw8HFvnyLIw85aLzbNoi6QxuBdXD4naVyCkzGY3Ok9zTGMI3lD0bVMJWr+pGF9O2LbpH3rz1NRfGa8QsloRDbUufcJLIQVwEYCdAHYDuDbl+wsAPA9gW2/7lM55XfQjzWsFq74nauh7bBLFdl24Q3N4F4mi9+9b1vrF4q9yjndCCqAD4AkAywEMAdgOYEXimAsAfNv03C6ENK+s5k0sFDxFR/Gk/dZ2zRKagJj6k1WWafJ+5r2EjazRq0UlpAOojzUAdjPzHmZ+BcA3AFxcY3oyGRnJ3q/6HgCeesp+eiplagrYsAGYnY2K5ewsQJR+bNqNGBsDZmaAubno79iY3fRt2gQMD8/fNzwc7feRMi/E7Gz0N+1+Zr2Es7PRM5yaKn5tQUmdQnoigKdjn/f29iU5h4i2E9F3iWil6mREtIGIpoloet++fbbTmltWN20y0xYvmZoCli0DBgaiv/1Ct3EjcODA/GOZF2a4LvEaGwMmJoDR0ShNo6PRZ9uCbYuyL4RKENNe0jgHDkTPUrBPmplaxQZgHYCvxj5fCuAvEsccA+Co3v9rAfxE59yuxtrntVDTRv4FMwN+li/T5oTI/Ws1MEqsjepemzbxVec2db0I2sBDH+k5AO6Ofb4OwHU5v5kBcFzeueuctCT+HheZL7g2svyMJj7IIrWN1zfGEWn3ycR3mieIRfqrCrn4KKSLAOwBcDKOBJtWJo55IwDq/b8GwFP9z1mbD7M/hRb/yIym6Ube844bH88u3G3HJJqfd7/yztXGyssC3glplCasBbALUfR+Y2/flQCu7P1/NYDHeiL7AIDf0DmvD0IaWo+czDHduh3a8/o4ZrkIvL0xFZPVpDEVwTwrVyovY1RC2rf2GsXq1at5enq61jQsW3YkwBpndDQKtHpHPzKfDCoBUQBDJ3gzMBAV0SJ4e2NqQvUCdTrALbeYBdJUz4UoivwL2hDRFmZendxfZ9S+0YTWI+dw5LvTWfidbrS3aDSayOMbUxOqLlJzc+a9EfL67gWOqrNJpaSZqaFvNpr2vgzNdoZpp26dpndaIElnKzuO1usbXRCbTvYGDx2tOmvw0UfqaisrpA1+7yKyMli0ABcZ9mhjMoKmPiybE8qMjkb3uWmVDVcf1BUhNSC4iLspRSa+yCuIRabRs1GYm/yw0ixtHeu7qZVLClUHdUVIDQgu4m6K6VRsWRNl5J0zubV9wpIy6Apk1pytDUMsUo+FtMlGDjObZ1Dn+KzCKxOW2EE3rzYXi/fc/yw+Uo+FtPEtI9MM6lh9dd20xj+sGLrWd9ZcpyYVTCD3tkqtFyFNIesBeF4Rl8ckg7rDDeu6aY1/WD10LdIyS10XuV6LECFNYDLqsQ1ldAHxjC9Zoi6YnlopjUT3pbU1Oa5n/mcfyqIIaQKdyjaQlo19inRlarGVUillhuqaLtfgkUXqS1kUIU2gU9l69B5VS5EZ3JsYJQ+VNNUp0mfXo5m6fCmLKiFt7RBRnVFzqlF6wc94n0eRDDZkuGEjSJvo+utfB77ylej7/phKImDRouhvcmzl1FQ0pp/5yD4iYP36SibMTg77TJt2AIj21zYsNE6auoa+2fKR6taCPvhurGJqkbbC39EQstw28edYowmoMqh9eAUhTfuF5AlgltgGO4GzDpOT0dLKqrc2ufZ6Q4cfNpK8SrIvlDUGmrJcvFlJd91lmVmEtDBpg3x0liQP2o86Ock8NKTOWLKfmA9RAEGPPDXqC2WNFmlep4M6G0oipBYwCWYHF3uJ1xidTraIxmsWVa0SdE3SYHQt0horyDwNNxFT26+hCGkJ8iYaD15HsizQtGq+kTVJS9D1kfaPrcFlk6fhdRo0IqQFKdKlMriWrY6vwnQLqiZpGXHLoN/68My3rRO/qKNhpBJSWWokh6yuF3GIokc3OhpN9u7rkuqpJNenL4vu0iSCYIm0lXJcvIay1EhBdLpUdrtRNz3maNmhxurHgMbrMjoqIuoaW2treLFGhx3Sus5W+hqmmamhbzab9rrzdXiLjp9Lp2k/NMQ8MJB9jPhF3WNrkgjpbVEIFPGRAjgGwCkp+/9D1u/q3lz7SIN530wKXbLf6MDA/L6ije/zFQg6NbvN0SaB4To+ZiykAC4B8AyAbYjWlj8r9t1W1e9MNgAXAdgJYDeAa1O+JwBf6n3/MIAzdc7rKmofXH9zk8KSt6xF4yJsgaIzxEcn8uLZzE42qMLoKSKk2wAs7f2/BsCPAfxW7/OPVL/T3QB0ADwBYDmAIQDbAaxIHLMWwHd7gno2gB/qnNtVP9LgKFNYTLorBFW7BE6RCWXSnnsDLdIqsqQS0qzoQYeZn+35UR8E8B8BbCSiPwDA5t7YBawBsJuZ9zDzKwC+AeDixDEXA7i1l4cHALyWiJZauHY7KLOe+caN80OgaQwPA5OTDY+wecamTdF9L0L/uU9NAf/2bwu/Hx6Ozh8odU4ylCWkLxDRKf0PPVG9AJG4rbRw7RMBPB37vLe3z/QY5+gEN70MgKYVOlVh0Z1uB6gpLCoAmB+eVjE0pH7u/X5C+/fP/77bDf55lrEbSpNmpkYWLFYBOBULm9uDAC5V/U53A7AOwFdjny8F8BeJY74D4NzY53sBvF1xvg0ApgFMj4yMWDPldfwuXgekdBy8JtPtBNz0axxZAcD+RDLA/E73DR7S66WP9PABwKMA/giRn/I1AP4CwA/yfqdx3nMA3B37fB2A6xLH/BWAD8c+70TPb5u1VdH9Kf7eBe9u0p1ux5vaQWDm/Nk9Gj05RDp1Re11OuS/A8CbANwP4CFEkfzfNLV8U3gIwKlEdDIRDQH4EIA7EsfcAeAjFHE2gOe557etCh2/S9ATQE9NqZvxzDX2cBZSibtgsgZIPPWUnp+7z7HHWkle3YyNRS77ublqXfeLNI45COAlRNboYgBPMvNc2Qsz8yEiuhrA3Ygi+Dcx82NEdGXv+78EcCeiyP1uAAcAfLTsdU0ZGUnXmbjfRecYL+n7y1SMjkZvo+AHyXGQr76qPnZkJJCavCGkmanxDVG3pD9B5Bt9I4C/B3B73u/q3KrskD85mb7IZhCt4KyuNEFkoGXodn3qPzuTrlINadq7BiWa9pcz86eY+SAz/5SZL+6JaSvIGsM7NQVcdhnw4osLf1fR0jblyLJY8prxXnZTaDhZzyv5ggLpXZxUE9R433zynDR1DX1z2SFfZ/5jIAqOek/RRanGxz3uptBgTJ5XWpCp22V+17taG0S0EYiCzEdaHtO5Sb2naN8u3a5RwY6t9RTd/j0qwe127SzTHCC2ukaJkFrAdHReEOSJXVE/m9edawNGp3LSmRshqwIMANM62lYXRRFSC5i8nyZNe68NN5NMN6pzbcCY1viBBZqK1NG25mhRCalM7GyAqT9eJ/7S79EyOxs92tnZ6HOtsRudvorJoEVy6GnQnWsDRzU0uNtNPz6wQFNa99gDB6L9KpwPH01T19C3Kn2kw8PqJZp1WrLeGW46juB+pou4BMQirQbVtIgNcLcUsS7FR+qRkDKrm+FFdcM7H6sqI52Ome8hK3IcWMH1GlO/kNd+JD2KljWJ2nskpCqK+mBUXag6nWrSvYC88dumYlrUVBcWIt3QmLlew1qENAMbNVXjLdKifQ+liW+HtOVgsiq8hqMqs94tNRLyZiKktmq3oufxTmdMZ8bPo4FLWtSCzppZLb+3Xk+jF+JmIqQ2haxIbeil/z8+Tjurqa9TYL2rKQJFV0Tj97YB/lAT6lxqpHbRc7GZCKkPBpO377tOf8S8BHtZUwSIiTU6Pp5+3wcH568M27BnUEVZFiFVIAZTBrqd8fOE0duaIiBUTfvFi9P91zqugIZVaL4uftcKdJY1au1ER7q9lfN6Q9c1226TuP76aC2mOENDwJIlkV7EOXBg4ZpMaeQ9t8DIKsvOy3Cauoa+2Yzat7plajpLi1ibbkl7UU3H1dfpw6oA1+MQIE37YrS+6Z98M9NmsW5lLWOZou4Pk5meWvYi581tXSTrIqQF8SEY5QUmM643uHA6oYzJlPXbuDh3u8xDQ62p9HQaU0XKsEpIW+8jzaPWtbJ9IT6zig4yMYkZRWbh6JO1hEOco44CLr+8NYsZ6qz7Z7UMp6lr6FuVaza1AtNp2UwsUonou2n2tPzFzXMdi4+0YiFllrJuFNAweUNbXtgPY+qI13khW+7cz/ONFn3FREiF4uRZpAMDxWqZlhf2w5hUKLrHtty576qOFiEVijM5mS2kRd/OKgu7780K3fTpVj5SSTl55F4JKYBjAXwPwE96f1+nOG4GwCMAtqkykLaJkJZA9fapRsqUWS5Vt7CXLRFNmhtVt/IRt4kTfBPSPwdwbe//awF8VnHcDIDjTM8vQlqQvK40WStQ2pyxJT77/lFHLRQNU0HIck24EBeX1q+qQkuzNH23wgPENyHdCWBp7/+lAHYqjhMhrZI8C3F8PH1cd5kJhnUmKy7bRM0Lltls7rq0BCcnF/YFBaLJSEQkK8E3If3XxOdfKo57EsBWAFsAbNA9f1EhbX0Fntds1J30uUyzX7erlYkfNe+cNn2yLn2TWaOYAkWnzOUN4a6yzFYupADuAfBoynaxgZCe0Pt7PIDtAM7PuN4GANMApkdGRoxvkLiUOF8EiozrNr2ButcwXe86y8q1aZG6DKA1LBKvU+ZMvU2uy6xvFqlW0z7xm08D+EOd8xexSCXIyflvrWqBKZvNZl2LdGjIrMRMTlazfpTLmcJN/KMBoHOrVMdkvYoub4dvQvq5RLDpz1OOWQLg6Nj/9wO4SOf8RYS0YZV9cXSnz9HdTG+gybWKzATvui1oy0xSTczcoPHyOmWuSCPIZZn1TUi7AO7tdX+6F8Cxvf0nALiz9//yXnN+O4DHAGzUPb9YpJbRsRIHBuzdwORkG1klxkefTFnHH3O2P7QhjvwyFqlO/eoCr4TU9VZESH0sj96gM3DZ5dLAWSXOpxpQ19rVedlM+osWEVYPIqtFfaR5r2JrfKSuN1dRew/evXrQHbisukEuOtT3S4wvPhmTmriMKRY/pmjt75HVYGq85/lGGxe1r3Nz0Y+07LsXtAiXybxNn2HaDfTFIjVJh47469y3onn35Z4VoO46QIS0JGXevbofvhWK1gRFb5zNZnIVmFjGutZm3D/cP39/0EJWKyHPGvfFii9InUaJCGlJyrx7ARsA5Sly43SdZ/GAVN3LDJs85Lz8lekl0WCL1IdWnQhpScoEUYM0AGy9tVkdAU0j1vHuTibTzsUtO1cTlZhaxln3t0io2sQa98WKN8CXJIuQliTtQQ4NRV378h5ucAaAzbdWx7oyjVjrDpWcnHQ3Nl3V39ZG5VN0ZdB+JEbn2j6YdwZklaEqsyJCaoGiA018qU21sa38OmFXk4h1ltDoWnZlajHbDzT5YuWt1KqzNWwik6xHbvooygivCKkDTJrsQRkAdY8XzxMqXYHMKn1l8qLr54lPB5jVt6eMPzRrC3gykyRZHiKTerJsHShC6oDgmuy6uB+smUwAABcUSURBVMyY7rmzap7JST2BdGWRFm16m/h9stKdFOis4xuCSgBNs1721RYhdYDrJnttVqzLvp+qOU3zemInb0DahM/JEpHlIx0fn++b6Q9x1bnRWcNWdYQwni/T36bRAiFlTn8dVMLYHz2cpGxjS4TUEa7Erna/qovRSGnRufgs+3m/j89ElTxP//zJdKZF7cfH0wU2vmVF98sIqY4ppfqNaXoa1LRXMTmpFse0ekcsUk+F1BXBuw1Mmqymb7xu1L5s2lTiVbRpDxSfijAvepKsGEynGQwY1W1TxSrER6q5NUFIg+x7GsdUbJJvctYN0I3a20ibicjrTO5iKqK6DzyoaOZ8yibd1OiQqH2LhLRVFmmaWVDEItU1L0zTlkRl1iSDQGlR+yLBpcDJixuWdWFV6QYTIQ2A5KhHnc7+3qLrI1WJRp6PtMzSIaoglMoiNA2Emd6XLOs7mAeeTtGebKb1R1UGuQip56h0p+4h5KVQjf7Rbcba6AKVlba0SUGqsAqT+Uqby1UVhAuMomMrfHVhiZBWSJHaUbePd3BimoYtM8S2/yOvie3yIQTs48yi6GhfXz0aIqQVUdRfoxv/CKp5r8JmP1WbzjHdIFIjHkI15All7d38DBEhrYiiNWzZ3kLBYWKB5TXxbVlyJv7LRjwE9+gIZVWP1wYipBVR1OdjMuTaV/+RE6owWZJRvrhjWh5CaWzG5eq2VkVIK6KMzyf5wjVsGfNiuA7rVhFW9s2sCgQf/acipBVhsxb1sUauHBth3ax+n3nTB5V9CPIQC+NjRF+EtEJsu+1abczYMEuKjERKTutX9CH4aFYFgo+3zishBbAOwGMA5gCszjjuIgA7AewGcK3u+esWUsEiNiw6Hcezq9Lqo1kVCD4a8yohHUA9PArgtwBsVh1ARB0ANwB4D4AVAD5MRCuqSZ7gDWNjwMQEMDoKEEV/Jyai/bp0OmbXJAI2bTL7jYqREbP9wmFsPPqqoEhka7o40X0A/pCZp1O+OwfAp5n5P/c+XwcAzPw/8867evVqnp5ecEqhrRCZ/8ZWuZiaAjZsAA4cOLJveNhfRRAyIaItzLw6ub8ui1SHEwE8Hfu8t7dPEMwYHXVz/NQUsGwZMDAQ/Z2aWnhMSGaVUBhnQkpE9xDRoynbxbqnSNmnNBOIaAMRTRPR9L59+4oluiKS5e+qq/LLo5BAR8T6bNoUWYFxhoeBd71robU6PKzXrO9bmrOzkfU6Oxt9VonpzAwwNxf9FRHNxOTRekOa47SqDcB9UASbAJwD4O7Y5+sAXKdzXp+DTTod7+t2qHtPkShEVj/SIhF5H0PKDcDHAFMc+BS1P3zxbCFdBGAPgJMBDAHYDmClznl9FlLdoaBSHjPwQcQkGm+Ebn3lw6PNQiWktfhIiegDRLQXkdX5HSK6u7f/BCK6EwCY+RCAqwHcDWAHgP/DzI/VkV6bPPWU3eNaiermVHnTJBp/mLymuIkXxIdHW4g0dQ19E4u04fhgtvjeBq0Indtg8rh8eLRZwCeLtM2kxT2S6MY7gEAd82VRBY9s9f3UQaLxmJoC1q+f37MLiD5v3Hjks4mV6cOjLUSauoa++WyRMqdPkG5zdpyi5wuK1o+drZe8oGncVawzJ6mN8lAF8DHY5GrzXUjjuBjGLfMQW8bW3KkeYrvTQlpT3HT5LZ/fVxFSDyn7EpVdVVjQwOQhBaYKZZKb9e6lnaMv2MCRCbdCnCpShNRDyjrWTWbVl145BWlSpCRBmeSqftvpqIW4CZOXq4RUgk01UrarR5pjXjWsPKtXTisDVrqYPKTA+u6USa4qKHTLLep428aNCwNTKkLrRSZCWiNluyKmBY6vvNIs6mnSx6+VmDykwPqWlklukU4LuvVJEFH6JGlmauhbKE17Vy41kwBCYK3R6hEfqTVU71pIS45DfKR+UneQV0Y6alAwav9Cd5Q/3p30WiCqfP9sC3cdZUeEVEglyyKtW+S9w+CGBGacVsLk5PwofbdbTkTruL8ipEIqWZ36RQhiGJbcNrlMdOoX28JX1/0VIRWUpBUE09EojRdYw5Jr22Xi6/3WFUjbwleXS0qEVDAi60VtZbPVsOTaFA6f77duhZt2TBnhE4tUhDQIsl7UNjVbD2OYaZvi5/P9VgkkoNcBv2gexEcqQhoEWS9qKyP9BUqurea4z/e7P9wzuXU6+SPvygqfRO1FSINA9aL6bCE5pSZHZZX32zSLWUKZNR7fJz+vCSKkLcRVubfZrPI1iOITVTVji1ynbS4gEdKW4brw2RBAn4MovlHkfpv+pojwNWmKPB1ESFuGDWvAtbXYRIvFF0xFbHIy/Vno+GKz3pOmtThESFtG2QBFFdZE0TRWUThDFwCTSiovui4V2xFESFuGq7lObRYq201JWzShSWpSSWVF10PLt2tESBtMmvVUVgxUBQuwm26bwQ1bNMHlYJKHrOi6iOh8REgbSp6zv2jzNKt/oO30m6Sxij6VPvfb1CXtvejnK3mfm1BxVIVXQgpgHYDHAMwBWJ1x3AyARwBsU2UgbWuKkOqIjItCkBV4sGmRFkEsUn3iwzOzFkRsgiujKnwT0tMAvAXAfRpCepzp+ZsgpLovt4vJMXwLPMQrlG6XeXDQbaFvmrDoVAyhB9eqwishPXxxEVIlulaRbevJt8BDmqgNDUWCWrYPa5ZwNEVYynRrEhYSqpA+CWArgC0ANuScawOAaQDTIyMj1m9g1ehamrZHGWU16esQE1euiyZZnCryWhfdbt0pDI/KhRTAPQAeTdkujh2TJ6Qn9P4eD2A7gPN1rt0mi5TZ3Sijupv0zG4CP6H5QIs+X9Wa8XHLvmmVh2tUQrrIeLU8TZj53RbO8Uzv78+J6FsA1gDYXPa8IbBpU7SaZ3z5WtXqimNj2as36pC1VG6dqzqOjEQrm6btL0pIqyb3V3ntP5v+Kq9A9jOfmgL2788+95IlB3HgwF7s2PGyncQ2iMWLF+Okk07C4OCg3g/S1LWqDRkWKYAlAI6O/X8/gIt0ztsEi5S5Wj+dr30JXTTDQ7JIi6Y1bwo7gPn66/fwD36wj+fm5irISTjMzc3xvn37eM+ePQu+g8IirWVdeyL6ABHtBXAOgO8Q0d29/ScQ0Z29w94A4J+JaDuABwF8h5nvqiO9dTE2BszMAHNz0d+yVmcWKgtvdNTtdeNMTQHLlgEDA9Hfqan566cDQKcTWWcbN0bfF2HTpsjKjuPrWupFrWcd6/rNb34Zr3lNF0RknrAGQ0Todrt4+WUDSz1NXUPfmmKRVomu5VfX1Hy2LVMfovIu+wnrWKR33fU4P/ec9Ww1hscff3zBPvgYtXe1tUVIbYrB5GT+Urkuo915ghFSc1wHk4qryD1P+93g4PxuYw89tFAoqubZZ5/l3/md3+Hly5fzaaedxu95z3t4586d/OSTT/LKlSudXPPll1/mSy65hE855RRes2YNP/nkk6nHiZC2QEhtd3tysRKkidBn+WjzvrNNFdZqFb0y8n6XJhRlzmfK3Nwcn3322XzjjTce3vejH/2IN2/e7FRIb7jhBv793/99Zma+7bbb+JJLLkk9ToS0BUJq00LTPZdJVyRToddpilZhkVbVx9SH8fwmQurivtx777183nnnpX4XF9Inn3ySzz33XD7jjDP4jDPO4O9///vMzPzMM8/weeedx6tWreKVK1fy5s2b+dChQ7x+/XpeuXIln3766fyFL3xhwbkvvPBCvv/++5mZ+eDBg9ztdlMDbiKkLRBSmwVR91wm4l3Ees1bcTK5uRC4olP7xYew6oy68sFVYSKkLtJ7/fXX8zXXXJP6XVxIX3zxRX7ppZeYmXnXrl3cL9+f//zn+TOf+QwzMx86dIh/9atf8fT0NL/73e8+fJ5f/vKXC869cuVKfvrppw9/Xr58Oe/bt2/BcSZCWkvUXiiPKspepH+l7rlMot2m0eZ4dF43iPya1+gdZ0Jan1VAne5+P8/Z2Uha9u+PNuYjfT7Tehfo3Mu0Xgx1UWff24MHD+KKK67AW9/6Vqxbtw6PP/44AOCss87C1772NXz605/GI488gqOPPhrLly/Hnj178PGPfxx33XUXjjnmmAXni/RwPqV7LqSpa+hbGyzSOnyk/WN1/GSuJpbOmsWoLJOTautclW4dl0QRH3IVvSjqtkjvuecerab9H//xH/MnP/lJfvXVV/ngwYPcic3l+C//8i88MTHBp59+Ot9yyy3MzPzCCy/w7bffzu973/v4ox/96IJzS9NehHQetqP2NgMJZYXepKlftjkcn25OJd6qdGcFwsq4W3RnbCpzj+v2kc7NzfGaNWt4YmLi8L4HH3yQ77vvvnlCes011/DnP/95Zma+6aabOLL/mGdmZvjgwYPMzPzFL36RP/GJT/C+ffv4+eefZ+YocLVq1aoF1/3yl788L9i0bt261PSJkLZESH2nrDgnu2TZFKr4NXQEW0UZizQLnZFmZa3EuqP2zJFFuW7dOl6+fDmvWLGC165dy7t27ZonpLt27eK3vvWt/I53vIOvvfZaXrJkCTMz33zzzbxy5Up+29vexueeey7v2bOHt23bxmeccQavWrWKV61axXfeeeeCa7700kv8wQ9+kE855RQ+66yz+IknnkhNmwipCGljcCVUts6fJ8RFJwbRmc6wbMDRVEjbhgSbhFpwERzJC2aUHdpZ9vzJIaxJmIulKy0Y1ac/RNZmwFEoSZq6hr6JRVo9rvpfZllmNpqXNs9vOyCTNylzlT7SNiJNexHSynHVLzJLLFzNw5onRqrr1jF3alVR+zYiQipCWjkuR+qkiYXt7l+6YpR1XVVgzNfZ/EVIsxEhFSGtnKpH6tQ1Mkh13W43Ciwl9w8O2ulK5mLsvwhpNhJsEiqn6jk+6xppozr//v3AK68s3H/MMeXnc61yXlqhGCKkghWSQzxHR6PPrgp9XRFr0/P/4hdu0tEUfvrTn+JDH/oQTjnlFKxYsQJr167Frl27MDMzg9NPP93JNTdv3owzzzwTixYtwu23327lnCKkgjWqtJzqmuVedd1uN/34osLu0zj7w1hOFDPjAx/4AC644AI88cQTePzxx/Fnf/Zn+NnPfmYluSpGRkZw880343d/93ftnTStvR/6Jj7SdlDXLPdVBL+qmMrPyEfqIFF1TaPXZ/369fzNb35T+b0Em0RIa8eHpTyqxlaeqwqkGQmpg0TVNY1eH5tC6mw5ZqG9FF1COHRsLIsNeLpcdI2JOnjwIK6++mps27YNnU4Hu3btAhBNo3fZZZfh4MGDeP/734+3ve1t86bRe+9734sLL7zQefoA8ZEKDti48YiI9ukPaxTy8XLop4NErVy5Elu2bMk97otf/CLe8IY3YPv27ZiensYrve4R559/PjZv3owTTzwRl156KW699Va87nWvw/bt23HBBRfghhtuwMc+9rHC6TNBhFSwjpcWVUB4uVy0g0S9853vxK9//Wv89V//9eF9Dz30EP7pn/5p3nHPP/88li5dioGBAXz961/Hq6++CgCYnZ3F8ccfjyuuuAKXX345tm7diueeew5zc3P47d/+bfzpn/4ptm7dWjh9RqS1911vAD4H4McAHgbwLQCvVRx3EYCdAHYDuFb3/OIjrRcfltEInSp8zMYd8h0kqo5p9B588EE+8cQTeXh4mI899lhesWJFatpMfKQUfVctRHQhgP/HzIeI6LM9Qf+jxDEdALsA/CcAewE8BODDzPx43vlXr17N09PT9hMuaJH0kQKR8eKyX6lgzo4dO3DaaafVnQxvSbs/RLSFmVcnj62lac/M/8DMh3ofHwBwUsphawDsZuY9zPwKgG8AuLiqNArFqbpzviDUjQ9R+8sA/O+U/ScCeDr2eS+Ad1SSIqE0tiLYghACzoSUiO4B8MaUrzYy89/3jtkI4BCAtCESacv6Kf0QRLQBwAYgGrkgCIJQFc6ElJnfnfU9Ea0H8D4A7+J0R+1eAG+KfT4JwDMZ15sAMAFEPlLjBAtCC2Hm8ksRNxDT2FEtPlIiugjAHwH4L8x8QHHYQwBOJaKTiWgIwIcA3FFVGgWh6SxevBj79+83Fo2mw8zYv38/Fi9erP2bunykXwbw7wB8r1cbPsDMVxLRCQC+ysxrexH9qwHcDaAD4CZmfqym9ApC4zjppJOwd+9e7Nu3r+6keMfixYtx0klpMfB0ahFSZn6zYv8zANbGPt8J4M6q0iUIbWJwcBAnn3xy3cloBDKySRAEoSQipIIgCCURIRUEQShJLUNEXUNE+wDMGvzkOADPOUqOXN/v67c5722/fpFrjzLz65M7GymkphDRdNr4Wbl+86/f5ry3/fo2ry1Ne0EQhJKIkAqCIJREhDRiQq7f2uu3Oe9tv761a4uPVBAEoSRikQqCIJSklUJKRJ8joh8T0cNE9C0ieq3iuIuIaCcR7Saiay1efx0RPUZEc0SkjBoS0QwRPUJE24jI2pT/Bte3nn8iOpaIvkdEP+n9fZ3iOKt5z8sLRXyp9/3DRHRm2WsaXv8CInq+l99tRPQpi9e+iYh+TkSPKr53nfe867vM+5uI6B+JaEfvnf9EyjHl85+2/kjTNwAXAljU+/+zAD6bckwHwBMAlgMYArAdwApL1z8NwFsA3AdgdcZxMwCOc5D/3Ou7yj+AP0dv/S0A16bde9t518kLojkevotoHtyzAfzQ4v3Wuf4FAL5t+1n3zn0+gDMBPKr43lneNa/vMu9LAZzZ+/9oRMsXWX/2rbRIuealTph5BzPvtHEuh9d3lf+LAdzS+/8WAO+3cM48dPJyMYBbOeIBAK8loqUVXt8ZzLwZwC8yDnGZd53rO4OZn2Xmrb3/XwCwA9HqG3FK57+VQprgMkS1UZK0pU6SD8A1DOAfiGhLbwWAKnGV/zcw87NA9JIDOF5xnM286+TF5fPWPfc5RLSdiL5LRCstXVsHH95153knomUAzgDww8RXpfPvw5pNTqh6qZMi19fgN5n5GSI6HtHcrT/u1e5VXL9w/rOurfP7HoXznpaklH3JvJR63hauvxXR8MN/I6K1AP4vgFMtXT8Pl3nXwXneiegoAH8L4Bpm/lXy65SfGOW/sULKFS91Ynp9zXM80/v7cyL6FqImopaYWLh+4fxnXZuIfkZES5n52V7z6eeKcxTOewo6eSn1vMteP164mflOIvoKER3HzFWMQ3eZ91xc552IBhGJ6BQz/13KIaXz38qmPQWw1AkRLSGio/v/IwqQpUY9HeEq/3cAWN/7fz2ABdaxg7zr5OUOAB/pRXDPBvB83wVhgdzrE9EbiaLlIohoDaKyud/S9fNwmfdcXOa9d96/AbCDmb+gOKx8/l1EynzfAOxG5BPZ1tv+srf/BAB3xo5biyjK9wSiJrGt638AUS34awA/A3B38vqIIrzbe9tjVV/fVf4BdAHcC+Anvb/HVpH3tLwAuBLAlb3/CcANve8fQUZvCkfXv7qX1+2IAqC/YfHatwF4FsDB3nO/vOK8513fZd7PRdRMfzhW3tfazr+MbBIEQShJK5v2giAINhEhFQRBKIkIqSAIQklESAVBEEoiQioIglASEVKh1RDRXUT0r0T07brTIoSLCKnQdj4H4NK6EyGEjQip0AqI6KzeXJOLeyOnHiOi05n5XgAv1J0+IWwaO9ZeEOIw80NEdAeAzwB4DYBJZq5yyK3QYERIhTbxJ4jGvb8M4A9qTovQIKRpL7SJYwEchWim9MU1p0VoECKkQpuYAPDfEc0/+9ma0yI0CGnaC62AiD4C4BAz/y8i6gC4n4jeCeB/APj3AI4ior0ALmfmu+tMqxAeMvuTIAhCSaRpLwiCUBIRUkEQhJKIkAqCIJREhFQQBKEkIqSCIAglESEVBEEoiQipIAhCSURIBUEQSvL/AXJor1kvvb3rAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "data_lin_sep_svm = pd.read_csv(\"./data/neural_networks/circles.csv\")\n",
    "X = data_lin_sep_svm.iloc[:,:2]\n",
    "y = data_lin_sep_svm[\"y\"]\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.2, random_state=1)\n",
    "sc = StandardScaler()\n",
    "X_train_scaled = sc.fit_transform(X_train)\n",
    "X_test_scaled = sc.transform(X_test)\n",
    "\n",
    "plt.figure(figsize=(5,5))\n",
    "plt.plot(X_train_scaled[y_train==0,0], X_train_scaled[y_train==0,1], \"bo\", label=\"Class 0\")\n",
    "plt.plot(X_train_scaled[y_train==1,0], X_train_scaled[y_train==1,1], \"ro\", label=\"Class 1\")\n",
    "plt.xlabel(\"x1\")\n",
    "plt.ylabel(\"x2\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Setting Up the network**. Let's review the ingredients of a Neural Network for binary classification: \n",
    "* a single output layer \n",
    "* cross-entropy for binary classification\n",
    "* sigmoid activation function for the output layer\n",
    "\n",
    "Since the classification taks is highly non-linear in this case, we use as an example a network with one hidden layer,  containing 15 neurons (and ReLU activation function). The other characteristics of the network we are going to build are the following ones: \n",
    "* ReLU activation function for the hidden layer (`activation='relu'`)\n",
    "* early stopping (`early_stopping=True` with max. 100 iterations without improvement on the validation set)\n",
    "* a maximum number of 2000 epochs (`max_iter=2000`)\n",
    "* SGD (`solver='sgd'`) with momentum optimizer with:\n",
    "    * batch size = 10 (`batch_size=64`) \n",
    "    * starting learning rate = 0.5 (`learning_rate_init=0.5`) with power scheduling (`learning_rate=invscaling` and `power_t=0.5`)\n",
    "    * momentum = 0.9 (`momentum=0.9`)\n",
    "* no $l_2$ penalty (`alpha=0`)\n",
    "\n",
    "As done in the regression example, we set `validation_fraction=0.1` to evaluate early stopping, and `random_state=1` for reproducibility. Last, note that MLPClassifier [uses the cross-entropy](https://scikit-learn.org/stable/modules/neural_networks_supervised.html#classification) as a loss function, and it automatically sets the `sigmoid` as the activation of the output layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "model = MLPClassifier(hidden_layer_sizes=(15, ), \n",
    "                     activation='relu', \n",
    "                     solver='sgd', \n",
    "                     alpha=0, \n",
    "                     batch_size=64, \n",
    "                     learning_rate='invscaling', \n",
    "                     learning_rate_init=0.5,\n",
    "                     power_t=0.5,                      \n",
    "                     max_iter=2000, \n",
    "                     random_state=1, \n",
    "                     tol=0.0001,   # the tolerance criterion for the optimization algorithm\n",
    "                     verbose=False, \n",
    "                     momentum=0.9, \n",
    "                     early_stopping=True, \n",
    "                     validation_fraction=0.1, \n",
    "                     n_iter_no_change=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now perform training: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPClassifier(activation='relu', alpha=0, batch_size=64, beta_1=0.9,\n",
       "              beta_2=0.999, early_stopping=True, epsilon=1e-08,\n",
       "              hidden_layer_sizes=(15,), learning_rate='invscaling',\n",
       "              learning_rate_init=0.5, max_fun=15000, max_iter=2000,\n",
       "              momentum=0.9, n_iter_no_change=500, nesterovs_momentum=True,\n",
       "              power_t=0.5, random_state=1, shuffle=True, solver='sgd',\n",
       "              tol=0.0001, validation_fraction=0.1, verbose=False,\n",
       "              warm_start=False)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT8AAADCCAYAAAA/3cXYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAYaUlEQVR4nO3de3Ccd33v8fd3dyXtStbVkixHvsg37DgXO67jXCFXIAmQBMLQJKWkXJqS07RlzswpZGA606GlpweGoefQFEwgodzbkJKckBAghARysS2HOLEjO/H9EtuSLNu6ry776x/7yF7LsrWxnvWzz+7nNaPZZ59d7X7MiE+e3efyNeccIiLFJhJ0ABGRIKj8RKQoqfxEpCip/ESkKKn8RKQoqfxEpCjFgnrj+vp619LSEtTbi0iBWr9+fadzrmGy5wVWfi0tLbS2tgb19iJSoMxsVzbP08deESlKKj8RKUoqPxEpSlmVn5ndYGZbzGyrmX1ugsf/l5m94v1sNLNRM6vzP66IiD8mLT8ziwL/CtwILAXuMLOlmc9xzn3ZObfcObccuA941jnX5WfQf3qijR+v3e3nS4pIEctmy28VsNU5t905NwT8GLjlNM+/A/iRH+EyPbFxP2t2+NqnIlLEsim/ZmBPxv293rqTmFk5cAPw06lHO1HUjNGULr8lIv7IpvxsgnWnaqEPAM+f6iOvmd1tZq1m1trR0ZFtRgCiEWNU1x4UEZ9kU357gdkZ92cBb53iubdzmo+8zrnVzrmVzrmVDQ2THoB9gmjESGnLT0R8kk35rQMWmdk8MyslXXCPjX+SmVUDVwGP+hsxLWLGiMpPRHwy6eltzrkRM7sXeAqIAt9xzm0ys097j3/De+oHgV865/pyEVRbfiLip6zO7XXOPQE8MW7dN8bdfwh4yK9g48X0nZ+I+Cg0Z3hEItrbKyL+CU356VAXEfFTeMpPW34i4iOVn4gUpXCVn3Z4iIhPQlV+OtRFRPwSnvLTQc4i4qPQlJ8OdRERP4Wm/GIRI6Xv/ETEJ6EpP235iYifQlN+OshZRPwUmvLTub0i4qfQlF8kYqRSQacQkUIRmvJLH+qi9hMRf/gyutJ7ztXe6MpNZvasvzEhGjVG1X0i4pNJr+eXMbry3aQvab/OzB5zzr2e8Zwa4H7gBufcbjNr9Dto1HSoi4j4x6/RlXcCjzjndgM459r9jZk+vW1Em34i4hO/Rle+A6g1s9+a2Xoz+5hfAcdEI4aOdBERv2RzGftsRlfGgD8CrgMSwItm9pJz7o0TXsjsbuBugDlz5rytoLqklYj4ya/RlXuBXzjn+pxzncBzwLLxLzSV0ZUR03F+IuIfv0ZXPgq808xiZlYOXAK0+Rk0pi0/EfGRL6MrnXNtZvYL4FUgBTzgnNvoZ1Cd2ysifvJzdOWXgS/7F+1EUUt/9ZhKOSKRib6GFBHJXmjO8IhF04WnC5qKiB9CU36RsS0/7fQQER+EpvyiXlJ97ycifghN+cUi6ajDOstDRHwQmvKrqygF4FDfUMBJRKQQhKb8GivLAGjvTgacREQKQXjKr8orv57BgJOISCEITfk1VMYB6OjRlp+ITF1oyq8qHqMsFuFgt7b8RGTqQlN+ZkZjVRnt2vITER+EpvwAGivj2uEhIr4IWfmVaYeHiPgiVOXXUFmmHR4i4otQlV9VvITe5AhO5/eKyBT5MrrSG1t51Btd+YqZ/Z3/UaEyHiPloH9oNBcvLyJFxJfRlZ7fOefen4OMx0yLp+P2JkeoKMvqUoQiIhPya3TlWVEZLwGgZ3A4iLcXkQLi1+hKgMvMbIOZPWlm5/mSbpxKb2uvZ3AkFy8vIkXEr9GVLwNznXO9ZnYT8DNg0UkvNIXRlXDix14RkanwZXSlc67bOdfrLT8BlJhZ/fgXmsroSkjv8ABt+YnI1PkyutLMmszS15k3s1Xe6x7yO2x1Iv2d3+F+XdNPRKbGl9GVwIeBe8xsBBgAbnc5OBhv7IKmXb0qPxGZGl9GVzrnvg583d9oJyuLRamKx3Q1ZxGZslCd4QFQP62Mzl6d4iYiUxO68ps+rVTlJyJTFrrya6pOsO/IQNAxRCTkQld+Cxoq2Ht4gMFhnd8rImcuhOU3DedgR2df0FFEJMRCV34LG6cBsK2jN+AkIhJmoSu/efUVmMG2dm35iciZC135xUuizKpN8MbBnqCjiEiIha78AC6aXUvrri5d0VlEzlgoy2/VvDoOdifZ3dUfdBQRCalQlt8l8+oAWLO9K+AkIhJWoSy/hY3TqKsoZc0OlZ+InJlQlp+ZcXFLLWt2+H7VLBEpEqEsP4ArFtaz9/CAjvcTkTPiy+jKjOddbGajZvZh/yJO7NoljQA83XYw128lIgVo0vLLGF15I7AUuMPMlp7ief9M+qKnOTertpwlTZX8uq39bLydiBQYP0dX/hXwU+CstdF15zayftdhjuiy9iLyNvkyutLMmoEPAt/gLLr+3BmMphy/2aytPxF5e7Ipv2xGV34N+Kxz7rTXmTKzu82s1cxaOzo6ss14Sstm1dBUFecXGw9M+bVEpLj4MroSWAn82Mx2kh5mdL+Z3Tr+haY6unK8SMR473kzePaNDvqHNM5SRLLny+hK59w851yLc64FeBj4H865n/medgI3nD+T5EiKZ7dMfUtSRIrHpOXnnBsBxkZXtgH/MTa6cmx8ZZAubqllekUpj20YvzEqInJqvoyuHLf+z6YeK3uxaISbl5/DD17azZH+IWrKS8/m24tISIX2DI9Mt62YxdBoiv+vrT8RyVJBlN9551SxpKmSh1/eF3QUEQmJgig/M+O2FbPYsOcIW9t1rq+ITK4gyg/glovOIRoxHl6/N+goIhICBVN+jZVxrlncwMPr95Ac0UxfETm9gik/gLsub6Gzd4ifv7o/6CgikucKqvyuXFjPwsZpPPj8Tg03EpHTKqjyMzPuuryF1/Yd5eXdR4KOIyJ5rKDKD+BDFzVTGY/x4PM7go4iInms4MqvoizGHavm8OTGA+zs7As6jojkqYIrP4BPXTmPWMT412e2Bh1FRPJUQZZfY1WcO1bN4ZE/7GOPBpuLyAQKsvwA7rl6AdGIcf9vtfUnIicr2PKbURXn9otn85+te9mh7/5EZJyCLT+Ae69dSLwkyj/+/PWgo4hInvFlbq+Z3WJmr5rZK96Mjiv9j/r2NVbGuffahfy6rZ1n39CVnkXkOL/m9j4NLHPOLQc+ATzgd9Az9fErWpg7vZwvPv46w6OpoOOISJ7wZW6vc67XHT+frIKTp7sFpiwW5QvvW8rW9l6+9+KuoOOISJ7wZW4vgJl90Mw2Az8nvfV3Er9HV2br+nMbueodDXzll1t06IuIAP7N7cU591/OuSXArcAXJ3ohv0dXZsvM+NKHLsCA+x55TRc9EBHf5vYe45x7DlhgZvVTzOar5poE9910Lr/f2smP1u6Z/BdEpKD5MrfXzBaamXnLK4BS4JDfYafqzlVzuHzBdP7h56+zrUOXuxcpZn7N7b0N2Ghmr5DeM/zHLg8/W0Yixlc/spyyWIR7f/gHBod1xWeRYmVBddTKlStda2trIO/9m80H+cRDrfzppXP54q3nB5JBRHLDzNY751ZO9ryCPsPjVK5dMoM/f+c8vvfSLn6ybnfQcUQkAEVZfgCfvWEJ71xUzxd+tpGXtufd15MikmNFW36xaISv37mCOXXlfPr763XhU5EiU7TlB1CdKOHbd10MwMe+s5aD3YMBJxKRs6Woyw+gpb6Chz6+ikO9Sf7022s43DcUdCQROQuKvvwAls+u4Vt3rWTnoX7uenAtRweGg44kIjmm8vNcvqCe++9cQdv+bu5Y/RKdvcmgI4lIDqn8Mly/dAbf+thKtnf28pFvvsj+owNBRxKRHFH5jXP14kb+/ROX0NGd5MP/9iJvHuwJOpKI5IDKbwKr5tXxwz+/lORIig/d/wLPbGkPOpKI+EzldwoXzKrm0XuvYHZdOZ98aB2rn9umS2GJFBCV32k01yR4+J7LeO95TXzpic18+vvr6R7UnmCRQqDym0R5aYz7/2QFX3jfuTzd1s77/+/vWbezK+hYIjJFKr8smBmfeud8fvIXl+JwfOSbL/KlJ9p0SSyREPNrdOWfeKMrXzWzF8xsmf9Rg/dHc+v4xd+8iztWzWH1c9v5wP/7Pa/tPRp0LBE5A36NrtwBXOWcu5D0/I7VfgfNFxVlMb70wQt46OMX0z04zK33P89XntrCwJC2AkXCxK/RlS845w57d18iPeejoF29uJFffuYqbll2Dl9/ZivXf/VZnnxtv/YIi4SEb6MrM3wSeHKiB4IaXZkr1eUlfPWPl/OTuy+lMh7jnh+8zEe/vYbNB7qDjiYik/BtdCWAmV1Duvw+O9HjQY2uzLVL5k/n8b+6kr+/+Txe23uUG//ld/zlD1/W2SEieSyWxXOyGl1pZhcCDwA3OueK7tLIsWiEuy5v4Zbl5/Ct323noed38sRr+/nAhefw19ctYmHjtKAjikiGSQcYmVkMeAO4DthHepTlnc65TRnPmQP8BviYc+6FbN44yAFGZ0NX3xDf+t12vvvCTgaHR7l5WboE5zeoBEVyKdsBRllNbzOzm4CvAVHgO865fxwbW+mc+4aZPUB6fOUu71dGJnvzQi+/MYd6k6x+bjv//uIukiOj3Lq8mb+4agGLmyqDjiZSkHwtv1wolvIb09mb5JvPbuN7L+1icDjFpfPr+LPLW7j+3BnEojrWXMQvKr88dbhviJ+07uF7L+5i35EBZlbH+eilc7n94tlMn1YWdDyR0FP55bnRlOPptoN898WdPL/1ECVR49oljdy2YhZXL26kNKatQZEzkW35ZbO3V3IgGjHec14T7zmvia3tPfxo7R4efWUfT206SF1FKTcvO4fbVszi/OYqzCY62khEpkJbfnlkZDTFc2928NP1+/jV6wcZGk0xv6GC918wk5sunMniGZUqQpFJ6GNvyB3tH+bx197i8Q37WbPjECkH8xsqeM/SJq5Z3MCKubWUaEeJyElUfgWkoyfJU5sO8OTG/azZ3sVIylFZFuPKRfVcs7iRqxY3MKMqHnRMkbyg8itQPYPDPL/1EM++0c4zmzs40D0IwNKZVVy9uIFrljRy0ewaHT4jRUvlVwScc2w52MMzmzv47ZZ2WncdZtTbKlzZUsuqedNZNa+OC5qrtfdYiob29hYBM2NJUxVLmqq45+oFdA8O8/ybnTz3ZidrdxzimS3pK+ckSqKsmFvDqpZ0GV40p4Z4STTg9CLB0pZfAevsTbJuRxdrdnSxdkcXbQe6cQ5iEeMdMypZNruGZbOqWTa7hkWN0/RRWQqCPvbKSY4ODNO6s4uXdx/m1b1H2bDnCN2DI0B66/D85iounFXDstk1XNhczZy6ciIRHVoj4aLyk0mlUo5dXf1s2HOEDXuPsGHPETa91U1yJAVAeWmUxU2VLGmqYunMSpbMrGJxUyVV8ZKAk4ucmspPzsjwaIotB3rY9NZR2vb30La/m80Hejg6cHxe8azaBEuaKlnQMI35DRXMq0/fTq8o1UHYEjjt8JAzUhKNcH5zNec3Vx9b55zjQPcgbfu7advfw+YDPWw50M1zb3QyNJo69ryqeIz5XiEuaJjGvPoK5jdU0DK9QjtYJO9kVX5mdgPwL6Sv5/eAc+5/j3t8CfAgsAL4vHPuK34HleCYGTOrE8ysTnDtkhnH1o+mHPsOD7Cts5ftHX1s7+hlR2cfL2w9xCMv78v4fWiuSaSL0SvEudMraK5JMKs2oWKUQExafhmjK99N+pL268zsMefc6xlP6wL+Grg1JyklL0Ujxpzp5cyZXs41i098rC85wo7OPrZ1eMXY2ceOzl5ad3bRP27MZ/20UpprEjTXJphVW55e9u431yb0HaPkRDZbfsdGVwKY2djoymPl55xrB9rN7H05SSmhU1EWO+njM6Q/Qh/sTrK7q5+9h/vZd3iAfUfSP5v39/DrtnaGRlIn/E5VPEazV4qzahPHbs+pSTCjKk79tFIdpiNvWzblN9HoykvO5M3M7G7gboA5c+acyUtIyJkZTdVxmqrjrJpXd9LjqZSjsy95vBQPD7DXW97T1c9L2w/RmxwZ95owvaKUhso4M6rKaKwso7EyTmNV5m0ZDZVllMX0EVvSsim/rEdXTsY5txpYDem9vWfyGlLYIhFLF1ZlnIvm1J70uHOO7oER9hzu560jA7T3JGnvSdLRM0h7d3r59be66exNkprgL6y2vORYITZUljGjKn5CWc6ojNNQWUaiVCVZ6HwbXSlyNpgZ1eUlVJef/JE602jKcagvSXt3ko6eJAe7B72iTJfkwZ4k29p76ehNMjx6cksmSqLUVZRSV1FKbUUpdeUl1FWUUVdR4t0vPeHx2vJSojogPFSyKb91wCIzm0d6dOXtwJ05TSUyRdGMLcjTSaUcRwaGae8Z5GB3kvbuQTp6kxzuG+JQ3xCH+4bo6h9mZ2cfXX1DJ33kHmMG1YkS6sq9svQKsqa8hKqE9xOPebclVCdKqErEqIqXaG93QCYtP+fciJndCzzF8dGVm8aNrmwCWoEqIGVmnwGWOue6c5hdZMoiETu2BbekafLnJ0dGOdI/zKHeIQ73D9HVd/wn8/4e78yZowPDx86YOZXSWISqeLoMq71yzCzL4+tiJzxWnSihMl6iK/acIZ3hIZJjg8OjdA8O0z0w4t0O0z04QvfAMEcHhk9+bNzjIxN9eZkhURI9oRjTpRijoizGNO8nvRylouzE9WOPVZRFC2ZnkM7wEMkT8ZIo8ZIojWcwp945x+Bwiu5BrygnKMv0eu/+4DAdPUm2dfTSlxyhNznC4PDptzzHlEQtoyhjJEqjlJdGSZTEKC+NUlF2fHnssfRP5jpvuSRdtOWlUcpikbw87VHlJ5LHzIyEVyxnOqpgZDRFX3KU3qGRY4XYlxyhd/D4ct/Q6Anr+4ZG6B8apX9olK6+AQa8+wNDo/QPjzI6ydZopoilt04TXjFmlmZmiSZKYiRKIyS8/1gkvBK9YmF9TsY0qPxEClwsGqG6PEJ1uT9nyjjnSI6kjhXhQEZR9mcsDxy7TZfr2HL/0CgDw+n7nb3JY8uDpyjWH3zqEpWfiATPzI59lD/5SMypGx5NMTA8yuDwKINDKRoqy3LwLio/EckzJdEIJdFIzs/p1j5yESlKKj8RKUoqPxEpSio/ESlKKj8RKUqBnd5mZh3Arrf5a/VAZw7i5IKy5k6Y8ipr7pwq71znXMNkvxxY+Z0JM2vN5py9fKCsuROmvMqaO1PNq4+9IlKUVH4iUpTCVn6rgw7wNihr7oQpr7LmzpTyhuo7PxERv4Rty09ExBehKD8zu8HMtpjZVjP7XNB5AMzsO2bWbmYbM9bVmdmvzOxN77Y247H7vPxbzOy9ZznrbDN7xszazGyTmf1NvuY1s7iZrTWzDV7Wv8/XrBnvHzWzP5jZ4yHIutPMXjOzV8ysNZ/zmlmNmT1sZpu9v93LfM3qnMvrH9JzQ7YB84FSYAPp+SBB53oXsALYmLHu/wCf85Y/B/yzt7zUy10GzPP+PdGzmHUmsMJbrgTe8DLlXV7So1KnecslwBrg0nzMmpH5fwI/BB7P578DL8NOoH7curzMC3wX+JS3XArU+Jn1rP2PPoX/AS4Dnsq4fx9wX9C5vCwt48pvCzDTW54JbJkoM+lhUJcFmPtR4N35nhcoB14GLsnXrKRHuT4NXJtRfnmZ1XvPicov7/KSHoa2A2+/RC6yhuFjbzOwJ+P+Xm9dPprhnNsP4N02euvz5t9gZi3ARaS3qPIyr/cx8hWgHfiVcy5vswJfA/4WyByUka9ZARzwSzNbb2Z3e+vyMe98oAN40PtK4QEzq/AzaxjKb6LJJ2HbRZ0X/wYzmwb8FPiMO/1Y0UDzOudGnXPLSW9VrTKz80/z9MCymtn7gXbn3Ppsf2WCdWf77+AK59wK4EbgL83sXad5bpB5Y6S/Vvo359xFQB/pj7mn8razhqH89gKzM+7PAt4KKMtkDprZTADvtt1bH/i/wcxKSBffD5xzj3ir8zYvgHPuCPBb4AbyM+sVwM1mthP4MXCtmX0/T7MC4Jx7y7ttB/4LWEV+5t0L7PW2+gEeJl2GvmUNQ/mtAxaZ2TwzKwVuBx4LONOpPAbc5S3fRfq7tbH1t5tZmZnNAxYBa89WKEvPDfw20Oac+2o+5zWzBjOr8ZYTwPXA5nzM6py7zzk3yznXQvrv8jfOuY/mY1YAM6sws8qxZeA9wMZ8zOucOwDsMbPF3qrrgNd9zXo2v2ydwpefN5HeQ7kN+HzQebxMPwL2A8Ok/6vzSWA66S+/3/Ru6zKe/3kv/xbgxrOc9UrSHwFeBV7xfm7Kx7zAhcAfvKwbgb/z1udd1nG5r+b4Do+8zEr6e7QN3s+msf8v5XHe5UCr97fwM6DWz6w6w0NEilIYPvaKiPhO5SciRUnlJyJFSeUnIkVJ5SciRUnlJyJFSeUnIkVJ5SciRem/AUj3ljLE1LPQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5,3))\n",
    "plt.plot(model.loss_curve_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SGD has used early stopping to terminate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "587"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.n_iter_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predicted probabilities of the first five test units: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.69870168, 0.30129832],\n",
       "       [0.30290545, 0.69709455],\n",
       "       [0.88417554, 0.11582446],\n",
       "       [0.36950992, 0.63049008],\n",
       "       [0.21821887, 0.78178113]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict_proba(X_test_scaled[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And their predicted classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 0, 1, 1], dtype=int64)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(X_test_scaled[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last, let's check plot the model decision boundary: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUwAAAFBCAYAAAAc8JaBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO2deZxU1bXvf6uqu+kJEQoUBLpb0KiAgtBR8zHyVIx6cUIRldvBCaO2iZqbmJsYPjcafRhzfYkhamK4Vz8q3dFrHJLcaJ5gHigap4bAdQwSoAFHaGWSprvpWu+P06f6VPUZ9hmqzjlV6/v57E93VZ1hnel31t577bWJmSEIgiA4kwjbAEEQhLgggikIgqCICKYgCIIiIpiCIAiKiGAKgiAoIoIpCIKgSGiCSUSVRPQ6Ea0loreJ6Mdh2SIIgqAChRWHSUQEoIaZ9xBROYCXANzIzK+GYpAgCIIDZWHtmDWl3tP3sbyvSBS9IAiRJdQ2TCJKEtEaAJ8CWMbMr4VpjyAIgh2heZgAwMy9AKYQ0YEAniaiScz8lnEZIroawNUAUFNTM+3II48MwVJBEIqZVatWbWfmEU7LhdaGmQsR3QLgC2b+P1bLNDY2cltbWwGtEgShFCCiVczc6LRcmL3kI/o8SxBRFYDTALwXlj2CIAhOhFklHwXgYSJKQhPux5n5TyHaIwiCYEuYveT/A+DYsPYvCILgFhnpIwiCoIgIpiAIgiIimIIgCIqIYAqCICgigikIgqCICKYgCIIiIpiCIAiKiGAKgiAoIoIpCIKgiAimIAiCIiKYgiAIiohgCoIgKCKCKQiCoIgIpiAIgiIimIIgCIqIYAqCICgigikIgqCICKYgCIIiIpiCIAiKiGAKgiAoIoIpCIKgiAimIAiCIiKYgiAIiohgCoIgKCKCKQiCoIgIpiAIgiIimIIgCIqIYAqCICgigikIgqCICKYgCIIiIpiCIAiKiGAKgiAoIoIpCIKgiAimIAiCIiKYgiAIiohgCoIgKBKaYBLRWCJaTkTvEtHbRHRjWLYIgiCoUBbivvcD+C4zryaiwQBWEdEyZn4nRJsEQRAsCc3DZOaPmHl13/+7AbwLYHRY9giCIDgRiTZMImoAcCyA18K1RChmWluBhgYgkdD+traGbZEQN8KskgMAiKgWwJMAvs3Mu0x+vxrA1QBQV1dXYOuEYqG1Fbj6amDvXu1ze7v2GQCamsKzS4gXxMzh7ZyoHMCfADzHzD93Wr6xsZHb2tryb5hQdDQ0aCKZS309sGlToa0RogYRrWLmRqflwuwlJwAPAHhXRSwFwQ+bN7v7XhDMCLMN80QA8wCcSkRr+srMEO0Rihir1hxp5RHcEGYv+UvMTMx8DDNP6SvPhmWPUNwsXAhUV2d/V12tfa/CddcBZWUAkfb3uuuCt1GIPpHoJReEfNPUBCxerLVZEml/Fy9W6/C57jrg178Genu1z7292mcRzdJDBFMInKiG7zQ1aR086bT2V7V3fPFid98LxYsIphAoevhOezvA3B++40Y0wxRcs33rnmUuVt8LxYsIphAoCxb0xzrq7N2rfa/T2goMH65VjYm0/3VRDEJwvWK27yuvtF4+mXS37Sh63YJLmDk2Zdq0aSxEGyJmTW6yC5H2e0sLc0XFwN/Ly7Xf6uvN16+v92aPvk0i7W9Li/VvqZT5vq1Kc7O6DdXV2etWV2fbIoQLgDZW0KDQRdBNEcGMPk6CZ/W7voyT4FphJowtLZoQmwmzmYjlQyxVzokQPiKYQt5w8trsvCk7EdK3Z/ZbKuV+nzU11vvxKpTGkkz2C5+dt+j1JSAUDhFMwRV2Ipi7nFP10m5bushYeVxm26+oGOgpGvdp57UWqthVscXDjD4imIIybtrY/D78dqKTSmkim0r1/2/XtqjvMyiP0a/3aTwHxpdGKmUv+HbXReUlJvhHBFNQxo0Iuq1eeu1YMQqK0z6t7E8k3IvejBkDXx6qImrs2DLzko0vARWxlI6iwiGCKShjJ0iqgmcmrmYPfXn5wF5yq/07dRTpv1uJS3OzeY+8k5do5dk5Vf1V7VVBqvGFRQRTUMauo8Vte6Lqdo2C5MVjU203daoaW+3TDLuedTcesUpV286rtTo+qbZ7RwRTUMZKkKy8SbseayOqD72KNxWUMNjFeqp4cMb1rXrJ7Y5Htbru5M3qHrRU24NBBLNE8SosZuvZCZ7K9lV6r1Opwj/4ZvGZunAFsU87j1jlnFiJYW6xijiQart7RDBLkKA7ClQebiLrIG7V4HBdIApVtbRrKggKqxeXageSLoZ2YVh210RwhwhmCZKPYYUqgqd3DlltQ0V4C+kVhRlI7iVm1Mpe8TCDQ1UwJflGERH0NAx6DslUyn455uzkGrnb2LRJyz9pR3t74RJSBJ193U1iDbNExk4wa0lKjFRXa4lC/CRFFjygoqpRKeJh2pOPUBQ3Xqbf7RSqwyLIpgsv22ppcZ/oQ7+OdlEA0kvuHUiVvPTIR7CzahVSRZRVhKJQ1cmghMbLS8rsOunVbqlmh4MIZokStMeh2knhZj8tLdbbiVuHhZf2ULdhRxIqlH9UBVPaMIsMr9MwWKHSrpdKudtPU5N1m2bcZnFUbQ81tnOazY8OaG3NfuYeEvKPCKZgi1MnRUUFsGhRMNuNY4eFynHkZnK3QhfZoF96QnCIYMacfE99YPR4zLATANXtxtmTUjkOs2k7conjy6IkUam3R6VIG2Y2fnNTukUSQnjD64gpu2GY0jseLJBOn+LHaxYfrw+XZA73RlA96XoxS4CiXxsRT2+oCqZUyWOMU6C6ygyObgg64LtU8NJea1eN7+4Genqyv+O+ppFCzrJZiohgxhgnAQt65E+xdNQUGi/ttV6vEeDvpSjYUxa2AYJ3Fi7UvAmjJ2IUsLo68xAWrx6h/oAvWKA90HV12r7y1VHz8ssv45ZbbsHu3bvzs4MCM2ZMOe6881u45JJLHJe1unaq+BFcwQaVentUirRhDsTPDI5R5vnnn+fq6moGUHTl/vvvdzx+t22YQXTElXJHEhTbMMXDjBmtrQM9vE2bzJcttEcYFM899xxmzZqFffv2YerUqTj66KPDNikQNm/ejOXLl+Paa69Fd3c3rr/+estljdeuvR1IJoHeXq06r9cg9N+IssO7vDST6LGiem1Fbws12iJAPMw4Yecxxto7MBj/xYgRPC+ZZAB84IEHhu4NBl2MXvNdd90V9OnzfO1LPWQMih4msfHVFHEaGxu5ra0tbDNCo6HBvF0rlQI6Owe2ZUYlEJyZ8be//Q07duwY8NvBzz+PI3/+cyS7ujLffQHgxqoqPNDZiUmTJim1+cWBF154AcuWLUNVVRU6OzsBAAsXLsQPf/jDkC3TBj6YSQGRNuKo2CGiVczc6LigiqpGpZSKh+k3W3eU2rH279/P8+fPt/S4NloYvxHgY489lrdv3+7PgAjR1dXFs2fPZgCcSqWYiBgAv/HGG2GbJh6mBK7HE7/zwRiL24DyoDuJenp6eN68eQyAq6qq+JRTThlQei2M7wX4s88+87bjCNPV1cUAmIj47LPPZgD81FNPhW1WrDsIg0AEM6Z4Sf3lZq5wr/t2S3d3N1988cUMgAcNGsSDBw9mIhpQrDzM3rFj3e/ULSE0/Pb29mYEc9asWYELpuohmS0X63Zwn4hgxhQvc1oH5R0ENfSxq6uLzz//fAbAlZWVmaqnWZkL8J6cHaarqvL/tIbkUnkRTDciqHJIpe5NmiGCGVO8enlh9ZQ+/vjjfOaZZ/LXvva1TJk0aVKmGq4L4x133MG9vb3mZckSTtfVcbqQrk1IjXa6YALg8847jwHw7373O8vl3Yib6iGVenulGbEQTAAPAvgUwFsqy5eCYIb59ne771//+teWnqMxfOZnP/tZ/o13S0iZRNLpNB9yyCEMgL/0pS8xAB43bhxv2rTJdHk34qZ6SJJEZSBxEczpAKaKYGYTZluS6r4XLVqUEcTTTz+d58+fnynnnHNO5rd77703+J37wan3rABu1sqVK7m2tpYB8NChQxkA19XV8fr16wcs60bc/HqYQc7LHjdiIZianWgQwXQmSg3y//7v/27qSRoLEfFvfvMb+w0ZDyqVGjjez41rrXKCnKau9DN1pMuL88orr/CQIUOyRHP06NH83nvvZS3nxsNsbjZfdsaMbPOam82HVlZUlG47pghmERGlET633357RhQrKysZAJ9yyin8ve99L6ssXbrU/oBU55nNVQY/vV52+/R68ny0obS1tWXEUv978MEH89tvv227+fJy7VByr7mVuOZ6qdXVzLW16kJcChSNYAK4GkAbgLa6uro8nKroY1eFKmR754oVKzJiecABBzAAvvDCC7m7u1t9I6oTnRuL3bqqcVX5mqrSZw/KmjVrePjw4QyAGxoaGABPmDBhwCkzOuIVFebX3O3AhqBPRZwpGsE0llL1MN0+CMlkfjzOBx54ICOSALimpoZ7enrcbcRL9L2TC6Xy9Oer3TKAHhT9vM6ZM4f1uFUr7PTZ7ekJ+lTEGVXBlATCMcBt/srevuEz7e3A178ODB8ebAbuQYMGAQCSySTKylwmvHKbqJG5Pxuu23WNJ84uuWRuap/WVu2kEWnF7gQGkIZeP5+JhPPjaJcU2izBM5H58ma7kmTQzoQqmET0KIBXABxBRFuJaH6Y9kQVq0znqZTa+h0dEZq2wEv2Yl0lrNZNpexTwdsdeCKRnaGktRW48krtpOl0dFi/eWbONN+u1fc+sdNns8zu115rPk1ybkKNVCo6yVqiTKiCycxzmXkUM5cz8xhmfiBMe6KK1RQHixbZzxluJIhpC9J9T5lWg/GI00TnZugqYbfuZZeZzwGhJ3q0wqgcra3adrq7zZft6ADmzdP2oc9p/Oyz5stafe8Tp2lCcuc0/9WvtFPh9HKtrdX+5nPK5qJApd4elVKqbZhGcjuJm5v7P+tTseajQX/jxo186KGHMgA+6qijGNACr5UNdZr716rn3GzeYJXldJwa9uym2HQqdsu7ONktLS0MIGvsvf3y7iMjVNo3S3m4JOLS6eOmlLpgOkWwOD3zXhv0169fz2PHjmWgP/xl8ODBvHLlSm+GOh2kkxoEMfwl1yavPSZW23cRBe5WML3g1HFo9bItlU4gEcwiREUn3DpgTrz33nuZoXy6WA4ZMoRfeeUVb4a69TzNsHv6VYMTk8nsQFYvYmlXIiaYdocYkKMca0QwixA3ESxBBLS/9dZbfPDBB2eJ5dChQ50T3jp5dWYeml2AYS6q9UunoPaWFufZxLyWPFbJvWBX+6ipsT4M8TBFMGOL1Y2dSATf1rR27VoeMWJEllgOHz6c16xZ47yynVfnVnjMnljV9kZj+6TZ20N1tJFV0Yfc+FQaXTAbGxsZACcSCV63bp3y+ur76b80KrG9xhDYYkcEs8iwG6hi54x5YdWqVTxs2LAssTzooIP4zTffVDfWypMMylMziqCTaFq52XbrmY1tN3tTNTf77i3ZunVrZsI3/SU1atQofuedd5S34QY374lSQQSzyFCphQZRfXr11VcHJIU45JBD+N1333W3IbfujJ+DcjOI2ihkTkKdSjmrS3V1dqiC2/aPvvOUJuLNiQTP9fqScrG7IE99sSCCWWSoVqFycdOWuXLlSh48eHCWWI4dO5bff/9974b76VBx07Ou6tEaVUDF1bIbq+5XWUzsTgP8KcBX9WWBSqVSvHr1am/bN0H1cpRSSBGzCGbR4cXDdBPds3z5cq6pqckSy0MPPZQ3btzoz3A33mUiYZ6Gx4zcjBTG9Dt2Amd8q7S0DOxoMitmWU7stqn6hrK5qPsA3gZtMrh2Im7/yU98XQYdu8uheuqLERHMiBBU+jUvqRxVwxWXLl2amU5Cb0s7/PDDefuiRd6M9xqqYxaKo5rOTdUrzO0IAtQi/ltaNEF32qab9kwXL5N9ZWWBqFgQyYOjlJs1KEQwI4Cf+G2r7RmdKiePQCUM6ZlnnuFBgwZlieWRRx7Jn91zjzfjvYyYMTPM7gSqVKXtct95SRNn5Y2WlzvHfFpV2d2+VAJoVPR7TwZ9T0cFEcwI4DNVYt73//vf/57Ly8tZD0YHwJMmTeKPP/7Yu/F+2ixzt+1nW7pXaOYKeUkwquKa2XmMZori9uUSUBS5Hw8x7Hs6X4hgRgAVDy+f1Rs7b+Dxxx/nsrKyLLGcMmUKb9u2Td14Nwdtth0nNyVfvet2x2Z1QVQa/+yq9lYp8pubvWefD4FinUBNBDMCOL2NnQahGJ+jVMqbmJo9/62trZxMJhnoz5ze2NjIHR0d6sZb7citqHkZM+5UnOqIXtykIIZPOjUTGNpbunLW7UwkuHfJEmv7CoR4mBEQQtUSN8F0au+xq+Xla5Kqhx56iImI9QQaAPiEE07gHTt2uDPeadkgvCXV7ao06Ho9Nh2rGcaCKCbn4s/z5vFGaL3kmwCeC/Cll17K+/fvdz5veUTaMCMghKolboLJbF/l9lLj9PMmX7x4cUYs9WleTzrpJN61a5d7442/2VVFzXqWvcwGabV9r3VBt20h+UjQYSw5+0+n03zzzTezPlRS75i75JJL3M2hlAeklzwmJY6CaYeXZ9CrPtx7770MaBOY6fGWp556Ku/Zs8d5ZbN2NxXPTzc2iCcs7Lqg3xnGnHr3TV4i6XSab731Vga0aYv1WTpnz57NXV1dhTnuEkEEMwZ4iZrRs5K54Z577hkglmeccQbv3bvXm5Gq4mGVOMNrbGeYdUE/HqYxlZyHhKV33HFH5vrp8bLnnHMO9/b2FubYSwARzJhgFZdtl/fBrU7oHTsnnHACA+CZM2dyZ2en2spehSJ38uzmZvN0S16q52HUBYOKL3UazG1xTHfddRfrGaP0eNmXX365QAdf/IhgxpzcXnJFZ8QUPXxIn8b10UcfVV/ZTVVUn9/XLLelW080iuQKtl0iSbvjs3sJWbxAduzYwXpUw2mnncYAeOnSpQU46NJAVTBlmt2I0tQEbN9uPU2q2xlnPaM6y2N1NfDww9rsW7W11hOJmVGwg/FJ7gxjv/kNUFFhvw7RwLlr7SZzC2K2ugBpbZWJ0YyIYBYYtzdgANNee0M3tL3dWrV1cmdptJsD3AzVg4na09vUBDz4oHb8VjAPnLtWnwbUioi8QPQJN9vbtcNob4/QdM1hoeKGRqXEvUrupd/Cb1/HRx99lAklmj59ulqV3G17nWpSDKv2Pa8p3KIUAOimF98pVMpkHb1KPmjQoMx1fPjhhyNzSHEH0oYZPbzegF77OrZu3cpHHHEEA/2ZvCsrK3n9+vXeDLUrqkkxcktzs9rBRP3ptYsmMF40L2mnWAsxmjx5MgPIzLNUVVWV13bMYh0GaYYIZgQp5A3Y3t7O48ePZ6A/v2VVVRU///zz3g0NsridiCgOT6/Rc7QaK2/3MrIb/9rSwj2jR3MvwBsBnt8XXjRo0CB+5pln8nI4UX9HBYkIZgQp1A24YcMGbmhoyBLLmpoaXrFihT9DUynv0fYKXpQnm6L49NrZanee7Mbp53ilewyiWV5ezk8//XTghxH1VpAgEcGMAKoDZLwm1jBjw4YNPGbMmCyxPOCAA9zF7Fk9KfrcNW7E0u+cN042RfHptfOG7YaRWnnLFud8I/qTp5SVlfGTTz7p23Sze7bYhkGaIYIZMnaaY9bUF9Sz/81vfpMB8MSJExnQUre9/vrr3g7Ay3BIoD8eMx/56uLw9AbtYVos3wtt9M9xxx3HAHjcuHG+zI7TOyloRDBDxu6ZyWftct68eQyAb7zxRgbA5557rv+NMrvzLKPUrhgGdsrj1Iapv3D0G6K52dJj3dU3FfKll17KgDYKyIupTjlUotjqETSqgilxmHnCKpRu82b736IWapjBTWwgc8SMLzB6nGV9vRbDaoxTtQpaTyaBjg7t/95e7W97O3D//dr5zIUIr513ni8zc+Ms9d3m0t5eupcyFxHMPGEXcG71GxHw9a9HNFB42DB3y0fK+BDIHRWkB6+biWkqZa1WZmLZ9/36447zZeKCBdrAIhWuuAIYPjyCL/ICI4KZJ8wciepq7XsrJyOdHvidm5FyzIyOPi9lz549Li22QHd5de/HDREb5hcZcsX0s8/cb8Mwuujzzz8HAOzbt8/VdXdTaejp0W6ByL3IC41KvT0qJU5tmMzO+XdV47xVmgTT6TR/73vfY0BLOKvnTrz99tv9HYDXDD3SnqmOU/uwRVjW66+/nhnFpUdEnHjiibxz585AdutUiqltE9LpE23caJHTjZlOpzOdPIlEIjMT5OWXX+5vSoMgsowX01OVL+xuBoewrEceeYQTiQQD/VOOHHfccfz555972q2bAVvF9C4UwYw4qlrkFNbR29vLzc3NDICTyWQmlds111zjP8FsEFnGSyEmxQqrKoZVEtRcpVIM0H3ssccGTGo3depU3r59uycTVV/mxfQuFMGMOCpa5PS8pNNpvuqqqxjQApd1sbz++us5nU6rG2P1YAeRZbxUsQvEdfO94jl88sknB8wxf8wxx/RPm+zBfMMklgNCjsrLi+vyBiKYAA4AMN7k+2NUNu64c+BMAH8HsB7AD5yWLybBdJrXSyUnxauvvsqANkZ8+PDhDIC/853vuBdLu6lfvXqZxVRf84Lbl00AQZD//d//zRUVFQwgMzT2lltu8X0oLS0D80EHMYNplPAtmAAuAvAhgDUA3gbwZcNvq1U2brtjIAngHwDGAagAsBbABLt1ikkwrao9boZJ/vnPf2ZAm5+nurqaAahNambEKYreJnC6ZOprXggqgYnLF89JJ52UqWUA4BtuuMH3ocRpGL9XVAXTLqzohwCmMfMUAFcAWEJEF/T95pBRVonjAKxn5g3M3A3gMQD+InFjhFk4XkuLlmUdKGDwul0UPQCceKL7GEw9fqqUcZvhOWHxKOY9U7QzTrdIKWEnmElm/ggAmPl1AKcAWEBENwDgAPY9GsAWw+etfd+VNAXPcm0XYa8b4yYGM5XqH9VSythNQ2GGWRBuRUUkXjyhZf2PIHaCuZuIxusf+sTzZGhe4MQA9m3mpQ4QYiK6mojaiKht27ZtAew2GlgJ4403Dhx94RT/vXv37sz/u3btcmeIXYS9mTFWGF3kUhdLnaoqf+sPHuz5XO7btw+Ah/vBBLtbpOSwqqsDmAzgcOS0KwIoBzBPpb5vVwB8BcBzhs83A7jZbp1iasN02ydg1pS1cePGTM/4yJEjGQBPmTLFfc+oVWyJimGlHjpkRhAB/3pxiZ6tSp+KFwDfd999gRxSHBJFeQVBhRUBeAvA96F5hFUA7gHwisrGHbZbBmADgEPR3+kz0W6dYhJMt30CVg3sv//97weEk0yaNIk/+eQTfwbaKXpu+rZif5rcEkTAv36eXfLFF1/w6aefzoCWNFoXzbvvvjv44ywighTMGgD3AnilTzxvBpBQ2bjCtmcCWAett3yB0/LFJJhunym7MKNnnnmGBw0alCWaRx55JH/44YfeDbRTdOPIE7M5yEvd6wxyig8PdHZ28llnncWAFnKmi+add94Z6GEW03sySMGsAHAXtPCi9QAuUdlwPkoxCaaXiRnteO655zLjx/Xq2GGHHcZbtmzxZqCVotfUuBsGUkxPlSpBeZiplGcTurq6eNasWQNE87bbblNa3+myFVuy4SAFcy2A2/raLkcC+AOAJ1Q2HnQpJsFkDiYBh/HGPuigvVxefjkDWvZtADx//nzvxvkdaFxsT5Uqdsftxvv0IZjMzN3d3XzRRRcxAK6rq8uMOXeaNVTlshVbbGaQgtlo8p3vTh8vpdgEk1ndGTF7dsxGYCQS3QzM5RtuuIEBnxnXc8fHuZlGN9+p5aNOEMNNAxgttWXLFgbAhxxySGYqi1deecV2HZXLFodJPN2gKpiO+TCZuc3kuyVO6wlqqAb/7t49MBbzxhuB7u7s79LpcgCLQBTA2AI9b+OSJUBnp3o8ph5zUsoRz1YJhM1idKyuVQCBjsb7QPWeULlspRqbKQmEQ0b1BuvuHhiLaa1fw/2YNBCV1Ny1tQOnY7A6OLcjh+JG7jwj113X/3nBAuCyy7KHeF17baQCHVXEsFRjM0UwC4DdPD1uBoSE5pip7Did1jzRXG+qvHzgsmbucrHQ2qrN52AckfDrX2d/fvhh7dzo3uevfmU9B1BAdHd3681p6OzstF3WSgxnzrTX/ZIY4KVSb49KiWMbpkoDem5zl1VTYW7Tn3WT4qfBzhqp2u5m1japejDFgmo7bypVkOiBzs5OHtY3u2R9fT0DWiajDRs22K6nMstyMfXfQfJhRgMv/R6qncstLVpewuxGd63TJ1DBdBMDlWtksfUOOKHaoWN2PvQbI2AVeumllzLZ2PWpLMaOHcvvv/++8jaKvf9OBDMi2EWROMW5qTggucudeOKvGMjDvOQtLdY5G+2UvdiftFy8CmaeXbfXXnstE5+ri+aoUaP43XffVVrf7j4uhhBbEcyIYKUXFvNa+WbevHn5EUxm9RhCY7b1UovFdBN6ZVfy8EJZtWpVpnqui+ZBBx3Eb775puO6Kq0ycb6sqoIpnT55xiqKhDn7u1jMSKvapd/b25+TzizxZzH0Dlj15C1apKVl80seevimTp2K5cuXY8SIEfj8889x+OGH49NPP8W1117ruK5K52Qs7mGfiGDmGTO9yBVLnciHJ7rp0jc+PVYxiXHFLmlpUxPw4IPZFzyVcr+PPAU0HnPMMfjud78LAJg+fToA4MMPP3RcT7+PnQ4l8vewT0QwC0CuXtTXmy8X+aBf/alJJtWWt3t67GKtoo5ZXKrdC2LRooEvmrIy64D1PAc0Jvqyu7sd3NDUpIXb2hH5e9gnIpghEOug36Ym8+zgZlg9PQVPKx8wqiOY9JfCvHlaMuFUShNJ/a+xqqGLV8SbLOzegbG5h30gghkChWjW279/PwAtYDlwVNwIu6fHyUPzQiE9VpWhMLkvhY4ObXjpkiWam9bTk70us3YjFKDJQr8n0qovPgNWh55MRlrng0OlZygqJY695IXmRz/6UVYvKAC+9957/W3ULIrZqYfXrrvUKTbTbUo4s5541bmKveA3nU+IsakrVwCGsxgAABtfSURBVK7MxGTW1dUxAD7llFOU1y/WoAdIWFFpYsy4rU+9C4B//vOfe9ug1RNSWWktmKmUvdhZiYlVnKfTE2kXu+XlSVYRbKdlnIIWCxRKZGT58uWZLOx6TOahhx7KGzdudLWdYkxxKoIZMQp5kwWacdvq4XaTFzPX0/My542dmDiNDnBDEC6UXd5L/eIX2E1bunRp370wl4naGejlsrKtvGjR9iyzzaZ2KjZxNEMEM0KEUY3p6uri888/nwFwZWUlExED6hm3M9hVH1XFzszTMz6JKiOI7KqrdlHVbqu5Xr0/leMxnocCKlH/FCZzGfjC9D5UfYcVQ/XbDBHMCBHW6EBjxu0DDzwwk3H75ZdfVt+IXw/Tr4do3J+VyDh5dG7w0r7oxmMuMHv27OHa2loGwGVlWy1PkZu8xsU4qlUEM0KEmX+ip6eHk8kkA8jM8fLoo4+qb8DvVBV+PUS9mM0lZHR3mpv9jze1Gy9vN0eRn2xOufsP2OvcunUrA9oQSKDX8tK4rTAUGyKYEcLOwyxEzUyfu3zOnDnuBZPZ3Ei3MyPmTs2bu32v83gbRcjPybSzwa7eqmq3UzaiPLXb6IJ5yCGHcEXFh5anUDxMEczIYPUsuM0x6FUPfAumGX5mRrTKVedlm0G5O3Y9907z8Vh5pfr3Kp5vntptjII5fvy/MbBH+V2geumKARHMiOGmJmf2jPhxQHTBvPDCCxkAtwRxx9vFPqp0gFgdpFlVv7ycOZHIi6BkUGk3sfOqcxOTAtY2m9mdp3Ybo2Aef/zxDMzlkSP3mb50W1q0lg8zE3STi1EsmUUwY4GbZ8SPA6Jn2v7Sl77EgFrGbVPMAtidXF6nkB+7NHCA9gTnTo1pfGPMmNEvysmk92B1lRNs1wFmZaNVyb3IBfAwNcG0njWy0PH/UUIEMwbY1fByNciPA/Lyyy8PyLg9ZswYXrdunbqxXl1cp2q2vg0v1d0ZM8x/c/OEG/ftVHUOqgNMF9kgzq8DbgTTzf1odgrjHKspghkDVNqNnPRE1QGxyrj9zjvvqG3AT3yi00HaDRe0K3ZiqoKVS6ULmtmIJePFUIkftbPda5p9F7gRTJVLoPoOiZtoimDGBNXmPjc3ptVz5yfjtmcX16pdMncbbjt8nMRKBat9mrU9Gk+2n159Y8n1MvNAEB6mndlhxRgHjQhmDAkiJ4WTsK5du5ZHjBiRJZqpVIr/9re/2Rvn5clQFRb9YHLbAZ3aBe1E03gyrXotvAicn+gAr8LuA79tmFYliKaiKCGCGUOCeFurbOPtt9/mkSNHZonm0KFD+Y033rDesJe6l4q4VFT0x7Xk9jSXlzPX1lqLl1PWJLNitNlPlboIBZNZfcSqfj+JhxnhUuyCGUR7kOob/+9//zuPHj06SzTLy8s5lUpllQsuuIB37drVb6CbNjaVRjF9fLXVk2e1jZoabR/Nzdm95Cr71J/mfAqh8cJZNUmEVCW/9dZ1SpexpcX+sunLSBtmREuxCyaz/3Z/N2/89evXZ3Ii6m2bZuWEE07gHTt2uD8Y1eqr1/ZAs5Ojuq4b+9yW3A4dq+aGPKtKSwvz6NE9DPRyMrmlL3B9LldW7lcWOCutD2qAVVQQwSwhjDdsKjWwZmv3QGzatInHjRvHAHj8+PE8a9asTDnjjDMyQtrY2MgdHR3uDVMVQ6fRMqpvAdXMR26Gt+g95n6EvMCqYnZoicReBj5VPpVW24mjB+mECGYR4LWTp6ysv2aqEsu9ZcuWTFC7WRkyZAgD4ClTpvC2bdu8HYQuVHZCYzV+1E74clFt19Sr77nhQ1bB+G7ENQJYO89p5VOpUwwepBOqgknasvGgsbGR29rawjajIOhTwhinvqmuHjhvSkODNm2MHWbr5bJr1y4sW7YMPYa5Zj7++GP867/+K3p6ejBkyBDs3LkTkyZNwg033JA14+CYMWNwxhlnOM9C2NoKXHaZNm95LvX12hxACxZoB5RMasvV1wPbtwNffGG+zqZNA7+/7jrg/vs1LVBB5QTp9i9YoM0ENmwYsHs3YJwzSXU7eeC1117Dm2++mfn8jW/MB2B2Pdj0+1RKm2po82Zt3p6FC0tgfh4DRLSKmRsdF1RR1aiUUvIwVdsiVeO9vfZaPvvss33JZ/unNTArzc3N3Nvb67xBpzqe2e8VFe7aGfTt5DsFT0Rcr/vuu8/kmmy0dICDOL3FBqRKHm9Ue7tVNcFPXNyyZcsyU11MnDiRL7rooky58MILM4I6f/583r9/v/MG7YTGbry21TpmY9zddiRZnaCIiKIVd999d0YkzzrrrMx1Of74RX1tlgNFMPeQVDp2ip1ICyaAOQDeBpAG0Ki6XikJpqqH6SY23A8rVqzITKCVW8rLy7miooIB8Ne//nXu6enxviO3kdBmPdB2xW32pAj3ePz0pz/NXAM9q3p2mdvnaaZttb5Ygs/9EHXBPArAEQBWiGCa43UopNtecje89NJLPG3aNK6vr8+UMWPGMAAmooynefHFF3N3d7e3nbiNhHbTe633jque2AhHZd92222Z815ZWcmANszVeG2OPPJIbm1tddxWhA+zYERaMDM7F8G0xWttsJC1yHQ6zbfeemvGq9Ef3gsuuIC7urrcb9CtV+em2q0rgOoJCtD1Wr16NX/lK1/h8ePH+y4NDQ0MgBOJROYlNWfOHM8vqYg70gVBBFMoKHfccUdGNPX2znPOOYf37dvnfmO5LrPdPOduBNNtfGRArpcxU1RQxdgM0tTU5K8ZxOE0lAKqgpm3sCIieh7ASJOfFjDzH/qWWQHgJma2jBUioqsBXA0AdXV109qdYmhKgNzoFgD47LPww0F+9rOf4aabbgIAVFdXY+/evTjzzDPx1FNPoaqqyv0GVWKrhg8HOjqct5VIDAxnctq+amyXDX/961/xT//0T9i1axeOOOIITJkyRWk9O9LpNP74xz+iq6sLl19+Of7zP/8TyWTS07aM91LY90+YxCKsCOJhusapkyfsqtQvf/nLjBekdxLNmDGD9+zZ435jKh6eWdIOs2IWva+6fY+u1wsvvJA5B0F7mAD4mmuucQzlsjNfquL9IGwPUwUVD9NIKQWuW6ESqG4Vz10oFi9ejGuuuQYAUFtbiz179mDUqFFYuHAhamtrlbdz4cUXg0zuTybCE//1X5nPY1euxNGPPorqjg7sTaWwa+RIHPzOO6B0GpxI4B+nnYY1V13lefte6OjowHe+8x10dnbiwAMPxI4dOzB+/HjcdNNNzgH+CowePRpnnXUWiMjSS3RykK3upbDvnzCItIcJ4HwAWwF0AfgEwHMq64mHqZ4AyIxCtlM9+OCDTEQMIDM9htuy0eIANwbkoeV7+zB4lkcccQRv3bo10HNslZtZNUu/hBP1gzh4mG4RD9O7h2nmbZSXAwcckL/2z9bWVlx66aVIp9M4++yzkUgkXK0/fetWfHPtWlQa2h73JZO4b/JkvDhmjG/7zLbfQ4S9ZWUY3NOD7VVVWHLUUZ721d3djVWrVmHbtm2YMGEC/vKXv2DkSLMmfW+YXU8j9fWax2n1eNfXW99H4mHaoKKqUSniYTq3YZaXuwspzHf71eOPP56Z5tdLmdvn8fX2/Z3r0+PL3d49hs+fAtyZc1L2+Nzn5MmT+dNPP1U+X6q1AKfr6WXWj3zdA3EA4mEWL3Y5LFIpLVdFLomEtbdhJB/exTPPPIMXX3wRixcvDnbDLpnd1YVfdnai2vDdXgA3VFXhyUGDsHbnTtSZnKTNRJg8ZIjr/Z1wwglobW3FMD2UwQE3nfJO11PPZXLFFYAhn4ot+jrSS26DiqpGpYiH2Y/b9ievY86LKj4v4o16bsI+7a6nSrL3kA4xskDRw3TXqCREhro6d98vXKh5K262q3s87e3aY9Xern1ubXVvbyTYvNn+e7cnNWCczDNidT1TqWyP9LPP1PZdoEOMPyqqGpUiHmY/XmLocgfQ5OasyF2/6MYYOx2QyUndg2q+PtVSEM/a7flW8f79tF0XVe3CAcRhaKTbIoKZjd8b2ml9uxCmWD5EZm8Zouyg9pYW3p2q514Qb0Q9z0VLwTpDvAaS211HqwT0NTX+pmsuNkQwBd+oeidWszpEkubmgW+CHCXw41nn+yVmtrydsHk9lqKrXTgggin4RjXXpoP+RAsFJfDa9xOGV+aUb9lrJ0+pBbWrCqZ0+giWNDVpHQj19YDdaD7m7M9792pD9SKJQs+K176fBQsGBpLn+1xYBZ93dNgPcHA6lpD7vyKLCKZgS1OTFpeZTmvCqYqVLoWOghKY9UBXV2vf2+Gml1uF1lZtZFciof01i07wkqRI5Vi8noOiR8UNjUqRKnnwuGkzs+oz8dLWFVoPrGK92Yt9btr9nLavWr13ai4xsyWKiajDBtKGKTjhpc2tubl/WpxkknnGDG/hTaH2wOZJCVSPS2U51YnJ3Ax/LNYOmyAQwRQc8RL3lxu7WVHhvpc8SE9MlUJ5S35iI43hoKqdNWbiW17uHGMrZCOCKZhifKDd9qBaeT2plDsbVHtgg/JEQ/doc3A6fjuvUfWlUkrV6SAQwSxhrB4W1TAhKw/Tbh03qHqYQcUCRi2m0MqeZFK7RnYvMxG+/CCCWaLYeVN+h8kFJZhOHp+TrW5jAaMWU2j34qquDs6TF9QRwSxR7Lwpr0MdnTxTLw+yHy/YzjM0227UPEzdTqvrkUpFqwmhFBDBLFHsvCmvwmGXIswqYbFXnLxgO+Gw8lybm6MnQFZjvI1Vb2mDLBwimCWKnSh6zXDk9GAHiZ0X7CQcTscelV5yp3ZKIhHIQiOCWaKotg/6DQHKV5XWT/U57LZK1ReSSluyxEwWFhHMEiZIb8qpxzaIfRm3YZanMzcDmxVht1Wq7t/PzJ9CfhDBFLLwKmx2PbZBxDeabSORGLg/1byQhWirtDqXqh6ueJjRQwRTyOAnMW15+cCHuaJCvffZSaiDHtqX7yBuL2FbuXY7RQKE3SFViohgChmCTiKrhxE5eVQqQq1SPfVTTXUzvls/Xn2svBuBd9upltsMkUpJj3iYiGAKGbx2hngdwqcLsYpQu/EwvcR7qiSxcAokVxV4K8GVEKHooyqYkg+zBPCaDNZpPaeciSr5IVVnswSAXbvUZ6xsbQWGD9cS6TrZYJb4Vyc3AbDVOSHqT9jb25t9Hopq5s1SR0VVo1LEw/SGnzZMp/XsvCc3bXqqnqZqO6ab0UJOzQJGT9xs23YjdsLuuRfUgFTJBSNeq4V+qpNuhdopoFu1HVNFfFWE3Ung7UZPORUJG4oWIphCaPjp0AjCw3QS3dy2UCeP1CkG1E409fZM8TCjjapgShumECitrdltdh0dQGcnsGSJNjdQU5P9+k7zBu3Z49z+Z9c2W10NLFrUb2tDAzBvHlBVpc2dY8azz9rvz26eG709M9eGkp8bJ66oqGpUiniY0cdvm51K+6OXKTB0z9JtblDV6rNdb7z0kkcfSJVcCIMgxnMbBcZrlTaMgPmoZUQS1FEVTKmSC4ESxHzWxql902nzZZymrjVuw6wpQHXqWyL16nNVVf//qZQ2p7tTE4QQL0QwhUAJej7rIATYz/rMzqKnt9saYz47O73bJkQXEUwhUJqaNM9K77xJJvuDv70EawctwHbbNcOpEwowD3zPDXgXigMRTCFwmpr6Bam3V/vO6wgXowATaX+DqOrmbjeVAioqspdRFWaVEU1CcUBae2c8aGxs5La2trDNEBRoaOgfKmikvl5rU4wira2aV7h5s1ZlX7hQTZjjeKxCNkS0ipkbnZYLxcMkoruI6D0i+h8iepqIDgzDDiF/xNHrcuoosiJfzQZC9AirSr4MwCRmPgbAOgA3h2SHkCfy1VkTRfLVbCBEj1AEk5mXMvP+vo+vAhgThh1C/ig1r8urdyrEiyh0+lwJ4M9WPxLR1UTURkRt27ZtK6BZgh/E6xKKkbx1+hDR8wBGmvy0gJn/0LfMAgCNAC5gBUOk00cQhHyg2ulTli8DmPk0u9+J6DIAZwOYoSKWgiAIYZM3wbSDiM4E8H0A/4uZLXJdC4IgRIuw2jDvBTAYwDIiWkNE94dkhyAIgjKheJjMfFgY+xUEQfBDFHrJBUEQYoEIpiAIgiIimIIgCIqIYAqCICgigikIgqCICKYgCIIiIpgB8PHHH+OSSy7B+PHjMWHCBMycORPr1q3Dpk2bMGnSpLzss6urCxdffDEOO+wwHH/88dgkiRcFIe+UnGDqc1EnEtpfL9MmGGFmnH/++Tj55JPxj3/8A++88w7uuOMOfPLJJ0GYa8kDDzyAoUOHYv369fiXf/kXfP/738/r/gRBKDHB1Ceram/XJrfyOm2CkeXLl6O8vBzXXntt5rspU6bgpJNOylpu06ZNOOmkkzB16lRMnToVf/3rXwEAH330EaZPn44pU6Zg0qRJWLlyJXp7e3H55Zdj0qRJOProo3H33XcP2O8f/vAHXHbZZQCACy+8EH/5y18gQ/IFIb+EMtInLOwmq/Kaduytt97CtGnTHJc76KCDsGzZMlRWVuL999/H3Llz0dbWht/+9rc444wzsGDBAvT29mLv3r1Ys2YNPvjgA7z11lsAgB07dgzY3gcffICxY8cCAMrKyjBkyBB0dHRg+PDh3g5EEARHSkoww5w2oaenB9/61rewZs0aJJNJrFu3DgDw5S9/GVdeeSV6enowa9YsTJkyBePGjcOGDRtw/fXX46yzzsLpp58+YHtm3iQR5f04BKGUKakqeT6mTZg4cSJWrVrluNzdd9+Ngw8+GGvXrkVbWxu6u7sBANOnT8eLL76I0aNHY968eXjkkUcwdOhQrF27FieffDLuu+8+XHXVVQO2N2bMGGzZsgUAsH//fuzcuRPDhg3zfiCCIDhSUoKZj2kTTj31VHR1deE//uM/Mt+98cYbeOGFF7KW27lzJ0aNGoVEIoElS5agt2/+2fb2dhx00EH4xje+gfnz52P16tXYvn070uk0Zs+ejdtvvx2rV68esN9zzz0XDz/8MADgiSeewKmnnioepiDkmZKqkuvtlF6mUrWCiPD000/j29/+Nu68805UVlaioaEBv/jFL7KWu+666zB79mz87ne/wymnnIKamhoAwIoVK3DXXXehvLwctbW1eOSRR/DBBx/giiuuQDqdBgD85Cc/GbDf+fPnY968eTjssMMwbNgwPPbYY94PQhAEJWReckEQSp5Iz0suCIIQR0QwBUEQFBHBFARBUEQEUxAEQRERTEEQBEVEMAVBEBQRwQyAMNK7vfjii5g6dSrKysrwxBNP5GUfgiBkU3qCGXB+t7DSu9XV1eGhhx7CP//zP+d1P4Ig9FNagpmH/G5hpXdraGjAMcccg0SitC6hIIRJSQ2NzEd+t7DSuwmCUHhKSzBDzO8WdHo3QRAKT2nV5/KQ3y2s9G6CIBSe0hLMPOR3Cyu9myAIIcDMsSnTpk1j37S0MNfXMxNpf1tafG/ygw8+4Dlz5vC4ceN4woQJPHPmTF63bh1v3LiRJ06cyMzM69at46OPPpqPP/54/sEPfsA1NTXMzPzQQw/xxIkTecqUKfzVr36VN2zYwGvWrOFjjz2WJ0+ezJMnT+Znn312wD5ff/11Hj16NFdXV/OwYcN4woQJvo9DEEoVAG2soEGS3k0QhJJH0rsJgiAEjAimIAiCIiKYgiAIiohgCoIgKCKCKQiCoIgIpiAIgiKhCCYR3U5E/0NEa4hoKREdEoYdgiAIbgjLw7yLmY9h5ikA/gTgRyHZIQiCoEwogsnMuwwfawDEJ3peEISSJbRsRUS0EMClAHYCOCUsOwRBEFTJ29BIInoewEiTnxYw8x8My90MoJKZb7HYztUAru77OAnAW0HbWgCGA9gethEeiavtcbUbiK/tcbUbAI5g5sFOC4U+lpyI6gE8w8yOk98QUZvKeM+oEVe7gfjaHle7gfjaHle7AXXbw+olP9zw8VwA74VhhyAIghvCasO8k4iOAJAG0A7gWoflBUEQQicUwWTm2R5XXRyoIYUjrnYD8bU9rnYD8bU9rnYDiraH3oYpCIIQF2RopCAIgiKxE8y4DqskoruI6L0+258mogPDtkkVIppDRG8TUZqIIt8LSkRnEtHfiWg9Ef0gbHtUIaIHiehTIopV6BwRjSWi5UT0bt99cmPYNqlARJVE9DoRre2z+8eO68StSk5EB+gjhYjoBgATmDnynUZEdDqA/8fM+4nopwDAzN8P2SwliOgoaB10vwFwEzNHdp4QIkoCWAfgawC2AngDwFxmfidUwxQgoukA9gB4RCXMLioQ0SgAo5h5NRENBrAKwKyon3MiIgA1zLyHiMoBvATgRmZ+1Wqd2HmYcR1WycxLmXl/38dXAYwJ0x43MPO7zPz3sO1Q5DgA65l5AzN3A3gMwHkh26QEM78I4LOw7XALM3/EzKv7/t8N4F0Ao8O1ypm++c/29H0s7yu2ehI7wQS0YZVEtAVAE+KZuONKAH8O24giZTSALYbPWxGDh7dYIKIGAMcCeC1cS9QgoiQRrQHwKYBlzGxrdyQFk4ieJ6K3TMp5AMDMC5h5LIBWAN8K19p+nOzuW2YBgP3QbI8MKrbHBDL5Lha1kLhDRLUAngTw7ZyaYGRh5t6+rGljABxHRLZNIaEl37CDmU9TXPS3AJ4BYDoOvdA42U1ElwE4G8AMjljjsYtzHnW2Ahhr+DwGwIch2VIy9LUBPgmglZmfCtsetzDzDiJaAeBM2OSriKSHaUdch1US0ZkAvg/gXGbeG7Y9RcwbAA4nokOJqALAJQD+GLJNRU1f58kDAN5l5p+HbY8qRDRCj1YhoioAp8FBT+LYS/4kgKxhlcz8QbhWOUNE6wEMAtDR99WrcejdBwAiOh/APQBGANgBYA0znxGuVdYQ0UwAvwCQBPAgMy8M2SQliOhRACdDy/rzCYBbmPmBUI1SgIi+CmAlgDehPZcA8ENmfjY8q5whomMAPAztPkkAeJyZb7NdJ26CKQiCEBaxq5ILgiCEhQimIAiCIiKYgiAIiohgCoIgKCKCKQiCoIgIplC0ENH/JaIdRPSnsG0RigMRTKGYuQvAvLCNEIoHEUwh9hDRl/vyjFYSUU1fbsNJzPwXALvDtk8oHiI5llwQ3MDMbxDRHwH8bwBVAFqYOVZJeIV4IIIpFAu3QRtHvg/ADSHbIhQpUiUXioVhAGoBDAZQGbItQpEigikUC4sB/Bu0PKM/DdkWoUiRKrkQe4joUgD7mfm3fXP6/JWITgXwYwBHAqgloq0A5jPzc2HaKsQbyVYkCIKgiFTJBUEQFBHBFARBUEQEUxAEQRERTEEQBEVEMAVBEBQRwRQEQVBEBFMQBEEREUxBEARF/j/TsM0lrvHNZwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xx, yy = np.meshgrid(np.arange(X_train_scaled[:,0].min(), X_train_scaled[:,0].max(), 0.1),\n",
    "                     np.arange(X_train_scaled[:,1].min(), X_train_scaled[:,1].max(), 0.1))\n",
    "zz = model.predict(np.c_[xx.ravel(),yy.ravel()]).reshape(xx.shape)\n",
    "\n",
    "\n",
    "plt.figure(figsize=(5,5))\n",
    "contour = plt.contour(xx, yy, zz, levels=1, alpha=1., vmin=0, vmax=1, linewidths=2, colors='k', linestyles='-')\n",
    "plt.plot(X_train_scaled[y_train==0,0], X_train_scaled[y_train==0,1], \"bo\", label=\"Class 0\")\n",
    "plt.plot(X_train_scaled[y_train==1,0], X_train_scaled[y_train==1,1], \"ro\", label=\"Class 1\")\n",
    "plt.xlabel(\"x1\")\n",
    "plt.ylabel(\"x2\")\n",
    "plt.ylim(-3,3)\n",
    "plt.xlim(-3,3)\n",
    "plt.legend(loc=3,  frameon=False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Multiclass Classification\n",
    "For the multiclass case, we see an example with the *iris dataset*. Unlike what done with other algorithms, however, this time we will use the dataset with all 4 original features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris \n",
    "X, y = load_iris(return_X_y=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "scikit-learn takes care internally (before training) of the encoding of the target variable, and so we don't need to perform one-hot encoding manually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now split the train/test set and standardize the features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.1, random_state=1)\n",
    "sc = StandardScaler()\n",
    "X_train_scaled = sc.fit_transform(X_train)\n",
    "X_test_scaled = sc.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The network**. In the multiclass case, a typical FNN has the following architecture: \n",
    "* as many output layers as there are classes (3 in this case)\n",
    "* cross-entropy loss \n",
    "* softmax activation function for the output layer\n",
    "\n",
    "We are going to create a netwrk with the following characteristics: \n",
    "* sigmoid activation function for the hidden layer (`activation='logistic'`)\n",
    "* early stopping (`early_stopping=True` with max. 1000 iterations without improvement on the validation set)\n",
    "* a maximum number of 2000 epochs (`max_iter=2000`)\n",
    "* ADAM (`solver='adam'`) with:\n",
    "    * batch size = 15 (`batch_size=15`) \n",
    "    * starting learning rate = 0.001 (`learning_rate_init=0.001`) \n",
    "    * decay of first moment = 0.9 (`beta_1=0.9`)\n",
    "    * decay of second moment = 0.999 (`beta_2=0.999`)\n",
    "    * numerical stability parameter = 1e-8 (epsilon=1e-8)\n",
    "* $l_2$ penalty (`alpha=0.001`)\n",
    "\n",
    "As usualy, we set `validation_fraction=0.1` to evaluate early stopping, and `random_state=1` for reproducibility. MLPClassifier  automatically sets the `softmax` as the activation of the output layer in the multi-class case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MLPClassifier(hidden_layer_sizes=(15, ), \n",
    "                     activation='logistic', \n",
    "                     solver='adam',\n",
    "                     beta_1=0.9, \n",
    "                     beta_2=0.999,\n",
    "                     epsilon=1e-8,\n",
    "                     alpha=0.0001, \n",
    "                     batch_size=15, \n",
    "                     learning_rate_init=0.001,                  \n",
    "                     max_iter=2000, \n",
    "                     random_state=1, \n",
    "                     tol=0.001,   # the tolerance criterion for the optimization algorithm\n",
    "                     verbose=False, \n",
    "                     early_stopping=True, \n",
    "                     validation_fraction=0.1, \n",
    "                     n_iter_no_change=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now train the model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPClassifier(activation='logistic', alpha=0.0001, batch_size=15, beta_1=0.9,\n",
       "              beta_2=0.999, early_stopping=True, epsilon=1e-08,\n",
       "              hidden_layer_sizes=(15,), learning_rate='constant',\n",
       "              learning_rate_init=0.001, max_fun=15000, max_iter=2000,\n",
       "              momentum=0.9, n_iter_no_change=1000, nesterovs_momentum=True,\n",
       "              power_t=0.5, random_state=1, shuffle=True, solver='adam',\n",
       "              tol=0.001, validation_fraction=0.1, verbose=False,\n",
       "              warm_start=False)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATwAAADCCAYAAADU6n7bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAaiklEQVR4nO3deXDc9Z3m8fenL6lbtyzZMpJPMIc5DI4xkABJJktiM0mcycxmYYdATg8zIZtJpiohlZrd7E7t7mTZZHKReBhCSDIEJ5ULJ2tyJxwJhw0YgzEG4VP4kg9Zsq5Wd3/2j26RRkhWS26pDz2vKpX6d3T3I7n0+Pvr32XujojITBAodAARkemiwhORGUOFJyIzhgpPRGYMFZ6IzBgqPBGZMUKFeuOmpiZfuHBhod5eRMrUE088ccTdm0dbVrDCW7hwIZs3by7U24tImTKzPWMt0yatiMwYKjwRmTFUeCIyY6jwRGTGKJnC+98bt7P+8b2FjiEiJaxkCu9X2w/xcPuRQscQkRJWMoVXHw1zon+o0DFEpISVTuHFInT1qfBEZPJKp/CiYbr644WOISIlrGQKry4W1ghPRE5LyRRefTRCz0CCRDJV6CgiUqJKpvBmVUcAONqrzVoRmZySKbw5tZUAHOoeKHASESlVJVN4LZnCO3hChScik1MyhTentgKAQz2DBU4iIqVq3MIzs7vM7LCZPTvGcjOzL5tZu5ltNbPl+Y8Js6orCAaMQxrhicgk5TLCuxtYdYrlq4Elma+1wNdPP9ZrBQNGc3WFPsMTkUkbt/Dc/UHg2ClWWQN829MeBerNbG6+AmabU1vBQRWeiExSPj7DawX2ZU13ZOa9hpmtNbPNZra5s7Nzwm80p7aSw936DE9EJicfhWejzPPRVnT3O9x9hbuvaG4e9R4bpzSntpJDPRrhicjk5KPwOoB5WdNtwP48vO5rzKmtoKtviIGh5FS8vIiUuXwU3gbgxsze2suBE+5+IA+v+xrDBx9rs1ZEJmPc2zSa2b3Am4AmM+sA/hsQBnD3dcBG4FqgHegD3j9VYYcL72D3APNnxabqbUSkTI1beO5+/TjLHfhI3hKdgk4vE5HTUTJnWsCfTi9T4YnIZJRU4dVGQ8QiQV7u6i90FBEpQSVVeGZGW0OUjuMqPBGZuJIqPIDW+igvq/BEZBJKrvDaGmJ0HO8rdAwRKUElV3itDVG6BxL0DOj+FiIyMSVXeG0NUQDtuBCRCSu5wmutTxdexzEVnohMTMkVXltD+gwLjfBEZKJKrvCaqiNUhALacSEiE1ZyhWdmtDZENcITkQkrucKD9Od4OvhYRCaqJAuvrSGmg49FZMJKtPCiHO2N0xdPFDqKiJSQki08QKM8EZmQEi289KEp+7SnVkQmoCQLb15jeoS3Twcfi8gElGThNVdXUBkOsO+YRngikruSLLz0dfFi2qQVkQnJqfDMbJWZ7TCzdjO7dZTldWb2UzN72sy2mdmU3chn2LyGqDZpRWRCxi08MwsCtwOrgaXA9Wa2dMRqHwGec/dlpO9w9nkzi+Q566vMa9QIT0QmJpcR3kqg3d13unscWA+sGbGOAzVmZkA1cAyY0oPk5jfG6BlIcKJP18UTkdzkUnitwL6s6Y7MvGxfBc4D9gPPAB9z91ReEo5Bh6aIyETlUng2yjwfMf02YAtwBnAx8FUzq33NC5mtNbPNZra5s7NzwmGzzW9MF96eoyo8EclNLoXXAczLmm4jPZLL9n7gR57WDuwCzh35Qu5+h7uvcPcVzc3Nk80MwIJZ6cLbfbT3tF5HRGaOXApvE7DEzBZldkRcB2wYsc5e4C0AZjYHOAfYmc+gI1VVhGiprWRnpwpPRHITGm8Fd0+Y2S3AL4AgcJe7bzOzmzPL1wH/BNxtZs+Q3gT+lLsfmcLcACxqqmLXkZNT/TYiUibGLTwAd98IbBwxb13W4/3AW/MbbXyLm6v4f88cmO63FZESVZJnWgxb1FRFV98Qx3vjhY4iIiWgpAtvcXMVADuP6HM8ERlfSRfeoqZqAHap8EQkByVdeG0NUUIB044LEclJSRdeOBhgfmNMIzwRyUlJFx6kd1zoWDwRyUXJF97i5ip2HekllRp5tpuIyKuVfOEtaqpmMJHiQPdAoaOISJErg8JLH5qyS5u1IjKOki+84WPxtKdWRMZT8oU3u6aCWCSog49FZFwlX3hmpj21IpKTki88GL5qigpPRE6tLArvzOZq9h3voz+eLHQUESliZVF4582twR1eONRT6CgiUsTKpPDSt8/YfqC7wElEpJiVReHNa4hRFQny/EGN8ERkbGVReIGAcU5LjUZ4InJKZVF4AOfOrWX7gW7cdU6tiIyubArvvLm1dA8kOHBC59SKyOhyKjwzW2VmO8ys3cxuHWOdN5nZFjPbZmYP5Dfm+M5rqQG040JExjZu4ZlZELgdWA0sBa43s6Uj1qkHvga8093PB/7jFGQ9pXMyhacdFyIyllxGeCuBdnff6e5xYD2wZsQ6/xn4kbvvBXD3w/mNOb6ayjALZsV4puPEdL+1iJSIXAqvFdiXNd2RmZftbKDBzH5vZk+Y2Y2jvZCZrTWzzWa2ubOzc3KJT2FZWz1Pd3Tl/XVFpDzkUng2yryRu0JDwOuAPwfeBvyjmZ39mie53+HuK9x9RXNz84TDjmfZvHoOnBjgkC4GKiKjyKXwOoB5WdNtwP5R1vm5u/e6+xHgQWBZfiLm7uJ59QBs2adRnoi8Vi6FtwlYYmaLzCwCXAdsGLHOfcBVZhYysxhwGbA9v1HHd/4ZtUSCAZ7Yc3y631pESkBovBXcPWFmtwC/AILAXe6+zcxuzixf5+7bzeznwFYgBdzp7s9OZfDRVIaDLJtXx+O7jk33W4tICRi38ADcfSOwccS8dSOmbwNuy1+0ybl0YSN3PLiT3sEEVRU5/XgiMkOUzZkWw1YuaiSRcjZrs1ZERijLwgsHjT+0Hyl0FBEpMmVXeLFIiOXzG3joRRWeiLxa2RUewFVLmth+oJsjJwcLHUVEikhZFt6VS9IHNWuzVkSylWXhXdhaR100zMParBWRLGVZeMGA8YazZvHAC52kUrogqIiklWXhAbx1aQuHewZ5ap8OTxGRtLItvD87bzaRYICNzxwsdBQRKRJlW3i1lWGuWtLEz589qPtciAhQxoUHsOqCFl7u6udpXRRURCjzwnvr+S1EQgF+8tTLhY4iIkWgrAuvLhrmmvPmsOHp/QwlU4WOIyIFVtaFB/Du5a0c643z+x35v6S8iJSWsi+8q89uprmmgm8/srvQUUSkwMq+8MLBAO97/UIeevEI2/Zr54XITFb2hQdww2ULiEWC/NuDOwsdRUQKaEYUXl0szPUr5/PTrQd4uau/0HFEpEBmROEBfODKRQDc9fCuAicRkULJqfDMbJWZ7TCzdjO79RTrXWpmSTP7q/xFzI/W+ijvuGgu6x/fy4n+oULHEZECGLfwzCwI3A6sBpYC15vZ0jHW+xzpu5sVpbVXn0lvPMk9j+0pdBQRKYBcRngrgXZ33+nucWA9sGaU9T4K/BA4nMd8ebX0jFquWtLEXQ/v5uRgotBxRGSa5VJ4rcC+rOmOzLxXmFkr8BfAq27dWIw+cc3ZHDk5yFd+82Kho4jINMul8GyUeSMvP/JF4FPunjzlC5mtNbPNZra5s7MwZz5cMr+B96xo4xsP76L98MmCZBCRwsil8DqAeVnTbcD+EeusANab2W7gr4Cvmdm7Rr6Qu9/h7ivcfUVzc/MkI5++T646l2gkyGc3bNOlo0RmkFwKbxOwxMwWmVkEuA7YkL2Cuy9y94XuvhD4AfB37v6TvKfNk6bqCv7hmrN5uP0IP39WFwgVmSnGLTx3TwC3kN77uh34vrtvM7ObzezmqQ44VW64fAHnttTwyR9uZd+xvkLHEZFpYIXapFuxYoVv3ry5IO89bM/RXt7+5Yc5d24N69deQTAw2seVIlJKzOwJd18x2rIZc6bFaBbMquJ/vOt8Nu0+zroHXip0HBGZYjO68ADedXEr71h2Bp//5Q4eeEHXzBMpZzO+8MyMz/3lhSyZXcMt9zzJjoM9hY4kIlNkxhceQCwS4us3LCcUND5w9yZdUUWkTKnwMhY3V3PHjSvo6ovz3jsfo2dAFxgQKTcqvCyXLmzkzpsuZc+xPj5w9yaOnBwsdCQRySMV3ghXnDmLf/lPF/PU3i7efNvveXTn0UJHEpE8UeGN4p3LzuAHf/t6ZlVHuPEbj3PfFt3XVqQcqPDGcPG8eu77yJVcMr+ev//eFn62deTpwyJSalR4p1AXC3P3+1eyYkEDH//eFo30REqcCm8c0UiQO2+6lGVt9Xxs/RY+8+NntAdXpESp8HJQFw1z79rL+ZurF3PPY3t5z78+ylHtwRUpOSq8HIWDAT597Xl8832XsrPzJGtu/wNbO7oKHUtEJkCFN0FvPnc23/+bK0imnHd/7Y/c/rt2UildRFSkFKjwJmHZvHru/9hVrLqghdt+sYObvvk4nT3axBUpdiq8SaqPRfjK9Zfwz+++kMd3HWP1lx5i4zMHdMl4kSKmwjsNZsZ1K+dz3y1vYE5tBX93z5N8+Nub6TiuKyiLFCMVXh6c21LLfR95A5+59jz+0H6Ua77wIF/89QsMDJ3yJm4iMs1UeHkSCgb48NWL+dUnruaqJU188dcv8o6vPMwjL+lcXJFiocLLs7aGGHfcuIJvfWAlffEk1//bo3zw7k28eEgXFhUptJwKz8xWmdkOM2s3s1tHWf7XZrY18/VHM1uW/6il5Y1nN/Obf3gjn1x1Do/vOsaqLz3EZzds43hvvNDRRGasce9aZmZB4AXgGtI35d4EXO/uz2Wt83pgu7sfN7PVwGfd/bJTvW4x3LVsuhzrjfP5X+7g3sf3UhkOcuMVC/nbN51JXTRc6GgiZed071q2Emh3953uHgfWA2uyV3D3P7r78czko0Db6QQuN41VEf7nX1zI/R+7mmuWzuFfH3yJN972O77x8C4GE9qxITJdcim8VmBf1nRHZt5YPgjcP9oCM1trZpvNbHNn58y7Q9g5LTV86bpL+NlHr+TC1jr+6WfPcfn/+g3feWQ3SZ2tITLlcim80e5OPepfp5m9mXThfWq05e5+h7uvcPcVzc3NuacsM+efUcd3PngZ93zoMs5tqeUf79vGNV94gB8/1UEimSp0PJGylUvhdQDzsqbbgNdcDdPMLgLuBNa4u47FyMEbzmriux++jHU3LCcSCvDx7z3NhZ/9JeseeIn+uDZ1RfItl8LbBCwxs0VmFgGuAzZkr2Bm84EfAe919xfyH7N8mRmrLpjLxv9yFetueB31sTD/fP/zXPMvD/DTp/drxCeSR6HxVnD3hJndAvwCCAJ3ufs2M7s5s3wd8F+BWcDXzAwgMdZeEhldIGCsuqCFt50/hz++dJTPbtjGR+99itb6KO+9YgHXXTqP+lik0DFFStq4h6VMlZl0WMpkJFPOr7cf4u4/7OaRnUepDAd457IzuH7lfC6eV0/mPxYRGeFUh6Wo8ErA8we7+dYfd3Pflv30xZOc21LDmotbec+KNmZVVxQ6nkhRUeGViZ6BIe7bsp8fPNHBln1dVIQCvHt5K++46AxWLmokFNSZgiIqvDLUfvgkdz60k59seZmBoRSNVRGuOW8Oqy5o4fVnzaIiFCx0RJGCUOGVsb54ggd2dHL/swf57fOHOTmYIBoOcvniRt6x7Azecu4c6mI6hU1mjlMV3rh7aaW4xSIhVl84l9UXzmUwkeQP7Ud48IUjbHzmAL/b0UnA0pekv/KsJq48q4nlCxoIa9NXZiiN8MpUKuVsffkEv91+iIfaj/D0vi5Snr7l5OWLG3ndggZet6CRC1prtfkrZUWbtMKJ/iEeeekov3ruEJt2H2PvsfRl6CPBABe21bF8fj1nza7mrNnVXDKvgUBAh71IadImrVAXDbPqghZWXdACwOGeAZ7c08WTe4/zxJ7jfOuRPcQTqVfWvaitjqVzazmnpYYLWutYMCumkaCUPI3wBIDBRJIdB3t4bn83W/Z1sbXjBO2HTxLPnNoWsPTVnBc3V3FmczWLm6tY3FRNa32UufWV+lxQioZGeDKuilCQi9rquaitnutWzgcgkUyx80gvz+3vZueRXnZ2nmRnZy+P7TxGf9YNiiKhAG0NUeY1xJjXGKWltpLZtZWc2VzN3LpKmmsqcE+vJ1JIKjwZUygY4Ow5NZw9p+ZV81Mp50D3ADs7T7LvWD97jvay91gf+473sWVfFyf6h0Z9vfmNMc5srqI2GqYuGmZ+Y4zZtZU0xMI0xCLUx8LMqqogGtGms0wNFZ5MWCBgtNZHaa2Pjrp8YCjJwRMD7DxykoMnBtnf1c/xvjjPH+yh8+QgLxw6SXf/ED2DiVGfXxEKUFMZoqm6guaaChqrItRWpkuyPhamNhqmtjJETWWY2sowtdEQnT2DnNNSQ02ljjmUsanwJO8qw0EWNlWxsKlqzHXcne7+BId7BjjeN8TxvjhdfXGO9sbp6huiZyDBkZODdPYMsudoH90DQ3T3DzHehaErwwGqIiGqKkJUhAJUVYSoqQy9Mi8SClAfC1MVCdLVN4QZLJhVhbvTUBWhKhKiIhwgGg5SGQ4SDQeJRYLURsNUhAK4Q288QVUk9MqebHfXxRxKhApPCsLMqIuFJ3QWSCrl9Awk6B5IF2LPwBDdAwm6+uIkU86BEwP0DyXpHUzQO5hgYChFbzzBycEEh7oHODmQIJ5M0dU3RGISl9QPBoxgwIgnUphBdUWIynCQE/1DNFVFqItF6IsnaK6uoKEqgsErpRmNBAkFjHAoQChgpNxJOaTcwSGeTNEYi9BSV0kkFCAUCBAKGuGgZT0OEAwY4cx0wIyBoSRDyRT1sQixSJBIKPDKJcqjkSCRzHNUyGkqPCkZgcDES3Isg4kkyZQTDgbYc7QXMJIpp38oSX88yUAiyUDm+8nBJD0DQ/QNJhkYStJYHaE/nnylfHHAoKtviGg4yLHeOPsyxzkOJlL0xRP0x5MkUk48kSLlTsDShYWl94AHzeid4qtcBwNGZShANBIiHEwX4PDvIBS0V8o0GDBCQSMUSJdtMFPUw4dmVoaCBING0NLrDX8PWPo5gYCxafcxmqsraG2IUlsZJpR5cv9QkpSn7xsx/B9IY1WEcDBAIJB+LYBD3QNEI0EaYmHeePZsWuoq8/I7UOHJjJR9TOFZs2tOseb0GRhKcrh7kHgyRTLlDCVTJFJOIpliKOkkUsPT6XkpT2/Ch4IBuvri9MXToz0Ad16ZTiRTnOgfojIcpH8oXdqZgSWhgL3qtZNJT79HKp0hkVnW158EdxzojydJupNKOUNJJ+VOMpX5yjzuGUiwqKmKJ/d20RdPMJRMj6ij4SABS7/38HPGG21/90OXqfBEyk1lOMj8WbFCx8iLVMpfc7bOaJ91Dn+Wm0ilMiUKSXdqKkMEzDjRP8Ssqvxd6VuFJyJ5N9qpiaN9jjj8We5YqivyW1E6ElREZoycCs/MVpnZDjNrN7NbR1luZvblzPKtZrY8/1FFRE7PuIVnZkHgdmA1sBS43syWjlhtNbAk87UW+Hqec4qInLZcRngrgXZ33+nucWA9sGbEOmuAb3vao0C9mc3Nc1YRkdOSS+G1Avuypjsy8ya6johIQeWyC2S0Q7RHHjiTyzqY2VrSm7wAJ81sRw7vn60JODLB50y3Ys9Y7PlAGfOh2PPB1GVcMNaCXAqvA5iXNd0G7J/EOrj7HcAdObznqMxs81jXuSoWxZ6x2POBMuZDseeDwmTMZZN2E7DEzBaZWQS4DtgwYp0NwI2ZvbWXAyfc/UCes4qInJZxR3junjCzW4BfAEHgLnffZmY3Z5avAzYC1wLtQB/w/qmLLCIyOTkdxuzuG0mXWva8dVmPHfhIfqONatKbw9Oo2DMWez5Qxnwo9nxQgIwFu6eFiMh006llIjJjlEThjXdq2zTmmGdmvzOz7Wa2zcw+lpnfaGa/MrMXM98bsp7z6UzuHWb2tmnKGTSzp8zsZ0War97MfmBmz2d+l1cUYcaPZ/6NnzWze82sstAZzewuMztsZs9mzZtwJjN7nZk9k1n2Zcvj1UHHyHhb5t96q5n92MzqC5bR3Yv6i/SOkpeAxUAEeBpYWqAsc4Hlmcc1wAukT7f7P8Ctmfm3Ap/LPF6ayVsBLMr8HMFpyPkJ4LvAzzLTxZbvW8CHMo8jQH0xZSR90PwuIJqZ/j7wvkJnBK4GlgPPZs2bcCbgceAK0sfP3g+snuKMbwVCmcefK2TGUhjh5XJq27Rw9wPu/mTmcQ+wnfQfxxrSf8Rkvr8r83gNsN7dB919F+m92CunMqOZtQF/DtyZNbuY8tWS/qP4BoC7x929q5gyZoSAqJmFgBjp40oLmtHdHwSOjZg9oUyZUz5r3f0RTzfLt7OeMyUZ3f2X7j58x6ZHSR+nW5CMpVB4RXnampktBC4BHgPmeOa4w8z32ZnVCpH9i8AngVTWvGLKtxjoBL6Z2ey+08yqiimju78M/F9gL3CA9HGlvyymjFkmmqk183jk/OnyAdIjNihAxlIovJxOW5tOZlYN/BD4e3fvPtWqo8ybsuxm9nbgsLs/ketTRpk31b/bEOlNnq+7+yVAL+lNsbFMe8bM52BrSG9mnQFUmdkNp3rKKPMKffjDWJkKltXMPgMkgHuGZ42RZcoylkLh5XTa2nQxszDpsrvH3X+UmX0oMwwn8/1wZv50Z38D8E4z20160//PzOzfiyjf8Ht2uPtjmekfkC7AYsr4H4Bd7t7p7kPAj4DXF1nGYRPN1MGfNimz508pM7sJeDvw15nN1IJkLIXCy+XUtmmR2VP0DWC7u38ha9EG4KbM45uA+7LmX2dmFWa2iPT1Ah+fqnzu/ml3b3P3haR/T7919xuKJV8m40Fgn5mdk5n1FuC5YspIelP2cjOLZf7N30L689piyjhsQpkym709ZnZ55me7Mes5U8LMVgGfAt7p7n0jsk9vxnzvSZqKL9Knrb1Aei/OZwqY40rSQ+utwJbM17XALOA3wIuZ741Zz/lMJvcO8rg3LIesb+JPe2mLKh9wMbA583v8CdBQhBn/O/A88CzwHdJ7EguaEbiX9GeKQ6RHQR+cTCZgRebnegn4KpkTEKYwYzvpz+qG/2bWFSqjzrQQkRmjFDZpRUTyQoUnIjOGCk9EZgwVnojMGCo8EZkxVHgiMmOo8ERkxlDhiciM8f8B+rwG3QYNFT8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(5,3))\n",
    "plt.plot(model.loss_curve_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The algorithm seems to have converged. Let's assess the predictions of the first ten test units:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.06137653e-02, 8.94082281e-01, 9.53039541e-02],\n",
       "       [1.24957860e-01, 8.68238360e-01, 6.80377961e-03],\n",
       "       [1.94671713e-03, 5.38950228e-01, 4.59103055e-01],\n",
       "       [9.74173103e-01, 2.57784764e-02, 4.84202629e-05],\n",
       "       [8.55910563e-03, 9.45845784e-01, 4.55951101e-02],\n",
       "       [9.43189855e-01, 5.67566377e-02, 5.35070347e-05],\n",
       "       [9.69202130e-01, 3.07095639e-02, 8.83060499e-05],\n",
       "       [5.55676110e-04, 1.24505831e-01, 8.74938493e-01],\n",
       "       [2.33373492e-02, 4.80090611e-01, 4.96572040e-01],\n",
       "       [2.14974750e-03, 1.47385879e-01, 8.50464373e-01]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Probabilities\n",
    "model.predict_proba(X_test_scaled[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 0, 1, 0, 0, 2, 2, 2])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predicted Classes\n",
    "model.predict(X_test_scaled[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test set accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Cross-Validation with the Heart Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now going to perform tuning with cross-validation using the Heart data. Let's first load the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train = pd.read_csv(\"./data/heart_data/heart_train_processed.csv\")\n",
    "data_test = pd.read_csv(\"./data/heart_data/heart_test_processed.csv\")\n",
    "X_train = data_train.drop(\"y\", axis=1)\n",
    "X_test = data_test.drop(\"y\", axis=1)\n",
    "y_train = data_train[\"y\"]\n",
    "y_test = data_test[\"y\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the large number of hyperparameters that need to be optimized, we will use random search, rather than Grid Search (which is nonetheless still possible, if you prefer manually specify a grid of value). As it is highly unlikely to find the best setting for a Neural Network with just grid/randomized search methods, a good tip would be to use these methodologies to find a good initial setting for (some of) the hyperparameters, and then manually refine these (and other) hyperparameter values, with a validation set. Moreover, for large datasets tuning a Neural Network with cross-validation is not recommended given the large computational costs involved in this operation. \n",
    "\n",
    "The ranodmized tuning will occur on a single-layer network (remember: networks with a single layers are universal approximators, as long as they have a sufficiently large enough number of neurons), and the hyperparameters will be tuned as follows: \n",
    "* we will choose between 1 and 20 neurons in the hidden layer from a uniform distribution in this range\n",
    "* we will use a [reciprocal distribution](https://en.wikipedia.org/wiki/Reciprocal_distribution) for the learning rate of SGD; this distribution will range between 1e-4 and 0.99, giving larger probabilities to smaller values\n",
    "* we will search how many optimal iterations to wait before stopping the algorithm (between 50, 100, 500, 1000, and 2000)\n",
    "* we will choose between SGD and ADAM optimizer\n",
    "\n",
    "The search will occur across 10 iterations (of course, in more intensive analyses you should increase such number). Let's initialize the random grid: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import reciprocal    # Reciprocal distribution in scipy.stats\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "param_distribs = {\n",
    "    \"hidden_layer_sizes\": [i for i in range(1, 20)],\n",
    "    \"learning_rate_init\": reciprocal(1e-4, 9e-1),\n",
    "    \"n_iter_no_change\": [50, 100, 500, 1000, 2000],\n",
    "    \"solver\" : [\"sgd\", \"adam\"]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to perform cross-validation (we will use 3-fold CV; the maximum number of epochs for the neural network is set equal to 10000): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 2 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  30 out of  30 | elapsed:   16.9s finished\n",
      "c:\\users\\u1253415\\appdata\\local\\programs\\python\\python38-32\\lib\\site-packages\\sklearn\\model_selection\\_search.py:823: FutureWarning: The parameter 'iid' is deprecated in 0.22 and will be removed in 0.24.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=3, error_score=nan,\n",
       "                   estimator=MLPClassifier(activation='relu', alpha=0.0001,\n",
       "                                           batch_size='auto', beta_1=0.9,\n",
       "                                           beta_2=0.999, early_stopping=False,\n",
       "                                           epsilon=1e-08,\n",
       "                                           hidden_layer_sizes=(100,),\n",
       "                                           learning_rate='constant',\n",
       "                                           learning_rate_init=0.001,\n",
       "                                           max_fun=15000, max_iter=10000,\n",
       "                                           momentum=0.9, n_iter_no_change=10,\n",
       "                                           nesterovs_momentum=True, power_t=0.5,\n",
       "                                           rand...\n",
       "                   iid=False, n_iter=10, n_jobs=-1,\n",
       "                   param_distributions={'hidden_layer_sizes': [1, 2, 3, 4, 5, 6,\n",
       "                                                               7, 8, 9, 10, 11,\n",
       "                                                               12, 13, 14, 15,\n",
       "                                                               16, 17, 18, 19],\n",
       "                                        'learning_rate_init': <scipy.stats._distn_infrastructure.rv_frozen object at 0x0F2BCEC8>,\n",
       "                                        'n_iter_no_change': [50, 100, 500, 1000,\n",
       "                                                             2000],\n",
       "                                        'solver': ['sgd', 'adam']},\n",
       "                   pre_dispatch='2*n_jobs', random_state=1, refit=True,\n",
       "                   return_train_score=False, scoring=None, verbose=1)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnd_search_cv = RandomizedSearchCV(MLPClassifier(max_iter=int(1e+4), random_state=1), \n",
    "                                   param_distribs, n_iter=10, cv=3, n_jobs=-1, iid=False, random_state=1, \n",
    "                                  verbose=1)\n",
    "rnd_search_cv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best set of hyperparameters was: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'hidden_layer_sizes': 10,\n",
       " 'learning_rate_init': 0.0015685425729289283,\n",
       " 'n_iter_no_change': 50,\n",
       " 'solver': 'sgd'}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnd_search_cv.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CV accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8429526748971193"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnd_search_cv.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the accuracy of the best model on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8360655737704918"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnd_search_cv.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last, you can store the best model in a separate object, with the `best_estimator_.model` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPClassifier(activation='relu', alpha=0.0001, batch_size='auto', beta_1=0.9,\n",
       "              beta_2=0.999, early_stopping=False, epsilon=1e-08,\n",
       "              hidden_layer_sizes=10, learning_rate='constant',\n",
       "              learning_rate_init=0.0015685425729289283, max_fun=15000,\n",
       "              max_iter=10000, momentum=0.9, n_iter_no_change=50,\n",
       "              nesterovs_momentum=True, power_t=0.5, random_state=1,\n",
       "              shuffle=True, solver='sgd', tol=0.0001, validation_fraction=0.1,\n",
       "              verbose=False, warm_start=False)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = rnd_search_cv.best_estimator_\n",
    "model"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}