new_NNCF10000_spotify.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Collaborative Filtering (NNCF)\n",
    "***\n",
    "Besides classical matrix factorization, we also implemented the neural network version of collaborative filtering: using deep network model to perform matrix factorization, as well as allowing extra playlist-track mixing. \n",
    "\n",
    "The code implemented here was adopted from this 2017 WWW paper on __[Neural Collaborative Filtering](https://www.comp.nus.edu.sg/~xiangnan/papers/ncf.pdf)__ and a comprehensive __[blogpost](https://nipunbatra.github.io/blog/2017/neural-collaborative-filtering.html)__ written by Dr. Nipun Batra, Assistant Professor in Computer Science Department at IIT Gandhinagar. We applied the architecture of the NNCF model to a subset of data consisting of 10,000 playlists genereated with a seed track (a popular song that belongs to >10,000 playlists), training the network to perform binary classifcation on playlist-track pairs (predicting 0 vs. 1), and evaluated the model performance.\n",
    "\n",
    "***\n",
    "First we import the libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import scipy.sparse as sps\n",
    "import scipy\n",
    "import pickle\n",
    "import keras\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "from keras.models import load_model\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import roc_auc_score\n",
    "import itertools\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data and model\n",
    "\n",
    "***\n",
    "Here we loaded the sparse matrix containing track-playlist contingency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load files\n",
    "sps_acc = sps.load_npz('sparse_10000.npz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<74450x10000 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 1030015 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert matrix to csc type\n",
    "sps_acc = sps_acc.tocsc()\n",
    "sps_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up model architecture\n",
    "***\n",
    "For building the model, we need to determine the number of latent factors for the embedding layers of ***matrix factorization stream***, as well as those for the ***multilayer perceptron stream***, for both playlist and track inputs. We set the former to 50, and the latter to 10, as a simple start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set latent factor number\n",
    "n_tracks, n_playlists = sps_acc.shape[0], sps_acc.shape[1]\n",
    "n_latent_mf = 50\n",
    "n_latent_playlist = 10\n",
    "n_latent_track = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network architecture\n",
    "***\n",
    "This architecture was adopted from the blogpost mentioned above. For the final prediction layer, we set it to a 2-unit dense connected layer for binary classification with softmax activation, and used binary crossentropy as loss function. We compiled the model with Adam optimizer, with a mild learning rate decay. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build model\n",
    "from keras.layers import Input, Embedding, Flatten, Dropout, Concatenate, Dot, BatchNormalization, Dense\n",
    "\n",
    "# track embedding stream\n",
    "track_input = Input(shape=[1],name='Track')\n",
    "track_embedding_mlp = Embedding(n_tracks + 1, n_latent_track, name='track-Embedding-MLP')(track_input)\n",
    "track_vec_mlp = Flatten(name='Flatten_tracks-MLP')(track_embedding_mlp)\n",
    "track_vec_mlp = Dropout(0.2)(track_vec_mlp)\n",
    "\n",
    "track_embedding_mf = Embedding(n_tracks + 1, n_latent_mf, name='track-Embedding-MF')(track_input)\n",
    "track_vec_mf = Flatten(name='Flatten_tracks-MF')(track_embedding_mf)\n",
    "track_vec_mf = Dropout(0.2)(track_vec_mf)\n",
    "\n",
    "# playlist embedding stream\n",
    "playlist_input = Input(shape=[1],name='Playlist')\n",
    "playlist_vec_mlp = Flatten(name='Flatten_playlists-MLP')(Embedding(n_playlists + 1, n_latent_playlist,name='playlist-Embedding-MLP')(playlist_input))\n",
    "playlist_vec_mlp = Dropout(0.2)(playlist_vec_mlp)\n",
    "\n",
    "playlist_vec_mf = Flatten(name='Flatten_playlists-MF')(Embedding(n_playlists + 1, n_latent_mf,name='playlist-Embedding-MF')(playlist_input))\n",
    "playlist_vec_mf = Dropout(0.2)(playlist_vec_mf)\n",
    "\n",
    "# MLP stream\n",
    "concat = Concatenate(axis=-1,name='Concat')([track_vec_mlp, playlist_vec_mlp])\n",
    "concat_dropout = Dropout(0.2)(concat)\n",
    "dense = Dense(200,name='FullyConnected')(concat_dropout)\n",
    "dense_batch = BatchNormalization(name='Batch')(dense)\n",
    "dropout_1 = Dropout(0.2,name='Dropout-1')(dense_batch)\n",
    "dense_2 = Dense(100,name='FullyConnected-1')(dropout_1)\n",
    "dense_batch_2 = BatchNormalization(name='Batch-2')(dense_2)\n",
    "dropout_2 = Dropout(0.2,name='Dropout-2')(dense_batch_2)\n",
    "dense_3 = Dense(50,name='FullyConnected-2')(dropout_2)\n",
    "dense_4 = Dense(20,name='FullyConnected-3', activation='relu')(dense_3)\n",
    "\n",
    "# end prediction for both streams\n",
    "pred_mf = Dot(1,name='Dot')([track_vec_mf, playlist_vec_mf])\n",
    "pred_mlp = Dense(1, activation='relu',name='Activation')(dense_4)\n",
    "\n",
    "# combine both stream\n",
    "combine_mlp_mf = Concatenate(axis=-1,name='Concat-MF-MLP')([pred_mf, pred_mlp])\n",
    "result_combine = Dense(100,name='Combine-MF-MLP')(combine_mlp_mf)\n",
    "deep_combine = Dense(100,name='FullyConnected-4')(result_combine)\n",
    "\n",
    "# final prediction layer\n",
    "result = Dense(2, activation = 'softmax', name='Prediction')(deep_combine)\n",
    "\n",
    "# build model\n",
    "model = keras.Model([playlist_input, track_input], result)\n",
    "\n",
    "# compile model\n",
    "opt = keras.optimizers.Adam(lr = 0.1, decay = 1e-5)\n",
    "model.compile(optimizer='adam', loss= 'binary_crossentropy', metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the visualization and summary of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"1359pt\" viewBox=\"0.00 0.00 948.50 1359.00\" width=\"949pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 1355)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-1355 944.5,-1355 944.5,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 4394662600 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>4394662600</title>\n",
       "<polygon fill=\"none\" points=\"163,-1314.5 163,-1350.5 282,-1350.5 282,-1314.5 163,-1314.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"222.5\" y=\"-1328.8\">Track: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4394662712 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>4394662712</title>\n",
       "<polygon fill=\"none\" points=\"229,-1241.5 229,-1277.5 448,-1277.5 448,-1241.5 229,-1241.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"338.5\" y=\"-1255.8\">track-Embedding-MLP: Embedding</text>\n",
       "</g>\n",
       "<!-- 4394662600&#45;&gt;4394662712 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>4394662600-&gt;4394662712</title>\n",
       "<path d=\"M251.1742,-1314.4551C266.232,-1304.979 284.8686,-1293.2508 301.053,-1283.0658\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"303.1543,-1285.8788 309.7537,-1277.5904 299.4259,-1279.9544 303.1543,-1285.8788\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4357081632 -->\n",
       "<g class=\"node\" id=\"node16\">\n",
       "<title>4357081632</title>\n",
       "<polygon fill=\"none\" points=\"0,-1241.5 0,-1277.5 211,-1277.5 211,-1241.5 0,-1241.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"105.5\" y=\"-1255.8\">track-Embedding-MF: Embedding</text>\n",
       "</g>\n",
       "<!-- 4394662600&#45;&gt;4357081632 -->\n",
       "<g class=\"edge\" id=\"edge15\">\n",
       "<title>4394662600-&gt;4357081632</title>\n",
       "<path d=\"M193.5786,-1314.4551C178.391,-1304.979 159.5937,-1293.2508 143.2699,-1283.0658\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"144.8309,-1279.9145 134.4941,-1277.5904 141.1255,-1285.8533 144.8309,-1279.9145\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4439404952 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>4439404952</title>\n",
       "<polygon fill=\"none\" points=\"637.5,-1314.5 637.5,-1350.5 765.5,-1350.5 765.5,-1314.5 637.5,-1314.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"701.5\" y=\"-1328.8\">Playlist: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4357007064 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>4357007064</title>\n",
       "<polygon fill=\"none\" points=\"708.5,-1241.5 708.5,-1277.5 940.5,-1277.5 940.5,-1241.5 708.5,-1241.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"824.5\" y=\"-1255.8\">playlist-Embedding-MLP: Embedding</text>\n",
       "</g>\n",
       "<!-- 4439404952&#45;&gt;4357007064 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>4439404952-&gt;4357007064</title>\n",
       "<path d=\"M731.9045,-1314.4551C748.0189,-1304.8912 767.9985,-1293.0334 785.2692,-1282.7833\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"787.2058,-1285.704 794.019,-1277.5904 783.6331,-1279.6843 787.2058,-1285.704\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112336498136 -->\n",
       "<g class=\"node\" id=\"node17\">\n",
       "<title>112336498136</title>\n",
       "<polygon fill=\"none\" points=\"466.5,-1241.5 466.5,-1277.5 690.5,-1277.5 690.5,-1241.5 466.5,-1241.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"578.5\" y=\"-1255.8\">playlist-Embedding-MF: Embedding</text>\n",
       "</g>\n",
       "<!-- 4439404952&#45;&gt;112336498136 -->\n",
       "<g class=\"edge\" id=\"edge16\">\n",
       "<title>4439404952-&gt;112336498136</title>\n",
       "<path d=\"M671.0955,-1314.4551C654.9811,-1304.8912 635.0015,-1293.0334 617.7308,-1282.7833\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"619.3669,-1279.6843 608.981,-1277.5904 615.7942,-1285.704 619.3669,-1279.6843\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4394663272 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>4394663272</title>\n",
       "<polygon fill=\"none\" points=\"261.5,-1168.5 261.5,-1204.5 437.5,-1204.5 437.5,-1168.5 261.5,-1168.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"349.5\" y=\"-1182.8\">Flatten_tracks-MLP: Flatten</text>\n",
       "</g>\n",
       "<!-- 4394662712&#45;&gt;4394663272 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>4394662712-&gt;4394663272</title>\n",
       "<path d=\"M341.2191,-1241.4551C342.4355,-1233.3828 343.8981,-1223.6764 345.2534,-1214.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"348.7449,-1215.0003 346.7741,-1204.5904 341.823,-1213.9572 348.7449,-1215.0003\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4635898768 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "<title>4635898768</title>\n",
       "<polygon fill=\"none\" points=\"708,-1168.5 708,-1204.5 897,-1204.5 897,-1168.5 708,-1168.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"802.5\" y=\"-1182.8\">Flatten_playlists-MLP: Flatten</text>\n",
       "</g>\n",
       "<!-- 4357007064&#45;&gt;4635898768 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>4357007064-&gt;4635898768</title>\n",
       "<path d=\"M819.0618,-1241.4551C816.6026,-1233.2951 813.6402,-1223.4652 810.9048,-1214.3887\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"814.1886,-1213.1551 807.9519,-1204.5904 807.4863,-1215.175 814.1886,-1213.1551\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4394663496 -->\n",
       "<g class=\"node\" id=\"node7\">\n",
       "<title>4394663496</title>\n",
       "<polygon fill=\"none\" points=\"303.5,-1095.5 303.5,-1131.5 431.5,-1131.5 431.5,-1095.5 303.5,-1095.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"367.5\" y=\"-1109.8\">dropout_1: Dropout</text>\n",
       "</g>\n",
       "<!-- 4394663272&#45;&gt;4394663496 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "<title>4394663272-&gt;4394663496</title>\n",
       "<path d=\"M353.9494,-1168.4551C355.9615,-1160.2951 358.3853,-1150.4652 360.6233,-1141.3887\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"364.0435,-1142.1375 363.0394,-1131.5904 357.247,-1140.4616 364.0435,-1142.1375\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4635899160 -->\n",
       "<g class=\"node\" id=\"node8\">\n",
       "<title>4635899160</title>\n",
       "<polygon fill=\"none\" points=\"693.5,-1095.5 693.5,-1131.5 821.5,-1131.5 821.5,-1095.5 693.5,-1095.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"757.5\" y=\"-1109.8\">dropout_3: Dropout</text>\n",
       "</g>\n",
       "<!-- 4635898768&#45;&gt;4635899160 -->\n",
       "<g class=\"edge\" id=\"edge6\">\n",
       "<title>4635898768-&gt;4635899160</title>\n",
       "<path d=\"M791.3764,-1168.4551C786.1299,-1159.9441 779.7636,-1149.6165 773.9724,-1140.2219\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"776.8785,-1138.2663 768.6516,-1131.5904 770.9197,-1141.9396 776.8785,-1138.2663\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112336498248 -->\n",
       "<g class=\"node\" id=\"node9\">\n",
       "<title>112336498248</title>\n",
       "<polygon fill=\"none\" points=\"451,-1022.5 451,-1058.5 584,-1058.5 584,-1022.5 451,-1022.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-1036.8\">Concat: Concatenate</text>\n",
       "</g>\n",
       "<!-- 4394663496&#45;&gt;112336498248 -->\n",
       "<g class=\"edge\" id=\"edge7\">\n",
       "<title>4394663496-&gt;112336498248</title>\n",
       "<path d=\"M404.5787,-1095.4551C424.6811,-1085.6719 449.7159,-1073.4883 471.1015,-1063.0806\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"472.8679,-1066.1135 480.328,-1058.5904 469.8047,-1059.8193 472.8679,-1066.1135\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4635899160&#45;&gt;112336498248 -->\n",
       "<g class=\"edge\" id=\"edge8\">\n",
       "<title>4635899160-&gt;112336498248</title>\n",
       "<path d=\"M698.1741,-1095.4551C664.4237,-1085.1893 621.9837,-1072.2804 586.7183,-1061.5539\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"587.5609,-1058.1519 576.9752,-1058.5904 585.5238,-1064.849 587.5609,-1058.1519\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112336934504 -->\n",
       "<g class=\"node\" id=\"node10\">\n",
       "<title>112336934504</title>\n",
       "<polygon fill=\"none\" points=\"453.5,-949.5 453.5,-985.5 581.5,-985.5 581.5,-949.5 453.5,-949.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-963.8\">dropout_5: Dropout</text>\n",
       "</g>\n",
       "<!-- 112336498248&#45;&gt;112336934504 -->\n",
       "<g class=\"edge\" id=\"edge9\">\n",
       "<title>112336498248-&gt;112336934504</title>\n",
       "<path d=\"M517.5,-1022.4551C517.5,-1014.3828 517.5,-1004.6764 517.5,-995.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-995.5903 517.5,-985.5904 514.0001,-995.5904 521.0001,-995.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112336934896 -->\n",
       "<g class=\"node\" id=\"node11\">\n",
       "<title>112336934896</title>\n",
       "<polygon fill=\"none\" points=\"443.5,-876.5 443.5,-912.5 591.5,-912.5 591.5,-876.5 443.5,-876.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-890.8\">FullyConnected: Dense</text>\n",
       "</g>\n",
       "<!-- 112336934504&#45;&gt;112336934896 -->\n",
       "<g class=\"edge\" id=\"edge10\">\n",
       "<title>112336934504-&gt;112336934896</title>\n",
       "<path d=\"M517.5,-949.4551C517.5,-941.3828 517.5,-931.6764 517.5,-922.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-922.5903 517.5,-912.5904 514.0001,-922.5904 521.0001,-922.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112336934224 -->\n",
       "<g class=\"node\" id=\"node12\">\n",
       "<title>112336934224</title>\n",
       "<polygon fill=\"none\" points=\"432.5,-803.5 432.5,-839.5 602.5,-839.5 602.5,-803.5 432.5,-803.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-817.8\">Batch: BatchNormalization</text>\n",
       "</g>\n",
       "<!-- 112336934896&#45;&gt;112336934224 -->\n",
       "<g class=\"edge\" id=\"edge11\">\n",
       "<title>112336934896-&gt;112336934224</title>\n",
       "<path d=\"M517.5,-876.4551C517.5,-868.3828 517.5,-858.6764 517.5,-849.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-849.5903 517.5,-839.5904 514.0001,-849.5904 521.0001,-849.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112337403184 -->\n",
       "<g class=\"node\" id=\"node13\">\n",
       "<title>112337403184</title>\n",
       "<polygon fill=\"none\" points=\"453,-730.5 453,-766.5 582,-766.5 582,-730.5 453,-730.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-744.8\">Dropout-1: Dropout</text>\n",
       "</g>\n",
       "<!-- 112336934224&#45;&gt;112337403184 -->\n",
       "<g class=\"edge\" id=\"edge12\">\n",
       "<title>112336934224-&gt;112337403184</title>\n",
       "<path d=\"M517.5,-803.4551C517.5,-795.3828 517.5,-785.6764 517.5,-776.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-776.5903 517.5,-766.5904 514.0001,-776.5904 521.0001,-776.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4439405008 -->\n",
       "<g class=\"node\" id=\"node14\">\n",
       "<title>4439405008</title>\n",
       "<polygon fill=\"none\" points=\"437.5,-657.5 437.5,-693.5 597.5,-693.5 597.5,-657.5 437.5,-657.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-671.8\">FullyConnected-1: Dense</text>\n",
       "</g>\n",
       "<!-- 112337403184&#45;&gt;4439405008 -->\n",
       "<g class=\"edge\" id=\"edge13\">\n",
       "<title>112337403184-&gt;4439405008</title>\n",
       "<path d=\"M517.5,-730.4551C517.5,-722.3828 517.5,-712.6764 517.5,-703.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-703.5903 517.5,-693.5904 514.0001,-703.5904 521.0001,-703.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112337276096 -->\n",
       "<g class=\"node\" id=\"node15\">\n",
       "<title>112337276096</title>\n",
       "<polygon fill=\"none\" points=\"426.5,-584.5 426.5,-620.5 608.5,-620.5 608.5,-584.5 426.5,-584.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-598.8\">Batch-2: BatchNormalization</text>\n",
       "</g>\n",
       "<!-- 4439405008&#45;&gt;112337276096 -->\n",
       "<g class=\"edge\" id=\"edge14\">\n",
       "<title>4439405008-&gt;112337276096</title>\n",
       "<path d=\"M517.5,-657.4551C517.5,-649.3828 517.5,-639.6764 517.5,-630.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-630.5903 517.5,-620.5904 514.0001,-630.5904 521.0001,-630.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112337853128 -->\n",
       "<g class=\"node\" id=\"node18\">\n",
       "<title>112337853128</title>\n",
       "<polygon fill=\"none\" points=\"451,-511.5 451,-547.5 580,-547.5 580,-511.5 451,-511.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"515.5\" y=\"-525.8\">Dropout-2: Dropout</text>\n",
       "</g>\n",
       "<!-- 112337276096&#45;&gt;112337853128 -->\n",
       "<g class=\"edge\" id=\"edge17\">\n",
       "<title>112337276096-&gt;112337853128</title>\n",
       "<path d=\"M517.0056,-584.4551C516.7845,-576.3828 516.5185,-566.6764 516.2721,-557.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"519.7683,-557.4907 515.9956,-547.5904 512.7709,-557.6825 519.7683,-557.4907\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4394663664 -->\n",
       "<g class=\"node\" id=\"node19\">\n",
       "<title>4394663664</title>\n",
       "<polygon fill=\"none\" points=\"55,-1168.5 55,-1204.5 222,-1204.5 222,-1168.5 55,-1168.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"138.5\" y=\"-1182.8\">Flatten_tracks-MF: Flatten</text>\n",
       "</g>\n",
       "<!-- 4357081632&#45;&gt;4394663664 -->\n",
       "<g class=\"edge\" id=\"edge18\">\n",
       "<title>4357081632-&gt;4394663664</title>\n",
       "<path d=\"M113.6573,-1241.4551C117.4254,-1233.1196 121.9812,-1223.0416 126.157,-1213.8042\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"129.3922,-1215.1443 130.3222,-1204.5904 123.0136,-1212.2608 129.3922,-1215.1443\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112336498528 -->\n",
       "<g class=\"node\" id=\"node20\">\n",
       "<title>112336498528</title>\n",
       "<polygon fill=\"none\" points=\"477.5,-1168.5 477.5,-1204.5 657.5,-1204.5 657.5,-1168.5 477.5,-1168.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"567.5\" y=\"-1182.8\">Flatten_playlists-MF: Flatten</text>\n",
       "</g>\n",
       "<!-- 112336498136&#45;&gt;112336498528 -->\n",
       "<g class=\"edge\" id=\"edge19\">\n",
       "<title>112336498136-&gt;112336498528</title>\n",
       "<path d=\"M575.7809,-1241.4551C574.5645,-1233.3828 573.1019,-1223.6764 571.7466,-1214.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"575.177,-1213.9572 570.2259,-1204.5904 568.2551,-1215.0003 575.177,-1213.9572\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112338090248 -->\n",
       "<g class=\"node\" id=\"node21\">\n",
       "<title>112338090248</title>\n",
       "<polygon fill=\"none\" points=\"432.5,-438.5 432.5,-474.5 592.5,-474.5 592.5,-438.5 432.5,-438.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"512.5\" y=\"-452.8\">FullyConnected-2: Dense</text>\n",
       "</g>\n",
       "<!-- 112337853128&#45;&gt;112338090248 -->\n",
       "<g class=\"edge\" id=\"edge20\">\n",
       "<title>112337853128-&gt;112338090248</title>\n",
       "<path d=\"M514.7584,-511.4551C514.4267,-503.3828 514.0278,-493.6764 513.6582,-484.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"517.1512,-484.4382 513.2434,-474.5904 510.1571,-484.7257 517.1512,-484.4382\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4394663328 -->\n",
       "<g class=\"node\" id=\"node22\">\n",
       "<title>4394663328</title>\n",
       "<polygon fill=\"none\" points=\"125.5,-1095.5 125.5,-1131.5 253.5,-1131.5 253.5,-1095.5 125.5,-1095.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"189.5\" y=\"-1109.8\">dropout_2: Dropout</text>\n",
       "</g>\n",
       "<!-- 4394663664&#45;&gt;4394663328 -->\n",
       "<g class=\"edge\" id=\"edge21\">\n",
       "<title>4394663664-&gt;4394663328</title>\n",
       "<path d=\"M151.1067,-1168.4551C157.114,-1159.8564 164.4168,-1149.4034 171.0341,-1139.9316\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"174.0036,-1141.7925 176.8615,-1131.5904 168.2652,-1137.7835 174.0036,-1141.7925\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112336497800 -->\n",
       "<g class=\"node\" id=\"node23\">\n",
       "<title>112336497800</title>\n",
       "<polygon fill=\"none\" points=\"485.5,-1095.5 485.5,-1131.5 613.5,-1131.5 613.5,-1095.5 485.5,-1095.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"549.5\" y=\"-1109.8\">dropout_4: Dropout</text>\n",
       "</g>\n",
       "<!-- 112336498528&#45;&gt;112336497800 -->\n",
       "<g class=\"edge\" id=\"edge22\">\n",
       "<title>112336498528-&gt;112336497800</title>\n",
       "<path d=\"M563.0506,-1168.4551C561.0385,-1160.2951 558.6147,-1150.4652 556.3767,-1141.3887\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"559.753,-1140.4616 553.9606,-1131.5904 552.9565,-1142.1375 559.753,-1140.4616\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112337853016 -->\n",
       "<g class=\"node\" id=\"node24\">\n",
       "<title>112337853016</title>\n",
       "<polygon fill=\"none\" points=\"431.5,-365.5 431.5,-401.5 591.5,-401.5 591.5,-365.5 431.5,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"511.5\" y=\"-379.8\">FullyConnected-3: Dense</text>\n",
       "</g>\n",
       "<!-- 112338090248&#45;&gt;112337853016 -->\n",
       "<g class=\"edge\" id=\"edge23\">\n",
       "<title>112338090248-&gt;112337853016</title>\n",
       "<path d=\"M512.2528,-438.4551C512.1422,-430.3828 512.0093,-420.6764 511.8861,-411.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"515.3846,-411.5414 511.7478,-401.5904 508.3852,-411.6374 515.3846,-411.5414\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112339506120 -->\n",
       "<g class=\"node\" id=\"node25\">\n",
       "<title>112339506120</title>\n",
       "<polygon fill=\"none\" points=\"365.5,-1022.5 365.5,-1058.5 431.5,-1058.5 431.5,-1022.5 365.5,-1022.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"398.5\" y=\"-1036.8\">Dot: Dot</text>\n",
       "</g>\n",
       "<!-- 4394663328&#45;&gt;112339506120 -->\n",
       "<g class=\"edge\" id=\"edge24\">\n",
       "<title>4394663328-&gt;112339506120</title>\n",
       "<path d=\"M241.1629,-1095.4551C276.3279,-1083.1726 322.3253,-1067.1065 355.4935,-1055.5214\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"357.0625,-1058.6808 365.3491,-1052.079 354.7542,-1052.0723 357.0625,-1058.6808\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112336497800&#45;&gt;112339506120 -->\n",
       "<g class=\"edge\" id=\"edge25\">\n",
       "<title>112336497800-&gt;112339506120</title>\n",
       "<path d=\"M512.1741,-1095.4551C490.5479,-1085 463.2508,-1071.8034 440.8269,-1060.9627\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"442.0722,-1057.6772 431.5457,-1056.4757 439.0254,-1063.9794 442.0722,-1057.6772\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112340224712 -->\n",
       "<g class=\"node\" id=\"node26\">\n",
       "<title>112340224712</title>\n",
       "<polygon fill=\"none\" points=\"436.5,-292.5 436.5,-328.5 554.5,-328.5 554.5,-292.5 436.5,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"495.5\" y=\"-306.8\">Activation: Dense</text>\n",
       "</g>\n",
       "<!-- 112337853016&#45;&gt;112340224712 -->\n",
       "<g class=\"edge\" id=\"edge26\">\n",
       "<title>112337853016-&gt;112340224712</title>\n",
       "<path d=\"M507.5449,-365.4551C505.7565,-357.2951 503.602,-347.4652 501.6126,-338.3887\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"505.0249,-337.6091 499.465,-328.5904 498.1872,-339.1079 505.0249,-337.6091\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112340365384 -->\n",
       "<g class=\"node\" id=\"node27\">\n",
       "<title>112340365384</title>\n",
       "<polygon fill=\"none\" points=\"346,-219.5 346,-255.5 537,-255.5 537,-219.5 346,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"441.5\" y=\"-233.8\">Concat-MF-MLP: Concatenate</text>\n",
       "</g>\n",
       "<!-- 112339506120&#45;&gt;112340365384 -->\n",
       "<g class=\"edge\" id=\"edge27\">\n",
       "<title>112339506120-&gt;112340365384</title>\n",
       "<path d=\"M398.5,-1022.062C398.5,-994.3663 398.5,-940.3785 398.5,-894.5 398.5,-894.5 398.5,-894.5 398.5,-383.5 398.5,-340.9537 415.5108,-294.1703 428.1339,-265.3111\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"431.4632,-266.4421 432.3808,-255.8871 425.0812,-263.5661 431.4632,-266.4421\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112340224712&#45;&gt;112340365384 -->\n",
       "<g class=\"edge\" id=\"edge28\">\n",
       "<title>112340224712-&gt;112340365384</title>\n",
       "<path d=\"M482.1517,-292.4551C475.7261,-283.7686 467.9007,-273.1898 460.8379,-263.642\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"463.6428,-261.5484 454.8819,-255.5904 458.0151,-265.7113 463.6428,-261.5484\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112340483320 -->\n",
       "<g class=\"node\" id=\"node28\">\n",
       "<title>112340483320</title>\n",
       "<polygon fill=\"none\" points=\"357,-146.5 357,-182.5 526,-182.5 526,-146.5 357,-146.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"441.5\" y=\"-160.8\">Combine-MF-MLP: Dense</text>\n",
       "</g>\n",
       "<!-- 112340365384&#45;&gt;112340483320 -->\n",
       "<g class=\"edge\" id=\"edge29\">\n",
       "<title>112340365384-&gt;112340483320</title>\n",
       "<path d=\"M441.5,-219.4551C441.5,-211.3828 441.5,-201.6764 441.5,-192.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"445.0001,-192.5903 441.5,-182.5904 438.0001,-192.5904 445.0001,-192.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112340481080 -->\n",
       "<g class=\"node\" id=\"node29\">\n",
       "<title>112340481080</title>\n",
       "<polygon fill=\"none\" points=\"361.5,-73.5 361.5,-109.5 521.5,-109.5 521.5,-73.5 361.5,-73.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"441.5\" y=\"-87.8\">FullyConnected-4: Dense</text>\n",
       "</g>\n",
       "<!-- 112340483320&#45;&gt;112340481080 -->\n",
       "<g class=\"edge\" id=\"edge30\">\n",
       "<title>112340483320-&gt;112340481080</title>\n",
       "<path d=\"M441.5,-146.4551C441.5,-138.3828 441.5,-128.6764 441.5,-119.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"445.0001,-119.5903 441.5,-109.5904 438.0001,-119.5904 445.0001,-119.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112340603792 -->\n",
       "<g class=\"node\" id=\"node30\">\n",
       "<title>112340603792</title>\n",
       "<polygon fill=\"none\" points=\"383.5,-.5 383.5,-36.5 499.5,-36.5 499.5,-.5 383.5,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"441.5\" y=\"-14.8\">Prediction: Dense</text>\n",
       "</g>\n",
       "<!-- 112340481080&#45;&gt;112340603792 -->\n",
       "<g class=\"edge\" id=\"edge31\">\n",
       "<title>112340481080-&gt;112340603792</title>\n",
       "<path d=\"M441.5,-73.4551C441.5,-65.3828 441.5,-55.6764 441.5,-46.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"445.0001,-46.5903 441.5,-36.5904 438.0001,-46.5904 445.0001,-46.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# visualize model\n",
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "SVG(model_to_dot(model, show_shapes=False, show_layer_names=True, rankdir='HB').create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Track (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Playlist (InputLayer)           (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "track-Embedding-MLP (Embedding) (None, 1, 10)        744510      Track[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "playlist-Embedding-MLP (Embeddi (None, 1, 10)        100010      Playlist[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "Flatten_tracks-MLP (Flatten)    (None, 10)           0           track-Embedding-MLP[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "Flatten_playlists-MLP (Flatten) (None, 10)           0           playlist-Embedding-MLP[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 10)           0           Flatten_tracks-MLP[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 10)           0           Flatten_playlists-MLP[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Concat (Concatenate)            (None, 20)           0           dropout_1[0][0]                  \n",
      "                                                                 dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, 20)           0           Concat[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected (Dense)          (None, 200)          4200        dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "Batch (BatchNormalization)      (None, 200)          800         FullyConnected[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "Dropout-1 (Dropout)             (None, 200)          0           Batch[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-1 (Dense)        (None, 100)          20100       Dropout-1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "Batch-2 (BatchNormalization)    (None, 100)          400         FullyConnected-1[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "track-Embedding-MF (Embedding)  (None, 1, 50)        3722550     Track[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "playlist-Embedding-MF (Embeddin (None, 1, 50)        500050      Playlist[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "Dropout-2 (Dropout)             (None, 100)          0           Batch-2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "Flatten_tracks-MF (Flatten)     (None, 50)           0           track-Embedding-MF[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "Flatten_playlists-MF (Flatten)  (None, 50)           0           playlist-Embedding-MF[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-2 (Dense)        (None, 50)           5050        Dropout-2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 50)           0           Flatten_tracks-MF[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 50)           0           Flatten_playlists-MF[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-3 (Dense)        (None, 20)           1020        FullyConnected-2[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "Dot (Dot)                       (None, 1)            0           dropout_2[0][0]                  \n",
      "                                                                 dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "Activation (Dense)              (None, 1)            21          FullyConnected-3[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "Concat-MF-MLP (Concatenate)     (None, 2)            0           Dot[0][0]                        \n",
      "                                                                 Activation[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "Combine-MF-MLP (Dense)          (None, 100)          300         Concat-MF-MLP[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-4 (Dense)        (None, 100)          10100       Combine-MF-MLP[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "Prediction (Dense)              (None, 2)            202         FullyConnected-4[0][0]           \n",
      "==================================================================================================\n",
      "Total params: 5,109,313\n",
      "Trainable params: 5,108,713\n",
      "Non-trainable params: 600\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# print out model summary\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spliting data\n",
    "***\n",
    "We split 5% of the data from the whole dataset, and distributed the 5% equally into validation and test set. From the summary statistics, we can see that the matrix is really sparse: only ~0.1% of pair had class 1 labels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of all sample = 744500000\n",
      "proportion of ones in all samples = 0.0013834989926124917\n",
      "# of split samples = 51086\n",
      "proportion of ones in split sample = 0.001372357286769644\n"
     ]
    }
   ],
   "source": [
    "# split data for training vs. validation/testing\n",
    "data_size = n_playlists*n_tracks\n",
    "\n",
    "# split a portion of pairs for validation and test\n",
    "split_ratio = 0.05\n",
    "split_num = int(data_size*split_ratio)\n",
    "\n",
    "split_tracks = np.random.choice(n_tracks, split_num).reshape(-1,)\n",
    "split_playlists = np.random.choice(n_playlists, split_num).reshape(-1,)\n",
    "split_values = np.array(sps_acc[split_tracks, split_playlists]).reshape(-1,)\n",
    "\n",
    "split_pair = [(tr, pl) for tr,pl in zip(split_tracks,split_playlists)]\n",
    "\n",
    "print('# of all sample =', data_size)\n",
    "print('proportion of ones in all samples =', sps_acc.count_nonzero()/data_size)\n",
    "print('# of split samples =',int(np.sum(split_values)))\n",
    "print('proportion of ones in split sample =', np.mean(split_values))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split the saved pairs into validation and test data\n",
    "test_num = int(split_num/2)\n",
    "\n",
    "test_tracks = split_tracks[:test_num]\n",
    "test_playlists = split_playlists[:test_num]\n",
    "test_values = split_values[:test_num]\n",
    "test_target = keras.utils.to_categorical(test_values, num_classes=2)\n",
    "\n",
    "val_tracks = split_tracks[test_num+1:]\n",
    "val_playlists = split_playlists[test_num+1:]\n",
    "val_values = split_values[test_num+1:]\n",
    "val_target = keras.utils.to_categorical(val_values, num_classes=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A trick here: We masked the splited data by setting the value to 2, and later excluded them during training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shihyitseng/anaconda3/lib/python3.6/site-packages/scipy/sparse/compressed.py:746: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient.\n",
      "  SparseEfficiencyWarning)\n"
     ]
    }
   ],
   "source": [
    "# put split sample values to 2\n",
    "sps_two = sps_acc\n",
    "sps_two[split_tracks,split_playlists] = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define training data generator\n",
    "***\n",
    "One important step we took for training was to artificially ***balance the class labels***. Since the class 1 pairs were extremely sparse, a network trained on the original data can end up predicting everything to 0. Actuaslly that was what we got when we started with the naive way. In the current version we presented here, the training generator yielded augemented data with equalized class number. This would force the network to pay more attention on the class 1 samples, and bias the network to have a higher prior to class 1 than the real prior (which was around 0.1%). But the \"high\" false positive rate is actually what we want! We need to make recommendation on extending the songs in a playlists, and the \"false positive\" samples would be the candidates for recommendation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define training sample generator (with balanced classes)\n",
    "\n",
    "# build one-hot encoder\n",
    "exampleCase = [0., 1.]\n",
    "ohe = OneHotEncoder(categories='auto').fit(np.array(exampleCase).reshape((len(exampleCase), 1)))\n",
    "\n",
    "def trainGenerator(sps_two, ohe, batch_size=10000, balance_ratio=0.5):\n",
    "    while True:\n",
    "        n_tracks = sps_two.shape[0]\n",
    "        n_playlists = sps_two.shape[1]\n",
    "\n",
    "        # find indices for non-zero values\n",
    "        index = scipy.sparse.find(sps_two)\n",
    "\n",
    "        # find number of ones and zeros in final yield\n",
    "        num_ones = np.ceil(batch_size*balance_ratio).astype(int)\n",
    "        num_zeros = batch_size - num_ones\n",
    "\n",
    "        # extract tracks, playlists and labels\n",
    "        tr = index[0] # track\n",
    "        pl = index[1] # playlist\n",
    "        la = index[2] # label\n",
    "\n",
    "        # find where label == 1\n",
    "        is_one = la == 1\n",
    "        tr = tr[is_one]\n",
    "        pl = pl[is_one]\n",
    "\n",
    "        # select class one samples\n",
    "        num = tr.shape[0]\n",
    "        one_ind = np.random.choice(num, num_ones)\n",
    "        one_tracks = tr[one_ind]\n",
    "        one_playlists = pl[one_ind]\n",
    "        one_values = np.ones((num_ones,1))\n",
    "\n",
    "        # over-sample 2 times of target number for class zero randomly (and later exclude non-zero samples)\n",
    "        rand_tracks = np.random.choice(n_tracks, num_zeros*2)\n",
    "        rand_playlists = np.random.choice(n_playlists, num_zeros*2)\n",
    "        rand_values = np.array(sps_two[rand_tracks, rand_playlists]).reshape(-1,)\n",
    "\n",
    "        # check to find class zero samples\n",
    "        is_zero = rand_values == 0\n",
    "        zero_tracks = rand_tracks[is_zero]\n",
    "        zero_playlists = rand_playlists[is_zero]\n",
    "        \n",
    "        # grab zero entries with the target number\n",
    "        zero_tracks = zero_tracks[:num_zeros]\n",
    "        zero_playlists = zero_playlists[:num_zeros]\n",
    "        zero_values = np.zeros((num_zeros,1))\n",
    "\n",
    "        # concatenate class one and zeros\n",
    "        sub_tracks = np.concatenate((one_tracks, zero_tracks))\n",
    "        sub_playlists = np.concatenate((one_playlists, zero_playlists))\n",
    "        values = np.concatenate((one_values, zero_values),axis = 0)\n",
    "        target = ohe.transform(values).toarray()\n",
    "        \n",
    "        yield [sub_playlists, sub_tracks], target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation generator was defined similarly, generating samples from the validation set with equalized class number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_ones_val = np.array(val_values).reshape(-1,)==1\n",
    "val_tracks_ones = val_tracks[is_ones_val]\n",
    "val_playlists_ones = val_playlists[is_ones_val]\n",
    "\n",
    "val_tracks_zeros = val_tracks[~is_ones_val]\n",
    "val_playlists_zeros = val_playlists[~is_ones_val]\n",
    "\n",
    "# define validation generator (with balanced classes)\n",
    "def valGenerator_balance(val_tracks_ones, val_playlists_ones, val_tracks_zeros, val_playlists_zeros, \n",
    "                         ohe, val_batch_size=10000, balance_ratio = 0.5):\n",
    "    while True:\n",
    "        num_ones = int(val_batch_size*balance_ratio)\n",
    "        num_zeros = val_batch_size - num_ones\n",
    "        \n",
    "        num_ones_total = val_tracks_ones.shape[0]\n",
    "        num_zeros_total = val_tracks_zeros.shape[0]       \n",
    "        \n",
    "        ind_ones = np.random.choice(num_ones_total, num_ones)\n",
    "        sub_val_tracks_ones = val_tracks_ones[ind_ones]\n",
    "        sub_val_playlists_ones = val_playlists_ones[ind_ones]\n",
    "        sub_val_values_ones = np.ones((num_ones,))\n",
    "        \n",
    "        ind_zeros = np.random.choice(num_zeros_total, num_zeros)\n",
    "        sub_val_tracks_zeros = val_tracks_zeros[ind_zeros]\n",
    "        sub_val_playlists_zeros = val_playlists_zeros[ind_zeros]\n",
    "        sub_val_values_zeros = np.zeros((num_zeros,))\n",
    "        \n",
    "        sub_val_tracks = np.concatenate((sub_val_tracks_ones,sub_val_tracks_zeros))\n",
    "        sub_val_playlists = np.concatenate((sub_val_playlists_ones,sub_val_playlists_zeros))\n",
    "        sub_val_values = np.concatenate((sub_val_values_ones,sub_val_values_zeros))\n",
    "        sub_val_target = ohe.transform(sub_val_values.reshape(-1,1)).toarray()\n",
    "        \n",
    "        yield [sub_val_playlists, sub_val_tracks], sub_val_target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model training\n",
    "***\n",
    "We set up checkpoints during the training process, saving models every 10 epochs. Later we can look at the training history and pick a model based on ***validation loss***, since the model can overfit during the extended training process (total 500 epochs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set checkpoints\n",
    "filepath = \"models_10000/model.{epoch:02d}-{val_loss:.2f}-{val_acc:.2f}.h5\"\n",
    "\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='val_loss', \n",
    "                             verbose=2, save_best_only=False, mode='auto', period=10)\n",
    "callbacks_list = [checkpoint]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit model with validation data\n",
    "history = model.fit_generator(trainGenerator(sps_two, ohe, batch_size=20000, balance_ratio=0.5), \n",
    "                              epochs=500, steps_per_epoch = 1, verbose=2, workers=1, use_multiprocessing=False,\n",
    "                              validation_data=valGenerator_balance(val_tracks_ones, val_playlists_ones, \n",
    "                                                                   val_tracks_zeros, val_playlists_zeros, \n",
    "                                                                   ohe, val_batch_size=10000, balance_ratio = 0.5),\n",
    "                              validation_steps=1,\n",
    "                              callbacks=callbacks_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save final model, training history, test/val data, and training matrix\n",
    "model.save('final_model_10000_new_181210.h5')\n",
    "\n",
    "with open('model_history_10000_new_181210.pkl', 'wb') as f0:\n",
    "    pickle.dump(history, f0)\n",
    "\n",
    "with open('test_tracks_10000_new_181210.pkl', 'wb') as f1:\n",
    "    pickle.dump(test_tracks, f1)\n",
    "with open('test_playlists_10000_new_181210.pkl', 'wb') as f2:\n",
    "    pickle.dump(test_playlists, f2)\n",
    "with open('test_values_10000_new_181210.pkl', 'wb') as f3:\n",
    "    pickle.dump(test_values, f3)\n",
    "    \n",
    "with open('val_tracks_10000_new_181210.pkl', 'wb') as f4:\n",
    "    pickle.dump(val_tracks, f4)\n",
    "with open('val_playlists_10000_new_181210.pkl', 'wb') as f5:\n",
    "    pickle.dump(val_playlists, f5)\n",
    "with open('val_values_10000_new_181210.pkl', 'wb') as f6:\n",
    "    pickle.dump(val_values, f6)\n",
    "    \n",
    "with open('sps_two_10000_new_181210.pkl', 'wb') as f7:\n",
    "    pickle.dump(sps_two, f7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training history\n",
    "***\n",
    "Plotted below is the training/validation loss during the training process. Both the training/validation loss dropped rapidly during the first 100 epochs, and and started to slow down between epoch 100 to 200. The training and validation loss started to diverge after epoch 200, as the training loss kept going down but the validation loss slowly increased. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(0,0.5,'Loss'), Text(0.5,0,'Epoch'), Text(0.5,1,'Loss')]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAGDCAYAAAAyM4nNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3XlcVOX+B/DPA4K444YbCpi7KIq4ZW6ZplZqaqnl/WVWZtb13mzT1mvLbb2tWlmW7ZnZZlq2a1ZuWIriBhoqLoi4sgs8vz++PJ4zw6CgDDMMn/frxeucOXPm8AzLfM+zfR+ltQYRERFVfH6eLgARERGVDQZ1IiIiH8GgTkRE5CMY1ImIiHwEgzoREZGPYFAnIiLyEQzqREREPoJBnaiSUkolKaUu83Q5iKjsMKgTERH5CAZ1InKglLpFKZWolDqqlFqilGpaeFwppV5QSh1WSp1QSsUppSILnxuulNqqlDqllNqvlLrbs++CqHJiUCeiM5RSlwJ4EsC1AJoA2ANgYeHTQwD0A9AGQDCAcQDSCp97C8CtWutaACIB/FyOxSaiQlU8XQAi8irXA3hba/0nACilZgE4ppQKB3AaQC0A7QCs01pvs73uNIAOSqlNWutjAI6Va6mJCABr6kTkqCmkdg4A0FqnQ2rjzbTWPwOYA2AugBSl1BtKqdqFp44BMBzAHqXUSqVU73IuNxGBQZ2IHB0AEGYeKKVqAKgPYD8AaK1f1lp3A9AR0gx/T+Hx9VrrkQBCAHwJYFE5l5uIwKBOVNkFKKWCzBckGN+olOqilKoK4L8A1mqtk5RS3ZVSPZVSAQAyAGQDyFdKBSqlrldK1dFanwZwEkC+x94RUSXGoE5UuX0DIMv21RfAQwA+A3AQwEUAxheeWxvAm5D+8j2QZvnnCp/7B4AkpdRJAFMBTCyn8hORjdJae7oMREREVAZYUyciIvIRDOpEREQ+gkGdiIjIRzCoExER+QgGdSIiIh9R4dLENmjQQIeHh3u6GEREROViw4YNR7TWDUtyboUL6uHh4YiNjfV0MYiIiMqFUmrPuc8SbH4nIiLyEQzqREREPoJBnYiIyEdUuD51IiLyvNOnTyM5ORnZ2dmeLorPCAoKQmhoKAICAs77GgzqRERUasnJyahVqxbCw8OhlPJ0cSo8rTXS0tKQnJyMiIiI876OW5vflVJDlVI7lFKJSqmZLp5/QSm1sfBrp1LquDvLQ0REZSM7Oxv169dnQC8jSinUr1//gls+3FZTV0r5A5gLYDCAZADrlVJLtNZbzTla6ztt5/8TQFd3lYeIiMoWA3rZKoufpztr6j0AJGqtd2utcwEsBDDyLOdPAPCxG8tDREQ+Ii0tDV26dEGXLl3QuHFjNGvW7Mzj3NzcEl3jxhtvxI4dO856zty5c/Hhhx+WRZHLhTv71JsB2Gd7nAygp6sTlVJhACIA/OzG8hARkY+oX78+Nm7cCAD4z3/+g5o1a+Luu+92OEdrDa01/Pxc118XLFhwzu9z++23X3hhy5E7a+qu2hF0MeeOB7BYa53v8kJKTVFKxSqlYlNTU8usgERE5FsSExMRGRmJqVOnIjo6GgcPHsSUKVMQExODjh074tFHHz1z7iWXXIKNGzciLy8PwcHBmDlzJqKiotC7d28cPnwYAPDggw/ixRdfPHP+zJkz0aNHD7Rt2xZ//PEHACAjIwNjxoxBVFQUJkyYgJiYmDM3HOXNnTX1ZADNbY9DARwo5tzxAIq9HdJavwHgDQCIiYkp7saAiIg84N//Bso6hnXpAhTG0lLbunUrFixYgNdffx0A8NRTT6FevXrIy8vDwIEDMXbsWHTo0MHhNSdOnED//v3x1FNPYcaMGXj77bcxc2aR8d3QWmPdunVYsmQJHn30USxfvhyvvPIKGjdujM8++wybNm1CdHT0+RW8DLizpr4eQGulVIRSKhASuJc4n6SUagugLoDVbiyLS/v/zsVPX5xEyq50pO3NwPEDmTh5OBsZR3OQdfI0cjPzkJeTD13A+wgioorioosuQvfu3c88/vjjjxEdHY3o6Ghs27YNW7duLfKaatWqYdiwYQCAbt26ISkpyeW1R48eXeSc3377DePHjwcAREVFoWPHjmX4bkrHbTV1rXWeUuoOAN8B8AfwttY6Xin1KIBYrbUJ8BMALNRal3vk3PHg+xj00c2lek0+/KChoKFQULgPANkIQoaqhSCVjf212mNd23+gw2MTUD2kJtq1A4KC3PEOiIg873xr1O5So0aNM/sJCQl46aWXsG7dOgQHB2PixIkup40FBgae2ff390deXp7La1etWrXIOR4IX8Vya/IZrfU3AL5xOvaw0+P/uLMMZ9Nreg/srPkcjqRqFOQXAAUaukBDFxRI7Ty/cKs1dH6BbAs0UCD7KCgofA9AYG46Ak5nIDs/EM33/Y6b103B3ssfw7/wEn6vcTn8albH/PnA0KFAFab8ISIqFydPnkStWrVQu3ZtHDx4EN999x2GDh1apt/jkksuwaJFi9C3b19s3rzZZUtAeanU4aV6z05o07MT2pT1hbXG0cU/o8kNV+OLrNE45N8accfaYvdVzdAp/EW8+0kQevQo629KRETOoqOj0aFDB0RGRqJly5bo06dPmX+Pf/7zn/i///s/dO7cGdHR0YiMjESdOnXK/PuUhPKmZoOSiImJ0RVmPfWkJGDFCuD555FfoOAfH4ePq9+Ep1vPx19/AczbQEQV1bZt29C+fXtPF8Mr5OXlIS8vD0FBQUhISMCQIUOQkJCAKufRLOvq56qU2qC1jinJ67lKmzuFhwOTJgFxcfDfsgmYPh3jct5FyqaDePHFM633RERUgaWnp6NPnz6IiorCmDFjMG/evPMK6GWBQb083XEH/PLz8FyreZgxA2jXDjhHMiMiIvJywcHB2LBhAzZt2oS4uDgMGTLEY2VhUC9PrVsDI0bg+sTZ2N1vEvTRY7jySqCYQZZERESlwqBe3mbPBgBE/PouVra+GYmJwOLFHi4TERH5BAb18talC7BvH3D//Wi65nMMbLkHr7zi6UIREZEvYFD3hNBQYNw4AMDdMb/gjz/Yt05ERBeOQd1TIiOBBg0wUP8Cf3+gBIsFERFRoQEDBuC7775zOPbiiy9i2rRpxb6mZs2aAIADBw5g7NixxV73XNOmX3zxRWRmZp55PHz4cBw/frykRXcrBnVP8fMD+vdHtXUrMXw4MG8ecPKkpwtFRFQxTJgwAQsXLnQ4tnDhQkyYMOGcr23atCkWX8BgJueg/s033yA4OPi8r1eWGNQ96eKLgT17cPvogzh+HAgJAdLSPF0oIiLvN3bsWCxduhQ5OTkAgKSkJBw4cABdunTBoEGDEB0djU6dOuGrr74q8tqkpCRERkYCALKysjB+/Hh07twZ48aNQ1ZW1pnzbrvttjNLtj7yyCMAgJdffhkHDhzAwIEDMXDgQABAeHg4jhw5AgB4/vnnERkZicjIyDNLtiYlJaF9+/a45ZZb0LFjRwwZMsTh+5SlSp0m1uN69wYAXF5nDZ555mrcey+wfDlw/fUeLhcRUWl4YO3V+vXro0ePHli+fDlGjhyJhQsXYty4cahWrRq++OIL1K5dG0eOHEGvXr0wYsQIqGJSeL722muoXr064uLiEBcX57Bs6hNPPIF69eohPz8fgwYNQlxcHKZPn47nn38ev/zyCxo0aOBwrQ0bNmDBggVYu3YttNbo2bMn+vfvj7p16yIhIQEff/wx3nzzTVx77bX47LPPMHHixLL5Wdmwpu5J5o9n9Gjc/fNwhDfMwNKlni0SEVFFYW+CN03vWmvcf//96Ny5My677DLs378fKSkpxV7j119/PRNcO3fujM6dO595btGiRYiOjkbXrl0RHx9/zoVafvvtN1x99dWoUaMGatasidGjR2PVqlUAgIiICHTp0gXA2Zd2vVCsqXtS1arAvfcC8fFQy5ZhZv8luGfZBJw8CdSu7enCERGVkIfWXh01ahRmzJiBP//8E1lZWYiOjsY777yD1NRUbNiwAQEBAQgPD3e51Kqdq1r833//jeeeew7r169H3bp1MWnSpHNe52xrqZglWwFZttVdze+sqXva008Dn38OBAbiqhYbceoU8M47ni4UEZH3q1mzJgYMGIDJkyefGSB34sQJhISEICAgAL/88gv27Nlz1mv069cPH374IQBgy5YtiIuLAyBLttaoUQN16tRBSkoKvv322zOvqVWrFk6dOuXyWl9++SUyMzORkZGBL774An379i2rt1siDOreIDAQ6NgRTQ/9hd69gZdfBvLzPV0oIiLvN2HCBGzatAnjx48HAFx//fWIjY1FTEwMPvzwQ7Rr1+6sr7/tttuQnp6Ozp0745lnnkGPwnWxo6Ki0LVrV3Ts2BGTJ092WLJ1ypQpGDZs2JmBckZ0dDQmTZqEHj16oGfPnrj55pvRtWvXMn7HZ8elV73F5MnA0qVY9EoKxo1X+PJLYORITxeKiMg1Lr3qHlx61VfExACpqRjdNh5NmrAJnoiISo9B3Vtccw1QtSqqDLkU9/T5A99+C7josiEiIioWg7q3aNgQ+Mc/gNRU3Lr+JuTkAO+95+lCERFRRcKg7k1efRV48EFU37Md1/VIxMMPAydOeLpQRESuVbQxWd6uLH6eDOreJCAAuOkmAMDjPZbg6FHg6689XCYiIheCgoKQlpbGwF5GtNZIS0tDUFDQBV2HyWe8TXg40Lw5wo/+iWbNgM8+A9yQSZCI6IKEhoYiOTkZqampni6KzwgKCkJoaOgFXYNB3Rt17Ai1NR5XXw3Mnw+kpwOFKwYSEXmFgIAAREREeLoY5ITN796oQwdg+3aMGZWP7GzAlsiIiIioWAzq3qhjRyA7G/1nX4pptd7Hq69yrXUiIjo3BnVvVLjOr1r1K15Jn4R1K7Pw0EMeLhMREXk9BnVv1L078PHHwJw58NMFmNB1O9av93ShiIjI2zGoeyOlgPHjgcLFAvrWi8eWLQBnjhAR0dkwqHuz1q2BgAB09tuCU6eAffs8XSAiIvJmDOreLCAAaNsWYae2AAA2b/ZweYiIyKsxqHu7iy9G3c0rEVIzE7NnA7m5ni4QERF5KwZ1bzduHFR6OvaocBxbn4BffvF0gYiIyFsxqHu7/v2BsDAEnUrFHZiDP//0dIGIiMhbMah7O39/ID4eGDIEV1f5GhtiOQSeiIhcY1CvCGrUAEaPRou8v3Fs7U5Pl4aIiLwUg3pF0aMHACB4/xYkJXm2KERE5J0Y1CuKiy6SDXbhgw88XBYiIvJKDOoVRe3aQMOG6Ns4EZ9+6unCEBGRN2JQr0hatUL7wERs3w7k53u6MERE5G0Y1CuSVq3QOD0RubnA3r2eLgwREXkbBvWKpHVr1DiWjFo4iZ0cBE9ERE4Y1CuSgQOhtMZTmIkjq7Z5ujRERORlGNQrkt69AQDT8BoGvn6thwtDRETehkG9IvH3B6ZPBwBUO3XYw4UhIiJvw6Be0bz0Er7o8ABq5x4BcnIAzbSxREQkGNQroPy2HeCPAuweNg26Vi3g1ClPF4mIiLwAg3oFVD26HQCg5S9vQ2VkAHv2eLhERETkDRjUK6CQvm2Rg0DrwKFDnisMERF5DbcGdaXUUKXUDqVUolJqZjHnXKuU2qqUildKfeTO8viKVlE1EIG/0QlxAAB9kEGdiIiAKu66sFLKH8BcAIMBJANYr5RaorXeajunNYBZAPporY8ppULcVR5fEhwMvPt9U/y1sibwBHB8+yHU9XShiIjI49xZU+8BIFFrvVtrnQtgIYCRTufcAmCu1voYAGitOU+rhAYPBnoPqYVMVMOxbaypExGRe4N6MwD7bI+TC4/ZtQHQRin1u1JqjVJqqKsLKaWmKKVilVKxqampbipuxdMxUuEQGiM7iUGdiIjcG9SVi2POk6qrAGgNYACACQDmK6WCi7xI6ze01jFa65iGDRuWeUErqnr1gLSAxtCHDgG5ucDttwP793u6WERE5CHuDOrJAJrbHocCOODinK+01qe11n8D2AEJ8lRC2cGNEXTsEPDtt8CrrwIzZni6SERE5CHuDOrrAbRWSkUopQIBjAewxOmcLwEMBAClVANIc/xuN5bJ5+gmTdEgex8K0jPlQEGBZwtEREQe47agrrXOA3AHgO8AbAOwSGsdr5R6VCk1ovC07wCkKaW2AvgFwD1a6zR3lckXqciOqIOTOLm2cNW2wMCzv4CIiHyW26a0AYDW+hsA3zgde9i2rwHMKPyi81D7kijgI0B//4McCAjwbIGIiMhjmFGugmsxPBIAUHfHGjmQkeHB0hARkScxqFdwdcNqY49/hHXg6FHPFYaIiDyKQd0H/N34YusBgzoRUaXFoO4D1FVXWg8Y1ImIKi0GdR/QZaYtEV8aJw8QEVVWDOo+oE5YMO7q/QcW1JouA+VycjxdJCIi8gAGdR/RaUpvrDvVTh4cO+bZwhARkUcwqPuIkSOBVP8m8uCWW4CvvvJsgYiIqNwxqPuIunWBnMuuwObAaGDpUmDUKOD0aU8Xi4iIyhGDug8ZdU0ABucuw/ExN8mBn37ybIGIiKhcMaj7kKgoIAWN8eu1c4BatYAvv/R0kYiIqBwxqPuQVq1ku3NvENC7N7B6tWcLRERE5YpB3YcEBwMNGgAJCQB69QK2bAHS0z1dLCIiKicM6j6mVSsgMRFAz56ytnpsrKeLRERE5YRB3ce0alVYU4+JkQObNnm0PEREVH4Y1H1Mz57Avn3AF781BAIDgQMHPF0kIiIqJwzqPubWW4FOnYBHH1NAkybAwYOeLhIREZUTBnUfExAADB8OxMcD2XWboGA/a+pERJUFg7oPioqSZHLfbGyCo1tYUyciqiwY1H1QVJRsD6Apgo6xpk5EVFkwqPugNm1kexBNUPP0cSAry7MFIiKicsGg7oOqVAHmzwdO1mgqBw4d8myBiIioXDCo+6ibbgLaDQ0HAOh33wPy8gCtPVsoIiJyKwZ1H1bQtz8+wPVQs/8D1KkDTJ/u6SIREZEbMaj7sJat/DAFbyCnUQsgMxOYM8fTRSIiIjdiUPdhoaFAFqpjxb3fyOi5WrXYBE9E5MMY1H1YaKhst/l1BO64Azh1CkhJ8WyhiIjIbRjUfVi9ekDVqsD+/bDmuSUkeLRMRETkPgzqPkwpoFmzwqDeurUc3LnTo2UiIiL3YVD3cc2aAcnJAMLCgIYNgUWLPF0kIiJyEwZ1H3empu7vD9x7L/D990BsrKeLRUREbsCg7uNCQyWoaw1gwgQ5uHatR8tERETuwaDu46KigJwcoH9/4Pe/m8q0tq1bPV0sIiJyAwZ1HzdhAtC3L7BqFXD/Awpo3x7Yts3TxSIiIjdgUPdx/v7At98CV1xRGMsZ1ImIfBaDeiVQowYweDCQmgqcCm0vq7YdP+7pYhERURljUK8kIiNluzuwreysXQscO+a5AhERUZljUK8kOnWSbVxGK9kZOtTKI0tERD6BQb2SCAmRZvgtmS2tg5mZnisQERGVOQb1SqRpU2BPanXHg1lZnikMERGVOQb1SqRpU+DgQaeDu3d7pCxERFT2GNQrkSZNgAMHINHd4KptREQ+g0G9EmnaVFLGHl++BvjqKzl49dXA5s2eLRgREZUJBvVKpGlT6UKv27k59nUdAfTqJU989JFnC0ZERGWCQb0SadLE2l+yBMDq1RLYV63yWJmIiKjsMKhXIsHB1v7SpYU7/foB69ZxFDwRkQ9gUK9EBg0C7rsPmDgRWLECOH0awKWXys6yZZ4uHhERXSAG9UqkalXgqadkcZfsbGDLFgCXXQaEhwNz5ni6eEREdIHcGtSVUkOVUjuUUolKqZkunp+klEpVSm0s/LrZneUh0b27bNevhyzjdvPNwMqVLiaxExFRReK2oK6U8gcwF8AwAB0ATFBKdXBx6ida6y6FX/PdVR6ytGwJ1KsnXekApOoOAN9/77EyERHRhXNnTb0HgESt9W6tdS6AhQBGuvH7UQkpBfTvD3z9NZCTA6BzZ6BRI2D5ck8XjYiILoA7g3ozAPtsj5MLjzkbo5SKU0otVko1d3UhpdQUpVSsUio2NTXVHWWtdG67DTh8GFi0CICfHzBgALBmjaeLRUREF8CdQV25OKadHn8NIFxr3RnAjwDedXUhrfUbWusYrXVMw4YNy7iYldNll0kT/B9/FB6IjASSkoCMDE8Wi4iILoA7g3oyAHvNOxTAAfsJWus0rXVO4cM3AXRzY3nIRilJRpOSUnigQ+Fwh23bPFYmIiK6MO4M6usBtFZKRSilAgGMB7DEfoJSypbjDCMAMKKUo0aNXAT1rVs9Vh4iIrowVdx1Ya11nlLqDgDfAfAH8LbWOl4p9SiAWK31EgDTlVIjAOQBOApgkrvKQ0U1agSsXVv4oFUrICCgcPI6ERFVREpr525u7xYTE6NjY2M9XQyfcOedwJtvAunphQf69gWOHJHaunI1JIKIiMqbUmqD1jqmJOcyo1wl1qiRjIvbt0/SxuKGG4Dt223VdyIiqkgY1CuxRo1k26IFMHAgkD/6Gjnw00+eKxQREZ03BvVKrHFjx8en/OrIousJCUBeHlduIyKqYBjUKzFTUzdOnADQurUE9XHjgOrVgQo25oKIqDJjUK/E2rYFBg8Gpk2Tx8ePwwrqn38uBxMSPFY+IiIqHQb1SqxGDVnDZcwYeXz8OKBbtQbsqXiZD56IqMJgUCcEB8t2wADgo/WtHZ9csED611NSbMu6ERGRN2JQJ9SpY+1/tdUW1Nu0ATZuBN5/XzLO9exZ/oUjIqISY1CnMzV1AFiecJH14I47gFq1gE2bgKNH5VheXvkWjoiISoxBnRxq6qfyqiGnUeE6PI0by6oviYnWCSdPlm/hiIioxBjUCVWcVgBIDS5sgm/cWL6WLbOePH68/ApGRESlwqBODqpUARJQGNQbNZKaut2xY+VfKCIiKhEGdXLQrx+wMj0GqFkTaNasaNo51tSJiLwWgzo56NMHeOLAjcjcslsmspuaulm1jUGdiMhrMaiTg5gYIE/7Y+P+hnLA1NRbtpQtgzoRkdeqcu5TqDI4eBDIyQECAuTxI48AQ4cCdzUNlAMREcCuXQzqRERejDV1AiAV8rAwWaStSRPgxx+Bu+8GdI/ChDP33gv4+TGoExF5MdbUqYj69aXmDgAHq7VEU7NSW506HP1OROTFWFOnIqKjrf3t221PBAezpk5E5MUY1KmIOXOAjz6S/R07bE+YoB4by3SxREReiEGdiqhVCxg/Xma0OQT1Ro2AtWuB7t1lkRciIvIqDOrkklKySNvOnbaDnToBR47I/vr1HikXEREVj0GdihUWBuzZYzsQFWXtb9pU7uUhIqKzK1FQV0pdpJSqWrg/QCk1XSkVfK7XUcVmgvrmzcCIEUB2W1tQj4sDCgo8VzgiIiqipDX1zwDkK6VaAXgLQASAj9xWKvIKYWFARgZw9dXA118Da4+3BapVk0739HSntnkiIvK0kgb1Aq11HoCrAbyotb4TQJNzvIYquBYtZLtrl2yT9gcAP/wgmWmqVAHmzfNc4YiIqIiSBvXTSqkJAG4AsLTwWIB7ikTeIizM8fGOHZAVX3r0kOHx8+cDp097pGxERFRUSYP6jQB6A3hCa/23UioCwAfuKxZ5A5dB3RgyRJrgExPLtUxERFS8EgV1rfVWrfV0rfXHSqm6AGpprZ9yc9nIwxo0kAFywcFAaKhTUO/YUbbx8R4pGxERFVXS0e8rlFK1lVL1AGwCsEAp9bx7i0aephTw1VeS7v2GGySonzhR+GS7dnICgzoRkdcoafN7Ha31SQCjASzQWncDcJn7ikXe5vLLJTPsDz8UHqheXdZYj48HzIIvRETkUSUN6lWUUk0AXAtroBxVIr17A3XrAsuW2Q526gT89pssybpggcfKRkREoqRB/VEA3wHYpbVer5RqCSDBfcUib1OlCtC3r8Tw224Ddu8GMGiQtUarmd6WmAgkJ3OJViIiDyjReupa608BfGp7vBvAGHcVirxTx47AkiUSt8PCgJnXDLOebNhQtq1bW8cWLgTGjSvfQhIRVWIlHSgXqpT6Qil1WCmVopT6TCkV6u7CkXfp0MHaj4sDcNFFsmIbIO3yV1/t+IKVK8utbEREVPLm9wUAlgBoCqAZgK8Lj1ElUiSoA8C6dcCdd8pguS+/dHxBvXrlVjYiIip5UG+otV6gtc4r/HoHQEM3lou8ULt21v6OHUBOTuGDiAjXL1DK7WUiIiJLSYP6EaXURKWUf+HXRABp7iwYeZ/q1YGHHwamTpXpbT//XPiE6U9/4AHg/fetFxw/Xu5lJCKqzEo0UA7AZABzALwAQAP4A5I6liqZ2bOBrCyZr37nncDQoYC65hogP18GxZnVXwAGdSKiclbSNLF7tdYjtNYNtdYhWutRkEQ0VAlVqwbMmCFN8Hv3AvD3B66/Xua9hYXJFmBQJyIqZyVtfndlRpmVgiqc6GjZbtzo9ERQkLTLR0VxrjoRUTm7kKDOUVCVWKdOMg7ur79cPNm3r0x3O34cePxx4JFHyr18RESVUUn71F1hwu9KrEYNoG3bYoI6IEu7HT8OfPopkJsrbfY33gg0alSu5SQiqkzOWlNXSp1SSp108XUKMmedKrGBAyXnzK+/uniybl1pft+/H9i+HZg1C/joo3IvIxFRZXLWoK61rqW1ru3iq5bW+kJq+eQDnnoKCA8Hrr0WeOghp3FxwcFAZiaQZpv5uGdPeReRiKhSuZA+darkatcGHnwQSEmRrvMnnrA9GRxc9AVJSeVVNCKiSolBnS7IdddJMpqmTYG5c4GMjMInGjcuejKDOhGRWzGo0wUJDAReew2YM0eS0mzdWvhEnz5FT05KsuWWJSKisubWoK6UGqqU2qGUSlRKzTzLeWOVUlopFePO8pD7dOwo2/j4wgNNmhQ96cQJGQW/bh2wb1+5lY2IqLJwW1BXSvkDmAtgGIAOACYopTq4OK8WgOkA1rqrLOR+LVsCVavagjpgLctqpzXQs6fr54iI6IK4s6beA0Ci1nq31joXwEIAI12c9xiAZwBku7Es5GZVqkg3+nPPATNNm8zPPwO7d0su2aNHgTfesF6QkuKRchKJMQe4AAAgAElEQVQR+TJ3BvVmAOxtrMmFx85QSnUF0FxrvdSN5aBy0qWLbJ9+GvjwQyC7Sk1ZlrV5c5m3fsst1sn163umkEREPsydQd1VGtkzWeiUUn6QVd/uOueFlJqilIpVSsWmpqaWYRGpLM2bB9x3n+xPnCjT3FascDrpkktkGxBQnkUjIqoU3BnUkwE0tz0OBXDA9rgWgEgAK5RSSQB6AVjiarCc1voNrXWM1jqmoVm7m7xOo0bA3Xdbj594QrLOFRTYTvrxR+Bf/wIOHZIE8kxIQ0RUZtwZ1NcDaK2UilBKBQIYD2CJeVJrfUJr3UBrHa61DgewBsAIrXWsG8tEbtaggXSlR0Zaxw4dsp1QtaoszwoAW7YAb71V9CKnT3PqGxHReXBbUNda5wG4A8B3ALYBWKS1jldKPaqUGuGu70ueN3CgJKMxilTG7a0tNWoUvUCHDq6nxBER0Vm5NX+71vobAN84HXu4mHMHuLMsVL5mzQK+/172n3kGaNUKGDcOiImBY1B/7z3JET97tnUsMbFcy0pE5CuU1hVrBdWYmBgdG8sW+oogIwOoWdN6XKWKxOuwo38B0dGOJ9v/DlXhGMtTpxwvQERUCSmlNmitS5ScjWliyW3sLetPPw3k5QE7dsCxpm6YoJ6bax3bvdut5SMi8jUM6lQurrpKtqmpAEJCZESd3YsvAtu2OY6qYzM8EVGpMKiTWy1dCnz+uUx3A6TyfTQ9UKJ7v37WiTNmyAC5b7+1ju3aVb6FJSKq4Nw6UI7oiitkW1AA+PsDDz8MLFsGrFkD4OTJoi+YOtXaP7PkGxERlQRr6lQu/PysFvdNmwoT0txzT/Ev6NkTWLWqXMpGROQrGNSp3JjxcdnZhV3n110H/PGHdYK96X3sWGl+37+/XMtIRFSRMaiTR6xdWzjgvU4d62CTJkBcnCSRv/RSOfbvfwODBknVPicHeOghID3dI2UmIvJ27FOncnPqlLU/erQs+vL+U7agXr8+EBoqOeHz84HatYHFi+W5tDRg4UJZJSYwUIK73X//K9/gySfd/0aIiLwUa+pUbpzHxX3wATDhtmDrgH05Vn9/a0U3QIL64cOyn59f9OIPPAA89VTZFZaIqAJiUKdyM3q0tV+vHnD//cA3K6pbB6tVc3xB//7W/pEjkqIOAIKC3FdIIqIKjEGdys2rr8q4t0OHgKQkWZo1brMq/gXXXAPUrSv7Tz8NfPKJ7O/eLUPojexsaz8rq8zLTURUUTD3O3meyfXu6m9x3z6gRQvXr5s3T7Lb3HYbMHy4HNu9G4iIcE85iYg8oDS53zlQjrzKsmXATz8Bzz9feMDez+7s1ltlu369dezQIQZ1Iqq02PxOHvf0TTvRtmoSkpOB668HXnhBVmMFAFSvfraXiuPHrX177ngiokqGQZ08TrVpjZ05YbjuOuDECTmWlHSOF4WGWvv2PnUGdSKqxBjUyeNM+thVqySRHAD8/beLE48csZZ7GzXK8bnQUMlFy6BORJUYgzp5nH159WnTZHvllRLgH3vMdmK9eta0tr59Hae89egBNG4M7Nzp9vISEXkrBnXyOHtQ79PH2v/sM1nVbe+YOyURjVJngvqxqo2BFStk5DsAdOkCDBsmI+3szfFGQQGQl+e+N0FE5AUY1MnjTPM7IBlgjRkzZPvzlc/juwdXYds2SI0cwKh/NpcnzYi65s2BceMkVezy5cB33wHvvmtd7B//AAIC3PcmiIi8AOepk8edOAEEF2aL1Rp48EHgwAHgtdckeVz16hK7mzcH/t6eg1v7bsVbf3bFyZNArSN/AzNnAm+/DVStKk3w7doBf/0F1KwJpKTIhc1c+JwcxzsHIiIvx3nqVKHUri3bf/xDto8/bj3XqJEVl/ftAx75b1UsO9AVgOSZiYqKsDLNAcCYMcAbb8h+ZqbU3GvVsp4/cAAID3fPGyHypPx8GSyqzpKlkXwem9/J45SS2vrbbxd9rkkT2c6YIcuvP/GENcB91y4XF7vxRqne33ST65OSk8us3ERepUoVa6QpVVoM6uQVateWzyRnpsu8a1fHLnLAdVBPqN8LecdOAXfcIQcSEx1P2L//wgtL5G1MN+rrr3u2HO6WmCj9cfHxZz8vI0O64Dzh88+BWbM8873BoE5e7uhR2UZFSdAPCbGecw7qBw8CHToAC96vAlx0kXWSfdwIa+rki3JyPF2C8rFpkyzaZF/QyZWxY4HoaNczYc6loAAYMABYsqRk569dCyxaJLNrnnoK+OIL4K23Sv99ywiDOnm1uXOlC7xdO3n844/A//2fzGBbs0bGvP34ozy3bp38X61aBWxLriV3AAkJ0q9uMKiTL6ooqxPGx8uKiyW1Y4dMZzV39wcOyPZcSaaWL5ftmXzTpZCeDqxcCYwcWbLzH38cmDJF1qCYNUsCvBn56wEM6uTVrr1WssuZ2WidOkkzfNu2crN++rSsyw4AGzbI9v33pcaeFxkF/PorMH26dcF9+8r3DRCVh/OpkbrL6tXAli2un3v7bZmtkpZWsmv961/A77/L/zFgdZ8dPFiy1x89Kh8SpWGvBJTEtm0yKCguTh7n5jKoE5WWaV0HrP/vP/90POdIh35SUzed8fXqyfz1kn6gEFUU3hTUb7lFArcrZlGHhISSXcvcHAQFydbU1Esa1Nu3B8LCXC/rXJzSBPXsbCuntbnxABjUiUrLOahnZFg1dWNXs/6OB/73Pznx2mvP/Y+bmgq89JJMEyLydu5ufi9NUDx8uPigu2ePbF2lcz5wQJq8d++2jpmaeXq642PT/J6dLTNeiutWy8uTskRFFX9OQQHw/ffWjVFpgvrOnfJ6QPr9DAZ1otKxB/X8fODbb+X/fNYsYOBAOb4xsIdjurr+/ZH6yBzg55+BTz89+zd47DHg3/8u+WAZIk9yZ01940aZ//7LL+c+t6BAWsJMcglnJqjv2FH0ucsuk/+3n36Sx/YbAxPUnWvqS5YA77wD3HOPde727UWvvXmzDLpxZfJk4PLLrZkDJ09az2kN3HeffLBs3y5rThw7Js/l5QFbt1rn2rv2GNSJSscE9a6ShwbvvSfb4cPlM6F6dWBXclWpcR89CixYgNOhEQj9T+H89dWrgcWLi154wACZ62uyzv3ww9kLsnmz5Jsn8iR3BnVzA1ySoH7ihAT2w4eL1u4zMmSlRaBoTT0vT/qmAeu9mBsAQGrP06dbAdvU1HNzZWu+19690uRuN368bE0wtsvOBj78UPbNte019SNHgGeekVHtP/0E/PabDIhLT5eBuGbQX3S043Xr1Cn6vcoJgzpVSE2bSiV85EjJBvv115LEpksX2YaH25ZvrVsXS+pNwuDBQC6qIhUNgPnzgWuucbyrz8+XUa+vvWbVDL755uwFefJJK9ENkae4s/l9717Zhoae+1wTtE+fBo4fd3zOBGk/PwnqBw8CgwdLs7i9dmyCrz2nRHo68Morst+kidyo5+RYo9vNSNrffy9apmbNZGtG0C9dar2nLVushZ4SEqTGf/XV1mvtTfbmsyIhQT5cjh2TVoyQEBnBa8eaOlHp+PnJ/+PMmdaNedu2EuABICJCpo++9578740cKfEaAA6iiXWh6Gi5C09JARYutI6b5sM9e6zagCupqfLFvndy9vDDwJAh7v8+p0459kOXNROMtZZgGBVlDT6Nj5catKnFm6AOFG2CN4E0JkYC45IlMh918WLHoG5uBuwBNT1dBsuNGiU1Z0CSyxw+LPv+/rL944+i5Q8JkSQXx47JzcZVVwE9e1rXAGR5yIQE4LnnHF+7fr21b6bJJSY6lq15c6BVK8fXMagTlV6jRrKGiwnkd95pPTdlisTaG26QWTEtWljPHUBT64Gfn/SXtWgBTJxoHbcntzh4UO7Mr7hCEkt8/rn13JEj0tyYmlq2b44qvscec+y+2bRJWojK2uzZ8gd/PnJypE/ZBFxXzIj1rCxZ6jguzupymjVLatCvvSZB8/33rdc5B3XTZD5ggFzLNOv/+KM02xumpp6cLP/gderI/1l2NtC9u/wf+vvLTYH5HqZlbfVquWm4/nrrejVqAHXrSk3dvE9Tlj//lOsPGeJ6uqu9i85kp0xIcDyXQZ2obD33nOR/uOUW69iIEfLZc8klwEcfOf4Pnqmpz5iB/OOnkFOletHa+N9/ywcBIB8u//2vNMWPHg3cfrvVh2dqJsUNDCIyfytdujj+kZYV59THpZmX/f33wIIFcucLSI05IAD48kt5XFBg/fNkZVkB3tSMzfTQtDRJzfzaa9a1U1LkRubNN+Wx+V/p00e2ZkDcypWO00yPH5dAP2+eNPnXqmWVoW5d+erXT5rRzf+deb1JVvPf/1rXq1FDprMeO2brk4P8XhISpKmvbduiP5uYGNdjahITiwb11q0dz2FQJzp/0dHAAw8UXZyqfXvJ1pifL/+/06ZJk3w+5AMpo0EYdiYoJORFnHlNfo9e1gW6dZPt/v2OiekPHZJjWlsfVOfKcEWVi72P23mKlJkCVRLmhkApqVGb/l875+s7Z1E7dUr+8F0xNwDm+6xfL9/j2Wflsb05PSvLCp6mZm2ays2AVLuUFKlVT5kiQTA1VQag2geVde0qtWzTbB4UJMF38GApd9260hRnatjmRjsmxuqXB6RcmZlyrUaNgGrVrO9Rvbq8btMmx5YEPz+5CWjQQG4EnPXube13727t795t3dwArKkTlSf7/1r79vK/GdNBPnC376+FnTuBJIQDAD7DaKx/4XcU1Chszy8cWr9nzhJkrXZaHGLtWvkQMSN1XdXUtbb6OnNzgV69ZO6dcfq01Chcjcol71FQUPosY/aFhJx/vyUdqf7440DjxlYgXbBAatGmayg+XoK9fX40IKPM7V57TYKWq/dgcsb7+0ut1dTQTX+VfVpZZmbRoG7eW2qqjF41QkKk79u0IixaJOc0bGgNXDNlA6xcz2Fhjj+vXbukpm76sE2wbNpUym5GzKelWV1gISFWshrAqqnv2GFNkzGSk+W5Zs0cF5YApP/e+OQT+QCZMkX+b+199y1aSBP+Bx/IcpL2cnoAgzr5ND8/uVEHpCVPKaD9UzcAAH7O748dO4A9CAMgwX3RYj8syRgEAMiq3QgAELbqQ1TbZKvp+PvLnNfvv7eOuaqpv/22zL1bu1ZqGmvXWtNnAGDFCmli+PjjMnu/dJ6+/lqyDbry9NOyjKBzTfRs7BnTnEeBlyQf+a5dwEMPyUAw+1xoQPqNAetvyT7IDCga1Hfvltq3q35z8578/WWBhTlz5LGZkmX/uz582GopcK6pHzni2C++dKnjYLIXXpCbkIYN5Z/wgQdkfnmPHhJUzSjWsDAJzqbZbcECqamba5uaurkxsDf/mxvrkBDXNXU7++P69WW7YYOVcxoALr1UprB9+aWMvN261eqr370biIyUGw4zr/b662VGTadO8j48hEGdfJ7JP2MqElWvGoLorhrv/dYSv/5qBfU9CMMLLwDP4F4AwL2fdne8UN++8uEUGSkd+aNHW8/ZP/yOH5cvM1p2927rA27lSqnBX3mlNTLaORUelc7UqdZ0p/M1YgQwdKj8Tp3nUH/1lWxLs5SnPRAnJjreEDgH3ddfL7qusL151zmlqpm+ZW8at3O+aTC1ZVcDwUxQzM937BYwQdReU7cH6RMnpKaclSX/YHl50l9dpYrUart3lxoyICswpabKjXDDhnLs8cdlFKtS0pRuhIXJ99QaeOMNa86qYYKxvVWgYUMpi/mZmdHupsusRg0JvkarVjIVzahXT7ahoXKzYdenj+PCLvamv7Fj5YaqTRvrWK9eMpjHXuZyxqBOPu/BB2Vrz0LXvbtMiVu2zGp+N9vVuBg1A3IwJ64fVsPWxz5unGSZa9u2aL/oCy/IXT0ADBokHz7mQzcnx/pATU6WDz97wprY2PN/c+vWcY34efMcF+25EF98IRkH7cyH9rBh0vpSHHuyFfuN2tix0rds2IOu1jKifNIkab41zFQtoGhQN/3gxQV1k+TF1FxLEtSd53ebGrgJ6k2bWkE9IkKCugn8ZpDYjh1yY3TttfLYBPBx46RGbj9m17mzbJVyDNbNm8vWVVC3N+EPGCBbE6hNM7qprVevbjXNP/+8jBuwl8MEdXPu2TSxTYft37/48zyIQZ183i23SAw2rWyA5JgwceB0n4FYrMZi5DOXYO5cmSmTcVoyyvXFKgQiB2M77ZAaIeB4Zw7IACZAalsFBdbKMitWyDYlxfED1dT8jPj4oslDZs6U2uO5jBpl3bV4i9zcki+4YcTHA088ITXGf/3r3OtlX6j8fCu3gHNfs3OwNNkFT58uPtHQe+/JeaY/ODYWuPhi6/k1a6x9E9STkiRHgrFokbVvD+rOLQfmexQ3ODMzU5qCx4+Xm4aSBHVzA5CYKDel9qBeu7bUxs11WraUn5lpXTBBPSPDMUBed51s27SRkf+A4/OGeb3Wjs3iJqjba9mmr9oeXE0GyCeflMcmYNuDurkRiomRa1SrZrUk2D8YzsU+GtfMdfcyDOpUKTiPjG/VStZrOXUKWLyyIYad+hQ33VMP06bJwFvTZH/rtCo4jUB8trkN2nX0x733Ail1HIO6fu5/WFdjIPLe+1Ca5p2ZoF6vnlz4rbfk+JgxViDbtUuCoQk0338vX65GO9ulpRW/zKWn3HKL1LhKM7Xqiivk5mTJEuDll60BR4DUCC+9tGjfMnD+6VHbtJFfNFA02DkHdXtfsfMoZ2PRIvldPfaYBNvkZMll7ooJ6gMGWH247dtLv/6998p1Dh+Wfu7g4KJB/ehRGeVt+tadHT8uK4Zt2SKDSswNwp491t9XQYHs299r/frSnFWvnmNQb9JEgqApd0ThbJF7pZvKYTqXPWg/9pjcrLVqBXTsKMec+/+dX28P1iaDnamp16xpdT1UrWqd17+/442tCdYmqNeoId0zDz3keKNlgr/zjUZs7NlXkZs+XfJR2/vtvQiDOlVqNWvKZ6f5HABk4GxiokxRnTtXpqUD0rr47LPAzc/KnNZ1HW/Ewe/icEwH4/eMKFTJzbJG444bZ13w8GEJHC1ayPza+Hg5Pnu29SFz+LB8UI0fLx+427dLs719FLWz3Fz52r5dFrrwlgF3JmGHWXyjJEx3xquvyrZxY+u5FSskY9lttxV93fnMHMjPl3EOv/witUvnAWSpqfLBbYLyyZNSO+3VS25UliwperNlAtCXX1rNwGZlIWeZmRJM7bnNzYIkzz4rwT0lRZqRmzRxXVM/W9OvWXHM+ebkvfek+wiQmnx4uOP8cNM/FRxsBfVDh6ygbrRs6Xjd4oK6nx/QoYPsm9YtezB29fpRo2TAXr161mA9E9SLS4WrlPTPO7PX1Fu0AB591JpfDxQf1Lt1K/7mDZDagBev98CgTuRCnTrWZ7r5n/fzk27PlYfbIcOvJl6L74vukzthzx5gM5xyP991l7WfkiKBIzTUqh0C8iFq+v9Mc/XixXKu+QA7Wy3cNBunp8uAnuuuK32ztzuYD+GzZSlzZn7IZmqT/QPc3BwcOiRfgwdbCU1KMyLdsJfr66+LlvPIEanZ/fST3FidPCkf8r16SSAeOVJqoXYmo9nevVZQN33Fhl/hx21GhvU+jUmT5CajSRMZn2GCeqNGRRMjJSfLoL3LLy86dapq1bPf3M2ZI83oCxfKdezpZe1B/dAhaUXYvVv+bu1BPSLC8Zr2BVRcNa8DcoMzb561AIqdvR89MFD+5nftso5df738LIYNc3zdxo3WTbSZgme/abAHdVeKC+oVHIM60TmY//kaNaQCXi+sNkIL9uJd3ID9++WzagMKE9U88oiMcO/eXar2V1whTaGbN0sf3E03yYd9+/bSJGCCur1p2XxQAfK6X3+V2oNz36/9sRn5u3mzbLUuOsraOHlSajeuVqkrC6YP1DRra110cJgz537NpCRr/rWpqe7dK/OEf/xRxhxofX5B3b7s5/r1roO68dFHUjOuXduxjCtXOp5nfhf5+TLroXHjosHC9BFnZkrrip+fjAL/97/l92Ga41etkgAfEuLYYmGsWiXvferUogGrTRvrBsN+LDHRuhG57z7rOfvPz0zDMjcKTz4pN1SRkVaAdP45ABIc27WTfeepY4ZS8rtz9by52TG1aNPtYC9/SorcgNlFRVnfF5Dfh/13Wa2aBHl77dy53EDp+tQrAAZ1onMwn801a8osmRUrgGun1MXGTX6IiJCgHocotKmejPda/geb6/aTGN2mjXwo5+ZKDeSWW6RP8K+/rNHR9erJh44JxoAVzBo0kA//u+6SwXc//STN1K++KoHZVTIRc50vv5RanvMcacCq/ZusYWXN1NRNFrEePSRAxcdLkGjYUEaRP/qoNVDMOVnIX39JV0VystW/mZ1tNXsePSpTh+zN76+84hikTLKRQYMcF9wxQb1hQ6mJ2muFgGOwnjxZzncV1Bs2lJrnqVOOgXTlSqvZ2WQlNN8PsIJ6eLhMoXrhBeucW2+Vv5uMjOKDurlB6tKlaMAytWYzKASQG5eLLgJuvFEef/ihY43WBFrzGufaf6dOVlBv0kR+FnbBwfK7As49BqQ4e/Y4Tpk7H/XrOyaQsQ+GcyU0VG6K7APxfACDOtE5mM88E6vCwyWQd+7smHQqIbMZbrhBjkdFFR40geDBByXIAlIzMR+Sfn7yYW8P6u+8I82JkZESBE3f4s8/y9ftt8vCIM41MkACHSABNCPD9XQ3M3DKuYaitTUo7MgRCS7O2cqMlBQJqPn51vc0zAf73r0yKDA2VsoyYoRkEEtLkxucRx6RBUEAx1aFyy+39vfulZq6GWhVUADcfLPsr1jhGMSnT5em3Ntvl5phu3bS1/rzz1Y/85tvSk26bl1pOdm1S1pG7AO0XHUbmBHgzmbOlBrzqVOOgyRNeWNjrd+tGTm/c6eM7rfXMo2AAOtmKyTEMSf54MHWEp916kjNurigbmZOBAVZQatZM8m1AMjcccMccx41bthr6s2aWbMBzGuDguTm5o47HJctLY0WLVzfwFyIatXOPkVtxgxpBXMeRVvBMagTnYOpuLjKJ2E+u53l5cln5C/d7paFYB56CEuWOI5LOiMkxDHZyMGD0n/btKnsm8FyixfLyHBAgqPJY23/kDUB1kxRMrVOraWv9aWXrAFazkF97lx5s/v2SZkTEhxzZQNWrbZxY/l68EG5g7npJmtJTFN73rtXpj1dcokE4MxMaW4GrOBjym6/QbHn2Y6Pl9r0lVdaI5/79ZNAsndv0eb3nBxrsJ3d22/LHOUpUySo33671F4TEqTW7Cr391tvWUHVVbPz+vUSGD75RFo/7AHY1FwBCYrHj1tTH//3P/me9sQJdlddJedMnuy4Tvf331tTwy6/XIKRc1C/+GKphZuBmo0bOwatf/5TtoMGWcdef11+LmZ+uUkdazRvbgX1pk2lDO+9JzcyJigGB0tLiTfVekNCiqZ+tatXz7ElxVdord32BWAogB0AEgHMdPH8VACbAWwE8BuADue6Zrdu3TRRefrjD60BrUeOLP4589W0qdZ33mk9vvlmrQsKtP77b3k8apSLbzB4sPWCkBDZPvec1nfdZR2/4grrOeevnj1lW7Om1n5+Wp84ofW118qxzz6T7/HJJ9b5N90k28mT5bmcHMfrLV2q9XXXWY979NA6L0/rZcusa5rnGjd2fK3WWlevbpUH0PqNN+T41KnyuHp1rbOytB43Tva11rpNG62rVtV67lyt58+3rtenj2x//FHrjh1lf+1aOf+aa7R+8EHXP5NnntF63z6tH35Y62nTHN/L5s3yPV9+2Tr+4otFr7F0qda9e8v+//6n9caN1nO7d8s1duywjk2erPXw4VrfeKP80p0VFDhe/803z/3Hd+yY48/23ntlPz7e+rkBWr/zjvW+0tO1TkuT4716Fb3mli1Slrg46zp2WVlaP/us1t9+q/Vbb8mxe+6R691337nL7C3S0uRvwAcAiNUljLtuq6krpfwBzAUwDEAHABOUUh2cTvtIa91Ja90FwDMAnndXeYjOV8+eMmbI1VLY9oG/APCPf1hT4AB5TevW1hLs9pbdzEzgs8+A/AaFtYmAAJlLC1g1deOmm2RecpcujnO4AStb18SJ0jw9a5Y18M40DdizlZmR16ZP3nmE/bZtUis0fafr1kmNzCSEsS+K4ZwAJTdX3li7dlL7rlrVqgGaWmdgoNS0o6Lk3LvvlibpiRMlkYh9UN3vv0sT6iWXWP3UrVtL7dFVTd1MRRo5UvpMZ892nAq3YIHVTO6cYjAw0PoegNTyTN+Lc/O7GQFuz/Fds6b0+b/9tusmXfuxBx5wPQ3LmXP/9iOPyO/WlNPU1KOjrfdVo4a8zt/fdZN2x45Slk6dHN+vERQkv5OhQ63WBfM7KesmcneqV8+aaliJuLP5vQeARK31bq11LoCFAEbaT9Ba2zMR1ADgYlgskWf5+UnXqasuVftn7tdfS+uy8zTeXbusacgmTubnS0wbOxbYnFaY8vL0abkj2L5dpqjZ+3m7dZML//WXNM1+8YX13GWXSSFNX/Orr1qB+sgRGVT33XfWvGnT/G6CunOa2vffl9fNmSNNy4AkfzGJUpzTqNqZpvdbb5WAd+211pgAE9RNU7tpzv7f/2Rr+jfGjZP3bwY5DRokNwejR0vTdN260ge7b59j5jVA3mOjRo5znyMjJbDPn+8YxHr2lOU1o6Pll/Hhh45Z3Ro1soJ6lSpWee3XrlrVOqc0Tc/Dh1vdCefSvbt141a9uuOd5LvvylQv5/XA/fzkxspV0D4f5vfqYyPFfVGVc59y3poBsKdqSgZQJK+eUup2ADMABAK41I3lIXKLN9+U/vMrr7SOHT0q47JefFEq3Cbnx4oVUkn64Qerq/w1PRXzUNgfrZT1Ae0qD7ZhD/hDhkhwq19fBouZfndAaurx8VKDvvVWmSplFK8kIlIAAB2sSURBVBfUTb/85ZdLbdWeSMf+uogIyWNvN3y4bENCZB6xfVS7CepmTrFzgDBBPSRE8uib/nbzwxs/Xr7MNQ4elNaLsDDrRuXZZ+Xmw7mm7KqfvX59xyU0x451fD4kxLprO3FC3sunnzqusw3ITcuxY6UL6s6/z7NZt67457p3l/EPrqxZ4zrZy/kwLSLFTVkjr+HOmrqrIYVFauJa67la64sA3AfAZRJrpdQUpVSsUio21STmJ/ISN9/suJATIJ99TZrIoODp02WKsL3S9PTTEnfGjAE+Wh2B9bM+x6bZXzhexB64nYOUPeBXrWoFyJdekosaR45YCUbODMmH3IVs2yYrYbkKGjEx1iCjBQtcNyfbR6kbJu993boSEO1BPThYapZmyVrnoO4cFN95R5p/XWVma97cymtuL0edOjI94UKsXy+/sKAgq/nZfI+xYx0XEwGsEdalWZnL/rt1F3ta1QtlRsg79zeR9ylp53tpvwD0BvCd7fEsALPOcr4fgBPnui4HylFFNWeO4zip7t21/ukn63FQkNML0tPlieuuK3qx3FzHAVR2d9xhPXfllVrPni37WVnWcftAOEDrevVke9ttWq9cqfWuXY7XNIPjOnWyXvP5564HqQFyjXOJi3N8zZw5Jf5Z6thY63WLFxf/sygPMTHyvd9999znerKcFyIvT+s9ezxdikoL3jBQDsB6AK2VUhFKqUAA4wEssZ+glLJ1TuEKAGfJok9UsTnnwRgzRrqqzWqe9pbSY8eAgydrIG1NAgreWlD0YmergdlrqmlpUlNv1kxqnn/9JYPPTD+38dZbMun+/vtlOpbzwAAzlSoyUvrJ27SR85ctc1xWFJA50KYP+GyKa34viW7dZFBhWJi19KZzWtbyYn6xHlxD2+38/a1uE/JuJY3+5/MFYDiAnQB2AXig8NijAEYU7r8EIB4ype0XAB3PdU3W1KmiysrS+tFHrcpaUpL13P33y7FJk7TeulXrsDDrvOnTi7lgcbW+zEyZgjRsmHVO376O55ipUYDWgYHymnNZvlymyzlP17r9dscad0llZzu+bvHikr/WMGX55BOtDx4s/evLgvk5L1ly7nPT0rQ+csT9ZSKfglLU1N05UA5a628AfON07GHb/r/c+f2JvElQkFQuO3WSLLH22VBmptA778jgcvvUt5dflrwpBQVOFfSICMdlQY1q1SQRzBNPAN9+a31zO3v/9QMPlGwZSVd96IAswTlokIxOLw3nQVz6PCa/mL5+M23OE0xNvbhc+3Y+tngIeR9mlCMqZ6NGFV3kyz7911WW0sGDZYbVlVfKyHkAMrfbeZ643QMPSCazMWMkhaedCeoTJgAPP1z0taXRooVjvtzSWLxYpq8BxS+t6e3M6HxXWemIyplba+pEVDLF5fRo3lymdZuZaMuWSQwdPBgyd/pc6tRxvRqbSVri3Ld+vpSSNLP2FK8lMWaM9Ik/+GDRKWUVRZ8+59fKQOQGDOpEXsBVUO/WTeJc7dqyWmZ8vIxJK5Ml008W5n0qq6AOSDa481G/vuSyJ6ILxuZ3Ii9gD+qffAL07w+sXSuZ7KZNkxVMW7SQ6c0HD0oSm7vvLnqdmTOt0fRnZeZJ21cWI6IKT+kK1mwUExOjY50zYBFVcFpLZk+zX5wbbpCm+C5dJC3tDz/I4Lunn5YuXdOqfs5/64ICyQE/eLDPLT1J5GuUUhu01jElOZc1dSIvoJSkHt+48eznNW0qKc9N+vXBgyUb6FtvFc3Yahw+LCtymvVYAMgdxJAhDOhEPoZBnchLXHedYyZXV0yrufPsqffec5xxlpUlOWYAyRy7erXrVeaIyLcwqBNVIPaU71u3Su3crNa6a5f13BNPSHN8797Af/8rx/Lyyq+cROQZDOpEFUjDhrJt0EDW1ggPl5wvzoPYV66U7Zo11kIyJugvWyaruxKR72FQJ6pAunQBevVyXG2zSxfJFX/ihCyoBsjKpYDkldm8WVZP3bVL+tWvvNKzCdiIyH0Y1IkqkDp1pH/cOceLUjKf3ay7AsiSr7Nny5i4Vq1kyXGzROzx47L96SfXS40TUcXE5DNEPsTe596pk7XfujWQny9J6Pr1kznw+flSa8/OBnr2lGQ3RFSxsaZO5EMaNgQmTpTZakOHWsfHjZOm+Y0bZa57Tg6QlGRlmn3uOdfXO3ECWL/e7cUmojLCmjqRD1EKeP/9oseDgoBJk2S/fXvZxsRIXnlAVo1zZfBgCerZ2UUXVSMi78OaOlElExkJVK9u9av36gUkJsrc9x9+AGbNsjLSmVq6q5XjiMj7sKZOVMnUqgWcOiXLgGdny+qra9YAjz8OfPABkJwsXyboAzIfvnVrz5WZiEqGNXWiSsjPT6a6PfusDJYDgKeekmAOSHBfutQ6f/du19dZvFia/NPS3FteIioZBnWiSqpVK1npLSICmDrVOj5vHnD0qKzO+sMP0pe+dasMmps92+qHB4B775Xt1q2O1377ban5E1H54iptRARAmuEXLpQavH1F1qAgGS1vvPoqcNttsh8YCJw+DXz0kbzeMOvEVLCPFyKvxFXaiKjU5s8HliwpusR6r16Oj6dNk2lzJ05IQAdkepwrmZkyJ57Bnah8MKgTEQAZOHfVVUWPf/KJzG+PjgaGD5djH34IvPOOdc7vvwO33w6sWOH42vffl5uCX3+1jh07JivHFRSU9TsgIo5+J6KzatRIvjZskEC8fr0E6n//W5rmW7WSRWIA4MABWbvd+Ppr2e7YAfTvL/szZwJvvAG0aSMJcj74ABgxouiiNERUeqypE1GJ+flJStmBA+Vxx46SU97YsQM4fNh6/P33sn3sMeBf/5L9zEzr3C1bgP/7P2DyZJkr/+ef1mvz8mTKHRGVHIM6EZXau+8CbdsC//wn8Mgj0nR/zz0SqO0ryJk+9+Rk4JVXZO67Wdc9NtaaKrd6tcyDt+efv/pqoFq18nk/RL6Cze9EVGrNmzuuyT5jBrBokTTP33qrHDNJbgytpW/d1OzXrgU6d5b9o0et8/LzAX9/a558VhaDO1FJsaZORBdMKcdlXwHgzjuLnnf77VIrB6S5/ccfZd8+ZS452XG0/N9/F71OZibw/PNAXJzjIDyiyo5BnYjKRNu2MmDOrPw2dao1Xx0ABgywMtZdc41sf/gBaNHC8Tq7dwP79lmPd+0q+r0+/hi46y4gKsoagEdEDOpEVIaGD5c5659+CjRpIs3xhw4Bb70F/PwzMH26nNezp7X2+7Bhjte49FJgzhzrsel3/+EH4Lrr5PHKlY6vOZ/pcfn5wNy50rxP5CvYp05EZapZM2DsWOtxo0Yyuh0AnnhCthMnysC4DRukD/7ii4EXXpD58IDkpA8KktHvf/0FjB4tme4SE2XFOOdV4/74Q1oK/PyAgweLJtAxCgqk6b5mTZlrf8cdkrf+4YfL9mdA5ClME0tEXsPeXP/YY8AXXzhOcwMkcJ+tZl6lCvDtt8Bll1nH8vPl2gsWyCh9Mxp/5kwZwf/yy2X7PojKEtPEElGFtH+/pJ/dtAl44AFg1CjH5y+91ArooaGur1G7tlXzPn1astc1aADcd58sMXvsmIzcP3jQOofIVzCoE5HXaNpUgnLnzlKzvvFGx+fvuku29etbKWuNpCSZ+37DDdKMn5cHPP20ZL4DgDfftJr3t20Ddu60Xmd36BCwfHlZviui8sOgTkReKzRU8smnpAC5ucCQITJnvV8/oHdvICTEOjcsTJLXdOsmg9+2bQO++w7o0QP47DNpATA9d9u2SaIcoGhQ791bBu/Zp9kRVRQM6kTk1fr3l+AdECD95Z9+Cjz5pNTIDxwoen5MYc/j1KnAb78BffvKdLrwcOucVausYJ6U5Dgv3hyfNQv46SfZf+MNq8ZP5M0Y1ImoQrniChnprpRkntu82XFee+vWkvHujz/k8SWXyOC6SZPkcc2akrCmoECS4WRny+vfeUdWmzNeeEEG22VlyQj9l14qvvaenm417RN5EoM6EVVokZGOg+b8/KS/fMkSyR9/6aVyfNo0qb0vXw40bgyMHw/cdps8N3Gi9N9fcknR6w8ZYu136CBz7J0T4gwYAHTtykF35Hmc0kZElY5ZVMbfH2jXTm4C+vSR7ZgxwOuvO57ft6802RuXXSZ9+PPny3Q5k0UvORk4eVJuDlavluVliS4Up7QREZ1FlSry9f/t3XeUVGWax/HvQwNNGBQkDUFBDzDTOAICIug6OxhBERTHkaAy2hzD6C4eHV3FMJhmEEZxAPMaB1S0FVAGAwuG41FUkjkhJgSkPUbYVRHe/eO517pVNKHtUNVVv885deret25X33oPzVNvel4zz1H/2996itvPPoObbkpd99FHPrnu6adTZcXFnrP+9tt9edzs2anX1q3zLwRffAH33rvj+1i0yFPeilQXBXURKWhnnOFpZ3fdNT35DXhe+j59vEt/5kzfVjaZrW7hQhg7NnW+dq2vtQf4/PNU+erVMH781mPyhxziqW83b97xfb799s5dJ4VNQV1EJMPcub5nfNKoUb7UrXnzVNlf/uJL5crK/Pzoo335HHgCndgll/iM/VGj0lv9sThr3s03b51wB7zHoKTEE+iIbI+CuohIhqFD4eSTd3zdm2/6JL0hQ9LLW7f2bWG3bIFx41JfEB5+GAYO9ICfbLUvWgQbN/rEvblzt56IF8/knzbNn994I9UjIJKkoC4iUglTp/rWsVdd5eejR/s4e2zOHPj7333C3N/+lsorf+qp3tJu1gwmTkxPenPrrek9AHPmpP/OxYv9+YcfvNU+ZIgvxxPJpNnvIiI/01dfQdOmnhgnHo//5BMP0O3bw7ff+m5z06bBSSd58B8zBu65J/UeXbvCe++lv2+7dj5eX1Li5/vv770CGzb4BL3SUmjRwsft62U0zULwe9myBSZM8C8TnTt7L0Hbtj7LX+oWzX4XEakFzZt7QE/q0MET3PzpT57HfvJkn0wXt+YPOyz9+hNOSD8fP963h917b7j22tT2s3HynAcf9Ocvv/RAn7RggfcEfPSRL8+78krPkR+CL9WraB2+5BcFdRGRajBnjq9bj1vsEyd6S/7ss9OvO+IIH3M/6ywP9kcdlf768OGem/644+DPf/bJeJs2+Uz5PfZI32zm2Wfhggtgv/18u9kZM3xsfu7c1J7zb72VnnHv2299Yt9ZZ/n+9Bs3Vn9dSPbUz/YNiIjkg2HDti7LXCIHHtDXr0+df/cd9OzpGemeeMK73Js08SV0K1fCpEl+3f77+2sff+zL75o0SR9XHzIktS3tvHne7R+LW/fg+fAfeMC742+80XPlv/zyz//cklvUUhcRyaJGjTxv/J13+gY1TZp4ecOGno++QQPPXteunXftg3fZx13pXbp4V3tRkZ/vu693w59+eup3XH996vjJJ/2LQbzmfckSz29fkS1b0je7kdynoC4ikqN69vRJdRMn+vk55/iY+7Rp/hp4EO/aFV5/3YP7Cy/4ennwyXotWnjym969fZ39o4/6mHvSgAHevZ8M7t98418Yxo3b+r4eeQSWLq3+zytVp9nvIiJ10LvvemB/9lkfU0967DGfIAew116wapUvs2vSxCfwJZ1zjnf1l5f7krv4C0RpKdxxhx8nw8Rrr0GPHn48b97WcwKSbrkFZs3ymfwVDUXIzsmZ2e9mNsjM3jGzlWZ2YQWvn2tmb5rZq2a20Mw61eT9iIjki27dvGWdGdAhtTNd376+ph58PX1mkpymTeG663yMv7TUA/+yZT6xLl52V6+el51yiifMeeklLy8q8h6BK67w83g8P+n+++Gpp+CDD9SNX1tqLKibWRFwAzAY6A6MNLPuGZctB/qGEHoAZcCkmrofEZFCUVzsy90ee8yXta1e7dvN7r67d9fHunVLtaAnT4Y2bbzlfeaZHoSvu86DdZ8+Pr6/cKFPqtt1V+9+r1fP95k//nho3BhefNET5DRv7sl54i8As2b5tbW9ec2mTYWXL78mW+r9gJUhhFUhhB+A+4G0+aEhhKdCCP8bnS4GOiIiIlVWUgKtWvlEu3iCHaTG2y+5BM49N1XeooXvPrdhg+9Y94c/eLrcpCee8Il1fft61//kyb4jXVmZB/NJkzzYf/01XHqpr7cHX5YHcNFFNfd5K9Kwoc8VKCQ1GdQ7AInVkayOyralFHisohfM7DQzW2JmS8rLy6vxFkVECst553m3+JVXwoknpr/WvbtPgispgYsvhj33hEGDvAXfpo3nsF++HPr39+v32Sf1s6NH+1r9mTPT33PgQG8xg+9i949/+Hr6detS1zz9tP/uG2+s/OcJIX0dfrIcfAe+QlJjE+XM7HjgiBDC2Oj8JKBfCOE/Krj2ROBs4N9DCN9nvp6kiXIiIrVvxgxPdQu+BK9nT99//pe/9LJVq3xSHnha2lGj4Nhj4ccffXY9+LBAvJHNccd5C/+77zypTnm5t6z/9S8/Hjky/ffPmuU9DplZ8crKYMQInzgY/37wFLqtW/txXR/Pz5WJcquB3RPnHYE1mReZ2aHAxcDQHQV0ERHJjmQ623j2e9u2/tyqlbfq+/Xz83PPhauv9m76/v3h/PO9637jRg+2I0d61rvx4/19y8vhttt8/Puww/wLwbp1/oUAfEhgxAg46CBPjwu+rr+szNfkb96cGr8H/7lkQp3hwwtnV7uazCj3MtDVzPYEPgVGAKOSF5jZvsAtwKAQwvqt30JERHJBgwaecrZevfTlaW++CS1b+nFZmSe2ydw0ZlJiCnTLlj4Z7777fBc78EBeWgrz58Ps2V7Wrp234MeP94l5scsu82Q8U6f6NRs2ePny5R74wX9m8uTUz8ye7cv5ZsxIlU2e7Pf7/PPeezBuHLzzjk8GzMznX5fU6Dp1MzsSuB4oAu4IIVxtZlcAS0IIj5jZ/wD7AGujH/k4hDB0G28HqPtdRKSuW7vWE+MMHuyBNc6it2SJz7yP/4svKfEvEu3bexd6s2bezR5r0iQ1GQ+86/7II73XILm1LfgXjeee8+PFi1NDAs884wl7LowWXS9fDr16VftHrpLKdL/XaO73EMJ8YH5G2WWJ40Nr8veLiEjuadfOk9h06+bj6LE4D/0uu/jGMytW+DK8NWt8tv6nn6YH9Tigd+jgrx11lL/erNnWv/P9970noKjI1+DXr+/d9Hfe6UMDnTp5pr1ly3Yc1J95xocPzj47fRw/FyhNrIiI1Lrf/CY9oCe9/bbPaG/Y0PeDLy6GM87wdLiZ6tf3Fvqll/pa+Suu8I1wMq1b50F/0CC4915/vxEjfP19ebl3zTdr5kE99uij6cv+wHsMhg+HKVM8Y17s3Xe3nUO/NmmXNhERySnt26eOJ0yA007z1vjJJ3ugfeGF1Ou9e/tM/J49fTz/7ru9fPp07wk4/PCt379HD9/WtnVrn70/YIDPqu/VC264wcfwjzkmtU5/7FhfcgeeyOeLL/z4wQfh17/2Wf6/+pUvEfznP6u9OipFLXUREclZxcU+Rg4e2J9/3vPdT5/uLeuBA1PXHnts6njAAJ9oF6sXRbsHHvBu/U6dfEx+yhRPtAMe6AH++tfUTH5IpczdsgVeecWP+/f39Lennuoz98Fz4WebgrqIiNQpBx3ke8kvW+bd7rEjjvC19Hfd5S34eFZ+06Y+6Q583H5bm8sMHQqXX751+bXXenn9+qku97FjU69PmODPjRtX5VNVDwV1ERGpk7p08YAda9TIW9Vjxvh5y5bedb9okbe8d9vNE+NsT58+6edLl/o6/AkTfDx93jyfHPfHP6aW38UT9tauhccfr4YPVgUK6iIikpfMvGXdr59vKbto0Y63gO0bLRzr3t27+nv3Tl93P2yYp7otKvJx947RjiXXXOPPgwenr4evbZooJyIieS/OX78jbdv6uPuBB6Ym7O23Hzz0kM+Wz9xpbv58n6B36KGeLW/6dF/qdvTR6UlzaouCuoiISEK8B30snjRX0fr1ffZJbWwzaZJnpvvgg+wEdFBQFxER2a4DDoDTT0+lod2eDh3St7qtbQrqIiIi21FcDDffnO272DmaKCciIpInFNRFRETyhIK6iIhInlBQFxERyRMK6iIiInlCQV1ERCRPKKiLiIjkCQV1ERGRPKGgLiIikicU1EVERPKEgrqIiEieUFAXERHJEwrqIiIiecJCCNm+h0oxs3Lgo2p8y1bA59X4foVK9Vh1qsOqUx1WD9Vj1VVnHXYKIbTemQvrXFCvbma2JITQN9v3UdepHqtOdVh1qsPqoXqsumzVobrfRURE8oSCuoiISJ5QUIdbs30DeUL1WHWqw6pTHVYP1WPVZaUOC35MXUREJF+opS4iIpInCjqom9kgM3vHzFaa2YXZvp9cZWZ3mNl6M3s9UbabmS0ws/ei5xZRuZnZ1KhOXzWz3tm789xhZrub2VNm9paZvWFm46Jy1WMlmFkjM3vJzF6J6vHyqHxPM3sxqsdZZtYwKi+OzldGr3fO5v3nEjMrMrPlZjYvOlcdVoKZfWhmr5nZCjNbEpVl/e+5YIO6mRUBNwCDge7ASDPrnt27yll3AYMyyi4EFoYQugILo3Pw+uwaPU4Dbqqle8x1PwLnhRBKgP7AWdG/N9Vj5XwPHBxC6An0AgaZWX/gGmBKVI9fAqXR9aXAlyGELsCU6Dpx44C3Eueqw8obGELolVi6lvW/54IN6kA/YGUIYVUI4QfgfmBYlu8pJ4UQngW+yCgeBtwdHd8NHJMovye4xUBzM2tXO3eau0IIa0MIy6Ljb/H/TDugeqyUqD42RKcNokcADgbKovLMeozrtww4xMyslm43Z5lZR+Ao4L+jc0N1WB2y/vdcyEG9A/BJ4nx1VCY7p20IYS14wALaROWq1x2Iui/3BV5E9VhpUbfxCmA9sAB4H/gqhPBjdEmyrn6qx+j1r4GWtXvHOel64AJgS3TeEtVhZQXgSTNbamanRWVZ/3uuXxNvWkdU9E1TSwGqTvW6HWb2C+Ah4JwQwjfbafCoHrchhLAZ6GVmzYHZQElFl0XPqscMZjYEWB9CWGpmv4uLK7hUdbh9B4YQ1phZG2CBmb29nWtrrQ4LuaW+Gtg9cd4RWJOle6mLPou7j6Ln9VG56nUbzKwBHtBnhhAejopVjz9TCOEr4Gl8jkJzM4sbKcm6+qkeo9d3ZeuhpEJzIDDUzD7Ehx0PxlvuqsNKCCGsiZ7X418u+5EDf8+FHNRfBrpGMz4bAiOAR7J8T3XJI8CY6HgMMDdRfnI027M/8HXcHVXIojHI24G3QgjXJV5SPVaCmbWOWuiYWWPgUHx+wlPA76PLMusxrt/fA4tCgSfnCCFcFELoGELojP+/tyiEMBrV4U4zs6Zm1iw+Bg4HXicX/p5DCAX7AI4E3sXH5C7O9v3k6gO4D1gLbMK/cZbiY2oLgfei592iaw1fVfA+8BrQN9v3nwsP4N/w7rZXgRXR40jVY6XrsQewPKrH14HLovK9gJeAlcCDQHFU3ig6Xxm9vle2P0MuPYDfAfNUh5Wut72AV6LHG3H8yIW/Z2WUExERyROF3P0uIiKSVxTURURE8oSCuoiISJ5QUBcREckTCuoiIiJ5QkFdpMCY2eZoZ6n4UW07FJpZZ0vs5icitauQ08SKFKr/CyH0yvZNiEj1U0tdRICf9oe+Jtqv/CUz6xKVdzKzhdE+0AvNbI+ovK2ZzY72Nn/FzA6I3qrIzG4z3+/8ySjzm4jUAgV1kcLTOKP7/YTEa9+EEPoB0/F84ETH94QQegAzgalR+VTgmeB7m/fGM2uB7xl9Qwhhb+Ar4Lga/jwiElFGOZECY2YbQgi/qKD8Q+DgEMKqaPOZdSGElmb2OdAuhLApKl8bQmhlZuVAxxDC94n36AwsCCF0jc7/C2gQQriq5j+ZiKilLiJJYRvH27qmIt8njjejuTsitUZBXUSSTkg8vxAdP4/v5gUwGnguOl4InAlgZkVmtktt3aSIVEzfoEUKT2MzW5E4fzyEEC9rKzazF/Ev/COjsv8E7jCz84Fy4JSofBxwq5mV4i3yM/Hd/EQkSzSmLiLAT2PqfUMIn2f7XkTk51H3u4iISJ5QS11ERCRPqKUuIiKSJxTURURE8oSCuoiISJ5QUBcREckTCuoiIiJ5QkFdREQkT/w/3b3RVDVObGEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot loss of training process\n",
    "fig, ax = plt.subplots(1,1,figsize = (8,6))\n",
    "ax.plot(history.history['loss'],'b',label = 'Training')\n",
    "ax.plot(history.history['val_loss'],'r',label='Validation')\n",
    "ax.legend()\n",
    "ax.set(xlabel='Epoch',ylabel = 'Loss', title= 'Loss')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the accuracy of training and validation data. It reflected similar trend as indicated by the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(0,0.5,'Accuracy'), Text(0.5,0,'Epoch'), Text(0.5,1,'Accuracy')]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAGDCAYAAAAyM4nNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd8VFX6x/HPIYTeO9ISitJ7sYKIHVkVbKyufe29r7quZde1scjP7tpYC9YFy6rYRUDpIBCkJkBC7x0S8vz+OJnMpEGImdxJ8n2/Xvc1t5y588xkMs895557rjMzREREpPSrEHQAIiIiUjyU1EVERMoIJXUREZEyQkldRESkjFBSFxERKSOU1EVERMoIJXUREZEyQkldpIxwzv3gnNvsnKscdCwiEgwldZEywDmXABwHGPCHEnzdiiX1WiJycErqImXDxcAvwBvAJaGVzrmqzrkRzrnlzrmtzrmJzrmqWduOdc5Nds5tcc6tdM5dmrX+B+fclRH7uNQ5NzFi2Zxz1zvnFgOLs9aNytrHNufcDOfccRHl45xz9zrnljrntmdtb+Gce845NyLyTTjnPnXO3RKND0ikPFBSFykbLgbezppOcc41zlr/FNALOBqoB9wFZDrnWgJfAM8ADYHuwOxDeL2zgH5Ax6zlaVn7qAe8A3zgnKuSte02YDhwOlALuBzYBYwGhjvnKgA45xoAg4Axh/LGRSRMSV2klHPOHQu0At43sxnAUuCPWcnycuBmM0szs/1mNtnM9gIXAt+Y2RgzSzezjWZ2KEn9n2a2ycx2A5jZW1n7yDCzEUBl4IisslcC95vZQvPmZJWdCmzFJ3KAC4AfzGzt7/xIRMotJXWR0u8S4Csz25C1/E7WugZAFXySz61FAesLa2XkgnPudufcgqwm/i1A7azXP9hrjQYuypq/CHjzd8QkUu6pk4tIKZZ1fvw8IM45tyZrdWWgDtAU2AO0AebkeupKoG8Bu90JVItYbpJPmezbO2adP78bX+Oeb2aZzrnNgIt4rTbAvHz28xYwzznXDegAjCsgJhEpBNXURUq3s4D9+HPb3bOmDsBP+PPsrwH/cs4dltVh7aisS97eBk50zp3nnKvonKvvnOuetc/ZwFDnXDXnXFvgioPEUBPIANYDFZ1zD+DPnYe8AjzinGvnvK7OufoAZpaKPx//JvBRqDlfRIpGSV2kdLsEeN3MVpjZmtAEPIs/b34PMBefODcBjwMVzGwFvuPa7VnrZwPdsvY5EtgHrMU3j799kBjG4zvdLQKW41sHIpvn/wW8D3wFbANeBapGbB8NdEFN7yK/mzOzg5cSEYkS51x/fDN8gpllBh2PSGmmmrqIBMY5Fw/cDLyihC7y+ympi0ggnHMdgC34Dn1PBxyOSJmg5ncREZEyQjV1ERGRMkJJXUREpIwodYPPNGjQwBISEoIOQ0REpETMmDFjg5k1LEzZUpfUExISmD59etBhiIiIlAjn3PLCllXzu4iISBmhpC4iIlJGKKmLiIiUEaXunHp+0tPTSU1NZc+ePUGHUmZUqVKF5s2bEx8fH3QoIiJSSGUiqaemplKzZk0SEhJwzh38CXJAZsbGjRtJTU0lMTEx6HBERKSQykTz+549e6hfv74SejFxzlG/fn21fIiIlDJlIqkDSujFTJ+niEjpU2aSepA2btxI9+7d6d69O02aNKFZs2bZy/v27SvUPi677DIWLlx4wDLPPfccb799sFtbi4hIeVUmzqkHrX79+syePRuABx98kBo1anDHHXfkKGNmmBkVKuR/HPX6668f9HWuv/763x+siIiUWaqpR9GSJUvo3Lkz11xzDT179mT16tVcddVV9O7dm06dOvHwww9nlz322GOZPXs2GRkZ1KlTh3vuuYdu3bpx1FFHsW7dOgDuv/9+nn766ezy99xzD3379uWII45g8uTJAOzcuZNhw4bRrVs3hg8fTu/evbMPOEREpGwrczX1W26B4s5h3bvD00W823NSUhKvv/46L774IgCPPfYY9erVIyMjg4EDB3LOOefQsWPHHM/ZunUrAwYM4LHHHuO2227jtdde45577smzbzNj6tSpfPLJJzz88MN8+eWXPPPMMzRp0oSPPvqIOXPm0LNnz6IFLiIipY5q6lHWpk0b+vTpk708ZswYevbsSc+ePVmwYAFJSUl5nlO1alVOO+00AHr16kVKSkq++x46dGieMhMnTuSCCy4AoFu3bnTq1KkY342ISPk0fz7s3x90FAdX5mrqRa1RR0v16tWz5xcvXsyoUaOYOnUqderU4aKLLsr3srFKlSplz8fFxZGRkZHvvitXrpynjJkVZ/giIuXeL7/AUUfBv/4Ft96afxkzGDfOz8+bB3feCVWqlFyMIaqpl6Bt27ZRs2ZNatWqxerVqxk/fnyxv8axxx7L+++/D8DcuXPzbQkQEZHC++gj/zh5MqxdC2lp4W2Zmf6U7zvvwNChfnrjDYiom5UoJfUS1LNnTzp27Ejnzp3585//zDHHHFPsr3HjjTeSlpZG165dGTFiBJ07d6Z27drF/joiIkF75x1YurTg7WYwalS4TGZm/vuYOTPnc/75Txg71je3790L773nt334ITRpAsccE97X7bdDjx5w0UXhfVx/PRRwoVP0hS61Ki1Tr169LLekpKQ868qr9PR02717t5mZLVq0yBISEiw9Pb1I+9LnKiKxKjXVDMwuvDC8LinJrF8/s2++8cvLl/sybdqYXXKJWcuWZj/+GC6flmYWF2d2/PF+ee9e/1yf2s1atDC78ko/37lzeD2YffedL1+nTs71xx9vtnlz8b5XYLoVMkeqpl7G7Nixg2OOOYZu3boxbNgwXnrpJSpWLHNdJ0SklFuxAtLT8982Zoxvwt6zx6fKhQthyRI///PPfv6aa3zZL7+E9ev946mnwpQpcN11fh8XXujLLF0Ko0f71xwwAP70J9i6FV5/3dfGf/wR1qzxzzvxRP+cV1+FlSvhlVfgvPPghRcgPh6++QZq14ZHH4WWLWHLFnj7bahRw5f5/nuoUyfqH1/BCpv9Y2VSTb3k6HMVkUjffmv2008Fbx83zuwf/zj4fpYtM4uPNzvhBLMdO8z27/ePn3xi1qNHzppv7drh+bi4nNtCU9u24fkbbsi/TEKC2fbtZvfd5/fTu7dZ9epmHTr47TfdFC4beg//+5/ZHXeY7dnjl/fv94/XXBMue+SRfntm5u/7bA+EQ6ipB56kD3VSUi85+lxFJGTVqnAiy09mplm7dmbOmaWk5F/mX//yyfXGG80qVPATmHXsaFa/fv7J+NxzzV54IdwMHpoqVzZ7/fWcif7WW30cxx+fdz/jxoXjeOQRvy4+3mz+fLM//zlc7osvDv5ZTJniy1555SF/jEVyKEld7bIiIpJtyRJITYXXXvNT6OzdU0+Fy2zdCrVqwdy58Ouvvgm7YkVYvNhvf+QRmDDB9wD/4QdYtQpeegmef95vf+YZGDbMN2v/8Y8Qukjn8MNh0SI/f+utcPLJvkkdfMe0fv1g/HjfYW3CBOjb1zfT//gjPPYY3H23Lzt+PCxf7l9j0CC/LesKYADuu883s1etCh07wrPPQv36/r2ccMLBP6M+fXznuUGDivIJR1lhs3+sTKqplxx9riKxbcuWoj3n00/NrrrK12xD0tPN7rwzZ+12zBjftH3eeWaJieH148ebXX113tpw48Zmxx2Xc93FF5tVrWpWqVLOTmWzZ/vXzcz0z7ngAr+8dq3Z5MkFx792rdnLL4ebu5980u9v0qRD/yxKC9T8LsVBn6tIbJk82ezyy8127TKbMcM3PR/oHHfII4+YXXut2ejRPsGGzk83aRJOjpdc4tf37583WYemu+/2zeunnOKXzznHbOZM3+v8ww99r+9p0/y2888Pl6tRw2zFCv86o0fnPe+emVn0c9I7dvh9RvOcdtCU1EvYgAED7Msvv8yxbuTIkXbttdcW+Jzq1aubmVlaWpoNGzaswP1OmzbtgK89cuRI27lzZ/byaaedZpuL6XqKoD9XkbIoM9PXgItSyx461P9qX3217/AFZgMGmLVvb9a8uU9uEyf65e7d/evceGPBSRrM+vQxu/lmn6xvu82/zvbt/jUqVjR77DGzI46w7Nrw0Uf7+apVfbn8fPaZ2erVZmvWmD36qO9gJ0WnpF7CXnzxRbv00ktzrOvXr59NmDChwOeEkvqBFCapt2rVytavX1+4QA9R0J+rSFmxa1d4fvZs/8s7aFDOMnPmmL3yitkHH/ge10lJZj/8YHbRRT7Brl9v1qtX/om5ZctwsgWzww7zndZCNfJQjRnM1q0z69Yt5/Pj4syqVfNJOCQjI9zre+VKX9vPyPCd3UK1dCkZSuolbMOGDdagQQPbk/UfkJycbC1atLBt27bZCSecYD169LDOnTvbuIjul6GknpycbJ06dTIzs127dtn5559vXbp0sfPOO8/69u2bndSvueYa69Wrl3Xs2NEeeOABMzMbNWqUxcfHW+fOne34rNETIpP8iBEjrFOnTtapUycbOXJk9uu1b9/errzySuvYsaOddNJJtivyFydC0J+rSCz69Vezs84y27nTX4K1ZIlf/+GHZs8+m7Pspk1mp53mf2lDjXnPPx9OpkuW+Jr7V1+F11Ws6B+d84916vjz0Wee6XtrX3NNuKd4fLxl9+zeu9ef0wazp54y27fPnzsPNXvfd58/F21m9uKLZkOG+LK9e/s4ly8v3PtfudKsVi1/uZeUjPKd1G++2bdHFed0880H/dBPP/307KT9z3/+0+644w5LT0+3rVu3mpnZ+vXrrU2bNpaZdeInv6Q+YsQIu+yyy8zMbM6cORYXF5ed1Ddu3GhmZhkZGTZgwACbM2eOmeWtqYeWp0+fbp07d7YdO3bY9u3brWPHjjZz5kxLTk62uLg4mzVrlpmZnXvuufbmm2/m+56U1EXyCl0HPXJkOBHv2BGenzXLX8/888++k1nosq0zz/TnnP/0J79ct65Zly7hjmU1aoT38dBD/tz5G2/4g4fbbw9ve/99fzDw889mGzb4mn3ofPKMGWannuqTdGEkJ+dsRZDYdChJXSPKFZPhw4fz7rvvAvDuu+8yfPhwzIx7772Xrl27cuKJJ5KWlsbatWsL3MeECRO4KGsA4a5du9K1a9fsbe+//z49e/akR48ezJ8//6A3apk4cSJnn3021atXp0aNGgwdOpSffvoJgMTERLp37w4c+NauImXdjh3w2Wc+XRZk48bwvBl8/rmfj7xb15tvhud79PCjnZ1yCmRk+BHGrr8ePv4Y6tb1Zc88049JPncu/PQTtGkDc+b4R/Ajnr36KlxyCVSr5vdXqRIccYS/FKtNGzjySH8Z1jnngHP+eT17whdf+NcpjIQEf1mXlB1l7zr1gO69etZZZ3Hbbbcxc+ZMdu/eTc+ePXnjjTdYv349M2bMID4+noSEhHxvtRrJhf47IyQnJ/PUU08xbdo06taty6WXXnrQ/dgBfqUqR1ywGRcXx+7duw/y7kTKpkcegSee8IkwdD00+ES8fDk0bw4nnQTvv++T58iRsGxZuNx99/nrr8eM8ctVqvihTf/9bz9U6I8/+qFEmzf3Q5lOmOAPDG67DXr18sl61y749lto1crf4Wv8eEhMzBln27b+2vF69SAuLvqfi5ReqqkXkxo1anD88cdz+eWXM3z4cAC2bt1Ko0aNiI+P5/vvv2f58uUH3Ef//v15++23AZg3bx6//vor4G/ZWr16dWrXrs3atWv54osvsp9Ts2ZNtm/fnu++xo0bx65du9i5cydjx47luOOOK663KxKzzHzNOOtfKXvda6/BxIl+eeVK2Lw5fPeuESNg+3Zf+1682A9actllPqGD399NN/k7cg0e7A8GunSBe+/1NeYJE3y5H3+EJ5/085dc4hM6QOvWfrCSFSv8a/fvD9Wr+7HJBw70CR38ICkzZuT/vho2VEKXgyt7NfUADR8+nKFDh2Y3w1944YUMGTKE3r170717d9q3b3/A51977bVcdtlldO3ale7du9O3b18AunXrRo8ePejUqROtW7fOccvWq666itNOO42mTZvy/fffZ6/v2bMnl156afY+rrzySnr06KGmdikz9u/3N9OoXz/n+iVL/Mhlzz/vb77x6KN+lLIrrvDbb73V17j79vUjo4Ev16WLr52/845P+JUqwb590K2bX//ss75m/dxz/raa99/vn9uvn2/CB+jUyY9QtmsX3HJL3pjj43Muv/RSuOkc/H4Du2WnlAnuQM20sah37942ffr0HOsWLFhAhw4dAoqo7NLnKkHYt883d593HvznPz7phRLy/v2+tty2rT/X/eCDsHYtNGoEGzb4puuNG+Hmm30te+JEf066cePwUKS5Pfgg1Kzp95uQAKHj3kWLYPJkOOMMn/xXrYJjj837/Llzfax//asf8lSkuDnnZphZ78KUVU1dRAK3YoUfI3zgQN/U/emnsHNn+PaaO3fC/Pm+RrxqVc7nNm7sxxJ/5RXf2Qx8s/f48bBggR/LOynJn//+979h3Tp46y0YOxY++sg3hQ8c6Kc2bXzHtYULoV07P4FvDWjdOv/Yu3TxryMSC1RTlwLpc5Xi9PLL0KCB7wyWW506vjbcuLGveYNvHp86NVymUiW/btAg3wS+YUPe/Vx+OWzaBEOG+HnwHcw2boSuXX2HuCee8PferlLF30O7SZPif68ixUk1dREJxHff+U5fM2bArFm+Vn3llb6T2ezZvsyiRf4c9Rdf+Gb0WrXC57bXrvU1606dfO0bfLP244/Dv/7la/EAv/zia+KXX+4v8frjH33v9FdfzRtT8+Z+Ajj9dD+FKKFLWRPVpO6cOxUYBcQBr5jZY7m2twJeAxoCm4CLzCy1KK9lZvleDiZFU9pacKRkZWb6RH3BBZCeDuPGwUUX5X8ryvfeg717w8uHH37gfT/4oE+2oaR+9dVw111Qo0a4TIMG/rF7dxg+3Nf0s/qEipRrUWt+d87FAYuAk4BUYBow3MySIsp8AHxmZqOdcycAl5nZnw603/ya35OTk6lZsyb169dXYi8GZsbGjRvZvn07ibkvmBXBn3fu398n1w4d/PKBtGnja9ZHHOE7u+V27LGwejUkJ8O2bf5yr8qVfae5zMycPcQBpkzxl5ItXuxr+yJlWaw0v/cFlpjZsqyg3gXOBCL7oHYEQuMyfQ+MK8oLNW/enNTUVNavX/87wpVIVapUoXmozVLKve3b/aVb69b5WvOSJX79hg2QNZxCvtq184n36KN9Yl+71p/TDk1NmsDFF/um83nzoGlTn9DBN9GvWZM3oYOPRY1JInlFM6k3A1ZGLKcC/XKVmQMMwzfRnw3UdM7VN7ONHIL4+HjVKEV+JzOfpBs2zLl+9Wo/OMuCBb7p/O67oWJFfxnZunX+fPhVV/mOcOATfr9+vnPaH/8IDz3kO6mB70V+223+PPqtt/r9dOkCnTv718/MDL9ukyY65y1yqKI5zEF+7eC5j63vAAY452YBA4A0ICPPjpy7yjk33Tk3XbVxkeJn5nulN2rkR0e74gr45BN/GdnRR/ukfsQR/pKx5s198v3uO99bHeC66/zoaO+952vk3br5oU5D59j79Am/VuPG/vKyillViu7d/Xx8vG9yF5Gii+Y59aOAB83slKzlvwCY2T8LKF8D+M3MDtjmm985dRE5NL/+Ci1a+Ou3jznGj6gWGhI1P2ef7WvWxx3nx0VfutT3at+8Gb76yg++EtlMPneub7I/+mg/36VL9N+TSFkVK+fUpwHtnHOJ+Br4BUCO8Zaccw2ATWaWCfwF3xNeRKJo/nxfkw757Tc/EEu1ar6X+v798MILfoCWwYPh3HNzdkYLDdQC/m5g55+f9zUik7gSukjJiVpSN7MM59wNwHj8JW2vmdl859zD+HvDfgIcD/zTOWfABOD6aMUjIl7oUrGQ0C0Jzj7bD686aZIfyS00mpuIlB5lYkQ5kfJo9mx/yVfo+uwlS2D3bt/pbOHCcLJes8bXqJ97Dh5+2Hdsu/hiP+ra2LH+PHj37v68+YABwb0fEcnfoTS/635AIqWIGbz7rh8L/eqrwzcQ+fRT36R+xhlwww3+2vELLoCePf1lYrfe6m8XunUrPPCAvzvYOefA6NF+3fTpSuhFtmWL7+Kfnh50JCKqqYvEmg0b/OAqgwfn3fb9934Y1XPOgRP+ez0NMtcR99EHnHMOVK3qb/l5IFOmaOS1YnfVVf5OMf/9rz+HIWXbihWwZ8/Bh0YsRqqpi5Rip5zia9xbtuTd9tVX/vHDD+HazOc5lw+56ip/s5MxY8LljjoKmrOSbx+eyCPXpFGZPRx2WAwn9ND9Tkuj5cv9Y35D5eW2ZYu/ZEDCli/3g/kfyEsv+ZsAFKe1a/2tAQ/V+ef7I+4YrRArqYvEADN/udiIETBzpl83f76/Icrgwb6Hupm/u1juwWEyNm6hf/+czefjx8NvzU7khAeO4543O/I4d3PaaRFPWrTIXzA+d27OgdkPxSuv+Nrp7/lxy8iAN97wF7VPnnzozx83zncWKA4ZGXnfi5m/O82B3uO2bf5xY9aYWevXF1z+iCOgXr1Di2vaND8O76WX+i/C7Nl+kADwNcZJkw5tf7Hmzjvh1FMLPigy8702//73nOt37gzfCQj8XX8efji8fOutcPLJBb/uI4/4uwFt3174WJOT/QHIkiV+NKapU2MvuZtZqZp69eplImXN4sVmYNaqlX/Mb6pZ0z8+9pjZuP/uz94woNJk++ADv5/O/GpHN1/uFyKevKdGPdu5aU/4BR96KLy9QwezNWvMhg0zGzfO7N//NnvvPR9UQTZvDj//llvMHnzQ7PjjzZ577tDeeOQbfPHFg5fPzPSP331nVr16+Lm7dvltq1aZTZt2aDGE9gtm112Xc/2oUX79V1/55ZQUsz17cpZp396Xeeghs7Vr/fxdd/ltM2aYLViQ9/1u2lT42CI/o+uuC8+vWWN24YV+PjU17/Patzc79VSzHj3MfvnFbO5cs+XLzZo3N5s0yX9mTz+dM77QZzFlSvizNjPbu9ds1qyc5f7v//zfYNeu/OP++Wez3r39Z3IgLVr49zBnjtmECWa1apn95z/h7b/+Gn7P6enh9Sec4Ne9/XbO72NGRs7PLdKECWY//GB21VVmder47YmJ/jv//PP+fR7I44/n/cf85puc+2/UyCwt7cD7OUT4K8YKlSMDT9KHOimpS2n3228+D37ySXjdSy/5/0bHfjuKSQaZBmb9+NmOZULWcqYt4Ajb+NAzZuvWZf+o7H/iKbPbb/eJIvRDc9pp4fkqVfzjf//rfxT37DG76aacP0z5HU0MGuSDW73abMmScLA//GB2ww35H3n07evLF2TuXLNt2/z87t05n3vvvT4pb9nif5j37cv53H/9y6x2bbMdO8x69cr72g88YHb00X7+++/zPn/lSrMbbzR79tm8ca1enX8SaNfOr3vuObOtW82qVfPJ++GHza64wiw5Ofy86683e/318HLoSC1yn5HxjhoVXr91q0/Yv/3m4774YrMvvjDbvz//zxnMTj45PD9jht9PaqrZlVeajR+f/3NeeME/Nmhg1rOnnx8+3H+mIR984Ne3b+9fIzPTf/Zg9vXXfvndd8P7fPdds4kT/fzcueH9DBsWLnP88f4gJGT37vBBWKjM88+b3X9/eDl0sPDUU+F1oYOXzEyzihX9utatc77H2bPDB2ngE/7o0T5516hh5lzBn+kTT/j3+txzZv37+8/rmWf8Pq67zqxJE3+Q9Ic/hJ/zyCP+9V5/PXyg8de/5v2O/Q5K6iIx7Oyz/X9ez55mO3eaff1Kir3R8XE7yv1sf2K0GdiNjLI+TMn+4cg88ihLoWX+P0SVKvnHyB/R0HTXXWYzZ5o1buynli3N+vXzCTtUpkuX/Pfbvbv/serTxyw+3mzECLN583KWqV8//+cuXJjzTf/vf2bXXBNOImZmS5fmfM6ZZ/rHOnV8DbBhQ7NLLvEf2KxZPobQgQOYDRxY8I8zmN13n08eycl+vk2b8LZFi3wMGRlmr7yS87MLmTo1vO7WW83Gjj3w651zjtnQoXn/LuA/x8zMvAll+nT/WmPGhNf16+cf69XLedAQmtq08Ykn9HmAP1j5/nuzJ588cIyhg578ppkz/QFP6PMNTZMn+/cPZgMG+OXI7SedFP5S165tdtxx/j116hQuU7myPwgy8weWHTuaHXus2Tvv+O3O+c/unHPCz/n+e18+snViyhS/LjU1b/xDhvjHc881u/TS8PqPPz7wZxI5NWmS//q2bcPz99/v/5bz5vn1Q4b4A5Lc5ffvL7bfDCV1kaBs2uT/2SNt3GiWlpbdqhnKg5Ur+0rVC1xtBramXnubffi5ZmAbqWtT6GP7qtS0zFCNJL8pVAsvaAo1rd5yS95tl1zik15qqq9J9enj13/zjW+ejCwbqrHmniJrVpHTk0/6mujIkb7JumHDnD+cZr6psrA/tr17m1WtmvNHMyMjZ204NF18cXjeOf+Bx8X5o6hXXvHrn3nGJ5cLLsj7/A8+MHvtNX9aonFjX8MbPNjs2mvzj+2ss/wBUOvWPtHecEPeZtrLLgsfxNx5p9myZX6+Zk0f7113+eXzz8/ZavLgg3lf75RT/OcX2RoTmnK/n6eeynkwA/7v/J//+KQZuf8//jFnuWrVfM22ShV/MJjfe+/ePe8653xTeuj78dtvZuedZ9a0qU+GkQcwVav6A4HQ97NuXX/gENo+ZkzOWnHjxv6f6Jtv/HLnzv6xQwe/79wHJJB/q05Cgv+s33vPH2wU9L077LDwfOj/8Jdfwv/bF1/sm9ubNcv5vL/9zf9vFRMldZFo2r3b1way/rnfecfsyy+ztmXVhD77cHd2ZTDzlFMss107m/yyPzfYmV9t0CCzCmQYZNov5Poh6tAhPP/sszmbH3NPf/1rzuURI/yRwqef+ibH0HnR1FSfbFau9LWk0A9+pM2bzd54wz8ndD4AfFPttm1mxxzjl0M1avCJcfZsn/ROP93s7rsLjjV07hR883Vof08+6WuakWUHDjT78Ud/UPD00+H1o0f7hBxqbk56nNwuAAAgAElEQVRPD28LJcOVK/3+I5PM5Mnh99m2rY/1kUf89lANL/cUF+c/x3PP9T/cjRqFD6Iia4Lp6T5phRJhWpo/5597f6HWgDfe8HFE1grbtfPNupF/i8iaeOR0/fW+zGOP5d3Wrp2vNT/zjK/Nm4UTZqi2GaoxmxXc+nDccWYbNoSTJ/iEGXlweP75/rRFQkL++6hVK3zaJnRaYtYsX7Nv29Z/T8F/Z7Zu9QkbzO65J2fNGMKnCkJT6L2Hvv8DBvjX2bPH9xeIbCUJvZ+XX/Yx/+EP/pRB5P/zmWf6g4VRo/x3Gfzf3czsww/N3nrL17xD37uQF1/M+77/8IcD/HgUjZK6SDHKzPS/kdl9m0b7JnI79VTLzPS/Re3a+Ypa6B/7eL4zMMtYucr245tcJ1c53gzsQR6wz8Zss5U0s/sqPmb7q1S1zMgaxrff+h/4OnXMtm/3PyYpKeHtoc5R4M+tn3mmT9BDhxaudrBrl/+hCp3bzk+oFt29e/jc9KxZ/sc2Kclv69gxXH737nBntVCN+vzz/Q8p+ObvffvMPvoo74/gtm0+Ef/97z7JJCb6I6WQHTt8s+zhh5utX5831qZNzc44wyfX0DnXn34Kx5D7h/juu80qVPBJe/jwnAcGoenrr31HL7OcrRE//eSXN240e/NN/2NvFu5j8OCDfjn0GeU3ffaZL9O8ec71kcnWLFxjTUzMWe4f//Db9+zJ/4Dv73/PuZ/Fi/2B09atvpVg/vzwti1b/IHos8+Gn9++vT+gCjnqKL9+6FCfMEPlIv8+ixblTcTvvRcus2aN/7xvusl/P26+2X93r77aP5qFW1Heesu3MP3yS/i7lF9tunbt8EHJsGE53/OqVf71Q2VDB1KFkZ7uOxj+738HL5uWFn6NUB+Ha68t/GsVkpK6yMFE9qLdv9/XrEK12s2bc5wPm/PdBpvEUTaqy799Tbh/f/+vM2SIpaaavc4l9iYX2hjOz/4Hf4T7DMwe4MGCf9wjp1DyAx/H4sV5e3F/9ZX/UTXzNdFGjaL3+ezb52tlKSn5b9+0yR9w5CfUZPnoo345smf2jh3+B3rEiLzJoahC56tzr/v44/wPctat883eLVqEj9Qef9yfggDfMhFp8WKzP//5wInhgw/Mjjwy3OEssjf2vff654aWQwcL337rm7xvusk3c4cuYQh59VXLPshr1Cj8/JEjc5bL/V2K7IFZWJEdy3ILJdSbbvL/Fy1b+r9fbrmb6HN3VIw8X/7xx3mfv3+/v/oi8uqCbt18+cirNT77zLfefP65P5C85BLfqz8/Q4b4g97c34/iFIor1J9BSV1JXUrY55/7r36oxvLhh3551Cj/QxSqDZuZTZli67vm6pAVF+cfjzjCPvlwb4GJejq+yXB532GW2uesAyf1uXN9G/6BLiMrLUKXCo0bV3CZjIziS+pFMXt23gOWH3/08fTu/fv3n5npO02AP78cutQNCv83zsw0W7HCzycl+Uu7zjwz7+Vwub9Luft0FNbEib6FIre//c3vN3SZXkH+8hdf7m9/860YuUV2PNy8uXAxnXiiL//KK74jX0EHmQXJ74CvuM2c6Q+kQn0Firnnu5mSusiBXX65/+o//LBfjjw3GdoGNu3LDTlrSBFTWryvlXze/1EzsO1Ut4n48+nnE+4ItO+Kq802brQ9aRtsdsOIHucJCf5ymRtu8NfZliWhDkWhTgUFefhhf147Vuza5Ztxf/utePYXOscfusTvUBNaYeX+fu7cWbz7D10Gd889By6XkXHwa9LXrPEHVIV1flbr1+uvF/45QcnI8JfCFXTd/u+gpC5yIKHLjgYO9MuXXZZv4r4Ff23u+rptbRBf26cMzt72FLflKHsYqQZm1dluYHYS4/OvkYV6lRdX4ohF//ynf4+RpzjKoyOPtBzN0JGnV4rT1Vfn/O4Wt927/bn4Qxkwp7gsXer/TzduLPnXjiGHktQ1TKyUP7/+CkDmjxNY/rdX/YDqAweS8dhTOYrdwLMAvNTnVRa3PJFtb3/GKpqSQRx9/30VKTU6s7tqXaxZcy75SzMuvxx2UoN33oEnZ58Mbdvmfe1//Qt+/NEPF1pW3XOPTy8VKwYdSbCaNPH3vI2P98tffgm33w7OFe/rvPiiH7Y0WqpUgSee8O+lpLVu7YfEPdShdcsx3aVNyryMDEhKgq5dgR07oFYtPku4njOSn80uM77RRfzS8Qr+9sNAAGbE9aXX/qnsI55OzbbSJLEqP/0EqX3OosraFTRYkTVA+759fgzqunXZtcsfH5x1VgBvUmLP22/DrFnw1FMHL/t77dgBNWv6+VL2my4Hp7u0iUR48sLZTOt2BUuS9vkbY5jxfMpgXjj1YyY08LfKHLNuEM//0CH7OWuqJgLwEcNYklaVypX9+ub/e5kGkz8J77xSpewaTLVqSugS4cILSyahA1Sv7h8feKBkXk9iVjlvH5Oy7KvxRub4r/nL+6cAkDp0OSz8ln3x1fg+YyAvvFiZNWv+QL0jN7GFOhjhZtF3Mi9gMO/R6pUHSPg7DBuWtaFRowDeichBOKcaugCqqUsZMHEiXHwxfPtteN3evbDu9Es5deQp2euaL/QFptObk4dUplUr6NcPPvmpHjfdXAFwbBl+LbfzFH3+cRbs28fRV3QgORmuvbaE35SISBHonLqUWpmZvoLSty9Mnw4NGsDq1VAxzvjqnQ2ccFFTkvpeCiOfpusx/nzj0tYnctKyl3jhy9acEs73bN3qb4180knBvBcRkYLonLqUeevWQae2eznvXGP6dDj5ZNiwAX5+exk89xwnX9SIiuynwz/+RNeja2Q/78RlL2MJrfMk79q1ldBFpPTTOXWJfZ99BikpcMMN2aseuGkLvyY35O3kC/lTld0c+9K/ObHzGo67NOelYvH9j/Izt9xC5gsv0fbYBK64EirocFZEyiAldYlp330HJwwZ4hfi41nf8xTGv7+Vjp+9RzwZXMpo2AN8dwqPtpwHC3zRzxhMszsvpEelSn7FyJFUGDmSrwN5FyIiJUPn1CV4773nL8k544w8mxo1gnXr8w7WsYPq1GBneEX//uybOZepOzqymHa83mkE38ysRyini4iUVodyTl01dQneBRf4xwIOMDdSj/psyrGuBjtZfuIVtJrxX+jWDX74gUrAjLP+Tq0/HM93f9KAZiJS/ujMosSM5x9aD/jRNMeOhfXrYdP6DOqymZ85ko3kHCoy/szTYdMmf869Th1ITOTm/w7gssuU0EWkfNJPnwRr797s2W8fnEDjzsM45xyoyi6aJVamEeuogPG/ehez0mZz3uaXs8s3GtTFz1SvDmlpfsjW4h5XW0SkFFFSl2CtWZM9+xB/49hzBgG1mUlPMpMrsOfy6+E1+PurTdmftA3u82XT46sSf3jr8H6qVfOTiEg5pqQuwcpK6k9zM7cwiut4nl84kvYsJDOuIhVey7qMrWlT4jIy/PxttxF/7LEQFxdQ0CIisUlJXQKTmgrTX1zDWcD09n+CzC94dNF92dsrfPoJnH66X2jaFFq0gM6d/Zit+d3WVESknFNHOQnG6NE8fUsKX7zha+pHDW0KvXNdsTFgAFxzjZ9v0gQOOwzmzlVCFxEpgK5TlxKTmQm7dsF/HlvFdf9olmPbnm37qLIsCW6+GX780a8089Pu3TpfLiLllsZ+l5h0//1QsyaM+8c8AHZTJXtblZrx2debM2cOzJjhNzinhC4iUkg6py4l5v/+zz92xif15+9ewS2XbCZuTVrOgl27lnBkIiJlg5K6RFXo7I5zUKsW7Nzpk3pGg8bc/lhDoCF0ODzQGEVEygoldYmK9HRY8d7PTH5yIoevnsCajgOpsvosZnM23fgV6zQg6BBFRMocJXUpdunpMHgwfPX10bQJrfzxM1axmPZxi1na9yLa3HhWkCGKiJRJSupS7M46C77/Oj3P+mt5kb0DT6fN128GEJWISNmn3u9SrLZsgc8/h1EXTst3e+WT1OwuIhItSupSbMaOhauu2M8OqnPZF+fm3Pjww/4xn3umi4hI8dDgM1Is9u6FxESotfo3fqODX9m2rb97WmjUmV27oEaNYAMVESllNPiMlJhtK7ZwZpMpWLXq1Fi9iG7MCW888kho3RoSEqBCBSV0EZEoU0c5+V1qJDbg48z9AJxf43Oq7Vgb3tivH3TvDqG7q4mISFQpqUvR7d1LhayEDnDeSZvZOvYnfuMI2t87DC64ABo0CDBAEZHyRUldimzn/BSqRyx3Ges7wz3d9HHa/+OuYIISESnHdE5dimzm+0sAWHjnK9CrFwD7TzqVa5OV0EVEgqCkLkWyZw9Mf28pAG1vHQL16gEQd+JAKlcOMjIRkfJLSV2K5LrrIC5lCelVaxLXpCFs3eo3HHVUsIGJiJRjOqcuh2brVn6YWo3XX49nSWIS8fUO97dge/55GDnSX8YmIiKBUE1dCs+MTc06k3T+Q1SpmEHr9VPCSbxXL3jrLYiPDzZGEZFyTEldCm3Zt8nU25nKkM2jmVq1P27HDjj66KDDEhGRLGp+l0Kb+59ZtAZakEqL7al+pZK6iEjMUFKXA1q9Gt664RfSe/Ql7p2ZnBm58Y47oFWroEITEZFclNSlQGZww4C5fLT4KIb89xPurj2H9BqtiK9ZFZ55Bk48MegQRUQkgpK6FGjdOti3OAWADiyga8NVxB/RGT77LNjAREQkX+ooJwVKToYmrAHgxtOXUitjc/YgMyIiEnuU1KVAycnQlNUAtNi3DDZtUlIXEYlhUU3qzrlTnXMLnXNLnHP35LO9pXPue+fcLOfcr86506MZjxTSTz/B2WeTsiQju6bOokWwbRvUrRtsbCIiUqConVN3zsUBzwEnAanANOfcJ2aWFFHsfuB9M3vBOdcR+BxIiFZMUkj9+wOwtXIqXSuvgb3AihV+m2rqIiIxK5o19b7AEjNbZmb7gHch5xVRgAG1suZrA6uiGI8Uxqrwn2D97FRaxq/OuV1JXUQkZkUzqTcDVkYsp2ati/QgcJFzLhVfS78xivFIIaT/76vs+Z0LU2nq1kCLFuECSuoiIjErmknd5bPOci0PB94ws+bA6cCbzrk8MTnnrnLOTXfOTV+/fn0UQpWvv4YPP4S09yeRQRwALVlB/fQ1OUeNU1IXEYlZ0UzqqUBEFY/m5G1evwJ4H8DMfgaqAA1y78jMXjaz3mbWu2HDhlEKt3z7+9/h3nuhysxJfF3hVKxWLR7tPRa3Zw+cdhpUquQLKqmLiMSsaCb1aUA751yic64ScAHwSa4yK4BBAM65Dvikrqp4AFJSYEfqFppsWsCqhKNxzZtTcfovEBcHQ4ZAYqIvqN7vIiIxK2pJ3cwygBuA8cACfC/3+c65h51zf8gqdjvwZ+fcHGAMcKmZ5W6ilyhLT4fUVGi9ex4AVY/qDlWr+o39+/vaeevWfrlOnYCiFBGRg4nqMLFm9jm+A1zkugci5pOAY6IZgxxcaipkZkIn5gPQaGAn6Hoe7NoFL7zgC3XpAr/+ChU1srCISKzSiHJCSop/7MR8tlODxAEt4a67ICkJjjjCb3zgAfj558BiFBGRg1O1S7KTemfmMZ9O9EnM58KF6tX9JCIiMUs19XIsIwNuugmefBKqVDa6MJd5dCYuLujIRESkKJTUy7F58/xt0Vcs2MFn/7eMhmxgVsW+QYclIiJFpKRejq1aBSfwLTuoyaDxdwHwwOdHBhyViIgUlZJ6OZaWBoP41i/8978AND6hU4ARiYjI76GOcuXYqlXQjSSsZk3cqadCq1bohLqISOmlpF6OpaXBZRVm4wYPhjFjgg5HRER+JzW/l2M7UjbQMnM5dO8edCgiIlIMlNTLqW++ge5fP0EmDk49NehwRESkGKj5vRz6+GMYeraxmRf5peX5HN2tW9AhiYhIMVBNvRwaMQJ6ttlKLbbTdEjvoMMREZFioqRezqxbB5MmwcUnrQYg8ZjDAo5IRESKi5J6ObJvH1x6KbjMDE5vv8yvPExJXUSkrNA59XJk0iRY/MVitlfpR9WbN/uVTZsGG5SIiBQb1dTLkeRkOI/3qbpnc3ilkrqISJmhpF6OJCdDJ5JyrqxZM5hgRESk2CmplyMpKdC10gI4/vigQxERkShQUi9HUpZl0jbjN+jRI+hQREQkCtRRrhxITYXNmyF9yXKqZO6GDh1gxQqooGM6EZGyREm9HLjmGpgyBfpvmOlXdO0KLVoEG5SIiBQ7JfUybs8e+O472L0b+jGFzPhKVNANXEREyiS1v5Zxkyb5hA5wVIUp/nx65crBBiUiIlGhmnoZN2UKXMvz9K2RRJ/d06lw5JVBhyQiIlGipF7GrVoFz3M97Mha0alToPGIiEj0qPm9jNu2fHPOFYmJwQQiIiJRp6RexjVeMinnitatgwlERESiTkm9jGuxZmp4oUIFaNkyuGBERCSqlNTLMDNovm0BW6tn3bSlRQuIjw82KBERiRol9TJs2zY4PHMBG1r1hkaNdD5dRKSMU+/3Mmz1ygzasZglrQfDxcdoFDkRkTLuoEndOXcD8LaZbT5YWYktm2cm0559uA4d4O5Lgw5HRESirDDN702Aac65951zpzrnXLSDkuKxasISAJoc1y7gSEREpCQcNKmb2f1AO+BV4FJgsXPuUedcmyjHJr/TujmrAajXpVnAkYiISEkoVEc5MzNgTdaUAdQFPnTOPRHF2KSItm+HIUMgdbpP6jRpEmxAIiJSIg6a1J1zNznnZgBPAJOALmZ2LdALGBbl+KQIbr4ZPvsMmrCGPVXrQJUqQYckIiIloDC93xsAQ81seeRKM8t0zp0RnbDk95gwAYYNg4u3raHyStXSRUTKi8I0v38ObAotOOdqOuf6AZjZgmgFJkWzcycsWwZdu0LtXatxTZsGHZKIiJSQwiT1Fwjf4wtgZ9Y6iUFJSX4kuS5dgDVrdD5dRKQcKUxSd1kd5QDf7I4GrYlZ8+b5x86d8UldNXURkXKjMEl9WVZnufis6WZgWbQDk6JJSoLKlaF1/a2+LV41dRGRcqMwSf0a4GggDUgF+gFXRTMoKbply/zdVeMmTfArevQINiARESkxB21GN7N1wAUlEIsUg2XLsu7b8umnULMm9O8fdEgiIlJCCjP2exXgCqATkH3Bs5ldHsW4pAjMIHXpXs7ots4n9VNOgUqVgg5LRERKSGGa39/Ej/9+CvAj0BzYHs2gpGg2b4Y7t/+VR0a39J3k/vjHoEMSEZESVJik3tbM/grsNLPRwGCgS3TDkqJYtgxO4uvwisGDgwtGRERKXGGSenrW4xbnXGegNpAQtYikSMzgtdegHptIb9IcvvpKTe8iIuVMYZL6y865usD9wCdAEvB4VKOSQzZpErz1wjZasYL4m66Dk04KOiQRESlhB+wo55yrAGwzs83ABKB1iUQlh+yNN6BP1fmwm6yRZ0REpLw5YE09a/S4G0ooFikiM/jwQzi3V9aYQIcfHmxAIiISiMI0v3/tnLvDOdfCOVcvNEU9Mim09eth61bo0Djrvjv19OcRESmPCjOGe+h69Osj1hlqio8ZKcnGfDqR+MM2v6JOnWADEhGRQBRmRLnEkghEim7V3I2cxQLYCNSoAfHxQYckIiIBKMyIchfnt97M/lP84UhR7Jy5MLxQt25wgYiISKAK0/zeJ2K+CjAImAkoqceI/QuU1EVEpHDN7zdGLjvnauOHjpUYUXV5RFLX+XQRkXKrML3fc9sFtCvuQKTo6q5XTV1ERAp3Tv1TfG938AcBHYH3oxmUFJ4ZNNm5NLyiRo3gghERkUAV5pz6UxHzGcByM0stzM6dc6cCo4A44BUzeyzX9pHAwKzFakAjM1P78SFYt9ZoZSlkVKxMxYy9sH9/0CGJiEhACpPUVwCrzWwPgHOuqnMuwcxSDvQk51wc8BxwEpAKTHPOfWJmSaEyZnZrRPkbgR6H/hbKt9RfN9GLHWxu04+6C6dARkbQIYmISEAKc079AyAzYnl/1rqD6QssMbNlZrYPeBc48wDlhwNjCrFfibBhegoArl1WNwcldRGRcqswSb1iVlIGIGu+MPf0bAasjFhOzVqXh3OuFZAIfFfA9qucc9Odc9PXr19fiJcuP3YlpQBQZfAgv+Koo4ILRkREAlWYpL7eOfeH0IJz7kxgQyGe5/JZZ/msA7gA+NDM8j0hbGYvm1lvM+vdsGHDQrx0+bFv8XIAqpx/JixYAHfcEXBEIiISlMKcU78GeNs592zWciqQ7yhzuaQCLSKWmwOrCih7ATnHlpdCyliSwq64mlSrU0eXs4mIlHOFGXxmKXCkc64G4MxseyH3PQ1o55xLBNLwifuPuQs5544A6gI/FzpqAWDLFqi0aTW7GhxGNZdfw4iIiJQnB21+d8496pyrY2Y7zGy7c66uc+7vB3uemWXg78U+HlgAvG9m851zD0c25+M7yL1rZgU1zUsuZnD99TBqFDRiHXGHNQ46JBERiQHuYLnUOTfLzHrkWjfTzHpGNbIC9O7d26ZPnx7ES8eM9euhUSM/v9C1p/VZXan4X40HJCJSFjnnZphZ78KULUxHuTjnXOWInVcFKh+gvETZ6tXh+eaV1lKxmWrqIiJSuI5ybwHfOudez1q+DBgdvZDkYFavMh7hr8xpNphqaVvC1XYRESnXCtNR7gnn3K/AifjL1L4EWkU7MCnYxiWbuZ9/QNo//IrGqqmLiEjh79K2Bj+q3DD8/dQXRC0iOagdS9fmXKGkLiIiHKCm7pw7HH8Z2nBgI/AevmPdwIKeIyVjz3IldRERyetAze+/AT8BQ8xsCYBz7tYDlJcSsn9VrqSuUfZERIQDJ/Vh+Jr69865L/E3ZNEIJzHArc9K6nPmwPjx0Lp1sAGJiEhMKDCpm9lYYKxzrjpwFnAr0Ng59wIw1sy+KqEYJZf4jWvZ7+KI69wZunYNOhwREYkRB+0oZ2Y7zextMzsDP377bOCeqEcm+crIgCrb1rKrWkOoUNh+jiIiUh4cUlYws01m9pKZnRCtgOTAtv31Sa6wV0mvo/PoIiKSk6p6pUy9x+7yj2lzA45ERERijZJ6KbO7RgMANt35z4AjERGRWKOkXpps3UrVHRu4xz1OrUfVrUFERHJSUi9NFi8GYHODdlQszKj9IiJSriiplyaLFvnHww8PNg4REYlJqu+VIpa0gEwqULN7m6BDERGRGKSkXors+XkWS+lA285Vgg5FRERikJrfS5EKs2cyk560bx90JCIiEouU1EuL1aupvGk1M+nJEUcEHYyIiMQiJfXSYtYsAJIq96RJk4BjERGRmKSkXlrMnAnAppbdcbpXnoiI5ENJvbSYOZMVldtRP7FW0JGIiEiMUlIvDcaOhbFjmUlPWrYMOhgREYlVSuqlwdChAEza20tJXURECqSkHuu2bwdgT4fuvMC1tGoVcDwiIhKzlNRj3apVACw84w52UkM1dRERKZCSeqzLSuo/LT2MypWhd++A4xERkZilpB7r0tIA+OiXZpx0EtSoEXA8IiISs5TUY11WTX3aqsMYPDjgWEREJKYpqce6tDTSq9ViJzXo2DHoYEREJJYpqce6VavYXvMwANq2DTgWERGJaUrqsW7FCtZVbkG1atC0adDBiIhILFNSj3XLlpFMIm3bojHfRUTkgJTUY9mOHbBhA/N3JdKuXdDBiIhIrFNSj2XJyQDM2JioTnIiInJQSuqxLCupL7VE+vULOBYREYl5SuqxLCupJ5NInz4BxyIiIjGvYtABSAFSUmDMGLZUbkT1xg1o1CjogEREJNYpqcei5GRo3RqAZ5o+T5eu6vYuIiIHp+b3WDRvHgB22eU8tukqDj884HhERKRUUFKPRUuXArDq5sfZtTdOl7OJiEihKKnHoqVLoVYtFm6oD6CauoiIFIqSeixauhTatGHRYn8uXTV1EREpDCX1WJSV1OfNg+rVoVmzoAMSEZHSQEk91mRm+svZWrfmm2+gf3+ooL+SiIgUgtJFrNmwAfbtY1O15ixcCCefHHRAIiJSWiipx5pVqwD4daNvcx80KMhgRESkNFFSjzVpaQAkbTmMihWhffuA4xERkVJDST3WZNXUZ65tRrt2EB8fcDwiIlJqKKnHmrQ0cI4py5uoli4iIodEST2WbNwIP/yANWrEb0vjldRFROSQ6IYusWLpUjjuOFi9GgdkAB06BB2UiIiUJqqpx4px42D1aoiLI/mYiwDo3j3gmEREpFRRUo8V69ZBlSqwcyfP9HqDqlVVUxcRkUOj5vdYsW4dNGwIlSszfZavpVfUX0dERA6BauqxYt06aNSI/fth1izo1SvogEREpLRRUo8VWUl90SLYsUNJXUREDl1Uk7pz7lTn3ELn3BLn3D0FlDnPOZfknJvvnHsnmvHEtKykPmOGX+zdO9hwRESk9InaWVvnXBzwHHASkApMc859YmZJEWXaAX8BjjGzzc65RtGKJ6aZ5UjqVatqeFgRETl00ayp9wWWmNkyM9sHvAucmavMn4HnzGwzgJmti2I8sWvnTtizBxo2ZPp0dZITEZGiiWZSbwasjFhOzVoX6XDgcOfcJOfcL865U/PbkXPuKufcdOfc9PXr10cp3ACt88cymQ0aMWuWmt5FRKRoopnUXT7rLNdyRaAdcDwwHHjFOVcnz5PMXjaz3mbWu2HDhsUeaOCWLwcgNb0xO3eqk5yIiBRNNJN6KtAiYrk5sCqfMh+bWbqZJQML8Um+/Jg6FR59FCpXZjJHA0rqIiJSNNFM6tOAds65ROdcJeAC4JNcZcYBAwGccw3wzfHLohhT7OnXD775Bk48kam/1VInORERKbKoJXUzywBuAMYDC4D3zWy+c+5h59wfsoqNBzY655KA74E7zWxjtGKKObt2hecffJDkZGjdWp3kRESkaKKaPszsc+DzXOseiJg34LasqfxZltUoMWYM9O5NWho0y92VUEREpJA0onVp15wAAA5nSURBVFyQlizxj23bArBqFRx2WIDxiIhIqaakHqSlS/1jmzbs3w9r1qimLiIiRaekHqQlS6BePahbl3XrYP9+JXURESk6JfUgpaRAYiIAaWl+lZrfRUSkqJTUgxTRMy6U1FVTFxGRolJSD1JEUl+wwK9q3jzAeEREpFRTUg/K7t2waRM0a0ZmJrzyChxzDDRpEnRgIiJSWimpB2X1av/YrBk//+w7wl93XbAhiYhI6aakHpSInnHTpvnZQYOCC0dEREo/JfWgRPSMmzHDn1pv3DjYkEREpHRTUg9KrqTes2ew4YiISOmnpB6UtDSoVo0dcbX57TfdblVERH4/JfWgZF3ONnuOw0w1dRER+f2U1IOSldRnzPCLqqmLiMjvpaQelKykPnOmvzZdw8OKiMjvpaQeBDN/n9VmzZg1C3r0CDogEREpC5TUg7BhA+zbB82asXx59u3URUREfhcl9SBkXc62u95hbNumm7iIiEjxUFIPwqRJAKyv0hJQUhcRkeKhpF7Stm+Hv/wFBg5kad3egDrJiYhI8VBSL2kLF/rEftNNpK32H79q6iIiUhyU1EtaSop/TExk1So/q6QuIiLFQUm9pIWSeqtWpKVBrVpQo0agEYmISBmhpF7SUlKgTh2oU4fkZGjePOiARESkrFBSL2kpKZCQQGam7wTfr1/QAYmISFmhpF7Sli+HhATmz4dNm2DAgKADEhGRskJJvSSZ+Zp6q1ZMmOBXKamLiEhxUVIvSZs2wY4dkJDAL79A06aQkBB0UCIiUlYoqZekUM/3hASmTYM+fQKNRkREyhgl9ZKUldS3109g4UIldRERKV5K6iUpK6m/+VMCoKQuIiLFS0m9JKWkkFmrNjfcX4eTT4YTTgg6IBERKUuU1EvS8uXsbtQKM7jpJoiPDzogEREpS5TUS1JKCtvqJgC6M5uIiBQ/JfWSknWN+rrqCYCSuoiIFD8l9ZKyeTNs305axQTi4qBhw6ADEhGRskZJvaRk9XxflplAkyZQQZ+8iIgUM6WWkjB9OvTqBcCC3QlqehcRkahQUi8JTz2VPTtnSysldRERiQol9Wgzg59/9rOVKzMvrS7NmgUck4iIlElK6tE2bx6sWAEvv8yiXzazdZsLtcSLiIgUKyX1aNq6Fd57z88PHsykmVUBOProAGMSEZEyq2LQAZRpp53mm97btYPDDmPiRKhbFw4/POjARESkLFJNPZqyzqVz/fVMnQpvvglDhuhyNhERiQ7V1KNl/36Ii4M774SbbuLFK6BmTRg1KujARESkrFKdMVo2bPCJvXlzcI5Zs/ytVuvUCTowEREpq5TUo2XNGv/YpAl798L8+dCjR7AhiYhI2aakHi2rVwOwMqMpVapAejp07x5wTCIiUqYpqUdLVk393R+aZK/q0yeoYEREpDxQUo+WrJr6y5825eSTISkJ2rQJOCYRESnTlNSjZfVq9lWrzZK0qlx/PXToEHRAIiJS1impR4lNncoCa0+3bv7adJH/b+/uY6yozjiOf38sqyBaX0GpiEggKBjdkg1aLWqptmqNtlVBa6q2GBKsEZumVGJC2sa0waa1wWKLVltMfauoFalppWg1RuVF3gSpsqJ0ESwQASUCCjz9Y87S23WhrPfenWXu75NMZs6Zw+S5T7j73Jm594yZWbW5qFfDhg0wdy6Pbb2AceNAyjsgMzOrBS7qZdqwAW66CbZtK+mcNQtF8HyPCxg1KrfQzMysxriol2n8+GyWuCefLOlcsIDtOpDDRwzloINyC83MzGqMp4kt06pV2Xrr1my9cCFsumMph8eJnHGW02tmZh3HZ+plWr8+Wzc3Z7PC3nr5YgZvX8AyhjB8eL6xmZlZbalqUZd0vqTXJTVJurmN/ddKWi9pUVquq2Y8lbZjB6xYkW03N8PSZ9fz6JsNHM066htOZujQfOMzM7PaUrXrw5LqgCnAecBqYJ6kGRHxWquhD0fEDdWKY68islPtXr1g1y54+WUYNgwWL4ZJk+Cii+Dqq/87fudOdvzxIeZ0O5u6VSt5eY7Ytm04PdjCi1NXEjObmZqGjpx4ItTn8qrMzKxGVfOm7zCgKSJWAkh6CLgEaF3Uc7P+F/dx6MQbeeE7v6dh7lSOmPc020ZfT7fHHoSNG4mZM1ny0ods27iV+rpdHPPRKj47/Q7OTP/+dGDqoF1M/uAGzlszjT+9c3m24+67/eN0MzPrcNUs6scCzSXt1cBpbYy7VNJZwBvA9yKiufUASWOAMQB9+/atWIDP7jqbflsHcdaUkXRlJwDd7rmTzXWHc9tps7hl0UhO/e3Y//k3q+hLb73LAfERAH/5+Wv0/PqfARjJI6w9cgi9r9uv7iKYmVlBVPOeeltTrkSr9pNAv4g4Bfg7MK2tA0XEXRHRGBGNPXv2rFiA517Xj6Pun7y7oI9hKlu6HMKonQ/w0znn0mv7v5jwrdU0L9zAiufW8Or1d/LU+Of4cMmb8MADAPSffhuH7NzM8qOGEz17cszYb1QsPjMzs/ZQROs6W6EDS58HfhQRX0ntCQAR8bM9jK8D3ouIQ/d23MbGxpg/f37lAo2AAQOgRw9ef2QJA47/mOhaz/Dh0L8/TJsGXfd0PWPgQHjrrexr7ytWZMcxMzOrIEmvRETjvoyt5uX3ecBASScA7wBXAN8sHSCpd0SsTc2LgeVVjKdtEjzxBHTpwqBB0PLtthdf3IfpXQcPhqYm6N49+wRgZmaWo6oV9YjYIekG4G9AHXBvRCyT9BNgfkTMAG6UdDGwA3gPuLZa8ezVySd/omuf5ms/6SSYMSNbd/FP/s3MLF9VnfIsIp4CnmrVN7FkewIwoZoxVFXL81QHD843DjMzMzyjXHlaivmQIfnGYWZmhot6eRoaYOxYGDky70jMzMz8QJey1NfDnXfmHYWZmRngM3UzM7PCcFE3MzMrCBd1MzOzgnBRNzMzKwgXdTMzs4JwUTczMysIF3UzM7OCcFE3MzMrCBd1MzOzgnBRNzMzKwgXdTMzs4JwUTczMysIF3UzM7OCUETkHUO7SFoPrKrgIY8CNlTweLXKeSyfc1g+57AynMfyVTKHx0dEz30ZuN8V9UqTND8iGvOOY3/nPJbPOSyfc1gZzmP58sqhL7+bmZkVhIu6mZlZQbiow115B1AQzmP5nMPyOYeV4TyWL5cc1vw9dTMzs6LwmbqZmVlB1HRRl3S+pNclNUm6Oe94OitJ90paJ2lpSd8RkmZJWpHWh6d+SZqccrpE0tD8Iu88JB0n6VlJyyUtkzQu9TuP7SCpm6S5khanPP449Z8gaU7K48OSDkj9B6Z2U9rfL8/4OxNJdZIWSpqZ2s5hO0h6W9KrkhZJmp/6cn8/12xRl1QHTAEuAAYDV0oanG9UndYfgPNb9d0MzI6IgcDs1IYsnwPTMgb4TQfF2NntAL4fEScBpwPfTf/fnMf22Q6MiIhTgQbgfEmnA5OA21MeNwKj0/jRwMaIGADcnsZZZhywvKTtHLbfFyOioeSna7m/n2u2qAPDgKaIWBkRHwEPAZfkHFOnFBHPA++16r4EmJa2pwFfK+m/LzIvA4dJ6t0xkXZeEbE2Ihak7Q/I/pgei/PYLikfW1KzPi0BjACmp/7WeWzJ73TgS5LUQeF2WpL6AF8FfpfawjmshNzfz7Vc1I8Fmkvaq1Of7ZujI2ItZAUL6JX6ndf/I12+/BwwB+ex3dJl40XAOmAW8CawKSJ2pCGludqdx7R/M3Bkx0bcKf0KGA/sSu0jcQ7bK4CnJb0iaUzqy/393LUaB91PtPVJ0z8FKJ/zuheSDgYeBW6KiPf3csLjPO5BROwEGiQdBjwOnNTWsLR2HluRdBGwLiJekXROS3cbQ53DvTszItZI6gXMkvTPvYztsBzW8pn6auC4knYfYE1OseyP/t1y+Sit16V+53UPJNWTFfT7I+Kx1O08fkoRsQn4B9l3FA6T1HKSUpqr3XlM+w/lk7eSas2ZwMWS3ia77TiC7MzdOWyHiFiT1uvIPlwOoxO8n2u5qM8DBqZvfB4AXAHMyDmm/ckM4Jq0fQ3wREn/1enbnqcDm1suR9WydA/yHmB5RPyyZJfz2A6SeqYzdCR1B84l+37Cs8BlaVjrPLbk9zLgmajxyTkiYkJE9ImIfmR/956JiKtwDveZpB6SDmnZBr4MLKUzvJ8jomYX4ELgDbJ7crfkHU9nXYAHgbXAx2SfOEeT3VObDaxI6yPSWJH9quBN4FWgMe/4O8MCfIHsctsSYFFaLnQe253HU4CFKY9LgYmpvz8wF2gCHgEOTP3dUrsp7e+f92voTAtwDjDTOWx33voDi9OyrKV+dIb3s2eUMzMzK4havvxuZmZWKC7qZmZmBeGibmZmVhAu6mZmZgXhom5mZlYQLupmNUbSzvRkqZalYk8olNRPJU/zM7OOVcvTxJrVqq0R0ZB3EGZWeT5TNzNg9/OhJ6Xnlc+VNCD1Hy9pdnoO9GxJfVP/0ZIeT882XyzpjHSoOkl3K3ve+dNp5jcz6wAu6ma1p3ury++jSva9HxHDgF+TzQdO2r4vIk4B7gcmp/7JwHORPdt8KNnMWpA9M3pKRAwBNgGXVvn1mFniGeXMaoykLRFxcBv9bwMjImJlevjMuxFxpKQNQO+I+Dj1r42IoyStB/pExPaSY/QDZkXEwNT+IVAfEbdW/5WZmc/UzaxU7GF7T2Pasr1keyf+7o5Zh3FRN7NSo0rWL6XtF8me5gVwFfBC2p4NjAWQVCfpMx0VpJm1zZ+gzWpPd0mLStp/jYiWn7UdKGkO2Qf+K1PfjcC9kn4ArAe+nfrHAXdJGk12Rj6W7Gl+ZpYT31M3M2D3PfXGiNiQdyxm9un48ruZmVlB+EzdzMysIHymbmZmVhAu6mZmZgXhom5mZlYQLupmZmYF4aJuZmZWEC7qZmZmBfEfuEB2l+KP+9YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot accuaracy of training process\n",
    "fig, ax = plt.subplots(1,1,figsize = (8,6))\n",
    "ax.plot(history.history['acc'],'b',label = 'Training')\n",
    "ax.plot(history.history['val_acc'],'r',label='Validation')\n",
    "ax.legend()\n",
    "ax.set(xlabel='Epoch',ylabel = 'Accuracy', title= 'Accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function to plot confusion matrix\n",
    "***\n",
    "We used the visualization function for confusion matrix from __[sklearn website](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py)__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model\n",
    "***\n",
    "First, we evaluted the performance of the final model on test set. The accuracy was 93% before class balancing, and 88% after class balancing, which was really close to the training and validation performance at the final epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18612500/18612500 [==============================] - 478s 26us/step\n",
      "Loss on test data = 0.18031106435448713 \n",
      "Accuracy on test data = 0.9366222699798394\n"
     ]
    }
   ],
   "source": [
    "# evaluate test data\n",
    "test_target = keras.utils.to_categorical(test_values, num_classes=2)\n",
    "test_result = model.evaluate([test_playlists, test_tracks],test_target)\n",
    "print('Loss on test data =', test_result[0],\n",
    "      '\\nAccuracy on test data =', test_result[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss on test data = 0.3529369857907295 \n",
      "Accuracy on test data = 0.8793360018730163\n"
     ]
    }
   ],
   "source": [
    "# evaluate test data (class balanced) \n",
    "is_ones_test = np.array(test_values).reshape(-1,)==1\n",
    "test_tracks_ones = test_tracks[is_ones_test]\n",
    "test_playlists_ones = test_playlists[is_ones_test]\n",
    "\n",
    "test_tracks_zeros = test_tracks[~is_ones_test]\n",
    "test_playlists_zeros = test_playlists[~is_ones_test]\n",
    "\n",
    "test_result_gen = model.evaluate_generator(valGenerator_balance(test_tracks_ones, test_playlists_ones, \n",
    "                                                                test_tracks_zeros, test_playlists_zeros, \n",
    "                                                                ohe, val_batch_size= 10000, balance_ratio = 0.5),\n",
    "                                           steps = 100, verbose = 0)\n",
    "print('Loss on test data =', test_result_gen[0],\n",
    "      '\\nAccuracy on test data =', test_result_gen[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction and confusion matrix\n",
    "***\n",
    "We then made the prediction on test data and plotted confusion matrix (both non-normalized and normalized). The final model had sensitivity 82% (predicting class 1 from real class 1) and specificity of 94% (predicting class 0 from real 0). Notice that there were a lot of false positive samples, since we have an extremely unbalanced class ratio. Those samples would potentially be the ones that we make recommendation on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted proportion of ones = 0.06425826729348556\n"
     ]
    }
   ],
   "source": [
    "# get prediction\n",
    "y_prob = model.predict([test_playlists, test_tracks])\n",
    "y_classes = y_prob.argmax(axis=-1)\n",
    "print('predicted proportion of ones =', np.mean(y_classes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[17411944  1175069]\n",
      " [    4549    20938]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAToAAAEYCAYAAADMJjphAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xm8VfP+x/HX55wjolJUVERIgzSqkAZJMlxkSKMbmS53NE/3XsNFuFxD/MxlnklEmUUURYVSiqKkeaCUzunz+2Otc+xznM5ep7332fvs8372WI/2XsP3+1177fPZ3+/6rrW+5u6IiGSznHQXQEQk1RToRCTrKdCJSNZToBORrKdAJyJZT4FORLJepQ10ZlbdzF42szVm9mwC6Qwys9eTWbZ0MbOuZjY7U/Izsz3NzM0sr6LKVFmY2Xwz6xW+vtzMHkhBHveY2T+TnW5lZKm+js7MBgLnA82Bn4BpwHXu/kGC6Q4B/gIc7O75CRc0w5mZA03dfW66y7IlZjYfOMPd3wzf7wl8C2yT7GNkZqOAhe5+ZTLTrSglP6skpDc0TO+QZKSXbVJaozOz84HbgOuBXYDGwN3AcUlIfg9gTlUIclGo1pQ6+myzgLunZAJ2BH4GTi5jnW0JAuEP4XQbsG24rAewELgAWAosBk4Ll10N/ApsCvMYBlwFPBaT9p6AA3nh+6HANwS1ym+BQTHzP4jZ7mDgE2BN+P/BMcveBa4FJobpvA7U3cK+FZb/4pjyHw8cBcwBVgKXx6zfCfgIWB2uOwKoFi6bEO7LunB/T4lJ/xLgR+DRwnnhNnuHebQP3zcElgM9Ihy7h4ELwteNwrzPDd/vE6ZrJfJ7FNgM/BKW8eKYY/BH4Lsw/ysiHv9ixyWc52H+Z4XH/tcwr5e3sB8OnAN8DawC7uK3VkwOcCWwIDw+jwA7lvjuDAvLPSFm3mnA92F65wAdgRnhcRsRk/fewNvAinC/HwdqxyyfD/QKX19F+N0Nj/vPMVM+cFW47FJgHsF3bybQN5zfAtgAFITbrA7njwL+E5PnmcDc8PiNARpG+ayyYUploOsTHqS8Mta5BpgE1AfqAR8C18YEivxwnW0IAsR6oE7JL8cW3hd+MfOAHYC1QLNwWQNgv5J/UMBO4UEeEm43IHy/c7j83fCLti9QPXw/fAv7Vlj+f4XlPxNYBjwB1AT2C7+ce4XrdwAODPPdE5gF/L3kH3kp6d9IEDCqExN4Yr7Ys4DtgfHAfyMeu9MJgwcwMNznp2OWvRRThtj85hP+8ZY4BveH5WsDbARaRDj+RceltM+AEn/EW9gPB14BahO0JpYBfWL2Yy6wF1ADeAF4tES5HyH47lSPmXcPsB3QOzx+o8PyNyIImN3DNPYBDg+PTT2CYHlbaZ8VJb67Meu0DcvcLnx/MsEPVg7Bj906oEEZn1fRZwT0JAi47cMy3QlMiPJZZcOUyqbrzsByL7tpOQi4xt2XuvsygprakJjlm8Llm9z9VYJfq2ZbWZ7NQCszq+7ui939y1LWORr42t0fdfd8d38S+Ar4Q8w6I919jrv/AjxD8GXckk0E5yM3AU8BdYHb3f2nMP8vgdYA7j7V3SeF+c4H7gW6R9inf7v7xrA8xbj7/QS/0JMJgvsVcdIr9B7Q1cxygG7ATUCXcFn3cHl5XO3uv7j7dGA6QcCD+Mc/GYa7+2p3/w54h9+O1yDgVnf/xt1/Bi4D+pdopl7l7utKfLbXuvsGd3+dINA8GZZ/EfA+0A7A3ee6+xvhsVkG3Er841nEzOoRBNG/uPtnYZrPuvsP7r7Z3Z8mOLadIiY5CHjI3T91943h/h4UnkcttKXPCjN7yMyWmtkXEcr+PzObFk5zzGx1xDKmTCoD3QqgbpzzGw0Jmg6FFoTzitIoESjXE/z6lou7ryP4BTwHWGxmY82seYTyFJapUcz7H8tRnhXuXhC+LvxjWRKz/JfC7c1sXzN7xcx+NLO1BOc165aRNsAyd98QZ537gVbAneEXPC53n0fwo9IW6ErwS/+DmTVj6wLdlj6zeMc/GcqTdx7BueRC35eSXsnjt6XjWd/MnjKzReHxfIz4x5Nw222A54An3P2pmPmnhsFjdRg8WkVNkxL7Gwb3FUT/bo8iaKXF5e7/cPe27t6WoOb4QsQypkwqA91HBFX748tY5weCToVCjcN5W2MdQROt0K6xC919vLsfTlCz+YogAMQrT2GZFm1lmcrj/wjK1dTdawGXE5wHK0uZXeZmVoPgvNeDwFVmtlM5yvMecBLBecJF4ftTgToEPeflLk8pyjr+xY6nmRU7nluRV5S88ykeuBLJ44Zw+9bh8RxM/ONZ6E6C83BFPcpmtgfBd/bPBKdSagNfxKQZr6zF9tfMdiBodUX6brv7BIJze0XMbG8zG2dmU83s/S1UHgYAT0bJI5VSFujcfQ3B+am7zOx4M9vezLYxsyPN7KZwtSeBK82snpnVDdd/bCuznAZ0M7PGZrYjQdUcADPbxcyODQ/uRoLaSkEpabwK7GtmA80sz8xOAVoS1GhSrSbBecSfwy/Mn0osX0JwPqk8bgemuvsZwFiC80sAmNlVZvZuGdu+R/BHNSF8/y7B5TwfxNRSSypvGcs6/tOB/cysrZltR3AeK5G8Ssv7H2bWJPxBuJ7gPGSyevFrEnYMmFkj4KIoG5nZ2QS15oHuvjlm0Q4EwWxZuN5pBDW6QkuA3cys2haSfgI4Lfw8tyXY38nhaZKtdR9B07oDcCHBFRWx+7IH0ISgUyatUnp5ibvfSnAN3ZUEB+h7gj+e0eEq/wGmEPRafQ58Gs7bmrzeAJ4O05pK8eCUQ9B7+wPBr1J34NxS0lgBHBOuu4Kg5/AYd1++NWUqpwsJTvz/RPDL/XSJ5VcBD4fNln7xEjOz4wiaGueEs84H2pvZoPD97gS9x1vyHsEfa2Gg+4CghjVhi1sEtZgrwzJeGK+MlHH83X0OQWfFmwTnokped/kg0DLMazTl9xBBT/EEgl74DQSBPFmuJjjxv4bgRyZq820AQQD/wcx+DqfL3X0mcAtBS2kJsD/Fj9/bBOd8fzSz331f3f0t4J/A8wS9+nsD/bdmx6CotXAw8KyZTSM4p9ygxGr9gefK+GGsMCm/YFgyU/jlPCwM7iJxhR0Xr7h7KzOrBcx295LBLXb9z4Dz3P3DCiriFlXaW8AkMeHJYgU52Sruvhb41sxOBrBAYW86YcdVHYIaaNop0IlIXGb2JEHQamZmC81sGMElK8PMbDpBszn2jqcBwFOeIU1GNV1FJOupRiciWS+jbla2vOpu1WqmuxhSDq2b757uIkg5fL9gAStWLI96PV8kubX2cM//3Y05pfJflo1390gXHidTZgW6ajXZtlncKyckg7z+3v/SXQQph97dD0x6mp7/S+S/2w3T7op6J0dSZVSgE5HKyMAy+yyYAp2IJMaAnNx0l6JMCnQikjhL6mm/pFOgE5EEqekqIlWBanQiktUM1ehEJNuZanQiUgWo11VEsps6I0Qk2xlquopIFaAanYhkNzVdRaQqyMnspmtmh2ERyXyF97pGmeIlFWGgbDPrEY5v+6WZRRpjWIFORBIUNl2jTPGNooyBss2sNsGwise6+37AyVESVdNVRBKXpF5Xd58Qjja2JQOBF9z9u3D9pVHSVY1ORBIXvUZX18ymxExnlTOnfYE6ZvaumU01s1OjbKQanYgkxsp1C9hydz8ggdzygA7AYUB14CMzmxQOeF7mRiIiiam4W8AWEgTLdcA6M5sAtAHKDHRquopIgpLaGRHPS0BXM8szs+2BzsCseBupRiciiUtSZ0Q4UHYPgnN5C4F/A9sAuPs97j7LzMYBM4DNwAPuvsVLUQop0IlIYpL4PDp3HxBhnZuBm8uTrgKdiCRIt4CJSFWgp5eISNbTgzdFJKuZmq4iUhWo6Soi2c4U6EQkmwVPUlegE5FsZuGUwRToRCRBRk6OOiNEJMup6SoiWU+BTkSym87RiUi2M0w1OhHJfuqMEJGspxqdiGS3SnCOLrPrmyJSKZhZpClCOnEHsA7X62hmBWZ2UpTyKdCJSEIKOyOSEeiIM4A1gJnlAjcC46OWUYFORBKWrEDn7hOAlXFW+wvwPBBp8GrQOToRSZSB5UQ+SVfXzKbEvL/P3e+LnJVZI6Av0BPoGHU7BToRSVg5el0THcD6NuASdy8oT0+vAp2IJKwCLy85AHgqzK8ucJSZ5bv76LI2UqATkYRU5J0R7t6kKF+zUcAr8YIcKNCJSDIkKc7FG8B6a9NVrytwz78HseCtG5jy7OVF8x4dfhqTnrqUSU9dyldjr2bSU5cW22b3XeuwbOIt/H3IYWWmA3BCr3ZMfe4K1k29g/YtGxfN3yYvl3uvGswnz1zO5KcvpWuHpr8r27O3nf279AD+PuQwfvlsBDvX3mGr9ztb/O3cM2m5VyO6dW5bNG/Mi8/RrVMbdt1xW6Z9OrVo/nNPP0HPLgcUTbvuuC1fzJgGQN+jenFw+/2Kli1bFnTqbdy4kTOHDqRzmxb0ObQL3y2YX5Tel1/M4KjDutKtUxu6H9iODRs2ADD6+WfocVB7unVqwzX/LP7dyTqW1F7XAe7ewN23cffd3P1Bd7+ntCDn7kPd/bkoRVSgAx59eRLHnXdXsXlDLh3Jgf2Hc2D/4Yx+axovvT2t2PKbLjyR1yd+GTcdgC/n/UD/C+7ng0/nFZt/+gldAOjY73qOOWcEw8/vW+zLcFzPNqxbv/F36e22S216Htic7xbH64WvGvoPOpWnXnil2LzmLffjocef4aAuXYvNP+mUgbw9cQpvT5zCiPtGsvsee9Kq9W8B8u4HHilaXq9efQCeeGQktWvXYfL0WZx93l+59t/BD09+fj7nnTmUm28bwYSPp/Pi2DfZZpttWLliBdf88zKee3k8Ez6ezrKlS5nw7tsp/hTSKycnJ9KUtvKlLecMMvHTeaxcs36Ly088vD3PjPutVvCHHq35duFyZs77MVI6s79dwtcLfn/JT/O9duWdj2cDsGzVz6z56Rc6hDW+HapX46+DezL8gXG/2+6mC0/kittH4+7RdjDLHdSlK7Xr1Ck2b99mLdinabMyt3vxuafpe1K/uOmPG/sy/QYMAeAPx5/IB+++g7vz7ltv0HK//dlv/zYA7LTzzuTm5rJg/rfstU9T6tatB0C3Hj0Z+9KLW7NrlYdFnNJEgS6OLu33ZsnKn5j33TIAtt+uGhecdjjX3ftqwml/PmcRf+ixP7m5OezRcGfatdyd3XYN/mD/fe4x3P7oW6z/5ddi2xzdfX9+WLqaz+csSjj/qu6l55+j70mnFJv3t3PPoGeXA7j1xuuKfkgWL15Eo912AyAvL4+atXZk5coVzJv7NWbGKccfTa+unRhx238BaLLX3sydM5vvFswnPz+f18aOYdGi7yt25ypYEu+MSImUdkaYWR/gdiAXeMDdh6cyv1To1+cAnh332/WN//zT0dz52NusKxGAtsbDL31E8ya7MPHxi/lu8UomTf+W/IICWu/biL12r8fFt7xA4wY7Fa1ffbttuGTYERxz7oiE867qpn7yMdW3r06Llq2K5t39wMM0aNiIn3/6idMHn8KzTz5Gv4FDoJSas2EUFOQzedKHjH/3Q6pX356T/nAErdu2p1uPntz4vzs5a+ggcnJy6Nj5IBbM/6Yid69CpTuIRZGyQBfej3YXcDiwEPjEzMa4+8xU5Zlsubk5HNezDV0G3lQ0r2OrPejbqy3X/f14dqxZnc2bnQ2/buKepyeUO/2Cgs1cfMsLRe/fGXU+c79bRtcO+9C+ZWO+Gns1ebk51NupJuPv/xvn3/gsezTamY+fvgyARvVr89ETl9B1yM0sWfFT4jtchYx+/pnf1eYaNGwEQI2aNTmhX38+mzqFfgOH0KDhbixauJCGjXYjPz+fn9auoc5OO9GgYSMO7tKVnXeuC0Cv3n34fPpndOvRkyOOPIYjjjwGgEdGPkBubnY3nqpsoAM6AXPd/RsAM3sKOA6oNIGuZ+dmzJm/hEVLVxfN6zXstqLXV5x9FOvWb9yqIAdBDc0w1m/4lZ6dm5NfsJmvvvmRr775kfuf/QCAxg124oU7zuGIM28HYI/DLiva/quxV9Nl0E2sWL1uq/KvqjZv3szLo59n9GtvFc3Lz89nzZrV7LxzXTZt2sQb48bSrUfQo37EUcfwzJOP0rHzgbw8+nkO6d4DM+PQw3pz1+23sH79eqpVq8aHE9/n7HP/CsCyZUupV68+q1etYtQD93D/qCfSsq8VpSoHukZA7ImJhUDnkiuZ2VnAWQBsUyOFxdmyh28YStcOTalbuwZzx13Ltfe8ysOjP+LkIzoU64TY2nSOPbQ1t15yMnXr1OCFO85hxuxFHHveXdSrU5OX7z6PzZudH5atZtiVD6dwL7PX2acN5sMPJrByxXLaNm/CRZf/izp16nD5Rf9gxfJlDDr5OFrt34anR48F4KOJ79OgYSP2bLJXURobN26kf9+j2bRpE5sLCuja4zAGDx0GwMBTT+PPZw2lc5sW1K5Th3tHPgZA7Tp1OOe8v9Gnx0FgRq/efTi8z1EAXHnx+cz8YgYA519yBXs33bciP5IKV457XdPCUtVzZ2YnA0e4+xnh+yFAJ3f/y5a2ydm+vm/bLH4vmGSOBRP+l+4iSDn07n4g0z6dmtSotO2uTX23QXdEWvebW4+amuC9rlsllTW6hcDuMe93A35IYX4ikgYGZHjLNaWXl3wCNDWzJmZWDegPjElhfiKSFkl98GZKpKxG5+75ZvZngqeA5gIPufuXcTYTkUoo02t0Kb2Ozt1fBRK/slZEMpdBToZ3RujpJSKSEEOBTkSqgCrddBWRqqEqXzAsIlWBZX6NLrtvwBORlAuuo6uYAazNbJCZzQinD82sTZQyKtCJSIKMnJxoUwSjKHsA62+B7u7eGrgWiDRUopquIpKwZJ2jc/cJZrZnGcs/jHk7ieCOq7gU6EQkMeU7R5fQANYlDANei7KiAp2IJKTwHF1EiQ5gHeRpdihBoDskyvoKdCKSsIrsdTWz1sADwJHuviLKNgp0IpKwirqOzswaAy8AQ9x9TtTtFOhEJDFJvNc1wgDW/wJ2Bu4Og2t+lKawAp2IJCSZz6Nz9wFxlp8BnFHedBXoRCRBVXgUMBGpOjI8zinQiUjiVKMTkaxmevCmiFQFqtGJSNbL8DinQCciiVONTkSyWyV48KYCnYgkxHQdnYhUBbnqdRWRbJfhFToFOhFJjFkl7owws1plbejua5NfHBGpjDK85Vpmje5LwAkeTlCo8L0DjVNYLhGpRCptjc7dd6/IgohI5ZXhcS7acIdm1t/MLg9f72ZmHVJbLBGpLAzINYs0pUvcQGdmI4BDgSHhrPXAPakslIhUIhEHr07SANZmZneY2dxwEOv2UYoYpUZ3sLufDWwAcPeVQLUoiYtI1WAWbYpgFGUPYH0k0DSczgL+L0qiUQLdJjPLIeiAwMx2BjZHSVxEsp8BOWaRpnjcfQKwsoxVjgMe8cAkoLaZNYiXbpRAdxfwPFDPzK4GPgBujLCdiFQR5ajR1TWzKTHTWeXMqhHwfcz7heG8MsW9YNjdHzGzqUCvcNbJ7l5q+1lEqp5yPngz0QGsS8vI420U9c6IXGBTmGCknloRqTqiNEuTZCEQe+nbbsAP8TaK0ut6BfAk0DBM9Akzu2wrCykiWcgiTkkwBjg17H09EFjj7ovjbRSlRjcY6ODu6wHM7DpgKnBDIqUVkeyRrDsjIgxg/SpwFDCX4FK306KkGyXQLSixXh7wTdSCi0h2C3pdk5NWhAGsHTivvOmWdVP//wjOya0HvjSz8eH73gQ9ryIiRRcMZ7KyanSFPatfAmNj5k9KXXFEpDKqtMMduvuDFVkQEamcktl0TZW45+jMbG/gOqAlsF3hfHffN4XlEpFKJNObrlGuiRsFjCQI3EcCzwBPpbBMIlLJVODlJVslSqDb3t3HA7j7PHe/kuBpJiIiwZ0RSbrXNVWiXF6y0YJ66TwzOwdYBNRPbbFEpDLJ8JZrpED3D6AG8FeCc3U7AqenslAiUrlU2l7XQu4+OXz5E789fFNEBAgGsE5nszSKsi4YfpEyngrg7iekpEQiUrlEf6hm2pRVoxtRYaUItWvRmImTKzxbkSojVeM2ZPrlJWVdMPxWRRZERCqvTH92W9Tn0YmIlMqoxDU6EZGo8jK8Shc50JnZtu6+MZWFEZHKJxgPIrNrdFGeMNzJzD4Hvg7ftzGzO1NeMhGpNHIs2pS28kVY5w7gGGAFgLtPR7eAiUiMZI3ramZ9zGx2OED1paUsb2xm75jZZ+EA1kdFKV+UQJfj7gtKzCuIkriIZL9kjetqZrkEw6seSfC0pAFm1rLEalcCz7h7O6A/cHeUMkY5R/e9mXUCPCzIX4A5URIXkaohNznN0k7AXHf/BsDMniIYsHpmzDoO1Apf70iEEcAgWqD7E0HztTGwBHgznCcigpXvySR1zWxKzPv73P2+8HVpg1N3LrH9VcDrZvYXYAd+G2+6TFHudV1KUEUUESlVOTpdyxrAOsrg1AOAUe5+i5kdBDxqZq3cfXNZmUZ5wvD9pWSGu58Vb1sRqRqS1KMaZXDqYUAfAHf/yMy2A+oCS8tKOErT9c2Y19sBfSlevRSRKqywMyIJPgGamlkTgude9gcGlljnO+AwYJSZtSCIScviJRyl6fp07HszexR4I1q5RaQqSEacc/d8M/szMB7IBR5y9y/N7BpgiruPAS4A7jezfxC0NIeGY72WaWtuAWsC7LEV24lINrLkPRXF3V8FXi0x718xr2cCXcqbbpRzdKv47RxdDrAS+N2FfCJSNVX64Q7DsSLaELSXATZHqSaKSNWS6YGuzDsjwqD2orsXhJOCnIj8jplFmtIlyi1gH5tZ+5SXREQqpcKmaybf1F/WmBF57p4PHAKcaWbzgHUE++XuruAnIpV+zIiPgfbA8RVUFhGphAzIy/CTdGUFOgNw93kVVBYRqaQqc42unpmdv6WF7n5rCsojIpWOkVPqbaqZo6xAlwvUoPQbbUVEgMLBcdJdirKVFegWu/s1FVYSEamc0tyjGkXcc3QiImUxIDfDI11Zge6wCiuFiFRqSXp6ScpsMdC5+8qKLIiIVF4ZHuc0gLWIJMaIdotVOinQiUhiKsEA1gp0IpKwzA5zmV/jFJEMZwQP3owyxU0rzgDW4Tr9zGymmX1pZk9EKaNqdCKSsGS0XGMGsD6cYKCcT8xsTPhU4cJ1mgKXAV3cfZWZ1Y+Stmp0IpKgaM+ii3Aer2gAa3f/FSgcwDrWmcBd7r4KioZjjUuBTkQSUtjrGmUiHMA6ZoodNrW0AawblchuX2BfM5toZpPMrE+UMqrpKiIJK0eva6IDWOcBTYEeBOO+vh8OYL26rExVoxORhFnEKY4oA1gvBF5y903u/i0wmyDwlUmBTkQSYpa0XteiAazNrBrBANZjSqwzGjg0yNfqEjRlv4mXsJquIpKwZFwwHHEA6/FAbzObCRQAF7n7inhpK9CJSMKSdcFwhAGsHTg/nCJToBORhGX4HWAKdCKSmODyksyOdAp0IpIw1ehEJMtZ5X3wpohIFGq6ikj2MzVdRaQKUKATkaxnGd501S1gCSgoKODAA9pxwnHHAHDm6UNp3rQJnTu0pXOHtkyfNq3Y+lM++YQdts3lheefK5p3xWWX0KFtKzq0bcWzzzxdoeXPdt9//z1H9DqUtvu3oH2b/Rhxx+0ArFy5kqP7HE6rFk05us/hrFq1CoBVq1bR76S+dGzXmkMO6sSXX3wBwIYNGzjkoE50at+G9m3249qr/12Uxztvv8VBHdvTuUNbenY/hHlz51b8jqZZMh+8mSoKdAkYccftNGvRoti864ffzOSp05g8dRpt2rYtml9QUMCVl1/C4b2PKJr32qtjmfbZp0yeMo0JEydz2y03s3bt2gorf7bLy8tj+E23MO3zWbz3wSTuvecuZs2cyX9vGk6Pnofxxayv6dHzMP5703AAbhp+PW3atOWTz2bw4MhHuPD8vwGw7bbbMu6Nt/n40+lMnjKN18ePY/KkSQD89c9/YuQjjzN56jRO6T+Q4df/J237m05m0aZ0UaDbSgsXLmTca2M57fQzIq1/94g7Ob7vidSr99sDUWfNmknXbt3Jy8tjhx12YP/WbXh9/LhUFbnKadCgAe3atwegZs2aNG/egh9+WMQrL7/E4CF/BGDwkD/y8pjRAHw1ayY9Dg2GM27WvDkLFsxnyZIlmBk1atQAYNOmTeRv2lR0b6eZFf04rV27hgYNG1boPmYKi/gvXRTottJFF/yd6264iZyc4h/hVf+6go7tWnPRBf9g48aNACxatIgxL73ImWefU2zd1q3bMH7ca6xfv57ly5fz3nvvsHDh90jyLZg/n2nTPqNjp84sXbKEBg0aAEEwXLY0eEjt/q3b8NLoFwD45OOP+W7BAhYtXAgENfLOHdrSuGF9evY6nE6dOwNw970P0PfYo9h7z9144vFHufDiUoc5yGoG5Fi0KV1SFujM7CEzW2pmX6Qqj3R5dewr1K9Xn/YdOhSbf811NzD9i6/4YNInrFq5kltuvhEIguJ/rr+R3NzcYuv3Orw3fY48ikO7HswfBw+gc+eDyMtV/1Cy/fzzzwzodyI333IbtWrV2uJ6F158KatXraJzh7b831130qZtO/LyguORm5vL5KnTmDt/IVM++bjo/N2dt/+PF8e8yrz5Cxnyx9O45MJy3WueJaLW59IX6VL5VzUKGAE8ksI80uKjDyfyyitjGDfuVTZu2MDatWs57dTBjHzkMSA4p3Pq0NO47db/AvDp1CmcOrg/ACuWL2f8uFfJy8vj2OOO55LLruCSy64A4I9DBrJP07jPEJRy2LRpEwP6ncgpAwZxfN8TAKi/yy4sXryYBg0asHjxYurVD04n1KpVi/seHAmAu9O8aRP2bNKkWHq1a9emW/cevP76OOrvsgufz5heVLs76eRTOO6YSE/2zi6V4Dq6lNXo3H0CsDJV6afTtdfdwLz5C5k9dz6PPP4UPQ7tychHHmPx4sVA8Ecy5qXRtNyvFQBfff0ts+fOZ/bc+fQ94SSyZX4rAAAJeklEQVRuu/Nujj3ueAoKClixIniU1uczZvDF5zPodXjvtO1XtnF3zjlzGM2at+Bv//itpnX0Mcfy2KMPA/DYow9zzB+C8VdWr17Nr7/+CsDIBx/gkEO6UatWLZYtW8bq1cGTun/55RfefutNmjVrTp06dVi7Zg1fz5kDwNtvvkGz5sU7p6qCytDrmvZ2Ujg4xlkAuzdunObSJOa0UwexfNkyHKd167bcefc9Za6/adMmeh3aFYCaNWvx0KjHippKkrgPJ07kiccfpVWr/encIegBv/o/13PhxZcyeEA/Hh75ILvv3pjHn3oWgK9mzeKM008lNzeX5i1acs99DwLw4+LFnHn6HykoKGCzb+bEk/px1NHBJUV33XM/A/qdSE5ODrXr1OHe+x9Kz86mWYZX6LDgOXYpStxsT+AVd28VZf0OHQ7wiZOnpKw8IlVdl84HMHXqlKTGpRb7t/ORo9+JtO5B+9SZWsbgOISjet1O8IThB9x9+BbWOwl4Fujo7nGDhnpdRSRhyeiMiBnA+kigJTDAzFqWsl5N4K/A5KjlU6ATkYQl6YLhKANYA1wL3ARsiFq+VF5e8iTwEdDMzBaa2bBU5SUi6VWO4Q4TGsDazNoBu7v7K+UpX8rOfLv7gFSlLSKZw6iYAazNLAf4HzC0POWDDOh1FZFKLnnX0cUbwLom0Ap4NwysuwJjzOzYeB0SCnQikrAkdeMWDWANLCIYwHpg4UJ3XwPULcrT7F3gQvW6ikjFKMdJui1x93ygcADrWcAzhQNYm9mxiRRPNToRSVDy7mONN4B1ifk9oqarQCciCSl8ekkmU6ATkcQp0IlItsv0MSMU6EQkYZn+mCYFOhFJWIbHOQU6EUlQhEtH0k2BTkQSEvS6ZnakU6ATkYRldphToBORZMjwSKdAJyIJ0+UlIpL1MvwUnQKdiCQuw+OcAp2IJKacD95MCwU6EUlMJRjAWoFORBKW4XFOgU5EkiDDI52eMCwiCYo6qmv8aGhmfcxstpnNNbNLS1l+vpnNNLMZZvaWme0RpYQKdCKSkMIHb0aZykwn2gDWnwEHuHtr4DmC8V3jUqATkcQlYcwIIgxg7e7vuPv68O0kgpHC4lKgE5GElaPpmtAA1iUMA16LUj51RohIwspxeclWD2BdPD8bDBwAdI+SqQKdiCQsSZ2u8QawDvIy6wVcAXR3941RElbTVUQSE14wHGWKo2gAazOrRjCA9ZhiWZm1A+4FjnX3pVGLqBqdiCQkWbeAuXu+mRUOYJ0LPFQ4gDUwxd3HADcDNYBnwzy/c/e4g1sr0IlIwpJ1vXC8AazdvdfWpKtAJyIJ072uIpL19OBNEcl+mR3nFOhEJHEZHucU6EQkMWYa7lBEqoLMjnMKdCKSuAyPcwp0IpK4DG+5KtCJSKKiPVQznRToRCQhwS1g6S5F2RToRCRhCnQikvXUdBWR7KZxXUUk20UbDiK9FOhEJHEZHukU6EQkYboFTESyXmaHOQU6EUmGDI90CnQikrBMv7zE3EsdNjEtzGwZsCDd5UiBusDydBdCyiVbj9ke7l4vmQma2TiCzyuK5e7eJ5n5R5FRgS5bmdmUMgbtlQykY5ZdNK6riGQ9BToRyXoKdBXjvnQXQMpNxyyL6BydiGQ91ehEJOsp0IlI1lOgE5Gsp0CXImbWzMwOMrNtzCw33eWRaHSsspM6I1LAzE4ArgcWhdMUYJS7r01rwWSLzGxfd58Tvs5194J0l0mSRzW6JDOzbYBTgGHufhjwErA7cLGZ1Upr4aRUZnYMMM3MngBw9wLV7LKLAl1q1AKahq9fBF4BqgEDzTL8wV1VjJntAPwZ+Dvwq5k9Bgp22UaBLsncfRNwK3CCmXV1983AB8A04JC0Fk5+x93XAacDTwAXAtvFBrt0lk2SR4EuNd4HXgeGmFk3dy9w9yeAhkCb9BZNSnL3H9z9Z3dfDpwNVC8MdmbW3syap7eEkig9jy4F3H2DmT0OOHBZ+IeyEdgFWJzWwkmZ3H2FmZ0N3GxmXwG5wKFpLpYkSIEuRdx9lZndD8wkqCVsAAa7+5L0lkzicfflZjYDOBI43N0XprtMkhhdXlIBwpPaHp6vkwxnZnWAZ4AL3H1GussjiVOgEymFmW3n7hvSXQ5JDgU6Ecl66nUVkaynQCciWU+BTkSyngKdiGQ9BbpKxMwKzGyamX1hZs+a2fYJpNXDzF4JXx9rZpeWsW5tMzt3K/K4yswujDq/xDqjzOykcuS1p5l9Ud4yStWgQFe5/OLubd29FfArcE7sQguU+5i6+xh3H17GKrWBcgc6kUyhQFd5vQ/sE9ZkZpnZ3cCnwO5m1tvMPjKzT8OaXw0AM+tjZl+Z2QfACYUJmdlQMxsRvt7FzF40s+nhdDAwHNg7rE3eHK53kZl9YmYzzOzqmLSuMLPZZvYm0CzeTpjZmWE6083s+RK11F5m9r6ZzQkfpYSZ5ZrZzTF5n53oBynZT4GuEjKzPILbkz4PZzUDHnH3dsA64Eqgl7u3J3jo5/lmth1wP/AHoCuw6xaSvwN4z93bAO2BL4FLgXlhbfIiM+tN8BiqTkBboIOZdTOzDkB/oB1BIO0YYXdecPeOYX6zgGExy/YEugNHA/eE+zAMWOPuHcP0zzSzJhHykSpM97pWLtXNbFr4+n3gQYInoixw90nh/AOBlsDE8NF31YCPgObAt+7+NUD4dI6zSsmjJ3AqFD2maE14S1Ss3uH0Wfi+BkHgqwm86O7rwzzGRNinVmb2H4LmcQ1gfMyyZ8Lb5r42s2/CfegNtI45f7djmPecCHlJFaVAV7n84u5tY2eEwWxd7CzgDXcfUGK9tgRPU0kGA25w93tL5PH3rchjFHC8u083s6FAj5hlJdPyMO+/uHtsQMTM9ixnvlKFqOmafSYBXcxsHwAz297M9gW+ApqY2d7hegO2sP1bwJ/CbXPDx7//RFBbKzQeOD3m3F8jM6sPTAD6mll1M6tJ0EyOpyawOHwE/aASy042s5ywzHsBs8O8/xSuj5ntGz4lWGSLVKPLMu6+LKwZPWlm24azr3T3OWZ2FjDWzJYTPPW4VSlJ/A24z8yGAQXAn9z9IzObGF6+8Vp4nq4F8FFYo/yZ4BFUn5rZ0wRPU15A0LyO55/A5HD9zykeUGcD7xE8x++c8Dl/DxCcu/vUgsyXAcdH+3SkqtJN/SKS9dR0FZGsp0AnIllPgU5Esp4CnYhkPQU6Ecl6CnQikvUU6EQk6/0/qBgd0agD4fsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compute and plot confusion matrix\n",
    "cm = confusion_matrix(test_values, y_classes)\n",
    "plot_confusion_matrix(cm, classes=[0,1],normalize=False,\n",
    "                      title='Confusion matrix, without normalization')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized confusion matrix\n",
      "[[0.93678011 0.06321989]\n",
      " [0.17848315 0.82151685]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAEYCAYAAAAnEYFiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XecVOW9x/HPd0FsWFBsgB3BLqJoLIhdbFhiNyZejS2JRo1JNCZGjUav5mr0aq7RxGiM3cTYg4kdRRGwYkFUVLAB9gYs/u4f58w6u+zunN0zszs7+33zOi9m5jzznGf2zP72eU55fooIzMxqVV1nN8DMrJIc5MyspjnImVlNc5Azs5rmIGdmNc1BzsxqWs0HOUkLS7pD0seSbs5Rz8GS7i1n2zqLpOGSXq6W7UlaRVJI6tlRbaoUSadL+lv6eCVJn0nqUeZtdOj+6+qqJshJOkjS+PRL8Y6keyRtWYaq9wGWA5aOiH3bW0lEXBsRO5ahPRWVBouBrZWJiEciYnBHtanp9iRNlbR9R22/s0TEmxHROyLm5amn6T7t6P3X1VVFkJN0IvB74LckAWkl4A/AHmWofmVgckTUl6GuLq8WekvlokRV/A5YBUVEpy7AEsBnwL6tlFmQJAi+nS6/BxZM120NTAN+ArwPvAP8V7ruDGAOMDfdxuHA6cDfiupeBQigZ/r8UOA14FPgdeDgotfHFL1vc+BJ4OP0/82L1j0I/AZ4NK3nXqBvC5+t0P6fFbV/T2AXYDLwAfCLovKbAGOBj9KylwC90nUPp5/l8/Tz7l9U/8+Bd4FrCq+l71k93cbQ9Hk/YCawdYZ9dzXwk/Rx/3TbP0ifD0zrVZPtXQN8DXyZtvFnRfvge8Cb6fZPbWW7VwGXAnelP98ngNXbsG/OTvfNl2k7HwTOAh5L23QHsDRwLfBJWscqRXVcBLyVrpsADC9adzrp96voc/UENkvrLixfAVPz7NOiba6VfoaPgEnAqKw/q+6wdH4DYCRQTxpkWihzJvA4sCywTPpl/E26buv0/WcCC5AEhy+APk2/dC08L/4iLpp+cQen61YA1kkfH0oa5IClgA+BQ9L3HZg+X7roF+lVYBCwcPr83BY+W6H9p6XtPwKYAVwHLAask/5CrJaW3wj4VrrdVYAXgeOL6gtgYDP1/zfJH4uFm/klOSKtZxFgNPC7jPvuMOCO9PFB6We+sWjdbUVtKN7eVGD7ZvbBFWn7NgBmA2u1sN2rSALoJunP4VrghjbsmzfTn2vP9Gf+IDCFJOAvAbxA8gdm+7TMX4G/FG3/OyRBsCfJH9d3gYWafr9o8ge06P2FbZ6TY59OK6prCvALoBewLUkwG1zqZ9Vdlmroqi8NzIzWh5MHA2dGxPsRMYOkh3ZI0fq56fq5EXE3yV+89h6z+BpYV9LCEfFORExqpsyuwCsRcU1E1EfE9cBLwO5FZf4SEZMj4kvgJmBIK9ucC5wdEXOBG4C+wEUR8Wm6/UnA+gARMSEiHk+3OxX4IzAiw2f6dUTMTtvTSERcAbxC8ld+BeDUEvUVPAQMT4d8WwHnAVuk60ak69vijIj4MiKeAZ4hCXYt+UdEjEu/N9fyzc83y765KiImpevnpq/9JSJejYiPgXuAVyPiP2n9NwMbFt4cEX+LiFnp+/+H5I9HW75vF5P0zE5N62vPPi34FtCb5I/onIi4H7iTJLgXtPSz6haqIcjNAvqWOFbUD3ij6Pkb6WsNdTQJkl+Q7Pg2iYjPSYYDRwPvSLpL0poZ2lNoU/+i5++2oT2z4puD04Ug9F7R+i8L75c0SNKdkt6V9AnJccy+rdQNMCMivipR5gpgXeB/I2J2ibIARMSrJH9QhgDDSX653pY0mPYFubb8zFoqm2XfvNVMfU1/3s3+/AEk/UTSi+kZ+49Ien+l9kHhvUeR9MQOioiv09fas08L+gFvFepK5fku1pxqCHJjSYZje7ZS5m2SEwgFK6WvtcfnJMOyguWLV0bE6IjYgaRH8xLJL3+p9hTaNL2dbWqL/yNp1xoRsTjJMEUl3tPqVDOSepMc5/wzcLqkpdrQnodIzmD3iojp6fPvAn2Ap9vTnpyy7Jt2b1/ScJLjm/uRHBJZkuTYX6l9UHjvb4A90h5jQXv2acHbwIpNTqB01HexS+j0IJfu7NOASyXtKWkRSQtI2lnSeWmx64FfSlpGUt+0/N/aucmnga3Sa5iWAE4prJC0nKRRkhYlOSb0GdDc6f+7gUHpZS89Je0PrE3Sk6m0xUiOG36W9jKPabL+PWC1NtZ5ETAhIr5PcoD6ssKK9LqvB1t570PAj0gOkENyrOlYkuOXLV060Z42ZlXpfbMYyTHOGUBPSacBi5d6k6QVgRuB70bE5GbqbO8+fYLkD/fP0t+brUmG5jdk+zi1r9ODHEBEXACcCPyS5MvzFskvzj/TImcB44FngeeAielr7dnWv0m+bM+SnBkr/vLXkRxIfpvkYO0I4AfN1DEL2C0tO4vkDOFuETGzPW1qo5NIDvJ/StLLvLHJ+tOBqyV9JGm/UpVJ2oPk5M/R6UsnAkMlHZw+X5HkTGRLHiL5JS0EuTEkPeWHW3wHnEPyR+sjSSeVamNbdMC+GU1yzG4yybDwK5of/ja1Hcmo4Zb0WtDPJBWO97Z7n0bEHGAUsDPJWek/kATSl9rx2WqSIjxpprVM0tPAdmnwMOtyHOTMrKZVxXDVzKxSHOTMrKY5yJlZTauqm7XVc+FQr8U6uxnWBhuutVJnN8Ha4I03pjJz5sys1+Bl0mPxlSPq57uRplnx5YzRETGynNsvpbqCXK/FWHBwyaserIo8+sQlnd0Ea4MtNt247HVG/ZeZf2+/evrSrHdylE1VBTkz64oEVTxjlYOcmeUjoK6skx+XlYOcmeWnsh7mKysHOTPLqbqHq9XbMjPrOqRsS8lqNFLSy5KmSDq5mfUrS7pP0rOSHpQ0oFSdDnJmlo9IenJZltaqSbKaXUoy2cDawIGS1m5S7HfAXyNifZLZwM8p1TwHOTPLKWMvrnRPbhNgSkS8ls6ucgPzJ7NaG7gvffxAM+vn4yBnZvnV9ci2tK4/jaetmkbjGY4hmRb/2+njvYDFJC3datPa8DHMzJqhtgxX+6b5lQvLkY0rmk/TaZJOAkZIeopkvsfpJJOYtshnV80sH9GWS0hmRkRLt11MI5mktWAATdIcRMTbwN7QMG3/t5tMJT8f9+TMLL8ynHggyW+7hqRVJfUCDgBub7QZqW9RPotTgCtLVeogZ2Y5tWm42qI0496PSKaYfxG4KSImSTpT0qi02NbAy5ImA8uRJApvlYerZpZfXXnueEjzJt/d5LXTih7fAtzSljod5MwsH9+7ama1rbpv63KQM7P8fIO+mdU09+TMrGZlvPm+szjImVl+PvFgZrXLJx7MrNZ5uGpmNaswn1yVcpAzs5w8XDWzWufhqpnVNJ9dNbOaJQ9XzazWebhqZrVMDnJmVquS2c+rN8hV70DazLoGtWEpVVXp5NIrSXpA0lNpguldStXpnpyZ5STq6vL3l4qSS+9AktTmSUm3R8QLRcV+STIt+v+liafvBlZprV735MwsN0mZlhKyJJcOYPH08RI0yebVHPfkzCy3Mh2Tay659KZNypwO3CvpWGBRYPtSlbonZ2b5tO2YXN7k0gcCV0XEAGAX4JqiFIXNck/OzHIRmYaiBbmSSwOHAyMBImKspIWAvsD7LW3QPTkzy62uri7TUkLJ5NLAm8B2AJLWAhYCZrRWqXtyZpZbOY7JRUS9pEJy6R7AlYXk0sD4iLgd+AlwhaQTSIayh0ZE0yFtIw5yZpZPxmvgssiQXPoFYIu21OkgZ2a5VfMdDw5yZpZLG088dDgHOTPLzUHOzGqXQHUOcmZWw9yTM7Oa5iBnZjXLJx7MrPZVb4zzbV157bD5Wjxz6694/rZfc9J/7TDf+pVW6MPdlx3LuBtPYfQVP6b/sks2Wr/Yogvx6uizuPDn+3ZUk7u9e0f/i/XXGcw6aw7k/PPOnW/97Nmz+c5B+7POmgMZvvmmvDF1asO65559lhFbbsbQDdZh4yHr8dVXX3Vgy6uUyjbVUkU4yOVQVyd+f/J+7PGjP7Dht89i35EbseZqyzcqc84Je3HtXePYZP9z+O3l93DmsaMarf/1D3blkQlTOrLZ3dq8efM4/rgfctsd9/DUsy9w8w3X8+ILLzQqc9WVf6bPkn2Y9NIUjv3xCZz6i58DUF9fz2Hf+w7/e+llTHxmEqPve5AFFligMz5G1SnTvauVaVunbLVGDFt3FV59ayZTp89ibv08bh49kd22Xr9RmTVXW4EHn3gZgIeenMxuW6/XsG7DtVZk2aUX5z9jX+zQdndnT44bx+qrD2TV1VajV69e7Lv/Adx5x22Nytx5x20cfMj3ANj72/vw4P33ERH859/3su5667P+BhsAsPTSS9OjR/XmG+1QZZr+vBIc5HLot+wSTHvvw4bn09/7kP7LLNGozHOTp7PndkMA2GPbDVi898IstcSiSOLcE/fmFxfe2qFt7u7efns6AwZ8M5tP//4DmD59+vxlVkzK9OzZk8WXWIJZs2bxyuTJSGL3XXZis2FD+Z/fndehba9m3Xa4WiopRVenZv40NZ0O4ZQLb2X4RgMZe/3PGb7RQKa/9yH18+Zx1H7DGT1mEtPe+6hjGmsANDdhRdNfvpbK1M+r57HHxvCXv17LfQ+N4fZ/3soD999XsbZ2FVkDXGcFuYqdXc2YlKJLm/7+RwxYrk/D8/7L9eHtGR83KvPOjI854KQ/AbDowr3Yc7shfPLZV2y6/qpsseHqHLnfcBZdeEF6LdCDz76cza8ubjp9lpVT//4DmDbtmxm2p0+fRr9+/eYv89ZbDBgwgPr6ej75+GOWWmop+vcfwPDhI+jbty8AI3fehaeemsg2227XoZ+hGlXzJSSV7MllSUrRpY2f9AYDV1qGlfstzQI9e7DvTkO568FnG5VZeslFG74APz1sJ66+7XEA/uvUqxm0y2msueuvOeXCW7nuznEOcB1g42HDmDLlFaa+/jpz5szh5htvYNfdGp8M2nW3UVx7zdUA/OPvtzBim22RxA477sTzzz3LF198QX19PY88/BBrrbV2Z3yMqtMte3JkS0pBOsd7Ms/7Ar0r2Jzymzfva07475u44w8/pEeduPq2x3nxtXf51TG7MvGFN7nroefYauM1OPPYUUTAmIlTOP6cmzq72d1az549ufCiS9h9152YN28e3zv0MNZeZx3OPP00hm60MbvtPopDDzucww49hHXWHEifPktxzbU3ANCnTx+OO/5EttxsGJLYaeQu7LzLrp38iapDNd+7qhKTara/YmlfYKeI+H76/BBgk4g4tqX31C2ybCw4eL+KtMcq48MnL+nsJlgbbLHpxkyYML6sEWnB5deIAQdfnKnsaxfsMqGVHA9IGglcRDIz8J8i4twm6y8EtkmfLgIsGxGNLz5topI9uSxJKcysixNQjpFoluP4EXFCUfljgQ1L1VvJY3JZklKYWZdXtrOrbT2OfyBwfalKK9aTaykpRaW2Z2adpw09ub6Sxhc9vzwiLk8fZzqOn2xPKwOrAveX2mBFb9BvLimFmdUYJbc4ZtRa3tUsyaULDgBuiYh5pTboWUjMLBfRpiDXmrYcxz8A+GGWSn1bl5nlJmVbSsh0HF/SYKAPMDZL2xzkzCy3cpx4iIh6oHAc/0XgpkJyaUnFV2wfCNxQKql0gYerZpZPtl5aJqWSS6fPT29LnQ5yZpZLcp1c9d7x4CBnZjmpXCceKsJBzsxyc0/OzGpXGY/JVYKDnJnl4mNyZlbzqjjGOciZWX7uyZlZ7WrbvasdzkHOzHIp13xyleIgZ2Y5dV7+hiwc5MwstyqOcQ5yZpafe3JmVrPkEw9mVuvckzOzmlbFMc5Bzszyq+aenGcGNrN8Mk59niUOShop6WVJUySd3EKZ/SS9IGmSpOtK1emenJnlojJdJ5clubSkNYBTgC0i4kNJy5aq10HOzHLrUZ6zqw3JpQEkFZJLv1BU5gjg0oj4ECAi3i9VqYerZpZbG4arfSWNL1qOLKqmueTS/ZtsahAwSNKjkh6XNLJU29yTM7NckgDWYcmlewJrAFuT5GV9RNK6EfFRSxtsMchJWry1lkbEJ62tN7Puo0zXAmdJLj0NeDwi5gKvS3qZJOg92VKlrfXkJpFE0eLmF54HsFLmpptZTSvTJSQNyaWB6STJpQ9qUuafJHlXr5LUl2T4+lprlbYY5CJixZbWmZkVK0eMi4h6SYXk0j2AKwvJpYHxEXF7um5HSS8A84CfRsSs1urNdExO0gHAahHxW0kDgOUiYkKeD2RmtUFAjzJdDFwquXREBHBiumRS8uyqpEuAbYBD0pe+AC7LugEzq3FKrpPLsnSGLD25zSNiqKSnACLiA0m9KtwuM+tCqviurkxBbq6kOtJTuZKWBr6uaKvMrMsQUFfFUS7LxcCXAn8HlpF0BjAG+O+KtsrMupRy3btaCSV7chHxV0kTgO3Tl/aNiOcr2ywz6ypqZdLMHsBckiGrbwUzs0a69HBV0qnA9UA/kiuQr5N0SqUbZmZdhzIunSFLT+47wEYR8QWApLOBCcA5lWyYmXUd1TxpZpYg90aTcj0pcRuFmXUfydnVzm5Fy1q7Qf9CkmNwXwCTJI1On+9IcobVzKzhYuBq1VpPrnAGdRJwV9Hrj1euOWbWFXXJs6sR8eeObIiZdU1ddrhaIGl14GxgbWChwusRMaiC7TKzLqSah6tZrnm7CvgLScDeGbgJuKGCbTKzLqaaLyHJEuQWiYjRABHxakT8kmRWEjOz5I4HKdPSGbJcQjJbSV/0VUlHk8zYWTINmJl1H1U8Ws3UkzsB6A0cB2xBkhLssEo2ysy6lro6ZVpKKZVcWtKhkmZIejpdvl+qziw36D+RPvyUbybONDMDkuTS5RiKZkkunboxIn6Utd7WLga+lfnTgTWIiL2zbsTMalj5plHKkly6zVrryV2Sp+L2WGfQAG679/yO3qzl0Gd/X07Zlcx+bWZF6m3DJSR9JY0ven55RFyePm4uufSmzdTxbUlbAZOBEyLirWbKNGjtYuD7srXZzLq7Nsy/lje59B3A9RExOz0RejWwbZnaZmY2P0G5EtmUTC4dEbMiYnb69Apgo1KVOsiZWW4967ItJTQkl06TZR0A3F5cQNIKRU9HAS+WbFvWDyFpwaIIamYGFPI35D/zkDG59HGSRgH1wAfAoaXqzXLv6ibAn4ElgJUkbQB8PyKObfenMbOaUq4b9DMklz4FaNPM5FmGqxcDuwGz0o08g2/rMrMiXTpbF1AXEW806Y7Oq1B7zKyLqfa8q1mC3FvpkDXSK5KPJbk+xcwMgB7VG+MyBbljSIasKwHvAf9JXzMzQ504w0gWWe5dfZ/kVK6ZWbOqOMZlOrt6Bc3cwxoRR1akRWbW5XTp6c9JhqcFCwF70fj+MjPrxrr8iYeIuLH4uaRrgH9XrEVm1uVUcYzLfsdDkVWBlcvdEDProgQ9qjjKZTkm9yHfHJOrI7mVYr4ZO82se+rSKQnT3A4bkOR1APg6IlqcSNPMuqdqDnKt3taVBrRbI2JeujjAmdl8yjTVUkVkuXd1nKShFW+JmXVJheFqlqUztJbjoWdE1ANbAkdIehX4nOQzRUQ48JlZOXM8VERrx+TGAUOBPTuoLWbWBQnoWcUH5VoLcgKIiFc7qC1m1kV11Z7cMpJObGllRFxQgfaYWZcj6prNQdOOmqSRwEUkMwP/KSLObaHcPsDNwLCIGN9cmYLWglwPoDfNZ9AxMwMKiWzKUE/G5NKSFgOOA56Yv5b5tRbk3omIM9vZXjPrLsp35jRrcunfAOcBJ2WptLVLSNyDM7OSBPSoU6aFNLl00VI8m1FzyaX7N9qWtCGwYkTcmbV9rfXktstaiZl1b22YhaTdyaUl1QEXkiFDV7EWg1xEfNCWisys+yrT2dVSyaUXA9YFHkzvnlgeuF3SqNZOPrRnFhIzswaibFnqG5JLk9wvfwBwUGFlRHwM9G3YrvQgcFKps6tlapuZdVsqz72r6R1WheTSLwI3FZJLpwml28U9OTPLrVxnKUsll27y+tZZ6nSQM7NcRBefNNPMrJQqjnEOcmaWV+fNFZeFg5yZ5VLGs6sV4SBnZrm5J2dmNa16Q5yDnJnlpK6ektDMrBQPV82splVviHOQM7MyqOKOnIOcmeWTXEJSvVHOQc7McnNPzsxqmNoyaWaHc5Azs1w8XDWz2iYPV82sxlVzkKvm+2rNrItQxn8l65FGSnpZ0hRJJzez/mhJz0l6WtIYSWuXqtNBLqeH7r+X7TfbgG02WZfLLv7dfOvHjR3DqO02Y9AKi3HPHbc2WnfuGacycvhG7LjFhpzxi58QEfO938pvhyH9eebib/P8Jfty0l7rz7d+xb6L8q8zdmbs+Xsy7oK92GnoAAC2Xb8fj563B09esBePnrcHI9ZdoaObXpUKk2ZmWVqt55vk0jsDawMHNhPErouI9SJiCEnu1QtKtc9BLod58+Zx+s9P4Mrr/8noMRO54x8388rLLzYq06//ipx38eXsvvf+jV6fMO5xJowby10PjuOeh8fz3FMTeOKxRzqy+d1SXZ34/RGbs8fZ97Lh8X9n3y1XY80BSzYq8/N9hvD3x15ns5/+k+9e8AAXHbE5ALM+nc0+5/ybYSfeyhH/+zBXHjeiMz5CVZKyLSU0JJeOiDlAIbl0g4j4pOjpohSlLGyJj8nl8MzE8ay86uqstMqqAOy21z785193ssbgtRrKDFhpZQDq6hr/PZHE7NlfMXfOHCKCufVz6bvMsh3X+G5q2MBlePXdT5j63qcA3DzmNXYbthIvTfuooUwELL5wLwCWWKQX73zwBQDPvD6rocwLb33Igr160KtnHXPqv+7AT1CdsgxFU30lFWfXujwiLk8fN5dcetP5tiX9EDgR6AVsW2qDDnI5vPfu26zQ/5sE38uv0J9nJj6Z6b1Dh23Kt7YYwbfWW42I4JDDj2LgoDUr1VRL9VtqEabN/Lzh+fQPvmCTNZZpVObsGydyx2kjOWaXtVlkwZ7sesY989Wz17dW4ZnXZznAkV5Ckv3EQ7uTSze8EHEpcKmkg4BfAt9rbYMVG65KulLS+5Ker9Q2Oluzx9Aynmaa+tqrvPrKSzz69Cs89swUHn/kIcaNHVPmFlpTze2epvtxv+Gr87cHXmHgkTew19n38ufjRjR631orLslZhwzjR5c9WuHWdhVZTzuU/N0olVy6qRuAPUtVWsljclcBIytYf6dbfoX+vDN9esPzd9+ZznLLZzsYfe/dtzNko01YtHdvFu3dmxHb7chT48dVqqmWmj7rCwb0XbThef+lFuHtdDha8L3tBvH3x14H4InJ77NQrx70XWyhhvI3/mx7vn/xQ7yeDnm7vYzH4zL8/W9ILi2pF0ly6dsbbUpao+jprsArpSqtWJCLiIeBDypVfzVYf8ONmPraFN56Yypz5szhzltvYbudds303n4DVmTcY2Oor69n7ty5PDF2DAMHDa5wi238lBkMXGFxVl62Nwv0rGPfLVfjrvFvNirz1ozP2Hr9fgAM7r8ECy3QgxmffMUSi/TiH6fuyGnXjmfsy+93RvOrUrnOrmZMLv0jSZMkPU1yXK7VoSpUwTE5SUcCR0Lyi9+V9OzZk1+fewGH7j+Kr+fNY5+DvsugNdfmwnPPZL0hQ9l+5G48+9R4jjn0AD7++CPuv/duLjrvLP71yAR23n0vxj7yILuMGIYkttpmh8wB0tpv3tfBCX8ayx2/GkmPOnH1/ZN58a2P+NUBQ5k4ZSZ3jX+Tk68exx+O2ZJjd1uHCDjikuSs99E7r83qyy/OyfsM4eR9hgCw+5n/YsYnX3XmR6oKHZVcOiJ+3NY6VclrsyStAtwZEetmKb/ekKFx2799nKMrWefo6zq7CdYGs+//DV9/OLWs9yestd6G8Zd/PpCp7GYD+0xo5cRDRXR6T87Mur42XELS4RzkzCy3bnnvqqTrgbHAYEnTJB1eqW2ZWedSxqUzVKwnFxEHVqpuM6sewtm6zKyWeT45M6t1VRzjHOTMrAyqOMo5yJlZTtkmxOwsDnJmlksbZyHpcA5yZpafg5yZ1TIPV82spvkSEjOraVUc4xzkzCynzrxnKwMHOTPLJTm7Wr1RzikJzSy3ct2gnyG59ImSXpD0rKT7JK1cqk4HOTPLrwxRLmNy6aeAjSNifeAWkgTTrXKQM7PcypStK0ty6QciopB56HGSjF6tcpAzs9zakK2rr6TxRcuRRdU0l1y6Py07HJg/KW4TPvFgZrm14bRD7uTSAJK+A2wMjCi1QQc5M8uljJNmZkouLWl74FRgRETMLlWph6tmlk/HJpfeEPgjMCoiMiW/dZAzs9zKcQlJxuTS5wO9gZslPS3p9haqa+DhqpnlV6ZrgTMkl96+rXU6yJlZTp4008xqmCfNNLPa5yBnZrXMw1Uzq2lVPAmJg5yZ5VfFMc5Bzsxyynahb6dxkDOzXMp4W1dFOMiZWW7VG+Ic5MysDKq4I+cgZ2b5+RISM6tt1RvjHOTMLL8qjnEOcmaWj1TdKQkd5Mwsv+qNcQ5yZpZfFcc4zwxsZvmVafrzLMmlt5I0UVK9pH2ytM1Bzsxyypp1tfUolzG59JvAocB1WVvn4aqZ5ZLc1lWWqhqSSwNIKiSXfqFQICKmpuu+zlqpe3JmllsnJZfOxD05M8utDXc8lCW5dFs4yJlZPuWbailTcum28nDVzHLJmnM1QxwsmVy6PRzkzCy/MkS5LMmlJQ2TNA3YF/ijpEmlmubhqpnlVq7bujIkl36SZBibmYOcmeVWzXc8OMiZWX5VHOUc5Mwst2qeNFMRuS9DKRtJM4A3OrsdFdAXmNnZjbA2qdV9tnJELFPOCiX9i+TnlcXMiBhZzu2XUlVBrlZJGt/KBZBWhbzPaocvITGzmuYgZ2Y1zUGuY1ze2Q2wNvM+qxE+JmdmNc09OTOraQ5yZlbTHOTMrKY5yFWIpMGSNpO0QDp3vXUB3le1xyceKkDS3sBvgenpMh64KiI+6dSGWYskDYqIyenjHhExr7PbZOUnqmxiAAAEWUlEQVThnlyZSVoA2B84PCK2A24jme30Z5IW79TGWbMk7QY8Lek6gIiY5x5d7XCQq4zFgTXSx7cCdwK9gIOkMk28ZWUhaVGSiRqPB+ZI+hs40NUSB7kyi4i5wAXA3pKGR8TXwBjgaWDLTm2czSciPgcOI8njeRKwUHGg68y2WXk4yFXGI8C9wCGStoqIeRFxHdAP2KBzm2ZNRcTbEfFZRMwEjgIWLgQ6SUMlrdm5LbQ8PJ9cBUTEV5KuJUmndkr6SzIbWA54p1MbZ62KiFmSjgLOl/QS0APYppObZTk4yFVIRHwo6QqS7N9HAV8B34mI9zq3ZVZKRMyU9CywM7BDREzr7DZZ+/kSkg6QHsCO9PicVTlJfYCbgJ9ExLOd3R7Lx0HOrBmSFoqIrzq7HZafg5yZ1TSfXTWzmuYgZ2Y1zUHOzGqag5yZ1TQHuS5E0jxJT0t6XtLNkhbJUdfWku5MH4+SdHIrZZeU9IN2bON0SSdlfb1Jmask7dOGba0i6fm2ttFqn4Nc1/JlRAyJiHWBOcDRxSuVaPM+jYjbI+LcVoosCbQ5yJlVAwe5rusRYGDag3lR0h+AicCKknaUNFbSxLTH1xtA0khJL0kaA+xdqEjSoZIuSR8vJ+lWSc+ky+bAucDqaS/y/LTcTyU9KelZSWcU1XWqpJcl/QcYXOpDSDoirecZSX9v0jvdXtIjkian0yEhqYek84u2fVTeH6TVNge5LkhST5Jbjp5LXxoM/DUiNgQ+B34JbB8RQ0km7DxR0kLAFcDuwHBg+Raqvxh4KCI2AIYCk4CTgVfTXuRPJe1IMpXUJsAQYCNJW0naCDgA2JAkiA7L8HH+ERHD0u29CBxetG4VYASwK3BZ+hkOBz6OiGFp/UdIWjXDdqyb8r2rXcvCkp5OHz8C/JlkZpM3IuLx9PVvAWsDj6ZT1/UCxgJrAq9HxCsA6SwbRzazjW2B70LDVEMfp7c5FdsxXZ5Kn/cmCXqLAbdGxBfpNm7P8JnWlXQWyZC4NzC6aN1N6a1wr0h6Lf0MOwLrFx2vWyLd9uQM27JuyEGua/kyIoYUv5AGss+LXwL+HREHNik3hGRWlHIQcE5E/LHJNo5vxzauAvaMiGckHQpsXbSuaV2RbvvYiCgOhkhapY3btW7Cw9Xa8ziwhaSBAJIWkTQIeAlYVdLqabkDW3j/fcAx6Xt7pFO2f0rSSysYDRxWdKyvv6RlgYeBvSQtLGkxkqFxKYsB76TTxh/cZN2+kurSNq8GvJxu+5i0PJIGpbP7mjXLPbkaExEz0h7R9ZIWTF/+ZURMlnQkcJekmSSzFa/bTBU/Bi6XdDgwDzgmIsZKejS9ROOe9LjcWsDYtCf5Gck0UhMl3UgyC/IbJEPqUn4FPJGWf47GwfRl4CGSefiOTufp+xPJsbqJSjY+A9gz20/HuiPfoG9mNc3DVTOraQ5yZlbTHOTMrKY5yJlZTXOQM7Oa5iBnZjXNQc7Matr/AwzB2ezkHPVoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot confusion matrix (normalized)\n",
    "plot_confusion_matrix(cm, classes=[0,1],normalize=True,\n",
    "                      title='Confusion matrix, with normalization')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also quantify the model performance using area under ROC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auROC = 0.8791484786391871\n"
     ]
    }
   ],
   "source": [
    "# calculate auROC\n",
    "auroc_test = roc_auc_score(test_values, y_classes)\n",
    "print('auROC =', auroc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation and prediction with the model of lowest validation loss\n",
    "***\n",
    "Since the final model might be a little bit overfitting according to the training histry, we went back and found the model with lowest validation loss (around epoch 220) that we saved as checkpoints, and evaluated the performance with it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load best model\n",
    "best_model_file = \"models_10000/model.220-0.30-0.88.h5\"\n",
    "best_model = load_model(best_model_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18612500/18612500 [==============================] - 503s 27us/step\n",
      "Loss on test data = 0.3220961782173249 \n",
      "Accuracy on test data = 0.8845281934184144\n"
     ]
    }
   ],
   "source": [
    "# evaluate test data with the best model\n",
    "test_target_best = keras.utils.to_categorical(test_values, num_classes=2)\n",
    "test_result_best = best_model.evaluate([test_playlists, test_tracks],test_target)\n",
    "print('Loss on test data =', test_result_best[0],\n",
    "      '\\nAccuracy on test data =', test_result_best[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss on test data (class balanced) = 0.313746480345726 \n",
      "Accuracy on test data (class balanced) = 0.8772099989652634\n"
     ]
    }
   ],
   "source": [
    "# evaluate test data (class balanced) with the best model \n",
    "test_result_best_gen = best_model.evaluate_generator(valGenerator_balance(test_tracks_ones, test_playlists_ones, \n",
    "                                                                          test_tracks_zeros, test_playlists_zeros, \n",
    "                                                                          ohe, val_batch_size= 10000, balance_ratio = 0.5), \n",
    "                                                 steps = 100, verbose = 0)\n",
    "print('Loss on test data (class balanced) =', test_result_best_gen[0],\n",
    "      '\\nAccuracy on test data (class balanced) =', test_result_best_gen[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the final model and the model with lowest validation loss performed similarly on class balanced test data, but the final model out-performed on the test data with real class ratio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted proportion of ones = 0.11648354600402955\n"
     ]
    }
   ],
   "source": [
    "# get prediction on test data with the best model \n",
    "y_prob_best = best_model.predict([test_playlists, test_tracks])\n",
    "y_classes_best = y_prob_best.argmax(axis=-1)\n",
    "print('predicted proportion of ones =', np.mean(y_classes_best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[16441122  2145891]\n",
      " [    3328    22159]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAToAAAEYCAYAAADMJjphAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XeYFFXWx/HvbwYBFRTfRQwI4ipRJAoYAVdAcM0RZHVVzKuuAROy5pwTuy66iBGziCwrrBFEQLIKimJAB1SCioKEYTjvH1Uz2zPMdNdMd9M9PefjU49d6d5bXc2Ze29V3ZKZ4ZxzuSwv0wVwzrl080DnnMt5HuiccznPA51zLud5oHPO5TwPdM65nFdtA52kLSW9JmmlpBeSSGegpAmpLFumSDpQ0oJsyU9SM0kmqdbmKlN1IelrSb3Cz0MkPZqGPB6W9LdUp1sdKd330Uk6CbgEaAX8CswBbjaz95JM92TgAmA/M9uQdEGznCQDmpvZwkyXpSKSvgbOMLM3wvlmwFfAFqk+R5JGAgVmNjSV6W4uZb+rFKR3apjeAalIL9ektUYn6RLgPuAWYAegKfB34MgUJL8r8FlNCHJReK0pffy7zQFmlpYJ2BZYBRwfZ5s6BIFwSTjdB9QJ1/UECoBLgaXAd8Bp4brrgfVAYZjHIOA64KmYtJsBBtQK508FviSoVX4FDIxZ/l7MfvsB04GV4f/3i1n3DnAjMDlMZwLQsIJjKy7/5THlPwo4FPgM+BEYErN9V2AK8HO47UNA7XDdxPBYVofHe2JM+lcA3wNPFi8L99k9zKNTOL8zsBzoGeHcPQ5cGn5uHOZ9Xji/R5iuyuT3JLARWBOW8fKYc/Bn4Jsw/6sjnv9S5yVcZmH+Z4Xnfn2Y12sVHIcB5wCfAz8Bw/hfKyYPGAosCs/PE8C2ZX47g8JyT4xZdhrwbZjeOUAX4MPwvD0Uk/fuwFvAivC4nwYaxKz/GugVfr6O8LcbnvdVMdMG4Lpw3ZXAFwS/vfnA0eHy1sBaoCjc5+dw+Ujgppg8zwQWhudvDLBzlO8qF6Z0Brq+4UmqFWebG4CpQCNge+B94MaYQLEh3GYLggDxG7Bd2R9HBfPFP8xawNbAL0DLcN1OwJ5l/0EB/xee5JPD/QaE878L178T/tBaAFuG87dVcGzF5b8mLP+ZwDLgGaA+sGf44/x9uH1nYJ8w32bAJ8BFZf+Rl5P+7QQBY0tiAk/MD/sTYCtgPHBXxHN3OmHwAE4Kj/m5mHWvxpQhNr+vCf/xljkHj4Tlaw+sA1pHOP8l56W874Ay/4grOA4DxgINCFoTy4C+McexEPg9UA94GXiyTLmfIPjtbBmz7GGgLtAnPH+jw/I3JgiYPcI09gB6h+dme4JgeV953xVlfrsx23QIy9wxnD+e4A9WHsEfu9XATnG+r5LvCPgDQcDtFJbpQWBilO8qF6Z0Nl1/Byy3+E3LgcANZrbUzJYR1NROjllfGK4vNLNxBH+tWlaxPBuBtpK2NLPvzGxeOdv8EfjczJ40sw1mNgr4FDg8ZpvHzOwzM1sDPE/wY6xIIUF/ZCHwLNAQuN/Mfg3znwe0AzCzmWY2Ncz3a+CfQI8Ix3Stma0Ly1OKmT1C8Bd6GkFwvzpBesXeBQ6UlAd0B+4A9g/X9QjXV8b1ZrbGzOYCcwkCHiQ+/6lwm5n9bGbfAG/zv/M1ELjHzL40s1XAVUD/Ms3U68xsdZnv9kYzW2tmEwgCzaiw/IuBSUBHADNbaGb/Dc/NMuAeEp/PEpK2JwiiF5jZ7DDNF8xsiZltNLPnCM5t14hJDgRGmNksM1sXHu++YT9qsYq+KySNkLRU0scRyn6vpDnh9JmknyOWMW3SGehWAA0T9G/sTNB0KLYoXFaSRplA+RvBX99KMbPVBH8BzwG+k/RvSa0ilKe4TI1j5r+vRHlWmFlR+Ln4H8sPMevXFO8vqYWksZK+l/QLQb9mwzhpAywzs7UJtnkEaAs8GP7AEzKzLwj+qHQADiT4S79EUkuqFugq+s4Snf9UqEzetQj6kot9W056Zc9fReezkaRnJS0Oz+dTJD6fhPtuAbwIPGNmz8YsPyUMHj+HwaNt1DQpc7xhcF9B9N/2SIJWWkJmdrGZdTCzDgQ1x5cjljFt0hnophBU7Y+Ks80SgosKxZqGy6piNUETrdiOsSvNbLyZ9Sao2XxKEAASlae4TIurWKbK+AdBuZqb2TbAEIJ+sHjiXjKXVI+g3+tfwHWS/q8S5XkXOI6gn3BxOH8KsB3BlfNKl6cc8c5/qfMpqdT5rEJeUfLeQOnAlUwet4b7twvP559IfD6LPUjQD1dyRVnSrgS/2fMJulIaAB/HpJmorKWOV9LWBK2uSL9tM5tI0LdXQtLukl6XNFPSpAoqDwOAUVHySKe0BTozW0nQPzVM0lGStpK0haR+ku4INxsFDJW0vaSG4fZPVTHLOUB3SU0lbUtQNQdA0g6SjghP7jqC2kpROWmMA1pIOklSLUknAm0IajTpVp+gH3FV+IM5t8z6Hwj6kyrjfmCmmZ0B/JugfwkASddJeifOvu8S/KOaGM6/Q3A7z3sxtdSyKlvGeOd/LrCnpA6S6hL0YyWTV3l5Xyxpt/APwi0E/ZCpuopfn/DCgKTGwGVRdpJ0NkGt+SQz2xizamuCYLYs3O40ghpdsR+AXSTVriDpZ4DTwu+zDsHxTgu7SapqOEHTujMwmOCOithj2RXYjeCiTEal9fYSM7uH4B66oQQn6FuCfzyjw01uAmYQXLX6CJgVLqtKXv8FngvTmknp4JRHcPV2CcFfpR7AeeWksQI4LNx2BcGVw8PMbHlVylRJgwk6/n8l+Mv9XJn11wGPh82WExIlJulIgqbGOeGiS4BOkgaG800Irh5X5F2Cf6zFge49ghrWxAr3CGoxQ8MyDk5URuKcfzP7jOBixRsEfVFl77v8F9AmzGs0lTeC4ErxRIKr8GsJAnmqXE/Q8b+S4I9M1ObbAIIAvkTSqnAaYmbzgbsJWko/AHtR+vy9RdDn+72kTX6vZvYm8DfgJYKr+rsD/atyYFDSWtgPeEHSHII+5Z3KbNYfeDHOH8bNJu03DLvsFP44Dw6Du3MJhRcuxppZW0nbAAvMrGxwi91+NvAXM3t/MxWxQtX2ETCXnLCz2IOcqxIz+wX4StLxAAoUX00nvHC1HUENNOM80DnnEpI0iiBotZRUIGkQwS0rgyTNJWg2xz7xNAB41rKkyehNV+dczvManXMu52XVw8qqtaWpdv1MF8NVwp4tdsl0EVwlLP72G35csTzq/XyR5G+zq9mGTR7MKZetWTbezCLdeJxK2RXoatenTsuEd064LPLqhDszXQRXCUf23j/xRpVkG9ZE/ne7ds6wqE9ypFRWBTrnXHUkUHb3gmV36Zxz2U9AXn60KVFSEQYPkNQzfOZ3nqRIz117oHPOJU+KNiU2kjiDB0hqQPCo2RFmtifB0FUJedPVOZek1DVdzWximaGjyjoJeDkcSgozWxolXa/ROeeSF71G11DSjJjprErm1ALYTtI74agpp0TZyWt0zrnkiMrU6Jab2d5J5FaLYDTugwlGfp4iaWo4CETcnZxzLgmR+99SoYAgWK4GVkuaSDBiddxA501X51zyUnTVNYJXCYb5ryVpK6AbwXtR4vIanXMuSam7GBEOHtCToC+vALiW4OVSmNnDZvaJpNcJxjDcCDxqZgnfY+GBzjmXHJGypquZDYiwzZ1ApR7J8UDnnEtelj8Z4YHOOZek7H8EzAOdcy55eZvtqmuVeKBzziWn+FnXLOaBzjmXJG+6Oudqgs13w3CVeKBzziXPa3TOuZwWfQimjPFA55xLnl+McM7lNr8Y4ZyrCbzp6pzLaZUbjy4jPNA555LkTVfnXE3gTVfnXM7zq67OuZwmb7o652oCb7o653KdsjzQZXd90zmX9YKR1BVpSpiWNELSUklx3wMhqYukIknHRSmjBzrnXHJUiSmxkUDfuNlJ+cDtwPioRfSmq3MuSSIvLzV1JjObKKlZgs0uAF4CukRN1wOdcy5pleijayhpRsz8cDMbXol8GgNHA3/AA51zbnOqRKBbbmZ7J5HVfcAVZlZUmQsgHuicc8mJ3v+WCnsDz4ZBriFwqKQNZjY63k4e6JxzSRHRrqimgpntVpKvNBIYmyjIgQc651wKpOpihKRRQE+CvrwC4FpgCwAze7iq6Xqgc84lLVU1OjMbUIltT426rQc651xyNm8fXZV4oHPOJS3bHwHzQOecS8rmvBhRVR7onHNJ80DnnMttAuV5oHPO5Tiv0Tnncp4HOudcTvOLEc65miG745wPvAnw8LUDWfTmrcx4YUip5ef278HcV/7GzBev5ua/HllqXZMdt2PZ5Lu56OSDSy3PyxNTRl3BS/efU7LsnBO78/Gr17Jm9kP8rsHWJctbNNuBdx6/lJ+n3VsqnV12aMDrwy9k9ktDmfni1fxlQM+SdbdcdBRzXh7KB89dxXN3n8m29bZMxVdQrS1ZXMBJR/elz/4d6XtgZx4bPgyAcWNepu+Bndljh635cM7MTfcr+Ja9mm3PI8PuK1nWvXMr+vXowmEHdePI3vuXLJ//0VyO7dejZPncWdMBWPnzT5zz5xM5tEdXjj7kQBZ8Mq9knyv+ejZd2uxK3+7JDNZRDSh1Iwyniwc64MnXpnLkX4aVWtZ97+Yc1nMvupxwK52Pu5n7nniz1Po7Bh/LhMnzKOv8kw5iwVc/lFo2Zc6XHHrOgyxasqLU8p9WrubS21/gvifeKrV8Q9FGrrznZToeexM9TrmLs0/sTqvf7wjAm1M/pfPxt9D1xFv5fNFSLju9T5WPO1fUqpXPkOtvZcLk2bz4n3d4asQ/+XzBJ7Ro1Ya/PzaKrvseUO5+N/3tcnocvOn39/TL/2Hs29N49b+TS5bdfsNQLhg8hLFvT+OiK/7G7TcMBeDv991Jm7btGPfuB9z10KPcOPSykn2O7X8yjz2b8HnznJCXlxdpylj5MpZzFpk86wt+XPlbqWVnHX8gdz32X9YXbgBg2U+rStYd3rMdXxUsZ/4X35fap3GjBvQ9YE8ee+X9UsvnLijgm+9+3CTfZT+tYub8byjcUFRq+ffLf2HOpwUArPptHZ9+9T07b98ACAJdUdFGAD746Csa79CgKoecUxrtsBNt23UEoF69+uzRoiU/fLeEPVq04vd7tCh3nwnjxtBk191o3rJ1pDwkserXXwH49ZdfaLTjTgAs/OwT9jvwIAB2b96Sxd8sYvnS4A9d130PoEGD/0vq2KqN1A2lnhYe6Cqwx66N2L/j7kx8YjATHv0rnds0BWCrurW59LTe3PzPcZvsc+dlx3L1/aPZuNFSVo6mO/0fHVruwvSPv95k3SlH7sv4yfNTllcuKPhmEfM+mkv7zhUPPvvb6tUMf/AeLhw8ZJN1kjj1hMM5otd+jHriXyXLh950B7ddP4T9OzTntuuu4rKrbwCg9Z57Mf7frwIwd9Z0Fhd8w3ffLU7xUWW/Gt10ldRX0gJJCyVdmc68Uq1Wfh7bbbMV3U+5iyH3juapO04H4G/n/pEHn3qL1WvWl9q+34FtWfrjr8z+5NuUlWHrLWsz6q4zuOyul/h19dpS6y4fdAhFRRt5dtz0lOVX3a1etYrzTh/A3268g/r1t6lwu/vuuInTzrmArevV22Td82PfZMybUxgxajRPjRjOB1PeA+DpkY8w9IY7mDznc66+8Q6uvOhcAM6+cDArV/7EYQd144lHH6bNXu2plV+zrvFFDXKZDHRpOyPhm3qGAb2BAmC6pDFmVi2qIIt/+JnRb84FYMa8RWzcaDTcrh5d2u7K0b06cPNFR7Ft/S3ZuNFYu76QnRs14LAee9H3gD2pU3sLttm6LiNuOoXThz5Rpfxr1cpj1F1n8tx/ZvDqW3NLrRt4eDcO7d6Wfmc/kPRx5orCwkL+cvpJHHlsfw457Ki4286dNZ3Xx77C7TdczS8rV5KXl0edunU4ZdC57LDjzgA03L4RfQ49nLmzZtB13wN4+bmnuebmuwA49IhjGHLxeQDUr78NdzwQvPLAzOixd2t22bVZ+g40S9Xk20u6AgvN7EsASc8CRwLVItC99s6H9OzagkkzP2ePpo2ovUUtlv+0il6D/neF7uqzD2X1b+t4+LmJAFzz4BgADuzcnItOObjKQQ6CK8ELvvqeB54qfaGi936tufTUXvQ5437WrC2scvq5xMy48qJz2b1FSwade2HC7Z977Y2Sz/ffcRNbbV2PUwady2+rV7PRNlKvXn1+W72aSe+8yQWDrwJghx13Ytr7k9hn/+68P+kddv397gD8svJn6m65FbVr1+a5px6jyz4HxK1N5qqaHOgaA7HtuAKgW9mNJJ0FnAXAFps2JTaHx289lQM7N6dhg3osfP1Gbnx4HI+PnsI/rxvIjBeGsL6wiDOuebLK6Z83oAeX/LkXO/xuG6Y/P4TX35vHeTc8ww6/q8/kpy+n/tZ12WjG+QN70vHYm9mr+c4MPKwbH322mKnPBi3+ax8aw/j35nPvFSdQp3Ytxv7jfAA++OhrLrz52ZR8D9XVzGlTGP3CM7Rs3ZbDDgp+YpdefT3r163jhiGX8uOK5Zxx0rG0aduOkc+PqTCd5cuWcu6p/QEoKtrA4cecQI8/BFdlb7l7GDcMHUzRhiLq1K3DzXc/BMDCzxYw+PwzyM/PZ48Wrbjtvn+UpPfXs//MtMkT+enHFezffg/+evlQThh4apq+hczK9mddZZa6jvNSCUvHA4eY2Rnh/MlAVzO7oKJ98rZqZHVanpCW8rj0mDfhzkwXwVXCkb3356M5s1Ialers2Nx2GRitG+XLew6dGe8tYJJGAIcBS82sbTnrBwJXhLOrgHPNbG7Z7cpK58WIAqBJzPwuwJI05uecywABUrQpgpFA3zjrvwJ6mFk74EYg0jth0xnopgPNJe0mqTbQH6i43eCcq6ZSd9XVzCYCm950+r/175vZT+HsVIIKVEJp66Mzsw2SzgfGA/nACDPb9FEC51y1V4lrEQ0lzYiZH25mkWpl5RgE/CfKhmm94cfMxgGb3lnrnMsdCp7xjmh5vD66yFlKBxEEuvKf7yujZt3Z6JxLOVGpQJd8flI74FGgn5mtSLQ9eKBzzqXA5rqNTlJT4GXgZDP7LOp+Huicc0lL1Q3DkkYBPQn68gqAa4EtAMzsYeAa4HfA38M8N0RpCnugc84lJ/qtIwmZ2YAE688Azqhsuh7onHNJCe6jy+4nIzzQOeeSpM16MaIqPNA555LmNTrnXG5LYR9dunigc84lxfvonHM1QpbHOQ90zrnkeY3OOZfbKvesa0Z4oHPOJaV4PLps5oHOOZekzL7hKwoPdM65pGV5nPNA55xLntfonHM5TX4xwjlXE3iNzjmX87I8znmgc84lz2t0zrnc5g/1O+dynarBfXTpfIG1c66GyM9TpCkRSSMkLZX0cQXrJekBSQslfSipU5TyeaBzziVNijZFMBLoG2d9P6B5OJ0F/CNKoh7onHNJCYKYIk2JmNlE4Mc4mxwJPGGBqUADSTslSrfCPjpJ2yQo0C+JEnfO1QyVuF+4oaQZMfPDzWx4JbJqDHwbM18QLvsu3k7xLkbMA4xgcIJixfMGNK1E4ZxzOawSFyOWR3kPa7ysyllmiXaqMNCZWZMkCuOcq0E240XXAiA2Nu0CLEm0U6Q+Okn9JQ0JP+8iqXOViuicyzkC8qVIUwqMAU4Jr77uA6w0s7jNVohwH52kh4AtgO7ALcBvwMNAl+TK65zLCREvNERLSqOAngR9eQXAtQTxBzN7GBgHHAosJIhFp0VJN8oNw/uZWSdJs8PMfpRUu9JH4JzLWalquprZgATrDfhLZdONEugKJeURdvhJ+h2wsbIZOedyk4C8HHgyYhjwErC9pOuB94Db01oq51y1ksIbhtMiYY3OzJ6QNBPoFS463szKfTzDOVfz5NLAm/lAIUHz1Z+mcM6VUu2brpKuBkYBOxPcs/KMpKvSXTDnXPWhiFOmRKnR/QnobGa/AUi6GZgJ3JrOgjnnqo9sH6YpSqBbVGa7WsCX6SmOc666Ca66ZroU8cV7qP9egj6534B5ksaH830Irrw651xKbxhOl3g1uuIrq/OAf8csn5q+4jjnqqNqe9XVzP61OQvinKueqnXTtZik3YGbgTZA3eLlZtYijeVyzlUj2d50jXJP3EjgMYLA3Q94Hng2jWVyzlUz2X57SZRAt5WZjQcwsy/MbChwUHqL5ZyrLqTghuEoU6ZEub1knYJ66ReSzgEWA43SWyznXHWS5S3XSIHuYqAecCFBX922wOnpLJRzrnqptlddi5nZtPDjr8DJ6S2Oc666EZltlkYR74bhV4jz0gkzOyYtJXLOVS8ZHoIping1uoc2WylCHVs3ZfK0zZ6tczVG7fz0DD6UwqHU+wL3E4yY9KiZ3VZmfVPgcaBBuM2VZjYuUbrxbhh+M6kSO+dqjFSET0n5BAP99iZ429d0SWPMbH7MZkOB583sH5LaELxDotnmKJ9zrgYTQY0uypRAV2ChmX1pZusJ7tc9ssw2BmwTft6WCK86hOgDbzrnXIVqRa8yNZQ0I2Z+uJkNDz83Br6NWVcAdCuz/3XABEkXAFvzv5HP45cvaukk1TGzdVG3d87VDMH7ICL30S03s70rSqqcZWUviA4ARprZ3ZL2BZ6U1NbM4r6wK8oIw10lfQR8Hs63l/Rgov2cczVHnqJNCRQATWLmd2HTpukggsdQMbMpBM/fN0xYvgjH8ABwGLAiTHwu/giYcy5Git4CNh1oLmm38N3R/YExZbb5Bjg4yFOtCQLdskQJR2m65pnZojJV06II+znnaoBUvdfVzDZIOh8YT3DryAgzmyfpBmCGmY0BLgUekXQxQbP21PCl1nFFCXTfSuoKWHj59wLgs6oejHMu9+Sn6Ibh8J64cWWWXRPzeT6wf2XTjRLoziVovjYFfgDeCJc55xzK8MgkUUR51nUpQVvZOefKleVxLtIIw49QzjOvZnZWWkrknKt2snzwkkhN1zdiPtcFjqb0TX3OuRosVRcj0ilK0/W52HlJTwL/TVuJnHPVTpbHuSo9ArYbsGuqC+Kcq6YE+Vke6aL00f3E//ro8oAfgSvTWSjnXPVR7V93GL4roj3BeyIANka5Oc85V7Nke6CL+whYGNReMbOicPIg55zbRIqGaUqbKM+6fiCpU9pL4pyrloqbril4qD9t4r0zopaZbQAOAM6U9AWwmuC4zMw8+Dnnqv07Iz4AOgFHbaayOOeqIQG1sryTLl6gE4CZfbGZyuKcq6aqc41ue0mXVLTSzO5JQ3mcc9WOyCt3cODsES/Q5QP1KH94Y+ecA4pfjpPpUsQXL9B9Z2Y3bLaSOOeqpwxfUY0iYR+dc87FIyA/yyNdvEB38GYrhXOuWsv20UsqvGHYzH7cnAVxzlVfKXo5DpL6SlogaaGkcp+pl3SCpPmS5kl6Jkr5/AXWzrmkiGiPWCVMJ3gnzTCgN8GrD6dLGhO+J6J4m+bAVcD+ZvaTpEZR0k5F+ZxzNZlS9qxrV2ChmX1pZuuBZ4Ejy2xzJjDMzH6Cklc9JOSBzjmXNEWcgIaSZsRMsa9kaEzp0csLwmWxWgAtJE2WNFVS3yjl86arcy4polIDby43s73jJFVW2RGTagHNgZ7ALsAkSW3N7Od4mXqNzjmXtBRdjCgAmsTM7wIsKWebV82s0My+AhYQBL64PNA555IUrX8uQh/ddKC5pN0k1SZ4zeqYMtuMBg4CkNSQoCn7ZaKEPdA555JSfNU1yhRPOCzc+cB44BPgeTObJ+kGSUeEm40HVkiaD7wNXGZmKxKV0fvonHNJS9XowWY2DhhXZtk1MZ8NuCScIvNA55xLWnY/F+GBzjmXJOXC6w6dcy6RTL74JgoPdM65pGV3mPNA55xLgSyv0Hmgc84lJ7i9JLsjnQc651zSvEbnnMtxyvqBNz3QOeeS4k1X51zuizh6cCZ5oHPOJc0DnXMu5ynLm64+ekkVrF27lgP27UrXTu3p1H5Pbrz+WgDOOXMQXTu1p0vHdgw48ThWrVoFwP333kPHdm3o0rEd/foczKJFi0rSGnLl5XRqvycd9mrNJRddSPDMskuFb7/9lkN6HUSHvVrTqf2ePPTA/QBcdcVltG/bii4d23HCcUfz88/BmI0rVqzgkF4H0bBBPS668PxSafU5uCft9mxJt84d6Na5A0uXBiN4L1q0iH59DqZLx3b0ObgnBQUFm/cgs0DxwJtRpkzxQFcFderU4fX/vsUHs+YybcYcJox/nWlTp3LH3ffyway5TJ/9IU2aNOUff38IgA4dOzJ56gymz/6Qo485jquvuhyAKe+/z5T3JzN91ofMnPMxM2dMZ9LEdzN5aDmlVq1a3HbH3cz56BPefW8q/3x4GJ/Mn8/BvXozc87HTJ/9Ic2bt+DO228FoG7dulxz3Y3cevtd5ab32ONPM23mHKbNnEOjRsE7Wa66YjAD/3QK02d/yJCh13DN1VdttuPLJql6C1i6eKCrAknUq1cPgMLCQjYUFiKJbbbZBgAzY+2aNSXP//XoeRBbbbUVAF277cPi8K++JNatXcv69etZt24dGwoLadRohwwcUW7aaaed6NipEwD169enVavWLFmymF69+1CrVtBrE3s+tt56a/Y/4ADq1q0bOY9PP5lPzz8Er0Du0fMgxr72aoqPonpQxP8yxQNdFRUVFdGtcwea7tyIP/TqTddu3QA4a9BpNNtlRxYs+JTz/nLBJvuNfOxfHNK3HwD77Lsv3XsexG5NdmK3JjvRq88htGrderMeR02x6OuvmTNnNl26diu1/ImRI0rORyJnn3Ea3Tp34NabbyzpYtirXXtGv/wSAK+OfoVff/2VFSsSjgOZUwTkKdqUKWkLdJJGSFoq6eN05ZFJ+fn5TJs5h4VfFzBj+gfM+zg4zOH/eowvv1lCq1atefH550rtM+rpp5g1cwYXX3oZAF8sXMiCTz9h4dcFfLFoMe+8/RbvTZq42Y8l161atYoBJxzLnXffV1LrBrj91pvJr1WL/icNTJjGY088zYw5H/HGO5OY/N4knnnqSQBuvf0uJk2F+hN5AAAK3klEQVR6l3327sikie+yc+PGJbXFmiNqfS43a3QjgUivIqvOGjRoQPcePZkw4fWSZfn5+Rx3womMfuWlkmVvvfkGt992My++MoY6deoA8Oqrr9C12z7Uq1ePevXqcUjffkybNnWzH0MuKywsZMAJx3LigIEcdfQxJcufeuJxxv17LCOfeDrSEEONGwdv3atfvz4n9j+J6dM/AGDnnXfmuRdeZuqM2Vx/480AbLvttmk4kiwWsX8uSh+dpL6SFkhaKOnKONsdJ8kkVfRGsVLSFujMbCLwY7rSz6Rly5aVXKlbs2YNb735Bi1atOSLhQuBoI/u32Nfo0XLVgDMmT2b8887mxdfHlPSiQ3QpElTJk18lw0bNlBYWMikie/SqpU3XVPFzDjnzEG0bNWav178v5G3J4x/nbvvup0XXxlT0ncaz4YNG1i+fDkQBM5x48ay555tAVi+fDkbN24E4M7bb+XPp56ehiPJbqm66iopHxgG9APaAAMktSlnu/rAhcC0qGXMeB07fIHtWQBNmjbNcGmi+f677zjz9D9TVFTERtvIscedQL9D/8jBPQ/k119+wTD22qs9Dwz7BwBDrryM1atWMbD/8UBwnC++MoZjjj2Od99+i7077oUkevfpyx8POzyTh5ZT3p88mWeefpK2bfeiW+cOAFx/0y1cevGFrFu3jsP69gaCCxIP/v1hAFru0Yxff/mF9evX89qY0YwdN4Gmu+7KEYceQmFhIUUbizjoD704/YwzAZj47jtcM/QqJHHAAd2578FhmTnYDEtRo7QrsNDMvgSQ9CxwJDC/zHY3AncAgyOXL533bUlqBow1s7ZRtu/ceW+bPG1G2srjXE23f7e9mTlzRko7y1rv1dEeG/12pG333WO7RcDymEXDzWw4BM1RoK+ZnRHOnwx0M7OSmxoldQSGmtmxkt4BBptZwqCR8Rqdc676q8SFhuVmVlG/WnmJlNTEJOUB9wKnVqpweKBzzqVAim4GLgCaxMzvAiyJma8PtAXeCS8g7QiMkXREolpdOm8vGQVMAVpKKpA0KF15OecySxGnBKYDzSXtJqk20B8YU7zSzFaaWUMza2ZmzYCpQMIgB2ms0ZnZgHSl7ZzLHiI1bwEzsw2SzgfGA/nACDObJ+kGYIaZjYmfQsW86eqcS04Kn2M1s3HAuDLLrqlg255R0/VA55xLWnYP0uSBzjmXClke6TzQOeeSlNnnWKPwQOecS0rx6CXZzAOdcy55Huicc7nOm67OuZznbwFzzuW8LI9zHuicc0mK+HxXJnmgc84lJbjqmt2RzgOdcy5p2R3mPNA551IhyyOdBzrnXNL89hLnXM7L8i46D3TOueRleZzzQOecS06qBt5MJw90zrnkpHDgzXTxQOecS1qWx7n0vRzHOVeDpOjtOJL6SlogaaGkK8tZf4mk+ZI+lPSmpF2jFM8DnXMuSYr8X9xUpHxgGNAPaAMMkNSmzGazgb3NrB3wInBHlBJ6oHPOJaV44M0oUwJdgYVm9qWZrQeeBY6M3cDM3jaz38LZqQTvfk3IA51zLnnRm64NJc2Imc6KSaUx8G3MfEG4rCKDgP9EKZ5fjHDOJa0ST0YsN7O9K0xmU1buhtKfgL2BHlEy9UDnnEtaim4vKQCaxMzvAizZNC/1Aq4GepjZuigJe9PVOZe0FF10nQ40l7SbpNpAf2BMqXykjsA/gSPMbGnU8nmNzjmXnBTdMGxmGySdD4wH8oERZjZP0g3ADDMbA9wJ1ANeCJ/G+MbMjkiUtgc651xSUvkImJmNA8aVWXZNzOdeVUnXA51zLmnZ/mSEBzrnXNL8WVfnXM7zgTedc7kvu+OcBzrnXPKyPM55oHPOJUfy1x0652qC7I5zHuicc8nL8jjngc45l7wsb7l6oHPOJSvxoJqZ5oHOOZeU4BGwTJciPg90zrmkeaBzzuU8b7o653Kbv9fVOZfrIg6qmVEe6JxzycvySOeBzjmXNH8EzDmX87I7zHmgc86lQpZHOg90zrmkZfvtJTIr9/2wGSFpGbAo0+VIg4bA8kwXwlVKrp6zXc1s+1QmKOl1gu8riuVm1jeV+UeRVYEuV0maEeft5C4L+TnLLf4Ca+dczvNA55zLeR7oNo/hmS6AqzQ/ZznE++iccznPa3TOuZzngc45l/M80Dnncp4HujSR1FLSvpK2kJSf6fK4aPxc5Sa/GJEGko4BbgEWh9MMYKSZ/ZLRgrkKSWphZp+Fn/PNrCjTZXKp4zW6FJO0BXAiMMjMDgZeBZoAl0vaJqOFc+WSdBgwR9IzAGZW5DW73OKBLj22AZqHn18BxgK1gZOkLB+4q4aRtDVwPnARsF7SU+DBLtd4oEsxMysE7gGOkXSgmW0E3gPmAAdktHBuE2a2GjgdeAYYDNSNDXaZLJtLHQ906TEJmACcLKm7mRWZ2TPAzkD7zBbNlWVmS8xslZktB84GtiwOdpI6SWqV2RK6ZPl4dGlgZmslPQ0YcFX4D2UdsAPwXUYL5+IysxWSzgbulPQpkA8clOFiuSR5oEsTM/tJ0iPAfIJawlrgT2b2Q2ZL5hIxs+WSPgT6Ab3NrCDTZXLJ8dtLNoOwU9vC/jqX5SRtBzwPXGpmH2a6PC55HuicK4ekuma2NtPlcKnhgc45l/P8qqtzLud5oHPO5TwPdM65nOeBzjmX8zzQVSOSiiTNkfSxpBckbZVEWj0ljQ0/HyHpyjjbNpB0XhXyuE7S4KjLy2wzUtJxlcirmaSPK1tGVzN4oKte1phZBzNrC6wHzoldqUClz6mZjTGz2+Js0gCodKBzLlt4oKu+JgF7hDWZTyT9HZgFNJHUR9IUSbPCml89AEl9JX0q6T3gmOKEJJ0q6aHw8w6SXpE0N5z2A24Ddg9rk3eG210mabqkDyVdH5PW1ZIWSHoDaJnoICSdGaYzV9JLZWqpvSRNkvRZOJQSkvIl3RmT99nJfpEu93mgq4Yk1SJ4POmjcFFL4Akz6wisBoYCvcysE8Ggn5dIqgs8AhwOHAjsWEHyDwDvmll7oBMwD7gS+CKsTV4mqQ/BMFRdgQ5AZ0ndJXUG+gMdCQJplwiH87KZdQnz+wQYFLOuGdAD+CPwcHgMg4CVZtYlTP9MSbtFyMfVYP6sa/WypaQ54edJwL8IRkRZZGZTw+X7AG2AyeHQd7WBKUAr4Csz+xwgHJ3jrHLy+ANwCpQMU7QyfCQqVp9wmh3O1yMIfPWBV8zstzCPMRGOqa2kmwiax/WA8THrng8fm/tc0pfhMfQB2sX0320b5v1ZhLxcDeWBrnpZY2YdYheEwWx17CLgv2Y2oMx2HQhGU0kFAbea2T/L5HFRFfIYCRxlZnMlnQr0jFlXNi0L877AzGIDIpKaVTJfV4N40zX3TAX2l7QHgKStJLUAPgV2k7R7uN2ACvZ/Ezg33Dc/HP79V4LaWrHxwOkxfX+NJTUCJgJHS9pSUn2CZnIi9YHvwiHoB5ZZd7ykvLDMvwcWhHmfG26PpBbhKMHOVchrdDnGzJaFNaNRkuqEi4ea2WeSzgL+LWk5wajHbctJ4q/AcEmDgCLgXDObImlyePvGf8J+utbAlLBGuYpgCKpZkp4jGE15EUHzOpG/AdPC7T+idEBdALxLMI7fOeE4f48S9N3NUpD5MuCoaN+Oq6n8oX7nXM7zpqtzLud5oHPO5TwPdM65nOeBzjmX8zzQOedyngc651zO80DnnMt5/w+mGCRw+Op4ZwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compute and plot confusion matrix with the best model \n",
    "cm_best = confusion_matrix(test_values, y_classes_best)\n",
    "plot_confusion_matrix(cm_best, classes=[0,1],normalize=False,\n",
    "                      title='Confusion matrix, without normalization')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized confusion matrix\n",
      "[[0.88454891 0.11545109]\n",
      " [0.13057637 0.86942363]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAEYCAYAAAAnEYFiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcVMW9/vHPM4OAuEfc2AQVEXBjdV9u4oJB0WhQ1GiMWzQXE7cYjcYo0ZvFJCZe9Rq5LokbStAISoJJfteoCQioaARFETUMuICixpXt+/vjnMFmmJnuobunl3nevM6LPudUV1X3mflO1VmqFBGYmVWrmlJXwMysmBzkzKyqOciZWVVzkDOzquYgZ2ZVzUHOzKpa1Qc5SetLmiTpfUnj88jnREmPFLJupSJpP0lzy6U8ST0lhaR2rVWnYpF0haQ709c9JH0oqbbAZbTq8at0ZRPkJJ0gaWb6Q/GGpD9K2rcAWX8V2ArYPCJGrmsmEXFXRBxSgPoUVRosdmguTUQ8HhF9WqtODcuT9Jqkg1qr/FKJiH9FxIYRsTKffBoe09Y+fpWuLIKcpPOBXwH/RRKQegA3AkcWIPttgZciYkUB8qp41dBaKhQlyuJ3wIooIkq6AJsAHwIjm0nTgSQILkqXXwEd0n0HAnXABcDbwBvAN9J9VwLLgOVpGacBVwB3ZuTdEwigXbp+CjAf+DfwKnBixvYnMt63NzADeD/9f++MfY8CPwL+nubzCNC5ic9WX/+LMup/FPBl4CXgXeD7GemHAlOB99K01wPt032PpZ/lo/TzHpeR//eAN4E76rel79k+LWNgut4FWAIcmMOx+y1wQfq6a1r2t9L1HdJ81aC8O4BVwCdpHS/KOAZfB/6Vln9pM+XeDtwAPJx+v08C27fg2FydHptP0no+ClwF/COt0yRgc+Au4IM0j54ZefwaWJDuewrYL2PfFaQ/Xxmfqx2wV5p3/fIp8Fo+xzSjzL7pZ3gPmA2MyPW7agtL6SsAw4AVpEGmiTRjgGnAlsAW6Q/jj9J9B6bvHwOsRxIcPgY2a/hD18R65g/iBukPbp903zZA//T1KaRBDvgCsBQ4KX3f8en65hm/SK8AOwLrp+s/aeKz1df/8rT+ZwCLgbuBjYD+6S/Edmn6QcCeabk9gReAczPyC2CHRvL/Kckfi/Ub+SU5I82nEzAF+HmOx+5UYFL6+oT0M9+bse/BjDpklvcacFAjx2BsWr/dgM+Avk2UeztJAB2afg93AeNacGz+lX6v7dLv/FFgHknA3wSYQ/IH5qA0ze+A2zLK/xpJEGxH8sf1TaBjw58vGvwBzXh/fZk/zuOY1mXkNQ/4PtAe+CJJMOuT7btqK0s5NNU3B5ZE893JE4ExEfF2RCwmaaGdlLF/ebp/eURMJvmLt67nLFYBO0taPyLeiIjZjaQZDrwcEXdExIqIuAd4ETgiI81tEfFSRHwC3Afs3kyZy4GrI2I5MA7oDPw6Iv6dlj8b2BUgIp6KiGlpua8BvwEOyOEz/TAiPkvrs4aIGAu8TPJXfhvg0iz51fsbsF/a5dsf+BmwT7rvgHR/S1wZEZ9ExLPAsyTBrin3R8T09OfmLj7/fnM5NrdHxOx0//J0220R8UpEvA/8EXglIv6S5j8eGFD/5oi4MyLeSd//C5I/Hi35ebuOpGV2aZrfuhzTensCG5L8EV0WEf8PeIgkuNdr6rtqE8ohyL0DdM5yrqgL8HrG+uvpttV5NAiSH5Mc+BaJiI9IugNnAW9IeljSTjnUp75OXTPW32xBfd6Jz09O1wehtzL2f1L/fkk7SnpI0puSPiA5j9m5mbwBFkfEp1nSjAV2Bv47Ij7LkhaAiHiF5A/K7sB+JL9ciyT1Yd2CXEu+s6bS5nJsFjSSX8Pvu9HvH0DSBZJeSK/Yv0fS+st2DOrf+02SltgJEbEq3bYux7ReF2BBfV6pfH4Wq045BLmpJN2xo5pJs4jkAkK9Hum2dfERSbes3taZOyNiSkQcTNKieZHklz9bferrtHAd69QS/0NSr94RsTFJN0VZ3tPsUDOSNiQ5z3kLcIWkL7SgPn8juYLdPiIWpusnA5sBs9alPnnK5disc/mS9iM5v3ksySmRTUnO/WU7BvXv/RFwZNpirLcux7TeIqB7gwsorfWzWBFKHuTSg305cIOkoyR1krSepMMk/SxNdg9wmaQtJHVO09+5jkXOAvZP72HaBLikfoekrSSNkLQByTmhD4HGLv9PBnZMb3tpJ+k4oB9JS6bYNiI5b/hh2so8u8H+t4DtWpjnr4GnIuJ0khPUN9XvSO/7erSZ9/4NGE1yghySc03nkJy/bOrWiXWpY66KfWw2IjnHuRhoJ+lyYONsb5LUHbgXODkiXmokz3U9pk+S/OG+KP29OZCkaz4ut49T/Uoe5AAi4pfA+cBlJD88C0h+cf6QJrkKmAk8B/wTeDrdti5l/Znkh+05kitjmT/8NSQnkheRnKw9APhWI3m8Axyepn2H5Arh4RGxZF3q1EIXkpzk/zdJK/PeBvuvAH4r6T1Jx2bLTNKRJBd/zko3nQ8MlHRiut6d5EpkU/5G8ktaH+SeIGkpP9bkO+DHJH+03pN0YbY6tkQrHJspJOfsXiLpFn5K493fhr5E0mv4fXov6IeS6s/3rvMxjYhlwAjgMJKr0jeSBNIX1+GzVSVFeNBMa5qkWcCX0uBhVnEc5MysqpVFd9XMrFgc5MysqjnImVlVK6uHtdVu/VD7jUpdDWuBAX17lLoK1gKvv/4aS5YsyfUevJzUbrxtxIq1HqRpVHyyeEpEDCtk+dmUV5BrvxEd+mS968HKyN+fvL7UVbAW2GePwQXPM1Z8kvPv7aezbsj1SY6CKasgZ2aVSFDGI1Y5yJlZfgTUFHTw44JykDOz/Kmgp/kKykHOzPLk7qqZVTu35Mysagm35MysmsktOTOrcr66ambVyxcezKyaCXdXzazKuSVnZtXL3VUzq3Y17q6aWbXys6tmVt3cXTWzauerq2ZW1dySM7OqJT/WZWbVrowvPJRvG9PMKkR64SGXJVtO0jBJcyXNk3RxI/t7SPo/Sc9Iek7Sl7Pl6SBnZvmr77JmW5rNQrXADcBhQD/geEn9GiS7DLgvIgYAo4Abs1XNQc7M8lM/nlz+LbmhwLyImB8Ry4BxwJEN0gSwcfp6E2BRtkx9Ts7M8tSi++Q6S5qZsX5zRNycvu4KLMjYVwfs0eD9VwCPSDoH2AA4KFuBDnJmlr/cr64uiYimJn9tLJNosH48cHtE/ELSXsAdknaOiFVNFeggZ2b5K8zV1Tqge8Z6N9bujp4GDAOIiKmSOgKdgbebrFohamZmbZgKdnV1BtBbUi9J7UkuLExskOZfwJeSYtUX6Agsbi5Tt+TMLH8FuBk4IlZIGg1MAWqBWyNitqQxwMyImAhcAIyVdB5JV/aUiGjYpV2Dg5yZ5U0FeuIhIiYDkxtsuzzj9Rxgn5bk6SBnZnlJRj/3Y11mVq1E49dFy4SDnJnlSdTUlO81TAc5M8ubu6tmVtUc5MysevmcnJlVMyG35MysuvnCg5lVNbfkzKx6+ZycmVU7t+TMrGr5woOZVT0HOTOrXgLVOMiZWRVzS87MqpqDnJlVLV94MLPqV74xzhPZ5Ovgvfvy7AM/4PkHf8iF3zh4rf3dt96MP938babe8z2m33sJh+6bTAjerl0NY8ecxIz7vs8zEy7jwlMPae2qt0mPTPkTu/bvQ/+dduCan/1krf1PPP4Yew0ZyIYd23H/hN+v3v7srFkcsO9eDNytP0MG7Mr4++5tzWqXNyXd1VyWUnCQy0NNjfjVxcdy5OgbGXDMVYwcNoidttt6jTTfO30YE/78NHsd/1NOvuQ2fn3JcQAcc9BAOrRvx5Bj/4u9T/wppx+zDz22+UIpPkabsXLlSs799n/y4KQ/8sxzcxg/7h5emDNnjTTdu/fg5ltu57hRJ6yxvVOnTtxy2+94+tnZPPjwn7jognN57733WrP6Za2mpianJRtJwyTNlTRP0sWN7L9W0qx0eUlS1oPg7moehuzck1cWLOG1he8AMH7K0xx+4K68OP/N1Wkigo036AjAJhuuzxuL30+2E3Tq2J7a2hrW79CeZctX8u+PPm39D9GGzJg+ne2334Fe220HwMjjRvHQpAfp26/f6jTb9uwJrP3Aee8dd1z9ukuXLmyxxZYsWbyYTTfdtPgVrwQFaKRJqgVuAA4mmYN1hqSJ6eQ1AETEeRnpzwEGZMvXQS4PXbbchLq3lq5eX/jWUobu3HONNFf/ZjKTbhzN2aMOoNP6HRh+1n8DcP9fnuHwA3fl1T9fTaeO7bno5/ez9IOPW7P6bc6iRQvp1u3zuYu7du3G9OlPtjifGdOns2z5MrbbfvtCVq+iFagrOhSYFxHz0zzHAUcCc5pIfzzww2yZFrW7mq3pWenUyJ+vhhNAHjtsMHdOmsYOw37AV875H2656mQkMaR/T1auXMV2h1xK3+E/5DsnfZGeXTdvnYq3UY1Nz9nSX8433niD075xEr8Ze1tZDy/UmnI9H5fDd90VWJCxXpdua6zMbYFewP/LlmnRjlJG0/MwoB9wvKR+zb+rsix8+z26bbXZ6vWuW23GorQ7Wu/rR+3FhEeeBuDJ516lY/v16LzpBhx72GAe+cccVqxYxeKlHzJ11nwG9evRqvVva7p27UZd3ee/QwsX1tGlS5ec3//BBx9w9Ijh/PDKq9hjzz2LUcWK1YIg11nSzIzlzMxsGsm6qYmjRwG/j4iV2epWzD9Fq5ueEbEMqG96Vo2Zs19nhx5bsG2XzVmvXS0jDx3Iw48+t0aaBW++y4FD+wDQp9dWdOywHouXfkjdm+9y4JBke6eO7Rm6a0/mvvZWq3+GtmTwkCHMm/cyr736KsuWLWP8veMYfviInN67bNkyjvvqVzjhaydzzFdHFrmmlacFQW5JRAzOWG7OyKYO6J6x3g1Y1ESRo4B7cqlbMYNcTk1PSWfWR/VY8UkRq1N4K1eu4ryf3sekG/+TWfdfxoRHnuGF+W/yg7OHM/yAXQC4+JcPcOrRe/PkvRfz2x9/gzMuvwOAm+59jA07teep31/KE3d9lzsenMbzLzd1PK0Q2rVrx7W/vp4jhh/K7rv05ZiRx9Kvf3/GXHE5D02aCMDMGTPYvmc37p8wnnO+9U0G7tYfgAnj7+OJxx/jzt/dzh6DdmePQbvz7KxZpfw4ZUU1ymnJYgbQW1IvSe1JAtnEtcqS+gCbAVNzqltj5ykKQdJI4NCIOD1dPwkYGhHnNPWemk5bRoc+xxalPlYcS2dcX+oqWAvss8dgnnpqZkFvWOuwde/oduJ1OaWd/8svPxURg5vaL+nLwK+AWuDWiLha0hhgZkRMTNNcAXSMiJzO8xfz6mpLmp5mVqEEFOo+34iYDExusO3yButXtCTPYnZXc2p6mlmlK9jV1aIoWksuIlZIGg1M4fOm5+xilWdmpVPGz+cX92bgxpqeZlZllDziWK78xIOZ5UU4yJlZlWuz3VUzaxs8aKaZVS+5JWdmVSy5T658o5yDnJnlSb7wYGbVzS05M6tePidnZtXM5+TMrOqVcYxzkDOz/LklZ2bVy8+umlk1K+R4csXgIGdmeSrdWHG5cJAzs7yVcYxzkDOz/LklZ2ZVS2V+4cFTgJtZ3go1x4OkYZLmSponqdHZuCQdK2mOpNmS7s6Wp1tyZpa3QvRWJdUCNwAHk8z2N0PSxIiYk5GmN3AJsE9ELJW0ZbZ83ZIzs7wVqCU3FJgXEfMjYhkwDjiyQZozgBsiYilARLydLVMHOTPLT/qAfi4L0FnSzIzlzIycugILMtbr0m2ZdgR2lPR3SdMkDctWPXdXzSwvatl9cksiYnCTWa0tGqy3A3oDB5JMWP+4pJ0j4r2mCnSQM7O81Rbm6mod0D1jvRuwqJE00yJiOfCqpLkkQW9GU5m6u2pmeWtBd7U5M4DeknpJag+MAiY2SPMH4D+SMtWZpPs6v7lM3ZIzs7wkASz/llxErJA0GpgC1AK3RsRsSWOAmRExMd13iKQ5wErguxHxTnP5NhnkJG2cpUIftPRDmFl1KtS9wBExGZjcYNvlGa8DOD9dctJcS242yUm/zOrXrwfQI9dCzKy6VeRjXRHRval9ZmaZyjjG5XbhQdIoSd9PX3eTNKi41TKzSiGgVsppKYWsQU7S9SRXM05KN30M3FTMSplZBcnxaYdSdWlzubq6d0QMlPQMQES8m17eNTMDyru7mkuQWy6phvTOY0mbA6uKWiszqxgCaso4yuVyTu4GYAKwhaQrgSeAnxa1VmZWUQp0M3BRZG3JRcTvJD0FHJRuGhkRzxe3WmZWKcp90Mxcn3ioBZaTdFn9KJiZraGiu6uSLgXuAbqQPDB7t6RLil0xM6scynEphVxacl8DBkXExwCSrgaeAn5czIqZWeWoyCceMrzeIF07sjz1b2ZtR3J1tdS1aFpzD+hfS3IO7mNgtqQp6fohJFdYzcxW3wxcrpprydVfQZ0NPJyxfVrxqmNmlagir65GxC2tWREzq0wV212tJ2l74GqgH9CxfntE7FjEeplZBSnn7mou97zdDtxGErAPA+4jmSrMzAwo71tIcglynSJiCkBEvBIRl5GOsW5mJiU3A+eylEIut5B8pqQt+oqks4CFQNZZq82s7Sjj3mpOLbnzgA2BbwP7kMxgfWoxK2VmlaWmRjkt2UgaJmmupHmSLm5k/ymSFkualS6nZ8szlwf0n0xf/pvPB840MwOSyaUL0RWVVEsy6tHBJPOrzpA0MSLmNEh6b0SMzjXf5m4GfoC1Z69eLSKOzrUQM6tihRtGaSgwLyLmA0gaBxwJNAxyLdJcS+76fDJeF7v37cFj/7iutYu1PGy29wWlroK1wGcv1hUl3xbcQtJZ0syM9Zsj4ub0dVdgQca+OmCPRvI4RtL+wEvAeRGxoJE0qzV3M/Bfc6uzmbV1LRh/bUlEDG5iX2ORsmFvchJwT0R8ll4I/S3wxQLVzcxsbYJCTWRTB2ROhdoNWJSZICLeiYjP0tWxQNaZAx3kzCxv7WpyW7KYAfSW1CudLGsUMDEzgaRtMlZHAC9krVuuH0JSh4wIamYG1M/fkP+Vh4hYIWk0MIVkNPJbI2K2pDHAzIiYCHxb0ghgBfAucEq2fHN5dnUocAuwCdBD0m7A6RFxzjp/GjOrKoV6QD8iJgOTG2y7POP1JUCLRibPpbt6HXA48E5ayLP4sS4zy1DRs3UBNRHxeoPm6Moi1cfMKky5z7uaS5BbkHZZI70j+RyS+1PMzACoLd8Yl1OQO5uky9oDeAv4S7rNzAyVcISRXOTy7OrbJJdyzcwaVcYxLqerq2Np5BnWiDizKDUys4pT0cOfk3RP63UEvsKaz5eZWRtW8RceIuLezHVJdwB/LlqNzKzilHGMy/2Jhwy9gG0LXREzq1CC2jKOcrmck1vK5+fkakgepVhrxE4za5sqekrCdG6H3UjmdQBYFRFNDqRpZm1TOQe5Zh/rSgPaAxGxMl0c4MxsLQUaaqkocnl2dbqkgUWviZlVpPruai5LKTQ3x0O7iFgB7AucIekV4COSzxQR4cBnZoWc46EomjsnNx0YCBzVSnUxswokoF0Zn5RrLsgJICJeaaW6mFmFqtSW3BaSzm9qZ0T8sgj1MbOKI2oanYOmPDQX5GqBDWl8Bh0zM6B+IptS16JpzQW5NyJiTKvVxMwqUwmvnOaiuVtIyrjaZlYuBNTWKKcla17SMElzJc2T1OSTVZK+KikkNTWH62rNteS+lLVGZmYUZhSSdOTxG4CDSeZgnSFpYkTMaZBuI+DbwJM51a2pHRHx7rpX18zakgJNZDMUmBcR8yNiGTAOOLKRdD8CfgZ8mkvdPLm0meVFJIEklyWLrqw5VmVduu3zsqQBQPeIeCjX+q3LUEtmZp9r2eTSnSXNzFi/OSJu/jyntax+Xl5SDXAtOUwonclBzszy1oIzcksioqmLBXVA94z1bsCijPWNgJ2BR9OgujUwUdKIiMgMnGtwkDOzvIiCDZo5A+gtqRfJ8G6jgBPqd0bE+0Dn1eVKjwIXNhfgwOfkzKwACnHhIR0QZDQwBXgBuC8iZksaI2nEutbNLTkzy1PhxoqLiMnA5AbbLm8i7YG55OkgZ2Z5qb+6Wq4c5Mwsb6Ua9TcXDnJmlrfyDXEOcmaWJ1X6lIRmZtm4u2pmVa18Q5yDnJkVQBk35BzkzCw/yS0k5RvlHOTMLG9uyZlZFVNBBs0sFgc5M8uLu6tmVt1yG/W3ZBzkzCxvDnJmVtVUxt3Vch48oCL8+ZE/MWCXvuzWb0d+cc1P19r/xOOPse+eg9l0g/b84f7fr97+r9dfZ7+9hrD30IEMGbALt4y9qTWr3aYdvGcfnh3/PZ6fcAkXnvzFtfZ332pT/nTj2Uy943ym33UBh+69EwCjDh3ItDvPX718NO0adu3dpbWrX3bqB83MZSkFt+TysHLlSi74zjk8+PAUunbrxgH77MHww49gp779Vqfp3r0HN429leuu/cUa7916m234y6NP0KFDBz788EP2GLgrXx4+gm26+JemmGpqxK8uOprho3/Dwrff54nfnstDj8/mxVffWp3me6cexIS/zmLshKns1Gsr/nDt6ex01NWMm/I046Y8DUD/7bdm/M9P5bmXFzVVVJtSzt1Vt+TyMHPGdLbbfnt6bbcd7du355iRx/HQpIlrpNm2Z0923mVXVLPmV92+fXs6dOgAwGeffcaqVatard5t2ZD+PXil7h1eW/Quy1esZPwjz3D4/v3XSBMBG2/QEYBNNuzIG0s+WCufYw8ZwH2PPNMqda4EyvFfKTjI5eGNRQvp2u3zeTe6du3KG4sW5vz+ugUL2HPw7vTdYVvOu/Ait+JaQZctNqHurfdWry98+326brHJGmmuHjuFUcMGMW/SD3jg2tM5/+cPrJXPVw/enfumOMhBeguJcltKoWhBTtKtkt6W9Hyxyii1iFhrW0tGY+jWvTvTZs7i2dkvcfedv+Ptt97K/ibLS2OHp+FRPPbQAdz50Ax2OOJHfOW8/+WWK45f47gO6d+Djz9dzpz5bxa3shUj13Zc9bXkbgeGFTH/kuvStRsL6z6fC3fhwoVsvU3LW2PbdOnCTn378Y+/P17I6lkjFr79Pt222nT1etctN2HR4vfXSPP1EXsw4S/PAvDkP1+nY4f16LzpBqv3jzxkd3dVM+U4iU2pztsVLchFxGPAu8XKvxwMGjyEV+bN47VXX2XZsmVMGH8vww8/Iqf3Lqyr45NPPgFg6dKlTJv6D3rv2KeY1TVg5pwF7NC9M9t2+QLrtatl5CEDePjx2WukWfDmUg4c0huAPj23pGP7dixe+iGQtNSP/uJujHeQW62QV1clDZM0V9I8SRc3sv8sSf+UNEvSE5L6NZZPppJfXZV0JnAmJFciK0m7du34+a+u46gjDmPVypWc9PVv0Ldff6668ocMGDSI4YeP4KmZMzjhuGN4b+lS/jj5Ia7+0ZXMeOafzH3xBb5/8XeRRETw7XPPp//Ou5T6I1W9lStXcd419zPpujOprRG/nTSdF+a/xQ/OPJSnX6jj4cdnc/GvJ3Hj90dyzgn7ExGcMWbc6vfvO2A7Fr79Pq8tquq/3y1WiEaapFrgBuBgkommZ0iaGBFzMpLdHRE3pelHAL8kS49RjZ1XKhRJPYGHImLnXNIPHDQ4HvvH9KLVxwpvi/2+W+oqWAt8NvsuVn30ZkE7jn13GRC3/eH/ckq71w6bPRURgxvbJ2kv4IqIODRdvwQgIn7cRPrjgZMj4rDmyix5S87MKl8LLip0lpQ54/3NEXFz+rorsCBjXx2wx1plSf8JnA+0B9a+m7sBBzkzy1sLLiosaaolR+O93rW6mhFxA3CDpBOAy4CvN1dgMW8huQeYCvSRVCfptGKVZWalpRyXLOqA7hnr3YDmHikZBxyVLdOiteQi4vhi5W1m5UMUbLauGUBvSb2AhcAo4IQ1ypJ6R8TL6epw4GWycHfVzPJToHvgImKFpNHAFKAWuDUiZksaA8yMiInAaEkHAcuBpWTpqoKDnJkVQKEu10bEZGByg22XZ7z+TkvzdJAzs/yV8SgkDnJmlqfSPZeaCwc5M8tL/Sgk5cpBzszy5yBnZtXM3VUzq2rlPPy5g5yZ5a2MY5yDnJnlKcdntkrFQc7M8pJcXS3fKOcgZ2Z5K98Q5yBnZoVQxlHOQc7M8uZbSMysqpXxKTkHOTPLXxnHOAc5M8tPAQfNLAoHOTPLTwknjs6Fg5yZ5a2MY5yDnJkVQBlHOQc5M8tTeQ+aWbQpCc2sbagfNDOXJWte0jBJcyXNk3RxI/vPlzRH0nOS/ipp22x5OsiZWf4KMPGqpFrgBuAwoB9wvKR+DZI9AwyOiF2B3wM/y1Y1Bzkzy5ty/JfFUGBeRMyPiGUkk0cfmZkgIv4vIj5OV6eRTEDdLAc5M8ublNsCdJY0M2M5MyObrsCCjPW6dFtTTgP+mK1uvvBgZnlrwWWHJRExuAXZRKMJpa8Bg4EDshXoIGdm+SnczcB1QPeM9W7AorWKkw4CLgUOiIjPsmXqIGdmeSngY10zgN6SegELgVHACWuUJQ0AfgMMi4i3c8nU5+TMLG8FuLhKRKwARgNTgBeA+yJitqQxkkakya4BNgTGS5olaWK2urklZ2Z5K9SzqxExGZjcYNvlGa8PammeDnJmlrdyfuLBQc7M8le+Mc5BzszyV8YxzkHOzPIjeUpCM6t25RvjHOTMLH9lHOMc5Mwsf2XcW3WQM7N8lfegmQ5yZpaX5LGuUteiaQ5yZpY3Bzkzq2rurppZ9fK8q2ZWzXIZYaSUHOTMLH9lHOUc5Mwsb36sy8yqWvmGOAc5MyuEMo5yDnJmlrdyvoVEEY3O+FUSkhYDr5e6HkXQGVhS6kpYi1TrMds2IrYoZIaS/kTyfeViSUQMK2T52ZRVkKtWkmY2M9eklSEfs+rh2brMrKo5yJlZVXOQax03l7oC1mI+ZlXC5+TMrKq5JWdmVc1BzsyqmoOcmVU1B7kikdRH0l6S1pNUW+r6WG58rKqPLzwUgaSjgf8CFqbLTOD2iPigpBWzJknaMSJeSl8dBq0oAAAEWklEQVTXRsTKUtfJCsMtuQKTtB5wHHBaRHwJeBDoDlwkaeOSVs4aJelwYJakuwEiYqVbdNXDQa44NgZ6p68fAB4C2gMnSGU88FYbJGkDYDRwLrBM0p3gQFdNHOQKLCKWA78Ejpa0X0SsAp4AZgH7lrRytpaI+Ag4FbgbuBDomBnoSlk3KwwHueJ4HHgEOEnS/hGxMiLuBroAu5W2atZQRCyKiA8jYgnwTWD9+kAnaaCknUpbQ8uHx5Mrgoj4VNJdQACXpL8knwFbAW+UtHLWrIh4R9I3gWskvQjUAv9R4mpZHhzkiiQilkoaC8whaR18CnwtIt4qbc0sm4hYIuk54DDg4IioK3WdbN35FpJWkJ7AjvT8nJU5SZsB9wEXRMRzpa6P5cdBzqwRkjpGxKelroflz0HOzKqar66aWVVzkDOzquYgZ2ZVzUHOzKqag1wFkbRS0ixJz0saL6lTHnkdKOmh9PUISRc3k3ZTSd9ahzKukHRhrtsbpLld0ldbUFZPSc+3tI5W/RzkKssnEbF7ROwMLAPOytypRIuPaURMjIifNJNkU6DFQc6sHDjIVa7HgR3SFswLkm4Enga6SzpE0lRJT6ctvg0BJA2T9KKkJ4Cj6zOSdIqk69PXW0l6QNKz6bI38BNg+7QVeU2a7ruSZkh6TtKVGXldKmmupL8AfbJ9CElnpPk8K2lCg9bpQZIel/RSOhwSkmolXZNR9jfz/SKtujnIVSBJ7UgeOfpnuqkP8LuIGAB8BFwGHBQRA0kG7DxfUkdgLHAEsB+wdRPZXwf8LSJ2AwYCs4GLgVfSVuR3JR1CMpTUUGB3YJCk/SUNAkYBA0iC6JAcPs79ETEkLe8F4LSMfT2BA4DhwE3pZzgNeD8ihqT5nyGpVw7lWBvlZ1cry/qSZqWvHwduIRnZ5PWImJZu3xPoB/w9HbquPTAV2Al4NSJeBkhH2TizkTK+CJwMq4caej99zCnTIenyTLq+IUnQ2wh4ICI+TsuYmMNn2lnSVSRd4g2BKRn77ksfhXtZ0vz0MxwC7Jpxvm6TtOyXcijL2iAHucrySUTsnrkhDWQfZW4C/hwRxzdItzvJqCiFIODHEfGbBmWcuw5l3A4cFRHPSjoFODBjX8O8Ii37nIjIDIZI6tnCcq2NcHe1+kwD9pG0A4CkTpJ2BF4EeknaPk13fBPv/ytwdvre2nTI9n+TtNLqTQFOzTjX11XSlsBjwFckrS9pI5KucTYbAW+kw8af2GDfSEk1aZ23A+amZZ+dpkfSjunovmaNckuuykTE4rRFdI+kDunmyyLiJUlnAg9LWkIyWvHOjWTxHeBmSacBK4GzI2KqpL+nt2j8MT0v1xeYmrYkPyQZRuppSfeSjIL8OkmXOpsfAE+m6f/JmsF0LvA3knH4zkrH6ftfknN1TyspfDFwVG7fjrVFfkDfzKqau6tmVtUc5MysqjnImVlVc5Azs6rmIGdmVc1BzsyqmoOcmVW1/w8xnSM7fAr9DwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot confusion matrix (normalized) with the best model \n",
    "plot_confusion_matrix(cm_best, classes=[0,1],normalize=True,\n",
    "                      title='Confusion matrix, with normalization')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auROC = 0.8769862664635207\n"
     ]
    }
   ],
   "source": [
    "# calculate auROC\n",
    "auroc_test_best = roc_auc_score(test_values, y_classes_best)\n",
    "print('auROC =', auroc_test_best)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison between models\n",
    "***\n",
    "When we looked at the normalized confusion matrix, we found that this model gave higher sensitivity on class 1 samples but lower specificity on class 0 samples compared to the final model. Since the validation data was also sampled in a class-balance way, it is biased toward the balanced statistics. But when we trained the model for longer, it learned the closer-to-real statistics of the original data, so it did well on the test set with original class values. The area under ROC metric for both models were quite similar. We think that both models were fine. The final model might give better accuracy on predicting playlist-track contingency, but the model with lowest validation loss had higher \"false positive rate\", therefore would give us a more extended list of recommendation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}