new_NNCF10000_rand_spotify.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Collaborative Filtering (NNCF)\n",
    "***\n",
    "Besides classical matrix factorization, we also implemented the neural network version of collaborative filtering: using deep network model to perform matrix factorization, as well as allowing extra playlist-track mixing. \n",
    "\n",
    "The code implemented here was adopted from this 2017 WWW paper on __[Neural Collaborative Filtering](https://www.comp.nus.edu.sg/~xiangnan/papers/ncf.pdf)__ and a comprehensive __[blogpost](https://nipunbatra.github.io/blog/2017/neural-collaborative-filtering.html)__ written by Dr. Nipun Batra, Assistant Professor in Computer Science Department at IIT Gandhinagar. We applied the architecture of the NNCF model to the same subset of data consisting of randomly selected 10,000 playlists, training the network to perform binary classifcation on playlist-track pairs (predicting 0 vs. 1), and evaluated the model performance.\n",
    "\n",
    "***\n",
    "First we import the libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import scipy.sparse as sps\n",
    "import scipy\n",
    "import pickle\n",
    "import keras\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "from keras.models import load_model\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import roc_auc_score\n",
    "import itertools\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data and model\n",
    "\n",
    "***\n",
    "Here we loaded the sparse matrix containing track-playlist contingency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load files\n",
    "sps_acc = sps.load_npz('sparse_10000_rand.npz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<171381x10000 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 657056 stored elements in Compressed Sparse Column format>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert matrix to csc type\n",
    "sps_acc = sps_acc.tocsc()\n",
    "sps_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up model architecture\n",
    "***\n",
    "For building the model, we need to determine the number of latent factors for the embedding layers of ***matrix factorization stream***, as well as those for the ***multilayer perceptron stream***, for both playlist and track inputs. We set the former to 50, and the latter to 10, as a simple start."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set latent factor number\n",
    "n_tracks, n_playlists = sps_acc.shape[0], sps_acc.shape[1]\n",
    "n_latent_mf = 50\n",
    "n_latent_playlist = 10\n",
    "n_latent_track = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network architecture\n",
    "***\n",
    "This architecture was adopted from the blogpost mentioned above. For the final prediction layer, we set it to a 2-unit dense connected layer for binary classification with softmax activation, and used binary crossentropy as loss function. We compiled the model with Adam optimizer, with a mild learning rate decay. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build model\n",
    "from keras.layers import Input, Embedding, Flatten, Dropout, Concatenate, Dot, BatchNormalization, Dense\n",
    "\n",
    "# track embedding stream\n",
    "track_input = Input(shape=[1],name='Track')\n",
    "track_embedding_mlp = Embedding(n_tracks + 1, n_latent_track, name='track-Embedding-MLP')(track_input)\n",
    "track_vec_mlp = Flatten(name='Flatten_tracks-MLP')(track_embedding_mlp)\n",
    "track_vec_mlp = Dropout(0.2)(track_vec_mlp)\n",
    "\n",
    "track_embedding_mf = Embedding(n_tracks + 1, n_latent_mf, name='track-Embedding-MF')(track_input)\n",
    "track_vec_mf = Flatten(name='Flatten_tracks-MF')(track_embedding_mf)\n",
    "track_vec_mf = Dropout(0.2)(track_vec_mf)\n",
    "\n",
    "# playlist embedding stream\n",
    "playlist_input = Input(shape=[1],name='Playlist')\n",
    "playlist_vec_mlp = Flatten(name='Flatten_playlists-MLP')(Embedding(n_playlists + 1, n_latent_playlist,name='playlist-Embedding-MLP')(playlist_input))\n",
    "playlist_vec_mlp = Dropout(0.2)(playlist_vec_mlp)\n",
    "\n",
    "playlist_vec_mf = Flatten(name='Flatten_playlists-MF')(Embedding(n_playlists + 1, n_latent_mf,name='playlist-Embedding-MF')(playlist_input))\n",
    "playlist_vec_mf = Dropout(0.2)(playlist_vec_mf)\n",
    "\n",
    "# MLP stream\n",
    "concat = Concatenate(axis=-1,name='Concat')([track_vec_mlp, playlist_vec_mlp])\n",
    "concat_dropout = Dropout(0.2)(concat)\n",
    "dense = Dense(200,name='FullyConnected')(concat_dropout)\n",
    "dense_batch = BatchNormalization(name='Batch')(dense)\n",
    "dropout_1 = Dropout(0.2,name='Dropout-1')(dense_batch)\n",
    "dense_2 = Dense(100,name='FullyConnected-1')(dropout_1)\n",
    "dense_batch_2 = BatchNormalization(name='Batch-2')(dense_2)\n",
    "dropout_2 = Dropout(0.2,name='Dropout-2')(dense_batch_2)\n",
    "dense_3 = Dense(50,name='FullyConnected-2')(dropout_2)\n",
    "dense_4 = Dense(20,name='FullyConnected-3', activation='relu')(dense_3)\n",
    "\n",
    "# end prediction for both streams\n",
    "pred_mf = Dot(1,name='Dot')([track_vec_mf, playlist_vec_mf])\n",
    "pred_mlp = Dense(1, activation='relu',name='Activation')(dense_4)\n",
    "\n",
    "# combine both stream\n",
    "combine_mlp_mf = Concatenate(axis=-1,name='Concat-MF-MLP')([pred_mf, pred_mlp])\n",
    "result_combine = Dense(100,name='Combine-MF-MLP')(combine_mlp_mf)\n",
    "deep_combine = Dense(100,name='FullyConnected-4')(result_combine)\n",
    "\n",
    "# final prediction layer\n",
    "result = Dense(2, activation = 'softmax', name='Prediction')(deep_combine)\n",
    "\n",
    "# build model\n",
    "model = keras.Model([playlist_input, track_input], result)\n",
    "\n",
    "# compile model\n",
    "opt = keras.optimizers.Adam(lr = 0.1, decay = 1e-5)\n",
    "model.compile(optimizer='adam', loss= 'binary_crossentropy', metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the visualization and summary of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"1359pt\" viewBox=\"0.00 0.00 948.50 1359.00\" width=\"949pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 1355)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-1355 944.5,-1355 944.5,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 4502921112 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>4502921112</title>\n",
       "<polygon fill=\"none\" points=\"163,-1314.5 163,-1350.5 282,-1350.5 282,-1314.5 163,-1314.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"222.5\" y=\"-1328.8\">Track: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4502921168 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>4502921168</title>\n",
       "<polygon fill=\"none\" points=\"229,-1241.5 229,-1277.5 448,-1277.5 448,-1241.5 229,-1241.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"338.5\" y=\"-1255.8\">track-Embedding-MLP: Embedding</text>\n",
       "</g>\n",
       "<!-- 4502921112&#45;&gt;4502921168 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>4502921112-&gt;4502921168</title>\n",
       "<path d=\"M251.1742,-1314.4551C266.232,-1304.979 284.8686,-1293.2508 301.053,-1283.0658\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"303.1543,-1285.8788 309.7537,-1277.5904 299.4259,-1279.9544 303.1543,-1285.8788\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4502936208 -->\n",
       "<g class=\"node\" id=\"node16\">\n",
       "<title>4502936208</title>\n",
       "<polygon fill=\"none\" points=\"0,-1241.5 0,-1277.5 211,-1277.5 211,-1241.5 0,-1241.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"105.5\" y=\"-1255.8\">track-Embedding-MF: Embedding</text>\n",
       "</g>\n",
       "<!-- 4502921112&#45;&gt;4502936208 -->\n",
       "<g class=\"edge\" id=\"edge15\">\n",
       "<title>4502921112-&gt;4502936208</title>\n",
       "<path d=\"M193.5786,-1314.4551C178.391,-1304.979 159.5937,-1293.2508 143.2699,-1283.0658\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"144.8309,-1279.9145 134.4941,-1277.5904 141.1255,-1285.8533 144.8309,-1279.9145\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4592173464 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>4592173464</title>\n",
       "<polygon fill=\"none\" points=\"637.5,-1314.5 637.5,-1350.5 765.5,-1350.5 765.5,-1314.5 637.5,-1314.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"701.5\" y=\"-1328.8\">Playlist: InputLayer</text>\n",
       "</g>\n",
       "<!-- 4465288920 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>4465288920</title>\n",
       "<polygon fill=\"none\" points=\"708.5,-1241.5 708.5,-1277.5 940.5,-1277.5 940.5,-1241.5 708.5,-1241.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"824.5\" y=\"-1255.8\">playlist-Embedding-MLP: Embedding</text>\n",
       "</g>\n",
       "<!-- 4592173464&#45;&gt;4465288920 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>4592173464-&gt;4465288920</title>\n",
       "<path d=\"M731.9045,-1314.4551C748.0189,-1304.8912 767.9985,-1293.0334 785.2692,-1282.7833\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"787.2058,-1285.704 794.019,-1277.5904 783.6331,-1279.6843 787.2058,-1285.704\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112432317496 -->\n",
       "<g class=\"node\" id=\"node17\">\n",
       "<title>112432317496</title>\n",
       "<polygon fill=\"none\" points=\"466.5,-1241.5 466.5,-1277.5 690.5,-1277.5 690.5,-1241.5 466.5,-1241.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"578.5\" y=\"-1255.8\">playlist-Embedding-MF: Embedding</text>\n",
       "</g>\n",
       "<!-- 4592173464&#45;&gt;112432317496 -->\n",
       "<g class=\"edge\" id=\"edge16\">\n",
       "<title>4592173464-&gt;112432317496</title>\n",
       "<path d=\"M671.0955,-1314.4551C654.9811,-1304.8912 635.0015,-1293.0334 617.7308,-1282.7833\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"619.3669,-1279.6843 608.981,-1277.5904 615.7942,-1285.704 619.3669,-1279.6843\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4502934248 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>4502934248</title>\n",
       "<polygon fill=\"none\" points=\"261.5,-1168.5 261.5,-1204.5 437.5,-1204.5 437.5,-1168.5 261.5,-1168.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"349.5\" y=\"-1182.8\">Flatten_tracks-MLP: Flatten</text>\n",
       "</g>\n",
       "<!-- 4502921168&#45;&gt;4502934248 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>4502921168-&gt;4502934248</title>\n",
       "<path d=\"M341.2191,-1241.4551C342.4355,-1233.3828 343.8981,-1223.6764 345.2534,-1214.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"348.7449,-1215.0003 346.7741,-1204.5904 341.823,-1213.9572 348.7449,-1215.0003\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4744235944 -->\n",
       "<g class=\"node\" id=\"node6\">\n",
       "<title>4744235944</title>\n",
       "<polygon fill=\"none\" points=\"708,-1168.5 708,-1204.5 897,-1204.5 897,-1168.5 708,-1168.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"802.5\" y=\"-1182.8\">Flatten_playlists-MLP: Flatten</text>\n",
       "</g>\n",
       "<!-- 4465288920&#45;&gt;4744235944 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>4465288920-&gt;4744235944</title>\n",
       "<path d=\"M819.0618,-1241.4551C816.6026,-1233.2951 813.6402,-1223.4652 810.9048,-1214.3887\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"814.1886,-1213.1551 807.9519,-1204.5904 807.4863,-1215.175 814.1886,-1213.1551\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4482710216 -->\n",
       "<g class=\"node\" id=\"node7\">\n",
       "<title>4482710216</title>\n",
       "<polygon fill=\"none\" points=\"303.5,-1095.5 303.5,-1131.5 431.5,-1131.5 431.5,-1095.5 303.5,-1095.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"367.5\" y=\"-1109.8\">dropout_1: Dropout</text>\n",
       "</g>\n",
       "<!-- 4502934248&#45;&gt;4482710216 -->\n",
       "<g class=\"edge\" id=\"edge5\">\n",
       "<title>4502934248-&gt;4482710216</title>\n",
       "<path d=\"M353.9494,-1168.4551C355.9615,-1160.2951 358.3853,-1150.4652 360.6233,-1141.3887\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"364.0435,-1142.1375 363.0394,-1131.5904 357.247,-1140.4616 364.0435,-1142.1375\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4744236392 -->\n",
       "<g class=\"node\" id=\"node8\">\n",
       "<title>4744236392</title>\n",
       "<polygon fill=\"none\" points=\"693.5,-1095.5 693.5,-1131.5 821.5,-1131.5 821.5,-1095.5 693.5,-1095.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"757.5\" y=\"-1109.8\">dropout_3: Dropout</text>\n",
       "</g>\n",
       "<!-- 4744235944&#45;&gt;4744236392 -->\n",
       "<g class=\"edge\" id=\"edge6\">\n",
       "<title>4744235944-&gt;4744236392</title>\n",
       "<path d=\"M791.3764,-1168.4551C786.1299,-1159.9441 779.7636,-1149.6165 773.9724,-1140.2219\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"776.8785,-1138.2663 768.6516,-1131.5904 770.9197,-1141.9396 776.8785,-1138.2663\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112432318280 -->\n",
       "<g class=\"node\" id=\"node9\">\n",
       "<title>112432318280</title>\n",
       "<polygon fill=\"none\" points=\"451,-1022.5 451,-1058.5 584,-1058.5 584,-1022.5 451,-1022.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-1036.8\">Concat: Concatenate</text>\n",
       "</g>\n",
       "<!-- 4482710216&#45;&gt;112432318280 -->\n",
       "<g class=\"edge\" id=\"edge7\">\n",
       "<title>4482710216-&gt;112432318280</title>\n",
       "<path d=\"M404.5787,-1095.4551C424.6811,-1085.6719 449.7159,-1073.4883 471.1015,-1063.0806\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"472.8679,-1066.1135 480.328,-1058.5904 469.8047,-1059.8193 472.8679,-1066.1135\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4744236392&#45;&gt;112432318280 -->\n",
       "<g class=\"edge\" id=\"edge8\">\n",
       "<title>4744236392-&gt;112432318280</title>\n",
       "<path d=\"M698.1741,-1095.4551C664.4237,-1085.1893 621.9837,-1072.2804 586.7183,-1061.5539\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"587.5609,-1058.1519 576.9752,-1058.5904 585.5238,-1064.849 587.5609,-1058.1519\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112438135552 -->\n",
       "<g class=\"node\" id=\"node10\">\n",
       "<title>112438135552</title>\n",
       "<polygon fill=\"none\" points=\"453.5,-949.5 453.5,-985.5 581.5,-985.5 581.5,-949.5 453.5,-949.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-963.8\">dropout_5: Dropout</text>\n",
       "</g>\n",
       "<!-- 112432318280&#45;&gt;112438135552 -->\n",
       "<g class=\"edge\" id=\"edge9\">\n",
       "<title>112432318280-&gt;112438135552</title>\n",
       "<path d=\"M517.5,-1022.4551C517.5,-1014.3828 517.5,-1004.6764 517.5,-995.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-995.5903 517.5,-985.5904 514.0001,-995.5904 521.0001,-995.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112438135944 -->\n",
       "<g class=\"node\" id=\"node11\">\n",
       "<title>112438135944</title>\n",
       "<polygon fill=\"none\" points=\"443.5,-876.5 443.5,-912.5 591.5,-912.5 591.5,-876.5 443.5,-876.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-890.8\">FullyConnected: Dense</text>\n",
       "</g>\n",
       "<!-- 112438135552&#45;&gt;112438135944 -->\n",
       "<g class=\"edge\" id=\"edge10\">\n",
       "<title>112438135552-&gt;112438135944</title>\n",
       "<path d=\"M517.5,-949.4551C517.5,-941.3828 517.5,-931.6764 517.5,-922.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-922.5903 517.5,-912.5904 514.0001,-922.5904 521.0001,-922.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112438435792 -->\n",
       "<g class=\"node\" id=\"node12\">\n",
       "<title>112438435792</title>\n",
       "<polygon fill=\"none\" points=\"432.5,-803.5 432.5,-839.5 602.5,-839.5 602.5,-803.5 432.5,-803.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-817.8\">Batch: BatchNormalization</text>\n",
       "</g>\n",
       "<!-- 112438135944&#45;&gt;112438435792 -->\n",
       "<g class=\"edge\" id=\"edge11\">\n",
       "<title>112438135944-&gt;112438435792</title>\n",
       "<path d=\"M517.5,-876.4551C517.5,-868.3828 517.5,-858.6764 517.5,-849.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-849.5903 517.5,-839.5904 514.0001,-849.5904 521.0001,-849.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112438598064 -->\n",
       "<g class=\"node\" id=\"node13\">\n",
       "<title>112438598064</title>\n",
       "<polygon fill=\"none\" points=\"453,-730.5 453,-766.5 582,-766.5 582,-730.5 453,-730.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-744.8\">Dropout-1: Dropout</text>\n",
       "</g>\n",
       "<!-- 112438435792&#45;&gt;112438598064 -->\n",
       "<g class=\"edge\" id=\"edge12\">\n",
       "<title>112438435792-&gt;112438598064</title>\n",
       "<path d=\"M517.5,-803.4551C517.5,-795.3828 517.5,-785.6764 517.5,-776.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-776.5903 517.5,-766.5904 514.0001,-776.5904 521.0001,-776.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112438599408 -->\n",
       "<g class=\"node\" id=\"node14\">\n",
       "<title>112438599408</title>\n",
       "<polygon fill=\"none\" points=\"437.5,-657.5 437.5,-693.5 597.5,-693.5 597.5,-657.5 437.5,-657.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-671.8\">FullyConnected-1: Dense</text>\n",
       "</g>\n",
       "<!-- 112438598064&#45;&gt;112438599408 -->\n",
       "<g class=\"edge\" id=\"edge13\">\n",
       "<title>112438598064-&gt;112438599408</title>\n",
       "<path d=\"M517.5,-730.4551C517.5,-722.3828 517.5,-712.6764 517.5,-703.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-703.5903 517.5,-693.5904 514.0001,-703.5904 521.0001,-703.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112438893984 -->\n",
       "<g class=\"node\" id=\"node15\">\n",
       "<title>112438893984</title>\n",
       "<polygon fill=\"none\" points=\"426.5,-584.5 426.5,-620.5 608.5,-620.5 608.5,-584.5 426.5,-584.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517.5\" y=\"-598.8\">Batch-2: BatchNormalization</text>\n",
       "</g>\n",
       "<!-- 112438599408&#45;&gt;112438893984 -->\n",
       "<g class=\"edge\" id=\"edge14\">\n",
       "<title>112438599408-&gt;112438893984</title>\n",
       "<path d=\"M517.5,-657.4551C517.5,-649.3828 517.5,-639.6764 517.5,-630.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"521.0001,-630.5903 517.5,-620.5904 514.0001,-630.5904 521.0001,-630.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112439262344 -->\n",
       "<g class=\"node\" id=\"node18\">\n",
       "<title>112439262344</title>\n",
       "<polygon fill=\"none\" points=\"451,-511.5 451,-547.5 580,-547.5 580,-511.5 451,-511.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"515.5\" y=\"-525.8\">Dropout-2: Dropout</text>\n",
       "</g>\n",
       "<!-- 112438893984&#45;&gt;112439262344 -->\n",
       "<g class=\"edge\" id=\"edge17\">\n",
       "<title>112438893984-&gt;112439262344</title>\n",
       "<path d=\"M517.0056,-584.4551C516.7845,-576.3828 516.5185,-566.6764 516.2721,-557.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"519.7683,-557.4907 515.9956,-547.5904 512.7709,-557.6825 519.7683,-557.4907\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4502934920 -->\n",
       "<g class=\"node\" id=\"node19\">\n",
       "<title>4502934920</title>\n",
       "<polygon fill=\"none\" points=\"55,-1168.5 55,-1204.5 222,-1204.5 222,-1168.5 55,-1168.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"138.5\" y=\"-1182.8\">Flatten_tracks-MF: Flatten</text>\n",
       "</g>\n",
       "<!-- 4502936208&#45;&gt;4502934920 -->\n",
       "<g class=\"edge\" id=\"edge18\">\n",
       "<title>4502936208-&gt;4502934920</title>\n",
       "<path d=\"M113.6573,-1241.4551C117.4254,-1233.1196 121.9812,-1223.0416 126.157,-1213.8042\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"129.3922,-1215.1443 130.3222,-1204.5904 123.0136,-1212.2608 129.3922,-1215.1443\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112432317776 -->\n",
       "<g class=\"node\" id=\"node20\">\n",
       "<title>112432317776</title>\n",
       "<polygon fill=\"none\" points=\"477.5,-1168.5 477.5,-1204.5 657.5,-1204.5 657.5,-1168.5 477.5,-1168.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"567.5\" y=\"-1182.8\">Flatten_playlists-MF: Flatten</text>\n",
       "</g>\n",
       "<!-- 112432317496&#45;&gt;112432317776 -->\n",
       "<g class=\"edge\" id=\"edge19\">\n",
       "<title>112432317496-&gt;112432317776</title>\n",
       "<path d=\"M575.7809,-1241.4551C574.5645,-1233.3828 573.1019,-1223.6764 571.7466,-1214.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"575.177,-1213.9572 570.2259,-1204.5904 568.2551,-1215.0003 575.177,-1213.9572\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112439428992 -->\n",
       "<g class=\"node\" id=\"node21\">\n",
       "<title>112439428992</title>\n",
       "<polygon fill=\"none\" points=\"432.5,-438.5 432.5,-474.5 592.5,-474.5 592.5,-438.5 432.5,-438.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"512.5\" y=\"-452.8\">FullyConnected-2: Dense</text>\n",
       "</g>\n",
       "<!-- 112439262344&#45;&gt;112439428992 -->\n",
       "<g class=\"edge\" id=\"edge20\">\n",
       "<title>112439262344-&gt;112439428992</title>\n",
       "<path d=\"M514.7584,-511.4551C514.4267,-503.3828 514.0278,-493.6764 513.6582,-484.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"517.1512,-484.4382 513.2434,-474.5904 510.1571,-484.7257 517.1512,-484.4382\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 4502934976 -->\n",
       "<g class=\"node\" id=\"node22\">\n",
       "<title>4502934976</title>\n",
       "<polygon fill=\"none\" points=\"125.5,-1095.5 125.5,-1131.5 253.5,-1131.5 253.5,-1095.5 125.5,-1095.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"189.5\" y=\"-1109.8\">dropout_2: Dropout</text>\n",
       "</g>\n",
       "<!-- 4502934920&#45;&gt;4502934976 -->\n",
       "<g class=\"edge\" id=\"edge21\">\n",
       "<title>4502934920-&gt;4502934976</title>\n",
       "<path d=\"M151.1067,-1168.4551C157.114,-1159.8564 164.4168,-1149.4034 171.0341,-1139.9316\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"174.0036,-1141.7925 176.8615,-1131.5904 168.2652,-1137.7835 174.0036,-1141.7925\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112432317440 -->\n",
       "<g class=\"node\" id=\"node23\">\n",
       "<title>112432317440</title>\n",
       "<polygon fill=\"none\" points=\"485.5,-1095.5 485.5,-1131.5 613.5,-1131.5 613.5,-1095.5 485.5,-1095.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"549.5\" y=\"-1109.8\">dropout_4: Dropout</text>\n",
       "</g>\n",
       "<!-- 112432317776&#45;&gt;112432317440 -->\n",
       "<g class=\"edge\" id=\"edge22\">\n",
       "<title>112432317776-&gt;112432317440</title>\n",
       "<path d=\"M563.0506,-1168.4551C561.0385,-1160.2951 558.6147,-1150.4652 556.3767,-1141.3887\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"559.753,-1140.4616 553.9606,-1131.5904 552.9565,-1142.1375 559.753,-1140.4616\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112440315800 -->\n",
       "<g class=\"node\" id=\"node24\">\n",
       "<title>112440315800</title>\n",
       "<polygon fill=\"none\" points=\"431.5,-365.5 431.5,-401.5 591.5,-401.5 591.5,-365.5 431.5,-365.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"511.5\" y=\"-379.8\">FullyConnected-3: Dense</text>\n",
       "</g>\n",
       "<!-- 112439428992&#45;&gt;112440315800 -->\n",
       "<g class=\"edge\" id=\"edge23\">\n",
       "<title>112439428992-&gt;112440315800</title>\n",
       "<path d=\"M512.2528,-438.4551C512.1422,-430.3828 512.0093,-420.6764 511.8861,-411.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"515.3846,-411.5414 511.7478,-401.5904 508.3852,-411.6374 515.3846,-411.5414\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112440544336 -->\n",
       "<g class=\"node\" id=\"node25\">\n",
       "<title>112440544336</title>\n",
       "<polygon fill=\"none\" points=\"365.5,-1022.5 365.5,-1058.5 431.5,-1058.5 431.5,-1022.5 365.5,-1022.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"398.5\" y=\"-1036.8\">Dot: Dot</text>\n",
       "</g>\n",
       "<!-- 4502934976&#45;&gt;112440544336 -->\n",
       "<g class=\"edge\" id=\"edge24\">\n",
       "<title>4502934976-&gt;112440544336</title>\n",
       "<path d=\"M241.1629,-1095.4551C276.3279,-1083.1726 322.3253,-1067.1065 355.4935,-1055.5214\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"357.0625,-1058.6808 365.3491,-1052.079 354.7542,-1052.0723 357.0625,-1058.6808\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112432317440&#45;&gt;112440544336 -->\n",
       "<g class=\"edge\" id=\"edge25\">\n",
       "<title>112432317440-&gt;112440544336</title>\n",
       "<path d=\"M512.1741,-1095.4551C490.5479,-1085 463.2508,-1071.8034 440.8269,-1060.9627\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"442.0722,-1057.6772 431.5457,-1056.4757 439.0254,-1063.9794 442.0722,-1057.6772\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112441417064 -->\n",
       "<g class=\"node\" id=\"node26\">\n",
       "<title>112441417064</title>\n",
       "<polygon fill=\"none\" points=\"436.5,-292.5 436.5,-328.5 554.5,-328.5 554.5,-292.5 436.5,-292.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"495.5\" y=\"-306.8\">Activation: Dense</text>\n",
       "</g>\n",
       "<!-- 112440315800&#45;&gt;112441417064 -->\n",
       "<g class=\"edge\" id=\"edge26\">\n",
       "<title>112440315800-&gt;112441417064</title>\n",
       "<path d=\"M507.5449,-365.4551C505.7565,-357.2951 503.602,-347.4652 501.6126,-338.3887\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"505.0249,-337.6091 499.465,-328.5904 498.1872,-339.1079 505.0249,-337.6091\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112441416840 -->\n",
       "<g class=\"node\" id=\"node27\">\n",
       "<title>112441416840</title>\n",
       "<polygon fill=\"none\" points=\"346,-219.5 346,-255.5 537,-255.5 537,-219.5 346,-219.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"441.5\" y=\"-233.8\">Concat-MF-MLP: Concatenate</text>\n",
       "</g>\n",
       "<!-- 112440544336&#45;&gt;112441416840 -->\n",
       "<g class=\"edge\" id=\"edge27\">\n",
       "<title>112440544336-&gt;112441416840</title>\n",
       "<path d=\"M398.5,-1022.062C398.5,-994.3663 398.5,-940.3785 398.5,-894.5 398.5,-894.5 398.5,-894.5 398.5,-383.5 398.5,-340.9537 415.5108,-294.1703 428.1339,-265.3111\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"431.4632,-266.4421 432.3808,-255.8871 425.0812,-263.5661 431.4632,-266.4421\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112441417064&#45;&gt;112441416840 -->\n",
       "<g class=\"edge\" id=\"edge28\">\n",
       "<title>112441417064-&gt;112441416840</title>\n",
       "<path d=\"M482.1517,-292.4551C475.7261,-283.7686 467.9007,-273.1898 460.8379,-263.642\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"463.6428,-261.5484 454.8819,-255.5904 458.0151,-265.7113 463.6428,-261.5484\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112441693016 -->\n",
       "<g class=\"node\" id=\"node28\">\n",
       "<title>112441693016</title>\n",
       "<polygon fill=\"none\" points=\"357,-146.5 357,-182.5 526,-182.5 526,-146.5 357,-146.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"441.5\" y=\"-160.8\">Combine-MF-MLP: Dense</text>\n",
       "</g>\n",
       "<!-- 112441416840&#45;&gt;112441693016 -->\n",
       "<g class=\"edge\" id=\"edge29\">\n",
       "<title>112441416840-&gt;112441693016</title>\n",
       "<path d=\"M441.5,-219.4551C441.5,-211.3828 441.5,-201.6764 441.5,-192.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"445.0001,-192.5903 441.5,-182.5904 438.0001,-192.5904 445.0001,-192.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112441693744 -->\n",
       "<g class=\"node\" id=\"node29\">\n",
       "<title>112441693744</title>\n",
       "<polygon fill=\"none\" points=\"361.5,-73.5 361.5,-109.5 521.5,-109.5 521.5,-73.5 361.5,-73.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"441.5\" y=\"-87.8\">FullyConnected-4: Dense</text>\n",
       "</g>\n",
       "<!-- 112441693016&#45;&gt;112441693744 -->\n",
       "<g class=\"edge\" id=\"edge30\">\n",
       "<title>112441693016-&gt;112441693744</title>\n",
       "<path d=\"M441.5,-146.4551C441.5,-138.3828 441.5,-128.6764 441.5,-119.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"445.0001,-119.5903 441.5,-109.5904 438.0001,-119.5904 445.0001,-119.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 112441822568 -->\n",
       "<g class=\"node\" id=\"node30\">\n",
       "<title>112441822568</title>\n",
       "<polygon fill=\"none\" points=\"383.5,-.5 383.5,-36.5 499.5,-36.5 499.5,-.5 383.5,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"441.5\" y=\"-14.8\">Prediction: Dense</text>\n",
       "</g>\n",
       "<!-- 112441693744&#45;&gt;112441822568 -->\n",
       "<g class=\"edge\" id=\"edge31\">\n",
       "<title>112441693744-&gt;112441822568</title>\n",
       "<path d=\"M441.5,-73.4551C441.5,-65.3828 441.5,-55.6764 441.5,-46.6817\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"445.0001,-46.5903 441.5,-36.5904 438.0001,-46.5904 445.0001,-46.5903\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# visualize model\n",
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "SVG(model_to_dot(model, show_shapes=False, show_layer_names=True, rankdir='HB').create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "Track (InputLayer)              (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "Playlist (InputLayer)           (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "track-Embedding-MLP (Embedding) (None, 1, 10)        1713820     Track[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "playlist-Embedding-MLP (Embeddi (None, 1, 10)        100010      Playlist[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "Flatten_tracks-MLP (Flatten)    (None, 10)           0           track-Embedding-MLP[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "Flatten_playlists-MLP (Flatten) (None, 10)           0           playlist-Embedding-MLP[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 10)           0           Flatten_tracks-MLP[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 10)           0           Flatten_playlists-MLP[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "Concat (Concatenate)            (None, 20)           0           dropout_1[0][0]                  \n",
      "                                                                 dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, 20)           0           Concat[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected (Dense)          (None, 200)          4200        dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "Batch (BatchNormalization)      (None, 200)          800         FullyConnected[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "Dropout-1 (Dropout)             (None, 200)          0           Batch[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-1 (Dense)        (None, 100)          20100       Dropout-1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "Batch-2 (BatchNormalization)    (None, 100)          400         FullyConnected-1[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "track-Embedding-MF (Embedding)  (None, 1, 50)        8569100     Track[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "playlist-Embedding-MF (Embeddin (None, 1, 50)        500050      Playlist[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "Dropout-2 (Dropout)             (None, 100)          0           Batch-2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "Flatten_tracks-MF (Flatten)     (None, 50)           0           track-Embedding-MF[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "Flatten_playlists-MF (Flatten)  (None, 50)           0           playlist-Embedding-MF[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-2 (Dense)        (None, 50)           5050        Dropout-2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 50)           0           Flatten_tracks-MF[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 50)           0           Flatten_playlists-MF[0][0]       \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-3 (Dense)        (None, 20)           1020        FullyConnected-2[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "Dot (Dot)                       (None, 1)            0           dropout_2[0][0]                  \n",
      "                                                                 dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "Activation (Dense)              (None, 1)            21          FullyConnected-3[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "Concat-MF-MLP (Concatenate)     (None, 2)            0           Dot[0][0]                        \n",
      "                                                                 Activation[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "Combine-MF-MLP (Dense)          (None, 100)          300         Concat-MF-MLP[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "FullyConnected-4 (Dense)        (None, 100)          10100       Combine-MF-MLP[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "Prediction (Dense)              (None, 2)            202         FullyConnected-4[0][0]           \n",
      "==================================================================================================\n",
      "Total params: 10,925,173\n",
      "Trainable params: 10,924,573\n",
      "Non-trainable params: 600\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# print out model summary\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spliting data\n",
    "***\n",
    "We split 5% of the data from the whole dataset, and distributed the 5% equally into validation and test set. From the summary statistics, we can see that the matrix is really sparse: only ~0.03% of pair had class 1 labels. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# of all sample = 1713810000\n",
      "proportion of ones in all samples = 0.00038338905713002026\n",
      "# of split samples = 32679\n",
      "proportion of ones in split sample = 0.00038136082762966724\n"
     ]
    }
   ],
   "source": [
    "# split data for training vs. validation/testing\n",
    "data_size = n_playlists*n_tracks\n",
    "\n",
    "# split a portion of pairs for validation and test\n",
    "split_ratio = 0.05\n",
    "split_num = int(data_size*split_ratio)\n",
    "\n",
    "split_tracks = np.random.choice(n_tracks, split_num).reshape(-1,)\n",
    "split_playlists = np.random.choice(n_playlists, split_num).reshape(-1,)\n",
    "split_values = np.array(sps_acc[split_tracks, split_playlists]).reshape(-1,)\n",
    "\n",
    "split_pair = [(tr, pl) for tr,pl in zip(split_tracks,split_playlists)]\n",
    "\n",
    "print('# of all sample =', data_size)\n",
    "print('proportion of ones in all samples =', sps_acc.count_nonzero()/data_size)\n",
    "print('# of split samples =',int(np.sum(split_values)))\n",
    "print('proportion of ones in split sample =', np.mean(split_values))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split the saved pairs into validation and test data\n",
    "test_num = int(split_num/2)\n",
    "\n",
    "test_tracks = split_tracks[:test_num]\n",
    "test_playlists = split_playlists[:test_num]\n",
    "test_values = split_values[:test_num]\n",
    "test_target = keras.utils.to_categorical(test_values, num_classes=2)\n",
    "\n",
    "val_tracks = split_tracks[test_num+1:]\n",
    "val_playlists = split_playlists[test_num+1:]\n",
    "val_values = split_values[test_num+1:]\n",
    "val_target = keras.utils.to_categorical(val_values, num_classes=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A trick here: We masked the splited data by setting the value to 2, and later excluded them during training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/shihyitseng/anaconda3/lib/python3.6/site-packages/scipy/sparse/compressed.py:746: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient.\n",
      "  SparseEfficiencyWarning)\n"
     ]
    }
   ],
   "source": [
    "# put split sample values to 2\n",
    "sps_two = sps_acc\n",
    "sps_two[split_tracks,split_playlists] = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define training data generator\n",
    "***\n",
    "One important step we took for training was to artificially ***balance the class labels***. Since the class 1 pairs were extremely sparse, a network trained on the original data can end up predicting everything to 0. Actuaslly that was what we got when we started with the naive way. In the current version we presented here, the training generator yielded augemented data with equalized class number. This would force the network to pay more attention on the class 1 samples, and bias the network to have a higher prior to class 1 than the real prior (which was around 0.1%). But the \"high\" false positive rate is actually what we want! We need to make recommendation on extending the songs in a playlists, and the \"false positive\" samples would be the candidates for recommendation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define training sample generator (with balanced classes)\n",
    "\n",
    "# build one-hot encoder\n",
    "exampleCase = [0., 1.]\n",
    "ohe = OneHotEncoder(categories='auto').fit(np.array(exampleCase).reshape((len(exampleCase), 1)))\n",
    "\n",
    "def trainGenerator(sps_two, ohe, batch_size=10000, balance_ratio=0.5):\n",
    "    while True:\n",
    "        n_tracks = sps_two.shape[0]\n",
    "        n_playlists = sps_two.shape[1]\n",
    "\n",
    "        # find indices for non-zero values\n",
    "        index = scipy.sparse.find(sps_two)\n",
    "\n",
    "        # find number of ones and zeros in final yield\n",
    "        num_ones = np.ceil(batch_size*balance_ratio).astype(int)\n",
    "        num_zeros = batch_size - num_ones\n",
    "\n",
    "        # extract tracks, playlists and labels\n",
    "        tr = index[0] # track\n",
    "        pl = index[1] # playlist\n",
    "        la = index[2] # label\n",
    "\n",
    "        # find where label == 1\n",
    "        is_one = la == 1\n",
    "        tr = tr[is_one]\n",
    "        pl = pl[is_one]\n",
    "\n",
    "        # select class one samples\n",
    "        num = tr.shape[0]\n",
    "        one_ind = np.random.choice(num, num_ones)\n",
    "        one_tracks = tr[one_ind]\n",
    "        one_playlists = pl[one_ind]\n",
    "        one_values = np.ones((num_ones,1))\n",
    "\n",
    "        # over-sample 2 times of target number for class zero randomly (and later exclude non-zero samples)\n",
    "        rand_tracks = np.random.choice(n_tracks, num_zeros*2)\n",
    "        rand_playlists = np.random.choice(n_playlists, num_zeros*2)\n",
    "        rand_values = np.array(sps_two[rand_tracks, rand_playlists]).reshape(-1,)\n",
    "\n",
    "        # check to find class zero samples\n",
    "        is_zero = rand_values == 0\n",
    "        zero_tracks = rand_tracks[is_zero]\n",
    "        zero_playlists = rand_playlists[is_zero]\n",
    "        \n",
    "        # grab zero entries with the target number\n",
    "        zero_tracks = zero_tracks[:num_zeros]\n",
    "        zero_playlists = zero_playlists[:num_zeros]\n",
    "        zero_values = np.zeros((num_zeros,1))\n",
    "\n",
    "        # concatenate class one and zeros\n",
    "        sub_tracks = np.concatenate((one_tracks, zero_tracks))\n",
    "        sub_playlists = np.concatenate((one_playlists, zero_playlists))\n",
    "        values = np.concatenate((one_values, zero_values),axis = 0)\n",
    "        target = ohe.transform(values).toarray()\n",
    "        \n",
    "        yield [sub_playlists, sub_tracks], target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation generator was defined similarly, generating samples from the validation set with equalized class number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_ones_val = np.array(val_values).reshape(-1,)==1\n",
    "val_tracks_ones = val_tracks[is_ones_val]\n",
    "val_playlists_ones = val_playlists[is_ones_val]\n",
    "\n",
    "val_tracks_zeros = val_tracks[~is_ones_val]\n",
    "val_playlists_zeros = val_playlists[~is_ones_val]\n",
    "\n",
    "# define validation generator (with balanced classes)\n",
    "def valGenerator_balance(val_tracks_ones, val_playlists_ones, val_tracks_zeros, val_playlists_zeros, \n",
    "                         ohe, val_batch_size=10000, balance_ratio = 0.5):\n",
    "    while True:\n",
    "        num_ones = int(val_batch_size*balance_ratio)\n",
    "        num_zeros = val_batch_size - num_ones\n",
    "        \n",
    "        num_ones_total = val_tracks_ones.shape[0]\n",
    "        num_zeros_total = val_tracks_zeros.shape[0]       \n",
    "        \n",
    "        ind_ones = np.random.choice(num_ones_total, num_ones)\n",
    "        sub_val_tracks_ones = val_tracks_ones[ind_ones]\n",
    "        sub_val_playlists_ones = val_playlists_ones[ind_ones]\n",
    "        sub_val_values_ones = np.ones((num_ones,))\n",
    "        \n",
    "        ind_zeros = np.random.choice(num_zeros_total, num_zeros)\n",
    "        sub_val_tracks_zeros = val_tracks_zeros[ind_zeros]\n",
    "        sub_val_playlists_zeros = val_playlists_zeros[ind_zeros]\n",
    "        sub_val_values_zeros = np.zeros((num_zeros,))\n",
    "        \n",
    "        sub_val_tracks = np.concatenate((sub_val_tracks_ones,sub_val_tracks_zeros))\n",
    "        sub_val_playlists = np.concatenate((sub_val_playlists_ones,sub_val_playlists_zeros))\n",
    "        sub_val_values = np.concatenate((sub_val_values_ones,sub_val_values_zeros))\n",
    "        sub_val_target = ohe.transform(sub_val_values.reshape(-1,1)).toarray()\n",
    "        \n",
    "        yield [sub_val_playlists, sub_val_tracks], sub_val_target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model training\n",
    "***\n",
    "We set up checkpoints during the training process, saving models every 10 epochs. Later we can look at the training history and pick a model based on ***validation loss***, since the model can overfit during the extended training process (total 500 epochs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set checkpoints\n",
    "filepath = \"models_10000_rand/model.{epoch:02d}-{val_loss:.2f}-{val_acc:.2f}.h5\"\n",
    "\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='val_loss', \n",
    "                             verbose=2, save_best_only=False, mode='auto', period=20)\n",
    "callbacks_list = [checkpoint]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit model with validation data\n",
    "history = model.fit_generator(trainGenerator(sps_two, ohe, batch_size=20000, balance_ratio=0.5), \n",
    "                              epochs=600, steps_per_epoch = 1, verbose=2, workers=1, use_multiprocessing=False,\n",
    "                              validation_data=valGenerator_balance(val_tracks_ones, val_playlists_ones, \n",
    "                                                                   val_tracks_zeros, val_playlists_zeros, \n",
    "                                                                   ohe, val_batch_size=10000, balance_ratio = 0.5),\n",
    "                              validation_steps=1,\n",
    "                              callbacks=callbacks_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save final model, training history, test/val data, and training matrix\n",
    "model.save('final_model_10000_rand_181211.h5')\n",
    "\n",
    "history = history.history\n",
    "with open('model_history_10000_rand_181211.pkl', 'wb') as f0:\n",
    "    pickle.dump(history, f0)\n",
    "\n",
    "with open('test_tracks_10000_rand_181211.pkl', 'wb') as f1:\n",
    "    pickle.dump(test_tracks, f1)\n",
    "with open('test_playlists_10000_rand_181211.pkl', 'wb') as f2:\n",
    "    pickle.dump(test_playlists, f2)\n",
    "with open('test_values_10000_rand_181211.pkl', 'wb') as f3:\n",
    "    pickle.dump(test_values, f3)\n",
    "    \n",
    "with open('val_tracks_10000_rand_181211.pkl', 'wb') as f4:\n",
    "    pickle.dump(val_tracks, f4)\n",
    "with open('val_playlists_10000_rand_181211.pkl', 'wb') as f5:\n",
    "    pickle.dump(val_playlists, f5)\n",
    "with open('val_values_10000_rand_181211.pkl', 'wb') as f6:\n",
    "    pickle.dump(val_values, f6)\n",
    "    \n",
    "with open('sps_two_10000_rand_181211.pkl', 'wb') as f7:\n",
    "    pickle.dump(sps_two, f7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training history\n",
    "***\n",
    "Plotted below is the training/validation loss during the training process. Both the training/validation loss dropped rapidly during the first 100 epochs, and and started to slow down between epoch 100 to 200. The training and validation loss started to diverge after epoch 200, as the training loss kept going down but the validation loss started to increase. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAGDCAYAAAAyM4nNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd4lGXWx/HvTRIEQg+dIE0E6SC2FRUFUVwLIhbEygqKvYvdxe666qrYlVVX4VVRREWwi4oFUEEQEaQL0ntPuN8/Th6emWQCATKZZPL7XBfXU+aZ4R4UTu52jvPeIyIiIiVfmUQ3QERERAqHgrqIiEiSUFAXERFJEgrqIiIiSUJBXUREJEkoqIuIiCQJBXUREZEkoaAuUko55+Y657oluh0iUngU1EVERJKEgrqIRHHO9XfOzXLOrXTOjXLO1cu575xzjzrnljrn1jjnpjjnWue8doJz7lfn3Drn3J/OuesT+y1ESicFdRHZwTl3DHA/cAZQF5gHDM95uTtwJLA/UBU4E1iR89qLwMXe+0pAa+CzImy2iORITXQDRKRY6Qu85L3/EcA5dzOwyjnXCNgGVAJaAD9476dHvG8b0NI5N9l7vwpYVaStFhFAPXURiVYP650D4L1fj/XG63vvPwOeBIYAS5xzzznnKuc8ehpwAjDPOfelc+6wIm63iKCgLiLRFgENgwvnXDqQAfwJ4L1/3Ht/INAKG4a/Ief+BO/9KUAtYCTwRhG3W0RQUBcp7dKcc+WCX1gwvtA51945tw9wH/C9936uc+4g59whzrk0YAOwGch2zpV1zvV1zlXx3m8D1gLZCftGIqWYgrpI6TYa2BTx6wjgdmAEsBhoCpyV82xl4HlsvnweNiz/cM5r5wJznXNrgUuAc4qo/SISwXnvE90GERERKQTqqYuIiCQJBXUREZEkoaAuIiKSJBTURUREkoSCuoiISJIocWlia9So4Rs1apToZoiIiBSJSZMmLffe1yzIsyUuqDdq1IiJEycmuhkiIiJFwjk3b9dPGQ2/i4iIJAkFdRERkSShoC4iIpIkStyceizbtm1j4cKFbN68OdFNSRrlypUjMzOTtLS0RDdFREQKKCmC+sKFC6lUqRKNGjXCOZfo5pR43ntWrFjBwoULady4caKbIyIiBZQUw++bN28mIyNDAb2QOOfIyMjQyIeISAmTFEEdUEAvZPrzFBEpeZImqCfSihUraN++Pe3bt6dOnTrUr19/x/XWrVsL9BkXXnghM2bM2OkzQ4YM4bXXXiuMJouISBJKijn1RMvIyODnn38G4K677qJixYpcf/31Uc947/HeU6ZM7J+jhg4dusvf57LLLtv7xoqISNJSTz2OZs2aRevWrbnkkkvo2LEjixcvZsCAAXTq1IlWrVoxePDgHc927tyZn3/+maysLKpWrcqgQYNo164dhx12GEuXLgXgtttu47HHHtvx/KBBgzj44INp3rw548ePB2DDhg2cdtpptGvXjj59+tCpU6cdP3CIiEhyS7qe+tVXQ2HHsPbtISeW7rZff/2VoUOH8swzzwDwwAMPUL16dbKysjj66KPp3bs3LVu2jHrPmjVrOOqoo3jggQe49tpreemllxg0aFCez/be88MPPzBq1CgGDx7MmDFjeOKJJ6hTpw4jRoxg8uTJdOzYcc8aLiIiJY566nHWtGlTDjrooB3Xw4YNo2PHjnTs2JHp06fz66+/5nlP+fLl6dGjBwAHHnggc+fOjfnZvXr1yvPM119/zVlnnQVAu3btaNWqVSF+GxERycN7iPFveSIkXU99T3vU8ZKenr7jfObMmfznP//hhx9+oGrVqpxzzjkxt42VLVt2x3lKSgpZWVkxP3ufffbJ84z3vjCbLyIiu/Laa3DuufDhh3D88QltinrqRWjt2rVUqlSJypUrs3jxYsaOHVvov0fnzp154403APjll19ijgSIiMhe8h6OOAKGDYNJk+ze9OmJbRNxDOrOuZecc0udc1Pzed055x53zs1yzk1xziX95G/Hjh1p2bIlrVu3pn///hx++OGF/ntcccUV/Pnnn7Rt25Z///vftG7dmipVqhT67yMiUqqtXg1ffw1nn20BHqAY5Pdw8Rqudc4dCawHXvHet47x+gnAFcAJwCHAf7z3h+zqczt16uRz11OfPn06BxxwQKG0u6TLysoiKyuLcuXKMXPmTLp3787MmTNJTd39mRb9uYqI5GPKFGjXzs6vvBIef9zmf6+6qtB/K+fcJO99p4I8G7c5de/9OOdco508cgoW8D3wnXOuqnOurvd+cbzaVBqsX7+erl27kpWVhfeeZ599do8CuoiI7MSCBeF5MeqpJ/Jf+/pAxJ8KC3Pu5QnqzrkBwACAfffdt0gaV1JVrVqVScH8joiIxMfCheH5tm12LAYLlRO5UC7WjzQx/0S898957zt57zvVrFkzzs0SERHZhcie+vz5dly2LAzwCZLInvpCoEHEdSawKEFtERER2bUhQ2Dlyuig/tdfdrz3Xst+9v77iWkbiQ3qo4DLnXPDsYVyazSfLiIixdrll9sxcj/68uXh+QcfFG17colbUHfODQO6ADWccwuBO4E0AO/9M8BobOX7LGAjcGG82iIiIrLXgmF2sEBeuTKsXRsd1BMsbnPq3vs+3vu63vs0732m9/5F7/0zOQEdby7z3jf13rfx3k/c1WcWV126dMmTSOaxxx7j0ksvzfc9FStWBGDRokX07t0738/NvX0vt8cee4yNGzfuuD7hhBNYvXp1QZsuIiIF9dln4fmCBdAgZwY54t/gRFNGuULQp08fhg8fHnVv+PDh9OnTZ5fvrVevHm+99dYe/965g/ro0aOpWrXqHn+eiIjk4913w/MlS8KgntusWUXTnhgU1AtB7969ef/999myZQsAc+fOZdGiRbRv356uXbvSsWNH2rRpw7uR/0PkmDt3Lq1bW26eTZs2cdZZZ9G2bVvOPPNMNm3atOO5gQMH7ijZeueddwLw+OOPs2jRIo4++miOPvpoABo1asTynKGgRx55hNatW9O6desdJVvnzp3LAQccQP/+/WnVqhXdu3eP+n1ERCSGrVth7Fho1Ci8l19Qb9YMcuJBUUu+rCQJqL2akZHBwQcfzJgxYzjllFMYPnw4Z555JuXLl+edd96hcuXKLF++nEMPPZSTTz4Zl0+CgqeffpoKFSowZcoUpkyZElU29d5776V69epkZ2fTtWtXpkyZwpVXXskjjzzC559/To0aNaI+a9KkSQwdOpTvv/8e7z2HHHIIRx11FNWqVWPmzJkMGzaM559/njPOOIMRI0ZwzjnnFM6flYhIMlqzBjZtgiOPhKByZmZm7GfbtoWcgltFTT31QhI5BB8MvXvvueWWW2jbti3dunXjzz//ZMmSJfl+xrhx43YE17Zt29K2bdsdr73xxht07NiRDh06MG3atF0Wavn666859dRTSU9Pp2LFivTq1YuvvvoKgMaNG9O+fXtg56VdRUQkRzCiWZCe+t//Hvfm5Cf5euoJqr3as2dPrr32Wn788Uc2bdpEx44d+e9//8uyZcuYNGkSaWlpNGrUKGap1UixevFz5szh4YcfZsKECVSrVo0LLrhgl5+zs5z++0T8BJmSkqLhdxEpniZPtsBZvfref9bKlbaP/L77IDsbNmyA3UlmVtCg/sQTMGDAXjV1b6inXkgqVqxIly5d6Nev344FcmvWrKFWrVqkpaXx+eefM2/evJ1+xpFHHslrr70GwNSpU5kyZQpgJVvT09OpUqUKS5Ys4cMPP9zxnkqVKrFu3bqYnzVy5Eg2btzIhg0beOeddzjiiCMK6+uKiMRf+/bQuXPhfNZNN8Ejj8A771jJ1Fq1Yj930klwzTV57wdBvWpVCHYsNWmS97nLL4eyZQunzXsg+XrqCdSnTx969eq1Yxi+b9++nHTSSXTq1In27dvTokWLnb5/4MCBXHjhhbRt25b27dtz8MEHA9CuXTs6dOhAq1ataNKkSVTJ1gEDBtCjRw/q1q3L559/vuN+x44dueCCC3Z8xkUXXUSHDh001C4iJUMw2rg3NcrPOQdmz4bx462nDlZ05ccf839PkA3u0Uej2xIE9fLlYfhwmDo1Oqg3bQrNm+95WwtJ3EqvxotKrxYd/bmKSMJs2QLlytn5nsapYDrTezjxRMv29uqrcO65dn/rVkhLy/89YHXTq1WD006DESPgiy/gqKPyPj90KFxwwZ61c5dfo+ClVzX8LiIixc8u1g3tlu3bw88LeuxgK9p3JZg2HTHCjuXLx36uSpU9b18hUlAXEZHipzAX8K5YEX5e7qDuPbz1Vv7V1ZYujb6uUCH2cwrqIiIi+SjMnvqiRZYBDvIG9U8+gdNPh5tvzv+9kfLrqReTTJ5JE9RL2tqA4k5/niKSUIUd1BfnFAHNHdSDYD90KGRlha9lZ9trf/4Z/Vkafo+/cuXKsWLFCgWiQuK9Z8WKFZQLFqmIiBS1vQ3qkcPps2eHRVcig/rq1WFd9JUrowuzvPgi1KkDjz8e/bnFPKgnxZa2zMxMFi5cyLJlyxLdlKRRrlw5MvNLgSgiEm+7CuqbNtnq+NwJu7y3gL5+fXjvt9/C89w99SCoAyxcGJ7feKMdc2cBVVCPv7S0NBo3bpzoZoiISGGJDOrr10NOuWrAAvG++9o2tffesx52SorlW3/+ebj4Yvj+++jnAcqUgb/+Cu+vWRNdI33OnOjXYskvp3vurXEJkhTD7yIikiQ++ghefjl69XulStHPBNvM3n/f5sHT06FDB7v3n//YccyY8PkgqHftGr4Xwp56RoZd55ecK6cKJpB3ZKCYUVAXEZHi47jjLInLzobfI1NjB8PsQea57dvtGJFOe8eweq9e0Z+zerX1zg85xK7zC+pnnJF/Wz7+GJ56Kv/Xi5iCuoiIJMbkyXDmmZbZLbfcW8kiV6ZHBvVq1cLzDRtg5kw7/+678H6w17xLl+jPnDrVPivIEJe7PsdFF0Ht2nl/GIjUrRsMHJj/60VMQV1ERBLjs8/gjTfCQBzpp5+ir9euDc9jFLECYNw424oWGegj5+KrV4/eTz5+vB0PPNCG8CPn1MGC9V9/5V/8pRhSUBcRkcTYsMGOs2fbMTJw5w7qq1fbcc2a6EVwgSpVYOxYWxF/+unh/cjyqFWrQr16eX//5s2hYUPIVVeEunUL/l2KCQV1EREpfJmZcNVVO38mmA8Pgnpkopdp0+z43//asWlT60l362Yr3HOrUsUW2R15ZHTN86CSWoUKVhI1CNRBjz09HerXh0svDd9z333Qs6cNvZcwCuoiIlL4/vwzb+KW3HIH9ch94lu22LFOnfDek0/m7U2DBfr5822x3HHHhcG4TBmoXNnOgyAe9NTL5IS//fe3Fe2nnBJ+3sknW931MhEh8rvv4Ouvd/59igEFdRERSYxg+DuYyw5SuQZ7vp2DmjXD53/5JfbnRAb+7t3DOfCyZcM59erV7XjzzTYkH5RfDWqgRw61x8qmecghcPjhu/5OCaagLiIihaugKbsje+qffQbDhtl1s2Z2LFcuemHbxx/H/pzIYfL99w+vI4N6MLd+wAHWq2/Xzq6DoJ6SEn5GCU6RnRQZ5UREpBiJ3H62M0FQnzbNEsMEmjSBX3+14FqQ9KtB4E9NtUAe9O4jg3rDhtHvCZ4Jgnqk/FLBlgDqqYuISOEqaDGWYPg9UpUq4fB5uXLhnDjAgAGxPycIwsGwff36cNBB8Mor4ahB7m1phx0Gp54a/cNEoAT31BXURUSkcEUG9W+/zf+59eutNx2pSpVw/rtCheic6s8+a/vQcwuCevBZaWnwww/QowesWGH3glSwgYwMePvt6GBfoYId88vvXgIoqIuISOGKzNv+t7+FveXXX7fV58Hw/Pr10KJF3vcHAXj//e343HMwYYKdl4kRtnL31GO1pSAJZH76CZ55Jnp+vYTRnLqIiOy9ceMsCNepk3f4ffVqy/J2+eWwapVlacvMtKB+6KE2DN+hA7z1lgX8oBZ6MN/dv3/0540ebT8ctG9v17l76pHuvdd64KeeuuvvsP/+4Q8SJZR66iIisne2b7f86Uceade5g3pQkzw93Y5B1bQNG2yofdYsuOQSu5eVFW5RO/bY2L9fjx7h6nXYeU+9Xj0YMqRED6nvDgV1ERHJa/z4gq9iX77cjkEO98jhd7Ce+aZN4Zz1ggU2JB9ZJz0YHs/Kgn/8w+bETzhh57/vtGlWhGVnPfVSRkFdRESiffedJVq5996CPZ+7olrunvqpp1pAD1aVL1hgz2zfHgb1YItZVpbNmx900K5/35YtYd99d95TL2U0py4iItGCDG9B/vVdCTLBBWLNqUd+7vXXQ+PGdh4MydeoYceC/iARKfhhQUFdQV1EJKktW2YLzyKrk+1KkHe9oPPQuXvquYffA5ElU19/3Y5BetbU1IJnostt+3Y7avhdw+8iIkmtVi1LxrI7guAbBPXNm20RXFDy9J574Morw+cje+rbtu08+UywunzuXDvuu+/utS2WoIeeO2tcKaSgLiIi0ZYts2NqzmDu5Mnw1VcwcKBd3347PPFE2LOOLJm6dGneoD52bDh3fu65lmDm55/tujCCeocO8OKL8MILe/9ZJZyCuoiIRFu61I5Bjz3IylahQjjUDbZVbeNGGDkyvDd7dnRQv/hiq5zWurVdlytn29iys224vCBJYXbFOejXr2B54pOcgrqISGkQJHTZuHHXzwZBfc0aOwb7yitUiO6V//KLbTv76y94913rjd9zD6xcGT4TDLfffrsd69cP08A2aBA7Q5zsMf1piogkq8gAvno1vPmmrTafOnXn78sd1OfPt2NqKsyYET7XvTt8+SXcfz+cfDLcdht89BHcequ9/tZbcPXVdn7CCfDHH3DmmZZdDgpn6F2iKKiLiCSrYG4crPc8apSd//hj/u/Ztg1++83O1661YxDUV6yw4XWAyy4L33P66Xa86aboTG+9ekX3xJs0sesgqMfK+y57RUFdRCRZBZnewIJ6EGAj58VzGzvWgnf16tZTz86GSZPCz1u0yOawH3ssfE+TJuF527Z2LFfOnoslmHNv1Wr3vo/skoK6iEiyyt1TD4JsVpYF9iZN4Omno98zfrwNs59xhgX1YcNg+nSoXds+b/Fiy/6Wmmo9+GnTooP3AQfYMVaJ1ECwr71ly73/jhJFQV1EJFnl11Nfu9aqqs2ZA9deG/2edeugUiXL8LZ2LXzxhQ2XX3GFvTZ3bpgwpkGDvIG5aVM7Hndc/u0KevNBFTYpNArqIiLJKljoBjakHgy7r1oFb79t50FltUBQZKVaNXv+q69sH3iwqO2zz8KgHsuJJ8Ldd8Nrr+X/zJNPwsSJu5flTgpEQV1EJFlt2BCeL1kSLnxbtSosh5o7UUwQ1INtZ7//bkE9KIOalbXzoF6hgq2Cr1x5588ceODufRcpEAV1EZFksWhRdP70IKjvu6/tLw967itXhuerVtkxyNe+fr0NvwdBHSxxTJ06YeU0JXkpthTURUSSwZw5ltjloYfCexs2WK84MxMWLowO5JEB/qqrbP/6uHF5e+oQ5o5/5hkblj/qqKL5TrLbFNRFRJJBkOntzTfDe+vXW7DOzLTXgxKokUF9xQp4/HHr4b/8cuygXqeOHTt2tOd79oz/95E9oqAuIpIMgvnyyHn0DRssqNevHx3UV6wIn4+cUx81yla45xfUIf+951IsKKiLiCSDYE96ZGrYDRssQGdm2nlQmGXePDuPrJfetattgVu2LFz9HsjIiH/7pVAoqIuIlGQrVlhPfPRou47VU4+sM965c1jzvFGj8P7f/mbHtWstqEcGfBVdKTH0X0pEpLibNg3efz/2a19+aave33jDrlevtm1nEAb1IGCD1TMPRAb1yC1mQe1zKXFSE90AERHZhaAWeeR2tRdegP32i74Hlp518mQL0uvX29B55L7ynj2txjlY5bTWrS1dbOQzlSrF53tI3Cmoi4iUFFlZlnMdoH9/O0bmbq9d25LKfPGFBfWgpw7wxBM2l16rVvh8vXpw5ZV2PndueD/oqf/xR/QwvBR7Gn4XESkp/v1v64kHCWMgzAwHllN9//3h+uutAtqSJWFQv/xy+Ne/7DwoulK+fPjemjXD8yCoN2kS7lGXEkFBXUSkpBg0CF56yXrQgcig3q4d9Otn57/+atvTgqAe6frr7Rg55B75XI0ahddmKVIK6iIixcnPP0ObNmHZ07vuin796afDdK0A774bnnftaj3yU04J78Va9Navn+1b79gx+v6++1qPPcjzLiWO5tRFRIqT22+HqVNtXvzss/O+/tNP0ddBbXKw7Wrp6TBypC2C+/DDMKd7brEqpE2ZYkPyKSl73HxJrLj21J1zxzvnZjjnZjnnBsV4fV/n3OfOuZ+cc1OccyfEsz0iIsVeWpod583L/5lu3aKvBw4M95cHHn/cFtXlLq26M1WqQNmyBX9eip24BXXnXAowBOgBtAT6OOda5nrsNuAN730H4CzgqXi1R0QkIT75BK67bufPZGeHe8uDoPr88/k///DD4ba0Bg1gyJC829D22w+2bo0eipekF8+e+sHALO/9bO/9VmA4kPv/Lg8ERXerAIsQEUkmxx4LjzwC27fn/0yLFjafDWFQnzo1OmnMjTeG5wccEC5sq1Ej/3zsytNe6sRzTr0+sCDieiFwSK5n7gI+cs5dAaQDucaURESSxOrV0UVSIs2aFZ5HJpPp18+SxRxxBBx6KFxwAUyfboE/COqVKyMSiGdPPdaPiLlSH9EH+K/3PhM4AXjVOZenTc65Ac65ic65icuCogUiIkVt0SJ47LG8WdwKYvnyXT/jfVg9DaB5c7jhBgvoYD30Xr3sPAjqSg4jEeIZ1BcCDSKuM8k7vP4P4A0A7/23QDkgzwZJ7/1z3vtO3vtONSMTJIiIFKWBA+Gaa2DSpN1/b1AhLbBhA0yYEH1v+fKwPCpYUM9PENSDhXUixDeoTwCaOecaO+fKYgvhRuV6Zj7QFcA5dwAW1NUVF5HiKdjqFTlcXlC5e+pXXw0HHwwLImYpFyyw/emBnSWBCYJ6qnYmSyhuQd17nwVcDowFpmOr3Kc55wY7507Oeew6oL9zbjIwDLjA+z0Z1xIRKQJBVbPZs+24YAFcdJGtMo8l8p+z5cstvevJJ8Pvv4ef0b17+MzChWFQHzhw5wvdghSvCuoSIa7/N3jvRwOjc927I+L8V+DweLZBRKTQBCvTZ8yw48UXW4KX3r3huOMs29vRR4e51devD9+7YoXNx7/3nuVUb5AzO/nbb+Ez8+ZZUL/8civAsjPBDwwK6hJBaWJFRApq82Y7zpxpx6CHnpJiAfuyy+D++8PnI+fRly+3HwAAVq6MnjsPTJ1qQb1q1V23JdjXrjl1iaCgLiJSUEHK1cWL7ZidbceUFBg61M4jK59FzqMvXx4WX5kyxQJ+ly7Rnz9ypO1nj5XCNbdgTj2yKIuUegrqIiIFFfTUFy2y4e+gt7xgAfzyi51Pngyff27nkSVSV6wIh+OnT4e//opeCNesGSxdaufHHLPrtvTsCU89BYMH7/n3kaSjoC4iUlBBT33rVgvYQVD/7LPwme+/t6B8xRXhyvaaNa2nvmEDZGba+2fNgoyMsGd/4ol2rFTJaqLvSpkytpiuXLnC+W6SFBTURUQKKuipgw3BB8PvQaW0OnXC15980nrSAE2bWs98yxY47LDwmYwMqFDBzs8/H0aPhrFjld5V9piCuohIQeUX1INMl7VrRz8fzL3vt19Yde3AA8PXI4N6hQrQo0d00BfZTQrqIiIFtWkT1K9v5wUJ6osW2SK6hg1h2za7V706tG5t5w0bwrXX2rmyZUohUFAXkdJh82ZbXZ7bzz/Hzsu+dSvcckv0a5s3hwloli4N59RzB/X09HBPe7Vq0Qvi0tPh00/hp58sj/tVV9kPBwXZxiayCwrqIlI63HgjnHoqfPdd9P0OHaBTp7zPv/aa7Tm/777w3qZN1qN2zvaTBwvntm2zBWtVqth1v35w/PF2XrVqdFCvWBFq1YL27e1znLNFbyKFQP8niUjp8Mcfdly+3Hrn778fvhbMd0f69Vc7RpY23bzZ5r6rVLHkMevWha+lp4fz41Wq2Ba14H5GRvRzInGi/IIiktxeeMG2iwXFWLKyoFs32ze+cGH+7wuyxkWuRN+8OeyRr14dXSY1MlhXqAAtW9r5okV5e+oicaKeuogkrwULoH9/OO20MKhv2RIG4Geeyf+9QU72yAQymzbZvvKqVeGDD+yzAunp0RnmgiH9ZctsQVxAQV3iSEFdRJJXkJv9zz/DoL5mTbigLeiNxxIE88igHtlTX7nSVsJXr26vpaeHC+dSUmxvOthK98iV7Rp+lzjS8LuIJK+NG+24fXsY1FevDsubBmlZYwmeWbXK0rr+73/2eUFPHaBzZ5g717LI5e6plylj76tRI3oIXz11iSP11EUkeQW51rdvt18QHdSDAiu5bdkSDq2vWmXV14JV8JGr3DMzw553ejoce6ydBwlkWrSInk8HBXWJKwV1EUleQVD3PjxfsyYse5pfTz0I+mBBPXLLWbly4Q8ImZnhivf0dDj5ZFs8d8gheT/zqKPsuM8+e/ZdRApAw+8ikrw2bLDj9u3h9rMlS8JeeKykMxAG9bJlYdo0+xUp+Kx69cIMc0Fwr1Qp9meOHm0r4ZXXXeJIPXURSV6Rw+/Beaw96WAJZCZMgJNOgiFD7F6w0A7gggvg5puhd+/ws6pUCVe5//XXzttSoYLlgBeJI/XURSR5RQ6/B73rIKg7Z/cjn3344eikNKedBiNG2Pmhh8LFF9v53/9u5VZbtAgrs02fHr/vIVJA6qmLSMm1dSvce290ZrdIsYbfgzztDRpEP7tuHXz9dbhFDeC222xv+qxZcNFF4f1rrrGh+4YNwyQzl1++999HZC8pqItIyfXMMxZ4n3wy9utBTz07OzwPZGZGX0+danPet9wS3qtSxRbGNW0abokD6+UHqV/T0qzHf8MNe/ddRAqBgrqIlFw//mjHChVs2DwobxoIAvm6dWHxlUDuoP7FF3YMVqlDuHVNpIRQUBeRkmvGDDt+841EBRmcAAAgAElEQVQtcAtqkwcie+oQXZwld1B/5x3butaqVXgv8nmREkBBXURKrlmz7Dhnjh3ffdeOPXpA377hnHqgdevwPDKop6XZZ+2/v2WM++47uOMOSNVaYilZFNRFpGRauzbcZx7kcF+wwOa3x4yB11/PO48eGdTr1g3PO3a0Y5s2djzkEPjnP+PTbpE40o+hIlIyBb1ziM4AN2VKeL5+vaVpDYJ/5NB6ZFA/7zybcx80KD5tFSkipbqn7n30vwsiUoLMnm3HtLTo+2++GZ6vXw9t24bXQfY3gMMPD88vvRQmTw577CIlVKkO6k89BU2awOLFiW6JiOzSsmXw++92/sorYfAOAnGtWnDggZaONbBhQ3Ta1sjz1FTrmZ92WnzbLVKESvXwe1BI6bPPbE2NiBRjBx1k2eCys+H88+1e1ar2k/n331tQz8yEDz4I37N+vVVFmzbN8r3n3tZ2//1F136RIlCqg3q79FkMqPAdE28sS8VPyrJs7T5s2pZKq2ZbWbnCU6Wqo3wFh3dlcGUcroyjTIrDpZTZcR5cB+fbvSNru12npDpS0+yYklYGV6kiZTKqsS2tAmnVKtKkSaL/BERKgL59bQ/5okV2/euv4WstW0K1anZes6bNk2dlha8HQT3I+vbLL0XSZJFEKdVBPeWrL3h2Y3/YCPy3aH/vJdRifNoBzKh2KF+0v5qqLeqQmWlJqnr31vZYKUV+/tmqoQWBN7fXX4++jszN3qKFpWoFaNQoevEb2AK5oN455F9BTSRJlOqgzpln4rsczbKFWyifspWKaVvYsn4bi5aXpWYtx9o1ns2bvK2o276d7dme7GyPz/Zsz9qO3+7Znu3tmHPt8KSm2HV2lic7y57J3pqN27Ce1HWrSN28ntQ5M6k1bzqHLHuYvh8/ytKPa7HBV2ABDbht4Olk9T2fhs3LcdppKuwkSa5DBztGFlfZmffeC88rVoTrroNjj7W/KMOHRz+7fbs9E1BQlyRXuoN6pUq4SpWoFRE0ywHBqHiR/PWfNYuUZ5+l/vIVZK3bRKOvv6Hbkkt47JXfeTD7eh58oA433+I47zyoXbsoGiRSzI0fH54fcYTlZA9+MKhXL+/zsYJ6rOdEkkCpXv1eLOy3H/zrX7ihL5H21jD2+XMOnH46V2c/wmLq8RIXcueNG2nTJv8y0CJJIbJ2+a4cdJAlnOndO/p+rVp5n40cfi9bFl58Eb76as/aKFLMKagXNykpMGSI9UCAnqtfZlXtFnRe9yGNGlm5Z5GkNH/+rp8JVpd26RJ7Xqp1a6t1/sAD4b3InjpAv35olaokKwX14qhmTRg3zuYDP/2UfTIqMWLL3zm9xS/ccINty41cKySSFCJXtefn7rtjF24JlC9vfzkiE8vkDuoiSUxBvThzDo45Br76CleuHMM5kwcHzuWnn6B//+jMmCIlUuT2s1NPhW+/hV698h+K79YNRo2COnV2/rmRgVxBXUoRBfWSoHp1uPtuyvw2nRt/OZcJP3iWLIGbbir4gmGRYimoopaRYSNTf/ublUAdPhxOOQVWrox+PtaceSyRgTxyTl0kySmolxTXXQfPPQdff02nOW9y1VXw7LPQsyds3pzoxonsoSCoX3ll9P1HH7UeeUZGeO/66wv+udWrh+fqqUspoqBekvTrB+3bw5ln8sjHrXlm0FxGjYL//Gf3Fg6LJMzWrbBkSXgdBPUmTaK3meXO/PbUU/DQQwX/fRTUpZRSUC9JUlJgxAho1w43bRoDZt3AfvtZTYpzzkl040QK4MwzbT58+3ZYvToM8BUrRldTy86Gc88NrytVsjUmuyPIUKegLqWIgnpJ06SJpdW8+WbciBEMHbyAihWtYNWQIYlunMgujBxpx+XLLWd7ztZN0tPzbjPr1ClM+7on8+Kffw7PP6+sTVKqKKiXVBddBN7T+dfnWLnSdvlceSVMmJDohonkY/v28PzPP6NfixXU69cP59T3pLddq5b9PREpRRTUS6omTawO9KOPkvbrZF55xba3Dx6c6IZJqbdsmeViD8qjBiLnyefOjX4tVlCvWzcM6rs79C5SSimol2QPPQRVqkDXrlRdNYe+fS3vhqpLSkJ99RV88gm88kr0/dtvD8979Yp+rWJF2Hff6Ht168JRR4Wvi8guKaiXZE2a2LafFSugSRMu7jiB1FTo3BlWrUp046TUWrs2771162D0aNuaGUt6et60r3Xrwh13wJdfwqGHFn47RZKQgnpJd+CBO7b67D/vY8aMsX9Tn3oqwe2S0isyqGdn23H8eDs/7rjY76lUyUadIrMplStnOz6OPDJ+bRVJMgrqyeCGG+CAA+Cbb+ja1RbN3XknPPlkohsmpVJkUP/sM0ukMG4cpKZaxrhnn41+vmZNZX0TKSSlu556Munc2VJrrl7NE09UZdIkuOIK2xWkkUspVN7Dpk1QoUL0/bVrLThHBvULLoBFiyyg77efvT5ggO1Pv+MOuPVWuOSS6M955x1Yvz7uX0MkGamnniwuvdTmLQcPpmFDmDHDtgHfeCOMGZPoxklSGTbMgvOUKeG9rCwbPh84MDqo77NP+HrknPntt9sPB/fcA5mZ0Z/fs6eyKYnsIQX1ZNG+vfV4Hn0UfviBihXt39evvoIePSzXh0ih+OEHO0auZl+xwo7PPx89vL5wYXjerFn82yZSyimoJ5MHH7RhzkMOgVde4aqrbJ0RKCmNFKLy5e0YmUBm6dLYz27bFp5Xrhy/NokIoKCeXCpXDuc5zz+fWrXCrW0nnAALFiSuaZJE1q2z45o14b1ly/J/vkULOx54YPzaJCKAgnryeeYZO5YrB9u3U6lSmKgreElkj7z5JjRuHP6kuGYNfPEFfPBB2FO/6qq87zvnHJg3z7ZliEhcKagnmz59bF5z82aYNQuwuhYNGtjuIpFd8t6GdnInO7joIkvvGqR4Xb0ajj4aTjwRpk2ze9265f28mjXzZosTkbhQUE9GwR62vn1h+3b23dd2Fv3wg027Z2UltHVSXHzzDbzxhp1v3Qr/93+wciXsvz98+CFcdln088H8eDCPEzlf/sgjUKaMLdjMrUaNwm+7iMSkoJ6MWre27DMTJ+7YdnTNNXDyyVZ7/ZFHEtw+KR46d7b65t5bUoOzzrKKQDkjPHTqFP381q12jLU4Y+NG65HXq2fXRx8dvhaUTxWRuFNQT1YXX2zH88+HdeuoVg3efhu6d4e77oKPPkpo6yRRfvgBmja1ofPAvHnw3HN2Huw9P/poC9733GO51wcNClO+5ta6tR1r1bLe+uzZ8N574esHHVT430NEYoprUHfOHe+cm+Gcm+WcG5TPM2c45351zk1zzr0ez/aUKnXrwuGH2z/SOflinbPh9/R0OOOM6H/XJUlF5lIHuPtuC7pffBHeez3ir92kSZYwpnVry/p2++22+O3BB6M/p2bN8LxxYzsecEB4HaR9PfZY22YpIkUibkHdOZcCDAF6AC2BPs65lrmeaQbcDBzuvW8FXB2v9pRKX34JHTtaCcx16+DPP2nf3qpirlljKeNz/5svSWThQus5v/MO/PyzzYEHe8XXroX69e388cfD96xda4va6tQJ702enPezGzQIz6tUsWO7dtHPrF1rldlEpMjEs6d+MDDLez/be78VGA6ckuuZ/sAQ7/0qAO99PhksZI+kpFjk/u03+8c8MxNWraJdO7j5ZnjhBbhaP0Ylr7fesuPQodChgw3PBEF9zZpwoduSJRbEgxwHDRvaUDpARkbszw5Su1aoEKaFDfZOBipVUi9dpIjFM6jXByJX1CzMuRdpf2B/59w3zrnvnHPHx7E9pdNZZ8H114fXkyYBNlU6cKB10n78MUFtk/j6+ms7Vqxox5Ejw2xwS5dG52ivVy9cpb7vvuHw+cEHx/7soCdfpYrVHQA46qjCa7uI7JECBXXnXFPn3D45512cc1c656ru6m0x7uUe7E0FmgFdgD7AC7E+1zk3wDk30Tk3cdnOMldJbOedF57nBPUyZeD++60jdtNNCWqXxNfUqXacPz+8N326HefOtVwGgRo1woVwrVuHK9Yj658/9FB43ry59cyfe86e8V6r3EWKgYL21EcA2c65/YAXgcbArha1LQQiJt7IBBbFeOZd7/027/0cYAYW5KN475/z3nfy3neqGblARwqmTRsbgm/Y0La55ahSBW67zebY33wzge2Twud9GMyDZDEAv/9uxyC4BzIywlzu7dpBly7w/fdw5ZXhM8cfH33+xx+WeEZEio2CBvXt3vss4FTgMe/9NcCufiyfADRzzjV2zpUFzgJG5XpmJHA0gHOuBjYcP7ugjZfd0Ly57Uv+8ktYvBh694aJExk4EBo1sunWc86B7dsT3VApFMuXW81ziC68Mjvnr9dvv0U/Hzl3Hix4O/hg2zIRqFMHXn3VVsu3bImIFD8FDerbnHN9gPOB93Pupe3sDTk/BFwOjAWmA29476c55wY7507OeWwssMI59yvwOXCD937F7n4JKaBjj7XCG4cdBiNGwJNPss8+tjj6tNPgtdeUSrbEys6GDRvC63nzdv588GwwZJ6RYRmKypbNuzjuvvvCZ845x1IRi0ixVNCgfiFwGHCv936Oc64x8L9dvcl7P9p7v7/3vqn3/t6ce3d470flnHvv/bXe+5be+zbe++F7+kWkAI491o7BP/g//wxYZs9XX7WFzCNGJKhtsneuv94WxAVZ34L/xuXK2bFs2XCRXKRgO1r16pZqcMuWvM/cfLMN55dRriqR4q5Af0u9979676/03g9zzlUDKnnvH4hz26Sw1asXbjFq1Qp++QV++gmwf+9PPBH+9z/rseeXPEyKqcces2NmJsyZE86nt2plx4yMcF96pEqV7KitZyJJoaCr379wzlV2zlUHJgNDnXPKIF4SBZnBnn8eate2lfEzZwLwwAOWTOycc+Dss3c9gitFbOpU61kH8+IXXwzDhkVX6Fm2zH4qW7zY/mM2bGj3a9QI95ZHql3bji7WZhURKWkKOp5WxXu/FugFDPXeHwjEqLEoxd4111h2ucMOs43qU6daVa7Jk2lcYx1zvv2L666z4l2NGoVbnSVBtm0Lt54NG2Z7y++9F2680baTnX02/Pqrvd6mjR3vv9/2pNepE/bEa9a0/84QFl2BcLtj7uItIlIiFTSopzrn6gJnEC6Uk5LIuTAZySGHhPe/+AJ69qTS/nW59YatO26ff77VY5cEOfTQcC68WjU7vvQS/Otf4TPBlrUXX7QFEhs32uhLnTrhf+sWLaBtWzsvWzZ87+mnw4oVKroikiQKGtQHYyvV//DeT3DONQFmxq9ZUiQis4W9/PKOpe/VPnmTadPs9uzZcMwx0XlKpAgF6f68t6F1yLtgbc4cO2ZmhtndwIbW16yx81atwmpq69ZFv7969cJts4gkTEEXyr3pvW/rvR+Ycz3be39afJsmcVe1qhXruPXWHQvmAHj5ZVo2z+ZU3iad9fyL6/n4zdVs2BCO9EocffyxDZGvXx/eW748DOq5h06+/NIWutWqBf37W2U1sJ59sDCiWbNw0VyQJS5tp7tSRaQEKuhCuUzn3DvOuaXOuSXOuRHOuRirbqTEadvWEsE3bWrXaWnWY7/xRt7mNMZXPp7r+Tc/n/dvKla0uLBxY2KbnPRuvNEWus2YEd6bNcvytXfoAEceGf38O+/YyvaUFLsOCqts2gTdcpa+tGlji+V++82G6W+8EcaPj/93EZEiVdDh96FYNrh6WFGW93LuSbLYd187Xned7Wd7xDY3tF37DQDlCMff09OtMylxEuwnDDLCgaVkXbYsrGM+bpxV4+nfP+/7g7n37GwbhQmqsIFlFixXznZBaHGcSNIpaFCv6b0f6r3Pyvn1X0BJ2JPJBRfYMZ9sYXX4K6qa20svxb9JpcqmTbZ4Yfz4MFfv8uVh8pjnn4cffghLoh5xBFxxBdxyi11HDp8cfzz06wcPP2y99+A9IpL0CppxYrlz7hxgWM51H0DpXJPJeedBjx7WE3z1VRuCX7XKtkYBp9b7nootNnEZLzKbJgwffgLNm1uysX32SXDbS7KvvrIa587ZXHn37tAgpw7SsmVhhrhx4+yYO0lMo0aWMah58/DePvvYELuIlDoFDer9gCeBR7HyqeOx1LGSTIKh3XPOsV8ff2wrradNo+KMGdCmDU/yB2tTqlIlexX//KctnI4s5CW7KZgfD5Lub9gQFltZsCC6wk5GhqWDza1v3/i2UURKjIKufp/vvT/Ze1/Te1/Le98TS0QjyezYYy0ZfJAQ/o8/AChbryaP9RjLSdW+ZvBVy4Ppd9kbs2blvRdsVevTx3rzkyaFK9hFRGLYmwoN1xZaK6R4a9UqzDwGlFuzhKsmnMOoVUewnJoMvm61veB9dJlPyd+yZTa9EQh+cLrzzvBeENSPP972mwcpX0VE8rE3QV3JokuTyICydq0t4srRmql8+SWW5SwoKCI7V6tWuI0QwqmOM88M7+VU0duR6lVEZBf2Jqj7QmuFFH/B4q0Y+jCMOV0uYNmtj9qNH34ookYVcz7nr8isWdF7AIMsb5E99e3bbR/5AQdYQqAzzghrniuoi0gB7TSoO+fWOefWxvi1DtuzLqVFkH0sqOoVkTf+Mp7iAl6mZtZfAGR/P4GRI+GZZ4q6kcXIt99az/unnyybW/fu4WtTpkQ/GwTtYOtZ27bwwgt5XxcR2YWdBnXvfSXvfeUYvyp571WAuTQ59VRLL/rWW7at6vrrYetW/FFH5Xl0+ZiJnHqqFYHzpXU8J1jNHtQ5B0v7etJJeTPCdelix8j95JUqhdfaMygiBbQ3w+9SmlSpAmPGQOfO8Ndf0Ls3pKXhcgX1NypeCNOnA547uYsVYycmpr2JFtQ4/+678N6558L7MYoc3norXH013H139P1337U/76BkqojILiioy+7LyAjPjzjCjo0bwx9/cPB5LajNUra5stzFP0kdcKElQpkyxRKsRFYRS0YjR1qN+kWL7Do4Bq9dcQV8+qkF7MBBB8Gjj8Lf/hb9WYceaslpKlSIf7tFJCk4X8LGRzt16uQnTiylvb/iaOtWG2e/5hor7fnOO9ArTGGwlJrUYln0e7Zvt8B25JHRtb1Lug0bwvrlzZpZTfPcFiywHQKLF1uN8yeeiNouKCKSm3Nukve+QMUa1FOXvVO2rPXEg1rdEUPFP7c4K29ABxuSPvZY+1WSbdsGPXvCN9/Y9MR994WvxQroYNXUAOrWhdWrFdBFpFApqEvhCvZeX3op7Z8aEPuZadPsOG6cBcaS6o8/bBi9Sxfbux8E9aDiXSzOxT4XESkECupSuMqVg5UrrSzo0UfDeecxu3HX6GeCleEAQ4eGi8pKmqDYSu72Dx1qWwD79Yu+36ZN0bRLREotBXUpfNWqWclPgP/+l1VvfsLRqV/xDBfbvWHDwmcvvpiNdzxQ9G0sDKtXx77fpg1s3gy33x7emzfPhulFROJIQV3iyzkOPBA+29qZE+Y9w+0MzvPIe/+ewdYxn8Hpp0dXJSvugsxwuWVkWOKZoOod2JC8ksiISJwpqEuRcM7iWqf/u5HNaRX5uOE/dryWvTWbsj26WmKb4lQQJjvbhtJXroy+P2sWzJ+ff0+9TM5fq/R0Ox5zTPzaKCISQVnhpEidcsY+cNpqunkg7UUAuvPRjtdfHTyHboMbULdughoY+O47GDwYPvzQVqpH7jdv1syOwRQD2Da1Nm1g3broz1m8GKpWjX97RURQUJdESEmJKvFXgxU7zre88AqLxj1N3UnPh3u+//lP2yLWvDncdFPR7G0/7LDwfPFiy8X++utw8cXh/ezs8Lx+fRg9Ou/n1KkTvzaKiOSioC6J8+WXtq3t9ttZTRWqsoaLeBF+h20XeJakZpJZbQO8+mpYsWzTJvjtNxsWr1Il+vMmT7ZiKJFbxTZvtopn99xjrxVEUNs80sCBtsr9889jvydGDnwRkaKmOXVJnCOPhNtugxEj2PhOOAT/OV1IG/F/ZP7fv63UWxDQAe6/37LW/d//hfe8t9Km7dtbTfeePW0POcD338N774U97C1brBceKwe791Y2tm/f8F6w7z4rK/o9zz8f/d7IKmwiIgmioC6J16sX9XoeDHfeyeab/0lX9znjiRj+jpWkZdUqC8LffGN741991e7fdJMlhHntNbteutSOweK1yZNtvrxnT7tesCBMhvPyy1ZSdssW2442Zw5MmGA/LJx8Mvz97+HvH1lp7X//02I4ESkWFNSl+LjrLsrddwfHHw+n8C438iBbU8rBP/6R99lBgyzYnn++JYEJgnrg00/tGPTYg0VtEybYMTvbet8HHmgpbs88Ey68MHx/y5bQqJHtuR83Dt54w+4PHWr3GzcOn+3bV9nhRKRYUFCXYmfUKJi3oSbTTriRzrVnkf34ELZWrcmW40+JfnDKFFi4MO8HtGoF335rvfTZs+3eipzFeN9+Gz7Xpw8sy8lNHwTtQGZmeF6pUljT/IILrAeflgavvBKOCIiIFAMK6lLspKZatdF+/WDCovqc+4+y7LN6KcdOvD/6wTlzrLb79OnhvcMPh+HDLaf8kCFhT33uXJgxw4J3//6Wp/2tt/JvRGRQz8+558LZZ+/29xMRiReVXpVia9s26NrVSooDlGULWyhnFzffHF0V7euvoUEDaNjQrk8/Hd5+2zLUOWfz7wCVK8Pvv1u2t4wMSyBzxhk2nN+xY/h5W7YkV1lYESmxVHpVkkJamtV+GT0afvwR9mu5D6fWHs/Mb5dHB3SAzp3DgA7w1FNhytnXXrO5c4AHH4TatW3hXK1adu+YY6Bdu+jPU0AXkRJIQV2KtdRU6NEDOnSwNWpfbj2MXv0z6NTJ4na+ata0+W+wD/juO5tjv+SS8Jlq1exYr54F+WXLYPlyWxEvIlICKahLiXHwwXD11TB1KkyaBJddFr2FPY8XXrCFdFWr2k8HkQVWIBxuD5LY1KhhQ/IFmU8XESmGlFFOSpTrrrMOdloaXHop9OoFY8bks6MsJcXSt+bn3/+2RDRHHBG39oqIFCUFdSlR0tPhiivsfOtWuOoqW8R++ul78GHly9sKdhGRJKHhdymxLr3U8s+ccYaNrj/+eKJbJCKSWArqUmKlpsLYsVanJTvbMsRu3x4uehcRKW0U1KVEq1XLksTdeqsVZEtPt0RxIiKlkYK6lHgVKoTz7Js3W9K4J56w/DEiIqWJgrokhdq14dBD7bxDB7jySq2BE5HSR0FdksaHH8LPP1tJ9PPPhxEjYP78RLdKRKToKKhL0qha1bK9pqbCnXfagrk330x0q0REio6CuiSlxo3hgAPg+edh2DD4299g7dpEt0pEJL4U1CVpHXecVVs9+2xbIb/TXPEiIklAQV2S1qBBYU0XgLvugsmTE9UaEZH4U1CXpFW7tlV2W7TICrSlpsKzzya6VSIi8aOgLkmvbl0r0Na9O4waBStWwN13W954EZFkooIuUmr07g3vvGMVVgOPPZZPhTcRkRJIPXUpNc44Axo0iL53881hNjoRkZJOQV1KjdRUmDLFtrkFHnwQnnwS5s1LXLtERAqLgrqUKlWrwkUXwcqV0fdffz0x7RERKUwK6lIqVasGr7wCp50GzZvDu+8mukUiIntPQV1KrXPPhbfegr594fvv4fDD4aOPEt0qEZE9F9eg7pw73jk3wzk3yzk3aCfP9XbOeedcp3i2RySW886zFfHjx9sK+WXLEt0iEZE9E7eg7pxLAYYAPYCWQB/nXMsYz1UCrgS+j1dbRHamYUNbQHf11bBunQ3Hb9qU6FaJiOy+ePbUDwZmee9ne++3AsOBU2I8dzfwELA5jm0R2am6deFf/4Jrr4VVqywTnYhISRPPoF4fWBBxvTDn3g7OuQ5AA+/9+3Fsh0iBpKbCww/DkUda3vi5c60m+7ZtiW6ZiEjBxDOox8rT5Xe86FwZ4FHgul1+kHMDnHMTnXMTl2nCU+LIOXj5ZRt+79jR5tiffjrRrRIRKZh4BvWFQGT+rkxgUcR1JaA18IVzbi5wKDAq1mI57/1z3vtO3vtONWvWjGOTRaBRI8s+t2qVXf/+e0KbIyJSYPEM6hOAZs65xs65ssBZwKjgRe/9Gu99De99I+99I+A74GTv/cQ4tkmkQB58MDwfMgQeegg2bkxce0RECiJuQd17nwVcDowFpgNveO+nOecGO+dOjtfvK1IYMjOtXOs119j1TTfBvfcmtk0iIrvivPe7fqoY6dSpk584UZ15KRqbNlk99ptugo8/thzxVasmulUiUpo45yZ57wuUx0UZ5UR2onx5aNoUbr0V1q61PezTpiW6VSIisSmoixRAhw5w5ZU2JH/KKTBuXKJbJCKSl4K6SAH95z/w4YfWY+/WDU49NbqMq4hIoimoi+yG44+HmTMhIwNGjoQBA2D58kS3SkTEKKiL7KYqVWDsWDjhBLsePtyOyjwnIommoC6yB9q2hQ8+sJzxP/wA//iHLarbf38YMybRrROR0io10Q0QKck6doRXXw2vZ86E++6zYXoRkaKmnrrIXujQwY4NG1owB9iyxY4//ABvvpmYdolI6aSeusheOP98WLIEbrgBmjWz82efhe3b4ZJL4I8/rCiMi1XeSESkkCmoi+yF/faD554Lr9u1g82bLcDPnm335s+3nryISLxp+F2kEJ1xhs2nBwEd4JdfEtceESldFNRFClF6elh/PSPDjv362a+vvkpcu0SkdFBQFylkjRrBG2/AhAlw+eWwYgUMHWppZkVE4klBXSQOTj8dGjeGJ56A33+HQw6BqVNh9epEt0xEkpmCukicNW0KTz4JWVnQpYv13CPn3EVECouCukgR6NTJcsVPmQI1aligf+YZ+O03WLAg0a0TkWShoC5SRE45BW65Jby+6io44AA45hjYtAlGjIAhQ8D7xLVRREo27VMXKUL//CcccQS0aGEL6pPEIGwAABXJSURBVABmzbJ97MuW2fXRR0PLlglrooiUYOqpixShlBQ47rjoZDQdO1pAr1HDrseNS0zbRKTkU1AXSZA77rDjZ59ZIJ8/H+rVg//8x2q0T52qcq4isnsU1EUS5K67LGhXqWJD8uXLw2232eK5mjWhTRu44opEt1JEShIFdZEEcQ5Sc61qGTgQLroovH7uOfj+e1izpmjbJiIlk4K6SDHz+OMwdqwF8rp14dBDoWpVuOce69nXrw/335/oVopIceR8Cds/06lTJz9x4sREN0OkSHz1FZx0Uuye+vz5NkxfrlzRt0tEio5zbpL3vlNBnlVPXaQYO+IIWLoUPvkk72v77gvXXKN97SISUlAXKebKlrXgXqcO3HADLFwIAwbYsPwzz0CDBjBtWvR7FOhFSiclnxEpAcqWtWBepowtsHv2Wfj0U+jWDf78E848E378ESZPhmrVoFkz+OgjOPbYRLdcRIqSgrpICZGSEn3dtavNq//4I/TsaXPr3tuwPMDw4bZN7qCDYJ99ir69IlL0NPwuUoI1aGA55d98E84+2+7Nn2/HH3+0YfvGjW1PfN++sH59wpoqIkVAq99FksiRR9qKeediz6t37w7Z2fD221C5ctG3T0R2n1a/i5RSL74ITz0FV14Z+/WPPrK5+FGjirZdIlI0FNRFkkizZpaV7sQTd/7c+PFaIS+SjBTURZJQt24wciRcfLFd161reeUDTz9t9558ElatSkwbRaTwaU5dpJTYvNkW1p18Mrz0Uni/Z08491wrIJOSAhkZVmRGRIqH3ZlT15Y2kVKiXDmr2w7Wk09Lg5tvth79yJG2v33VKtsqFyuDnYgUf+qpi5RiGzdCly4wYUL0/RL2z4JIUtPqdxEpkAoV4NtvrQffv7/dS0mBTZtg61b46y+7d/nl8MgjiWuniBSMgrpIKZeSAjVqWO32d96xfexHHgmDB9tiuh49YMgQuO46e37ePNi+PbFtFpHYFNRFZIfjjoNevWDiRLj3Xrs3Zkz4+vz50KiRZagTkeJHC+VEZIfy5S3l7H77wZw5cOedVh1u+nR4/HEYMcKeu/tuqFXLnjn1VDvff//Etl1EtFBORGKYMwfuuAMeeADq14exY+H443f+nunToUWLommfSGmihXIislcaN4ZXX7WADtC+vS2qA1st/8EHMHs2TJoUvue99ywNbQnrJ4gkFQ2/i8gu1a4Na9bAihVW271aNbvfuDH8/DOcdBLceKPdGz3aFtdFWrEC5s6FAw8s0maLlDrqqYtIgaSmWnAPAnqgXTsYNCi8vuQSuOgiC/jHHAO//QZNmkCnTrBtW9G2WaS0UVAXkb126aXwyy9WJc57eOstm1//5Rc44ABYu9aeK1sWTjvNKsmtWQOzZml7nEhh0kI5EYmb0aPh73+P/Vr37jYH/+CD4dC9iOSlhXIiUiwcd5yVg73hhryvffSRHd96K7w3Z44F+OzsommfSLLRQjkRiZuUFPj9dzvv0sVW07dvb9epqZCVBTNm2Cr6zz+3Yfk5c6BvX5urF5Hdo6AuIkXihBPs+O67tuDul18s5ez999siukizZyuoi+wJBXURKVInn2zHQw6x40EHwaefWi/+//4PfvwRZs601777ztLSli0Lb79tc/RbttiCvDp1EtJ8kWJNC+VEpFhxzo6dO8PXX9t5UOs9cM01qhonpYcWyolIiRcEdIgO6ACPPgr/+x9s3gz//KfNy193XVgqVqS0UlAXkWLlk0+scMxvv8GSJbYS/q+/8laGe/BBeP99u9+ihfXcL70U+vSxLHe5/fILdO0K69YVxbcQSQwNv4tIibBhA1SsaOf9+8Pzz9t8+9y5sZ//3/9sFX3gqKNg3DgYNcrS2oqUFBp+F5Gkk55ue9jHjAlX0s+dCy1bxn7+3nttFf3s2Xa9dasd58yJe1NFEkar30WkxHjwQTtmZcHTT9te92OOgaZN7X5qKpx+uqWmveMOOPTQcPvcn3/aM2PG2N75+++HSpVstX1mpi3GS0tLzPcSKSwK6iJS4qSmWuGYwIIFVhq2fHkoUwa+/NLuL1tmv2rUsEpxAB9+aMeMDLjzzrByXMWKNt++cKEF+ypViu77iBQWDb+LSImXmQnVq1tQ32cfOOwwWzz38MN2vWKFpasdMSJ8z3PPRc/Hr18PvXpBgwZw4olF/hVECoUWyolIUluyxHr2qanW+16+3PLNDxxoC+beey/2+7ZtgyeegPPPh2+/tZXz5coVbdtFoBgtlHPOHe+cm+Gcm+WcGxTj9Wudc78656Y45z51zjWMZ3tEpPSpXduG2oPh9Bo1LA89hAH922/D5+vWtePFF8O110KtWtZzv/XWImuyyB6LW1B3zqUAQ4AeQEugj3Mu9zrVn4BO3vu2wFvAQ/Fqj4hIoHlzW2AXCFLWtmwJU6dCvXrw0kt2L6gYF6SuBVi50urGixQ38eypHwzM8t7P9t5vBYYDp0Q+4L3/3Hu/MefyOyAzju0REQEsFe2nn9pQ/IwZdj13rvXYq1eH//4373uWL4dZs2x1fUYGPPQQnHUWjByZ99nPPoMPPlDgl6IXtzl151xv4Hjv/UU51+cCh3jvL8/n+SeBv7z398R4bQAwAGDfffc9cN68eXFps4gIRCe6ufBCGDp0588vXmwFZpYssZXz6el2f8QIW3wH9lrt2vFrsySv4jKn7mLci/kThHPuHKAT8K9Yr3vvn/Ped/Led6pZs2YhNlFEJK/0dMta9+ijNgz/0Ufha/feG26ZC3z8sc3PN2wY/jAA0K+f1Yr/6isL+oMGqfcu8RXPfeoLgQYR15nAotwPOee6AbcCR3nvt8SxPSIiBfbcc+H5scdaMP7zT5tvd84y023fbolvzjsv7/v/8Q8rEXv11bbYDix5zujRNmTfpIltozvnHMtZ37170XwvSW7xDOoTgGbOucbAn8BZwNmRDzjnOgDPYsP0S+PYFhGRvVa/fnjeqJEdL7oIXnjBzkeMgNNOs/MnnoD994ebbrLra66Bdu0sgPfta9nrmjeHd9+1X6tXQ+XK9gPD9u3hZ150kSXUESmIuO5Td86dADwGpAAvee/vdc4NBiZ670c55z4B2gCLc94y33t/8s4+U/vURaQ4ycqyErDffGO97XvugQkTrHDMsmVhL33uXBuev/ZaG9bPrVs3y1P/3ntw333w2mt2//vv4eCDi+zrSDG0O3PqSj4jIhJH8+bZqvmuXe16yxb44w/b9z5ypA3HX3pp3mpz550Hr7xiOe5794bx4+Hzz21Iv0wZ+OILaN++qL+NJIKCuohIMbd0qQXmM86Ar7+2nv62bXD77VZcZskSKFs2//dfdRU89lh4vXw53HKL3W/Vyu6tXWs9/7PPtmF9KZmKy+p3ERHJR61aFtABOne2ufcbb7Re+Ztvxq4Yl5ICjz8O++1nwXruXFvAt2iRLeZ7/nnbQrdwoT0/aJAtxMu9Wl+Sl4K6iEgxUbasBfRgqD7YwXv88Za7fuv/t3fvMVKVZxzHv48LuFsQULBoWAtuIC3WcFXE0iC1tFhSa7BGRKNgSDDECzVCK5YoxBoj4gUC0drWSw0KagsaTJB1XWoqCKIsCMJWLFsERdAiYDXIyts/nneYmXUF0WFn95zfJ5nMOe85jOc8Mjxz3uvncP31cPvt3v5++un+6toVamp8GN727Z7gq6t99TrwNvobbvC2/1zbt3sPfMiuOy8tm5K6iEgztWpVttPd7t3ZXvCXXurT3Hbu7EvOjhwJCxb4MLy774ZNm/z44sV+fmWl98bP7G/f7jPjlZf7ZDkDBvjQvNxV7BrTwlprU0lt6iIiCfLJJzB6dDaBX3OND6cbMsTb8UeOhIULG/+zgwd7+/5HH3kzwJgxMGiQL2P71FMwapRX9WcWvZGmoTZ1EZGUatfO29s3b/ZOd1Om+Hj4yZP9eMOEnvt0/sor3r7fubN34rv6aujVy4/dc4+/jx3r69XPng2vvgp79vgQvfp6780/bdoxvkE5LD2pi4ikxJYt3mN+xQqv1gdfha6kBM47z8fEN2x3B2/jr6pq/DN79PAfEJMmwcyZXvbpp15Vf/CgV/WXlHj1vnrgfzMa0iYiIoe1Z4/PYtetm2+XlsKaNd4Z7/PPfVGbNWv8KXzvXv8zU6fCiy/C2Wf74jRTpzb+2UuWwPjxsHVrtuyRR/wpH+Dll328/qZN3kwwbZo//R9OXZ33AXjwQZ9lr7T0WwagBVFSFxGRggjBk39dHfTu7U/dGRdd5J34MgYP9ir8E06Affu8rKLCe9aXlcHw4XDlldmpdDNKS2H1ah9ff8cdsHatd/zLPNkvXgwXXuid+55+GqZPh1tvzb/GJNcCqE1dREQKwswnw+nXLz+hAzzxhCf8jOpqH2K3b58n/Pp6nz2vstKr4hct+nJCb9/eV8U780yfZnfqVE/cFRXe+3/gQE/o4OWQHaoHPuNemzY+Wc/hvPVWdvhekimpi4jIN9K2LXToAOvXe+Ju3Roee8wT8m23ZX8EDBsGtbVeJd9QeTnMmOHblZXZ8ro6OOecbNt/rszkOvv2+bj9+npf2vaqq/yJ/6ab/Nhrr3lTQG2t1wJcfrn3IVi9Oju3/o4dyUr2qn4XEZGC+qrq8P37YcIE+Owzf7KurvZe+TNm+LHHH/cheFu2+A+FsWPhsst8jH1Dc+bAvfce/aQ5ZWX+3wfvS9Chg3cSXLbsaO/SZToaHktqUxcRkWZv40bvPZ87Je6BA9n9THp69lnfnj3bl7N94w1/2s5YudLb+2fO9M88cCBbVX84t9zis+2BL6zTtat/Tqbn/pGS9fLl3o9g+fIjd/T7NpTURUQksQ4e9ARfWwtz5355vfkQ4MknvS9AWZmvTZ+pbu/Xz3v1N6ZTJ+8DMHy4v9fUwKOP+miA6dO/fP6NN/oQwT59vP1/zhzo3x+GDi3k3Sqpi4iI5NmwwWfJW7oUOnaE55/3Nv4+fXwWvYzJk32qXcjvxV9Z6dX248b5D4n58318/rp1fnziRJg1y7c/+MAX7CkUJXUREZEjCME72c2a5c0AF1/sZQMGwJ13em988MV0unTx5W3378//jOuu8457ixZly045xcfgd+hQmOs8mqTeqjD/SRERkZbFzNvvJ03y/WXLvH189GiflGfECG9rnzfPO+yF4PPgV1R4r/m9e72n/dat2aS+YoW39xcqoR8tJXURERF80ZshQ7L7CxZ4gh46FLp39yQ+bJgfO3jQ3487zo/Nn+/JfdAgfxWLqt9FRESaMc0oJyIikkJK6iIiIgmhpC4iIpIQSuoiIiIJoaQuIiKSEErqIiIiCaGkLiIikhBK6iIiIgmhpC4iIpIQSuoiIiIJoaQuIiKSEErqIiIiCaGkLiIikhAtbpU2M9sF/KeAH9kZ+LCAn9fSKR75FI98ikeWYpFP8chXyHh0CyGc/HVObHFJvdDMbPXXXdIuDRSPfIpHPsUjS7HIp3jkK1Y8VP0uIiKSEErqIiIiCaGkDg8V+wKaGcUjn+KRT/HIUizyKR75ihKP1Lepi4iIJIWe1EVERBIi1UndzC4ws1oz22xmNxf7epqCmT1sZjvNbH1O2UlmVmlmb8f3E2O5mdnsGJ91Zta/eFdeeGZ2mplVm9lGM9tgZhNjeVrjUWpmq8xsbYzH9Fh+upmtjPFYYGZtYvnxcX9zPN69mNd/LJhZiZmtMbPFcT/NsagzszfNrMbMVseyVH5XAMyso5k9Y2ab4r8h5zaHeKQ2qZtZCTAX+AVwBjDazM4o7lU1iUeBCxqU3QxUhRB6AlVxHzw2PeNrPPBAE11jU6kHbgoh9AIGAdfGvwNpjcd+4PwQQh+gL3CBmQ0C7gLui/HYDYyL548DdocQegD3xfOSZiKwMWc/zbEA+EkIoW/OUK20flcAZgFLQgg/APrgf0+KH48QQipfwLnACzn7U4Apxb6uJrr37sD6nP1a4NS4fSpQG7f/CIxu7LwkvoBngZ8pHgHgO8AbwDn4BBqtYvmh7w3wAnBu3G4Vz7NiX3sBY1CO/8N8PrAYsLTGIt5XHdC5QVkqvytAe2BLw//HzSEeqX1SB7oC7+bsb4tladQlhPA+QHz/bixPTYxidWk/YCUpjkesbq4BdgKVwDvAxyGE+nhK7j0fikc8vgfo1LRXfEzdD/wWOBj3O5HeWAAEYKmZvW5m42NZWr8rFcAu4JHYPPNnM2tLM4hHmpO6NVKmoQD5UhEjM2sH/A34TQhh7+FObaQsUfEIIXwRQuiLP6UOBHo1dlp8T2w8zOyXwM4Qwuu5xY2cmvhY5BgcQuiPVyVfa2ZDDnNu0uPRCugPPBBC6Af8j2xVe2OaLB5pTurbgNNy9suB94p0LcX2gZmdChDfd8byxMfIzFrjCX1eCOHvsTi18cgIIXwMLMP7GnQ0s1bxUO49H4pHPN4B+G/TXukxMxj4lZnVAfPxKvj7SWcsAAghvBffdwIL8R99af2ubAO2hRBWxv1n8CRf9HikOam/BvSMvVnbAJcBzxX5morlOWBM3B6Dty1nyq+KPTcHAXsyVUtJYGYG/AXYGEK4N+dQWuNxspl1jNtlwDC88081cEk8rWE8MnG6BHgpxAbDli6EMCWEUB5C6I7/2/BSCOEKUhgLADNra2YnZLaBnwPrSel3JYSwA3jXzL4fi34KvEVziEexOxwUubPDCOBfeLvh74t9PU10z08C7wMH8F+P4/C2vyrg7fh+UjzX8BEC7wBvAmcV+/oLHIsf41Vg64Ca+BqR4nj0BtbEeKwHbo3lFcAqYDPwNHB8LC+N+5vj8Ypi38MxistQYHGaYxHve218bcj8e5nW70q8x77A6vh9WQSc2BzioRnlREREEiLN1e8iIiKJoqQuIiKSEErqIiIiCaGkLiIikhBK6iIiIgmhpC6SMmb2RVxpK/Mq2AqFZtbdclYAFJGm1erIp4hIwnwWfCpYEUkYPamLCHBovey7zNdUX2VmPWJ5NzOriutAV5nZ92J5FzNbaL7++loz+1H8qBIz+5P5muxL4+x0ItIElNRF0qesQfX7qJxje0MIA4E5+FznxO2/hhB6A/OA2bF8NvCP4Ouv98dnGgNfM3puCOGHwMfAr4/x/YhIpBnlRFLGzD4JIbRrpLwOOD+E8O+40M2OEEInM/sQX/v5QCx/P4TQ2cx2AeUhhP05n9EdqAwh9Iz7vwNahxD+cOzvTET0pC4iucJXbH/VOY3Zn7P9Beq7I9JklNRFJNeonPcVcXs5vlIZwBXAP+N2FTABwMxKzKx9U12kiDROv6BF0qfMzGpy9peEEDLD2o43s5X4D/7RsewG4GEzmwzsAq6O5ROBh8xsHP5EPgFfAVBEikRt6iICHGpTPyuE8GGxr0VEvhlVv4uIiCSEntRFREQSQk/qIiIiCaGkLiIikhBK6iIiIgmhpC4iIpIQSuoiIiIJoaQuIiKSEP8HupGpqReqal8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot loss of training process\n",
    "fig, ax = plt.subplots(1,1,figsize = (8,6))\n",
    "ax.plot(history['loss'],'b',label = 'Training')\n",
    "ax.plot(history['val_loss'],'r',label='Validation')\n",
    "ax.legend()\n",
    "ax.set(xlabel='Epoch',ylabel = 'Loss', title= 'Loss');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the accuracy of training and validation data. It reflected similar trend as indicated by the loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAGDCAYAAAAyM4nNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xd4VFX+x/H3ofdeBWlSpFfBgoANFRcLVtRdG6L8FBuuYt1dC3YURUV0BXVRVkUQVwQbFixUESlKkd470gk5vz++ubkzSYCAmUxm8nk9zzz33nPLnEyS+d5Tr/PeIyIiIomvQLwzICIiIjlDQV1ERCRJKKiLiIgkCQV1ERGRJKGgLiIikiQU1EVERJKEgrqIiEiSUFAXSRLOua+cc5udc0XjnRcRiQ8FdZEk4JyrA5wMeODcXHzfQrn1XiJyaArqIsnhb8CPwHDgqiDROVfcOfeMc26pc26rc26Sc6542r6OzrnvnXNbnHPLnXNXp6V/5ZzrFXGNq51zkyK2vXPuJufcAmBBWtqgtGtsc85Nd86dHHF8Qefcvc65Rc65P9L2H+2ce9E590zkD+Gc+8g5d1ssPiCR/EBBXSQ5/A0YkfY60zlXNS39aaAtcCJQAbgLSHXO1QI+AV4AKgOtgJmH8X7nAx2AJmnbU9OuUQF4G3jPOVcsbd8dQE+gG1AGuBbYCbwB9HTOFQBwzlUCTgPeOZwfXERCCuoiCc451xGoDbzrvZ8OLAIuTwuW1wK3eu9Xeu/3e++/997vAa4APvfev+O93+e93+i9P5yg/pj3fpP3fheA9/4/addI8d4/AxQFGqUd2wu433v/mzc/px07BdiKBXKAy4CvvPdr/+RHIpJvKaiLJL6rgE+99xvStt9OS6sEFMOCfEZHHyA9u5ZHbjjn+jnn5qVV8W8Byqa9/6He6w3gyrT1K4G3/kSeRPI9dXIRSWBp7eOXAAWdc2vSkosC5YDqwG7gGODnDKcuB9of4LI7gBIR29WyOCb98Y5p7ed3YyXuOd77VOfcZsBFvNcxwOwsrvMfYLZzriXQGBhzgDyJSDaopC6S2M4H9mNt263SXo2Bb7F29teBgc65o9I6rJ2QNuRtBHC6c+4S51wh51xF51yrtGvOBHo450o45+oD1x0iD6WBFGA9UMg59yDWdh54DXjYOdfAmRbOuYoA3vsVWHv8W8CooDpfRI6MgrpIYrsKGOa9X+a9XxO8gMFYu3l/4BcscG4CngAKeO+XYR3X+qWlzwRapl3zWWAvsBarHh9xiDxMwDrdzQeWYrUDkdXzA4F3gU+BbcC/geIR+98AmqOqd5E/zXnvD32UiEiMOOc6YdXwdbz3qfHOj0giU0ldROLGOVcYuBV4TQFd5M9TUBeRuHDONQa2YB36notzdkSSgqrfRUREkoRK6iIiIklCQV1ERCRJJNzkM5UqVfJ16tSJdzZERERyxfTp0zd47ytn59iEC+p16tRh2rRp8c6GiIhIrnDOLc3usap+FxERSRIK6iIiIklCQV1ERCRJJFybelb27dvHihUr2L17d7yzkjSKFStGzZo1KVy4cLyzIiIi2ZQUQX3FihWULl2aOnXq4Jw79AlyUN57Nm7cyIoVK6hbt268syMiItmUFNXvu3fvpmLFigroOcQ5R8WKFVXzISKSYJIiqAMK6DlMn6eISOJJmqAeTxs3bqRVq1a0atWKatWqUaNGjfTtvXv3Zusa11xzDb/99ttBj3nxxRcZMeJQj7YWEZH8Kina1OOtYsWKzJw5E4B//vOflCpVijvvvDPqGO893nsKFMj6PmrYsGGHfJ+bbrrpz2dWRESSlkrqMbRw4UKaNWvGjTfeSJs2bVi9ejW9e/emXbt2NG3alIceeij92I4dOzJz5kxSUlIoV64c/fv3p2XLlpxwwgmsW7cOgPvvv5/nnnsu/fj+/fvTvn17GjVqxPfffw/Ajh07uPDCC2nZsiU9e/akXbt26TccIiKS3JKupH7bbZDTMaxVK3juCJ/2PHfuXIYNG8aQIUMAePzxx6lQoQIpKSmccsopXHTRRTRp0iTqnK1bt9K5c2cef/xx7rjjDl5//XX69++f6dree6ZMmcLYsWN56KGHGD9+PC+88ALVqlVj1KhR/Pzzz7Rp0+bIMi4iIglHJfUYO+aYYzjuuOPSt9955x3atGlDmzZtmDdvHnPnzs10TvHixTn77LMBaNu2LUuWLMny2j169Mh0zKRJk7jssssAaNmyJU2bNs3Bn0ZERAA2boTly+Odi8ySrqR+pCXqWClZsmT6+oIFCxg0aBBTpkyhXLlyXHnllVkOGytSpEj6esGCBUlJScny2kWLFs10jPc+J7MvIpJvLV8Oo0dD377wyScwZAiceCIceyxcfz1s2AApKfaaNMkC/SWXxDfPKqnnom3btlG6dGnKlCnD6tWrmTBhQo6/R8eOHXn33XcB+OWXX7KsCRARkWgpKbB+vQXvzZvhu+/gzDPh1luhXj045xz46CO45x644AIL6ADPPw81asDpp8Oll9r6XXfBzp3x+TkU1HNRmzZtaNKkCc2aNeP666/npJNOyvH36Nu3LytXrqRFixY888wzNGvWjLJly+b4+4iI5LbVq2H79iM7d9486NAB0vodM2YM/PQT/P47nHsuVKtm+7t1gwoVoGNHOwcgaAFt2za8XrlyUKwY3HEH7N9v1wBYtQpGjbJ98eASrbq2Xbt2PuPz1OfNm0fjxo3jlKO8JSUlhZSUFIoVK8aCBQvo2rUrCxYsoFChw29p0ecqInlFSgoULmwl4s8+g19/Be/hQF9R330Hd98NlSrZ8tpr7Zx777VrPflk1uedfjpUr24lce9h/HhL370bNm2Cf/8bTj4ZmjeHK6+0kn3//vDII3YzcMEF8MorkNY6miOcc9O99+2yc2zStannd9u3b+e0004jJSUF7z2vvPLKEQV0EZFYmTMHhg2DJ56AggUtLTUVChSwduk//oBSpaw6vHFj+PlnCObm+vxz+PJLC77ew0svWcBOSYGpU2HfPli6FG65BcqUge+/hw8/DN97wICs8/Tii3D00dC9e3T6ffdZvooWtWB///3hvocfhoUL4cYb7edYtw4KFQp/prgIJkVJlFfbtm19RnPnzs2UJn+ePlcRyY6BA72//PKDH7N8ufctW3pfp4739et7D9737ev9xo3eDxjgfZUq3g8b5n3x4ravdm1b9u9vy4yvkiW979Il633gfevW3q9d6/3992feN3y492+/7f3554dpeRkwzWczRqr6XQ5In6tI/hSUmsFKyd9/b6XRjI+EmD0b1q61UjNYe/LOnTZfyFlnWU/xAQOsw9mIEfDVV5nfq0uXrNMzKlMGataEoO/vp5/CKadYlTxA/frw6KO2XasWtGxppWbvw6r6EiWsJN+gQXjdV1+12oCOHQ/jA8plqn4XEZFs+/vfrZPYmWfC44/DW2/B119bNXKrVnbM5MkwcKAFzVKlLMBffbV1NgvccotN/rVwIfzvf2H6++/bTcKbb1r1+ZtvhvuCgH7XXRZ4L7zQqtqvusrazkeMgM6d7b2GDoUbbrDObGecYecNH255njYNIkYQp3PuwO3uYEPTkomCuohIPuK99dQ+80x7Va4MTz9t++rXt4AM0LUrHH98eN4bb1insfXr4fLLrTQ8fbq1Ne/ZY8e8/z7UrQsTJsBrr8GiRdCmja136QJ//avVArz5ppXi77wT0ubQ4vHHw5qADh0suJcsGdYCgJXWwUrhgauuspcYBXURkSSzaxcsWGA9wLt3t+FahQpZqXrcOCtFByXpRx8Nz1u40Kqh773Xxlz/5z9w/vlWMu7d26rawdIBihe3TmuffAI33wxjx9p5pUrZTQFYj/F168Ibhx49rPT9xBPQurXlo2DBzFX7WZW6L7rIbiqSrXSdo7Lb+J5XXnmxo1znzp39+PHjo9KeffZZ36dPnwOeU7JkSe+99ytXrvQXXnjhAa87derUg773s88+63fs2JG+ffbZZ/vNmzdnN+sHFe/PVUQObsMGW+7aFaZt3+59kybRHcPq1vX+0ksP3KkseJUo4f2+fXadWbO8f+IJu5733g8ZYseMHev9TTd5/8sv3kd89UgMcRgd5TT5TA7o2bMnI0eOjEobOXIkPXv2POS5Rx11FO+///4Rv/dzzz3Hzoipi8aNG0e5cuWO+Hoiknds3myl6zPPhIsvDtPfe89KspUqWVV16dI2VnrOHJvKdO5cq1YPLF5sk6384x9WQn7kEYj8eho61Kq/58+3Ej3YOOy77gpLzL17W366d4fBg6FZM+t4JnlMdqN/XnnlxZL6hg0bfKVKlfzu3bu9994vXrzYH3300X7btm3+1FNP9a1bt/bNmjXzY8aMST8nKKkvXrzYN23a1Hvv/c6dO/2ll17qmzdv7i+55BLfvn379JL6jTfe6Nu2beubNGniH3zwQe+994MGDfKFCxf2zZo18126dPHee1+7dm2/fv167733zzzzjG/atKlv2rSpf/bZZ9Pf79hjj/W9evXyTZo08WeccYbfuXNnlj9XvD9XkWS2cKH3gwZ5n5oanf7TT96vWWPrbdpEl6SXLPF+zpxDl7jbtrXrnnCCbV9wgfeLFkW/z/Ll4fGSt3EYJfXka1OPw7NXK1asSPv27Rk/fjznnXceI0eO5NJLL6V48eKMHj2aMmXKsGHDBo4//njOPfdcXMbGozQvv/wyJUqUYNasWcyaNSvqsamPPvooFSpUYP/+/Zx22mnMmjWLW265hYEDBzJx4kQqVaoUda3p06czbNgwJk+ejPeeDh060LlzZ8qXL8+CBQt45513ePXVV7nkkksYNWoUV155Zc58ViJyUJs2WQezoM25e3fr7f3RRza/+DnnWPq778KMGdHn1qljy8qVrf362GNtuFaNGjb0bOlSG1J25ZXWRt29O/zwg5XC69WLvlaNGrbs3DlmP6rEgarfc0hkFXxQ9e69595776VFixacfvrprFy5krVBT5MsfPPNN+nBtUWLFrRo0SJ937vvvkubNm1o3bo1c+bMOeSDWiZNmsQFF1xAyZIlKVWqFD169ODbb78FoG7durRKG6dysEe7ikj2pKbabGZgncaCTmFg05H+5S+wZo0F2OuuCwM6WPX52WfbzGhBQK9dO3zaV+3a1pu8WzeoWhV69bJe6Mcfb/OPH3+8zYT2l7/ATTfZ8LTq1e3cu++2HugNG2bOs3OwcqV1nJPkkXwl9Tg9e/X888/njjvuYMaMGezatYs2bdowfPhw1q9fz/Tp0ylcuDB16tTJ8lGrkbIqxS9evJinn36aqVOnUr58ea6++upDXsdqbLJWNGJS4oIFC7Jr165D/HQiktHevTY16a5d1iZ97702Icptt1kb9k03Wdvz3Xfb8UGgzWj9els+9xy8/LJNQ9qqlbVp16pl7dxFikTfCGRXgQKZS+iRjjrq8K8peZtK6jmkVKlSdOnShWuvvTa9g9zWrVupUqUKhQsXZuLEiSxduvSg1+jUqRMjRowAYPbs2cyaNQuwR7aWLFmSsmXLsnbtWj755JP0c0qXLs0ff/yR5bXGjBnDzp072bFjB6NHj+bkk0/OqR9XJF9KTbUq7sWLoVEjG4N9/PEW0MEmRpk/32Yt69XLOppdeqlVh2cUWf7YutUe8fnrr3Zss2ZW8v/5ZwvoItmVfCX1OOrZsyc9evRIr4a/4oor6N69O+3ataNVq1Yce+yxBz2/T58+XHPNNbRo0YJWrVrRvn17AFq2bEnr1q1p2rQp9erVi3pka+/evTn77LOpXr06EydOTE9v06YNV199dfo1evXqRevWrVXVLpJNn39upeu6dW0c9QMPWMl71y4oX956gme0eLH1CN+5E95+26rQ33rLuqN162Y3BMOH2742bWxilUKFwklVIrXL1qSgItE097sckD5XSXZLltgzuk84wbZTUizIfvaZVXc3aWJPDcuqK8wpp9hMat9/bwH/iitsMpWrrrKgX6MG/N//ZZ5UReRwae53EZFsaNjQqspTU23c9qhRVooePNj2Z+yP2q0b9O0LTz1lj+Rs0sTa0++5x+ZEnzXLSvdZzYYmkhsU1EUk6a1aZcG7du0wbeJESwOYN8+GkHkf9jovXtyq2gMvvGBToYI9gSzw4IPhev36scm/SHapo5yIJKXNm+GDD2xM+Ikn2hjvoBr9iy/g1FPDY5s2tYB+001h2mOP2bJFCxtbHgR0kbwsaUrq3vsDTuoihy/R+lqIAGzfblXfzlmJ+/PPo/c/9JANFevTJ+vz+/aFF1+0NvYuXSytZ09rIxdJBElRUi9WrBgbN25UIMoh3ns2btxIsWLF4p0VkQMaORI6dbIJXQBeecXmQL/iCtuXMaCDTfASBPRFi2ys+RNPhPsbNLBx4599Zo/3nD07HGcukgiSovf7vn37WLFixSEnZJHsK1asGDVr1qRw4cLxzopIukcegZNOslJ006bWFt65M5Qta9OmRqpRw8Z9p6ZakB4xAkaPtn3PPWfjwgPLltlwNE2ZKnlRvuv9XrhwYerWrRvvbIhIjKxbZ0H5gQdse8oUC+hly8LXX4fHde0K+/fbhDCXXWbP9Qa48EL45htb//BDOPfc6OvXqmUvkUSXFEFdRJLL5MnWye3eey0wH3OMtZcH0uZUYvBg+Otfbb1IEZti9UATJz78sHV66949tnkXiaekaFMXkcQRPE0ssGdP9OQuDz5oJe0nn7QHlhQqFB3QI58DfsUV8L//2TSre/YcOKCDzdp23XWaDEaSm4K6iOSqVq3CR4iuXm0TuFSrZr3OR4yAZ56xTmpvvJH1+UOG2PKiiyxAn3NO1tOsiuRHqn4XkZhJSbESdDDD2v794b5jj7UZ2pYssUlhIseBX3st/O1vcNppNqZ81Sp7yMns2RbA1661nu4iEi0per+LSN50wQUwZoxN7AKwYoU9+ztSs2bw5ZfQr589/ASsTT1oNxfJ7/Jd73cRyXu8t4AONvb7hhus5J5RixZQuTK8+abN4jZ8uJ5QJnKkYhrUnXNnAYOAgsBr3vvHM+yvDbwOVAY2AVd671fEMk8iElsLF8Knn0aXyK++GsaNC7dnz7aHpVxySfR87DVq2INSROTIxCyoO+cKAi8CZwArgKnOubHe+8jnHj0NvOm9f8M5dyrwGPDXWOVJRGJrwQJ7Tnhkb3WIDuhggbxxY/jPfzKPGReRIxfL3u/tgYXe+9+993uBkcB5GY5pAnyRtj4xi/0iksfs2QM//RRuz5kDN94IRx1ljzINnkd+wQX2GFPnoGpV+OMPq47v08fGnhcoYEPS1OFNJOfEsvq9BrA8YnsF0CHDMT8DF2JV9BcApZ1zFb33G2OYLxE5Qtu2waWXwvjxNkPb8uVWtR75WNMhQ+D00+0FcMopNt68WDE47zx7iUhsxDKoZzXFQ8au9ncCg51zVwPfACuBTF1pnHO9gd4AtTSXo0iuWr8eiha1kvd331lJHexhKoE33oArr7ThafXqRZ9frVquZVUk34tl9fsKIHLwSk1gVeQB3vtV3vse3vvWwH1paVszXsh7P9R73857365y5coxzLKIAMyfb48fHTjQgnLZsjbsrHZtePxxK40H86qDTb1aoEDmgC4iuSuWJfWpQAPnXF2sBH4ZcHnkAc65SsAm730qcA/WE15EctH27dbeXb26bS9dCtdcAz/+aK9Ic+ZYmznYELVgylU9b1wkb4hZSd17nwLcDEwA5gHveu/nOOcecs4F/V27AL855+YDVYFHY5UfEcnaiSdaJ7dly6zDW926NvnLOefYI06nTLGx5NdfHwb0wE8/ZQ78IhI/mlFOJB/7/Xd7AhpA4cLW4a1JE3j/fRtyFkhNtep1Ecl9hzOjnP5NRfKJ3butuvz5560kXqOGlcYD+/bZQ1K+/DI6oIMCukii0DSxIvnEsmW2vOce6N3bHpKyahWUKAGvv27D1a65JnMVu4gkDv37iiSpRYus2nzyZGsr/2vaXI07d8I774TH7dxpY89FJPGpUk0kCY0eDfXr2wxvf/0r7NgRPocc7NGlL71k61dfHZcsikgMqKQukkSWLoWvv7bXwfTrZ9O1XnCBhqOJJBMFdZEk8PPPNt/6hAnw+ecHP7ZnT3jwQVvXbG8iyUVBXSQJXH89TJ0anda1qz0CNWPa22/nXr5EJHepTV0kCezcGa6XKWPLyy6DGTPgk0/Ch6vcemvu501Eco9K6iIJ7Kuv4NVXbfrWdu1shrexY62dvEmTcHha584wc6bN5y4iyUtBXSQBpKRA8+b2/PE77oDrrrPq9pUrbVIZgNtvt6FpBQtmPr94cQV0kfxAQV0kD0tNteW338Kvv8IDD9gT1EaOtPRSpayn+/LlcPHFWQd0Eck/FNRF8rDrroO5c6OHnb31lj3q9Pnnrf28QoX45U9E8hYFdZE86pdfYPhwWy9UCJ54wh7AMmYMDB4MtWrFNXsikgcpqIvEyeLFVnWecUa3L7+09GeesRL6Y4/BKafY7HD791sJvUiRuGRZRPI4BXWRODn7bPjtN6tKr1jR0iZOhNNOs/WGDW0imaOPDs8pWFDt5iJyYArqInGyaJEtp0+3jm7Nm8M//gHVq8P330PNmnpimogcHn1liMRJkSI2VO2cc2zpHHgPAwZAnTrxzp2IJCLNKCcSB3v2hOPLU1Js6b0tr7giPnkSkcSnoC6SS3btsuXTT0OxYjYGvW1bm4998mQ480x7zrl6tYvIkVL1u0gu+OUXaN3a5mN/911Lu/VWeOopKFzYtsePj1/+RCQ5KKiLxJj31ot9/34YMcJ6s0+dClWrxjtnIpJsFNRFYmj3bujSxarXS5a0R6EefbQCuojEhoK6SA7zHvr0gQsvhO++s4AOsGMHnHhifPMmIslNHeVEctjq1fDKK9YBbuBA6NbN0oNnmouIxIqCukgO++mncP2PP+CCC2zO9lGj4pcnEckfVP0uksNmzozebtYM6taNT15EJH9RSV3kT9q7N5xI5tdf4bnnoEmTcH/kuohILCmoi/wJM2ZA5cr2UJZdu+CRR2D7dnjvPWjZ0o4pUya+eRSR/EPV7yJ/Qv/+sG2bjUM/80z49lu49FIrnX/3XTiLnIhIblBJXeQILVwIn30GPXva9rff2vKhh2xZsiRUqhSfvIlI/qSgLnKEPvrIlo8+CtWq2fonn9hz0EVE4kHV7yKHYckSGDsWpkyBMWOgaVPr2f799zBkCJx2WrxzKCL5mYK6yGHo0gWWLg23X37ZlnXrwhNPxCVLIiLpFNRFsumtt6ID+ubNUK5c/PIjIpKR2tRFssF7uPNOqF49TFNAF5G8RiV1kWxYvRrWrbOJZerXhxIl4p0jEZHMFNRFsmHsWFu2bg2dOsU3LyIiB6Lqd5GDSE2Fe+6xR6kCtGoV3/yIiByMgrpIFgYPtjbzSpXg8cfhootgwgRN+SoieZuq30Uy+P13uOMO2LfPtjt0gHffBefimy8RkUNRSV0kg9GjLaDfcINtN2+ugC4iiUEldZEMPv0UGjeGf/3L5ne/555450hEJHsU1EUi7NoF33wDN94IVava09dERBKFqt9FInzzDezeDV27xjsnIiKHT0FdJMI770CRItC5c7xzIiJy+FT9LvnewoVwySU2l/uSJfD3v2vGOBFJTCqpS773ySfw009QsyY89JA9H11EJBGppC753syZULmytadr6JqIJDKV1CXfmzLFpn9VQBeRRKegLvnW/v1wyy0wezYcf3y8cyMi8uep+l3ypT17oFYte5zq1VdrghkRSQ4K6pLvLFgAS5daQD/3XHjlFRvGJiKS6FT9LvnKXXdBw4bQvbu1oQ8bpoAuIslDJXXJN374AZ56yp66tnw51KsHFSrEO1ciIjlHQV3yjZdfhrJl4YsvoHBhSEmJd45ERHKWgrokrUWLYOxYaNAATjgBPvgALr8cSpa0/ap2F5FkE9Og7pw7CxgEFARe894/nmF/LeANoFzaMf299+NimSfJPy68EH7+2dbLlYMdO+Dii+ObJxGRWIpZRznnXEHgReBsoAnQ0znXJMNh9wPveu9bA5cBL8UqP5L/bNsWrm/ZYvO5n3xy/PIjIhJrsez93h5Y6L3/3Xu/FxgJnJfhGA+USVsvC6yKYX4knylfPnr74ouhWLH45EVEJDfEsvq9BrA8YnsF0CHDMf8EPnXO9QVKAqdndSHnXG+gN0CtWrVyPKOSnHbsCNcnTdKscSKS/GJZUs9qJm2fYbsnMNx7XxPoBrzlnMuUJ+/9UO99O+99u8qVK8cgq5JsvIcVK6yX+5tvwkknQcGC8c6ViEhsxTKorwCOjtiuSebq9euAdwG89z8AxYBKMcyT5BNbt1pJ/bHH4K9/jXduRERyRyyD+lSggXOurnOuCNYRbmyGY5YBpwE45xpjQX19DPMkSeybb+Dee2HNGrjtNktrkrFrpohIEotZm7r3PsU5dzMwARuu9rr3fo5z7iFgmvd+LNAPeNU5dztWNX+19z5jFb1ItnTubMvHHrNl9+5w1lnxy4+ISG6L6Tj1tDHn4zKkPRixPhc4KZZ5kPxhxYro7UmT4MQT9Yx0EclfNKOcJIXZs23ZubO1oZ+kW0URyYcU1CUprErrgjlsGNStG9+8iIjEix69Kklh5UpbHnVUfPMhIhJPCuqSsObNg//9z9ZXrYKKFaFo0cO4QGoqTJsWk7yJiMSDgrokrObNrYf7jz/CkCFQo8ZhXuDRR+G44xTYRSRpqE1dEkpqKnz+OZQpA/v3W9oJJ9hy9erDvNikSbZcty7H8iciEk8K6pJQRo+Giy7Ket/55x/mxXbtsmWhiH+DrVvtkW61ax9R/kRE4klBXRLKZ59Fbz/+uD1OtVkzKF78MC8WBPXt28O0jh1tfJzmQBKRBKSgLglj00bPkFcKcGr9eyj0xACaN4cGDbI4cO9e2LgRKlWyqvUaNWDTJpuJJvJ5rEFQ/+OPMC0Y8L5r1xHcJYiIxJc6yknCuPfmbQBcsvAxevTIENC3bIHdu2395pttbNuNN0LNmha0zz4bTjvNGuUDQVDfti3zmy1bFpsfQkQkhhTUJc/auxfatYPhw2178ZQmciuvAAAgAElEQVQsnvUzZgwsWGAl8HPOsbSxac8Nev11W779NkyZAj/9BB99FJ4bBPVbboHevWHw4HDfgYL6kCFQoADs22fXff/9I/75DunDD612YdOm2L2HiCQVVb9LnrVgAUyfDtdcY9O/7liSIai/9Rb87W/QooVtf/mlLWvXhrVrw+P+8Q8LxEcfDQ89BNWrQ9++0d3lX301+tpLl2adqT59bLlhA1xxha2XKQOLF0OFCpmPP/VU6NkTrr8+ez90pLvusuWBri0ikoFK6pJnzZ9vywLsp0DzJvRJHRx9wOjRtpw1K0w7+eTMT3FZuxbatrXgPmOGdZ+fMuXgb750qT3uLbJkH2n8+HB92zaYMMGq9iOr91NTYeJEqwU4EosW2XLv3iM7X0TyHQV1ybPmz4dufMwdp/9C7R3zuIK3w53eh0Ev0qRJ8NtvmdM7doQrr4Q6dWD5ckuLHBvXoUP08cHD2c891wbEv/12WF0P8Pe/Rx8/daoNmK9TJ0yL7IB3JIKB+JG980VEDkJBXfKcLVtg82ZYNXMdH/MXnpzYLvNB48ZZUK9Y0babNo2+QL9+8N13VsXep49VtxcubNX1YM9lfe+98Jw337QS8YgR8M9/WlAP9OtnVe2Rbe4bN0bn5/vvrfQf3DCA/RDZdcst8NRT4fbzz4frf/wBl1xy5BPbR9YexIL30T93fjRzZtYdLkVymYK65DkXXwwVKniWjPwBABeUWCP95S+wYwfceqv1cv/ii+jpXqtVs8BdrRq89FL46LZzz7VlyZLR16te3YL+5ZdDt27R+wYNsuUbb0Snn312uD55crgejHHfsiVzWlYmT4YXXrA29NRUu4kYMADKlrX9f/xhNyCrV4c9/LNr8GAoWDDzDcYnn8D6LDoeHsqAAdE3H8F71KoFc+ZEp7/4otVgxMvq1bG/oQHYsweOPz76Riy3fP55WKMjgoK65AU7dqS3G6ek2PfUPTzGh2QxRVzZsnD//eF2mzbw8stQtWp01XfVqlm/V5s28NxzMHRodHrp0uF6rVpZn5sxaLVsmfVxO3faMjKQbtpkve9HjQrTNm609A8/tO0qVWyc/MCB1g8g6AAYWQKcOzfz+3Xvbudk5c03bTlvXpi2Y4fduHToAD//bCMHVqzI+vyM7rvPbj4iby6++sqWP/0UpnlvQwvbt8/edXPa5s1Ws3HccbGfSGjFCgvsuT0M8rPP4Iwz4Mknc/d9k0ES1ywpqEt8eQ+lStlwtOnTSWnUhI58y//xUtbHX3wxPPywBaNbb4UuXcJ9kT3EI9MjOWfnBTcAJUpkPqZy5cxpd96ZOS0IuhkNHmw3GpGl9w8/tBuKiy6y9XPPtclx6tULJ7z544/oG4dgqtrIPgIzZ0a/17Zt9qi6fv1se/lya/8fOtSmvK1WzdKXLrU7puuuC28sFi+2UveWLdac8fe/Z+5l7709MSdjaXDChHA9qFGI7OMQ2Q/A++jRCIdyzTVw3nnZPz4rwfvNmHHwG5Y334Tbb/9z7xUEiDVrDn7cyJE2V0JO3WQEozd++SVnrgdW27V4cc5dLzf99JP9jWclssZmxAi7cf/uu9zJV27z3ifUq23btl6SyIwZ3tvXnPetW3sP/iPO8TtciTAdvB82zPvJk71PTT349a691vt//Sv77792rfe//545PfK9wfsFC2zZsmWY9u23mY870KtAgQPvK1Uq6/S77vK+YMHotNq1vb/33jCfn38e7vPe+06dwu2bb/b+uuts/V//8v6TTzK/R/Pmtnz11TBt8+bw+m++aWkvv+z9L7+Exzz2mPfLlnm/f7/3551naRde6P2+fd6fe673gwaFx774oi3Hjg2vm5Li/ZYt0Z/57t3eX3JJ9M9zpH78MbzO1KmWtn9/5r+fI32vVau8f/RR+zmCz6h9++hjJk3y/plnMr/Xhg2H/35ZGTHCrnfRRTlzPe/DPC5datv9+0f/DDll5Ur7m8v4+1i1yvu9ew//evPnW7779cu874svbN/s2bZ94YW2/Z//ZD521SrvP/zw8N77hx+8f+edw8/zYQCm+WzGyLgH6cN9KagnmX/9K/2LJLVAGMBSixa19bp1LVDktuDL7YwzbJmS4v26dd5v3RruC4Lc0Ufb8oILsg7ObdrYctw478uUyTqYV6mS+bwnngjXy5e3V7A9Z473773n/XHHhWlr13rfqFG43bWr93/7m61fcYX311yT+T0qV7blk0+Gad9+a5/BDz94X65c1vm74gpb3nCD9+3aZf1zB68OHWx5yy3h59unj6Xt3Rt+sb/3XvR5Wd3AzZnjfZEi3v/668F/f59+Gl7no4+8f/99W3/nHe8nTLDf5Z494TEZA8kPP3h///12bKdO3n/1lfczZ4b7r73Wzhs0yPsHH/TpN1yRChWy9E2bov+mpk2z7fnz/9zf9iuv2PXOO+/wz1282G74tm3zfvt2S9uyJczj66/b55/xpic11fuHHrK833ij3WhnNG2a94sWHfz9zzor/Dv23vuFCy0v4H2vXof/83z/vZ3bunXmfffcY/t697btE0+07bffjj4uJcX7ChVs35492X/vnLgJPeRbKKhLogiCTtrrjXJ9/exS7b2fMsVKbofzz5WTgjzt3u39+vXR+woX9r5GDe9XrLBj/vY3KwUGpXmw/WAl9J9/9v6//7Vzf/nFAujatZYeHH/RRZmDYWTpedQo79u2zTpo1q1ry1tvjU4vX95KzcGNRXBcVq9+/cL1l16yvJ5zjvfVqnl/8smZj69WLXq7atXobefC9SJFws/Je++XLw/33Xhj+Dl37x59jY0bw8/8pZe8nzXLaikgurbCe++/+877JUvsOlddFXWz6AcN8r5SpejPqGFDC0jBMWXL2u8j4+8/uHkJXl9+aX+TQb4jX0WLRt+IlCxp6aNHW/AOjnv/fft7ACvlb9hgAeaHH+z3FJmPg3n8cbtGt25289G1q/0dBvr0sd9dVv9DJdJqwh591JYXXBD9eTzwgJXWI/8PvLfSLnh/1FG2rFDBzgtqd3btCs/55RdLe/RRKxXv3Ws3BBs2hDejQ4eG79u7ty1LlMic3+ees99xRl9/bTdb48fbuY0bh/s2bbLfR3DT1aiRpQf/By+9ZDc0a9daenDzCfZzfv559Hu9917WJfLgnBh+VymoS563f7/3O3f66Opi8EexwvftG+/cee+HD/f+/POz3rdjh315eW9fxDt32npQ0ilQwPu//jX80juQ7dvDnz2owo18jRoVrs+YEZZuMr4iv3zBqsaDG4IDBfLIGgPwvkePcP3qq+0LsXBh7++8076cmzTJ+jrB6777vL/4Yu//9z/vzzzzwMf9+99Zp0+fHpZsI4PCjBkWgIO04Au6d2/v777bSndvvWVp3bvb7yPjtSPzfv754XpwXvBq2tSCwKZN0WlHHRWWzMH7p5/OHOyDV2STQtOmPv1mJrLp4qmnvK9Z09ZPOSVMP/tsW9aoYaXwxx+366SmZl1r0b9/5vdftizcH6S9+mr0eevWhfuOOSZcv/vucP3KK625JNgOaimGD8/6577yStv/5ZdhWlAVXqmS3dgFv7ubbw4D6MUXhzdzwY1hxqCekhL9dzJnTvh5BOnDhtmyTh1LD2rUbr89vAErUsSuFdQC3nyzLatXtxuOyJ+neHFbBjeWWdVaZPyclyw58P/6n5SjQR24GSif3QvG+qWgnhyuv97++lJr1/brqJT+j1G0qMWyhJSaal+M06aFVX5Nmhz8nOALYe1aWz78cFgKjvyC3Lw5U61G+ivyCwe8nzs3uk058hUEyK5dM7fXB6+yZcP22kmTMuf12GNtWaCAfZE99ljYBut9WLUOB36PjK/LL8+cNmGC96efnnX+g9fDD4fv17591jdHQa1B5creN2sWfW7GY2+6KXOtyXnnRZe0777bgnGjRmHzRPCaM8f7MWPsdxKUZsH7nj3D9chAnvGVse/Fnj12YzZggI8KlN5b00fG8ydOtH0pKeFN0t13R//NffZZ9DmFCoXvW6iQVU+fdFL05/Pvf9u5//d/Wee7ShX7jM46y37nnTp5X6+elfCz8/uPfAVB/cUX7fcRWbNTuHD07yrjuVWr2rkTJ4Zp55wT/fkE6xUrZk4PaguC19NPe3/ppdG1agf6H478X8lhOR3UHwEWAu8CZwEuuxePxUtBPYEtX+59gwbez57tC7LPf4CVmoKlv+qq9EJvwguqNYMq5wNp1Mj7006z9aBdt0ULO3fq1Ogvksgq3xo1rM33449tX8YvlqB9EsLOcJFfWg88EJZYMgbJYL148egqxSA96MyWsQ05488O3pcubcsXX/R+9Wor+U+alPWXfefOVpUcbL/+enTes3rdd1/Y8alhQ/tMIvc3aGDLggWtajvyZ84YXCObLyKrYm+91X6uu+6y7QcftPe65BLvTz3V0oK+DEFNwH//awEyyBtYsAzacyNfQTsuWLNCZO3K0KG2jOx/EVSFX3pp5msNGWIl8aAPQdr/lU9N9f6RR+xm8+mnfXogBmuiCfJ13HFWUxPsr1fPboa6dbNq83r1Dvy7CG7A+va1n6NgQWu7B+//8pfMx0fe/EW+ihe3qrxg+3A6pJYubZ9N8DNC2AxysPfs29eW77yT9f7I8/bu9f6336xkEtncEDSxHaq/xxHI8ep3wAFnAiPTAvwA4JjsvklOvhTUE9izz6b/A6QHcvD9eMrPGPBJ/NrPYyHo9DV+/MGPy6pqNQi8CxdaGyFEp/fpE91D3fuwVLhtW5gWBIerr7aq3Pnzw2r2N94Iv4x69bLq048/trx07WrpnTpFv0f16pY+fLiVmIKORxlFXjt4ZdVOnPGY4cOtxmLAAPtybtEi+gs5q+rmPn2879jR1suVi+49D3ajAFYqO+206H0lSkSXpiObMe64I1x/9tkwz6VKWZVuqVLe33Zb2PP/yivDPES+x2OPhet33WVty2C/h+B3+8AD4THTplmNQ7Cdsa9CEKQy/iwHe512WnQTRNeu9nMHNxNDh4Z9Knr1Cm8kgnxGNs1AdMAE7//5z+hmtJ07vR84MPqYceOib/YKFrQ+KMFNX2QNSpEi4ecEYd8BsBqFjDcIGWuwevTw/rLLotOCz7RCBbt+8L8R3EAHfWCCfjIZXyUiRuMsWxauR95wDBwY1q698srB/+8PU0za1IGWwHPAr8DLwE/Ak9k9P6deCuqJZ/ly76/ouMRvvez6LP9h6rEwls1R8ZGamvVQuezYvdt68wbrQe/kXr3sM3v55cznzJ9vbcSRgrbKyF7nP/9spdDNm8PfQcYbhE2brMQe9IIPzJljJeFDDStctMh6IV9zTdjzPyUl83H/+pcFhCAfP/wQ7stq+N2MGdY0kVUv/uDVqFH4RQ1habZ+/czBCcJ2bIiuYo9sO54wIcxXtWrhjcNTT4UBJuKGNer19tvh+quv2ucwdKi1vQel+5de8v6EE6xE7H3Yft+ihQWb00+PHtUQWQVdseKB+ymA/R6CqvXIWo+zz7bzLrzQ8vTBB2Eeg98N2M3K11+H5xUqFF06BSvdRjZ7eJ/5xu6nnyw9uJGYMcO2g9EXkSM9Dvbq2tXeP+izEuQ5WI+8qQtuTsFu/oLmiObNvT/+eFt/8EFrbgKreYpsyspYkxME9iFDwrTBg8P1W2/1/vnnbT1o288hOV39fgswHZgAXAwUTksvACzK7hvl1EtBPfH84x/e7yPrttW2TPWFCmX9nS8ZfPyxfW6zZmXv+KCn+H33Zb0/KOHG0saNB6+OjKxmXbcuet/dd0cH3VWrLD1jHwIIe9gHX9SRX7Rg1cqRnd0iv+zr1bObGO/D9Mjx/5HDzho0CPsUvPde2Hnxgw+y7j8Q9MoGGxYXKeik9sEHVksVdL7cutVu3CJvnr74Ihw62atXOL9DqVLejxyZdQB8993oNuLZsy3YgNV6REpNtd7lwXsuW2YB98cfbXvcODvvqKNs+5dfLN+9e1u+Fy4M38f78G81eAU9zIOgFzQhBB0GMx4P0bUcYD30V6yw815+OUyfOTNcT00N+1BEBtwBA8LP+9xzw9/bxx+Hbe6XXhr9N7B7d/SNXlZ9WoIbbfC+S5ew813XruHNeA7I6aD+EFD7APsaZ/eNcuqloJ54Hn/cZ/2lA37f5j/S/78lGw5nXHNQSgx6UWe0e7f3f/yRM/n6MyK/kA+2P3Is+d690cPW0iYuSg+ekV/mwZdsUKUe2Yb99NNZv9e8eVYiGzMmen9Q8wBWVR7UKKxaFTZPPP20bX/xRfQNyMqV0deqVcvSI2soDiYYThaUckeNst7gmzbZzx9MNFS0qFUDe2931GA3HPv2hdXkw4Zl7z0DO3ZYU0bG2ptAaqrdOL32mm1n7KgZOdQuUtBGP3263fxF1jpMmmTLDh2stiBScJMBNuQ08oYiGMYZOeJg0aKw5H7LLWHHxfXrw+aPO+6w8599NnroWtCh7tdfw+udeqqV8IM5GmrXDvc1a3Z4n202HE5QL5SNSefGAZuCDedcaaCJ936y937egU8TMatWZUiYMsWmMJ0yhULlSmXrj1DSFDqMT6t5c1sG07hmVLSovfIK5w6+v3Dh6PUHH7Q51wcMgDJlwn2Rj9ENpsktXx7KlbP1ChVszn2IfrpfpBo14IYbMqdHPiOgXj1o29a+ysGeN7B6NXTqZA8Iql7d0s85Bz7+ONwO7NtnyypVDvwzR2raNHwvgB49wvUZM+zJgv/+N1x9NZxyiqXXrGnLdu3sb+e+++y4zp2z956BEiXCOf6z4pz9XwcqVQrXP/0UChxgRvKLL4Ynngg/r9q1bSpjgIYNbbrhYsXsoUSRgr/tChXsNXeuHQc29e+nn0KzZvDaa/Y7qVcv/P3XrQsNGth3UKVKcNNN9sjmm2+2/bfdFv1e06bZz1ezpj3w6aKLbJrnNm3scwd77PLSpWGe4ulQUR9rO3cR2wWAGdm9a8jpl0rqCWLvXu+XLPEbv5zp76qVoUep5I7UVGuvzTh5Tl7z979bW/2BBCXgrPzvfz69qrZRI6ty9t5KymvXhtW6N95ovb8hesha5Lhu78NS3oEEbejlymXed+aZViIOqtEDu3aFTQeRgrbgjMcfqdRUa8veujVM277dSp3BrHa5JbLPxsGkpEQPh/T+0DU3gR077JVdQen89dezf87BRHbYmzzZfv8vvGD9W3IYOVz9PjOLtFnZfYOcfimo53GpqfYFfbChSCKHY8uWrINi4IcfDly9O326/c3dc0/YDhs5hWzGwLF798HnZg8CQ1bTkT72WM7Ow57IDjZZy6HcdJNVlee0YLha0HzxZ915p12vSJFD34D8SYcT1LNTl/e7c+4WrMc7wP8Bv+dsfYEkjQUL4JFH4p0LSSZlyx64CQHsWeYHElS/V6gAvXrBiSfa0/VGjYKFCzNX+R+qSSKofm/YMPO+/v0PfF5+E3yuNWoc/rmDB+dsXgKXXQZdu+Zc9XizZuH6oZqOclF2Hr16I3AisBJYAXQAescyU5LAvvkmfXV/vfoA3MpzjHl5dbxyJPlZ9erw/PPQs6e1KQePy+3Rw54Lf7iCtt3GjXMuj8nqt9/sEcl5SU62dwdBfe/enLtmDjhkSd17vw64LBfyIolqxgx46CHreFKrFhQrhn9hMJ2eu4gTeJW/fPx/nNGtMPQh7Lgjkhucg759c+56Qa/PY4/NuWsmq6xqM5JJcGN3wgnxzUcGhwzqzrliwHVAU6BYkO69vzaG+ZJEMXlyVPWnL1+emUd144e91/H9HNh33J08elrazmnTFNQlsQW95mvVim8+JP5KlIDvvoNGjeKdkyjZqX5/C6iGTRP7NVAT+COWmZIE8txzUZtu82aG/96Jm26y7XfeiWiibNvWhv2IJKohQ6B3bzjuuHjnRPKCE0+EihXjnYso2Qnq9b33DwA7vPdvAOcAzWObLUkYc+ZkSvoaGwNboIANDxVJGk2awCuvHN58ASK5KDtBPW2GBLY455oBZYE6McuRJJb169NXt8z4nV5HT+BnWvHTT/Drr3mqU6iISNLLTlAf6pwrD9wPjAXmAk/ENFeS98yfbxH600/DNO9hwwYbyrN/P89/VJd/L+8KQMuWNmmTiIjknoPWITnnCgDbvPebgW8AVabmV19+acv//tfGeq5eDaeeCikpjJ5UmVe6FWDCBDukVy+V0EVE4uGgJXXvfSpwcy7lRfKynTttWayYzVndq5fVrwOjJ1VKD+jt28Orr8YpjyIi+Vx2qt8/c87d6Zw72jlXIXjFPGeSt6xZY0vn7AEM48al71pP5fT1du1yO2MiIhLIThfOYDz6TRFpHlXF5y9LltjyxRdt2bVrevv6BirRrRs88EA4YZeIiOS+7MwoVzc3MiJ52MKF8N570Wn9+tnECzt2sLVQJX4cm/npiCIikruyM6Pc37JK996/mfPZkTzphReit4cOha5d+fnyJ2j56s2sSKmqgC4ikgdkp039uIjXycA/gXNjmCfJa1autHmOy5Sx7bS5jgf7m3B4Op1ZIo6ZExGRQHaq36OehuCcK4tNHSv5xapVcNRRVuV+113pDzJYtgyOOQbefTfO+RMRESB7JfWMdgKaViQ/2LjRHsISBPXrrrO0tLr2336DDh3CAryIiMRXdtrUP8J6u4PdBDQBVDbLD+66C15/3darV09Pnj0bBg6EpUvhWj2rT0Qkz8jOkLanI9ZTgKXe+xUxyo/kFTt3pk8uA0QF9ddeg2HDoGxZOOOMOORNRESylJ2gvgxY7b3fDeCcK+6cq+O9XxLTnEn8vP02XHFFdFpalfukSfY41datYcoUPaxKRCQvyc5X8nvAiRHb+9PS9EDhZDVyZLh+223s2LyHGUdfzsnAySdb8jnnKKCLiOQ12ekoV8h7vzfYSFsvErssSdz9/nu4XrMmnWe/RKcLKrIiotGlXLncz5aIiBxcdoL6eudc+rh059x5wIbYZUniynsL6l262PZZZ/Hbb7Y6cKAt69SB22+PR+ZERORgshPUbwTudc4tc84tA+4GbohttiRu1qyBXbvgoosswDdtytFH265nn7Xl11+TniYiInlHdiafWQQc75wrBTjv/R/Zvbhz7ixgEFAQeM17/3iG/c8Cp6RtlgCqeO9VsZvb1q2zedxvusl6wIHNKpNm0yZbNmsGDRoooIuI5FXZGac+AHjSe78lbbs80M97f/8hzisIvAicAawApjrnxnrv5wbHeO9vjzi+L9D6iH4K+XPatLGpYAFWr4amTeFE6xu5Zw+sXQv/+hc8+GAc8ygiIoeUner3s4OADuC93wx0y8Z57YGF3vvf0zrXjQTOO8jxPYF3snFdySnew803hwE98OabUKYM69bBRx9ZkkrnIiJ5X3YGJRV0zhX13u8BG6cOFM3GeTWA5RHbK4AOWR3onKsN1AW+zMZ1Jads3Ro+Hz1Sq1asXQvHHx8+Rr2uHsArIpLnZSeo/wf4wjk3LG37GuCNbJznskjzWaQBXAa8773fn+WFnOsN9AaoVatWNt5asmXr1ujtd96xhvMCBahWLXpX+/a5ly0RETky2eko96RzbhZwOhaoxwO1s3HtFUBkpW1NYNUBjr0MuOkgeRgKDAVo167dgW4M5HBt3hyuDxwIl10GZI71ACX0dFURkTwvu3OCrQFSgUuAxcCobJwzFWjgnKsLrMQC9+UZD3LONQLKAz9kMy+SU7ZsCdcj5nZfutSW9epBp06a311EJFEcMKg75xpigbgnsBH4Lzak7ZQDnRPJe5/inLsZmIANaXvdez/HOfcQMM17Pzbt0J7ASO+9SuC5LTKoly6dvhq0o48cCcdpMmARkYRxsJL6r8C3QHfv/UIA59xhzSPmvR8HjMuQ9mCG7X8ezjUlB0VWvzdqlL4aBPU6dXI1NyIi8icdbEjbhVi1+0Tn3KvOudPIuvObJKqgpL5sGdSvD1icHzbM2tArVYpj3kRE5LAdMKh770d77y8FjgW+Am4HqjrnXnbOdc2l/ElOW7gQ+vWD/fstqDsHNWoAkJICHTvC7Nlw9922S0REEschJ5/x3u/w3o/w3v8F68E+E+gf85xJbPToYT3dFyywYnnZslDA/gzGjYO5c23uGc0eJyKSeLIzo1w67/0m7/0r3vtTY5UhibGga/v69TBvngX1NP/5D1SrBhdfHKe8iYjIn3JYQV2SwLZttnz2Wfj88/SGc+9h4kTo2hUKZXego4iI5CkK6vnJ7t3h+ujRthw+HLBq9w0bwseoi4hI4lFQz0+WL4/evu46mxYW+OorS1JQFxFJXArq+cmaNdHbPXqkr371FdSqpbHpIiKJTEE9P8kY1LvayMT774f337dSuoaxiYgkLnWJyk+CoP7xx9brPa1H3Ni0CXtvP6z5AkVEJK9RUM9P1qyxQH7WWelj0/fuhV9/hf79oVWrOOdPRET+FFW/5yerV0PVqukBHSyg79sHLVrEMV8iIpIjFNTzkzVrbHaZNN7DgAG23rZtnPIkIiI5RkE9v/jmG/j9dzjqqPSkBQvgv/+1qveGDeOYNxERyRFqU88Ppk2Dzp1t/Ywz0pM/+8yW110XhzyJiEiOU0k9P/jhh3C9du301c8/t3HpxxyT+1kSEZGcp6CeH3z7bbieNrvMvn3w5Zc2VF1j00VEkoOCerLbssXGpQfSSupTptizXdLmnxERkSSgoJ7sXn8ddu4Me72nBfVPP7WRbafqIboiIklDQT2Z7d8PgwdDx4727PRPPoEqVQAL6scdB+XLxzmPIiKSYxTUk9kTT8DixXDHHVCunM0kB2zaZNXvER3hRUQkCSioJ7NXX4Uzz4Tzz49KHj8eUlPhnHPilC8REYkJBfVktX+/PT+9bduo7u379lkze5Uq0L59HPMnIiI5TpPPJKtVqyywR4xLBxg4EL74Ap5/PmoKeBERSQL6Wk9WS5faMiKo790Lz+vOTG4AABP/SURBVD1nw9j69o1TvkREJGYU1JNVENRr1UpPmjHDnunSq1ec8iQiIjGloJ6MvIe33rL1iKAezBZ70klxyJOIiMScgnoymjkTJkyAa6+FkiUBa2J/5hmL8REPahMRkSSijnLJaP16W157bXrS8OGwciUMGhSfLImISOyppJ6MNm2yZYUK6UlLl9owtltuiVOeREQk5hTUk81334WROyKoL1sW1bwuIiJJSEE92XTsGFa/p03sfv31NoucgrqISHJTUE9mRYoA8Nprtlm9ehzzIiIiMaegnuS8D9e3b49fPkREJPYU1JNNoegBDVu3huv335/LeRERkVyloJ5sihWL2lyzxpYjRkD9+nHIj4iI5BoF9WSSmgq7dkUlrV5ty2rV4pAfERHJVQrqyWTrVnsyW+XKMGoUEAZ1dZITEUl+CurJJBjKNnAg9OjBkiVwxRWWpJK6iEjyU1BPJjNn2vLoowEYNsw2+/ZNH7IuIiJJTEE9mQweDPXq2QQ0wAcfwKmnwvPPxzlfIiKSKxTUk8X27TZF7OWXQ8GCpKbCggXQpk28MyYiIrlFQT1ZTJ9uvd+PPx6wJ7Lt2QPHHBPnfImISK5RUE8Wkyfbsn17ABYtsk0FdRGR/ENBPVnMn29d3CtXBmDuXEtWUBcRyT8U1JPFunVQtSoAX34Jt95qPd71ZDYRkfxDQT1ZrFsHVaoA9qjVY46Br7/ONBW8iIgkMQX1ZJEW1HfsgN9/hyuvhObN450pERHJTQrqyWDKFFi8GKpUYf58S2rUKL5ZEhGR3Kegngw6dLClgrqISL6moJ7otm8P1ytX5scfbVWPWRURyX8U1BPd2rXpq2tWpDBoEPTsCSVKxDFPIiISFwrqiW7NGlsWLcrkGj3wHvr3j2+WREQkPhTUE10Q1CdP5teNNvFM3bpxzI+IiMSNgnqiC4J6tWosXgwVK0Lp0vHNkoiIxIeCeiLzHiZMsPVKlVi8WKV0EZH8LKZB3Tl3lnPuN+fcQudcli29zrlLnHNznXNznHNvxzI/SeeLL+Cjj6x4XrAgixfb49RFRCR/illQd84VBF4EzgaaAD2dc00yHNMAuAc4yXvfFLgtVvlJSlOm2PLjj9m/H5YsUUldRCQ/i2VJvT2w0Hv/u/d+LzASOC/DMdcDL3rvNwN479fFMD/JZ9Ysi+IdOrBqFezbp6AuIpKfxTKo1wCWR2yvSEuL1BBo6Jz7zjn3o3PurKwu5Jzr7Zyb5pybtn79+hhlNwH9/DO0aAHYLLGgoC4ikp/FMqi7LNJ8hu1CQAOgC9ATeM05Vy7TSd4P9d638963q5z2vPB8b8MG+O03aNsWUFAXEZHYBvUVwNER2zWBVVkc86H3fp/3fjHwGxbk5VAmTLDe72dZ5ca8eeCcnp8uIpKfxTKoTwUaOOfqOueKAJcBYzMcMwY4BcA5Vwmrjv89hnlKHp99BpUrQ9u2vPkmPP00nHIKFC0a74yJiEi8xCyoe+9TgJuBCcA84F3v/Rzn3EPOuXPTDpsAbHTOzQUmAn/33m+MVZ6Sxq+/wvTp0KIFe1MK0KcPnHQSjBkT74yJiEg8FYrlxb3344BxGdIejFj3wB1pL8mO3buhcWNb79iRGTNg50645RbNJCcikt9pRrlEM3duuN6wIZMm2epJJ8UnOyIikncoqCeaX34J16tXZ+JEaNAAqlWLX5ZERCRvUFBPNLNm2bJsWXZ36MxXX8GZZ8Y1RyIikkcoqCeaefOgdWvYsoVJi6qzc2f6qDYREcnnFNQTzcKFUL8+AOPHQ5Ei0KVLfLMkIiJ5g4J6IklJsae2HHMMYEH95JOhZMn4ZktERPIGBfVEsny5PbWlfn22bYM5c1RKFxGRkIJ6Ilm0yJbHHJPeCb5Vq/hlR0RE8hYF9UQybZotmzRJ7wSf9pA2ERERBfWE8tln0Lw5VKnCrFlQtiwcffShTxMRkfxBQT1RrF0LkybBGWcA9tTVxo3tyWwiIiKgoJ44BgyA1FS44QbAmtfTRraJiIgACuqJ49NPbZaZhg3Zvds6wiuoi4hIJAX1RLBtm9W3t28PwOLF4L2CuoiIRFNQTwQ//WRRvG1bABYssGQFdRERiaSgngiCoWxpQT14+mrwWHURERFQUE8M06fb2LWqVQGYPRtq1YIyZeKcLxERyVMU1BPBtGnppfTPP4cRI+DYY+OcJxERyXMU1POyPXvg+eetEb19exYuTB+mHsR4ERGRdIXinQE5iLfegltvhTZtoE8f/v0EFCwI774Lp58e78yJiEheo6CeV40cCUOG2Pr33+OLFOWDD+CUU6BHj/hmTURE8iZVv+dVjzxiHeSKF4eiRZk3D+bPhwsuiHfGREQkr1JQz4u8hyVLbL1sWQBGj7bN886LT5ZERCTvU1DPizZtgh07bL1cOQA++QSOOw5q1IhjvkREJE9TUM+Lli4N18uWZc8eG9XWqVP8siQiInmfgnpeFFS9AxQsyPTpNrrtpJPiliMREUkACup5UWRJff9+Jk2yVQV1ERE5GAX1vChDUP/uO2jQAKpUiV+WREQk71NQz4sigvrqVamMHatSuoiIHJqCel60ZEn6w1s2rd8PwKWXxjE/IiKSEBTU86KlS6F9ewC+2N+F/v3hrLPinCcREcnzFNTzmq1bYfNm6NiRpZ/MpV/qUzRsGO9MiYhIIlBQz2sefdSWxx/P7P2NSaEwjRrFN0siIpIYFNTzmpEjbYL3Tp14/30oVAgaN453pkREJBEoqOclf/wBy5dDu3YMGgTDh0O/flC+fLwzJiIiiUBBPS/59VdbNm7MK6/YMLaHH45vlkREJHEoqOcFv/xi0Tutx7tv3ITFi6FDByhcOM55ExGRhFEo3hkQoFs3WLHC1kuXZk2JeuzeDfXqxTdbIiKSWFRSj7ehQ8OADrB1K4tXWPG8bt045UlERBKSgnq83XOPLa+7DsaMYcBjLn1KWJXURUTkcKj6PZ727IFNm+Chh+CBBwAYeqvtatlSJXURETk8KqnH06pVtqxZE4B9+2xE2/33w8yZULRoHPMmIiIJR0E9noK29LSgvngxpKZCnTrxy5KIiCQuBfU4+eYbWDcjDOojR5I+Hayq3UVE5EioTT0OvIfOneHvLOdJwNeoye2n/n979x5jRXnGcfz78+AFLyiIt3ARLbReEgQkCqJGqRc0jbZqqtZ4qajVKFqDWoiprcZotLYgSkytULWK2Npq0VpZu3iJ0YJYUfCCgmJYL4DigkUFV57+MS/s2XW1uu45B2Z+n+TkzPvOu8M7Dzv7nHdmzrzN6z1SNzOz9vBIvQaWLIHuLOMyrmftrruxuHEb3nsPunXL1vfqVdv+mZnZxskj9RpYMPMDTuNOuvMBo7vcw9LLs/q6Ohg4EDbxRy0zM2sHJ/Ua2PtnwziQ+QA8MHd33pibjdL793dCNzOz9nNSr7KnnoIDl8xfXz778p3Yckc47TQ/593MzL4dJ/Uq+ugjOOggiLK6MVdvXbP+mJlZvvhkbxVNn17rHpiZWZ45qVfLxx/zyW9uZgs+qXVPzMwsp5zU22HVKjjuOFi48Bv80KRJnDprFHU9z6xYv8zMrNic1Nth2jS4//7mCdbKNTbC5MnZA2bW++gjmsbdBMBBDVOb66+5prIdNTOzQvGNcu2wYkX23tbd6qNGwV13wT7bNzBoxg3Uvdqbw2aMpVPTGmayH/szK2v48MNw1FHV67SZmeVeRUfqkkZImi9pgaQxbaw/Q9IySXPS66xK9ufbWrkSLrkEnn46K69d+8U2818NBvA8PUf9CE24kSPrRlNqWsMvuYopJz3Y3HDIkOp02szMCqNiI3VJJWAicDjQADwraVpEvNyq6b0RcUGl+tEey5fD+OvXcOkFn7B1j2254grYZ9dG3pr6DH3q/0E9I4GBvP12y5+LgHNfHsWZTITFzfWXcR2/ZTRvjyvBDqPg4IOha9eq7pOZmeVfJU+/7wcsiIg3ACRNBY4FWif1mluzJjuVLmXlcaMbOPX24XS+7g0e3+ooeqzqwbFMYlOaADiL2xjBI7y1+JAW23n+z69z5scT+fcmQxmy9hkAFnXek1HzL+PHS2DnnYEJE6q4Z2ZmViSVTOo9aDFepQHYv412x0s6GHgNuDgiFrfRpjIiePHQC+n9xJ8AENCorpweJfqS3dq+/6p6Dk1fQ1uL+NeJt/GdB25gxprvs3LRNjSqeXODyC62f/fJSZx+5XY8Pq87/6wrsVcvT9JiZmaVV8mkrjbqolX5QeCeiFgt6VzgDmB46x+SdA5wDkDv3r07rodTptD/iZuZvsWxdN6zDxGw95sP0n3FQj49+wIWj76Rnr03gdWNNN12Ox/8cCRH9N0GGo/jw1+NZ+5TjYhshF/S5wydfTMA3Q7Ygzvq2tp9MzOzylFE6zzbQRuWhgK/jogjU3ksQERc+yXtS8DyiNj2q7Y7ePDgmD17dof08YOZC5gy5EY+vXY8l44pZZUrV8K8edmNbN90dpUbboBSCS6+uEP6Z2ZmJum5iBj8ddpWcqT+LNBP0m7A28BJwE/KG0jaJSLeTcVjgFcq2J8veHppXy7kJp4cVlbZpQsccED7NnjJJR3SLzMzs/aoWFKPiCZJFwDTgRIwOSJeknQVMDsipgEXSjoGaAKWA2dUqj9tKZVg2DDYd99q/qtmZmaVUbHT75XSkaffzczMNnTf5PS7HxNrZmaWE07qZmZmOeGkbmZmlhNO6mZmZjnhpG5mZpYTTupmZmY54aRuZmaWE07qZmZmOeGkbmZmlhNO6mZmZjnhpG5mZpYTTupmZmY54aRuZmaWExvdLG2SlgFvdeAmuwPvd+D2NnaOR0uOR0uORzPHoiXHo6WOjMeuEbHD12m40SX1jiZp9ted0q4IHI+WHI+WHI9mjkVLjkdLtYqHT7+bmZnlhJO6mZlZTjipw6217sAGxvFoyfFoyfFo5li05Hi0VJN4FP6aupmZWV54pG5mZpYThU7qkkZImi9pgaQxte5PNUiaLGmppHlldd0kPSrp9fTeNdVL0oQUnxclDapdzzuepF6SHpP0iqSXJF2U6osajy0kzZL0QorHlal+N0kzUzzulbRZqt88lRek9X1q2f9KkFSS9Lykh1K5yLFYJGmupDmSZqe6Qh4rAJK2k3SfpFfT35ChG0I8CpvUJZWAicBRwF7AyZL2qm2vquJ2YESrujFAfUT0A+pTGbLY9Euvc4BbqtTHamkCRkfEnsAQ4Pz0O1DUeKwGhkfEPsAAYISkIcB1wLgUjw+Bkan9SODDiOgLjEvt8uYi4JWycpFjAXBoRAwo+6pWUY8VgBuBRyJiD2Afst+T2scjIgr5AoYC08vKY4Gxte5Xlfa9DzCvrDwf2CUt7wLMT8u/B05uq10eX8DfgcMdjwDYEvgPsD/ZAzQ6pfr1xw0wHRialjuldqp13zswBj3J/jAPBx4CVNRYpP1aBHRvVVfIYwXoArzZ+v94Q4hHYUfqQA9gcVm5IdUV0U4R8S5Aet8x1RcmRul06UBgJgWORzrdPAdYCjwKLAQaI6IpNSnf5/XxSOtXANtXt8cVNR64DFibyttT3FgABFAn6TlJ56S6oh4ruwPLgD+myzO3SdqKDSAeRU7qaqPOXwVoqRAxkrQ18Ffg5xGx8quatlGXq3hExOcRMYBslLofsGdbzdJ7buMh6QfA0oh4rry6jaa5j0WZYRExiOxU8vmSDv6KtnmPRydgEHBLRAwEVtF8qr0tVYtHkZN6A9CrrNwTeKdGfam1JZJ2AUjvS1N97mMkaVOyhH53RPwtVRc2HutERCPwONm9BttJ6pRWle/z+nik9dsCy6vb04oZBhwjaREwlewU/HiKGQsAIuKd9L4UuJ/sQ19Rj5UGoCEiZqbyfWRJvubxKHJSfxbol+5m3Qw4CZhW4z7VyjTg9LR8Otm15XX1p6U7N4cAK9adWsoDSQImAa9ExO/KVhU1HjtI2i4tdwYOI7v55zHghNSsdTzWxekEYEakC4Ybu4gYGxE9I6IP2d+GGRFxCgWMBYCkrSRts24ZOAKYR0GPlYh4D1gs6Xup6vvAy2wI8aj1DQc1vtnhaOA1suuGl9e6P1Xa53uAd4HPyD49jiS79lcPvJ7eu6W2IvuGwEJgLjC41v3v4FgcSHYK7EVgTnodXeB49AeeT/GYB1yR6ncHZgELgL8Am6f6LVJ5QVq/e633oUJxOQR4qMixSPv9Qnq9tO7vZVGPlbSPA4DZ6Xh5AOi6IcTDT5QzMzPLiSKffjczM8sVJ3UzM7OccFI3MzPLCSd1MzOznHBSNzMzywkndbOCkfR5mmlr3avDZiiU1EdlMwCaWXV1+v9NzCxnPonsUbBmljMeqZsZsH6+7OuUzak+S1LfVL+rpPo0D3S9pN6pfidJ9yubf/0FSQekTZUk/UHZnOx16el0ZlYFTupmxdO51en3E8vWrYyI/YCbyZ51Tlq+MyL6A3cDE1L9BOCJyOZfH0T2pDHI5oyeGBF7A43A8RXeHzNL/EQ5s4KR9N+I2LqN+kXA8Ih4I010815EbC/pfbK5nz9L9e9GRHdJy4CeEbG6bBt9gEcjol8q/wLYNCKurvyemZlH6mZWLr5k+cvatGV12fLn+N4ds6pxUjezcieWvT+Tlp8mm6kM4BTgqbRcD5wHIKkkqUu1OmlmbfMnaLPi6SxpTln5kYhY97W2zSXNJPvAf3KquxCYLOlSYBnw01R/EXCrpJFkI/LzyGYANLMa8TV1MwPWX1MfHBHv17ovZtY+Pv1uZmaWEx6pm5mZ5YRH6mZmZjnhpG5mZpYTTupmZmY54aRuZmaWE07qZmZmOeGkbmZmlhP/A/zzzdnJ+ckfAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot accuaracy of training process\n",
    "fig, ax = plt.subplots(1,1,figsize = (8,6))\n",
    "ax.plot(history['acc'],'b',label = 'Training')\n",
    "ax.plot(history['val_acc'],'r',label='Validation')\n",
    "ax.legend()\n",
    "ax.set(xlabel='Epoch',ylabel = 'Accuracy', title= 'Accuracy');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function to plot confusion matrix\n",
    "***\n",
    "We used the visualization function for confusion matrix from __[sklearn website](https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py)__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model\n",
    "***\n",
    "First, we evaluted the performance of the final model on test set. The accuracy was 97% before class balancing, and 75% after class balancing, which was really close to the training and validation performance at the final epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42845250/42845250 [==============================] - 1265s 30us/step\n",
      "Loss on test data = 0.0913167645157834 \n",
      "Accuracy on test data = 0.9683511357735105\n"
     ]
    }
   ],
   "source": [
    "# evaluate test data\n",
    "test_target = keras.utils.to_categorical(test_values, num_classes=2)\n",
    "test_result = model.evaluate([test_playlists, test_tracks],test_target)\n",
    "print('Loss on test data =', test_result[0],\n",
    "      '\\nAccuracy on test data =', test_result[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss on test data = 0.9937882089614868 \n",
      "Accuracy on test data = 0.7507510006427764\n"
     ]
    }
   ],
   "source": [
    "# evaluate test data (class balanced) \n",
    "is_ones_test = np.array(test_values).reshape(-1,)==1\n",
    "test_tracks_ones = test_tracks[is_ones_test]\n",
    "test_playlists_ones = test_playlists[is_ones_test]\n",
    "\n",
    "test_tracks_zeros = test_tracks[~is_ones_test]\n",
    "test_playlists_zeros = test_playlists[~is_ones_test]\n",
    "\n",
    "test_result_gen = model.evaluate_generator(valGenerator_balance(test_tracks_ones, test_playlists_ones, \n",
    "                                                                test_tracks_zeros, test_playlists_zeros, \n",
    "                                                                ohe, val_batch_size= 20000, balance_ratio = 0.5),\n",
    "                                           steps = 100, verbose = 0)\n",
    "print('Loss on test data =', test_result_gen[0],\n",
    "      '\\nAccuracy on test data =', test_result_gen[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction and confusion matrix\n",
    "***\n",
    "We then made the prediction on test data and plotted confusion matrix (both non-normalized and normalized). The final model had sensitivity 53% (predicting class 1 from real class 1) and specificity of 97% (predicting class 0 from real 0). Notice that there were a lot of false positive samples, since we have an extremely unbalanced class ratio. Those samples would potentially be the ones that we make recommendation on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted proportion of ones = 0.031673802813614114\n"
     ]
    }
   ],
   "source": [
    "# get prediction\n",
    "y_prob = model.predict([test_playlists, test_tracks])\n",
    "y_classes = y_prob.argmax(axis=-1)\n",
    "print('predicted proportion of ones =', np.mean(y_classes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[41480535  1348361]\n",
      " [    7643     8711]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAToAAAEYCAYAAADMJjphAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xu8VXP+x/HX+5zqFJVQoXuKIqMbJZWiNLlV7uU27mLcB0MiGYwZv3GNMQxjmFFhXBImkaTQVSLp5pqiC0n3zunz+2Otc2Z3Oufsddp7t/fZ5/P0WA973b7ru/bafc73u75rfb8yM5xzLpvlpDsDzjmXah7onHNZzwOdcy7reaBzzmU9D3TOuazngc45l/UqbKCTVEPSq5J+lvR8AumcKenNZOYtXSR1lzQ/U44nqZkkk1RlZ+WpopD0laTe4echkv6egmM8KumWZKdbESnVz9FJOgO4FmgN/ALMBu40s8kJpns2cAVwuJnlJ5zRDCfJgP3MbFG681IaSV8BF5rZW+F8M+BLoGqyr5Gkp4AlZjY0menuLMW/qySkd26YXrdkpJdtUlqik3QtcD9wF7AX0AR4BOifhOSbAgsqQ5CLwktNqePfbRYws5RMwG7AWuDUMrbJIwiES8PpfiAvXNcTWAL8DlgOLAPOC9cNBzYDW8JjXADcBvwrJu1mgAFVwvlzgS8ISpVfAmfGLJ8cs9/hwHTg5/D/h8esmwj8AZgSpvMmULeUcyvM/w0x+R8AHAssAH4EhsRs3wn4AFgdbjsCqBaumxSey7rwfE+PSf/3wPfAM4XLwn1ahMfoEM43AFYCPSNcu38Cvws/NwyPfVk43zJMV8WO9wywFdgQ5vGGmGvwG+Cb8Pg3R7z+21yXcJmFx784vPabw2O9Wsp5GDAYWAj8BDzM/2oxOcBQ4Ovw+jwN7Fbst3NBmO9JMcvOA74N0xsMHArMCa/biJhjtwAmAKvC8/43UCdm/VdA7/DzbYS/3fC6r42Z8oHbwnU3AosJfnufASeGyw8ANgIF4T6rw+VPAXfEHPMiYFF4/cYADaJ8V9kwpTLQ9Q0vUpUytrkd+BCoD9QD3gf+EBMo8sNtqhIEiPXA7sV/HKXMF/4wqwC7AmuAVuG6fYA2xf9BAXuEF/nscL9B4fye4fqJ4Q9tf6BGOH93KedWmP9bw/xfBKwAngVqAW3CH+e+4fYdgcPC4zYD5gFXF/9HXkL6fyIIGDWICTwxP+x5wC7AOOD/Il678wmDB3BGeM6jY9a9EpOH2ON9RfiPt9g1eDzMX1tgE3BAhOtfdF1K+g4o9o+4lPMwYCxQh6A2sQLoG3Mei4B9gZrAi8AzxfL9NMFvp0bMskeB6kCf8Pq9HOa/IUHA7BGm0RI4Orw29QiC5f0lfVcU++3GbNMuzHP7cP5Ugj9YOQR/7NYB+5TxfRV9R8BRBAG3Q5inh4BJUb6rbJhSWXXdE1hpZVctzwRuN7PlZraCoKR2dsz6LeH6LWb2OsFfq1Y7mJ+twEGSapjZMjObW8I2xwELzewZM8s3s5HA58AJMdv8w8wWmNkG4DmCH2NpthDcj9wCjALqAg+Y2S/h8ecCBwOY2Uwz+zA87lfA34AeEc5pmJltCvOzDTN7nOAv9FSC4H5znPQKvQt0l5QDHAH8GegarusRri+P4Wa2wcw+Bj4mCHgQ//onw91mttrMvgHe4X/X60zgXjP7wszWAjcBA4tVU28zs3XFvts/mNlGM3uTINCMDPP/HfAe0B7AzBaZ2fjw2qwA7iX+9SwiqR5BEL3CzD4K03zezJaa2VYzG01wbTtFTPJM4Ekzm2Vmm8Lz7RLeRy1U2neFpCclLZf0aYS83ydpdjgtkLQ6Yh5TJpWBbhVQN879jQYEVYdCX4fLitIoFijXE/z1LRczW0fwF3AwsEzSa5JaR8hPYZ4axsx/X478rDKzgvBz4T+WH2LWbyjcX9L+ksZK+l7SGoL7mnXLSBtghZltjLPN48BBwEPhDzwuM1tM8EelHdCd4C/9Ukmt2LFAV9p3Fu/6J0N5jl2F4F5yoW9LSK/49SvtetaXNErSd+H1/BfxryfhvlWBF4BnzWxUzPJzwuCxOgweB0VNk2LnGwb3VUT/bT9FUEuLy8yuMbN2ZtaOoOT4YsQ8pkwqA90HBEX7AWVss5SgUaFQk3DZjlhHUEUrtHfsSjMbZ2ZHE5RsPicIAPHyU5in73YwT+XxV4J87WdmtYEhBPfBylJmk7mkmgT3vZ4AbpO0Rzny8y5wCsF9wu/C+XOA3QlazsudnxKUdf23uZ6StrmeO3CsKMfOZ9vAlcgx/hjuf3B4Pc8i/vUs9BDBfbiiFmVJTQl+s5cT3EqpA3wak2a8vG5zvpJ2Jah1Rfptm9kkgnt7RSS1kPRfSTMlvVdK4WEQMDLKMVIpZYHOzH4muD/1sKQBknaRVFXSMZL+HG42EhgqqZ6kuuH2/9rBQ84GjpDURNJuBEVzACTtJalfeHE3EZRWCkpI43Vgf0lnSKoi6XTgQIISTarVIriPuDb8wVxabP0PBPeTyuMBYKaZXQi8RnB/CQBJt0maWMa+7xL8o5oUzk8keJxnckwptbjy5rGs6/8x0EZSO0nVCe5jJXKsko59jaTm4R+EuwjuQyarFb8WYcOApIbA9VF2knQJQan5DDPbGrNqV4JgtiLc7jyCEl2hH4BGkqqVkvSzwHnh95lHcL5Tw9skO+oxgqp1R+A6gicqYs+lKdCcoFEmrVL6eImZ3UvwDN1Qggv0LcE/npfDTe4AZhC0Wn0CzAqX7cixxgOjw7Rmsm1wyiFovV1K8FepB3BZCWmsAo4Pt11F0HJ4vJmt3JE8ldN1BDf+fyH4yz262PrbgH+G1ZbT4iUmqT9BVWNwuOhaoIOkM8P5xgStx6V5l+Afa2Ggm0xQwppU6h5BKWZomMfr4uWRMq6/mS0gaKx4i+BeVPHnLp8ADgyP9TLl9yRBS/Ekglb4jQSBPFmGE9z4/5ngj0zU6tsgggC+VNLacBpiZp8BfyGoKf0A/Iptr98Egnu+30va7vdqZm8DtwD/IWjVbwEM3JETg6LawuHA85JmE9xT3qfYZgOBF8r4w7jTpPyBYZeZwh9nrzC4OxdX2HAx1swOklQbmG9mxYNb7PYfAb81s/d3UhZLVWFfAXOJCW8We5BzO8TM1gBfSjoVQIHC1nTChqvdCUqgaeeBzjkXl6SRBEGrlaQlki4geGTlAkkfE1SbY994GgSMsgypMnrV1TmX9bxE55zLehn1srKq1DBVq5XubLhyaNu6Sbqz4Mrhm2++YtXKlVGf54skt3ZTs/ztXswpkW1YMc7MIj14nEyZFeiq1SKvVdwnJ1wGmTjlgXRnwZVDz66dk56m5W+I/O924+yHo77JkVQZFeiccxWRQJl9F8wDnXMuMQJyctOdizJ5oHPOJU5Jve2XdJld3nTOVQBh1TXKFCU1KVfSR5K2e8dcUp6k0ZIWSZparJupUnmgc84lToo2RXMVQYexJbkA+MnMWgL3EXQ8G5cHOudcYkTSSnSSGhF0gFvaqGj9Cbr6h6DPvl5S/Ajqgc45l6CIpbkgHtWVNCNmurhYYvcT9Bq0dbvDBBoSdogadqn1M0G/emXyxgjnXOKit7quNLNDSloh6XhguZnNlNSzlP1LKr3FfY/VS3TOuQQlrTGiK9AvHPN2FHCUpOId8S4h6EuxcBjK3SjW83FJPNA55xIjktIYYWY3mVkjM2tG0GnnBDM7q9hmYwiGz4Sgq/8JUXpI8aqrcy5xKXwzQtLtwAwzG0PQs/QzkgrHp43US7IHOudcgpL/CpiZTSQYpwQzuzVm+UaC8W3LxQOdcy5xOZn9ZoQHOudcYvxdV+dc9vPeS5xzlUGGv9Tvgc45lzgv0Tnnslr5XthPCw90zrnEeWOEcy67eWOEc64y8Kqrcy6rFfZHl8E80DnnEuRVV+dcZeBVV+dc1vNWV+dcVpNXXZ1zlYFXXZ1z2S7CQFxp5YHOOZeQoCf1zA50mV2xds5lPpVjipeUVF3SNEkfS5oraXgJ25wraYWk2eF0Ybx0vUTnnEuQyMlJWplpE3CUma2VVBWYLOkNM/uw2HajzezyqIl6oHPOJSxZVddwRK+14WzVcIo7ylc8XnV1ziVMUqQJqCtpRsx0cQlp5UqaDSwHxpvZ1BIOebKkOZJekNQ4Xv68ROecS0zE+2+hlWZ2SFkbmFkB0E5SHeAlSQeZ2acxm7wKjDSzTZIGA/8EjiorTS/ROecSIqKV5spbvTWz1QRDHvYttnyVmW0KZx8HOsZLywOdcy5hOTk5kaZ4JNULS3JIqgH0Bj4vts0+MbP9gHnx0vWqq3MuYUl8jm4f4J+ScgkKYs+Z2VhJtwMzzGwMcKWkfkA+8CNwbrxEPdA55xJTvnt0ZTKzOUD7EpbfGvP5JuCm8qTrgc45l7BMfzPCA51zLiGFjRGZzAOdcy5hHuicc9lNoBwPdM65LOclOudc1vNA55zLat4Y4ZyrHDI7zvkrYIVycsQHI3/Pfx4YDMDg04/g01eGseGjEexZZ9fttu94YBPWzniQE3u3K1p251X9mfnCzXz0n6H85YZTipa3P6Ax058bwqevDNtm+c2XHMvicXfw4agb+XDUjfy624EAHNKmadGyqaNvpN+RBxft8/lrw5n+3BA+HHUjk/99Q9K/h4rot5dcSMum+9DlkLZFy+4YfiuHd2pPt84dOfGEvixbunSbfWbNmM4eNavxykv/KVp2682/57COB9Op/UHc8LurCXoMgpP7HUvXzh04rOPBXHPFZRQUFBTt87e/juCQtgdyWMeDufXm3wPw46pVHN+3Fw3r7cb111yZylPPDCpX7yVp4SW60OVnHMn8L3+g1q7VAfhg9he8PulT3vz7Vdttm5Mj7riqP+M/+N8rdoe1bU6Xdvty6Gl3ATDhH9fSveN+vDdzIQ8OOZ3L7xjJ1Dlf8vKIS+nT9UDenPIZAA/96x3uf+btbdKfu3gpXc/8MwUFW9m7bm2mjr6J1yZ9SkHBVgD6XvwAq1avS8n3UBGdcfY5XDT4Mi696LyiZVdecx1Dh90OwKOPPMSf/3gH9z30CAAFBQUMu+UmevXuU7T91A/fZ+oH7zNl2kcA9O11BJPfe5fuR/TkH/8aRe3atTEzzjnjNF5+8QVOPvV0Jr37Dq+PHcOUaR+Rl5fHiuXLAcirXp2bbx3OvLlzmffZ3J31NaRVEjveTInMzt1O0rB+Hfp2a8M/Xnq/aNnH85fwzbIfS9z+soE9ePntj1nx4y9Fy8wgr1pVqlWtQl61KlSpksvyH9ewd93a1Nq1OlPnfAnAs2OncULPg0tMt9CGjVuKglpetapFJQtXsq7djmD3PfbYZlnt2rWLPq9ft26b0sTf/jqCfv1Pom79+kXLJLFx4yY2b97Mpk2b2LIln/r199omrfz8fDZv3lyU1pOP/41rfncDeXl5ANQL09t1113pcng38qpXT8HZZqgkdaWeKh7ogHuuP5mbH3iZrVvjB5QG9Xaj31FtefyF97ZZPnXOl0yasZAvx9/Jl2/exVvvz2P+lz/QoH4dvlu+umi7735YTYP6dYrmBw88gmmjb+LRYWdSp1aNouWHHtSUmS/czIznh3DlnaOKAp+Z8eojlzPl3zdw/kldEz31rPaHYUNps18znh89kiG33AbA0u++Y+yYlzn/oku22bZT5y5079GDVvs2ovW+jejV+2hatT6gaP1J/Y6hZdN9qFWrFv1PPBmARQsX8v6UyfQ6ogvH9jmSWTOm77RzyzSZXnVNaaCT1FfSfEmLJN2YymPtqGO6H8TyH3/ho3nfRtr+nutPZugDr2wXFPdtXJdWzfei5a+H0uLXN9Oz0/507dCixD9ihSW0x59/jwNPuI3OA+/m+5VruPvak4q2mf7p13Q85U66nfVnrj+/D3nVgrsMR513H4ef8ScGXP4Il5zena4dWuzYiVcCtwy/g7kLv+LU0wfx2KMPA3DTDdcy/I4/kpu77cjyXyxexILPP+ezhV/z2aJvmPTuO0yZPKlo/Ytj3mD+F0vYtGkTkyZOAKCgIJ/Vq1fz1rvv84c7/8S5Zw+qlKXvqEEuK+/Rhd2sPAwcDSwBpksaY2afpeqYO6JLu305vsev6NutDXnVqlJ71+o8ecc5nD/06RK373BgE56+O7gXtGedmvy6Wxvy87fSskk9pn3yFes2bAZg3JS5dP5Vc559bRoNY0pwDfeqw7IVPwOwPKbq++SLU3jxwcHbHW/+lz+wbsNm2rRswKzPvinad8VPaxkzYQ6HtmnGlFmLk/NlZKlTTh/E6Sf1Y8gtt/HRrJmcf86ZAPy4aiXjx71Bbm4Vvli8kEM6daZmzZoA9O7TlxnTptK12xFF6VSvXp1jjjuB18e+ypG9jqZBg4ac0H8Akuh4aCdycnJYtXIldevVS8t5plOmP16SyhJdJ2CRmX1hZpuBUUD/FB5vh9z60Bha9r2F1scN45wb/8HE6QtKDXIABxx/G62PG0br44bx0lsfcfUfR/PqxDl8+/1PdO/YktzcHKpUyaF7h/34/Mvv+X7lGtau30SnXzUD4IzjOzH23TkA7F33f/eR+h/Vls8WLwOgaYM9yc0NLk2TfXZn/2Z78fXSVexSvRo1dwnuB+1SvRq9u7Rm7uJtWxNdYPGihUWf33jtVfbbvxUAc+Yt4pPPF/PJ54vpd+LJ/OX+ERzfrz+NGjdhyuRJ5Ofns2XLFqZMnsT+rVqzdu1avl8WXJf8/HzGj3ujKK3jTujPpInvALBo4QK2bN7MnnXr7uQzzQyVtkQHNARi64NLgM7FNwoHxwgGyKhaM4XZKZ/LBvXg2t/0Zq89azP9uSH8d/JcLrv92VK3f/Gtj+hx6P7MeG4IhjH+/Xm8Pino5v7Ku0bz2PCzqJFXlTenfMa4yUGh9s6rBnBwq0aYGV8v+5Er7hgJwOHt9+W68/qwJb+ArVuNq+4azarV62jWcE9G33sRAFVycxn9xgzGvx+3c9Wsd8FvzmTypHdZtWolB7Zsyo1DhzF+3BssWrgA5eTQuHET7nvwkTLT6H/iyUya+A6HH9oOSfQ6ug/HHHcCy3/4gUGnnsimzZvYWlBA9x5HFt3fO+s353H54AvpckhbqlatxiOPP1n0j/lXrVvwyy9r2LJ5M6+9+govvvoGrQ84MOXfRbpk+ruuStU9BUmnAr82swvD+bOBTmZ2RWn75OxS3/JanZaS/LjU+P79B9KdBVcOPbt25qNZM5IalfL23s8anflgpG2/uPfYmfEGx0mFVFZdlwCxw5A1Arye5VyWESBFm+KmJVWXNE3Sx5LmShpewjZ5kkaHjZxTJTWLl24qA910YD9JzSVVAwYCY1J4POdcWiS11XUTcJSZtQXaAX0lHVZsmwuAn8ysJXAf8Kd4iaYs0JlZPnA5MI5glJ7nzKxyPCbuXCWTrBKdBdaGs1XDqfj9tf4EY7kCvAD0UpwomtJXwMzsdeD1VB7DOZdmCl6LTFpywaNpM4GWwMNmNrXYJkUNnWaWL+lnYE9gZWlp+psRzrmEiCDQRZmAupJmxEwXF0/PzArMrB3Bff1Okg4q4ZDb7VZWHv2lfudcwsrxiNzKqK2uZrZa0kSgL/BpzKrChs4lkqoAuxGM71oqL9E55xKWrMYISfUk1Qk/1wB6A58X22wM8Jvw8ynABIvznJyX6JxziYnY0BDRPsA/w/t0OQSNmGMl3Q7MMLMxwBPAM5IWEZTkBsZL1AOdcy4hwXN0yYl0ZjYHaF/C8ltjPm8ETi1Puh7onHMJUlJbXVPBA51zLmGZ3nuJBzrnXGKSe48uJTzQOecSksx7dKnigc45l7AMj3Me6JxzifMSnXMuuyX5XddU8EDnnEtIYX90mcwDnXMuQekdDyIKD3TOuYRleJzzQOecS5yX6JxzWU3eGOGcqwy8ROecy3oZHuc80DnnEuclOudcdvOX+p1z2U7+HJ1zrjLIzfBWVx8cxzmXsGQNYC2psaR3JM2TNFfSVSVs01PSz5Jmh9OtJaUVy0t0zrmEBEEsaSW6fOB3ZjZLUi1gpqTxZvZZse3eM7PjoyZaaqCTVLusHc1sTdSDOOeyW7Jqrma2DFgWfv5F0jygIVA80JVLWSW6uQSjX8eeQuG8AU0SObBzLnuUo0RXV9KMmPnHzOyxUtJsRjAi2NQSVneR9DGwFLjOzOaWddBSA52ZNY6XY+ecg3I9XrLSzA6Jn55qAv8Bri6h9jgLaGpmayUdC7wM7FdWepEaIyQNlDQk/NxIUsco+znnsp+AXCnSFCk9qSpBkPu3mb1YfL2ZrTGzteHn14GqkuqWlWbcQCdpBHAkcHa4aD3waKQcO+eyn4Ln6KJM8ZOSgCeAeWZ2bynb7B1uh6ROBHFsVVnpRml1PdzMOkj6CMDMfpRULcJ+zrlKIonPC3clKFR9Iml2uGwIYZuAmT0KnAJcKikf2AAMNDMrK9EogW6LpByCBggk7Qls3aFTcM5lHQE5SYp0ZjaZbRtAS9pmBDCiPOlGuUf3MEF9uZ6k4cBk4E/lOYhzLrsl64HhVIlbojOzpyXNBHqHi041s09Tmy3nXEWRTR1v5gJbCKqv/tqYc24byaq6pkqUVtebgZFAA6AR8Kykm1KdMedcxaGIU7pEKdGdBXQ0s/UAku4EZgJ/TGXGnHMVRzZ00/R1se2qAF+kJjvOuYomaHVNdy7KVtZL/fcR3JNbD8yVNC6c70PQ8uqcc0UPDGeyskp0hS2rc4HXYpZ/mLrsOOcqogrb6mpmT+zMjDjnKqYKXXUtJKkFcCdwIFC9cLmZ7Z/CfDnnKpBMr7pGeSbuKeAfBIH7GOA5YFQK8+Scq2Ay/fGSKIFuFzMbB2Bmi81sKEFvJs45F7wZIUWa0iXK4yWbwi5RFksaDHwH1E9ttpxzFUmG11wjBbprgJrAlQT36nYDzk9lppxzFUuFbXUtZGaF/bX/wv8633TOOSAYwDrT33Ut64Hhlwj7oCuJmZ2Ukhw55yqWNHfBFEVZJbpydWyXDO0PaMKUqTv9sM5VGqmqYWb64yVlPTD89s7MiHOu4kpW322SGgNPA3sT9GT+mJk9UGwbAQ8AxxK8onqumc0qK92o/dE551yJRFJLdPnA78xslqRawExJ480sdgDrYwiGN9wP6Az8Nfx/qTzQOecSViVJRTozWwYsCz//Imke0BCIDXT9gafDAXE+lFRH0j7hviWKnD1JeTuWdedcNgvGg4g83GFdSTNipotLT1fNgPbA1GKrGgLfxswvCZeVKsq7rp0IxlncDWgiqS1woZldEW9f51zlUI5GjpVmdki8jSTVJBiU62ozW1N8dQm7lDncYZQS3YPA8YQDxJrZx/grYM65GMkcBUxSVYIg928ze7GETZYAjWPmGwFLy0ozSqDLMbOviy0riLCfc64SKBzXNRnvuoYtqk8A88zs3lI2GwOco8BhwM9l3Z+DaI0R34bVV5OUC1wBLIiwn3OukshN3mN0XQnewPpE0uxw2RCgCYCZPQq8TvBoySKCx0vOi5dolEB3KUH1tQnwA/BWuMw551ASeyYxs8nE6dEpbG39bXnSjfKu63JgYHkSdc5VLhn+YkSkVtfHKaFFw8xKbRZ2zlUuGd55SaSq61sxn6sDJ7LtMyzOuUqssDEik0Wpuo6OnZf0DDA+ZTlyzlU4GR7ndugVsOZA02RnxDlXQQlyMzzSRblH9xP/u0eXA/wI3JjKTDnnKo4KP9xh+PBeW4JxIgC2hk27zjlXJNMDXZlvRoRB7SUzKwgnD3LOue2U46X+tIjyCtg0SR1SnhPnXIVUWHWNMqVLWWNGVDGzfKAbcJGkxcA6gvMyM/Pg55yr8GNGTAM6AAN2Ul6ccxWQgCoZfpOurEAnADNbvJPy4pyroCpyia6epGtLW1lGFyrOuUpF5JT9Hn7alRXocoGaxOlJwDlXuQWD46Q7F2UrK9AtM7Pbd1pOnHMVU5pbVKOIe4/OOefKIiA3wyNdWYGu107LhXOuQsv03ktKfWDYzH7cmRlxzlVcyRocR9KTkpZL+rSU9T0l/SxpdjjdGiV/PoC1cy4hohwDRMf3FDACeLqMbd4zs+PLk6gHOudcYsIBrJPBzCaFA1cnVRIDsXOuslLECagraUbMtCNDMnSR9LGkNyS1ibKDl+iccwkR5ep4c6WZHZLA4WYBTc1sraRjgZeB/eLt5CU651zCktUYEY+ZrTGzteHn14GqkurG289LdM65BO28vuYk7Q38YGYmqRNBYW1VvP080DnnEpLMVldJI4GeBPfylgDDgKoAZvYocApwqaR8YAMwMEqHwB7onHMJS2Kr66A460cQPH5SLh7onHMJy+z3IjzQOecSpGwY7tA55+JJ58A3UXigc84lLLPDnAc651wSZHiBzgOdcy4xweMlmR3pPNA55xLmJTrnXJZTxne86YHOOZcQr7o657Jfkl7YTyUPdM65hHmgc85lPWV41dX7o9sBC+bPp3PHdkVT/T1q89AD9wPwyIiHOLhNKzq0bcOQG2/YZr9vvvmGunVqct+9/wfAxo0b6dalE506tKVD2zb8YfiwnX4ulcmD999Hh7Zt6NjuIM45axAbN26kV8/uRdexeZMGnHryAADmf/45Pbp1Ybdd84quV6FLLjyfJg3q07HdQek4jYxT2PFmlCldvES3A/Zv1YqpM2cDUFBQQIumDek34ETenfgOY199hemz5pCXl8fy5cu32e+G666hT99jiubz8vL47/gJ1KxZky1btnBUj270+fUxdD7ssJ16PpXBd999xyMPP8hHcz6jRo0anDnoNJ4fPYq3J75XtM3A007mhBP6A7D7Hnvwl/se5NUxL2+X1tm/OZfBl13Oheefs9Pyn+kyverqJboEvTPhbZrv24KmTZvy2N/+ynU33EheXh4A9evXL9puzCsv07z5vhx44P+6uJdEzZo1AdiyZQv5W7Zk/DuDFVl+fj4bNmwI/r9+Pfs0aFC07pdffuHddyZwQv+gRFe/fn0OOfRQqlatul063bofwR577LHT8l0RKOJ/6eKBLkHPjx7FaacHXWgtWrCAKZPfo/vhnTn6qB7MmD4dgHXr1vGXe/7EzbdCVuu4AAAKyklEQVRsXzUtKCigc8d2NGlQn6N6H02nzp13av4ri4YNG3L1Ndex/75NaN54H2rX3o3eR/cpWj/m5ZfoeVQvateuncZcVkwCchRtSpeUBbp4A9Fmg82bN/Pa2DGcdMqpAOQX5PPTTz8xacqH3HX3PZx1xmmYGX8YPowrrrqmqPQWKzc3l6kzZ7PoqyXMmD6NuZ9m7deVVj/99BNjX32FeQu/5ItvlrJu/TpG/vtfReufGz2y6A+WK6+o5bnsLNE9BfRNYfppN+6/b9CufQf22msvABo2bMSAE09CEod26kROTg4rV65k+rSp3HzTDbRq2YwRD97PPXffxV8f3raT1Dp16nBEj568+eZ/03EqWW/C22/RrFlz6tWrR9WqVRkw4CQ+/OB9AFatWsWM6dM45tjj0pzLCiriwDjpvCuTskBnZpOAH1OVfiYoXgo4od8AJr4zAYCFCxawefNm6taty9sT32P+oq+Yv+grLr/yaq6/cQiX/vZyVqxYwerVqwHYsGEDE95+i1atWqflXLJd48ZNmDbtQ9avX4+Z8c6Et2nV+gAAXnzheY459niqV6+e5lxWTMlsdY1XE1TgQUmLJM2R1CFKHtN+j07SxYWD2a5YuSLd2Yls/fr1THhrPP1PPKlo2W/OO58vv/gieHzhzIH8/cl/ltm48P2yZfTtfSSHtj+Ybl0OpVfvozn2uON3RvYrnU6dO3PiSafQpVMHDmn/K7Zu3coFFwVjJz//3ChOG7httfX777+nRbNGPHj/vfzprjto0awRa9asAeCcswbRs3sXFsyfT4tmjXjqySd2+vlkmnIMYB3PU5RdEzyGYBzX/YCLgb9Gyl+EAXR2mKRmwFgzi/TAUceOh9iUqTNSlh/nKruunQ9h5swZSa1EHvCr9vaPl9+JtG2XlrvPjDeAdVlxQ9LfgIlmNjKcnw/0NLNlZaXpz9E55xJWjoaGupJiSzOPmdlj5ThUQ+DbmPkl4TIPdM651CpHQ8PKeCW6eIcqYVncamkqHy8ZCXwAtJK0RNIFqTqWcy69kniPLp4lQOOY+UbA0ng7paxEF28gWudcdhA7dRSwMcDlkkYBnYGf492fA6+6OucSlcRn5MKaYE+Ce3lLgGFAVQAzexR4HTgWWASsB86Lkq4HOudcwpJVnotXE7TgMZHfljddD3TOucRleF8UHuiccwlK73usUXigc84lpLD3kkzmgc45lzgPdM65bOdVV+dc1sv0jrE90DnnEpbhcc4DnXMuQUl8vytVPNA55xIStLpmdqTzQOecS1hmhzkPdM65ZMjwSOeBzjmXMH+8xDmX9TL8Fp0HOudc4jI8znmgc84lZid3vLlDPNA55xKT5sGpo/BA55xLWIbHufQPYO2cywJJGh1HUl9J8yUtknRjCevPlbRC0uxwujBK9rxE55xLUHI63pSUCzwMHE0w2td0SWPM7LNim442s8vLk7aX6JxzCSnseDPKFEcnYJGZfWFmm4FRQP9k5NEDnXMucdGrrnUlzYiZLo5JpSHwbcz8knBZcSdLmiPpBUmNS1i/Ha+6OucSVo6q60ozO6TUZLZnxeZfBUaa2SZJg4F/AkfFO6iX6JxzCZOiTXEsAWJLaI2ApbEbmNkqM9sUzj4OdIySPw90zrmEJanRdTqwn6TmkqoBA4Ex2xxH2idmth8wL0r+vOrqnEtMkh4YNrN8SZcD44Bc4EkzmyvpdmCGmY0BrpTUD8gHfgTOjZK2BzrnXEKS+QqYmb0OvF5s2a0xn28Cbipvuh7onHMJy/Q3IzzQOecS5u+6Oueynne86ZzLfpkd5zzQOecSl+FxzgOdcy4xkg936JyrDDI7znmgc84lLsPjnAc651ziMrzm6oHOOZeo5HS8mUoe6JxzCQleAUt3Lsrmgc45lzAPdM65rOdVV+dcdvNxXZ1z2S5ip5pp5YHOOZe4DI90HuiccwnzV8Ccc1kvs8OcBzrnXDJkeKTzQOecS1imP14is+Ljw6aPpBXA1+nORwrUBVamOxOuXLL1mjU1s3rJTFDSfwm+ryhWmlnfZB4/iowKdNlK0owyRid3GcivWXbxAaydc1nPA51zLut5oNs5Hkt3Bly5+TXLIn6PzjmX9bxE55zLeh7onHNZzwOdcy7reaBLEUmtJHWRVFVSbrrz46Lxa5WdvDEiBSSdBNwFfBdOM4CnzGxNWjPmSiVpfzNbEH7ONbOCdOfJJY+X6JJMUlXgdOACM+sFvAI0Bm6QVDutmXMlknQ8MFvSswBmVuAlu+zigS41agP7hZ9fAsYC1YAzpAzvuKuSkbQrcDlwNbBZ0r/Ag1228UCXZGa2BbgXOElSdzPbCkwGZgPd0po5tx0zWwecDzwLXAdUjw126cybSx4PdKnxHvAmcLakI8yswMyeBRoAbdObNVecmS01s7VmthK4BKhRGOwkdZDUOr05dIny/uhSwMw2Svo3YMBN4T+UTcBewLK0Zs6VycxWSboEuEfS50AucGSas+US5IEuRczsJ0mPA58RlBI2AmeZ2Q/pzZmLx8xWSpoDHAMcbWZL0p0nlxh/vGQnCG9qW3i/zmU4SbsDzwG/M7M56c6PS5wHOudKIKm6mW1Mdz5ccnigc85lPW91dc5lPQ90zrms54HOOZf1PNA557KeB7oKRFKBpNmSPpX0vKRdEkirp6Sx4ed+km4sY9s6ki7bgWPcJum6qMuLbfOUpFPKcaxmkj4tbx5d5eCBrmLZYGbtzOwgYDMwOHalAuW+pmY2xszuLmOTOkC5A51zmcIDXcX1HtAyLMnMk/QIMAtoLKmPpA8kzQpLfjUBJPWV9LmkycBJhQlJOlfSiPDzXpJekvRxOB0O3A20CEuT94TbXS9puqQ5kobHpHWzpPmS3gJaxTsJSReF6Xws6T/FSqm9Jb0naUHYlRKSciXdE3PsSxL9Il3280BXAUmqQvB60ifholbA02bWHlgHDAV6m1kHgk4/r5VUHXgcOAHoDuxdSvIPAu+aWVugAzAXuBFYHJYmr5fUh6Abqk5AO6CjpCMkdQQGAu0JAumhEU7nRTM7NDzePOCCmHXNgB7AccCj4TlcAPxsZoeG6V8kqXmE47hKzN91rVhqSJodfn4PeIKgR5SvzezDcPlhwIHAlLDru2rAB0Br4EszWwgQ9s5xcQnHOAo4B4q6Kfo5fCUqVp9w+iicr0kQ+GoBL5nZ+vAYYyKc00GS7iCoHtcExsWsey58bW6hpC/Cc+gDHBxz/2638NgLIhzLVVIe6CqWDWbWLnZBGMzWxS4CxpvZoGLbtSPoTSUZBPzRzP5W7BhX78AxngIGmNnHks4FesasK56Whce+wsxiAyKSmpXzuK4S8apr9vkQ6CqpJYCkXSTtD3wONJfUItxuUCn7vw1cGu6bG3b//gtBaa3QOOD8mHt/DSXVByYBJ0qqIakWQTU5nlrAsrAL+jOLrTtVUk6Y532B+eGxLw23R9L+YS/BzpXKS3RZxsxWhCWjkZLywsVDzWyBpIuB1yStJOj1+KASkrgKeEzSBUABcKmZfSBpSvj4xhvhfboDgA/CEuVagi6oZkkaTdCb8tcE1et4bgGmhtt/wrYBdT7wLkE/foPDfv7+TnDvbpaCg68ABkT7dlxl5S/1O+eynlddnXNZzwOdcy7reaBzzmU9D3TOuazngc45l/U80Dnnsp4HOudc1vt/o/YxwHAQ3Q0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compute and plot confusion matrix\n",
    "cm = confusion_matrix(test_values, y_classes)\n",
    "plot_confusion_matrix(cm, classes=[0,1],normalize=False,\n",
    "                      title='Confusion matrix, without normalization')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized confusion matrix\n",
      "[[0.96851749 0.03148251]\n",
      " [0.46734744 0.53265256]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAEYCAYAAAAnEYFiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcVNWZ//HPt7tBAQFRXFhFZXHBqIBoYlziFo2IZtG4xMToxERHf5MYkzExcQyJYxJnkuiok9EkmrjrGCMqitHEdQRB4wYqAoqyqIAsLqzN8/vj3sai6aWaquqquv1987ov6t576tyn6lY9fc5dTikiMDPLqppyB2BmVkpOcmaWaU5yZpZpTnJmlmlOcmaWaU5yZpZpmU9ykrpIukfSMkl3FFDPKZIeLGZs5SLpAEmvVsr2JA2SFJLq2iumUpF0saQb08cDJX0gqbbI22jX/VftKibJSTpZ0tT0Q7FA0v2SPl2Eqr8EbAdsHRHHb2olEXFTRBxRhHhKKk0Wg1sqExGPR8Sw9oqp8fYkvSHpsPbafrlExJsRsUVE1BdST+N92t77r9pVRJKTdB7wG+DfSRLSQOBq4NgiVL8DMCMi1hahrqqXhdZSsShREd8BK6GIKOsE9AQ+AI5vocxmJElwfjr9BtgsXXcwMBf4LvAusAD4erruJ8BqYE26jTOAi4Ebc+oeBARQl86fBswG3gdeB07JWf5EzvM+BUwBlqX/fypn3SPAT4En03oeBHo389oa4v9+TvzHAZ8DZgDvAT/MKT8aeApYmpa9EuicrnssfS0fpq/3yzn1/yvwNnBDw7L0OTun2xiRzvcFFgEH57Hv/gh8N33cL9322en84LReNdreDcA6YEUa4/dz9sHXgDfT7V/YwnavB64C7kvf38nAzm3YN5ek+2ZFGucjwM+A/0tjugfYGrgJWJ7WMSinjsuBt9J1zwAH5Ky7mPTzlfO66oBPpnU3TCuBNwrZpznb3DV9DUuBacDYfN+rjjCVPwA4ElhLmmSaKTMOmARsC2yTfhh/mq47OH3+OKATSXL4COjV+EPXzHzuB7Fb+sEdlq7rA+yePj6NNMkBWwFLgFPT552Uzm+d80WaBQwFuqTzP2/mtTXEf1Ea/zeAhcDNQHdg9/QLsVNafiSwX7rdQcDLwLdz6gtgcBP1/4Lkj0WXJr4k30jr6QpMBP4jz313OnBP+vjk9DXflrPu7pwYcrf3BnBYE/vg2jS+PYFVwK7NbPd6kgQ6On0fbgJubcO+eTN9X+vS9/wRYCZJwu8JTCf5A3NYWuZPwHU52/8KSRKsI/nj+jaweePPF43+gOY8v2GblxawT+fm1DUT+CHQGTiEJJkNa+296ihTJTTVtwYWRcvdyVOAcRHxbkQsJGmhnZqzfk26fk1ETCD5i7epxyzWAcMldYmIBRExrYkyRwOvRcQNEbE2Im4BXgGOySlzXUTMiIgVwO3AXi1scw1wSUSsAW4FegOXR8T76fanAZ8AiIhnImJSut03gP8BDsrjNf1bRKxK49lARFwLvEbyV74PcGEr9TV4FDgg7fIdCPwS2D9dd1C6vi1+EhErIuJ54HmSZNecP0fE0+nn5iY+fn/z2TfXR8S0dP2adNl1ETErIpYB9wOzIuKhtP47gL0bnhwRN0bE4vT5/0nyx6Mtn7crSFpmF6b1bco+bbAfsAXJH9HVEfE34F6S5N6gufeqQ6iEJLcY6N3KsaK+wJyc+TnpsvV1NEqSH5Hs+DaJiA9JugPfAhZIuk/SLnnE0xBTv5z5t9sQz+L4+OB0QxJ6J2f9iobnSxoq6V5Jb0taTnIcs3cLdQMsjIiVrZS5FhgO/FdErGqlLAARMYvkD8pewAEkX675koaxaUmuLe9Zc2Xz2TdvNVFf4/e7yfcfQNJ3Jb2cnrFfStL6a20fNDz3myQtsZMjYl26bFP2aYO+wFsNdaUK+SxmTiUkuadIumPHtVBmPskJhAYD02Wb4kOSblmD7XNXRsTEiDicpEXzCsmXv7V4GmKat4kxtcV/k8Q1JCJ6kHRT1MpzWhxqRtIWJMc5fw9cLGmrNsTzKMkZ7M4RMS+d/yrQC3huU+IpUD77ZpO3L+kAkuObJ5AcEtmS5Nhfa/ug4bk/BY5NW4wNNmWfNpgPDGh0AqW9PotVoexJLt3ZFwFXSTpOUldJnSQdJemXabFbgB9J2kZS77T8jZu4yeeAA9NrmHoCP2hYIWk7SWMldSM5JvQB0NTp/wnA0PSylzpJXwZ2I2nJlFp3kuOGH6StzLMarX8H2KmNdV4OPBMR/0RygPq3DSvS674eaeG5jwLnkBwgh+RY07kkxy+bu3RiU2LMV6n3TXeSY5wLgTpJFwE9WnuSpAHAbcBXI2JGE3Vu6j6dTPKH+/vp9+Zgkq75rfm9nOwre5IDiIhfAecBPyL58LxF8sX5S1rkZ8BU4AXgReDZdNmmbOuvJB+2F0jOjOV++GtIDiTPJzlYexBwdhN1LAbGpGUXk5whHBMRizYlpjY6n+Qg//skrczbGq2/GPijpKWSTmitMknHkpz8+Va66DxghKRT0vkBJGcim/MoyZe0Ick9QdJSfqzZZ8ClJH+0lko6v7UY26Id9s1EkmN2M0i6hStpuvvb2KEkvYb/Ta8F/UBSw/HeTd6nEbEaGAscRXJW+mqSRPrKJry2TFKEB8205kl6Djg0TR5mVcdJzswyrSK6q2ZmpeIkZ2aZ5iRnZplWUTdrq65LqHP3codhbbD3rgPLHYK1wZw5b7Bo0aJ8r8HLS22PHSLWbnQjTZNixcKJEXFkMbffmspKcp27s9mwVq96sAry5OQryx2CtcH++44qep2xdkXe39uVz12V750cRVNRSc7MqpGggkescpIzs8IIqCnq4MdF5SRnZoVTUQ/zFZWTnJkVyN1VM8s6t+TMLLOEW3JmlmVyS87MMs5nV80su3ziwcyyTLi7amYZ55acmWWXu6tmlnU17q6aWVb53lUzyzZ3V80s63x21cwyzS05M8ss+bYuM8s6n3gws+zyiQczyzp3V80sszyenJllm7urZpZ17q6aWab57KqZZZbcXTWzrHN31cyyTE5yZpZVyejnTnJmllVKpwrlJGdmBRI1NT7xYGYZ5u6qmWWak5yZZVeFH5Or3I60mVUFIaT8plbrko6U9KqkmZIuaGL9QEl/l/QPSS9I+lxrdbolZ2YFK8aJB0m1wFXA4cBcYIqk8RExPafYj4DbI+K/Je0GTAAGtRhbwZGZWYdXpJbcaGBmRMyOiNXArcCxjcoE0CN93BOY31qlbsmZWWGKd0yuH/BWzvxcYN9GZS4GHpR0LtANOKy1St2SM7OCtaEl11vS1JzpzNxqmqg6Gs2fBFwfEf2BzwE3SC2PDuCWnJkVpOHEQ54WRcSoZtbNBQbkzPdn4+7oGcCRABHxlKTNgd7Au81t0C05MytYkY7JTQGGSNpRUmfgRGB8ozJvAoem29wV2BxY2FKlbsmZWWEEqin8oFxErJV0DjARqAX+EBHTJI0DpkbEeOC7wLWSvkPSlT0tIhp3aTfgJGdmBSvWHQ8RMYHkspDcZRflPJ4O7N+WOp3kzKxgvq3LzDKrjSce2p2TnJkVrnJznM+uFurwT+3K83f9mJfu/jfO//rhG60f2KcXE357Lk/f9gMmXvsv9Nt2SwAOHDWESbdesH5aMunXHHPwJ9o7/A7pwYkP8Indh7H7LoO57Jc/32j9qlWr+MrJX2b3XQZzwKf2Zc4bbwAw5emn2XfkXuw7ci9Gj9iTu/9yVztHXqFUtLOrJeGWXAFqasRvLjiBo8+6knnvLOWJm77HvY++yCuz315f5tLvfJ6b7nuam+6ZzEH7DGXcuWM548d/4rGpr7HfickXrFePrrw0/t94aNLL5XopHUZ9fT3f/n//zH33/5V+/fvz6f32YcyYsey6227ry1z/h9/Ta8teTHtlJrffdisX/vBfufHm29h9+HCenDyVuro6FixYwL4j9+ToMcdQV+evUSUPmlm5kVWBfYYPYtZbi3hj3mLWrK3njonPMqZRa2yXnfrwyORXAXh0ygzGHLzHRvV8/rC9efDJ6axYuaZd4u7Ipjz9NDvvPJgdd9qJzp07c/yXT+Tee+7eoMy999zNKad+DYAvfPFLPPK3h4kIunbtuj6hrVq5sqKPQ7U75TmVgZNcAfpu25O57yxZPz/vnSX026bnBmVenDGP4w7dC4BjD9mTHlt0Yaue3TYoc/xnR3D7A8+UPmBj/vx59O//8UX1/fr1Z968eRuXGZCUqauro0fPnixevBiApydPZsSeuzNq7z244qrfuhWXquTuakmTXGtjQ1U7NfGnqfFViT/49V0cMHIwT93yrxwwcjDz3lnC2vr69eu3792D3Yf05a9PTcdKr6nrRht/+VoqM3rffXn2+Wk88dQULvvFpaxcubI0gVaRfBNc5o7J5Tk2VFWb9+5S+m/Xa/18v+16MX/hsg3KLFi4jBPP/x0A3bp05rhD92L5Bx9/Mb54+AjG/+0F1q5d1z5Bd3D9+vVn7tyPB7qYN28uffv23bjMW2/Rv39/1q5dy/Jly9hqq602KLPLrrvSrVs3pr30EiNHNXcrZsdRyV33Urbk8hkbqqpNnTaHwQO3YYe+W9OprpbjPzuC+x55YYMyW2/Zbf0H4Hunf5Y/3j1pg/UnHDmS2x+Y2m4xd3Sj9tmHmTNf443XX2f16tXccdutHD1m7AZljh4zlptu+CMAf77zfznoM4cgiTdef521a9cCMGfOHGbMeJUdBg1q75dQkTpkS478xoYiHWolGW6l0xYlDKf46uvX8Z1f3M49V/8ztTXij3dP4uXZb/Pjs47m2elvct+jL3LgqCGMO3csEfDEszP59qW3r3/+wD5b0X/7Xjz+zMwyvoqOpa6ujl9ffiXHHP1Z6uvr+dppp7Pb7rsz7uKLGDFyFGOOGctpp5/B6aedyu67DKZXr6244aZbAfi/J5/gPy77OZ3qOlFTU8Pl/3U1vXv3LvMrqgzFuHe1VNTKva2bXrF0PPDZiPindP5UYHREnNvcc2q6bhubDTuhJPFYaSyZcmW5Q7A22H/fUTzzzNSiZqTNth8S/U+5Iq+ys3/1uWdaGGqpJErZkstnbCgzq3ICKviQXEmPyeUzNpSZVb0Oena1ubGhSrU9MyufSm7JlfRKxqbGhjKzjFFyi2Ol8uXaZlYQ4SRnZhnXYburZtYxVPIdD05yZlYYuSVnZhmWXCdXuVnOSc7MCiSfeDCzbHNLzsyyy8fkzCzLfEzOzDKvgnOck5yZFc4tOTPLLt+7amZZVunjyTnJmVmByjdWXD6c5MysYBWc45zkzKxwbsmZWWbJJx7MLOvckjOzTKvgHOckZ2aFc0vOzLLLN+ibWZapwq+TK+WPS5tZB1Fbo7ym1kg6UtKrkmZKuqCZMidImi5pmqSbW6vTLTkzK1gxGnKSaoGrgMOBucAUSeMjYnpOmSHAD4D9I2KJpG1bq9ctOTMriJSceMhnasVoYGZEzI6I1cCtwLGNynwDuCoilgBExLutVdpsS05Sj5aeGBHLW6vczDqGNlwL3FvS1Jz5ayLimvRxP+CtnHVzgX0bPX8ogKQngVrg4oh4oKUNttRdnQYEySADDRrmAxjYUsVm1nG04cTDoogY1Vw1TSyLRvN1wBDgYKA/8Lik4RGxtLkNNpvkImJAy7GamSWKdHJ1LpCbd/oD85soMyki1gCvS3qVJOlNaa7SvI7JSTpR0g/Tx/0ljWxL5GaWXQJqpbymVkwBhkjaUVJn4ERgfKMyfwE+AyCpN0n3dXZLlbaa5CRdmVZ6arroI+C3rT3PzDqIPE86tNaljYi1wDnAROBl4PaImCZpnKSxabGJwGJJ04G/A9+LiMUt1ZvPJSSfiogRkv6RBvJemmXNzIDi3fEQEROACY2WXZTzOIDz0ikv+SS5NZJqSA8AStoaWJfvBsws2wTUVPkdD1cBdwLbSPoJ8ATwi5JGZWZVRcpvKodWW3IR8SdJzwCHpYuOj4iXShuWmVWLrAyaWQusIemy+i4JM9tAVXdXJV0I3AL0Jblu5WZJPyh1YGZWPZTnVA75tOS+AoyMiI8AJF0CPANcWsrAzKx6VPJQS/kkuTmNytXRysV3ZtZxJGdXyx1F81q6Qf/XJMfgPgKmSZqYzh9BcobVzGz9xcCVqqWWXMMZ1GnAfTnLJ5UuHDOrRlV5djUift+egZhZdara7moDSTsDlwC7AZs3LI+IoSWMy8yqSCV3V/O55u164DqShH0UcDvJiJ1mZkBlX0KST5LrGhETASJiVkT8iHSoEzMzKbkYOJ+pHPK5hGSVkrboLEnfAuYBrf54hJl1HBXcW80ryX0H2AL4fyTH5noCp5cyKDOrLlV5drVBRExOH77PxwNnmpkByY9LV/K9qy1dDHwXG/+IxHoR8YWSRGRm1aWMwyjlo6WW3JXtFkVqu37bctol57b3Zq0AZ93xQrlDsDaYs2RFSeqt5EtIWroY+OH2DMTMqlclj7+W73hyZmZNElXakjMzy1ddBTfl8k5ykjaLiFWlDMbMqk/y+w2V25LLZ2Tg0ZJeBF5L5/eU9F8lj8zMqkaN8pvKElseZa4AxgCLASLieXxbl5nlqOpf6wJqImJOo+ZofYniMbMqU+m/u5pPkntL0mggJNUC5wIzShuWmVWT2srNcXklubNIuqwDgXeAh9JlZmaojCOM5COfe1ffBU5sh1jMrEpVcI7La2Tga2niHtaIOLMkEZlZ1angQUjy6q4+lPN4c+DzwFulCcfMqk3Vn3iIiNty5yXdAPy1ZBGZWdWp4By3Sbd17QjsUOxAzKxKCWorOMvlc0xuCR8fk6sB3gMuKGVQZlY9qvonCdPfdtiT5HcdANZFRLMDaZpZx1TJSa7F27rShHZXRNSnkxOcmW1EUl5TOeRz7+rTkkaUPBIzq0oN3dVKvUG/pd94qIuItcCngW9ImgV8SPKaIiKc+Mysqn/j4WlgBHBcO8ViZlVIQF2RmmmSjgQuB2qB30XEz5sp9yXgDmCfiJjaUp0tJTkBRMSsTQvXzDqKYrTk0gFArgIOB+YCUySNj4jpjcp1J/kd6Mkb17KxlpLcNpLOa25lRPwqnw2YWdaJGorSkhsNzIyI2QCSbgWOBaY3KvdT4JfA+flU2tKJh1pgC6B7M5OZWfpDNkUZNLMfG94yOjdd9vG2pL2BARFxb77xtdSSWxAR4/KtyMw6qLadOe0tKfcY2jURcc3HNW1k/WVrkmqAXwOntSW8Vo/JmZm1REBt/lluUUSMambdXGBAznx/YH7OfHdgOPBIes3d9sB4SWNbOvnQUpI7NK+QzazDK9IoJFOAIZJ2JLnL6kTg5IaVEbEM6N0wL+kR4PzWzq42e0wuIt4rMGAz6yCKcUwuvS73HGAi8DJwe0RMkzRO0thNjc0/Lm1mBRH53TqVj4iYAExotOyiZsoenE+dTnJmVpgK/3FpJzkzK1jlpjgnOTMrkKjyQTPNzFpTwTnOSc7MClW+seLy4SRnZgUp5tnVUnCSM7OCuSVnZplWuSnOSc7MCqRq/0lCM7PWuLtqZplWuSnOSc7MiqCCG3JOcmZWmOQSksrNck5yZlYwt+TMLMNUrEEzS8JJzswK4u6qmWVbfr/EVTZOcmZWMCc5M8s0ubuaXbOnPs5D11zCunXr2POIL/HJE85sstwrTzzAXy79Nl/7zR30GbIH0/5+D5Pv/P369e++8Spfv/zPbLfzru0Veoc1vM8WnDyiHzWCx2a9x4SXF26wfv8de/HlvfqwZMUaAB6esZjHZr/H1l07cc4BO1AjUVsjHpqxiEdm+veePGhmhq2rr+fB/x7HiT/7A917b8f13zmeIfsdQu+Bgzcot+qjD5g6/kb6Dttz/bLdP3MMu3/mGCBJcHeO+2cnuHYgwakj+/Eff3+d91as4aIjBvPcvOXMX75qg3JPv7mUG5+Zv8GypSvXcslfZ7F2XbBZXQ0/O2ooz81bztIVa9vzJVSkCs5xFT0MVMVbMOMFevUdyJZ9BlDbqTO7Hfg5Xpv08EblHr/xCvb70hnUdu7cZD0vP3ofux10dKnDNWCnrbry7gerWfjhaurXBU+/uZS9+/fI67n164K165IfdK+rUUV/sdub8vxXDk5yBXh/8Tt0791n/Xz33tvz/uJ3Nijz9qzpLF+4gMGjP9NsPS8/dr+TXDvp1bUT7320Zv38ex+toVeXThuVGzmgJ+OOGsLZ+w9kq64fr9+qayfGHTWE/zx2Vya8vNCtONJLSJTfVA4l665K+gMwBng3IoaXajtlFU0t/HhPxrp1PHztpRz9nUubrWL+K8/TabPN2WbQ0OLHZ3lpvBufm7ecyXOWsnZdcPDgrfin/Qbwy7/NBpKkeNH9r7FllzrOPWAQU99axvKVHT3Rla+Vlo9StuSuB44sYf1l1733dry/aMH6+fcXvU33rbddP79qxYcsmvMaN1/wVa7++iHMf+V57hx3Ngtee3F9memPTWBXt+LazZKP1mzUMlu6Ys0GZT5cXb++W/rorPfYoVeXjepZumIt85atZOg23UobcDVIr5PLZyqHkiW5iHgMyPSppz5D9+C9eXNY+vZc6tesZvpjExi87yHr12/erTv/csskzr7ub5x93d/ou8uefPGiq+kzZA8gaem9+sQD7Hagk1x7ef29j9i2e2d6d+tEbY0YPXBL/jF3+QZlem7+cQdn7349WLB8JQC9unSiU23yTe3aqZYhvbvxdqMTFh1Rw9nVfKZyKPvZVUlnAmcC9Nimb5mjaZua2jqOOOvH3PbjM4h16/jE4V9kmx2G8NgNV9BnyHCG7HdIi89/86UpdO+9PVv2GdBOEdu6gJumzue7B+9EjeDx2UuYv3wVx+2xHW+8t4Ln5i3n8GG92atfD+rXBR+urud3k+YC0KfnZpy49yAiklbJA68sZO6ylWV+RZWhcjuroIgmDywVp3JpEHBvvsfk+gwZHqddfmfJ4rHie3uZWzLV5J4LT2LR7GlFzUm77rF3XPeXv+dV9pODez0TEaOKuf3WlL0lZ2bVr5JPPDjJmVnBKvmawZKdeJB0C/AUMEzSXElnlGpbZlZeynMqh5K15CLipFLVbWaVQ/jXuswsyzyenJllXQXnOCc5MyuCCs5yTnJmVqDKvnfVSc7MCtIwCkmlcpIzs8JVcJLzeHJmVrBiDZop6UhJr0qaKemCJtafJ2m6pBckPSxph9bqdJIzs4IVY6glSbXAVcBRwG7ASZJ2a1TsH8CoiPgE8L/AL1uLzUnOzApWpDseRgMzI2J2RKwGbgWOzS0QEX+PiI/S2UlA/9YqdZIzs8Lkm+GSLNdb0tScKffn7foBb+XMz02XNecM4P7WwvOJBzMrSHJ2Ne8zD4taGGqpqUqa/pEB6SvAKOCg1jboJGdmBSvSydW5QO4Isv2B+Y0LSToMuBA4KCJaHdDQ3VUzK1xxDspNAYZI2lFSZ+BEYPwGm5H2Bv4HGBsR7+YTmltyZlawYtzxEBFrJZ0DTARqgT9ExDRJ44CpETEeuAzYArgjHfnkzYgY21K9TnJmVrBijUISEROACY2WXZTz+LC21ukkZ2YFq+AbHpzkzKwwHjTTzLLNg2aaWdZVcI5zkjOzIqjgLOckZ2YF8qCZZpZhHjTTzLLPSc7MsszdVTPLNF9CYmaZVsE5zknOzArki4HNLMt8W5eZZV7lpjgnOTMrggpuyDnJmVnhfAmJmWVb5eY4JzkzK1wF5zgnOTMrjNSmnyRsd05yZla4ys1xTnJmVrgKznFOcmZWuArurTrJmVmhPGimmWVYcltXuaNonpOcmRXMSc7MMs3dVTPLLg+1ZGZZJnwJiZllXQVnOSc5MyuYb+sys0yr3BTnJGdmxVDBWc5JzswKVsmXkCgiyh3DepIWAnPKHUcJ9AYWlTsIa5Os7rMdImKbYlYo6QGS9ysfiyLiyGJuvzUVleSyStLUiBhV7jgsf95n2VFT7gDMzErJSc7MMs1Jrn1cU+4ArM28zzLCx+TMLNPckjOzTHOSM7NMc5Izs0xzkisRScMkfVJSJ0m15Y7H8uN9lT0+8VACkr4A/DswL52mAtdHxPKyBmbNkjQ0Imakj2sjor7cMVlxuCVXZJI6AV8GzoiIQ4G7gQHA9yX1KGtw1iRJY4DnJN0MEBH1btFlh5NcafQAhqSP7wLuBToDJ0sVPPBWBySpG3AO8G1gtaQbwYkuS5zkiiwi1gC/Ar4g6YCIWAc8ATwHfLqswdlGIuJD4HTgZuB8YPPcRFfO2Kw4nORK43HgQeBUSQdGRH1E3Az0BfYsb2jWWETMj4gPImIR8E2gS0OikzRC0i7ljdAK4fHkSiAiVkq6CQjgB+mXZBWwHbCgrMFZiyJisaRvApdJegWoBT5T5rCsAE5yJRIRSyRdC0wnaR2sBL4SEe+UNzJrTUQskvQCcBRweETMLXdMtul8CUk7SA9gR3p8ziqcpF7A7cB3I+KFcsdjhXGSM2uCpM0jYmW547DCOcmZWab57KqZZZqTnJllmpOcmWWak5yZZZqTXBWRVC/pOUkvSbpDUtcC6jpY0r3p47GSLmih7JaSzt6EbVws6fx8lzcqc72kL7VhW4MkvdTWGC37nOSqy4qI2CsihgOrgW/lrlSizfs0IsZHxM9bKLIl0OYkZ1YJnOSq1+PA4LQF87Kkq4FngQGSjpD0lKRn0xbfFgCSjpT0iqQngC80VCTpNElXpo+3k3SXpOfT6VPAz4Gd01bkZWm570maIukFST/JqetCSa9KeggY1tqLkPSNtJ7nJd3ZqHV6mKTHJc1Ih0NCUq2ky3K2/c1C30jLNie5KiSpjuSWoxfTRcOAP0XE3sCHwI+AwyJiBMmAnedJ2hy4FjgGOADYvpnqrwAejYg9gRHANOACYFbaivyepCNIhpIaDewFjJR0oKSRwInA3iRJdJ88Xs6fI2KfdHsvA2fkrBsEHAQcDfw2fQ1nAMsiYp+0/m9I2jGP7VgH5XtXq0sXSc+ljx8Hfk8yssmciJiULt8P2A14Mh26rjPwFLAL8HpEvAaQjrJxZhPbOAT4KqwfamhZeptTriPS6R/p/BYkSa8T4qCYAAABeElEQVQ7cFdEfJRuY3wer2m4pJ+RdIm3ACbmrLs9vRXuNUmz09dwBPCJnON1PdNtz8hjW9YBOclVlxURsVfugjSRfZi7CPhrRJzUqNxeJKOiFIOASyPifxpt49ubsI3rgeMi4nlJpwEH56xrXFek2z43InKTIZIGtXG71kG4u5o9k4D9JQ0GkNRV0lDgFWBHSTun5U5q5vkPA2elz61Nh2x/n6SV1mAicHrOsb5+krYFHgM+L6mLpO4kXePWdAcWpMPGn9Jo3fGSatKYdwJeTbd9VloeSUPT0X3NmuSWXMZExMK0RXSLpM3SxT+KiBmSzgTuk7SIZLTi4U1U8S/ANZLOAOqBsyLiKUlPppdo3J8el9sVeCptSX5AMozUs5JuIxkFeQ5Jl7o1PwYmp+VfZMNk+irwKMk4fN9Kx+n7HcmxumeVbHwhcFx+7451RL5B38wyzd1VM8s0JzkzyzQnOTPLNCc5M8s0JzkzyzQnOTPLNCc5M8u0/w9kF1Qkk2LnJgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot confusion matrix (normalized)\n",
    "plot_confusion_matrix(cm, classes=[0,1],normalize=True,\n",
    "                      title='Confusion matrix, with normalization')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also quantify the model performance using area under ROC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auROC = 0.7505850277437294\n"
     ]
    }
   ],
   "source": [
    "# calculate auROC\n",
    "auroc_test = roc_auc_score(test_values, y_classes)\n",
    "print('auROC =', auroc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation and prediction with the model of lowest validation loss\n",
    "***\n",
    "Since the final model might be a little bit overfitting according to the training histry, we went back and found the model with lowest validation loss (around epoch 220) that we saved as checkpoints, and evaluated the performance with it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load best model\n",
    "best_model_file = \"models_10000_rand/model.200-0.50-0.76.h5\"\n",
    "best_model = load_model(best_model_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42845250/42845250 [==============================] - 1337s 31us/step\n",
      "Loss on test data = 0.3521598268120076 \n",
      "Accuracy on test data = 0.8817210542592236\n"
     ]
    }
   ],
   "source": [
    "# evaluate test data with the best model\n",
    "test_target_best = keras.utils.to_categorical(test_values, num_classes=2)\n",
    "test_result_best = best_model.evaluate([test_playlists, test_tracks],test_target)\n",
    "print('Loss on test data =', test_result_best[0],\n",
    "      '\\nAccuracy on test data =', test_result_best[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss on test data (class balanced) = 0.4995765554904938 \n",
      "Accuracy on test data (class balanced) = 0.764355999827385\n"
     ]
    }
   ],
   "source": [
    "# evaluate test data (class balanced) with the best model \n",
    "test_result_best_gen = best_model.evaluate_generator(valGenerator_balance(test_tracks_ones, test_playlists_ones, \n",
    "                                                                          test_tracks_zeros, test_playlists_zeros, \n",
    "                                                                          ohe, val_batch_size= 20000, balance_ratio = 0.5), \n",
    "                                                 steps = 100, verbose = 0)\n",
    "print('Loss on test data (class balanced) =', test_result_best_gen[0],\n",
    "      '\\nAccuracy on test data (class balanced) =', test_result_best_gen[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the final model and the model with lowest validation loss performed similarly on class balanced test data (75% vs. 76%), but the final model out-performed on the test data with real class ratio (97% vs. 88%).\n",
    "***\n",
    "We can look at the confusion matrix, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted proportion of ones = 0.11839219049953029\n"
     ]
    }
   ],
   "source": [
    "# get prediction on test data with the best model \n",
    "y_prob_best = best_model.predict([test_playlists, test_tracks])\n",
    "y_classes_best = y_prob_best.argmax(axis=-1)\n",
    "print('predicted proportion of ones =', np.mean(y_classes_best))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[37766956  5061940]\n",
      " [    5751    10603]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAToAAAEYCAYAAADMJjphAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XecFPX9x/HX++5oAgpSpBepCipdBQtBQaJYoxSxoCiiiSVG/SF204wajS1RiQUbIBYEYgMbYBQFBBRRQNBIkSYgTfCOz++PmTuX8253j93l9vY+Tx7zYKfsd76zs/fZb5n5jswM55zLZFmlnQHnnEs1D3TOuYzngc45l/E80DnnMp4HOudcxvNA55zLeGU20EmqImmSpE2SxieQzmBJbyYzb6VF0tGSvkyX/UlqJskk5eytPJUVkr6WdHz4eqSkf6dgHw9LuinZ6ZZFSvV1dJLOBq4G2gKbgbnAn81sRoLpngtcDnQ3s9yEM5rmJBnQysyWlHZeiiPpa+AiM5sazjcDlgEVkn2OJD0JLDezG5OZ7t5S+LNKQnpDwvSOSkZ6mSalJTpJVwP/AP4CHAA0Af4JnJqE5JsCi8pDkIuHl5pSxz/bDGBmKZmA/YAtwFlRtqlEEAhXhtM/gErhup7AcuAPwBpgFXBBuO42YCfwU7iPocCtwDMRaTcDDMgJ54cASwlKlcuAwRHLZ0S8rzvwMbAp/L97xLp3gT8C74fpvAnULubY8vN/XUT+TwNOBBYB3wMjI7bvBnwAbAy3fRCoGK6bFh7L1vB4B0Sk/3/Ad8DT+cvC97QI99EpnG8ArAN6xnHuRgN/CF83DPd9WTjfMkxXhfb3NLAL2B7m8bqIc3A+8L9w/zfEef53Oy/hMgv3Pyw89zvDfU0q5jgMGA4sBjYAD/FzLSYLuBH4Jjw/TwH7FfruDA3zPS1i2QXAt2F6w4GuwPzwvD0Yse8WwNvA+vC4nwVqRKz/Gjg+fH0r4Xc3PO9bIqZc4NZw3QjgK4Lv3ufA6eHyg4AfgbzwPRvD5U8Cf4rY58XAkvD8TQQaxPNZZcKUykDXNzxJOVG2uR34EKgL1AH+C/wxIlDkhttUIAgQ24Cahb8cxcznfzFzgKrAD0CbcF19oF3hPyhg//Aknxu+b1A4Xytc/274RWsNVAnn7yjm2PLzf3OY/4uBtcBzQHWgXfjlPDDcvjNwRLjfZsBC4KrCf+RFpP83goBRhYjAE/HFXgjsA7wB3B3nubuQMHgAZ4fHPC5i3SsReYjc39eEf7yFzsGoMH+HATuAg+I4/wXnpajPgEJ/xMUchwGTgRoEtYm1QN+I41gCHAhUA14Cni6U76cIvjtVIpY9DFQG+oTnb0KY/4YEAfPYMI2WQO/w3NQhCJb/KOqzotB3N2KbDmGeO4bzZxH8YGUR/NhtBepH+bwKPiOgF0HA7RTm6QFgWjyfVSZMqay61gLWWfSq5WDgdjNbY2ZrCUpq50as/ylc/5OZvUrwa9VmD/OzC2gvqYqZrTKzBUVscxKw2MyeNrNcMxsDfAGcHLHNE2a2yMy2A88TfBmL8xNBe+RPwFigNnCfmW0O978AOBTAzGab2Yfhfr8GHgGOjeOYbjGzHWF+dmNmowh+oWcSBPcbYqSX7z3gaElZwDHAnUCPcN2x4fqSuM3MtpvZPGAeQcCD2Oc/Ge4ws41m9j/gHX4+X4OBe8xsqZltAa4HBhaqpt5qZlsLfbZ/NLMfzexNgkAzJsz/CmA60BHAzJaY2ZTw3KwF7iH2+SwgqQ5BEL3czD4J0xxvZivNbJeZjSM4t93iTHIw8LiZzTGzHeHxHhm2o+Yr7rNC0uOS1kj6LI683ytpbjgtkrQxzjymTCoD3Xqgdoz2jQYEVYd834TLCtIoFCi3Efz6loiZbSX4BRwOrJL0H0lt48hPfp4aRsx/V4L8rDezvPB1/h/L6oj12/PfL6m1pMmSvpP0A0G7Zu0oaQOsNbMfY2wzCmgPPBB+wWMys68IflQ6AEcT/NKvlNSGPQt0xX1msc5/MpRk3zkEbcn5vi0ivcLnr7jzWVfSWEkrwvP5DLHPJ+F7KwAvAM+Z2diI5eeFwWNjGDzax5smhY43DO7rif+7/SRBLS0mM/u9mXUwsw4EJceX4sxjyqQy0H1AULQ/Lco2Kwk6FfI1CZftia0EVbR89SJXmtkbZtaboGTzBUEAiJWf/Dyt2MM8lcS/CPLVysz2BUYStINFE7XLXFI1gnavx4BbJe1fgvy8B5xJ0E64Ipw/D6hJ0HNe4vwUIdr53+18StrtfO7BvuLZdy67B65E9vHX8P2HhufzHGKfz3wPELTDFfQoS2pK8J39HUFTSg3gs4g0Y+V1t+OVVJWg1hXXd9vMphG07RWQ1ELS65JmS5peTOFhEDAmnn2kUsoCnZltImifekjSaZL2kVRB0q8l3RluNga4UVIdSbXD7Z/Zw13OBY6R1ETSfgRFcwAkHSDplPDk7iAoreQVkcarQGtJZ0vKkTQAOJigRJNq1QnaEbeEX5hLC61fTdCeVBL3AbPN7CLgPwTtSwBIulXSu1He+x7BH9W0cP5dgst5ZkSUUgsraR6jnf95QDtJHSRVJmjHSmRfRe3795Kahz8IfyFoh0xWL351wo4BSQ2Ba+N5k6RLCErNZ5vZrohVVQmC2dpwuwsISnT5VgONJFUsJunngAvCz7MSwfHODJtJ9tSjBFXrzsA1BFdURB5LU6A5QadMqUrp5SVmdg/BNXQ3Epygbwn+eCaEm/wJmEXQa/UpMCdctif7mgKMC9Oaze7BKYug93Ylwa/SscBlRaSxHugXbrueoOewn5mt25M8ldA1BA3/mwl+uccVWn8rMDqstvSPlZikUwmqGsPDRVcDnSQNDucbE/QeF+c9gj/W/EA3g6CENa3YdwSlmBvDPF4TK49EOf9mtoigs2IqQVtU4esuHwMODvc1gZJ7nKCneBpBL/yPBIE8WW4jaPjfRPAjE2/1bRBBAF8paUs4jTSzz4G/E9SUVgOHsPv5e5ugzfc7Sb/4vprZW8BNwIsEvfotgIF7cmBQUFvoDoyXNJegTbl+oc0GAi9E+WHca1J+wbBLT+GX87gwuDsXU9hxMdnM2kvaF/jSzAoHt8jtPwF+a2b/3UtZLFaZvQXMJSZsLPYg5/aImf0ALJN0FoAC+b3phB1XNQlKoKXOA51zLiZJYwiCVhtJyyUNJbhkZaikeQTV5sg7ngYBYy1NqoxedXXOZTwv0TnnMl5a3aysnCqmitVLOxuuBNq1blTaWXAlsOLb//H9+nXxXs8Xl+x9m5rl/uLGnCLZ9rVvmFlcFx4nU3oFuorVqdQm5pUTLo1MeOOu0s6CK4HT+vSIvVEJWe72uP9uf5z7ULx3ciRVWgU651xZJFB6t4J5oHPOJUZAVnZp5yIqD3TOucQpqc1+SeeBzjmXIK+6OufKAy/ROecymvASnXMu08lLdM65csB7XZ1zmc07I5xzmU541dU5Vw6keYkuvXPnnCsDwqprPFOslKTKkj6SNE/SAkm3FbHNEElrIx6peFGsdL1E55xLXFbSqq47gF5mtiV87OMMSa+Z2YeFthtnZr+LN1EPdM65xCTxXtdwROIt4WyFcEp4dGCvujrnElSiqmttSbMipmG/SE3KDh/etAaYYmYzi9jpbyTNl/SCpMaxcuglOudc4uLvdV1nZl2ibRA+HrGDpBrAy5Lam9lnEZtMAsaY2Q5Jw4HRQK9oaXqJzjmXuCR1RkQys40ED07vW2j5ejPbEc6OAjrHSssDnXMuMVL8U8ykVCcsySGpCnA88EWhbSKfJXsKsDBWul51dc4lLnm3gNUHRkvKJiiIPW9mkyXdDswys4nAFZJOAXKB74EhsRL1QOecS1DybgEzs/lAxyKW3xzx+nrg+pKk64HOOZc4vwXMOZfRfDw651zm89FLnHPlgVddnXMZzwfedM5lNHnV1TlXHnjV1TmX6eSBzjmXyYKR1D3QOecymcIpjXmgc84lSGRleWeEcy7DedXVOZfxPNA55zKbt9E55zKdkJfonHOZzzsjnHMZz0t0zrnM5m10zrnywEt0zrmMVhY6I9K7BdE5VyZIimuKI53Kkj6SNE/SAkm3FbFNJUnjJC2RNFNSs1jpeqBzziVGoCzFNcVhB9DLzA4DOgB9JR1RaJuhwAYzawncC/wtVqIe6JxzCUtWic4CW8LZCuFkhTY7FRgdvn4BOE4xEvdA55xLWAkCXW1JsyKmYUWklS1pLrAGmGJmMwtt0hD4FsDMcoFNQK1o+fPOCOdcQkrYGbHOzLpE28DM8oAOkmoAL0tqb2af7bbLIt4WLU0v0TnnEqc4pxIws43Au0DfQquWA40BJOUA+wHfR0ur3JfoKlXMYepjV1GxYg452dm8PPUT/vTwq0x97CqqVa0MQN39qzPrs6/pf/Uofn/ecQw4sSsAOdlZtG1ej8a9RrDhh23sV60K/7rlbA5uUR8zGH7bs8ycvwyASwcey/ABx5Cbt4vXp3/GDfe9QoWcbB68cRCdDm7CLtvFNXe+yPTZiwF4Y9SV1Ku9L9t3/ATAyZc+yNoNQdPFb3p35IbhJ2IGny5awZCRT+7lTy39HNulLVWrVic7O4vsnBwmvPk+Gzd8z5XDzmP5t9/QqHFT7h/1NPvVqAnAh+9P4883XctPubnU3L8WYya8CcCIKy/h7SmvU6t2HV6bNqsg/YUL5nPTtVewbetWGjZuwj3/eoLq1fctWL9y+bf0PboTV1x7AxdddhUA7739Jn+68Vry8vLoP3gIw6+4Zi9+InuRkncdnaQ6wE9mtlFSFeB4ftnZMBE4H/gAOBN428yilujKfaDbsTOXvsPuZ+v2neTkZPH241fz5vufc/zQfxRsM+bui5j07nwA7n3qLe596i0ATjymPZcP/hUbftgGwN3Xncmb//2cs699jAo52exTuSIAx3RpRb+eh9C1/1/Z+VMudWpWA+DCM3oA0LX/X6hTsxoTHryMo865i/xzdsENo5nz+f92y2+LJnW45sI+9BpyDxs3by9Iy8EzL73G/rVqF8w/8sDfOfLongy/4hoevv9uHnng71x305/4YdNGbhlxFU+MeYUGjRqzfu2agvecMfBczhk6nGt/d/FuaY+8+jJG3PJXDu9+NOOfG82/H7qX34+4pWD9n2++jmOO61Mwn5eXx60jfs/o5ydTr0FDzjjhaI474SRatTkohZ9A6Univa71gdGSsglqnM+b2WRJtwOzzGwi8BjwtKQlBCW5gTHzl6zclWVbt+8EoEJONjk52UT+OFTbpxLHdm3NpHfm/+J9/ft24fnXZwNQvWpljurUgidf/gCAn3Lz2LRlOwDDzjqau5+Yws6fcgEKSmZtD6zHOx99WbBs0+btdD64SdS8Xnh6dx55fhobN2/fLS33S1Nfn8wZAwYDcMaAwUx5bRIAE18axwknnkKDRo0BqFWnbsF7uh15FDVq7P+LtJYuWUy3I48CoMexx/H6f14pWDfl1Yk0btp8tyA2b84smjZvQZNmzalYsSInnXYmU1+fnPyDTBdJqrqa2Xwz62hmh5pZezO7PVx+cxjkMLMfzewsM2tpZt3MbGmsdD3QAVlZ4sOxI/jfW3fw9odf8PFn3xSsO6XXYbz70Zds3vrjbu+pUrkCvbsfxIS35gLQvGEt1m3YwqO3ncMHY/6Pf958dkGJrmXTuvTo2IJpT13Dm/++siCYfbpoBSf3PITs7CyaNqhFx4Mb06hezYJ9PHLrOXw4dgQjLv65iaJV07q0alKXt5/4Pe+N/gO9u2dmCaGkhBgy4GRO7d2dsU89BsC6tWuoe0B9AOoeUJ/169YC8PVXS9i0aSNnn34Cp/buzsvPPxsz/dZtDy4IVK9NeonvViwHYNvWrTzy4D1cfs3I3bZf/d1K6jdoWDBfr0FDVn+3MvEDTVPJurwkVVIa6CT1lfRleAXziFTuKxG7dhlHDLyDlifcSJf2TTm4Rf2Cdf37di4otUU66ZhD+GDu0oJqa05ONh3aNmbU+OkcOehvbNu+g2su7B2sy86i5r77cMx5dzPy3gk8c+eFAIx+5QNWrN7I+89ex13X/oYP5y0jNy8PgAtGPknX/n/h+AvvpUfHFpzdrxsA2dnZtGxSlz4X38d51z/Jv24+m/2qVUnp51MWjJv8FhOnfsDjz03gmSce5aMPZhS7bW5eLp/N+4R/P/MST4ydyIP33MGyrxZHTf+OfzzMM088yqm9u7N1y2YqVAx+xO67609ccMnlVK26exNCUU1GSvc73/dQvEGuNANdytrowjr2Q0Bvgl6SjyVNNLPPU7XPRG3asp1psxbTp/vBfP7VKvbfrypd2jVjwNWjfrHtWSd0ZnxEAFyxegMr1mwsKA2+PHUuf7igd7huIxPemgfArAXfsGuXUbtmNdZt2MJ1f3+pII13nryaJf8LSh0r124CYMu2HYx7bRZd2zXluckfsWLNRj6av4zc3F18s3I9i75eQ8smdZhdqC2vvDmgXgMgqIb2PvFk5n8yi9p16rJm9SrqHlCfNatXUat2HQDq1W9Izf1rsU/VquxTtSpdj+jBwgWf0rxFq2LTb9GqDaOfD6q+y75azLtTXgdg3pyPeX3yy9z5xxv4YdMmsrKyqFipEu0P7cSqlSsK3v/dyhXUrVe/yLQzQXm+17UbsMTMlprZTmAswRXNaaV2zWoFJaLKlSrQ6/A2fPn1agDO6N2R16Z/xo6dubu9Z99qlTmqc8uCDgqA1es3s/y7DbRqGrT39OzWhi+WfgfApHfn07NbawBaNqlLxQo5rNuwhSqVKxRUb3sd3pbcvF18sfQ7srOzqFWjKgA5OVmceEx7Fny1KkjrnXkc2zVIq1aNqrRqWpdlK9an5LMpK7Zt3cqWLZsLXs949y1atT2Y4044iZfGBdXSl8Y9y/F9+wFwfN9+zPrwv+Tm5rJ92zbmzZlFy1Ztou4jv8Ni165dPHTv3xh0/kUAjJ04lfdmfcF7s75gyLDfcumV13Le0Es5tGNnvlm6hG+/+ZqdO3fynwkvcNwJJ6XqIyh15bZER8TVy6HlwOGFNwqvjA6ujq6w93sQ69Xel1G3n0t2VhZZWeLFKXN4bXpwbeJZJ3Tm7ife/MV7TvnVYbz14Rds+3Hnbsuv/tt4nvjLECrmZPP1inUMu+UZAEZP+IBHbh3MrPEj2flTHhfd/DQAdWpWZ9I/f8uuXcbKtRsZemNwV0ulCjlMfOi3VMjJJjs7i3dmfsHjL70PwJT/LuT4Iw9izos3kJdnjPzHBL7ftDVln09ZsG7tGi67IOh4y83L5ZTT+3Nsrz4c2qEzV1x8LuOfG02Dho154N/B+WjZui3H9OrNSb/qRpay6D94CK0PagfAVZecz8z/TmPD9+vp0aElV157I/0HD2HSy+N55olHAOhz4qmcOei8qHnKycnhlr/ewwUDTyEvL4+zBp1H67YHp/BTKF1x3sdaahTj8pM9T1g6CzjBzC4K588FupnZ5cW9J2ufulapTf+U5Melxmdv3FXaWXAlcFqfHnw6d05So1Kleq2s0eD749p26T0nzo51Z0QqpLJEV3D1cqgRkLndTs6VUwLSvIkupW10HwOtJDWXVJHgor6JKdyfc65UlONeVzPLlfQ74A0gG3jczBakan/OudKT7iW6lN4CZmavAq+mch/OuVKm4KL7dFbu73V1ziVGeKBzzpUD5brq6pwrH9L9zggPdM65xMhLdM65DBdcR5fekc4DnXMuQfLOCOdc5vMSnXMus3kbnXMu03kbnXOuXEjzOOfPjHDOJS5ZN/VLaizpHUkLJS2QdGUR2/SUtEnS3HC6OVa6XqJzziUmufe65gJ/MLM5kqoDsyVNKeIRDNPNrF+8iXqgc84lJJnj0ZnZKmBV+HqzpIUEo5Un9KwZr7o65xJUovHoakuaFTENKzZVqRnQEZhZxOojJc2T9JqkdrFy6CU651zCSlCiWxfPUOqSqgEvAleZ2Q+FVs8BmprZFkknAhOA4h/hhpfonHNJkMwRhiVVIAhyz5rZS4XXm9kPZrYlfP0qUEFS7WhpeonOOZcQJbEzQkE0fAxYaGb3FLNNPWC1mZmkbgQFtqjP/PRA55xLWBIvGO4BnAt8KmluuGwk0ATAzB4GzgQulZQLbAcGWozHGXqgc84lLIm9rjMIOnKjbfMg8GBJ0vVA55xLmN8C5pzLbH5Tv3Mu04nSfWZrPDzQOecSlu0DbzrnMl2aF+g80DnnEiOV4c4ISftGe2MRt2U458qpNK+5Ri3RLQCM3a9pyZ83wgv4nHOuzJbozKzx3syIc67sSvM4F99N/ZIGShoZvm4kqXNqs+WcKysEZEtxTaUlZqCT9CDwK4L7zwC2AQ+nMlPOuTIkzpFLSrN6G0+va3cz6yTpEwAz+15SxRTnyzlXhqR71TWeQPeTpCyCDggk1QJ2pTRXzrkyQ0BWmke6eNroHiIYBK+OpNuAGcDfUpor51yZIsU3lZaYJToze0rSbOD4cNFZZvZZarPlnCsrkjnwZqrEe2dENvATQfXVh193zu2mzFddJd0AjAEaAI2A5yRdn+qMOefKDsU5lZZ4SnTnAJ3NbBuApD8Ds4G/pjJjzrmyo8zeGRHhm0Lb5QBLU5Md51xZE/S6lnYuoot2U/+9BG1y24AFkt4I5/sQ9Lw651zBBcPpLFqJLr9ndQHwn4jlH6YuO865siiJjztsDDwF1CO4XvdRM7uv0DYC7gNOJCiIDTGzOdHSjXZT/2OJZto5l/mSXHXNBf5gZnMkVQdmS5piZp9HbPNroFU4HQ78K/y/WDHb6CS1AP4MHAxUzl9uZq1LfAjOuYyUrKqrma0CVoWvN0taCDQEIgPdqcBT4bNcP5RUQ1L98L1FiueauCeBJwgC96+B54Gxe3QUzrmMVILLS2pLmhUxDSs2TakZ0BGYWWhVQ+DbiPnl4bJixdPruo+ZvSHpbjP7CrhR0vQ43uecKwekEl0wvM7MusROU9UIbj29qojRzIvamUVLL55AtyNs/PtK0nBgBVA3jvc558qJZHa6SqpAEOSeNbOXithkORA5MHAjYGW0NOOpuv4eqAZcAfQALgYujCfDzrnyIStLcU2xhIWqx4CFZnZPMZtNBM5T4AhgU7T2OYjvpv78+vFmfh580znngOAB1km817UHQZz5VNLccNlIwmfUmNnDwKsEl5YsIbi85IJYiUa7YPhlotR7zeyMeHPunMtgSRyCycxmEOO22LC39bclSTdaie7BkiSUDB0PasL7M/f6bp0rNypmp2bwoTJ7Z4SZvbU3M+KcK7vSfey2eMejc865IokyXKJzzrl45aR5kS7uQCepkpntSGVmnHNlT/A8iPQu0cUzwnA3SZ8Ci8P5wyQ9kPKcOefKjCzFN5Va/uLY5n6gH7AewMzmETzQ2jnngAx4ChiQZWbfFCqa5qUoP865MqYsPNc1nkD3raRugEnKBi4HFqU2W865siQ7veNcXIHuUoLqaxNgNTA1XOacc0hJvQUsJeK513UNMHAv5MU5V0aleZyLa4ThURRxz6uZFTtgnnOufCmzTwGLMDXidWXgdHYf3dM5V45lRGeEmY2LnJf0NDAlZTlyzpU5aR7n9ugWsOZA02RnxDlXRgmy0zzSxdNGt4Gf2+iygO+BEanMlHOu7Ejy4w5TImqgC4c1PozgOREAu8JB75xzrkC6B7qot4CFQe1lM8sLJw9yzrlfkBTXVFriudf1I0mdUp4T51yZlF91Teeb+qM9MyLHzHKBo4CLJX0FbCU4LjMzD37OuaQ+MyJVorXRfQR0Ak7bS3lxzpVBAnKSVFyT9DjBaElrzKx9Eet7Aq8Ay8JFL5nZ7bHSjRboBGBmX5U4t865ciWJJbonCR7M9VSUbaabWb+SJBot0NWRdHVxK6M8XNY5V66IrOhPKIybmU2T1CwpiUWIFuiygWrEeMaic658Cx6OE/fmtSXNiph/1MweLeEuj5Q0D1gJXGNmC2K9IVqgWxVP3dc5V86VrEd1nZl1SWBvc4CmZrZF0onABKBVrDdFu7zES3LOuZgEZGcprilRZvaDmW0JX78KVJBUO9b7opXojks4V865cmFvjV4iqR6w2swsHPk8i/B5NtEUG+jM7Psk5s85l8GSFeckjQF6ErTlLQduASoAmNnDwJnApZJyge3AwHju2PIHWDvnEiLiu8UqHmY2KMb6BwkuPykRD3TOucSUgQdYe6BzziUsvcOcBzrnXIJEBgy86ZxzsaR5nPNA55xLVOmONRcPD3TOuYQks9c1VTzQOecS5iU651zGS+8w54HOOZcgZcLjDp1zLhavujrnMl56hzkPdM65JEjzAp0HOudcYoLLS9I70nmgc84lzEt0zrkMp7028Oae8kDnnEuIV12dc5lPXnV1zpUDHuiccxlPaV51TfdBB9JWm5bN6NLhEA7v3IEehwePqTzn7AEc3rkDh3fuQJuWzTi8cwcAvvn6a2pWr1Kw7vLLhhekc8tNN9CyeWNq16hWKseRyS656EKaNKhL5w7tC5Z9//33nNS3N+0PasVJfXuzYcOGgnXT3nuXwzt3oNNh7ejd69iC5W++8TqHtmtDu7YtuevOOwqWD794KN06HUbXjocyaMCZbNmyZe8cWJrJH3gznilmWtLjktZI+qyY9ZJ0v6QlkuZL6hRPHj3QJeD1qe8wc/Zc3p8ZPHj8mefGMXP2XGbOnstpp/+GU08/o2DbA1u0KFj3wD8fLlh+4kknM/2/H+31vJcH554/hFcmv77bsrvvvIOevY7js4WL6dnrOO4OA9fGjRu58vLLGP/yRObMW8CzY8cDkJeXx1VX/JZXJr3GJ/M/Z/zYMSz8/HMA7vz7vXw0Zx4ffzKfxo2b8K9/lviZLRlDim+Kw5NA3yjrf03wwOpWwDDgX/Ek6oEuBcyMF194nv4Doj7QCIDDjziC+vXr74VclT9HHX0M+++//27LJk96hXPOPR+Ac849n0kTJwAwbsxznHraGTRp0gSAunXrAvDxRx/RokVLmh94IBUrVuSsAQOZPOkVAPbdd18gON8/bt+e9vd7ppLi/BeLmU0Doj1q9VTgKQt8CNSQFPMPyAPdHpJjFvBJAAALJUlEQVTEyb/uQ/dunXls1KO7rXt/xnQOqHsALVu1Klj29bJlHNGlI717HcuMGdP3dnZdaM3q1QU/LPXr12ftmjUALF68iI0bNtDnuJ5079aZZ59+CoCVK1fQqFHjgvc3bNiIFStWFMwPG3oBzRrV48svv+Cy316+F48kfQjIUnxTEjQEvo2YXx4uiyplnRGSHgf6AWvMrH2s7cuat997nwYNGrBmzRr69e1Nm7ZtOeroYwB4fuwYzhr4c2muXv36LFr6P2rVqsWc2bPpf+ZpzJm3oKBE4Epfbm4uc+bM5rU332L79u30PPpIuh1+BEU9Gzmy5PboY0+Ql5fH1VdezgvPj+O8IRfszWynifhKa6HakmZFzD9qZo8Wu3VRO/ulmA+wTmWJ7kmi17XLtAYNGgBBFeeU007n44+Ddrbc3FxemfASZ541oGDbSpUqUatWLQA6de7MgQe2YPGiRXs/0466BxzAqlWrAFi1ahV1wipqw0aN6HNCX6pWrUrt2rU56qhjmD9/Hg0bNmL58p8LECtWLC849/mys7M5s/8AJrz84t47kHQSZ/tc+Puwzsy6REwlCXIQlOAaR8w3AlbGelPKAl0cde0ya+vWrWzevLng9dQpb9KuXVBoffutqbRu05ZGjRoVbL927Vry8vIAWLZ0KUuWLKb5gQfu/Yw7Tup3Cs88PRqAZ54eTb+TTwXg5JNP5f0Z08nNzWXbtm18/PFM2rY9iC5du7JkyWK+XraMnTt3Mn7cWE7qdwpmxldLlgBBG91/Jk+idZu2pXZcpSmZva5xmAicF/a+HgFsMrNVsd5U6tfRSRpG0HtC47AhON2tWb2aAWeeDkBuXi4DBp5NnxOCwuv4cWN/0QkxY/o0/njbzeRk55Cdnc0DDz1c0Eg+csR1jBv7HNu2baNFs0ZccOFF3HjzrXv1eDLVeecMYvp777Ju3TpaNGvETTffxjXXjeCcQf0Z/cRjNG7cpKB3te1BB9H7hL507XQoWVlZDLngItq1D3687r3vQU4+6QTy8vI4f8iFHNyuHbt27eKiC89n8w8/YBiHHHIY9z8UVwdgRkpWN4ykMUBPgirucuAWoAKAmT0MvAqcCCwBtgFxtRWoqDaIZJHUDJgcbxtd585dLP9SDedc8vU4vAuzZ89KavfwQYd0tCcmvBPXtke2rDnbzLokc//xKPUSnXOu7Ev3OyM80DnnEpbulxCmrDMirGt/ALSRtFzS0FTtyzlXuhTnVFpSVqIzs9i3BTjnyjzhTwFzzmU6H4/OOVcepHmc80DnnEuCNI90Huiccwkq0b2upcIDnXMuIfmjl6QzD3TOucR5oHPOZTqvujrnMp5fXuKcy3hpHuc80DnnElTa93fFwQOdcy4hQa9rekc6D3TOuYSld5jzQOecS4Y0j3Qe6JxzCfPLS5xzGS/Nm+g80DnnEpfmcS6lz3V1zpUD+QNvxjPFTEvqK+lLSUskjShi/RBJayXNDaeL4smjl+icc4lJ0sCbkrKBh4DeBA+q/ljSRDP7vNCm48zsdyVJ20t0zrmEJemZEd2AJWa21Mx2AmOBU5ORPw90zrnExR/pakuaFTENi0ilIfBtxPzycFlhv5E0X9ILkhrHkz2vujrnElSigTfXRXmAdVGJWKH5ScAYM9shaTgwGugVa6deonPOJSR/4M14phiWA5EltEbAysgNzGy9me0IZ0cBnePJowc651ziktNI9zHQSlJzSRWBgcDE3XYj1Y+YPQVYGE/2vOrqnEtYMu6MMLNcSb8D3gCygcfNbIGk24FZZjYRuELSKUAu8D0wJJ60PdA55xKWrDsjzOxV4NVCy26OeH09cH1J0/VA55xLWLrfGeGBzjmXmCRdMJxKHuiccwnJvwUsnXmgc84lLL3DnAc651wSpHmBzgOdcy5xPvCmcy7zpXec80DnnEtcmsc5D3TOucRI/rhD51x5kN5xzgOdcy5xaR7nPNA55xKX5jVXD3TOuUSVaODNUuGBzjmXkOAWsNLORXQe6JxzCfNA55zLeF51dc5lNh+myTmX6eJ8Zmup8kDnnEtcmkc6D3TOuYT5LWDOuYyX3mHOA51zLhnSPNJ5oHPOJSzdLy+RmZV2HgpIWgt8U9r5SIHawLrSzoQrkUw9Z03NrE4yE5T0OsHnFY91ZtY3mfuPR1oFukwlaZaZdSntfLj4+TnLLFmlnQHnnEs1D3TOuYzngW7veLS0M+BKzM9ZBvE2OudcxvMSnXMu43mgc85lPA90zrmM54EuRSS1kXSkpAqSsks7Py4+fq4yk3dGpICkM4C/ACvCaRbwpJn9UKoZc8WS1NrMFoWvs80sr7Tz5JLHS3RJJqkCMAAYambHAa8AjYHrJO1bqplzRZLUD5gr6TkAM8vzkl1m8UCXGvsCrcLXLwOTgYrA2VKaD9xVzkiqCvwOuArYKekZ8GCXaTzQJZmZ/QTcA5wh6Wgz2wXMAOYCR5Vq5twvmNlW4ELgOeAaoHJksCvNvLnk8UCXGtOBN4FzJR1jZnlm9hzQADisdLPmCjOzlWa2xczWAZcAVfKDnaROktqWbg5donw8uhQwsx8lPQsYcH34h7IDOABYVaqZc1GZ2XpJlwB3SfoCyAZ+VcrZcgnyQJciZrZB0ijgc4JSwo/AOWa2unRz5mIxs3WS5gO/Bnqb2fLSzpNLjF9esheEjdoWtte5NCepJvA88Aczm1/a+XGJ80DnXBEkVTazH0s7Hy45PNA55zKe97o65zKeBzrnXMbzQOecy3ge6JxzGc8DXRkiKU/SXEmfSRovaZ8E0uopaXL4+hRJI6JsW0PSZXuwj1slXRPv8kLbPCnpzBLsq5mkz0qaR1c+eKArW7abWQczaw/sBIZHrlSgxOfUzCaa2R1RNqkBlDjQOZcuPNCVXdOBlmFJZqGkfwJzgMaS+kj6QNKcsORXDUBSX0lfSJoBnJGfkKQhkh4MXx8g6WVJ88KpO3AH0CIsTd4VbnetpI8lzZd0W0RaN0j6UtJUoE2sg5B0cZjOPEkvFiqlHi9puqRF4VBKSMqWdFfEvi9J9IN0mc8DXRkkKYfg9qRPw0VtgKfMrCOwFbgRON7MOhEM+nm1pMrAKOBk4GigXjHJ3w+8Z2aHAZ2ABcAI4KuwNHmtpD4Ew1B1AzoAnSUdI6kzMBDoSBBIu8ZxOC+ZWddwfwuBoRHrmgHHAicBD4fHMBTYZGZdw/QvltQ8jv24cszvdS1bqkiaG76eDjxGMCLKN2b2Ybj8COBg4P1w6LuKwAdAW2CZmS0GCEfnGFbEPnoB50HBMEWbwluiIvUJp0/C+WoEga868LKZbQv3MTGOY2ov6U8E1eNqwBsR654Pb5tbLGlpeAx9gEMj2u/2C/e9KI59uXLKA13Zst3MOkQuCIPZ1shFwBQzG1Rouw4Eo6kkg4C/mtkjhfZx1R7s40ngNDObJ2kI0DNiXeG0LNz35WYWGRCR1KyE+3XliFddM8+HQA9JLQEk7SOpNfAF0FxSi3C7QcW8/y3g0vC92eHw75sJSmv53gAujGj7ayipLjANOF1SFUnVCarJsVQHVoVD0A8utO4sSVlhng8Evgz3fWm4PZJah6MEO1csL9FlGDNbG5aMxkiqFC6+0cwWSRoG/EfSOoJRj9sXkcSVwKOShgJ5wKVm9oGk98PLN14L2+kOAj4IS5RbCIagmiNpHMFoyt8QVK9juQmYGW7/KbsH1C+B9wjG8RsejvP3b4K2uzkKdr4WOC2+T8eVV35Tv3Mu43nV1TmX8TzQOecyngc651zG80DnnMt4HuiccxnPA51zLuN5oHPOZbz/B6wojAs5PUkvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# compute and plot confusion matrix with the best model \n",
    "cm_best = confusion_matrix(test_values, y_classes_best)\n",
    "plot_confusion_matrix(cm_best, classes=[0,1],normalize=False,\n",
    "                      title='Confusion matrix, without normalization')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized confusion matrix\n",
      "[[0.88181017 0.11818983]\n",
      " [0.35165709 0.64834291]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAEYCAYAAAAnEYFiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcFNW5xvHfMzOyibihIjsiokjcd6OSxAWXq8ZEBY3GuEUTTKIxXo1KjNEk6r0xixij1yVxQVwjKglmcU1cAEUjKIgosomguIPA8N4/qgabYWa6h+6e7ul5vnzqQ1fV6XNOT/W8c06dqlOKCMzMKlVVqStgZlZMDnJmVtEc5MysojnImVlFc5Azs4rmIGdmFa3ig5ykjpIelPSBpLvzyOd4SY8Usm6lImkfSdPKpTxJfSWFpJqWqlOxSLpE0m3p696SPpZUXeAyWvT4tXZlE+QkHSdpYvqlmC/pL5K+WICsvw5sBmwcEUevbSYRcXtEHFiA+hRVGiy2bCpNRDwZEQNbqk71y5P0pqT9W6r8UomItyKic0TU5pNP/WPa0sevtSuLICfpHODXwM9JAlJv4FrgiAJk3weYHhErCpBXq1cJraVCUaIsfgesiCKipAuwPvAxcHQTadqTBMF56fJroH26bwgwB/gh8A4wH/hWuu+nwDJgeVrGKcAlwG0ZefcFAqhJ108CZgIfAW8Ax2dsfyrjfXsBE4AP0v/3ytj3GPAz4F9pPo8AXRv5bHX1Py+j/kcChwDTgfeAH2ek3w14Gng/TXsN0C7d90T6WT5JP++xGfn/N/A2cGvdtvQ9/dMydkrXuwOLgCE5HLs/Aj9MX/dIy/5Our5lmq/qlXcrsBJYktbxvIxj8E3grbT8C5so9xZgFPBw+vN9FujfjGNzeXpslqT1fAy4DPh3WqcHgY2B24EP0zz6ZuTxG2B2um8SsE/GvktIv18Zn6sG2DPNu25ZCryZzzHNKHOb9DO8D0wBDs/1Z9UWltJXAIYCK0iDTCNpLgWeATYFNkm/jD9L9w1J338psA5JcPgU2LD+l66R9cwv4rrpF3dgum9zYNv09UmkQQ7YCFgMnJC+b3i6vnHGL9LrwFZAx3T9l418trr6j0zrfxqwELgDWA/YNv2F2CJNvzOwR1puX+AV4AcZ+QWwZQP5X0Hyx6JjA78kp6X5dALGA/+T47E7GXgwfX1c+pnHZOx7IKMOmeW9CezfwDG4Ia3f9sBnwDaNlHsLSQDdLf053A7c2Yxj81b6c61Jf+aPATNIAv76wFSSPzD7p2n+BNycUf43SIJgDckf17eBDvW/X9T7A5rx/royf5HHMZ2TkdcM4MdAO+DLJMFsYLafVVtZyqGpvjGwKJruTh4PXBoR70TEQpIW2gkZ+5en+5dHxDiSv3hre85iJTBYUseImB8RUxpIcyjwWkTcGhErImI08CrwXxlpbo6I6RGxBLgL2KGJMpcDl0fEcuBOoCvwm4j4KC1/CrAdQERMiohn0nLfBP4A7JfDZ/pJRHyW1mc1EXED8BrJX/nNgQuz5FfncWCftMu3L3AlsHe6b790f3P8NCKWRMSLwIskwa4x90XEc+n35nY+//nmcmxuiYgp6f7l6babI+L1iPgA+AvwekT8Pc3/bmDHujdHxG0R8W76/v8l+ePRnO/bb0laZhem+a3NMa2zB9CZ5I/osoj4J/AQSXCv09jPqk0ohyD3LtA1y7mi7sCsjPVZ6bZVedQLkp+SHPhmiYhPSLoDZwDzJT0saesc6lNXpx4Z6283oz7vxucnp+uC0IKM/Uvq3i9pK0kPSXpb0ock5zG7NpE3wMKIWJolzQ3AYOB3EfFZlrQARMTrJH9QdgD2IfnlmidpIGsX5JrzM2ssbS7HZnYD+dX/eTf48weQ9ENJr6Qj9u+TtP6yHYO6936bpCV2XESsTLetzTGt0x2YXZdXKp/vYsUphyD3NEl37Mgm0swjGUCo0zvdtjY+IemW1emWuTMixkfEASQtmldJfvmz1aeuTnPXsk7N8XuSeg2IiC4k3RRleU+TU81I6kxynvNG4BJJGzWjPo+TjGC3i4i56fqJwIbA5LWpT55yOTZrXb6kfUjObx5DckpkA5Jzf9mOQd17fwYckbYY66zNMa0zD+hVbwClpb6LrULJg1x6sEcCoyQdKamTpHUkHSzpyjTZaOAiSZtI6pqmv20ti5wM7Jtew7Q+cEHdDkmbSTpc0rok54Q+Bhoa/h8HbJVe9lIj6VhgEElLptjWIzlv+HHayjyz3v4FwBbNzPM3wKSIOJXkBPV1dTvS674ea+K9jwMjSE6QQ3Ku6SyS85eNXTqxNnXMVbGPzXok5zgXAjWSRgJdsr1JUi9gDHBiRExvIM+1PabPkvzhPi/9vRlC0jW/M7ePU/lKHuQAIuJXwDnARSRfntkkvzh/TpNcBkwEXgL+Azyfblubsv5G8mV7iWRkLPPLX0VyInkeycna/YDvNJDHu8Bhadp3SUYID4uIRWtTp2Y6l+Qk/0ckrcwx9fZfAvxR0vuSjsmWmaQjSAZ/zkg3nQPsJOn4dL0XyUhkYx4n+SWtC3JPkbSUn2j0HfALkj9a70s6N1sdm6MFjs14knN200m6hUtpuPtb31dIeg33pNeCfiyp7nzvWh/TiFgGHA4cTDIqfS1JIH11LT5bRVKEJ820xkmaDHwlDR5mrY6DnJlVtLLorpqZFYuDnJlVNAc5M6toZXWztmo6htqtV+pqWDPsuE3vUlfBmmHWrDdZtGhRrtfg5aS6S5+IFWvcSNOgWLJwfEQMLWT52ZRXkGu3Hu0HZr3qwcrIv569ptRVsGbYe/ddCp5nrFiS8+/t0smjcr2To2DKKsiZWWskKOMZqxzkzCw/AqoKOvlxQTnImVn+VNDTfAXlIGdmeXJ31cwqnVtyZlaxhFtyZlbJ5JacmVU4j66aWeXywIOZVTLh7qqZVTi35Myscrm7amaVrsrdVTOrVL531cwqm7urZlbpPLpqZhXNLTkzq1jybV1mVunKeOChfNuYZtZKpAMPuSzZcpKGSpomaYak8xvY31vSo5JekPSSpEOy5ekgZ2b5q+uyZluazELVwCjgYGAQMFzSoHrJLgLuiogdgWHAtdmq5iBnZvmpm08u/5bcbsCMiJgZEcuAO4Ej6qUJoEv6en1gXrZMfU7OzPLUrOvkukqamLF+fURcn77uAczO2DcH2L3e+y8BHpF0FrAusH+2Ah3kzCx/uY+uLoqIxh7+2lAmUW99OHBLRPyvpD2BWyUNjoiVjRXoIGdm+SvM6OocoFfGek/W7I6eAgwFiIinJXUAugLvNFq1QtTMzNowFWx0dQIwQFI/Se1IBhbG1kvzFvCVpFhtA3QAFjaVqVtyZpa/AlwMHBErJI0AxgPVwE0RMUXSpcDEiBgL/BC4QdLZJF3ZkyKifpd2NQ5yZpY3FeiOh4gYB4yrt21kxuupwN7NydNBzszyksx+7tu6zKxSiYbHRcuEg5yZ5UlUVZXvGKaDnJnlzd1VM6toDnJmVrl8Ts7MKpmQW3JmVtk88GBmFc0tOTOrXD4nZ2aVzi05M6tYHngws4rnIGdmlUugKgc5M6tgbsmZWUVzkDOziuWBBzOrfOUb4/wgm3wdsNc2vHj/xbz8wE8491sHrLG/V7cN+ev13+Pp0f/Nc2Mu4KAvJg8Er6mp4oZLT2DCXT/mhXsv4tyTD2zpqrdJj4z/K9ttO5Btt96Sq6785Rr7n3ryCfbcdSc6d6jhvnvvWbX9xcmT2e+Le7LT9tuy647bcfddY1qy2uVNSXc1l6UUHOTyUFUlfn3+MRwx4lp2/NplHD10Z7beottqaf771KHc+7fn2XP4FZx4wc385oJjAfja/jvRvl0Nux7zc/Y6/gpO/dre9N58o1J8jDajtraWH3zvuzzw4F944aWp3H3naF6ZOnW1NL169eb6G2/h2GHHrba9U6dO3Hjzn3j+xSk88PBfOe+HP+D9999vyeqXtaqqqpyWbCQNlTRN0gxJ5zew/2pJk9NluqSsB8Hd1TzsOrgvr89exJtz3wXg7vHPc9iQ7Xh15tur0kQEXdbtAMD6nTsyf+EHyXaCTh3aUV1dRcf27Vi2vJaPPlna8h+iDZnw3HP0778l/bbYAoCjjx3GQw8+wDaDBq1K06dvX2DNG84HbLXVqtfdu3dnk002ZdHChWywwQbFr3hrUIBGmqRqYBRwAMkzWCdIGps+vAaAiDg7I/1ZwI7Z8nWQy0P3TddnzoLFq9bnLljMboP7rpbm8j+M48FrR3DmsP3o1LE9h57xOwDu+/sLHDZkO9742+V06tCO8/7nPhZ/+GlLVr/NmTdvLj17fv7s4h49evLcc882O58Jzz3HsuXL2KJ//0JWr1UrUFd0N2BGRMxM87wTOAKY2kj64cBPsmVa1O5qtqZna6cG/nzVfwDkMUN34bYHn2HLoRfz1bN+z42XnYgkdt22L7W1K9niwAvZ5tCf8P0TvkzfHhu3TMXbqIYez9ncX8758+dzyrdO4A833FzW0wu1pFzPx6U/666SJmYsp2dk1QOYnbE+J93WUJl9gH7AP7PVr2gtuVyanq3d3Hfep+dmG65a77HZhsxLu6N1vnnknhzx3VEAPPvSG3Rotw5dN1iXYw7ehUf+PZUVK1aycPHHPD15JjsP6r2q62uF16NHT+bM+fx3aO7cOXTv3j3n93/44Yccdfih/OSnl7H7HnsUo4qtVjP+WCyKiF0ay6aBbY09OHoYcE9E1GYrsJh/ilY1PSNiGVDX9KwYE6fMYsvem9Cn+8asU1PN0QftxMOPvbRamtlvv8eQ3QYCMLDfZnRovw4LF3/MnLffY8iuyfZOHdqx23Z9mfbmghb/DG3JLrvuyowZr/HmG2+wbNky7h5zJ4cednhO7122bBnHfv2rHPeNE/na148uck1bnwKNrs4BemWs9wTmNZJ2GDA6l7oVM8jl1PSUdHpd0zVWLClidQqvtnYlZ19xFw9e+10m33cR9z7yAq/MfJuLzzyUQ/f7AgDn/+p+Tj5qL54dcz5//MW3OG3krQBcN+YJOndqx6R7LuSp23/ErQ88w8uvNXY8rRBqamq4+jfX8F+HHsQOX9iGrx19DIO23ZZLLxnJQw+OBWDihAn079uT++69m7O+82122n5bAO69+y6eevIJbvvTLey+8w7svvMOvDh5cik/TllRlXJaspgADJDUT1I7kkA2do2ypIHAhsDTOdWtofMUhSDpaOCgiDg1XT8B2C0izmrsPVWdNo32A48pSn2sOBZPuKbUVbBm2Hv3XZg0aWJBL1hr321A9Dz+tzmlnfmrQyY10V1F0iHAr4Fq4KaIuFzSpcDEiBibprkE6BAROZ3nL+boanOanmbWSgko1HW+ETEOGFdv28h665c0J89idldzanqaWWvXrNHVFle0llxErJA0AhjP503PKcUqz8xKp4zvzy/uxcANNT3NrMIoucWxXPmOBzPLi3CQM7MK12a7q2bWNnjSTDOrXHJLzswqWHKdXPlGOQc5M8uTPPBgZpXNLTkzq1w+J2dmlczn5Mys4pVxjHOQM7P8uSVnZpXL966aWSUr5HxyxeAgZ2Z5Kt1ccblwkDOzvJVxjHOQM7P8uSVnZhVLZT7w4EeAm1neCvWMB0lDJU2TNENSg0/jknSMpKmSpki6I1uebsmZWd4K0VuVVA2MAg4gedrfBEljI2JqRpoBwAXA3hGxWNKm2fJ1S87M8lagltxuwIyImBkRy4A7gSPqpTkNGBURiwEi4p1smTrImVl+0hv0c1mArpImZiynZ+TUA5idsT4n3ZZpK2ArSf+S9Iykodmq5+6qmeVFzbtOblFE7NJoVmuKeus1wABgCMkD65+UNDgi3m+sQAc5M8tbdWFGV+cAvTLWewLzGkjzTEQsB96QNI0k6E1oLFN3V80sb83orjZlAjBAUj9J7YBhwNh6af4MfCkpU11Juq8zm8rULTkzy0sSwPJvyUXECkkjgPFANXBTREyRdCkwMSLGpvsOlDQVqAV+FBHvNpVvo0FOUpcsFfqwuR/CzCpToa4FjohxwLh620ZmvA7gnHTJSVMtuSkkJ/0yq1+3HkDvXAsxs8rWKm/rioheje0zM8tUxjEut4EHScMk/Th93VPSzsWtlpm1FgKqpZyWUsga5CRdQzKacUK66VPgumJWysxakRzvdihVlzaX0dW9ImInSS8ARMR76fCumRlQ3t3VXILccklVpFceS9oYWFnUWplZqyGgqoyjXC7n5EYB9wKbSPop8BRwRVFrZWatSoEuBi6KrC25iPiTpEnA/ummoyPi5eJWy8xai3KfNDPXOx6qgeUkXVbfCmZmq2nV3VVJFwKjge4kN8zeIemCYlfMzFoP5biUQi4tuW8AO0fEpwCSLgcmAb8oZsXMrPVolXc8ZJhVL10NWe76N7O2IxldLXUtGtfUDfpXk5yD+xSYIml8un4gyQirmdmqi4HLVVMtuboR1CnAwxnbnyledcysNWqVo6sRcWNLVsTMWqdW212tI6k/cDkwCOhQtz0itipivcysFSnn7mou17zdAtxMErAPBu4ieVSYmRlQ3peQ5BLkOkXEeICIeD0iLiKdY93MTEouBs5lKYVcLiH5TElb9HVJZwBzgaxPrTaztqOMe6s5teTOBjoD3wP2JnmC9cnFrJSZtS5VVcppyUbSUEnTJM2QdH4D+0+StFDS5HQ5NVueudyg/2z68iM+nzjTzAxIHi5diK6opGqSWY8OIHm+6gRJYyNiar2kYyJiRK75NnUx8P2s+fTqVSLiqFwLMbMKVrhplHYDZkTETABJdwJHAPWDXLM01ZK7Jp+M10b/fpvz61svbuliLQ97XPaPUlfBmmH6/I+Kkm8zLiHpKmlixvr1EXF9+roHMDtj3xxg9wby+JqkfYHpwNkRMbuBNKs0dTGwv71mlpNmzL+2KCJ2aWRfQ5Gyfm/yQWB0RHyWDoT+EfhygepmZrYmQaEeZDMHyHwUak9gXmaCiHg3Ij5LV28Asj450EHOzPJWU5XbksUEYICkfunDsoYBYzMTSNo8Y/Vw4JWsdcv1Q0hqnxFBzcyAuuc35D/yEBErJI0AxpPMRn5TREyRdCkwMSLGAt+TdDiwAngPOClbvrncu7obcCOwPtBb0vbAqRFx1lp/GjOrKIW6QT8ixgHj6m0bmfH6AqBZM5Pn0l39LXAY8G5ayIv4ti4zy9Cqn9YFVEXErHrN0doi1cfMWplyf+5qLkFudtpljfSK5LNIrk8xMwOgunxjXE5B7kySLmtvYAHw93SbmRkq4Qwjucjl3tV3SIZyzcwaVMYxLqfR1Rto4B7WiDi9KDUys1anVU9/TtI9rdMB+Cqr319mZm1Yqx94iIgxmeuSbgX+VrQamVmrU8YxLvc7HjL0A/oUuiJm1koJqss4yuVyTm4xn5+TqyK5lWKNGTvNrG1q1Y8kTJ/tsD3Jcx0AVkZEoxNpmlnbVM5BrsnbutKAdn9E1KaLA5yZraFAUy0VRS73rj4naaei18TMWqW67mouSyk09YyHmohYAXwROE3S68AnJJ8pIsKBz8wK+YyHomjqnNxzwE7AkS1UFzNrhQTUlPFJuaaCnAAi4vUWqouZtVKttSW3iaRzGtsZEb8qQn3MrNURVQ0+g6Y8NBXkqoHONPwEHTMzoO5BNqWuReOaCnLzI+LSFquJmbVOJRw5zUXWc3JmZk0RUF3GUa6p6+S+0mK1MLNWrSqdODPbko2koZKmSZohqdHbRyV9XVJIauxB1Z/XrbEdEfFe1hqZmVGYB9mkj1cYBRwMDAKGSxrUQLr1gO8Bz+ZSNz9c2szyIpJAksuSxW7AjIiYGRHLgDuBIxpI9zPgSmBpLvVzkDOz/KhZ9652lTQxY8mcYbwHq0/IOyfd9nlR0o5Ar4h4KNfqrc18cmZmq2nGsMOiiGjsPFpD2ayaFERSFXA1cFIzquYgZ2b5EQWbNHMO0CtjvScwL2N9PWAw8FjaKuwGjJV0eERMbCxTBzkzy1uBLgaeAAyQ1I9kDsthwHF1OyPiA6Dr52XqMeDcpgIc+JycmeUtt/Nx2eaTS2c9GgGMB14B7oqIKZIulXT42tbOLTkzy0vd6GohRMQ4YFy9bSMbSTsklzwd5Mwsb6Wa9TcXDnJmlrfyDXEOcmaWJ7X2RxKamWXj7qqZVbTyDXEOcmZWAGXckHOQM7P8JJeQlG+Uc5Azs7y5JWdmFSy3CTFLxUHOzPLi7qqZVbYcZv0tJQc5M8ubg5yZVTS5u1q5Jj31T66/4mJW1tZy4FHHc/SpZ622f9xdf+Th0TdTVV1Nx07rMuInV9G7/0AWzH2LM4/Ylx59+wMwcLudGTHyylJ8hDZlry034ryhW1FVJe5/fh43PzVrjTQHbrsp3x6yBUQwfcHHXHDvFAAmjfwyM975GID5HyzlB6NfatG6l6sCTppZFA5yeaitreX3l1/AZdffxcbdNufsYUPZ/UsH0rv/wFVphhxyFIcc800Ann10PP931SVcet1oALr16sPv7vlHSereFlUJLjhkIGfc+gILPvyM20/blcenLWLmwk9Wpem9UUdO/mJfTrpxIh8tXcGG666zat9nK2o59rrnSlH1slfGMc6TZuZj+n9eYPPe/ejWqw/rrNOOfQ8+kmceHb9amk6d11v1eumST8u4UV/5Bvfowuz3ljB38VJW1AbjX17AkIFdV0tz1M49GDNhDh8tXQHA4k+Wl6KqrY5y/FcKbsnl4d135rNJt+6r1rtutjnTXnp+jXQPjb6JP//pD6xYvpzLb7xn1fYFc9/ie0fvT6d1O/ONs85n8M57tEi926pNu3Tg7Q8/f4rdgg8/4ws9u6yWps/GnQC45eSdqaoS1z02k3/PSB5B3K6mittP35XalcHNT73Jo68uarnKlzGRtJLLVdGCnKSbgMOAdyJicLHKKamINTY1NBvDYcNP5rDhJ/PYw/cx5vqrOefy37HRJptx8yOT6LLBRsyY8iKXff9bXPvnx1dr+VlhNfgoqHqHsLpK9N6oI6fe8jybdmnPzSfvzNevfZaPlq7g4Kv/xcKPltFjww7c8M2deG3BJ8xZvKRF6l7eStdKy0Uxu6u3AEOLmH/JbbxZdxa+/fnDhBYtmM9Gm3ZrNP2+Bx/JM//8KwDrtGtPlw02AmDLbbenW68+zJ31enEr3MYt+HAp3bp0WLW+WZf2LPzoszXSPDZtEStWBvPeX8qbiz6l90YdAVj40TIA5i5eysQ3F7P15v6DBKy6Ti6XpRSKFuQi4gngvWLlXw62GrwD82bN5O05s1i+fBlP/OXP7D7kwNXSzJ01c9XrCU/8ne69+wHwwXuLqK2tBeDt2bOY99YbdOvZp+Uq3wZNmfcRvTfuRPcNOlBTLQ4avBmPT1u9y/noqwvZte+GAGzQaR36bNyJOYuXsF6HGtap1qrtO/TaYLUBi7asbnQ1lyVrXtJQSdMkzZB0fgP7z5D0H0mTJT0laVC2PEt+Ti59gvbpAJts3rPEtWme6poazvjxzxl5xnBW1tZywFeH02fLrbntmisYsO0O7P6lg3ho9E28+MwTVNesQ+cu63P25b8F4OVJz3D7qCupqq6hurqa7158Jeutv2GJP1Flq10Z/HLcNH5/wo5UCR54YT6vL/yEM7+0BVPnfcjj0xbx7xnvsWf/jbn3u3uwcmVw9d9m8MGSFWzfa30uOmxrVkZQJXHTU286yGUoRCNNUjUwCjiA5BmsEySNjYipGcnuiIjr0vSHA78iS49R0cB5pUKR1Bd4KNdzcgO23T5+PeaRotXHCu/Ce14udRWsGaZf/x0+nTetoB3Hbb6wY9z850dzSrvnlhtOiohdGtonaU/gkog4KF2/ACAiftFI+uHAiRFxcFNllrwlZ2atXzMGHrpKynwY9PURcX36ugcwO2PfHGD3NcqSvgucA7QDvpytQAc5M8tbMwYVFjXWkqORAfA1NkSMAkZJOg64CPhmUwUWbeBB0mjgaWCgpDmSTilWWWZWWspxyWIO0CtjvScwr5G0AHcCR2bLtGgtuYgYXqy8zax8iII9rWsCMEBSP2AuMAw4brWypAER8Vq6eijwGlm4u2pm+SnQNXARsULSCGA8UA3cFBFTJF0KTIyIscAISfsDy4HFZOmqgoOcmRVAoYZrI2IcMK7etpEZr7/f3Dwd5Mwsf+V7V5eDnJnlq7zvXXWQM7O8tNlZSMysDXGQM7NK5u6qmVW0cp7+3EHOzPJWxjHOQc7M8pTjPVul4iBnZnlJRlfLN8o5yJlZ3so3xDnImVkhlHGUc5Azs7z5EhIzq2hlfErOQc7M8lfGMc5BzszyU8BJM4vCQc7M8lPCB0fnwkHOzPJWxjHOQc7MCqCMo5yDnJnlyZNmmlkFK/dJM4v23FUza0MK9OBVSUMlTZM0Q9L5Dew/R9JUSS9J+oekPtnydJAzs7wpx39N5iFVA6OAg4FBwHBJg+olewHYJSK2A+4BrsxWNwc5M8ublNuSxW7AjIiYGRHLgDuBIzITRMSjEfFpuvoM0DNbpg5yZpa3ZvRWu0qamLGcnpFND2B2xvqcdFtjTgH+kq1uHngws/w072LgRRGxS+M5rSEaTCh9A9gF2C9bgQ5yZpaXAt7WNQfolbHeE5i3RnnS/sCFwH4R8Vm2TN1dNbO8FWhwdQIwQFI/Se2AYcDY1cqRdgT+ABweEe/kUje35Mwsb4VoyEXECkkjgPFANXBTREyRdCkwMSLGAlcBnYG709bjWxFxeFP5OsiZWd4KdcdDRIwDxtXbNjLj9f7NzdNBzszyV8Z3PDjImVneyjjGOciZWX4kP5LQzCpd+cY4Bzkzy18ZxzgHOTPLXxn3Vh3kzCxfnjTTzCpYcltXqWvROAc5M8ubg5yZVTR3V82scvm5q2ZWyXKcYaRkHOTMLH9lHOUc5Mwsb76ty8wqWvmGOAc5MyuEMo5yDnJmlrdyvoREEQ0+DKckJC0EZpW6HkXQFVhU6kpYs1TqMesTEZsUMkNJfyX5eeViUUQMLWT52ZRVkKtUkiY28Rg2K0M+ZpXDT+sys4rmIGdmFc1BrmVcX+oKWLP5mFUIn5Mzs4rmlpyZVTQHOTNh+5KvAAAEkklEQVSraA5yZlbRHOSKRNJASXtKWkdSdanrY7nxsao8HngoAklHAT8H5qbLROCWiPiwpBWzRknaKiKmp6+rI6K21HWywnBLrsAkrQMcC5wSEV8BHgB6AedJ6lLSylmDJB0GTJZ0B0BE1LpFVzkc5IqjCzAgfX0/8BDQDjhOKuOJt9ogSesCI4AfAMsk3QYOdJXEQa7AImI58CvgKEn7RMRK4ClgMvDFklbO1hARnwAnA3cA5wIdMgNdKetmheEgVxxPAo8AJ0jaNyJqI+IOoDuwfWmrZvVFxLyI+DgiFgHfBjrWBTpJO0naurQ1tHx4PrkiiIilkm4HArgg/SX5DNgMmF/SylmTIuJdSd8GrpL0KlANfKnE1bI8OMgVSUQslnQDMJWkdbAU+EZELChtzSybiFgk6SXgYOCAiJhT6jrZ2vMlJC0gPYEd6fk5K3OSNgTuAn4YES+Vuj6WHwc5swZI6hARS0tdD8ufg5yZVTSPrppZRXOQM7OK5iBnZhXNQc7MKpqDXCsiqVbSZEkvS7pbUqc88hoi6aH09eGSzm8i7QaSvrMWZVwi6dxct9dLc4ukrzejrL6SXm5uHa3yOci1LksiYoeIGAwsA87I3KlEs49pRIyNiF82kWQDoNlBzqwcOMi1Xk8CW6YtmFckXQs8D/SSdKCkpyU9n7b4OgNIGirpVUlPAUfVZSTpJEnXpK83k3S/pBfTZS/gl0D/tBV5VZruR5ImSHpJ0k8z8rpQ0jRJfwcGZvsQkk5L83lR0r31Wqf7S3pS0vR0OiQkVUu6KqPsb+f7g7TK5iDXCkmqIbnl6D/ppoHAnyJiR+AT4CJg/4jYiWTCznMkdQBuAP4L2Afo1kj2vwUej4jtgZ2AKcD5wOtpK/JHkg4kmUpqN2AHYGdJ+0raGRgG7EgSRHfN4ePcFxG7puW9ApySsa8vsB9wKHBd+hlOAT6IiF3T/E+T1C+HcqyN8r2rrUtHSZPT108CN5LMbDIrIp5Jt+8BDAL+lU5d1w54GtgaeCMiXgNIZ9k4vYEyvgycCKumGvogvc0p04Hp8kK63pkk6K0H3B8Rn6ZljM3hMw2WdBlJl7gzMD5j313prXCvSZqZfoYDge0yztetn5Y9PYeyrA1ykGtdlkTEDpkb0kD2SeYm4G8RMbxeuh1IZkUpBAG/iIg/1CvjB2tRxi3AkRHxoqSTgCEZ++rnFWnZZ0VEZjBEUt9mlmtthLurlecZYG9JWwJI6iRpK+BVoJ+k/mm64Y28/x/Amel7q9Mp2z8iaaXVGQ+cnHGur4ekTYEngK9K6ihpPZKucTbrAfPTaeOPr7fvaElVaZ23AKalZZ+ZpkfSVunsvmYNckuuwkTEwrRFNFpS+3TzRRExXdLpwMOSFpHMVjy4gSy+D1wv6RSgFjgzIp6W9K/0Eo2/pOfltgGeTluSH5NMI/W8pDEksyDPIulSZ3Mx8Gya/j+sHkynAY+TzMN3RjpP3/+RnKt7XknhC4Ejc/vpWFvkG/TNrKK5u2pmFc1BzswqmoOcmVU0Bzkzq2gOcmZW0RzkzKyiOciZWUX7f9SYSrIldKNDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot confusion matrix (normalized) with the best model \n",
    "plot_confusion_matrix(cm_best, classes=[0,1],normalize=True,\n",
    "                      title='Confusion matrix, with normalization')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "auROC = 0.7650765407927385\n"
     ]
    }
   ],
   "source": [
    "# calculate auROC\n",
    "auroc_test_best = roc_auc_score(test_values, y_classes_best)\n",
    "print('auROC =', auroc_test_best)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model with lowest validation loss had a higher sensitivity (65%) and lower specificity (88%) compared to the final model. The area under ROC was similar. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison between models\n",
    "***\n",
    "When we looked at the normalized confusion matrix, we found that this model gave higher sensitivity on class 1 samples but lower specificity on class 0 samples compared to the final model. Since the validation data was also sampled in a class-balance way, it is biased toward the balanced statistics. But when we trained the model for longer, it learned the closer-to-real statistics of the original data, so it did well on the test set with original class values. The area under ROC metric for both models were quite similar. We think that both models were fine. The final model might give better accuracy on predicting playlist-track contingency, thus a talored recommendation to individual playlists, whereas the model with lowest validation loss had higher \"false positive rate\", which would generate a more extended list of recommendation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}