04-Model-Comparison-Word2vec-Doc2vec-TfIdfWeighted.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Performance of Different Word Embeddings on Text Classification\n",
    "\n",
    "## Compared among word2vec, TF-IDF weighted, GloVe and doc2vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from numpy import random\n",
    "import gensim\n",
    "import spacy\n",
    "import nltk\n",
    "import os\n",
    "import pickle\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import re\n",
    "import warnings\n",
    "warnings.simplefilter('ignore')\n",
    "pd.set_option('max_colwidth',1000)\n",
    "\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Incentive\n",
    "\n",
    "It's been a while not able to write new posts, so sad, but now finally I am back again to share some of the knowledge I've just acquired. This time is about NLP. \n",
    "\n",
    "As a fresh rookie in NLP, I'd like to play around and test out how different methods of creating doc vector perform on text classification. This post will be highly focused on feature engineering side, that is word vectorization, and less on modeling. Thus, without further due, let's get started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Brief Introduction\n",
    "\n",
    "The word embeddings being investigated here are word2vec, TF-IDF weighted word2vec, pre-train GloVe word2vec and doc2vec. The packages needed are Gensim, Spacy and Scikit-Learn. Spacy is used in doc preprocessing, including stop word removal and custom token selection based on its part of speech. Gensim is heavily applied for training word2vec and doc2vec, and lastly, Scikit-Learn is for classifier building and training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick Summary\n",
    "\n",
    "After a series of comparison on different word embedding/averaging methods, it turns out that custom-trained word embedding and its averaging method, either simple mean or TF-IDF weighted has the best performance, while on the contrary, GloVe word embedding or custom-trained Doc2vec perform slightly worse than the former word embedding.\n",
    "\n",
    "Besides, even if we try to concatenate both word2vec and doc2vec as a whole feature set, it performs equally the same to just using averaging word embedding alone. In other words, no need to use both word2vec and doc2vec at the same time.\n",
    "\n",
    "\n",
    "| WordEmbedding Method        | F1 Score - Training | F1 Score - Testing | Accuracy - Training | Accuracy - Testing |\n",
    "| :---:                       | :---:               | :---:              | :---:               | :---:              |\n",
    "| Mean Word2vec               | 0.82                | 0.81               | 0.82                | 0.81               |\n",
    "| Tf-Idf Mean Word2vec        | 0.82                | 0.81               | 0.82                | 0.81               |\n",
    "| GloVe Mean Word2vec         | 0.72                | 0.71               | 0.73                | 0.72               |\n",
    "| PV-DM Doc2vec               | 0.79                | 0.78               | 0.79                | 0.78               |\n",
    "| Tf-Idf Word2vec + Doc2vec   | 0.84                | 0.81               | 0.85                | 0.82               |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Special Credits to the Following Posts and Authors\n",
    "\n",
    "In creating my **python class object** used for text preprocessing, I referred from these well-written posts.\n",
    "\n",
    "* The post [\"Text Classification with Word2vec\"](http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/) by nadbor demos how to write your own class to compute average word embedding for doc, either simple averaging or TF-IDF weighted one.\n",
    "\n",
    "* [\"Multi-Class Text Classification Model Comparison and Selection\"](https://towardsdatascience.com/multi-class-text-classification-model-comparison-and-selection-5eb066197568) by Susan Li teaches me how to write beautiful averaging function for word embedding.\n",
    "\n",
    "* This tutorial [\"Gensim Doc2vec Tutorial on the IMDB Sentiment Dataset\"](https://github.com/RaRe-Technologies/gensim/blob/3c3506d51a2caf6b890de3b1b32a8b85f7566ca5/docs/notebooks/doc2vec-IMDB.ipynb) has step by step guidance on how to create doc2vec via Gensim.\n",
    "\n",
    "* [\"Distributed representations of sentences and documents\"](https://blog.acolyer.org/2016/06/01/distributed-representations-of-sentences-and-documents/) by Le & Mikolov presents a clear and easy-to-understand explanation on what's going under doc2vec."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset I am gonna use here is consumer complaints dataset on financial product/service as referred from [the post](https://towardsdatascience.com/multi-class-text-classification-with-doc2vec-logistic-regression-9da9947b43f4). The dataset is collected and published by [US GOV CFPB](https://catalog.data.gov/dataset/consumer-complaint-database), while we can also download the dataset from [Kaggle](https://www.kaggle.com/cfpb/us-consumer-finance-complaints).\n",
    "\n",
    "The original dataset contains more than 500 thousands records, and columns include product, sub_product, issue, consumer_complaint_narrative, and company_response_to_consumer etc.. We will just use **product** as text label and **consumer_complaint_narrative** as text itself. After dropping rows of missing values on consumer complaint we are left with around 60 thousands records. In order to lessen the computing pressure, I will just experiment on the first 25 thousands records only."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in file.\n",
    "dir_path = '../Private-Dataset'\n",
    "\n",
    "df = pd.read_csv(os.path.join(dir_path, 'consumer_complaints_simple.csv'))\n",
    "df = df[pd.notnull(df['consumer_complaint_narrative'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Familiar with Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(66806, 18)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get familiar with dataset.\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 66806 entries, 0 to 66805\n",
      "Data columns (total 18 columns):\n",
      "date_received                   66806 non-null object\n",
      "product                         66806 non-null object\n",
      "sub_product                     46351 non-null object\n",
      "issue                           66806 non-null object\n",
      "sub_issue                       32932 non-null object\n",
      "consumer_complaint_narrative    66806 non-null object\n",
      "company_public_response         32776 non-null object\n",
      "company                         66806 non-null object\n",
      "state                           66620 non-null object\n",
      "zipcode                         66617 non-null object\n",
      "tags                            11417 non-null object\n",
      "consumer_consent_provided       66806 non-null object\n",
      "submitted_via                   66806 non-null object\n",
      "date_sent_to_company            66806 non-null object\n",
      "company_response_to_consumer    66806 non-null object\n",
      "timely_response                 66806 non-null object\n",
      "consumer_disputed?              66806 non-null object\n",
      "complaint_id                    66806 non-null int64\n",
      "dtypes: int64(1), object(17)\n",
      "memory usage: 9.7+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(os.path.join(dir_path, 'consumer_complaints_simple.csv'), index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter on Text and Label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter on text and label.\n",
    "df = df[['product','consumer_complaint_narrative']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Experiment on smaller subset.\n",
    "df = df[:25000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>**Chart of Label Frequency**</mark>\n",
    "\n",
    "Now, let's see how frequency distributed among each label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Distribution of target variable.\n",
    "display(df['product'].value_counts())\n",
    "\n",
    "# df[['product', 'sub_product']].apply(lambda x: str(x[0]) + '///' + str(x[1]),\n",
    "#                                      axis=1).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frequency(Top 30)...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>freq</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Debt collection</th>\n",
       "      <td>7093</td>\n",
       "      <td>0.28372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Mortgage</th>\n",
       "      <td>5703</td>\n",
       "      <td>0.22812</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Credit reporting</th>\n",
       "      <td>4584</td>\n",
       "      <td>0.18336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Credit card</th>\n",
       "      <td>2750</td>\n",
       "      <td>0.11000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bank account or service</th>\n",
       "      <td>1923</td>\n",
       "      <td>0.07692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Consumer Loan</th>\n",
       "      <td>1292</td>\n",
       "      <td>0.05168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Student loan</th>\n",
       "      <td>917</td>\n",
       "      <td>0.03668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Payday loan</th>\n",
       "      <td>273</td>\n",
       "      <td>0.01092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Money transfers</th>\n",
       "      <td>270</td>\n",
       "      <td>0.01080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Prepaid card</th>\n",
       "      <td>154</td>\n",
       "      <td>0.00616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Other financial service</th>\n",
       "      <td>41</td>\n",
       "      <td>0.00164</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         count     freq\n",
       "Debt collection           7093  0.28372\n",
       "Mortgage                  5703  0.22812\n",
       "Credit reporting          4584  0.18336\n",
       "Credit card               2750  0.11000\n",
       "Bank account or service   1923  0.07692\n",
       "Consumer Loan             1292  0.05168\n",
       "Student loan               917  0.03668\n",
       "Payday loan                273  0.01092\n",
       "Money transfers            270  0.01080\n",
       "Prepaid card               154  0.00616\n",
       "Other financial service     41  0.00164"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA38AAAIqCAYAAACUg/N/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XtYlHX+//EXB/GEgro2muKhkMDE1MzME4UiLm6iQn3LtlrLQ5uWtbbaEc3VDpvlIUw081BmKZWa4WqGlnho19QNTSrFMDRFy1BTERnu3x/9mHVCayaHGfDzfFyX18Xc9+f+zPt+AyOvuQ/jZ1mWJQAAAADAJc3f1wUAAAAAACoe4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAwHDp6emKjo6usPmnTJmiPn36VNj8AADXBPq6AACA5zz66KNaunRpueUvvfSS+vbt64OKcLF69OihgoKCcst79uypV155xQcVOTt16pTmzp2r1atX69tvv1WNGjXUvHlzJSYmKikpSTVq1PB1iQCA/4/wBwCXmI4dO2rq1KlOy+rWrXvesWfPnlW1atW8URZ+RWlpqSzLUkBAwHnX33ffffrzn//stKx69ereKO1XHT9+XHfccYd+/PFHPfjgg2rbtq1q166tHTt26PXXX9fll1+um266yddlAgD+P077BIBLTLVq1dSwYUOnf2VB4ZFHHtG9996r+fPn66abblJ0dLTOnj0ry7K0YMECxcfHKzo6WvHx8Zo1a5ZKSkoc85b9gd+uXTt16dJF06dPd8xX5vbbb1dKSopTPS+//LLi4uKclr3//vvq16+foqOjFRsbq+eff16nT58uN09qaqq6dOmiTp066bHHHtOpU6ec5lmxYoX69++v6OhoXX/99Ro6dKh++uknpaenq1OnTjpz5ozT+GnTpv3q6YeWZWn27NmKjY1VmzZt1KtXL73xxhuO9S+88IISEhLKbffkk0/qzjvvdDzOzs7W4MGD1b59e91www168MEHdfDgQcf6stMgP/jgA8XHx6tNmzbKz8+/YF21a9cu9z09N9DPnz9f/fr1U/v27dWtWzeNHj1aR44ccZojLy9PDzzwgDp16qRrrrlG/fr10yeffOI05rPPPlNiYqKuueYaJSUlaefOnResSZJefPFF5efna8mSJbr11lsVGRmpsLAwJSQk6K233lLHjh3Pu923336rESNGqFu3brrmmmt08803a8WKFU5jtmzZottuu03t27dXhw4dlJiYqE2bNjnWz5gxw/F9uuGGG3TvvfequLj4V+sFANMR/gDAMNu2bdO2bds0c+ZMLVu2TAEBAZo6daoWLFigv//971q5cqUee+wxvfnmm0pLS3Ns99hjj+mrr75SWlqa5s+fr7y8PK1du9bt509PT9ekSZN07733auXKlXruueeUlZWlCRMmOI1buXKlTp48qYULF2ry5Mlas2aN5s6d61i/ZMkSPfroo4qPj9eyZcu0YMECde3aVSUlJerbt6/sdrtWr17tGG+32/Xee+/p1ltvvWBtr7/+umbMmKG//vWv+uCDDzR48GA9//zzjlNpBwwYoNzcXKdQdObMGa1atUqJiYmSpK+++kp33XWXrr32Wr3zzjuaN2+eLMvS4MGDncLJoUOHtHjxYr3wwgv64IMP1LBhQ7d7ea7HHntM77//vqZPn678/Hw98sgjjnUFBQW6/fbbdfLkSaWlpWnFihV68MEH5e//vz8DSkpKNHXqVKWkpOi9995TSEiIHn74Ydnt9vM+n91uV0ZGhvr166fLL7+83Ho/Pz/VqVPnvNueOnVKXbp00Zw5c7RixQolJSVpzJgx2rJli6Sfj0jfd999at++vZYtW6b33ntPI0aMcLyJsXLlSs2dO1cpKSn68MMPNXfuXHXv3v139w4AjGEBAC4ZY8eOtaKioqx27do5/vXu3duxfvTo0dZ1111nnTp1yrHsp59+sqKjo62NGzc6zZWenm516tTJsizL2rNnjxUREWFt3rzZsb6oqMjq0qWLdc899ziW3XbbbdZTTz3lNM/06dOtXr16OR53797dWrJkidOYTZs2WVdddZV14sQJxzz9+/d3GvPEE09Yt99+u2VZllVaWmp17drVmjhx4gV7MW7cOOvPf/6z4/G6deusq6++2vrhhx8uuE3Xrl2tF1980WnZhAkTnHo4YMAA6x//+Ifj8QcffGBdc801jtpHjx5tjR492mmO06dPW1dffbW1du1ay7Is66WXXrIiIyOtQ4cOXbCWMt27d7euvvpqp+9pu3btrLlz515wm88//9yKiIiwjhw5YlmWZU2ePNnq2rWr0/f9XEuWLLEiIiKsnJwcx7LPPvvMioiIsPbt23febQ4dOmRFRERYCxYs+M19eOmll6z4+PhfHTN06FArJSXFsizL+uGHH6yIiAhry5Yt5x376quvWn369LGKi4t/87kBAP/DNX8AcIlp27atnn/+ecfjX15HFh4erpo1azoef/311zpz5ozuv/9++fn5OZbb7XadOXNGx44dU25urvz8/NS+fXvH+urVq6tNmzZOp4b+lsOHD6ugoEATJ07UM88841huWZYsy9K3336r1q1bS5IiIyOdtr3sssscR4YOHz6sI0eOqFu3bhd8rttuu02JiYn65ptv1LJlSy1ZskRxcXGqX7/+eccXFhbqyJEj5U5VvO6667Ro0SIVFxcrKChIAwYM0MyZM/Xoo48qMDBQy5YtU8+ePRUcHCxJ2rFjhw4cOKDMzEyneUpKSrRv3z6n/bHZbL/VMknSnXfeqdtuu81pWb169Rxfb968Wa+++qpyc3N1/PhxWZYlSfruu+/0hz/8QV988YWuvfZap+/7LwUGBioiIsKpPkn6/vvv1axZM5fqdNWpU6c0Y8YMffzxxzpy5IjOnj2r4uJidenSRZJUv359DRw4UIMHD1bnzp113XXXqXfv3mrRooUkKSEhQW+++aZiY2PVrVs3de7cWb169VLt2rU9WicAXGoIfwBwiSm72+KF/DIAlJaWSpJSU1MVFhZWbnxZqHHFuacRljk3HJaFkqeeekrXXXddubGNGzd2fP3LG9H4+fk5anVFZGSk2rVrp/T0dA0ePFiffPKJ5syZ4/L2F9K3b189//zzysrKUnR0tDZu3KjZs2c71luWpYEDBzpdC1nm3MD2a0HsfNtd6Huan5+v4cOHa8CAARo5cqRCQ0P13XffuX0NnL+/v9P3r+yNgAv1/A9/+IPq1KmjPXv2uPwcZZ577jmtX79eY8eOVcuWLVWzZk0988wzTtdoPvvss/rLX/6ijRs3auPGjZo+fbrGjRunW265RZdffrlWrVqlTz/9VJ9++qlmzJihF198Uenp6S4HagAwEdf8AYDhrrrqKgUFBSk/P1/Nmzcv9y8gIEBXXnmlLMvS9u3bHdsVFxfriy++cJqrQYMGOnz4sNOyc8dcdtllatiwofLy8s77XEFBQS7VXDbPhg0bfnXc//3f/2nZsmVavHixmjRpos6dO19wbGhoqBo2bKjPPvvMafmWLVvUrFkzR23169dXjx49tHz5cq1YsUINGjRwHLGSpDZt2uirr7467/5d6K6rF2PHjh0qLi7WE088oQ4dOuiKK67Q999/7zTm6quv1rZt25xuqnOxAgIClJCQoPfff1/fffddufWWZenEiRPn3XbLli1KTEzUH//4R0VGRqpp06bKy8srN+6qq67SPffco9dee02JiYlasmSJY1316tUVExOjsWPHasWKFTpx4kS5o60AAGeEPwAwXHBwsIYOHarJkydr0aJF+uabb7R7926tWLFCL774oiTpyiuvVExMjMaPH6///Oc/2r1793nvvnnDDTdow4YNWr16tfLy8pSWlqb//ve/jvV+fn56+OGHNX/+fM2aNUu7d+/W3r17tWbNGo0bN87lmv38/DRixAgtWrRIaWlpys3N1e7du/X666+rsLDQMS4hIUFnz55VWlqakpOTnU5rPZ9hw4ZpwYIFSk9PV15enhYtWqTFixfrvvvucxrXv39/rV27VkuWLFG/fv2cjpjdd999+vrrrzVmzBhlZ2crPz9fmzdv1j/+8Q8dOHDA5X0818mTJ3XkyBGnfz/++KMkqUWLFrIsS/PmzVN+fr7WrFmjmTNnOm3/5z//WcXFxRo5cqS2bdum/Px8rV27VllZWb+rnjKjR49W06ZNdeuttyo9PV1ffvml8vPztXr1ag0aNKhckC7TsmVLffTRR8rOztbu3bv15JNPOgXWvXv36sUXX9TWrVt14MABx02KwsPDJf18s5+y5ztw4ICWL1+u06dPO9YDAM6P0z4BAHrwwQdls9n05ptv6plnnlHNmjXVsmVLDRw40DHmueee07hx4zR06FDVqlVLt956q2JjYx0hRJKSk5O1Z88ejR8/XiUlJUpMTNQdd9yhlStXOsYkJSWpTp06mjNnjmbMmKHAwEA1a9ZMvXv3dqvm22+/XTVr1tRrr72m1NRU1a5dW+3atXOquUaNGurXr58WL16spKSk35zzzjvvVFFRkWbOnKmnn35ajRo10pgxYzRgwACncTfeeKNq1qypvXv3avr06U7rIiIi9NZbb2nq1Km65557VFxcLJvNphtuuOGCd7/8LWlpaU53XpV+Pq11+fLlat26tZ544glHP6Ojo/X4449r2LBhjrE2m02LFi3S5MmTNXToUJWUlKhFixYaPXr076qnTEhIiBYvXqy5c+dqwYIFjg95b9Gihfr37+90RPRcTzzxhJ588kndddddCg4O1m233aa4uDgdOnRI0s8fbbF3714tW7ZMP/74o+rVq6ebbrpJY8aMkSTVqVNH8+fP1/PPP6+zZ8+qWbNmeuaZZ9SpU6eL2h8AuNT5WWUXYAAA4KZHHnlEP/74o1577TVfl3JBI0eOlL+/f7mQBgCAaTjyBwC4JBUWFurzzz/X2rVrtXDhQl+XAwCAzxH+AACXpH79+unEiRMaPny4OnTo4OtyAADwOU77BAAAAAADcLdPAAAAADDAJXXa59atW31dAgAAAAD41LXXXnve5ZdU+JMuvKO+lJOTo6ioKF+XUWXQL/fQL/fQL/fQL/fQL/fRM/fQL/fQL/fQL/dU1n792gExTvsEAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwQKCvC6ishmct8uyEWds9Ms2s7oM8Mg8AAAAAs3DkDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAwQ6M0nW79+vSZNmqTS0lLdcsstGjZsmNP6efPmKT09XQEBAapfv76eeeYZNWnSRJIUFRWliIgISVLjxo2VlpbmzdIBAAAAoErzWviz2+2aMGGC5s2bJ5vNpuTkZMXGxio8PNwxJioqSu+++65q1qypRYsW6YUXXtDUqVMlSTVq1NDy5cu9VS4AAAAAXFK8dtpndna2mjdvrrCwMAUFBalv377KzMx0GtO5c2fVrFlTktSuXTsdOnTIW+UBAAAAwCXNa0f+CgoK1KhRI8djm82m7OzsC45/55131KNHD8fjM2fOaODAgQoMDNSwYcPUq1ev826Xk5PjuaIroUt9/ySpqKjIiP30FPrlHvrlHvrlHvrlPnrmHvrlHvrlHvrlnqrYL69e8+eq5cuXa+fOnVq4cKFj2bp162Sz2ZSfn6+7775bERERatasWblto6KiPFNE1nbPzONhHtu/SiwnJ8eI/fQU+uUe+uUe+uUe+uU+euYe+uUe+uUe+uWeytqvrVu3XnCd1077tNlsTqdxFhQUyGazlRu3adMmpaWlaebMmQoKCnLaXpLCwsLUqVMn7dq1q+KLBgAAAIBLhNfCX3R0tPLy8pSfn6/i4mJlZGQoNjbWacyuXbuUkpKimTNnqkGDBo7lx44dU3FxsSTp6NGj2rZtm9ONYgAAAAAAv85rp30GBgYqJSVFQ4YMkd1uV1JSklq1aqVp06apTZs26tmzp/75z3/q1KlTGjVqlKT/faRDbm6uxo0bJz8/P1mWpaFDhxL+AAAAAMANXr3mLyYmRjExMU7LyoKeJM2fP/+823Xo0EErVqyoyNIAAAAA4JLmtdM+AQAAAAC+Q/gDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAME+roAXBqGZy3y7IRZ2z0yzazugzwyDwAAAFDVceQPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAzg1fC3fv16xcfHKy4uTrNnzy63ft68eUpISNDNN9+su+++WwcOHHCsW7p0qXr37q3evXtr6dKl3iwbAAAAAKo8r4U/u92uCRMmaM6cOcrIyNAHH3ygPXv2OI2JiorSu+++qxUrVig+Pl4vvPCCJKmwsFCpqalasmSJ0tPTlZqaqmPHjnmrdAAAAACo8rwW/rKzs9W8eXOFhYUpKChIffv2VWZmptOYzp07q2bNmpKkdu3a6dChQ5KkDRs2qGvXrgoNDVVISIi6du2qrKwsb5UOAAAAAFVeoLeeqKCgQI0aNXI8ttlsys7OvuD4d955Rz169LjgtgUFBefdLicnx0MVV06X+v55mgn9KioqMmI/PYV+uYd+uYd+uY+euYd+uYd+uYd+uacq9str4c8dy5cv186dO7Vw4UK3t42KivJMEVnbPTOPh3ls/zyNfvlMTk6OEfvpKfTLPfTLPfTLffTMPfTLPfTLPfTLPZW1X1u3br3gOq+d9mmz2RyncUo/H82z2Wzlxm3atElpaWmaOXOmgoKC3NoWAAAAAHB+Xgt/0dHRysvLU35+voqLi5WRkaHY2FinMbt27VJKSopmzpypBg0aOJZ369ZNGzZs0LFjx3Ts2DFt2LBB3bp181bpAAAAAFDlee20z8DAQKWkpGjIkCGy2+1KSkpSq1atNG3aNLVp00Y9e/bUP//5T506dUqjRo2SJDVu3FhpaWkKDQ3V/fffr+TkZEnSiBEjFBoa6q3SAQAAAKDK8+o1fzExMYqJiXFaVhb0JGn+/PkX3DY5OdkR/gAAAAAA7vHqh7wDAAAAAHyD8AcAAAAABiD8AQAAAIABCH8AAAAAYADCHwAAAAAYgPAHAAAAAAYg/AEAAACAAQh/AAAAAGAAwh8AAAAAGIDwBwAAAAAGIPwBAAAAgAEIfwAAAABgAMIfAAAAABiA8AcAAAAABiD8AQAAAIABCH8AAAAAYADCHwAAAAAYgPAHAAAAAAYg/AEAAACAAQJ9XQBgouFZizw7YdZ2j0wzq/sgj8wDAACAyocjfwAAAABgAMIfAAAAABiA8AcAAAAABiD8AQAAAIABCH8AAAAAYADCHwAAAAAYgPAHAAAAAAYg/AEAAACAAQh/AAAAAGAAwh8AAAAAGIDwBwAAAAAGIPwBAAAAgAEIfwAAAABgAMIfAAAAABiA8AcAAAAABiD8AQAAAIABXA5/8fHxeu2113T06NGKrAcAAAAAUAFcDn8333yz3nzzTcXExGjUqFHavHlzRdYFAAAAAPAgl8PfyJEjlZmZqVdeeUWWZWno0KHq1auXZs+ere+//74iawQAAAAAXCS3rvnz8/NT9+7dNX36dK1fv15JSUlKTU3VjTfeqAceeEBbt26tqDoBAAAAABfhd93wZe/evXr11Ve1YMEC1apVS3fccYcCAgJ09913a9q0aZ6uEQAAAABwkQJdHXjmzBn961//Unp6urZt26YOHTro8ccfV58+fRQUFCRJysrK0kMPPaRRo0ZVWMEAAAAAAPe5HP66desmf39/3XzzzXr66acVHh5ebsw111yjkJAQjxYIAAAAALh4Loe/xx9/XAkJCapevfoFx9StW1dr1671SGEAAAAAAM9x+Zq/Xr166fTp0+WWFxYW6qeffvJoUQAAAAAAz3I5/I0ePVorVqwotzwjI0OPPPKIR4sCAAAAAHiWy+Hv888/1/XXX19ueadOnfTf//7Xo0UBAAAAADzL5fB3+vRpBQQElJ/A31+nTp3yaFEAAAAAAM9yOfyFh4drzZo15ZavXr1aV1xxhUeLAgAAAAB4lst3+xwyZIjGjBmj77//Xl27dpUkbdy4UW+//baee+65CisQAAAAAHDxXA5/CQkJOn36tFJTU7Vw4UJJUqNGjTRu3Dj96U9/qrACAQAAAAAXz+XwJ0lJSUlKSkrS0aNHJUn169evkKIAAAAAAJ7lVvgrQ+gDAAAAgKrF5fB3+vRpvfrqq9q0aZN++OEHlZaWOq3PzMz0eHEAAAAAAM9wOfyNHz9eH3/8sf74xz/qsssuk5+fX0XWBQAAAADwIJfD38cff6wpU6aoS5cuFVkPAAAAAKACuPw5f9WqVVPjxo0rshYAAAAAQAVxOfwNGjRIixcvrshaAAAAAAAVxOXTPr/77jt9+OGH+vTTTxUZGalq1ao5rf/HP/7h8eIAAAAAAJ7hcvjbt2+frrrqKknSgQMHnNZx8xcAAAAAqNxcDn9vvPFGRdYBAAAAAKhALl/zV+bEiRPauXOniouLK6IeAAAAAEAFcDn8nT59WmPHjtV1112nW265RQUFBZKklJQUvfLKKxVWIAAAAADg4rkc/qZNm6Yvv/xSCxcuVI0aNRzLe/TooTVr1lRIcQAAAAAAz3A5/H344Yd66qmn1LFjR6flV155pfLz812aY/369YqPj1dcXJxmz55dbv2WLVs0YMAAtW7dWqtWrXJaFxUVpcTERCUmJuq+++5ztWwAAAAAgNy44cv3339/3g95Lykpkd1u/83t7Xa7JkyYoHnz5slmsyk5OVmxsbEKDw93jGncuLGeffZZzZ07t9z2NWrU0PLly10tFwAAAABwDpeP/F1xxRXatm1bueWZmZmOj4D4NdnZ2WrevLnCwsIUFBSkvn37KjMz02lM06ZNFRkZKX9/t+9DAwAAAAD4FS4f+Rs6dKgmTJigU6dOSZL+/e9/66233tIbb7yhqVOn/ub2BQUFatSokeOxzWZTdna2y4WeOXNGAwcOVGBgoIYNG6ZevXqdd1xOTo7Lc1ZFl/r+eRr9co8J/SoqKjJiPz2FfrmHfrmPnrmHfrmHfrmHfrmnKvbL5fDXt29fnTlzRqmpqTp9+rSefPJJNW7cWBMnTlTPnj0rskZJ0rp162Sz2ZSfn6+7775bERERatasWblxUVFRnnnCrO2emcfDPLZ/nka/3EO/fCYnJ8eI/fQU+uUe+uU+euYe+uUe+uUe+uWeytqvrVu3XnCdy+FPkgYOHKiBAwfq6NGjsixLDRo0cHlbm82mQ4cOOR4XFBTIZrO5tb0khYWFqVOnTtq1a9d5wx8AAAAAoLzfdXFd/fr13Qp+khQdHa28vDzl5+eruLhYGRkZio2NdWnbY8eOOT5U/ujRo9q2bZvTjWIAAAAAAL/O5SN/sbGx8vPzu+D6X968pdwTBQYqJSVFQ4YMkd1uV1JSklq1aqVp06apTZs26tmzp7KzszVy5EgdP35c69at08svv6yMjAzl5uZq3Lhx8vPzk2VZGjp0KOEPAAAAANzgcvi75ZZbnB6XlJToiy++0NatW3XXXXe5NEdMTIxiYmKclo0aNcrxddu2bbV+/fpy23Xo0EErVqxwtVQAAAAAwC+4HP7++te/nnf57NmzVVBQ4LGCAAAAAACed9EfqNenTx+OygEAAABAJXfR4W/Xrl2qVq2aJ2oBAAAAAFQQl0/7fOqpp5weW5alw4cPa9OmTbrjjjs8XhgAAAAAwHNcDn95eXlOj/39/dWgQQONGzdOAwYM8HRdAAAAAAAPcjn8vfHGGxVZBwAAAACgAl30NX8AAAAAgMrP5SN/gwcP/tUPeT/X3Llzf3dBAAAAAADPczn82Ww2rVmzRrVr11Z0dLQkaceOHTp16pR69erlcjAEAAAAAHify+GvUaNGio2N1TPPPOP4aIezZ8/qySeflM1m08MPP1xhRQIAAAAALo7L1/y98847uu+C3QKEAAAgAElEQVS++5w+069atWoaOnSo0tPTK6Q4AAAAAIBnuBz+Tp48qR9++KHc8h9++EGnT5/2aFEAAAAAAM9yOfzFxMToqaee0ubNm1VUVKSioiJt3rxZ48aN04033liBJQIAAAAALpbL1/w9/fTTevzxx8vd9fOmm27S+PHjK6I2AAAAAICHuBz+QkJCNGPGDO3bt0+5ubmSpPDwcDVr1qzCigMAAAAAeIbL4a9M8+bNVbduXYWGhvLxDgAAAABQRbh8zZ/dbtf06dN13XXXqWvXrtq/f78kafLkyXr77bcrrEAAAAAAwMVzOfy9+uqrWrZsmZ544gmnj3to3bq1li5dWiHFAQAAAAA8w+Xwt3TpUo0fP179+/eXv///NouIiFBeXl5F1AYAAAAA8BCXw9/Bgwd15ZVXllseEBCgoqIijxYFAAAAAPAsl8NfkyZN9OWXX5ZbvmnTJl1xxRUeLQoAAAAA4Fku3+1z0KBBmjRpkqpXry5JysvL0/r16zVlyhQ9+uijFVYgAAAAAODiuRz+7rzzThUWFmrkyJEqKirS0KFDVb16dQ0fPlzJyckVWSMAAAAA4CK5HP6Ki4s1YsQIDR06VLt375ZlWQoPD1etWrUqsj4AAAAAgAe4dM1fSUmJ2rVrp9zcXNWoUUPR0dFq27YtwQ8AAAAAqgiXwl9gYKAaNWqk0tLSiq4HAAAAAFABXL7b5z333KPU1FQ+1gEAAAAAqiCXr/nLzMxUdna2evTooSuuuKLcKZ9z5871eHEAAAAAAM/41fC3bNkyJSQkKCgoSI0aNVKjRo28VRcAAAAAwIN+Nfw99thj6tGjh+rXr69ly5Zpw4YNatCggbdqAwAAAAB4yK9e81e/fn19/vnnkiTLsuTn5+eVogAAAAAAnvWrR/4GDhyov/71rwoICJCfn5969OhxwbE7d+70eHEAAAAAAM/41fA3evRoxcXF6ZtvvtHYsWM1ZswY1alTx1u1AQAAAAA85Dfv9tm2bVu1bdtWn376qQYOHKjg4GBv1AUAAAAA8CCXP+rh2Wefrcg6AAAAAAAVyOUPeQcAAAAAVF2EPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMECgrwsAgN8yPGuRZyfM2u6RaWZ1H+SReQAAALyBI38AAAAAYADCHwAAAAAYgPAHAAAAAAYg/AEAAACAAQh/AAAAAGAAwh8AAAAAGIDwBwAAAAAGIPwBAAAAgAEIfwAAAABgAMIfAAAAABjAq+Fv/fr1io+PV1xcnGbPnl1u/ZYtWzRgwAC1bt1aq1atclq3dOlS9e7dW71799bSpUu9VTIAAAAAXBICvfVEdrtdEyZM0Lx582Sz2ZScnKzY2FiFh4c7xjRu3FjPPvus5s6d67RtYWGhUlNT9e6778rPz08DBw5UbGysQkJCvFU+AAAAAFRpXjvyl52drebNmyssLExBQUHq27evMjMzncY0bdpUkZGR8vd3LmvDhg3q2rWrQkNDFRISoq5duyorK8tbpQMAAABAlee1I38FBQVq1KiR47HNZlN2dvbv3ragoOC8Y3Nyci6u0EruUt8/T6Nf7qFf7jGhX0VFRUbsp6fQL/fRM/fQL/fQL/fQL/dUxX55Lfx5S1RUlGcmytrumXk8zGP752n0yz30yz30y2dycnKM2E9PoV/uo2fuoV/uoV/uoV/uqaz92rp16wXXee20T5vNpkOHDjkeFxQUyGazVfi2AAAAAAAvhr/o6Gjl5eUpPz9fxcXFysjIUGxsrEvbduvWTRs2bNCxY8d07NgxbdiwQd26davgigEAAADg0uG10z4DAwOVkpKiIUOGyG63KykpSa1atdK0adPUpk0b9ezZU9nZ2Ro5cqSOHz+udevW6eWXX1ZGRoZCQ0N1//33Kzk5WZI0YsQIhYaGeqt0AAAAAKjyvHrNX0xMjGJiYpyWjRo1yvF127ZttX79+vNum5yc7Ah/AAAAAAD3ePVD3gEAAAAAvkH4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwQKCvCwAAeNbwrEWenTBru0emmdV9kEfmAQAAvw9H/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAM4NXwt379esXHxysuLk6zZ88ut764uFgPPfSQ4uLidMstt2j//v2SpP3796tt27ZKTExUYmKiUlJSvFk2AAAAAFR5gd56IrvdrgkTJmjevHmy2WxKTk5WbGyswsPDHWPS09NVt25drVmzRhkZGZo8ebKmTp0qSWrWrJmWL1/urXIBAIYYnrXIsxNmbffINLO6D/LIPAAAlPHakb/s7Gw1b95cYWFhCgoKUt++fZWZmek0Zu3atRowYIAkKT4+Xps3b5ZlWd4qEQAAAAAuWV478ldQUKBGjRo5HttsNmVnZ5cb07hx458LCwxUnTp19OOPP0r6+dTP/v37Kzg4WA899JA6dux43ufJycmpoD2oHC71/fM0+uUe+uUe+uUe+uUeU/pVVFRkzL56Av1yD/1yD/1yT1Xsl9fC38W47LLLtG7dOtWrV087d+7UiBEjlJGRoeDg4HJjo6KiPPOkHjptx9M8tn+eRr/cQ7/cQ7/cQ7/cQ798Kicnx5h99QT65R765R765Z7K2q+tW7decJ3XTvu02Ww6dOiQ43FBQYFsNlu5MQcPHpQklZSU6MSJE6pXr56CgoJUr149SVKbNm3UrFkzffPNN94qHQAAAACqPK+Fv+joaOXl5Sk/P1/FxcXKyMhQbGys05jY2FgtXbpUkrR69Wp17txZfn5+Onr0qOx2uyQpPz9feXl5CgsL81bpAAAAAFDlee20z8DAQKWkpGjIkCGy2+1KSkpSq1atNG3aNLVp00Y9e/ZUcnKy/v73vysuLk4hISGaMmWKJGnLli2aPn26AgMD5e/vr6efflqhoaHeKh0AAAAAqjyvXvMXExOjmJgYp2WjRo1yfF29enVNnz693Hbx8fGKj4+v8PoAAAAA4FLl1Q95BwAAAAD4BuEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAwQ6OsCAABA1TI8a5FnJ8za7pFpZnUf5JF5AOBSxZE/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIQ/AAAAADAA4Q8AAAAADED4AwAAAAADEP4AAAAAwACEPwAAAAAwAOEPAAAAAAxA+AMAAAAAAxD+AAAAAMAAhD8AAAAAMADhDwAAAAAMQPgDAAAAAAMQ/gAAAADAAIG+LgAAAOBSNjxrkWcnzNrukWlmdR/kkXkAVB2EPwAAAFQahGWg4nDaJwAAAAAYgPAHAAAAAAYg/AEAAACAAQh/AAAAAGAAwh8AAAAAGIC7fQIAAABVFHdHhTs48gcAAAAABiD8AQAAAIABCH8AAAAAYADCHwAAAAAYgPAHAAAAAAbwavhbv3694uPjFRcXp9mzZ5dbX1xcrIceekhxcXG65ZZbtH//fse6WbNmKS4uTvHx8crKyvJm2QAAAABQ5Xntox7sdrsmTJigefPmyWazKTk5WbGxsQoPD3eMSU9PV926dbVmzRplZGRo8uTJmjp1qvbs2aOMjAxlZGSooKBAgwcP1urVqxUQEOCt8gEAAABUcaZ/NIbXjvxlZ2erefPmCgsLU1BQkPr27avMzEynMWvXrtWAAQMkSfHx8dq8ebMsy1JmZqb69u2roKAghYWFqXnz5srOzvZW6QAAAABQ5flZlmV544lWrVqlrKwsTZo0SZK0bNkyZWdnKyUlxTHmT3/6k+bMmaNGjRpJknr16qUlS5YoNTVV11xzjRITEyVJjz/+uHr06KE+ffo4PcfWrVu9sSsAAAAAUGlde+21513utdM+veFCOwkAAAAApvPaaZ82m02HDh1yPC4oKJDNZis35uDBg5KkkpISnThxQvXq1XNpWwAAAADAhXkt/EVHRysvL0/5+fkqLi5WRkaGYmNjncbExsZq6dKlkqTVq1erc+fO8vPzU2xsrDIyMlRcXKz8/Hzl5eWpbdu23iodAAAAAKo8r532GRgYqJSUFA0ZMkR2u11JSUlq1aqVpk2bpjZt2qhnz55KTk7W3//+d8XFxSkkJERTpkyRJLVq1Up//OMflZCQoICAAKWkpHCnTwDndfr0adWsWdPXZQAAAFQ6XrvhCwD3WZYlPz8/X5dRZaxfv14bNmzQww8/TAAEUOXwmu8e+uWevLw8NW7cWNWrV/d1KVVCaWmp/P39L7mfM69+yPulprS01Ncl4BJmt9sdLzZbt24V79P8utLSUlWvXl2lpaVavHixr8upEux2u69LqHJ++brP7+WF8fPlnnNf88t+rvg748JKS0sd/crNzVVJSYmPK6rcDhw4oL/97W9au3atzp496+tyKj273S5//59j0qXWr4Dx48eP93URVVHZuwGStGXLFu3bt09hYWE+rqryO/eXCRdWWlrqOLX5vffe08KFCyVJkZGRviyr0rLb7QoICFBoaKh27typ7OxsBQQEKDw83NelVVplP2N2u11vvfWWAgICVFxcrLp16/q6tErLsiz5+/vr+PHjKiwsVHBwsPz8/C65d4U94dzXsMLCQvn7+3O5xm/w9/fXt99+q3/+85/Kzs7WVVddpVq1ajmFHPxPWU9SU1M1e/Zs3XTTTQoODvZxVZVTSUmJQkNDFRISooyMDDVp0kSNGzf2dVmVWtnfqikpKdq4caMOHz6s2rVrKzQ01MeVXTzC3+9Q9geAJM2aNUtz5sxRVlaWDh48qIiICNWqVcvHFVZO5/4x8MILL2jfvn366aefFBYWxh9Pv+Dn56eTJ0/qrrvu0pkzZ3Ty5EnHHW8JNOX5+/tr//79+stf/qImTZro0KFDOnjwoKpXr64WLVr4urxKyc/PTwcPHtSwYcNUo0YN5eTkaN26derWrZuCgoJ8XV6lUvb65Ofnp23btunee+9Vdna2PvnkE/Xu3ZsA+Atlb47+9NNPuuuuu/Sf//xH69atU2xsLAHwV+zbt09PPvmk2rZtq++//14zZszQoEGD+Pn6hXPD8Ny5c/Xpp59q3rx5Cg0N1cmTJxUQEMCbzL/g7++vs2fP6v3339euXbv0r3/9S127dlW9evV8XVqldebMGY0cOVJRUVFq166dXnnlFbVs2VKtWrXydWkXjd+O36HsReftt9/W9u3b9f7772vBggXKy8vTypUrdfz4cR9XWDn5+/vrxIkTGj58uKpVq6ZatWrpnnvu0f79+/lP7Rxlp/ts2bJFLVq00MSJE/Xcc8+pe/fu+uSTT5Sdne3jCiuHc0+3Ky0t1Weffabu3btr7Nixmjhxoq6//nqtWbNGu3fv9mGVlcsvT1HMzMzUsGHD9PDDD+uzzz5Tu3btVLt2bR9VV3mVvT7t2LFDixcv1qRJk5SamqodO3Zo8uTJTmPw82v9wYMH9frrrys2NlZPP/20goKC9MADD/i6tEprx44dmjZtmiIjIzVs2DCNHz9eTZs21fDhwyXx81Wm7M33wsJCffjhhwoNDVVCQoKWLl2qV155RUOGDNGiRYt04sQJX5daqRQXF2vEiBEKCAjQG2+8ocjISM2ePVsFBQW+Lq3S+OUp1oWFhWrZsqVuu+02paen68Ybb1RCQoJ++umnKn+6P+HPDed+sw8ePKjc3Fzt3btX+/fvV/369TV48GBt3rxZa9eu1alTp3xYaeXxy1+Qr7/+Wt27d9fgwYO1du1a3XnnnWratOkFx5vifNd12O12ffHFFzp+/LgaNmyotm3bKjc3V++9954OHDjggyorj3PfBX/99de1YsUKWZalrVu3qqioSE2aNFFUVJR27dqlWbNm6fDhwz6u2PfOfbe87D/8EydOaNmyZRo8eLDuuOMO3XnnnTpx4oS+/PJLX5ZaaZT9XlqWpSNHjigtLU379+9X48aNFRgYqAULFmjp0qWOjygy9fVL+l+vSktLZbfb9eyzz2rt2rXq3Lmz6tWrpwkTJqi4uFiTJk3ycaWVwy+vhwwLC1NISIgOHz6sPXv2SJLS0tJ04MABPfroo74osVLy8/PToUOH9Le//U3Hjh1TaGioduzYoe3bt6t169bq37+/8vLyjL+hybmvRbt375a/v79CQ0MVGxur4OBgTZ8+XQUFBZoyZQoHLOR8KdfGjRuVm5urOnXq6JtvvlGfPn0UGxurMWPGSJLef/99HTlyxJflXjRO+3TBuaf85OXl6ejRo6pXr57at2+vwsJCffXVV4qOjlbz5s1lt9v10UcfqXv37sbfbfDcX6bdu3erTp06ysvLU0ZGht555x316NFDDzzwgIqLi7Vy5UpdddVVRr67eW6f0tPTtW7dOjVp0kQRERE6cuSIvvrqK3Xs2FH169fXf/7zH0lScHCwrrzySl+W7VNlPyfz5s3TF198oQEDBujqq6/Wt99+q9zcXHXo0EE2m01ZWVnq1q2bOnXq5OOKfevcn7Hnn39eW7ZsUYcOHVSnTh298847SkxM1KBBg2S32x13Sm3Tpo2Pq/atsutIpZ8v9q9bt65atmypXbt2qW7dumrYsKH+8Ic/6Oqrr9YDDzygxMREhYSE+Lhq3zi3VydPnlSNGjXUtm1bffrpp6pdu7ZatWql6tWrq0uXLpo4caJsNpsiIiJ8XLXvlF0CYVmWZsyYodOnT6tu3brq0qWL/v3vf+vMmTNq2LCh6tSpoz59+mjPnj264YYbfF22z5z7Zl9+fr5mzZqloqIijR07Vi1bttT111+vhIQEtWjRQtnZ2fr3v/+tuLg41ahRw8eV+05Zv/71r3/ppZdeUlRUlPbs2aPg4GBdfvnlCgoKks1m00svvaTWrVvryiuvNPLvrzJl+56Wlqa5c+cqPj5eISEhOnHihIKDg9WlSxddfvnleuSRR/T1119r4MCBVfoUdsKfC8peeFatWqWxY8fqwIEDmjFjhrp16yabzaY9e/Zo7969uvbaaxUZGakuXboYfx71uX9sfvLJJ5ozZ46Ki4sVHx+vRYsW6YorrtDo0aMlSU888YT279+vmJiYKv3L9Hude9H6J598Ij8/P7377ruKiIiQzWbTxo0blZ6errfffluRkZEKDg7WmTNn1L59ex9X7n3n/hFQUFCgxx9/XE2aNFFycrLsdruCgoK0cuVKrVy5UrNnz1bHjh117733+rhq3ykuLlZAQICjZ1OmTFFubq4mTpyomjVrqnr16qpZs6bee+89FRYW6sUXX1R4eLjuv/9+H1fue/7+/jpy5IhGjx6t7Oxs5eXlqUePHqpVq5Y++ugjhYSEyGazqWXLlurVq5exb8acew38pEmTtGbNGh0/flzR0dFq1aqVFixYoMsuu0xNmjRRnTp1FBcXp44dO/q4at/y8/PTkSNHdO+996pBgwbKz8/XRx99pGuvvVbR0dFatWqV7Ha7GjdurPr16xsd/H55k7hq1arpm2++0Z49e3T55ZeradOmqlatmgoKCvTyyy9r06ZNmj59uvF/g0nSxx9/rLffflsPP/yw2rVrp//H3pmGVVmuDftciwWLeZ7nUeZBRhEQUAZFAWfDtDTbWzPLTMtKyzQ1NYctac6mmTMqJGoKgvOEqCCTCiiiooIogoiM3w+PtTZYe3/v+37vt1f1eP6pw3o6rnV3D9d8NTY2cuDAAVpbW1FVVSUtLY2AgACSkpIEqXvJgjWyuzs3N5fdu3ezadMmzMzMUFZWRl9fn7a2NpKTk8nOzkZbW5tly5b96dfrtfH3b9iyZQtubm5IJBLu37/P0qVL+fbbbxk5ciQNDQ1kZWUxYMAAtLS0OHbsGFKpFDs7O0F7m2TIlM2pU6dy5coVXrx4QUVFBbq6uowcOZL169dz9epVNm3ahI6ODt9+++2f/jD9d6ivr2f16tX06NGD9vZ2tm3bRmZmJj///DN9+vShoqKCvLw8QkJCGDhwIFpaWoSEhDB8+HDWrVuHi4uLoCIzshQWkUhEc3MzbW1t6Ojo4OjoyE8//YSrqyt2dnZYW1vTp08fDAwMCAsLY/jw4QqWXHE0Nzezdu1atLW1MTQ0BODgwYMMHToUW1tbmpub0dDQwM3NDUdHRyQSCR4eHvztb39TsOR/DCorK/nwww8ZOXIkBgYGbN26lQcPHjBy5Ejq6uo4evQopqammJubY2BgIFivuexMfvLJJ2hra9OvXz/+8Y9/8OjRI2JjY9HX12fDhg04OjpiZmYm2OiozHHV0dFBS0sLKSkphIaGMmrUKP7xj3+gqanJlStXiIuLQ11dndOnTxMSEiL4DCKZ4Td//nyKi4u5f/8+SUlJVFZWcv/+fQwNDdHX10cikSAWi/n4449RU1MTZGfxV39zTk4Oly9fRkVFBX9/f5ycnGhvb6e0tJSdO3dibGzM9OnTEYvFguwom5eXh0QiwcLCApFIxMOHD6mqqqJv3748e/YMZWVldHR0cHZ2JiEhgbCwMBISEhQt9v8Kr42/f0Frayt37tzB1dUVAFVVVU6cOIG3tzcmJib4+/uzd+9e7t27x+DBgzE3Nxe0d+73yM/P5/Tp06xevVre5S0rKwsfHx9GjhyJq6srrq6ujB07FhDWGIjW1lY0NDTQ09NDWVmZ8vJyzpw5g5qaGq6urgQHB5OVlUV+fj7e3t54eXnR0dHBtGnT8Pf3l6+ZUJClXZ86dYq5c+dy5swZdHV15VH2lStXEhwcjK6uLsrKytjY2GBtba1osRWOu7s76urqZGVl0a1bN44cOYKpqSlOTk5yZ0teXh4uLi64u7vj7OysYIkVh0z5kSnpZWVl9OzZE0dHRxYsWEBkZCTV1dU8ePCApKQkqqqqCAgIQENDQ3BK06udJ2/cuIGamhpvv/02y5YtQ1tbG4lEQkVFBcOGDaOurg5DQ0MsLCwUKLXi6Py2iUQi+RgaMzMzJkyYwKBBg3jjjTfYtm0bJSUljBkzhp49ewrWUH7w4AGnTp3C0dGR5uZmpkyZgra2NpGRkYwfP56YmBgcHR0pLi6moqICS0tL9PT05J2dO6chCwVZBL66upqcnBykUilBQUF0dHTIewTY2Njg7OxMSEgI4eHhxMbGAl0ztYSA7P4yMzPDyMiI77//nvr6ehwdHVm8eDHBwcGYm5sjEok4ePAglZWVuLq6/qXGiLw2/l6htbUVsViMWCzG2dmZtWvXkpKSQkxMDKdOnUJFRQVTU1PU1NQQi8W0tbXh5+eHqampokVXKL/Xhvr+/fvs2LGDuLg49PT00NDQYP/+/Vy7dg1XV1ccHBzkCnrnMRB/ZRobG1FWVkZFRQUjIyMmTpzIw4cPGTlyJGKxmIKCAjQ1NbGwsMDT05NTp07Ro0cP+aXj4eFBXFycgn+FYkhNTWXNmjV8++233LhxgxUrVuDv709ERASPHz9myZIlDB8+XBD76L+CrNX+5cuX+fHHH/H09MTCwoKVK1dibW2NlpYW69ev58iRI0RFRQl6vENnZbG8vBwdHR0sLCywsLBg2bJlJCYmMmrUKI4cOcLp06exsLAgISFBkJ1ROyuK9+/fR1NTE0NDQ1xdXdmyZQtSqZRvvvmGvLw8Dh06hLq6OklJSYI1/Dq/bRs3buTIkSNUVlZiZ2eHiooKx44d47PPPkNHR4fTp0/j5uaGt7c3qqqqghzvUFVVxYQJE4iKisLS0pLHjx/z+PFjxo4dy4oVK/Dw8GDEiBEYGRkhkUjIz8/Hw8OjS5qnkAwZGSKRiJycHKZPn45IJOLw4cO0t7cTHx9PUVERt27dQlVVVT7bT01NDZFIJDjDD/6ZmVZaWkpDQwMPHz7k8uXLBAUFYW9vz4wZM1BSUuLQoUPs3r2bpKQk9PX1FSz1/y6vjb9ONDc3k56ejo6ODq2trRw+fJiQkBBSUlJQV1cnLi6OLVu2UFRURGFhIRs3bmTw4MHY2NgoWnSF0vnyuHDhAoWFhfKGJI8ePaK4uJjAwEB0dHTIy8mEYJ8AACAASURBVMuTe6hcXV27NNP5q1NbW8v27dtxdXXl5MmT3L59m169erFhwwYsLCyIiYnh5s2bXLp0CTMzM6ytrYmMjERLS4v29nbU1NQwMjJS9M/4j/FqJPj+/fu88cYbFBYWkpGRgb+/Pz/99BORkZH07t2bjo4OvLy8BPeQ/SvOnDnDsWPHGDhwIG1tbezcuZPRo0ejra3N4cOH2b9/P0+ePGHhwoV/KY/m/4TOw3z37t1Lbm4uBQUF9OjRgy1btmBtbY27uzupqakkJSXRp08fBUusODp32f3hhx+4cOECZWVleHl5cerUKe7cuUNMTAyZmZm4uLgwdOhQQToW2traaG1tRSKRAC/3lqx5V3FxMSkpKQwbNozVq1eTn5/P6tWrcXNzY9KkSYjFYkEafgBlZWUcO3aMAQMGsH//ftra2khLS2PPnj306NGDadOmAbBz5055/aiJiYmCpVYMnVM17927R3JyMjNnzsTU1JTdu3dz7do1TExMiImJ4dixY7S2tuLu7o5YLJZ/J8Q9Bi9n+H3wwQdYWVkxePBgysrKyM7OZtSoUTg7O/P06VMeP37MggULMDc3V7S4/+uIOoTcm/p3OHbsGDNnzkRTU5PZs2cTFBTExYsXmT17NvPmzcPU1JTz589TVFREQkKCPC30NbBp0yb27dtHnz59yMjIYO7cuTQ2NnLo0CFu376NgYGB/PKpq6vjk08+UbTI/zFkkYV169axbt06nJycWLt2LRoaGhw+fJgVK1bw/fffo62tzcqVK/Hw8GDQoEGCVQBkDoWOjg6WLVvGiBEjMDU15eHDh0yfPp3FixdjbGxMnz59sLOz44cffhCkgtmZVz24ly5dYtq0aezcuRMtLS3WrVvHvXv3mDVrFqqqqlRWVmJlZaVAif9YrFu3jqKiIubOnUtZWRlLly4lPj4eZ2dnpk+fjrKyMuHh4UyZMkXRoiqEsrIy7OzsEIvFnDhxguTkZJKTkykuLubcuXOoq6szduxYRo8ejaamJjo6OqxatUqQ99ezZ8/YvHkzRkZGDBs2DIDPP/+cGTNmyB0tn3zyCT4+PgwcOJADBw6goqLCwIEDAeGl4UHX39y3b1/u3r0rb3w2depU7t69y44dOwD45ptvuHfvHkuWLEFdXV2RYiuMztkKsrv87t271NTUMHPmTGbOnElpaSlpaWnMmTMHfX19eZmJEPm9M5WZmcmePXtYsmQJVVVVpKen09TUxIQJE/7yKdevI390jTDU1NTIa69k3SjNzc3R1tZm8eLFhIWF0bNnT0JCQgQVhfk9Oq/b+fPnOXjwIBs3buThw4ccPnyYJ0+eEBYWRnx8PBKJBBsbG6ZMmUJ6ejqqqqr06NFDEIqBLO3n+fPnHDt2jJaWFgwNDUlMTATA0dGRFy9e8P333zN06FD8/Pzw9fUFhOuVE4lE1NbWMmnSJCQSCQMGDEAikdDS0kJOTg7m5ubydKCYmBjBO2Fk0XRZYyV1dXWsrKyoqanh9u3bBAQEYGlpyeXLl8nKyiIqKgptbW3B7i8ZnZ0rJ06cwMfHBy8vL0xMTLC3t2fHjh2MGTOGqKgoYmNj5TUyQmPXrl0cPHgQMzMzjI2NuXTpEkpKSvTv31+eHpuVlUVQUBBDhw7F29ubcePGCXZ/SSQSCgsLuXPnDm1tbdjY2LBixQpUVVVxd3cHXo4PqayspFevXri5ucnvMCEafp0NmebmZsrKymhqauLp06f06tWLqKgo9u3bx9mzZ9m6dStKSkosX75c0A4/2R6ZOXMmly9fJiQkBD09PU6fPo27uzt9+/bl1q1b3Lx5E4lEQkhICEpKSoJs7CI7Uw0NDWzcuBGRSISWlhZWVlZcvnwZLy8vrK2tEYvFFBUVYWBg8JdPUxe88dc5H3/9+vVYW1szevRobt++zf79++WPvaOjIw0NDRQXF9OjRw9AuIo5/FPZbGlpQUlJCUtLS8LCwvjll1/kedJpaWmcOHGCoKAgAgMDMTEx4euvv6atrY0vv/xSMOsn6yL1xRdfoKury4IFCzh06BB5eXmEhYUB4OTkxP3791FRUZE33RBq1E/GsWPHeP78ObNnz5Y/dC0tLdy/f5+srCw2b97MxIkTBZ2CJ0MkElFcXMyCBQsoKyvjzJkzhIeH8+TJE+7evUtwcDCamppYWlri7u6OsbGxYPdWVlYWx48fx8fHp0uDl6qqKtauXcuQIUNQVlZGJBJx7tw5IiMj0dPT+8t7gv8dNjY2XL58mbt372JnZ4eenh4bN27E1dVV3mxD1jbewsJCsGl48M+30cnJidLSUq5fv46NjQ3BwcHMnz8fZ2dnLC0t2bdvHw0NDYSHhwMINg1Ptl7Nzc2MHz8efX19xo4dy8iRI/niiy9ob2/H39+fmJgYAgICcHd3Z8yYMYhEIkE1iXuVtrY2Zs6ciZKSErNnz5anF586dYrMzExUVVVZt24dkyZNIj4+Xv6d0PYXvPzNN2/eZPny5VRVVVFdXc3evXuJiYnh0KFDPHz4kMDAQMzNzXF3dxfEDFLBG3+yNtVTpkzhwYMHDB8+HF1dXTw9PTlw4ADl5eU4OjqyZs0a3nrrrd9c1EJEpixVV1fzwQcfcPr0aWJiYpBKpaSlpeHn50f37t159OgRBQUFREZGoq+vT3t7Ozo6OoLrVFlbW8vMmTMxNDTkyy+/RCKR4ObmxpYtW+jo6ODOnTucOHGC9957Dzs7O/l3Qt5jANnZ2Tx58oTIyEiampqQSCRIpVKcnJzo2bMngwcPFnzET8aFCxdYtmwZQ4YMoV+/fmRkZHD8+HEMDAzYtGkT3t7eWFhYYGBggLGxsaLFVSjV1dUkJydjbW2Nra2tXIF0dXWloqKCVatW4enpycaNG2lsbCQ2NlauWAmR9vZ2pFIp9vb2ZGdnU11djZubG0ZGRqxZswZHR0fOnj3LpUuX6Nu3r2CNZFlERdZEQyqVYmZmxvXr1ykqKiI4OBg7OzsWL17M2bNnuXfvHnPmzEFVVVXQd70sy2PMmDEEBAQwYsQI+T/r2bMnX375JXZ2dri6uqKlpSWvvxJKkzj4fUfw8+fPSU1NZfr06WhqasrfSD8/Px49ekRRURGJiYly56gQI34yysrKWLRoEYMGDWL8+PEEBgZy+fJlTp48iba2NidOnJD3VxBKEy/BG38A586do6ioiOTkZJSVleno6EBdXR1/f382btzIjh07iImJwcvLS9Gi/iEQiUScPHmSRYsW4eHhQVpaGq2trQQGBlJdXc3hw4e5fPkyubm5zJo1i27dutHR0YGKioog2u+/6o1sa2ujtLSU7Oxs+vbti6amJnp6eri5ubF27VoKCwt5//33BTuUVhY9liF7pExNTUlOTsbKygonJycApk2bhpGRES4uLmhpaSlKZIXTWRkoKiriH//4h3xYr5aWFn379qW6upqnT59y9OhRunXrho+Pj4Kl/mNgYWGBoaEhycnJBAQEYGhoSHNzM0pKSvTq1YubN29y69YtGhoaWLx4sSANP9kdJovKyBx3RkZGZGdnAxAREYFYLObo0aMUFBTwzTffCOJ+/1f8nmKtra2Njo4ORUVFVFRUMHToUBISEvD29mbs2LFIpVJBRq9eNURu3bpFY2Mjb7/9NiUlJWzfvp0HDx4QGhqKiYkJCxYs4M033+zyTgjJkGlsbERFRUXejR5ARUWFnJwcHj16hKenpzwF9uzZsyQmJhIRESGPYMnOsVDofKaePn3K5s2buXz5Mv3798fMzAwlJSXCw8MRiURUVFRw9+5dRo8eLai7XnDG3+95P0pLS6moqCA6Olp+yGRDpPv27UtsbCz+/v4KkviPgUw5gpdruGjRIuLi4hg3bhwRERHMnz8fKysroqKi0NLSoqKigi+++AI7OztBpS929kbu2bOHa9euYWBgQGBgIC9evODo0aNypUnWsGTEiBHo6OgI0jN38eJF2tvb0dXVJTU1FRcXF7nnXEdHB1tbW2bNmkV9fT3JyckYGRnx9ttvK1pshfKqstja2kp5eTlXr14lODhYbhR7eHjg7++PoaEho0aNUpS4f0icnJxobGxk/fr1xMTEyJtGVFZWoqysTFJSElFRUQqWUjFcv34ddXV1VFRUuHbtGoaGhvJ7ydTUFLFYzKlTp5BKpQwaNIjo6GgGDBiArq6ugiVXLB0dHUyePBl1dXX5vDmRSCQfSZCbm0tFRQW+vr4YGhoCwp5HBy+zYtTU1GhqaiIlJYW9e/fy4sULGhoayM3NxdramvDwcIYMGSLYxi7Hjx/nzTff5N1330VJSanL/d/Q0MDt27e5du0avr6+zJgxg4KCAiIiIroYMkLSKzqfqQcPHqCjo4OHhweVlZXU19dja2sr30u2traEh4eTlJQkKMMPBGb8dS6k3rFjBwUFBVRUVNCrVy8WLFiAvb29PMIwf/58nj17hru7u6AjDAA3b97k6tWr2NnZkZmZiYqKiry1t6WlJUZGRkilUr7++mv69etHQEAAYWFh6OjoCM6rKbtkP/nkEwoKCmhoaGDfvn0YGRnh6+vLtWvXyMnJITQ0FEA+L1JoRf51dXW0tbVx7949Fi1axK5du6irqyMmJkauHHR0dGBra4u3tzfGxsZ069aN8ePHK1p0hSJbG1lUqri4WL5GT548obCwkO7duyORSGhtbUVJSQkPDw9Fi/2HQJYWJXNG+fj4cO3aNfbt20f//v05fvw4Y8eO7eIxFyKHDx/ml19+YdeuXVy6dIl+/fohawouEomwt7fnwYMHZGdn4+Ligp6enqCUy3+FSCSipqaGhw8f0r179y5rYmlpSVNTEx4eHl1mAgvpzoeujqv58+ezYcMGSkpKcHNzY/jw4YSEhBAXF0dERATp6elYWFjg5OSEioqKoJzIMmRvYGFhIampqcTHxyMWi+URQAsLC8RiMb/88gtHjx5FJBLx3XffCc6h0BlZ/eiECRMoLi4mOTmZ0NBQ7OzsuHr1KnV1dTg4OKCsrCxIh7sMQRl/sv/JCxYs4NKlS0RHR/Pxxx8TFBREeHg48+bN4969e/z444+8ePGCDz/8ULAbQ0ZHRwd1dXWkpqaydOlS6uvrGTJkCJcuXeLs2bNyD5NUKuXSpUtkZmYybNgwlJSUBJdqIKOoqIisrCzWr19PZGQkLS0t5ObmEh4ejpWVFZmZmTg5Ocm9vyAsz1x9fT3bt2/H3t4eZ2dnDh06xKNHj1i+fHmX+hfZX83MzLqkfgoZkUjE3bt3+fjjjwkODqa6ulo+CkNbW5sbN25QXFxMQECAoBUA6Joae/LkSdatW0fPnj3lzVxEIhG+vr4cO3aM7777jjNnzrBkyRIiIiIUK7iCkK2XpaUl3333HR0dHaxZswYlJSX5esn+HS8vLywsLHBxcVG02Arn6tWr8gY3NTU1nD9/npiYGPk/l62Zk5OT4DuEy/SBAwcOcOfOHSZNmkRDQwNbtmzB398fW1tbLl68yKeffoqDgwPjxo0DEMws4M7IDOXW1laamprYtWsXNTU1hIWFIRaLaWtrQ1VVFVtbW/r3709ISAgJCQldvhUa7e3ttLS08PnnnxMUFMRHH33EDz/8gK6uLvHx8TQ0NJCTk8OLFy/kWUZC5S9v/DU1NbFz507a29sxNTWVpxMsWLCAlJQUxGIxSUlJdOvWjaCgIExMTLCwsODjjz8W9MaAf0ZK6+rq2LhxI0pKSowYMQIHBweCg4NJTU3l/PnzXLx4kR9//JHPPvuMBw8e4OLigo6OjiDWr7OCKfPGtbe3c+LECczNzTEzM8Pd3Z3169ejpqZGZGQkoaGhWFpaKlhyxZCfn4+ZmRmBgYHU1taSkZHBW2+9xaNHjzh//jzdunXrMnBcyJ45Ga+uwYULF7CwsCA8PJytW7cSFxdHr169sLS05Pnz59y8eRN3d3fU1NQUKLXika3ZxYsXWbt2Le+//z7m5uZd1lJFRQVPT09KS0v57rvvBGnMyO4wkUhEY2Mjmpqa2NnZUVpaSktLC9bW1kilUuDlmspqdIXa0bNzPaRIJGLy5MlcuXKFlpYWoqOj2bFjByoqKnJnldDvr1fZtGkTCxYsYPTo0QQHB2Nvb8+jR4/45Zdf6N27NyUlJTg6OsqzPIT2BshSYcViMffv32f48OE4ODgQGhrK1q1bAbqMguro6EBJSUl+3wupEQ78dn9IJBLu3r2Lh4cH06dPJzo6mokTJ9LU1ISrqytPnz7F19dXsI2pZPyljb+mpiaeP38ujywYGxujpKTEhg0bOHToECoqKixduhRVVVV+/fVXnJyccHJyEqQC8HuIRCLy8/NZvnw5iYmJhISEcPr0aUQiEba2tsTGxso9UOPHj6etrY09e/YwcODALgr8X5XOht++fftIT0/nxo0b2NvbU1FRAYBUKkVfX5/8/Hy6deuGvb09qqqqv/leCNy6dYvNmzdz8+ZN/P39OXDgAFevXsXGxobExET27t3LkydPsLOzY9WqVbi5uQnegOlcv5Cbm4u5uTk3btwgOzub7du3M2rUKN544w3u3r1LTk4O0dHRBAYGCjpVvbMycPfuXTZt2kRFRQVvvvkm6urqv1EWtLW16devnyDXrHO6eWpqKsuWLaOlpYXw8HDc3d1Zt24dZmZm2NracuTIERwcHASlWL5K5/N448YNDAwMSEhIQCQSkZGRQXZ2NlZWVgD4+PgINgIjo/Mb19jYiLKyMj4+PpSWlnLt2jX69euHVCrFzs6OnJwccnNzGTt2rGBnHpaXl5OWloanpycSiYQ7d+5QVVXFp59+ipeXFxEREUyfPh1PT0+sra1/d38JSafovD/S09O5evUqrq6urF+/nnXr1jFnzhwGDx4MIO+sGx0dLXjDD/7Cxt+tW7dITk7Gx8cHd3d3Ll68SE1NDSEhIVRVVXHs2DG2bNkCwPbt29m3bx8RERGCVAD+Ffn5+WzYsIGoqCiGDBmChoYGjx494sqVK1hbW3Pu3DmsrKyIjo4mJyeHuXPnMnfuXBwdHRUt+n8E2SW7Z88eduzYQXx8PLt27UIsFmNgYEBFRQV79+7lyJEj3L9/Xz6w/NXvhYKuri5isZj8/Hza2tro378/d+/eJTc3FxsbG/kg37S0NJ4+fcqQIUMULbLCkT1sc+fO5ejRo/Tu3RstLS127txJQkICw4cPB2Dq1Kk8efLkN4X+QkOmDLS2tvLkyROMjIywt7envLyc58+f4+TkJO/oLLTz93vI1uDAgQPy/XX16lUKCwsJDw/HxMSEnTt3snv3bi5evEhiYqKgjT+xWExTUxOTJ08mJyeHgoIClJSUiIiIICYmhpKSEq5cucKJEycYM2aMoAyX36PzG7lt2zZu3ryJs7Mzffv2ZdOmTTx48ICgoCDU1dXx8PAgICCgS6t9oZ3Rjo4OevTowY0bN6ivr6e9vZ3du3czbNgwRCIR+vr6lJSUsHjxYoYNGyZ4fVW2P+bOnUt2djbDhg3DyMiI7t27s3v3bjw8PKivr2fhwoU0NTUxbtw4wZ9JGX9Z46+6uprbt29z5coVevfujY6ODhcuXODRo0e8+eabXL16lb1795KVlUV+fj7z5s0TbCqejFe9SBcvXuTMmTNIJBJ69uyJtrY22tra1NbWsnTpUh48eMBbb72FkpIS7e3tjB49usucOiGQnp5OSkoKY8aMISYmhh49epCdnY2NjQ2jRo2iW7duWFtbM23aNCQSieBSWDrT1NTEhg0bqKmpIS8vD1tbW8LCwigsLKSkpARPT08SEhLw9fV93Z2yE99//z1PnjxhyZIlqKqqoqOjg6urKykpKWRmZrJ582Y8PT354osvFC2qwhGJRNy5c4dx48ZRVVXFsmXLGDlyJMrKyhQWFtLU1ISTk5Ngz6CMzsbv8ePHmTZtGtOnT6dv377o6elRXl7O9evXGThwID4+PtjZ2TFt2jRBGn5VVVVcv34dc3NzGhoa+OyzzwgJCSEpKYkffviByspKVFVVsbe3x9fXl0GDBpGfn49EIsHe3l7R4iuEzu/cunXrOHz4MNOmTWPJkiXcvHkTT09P4uLiWLhwIUpKSnh6espnrAnxjZT9ZjU1NR4/fsy2bdsoLCxk0KBBXLlyhR07dtCvXz8aGxspKysjNDRUsLXJr1JYWMjhw4fZtGkTKioqXL16lfb2dkaOHMmRI0fIy8vDxsaG2bNnvzb8OvGXM/5kBoyBgQEaGhpcv36d4uJiEhISaGlp4eLFiygrK/Pee+9hb2+PpaUlH3zwAfr6+ooWXaF0zhPPzMyksbGR0NBQ1NTUuHfvHu3t7djZ2aGvr4+vry8BAQG8/fbb8toHQ0NDQaTovfowvXjxgoKCAmpra/H09MTU1BSRSMSGDRtITEzEwsJCrgAIsa23jOfPnzNp0iQ8PDwYMmQIjx8/Ji8vD2dnZ7y9vTl16hSVlZV07969SyMcodPe3k5aWhru7u5UVFRw8OBBZsyYQWBgIGPHjsXLy4vAwEB5aouQ6ejo4MmTJ3z22Wf8/e9/p3///ixZsgQ/Pz8iIyOpqqri4sWLqKqqCnoenSw6Kqvds7KyoqSkhNOnTzNo0CDMzMxob28nLy+PW7duER4eLjinnoznz5+zZcsWiouLMTExQV9fHzMzM/z9/ZkyZQphYWHY2dlx6tQprKys5Pf/6dOnsbS0FEwWTGc6p+LJho1//vnnZGRkUFpaio6ODjdu3CA6OhpnZ2dKSkoIDg6Wfy9Ew69zB091dXWMjY0pKiqitLSUTz75hOzsbLKzs/n+++9xdXXl/fffl38rpPX6vTRXdXV1tmzZQmZmJgUFBVy8eJFFixYRHh7OqFGjiIiIoGfPngqS+I/LX8r4kynXMq+miYkJGhoaXLx4kYcPH5KYmEhNTQ3nzp1DJBIRGBiInZ2dYBXyzohEImpra5kwYQIVFRVkZWVx7tw5xo4dy61btygtLUVZWRkLCwsADAwMAOF1lRKJROTl5fHDDz9QWFhIz549sbW1pby8nHv37tG9e3euX7/O3bt3iYqKQllZWf6tkNbp1Ufp8ePH/PLLL3z11VdYWlpiYWFBUVER+fn5hISEYG9vj7e3t2AH3cPv14DKht2vWbMGVVVVgoKCCAgIYOvWrfTv3x8TE5MureOFRuf7R9a0pL6+Hn19fWbMmMHkyZPlHnMPDw+ePn1KcHCwvO5WiMhqub///nvKyspQU1MjPj6ekydPUlBQQGhoKNbW1jQ1NeHt7S3YDpWtra1IpVKsrKwoKCigsrISZ2dnHBwcyMrKQk9Pjw8++ID79+9z8uRJ2tra8PPzo6KiggMHDpCYmChIp7JIJKKhoYF3330XExMTBg0aRFFREXv37uXnn3/GyMiIRYsWIRKJSEhI6GL4CRGRSMTNmzeZMmUKd+7cobCwkOjoaNTV1Tlz5gyNjY1MnjyZnj17EhwczIABAwDhDW7Pz88nJSUFFxcX+f3d3t6OVColKCgIqVTK8OHDGTp0KGKxWN50SSwWC8pA/q/ylzL+xGIxDx484MMPPyQ3N5djx46RlJRER0cHZ8+eRSQSERsbS0lJCQ4ODpibmytaZIVy5swZzp8/j5ubGwBbt25FJBKxaNEiBg0axKZNm6ipqeGdd97h9OnTVFdX4+npKViDBl6mSC1evJh3332XjIwMUlNTmTBhArW1taSmppKRkUFubi5Tp07FzMxM0eIqhM6e34KCArS1tdHR0aG4uJjCwkJ69OiBnp4eZWVlnDx5EiUlJSIjIwVdv9DZiDl+/DhqampoamrS1taGqakpcXFxREZGYm1tTVZWFi9evCA6OlrQjqvOys/169flmQffffcdhw4dYu3atQQFBdHa2srMmTNxdXUlJCRE0IYfvGwc9NVXXzFp0iQOHDhATk4Offr0ISgoiJ9//pna2lr8/Px+M45GaMj21vr16ykpKeHChQtyhTI/P599+/aRkJDAmjVriIiI4G9/+xtKSkro6enRr18/wRrNNTU1TJo0ibCwMIYNG4ZYLObGjRtkZGSQlJREcXExNTU1JCUlyZ3IQq7BLSwsZOrUqUyaNAl4qYc1NjYSHx8PwMGDBxGLxbi5ucn3lNAa4cDLN3LDhg2IxWK6deuGRCJBJBLR3t6Ovr4+np6eqKurs2zZMjIzM/n73/+OpqamYPfV/42/jPHX0dFBTU0NU6ZMoU+fPgwfPpzt27dTWlrK6NGjqa+v59ChQ1hbW9OvXz9B1/d1dHTQ0NDAiBEjyMvLw8TEBCcnJ4qLi5FIJAQGBgLQs2dPFi5cyIABA7C3tycwMFBwXZJejWBdvnyZQYMG8ezZMzIzM5k4cSJ2dnY4OTnR2tpKR0cHgwcPxs/PT7APmuw3f/XVV2RlZVFQUMCNGzfw8vLixo0b5Obm4ufnx08//URUVBRvvvmmgiVWPLKH/MiRI2zatInExESkUqn8zyUSCY8fP2bixIk8ffqUhQsXoqKiokiRFUrns/XBBx+wZ88eSkpK0NfXp2/fvuzcuZPAwEAqKiqYOXMmdnZ2cmVKaGzYsIGdO3cSHR0NwOnTpwkODsbY2Ji9e/cyadIkunXrhrq6OlZWVpw9e5bIyEhBzlaDrntr06ZN5OXlsXbtWtra2qisrKSxsZGhQ4dy+fJlUlNTMTY25qOPPkIkEslT94TklLl16xa3b9/GxMSES5cuoa6uTkpKCs3NzcTFxdHe3o6JiQl5eXls3LiRU6dOMXv27C5zW4W0zzo7+trb26moqCA+Ph5NTU0WL17M8OHDOXXqFCoqKsTGxiKRSOjRo4d83AoIa706Ojro6OhAW1sbBwcHfvrpJ3R0dLCxsekS1WtpaSEjI4OzZ8+yfv16dHV1FSz5H5s/tfHXWTEXiUS8ePGCy5cvM3HiRAwNDUlMTGT+/PkYGBgQERFBe3s7Hh4eghhD8O8QiURIpVKUlZWRSCTk5eXJZ9Jt2rSJkJAQVx1RyAAAIABJREFUtLW10dLSknuFzc3NUVdXF5RB07lG79tvv6Wuro7i4mLS0tK4evUqM2fOxM/Pj5ycHO7fv09ISAjXrl2jpKQEGxsbwaUwdj6PW7ZsoampiYULF7Jq1Sp0dHQYOHAg1tbWpKWlcejQIaysrJg4caKCpVY8sjOVnJzM8uXL+fbbb7GxsflNSqOamhq6urq89957glIuX0W2Xk1NTVy5coWOjg7mzJlDS0sL27ZtIyoqisDAQE6ePElubi59+vRhwoQJihZbYbi6ujJ37lw0NDTw8PCgqqqK1atXk52dzfLly/Hw8ODatWsUFxcTGhpKdHS0oFOlRCIRdXV1SKVSTp8+jZaWFiEhIfj4+FBRUcH+/fsxMzPjnXfeISIigr59+wLCm68mo6mpibfffpvS0lIyMzOJjY2ld+/e8nmH7u7uKCsrExkZKb/zTU1NBVevBl33iKxRkI2NDZqamixYsIBx48aRmJhIRkYG6enpBAUFERgYiFQqFeR6yd5AWYTP3NwcNTU1tm/fjo2NDWZmZvI1UVJSwsnJiQEDBgi64/V/lT+t8dc57L127Vqam5vR0tIiPT0dT09PjIyMUFJSQkVFhcbGRvz9/XFxcenSRlhoNDc3y4dsw8sZWNXV1cTFxfHzzz8zePBgxGIxP/74I+Xl5axZswZjY2Pi4uLk/w0hXT5isZjnz5+za9cuKisrmThxIhYWFqxatYrPPvsMX19fysrK+Oqrr7C1tcXHxwcdHR10dXXp3r27osX/j9HR0QG8XK/Kykp0dHQoKSlBVVWVXbt2oaury6xZs5BKpUilUgYPHkxkZCRRUVEKllyxyIwY2V3m7e3N7t27AQgNDe0SeZH9u7a2tgqU+I+BSCQiNzeXWbNmceLECUxMTAgLC8Pc3JwXL16wc+dO4uPjGTBgAOHh4YI6i52RKYvKyso8f/6c5ORkevTogYuLC+fPnychIYGePXvK641sbGzw9PRUtNgKp7GxkVWrViGRSDA1NaWwsBBTU1OMjIxwdXXlp59+QiQS4eHhIc+EEWIaHrz83VpaWhQWFnLw4EGWL1+Ora0t+vr62NnZsXr1amxsbLCyspI3F1JRURFk8zPZb25ra2PChAkcPnyYzMxMXrx4gZeXF7t37yYqKor29nZycnJ45513CAgIkH8vJN0L/mkoV1dXM3/+fK5cuYK+vj7BwcHU19fzyy+/4OrqKsi62v8N/rTGn+wgTJs2jfLychISErCwsODhw4ds27YNU1NTmpubWblyJT4+Pri6ugru8HSmo6OD4cOHs2bNGkxNTTEwMKB79+5s27YNCwsLjI2N2bNnDx999BGOjo50dHTg4uIij8wIKeLXmQMHDrBmzRr8/PzktWrGxsZs2LCBCxcusHPnTsaMGcPAgQMBMDQ0FFxnPJmRkp+fz/vvv4+DgwNqamosX74cd3d3ZFfMDz/8wP3793F3dxd83RW8XLfLly/z888/U1lZibW1NfHx8cyYMQN7e3scHBy6eD5f85IzZ86wefNmxo0bh1QqJS8vDy8vL4yNjbG0tKSsrIzc3FzCwsJQUlIS7NqJRCJu3brFqFGj8Pb2Rk1NjTVr1jBy5EisrKw4efIkP/74I/v372fcuHEMHTpU0SL/YTh06BDq6uqEh4dTWFjIuXPnUFNT48CBA3R0dPDxxx93SSsT8h4DMDY2Rl9fn1WrVjFs2LAuht7ChQsZNGhQl7RFIRrKsp4UO3fuRF9fn3nz5mFsbMyMGTOIiIhAKpWyfft21q9fT79+/UhMTASE19GzoaEBFRUVRCIR165dY+rUqSQkJFBZWcmuXbswMTEhISGBkpISjhw5gq+vr+Cz+f4niDpkbvs/IWVlZaxcuZKlS5d2+fM1a9bw4MEDrl27RlRUFGPHjlWQhH8sHjx4QJ8+fejbty8uLi6IRCJcXFx4+PAhPXv2ZMOGDdTX1/Ptt992+U4oXk3Z75QdCdmF+/3331NeXs706dPlnRVv375Na2srSkpK2NjYdPleKKSmpmJlZUW3bt14+PAhU6dOlRvCZWVlpKSk0NjYSEJCAr/++iuFhYUsW7YMExMTRYv+h+D48ePMnj2bd955h8uXL6Ours5bb71FbW0tH3zwAVu2bMHZ2VnRYiqczueqrq6OL7/8kqqqKnmUdO7cufKOeCYmJjx69Ahtbe0ujamEwqtOutTUVPLy8pg1axbw8i7LyMggLS0NkUhEeXk5Ghoar88kL2shtbW18fT05MqVK/KU9WfPnnHq1ClOnz6NsrIy8+fPl6fhCem+f5WWlpbfnLEPP/yQ+vp6fvzxR86ePYu+vj5qamqCHa3y6h7ZuHEjS5cu5euvv5Y7W1JSUkhPT2fTpk1UVVXR2tqKlZUVIDyne15eHtnZ2UyaNAmJREJqaiqGhobY2NgwZcoUzM3NkUqlvPvuuzg7OzNv3jzGjx8v6MZU/1P+tJE/eGn87dixgyFDhqCsrCz3kPj7+xMeHk5YWBihoaGKFvMPg6amptzImzlzJvv37yclJYV79+7x5ptvYmJiQnl5+W/SY4Vw+Tx48EDebbK5uRmJRCKPugQFBZGRkcHNmzdxcXFBTU0NHR0d9PT00NXVlV/QQlgnGV988QVXrlyRD+dta2vjwoULlJeXEx8fL5+HVVdXx4ULF2htbWX58uWC7uj5KkeOHKF3794MHz4cb29vWlpaOHv2LMOHD6ehoYHs7GxiY2MVLabCkZ2rrVu3EhAQgIWFBcePH6e1tRVPT09CQ0NJS0ujqKiI4OBgtLS0BJdSJkO2VhcvXsTMzIyioiJu3bpFdHQ0LS0t+Pn5sXnzZg4cOMCwYcPQ09MTrNe8c0SlqqqKzMxMVqxYgZmZGVpaWohEIiwsLLCxscHDw4Pw8HB5PZEQ0xY7c+vWLbZu3Yqvr688lVEsFtOvXz9+/PFHjhw5wq+//kr//v3lmTBCM2Q6G36yZlS+vr40NjZy9uxZeWRPX1+fgoICoqKi0NLSkqcSC229AExNTfH19SUvLw8NDQ18fHzQ19fn66+/ZtKkSbz77rssW7aMW7du4ePjQ1xcHOrq6ooW+0/Jn9L4k13apqam3Llzh1u3buHr64tIJGLp0qVcuXKFgIAA1NXVBXd4/m+YmpqioqLCzp07WbJkCQ4ODujp6WFvb4+xsTEhISGCUwaam5vZsmULtbW1bNiwgby8PEJCQhCLxfJHLSAggG3btvHo0SN8fHy6PPxC2mOtra189tlnACQnJ2NlZYWmpia6urq4uLiQn59PRUUFfn5+6Ovr0717d8LCwujTp4+gveS/95Dn5uZy/PhxYmNj0dHRobGxkUOHDhEbG0tYWNhrw68THR0dvPXWW7S2thIfH4+RkRHp6eloaGhgb2+Pn58fSkpKuLi4KFpUhdDZkKmoqGDDhg3U1NSQmJjIokWL0NDQwNPTk+fPn/P48WO0tLSIjIxUsNSKQ6aY19bWUlxcjKmpKb169ZLP9Dt69CgHDhxATU1N3v1aFuUSYmOXV1MPL126RH5+PjExMQBdhpQPHjwYXV1d3n//fczNzQXpHIV/6gVLly4lJSWFsrIy8vPzmTx5Mnv27CE7Oxt1dXU2bNjAixcv6Nu3b5c1EtJ6vTov+rPPPuPRo0d4eXnR0dHBtm3b6NWrF2ZmZpw8eZLw8HCCgoIUKPGfnz+88Sebr+Pk5ERaWhp6enpdogfKysocPXqUtLQ0srOzuXfvHtOmTUMqlQrq8Px38PLy4vjx45w/f56RI0fi6ekpbysvJAX92bNn/PLLL3h6etLQ0MCnn36KpqYm8+fPlz/uYrGY9vZ21NXVcXBwoKmpSdBNEcRiMVlZWXz88cdoaGjIH3x4We+op6dHVlYWDQ0N8vmRQtpTr9LU1CSfR9Tc3IySkpJckTI0NOTu3bsUFRUREBBAfX09Z86cITQ0FA0Njdf3F/D48WMaGhrQ0NBgwIABzJgxAwsLC2JiYmhubmbr1q24uLhgZ2eHo6OjosVVGCKRSK5ASaVSNDQ0OHXqFLq6uowcOZJ58+ZRXl7OggULCA0N5aOPPlK0yApDNh/yypUrvPfeezQ3N7NixQo8PDwICgrCz88Pc3NzqqqqePToEd7e3mhra8u/F+K5lP1mWUMvMzMz1q5di46Ojvzcyd5KZWVl7O3tUVNT+41SLwQ6G8oyo2/ZsmVs376duro6+vbtS2RkJD///DPFxcUMGDCAyZMnC3JfwT/P49OnT1m9ejXdunUjPDyc7du309raSvfu3Wlra2PNmjX89NNPREZG8tZbbyla7D89f3jjz9DQkHHjxnHw4EGqq6uJiYmRN4sQiURYWVkRHh4un1EkM/xe8+/p06cPS5YsoaGhAT8/P0WL8x+nubmZhoYGVFVV0dXV5cmTJ+jp6fHkyRP5sFCJRCL3EMtmFckMGqFSW1vLypUrCQsLw8jI6Dce3WfPnqGnp0dmZiahoaHywm2hsmfPHoqLizE0NGTZsmUEBgbK5/NpaGigoaHBnj17yMjI4Oeff2bChAl4eXkJds06K051dXV88803iEQizMzMMDY2xtPTk8mTJxMZGUmvXr14/vw5bm5ugkwn7qxYt7a2MmzYMMzMzHB0dERXVxeJREJaWhohISEkJSVhYWFBUFAQCQkJCpZcMeTl5aGmpkZraytPnz7ls88+Y/78+fTo0YNVq1Zx//59HBwcMDIywszMDA8PD86fP09gYKBgZ4Z1Po/p6emsWLFC3ulUT0+PyspKfHx8/mV0T2iGn8yQqa2tpaWlhaKiImxtbdm/fz/Pnj1j2bJliEQieTnJjh07iI6OxsbGposjVUiIRCIqKirkdXw9evRAX18fU1NTtm7dioGBAX379sXb25tevXoxYMAARYv8l+APZ/x1bn8u+/vy8nLOnDnD9OnTcXJyknvQ4eXlpKKigqOj4+vmCP8NRCIRgwcPpq6uDgcHB0WL8x/lxo0brF69Gl9fXxwcHFi2bBlPnjzh/fff5969e+zfvx9/f380NTW5c+cOOjo6glXGX0VNTY2amhoKCwvx8fFBKpXKo1v37t2jtLSUkJAQef2C0NfN0dGRkSNHkp6ezgcffCBvDgQv5xKZm5vTv39/3NzceOONN/Dy8lKgtIqlcx3VsWPHcHR05MWLF1y8eBENDQ0MDQ2xt7enqKiIRYsWMXbsWPz8/ARp+NXV1ZGeno67uzvnz5+nsbERa2tr5syZQ1xcHEZGRqirq5OVlcXx48fx8/PDxcWly/4TEvPmzWPr1q3k5uZSUVFBr1696NOnD7W1tXzyyScsXbqU8vJyDh06hJeXF/r6+ujq6rJu3TrMzMwEqVvIDJmamhpaW1sxMzPDy8uL7du3U1ZWxoULF2hoaCA6Olrw9zz8U3e9efMmkyZNwtnZGQ0NDb766itsbGxYtGgRIpGIrVu3UlxcTHh4ONra2ixYsIDExERB167t27cPExMTJk+ejEQioaOjA0tLSyQSCWvWrMHLywt3d3f5mLLX/L/zhzP+Ghsb5dGCx48fI5FI8PT0JDAwkKlTp9KnT58unX1eXzr/cyQSieAMP3jZ3KWsrIzr168TEBDAs2fPKCoqoqmpiREjRnD27Fny8vLYtm0bWVlZxMXFCa7G49+hoaFBUVERlZWVdO/eXT5QdcaMGTQ1NRERESGPbgkRWUSmra2N1tZWysvLuXHjBn/729/Q0dHp4rxqa2tDWVkZfX19wdXavorM6/3JJ5+Ql5dHcHAwvr6+3Lx5k7y8PNTV1dHS0uLOnTvyWlIhUltbi56eHikpKSxatIgzZ87Qo0cPevfuTV1dHUuWLGH06NFoa2vLSyaCgoIEOV6ltbWVN998Ez09PVavXo2GhgZ5eXny6NWBAwcwNzdn8ODBSKVSUlNTCQwMxNramgcPHnD9+nUmTpwoSD1DJBKRlZXFnDlzKC0tpaCggIiICMLDw9HV1eXo0aNcuXIFQ0NDwdbawj8jyh0dHVRXVzNu3DjGjBnTJUvt0aNHODk5sWnTJn799Vfeeecd9PT0cHFxITY2VlDdKn8vFbioqIgnT54QGBhIR0cHSkpKPHv2TJ6F5ePjI2id4v8Hfyjj78aNG8yZM4fY2FguXLjAtGnTyMrKorGxkf79+6OmpsY333zD22+/ze7du3n8+LFgWwi/5r+P7NIxMjLi2rVrFBUVyUcRPH78mNzcXDQ1NRk2bBh1dXVoaGjw9ddfy42b17xE1hb+2LFjpKam0tDQwDfffIONjQ2ff/65gqVTLJ2bQZw/fx4DAwMGDRqEubk5kydPJiEhAW1tbZ49e4aKioog03x+D9l4FVnzg0WLFqGurk5tbS0hISFUVFRw6tQpFi9eTEBAgHz+qND49NNPuXHjBn5+fkilUo4cOUJAQACjRo0CIDg4mAsXLrBr1y52796NVCrliy++EKThBy8dCsnJyXh7exMcHIytrS2ZmZkYGhri6upKbW0tv/76K42Njezdu5d33nmHqKgo4GV3bCFHtcrLy1m+fDkrV67k3r17HDlyRN4Ux9zcnODgYPnIC29vb8HNo4OuEeXy8nLc3Nw4d+4ct27dIiEhAU1NTezt7amoqODcuXPU1NSQnJyMiYmJPM1TSPXdsmgyvLzrHz58KB/Lc+7cObp16yY3hGfNmoW+vr7gncn/v/hDGX8vXrygvLyc1NRULl68yJw5c3B0dOTOnTscPXqUDz/8kJKSEtauXUtpaSlvvfUWampqihb7NX8SZB3Jxo8fz9OnT2lubqa8vByxWMyAAQOoqKggJycHPT09IiMjCQgI6NLF7DX/xNbWlrCwMGpqapBKpXTv3p0JEyYoWiyFInvYGhoamDBhAqdPn5Y3JImMjKS2tpbk5GQ8PT2ZPXs23t7egq0lgq4dUGX1Qrm5uZSUlPDkyRP279/PihUryM3N5dNPPyU4OJjQ0FBBdkFtaGjg3XffRVdXl6lTp6KmpoaRkRHx8fGsXbuWtrY2vL29AYiLi8PKygo7OzsmTZokGMXyXzFgwAC+/PJLzM3N5dEXPz8/HB0d0dDQQEdHh4yMDIYPH05cXBwgvMHa8NvffPfuXaqrq2lsbGTfvn3MmjULNzc37t69i7q6Ojo6OrS0tHDgwAH69+8vqDfy9yLKBQUF6OrqkpSUxL59+6ivr5cPIA8ODiYyMpLY2FhUVFRoa2uTO5WFss9k931raytfffUV+fn5tLa2smnTJiZMmEBVVRXp6elkZmayZs0aXFxcGDx4sKLF/svyhzD+ZBEZLS0tLC0tyc/Pp6ysjPfffx9ra2s0NDS4cOEC1tbWjBgxAnd3d8aPH//a8HvNfxnZw5afn09+fj7Lly8nIiICqVTKiRMn5J7Mq1ev4uTk1GXosZAetf8OKioq+Pv74+npiaurq6LFURidmx3U1NTw3Xff4e3tzbx583j27BnZ2dnY2NiQmJhIeXk5GRkZJCYmylvIC5HOM7DOnz/PhQsXsLGxwdfXl/Pnz6OiokLv3r0ZP348aWlpBAUFoa+vj7GxsYIlVwwZGRk8efKEb7/9FmVlZRoaGmhsbMTY2Bh/f3/mzp2Lm5sbpaWlpKenM3DgQLp166Zosf8QyOaFTZ48mV27dvHOO+/Im0Zoamri7OxMbGysvK6vc3RCKHRubCarc9fQ0GDz5s1cuHCBzZs3Y21tTUFBAXv27MHT0xM1NTU2b96MkpKSPFoqFH4vopyRkYGBgQEBAQG4uLiwfv16tLW1cXJy6vKtEPcXvDRya2trWbhwIerq6ixdupSwsDAKCwtJSUnhyy+/xMPDQz6GZvjw4YoW+S+Nwo0/WZpUc3Mzjx8/xsLCAjMzM86cOUNDQwM+Pj4YGxuzf/9+DA0NcXJywsjISJEiv+ZPQmdPZue5Tps3b5bXjmpqapKWlsbly5fx9/enT58+mJubK1jy1/yZ6Oy5vXLlCmlpaWhpadGrVy/8/PzIz88nJycHJycn4uPj6d27Nx4eHgqUWPHI1iw1NZXly5dTWVnJpUuXsLKyIikpicDAQAwNDVm5ciWPHz9m8ODBgq67ra6ulnfQ3bdvH7/++itLliwhLy8PHx8fgoODWbduHceOHSM+Ph5bW1tFi/yHwtzcHGtraw4cOMCyZcuAl5lGsuiLWCzuEoUWGiKRiHv37jFmzBhOnDjBuXPniImJob6+Hk1NTerr63n06BFz584lNDSUgIAAALy9venfv7+CpVcM/y6ibGJiwv9p7+7Doi6zBo5/h2EYhqEZJBFQRkV51VBRMIwXBxXDd9d8ucrMUio3vUi9bBO1rGyxTU1WDF3Tq1rNDE3Wyi1EW4EMoRQUTQ1JA0FREONNRWCeP3qYVbctfZ5kgDmf/xwGOcw1zO937vs+59x3332sX7+e0aNH39L12preX7fvJqtUKnbu3MnFixcJCQnB0dERo9HIZ599xp49e5gyZQpeXl4YDAYLRm0dLJ78NXfznDNnDidOnCAnJ4f+/fvj7+/P9u3buXz5MjY2NiQnJzN69Gi5MRd3rPlD55NPPiElJYXr16+bE74jR44QFBSEXq8nJycHLy8vgoODrb7phvi/+fzzz/nwww95/PHH0el0nDp1ChsbG3r06EFISAgpKSk0NDTQt29fqV/4XykpKezYsYMPP/yQRx55hKNHj3L69Gm0Wi06nY4XXniBuro6EhMTrTrxA9DpdJSVlREfH09JSQlhYWEYjUY6dOhAamoqM2fOJDg4mEmTJll1841f4+3tTVVVFa+99hrTpk27pZbbmm7If0lxcTGJiYmMHTuWuLg43nnnHWpqahg7diwODg6kpqZy6tQpZsyYYU72TCaT1daSwq/vKMPP77eHH34YJycnq3x/3Xy6Iz09nYqKCrp06UJoaCh79uzBZDLh5eWFSqUiLCyMr776irCwMFQqlVW+Xi3NYslf81Gp0tJSli1bxrx588xHpaKjo+nXrx8KhYJVq1ZRVlbGggULCAoKskSoog1LSkoiLS2N6OhoNm3aBEC/fv3Iy8tj48aN7N27l/r6epYsWYJWq72lDkmI/+b2FU2dTsfatWtRqVT84Q9/4OLFixw6dAi9Xk+XLl0ICwuz6mOecOtr1jwDa9euXfTt2xeDwUDXrl05dOgQFy5cwGAwEBISwpQpUywcdetgb2/PoEGDCAoKYvbs2fj4+ODn50dVVRVnzpxhyJAh3HfffVZ9M34nQkND2bdvH0ql0irHNzRr/ls0mUzU1tby7rvvcujQIcaNG4e7uzuDBg1izZo16PV6RowYwfDhwxk6dCienp7m5kxynfztHeXmLqDW+Fo1/86vvPIKGRkZFBUVkZ2dTd++fRk4cCDvvvsujo6OdOvWDa1Wa66HtMbXyhJaPPk7e/YsDQ0NXLt2DQcHB+rq6qisrESpVLJmzRpiY2MxGo1cv34dLy8vevbsyfTp061yJIG4e7fflO/bt4+VK1eSnZ3N6dOniYuLo0ePHkRERNCpUyd69uzJ3LlzUSqVt6xUCfFrFAoF9fX15Ofn4+zsjF6vp0+fPixfvhxvb2+GDRvGiRMnyM7ONu8wW/NFrbnOpbS0lBMnTtDU1MRDDz2ERqPhiy++ICAgAA8PD+677z7zDUKPHj0sHXar01zzqFQquXTpEsuXL8fX19fqFxbuxvjx46068bt5nmZtbS2Ojo54eXlRVFREbW0tHh4edO7cGU9PT5YuXcrAgQNxc3Mzf88vDXO3Zr+2owzWmSQ3J7xbt26lurqalStXkpqayvHjxykvL2fo0KG4urqybt06wsPD0ev1lg7Z6rRo8venP/2JtLQ0du3axaeffoqHhwedOnVi+/bt7N+/nyVLlmA0GikrK2PFihU8+OCD9OnTRxq7iDvSfIN56dIl5s6dy+DBg/nyyy9ZsWIFjY2NbNiwAbVaTWZmJvb29gQEBJiLsW++IArxS25fwd27dy9vvfUWDzzwAB07dsTd3R2dTsfq1auJjIykb9++eHt7YzAYrPIG4GYKhYLs7Gxmz55NXV0db7/9Ni4uLvTu3Zva2lrS0tIICwvDYDAQFBQkw3x/xeXLl8nIyODFF19kzJgxPPPMM5YOqU2y1o6ezde5efPmkZaWxubNm3nooYfw9vbm8OHDVFVV0b17dzw9PQkKCqJPnz4Wjrr1kx3ln93c/AygQ4cODB48mPj4eBobG4mNjeXjjz+mtLSUyZMnM3DgQKlPtpAWSf4qKytZsGABDg4OJCQkMGjQILRarXmmn1Kp5OrVq7i6unLjxg3i4uLo3r07UVFR9zo00Q6UlZVhZ2eHra0tRUVFvPnmmwQHBxMSEkJVVRVXrlxh+vTpuLu7s337dhITE4mKisLZ2dn8f8iOn/g1N+8KFxUVodFo8PX1pampiZSUFAICAnByckKv15Oamkp2djaPPvqo1XanhJ+budTW1mJvb4+trS1r1qwhJiaGmTNn4ubmRk5Ojvl4Z2ZmJqWlpQwYMECOLv4GOzs7fvrpJ4YMGWIeTSDunjUlfmVlZdTX1+Pg4EB9fT2vvvoqarWaN954g5qaGhISEnjiiSfQarUcOHCAq1ev4u/vj6urKwqFwioT5btlzTvKNyd9hw4dYvfu3bi6uuLm5kZTUxM7d+4kISEBV1dX0tLScHBwwN/fXxq7WNA9T/6ampp47LHHAHj77beBnwtlAwICqKqqYv369cTHx1NXV8ehQ4fIzMxk2LBhVj8zTNyZrVu3smnTJvR6PRqNhoyMDDZv3szzzz9Pp06dUKvVmEwmEhMTOXz4MBkZGaxevRovLy9Lhy7akOYbny1btpCQkEBubi4HDhzgueeeo6CggPT0dHQ6HTt37iQ4OJjY2FirPrGwaNEi8vLyCAgIQK/Xo9PpyMjIQKPREBAQgJeXF7m5uezfv988vsdoNMoN5h2wsbGhc+fO0vxM/CaTycTp06cZM2YM58+fZ/DgwajfutZPAAAQsElEQVTVarKzs4mIiMDT05PAwEDOnj1LSkoKs2fP5tKlS/Tv3x9nZ2er7FD5/2WNiXLz77t3715WrVrF9evXSU9Px9HRER8fHz766CPS09M5evQo5eXlLFq0SLr2W9g9T/4UCgVeXl5s3ryZ4OBg3NzcuHHjBkqlkkGDBvH555+jVCoZP348RqOR8PBwBgwYcC9DEu3E66+/TnZ2NosXL6Z37944OTlhMBiorq4mMzOTyMhIXFxc6NOnD+Hh4TzwwAPMnDkTV1dXq/yAFnfv5qOe33zzDRs3biQxMZF+/fpx/Phxdu/ezcsvv0xBQQFZWVnmC5u1Jn4NDQ0sXLgQgDVr1mAwGHB0dDTPD6upqcHR0REXFxecnZ354YcfiIiIsPqaSCHuBYVCgZOTE5mZmVRXV/PDDz8QHh5ORkYGFRUVhISEAODv78+RI0cYOnQovXr14v7777dw5G2XNX2Obdq0if79+wM/91dISkpi7dq1TJ48mYsXL5Kfn0+HDh14/PHHKSws5Pr168THx0tX9VagRY59du7cGScnJxYvXsy4cePQ6XTU1dWhUqnYv38/vr6+5toraYMu7kRhYSFffPEF69atw8XFBaVSiUKhQK1W07NnT77//ntycnIIDw9HoVCg1+txcXFBpVJJfZ+4I42NjebBx80jaaqrqxk7diw6nY6BAwfy6aefYjAYGDVqFEajkbFjx1rVxf92NjY2fPnll8yfPx+tVktDQwMKhQIbGxvc3Nw4ePCgeQU4KSmJkSNH0rt3b0uHLUS7c/XqVVQqFfX19VRUVNCrVy8uX75McXEx06ZNIyEhgerqamxsbNiwYQNNTU0MGzZMGrqI31RbW0tFRQVFRUXmmtDz58+za9cunJyc6NevH7169eK7777j6NGjeHh4MHbsWCIiIuTeq5VosUKnSZMmMWHCBKZPnw6Ag4MD8HPLb1llEnfr3Llz2NjYYGdnx40bN8xtqxUKBQaDgQkTJlBWVsbatWv/43vlw0fcCaVSSXFxMYsXL+bkyZN4enqSnp5OVlYWtra2aLVa8wwnhUKBSqWydMgWd/nyZXJzcykvLwcwL8oAuLm5ERUVxfTp03FxcSE+Pp5HHnnEkuEK0e40NDQwc+ZMXnnlFaqqqrC3t8fX15fU1FRGjRpFdnY2+fn5rFmzhp9++onNmzfj5OTEX/7yF0n8xG8qLS0lOjqaH3/8kccff5y5c+eSmJhIaGgoixYtIisri6ysLNRqNZMnT6axsZH6+npLhy1u06LdPkNDQzlw4AB79uzBaDQSExODwWBg6tSpLRWCaCfs7OzYuXMnvXr1ws3NDfh3U46ioiJu3LiBr68vnp6e5q8LcTe+//57Zs+ezfDhwzEajej1etzc3Hjttdfo1q0bX3/9NQcOHGDUqFG3NA+yZhqNhvLyco4fP06/fv1Qq9Vcu3YNW1tbSktLOXv2LGFhYQQGBkrNmhD3gI2NDW+//TY5OTloNBqOHTtGZGQk169fN4912LZtG7169eKRRx5h8ODBREZGAv8+7SDEf3PmzBmqq6vx9fXl5MmTPPzww7z88sv4+fkxbNgwqqqq2L17Nz4+PnTt2pXg4GBp7NIKtfhfeVJSEqdOnSIwMJDhw4ezdOnSlg5BtAMdO3YkIiKC1NRULly4APx7R++jjz6ipKSEgQMH0q9fP0uGKdqQxsbGW/595MgRIiMjefTRR80d70aMGMGcOXPIyckhOzubhIQEmUF6m6FDh3Ljxg2Sk5MBzN0733jjDfLz8y0ZmhBWITk5mc6dO1NRUUGXLl149tln+eabb7h8+TJRUVEMGjSIDRs2UFNTYz6xcPMYCCH+GxcXF0pKSnjxxReprKykZ8+evPrqq8yfP5+ioiKmTp2Kh4cHH330ESaTyXzKT7QuCpPJZGrpH1pdXc3x48fNxcZC/F8UFhaSnJzMlStXiImJoVOnTqxcuZLS0lJWr16NTqezdIiijbi5DvTbb78lKCiI1NRU9uzZw5IlS+jQoQPw89gae3t7NBrNf8z9E/+Wnp7Ozp07uXbtGkajkZSUFHr37i2LfUK0kLy8PGbMmMGOHTuoqqpiy5YtdOnShXnz5nH58mVMJpOU3Ii7VlZWRnx8PEqlkmHDhmE0GnFwcGD9+vUkJyfzxRdfmK+n0sOj9WrRY5/N1Go1Hh4eLf1jRTvj7OxM9+7dOX36NNu2bePAgQOoVCoSExNRq9XS0VPcseajTnPnziU3N5fIyEjq6uo4c+YMSqWSrl27UlNTQ1xcHHq9Hk9PT3lv/Yru3bsTHh5OeXk5arWawMBAGd8jRAtyc3PDycmJuLg4Zs2axbhx4/Dx8cHBwQGNRoODg4MsYInfdPtRYI1Gw4gRI8wz/a5evYqPjw9BQUFkZWVx6dIlHnzwQdlFbuUssvMnxO+tpqaGa9eu0bFjRwDp6Cnu2ooVK7hy5Qp//vOfzY99/PHH5Obmcu7cOSorKxk/fjxPPfWUBaMUQog7t2LFCvbu3UtqaqqlQxFtTPPiQGVlJaWlpfj7+5ubAtXX17Nt2zbOnz9PcHAwQ4YMsXS44i5I8ifajeYPqubGL0L8N7+04r18+XKMRiODBg2iqqoKnU5HQ0MD165d48cff8TOzs48kkYIIdqK6dOnM3HiRMaMGWPpUEQbU1payuuvv87w4cMZP348cGtS+N5779HU1ERMTAw6nU52ktsISf6EEFbl5sWBM2fO4OnpCcDq1avJy8vj/fffNz93x44dREdHy1BaIYQQ7V5zYtfY2EhxcTETJkzgySefJDY29hefd+7cOezt7c2nrkTbIMmfEMIqvfXWW+Tl5eHt7U1oaCgRERHExsaiUqmYOXMmW7ZsoaKignXr1knhuhCizZNTMeLX/FK5zOLFi8nKyuLLL78E5D3UXkjyJ4SwOmvXrqWoqIg333yTqVOnYm9vz8SJExk6dCjx8fHY2tpia2vLwoULLR2qEEIIcU/dXAqxdu1aHB0d6du3L4GBgTz22GN4eHjw5ptv/sdzRdskyZ8Qwuqkp6cTHBzMihUruHDhAiEhIRw8eJCYmBgGDBhg6fCEEEKIFnX9+nVmzZpF79690el0pKWlsWzZMnr06MGYMWMYOXIkzz//vKXDFL8D2bsVQrRbt69tNQ9yHzx4MCUlJfz000+sW7eOiRMn8uOPP5KammqegSWEEEK0V7df5woKCujfvz8LFizg8OHD9O/fHz8/P+zs7NiwYQObN2/m/Pnzcn1sByT5E0K0W81HUy5cuABwSz2DyWRi3759FBQUsH37dvz8/Hj66adxdnaWIy1CCCHatebrXPOiqIODAxkZGURHRxMSEkJcXBwAe/fupVu3buzfvx93d3e5PrYDFhnyLoQQ91JeXh56vR6VSsW+ffvYsWMHgwcPNtcqmEwmOnbsiL29Pe+//z7FxcUsX74cZ2dnS4cuhBBC3DMHDx5EpVLh6OjI1q1byczMxM7ODn9/f06cOEHnzp2ZN28eAHFxcRw7doyhQ4ei0WgsHLn4vUjNnxCiXSkoKGD+/PnMmzePIUOG8Mknn3DhwgWeeeaZX+xmdvHiRTp16mShaIUQQoiWcfr0aTZt2oSfnx/V1dVkZWURGhrK1q1beeedd6iqquJf//oXBw8e5P7776dDhw6sXLnS0mGL35mtpQMQQojfk7e3N08//TRbtmzBYDBQWFhIz549AW5pUd3Q0ICtra0kfkIIIdq15oVPLy8vjEYj+fn5FBYWkpiYiLOzMxqNhvj4eN544w0WLlzIsWPHuHHjBoGBgZYOXdwDcuxTCNHm3d562tfXl9LSUv75z39SVlZGWVkZJpOJkpISrl69iqOjI2q12oIRCyGEEPdefX093377LZWVlTg4OFBaWoqdnR1Hjx5FqVQSEBBAYGAgJ0+eZNu2bQwfPhwPDw/c3d0tHbq4R6ThixCiTWtqajInfpmZmWzatIlz587xxz/+ER8fH86cOYOrqyvHjx/ns88+Y+PGjZSXl1s4aiGEEOLea2xspKGhgRUrVjB69Gi0Wi2TJ08mKiqKkpISDhw4AMCiRYtwd3enoqLCwhGLe02OfQoh2rTmo5xpaWkkJSXRrVs3CgoKGDZsGM899xzFxcU4ODjwxBNPoNVqqa+vx87OzsJRCyGEEPdObm4unTp1QqPRYDAYOHbsGCEhIfj6+gIQHR3Nxx9/TE5ODmq1mqCgIPMgd9G+ScMXIUSb98EHH5CSksLGjRtxcnIiOTmZkydPMnr0aDw9PYmJiWHOnDlERkZaOlQhhBDinnr++eepqKhAp9NRUlLC0qVLUalU5OXlUVxczFNPPYW7uzunTp0iOTmZAQMGMHLkSEuHLVqI1PwJIdqcm496AlRVVfH+++/j5+eHt7c3rq6ulJWVkZWVhZ+fH5MnT6ZXr14WjFgIIYS4txoaGliwYAEajYakpCSGDx+OXq/nr3/9K8HBwXTt2pWCggLOnj2Ls7MzeXl5hIaGEhYWZunQRQuSmj8hRJtiMpmwsbHh3Llz5ObmUlJSQlhYGEuXLmX9+vUUFhbSsWNHwsPDcXFxoa6uTub3CSGEaPcUCgV2dna88MIL5sdGjx7NhAkTWLVqFT4+PowYMYKLFy8ybdo0APr06WOpcIWFyLFPIUSb89VXX7Fs2TJGjx5NcnIymzZtwsfHh/Xr1/PVV1/xt7/9Da1WS21tLVqt1tLhCiGEEPdcRUUFjz32GAkJCfj7+98y2/bZZ58lLCyMadOm0djYyLlz5+jWrZuFIxaWIDt/QohWr76+nsbGRuDnIe6JiYkkJSURFRVFTU0NcXFxlJSUMGvWLFxcXEhISACQxE8IIYTVuP/++xkxYgS7du2ipqYGpVJJTU0N8PMM3OZrolKplMTPisnOnxCiVVu4cCEKhQKlUsnrr78OQHl5OadOnWLVqlUkJSWxevVqCgsLee+997C1tUWtVt9SEyiEEEJYg/z8fP7xj3/QpUsXZsyYYX48JiaGMWPGMG7cOAtGJ1oD2fkTQrRKlZWVzJkzB4VCwYwZM6iqqiItLQ2Ajh07kp+fz5AhQ3BzcyMqKori4mIOHz6Mvb29JH5CCCGsUkBAABERERw5coRZs2axY8cOpkyZgqurqyR+ApCdPyFEK9TU1MSECRPQarV88MEHAMydO5egoCB0Oh1jx45l586dfP3113h7e5Odnc1TTz1FeHi4hSMXQgghLK+mpoa///3vaLVatFotEydOtHRIopWQIe9CiFbHxsaGJUuWMHv2bL777jvS09M5deoUERERJCQkcOnSJaZOnYqNjQ179uzhySeflMRPCCGE+F+Ojo4899xzlg5DtEKy8yeEaLW2b9/OSy+9hNFoZP369QAUFhYyadIkPvnkEzw8PGhoaMDWVtaxhBBCCCF+i9wxCSFarUmTJnH+/Hn27Nljfqxbt24MHDjQ3L5aEj8hhBBCiDsjDV+EEK1abGwsBoOB2NhYrl69yrPPPoubmxvu7u6WDk0IIYQQok2RY59CiDYhOjqas2fP8tJLLzF16lRLhyOEEEII0eZI8ieEaBOqq6s5fvw4ISEhlg5FCCGEEKJNkuRPCCGEEEIIIayA1PwJIYQQQgghhBWQ5E8IIYQQQgghrIAkf0IIIYQQQghhBST5E0IIIYQQQggrIMmfEEIIIYQQQlgBSf6EEEIIIYQQwgpI8ieEEEIIIYQQVuB/AFJhVbMOK7iRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from UtilTextClassification import plot_freq\n",
    "\n",
    "plot_freq(df, col=['product'], top_classes=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>**Sample of Dataset**</mark>\n",
    "\n",
    "Demo of (product, complaints) example to get a feeling of what complaints will be like to corresponding product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Demo of product and its complaint example...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product</th>\n",
       "      <th>consumer_complaint_narrative</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Debt collection</td>\n",
       "      <td>XXXX has claimed I owe them {$27.00} for XXXX years despite the PROOF of PAYMENT I sent them : canceled check and their ownPAID INVOICE for {$27.00}! \\nThey continue to insist I owe them and collection agencies are after me. \\nHow can I stop this harassment for a bill I already paid four years ago? \\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Consumer Loan</td>\n",
       "      <td>Due to inconsistencies in the amount owed that I was told by M &amp; T Bank and the amount that was reported to the credit reporting agencies, I was advised to write a good will letter in order to address the issue and request the negative entry be removed from my credit report all together. I had a vehicle that was stolen and it was declared a total loss by insurance company. The insurance company and the GAP insurancw companypaid the outstanding balance of the loan, but I was told by M &amp; T Bank that there was still a balance due on the loan. In good faith, without having received any proof as to why there was still a balance, I made a partial payment towards the remaining debt. I then sent the goodwill letter still offering to pay the remainder of the debt, but in exchange for the removal of the negative entry on my credit report. At one point, in XXXX 2015, per my credit monitoring agency, it showed a delinquent balance of {$0.00}, but when I checked my credit report again on XXXX X...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Mortgage</td>\n",
       "      <td>In XX/XX/XXXX my wages that I earned at my job decreased by almost half, by XX/XX/XXXX I knew I was in trouble with my home loan. I began contacting WFB whom my home loan is with, for assitance and options. \\nIn early XX/XX/XXXX I began the Loan Modification process with Wells Fargo Bank. I was told that they would not assist me with anything financial on my home loan until I fell 90 days behind, though at the time I started to inquire for assistance from WFB I was only a few weeks behind. So, I began working with a program called XXXX. They approved me for a variety of assistence and reached out to Wells Fargo Bank to determine what they could assist with. Wells Fargo then turned down the assistance from XXXX and finally offered to do a Loan Modification for me. The outcome was totally unknow about what I would be offered in the end by WFB for assistance. Wells Fargo lost my paperwork twice during this process, so it took 2 months from the time I started to the time my paperwork b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mortgage</td>\n",
       "      <td>I have an open and current mortgage with Chase Bank # XXXX. Chase is reporting the loan payments to XXXX but XXXX is surpressing the information and reporting the loan as Discharged in BK. This mortgage was reaffirmed in a Chapter XXXX BK discharged dated XXXX/XXXX/2013. Chase keeps referring to BK Law for Chapter XXXX and we keep providing documentation for Chapter XXXX, and the account should be open and current with all the payments \\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Mortgage</td>\n",
       "      <td>XXXX was submitted XX/XX/XXXX. At the time I submitted this complaint, I had dealt with Rushmore Mortgage directly endeavoring to get them to stop the continuous daily calls I was receiving trying to collect on a mortgage for which I was not responsible due to bankruptcy. They denied having knowledge of the bankruptcy, even though I had spoken with them about it repeatedly and had written them repeatedly referencing the bankruptcy requesting them to cease the pursuit, they continued to do so. When they were unable to trick me into paying, force me into paying in retaliation they placed reported to my credit bureaus a past due mortgage amount that had been discharged in Federal Court. On XX/XX/XXXX Rushmore responded the referenced complaint indicating that they would remove the reporting from my bureau, yet it is still there now in XX/XX/XXXX. I would like them to remove it immediately and send me a letter indicating that it should not have been there in the first place and they ar...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           product  \\\n",
       "0  Debt collection   \n",
       "1    Consumer Loan   \n",
       "2         Mortgage   \n",
       "3         Mortgage   \n",
       "4         Mortgage   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              consumer_complaint_narrative  \n",
       "0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           XXXX has claimed I owe them {$27.00} for XXXX years despite the PROOF of PAYMENT I sent them : canceled check and their ownPAID INVOICE for {$27.00}! \\nThey continue to insist I owe them and collection agencies are after me. \\nHow can I stop this harassment for a bill I already paid four years ago? \\n  \n",
       "1  Due to inconsistencies in the amount owed that I was told by M & T Bank and the amount that was reported to the credit reporting agencies, I was advised to write a good will letter in order to address the issue and request the negative entry be removed from my credit report all together. I had a vehicle that was stolen and it was declared a total loss by insurance company. The insurance company and the GAP insurancw companypaid the outstanding balance of the loan, but I was told by M & T Bank that there was still a balance due on the loan. In good faith, without having received any proof as to why there was still a balance, I made a partial payment towards the remaining debt. I then sent the goodwill letter still offering to pay the remainder of the debt, but in exchange for the removal of the negative entry on my credit report. At one point, in XXXX 2015, per my credit monitoring agency, it showed a delinquent balance of {$0.00}, but when I checked my credit report again on XXXX X...  \n",
       "2  In XX/XX/XXXX my wages that I earned at my job decreased by almost half, by XX/XX/XXXX I knew I was in trouble with my home loan. I began contacting WFB whom my home loan is with, for assitance and options. \\nIn early XX/XX/XXXX I began the Loan Modification process with Wells Fargo Bank. I was told that they would not assist me with anything financial on my home loan until I fell 90 days behind, though at the time I started to inquire for assistance from WFB I was only a few weeks behind. So, I began working with a program called XXXX. They approved me for a variety of assistence and reached out to Wells Fargo Bank to determine what they could assist with. Wells Fargo then turned down the assistance from XXXX and finally offered to do a Loan Modification for me. The outcome was totally unknow about what I would be offered in the end by WFB for assistance. Wells Fargo lost my paperwork twice during this process, so it took 2 months from the time I started to the time my paperwork b...  \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               I have an open and current mortgage with Chase Bank # XXXX. Chase is reporting the loan payments to XXXX but XXXX is surpressing the information and reporting the loan as Discharged in BK. This mortgage was reaffirmed in a Chapter XXXX BK discharged dated XXXX/XXXX/2013. Chase keeps referring to BK Law for Chapter XXXX and we keep providing documentation for Chapter XXXX, and the account should be open and current with all the payments \\n  \n",
       "4  XXXX was submitted XX/XX/XXXX. At the time I submitted this complaint, I had dealt with Rushmore Mortgage directly endeavoring to get them to stop the continuous daily calls I was receiving trying to collect on a mortgage for which I was not responsible due to bankruptcy. They denied having knowledge of the bankruptcy, even though I had spoken with them about it repeatedly and had written them repeatedly referencing the bankruptcy requesting them to cease the pursuit, they continued to do so. When they were unable to trick me into paying, force me into paying in retaliation they placed reported to my credit bureaus a past due mortgage amount that had been discharged in Federal Court. On XX/XX/XXXX Rushmore responded the referenced complaint indicating that they would remove the reporting from my bureau, yet it is still there now in XX/XX/XXXX. I would like them to remove it immediately and send me a letter indicating that it should not have been there in the first place and they ar...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Demo of product and its complaint example...')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now comes the first step -Doc Preprocessing. Before we create our own word embedding based on the input texts, we need to preprocess the text so that it complies with the input format as Gensim requires. It involves multiple steps starting from word tokenization, bi-gram detection, lemmatization etc..\n",
    "\n",
    "Here, I wrote a python class called **DocProcess**. This class implements all the nitty-gritty jobs mentioned above for us under the hood, such as:\n",
    "\n",
    "1. First, the class takes in a series of texts, then tokenizes the text and removes all punctuations.\n",
    "\n",
    "2. It has the option build_bi, meaning whether to build up bi-gram, function adopted from Gensim. The default is False, if option build_bi is set to True, then the class will train a bi-gram detector and create bi-gram words for the text.\n",
    "\n",
    "3. Now, all the processed tokens are concatenated back to form a sentence again.\n",
    "\n",
    "4. The texts are tokenized once again, but this time, both **stop words** and **parts of speech** that are not allowed in the text will be removed and all tokens are **lemmatized**. These tokens are stored as `self.doc_words` - list of the tokens for each text(doc).\n",
    "\n",
    "5. Finally, these `self.doc_words` are wrapped up into **TaggedDocument**, a object type in Gensim for later use in doc2vec training. It's stored in `self.tagdocs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "\n",
    "nlp = spacy.load('en_core_web_md')\n",
    "stop_words = spacy.lang.en.stop_words.STOP_WORDS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from UtilWordEmbedding import DocPreprocess\n",
    "\n",
    "\n",
    "\n",
    "all_docs = DocPreprocess(nlp, stop_words, df['consumer_complaint_narrative'], df['product'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "\n",
    "# Save all_docs as pickle.\n",
    "with open(os.path.join(dir_path, 'all_docs.pickle'), 'wb') as f:\n",
    "    pickle.dump(all_docs, f, pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "\n",
    "# Read pickle.\n",
    "with open(os.path.join(dir_path, 'all_docs.pickle'), 'rb') as f:\n",
    "    all_docs = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deomo - Structure of Preprocessed Docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UtilWordEmbedding.DocPreprocess"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Demo: structure of preprocessed docs.\n",
    "type(all_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(all_docs.tagdocs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Demo of tagged document...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TaggedDocument(words=['xxxx', 'submit', 'xx', 'time', 'submit', 'complaint', 'deal', 'rushmore', 'mortgage', 'directly', 'endeavor', 'stop', 'continuous', 'daily', 'call', 'receive', 'try', 'collect', 'mortgage', 'responsible', 'bankruptcy', 'deny', 'have', 'knowledge', 'bankruptcy', 'speak', 'repeatedly', 'write', 'repeatedly', 'reference', 'bankruptcy', 'request', 'cease', 'pursuit', 'continue', 'unable', 'trick', 'pay', 'force', 'pay', 'retaliation', 'place', 'report', 'credit', 'bureau', 'mortgage', 'discharge', 'federal', 'court', 'xx', 'xx', 'xxxx', 'rushmore', 'respond', 'reference', 'complaint', 'indicate', 'remove', 'reporting', 'bureau', 'xx', 'xx', 'like', 'remove', 'immediately', 'send', 'letter', 'indicate', 'place', 'go', 'remove', 'bureaus', 'rushmore', 'speak', 'represent', 'new', 'note', 'holder', 'cfpb', 'involve', 'identify', 'servicing', 'agency', 'xxxx', 'credit', 'bullying', 'racial', 'discrimination', 'practice', 'damaging', 'expose', 'tactic', 'need', 'stop', 'deny', 'intent', 'walk', 'away', 'penalty', 'kind', 'reason', 'continue', 'assist', 'procure', 'resolution'], tags=[4])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Demo of tagged document...')\n",
    "all_docs.tagdocs[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Demo of doc words...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['xxxx',\n",
       " 'submit',\n",
       " 'xx',\n",
       " 'time',\n",
       " 'submit',\n",
       " 'complaint',\n",
       " 'deal',\n",
       " 'rushmore',\n",
       " 'mortgage',\n",
       " 'directly']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Demo of doc words...')\n",
    "all_docs.doc_words[4][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label of tagged document...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Mortgage'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print('Label of tagged document...')\n",
    "all_docs.labels.iloc[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build up Word Model - Word2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the text are properly processed, we're ready to train our word2vec via Gensim. Here I chose the dimension size 100 for each word embedding and window size of 5. The training iterates for 100 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of cpu: 8\n"
     ]
    }
   ],
   "source": [
    "import multiprocessing\n",
    "import sys\n",
    "from gensim.models.word2vec import Word2Vec\n",
    "\n",
    "workers = multiprocessing.cpu_count()\n",
    "print('number of cpu: {}'.format(workers))\n",
    "assert gensim.models.doc2vec.FAST_VERSION > -1, \"This will be painfully slow otherwise.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_model = Word2Vec(all_docs.doc_words,\n",
    "                      min_count=2,\n",
    "                      size=100,\n",
    "                      window=5,\n",
    "                      workers=workers,\n",
    "                      iter=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Averaging Word Embedding for Each Doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK! Now we have the word embedding at hand, we'll be using the word embedding to compute for representative vector for whole text. It then serves as feature input for text classification model. There are various ways to come up with doc vector. First, let's start with the simple one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Averaging on Word Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a rather straightforward method. It directly averages all word embedding occurred in the text. Here I adapted the code from these two posts [2](http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/), [3](https://towardsdatascience.com/multi-class-text-classification-model-comparison-and-selection-5eb066197568) and created the python class **MeanWordEmbeddingVectorizer**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:cannot compute average owing to no vector for []\n"
     ]
    }
   ],
   "source": [
    "from UtilWordEmbedding import MeanEmbeddingVectorizer\n",
    "\n",
    "\n",
    "mean_vec_tr = MeanEmbeddingVectorizer(word_model)\n",
    "doc_vec = mean_vec_tr.transform(all_docs.doc_words)\n",
    "\n",
    "# print('Demo of word averaging doc vector...')\n",
    "# display(doc_vec[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of word-mean doc2vec...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(25000, 100)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Save word-mean doc2vec as csv file...\n"
     ]
    }
   ],
   "source": [
    "# Save word averaging doc2vec.\n",
    "print('Shape of word-mean doc2vec...')\n",
    "display(doc_vec.shape)\n",
    "print('Save word-mean doc2vec as csv file...')\n",
    "np.savetxt(os.path.join(dir_path,'doc_vec.csv'), doc_vec, delimiter=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF Weighted Averaging on Word Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not just satisfied with simple averaging? We can further adopt TF-IDF as weights for each word embedding. This will amplify the role of significant word in computing doc vector. Here, the whole process is implemented under class **TfidfEmbeddingVectorizer**. Again, the code is adapted from the same post source.\n",
    "\n",
    "One thing worth noted is that, the Term Frequency has already been considered when we conduct averaging over the text, but not Inverse Document Frequency, thus the weight literally being the IDF, and the unseen word is assigned the max IDF in default setting.\n",
    "\n",
    "And the other thing to note is that we need to fit the class with tokens first, for it must loop through all the words before hand in order to compute IDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from UtilWordEmbedding import TfidfEmbeddingVectorizer\n",
    "\n",
    "\n",
    "\n",
    "tfidf_vec_tr = TfidfEmbeddingVectorizer(word_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:cannot compute average owing to no vector for []\n"
     ]
    }
   ],
   "source": [
    "tfidf_vec_tr.fit(all_docs.doc_words)  # fit tfidf model first\n",
    "tfidf_doc_vec = tfidf_vec_tr.transform(all_docs.doc_words)\n",
    "\n",
    "\n",
    "# print('Demo of tf-idf word averaging doc vector...')\n",
    "# display(tfidf_doc_vec[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of tfidf-word-mean doc2vec...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(25000, 100)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Save tfidf-word-mean doc2vec as csv file...\n"
     ]
    }
   ],
   "source": [
    "# Save tfidf word averaging doc2vec.\n",
    "print('Shape of tfidf-word-mean doc2vec...')\n",
    "display(tfidf_doc_vec.shape)\n",
    "print('Save tfidf-word-mean doc2vec as csv file...')\n",
    "np.savetxt(os.path.join(dir_path, 'tfidf_doc_vec.csv'), tfidf_doc_vec, delimiter=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leverage Pre-train GloVe Word Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load in GloVe Vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's include another option - leveraging the existing pre-trained word embedding and see how it performs in text classification. Here I follow up the instructions from [Stanford NLP course(CS224N) notebook](http://web.stanford.edu/class/cs224n/materials/Gensim%20word%20vector%20visualization.html), importing GloVe word embedding into Gensim to compute for averaging word embedding on text.\n",
    "\n",
    "As a side note, I've also tried to apply Tf-IDF weighted method on GloVe vector, but found out that the result is basically the same as the ones from TF-IDF weighted averaging doc vector. Thus, I omit the demonstration and just include simple averaging on GloVe word vector here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>**APPENDIX: The explanation for logic behind loading pre-train word vector.**</mark>\n",
    "\n",
    "The result of `datapath()` shows that Gensim will try to load in dataset from */Users/XXX/miniconda3/lib/python3.7/site-packages/gensim/test/test_data/*, and calls it `glove_vec_fi`.\n",
    "\n",
    "It then uses `get_tmpfile()` to create a temporary file path to store the word2vec `tmp_word2vec_fi`, which is converted from `glove_vec_fi`.\n",
    "\n",
    "At the final step, **KeyedVectors** then loads in the `tmp_word2vec_fi` as word model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/smart_open/smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
     ]
    }
   ],
   "source": [
    "from gensim.test.utils import get_tmpfile, datapath\n",
    "from gensim.models import KeyedVectors\n",
    "from gensim.scripts.glove2word2vec import glove2word2vec\n",
    "\n",
    "\n",
    "\n",
    "# Load in GloVe vector.\n",
    "glove_vec_fi = datapath('glove.twitter.27B/glove.twitter.27B.100d.txt')\n",
    "tmp_word2vec_fi = get_tmpfile('tmp_glove2word2vec.txt')\n",
    "\n",
    "glove2word2vec(glove_vec_fi, tmp_word2vec_fi)\n",
    "\n",
    "glove_word_model = KeyedVectors.load_word2vec_format(tmp_word2vec_fi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple Averaging on GloVe Word Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply word averaging on GloVe word vector.\n",
    "glove_mean_vec_tr = MeanEmbeddingVectorizer(glove_word_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:cannot compute average owing to no vector for []\n"
     ]
    }
   ],
   "source": [
    "glove_doc_vec = glove_mean_vec_tr.transform(all_docs.doc_words)\n",
    "\n",
    "\n",
    "# print('Demo of GloVe word averaging doc vector...')\n",
    "# display(glove_doc_vec[4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Examine if glove_doc_vec is equal to self-trained doc_vec...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False, False, False, False, False, False, False, False, False,\n",
       "       False])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Examine if glove_doc_vec is equal to self-trained doc_vec.\n",
    "print('Examine if glove_doc_vec is equal to self-trained doc_vec...')\n",
    "glove_doc_vec[4] == doc_vec[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of glove-word-mean doc2vec...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(25000, 100)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Save glove-word-mean doc2vec as csv file...\n"
     ]
    }
   ],
   "source": [
    "# Save glove word averaging doc2vec.\n",
    "print('Shape of glove-word-mean doc2vec...')\n",
    "display(glove_doc_vec.shape)\n",
    "print('Save glove-word-mean doc2vec as csv file...')\n",
    "np.savetxt(os.path.join(dir_path, 'glove_doc_vec.csv'), glove_doc_vec, delimiter=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply Doc2vec Training Directly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PV-DM Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last but not least, we still have one more option - to directly train doc2vec, and no need to average all word embeddings. Here I chose **PV-DM model** to train my doc2vec.\n",
    "\n",
    "The script is mostly referred from [Gensim tutorial](https://github.com/RaRe-Technologies/gensim/blob/3c3506d51a2caf6b890de3b1b32a8b85f7566ca5/docs/notebooks/doc2vec-IMDB.ipynb). And again, to save all the labor, I create a class **DocModel** for it. The class just needs to take in the **TaggedDocument** and then we call `self.custom_train()` method, the doc model will train itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>**APPENDIX: Training with Fixed Learning Rate.**</mark>\n",
    "\n",
    "It's said to achieve better result, but the statement is from a rather old-version gensim tutorial. I found no better training result out of using fixed learning rate. Instead, using the default one, which is also recommended by the new gensim document achieving better performance.\n",
    "\n",
    "Excerpted from [Doc2vec tutorial](https://rare-technologies.com/doc2vec-tutorial/)\n",
    "\n",
    "I have obtained better results by iterating over the data several times and either\n",
    "\n",
    "1. randomizing the order of input sentences, or\n",
    "2. manually controlling the learning rate over the course of several iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from UtilWordEmbedding import DocModel\n",
    "\n",
    "# Configure keyed arguments for Doc2Vec model.\n",
    "dm_args = {\n",
    "    'dm': 1,\n",
    "    'dm_mean': 1,\n",
    "    'vector_size': 100,\n",
    "    'window': 5,\n",
    "    'negative': 5,\n",
    "    'hs': 0,\n",
    "    'min_count': 2,\n",
    "    'sample': 0,\n",
    "    'workers': workers,\n",
    "    'alpha': 0.025,\n",
    "    'min_alpha': 0.025,\n",
    "    'epochs': 100,\n",
    "    'comment': 'alpha=0.025'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a pv-dm model.\n",
    "dm = DocModel(docs=all_docs.tagdocs, **dm_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "dm.custom_train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Demo: doc vector.\n",
    "# dm.model.docvecs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of dm doc2vec...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(25000, 100)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Save dm doc2vec as csv file...\n"
     ]
    }
   ],
   "source": [
    "# Save doc2vec as feature dataframe.\n",
    "dm_doc_vec_ls = []\n",
    "for i in range(len(dm.model.docvecs)):\n",
    "    dm_doc_vec_ls.append(dm.model.docvecs[i])\n",
    "\n",
    "\n",
    "dm_doc_vec = pd.DataFrame(dm_doc_vec_ls)\n",
    "print('Shape of dm doc2vec...')\n",
    "display(dm_doc_vec.shape)\n",
    "\n",
    "print('Save dm doc2vec as csv file...')\n",
    "dm_doc_vec.to_csv(os.path.join(dir_path, 'dm_doc_vec.csv'), index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Target Label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Shape of target labels...')\n",
    "display(all_docs.labels.shape)\n",
    "target_labels = all_docs.labels\n",
    "\n",
    "print('Save target labels...')\n",
    "target_labels.to_csv(os.path.join(dir_path, 'target_labels.csv'), index=False, header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare the Classification Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we've prepared all the necessary ingredients - different types of features. Let's experiment to observe their effect on classification performance. Here, I'll use **basic logistic model** as the base model and feed in different kind of features created earlier. Hence, to compare their effectiveness.\n",
    "\n",
    "In addition to compare effects of each word embedding averaging method, I also try to **concatenate word2vec and doc2vec** together, and see if it can boost up the performance even more.\n",
    "\n",
    "I used TF-IDF weighted word embedding and PV-DM doc2vec together. The result shows that it increases the accuracy on training dataset (perhaps a sign of over-fitting?), but not so significant improvement on testing dataset compared using TF-IDF word2vec alone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Read in saved files.\n",
    "dir_path = '../Private-Dataset'\n",
    "\n",
    "doc_vec = pd.read_csv(os.path.join(dir_path, 'doc_vec.csv'), header=None)\n",
    "tfidf_doc_vec = pd.read_csv(os.path.join(dir_path, 'tfidf_doc_vec.csv'), header=None)\n",
    "glove_doc_vec = pd.read_csv(os.path.join(dir_path, 'glove_doc_vec.csv'), header=None)\n",
    "dm_doc_vec = pd.read_csv(os.path.join(dir_path, 'dm_doc_vec.csv'), header=None)\n",
    "target_labels = pd.read_csv(os.path.join(dir_path, 'target_labels.csv'), header=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "\n",
    "# Classification via Logistic Model\n",
    "logistic = LogisticRegression(random_state=1, multi_class='multinomial', solver='saga')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "# (Optional) Classification via stochastic gradient descent classifier.\n",
    "sgd = SGDClassifier(loss='hinge',\n",
    "                    penality='l2',\n",
    "                    verbose=1,\n",
    "                    random_state=1,\n",
    "                    learning_rate='invscaling',\n",
    "                    eta0=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Demo) Hyper-Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = logistic  # or choose sgd.\n",
    "df = tfidf_doc_vec\n",
    "concate = False\n",
    "concat_df = dm_doc_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define Train Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from UtilTextClassification import split_size\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "def main(model, df, concate, concat_df):\n",
    "    if concate:\n",
    "        df = pd.concat([df, concat_df], axis=1, ignore_index=True)\n",
    "    else:\n",
    "        df = df\n",
    "\n",
    "    # Specify train/valid/test size.\n",
    "    train_size, valid_size, test_size = split_size(df, train=0.7, valid=0.)  # no need to use valid dataset here\n",
    "    # Prepare test dataset.\n",
    "    train_X, test_X, train_y, test_y = train_test_split(df,\n",
    "                                                    target_labels,\n",
    "                                                    test_size=test_size,\n",
    "                                                    random_state=1,\n",
    "                                                    stratify=target_labels)\n",
    "\n",
    "    # Prepare valid dataset.\n",
    "    if valid_size != 0:\n",
    "        train_X, valid_X, train_y, valid_y = train_test_split(train_X,\n",
    "                                                      train_y,\n",
    "                                                      test_size=valid_size,\n",
    "                                                      random_state=1,\n",
    "                                                      stratify=train_y)\n",
    "    \n",
    "    print('Shape of train_X: {}'.format(train_X.shape))\n",
    "    print('Shape of valid_X: {}'.format(valid_X.shape if 'valid_X' in vars() else (0,0)))\n",
    "    print('Shape of text_X: {}'.format(test_X.shape))\n",
    "    \n",
    "    model.fit(train_X, train_y)\n",
    "    \n",
    "    if valid_size != 0:\n",
    "        return model, train_X, valid_X, test_X, train_y, valid_y, test_y\n",
    "    else:\n",
    "        return model, train_X, None, test_X, train_y, None, test_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance on Simple Averaging Word Vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper-Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = logistic  # or choose sgd.\n",
    "df = doc_vec\n",
    "concate = False\n",
    "concat_df = dm_doc_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of train_X: (17500, 100)\n",
      "Shape of valid_X: (0, 0)\n",
      "Shape of text_X: (7500, 100)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/sag.py:326: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "# __main__\n",
    "clf, train_X, valid_X, test_X, train_y, valid_y, test_y = main(model, \n",
    "                                                               df, \n",
    "                                                               concate=concate, \n",
    "                                                               concat_df=concat_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Training Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of Mean Word Vector on training dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[1048   11  141   32   33   22   47    0    0    9    3]\n",
      " [  27  475   55   79  162    3   81    0   16    0    6]\n",
      " [ 114   16 1483  120  157    5   25    0    1    3    1]\n",
      " [  17   40   83 2609  369    2   72    0    1    1   15]\n",
      " [  34   48   84  295 4354    3   87    0   12    1   47]\n",
      " [  37    1    8    0    8  128    3    0    1    3    0]\n",
      " [  29   24   27   65   68    2 3769    0    2    0    6]\n",
      " [  10    0    3    1    8    4    3    0    0    0    0]\n",
      " [   8   19    5    7   67    3   12    0   67    0    3]\n",
      " [  23    0   40    1    1    2    0    0    0   41    0]\n",
      " [   4    8    4   16   71    1   21    0    0    0  517]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.78      0.78      0.78      1346\n",
      "          Consumer Loan       0.74      0.53      0.61       904\n",
      "            Credit card       0.77      0.77      0.77      1925\n",
      "       Credit reporting       0.81      0.81      0.81      3209\n",
      "        Debt collection       0.82      0.88      0.85      4965\n",
      "        Money transfers       0.73      0.68      0.70       189\n",
      "               Mortgage       0.91      0.94      0.93      3992\n",
      "Other financial service       0.00      0.00      0.00        29\n",
      "            Payday loan       0.67      0.35      0.46       191\n",
      "           Prepaid card       0.71      0.38      0.49       108\n",
      "           Student loan       0.86      0.81      0.83       642\n",
      "\n",
      "            avg / total       0.82      0.83      0.82     17500\n",
      "\n",
      "\n",
      "accuracy: 0.828\n",
      "f1 score: 0.824\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "from UtilTextClassification import sk_evaluate\n",
    "\n",
    "\n",
    "# Score on valid dataset.\n",
    "print('Performance of Mean Word Vector on training dataset...')\n",
    "_, _ = sk_evaluate(clf, train_X, train_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Testing Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of Mean Word Vector on testing dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[ 435    5   54   13   17   14   33    1    3    2    0]\n",
      " [  11  200   16   28   80    1   39    1    4    0    8]\n",
      " [  41    6  629   64   64    2   13    0    0    4    2]\n",
      " [   9   26   44 1101  144    1   40    0    2    0    8]\n",
      " [  13   27   50  135 1833    0   42    0   11    1   16]\n",
      " [  23    0    7    2    1   43    3    1    1    0    0]\n",
      " [  12   12    5   29   29    0 1621    0    0    1    2]\n",
      " [   4    0    1    2    4    0    0    1    0    0    0]\n",
      " [   2   15    3    1   26    0    7    0   25    0    3]\n",
      " [   8    0   16    0    2    3    1    0    2   14    0]\n",
      " [   2    3    2   14   29    0    9    0    1    0  215]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.78      0.75      0.77       577\n",
      "          Consumer Loan       0.68      0.52      0.59       388\n",
      "            Credit card       0.76      0.76      0.76       825\n",
      "       Credit reporting       0.79      0.80      0.80      1375\n",
      "        Debt collection       0.82      0.86      0.84      2128\n",
      "        Money transfers       0.67      0.53      0.59        81\n",
      "               Mortgage       0.90      0.95      0.92      1711\n",
      "Other financial service       0.25      0.08      0.12        12\n",
      "            Payday loan       0.51      0.30      0.38        82\n",
      "           Prepaid card       0.64      0.30      0.41        46\n",
      "           Student loan       0.85      0.78      0.81       275\n",
      "\n",
      "            avg / total       0.81      0.82      0.81      7500\n",
      "\n",
      "\n",
      "accuracy: 0.816\n",
      "f1 score: 0.811\n"
     ]
    }
   ],
   "source": [
    "# Score on test dataset.\n",
    "print('Performance of Mean Word Vector on testing dataset...')\n",
    "_, _ = sk_evaluate(clf, test_X, test_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance on Tf-Idf Weighted Averaging Word Vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper-Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = logistic  # or choose sgd.\n",
    "df = tfidf_doc_vec\n",
    "concate = False\n",
    "concat_df = dm_doc_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of train_X: (17500, 100)\n",
      "Shape of valid_X: (0, 0)\n",
      "Shape of text_X: (7500, 100)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/sag.py:326: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "# __main__\n",
    "clf, train_X, valid_X, test_X, train_y, valid_y, test_y = main(model, \n",
    "                                                               df, \n",
    "                                                               concate=concate, \n",
    "                                                               concat_df=concat_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Training Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of Tf-Idf Mean Word Vector on training dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[1035   14  155   26   37   16   51    0    1    7    4]\n",
      " [  24  474   56   82  160    2   87    0   14    0    5]\n",
      " [ 108   18 1471  122  161    5   32    0    2    3    3]\n",
      " [  23   39   77 2608  376    3   66    0    1    0   16]\n",
      " [  38   53  101  301 4322    3   88    0   12    1   46]\n",
      " [  38    1    9    1    8  126    3    0    0    3    0]\n",
      " [  33   24   27   70   65    3 3763    0    1    0    6]\n",
      " [   8    0    4    1    5    5    2    4    0    0    0]\n",
      " [   7   21    8    7   66    3   10    0   68    0    1]\n",
      " [  21    0   36    0    5    2    0    0    0   44    0]\n",
      " [   5    7    4   12   72    1   22    0    1    0  518]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.77      0.77      0.77      1346\n",
      "          Consumer Loan       0.73      0.52      0.61       904\n",
      "            Credit card       0.76      0.76      0.76      1925\n",
      "       Credit reporting       0.81      0.81      0.81      3209\n",
      "        Debt collection       0.82      0.87      0.84      4965\n",
      "        Money transfers       0.75      0.67      0.70       189\n",
      "               Mortgage       0.91      0.94      0.93      3992\n",
      "Other financial service       1.00      0.14      0.24        29\n",
      "            Payday loan       0.68      0.36      0.47       191\n",
      "           Prepaid card       0.76      0.41      0.53       108\n",
      "           Student loan       0.86      0.81      0.83       642\n",
      "\n",
      "            avg / total       0.82      0.82      0.82     17500\n",
      "\n",
      "\n",
      "accuracy: 0.825\n",
      "f1 score: 0.821\n"
     ]
    }
   ],
   "source": [
    "from UtilTextClassification import sk_evaluate\n",
    "\n",
    "\n",
    "# Score on valid dataset.\n",
    "print('Performance of Tf-Idf Mean Word Vector on training dataset...')\n",
    "_, _ = sk_evaluate(clf, train_X, train_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Testing Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of Tf-Idf Mean Word Vector on testing dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[ 440    3   52   14   16   13   33    0    3    3    0]\n",
      " [  12  192   24   26   79    1   42    1    5    0    6]\n",
      " [  49    4  623   66   65    1   13    0    0    3    1]\n",
      " [  12   23   38 1113  144    1   34    0    1    0    9]\n",
      " [  16   26   42  139 1833    1   40    0   10    1   20]\n",
      " [  21    0    7    1    4   44    2    1    1    0    0]\n",
      " [  16    7    9   29   34    0 1612    0    1    1    2]\n",
      " [   4    0    1    2    4    0    0    1    0    0    0]\n",
      " [   3   12    3    1   28    0    5    0   27    0    3]\n",
      " [   5    0   19    0    2    3    1    0    2   14    0]\n",
      " [   1    4    2   11   34    0   10    0    0    0  213]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.76      0.76      0.76       577\n",
      "          Consumer Loan       0.71      0.49      0.58       388\n",
      "            Credit card       0.76      0.76      0.76       825\n",
      "       Credit reporting       0.79      0.81      0.80      1375\n",
      "        Debt collection       0.82      0.86      0.84      2128\n",
      "        Money transfers       0.69      0.54      0.61        81\n",
      "               Mortgage       0.90      0.94      0.92      1711\n",
      "Other financial service       0.33      0.08      0.13        12\n",
      "            Payday loan       0.54      0.33      0.41        82\n",
      "           Prepaid card       0.64      0.30      0.41        46\n",
      "           Student loan       0.84      0.77      0.81       275\n",
      "\n",
      "            avg / total       0.81      0.81      0.81      7500\n",
      "\n",
      "\n",
      "accuracy: 0.815\n",
      "f1 score: 0.810\n"
     ]
    }
   ],
   "source": [
    "# Score on test dataset.\n",
    "print('Performance of Tf-Idf Mean Word Vector on testing dataset...')\n",
    "_, _ = sk_evaluate(clf, test_X, test_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance on Simple Averaging of GloVe Word Vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper-Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = logistic  # or choose sgd.\n",
    "df = glove_doc_vec\n",
    "concate = False\n",
    "concat_df = dm_doc_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of train_X: (17500, 100)\n",
      "Shape of valid_X: (0, 0)\n",
      "Shape of text_X: (7500, 100)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/sag.py:326: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "# __main__\n",
    "clf, train_X, valid_X, test_X, train_y, valid_y, test_y = main(model, \n",
    "                                                               df, \n",
    "                                                               concate=concate, \n",
    "                                                               concat_df=concat_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Training Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of GloVe Mean Word Vector on training dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[ 882    8  197   45  120    6   82    0    1    3    2]\n",
      " [  41  299   55   85  187    0  214    0   10    0   13]\n",
      " [ 121   17 1231  157  319    1   72    0    1    0    6]\n",
      " [  39   34  107 2385  477    1  148    0    1    0   17]\n",
      " [  67   36  111  407 4062    0  240    0    5    0   37]\n",
      " [  67    3   24    4   39   40   12    0    0    0    0]\n",
      " [  45   17   37  101  194    1 3573    0    0    0   24]\n",
      " [   5    0    4    2   10    3    5    0    0    0    0]\n",
      " [  14    5   13   10   56    1   48    0   34    0   10]\n",
      " [  18    0   56    3   15    1    0    0    2   13    0]\n",
      " [   9    6   13   24  104    0  121    0    0    0  365]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.67      0.66      0.66      1346\n",
      "          Consumer Loan       0.70      0.33      0.45       904\n",
      "            Credit card       0.67      0.64      0.65      1925\n",
      "       Credit reporting       0.74      0.74      0.74      3209\n",
      "        Debt collection       0.73      0.82      0.77      4965\n",
      "        Money transfers       0.74      0.21      0.33       189\n",
      "               Mortgage       0.79      0.90      0.84      3992\n",
      "Other financial service       0.00      0.00      0.00        29\n",
      "            Payday loan       0.63      0.18      0.28       191\n",
      "           Prepaid card       0.81      0.12      0.21       108\n",
      "           Student loan       0.77      0.57      0.65       642\n",
      "\n",
      "            avg / total       0.73      0.74      0.72     17500\n",
      "\n",
      "\n",
      "accuracy: 0.736\n",
      "f1 score: 0.724\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "from UtilTextClassification import sk_evaluate\n",
    "\n",
    "\n",
    "# Score on valid dataset.\n",
    "print('Performance of GloVe Mean Word Vector on training dataset...')\n",
    "_, _ = sk_evaluate(clf, train_X, train_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Testing Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of GloVe Mean Word Vector on testing dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[ 384    5   68   17   54    3   43    0    2    1    0]\n",
      " [   9  139   23   28   77    0  100    0    1    0   11]\n",
      " [  51   13  515   77  141    0   24    0    1    2    1]\n",
      " [  19   22   48 1007  209    3   62    0    0    0    5]\n",
      " [  36   22   57  171 1706    1  113    0    7    1   14]\n",
      " [  28    1   14    3   17   14    4    0    0    0    0]\n",
      " [  19   10   12   36   88    1 1535    0    1    0    9]\n",
      " [   3    0    0    2    6    1    0    0    0    0    0]\n",
      " [   5    5    2    1   22    0   29    0   12    0    6]\n",
      " [   6    0   28    0    5    2    0    0    2    3    0]\n",
      " [   4    6    5   10   46    0   52    0    2    0  150]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.68      0.67      0.67       577\n",
      "          Consumer Loan       0.62      0.36      0.45       388\n",
      "            Credit card       0.67      0.62      0.64       825\n",
      "       Credit reporting       0.74      0.73      0.74      1375\n",
      "        Debt collection       0.72      0.80      0.76      2128\n",
      "        Money transfers       0.56      0.17      0.26        81\n",
      "               Mortgage       0.78      0.90      0.84      1711\n",
      "Other financial service       0.00      0.00      0.00        12\n",
      "            Payday loan       0.43      0.15      0.22        82\n",
      "           Prepaid card       0.43      0.07      0.11        46\n",
      "           Student loan       0.77      0.55      0.64       275\n",
      "\n",
      "            avg / total       0.72      0.73      0.72      7500\n",
      "\n",
      "\n",
      "accuracy: 0.729\n",
      "f1 score: 0.717\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "# Score on test dataset.\n",
    "print('Performance of GloVe Mean Word Vector on testing dataset...')\n",
    "_, _ = sk_evaluate(clf, test_X, test_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance on PV-DM Doc2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper-Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = logistic  # or choose sgd.\n",
    "df = dm_doc_vec\n",
    "concate = False\n",
    "concat_df = dm_doc_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of train_X: (17500, 100)\n",
      "Shape of valid_X: (0, 0)\n",
      "Shape of text_X: (7500, 100)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/sag.py:326: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "# __main__\n",
    "clf, train_X, valid_X, test_X, train_y, valid_y, test_y = main(model, \n",
    "                                                               df, \n",
    "                                                               concate=concate, \n",
    "                                                               concat_df=concat_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Training Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of Doc2vec on training dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[ 996   10  133   52   69   22   55    0    0    6    3]\n",
      " [  29  477   40   87  168    3   68    0   18    0   14]\n",
      " [  89   28 1414  154  192    6   22    0    3    9    8]\n",
      " [  25   48   89 2430  514    0   84    0    2    0   17]\n",
      " [  41   53  101  378 4219    3   98    0   21    1   50]\n",
      " [  40    1   12    2   15  109    5    0    2    3    0]\n",
      " [  33   36   28   81   98    3 3695    1    4    0   13]\n",
      " [  10    0    2    2    9    1    0    4    0    0    1]\n",
      " [  16   29    9    8   67    1   12    0   38    0   11]\n",
      " [  11    0   29    1    2    5    0    1    0   59    0]\n",
      " [   8   12    3   17   69    1   22    0    0    0  510]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.77      0.74      0.75      1346\n",
      "          Consumer Loan       0.69      0.53      0.60       904\n",
      "            Credit card       0.76      0.73      0.75      1925\n",
      "       Credit reporting       0.76      0.76      0.76      3209\n",
      "        Debt collection       0.78      0.85      0.81      4965\n",
      "        Money transfers       0.71      0.58      0.64       189\n",
      "               Mortgage       0.91      0.93      0.92      3992\n",
      "Other financial service       0.67      0.14      0.23        29\n",
      "            Payday loan       0.43      0.20      0.27       191\n",
      "           Prepaid card       0.76      0.55      0.63       108\n",
      "           Student loan       0.81      0.79      0.80       642\n",
      "\n",
      "            avg / total       0.79      0.80      0.79     17500\n",
      "\n",
      "\n",
      "accuracy: 0.797\n",
      "f1 score: 0.793\n"
     ]
    }
   ],
   "source": [
    "from UtilTextClassification import sk_evaluate\n",
    "\n",
    "\n",
    "# Score on valid dataset.\n",
    "print('Performance of Doc2vec on training dataset...')\n",
    "_, _ = sk_evaluate(clf, train_X, train_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Testing Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of Doc2vec on testing dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[ 409    8   56   20   32   12   32    0    1    4    3]\n",
      " [   9  194   20   35   76    0   31    0    9    1   13]\n",
      " [  48    8  594   75   83    1   10    0    0    5    1]\n",
      " [  15   24   60 1046  186    0   33    0    1    0   10]\n",
      " [  17   39   46  157 1790    0   44    3    8    2   22]\n",
      " [  20    0   13    5    3   34    2    0    0    3    1]\n",
      " [  13   16    3   45   50    2 1569    0    5    0    8]\n",
      " [   4    1    0    3    3    0    0    0    0    0    1]\n",
      " [   0   11    1    5   34    0    4    0   21    0    6]\n",
      " [   6    0    9    1    1    3    1    0    1   24    0]\n",
      " [   1    4    0   11   29    0    9    0    1    0  220]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.75      0.71      0.73       577\n",
      "          Consumer Loan       0.64      0.50      0.56       388\n",
      "            Credit card       0.74      0.72      0.73       825\n",
      "       Credit reporting       0.75      0.76      0.75      1375\n",
      "        Debt collection       0.78      0.84      0.81      2128\n",
      "        Money transfers       0.65      0.42      0.51        81\n",
      "               Mortgage       0.90      0.92      0.91      1711\n",
      "Other financial service       0.00      0.00      0.00        12\n",
      "            Payday loan       0.45      0.26      0.33        82\n",
      "           Prepaid card       0.62      0.52      0.56        46\n",
      "           Student loan       0.77      0.80      0.79       275\n",
      "\n",
      "            avg / total       0.78      0.79      0.78      7500\n",
      "\n",
      "\n",
      "accuracy: 0.787\n",
      "f1 score: 0.783\n"
     ]
    }
   ],
   "source": [
    "# Score on test dataset.\n",
    "print('Performance of Doc2vec on testing dataset...')\n",
    "_, _ = sk_evaluate(clf, test_X, test_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performance on Tf-Idf and Doc2vec Concatenated Feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper-Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = logistic  # or choose sgd.\n",
    "df = tfidf_doc_vec\n",
    "concate = True  # set to True.\n",
    "concat_df = dm_doc_vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of train_X: (17500, 200)\n",
      "Shape of valid_X: (0, 0)\n",
      "Shape of text_X: (7500, 200)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py:578: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/sag.py:326: ConvergenceWarning: The max_iter was reached which means the coef_ did not converge\n",
      "  \"the coef_ did not converge\", ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "# __main__\n",
    "clf, train_X, valid_X, test_X, train_y, valid_y, test_y = main(model, \n",
    "                                                               df, \n",
    "                                                               concate=concate, \n",
    "                                                               concat_df=concat_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Training Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of Tf-Idf Mean Word Vector and Doc2vec Combined on training dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[1072   14  132   31   29   15   45    0    0    6    2]\n",
      " [  25  546   47   71  131    1   54    0   20    0    9]\n",
      " [  88   14 1533  104  150    3   25    0    0    4    4]\n",
      " [  14   39   66 2690  339    0   46    0    1    0   14]\n",
      " [  30   57   93  269 4373    5   80    1   20    0   37]\n",
      " [  29    0    7    0    3  145    2    0    1    2    0]\n",
      " [  29   25   21   59   56    1 3792    0    3    0    6]\n",
      " [   6    0    3    1    6    2    0    9    1    0    1]\n",
      " [  10   21    7    5   50    0    6    0   89    0    3]\n",
      " [  11    0   23    0    0    0    0    0    0   74    0]\n",
      " [   6    8    1   12   50    1   11    0    2    0  551]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.81      0.80      0.80      1346\n",
      "          Consumer Loan       0.75      0.60      0.67       904\n",
      "            Credit card       0.79      0.80      0.79      1925\n",
      "       Credit reporting       0.83      0.84      0.83      3209\n",
      "        Debt collection       0.84      0.88      0.86      4965\n",
      "        Money transfers       0.84      0.77      0.80       189\n",
      "               Mortgage       0.93      0.95      0.94      3992\n",
      "Other financial service       0.90      0.31      0.46        29\n",
      "            Payday loan       0.65      0.47      0.54       191\n",
      "           Prepaid card       0.86      0.69      0.76       108\n",
      "           Student loan       0.88      0.86      0.87       642\n",
      "\n",
      "            avg / total       0.85      0.85      0.85     17500\n",
      "\n",
      "\n",
      "accuracy: 0.850\n",
      "f1 score: 0.848\n"
     ]
    }
   ],
   "source": [
    "from UtilTextClassification import sk_evaluate\n",
    "\n",
    "\n",
    "# Score on valid dataset.\n",
    "print('Performance of Tf-Idf Mean Word Vector and Doc2vec Combined on training dataset...')\n",
    "_, _ = sk_evaluate(clf, train_X, train_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test on Testing Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance of Tf-Idf Mean Word Vector and Doc2vec Combined on testing dataset...\n",
      "Score on dataset...\n",
      "\n",
      "Confusion Matrix:\n",
      " [[ 440    4   50   13   13   14   32    0    4    6    1]\n",
      " [   9  202   22   25   70    0   32    0   18    1    9]\n",
      " [  44    8  625   60   67    4   10    0    0    6    1]\n",
      " [  11   21   36 1137  130    1   28    0    2    1    8]\n",
      " [  16   36   50  130 1815    0   39    1   18    0   23]\n",
      " [  16    0   12    2    3   43    1    1    2    1    0]\n",
      " [  11   10    6   26   31    1 1617    0    3    1    5]\n",
      " [   4    1    0    2    4    0    0    1    0    0    0]\n",
      " [   3   10    2    1   34    0    4    0   24    0    4]\n",
      " [   8    0    8    0    1    3    1    0    2   23    0]\n",
      " [   1    7    2   13   22    1    6    0    0    0  223]]\n",
      "\n",
      "Classification Report:\n",
      "                          precision    recall  f1-score   support\n",
      "\n",
      "Bank account or service       0.78      0.76      0.77       577\n",
      "          Consumer Loan       0.68      0.52      0.59       388\n",
      "            Credit card       0.77      0.76      0.76       825\n",
      "       Credit reporting       0.81      0.83      0.82      1375\n",
      "        Debt collection       0.83      0.85      0.84      2128\n",
      "        Money transfers       0.64      0.53      0.58        81\n",
      "               Mortgage       0.91      0.95      0.93      1711\n",
      "Other financial service       0.33      0.08      0.13        12\n",
      "            Payday loan       0.33      0.29      0.31        82\n",
      "           Prepaid card       0.59      0.50      0.54        46\n",
      "           Student loan       0.81      0.81      0.81       275\n",
      "\n",
      "            avg / total       0.82      0.82      0.82      7500\n",
      "\n",
      "\n",
      "accuracy: 0.820\n",
      "f1 score: 0.817\n"
     ]
    }
   ],
   "source": [
    "# Score on test dataset.\n",
    "print('Performance of Tf-Idf Mean Word Vector and Doc2vec Combined on testing dataset...')\n",
    "_, _ = sk_evaluate(clf, test_X, test_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (Optional) Grid-Search for Best Hyper-Paremeters on Logistic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>**NOTE:**</mark>\n",
    "\n",
    "The `log_loss()` from sklearn expects to have predicted probs for each class. Thus, instead of simply using `clf.predict(X)`, we need to use `clf.predict_proba(X)` when computing for y_pred. In other words, if we want to use `GridSearchCV`, it's probabily best to directly specify scoring as **'neg_log_loss'**, instead of creating our own scoring func."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import make_scorer, log_loss\n",
    "\n",
    "logistic = LogisticRegression(random_state=1,\n",
    "                              multi_class='multinomial',\n",
    "                              solver='saga')\n",
    "\n",
    "params_log = {\"max_iter\": [100, 200, 500], \"C\": [1, 2, 5]}\n",
    "\n",
    "\n",
    "# Ref: https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter\n",
    "# whether the python function returns a score \n",
    "# (greater_is_better=True, the default) or a loss (greater_is_better=False). \n",
    "# If a loss, the output of the python function is negated by the scorer object, \n",
    "# conforming to the cross validation convention that scorers return higher values for better models.\n",
    "grid_log = GridSearchCV(logistic, param_grid=params_log, scoring='neg_log_loss', cv=5)\n",
    "\n",
    "grid_log.fit(train_X, train_y)\n",
    "\n",
    "print(\n",
    "    \"the best neg_log_loss of all model parameters' combination on model: {:.4f}\"\n",
    "    .format(grid_log.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"the parameter setting of optimized model: {}\".format(\n",
    "    grid_log.best_estimator_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"the accuracy of the optimized model on testing dataset...\")\n",
    "_, _ = sk_evaluate(grid_log, test_X, test_y, label_names=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (Optional) Grid-Search for Best Hyper-Parameters on SGD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<mark>**NOTE:**</mark>\n",
    "\n",
    "The `roc_auc_score` only accepts **binary class**, thus if we have multiple classes, the `roc_auc_score` does not apply."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "sgd = SGDClassifier(loss='hinge', penality='l2', verbose=1, random_state=1)\n",
    "\n",
    "params_sgd = {'learning_rate': ['adaptive', 'invscaling'], \n",
    "              'eta0': [1, 0.5, 0.1]}\n",
    "\n",
    "grid_sgd = GridSearchCV(sgd, param_grid=params_sgd, scoring='accuracy', cv=5)\n",
    "\n",
    "grid_sgd.fit(train_X, train_y)\n",
    "\n",
    "print(\"the best auc of all model parameters' combination on model: {:.4f}\".\n",
    "      format(grid_sgd.best_score_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"the parameter setting of optimized model: {}\".format(\n",
    "    grid_sgd.best_estimator_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"the accuracy of the optimized model on testing dataset...\")\n",
    "_, _ = sk_evaluate(grid_sgd, test_X, test_y, label_names=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}