NearestCentroid.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problema\n",
    "\n",
    "## Introdução\n",
    "\n",
    "O [Reuters](https://www.reuters.com/) é uma agência de notícias britânica mundialmente conhecida por conta do seu conteúdo diversificado e qualidade das postagens. Por possuir uma diversidade muito grande, o Reuters disponibilizou mundialmente um *dataset* contendo mais de 800 mil documentos manualmente categorizados para fins de pesquisa, para o insentivo no aprendizado de máquina e classificação de linguagem natural utilizando mecanismos de *Machine Learning*.\n",
    "\n",
    "## Problemática\n",
    "\n",
    "A problemática do [Reuters RCV1 RCV2 Multilingual, Multiview Text Categorization Test collection Data Set ](https://archive.ics.uci.edu/ml/datasets/Reuters+RCV1+RCV2+Multilingual,+Multiview+Text+Categorization+Test+collection) consiste em realizar a identificação da classe dos documentos dispostos do *Reuters Corpus, Volume 1* que são publicações em Inglês e do *Reuters Corpus, Volume 2* publicados em outras quatro linguas, sendo elas Francês, Alemão, Espanhol e Italiano, totalizando cinto linguas diferentes dos mesmos documentos. Dentro das diversas 134 classes conhecidas das postagens do *Reuters*, o *dataset* possui seis classes selecionadas. São elas: PERFORMANCE, CORPORATE/INDUSTRIAL, GOVERNMENT FINANCE, ECONOMICS, GOVERNMENT/SOCIAL, EQUITY MARKETS.\n",
    "\n",
    "## Disponibilização dos dados\n",
    "\n",
    "Os dados foram disponibilizados no formato parecido com o SVM *light*. Cada linha dos arquivos de dados consiste em um tipo de documento, disponibilizado da maneira:\n",
    "```\n",
    "class feature:value feature:value feature:value ... feature:value\n",
    "```\n",
    "* *Class*: é a categoria do documento, disposta no formato de string;\n",
    "* *Feature*: é uma palavra disposta no texto;\n",
    "* *Value*: é o peso dessa palavra dentro da identificação da classe.\n",
    "\n",
    "Os dados de feature:value são ordenados de acordo com seu peso dentro de cada registro."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import itertools\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_svmlight_file\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.neighbors.nearest_centroid import NearestCentroid\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorias do Documento  \n",
    "Como explicado anteriormente, os textos são divididos em algumas categorias. Para processar os dados, deve-se converter essas strings em números.\n",
    "É importante lembrar que antes de executar o programa deve-se descompactar o arquivo *rcv1rcv2aminigoutte.tar.gz* na pasta do projeto. Isso criará a pasta *rcv1rcv2aminigoutte*, que contém os arquivos necessários para o problema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = {'C15': 4587, 'CCAT': 8745, 'E21': 9625, 'ECAT': 5656, 'GCAT': 5745, 'M11': 45845}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "É necessário realizar um breve tratamento de dados para a permitir realizar a importação dos dados no formato SML Light. Para isso, criou-se o método abaixo que realiza a substituição das strings correspondentes às classes em seus respectivos números."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_datas(file, target_file):\n",
    "    with open(file, 'r') as document_read:\n",
    "        with open(target_file, 'w') as document_write:\n",
    "            for line in document_read:\n",
    "                target = line.split(None, 1)[0]\n",
    "                line_to_write = '%s %s' % (categories[target], line[len(target)+1:len(line)])\n",
    "                document_write.write(line_to_write)\n",
    "        \n",
    "    return target_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Carregando o dataset no formato SVMLight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "file = filter_datas('rcv1rcv2aminigoutte/EN/Index_EN-EN', 'svml_en_en.txt')\n",
    "X,  y = load_svmlight_file(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aplicando a seleção das features\n",
    "\n",
    "Dentro dos problemas que envolvem classificação, uma das partes mais importantes para obtenção de melhores resultados se dá na filtragem e identificação da relevância entre as demais características dos dados. Para isso, existem os métodos de *feature selection* que conseguem, a partir de cálculos matemáticos, identificar quais dessas características de fato são relevantes para a correta categorização ou classificação, aumentando assim a assertividade dos modelos. \n",
    "\n",
    "### Feature selection utilizado\n",
    "\n",
    "Dentro dos diversos métodos de *feature selection*, foram testados os métodos utilizando o LinearSVC e o ExtraTreesClassifier, ambos em conjunto com o SelectFromModel, que é um modelo que remove as *features* identificadas como não importantes à partir de pesos de coeficientes ou *feature_importances*. Em todos os testes, o modelo utilizando o LinearSVC conseguiu reduzir consideravelmente mais *features* do que o ExtraTreesClassifier, e por isso foi o utilizado."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data matrix shape before feature selection: (18758, 21531)\n",
      "Data matrix shape after ExtraTreesClassifier feature selection: (18758, 3030)\n",
      "Data matrix shape after LinearSVC feature selection: (18758, 2683)\n"
     ]
    }
   ],
   "source": [
    "clf = ExtraTreesClassifier()\n",
    "clf = clf.fit(X, y)\n",
    "model = SelectFromModel(clf, prefit=True)\n",
    "Trees_X_new = model.transform(X)\n",
    "\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "lsvc = LinearSVC(C=0.01, penalty=\"l1\", dual=False).fit(X, y)\n",
    "model = SelectFromModel(lsvc, prefit=True)\n",
    "Linear_X_new = model.transform(X)\n",
    "\n",
    "print('Data matrix shape before feature selection: ' + str(X.shape))\n",
    "print('Data matrix shape after ExtraTreesClassifier feature selection: ' + str(Trees_X_new.shape))\n",
    "print('Data matrix shape after LinearSVC feature selection: ' + str(Linear_X_new.shape))\n",
    "X_new = Linear_X_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dividindo o dataset\n",
    "O método *train_test_split* é bem útil para dividir os dados em partes que serão usados para o treino do modelo e os testes, que serão convenientes para avaliar o modelo. Os dados de treino estarão em *x_train* e *y_train*, e os dados de teste em *x_test* e *y_test*, sendo que 25% dos dados serão usados para os testes e os outros 75% para o treinamento."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X_new,\n",
    "    y,\n",
    "    test_size=0.25,\n",
    "    random_state=42,\n",
    "    stratify=y\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Métodos de Classificação\n",
    "\n",
    "### Nearest Centroid/Rocchio classifier\n",
    "O Nearest Centroid é um algorítimo parecido com o KNearestNeighbours(KNN). Enquanto o KNN se basea na classe dos *k* vizinhos mais próximos ao dado a ser observado, o NearestCentroid basea-se no centroide das classes. O centroide mais próximo (distância euclidiana) ao dado a ser observado determina a classe deste dado.\n",
    "\n",
    "O Nearest Centroid utilizado para classificação de texto com vetores tf-idf é conhecido como classificador Rocchio."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Avaliando o modelo\n",
    "\n",
    "Os dados de treino são usados para treinar o modelo através do método *fit*, enquanto os dados de teste são usados no método *predict* para testar o modelo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "nc = NearestCentroid(metric='euclidean')\n",
    "nc.fit(X_train, y_train)\n",
    "y_pred = nc.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matriz de confusão\n",
    "A matriz de confusão nos mostra os números de observações previstas em um grupo conhecido *i*, mas que foram previstos em um grupo *j*, ou seja, relaciona os valores reais com os previstos pelo classificador."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion_matrix(cm, classes,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j]),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_as_number = [4587, 8745, 9625, 5656, 5745, 45845]\n",
    "confusion = confusion_matrix(y_test, y_pred, labels=class_as_number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVcAAAEmCAYAAADWT9N8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd8zdf/wPHXOwmhtlqRGDVjtRESu3btVVtrqw7a0qVLq9uv80vp0KmTbrNGaRQ1Y7W6UFSCorVXhvfvj/tJGkpyxV3S99Pj83A/87xzk7xz7vmczzmiqhhjjPGsIH8HYIwxOZElV2OM8QJLrsYY4wWWXI0xxgssuRpjjBdYcjXGGC+w5GqyJCJ5RWSWiBwWkU8v4To3iMgCT8bmLyLSRER+9XccJnCJ9XPNOUSkH3AXEAkcBTYAT6nqsku8bn/gdqChqqZccqABTkQUqKyqW/0di7l8Wc01hxCRu4D/AU8DJYGywCtAFw9cvhzw238hsbpDREL8HYO5DKiqLZf5AhQCjgE9MzkmFFfy3e0s/wNCnX3NgATgbmAfsAcY7Ox7DEgCkp0yhgLjgA8yXLs8oECIsz4I+B1X7Xk7cEOG7csynNcQWAMcdv5vmGFfHPAEsNy5zgKg2AW+trT478sQf1egPfAb8DfwYIbjY4EVwCHn2ElAbmffd87Xctz5entnuP4YYC/wfto255yKThnRznppYD/QzN8/G7b4b7Gaa87QAMgDfJnJMQ8B9YEo4BpcCebhDPtL4UrS4bgS6GQRKaKqj+KqDU9X1fyq+lZmgYhIPmAi0E5VC+BKoBvOc1xRYI5z7JXAi8AcEbkyw2H9gMFACSA3cE8mRZfC9R6EA48AbwA3AnWAJsBYEbnKOTYVGA0Uw/XetQRuA1DVa51jrnG+3ukZrl8UVy1+eMaCVXUbrsT7gYhcAbwDTFXVuEziNTmcJdec4UrggGb+sf0G4HFV3aeq+3HVSPtn2J/s7E9W1bm4am1VsxnPGaCmiORV1T2quvk8x3QAtqjq+6qaoqofA78AnTIc846q/qaqJ4FPcP1huJBkXO3LycA0XIlzgqoedcr/CdcfFVQ1XlVXOuXuAF4HmrrxNT2qqqedeM6iqm8AW4FVQBiuP2bmP8ySa87wF1Asi7bA0sDODOs7nW3p1zgnOZ8A8l9sIKp6HNdH6VuAPSIyR0Qi3YgnLabwDOt7LyKev1Q11Xmdlvz+zLD/ZNr5IlJFRGaLyF4ROYKrZl4sk2sD7FfVU1kc8wZQE3hZVU9ncazJ4Sy55gwrgNO42hkvZDeuj7RpyjrbsuM4cEWG9VIZd6rqfFVtjasG9wuupJNVPGkxJWYzpovxKq64KqtqQeBBQLI4J9NuNSKSH1c79lvAOKfZw/yHWXLNAVT1MK52xski0lVErhCRXCLSTkSedQ77GHhYRIqLSDHn+A+yWeQG4FoRKSsihYAH0naISEkR6eK0vZ7G1bxw5jzXmAtUEZF+IhIiIr2B6sDsbMZ0MQoAR4BjTq361nP2/wlUuMhrTgDWquowXG3Jr11ylOayZsk1h1DVF3D1cX0Y153qXcBI4CvnkCeBtcAm4AdgnbMtO2UtBKY714rn7IQY5MSxG9cd9Kb8O3mhqn8BHXH1UPgL153+jqp6IDsxXaR7cN0sO4qrVj39nP3jgKkickhEemV1MRHpArTln6/zLiBaRG7wWMTmsmMPERhjjBdYzdUYY7zAkqsxxniBJVdjjPECS67GGOMFOX4ACsmdTyVPYX+HkS6yXAl/h3CWvLmD/R3CWQLx9mpWHWB9LTXAbkJvXL/ugKoW99T1gguWU03510Nw56Un989X1baeKtuTcn5yzVOY0Loj/B1GuvfeCpxYAGqWKeTvEM6Sknq+LrH+FRIcWB/wjp8KrMHJihXIde6TdpdEU04SWjXLHnAAnNowOasn6/wmxydXY8zlRkAC6w9adlhyNcYEFgGCAqu5KjssuRpjAo8EWkv3xbPkaowJMDmjWeDy/wqMMTmPiHtLlpeRt0Vkn4j8mGFbURFZKCJbnP+LONtFRCaKyFYR2SQi0RnOGegcv0VEBrrzJVhyNcYEFsFVc3Vnydq7uAbVyeh+YJGqVgYWOesA7YDKzjIc19CUabNmPArUwzWDx6NpCTkzllyNMQHGzVqrGzVXVf0O1+hsGXUBpjqvp/LPOMhdgPfUZSVQWETCgDbAQlX9W1UPAgv5d8L+F2tzNcYEHvd7CxQTkbUZ1qeo6pQszimpqnuc13txzZYMrlkwdmU4LsHZdqHtmbLkaowJMBd1Q+uAqtbNbkmqqiLilUferFnAMeL6GNa+dRPxb9/EyO4xALw/tisrpwxl5ZSh/PLRbaycMjT9+Hv6NuDH929h49SbaVX3qgtdNlv27k7gln4d6XVdPXq1qc/H77wKwIRnxtKjVQx92zXk3ltu4OiRQwCkJCcz7p5b6NO2IT1bx/LOKy96NJ6sLJg/j6trVKVGZCWee3a8T8sGuHX4UK4qU4rY6KvTt23auIHm1zakYWw01zaMZe2a1T6PC2DXrl20adWc2ldXJ/qaGkyaOMEvcdSuUYkm9aJo1rAOLa+tB8APmzbQpnmj9G3r1vrnPfoXwWPNAhfwp/NxH+f/fc72RKBMhuMinG0X2p4pS65A9fLFGdwhiia3vUPssDdpV78SFUoXof8TX1F/+FvUH/4WX333KzOW/gpAZLli9GxRneghb9B5zDQmjGpLUJDn+uWFhIQw6sEn+WTBKt75fCGfvf8mv2/5hXqNmzNt3go+/vp7ypavxLuvvATAN3O/IikpiWnzvuf9mXF8+fE77E7w6BOJF5SamsqoO0YwY9bXrN/0E59O+5iff/rJJ2WnuaH/QL6cOfesbWMfHMMDD43l+9XreOiRcYx98P4LnO1dISEhjH/2BdZv+okly1by+muTff7+pPlqzjfEfR/Pou9WAfDY2Ae494GxxH0fz/0PjWPc2AeyuIIPee6G1vnMBNLu+A8EZmTYPsDpNVAfOOw0H8wHrhORIs6NrOucbZmy5ApElruSNT8ncvJ0CqlnlKUb/6Brk7Nnle7erBqfLHbNEN2xYWU+XfwTScmp7Nx7mG2JB4mJLH2+S2dLsRKliKzpmkU6X/4ClK9Uhf1791C/SQtCQlwtOTVr1+XPva75BUWEkyeOk5KSwqlTp8iVKzf58hf0WDyZWbN6NRUrVuKqChXInTs3PXv3YfasGVmf6EGNm1xLkSJnzwcoIhw9cgSAI4cPExYW5tOY0oSFhVE72tWjp0CBAkRGVmP3bl/MwZg1EeHoUec9OnKYUmGe+xm+NOKx5CoiH+OawLOqiCSIyFBgPNBaRLYArZx1cM3r9juuKdLfAG4DUNW/gSeANc7yuLMtU9bmCmzevp9xQ5pStGBeTp5Opm29iqz77Z9ZnRtdXYY/Dx5nW+JBAMKLF2DVT//8giTuP0LpYgW8EtvuhJ38uvkHakTVOWv7zE8/oHXH6wFo2a4LS76ZS7v6VTl18iSjH36aQoWz7Cnimfh2JxIR8c8npvDwCFavXuWTsjMz/vmX6NaxHQ/dfx9n9AzffLvM3yGxc8cONmxYT0xsPZ+XLSL06NoOEWHg4JsYOOQmnhr/Aj27deDRh8Zw5swZvv7mO5/HdV4CBHvm8VdV7XuBXS3Pc6wC5x1ZSVXfBt6+mLL9mlxFpBSu6YhjgEO4Zt0cBUwE6gPLVLVjhuPfxTXh3WFn0yBV3XCpcfz6x1+8MG0ls57tw4lTyWzcto/UM/+MztSrRQ0+dWqtvnTi+DHG3DaAu8Y+Tf4C/9RE3578PCEhIbTr4ho5aPPGeIKCgvl6xS8cOXyIm3q3I7ZRMyLKlvd5zIHirSmvMf65F+jSrTtffPYJI265iVlfL/BbPMeOHaNvr+4898L/KFjQN58qMpqzII6w0uHs37+PHp3bUrlKJLNmfM6T45+nU5fr+eqLT7lzxHC+mJXlp13fyAGPv/qtWUBEBPgSiFPViqpaB9cUzSWB54D+Fzj1XlWNcpZLTqxppn69kUa3vEPrUR9w6OhJtuxy1fqDg4Qujavy2bc/px+buP8oEcX/+QUJL16Q3QeOeioUwHWTasxtA2jbuSct2nZO3z7rsw9Ztng+T7z0BuL8AM6b+RkNm7YkJFcuihYrzjV16vHzD+s9Gs+FlC4dTkLCP71UEhMTCA/PspeK1330wXt07uqq2Xfr3pN4P96sSU5Opm+v7vTuewNdu13vlxjCSru+J8WLl6B9p66si1/DtI/ep2PnbgB06daDdfFr/BLbv3muWcCf/BldcyBZVdPnd1fVjaq6VFUX4Zr22GeKF74CgDIlCtKlSSTTF7lqqi3qXMVvu/4iMUPynLNiCz1bVCd3rmDKlSpEpfAirPllt8diUVWeuH8k5StW4YZhI9O3f7/kG96fMpEXpnxMnrxXpG8vVTqCNd+7PtKdPHGcHzespXyFyh6LJzN1Y2LYunULO7ZvJykpiU+nT6NDx85Zn+hlpcJKs+y7JQAs+XYxFSv55v04l6pyy01DqRpZjTtH3+WXGI4fP87Ro0fTX8ctWki16jUoVao0y5e5fm6WLvmWChUr+SW+8/JubwGf8GezQE1cc95frKdE5BGcx9ZU9fS5B4jIcFyPr0Goe4NBfzyuO0UL5iU5NZVRE+Zz+Ljrsj2bV0+/kZXm5x0H+DzuZ9a/M5yU1DOMmjifM2c811Vu49qVzP1yOpWqVqdfh8YAjLjnEZ5/fAxJSUmMGOB6oKRWVAwPPPUSPfsP4/H7RtCrTX1QpVOPG6hcrabH4slMSEgIL02YRKcObUhNTWXgoCFUr1HDJ2WnGdy/H0uXLuGvAweoWrEsDz78KC+/8jpj7hlNSkoKefLkYeLk17K+kBd8v3w5H334PjVr1qJeHddNyseefJq27dr7LIb9+/5kYL8eAKSkpNK9Vx9atm5Dvnz5eHDMXaSmpBCaJw8vTnzVZzFlKcBrpe4Q9dOUESJyB3CVqo6+wP5mwD3ntLmG4XqiIjcwBdimqo9nVk5QwXANpJkIltpMBJmymQiyFoAzEcRfSkf+cwUVKqOh9e9069hTC+71aNme5M+fms1AnSyPykBV9zjP/Z4G3sE1iIIxJqcJCnZvCWD+TK6LgVDnIzwAInK1iDS50AkZnqoQXIMt/HihY40xlyu7oXVJnD5l3YBWIrJNRDYDzwB7RWQp8CnQ0un428Y57UMR+QH4ASgGPOmP2I0xXmY3tC6Nqu4GzjfN43lrr6rawrsRGWP8Lm0818ucPaFljAkwOWOaF0uuxpjAE+A3q9xhydUYE3gCvD3VHZZcjTGBRaxZwBhjvMNqrsYY43liydUYYzzLNcuLJVdjjPEsEcSD0yb5iyVXY0zAsZqrMcZ4gSVXY4zxAkuuxhjjaeIsl7kcn1yjKoexdF7gzMderLt/RsS/kP2f3ezvEM5yNMAGggbIHxpYvyZXhF7+j4ZmRhCCguwhAmOM8ThrFjDGGC+w5GqMMZ5mba7GGOMdVnM1xhgPE8SSqzHGeIM9/mqMMZ4m1ixgjDFeYcnVGGO8wJKrMcZ4WE65oXX5P2PmBbcOH0L5iJLE1K6Vvu2pJ8ZR+aoIGsTUpkFMbeZ/PderMYzoVIu1k3oTP7k3IztfDcDTgxuw4dU+rJ7Yi+kPtqFQvtwAlC1RgL8/u4mVE3qyckJPJt52rVdjS9i1i/bXtaRuVE1iatfilUkTAXhi3CPUrxtFw9hounRow57du70aR0ZTJk+gWf0omjeoza1D+3Pq1ClUlfFPPELjOjW4NvZq3nxtkk9iSdi1iw5tWhJTuyax0f+8P5s2bqDFtQ1pVC+apo1iWbtmtU/iOdehQ4fo17snUTWrUbtWdVatXOGXOC5IXDe03FkCmdVcz+OG/oO4+daR3DRk4FnbR94+ijvvusfr5VcvW5TBbarT5O7PSUpOZeZjHZm7ZgeLNuxi7NSVpJ5RnhxYn3t7RPPw1JUA/L73CPXv/NTrsQGEhITw9P89R1TtaI4ePUqTBjG0aNmKO++6h7HjHgfg1ckvM/7pJ5gw6VWvx7NndyJvvT6ZuFUbyZs3LzcP6seMzz9BVdmdkMB3a34gKCiIA/v3eT0WcL0/T43/5/25tqHr/Rn70Bjuf2gs17Vpx/x5c3nkofuZu2CxT2LK6N67RtG6TRs+mv4pSUlJnDhxwucxZMWTNVcRGQ0MAxT4ARgMhAHTgCuBeKC/qiaJSCjwHlAH+Avorao7slOu1VzPo3GTaylSpKjfyo8sU5g1v/7JydMppJ5Rlv64m64NKrBofQKpZxSA1b/+SXixfH6Jr1RYGFG1owEoUKAAVSMj2Z2YSMGCBdOPOX78uE8/2qWkpnLq1ElSUlI4efIEJcPCeO/tKYwe82D6ICDFipfwSSznfX92JyIiHD1yBIAjhw9TKizMJ/FkdPjwYZYt+45Bg4cCkDt3bgoXLuzzOLIiIm4tblwnHLgDqKuqNYFgoA/wf8BLqloJOAgMdU4ZChx0tr/kHJctllwvwuuvTaZenWu4dfgQDh486LVyNu/8m0Y1wihaIJS8oSG0rVuWiGL5zzpmQOtI5sf/kb5evmQBVvyvBwue6UKj6r77pd25YwebNmygbmw9AB575GEiK5bjk2kf8dAjj/kkhrDS4dw6chQxNSsRVbUcBQoWolmL1uzc/jszv/iMts0acEOPTvy+bYtP4slo507n/Ympx/899xJjHxxDtUrlePiB+xj3+NM+j2fH9u0UK1acm4cNoX5MNLfePIzjx4/7PI4siZuLe0KAvCISAlwB7AFaAJ85+6cCXZ3XXZx1nP0tJZu1BK8nVxEpJSLTRGSbiMSLyFwRqeIsc0Vki4isE5FPRKRkhvP+JyKJIq4JzEVksIhscJYkEfnBeT3e218DwLDht/LDz1tZsWY9JUuF8eCYu71W1q8Jh3jh8/XMerwTM8d1YOPvf6XXWAHu6xVNauoZpsW5ksXev49TZcj7NBj1GWPeXM6797SiQN5cXosvzbFjx7ixb0/GP/9ieq310cef5JdtO+nVpx9TXp3s9RgADh06yPy5s1m18VfW/7KDE8eP8/n0jziddJrQ0FDmxa3ghgFDuWukb4dXPHbsGP379mT8c673580pr/HMsy/w89adPPPsC4y89SafxgOQkprChvXrGHbzLaxcs458+fLx/LM++RW6KBdRcy0mImszLMMzXkdVE4HngT9wJdXDuJoBDqlq2viWCUC48zoc2OWcm+Icf2V2vgavJlcn438JxKlqRVWtAzwAlATmAK+qamVVjQZeAYo75wUB3XB9kU0BVPUdVY1S1ShgN9DcWb/fm19DmpIlSxIcHExQUBCDh9zE2jVrvFre1IW/0Gj0Z7R+YAaHjp1my+5DANzYsirtY8ox6IVF6ccmpZzh76OnAVi/7QC/7z1M5XDvftRLTk7mxj496NWnH126Xv+v/b379GPGV194NYY0S+MWU6Zcea4sVpxcuXLRvlNX1q5eQVjpcNp3clVI2nXqws+bf/BJPOC8P3170Kt3Pzo778/HH76X/rpb957Er/X9Da3w8AjCIyKIdT5pdLu+Bxs2rPd5HJlxN7E6yfWAqtbNsEw551pFcNVGrwJKA/mAtr74Orxdc20OJKtq+gjRqroRqAysUNVZGbbHqeqPzmozYDPwKtDXyzG6Ze+ePemvZ834kuo1anq1vOKF8gJQpnh+ujS8iulLttA6ugx3XR9Fjye+5uTpfwaVLlYwD0HOndPyJQtQqXQhtu894rXYVJURNw+jamQ1br9zdPr2rVv/+dg9Z/ZMqlSt6rUYMgqPKMO6tas4ceIEqsqyJd9SqUokbTt0ZvnSJQCsWPYdFSpW9kk8qsqIW4ZRtWo1RmZ4f0qFlWaZE8+SuMVUrOSbeDIqVaoUERFl+O3XXwH4dvEiqlWr5vM4shIUFOTW4oZWwHZV3a+qycAXQCOgsNNMABABJDqvE4EyAM7+QrhubF00b/cWqImrCu7u9jR9gY+BGcDTIpLLeWN8YlD/fiz9Lo6/DhygSoUyPDR2HEu/W8KmjRsQEcqVK8/Eyd6dUeDjB9pQtEAoyalnGPXqUg4fT+Klm5sQmiuY2U90Alw3te545Tsa1yzN2BtiSE45wxlVbp/8HQePnfZabCu+X87HH31AjZq1aBjrunHz6ONP8t67b7Plt98ICgqiTNmyTHjZ+z0FAKLrxtKh8/W0aVqPkJAQataK4sZBwzh18iQjhw/kjVcnki9ffp6f6JtZIFZ+v5xpzvvTqJ7r/XnksSd5efLrjLl3NCkpKYSG5mHCJP/MSvHCSxMZPPBGkpOSKH9VBV5/822/xJEpz90L/QOoLyJXACeBlsBa4FugB64eAwNx5RqAmc76Cmf/YlXVcy/qDsnmee5dXOQO4CpVHX3O9heBnao64Tzn5Aa2A5GqelREvgDeVtXZGY7Zgevu34ELlDscGA5QpmzZOj9v2eGhr+jS2TQvmbNpXrIWEhxY/TuvyB0Ur6p1PXW90JKVNfyGf6WG89r+UocsyxaRx4DeQAqwHle3rHBcibWos+1GVT0tInmA94HawN9AH1X9PTtfh7d/ajbjyv7n2970Aue0AQoDPzhtKml/cWZf4Ph/cdpdpgBE16nrvb8exhjP8/DALar6KPDoOZt/B2LPc+wpoKcnyvV2m+tiIDTjHTwRuRr4DWgoIh0ybL9WRGriahIYpqrlVbU8robo1k613hiTwwkg4t4SyLyaXJ22im5AK6cr1mbgGWAv0BG43emK9RNwG3AU1528ORmucRxYBnTyZqzGmEBxUb0FApbXG5NUdTfQ6wK7z9cl4l+PRqnq9eesl7/0yIwxgSoowMcNcEdgtdQbY8xl8JHfHZZcjTEBRbCaqzHGeIXVXI0xxgsC/WaVOyy5GmMCiog1CxhjjBcEfjcrd1hyNcYEnByQWy25GmMCj9VcjTHG06yfqzHGeJ5rbIHLP7tacjXGBBzrLWCMMV6QAyquOT+5ChAcQH8FA21w6oS/T/o7hLMUusL7Eyte7nLCR+ZMeXg8V3/J8cnVGHN5SRvP9XJnydUYE2DsIQJjjPEKu6FljDGeZv1cjTHG86yfqzHGeIklV2OM8YIckFstuRpjAo/VXI0xxsNExHoLGGOMN+SAiqslV2NM4AnKAdk1yN8BBLoF8+dxdY2q1IisxHPPjvd5+Qm7dtH+upbUjapJTO1avDJpIgBPjHuE+nWjaBgbTZcObdize7dX43ho9K00qlWeTs1j0reNvnkA3Vo1oFurBrSMrU63Vg3S9015+XnaNLyado1rsyzuG6/GBjBl8gSa1Y+ieYPa3Dq0P6dOnaJruxa0ahxDq8Yx1I4sz+B+PbweB7i+Zx3atCSmdk1io//5nm3auIEW1zakUb1omjaKZe2a1T6J51z+/pl2h4h7SyATVfV3DF5Vp05dXb5qbbbOTU1NpVb1Ksz5eiHhERE0rh/D1A8+plr16tmOJyX1zEUdv3fPHvbu3UNU7WiOHj1KkwYxTPv0C0qHR1CwYEEAXp38Mr/8/BMTJr160fG4O3DLmpXLuOKK/Nx/503M+nbNv/b/32MPkL9AQUbc9QBbf/uZe24bzCdzlrDvzz0M6d2Jr5dtIDg4OMtysjNwy57diXRt25y4VRvJmzcvNw/qR4vWbel9w4D0Y4b1702b9p3o2ffGi75+/tCL+4B37vfs2oYxfPzJF4y5dzQjbh/FdW3aMX/eXCa8+DxzFyy+6HhyhWS/TuSNn+m8uSReVetm+wLnKFSumtYf865bxy4YUd+jZXvSBb9LIlIws8WXQfrLmtWrqVixEldVqEDu3Lnp2bsPs2fN8GkMpcLCiKodDUCBAgWoGhnJ7sTE9MQKcPz4ca/fXY2p35jCRYqcd5+qMm/mF3To2hOAxfPn0L5LD3KHhhJRtjxly1dg0/rs/YFzV0pqKqdOnSQlJYWTJ09QMiwsfd/RI0dY/l0cbTt09moMac77PdudiIhw9MgRAI4cPkypDDH6SiD8TLsjOEjcWgJZZn+SNwOK64GJNGnrCpT1YlwBYffuRCIiyqSvh4dHsHr1Kr/Fs3PHDjZt2EDd2HoAPPbIw3z84fsULFSIOfMX+S2utauWc2XxEpSvUAmAP/fs5po6sen7S4aFs2+v95otwkqHc+vIUcTUrESePHlp2qIVzVq0Tt8/b85MGjdtToGCvq8T7NzpfM9i6vF/z71Et07tePiB+zhz5gwLv13m83gC7Wf6QgL9I787LlhzVdUyqlrW+b/MOeseSawikioiGzIs9zvbPxSRX0XkRxF5W0RyOdsjRWSFiJwWkXs8EcPl4tixY9zYtyfjn38xvdb66ONP8su2nfTq048pr072W2xzvvo0vdbqD4cOHWT+3Nms2vgr63/ZwYnjx/l8+kfp+7/6fDpdu/f2eVzHjh2jf9+ejH/O9T17c8prPPPsC/y8dSfPPPsCI2+9yecxXQ4EEDf/BTK3Gm9EpI+IPOi8jhCROh4q/6SqRmVY0lrXPwQigVpAXmCYs/1v4A7geQ+Vn6nSpcNJSNiVvp6YmEB4eLgvij5LcnIyN/bpQa8+/ejS9fp/7e/dpx8zvvrC53EBpKSk8M3cmbTr3D19W8mw0uzdnZC+/ueeREqUKu21GJbGLaZMufJcWaw4uXLlon2nrqxdvQKAv/46wIb4tbRs085r5Z9PcnIyN/btQa/e/ejsfM8+/vC99Nfduvckfq3vb2gFys90VoLEvcUdIlJYRD4TkV9E5GcRaSAiRUVkoYhscf4v4hwrIjJRRLaKyCYRic721+BGYJOA5kB/Z9MJ4LXsFugOVZ2rDmA1EOFs36eqa4Bkb5afpm5MDFu3bmHH9u0kJSXx6fRpdOjom3a7NKrKiJuHUTWyGrffOTp9+9atW9Jfz5k9kypVq/o0rjQrln7LVZWqUKr0P7+gza9rz9wZn5F0+jQJf+xg5/ZtXF3be/ccwiPKsG7tKk6cOIGqsmzJt1SqEgnAnBlf0KpNe/LkyeO18s+lqoy4ZRhVq1ZjZIbvWamw0ixbugSAJXGLqVipss9iShMIP9NZEtd4ru4sbpoAzFPVSOAa4GfgfmDD9DAWAAAgAElEQVSRqlYGFjnrAO2Ays4yHLj4u8QOd26DNlTVaBFZD6Cqf4tI7uwWeI68IrIhw/ozqjo9bcVpDugP3HkxFxWR4bjeGMqUzX4LRkhICC9NmESnDm1ITU1l4KAhVK9RI9vXy44V3y/n448+oEbNWjSMdf0RffTxJ3nv3bfZ8ttvBAUFUaZsWSa8nO2fAbfcfesgVq9YyqG//6JZnSqMvPshevQbyNwZn/2rSaBy1eq07XQ9HZvVJTg4hLFPv+hWT4Hsiq4bS4fO19OmaT1CQkKoWSuKGwe5PuzM+PxTRo72bQvSyu+XM835njWq5/qePfLYk7w8+XXG3DualJQUQkPzMGGSV+so5xUIP9Pu8FSbq4gUAq4FBgGoahKQJCJdgGbOYVOBOGAM0AV4z6nYrXRqvWGquueiy86qK5aIrAIaAGudJHsl8I2q1r7Yws5z7WOqmj+T/W8Ax1V11DnbxwHHVDXL5oFL6YrlDRfbFcvbbA6trF1sVyxvu5SuWN7g6a5YRcpX1xaPvO/WsV8MrbsTOJBh0xRVnZK2IiJRwBTgJ1y11nhclbVEVS3sHCPAQVUtLCKzgfGquszZtwgYo6oXnUTc+amZDHwOFBeRx4BewGMXW9DFEpFHgeJAYM3oZ4zxuov4yH8gi8QeAkQDt6vqKhGZwD9NAACoqoqIxzv8Z5lcVfU9EYkHWjmbeqrqj54OJCMRGQa0AVqqamBV9YwxXuXhp68SgARVTetv9hmu5Ppn2sd9EQkD9jn7E4EyGc6PcLZdNHc/XwTjuomUdBHnuCPvOV2x0noLvAaUBFY42x8BEJFSIpIA3AU8LCIJ/5UHGoz5LwkScWvJiqruBXaJSNod35a4mghmAgOdbQOBtCcpZgIDnF4D9YHD2WlvBTdqriLyENAP+BJXF7SPRORDVX0mOwVmpKrnvcuhqueNy3mjIi61XGNMYPNwD9bbgQ+dG/G/A4NxVRI/EZGhwE5czZ0Ac4H2wFZcPaMGZ7dQd9pcBwC1VfUEgIg8BawHLjm5GmPMuQQ8+mirqm4Aztcu2/I8xyowwhPlupNc95xzXIizzRhjPO/i+rAGrAsmVxF5CdcYAn8Dm0VkvrN+HfDvYZGMMcZDckBuzbTmmtYjYDMwJ8P2ld4LxxhjcvgcWqr6li8DMcYYcLW5Bvhogm5xp7dAReApoDqQ/oC2qlbxYlzGmP+wnFBzdafP6rvAO7j+oLQDPgGmZ3aCMcZklwgEi7i1BDJ3kusVqjofQFW3qerDuJKsMcZ4RU6YQ8udrlinRSQI2CYit+B6FKyAd8MyxvyX5YRmAXeS62ggH65Bqp8CCgFDvBmUMea/LQfkVrcGbkkb8OAo/wyYbYwxXiG4N25AoMvsIYIvcT00cF6q+u/5RkyWTqcE1iBf5Yvn83cIZ/lsY0LWB/lYq0ol/B3CWQqHeGqs+gAlEJQD+mJlVnOd5LMojDEmg8AaDjx7MnuIwH9zNRtj/rOE/84NLWOM8akc0CpgydUYE3j+U8lVREJV9bQ3gzHGGNcDApd/ds2y3VhEYkXkB2CLs36NiLzs9ciMMf9ZwUHuLYHMnfAmAh2BvwBUdSPQ3JtBGWP+u1yjYnlmDi1/cqdZIEhVd55TTU/1UjzGGJOzu2JlsEtEYgEVkWBck3395t2wjDH/ZQFeKXWLO8n1VlxNA2WBP4FvnG3GGONxchl85HeHO2ML7AP6+CAWY4wB/iM1VxF5g/OMMaCqw70SUYBZMH8e99x1J6mpqQwaMox777vf5zHUrl6J/PnzExwcTHBICIuWusbSeePVSbw15TWCg4Np3bYd454c7/PYdu3axbDBA9i3709EhCFDhzPyjju9Xm7S6VM8dVMPkpOTOJOaSkzL9nS/+W6eGHY9p04cB+DI3weoUCOK0S+8RXzcfD5/7XkkKIjg4GBuuHscVaNivRbf65Mn8NH77yAiVKtek5cmv8Fdt9/MpvXxhOTKRe3oGJ7932Ry5crltRjO5+ZhQ/h67myKlyhB/IYfsz7BDwQIyQEdXd1pFvgmw+s8QDdgl3fCCSypqamMumMEc75eSHhEBI3rx9CxY2eqVa/u81i+mvsNVxYrlr6+dEkcX8+ZxZKV8YSGhrJ/3z6fxwQQEhLC+GdfoHZ0NEePHqVhvTq0bNXa6+9RrtyhPPDadPJckY+UlGSeGHo91zRsztg3v0g/ZsK9w6nT9DoAasQ2JrrpdYgIf2z5mUn338qzn8d5JbY9uxN56/XJLFm1kbx58zJ8UD9mfP4J3Xv2YfKUdwG4bdgAPnrvbQYOvdkrMVxI/4GDuOW2kQwbMsCn5V6snFBzzfKmnKpOz7BMBa4H6ng/NP9bs3o1FStW4qoKFcidOzc9e/dh9qwZ/g4LgHfffJ07776P0NBQAIqX8M/ITWFhYdSOjgagQIECREZWY/fuRK+XKyLkucI1oldqSgqpKSln/UaePHaUn9Z+T51mbQDIc0W+9I7pp0+e8Hon9dTUVE6dOklKSgonT56gZFgYLa9rh4ggIkRF1/XJ+3Suxk2upWjRoj4v96KI6wktd5ZAlp0eD1cBJT0dSCDavTuRiIgy6evh4REkJvr+F0JE6NGlHS0axzL17TcA2Lb1N1YsX8Z1zRrSqU0L1sWv8Xlc59q5YwcbNqwnJraeT8o7k5rKQ/3aMKJ1FDXrNaFSzdrp++Lj5lMjphF58/8zacbab7/mvu7NeGHUQIY98rzX4gorHc4tI0dRt2YlrqlajgIFC9GsRev0/cnJyXw2/SOat7zOazFc7sTNf4HMnSe0DorI385yCFgIPOBuASKSKiIbMiz3O9tzich4EdkiIutEZIWItMtwXpSIqIi0ddavzHCNvSKSmGE9Rw9wOWdhHN8uX8P0L2bz9pRX+X7ZUlJSUjl08G/mf7ucx54az7AB/VC94PC7Xnfs2DH69urOcy/8j4IFC/qkzKDgYJ76aD4T5q7m980b2LX1l/R9KxbMoEGbLmcdX7d5O579PI5Rz7/J5695L7keOnSQ+XNns2rjr2z4ZQcnjh/ns+kfpe+//+47qN+wMfUbNvZaDJeztKm1c3TNVVyfna4BijtLEVWtoKqfXEQZJ1U1KsOSdtflCSAMqKmq0UBXzp6bqy+wzPkfVf0r7RrAa8BLGa6ZdBHxuK106XASEv5pXk5MTCA8PNwbRWUqrLSrzOIlStC+U1fWxa+hdHg4HTp3Q0SIrhtLUFAQfx044PPYwFUT69urO7373kDXbr4fQz1fgUJUq9uQTSviADh66G9+37yBaxq3OO/xkdH12Zf4B0cP/e2VeJbGLaZsufIUK1acXLly0b5TV9auXgHAC+Of5K8D+3ns6ee8UnZOERwkbi2BLNPkqq6q0FxVTXUWj1SNROQK4Cbg9rTBYFT1z7Sk7ST1nsAgoLWI5PFEuRerbkwMW7duYcf27SQlJfHp9Gl06NjZpzEcP36co0ePpr+OW7yQatVr0K5jZ5Z9FwfA1i2/kZSUdNYNL19RVW65aShVI6tx5+i7fFbukYN/cfzoYQCSTp3kx1XfUbp8JQBWfzOHqMatyB36z4/Nn7u2p9fsd/zyAylJp8lfqIhXYguPKEP82lWcOHECVWXZkm+pXCWSD997m7jFC3n1rfcJCsoJzyB5R06pubrTW2CDiNRW1fXZLCOviGzIsP4M8DPwh6oeucA5DYHtqrpNROKADsDn7hYoIsOB4QBlypbNVtDguhP+0oRJdOrQhtTUVAYOGkL1GjWyfb3s2L/vTwb27QFASkoq3Xv1oWXrNiQlJXHHrcNoHBNFrty5mPT6234ZSej75cv56MP3qVmzFvXqRAHw2JNP07Zde6+We+jAPqY8OpozZ1I5c+YM9Vp3onaTVgCsXDCTToNuO+v4NYu+ZtnczwkOCSF3aB5GPPOK196v6LqxdOx8Pdc1rUdISAg1a0Vx46BhVCxdhIgyZenU+loA2nfqyl1jHvJKDBcy4Ma+LF0Sx4EDB6hYPoKxjzzGoCFDfRpDli6DabPdIReqjIpIiKqmiMhmoCqwDTiO6w+LOh/lsy5A5Jiq5j9n29XAVFWtfYFzJgEbVfUNEekMDFDVHhn2jwOOqWqWDWd16tTV5avWuhOqTxw/neLvEM6SLzSwhvS1ObSyVjhfYN1iyJtL4lW1rqeuVyaylt79xky3jh19bQWPlu1Jmf1mrQaiAW98Dt4KlBWRgufWXp3xC7oDXUTkIVzJ/EoRKaCqR70QizEmgKQ1C1zuMkuuAqCq2zxdqKqeEJG3gAkicrOqJolIcaAZcBjYpKpt0gMRmYrr4YX3PB2LMSbw5IRmgcySa3ERueAdClV90c0yzm1znaeq9wMPA08CP4nIKVxNDo/g6h3w5TnX+BzXYDGWXI3J4QQh2IPZ1fk0vBZIVNWOInIVMA24EogH+jsVvFBcOaYOrvGre6vqjuyWm1lyDQbyw6X11FXV4AtsTwLuc5aM5p/n2JnAzAzr4y4lJmNMAPN8T4A7cd1ET+uA/X+4unJOE5HXgKHAq87/B1W1koj0cY7rnd1CM0uue1T18exe2BhjsstTQw6KSASu3kZPAXc53TxbAP2cQ6YC43Al1y7Oa4DPgEkiItntgppZZ7sc0OphjLncCGmTFGa9AMVEZG2G5dzR+v6H69PxGWf9SuCQqqZ120kA0p4MCscZlMrZf9g5Plsyq7m2zO5FjTHmUlxEzfXAhbpiiUhHYJ+qxotIM0/F5q4LJldV9c6zgcYYkwkBgj3zubkR0FlE2uMaLrUgMAEonNaPH4gA0kZjSgTKAAkiEgIUwpmYNTvsGTxjTGAR0odmzGrJjKo+oKoRqloe12wqi1X1BuBbIO2hpIFA2jiiM511nP2LL+WRf0uuxpiAI24u2TQG182trbjaVN9ytr+F64GlrcBdwCVNOxJYzz4aY/7zXE9oefZ+uqrGAXHO69+Bf83xo6qncA0Y5RGWXI0xAScndFWy5GqMCTg5/fFXY4zxOU8//uovllyNMQHHH2MTe5olV2NMwLn8U6slV587fjrV3yGcJdAGy24fWcrfIfxLeONR/g7hLAfXTPJ3CN4lVnM1xhiPE3JGB3xLrsaYgOPpfq7+YMnVGBNwckButeRqjAksrmaByz+7WnI1xgQcq7kaY4zHCWI1V2OM8TyruRpjjIeJYI+/GmOMN+SA3GrJ1RgTeHJCm2tOeBDCqxbMn8fVNapSI7ISzz073idl3nP7cKKrlqF1o+j0bXNmfE6rhrUpXywvm9bHp29f+u03dGjRgOsa16FDiwYs/+5bn8SY5tSpUzRuEEts9DVEX1ODJx571Kflp0lNTaVZw7r07dEFgO/ivqV5oxgaxURx2/DBpKSkZHGFi/faozewc9EzrP30wfRtRQpewexXR/LDjEeY/epIChfIe9Y5daqX5eiaCXRrFQXA1VXCiZt6N/GfPcTq6Q/Q47povOnmYUMoW7oEdaJqerWcS+EaLNu9JZBZcs1Eamoqo+4YwYxZX7N+0098Ou1jfv7pJ6+X27Nvf6Z+MvOsbVUia/D61OnUa9j4rO1FrizG2x9+zoJl8bw4+U1G3zrU6/FlFBoayryFi1m9biOr1m5gwfx5rFq50qcxALz+ykSqVK0GwJkzZxhx8xDeePdDlq/ZQJky5Zj24XseL/P9WSvpMmLyWdvuGdyauNW/UqvL48St/pV7Bl+Xvi8oSHjyzi58s/KX9G0nTiUzdOx71OnxFF1GvsKz93SnUP6zE7In9R84iBmz53nt+p4ibv4LZJZcM7Fm9WoqVqzEVRUqkDt3bnr27sPsWTOyPvES1WvYhMJFipy1rXLVSCpWrvKvY2teHUXJsNIAVImszqlTJzl9+rTXY0wjIuTPnx+A5ORkUpKTfT7oRmJiAgvmfc2NA4cA8Pdff5E7d24qOe9XsxatmDXjS4+Xu3zdNv4+fOKsbR2bXc0Hs1YB8MGsVXRqfnX6vtv6NOWrRRvZ//fR9G1b/9jHtj/2A7Bn/2H2HzxKsaL5PR5rmsZNrqVo0aJeu76nBIm4tQQyS66Z2L07kYiIMunr4eERJCYmZnKGf82d9SU1r44iNDTUp+WmpqZSr04UZUuXoEWr1sTWq+fT8h+6727GPfkMQUGuH+crixUjJSWF9evWAjDzq89JTNjlk1hKXFmAvQeOALD3wBFKXFkAgNLFC9G5xTVM+XTpBc+tW6McuUNC+H3XAZ/EGqisWcANIlJSRD4Skd9FJF5EVohIN2dfrIh8JyK/ish6EXlTRK7IcO5XIrIyw/pDIrLBWVIzvL7Dm1/D5eK3X35i/GMP8cyLvh+OLjg4mFXxG9i6I4G1a1az+ccffVb2/K/nUKx4caJq10nfJiK8+e4HPDzmHlo1bUD+/AUIDg72WUwZpU3M/Ny93Xl4wgwuNFNzqWIFeevJAdw87oMLHvPf4W6jQGBnV6/1FhDXZ8OvgKmq2s/ZVg7oLCIlgU+BPqq6wtnXAygAnBCRwkAd4JiIVFDV31X1KeAp59hjqhrlrdjTlC4dTkKGGk9iYgLh4eHeLvai7UlMYPiAXrz4yluUu6qi3+IoXLgwTZs1Z8GCedSo6ZsbJqtWfs+8ubP5ZsE8Tp86xdGjR7h56ABef+s95iyMA+DbRQvZtnWLT+LZ99dRShUryN4DRyhVrGB6E0B09bK8N34wAFcWzk+bxjVISTnDrLhNFMiXhy8m3sq4ybNY/cMOn8QZ0CRndMXyZs21BZCkqq+lbVDVnar6MjACV9JdkWHfZ6r6p7N6PTALmAb08WKMmaobE8PWrVvYsX07SUlJfDp9Gh06dvZXOOd1+PAhBvftxpixTxJTr6HPy9+/fz+HDh0C4OTJkyz6ZiFVq0b6rPxHHnuKH3/bwYaftvLGux/SpGlzXn/rPfbv2wfA6dOnmfDicwwaOtwn8cxZ8gM3dnI1i9zYqR6z4zYBUK3jOCI7PEpkh0f58pv1jHpmOrPiNpErJJjpL9zER7NX8eU3G3wS4+VA3FwCmTeTaw1g3QX21QTiL7APoC/wsbP0vdiCRWS4iKwVkbX7D+y/2NPThYSE8NKESXTq0IaoWtXo3rMX1WvUyPb13HX7Tf3p1rYZv2/9jXo1KzLtg3eYN3sG9WpWZN2aVQzu243+PToCMPWNV9mxfRsTn3+adk1jadc0lgP793k9xjR79+yhbavmxNS+msYNYmjZqjXtO3T0WfkXMmnCC9SPrkWT+tG0bd+Ba5s193gZU58ZRNzUu6lSriRb5z3BwK4NeP6dhbSoF8kPMx6heb2qPP/Owkyv0f26aBpHV+LGzvVZOe1+Vk67n6ureO/T0YAb+9KsSQN++/VXKpaP4N233/JaWdnlanO9/G9oibfad5y20KtUdbSzPhloDCQBu3DVXP91691pMljlnKsisg4YoKo/ZjjmmKq6dUu1Tp26unzV2kv/gjxk3xHf3cl3R4mCvr35lZUTpz3fH/VS2TQvmcubS+JVta6nrletVm195yv3+ms3qFTEo2V7kjdrrpuB9B7RqjoCaAkUd/bVucB5vYAiwHYR2QGUJxu1V2PM5Ssn3NDyZnJdDOQRkVszbEvrDTAJGCgi6X12ROR6p9baF2irquVVtTyuJOy3dldjjO+JuLcEMq8lV3W1N3QFmorIdhFZDUwFxjg3rvoAzztdsX4G2gBXAuWAlRmusx04nDERG2NytpxwQ8urA7eo6h4uUOt0ego0Oc+uf7Xmq2r0Oevee4TFGON/gZ453WCjYhljAoqIzf5qjDFecfmnVkuuxphAlAOyqw3cYowJMJ4bW0BEyojItyLyk4hsFpE7ne1FRWShiGxx/i/ibBcRmSgiW0Vkk4hke4BdS67GmIDjwa5YKcDdqlodqA+MEJHqwP3AIlWtDCxy1gHaAZWdZTjwana/BkuuxpiA4m43LHdyq6ruUdV1zuujwM+4eiR1wdU1FOf/rs7rLsB76rISKCwiYdn5OqzN1RgTcLwx4LqIlAdq43q8vqTTVRRgL1DSeR2O6/H8NAnOtj1cJEuuxpiAcxG5tZiIZBw8ZIqqTvn39SQ/8DkwSlWPZEzezhgmHh9kxZKrMSbgXES99UBWA7eISC5cifVDVf3C2fyniISp6h7nY3/aUHKJQJkMp0c42y6atbkaYwKLBxtdnUH73wJ+VtUXM+yaCQx0Xg8EZmTYPsDpNVAfOJyh+eCiWM3VGBNwPDjiVSOgP/CDiKSNRv4gMB74RESGAjtxjcYHMBdoD2wFTgCDs1uwJVdjTEARPDfilaou48J13JbnOV5xzZRyySy5+lihvPaWZyZvbv9MJJiZQBucesOOQ/4OwetywNACllyNMYEn0AfCdoclV2NMwLGaqzHGeEEOyK2WXI0xASgHZFdLrsaYgGKDZRtjjJdc/qnVkqsxJhDlgOxqydUYE2DcGwg70FlyNcYEnBzQ5GrJ1RgTWNwdCDvQ2ahYWbh52BDKli5Bnaiafin/1KlTtGhSn0b1oqlf52qefmLcWfvvu3sU4cUL+SW2NAvmz+PqGlWpEVmJ554d79dYAF6e8BJ1rqlJ3ahaDLyxH6dOnfJbLKdOnaJxg1hio68h+poaPPHYoz4p9889CYy4sRN929anX7sGTH/3NQDenDieTo2rM6BTEwZ0asL3cQvSz9n6y4/c1PM6+rVrwA0dGnL6tP/eNxFxawlkVnPNQv+Bg7jltpEMGzLAL+WHhoYy8+tvyJ8/P8nJybRteS2t27QlJrY+6+PXcujQQb/ElSY1NZVRd4xgztcLCY+IoHH9GDp27Ey16tX9Ek9iYiKvTH6ZdRs3kzdvXm7s25tPP5lG/wGD/BJPaGgo8xYuTv/+tWjamOvatKNe/fpeLTc4OIQ7HniSqjWu4fixowzu1pzYRs0A6DPoVm4YdvtZx6ekpDDunpt59LnXqFytFocP/k1ISC6vxpiZAM+bbrGaaxYaN7mWokWL+q18ESF//vwAJCcnk5ycgiCkpqYy9qExPP6kf2uKa1avpmLFSlxVoQK5c+emZ+8+zJ41I+sTvSglJYWTJ0+SkpLCiZMnCAsr7bdYzv3+pSQn+6TGVaxEKarWuAaAfPkLUL5iFfb/eeFhSVcvW0ylqjWoXK0WAIWKFCU42H+D6HhqDi1/suR6GUhNTaVxvTpULhdG85YtqRtbjymvTaZdh06UCsvW3Gkes3t3IhER/wzcHh4eQWJitgZu94jw8HBGjb6bqhXLUaFsaQoVLESr1tf5LR5wff/q1YmibOkStGjVmth69Xxa/p6EP/jtp03UuKYOAJ998AY3dmzEk/eP5Mhh1whbf2zfhogwanB3BnZpygdTJvg0xrO4OfNroNdufZ5cRURF5IMM6yEisl9EZjvrkSKyQkROi8g955z7tojsE5EffR23PwUHB7NsVTybt+wkfu0ali/7jhlffMbNt470d2gB5+DBg8yeNZOffvudbTsTOX78OB9/+EHWJ3pRcHAwq+I3sHVHAmvXrGbzj7778T1x/BgPjBzAqIeeIV+BglzfbwifLVrPezOXUqxESSY+8zAAqakpbIxfybgXpvD6tK9ZsnAOa75f4rM4/+3yr7v6o+Z6HKgpInmd9dacPUfN38AdwPPnOfddoK1XowtghQsXpsm1zVi6JI7ft22jds2q1IqsyIkTJ6hds6pfYipdOpyEhH8my0xMTCA8PNwvsQB8u+gbypUvT/HixcmVKxddunZj5crv/RZPRoULF6Zps+YsWDDPJ+WlJCfz4MiBtOnck2ZtOgFQtFgJgoODCQoKokuvgfy8KR6AEqVKExXTkMJFryRP3ito0LQ1v27e6JM4zyVAkLi3BDJ/NQvMBTo4r/sCH6ftUNV9qroGSD73JFX9Dlfy/c84sH8/hw65PrqdPHmSuMXfEFU7mt92JPLDL9v44ZdtXHHFFaz/8Ve/xFc3JoatW7ewY/t2kpKS+HT6NDp07OyXWAAiypZlzapVnDhxAlUl7tvFREZW81s8+8/5/i36ZiFVq0Z6vVxV5akHb6dcxSr0HfLPwPoH9u1Nfx23cDYVqrjem3pNWrLt1584dfIEKSkprF+znKsq+ecPNuSMZgF/9RaYBjziNAVcDbwNNPHUxUVkODAcoEzZspd0rQE39mXpkjgOHDhAxfIRjH3kMQYNGeqJMN2yd+8ebr1pCKlnUtEzZ+h6fQ/atu/os/KzEhISwksTJtGpQxtSU1MZOGgI1WvU8Fs8sbH16Hp9dxrG1iEkJIRromozZNhwv8Wzd88ebhoykNTUVM7oGbr36EX7Dt7//m2KX8m8r6ZTsaqr2xXALXePZeHsz/nt5x8QEcLCyzLmiZcAKFioMH2H3MaQ61siAg2atqZR8zZej/NCcsITWuKaMsaHBYocU9X8zlzjk4HKwALgHlXtmOG4ccAxVX3+nPPLA7NV1a2Op3Xq1NXlq9ZmfaCPnE5O9XcIZwnNFVjTqvj659EdgdafMtCmeWlQuUh8VtNbX4xratfR+UtWunVsWKHcHi3bk/zZz3UmrnbVZsCVfozDGBNgAuvPWfb4M7m+DRxS1R9EpJkf4zDGBJDLoT3VHX7r56qqCao68dztIlJKRBKAu4CHRSRBRAo6+z4GVgBVne2+a/w0xviMPf6aDaqa/zzb4oA45/VeIOIC5/b1ZmzGmMAQ2GnTPTa2gDEm4AR4pdQtllyNMQHGBss2xhiPE6zmaowxXmHJ1RhjvMCaBYwxxtNySD9XS67GmIAS+IMJuseSqzEm8OSA7GrJ1RgTcHJCm6tN82KMCTieHCxbRNqKyK8islVE7vdu5P+w5GqMCTwemuVFRIJxDW3aDqgO9BURn0xNbMnVGBNwxM1/bogFtqrq76qahGug/i5eDd6R49tc162LP5A3l+z0wKWKAQc8cB1PsXgyZ/FkzVMxlfPANdKtXxc//4rcUszNw/M4A++nmaKqUzKshwO7MqwnAIcy7kQAAA6NSURBVD6ZfjfHJ1dVLe6J64jI2kAa8dziyZzFk7VAjAlAVXPEJKTWLGCMyckSgTIZ1iM4e7Zpr7HkaozJydYAlUXkKhHJDfTBNcWU1+X4ZgEPmpL1IT5l8WTO4slaIMbkUaqaIiIjgflAMPC2qm72Rdk+n/3VGGP+C6xZwBhjvMCSqzF+JoE+057JFkuu5j/DeVonYIi4+nKqtc3lSJZccwgRiRKRiv6OI1CJSBfgXREJiJ95kf9v79zDtarqPP75AqIoxCWMUksYRHNUINC8X1IC5BJ4KbnZqCjJBFlWT3iJGTOLspgxGrUU62mc8TFLydGKoikBQwckMccL5jg5XZ4ZvKAioHj4zh+/9XK273OQE56934NnfZ5nP++7917nXb+z936/71q/31q/pdHAFyT1brQtmXJoFw/aroCkIZL2bbQdLSHpVOBaoGsDbdhPUs9G1f9GSBoF/B1wne2t7cCekcBVwL/Yfr7BtnQuvmbajiyurSC1Mn4A9Cwcaxd+siQc84FZth9thF2SJgJLgOm1rm57IQnZzcA9tn+djjXsuU/3625gre1707GGCJukccAiSX1tN2WBbVuyuO4ASR8AvgGcb/sRSbvXTqXzDRPZ1GK9AegHPFe0q0Ib9gZmAyuA3sCk9iKwkj4EzANuAjZImiGph+2tDfoROhn4CjAVOFLSlQBJ2Cr9Lko6HPgmMfbztiywbU8W1zcgzej4CHA/sDS5BeZL+ipwuaRejQpGSBoCfAkYBXwCWCzp/VV1ewvi9CJwEXAh8CAwiEjr9o66cpWS6j0WmG37c8DvgEOBsyR1t+2qbFOwJ3B0suf7wMlES/8LAEnwq/w+bgausj0GWAX8MAts25InEewASQOAzwBbgNFEK+gZ4JBU5HNAU1UiK0lJGIYBr9Rmm0iaDcwCptleWYEdfW0/I6mL7dcKx88ATgSesL1A0lDbD5ZtT2uQdDZwOPAwcIvtDbXrWXK9nZJ4drO9SdIetjdLeg9wH3Cj7bnFsmXaU7CrZk8X4of6KOAM2+sk7Wf7D1XY8VYli2sLSNrX9h8L+wOAucBK29emY6OBcbZnVWzbQNtPFvZ3t/1Kev9xohVbqsAmd8SniVRujwI3FAMzSWCHAIOBU4CDbP+pLHvqbBtN5PBcUvOx1p0/G3gf8BQxFfLlku05BrgYmAM8W7tOknazvUXSu4FlwO22Ly7ZloOJkV+Ppf1tPyypl/ZFonX/S6IH8qmyr89bGtt5K2zAeOBx4JK64z2BroX984BFwJ6kH6kKbDuA6IbPB04H3lb7bhTKzATWAcNKsuGD6fqcQLhMrgGOSec6F8rdDPw3cFjF9+9LwPPAr4m584cBe9aVmUH4YntVYM88YCvwPeDrwJQWyuwPPALsXdazlJ7rF5MNwwrHVVduGfAsMKTK+/ZW3LLPtUDqpl1NZCs/SK9fb+dFRyZzJM0kuuCX2d7o9FRWYSIRuBpEtDCWSzoa2KdWwPZ1wGXAC21eefgETwXm2V7q8B12IWV2d/LXSXoXMAyYaPu3bW3HDriLGLlwOrARmAzcLGn/5PfEkUz5y7bXV2DPVcDthHguA2ZJ+kdJp9UK2P49MNj2ujKepfR/jyF+bNYBpyW3Enaz71nSWOCdwIm217S1HR2N7BZI1HyYhGgtJrq0s4DVtucVyvUBbgTm2n64AXZOA8YBHyOE7kJgPXAH8AOX1I1L0eWNwNPEqIA/OvyIpwGnOLlHJHW1/WqKyr9Uhi0t2LYP4ff+37R/J/CI7TmSphCt6CVEFvo7bP9byfYMJPzyL6Vr9BnCP74g3b/rgKXAu4ALgNVQzkwtSb1tPy9pP+BPxPM9BXgVuNP2qkLZ4cB6F9xOmZ0nt1zZ5qdbSPgJV6UWzQpgATBM0iWp3EG2nwPOqkpYJR0raWpqVUPkp3wxbauBg4F/By4nRjDsVYINpwLfItwiG2z/j5uDLi8D/VO5ScCM1MLd0NZ2bMe2UcA/A+Ml1ZYb+XSc0nTgSsLvOxu4Byi1JZ2u1UJCwHqkwyuBc5NP/BLCJz6WyCu6zokSbBkJLJE0wvYfbG+1/RDRM+sKTJDUT9JYSf1sP5CFtQ1ptF+i0Rspsg0c0cK5PQjf4kKi5fNboG+Fto0lvpgzgP0Lx/+JiHj/juh6Q7SC+pVgw2jgXmBk2u8NDCicPz5dnwnAGuDACq/PmFTnB+qO9yEE5GXghMLxUn3jRE/iUSKg1q3u3KXJnjMrvD6ziSFXP6s9J4VzQ4mRLkuIHkll962jbA03oNEbEcm9KL3vkl7rnfzziaUhBldo15FE4Kgl0d89Cd6ctL9bSTb0IYIxNQEfSPgNTyyUGQC8BCwHDqno2gjoDtwGjCreM1JQjXDrPAAMrNCmm4DxdXbUXt+f7N0t7XeqwKa+wD8AnyL8vh+uOz+fGPFRyX3raFuHdQsUBpAPIKK0AE3Q7PuSNFgxQ2ssMNrRpaqKvYHv2F6ZhskUbW4CfkrzShKvtfD3bxqHC2Q8MFfSYOB6YJHtewrFnid8mRe6ogzvDjYQLa5aUKpTOteUrtNLhLvkpLIHxUvqnJ6ZgTS72rbW7En7DwF7EQKMSxrLmp7ZwWn3OcK3egjh552WhskhaQ9iPakJVd23jkaHFdeagBKBoKMkDbdtSZ0KM2WOJwI4x7miqLekwyUdARxBzOLBaZRCgb7El/VcSb3KtMf23USX9kHgF7a/ruZkH2OJkQqHujof9HBJx6SB730It05NVLuk9waOA9YCiwsCV4Y9xwHnSOpP5J8YpOYZYJ1SmT7ANKKbflmJtryduE93SzoTGJ7qe4X4rv8r8FFJk21vBibbXl2WPR2dDiuuBe4nurRnJYHd6ojwTgLOB161va4KQ1Jg7XqiNf1j4GlJEwpiX3s9n2iVHGZ7feGHohRs/5SYZnuOYspvk6RziExTm8sUryIpWPRtYKNjVthcYIqkv0l2vpbKTScWorvFJc4ySvdrAdFzeA8xMeFI4HhJPQut01HEkLD1tp8uyx7bzwIjgH0Jt8hoYnztRmBv27cSjYkJknqQWteZcshDsYgZWcB0Iqq8CtgEnEkEH6pqkZ1IDPGaklwBexC+sn7AUtu3p3KTiNk+Z7jiyG4St68S6Q2nUKErIAnZ54ErbP9MkRymOzFJ4CtEq+w/iAH5nyBaZaXdu3S/FgJTbd9fOL6QGCnxLBFw7Al8PNlTVe/nFML9MIx4jqcQvtXzCH89rmiYXEcmi2tCUjeiGzUC+DPwS9trK6z/YmKs5jWFsaJ9gXOBvyb8eSsI/29lX9QW7BxHBEfeV6Gw9iHGjZ5ue5GkA4DvELPolicf4xxi4kQ34Gtl/yhK+iThgbimcOyLRPf/cWLo2tnE2NJrq/ZrShpD/Ogc7cihMMD2U1Xa0NHJS2snbG8i3APLq6xX2ja/ewDNs6q2KBJ4PCNpATHw+zjCn3Z9I78ktu9KroGNFdb5nKTxwJWS/ouYwvmjJKydbD8kaWryc27LtVAGhfs1kMIsuNSq709MCb4VeMb2aS1+SAXY/nGKf66UdGztmSnYnymZLK4Npi6wdmny+z6goIsje9Jw4FduJ9mlqhTWQp13S2oifmAutf21FKVvSsK2hRizWR/8a2s7avdrETBH0rAUFFpCBPxelXQTEYlvKElgdyMmEhxOGmjRaLs6Cjmg1X5oKbD2WvKxXkBzMuwOyxsE1q4AnkxlqhKP+4ixxpMUeXS3JGGdTASS7qvIjjfE9o+IiRRbs7BWS/a5tiPaQ2BtV6CRgbU6O2r362TgNzTfr4m2H6nankz7IotrO6PRgbVdhUYE1rZjRzciKv9BYhbfr2w/0Sh7Mu2HLK6ZXRZJezbC/5vJtIYsrplMJlMCOaCVyWQyJZDFNZPJZEogi2smk8mUQBbXTCaTKYEsrplMJlMCWVw7OJKaJD0o6WFJtymtkLqTn3WSpLvS+w/p9avn1pftJelvd6KOv1cs+Neq43VlvpvynLa2rv6S8uSNzE6RxTWzyfZQ24cS8/IvLJ5MOQ7+4ufE9p0urJrbAr2Av1hcM5ldhSyumSLLgANSi+1xSd8jFkJ8t6SRklZIWp1auN0h8qxKekzSauD02gdJOkfSN9P7fpLukLQmbccA84CBqdV8dSr3WUkrJT0k6YrCZ10maa2k5cBBO/onJF2QPmeNpB/WtcZHSFqVPm9cKt9Z0tWFuj/2Zi9kJpPFNQOAYomUU2leenoQkYf0EGLV0suBEbaHEXkPLlYk9L6BWGdrOPDO7Xz8N4B7bA8hpor+J5F/9cnUav6sYhnoQcRCfkOB4ZJOSBnBJqVjY4jlb3bE7baPSPU9Ssz/r9E/1TEWuD79D9OBF2zXlte5QNKAVtSTyWyXnHIw001SLZXhMiK7/j7A723XMjsdRSTsvjflCO1KJO5+L/BUbS69pJuJZcDrORn4KGxbsO8FSb3ryoxM22/SfndCbHsAd9SmuUq6sxX/06EpcXWv9DmLC+e+n5ZfeSLlhn1vqndwwR/bM9WdczpkdposrplNtocWDyQBfbl4CPi57cl15V73d28SAV+2/a26Oj65E5/1XSIz1ZqUkvCkwrn6+d5Odc+2XRRhFIsOZjI7RXYLZFrDfcCxiuVVkLSXpAOBx4D+kgamcpO38/e/AGamv+0sqSex9HWPQpnFwHkFX+6+kt4BLAUmSuqmWFRvfCvs7QH8OSWKnlp37sOKFX4HAn9FLMmyGJiZyiPpQEl7taKeTGa75JZrZofYXpdagLdI2j0dvtz2WkkziKWcNxJuhR4tfMRFwLcVq7I2ATNtr5B0bxrq9JPkdz0YWJFazhuAabZXS7oVWAP8H7CyFSZ/nkg+vi69Fm16mljI8G1EHtjNkm4kfLGrFZWvAya27upkMi2Ts2JlMplMCWS3QCaTyZRAFtdMJpMpgSyumUwmUwJZXDOZTKYEsrhmMplMCWRxzWQymRLI4prJZDIl8P8iCxtqW7taQQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class_as_string = ['C15', 'CCAT', 'E21', 'ECAT', 'GCAT', 'M11']\n",
    "\n",
    "plt.figure() \n",
    "plot_confusion_matrix(confusion, classes=class_as_string)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Relatório de classificação"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        C15       0.98      0.76      0.86      1276\n",
      "       CCAT       0.67      0.85      0.75      1083\n",
      "        E21       0.86      0.55      0.67       308\n",
      "       ECAT       0.60      0.73      0.66       514\n",
      "       GCAT       0.88      0.86      0.87      1207\n",
      "        M11       0.80      0.85      0.82       302\n",
      "\n",
      "avg / total       0.82      0.80      0.80      4690\n",
      "\n"
     ]
    }
   ],
   "source": [
    "report = classification_report(y_test, y_pred, labels=class_as_number, target_names=class_as_string)\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}