Final_project_DeepTriage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation Study of DeepTriage: Deep Learning for Bug Report Classification\n",
    "\n",
    "#### Author: Patrick de Guzman\n",
    "\n",
    "####  Emails: pdeguzman@ryerson.ca"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction:\n",
    "\n",
    "#### Problem Description:\n",
    "\n",
    "Software engineering teams for large software applications receive a plethora of bug reports outlining issues with various aspects of their applications. With the increasing complexity of software, bug reports come in larger numbers and in greater variety, presenting a problem for both developers solving bugs and the processes/members responsible for assigning bugs to appropriate developers for effective resolution (the bug triaging problem). \n",
    "\n",
    "Bug triaging presents a challenging NLP classification task given the nature of bug reports. Bug reports are generated by end users and are typically a combination of formal language/sentences as well as code stacks (e.g., java/python functions, other code, bug traces). This generates noise in the data which makes it difficult for models to separate classes in classification. \n",
    "\n",
    "The paper \"DeepTriage: Exploring the Effectiveness of Deep Learning for Bug Triaging\" explores a deep neural network to capture syntactical and sequential word information present in bug reports. More specifically, they present a deep bi-directional recurrent neural network with Word2Vec embeddings and a soft attention mechanism (DBRNN-A). After processing bug report data for several different software data sources, they compare rank-10 classification performance between their DBRNN-A and standard bag-of-word (BOW) models found in related literature. \n",
    "\n",
    "#### Context of the Problem:\n",
    "\n",
    "In the domain of software engineering, the bug triaging classification problem is relevant because of the growing complexity of softwares, teams, and the processes underpinning their effective operations. Bottlenecks in effective bug triaging processes can strain developer resources and time which already face severe constraints, particularly in technology companies where there are many ongoing projects and little time to waste on menial tasks. Automated bug triaging can help tremendously in operational efficiency, thus, trickling its benefits to the team's general operations through time savings. In addition, an automated bug triaging solution that is capable of finding the right developer for a particular problem should, theoretically, reduce total time to resolution for bugs by assigning developers based on past experience on certain types of bugs because developers with repeated experience in special areas of troubleshooting benefit from higher transfer learning with their past experience.  \n",
    "\n",
    "Outside of the software engineering domain, the bug triaging problem can be viewed as an important area for understanding noisy textual data in general, as well as the types of processing and modeling methods that are effective in extracting the relevant contextual/semantic information for related problems. \n",
    "\n",
    "#### Limitation About other Approaches:\n",
    "\n",
    "Prior approaches to bug triaging have limitations in their ability to capture semantic information from the text descriptions of bugs. More specifically, BOW models treat text as an unordered collection of words, eliminating the hidden meaning behind their sequences. \n",
    "\n",
    "#### Solution:\n",
    "\n",
    "Deep RNNs (as used in the DeepTriage paper) can extract such sequential information to improve classification performance. It does this through long short-term memory (LSTM) layers, treating each bug report description as a sequence and learning from word to word in order to capture additional information from the text. The authors also employ a soft attention mechanism which further allows the model to distinguish which parts of the bug description are most important for the task at hand which, theoretically, reduces noise in the data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Background\n",
    "\n",
    "Explain the related work using the following table\n",
    "\n",
    "| Reference |Explanation |  Dataset/Input |Accuracy | Weakness\n",
    "| --- | --- | --- | --- | --- |\n",
    "| Bhattacharya et al., 2010 [2] | Trained Naive Bayes model with BOW and TF-IDF feature vectors | Eclipse & Mozilla bug data: bug titles, descriptions, product, component, developer activity | Rank-5 accuracy: 77% | Does not capture semantic context through BOW\n",
    "| Tamrawi et al., 2011 [3] | Trained 'fuzzy-set' features for word tokens | Eclipse bug data: bug titles, descriptions | Rank-5 accuracy: 68% | Does not capture semantic context through BOW\n",
    "| Mani et al. [1] | Trained Bi-LSTM model with attention mechanism & Word2Vec embeddings, compared against BOW classifiers | Mozilla, Chromium data: bug titles, descriptions for triaged & untriaged bugs | Rank-10 accuracy: 46% | No rank-5 accuracy as baseline, lower rank-10 accuracy, \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Methodology\n",
    "\n",
    "The model proposed by Mani et al. [1] is a bi-directional LSTM with a soft attention layer (termed as 'DBRNN-A'). The figures below are directly from the DeepTriage paper. \n",
    "To implement the DBRNN-A, the approach is shown below where all bug reports in the dataset are first cleaned (stripped of URLs, stack traces, tokenized, and case folded) (steps 1-3). Afterwards, all bugs (untriaged and triaged) are used for training a Deep learning-based bug feature representation via Word2Vec (step 5). Using these Word2Vec embeddings, the DBRNN-A model is trained only on the triaged bugs (since these are the instances with actual labels) (steps 6-10). \n",
    "\n",
    "This project will implement this DBRNN-A model, along with the BOW models (MNB and Logistic Regression/Softmax). The model runs are repeated for 10-fold validation on the dataset, and re-run for varying training thresholds per label. Training thresholds are defined as the minimum number of training instances per unique label in the dataset, and the parameters run are 0, 5, 10, and 20 threshold instances. This training threshold filtration is performed as there are many developers with very few bug report instances, generating noise and imbalance in the dataset.\n",
    "\n",
    "![DT_approach](figures/DT_approach.png \"DeepTriage Approach\")\n",
    "\n",
    "The DBRNN-A model architecture is shown below: \n",
    "\n",
    "![DT_model](plots/model_plots/dbrnna_model_plot.png \"DeepTriage Model\")\n",
    "\n",
    "Based on their implementation, below shows the results of the DBRNN-A model that we'll use to compare against the performance obtained in this replication project. Most notably, a subset of the Chromium bug report dataset will be used for this project, and results will be compared against the highest performing model (with minimum training threshold of 20 instances per label).\n",
    "\n",
    "![DT_results](figures/DT_results.png \"DeepTriage Results\")\n",
    "\n",
    "In extension of the paper, I also explore tuning of the size of the units for LSTM and Dense layers in the DBRNN-A model to assess whether the proposed architecture by Mani et al. [1] is optimal, or might have room for exploration in hyperparameter tuning. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#LIBRARIES\n",
    "import src.processing as p\n",
    "import imp; imp.reload(p)\n",
    "\n",
    "import pandas as pd\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To generate the Word2vec embeddings, we first gather both triaged and untriaged bugs in the dataset (at a sample 'frac' size for each) and concatenate these to generate an 'all_bugs' set. The function read_process_data() currently cleans and tokenizes the full dataset per the processing steps outlined by Mani et al. [1]. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                   X     y\n",
      "0  add uma tracking to ntp  sign in  link see tit...   300\n",
      "1   chaos lab  install initial aps on chromeos3-r...   370\n",
      "2  windows should not open maximized on aura we w...  1317\n",
      "3  replace default thumbnail for themes on most v...   437\n",
      "4  add gallery browser tests to crop an image  ve...   564\n",
      "   code                  label\n",
      "0     0   *idanan@chromium.org\n",
      "1     1      *mdu@chromium.org\n",
      "2     2      *sjl@chromium.org\n",
      "3     3      a...@chromium.org\n",
      "4     4  a.suchit@chromium.org\n"
     ]
    }
   ],
   "source": [
    "#READING & PROCESSING DATA\n",
    "triaged, untriaged, label_map = p.read_process_data(overwrite=False, \n",
    "                                                    triaged_frac=0.1, \n",
    "                                                    untriaged_frac=0.3)\n",
    "untriaged.columns = ['X']\n",
    "print(triaged.head())\n",
    "print(label_map.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#BUILD WORD2VEC EMBEDDINGS ON TRIAGED + UNTRIAGED DATA\n",
    "all_bugs = pd.concat([triaged['X'], untriaged['X']], axis=0).reset_index(drop=True)\n",
    "all_bugs = all_bugs.astype(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using all bugs, we then train our Word2Vec model using a vector size of 200 and window of 5 per word. The embedding indices are also generated, and stored separately for ease of access to all models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "p.build_custom_emb_model(data=all_bugs, \n",
    "                        vector_size=200, \n",
    "                        min_count=5, \n",
    "                        window=5, \n",
    "                        overwrite=False)\n",
    "p.build_emb_indices()\n",
    "p.build_results_dfs(overwrite = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Runs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import src.model_dbrnna as dbrnna\n",
    "import src.model_dbrnna_mod as dbrnna_mod\n",
    "import src.model_bow as bows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All models are run below (beware, these take a long time to run!). The 'bows' script contains functionality to run BOW models for 2 different classifiers: MNB and LOG. \n",
    "In addition, 'dbrnna_mod' utilizes the same architecture as the proposed DBRNN-A model, but with additional functionality to adjust the number of units in both LSTM and intermediate Dense layers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "bows.run() \n",
    "dbrnna.run()\n",
    "dbrnna_mod.run(lstm_units=512, layer_units=300)\n",
    "dbrnna_mod.run(lstm_units=300, layer_units=512)\n",
    "dbrnna_mod.run(lstm_units=300, layer_units=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the first round of results, we compare the BOW models against the proposed DBRNN-A, similar to how Mani et al. [1] has formatted their results. \n",
    "In the below implementation, we only view the accuracy by top-10 since this will be the highest performing of the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Results processing\n",
    "models = ['bow_mnb', 'bow_log', 'dbrnna']\n",
    "results = pd.read_csv(os.path.join(os.getcwd(), 'results/results.csv'))\n",
    "results.query(f\"classifier in {models} and metric == 'top_n' and metric_n == 10\", inplace=True)\n",
    "results[[c for c in results.columns if 'cv_' in c]] = results[[c for c in results.columns if 'cv_' in c]].round(4)*100.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Plotting\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "def bar_plots(models, results, orient_x='horizontal', leg_loc=2):\n",
    "    classes = models\n",
    "    thresholds = [0, 5, 10, 20]\n",
    "    colors = ['cyan', 'orange', 'green', 'brown']\n",
    "    widths = [0.2*n for n in [-2, -1, 0, 1]]\n",
    "    x = np.arange(len(models))\n",
    "    for idx, thresh in enumerate(thresholds):\n",
    "        sub = results.query(f\"threshold == {thresh}\")\n",
    "        plt.bar(x+widths[idx], sub['average'], 0.2, color = colors[idx])\n",
    "    plt.xticks(x, classes, rotation=orient_x)\n",
    "    plt.xlabel('Models')\n",
    "    plt.title(\"Top-10 Accuracy by Model and Threshold\")\n",
    "    plt.legend([\"Min. \"+str(thresh)+\" samples\" for thresh in thresholds], loc=leg_loc)\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are the performance results across 10-fold validation runs per model and for training threshold 20 (i.e., each instance had a minimum of 20 instances in the dataset). \n",
    "We see that the top performing model, DBRNN-A, has very similar performance at 41.8% to the final model shown in the results of the DeepTriage paper for the Chromium dataset (42%). \n",
    "\n",
    "In addition, the DBRNN-A model clearly outperforms the BOW models across all training thresholds. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>classifier</th>\n",
       "      <th>average</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>dbrnna</td>\n",
       "      <td>0.417897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bow_mnb</td>\n",
       "      <td>0.233316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>bow_log</td>\n",
       "      <td>0.223729</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   classifier   average\n",
       "23     dbrnna  0.417897\n",
       "7     bow_mnb  0.233316\n",
       "15    bow_log  0.223729"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.query('threshold == 20').sort_values(by='average', ascending=False)[['classifier', 'average']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEXCAYAAACjyo8UAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAs2klEQVR4nO3de3wV5bn3/88XDAKChyJtlaDAFpVDMNgAumktFi2ILfiz9hFFK9VqrfrTx9P2UKpI7cHDtm63VLFK8VC01lZlK61oFQ+tbAgUQRAEESVoWwSloqCg1/PHTOIiJGSFZAUyfN+v13pl5r7ncM9as67MumfmGkUEZmaWXS22dwPMzKywHOjNzDLOgd7MLOMc6M3MMs6B3sws4xzozcwyzoHeMk3SJEnXbu921EbSIEkVeU47VtJ9hW5Tuq7lko7ahvny3p6GkNRFUkjaZRvm3Wobd/R9Zls40DcCSetyXp9KWp8zPqqR1tFK0kPpFzAkDapWL0nXSVqdvq6TpDqW2TVt722N0casS9/3f+YGF0lFadlOcUOKpD/m7NsbJX2cM3779m6f1cyBvhFERLvKF/Am8M2cst804qpeAE4B/l5D3VnAccAhQB/gm8D361jed4B3gRMl7dp4zaybpJZNub5G9C5wTM74MWnZTiEijsnZ138DXJ+zr59d3+U14/2gWXGgLyBJu0q6WdJb6evmyoBa+fNR0pWS3kmP1Gs9+o+IjyPi5oh4AfikhklOA/4zIioiYiXwn8DorbRNJIF+DLCR5B9Dbv0ISXMl/UvSa5KGpuWfk/TrdHvelfRIWj5a0gvVlhGSDkiHJ0m6TdJUSR8AR0o6VtLf0nWskDS22vxflvRXSe+l9aMl9ZP0j9wAIel4SS/Vtq3A3pKelPS+pGcl7Z/ON17Sf1Zb5xRJF25lWfem71ul7wD3VFvGvuly1khaKunMnLo26XvxrqSFQL8a5v29pFWSXpd0/lbakjvfXpIeS+d7Nx0uzqmfLunHkv6Svg/TJO2dU3+qpDfSX4M/zGeddbTn4vSXztuSvptTXtN+UOs2S+ovqTzdR/4h6aZqqxol6c30O/TDnPlq/e7V0Na+kuak78tvgdYN3f4dTkT41YgvYDlwVDo8DpgBfB7oCPwV+HFaNwjYBNwE7Ap8FfgAOCiPdVQAg6qVrQUG5IyXAe9vZRlfAT4C9gL+G/ifnLr+6fKOJjkY6AQcnNY9Dvw2na8I+GpaPhp4odo6AjggHZ6ULnNguszW6XtQko73Af4BHJdOvz/wPnBSup4OQGlatxA4Jmc9DwMX17Kdk9LlHJG+z/9V2c50O98CWqTjewMfAl+oZVkB9E7buWf6HvwjLYuc6Z4DfpluYymwCvhaWvdz4Hngc0Bn4GWgIq1rAcwGrgJaAd2AZcCQtH4scF8tbesAfAtoC7QHfgc8klM/HXgNOBBok47/PK3rCazLeY9uItk3j6pjP5wEXFutbFA677j0cxuWvqd71bIftK1jm18ETk2H2wGHpcNd0s/jV+n2HEKyP/fI87tX+Z63At4ALkzbewLJgc+1W9v25vba7g3I2ovNA/1rwLCcuiHA8nS48guxW079g8CP8lhHTYH+E9JgnI53T78IqmUZd1YGAuDwdOf+fDo+AfhFDfPsA3xa+aWtVjeaugP9PXVs182V6wWuAB6uZbrLgN+kw59LA8k+tUw7CXggZ7xd+l51TsdfAY5Oh88Dpm6lfQEckL533wfOTgPNAaSBniR4fwK0z5nvZ8CkdHgZMDSn7qycoDMAeLPaOq8Afp0Oj6WWQF9DW0uBd3PGpwNjcsbPAf6UDl9V7T3aDfiYbQ/064Fdcsr+yWcBerP9II9tfg64Bti72jRd0s+jOKdsJjAyz+9e5Xt+BMk/e+VM+9fq29XcX+66Kax9SY4WKr2RllV6NyI+qF4vaT/lnODNc13rgN1zxncH1lVGn1yS2gDfJuljJSJeJDm3cHI6SWeSL0p1nYE1EbGtfdIrqrVjgKRn0p/sa0kCZ2V3Qm1tALgP+Kak3YD/AzwfEW/ns96IWAes4bPP4W6S8x6kf+/NYzvuIemy2aLbJl3umoh4P6fsDZJfRZX1K6rVVdqf5PN/r/IFXAl8oa4GSWoraULa/fIvkgC5pzbvA889t/MhyT+9LdqU7pOr61rnVqyOiE21rAs23/66tvkMkl8hiyTNkvSNauva2jZt7btHznQrq31P3qhhumbNgb6w3iLZkSvtl5ZV2isNVpvVR8SbsfkJ3nwsIPn5WumQtKwm/x/JP4JfSvq7pL+TBKLT0voVwL/VMN8K4HOS9qyh7gOSn+EASPpiDdNU/6czGZhCcnS9B3A7UHmlUG1tIJJzEC8CxwOnUndw7pzTrnYkvwIqP4f7gBGSDgF6AI/UsSxIul72IQlGL1Sre4vkPWqfU7YfsDIdfju3PWldpRXA6xGxZ86rfUQMy6NNFwMHkXTf7U5ypAqfvZ9bs1mbJLUl6QoqlNz9YKvbHBFLIuIkki6Y64CHqn1nalPXd6/S20AnabMr1ParYbpmzYG+sO4HxkjqmJ74uooksOS6Rsmlk18BvkHSt1qj9ART5YmiVpJa5+yg9wAXSeokaV+SL/6kWhZ1GjCRpH+8NH0NBA6RVALcBXxX0mBJLdJlHpweNf+R5B/EXkouLawMKC8BvSSVpm0cm8f7057k6HeDpP589osCkl8bR0n6P5J2kdRBUmlO/T3Af6Tb8Ic61jNMyYndVsCPgRkRsQIgIiqAWST/LH4fEevranR69PdNYHj1X0zpcv8K/Cz9fPqQHJVWfu4PAlek718x8P/nzD4TeF/SZUpO2raU1FvSZidsa9GepMvkPUmfA67OY55KDwHfyHmPxtF0sWGr2yzpFEkdI+JT4L10nk/zWG4+3z1IDhg2Aeen+/PxJOduMsWBvrCuBcqBecB8YE5aVunvJJfmvUUS2M6OiEVbWd5iki9zJ+CJdLjyqGUC8D/pel4mOWk6ofoCJHUCBgM3R8Tfc16zgT8Bp0XETOC7wC9ITpw9m7OeU0n68xeR9L3+X4CIeJUkQDwFLGHLI92anAOMk/Q+yRfxwcqKiHiT5ETexSRdLXPZ/BfLw2mbHo6ID+tYz2SSwLcG+BKfddVUupvkH0Y+3TaV7VsQEbX9YjqJpA/5rbSdV0fEU2ndNSRdA68D03LXGRGfkPyzL03r3yE5H7BHHk26meSk5DskJyH/VJ9tAc4leZ/eJtknC37TU7ruurZ5KLAg7cL8L5I++Dr/GVP3d69y/R+T/DIcTbJ/nEjdBw7NjmrowrUmoOSGp/sioriOSa0Wkl4Dvp8TRLd1OUeQHO3tX9M5DbPmzkf01ixJ+hZJX+/TDVxOEXABcKeDvGVVvfNEmG1vkqaTXPt9atp3u63L6UHy8/4lkq4qs0xy142ZWca568bMLON2uK6bvffeO7p06bK9m2Fm1qzMnj37nYjoWFPdDhfou3TpQnl5+fZuhplZsyKp1jt63XVjZpZxDvRmZhnnQG9mlnE7XB99TTZu3EhFRQUbNmzY3k2xRtK6dWuKi4spKira3k0xy7xmEegrKipo3749Xbp0QVt/DKo1AxHB6tWrqaiooGvXrtu7OWaZ1yy6bjZs2ECHDh0c5DNCEh06dPAvNLMm0iwCPeAgnzH+PM2aTrMJ9GZmtm2aZaBXI7/yWqfEKad8lsZ806ZNdOzYkW98I3my2ZQpU/j5z3++zdv0+uuvM2DAAA444ABOPPFEPv74421eVmNYvnw5vXv33q5tMLPG0SxOxu4IdtttN15++WXWr19PmzZtePLJJ+nUqVNV/fDhwxk+fPg2L/+yyy7jwgsvZOTIkZx99tncdddd/OAHP2iMpptZI5vcq1dBlnvygtqeZdMwzfKIfnsZNmwYjz/+OAD3338/J510UlXdpEmTOO+88wAYPXo0559/Pv/+7/9Ot27deOihh7a63Ijg6aef5oQTTgDgtNNO45FHHtliumeffZbS0lJKS0vp27cv77//PuvWrWPw4MEceuihlJSU8OijjwLJEfnBBx/M6NGjOfDAAxk1ahRPPfUUAwcOpHv37sycOROAsWPHcuqpp3L44YfTvXt3fvWrX22x3k8++YRLL72Ufv360adPHyZMSB5c9fbbb3PEEUdQWlpK7969ef755+v5jppZU3Cgr4eRI0fywAMPsGHDBubNm8eAAQNqnfbtt9/mhRde4LHHHuPyyy/f6nJXr17NnnvuyS67JD+wiouLWbly5RbT3XjjjYwfP565c+fy/PPP06ZNG1q3bs3DDz/MnDlzeOaZZ7j44oupTD29dOlSLr74YhYtWsSiRYuYPHkyL7zwAjfeeCM//elPq5Y7b948nn76aV588UXGjRvHW29t/gzlu+66iz322INZs2Yxa9YsfvWrX/H6668zefJkhgwZwty5c3nppZcoLS3N9600sybkrpt66NOnD8uXL+f+++9n2LBhW532uOOOo0WLFvTs2ZN//OMfjbL+gQMHctFFFzFq1CiOP/54iouL2bhxI1deeSXPPfccLVq0YOXKlVXr69q1KyUlJQD06tWLwYMHI4mSkhKWL19etdwRI0bQpk0b2rRpw5FHHsnMmTM3C9rTpk1j3rx5Vb9M1q5dy5IlS+jXrx+nn346Gzdu5LjjjnOgN9tBOdDX0/Dhw7nkkkuYPn06q1evrnW6XXfdtWq4roe7dOjQgffee49Nmzaxyy67UFFRsVn/f6XLL7+cY489lqlTpzJw4ECeeOIJZsyYwapVq5g9ezZFRUV06dKl6vr03Da0aNGiarxFixZs2rSpqq76pY7VxyOC//7v/2bIkCFbtOm5557j8ccfZ/To0Vx00UV85zvf2eq2mlnTy6vrRtJQSYslLZVUaz+EpG9JCkllOWVXpPMtlrRlpGhmTj/9dK6++uqqI+XGIIkjjzyy6oj57rvvZsSIEVtM99prr1FSUsJll11Gv379WLRoEWvXruXzn/88RUVFPPPMM7zxRq2ZSmv16KOPsmHDBlavXs306dPp16/fZvVDhgzhtttuY+PGjQC8+uqrfPDBB7zxxht84Qtf4Mwzz+R73/sec+bM2YatN7NCq/OIXlJLYDxwNFABzJI0JSIWVpuuPclDlv83p6wnMBLoBewLPCXpwIj4pCGN3p4PPywuLub888/f5vlLS0uZO3fuFuXXXXcdI0eOZMyYMfTt25czzjhji2luvvlmnnnmGVq0aEGvXr045phjeP/99/nmN79JSUkJZWVlHHzwwfVuU58+fTjyyCN55513+NGPfsS+++67WdfO9773PZYvX86hhx5KRNCxY0ceeeQRpk+fzg033EBRURHt2rXjnnvuqfe6zazw6nxmrKTDgbERMSQdvwIgIn5WbbqbgSeBS4FLIqK8+rSSnkiX9WJt6ysrK4vqDx555ZVX6NGjR/22zPIyduxY2rVrxyWXXNLk6/bnas3Vjnh5paTZEVFWU10+XTedgBU54xVpWe4KDgU6R8Tj9Z3XzMwKq8EnYyW1AG4CRjdgGWcBZwHst99+DW2S1cPYsWO3dxPMrMDyOaJfCXTOGS9Oyyq1B3oD0yUtBw4DpqQnZOuaF4CIuCMiyiKirGPHGp9ta2Zm2yifQD8L6C6pq6RWJCdXp1RWRsTaiNg7IrpERBdgBjA8IsrT6UZK2lVSV6A7MLPRt8LMzGpVZ9dNRGySdB7wBNASmBgRCySNA8ojYspW5l0g6UFgIbAJOLehV9yYmVn95NVHHxFTganVyq6qZdpB1cZ/AvxkG9tnZmYN1DzvjJ3cyA+tOLnuK/MlMWrUKO677z4gSVO8zz77MGDAAB577DGmTJnCwoUL68xrU5vRo0fz7LPPssceewBJkrTtnVKgXbt2rFu3bru2wcwarnkG+u2g0GmKAW644YaqDJZmZo3F2SvroVBpivO1YMEC+vfvT2lpKX369GHJkiVAkkDtS1/6Er169eKOO+6omr5du3Zceuml9OrVi6OOOoqZM2cyaNAgunXrxpQpU6raPWLECAYNGkT37t255ppralz3DTfcUJWm+Oqrrwbggw8+4Nhjj+WQQw6hd+/e/Pa3v22U7TSzxuVAXw+FSlNc6Yc//CF9+vThwgsv5KOPPtqi/vbbb+eCCy5g7ty5lJeXU1xcDMDEiROZPXs25eXl3HLLLVXJ1j744AO+9rWvsWDBAtq3b8+YMWN48sknefjhh7nqqs9OscycOZPf//73zJs3j9/97ndUvzN52rRpLFmyhJkzZzJ37lxmz57Nc889x5/+9Cf23XdfXnrpJV5++WWGDh2a13aaWdNyoK+HQqYp/tnPfsaiRYuYNWsWa9as4brrrttimsMPP5yf/vSnXHfddbzxxhu0adMGgFtuuYVDDjmEww47jBUrVlQd6bdq1aoq+JaUlPDVr36VoqKiLdIUH3300XTo0IE2bdpw/PHH88ILL2y23mnTpjFt2jT69u3LoYceyqJFi1iyZAklJSU8+eSTXHbZZTz//PNV5xfMbMfiPvp6KkSaYoB99tmnar7vfve73HjjjVtMc/LJJzNgwAAef/xxhg0bxoQJE2jRogVPPfUUL774Im3btmXQoEFVaYqLioqqUg43NE3xFVdcwfe///0t2jRnzhymTp3KmDFjGDx48Ga/FMxsx+Aj+noqRJpiSLp6IAmqjzzySI0P5l62bBndunXj/PPPZ8SIEcybN4+1a9ey11570bZtWxYtWsSMGTPqve4nn3ySNWvWsH79eh555BEGDhy4Wf2QIUOYOHFi1RU4K1eu5J///CdvvfUWbdu25ZRTTuHSSy91mmKzHVTzPKLP43LIQilUmuJRo0axatUqIoLS0lJuv/32LaZ58MEHuffeeykqKuKLX/wiV155Jbvtthu33347PXr04KCDDuKwww6rd5v69+/Pt771LSoqKjjllFMoK9s8Ad7Xv/51XnnlFQ4//HAgOcl73333sXTpUi699FJatGhBUVERt912W73XbWaFV2ea4qbmNMVNa9KkSZSXl3Prrbc2+br9uVpzlcU0xWZm1ow1z64bazSjR49m9OjR27sZZlZAPqI3M8s4B3ozs4xzoDczyzgHejOzjGuWJ2N1TeOmKY6rt3+a4ltvvZWbb76Z1157jVWrVrH33nsnbYvgggsuYOrUqbRt25ZJkyZx6KGHbtM6GsugQYO48cYbt7je3sx2THkd0UsaKmmxpKWStohkks6WNF/SXEkvSOqZlneRtD4tnytpy7uAmoncNMVAjWmKtzXIAwwcOJCnnnqK/ffff7PyP/7xjyxZsoQlS5Zwxx138IMf/GCb12FmO6c6A72klsB44BigJ3BSZSDPMTkiSiKiFLgeuCmn7rWIKE1fZzdSu7eLQqYp7tu3L126dNmi/NFHH+U73/kOkjjssMN47733qtIlVKotXfC4cePo168fvXv35qyzzqrKuTNo0CAuvPBCysrK6NGjB7NmzeL444+ne/fujBkzBoDly5dz8MEHM2rUKHr06MEJJ5zAhx9+uEX7pk2bxuGHH86hhx7Kt7/97ao0CZdffjk9e/akT58+XHLJJXVuv5kVTj5H9P2BpRGxLCI+Bh4ARuROEBH/yhndDdixbrdtJIVOU1yTlStX0rlz56rx4uJiVq5cudk0taULPu+885g1a1bVL5HHHnusap5WrVpRXl7O2WefzYgRIxg/fjwvv/wykyZNqkrWtnjxYs455xxeeeUVdt99d375y19utt533nmHa6+9lqeeeoo5c+ZQVlbGTTfdxOrVq3n44YdZsGAB8+bNq/rnYWbbRz6BvhOwIme8Ii3bjKRzJb1GckSfmwymq6S/SXpW0ldqWoGksySVSypftWpVPZrftAqZprghaksX/MwzzzBgwABKSkp4+umnWZBze3Xl07BKSkro1asX++yzD7vuuivdunVjxYrk4+7cuXNVgrNTTjlli/TFM2bMYOHChQwcOJDS0lLuvvtu3njjDfbYYw9at27NGWecwR/+8Afatm1b0O03s61rtKtuImJ8RPwbcBlQeQj3NrBfRPQFLgImS9q9hnnviIiyiCjr2LFjYzWpICrTFOd229SkvmmKa9OpU6eqwAtQUVGx2bkBgAMPPJA5c+ZQUlLCmDFjGDduHBs2bOCcc87hoYceYv78+Zx55plV6Ytz25ebvrhyvDKFcT7pi48++mjmzp3L3LlzWbhwIXfddRe77LILM2fO5IQTTuCxxx7zA0nMtrN8Av1KoHPOeHFaVpsHgOMAIuKjiFidDs8GXgMO3KaW7iAKlaa4NsOHD+eee+4hIpgxYwZ77LFHVe76SjWlC64M6nvvvTfr1q3bpscZvvnmm7z44osATJ48mS9/+cub1R922GH85S9/YenSpUByruDVV19l3bp1rF27lmHDhvGLX/yCl156aVs23cwaST6XV84CukvqShLgRwIn504gqXtELElHjwWWpOUdgTUR8YmkbkB3YFlDG53P5ZCFUqg0xbfccgvXX389f//73+nTpw/Dhg3jzjvvZNiwYUydOpUDDjiAtm3b8utf/3qLeefPn79FuuA999yTM888k969e/PFL36Rfv361butBx10EOPHj+f000+nZ8+eW1zx07FjRyZNmsRJJ51U9ejDa6+9lvbt2zNixAg2bNhARHDTTTfVtHgzayJ5pSmWNAy4GWgJTIyIn0gaB5RHxBRJ/wUcBWwE3gXOi4gFkr4FjEvLPwWujoj/2dq6nKZ4x7B8+XK+8Y1v8PLLLxdsHf5crblqbmmK87phKiKmAlOrlV2VM3xBLfP9Hvh9/k01M7PG5hQIVqMuXboU9GjezJqOA72ZWcY50JuZZZwDvZlZxjnQm5llXLNMU9zYlzblc0lTodMUjxo1ivLycoqKiujfvz8TJkygqKjIaYrNrMF8RJ+nQqcpHjVqFIsWLWL+/PmsX7+eO++8E3CaYjNrOAf6eihkmuJhw4YhCUn079+fiooKwGmKzazhHOjroSnSFG/cuJF77723KhGY0xSbWUM50NdDU6QpPuecczjiiCP4yldqzOhcI6cpNrOtaZYnY7enyjTF06dPrzryrcm2pCm+5pprWLVqFRMmTKgqq0+a4qlTpzJmzBgGDx7Mf/zHf3DOOedQXl5O586dGTt2bMHSFN9///1bbMvMmTP585//zEMPPcStt97K008/ndd7YGaNz0f09VSoNMV33nknTzzxBPfffz8tWnz2sThNsZk1VLM8om9IhreGKlSa4rPPPpv999+fww8/HIDjjz+eq666ymmKzazB8kpT3JScpnjH4DTFZrVrbmmK3XVjZpZxDvRWI6cpNsuOvAK9pKGSFktaKmmLi8IlnS1pvqS5kl6Q1DOn7op0vsWShmxrQ3e0LiZrGH+eZk2nzkAvqSUwHjgG6AmclBvIU5MjoiQiSoHrgZvSeXuSPGO2FzAU+GW6vHpp3bo1q1evdnDIiIhg9erVtG7dens3xWynkM9VN/2BpRGxDEDSA8AIYGHlBBHxr5zpdwMqI/II4IGI+Ah4XdLSdHkv1qeRxcXFVFRUsGrVqvrMZjuw1q1bU1xcvL2bYbZTyCfQdwJW5IxXAFvc+y/pXOAioBXwtZx5Z1Sbt1O1WZF0FnAWwH777bdFA4qKiujatWseTTUzs+oa7WRsRIyPiH8DLgPqldwkIu6IiLKIKOvYsWNjNcnMzMgv0K8EOueMF6dltXkAOG4b5zUzs0aWT6CfBXSX1FVSK5KTq1NyJ5DUPWf0WGBJOjwFGClpV0ldge7AzIY328zM8lVnH31EbJJ0HvAE0BKYGBELJI0DyiNiCnCepKOAjcC7wGnpvAskPUhy4nYTcG5EfFKgbTEzsxrklesmIqYCU6uVXZUzfMFW5v0J8JNtbaCZmTWM74w1M8s4B3ozs4xzoDczy7hmmY/ezCxfukZ1T1RPv6F6Fpgdm4/ozcwyzoHezCzjHOjNzDLOgd7MLOMc6M3MMs6B3sws4xzozcwyzoHezCzjHOjNzDLOgd7MLOMc6M3MMs6B3sws4/IK9JKGSlosaamky2uov0jSQknzJP1Z0v45dZ9Impu+plSf18zMCqvO7JWSWgLjgaOBCmCWpCkRsTBnsr8BZRHxoaQfANcDJ6Z16yOitHGbbWZm+crniL4/sDQilkXEx8ADwIjcCSLimYj4MB2dARQ3bjPNzGxb5RPoOwErcsYr0rLanAH8MWe8taRySTMkHVfTDJLOSqcpX7VqVR5NMjOzfDXqg0cknQKUAV/NKd4/IlZK6gY8LWl+RLyWO19E3AHcAVBWVhaN2SYzs51dPkf0K4HOOePFadlmJB0F/BAYHhEfVZZHxMr07zJgOtC3Ae01M7N6yifQzwK6S+oqqRUwEtjs6hlJfYEJJEH+nznle0naNR3eGxgI5J7ENTOzAquz6yYiNkk6D3gCaAlMjIgFksYB5RExBbgBaAf8ThLAmxExHOgBTJD0Kck/lZ9Xu1rHzMwKLK8++oiYCkytVnZVzvBRtcz3V6CkIQ00M7OG8Z2xZmYZ50BvZpZxjXp5pe0YJvfqVZDlnrxgQUGWa2aF5SN6M7OMc6A3M8s4B3ozs4xzoDczyzifjDXbARTiBLpPnlslH9GbmWWcA72ZWcY50JuZZZz76M1sxzBZ27sFmeUjejOzjHOgNzPLOAd6M7OMc6A3M8u4vAK9pKGSFktaKunyGuovkrRQ0jxJf5a0f07daZKWpK/TGrPxZmZWtzoDvaSWwHjgGKAncJKkntUm+xtQFhF9gIeA69N5PwdcDQwA+gNXS9qr8ZpvZmZ1yeeIvj+wNCKWRcTHwAPAiNwJIuKZiPgwHZ0BFKfDQ4AnI2JNRLwLPAkMbZymm5lZPvIJ9J2AFTnjFWlZbc4A/lifeSWdJalcUvmqVavyaJKZmeWrUU/GSjoFKANuqM98EXFHRJRFRFnHjh0bs0lmZju9fAL9SqBzznhxWrYZSUcBPwSGR8RH9ZnXzMwKJ59APwvoLqmrpFbASGBK7gSS+gITSIL8P3OqngC+Lmmv9CTs19MyMzNrInXmuomITZLOIwnQLYGJEbFA0jigPCKmkHTVtAN+JwngzYgYHhFrJP2Y5J8FwLiIWFOQLTEzsxrlldQsIqYCU6uVXZUzfNRW5p0ITNzWBpqZWcP4zlgzs4xzoDczyzgHejOzjHOgNzPLOAd6M7OMc6A3M8s4B3ozs4xzoDczyzgHejOzjHOgNzPLOAd6M7OMc6A3M8s4B3ozs4xzoDczyzgHejOzjMsr0EsaKmmxpKWSLq+h/ghJcyRtknRCtbpPJM1NX1Oqz2tmZoVV54NHJLUExgNHAxXALElTImJhzmRvAqOBS2pYxPqIKG14U83MbFvk84Sp/sDSiFgGIOkBYARQFegjYnla92kB2mhmZg2QT6DvBKzIGa8ABtRjHa0llQObgJ9HxCP1mDfTdI0Kstzf0LMgyzWz5imvZ8Y20P4RsVJSN+BpSfMj4rXcCSSdBZwFsN9++zVBk8zMdh75nIxdCXTOGS9Oy/ISESvTv8uA6UDfGqa5IyLKIqKsY8eO+S7azMzykM8R/Sygu6SuJAF+JHByPguXtBfwYUR8JGlvYCBw/bY21mx7c3ebNUd1HtFHxCbgPOAJ4BXgwYhYIGmcpOEAkvpJqgC+DUyQtCCdvQdQLukl4BmSPvqFW67FzMwKJa8++oiYCkytVnZVzvAski6d6vP9FShpYBvNzKwBfGesmVnGOdCbmWVcU1xeaWYZUpjT0RAFWq75iN7MLPMc6M3MMs6B3sws4xzozcwyzoHezCzjHOjNzDLOl1fmY3KhLigzMys8H9GbmWWcA72ZWcY50JuZZZwDvZlZxjnQm5llnK+6sWzylVJmVfI6opc0VNJiSUslXV5D/RGS5kjaJOmEanWnSVqSvk5rrIbX2tYCvMzMmrM6A72klsB44BigJ3CSpOoPuHwTGA1Mrjbv54CrgQFAf+Dq9DmyZmbWRPI5ou8PLI2IZRHxMfAAMCJ3gohYHhHzgE+rzTsEeDIi1kTEu8CTwNBGaLeZmeUpn0DfCViRM16RluUjr3klnSWpXFL5qlWr8ly0mZnlY4e46iYi7oiIsogo69ix4/ZujplZpuQT6FcCnXPGi9OyfDRkXjMzawT5BPpZQHdJXSW1AkYCU/Jc/hPA1yXtlZ6E/XpaZmZmTaTOQB8Rm4DzSAL0K8CDEbFA0jhJwwEk9ZNUAXwbmCBpQTrvGuDHJP8sZgHj0jIzM2sied0wFRFTganVyq7KGZ5F0i1T07wTgYkNaKNlWKHuU4gCLdesOdohTsaamVnhONCbmWWcA72ZWcY50JuZZZwDvZlZxjnQm5llnAO9mVnGOdCbmWWcA72ZWcY50JuZZZwDvZlZxjnQm5llnAO9mVnGOdCbmWWcA72ZWcY50JuZZVxegV7SUEmLJS2VdHkN9btK+m1a/7+SuqTlXSStlzQ3fd3eyO03M7M61PmEKUktgfHA0UAFMEvSlIhYmDPZGcC7EXGApJHAdcCJad1rEVHauM02M7N85XNE3x9YGhHLIuJj4AFgRLVpRgB3p8MPAYMlFeopcWZmVg/5BPpOwIqc8Yq0rMZp0oeJrwU6pHVdJf1N0rOSvlLTCiSdJalcUvmqVavqtQFmZrZ1hT4Z+zawX0T0BS4CJkvavfpEEXFHRJRFRFnHjh0L3CQzs51LPoF+JdA5Z7w4LatxGkm7AHsAqyPio4hYDRARs4HXgAMb2mgzM8tfPoF+FtBdUldJrYCRwJRq00wBTkuHTwCejoiQ1DE9mYukbkB3YFnjNN3MzPJR51U3EbFJ0nnAE0BLYGJELJA0DiiPiCnAXcC9kpYCa0j+GQAcAYyTtBH4FDg7ItYUYkPMzKxmdQZ6gIiYCkytVnZVzvAG4Ns1zPd74PcNbKOZmTWA74w1M8s4B3ozs4xzoDczyzgHejOzjHOgNzPLOAd6M7OMc6A3M8s4B3ozs4xzoDczyzgHejOzjHOgNzPLOAd6M7OMc6A3M8s4B3ozs4xzoDczyzgHejOzjMsr0EsaKmmxpKWSLq+hfldJv03r/1dSl5y6K9LyxZKGNGLbzcwsD3UG+vSZr+OBY4CewEmSelab7Azg3Yg4APgFcF06b0+Sxwr2AoYCv6x8hqyZmTWNfI7o+wNLI2JZRHwMPACMqDbNCODudPghYLAkpeUPRMRHEfE6sDRdnpmZNZF8nhnbCViRM14BDKhtmvRh4muBDmn5jGrzdqq+AklnAWelo+skLc6r9U1E27sBib2Bd/KZcBQLC9KAUdpB3ok87CAt3a6fWXP6vGCH+Mya+3ds/9oq8no4eKFFxB3AHdu7HTsySeURUba922H582fWvGT588qn62Yl0DlnvDgtq3EaSbsAewCr85zXzMwKKJ9APwvoLqmrpFYkJ1enVJtmCnBaOnwC8HRERFo+Mr0qpyvQHZjZOE03M7N81Nl1k/a5nwc8AbQEJkbEAknjgPKImALcBdwraSmwhuSfAel0DwILgU3AuRHxSYG2JevctdX8+DNrXjL7eSk58DYzs6zynbFmZhnnQG9mlnEO9GaWOZLGSrpE0nRJmbxksj4c6BuRpC6SXt7e7dga7/i1a4rPT9IkSScUch1WP+kl4ZmW+Q00s52DpB+SXOb9T5I79WenVadKupMk3p0eETMljQX+DegGvJnejb9fOr4fcHNE3JImaPwj8ALw7yT3AY2IiPWSziS5o78VSXqXUyPiwybZ2HryEX3j20XSbyS9IukhSW0lDZb0N0nzJU1M7yvoJ+kPAJJGSFovqZWk1pKW1bbw9Ij8F5LK03X0k/QHSUskXZtO0yWt+5WkBZKmSWqTs5hTJc2V9LIk5x7aXEE/v1w1LTctHyZpkaTZkm6R9FghNzgLJH2J5LLuUmAY0C+num1ElALnABNzynsCR0XESen4wcAQknxcV0sqSsu7A+MjohfwHvCttPwPEdEvIg4BXiFJ7rhDcqBvfAcBv4yIHsC/gIuAScCJEVFCclTxA+BvJDslwFeAl0l2zgHA/9axjo/TW7VvBx4FzgV6A6MldUinqW3nhNp3fGuazw9JrWtablo+ATgmIr4EdGysDcu4rwAPR8SHEfEvNr+p836AiHgO2F3Snmn5lIhYnzPd42kCxndIfhV8IS1/PSLmpsOzgS7pcG9Jz0uaD4wiydK7Q3Kgb3wrIuIv6fB9wGCSHeXVtOxu4IiI2AS8JqkHyRHETcARJDvs83Wso3Inng8siIi3I+IjYBmfpZyobeeE2nd8a5rPD5J/KFssl+Soclma7RXSz8oapPrNQpXjH1Qr/yhn+BM+69qurXwScF76j/oaoHWDW1ogDvSNr/pO9d5Wpn2OJM//RuAp4Mvpq65AUbnjfcrmO+Gn1L1z1tRG3zX3mab4/KzxPQccJ6mNpPbAN3PqTgSQ9GVgbUSsbaR1tgfeTrt4RjXSMgvCgb7x7Sfp8HT4ZKAc6CLpgLTsVODZdPh54P8CL0bEKpLUzgeRdAMUUqF2/Cxoqs9vcS3LXQx0y3lK24nbvik7j4iYA/wWeInk5OmsnOoNkv5G0tXZmP3oPyLppvsLsKgRl9vofNVN41sMnCtpIkmOn/NJcvL/Lr2MaxbJDgfJTvIFkqMRgHnAF6PweSkqd/wi4PQCr6u5aZLPLyI2SPpu9eVGxEeSzgH+JOkDNg9YthUR8RPgJ9WKb6xl2rF1jPfOGe2dU35jzvBtwG3b1tqm5Vw3ZjsYSe0iYp0kkTzGc0lE/GJ7t8uaL3fdmO14zpQ0F1hA8myHCdu3Odbc+Yh+ByVpPDCwWvF/RcSvt0d7rH78+dmOxIHezCzj3HVjZpZxDvRmZhnnQG87DUkh6b6c8V0krapvLhlJyyXt3dBpzJqKA73tTD4gyU9SmeDtaJJshGaZ5kBvO5upwLHp8Enk5JKR9DlJj0iaJ2mGpD5peYc0A+iCNN2tcuY5RdLMNBvoBEktc1cmaTdJj0t6Kc0W6jtdrck50NvO5gFgZJolsg+bZ5q8BvhbRPQBrgTuScuvBl5IM4E+TJKvnDSh2YnAwDQb6CdsmfNkKPBWRByS3m35p4JsldlWOAWC7VQiYl6aR+YkkqP7XF8mTeccEU+nR/K7k2SVPD4tf1zSu+n0g4EvAbOSm1hpQ5LeNtd84D8lXQc8FhFOeGZNzoHedkZTSHKgDCJJRLatBNwdEVfUNkFEvCrpUJKHYVwr6c8RMa4B6zSrN3fd2M5oInBNRMyvVv48adeLpEHAO+lDLJ4jyWSJpGOAvdLp/wycIOnzad3nJO2fu0BJ+wIfRsR9wA3AoYXYILOt8RG97XQiogK4pYaqscBESfOAD0mePwpJ3/39khYAfwXeTJezUNIYYJqkFiR56c8F3shZZglwg6RP0/ofNP4WmW2dUyCYmWWcu27MzDLOgd7MLOMc6M3MMs6B3sws4xzozcwyzoHezCzjHOjNzDLu/wFAtmJed0F0AQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bar_plots(models, results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Comparison: LSTM and Dense Layer Unit sizes (DBRNN-A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we compare the results of the original DBRNN-A against the modified DBRNN-A models with varying layer units sizes. \n",
    "The original DBRNN-A utilizes 1024 LSTM units and 1000 Dense units per layer. \n",
    "\n",
    "3 permutations are run for the modified models: \n",
    "- 512 LSTM units, 300 Dense units\n",
    "- 300 LSTM units, 512 Dense units\n",
    "- 300 LSTM units, 300 Dense units\n",
    "\n",
    "These are selected to explore the possibility that a simpler architecture for the DBRNN-A could improve results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Results processing\n",
    "models_ = ['dbrnna', 'dbrnna_lstm512_dense300', 'dbrnna_lstm300_dense512', 'dbrnna_lstm300_dense300']\n",
    "results_ = pd.read_csv(os.path.join(os.getcwd(), 'results/results.csv'))\n",
    "results_.query(f\"classifier in {models_} and metric == 'top_n' and metric_n == 10\", inplace=True)\n",
    "results_[[c for c in results_.columns if 'cv_' in c]] = results_[[c for c in results_.columns if 'cv_' in c]].round(4)*100.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>classifier</th>\n",
       "      <th>average</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>dbrnna_lstm300_dense300</td>\n",
       "      <td>0.470646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>dbrnna_lstm300_dense512</td>\n",
       "      <td>0.467710</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>dbrnna_lstm512_dense300</td>\n",
       "      <td>0.429550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>dbrnna</td>\n",
       "      <td>0.417897</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 classifier   average\n",
       "47  dbrnna_lstm300_dense300  0.470646\n",
       "39  dbrnna_lstm300_dense512  0.467710\n",
       "31  dbrnna_lstm512_dense300  0.429550\n",
       "23                   dbrnna  0.417897"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_.query('threshold == 20').sort_values(by='average', ascending=False)[['classifier', 'average']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAGUCAYAAAAlJ9nRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA2d0lEQVR4nO3dd7wU5dn/8c8XAUHEhthARcXYEQ2KimJNIqioiIgd9flpNEYTU9TEGDXJE40liYkxlhgssfAYMUQx9horqEGxRESkiA3FAirF6/fHzIHlcMoedg+zM37fr9e+2Ck7c+3F7nXuvWfmHkUEZmaWf22yDsDMzKrDBd3MrCBc0M3MCsIF3cysIFzQzcwKwgXdzKwgXNAt9ySNkPTLrONojKTdJE0rc91zJN3Q2jGl+5osaa+leF3Z76cSknpICkltl+K1TcZY65+ZpeWCXiZJn5Y8vpT0Wcn04VXaR3tJt6ZftJC0W73lknSBpJnp4wJJamabG6TxXl6NGIsuzfu7pUVEUrt03lfiog1Jd5V8tudJmlsy/ees47PGuaCXKSJWrHsAU4D9Sub9rYq7egw4Ani7gWXHAwcAWwO9gP2AE5rZ3lHAh8AhkpavXpjNk7TcstxfFX0IDCiZHpDO+0qIiAEln/W/Ab8p+ax/u6Xby/HnIHdc0CskaXlJv5P0Vvr4XV3hrPvZJ+knkt5PW96NtuYjYm5E/C4iHgMWNLDK0cDFETEtIqYDFwPDm4hNJAX9LGAeyR+A0uX7S3pe0seSXpe0dzp/NUl/Td/Ph5JuT+cPl/RYvW2EpJ7p8xGSLpc0RtJsYHdJ+0h6Lt3HVEnn1Hv9zpIelzQrXT5c0naS3iktBJIGS/pPY+8VWF3SvZI+kfSwpPXT110m6eJ6+xwt6ftNbOv6NG91jgKuq7eNddLtfCBpoqT/V7KsY5qLDyW9BGzXwGv/Luk9SW9IOqWJWEpft6qkO9LXfZg+716y/CFJv5D07zQP90havWT5kZLeTH/d/bScfTYTzw/SXy4zJB1TMr+hz0Gj71nS9pLGpp+RdyRdUm9Xh0uakn6Hflryuka/ew3Euo2kZ9O83AJ0qPT916SI8KOFD2AysFf6/DzgSWANoCvwOPCLdNluwHzgEmB5YFdgNrBJGfuYBuxWb95HQN+S6T7AJ01sYxfgC2BV4A/AP0uWbZ9u7xskf9i7AZumy+4Ebklf1w7YNZ0/HHis3j4C6Jk+H5Fus1+6zQ5pDrZKp3sB7wAHpOuvD3wCHJrupwvQO132EjCgZD+jgB808j5HpNvpn+b593Vxpu/zLaBNOr06MAdYs5FtBbBlGucqaQ7eSedFyXqPAH9K32Nv4D1gj3TZ+cCjwGrAusCLwLR0WRtgHHA20B7YEJgEfCtdfg5wQyOxdQEOAlYAOgP/B9xesvwh4HXga0DHdPr8dNnmwKclObqE5LO5VzOfwxHAL+vN2y197Xnp/9vANKerNvI5WKGZ9/wEcGT6fEVgh/R5j/T/46r0/WxN8nnerMzvXl3O2wNvAt9P4x1C0sD5ZVPvPY+PzAPI44PFC/rrwMCSZd8CJqfP6z74nUqWjwR+VsY+GiroC0iLbjq9cfqBVyPbuLruCw/smH6I10inrwB+28Br1ga+rPty1ls2nOYL+nXNvK/f1e0XOBMY1ch6pwN/S5+vlhaMtRtZdwRwc8n0immu1k2nXwa+kT4/GRjTRHwB9ExzdwLw7bSg9CQt6CRFegHQueR1vwZGpM8nAXuXLDu+pLj0BabU2+eZwF/T5+fQSEFvINbewIcl0w8BZ5VMnwT8K31+dr0cdQLmsvQF/TOgbcm8d1lUiBf7HJTxnh8BzgVWr7dOj/T/o3vJvKeBYWV+9+py3p/kj7pK1n28/vsqwsNdLpVbh+Svf50303l1PoyI2fWXS1pPJQday9zXp8BKJdMrAZ/WVZlSkjoCB5P0gRIRT5D0/R+WrrIuyReivnWBDyJiafuMp9aLo6+kB9Of2h+RFMi6boDGYgC4AdhPUidgKPBoRMwoZ78R8SnwAYv+H64lOS5B+u/1ZbyP60i6Wpbobkm3+0FEfFIy702SXzl1y6fWW1ZnfZL//1l1D+AnwJrNBSRpBUlXpN0mH5MUwlW0eB916bGXOSR/3JaIKf1Mzmxun02YGRHzG9kXLP7+m3vPx5H8qnhF0jOS9q23r6beU1PfPUrWm17ve/JmA+vlngt65d4i+cDWWS+dV2fVtCgttjwipsTiB1rLMYHkZ2edrdN5DTmQpOD/SdLbkt4mKThHp8unAhs18LqpwGqSVmlg2WySn88ASFqrgXXq/3G5ERhN0lpeGfgzUHdmTmMxEMkxgieAwcCRNF+E1y2Ja0WSVn3d/8MNwP6StgY2A25vZluQdJmsTVJ0Hqu37C2SHHUumbceMD19PqM0nnRZnanAGxGxSsmjc0QMLCOmHwCbkHS7rUTS8oRF+WzKYjFJWoGkC6e1lH4OmnzPEfFaRBxK0nVyAXBrve9MY5r77tWZAXSTFjsjbL0G1ss9F/TK3QScJalregDqbJICUupcJack7gLsS9L32aD0QE/dAZv2kjqUfBCvA06T1E3SOiRf8BGNbOpo4BqS/uve6aMfsLWkrYC/AMdI2lNSm3Sbm6at4LtI/hCsquSUvbrC8R9gC0m90xjPKSM/nUlas59L2p5FvxAg+fWwl6ShktpK6iKpd8ny64Afp+/htmb2M1DJAdb2wC+AJyNiKkBETAOeIfmj8PeI+Ky5oNPW3H7AoPq/gNLtPg78Ov3/6UXSyqz7fx8JnJnmrzvw3ZKXPw18Iul0JQdPl5O0paTFDpw2ojNJV8csSasBPy/jNXVuBfYtydF5LLvvf5PvWdIRkrpGxJfArPQ1X5ax3XK+e5A0DOYDp6Sf58Ekx1YKxwW9cr8ExgLjgReAZ9N5dd4mOeXtLZIC9u2IeKWJ7b1K8qXtBtydPq9rhVwB/DPdz4skBy+vqL8BSd2APYHfRcTbJY9xwL+AoyPiaeAY4LckB7AeLtnPkST97a+Q9I1+DyAi/ktSCO4DXmPJlmtDTgLOk/QJyRduZN2CiJhCckDtByRdJM+z+C+QUWlMoyJiTjP7uZGkwH0AfJ1FXSx1riX5w1BOd0tdfBMiorFfQIeS9PG+lcb584i4L112LslP+jeAe0r3GRELSP6o906Xv0/SX79yGSH9juTg4PskBwP/1ZL3AnyHJE8zSD6TrX5xULrv5t7z3sCEtOvx9yR95M3+0aX5717d/ueS/NIbTvL5OITmGwi5pAa6X61KlFwYdENEdG9mVWuEpNeBE0qK5dJupz9J6239ho45mBWBW+hWsyQdRNIX+0CF22kHnApc7WJuRdbiMRLMlgVJD5GcO31k2re6tNvZjORn+X9IupjMCstdLmZmBeEuFzOzgnBBNzMriMz60FdfffXo0aNHVrs3M8ulcePGvR8RXRtalllB79GjB2PHjs1q92ZmuSSp0WEL3OViZlYQLuhmZgXhgm5mVhAu6GZmBeGCbmZWEC7oZmYF4YJuZlYQLuhmZgXh0RbNzFroxi22qOj1h01o7L4plXEL3cysINxCN/sKqtUWplXGLXQzs4JwQTczKwgXdDOzgnBBNzMrCBd0M7OCcEE3MysIF3Qzs4JwQTczKwgXdDOzgnBBNzMrCBd0M7OCcEE3MysIF3Qzs4JwQTczKwgXdDOzgnBBNzMrCN/gwnLHN2cwa5hb6GZmBeGCbmZWEC7oZmYF4YJuZlYQPiiaAR/UM7PW4Ba6mVlBuKCbmRWEC7qZWUGUVdAl7S3pVUkTJZ3RxHoHSQpJfaoXopmZlaPZgi5pOeAyYACwOXCopM0bWK8zcCrwVLWDNDOz5pXTQt8emBgRkyJiLnAzsH8D6/0CuAD4vIrxmZlZmcop6N2AqSXT09J5C0naFlg3Iu6sYmxmZtYCFR8UldQGuAT4QRnrHi9prKSx7733XqW7NjOzEuUU9OnAuiXT3dN5dToDWwIPSZoM7ACMbujAaERcGRF9IqJP165dlz5qMzNbQjkF/RlgY0kbSGoPDANG1y2MiI8iYvWI6BERPYAngUERMbZVIjYzswY1W9AjYj5wMnA38DIwMiImSDpP0qDWDtDMzMpT1lguETEGGFNv3tmNrLtb5WGZmVlL+UpRM7OCcEE3MysIF3Qzs4JwQTczKwgXdDOzgnBBNzMrCBd0M7OCcEE3MysIF3Qzs4JwQTczKwgXdDOzgnBBNzMrCBd0M7OCcEE3MysIF3Qzs4JwQTczKwgXdDOzgnBBNzMrCBd0M7OCcEE3MysIF3Qzs4JwQTczKwgXdDOzgnBBNzMriLZZB2BmLadzVdHr/8bmVYrEaolb6GZmBeGCbmZWEC7oZmYF4YJuZlYQPihqZl85RT2o7IJuy1xRv0xmWXOXi5lZQbigm5kVhAu6mVlBuKCbmRVEWQVd0t6SXpU0UdIZDSz/tqQXJD0v6TFJPmplZraMNVvQJS0HXAYMADYHDm2gYN8YEVtFRG/gN8Al1Q7UzMyaVk4LfXtgYkRMioi5wM3A/qUrRMTHJZOdgKheiGZmVo5yzkPvBkwtmZ4G9K2/kqTvAKcB7YE9qhKdmZmVrWoHRSPisojYCDgdOKuhdSQdL2mspLHvvfdetXZtZmaUV9CnA+uWTHdP5zXmZuCAhhZExJUR0Sci+nTt2rXsIM3MrHnlFPRngI0lbSCpPTAMGF26gqSNSyb3AV6rXohmZlaOZvvQI2K+pJOBu4HlgGsiYoKk84CxETEaOFnSXsA84EPg6NYM2szMllTW4FwRMQYYU2/e2SXPT61yXGZm1kIebdEsCzdWNuKkWUN86b+ZWUG4hb4UPJ63mdUiF3Qzyx93WTXIXS5mZgXhgm5mVhAu6GZmBeGCbmZWEC7oZmYF4YJuZlYQLuhmZgXhgm5mVhAu6GZmBeErRa3lfJWeWU1yC93MrCBc0M3MCsIF3cysIFzQzcwKwgXdzKwgXNDNzArCBd3MrCBc0M3MCsIF3cysIFzQzcwKwgXdzKwgXNDNzArCBd3MrCBc0M3MCsIF3cysIFzQzcwKwgXdzKwgvpp3LPIdd8ysgNxCNzMrCBd0M7OCcEE3MyuIsgq6pL0lvSppoqQzGlh+mqSXJI2XdL+k9asfqpmZNaXZgi5pOeAyYACwOXCopM3rrfYc0CciegG3Ar+pdqBmZta0clro2wMTI2JSRMwFbgb2L10hIh6MiDnp5JNA9+qGaWZmzSmnoHcDppZMT0vnNeY44K5KgjIzs5ar6nnoko4A+gC7NrL8eOB4gPXWW6+auzYz+8orp4U+HVi3ZLp7Om8xkvYCfgoMiogvGtpQRFwZEX0iok/Xrl2XJl4zKwBV+LCGlVPQnwE2lrSBpPbAMGB06QqStgGuICnm71Y/TKsmf5nMiqnZgh4R84GTgbuBl4GRETFB0nmSBqWrXQisCPyfpOcljW5kc2Zm1krK6kOPiDHAmHrzzi55vleV4zIzsxbylaJmZgXhgm5mVhBfzeFzzSpQjQPDUYVtmNXnFrqZWUG4oJuZFYQLuplZQbigm5kVhAu6mVlBuKCbmRWEC7qZWUG4oJuZFYQLuplZQbigm5kVhAu6mVlBuKCbmRWEB+cyK4BV26/KOdueQ8+VetKmjHba6ju3q2h/L7/8ckWvr/Qu8i9/rbIt3LVBZftfFvnr0KED3bt3p1278vflgm5WAOdsew7bb7A9bTu1LWs4yA0+6FDR/rpstllFr59d0aths5mVbWF2g3c9Ll9r5y8imDlzJtOmTWODDcr/6+MuF7MC6LlSz7KLudU+SXTp0oXPP/+8Ra9zQTcrgDa0cTEvGKnl/6Eu6GZWFZI44ogjFk7Pnz+frl27su+++wIwevRozj///KXe/vQ33mB4374c2LMnhxx3JnPnzqs45kpMmT6dnQ88MNMY6nMfulkBbbdOn6ZXWKdl23v/xRebXadTp068+OKLfPbZZ3Ts2JF7772Xbt26LVw+aNAgBg0a1LIdl/jj6adz2Pe/zzeHDePq4Qfxlxv+wYnHDlnq7RWRW+hmVjUDBw7kzjvvBOCmm27i0EMPXbhsxIgRnHzyyQCcM3w4F51yCsfutBP7b7gh9996a5PbjQieeeAB9hiSFPCjh+3D7Xc9vMR6D/97HL13O4zeux3GNrsfziefzObTT+ew54Ensu3uR7DVLsP4x5jkdW9NfYsh/YdwzvfO4aCdD+Ksk8/iqUee4rj9j2Nwv8FMeG4CAFdefCVnf/dsjt3vWAb3G8yov41aYr8LFizg5xdfzF7DhtF/8GBGjBwJwNvvvce+Rx/NbkOGsPOBB/LEuHEtTWmLuIVuZlUzbNgwzjvvPPbdd1/Gjx/Psccey6OPPtrguu/PmMHVjz3G5Fde4QeDBrHnkMZb2x/NnEnnVVahbdukZHVfZw2mz3h3ifUuuuwGLrvgdPr13ZpPP51Dhw7tARh13YWs1HlF3p85ix32Poab9ugPwLTJ0zj/ivPZ8JINOXrg0dx9+91cffvVPHLPI/z1D3/lomsuAmDiyxO55p/X8Plnn3P4Nw/n0K/vudh+b7jtNlZacUXuu/lmvpg7l4FHHsnuO+3EHfffzx79+nHa8cezYMEC5rTwIGdL5bKgV3rsxzfoNWsdvXr1YvLkydx0000MHDiwyXV3PeAA2rRpw4abb84H77xTlf3367s1p/3stxw+ZG8G77s73Vdck3nz5vOTX/6JR554jjZtxPQZ7zHzvZkArLPuOvTcrCcAG35tQ7bbeTsksdGmG/HW1LcWbrf/t/rToWMHOnTsQJ+d+vDcCy+w5aabLlz+0BNPMOG//+Wf994LwMeffsqkKVPYZostOPXss5k3fz4D99iDrUpe0xpyWdDNrHYNGjSIH/7whzz00EPMnDmz0fXaL7/8wucRTTezVu7ShU9mzWL+/Pm0bduWaW+9S7e111hivTNOHc4+39iZMff9m34D/4e7/+8PPDn2Bd57/0PG3X897dq1pcc2g5j7xVwA2i2/6KIdtRHt2yct+jZt2rBgwYJFy+qfcVJvOiI4/8wz2aNfvyViGj1iBPc+8gjfPessTjzqKA6p4DhCc9yHbmZVdeyxx/Lzn/+crbbaqmrblESf3XfngbSv/dqb72T/Af2XWO/1N6ax1eY9Of2Uo9lum8155bXJfPTxp6zRdTXatWvLg4+O5c2pM1q8/4fvfpgvPv+CWR/MYtwT49hmyy0XW777Tjvx11tuYd685MybiZMnM3vOHKa+9RZrdOnCUUOGcMTgwYyv8Arb5riFbmZV1b17d0455ZSlfv1hvXtz4/PPLzH/5Asu4KfDhnH5WWexwxY9OO7w/ZdY53dX3MSDj42lTZs2bLHJhgzYcyc++XQO+x1+GlvtMow+vTdj0417tDimjTfbmBMPPpFZH8ziuO8dx9prrMGU6dMXLj/yoIOY+tZb7DF0KAF0WXVVrv/97/n3M8/wxxEjaNe2LZ1WWIHLfvWrFu+7JdTcT53W0qdPnxg7duxSvbbiPvQbK9uCXqts/38buXlFrz9swoSKXu/8ZZs/qH4O7/rmXay+/uplv77iS9frtVBbaum++Yv0mVnZFsa24NL/Ky++ko6dOnLkt49cOG9Z5e/ll19ms3rDBEgaFxENnpfqLhczs4Jwl4uZWROO/8HxWYdQNrfQzcwKwgXdzKwgXNDNzArCBd3MrCBc0M2sKlp7+Nxzhg9n/w024LDevem922E8/8KrFcdcqfW33z7rEBZT1lkukvYGfg8sB1wdEefXW94f+B3QCxgWEU0PnWZmrarPQ9tVdXsze73Q7DqtPXwuwCkXXsieQ4ZUfB56UTXbQpe0HHAZMADYHDhUUv0rO6YAw4Ebqx2gmeVHaw2fW64Jr7zO9t84mt67HUav/ofy2utTADjgyB/y9T2OZIt+Q7ny2tsWrt9/4/78/he/Z+juQznpkJOY8NwEThhyAvvvuD8P35MMs/vPW/7JD475AScMOYHB/QZz1SVXNbjvP/z1rwuHzz3/sssAmD1nDsNOOoldDzqInQ88kFH/+ldV3mdjyuly2R6YGBGTImIucDOw2DW3ETE5IsYDX7ZCjGaWE8OGDePmm2/m888/Z/z48fTt27fRdeuGz/3tHXfwxzPOKGv7f/rpTzm0Vy++/9NL+CIdYKvUn0fcxqnHD+P5h25k7H3X0X2dZACvay79GeMeuJ6x913HpVfdwqwPZgHw2ZzP6NOvDyMfHEmnFTtx+W8u57KbLuPCqy/kiguvWLjdCc9P4IIrL+DG+27kvjvu47l6Vxs/+PjjTHrzTe696SYeuvVW/vPSSzw+diwP/PvfrLXGGjz897/z2KhR7NnA4F3VVE5B7wZMLZmels4zM1tMaw6fe/Kvf82tr7zCtc88wwezPuaCS69dYp0d+2zF//7ur1xw6bW8OXUGHTsml+hfeuUtbL3rYeyw97FMnf4OU99ISlq79u3YafedANho043YdodtaduuLT0368mMaYsG8eq7S19WWW0VOnTswO4DduepZ59dbL8PPv44Dz3xBLsffDB7DB3KxDfeYNKUKWy28cY8/MQTnHvJJTwxbhwrde7c7PusxDK9UlTS8cDxAOutt96y3LWZLSOtMXwuwOprr73wdcccuh8XXXbDEuscNmRv+n59S+689zEGDvseV1x8Jm3atOG+R57mibuuYYUVOrDboBMWDp/btm3bhUPjtmnTZuFwum3atGHB/MaHz60/HRGcetxxDB86dImYHhg5knsfeYT//cMf6N+3Lz868cRm3+vSKqeFPh1Yt2S6ezqvxSLiyojoExF9unbtujSbMLMa1xrD50LSRQNJ8bz9rofYcrMNl1hn0uRpbNijG6ccP4z9B/Rn/Euv8dHHn7Lqyp1ZYYUOvPLaZJ4c1/z9Uet76tGn+OjDj/j8s895+O6H2X6bbRZbvke/ftx4++18OmcOADPeeYf3Zs5kxrvv0rFDB4butx8nH3NMTQyf+wywsaQNSAr5MOCwVo3KzHKrtYbP/dnhh/Phe+8REey42br8+aIzl1hn5D/u4/qRY2jXri1rrdGFn3z/GDqt0JE/X/t3NtvxYDbpuT47fL3lI0Vu0XsLTv9/p/PujHcZcNAAttlii8WW777TTvx30iQGHH44AJ1WWIHLzz+fN6ZM4ZyLL6ZNmza0bduWC3/2sxbvuyXKGj5X0kCS0xKXA66JiF9JOg8YGxGjJW0HjAJWBT4H3o6ILRrdIB4+txJZD//q/FXOw+dWZlkOn/vPW/7Jy+Nf5se/+vHCebU6fG5ZfegRMQYYU2/e2SXPnyHpijEzs4x4+Fwzsybsd8h+7HfIflmHURZf+m9mVhAu6GZmBeGCbmZWEC7oZmYF4YJuZlXR2sPnjvzjHzmwZ0+2k3h/5qyF8yOCU868iJ7bHUiv/ofy7H9eWep9VMugY45ZYryXZcFnuZgV0HZXVXf43PcPyn743K379WPnfffl27vtttj8u+57nNcmTeG1p2/jqXEvcuKPzuepe0Ys9X7yzC10M6ua1hw+d5NttmGdHj2WmP+Pux7mqKH7IIkd+mzFrI8+Ycbb7y+2zuzZn7HPsO+x9a6HseXOh3DPP+4B4KrfXsVRA4/ikD0O4Vc//tXCMWVOGHICl/z8Eo4acBQH73owE56fwI/+50cM7jeYyy+4HIAp06ezw377ccLpp7PjoEEcc9ppzPnssyXie/Dxx9n78MPZfehQjj3ttIXDA5xxxhlsvvnm9OrVix/+8IfNvv9yuKCbWdW09vC5DZk+4z3W7bbmwunu66zB9BnvLrbOvx54gnXWWp3/PHwjLz52y8IRFocOH8p1Y67jlgdu4YvPvuDRex9d+Jp27dtx3V3XMfjIwfzw2B9y+q9O5+YHbuaOkXfwwaxZAEycPJljhw3jidGj6dypE9fccsti+5354YdcfMUV/P2qq3hw5Eh6b7EFl197LTNnzmTUqFFMmDCB8ePHc9ZZZy31+y/lgm5mVdOaw+dWYqvNNuLeh5/m9HP/wKNPPMeKK60IwLjHxzF83+EM23MYYx8fy6T/Tlr4mv7f7A9Az017suHXNmT1NVen/fLt6bZ+N6a//TYA3dZai77pQF1D9t13iWF1x44fz38nTWKfo45ityFDuHn0aKbNmMHKK69Mhw4dOO6447jttttYYYUVqvI+3YduZlXVWsPnNqbb2l2ZOn3RH4Rpb71Lt7XXWGydr/Vcn2cfuJ4x9/6bs/73cjbttx1HnXgUF/zkAq4dcy1rdVuLKy++cuGwupC00CEZSrd9+/YL56uNmL8gGVq3nGF1d91xR676zW8Wm9+2bVuefvpp7r//fm699Vb++Mc/8sADDyx1Duq4hW5mVdVaw+c2ZtDe/blu5J1EBE+OfYGVV1qRtddafKCyt2a8xwodO3DE0IH86OQjefWFVxcW71VWW4U5s+dw/533t3jf02bM4Jl0ZMi/jxlD3223XWx5n169ePq555g0JbkV3uw5c5g4eTKffvopH330EQMHDuS3v/0t//nPf5binS/JLXQzq6rWGj735ksv5frf/IaZb79Nr/6HMnCvflz9+7MY+I1+jLnv3/Tc7kBW6NiBv1569hKvfeHlifzonEtp00a0a9uW7/7vGXReuTMHHHYAw/YcRpeuXdh865aP4tmzRw/+cvPNnHL22Wyy0UYcU+8GF6uvthp/+OUvOf7HP2bu3OQPyJnf/S7rff3r7L///nz++edEBJdcckmL992QsobPbQ0ePnfpZT38q/NXOQ+fW5llOXxuQzb4oANTpk/nsJNP5rFRo1r8+tYaPtddLmZmBeGCbma2FNbr1m2pWuetyQXdzKwgXNDNzArCBd3MrCBc0M3MCsIF3cyqorWHzz3r8MM5aJNNOGTLLTn2lPOYN28+4OFzS/nCIrMC+u83jm56eQu39616g041pLWHzx1w+OH84oYbALjkoL25+vrbOfHYIR4+t4Rb6GZWNa05fG6/gQORhCS233YLpqUjKnr43EVc0M2sapbF8Lnz583j+pFj2HuPHQEPn1vKBd3MqmZZDJ97/kkn0X+nbdhlx23Kfo2HzzUzWwqtOXzuVeeey6z33uOSq36ycJ6Hz13ELXQzq6rWGj739quv5om77+aXN91EmzaLSpeHz13ELXQzq6rWGj73/G9/m7XWX59jd9yRFRbMYfA+u3P2j/6fh88t4eFzl2b/X/HhX52/ynn43Mp4+FwPn2tmVmgu6GZmS8HD55qZWatxQTcrgC/5ErI5HGatZGmOb7qgmxXAxI8nMn/2fBf1gogIZs6cSYcOLTt47dMWzQrgnGfP4RzOoedKPWlTTjttTruK9vfucstV9Pr3m1+lSS/PrmwL78+rMIBlkL8OHTrQvXv3Fm23rIIuaW/g98BywNURcX695csD1wFfB2YCh0TE5BZFYmZL7cO5H3Lqk6eWvX7Wp35WtneIGyvbwuY5P3W2Mc3+KZe0HHAZMIDk/+FQSfXfzXHAhxHRE/gtcEG1AzUzs6aV04e+PTAxIiZFxFzgZmD/euvsD1ybPr8V2FP1BzUwM7NWVU5B7wZMLZmels5rcJ2ImA98BHSpRoBmZlaeZi/9lzQE2Dsi/iedPhLoGxEnl6zzYrrOtHT69XSd9+tt63jg+HRyE+DVar2RGrM6lR/3+Spz/irnHFamlvO3fkR0bWhBOQdFpwPrlkx3T+c1tM40SW2BlUkOji4mIq4Eriwn4jyTNLaxsRasec5f5ZzDyuQ1f+V0uTwDbCxpA0ntgWHA6HrrjAbqbmI4BHggshr1y8zsK6rZFnpEzJd0MnA3yWmL10TEBEnnAWMjYjTwF+B6SROBD0iKvpmZLUNlnYceEWOAMfXmnV3y/HPg4OqGlmuF71ZqZc5f5ZzDyuQyf5mNh25mZtXlsVzMzArCBd3MrCBc0M3MCsKjLVaJpA4kY9psASwc8zIijs0sqJyQtCnJ8BF1VyBPB0ZHxMvZRZUfklYCukbE6/Xm94qI8RmFlTuS1qTkMxgR72QZz9JwC716rgfWAr4FPExyAdYnmUaUA5JOJxkfSMDT6UPATZLOyDK2PJA0FHgF+LukCZK2K1k8Ipuo8kVSb0lPAg8Bv0kfD0t6UtK2mQbXQj7LpUokPRcR20gaHxG9JLUDHo2IHbKOrZZJ+i+wRUTMqze/PTAhIjbOJrJ8kPQ8MCAiZkjanmQY6zMjYlTdZzLbCGtfmsMTIuKpevN3AK6IiK0zCWwpuMuleuoK0ixJWwJvA2tkGE9efAmsA7xZb/7a6TJr2nIRMQMgIp6WtDtwh6R18f2LytWpfjEHiIgnJXXKIqCl5YJePVdKWhX4GclQCCsCZzf9EgO+B9wv6TUWjeq5HtATOLmxF9lCn0jaqK7/PG2p7wbcTnI8x5p3l6Q7SX7d1H0G1wWOAv6VWVRLwV0uljlJbUjG3S89KPpMRCzILqp8kLQ1MDsiJtab3w4YGhF/yyayfJE0gIYPzI9p/FW1xwW9StLb8B0E9KDkl09EnJdVTHmR3gylfkF/2gO8mbWMu1yq5x8kN/YYB3yRcSy5IembwJ+A11g0LHN3oKekkyLinsyCyzlJL0TEVlnHUeskrQycSdJCX5Pk2MO7JN/p8yNiVnbRtYwLevV0j4i9sw4ih34P7FX/puKSNiAZEG6zLILKC0mDG1tEchqtNW8k8ACwe0S8DSBpLWB4uuyb2YXWMi7o1fO4pK0i4oWsA8mZtiS3NaxvOtBuGceSR7cAf6PhM1o6NDDPltQjIha7sX1a2M+XdExGMS0VF/Tq2RkYLukNki4XARERvbINq+ZdAzwj6WYWP8NgGMk4+9a08cBFEfFi/QWS9sognjx6U9KPgWvrrg5NrxodzuL3U655PihaJZLWb2h+RNQ/v9rqkbQ5MIglzzB4Kbuo8kHSLsCbETGlgWV9ImJsBmHlSnq68Rkkfeh11468Q3L68QUR8UFWsbWUC3oVSVqO5KBK6VkuS3zRzMxag8dyqRJJ3yX5q34vcGf6uCPToHJA0qaS7pJ0p6SNJI2QNEvS05J8QLQZklavN32EpEslHZ+eDmrNUGKopIPT53umOTwpvUYiN9xCr5L0fqp9I2Jm1rHkiaRHgAtJrqw9Hzid5EDfvsD3ImLPDMOreZKejYht0+dnAbsAN5Lkb1pEfD/L+PJA0p9IulraAx8Dy5N0t+wDvBMRp2YYXou4oFeJpAeBb0TE/KxjyZPSAaQkTYyIniXLFhYra1i9/D0L7BIRs9MrRZ/1eejNqztfP83Z28DaETFXUluSHObmxAaf5VI9k4CH0jEhFl5YFBGXZBdSLixX8rx+rtovy0ByqqOkbUi6T5eLiNkAETFPkodOKM98WJizZyJibjo9X1KuBohzQa+eKemjPS5ELXGZpBUj4tOI+FPdTEk9gfsyjCsvZrDoD+EHktZOB+jqQlqorFlvl3wGF14cmF5cNDfDuFrMXS5VkJ7dcl1EHJ51LGaw8DO5fETMyTqWvEqHzu0UEe9mHUu5cnUEt1alowKun96UwZaCpK9Jul/Si+l0r/Qgn5UhPTvjCEl1QzZ3A7bMMqa8aSCHXUgG28sNt9CrRNJ1JOOOjAZm1813H3p5JD0M/IjkDjF1B/lejAgXpTJIupzkhiB7RMRm6cUy90TEds281FJFyKH70Kvn9fTRBuiccSx5tEJ6x53See4DLl/fiNhW0nMAEfGhfzG2WO5z6IJeJRFxLiy8A3tEhG8Q3TLvS9qIdJApSUNIDvhZeeal/eZ1+euKb+HXUrnPoQt6lUjqA/yVtHUu6SPg2IgYl2lg+fEd4EpgU0nTgTeAI7INKVcuBUYBa0j6FTAE8DGIlsl9Dt2HXiWSxgPfiYhH0+mdgT/l6aKEWpCeWdDGv3BaTtKmwJ4kI33eHxEvZxxS7uQ9hz7LpXoW1BVzgIh4DPcBl03SqWl31Rzgt5KeVXI3IytD2l31RkRcBrwIfEPSKtlGlS9FyKELeoUkbStpW+BhSVdI2k3Srun4EA9lHF6eHBsRH5PcHaYLcCTJ2C5Wnr8DC9ILsq4gGVP+xmxDyp3c59B96JW7mOQgSt3pGXXnsIqG7yJjDavL30CSi7QmeLTAFvkyvVR9MPDHiPhD3dkaVrbc59AFvXKlQ+SWFnYX85YZJ+keYAPgTEmdydkZBhmbJ+lQ4Chgv3Seb+HXMrnPoQt65VZM/90E2I7kTuEi+UA8nVVQOXQc0BuYFBFz0rFIcnU/x4wdA3wb+FVEvKHkJtvXZxxT3uQ+hz7LpUrScb33qTs7I21h3hkR/bONLD8kdQPWZ/E7Pj2SXURm+eIWevWsyeIjs81N51kZJF0AHAK8BNQN+xqAC3oZJPUDzmHRH8S6m5RvmGVceVKEHLqgV891wNOSRqXTBwAjMosmfw4ANomIL5pb0Rr0F+D7wDgW/UG0lsl9Dt3lUkXp6Yu7pJOPRESujpBnSdJdwMER8WnWseSRpKciom/WceRZEXLogm41QdLfga2B+1n8jk+nZBZUjkg6n+TuT7exeP6ezSyonClCDl3QrSZIOrqh+RFx7bKOJY/Se9rWFxGxxzIPJqeKkEMXdKsZkjoC60XEq1nHYpZHvvTfaoKk/YDngX+l070ljc40qByRtKakv6THIpC0uaTjso4rT4qQQxd0qxXnANsDswAi4nkgN6eL1YARwN3AOun0f4HvZRVMTo0g5zl0QbdaMS8iPqo3z5f+l2/1iBhJmrOImE9OT73LUO5z6PPQrVZMkHQYsJykjYFTgMczjilPZqfDJdTdbWcHoP4fSGta7nPog6JWEyStAPyUZPhckfz0/UVEfJ5pYDmRXgPxB2BLkrG8uwJDImJ8poHlSBFy6IJuVhCS2pIMEifg1YiYl3FIuZP3HLqgW6Yk/ZMmhhqOiEHLMJzcScfublRE3LasYsmrIuXQfeiWtYvSfwcDawE3pNOHAu9kElG+1I3bvQawE/BAOr07yTGI3BSjDBUmh26hW02QNDYi+jQ3zxqW3hzk6IiYkU6vDYyIiG9lG1l+FCGHPm3RakUnSQvPO09vLtApw3jyZt26QpR6B1gvq2ByKvc5dJeL1YrvAw9JmkRyQGp94PhsQ8qV+yXdDdyUTh8C3JdhPHmU+xy6y8VqhqTlgU3TyVdKx0aX9I2IuDebyPJB0oFA3R2yHomIUU2tb0vKew5d0C0XJD0bEdtmHUdeSXoiInbMOo48y0MO3YdueaGsA8i5DlkHUAA1n0MXdMsL/5SsjPNXuZrPoQu6mVlBuKBbXkzOOoCcc5dV5Wo+hz4oapmTtBLQNSJerze/V54GRsqSpDWBbunk9Ih4p97yLSPixWUfWX4UIYcu6JYpSUOB3wHvAu2A4RHxTLrMZ7Y0Q1Jv4M/AysD0dHZ3khuFnJSnGxxnpUg5dEG3TEl6HhgQETMkbQ9cB5wZEaMkPRcR22QbYW1L83dCRDxVb/4OwBURsXUmgeVIkXLoK0Uta8vVXW4dEU9L2h24Q9K65OCsghrQqX4hAoiIJyV56ITyFCaHLuiWtU8kbVTXf5621HcDbge2yDCuvLhL0p0kv2ympvPWBY4iveG2NaswOXSXi2VK0tbA7IiYWG9+O2BoRPwtm8jyQ9IAYH9KDugBoyNiTHZR5UtRcuiCbmZWED4P3WqWpBeyjqHWSVpZ0vmSXpb0gaSZ6fPzJa2SdXx5UKQcug/dMtXE7b9Ecgcja9pIkjvs7B4RbwNIWgsYni77Znah5UZhcuguF8uUpHnA32j4jJYhEdF5GYeUK5JejYhNWrrMFilSDt1Ct6yNBy5q6Ao8SXtlEE/evCnpx8C1dVc2plc8DmfRGRvWtMLk0H3olrXvAR83suzAZRhHXh0CdAEeTvt/PwAeAlYDhmYZWI4UJofucjEzKwh3uVimJK0eEe+XTB8BbA+8CFwVbnE0SZKAg0mOQdwK7EFyPvUrwJ8j4ssMw8uFIuXQLXTLVOkAXJLOAnYBbgT2BaZFxPezjK/WSfoTsAbQnqTranlgNLAP8E5EnJpheLlQpBy6oFumSgfgkvQssEtEzE6vFH02IrbKNsLaJumFiNgqzdfbwNoRMVdSW5L89co4xJpXpBy6y8Wy1lHSNiQH6JeLiNkAETFP0oJsQ8uF+bAwX89ExNx0er6k3HQVZKwwOXRBt6zNAC5Jn38gae10gK4upF80a9LbklaMiE8jYu+6memFMXMzjCtPCpNDd7lYTZLUBugQEXOyjiWPJK0ArBgR72YdS17lMYduoVvNkNQL6MHin8vbsokmf5y/yuU9hy7oVhMkXQP0AiYAdf2WQY6+TFly/ipXhBy6y8VqgqSXImLzrOPIK+evckXIoS/9t1rxhKRcf5ky5vxVLvc5dAvdaoKkXUku5ngb+IJk+NzI0znAWXL+KleEHLqgW02QNBE4DXiBRf2XRMSbmQWVI85f5YqQQx8UtVrxXkSMzjqIHHP+Kpf7HLqFbjUhHU9jFeCfJD93AYiI3JxhkCXnr3JFyKFb6FYrOpJ8iUpv95WrU8Yy5vxVLvc5dEG3WnF1RPy7dIakflkFk0POX+Vyn0N3uVhNKB1Gt6l51jDnr3JFyKFb6JYpSTsCOwFdJZ1WsmglYLlsosoP569yRcqhC7plrT2wIslnsXPJ/I+BIZlElC/OX+UKk0N3uVhNkLR+3fm+6UiLK0ZEYzePtnqcv8oVIYe+9N9qxa8lrSSpE8n9RF+S9KOsg8oR569yuc+hC7rVis3T1tABwF3ABsCRmUaUL85f5XKfQxd0qxXt0ns6HgCMjoh5JOcAW3mcv8rlPocu6FYrrgAmA52ARyStT3JQysrj/FUu9zn0QVGrWZLaRoTvK7qUnL/K5S2HPm3RMlXvvN+GXNLM8q80569yRcqhC7plrXPzq1gTnL/KFSaH7nKxXJB0ZkT8Ous48sr5q1wecuiDopYXB2cdQM45f5Wr+Ry6oFteKOsAcs75q1zN59AF3fLCfYOVcf4qV/M5dEG3vKj51lGNc/4qV/M5dEG3vPi/rAPIOeevcjWfQ5/lYjVBUgfgOGALoEPd/Ig4NrOgcsT5q1wRcugWutWK64G1gG8BDwPdgU8yjShfnL/K5T6HbqFbTZD0XERsI2l8RPRKB0l6NCJ2yDq2PHD+KleEHLqFbrViXvrvLElbAisDa2QYT944f5XLfQ596b/ViislrQr8DBhNckuws7MNKVecv8rlPofucjEzKwi30K0mSFoeOAjoQcnnMiLOyyqmPHH+KleEHLqgW634B/ARMA74IuNY8sj5q1zuc+guF6sJkl6MiC2zjiOvnL/KFSGHPsvFasXjkrbKOogcc/4ql/scuoVuNUHSS0BP4A2Sn7sCIiJ6ZRpYTjh/lStCDl3QrSakN+RdQkS8uaxjySPnr3JFyKELutUMScsBa7L4GQZTsosoX5y/yuU9hz7LxWqCpO8CPwfeAb5MZweQm5+7WXL+KleEHLqFbjVB0kSgb0TMzDqWPHL+KleEHPosF6sVU0nOAbal4/xVLvc5dJeL1YpJwEOS7qTkoo6IuCS7kHLF+atc7nPogm61Ykr6aJ8+rGWcv8rlPocu6Ja59MyCr0XE4VnHkkfOX+WKkkP3oVvmImIBsL6kXLaKsub8Va4oOXQL3WrFJODfkkYDs+tm5qn/MmPOX+Vyn0MXdKsVr6ePNkDnjGPJI+evcrnPoc9Dt5oiaSWS8TNydXPeWuH8VS7POXQfutUESX0kvQCMB16Q9B9JX886rrxw/ipXhBy6hW41QdJ44DsR8Wg6vTPwpzyNdJcl569yRcihW+hWKxbUfZEAIuIxYH6G8eSN81e53OfQLXTLlKRt06dHAR2Bm0gGRDoE+DwiTssqtjxw/ipXpBy6oFumJD1I8uVROqvuA1l3c4E9MgksJ5y/yhUphz5t0bJ2R8nzhr5U1jTnr3KFyaELumVtxfTfTYDtSO68LmA/4OmsgsoR569yhcmhu1ysJkh6BNin7txfSZ2BOyOif7aR5YPzV7ki5NBnuVitWBOYWzI9N51n5XH+Kpf7HLrLxWrFdcDTkkal0wcAIzKLJn+cv8rlPofucrGakZ4+tks6+UhEPJdlPHnj/FUu7zl0QTczKwj3oZuZFYQLuplZQbigW+FICkk3lEy3lfSepDuael0D25ksafVK1zFbVlzQrYhmA1tK6phOfwOYnmE8ZsuEC7oV1Rhgn/T5oSQDLgEgaTVJt0saL+lJSb3S+V0k3SNpgqSrWXQJOJKOkPS0pOclXZHeVJiS5Z0k3ZmOof2ipENa/y2aLc4F3YrqZmCYpA5AL+CpkmXnAs+l41z/hOT8Y4CfA49FxBbAKGA9AEmbkYy81y8iegMLgPp3h98beCsito6ILYF/tcq7MmuCLyyyQoqI8ZJ6kLTOx9RbvDNwULreA2nLfCWgPzA4nX+npA/T9fcEvg48IwmSIVbfrbfNF4CLJV0A3FE6rrbZsuKCbkU2GrgI2A3oUsF2BFwbEWc2tkJE/De9KGUg8EtJ90fEeRXs06zF3OViRXYNcG5EvFBv/qOkXSaSdgPej4iPgUeAw9L5A4BV0/XvB4ZIWiNdtpqk9Us3KGkdYE5E3ABcCGyL2TLmFroVVkRMAy5tYNE5wDXpPSTnAEen888FbpI0AXgcmJJu5yVJZwH3SGoDzAO+A7xZss2tgAslfZkuP7H678isab7038ysINzlYmZWEC7oZmYF4YJuZlYQLuhmZgXhgm5mVhAu6GZmBeGCbmZWEC7oZmYF8f8BA3bw4Ft6SE8AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bar_plots(models_, results_, 'vertical', 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the results, it appears that 300 LSTM units and 300 Dense units (the smallest permutation) improved performance against the originally large DBRNN-A with 1000+ units. This improvement is also noted across all training thresholds. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion and Future Direction\n",
    "### Results\n",
    "Given the data used in this project is different than the dataset used by Mani et al. [1], I anticipated different performance results, even when running identical models. However, a key limitation exists where Mani et al. implemented chronological time cross-validation (using a sliding window for training and testing sets) whereas this implementation implements a standard cross-validation procedure given the absence of a date/time feature in the available dataset. In addition, the DBRNN-A model proposed does appear to be heavily overfitting, possibly due to the size of the layers used and the general complexity of the model. \n",
    "\n",
    "The comparison analysis performed using different sizes of LSTM and Dense layer units revealed that performance of the proposed DBRNN-A was improved with a smaller, less complex model compared to the architecture size proposed by Mani et al. [1]. Therefore, there is additional work that can be done to explore hyper-parameter tuning to improve model performance since the smaller models shown here are more favorable due to their decreased complexity, faster training time, and increased resistance to overfitting. In addition, pretrained models like BERT (or other variants) can be employed in future work to attempt to capture more semantic information from bug reports to better enable developer assignment.   \n",
    "\n",
    "### Learnings\n",
    "In general, it was found that the text classification task becomes much more difficult when text input is extremely noisy and variable. Compared to other multi-class NLP tasks, bug triaging is particularly difficult because of the combination of both natural language as well as code traces and non-english word in the input data (bug reports) (compared to simpler classification tasks with english-only words like sentiment analysis for movies or news article classification). As a result, the processing steps and modeling become increasingly important due to this noise in order to achieve reasonable model accuracy.\n",
    "In addition, it's been shown the importance of exploring less complex architectures as they may be helpful in improving overall model performance in noisy input environments. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References:\n",
    "\n",
    "[1]: Senthil Mani, Anush Sankaran, and Rahul Aralikatte. 2019. DeepTriage: Exploring the Effectiveness of Deep Learning for Bug Triaging. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data (CoDS-COMAD '19). Association for Computing Machinery, New York, NY, USA, 171–179. DOI:https://doi.org/10.1145/3297001.3297023\n",
    "\n",
    "[2]:  Pamela Bhattacharya and Iulian Neamtiu. Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging. In International Conference on Software Maintenance, pages 1–10, 2010.\n",
    "\n",
    "[3]: Ahmed Tamrawi, Tung Thanh Nguyen, Jafar Al-Kofahi, and Tien N Nguyen. Fuzzy set-based automatic bug triaging: Nier track. In International Conference on Software Engineering, pages 884–887, 2011.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}