project_6_starter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 6: Analyzing Stock Sentiment from Twits\n",
    "## Instructions\n",
    "Each problem consists of a function to implement and instructions on how to implement the function.  The parts of the function that need to be implemented are marked with a `# TODO` comment.\n",
    "\n",
    "## Packages\n",
    "When you implement the functions, you'll only need to you use the packages you've used in the classroom, like [Pandas](https://pandas.pydata.org/) and [Numpy](http://www.numpy.org/). These packages will be imported for you. We recommend you don't add any import statements, otherwise the grader might not be able to run your code.\n",
    "\n",
    "### Load Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import nltk\n",
    "import os\n",
    "import random\n",
    "import re\n",
    "import torch\n",
    "import numpy as np\n",
    "from tqdm import tqdm\n",
    "\n",
    "from torch import nn, optim\n",
    "import torch.nn.functional as F\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "When deciding the value of a company, it's important to follow the news. For example, a product recall or natural disaster in a company's product chain. You want to be able to turn this information into a signal. Currently, the best tool for the job is a Neural Network. \n",
    "\n",
    "For this project, you'll use posts from the social media site [StockTwits](https://en.wikipedia.org/wiki/StockTwits). The community on StockTwits is full of investors, traders, and entrepreneurs. Each message posted is called a Twit. This is similar to Twitter's version of a post, called a Tweet. You'll build a model around these twits that generate a sentiment score.\n",
    "\n",
    "We've collected a bunch of twits, then hand labeled the sentiment of each. To capture the degree of sentiment, we'll use a five-point scale: very negative, negative, neutral, positive, very positive. Each twit is labeled -2 to 2 in steps of 1, from very negative to very positive respectively. You'll build a sentiment analysis model that will learn to assign sentiment to twits on its own, using this labeled data.\n",
    "\n",
    "The first thing we should to do, is load the data.\n",
    "\n",
    "## Import Twits \n",
    "### Load Twits Data \n",
    "This JSON file contains a list of objects for each twit in the `'data'` field:\n",
    "\n",
    "```\n",
    "{'data':\n",
    "  {'message_body': 'Neutral twit body text here',\n",
    "   'sentiment': 0},\n",
    "  {'message_body': 'Happy twit body text here',\n",
    "   'sentiment': 1},\n",
    "   ...\n",
    "}\n",
    "```\n",
    "\n",
    "The fields represent the following:\n",
    "\n",
    "* `'message_body'`: The text of the twit.\n",
    "* `'sentiment'`: Sentiment score for the twit, ranges from -2 to 2 in steps of 1, with 0 being neutral.\n",
    "\n",
    "\n",
    "To see what the data look like by printing the first 10 twits from the list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'message_body': '$FITB great buy at 26.00...ill wait', 'sentiment': 2, 'timestamp': '2018-07-01T00:00:09Z'}, {'message_body': '@StockTwits $MSFT', 'sentiment': 1, 'timestamp': '2018-07-01T00:00:42Z'}, {'message_body': '#STAAnalystAlert for $TDG : Jefferies Maintains with a rating of Hold setting target price at USD 350.00. Our own verdict is Buy  http://www.stocktargetadvisor.com/toprating', 'sentiment': 2, 'timestamp': '2018-07-01T00:01:24Z'}, {'message_body': '$AMD I heard there’s a guy who knows someone who thinks somebody knows something - on StockTwits.', 'sentiment': 1, 'timestamp': '2018-07-01T00:01:47Z'}, {'message_body': '$AMD reveal yourself!', 'sentiment': 0, 'timestamp': '2018-07-01T00:02:13Z'}, {'message_body': '$AAPL Why the drop? I warren Buffet taking out his position?', 'sentiment': 1, 'timestamp': '2018-07-01T00:03:10Z'}, {'message_body': '$BA bears have 1 reason on 06-29 to pay more attention https://dividendbot.com?s=BA', 'sentiment': -2, 'timestamp': '2018-07-01T00:04:09Z'}, {'message_body': '$BAC ok good we&#39;re not dropping in price over the weekend, lol', 'sentiment': 1, 'timestamp': '2018-07-01T00:04:17Z'}, {'message_body': '$AMAT - Daily Chart, we need to get back to above 50.', 'sentiment': 2, 'timestamp': '2018-07-01T00:08:01Z'}, {'message_body': '$GME 3% drop per week after spike... if no news in 3 months, back to 12s... if BO, then bingo... what is the odds?', 'sentiment': -2, 'timestamp': '2018-07-01T00:09:03Z'}]\n"
     ]
    }
   ],
   "source": [
    "with open(os.path.join('..', '..', 'data', 'project_6_stocktwits', 'twits.json'), 'r') as f:\n",
    "    twits = json.load(f)\n",
    "\n",
    "print(twits['data'][:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Length of Data\n",
    "Now let's look at the number of twits in dataset. Print the number of twits below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1548010"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"print out the number of twits\"\"\"\n",
    "\n",
    "# TODO Implement \n",
    "\n",
    "len(twits['data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split Message Body and Sentiment Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = [twit['message_body'] for twit in twits['data']]\n",
    "# Since the sentiment scores are discrete, we'll scale the sentiments to 0 to 4 for use in our network\n",
    "sentiments = [twit['sentiment'] + 2 for twit in twits['data']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing the Data\n",
    "With our data in hand we need to preprocess our text. These twits are collected by filtering on ticker symbols where these are denoted with a leader $ symbol in the twit itself. For example,\n",
    "\n",
    "`{'message_body': 'RT @google Our annual look at the year in Google blogging (and beyond) http://t.co/sptHOAh8 $GOOG',\n",
    " 'sentiment': 0}`\n",
    "\n",
    "The ticker symbols don't provide information on the sentiment, and they are in every twit, so we should remove them. This twit also has the `@google` username, again not providing sentiment information, so we should also remove it. We also see a URL `http://t.co/sptHOAh8`. Let's remove these too.\n",
    "\n",
    "The easiest way to remove specific words or phrases is with regex using the `re` module. You can sub out specific patterns with a space:\n",
    "\n",
    "```python\n",
    "re.sub(pattern, ' ', text)\n",
    "```\n",
    "This will substitute a space with anywhere the pattern matches in the text. Later when we tokenize the text, we'll split appropriately on those spaces."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to /home/aswin/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "nltk.download('wordnet')\n",
    "import re\n",
    "\n",
    "def preprocess(message):\n",
    "    \"\"\"\n",
    "    This function takes a string as input, then performs these operations: \n",
    "        - lowercase\n",
    "        - remove URLs\n",
    "        - remove ticker symbols \n",
    "        - removes punctuation\n",
    "        - tokenize by splitting the string on whitespace \n",
    "        - removes any single character tokens\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "        message : The text message to be preprocessed.\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "        tokens: The preprocessed text into tokens.\n",
    "    \"\"\" \n",
    "    #TODO: Implement \n",
    "    url_regex = re.compile(r\"(https?://[^\\s]+)\")\n",
    "    stock_regex = re.compile(r\"(\\$[a-zA-Z]+[^\\s]+)\")\n",
    "    user_regex = re.compile(r\"(\\@[\\w]+[^\\s]+)\")\n",
    "    noletter_regex = re.compile(r\"[^a-zA-Z]\")\n",
    "    \n",
    "    # Lowercase the twit message\n",
    "    text = message.lower()\n",
    "    \n",
    "    # Replace URLs with a space in the message\n",
    "    text, _ = url_regex.subn(' ', text)\n",
    "    \n",
    "    # Replace ticker symbols with a space. The ticker symbols are any stock symbol that starts with $.\n",
    "    text, _ = stock_regex.subn(' ', text)\n",
    "    \n",
    "    # Replace StockTwits usernames with a space. The usernames are any word that starts with @.\n",
    "    text, _ = user_regex.subn(' ', text)\n",
    "\n",
    "    # Replace everything not a letter with a space\n",
    "    text, _ = noletter_regex.subn(' ', text)\n",
    "    \n",
    "    # Tokenize by splitting the string on whitespace into a list of words\n",
    "    tokens = text.split()\n",
    "\n",
    "    # Lemmatize words using the WordNetLemmatizer. You can ignore any word that is not longer than one character.\n",
    "    wnl = nltk.stem.WordNetLemmatizer()\n",
    "    tokens = [wnl.lemmatize(w) for w in tokens if len(w) > 1]\n",
    "    \n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess All the Twits \n",
    "Now we can preprocess each of the twits in our dataset. Apply the function `preprocess` to all the twit messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# TODO Implement\n",
    "from tqdm import tqdm\n",
    "import numpy as np\n",
    "from collections import Counter\n",
    "from concurrent import futures\n",
    "import time\n",
    "\n",
    "start = time.time()\n",
    "tokenized = []\n",
    "pool_executor = futures.ProcessPoolExecutor(max_workers=5)\n",
    "for token in tqdm(pool_executor.map(preprocess, messages, chunksize=1000)):\n",
    "    tokenized.append(token)\n",
    "end = time.time()\n",
    "print(\"%.3f seconds\", float(end - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_string = ' '.join([w for t in tqdm(tokenized) for w in t])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag of Words\n",
    "Now with all of our messages tokenized, we want to create a vocabulary and count up how often each word appears in our entire corpus. Use the [`Counter`](https://docs.python.org/3.1/library/collections.html#collections.Counter) function to count up all the tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from functools import reduce\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import threading\n",
    "from time import sleep\n",
    "\n",
    "\"\"\"\n",
    "Create a vocabulary by using Bag of words\n",
    "\"\"\"\n",
    "start = time.time()\n",
    "bow = Counter(full_string.split())\n",
    "end = time.time()\n",
    "print(\"%.3f seconds\", float(end - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency of Words Appearing in Message\n",
    "With our vocabulary, now we'll remove some of the most common words such as 'the', 'and', 'it', etc. These words don't contribute to identifying sentiment and are really common, resulting in a lot of noise in our input. If we can filter these out, then our network should have an easier time learning.\n",
    "\n",
    "We also want to remove really rare words that show up in a only a few twits. Here you'll want to divide the count of each word by the number of messages. Then remove words that only appear in some small fraction of the messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freqs = dict(zip(list(bow.keys()), np.array(list(bow.values()))/len(messages)))\n",
    "np.max(list(freqs.values())), np.min(list(freqs.values()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Set the following variables:\n",
    "    freqs\n",
    "    low_cutoff\n",
    "    high_cutoff\n",
    "    K_most_common\n",
    "\"\"\"\n",
    "\n",
    "# TODO Implement \n",
    "\n",
    "# Dictionart that contains the Frequency of words appearing in messages.\n",
    "# The key is the token and the value is the frequency of that word in the corpus.\n",
    "freqs = dict(zip(list(bow.keys()), np.array(list(bow.values()))/len(messages)))\n",
    "\n",
    "# Float that is the frequency cutoff. Drop words with a frequency that is lower or equal to this number.\n",
    "low_cutoff = 5e-4\n",
    "\n",
    "# Integer that is the cut off for most common words. Drop words that are the `high_cutoff` most common words.\n",
    "high_cutoff = 0.078\n",
    "\n",
    "# The k most common words in the corpus. Use `high_cutoff` as the k.\n",
    "K_most_common = [word for word in freqs if (freqs[word] > high_cutoff)]\n",
    "\n",
    "\n",
    "filtered_words = [word for word in freqs if (freqs[word] > low_cutoff and word not in K_most_common)]\n",
    "print(K_most_common)\n",
    "len(filtered_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Updating Vocabulary by Removing Filtered Words\n",
    "Let's creat three variables that will help with our vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from time import sleep\n",
    "\n",
    "\"\"\"\n",
    "Set the following variables:\n",
    "    vocab\n",
    "    id2vocab\n",
    "    filtered\n",
    "\"\"\"\n",
    "\n",
    "#TODO Implement\n",
    "\n",
    "# A dictionary for the `filtered_words`. The key is the word and value is an id that represents the word.\n",
    "sorted_vocab = list(bow.keys())\n",
    "vocab = {word: ii for ii, word in tqdm(enumerate(sorted_vocab))}\n",
    "# Reverse of the `vocab` dictionary. The key is word id and value is the word. \n",
    "id2vocab = {ii: word for ii, word in tqdm(enumerate(sorted_vocab))}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fw = np.array(filtered_words)\n",
    "\n",
    "filtered = [[w for w in token if len(np.where(fw == w)[0]) > 0] for token in tqdm(tokenized)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "pickle.dump(tokenized, open('tokenized.pkl', 'wb'))\n",
    "pickle.dump(bow, open('bow.pkl', 'wb'))\n",
    "pickle.dump(K_most_common, open('K_most_common.pkl', 'wb'))\n",
    "pickle.dump(filtered_words, open('filtered_words.pkl', 'wb'))\n",
    "pickle.dump(vocab, open('vocab.pkl', 'wb'))\n",
    "pickle.dump(id2vocab, open('id2vocab.pkl', 'wb'))\n",
    "pickle.dump(filtered, open('filtered.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "tokenized = pickle.load(open('tokenized.pkl', 'rb'))\n",
    "bow = pickle.load(open('bow.pkl', 'rb'))\n",
    "K_most_common = pickle.load(open('K_most_common.pkl', 'rb'))\n",
    "filtered_words = pickle.load(open('filtered_words.pkl', 'rb'))\n",
    "vocab = pickle.load(open('vocab.pkl', 'rb'))\n",
    "id2vocab = pickle.load(open('id2vocab.pkl', 'rb'))\n",
    "filtered = pickle.load(open('filtered.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Balancing the classes\n",
    "Let's do a few last pre-processing steps. If we look at how our twits are labeled, we'll find that 50% of them are neutral. This means that our network will be 50% accurate just by guessing 0 every single time. To help our network learn appropriately, we'll want to balance our classes.\n",
    "That is, make sure each of our different sentiment scores show up roughly as frequently in the data.\n",
    "\n",
    "What we can do here is go through each of our examples and randomly drop twits with neutral sentiment. What should be the probability we drop these twits if we want to get around 20% neutral twits starting at 50% neutral? We should also take this opportunity to remove messages with length 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "balanced = {'messages': [], 'sentiments':[]}\n",
    "\n",
    "n_neutral = sum(1 for each in sentiments if each == 2)\n",
    "N_examples = len(sentiments)\n",
    "keep_prob = (N_examples - n_neutral)/4/n_neutral\n",
    "\n",
    "for idx, sentiment in enumerate(sentiments):\n",
    "    message = filtered[idx]\n",
    "    if len(message) == 0:\n",
    "        # skip this message because it has length zero\n",
    "        continue\n",
    "    elif sentiment != 2 or random.random() < keep_prob:\n",
    "        balanced['messages'].append(message)\n",
    "        balanced['sentiments'].append(sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you did it correctly, you should see the following result "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.19301364362040765"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_neutral = sum(1 for each in balanced['sentiments'] if each == 2)\n",
    "N_examples = len(balanced['sentiments'])\n",
    "n_neutral/N_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally let's convert our tokens into integer ids which we can pass to the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_ids = [[vocab[word] for word in message] for message in balanced['messages']]\n",
    "sentiments = balanced['sentiments']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network\n",
    "Now we have our vocabulary which means we can transform our tokens into ids, which are then passed to our network. So, let's define the network now!\n",
    "\n",
    "Here is a nice diagram showing the network we'd like to build: \n",
    "\n",
    "#### Embed -> RNN -> Dense -> Softmax\n",
    "### Implement the text classifier\n",
    "Before we build text classifier, if you remember from the other network that you built in  \"Sentiment Analysis with an RNN\"  exercise  - which there, the network called \" SentimentRNN\", here we named it \"TextClassifer\" - consists of three main parts: 1) init function `__init__` 2) forward pass `forward`  3) hidden state `init_hidden`. \n",
    "\n",
    "This network is pretty similar to the network you built expect in the  `forward` pass, we use softmax instead of sigmoid. The reason we are not using sigmoid is that the output of NN is not a binary. In our network, sentiment scores have 5 possible outcomes. We are looking for an outcome with the highest probability thus softmax is a better choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TextClassifier(nn.Module):\n",
    "    def __init__(self, vocab_size, embed_size, lstm_size, output_size, lstm_layers=1, dropout=0.1):\n",
    "        \"\"\"\n",
    "        Initialize the model by setting up the layers.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "            vocab_size : The vocabulary size.\n",
    "            embed_size : The embedding layer size.\n",
    "            lstm_size : The LSTM layer size.\n",
    "            output_size : The output size.\n",
    "            lstm_layers : The number of LSTM layers.\n",
    "            dropout : The dropout probability.\n",
    "        \"\"\"\n",
    "        \n",
    "        super().__init__()\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embed_size = embed_size\n",
    "        self.lstm_size = lstm_size\n",
    "        self.output_size = output_size\n",
    "        self.lstm_layers = lstm_layers\n",
    "        self.dropout = dropout\n",
    "        \n",
    "        # TODO Implement\n",
    "\n",
    "        # Setup embedding layer\n",
    "        self.embedding = nn.Embedding(self.vocab_size, self.embed_size)\n",
    "        self.lstm = nn.LSTM(self.embed_size, self.lstm_size, self.lstm_layers, dropout=dropout, batch_first=True)\n",
    "        \n",
    "        # Setup additional layers\n",
    "        self.fc = nn.Linear(self.lstm_size, self.output_size)\n",
    "        \n",
    "        self.dropout = nn.Dropout(0.1)\n",
    "        \n",
    "        self.log_softmax = nn.LogSoftmax(dim=2)\n",
    "\n",
    "\n",
    "    def init_hidden(self, batch_size):\n",
    "        \"\"\" \n",
    "        Initializes hidden state\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "            batch_size : The size of batches.\n",
    "        \n",
    "        Returns\n",
    "        -------\n",
    "            hidden_state\n",
    "            \n",
    "        \"\"\"\n",
    "        \n",
    "        # TODO Implement \n",
    "        \n",
    "        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,\n",
    "        # initialized to zero, for hidden state and cell state of LSTM\n",
    "        \n",
    "        weights = next(self.parameters()).data\n",
    "        \n",
    "        return tuple([weights.new(self.lstm_layers, batch_size, self.lstm_size).zero_(), \n",
    "                     weights.new(self.lstm_layers, batch_size, self.lstm_size).zero_()])\n",
    "\n",
    "\n",
    "    def forward(self, nn_input, hidden_state):\n",
    "        \"\"\"\n",
    "        Perform a forward pass of our model on nn_input.\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "            nn_input : The batch of input to the NN.\n",
    "            hidden_state : The LSTM hidden state.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "            logps: log softmax output\n",
    "            hidden_state: The new hidden state.\n",
    "\n",
    "        \"\"\"\n",
    "        \n",
    "        # TODO Implement\n",
    "        \n",
    "        embedding = self.embedding(nn_input)\n",
    "        output, hidden_state = self.lstm(embedding, hidden_state)\n",
    "        output = self.dropout(output)\n",
    "        output = self.fc(output)\n",
    "        \n",
    "        softmax_out = self.log_softmax(output)\n",
    "        softmax_out = softmax_out[:, -1, :]\n",
    "        \n",
    "        return softmax_out, hidden_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.2597, 0.1586, 0.1541, 0.1936, 0.2340],\n",
       "        [0.2604, 0.1508, 0.1449, 0.2113, 0.2327],\n",
       "        [0.2538, 0.1587, 0.1515, 0.1950, 0.2410],\n",
       "        [0.2561, 0.1570, 0.1533, 0.1943, 0.2393]], grad_fn=<ExpBackward>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = TextClassifier(len(vocab), 10, 6, 5, dropout=0.1, lstm_layers=2)\n",
    "model.embedding.weight.data.uniform_(-1, 1)\n",
    "input = torch.randint(0, 1000, (4, 5), dtype=torch.int64)\n",
    "hidden = model.init_hidden(4)\n",
    "logps, _ = model.forward(input, hidden)\n",
    "torch.exp(logps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "### DataLoaders and Batching\n",
    "Now we should build a generator that we can use to loop through our data. It'll be more efficient if we can pass our sequences in as batches. Our input tensors should look like `(sequence_length, batch_size)`. So if our sequences are 40 tokens long and we pass in 25 sequences, then we'd have an input size of `(40, 25)`.\n",
    "\n",
    "If we set our sequence length to 40, what do we do with messages that are more or less than 40 tokens? For messages with fewer than 40 tokens, we will pad the empty spots with zeros. We should be sure to **left** pad so that the RNN starts from nothing before going through the data. If the message has 20 tokens, then the first 20 spots of our 40 long sequence will be 0. If a message has more than 40 tokens, we'll just keep the first 40 tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dataloader(messages, labels, sequence_length=30, batch_size=32, shuffle=False):\n",
    "    \"\"\" \n",
    "    Build a dataloader.\n",
    "    \"\"\"\n",
    "    if shuffle:\n",
    "        indices = list(range(len(messages)))\n",
    "        random.shuffle(indices)\n",
    "        messages = [messages[idx] for idx in indices]\n",
    "        labels = [labels[idx] for idx in indices]\n",
    "\n",
    "    total_sequences = len(messages)\n",
    "\n",
    "    for ii in range(0, total_sequences, batch_size):\n",
    "        batch_messages = messages[ii: ii+batch_size]\n",
    "        if len(batch_messages) < batch_size:\n",
    "            b = [[0]*token_ids.shape[1]]*(batch_size-len(batch_messages))\n",
    "            batch_messages = np.vstack([batch_messages, b])\n",
    "        \n",
    "        # First initialize a tensor of all zeros\n",
    "        batch = torch.tensor(batch_messages, dtype=torch.int64)\n",
    "#         for batch_num, tokens in enumerate(batch_messages):\n",
    "#             token_tensor = torch.tensor(tokens)\n",
    "#             # Left pad!\n",
    "#             start_idx = max(sequence_length - len(token_tensor), 0)\n",
    "#             batch[batch_num, start_idx:] = token_tensor[:sequence_length]\n",
    "        \n",
    "        labels_new = labels[ii: ii+len(batch_messages)]\n",
    "        if len(labels_new) < batch_size:\n",
    "            b = [0]*(batch_size-len(labels_new))\n",
    "            for i in b:\n",
    "                labels_new.append(i)\n",
    "        \n",
    "        label_tensor = torch.tensor(labels_new)\n",
    "        \n",
    "        yield batch, label_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and  Validation\n",
    "With our data in nice shape, we'll split it into training and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "def pad_features(reviews_ints, seq_length):\n",
    "    features=np.zeros((len(reviews_ints), seq_length), dtype=int)\n",
    "    \n",
    "    for i, row in enumerate(reviews_ints):\n",
    "        features[i, -len(row):] = np.array(row)[:seq_length]\n",
    "    \n",
    "    return features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "l = np.vectorize(len)\n",
    "token_ids = pad_features(token_ids, seq_length=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Split data into training and validation datasets. Use an appropriate split size.\n",
    "The features are the `token_ids` and the labels are the `sentiments`.\n",
    "\"\"\"   \n",
    "train_split = 0.7\n",
    "val_split = 1 - train_split\n",
    "\n",
    "# TODO Implement \n",
    "\n",
    "train_idx = int(len(token_ids)*train_split)\n",
    "\n",
    "train_features = token_ids[:train_idx]\n",
    "valid_features = token_ids[train_idx:]\n",
    "train_labels = sentiments[:train_idx]\n",
    "valid_labels = sentiments[train_idx:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.return_types.topk(\n",
       "values=tensor([[0.2107],\n",
       "        [0.2141],\n",
       "        [0.2338],\n",
       "        [0.2264],\n",
       "        [0.2277],\n",
       "        [0.2255],\n",
       "        [0.2190],\n",
       "        [0.2161],\n",
       "        [0.2268],\n",
       "        [0.2273],\n",
       "        [0.2256],\n",
       "        [0.2333],\n",
       "        [0.2349],\n",
       "        [0.2188],\n",
       "        [0.2254],\n",
       "        [0.2081],\n",
       "        [0.2252],\n",
       "        [0.2496],\n",
       "        [0.2356],\n",
       "        [0.2267],\n",
       "        [0.2356],\n",
       "        [0.2337],\n",
       "        [0.2106],\n",
       "        [0.2319],\n",
       "        [0.2141],\n",
       "        [0.2262],\n",
       "        [0.2344],\n",
       "        [0.2195],\n",
       "        [0.2292],\n",
       "        [0.2151],\n",
       "        [0.2347],\n",
       "        [0.2246],\n",
       "        [0.2389],\n",
       "        [0.2288],\n",
       "        [0.2229],\n",
       "        [0.2153],\n",
       "        [0.2244],\n",
       "        [0.2290],\n",
       "        [0.2256],\n",
       "        [0.2152],\n",
       "        [0.2281],\n",
       "        [0.2460],\n",
       "        [0.2266],\n",
       "        [0.2320],\n",
       "        [0.2189],\n",
       "        [0.2226],\n",
       "        [0.2170],\n",
       "        [0.2059],\n",
       "        [0.2292],\n",
       "        [0.2235],\n",
       "        [0.2190],\n",
       "        [0.2392],\n",
       "        [0.2218],\n",
       "        [0.2243],\n",
       "        [0.2246],\n",
       "        [0.2269],\n",
       "        [0.2217],\n",
       "        [0.2213],\n",
       "        [0.2399],\n",
       "        [0.2308],\n",
       "        [0.2316],\n",
       "        [0.2327],\n",
       "        [0.2358],\n",
       "        [0.2298],\n",
       "        [0.2104],\n",
       "        [0.2217],\n",
       "        [0.2182],\n",
       "        [0.2276],\n",
       "        [0.2299],\n",
       "        [0.2148],\n",
       "        [0.2242],\n",
       "        [0.2359],\n",
       "        [0.2468],\n",
       "        [0.2126],\n",
       "        [0.2343],\n",
       "        [0.2278],\n",
       "        [0.2223],\n",
       "        [0.2352],\n",
       "        [0.2088],\n",
       "        [0.2152],\n",
       "        [0.2645],\n",
       "        [0.2316],\n",
       "        [0.2209],\n",
       "        [0.2187],\n",
       "        [0.2467],\n",
       "        [0.2277],\n",
       "        [0.2133],\n",
       "        [0.2174],\n",
       "        [0.2108],\n",
       "        [0.2130],\n",
       "        [0.2207],\n",
       "        [0.2098],\n",
       "        [0.2260],\n",
       "        [0.2440],\n",
       "        [0.2247],\n",
       "        [0.2103],\n",
       "        [0.2295],\n",
       "        [0.2448],\n",
       "        [0.2187],\n",
       "        [0.2306],\n",
       "        [0.2279],\n",
       "        [0.2305],\n",
       "        [0.2273],\n",
       "        [0.2128],\n",
       "        [0.2191],\n",
       "        [0.2285],\n",
       "        [0.2132],\n",
       "        [0.2085],\n",
       "        [0.2315],\n",
       "        [0.2225],\n",
       "        [0.2260],\n",
       "        [0.2288],\n",
       "        [0.2083],\n",
       "        [0.2154],\n",
       "        [0.2261],\n",
       "        [0.2310],\n",
       "        [0.2233],\n",
       "        [0.2302],\n",
       "        [0.2142],\n",
       "        [0.2119],\n",
       "        [0.2261],\n",
       "        [0.2127],\n",
       "        [0.2812],\n",
       "        [0.2173],\n",
       "        [0.2202],\n",
       "        [0.2258],\n",
       "        [0.2289],\n",
       "        [0.2163]], grad_fn=<TopkBackward>),\n",
       "indices=tensor([[3],\n",
       "        [1],\n",
       "        [1],\n",
       "        [1],\n",
       "        [3],\n",
       "        [4],\n",
       "        [2],\n",
       "        [1],\n",
       "        [0],\n",
       "        [2],\n",
       "        [2],\n",
       "        [0],\n",
       "        [2],\n",
       "        [3],\n",
       "        [0],\n",
       "        [3],\n",
       "        [0],\n",
       "        [2],\n",
       "        [0],\n",
       "        [1],\n",
       "        [2],\n",
       "        [4],\n",
       "        [0],\n",
       "        [3],\n",
       "        [3],\n",
       "        [0],\n",
       "        [2],\n",
       "        [0],\n",
       "        [0],\n",
       "        [1],\n",
       "        [0],\n",
       "        [2],\n",
       "        [0],\n",
       "        [0],\n",
       "        [0],\n",
       "        [0],\n",
       "        [0],\n",
       "        [0],\n",
       "        [3],\n",
       "        [1],\n",
       "        [4],\n",
       "        [2],\n",
       "        [2],\n",
       "        [1],\n",
       "        [0],\n",
       "        [2],\n",
       "        [2],\n",
       "        [2],\n",
       "        [2],\n",
       "        [4],\n",
       "        [3],\n",
       "        [1],\n",
       "        [0],\n",
       "        [0],\n",
       "        [0],\n",
       "        [2],\n",
       "        [2],\n",
       "        [0],\n",
       "        [0],\n",
       "        [2],\n",
       "        [2],\n",
       "        [1],\n",
       "        [0],\n",
       "        [2],\n",
       "        [4],\n",
       "        [2],\n",
       "        [4],\n",
       "        [2],\n",
       "        [0],\n",
       "        [2],\n",
       "        [2],\n",
       "        [0],\n",
       "        [0],\n",
       "        [0],\n",
       "        [2],\n",
       "        [2],\n",
       "        [0],\n",
       "        [2],\n",
       "        [1],\n",
       "        [0],\n",
       "        [2],\n",
       "        [1],\n",
       "        [0],\n",
       "        [0],\n",
       "        [0],\n",
       "        [2],\n",
       "        [1],\n",
       "        [2],\n",
       "        [0],\n",
       "        [0],\n",
       "        [2],\n",
       "        [0],\n",
       "        [1],\n",
       "        [2],\n",
       "        [0],\n",
       "        [4],\n",
       "        [0],\n",
       "        [0],\n",
       "        [0],\n",
       "        [2],\n",
       "        [2],\n",
       "        [0],\n",
       "        [0],\n",
       "        [3],\n",
       "        [0],\n",
       "        [2],\n",
       "        [0],\n",
       "        [4],\n",
       "        [3],\n",
       "        [2],\n",
       "        [0],\n",
       "        [2],\n",
       "        [2],\n",
       "        [0],\n",
       "        [0],\n",
       "        [2],\n",
       "        [1],\n",
       "        [2],\n",
       "        [0],\n",
       "        [1],\n",
       "        [4],\n",
       "        [3],\n",
       "        [2],\n",
       "        [0],\n",
       "        [1],\n",
       "        [3],\n",
       "        [2],\n",
       "        [4]]))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_batch, labels = next(iter(dataloader(train_features, train_labels, sequence_length=20, batch_size=128)))\n",
    "model = TextClassifier(len(vocab)+1, 200, 128, 5, dropout=0.)\n",
    "hidden = model.init_hidden(128)\n",
    "logps, hidden = model.forward(text_batch, hidden)\n",
    "torch.exp(logps).topk(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "It's time to train the neural network!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TextClassifier(\n",
       "  (embedding): Embedding(98408, 200)\n",
       "  (lstm): LSTM(200, 128, batch_first=True)\n",
       "  (fc): Linear(in_features=128, out_features=5, bias=True)\n",
       "  (dropout): Dropout(p=0.1)\n",
       "  (log_softmax): LogSoftmax()\n",
       ")"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "model = TextClassifier(len(vocab)+1, 200, 128, 5, lstm_layers=1, dropout=0.)\n",
    "model.embedding.weight.data.uniform_(-1, 1)\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "  0%|          | 0/4 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 1\n",
      "Step: 1/1000, Accuracy: 1.00000000\n",
      "Epoch: 1/4... Step: 1000... Loss: 1.054255... Val Loss: 0.960143\n",
      "Step: 1/2000, Accuracy: 1.00000000\n",
      "Epoch: 1/4... Step: 2000... Loss: 0.969333... Val Loss: 0.895112\n",
      "Step: 1/3000, Accuracy: 1.00000000\n",
      "Epoch: 1/4... Step: 3000... Loss: 0.926033... Val Loss: 0.874179\n",
      "Step: 1/4000, Accuracy: 1.00000000\n",
      "Epoch: 1/4... Step: 4000... Loss: 0.898500... Val Loss: 0.843888\n",
      "Step: 1/5000, Accuracy: 1.00000000\n",
      "Epoch: 1/4... Step: 5000... Loss: 0.878852... Val Loss: 0.834555\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 25%|██▌       | 1/4 [1:13:10<3:39:31, 4390.58s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 2\n",
      "Step: 2/6000, Accuracy: 1.00000000\n",
      "Epoch: 2/4... Step: 6000... Loss: 0.785032... Val Loss: 0.825089\n",
      "Step: 2/7000, Accuracy: 1.00000000\n",
      "Epoch: 2/4... Step: 7000... Loss: 0.782902... Val Loss: 0.826481\n",
      "Step: 2/8000, Accuracy: 1.00000000\n",
      "Epoch: 2/4... Step: 8000... Loss: 0.780123... Val Loss: 0.819801\n",
      "Step: 2/9000, Accuracy: 1.00000000\n",
      "Epoch: 2/4... Step: 9000... Loss: 0.778950... Val Loss: 0.810593\n",
      "Step: 2/10000, Accuracy: 1.00000000\n",
      "Epoch: 2/4... Step: 10000... Loss: 0.777000... Val Loss: 0.812058\n",
      "Step: 2/11000, Accuracy: 1.00000000\n",
      "Epoch: 2/4... Step: 11000... Loss: 0.775745... Val Loss: 0.802455\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 50%|█████     | 2/4 [2:30:32<2:28:52, 4466.04s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 3\n",
      "Step: 3/12000, Accuracy: 1.00000000\n",
      "Epoch: 3/4... Step: 12000... Loss: 0.761714... Val Loss: 0.807732\n",
      "Step: 3/13000, Accuracy: 1.00000000\n",
      "Epoch: 3/4... Step: 13000... Loss: 0.759037... Val Loss: 0.807115\n",
      "Step: 3/14000, Accuracy: 1.00000000\n",
      "Epoch: 3/4... Step: 14000... Loss: 0.757159... Val Loss: 0.803555\n",
      "Step: 3/15000, Accuracy: 1.00000000\n",
      "Epoch: 3/4... Step: 15000... Loss: 0.755217... Val Loss: 0.806277\n",
      "Step: 3/16000, Accuracy: 1.00000000\n",
      "Epoch: 3/4... Step: 16000... Loss: 0.753905... Val Loss: 0.799041\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 75%|███████▌  | 3/4 [3:43:23<1:13:57, 4437.48s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 4\n",
      "Step: 4/17000, Accuracy: 1.00000000\n",
      "Epoch: 4/4... Step: 17000... Loss: 0.719654... Val Loss: 0.808438\n",
      "Step: 4/18000, Accuracy: 1.00000000\n",
      "Epoch: 4/4... Step: 18000... Loss: 0.718484... Val Loss: 0.811749\n",
      "Step: 4/19000, Accuracy: 1.00000000\n",
      "Epoch: 4/4... Step: 19000... Loss: 0.717403... Val Loss: 0.814007\n",
      "Step: 4/20000, Accuracy: 1.00000000\n",
      "Epoch: 4/4... Step: 20000... Loss: 0.716855... Val Loss: 0.809772\n",
      "Step: 4/21000, Accuracy: 1.00000000\n",
      "Epoch: 4/4... Step: 21000... Loss: 0.716506... Val Loss: 0.808020\n",
      "Step: 4/22000, Accuracy: 1.00000000\n",
      "Epoch: 4/4... Step: 22000... Loss: 0.716163... Val Loss: 0.807513\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [5:00:32<00:00, 4494.96s/it]  \n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Train your model with dropout. Make sure to clip your gradients.\n",
    "Print the training loss, validation loss, and validation accuracy for every 100 steps.\n",
    "\"\"\"\n",
    "\n",
    "epochs = 4\n",
    "batch_size = 128\n",
    "learning_rate = 0.001\n",
    "\n",
    "print_every = 1000\n",
    "clip = 5\n",
    "criterion = nn.NLLLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=learning_rate)\n",
    "model.train()\n",
    "\n",
    "steps = 0\n",
    "train_losses = []\n",
    "test_losses = []\n",
    "for epoch in tqdm(range(epochs)):\n",
    "    print('Starting epoch {}'.format(epoch + 1))\n",
    "    \n",
    "    hidden = model.init_hidden(batch_size)\n",
    "    running_loss = 0.0\n",
    "    \n",
    "    for text_batch, labels in dataloader(\n",
    "            train_features, train_labels, batch_size=batch_size, sequence_length=20, shuffle=True):\n",
    "        \n",
    "        steps += 1\n",
    "        \n",
    "        # Set Device\n",
    "        text_batch, labels = text_batch.to(device), labels.to(device)\n",
    "        \n",
    "        for each in hidden:\n",
    "            each.to(device)\n",
    "        hidden = tuple([each.data for each in hidden])\n",
    "        \n",
    "        # TODO Implement: Train Model\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        output, hidden = model.forward(text_batch, hidden)\n",
    "        loss = criterion(output, labels.long())\n",
    "        loss.backward()\n",
    "        nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "        optimizer.step()\n",
    "        \n",
    "        running_loss += loss.item()\n",
    "        \n",
    "        val_losses = []\n",
    "        if steps % print_every == 0:\n",
    "            val_h = model.init_hidden(batch_size)\n",
    "            \n",
    "            accuracy = 0.0\n",
    "            \n",
    "            # TODO Implement: Print metrics\n",
    "            with torch.no_grad():\n",
    "                model.eval()\n",
    "                for val_text_batch, val_labels in dataloader(valid_features, valid_labels, batch_size=batch_size, \n",
    "                                                     sequence_length=20, shuffle=True):\n",
    "\n",
    "                    for each in val_h:\n",
    "                        each.to(device)\n",
    "                    val_h = tuple([each.data for each in val_h])\n",
    "                    val_text_batch, val_labels = val_text_batch.to(device), val_labels.to(device)\n",
    "                    val_output, val_h = model.forward(val_text_batch, val_h)\n",
    "                    val_loss = criterion(val_output, val_labels.long())\n",
    "\n",
    "                    val_losses.append(val_loss.item())\n",
    "\n",
    "                    topval, topclass = torch.exp(val_output).topk(1)\n",
    "                    accuracy += torch.sum(topclass.squeeze() == val_labels)\n",
    "\n",
    "            train_losses.append(running_loss/print_every)\n",
    "            running_loss = 0.0\n",
    "            print(\"Step: {}/{}, Accuracy: {:.8f}\".format(epoch+1, steps, \n",
    "                        accuracy/(len(val_labels)*1000)))\n",
    "            \n",
    "            print(\"Epoch: {}/{}...\".format(epoch+1, epochs),\n",
    "              \"Step: {}...\".format(steps),\n",
    "              \"Loss: {:.6f}...\".format(np.mean(train_losses)),\n",
    "              \"Val Loss: {:.6f}\".format(np.mean(val_losses)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making Predictions\n",
    "### Prediction \n",
    "Okay, now that you have a trained model, try it on some new twits and see if it works appropriately. Remember that for any new text, you'll need to preprocess it first before passing it to the network. Implement the `predict` function to generate the prediction vector from a message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_text(text):\n",
    "    return preprocess(text)\n",
    "\n",
    "def get_word_ints(text_split):\n",
    "    token_ints = []\n",
    "    other_ints = []\n",
    "    token_ints = [[vocab[t] for t in text_split if t in filtered_words]]\n",
    "    \n",
    "    return token_ints\n",
    "\n",
    "def predict(text, model, vocab):\n",
    "    \"\"\" \n",
    "    Make a prediction on a single sentence.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "        text : The string to make a prediction on.\n",
    "        model : The model to use for making the prediction.\n",
    "        vocab : Dictionary for word to word ids. The key is the word and the value is the word id.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "        pred : Prediction vector\n",
    "    \"\"\"    \n",
    "    \n",
    "    # TODO Implement\n",
    "    \n",
    "    tokens = preprocess(text)\n",
    "    \n",
    "    # Filter non-vocab words\n",
    "    vocab_tokens = get_word_ints(tokens)\n",
    "    # Convert words to ids\n",
    "    \n",
    "    # Adding a batch dimension\n",
    "    text_input = torch.tensor(pad_features(vocab_tokens, seq_length=batch_size), dtype=torch.int64)\n",
    "    # Get the NN output\n",
    "    hidden = model.init_hidden(1)\n",
    "    logps, _ = model.forward(text_input, hidden)\n",
    "    # Take the exponent of the NN output to get a range of 0 to 1 for each label.\n",
    "    predval, predclass = torch.exp(logps).topk(1)\n",
    "    print(predval)\n",
    "    \n",
    "    pred = predclass\n",
    "    \n",
    "    return pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.9222]], grad_fn=<TopkBackward>)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[3]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"Google is working on self driving cars, I'm bullish on $goog\"\n",
    "model.eval()\n",
    "model.to(\"cpu\")\n",
    "predict(text, model, vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Questions: What is the prediction of the model? What is the uncertainty of the prediction?\n",
    "** TODO: Answer Question**\n",
    "\n",
    "**The model is predicted to have label 3, or sentiment value of 3. \n",
    "The odds of prediction are termed as logits which go by the equation, p / (1 - p), here p is the prediction probability and (1 - p) is the uncertainty of prediction.\n",
    "Uncertainty of prediction = 1 - 0.9222 = 0.0778**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a trained model and we can make predictions. We can use this model to track the sentiments of various stocks by predicting the sentiments of twits as they are coming in. Now we have a stream of twits. For each of those twits, pull out the stocks mentioned in them and keep track of the sentiments. Remember that in the twits, ticker symbols are encoded with a dollar sign as the first character, all caps, and 2-4 letters, like $AAPL. Ideally, you'd want to track the sentiments of the stocks in your universe and use this as a signal in your larger model(s).\n",
    "\n",
    "## Testing\n",
    "### Load the Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join('..', '..', 'data', 'project_6_stocktwits', 'test_twits.json'), 'r') as f:\n",
    "    test_data = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Twit Stream"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'message_body': '$JWN has moved -1.69% on 10-31. Check out the movement and peers at  https://dividendbot.com?s=JWN',\n",
       " 'timestamp': '2018-11-01T00:00:05Z'}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def twit_stream():\n",
    "    for twit in test_data['data']:\n",
    "        yield twit\n",
    "\n",
    "next(twit_stream())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the `prediction` function, let's apply it to a stream of twits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score_twits(stream, model, vocab, universe):\n",
    "    \"\"\" \n",
    "    Given a stream of twits and a universe of tickers, return sentiment scores for tickers in the universe.\n",
    "    \"\"\"\n",
    "    for twit in stream:\n",
    "\n",
    "        # Get the message text\n",
    "        text = twit['message_body']\n",
    "        symbols = re.findall('\\$[A-Z]{2,4}', text)\n",
    "        score = predict(text, model, vocab)\n",
    "\n",
    "        for symbol in symbols:\n",
    "            if symbol in universe:\n",
    "                yield {'symbol': symbol, 'score': score, 'timestamp': twit['timestamp']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.9985]], grad_fn=<TopkBackward>)\n",
      "tensor([[0.4792]], grad_fn=<TopkBackward>)\n",
      "tensor([[0.7324]], grad_fn=<TopkBackward>)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'symbol': '$AAPL',\n",
       " 'score': tensor([[4]]),\n",
       " 'timestamp': '2018-11-01T00:00:18Z'}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "universe = {'$BBRY', '$AAPL', '$AMZN', '$BABA', '$YHOO', '$LQMT', '$FB', '$GOOG', '$BBBY', '$JNUG', '$SBUX', '$MU'}\n",
    "score_stream = score_twits(twit_stream(), model, vocab, universe)\n",
    "\n",
    "next(score_stream)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it. You have successfully built a model for sentiment analysis! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission\n",
    "Now that you're done with the project, it's time to submit it. Click the submit button in the bottom right. One of our reviewers will give you feedback on your project with a pass or not passed grade. You can continue to the next section while you wait for feedback."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}