8.3 - NaiveBayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos usar o sklearn.naive_bayes para treinar um classificador de spam! A maior parte do código é apenas carregar nossos dados de treinamento em um DataFrame pandas com o qual podemos brincar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import io\n",
    "import numpy\n",
    "from pandas import DataFrame\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "def readFiles(path):\n",
    "    for root, dirnames, filenames in os.walk(path):\n",
    "        for filename in filenames:\n",
    "            path = os.path.join(root, filename)\n",
    "\n",
    "            inBody = False\n",
    "            lines = []\n",
    "            f = io.open(path, 'r', encoding='latin1')\n",
    "            for line in f:\n",
    "                if inBody:\n",
    "                    lines.append(line)\n",
    "                elif line == '\\n':\n",
    "                    inBody = True\n",
    "            f.close()\n",
    "            message = '\\n'.join(lines)\n",
    "            yield path, message\n",
    "\n",
    "\n",
    "def dataFrameFromDirectory(path, classification):\n",
    "    rows = []\n",
    "    index = []\n",
    "    for filename, message in readFiles(path):\n",
    "        rows.append({'message': message, 'class': classification})\n",
    "        index.append(filename)\n",
    "\n",
    "    return DataFrame(rows, index=index)\n",
    "\n",
    "data = DataFrame({'message': [], 'class': []})\n",
    "\n",
    "data = data.append(dataFrameFromDirectory('spam', 'spam'))\n",
    "data = data.append(dataFrameFromDirectory('/ham', 'ham'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos dar uma olhada nesse DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>spam/00228.cf58326ab05a757c7e759acc8d6b360d</th>\n",
       "      <td>spam</td>\n",
       "      <td>REQUEST FOR URGENT BUSINESS ASSISTANCE\\n\\n---...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spam/00400.cc74b7994a7282f32ee2a3b7e3634d31</th>\n",
       "      <td>spam</td>\n",
       "      <td>Our delightful garden ornaments combine the fi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spam/00461.1a27d007492d1c665d07db820b7dc3b8</th>\n",
       "      <td>spam</td>\n",
       "      <td>&lt;html&gt;\\n\\n&lt;head&gt;\\n\\n&lt;title&gt;Tell Me Where to Se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spam/00175.e672ac062c93c15915d4d264c4282b47</th>\n",
       "      <td>spam</td>\n",
       "      <td>&lt;FONT face=3D\"MS Sans Serif\"&gt;\\n\\n&lt;FONT size=3D...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spam/00034.8e582263070076dfe6000411d9b13ce6</th>\n",
       "      <td>spam</td>\n",
       "      <td>As seen on NBC, CBS, CNN, and even Oprah! The ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            class  \\\n",
       "spam/00228.cf58326ab05a757c7e759acc8d6b360d  spam   \n",
       "spam/00400.cc74b7994a7282f32ee2a3b7e3634d31  spam   \n",
       "spam/00461.1a27d007492d1c665d07db820b7dc3b8  spam   \n",
       "spam/00175.e672ac062c93c15915d4d264c4282b47  spam   \n",
       "spam/00034.8e582263070076dfe6000411d9b13ce6  spam   \n",
       "\n",
       "                                                                                       message  \n",
       "spam/00228.cf58326ab05a757c7e759acc8d6b360d   REQUEST FOR URGENT BUSINESS ASSISTANCE\\n\\n---...  \n",
       "spam/00400.cc74b7994a7282f32ee2a3b7e3634d31  Our delightful garden ornaments combine the fi...  \n",
       "spam/00461.1a27d007492d1c665d07db820b7dc3b8  <html>\\n\\n<head>\\n\\n<title>Tell Me Where to Se...  \n",
       "spam/00175.e672ac062c93c15915d4d264c4282b47  <FONT face=3D\"MS Sans Serif\">\\n\\n<FONT size=3D...  \n",
       "spam/00034.8e582263070076dfe6000411d9b13ce6  As seen on NBC, CBS, CNN, and even Oprah! The ...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Agora vamos usar um CountVectorizer para dividir cada mensagem em sua lista de palavras e lançá-la em um classificador MultinomialNB. Execute a função fit() e temos um filtro de spam treinado pronto para começar!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = CountVectorizer()\n",
    "counts = vectorizer.fit_transform(data['message'].values)\n",
    "\n",
    "classifier = MultinomialNB()\n",
    "targets = data['class'].values\n",
    "classifier.fit(counts, targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos experimentar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = ['Free Viagra now!!!', \"Hi Bob, how about a game of golf tomorrow?\"]\n",
    "example_counts = vectorizer.transform(examples)\n",
    "predictions = classifier.predict(example_counts)\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Atividade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nosso conjunto de dados é pequeno, então nosso classificador de spam não é realmente muito bom. Tente executar alguns e-mails de teste diferentes e veja se você obtém os resultados esperados.\n",
    "\n",
    "Se você realmente quiser desafiar a si mesmo, tente aplicar train / test nesse classificador de spam - veja como ele pode prever alguns subconjuntos de e-mails de spam e spam."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}