01_IntroSML_Solution.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Kopie von Kopie von IntroMLSklearn.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyMbrJWhuln/Cj4YZ3Myjd2q",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nicolaiberk/Imbalanced/blob/master/01_IntroSML_Solution.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# An Introduction to Supervised Learning with Scikit learn"
      ],
      "metadata": {
        "id": "_KVhbic470nv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Classifications can take many forms. Today, we will train a simple binary sentiment classifier, using a subset of 10,000 [Amazon Reviews provided as part of a Kaggle competition](https://www.kaggle.com/datasets/bittlingmayer/amazonreviews?resource=download). We will use the amazing `scikit-learn` package to transform the data and train our classifier."
      ],
      "metadata": {
        "id": "xmoWfquYODzt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## A Minimal Example"
      ],
      "metadata": {
        "id": "WCXdH4LiOs36"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We first load some basic packages and the data:"
      ],
      "metadata": {
        "id": "2qmx9Tbptb4I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# fundamental packages\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "# load some data to train our classifier on\n",
        "reviews = pd.read_csv(\"https://www.dropbox.com/s/zup4rcr8j5jz0wr/sub_sample.csv?dl=1\")\n",
        "reviews[\"bin_label\"] = reviews.label == \"good\"\n"
      ],
      "metadata": {
        "id": "Gyx-mllorlZR"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "reviews.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QXFZKxn7ziuB",
        "outputId": "c385cfa9-da15-42eb-b584-30aa5b62af68"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10000, 3)"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "reviews.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "r8N0qUySxsAe",
        "outputId": "59ca5539-3650-4236-9fe8-66261bfa470c"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "  label                                               text  bin_label\n",
              "0  good  District 9 Blu-Ray Edition: This Blu-Ray disc ...       True\n",
              "1   bad  Great for it's age I guess...: This book smell...      False\n",
              "2   bad  OOps !.. too much religious stuff here: I gues...      False\n",
              "3  good  A Great Surprise!: When I first heard about th...       True\n",
              "4   bad  Useless: Worthless for med school admission. H...      False"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-95f12a5e-3e2b-432b-b9cd-8245bfa45d80\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "      <th>bin_label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>good</td>\n",
              "      <td>District 9 Blu-Ray Edition: This Blu-Ray disc ...</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>bad</td>\n",
              "      <td>Great for it's age I guess...: This book smell...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>bad</td>\n",
              "      <td>OOps !.. too much religious stuff here: I gues...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>good</td>\n",
              "      <td>A Great Surprise!: When I first heard about th...</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>bad</td>\n",
              "      <td>Useless: Worthless for med school admission. H...</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-95f12a5e-3e2b-432b-b9cd-8245bfa45d80')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-95f12a5e-3e2b-432b-b9cd-8245bfa45d80 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-95f12a5e-3e2b-432b-b9cd-8245bfa45d80');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The data has a simple structure, with 10,000 observations and two variables/columns, \"label\" and \"text\". The label is either \"good\"or \"bad\". We added a binary version of the label as a third variable. Our task is now to train a classifier that cann tell tehse two labels apart, based on the text of the review. For that, we need some tools!"
      ],
      "metadata": {
        "id": "kJ_bKnN_zOBq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### A Quick Intro to `scikit-learn`"
      ],
      "metadata": {
        "id": "xSMvlWwqkdAL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import IPython\n",
        "url = 'https://scikit-learn.org/stable/'\n",
        "iframe = '<iframe src=' + url + ' width=1600 height=350></iframe>'\n",
        "IPython.display.HTML(iframe)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 430
        },
        "id": "26Z1bjMoiA1G",
        "outputId": "a89bdda3-b659-4998-f068-466386cc658f"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/IPython/core/display.py:724: UserWarning: Consider using IPython.display.IFrame instead\n",
            "  warnings.warn(\"Consider using IPython.display.IFrame instead\")\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<iframe src=https://scikit-learn.org/stable/ width=1600 height=350></iframe>"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "`scikit-learn` is an amazing package, catering to pretty much every need of data scientist. **In order to train a classifier, we need a model that we can train and a vectorizer to transform our data**, that's pretty much it. `scikit-learn` offers much more (please go check it out already!), like a function to transform our data in training and testing data and functions to bind them together and produce our metrics. **We load all of this below**:"
      ],
      "metadata": {
        "id": "MQli_u4Mrizc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# load relevant tools\n",
        "\n",
        "## A model (choose from API)\n",
        "from sklearn.linear_model import LogisticRegression as LogReg\n",
        "\n",
        "## A vectorizer to transform our text into numbers\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "\n",
        "## A function to split our data into train and test set\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "## A pipeline to put it all together, and a few functions to compute how well our classifier performs\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.metrics import accuracy_score, f1_score, recall_score, precision_score"
      ],
      "metadata": {
        "id": "ALUy1tqOk651"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's split our data into train and test set:"
      ],
      "metadata": {
        "id": "2SeFv29i1Qiz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    reviews.text, reviews.bin_label, test_size=0.33, random_state=42)"
      ],
      "metadata": {
        "id": "iKTbFt-41Pjf"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we need literally two lines of code to train the classifier."
      ],
      "metadata": {
        "id": "zoBhnCAo1ZaZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipe = Pipeline([('Tfidf', TfidfVectorizer()), ('LogReg', LogReg())])\n",
        "pipe.fit(X_train, y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "id": "CnR9ykwu1Y1M",
        "outputId": "2fb66dc8-c7f9-4d00-999b-3b5af5509d23"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(steps=[('Tfidf', TfidfVectorizer()), ('LogReg', LogisticRegression())])"
            ],
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;Tfidf&#x27;, TfidfVectorizer()), (&#x27;LogReg&#x27;, LogisticRegression())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;Tfidf&#x27;, TfidfVectorizer()), (&#x27;LogReg&#x27;, LogisticRegression())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TfidfVectorizer</label><div class=\"sk-toggleable__content\"><pre>TfidfVectorizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "![](https://media.giphy.com/media/zXMRfbsHOAire/giphy.gif)\n",
        "\n",
        "Don't believe me? Check for yourself:"
      ],
      "metadata": {
        "id": "LEPJfOxx1g7e"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipe.predict([\"This is a great movie\",\n",
        "              \"Never hated something as much as this movie\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6raSEFyK56Nn",
        "outputId": "4fd892dc-3127-4184-9daf-833b1fd617cb"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ True, False])"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "It does predict our examples well, but how good is the accuracy?"
      ],
      "metadata": {
        "id": "OFBMZjISUc98"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y_pred = pipe.predict(X_test)\n",
        "pd.crosstab(y_test, y_pred)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "gOZzD9jeUhxO",
        "outputId": "357146d9-abe9-4996-8ac3-be39de5fbb8b"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "col_0      False  True \n",
              "bin_label              \n",
              "False       1434    208\n",
              "True         235   1423"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-04849d43-bbc4-4524-b4cc-afc719503dde\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>col_0</th>\n",
              "      <th>False</th>\n",
              "      <th>True</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bin_label</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>False</th>\n",
              "      <td>1434</td>\n",
              "      <td>208</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>True</th>\n",
              "      <td>235</td>\n",
              "      <td>1423</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-04849d43-bbc4-4524-b4cc-afc719503dde')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-04849d43-bbc4-4524-b4cc-afc719503dde button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-04849d43-bbc4-4524-b4cc-afc719503dde');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "## define a custom function to report metrics\n",
        "def accuracy_report(y_test, y_pred):\n",
        "  print(\"Accuracy: \",  round(accuracy_score(y_test, y_pred), 3))\n",
        "  print(\"Recall: \",    round(recall_score(y_test, y_pred), 3))\n",
        "  print(\"Precision: \", round(precision_score(y_test, y_pred), 3))\n",
        "  print(\"F1: \",        round(f1_score(y_test, y_pred), 3))\n",
        "\n",
        "accuracy_report(y_test, y_pred)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WNoZR8vjU6Nj",
        "outputId": "a25fa568-f36c-4f5a-d7b6-33b7f37ae51f"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy:  0.866\n",
            "Recall:  0.858\n",
            "Precision:  0.872\n",
            "F1:  0.865\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pretty good, huh? Let's see how this works in more detail!"
      ],
      "metadata": {
        "id": "FDlTixZSWPxI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Under the Hood"
      ],
      "metadata": {
        "id": "qDlWwbv-k7Yw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's show this based on a very simple example. We generate a set of example texts that are positive or negative reviews and check what the classifier does:"
      ],
      "metadata": {
        "id": "Wnfmt7h_fYyG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "example_revs = [\"This is a great, great movie\",\n",
        "                \"This is a horrible movie\",\n",
        "                \"Waste of time\",\n",
        "                \"Beautiful\"]\n",
        "\n",
        "example_y = [True, False, False, True]"
      ],
      "metadata": {
        "id": "PAMu9ilAfYgO"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Vectorization"
      ],
      "metadata": {
        "id": "BsnEMd83-70P"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We choose a vectorizer for our text [from `scikit-learn`](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.feature_extraction.text) and assign it to an object so we can fit it:"
      ],
      "metadata": {
        "id": "KcqIremTkvAv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "vec  = CountVectorizer()"
      ],
      "metadata": {
        "id": "QmkdS8WtkuPt"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Then, we fit it to our example reviews and transform the text into numbers:"
      ],
      "metadata": {
        "id": "fU1XtT1nlSZq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sparse_mtrx = vec.fit_transform(example_revs)\n",
        "print(vec.get_feature_names_out(), \"\\n\", sparse_mtrx.toarray())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bpBCdkEKfVfw",
        "outputId": "ac1a3579-2640-4d08-ce8a-eef91a32abe3"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['beautiful' 'great' 'horrible' 'is' 'movie' 'of' 'this' 'time' 'waste'] \n",
            " [[0 2 0 1 1 0 1 0 0]\n",
            " [0 0 1 1 1 0 1 0 0]\n",
            " [0 0 0 0 0 1 0 1 1]\n",
            " [1 0 0 0 0 0 0 0 0]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that the vectorizer simply counts the occurence of each word in each text. The vectorizer by default strips all accents and converts all words into lowercase. Now we can use the `transform()` function to transform new texts into the same format. This is particularly important when we need to transform texts in the test set into a matrix based on the training set."
      ],
      "metadata": {
        "id": "KJmaZUMS9r9I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "vec.transform([\"This movie is not good.\"]).toarray()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8-boZz6uA729",
        "outputId": "8c037163-3b88-4764-bd00-4d3e6c8949af"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0, 0, 0, 1, 1, 0, 1, 0, 0]])"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that some features ('not' and 'good') from this new text are not encoded, as the vectorizer does not have an appropriate column in the document-term-matrix."
      ],
      "metadata": {
        "id": "D0Yaj6vwBNkL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Vectorizers have many more features that can be used to preprocess the text. Look up the arguments of the [`CountVectorizer()`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer) and test it yourself:"
      ],
      "metadata": {
        "id": "8evd5J46AXk5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "vec = CountVectorizer(stop_words=[\"this\", \"is\", \"of\"])\n",
        "sparse_mtrx = vec.fit_transform(example_revs)\n",
        "\n",
        "## use the command from above to rpint your transformed matrix\n",
        "print(vec.get_feature_names_out(), \"\\n\", sparse_mtrx.toarray())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Cd-T9_C5-3_3",
        "outputId": "5e9dec4e-3393-4502-bedb-0e5ce5e3232e"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['beautiful' 'great' 'horrible' 'movie' 'time' 'waste'] \n",
            " [[0 2 0 1 0 0]\n",
            " [0 0 1 1 0 0]\n",
            " [0 0 0 0 1 1]\n",
            " [1 0 0 0 0 0]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Fitting the Model"
      ],
      "metadata": {
        "id": "4ikxcQYF_yMR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we know how to convert text into numbers, we can fit a classifier to the data in order to predict observations in the test set (we use our initial data again)."
      ],
      "metadata": {
        "id": "3bJOV4iNARRL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## train-test-split\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "  reviews.text, reviews.bin_label, test_size=0.33, random_state=42)\n",
        "\n",
        "## vectorize data\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "vec = TfidfVectorizer(ngram_range=(1,2))\n",
        "\n",
        "X_train = vec.fit_transform(X_train)\n",
        "\n",
        "## load a classifier of your choosing\n",
        "from sklearn.linear_model import SGDClassifier as SVM\n",
        "clsfr = SVM()\n",
        "\n",
        "## fit\n",
        "clsfr.fit(X_train, y_train)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        },
        "id": "XlJhj4_n_8Zz",
        "outputId": "79710fac-3322-4a2b-b6d5-d6a9442488a9"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "SGDClassifier()"
            ],
            "text/html": [
              "<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SGDClassifier()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" checked><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SGDClassifier</label><div class=\"sk-toggleable__content\"><pre>SGDClassifier()</pre></div></div></div></div></div>"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we can assess performance same as before:"
      ],
      "metadata": {
        "id": "7x1_QmlwRd00"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X_test  = vec.transform(X_test)\n",
        "y_pred = clsfr.predict(X_test)\n",
        "\n",
        "pd.crosstab(y_test, y_pred)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "WCUBhveLRcb8",
        "outputId": "b4cf2bff-ee01-4fd5-f49b-83cb618e4d45"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "col_0      False  True \n",
              "bin_label              \n",
              "False       1472    170\n",
              "True         206   1452"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5bf3052f-1857-44bd-a9c6-419116889223\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>col_0</th>\n",
              "      <th>False</th>\n",
              "      <th>True</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bin_label</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>False</th>\n",
              "      <td>1472</td>\n",
              "      <td>170</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>True</th>\n",
              "      <td>206</td>\n",
              "      <td>1452</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5bf3052f-1857-44bd-a9c6-419116889223')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-5bf3052f-1857-44bd-a9c6-419116889223 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-5bf3052f-1857-44bd-a9c6-419116889223');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "accuracy_report(y_test, y_pred)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X9XTTP1yRoeo",
        "outputId": "f051aec8-39c0-49f0-fe09-c70ddd55c673"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy:  0.886\n",
            "Recall:  0.876\n",
            "Precision:  0.895\n",
            "F1:  0.885\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## BONUS: Improving your Model"
      ],
      "metadata": {
        "id": "A0nLXD2MlRT6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Preprocessing with [`spaCy`](https://spacy.io/)\n",
        "\n",
        "Sometimes, you might want to pre-select features based on your classification problem. For example, when you are interested in the topic of a text, it might be sufficient to assess the nouns which are used, whereas other words might introduce mostly noise. Other tasks might require you to identify the object in a sentence or the organisation mentioned in a text. `spaCy` can identify these words through **parts-of-speech tagging**, **Dependency Parsing**, and **named entity recognition**. However, `spaCy` can do much more. Their [website](https://course.spacy.io/en/) provides an entire course from finding words to training a neural network."
      ],
      "metadata": {
        "id": "vtVbJUoKTmuh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Parts-of-speech tagging"
      ],
      "metadata": {
        "id": "AH91FNjMiVXz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import spacy\n",
        "nlp = spacy.load('en_core_web_sm')\n",
        "\n",
        "\n",
        "# Process a text\n",
        "doc = nlp(\"Not the hero we deserve, but the hero we need.\")\n",
        "\n",
        "# Iterate over the tokens\n",
        "for token in doc:\n",
        "    # Print the text and the predicted part-of-speech tag\n",
        "    print(token.text, token.pos_)\n",
        "\n",
        "# what's PRON? get an explanation:\n",
        "spacy.explain(\"PRON\")"
      ],
      "metadata": {
        "id": "xUfFxHG4VM9G",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 247
        },
        "outputId": "e1fb60cc-c170-4d97-97f9-5b250d5cedcb"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Not PART\n",
            "the DET\n",
            "hero NOUN\n",
            "we PRON\n",
            "deserve VERB\n",
            ", PUNCT\n",
            "but CCONJ\n",
            "the DET\n",
            "hero NOUN\n",
            "we PRON\n",
            "need VERB\n",
            ". PUNCT\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'pronoun'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# retain only the nouns of a set of texts:\n",
        "docs = [\"May the force be with you.\",\n",
        "        \"You're gonna need a bigger boat!\",\n",
        "        \"Fly, you fools!\",\n",
        "        \"And I will strike down upon thee with great vengeance and furious anger!\",\n",
        "        \"You can't handle the truth!\",\n",
        "        \"You take the blue pill, the story ends; you wake up in your bed and believe whatever you want to believe.\",\n",
        "        \"I love the smell of napalm in the morning.\"]\n",
        "\n",
        "\n",
        "for doc in nlp.pipe(docs):\n",
        "  print([token.text for token in doc if token.pos_ == 'NOUN'])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dnm7Acftcc35",
        "outputId": "74cb3953-2864-4895-9b90-965ca81bf13d"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['force']\n",
            "['boat']\n",
            "['fools']\n",
            "['vengeance', 'anger']\n",
            "['truth']\n",
            "['pill', 'story', 'bed']\n",
            "['smell', 'napalm', 'morning']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Another package commonly used in text analysis is [`nltk`](https://www.nltk.org/). It has similar functionalities as `spacy` (e.g. parts-of-speech-tagging) but a slightly different implementation. Below, it is shown how to remove stopwords and stem with `nltk`."
      ],
      "metadata": {
        "id": "l-b3kdUn9NAv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Stopword removal"
      ],
      "metadata": {
        "id": "spp0rtypigha"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Many words that are constantly used in everyday language are usually not very informative about the content of text (see [Pennebaker 2011](http://secretlifeofpronouns.com/) for a contrasting perspective). These words are called 'stopwords' in NLP and usually considered clutter that could and should be removed.\n",
        "\n",
        "Note however that preprocessing can heavily affect model results ([Denny and Spirling 2018](https://www.cambridge.org/core/journals/political-analysis/article/text-preprocessing-for-unsupervised-learning-why-it-matters-when-it-misleads-and-what-to-do-about-it/AA7D4DE0AA6AB208502515AE3EC6989E)). How to preprocess text in general is a decision that should be made based on careful consideration of the problem at hand ([Grimmer and Stewart 2013](https://www.cambridge.org/core/services/aop-cambridge-core/content/view/F7AAC8B2909441603FEB25C156448F20/S1047198700013401a.pdf))."
      ],
      "metadata": {
        "id": "k4OASjJgBvn1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "from nltk.corpus import stopwords\n",
        "print(stopwords.words('english'))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lGn0m_hJiqAI",
        "outputId": "f71045d1-88ef-4ea7-ba55-37fae2f4de21"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.tokenize import word_tokenize\n",
        "nltk.download('punkt')\n",
        "stop_words = set(stopwords.words('english'))\n",
        "\n",
        "filtered_docs = []\n",
        "\n",
        "for doc in docs:\n",
        "  filtered_doc = \" \".join([w for w in word_tokenize(doc) if not w.lower() in stop_words])\n",
        "  filtered_docs.append(filtered_doc)\n",
        "\n",
        "filtered_docs"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ccJ53_LZ-jbh",
        "outputId": "b664d985-eee8-4f04-dd1d-0be1f5089932"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['May force .',\n",
              " \"'re gon na need bigger boat !\",\n",
              " 'Fly , fools !',\n",
              " 'strike upon thee great vengeance furious anger !',\n",
              " \"ca n't handle truth !\",\n",
              " 'take blue pill , story ends ; wake bed believe whatever want believe .',\n",
              " 'love smell napalm morning .']"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Stemming"
      ],
      "metadata": {
        "id": "j5z31OiwicQZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.stem.snowball import SnowballStemmer\n",
        "stemmer = SnowballStemmer('english')\n",
        "\n",
        "stemmed_docs = []\n",
        "\n",
        "for doc in docs:\n",
        "  stemmed_doc = \" \".join([stemmer.stem(w) for w in word_tokenize(doc)])\n",
        "  stemmed_docs.append(stemmed_doc)\n",
        "\n",
        "stemmed_docs"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e6NyTTTnigJp",
        "outputId": "9f8c4d26-f352-4fa6-cfdf-ce0a6549655a"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['may the forc be with you .',\n",
              " 'you re gon na need a bigger boat !',\n",
              " 'fli , you fool !',\n",
              " 'and i will strike down upon thee with great vengeanc and furious anger !',\n",
              " \"you ca n't handl the truth !\",\n",
              " 'you take the blue pill , the stori end ; you wake up in your bed and believ whatev you want to believ .',\n",
              " 'i love the smell of napalm in the morn .']"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Optimizing Model Fit and Avoiding Overfit with CrossValidation"
      ],
      "metadata": {
        "id": "y3_Xrdn8PCqR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Overfitting** describes the problem that we might have a classifier fitting our data a little too well, in that it **does not describe general patterns** anymore, but potentially incorporate some **idiosyncratic noise** in the training data. The visualisation below from the [`scikit-learn` guide on overfitting](https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html?highlight=crossvalidation#underfitting-vs-overfitting) tries to describe this problem.\n",
        "\n",
        "![](https://scikit-learn.org/stable/_images/sphx_glr_plot_underfitting_overfitting_001.png)\n"
      ],
      "metadata": {
        "id": "9eOKnsmJcINb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Often, we might want to test models with different parameters and choose the one that fits best to solve our problem. This will generally increase the fit of our data but also potentially lead to overfitting. A particularly good way to deal with this is called **cross-validation**. In this process, the training data is divided into equally sized subsets and then, several classifiers are trained to predict each subset from the other subsets. This way, the influence of single observations is reduced, because each observation is in the test set once. More on this [here](https://scikit-learn.org/stable/modules/cross_validation.html) and [here](https://cssbook.net/chapter08.html#8_5_3).\n",
        "\n",
        "`scikit-learn` contains many models like `LogisticRegressionCV()`, which implement this by default. However, using `GridSearchCV()`, we can optimise the parameters of any model. In order to optimise the parameters of our model, we first need to check which parameters *can* be tuned:"
      ],
      "metadata": {
        "id": "c3rrA3v9btQA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "?LogReg()"
      ],
      "metadata": {
        "id": "KyMI5UGMZjQA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e19db5a4-2d72-4e5d-ebc5-5bdfe0b485f4"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Object `LogReg()` not found.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In `LogisticRegression()`, we can change the regularization technique with `penalty` (regularisation is a [process](https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c) decreasing the importance of unimportant features, which avoids overfitting when classifying with many features). We can also change the degree of regularization using `C`. "
      ],
      "metadata": {
        "id": "4YXn33ytqYsA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.metrics import classification_report\n",
        "\n",
        "# Set the parameters by cross-validation\n",
        "tuned_parameters = [\n",
        "    {\"penalty\": [\"l1\", \"l2\"], \"C\": [1, 10, 100, 1000]}\n",
        "]\n",
        "\n",
        "score = \"f1\"\n",
        "\n",
        "clf = GridSearchCV(LogReg(solver = 'liblinear'), tuned_parameters, scoring=\"%s_macro\" % score)\n",
        "clf.fit(X_train, y_train)\n",
        "\n",
        "print(\"Best parameters set found on development set:\")\n",
        "print(clf.best_params_)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hli4P-0Wrmm7",
        "outputId": "7d08c5b2-2f03-4767-c230-f80a4062e2bc"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Best parameters set found on development set:\n",
            "{'C': 1000, 'penalty': 'l2'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see if a classifier with these parameters outperforms the standard model:"
      ],
      "metadata": {
        "id": "op8Cd0gUr5kf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "clsfr_std = LogReg(solver = 'liblinear')\n",
        "clsfr_new = LogReg(solver = 'liblinear', C = 1000, penalty = \"l2\")\n",
        "\n",
        "clsfr_std.fit(X_train, y_train)\n",
        "clsfr_new.fit(X_train, y_train)\n",
        "\n",
        "y_pred_std = clsfr_std.predict(X_test)\n",
        "y_pred_new = clsfr_new.predict(X_test)\n",
        "\n",
        "accuracy_report(y_test, y_pred_std)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XtvGLq2ntKDp",
        "outputId": "cac34029-5dca-4e8b-80e6-52d53cf996e1"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy:  0.872\n",
            "Recall:  0.849\n",
            "Precision:  0.89\n",
            "F1:  0.869\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "accuracy_report(y_test, y_pred_new)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WYeA1fwZtyYY",
        "outputId": "40f53cb6-bdcc-42fd-d938-40b16c8ec302"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy:  0.886\n",
            "Recall:  0.873\n",
            "Precision:  0.898\n",
            "F1:  0.885\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We managed to improve our classifier!\n",
        "\n",
        "![](https://media.giphy.com/media/a0h7sAqON67nO/giphy.gif)"
      ],
      "metadata": {
        "id": "f3M7vdfKt8hr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Feature Assessment with [`eli5`](https://eli5.readthedocs.io/en/latest/index.html)"
      ],
      "metadata": {
        "id": "uKBWEKavPHn9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "`eli5` is a great package to understand how our classifier makes decisions. It has two main functions: `show_weights()` tells us which features are most predictive for the classification, and `show_prediction()` explains us how each feature affects the prediction for a single example. This can be particularly useful for iterative feature selection and the exclusion of stopwords, etc."
      ],
      "metadata": {
        "id": "MDP6Q1o2u19M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install eli5\n",
        "import eli5\n",
        "eli5.show_weights(clsfr, vec=vec, feature_names=vec.get_feature_names_out())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 878
        },
        "id": "eA0ugrF1Zeaw",
        "outputId": "d871a33a-d065-4d4e-fce5-f03d85663376"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting eli5\n",
            "  Downloading eli5-0.13.0.tar.gz (216 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m216.2/216.2 kB\u001b[0m \u001b[31m5.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: attrs>17.1.0 in /usr/local/lib/python3.10/dist-packages (from eli5) (23.1.0)\n",
            "Requirement already satisfied: jinja2>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from eli5) (3.1.2)\n",
            "Requirement already satisfied: numpy>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from eli5) (1.22.4)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from eli5) (1.10.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from eli5) (1.16.0)\n",
            "Requirement already satisfied: scikit-learn>=0.20 in /usr/local/lib/python3.10/dist-packages (from eli5) (1.2.2)\n",
            "Requirement already satisfied: graphviz in /usr/local/lib/python3.10/dist-packages (from eli5) (0.20.1)\n",
            "Requirement already satisfied: tabulate>=0.7.7 in /usr/local/lib/python3.10/dist-packages (from eli5) (0.8.10)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2>=3.0.0->eli5) (2.1.2)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.20->eli5) (1.2.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.20->eli5) (3.1.0)\n",
            "Building wheels for collected packages: eli5\n",
            "  Building wheel for eli5 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for eli5: filename=eli5-0.13.0-py2.py3-none-any.whl size=107747 sha256=d1ac81b4da6efcdf709486e3b2481be83847c723a5d864a192b0daee31021b75\n",
            "  Stored in directory: /root/.cache/pip/wheels/b8/58/ef/2cf4c306898c2338d51540e0922c8e0d6028e07007085c0004\n",
            "Successfully built eli5\n",
            "Installing collected packages: eli5\n",
            "Successfully installed eli5-0.13.0\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <style>\n",
              "    table.eli5-weights tr:hover {\n",
              "        filter: brightness(85%);\n",
              "    }\n",
              "</style>\n",
              "\n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "        \n",
              "\n",
              "    \n",
              "\n",
              "        \n",
              "            \n",
              "                \n",
              "                \n",
              "    \n",
              "        <p style=\"margin-bottom: 0.5em; margin-top: 0em\">\n",
              "            <b>\n",
              "    \n",
              "        y=True\n",
              "    \n",
              "</b>\n",
              "\n",
              "top features\n",
              "        </p>\n",
              "    \n",
              "    <table class=\"eli5-weights\"\n",
              "           style=\"border-collapse: collapse; border: none; margin-top: 0em; table-layout: auto; margin-bottom: 2em;\">\n",
              "        <thead>\n",
              "        <tr style=\"border: none;\">\n",
              "            \n",
              "                <th style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\" title=\"Feature weights. Note that weights do not account for feature value scales, so if feature values have different scales, features with highest weights might not be the most important.\">\n",
              "                    Weight<sup>?</sup>\n",
              "                </th>\n",
              "            \n",
              "            <th style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">Feature</th>\n",
              "            \n",
              "        </tr>\n",
              "        </thead>\n",
              "        <tbody>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 80.00%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +5.940\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        great\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 86.18%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +3.502\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        excellent\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 86.32%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +3.454\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        best\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 87.65%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +2.985\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        love\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 88.88%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +2.568\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        perfect\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 88.93%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +2.553\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        good\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 89.04%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +2.516\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        wonderful\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 89.44%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +2.387\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        amazing\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 89.56%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +2.347\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        fun\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 89.56%); border: none;\">\n",
              "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
              "                    <i>&hellip; 111770 more positive &hellip;</i>\n",
              "                </td>\n",
              "            </tr>\n",
              "        \n",
              "\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 89.57%); border: none;\">\n",
              "                <td colspan=\"2\" style=\"padding: 0 0.5em 0 0.5em; text-align: center; border: none; white-space: nowrap;\">\n",
              "                    <i>&hellip; 111292 more negative &hellip;</i>\n",
              "                </td>\n",
              "            </tr>\n",
              "        \n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 89.57%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -2.345\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        money\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 89.52%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -2.358\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        waste\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 89.32%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -2.425\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        disappointment\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 88.92%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -2.556\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        disappointing\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 88.60%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -2.661\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        terrible\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 88.28%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -2.770\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        worst\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 87.87%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -2.908\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        bad\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 87.48%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -3.042\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        disappointed\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 86.45%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -3.408\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        poor\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 85.84%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -3.627\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        boring\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 82.00%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -5.111\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        not\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "\n",
              "        </tbody>\n",
              "    </table>\n",
              "\n",
              "            \n",
              "        \n",
              "\n",
              "        \n",
              "\n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "\n",
              "\n"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "eli5.show_prediction(clsfr, reviews.text[400], vec=vec, feature_names=vec.get_feature_names_out())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "zXZQeOJHvmOV",
        "outputId": "ecaa60b9-f8b9-4f6c-abd9-3ce5c8d6eae0"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <style>\n",
              "    table.eli5-weights tr:hover {\n",
              "        filter: brightness(85%);\n",
              "    }\n",
              "</style>\n",
              "\n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "        \n",
              "\n",
              "    \n",
              "\n",
              "        \n",
              "\n",
              "        \n",
              "    \n",
              "        \n",
              "        \n",
              "    \n",
              "        <p style=\"margin-bottom: 0.5em; margin-top: 0em\">\n",
              "            <b>\n",
              "    \n",
              "        y=False\n",
              "    \n",
              "</b>\n",
              "\n",
              "    \n",
              "    (score <b>-1.320</b>)\n",
              "\n",
              "top features\n",
              "        </p>\n",
              "    \n",
              "    <table class=\"eli5-weights\"\n",
              "           style=\"border-collapse: collapse; border: none; margin-top: 0em; table-layout: auto; margin-bottom: 2em;\">\n",
              "        <thead>\n",
              "        <tr style=\"border: none;\">\n",
              "            \n",
              "                <th style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\" title=\"Feature contribution already accounts for the feature value (for linear models, contribution = weight * feature value), and the sum of feature contributions is equal to the score or, for some classifiers, to the probability. Feature values are shown if &quot;show_feature_values&quot; is True.\">\n",
              "                    Contribution<sup>?</sup>\n",
              "                </th>\n",
              "            \n",
              "            <th style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">Feature</th>\n",
              "            \n",
              "        </tr>\n",
              "        </thead>\n",
              "        <tbody>\n",
              "        \n",
              "            <tr style=\"background-color: hsl(120, 100.00%, 80.00%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        +1.398\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        Highlighted in text (sum)\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "        \n",
              "\n",
              "        \n",
              "        \n",
              "            <tr style=\"background-color: hsl(0, 100.00%, 97.36%); border: none;\">\n",
              "    <td style=\"padding: 0 1em 0 0.5em; text-align: right; border: none;\">\n",
              "        -0.078\n",
              "    </td>\n",
              "    <td style=\"padding: 0 0.5em 0 0.5em; text-align: left; border: none;\">\n",
              "        &lt;BIAS&gt;\n",
              "    </td>\n",
              "    \n",
              "</tr>\n",
              "        \n",
              "\n",
              "        </tbody>\n",
              "    </table>\n",
              "\n",
              "    \n",
              "\n",
              "\n",
              "\n",
              "    <p style=\"margin-bottom: 2.5em; margin-top:-0.5em;\">\n",
              "        <span style=\"background-color: hsl(120, 100.00%, 83.70%); opacity: 0.86\" title=\"0.047\">caveat</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 81.52%); opacity: 0.87\" title=\"0.056\">emptor</span><span style=\"opacity: 0.80\">...: i </span><span style=\"background-color: hsl(120, 100.00%, 92.96%); opacity: 0.82\" title=\"0.014\">was</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 82.65%); opacity: 0.86\" title=\"-0.051\">also</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 95.40%); opacity: 0.81\" title=\"0.008\">looking</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 96.35%); opacity: 0.81\" title=\"0.005\">forward</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 95.91%); opacity: 0.81\" title=\"-0.006\">to</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 97.35%); opacity: 0.80\" title=\"-0.003\">seeing</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 91.86%); opacity: 0.82\" title=\"0.017\">this</span><span style=\"opacity: 0.80\"> (</span><span style=\"background-color: hsl(120, 100.00%, 84.79%); opacity: 0.85\" title=\"0.042\">it</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 69.99%); opacity: 0.93\" title=\"0.111\">would</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 82.22%); opacity: 0.86\" title=\"0.053\">have</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 85.69%); opacity: 0.85\" title=\"0.039\">been</span><span style=\"opacity: 0.80\"> a </span><span style=\"background-color: hsl(0, 100.00%, 60.00%); opacity: 1.00\" title=\"-0.168\">great</span><span style=\"opacity: 0.80\"> souvenir </span><span style=\"background-color: hsl(120, 100.00%, 87.53%); opacity: 0.84\" title=\"0.032\">of</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 80.94%); opacity: 0.87\" title=\"0.058\">that</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 87.65%); opacity: 0.84\" title=\"0.031\">tour</span><span style=\"opacity: 0.80\">), </span><span style=\"background-color: hsl(120, 100.00%, 88.90%); opacity: 0.83\" title=\"0.027\">but</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 96.06%); opacity: 0.81\" title=\"0.006\">let</span><span style=\"opacity: 0.80\">&#x27;s </span><span style=\"background-color: hsl(120, 100.00%, 87.98%); opacity: 0.84\" title=\"0.030\">be</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 81.12%); opacity: 0.87\" title=\"-0.057\">honest</span><span style=\"opacity: 0.80\">...</span><span style=\"background-color: hsl(0, 100.00%, 88.19%); opacity: 0.84\" title=\"-0.029\">the</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 87.00%); opacity: 0.84\" title=\"-0.034\">sound</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 90.46%); opacity: 0.83\" title=\"-0.022\">quality</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 68.58%); opacity: 0.94\" title=\"0.119\">sucks</span><span style=\"opacity: 0.80\">...</span><span style=\"background-color: hsl(120, 100.00%, 94.64%); opacity: 0.81\" title=\"0.009\">it</span><span style=\"opacity: 0.80\">&#x27;s </span><span style=\"background-color: hsl(120, 100.00%, 71.45%); opacity: 0.92\" title=\"0.104\">not</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 89.76%); opacity: 0.83\" title=\"0.024\">much</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 94.94%); opacity: 0.81\" title=\"0.009\">better</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 82.95%); opacity: 0.86\" title=\"-0.050\">than</span><span style=\"opacity: 0.80\"> a </span><span style=\"background-color: hsl(120, 100.00%, 96.07%); opacity: 0.81\" title=\"0.006\">bootleg</span><span style=\"opacity: 0.80\">...</span><span style=\"background-color: hsl(0, 100.00%, 77.70%); opacity: 0.89\" title=\"-0.073\">in</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 79.86%); opacity: 0.88\" title=\"-0.063\">fact</span><span style=\"opacity: 0.80\">, i&#x27;</span><span style=\"background-color: hsl(120, 100.00%, 91.47%); opacity: 0.82\" title=\"0.018\">ve</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 82.85%); opacity: 0.86\" title=\"0.050\">heard</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 91.28%); opacity: 0.82\" title=\"0.019\">boots</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 86.51%); opacity: 0.84\" title=\"0.036\">from</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 83.71%); opacity: 0.86\" title=\"0.047\">this</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 82.34%); opacity: 0.86\" title=\"0.052\">tour</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 96.01%); opacity: 0.81\" title=\"0.006\">that</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 92.00%); opacity: 0.82\" title=\"-0.017\">sound</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 94.58%); opacity: 0.81\" title=\"0.010\">way</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 89.64%); opacity: 0.83\" title=\"-0.024\">better</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 88.91%); opacity: 0.83\" title=\"0.027\">than</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 86.91%); opacity: 0.84\" title=\"0.034\">this</span><span style=\"opacity: 0.80\">. </span><span style=\"background-color: hsl(0, 100.00%, 90.61%); opacity: 0.83\" title=\"-0.021\">the</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 86.37%); opacity: 0.84\" title=\"-0.036\">video</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 91.66%); opacity: 0.82\" title=\"-0.018\">quality</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 93.14%); opacity: 0.82\" title=\"-0.014\">is</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 97.64%); opacity: 0.80\" title=\"0.003\">decent</span><span style=\"opacity: 0.80\">, </span><span style=\"background-color: hsl(120, 100.00%, 66.29%); opacity: 0.96\" title=\"0.132\">not</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 89.97%); opacity: 0.83\" title=\"-0.023\">spectacular</span><span style=\"opacity: 0.80\">. </span><span style=\"background-color: hsl(120, 100.00%, 85.80%); opacity: 0.85\" title=\"0.038\">furthermore</span><span style=\"opacity: 0.80\">, </span><span style=\"background-color: hsl(120, 100.00%, 84.91%); opacity: 0.85\" title=\"0.042\">the</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 89.08%); opacity: 0.83\" title=\"0.026\">whole</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 74.48%); opacity: 0.91\" title=\"0.088\">thing</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 77.26%); opacity: 0.89\" title=\"0.075\">is</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 62.03%); opacity: 0.99\" title=\"0.156\">too</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 83.16%); opacity: 0.86\" title=\"0.049\">short</span><span style=\"opacity: 0.80\">...</span><span style=\"background-color: hsl(0, 100.00%, 74.75%); opacity: 0.90\" title=\"-0.087\">some</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 78.65%); opacity: 0.88\" title=\"-0.068\">of</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 90.34%); opacity: 0.83\" title=\"-0.022\">the</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 78.74%); opacity: 0.88\" title=\"-0.068\">show</span><span style=\"opacity: 0.80\">&#x27;s </span><span style=\"background-color: hsl(120, 100.00%, 97.39%); opacity: 0.80\" title=\"0.003\">highlights</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 92.22%); opacity: 0.82\" title=\"0.016\">such</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 87.23%); opacity: 0.84\" title=\"-0.033\">as</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 81.14%); opacity: 0.87\" title=\"-0.057\">the</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 86.22%); opacity: 0.84\" title=\"-0.037\">time</span><span style=\"opacity: 0.80\"> &amp; a </span><span style=\"background-color: hsl(120, 100.00%, 80.48%); opacity: 0.87\" title=\"0.060\">word</span><span style=\"opacity: 0.80\"> -&gt; -&gt; </span><span style=\"background-color: hsl(120, 100.00%, 97.17%); opacity: 0.80\" title=\"0.004\">soon</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 94.72%); opacity: 0.81\" title=\"-0.009\">medley</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 74.64%); opacity: 0.90\" title=\"0.088\">should</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 75.96%); opacity: 0.90\" title=\"0.081\">have</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 91.08%); opacity: 0.82\" title=\"0.020\">been</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 96.12%); opacity: 0.81\" title=\"-0.006\">included</span><span style=\"opacity: 0.80\">. i </span><span style=\"background-color: hsl(120, 100.00%, 77.17%); opacity: 0.89\" title=\"0.075\">would</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 88.06%); opacity: 0.84\" title=\"0.030\">have</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 91.52%); opacity: 0.82\" title=\"-0.018\">thought</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 93.96%); opacity: 0.81\" title=\"0.011\">something</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 93.27%); opacity: 0.82\" title=\"0.013\">released</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 84.85%); opacity: 0.85\" title=\"0.042\">under</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 90.92%); opacity: 0.82\" title=\"0.020\">the</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 88.47%); opacity: 0.83\" title=\"-0.028\">yes</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 86.89%); opacity: 0.84\" title=\"0.034\">name</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 79.01%); opacity: 0.88\" title=\"0.067\">could</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 71.78%); opacity: 0.92\" title=\"0.102\">have</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 84.00%); opacity: 0.85\" title=\"0.045\">had</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 77.85%); opacity: 0.89\" title=\"0.072\">better</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 87.80%); opacity: 0.84\" title=\"0.031\">sound</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 74.76%); opacity: 0.90\" title=\"0.087\">than</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 79.87%); opacity: 0.87\" title=\"0.063\">this</span><span style=\"opacity: 0.80\">, </span><span style=\"background-color: hsl(120, 100.00%, 92.26%); opacity: 0.82\" title=\"0.016\">it</span><span style=\"opacity: 0.80\">&#x27;s </span><span style=\"background-color: hsl(120, 100.00%, 65.05%); opacity: 0.96\" title=\"0.139\">not</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 94.47%); opacity: 0.81\" title=\"-0.010\">like</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 94.61%); opacity: 0.81\" title=\"-0.010\">1979</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 86.04%); opacity: 0.84\" title=\"0.037\">was</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(120, 100.00%, 93.70%); opacity: 0.81\" title=\"0.012\">the</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 94.82%); opacity: 0.81\" title=\"-0.009\">sonic</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 82.98%); opacity: 0.86\" title=\"-0.050\">dark</span><span style=\"opacity: 0.80\"> </span><span style=\"background-color: hsl(0, 100.00%, 80.45%); opacity: 0.87\" title=\"-0.060\">ages</span>\n",
              "    </p>\n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "    \n",
              "\n",
              "\n",
              "\n"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    }
  ]
}