TFIDF_Intro.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TFIDF_Intro.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tawfiqam/MI564/blob/main/TFIDF_Intro.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qZEBn77ibrvH"
      },
      "source": [
        "#Revisiting Bag of Words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V4tpaRancBV2"
      },
      "source": [
        "Let's revisit bag of words (BoA) from [the Naive Bayes classifier example](https://github.com/tawfiqam/MI564/blob/main/Naive_Bayes_Intro.ipynb).\n",
        "\n",
        "\n",
        "The documents:\n",
        "\n",
        "\n",
        "```\n",
        "[Today is a great day,Today is a sunny day]\n",
        "```\n",
        "\n",
        "\n",
        "BoW = {\"Today\":2,\"great\":1,\"day\":2,\"sunny\":1}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WPeddAemcb9p"
      },
      "source": [
        "![1-Bag-of-words.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAu4AAAHCCAMAAACHRbwoAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAD8UExURUyP/8f2vozE3HxFLwAAAP3Ep////68/ciXTZi97yQJNqPajygsPFfW5oBQVFeLi4kmK9/r8/1qW/AMLCkKf202F7S9ZoP+8pB45ZQ4aLRQlQSlLhUKA5Tixsi7CjBgvU9bm/MTa+tPR0+ry/SxSkzRhrjZpvHdnrnGd7iJCc464/wYmEqLF/zp93hBhLmZ40RV8OyLFXzx20wtCH4G143So/xZhujxvyD2EzcWty6mp0pZTlXtKOrfR822i2VKQ05Ch4Na4uaPE5u2+rqndzhqXSR6uVJrR1bjqxeOfktOHhMVjgCs0dbN2Y21bbpJXR+zOwJ9AXMCLpcfEL5wWkzAAACAASURBVHja7J1Rb6M6E4YjNd0rV3IXgbBBokcFKqqiSkgV30VWe70rbfv//82XpAnY2GMbAiEh855ztQlpMI/Hr8djZ7VCoVAoFAqFQqFQKBQKhUKhUCgUCoVCoVAoFAqFQqFQKBQKhUKhUCgUCoVCoVAoFAqFQqFQKBQKhUKhUCgUCoVCoVAoFAqFQqFQKBQKhUKhUCgUCoVCoVAoFAqFQi1SNK5ibAXULaAeJGS9Ewszegs3nPGDcnz4gqpAo7xaGBLpN+vfcr23vGmP0tRufa88j6Lj3QbIuAjCWisvWVBU8BPp1lwvK5pLCvXF9iNr9cW2c1WIew89/lB0Hty34tVCaKdcui/mel1uuIR6zYuRGvmb18j8YeyacH9SdTbc195CxsGw042du0mLdAx3BU1faNs0nO2m0cz0xX29ThcxO+nclDuCrWHJYKOj6QvtdQHifkW4q4/5CiUYd48Qnfmwt00Iu3MNUO2o4CPu14S751/9DcbNzSS7yQjtkXsvQcMijRhJ58W6v29C7z77VBXKSVyZAohKBzHIsERSTKDQi9F8t42ZmSG4dx/lFU9U61MmuR1ouMny8fUJf/Kmcf9QNRnuYRSlh/9Dshz3zvum2wVlgGGJTZPfNqFDVoj7xXp3EWyazJ9KG0nsBPYokEAPZNzlF/P1Cfbptr37XLivKJt/tjU/7oIvqYBkj2paivUlkHaV0f1N1Xlwb/6dXTvu5BTcU+0ShLD+pJmScjghj7hfzlQ109pWI+5VUIRhEWWu6Uo/i8IwjNJ6lAlwnRa7T7P89ZNwr7TGJO9O6JnWAJkjRRns26JHEdm29YoCvoDu22P7NEon3P1sO00LLqtQ5PXt7bXz38VE97hoJ7Q82AIcH0pOobXYPBEq0HbtHB0uGJLcr0JP+Ovw+/LmbceK2F6Tb6IrLAuVBFasm94akrhx1LYdi6D7rw7feD9/qhPjDFhsD7L9RAvuGW/ee1N7ACDcmWWqSiN5RGd1M4HTX1J2DG9Cy+Frj3GHOJbZxvSBlRGhzp83qBINU611B2tKu21HUv1ol7ddjRZrU92ZX3Tyx5kR95qJ701vHncaWsbCkikZ+iZ4a3HPPMUB8MG4B55av6n7EJ+fWgiUafx51YKisTrMumRRE+U7scqE+7qiibGkM1c/kRgeYHetJaE3iHvRbu+ImmdGqOsTW3ueAXfjalZf3AvdhzDV01Jyct0b1RQENIEziVW0Y2saUu34+2Bhwj0oTEULQp/USMU9UN6T3CDujo21j2+e6RoN7sF6RNwT/aeQEmZ1eJknV4lm7XjIlKExsKUhU/dqvAZ3ZmzfrN8TrDRvihB3uNv7bN0Pd3P36Il7BH0M8yfAPVKseCzsAoyUew4tacgaLNeoYNyNbJZeP9x1Bs+LEfftAE77hFcYd3P36Id77t45x8C9UhItgWBvamX6SMxpyJiA3574jrcqA0xZv/G5culCt4l7bg1QLMryLE3MuGfdCyI2GHfRj5PtZwUJbAjGwL3ll3c7eyq+eojNpSUNKRbfh0EeCOlD9YoO7h4vorRISA76RL5vXG7EvZkGkIz6GVvezuQTzExqtLNrcmzLODHhztT9gBkZiHv7fb1DRKraj2ed0ajcqllmKg/qmYYIOycYtEuqsfhq1IUvN48V6/D7rsVt46UJd6bPVgrdn9VNvt6AeyJlVinfsj75UST++/vLy0v+/u5fNu66+VYb3FmsddQh+NRI609jNgj39um2uQxh63UAR+eh5YlZZ+zIJbeSd7xLYk5Dco17KEAzJuIeAkgGunSimLcMgNhzaI94etY/N3eNNp/vl4y7pmS7CWgk1g/TIZg3rPWWqAfu2sSHbzLMp+Lud8yGHM/lWC+YG+0kP9a2EYcaInfw1lw7sgmGHsL9TBsL6eeW8TtRm/cLxp1R0MxmwDgdQg1cAC6hR8Mn2n1xmSEdciruAlDy7VfyFwrkRkiNjS1tiowhNHP7LkAKnGgD50Ob2+HnOPLpt4z6tz7pBeCeRI0KDtv3GIqkHMIdYLEegrunt8YMdjMn4x5J37/q5GLkvYEp6MPlvgF0/BDCvbLmqTqjSTvqgFPVLfBhOvF+q5c7rTa/58ddCtY16A9yKJFQAA8N7B9ef9xj4CiwAs6HnIx7LbVQN9PuS2Y9MachiR7fHLgqt9dWptYgHpgT/1NOVOn73d2F8Q6ViNWQew+g1EMA4F5BdnaAi4SCWQ5b5pNxbyENxS+dd7nKxR6sL5MDOisF/j23L/IXkIsrwPFOWTUhU9WIaZ3MvLxDuLetEgEXVBCKrrgPKBHLbH+DT4C7WHmrVsmkwleqzVt8KRStmw7l1qI6H0QhC6bg7msq1KZZVP28g7Whl4U7dGBFb9xLCEVyAu4FYHKmwD0QzLvaKqXQGSLzKcM+9C2ZBffCjvvKGXcpB6xmiCc37sf56mXhngPPpreZoUAqOl6fYGZcR5AxcBczJ4lKEWv7P7dkUmzR3esd3QeYGbG825CAOz3ffmfWy2XiDs2fIiiehG6Zy3QA7tBIkU04VRWATqhmet2AFbWvpuZZQKfj+7amDu2PL3Odqn6Hh5BMfhxo18psvi7AzoC4RwBZoDcBE5GJfiWEnJKI7ABTwE9tBNyFMhPNzTeGnVkTh1w//c9sE/Cw93AHJyKbrx0lbMgx4AOD++bv/f3Dv83c4R3EnUH+gOjdTAkuMwXaV8JBq6r6oEUJzNkIuAuHPuq6VZu5sZ2nBAyAHOisDt7dBxY1MqfTP/zIm+zYBNm5f/253+nhn/iP/uXgXoDr16G+vhwuIvA9tfJDso/+gK8rlcsWExYRSCsE2lWk0GWDizzFkLplDiHnEN2B6YKhiACato694CQF8q+H+4P+bGYN78BeVWHtDczASOXwhhIxcSGPpLtH6ku/ldQH97brCBVRmam0ZAzcE9NJG5rdRJkVTiH1VxLIHrrgrj2U21Qi9v0GX3k2I5cA/xZp/3ffSgjwmzlxZ7wR80ymjqklp+YCYF+KjoSRE7Z3tN2Kx8ot6HbWjoF7YNwNoZyyBN+QpjhU2PZbD8BdmAI1RTAVM9e00pQcI5VPrHUKp09U/95Lanl/nxF35z2UYjTjQV1W0iYF3cNJx9vNJCySeEVNV3HAzZFsDNxjc51oN/hzB++xbabcX/nC2TtqEtXBu0vDqpdkVVl3NtsobUL3Z9ywfTQvuXmlYAzcD7Zdx/vnxeHOzc/McWu2ebsfPSnSWrILY+De3X1IzF/JsBPOsGvXi1cDojtw3AKIezs7JYyzyQ4joBrb3uprNjdjwd3TjXEx6Yu7eTN3v28cgps948lwL4x32A3+Jl+Q9dgp6YQ7vNlbizvUO0a27k1x2EZD+/3DZi43Y8E9t1hQR9ylydORPzIMd8qdd6KMhntubhVuiv3mnmNa53HD3fIIA6fRceRzzekL4NuP+Zm5cjPGtvKgFIPuaCDjsUrKQXHrpIn4fX9ogGqtEQEyaaPgLs9GlY15UR90IuddwU7efWU4ikTr3UP3IydOt+5f93p9zWTeTbgzeFTWHCOWJebHLR0SuZspDQdREyHBor5RcJcnHwmcT7dnurWxguTGMcUWe9WI7RlOPSymp73Juv8BcH+YybzDuBPjY/NDZSBIbQ8nTvkOPsKjSoyYA9ava94t2QYf2Di4p2aeWa9EU/dA13Xhr07CXTlvdhuoGNz5OkdKTnEmqiW4N+HdvwjcPRZaN7lU0pkpcRvjzA+HKg9zUFIg4+J5M4ZWGwf30sxz0bP6RDye2gtL24zBwVnnQnvsz/RODWMNDZjQeBMUu//WrC9p3fu85xL0k58VCedJOOznCUJHZ2r969XqCkXzKNkd3R7lYxmJONi3R+H24xLbd4dJUkzUeO/GiaoQ3l9WN6LK2euirk4vFuverDUtE/dYXdtjk9XioS4Gd5j2+78z7mmaHPd1IY/Zwg8NMKRjaaIOuD8sObrLUyIqHhGZIh6L0zHt/mDgfbNo3HdxvAjyus4DaecYoUjHYnE3ePfvueqScXesuERdve7smZnvuernjeEeIRvLk39nzbsfcL+x6F4gGwvUO+I+xi9poK4rMWMoIli0maGRrsyaV4jGomeqxkzkgnHf7Y/slqnzHLlYPO4PtmXV5TZCHIScePsCNO5Y2YG6ctz/2Ipm3hfeFJRS5OFmcP9nWWZarJtBIe6SZjtrBoWaCPcvS83MXKnI5VoMNE9nV3s+5MZSETlXeI/WPC2X1/BlynHh9tz6Wd/ZM5HNWTNzhPd4v5mMLWynxWGPWokEnlkbe2pmzpMijzvtlXWfj1eNPn726Omvv56eH388Pj/99/ZhfGf7pz50H3PUzx5Xlp7tEDvUxG7mn3mv6jzhvYbqtf77odXzrw+nz319ki97M3SUj+ZtT5oPal58Wxm+5Jti0aY4Igtlk7+xmXfxh+PP/vU49AM9AO4/Hh9/2UP867N6HXzZz8fmXep7nkx94Rm+kOGGqYvMzfwVz8M+91JTDpaeQ7jvQrUlwH886TvKG3RB+/5XQ1d4/AkPC8897gx1ptyMzs782cz5I3wcjIEG3B8fjby/PULXPQEB/q15yy/Yy2j6wit83fHW0L2fWf9rcd98aY5E/T9759qbtrKFYYdukg8OR2EiStJdqbbcFAf5CKkCbdmEbihICSAUpf//xxxjLp7rmrHxmJ50vemXJsEY5fHyu965+Jy4f1c7XAD3tJgCfgZ6oeK+AFTpCfVqX/1WfWVX8g8SWKsGFO5Pr12+tj+d8xGrd2qDC+Iu89FGr1PwrvbgAX1T4V/WVvucY3m/QwLPh/uHVqvL0n5O3PebNX4sju3lqBTtqvvCRFWmR9CbjuCrb7eh+i1OwDyXd0+1bG3V3TLfbbVW58X9I7AjgIZbT+fCC/HeVxkW9niR6ofyJvgatyo7ZzLzlH4tu61l+rX/t+Jor3nazBdg1SgVaefyochQKMaXgZ++ZsLlND4YRQZKLyP+0NPcax6xWa1fT2x176ZF/ci7gHut1b0DNXO+FGziAc0hR6d/gJAwWU27D0aRDLmEuzUQuXUPwEYct+arUdRjJrfOJcW9m1d3vlN9qhX3B2gkRo57XocjjZUJaKqJr67R3Cv7kDeK5LcSX/EB79HNnM/LbM3MU2rbU20tTUb9B6661zrOdAftgKHA/fj9CWRJUm9P1FdCBLkgX170JR1DBN4vtvrbxqPlUGZeZhdE7pSV9u6y9crjXmd5v4emlahwn6gragR1sn24vB9tUFtx/YiBowf3Ec4her9GCM8QQ2a4r7oHLbMSv+Jxr7FZ/Sx/upZhdY8g5y6LXyaX0KCQLzPvfSHX6ct8vaf8iLc2HguNMvIy21HVVpdSZt553OubJfYABhc6796H6vcI7mM9KIqMJOcQSKxOXxND5tETmvfawg+GZKq47z1N95XH/cO/dZ3b3+Dmdbpkhqhf0ZaX2/4l4D6I7MVHyvuSgdWJdshL9xFRdov7pitILO+1Natw6aPmo+SKAiAMCXT8BZCb8UTzPsrNfiAe2tPEkPobGMpio/pBRnu3uzkb7vegsYVHVSVzIsmljr8ImsAYiRdLlP/6RDAuRBtDHtsT7FVr0r9a2kXea8N9NwhzUwr3CPIqE8UhwTkuI/HYVEkfCZdS/xIc8GJ61RsksRb1KIxXXYVY+/40GAw6nRr+QJrK5xek3SAGz/mV1f82fy0Q2t60edvva2PI/A6GK7Rrt+4bFe2tNYP761WmX4ntc/sOr27zi07zMmgd86QcesNDuB7RbsXnr7RAH0M6h0nAOOe9btyf1ks57curNwb3He3rZnNG7J7bN3jI0S9o3Sncif6YBDJDI+7i6NM/9XjrHunbcVzBVzPuT1uGVy2hsq/Sb79Jivt2kw7LvD/Aqx8Kt6r+ibgTzvsfE/6s2rP/k1wcct1h8F6fjrsQvF3ti/aKrvGrdfbNJ2lxt877x5NwF/2MQXUHzUz+U48F2hNrPW19wA/5iI8KqUuD8dUrS/ue5fVqtVqvD/9livv+N/d7wcf2cX8sh7skfokKePc2nFMSR3TrfTZ3DAxiSMS9Nt30Mm5F2AW9ilZmfVjcZ7Nf/WqKO7W8I/LUdsYgGYQnqI/YAwTszYIwA6sjoxjyMKyKe0Xa1fTXsXZrYGe9zFXu3Hea/gbVncihnihpVRVc3W+06UMLSbtH3zwioxgSq3sdCmfN1vrKVG9PnJM5eplUc2IZ97uCuFPxurLVVNnpiSZN8WnCI/53mW/4xx7CMcAdW1WLSrL9Y66K6e0tvw+sqb0KZpaTmS9FcZdNX+F+oDAY1Ox1ubvv028pvA1T7ttGMSQmM/YV7/dLuiotZqtga3bmW0ncfRXUkWbSTKQr/4S6Xoh4LMrMj8xiSMzd66ntW61L4848o8+anTEeVVUtw4uU1lxac0dtrbmn4pZIbBGoaWKRWQyJo6rWffuR025p3NmNxmylkR3TOTOmuNP7EEgmTHr6PZmivKR74u+O8hnxnlkMedhqBvcisCQyp7b2LUn7mttGMrR0rreGMyI53D0l7n1g1JWm3XO00c3xVtCWJDftvAvoaz7jbtYncmkrgaR3O60Id1vdqul8d6I02EB556ZF0rQDjOazImXl+3hKvmEM6fwX57tb1ZzmtKR7XzXrKe+mq5mIvLhLLAmzlNqnXsfsq+SpT8mH1mLnh9ftp2SaPqGqcu7ZhqcV4W6pvJ+2VrUt6xKZfWHa/u5RTqMo0M2mFMMb6Zu3tTMZOD3iWtW6vExpNyPgbimMNN2JwMsVaFATgAyCoG2wNER0StI7gWe6DzHmkPWmkKe4GRH3uZWT/Qyu3pNOEWtfXoJbqvN7ojKvMMlSAvDa4Iu/Lobcb+iNwYzWlSSlDHNsB3dL5d1wFzHz9XuOfsdrDz6lCbgHKil2sP1IGj6OzITbWQngZ80KzPtaxN2OezfcI7IYtjDvniZK4XfMBot/OzKy7vj8DjNXMo+T8MfA6YRhSM6Lu51w5gGK6fwSD+HY5Sft8rTz5j+CryXd0e7RuptpP1zU2myWm93z8mITQ8GZmVZluFsp73tr+09x3KFHkTmjoKABUnejI7D462LI7+AmmCjavG95b62Wq83x6ZBzPfBcq9qtYhKBxfL+BbjZQ7j7cFklk1KPYxXrdwC3smYxJKbuhry3Ninvm1aB6Ssc7s0KZkTaLO8PQPlT4+7psR2Jrw76RqdE4BRncmk0Oru7eV2jlyngZ+JNq7XarIpM15pWgvsvWXm3Ec7s3cxX8yAy8KKR0aHZsaW23zc9pwCM1UeXuig0F/SgNZSg3mC2St1MEejCSnAfzyW4W8neH20mdaP+xN+OTfmGV4idnBWHVM3U6aXEb5os7prZ5zzu43K8J3WV990Uqvd5v/+GDyIrVNwdZ9DrTblBH83eAByiv8rhzgea9tZ5fHm/AzH32KiaN6phb5AS33NuerMinoJHtBzunVBW3m3swrF7etF7XKr/8S9cyGRMu+v2tnvF9NICT2YFEsFmJW6mI/S8meyV9/fXz+1jGSzuJqGMm4qkxX3gDAYOyztcY3kX0iuF+0AYsLK2jG9f3t/dQPuuCcenkJnQHm9xj7deZos7y/u8EO5JWdzJrKaxpl15V00U+3/Ve72KrdHuukmv0Qh/JEN2BSqckFSDe8fZD+vWEEZ+zrL323eW1z1mn+q6gzRrtafddX80Go3Fxbakhqbjm7wHScZlcZfadxvd6te/rh+/vT/v/u3xGrdTMlByoD1pbHFPwuGw8bwwtBR8Yj4rZd7lx7JlZ96tv0XjXoB2d7jF/fki08vMrLxXgvtYcauwuoce6k/U1GWK+wH3C9q+TwvgPjgBd1m7OsU/EcoC7XFjpz3utJWeF8C9UwL33uFoknZ1TvCPhKqcdrfB4k7o0CWxi3v+ZNUQ7QzKmkJXKO4HMxPSaaS6xAq4O6U7VVW7inYGVYmIKxb3xRF3Gr3YJu5j+oBxTQubUH9wbT/S3njZ484NNoXGuI/LW/dMM7QzKBtyRStztO4XDccx6VYTYZrL+ATrnt1y5rUMNqH+MCcTy4r7M427QbeaCGQWx/2Gu+nM0c6g7NM+pHF/5oOS0BbuYyEuwjQSVbFiqZU5dqoXC75xnBnhHjq906y7PJ5BO4M6RYkrLe487kQXCFaAu2QWX4x2BmWL9p95cX8+BDMXLwJ4cxlyM2EBUu9UL+PIZhOgnUGV1tSVW5nG4oj7hSOU95ke97lTGPee7ATFdhXtDKoK2l2K9kZO+wURfUWixX1WHPeB9BRDtDMoG7T/hHEnMHJiDe6Vn0EAxjM42IQ6nfaDldlNdl/kuIeStnFGNLin7ez4dOsub1en+KdDFVboKq1MY0HhPnQk5T3ReA5SGPeB6jyFdnWOfzvUibT/pIt74+VZxJ0ts1PQcmyJHJ8cQ6raVXTvqIIirqtKZbYTZijcG46svHOBYCJuDFONdZfZdwxnUCfV9qOVGe4nzMhwZ9OXGWSw09rfqSCGVMQ+WN1RxcTRnrDFfXFBefdnhUFnauxcsO6Diqy7+Mbo3VHFnEwMWpnGC437QlVlqSJLxKywd9JsSAh39DKoU2h3OdobFy9U7v6irLJE5a6nhTvVMXjCMU4jQFVGe8I699S6L+S4c55lprLuYeFOFbTuBIs7qjLaeSuTWneqUz1MmpFV8UR9GVRp3WdY3FFV0S5YmdS6M7gTdUsaykxOdhX0KoshuZATYxnUKbTnqczwuExViftUOh13Kj594Koy6459KqqsQoF2wcps1+01aNxDpbE42PfZqV4GsO7sCm20MqgCtd0V9FMo7gsO96G61u6qLTk1lwGse6En5qBQtBURaU8aEuvOmpkh4KQzuKfCKNBNZTMIYrQyqHKlPRFpF61MtsMMg3sDMBfZUr741EZ1bNim4lR3lLFtd13IytDW/WVB4/7M3SKE1aNCo3pzVRHuU1yniiqlREa7rLinqC8g3IVV2HNhMmQly1TFRgGNO6q0a+doH9LW/YXGfcGboiaksHhxV3Wq3ET3Kf4ZUeVhl1oZ0bozswik3eqJs8NUuJO59Yerot5fh6qCXWplst3yGjDuzhws7sUfUyPFnYsgZ2jcUfoGNXGVktGebR/G4n6h89RcCTbP3NetZmupwJ2jHdtUVJnoUU77kNlyg8OdOMZ2hhSyMqvsNcu1DHdcwYQq5thjCHa5ldnt687hHupsNV3cC1mZHe/NX+LJx7jZBqqqwq4u7s9GuEufbJ05joJLVNerlrQNjXExNqqqwq407vudf9lkhptFANiZqVPiIQbrlli94yaGMijT9tTVS2FlMuv+YoC7zM6U2BhyW99Xwq7CMW6Th6oUdwXtmY1hB1X5STPKdCZFtnAGuZTljJxTwggSBXsZLe0/FbRndZ0dVBVnESjs+7Socb/aGXfesEyFqwhVl77dfbrOdP/py6f/ZLq/+/j5d29US1qZnXXnOlUF7qLluBkXN+0i70j7ufT57jrF+zb7YvXp4Tc/9biUldltdM3jvlC8B79kuiDt+9A91ZLiPcT9fs8G+62U9f+xdy68bSJdGE4svnVsuZGJ11WT+lOw6MauUIgJKCGtbLrU2E7z///PzpkBfGEG4wuXGeZIqQkMRFTPHL/nMkCAf+VX0Gyv6XB2pPuls4P7ivUd8u8pwh3h/um9+fH+Hjv5NyTSf/2WtJdin6871yzWsX3nNWBlSpnw9ZK7uF9e7OX99+HCvfnx0fx43xYub7/l00/Lsa8dzHrn+u5+k/HO9W38+12fT0HDpD18315m3GPeQWAfnnGPiqr/S23BkVaAfSG0334efA8D1M+f77G4ub4bPHwJ9z3zKGh+NNKle0K7X6ZkAnG56d+j+iCbRMV8+gBP/y5pL5V2cOzX35CXJ2B/BU/+eof1DfLqr8TF31bcv98cJGVC6Z7EPS05Mv71A/R1/3DaP0ioSvLvnxi989KK0O0E68HFxT9EtjxvTIPe9T3aIk7/rup38naAlAml+25RNR330I6QMs2PPx9NNuyyvFSUvRKoket+vt0R6d86vTBKJRPha9Xv5Ve6lHGS0n23qErvIti2I6RMDP07Y12UpL0guwPhjh33fSIkxSFsZxDz/lz1mxlnljKhdN8tqjK6CDatezTsfz4xolS5nqNAKdMjsvwVEz1IBrFfLiI9c1/9+3nLJmVi6X447qNjaQ9h//Tn4+PjTRZTS7FbABpXkbCU2akn9dMPV1vQ/Gjsle7JSJXVRbD+Lznaub/D8r0/4eM3fknaS7B/OmE0Spz7PcX5d4h7f4DjXzi4pXEWKRNK92TandlFcEqcSnu40s3bRn1VWlFJSIQ7FjBf6d77FsLVi1jac3FTb9RGSIci3Q/H/al5HnuCxyb8+PHjl4S9QLvuhHEqFiu3yQEoWu10cIj6GXB/4OKufu2TMpF0p+C+yilOPeSNNdLysWfIueP84oCRaXzudDZHfOfjvsZ7pEwk3Sm4X+YTpx70Dg9p+dhrpxNGoq8s342ceyjp7/jIzWC7afhZpDslVE3rIjghTj3gRfHS8jIsVfpxJEpLvNwj734fi3decL8YN7JI92RRNR338zl3qWZKwb133bmII1XakLtOmLq5+EJX91zyHsWl8yTu40Kc+563q0rLybv39uD+HXI36UOqaU4K75FTX1Jwd/JOQmZ5d7a0fHDvhN79G4vl9RDOcE/w7iSle7KHIA33czr3Pa9XlVaSd18P+cYZ7jds975iR6opXQRnde5SvFdSu2cYwp98v2TnIdm4P52VdineS8rM7BMz0RDucGfK92Ua7stinLtMRVbcu3/jDvck79vSnYr7vBDlLsV7qd79q4jePSnfG8tN6U7V7vOCnLtUMwXYILJHsM/3yPAWZvmRYush39dDBt3IeON9uSndqbivCnLuso8gd3tuh/bS2jHMcivVWwL1ygAAFqBJREFUqEMeKw78mMJ7LN1XNNwvC3LuUrznbN2XiPaMLGcaUnHeE/J9vpbutB4CRhdBDs5divd8rcX07afg3qr4XSfleyzdaT0EDNxzcO4y856vc2+znfspuFf8EcGRfHccZ0lsnXNc0XAfF+TcZayab5Sa4txPwf2Rm7RkoqJEo53aRTDKA3fZNpOnPeaEu8vLf8DyaNy7udAuxTuPuLc4uf+by2y4N4py7lK8lxWp1gH3JNwOFXdKF0E+tEvxnqe91Bz3VUbvnsT9SeLOMe4PSWJf8GM1XlJxx0++pmV1+Ii4xpcZcZ8XkoWUsWpRuFNob2Pc22m8h0PavOK+zIr7qijnLmPVPI2ddn9os1nenRGUMVwUw28oXC+z4f5T4i4U7i8x7mz33mbjPuDh7imefEUtqiaaZvq50S7Fe36WUlRt78d9/QWQHMMF7pQC6pyB+00hWUiJe1m4I+8OaZfbdO/+hQxpc1lWpa5bWtFx3+4i6DZzNIllXpbSQwBB7O317fd03MMhfOJOI5seqe7iPsoTd9kDnJelFVXTmsc2pgTjdA66CMaXB+DuFBSolherXglhfx+L+8te2lPHVB/3+dG4P+VJe1ltBDXAPbWHoMU8sMP7C5dl1fHlIbg3CnPuEvfccE/tITjNuFTubNyXxWQhS0zN1An3h7Pj3uUwLZMR91G+uDcl7nnj3qod7oyE45KB+2bTTM60l5SakbifYn0unfuKhfuqqEC1tNRMDXBv54d7tcuqNyyq5/tx/ylx5xP3bm1xZzpxhsbZaJrp5k17SbGqxP0Ue+RQyiDcWQfWTTMjiTunuA/yy0NWG/fLw3EfF6ZlSkrNHEWXZQa03aZhnU5uYJ4Z98e64s5S6Jfzvbj3m/XEfaEQm2zjrlCZtHtH4W5NFptzRuEI92rXmW4ceusjG3enMC1TUiZyL4oTU7EnW0AycT/Wn29NprPj3qot7iHy8+y4NwpKupeWmslMY2D3DNhYwCfCfWEbhr0gAsbCR9D21dXE6BlkLsDWJJY5hg0jTXtiBOQSV9EVAkMxjEU0zLQV+Ftow7qybfh7xuI03F9qjTtGftyYH4T7UxG4j6qJ+wLjHihAIqK81zNNBW1MDNPsGRaeDBNlQcQMGmZOTIy7ZZgTO/TbtmKjkxUEsNKzFwsEu7m+wgJ9fZgWcexoNigId9uG7xT0O7qsbVxVGPfuBS+GkF/txX1ZWKBaWdyJdzcMjK2FEZyEYgYwhwOGHWp3ROhkU7/3bDJhTHIy+kFnmyDxTcWKrhCJGQsRjmaJEgUCWDNZO2HDCbi3ao07Rp6IeTbu84KS7qVlIrN6d0ARwRlgZwscLoweCmED2BcoQUwo8s6hhDHhuLG+AgCORgD32BbRFSLcg3AY6CAYYMHEMDOFv6Xhzt+inBunsZyzrFFYoFpZ3De8O3Lt4KNBsqAdC2sCnPfsSG9gmBcBduFwmmXhs2K3HeO+sJDFV4hwJ18CyLuj8ZYFswPNk555aqjazhP3wYWAVoiWKScTmdW7TxQ7AD2NNHww6RHcFwbgbsZpStDudrCB+4R4dzTEsHtKhPsC+f/AnsRXQHQHRPsrJjqZ4B70YF6h3xZXFpFEEveCrN+sM+6h80WMKxBRBobSg0AT8Yg+A/Ddkd5AMIMM6eHxFkLVNoyodgTBZ+j/4RJo0qyvEE0YOMWAzAz6W0b4NQLfCyfh3s0V90cBcR8VhHu/krivs+20TwvRaFKHbWyB87Z6GykWy9oeYF3Rr02iAusUMTOQuFdTy5SSeD+1RmQbvf3BpK300HdCcGi1FZ1kn1xmyrWoys8bDaqnZbjEfWGaGVInwcScHN5gEGQ+6XDcZ8S8bXpnmsckG41O7nyWWqZWuFe+I5LeQ+CpxKbb9DrqjAW7jkbrHq+PeK+glimlziQ87vSiqut5M3XqecCv64b73BB3soPsdcNPR5uhM4bi4/7UlLgLhzt28D7RL6qqIcjdKfLeU4Q77NCd1kx10NGp7qlafIqmc15WrZKWKaXOVB/cW1TcEc0zBLjXmqr+zFcB9+lsNkQ7dIS5q/quPlyfMRUf96bEXUjcXYy7j8AGjl2Msk/EjAeuHYRNQ3U35oeuu8LjXpyWKaXOJDzu7XTvPlRhWx96WLsA4jgo/avRggmgT7do98Qvq44k7mLivuXdfeLdQbtrmot2NOCQAwfdNNqFw71A2st4H5nouHfb+7S77jg61u5Tot013fOmgLunQpQaandPV33HcTzBcS9Sy5TRRSA67oM93r3lRZkZxLoG2h12+IA7EjqzGHcfFM5fybT8o9QyPNWZ/hbCTushcBmfQ33zt1o0zTQFx110O7plxtE1ZolV2DrTk8Sdb3s+FnfP97PQLhbuhWqZsl5pILLlujBbONybEneJu0CLs9OtL3EXBfeWxL1iWka+S1jiXqr9lLhzbu28cReozlSwlpG4S9zrk4ZsyjfFn93O8RwCj7psT0Dcf0rcObejn0PgO/HmVFXVOpRVu02Juyi4v1B7B1zy47Y21/C1SEtwvO16w1rgXriWKaMlUmxj9xBAW/tQ0wBsWLvkoF+nbguRDeu1XWgI81tauLpjqtbh0RujwnHvS0DPa8weghks2htCh6+uao7rqL7n6IjzoTPzVQcf9lpDfT/u4pRVmxJ33o1ZVJ2C43ZhPSpekqdpruv6eJ+Ll6Tqw63BNcC9eC0je8QKxB0/WkALwQbxgn48F8QMPF+jfriPJO7i4J7IvMDa1Jka4a5p0W6P7NkIVffgLkq89VPiLg7uD7sRpqZqmhrj7qhDx9Fm8CgCzwfvrukzWMfn1gb3flPizr2x0+6uM/W9tR93gH2vBWJmqA1x4kbLFqoKUmcqQbrLlsjicHdcEDP+Bv/u1so9vOHWp4vgp8Sde0t7DgGIGd2VTTOlpSEl7ue2lB4Cz/F95wy0C1JWfZK4C4R7XmuZhMF9JHHn3x4LwF2MLoIyaC+h4f3/QhgrGfhcAO5C1Jn6NcFd7KeIvUjcK6xlJO4S9/qkISXuueH+kCPuXSndJe7Vwr0lca9cGlLifm7c20XgLkCdaSRxrw/uQz9tWbZXA9zLke4lLFYVGveMzyHQh6wjsCpb9YXHvduU3l0A3LM8h8DdXoYd/Us+Nd/zhvuee81/WbUk6S69e064M/OQsEB7inAPl2STN6d6qqOtX7lHXrAqNO4jibsIuO/tIQjfqjqMl2Tj12b7qjtc90pOVU/0LoKmxJ1tgUnbuzAmZ2LXMoMz4b63h2BK3rs3bG0uyXZ1f2uVny96nakvccdmK9iM7b2mQp0EytlwV/5j7+x7E1XaMG421JyeaIoSXN/yqPWlrawU31a3Rrer0eSc4x/9/p/mue8ZUFBbqSuuwHVltyIOg4Eft9c9DDPmmXC/PYa7nH2JEXceyZ5nfg7c4dwH7aHH/Rm4S4Qty1Asy/KD+/l0PtyP9iGYO9F980j2ODOv1rbPMfmhPfS4T4D7JsBr0l8YRn/FqBuG2SfcLUMj89IVBsYyusLMLP9a9e1ivIFm2J6Ey5qylGWs/jL7FlVCbwyTa/NUQjVQvYS7XZPJa7vHbdIx3Fvvencxq2pt+0g2tz3Obp2HsqeZ6mAwmEX8tuofo/3yo+b5w91QTFPTCD+FWFUUItC0+uRexKdaX4BKbw3DsiTuKypgKCvpckzLNGlTxp1W9RWNrhjaRlMM09ipRNNMU1H44qAdGl3hkCxZz+/g/i6pM55EtVrbPpLN2atwOQJ3MZlqshpt3HXg7sVdULdSzC5T2jVsM0OE8gdLSSPjrvTdYK6kmzcVqyteNrivZLUa5QRdbyUmv1i2meHCBpUx+qeaGT93mbyTqfL88IPP9iII+dhvf866X37UPF+4CwoJb5mOsnc3NZnCEo6GsYHbUhStL0xJt88FBLZ0eShkR7a4azb8IqAbnkr6hu3dVwZvv6QNlktlGSTuXvbrtXo1bp1mJojuXtwtJo5C+0oATNGduO92GVFLsWwaRSzvLi3pSugC6a6clHO1pIguLIl5EPdtJX2+Jsj90OpVV6zU+n3j1FT1hLkMBvMvY+Ae8+je1TRLWPG+YhK6EndLEWbEaaZk3PvWcungvlyZEvdlf8m4ryjEky0/hLu7EoMuGIn7ymDcaROLf0xWJ+CuX6RDZNhvq/5B636luP+1Isa5oaXLr9wyQ699YUBMp7mdcBcOxjYz5GBs3Dm31Tjj5NT0IO7bSriwxtstNX4Vvyla13FTn8X9EuMQhB/3Z+B+qDXc+9qVVFPc7R4q5V7abvNeze5KPPvpUuQ3P9j0Y9w7F8I93L0IHoC7T6004/fvpX5USd/QtO7Jt5l6F8I93PeZJsDdb8S3Pt+95VOVrEzTD+3v4H4L3IH7GXG/8h6R34E7zEwMcW8FjHuob6siVY0a7rfA/X2pwD0auP99Kdx1uJmQTN8RX9z9DDLg6U32fvlw31ZVJ4juEcD94z4EYpCBT0xn8PODMQlC3ovgz/EO3C+E+yAzH49n883YA/aE8c5oBOO9UQno1+Bd3EP/cPYDcA897h/eVJ07T+iJ5/e+0LtpfU7hfna7WZBF6lMH85/RxT2hT+KB+/8iIfXTuM8y9fls7MGdn1ziKbSdBYH3LDObZb4cwT0KMxo8T+KAe4T1cR+CWU2acRfuP3cXiP1a1eXe51Ee81p9AO5h1rE+BONZNTOQuM8F5SLoz7YLs8zA/WxTxHEnR/MA3MOrD/sQjB2Apzxe2NTBfeDgLsZXqlbn9fHm+b7I4355Cw/cA8D9UB+Cef3LbMbDP/L08JSaHoru9Jm078dS1ahMFH9xCw/cA8D9YDM6t7vXB8LIZKrzw9H9tp4Rc2t/iQ/ul+1DA9zPp2N9CMbjncEI9vzOuD699TVVfLRO2zNwjyDuxzSt1us+uxmUI3boLmVpVFB6NbjP5nO/nWqihntCvQzwwP18Z+xiHSKjMlH85YEH7ufH/Ttwv1ZLA0pDifs4oscw6BtPoPRsKl8Q99voBo0H4A7cY4N7sJ4GlAL3GHkaUArcrzPEPwB3pKqRbpkJ3NRMQOn59P1y7e5qTCLImYlHH4Ig3MzfCO7ns/HnI/5BBaNnDUaXGTWvF7fT9vxwBuQfENqh8LTV/PdbrD/jEEJh0kv6n39PQn6CuA6FTtk0i5D/DPOTyTMMOxTO8C51c/P1Hx/UTyYPQB0KrxYM+4J5l/qaFtj/50ac9PDw/KwDdCjkenWiu1cLHBkoivrFsX0P9xccGCi64X0X90ccGCii4d3t3eFloOiHd3gZKD7hPQ0vA8UovMPLQDEK7/AyUIzCO7wMFBepix3ccUigCOvFi/svHBEownpMw7pD8dEC1h2Kj36gGRKKKe6w7lCMcId1h6KtX7DuUExT1SyOBxRp3QB3KDZ6dNP+Fd4dig/uaTREQpHWqwf39CuOCBRhvXhxR8M7FGX98OKeRlMkFGH92sH9Bw4JFF3d7OCeRlskFFll93BHWyQUWb3u0o62SCi6evFGdx5E7xVHBYqofuxFdySrUGS12McdySoUi0z1RgwZiWQViqgebw5EdySrUGwy1cU13lnVm431utEs44xB58pUnej+mWQ1265Vq9VpwHa/uV6XGo3Sep1Tcc6g82SqgvfFp5LVdiaZrFbpTy1I4BvrhpjXXM+tS+AdOo91tzNV+vPqc/tvyXqbOR/WktXgeG+uc87iaN3AWYPOg/vGzfjsBtzeQk6LQX3Jsk37qCzQH+G0QafpdZ/2hf+m92ymTuWy0sd8S34L6Evm1roM7Ay6ui7htEGn6WV3jsm0TbwvN/Mt2WYjk5Ehvl4P6EtKwEd2WM+tcdqg0/TjsJfx6WaqGfFi8z5NBoV700V7ogXcoRP16+ZAripCvJ+m94xt1yXvw2RAyeq6xR6msUlWcdqgc+HuyE9Hgk12WmNXExzuOVe+iugOnazFAdxliPfjZupVF+2JdlC4lwTgDu8l4A6dqN0Js+12GZ9upiYBl7TTS0BfsiVte1m8dLZt8BD0W7i/pbfm3YebaYu2x6GkPes4+SDCu+grw82Rammt4rRB58D97W4T2xe+3Ew1M2TQ5XJyGNS3LK/XHXuphLtM0KnK7tB+l97eaPJzp2mYydiMZ2vJaXDfk3hvjDqdUW4N2qHTcX9z437HuC/sbNXfnaZhJlltD7PDaSZZC/KLqgQ6K6fjpEGn6vHuzRPc77ZexmcvYIrqQvV20N9V1XW4duh3NCTC37axXeC+aZ1Z+GxXzLJwLKGr11ww/vZ28/Ymlu4WG9bp/z2Gi4Sih7tL27uq6cX9/RxHCIqQpru439//ent7W9D/exaOEBRt3D2CJYcipLsjuMO8Q8AdgkKo4d0R3gc4RlBk9HgM91ccIwjRHYLCp/Yx3OHdoehofgx3NERCMYruOERQfKI7OhFAEdJ0H/cpgns8NGqqwP3ucb4J7Y9w7lepSuEMo8L2FOUJZuZumEhkH0lA/WpV2AW13CsHhLve60Q7VQVN4cO9olQCMjMNJRWhAzfYox3J6XVLVV24Ow8zunBX1cTHi4l9yNW9Mps3e7gfrnOz/srTgfsDXga6WjUKilJoCtzVEi3Tm0SildIULZWipVxRU5SUmN5kVEnRYp4fZNdLvFhs8mVR4mXeKJHQlJxY00gpqrt4SSnSHqga2keK60ttfkuaRUXR8mycxN7tPSlKryjKj/jjp2smfrcXQRtIXbEIJ0Xjf08cd4lxep9LtBShJttxsSrPoBKAVLQgfgwIWUVp8PZia+naeVNawxuo7uIlUUIUUuVC3t59Xu4+1UnoBflJQRe4yw0qcl3lmo9gdg4rE5rYTgiqiV5B4KpWRpSjFoTZcMzME9lxlaCk9DJXIhJzjDThOOKBS1RxuRTLiTJx2drgrhRanuKEu1IYJTr0ie41M/Txk067p0/zipZT1ZzG+6X6Uy21wxdATtWp7t5VH8TssD0l3U2nc7TGXHmKKqjuuNpUmgJKT6qqc6B3JbUtxtD5dUjxEsXy/AZ3Td/JgUsS2J64Jty427unhEEXvxXi+isz7jxoZ0v+Cujy3XXnP9eeYkAJx35sWmZalUKhkGLgNrj38kVaJ4rpJV7UCEFVePx8S+BesV1JYYO7XLMtTrgLwlVx1bhx34I8Ej8g9OPCl8RIXh5l+ypLKZgqDjo77hTXC8ViQYBn497RKCMtFrkYMa7Rogjj+lNBWnGCO7+Hu1jjLl7iD529Hca9tYN7WYb1FnCHzm5mygJdGeJHbtyfRKopsGwqWjmx5ZviPiWkGzMjKnLj7i7+Lu5bM1PeMTPAHTq/yFWX1ERHpqoppSQbSzo25yq9KSbshDMnPHlPuJOnnmPyKfBX9ITuSVXzMgvdFvfg3uT6eZ+FHsMtdk+pakXRyLk0nVQVuEMBpFhF2dIoGiKfCN08u5SOMBepVIVfivmiaJwsk6/JV0QjIuGfYkNfkQ2Zit1U6cHdVdyFOxGtpxStkOIL7Yl/RMTWWk80RGrcJlkG7lBgvIsbRrmKaIh84nbypnTRjZTAOccfNwrM8YiBzOeJ37Kw7uKWEMEtbjPlEju4u4p7o3uixTe2/t/eHasACAJhHHfoAT6Qg4aDZtfe/+HipEzCxiDq/5tEhHO4xVM8X3OOf8bjlmtaIqBKhCqWSHc8yLq6oXfj/T2BnW/F5LoOa3L7uAqnm/mk2jNFx6rhToD3aSdXgHQHvlTamWlaCAAAAAAAgJ/ZAOF5n7MgcXZlAAAAAElFTkSuQmCC)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "id": "HQN2bu0ScmCM",
        "outputId": "2317ec86-9d01-4a40-a73c-3b6be5d60d9f"
      },
      "source": [
        "#nltk is the natural language toolkit https://www.nltk.org/\n",
        "#it provides a lot of tools needed for NLP analysis\n",
        "import nltk\n",
        "#use nltk to get stopwords\n",
        "nltk.download('stopwords')\n",
        "from nltk.corpus import stopwords\n",
        "#gensim is a python library used mostly for NLP analysis\n",
        "#corpora allows us to build a corpus out of different\n",
        "#documents\n",
        "from gensim import corpora\n",
        "#Defaultdict is a container like dictionaries present in the module collections.\n",
        "#Defaultdict is a sub-class of the dict class that returns a dictionary-like object.\n",
        "#The functionality of both dictionaries and defualtdict are almost same except for\n",
        "#the fact that defualtdict never raises a KeyError. It provides a default value\n",
        "#for the key that does not exists.\n",
        "from collections import defaultdict\n",
        "\n",
        "#these are the document samples we are going to use here as an example...\n",
        "documents = [\n",
        "    \"Human machine interface for lab abc computer applications\",\n",
        "    \"A survey of user opinion of computer system response time\",\n",
        "    \"The EPS user interface management system\",\n",
        "    \"System and human system engineering testing of EPS\",\n",
        "    \"Relation of user perceived response time to error measurement\",\n",
        "    \"The generation of random binary unordered trees\",\n",
        "    \"The intersection graph of paths in trees\",\n",
        "    \"Graph minors IV Widths of trees and well quasi ordering\",\n",
        "    \"Graph minors A survey\"\n",
        "]\n",
        "\n",
        "#create a stoplist that includes all the english stop words\n",
        "stoplist = stopwords.words('english')\n",
        "display('here is a subset of the stop words')\n",
        "display(stoplist[0:10])\n",
        "\n",
        "#Now, let's remove words that are not in the stoplist and change all\n",
        "#text to lower case\n",
        "texts = [\n",
        "    [word for word in document.lower().split() if word not in stoplist]\n",
        "    for document in documents\n",
        "]\n",
        "\n",
        "# remove words that appear only once\n",
        "\n",
        "#first, get the frequncy of each word\n",
        "frequency = defaultdict(int)\n",
        "for text in texts:\n",
        "    for token in text:\n",
        "        frequency[token] += 1\n",
        "\n",
        "texts = [\n",
        "    [token for token in text if frequency[token] > 1]\n",
        "    for text in texts\n",
        "]\n",
        "\n",
        "\n",
        "#Next, we'll create a gensim dictionary...\n",
        "dictionary = corpora.Dictionary(texts)\n",
        "\n",
        "#and then we create a corpus of different documents\n",
        "#so, a collection of documents is a corpus (corpora is plural)\n",
        "corpus = [dictionary.doc2bow(text) for text in texts]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'here is a subset of the stop words'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\"]"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WWfoaE2aurpC",
        "outputId": "3f5089a1-5c30-40a3-e090-7392d8570b0e"
      },
      "source": [
        "#Let's take a look at how the tokens look like...\n",
        "print(texts)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[['human', 'interface', 'computer'], ['survey', 'user', 'computer', 'system', 'response', 'time'], ['eps', 'user', 'interface', 'system'], ['system', 'human', 'system', 'eps'], ['user', 'response', 'time'], ['trees'], ['graph', 'trees'], ['graph', 'minors', 'trees'], ['graph', 'minors', 'survey']]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8LDD9LmxeXUZ",
        "outputId": "736d2e98-000e-466d-ada4-e0805bbd18c0"
      },
      "source": [
        "#Now...recall that we created a gensim dictionary...\n",
        "#Let's see what that looks like...\n",
        "#Notice that each word has a related number, so...\n",
        "#computer --> 0\n",
        "#system   --> 5 [there's a reason I point this one out]\n",
        "print(\"The dictionary has: \" +str(len(dictionary)) + \" tokens\")\n",
        "#token.2id shows the features and the ids associated with them...\n",
        "for k, v in dictionary.token2id.items():\n",
        "    print(f'{k:{15}} {v:{10}}')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The dictionary has: 12 tokens\n",
            "computer                 0\n",
            "human                    1\n",
            "interface                2\n",
            "response                 3\n",
            "survey                   4\n",
            "system                   5\n",
            "time                     6\n",
            "user                     7\n",
            "eps                      8\n",
            "trees                    9\n",
            "graph                   10\n",
            "minors                  11\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4lDWM9bee8SV",
        "outputId": "e8e06f45-77ba-4df3-e13b-0044cd5b3978"
      },
      "source": [
        "#The corpus contains what we call a word vector\n",
        "#so each of the document is now described as a tuple\n",
        "#(token_number,frequency)\n",
        "#take a look @ the fourth row from the top\n",
        "#see how 5,systems is counted twice...why is that\n",
        "#(take a look at the 4th document from the top in our example)\n",
        "#\"System and human system engineering testing of EPS\"\n",
        "corpus"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[(0, 1), (1, 1), (2, 1)],\n",
              " [(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)],\n",
              " [(2, 1), (5, 1), (7, 1), (8, 1)],\n",
              " [(1, 1), (5, 2), (8, 1)],\n",
              " [(3, 1), (6, 1), (7, 1)],\n",
              " [(9, 1)],\n",
              " [(9, 1), (10, 1)],\n",
              " [(9, 1), (10, 1), (11, 1)],\n",
              " [(4, 1), (10, 1), (11, 1)]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pVVODUWslNFS"
      },
      "source": [
        "    \"System and human system engineering testing of EPS\",\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "96bLOmr7eMnP"
      },
      "source": [
        "#TF-IDF"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dFYr5hefZoOx"
      },
      "source": [
        "TFIDF is the Term Frequency-Inverse Document Frequency model. Much like count vectorizer introduced in  is also a bag-of-words model.\n",
        "\n",
        "Term frequency-inverse document frequency(tf-idf) that can be used to downweight these frequently occurring words in the feature vectors [PML - P 261].\n",
        "\n",
        "The difference here is that we are weighting the words so that those words that appear more rarely have a higher weight than those that appear at a higher frequency. Words appearing frequently across documents are less important. Those occuring more rarely, but not too rarely, are more important.\n",
        "\n",
        " The output vector will have the same dimensionality but the value of the rare features (at the time of training) will be increased. It basically converts integer-valued vectors into real-valued vectors.\n",
        "\n",
        "Compute TF-IDF by multiplying a local component (term frequency) with a global component (inverse document frequency), and normalizing the resulting documents to unit length. Formula for non-normalized weight of term i in document j in a corpus of D documents"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "96nDMGjIdirk"
      },
      "source": [
        "\n",
        "![b51a7ddc6375895ce7cc6aeb2f46dce720ae7ab3.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAeoAAAAUBAMAAACwrGPUAAAAMFBMVEX///8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAv3aB7AAAAD3RSTlMAYIu/GPtI6TKdr8/d83RAGjvAAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAGOElEQVRYw81YX4wTZRCf7W3b3W27t0Ee1GjcFPQ8/pyXe7vT6N4ZDCGGHochxAdSgmhCCFR9QRJ1/XMhikmb81SOO0ipiAECVOPLHS8XognwYMCQaGJiGtFwoFeKuZiLD+Jvvt2213bbaw1EN9nuN7Mzv5n5vpnvmy3RnboGWtbwWXcTvVWzyteLar5XS3e3bF6uo6KknOfHC2ghut++E1FXm/3VeewcZFPxBoraUA2tsrzc21Ls93mzPzPF44NCmRbogW8WAyw6/fSHnq9P/XXiCaPG7FrnoWfxEzYbwf9QQwf5Ec63FHXQmx1xllSaLtOO5G+LARad/vaq9+ssyd/VMxti3fZG2SQP1tBRfsaslnJNzXmy17gJlC7TAp16FgN0nZbn6rxGYA9Zdcz6WXd7I3SRDZX0eX7OtlZhAQGjVbPdJQ0aZfq8Z4bVXK7T+iXv14iYYgablZZRm7nGougKWoL0XLesl4Z3raQdr2e89FxpzobrK+Kkd82QevgaZwev/rrvj5A+8vYIMRqNzGQhjgnxRdP62MBKBPFz3t7VvSBRRcpEFgAL+1/C75FjFMsfJhrLJ5hmwbH8oOAzyhJG6RyNfjGwMi6sXce46LR8aq9B1x/tNItulOoad9JktAF/3B8PGmEzsNTcRMoV+RY9HEd9nPM+QBxpZENbCmndR3sheJazQwQApf6kOYX6ChjhXGga4qEUuFvB1RPUlg4ehZGpEhzvV1Mh33QJWNjfTNSLtL5p+62wGcoyPcg1iyHzXZRJ+d4/aNaEMvuO8TUqOg338TKULrpRNPcqr7fBZo0kJUm1Pqd1/UqBgibN0UWK5Gie6EZpJR4/i4uddaWHiTqIbvkSFPfxWgyTIpJqEyQ4y4FmTVA4DXF/XEvIR8FVp6ETkGBkg4MMjStc3GcnrRIw21cGCaDtRh/5jQ6kEWhG56HguyibJCTyatKybK0f4wlip0WJxolfltzQ3R3ib9xnLGH2JK1CaW2MLietG5kvXYJuuyVhdsXWpFQutiM9TjRE8p9+ZAvuzaBF1DKvXR9uRhuCgxBH3D1RC1wsOp93WjZQcJBtJ2o6dLQMzPbVLIPG4pdxDyGNQDM6DwXfQUHcbWnitBLWMJ4kyd1kY7jBKLkhu+XJm9yPJMz+BHFVnnem6BBCDwwiWbSs94YgpJFJyjzphSQcT9qQP+fs6hKHvgHRAw0Cu1l8FcVswW23FTYSyamJirpeve2ldAmY7SMOAN/ERnwTGttFvg6SGDLfRUFm+w3AxmzHdwOLVXT6QYdRdsM9FOCeb84xOweDL/Jeb81KmPFgDnN7QQqb/Spj+I5VZLgjTfNruRpSwFN2I3WN+bUiANbAC3FyPFOgT1h8SsJxJoM7K2GRyW63wmmBjM1QKBlBMkrAbL/9XTtJ9BoS+yI0JiTQEBRD5rsoIS4fwHZIwvckNm5sTv0ivHHetRa4Me6e+5jdYcOJerO8hUx6mXzxEyr9To/ZqK35rZF4yuT8++VG5VoLablg4hQZ5/bhgN/al14Oms8WJDGXJDHaCwXlecSibHkLJ9Aj4Pap0FHhb9JgZH192j25/GVgYT/mAzAm5pKUoOeUvT7QjM5DwXdQhmdQ4FjwXlX43oGUTsFpgTVJYt8puuGLWMpXJA4t6bJ7YN4T3daJEwMlsA2lffgjjiVj6hlbYf+tJyujdqQzOXoHB4OSiVrS6ZlJCzT3ERFMpMYtDqPd6LpF1Nl11ZQhBe6OUXofbF/mjMXIclvObRfkBcBsX+sipQen3/0Hcef3j4JmdB4KvkAZyfdQhhxYtpahwHELTt++fZuIQ8sscGMfBbAZaG9OfTpZpzk6WW6uRXu3p8mO41nufSqb4jod0oSDHLQ9W8OTddGbvwpVbmAG9MaN8CvldniAlrqNRBOXFifrgcovOs8OaQNWIjIgWZzYS6iR/Sr05r+pSg180Y0rZkXU1Wa1hF7eYLUDdIEC080a66ahIxWMtm7Pzzycj9qBcIp2enxpVtivQm/68q8wqtw4btNogw9c/fTBSprUVPPf/ZkKWeWpN7zEok6nm6Mxj8/7avv/6l8FrdPDjVwLf2as14J35EO+dslz6T30f71y4a67A6zTudx/GNc/6F/r1W011zgAAAAASUVORK5CYII=)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tGgQ1-tJdi8Y"
      },
      "source": [
        "![TF_IDF-final-1024x399.webp](data:image/webp;base64,UklGRqQuAABXRUJQVlA4TJguAAAv/4NjAI+gJpKt5n+gwgcGGPwLoc051cmGoraNpE81/EkNiSFwH7tLYBU1kqQs6dgvnH81p4DRwM1//P8ziPkug4YrSV71WeY5ag19lv2esVu0Qkt2OBFQihIJw1HQto1U84e93X8AImICMi9y+tyz4tDt+hltgX00sV6CNLLh6xZt2w3a2qJCNiSBE/j/v42lvWVKLwX3PSL6Lwu2rbptDslxubXcc2/MFULoEffPt21LkiRJkkQICwBHAPeI//9bgnuB2MRFxVhZXiL6Lwu23bhtnhOkQi31QbUgLiAoL/npmn+5/cftP27/cfuP23/8fz7IjLy5/d5/zZY31997dMCRX3xEYv+ONhGklDdgqxWY5BnmifUKkHgFjmfCqlUxyhUszQY6zVuUrEDoq0IWGl8QtubciMFsBHj5BGJH30UJOD69Bt0+RpM2G6Tpa4viBTUCwf4FOccECACAjuj0DvISED8/wBBFpMzzNKpEMFe4PjcczHvArKVj2D0dxuI1dfrNxfZHiHiAsOPzqNuihdG8BU1wAfu+w0C2OhyoeEgINe4MyittCrugxMtEc+mI3NY4xCxtTGjbRLfrI82pdVWqO/3PNvgJvB/hCcYjQ+aq8tL2VdFdPWXEPyFeTRWCdTtz2yMHHdvjkV608LOy1Jlrv9oXzaCQnEHmHXSM+LQEMIcIYoPmcTLAfQ17KDm8AhiqE99TgmhH8w4kZVhQw34NUE5ieGZSApP+hNxLDPkHtZw5Upkf4/h4vqaZ/ruCUOF6lq4NFbhZIDU7/cf6XLa4QeIFjzRV0eTmKQ2Pme1SCGHoeGlgmEBPoB7gN07UEQlVLxQZYpD8YqZWiM1I4DsM7h72z0WGhJx84IfcziCVlUp6dhlTNPDrZXWMyJmL4OwDaamkO6gMIZNKDQ+r1RIsT7xfs51hB6SbaRchzd7lK9Xwj/BhaHLovoksiiIE/2hbvm41du5CXG2nZhnBToV/bPsfCJ5lZmmM5ulFyjOReUyn8iAzWgz92tWwRvjXySHu+iu1MYqXgRtn4atuX/K1q5EyQM/qXc9NjCqO+Rn4S1Rk05V8pag4JOqxQeprW4vtoNF/nboCzxZHXYC+E1+3WoWnr1ZG8l2V76F5E4JouJX/qVphpQli+tHmR+NcbLeUkWtjgzOPVZWZT/rZqVkecN7RBq43tuMORQGXmuxnsbijswbmHSPXjefS1uYdBs47tB9zvD41wuMOahr9Ps0ywb9pZ+cd+DbapVlGfN6R5nje0Z9Agi9HMOkWaMTm6Pc8dE2Ouh9jHkKk48wuPdYazzu4e7PkiU9raozmr92a56ya8vCaIxcebIPmCdAOzdW3HtAKad43Vd7TQJumeQPCVy1pjtcd2Nti9d/tsLJb8xZ/27wXq77tsHZonn+a2d7NTLun6eXOJNbizfM8PQaVrs0hD+OHS01DmqyooukbSxqKV1i8FiWWraScIhY/IXkYfFha3i8qsBPtzb/c/uP2H7f/uP3H7T9u/3HXi9t/dLf/H8Px2XgXXEW5JJHi3sxYiqLfomUQIle4dvDxHITUh93uSZGQmIBCkYBwDys+AuJIwI8KhUJDtHIpNFYSE99DMQL10vivWArRC0QypcOYgj9JUQFviCIiokcUyoF62O9A6d9u6JeiDHgoqirEZchnCYl8Xm/+kzV9Qi1P9/ZL4zypzzX6KiKZ9mmeXq4tBNRPzyjWNy6LbV/9gsX0HRWzLLD190aJf+MvRnwE2JHFAbmBlRKXV9im1sWqXI1IHwsWKy9pJeKhmEWQSJzs1oUFtIfxW4r2QJtPOs1D0ujnr4Y++CwkAupR+BmrCdHjC9aOPpa1RLzX5NR8UkxIdzppn9efw4R/YQ73WVxO09f8mjKSOw8e+EZbR/kgpD6Mf7i8Hd5YjG8gUAzMLSvU261FG84WIB9BP3zbx/6uAoBdNvCZwml02cUFqLgJJcTELzhCSKQFjTNKTCZAQGMgleZpmr+m6SWx26ocAIoIly0S0qMD05B9b3OAJYH+DSo2Gfk8BtSAlRHRLScTOpF7dVWMzO6yW74RSwyA/uEvJnyB6eUdoiDIFwBX4yIRlfNvEwCymoEbsO8VyVKrD6DYDMQx2rfLFHgQA58ITG5ICALTEco1ycnRGDSzr5Cc3J05yDCO48PcQO/4KYowsXdVKlrmQWOY3YftN7kP2A/i5d/GUfTrpywmkxQF4tqw0eEKbvpGzBsnl951WFwDjlJnNdvizSUnGK+tH0Shl4i+TWiKRihSbJ8gijcXv7gIeZGCv9/FKS8dpJuiijzSX2hwStVkf+aKCCWY7JRBmclkFlcEas/pIyRlnE2yk2PcwVBfTtWLKfpMXR0Cd+wMPoLDcGVOZoVWWsOcKtSLsoFIupIFzZxkD0gr2AJZxZHrh9itizpaqB+mTxNIGWQ1j4YzTKcISZDqyjvUV1h934pBbNYVu3GzCom9i3EXwOBIgpSACNBbkMT7I4rqGPg9CZSpV5uVuyd2OCLgRHc+MU3zPAEUZQzoM/YZa9QX7ZYfDMsGMC0wkBt63JrjEAJIVmIA3GX7DdPESrBV8vKbWKur4McF7I9goRL7GII1T7oFuWSQsJ5ks4ZXA5ECPRyQTIQj+50LmLrziKdKKWClZBbLEqKMIFYxU292WvE7bn9VEe78S4SMAqz5bXQsRUG4YmDjSAioWmNbKDg1oN0HCOsmlV5whEzSchQLOk2+YFMkkA2yTvOYUreF2W0uGGI9BJBKMCW62aBGGgQeo3SvEqucQMyTLQLelo/hoyFudrJfZjQsrl6fz5fjAeLNSOzWvW2mPSHUA4HpYzG65V0UhCt2077w0lMSMrK/4MFNkhdxUJZ6cjkG/0umI7EYFyCJUdJ14Vj5AYqKeQdbkN3LwHoY4tSpG0qAT3HjiYkR2qzAT3F3V08AujpnD6igNYiZeeIrxIzwJEBF+NQUqSO8pDgs7ClQ82qfVxr+Egbetwlp8U9ZQvhHAOK0DWDgFkyL6U3Y5aKbRJHo00f6LtjCLHY40RLbxADJCL+QA0kmkad0ClQmCSGdSVTsrkQVRq8IZnUeqoqp6Qbw046YGL/x7vNYD1qzWccvVfMO3haAgB9FK1HoZwzxuPBeT3wxLH7xZgO3TVWeJ15OpGtBhlSrGvA2lMV+2RRDtibELkeRh/ypQQpPPPj44/soGEZ1PoWYn7ZIUOkcIDSw8JS4ep2maTa16YErsadIbxiZ8Z5/oNK25tVnD05kbN+U1ROdSYC3DnqxLhOzYDPOQTAFEfEsbp1Ft3pfyXQkHyASLZMU+8r9yJuY/b+IyS2fPtCOIIsdnYEZin2TCDmDQIgXftIQ22OwKWT8QnqEqK9/yhaLuXU5E9mbnXafTonNF+HKO7YY8mMJKwQpz/iKNeAxCLHdh4KqxUxK/EAWB4ns1DBDnYGfx/eTyx2Y3Gu6ZwNzfLEh5zwwUX64IWHg/ZZnFyn2FEmP2UyGq9p7+Zocj9vPO3Arc7feJ2tz05jBC65RM5ADaXV+9Eyh8C0A19kpRx+QWY9+zsBHvUk19fMOFtiBAls6zikp4peeK62izczlDREgweOLW8w7UMGTCkwmSyoSmcM8nF+E5x18Ta9psvMOc5V6BUWqF3svrvS0j2/2LbAK6vYvm887AFP8WSZxhY9GLJ9eilGJOkNej8kddeJldQnLbyTxURbnE4yMwS4IfClJMkEpGcwLE6hdeUJLg9Vo/wFcQYewnlz8k3EHWbZSv9FicIxvcRCyiu9NP+GjqjRqR5vOO0SjCqDCFuQHPdAQrUQlApaJfYdiQ2H2QSP7O77f7LoZgNggoZBURHYc4hFKL1T5UUdYjDLdZ9r88Cw7pfseLBUIjPGtuH2uvjWbEb468xojMCgJXogA9gHe8JhHielZpRZm5ygcu5DIQY30mOkdEcgaRCCsx447ImT6GmTeQYLHIgWhpwr+GML6+OgYKrHdT2cch9vZY+qEqt/E7b5MB4J6ZVmKEjNPCPbbvQYxqqBmmFGKOAAmBwKo7630ol2AQsR4I6CcG8D2eD4SoA9FT9Hd4w2pBUlHGdJGCHjKIl8klv0dBObnsf2BpClngJQyOKgw50yg5r85hypBDAA0aPECDlOm5M46AMD+B3Wu5pOm+Z0LJZXwQcVe9uCW9P/NDVWu9hMvLug5jb70i5v31OzRSW+nLt45l9t/dHf/dy5iuUJUXA/XASVheXGxYplLi7oSVFnpQ1Rllf6oSKwhJj4uMuRUjUMX35/Sp2R7UjveAFTRWSz4lUqZHnTRUlF4/dKiAV1Cy4gVLMnbwfn0cgLKoP4iD9W0C4UBeKGBcOKwKfI1YQdEQiE27kA3oJtOPg4x9VmqnNJEw3vtAZ/WSllGyuB0VUKaeUCPwAtoZH7RiyzYQ4/o+bSgCONK4Nx7wbNwp4gLkmk4Qx5odYiVniEMBTl+fj9F0ZOIUzkpDIoiCRcdorewEBWUk2zRyhOQleFFb2D2IEhBIJ3y7Qiebk9S0AFlwuuRQxAd2LWAJeiUco+S8fT6o9Kfz+xLsls2u7Pjo6aLL3ohgRNrJkwZt8YIofsdwMhCSf3JtG1PP4jMTs0t2WlhZ/LEClM0lEOSEPIH3s2geHWaPXGnXADwg4CbDbj2kE1n+1PSQO93wQlSkpAU6UlFZWBjI9OTXnsZacCwl8wA18MiGJIwGG3GNXP1rCASc0F8dKyDGlIooRDbM4FjdLBSLd5o9hRziqc0kKSvv4qirrP6R7AW/UijPYzshxhRfz5MKI9IHgKU44MgiYD7E0VFOE0irBIvmfYvMo5amDTw8xvclyfslBcsNFUady47G1lBh+6Ii9/PgtSNk+XwU1CySRSphowPwpGrEetOeVVZY97ijMFCvAmpDxmh4qIj5lQwTan5+2sQsFVCsJZZ0Y7Y9a8S7bmX10u29afAYtlwIvfTZFPe9pQ1GRYeTuU0ZMsqNNpIZdCcMYZs1StOJMluGxkiToVj5Ts8F6CZNLRIKxeGLFUkvJLMel/Naja6GIVbG28LvIsBhjOJivEiB/dQdCEIxdBVLgYgBmq8NVQB2iVLlx8UImUGCSNCiVUUOqWS3VOFUAXs1njastJhCGzW+VPYYgxbkgzQTMQTcptrAQoHyz1ejJUUr/gGWGFOYfLtVCMI3k+fuJwUIq0i938EHOscJLaFgpxFw60sxLgCu1sx8bRbxDcnnFMVD+lShKJVO9nXIFgDaVRso0BAP4aY/WiiECSyGIFbs29aQbskJTadUirFi01NOKy4D4KitMFiCea70ExKuss0ucfLsCK2zbF72BQwcQ6P0g3po6KkkcR6EeJXyqTi0kjqGFXAq//2xUx47N9XI4F9o1aCYu1ZQEaOuRnIvGq/shCnynO1/Jx4proAIRnfwtagwrrFyelJgnTCGX2JdPM/IlNkYxz02msqgWtya/4GiqVT0Gci63rgLZsdaUJM0pQSPUqgvopD7MXi+OKfU1aBsHQFVlgXPXzB+tCBChhYZZZAZ9wopJWQUgEadBeU1e8orT/uINMQx2QmmKkjkRcDOAoi0RsHuCChoBXR1aJARUmCSjPSIykHc5Rt82b9TKkWXaqnuKTAnTnguDFILMTYV0049JXJywDmJYn9nPSSrACkRKf0q2sx/MQDosl1yInKlJGaSGPsGKQeXd56MQ0SfwgQ+YM/GAjqgaFHqKBgh0Kc+Ex/rpqvQerMcH2pRTWpSFRrRE9IJHOHiBXnsEBxxH/qGb9vvlpNKDCCVQnwlksVN+5aBmC0NvTiS0WUuE9d7OdkZTfFohkNnDT5gvYZpoZ/CtaUsjmxYcP/MqMbzb/c/uP2H7f/uP3H7T9u/3H7j/9JQV8+up5Dtx/Dh4EhokdffnVY/x19OQjk4eiL/OroHGI4fvzppHu4ynDhQADeR1uOhi6hyrL2h1rW90EhHWvpb+bCcrThXo6pkuVgS//o4KAoHe4d6b+XC6ucQUpyrHg7hZxByodLvwjqa6jS/qnVm1e02J9wumRwEZvlN1c5m+a/ESRKWxJkUXMR4IJ5TCsOAqQrtsytpFRQfCDyMguyuNhSYVFt9MMuqi2VXx/1PPAQ6/BuT8ncmsy9I5c8hnXjK8INRztwOHBoE6Nt/LJzP70TDNcAirN4utQreT2wQE7qE400ZLvAqIE2yEKjDXYo7oe1lIin2frrs9oJOHIHfDTK9cDR32OlE1ph4Bfi7L0nwNvZJ2IQZXcP9BrhSW6U5s/m+9w8jX+nEnzfemRhyAJ8mbaNn65qIL1aQ2Pboc9vftoD/kpWi56UDvjGs6Pl7vjNiDuUrSD0bg95vwShJ9c1LGuYuxbPb85DRNQ9WUjETcjgwLc25ZIHQ2eMm+Mk5+LAYoSDB0EWfVwFLsKEj1azrCC4m5DVsuANaVBH7sle5I2NO/wLFx6uKdhnsCdQd1PVAoWvEm+zeK3VYoO6FpGGy4YtjVjAaQGLHgsJvoTwsHR5gTjRYS4RwM3pCAjcCZ52/u6oKx8yYq5YDHOXm4Mimt+3XgkLp7aBiD0ERcALz/+AgQ5tQCXk5l2sO6O/jhXDT/osBhg9CePEvAUj3lCccfMmESPrBgTZYZZFhUpthHNO9dZxzWyDIHU2D8jDQi9wVZho5DOkEH3F9KMENHoQeu6bSsT7uaH11JvHZRHuxuUDKBy8oG7u0ci/ZCfACAdyseAJz78g9GFBdsOFUyKOoFNBYRoRQ5wXLK1mXL15kJC3dOGUmwsBb0UcQTzYN2hCRYiOaqcTf/eu0hH9hojIE0R0xU4yepKYzjiCxq7bkrnItQyoyygKk9Vuw3jhvksNma1V4hdDPW5vBshuTmwndC6Walc+1GXGLH9NSkap3vVCUil+EwvsRHhwkUGR8hbmf9vWi9ZK3TcHFHrQoE4AJl7d5d1dKF/ysocXv+HhFFOQeSh2EuPXWD7wKMkyy9dMfoniA8l3Towj6BeSpu6lxP5sISw/KbFYkGx0ISed5+yOFM6Bd42kNv840Crc4T57iBQsKk5281/k7/omw/dEPwQ2H2zCAHK2J2HtVyG/xqfY2idlh8cYUJj06Dt7Tz5DbcNHyLYStoCn5EMpWa4u9LgCW6h0a3OvaczUygeeIb+OHDoYOKVkq/DonQqM+1/SuAP5GkZfvrPe27gD2+4tHKM9PaowTyiJBrVLOdYDe7RRn86qwupClug1JvChQubAdxBsCDVJC7DUkvcgrgZVUK3ZGtVtoZH8lyrvgKj2djCaRMOtuKgLD1PrcQjpgGS3B8yYTHPFUDUPJlEEqFjzYTYCvdBgGPEu+GFTkzgied5soeiiyaXHlt+jvAP8hawtan1+ij7FFisikXVSg86iUsQAloUnmWEUeFbSN9wuVk/ag4hR8JvhbQlBT2ChhZrLlmJfmfQ11xzaeGjlUIZ5YHsBtz5D9DkiS//HVzyJM+c0wa7k20tsht/0scMpnF2aobhQ4ELy2ygy3mww4tS11BM70IqZDRVLWxH9PzcJRyFa1V7gMef0mNT7o356rP4kGrwzqFiSWHPuOFL0RvkWwEEctxfUM5LfIQ7dIozQHmoZPagbit6rvl8kdREUzsKrgSXeIoLfG1R3GBCi/VFMftFtF7B0u2ihbL5Q9G2u1WDXzRgFv+yLHuGBcLenfvPJPVkwOrbxnHdorTNLQc/DbMYSwmXetjyKygutGDbcCx9S+qatEJk60bIw9G8l7wKyqIAhC9kOZVJGWPftnVeXiYd4BOKL6YG0Yir1lmp+VQqiR7aiRP59vUudea+1phbqQ+8T5I1ruV3PoieKloRJG1px0j/cpDCEbG+nAozM8LWD+I2G/Mi1YMVHC7Ynoj8AmGdj5K/dLO5xy5nVUmBw0Gw9kTR2mamI24b2obRDJW9MUoCku/fqDSv2KZgJKTnuI2+r4GaNySLgBqf+45KeCP/3E6L4mCiFF5hJVnhEe/2r9iTDk4OKFCw4p9Uo37H4Aa7FOIRVk982qfEXEHegEeu+GZyrfK0TblSp3FkepF6iOvmQhwDhXfKeSKUdrYqeN9uzHUcnuSsMUXyDhfzvw8q+jLjstpcXh8h3eMz8zs1P3UE4ZNiwxcooKIR8kMGqPIbhQjzwRmSldrRRsu3UR72halplJP37yzug2/3Yuzru8KehyA/UNvYqIY6Zpgss6p6ym70Eh5dhnDh4dMZ8iGc70amDu+9z37q8NO7gbJ134OFV738vxY+r5DAjwSIEJBxcAOw/5nGDVJEYL6EaX3gT7+HaE89R5RZO9U+F0ZX2q5F34Mfh3FBQx3Ev1LHXnlZjtiEvAPgt4zpIxq0FYLUashuxF1FR/fvMO/D3ppbwYgETDliRkJTDwUZbsHCaVRkPekyqtwA/hpxwXphS/2iHrICOiiEfyXkH7PjTbb50LMh8JHfsp1dLSUFGzAq3xD5IGEy2qSlyQK5iDy5rKKc3siSm3tqo2Q55MlDosyfwkZx3QPaX/D4SjHgkt+TbqX8ysIEQERhPUCQfatx/LCcpkX9S4mdkQzmQ3Ry7qglPo0w/lfMOmIO3CVx67dP9AOfVYGoNqf3oQtasvO6A4gLGlq2Tt8UhIztnIjKjkleVfEhKfgphCjKOXQW/xbUo3Q8+/GOpntHamZQAcPpjoYIm0YmbFjI78GYr/zuOO5AcwOrn++ySz+W8A3Ja80m3RBihfKPF+6UDeC2/Wa6p8q847lBsZZLB74JP5rwDGg1GWB9Yiudqhd5+aYv0c5DWDV/6b5l3QPR7eYbPZdzBuEHfx7MZot6gTCiJ2EbPBN/b3rO8Ax/MuANZ+IV/fFRZZFeErMIphZ8hH8a8A29g3KE5uBO6Q7ZDSt0qcfTq8CcP4Hdv9X0UcsXMO9glJAML4Ad/ngJGXSexHHF1nnTfT33F0O/yuAPdpQpWsapEMi8ZQAh0+ObK/yarK/9CG8GvJRyV6HIZlEhIB1v8d1V/pOMOYlOL8YsLGIHxdIG6U23x2PUhFOtvwLyzkireCKsssg9SRr3F1H17tTyld1CgIThwRx7e2cDHdycw6alpwdy6ETNRAxIP5rB5atW7OIicib7epjxYItqZouEOEKBd7AIt2FxA8SluQbt+MPMOqON9Ch+zBK6e6XmL+g1PwPNZzIhRklvhhtlkpRrZQ/uKhxDZjgiHSeWSkwJVBY6nX0AnkplULi9JbaPI6+THVbATPSnnfo8lOZhwcMZ5PXFzTGPFG1jS+dIkbnZVDkoDIhHlYGPKcqofzLwDAz9OGVXwQU/duHQ9b/Hbq9WzseMhKmJJXsQDDfGrLi1m5t4hbrE9pmQ/afoKcAgazTfkgE46sfeeBXqEH5qHdgEncnXI3OKmq94G7IYJE09SRumkeGPdFgW44dEcJrhITDABnLOD0WEVeo7q/EDGHVQRBs8bDAg4HtkZ+S11pgG+hhEe0StkjUjVC9CkkugF2hJK8G5ZPwDAtUVYIobYbQM6k4drSMj7qkWbgFNzms0x2JJKL6cNsAQxiGKWylxFHhHOJuQuSZ9V/8emo3sfxa/TfF2Gqir4ADahD1uU65uKx3RdY7ctpYg2RwuABTjHYlpn7CGVmbUBk6sQBZQRXnaZGtYNVnMBkXnIa047UwkNsFSzfHGcBFjNOcnI/NsjnoMubELr7RLATIyrtCfN/SzGHTwPMExV9VUEHPsJ3CLc6duv1iP6aG6lH1oeHjtIRIQbFjam0Yldut+EZrIEkGgBxRUqVEW8goOoOAHHGWfEwXni9AAtBjQmFOUTDs3KjA2j4HAQkftC67T1dtqWD2beQVkPG4QjdpzkQT/v7sHvKx6qGRj4UO2NkfeZhMCkf+yCsFmhd1Eo1MOVQMloToCa/4sLS5LqwtUfPqYcI6sz0GipiSShgzGTfSFoihJISoaiWAB39fATHZG9owsPb0qV0jtxBI8ZZKMPpzpDAjZMgIcKMe8G0NGBljKH6+ZQmgZ4pNof7MPHFZFFIgaHWbkzhrz4FvnY6PKleYf3/oJbbacPqnTfXXD2byjma+WIpmZZhMXCWeJltyQIh7blXvG2sWESKcDCNI9Jdf6bbnWgT6TntPpTBQzaW2G4NRVvSvUB+pUgtbA64ANthWooZpQXN+ywm6Qed/A2LnRv0fAcNONoPljysTyRTd326+FUukw+0gNG2YqdPC1MweosPKqHQDuS0SXo9SDTioPmRBBVA06FfSeAxwAzoQwaSbcHXsoikApwaJ4nzlniNO9fmHd4tyZS5rEM19JUbDHx/Veb7ZhqjzZKcY0hCAtOfGgB8z3vLB3pgl0DPYzzpoAlAtjGhIcX/XK1lhM/JtPsqXwGFRn0mmQ6tB1IsRrmXavUxV1IDbdORGj3Ef3cTdnS8Z6KOfMAHomWwNPAapGniVIsfvDpgYj60wDmzddp5BdC4GxTXEcMtDzrQNMQSHXfVAeWhndp30gT9rQkhGhtQNvXwuscWERyp7VRwqDE/QfPOyC7XsX9gA+qRhv/EUKxXFjGAdG8zQplPDX8nJ9hXQN5wnmjqDmI9UHBRklFkM9Npvhd6yzo3UzZoqcfSVB7l6vYBULm8ceMiCxG2r3YUUofYzQiIuXu60xM3lO1YE3ySriKcTfwebrE5Ewlx5eJnecPOM7h3OYmulpoCRBZ+rzxYNp1lVSpRNFFpDygktRVwzVP6uDXgDCQ9Ob9Yb/slsWhr5t3YLcm2gA1QLBuVt77QznuEAoKWNfIjmsSDa8Zsar87R73Dwohmye8TfAyC7wEoMW9wMIjplPp0WPVR+gUXBcru+eU6XqwPwwIIU7Mc9swq7bpItp/al3ptnR4f/K8Q70VA5CKEITHiKP3b9zB27oatCEXXb+Nw8CqqTSoDdkiFgaBLo/wzcCglkIUBZdRNR+XUSQV1TBuAt8P0KsWM13NbrTA170pltXfyplliO1urn/yvEOyWnUff97id1Z/csT/Xc/CTTurz8MlJo+iqwrCyXc2SxRdT6K7qERkQ2YHbF2t1aJc6JcepjpyYpkH8QA7N/lFwrnr7N5kMpe61B+t9S+bd3DHpupHoZPrcymuWhR3YXE2TBFZ9DXsfbLvB55QhjXXWdzleIuB7AI6vhCip0BVRCip8ur1bdQibbQRrgsF7QyaATOqVu8ZGbXu7cgfPe/gKDb4HHH0YVT3IfMLWPThQKqq9V1ENzA7N4du1y2SuEGL+xAaxAadWISrV+pZt04foEfNi3Bg/hO15uhZFnAbMPgtrdW6KiMQx/JV8w7v2uTnSSfXdxcfnQUCX/oXzvCMrfli+nhlSfLADLdEnCwJMoeCjSLdB9Oukp4UBjGlggiZ+GAjW1gwqW0pdssfLHgY7oGtJ9oNiDj6k9Qiyk/gp2LwdC3ZtKWCakWpYPPAlXg+0ikNK7m+wO+JVE3hpd1Q7KWa3gOhd3vckVzyM516/jGpP1hQshHbxdoJletPEnLCaUi/nQi8SlSZEVz+oaGEK1wrX7mEZgst+nFqrph6wNEzQHIANsGLgQTFlSJRhJVEUqoTLdjyOEPhhLb9pgucJtgv2CilvJfcsF25oNcdo+LGh3LcYXXgnYyRkHnKu+Tf/0lqWRme75T4yO9bYicVrEJ22BN2kyuweIige9UEJAu5DkC/efZnOgadVM5+D80jghJpRdkIZA6t1mC6A7+dnHa0iaMP8UMt5VDtDSvp6Y0Bo7izm+tzOe4QT8pw1Oniv+Y05H+UWhX6jgSOdFNRwVF7etNYlL+jQ1KkR2QQufWhFY5m7D/3E7UjBZ0c2pzwrsJuFUPDzQ54Wm0xFFuUq0h1HVspz4FRZPIGR+0F2hDI1o70OtCjELaRqIV8jsYd7usihCuTjRTfoNI/QP1it41FO3tPRQgMKk7kqTwiHgS6KXh2YQWQ6xaMOB0RRnQ0BxuAiDRMyrhPKSo2uZ5WUGQjuD8Mmro3INq4bSJEOAxP7gV/NGbc9Ly/o+L/rvdSHCzTVxvykz/29xevg0YL0ShDrCz0rA/cPJ5QjB47pYh9IiZGnB6KLpUJdKCbyh4/0JDELNI+SZ/kVVSIsQvC0DSMVoGFHe6xCO7Iw9JBltlNRnjuuzY/m2rQiezjp1hMVfTvHnfwKqDHF5P/5ogoKBIBdgaXDvdntaqthxGLLXj0sLeMoXAr6aAYdTpiGkaCri5dXj1UMSCxIdWHxgXR6iBm/uWggozPm+y4gfQk6W3BAZUHJnV9S+MOBHJHK7b1Gl8+XpKhbj0tslLXj3urtsxbFSsokC0lx4oY0VfDDRfc0Iiij7JkpD+iMG95lp6o4T+g6o8VA7J3Q+G973YQR71te0HMOH3WqLiObDL01ibTiaMViD9Ggo2BIeTaYo6oLz65xRqHSkEx+kGntG8K0XpnoSf3C9ObSa/TcHZLy9/TuIPuxS+y9kAG2eyzgUEX6HQ6cCKIKS8luDbqYiooQFV3lbmqZfIj0DzJbPAwYO7W4nBxF9gHPc2vO+stz1k/oJDUyWUUcRDOm4XT9B+mPgjHT20eYIllpx6IbP4zf8agXtJaTR2Bp20c9a9AMulUC7HqSyxC6qReZaIJ2WmW9+IXtdjHBWLoXMo42k+/UdMTSkl6m/WGiv/DUEGocS7FWONzFjX7HuyHkgMOfz2M6lfoVSqyUAc3avnhFHaxz0GimPcWcBO4OW4uzHpVMp1Yao3c0no4UrNgaj+fkInTDC0KOCy+QDX+KMGX7qVzwSVd3+j3ejI2ZJaC9URnmUZ8AVEbaxXDOGuuMicUBpOVGNABprrQsPIMcZuKRkuS1FkOC5U0y5jCnuguO1LRNehChOhpZlVMBQ0akjzsek8XHvMOSJIBKpiZUqA3XeH8+ZTg2ghdM48MRgzrolPdLp5PHJRbFeGm2PBkLZZaLywhNmDdrKFZ5h0/mXiaZPQ4gEYgs8rtfRx38K6uBnFfD2ONNe8rHohofUYUArjM0xZBMzEcATqlKNTlDIAHc7SkB4DJZ+q0ATNQ3Lf5gNlOZqn66YSCby4nGcUfuAS5vj6neQeCGo5OC25rPtkZFL5L9FsLCa6xqgBNbvcCBaUlwAJ2NjYjhEPVgnzQirrXKpnhIOQsZ+y6gtOzaeqnE7Ohjr/h6+FiBD6UeQf+n3AS5QIieCxxZOfQmgwoVmBLNAIaGymoLbKl6GGhrJKLCxbzbJONds04vVE2lwNyNHMrxYybdbZx/GSimGR04j+Ufxs3njb44RRRYM4xFNXQ6QsYWbkYlTcwz2p1ehBSJyFYAAoY6jLaN4fOX6YxTm6qqpvBNuOs+4/96S52r667hDiyDv3RhNQRR3Rfx/UKKlu+7/MOvobOO8J5B+jyWVENLkXyqBoy51UJwdh2zhsYVmYMeLgk16LoMU9lTstHgzW7qaT9PPRzXYOdmKRXO3UFRzNofv5I4vkCY8pkI5tSkEBJ5ETcqvdTyFv5vKPo7KX0azbGc4eMYTiNCOeKfuApPQkpz/bxQCHZNiHAJ6BR1K4ztD99GghGBzLp6HI3PJlyM/nJhDleYgAmHMBtdlmPKrf3T/23dd0BxaV9vTe+mdinDV1MwPcSTEKId2bVuyfkXnpY+BSOOxTe4vHDuqN8OTLlBmP+YHEH6scgaUEODnhIk67b15O728UrCZQh1kOZN9M0ToCAzzyTuS54VlpAJqkNomS7mJRpZyiinVPv21O9bEFw+0iOO8QpUWRWA43pafK8oAr7uYTCPhEYm6gRJUbZZcL7MtpR7JkCCsib4K0zgQiPLeiYMa5doQEmEjM8MaLo3rqgl9siygIdZbrmHisP+UCOO9TUSV5gjNMW+VQJKvdapT+XaJ7I/WQl0Lm0VTvZWGZ74rx0w/rNinhzE4wx4UkJIq0VXDWjUk2xm0mgv3bflNC1wWRWNUaLad05nQGMWPtA5h2afHlh4F5wi1Zejgz9oOgj+flEOxZe4SaRAaEFFl78h5SThbq+LHyuaQU1ikGHEFICBpsBxwEiYvUK4pxqXgGjd/s8qiGD35TUVeONikNCzmPwQej3XG0/yEG/Hiuil9WMlYMPC5y2mdJNj7qaa5SiUVPLbqest0AEGSxc0CdduDABk3cJ8GxUDzKKiP08XndgGXHEw42WXgyjVQpa58cTo5c9pHT1UXXQ56IyBJYKqPxQI0AArd276aO4XhQRIKLP9M2PTm5wQNpZa7ypd5HHp4ys+sxBROVxIgAptqRa81EtX5+64yMZcJTKBufCupLXxJkIyxHIP5Y6QB0Te1GK3YaXDRO9LGadji8a3M1pZl7QCbxp7jj1VhBY9NBT5sqCgbWTbFmg1U12lBjZyDECVrUG10zypsYdzIjgHiSp089rdh4tOCNBhKHDY+XExjwhHTlDmzI/NXhzTCdnyvgfRruylVt85dKCNVW8EdzgTyWcABmCO9PxbDx8uAIQzC0CnYgbKcDJpuVt88VFuGYbJjIouG6t1MVq3ZNSWCLrF+olroWJTJIUJUp6U9UrvyO28CjCLGlSo3OOEuSWocGtRGEwbHhg67Ofc7AgGp36jeaZyvp2kP8u2t8rFiEHZ+sLvZK8eEAlP5JaDniSsa18UXIKCn3X07uvw7lF+i4onH3OGfAa5SQBodeH4iBhW5tTGvd8MKItaNrNj6yaW5Piqv/X1uEsKmtJ4g1doIqhrmjU9JSZQXjm0a4GAtFCKLs2HE8dcCQrLWW/ESzkolmOjgRGSM7pHh7BAceP5FZAjzktpUrejLhDecvHHfQMK5xbeQ1WFWgqFkokfT0ISeHZV8zi7ENxIwFXJlGWZI/O3APa1svnHfjSU49Uf+FXC/WTmHfo/3GhHW7lpXl7F18OFv9v1wsRN604ToCqIc0s7O0Q7HGGB0d6XvndAd32sneqP1jx8lHMO0ALmA6+yPbusP6BK4+orkhzUVtxGGaHpwXum/DW8xgjHCHZyZvXH8b+01+rnZTkoB3f3sZxh2/lwhjeHNePOU4LpU6muemaEGNkHxMKzZaKQ8L5gzWVzcczCHqEBHUy6Ytvc66XK8Oig7ml3j5/YoUo6g78BFKh+bxjgcSTA3Qsb8oSAmqx3YUBtzyUPPuNzk8pfngD/ACqe2a2ed+68n/sXFyWlWSEHGSsOZTSwlPKeGXOEyiMXE/nQMqPQX8xnzvAR1H9vXuOfjH0r6KWufeeRaDS2GtIxaELT1hyjZXkbFOAw4Bz1M4bZJraPLHIpRtuEynvm2G98HeVd+Ack2Rvc1BQ4YjkgClIzIGD4mQTfGlLAwIVNXC0oidnnFgUpmz1GNRPg8Qs7Xjqp/m7yjtwjs3iDccG0ACDcXM46FOyMTvRARjEguM0ZkO34o1etSoc7YAQ62mLXuiNxpnvrPh/+fwImJqURh8vKizA1LE4kkmpCbpEmxqFWL0XcMNQFrpuZcTq+vejvv7y1XgVAS19zcYJZeNmIpOrCySE4e4rPEhkTHanC4tvyxvCKUGLnhyo0a+3duEzJO6iBYmjzAQnpStDUD1QfSBiPdzAv5y1JorV6SrJbhYsRPvKQcnv57iD9WIyD9HTsIEhZnJwpYyCuihQbcJlDwB2RrMZC3CagJ05HCoBWEm9BfiiN1NcS2bMDWyYlLiYrBIiUItrlYnA7gPJuetOWIGIgCQGq+1Me/ykFg8mPMN+LdXHo0i1Jv2E74oD8ZHhTKH7ukQuA3Z6sgECMGLC3AL9ql+5JFlsx3D9xYg7+M+Jx6tsvoM9v0NxBz+1u/3ViDsYj/S4g5ZhvmJwEs1XDtZL1tws5h38CTB3107l0iW5/OcdXDpP+WNuvnawHmw5KtJjLemXWGn9/+/oXX/cOKLjDq4aNH9aTyQNf/RobA00/hGg8QCf95tc7CI2y28Oy+EXOQSkwy+/ue+unPr5w0ddK/aDePOYGIJ4HiAH4tz8y+0/bv9x+4/bf9z+4/Yft/+4/cftP27/cfuP23/c/uP2H3+q0AE=)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fKyq-NgFefju",
        "outputId": "fe65d040-92e0-4728-db78-67081d31afad"
      },
      "source": [
        "from gensim.models import TfidfModel\n",
        "from gensim.corpora import Dictionary\n",
        "\n",
        "#now let's fir a TF-IDF model on the corpus\n",
        "model = TfidfModel(corpus)\n",
        "\n",
        "#apply the model to our corpus\n",
        "corpus_tfidf = model[corpus]\n",
        "\n",
        "#creating a dictionary that will hold a word and associated weight\n",
        "topWords = {}\n",
        "\n",
        "for doc in corpus_tfidf:\n",
        "    for iWord, tf_idf in doc:\n",
        "        if iWord not in topWords:\n",
        "            topWords[iWord] = 0\n",
        "\n",
        "        if tf_idf > topWords[iWord]:\n",
        "            topWords[iWord] = tf_idf\n",
        "\n",
        "wordimportance = []\n",
        "for i, item in enumerate(sorted(topWords.items(), key=lambda x: x[1], reverse=True), 1):\n",
        "    wordimportance.append((dictionary[item[0]],item[1]))\n",
        "    print(\"%2s: %-13s %s\" % (i, dictionary[item[0]], item[1]))\n",
        "    if i == 100: break"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " 1: trees         1.0\n",
            " 2: system        0.7184811607083769\n",
            " 3: graph         0.7071067811865475\n",
            " 4: minors        0.695546419520037\n",
            " 5: response      0.6282580468670046\n",
            " 6: survey        0.6282580468670046\n",
            " 7: time          0.6282580468670046\n",
            " 8: computer      0.5773502691896257\n",
            " 9: human         0.5773502691896257\n",
            "10: interface     0.5773502691896257\n",
            "11: eps           0.5710059809418182\n",
            "12: user          0.45889394536615247\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ycPasfBeQs10",
        "outputId": "0c565f17-4059-4ccf-806a-332e38bf5e96"
      },
      "source": [
        "model.idfs"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: 2.1699250014423126,\n",
              " 1: 2.1699250014423126,\n",
              " 2: 2.1699250014423126,\n",
              " 3: 2.1699250014423126,\n",
              " 4: 2.1699250014423126,\n",
              " 5: 1.5849625007211563,\n",
              " 6: 2.1699250014423126,\n",
              " 7: 1.5849625007211563,\n",
              " 8: 2.1699250014423126,\n",
              " 9: 1.5849625007211563,\n",
              " 10: 1.5849625007211563,\n",
              " 11: 2.1699250014423126}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b664MEV6Qi2c",
        "outputId": "0d530cf1-28c0-463e-8bfd-d245dfc9cb43"
      },
      "source": [
        "corpus_tfidf.corpus"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[(0, 1), (1, 1), (2, 1)],\n",
              " [(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)],\n",
              " [(2, 1), (5, 1), (7, 1), (8, 1)],\n",
              " [(1, 1), (5, 2), (8, 1)],\n",
              " [(3, 1), (6, 1), (7, 1)],\n",
              " [(9, 1)],\n",
              " [(9, 1), (10, 1)],\n",
              " [(9, 1), (10, 1), (11, 1)],\n",
              " [(4, 1), (10, 1), (11, 1)]]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ak1eK7KLBxm9"
      },
      "source": [
        "By creating a TF-IDF"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Please forget about the Pushshift API. I will keep the code in case the API comes back to life (doubted). **\n",
        "\n",
        "**The dataset used in this analysis is [here](https://drive.google.com/file/d/1JAYNVL1mVI-a2EbDKvRl0U0-xuivQ_-R/view?usp=sharing). Download it and then use it in the analysis. Basically read it as a json file to a pandas dataframe. **"
      ],
      "metadata": {
        "id": "lUfrPru3YeqH"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2roNlIxWsKLO",
        "outputId": "80e36642-84ea-4b70-a36c-f4554fe94249"
      },
      "source": [
        "!pip install psaw"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting psaw\n",
            "  Downloading https://files.pythonhosted.org/packages/01/fe/e2f43241ff7545588d07bb93dd353e4333ebc02c31d7e0dc36a8a9d93214/psaw-0.1.0-py3-none-any.whl\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from psaw) (2.23.0)\n",
            "Requirement already satisfied: Click in /usr/local/lib/python3.7/dist-packages (from psaw) (7.1.2)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->psaw) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->psaw) (2020.12.5)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->psaw) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->psaw) (1.24.3)\n",
            "Installing collected packages: psaw\n",
            "Successfully installed psaw-0.1.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tzCGcpPksTTr"
      },
      "source": [
        "import pandas as pd\n",
        "#we will need datetime in order to specify the timeline we need to collect the data\n",
        "import datetime as dt\n",
        "\n",
        "#now we import the wrapper in order to use the API\n",
        "from psaw import PushshiftAPI\n",
        "\n",
        "api = PushshiftAPI()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BDNT8oQHs7Uq"
      },
      "source": [
        "#this function will allow us to find the last day of each month\n",
        "#for example, there are 31 days in January, but 28 this February\n",
        "def last_day_of_month(any_day):\n",
        "    # this will never fail\n",
        "    # get close to the end of the month for any day, and add 4 days 'over'\n",
        "    next_month = any_day.replace(day=28) + datetime.timedelta(days=4)\n",
        "    # subtract the number of remaining 'overage' days to get last day of current month, or said programattically said, the previous day of the first of next month\n",
        "    return next_month - datetime.timedelta(days=next_month.day)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5uVu-Sz5I3iL",
        "outputId": "67529278-1b28-4339-e0ff-0cbc82dec363"
      },
      "source": [
        "\n",
        "\n",
        "import datetime\n",
        "subredditlist = ['Ex_Foster']\n",
        "for reddit in subredditlist:\n",
        "    for y in range(2019,2021):\n",
        "      for i in range(1,12):\n",
        "          file_name= str(reddit)+\"_\"+str(y)+\"_\"+str(i)+\".json\"\n",
        "          print(\"starting with the month \"+str(i))\n",
        "          print(\"for subreddit...\"+str(reddit))\n",
        "          print(\"setting start epoch...\")\n",
        "          start_epoch=int(dt.datetime(y, i, 1).timestamp())\n",
        "          print(\"setting end epoch...\")\n",
        "          last_day = last_day_of_month(datetime.date(y, i, 1))\n",
        "          print(\"the last day of the month is...\")\n",
        "          print(last_day.day)\n",
        "          last_day = int(last_day.day)\n",
        "          end_epoch = int(dt.datetime(y,i,last_day).timestamp())\n",
        "          print(\"setting up the generator...\")\n",
        "          gen = api.search_comments(after=start_epoch, before=end_epoch,subreddit=reddit)\n",
        "          print(\"setting up the dataframe...\")\n",
        "          df = pd.DataFrame([obj.d_ for obj in gen])\n",
        "          print(\"The number of comments for year \"+ str(y)+\" and month \"+str(i)+\" is \"+str(len(df.index)))\n",
        "          df.to_json(file_name)\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "starting with the month 1\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 1 is 0\n",
            "starting with the month 2\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "28\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 2 is 0\n",
            "starting with the month 3\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 3 is 69\n",
            "starting with the month 4\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "30\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 4 is 233\n",
            "starting with the month 5\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/psaw/PushshiftAPI.py:192: UserWarning: Got non 200 code 429\n",
            "  warnings.warn(\"Got non 200 code %s\" % response.status_code)\n",
            "/usr/local/lib/python3.7/dist-packages/psaw/PushshiftAPI.py:180: UserWarning: Unable to connect to pushshift.io. Retrying after backoff.\n",
            "  warnings.warn(\"Unable to connect to pushshift.io. Retrying after backoff.\")\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "The number of comments for year 2019 and month 5 is 119\n",
            "starting with the month 6\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "30\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 6 is 183\n",
            "starting with the month 7\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 7 is 78\n",
            "starting with the month 8\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 8 is 80\n",
            "starting with the month 9\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "30\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 9 is 229\n",
            "starting with the month 10\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 10 is 351\n",
            "starting with the month 11\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "30\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2019 and month 11 is 388\n",
            "starting with the month 1\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 1 is 134\n",
            "starting with the month 2\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "29\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 2 is 210\n",
            "starting with the month 3\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 3 is 275\n",
            "starting with the month 4\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "30\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 4 is 279\n",
            "starting with the month 5\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 5 is 501\n",
            "starting with the month 6\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "30\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 6 is 185\n",
            "starting with the month 7\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 7 is 353\n",
            "starting with the month 8\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 8 is 217\n",
            "starting with the month 9\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "30\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 9 is 217\n",
            "starting with the month 10\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "31\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 10 is 222\n",
            "starting with the month 11\n",
            "for subreddit...Ex_Foster\n",
            "setting start epoch...\n",
            "setting end epoch...\n",
            "the last day of the month is...\n",
            "30\n",
            "setting up the generator...\n",
            "setting up the dataframe...\n",
            "The number of comments for year 2020 and month 11 is 100\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qNBcWMbjJITl"
      },
      "source": [
        "#this function from sklearn allow us to create a TFIDF vector for our text\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from collections import defaultdict\n",
        "\n",
        "#We'll create a vector here, and as we did before use the stop words provided by sklearn\n",
        "#Now notice the ngram range. The reason we use the word ngram is that the n can be changed!\n",
        "#So, for example, 1-gram is a one word...2-grams is a collection of two words...etc...\n",
        "#In this example, we're creating a collection that includes between 1 and 3 grams\n",
        "vectorizer = TfidfVectorizer(ngram_range=(1,3),stop_words=stoplist)\n",
        "#Fitting the model\n",
        "X = vectorizer.fit_transform(df['body'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HQBUwejMKLeW"
      },
      "source": [
        "#since we have 3 grams (1,2,3), we need to create a dictionary for each\n",
        "features_by_gram = defaultdict(list)\n",
        "\n",
        "#zip is a helpful function in python  that aggregates iteratbles in a tuple (list1,list2)\n",
        "#list one in this case are all the features in the mode, and idf_ The inverse document frequency (IDF) vector\n",
        "for f, w in zip(vectorizer.get_feature_names(), vectorizer.idf_):\n",
        "    features_by_gram[len(f.split(' '))].append((f, w))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mN1TkSA3KOd2",
        "outputId": "f1579152-8b10-40ea-f457-e5457cd2fde7"
      },
      "source": [
        "#lets get the top tokens for each of the grams\n",
        "top_n = 20\n",
        "\n",
        "#for each of the grams, we want to get the top features\n",
        "for gram, features in features_by_gram.items():\n",
        "    #get sorted top features\n",
        "    top_features = sorted(features, key=lambda x: x[1], reverse=False)[:top_n]\n",
        "    #get the top features and their weights\n",
        "    #now just focus on getting the first element of the tuple\n",
        "    top_features = [f[0] for f in top_features]\n",
        "    print('{}-gram top:'.format(gram), top_features)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1-gram top: ['like', 'know', 'sorry', 'people', 'family', 'going', 'life', 'good', 'want', 'care', 'foster', 'really', 'say', 'better', 'even', 'love', 'things', 'feel', 'get', 'help']\n",
            "2-gram top: ['foster care', 'sounds like', 'even though', 'put username', 'sorry loss', 'feel free', 'feels like', 'easier said', 'feel like', 'former foster', 'foster youth', 'go back', 'people know', 'said done', 'sorry going', 'wish best', '20 years', 'beautiful day', 'buy something', 'care even']\n",
            "3-gram top: ['easier said done', 'ever need someone', 'feel free dm', 'one year care', '10 years like', '10x definitely blessing', '12 best answers', '13 years hurts', '14 imagining future', '16 especially home', '16 matter wonderful', '18 held resentment', '18 moved another', '20 years ago', '20 years every', '20s man felt', '25 still close', '30 hard resentful', '34 still get', '70 horses boys']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 744
        },
        "id": "tvk8hpRpKxpx",
        "outputId": "91944608-945c-42ab-a4d7-167054344ce9"
      },
      "source": [
        "#lets get the top tokens for each of the grams\n",
        "top_n = 20\n",
        "\n",
        "#for each of the grams, we want to get the top features\n",
        "for gram, features in features_by_gram.items():\n",
        "    #get sorted top features\n",
        "    top_features = sorted(features, key=lambda x: x[1], reverse=False)[:top_n]\n",
        "    #get the top features and their weights\n",
        "    display(top_features[0:5])\n",
        "    display(50*'*')\n",
        "    #now just focus on getting the first element of the tuple\n",
        "    top_features = [f[0] for f in top_features]\n",
        "    display(top_features[0:5])\n",
        "    display(50*'*')\n",
        "    print('{}-gram top:'.format(gram), top_features)\n",
        "    display(50*'*')\n",
        "    display(50*'*')\n",
        "    display(50*'*')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "[('like', 2.2139231351791038),\n",
              " ('know', 2.570598079117836),\n",
              " ('sorry', 2.570598079117836),\n",
              " ('people', 2.670681537674819),\n",
              " ('family', 2.7819071727850435)]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "['like', 'know', 'sorry', 'people', 'family']"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "1-gram top: ['like', 'know', 'sorry', 'people', 'family', 'going', 'life', 'good', 'want', 'care', 'foster', 'really', 'say', 'better', 'even', 'love', 'things', 'feel', 'get', 'help']\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "[('foster care', 3.5356789751614235),\n",
              " ('sounds like', 3.5356789751614235),\n",
              " ('even though', 3.8233610476132043),\n",
              " ('put username', 3.8233610476132043),\n",
              " ('sorry loss', 3.8233610476132043)]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "['foster care', 'sounds like', 'even though', 'put username', 'sorry loss']"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "2-gram top: ['foster care', 'sounds like', 'even though', 'put username', 'sorry loss', 'feel free', 'feels like', 'easier said', 'feel like', 'former foster', 'foster youth', 'go back', 'people know', 'said done', 'sorry going', 'wish best', '20 years', 'beautiful day', 'buy something', 'care even']\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "[('easier said done', 4.228826155721369),\n",
              " ('ever need someone', 4.51650822817315),\n",
              " ('feel free dm', 4.51650822817315),\n",
              " ('one year care', 4.51650822817315),\n",
              " ('10 years like', 4.921973336281314)]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "['easier said done',\n",
              " 'ever need someone',\n",
              " 'feel free dm',\n",
              " 'one year care',\n",
              " '10 years like']"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "3-gram top: ['easier said done', 'ever need someone', 'feel free dm', 'one year care', '10 years like', '10x definitely blessing', '12 best answers', '13 years hurts', '14 imagining future', '16 especially home', '16 matter wonderful', '18 held resentment', '18 moved another', '20 years ago', '20 years every', '20s man felt', '25 still close', '30 hard resentful', '34 still get', '70 horses boys']\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'**************************************************'"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    }
  ]
}