00-Python-Collections/01.01 Data-Wrangling-with-Plain-Old-Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7e9f4107-83a1-4cf6-8340-9f9454b57eae",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Data Wrangling in Python  \n",
    "*Exploring the __MovieLens__ dataset using the Python Collections module*  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58653967-42e8-4130-a8ad-905eabbb54b8",
   "metadata": {},
   "source": [
    "**Part 1: Basic collections in Python**  \n",
    "  \n",
    "![Data Wrangling with Python Collections Module](./../images/data_munging_00-Python-Collections-01.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f21bef00-086e-44fd-b516-01016a08432d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### <font color='green'>__Support for Google Colab__  </font>  \n",
    "    \n",
    "open this notebook in Colab using the following button:  \n",
    "  \n",
    "<a href=\"https://colab.research.google.com/github/shauryashaurya/learn-data-munging/blob/main/00-Python-Collections/01.01%20Data-Wrangling-with-Plain-Old-Python.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>  \n",
    "\n",
    "  \n",
    "<font color='green'>uncomment and execute the cell below to setup and run this notebook on Google Colab.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "cbba6f93-2b10-45bf-9975-a41d30ffd219",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # SETUP FOR COLAB: select all the lines below and uncomment (CTRL+/ on windows)\n",
    "# # Let's download and unzip the Small MovieLens Dataset\n",
    "# ! mkdir ./../data\n",
    "# ! wget -q https://files.grouplens.org/datasets/movielens/ml-latest-small.zip\n",
    "# ! unzip ./ml-latest-small.zip -d ./../data/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "868129c9-cedd-4513-997b-53f9427b6f1a",
   "metadata": {},
   "source": [
    "### Get the _Small_ MovieLens Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67afc9a9-31b8-483d-89c1-699867c8700b",
   "metadata": {},
   "source": [
    "We'll use the [small MovieLens dataset](https://grouplens.org/datasets/movielens/#:~:text=Small%3A%20100%2C000%20ratings%20and%203%2C600%20tag%20applications) here.\n",
    "\n",
    "Download it and unzip to the data folder under the name `ml-latest-small`.\n",
    "\n",
    "This dataset expands to about 3.2 MB on your local disk. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4a045b14-f578-47ad-9972-a3f5aaba0911",
   "metadata": {},
   "outputs": [],
   "source": [
    "datalocation = \"./../data/ml-latest-small/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0428c276-5ef7-4bde-addf-3bb79337783d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify file names\n",
    "file_path_movies = datalocation + \"movies.csv\"\n",
    "file_path_links = datalocation + \"links.csv\"\n",
    "file_path_ratings = datalocation + \"ratings.csv\"\n",
    "file_path_tags = datalocation + \"tags.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef2dcf83-2665-4c52-9134-39afdd9a53c8",
   "metadata": {},
   "source": [
    "# Starting small: What does it look like? The Data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c30d8a75-d0e1-4b82-b419-fdcfa503cb31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# old school print 5 lines of the file\n",
    "def renderlines(file_name=file_path_movies, numlines=5):\n",
    "    file_data = open(file_name, \"r\")\n",
    "    c = 0\n",
    "    lines_limit = 5\n",
    "    for line in file_data:\n",
    "        print(line)\n",
    "        # condition tested after printing,\n",
    "        # so at least lines_limit+1 lines will be printed\n",
    "        c = c + 1\n",
    "        if c > lines_limit:\n",
    "            break\n",
    "    file_data.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e1d229de-29ca-4411-a1b7-0c3c5cd4b1c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movieId,title,genres\n",
      "\n",
      "1,Toy Story (1995),Adventure|Animation|Children|Comedy|Fantasy\n",
      "\n",
      "2,Jumanji (1995),Adventure|Children|Fantasy\n",
      "\n",
      "3,Grumpier Old Men (1995),Comedy|Romance\n",
      "\n",
      "4,Waiting to Exhale (1995),Comedy|Drama|Romance\n",
      "\n",
      "5,Father of the Bride Part II (1995),Comedy\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# movies file is default\n",
    "renderlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e754afd7-ad20-44bd-9240-2d8577d200cf",
   "metadata": {},
   "source": [
    "Notice how the first line is made up of headers - names of the columns.  \n",
    "This is going to be important later.  \n",
    "\n",
    "There's files without headers, files with headers, both carry their own idiosyncracies.  \n",
    "\n",
    "If the file has multiple parts, each part carrying the headers, you can use the headers to find out the values for that column - this means the order of the columns can vary from file to file.  \n",
    "\n",
    "But if we do not have the headers (and no other indication to inform us which column contains what values) the order in which the columns appear in the data files cannot be inconsistent.  \n",
    "  \n",
    "This means you have to decide different strategies for loading as well as writing data files with headers and without headers.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6960b93e-93b1-4e3d-b3b4-7928394d7210",
   "metadata": {},
   "source": [
    "For now, let's just try to load more lines from other files.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "411c845a-bd4d-4522-b9c8-2630c18e768b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# renderlines(file_path_links, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c293c474-0cd8-43e3-956d-59cc4ccbee37",
   "metadata": {},
   "outputs": [],
   "source": [
    "# renderlines(file_path_tags, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cd8e2849-2d59-42bc-ac79-199a7dca7e62",
   "metadata": {},
   "outputs": [],
   "source": [
    "# renderlines(file_path_ratings, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90c84a07-f77c-422a-93d6-948195f63db9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Outline: Learning more about our data\n",
    "\n",
    "We'll work on building some intuition around our data with the core Python features like ```Lists```, ```Tuples``` and ```Dictionaries```, then explore more data types in the ```collections``` module that provide additional convinient features.  \n",
    "  \n",
    "1. To get some more intuition on navigating the movies file, extract 5 titles from the start of the movies file and 5 from the end of the file.    \n",
    "1. **How much, how many, how big**? Can we get the different markers of size and cost for the data we load?    \n",
    "1. Is there a way to do database style access? With ```movieIds``` and ```titles```? For e.g. is there a way for me to do a ```SELECT * from movies where movieId='206'```?    \n",
    "1. What is the frequency of each Rating? Or hHow many 1.0 ratingscan be  found in the ratings.csv data.? How many 2.0, 3.0 etc.?  \n",
    "    1. Also, what are the 3 most frequent ratings?  \n",
    "1. Is there a way for us to find out the he average rating of a movbased on it's ```movieId```?  \n",
    "1. Finally, can we find out the number of movies in each genre? How many adventure movies? How many romance ones etc.?  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35aa5acf-5afc-4392-87d4-1dbcedc87d74",
   "metadata": {},
   "source": [
    "# **Basic Python Collections**\n",
    "Examples using the MovieLens Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c21b381-7b00-4fe4-9def-e6cf171bb85a",
   "metadata": {},
   "source": [
    "## **Lists**\r",
    "Ordered sequences of elements, and they are mutable.\r\n",
    "\r\n",
    "**Example:** Extract all movie titles from the dataset into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "38179ae2-a3fa-495f-9851-854d1f71de96",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FIRST 5\n",
      "Toy Story (1995)\n",
      "Jumanji (1995)\n",
      "Grumpier Old Men (1995)\n",
      "Waiting to Exhale (1995)\n",
      "Father of the Bride Part II (1995)\n",
      "---\n",
      "LAST 5\n",
      "Black Butler: Book of the Atlantic (2017)\n",
      "No Game No Life: Zero (2017)\n",
      "Flint (2017)\n",
      "Bungo Stray Dogs: Dead Apple (2018)\n",
      "Andrew Dice Clay: Dice Rules (1991)\n"
     ]
    }
   ],
   "source": [
    "# using list comprehensions\n",
    "movie_titles = [\n",
    "    line.split(\",\")[1]\n",
    "    for line in open(\n",
    "        file_path_movies, \"r\", encoding=\"utf-8\", newline=\"\\r\\n\"\n",
    "    ).readlines()[1:]\n",
    "]\n",
    "#\n",
    "print(\"FIRST 5\")\n",
    "print(\"\\n\".join(movie_titles[:5]))  # Print the first 5 movie titles, each in a new line\n",
    "print(\"---\\nLAST 5\")\n",
    "print(\"\\n\".join(movie_titles[-5:]))  # Print the last 5 movie titles, each in a new line"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b466cb4b-e55f-4dc6-8091-d54ea47c22a7",
   "metadata": {},
   "source": [
    "## Sidebar: List Comprehensions, Nested List Comprehensions and Flattening Nested Lists\n",
    "<font color='red'>**Warning: Here be dragons!**</font>  \n",
    "  \n",
    "_skip this if you are new to data wrangling or running these notebooks for the first time_  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3115a3a7-26d5-442b-aaf6-9413bb6f6e7f",
   "metadata": {},
   "source": [
    "Let's try to build a list of genres found in the moves data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5a1d1768-c211-436a-829c-5302e7fef256",
   "metadata": {},
   "outputs": [],
   "source": [
    "# _lc for this list comprehensions sidebar\n",
    "genres_lc = []\n",
    "with open(file_path_movies, \"r\", encoding=\"utf-8\", newline=\"\\r\\n\") as file:\n",
    "    next(file)  # skip the header\n",
    "    for line in file:\n",
    "        row_data = line.split(\",\")\n",
    "        # read from end of line, avoid running into commas in titles\n",
    "        genres_in_row = row_data[-1].split(\"|\")\n",
    "        # strip the newlines and other cruft\n",
    "        genres_in_row = [g.strip() for g in genres_in_row]\n",
    "        genres_lc.append(genres_in_row)\n",
    "\n",
    "# .extend() instead of .append() will add individual elements instead of the full list\n",
    "# thus generating a flat list in the first place\n",
    "# but we are trying to create a nested list, then learning how to flatten it\n",
    "# because it's fun :P"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3fa39042-0531-48ba-9725-4053c821c0c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['Adventure', 'Animation', 'Children', 'Comedy', 'Fantasy'], ['Adventure', 'Children', 'Fantasy'], ['Comedy', 'Romance'], ['Comedy', 'Drama', 'Romance'], ['Comedy'], ['Action', 'Crime', 'Thriller'], ['Comedy', 'Romance'], ['Adventure', 'Children'], ['Action'], ['Action', 'Adventure', 'Thriller']]\n"
     ]
    }
   ],
   "source": [
    "print(genres_lc[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a26b37c-7d08-4b12-a894-ae030a1628a5",
   "metadata": {},
   "source": [
    "Try this again with a lambda and a map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d918b926-2a03-4527-a5e9-f5709364073b",
   "metadata": {},
   "outputs": [],
   "source": [
    "genres2_lc = list( \\\n",
    "\tmap( \\\n",
    "\t\tlambda gen: gen.split(',')[-1].strip().split('|'), \\\n",
    "\t\topen(file_path_movies, \"r\", encoding=\"utf-8\", newline=\"\\r\\n\") \\\n",
    "\t) \\\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8fa76f2a-e7ec-478a-b811-172f7be45565",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['Adventure', 'Animation', 'Children', 'Comedy', 'Fantasy'], ['Adventure', 'Children', 'Fantasy'], ['Comedy', 'Romance'], ['Comedy', 'Drama', 'Romance'], ['Comedy'], ['Action', 'Crime', 'Thriller'], ['Comedy', 'Romance'], ['Adventure', 'Children'], ['Action']]\n"
     ]
    }
   ],
   "source": [
    "print(genres2_lc[1:10]) #skip first line, header"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f1b3db2-1cbf-4312-a720-f7d3cf802d2b",
   "metadata": {},
   "source": [
    "```genres_lc``` and ```genres2_lc``` are similar.  \n",
    "\n",
    "However, I submit that while genres2 is a more concise form, it is not necessarily a more readable form.  \n",
    "\n",
    "List Comprehensions, like Regular Expressions are a double edged sword.  \n",
    "Too complex and they may create maintenance overhead later. \n",
    "\n",
    "erm...\n",
    "_Having said that..._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bf699f70-aed3-4808-a4b7-0ad990d4887d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# flatten genres - list comprehensions\n",
    "# they need you to be smart\n",
    "# and sometimes make people who read your code feel dumb\n",
    "genres_flat = [genre for genre_row in genres_lc for genre in genre_row]\n",
    "# whoa! quick note on how to read this follows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "778d0d2e-d011-45b1-a508-51068cb31718",
   "metadata": {},
   "source": [
    " \n",
    "reading [list comprehensions](https://docs.python.org/3.11/tutorial/datastructures.html#list-comprehensions)...is kinda simple.\n",
    "\n",
    "Imagine a basic scenario - we have elements (comma separate) arranged in multiple lines in a file.\n",
    "\n",
    "```\n",
    "FILE\n",
    "   |\n",
    "   |--line 01\n",
    "   |      |\n",
    "   |      |--CSV elements\n",
    "   |\n",
    "   |--line 02\n",
    "   |      |\n",
    "   |      |--CSV elements\n",
    "   |\n",
    "   |--line 03\n",
    "   |      |\n",
    "   |      |--CSV elements\n",
    "   |...so on\n",
    "```\n",
    "a basic for loop that reads lines from a file into a list:\n",
    "``` Python\n",
    "output = []\n",
    "# iterate over container\n",
    "# each iteration returns an element from the container\n",
    "for line in open(file,'r'):\n",
    "    # do some processing/transformation on the element\n",
    "    output.append(line)\n",
    "```\n",
    "\n",
    "using [```map(function, iterable)```](https://docs.python.org/3/library/functions.html#map) this can be written as:\n",
    "``` Python\n",
    "# map a lambda function on to every element of the file iterable\n",
    "# \n",
    "output = list(map(lambda line: line, open(file,'r')))\n",
    "```\n",
    "  \n",
    "and...\n",
    "translates to a list comprehension of:\n",
    "``` Python\n",
    "# output_list = [some_xform_on_element for element in container]\n",
    "output = [line for line in open(file,'r')]\n",
    "```\n",
    "\n",
    "---\n",
    "__nesting list comprehensions__:  \n",
    "now if the line had comma separated elements, we'll have to use a nested for loop:\n",
    "``` Python\n",
    "output = []\n",
    "# Loop 1 - Outer Loop\n",
    "for line in open(file,'r'):\n",
    "    # Loop 2 - Inner Loop - extract elements\n",
    "    line_list = line.split(',')\n",
    "    for element in line_list:\n",
    "        # do some processing/transformation on the element\n",
    "        output.append(element)\n",
    "```\n",
    "  \n",
    "compress outer loop using a ```map()```:  \n",
    "``` Python\n",
    "output = list( \\ # Loop 1 - Outer Loop\n",
    "    map( \\\n",
    "        lambda line: line, \\\n",
    "        open(file,'r') \\\n",
    "    ) \\\n",
    ")\n",
    "```\n",
    "build the inner loop now:  \n",
    "``` Python\n",
    "output = list( \\ # Loop 2 - Inner Loop - extract elements\n",
    "    map( \\\n",
    "        lambda element: element, \n",
    "        list( \\ # Loop 1 - Outer Loop\n",
    "            map( \\\n",
    "                lambda line: line, \\\n",
    "                open(file,'r') \\\n",
    "            ) \\\n",
    "        )\n",
    "    )\n",
    ")\n",
    "```\n",
    "Let's do a nested list comprehension now.  \n",
    "What we'd like to think will work is:  \n",
    "``` Python\n",
    "# WRONG!!\n",
    "out = [x # inner loop first #outer loop here]\n",
    "```\n",
    "but this is where it gets counter intuitive, the correct way to nest comprehensions is:  \n",
    "``` Python\n",
    "# Correct: LOOPS FOLLOW THE SAME ORDER AS THE FOR LOOP EQUIVALENT\n",
    "out = [x #outer-loop-here-that-returns-inner-iterable #inner-loop-here-that-returns-x]\n",
    "# Notice how the right most (or inner most) loop is the one that returns x\n",
    "# From left to right, each loop retuns an iterator\n",
    "# that the subsequent loop on the right then iterates over...\n",
    "# on and on...till the right most loop returns x\n",
    "```\n",
    "So, in our example:\n",
    "``` Python\n",
    "# returns a flat list of \n",
    "output = [element for line in open(file,'r') for element in line]\n",
    "```\n",
    "\n",
    "[This blog post](https://treyhunner.com/2015/12/python-list-comprehensions-now-in-color/) does a much better job of explaining list comprehensions than the note above. There's a separate notebook in this folder that helps practice list comprehensions.  \n",
    "  \n",
    "If you read the code we used to flatten the genres list, it'll make more sense now.  \n",
    "\n",
    "---  \n",
    " \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80b85d62-963f-4a64-b1bb-f6c9dda032cf",
   "metadata": {},
   "source": [
    "## How much, how many, how big?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90e0d5ed-839b-427b-9fae-347540b1783e",
   "metadata": {},
   "source": [
    "Let's think about what each item in the ```movie_titles``` list is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "298969dd-69af-4bc4-b8c7-59099583c621",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# all the methods built into movie_titles list (which is a kind of an object)\n",
    "movie_titles_methods = dir(movie_titles)\n",
    "\n",
    "# all the methods built into each element of movie_titles (each element is an object too, in our case, a string object)\n",
    "methods_for_each_movie_in_movie_titles = dir(movie_titles[0])\n",
    "\n",
    "# what is the data type of an element in our movie_titles list?\n",
    "type(movie_titles[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ddef5b48-9fc4-41b7-be08-b605ac5cf1bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48 methods support lists: \n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __imul__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __reversed__, __rmul__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, append, clear, copy, count, extend, index, insert, pop, remove, reverse, sort\n",
      "\n",
      "81 methods support each element in the list: \n",
      "__add__, __class__, __contains__, __delattr__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getnewargs__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mod__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __rmod__, __rmul__, __setattr__, __sizeof__, __str__, __subclasshook__, capitalize, casefold, center, count, encode, endswith, expandtabs, find, format, format_map, index, isalnum, isalpha, isascii, isdecimal, isdigit, isidentifier, islower, isnumeric, isprintable, isspace, istitle, isupper, join, ljust, lower, lstrip, maketrans, partition, removeprefix, removesuffix, replace, rfind, rindex, rjust, rpartition, rsplit, rstrip, split, splitlines, startswith, strip, swapcase, title, translate, upper, zfill\n"
     ]
    }
   ],
   "source": [
    "print(str(len(movie_titles_methods))+ \" methods support lists: \\n\"+ (', ').join(movie_titles_methods)+'\\n')\n",
    "print(str(len(methods_for_each_movie_in_movie_titles))+ \" methods support each element in the list: \\n\"+ (', ').join(methods_for_each_movie_in_movie_titles))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc295046-b0fb-4055-bd99-ffa8e610cf1d",
   "metadata": {},
   "source": [
    "WHOA!!! that's **81** method instances per line in the movies data.  \n",
    "We probably don't need that."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56e69ef5-b2e4-4f76-baf1-6802c010b446",
   "metadata": {},
   "source": [
    "##### *spoiler alert*  \n",
    "That's one of the optimizations NumPy builds upon.  \n",
    "  \n",
    "That for analysis of strcutured data, we don't need to be as generally flexible as Python, we could just use C-style arrays without the overhead of Python Objects.  \n",
    "We'll check those out in just a bit."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "973127d6-4944-4522-9894-862631158975",
   "metadata": {},
   "source": [
    "## Sidebar: Helper methods to figure out usage overhead  \n",
    "Before moving on, let's define a couple of helper methods that allows us to look into a python datastructure.  \n",
    "*If you don't grok the code below, you can safely ignore it and come back to it later.*  \n",
    "These are just meant to help us grasp the memory and complexity for each data structure.  \n",
    "  \n",
    "  [TODO: extract this into a separate file for general use across all notebooks]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "80e88ece-73ef-4b99-8b60-93d934bac528",
   "metadata": {},
   "outputs": [],
   "source": [
    "# before moving on, let's define a couple of helper methods that allows us to look into a python datastructure\n",
    "\n",
    "import sys\n",
    "\n",
    "# recursively calculate the total size of a datastructure\n",
    "def get_total_size(data):\n",
    "    recurse_count = 1\n",
    "    total_size = sys.getsizeof(data)\n",
    "    if isinstance(data, (list, tuple)):\n",
    "        for element in data:\n",
    "            size = get_total_size(element)\n",
    "            total_size += size[0]\n",
    "            recurse_count += size[1]\n",
    "    elif isinstance(data, dict):\n",
    "        for key, value in data.items():\n",
    "            size_dict_value = get_total_size(value)\n",
    "            total_size += sys.getsizeof(key) + size_dict_value[0]\n",
    "            recurse_count += size_dict_value[1]\n",
    "    return (total_size,recurse_count)\n",
    "\n",
    "# find out the number of methods associated with a data_structure and the memory it occupies\n",
    "def show_overhead(data_structure):\n",
    "\tlist_of_methods = dir(data_structure)\n",
    "\tnum_methods = str(len(list_of_methods))\n",
    "\tsize = get_total_size(data_structure)\n",
    "\tmemory_size = str(round(size[0]/1024,2))\n",
    "\tif hasattr(data_structure, \"__len__\"):\n",
    "\t\tmessage = num_methods+ ' methods support '+ str(type(data_structure))+ ', of length: '+str(len(data_structure))+' covering '+str(size[1])+' levels of depth and occupying '+memory_size+' Kb in memory:\\n'+ (', ').join(list_of_methods)+'\\n'\n",
    "\telse:\n",
    "\t\tmessage = num_methods+ ' methods support '+ str(type(data_structure))+ ', '+str(size[1])+' levels of depth occupying '+memory_size+' Kb in memory:\\n'+ (', ').join(list_of_methods)+'\\n'\n",
    "\tprint(message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b4664c35-bbd6-4a77-b365-b0266865fbe4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "73 methods support <class 'int'>, 1 levels of depth occupying 0.03 Kb in memory:\n",
      "__abs__, __add__, __and__, __bool__, __ceil__, __class__, __delattr__, __dir__, __divmod__, __doc__, __eq__, __float__, __floor__, __floordiv__, __format__, __ge__, __getattribute__, __getnewargs__, __getstate__, __gt__, __hash__, __index__, __init__, __init_subclass__, __int__, __invert__, __le__, __lshift__, __lt__, __mod__, __mul__, __ne__, __neg__, __new__, __or__, __pos__, __pow__, __radd__, __rand__, __rdivmod__, __reduce__, __reduce_ex__, __repr__, __rfloordiv__, __rlshift__, __rmod__, __rmul__, __ror__, __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, __rxor__, __setattr__, __sizeof__, __str__, __sub__, __subclasshook__, __truediv__, __trunc__, __xor__, as_integer_ratio, bit_count, bit_length, conjugate, denominator, from_bytes, imag, numerator, real, to_bytes\n",
      "\n",
      "59 methods support <class 'float'>, 1 levels of depth occupying 0.02 Kb in memory:\n",
      "__abs__, __add__, __bool__, __ceil__, __class__, __delattr__, __dir__, __divmod__, __doc__, __eq__, __float__, __floor__, __floordiv__, __format__, __ge__, __getattribute__, __getformat__, __getnewargs__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __int__, __le__, __lt__, __mod__, __mul__, __ne__, __neg__, __new__, __pos__, __pow__, __radd__, __rdivmod__, __reduce__, __reduce_ex__, __repr__, __rfloordiv__, __rmod__, __rmul__, __round__, __rpow__, __rsub__, __rtruediv__, __setattr__, __sizeof__, __str__, __sub__, __subclasshook__, __truediv__, __trunc__, as_integer_ratio, conjugate, fromhex, hex, imag, is_integer, real\n",
      "\n",
      "81 methods support <class 'str'>, of length: 15 covering 1 levels of depth and occupying 0.06 Kb in memory:\n",
      "__add__, __class__, __contains__, __delattr__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getnewargs__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mod__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __rmod__, __rmul__, __setattr__, __sizeof__, __str__, __subclasshook__, capitalize, casefold, center, count, encode, endswith, expandtabs, find, format, format_map, index, isalnum, isalpha, isascii, isdecimal, isdigit, isidentifier, islower, isnumeric, isprintable, isspace, istitle, isupper, join, ljust, lower, lstrip, maketrans, partition, removeprefix, removesuffix, replace, rfind, rindex, rjust, rpartition, rsplit, rstrip, split, splitlines, startswith, strip, swapcase, title, translate, upper, zfill\n",
      "\n",
      "48 methods support <class 'list'>, of length: 4 covering 5 levels of depth and occupying 0.2 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __imul__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __reversed__, __rmul__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, append, clear, copy, count, extend, index, insert, pop, remove, reverse, sort\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# tests\n",
    "\n",
    "# integer\n",
    "test1 = 1\n",
    "show_overhead(test1)\n",
    "\n",
    "# float\n",
    "test2 = 2.4\n",
    "show_overhead(test2)\n",
    "\n",
    "# string\n",
    "test3 = \"Happy Birthday!\"\n",
    "show_overhead(test3)\n",
    "\n",
    "# list\n",
    "test4 = [1,2,3,4]\n",
    "show_overhead(test4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "506767be-399a-4515-adc5-c0615cf93ab3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### <font color='green'>And Now, Back To Our Regularly Scheduled Programming...</font>  \n",
    "Tuples, Dictionaries, let's explore the rest of 'em."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f0c9ee5-8082-4d98-9ac8-3baf642a0ad2",
   "metadata": {},
   "source": [
    "## **Tuples**\r\n",
    "Tuples are ordered, immutable sequences.\r\n",
    "\r\n",
    "**Example:** Pair each movie with its genres in a tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "38472cf7-6a34-4000-bbc4-30467712996b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FIRST 5\n",
      "[('Toy Story (1995)', 'Adventure|Animation|Children|Comedy|Fantasy\\r\\n'), ('Jumanji (1995)', 'Adventure|Children|Fantasy\\r\\n'), ('Grumpier Old Men (1995)', 'Comedy|Romance\\r\\n'), ('Waiting to Exhale (1995)', 'Comedy|Drama|Romance\\r\\n'), ('Father of the Bride Part II (1995)', 'Comedy\\r\\n')]\n",
      "---\n",
      "LAST 5\n",
      "[('Black Butler: Book of the Atlantic (2017)', 'Action|Animation|Comedy|Fantasy\\r\\n'), ('No Game No Life: Zero (2017)', 'Animation|Comedy|Fantasy\\r\\n'), ('Flint (2017)', 'Drama\\r\\n'), ('Bungo Stray Dogs: Dead Apple (2018)', 'Action|Animation\\r\\n'), ('Andrew Dice Clay: Dice Rules (1991)', 'Comedy\\r\\n')]\n"
     ]
    }
   ],
   "source": [
    "# choose the second and third values in the row\n",
    "# read the file\n",
    "# skip the header\n",
    "# built a list of tuples\n",
    "movie_genre_pairs = [\n",
    "    (line.split(\",\")[1], line.split(\",\")[2])\n",
    "    for line in open(\n",
    "        file_path_movies, \"r\", encoding=\"utf-8\", newline=\"\\r\\n\"\n",
    "    ).readlines()[1:]\n",
    "]\n",
    "#\n",
    "print(\"FIRST 5\")\n",
    "print(movie_genre_pairs[:5])  # Print the first 5 (movie, genre) pairs\n",
    "print(\"---\\nLAST 5\")\n",
    "print(movie_genre_pairs[-5:])  # Print the last 5 (movie, genre) pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23001920-6a59-47e8-b945-97e2cf38ff4f",
   "metadata": {},
   "source": [
    "Like before, let's look at what it takes to support a tuple in memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "df81fad6-9435-471b-a537-b58fc25379c9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48 methods support <class 'list'>, of length: 9742 covering 29227 levels of depth and occupying 1934.81 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __imul__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __reversed__, __rmul__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, append, clear, copy, count, extend, index, insert, pop, remove, reverse, sort\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# remember that movie_genre_pairs is a list\n",
    "# list of all the movie_genre_pairs\n",
    "# len(movie_genre_pairs)\n",
    "show_overhead(movie_genre_pairs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "11568182-6413-4c60-935e-c9420917986e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "35 methods support <class 'tuple'>, of length: 2 covering 3 levels of depth and occupying 0.21 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getnewargs__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __rmul__, __setattr__, __sizeof__, __str__, __subclasshook__, count, index\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# all the methods for one tuple\n",
    "show_overhead(movie_genre_pairs[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8cefa427-bec4-4a1f-8601-5cc4d043d6dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "81 methods support <class 'str'>, of length: 16 covering 1 levels of depth and occupying 0.06 Kb in memory:\n",
      "__add__, __class__, __contains__, __delattr__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getnewargs__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mod__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __rmod__, __rmul__, __setattr__, __sizeof__, __str__, __subclasshook__, capitalize, casefold, center, count, encode, endswith, expandtabs, find, format, format_map, index, isalnum, isalpha, isascii, isdecimal, isdigit, isidentifier, islower, isnumeric, isprintable, isspace, istitle, isupper, join, ljust, lower, lstrip, maketrans, partition, removeprefix, removesuffix, replace, rfind, rindex, rjust, rpartition, rsplit, rstrip, split, splitlines, startswith, strip, swapcase, title, translate, upper, zfill\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# all the methods for each element in a tuple (which is an object, in our case, a string object)\n",
    "show_overhead(movie_genre_pairs[0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "369867f5-25a4-4e19-9238-80e38b81c13e",
   "metadata": {},
   "source": [
    "Let's consider Dictionaries next."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eb72c3c-799c-485e-8980-0e98655c8cf5",
   "metadata": {},
   "source": [
    "## **Dictionaries**\r\n",
    "Dictionaries map keys to values. They are mutable and keys are unique.\r\n",
    "\r\n",
    "**Example:** Create a dictionary where movie IDs are keys and movie titles are values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a61edd7e-59bd-4783-9c18-fb86526d325e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FIRST 5\n",
      "[('1', 'Toy Story (1995)'), ('2', 'Jumanji (1995)'), ('3', 'Grumpier Old Men (1995)'), ('4', 'Waiting to Exhale (1995)'), ('5', 'Father of the Bride Part II (1995)')]\n",
      "---\n",
      "LAST 5\n",
      "[('193581', 'Black Butler: Book of the Atlantic (2017)'), ('193583', 'No Game No Life: Zero (2017)'), ('193585', 'Flint (2017)'), ('193587', 'Bungo Stray Dogs: Dead Apple (2018)'), ('193609', 'Andrew Dice Clay: Dice Rules (1991)')]\n"
     ]
    }
   ],
   "source": [
    "movie_dict = {}\n",
    "with open(file_path_movies, \"r\", encoding=\"utf-8\", newline=\"\\r\\n\") as file:\n",
    "    next(file)  # skip the header\n",
    "    for line in file:\n",
    "        data = line.split(\",\")\n",
    "        movie_id, title = data[0], data[1]\n",
    "        movie_dict[movie_id] = title\n",
    "#\n",
    "print(\"FIRST 5\")\n",
    "print(list(movie_dict.items())[:5])  # Print the first 5 id-title pairs\n",
    "print(\"---\\nLAST 5\")\n",
    "print(list(movie_dict.items())[-5:])  # Print the last 5 id-title pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57a276a3-e3d8-4336-9212-251e2535a38d",
   "metadata": {},
   "source": [
    "with this dictionary, we can query a movie title using it's id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f98a8ba-28df-407d-97f5-27194d4fd068",
   "metadata": {},
   "source": [
    "### That gives us a way to do database style access using MovieIds! Yay!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "98670faa-8c56-430c-99a7-f57c1a107037",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Star Wars: Episode IV - A New Hope (1977)'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_dict[\"260\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "61ee9524-e535-4e02-ac5d-ec5a347b263c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9742"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(movie_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "80bbf4b2-d8e0-4eaf-ace9-9856682e84b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46 methods support <class 'dict'>, of length: 9742 covering 9743 levels of depth and occupying 1398.72 Kb in memory:\n",
      "__class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __ior__, __iter__, __le__, __len__, __lt__, __ne__, __new__, __or__, __reduce__, __reduce_ex__, __repr__, __reversed__, __ror__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, clear, copy, fromkeys, get, items, keys, pop, popitem, setdefault, update, values\n",
      "\n",
      "81 methods support <class 'str'>, of length: 41 covering 1 levels of depth and occupying 0.09 Kb in memory:\n",
      "__add__, __class__, __contains__, __delattr__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getnewargs__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mod__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __rmod__, __rmul__, __setattr__, __sizeof__, __str__, __subclasshook__, capitalize, casefold, center, count, encode, endswith, expandtabs, find, format, format_map, index, isalnum, isalpha, isascii, isdecimal, isdigit, isidentifier, islower, isnumeric, isprintable, isspace, istitle, isupper, join, ljust, lower, lstrip, maketrans, partition, removeprefix, removesuffix, replace, rfind, rindex, rjust, rpartition, rsplit, rstrip, split, splitlines, startswith, strip, swapcase, title, translate, upper, zfill\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# consider the overheads here\n",
    "show_overhead(movie_dict)\n",
    "show_overhead(movie_dict[\"260\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aac5359-3933-4ac8-bf7a-d2dabcbd5e9d",
   "metadata": {},
   "source": [
    "## **Sets**\r\n",
    "Sets are unordered collections of unique elements.\r\n",
    "\r\n",
    "**Example:** Extract all unique genres from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "2ce0da00-c2c1-479e-b7d2-b5cab5e48d3b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' The (1995)\"']\n",
      "[' The (Cité des enfants perdus']\n",
      "[' the Beloved Country (1995)\"']\n",
      "[' The (1995)\"']\n",
      "[' The (1995)\"']\n",
      "[' The (Postino']\n",
      "[' The (1995)\"']\n",
      "[' Les (1995)\"']\n",
      "[' The (1995)\"']\n",
      "[' The (1996)\"']\n"
     ]
    }
   ],
   "source": [
    "genres_set = set()\n",
    "with open(file_path_movies, \"r\", encoding=\"utf-8\", newline=\"\\r\\n\") as file:\n",
    "\tnext(file)  # skip the header\n",
    "\t# some titles may have commas\n",
    "\t# so, we'll use the following counter to \n",
    "\t# limit the number of lines this block of code prints\n",
    "\tlines_with_too_many_commas = 0\n",
    "\tlimit_output_to_x_lines = 10\n",
    "\t# \n",
    "\tfor line in file:\n",
    "\t\trow_data = line.split(\",\")\n",
    "\t\tgenres = row_data[2].split(\"|\")\n",
    "\t\tgenres_set.update(genres)\n",
    "\t\t# count if the row has too many commas, limit the output\n",
    "\t\tif len(row_data) > 3 and lines_with_too_many_commas < limit_output_to_x_lines:\n",
    "\t\t\tlines_with_too_many_commas += 1\n",
    "\t\t\t# this will only print as many times as limit_output_to_x_lines\n",
    "\t\t\tprint(genres)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979e5333-aa73-4524-bd03-85ec68adf4f8",
   "metadata": {},
   "source": [
    "##### Defer CSV Reading  \n",
    "In a future notebook, we'll use Python's CSV reader and also make a CSV reader of our own.  \n",
    "*For now, it's probably a distraction...*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "8ea44deb-4e47-499e-bd9e-562b436ba2f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the movie titles have commas in them, find a way to escape those\n",
    "# blows up on screen - careful.\n",
    "# print(genres_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "383c219c-235c-40fd-8d4f-275f8ef1b955",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Comedy', 'Drama', 'Romance']\n",
      "['Adventure', 'Drama', 'Fantasy', 'Mystery', 'Sci-Fi']\n",
      "['Drama']\n",
      "['Crime', 'Mystery', 'Thriller']\n",
      "['Children', 'Comedy']\n",
      "['Comedy', 'Drama', 'Romance']\n",
      "['Adventure', 'Children', 'Fantasy']\n",
      "['Drama', 'War']\n",
      "['Action', 'Crime', 'Drama', 'Thriller']\n",
      "['Drama', 'Thriller']\n",
      "\n",
      "Total generes (including duplicates) found in the dataset:  22084\n",
      "\n",
      "Unique generes found in the dataset: {'Action', 'Drama', 'Comedy', 'Horror', 'Sci-Fi', 'Film-Noir', 'Crime', '(no genres listed)', 'Musical', 'Documentary', 'Western', 'Animation', 'IMAX', 'Fantasy', 'Adventure', 'Thriller', 'Children', 'Mystery', 'Romance', 'War'}\n",
      "Number of unique genres found:  20\n"
     ]
    }
   ],
   "source": [
    "# for now lets read the last element in each row\n",
    "# same code as before...\n",
    "unique_genres_set = set()\n",
    "# we'll collect all genres in a list, allowing duplicates.\n",
    "# this list also enables us to analyse the computational complexity of building a set\n",
    "genres_with_duplicates_list = []\n",
    "with open(file_path_movies, \"r\", encoding=\"utf-8\", newline=\"\\r\\n\") as file:\n",
    "\tnext(file)  # skip the header\n",
    "\t# same as before, counter to limit the number of lines printed in this code block\n",
    "\tlines_with_too_many_commas = 0\n",
    "\tlimit_output_to_x_lines = 10\n",
    "\t# \n",
    "\tfor line in file:\n",
    "\t\t# adding .strip() to remove additional \\r\\n at the end \n",
    "\t\trow_data = line.strip().split(\",\")\n",
    "\t\tgenres = row_data[-1].split(\"|\")\n",
    "\t\t# use extend() instead of append() to add all elements to the list \n",
    "\t\t# instead of adding a list of genres as an element at the end\n",
    "\t\tgenres_with_duplicates_list.extend(genres)\n",
    "\t\t# some titles may have commas\n",
    "\t\tif len(row_data) > 3 and lines_with_too_many_commas < limit_output_to_x_lines:\n",
    "\t\t\tlines_with_too_many_commas += 1\n",
    "\t\t\tprint(genres)\n",
    "\n",
    "unique_genres_set = set(genres_with_duplicates_list)\n",
    "\n",
    "# the movie titles have commas in them, but now we should only see genres...\n",
    "print('\\nTotal generes (including duplicates) found in the dataset: ',len(genres_with_duplicates_list))\n",
    "print('\\nUnique generes found in the dataset:',unique_genres_set)\n",
    "print('Number of unique genres found: ',len(unique_genres_set))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "a1a9e568-6a2d-407d-a8d2-3e3127133fa7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48 methods support <class 'list'>, of length: 22084 covering 22085 levels of depth and occupying 1389.94 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __imul__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __reversed__, __rmul__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, append, clear, copy, count, extend, index, insert, pop, remove, reverse, sort\n",
      "\n",
      "57 methods support <class 'set'>, of length: 20 covering 1 levels of depth and occupying 2.21 Kb in memory:\n",
      "__and__, __class__, __class_getitem__, __contains__, __delattr__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getstate__, __gt__, __hash__, __iand__, __init__, __init_subclass__, __ior__, __isub__, __iter__, __ixor__, __le__, __len__, __lt__, __ne__, __new__, __or__, __rand__, __reduce__, __reduce_ex__, __repr__, __ror__, __rsub__, __rxor__, __setattr__, __sizeof__, __str__, __sub__, __subclasshook__, __xor__, add, clear, copy, difference, difference_update, discard, intersection, intersection_update, isdisjoint, issubset, issuperset, pop, remove, symmetric_difference, symmetric_difference_update, union, update\n",
      "\n",
      "In our specific case, the number of operations involved in building the set:  441680\n"
     ]
    }
   ],
   "source": [
    "# before moving on, let's consider the memory implications here:\n",
    "show_overhead(genres_with_duplicates_list)\n",
    "show_overhead(unique_genres_set)\n",
    "\n",
    "# Realize that sets have a big processing implication too (counting all the uniqueness)\n",
    "# sets involve checking each element for membership, \n",
    "# which can be O(n) for each element in the worst case. \n",
    "# So, to generalize, the overall processing time could be O(n^2) in the worst case.\n",
    "# In our specific case, the number of operations involved in building the set:\n",
    "print('In our specific case, the number of operations involved in building the set: ',len(genres_with_duplicates_list)*len(unique_genres_set))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ac511b6-efda-439d-9641-d0627a6b8046",
   "metadata": {},
   "source": [
    "Bruh! Counting unique elements can be expensive!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39f879ef-7625-42b4-beea-fa243784039d",
   "metadata": {},
   "source": [
    "# **The ```collections``` Module**\n",
    "\n",
    "The `collections` module in Python offers datatypes with additional capabilites that can augment the basic ones.\n",
    "This gives us powerful tools for specialized tasks. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a861549-76f5-4543-a3d9-8f816810a240",
   "metadata": {},
   "source": [
    "## **`namedtuple`: Tuple Subclass with Named Fields**\n",
    "- Provides clarity without the memory overhead of a full class.\n",
    "- Useful in scenarios where you might use structs in C.\n",
    "\n",
    "**Example:** Representing a movie with its title, genre, and rating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "9d6764be-d2a5-4abd-9ad9-ebc636a977f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "1be4ffd2-75cc-40bd-9029-be7c7a6b3fba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The first space opera blockbuster \n",
      " that excited all he interest in the \n",
      " idea of A Hero's Journey was: \n",
      " *Star Wars: Episode IV - A New Hope (1977)*\n"
     ]
    }
   ],
   "source": [
    "Movie = namedtuple(\"Movie\", [\"title\", \"genre\", \"rating\"])\n",
    "space_opera_the_first = Movie(\"Star Wars: Episode IV - A New Hope (1977)\", \"Sci-Fi\", 8.8)\n",
    "#\n",
    "print(f\"The first space opera blockbuster \\n \\\n",
    "that excited all he interest in the \\n \\\n",
    "idea of A Hero's Journey was: \\n \\\n",
    "*{space_opera_the_first.title}*\")  # Output: Inception :P"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2b2920c0-fc00-4011-a37f-ec6e9c6c973a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a Rating namedtuple\n",
    "Rating = namedtuple('Rating', ['user_id', 'movie_id', 'rating', 'timestamp'])\n",
    "# \n",
    "# load the ratings data into a list of namedtuples\n",
    "# * means the value is a tuple of multiple parameters\n",
    "# so it basically instructs the Rating() namedtuple to unpack the value\n",
    "ratings = [Rating(*line.strip().split(',')) \\\n",
    "\t\t   for line in open(file_path_ratings, 'r') \\\n",
    "\t\t   .readlines()[1:] \\\n",
    "\t\t  ]\n",
    "# the * operator is unpacking the list resulting from line.strip().split(',')\n",
    "# and creating the named tuple"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffe370f6-422e-4d6e-a311-6139dd8618b5",
   "metadata": {},
   "source": [
    "### Sidebar: * and ** in arguments \n",
    "The * and ** in arguments a useful but counter intuitive IMO. \n",
    "* and ** often come up in so many sessions that I run. \n",
    "  \n",
    "see [Arbitrary Argument Lists](https://docs.python.org/3/tutorial/controlflow.html#arbitrary-argument-lists) \n",
    "  \n",
    "- *xx means the value xx is a tuple, unpack it to find the arguments needed for the function in the respective positions, i.e. the first argument is tuple[0]th position, the second arg in the tuple[1]st position and so on for an arbitrary number of arguments.  \n",
    "- **xx means that the value xx is a dictionary, unpack it to find the *named* arguments, with argument names as keys. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "82c0301e-f35d-49e7-afb6-4be978de7de3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.0\n",
      "223\n",
      "1\n",
      "986935199\n"
     ]
    }
   ],
   "source": [
    "print(ratings[10].rating)\n",
    "print(ratings[12].movie_id)\n",
    "print(ratings[25].user_id)\n",
    "print(ratings[300].timestamp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "bb1d1400-1feb-40c3-ae8e-8712e1bf12e7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48 methods support <class 'list'>, of length: 100836 covering 504181 levels of depth and occupying 29203.81 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __imul__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __reversed__, __rmul__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, append, clear, copy, count, extend, index, insert, pop, remove, reverse, sort\n",
      "\n",
      "47 methods support <class '__main__.Rating'>, of length: 4 covering 5 levels of depth and occupying 0.28 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getnewargs__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __match_args__, __module__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __rmul__, __setattr__, __sizeof__, __slots__, __str__, __subclasshook__, _asdict, _field_defaults, _fields, _make, _replace, count, index, movie_id, rating, timestamp, user_id\n",
      "\n"
     ]
    }
   ],
   "source": [
    "show_overhead(ratings)\n",
    "show_overhead(ratings[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "da18e8cc-b04e-4da5-84eb-01bff2281be3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48 methods support <class 'list'>, of length: 100836 covering 100837 levels of depth and occupying 6101.89 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __imul__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __reversed__, __rmul__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, append, clear, copy, count, extend, index, insert, pop, remove, reverse, sort\n",
      "\n",
      "81 methods support <class 'str'>, of length: 1 covering 1 levels of depth and occupying 0.05 Kb in memory:\n",
      "__add__, __class__, __contains__, __delattr__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getnewargs__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mod__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __rmod__, __rmul__, __setattr__, __sizeof__, __str__, __subclasshook__, capitalize, casefold, center, count, encode, endswith, expandtabs, find, format, format_map, index, isalnum, isalpha, isascii, isdecimal, isdigit, isidentifier, islower, isnumeric, isprintable, isspace, istitle, isupper, join, ljust, lower, lstrip, maketrans, partition, removeprefix, removesuffix, replace, rfind, rindex, rjust, rpartition, rsplit, rstrip, split, splitlines, startswith, strip, swapcase, title, translate, upper, zfill\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# hold on...is that 28+ megs of memory?\n",
    "simple_ratings_list = [\n",
    "    line.split(\",\")[1]\n",
    "    for line in open(\n",
    "        file_path_ratings, \"r\", encoding=\"utf-8\", newline=\"\\r\\n\"\n",
    "    ).readlines()[1:]\n",
    "]\n",
    "show_overhead(simple_ratings_list)\n",
    "show_overhead(simple_ratings_list[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "b2174214-ddac-49a8-ab5e-45756cba946d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# levels in the list of named tuples vs simple_ratings_list\n",
    "504180/100836"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63b3e714-53ac-4304-afb5-9151271dee2d",
   "metadata": {},
   "source": [
    "Compared to the 2.5Mb CSV data and the 6-ish Mb list, the 28Mb Named Tuple is certainly more expensive, but very convinent."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc02c6bb-fe59-4100-8e90-1d701306dc39",
   "metadata": {},
   "source": [
    "## **`deque`: Double-Ended Queue**\n",
    "- Allows for fast appends and pops from both ends.\n",
    "- Useful for maintaining a sliding window or implementing certain types of parsers.\n",
    "\n",
    "**Example:** Maintaining a fixed-size window of the last N ratings for a movie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "962016cd-35e9-4a31-8fe4-83982e9728e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "f38c44a8-0c3f-4ee8-992c-30647b0c40e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration  1 \tlast_five_ratings:  deque([1.0], maxlen=5)\n",
      "iteration  2 \tlast_five_ratings:  deque([1.0, 2.0], maxlen=5)\n",
      "iteration  3 \tlast_five_ratings:  deque([1.0, 2.0, 3.0], maxlen=5)\n",
      "iteration  4 \tlast_five_ratings:  deque([1.0, 2.0, 3.0, 4.0], maxlen=5)\n",
      "iteration  5 \tlast_five_ratings:  deque([1.0, 2.0, 3.0, 4.0, 5.0], maxlen=5)\n",
      "iteration  6 \tlast_five_ratings:  deque([2.0, 3.0, 4.0, 5.0, 6.0], maxlen=5)\n",
      "iteration  7 \tlast_five_ratings:  deque([3.0, 4.0, 5.0, 6.0, 7.0], maxlen=5)\n",
      "iteration  8 \tlast_five_ratings:  deque([4.0, 5.0, 6.0, 7.0, 8.0], maxlen=5)\n",
      "iteration  9 \tlast_five_ratings:  deque([5.0, 6.0, 7.0, 8.0, 9.0], maxlen=5)\n",
      "iteration  10 \tlast_five_ratings:  deque([6.0, 7.0, 8.0, 9.0, 10.0], maxlen=5)\n",
      "\n",
      "final state: \n",
      " deque([6.0, 7.0, 8.0, 9.0, 10.0], maxlen=5)\n"
     ]
    }
   ],
   "source": [
    "last_five_ratings = deque(maxlen=5)\n",
    "# 10 ratings below\n",
    "iter = 0\n",
    "for rating in [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0]:\n",
    "\t# ratings go in one end, come out the other\n",
    "\titer += 1\n",
    "\tlast_five_ratings.append(rating)\n",
    "\tprint('iteration ',iter, '\\tlast_five_ratings: ',last_five_ratings)\n",
    "# finally\n",
    "print('\\nfinal state: \\n',last_five_ratings)  # Output: deque([6.0, 7.0, 8.0, 9.0, 10.0], maxlen=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "fea8e110-9ccc-4895-9167-e55fe0c17a8e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "53 methods support <class 'collections.deque'>, of length: 5 covering 1 levels of depth and occupying 0.74 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __copy__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __imul__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __reversed__, __rmul__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, append, appendleft, clear, copy, count, extend, extendleft, index, insert, maxlen, pop, popleft, remove, reverse, rotate\n",
      "\n",
      "59 methods support <class 'float'>, 1 levels of depth occupying 0.02 Kb in memory:\n",
      "__abs__, __add__, __bool__, __ceil__, __class__, __delattr__, __dir__, __divmod__, __doc__, __eq__, __float__, __floor__, __floordiv__, __format__, __ge__, __getattribute__, __getformat__, __getnewargs__, __getstate__, __gt__, __hash__, __init__, __init_subclass__, __int__, __le__, __lt__, __mod__, __mul__, __ne__, __neg__, __new__, __pos__, __pow__, __radd__, __rdivmod__, __reduce__, __reduce_ex__, __repr__, __rfloordiv__, __rmod__, __rmul__, __round__, __rpow__, __rsub__, __rtruediv__, __setattr__, __sizeof__, __str__, __sub__, __subclasshook__, __truediv__, __trunc__, as_integer_ratio, conjugate, fromhex, hex, imag, is_integer, real\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# we didn't load actual data here, but doesn't hurt to see the overheads\n",
    "# \n",
    "show_overhead(last_five_ratings)\n",
    "# each element is a float\n",
    "show_overhead(last_five_ratings[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a4d14a-7e80-4272-a287-127e7a4f8e82",
   "metadata": {},
   "source": [
    "## **`Counter`: Counting Elements**\r\n",
    "- Facilitates counting elements in an iterable.\r\n",
    "- Useful for tasks like token counting or histogram creation.\r\n",
    "\r\n",
    "**Example:** Counting genres in a list of movies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "48e6713f-ad13-45db-8545-8689e0fcf630",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "042a887a-8945-4872-a105-2751205cc708",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'Action': 3, 'Drama': 2, 'Sci-Fi': 2})\n"
     ]
    }
   ],
   "source": [
    "# usage\n",
    "arbitrary_list_of_genres = [\"Action\", \"Drama\", \"Action\", \"Sci-Fi\", \"Drama\", \"Sci-Fi\", \"Action\"]\n",
    "arbitrary_genre_count = Counter(arbitrary_list_of_genres)\n",
    "print(arbitrary_genre_count)  # Output: Counter({'Action': 3, 'Drama': 2, 'Sci-Fi': 2})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "36da966e-ad24-443b-a3aa-57d774598379",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'Drama': 4361, 'Comedy': 3756, 'Thriller': 1894, 'Action': 1828, 'Romance': 1596, 'Adventure': 1263, 'Crime': 1199, 'Sci-Fi': 980, 'Horror': 978, 'Fantasy': 779, 'Children': 664, 'Animation': 611, 'Mystery': 573, 'Documentary': 440, 'War': 382, 'Musical': 334, 'Western': 167, 'IMAX': 158, 'Film-Noir': 87, '(no genres listed)': 34})\n"
     ]
    }
   ],
   "source": [
    "# using the list we created for Sets\n",
    "movielens_genre_count = Counter(genres_with_duplicates_list)\n",
    "print(movielens_genre_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fa6fc2f-25d7-44c8-ab06-bce86a372e49",
   "metadata": {},
   "source": [
    "### A bit more involved: Count the frequency of each Rating \n",
    "* Find the frequency of each rating. How many 1.0 ratings found in the ratings.csv data.? How many 2.0, 3.0 etc. ?\n",
    "* Also, what are the 3 most frequent ratings?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "5e0944b3-cb00-4a03-8ede-9c696b37f937",
   "metadata": {},
   "outputs": [],
   "source": [
    "# should not be needed, here just in case \n",
    "from collections import namedtuple, Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "160e321f-67a4-42b9-a750-a3bc61540a34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Frequency of each rating.\n",
    "\n",
    "# define a Rating namedtuple\n",
    "Rating = namedtuple('Rating', ['user_id', 'movie_id', 'rating', 'timestamp'])\n",
    "# \n",
    "# load the ratings data into the tuple\n",
    "ratings = [Rating(*line.strip().split(',')) \\\n",
    "\t\t   for line in open(file_path_ratings, 'r') \\\n",
    "\t\t   .readlines()[1:] \\\n",
    "\t\t  ]\n",
    "\n",
    "# ratings is re-used in OrderedDict example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "8b695f2d-e9cf-474d-9ef7-4261e77ceb05",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'4.0': 26818, '3.0': 20047, '5.0': 13211, '3.5': 13136, '4.5': 8551, '2.0': 7551, '2.5': 5550, '1.0': 2811, '1.5': 1791, '0.5': 1370})\n"
     ]
    }
   ],
   "source": [
    "# Counter(list of ratings)\n",
    "list_of_ratings = [rating.rating for rating in ratings] # list comprehensions are fun amiright?\n",
    "rating_counts = Counter(list_of_ratings) \n",
    "print(rating_counts) #answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "dc22b2fd-5505-4b33-9084-1a2ab689f2a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('4.0', 26818), ('3.0', 20047), ('5.0', 13211)]\n",
      "100836\n"
     ]
    }
   ],
   "source": [
    "print(rating_counts.most_common(3)) #answer\n",
    "print(rating_counts.total())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "5fbea8ac-8ebc-46e6-b6fe-02a983b9de99",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('1.5', 1791)\n",
      "[('1.0', 2811), ('1.5', 1791), ('0.5', 1370)]\n"
     ]
    }
   ],
   "source": [
    "# how about the second least common rating?\n",
    "common = rating_counts.most_common()\n",
    "print(common[-2])\n",
    "# how about the last 3 least common ratings?\n",
    "print(common[-3:])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed655a48-fcb3-4836-9248-c509258b45e2",
   "metadata": {},
   "source": [
    "As you can see the Counter class is kinda fun!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "ff7ca79e-71e7-455b-9ea1-94f0dee1e74b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "48 methods support <class 'list'>, of length: 100836 covering 504181 levels of depth and occupying 29203.81 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __imul__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __reversed__, __rmul__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, append, clear, copy, count, extend, index, insert, pop, remove, reverse, sort\n",
      "\n",
      "48 methods support <class 'list'>, of length: 100836 covering 100837 levels of depth and occupying 6000.6 Kb in memory:\n",
      "__add__, __class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __imul__, __init__, __init_subclass__, __iter__, __le__, __len__, __lt__, __mul__, __ne__, __new__, __reduce__, __reduce_ex__, __repr__, __reversed__, __rmul__, __setattr__, __setitem__, __sizeof__, __str__, __subclasshook__, append, clear, copy, count, extend, index, insert, pop, remove, reverse, sort\n",
      "\n",
      "63 methods support <class 'collections.Counter'>, of length: 10 covering 11 levels of depth and occupying 1.07 Kb in memory:\n",
      "__add__, __and__, __class__, __class_getitem__, __contains__, __delattr__, __delitem__, __dict__, __dir__, __doc__, __eq__, __format__, __ge__, __getattribute__, __getitem__, __getstate__, __gt__, __hash__, __iadd__, __iand__, __init__, __init_subclass__, __ior__, __isub__, __iter__, __le__, __len__, __lt__, __missing__, __module__, __ne__, __neg__, __new__, __or__, __pos__, __reduce__, __reduce_ex__, __repr__, __reversed__, __ror__, __setattr__, __setitem__, __sizeof__, __str__, __sub__, __subclasshook__, __weakref__, _keep_positive, clear, copy, elements, fromkeys, get, items, keys, most_common, pop, popitem, setdefault, subtract, total, update, values\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# what about the memory costs?\n",
    "show_overhead(ratings)\n",
    "show_overhead(list_of_ratings)\n",
    "show_overhead(rating_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f502b8f3-fc06-4267-a651-2b4ede53426e",
   "metadata": {},
   "source": [
    "## **`OrderedDict`: Dict subclass that remembers the order entries were added**\n",
    "- Maintains the order of insertion.\n",
    "- Especially relevant for tasks where order matters, like configuration parsing or specific serialization tasks.\n",
    "\n",
    "**Example:** Storing movies and their ratings in the order they were rated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "898d69e8-4715-4f36-b77b-fa9c9ea4f2ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import OrderedDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "3f428f1d-896a-4605-85df-edbe246fc3c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inception: 8.8\n",
      "Interstellar: 8.6\n",
      "Dunkirk: 7.9\n"
     ]
    }
   ],
   "source": [
    "movie_ratings = OrderedDict()\n",
    "movie_ratings[\"Inception\"] = 8.8\n",
    "movie_ratings[\"Interstellar\"] = 8.6\n",
    "movie_ratings[\"Dunkirk\"] = 7.9\n",
    "\n",
    "for movie, rating in movie_ratings.items():\n",
    "    print(f\"{movie}: {rating}\")  # same order as added"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04d430cc-843d-44fc-8ab8-eec4e500fdfa",
   "metadata": {},
   "source": [
    "### A bit more involved: What is the average rating of a movie (```movie_id```) in the ratings file?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "2e47b857-aab2-425a-9a48-5246e0115f79",
   "metadata": {},
   "outputs": [],
   "source": [
    "avg_rating = OrderedDict()\n",
    "\n",
    "# ratings from the Counter example above\n",
    "for rating in ratings:\n",
    "\t# if the movie_id doesn't exist in our OrderedDict, add it as a key and create a list of ratings as value\n",
    "\t# else add the new rating found for the movie_id into the list of ratings\n",
    "\tif rating.movie_id not in avg_rating:\n",
    "\t\tavg_rating[rating.movie_id] = [float(rating.rating)]\n",
    "\telse:\n",
    "\t\tavg_rating[rating.movie_id].append(float(rating.rating))\n",
    "\n",
    "for movie, movie_ratings in avg_rating.items():\n",
    "\tavg_rating[movie] = sum(movie_ratings)/len(movie_ratings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "57c297b7-6821-4565-bfab-25c5db33baff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.231075697211155"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avg_rating['260'] #Star Wars: Episode IV - A New Hope (1977)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7286625-08cf-4647-85b3-67eb7db9aa64",
   "metadata": {},
   "source": [
    "## **`defaultdict`: Dict subclass with a default value for missing keys**\r\n",
    "- Helps in avoiding \"key not found\" errors.\r\n",
    "- Commonly used in graph algorithms, multi-value dictionaries, or accumulators.\r\n",
    "\r\n",
    "**Example:** Storing a list of ratings for each movie."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "f0b5b694-5187-48c0-85a0-05c6dc258c26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8.5, 8.6, 8.7]\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "movie_ratings = defaultdict(list)\n",
    "movie_ratings[\"Inception\"].extend([8.5, 8.6, 8.7])\n",
    "print(movie_ratings[\"Inception\"])  # Output: [8.5, 8.6, 8.7]\n",
    "print(movie_ratings[\"Unknown Movie\"])  # Output: [], without error"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ba388fe-a610-4a58-bbbd-32768bb8b6cb",
   "metadata": {},
   "source": [
    "### A bit more involved: Can we find out the number of movies for each genre? \n",
    "* Just like ```Counter``` above we want to find out the number of movies for each genre."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "9aa4c109-2eee-46ac-8177-8dd9caa35eff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# \n",
    "count_movie_by_genre = defaultdict(int)\n",
    "# \n",
    "# loop through all the lines\n",
    "line_num = 0\n",
    "for movie in open(file_path_movies,'r',encoding='utf-8'):\n",
    "\tline_num += 1\n",
    "\tif line_num > 1:\n",
    "\t\tfor genre in movie.strip().split(',')[-1].split('|'):\n",
    "\t\t\tcount_movie_by_genre[genre] +=1 #still captures the header"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "5c90ad76-3592-4ed7-8c9f-fc67b94648d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(int,\n",
       "            {'Adventure': 1263,\n",
       "             'Animation': 611,\n",
       "             'Children': 664,\n",
       "             'Comedy': 3756,\n",
       "             'Fantasy': 779,\n",
       "             'Romance': 1596,\n",
       "             'Drama': 4361,\n",
       "             'Action': 1828,\n",
       "             'Crime': 1199,\n",
       "             'Thriller': 1894,\n",
       "             'Horror': 978,\n",
       "             'Mystery': 573,\n",
       "             'Sci-Fi': 980,\n",
       "             'War': 382,\n",
       "             'Musical': 334,\n",
       "             'Documentary': 440,\n",
       "             'IMAX': 158,\n",
       "             'Western': 167,\n",
       "             'Film-Noir': 87,\n",
       "             '(no genres listed)': 34})"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_movie_by_genre"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e8449e4-845e-494b-b3e5-c69adc40d9d2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e9cd9648-5a13-446f-a775-76616ace03c1",
   "metadata": {},
   "source": [
    "# Next"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f30f9979-d5c1-4fa6-875d-76e2c017ec76",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "We look at itertools and functools"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}