04-Dask/02.01-MovieLens-and-Dask.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7e9f4107-83a1-4cf6-8340-9f9454b57eae",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Distributed Data Analysis with Dask  \n",
    "*__[Dask](https://www.dask.org/)__ with the __MovieLens__ dataset*  \n",
    "\n",
    "**Part 1: Getting Started, Load the MovieLens dataset**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f21bef00-086e-44fd-b516-01016a08432d",
   "metadata": {},
   "source": [
    "### <font color='green'>__Support for Google Colab__  </font>  \n",
    "    \n",
    "open this notebook in Colab using the following button:  \n",
    "  \n",
    "<a href=\"https://colab.research.google.com/github/shauryashaurya/learn-data-munging/blob/main/04-Dask/02.01-MovieLens-and-Dask.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>  \n",
    "\n",
    "**Note**: The Dask Dashboard is not available on Google Colab unless you register with tunnelling systems like Saturn Cloud or NGrok - these are both good approaches - for folks running this on colab I have not built support for bit for the workshop. Your contributions / PRs would be very welcome.\n",
    "  \n",
    "<font color='green'>uncomment and execute the cell below to setup and run this notebook on Google Colab.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "303cf188-3333-4207-98d6-ec8af6d83f28",
   "metadata": {},
   "source": [
    "### Graphviz or Cytoscape (ipycytoscape)\n",
    "\n",
    "For some Dask exercises, we may want to visualize the task graph. \n",
    "To do so we'll need: [GraphViz](https://graphviz.org/) or [Cytoscape](https://cytoscape.org/download.html) and [ipycytoscape](https://ipycytoscape.readthedocs.io/en/latest/installing.html)\n",
    "\n",
    "These come with ready installers. Install, restart Jupyter if needed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d40fdc1e-8800-47e9-bd9a-cf007406461c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # SETUP FOR COLAB: select all the lines below and uncomment (CTRL+/ on windows)\n",
    "\n",
    "# # grab dask - in most cases it should already be available in colab\n",
    "# ! python -m pip install --quiet --upgrade --no-cache-dir \"dask[complete]\"\n",
    "# # Let's download and unzip the MovieLens 25M Dataset as well.\n",
    "# ! mkdir ./../data\n",
    "# ! wget -q https://files.grouplens.org/datasets/movielens/ml-25m.zip\n",
    "# ! unzip ./ml-25m.zip -d ./../data/\n",
    "\n",
    "# ! echo \"DONE\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "618beb44-e43c-413d-bbaf-6aee638a79d3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Overview\n",
    "Select, filter, join, groupby, pivot, and windows.  \n",
    "\n",
    "Instead of toy examples and '10 minutes to xx' we load an actual dataset and ask meaningful questions about it.\n",
    "  \n",
    "We'll use the [MovieLens](https://grouplens.org/datasets/movielens/) dataset for these exercises.  \n",
    "This dataset is non trivial and should expand to about __1GB__ on you local disk.  \n",
    "\n",
    "Download and unzip [MovieLens 25M Dataset](https://grouplens.org/datasets/movielens/25m/) for this analysis.\n",
    "\n",
    "Either ensure the data is in ```\"./data/ml-25m\"``` folder or update the path to the data below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1999f3bc-1a3d-4a4e-a4c2-5634a9cfd0ed",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "**Citation**:  \n",
    "*F. Maxwell Harper and Joseph A. Konstan.* 2015.  \n",
    "The MovieLens Datasets: History and Context.  \n",
    "ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4: 19:1–19:19. <https://doi.org/10.1145/2827872>  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e800f817-bb56-4a7e-bbff-94320a85c234",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Approach\n",
    "\n",
    "The idea is to tackle simple Dask use-cases first and move on to more complex ones.  \n",
    "\n",
    "Starting with simply loading the data into a dask distributed dataframe, we then perform a data evaluation, some cleanup and finally analysis. We first ask questions based on individual data files, then move on to combining data from multiple files.\n",
    "\n",
    "We are going to try and avoid the more mathematically involved parts of exploratory data analysis - for e.g. statistical analysis on various features etc. - the core focus in the ability to grok pyspark functions and have fun while doing it.  \n",
    "\n",
    "By the end you'd not only have an idea of Dask, but also how we ask questions and analyze a chunk of data.  \n",
    "\n",
    "_You may also end up with a watch-list to binge on your next weekend._ :)   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b518bef-b08d-40ff-b63b-7395f06de060",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Setup Dask, Pandas and Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9d77b71-0373-43bd-8011-1c05b5b18e60",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Setup the Dask Cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "938fa94c-777e-41ed-8f50-d9d6d37f2a2c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### Installation\n",
    "\n",
    "A local install is as simple as ```pip install \"dask[complete]\"```  \n",
    "  \n",
    "Unlike Spark, Dask is incredibly easy to setup - checkout [Dask Installation Docs](https://docs.dask.org/en/stable/install.html)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cfe72515-0873-4f40-b574-024ef66bc31b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numpy version:  1.26.2\n",
      "pandas version:  2.1.3\n"
     ]
    }
   ],
   "source": [
    "# Step 1: numpy and pandas\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "print(\"numpy version: \", np.__version__)\n",
    "print(\"pandas version: \", pd.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "19defc56-381c-45fb-b93c-a3ddc5444e70",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dask version:  2023.12.0\n"
     ]
    }
   ],
   "source": [
    "# Step 2: import dask and related\n",
    "import dask\n",
    "import dask.array as da\n",
    "import dask.bag as db\n",
    "import dask.dataframe as dd\n",
    "from dask.distributed import Client, LocalCluster\n",
    "\n",
    "print(\"dask version: \", dask.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d4028755-bd5d-40e0-a1bd-1de6f835cb19",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Step 3: Create a Dask Cluster and a Client\n",
    "try:\n",
    "    cluster = LocalCluster()\n",
    "    # alternative, when you want to specify the dashboard address/port\n",
    "    # cluster = LocalCluster(dashboard_address = 'localhost:8786')\n",
    "    if (cluster.shutdown_on_close == False):\n",
    "        cluster.shutdown_on_close = True\n",
    "except Exception:\n",
    "    pass\n",
    "#\n",
    "try:\n",
    "    client = Client(cluster)\n",
    "except Exception:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9ebfabd9-4706-4ba3-a0e9-5cd3f795444f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# see cluster information\n",
    "# cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "59053c50-23ba-49b7-b64c-23ea049d86f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# see client information\n",
    "# client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "296f5b9b-c27b-44e3-9552-bca987d9fbba",
   "metadata": {},
   "source": [
    "# Client, Workers, and Scheduler - How Dask does it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eabb7b71-b559-4cb6-800f-fdb37364a9a3",
   "metadata": {},
   "source": [
    "Dask is very similar to Spark.  \r\n",
    "Both lazily construct directed acyclic graphs (DAGs) of tasks,  \r\n",
    "and split large datasets into small portions (partitions/chunks) \n",
    "\n",
    "  \n",
    "\n",
    "    \n",
    "<img src=\"https://docs.dask.org/en/stable/_images/dask-overview.svg\" width=\"80%\" height=\"80%\" />\n",
    "\n",
    "<font size = '2px' color='green'>Image from docs.dask.org</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1c2535f-39b4-4429-b0c2-c58c5577341a",
   "metadata": {},
   "source": [
    "* The **client** is the machine that submits tasks to the **scheduler** (create Task Graphs).  \n",
    "* The scheduler is responsible for managing the tasks and distributing them to the **workers** (execute Task Graphs).  \n",
    "* The workers are responsible for executing the tasks.  \n",
    "* The scheduler will then return the results to the client.\n",
    "\n",
    "   \n",
    " \r\n",
    "\r\n",
    "The client, scheduler, and workers communicate with each other using a message passing protocol. The client sends a message to the scheduler, which then sends a message to the worker. The worker executes the task and sends a message back to the scheduler, which then sends the results back to the cli  ent."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "717eedf9-5b2e-4d48-93ef-f4893ee1c17f",
   "metadata": {},
   "source": [
    "Instead of just distributing data across nodes, Dask distributes the functions too - for e.g. you can imagine that each worker node in the dask cluster that's working on Dask.dataframe is running it's own copy of Pandas, the scheduler builds a task graph that let's these workers perform the computations in parallel. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a16086fa-617e-4f96-a65c-e337f2c978f5",
   "metadata": {},
   "source": [
    "Dask structures (Array, DataFrame, and Bag) are *lazy* by default, meaning they don't evaluate until you explicitly ask for a result using ```compute()```.  \n",
    "\n",
    "* Dask Bag is like Spark RDD - a general purpose distributed, parallel compute engine.  \n",
    "* Dask Dataframes, built on top of Dask Bag and Distributed, are like Spark Dataframes.  \n",
    "* Spark DataFrames have an advantage with optimizations from the Catalyst engine and Tungsten, but Dask DataFrames provide a more \"Pandas-like\" experience."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb164ece-2d5e-4451-b455-24fe2add5bee",
   "metadata": {},
   "source": [
    "# Locate the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4a045b14-f578-47ad-9972-a3f5aaba0911",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "datalocation = \"../data/ml-25m/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0428c276-5ef7-4bde-addf-3bb79337783d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# specify file names\n",
    "file_path_movies = datalocation + \"movies.csv\"\n",
    "file_path_links = datalocation + \"links.csv\"\n",
    "file_path_ratings = datalocation + \"ratings.csv\"\n",
    "file_path_tags = datalocation + \"tags.csv\"\n",
    "file_path_genome_tags = datalocation + \"genome-tags.csv\"\n",
    "file_path_genome_scores = datalocation + \"genome-scores.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96574358-90c0-4eed-8f4e-1dd5692bd60c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Load the dataset(s)\n",
    "\n",
    "From the ```README.txt``` file in the small MovieLens dataset:\n",
    "The dataset files are written as [**comma-separated values**](http://en.wikipedia.org/wiki/Comma-separated_values) files with a **single header row**. Columns that contain commas (`,`) are **escaped using double-quotes (`\"`)**. These files are encoded as **UTF-8**. If accented characters in movie titles or tag values (e.g. Misérables, Les (1995)) display incorrectly, make sure that any program reading the data, such as a text editor, terminal, or script, is configured for UTF-8.\n",
    "\n",
    "So, we specify:\n",
    "* Separator - ```,```\n",
    "* Escape Character - ```\"```\n",
    "* Encoding - ```UTF-8```\n",
    "\n",
    "Often this is called the **dialect** of the CSV file.\n",
    "These dialects vary often, so need our attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "51505e23-1133-4302-beaf-1690963ec807",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# dask dataframes parallelize pandas dataframes\n",
    "# so many of the idioms are similar\n",
    "csv_separator = \",\"\n",
    "csv_escapechar = '\"'\n",
    "csv_quotechar = csv_escapechar\n",
    "csv_encoding = \"utf-8\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "503f86f0-c81b-4f2c-80ab-090afbe049d2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Movies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2583f6a3-72c2-4414-9c12-b56ac69fb00a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Let's specify the [-  ```dtypes```  ](https://pandas.pydata.org/docs/user_guide/basics.html#dtypes) of each of the columns in the movies file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c9a1e502-96a1-4ae1-91e9-cba5e695e7a1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# schema, inferred from the README.txt file\n",
    "movies_schema = {\"movieId\": \"Int32\", \"title\": \"string\", \"genres\": \"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39569d7e-8c3c-44f2-a40b-776adc33d13d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Two of the columns are [strings of text](https://pandas.pydata.org/docs/user_guide/text.html#working-with-text-data). Pandas may treat those as ```object```, but we wanted to use the [```pandas.StringDType```](https://pandas.pydata.org/docs/reference/api/pandas.StringDtype.html#pandas-stringdtype) here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5c90ad76-3592-4ed7-8c9f-fc67b94648d2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# we are using dd - dask.dataframe\n",
    "movies = dd.read_csv(\n",
    "    file_path_movies,\n",
    "    dtype=movies_schema,\n",
    "    sep=csv_separator,\n",
    "    quotechar=csv_quotechar,\n",
    "    encoding=csv_encoding,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3440485a-172d-4fbf-8970-d88c5f1c7cbb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# needs GraphViz or ipycytoscape\n",
    "# movies.visualize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f1b032d0-cd24-4aa7-8d98-422ba8d4900e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>Comedy|Drama|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId                               title  \\\n",
       "0        1                    Toy Story (1995)   \n",
       "1        2                      Jumanji (1995)   \n",
       "2        3             Grumpier Old Men (1995)   \n",
       "3        4            Waiting to Exhale (1995)   \n",
       "4        5  Father of the Bride Part II (1995)   \n",
       "\n",
       "                                        genres  \n",
       "0  Adventure|Animation|Children|Comedy|Fantasy  \n",
       "1                   Adventure|Children|Fantasy  \n",
       "2                               Comedy|Romance  \n",
       "3                         Comedy|Drama|Romance  \n",
       "4                                       Comedy  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the first 15 lines\n",
    "# just like Pandas\n",
    "movies.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9fd113e-b20d-4877-85ed-70bc81d14a89",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "The dask dataframe understands when to internally call Dask's ```compute()``` or ```gather()``` methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "521d5f6a-1be8-496b-9d75-ebd120a3475f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "movieId              Int32\n",
       "title      string[pyarrow]\n",
       "genres     string[pyarrow]\n",
       "dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data types of each column\n",
    "movies.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "822b6e03-cff0-41fb-9bfa-8b760ab40bef",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [movieId, title, genres]\n",
       "Index: []"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to distribute data across nodes, \n",
    "# dask maintains an empty _meta object\n",
    "movies._meta"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e863b90d-98b6-4b6e-bda1-53bee108a426",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### From [Dask DataFrames Internal Design](https://docs.dask.org/en/stable/dataframe-design.html)\n",
    "\n",
    "Many DataFrame operations rely on knowing the name and dtype of columns. To keep track of this information, all Dask DataFrame objects have a ```_meta``` attribute which contains an empty Pandas object with the same dtypes and names.\n",
    "\n",
    "Internally, Dask DataFrame does its best to propagate this information through all operations, so most of the time a user shouldn’t have to worry about this. Usually this is done by evaluating the operation on a small sample of fake data, which can be found on the ```_meta_nonempty``` attribute.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1c692e4c-ac28-45ed-aca2-94b9dbca4f4a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>title</th>\n",
       "      <th>genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId title genres\n",
       "0        1     a      a\n",
       "1     <NA>  <NA>   <NA>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies._meta_nonempty"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21309cbb-2489-4b4d-96cc-6532de8cc6bb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Just for practice, let's load the other datasets too..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4ca7df0-19e3-4535-99ec-efe5dd1c6dfd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c2e6aa7b-4f71-4e33-95fb-a74b9014368d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# schema, inferred from the README.txt file\n",
    "# load imdbId,tmdbId as strings because the are a part of a URL.\n",
    "#   IMDB: http://www.imdb.com/title/imdbId/\n",
    "#   TMDB: https://www.themoviedb.org/movie/tmdbId\n",
    "links_schema = {\"movieId\": \"Int32\", \"imdbId\": \"string\", \"tmdbId\": \"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8e0390f3-e09e-4a66-a335-23e71a15ba25",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "links = dd.read_csv(\n",
    "    file_path_links,\n",
    "    dtype=links_schema,\n",
    "    sep=csv_separator,\n",
    "    quotechar=csv_quotechar,\n",
    "    encoding=csv_encoding,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "39b4d11d-cc6e-404c-ab23-31c282d89142",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# links.visualize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ebaa2447-2217-46e3-968f-6da807d7fb1a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>imdbId</th>\n",
       "      <th>tmdbId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0114709</td>\n",
       "      <td>862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0113497</td>\n",
       "      <td>8844</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0113228</td>\n",
       "      <td>15602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0114885</td>\n",
       "      <td>31357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0113041</td>\n",
       "      <td>11862</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId   imdbId tmdbId\n",
       "0        1  0114709    862\n",
       "1        2  0113497   8844\n",
       "2        3  0113228  15602\n",
       "3        4  0114885  31357\n",
       "4        5  0113041  11862"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "035a2d76-69ab-4f49-a6d2-262c84e14257",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "movieId              Int32\n",
       "imdbId     string[pyarrow]\n",
       "tmdbId     string[pyarrow]\n",
       "dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "9bea6ef0-eab7-4b09-861a-ae011b92370b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>imdbId</th>\n",
       "      <th>tmdbId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [movieId, imdbId, tmdbId]\n",
       "Index: []"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links._meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "54322fe8-7e19-448a-b232-9c093af02ce6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movieId</th>\n",
       "      <th>imdbId</th>\n",
       "      <th>tmdbId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movieId imdbId tmdbId\n",
       "0        1      a      a\n",
       "1     <NA>   <NA>   <NA>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links._meta_nonempty"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25e7b872-1db9-49c9-9288-d327cae2fdf1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Ratings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "342bbe4a-c904-4313-8b31-252dc87cda76",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Reading through the ```README``` file:  \n",
    "Ratings are made on a 5-star scale, with half-star increments (0.5 stars - 5.0 stars).  \n",
    "Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970.  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dee402a-94df-4afb-b691-2c1397ea3f36",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Ooh! Ooh! We got our first [DateTime](https://pandas.pydata.org/docs/reference/api/pandas.DatetimeTZDtype.html#pandas.DatetimeTZDtype)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "672561a7-1b7f-4168-aaec-85245faec9b0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# schema, inferred from the README.txt file\n",
    "# read timestamps as integers then convert to dates later.\n",
    "\n",
    "ratings_schema = {\n",
    "    \"userId\": \"Int32\",\n",
    "    \"movieId\": \"Int32\",\n",
    "    \"rating\": \"Float32\",\n",
    "    \"timestamp\": \"Int64\",\n",
    "}\n",
    "#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "5d102b11-3aca-43ef-977a-21c67ca7f610",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ratings = dd.read_csv(\n",
    "    file_path_ratings,\n",
    "    dtype=ratings_schema,\n",
    "    sep=csv_separator,\n",
    "    quotechar=csv_quotechar,\n",
    "    encoding=csv_encoding,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "5f36be99-fb8b-48d4-897a-d1076092022c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>296</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1147880044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>306</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1147868817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>307</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1147868828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>665</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1147878820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>899</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1147868510</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating   timestamp\n",
       "0       1      296     5.0  1147880044\n",
       "1       1      306     3.5  1147868817\n",
       "2       1      307     5.0  1147868828\n",
       "3       1      665     5.0  1147878820\n",
       "4       1      899     3.5  1147868510"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e6261239-e395-4199-b2a4-8ce39d8a73e5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ratings.visualize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "5fda8643-8179-4e79-9c56-663a66830e9c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# now let's add a datetime column that we derive from the raw timestamp\n",
    "ratings[\"datetime\"] = dd.to_datetime(ratings[\"timestamp\"], unit=\"s\", utc=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "3c1a2a42-ed8b-4a45-a90c-a8e97d5b7cbd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                     Int32\n",
       "movieId                    Int32\n",
       "rating                   Float32\n",
       "timestamp                  Int64\n",
       "datetime     datetime64[ns, UTC]\n",
       "dtype: object"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "fa2d7b97-64ab-4974-ae49-b79e9b1405f8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [userId, movieId, rating, timestamp, datetime]\n",
       "Index: []"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings._meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "43b1ccf1-32da-4350-bbda-34fde9186850",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1970-01-01 00:00:00+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>1970-01-01 00:00:00+00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp                  datetime\n",
       "0       1        1     1.0          1 1970-01-01 00:00:00+00:00\n",
       "1    <NA>     <NA>    <NA>       <NA> 1970-01-01 00:00:00+00:00"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings._meta_nonempty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "aa55cc44-f2f2-435f-ab47-2553bbbdcc2c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ratings[\"date\"] = ratings[\"datetime\"].dt.date"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "b634931a-1b8f-48d2-a399-b932dcba0cd7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>296</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1147880044</td>\n",
       "      <td>2006-05-17 15:34:04+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>306</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1147868817</td>\n",
       "      <td>2006-05-17 12:26:57+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>307</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1147868828</td>\n",
       "      <td>2006-05-17 12:27:08+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>665</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1147878820</td>\n",
       "      <td>2006-05-17 15:13:40+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>899</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1147868510</td>\n",
       "      <td>2006-05-17 12:21:50+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating   timestamp                  datetime        date\n",
       "0       1      296     5.0  1147880044 2006-05-17 15:34:04+00:00  2006-05-17\n",
       "1       1      306     3.5  1147868817 2006-05-17 12:26:57+00:00  2006-05-17\n",
       "2       1      307     5.0  1147868828 2006-05-17 12:27:08+00:00  2006-05-17\n",
       "3       1      665     5.0  1147878820 2006-05-17 15:13:40+00:00  2006-05-17\n",
       "4       1      899     3.5  1147868510 2006-05-17 12:21:50+00:00  2006-05-17"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b8a743c-d528-4e02-8b3f-4658951b7dc1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Niiice!  \n",
    "Wait, let's check the data types once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "ed38a86b-2820-43e3-9cd5-e1752e481f54",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                     Int32\n",
       "movieId                    Int32\n",
       "rating                   Float32\n",
       "timestamp                  Int64\n",
       "datetime     datetime64[ns, UTC]\n",
       "date             string[pyarrow]\n",
       "dtype: object"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17bfb7b-4963-4520-bb35-47b859392b42",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "We'd prefer if date was a datetime type as well.  \n",
    "Let's prepare the date column again, wrapping it in ```pd.to_datetime()```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "2b2d2175-123f-4866-83e9-678ea4655e74",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "ratings[\"date\"] = dd.to_datetime(ratings[\"datetime\"].dt.date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "3eeec631-6ec1-45a4-a1a9-8f25d691c2b4",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                     Int32\n",
       "movieId                    Int32\n",
       "rating                   Float32\n",
       "timestamp                  Int64\n",
       "datetime     datetime64[ns, UTC]\n",
       "date              datetime64[ns]\n",
       "dtype: object"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the data types again\n",
    "ratings.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e558931-f805-49c1-acff-3bc99c80418d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Ah! much better.  \n",
    "Why you say?  \n",
    "We could easily extract and manipulate dates this way.  \n",
    "for e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "e9207f2d-cf0d-41b9-bc43-8eefce4ab1f8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# extract the day, month and year of each rating\n",
    "ratings[\"day\"] = ratings[\"date\"].dt.day\n",
    "ratings[\"month\"] = ratings[\"date\"].dt.month\n",
    "ratings[\"year\"] = ratings[\"date\"].dt.year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "89f4fab2-5691-4bf2-89cd-aef1c90dcd18",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                     Int32\n",
       "movieId                    Int32\n",
       "rating                   Float32\n",
       "timestamp                  Int64\n",
       "datetime     datetime64[ns, UTC]\n",
       "date              datetime64[ns]\n",
       "day                        int32\n",
       "month                      int32\n",
       "year                       int32\n",
       "dtype: object"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "d872bebb-b7ab-4580-9707-e156d71e9fce",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [userId, movieId, rating, timestamp, datetime, date, day, month, year]\n",
       "Index: []"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings._meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "ba6f4188-dde4-4c0c-ae0d-15ebd725d883",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1970-01-01 00:00:00+00:00</td>\n",
       "      <td>1970-01-01</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>1970-01-01 00:00:00+00:00</td>\n",
       "      <td>1970-01-01</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp                  datetime       date  \\\n",
       "0       1        1     1.0          1 1970-01-01 00:00:00+00:00 1970-01-01   \n",
       "1    <NA>     <NA>    <NA>       <NA> 1970-01-01 00:00:00+00:00 1970-01-01   \n",
       "\n",
       "   day  month  year  \n",
       "0    1      1     1  \n",
       "1    1      1     1  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings._meta_nonempty"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c962401-b2e3-4678-911d-8a9cd9ac9064",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "very clean!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "6510354d-0981-4f9a-913f-aac00e23e806",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Holy Moly!\n",
    "# ratings.visualize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "1fa35971-7b52-44ef-86d8-d413e9924420",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>296</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1147880044</td>\n",
       "      <td>2006-05-17 15:34:04+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "      <td>17</td>\n",
       "      <td>5</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>306</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1147868817</td>\n",
       "      <td>2006-05-17 12:26:57+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "      <td>17</td>\n",
       "      <td>5</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>307</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1147868828</td>\n",
       "      <td>2006-05-17 12:27:08+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "      <td>17</td>\n",
       "      <td>5</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>665</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1147878820</td>\n",
       "      <td>2006-05-17 15:13:40+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "      <td>17</td>\n",
       "      <td>5</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>899</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1147868510</td>\n",
       "      <td>2006-05-17 12:21:50+00:00</td>\n",
       "      <td>2006-05-17</td>\n",
       "      <td>17</td>\n",
       "      <td>5</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating   timestamp                  datetime       date  \\\n",
       "0       1      296     5.0  1147880044 2006-05-17 15:34:04+00:00 2006-05-17   \n",
       "1       1      306     3.5  1147868817 2006-05-17 12:26:57+00:00 2006-05-17   \n",
       "2       1      307     5.0  1147868828 2006-05-17 12:27:08+00:00 2006-05-17   \n",
       "3       1      665     5.0  1147878820 2006-05-17 15:13:40+00:00 2006-05-17   \n",
       "4       1      899     3.5  1147868510 2006-05-17 12:21:50+00:00 2006-05-17   \n",
       "\n",
       "   day  month  year  \n",
       "0   17      5  2006  \n",
       "1   17      5  2006  \n",
       "2   17      5  2006  \n",
       "3   17      5  2006  \n",
       "4   17      5  2006  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae72f8b9-6d60-47f6-a9c3-33a21c985bc1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Let's do this for the Tags data set too - just for practice."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f67371-838c-48fd-967d-c2d9c9c210f8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "\n",
    "## Tags"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a9e1cbe-bd07-4926-b770-b7e476ed09e3",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "From ```README```:  \n",
    "Tags are user-generated metadata about movies. Each tag is typically a single word or short phrase. The meaning, value, and purpose of a particular tag is determined by each user.  \n",
    "  \n",
    "Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "29e98718-59dd-4682-87bd-38019d1a792c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# schema, inferred from the README.txt file\n",
    "# read timestamps as integers then convert to dates later.\n",
    "# userId,movieId,tag,timestamp\n",
    "tags_schema = {\n",
    "    \"userId\": \"Int32\",\n",
    "    \"movieId\": \"Int32\",\n",
    "    \"tag\": \"string\",\n",
    "    \"timestamp\": \"Int64\",\n",
    "}\n",
    "#"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "823b2316-bdc9-4081-b903-c67b2cac472d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "tags = dd.read_csv(\n",
    "    file_path_tags,\n",
    "    dtype=tags_schema,\n",
    "    sep=csv_separator,\n",
    "    quotechar=csv_quotechar,\n",
    "    encoding=csv_encoding,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "85a3a68a-30fc-4ad1-bcc3-8f8d650704e8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# So you say this is simple?...\n",
    "# tags.visualize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "8c5bb70a-e903-4962-b4e9-abc7521937ff",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>260</td>\n",
       "      <td>classic</td>\n",
       "      <td>1439472355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>260</td>\n",
       "      <td>sci-fi</td>\n",
       "      <td>1439472256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>1732</td>\n",
       "      <td>dark comedy</td>\n",
       "      <td>1573943598</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1732</td>\n",
       "      <td>great dialogue</td>\n",
       "      <td>1573943604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>7569</td>\n",
       "      <td>so bad it's good</td>\n",
       "      <td>1573943455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId               tag   timestamp\n",
       "0       3      260           classic  1439472355\n",
       "1       3      260            sci-fi  1439472256\n",
       "2       4     1732       dark comedy  1573943598\n",
       "3       4     1732    great dialogue  1573943604\n",
       "4       4     7569  so bad it's good  1573943455"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb95a477-114e-4372-9119-1ffc0c18de23",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "just like before let's add a more readable ```datetime``` column here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "91bbfb1e-95b3-4448-a639-5a06f625b799",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "tags[\"datetime\"] = dd.to_datetime(tags[\"timestamp\"], unit=\"s\", utc=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "52c21c7a-bf91-417f-bcb4-6f86e86930ce",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                     Int32\n",
       "movieId                    Int32\n",
       "tag              string[pyarrow]\n",
       "timestamp                  Int64\n",
       "datetime     datetime64[ns, UTC]\n",
       "dtype: object"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "2de8efdd-9f30-46b5-84b5-195ab949c4f6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# extract date into a new column\n",
    "tags[\"date\"] = dd.to_datetime(tags[\"datetime\"].dt.date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "702e7d50-0b07-4062-929c-e292aa57399f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId                     Int32\n",
       "movieId                    Int32\n",
       "tag              string[pyarrow]\n",
       "timestamp                  Int64\n",
       "datetime     datetime64[ns, UTC]\n",
       "date              datetime64[ns]\n",
       "dtype: object"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "1723c691-f631-4550-b4f5-cb2f787b56ca",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>260</td>\n",
       "      <td>classic</td>\n",
       "      <td>1439472355</td>\n",
       "      <td>2015-08-13 13:25:55+00:00</td>\n",
       "      <td>2015-08-13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>260</td>\n",
       "      <td>sci-fi</td>\n",
       "      <td>1439472256</td>\n",
       "      <td>2015-08-13 13:24:16+00:00</td>\n",
       "      <td>2015-08-13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>1732</td>\n",
       "      <td>dark comedy</td>\n",
       "      <td>1573943598</td>\n",
       "      <td>2019-11-16 22:33:18+00:00</td>\n",
       "      <td>2019-11-16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1732</td>\n",
       "      <td>great dialogue</td>\n",
       "      <td>1573943604</td>\n",
       "      <td>2019-11-16 22:33:24+00:00</td>\n",
       "      <td>2019-11-16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>7569</td>\n",
       "      <td>so bad it's good</td>\n",
       "      <td>1573943455</td>\n",
       "      <td>2019-11-16 22:30:55+00:00</td>\n",
       "      <td>2019-11-16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId               tag   timestamp                  datetime  \\\n",
       "0       3      260           classic  1439472355 2015-08-13 13:25:55+00:00   \n",
       "1       3      260            sci-fi  1439472256 2015-08-13 13:24:16+00:00   \n",
       "2       4     1732       dark comedy  1573943598 2019-11-16 22:33:18+00:00   \n",
       "3       4     1732    great dialogue  1573943604 2019-11-16 22:33:24+00:00   \n",
       "4       4     7569  so bad it's good  1573943455 2019-11-16 22:30:55+00:00   \n",
       "\n",
       "        date  \n",
       "0 2015-08-13  \n",
       "1 2015-08-13  \n",
       "2 2019-11-16  \n",
       "3 2019-11-16  \n",
       "4 2019-11-16  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6eb6c9f-d156-48fa-801c-754e0c899aac",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "umm... go nuts.  \n",
    "Extract the day, month and year from date, because why not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "64f6aeb1-3a1b-46da-be87-292161d9d999",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "tags[\"day\"] = tags[\"date\"].dt.day\n",
    "tags[\"month\"] = tags[\"date\"].dt.month\n",
    "tags[\"year\"] = tags[\"date\"].dt.year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "371cf9ef-cc84-4768-8251-1c76ca78495d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [userId, movieId, tag, timestamp, datetime, date, day, month, year]\n",
       "Index: []"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags._meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "e0d8b28d-ac99-45f6-8719-6f9c9771a2bd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>a</td>\n",
       "      <td>1</td>\n",
       "      <td>1970-01-01 00:00:00+00:00</td>\n",
       "      <td>1970-01-01</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>1970-01-01 00:00:00+00:00</td>\n",
       "      <td>1970-01-01</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId   tag  timestamp                  datetime       date  day  \\\n",
       "0       1        1     a          1 1970-01-01 00:00:00+00:00 1970-01-01    1   \n",
       "1    <NA>     <NA>  <NA>       <NA> 1970-01-01 00:00:00+00:00 1970-01-01    1   \n",
       "\n",
       "   month  year  \n",
       "0      1     1  \n",
       "1      1     1  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags._meta_nonempty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "fb38b233-df8b-4176-93db-8760067e4010",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ...would that it were so simple\n",
    "# tags.visualize()\n",
    "# #Hail,Caesar! #CoenBros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "1618bd04-750f-499a-a635-8415854b17c8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>tag</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>datetime</th>\n",
       "      <th>date</th>\n",
       "      <th>day</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>260</td>\n",
       "      <td>classic</td>\n",
       "      <td>1439472355</td>\n",
       "      <td>2015-08-13 13:25:55+00:00</td>\n",
       "      <td>2015-08-13</td>\n",
       "      <td>13</td>\n",
       "      <td>8</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>260</td>\n",
       "      <td>sci-fi</td>\n",
       "      <td>1439472256</td>\n",
       "      <td>2015-08-13 13:24:16+00:00</td>\n",
       "      <td>2015-08-13</td>\n",
       "      <td>13</td>\n",
       "      <td>8</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>1732</td>\n",
       "      <td>dark comedy</td>\n",
       "      <td>1573943598</td>\n",
       "      <td>2019-11-16 22:33:18+00:00</td>\n",
       "      <td>2019-11-16</td>\n",
       "      <td>16</td>\n",
       "      <td>11</td>\n",
       "      <td>2019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1732</td>\n",
       "      <td>great dialogue</td>\n",
       "      <td>1573943604</td>\n",
       "      <td>2019-11-16 22:33:24+00:00</td>\n",
       "      <td>2019-11-16</td>\n",
       "      <td>16</td>\n",
       "      <td>11</td>\n",
       "      <td>2019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>7569</td>\n",
       "      <td>so bad it's good</td>\n",
       "      <td>1573943455</td>\n",
       "      <td>2019-11-16 22:30:55+00:00</td>\n",
       "      <td>2019-11-16</td>\n",
       "      <td>16</td>\n",
       "      <td>11</td>\n",
       "      <td>2019</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId               tag   timestamp                  datetime  \\\n",
       "0       3      260           classic  1439472355 2015-08-13 13:25:55+00:00   \n",
       "1       3      260            sci-fi  1439472256 2015-08-13 13:24:16+00:00   \n",
       "2       4     1732       dark comedy  1573943598 2019-11-16 22:33:18+00:00   \n",
       "3       4     1732    great dialogue  1573943604 2019-11-16 22:33:24+00:00   \n",
       "4       4     7569  so bad it's good  1573943455 2019-11-16 22:30:55+00:00   \n",
       "\n",
       "        date  day  month  year  \n",
       "0 2015-08-13   13      8  2015  \n",
       "1 2015-08-13   13      8  2015  \n",
       "2 2019-11-16   16     11  2019  \n",
       "3 2019-11-16   16     11  2019  \n",
       "4 2019-11-16   16     11  2019  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tags.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e13fbf25-2912-42ce-bb00-3862404de8d0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Wrap Up the cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "cc23986a-d022-4dac-a0b6-148c69941bbb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-12-07 15:29:56,536 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:49176'.\n"
     ]
    }
   ],
   "source": [
    "# wrap up like this\n",
    "client.retire_workers()\n",
    "# QQ - do we really need cluster.close() here?\n",
    "# cluster.close()\n",
    "client.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00e87a98-9330-49cd-8bbb-78bf7f38ac1d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Insights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c32a1f8-01d0-4685-91a4-48abb08dc38a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "1. Client, Workers, and Scheduler\r\n",
    "\t* The client is the machine that submits tasks to the scheduler (create Task Graphs).\r\n",
    "\t* The scheduler is responsible for managing the tasks and distributing them to the workers (execute Task Graphs).\r\n",
    "\t* The workers are responsible for executing the tasks.\r\n",
    "\t* The scheduler will then return the results to the client.\r\n",
    "1. ```.visualize()```, ```._meta``` and ```._meta_nonempty```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9cd9648-5a13-446f-a775-76616ace03c1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Next"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f30f9979-d5c1-4fa6-875d-76e2c017ec76",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "* Let's play with the MovieLens dataset some more."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}