OverviewNotes_PYTDS_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### PYT-DS SAISOFT\n",
    "\n",
    "* [Overview 1](https://github.com/4dsolutions/Python5/blob/master/OverviewNotes_PYTDS.ipynb)\n",
    "* [Overview 3](https://github.com/4dsolutions/Python5/blob/master/OverviewNotes_PYTDS_3.ipynb)\n",
    "\n",
    "<a data-flickr-embed=\"true\"  href=\"https://www.flickr.com/photos/kirbyurner/27963484878/in/album-72157693427665102/\" title=\"Barry at Large\"><img src=\"https://farm1.staticflickr.com/969/27963484878_b38f0db42a_m.jpg\" width=\"240\" height=\"180\" alt=\"Barry at Large\"></a><script async src=\"//embedr.flickr.com/assets/client-code.js\" charset=\"utf-8\"></script>\n",
    "\n",
    "# DATA SCIENCE WITH PYTHON\n",
    "\n",
    "## Where Have We Been, What Have We Seen?\n",
    "\n",
    "Data Science includes Data Management.  This means we might call a DBA (Database Administrator) a kind of data scientist?  Why not?  Their speciality is efficiently warehousing data, meaning the same information is not redundantly scattered.  \n",
    "\n",
    "In terms of rackspace and data center security, of course we want redundancy, but in databases the potential for data corruption increases exponentially with the number of places the same information must be kept up to date.  If a person changes their legal name, you don't want to have to break your primary key, which should be based on something less mutable.\n",
    "\n",
    "Concepts of mutability versus immutability are important in data science.  In consulting, I would often advertise spreadsheets as ideal for \"what if\" scenarios, but if the goal is to chronicle \"what was\" then the mutability of a spreedsheet becomes a liability.  The bookkeeping community always encourages databases over spreadsheets when it comes to keeping a company or agency's books.\n",
    "\n",
    "DBAs also concern themselves with missing data.  If the data is increasingly full of holes, that's a sign the database may no longer be loved.  DBAs engage in load balancing, meaning they must give priority to services most in demand.  However \"what's in demand\" may be a changing vista."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rng_years = pd.period_range('1/1/2000', '1/1/2018', freq='Y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "People needing to divide a fiscal year starting in July, into quarters, are in luck with ```pandas```.  I've been looking for lunar year and other periodic progressions.  The whole timeline thing still seems difficult, even with a proleptic Gregorian plus UTC timezones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "head_count = np.random.randint(10,35, size=19)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As usual, I'm recommending telling yourself a story, in this case about an exclusive party you've been hosting ever since 2000, all the way up to 2018.  Once you get the interactive version of this Notebook, you'll be able to extend this record by as many more years as you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "new_years_party = pd.DataFrame(head_count, index = rng_years,\n",
    "                               columns=[\"Attenders\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DBAs who know SQL / noSQL, will find pandas, especially its ```inner``` ```outer``` ```left``` and ```right``` merge possibilities somewhat familiar.  We learn about the ```set``` type through maths, through Python, and understand about unions and intersections, differences.\n",
    "\n",
    "We did a fair amount of practicing with merge, appreciating that pandas pays a lot of attention to the DataFrame labels, synchronizing along indexes and columns, creating NaN empty cells where needed.\n",
    "\n",
    "We're spared a lot of programming, and yet even so though these patchings- together can become messy and disorganized.  At least the steps are chronicled.  That's why spreadsheets are not a good idea.  You lose your audit trail.  There's no good way to find and debug your mistakes.\n",
    "\n",
    "Keep the whole pipeline in view, from raw data sources, through [numerous cleaning and filtering steps](https://youtu.be/qvHXRuGPHl0).  The linked Youtube is a good example:  the data scientist vastly shrinks the data needed, by weeding out what's irrelevant.  Data science is all about dismissing the irrelevant, which takes work, real energy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Attenders</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2000</th>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2001</th>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2002</th>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2003</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2004</th>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2005</th>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007</th>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008</th>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009</th>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2011</th>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012</th>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013</th>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2015</th>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2016</th>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017</th>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018</th>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Attenders\n",
       "2000         31\n",
       "2001         16\n",
       "2002         11\n",
       "2003         10\n",
       "2004         24\n",
       "2005         14\n",
       "2006         26\n",
       "2007         18\n",
       "2008         26\n",
       "2009         25\n",
       "2010         10\n",
       "2011         17\n",
       "2012         19\n",
       "2013         12\n",
       "2014         33\n",
       "2015         16\n",
       "2016         24\n",
       "2017         14\n",
       "2018         33"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_years_party"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's the average number of party-goers over this nine-year period?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20.0"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.round(new_years_party.Attenders.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Might you also want the median and mode?  Do you remember what those are?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    10\n",
       "1    14\n",
       "2    16\n",
       "3    24\n",
       "4    26\n",
       "5    33\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_years_party.Attenders.mode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that seems strange.  Isn't the mode of a column of numbers, a number?\n",
    "\n",
    "We're looking at the numbers that appear most often, the top six in the ranking.  Surely there must be some tie breaking rule."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18.0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_years_party.Attenders.median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's not years of age, lets remember, but the number clocked in at just after midnight, so still there for the beginning of the New Year.  What other columns (features) were collected on these people?  Do they know they were being surveilled?  Would they recognize themselves, even if they saw this data?  Or is this data totally made up.  Come to think of it, we did use a randomizer now didn't we.\n",
    "\n",
    "<a data-flickr-embed=\"true\"  href=\"https://www.flickr.com/photos/kirbyurner/16149955921/in/photolist-JdgRnt-JdgNxa-239WrJR-JdgXVD-JdgQJp-Dve8wV-JdgZax-DveefD-Dve7S8-JdgXdr-239W9Ea-239Wb5V-DveeFD-DvedJD-239WsoB-JdgYxF-239Wahn-DvedjR-JdgZJZ-225jXoh-239VXTF-Dve75X-JdgJ9R-JdgJGK-JdgKo4-JdgL42-JdgHrt-239VX7a-F2iuns-21K5vNc-F2it6E-F2irFq-F2isp9-F2iqYd-F2itHm-F2ipcN-F2ipLo-F2imu3-237fAbJ-F2inqG-F2iqgb-F2ikwb-239VKTZ-JdfMD8-239VJa8-239VJZz-239VHGV-QfwD7s-QHZTcw-qB7KeT\" title=\"New Years Ball\"><img src=\"https://farm8.staticflickr.com/7498/16149955921_514a579c6b_n.jpg\" width=\"320\" height=\"303\" alt=\"New Years Ball\"></a><script async src=\"//embedr.flickr.com/assets/client-code.js\" charset=\"utf-8\"></script>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}