sliderule_dsi_inferential_statistics_exercise_2-TForsyth.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examining Racial Discrimination in the US Job Market\n",
    "\n",
    "### Background\n",
    "Racial discrimination continues to be pervasive in cultures throughout the world. Researchers examined the level of racial discrimination in the United States labor market by randomly assigning identical résumés to black-sounding or white-sounding names and observing the impact on requests for interviews from employers.\n",
    "\n",
    "### Data\n",
    "In the dataset provided, each row represents a resume. The 'race' column has two values, 'b' and 'w', indicating black-sounding and white-sounding. The column 'call' has two values, 1 and 0, indicating whether the resume received a call from employers or not.\n",
    "\n",
    "Note that the 'b' and 'w' values in race are assigned randomly to the resumes when presented to the employer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "You will perform a statistical analysis to establish whether race has a significant impact on the rate of callbacks for resumes.\n",
    "\n",
    "Answer the following questions **in this notebook below and submit to your Github account**. \n",
    "\n",
    "   1. What test is appropriate for this problem? Does CLT apply?\n",
    "   2. What are the null and alternate hypotheses?\n",
    "   3. Compute margin of error, confidence interval, and p-value. Try using both the bootstrapping and the frequentist statistical approaches.\n",
    "   4. Write a story describing the statistical significance in the context or the original problem.\n",
    "   5. Does your analysis mean that race/name is the most important factor in callback success? Why or why not? If not, how would you amend your analysis?\n",
    "\n",
    "You can include written notes in notebook cells using Markdown: \n",
    "   - In the control panel at the top, choose Cell > Cell Type > Markdown\n",
    "   - Markdown syntax: http://nestacms.com/docs/creating-content/markdown-cheat-sheet\n",
    "\n",
    "#### Resources\n",
    "+ Experiment information and data source: http://www.povertyactionlab.org/evaluation/discrimination-job-market-united-states\n",
    "+ Scipy statistical methods: http://docs.scipy.org/doc/scipy/reference/stats.html \n",
    "+ Markdown syntax: http://nestacms.com/docs/creating-content/markdown-cheat-sheet\n",
    "+ Formulas for the Bernoulli distribution: https://en.wikipedia.org/wiki/Bernoulli_distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.io.stata.read_stata('data/us_job_market_discrimination.dta')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "235.0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Number of callbacks for black-sounding names\n",
    "sum(data[data.race=='w'].call)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>ad</th>\n",
       "      <th>education</th>\n",
       "      <th>ofjobs</th>\n",
       "      <th>yearsexp</th>\n",
       "      <th>honors</th>\n",
       "      <th>volunteer</th>\n",
       "      <th>military</th>\n",
       "      <th>empholes</th>\n",
       "      <th>occupspecific</th>\n",
       "      <th>...</th>\n",
       "      <th>compreq</th>\n",
       "      <th>orgreq</th>\n",
       "      <th>manuf</th>\n",
       "      <th>transcom</th>\n",
       "      <th>bankreal</th>\n",
       "      <th>trade</th>\n",
       "      <th>busservice</th>\n",
       "      <th>othservice</th>\n",
       "      <th>missind</th>\n",
       "      <th>ownership</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>316</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>19</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>313</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>313</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Nonprofit</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 65 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  id ad  education  ofjobs  yearsexp  honors  volunteer  military  empholes  \\\n",
       "0  b  1          4       2         6       0          0         0         1   \n",
       "1  b  1          3       3         6       0          1         1         0   \n",
       "2  b  1          4       1         6       0          0         0         0   \n",
       "3  b  1          3       4         6       0          1         0         1   \n",
       "4  b  1          3       3        22       0          0         0         0   \n",
       "\n",
       "   occupspecific    ...      compreq  orgreq  manuf  transcom  bankreal trade  \\\n",
       "0             17    ...          1.0     0.0    1.0       0.0       0.0   0.0   \n",
       "1            316    ...          1.0     0.0    1.0       0.0       0.0   0.0   \n",
       "2             19    ...          1.0     0.0    1.0       0.0       0.0   0.0   \n",
       "3            313    ...          1.0     0.0    1.0       0.0       0.0   0.0   \n",
       "4            313    ...          1.0     1.0    0.0       0.0       0.0   0.0   \n",
       "\n",
       "  busservice othservice  missind  ownership  \n",
       "0        0.0        0.0      0.0             \n",
       "1        0.0        0.0      0.0             \n",
       "2        0.0        0.0      0.0             \n",
       "3        0.0        0.0      0.0             \n",
       "4        0.0        1.0      0.0  Nonprofit  \n",
       "\n",
       "[5 rows x 65 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 4870 entries, 0 to 4869\n",
      "Data columns (total 65 columns):\n",
      "id                    4870 non-null object\n",
      "ad                    4870 non-null object\n",
      "education             4870 non-null int8\n",
      "ofjobs                4870 non-null int8\n",
      "yearsexp              4870 non-null int8\n",
      "honors                4870 non-null int8\n",
      "volunteer             4870 non-null int8\n",
      "military              4870 non-null int8\n",
      "empholes              4870 non-null int8\n",
      "occupspecific         4870 non-null int16\n",
      "occupbroad            4870 non-null int8\n",
      "workinschool          4870 non-null int8\n",
      "email                 4870 non-null int8\n",
      "computerskills        4870 non-null int8\n",
      "specialskills         4870 non-null int8\n",
      "firstname             4870 non-null object\n",
      "sex                   4870 non-null object\n",
      "race                  4870 non-null object\n",
      "h                     4870 non-null float32\n",
      "l                     4870 non-null float32\n",
      "call                  4870 non-null float32\n",
      "city                  4870 non-null object\n",
      "kind                  4870 non-null object\n",
      "adid                  4870 non-null float32\n",
      "fracblack             4784 non-null float32\n",
      "fracwhite             4784 non-null float32\n",
      "lmedhhinc             4784 non-null float32\n",
      "fracdropout           4784 non-null float32\n",
      "fraccolp              4784 non-null float32\n",
      "linc                  4784 non-null float32\n",
      "col                   4870 non-null float32\n",
      "expminreq             4870 non-null object\n",
      "schoolreq             4870 non-null object\n",
      "eoe                   4870 non-null float32\n",
      "parent_sales          1672 non-null float32\n",
      "parent_emp            1722 non-null float32\n",
      "branch_sales          608 non-null float32\n",
      "branch_emp            658 non-null float32\n",
      "fed                   3102 non-null float32\n",
      "fracblack_empzip      1918 non-null float32\n",
      "fracwhite_empzip      1918 non-null float32\n",
      "lmedhhinc_empzip      1908 non-null float32\n",
      "fracdropout_empzip    1918 non-null float32\n",
      "fraccolp_empzip       1918 non-null float32\n",
      "linc_empzip           1918 non-null float32\n",
      "manager               4870 non-null float32\n",
      "supervisor            4870 non-null float32\n",
      "secretary             4870 non-null float32\n",
      "offsupport            4870 non-null float32\n",
      "salesrep              4870 non-null float32\n",
      "retailsales           4870 non-null float32\n",
      "req                   4870 non-null float32\n",
      "expreq                4870 non-null float32\n",
      "comreq                4870 non-null float32\n",
      "educreq               4870 non-null float32\n",
      "compreq               4870 non-null float32\n",
      "orgreq                4870 non-null float32\n",
      "manuf                 4870 non-null float32\n",
      "transcom              4870 non-null float32\n",
      "bankreal              4870 non-null float32\n",
      "trade                 4870 non-null float32\n",
      "busservice            4870 non-null float32\n",
      "othservice            4870 non-null float32\n",
      "missind               4870 non-null float32\n",
      "ownership             4870 non-null object\n",
      "dtypes: float32(42), int16(1), int8(12), object(10)\n",
      "memory usage: 1.3+ MB\n"
     ]
    }
   ],
   "source": [
    "#Inspect dataframe\n",
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NULL count for 'call' column: 0\n",
      "NULL count for 'race' column: 0\n",
      "Count of duplicate records: 0\n"
     ]
    }
   ],
   "source": [
    "#Get null counts\n",
    "print(\"NULL count for 'call' column:\", data.call.isnull().sum())\n",
    "print(\"NULL count for 'race' column:\", data.race.isnull().sum())\n",
    "\n",
    "#Get duplicate counts\n",
    "print(\"Count of duplicate records:\", data.duplicated().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: There are no null values or duplicates in our sample."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question 1: What test is appropriate for this problem? Does CLT apply?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to apply the appropriat test and use the CLT, we consider two things: \n",
    "* Size of sample: is it large? \n",
    "* Nature of observations: are they independent? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of observations for white resumes: 2435\n",
      "Number of observations for black resumes: 2435\n"
     ]
    }
   ],
   "source": [
    "#Let's get a closer look at the data in terms of frequencies\n",
    "##Special Note: both of the variables of interest (b/w and 1/0 are Bernoulli random variables)\n",
    "w = data[data.race=='w']\n",
    "b = data[data.race=='b']\n",
    "\n",
    "print(\"Number of observations for white resumes:\", len(w))\n",
    "print(\"Number of observations for black resumes:\", len(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The sample includes 2435 observations for each category (b, w), which is larger than 30, making it a large sample. We assume that the observations are independent as the resumes were randomly selected and one person's resume does not affect the others'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Since the rate of callbacks per b/w category is compared, the sample size is large, and the observations are independent, both a binomial test and z-test is appropriate for this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of black sounding names: 2435\n",
      "Total number of white sounding names: 2435\n",
      "Total callbacks: 392\n",
      "Total no callbacks: 4478\n",
      "Number of callbacks for black sounding names: 157\n",
      "Number of no callbacks for black sounding names: 2278\n",
      "Number of callbacks for white sounding names: 235\n",
      "Number of no callbacks for white sounding names: 2200\n"
     ]
    }
   ],
   "source": [
    "#Totals, Callbacks, and Proportions filtered by race\n",
    "\n",
    "b_all = data.race[data.race == \"b\"].count()\n",
    "w_all = data.race[data.race == \"w\"].count()\n",
    "\n",
    "call_all = data.call[data.call == 1].count()\n",
    "nocall_all = data.call[data.call == 0].count()\n",
    "\n",
    "b_callbacks = data.loc[(data.race == \"b\") & (data.call == 1), \"call\"].count()\n",
    "b_nocallbacks = data.loc[(data.race=='b') & (data.call == 0), \"call\"].count()\n",
    "\n",
    "w_callbacks = data.loc[(data.race == \"w\") & (data.call == 1), \"call\"].count()\n",
    "w_nocallbacks = data.loc[(data.race == \"w\") & (data.call == 0), \"call\"].count()\n",
    "\n",
    "print(\"Total number of black sounding names:\", b_all)\n",
    "print(\"Total number of white sounding names:\", w_all)\n",
    "\n",
    "print(\"Total callbacks:\", call_all)\n",
    "print(\"Total no callbacks:\", nocall_all)\n",
    "\n",
    "print(\"Number of callbacks for black sounding names:\", b_callbacks)\n",
    "print(\"Number of no callbacks for black sounding names:\", b_nocallbacks)\n",
    "print(\"Number of callbacks for white sounding names:\", w_callbacks)\n",
    "print(\"Number of no callbacks for white sounding names:\", w_nocallbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>race</th>\n",
       "      <th>b</th>\n",
       "      <th>w</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>call</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>2278</td>\n",
       "      <td>2200</td>\n",
       "      <td>4478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>157</td>\n",
       "      <td>235</td>\n",
       "      <td>392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>2435</td>\n",
       "      <td>2435</td>\n",
       "      <td>4870</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "race     b     w   All\n",
       "call                  \n",
       "0.0   2278  2200  4478\n",
       "1.0    157   235   392\n",
       "All   2435  2435  4870"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "##Create a frequency table for callback status by race \n",
    "callback_race_table = pd.crosstab(index=data.call, columns=data.race, margins=True)\n",
    "callback_race_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Callbacks by Race (Black (B) and White (W)):\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>B</th>\n",
       "      <th>W</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>No</th>\n",
       "      <td>2278</td>\n",
       "      <td>2200</td>\n",
       "      <td>4478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Yes</th>\n",
       "      <td>157</td>\n",
       "      <td>235</td>\n",
       "      <td>392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Total</th>\n",
       "      <td>2435</td>\n",
       "      <td>2435</td>\n",
       "      <td>4870</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          B     W  Total\n",
       "No     2278  2200   4478\n",
       "Yes     157   235    392\n",
       "Total  2435  2435   4870"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "callback_race_table.index=['No', 'Yes', 'Total']\n",
    "callback_race_table.columns=['B', 'W', 'Total']\n",
    "print(\"Callbacks by Race (Black (B) and White (W)):\")\n",
    "callback_race_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion: The sample size is large enough to use CTL, and we will test with a proportional z-test to evaluate the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question 2: What are the null and alternate hypotheses?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the table above, it appears that white names seem to receive more callbacks (235/2435) and black names receive less (157/2435). \n",
    "\n",
    "In order to further evaluate this, the null and alternative hypothesises are stated as follows: \n",
    "* Null Hypothesis: The effect of \"race\" (B) = 0\n",
    "* Alternative Hypoethesis: The effect of \"race\" (B) > 0\n",
    "\n",
    "Our significant level is 95%, with a = 0.05. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question 3: Compute margin of error, confidence interval, and p-value. \n",
    "* Try using both the bootstrapping and the frequentist statistical approaches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The mean callback is: 0.08049281686544418\n",
      "The std callback is: 0.27205151319503784\n",
      "The length of the data is: 4870\n"
     ]
    }
   ],
   "source": [
    "#Start by computing the mean and standard deviation\n",
    "mean_callbacks = np.mean(data.call)\n",
    "std_callbacks = np.std(data.call)\n",
    "n_data = len(data)\n",
    "\n",
    "#NOTE: the \"call\" field is a binary variable\n",
    "print(\"The mean callback is:\", mean_callback)\n",
    "print(\"The std callback is:\", std_callback)\n",
    "print(\"The length of the data is:\", n_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The empirical difference of the mean between w and b callbacks: 0.03203285485506058\n"
     ]
    }
   ],
   "source": [
    "mean_b_callbacks = np.mean(data.call[data.race == \"b\"])\n",
    "mean_w_callbacks = np.mean(data.call[data.race == \"w\"])\n",
    "\n",
    "empirical_diff_means = abs(mean_b_callbacks - mean_w_callbacks)\n",
    "print(\"The empirical difference of the mean between w and b callbacks:\", empirical_diff_means)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The empirical difference of the mean between w and b callbacks can also be interpretted as the difference in proportion of callbacks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First, start with bootstrapping for p-value hypothesis testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Start bootstrapping approach to statistical evaluation\n",
    "##Replicate of resampled data array function\n",
    "\n",
    "def reps(data, function): \n",
    "    \"\"\"Return replicate of a resampled data array.\"\"\"\n",
    "    \n",
    "    #Create resampled array and return stat:\n",
    "    return function(np.random.choice(data, size=len(data)))\n",
    "\n",
    "#Bootstrap replicates function\n",
    "\n",
    "def draw_bsreps(data, func, size=1):\n",
    "    \"\"\"Draw bootstrap replicates.\"\"\"\n",
    "    \n",
    "    #Initialize array of replicates\n",
    "    bs_replicates = np.empty(size)\n",
    "    \n",
    "    #Generate replicates\n",
    "    for i in range(size): \n",
    "        bs_replicates[i] = reps(data, func)\n",
    "        \n",
    "    return bs_replicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The p-value of the replicates is: 0.0\n"
     ]
    }
   ],
   "source": [
    "#Seed the random number generator\n",
    "np.random.seed(25)\n",
    "\n",
    "#Generate shifted arrays - we are doing a two-sided test\n",
    "b_shift = b_callbacks - mean_b_callbacks + mean_callbacks\n",
    "w_shift = w_callbacks - mean_w_callbacks + mean_callbacks\n",
    "\n",
    "#Generate 10,000 bootstrap replicates from our shift arrays\n",
    "bsreps_b = draw_bsreps(b_shift, np.mean, 10000)\n",
    "bsreps_w = draw_bsreps(w_shift, np.mean, 10000)\n",
    "\n",
    "#Compute the difference of means of the replicates\n",
    "bsreps_diff_means = bsreps_b - bsreps_w\n",
    "\n",
    "#Compute p-value\n",
    "p = np.sum(bsreps_diff_means <= empirical_diff_means) / len(bsreps_diff_means)\n",
    "print(\"The p-value of the replicates is:\", p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: This small p-value gives us strong evidence against the null hypothesis (there is a higher than 0.05 chance of observing proof in favour of the alternative hypothesis). Therefore, we cannot claim that the mean callback rate is the same for the black names and the white names. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Second, try with frequentist statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.6448536269514722\n"
     ]
    }
   ],
   "source": [
    "from scipy.stats import norm\n",
    "\n",
    "#Start by computing margin of error and confidence interval\n",
    "\n",
    "z_critic = norm.ppf(0.95)\n",
    "print(z_critic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The proportion of all white sounding names is: 0.5\n",
      "The proportion of all black sounding names is: 0.5\n",
      "The proportion of black sounding names with callback is: 0.06447638603696099\n",
      "The proportion of black sound names with no callback is: 0.9355236139630391\n",
      "The proportion of white sounding names with callback is: 0.09650924024640657\n",
      "The proportion of white sounding names with no callback is: 0.9034907597535934\n"
     ]
    }
   ],
   "source": [
    "#To calculate the standard error, we need to compute probabilities of the Bernoulli distribution\n",
    "\n",
    "#b_all\n",
    "#w_all\n",
    "\n",
    "#call_all\n",
    "#nocall_all\n",
    "\n",
    "#b_callbacks \n",
    "#b_nocallbacks\n",
    "\n",
    "#w_callbacks\n",
    "#w_nocallbacks \n",
    "\n",
    "w_prop_all = (w_all) / len(data)\n",
    "b_prop_all = (b_all) / len(data)\n",
    "\n",
    "b_prop_call = (b_callbacks) / (b_all)\n",
    "b_prop_nocall = (b_nocallbacks) / (b_all)\n",
    "\n",
    "w_prop_call = (w_callbacks) / (w_all)\n",
    "w_prop_nocall = (w_nocallbacks) / (w_all)\n",
    "\n",
    "\n",
    "print(\"The proportion of all white sounding names is:\", w_prop_all)\n",
    "print(\"The proportion of all black sounding names is:\", b_prop_all)\n",
    "\n",
    "print(\"The proportion of black sounding names with callback is:\", b_prop_call)\n",
    "print(\"The proportion of black sound names with no callback is:\", b_prop_nocall)\n",
    "\n",
    "print(\"The proportion of white sounding names with callback is:\", w_prop_call)\n",
    "print(\"The proportion of white sounding names with no callback is:\", w_prop_nocall)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The standard error for white names is: 0.001955777357746891\n",
      "The standard error for black names is: 0.001306625724111752\n",
      "The standard error is: 0.0023520918043277925\n"
     ]
    }
   ],
   "source": [
    "#Compute the standard error of each b and w\n",
    "w_std_error = np.sqrt((w_prop_call * (1-w_prop_nocall))/(w_all))\n",
    "b_std_error = np.sqrt((b_prop_call * (1-b_prop_nocall))/(b_all))\n",
    "\n",
    "std_error = np.sqrt((w_prop_call * (1-w_prop_nocall))/(w_all) + (b_prop_call * (1-b_prop_nocall))/(b_all))\n",
    "\n",
    "print(\"The standard error for white names is:\", w_std_error)\n",
    "print(\"The standard error for black names is:\", b_std_error)\n",
    "print(\"The standard error is:\", std_error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The z critical value is 1.959963984540054\n",
      "The margin of error is 0.0046100152248143055\n"
     ]
    }
   ],
   "source": [
    "#Finally, compute the margine of error, confidence interval and p-value\n",
    "\n",
    "##Calcaulte the z critical value\n",
    "z_critic = norm.ppf(0.975)\n",
    "print(\"The z critical value is\", z_critic)\n",
    "\n",
    "##Calculate the moe\n",
    "moe = z_critic * std_error\n",
    "print(\"The margin of error is\", moe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: This means that the difference in the proportion of names receiving call backs is the difference of the means +/- the margin of error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The empirical difference of the mean between the two groups receiving callbacks is: 0.03203285485506058\n",
      "With +/- 0.0046100152248143055\n",
      "Which equals 0.027422839630246273 and 0.036642870079874885\n"
     ]
    }
   ],
   "source": [
    "print(\"The empirical difference of the mean between the two groups receiving callbacks is:\", empirical_diff_means)\n",
    "print(\"With +/-\", moe)\n",
    "print(\"Which equals\", empirical_diff_means - moe, \"and\", empirical_diff_means + moe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With a 95% confidence interval, the range is: 0.027422839630246273 and 0.036642870079874885\n"
     ]
    }
   ],
   "source": [
    "#Computing the confidence interval another way\n",
    "[CIU, CIL] = z_critic * std_error * np.array([-1,1])\n",
    "CIUpper = empirical_diff_means + CIU\n",
    "CILower = empirical_diff_means + CIL\n",
    "\n",
    "print(\"With a 95% confidence interval, the range is:\", CIUpper, \"and\", CILower)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.108412152434346 3.983886837585077e-05\n"
     ]
    }
   ],
   "source": [
    "#Computing the z-statistic with proportions\n",
    "import statsmodels.stats.proportion as props\n",
    "\n",
    "z_test, p = props.proportions_ztest([w_callbacks, b_callbacks], [w_all, b_all])\n",
    "print(z_test, p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The z-statistic of 4.11 from a proportional test and small p-value also indicate that we should reject the null hypothesis that race has no effect on callbacks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion: Both boostrapping a p-value hypothesis test and computing frequentist statistics indicate that we should reject the null hypothesis and that race has an effect on callbacks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question 4: Write a story.\n",
    "* Describing the statistical significance in the context or the original problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The original problem asked us to evaluate whether or not race, as identified by either a \"black\" name or a \"white\" name (which were randomly distributed), has a statistically significant impact on resume callback rate. This analysis proposed a null hypothesis that race does not have an effect and an alternative hypothesis that race does have an effect. The analysis used both p-value bootstrapping and frequentist statistical methods to evaluate the data provided. The analysis rejects the null hypothesis with evidence from both methods, as Z is far outside the critical value of +/- 1.96 at 4.11. Both tests also found a low p-value below 0.05. We are 95% confident that black sounding names are less likely to be called back and conclude that black sounding names are more than 4 standard deviations from the mean, compared to white sounding names. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question 5: Does your analysis mean that race/name is the most important factor in callback success? \n",
    "* Why or why not? If not, how would you amend your analysis?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The analysis does not conclude findings on other variables in the data. The analysis would have to include those variables and compare the effects on callback rate and compared to the effect of race/name. This could include a correlation analysis or logistical regression methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['id', 'ad', 'education', 'ofjobs', 'yearsexp', 'honors', 'volunteer',\n",
       "       'military', 'empholes', 'occupspecific', 'occupbroad', 'workinschool',\n",
       "       'email', 'computerskills', 'specialskills', 'firstname', 'sex', 'race',\n",
       "       'h', 'l', 'call', 'city', 'kind', 'adid', 'fracblack', 'fracwhite',\n",
       "       'lmedhhinc', 'fracdropout', 'fraccolp', 'linc', 'col', 'expminreq',\n",
       "       'schoolreq', 'eoe', 'parent_sales', 'parent_emp', 'branch_sales',\n",
       "       'branch_emp', 'fed', 'fracblack_empzip', 'fracwhite_empzip',\n",
       "       'lmedhhinc_empzip', 'fracdropout_empzip', 'fraccolp_empzip',\n",
       "       'linc_empzip', 'manager', 'supervisor', 'secretary', 'offsupport',\n",
       "       'salesrep', 'retailsales', 'req', 'expreq', 'comreq', 'educreq',\n",
       "       'compreq', 'orgreq', 'manuf', 'transcom', 'bankreal', 'trade',\n",
       "       'busservice', 'othservice', 'missind', 'ownership'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Let's take a look at the other variables\n",
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Callback rate would likely be influenced by a few of these: computerskills, specialskills, city, etc. as they are likely to indicate match potential for a job. Additionally, all humans are susceptible to bias and error in judgment. Likely variables that could bias callback rate could include sex, as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAFfCAYAAACY4WFaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAH2JJREFUeJzt3XmYJVV9//H3B1BBNlGWIIsrEokLy4BrFFwQcAGMG6LgElEB0agoGgMEojGiYhA3VCIYFVFB0SCLRERcGQRZBH+gEBhAGURBBVHw+/ujTst16O65s5zpmeH9ep5++ta5p059b/dMf25VnVuVqkKSpJ5WmOkCJEnLP8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2mnFJKslD2+NPJfm39njbJHM6bO8v21jajf5spGWZYaPFIsmLk8xO8rsk1yX5epInznRdiyrJmUn+0F7XDUlOSLL+Aqy/VIRFknsmeV+SOe21XJHk8JmuS3cfho0WWZI3Ah8A3gWsB2wMfBjYeSbrWoz2rarVgIcCqwHvneF6FsbbgFnANsDqwHbAeTNake5WDBstkiRrAocA+1TVCVX1+6r6U1V9tar2b322SfK9JL9pez1HJrnnAmzj7W2v4soku4+0PzPJeUluTnJ1koPnWe+JSb7btnt1kpdNMvbqSb6Z5Igkma6OqvoN8GVg85H1p3xtSc5q3X7c9iZe2NqfleT8ts53kzxqPj+CnZL8vP0MDkuyQpJ7JbkxySNHalk3ya1J1plkjK2BE6vq2hpcWVXHjqx7/yRfSjK37fXsN/LcyUneN7L8+SRHz6dm6a8YNlpUjwNWBk6cps8dwD8Ba7f+TwX2HnP8v2nrbQDsCRyVZNP23O+BPYD7AM8EXptkF4AkGwNfBz4IrMMQEOePDpzkfsAZwHeqar+az7WbWv/nApeP89qq6kmtz6OrarWq+nySLYGjgVcD9wM+BpyU5F7TbHpXhr2SLRn2Fl9RVbcBxwEvGem3G/CNqpo7yRjfB96YZO8kjxwN1iQrAF8Ffszwc34q8IYkz2hdXgG8NMlTWthvDbx+mnqlu6oqv/xa6C9gd+AXC7jOGxjeZU8sF/DQ9vhTwL+1x9sCtwOrjvQ9HviXKcb9AHB4e/y20W3M0+9TDH/wLwL2n0+tZwK3ADe1Os8HNl6Y19aWPwIcOs86PwWePMV4Bewwsrw3cEZ7/BjgamCFtjwbeMEU46wI7AN8B7gNuBbYc2Scq+bp/zbgv0aWn9u2dQPwxJn+d+fXsvflno0W1a+AtZOsNFWHJA9L8rUkv0hyM8O5nbXHHP/XVfX7keX/A+7fxn1MOwQ2N8lNwGtGxt0I+Nk04z4TWAX46Bg17FdVawKPAtYCNlyE1/YA4E3tENpvkvym1Xr/ada5euTxX15/Vf2AYe/uyUn+luGc0kmTDVBVd1TVh6rqCQx7gu8Ejk7y8FbT/eep6e0M598mfI0hsH5aVWdPU6s0KcNGi+p7wB+AXabp8xHgUmCTqlqD4Q/ZtOdHRqyVZNWR5Y0Z3pUDfJbhj+tGLQw+OjLu1cBDphn348ApwMnzjD+lqroQ+DfgQyOHoRb0tV0NvLOq7jPyde+q+tw062w08nj09QMcw3Ao7aXAF6vqD2O8jlur6kPAr4HNWk1XzFPT6lW108hq7wQuAdZPstv8tiHNy7DRIqmqm4ADGf4A75Lk3knukWTHJO9p3VYHbgZ+196Bv3YBN/Ovberu3wPPAr4wMu6NVfWHJNsALx5Z5zPA05K8IMlKSe6XZPN5xt2X4RDW15KsMmYtxwDrAs8Z87X9EnjwyPLHgde0vbIkWbVNdFh9mm3un2StJBsxnCv5/Mhzn2Y4p/MS4NjJVgZI8oYMn1tapf089my1nwf8ELg5yVvb8ysmeUSSrdu6TwJeznB+bA/gg0k2mKZe6S4MGy2yqno/8EbgHcBchnfK+zLM3AJ4M0MQ/Jbhj+3nJxlmKr9geAd+LUOAvKaqLm3P7Q0ckuS3DIF3/EhNVwE7AW8CbmQ41/LoeeouYK9W71eSrDzGa/0jcATwL2O+toOBY9rhqRdU1WzgVcCR7XVdDrxsPpv9CnBuew3/A3xypJ45wI8Yzu18e5oxbgXex/DzvIHh/M0/VNXPq+oO4NkMkyiuaM9/AlgzyRoMIbZvVV3TDqF9Eviv+c3ek0Zl+P8maVnVpiFfW1XvmOlapKlMeVJX0tIvyQMZZoptMbOVSNPzMJq0jEpyKMP07cOq6oqZrkeajofRJEnduWcjSerOsJEkdbdcThDYYYcd6pRTTpnpMiRpcVguppgvl3s2N9xww0yXIEkasVyGjSRp6WLYSJK6M2wkSd0ZNpKk7gwbSVJ3ho0kqTvDRpLUnWEjSerOsJEkdWfYSJK6M2wkSd0ZNpKk7pbLqz5rwV11yCNnuoQZsfGBF850CdLdgns2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuvILAJLba/9iZLmGJO3H1ma5A0vLMPRtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR11y1skmyU5JtJLklycZLXt/b7Jjk9yWXt+1qtPUmOSHJ5kguSbDky1p6t/2VJ9uxVsySpj557NrcDb6qqhwOPBfZJshlwAHBGVW0CnNGWAXYENmlfewEfgSGcgIOAxwDbAAdNBJQkadnQLWyq6rqq+lF7/FvgEmADYGfgmNbtGGCX9nhn4NgafB+4T5L1gWcAp1fVjVX1a+B0YIdedUuSFr8lcs4myQOBLYAfAOtV1XUwBBKwbuu2AXD1yGpzWttU7ZKkZUT3sEmyGvAl4A1VdfN0XSdpq2na593OXklmJ5k9d+7chStWktRF17BJcg+GoPlMVZ3Qmn/ZDo/Rvl/f2ucAG42sviFw7TTtf6WqjqqqWVU1a5111lm8L0SStEh6zkYL8Engkqp6/8hTJwETM8r2BL4y0r5Hm5X2WOCmdpjtVGD7JGu1iQHbtzZJ0jJipY5jPwF4KXBhkvNb29uBdwPHJ3klcBXw/PbcycBOwOXALcDLAarqxiSHAue0fodU1Y0d65YkLWbdwqaqzmby8y0AT52kfwH7TDHW0cDRi686SdKS5BUEJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHXXLWySHJ3k+iQXjbQdnOSaJOe3r51GnntbksuT/DTJM0bad2htlyc5oFe9kqR+eu7ZfArYYZL2w6tq8/Z1MkCSzYAXAX/X1vlwkhWTrAh8CNgR2AzYrfWVJC1DVuo1cFWdleSBY3bfGTiuqm4DrkhyObBNe+7yqvo5QJLjWt+fLOZyJUkdzcQ5m32TXNAOs63V2jYArh7pM6e1TdV+F0n2SjI7yey5c+f2qFuStJCWdNh8BHgIsDlwHfC+1p5J+tY07XdtrDqqqmZV1ax11llncdQqSVpMuh1Gm0xV/XLicZKPA19ri3OAjUa6bghc2x5P1S5JWkYs0T2bJOuPLO4KTMxUOwl4UZJ7JXkQsAnwQ+AcYJMkD0pyT4ZJBCctyZolSYuu255Nks8B2wJrJ5kDHARsm2RzhkNhVwKvBqiqi5Mcz3Di/3Zgn6q6o42zL3AqsCJwdFVd3KtmSVIfPWej7TZJ8yen6f9O4J2TtJ8MnLwYS5MkLWFeQUCS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKm7scImySN6FyJJWn6Nu2fz0SQ/TLJ3kvt0rUiStNwZK2yq6onA7gxXYJ6d5LNJnt61MknScmPsczZVdRnwDuCtwJOBI5JcmuS5vYqTJC0fxj1n86gkhwOXAE8Bnl1VD2+PD+9YnyRpOTDuVZ+PBD4OvL2qbp1orKprk7yjS2WSpOXGuGGzE3DryD1mVgBWrqpbqurT3aqTJJHkn4EXA3cAf2a4F9h/AOsDEzsAl1fV85IcAcytqkNH1r1/Ve2z5Cu/07hh8w3gacDv2vK9gdOAx/coSpI0SPI44FnAllV1W5K1gXu2p3evqtnzrPIO4Pwkn2G4UeU/AlsssYKnMG7YrFxVE0FDVf0uyb071SRJutP6wA1VdRtAVd0AkGTSzlV1c9ubObI1HVhVv1kShU5n3Nlov0+y5cRCkq24c9dNktTPacBGSf5fkg8nefLIc59Jcn77Omyisao+B6wFrLG0nOoYd8/mDcAXklzbltcHXtinJEnShHYkaSvg74HtgM8nOaA9PdlhNJJsCPwNUElWGz0yNVPGCpuqOifJ3wKbAgEurao/da1MkgRAm5x1JnBmkguBPeezyn8CBwMPBw4C9u9Z3zjG3bMB2Bp4YFtniyRU1bFdqpIkAZBkU+DP7YP1AJsD/wdMes3KJDsC6wLHMkzm+nGS/6qqnyyJeqcyVtgk+TTwEOB8hql3MMxyMGwkqa/VgA+261LeDlwO7AV8keGczcT58xsYZq19AHheVRXD+fa3MEwWeMoSr3zEuHs2s4DNWvGSpCWkqs5l8o+ZbDvFKpvOs/4JwAmLuawFNu5stIsYTjZJkrTAxt2zWRv4SZIfArdNNFbVc7pUJUlarowbNgf3LEKStHwbd+rzt5I8ANikqr7Rrh6wYt/SJEnLi3FvMfAqhpkPH2tNGwBf7lWUJGn5Mu4EgX2AJwA3w19upLZur6IkScuXcc/Z3FZVf5y48FuSlRg+ZyNJd3tb7b94P3J47mF7zLdPkjuAC0eadqmqKxdrIXdu62XArKrad2HHGDdsvpXk7cAqSZ4O7A18dWE3KklaZLdW1eYzXcS4xj2MdgAwlyFFXw2czHDPBEnSUiLJikkOS3JOkguSvLq1b5vkW0mOb1ePfneS3ZP8MMmFSR7S+j07yQ+SnJfkG0nWm2Qb6yT5UtvGOUmeME5t485G+zPDbaE/Pv7LliR1tEqS89vjK6pqV+CVwE1VtXWSewHfSXJa6/Nohgtz3gj8HPhEVW2T5PXA6xiu7n828NiqqiT/CLwFeNM82/1P4PCqOjvJxsCpbdxpjXtttCuY5BxNVT14nPUlSYvdZIfRtgceleR5bXlNYBPgj8A5VXUdQJKfMdwnB4YjVtu1xxsy3MJgfYa7gV4xyXafBmw2cvO2NZKsXlW/na7YBbk22oSVgecD9x1zXUnSkhHgdVV16l81JtsycvUX4M8jy3/mziz4IPD+qjqprXPwJNtYAXhcVS3QDTTHOmdTVb8a+bqmqj7ADF9BVJJ0F6cCr01yD4AkD0uy6gKsvyZwTXs81T1zTgP+MistyViTFMY9jLblyOIKDHs6q4+zriQt78aZqryEfILhvmM/ynCcay6wywKsfzDDXZmvAb4PPGiSPvsBH0pyAUOGnAW8Zn4DZ5y7BiT55sji7cCVwHur6qfzXXkGzJo1q2bPvsudUse2uOfMLwtOXP2w+XdaDm184IXz7yTNrMy/y9Jv3Nlo282/lyRJkxv3MNobp3u+qt6/eMqRJC2PFmQ22tbASW352QzH6a7uUZQkafmyIDdP23JiHnWSg4EvVNU/9ipMkrT8GPdyNRszfChowh8ZZjxIkjRf4+7ZfBr4YZITGa4ksCtw95uyJUlaKOPORntnkq8Df9+aXl5V5/UrS5KWHVcd8sjFOt44U/KTFPDfVfXStrwScB3wg6p61jTrbQu8ebo+PYx7GA3g3sDNVfWfwJwkk33YR5K0ZPweeESSVdry07nz0/9LnXFvC30Q8Fbgba3pHsB/9ypKkjSWrwPPbI93Az438USSbZJ8t90u4LtJNp135SSrJjm63SrgvCQ79yp03D2bXYHnMCQpVXUtXq5GkmbaccCLkqwMPAr4wchzlwJPqqotgAOBd02y/j8D/1tVWzNc+fmwBbyW2tjGnSDwx3Z/g4IhDXsUI0kaX1VdkOSBDHs1J8/z9JrAMUk2YZjYdY9JhtgeeE6SN7fllRlmH1+yuGsdN2yOT/Ix4D5JXgW8Am+kJklLg5OA9wLbAvcbaT8U+GZV7doC6cxJ1g3wD0viOpfjzkZ7b5KnAzcDmwIHVtXpXSuTJI3jaIa7c17YZppNGL1dwMumWPdU4HVJXteOXm3Ra6bxfMMmyYrAqVX1NMCAkaR5zOTVw6tqDsOtmuf1HobDaG8E/neK1Q8FPgBc0G5JcCXQZUr0fMOmqu5IckuSNavqph5FSJIWTFWtNknbmbTDZVX1PeBhI0//yyR9bgVe3bXQZtxzNn8ALkxyOm1GGkBV7delKknScmXcqc//w5CKZwHnjnxNqc3dvj7JRSNt901yepLL2ve1WnuSHJHk8iQXjN4ZNMmerf9lSaa6TakkaSk27Z5Nko2r6qqqOmYhxv4UcCR/fQ21A4AzqurdSQ5oy28FdgQ2aV+PAT4CPCbJfYGDGG5xUMC5SU6qql8vRD2SpBkyvz2bL088SPKlBRm4qs4CbpyneWdgIriO4c57Y+8MHFuD7zNMsV4feAZwelXd2ALmdGCHBalDkjTz5hc2o/e+fvBi2N56VXUdQPu+bmvfgL++Educ1jZVuyRpGTK/sKkpHi9umaStpmm/6wDJXklmJ5k9d+7cxVqcJGnRzC9sHp3k5iS/BR7VHt+c5LdJbl6I7f2yHR6jfb++tc8BNhrptyFw7TTtd1FVR1XVrKqatc466yxEaZKkXqYNm6pasarWqKrVq2ql9nhieY2F2N5JwMSMsj2Br4y079FmpT2W4dOw1zF8unX7JGu1mWvbtzZJ0jJk3M/ZLLAkn2O4Vs/aSeYwzCp7N8N11l4JXAU8v3U/GdgJuBy4BXg5QFXdmORQ4JzW75CqmnfSgSRpKdctbKpqtymeeuokfQvYZ4pxjma49o8kaRm1IHfqlCRpoRg2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1NyNhk+TKJBcmOT/J7NZ23ySnJ7msfV+rtSfJEUkuT3JBki1nomZJ0sKbyT2b7apq86qa1ZYPAM6oqk2AM9oywI7AJu1rL+AjS7xSSdIiWZoOo+0MHNMeHwPsMtJ+bA2+D9wnyfozUaAkaeHMVNgUcFqSc5Ps1drWq6rrANr3dVv7BsDVI+vOaW1/JcleSWYnmT137tyOpUuSFtRKM7TdJ1TVtUnWBU5Pcuk0fTNJW92loeoo4CiAWbNm3eV5SdLMmZE9m6q6tn2/HjgR2Ab45cThsfb9+tZ9DrDRyOobAtcuuWolSYtqiYdNklWTrD7xGNgeuAg4CdizddsT+Ep7fBKwR5uV9ljgponDbZKkZcNMHEZbDzgxycT2P1tVpyQ5Bzg+ySuBq4Dnt/4nAzsBlwO3AC9f8iVLkhbFEg+bqvo58OhJ2n8FPHWS9gL2WQKlSZI6WZqmPkuSllOGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6m6lmS5Aurvaav9jZ7qEJe7cw/aY6RI0Q9yzkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR15xUEJC0xVx3yyJkuYUZsfOCFM13CjHPPRpLUnWEjSerOsJEkdWfYSJK6M2wkSd0ZNpKk7gwbSVJ3ho0kqTvDRpLUnWEjSerOsJEkdWfYSJK6M2wkSd0ZNpKk7gwbSVJ3y0zYJNkhyU+TXJ7kgJmuR5I0vmUibJKsCHwI2BHYDNgtyWYzW5UkaVzLRNgA2wCXV9XPq+qPwHHAzjNckyRpTMtK2GwAXD2yPKe1SZKWASvNdAFjyiRt9Vcdkr2Avdri75L8tHtVy5EHwNrADTNdxxJ30GT/tNSL/84WyilVtcPiKmWmLCthMwfYaGR5Q+Da0Q5VdRRw1JIsanmSZHZVzZrpOrR889/Z3deychjtHGCTJA9Kck/gRcBJM1yTJGlMy8SeTVXdnmRf4FRgReDoqrp4hsuSJI1pmQgbgKo6GTh5putYjnkIUkuC/87uplJV8+8lSdIiWFbO2UiSlmGGzVIqSSV538jym5McvIBj7JhkdpJLklya5L3z6f+yJEe2xwcneXN7fGaSRZ5BlOTKJGsv6jgaJPmbJMcl+VmSnyQ5OcnD5rPO79r3Bya5qD3+y+99Eev5y7+Z+fS5Jsn5rebdFnW7k2zj5CT3WQzjzPf1aHyGzdLrNuC5C/vHOckjgCOBl1TVw4FHAD9fjPVpBiUJcCJwZlU9pKo2A94OrDezlY3l8KranOEqIB9Lco/FOXhV7VRVv1mcY2rRGTZLr9sZTqb+07xPJHlAkjOSXNC+bzzJ+m8B3llVl8Iwo6+qPtzWf3aSHyQ5L8k3kozzB+olSb6b5KIk27Rxtmlt57Xvm7b2FZO8N8mFrcbXzVP/KklOSfKqBfqJaNR2wJ+q6qMTDVV1flV9O8lq7d/Fj9rvYJxLO23Ufic/TXLQRGOSLyc5N8nF7YPTE+07tPF/nOSMeQdL8qokX0+yylQbrKrLgFuAtdo6D2k1nJvk20n+trWvl+TEtq0fJ3l8a39Jkh+2vaSPtWso/mUPOsl/JNl7pKaDk7ypPd4/yTnt3+e/jvT55/Yz+Aaw6Rg/N43JsFm6fQjYPcma87QfCRxbVY8CPgMcMcm6jwDOnWLcs4HHVtUWDNeZe8sYtaxaVY8H9gaObm2XAk9q4xwIvKu17wU8CNhipMYJqwFfBT5bVR8fY7ua3HS/3z8Au1bVlgyh9L62JzSdbYDdgc2B548cNn1FVW0FzAL2S3K/JOsAHwf+oaoeDTx/dKD2MYVnA7tU1a1TbTDJlsBlVXV9azoKeF3b3puBD7f2I4BvtW1tCVyc5OHAC4EntL2kO1r9o45rfSa8APhCku2BTdpr3hzYKsmTkmzF8Bm+LYDnAltP/yPTglhmpj7fHVXVzUmOBfYDRv/TPo7hPwPAp4H3LODQGwKfT7I+cE/gijHW+Vyr6awka7Rj4qsDxyTZhOHyQROHQ54GfLSqbm/r3DgyzleA91TVaABp8QrwriRPAv7McB3B9YBfTLPO6VX1K4AkJwBPBGYzBMyurc9GDH+k1wHOqqor4C6/35cyXPFjl6r60xTb+qe2V/tgYIe2zdWAxzOEwUS/e7XvTwH2aNu6A7gpyUuBrYBzWv9VgInQovU9L8m6Se7fav51VV2VZD9ge+C81nW19rpWB06sqltaTX5wfDFyz2bp9wHglcCq0/SZbP76xQz/GSfzQeDIqnok8Gpg5THqmHcbBRwKfLOqHsHwTnZinExRE8B3gB3HeKet6U33+92d4Y/rVu1d/y+Z/+/4Lr/fJNsyvHF4XNurOK+NM93v9yLggQxvaKZyeFVtyrDXcWySlRn+Fv2mqjYf+Xr4NGMEOGak76ZVdfAk/b4IPK9t67iRdf99ZN2HVtUnJ173NNvUIjBslnLtXePxDIEz4bsMu/sw/GE5e5JVDwPenjY7KckKSd7YnlsTuKY93nPMUl7YxnkicFNV3TTPOC8b6Xsa8JokK7V17jvy3IHAr7jzEIkWzv8C9xo975Vk6yRPZvi9XF9Vf0qyHfCAMcZ7epL7tnMsuzC8KViTYW/glnb+5LGt7/eAJyd5UNvu6O/3PIY3MCe1PYopVdUJDHtPe1bVzcAVSZ7fxkySR7euZwCvbe0rJlmjtT0vyboTNSSZ7HUex/B/5XkMwQPDlUhe0famSLJBG+csYNd2TnF1hjdQWkwMm2XD+xiuljthP+DlSS5gOGzx+nlXqKoLgDcAn0tyCcM7zvXb0wczHK74NuNfgffXSb4LfJQ7g+89wL8n+Q7DZYQmfAK4CrggyY+BF88z1huAlZMs6OE/NTV8GntXhpD4WZKLGX6v1zKcI5uVZDbDm5FLxxjybIZDsucDX6qq2cApwErt39mhwPfbtucynJc7of1+Pz9PbWcznHP5n8x/NuUhwBuTrNBqfWUb82LuvGfV64HtklzIcJ7q76rqJ8A7gNNafadz57/v0VouZjg8dk1VXdfaTgM+C3yvjflFYPWq+lF7LecDXwK+Pf8fm8blFQQkSd25ZyNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktTd/wdc87UiOTyhJgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 404.125x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "#Let's quickly get a glimpse to see if sex could have an effect\n",
    "fig = sns.catplot(x='call',hue='race', data=data, kind='count')\n",
    "\n",
    "fig._legend.set_title('SEX')\n",
    "for text, label in zip(\n",
    "    fig._legend.texts, \n",
    "    ['Female', 'Male']\n",
    "): text.set_text(label)\n",
    "fig.set_xticklabels(['No Callback','Callback Received'])\n",
    "fig = plt.xlabel('')\n",
    "fig = plt.ylabel('Frequency')\n",
    "fig = plt.title('Callback Rate by Sex')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: This just shows that evaluating race in its effect on callback rate should be put in context / compared to other factors in the data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}