african_wells.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<style>\n",
    "@font-face {\n",
    "  font-family: CharisSILW;\n",
    "  src: url(files/CharisSIL-R.woff);\n",
    "}\n",
    "@font-face {\n",
    "  font-family: CharisSILW;\n",
    "  font-style: italic;\n",
    "  src: url(files/CharisSIL-I.woff);\n",
    "}\n",
    "@font-face {\n",
    "\tfont-family: CharisSILW;\n",
    "\tfont-weight: bold;\n",
    "\tsrc: url(files/CharisSIL-B.woff);\n",
    "}\n",
    "@font-face {\n",
    "\tfont-family: CharisSILW;\n",
    "\tfont-weight: bold;\n",
    "\tfont-style: italic;\n",
    "\tsrc: url(files/CharisSIL-BI.woff);\n",
    "}\n",
    "\n",
    "div.cell, div.text_cell_render{\n",
    "    max-width:1000px;\n",
    "}\n",
    "\n",
    "h1 {\n",
    "    text-align:center;\n",
    "    font-family: Charis SIL, CharisSILW, serif;\n",
    "}\n",
    "\n",
    ".rendered_html {\n",
    "    font-size: 130%;\n",
    "    line-height: 1.3;\n",
    "}\n",
    "\n",
    ".rendered_html li {\n",
    "    line-height: 2;\n",
    "}\n",
    "\n",
    ".rendered_html h1{\n",
    "    line-height: 1.3;\n",
    "}\n",
    "\n",
    ".rendered_html h2{\n",
    "    line-height: 1.2;\n",
    "}\n",
    "\n",
    ".rendered_html h3{\n",
    "    line-height: 1.0;\n",
    "}\n",
    "\n",
    ".text_cell_render {\n",
    "    font-family: Charis SIL, CharisSILW, serif;\n",
    "    line-height: 145%;\n",
    "}\n",
    "\n",
    "li li {\n",
    "    font-size: 85%;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End-to-End Data Science in Python\n",
    "\n",
    "<img src=\"scikit-learn.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    " This is the workbook for the \"End-to-End Data Analysis in Python\" workshop\n",
    " at the Open Data Science Conference 2015, in beautiful San Francisco. \n",
    " This notebook contains starter code only; the goal is that we will fill in the\n",
    " gaps together as we progress through the workshop.  If, however, you're doing this\n",
    " asynchronously or you get stuck, you can reference the solutions workbook.\n",
    "\n",
    " The objective is to complete the \"Pump it Up: Mining the Water Table\" challenge\n",
    " on [drivendata.org](www.drivendata.org/competitions/7/); the objective here is to predict\n",
    " African wells that are non-functional or in need of repair.  Per the rules of the\n",
    " competition, you should register for an account with drivendata.org, at which point you\n",
    " can download the training set values and labels.  We will be working with those datasets\n",
    " during this workshop.  You should download those files to the directory in which this\n",
    " notebook lives, and name them wells_features.csv and wells_labels.csv (to be consistent\n",
    " with our nomenclature).  You are also encouraged to continue developing your solution\n",
    " after this workshop, and/or to enter your solution in the competition on the drivendata\n",
    " website!\n",
    " \n",
    " ### Code requirements\n",
    " Here's the environment you'll need to work with this code base:\n",
    "\n",
    " * python 3 (2.x may work with minor changes, but no guarantees)\n",
    " * pandas\n",
    " * scikit-learn\n",
    " * numpy\n",
    "\n",
    "## First Draft of an Analysis\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "  \n",
    "features_df = pd.DataFrame.from_csv(\"well_data.csv\")\n",
    "labels_df   = pd.DataFrame.from_csv(\"well_labels.csv\")  \n",
    "print( labels_df.head() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " One nice feature of ipython notebooks is it's easy to make small changes to code and\n",
    " then re-execute quickly, to see how things change.  For example, printing the first 5 lines\n",
    " of the labels dataframe (which is the default) isn't really ideal here, since there's a label\n",
    " (\"functional needs repair\") which doesn't appear in the first five lines.  Type 20 in the\n",
    " parentheses labels_df.head(), so it now reads labels_df.head(20), and press shift-enter to\n",
    " rerun the code.  See the difference?\n",
    " \n",
    " Now take a quick look at the features, again by calling .head() (set up for you in the code box\n",
    " below, or add your own code to the code box above).  You can print or as few\n",
    " rows as you like.  Take a quick look at the data--approximately how many features are there?\n",
    " Are they all numeric, or will you have to do work to transform non-numeric features into\n",
    " numbers?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print( features_df.head() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming string labels into integers\n",
    " The machine learning algorithms downstream are not going to handle it well if the class labels\n",
    " used for training are strings; instead, we'll want to use integers.  The mapping that we'll use\n",
    " is that \"non functional\" will be transformed to 0, \"functional needs repair\" will be 1, and\n",
    " \"functional\" becomes 2.\n",
    " \n",
    " There are a number of ways to do this; the framework below uses applymap() in pandas.\n",
    " [Here's](http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.applymap.html)\n",
    " the documentation for applymap(); in the code below, you should fill in the function body for\n",
    " label_map(y) so that if y is \"functional\", label_map returns 2; if y is \"functional needs\n",
    " repair\" then it should return 1, and \"non functional\" is 0. \n",
    " There's a print statement there to help you confirm that the label transformation is working\n",
    " properly.\n",
    " \n",
    " As an aside, you could also use apply() here if you like.  The difference between apply()\n",
    " and applymap() is that applymap() operates on a whole dataframe while apply() operates on a series\n",
    " (or you can think of it as operating on one column of your dataframe).  Since labels_df only has\n",
    " one column (aside from the index column), either one will work here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def label_map(y):\n",
    "   ### your code goes here\n",
    "labels_df = labels_df.applymap(label_map)\n",
    "\n",
    "print(labels_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming string features into integers\n",
    " \n",
    "Now that the labels are ready, we'll turn our attention to the features.  Many of the features\n",
    "are categorical, where a feature can take on one of a few discrete values, which are not ordered.\n",
    "Fill in the function body of ``transform_feature( df, column )`` below so that it takes our ``features_df`` and\n",
    "the name of a column in that dataframe, and returns the same dataframe but with the indicated\n",
    "feature encoded with integers rather than strings.\n",
    "\n",
    "We've provided code to wrap your transformer function in a loop iterating through all the columns that should\n",
    "be transformed.\n",
    "\n",
    "Last, add a line of code at the bottom of the block below that removes the ``date_recorded`` column from ``features_df``.  Time-series information like dates and times need special treatment, which we won't be going into today."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def transform_feature( df, column_name ):\n",
    "    ### your code goes here\n",
    "    return df\n",
    "\n",
    "### list of column names indicating which columns to transform; \n",
    "### this is just a start!  Use some of the print( labels_df.head() )\n",
    "### output upstream to help you decide which columns get the\n",
    "### transformation\n",
    "names_of_columns_to_transform = [\"funder\", \"installer\", \"wpt_name\"]\n",
    "for column in names_of_columns_to_transform:\n",
    "    features_df = transform_feature( features_df, column )\n",
    "    \n",
    "### remove the \"date_recorded\" column--we're not going to make use\n",
    "### of time-series data today"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, a couple last steps to get everything ready for sklearn.  The features and labels are taken out of their dataframes and put into a numpy.ndarray and list, respectively.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'features_df' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-dede2d6e4ee5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeatures_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_matrix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlabels_df\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"status_group\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtolist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'features_df' is not defined"
     ]
    }
   ],
   "source": [
    "X = features_df.as_matrix()\n",
    "y = labels_df[\"status_group\"].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Predicting well failures with logistic regression\n",
    "\n",
    "The cheapest and easiest way to train on one portion of your dataset and test on another, and to get a measure of model quality at the same time, is to use ``sklearn.cross_validation.cross_val_score()``.  This splits your data into 3 equal portions, trains on two of them, and tests on the third.  This process repeats 3 times.  That's why 3 numbers get printed in the code block below.\n",
    "\n",
    "You don't have to add anything to the code block, it's ready to go already.  However, use it for reference in the next part of the tutorial, where you will be looking at other sklearn algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(compute_importances=None,\n",
       "           criterion=<sklearn.tree._tree.FriedmanMSE object at 0x3e4eb28>,\n",
       "           max_depth=3, max_features=None, max_leaf_nodes=None,\n",
       "           min_density=None, min_samples_leaf=1, min_samples_split=2,\n",
       "           random_state=<mtrand.RandomState object at 0x7feca45d6660>,\n",
       "           splitter=<sklearn.tree._tree.PresortBestSplitter object at 0x3da15b0>)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sklearn.linear_model\n",
    "import sklearn.cross_validation\n",
    "\n",
    "clf = sklearn.linear_model.LogisticRegression()\n",
    "score = sklearn.cross_validation.cross_val_score( clf, X, y )\n",
    "print( score )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Comparing logistic regression to tree-based methods\n",
    "\n",
    "We have a baseline logistic regression model for well failures.  Let's compare to a couple of other classifiers, a decision tree classifier and a random forest classifier, to see which one seems to do the best.  \n",
    "\n",
    "Code this up on your own.  You can use the code in the box above as a kind of template, and just drop in the new classifiers.  The sklearn documentation might also be helpful:\n",
    "* [Decision tree classifier](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)\n",
    "* [Random forest classifier](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)\n",
    "\n",
    "We will talk about all three of these models more in the next part of the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations!  You have a working data science setup, in which you have:\n",
    "* read in data\n",
    "* transformed features and labels to make the data amenable to machine learning\n",
    "* made a train/test split (this was done implicitly when you called ``cross_val_score``)\n",
    "* evaluated several models for identifying wells that are failed or in danger of failing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Paying down technical debt and tuning the models\n",
    "\n",
    "We got things running really fast, which is great, but at the cost of being a little quick-and-dirty about some details.  First, we got the features encoded as integers, but they really should be dummy variables.  Second, it's worth going through the models a little more thoughtfully, to try to understand their performance and if there's any more juice we can get out of them.\n",
    "\n",
    "###One-hot encoding to make dummy variables\n",
    "A problem with representing categorical variables as integers is that integers are ordered, while categories are not.  The standard way to deal with this is to use dummy variables; one-hot encoding is a very common way of dummying.  Each possible category becomes a new boolean feature.  For example, if our dataframe looked like this: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``index      country\n",
    "1          \"United States\"\n",
    "2          \"Mexico\"\n",
    "3          \"Mexico\"\n",
    "4          \"Canada\"\n",
    "5          \"United States\"\n",
    "6          \"Canada\"``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "then after dummying it will look something like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``index      country_UnitedStates     country_Mexico    country_Canada\n",
    "1          1                        0                 0\n",
    "2          0                        1                 0\n",
    "3          0                        1                 0\n",
    "4          0                        0                 1\n",
    "5          1                        0                 0\n",
    "6          0                        0                 1``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully the origin of the name is clear--each variable is now encoded over several boolean columns, one of which is true (hot) and the others are false.\n",
    "\n",
    "Now we'll write a hot-encoder function that takes the data frame and the title of a column, and returns the same data frame but one-hot encoding performed on the indicated feature.\n",
    "\n",
    "Protip: sklearn has a [one-hot encoder](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) function available that will be your friend here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def hot_encoder(df, column_name)\n",
    "    ### your code goes here\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll take the ``to_transform`` list that you populated above with categorical variables, and use that to loop through columns that will be one-hot encoded.\n",
    "\n",
    "One note before you code that up: one-hot encoding comes with the baggage that it makes your dataset bigger--sometimes a lot bigger.  In the countries example above, one column that encoded the country has now been expanded out to three columns.  You can imagine that this can sometimes get really, really big (imagine a column encoding all the counties in the United States, for example).  \n",
    "\n",
    "There are some columns in this example that will really blow up the dataset, so we'll remove them before proceeding with the one-hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features_df.drop( \"funder\", axis=1, inplace=True )\n",
    "features_df.drop( \"installer\", axis=1, inplace=True )\n",
    "features_df.drop( \"wpt_name\", axis=1, inplace=True )\n",
    "features_df.drop( \"subvillage\", axis=1, inplace=True )\n",
    "features_df.drop( \"ward\", axis=1, inplace=True )\n",
    "\n",
    "for feature in to_transform:\n",
    "    features_df = hot_encode( features_df, feature )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the features are a little fixed up, I'd invite you to rerun the models, and see if the cross_val_score goes up as a result.  It is also a great chance to take some of the theory discussion from the workshop and play around with the parameters of your models, and see if you can increase their scores that way.  There's a blank code box below where you can play around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'features_df' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-4fa70eb56ab4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeatures_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_matrix\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlabels_df\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"status_group\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtolist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mclf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensemble\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRandomForestClassifier\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mscore\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcross_validation\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcross_val_score\u001b[0m\u001b[0;34m(\u001b[0m \u001b[0mclf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'features_df' is not defined"
     ]
    }
   ],
   "source": [
    "### your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "##End-to-end workflows using Pipeline and GridSearchCV\n",
    "\n",
    "So far we have made a nice workflow using a few ideas assembled in a script-like workflow.  A few spots remain where we can tighten things up though:\n",
    "\n",
    "* the best model, the random forest, has a lot of parameters that we'd have to work through if we really wanted to tune it\n",
    "* after dummying, we have _lots_ of features, probably only a subset of which are really offering any discriminatory power (this is a version of the bias-variance tradeoff)\n",
    "* maybe there's a way to make the code more streamlined (hint: there is)\n",
    "\n",
    "We will solve all these with two related and lovely tools in sklearn: Pipeline and GridSearchCV.\n",
    "\n",
    "Pipeline in sklearn is a tool for chaining together multiple pieces of a workflow into a single coherent analysis.  In our example, we will chain together a tool for feature selection, to will address the second point, which then feeds our optimized feature set into the random forest model, all in a few lines of code (which addresses the third point).\n",
    "\n",
    "To get to the first point, about finding the best parameters--that's where the magic of GridSearchCV comes in.  But first we need to get the feature selector and pipeline up and running, so let's do that now.\n",
    "\n",
    "In ``sklearn.feature_selection`` there is a useful tool, ``SelectKBest`` (link)[http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html] that you should use.  By default, this will select the 10 best features; that seems like it might be too few features to do well on this problem, so change the number of features to 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'X' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-8bca07a79e38>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mselect\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_selection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSelectKBest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mselected_X\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mselect\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit_transform\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m \u001b[0mselected_X\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m \u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'X' is not defined"
     ]
    }
   ],
   "source": [
    "import sklearn.feature_selection\n",
    "\n",
    "### your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipeline\n",
    "\n",
    "After selecting the 100 best features, the natural next step would be to run our random forest again to see if it does a little better with fewer features.  So we would have ``SelectKBest`` doing selection, with the output of that process going straight into a classifier.  A [Pipeline](http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) packages the transformation step of ``SelectKBest`` with the estimation step of ``RandomForestClassifier`` into a coherent workflow.\n",
    "\n",
    "Why might you want to use ``Pipeline`` instead of keeping the steps separate?\n",
    "\n",
    "  * makes code more readable\n",
    "  * don't have to worry about keeping track data during intermediate steps, for example between transforming and estimating\n",
    "  * makes it trivial to move ordering of the pipeline pieces, or to swap pieces in and out\n",
    "  * *Allows you to do GridSearchCV on your workflow*\n",
    "\n",
    "This last point is, in my opinion, the most important.  We will get to it very soon, but first let's get a pipeline up and running that does ``SelectKBest`` followed by ``RandomForestClassifier``.\n",
    "\n",
    "In the code box below, I've also set up a slightly better training/testing structure, where I am explicitly splitting the data into training and testing sets which we'll use below.  The  training/testing split before was handled automatically in ``cross_val_score,`` but we'll be using a different evaluation metric from here forward, the classification report, which requires us to handle the train/test split ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-6-eeda30413ef3>, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-6-eeda30413ef3>\"\u001b[0;36m, line \u001b[0;32m3\u001b[0m\n\u001b[0;31m    select = ### fill this in--SelectKBest, k=100\u001b[0m\n\u001b[0m                                                 ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "import sklearn.pipeline\n",
    "\n",
    "select = ### fill this in--SelectKBest, k=100\n",
    "clf = ### fill this in--RandomForestClassifier\n",
    "\n",
    "pipeline = ### fill this in, using sklearn docs as a guide\n",
    "\n",
    "X_train, X_test, y_train, y_test = sklearn.cross_validation.train_test_split(X, y, test_size=0.33, random_state=42)\n",
    "\n",
    "### fit your pipeline on X_train and y_train\n",
    "### call pipeline.predict() on your X_test data to make a set of test predictions\n",
    "### test your predictions using sklearn.classification_report()\n",
    "### and print the report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading the classification report\n",
    "\n",
    "A brief aside--we've switched from ``cross_val_score`` to ``classification_report`` for evaluation, mostly to show you two different ways to evaluating a model.  The classification report has the advantage of giving you a lot more information, and if (for example) one class is more important to get right than the others (say you're trying to zero in on non-functional wells, so finding those correctly is more important than getting the functional wells right).\n",
    "\n",
    "For more information, the [sklearn docs](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) on ``classification_report`` are, like all the sklearn docs, incredibly helpful.  For interpreting the various metrics, [this page](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html) may also help.\n",
    "\n",
    "### GridSearchCV\n",
    "\n",
    "We're in the home stretch now.  When we decided to select the 100 best features, setting that number to 100 was kind of a hand-wavey decision.  Similarly, the ``RandomForestClassifier`` that we're using right now has all its parameters set to their default values, which might not be optimal.\n",
    "\n",
    "So, a straightforward thing to do now is to try different values of ``k`` and any ``RandomForestClassifier`` parameters we want to tune (for the sake of concreteness, let's play with ``n_estimators`` and ``min_samples_split``).  Trying lots of values for each of these free parameters is tedious, and there can sometimes be interactions between the choices you make in one step and the optimal value for a downstream step.  In other words, to avoid local optima, you should try all the **combinations** of parameters, and not just vary them independently.  So if you want to try 5 different values each for ``k``, ``n_estimators`` and ``min_samples_split``, that means 5 x 5 x 5 = 125 different combinations to try.  Not something you want to do by hand.\n",
    "\n",
    "``GridSearchCV`` allows you to construct a grid of all the combinations of parameters, tries each combination, and then reports back the best combination/model.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}