h2o-world-2017/automl/Python/automl_regression_powerplant_output.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# H2O AutoML Regression Demo\n",
    "\n",
    "This is a [Jupyter](https://jupyter.org/) Notebook. When you execute code within the notebook, the results appear beneath the code. To execute a code chunk, place your cursor on the cell and press *Shift+Enter*. \n",
    "\n",
    "### Start H2O\n",
    "\n",
    "Import the **h2o** Python module and `H2OAutoML` class and initialize a local H2O cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321..... not found.\n",
      "Attempting to start a local H2O server...\n",
      "  Java Version: java version \"1.8.0_45\"; Java(TM) SE Runtime Environment (build 1.8.0_45-b14); Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)\n",
      "  Starting server from /usr/local/lib/python3.5/site-packages/h2o/backend/bin/h2o.jar\n",
      "  Ice root: /var/folders/2j/jg4sl53d5q53tc2_nzm9fz5h0000gn/T/tmpyxils1hk\n",
      "  JVM stdout: /var/folders/2j/jg4sl53d5q53tc2_nzm9fz5h0000gn/T/tmpyxils1hk/h2o_me_started_from_python.out\n",
      "  JVM stderr: /var/folders/2j/jg4sl53d5q53tc2_nzm9fz5h0000gn/T/tmpyxils1hk/h2o_me_started_from_python.err\n",
      "  Server is running at http://127.0.0.1:54321\n",
      "Connecting to H2O server at http://127.0.0.1:54321... successful.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O cluster uptime:</td>\n",
       "<td>01 secs</td></tr>\n",
       "<tr><td>H2O cluster version:</td>\n",
       "<td>3.16.0.2</td></tr>\n",
       "<tr><td>H2O cluster version age:</td>\n",
       "<td>2 days </td></tr>\n",
       "<tr><td>H2O cluster name:</td>\n",
       "<td>H2O_from_python_me_bwxa8k</td></tr>\n",
       "<tr><td>H2O cluster total nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O cluster free memory:</td>\n",
       "<td>3.556 Gb</td></tr>\n",
       "<tr><td>H2O cluster total cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster allowed cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster status:</td>\n",
       "<td>accepting new members, healthy</td></tr>\n",
       "<tr><td>H2O connection url:</td>\n",
       "<td>http://127.0.0.1:54321</td></tr>\n",
       "<tr><td>H2O connection proxy:</td>\n",
       "<td>None</td></tr>\n",
       "<tr><td>H2O internal security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>H2O API Extensions:</td>\n",
       "<td>XGBoost, Algos, AutoML, Core V3, Core V4</td></tr>\n",
       "<tr><td>Python version:</td>\n",
       "<td>3.5.2 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  ----------------------------------------\n",
       "H2O cluster uptime:         01 secs\n",
       "H2O cluster version:        3.16.0.2\n",
       "H2O cluster version age:    2 days\n",
       "H2O cluster name:           H2O_from_python_me_bwxa8k\n",
       "H2O cluster total nodes:    1\n",
       "H2O cluster free memory:    3.556 Gb\n",
       "H2O cluster total cores:    8\n",
       "H2O cluster allowed cores:  8\n",
       "H2O cluster status:         accepting new members, healthy\n",
       "H2O connection url:         http://127.0.0.1:54321\n",
       "H2O connection proxy:\n",
       "H2O internal security:      False\n",
       "H2O API Extensions:         XGBoost, Algos, AutoML, Core V3, Core V4\n",
       "Python version:             3.5.2 final\n",
       "--------------------------  ----------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import h2o\n",
    "from h2o.automl import H2OAutoML\n",
    "h2o.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data\n",
    "\n",
    "For the AutoML regression demo, we use the [Combined Cycle Power Plant](http://archive.ics.uci.edu/ml/datasets/Combined+Cycle+Power+Plant) dataset.  The goal here is to predict the energy output (in megawatts), given the temperature, ambient pressure, relative humidity and exhaust vacuum values.  In this demo, you will use H2O's AutoML to outperform the [state of the art results](https://www.sciencedirect.com/science/article/pii/S0142061514000908) on this task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "# Use local data file or download from GitHub\n",
    "import os\n",
    "docker_data_path = \"/home/h2o/data/automl/powerplant_output.csv\"\n",
    "if os.path.isfile(docker_data_path):\n",
    "  data_path = docker_data_path\n",
    "else:\n",
    "  data_path = \"https://github.com/h2oai/h2o-tutorials/raw/master/h2o-world-2017/automl/data/powerplant_output.csv\"\n",
    "\n",
    "\n",
    "# Load data into H2O\n",
    "df = h2o.import_file(data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows:9568\n",
      "Cols:5\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>       </th><th>TemperatureCelcius  </th><th>ExhaustVacuumHg   </th><th>AmbientPressureMillibar  </th><th>RelativeHumidity  </th><th>HourlyEnergyOutputMW  </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>type   </td><td>real                </td><td>real              </td><td>real                     </td><td>real              </td><td>real                  </td></tr>\n",
       "<tr><td>mins   </td><td>1.81                </td><td>25.36             </td><td>992.89                   </td><td>25.56             </td><td>420.26                </td></tr>\n",
       "<tr><td>mean   </td><td>19.651231187291014  </td><td>54.305803720735604</td><td>1013.2590781772534       </td><td>73.30897784280899 </td><td>454.36500940635517    </td></tr>\n",
       "<tr><td>maxs   </td><td>37.11               </td><td>81.56             </td><td>1033.3                   </td><td>100.16            </td><td>495.76                </td></tr>\n",
       "<tr><td>sigma  </td><td>7.4524732296110825  </td><td>12.707892998326784</td><td>5.938783705811579        </td><td>14.600268756728964</td><td>17.066994999803402    </td></tr>\n",
       "<tr><td>zeros  </td><td>0                   </td><td>0                 </td><td>0                        </td><td>0                 </td><td>0                     </td></tr>\n",
       "<tr><td>missing</td><td>0                   </td><td>0                 </td><td>0                        </td><td>0                 </td><td>0                     </td></tr>\n",
       "<tr><td>0      </td><td>14.96               </td><td>41.76             </td><td>1024.07                  </td><td>73.17             </td><td>463.26                </td></tr>\n",
       "<tr><td>1      </td><td>25.18               </td><td>62.96             </td><td>1020.04                  </td><td>59.08             </td><td>444.37                </td></tr>\n",
       "<tr><td>2      </td><td>5.11                </td><td>39.4              </td><td>1012.16                  </td><td>92.14             </td><td>488.56                </td></tr>\n",
       "<tr><td>3      </td><td>20.86               </td><td>57.32             </td><td>1010.24                  </td><td>76.64             </td><td>446.48                </td></tr>\n",
       "<tr><td>4      </td><td>10.82               </td><td>37.5              </td><td>1009.23                  </td><td>96.62             </td><td>473.9                 </td></tr>\n",
       "<tr><td>5      </td><td>26.27               </td><td>59.44             </td><td>1012.23                  </td><td>58.77             </td><td>443.67                </td></tr>\n",
       "<tr><td>6      </td><td>15.89               </td><td>43.96             </td><td>1014.02                  </td><td>75.24             </td><td>467.35                </td></tr>\n",
       "<tr><td>7      </td><td>9.48                </td><td>44.71             </td><td>1019.12                  </td><td>66.43             </td><td>478.42                </td></tr>\n",
       "<tr><td>8      </td><td>14.64               </td><td>45.0              </td><td>1021.78                  </td><td>41.25             </td><td>475.98                </td></tr>\n",
       "<tr><td>9      </td><td>11.74               </td><td>43.56             </td><td>1015.14                  </td><td>70.72             </td><td>477.5                 </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's identify the response column and save the column name as `y`.  In this dataset, we will use all columns except the response as predictors, so we can skip setting the `x` argument explicitly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = \"HourlyEnergyOutputMW\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, let's split the data into two frames, a `train` (80%) and a `test` frame (20%).  The `test` frame will be used to score the leaderboard and to demonstrate how to generate predictions using an AutoML leader model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = df.split_frame(ratios = [0.8], seed = 1)\n",
    "train = splits[0]\n",
    "test = splits[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run AutoML \n",
    "\n",
    "Run AutoML, stopping after 60 seconds.  The `max_runtime_secs` argument provides a way to limit the AutoML run by time.  When using a time-limited stopping criterion, the number of models train will vary between runs.  If different hardware is used or even if the same machine is used but the available compute resources on that machine are not the same between runs, then AutoML may be able to train more models on one run vs another. \n",
    "\n",
    "The `test` frame is passed explicitly to the `leaderboard_frame` argument here, which means that instead of using cross-validated metrics, we use test set metrics for generating the leaderboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "aml = H2OAutoML(max_runtime_secs = 60, seed = 1, project_name = \"powerplant_lb_frame\")\n",
    "aml.train(y = y, training_frame = train, leaderboard_frame = test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: If you see the following error, it means that you need to install the pandas module.*\n",
    "```\n",
    "H2OTypeError: Argument `python_obj` should be a None | list | tuple | dict | numpy.ndarray | pandas.DataFrame | scipy.sparse.issparse, got H2OTwoDimTable \n",
    "``` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For demonstration purposes, we will also execute a second AutoML run, this time providing the original, full dataset, `df` (without passing a `leaderboard_frame`).  This is a more efficient use of our data since we can use 100% of the data for training, rather than 80% like we did above.  This time our leaderboard will use cross-validated metrics.\n",
    "\n",
    "*Note: Using an explicit `leaderboard_frame` for scoring may be useful in some cases, which is why the option is available.*  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "aml2 = H2OAutoML(max_runtime_secs = 60, seed = 1, project_name = \"powerplant_full_data\")\n",
    "aml2.train(y = y, training_frame = df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: We specify a `project_name` here for clarity.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leaderboard\n",
    "\n",
    "Next, we will view the AutoML Leaderboard.  Since we specified a `leaderboard_frame` in the `H2OAutoML.train()` method for scoring and ranking the models, the AutoML leaderboard uses the performance on this data to rank the models.\n",
    "\n",
    "After viewing the `\"powerplant_lb_frame\"` AutoML project leaderboard, we compare that to the leaderboard for the `\"powerplant_full_data\"` project.  We can see that the results are better when the full dataset is used for training.  \n",
    "\n",
    "A default performance metric for each machine learning task (binary classification, multiclass classification, regression) is specified internally and the leaderboard will be sorted by that metric.  In the case of regression, the default ranking metric is mean residual deviance.  In the future, the user will be able to specify any of the H2O metrics so that different metrics can be used to generate rankings on the leaderboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>model_id                                             </th><th style=\"text-align: right;\">  mean_residual_deviance</th><th style=\"text-align: right;\">   rmse</th><th style=\"text-align: right;\">    mae</th><th style=\"text-align: right;\">   rmsle</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>StackedEnsemble_AllModels_0_AutoML_20171203_023916   </td><td style=\"text-align: right;\">                 11.3313</td><td style=\"text-align: right;\">3.3662 </td><td style=\"text-align: right;\">2.36832</td><td style=\"text-align: right;\">0.007408</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_023916_model_3            </td><td style=\"text-align: right;\">                 11.4702</td><td style=\"text-align: right;\">3.38677</td><td style=\"text-align: right;\">2.38687</td><td style=\"text-align: right;\">0.007449</td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_0_AutoML_20171203_023916</td><td style=\"text-align: right;\">                 11.5596</td><td style=\"text-align: right;\">3.39995</td><td style=\"text-align: right;\">2.38938</td><td style=\"text-align: right;\">0.007483</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_023916_model_2            </td><td style=\"text-align: right;\">                 11.7115</td><td style=\"text-align: right;\">3.4222 </td><td style=\"text-align: right;\">2.43157</td><td style=\"text-align: right;\">0.007532</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_023916_model_1            </td><td style=\"text-align: right;\">                 12.3146</td><td style=\"text-align: right;\">3.50922</td><td style=\"text-align: right;\">2.5227 </td><td style=\"text-align: right;\">0.007719</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_023916_model_0            </td><td style=\"text-align: right;\">                 12.5269</td><td style=\"text-align: right;\">3.53934</td><td style=\"text-align: right;\">2.55044</td><td style=\"text-align: right;\">0.007775</td></tr>\n",
       "<tr><td>XRT_0_AutoML_20171203_023916                         </td><td style=\"text-align: right;\">                 13.3574</td><td style=\"text-align: right;\">3.65478</td><td style=\"text-align: right;\">2.58698</td><td style=\"text-align: right;\">0.00805 </td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_023916_model_10           </td><td style=\"text-align: right;\">                 13.506 </td><td style=\"text-align: right;\">3.67506</td><td style=\"text-align: right;\">2.65552</td><td style=\"text-align: right;\">0.008093</td></tr>\n",
       "<tr><td>DRF_0_AutoML_20171203_023916                         </td><td style=\"text-align: right;\">                 13.7595</td><td style=\"text-align: right;\">3.70938</td><td style=\"text-align: right;\">2.62833</td><td style=\"text-align: right;\">0.008173</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_023916_model_4            </td><td style=\"text-align: right;\">                 13.9662</td><td style=\"text-align: right;\">3.73714</td><td style=\"text-align: right;\">2.71332</td><td style=\"text-align: right;\">0.008218</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aml.leaderboard.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will view a snapshot of the top models.  Here we should see the two Stacked Ensembles at or near the top of the leaderboard.  Stacked Ensembles can almost always outperform a single model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>model_id                                             </th><th style=\"text-align: right;\">  mean_residual_deviance</th><th style=\"text-align: right;\">   rmse</th><th style=\"text-align: right;\">    mae</th><th style=\"text-align: right;\">   rmsle</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>StackedEnsemble_AllModels_0_AutoML_20171203_024210   </td><td style=\"text-align: right;\">                 10.4436</td><td style=\"text-align: right;\">3.23166</td><td style=\"text-align: right;\">2.33147</td><td style=\"text-align: right;\">0.007102</td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_0_AutoML_20171203_024210</td><td style=\"text-align: right;\">                 10.6979</td><td style=\"text-align: right;\">3.27076</td><td style=\"text-align: right;\">2.34774</td><td style=\"text-align: right;\">0.007189</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_024210_model_3            </td><td style=\"text-align: right;\">                 10.7952</td><td style=\"text-align: right;\">3.2856 </td><td style=\"text-align: right;\">2.36773</td><td style=\"text-align: right;\">0.007216</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_024210_model_0            </td><td style=\"text-align: right;\">                 11.0041</td><td style=\"text-align: right;\">3.31724</td><td style=\"text-align: right;\">2.42454</td><td style=\"text-align: right;\">0.007283</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_024210_model_2            </td><td style=\"text-align: right;\">                 11.067 </td><td style=\"text-align: right;\">3.32671</td><td style=\"text-align: right;\">2.42077</td><td style=\"text-align: right;\">0.007309</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_024210_model_1            </td><td style=\"text-align: right;\">                 11.4219</td><td style=\"text-align: right;\">3.37964</td><td style=\"text-align: right;\">2.46841</td><td style=\"text-align: right;\">0.00742 </td></tr>\n",
       "<tr><td>XRT_0_AutoML_20171203_024210                         </td><td style=\"text-align: right;\">                 11.9444</td><td style=\"text-align: right;\">3.45607</td><td style=\"text-align: right;\">2.51927</td><td style=\"text-align: right;\">0.007597</td></tr>\n",
       "<tr><td>DRF_0_AutoML_20171203_024210                         </td><td style=\"text-align: right;\">                 11.9698</td><td style=\"text-align: right;\">3.45974</td><td style=\"text-align: right;\">2.51205</td><td style=\"text-align: right;\">0.00761 </td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_024210_model_4            </td><td style=\"text-align: right;\">                 12.8725</td><td style=\"text-align: right;\">3.58783</td><td style=\"text-align: right;\">2.66669</td><td style=\"text-align: right;\">0.007878</td></tr>\n",
       "<tr><td>GBM_grid_0_AutoML_20171203_024210_model_8            </td><td style=\"text-align: right;\">                 14.75  </td><td style=\"text-align: right;\">3.84057</td><td style=\"text-align: right;\">2.85527</td><td style=\"text-align: right;\">0.008425</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aml2.leaderboard.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset comes from the [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/datasets/Combined+Cycle+Power+Plant) of machine learning datasets.  The data was used in a [publication](https://www.sciencedirect.com/science/article/pii/S0142061514000908) in the *International Journal of Electrical Power & Energy Systems* in 2014.  In the paper, the authors achieved a mean absolute error (MAE) of 2.818 and a Root Mean-Squared Error (RMSE) of 3.787 on their best model.  So, with H2O's AutoML, we've already beaten the state-of-the-art in just 60 seconds of compute time!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict Using Leader Model\n",
    "\n",
    "If you need to generate predictions on a test set, you can make predictions on the `\"H2OAutoML\"` object directly, or on the leader model object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "stackedensemble prediction progress: |████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  predict</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">  487.682</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  474.604</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  465.741</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  452.552</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  447.689</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  470.933</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  443.294</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  463.253</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  443.096</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  432.704</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = aml.predict(test)\n",
    "pred.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If needed, the standard `model_performance()` method can be applied to the AutoML leader model and a test set to generate an H2O model performance object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ModelMetricsRegressionGLM: stackedensemble\n",
      "** Reported on test data. **\n",
      "\n",
      "MSE: 11.331286701529518\n",
      "RMSE: 3.3661976622785414\n",
      "MAE: 2.368315617370515\n",
      "RMSLE: 0.0074080999439092325\n",
      "R^2: 0.9612334999993339\n",
      "Mean Residual Deviance: 11.331286701529518\n",
      "Null degrees of freedom: 1890\n",
      "Residual degrees of freedom: 1881\n",
      "Null deviance: 552751.8588072016\n",
      "Residual deviance: 21427.463152592318\n",
      "AIC: 9978.955929384452\n"
     ]
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perf = aml.leader.model_performance(test)\n",
    "perf"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}