Feature Engeenering (Part 3) Cold Start Energy Forecasting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recap\n",
    "\n",
    "\n",
    "Before we start, I will copy some idea that we got in previous stages :\n",
    "#### Part 1\n",
    "\n",
    "- Here we have 1383 building, 758 in training data and the rest in test data.\n",
    "- We have 276923 training data, and 81827 test data\n",
    "- We have many missing value in temperature\n",
    "- etc\n",
    "\n",
    "- In the data train each building is provided with the same number of sample trainings, namely samples in units of hours in 28 days. while in the data test, the number of data samples for each building varies, there are 4 days, 12 days, etc.\n",
    "\n",
    "For more detail you can check direcly to the notebook https://github.com/fauzantaufik/Cold-Start-Energy-Forecasting/blob/master/EXPLOLATORY%20DATA%20TO%20DEFINE%20PROBLEM%20DEEPLY.ipynb\n",
    "\n",
    "#### Part 2\n",
    "\n",
    "- seen that consumption from year to year is increasing, so maybe years will be a good feature to be the input in our model\n",
    "- The month will also be a good feature. there is a seasonal pattern from month to month.\n",
    "- There is some noise in data\n",
    "- Consumption nearly uniform from hour to hour, but some date have pattern in consomption by hour to hour\n",
    "- There is some correlation bettwen temperature/hour and consumption/hour\n",
    "\n",
    "\"\"\"One of the challenges is in the way we fill in the missing values. because many buildings have a temperature value that is lost up to 100%. meaning that for one year the temperature inside the building is unknown!!.\n",
    "\n",
    "we cannot fill this missing value with linear interpolation, even if it has to be non-linear we will have difficulty determining the degree of non-linearity.\n",
    "\n",
    "Because all series IDs have the same pattern regarding the temperature value. then it's have to be good if we try to build a model that will predict the temperature value. this is a gonna be like other project though, but it's worth it since we have missinig value almost 50% in temperature\"\"\"\"\n",
    "\n",
    "=======================================\n",
    "\n",
    "\"\"\"For electricity consumption in each building, it seems that each building has a different level of electricity consumption. may depend on the type of building. there are buildings that always need electricity every time. there are also buildings that only require electricity when at certain times such as working hours and then a little consumption at night.\"\"\"\n",
    "\n",
    "Talk about electricity consumption : \n",
    "- There is a changes in consumption pattern over year. but it's not abvious the rule of pattern. and I dont know is it significant or not. but it's in my hipotesis.\n",
    "- There is no really abvious pattern of electricity consumption over month and day of month.\n",
    "- There is a pattern in the name of day, the consumpton is low in holiday (saturday and sunday) than in work days.\n",
    "- There is a pattern in series of hours, which is the high consumption at work time.\n",
    "\n",
    "We need to talk about Temperature due to we wanna build predictive model for temperature because of the need of fill in the missing value :\n",
    "- The temperature seem to be increase over year.\n",
    "- The temperature seem to be have seasonal pattern in series of month.\n",
    "- The temperature seem to be constant in the day of month and in the day of week.\n",
    "- Of course, the temperature is higher in the daytime hours\n",
    "\n",
    "- the variance in consumption vs. lag increases with further lag\n",
    "- temperature vs temperature lag-k seem have to be same \n",
    "\n",
    "One more problem is we have no sample for electricity consumption and temperature in 2012.\n",
    "\n",
    "For more detail you check the notebook : https://github.com/fauzantaufik/Cold-Start-Energy-Forecasting/blob/master/Visualization%20Cold%20Start%20Energy%20Forecasting.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature Engeenering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Substep of Feature Engeenering :\n",
    "\n",
    "1. Feature Creation\n",
    "\n",
    "    1.1 Feature Extractin of Time (Years, Months, Day of Week, Hours)\n",
    "    \n",
    "    1.2 Feature Over Time\n",
    "    \n",
    "    1.3 Adding Lag Feature\n",
    "    \n",
    "    1.4 Adding SMA (smooth moving avarage), pct_change\n",
    "    \n",
    "    1.5 Adding National Holiday Feature\n",
    "    \n",
    "    1.6 Adding meta data information\n",
    "    \n",
    "    \n",
    "2. Data Cleaning\n",
    "\n",
    "    2.1 Build Predictive Model for temperature\n",
    "    \n",
    "    2.2 Filling missing value of temperature by Predictive Model that have been build\n",
    "    \n",
    "    2.3 Cleaning Outlier\n",
    "    \n",
    "    \n",
    "3. Feature Selection\n",
    "    \n",
    "    3.1 Inspecting by Heatmap\n",
    "\n",
    "    3.2 Find coeffecient with linear model first\n",
    "    \n",
    "    3.3 RFECV\n",
    "    \n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta_data = pd.read_csv('meta.csv')\n",
    "train_data = pd.read_csv('consumption_train.csv')\n",
    "test_data = pd.read_csv('cold_start_test.csv')\n",
    "submission = pd.read_csv('submission_format.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pred_id</th>\n",
       "      <th>series_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>temperature</th>\n",
       "      <th>consumption</th>\n",
       "      <th>prediction_window</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>31</td>\n",
       "      <td>101878</td>\n",
       "      <td>2014-04-08 00:00:00</td>\n",
       "      <td>18.85</td>\n",
       "      <td>0.0</td>\n",
       "      <td>hourly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>32</td>\n",
       "      <td>101878</td>\n",
       "      <td>2014-04-08 01:00:00</td>\n",
       "      <td>19.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>hourly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>33</td>\n",
       "      <td>101878</td>\n",
       "      <td>2014-04-08 02:00:00</td>\n",
       "      <td>19.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>hourly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>34</td>\n",
       "      <td>101878</td>\n",
       "      <td>2014-04-08 03:00:00</td>\n",
       "      <td>18.15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>hourly</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>35</td>\n",
       "      <td>101878</td>\n",
       "      <td>2014-04-08 04:00:00</td>\n",
       "      <td>17.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>hourly</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    pred_id  series_id            timestamp  temperature  consumption  \\\n",
       "31       31     101878  2014-04-08 00:00:00        18.85          0.0   \n",
       "32       32     101878  2014-04-08 01:00:00        19.00          0.0   \n",
       "33       33     101878  2014-04-08 02:00:00        19.00          0.0   \n",
       "34       34     101878  2014-04-08 03:00:00        18.15          0.0   \n",
       "35       35     101878  2014-04-08 04:00:00        17.00          0.0   \n",
       "\n",
       "   prediction_window  \n",
       "31            hourly  \n",
       "32            hourly  \n",
       "33            hourly  \n",
       "34            hourly  \n",
       "35            hourly  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission[submission.prediction_window=='hourly'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>series_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>consumption</th>\n",
       "      <th>temperature</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4261</th>\n",
       "      <td>4261</td>\n",
       "      <td>101464</td>\n",
       "      <td>2013-11-05 13:00:00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4262</th>\n",
       "      <td>4262</td>\n",
       "      <td>101464</td>\n",
       "      <td>2013-11-05 14:00:00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>23.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78730</th>\n",
       "      <td>78730</td>\n",
       "      <td>100721</td>\n",
       "      <td>2012-06-05 10:00:00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78731</th>\n",
       "      <td>78731</td>\n",
       "      <td>100721</td>\n",
       "      <td>2012-06-05 11:00:00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0  series_id            timestamp  consumption  temperature\n",
       "4261         4261     101464  2013-11-05 13:00:00          0.0        22.15\n",
       "4262         4262     101464  2013-11-05 14:00:00          0.0        23.00\n",
       "78730       78730     100721  2012-06-05 10:00:00          0.0          NaN\n",
       "78731       78731     100721  2012-06-05 11:00:00          0.0          NaN"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data[test_data.consumption==0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>series_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>consumption</th>\n",
       "      <th>temperature</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>78624</th>\n",
       "      <td>78624</td>\n",
       "      <td>100721</td>\n",
       "      <td>2012-06-01 00:00:00</td>\n",
       "      <td>80921.671453</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78625</th>\n",
       "      <td>78625</td>\n",
       "      <td>100721</td>\n",
       "      <td>2012-06-01 01:00:00</td>\n",
       "      <td>63264.995507</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78626</th>\n",
       "      <td>78626</td>\n",
       "      <td>100721</td>\n",
       "      <td>2012-06-01 02:00:00</td>\n",
       "      <td>61655.566815</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78627</th>\n",
       "      <td>78627</td>\n",
       "      <td>100721</td>\n",
       "      <td>2012-06-01 03:00:00</td>\n",
       "      <td>69971.508388</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78628</th>\n",
       "      <td>78628</td>\n",
       "      <td>100721</td>\n",
       "      <td>2012-06-01 04:00:00</td>\n",
       "      <td>100214.655903</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0  series_id            timestamp    consumption  temperature\n",
       "78624       78624     100721  2012-06-01 00:00:00   80921.671453          NaN\n",
       "78625       78625     100721  2012-06-01 01:00:00   63264.995507          NaN\n",
       "78626       78626     100721  2012-06-01 02:00:00   61655.566815          NaN\n",
       "78627       78627     100721  2012-06-01 03:00:00   69971.508388          NaN\n",
       "78628       78628     100721  2012-06-01 04:00:00  100214.655903          NaN"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data[test_data.series_id==100721].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['daily', 'weekly', 'hourly'], dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission.prediction_window.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(submission.series_id) == set(test_data.series_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"Change the types of timestamp columns and set the index\"\"\"\n",
    "\n",
    "train_data.timestamp = pd.to_datetime(train_data.timestamp)\n",
    "test_data.timestamp = pd.to_datetime(test_data.timestamp)\n",
    "\n",
    "train_data.index = train_data.timestamp\n",
    "test_data.index = test_data.timestamp\n",
    "\n",
    "train_data.drop('timestamp', axis=1, inplace=True)\n",
    "train_data.drop('Unnamed: 0', axis=1, inplace=True)\n",
    "\n",
    "test_data.drop('timestamp', axis=1, inplace=True)\n",
    "test_data.drop('Unnamed: 0', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>series_id</th>\n",
       "      <th>surface</th>\n",
       "      <th>base_temperature</th>\n",
       "      <th>monday_is_day_off</th>\n",
       "      <th>tuesday_is_day_off</th>\n",
       "      <th>wednesday_is_day_off</th>\n",
       "      <th>thursday_is_day_off</th>\n",
       "      <th>friday_is_day_off</th>\n",
       "      <th>saturday_is_day_off</th>\n",
       "      <th>sunday_is_day_off</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100003</td>\n",
       "      <td>x-large</td>\n",
       "      <td>low</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100004</td>\n",
       "      <td>x-large</td>\n",
       "      <td>low</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100006</td>\n",
       "      <td>x-small</td>\n",
       "      <td>low</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100008</td>\n",
       "      <td>x-small</td>\n",
       "      <td>low</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100010</td>\n",
       "      <td>x-small</td>\n",
       "      <td>low</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   series_id  surface base_temperature  monday_is_day_off  tuesday_is_day_off  \\\n",
       "0     100003  x-large              low              False               False   \n",
       "1     100004  x-large              low              False               False   \n",
       "2     100006  x-small              low              False               False   \n",
       "3     100008  x-small              low              False               False   \n",
       "4     100010  x-small              low              False               False   \n",
       "\n",
       "   wednesday_is_day_off  thursday_is_day_off  friday_is_day_off  \\\n",
       "0                 False                False              False   \n",
       "1                 False                False              False   \n",
       "2                 False                False              False   \n",
       "3                 False                False              False   \n",
       "4                 False                False              False   \n",
       "\n",
       "   saturday_is_day_off  sunday_is_day_off  \n",
       "0                 True               True  \n",
       "1                 True               True  \n",
       "2                 True               True  \n",
       "3                 True               True  \n",
       "4                 True               True  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meta_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'friday_is_day_off count': 17,\n",
       " 'monday_is_day_off count': 4,\n",
       " 'saturday_is_day_off count': 1258,\n",
       " 'sunday_is_day_off count': 1274,\n",
       " 'thursday_is_day_off count': 7,\n",
       " 'tuesday_is_day_off count': 4,\n",
       " 'wednesday_is_day_off count': 7}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "day_off = meta_data.columns[3:]\n",
    "\n",
    "count_day_off = {'{} count'.format(day) : meta_data[day].sum() for day in day_off}\n",
    "\n",
    "count_day_off"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "def evaluate_feature_rf(x, y, metrics) :\n",
    "    model_evaluate = RandomForestRegressor(criterion='mae')\n",
    "    model_evaluate.fit(x, y)\n",
    "    score = metrics(y, model_evaluate.predict(x))\n",
    "    \n",
    "    plt.barh(model_evaluate.feature_importances_, x.columns)\n",
    "    plt.title('Feature Importances')\n",
    "    plt.show()\n",
    "    \n",
    "    print('Score : {}'.format(score))\n",
    "    \n",
    "def evaluate_feature_dtr(x, y, metrics) :\n",
    "    model_evaluate = DecisionTreeRegressor(criterion='mae')\n",
    "    model_evaluate.fit(x, y)\n",
    "    score = metrics(y, model_evaluate.predict(x))\n",
    "    \n",
    "    plt.barh(model_evaluate.feature_importances_, x.columns)\n",
    "    plt.title('Feature Importances')\n",
    "    plt.show()\n",
    "    \n",
    "    print('Score : {}'.format(score))\n",
    "    \n",
    "def feature_summary(x, y, metrics) :\n",
    "    linear_features = sm.add_constant(x)\n",
    "    model = sm.OLS(y, x)\n",
    "    results = model.fit()  # fit the model\n",
    "    \n",
    "    print(results.summary())\n",
    "    print(results.pvalues)\n",
    "    \n",
    "    prediction = results.predict(x)\n",
    "    score = metrics(y.values, prediction)\n",
    "    print(\"Mean Absolute Error {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Feature Extraction of Time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data['year'] = train_data.index.year\n",
    "train_data['month'] = train_data.index.month_name()\n",
    "train_data['dayofweek'] = train_data.index.day_name()\n",
    "train_data['hour'] = train_data.index.hour\n",
    "\n",
    "test_data['year'] = test_data.index.year\n",
    "test_data['month'] = test_data.index.month_name()\n",
    "test_data['dayofweek'] = test_data.index.day_name()\n",
    "test_data['hour'] = test_data.index.hour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>series_id</th>\n",
       "      <th>consumption</th>\n",
       "      <th>temperature</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>dayofweek</th>\n",
       "      <th>hour</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2014-12-24 00:00:00</th>\n",
       "      <td>103088</td>\n",
       "      <td>101842.233424</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014</td>\n",
       "      <td>December</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-24 01:00:00</th>\n",
       "      <td>103088</td>\n",
       "      <td>105878.048906</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014</td>\n",
       "      <td>December</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-24 02:00:00</th>\n",
       "      <td>103088</td>\n",
       "      <td>91619.105008</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014</td>\n",
       "      <td>December</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-24 03:00:00</th>\n",
       "      <td>103088</td>\n",
       "      <td>94473.706203</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014</td>\n",
       "      <td>December</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-12-24 04:00:00</th>\n",
       "      <td>103088</td>\n",
       "      <td>96976.755526</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014</td>\n",
       "      <td>December</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     series_id    consumption  temperature  year     month  \\\n",
       "timestamp                                                                    \n",
       "2014-12-24 00:00:00     103088  101842.233424          NaN  2014  December   \n",
       "2014-12-24 01:00:00     103088  105878.048906          NaN  2014  December   \n",
       "2014-12-24 02:00:00     103088   91619.105008          NaN  2014  December   \n",
       "2014-12-24 03:00:00     103088   94473.706203          NaN  2014  December   \n",
       "2014-12-24 04:00:00     103088   96976.755526          NaN  2014  December   \n",
       "\n",
       "                     dayofweek  hour  \n",
       "timestamp                             \n",
       "2014-12-24 00:00:00  Wednesday     0  \n",
       "2014-12-24 01:00:00  Wednesday     1  \n",
       "2014-12-24 02:00:00  Wednesday     2  \n",
       "2014-12-24 03:00:00  Wednesday     3  \n",
       "2014-12-24 04:00:00  Wednesday     4  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>series_id</th>\n",
       "      <th>consumption</th>\n",
       "      <th>temperature</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>dayofweek</th>\n",
       "      <th>hour</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2013-02-27 00:00:00</th>\n",
       "      <td>102781</td>\n",
       "      <td>15295.740389</td>\n",
       "      <td>17.00</td>\n",
       "      <td>2013</td>\n",
       "      <td>February</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-02-27 01:00:00</th>\n",
       "      <td>102781</td>\n",
       "      <td>15163.209562</td>\n",
       "      <td>18.25</td>\n",
       "      <td>2013</td>\n",
       "      <td>February</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-02-27 02:00:00</th>\n",
       "      <td>102781</td>\n",
       "      <td>15022.264079</td>\n",
       "      <td>18.00</td>\n",
       "      <td>2013</td>\n",
       "      <td>February</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-02-27 03:00:00</th>\n",
       "      <td>102781</td>\n",
       "      <td>15370.420458</td>\n",
       "      <td>17.00</td>\n",
       "      <td>2013</td>\n",
       "      <td>February</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-02-27 04:00:00</th>\n",
       "      <td>102781</td>\n",
       "      <td>15303.103213</td>\n",
       "      <td>16.90</td>\n",
       "      <td>2013</td>\n",
       "      <td>February</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     series_id   consumption  temperature  year     month  \\\n",
       "timestamp                                                                   \n",
       "2013-02-27 00:00:00     102781  15295.740389        17.00  2013  February   \n",
       "2013-02-27 01:00:00     102781  15163.209562        18.25  2013  February   \n",
       "2013-02-27 02:00:00     102781  15022.264079        18.00  2013  February   \n",
       "2013-02-27 03:00:00     102781  15370.420458        17.00  2013  February   \n",
       "2013-02-27 04:00:00     102781  15303.103213        16.90  2013  February   \n",
       "\n",
       "                     dayofweek  hour  \n",
       "timestamp                             \n",
       "2013-02-27 00:00:00  Wednesday     0  \n",
       "2013-02-27 01:00:00  Wednesday     1  \n",
       "2013-02-27 02:00:00  Wednesday     2  \n",
       "2013-02-27 03:00:00  Wednesday     3  \n",
       "2013-02-27 04:00:00  Wednesday     4  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "DatetimeIndex: 509376 entries, 2014-12-24 00:00:00 to 2017-12-18 23:00:00\n",
      "Data columns (total 7 columns):\n",
      "series_id      509376 non-null int64\n",
      "consumption    509376 non-null float64\n",
      "temperature    280687 non-null float64\n",
      "year           509376 non-null int64\n",
      "month          509376 non-null object\n",
      "dayofweek      509376 non-null object\n",
      "hour           509376 non-null int64\n",
      "dtypes: float64(2), int64(3), object(2)\n",
      "memory usage: 31.1+ MB\n"
     ]
    }
   ],
   "source": [
    "train_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Data Scientist\\Anaconda3\\lib\\site-packages\\sklearn\\ensemble\\weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.\n",
      "  from numpy.core.umath_tests import inner1d\n"
     ]
    }
   ],
   "source": [
    "## Evaluation Feature Performance\n",
    "\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "train_data.year = train_data.year.astype('object')\n",
    "train_data.hour = train_data.hour.astype('object')\n",
    "\n",
    "feature_test = ['year', 'month', 'dayofweek', 'hour']\n",
    "df_test=train_data[feature_test]\n",
    "y= train_data.consumption\n",
    "\n",
    "X = pd.get_dummies(df_test, drop_first=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            OLS Regression Results                            \n",
      "==============================================================================\n",
      "Dep. Variable:            consumption   R-squared:                       0.334\n",
      "Model:                            OLS   Adj. R-squared:                  0.333\n",
      "Method:                 Least Squares   F-statistic:                     5793.\n",
      "Date:                Sat, 27 Oct 2018   Prob (F-statistic):               0.00\n",
      "Time:                        15:02:19   Log-Likelihood:            -6.8241e+06\n",
      "No. Observations:              509376   AIC:                         1.365e+07\n",
      "Df Residuals:                  509332   BIC:                         1.365e+07\n",
      "Df Model:                          44                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "=======================================================================================\n",
      "                          coef    std err          t      P>|t|      [0.025      0.975]\n",
      "---------------------------------------------------------------------------------------\n",
      "year_2014            4.172e+04   1443.874     28.896      0.000    3.89e+04    4.46e+04\n",
      "year_2015            3.699e+04   1199.275     30.843      0.000    3.46e+04    3.93e+04\n",
      "year_2016            4.716e+04   1143.283     41.251      0.000    4.49e+04    4.94e+04\n",
      "year_2017            6.163e+04   1101.734     55.939      0.000    5.95e+04    6.38e+04\n",
      "month_August         1.387e+04   1079.658     12.843      0.000    1.18e+04     1.6e+04\n",
      "month_December      -5684.3090   1194.142     -4.760      0.000   -8024.791   -3343.827\n",
      "month_February       1.756e+04   1138.893     15.415      0.000    1.53e+04    1.98e+04\n",
      "month_January       -1.861e+04   1122.716    -16.576      0.000   -2.08e+04   -1.64e+04\n",
      "month_July          -3251.0889   1043.314     -3.116      0.002   -5295.952   -1206.225\n",
      "month_June           1.782e+04   1061.365     16.793      0.000    1.57e+04    1.99e+04\n",
      "month_March          -2.32e+04   1065.992    -21.767      0.000   -2.53e+04   -2.11e+04\n",
      "month_May           -8503.9225   1036.683     -8.203      0.000   -1.05e+04   -6472.056\n",
      "month_November       2144.6621   1021.185      2.100      0.036     143.172    4146.152\n",
      "month_October        2.278e+04   1050.804     21.679      0.000    2.07e+04    2.48e+04\n",
      "month_September      5899.9240   1027.818      5.740      0.000    3885.433    7914.415\n",
      "dayofweek_Monday     2238.3932    819.302      2.732      0.006     632.588    3844.199\n",
      "dayofweek_Saturday  -2.269e+04    818.299    -27.729      0.000   -2.43e+04   -2.11e+04\n",
      "dayofweek_Sunday    -3.401e+04    818.669    -41.546      0.000   -3.56e+04   -3.24e+04\n",
      "dayofweek_Thursday    1.13e+04    819.208     13.789      0.000    9690.666    1.29e+04\n",
      "dayofweek_Tuesday    1.121e+04    819.300     13.688      0.000    9609.012    1.28e+04\n",
      "dayofweek_Wednesday   1.28e+04    819.294     15.623      0.000    1.12e+04    1.44e+04\n",
      "hour_1               2.552e+04   1442.776     17.690      0.000    2.27e+04    2.84e+04\n",
      "hour_2               2.429e+04   1442.776     16.833      0.000    2.15e+04    2.71e+04\n",
      "hour_3               2.365e+04   1442.776     16.394      0.000    2.08e+04    2.65e+04\n",
      "hour_4               2.459e+04   1442.776     17.045      0.000    2.18e+04    2.74e+04\n",
      "hour_5                2.96e+04   1442.776     20.517      0.000    2.68e+04    3.24e+04\n",
      "hour_6               4.192e+04   1442.776     29.058      0.000    3.91e+04    4.48e+04\n",
      "hour_7                5.48e+04   1442.776     37.985      0.000     5.2e+04    5.76e+04\n",
      "hour_8               7.151e+04   1442.776     49.564      0.000    6.87e+04    7.43e+04\n",
      "hour_9               8.547e+04   1442.776     59.242      0.000    8.26e+04    8.83e+04\n",
      "hour_10              8.868e+04   1442.776     61.463      0.000    8.59e+04    9.15e+04\n",
      "hour_11              8.893e+04   1442.776     61.637      0.000    8.61e+04    9.18e+04\n",
      "hour_12              8.657e+04   1442.776     60.006      0.000    8.37e+04    8.94e+04\n",
      "hour_13              7.959e+04   1442.776     55.165      0.000    7.68e+04    8.24e+04\n",
      "hour_14              8.245e+04   1442.776     57.146      0.000    7.96e+04    8.53e+04\n",
      "hour_15              8.287e+04   1442.776     57.436      0.000       8e+04    8.57e+04\n",
      "hour_16              8.201e+04   1442.776     56.840      0.000    7.92e+04    8.48e+04\n",
      "hour_17              7.827e+04   1442.776     54.248      0.000    7.54e+04    8.11e+04\n",
      "hour_18              6.918e+04   1442.776     47.949      0.000    6.64e+04     7.2e+04\n",
      "hour_19              5.945e+04   1442.776     41.203      0.000    5.66e+04    6.23e+04\n",
      "hour_20              5.148e+04   1442.776     35.681      0.000    4.87e+04    5.43e+04\n",
      "hour_21              4.042e+04   1442.776     28.015      0.000    3.76e+04    4.32e+04\n",
      "hour_22              3.221e+04   1442.776     22.326      0.000    2.94e+04     3.5e+04\n",
      "hour_23              2.947e+04   1442.776     20.427      0.000    2.66e+04    3.23e+04\n",
      "==============================================================================\n",
      "Omnibus:                   382738.663   Durbin-Watson:                   0.047\n",
      "Prob(Omnibus):                  0.000   Jarque-Bera (JB):          7948804.166\n",
      "Skew:                           3.498   Prob(JB):                         0.00\n",
      "Kurtosis:                      21.044   Cond. No.                         18.0\n",
      "==============================================================================\n",
      "\n",
      "Warnings:\n",
      "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
      "year_2014              1.882648e-183\n",
      "year_2015              1.090080e-208\n",
      "year_2016               0.000000e+00\n",
      "year_2017               0.000000e+00\n",
      "month_August            9.536257e-38\n",
      "month_December          1.934924e-06\n",
      "month_February          1.333031e-53\n",
      "month_January           1.085813e-61\n",
      "month_July              1.832601e-03\n",
      "month_June              2.862598e-63\n",
      "month_March            5.294472e-105\n",
      "month_May               2.349770e-16\n",
      "month_November          3.571436e-02\n",
      "month_October          3.641591e-104\n",
      "month_September         9.459462e-09\n",
      "dayofweek_Monday        6.293912e-03\n",
      "dayofweek_Saturday     4.184516e-169\n",
      "dayofweek_Sunday        0.000000e+00\n",
      "dayofweek_Thursday      3.010450e-43\n",
      "dayofweek_Tuesday       1.213821e-42\n",
      "dayofweek_Wednesday     5.208232e-55\n",
      "hour_1                  5.280704e-70\n",
      "hour_2                  1.447131e-63\n",
      "hour_3                  2.184187e-60\n",
      "hour_4                  3.970404e-65\n",
      "hour_5                  1.643951e-93\n",
      "hour_6                 1.750054e-185\n",
      "hour_7                  0.000000e+00\n",
      "hour_8                  0.000000e+00\n",
      "hour_9                  0.000000e+00\n",
      "hour_10                 0.000000e+00\n",
      "hour_11                 0.000000e+00\n",
      "hour_12                 0.000000e+00\n",
      "hour_13                 0.000000e+00\n",
      "hour_14                 0.000000e+00\n",
      "hour_15                 0.000000e+00\n",
      "hour_16                 0.000000e+00\n",
      "hour_17                 0.000000e+00\n",
      "hour_18                 0.000000e+00\n",
      "hour_19                 0.000000e+00\n",
      "hour_20                1.726600e-278\n",
      "hour_21                1.456382e-172\n",
      "hour_22                2.325718e-110\n",
      "hour_23                 1.039082e-92\n",
      "dtype: float64\n",
      "Mean Absolute Error 98753.94693924116\n"
     ]
    }
   ],
   "source": [
    "feature_summary(X, y, mean_absolute_error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Feature Over Time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "temperature_rolling = train_data.dropna()['temperature'].rolling(24).agg(['mean', 'min', 'max', 'std'])\n",
    "temperature_rolling = pd.concat([train_data.dropna().temperature, temperature_rolling], axis=1)\n",
    "y = train_data.dropna()['consumption']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "DatetimeIndex: 280687 entries, 2014-12-30 17:00:00 to 2017-12-14 23:00:00\n",
      "Data columns (total 5 columns):\n",
      "temperature    280687 non-null float64\n",
      "mean           280664 non-null float64\n",
      "min            280664 non-null float64\n",
      "max            280664 non-null float64\n",
      "std            280664 non-null float64\n",
      "dtypes: float64(5)\n",
      "memory usage: 12.8 MB\n"
     ]
    }
   ],
   "source": [
    "temperature_rolling.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            OLS Regression Results                            \n",
      "==============================================================================\n",
      "Dep. Variable:            consumption   R-squared:                       0.270\n",
      "Model:                            OLS   Adj. R-squared:                  0.270\n",
      "Method:                 Least Squares   F-statistic:                 2.077e+04\n",
      "Date:                Sat, 27 Oct 2018   Prob (F-statistic):               0.00\n",
      "Time:                        15:20:56   Log-Likelihood:            -3.7669e+06\n",
      "No. Observations:              280664   AIC:                         7.534e+06\n",
      "Df Residuals:                  280659   BIC:                         7.534e+06\n",
      "Df Model:                           5                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "===============================================================================\n",
      "                  coef    std err          t      P>|t|      [0.025      0.975]\n",
      "-------------------------------------------------------------------------------\n",
      "temperature  3134.3997     91.158     34.384      0.000    2955.733    3313.066\n",
      "mean        -9238.2956    405.050    -22.808      0.000      -1e+04   -8444.408\n",
      "min         -1.066e+04    350.140    -30.434      0.000   -1.13e+04   -9969.851\n",
      "max          2.018e+04    399.556     50.498      0.000    1.94e+04     2.1e+04\n",
      "std         -3.461e+04    948.488    -36.486      0.000   -3.65e+04   -3.27e+04\n",
      "==============================================================================\n",
      "Omnibus:                   208417.331   Durbin-Watson:                   0.039\n",
      "Prob(Omnibus):                  0.000   Jarque-Bera (JB):          4117187.627\n",
      "Skew:                           3.457   Prob(JB):                         0.00\n",
      "Kurtosis:                      20.443   Cond. No.                         121.\n",
      "==============================================================================\n",
      "\n",
      "Warnings:\n",
      "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
      "temperature    1.494910e-258\n",
      "mean           4.889404e-115\n",
      "min            4.205241e-203\n",
      "max             0.000000e+00\n",
      "std            9.064880e-291\n",
      "dtype: float64\n",
      "Mean Absolute Error 98842.23060077435\n"
     ]
    }
   ],
   "source": [
    "feature_summary(temperature_rolling.interpolate()[23:], y[23:], mean_absolute_error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data =train_data.join(temperature_rolling, rsuffix='_r')\n",
    "train_data.drop('temperature_r', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Adding Lag Feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shifts = np.arange(1, 24)\n",
    "\n",
    "temperature_shifts = pd.DataFrame({'temperature_shift_{}'.format(n_shift) : train_data.temperature.shift(n_shift) for n_shift in\n",
    "                                  shifts})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>series_id</th>\n",
       "      <th>consumption</th>\n",
       "      <th>temperature</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>dayofweek</th>\n",
       "      <th>hour</th>\n",
       "      <th>mean</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>timestamp</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2013-01-02 00:00:00</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 00:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>135802.353408</td>\n",
       "      <td>-3.766667</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>0</td>\n",
       "      <td>11.975000</td>\n",
       "      <td>-3.766667</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>4.562769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 00:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>135802.353408</td>\n",
       "      <td>-3.766667</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>0</td>\n",
       "      <td>11.355903</td>\n",
       "      <td>-3.766667</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>4.798539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 00:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>28843.506793</td>\n",
       "      <td>-3.766667</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>0</td>\n",
       "      <td>11.975000</td>\n",
       "      <td>-3.766667</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>4.562769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 01:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>28843.506793</td>\n",
       "      <td>-3.766667</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>0</td>\n",
       "      <td>11.355903</td>\n",
       "      <td>-3.766667</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>4.798539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 01:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>135586.622902</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>1</td>\n",
       "      <td>11.290972</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>5.583897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 01:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>135586.622902</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>1</td>\n",
       "      <td>10.876389</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>5.676967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 01:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>26445.243198</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>1</td>\n",
       "      <td>11.290972</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>5.583897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 02:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>26445.243198</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>1</td>\n",
       "      <td>10.876389</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>5.676967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 02:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>141357.413933</td>\n",
       "      <td>-3.733333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "      <td>10.631250</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>6.364874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 02:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>141357.413933</td>\n",
       "      <td>-3.733333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "      <td>10.429167</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>6.375430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 02:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>28679.722938</td>\n",
       "      <td>-3.733333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "      <td>10.631250</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>6.364874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 03:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>28679.722938</td>\n",
       "      <td>-3.733333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "      <td>10.429167</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>6.375430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 03:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>138645.373288</td>\n",
       "      <td>-3.666667</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>3</td>\n",
       "      <td>10.034722</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>7.002041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 03:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>138645.373288</td>\n",
       "      <td>-3.666667</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>3</td>\n",
       "      <td>9.972222</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>6.974332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 03:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>26100.127218</td>\n",
       "      <td>-3.666667</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>3</td>\n",
       "      <td>10.034722</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>7.002041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 04:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>26100.127218</td>\n",
       "      <td>-3.666667</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>3</td>\n",
       "      <td>9.972222</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>6.974332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 04:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>144739.760077</td>\n",
       "      <td>-3.633333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>4</td>\n",
       "      <td>9.481944</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>7.538245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 04:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>144739.760077</td>\n",
       "      <td>-3.633333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>4</td>\n",
       "      <td>9.490625</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>7.501025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 04:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>28568.583893</td>\n",
       "      <td>-3.633333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>4</td>\n",
       "      <td>9.481944</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>7.538245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 05:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>28568.583893</td>\n",
       "      <td>-3.633333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>4</td>\n",
       "      <td>9.490625</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>7.501025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 05:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>152267.213798</td>\n",
       "      <td>-3.500000</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>5</td>\n",
       "      <td>8.947222</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>7.990823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 05:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>152267.213798</td>\n",
       "      <td>-3.500000</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>5</td>\n",
       "      <td>9.018750</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>7.952964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 05:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>19852.943025</td>\n",
       "      <td>-3.500000</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>5</td>\n",
       "      <td>8.947222</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>7.990823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 06:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>19852.943025</td>\n",
       "      <td>-3.500000</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>5</td>\n",
       "      <td>9.018750</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>7.952964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 06:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>258221.710795</td>\n",
       "      <td>-3.433333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>6</td>\n",
       "      <td>8.425694</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>8.380507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 06:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>258221.710795</td>\n",
       "      <td>-3.433333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>6</td>\n",
       "      <td>8.571528</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>8.345897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 06:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>26883.949953</td>\n",
       "      <td>-3.433333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>6</td>\n",
       "      <td>8.425694</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>8.380507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 07:00:00</th>\n",
       "      <td>102642.0</td>\n",
       "      <td>26883.949953</td>\n",
       "      <td>-3.433333</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>6</td>\n",
       "      <td>8.571528</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>17.075000</td>\n",
       "      <td>8.345897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2013-01-02 07:00:00</th>\n",
       "      <td>103298.0</td>\n",
       "      <td>275980.954220</td>\n",
       "      <td>-3.066667</td>\n",
       "      <td>2013</td>\n",
       "      <td>January</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>7</td>\n",
       "      <td>7.920833</td>\n",
       "      <td>-3.833333</td>\n",
       "      <td>18.633333</td>\n",
       "      <td>8.700135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-28 18:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>60846.658433</td>\n",
       "      <td>0.188889</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>17</td>\n",
       "      <td>-5.340278</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.834108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-28 19:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>59795.244831</td>\n",
       "      <td>-0.488889</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>18</td>\n",
       "      <td>-5.131944</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.959474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-28 20:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>59701.554510</td>\n",
       "      <td>-1.222222</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>19</td>\n",
       "      <td>-4.923611</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>4.030517</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-28 21:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>60732.148041</td>\n",
       "      <td>-1.955556</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>20</td>\n",
       "      <td>-4.715278</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>4.050097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-28 22:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>61096.499289</td>\n",
       "      <td>-2.577778</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>21</td>\n",
       "      <td>-4.512500</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>4.029291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-28 23:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>186953.830418</td>\n",
       "      <td>-3.144444</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>22</td>\n",
       "      <td>-4.318056</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.975427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 00:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>61533.720786</td>\n",
       "      <td>-3.711111</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>23</td>\n",
       "      <td>-4.131944</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.890525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 01:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>72703.689050</td>\n",
       "      <td>-4.277778</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>0</td>\n",
       "      <td>-3.958796</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.781718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 02:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>61554.540858</td>\n",
       "      <td>-4.844444</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>1</td>\n",
       "      <td>-3.800926</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.655055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 03:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>61575.360929</td>\n",
       "      <td>-5.411111</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>2</td>\n",
       "      <td>-3.658333</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.514235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 04:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>83259.465209</td>\n",
       "      <td>-5.955556</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>3</td>\n",
       "      <td>-3.531019</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.364020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 05:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>85081.221449</td>\n",
       "      <td>-6.488889</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>4</td>\n",
       "      <td>-3.418981</td>\n",
       "      <td>-9.344444</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.208984</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 06:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>84820.970558</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>5</td>\n",
       "      <td>-3.322222</td>\n",
       "      <td>-8.800000</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>3.053806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 07:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>64937.802447</td>\n",
       "      <td>-6.333333</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>6</td>\n",
       "      <td>-3.219444</td>\n",
       "      <td>-7.900000</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>2.899018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 08:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>58972.852014</td>\n",
       "      <td>-5.033333</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>7</td>\n",
       "      <td>-3.100000</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>2.753174</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 09:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>60253.286400</td>\n",
       "      <td>-3.733333</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>8</td>\n",
       "      <td>-2.963889</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>2.629976</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 10:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>99780.191801</td>\n",
       "      <td>-2.188889</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>9</td>\n",
       "      <td>-2.822222</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>2.573393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 11:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>121797.417222</td>\n",
       "      <td>-0.522222</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>10</td>\n",
       "      <td>-2.680556</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>2.603613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 12:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>95231.006218</td>\n",
       "      <td>1.144444</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>11</td>\n",
       "      <td>-2.538889</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>1.322222</td>\n",
       "      <td>2.717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 13:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>96334.469998</td>\n",
       "      <td>2.188889</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>12</td>\n",
       "      <td>-2.408333</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>2.188889</td>\n",
       "      <td>2.868732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 14:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>138484.704385</td>\n",
       "      <td>2.922222</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>13</td>\n",
       "      <td>-2.294444</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>2.922222</td>\n",
       "      <td>3.026257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 15:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>60961.168825</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>14</td>\n",
       "      <td>-2.197222</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>3.181021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 16:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>63022.355886</td>\n",
       "      <td>3.033333</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>15</td>\n",
       "      <td>-2.125926</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>3.280939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 17:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>61721.101428</td>\n",
       "      <td>1.733333</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>16</td>\n",
       "      <td>-2.085185</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>3.324062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 18:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>63782.288489</td>\n",
       "      <td>0.433333</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>17</td>\n",
       "      <td>-2.075000</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>3.331699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 19:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>63480.397455</td>\n",
       "      <td>-0.488889</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>18</td>\n",
       "      <td>-2.075000</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>3.331699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 20:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>82541.172748</td>\n",
       "      <td>-1.222222</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>19</td>\n",
       "      <td>-2.075000</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>3.331699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 21:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>62939.075601</td>\n",
       "      <td>-1.955556</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>20</td>\n",
       "      <td>-2.075000</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>3.331699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 22:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>61637.821143</td>\n",
       "      <td>-2.200000</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>21</td>\n",
       "      <td>-2.059259</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>3.330112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017-12-29 23:00:00</th>\n",
       "      <td>103436.0</td>\n",
       "      <td>105235.050487</td>\n",
       "      <td>-2.200000</td>\n",
       "      <td>2017</td>\n",
       "      <td>December</td>\n",
       "      <td>Friday</td>\n",
       "      <td>22</td>\n",
       "      <td>-2.019907</td>\n",
       "      <td>-7.022222</td>\n",
       "      <td>3.655556</td>\n",
       "      <td>3.322302</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6207224 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     series_id    consumption  temperature  year     month  \\\n",
       "timestamp                                                                    \n",
       "2013-01-02 00:00:00        NaN            NaN          NaN   NaN       NaN   \n",
       "2013-01-02 00:00:00   103298.0  135802.353408    -3.766667  2013   January   \n",
       "2013-01-02 00:00:00   103298.0  135802.353408    -3.766667  2013   January   \n",
       "2013-01-02 00:00:00   102642.0   28843.506793    -3.766667  2013   January   \n",
       "2013-01-02 01:00:00   102642.0   28843.506793    -3.766667  2013   January   \n",
       "2013-01-02 01:00:00   103298.0  135586.622902    -3.833333  2013   January   \n",
       "2013-01-02 01:00:00   103298.0  135586.622902    -3.833333  2013   January   \n",
       "2013-01-02 01:00:00   102642.0   26445.243198    -3.833333  2013   January   \n",
       "2013-01-02 02:00:00   102642.0   26445.243198    -3.833333  2013   January   \n",
       "2013-01-02 02:00:00   103298.0  141357.413933    -3.733333  2013   January   \n",
       "2013-01-02 02:00:00   103298.0  141357.413933    -3.733333  2013   January   \n",
       "2013-01-02 02:00:00   102642.0   28679.722938    -3.733333  2013   January   \n",
       "2013-01-02 03:00:00   102642.0   28679.722938    -3.733333  2013   January   \n",
       "2013-01-02 03:00:00   103298.0  138645.373288    -3.666667  2013   January   \n",
       "2013-01-02 03:00:00   103298.0  138645.373288    -3.666667  2013   January   \n",
       "2013-01-02 03:00:00   102642.0   26100.127218    -3.666667  2013   January   \n",
       "2013-01-02 04:00:00   102642.0   26100.127218    -3.666667  2013   January   \n",
       "2013-01-02 04:00:00   103298.0  144739.760077    -3.633333  2013   January   \n",
       "2013-01-02 04:00:00   103298.0  144739.760077    -3.633333  2013   January   \n",
       "2013-01-02 04:00:00   102642.0   28568.583893    -3.633333  2013   January   \n",
       "2013-01-02 05:00:00   102642.0   28568.583893    -3.633333  2013   January   \n",
       "2013-01-02 05:00:00   103298.0  152267.213798    -3.500000  2013   January   \n",
       "2013-01-02 05:00:00   103298.0  152267.213798    -3.500000  2013   January   \n",
       "2013-01-02 05:00:00   102642.0   19852.943025    -3.500000  2013   January   \n",
       "2013-01-02 06:00:00   102642.0   19852.943025    -3.500000  2013   January   \n",
       "2013-01-02 06:00:00   103298.0  258221.710795    -3.433333  2013   January   \n",
       "2013-01-02 06:00:00   103298.0  258221.710795    -3.433333  2013   January   \n",
       "2013-01-02 06:00:00   102642.0   26883.949953    -3.433333  2013   January   \n",
       "2013-01-02 07:00:00   102642.0   26883.949953    -3.433333  2013   January   \n",
       "2013-01-02 07:00:00   103298.0  275980.954220    -3.066667  2013   January   \n",
       "...                        ...            ...          ...   ...       ...   \n",
       "2017-12-28 18:00:00   103436.0   60846.658433     0.188889  2017  December   \n",
       "2017-12-28 19:00:00   103436.0   59795.244831    -0.488889  2017  December   \n",
       "2017-12-28 20:00:00   103436.0   59701.554510    -1.222222  2017  December   \n",
       "2017-12-28 21:00:00   103436.0   60732.148041    -1.955556  2017  December   \n",
       "2017-12-28 22:00:00   103436.0   61096.499289    -2.577778  2017  December   \n",
       "2017-12-28 23:00:00   103436.0  186953.830418    -3.144444  2017  December   \n",
       "2017-12-29 00:00:00   103436.0   61533.720786    -3.711111  2017  December   \n",
       "2017-12-29 01:00:00   103436.0   72703.689050    -4.277778  2017  December   \n",
       "2017-12-29 02:00:00   103436.0   61554.540858    -4.844444  2017  December   \n",
       "2017-12-29 03:00:00   103436.0   61575.360929    -5.411111  2017  December   \n",
       "2017-12-29 04:00:00   103436.0   83259.465209    -5.955556  2017  December   \n",
       "2017-12-29 05:00:00   103436.0   85081.221449    -6.488889  2017  December   \n",
       "2017-12-29 06:00:00   103436.0   84820.970558    -7.022222  2017  December   \n",
       "2017-12-29 07:00:00   103436.0   64937.802447    -6.333333  2017  December   \n",
       "2017-12-29 08:00:00   103436.0   58972.852014    -5.033333  2017  December   \n",
       "2017-12-29 09:00:00   103436.0   60253.286400    -3.733333  2017  December   \n",
       "2017-12-29 10:00:00   103436.0   99780.191801    -2.188889  2017  December   \n",
       "2017-12-29 11:00:00   103436.0  121797.417222    -0.522222  2017  December   \n",
       "2017-12-29 12:00:00   103436.0   95231.006218     1.144444  2017  December   \n",
       "2017-12-29 13:00:00   103436.0   96334.469998     2.188889  2017  December   \n",
       "2017-12-29 14:00:00   103436.0  138484.704385     2.922222  2017  December   \n",
       "2017-12-29 15:00:00   103436.0   60961.168825     3.655556  2017  December   \n",
       "2017-12-29 16:00:00   103436.0   63022.355886     3.033333  2017  December   \n",
       "2017-12-29 17:00:00   103436.0   61721.101428     1.733333  2017  December   \n",
       "2017-12-29 18:00:00   103436.0   63782.288489     0.433333  2017  December   \n",
       "2017-12-29 19:00:00   103436.0   63480.397455    -0.488889  2017  December   \n",
       "2017-12-29 20:00:00   103436.0   82541.172748    -1.222222  2017  December   \n",
       "2017-12-29 21:00:00   103436.0   62939.075601    -1.955556  2017  December   \n",
       "2017-12-29 22:00:00   103436.0   61637.821143    -2.200000  2017  December   \n",
       "2017-12-29 23:00:00   103436.0  105235.050487    -2.200000  2017  December   \n",
       "\n",
       "                     dayofweek hour       mean       min        max       std  \n",
       "timestamp                                                                      \n",
       "2013-01-02 00:00:00        NaN  NaN        NaN       NaN        NaN       NaN  \n",
       "2013-01-02 00:00:00  Wednesday    0  11.975000 -3.766667  18.633333  4.562769  \n",
       "2013-01-02 00:00:00  Wednesday    0  11.355903 -3.766667  17.075000  4.798539  \n",
       "2013-01-02 00:00:00  Wednesday    0  11.975000 -3.766667  18.633333  4.562769  \n",
       "2013-01-02 01:00:00  Wednesday    0  11.355903 -3.766667  17.075000  4.798539  \n",
       "2013-01-02 01:00:00  Wednesday    1  11.290972 -3.833333  18.633333  5.583897  \n",
       "2013-01-02 01:00:00  Wednesday    1  10.876389 -3.833333  17.075000  5.676967  \n",
       "2013-01-02 01:00:00  Wednesday    1  11.290972 -3.833333  18.633333  5.583897  \n",
       "2013-01-02 02:00:00  Wednesday    1  10.876389 -3.833333  17.075000  5.676967  \n",
       "2013-01-02 02:00:00  Wednesday    2  10.631250 -3.833333  18.633333  6.364874  \n",
       "2013-01-02 02:00:00  Wednesday    2  10.429167 -3.833333  17.075000  6.375430  \n",
       "2013-01-02 02:00:00  Wednesday    2  10.631250 -3.833333  18.633333  6.364874  \n",
       "2013-01-02 03:00:00  Wednesday    2  10.429167 -3.833333  17.075000  6.375430  \n",
       "2013-01-02 03:00:00  Wednesday    3  10.034722 -3.833333  18.633333  7.002041  \n",
       "2013-01-02 03:00:00  Wednesday    3   9.972222 -3.833333  17.075000  6.974332  \n",
       "2013-01-02 03:00:00  Wednesday    3  10.034722 -3.833333  18.633333  7.002041  \n",
       "2013-01-02 04:00:00  Wednesday    3   9.972222 -3.833333  17.075000  6.974332  \n",
       "2013-01-02 04:00:00  Wednesday    4   9.481944 -3.833333  18.633333  7.538245  \n",
       "2013-01-02 04:00:00  Wednesday    4   9.490625 -3.833333  17.075000  7.501025  \n",
       "2013-01-02 04:00:00  Wednesday    4   9.481944 -3.833333  18.633333  7.538245  \n",
       "2013-01-02 05:00:00  Wednesday    4   9.490625 -3.833333  17.075000  7.501025  \n",
       "2013-01-02 05:00:00  Wednesday    5   8.947222 -3.833333  18.633333  7.990823  \n",
       "2013-01-02 05:00:00  Wednesday    5   9.018750 -3.833333  17.075000  7.952964  \n",
       "2013-01-02 05:00:00  Wednesday    5   8.947222 -3.833333  18.633333  7.990823  \n",
       "2013-01-02 06:00:00  Wednesday    5   9.018750 -3.833333  17.075000  7.952964  \n",
       "2013-01-02 06:00:00  Wednesday    6   8.425694 -3.833333  18.633333  8.380507  \n",
       "2013-01-02 06:00:00  Wednesday    6   8.571528 -3.833333  17.075000  8.345897  \n",
       "2013-01-02 06:00:00  Wednesday    6   8.425694 -3.833333  18.633333  8.380507  \n",
       "2013-01-02 07:00:00  Wednesday    6   8.571528 -3.833333  17.075000  8.345897  \n",
       "2013-01-02 07:00:00  Wednesday    7   7.920833 -3.833333  18.633333  8.700135  \n",
       "...                        ...  ...        ...       ...        ...       ...  \n",
       "2017-12-28 18:00:00   Thursday   17  -5.340278 -9.344444   1.322222  3.834108  \n",
       "2017-12-28 19:00:00   Thursday   18  -5.131944 -9.344444   1.322222  3.959474  \n",
       "2017-12-28 20:00:00   Thursday   19  -4.923611 -9.344444   1.322222  4.030517  \n",
       "2017-12-28 21:00:00   Thursday   20  -4.715278 -9.344444   1.322222  4.050097  \n",
       "2017-12-28 22:00:00   Thursday   21  -4.512500 -9.344444   1.322222  4.029291  \n",
       "2017-12-28 23:00:00   Thursday   22  -4.318056 -9.344444   1.322222  3.975427  \n",
       "2017-12-29 00:00:00   Thursday   23  -4.131944 -9.344444   1.322222  3.890525  \n",
       "2017-12-29 01:00:00     Friday    0  -3.958796 -9.344444   1.322222  3.781718  \n",
       "2017-12-29 02:00:00     Friday    1  -3.800926 -9.344444   1.322222  3.655055  \n",
       "2017-12-29 03:00:00     Friday    2  -3.658333 -9.344444   1.322222  3.514235  \n",
       "2017-12-29 04:00:00     Friday    3  -3.531019 -9.344444   1.322222  3.364020  \n",
       "2017-12-29 05:00:00     Friday    4  -3.418981 -9.344444   1.322222  3.208984  \n",
       "2017-12-29 06:00:00     Friday    5  -3.322222 -8.800000   1.322222  3.053806  \n",
       "2017-12-29 07:00:00     Friday    6  -3.219444 -7.900000   1.322222  2.899018  \n",
       "2017-12-29 08:00:00     Friday    7  -3.100000 -7.022222   1.322222  2.753174  \n",
       "2017-12-29 09:00:00     Friday    8  -2.963889 -7.022222   1.322222  2.629976  \n",
       "2017-12-29 10:00:00     Friday    9  -2.822222 -7.022222   1.322222  2.573393  \n",
       "2017-12-29 11:00:00     Friday   10  -2.680556 -7.022222   1.322222  2.603613  \n",
       "2017-12-29 12:00:00     Friday   11  -2.538889 -7.022222   1.322222  2.717742  \n",
       "2017-12-29 13:00:00     Friday   12  -2.408333 -7.022222   2.188889  2.868732  \n",
       "2017-12-29 14:00:00     Friday   13  -2.294444 -7.022222   2.922222  3.026257  \n",
       "2017-12-29 15:00:00     Friday   14  -2.197222 -7.022222   3.655556  3.181021  \n",
       "2017-12-29 16:00:00     Friday   15  -2.125926 -7.022222   3.655556  3.280939  \n",
       "2017-12-29 17:00:00     Friday   16  -2.085185 -7.022222   3.655556  3.324062  \n",
       "2017-12-29 18:00:00     Friday   17  -2.075000 -7.022222   3.655556  3.331699  \n",
       "2017-12-29 19:00:00     Friday   18  -2.075000 -7.022222   3.655556  3.331699  \n",
       "2017-12-29 20:00:00     Friday   19  -2.075000 -7.022222   3.655556  3.331699  \n",
       "2017-12-29 21:00:00     Friday   20  -2.075000 -7.022222   3.655556  3.331699  \n",
       "2017-12-29 22:00:00     Friday   21  -2.059259 -7.022222   3.655556  3.330112  \n",
       "2017-12-29 23:00:00     Friday   22  -2.019907 -7.022222   3.655556  3.322302  \n",
       "\n",
       "[6207224 rows x 11 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}