notebook-without-output.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "54e0459f",
   "metadata": {
    "id": "54e0459f"
   },
   "source": [
    "<a id='toc'></a>\n",
    "## Table of Contents\n",
    "\n",
    "* [1. About the project](#about-project)\n",
    "* [2. Import libraries and load data](#import-libraries-load-data)\n",
    "* [3 Exploratory data analysis (EDA)](#eda)\n",
    "  * [3.1 EDA - basic](#eda-basic)\n",
    "  * [3.2 EDA - additional](#eda-additional)\n",
    "* [4 Baseline model](#baseline-model)\n",
    "* [5 Improvement over baseline model](#baseline-improvement)\n",
    "  * [5.1 Linear Regression](#linear-regression)\n",
    "  * [5.2 Decision Tree](#decision-tree)\n",
    "  * [5.3 Random Forest](#random-forest)\n",
    "  * [5.4 XGBoost](#xgb)\n",
    "* [6. Final model](#final-model)\n",
    "  * [6.1 Compare results from hyper-parameter tuning for the different models and choose final model](#choose-final-model)\n",
    "  * [6.2 Train final model](#train-final-model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4b7748c",
   "metadata": {
    "id": "b4b7748c"
   },
   "source": [
    "<a class='anchor' id='about-project'></a>\n",
    "[back to TOC](#toc)\n",
    "### 1. About the project:\n",
    "\n",
    "The purpose of this project is to try predict the future bike shares in London based on the past information on bike sharing in London.\n",
    "\n",
    "With cost of living increasing day by day, congestion charges levied in many parts of London, the choice of using public transport makes sense to people. But what if you want to travel shorter distances, still have your independence and maintain a healthy lifestyle - thats where bike sharing started (is my assumption). Many organizations like Santander offer public bike sharing schemes with several docking stations across London to help and encourage this lifestyle. However it is also a challenge to maintain the requisite number of bikes. The goal of this project is to try and predict the bike share numbers using Machine Learning.\n",
    "\n",
    "This is a regression problem.\n",
    "\n",
    "The trained model can then be deployed as a web service (locally / on a Docker container / in Cloud). Organizations managing bikes for sharing, can then use this service to get predictions on bikes required to be maintained at different times or in different weather conditions. This will also help them plan maintainance of bikes when there could be less demand.\n",
    "\n",
    "Acknowledgements: The dataset is Powered by TfL Open Data. The data contains OS data © Crown copyright and database rights 2016' and Geomni UK Map data © and database rights [2019].\n",
    "\n",
    "\n",
    "Data Source: https://www.kaggle.com/hmavrodiev/london-bike-sharing-dataset\n",
    "\n",
    "Acknowledgements: The dataset is Powered by TfL Open Data. The data contains OS data © Crown copyright and database rights 2016' and Geomni UK Map data © and database rights [2019].\n",
    "\n",
    "Reference to orignal data - https://cycling.data.tfl.gov.uk/\n",
    "\n",
    "#### Data description\n",
    "* \"timestamp\" - timestamp field for grouping the data\n",
    "* \"cnt\" - the count of a new bike shares\n",
    "* \"t1\" - real temperature in C\n",
    "* \"t2\" - temperature in C \"feels like\"\n",
    "* \"hum\" - humidity in percentage\n",
    "* \"windspeed\" - wind speed in km/h\n",
    "* \"weather_code\" - category of the weather\n",
    "* \"is_holiday\" - boolean field - 1 holiday / 0 non holiday\n",
    "* \"is_weekend\" - boolean field - 1 if the day is weekend\n",
    "* \"season\" - category field meteorological seasons: 0-spring ; 1-summer; 2-fall; 3-winter.\n",
    "\n",
    "* \"weather_code\" category description:\n",
    "1 = Clear ; mostly clear but have some values with haze/fog/patches of fog/ fog in vicinity 2 = scattered clouds / few clouds 3 = Broken clouds 4 = Cloudy 7 = Rain/ light Rain shower/ Light rain 10 = rain with thunderstorm 26 = snowfall 94 = Freezing Fog\n",
    "\n",
    "\n",
    "#### Additional info\n",
    "The data is acquired from 3 sources:\n",
    "\n",
    "https://cycling.data.tfl.gov.uk/ 'Contains OS data © Crown copyright and database rights 2016' and Geomni UK Map data © and database rights [2019] 'Powered by TfL Open Data'\n",
    "freemeteo.com - weather data\n",
    "https://www.gov.uk/bank-holidays\n",
    "From 1/1/2015 to 31/12/2016\n",
    "The data from cycling dataset is grouped by \"Start time\", this represent the count of new bike shares grouped by hour. The long duration shares are not taken in the count."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1b8ea53",
   "metadata": {
    "id": "f1b8ea53"
   },
   "source": [
    "<a id='import-libraries-load-data'></a>\n",
    "### 2. Import libraries and load data\n",
    "[back to TOC](#toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd4df8e8",
   "metadata": {
    "id": "bd4df8e8"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import plotly.express as px  # For interactive graphs\n",
    "\n",
    "from sklearn.model_selection import train_test_split, TimeSeriesSplit\n",
    "from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
    "from sklearn.metrics import mutual_info_score, normalized_mutual_info_score, adjusted_mutual_info_score, r2_score, mean_squared_error, mean_absolute_error, mean_squared_log_error\n",
    "\n",
    "from sklearn.tree import DecisionTreeRegressor, export_text\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "import xgboost as xgb\n",
    "\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn import preprocessing\n",
    "\n",
    "import datetime\n",
    "\n",
    "import pickle\n",
    "\n",
    "from IPython.display import display\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c62e54c",
   "metadata": {
    "id": "0c62e54c"
   },
   "source": [
    "#### Load data and have a look"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d0ede60",
   "metadata": {
    "id": "0d0ede60"
   },
   "outputs": [],
   "source": [
    "datafile = 'london_merged.csv'\n",
    "df = pd.read_csv(datafile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59f0e6a0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "id": "59f0e6a0",
    "outputId": "e57decea-a51a-4687-9ca3-00e174fb8d9b"
   },
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3865b41b",
   "metadata": {
    "id": "3865b41b"
   },
   "source": [
    "**Observations:** Actually the data has certain categorical features like weather_code, is_holiday, is_weekend and season, however these have already been encoded into numbers as part of the dataset that has been made available. Hence, we will hopefully not need much pre-processing for these features, lets see"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83a1d908",
   "metadata": {
    "id": "83a1d908"
   },
   "source": [
    "<a id='eda'></a>\n",
    "### 3. Exploratory Data Analysis\n",
    "\n",
    "[back to TOC](#toc)\n",
    "\n",
    "This section performs various analysis of the dataset, split it into training, validation, test.\n",
    "\n",
    "* [3.1 EDA - basic](#eda-basic)\n",
    "* [3.2 EDA - additional](#eda-additional)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8d5b5e5",
   "metadata": {
    "id": "b8d5b5e5"
   },
   "source": [
    "<a id='eda-basic'></a>\n",
    "#### 3.1  EDA - Basic\n",
    "\n",
    "* Check if columns are correctly classified as numerical and categorical *(sometimes numerical columns are marked categorical or vice versa)*\n",
    "* Check if any numerical features have extremely high values *(sometimes NaNs are coded as high number like 99999999)*\n",
    "* Check cardinality of categorical features *(if very high cardinality then using one-hot encoding may create a lot of features)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a67a5ac9",
   "metadata": {
    "id": "a67a5ac9"
   },
   "source": [
    "#### Let us check for the data types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04a1e75d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "04a1e75d",
    "outputId": "6879fd6e-7862-4331-e581-252a285d1c44"
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45aaacf2",
   "metadata": {
    "id": "45aaacf2"
   },
   "source": [
    "**Observations:**  \n",
    "* The actual categorical features (weather_code, is_holiday, is_weekend and season) are already encoded as numbers. We will later perform experiments to see if this numerical encoding helps the model or instead using one-hot encoding proves to be better.\n",
    "* The encoded categorical values have float64 as the dtype. Let us check if they have any float or negative values, else we can convert these to uint8 (unsigned 8 bit integer) to save on memory. Infact most features are dtype float, so we will also see if any dtype conversions are possible to save memory.\n",
    "* Current memory usage is 1.3MB - which is not huge, but every MB helps :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "668ff0eb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "668ff0eb",
    "outputId": "17c9027f-460d-4960-fce7-86b2dea8d8b7"
   },
   "outputs": [],
   "source": [
    "df['weather_code'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc3df54b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dc3df54b",
    "outputId": "708a9e71-f90e-4364-e590-3fcd317dbf8c"
   },
   "outputs": [],
   "source": [
    "df['is_holiday'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26929c67",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "26929c67",
    "outputId": "cf57dc23-baad-4f1b-9063-6d5de0bcffe7"
   },
   "outputs": [],
   "source": [
    "df['is_weekend'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac063c1e",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ac063c1e",
    "outputId": "49853f81-1064-4e04-8c9f-11c5e5199a3e"
   },
   "outputs": [],
   "source": [
    "df['season'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd818781",
   "metadata": {
    "id": "fd818781"
   },
   "source": [
    "**Observations:** None of the encoded categorical features actually have float values (as expected) and the values are below 255, so we can convert these to uint8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96585efc",
   "metadata": {
    "id": "96585efc"
   },
   "source": [
    "**Useful tip:** \n",
    "\n",
    "* Numpy provides numpy.iinfo() to know the range (min/max values) of a particular integer dtype. \n",
    "    * e.g. numpy.iinfo(numpy.int16) can show the range for dtype of int16. \n",
    "* You can also use max and min to find the max or min\n",
    "    * e.g. numpy.iinfo(numpy.int16).max or numpy.iinfo(numpy.int16).min\n",
    "* Similar to iinfo, Numpy also provides finfo() to know the range of a particular float dtype. \n",
    "    * e.g. numpy.finfo(numpy.float64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a14893a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7a14893a",
    "outputId": "a1d50bcf-f5ff-4b93-f9b8-6e939bff5ef3"
   },
   "outputs": [],
   "source": [
    "print(np.iinfo(np.uint8))\n",
    "print(\"Min Max for uint8 :\", np.iinfo(np.uint8).min, np.iinfo(np.uint8).max)\n",
    "print('-'*50)\n",
    "\n",
    "print(np.finfo(np.float16))\n",
    "print(\"Min Max for float16:\", np.finfo(np.float16).min, np.finfo(np.float16).max)\n",
    "print('-'*50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1e6da57",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c1e6da57",
    "outputId": "d705304d-f640-4450-8df0-04a64594a3d9"
   },
   "outputs": [],
   "source": [
    "df['weather_code'] = df['weather_code'].astype('uint8')\n",
    "df['is_holiday'] = df['is_holiday'].astype('uint8')\n",
    "df['is_weekend'] = df['is_weekend'].astype('uint8')\n",
    "df['season'] = df['season'].astype('uint8')\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "641c12cb",
   "metadata": {
    "id": "641c12cb"
   },
   "source": [
    "#### Let us also check distinct values for 't1', 't2', 'hum' and 'wind_speed' to see if any of these also need to be float64 or can be converted to int/uint to reduce memory consumption"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2c6eb57",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d2c6eb57",
    "outputId": "60b442e6-f49d-44e6-b59e-16cb04669254"
   },
   "outputs": [],
   "source": [
    "df['t1'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "906e96c2",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "906e96c2",
    "outputId": "da7d8dda-d68e-42b1-e73f-e9058845bb74"
   },
   "outputs": [],
   "source": [
    "df['t2'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c7b02ac",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7c7b02ac",
    "outputId": "a59423ca-6660-4d0f-b1f7-588ca97b2ccc"
   },
   "outputs": [],
   "source": [
    "df['hum'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92e24d0b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "92e24d0b",
    "outputId": "c17956da-4955-4040-9c78-9b5c62e95f06"
   },
   "outputs": [],
   "source": [
    "df['wind_speed'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbf2ca74",
   "metadata": {
    "id": "cbf2ca74"
   },
   "source": [
    "Let us check for significantly high values (sometimes NaNs are set as high values in a processed dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32ca4f5e",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 294
    },
    "id": "32ca4f5e",
    "outputId": "18cab0b8-20b7-432a-da61-4b2404df7485"
   },
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3cadf88",
   "metadata": {
    "id": "a3cadf88"
   },
   "source": [
    "**Observations:** \n",
    "* We can see that there are no significantly high values, so there is no encoding done for NaNs and there are no NaNs as seen earlier.\n",
    "\n",
    "**Additional Observations**\n",
    "* We can see that since t1 and t2 relate to temperature in degree C, these can have negative and positive values, as well as float values. However these cannot have values beyond -100 or +100 for instance. So we can use float16 for these.\n",
    "* hum (humidity) and wind_speed will have positive values and can have float values. Again, these cannot have very high values, so we can use float16 here also."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6367e83",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b6367e83",
    "outputId": "13028481-ab19-4665-9add-29298c037fb1"
   },
   "outputs": [],
   "source": [
    "df['t1'] = df['t1'].astype('float16')\n",
    "df['t2'] = df['t2'].astype('float16')\n",
    "df['hum'] = df['hum'].astype('float16')\n",
    "df['wind_speed'] = df['wind_speed'].astype('float16')\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc53e82a",
   "metadata": {
    "id": "dc53e82a"
   },
   "source": [
    "**Observations:** We have managed to bring down the memory consumption from 1.3MB to 480MB which is like 63% saving."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6d3307c",
   "metadata": {
    "id": "b6d3307c"
   },
   "source": [
    "<a id='eda-additional'></a>\n",
    "#### 3.2. EDA - additional\n",
    "[back to TOC](#toc)\n",
    "\n",
    "**Note:** Since this is a time series dataset, will need the dataset to be sorted according to timestamp first and then split in sequence across train, validation and test. This is necessary to avoid data leakage (because if the model sees data from future timestamps, it will already know what to expect and how to predict)\n",
    "\n",
    "* Sort the data as per timestamp\n",
    "* Then split the data into Train (70%), Validation (20%) and Test (10%) without shuffling (so that time sequence is mainted when distributing the data across Train, Val, Test)\n",
    "* Check for missing data and impute if data is missing\n",
    "* Look at the distribution of features\n",
    "* Feature importance - using mutual information score for categorical features and using correlation for numerical features\n",
    "* Additional EDA like adding/creating new features, applying transformations and encoding etc. is taken up in the Model training / evaluation part of this notebook. This is done so that we do not just create additional features but also evaluate if it actually helps."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f059e354",
   "metadata": {
    "id": "f059e354"
   },
   "source": [
    "#### Sort data according to timestamp\n",
    "\n",
    "The dataset is possibly already sorted as per timestamp, but cant be very sure, so let us do it again. Since the timestamp feature is currently dtype object, we will first convert it to pandas datetime format and then sort the data according to timestamp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb6e14f2",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "id": "cb6e14f2",
    "outputId": "e1a29213-263b-4809-caf9-67deeb520df7"
   },
   "outputs": [],
   "source": [
    "df['timestamp'] = pd.to_datetime(df['timestamp'])\n",
    "df = df.sort_values(by=['timestamp'],ascending=True)\n",
    "df.reset_index(drop=True,inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa9defd0",
   "metadata": {
    "id": "fa9defd0"
   },
   "source": [
    "#### Splitting data as Train (70%), Val (20%), Test (10%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d873737",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6d873737",
    "outputId": "35470383-9e0c-4c44-b3ea-e2423fa342cb"
   },
   "outputs": [],
   "source": [
    "df_full_train, df_test = train_test_split(df,test_size=0.1,shuffle=False,random_state=1)\n",
    "df_train, df_val = train_test_split(df_full_train,test_size=0.22,shuffle=False,random_state=1)\n",
    "\n",
    "print(f'train : {round(df_train.shape[0]/df.shape[0],2)}, val: {round(df_val.shape[0]/df.shape[0],2)}, test: {round(df_test.shape[0]/df.shape[0],2)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9eacb35",
   "metadata": {
    "id": "e9eacb35"
   },
   "source": [
    "Let us have a look at the train, val, test datasets and see that they are still time sorted and have not been shuffled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88fc7e76",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "id": "88fc7e76",
    "outputId": "3692f058-9bc8-4080-bf97-53e1efaba942"
   },
   "outputs": [],
   "source": [
    "df_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51f5f131",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "id": "51f5f131",
    "outputId": "97b2ef58-b6b3-42fc-8f08-4225e3940182"
   },
   "outputs": [],
   "source": [
    "df_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae5f98aa",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "id": "ae5f98aa",
    "outputId": "de693954-41d4-431e-ab19-01b18635ae8b"
   },
   "outputs": [],
   "source": [
    "df_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72710ab4",
   "metadata": {
    "id": "72710ab4"
   },
   "source": [
    "**Observations:** Data seems time sorted and not shuffled. Now we can do the EDA on train dataset (EDA is being done after split to avoid data snooping)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f970dace",
   "metadata": {
    "id": "f970dace"
   },
   "source": [
    "#### Check for missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31bc74f8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "31bc74f8",
    "outputId": "7140d9a2-c650-447e-8bc1-b2b63ab751a8"
   },
   "outputs": [],
   "source": [
    "df_train.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91c441c2",
   "metadata": {
    "id": "91c441c2"
   },
   "source": [
    "**Observations:** There is no missing data fortunately."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35916fd8",
   "metadata": {
    "id": "35916fd8"
   },
   "source": [
    "#### Let us delete the target variable from the datasets after saving it as y_train, y_val and y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3339c78e",
   "metadata": {
    "id": "3339c78e"
   },
   "outputs": [],
   "source": [
    "y_train = df_train['cnt']\n",
    "y_val = df_val['cnt']\n",
    "y_test = df_test['cnt']\n",
    "\n",
    "del df_train['cnt']\n",
    "del df_val['cnt']\n",
    "del df_test['cnt']\n",
    "\n",
    "# Have kept df_full_train as it is, without deleting the target variable, \n",
    "# so that we can use this data for cross validation during model training or hyper parameter tunning if required"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e6b965d",
   "metadata": {
    "id": "3e6b965d"
   },
   "source": [
    "#### Check distribution of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4b28af8",
   "metadata": {
    "id": "b4b28af8"
   },
   "outputs": [],
   "source": [
    "num_features = ['t1', 't2', 'hum', 'wind_speed']\n",
    "cat_features = ['weather_code', 'is_holiday', 'is_weekend', 'season']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fc60c61",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 606
    },
    "id": "0fc60c61",
    "outputId": "c21ab2f0-bbc2-4bce-d936-dd79307c5637"
   },
   "outputs": [],
   "source": [
    "# Distribution of numerical features\n",
    "\n",
    "rows, columns = 2, 2\n",
    "n_row, n_col = 0, 0\n",
    "fig, axes = plt.subplots(rows, columns, figsize=(20,10))\n",
    "for num_ft in num_features:\n",
    "    if n_col < columns:\n",
    "        axes[n_row][n_col].hist(df[num_ft])\n",
    "        axes[n_row][n_col].set_title(num_ft)\n",
    "        n_col = n_col + 1\n",
    "    else:\n",
    "        n_row = 1\n",
    "        axes[n_row][n_col-columns].hist(df[num_ft])\n",
    "        axes[n_row][n_col-columns].set_title(num_ft)\n",
    "        n_col = n_col + 1\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7acbb40",
   "metadata": {
    "id": "f7acbb40"
   },
   "source": [
    "**Observations:** Numerical features although not perfectly normalised, have a good distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b85fd54",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 606
    },
    "id": "8b85fd54",
    "outputId": "76eefc11-672a-4a94-a286-de4db1d6b9f3"
   },
   "outputs": [],
   "source": [
    "# Distribution of categorical features\n",
    "\n",
    "rows, columns = 2, 2\n",
    "n_row, n_col = 0, 0\n",
    "fig, axes = plt.subplots(rows, columns, figsize=(20,10))\n",
    "for num_ft in cat_features:\n",
    "    if n_col < columns:\n",
    "        axes[n_row][n_col].hist(df[num_ft])\n",
    "        axes[n_row][n_col].set_title(num_ft)\n",
    "        n_col = n_col + 1\n",
    "    else:\n",
    "        n_row = 1\n",
    "        axes[n_row][n_col-columns].hist(df[num_ft])\n",
    "        axes[n_row][n_col-columns].set_title(num_ft)\n",
    "        n_col = n_col + 1\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62aee8a3",
   "metadata": {
    "id": "62aee8a3"
   },
   "source": [
    "#### Feature weather_code is skewed and not normalised. Let us check if log transforming improves the distribution.\n",
    "\n",
    "Transformation may not help in distribution of is_holiday and is_weekend since they are binary anyway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9ec51dd",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 353
    },
    "id": "d9ec51dd",
    "outputId": "4d253ec3-4047-477d-e4af-c77e154a7673"
   },
   "outputs": [],
   "source": [
    "plt.hist(np.log1p(df_train['weather_code']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46dbc291",
   "metadata": {
    "id": "46dbc291"
   },
   "source": [
    "**Observations:** Now the distribution looks comparatively better. We will try this transformation in model training."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf541bee",
   "metadata": {
    "id": "cf541bee"
   },
   "source": [
    "#### Check distribution of target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9032384",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 353
    },
    "id": "a9032384",
    "outputId": "4e8f91b2-788a-4586-8b48-da5b06e22b63"
   },
   "outputs": [],
   "source": [
    "plt.hist(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f42a2ba",
   "metadata": {
    "id": "0f42a2ba"
   },
   "source": [
    "#### Let us check if log transformation on the target feature helps in normalising the feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12ac3e56",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 371
    },
    "id": "12ac3e56",
    "outputId": "d44dd621-cf77-4a5f-80f7-7115452236e4"
   },
   "outputs": [],
   "source": [
    "plt.hist(np.log1p(y_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "497c0887",
   "metadata": {
    "id": "497c0887"
   },
   "source": [
    "**Observations:** The distribution is much better with log transformation. We will use this in model evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ab01af2",
   "metadata": {
    "id": "9ab01af2"
   },
   "source": [
    "#### Let us further visualize the distribution of various features and target to see outliers, the range of values etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a902c123",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 296
    },
    "id": "a902c123",
    "outputId": "058b97cc-1c12-4a89-8de1-b8a425cfb2d5"
   },
   "outputs": [],
   "source": [
    "# Boxplot\n",
    "# plt.figure(figsize = (8,6))\n",
    "sns.boxplot(x = y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9278c9df",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 386
    },
    "id": "9278c9df",
    "outputId": "74473dbd-a4ce-4ead-ea28-1a5c67da5c5f"
   },
   "outputs": [],
   "source": [
    "sns.displot(x = y_train, aspect = 2, kde = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03e5ab8f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "03e5ab8f",
    "outputId": "5a819236-d635-4a60-ee1e-aa9db1478a0f"
   },
   "outputs": [],
   "source": [
    "sns.violinplot(y = y_train, x = df_train['is_holiday'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5be4e22a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "5be4e22a",
    "outputId": "493584e9-9fc9-4298-82d8-d4f366122ce2"
   },
   "outputs": [],
   "source": [
    "sns.violinplot(y = y_train, x = df_train['is_weekend'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ab313c6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "8ab313c6",
    "outputId": "d314cf01-3995-478b-cdd6-5c2cb491eac5"
   },
   "outputs": [],
   "source": [
    "sns.violinplot(y = y_train, x = df_train['weather_code'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa4e2bb5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 296
    },
    "id": "fa4e2bb5",
    "outputId": "3f7c8dc5-4c8e-4f9a-ad38-cb3aaf168ceb"
   },
   "outputs": [],
   "source": [
    "sns.violinplot(y = y_train, x = df_train['season'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b3d03db",
   "metadata": {
    "id": "0b3d03db"
   },
   "source": [
    "#### Feature Importance\n",
    "\n",
    "#### Mutual information with categorical features\n",
    "\n",
    "Apart from the sklearn.metrics.mutual_info_score, I found that there are sklearn.metrics.cluster.normalized_mutual_info_score and sklearn.metrics.cluster.adjusted_mutual_info_score available which provide mutual_info_score normalised (i.e. scaled between 0 and 1) and adjusted against chance. Will check all these scores.\n",
    "\n",
    "* https://scikit-learn.org/stable/modules/generated/sklearn.metrics.normalized_mutual_info_score.html\n",
    "* https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_mutual_info_score.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c7ca36e",
   "metadata": {
    "id": "4c7ca36e"
   },
   "outputs": [],
   "source": [
    "def fn_mutual_info_score(series):\n",
    "    return mutual_info_score(series,y)\n",
    "\n",
    "def fn_norm_mutual_info_score(series):\n",
    "    return normalized_mutual_info_score(series,y)\n",
    "\n",
    "def fn_adj_mutual_info_score(series):\n",
    "    return adjusted_mutual_info_score(series,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cca03b18",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 171
    },
    "id": "cca03b18",
    "outputId": "840b5bf2-d13b-4516-e1d8-b1bb75bc9aca"
   },
   "outputs": [],
   "source": [
    "y = y_train\n",
    "mi = df_train[cat_features].apply(fn_mutual_info_score)\n",
    "mi =pd.concat([mi,df_train[cat_features].apply(fn_norm_mutual_info_score)],axis=1)\n",
    "mi =pd.concat([mi,df_train[cat_features].apply(fn_adj_mutual_info_score)],axis=1)\n",
    "mi.columns = ['mi','norm_mi','adj_mi']\n",
    "mi.sort_values(by=['mi'],ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7582bb41",
   "metadata": {
    "id": "7582bb41"
   },
   "source": [
    "**Observations:** We can see that weather_code, season and is_weekend have good mutual info with the target and hence are useful for the model, while is_holiday has lesser score."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "600b183d",
   "metadata": {
    "id": "600b183d"
   },
   "source": [
    "#### Co-relation of numerical features with target variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4af1a898",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4af1a898",
    "outputId": "c8313266-9ece-4c2c-9133-864240c871ab"
   },
   "outputs": [],
   "source": [
    "df_train[num_features].corrwith(y_train).sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db9b3bae",
   "metadata": {
    "id": "db9b3bae"
   },
   "source": [
    "**Observations:** We can see that humidity (hum) has good correlation with target and is inversely correlated, while the temperatures (t1 and t2) also have good correlation with target. wind_speed has correlation with target but is less"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01dc0a18",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 505
    },
    "id": "01dc0a18",
    "outputId": "5c8ab5e9-c389-4fd3-f617-7ad667c0909b"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16, 8))\n",
    "sns.heatmap(df_train[num_features].corr(), annot=True, fmt='.3f')\n",
    "# sns.heatmap(df_full_train[t_num_cols].corr(), annot=True, fmt='.3f')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12c3bd4b",
   "metadata": {
    "id": "12c3bd4b"
   },
   "source": [
    "**Observations:** We can see that temperatures t1 and t2 are highly correlated, obviously. Also hum and t1, t2 are related in a negative way. There is some correlation between wind_speed and hum and wind_speed and the temperatures also."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da768eba",
   "metadata": {
    "id": "da768eba"
   },
   "source": [
    "<a class='anchor' id='baseline-model'></a>\n",
    "[back to TOC](#toc)\n",
    "### Baseline model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b33ea6b1",
   "metadata": {},
   "source": [
    "#### Common functions across experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c98a8ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to train the model and predict on validation data\n",
    "\n",
    "def train_predict(df_train_copy,df_val_copy,y_train_copy,model):\n",
    "    X_train = df_train_copy.values\n",
    "    model.fit(X_train, y_train_copy)\n",
    "\n",
    "    X_val = df_val_copy.values\n",
    "    y_pred = model.predict(X_val)\n",
    "    \n",
    "    y_train_pred = model.predict(X_train)\n",
    "    \n",
    "    return y_pred, y_train_pred, model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd650179",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to evaluate various metrics/scores on predictions on validation and training\n",
    "\n",
    "def evaluate_scores(y_val_eval, y_pred_eval, y_train_eval, y_pred_train_eval):\n",
    "    scores = {}\n",
    "#     print(\"evaluating scores for val\")\n",
    "#     print(len(y_val_copy),len(y_pred_eval),len(y_train_copy),len(y_pred_train_eval))\n",
    "    scores['val_r2'] = r2_score(y_val_eval, y_pred_eval)\n",
    "    scores['val_mse'] = mean_squared_error(y_val_eval, y_pred_eval,squared=True)\n",
    "    scores['val_rmse'] = mean_squared_error(y_val_eval, y_pred_eval,squared=False)\n",
    "    scores['val_mae'] = mean_absolute_error(y_val_eval, y_pred_eval)\n",
    "#     scores['val_msle'] = mean_squared_log_error(y_val_eval, y_pred_eval)\n",
    "#     scores['val_rmsle'] = np.sqrt(mean_squared_log_error(y_val_eval, y_pred_eval))\n",
    "    if exp == \"baseline\":\n",
    "        scores['diff'] = 0\n",
    "    else:\n",
    "        scores['diff'] = baseline_rmse - scores['val_rmse']\n",
    "    \n",
    "#     print(\"evaluating scores for train\")\n",
    "    scores['train_r2'] = r2_score(y_train_eval, y_pred_train_eval)\n",
    "    scores['train_mse'] = mean_squared_error(y_train_eval, y_pred_train_eval,squared=True)\n",
    "    scores['train_rmse'] = mean_squared_error(y_train_eval, y_pred_train_eval,squared=False)\n",
    "    scores['train_mae'] = mean_absolute_error(y_train_eval, y_pred_train_eval)\n",
    "#     scores['train_msle'] = mean_squared_log_error(y_train_eval, y_pred_train_eval)\n",
    "#     scores['train_rmsle'] = np.sqrt(mean_squared_log_error(y_train_eval, y_pred_train_eval))\n",
    "\n",
    "    rnd_digits = 5 #round upto how many digits\n",
    "    for metric, value in scores.items():\n",
    "        scores[metric] = round(scores[metric],rnd_digits)\n",
    "    \n",
    "    return scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddbc3ce8",
   "metadata": {},
   "source": [
    "#### Baseline with LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26dafe31",
   "metadata": {
    "id": "26dafe31"
   },
   "outputs": [],
   "source": [
    "#Save all scores into a pandas dataframe so that we can have a comparative study across experiments\n",
    "\n",
    "exp_columns = [\n",
    "    \"algo\", \n",
    "    \"experiment\",\n",
    "    \"desc\", \n",
    "    \"val_r2\", \n",
    "    \"val_mse\", \n",
    "    \"val_rmse\", \n",
    "    \"val_mae\", \n",
    "#     \"val_msle\", # Commented because msle cannot be used when target has negative values. And some preds are coming negative\n",
    "#     \"val_rmsle\", \n",
    "    \"train_r2\", \n",
    "    \"train_mse\",\n",
    "    \"train_rmse\", \n",
    "    \"train_mae\", \n",
    "#     \"train_msle\", \n",
    "#     \"train_rmsle\", \n",
    "    \"diff\"]\n",
    "exp_scores = pd.DataFrame(columns = exp_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f955cd38",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 78
    },
    "id": "f955cd38",
    "outputId": "e3b92913-6f59-47b2-e8d1-f0917db61c27"
   },
   "outputs": [],
   "source": [
    "#Experiment 0 - baseline\n",
    "\n",
    "# For every experiment we will use a copy of the train, validation dataframes (df_train, df_val) \n",
    "# and target series (y_train, y_val) to avoid any intermediate processing to affect subsequent experiments\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "features = ['t1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = LinearRegression()\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "exp = \"baseline\"\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"linear\", \"experiment\": 0, \"desc\": \"baseline\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "baseline_rmse = exp_scores.iloc[0]['val_rmse']  # Save baseline_rmse for comparison of further experiments score\n",
    "exp = \"rest\"\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b18661b5",
   "metadata": {},
   "source": [
    "<a class='anchor' id='baseline-improvement'></a>\n",
    "[back to TOC](#toc)\n",
    "    \n",
    "## 5. Improvement over baseline model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "434a51ef",
   "metadata": {},
   "source": [
    "<a id='linear-regression'></a>\n",
    "#### 5.1. Linear Regression\n",
    "[back to TOC](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "N9QSrxwwDNTx",
   "metadata": {
    "id": "N9QSrxwwDNTx"
   },
   "source": [
    "#### Further experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f751e44",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 110
    },
    "id": "9f751e44",
    "outputId": "0da853c2-dc44-460e-911c-5279a2273774"
   },
   "outputs": [],
   "source": [
    "#Experiment 1 - with time in epoch\n",
    "\n",
    "# Convert timestamp into epoch format\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "# Below code was used when timestamp was in str format\n",
    "# df_train_copy['timestamp'] = df_train_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "# df_val_copy['timestamp'] = df_val_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "\n",
    "# Below code used when timestamp is in pandas datetime format, so we convert it to str, then convert to python datetime format, then convert to number of seconds\n",
    "df_train_copy['timestamp'] = df_train_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(str(x), '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "df_val_copy['timestamp'] = df_val_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(str(x), '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "\n",
    "\n",
    "features = ['timestamp', 't1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = LinearRegression()\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"linear\", \"experiment\": 1, \"desc\": \"epoch timestamp\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02aacd26",
   "metadata": {
    "id": "02aacd26"
   },
   "source": [
    "**Observations:** Using timestamp in epoch (number of seconds) format has increased the rmse very slightly (worsened the score), so this experiment did not help."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4916f78",
   "metadata": {
    "id": "b4916f78"
   },
   "source": [
    "Will check if scaling the timestamp and hum, wind_speed, t1 and t2 helps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6735343b",
   "metadata": {
    "id": "6735343b"
   },
   "outputs": [],
   "source": [
    "# Function to scale data using StandardScaler\n",
    "\n",
    "def pre_process_stdscale(df_train_to_process, df_val_to_process,scale_features):\n",
    "    std_scaler = preprocessing.StandardScaler()\n",
    "    # scale_features = ['timestamp', 't1', 't2', 'hum', 'wind_speed']\n",
    "    df_train_to_process[scale_features] = std_scaler.fit_transform(df_train_to_process[scale_features])\n",
    "    df_val_to_process[scale_features] = std_scaler.transform(df_val_to_process[scale_features])\n",
    "    \n",
    "    return df_train_to_process, df_val_to_process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59fce088",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 141
    },
    "id": "59fce088",
    "outputId": "91f7cefe-510b-4f3f-f56d-cd9516f287f9"
   },
   "outputs": [],
   "source": [
    "#Experiment 2 - with time in epoch, t1, t2, hum and wind_speed scaled using Standard Scaler\n",
    "\n",
    "# Convert timestamp into epoch format\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "# Below code was used when timestamp was in str format\n",
    "# df_train_copy['timestamp'] = df_train_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "# df_val_copy['timestamp'] = df_val_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "\n",
    "# Below code used when timestamp is in pandas datetime format, so we convert it to str, then convert to python datetime format, then convert to number of seconds\n",
    "df_train_copy['timestamp'] = df_train_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(str(x), '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "df_val_copy['timestamp'] = df_val_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(str(x), '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "\n",
    "features = ['timestamp', 't1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = LinearRegression()\n",
    "\n",
    "scale_features = ['timestamp', 't1', 't2', 'hum', 'wind_speed']\n",
    "\n",
    "# Preprocess by scaling using StandardScaler\n",
    "df_train_copy, df_val_copy = pre_process_stdscale(df_train_copy, df_val_copy,scale_features)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"linear\", \"experiment\": 2, \"desc\": \"epoch timestamp, scaled features\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae4176be",
   "metadata": {
    "id": "ae4176be"
   },
   "source": [
    "**Observations:** Scaling timestamp and other features has not helped at all and the score is exactly the same as without scaling."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44534eb",
   "metadata": {
    "id": "d44534eb"
   },
   "source": [
    "Will check if normalizing target with log transformation helps with better score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cbd1df3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 171
    },
    "id": "6cbd1df3",
    "outputId": "d9072504-04e9-496f-87e2-4b5c1ed49af4"
   },
   "outputs": [],
   "source": [
    "#Experiment 3 - Normalize target using log transformation\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "features = ['t1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = LinearRegression()\n",
    "\n",
    "# Normalize target feature\n",
    "y_train_copy = np.log1p(y_train_copy)\n",
    "y_val_copy = np.log1p(y_val_copy)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(np.expm1(y_val_copy), np.expm1(y_pred), np.expm1(y_train_copy), np.expm1(y_train_pred))\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"linear\", \"experiment\": 3, \"desc\": \"normalize target\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e987e6e7",
   "metadata": {
    "id": "e987e6e7"
   },
   "source": [
    "**Observations:** Log transform of target has not helped the score at all, infact rmse has increased."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "163c686d",
   "metadata": {
    "id": "163c686d"
   },
   "outputs": [],
   "source": [
    "# Use below code to drop certain experiment scores from the scoring dataframe to be able to rerun the experiment\n",
    "# exp_scores.drop(labels=[2],axis=0,inplace=True)\n",
    "# exp_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e131530",
   "metadata": {
    "id": "8e131530"
   },
   "source": [
    "##### Let us add more features by creating them from the timestamp feature.\n",
    "Will add the following features, because the bike count may relate to specific months, days, day of the week etc. Will then perform model evaluation and determine which of these features to keep and which to delete (if at all):\n",
    "* year\n",
    "* month\n",
    "* date\n",
    "* hour\n",
    "* minute\n",
    "* second\n",
    "* day-of-week\n",
    "* week-of-year\n",
    "* day-of-year"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcd1a5a8",
   "metadata": {
    "id": "dcd1a5a8"
   },
   "source": [
    "**Useful tips:**\n",
    "\n",
    "* Pandas Series **str.split** function seems useful in splitting. \n",
    "* There is also a variation - **str.rsplit** - which does a split from right-hand end. \n",
    "* The **expand=True** flag expands the list from the split into individual series (columns). \n",
    "* Also by specifying a number **n** we can define how many splits should happen (e.g. n=1 means split only once using the delimiter and keep rest of the part intact)\n",
    "\n",
    "https://pandas.pydata.org/docs/reference/api/pandas.Series.str.split.html\n",
    "\n",
    "* Pandas provides dt.dayofweek method to get the day of week from pandas datetime formated value\n",
    "* Pandas provides dt.weekofyear method to get the week of the year from pandas datetime formated value\n",
    "* There are other useful methods also like dt.day_name , dt.isocalendar().week, dt.dayofyear"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cc05ebb",
   "metadata": {
    "id": "2cc05ebb"
   },
   "source": [
    "First we will create different features one by one, analyse the usefulness and then update the pre_process function to handle this for a given dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75d7540c",
   "metadata": {
    "id": "75d7540c"
   },
   "outputs": [],
   "source": [
    "# We will create a copy of the original dataframe and perform processing on this to avoid changes to original dataframe\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b27f0072",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 597
    },
    "id": "b27f0072",
    "outputId": "08fe6c7b-8208-4f5e-ab72-61e52c30f5f8"
   },
   "outputs": [],
   "source": [
    "df_train_copy['year'] = df_train_copy['timestamp'].dt.year\n",
    "df_train_copy['month'] = df_train_copy['timestamp'].dt.month\n",
    "df_train_copy['day'] = df_train_copy['timestamp'].dt.day\n",
    "df_train_copy['hour'] = df_train_copy['timestamp'].dt.hour\n",
    "df_train_copy['minute'] = df_train_copy['timestamp'].dt.minute\n",
    "df_train_copy['second'] = df_train_copy['timestamp'].dt.second\n",
    "df_train_copy['day-of-week'] = pd.to_datetime(df_train_copy['timestamp']).dt.dayofweek.values\n",
    "df_train_copy['week-of-year'] = pd.to_datetime(df_train_copy['timestamp']).dt.isocalendar().week.values\n",
    "df_train_copy['day-of-year'] = pd.to_datetime(df_train_copy['timestamp']).dt.dayofyear\n",
    "df_train_copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c69bcb0b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 597
    },
    "id": "c69bcb0b",
    "outputId": "a3587e81-4362-4eae-9e68-875b75e83c6d"
   },
   "outputs": [],
   "source": [
    "df_val_copy['year'] = df_val_copy['timestamp'].dt.year\n",
    "df_val_copy['month'] = df_val_copy['timestamp'].dt.month\n",
    "df_val_copy['day'] = df_val_copy['timestamp'].dt.day\n",
    "df_val_copy['hour'] = df_val_copy['timestamp'].dt.hour\n",
    "df_val_copy['minute'] = df_val_copy['timestamp'].dt.minute\n",
    "df_val_copy['second'] = df_val_copy['timestamp'].dt.second\n",
    "df_val_copy['day-of-week'] = pd.to_datetime(df_val_copy['timestamp']).dt.dayofweek.values\n",
    "df_val_copy['week-of-year'] = pd.to_datetime(df_val_copy['timestamp']).dt.isocalendar().week.values\n",
    "df_val_copy['day-of-year'] = pd.to_datetime(df_val_copy['timestamp']).dt.dayofyear\n",
    "df_val_copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "403cf97a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "403cf97a",
    "outputId": "f8e1cab5-1de0-4454-aa37-9c20e663efa1"
   },
   "outputs": [],
   "source": [
    "df_train_copy.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10610dcd",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "10610dcd",
    "outputId": "fe886c16-5367-4ce6-993d-c6bd2b2bad58"
   },
   "outputs": [],
   "source": [
    "df_val_copy.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c521f8f",
   "metadata": {
    "id": "6c521f8f"
   },
   "source": [
    "**Observations:** The dtypes of the newly added features is int64. \n",
    "\n",
    "We can convert dtypes of the new features to save memory\n",
    "* The features 'day', 'month' and 'hour' cannot have values > 255 and will not be negative, hence lets convert to to uint8 dtype\n",
    "* Feature 'year' will not have negative values and values > 65535, hence lets convert to uint16 dtype\n",
    "* Feature day-of-week will not have negative values and values > 7, hence let us convert to uint8 dtype\n",
    "* Feature week-of-year will not have negative values and values > 52, hence let us convert to uint8 dtype\n",
    "* Feature day-of-year will not have negative values and values > 65535, hence lets convert to uint16 dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6ba499b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f6ba499b",
    "outputId": "659297e2-ea1d-42b3-d6fc-fe8ac234f66f"
   },
   "outputs": [],
   "source": [
    "df_train_copy['year'] = df_train_copy['year'].astype('uint16')\n",
    "df_train_copy['month'] = df_train_copy['month'].astype('uint8')\n",
    "df_train_copy['day'] = df_train_copy['day'].astype('uint8')\n",
    "df_train_copy['hour'] = df_train_copy['hour'].astype('uint8')\n",
    "df_train_copy['minute'] = df_train_copy['minute'].astype('uint8')\n",
    "df_train_copy['second'] = df_train_copy['second'].astype('uint8')\n",
    "df_train_copy['day-of-week'] = df_train_copy['day-of-week'].astype('uint8')\n",
    "df_train_copy['week-of-year'] = df_train_copy['week-of-year'].astype('uint8')\n",
    "df_train_copy['day-of-year'] = df_train_copy['day-of-year'].astype('uint16')\n",
    "df_train_copy.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18c93de3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "18c93de3",
    "outputId": "b235abb1-3d2f-4910-8e50-003871049a87"
   },
   "outputs": [],
   "source": [
    "df_val_copy['year'] = df_val_copy['year'].astype('uint16')\n",
    "df_val_copy['month'] = df_val_copy['month'].astype('uint8')\n",
    "df_val_copy['day'] = df_val_copy['day'].astype('uint8')\n",
    "df_val_copy['hour'] = df_val_copy['hour'].astype('uint8')\n",
    "df_val_copy['minute'] = df_val_copy['minute'].astype('uint8')\n",
    "df_val_copy['second'] = df_val_copy['second'].astype('uint8')\n",
    "df_val_copy['day-of-week'] = df_val_copy['day-of-week'].astype('uint8')\n",
    "df_val_copy['week-of-year'] = df_val_copy['week-of-year'].astype('uint8')\n",
    "df_val_copy['day-of-year'] = df_val_copy['day-of-year'].astype('uint16')\n",
    "df_val_copy.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbc7ae85",
   "metadata": {
    "id": "cbc7ae85"
   },
   "source": [
    "#### Now that we have created all the required features, let us check if minute and seconds are all 00 or they have other values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c792eb99",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c792eb99",
    "outputId": "52444522-d0a5-4ded-cec8-b46075b9e921"
   },
   "outputs": [],
   "source": [
    "# Let us check if minute and second have meaningful values (since possibly the data aggregates the count of bike rentals per hour)\n",
    "print(f\"number of distict values for second are : {df_train_copy['second'].nunique()}\")\n",
    "print(f\"number of distict values for minute are : {df_train_copy['minute'].nunique()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b14a06f9",
   "metadata": {
    "id": "b14a06f9"
   },
   "source": [
    "**Observations:** Since minutes and seconds are all 00, we do not need these features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6526cb3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 995
    },
    "id": "d6526cb3",
    "outputId": "ddc21fbd-7868-46cb-97b0-754d72905707"
   },
   "outputs": [],
   "source": [
    "del_cols = ['second', 'minute']\n",
    "for col in del_cols:\n",
    "    del df_train_copy[col]\n",
    "    del df_val_copy[col]\n",
    "display(df_train_copy)\n",
    "display(df_val_copy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a192026",
   "metadata": {
    "id": "0a192026"
   },
   "source": [
    "#### Let us check feature importance now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba4f1c79",
   "metadata": {
    "id": "ba4f1c79"
   },
   "outputs": [],
   "source": [
    "new_cat_features = ['weather_code', 'is_holiday', 'is_weekend', 'season', 'year', 'month', 'day', 'hour', 'day-of-week', 'week-of-year', 'day-of-year']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e944e78a",
   "metadata": {
    "id": "e944e78a"
   },
   "outputs": [],
   "source": [
    "def fn_mutual_info_score(series):\n",
    "    return mutual_info_score(series,y)\n",
    "\n",
    "def fn_norm_mutual_info_score(series):\n",
    "    return normalized_mutual_info_score(series,y)\n",
    "\n",
    "def fn_adj_mutual_info_score(series):\n",
    "    return adjusted_mutual_info_score(series,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "259ac9d0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 385
    },
    "id": "259ac9d0",
    "outputId": "78be820c-f9b8-48e1-9fbc-884ed253e47c"
   },
   "outputs": [],
   "source": [
    "y = y_train_copy\n",
    "mi = df_train_copy[new_cat_features].apply(fn_mutual_info_score)\n",
    "mi =pd.concat([mi,df_train_copy[new_cat_features].apply(fn_norm_mutual_info_score)],axis=1)\n",
    "mi =pd.concat([mi,df_train_copy[new_cat_features].apply(fn_adj_mutual_info_score)],axis=1)\n",
    "mi.columns = ['mi','norm_mi','adj_mi']\n",
    "mi.sort_values(by=['mi'],ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cd9ea98",
   "metadata": {
    "id": "8cd9ea98"
   },
   "source": [
    "**Observations:** We can see that almost all features are useful for model and related to target, except is_holiday. However we will still keep is_holiday."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf807fab",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "cf807fab",
    "outputId": "0b3f3ffa-6a18-4886-e441-66461ecbbf62"
   },
   "outputs": [],
   "source": [
    "## Let us check how bike share count is on holidays vs not holidays\n",
    "plt.figure(figsize = (12,4))\n",
    "sns.pointplot(x = df_train_copy['is_holiday'] , y = y_train_copy);\n",
    "plt.ylabel(\"cnt\", fontsize = 12)\n",
    "plt.xlabel(\"is_holiday\", fontsize = 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f207abd5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "f207abd5",
    "outputId": "6156ea00-b04f-4c0c-9a7f-e7c399751500"
   },
   "outputs": [],
   "source": [
    "## Let us check how bike share count is on weekends vs not weekends\n",
    "plt.figure(figsize = (12,4))\n",
    "sns.pointplot(x = df_train_copy['is_weekend'] , y = y_train_copy);\n",
    "plt.ylabel(\"cnt\", fontsize = 12)\n",
    "plt.xlabel(\"is_weekend\", fontsize = 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf8f91c1",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "bf8f91c1",
    "outputId": "aa7717b6-6a91-46e0-a9a2-12eb195e450f"
   },
   "outputs": [],
   "source": [
    "## Let us check how bike share count is on different days\n",
    "\n",
    "plt.figure(figsize = (12,4))\n",
    "sns.pointplot(x = df_train_copy['day'] , y = y_train_copy);\n",
    "plt.ylabel(\"cnt\", fontsize = 12)\n",
    "plt.xlabel(\"day\", fontsize = 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b1b7db7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "7b1b7db7",
    "outputId": "c7e30a6a-0f69-4e68-af7c-4a593a392a90"
   },
   "outputs": [],
   "source": [
    "## Let us check how bike share count is in different months\n",
    "\n",
    "plt.figure(figsize = (12,4))\n",
    "sns.pointplot(x = df_train_copy['month'] , y = y_train_copy);\n",
    "plt.ylabel(\"cnt\", fontsize = 12)\n",
    "plt.xlabel(\"month\", fontsize = 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e5d6199",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "0e5d6199",
    "outputId": "232d42a9-a404-4079-cc1f-d4d26396f1e0"
   },
   "outputs": [],
   "source": [
    "## Let us check how bike share count is on different days of the week\n",
    "\n",
    "plt.figure(figsize = (12,4))\n",
    "sns.pointplot(x = df_train_copy['day-of-week'] , y = y_train_copy);\n",
    "plt.ylabel(\"cnt\", fontsize = 12)\n",
    "plt.xlabel(\"day-of-week\", fontsize = 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbcd12e3",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "bbcd12e3",
    "outputId": "baf73011-849f-4369-b22c-27865c0ebb70"
   },
   "outputs": [],
   "source": [
    "## Let us check how bike share count is at different hours\n",
    "\n",
    "plt.figure(figsize = (12,4))\n",
    "sns.pointplot(x = df_train_copy['hour'] , y = y_train_copy);\n",
    "plt.ylabel(\"cnt\", fontsize = 12)\n",
    "plt.xlabel(\"Hour\", fontsize = 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a75ca776",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "a75ca776",
    "outputId": "8c0b1eea-42de-4a04-cbc1-62022dc48dab"
   },
   "outputs": [],
   "source": [
    "## Let us check how bike share count is in different seasons\n",
    "\n",
    "plt.figure(figsize = (12,4))\n",
    "sns.pointplot(x = df_train_copy['season'] , y = y_train_copy);\n",
    "plt.ylabel(\"cnt\", fontsize = 12)\n",
    "plt.xlabel(\"season\", fontsize = 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffd27c20",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "ffd27c20",
    "outputId": "bb180bea-c4aa-46de-e325-c5d361382123"
   },
   "outputs": [],
   "source": [
    "## Let us check how bike share count is in different weathers\n",
    "\n",
    "plt.figure(figsize = (12,4))\n",
    "sns.pointplot(x = df_train_copy['weather_code'] , y = y_train_copy);\n",
    "plt.ylabel(\"cnt\", fontsize = 12)\n",
    "plt.xlabel(\"weather_code\", fontsize = 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d406fb7b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "d406fb7b",
    "outputId": "beed66c9-cdeb-4f09-b9cc-e0c8477adac0"
   },
   "outputs": [],
   "source": [
    "## Let us check how bike share count is in different months\n",
    "\n",
    "plt.figure(figsize = (12,4))\n",
    "sns.pointplot(x = df_train_copy['month'] , y = y_train_copy);\n",
    "plt.ylabel(\"cnt\", fontsize = 12)\n",
    "plt.xlabel(\"month\", fontsize = 12)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60b02d27",
   "metadata": {
    "id": "60b02d27"
   },
   "source": [
    "#### Using Cyclical feature encoding for time related features\n",
    "* Using sine and cosine transformations of time related features like day, month, hour etc."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nx-GdIYMG3sJ",
   "metadata": {
    "id": "nx-GdIYMG3sJ"
   },
   "source": [
    "**Useful tip:**\n",
    "* When using datetime related features like day, month, week, hour, minute etc., these are cyclic in nature - e.g. after month 1 (Jan) through month 12 (Dec), we again have month 1.\n",
    "* Thus month 12 and month 1 have only 1 month difference and not 12.\n",
    "* To handle this cyclic nature, it may not be useful if we leave the month/hour/day etc. numbers as they are, since model may not understand the cyclic nature from this.\n",
    "* A suggested approach is to use both sine and cosine transformations to represent each of these features. More can be found out in the reference link below\n",
    "\n",
    "https://towardsdatascience.com/cyclical-features-encoding-its-about-time-ce23581845ca"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cebb7b89",
   "metadata": {
    "id": "cebb7b89"
   },
   "outputs": [],
   "source": [
    "def create_cyclical_features(df_to_process):\n",
    "    # We normalize x values to match with the 0-2π cycle\n",
    "    cols = df_to_process.columns\n",
    "    for col in cols:\n",
    "        df_to_process[f\"{col}_x_norm\"] = 2 * math.pi * df_to_process[col] / df_to_process[col].max()\n",
    "        df_to_process[f\"{col}_cos_x\"] = np.cos(df_to_process[f\"{col}_x_norm\"])\n",
    "        df_to_process[f\"{col}_sin_x\"] = np.sin(df_to_process[f\"{col}_x_norm\"])\n",
    "        del df_to_process[f\"{col}_x_norm\"]\n",
    "    return df_to_process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "244430bb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "244430bb",
    "outputId": "63c0b749-9d3a-4786-e201-0e79bccc859e"
   },
   "outputs": [],
   "source": [
    "cyclical_features = ['year', 'month', 'day', 'hour', 'day-of-week', 'week-of-year', 'day-of-year']\n",
    "\n",
    "df_train_to_process = df_train_copy[cyclical_features].copy()\n",
    "df_train_cyclical = create_cyclical_features(df_train_to_process)\n",
    "display(df_train_cyclical)\n",
    "\n",
    "df_val_to_process = df_val_copy[cyclical_features].copy()\n",
    "df_val_cyclical = create_cyclical_features(df_val_to_process)\n",
    "display(df_val_cyclical)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53d72b10",
   "metadata": {
    "id": "53d72b10"
   },
   "source": [
    "#### Let us check the correlation of the new numerial features (cyclical encoded features) with target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53dfcf84",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "53dfcf84",
    "outputId": "fe93c496-ff8c-4aa5-9c6c-69dd55eaf65a"
   },
   "outputs": [],
   "source": [
    "new_num_features = df_train_cyclical.columns\n",
    "df_train_cyclical[new_num_features].corrwith(y_train_copy).sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9dbbc83",
   "metadata": {
    "id": "f9dbbc83"
   },
   "source": [
    "#### Taking the absolute value of the correlation - since both positive and negative correlations but with higher values should be useful"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9df9129",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f9df9129",
    "outputId": "0e21e1bd-6370-41a4-bb68-a6907e6eb2ce"
   },
   "outputs": [],
   "source": [
    "new_num_features = df_train_cyclical.columns\n",
    "np.abs(df_train_cyclical[new_num_features].corrwith(y_train_copy)).sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77d20b68",
   "metadata": {
    "id": "77d20b68"
   },
   "source": [
    "**Observations:** We can see that hour has good correlation, followed by day-of-year, week-of-year, month. Other features have weak correlation but still some correlation is there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53e6dae6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "53e6dae6",
    "outputId": "80b64603-09e7-46d8-d313-87875bdefe2d"
   },
   "outputs": [],
   "source": [
    "len(df_train_cyclical.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e1cfbdd",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "4e1cfbdd",
    "outputId": "61d3c9ff-a209-494c-c4d4-acf05ba715a8"
   },
   "outputs": [],
   "source": [
    "for col in df_train_cyclical.columns:\n",
    "    df_train_copy[col] = df_train_cyclical[col]\n",
    "    df_val_copy[col] = df_val_cyclical[col]\n",
    "    \n",
    "display(df_train_copy)\n",
    "display(df_val_copy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea6ed5ba",
   "metadata": {
    "id": "ea6ed5ba"
   },
   "source": [
    "#### Further experiments\n",
    "Perform further experiments based on above analysis of new features and encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "350755ee",
   "metadata": {},
   "source": [
    "#### Common used function for rest of the experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "273d5bdb",
   "metadata": {
    "id": "273d5bdb"
   },
   "outputs": [],
   "source": [
    "# Function to perform pre processing on data before training\n",
    "# Combining all the step by step processing done above into a function\n",
    "\n",
    "\n",
    "# Function to now create different features from timestamp\n",
    "def pre_process_new_ft(df_to_process):\n",
    "\n",
    "    df_to_process['year'] = df_to_process['timestamp'].dt.year\n",
    "    df_to_process['month'] = df_to_process['timestamp'].dt.month\n",
    "    df_to_process['day'] = df_to_process['timestamp'].dt.day\n",
    "    df_to_process['hour'] = df_to_process['timestamp'].dt.hour\n",
    "    df_to_process['day-of-week'] = pd.to_datetime(df_to_process['timestamp']).dt.dayofweek.values\n",
    "    df_to_process['week-of-year'] = pd.to_datetime(df_to_process['timestamp']).dt.isocalendar().week.values\n",
    "    df_to_process['day-of-year'] = pd.to_datetime(df_to_process['timestamp']).dt.dayofyear\n",
    "\n",
    "\n",
    "    df_to_process['year'] = df_to_process['year'].astype('uint16')\n",
    "    df_to_process['month'] = df_to_process['month'].astype('uint8')\n",
    "    df_to_process['day'] = df_to_process['day'].astype('uint8')\n",
    "    df_to_process['hour'] = df_to_process['hour'].astype('uint8')\n",
    "    df_to_process['day-of-week'] = df_to_process['day-of-week'].astype('uint8')\n",
    "    df_to_process['week-of-year'] = df_to_process['week-of-year'].astype('uint8')\n",
    "    df_to_process['day-of-year'] = df_to_process['day-of-year'].astype('uint16')\n",
    "\n",
    "\n",
    "    # Create cyclical encoded features\n",
    "    cyclical_features = ['year', 'month', 'day', 'hour', 'day-of-week', 'week-of-year', 'day-of-year']\n",
    "    for col in cyclical_features:\n",
    "        df_to_process[f\"{col}_x_norm\"] = 2 * math.pi * df_to_process[col] / df_to_process[col].max()\n",
    "        df_to_process[f\"{col}_cos_x\"] = np.cos(df_to_process[f\"{col}_x_norm\"])\n",
    "        df_to_process[f\"{col}_sin_x\"] = np.sin(df_to_process[f\"{col}_x_norm\"])\n",
    "        del df_to_process[f\"{col}_x_norm\"]\n",
    "\n",
    "    return df_to_process\n",
    "\n",
    "\n",
    "# Function to encode the time features using cyclical encoding using sine and cosine. Drop original time features\n",
    "def pre_process_cyclic_encode(df_to_process,drop_org=True):\n",
    "    # Create cyclical encoded features\n",
    "    cyclical_features = ['year', 'month', 'day', 'hour', 'day-of-week', 'week-of-year', 'day-of-year']\n",
    "    for col in cyclical_features:\n",
    "        df_to_process[f\"{col}_x_norm\"] = 2 * math.pi * df_to_process[col] / df_to_process[col].max()\n",
    "        df_to_process[f\"{col}_cos_x\"] = np.cos(df_to_process[f\"{col}_x_norm\"])\n",
    "        df_to_process[f\"{col}_sin_x\"] = np.sin(df_to_process[f\"{col}_x_norm\"])\n",
    "        del df_to_process[f\"{col}_x_norm\"]\n",
    "\n",
    "    if drop_org:\n",
    "        for col in cyclical_features:\n",
    "            del df_to_process[col]\n",
    "            \n",
    "    return df_to_process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0eb89c4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "id": "a0eb89c4",
    "outputId": "62768998-a238-4da5-de6d-9a9ef5b076fc"
   },
   "outputs": [],
   "source": [
    "#Experiment 4 - Use additional created features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = LinearRegression()\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[new_features],df_val_copy[new_features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"linear\", \"experiment\": 4, \"desc\": \"new time features no encoding\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20f9af45",
   "metadata": {
    "id": "20f9af45"
   },
   "source": [
    "**Observations:** We can see that there is improvement in score when using the new created features."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bf407f7",
   "metadata": {
    "id": "2bf407f7"
   },
   "source": [
    "#### Now let us experiment by using cyclical encoding of newly created time features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63da65ea",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 232
    },
    "id": "63da65ea",
    "outputId": "e77bf376-3ffb-4f08-c4b1-c9da1c0f034e"
   },
   "outputs": [],
   "source": [
    "#Experiment 5 - Use additional created features with encoding, replacing the non encoded features\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = LinearRegression()\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# Encode time features with cyclic encoding - using sine and cosine\n",
    "df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[new_features],df_val_copy[new_features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"linear\", \"experiment\": 5, \"desc\": \"new cyclic encoded time features\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d52424fe",
   "metadata": {
    "id": "d52424fe"
   },
   "source": [
    "**Observations:** The score with using cyclic encoded new time features is almost same as without encoding. Ideally it should have helped."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f3fa440",
   "metadata": {
    "id": "2f3fa440"
   },
   "source": [
    "#### Let us see the coefficients of the linear regression model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5ce0df4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f5ce0df4",
    "outputId": "d0dbe840-c052-4ee6-d340-b75d8bcaaba4"
   },
   "outputs": [],
   "source": [
    "dict(zip(new_features, model.coef_))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "323ea6fe",
   "metadata": {
    "id": "323ea6fe"
   },
   "source": [
    "#### Let us check using one-hot encoding for the categorical features instead of the current numeric ordinal encoding.\n",
    "\n",
    "Since the dataset we have already has the categorical features being encoded, before we can use one-hot encoding, we will have to convert the categorical features to have labels and then one-hot encode these"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28fc5ebc",
   "metadata": {
    "id": "28fc5ebc"
   },
   "outputs": [],
   "source": [
    "# Function to convert the numerical encoded categorical features into corresponding labels\n",
    "# So that we can use it with DictVectorizer for one-hot encoding\n",
    "\n",
    "def set_cat_labels(df_to_process):\n",
    "    df_to_process['is_holiday'].replace([0,1],['no_hol','hol'],inplace=True)\n",
    "    df_to_process['is_weekend'].replace([0,1],['no_wknd','wknd'],inplace=True)\n",
    "    df_to_process['season'].replace([0,1,2,3],['spring','summer','fall','winter'],inplace=True)\n",
    "    df_to_process['weather_code'].replace([1,2,3,4,7,10,26,94],['clear','few_clouds','broken_clouds','cloudy','rain','thunderstorm','snowfall','freezing_fog'],inplace=True)\n",
    "\n",
    "    return df_to_process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f15a3e32",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 263
    },
    "id": "f15a3e32",
    "outputId": "f63f181d-8963-4741-ac2b-13db5a8325ee"
   },
   "outputs": [],
   "source": [
    "# Experiment 6 - one hot encoding of categorical features, new time features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = LinearRegression()\n",
    "\n",
    "# Convert the numerical encoded categorical features into corresponding labels\n",
    "df_train_copy = set_cat_labels(df_train_copy)\n",
    "df_val_copy = set_cat_labels(df_val_copy)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# # Encode time features with cyclic encoding - using sine and cosine\n",
    "# df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "# df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "dict_train_copy = df_train_copy.to_dict(orient='records')\n",
    "X_train_copy = dv.fit_transform(dict_train_copy)\n",
    "df_train_copy = pd.DataFrame(X_train_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "dict_val_copy = df_val_copy.to_dict(orient='records')\n",
    "X_val_copy = dv.transform(dict_val_copy)\n",
    "df_val_copy = pd.DataFrame(X_val_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy,df_val_copy,y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"linear\", \"experiment\": 6, \"desc\": \"one hot encoding cat with new time features no encoding\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a583f1b",
   "metadata": {
    "id": "8a583f1b"
   },
   "outputs": [],
   "source": [
    "# # Use below code to drop certain experiment scores from the scoring dataframe to be able to rerun the experiment\n",
    "# exp_scores.drop(labels=[7],axis=0,inplace=True)\n",
    "# exp_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "289ade67",
   "metadata": {
    "id": "289ade67"
   },
   "source": [
    "**Observations:** There is very tiny improvement or difference in score between one hot encoding or ordinal encoding of the categorical features."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "qAKGv9_ugy0f",
   "metadata": {
    "id": "qAKGv9_ugy0f"
   },
   "source": [
    "**Intermediate Observations Summary**: From the experiments so far, following is the summary:\n",
    "* Using StandardScaler to normalize the timestamp and the numerical features did not help (Hence, we will not be scaling in future experiments)\n",
    "* Normalizing the target ('cnt') using log transformation did not help (So, will use target as it is in future experiments)\n",
    "* Creating time features like day, month, hour etc. helped (so we will use these new features further)\n",
    "* Keeping the newly created time features as they are Vs cyclical encoding them with sine/cosine did not show any difference in results (So we will not perform cyclical scaling in further experiments)\n",
    "* Using One-hot encoding of categorical features instead of the ordinal encoding available as part of original dataset did not make any differece (So we will leave the features as they are in future experiments)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "uY2u6xRPZZn_",
   "metadata": {
    "id": "uY2u6xRPZZn_"
   },
   "source": [
    "#### Time series Cross-validation\n",
    "* So far the experiments were done using standard train_test_split and there was no cross-validation. In such scenarios, it is possible that by chance the data selection led to a good score. \n",
    "* Will perform a few experiments with cross-validation suitable for timeseries data. Found two options of cross-validation\n",
    "  * Scikit learn provides TimeSeriesSplit. Reference provided below.\n",
    "  * There is also a method called Blocked time series split. Have used the function for BlockingTimeSeriesSplit (with a small modification of mine) from reference provided below.\n",
    "* Will test with and without standardization of data (StandardScaler)\n",
    "* These experiments will be done using Lasso and Ridge regressions with different alpha values and max_iterations values to find the best hyper-parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "PyJ9cKq3Z6DG",
   "metadata": {
    "id": "PyJ9cKq3Z6DG"
   },
   "source": [
    "**Useful tip:**\n",
    "* When using cross validation for Time series data, we cannot use the typical way of KFold or similar cross validation, since it is important to maintain the sequence of time across train and val/test. Found two methods to achieve the data split for cross-validation for time series:\n",
    "  * Scikit learn provides TimeSeriesSplit for this - https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\n",
    "  *  BlockingTimeSeriesSplit is another method, and code available here - https://goldinlocks.github.io/Time-Series-Cross-Validation/#Blocked-and-Time-Series-Split-Cross-Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rWwGHtVUbHfd",
   "metadata": {
    "id": "rWwGHtVUbHfd"
   },
   "source": [
    "#### TimeSeriesSplit\n",
    "Cross-validation using TimeSeriesSplit method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "oyNpPMCFbMht",
   "metadata": {
    "id": "oyNpPMCFbMht"
   },
   "outputs": [],
   "source": [
    "# Experiment 7 \n",
    "# Linear using TimeSeriesSplit cross validation\n",
    "# one hot encoding of categorical features, new time features without encoding\n",
    "# with no scaling\n",
    "\n",
    "df_full_train_copy = df_full_train.copy()\n",
    "df_full_train_copy.reset_index(drop=True,inplace=True)\n",
    "y_full_train_copy = df_full_train_copy['cnt']\n",
    "del df_full_train_copy['cnt']\n",
    "\n",
    "n_splits = 5\n",
    "tscv = TimeSeriesSplit(n_splits=n_splits, test_size=len(df_full_train_copy)//5)\n",
    "\n",
    "t_scores = []\n",
    "iter_no = 1\n",
    "\n",
    "for train_index, val_index in tscv.split(df_full_train_copy):\n",
    "    # print(\"TRAIN:\", train_index, \"VAL:\", val_index)\n",
    "    df_split_train, df_split_val = df_full_train_copy.iloc[train_index], df_full_train_copy.iloc[val_index]\n",
    "    y_split_train, y_split_val = y_full_train_copy.iloc[train_index], y_full_train_copy.iloc[val_index]\n",
    "    \n",
    "    df_split_train.reset_index(drop=True,inplace=True)\n",
    "    df_split_val.reset_index(drop=True,inplace=True)\n",
    "    y_split_train.reset_index(drop=True,inplace=True)\n",
    "    y_split_val.reset_index(drop=True,inplace=True)\n",
    "\n",
    "    # Convert the numerical encoded categorical features into corresponding labels for one-hot encoding\n",
    "    df_split_train = set_cat_labels(df_split_train)\n",
    "    df_split_val = set_cat_labels(df_split_val)\n",
    "\n",
    "    # Preprocess by creating new time related features\n",
    "    df_split_train = pre_process_new_ft(df_split_train)\n",
    "    df_split_val = pre_process_new_ft(df_split_val)\n",
    "\n",
    "    # Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "    del df_split_train['timestamp']\n",
    "    del df_split_val['timestamp']\n",
    "\n",
    "    # One-hot encoding\n",
    "    dv = DictVectorizer(sparse=False)\n",
    "    dict_split_train = df_split_train.to_dict(orient='records')\n",
    "    X_split_train = dv.fit_transform(dict_split_train)\n",
    "    df_split_train = pd.DataFrame(X_split_train,columns=dv.get_feature_names_out())\n",
    "\n",
    "    dict_split_val = df_split_val.to_dict(orient='records')\n",
    "    X_split_val = dv.transform(dict_split_val)\n",
    "    df_split_val = pd.DataFrame(X_split_val,columns=dv.get_feature_names_out())\n",
    "\n",
    "    new_features = list(df_split_train.columns)\n",
    "\n",
    "    model = LinearRegression()\n",
    "\n",
    "    # Train and get predictions\n",
    "    y_pred, y_train_pred, model = train_predict(df_split_train,df_split_val,y_split_train,model)\n",
    "\n",
    "    # Score\n",
    "    iter_scores = evaluate_scores(y_split_val, y_pred, y_split_train, y_train_pred)\n",
    "\n",
    "    # Update scoring dataframe\n",
    "    score_entry = {\"algo\": \"linear\", \"experiment\": 7, \"iter\": iter_no, \"desc\": f\"timeseriesplit cv noscaling\"}\n",
    "    score_entry.update(iter_scores)\n",
    "    t_scores.append(score_entry)\n",
    "    iter_no = iter_no + 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "DMs6zXumdQVT",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "id": "DMs6zXumdQVT",
    "outputId": "4cb3ca44-cd35-4c8e-b7a8-c4d598844719"
   },
   "outputs": [],
   "source": [
    "# Will create a dataframe to store the scores from cross-validation\n",
    "s_df1 = pd.DataFrame(t_scores)\n",
    "display(s_df1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "260nVMq6e_st",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 110
    },
    "id": "260nVMq6e_st",
    "outputId": "3e266356-fe0e-4914-e513-46ad8ab22160"
   },
   "outputs": [],
   "source": [
    "# Aggregate scores from one experiment (i.e. same desc) and find average of the scores from the cross-validation\n",
    "s_df1_agg = s_df1.groupby(by='desc')[['val_r2', 'val_mse', 'val_rmse', 'val_mae', 'train_r2', 'train_mse', 'train_rmse', 'train_mae', 'diff']].agg('mean')\n",
    "s_df1_agg.sort_values(by='val_rmse',ascending=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "SESpjKatiYC5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SESpjKatiYC5",
    "outputId": "52303e86-3765-47b3-a67f-87eb0c23c59b"
   },
   "outputs": [],
   "source": [
    "# Prepare score_entry dict to then add the results to the scoring dataframe\n",
    "algo = s_df1.loc[0]['algo']\n",
    "experiment = s_df1.loc[0]['experiment']\n",
    "desc = s_df1.loc[0]['desc']\n",
    "score_entry = {\"algo\": algo, \"experiment\": experiment, \"desc\": desc}\n",
    "scores_dict = dict(s_df1_agg.loc['timeseriesplit cv noscaling',['val_r2', 'val_mse', 'val_rmse', 'val_mae', 'train_r2', 'train_mse', 'train_rmse', 'train_mae', 'diff']])\n",
    "score_entry.update(scores_dict)\n",
    "score_entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "N7mWh9apmWFG",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 294
    },
    "id": "N7mWh9apmWFG",
    "outputId": "060feb33-3fe7-4ef5-f481-99b32ed7d75b"
   },
   "outputs": [],
   "source": [
    "# Add score_entry to scoring dataframe\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "SEoQOJRJmydr",
   "metadata": {
    "id": "SEoQOJRJmydr"
   },
   "source": [
    "**Observations:** We can see that the scores we get from using the standard train_test_split and from TimeSeriesSplit cross-validation are similar, so we can possibly conclude that it is safe to use the train_test_split for future experiments. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fckovxHDag1g",
   "metadata": {
    "id": "fckovxHDag1g"
   },
   "source": [
    "#### BlockingTimeSeriesSplit\n",
    "Before concluding on use of train_test_split, let us also check the Cross-validation using BlockingTimeSeriesSplit method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "RqEEe2XNajog",
   "metadata": {
    "id": "RqEEe2XNajog"
   },
   "outputs": [],
   "source": [
    "# Modified the code a bit from https://goldinlocks.github.io/Time-Series-Cross-Validation/#Blocked-and-Time-Series-Split-Cross-Validation\n",
    "# To accept train_size and split data into train:test, instead of 50:50 in the code ine reference link\n",
    "\n",
    "class BlockingTimeSeriesSplit():\n",
    "    def __init__(self, n_splits, train_size):\n",
    "        self.n_splits = n_splits\n",
    "    \n",
    "    def get_n_splits(self, X, y, groups):\n",
    "        return self.n_splits\n",
    "    \n",
    "    def split(self, X, y=None, groups=None):\n",
    "        n_samples = len(X)\n",
    "        k_fold_size = n_samples // self.n_splits\n",
    "        indices = np.arange(n_samples)\n",
    "        margin = 0\n",
    "        for i in range(self.n_splits):\n",
    "            start = i * k_fold_size\n",
    "            stop = ((i+1) * k_fold_size)\n",
    "            split_point = start + int(k_fold_size * train_size)\n",
    "            yield indices[start: split_point], indices[split_point + margin: stop]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "IvFRFhrIa5oD",
   "metadata": {
    "id": "IvFRFhrIa5oD"
   },
   "outputs": [],
   "source": [
    "# Experiment 8 \n",
    "# Linear using BlockingTimeSeriesSplit cross validation\n",
    "# one hot encoding of categorical features, new time features without encoding\n",
    "# with no scaling\n",
    "\n",
    "df_full_train_copy = df_full_train.copy()\n",
    "df_full_train_copy.reset_index(drop=True,inplace=True)\n",
    "y_full_train_copy = df_full_train_copy['cnt']\n",
    "del df_full_train_copy['cnt']\n",
    "\n",
    "n_splits = 3\n",
    "train_size = 0.7\n",
    "btscv = BlockingTimeSeriesSplit(n_splits=n_splits,train_size=train_size)\n",
    "\n",
    "t_scores = []\n",
    "iter_no = 1\n",
    "\n",
    "for train_index, val_index in btscv.split(df_full_train_copy):\n",
    "    # print(\"TRAIN:\", train_index, \"VAL:\", val_index)\n",
    "    df_split_train, df_split_val = df_full_train_copy.iloc[train_index], df_full_train_copy.iloc[val_index]\n",
    "    y_split_train, y_split_val = y_full_train_copy.iloc[train_index], y_full_train_copy.iloc[val_index]\n",
    "    \n",
    "    df_split_train.reset_index(drop=True,inplace=True)\n",
    "    df_split_val.reset_index(drop=True,inplace=True)\n",
    "    y_split_train.reset_index(drop=True,inplace=True)\n",
    "    y_split_val.reset_index(drop=True,inplace=True)\n",
    "\n",
    "    # Convert the numerical encoded categorical features into corresponding labels\n",
    "    df_split_train = set_cat_labels(df_split_train)\n",
    "    df_split_val = set_cat_labels(df_split_val)\n",
    "\n",
    "    # Preprocess by creating new time related features\n",
    "    df_split_train = pre_process_new_ft(df_split_train)\n",
    "    df_split_val = pre_process_new_ft(df_split_val)\n",
    "\n",
    "    # Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "    del df_split_train['timestamp']\n",
    "    del df_split_val['timestamp']\n",
    "\n",
    "    dv = DictVectorizer(sparse=False)\n",
    "    dict_split_train = df_split_train.to_dict(orient='records')\n",
    "    X_split_train = dv.fit_transform(dict_split_train)\n",
    "    df_split_train = pd.DataFrame(X_split_train,columns=dv.get_feature_names_out())\n",
    "\n",
    "    dict_split_val = df_split_val.to_dict(orient='records')\n",
    "    X_split_val = dv.transform(dict_split_val)\n",
    "    df_split_val = pd.DataFrame(X_split_val,columns=dv.get_feature_names_out())\n",
    "\n",
    "    new_features = list(df_split_train.columns)\n",
    "\n",
    "    model = LinearRegression()\n",
    "\n",
    "    # Train and get predictions\n",
    "    y_pred, y_train_pred, model = train_predict(df_split_train,df_split_val,y_split_train,model)\n",
    "\n",
    "    # Score\n",
    "    iter_scores = evaluate_scores(y_split_val, y_pred, y_split_train, y_train_pred)\n",
    "\n",
    "    # Update scoring dataframe\n",
    "    score_entry = {\"algo\": \"linear\", \"experiment\": 8, \"iter\": iter_no, \"desc\": f\"blockingtime cv noscaling\"}\n",
    "    score_entry.update(iter_scores)\n",
    "    t_scores.append(score_entry)\n",
    "    iter_no = iter_no + 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Eie0NMA2bbuj",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 141
    },
    "id": "Eie0NMA2bbuj",
    "outputId": "aad83d15-5f1e-4d7a-cd08-1bcfe4246064"
   },
   "outputs": [],
   "source": [
    "# Will create a dataframe to store the scores\n",
    "s_df2 = pd.DataFrame(t_scores)\n",
    "s_df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bB8pHvlNbe2h",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 110
    },
    "id": "bB8pHvlNbe2h",
    "outputId": "1d3d570a-ddcf-4511-ed4b-9b99af95e0ba"
   },
   "outputs": [],
   "source": [
    "# Aggregate scores from one experiment (i.e. same desc) and find average of the scores from the cross-validation\n",
    "s_df2_agg = s_df2.groupby(by='desc')[['val_r2', 'val_mse', 'val_rmse', 'val_mae', 'train_r2', 'train_mse', 'train_rmse', 'train_mae', 'diff']].agg('mean')\n",
    "s_df2_agg.sort_values(by='val_rmse',ascending=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "myX_9118nfbR",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "myX_9118nfbR",
    "outputId": "c80c6a92-63c4-48be-f4a3-a431e1f24c6a"
   },
   "outputs": [],
   "source": [
    "# Prepare score_entry dict to then add the results to the scoring dataframe\n",
    "algo = s_df2.loc[0]['algo']\n",
    "experiment = s_df2.loc[0]['experiment']\n",
    "desc = s_df2.loc[0]['desc']\n",
    "score_entry = {\"algo\": algo, \"experiment\": experiment, \"desc\": desc}\n",
    "scores_dict = dict(s_df2_agg.loc['blockingtime cv noscaling',['val_r2', 'val_mse', 'val_rmse', 'val_mae', 'train_r2', 'train_mse', 'train_rmse', 'train_mae', 'diff']])\n",
    "score_entry.update(scores_dict)\n",
    "score_entry"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "i0ne5WatqnjK",
   "metadata": {
    "id": "i0ne5WatqnjK"
   },
   "source": [
    "**Observations:** \n",
    "* BlockingTimeSplit cross-validation method has given uncomprehensible results - did not understand why such results. \n",
    "* Will not add the results to scoring dataframe and will consider this experiment void\n",
    "* No point using this approach at all. So we will not be using this cross-validation approach, and our conclusion on suitability of train_test_split remains."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57c34448",
   "metadata": {
    "id": "57c34448"
   },
   "source": [
    "#### Cannot think of any more experiments, so let us try Ridge and Lasso regression with the best experiment (one hot of cat features and new time features without cyclical encoding using train_test_split)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61324423",
   "metadata": {
    "id": "61324423"
   },
   "source": [
    "**Useful tip:** Found a great article on Lasso and Ridge regression and their importance over Linear regression.\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2016/01/ridge-lasso-regression-python-complete-tutorial/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1eff71f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 324
    },
    "id": "b1eff71f",
    "outputId": "e959f385-fc66-425c-9f65-ef55a673c2f9"
   },
   "outputs": [],
   "source": [
    "# Experiment 8 (since previous experiment was considered void)\n",
    "# Ridge regression with one hot encoding of categorical features, new time features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = Ridge()\n",
    "\n",
    "# Convert the numerical encoded categorical features into corresponding labels\n",
    "df_train_copy = set_cat_labels(df_train_copy)\n",
    "df_val_copy = set_cat_labels(df_val_copy)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# # Encode time features with cyclic encoding - using sine and cosine\n",
    "# df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "# df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "dict_train_copy = df_train_copy.to_dict(orient='records')\n",
    "X_train_copy = dv.fit_transform(dict_train_copy)\n",
    "df_train_copy = pd.DataFrame(X_train_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "dict_val_copy = df_val_copy.to_dict(orient='records')\n",
    "X_val_copy = dv.transform(dict_val_copy)\n",
    "df_val_copy = pd.DataFrame(X_val_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy,df_val_copy,y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"ridge\", \"experiment\": 8, \"desc\": \"one hot encoding cat with new time features no encoding\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f72175c1",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 355
    },
    "id": "f72175c1",
    "outputId": "7d9d61b2-91ff-4670-e515-8cf261f647f3"
   },
   "outputs": [],
   "source": [
    "# Experiment 9 - Lasso regression with one hot encoding of categorical features, new time features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = Lasso()\n",
    "\n",
    "# Convert the numerical encoded categorical features into corresponding labels\n",
    "df_train_copy = set_cat_labels(df_train_copy)\n",
    "df_val_copy = set_cat_labels(df_val_copy)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# # Encode time features with cyclic encoding - using sine and cosine\n",
    "# df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "# df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "dict_train_copy = df_train_copy.to_dict(orient='records')\n",
    "X_train_copy = dv.fit_transform(dict_train_copy)\n",
    "df_train_copy = pd.DataFrame(X_train_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "dict_val_copy = df_val_copy.to_dict(orient='records')\n",
    "X_val_copy = dv.transform(dict_val_copy)\n",
    "df_val_copy = pd.DataFrame(X_val_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy,df_val_copy,y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"lasso\", \"experiment\": 9, \"desc\": \"one hot encoding cat with new time features no encoding\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "QSFVdH3Rrgks",
   "metadata": {
    "id": "QSFVdH3Rrgks"
   },
   "source": [
    "#### Hyper parameter tuning of Lasso and Ridge\n",
    "* Linear regression does not have any hyper parameters\n",
    "* Will try tuning Lasso and Ridge models with different alpha and max_iter values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8cbe585",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "id": "b8cbe585",
    "outputId": "11440ba8-74a9-4397-85b6-9495a5ef1aa8"
   },
   "outputs": [],
   "source": [
    "# Experiment 10\n",
    "# Lasso regression with diff alpha values and max_iter\n",
    "# with one hot encoding of categorical features, new time features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "# Convert the numerical encoded categorical features into corresponding labels\n",
    "df_train_copy = set_cat_labels(df_train_copy)\n",
    "df_val_copy = set_cat_labels(df_val_copy)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# # Encode time features with cyclic encoding - using sine and cosine\n",
    "# df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "# df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "dict_train_copy = df_train_copy.to_dict(orient='records')\n",
    "X_train_copy = dv.fit_transform(dict_train_copy)\n",
    "df_train_copy = pd.DataFrame(X_train_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "dict_val_copy = df_val_copy.to_dict(orient='records')\n",
    "X_val_copy = dv.transform(dict_val_copy)\n",
    "df_val_copy = pd.DataFrame(X_val_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "alpha_range = [1e-15, 1e-10, 1e-8, 1e-4, 1e-3,1e-2, 1, 5, 10, 20]\n",
    "max_iter_range = [10, 50, 100, 500, 1000, 2000, 3000, 4000]\n",
    "\n",
    "for alpha in alpha_range:\n",
    "    for max_iter in max_iter_range:\n",
    "        model = Lasso(alpha=alpha,max_iter=max_iter)\n",
    "\n",
    "        # Train and get predictions\n",
    "        y_pred, y_train_pred, model = train_predict(df_train_copy,df_val_copy,y_train_copy,model)\n",
    "\n",
    "        # Score\n",
    "        scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "        # Update scoring dataframe\n",
    "        score_entry = {\"algo\": \"lasso\", \"experiment\": 10, \"desc\": f\"hyper-parameters max_iter: {max_iter}, alpha: {alpha}\"}\n",
    "        score_entry.update(scores)\n",
    "        exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Cj3exNCAvzGQ",
   "metadata": {
    "id": "Cj3exNCAvzGQ"
   },
   "source": [
    "**Observations:** The score is better  with hyper-parameter optimization compared to baseline, but maybe only tiny bit from previous top scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32SxStWlLf9P",
   "metadata": {
    "id": "32SxStWlLf9P"
   },
   "source": [
    "#### Visualizing the train_rmse Vs val_rmse scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e82QxCiD_gM6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 560
    },
    "id": "e82QxCiD_gM6",
    "outputId": "88842257-5993-4d20-c8ff-31783ad6832c"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize = (8,6))\n",
    "# Scatterplot\n",
    "# Using Plotly graphs as they are interactive and we can see the values of the data points.\n",
    "# Below we plot the train_rmse Vs val_rmse and showing the index (row), \"desc\", \"algo\" on hovering on any point\n",
    "# (so that we know which entry these observations relate to)\n",
    "fig = px.scatter(exp_scores, x=\"val_rmse\", y=\"train_rmse\", hover_data=[\"desc\",\"algo\"], hover_name=exp_scores.index)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "HmGEC0LTOq7H",
   "metadata": {
    "id": "HmGEC0LTOq7H"
   },
   "source": [
    "**Observations:** As we can see most of the scores are concetrated around 934-945 range for val_rmse, where train_rmse ranges from 787-796. There seems to be some level of overfitting across all experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "jYjf0nK2LLRr",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 968
    },
    "id": "jYjf0nK2LLRr",
    "outputId": "0fedf1e5-ac0e-43ab-a295-e3c77e52b348"
   },
   "outputs": [],
   "source": [
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True).head(30))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "OC5x9W4vviQo",
   "metadata": {
    "id": "OC5x9W4vviQo"
   },
   "source": [
    "#### Let us compare the top scores from hyper-parameter tunning [index 33, 25, 17] with previous top 3 scores [indices and experiments 6, 8, 9] and baseline [index 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eK1872P-vFS9",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 263
    },
    "id": "eK1872P-vFS9",
    "outputId": "12764360-83fc-4655-8fd0-9e3a5b4a6b9c"
   },
   "outputs": [],
   "source": [
    "exp_scores.loc[[0, 6, 8, 9, 33, 25, 17]].sort_values(by=['val_rmse'],ascending=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "PKcFnlsHwLoD",
   "metadata": {
    "id": "PKcFnlsHwLoD"
   },
   "source": [
    "**Observations:** Cannot see any useful improvement with hyper-parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0QlhtuNjwTO9",
   "metadata": {
    "id": "0QlhtuNjwTO9"
   },
   "source": [
    "#### Will use Ridge regression with hyper-parameter tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7xPv95qpwSe9",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "id": "7xPv95qpwSe9",
    "outputId": "705b4693-4ab3-4a0c-9e4c-1ae09de3ec98"
   },
   "outputs": [],
   "source": [
    "# Experiment 10\n",
    "# Ridge regression with diff alpha values and max_iter\n",
    "# with one hot encoding of categorical features, new time features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "# Convert the numerical encoded categorical features into corresponding labels\n",
    "df_train_copy = set_cat_labels(df_train_copy)\n",
    "df_val_copy = set_cat_labels(df_val_copy)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# # Encode time features with cyclic encoding - using sine and cosine\n",
    "# df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "# df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "dict_train_copy = df_train_copy.to_dict(orient='records')\n",
    "X_train_copy = dv.fit_transform(dict_train_copy)\n",
    "df_train_copy = pd.DataFrame(X_train_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "dict_val_copy = df_val_copy.to_dict(orient='records')\n",
    "X_val_copy = dv.transform(dict_val_copy)\n",
    "df_val_copy = pd.DataFrame(X_val_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "alpha_range = [1e-15, 1e-10, 1e-8, 1e-4, 1e-3,1e-2, 1, 5, 10, 20]\n",
    "max_iter_range = [10, 50, 100, 500, 1000, 2000, 3000, 4000]\n",
    "\n",
    "for alpha in alpha_range:\n",
    "    for max_iter in max_iter_range:\n",
    "        model = Ridge(alpha=alpha,max_iter=max_iter)\n",
    "\n",
    "        # Train and get predictions\n",
    "        y_pred, y_train_pred, model = train_predict(df_train_copy,df_val_copy,y_train_copy,model)\n",
    "\n",
    "        # Score\n",
    "        scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "        # Update scoring dataframe\n",
    "        score_entry = {\"algo\": \"ridge\", \"experiment\": 11, \"desc\": f\"hyper-parameters max_iter: {max_iter}, alpha: {alpha}\"}\n",
    "        score_entry.update(scores)\n",
    "        exp_scores = exp_scores.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3972961",
   "metadata": {
    "id": "f3972961"
   },
   "source": [
    "**Observations:** The scores with hyper-parameter tuning for Ridge are not the top scores.\n",
    "\n",
    "Let us check the top 5 scores of Ridge with hyper-parameter tuning in experiment 11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1DS5v-ia7N5P",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "id": "1DS5v-ia7N5P",
    "outputId": "fe7c4577-61d2-4af4-d692-44c9bf1b98c6"
   },
   "outputs": [],
   "source": [
    "exp_scores.loc[np.where(exp_scores['experiment'] == 11)].sort_values(by=['val_rmse'],ascending=True).head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "-ppPvV0h7i9p",
   "metadata": {
    "id": "-ppPvV0h7i9p"
   },
   "source": [
    "Let us compare the top 3 scores of Ridge with previous top scores (indices 6, 8, 9, 33, 25, 17 of exp_scores dataframe) and baseline like we did after Lasso hyper-parameter tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5alB5m1f52vL",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 355
    },
    "id": "5alB5m1f52vL",
    "outputId": "a02c36c0-c8fd-46ca-db04-3de4817696ab"
   },
   "outputs": [],
   "source": [
    "# exp_scores.loc[np.where(exp_scores['experiment'] == 11)].sort_values(by=['val_rmse'],ascending=True).head(3).index.values\n",
    "exp_scores.loc[[0, 6, 8, 9, 33, 25, 17, 137, 136, 135]].sort_values(by=['val_rmse'],ascending=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "s2irdnpZ6XdU",
   "metadata": {
    "id": "s2irdnpZ6XdU"
   },
   "source": [
    "**Observations:** \n",
    "* We can see that with lower values of alpha there is very tiny improvement over baseline (practically may be not that useful), while with larger values of alpha the score worsens. \n",
    "* Results from Lasso are better than Ridge, even though extremely slightly.\n",
    "* Results of Lasso and Ridge are not much better than the top experiments with Linear regression (index 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "GLBsz8u3N0dX",
   "metadata": {
    "id": "GLBsz8u3N0dX"
   },
   "source": [
    "#### Will rerun experiments 6 (Linear), 8 (Ridge), 9 (Lasso) to check the coefficients of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8jkI6WvP-gTD",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8jkI6WvP-gTD",
    "outputId": "f060afa8-bbda-4998-dd93-8d9dda004460"
   },
   "outputs": [],
   "source": [
    "# Will run Experiment 6 (Linear) again and check the coefficients\n",
    "# one hot encoding of categorical features, new time features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = LinearRegression()\n",
    "\n",
    "# Convert the numerical encoded categorical features into corresponding labels\n",
    "df_train_copy = set_cat_labels(df_train_copy)\n",
    "df_val_copy = set_cat_labels(df_val_copy)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# # Encode time features with cyclic encoding - using sine and cosine\n",
    "# df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "# df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "dict_train_copy = df_train_copy.to_dict(orient='records')\n",
    "X_train_copy = dv.fit_transform(dict_train_copy)\n",
    "df_train_copy = pd.DataFrame(X_train_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "dict_val_copy = df_val_copy.to_dict(orient='records')\n",
    "X_val_copy = dv.transform(dict_val_copy)\n",
    "df_val_copy = pd.DataFrame(X_val_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy,df_val_copy,y_train_copy,model)\n",
    "\n",
    "dict(zip(new_features, model.coef_))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99XDcfhzMVlc",
   "metadata": {
    "id": "99XDcfhzMVlc"
   },
   "source": [
    "**Observations:** The coefficients for many features are very very high. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "LkvyPJ2NL_J5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "LkvyPJ2NL_J5",
    "outputId": "d8b13115-12c7-4c0a-a953-d3758b936acc"
   },
   "outputs": [],
   "source": [
    "# Will run Experiment 8 (Ridge) again and check the coefficients\n",
    "# one hot encoding of categorical features, new time features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = Ridge()\n",
    "\n",
    "# Convert the numerical encoded categorical features into corresponding labels\n",
    "df_train_copy = set_cat_labels(df_train_copy)\n",
    "df_val_copy = set_cat_labels(df_val_copy)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# # Encode time features with cyclic encoding - using sine and cosine\n",
    "# df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "# df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "dict_train_copy = df_train_copy.to_dict(orient='records')\n",
    "X_train_copy = dv.fit_transform(dict_train_copy)\n",
    "df_train_copy = pd.DataFrame(X_train_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "dict_val_copy = df_val_copy.to_dict(orient='records')\n",
    "X_val_copy = dv.transform(dict_val_copy)\n",
    "df_val_copy = pd.DataFrame(X_val_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy,df_val_copy,y_train_copy,model)\n",
    "\n",
    "dict(zip(new_features, model.coef_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7266d872",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7266d872",
    "outputId": "782c7229-c040-466e-d633-4f5536e46c35"
   },
   "outputs": [],
   "source": [
    "# Will run Experiment 9 (Lasso) again and check the coefficients\n",
    "# one hot encoding of categorical features, new time features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = Lasso()\n",
    "\n",
    "# Convert the numerical encoded categorical features into corresponding labels\n",
    "df_train_copy = set_cat_labels(df_train_copy)\n",
    "df_val_copy = set_cat_labels(df_val_copy)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# # Encode time features with cyclic encoding - using sine and cosine\n",
    "# df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "# df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "dict_train_copy = df_train_copy.to_dict(orient='records')\n",
    "X_train_copy = dv.fit_transform(dict_train_copy)\n",
    "df_train_copy = pd.DataFrame(X_train_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "dict_val_copy = df_val_copy.to_dict(orient='records')\n",
    "X_val_copy = dv.transform(dict_val_copy)\n",
    "df_val_copy = pd.DataFrame(X_val_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy,df_val_copy,y_train_copy,model)\n",
    "\n",
    "dict(zip(new_features, model.coef_))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "_8CMcVTxOFAr",
   "metadata": {
    "id": "_8CMcVTxOFAr"
   },
   "source": [
    "**Observations:** Coefficients with Lasso and Ridge are smaller values compared to Linear regression. Higher coefficients means the increased model complexity and that the model gives those features far too importance. This is my interpretation after reading the below reference link\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2016/01/ridge-lasso-regression-python-complete-tutorial/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "F9kPVf4zQ5DQ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 294
    },
    "id": "F9kPVf4zQ5DQ",
    "outputId": "3bbd7122-9e30-4836-cb01-26181f6340f7"
   },
   "outputs": [],
   "source": [
    "## Saving experiment scores to file\n",
    "display(exp_scores.describe())\n",
    "exp_scores.to_csv('linear-scores.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wdyCJkiCP3qd",
   "metadata": {
    "id": "wdyCJkiCP3qd"
   },
   "source": [
    "#### Summary of Observations for Linear Regression\n",
    "* Amongst all experiment, using one-hot encoding, new time features without cyclical encoding were having good results with Ridge or Lasso regression without any hyper-parameter tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vGi9yTrTQpoz",
   "metadata": {
    "id": "vGi9yTrTQpoz"
   },
   "outputs": [],
   "source": [
    "# # Below line to copy the saved scores file to Google Drive, since files stored on Colab get deleted on restart\n",
    "# !cp linear-scores.csv /content/drive/MyDrive/ml-zoomcamp-capstone/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a70ef5bf",
   "metadata": {
    "id": "a70ef5bf"
   },
   "source": [
    "<a id='decision-tree'></a>\n",
    "#### 5.2. Decision Tree\n",
    "[back to TOC](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "OuUeSnbcS8ra",
   "metadata": {
    "id": "OuUeSnbcS8ra"
   },
   "source": [
    "#### Will perform following experiments (from Linear regression) with DecisionTree\n",
    "* Baseline\n",
    "* Scaled features using StandardScaler\n",
    "* Normalize target with Log transformation\n",
    "* Create new time features and no cyclical encoding\n",
    "* New time features with cyclical encoding\n",
    "* One hot encoding of categorical features (that were orinal encoded in given dataset), and new time features without encoding\n",
    "* Using cross-validation with TimeSerieSplit method"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "H6ou2qvzWddD",
   "metadata": {
    "id": "H6ou2qvzWddD"
   },
   "source": [
    "#### Re-defining the scoring dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f45e4743",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 78
    },
    "id": "f45e4743",
    "outputId": "e6ac2b50-c71f-4273-8a2d-ed88eca1c641"
   },
   "outputs": [],
   "source": [
    "# #Save all scores into a pandas dataframe so that we can have a comparative study across experiments\n",
    "\n",
    "exp_columns = [\n",
    "    \"algo\", \n",
    "    \"experiment\",\n",
    "    \"desc\", \n",
    "    \"val_r2\", \n",
    "    \"val_mse\", \n",
    "    \"val_rmse\", \n",
    "    \"val_mae\", \n",
    "#     \"val_msle\", # Commented because msle cannot be used when target has negative values. And some preds are coming negative\n",
    "#     \"val_rmsle\", \n",
    "    \"train_r2\", \n",
    "    \"train_mse\",\n",
    "    \"train_rmse\", \n",
    "    \"train_mae\", \n",
    "#     \"train_msle\", \n",
    "#     \"train_rmsle\", \n",
    "    \"diff\"]\n",
    "exp_scores_dt = pd.DataFrame(columns = exp_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e654c8b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Saving the LinearRegression baseline score, if required after initializing notebook and populating the scoring dataframe\n",
    "\n",
    "get_from_previous_run = False\n",
    "\n",
    "dict_linear_baseline = {'algo': 'linear',\n",
    " 'desc': 'baseline',\n",
    " 'diff': 0,\n",
    " 'experiment': 0,\n",
    " 'train_mae': 657.71943,\n",
    " 'train_mse': 796550.98649,\n",
    " 'train_r2': 0.26809,\n",
    " 'train_rmse': 892.49705,\n",
    " 'val_mae': 820.14977,\n",
    " 'val_mse': 1122926.38592,\n",
    " 'val_r2': 0.27086,\n",
    " 'val_rmse': 1059.68221}\n",
    "\n",
    "# Insert the Linear regression baseline for reference\n",
    "if get_from_previous_run:\n",
    "    exp_scores_dt = exp_scores_dt.append(dict(exp_scores.loc[0,:]),ignore_index=True)\n",
    "    baseline_rmse = exp_scores_dt.loc[0]['val_rmse']\n",
    "    \n",
    "else:\n",
    "    exp_scores_dt = exp_scores_dt.append(dict_linear_baseline,ignore_index=True)\n",
    "    baseline_rmse = dict_linear_baseline['val_rmse']\n",
    "\n",
    "# Defining experiments as 'rest' and not the LinearRegression 'baseline'\n",
    "exp = 'rest'\n",
    "\n",
    "exp_scores_dt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54dbf1ed",
   "metadata": {},
   "source": [
    "#### DecisionTree baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c1b1653",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 110
    },
    "id": "5c1b1653",
    "outputId": "305ab043-ca9b-4fce-ea0f-31945544f00e"
   },
   "outputs": [],
   "source": [
    "#Experiment 12 - DecisonTree baseline\n",
    "\n",
    "# For every experiment we will use a copy of the train, validation dataframes (df_train, df_val) \n",
    "# and target series (y_train, y_val) to avoid any intermediate processing to affect subsequent experiments\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "features = ['t1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = DecisionTreeRegressor(max_depth=4)  # We will use max_depth=4, since default unlimited max_depth can result into overfitting\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"dt\", \"experiment\": 12, \"desc\": \"baseline\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores_dt = exp_scores_dt.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_dt.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "yujOzPh0YmAj",
   "metadata": {
    "id": "yujOzPh0YmAj"
   },
   "source": [
    "**Observations:** DecisionTree baseline is not very different than Linear Regression baseline model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4RlUgCK7YvzG",
   "metadata": {
    "id": "4RlUgCK7YvzG"
   },
   "source": [
    "#### Experiments using DecisionTree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "gMdFGFpUXa0P",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 141
    },
    "id": "gMdFGFpUXa0P",
    "outputId": "dc3a8a79-b529-4b8b-a7f7-88160b05285b"
   },
   "outputs": [],
   "source": [
    "#Experiment 13 - with time in epoch, t1, t2, hum and wind_speed scaled using Standard Scaler\n",
    "\n",
    "# Convert timestamp into epoch format\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "# Below code was used when timestamp was in str format\n",
    "# df_train_copy['timestamp'] = df_train_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "# df_val_copy['timestamp'] = df_val_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "\n",
    "# Below code used when timestamp is in pandas datetime format, so we convert it to str, then convert to python datetime format, then convert to number of seconds\n",
    "df_train_copy['timestamp'] = df_train_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(str(x), '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "df_val_copy['timestamp'] = df_val_copy['timestamp'].apply(lambda x: datetime.datetime.strptime(str(x), '%Y-%m-%d %H:%M:%S').strftime('%s'))\n",
    "\n",
    "features = ['timestamp', 't1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = DecisionTreeRegressor(max_depth=4)\n",
    "\n",
    "scale_features = ['timestamp', 't1', 't2', 'hum', 'wind_speed']\n",
    "\n",
    "# Preprocess by scaling using StandardScaler\n",
    "df_train_copy, df_val_copy = pre_process_stdscale(df_train_copy, df_val_copy,scale_features)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"dt\", \"experiment\": 13, \"desc\": \"epoch timestamp, scaled features\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores_dt = exp_scores_dt.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_dt.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "T0HO4RK8ZdF5",
   "metadata": {
    "id": "T0HO4RK8ZdF5"
   },
   "source": [
    "**Observations:** Sclaing features did not help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Xj-Urqf-ZZDR",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 171
    },
    "id": "Xj-Urqf-ZZDR",
    "outputId": "b2b9ceac-5cc8-4009-c6c7-accefc7e45a5"
   },
   "outputs": [],
   "source": [
    "#Experiment 14 - Normalize target using log transformation\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "features = ['t1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = DecisionTreeRegressor(max_depth=4)\n",
    "\n",
    "# Normalize target feature\n",
    "y_train_copy = np.log1p(y_train_copy)\n",
    "y_val_copy = np.log1p(y_val_copy)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(np.expm1(y_val_copy), np.expm1(y_pred), np.expm1(y_train_copy), np.expm1(y_train_pred))\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"dt\", \"experiment\": 14, \"desc\": \"normalize target\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores_dt = exp_scores_dt.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_dt.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8OJ5Yag5Zt_n",
   "metadata": {
    "id": "8OJ5Yag5Zt_n"
   },
   "source": [
    "**Observations:** Normalizing target did not help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "U7wf53S4ZsJI",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "id": "U7wf53S4ZsJI",
    "outputId": "239726d1-7020-40d3-b8ea-d522cbb70749"
   },
   "outputs": [],
   "source": [
    "#Experiment 15 - Use additional created features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = DecisionTreeRegressor(max_depth=4)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[new_features],df_val_copy[new_features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"dt\", \"experiment\": 15, \"desc\": \"new time features no encoding\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores_dt = exp_scores_dt.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_dt.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "RR8lKnAlaF8M",
   "metadata": {
    "id": "RR8lKnAlaF8M"
   },
   "source": [
    "**Observations:** New time features has improved the score very much and is even better than the experiments with LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vkmgI7SJaCKc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 232
    },
    "id": "vkmgI7SJaCKc",
    "outputId": "247fca5a-7262-4214-c7ac-8e0b4264dafd"
   },
   "outputs": [],
   "source": [
    "#Experiment 16 - Use additional created features with encoding, replacing the non encoded features\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = DecisionTreeRegressor(max_depth=4)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# Encode time features with cyclic encoding - using sine and cosine\n",
    "df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[new_features],df_val_copy[new_features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"dt\", \"experiment\": 16, \"desc\": \"new cyclic encoded time features\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores_dt = exp_scores_dt.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_dt.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "K2K3LGi4ajob",
   "metadata": {
    "id": "K2K3LGi4ajob"
   },
   "source": [
    "**Observations:** Encoding the newly created time features with cyclical encoding did not help compared to no encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "-h0UqpkRaf8C",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-h0UqpkRaf8C",
    "outputId": "1a1452d9-7c11-406b-cd35-9c1ede097060"
   },
   "outputs": [],
   "source": [
    "print(export_text(model,feature_names=new_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59ad6807",
   "metadata": {},
   "source": [
    "**Observations:** We can see that top features used for decision making by the algorithm are hour, temperature, day-of-year, is_weekend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Tk_1L9Q9a4H4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 263
    },
    "id": "Tk_1L9Q9a4H4",
    "outputId": "490e1f3c-de31-4219-b2b1-b65f18775073"
   },
   "outputs": [],
   "source": [
    "# Experiment 17 - one hot encoding of categorical features, new time features without encoding\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "model = DecisionTreeRegressor(max_depth=4)\n",
    "\n",
    "# Convert the numerical encoded categorical features into corresponding labels\n",
    "df_train_copy = set_cat_labels(df_train_copy)\n",
    "df_val_copy = set_cat_labels(df_val_copy)\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# # Encode time features with cyclic encoding - using sine and cosine\n",
    "# df_train_copy = pre_process_cyclic_encode(df_train_copy,drop_org=True)\n",
    "# df_val_copy = pre_process_cyclic_encode(df_val_copy,drop_org=True)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "dv = DictVectorizer(sparse=False)\n",
    "dict_train_copy = df_train_copy.to_dict(orient='records')\n",
    "X_train_copy = dv.fit_transform(dict_train_copy)\n",
    "df_train_copy = pd.DataFrame(X_train_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "dict_val_copy = df_val_copy.to_dict(orient='records')\n",
    "X_val_copy = dv.transform(dict_val_copy)\n",
    "df_val_copy = pd.DataFrame(X_val_copy,columns=dv.get_feature_names_out())\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy,df_val_copy,y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"dt\", \"experiment\": 17, \"desc\": \"one hot encoding cat with new time features no encoding\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores_dt = exp_scores_dt.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_dt.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "EqMcGLPLb3Ej",
   "metadata": {
    "id": "EqMcGLPLb3Ej"
   },
   "source": [
    "**Observations:** One-hot encoding score is same as using the ordinal encoding instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "To0276Tlbxep",
   "metadata": {
    "id": "To0276Tlbxep"
   },
   "outputs": [],
   "source": [
    "# Experiment 18\n",
    "# DT using TimeSeriesSplit cross validation\n",
    "# one hot encoding of categorical features, new time features without encoding\n",
    "# with no scaling\n",
    "\n",
    "df_full_train_copy = df_full_train.copy()\n",
    "df_full_train_copy.reset_index(drop=True,inplace=True)\n",
    "y_full_train_copy = df_full_train_copy['cnt']\n",
    "del df_full_train_copy['cnt']\n",
    "\n",
    "n_splits = 5\n",
    "tscv = TimeSeriesSplit(n_splits=n_splits, test_size=len(df_full_train_copy)//5)\n",
    "\n",
    "t_scores = []\n",
    "iter_no = 1\n",
    "\n",
    "for train_index, val_index in tscv.split(df_full_train_copy):\n",
    "    # print(\"TRAIN:\", train_index, \"VAL:\", val_index)\n",
    "    df_split_train, df_split_val = df_full_train_copy.iloc[train_index], df_full_train_copy.iloc[val_index]\n",
    "    y_split_train, y_split_val = y_full_train_copy.iloc[train_index], y_full_train_copy.iloc[val_index]\n",
    "    \n",
    "    df_split_train.reset_index(drop=True,inplace=True)\n",
    "    df_split_val.reset_index(drop=True,inplace=True)\n",
    "    y_split_train.reset_index(drop=True,inplace=True)\n",
    "    y_split_val.reset_index(drop=True,inplace=True)\n",
    "\n",
    "    # Convert the numerical encoded categorical features into corresponding labels for one-hot encoding\n",
    "    df_split_train = set_cat_labels(df_split_train)\n",
    "    df_split_val = set_cat_labels(df_split_val)\n",
    "\n",
    "    # Preprocess by creating new time related features\n",
    "    df_split_train = pre_process_new_ft(df_split_train)\n",
    "    df_split_val = pre_process_new_ft(df_split_val)\n",
    "\n",
    "    # Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "    del df_split_train['timestamp']\n",
    "    del df_split_val['timestamp']\n",
    "\n",
    "    # One-hot encoding\n",
    "    dv = DictVectorizer(sparse=False)\n",
    "    dict_split_train = df_split_train.to_dict(orient='records')\n",
    "    X_split_train = dv.fit_transform(dict_split_train)\n",
    "    df_split_train = pd.DataFrame(X_split_train,columns=dv.get_feature_names_out())\n",
    "\n",
    "    dict_split_val = df_split_val.to_dict(orient='records')\n",
    "    X_split_val = dv.transform(dict_split_val)\n",
    "    df_split_val = pd.DataFrame(X_split_val,columns=dv.get_feature_names_out())\n",
    "\n",
    "    new_features = list(df_split_train.columns)\n",
    "\n",
    "    model = DecisionTreeRegressor(max_depth=4)\n",
    "\n",
    "    # Train and get predictions\n",
    "    y_pred, y_train_pred, model = train_predict(df_split_train,df_split_val,y_split_train,model)\n",
    "\n",
    "    # Score\n",
    "    iter_scores = evaluate_scores(y_split_val, y_pred, y_split_train, y_train_pred)\n",
    "\n",
    "    # Update scoring dataframe\n",
    "    score_entry = {\"algo\": \"dt\", \"experiment\": 18, \"iter\": iter_no, \"desc\": f\"timeseriesplit cv noscaling\"}\n",
    "    score_entry.update(iter_scores)\n",
    "    t_scores.append(score_entry)\n",
    "    iter_no = iter_no + 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "yt4H3PmfcPRr",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "id": "yt4H3PmfcPRr",
    "outputId": "b70277f3-ecc0-45e4-9ea3-349478634e6f"
   },
   "outputs": [],
   "source": [
    "# Will create a dataframe to store the scores from cross-validation\n",
    "s_df3 = pd.DataFrame(t_scores)\n",
    "display(s_df3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "y1GDoCmGcdEZ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 110
    },
    "id": "y1GDoCmGcdEZ",
    "outputId": "6a5e83a9-f762-4b71-934f-b5abd5bc1e16"
   },
   "outputs": [],
   "source": [
    "# Aggregate scores from one experiment (i.e. same desc) and find average of the scores from the cross-validation\n",
    "s_df3_agg = s_df3.groupby(by='desc')[['val_r2', 'val_mse', 'val_rmse', 'val_mae', 'train_r2', 'train_mse', 'train_rmse', 'train_mae', 'diff']].agg('mean')\n",
    "s_df3_agg.sort_values(by='val_rmse',ascending=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "CB2QiUfccjs7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CB2QiUfccjs7",
    "outputId": "a1268dac-cf3f-4b6b-b812-7538397fe930"
   },
   "outputs": [],
   "source": [
    "# Prepare score_entry dict to then add the results to the scoring dataframe\n",
    "algo = s_df3.loc[0]['algo']\n",
    "experiment = s_df3.loc[0]['experiment']\n",
    "desc = s_df3.loc[0]['desc']\n",
    "score_entry = {\"algo\": algo, \"experiment\": experiment, \"desc\": desc}\n",
    "scores_dict = dict(s_df3_agg.loc['timeseriesplit cv noscaling',['val_r2', 'val_mse', 'val_rmse', 'val_mae', 'train_r2', 'train_mse', 'train_rmse', 'train_mae', 'diff']])\n",
    "score_entry.update(scores_dict)\n",
    "score_entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "xqr1mDR3cxOT",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 294
    },
    "id": "xqr1mDR3cxOT",
    "outputId": "bca2d1ac-f09c-4b37-a7ee-61945483da35"
   },
   "outputs": [],
   "source": [
    "# Add score_entry to scoring dataframe\n",
    "exp_scores_dt = exp_scores_dt.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_dt.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "kkU4l7ONc8-F",
   "metadata": {
    "id": "kkU4l7ONc8-F"
   },
   "source": [
    "**Observations:** Using TimeSeriesSplit cross-validation scores worsened and there was further overfitting, so will use the train_test_split method only"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eueBiRwdWhq",
   "metadata": {
    "id": "1eueBiRwdWhq"
   },
   "source": [
    "#### Will perform hyper-parameter tuning now on the DecisionTree Algorithm using the top experiment (new time features with no encoding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "LWkZt4rAc3Nz",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "id": "LWkZt4rAc3Nz",
    "outputId": "3a73fdd3-2555-4131-cf59-10ee52406862"
   },
   "outputs": [],
   "source": [
    "#Experiment 19\n",
    "\n",
    "# Hyper-parameter tuning DT\n",
    "max_depths = [1, 2, 3, 4, 5, 6, 10, 15, 20, 50]\n",
    "min_samples_leafs = [1, 2, 5, 10, 15, 20, 100, 200, 500]\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "for max_depth in max_depths:\n",
    "  for min_samples_leaf in min_samples_leafs:\n",
    "    model = DecisionTreeRegressor(max_depth=max_depth,min_samples_leaf=min_samples_leaf)\n",
    "    # Train and get predictions\n",
    "    y_pred, y_train_pred, model = train_predict(df_train_copy[new_features],df_val_copy[new_features],y_train_copy,model)\n",
    "\n",
    "    # Score\n",
    "    scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "    # Update scoring dataframe\n",
    "    score_entry = {\"algo\": \"dt\", \"experiment\": 19, \"desc\": f\"hyper-params max_depth: {max_depth}, min_samples_leaf: {min_samples_leaf} \"}\n",
    "    score_entry.update(scores)\n",
    "    exp_scores_dt = exp_scores_dt.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_dt.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rMc9e9kYhEzM",
   "metadata": {
    "id": "rMc9e9kYhEzM"
   },
   "source": [
    "**Observations:** There was significant improvement with tuned hyper-parameters. Best score of val_rmse 399.5 was achieved with max_depth: 10, min_samples_leaf: 5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6q_9gsiohzan",
   "metadata": {
    "id": "6q_9gsiohzan"
   },
   "source": [
    "#### Let us visualize the train_rmse Vs val_rmse for different hyper-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "JI4mW2LVg_gU",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 560
    },
    "id": "JI4mW2LVg_gU",
    "outputId": "ff9b1dc3-4fd0-4880-bb0d-42dded74c35f"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize = (8,6))\n",
    "# Scatterplot\n",
    "# Using Plotly graphs as they are interactive and we can see the values of the data points.\n",
    "# Below we plot the train_rmse Vs val_rmse and showing the index (row), \"desc\", \"algo\" on hovering on any point\n",
    "# (so that we know which entry these observations relate to)\n",
    "fig = px.scatter(exp_scores_dt, x=\"val_rmse\", y=\"train_rmse\", hover_data=[\"desc\",\"algo\"], hover_name=exp_scores_dt.index)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "VIMji5_Yi2rT",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 294
    },
    "id": "VIMji5_Yi2rT",
    "outputId": "32d238dc-5a2d-411d-9b05-0dbd3b3c2fe1"
   },
   "outputs": [],
   "source": [
    "## Saving experiment scores to file\n",
    "display(exp_scores_dt.describe())\n",
    "exp_scores_dt.to_csv('decision-tree-scores.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "TGSkYCafjFnY",
   "metadata": {
    "id": "TGSkYCafjFnY"
   },
   "outputs": [],
   "source": [
    "# # Below line to copy the saved scores file to Google Drive, since files stored on Colab get deleted on restart\n",
    "# !cp decision-tree-scores.csv /content/drive/MyDrive/ml-zoomcamp-capstone/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "xFKqOviJiA1Z",
   "metadata": {
    "id": "xFKqOviJiA1Z"
   },
   "source": [
    "<a id='random-forest'></a>\n",
    "#### 5.3. Random Forest\n",
    "[back to TOC](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "NqDN5ywciDr0",
   "metadata": {
    "id": "NqDN5ywciDr0"
   },
   "source": [
    "#### Will perform following experiments with RandomForest\n",
    "* Baseline\n",
    "* Hyper-parameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "HxM4Dlzvjknh",
   "metadata": {
    "id": "HxM4Dlzvjknh"
   },
   "source": [
    "#### Redefining scoring dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Zu0A3LAojebr",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 78
    },
    "id": "Zu0A3LAojebr",
    "outputId": "f549e534-7564-476c-c3c2-41d754c5f96f"
   },
   "outputs": [],
   "source": [
    "# Defining experiments as 'rest' and not the LinearRegression 'baseline'\n",
    "exp = 'rest'\n",
    "\n",
    "# #Save all scores into a pandas dataframe so that we can have a comparative study across experiments\n",
    "\n",
    "exp_columns = [\n",
    "    \"algo\", \n",
    "    \"experiment\",\n",
    "    \"desc\", \n",
    "    \"val_r2\", \n",
    "    \"val_mse\", \n",
    "    \"val_rmse\", \n",
    "    \"val_mae\", \n",
    "#     \"val_msle\", # Commented because msle cannot be used when target has negative values. And some preds are coming negative\n",
    "#     \"val_rmsle\", \n",
    "    \"train_r2\", \n",
    "    \"train_mse\",\n",
    "    \"train_rmse\", \n",
    "    \"train_mae\", \n",
    "#     \"train_msle\", \n",
    "#     \"train_rmsle\", \n",
    "    \"diff\"]\n",
    "exp_scores_rf = pd.DataFrame(columns = exp_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59f94be0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Saving the LinearRegression baseline score, if required after initializing notebook and populating the scoring dataframe\n",
    "\n",
    "get_from_previous_run = False\n",
    "\n",
    "dict_linear_baseline = {'algo': 'linear',\n",
    " 'desc': 'baseline',\n",
    " 'diff': 0,\n",
    " 'experiment': 0,\n",
    " 'train_mae': 657.71943,\n",
    " 'train_mse': 796550.98649,\n",
    " 'train_r2': 0.26809,\n",
    " 'train_rmse': 892.49705,\n",
    " 'val_mae': 820.14977,\n",
    " 'val_mse': 1122926.38592,\n",
    " 'val_r2': 0.27086,\n",
    " 'val_rmse': 1059.68221}\n",
    "\n",
    "# Insert the Linear regression baseline for reference\n",
    "if get_from_previous_run:\n",
    "    exp_scores_rf = exp_scores_rf.append(dict(exp_scores.loc[0,:]),ignore_index=True)\n",
    "    baseline_rmse = exp_scores_rf.loc[0]['val_rmse']\n",
    "    \n",
    "else:\n",
    "    exp_scores_rf = exp_scores_rf.append(dict_linear_baseline,ignore_index=True)\n",
    "    baseline_rmse = dict_linear_baseline['val_rmse']\n",
    "\n",
    "# Defining experiments as 'rest' and not the LinearRegression 'baseline'\n",
    "exp = 'rest'\n",
    "\n",
    "exp_scores_rf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "URr7MkwCihkU",
   "metadata": {
    "id": "URr7MkwCihkU"
   },
   "source": [
    "#### Baseline with RandomForest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2K0ZiJf5j8U4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 110
    },
    "id": "2K0ZiJf5j8U4",
    "outputId": "b8d6f969-2378-47e8-f5aa-ecebf2462fff"
   },
   "outputs": [],
   "source": [
    "#Experiment 20 - RandomForest baseline\n",
    "\n",
    "# For every experiment we will use a copy of the train, validation dataframes (df_train, df_val) \n",
    "# and target series (y_train, y_val) to avoid any intermediate processing to affect subsequent experiments\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "features = ['t1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = RandomForestRegressor(random_state=42, n_jobs=-1)  # We will use max_depth=4, since default unlimited max_depth can result into overfitting\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"rf\", \"experiment\": 20, \"desc\": \"baseline\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores_rf = exp_scores_rf.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_rf.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b7c2e97",
   "metadata": {},
   "source": [
    "#### Hyper-parameter tuning with RandomForest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "n6WMdiankaia",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "id": "n6WMdiankaia",
    "outputId": "3e19dc93-fdbc-43c3-ec2b-5b686fdc5edf"
   },
   "outputs": [],
   "source": [
    "#Experiment 21\n",
    "\n",
    "# Hyper-parameter tuning RF\n",
    "max_depths = [1, 2, 3, 4, 5, 6, 10, 15, 20, 50]\n",
    "min_samples_leafs = [1, 2, 5, 10, 15, 20, 100, 200, 500]\n",
    "n_estimators_range = range(10, 200, 10)\n",
    "\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "for max_depth in max_depths:\n",
    "  for min_samples_leaf in min_samples_leafs:\n",
    "    for n_estimators in n_estimators_range:\n",
    "      model = RandomForestRegressor(random_state=42, n_jobs=-1, warm_start=True, n_estimators=n_estimators, max_depth=max_depth, min_samples_leaf=min_samples_leaf)\n",
    "      # Train and get predictions\n",
    "      y_pred, y_train_pred, model = train_predict(df_train_copy[new_features],df_val_copy[new_features],y_train_copy,model)\n",
    "\n",
    "      # Score\n",
    "      scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "      # Update scoring dataframe\n",
    "      score_entry = {\"algo\": \"rf\", \"experiment\": 21, \"desc\": f\"hyper-params n_estimators: {n_estimators} max_depth: {max_depth}, min_samples_leaf: {min_samples_leaf} \"}\n",
    "      score_entry.update(scores)\n",
    "      exp_scores_rf = exp_scores_rf.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_rf.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "iAH2TP5R_r6q",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iAH2TP5R_r6q",
    "outputId": "b40ab0b6-5325-4d61-f3bd-a489bd6a21e4"
   },
   "outputs": [],
   "source": [
    "exp_scores_rf.sort_values(by=['val_rmse'],ascending=True).head(2)['desc'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "y3q0XzHcmMuN",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 294
    },
    "id": "y3q0XzHcmMuN",
    "outputId": "defea600-6049-4ce0-bc8d-1c2c59b11a3d"
   },
   "outputs": [],
   "source": [
    "## Saving experiment scores to file\n",
    "display(exp_scores_rf.describe())\n",
    "exp_scores_rf.to_csv('random-forest-scores.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Kze2YfbDtONR",
   "metadata": {
    "id": "Kze2YfbDtONR"
   },
   "outputs": [],
   "source": [
    "# # Below line to copy the saved scores file to Google Drive, since files stored on Colab get deleted on restart\n",
    "# !cp random-forest-scores.csv /content/drive/MyDrive/ml-zoomcamp-capstone/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "xiF-1Hh0_23u",
   "metadata": {
    "id": "xiF-1Hh0_23u"
   },
   "source": [
    "**Observations:** Scores with Random Forest are good and better than DecisionTree.\n",
    "The best score with RadomForest was val_rmse of 349.7. This was achived with hyper-parameters n_estimators: 50 max_depth: 50, min_samples_leaf: 2 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6n3leCuAmal",
   "metadata": {
    "id": "c6n3leCuAmal"
   },
   "source": [
    "<a id='xgb'></a>\n",
    "#### 5.4. XGBoost\n",
    "[back to TOC](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Pl3S4643Axps",
   "metadata": {
    "id": "Pl3S4643Axps"
   },
   "source": [
    "#### Will perform following experiments with XGBoost\n",
    "* Baseline\n",
    "* Outlier removal from target\n",
    "* Hyper-parameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "EcmvfpecA1zi",
   "metadata": {
    "id": "EcmvfpecA1zi"
   },
   "source": [
    "#### Redefining scoring dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "V54Jyans_FNG",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 78
    },
    "id": "V54Jyans_FNG",
    "outputId": "7865512e-09f8-45b5-c2ea-a33f81da5345"
   },
   "outputs": [],
   "source": [
    "# #Save all scores into a pandas dataframe so that we can have a comparative study across experiments\n",
    "\n",
    "exp_columns = [\n",
    "    \"algo\", \n",
    "    \"experiment\",\n",
    "    \"desc\", \n",
    "    \"val_r2\", \n",
    "    \"val_mse\", \n",
    "    \"val_rmse\", \n",
    "    \"val_mae\", \n",
    "#     \"val_msle\", # Commented because msle cannot be used when target has negative values. And some preds are coming negative\n",
    "#     \"val_rmsle\", \n",
    "    \"train_r2\", \n",
    "    \"train_mse\",\n",
    "    \"train_rmse\", \n",
    "    \"train_mae\", \n",
    "#     \"train_msle\", \n",
    "#     \"train_rmsle\", \n",
    "    \"diff\"]\n",
    "exp_scores_xgb = pd.DataFrame(columns = exp_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "p8bkleBtKS07",
   "metadata": {
    "id": "p8bkleBtKS07"
   },
   "outputs": [],
   "source": [
    "# # Saving the LinearRegression baseline score, if required after initializing notebook and populating the scoring dataframe\n",
    "\n",
    "get_from_previous_run = False\n",
    "\n",
    "dict_linear_baseline = {'algo': 'linear',\n",
    " 'desc': 'baseline',\n",
    " 'diff': 0,\n",
    " 'experiment': 0,\n",
    " 'train_mae': 657.71943,\n",
    " 'train_mse': 796550.98649,\n",
    " 'train_r2': 0.26809,\n",
    " 'train_rmse': 892.49705,\n",
    " 'val_mae': 820.14977,\n",
    " 'val_mse': 1122926.38592,\n",
    " 'val_r2': 0.27086,\n",
    " 'val_rmse': 1059.68221}\n",
    "\n",
    "# Insert the Linear regression baseline for reference\n",
    "if get_from_previous_run:\n",
    "    exp_scores_xgb = exp_scores_xgb.append(dict(exp_scores.loc[0,:]),ignore_index=True)\n",
    "    baseline_rmse = exp_scores_xgb.loc[0]['val_rmse']\n",
    "    \n",
    "else:\n",
    "    exp_scores_xgb = exp_scores_xgb.append(dict_linear_baseline,ignore_index=True)\n",
    "    baseline_rmse = dict_linear_baseline['val_rmse']\n",
    "\n",
    "# Defining experiments as 'rest' and not the LinearRegression 'baseline'\n",
    "exp = 'rest'\n",
    "\n",
    "exp_scores_xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1IwlC8_kBG1R",
   "metadata": {
    "id": "1IwlC8_kBG1R"
   },
   "source": [
    "#### Baseline with XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "BujKhEGjA83L",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 127
    },
    "id": "BujKhEGjA83L",
    "outputId": "1566e48c-4889-43b9-b799-05a8deff8066"
   },
   "outputs": [],
   "source": [
    "#Experiment 22 - XGBoost baseline\n",
    "\n",
    "# For every experiment we will use a copy of the train, validation dataframes (df_train, df_val) \n",
    "# and target series (y_train, y_val) to avoid any intermediate processing to affect subsequent experiments\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "features = ['t1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = xgb.XGBRegressor(random_state=42, n_jobs=-1, objective=\"reg:squarederror\")\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"xgb\", \"experiment\": 22, \"desc\": \"baseline\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores_xgb = exp_scores_xgb.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_xgb.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "Vd5VcUnzFXzu",
   "metadata": {
    "id": "Vd5VcUnzFXzu"
   },
   "source": [
    "**Observations:** Baseline of XGBoost is only little bit better than LinearRegression baseline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "MaLEu4ikQVTd",
   "metadata": {
    "id": "MaLEu4ikQVTd"
   },
   "source": [
    "#### Outlier removal on target\n",
    "\n",
    "Outlier is a data point which differs significantly from other observations. There are various techniques that can be used to define outliers (99 percentile, 1.5*IQR (inter quartile range) etc.) and process them (remove, cap them etc.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e2f1d4",
   "metadata": {},
   "source": [
    "**Useful tip:** Some important points with regards to Outliers that I learned from the discussion on Slack and guidance from Alexey.\n",
    "\n",
    "**Important points about Outliers**\n",
    "* Derive rules from train data and apply on train data (for outlier detection and handling)\n",
    "* If processing outliers on target, do not perform same processing on val/test data, simply train the processed training data and evaluate on validation data\n",
    "* If processing outliers on features, test different strategies for dealing with outliers- capping big values, removing outliers altogether etc. on train and validation data both and check if it actually helping\n",
    "* If outliers are due to mistake in input data, you can correct it\n",
    "* Save / make a note of the rules you have deviced for outlier handling. You will need these on final model training and evaluation on test data. You will also need these when you deploy model in production"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "SOR5_hgsLDnp",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 158
    },
    "id": "SOR5_hgsLDnp",
    "outputId": "7a4ea3fc-5e1b-47f5-a4c0-92391a5e3eb7"
   },
   "outputs": [],
   "source": [
    "# Experiment 23 - Remove outliers on target 'cnt'\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "# For Outliers typically the IQR (inter quantile range) is the range between 25th percentile (Q1 - quartile 1) and 75th percentile (Q3 - quartile 3)\n",
    "# Thus IQR = Q3 - Q1. Typically outliers are the values outside a calculated outlier range\n",
    "# The lower bound of the range is calculated as (Q1 - 1.5 * IQR)\n",
    "# The upper bound of the range is calculated as (Q3 + 1.5 * IQR)\n",
    "q3 = y_train_copy.quantile(0.75)\n",
    "q1 = y_train_copy.quantile(0.25)\n",
    "IQR = q3 - q1\n",
    "upper = q3 + (1.5 * IQR)\n",
    "lower = q1 - (1.5 * IQR)\n",
    "\n",
    "# We will calculate the outlier range based on training dataset only and apply to validation dataset (to avoid data leakage)\n",
    "train_inlier_indices = y_train_copy[(y_train_copy > lower) & (y_train_copy < upper)].index\n",
    "y_train_copy = y_train_copy.loc[train_inlier_indices]\n",
    "df_train_copy = df_train_copy.loc[train_inlier_indices]\n",
    "\n",
    "# Outliers in validation target should not be removed\n",
    "# val_inlier_indices = y_val_copy[(y_val_copy > lower) & (y_val_copy < upper)].index\n",
    "# y_val_copy = y_val_copy.loc[val_inlier_indices]\n",
    "# df_val_copy = df_val_copy.loc[val_inlier_indices]\n",
    "\n",
    "features = ['t1', 't2', 'hum', 'wind_speed', 'weather_code', 'is_holiday', 'is_weekend', 'season']\n",
    "model = xgb.XGBRegressor(random_state=42, n_jobs=-1, objective=\"reg:squarederror\")\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_train_pred, model = train_predict(df_train_copy[features],df_val_copy[features],y_train_copy,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "# Update scoring dataframe\n",
    "score_entry = {\"algo\": \"xgb\", \"experiment\": 23, \"desc\": \"remove outliers on target cnt\"}\n",
    "score_entry.update(scores)\n",
    "exp_scores_xgb = exp_scores_xgb.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_xgb.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wu-UsFqGSE9W",
   "metadata": {
    "id": "wu-UsFqGSE9W"
   },
   "source": [
    "**Observations:** After removing outliers, the score has not improved, so we will decide to keep all the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5GKhOFlgFVdG",
   "metadata": {
    "id": "5GKhOFlgFVdG"
   },
   "outputs": [],
   "source": [
    "#Experiment 24\n",
    "\n",
    "# Hyper-parameter tuning XGB in a manual fashion\n",
    "# First we will find learning_rate (eta) and num_rounds (n_estimators)\n",
    "\n",
    "# max_depths = [1, 2, 3, 4, 5, 6, 10, 15, 20, 50]\n",
    "# max_depths = [3, 4, 5, 6, 10, 15]\n",
    "# min_samples_leafs = [1, 2, 5, 10, 15, 20, 100, 200, 500]\n",
    "# min_child_weights = [4, 5, 6, 8, 10]\n",
    "# colsample_bytrees = [0.4, 0.6, 0.8]\n",
    "\n",
    "learning_params = [\n",
    "    (1.0, 500),\n",
    "    (0.3, 800),\n",
    "    (0.1, 1000),\n",
    "    (0.05, 1500),\n",
    "    (0.01, 3000),\n",
    "]\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "# for max_depth in max_depths:\n",
    "#     for min_samples_leaf in min_samples_leafs:\n",
    "for eta, n_estimators in learning_params:\n",
    "    model = xgb.XGBRegressor(random_state=42, n_jobs=-1, objective=\"reg:squarederror\", booster='gbtree', learning_rate=eta, n_estimators=n_estimators)\n",
    "    # Train and get predictions\n",
    "    y_pred, y_train_pred, model = train_predict(df_train_copy[new_features],df_val_copy[new_features],y_train_copy,model)\n",
    "\n",
    "    # Score\n",
    "    scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "    # Update scoring dataframe\n",
    "    score_entry = {\"algo\": \"xgb\", \"experiment\": 24, \"desc\": f\"hyper-params booster='gbtree', learning_rate: {eta}, n_estimators: {n_estimators}\"}\n",
    "    score_entry.update(scores)\n",
    "    exp_scores_xgb = exp_scores_xgb.append(score_entry,ignore_index=True)\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_xgb.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4c4e32c",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_scores_xgb.sort_values(by=['val_rmse'],ascending=True)['desc'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8ad8728",
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_df = pd.DataFrame()\n",
    "tmp_df[['val_rmse','train_rmse']] = exp_scores_xgb.loc[3:7,['val_rmse','train_rmse']]\n",
    "tmp_df.reset_index(drop=True,inplace=True)\n",
    "tmp_df[['booster','eta','n_estimators']] = exp_scores_xgb.loc[3:7]['desc'].str.split(',').to_list()\n",
    "tmp_df.drop(columns=['booster'],inplace=True)\n",
    "tmp_df['eta']=tmp_df['eta'].str.replace('learning_rate: ','')\n",
    "tmp_df['n_estimators']=tmp_df['n_estimators'].str.replace('n_estimators: ','')\n",
    "tmp_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "014c1650",
   "metadata": {},
   "outputs": [],
   "source": [
    "# plt.figure(figsize = (8,6))\n",
    "fig = px.scatter(tmp_df, x=\"val_rmse\", y=\"train_rmse\", hover_data=[\"n_estimators\",\"eta\"], hover_name=tmp_df.index)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f8a5ce4",
   "metadata": {},
   "source": [
    "**Observations:** The best score val_rmse of 341 was achieved with hyper-params booster='gbtree', learning_rate: 0.1, n_estimators: 1000. We will use these for further tuning other parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6847abcb",
   "metadata": {},
   "source": [
    "#### Hyper-parameter tuning of XGB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "632625bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Experiment 25\n",
    "\n",
    "# Hyper-parameter tuning XGB in a manual fashion\n",
    "# After having tunerd learning_rate (eta) and num_rounds (n_estimators), now we will tune\n",
    "# max_depth, min_samples_leaf, min_child_weight, colsample_bytrees\n",
    "\n",
    "eta = 0.1\n",
    "n_estimators = 1000\n",
    "\n",
    "# learning_params = [\n",
    "#     (1.0, 500),\n",
    "#     (0.3, 800),\n",
    "#     (0.1, 1000),\n",
    "#     (0.05, 1500),\n",
    "#     (0.01, 3000),\n",
    "# ]\n",
    "\n",
    "max_depth_range = [3, 4, 5, 6, 10, 15]\n",
    "# min_samples_leaf_range = [1, 2, 5, 10, 15, 20, 100, 200, 500]  # COmmented since this xgb does not seem to have this parameter\n",
    "min_child_weight_range = [4, 5, 6, 8, 10]\n",
    "colsample_bytree_range = [0.4, 0.6, 0.8]\n",
    "\n",
    "df_train_copy = df_train.copy()\n",
    "df_val_copy = df_val.copy()\n",
    "\n",
    "y_train_copy = y_train.copy()\n",
    "y_val_copy = y_val.copy()\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_train_copy = pre_process_new_ft(df_train_copy)\n",
    "df_val_copy = pre_process_new_ft(df_val_copy)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_train_copy['timestamp']\n",
    "del df_val_copy['timestamp']\n",
    "\n",
    "new_features = list(df_train_copy.columns)\n",
    "\n",
    "for max_depth in max_depth_range:\n",
    "    for min_child_weight in min_child_weight_range:\n",
    "        for colsample_bytree in colsample_bytree_range:\n",
    "            model = xgb.XGBRegressor(random_state=42, n_jobs=-1, objective=\"reg:squarederror\", booster='gbtree', learning_rate=eta, n_estimators=n_estimators, max_depth=max_depth, min_child_weight=min_child_weight, colsample_bytree=colsample_bytree)\n",
    "            # Train and get predictions\n",
    "            y_pred, y_train_pred, model = train_predict(df_train_copy[new_features],df_val_copy[new_features],y_train_copy,model)\n",
    "\n",
    "            # Score\n",
    "            scores = evaluate_scores(y_val_copy, y_pred, y_train_copy, y_train_pred)\n",
    "\n",
    "            # Update scoring dataframe\n",
    "            score_entry = {\"algo\": \"xgb\", \"experiment\": 25, \"desc\": f\"hyper-params booster='gbtree', learning_rate: {eta}, n_estimators: {n_estimators}, max_depth: {max_depth}, min_child_weight: {min_child_weight}, colsample_bytree: {colsample_bytree}\"}\n",
    "            score_entry.update(scores)\n",
    "            exp_scores_xgb = exp_scores_xgb.append(score_entry,ignore_index=True)\n",
    "#             display(exp_scores_xgb.tail(1))\n",
    "\n",
    "# Print scoring dataframe\n",
    "display(exp_scores_xgb.sort_values(by=['val_rmse'],ascending=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45eddcbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "exp_scores_xgb.loc[8:10,:]['desc'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52372433",
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_df2 = pd.DataFrame()\n",
    "tmp_df2[['val_rmse','train_rmse']] = exp_scores_xgb.loc[8:,['val_rmse','train_rmse']]\n",
    "tmp_df2.reset_index(drop=True,inplace=True)\n",
    "tmp_df2[['booster','eta','n_estimators', 'max_depth', 'min_child_weight', 'colsample_bytree']] = exp_scores_xgb.loc[8:]['desc'].str.split(',').to_list()\n",
    "tmp_df2.drop(columns=['booster'],inplace=True)\n",
    "tmp_df['eta']=tmp_df2['eta'].str.replace(' learning_rate:','')\n",
    "tmp_df2['n_estimators']=tmp_df2['n_estimators'].str.replace(' n_estimators:','')\n",
    "tmp_df2['max_depth']=tmp_df2['max_depth'].str.replace(' max_depth:','')\n",
    "tmp_df2['min_child_weight']=tmp_df2['min_child_weight'].str.replace(' min_child_weight:','')\n",
    "tmp_df2['colsample_bytree']=tmp_df2['colsample_bytree'].str.replace(' colsample_bytree:','')\n",
    "tmp_df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffe21086",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize = (8,6))\n",
    "fig = px.scatter(tmp_df2, x=\"val_rmse\", y=\"train_rmse\", hover_data=[\"max_depth\",\"min_child_weight\",\"colsample_bytree\"], hover_name=tmp_df2.index)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ed0b404",
   "metadata": {},
   "source": [
    "**Observations:** Best score is val_rmse: 299, train_rmse: 101, achieved with eta=0.1, n_estimators=1000, max_depth=4, min_child_weight=5, colsample_bytree=0.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01b3182b",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Saving experiment scores to file\n",
    "display(exp_scores_xgb.describe())\n",
    "exp_scores_xgb.to_csv('xgb-scores.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47fe2892",
   "metadata": {},
   "source": [
    "<a class='anchor' id='final-model'></a>\n",
    "[back to TOC](#toc)\n",
    "### 6. Final model:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c94b8784",
   "metadata": {},
   "source": [
    "<a id='choose-final-model'></a>\n",
    "#### 6.1 Compare results from hyper-parameter tuning for the different models and choose final model\n",
    "[back to TOC](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6c76f00",
   "metadata": {},
   "source": [
    "#### Load saved scores from different experiments with different algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7154970",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -1 work-dump/*.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d48793ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_files = ['work-dump/linear-scores.csv', 'work-dump/decision-tree-scores.csv', 'work-dump/random-forest-scores.csv', 'work-dump/xgb-scores.csv']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce177fa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cons_scores = pd.DataFrame()\n",
    "\n",
    "for file in scores_files:\n",
    "    df_tmp = pd.read_csv(file)\n",
    "    df_cons_scores = pd.concat([df_cons_scores,df_tmp],axis=0,ignore_index=True)\n",
    "\n",
    "del df_tmp\n",
    "df_cons_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c97eeb3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cons_scores.sort_values(by=['val_rmse'],ascending=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b8d7cc7",
   "metadata": {},
   "source": [
    "#### Let us look at the val_rmse and train_rmse scores for different algorithms/models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a5d79b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_cons_scores.groupby(['algo'])['val_rmse','train_rmse'].agg(['mean','min','max','std'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d39883fc",
   "metadata": {},
   "source": [
    "#### Let us visualize the val_rmse scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bcf6761",
   "metadata": {},
   "outputs": [],
   "source": [
    "px.line(x=df_cons_scores.index,y=df_cons_scores.val_rmse,color=df_cons_scores.algo)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8130902e",
   "metadata": {},
   "source": [
    "**Observations:** We can see that results from XGB are the best with lower val_rmse overall."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78af170d",
   "metadata": {},
   "source": [
    "<a id='train-final-model'></a>\n",
    "#### 6.2 Train final model\n",
    "[back to TOC](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0b36088",
   "metadata": {},
   "source": [
    "Will train final model with full_train and test on test data using XGB with hyper-parameters of eta=0.1, n_estimators=1000, max_depth=4, min_child_weight=5, colsample_bytree=0.8\n",
    "\n",
    "Will copy only relevant code which can then be used for the train.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16cda7cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error\n",
    "\n",
    "import xgboost as xgb\n",
    "\n",
    "import datetime\n",
    "\n",
    "import pickle\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d55194f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load data to dataframe\n",
    "datafile = 'london_merged.csv'\n",
    "df = pd.read_csv(datafile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e25efec8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert dtypes to save memory\n",
    "df['weather_code'] = df['weather_code'].astype('uint8')\n",
    "df['is_holiday'] = df['is_holiday'].astype('uint8')\n",
    "df['is_weekend'] = df['is_weekend'].astype('uint8')\n",
    "df['season'] = df['season'].astype('uint8')\n",
    "df['t1'] = df['t1'].astype('float16')\n",
    "df['t2'] = df['t2'].astype('float16')\n",
    "df['hum'] = df['hum'].astype('float16')\n",
    "df['wind_speed'] = df['wind_speed'].astype('float16')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a6fce34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sort data according to timestamp\n",
    "df['timestamp'] = pd.to_datetime(df['timestamp'])\n",
    "df = df.sort_values(by=['timestamp'],ascending=True)\n",
    "df.reset_index(drop=True,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af7231f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting data as Full train (80%), Test (20%)\n",
    "df_full_train, df_test = train_test_split(df,test_size=0.2,shuffle=False,random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be04aa4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set target and delete it from dataframe\n",
    "y_full_train = df_full_train['cnt']\n",
    "y_test = df_test['cnt']\n",
    "\n",
    "del df_full_train['cnt']\n",
    "del df_test['cnt']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "683acb07",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to train the model and predict on validation data\n",
    "\n",
    "def train_predict(df_full_train,df_test,y_full_train,model):\n",
    "    X_full_train = df_full_train.values\n",
    "    model.fit(X_full_train, y_full_train)\n",
    "\n",
    "    X_test = df_test.values\n",
    "    y_pred = model.predict(X_test)\n",
    "    \n",
    "    y_train_pred = model.predict(X_full_train)\n",
    "    \n",
    "    return y_pred, y_train_pred, model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bf357fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to evaluate various metrics/scores on predictions on validation and training\n",
    "\n",
    "def evaluate_scores(y_test_eval, y_pred_eval, y_full_train_eval, y_pred_full_train_eval):\n",
    "    scores = {}\n",
    "    scores['val_r2'] = r2_score(y_test_eval, y_pred_eval)\n",
    "    scores['val_mse'] = mean_squared_error(y_test_eval, y_pred_eval,squared=True)\n",
    "    scores['val_rmse'] = mean_squared_error(y_test_eval, y_pred_eval,squared=False)\n",
    "    scores['val_mae'] = mean_absolute_error(y_test_eval, y_pred_eval)\n",
    "    \n",
    "    scores['train_r2'] = r2_score(y_full_train_eval, y_pred_full_train_eval)\n",
    "    scores['train_mse'] = mean_squared_error(y_full_train_eval, y_pred_full_train_eval,squared=True)\n",
    "    scores['train_rmse'] = mean_squared_error(y_full_train_eval, y_pred_full_train_eval,squared=False)\n",
    "    scores['train_mae'] = mean_absolute_error(y_full_train_eval, y_pred_full_train_eval)\n",
    "\n",
    "    rnd_digits = 5 #round upto how many digits\n",
    "    for metric, value in scores.items():\n",
    "        scores[metric] = round(scores[metric],rnd_digits)\n",
    "    \n",
    "    return scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30b4efbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to perform pre processing on data before training\n",
    "# Combining all the step by step processing done above into a function\n",
    "\n",
    "\n",
    "# Function to now create different features from timestamp\n",
    "def pre_process_new_ft(df_to_process):\n",
    "\n",
    "    df_to_process['year'] = df_to_process['timestamp'].dt.year\n",
    "    df_to_process['month'] = df_to_process['timestamp'].dt.month\n",
    "    df_to_process['day'] = df_to_process['timestamp'].dt.day\n",
    "    df_to_process['hour'] = df_to_process['timestamp'].dt.hour\n",
    "    df_to_process['day-of-week'] = pd.to_datetime(df_to_process['timestamp']).dt.dayofweek.values\n",
    "    df_to_process['week-of-year'] = pd.to_datetime(df_to_process['timestamp']).dt.isocalendar().week.values\n",
    "    df_to_process['day-of-year'] = pd.to_datetime(df_to_process['timestamp']).dt.dayofyear\n",
    "\n",
    "\n",
    "    df_to_process['year'] = df_to_process['year'].astype('uint16')\n",
    "    df_to_process['month'] = df_to_process['month'].astype('uint8')\n",
    "    df_to_process['day'] = df_to_process['day'].astype('uint8')\n",
    "    df_to_process['hour'] = df_to_process['hour'].astype('uint8')\n",
    "    df_to_process['day-of-week'] = df_to_process['day-of-week'].astype('uint8')\n",
    "    df_to_process['week-of-year'] = df_to_process['week-of-year'].astype('uint8')\n",
    "    df_to_process['day-of-year'] = df_to_process['day-of-year'].astype('uint16')\n",
    "\n",
    "\n",
    "    # Create cyclical encoded features\n",
    "    cyclical_features = ['year', 'month', 'day', 'hour', 'day-of-week', 'week-of-year', 'day-of-year']\n",
    "    for col in cyclical_features:\n",
    "        df_to_process[f\"{col}_x_norm\"] = 2 * math.pi * df_to_process[col] / df_to_process[col].max()\n",
    "        df_to_process[f\"{col}_cos_x\"] = np.cos(df_to_process[f\"{col}_x_norm\"])\n",
    "        df_to_process[f\"{col}_sin_x\"] = np.sin(df_to_process[f\"{col}_x_norm\"])\n",
    "        del df_to_process[f\"{col}_x_norm\"]\n",
    "\n",
    "    return df_to_process\n",
    "\n",
    "\n",
    "# Function to encode the time features using cyclical encoding using sine and cosine. Drop original time features\n",
    "def pre_process_cyclic_encode(df_to_process,drop_org=True):\n",
    "    # Create cyclical encoded features\n",
    "    cyclical_features = ['year', 'month', 'day', 'hour', 'day-of-week', 'week-of-year', 'day-of-year']\n",
    "    for col in cyclical_features:\n",
    "        df_to_process[f\"{col}_x_norm\"] = 2 * math.pi * df_to_process[col] / df_to_process[col].max()\n",
    "        df_to_process[f\"{col}_cos_x\"] = np.cos(df_to_process[f\"{col}_x_norm\"])\n",
    "        df_to_process[f\"{col}_sin_x\"] = np.sin(df_to_process[f\"{col}_x_norm\"])\n",
    "        del df_to_process[f\"{col}_x_norm\"]\n",
    "\n",
    "    if drop_org:\n",
    "        for col in cyclical_features:\n",
    "            del df_to_process[col]\n",
    "            \n",
    "    return df_to_process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1892eb0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train final model\n",
    "\n",
    "# Preprocess by creating new time related features\n",
    "df_full_train = pre_process_new_ft(df_full_train)\n",
    "df_test = pre_process_new_ft(df_test)\n",
    "\n",
    "# Drop the timestamp feature, since we added more meaningful features and experiment with timestamp had not helped\n",
    "del df_full_train['timestamp']\n",
    "del df_test['timestamp']\n",
    "\n",
    "new_features = list(df_full_train.columns)\n",
    "\n",
    "#Define the hyper-parameters for the XGB model\n",
    "eta=0.1\n",
    "n_estimators=1000\n",
    "max_depth=4\n",
    "min_child_weight=5\n",
    "colsample_bytree=0.8\n",
    "\n",
    "model = xgb.XGBRegressor(random_state=42, n_jobs=-1, objective=\"reg:squarederror\", booster='gbtree', learning_rate=eta, n_estimators=n_estimators, max_depth=max_depth, min_child_weight=min_child_weight, colsample_bytree=colsample_bytree)\n",
    "\n",
    "# Train and get predictions\n",
    "y_pred, y_full_train_pred, model = train_predict(df_full_train[new_features],df_test[new_features],y_full_train,model)\n",
    "\n",
    "# Score\n",
    "scores = evaluate_scores(y_test, y_pred, y_full_train, y_full_train_pred)\n",
    "\n",
    "print(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc1763a7",
   "metadata": {},
   "source": [
    "**Observations:** Score on test data is almost same as our validation, which is a good sign that the model is behaving consistently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1b9e0a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save model to file\n",
    "model_output_file = f'capstone-xgb_model.bin'\n",
    "\n",
    "with open(model_output_file,'wb') as f_out:\n",
    "    pickle.dump((model),f_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65a303c1",
   "metadata": {},
   "source": [
    "## END of Notebook"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "ml-zoomcamp-capstone-v2.3.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}