Lecture05_03 - Convexity and Jensen's Inequality.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convexity and Jensen's Inequality\n",
    "\n",
    "##### Keywords: convex, Jensen's inequality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hide": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "pd.set_option('display.width', 500)\n",
    "pd.set_option('display.max_columns', 100)\n",
    "pd.set_option('display.notebook_repr_html', True)\n",
    "import seaborn as sns\n",
    "sns.set_style(\"whitegrid\")\n",
    "sns.set_context(\"poster\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convexity\n",
    "\n",
    "Let $f$ be a function with domain the set of real numbers. If the second derivative is greater than zero \n",
    "for all $x\\in R$ this function is convex\\footnote{In the case of vector inputs this is called positive \n",
    "semi-definite}. \n",
    "\n",
    "\n",
    "Consider the case of two random variables $x_1$ and $x_2$, as seen in the diagram below:\n",
    "\n",
    "![](../images/EM1.png)\n",
    "\n",
    "\n",
    "**Defnition** Let f be a real valued function defined on an interval $I = [a, b]$.\n",
    "$f$ is said to be convex on I if $\\forall x_1, x_2 \\in I, \\lambda \\in [0, 1]$,\n",
    "\n",
    "\\begin{equation}\n",
    "f(\\lambda x_1 + (1 - \\lambda)\\,x_2) \\le \\lambda f(x_1) + (1- \\lambda)\\,f(x_2).\n",
    "\\end{equation}\n",
    "\n",
    "$f$ is said to be strictly convex if the inequality is strict. Intuitively, this definition\n",
    "states that the function falls below  the\n",
    "straight line (the secant) from points $(x_1, f(x_1))$ to $(x_2, f(x_2))$. In other words, the equality is satisfied only for $\\lambda = 0$ and $\\lambda = 1$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Jensen's Inequality\n",
    "\n",
    "Let $f$ be a convex function defined on an interval $I$. \n",
    "If $x_1,x_2,\\dots,x_n \\in I {\\rm and} \\lambda_1, \\lambda_2,\\ldots,\\lambda_n \\ge  0$ with $\\sum^n_{i=1} \\lambda_i = 1$,\n",
    "\n",
    "\\begin{equation}\n",
    "f \\left( \\sum_{i=1}^n \\lambda_i \\,  x_i \\right) \\le  \\sum_{i=1}^n \\lambda_i f(x_i) \n",
    "\\end{equation}\n",
    "\n",
    "**Proof:**\tFor $n = 1$ this is trivial. The case $n = 2$ corresponds to the definition of convexity (see above). \n",
    "To show that this is true for all natural numbers, we proceed by induction. Assume the theorem is true for some $n$ then,\n",
    "\n",
    "$$\n",
    "\\begin{eqnarray}\n",
    "f \\left( \\sum_{i=1}^{n+1} \\lambda_i \\,  x_i \\right) &=& f\\left( \\lambda_{n+1} x_{n+1} + \\sum_{i=1}^n \\lambda_i \\,  x_i  \\right) \\nonumber \\\\\n",
    "\t\t&=&  f\\left( \\lambda_{n+1} x_{n+1} + \\frac{(1-\\lambda_{n+1})}{(1-\\lambda_{n+1})}\\sum_{i=1}^n \\lambda_i \\,  x_i  \\right) \\nonumber \\\\\n",
    "\t\t& \\le & \\lambda_{n+1} f(x_{n+1}) + (1-\\lambda_{n+1}) f \\left( \\frac{1}{(1-\\lambda_{n+1})} \\sum_{i=1}^n \\lambda_i \\,  x_i  \\right) \\nonumber \\\\\n",
    "\t\t& = & \\lambda_{n+1} f(x_{n+1}) + (1-\\lambda_{n+1}) f \\left( \\sum_{i=1}^n \\frac{\\lambda_i}{(1-\\lambda_{n+1})} \\,  x_i  \\right)  \\nonumber \\\\\n",
    "\t\t& \\le & \\lambda_{n+1} f(x_{n+1}) + (1-\\lambda_{n+1})  \\sum_{i=1}^n \\frac{\\lambda_i}{(1-\\lambda_{n+1})} \\,  f(x_i)  \\nonumber \\\\ \n",
    "\t\t& =&  \\lambda_{n+1} f(x_{n+1}) + \\sum_{i=1}^n \\lambda_i f(x_i) \\nonumber \\\\ \n",
    "\t\t& =&  \\sum_{i=1}^{n+1}  \\lambda_i f(x_i)\n",
    "\\end{eqnarray}\n",
    "$$\n",
    "\n",
    "By interpreting the $\\lambda_i$ as the probability distribution over a discrete variable $x$ taking the values $\\{x_i\\}$:\n",
    "\n",
    "$$f(\\mathrm{E}[x]) \\le \\mathrm{E}[f(x)]$$\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theorem**. Let $f$ be a convex function, and $X$ be a random variable, then\n",
    "\n",
    "$$ E[f(X)] \\ge f(E[X]) $$ \n",
    "\n",
    "Furthermore, if $f$ is stricly convex (i.e. $f''(x)>0$), then $E[f(x)]=f(E[X])$ only if $X=E[X]$ with\n",
    "probability 1 ($X$ is constant).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}