seminars/seminar1/seminar1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1f858ee3",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "import itertools\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7f669e7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center>Deep Generative Models</center>\n",
    "## <center>Seminar 1</center>\n",
    "\n",
    "<center>10.09.2024</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13c3a67c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Kondratenko Vladimir\n",
    "\n",
    "\n",
    "tg: @username27\n",
    "\n",
    "e-mail: vladimir.kondratenko@phystech.edu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a9d090e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Plan\n",
    "\n",
    "Part 1\n",
    "- Maximum Likelihood Estimation\n",
    "- Histograms\n",
    "- Bayes' theorem\n",
    "\n",
    "Part 2\n",
    "- PixelCNN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8a5f2e3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Maximum Likelihood Estimation (MLE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d79e097d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Recall the general task of Generative Modeling. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca41310b",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We have samples $X \\sim \\pi(x)$. We assume, that $\\pi(x) = p(x|\\theta), \\theta \\in \\Theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41176b8c",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We want to estimate $\\hat{\\theta} = \\hat{\\theta}(X)$ in order to sample from $p(x|\\hat{\\theta})$ and estimate the corresponding pdf. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d8e0f41",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How?\n",
    "\n",
    "$\\theta^* = \\arg\\max\\limits_{\\theta} p(X | \\theta) = \\arg\\max\\limits_{\\theta} \\prod_{i=1}^n p(x_i | \\theta) = \\arg\\max\\limits_{\\theta} \\sum_{i=1}^n \\log p(x_i | \\theta)$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44846027",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MLE examlpes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d69d6e5",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Problem 1\n",
    "\n",
    "$x_1, ... x_n \\sim U[0,\\theta]$. \n",
    "\n",
    "$\\hat{\\theta}_{ML} = ?$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c36b2e90",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For a single $x \\sim U[0,\\theta]$:\n",
    "\n",
    "$p(x) = \\dfrac{1}{\\theta}, x \\in [0,\\theta]$\n",
    "\n",
    "$p(x) = 0, x \\notin [0,\\theta]$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "585637e7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We have $x_1, \\dots x_n \\sim U[0,\\theta]$:\n",
    "\n",
    "$p(X | \\theta) = \\begin{cases} \\dfrac{1}{\\theta^{n}}, &X \\in [0,\\theta]^{n}, \\\\ 0, &X \\notin [0,\\theta]^{n} \\end{cases}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69a58da5",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "It also can be written as:\n",
    "\n",
    "$p(X | \\theta) = \\begin{cases} 0, &\\theta < \\max(x_1, \\dots x_n), \\\\\n",
    "\\dfrac{1}{\\theta^{n}}, &\\theta \\geq \\max(x_1, \\dots x_n) \\end{cases}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "541edd00",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "So, we can find:\n",
    "\n",
    "$\\hat{\\theta}_{ML} = \\max(x_1, ... x_n)$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d41cca99",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$\\hat{\\theta}_{ML} = \\max(x_1, ... x_n)$ is also random variable\n",
    "\n",
    "How does Cumulative distribution function $F_{\\hat{\\theta}_{ML}}(x) = P(\\hat{\\theta}_{ML} < x )$ looks like?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1507cee2",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "$P_{X \\sim U[0,\\theta]}(\\max(x_1, ... x_n) < x) = \\dfrac{x^n}{\\theta^n}$, $x \\in [0,\\theta]$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71066e2d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Check if the $\\hat{\\theta}_{ML}$ is **unbiased** estimation of $\\theta$, i.e. $\\mathbb{E}\\hat{\\theta}_{ML} = \\theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8309a859",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's find this $\\mathbb{E}\\hat{\\theta}_{ML} = ?$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52dd5855",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We know that:\n",
    "\n",
    "$\\mathbb{E}x = \\int_{-\\infty}^{+\\infty} x f(x) dx = \\int_{-\\infty}^{+\\infty} x dF(x)$\n",
    "\n",
    "$\\mathbb{E}_{X \\sim U[0, \\theta]} \\hat{\\theta}_{ML} = \\mathbb{E}_{\\hat{\\theta}_{ML}} x$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e46620c",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$\\mathbb{E}_{X \\sim U[0,\\theta]}\\hat{\\theta}_{ML} =\n",
    "\\int_{0}^{\\theta} x d \\dfrac{x^n}{\\theta^n} = ...\n",
    "%\\int_{0}^{\\theta} \\dfrac{xnx^{n-1}}{\\theta^n} d x =\n",
    "%\\int_{0}^{\\theta} \\dfrac{nx^{n}}{\\theta^n} d x =\n",
    "%\\dfrac{n\\theta^{n+1}}{(n+1)\\theta^n}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9561d440",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$\\Rightarrow \\mathbb{E}_{X \\sim U[0,\\theta]}\\hat{\\theta}_{ML} = \\dfrac{n}{n+1}\\theta \\Rightarrow$ MLE is **biased**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9429619d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### MLE\n",
    "\n",
    "- not unbiased\n",
    "- *(regularity conditions)* consistent (состоятельные) i.e. $P_{\\theta_{\\text{true}}}( \\Vert \\hat{\\theta} - \\theta_{\\text{true}} \\Vert > \\varepsilon ) \\rightarrow 0 \\quad \\forall \\varepsilon > 0$\n",
    "- *(regularity conditions ++)* asymptotically normal (асимптотически нормальные) i.e $\\sqrt{n} \\left(\\hat{\\theta} - \\theta \\right) \\overset{d_{\\theta}}{\\rightarrow} \\mathcal{N}(0, \\Sigma(\\theta))$\n",
    "- *(regularity conditions ++)* asymptotically efficient (асимптотически эффективные) i.e. $\\sqrt{n} \\left(\\hat{\\theta} - \\theta \\right) \\overset{d_{\\theta}}{\\rightarrow} \\mathcal{N}(0, I(\\theta)) \\text{  ,where } I(\\theta) \\text{ is Fisher matrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04300b36",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Simple distributions\n",
    "### Example 1\n",
    "\n",
    "Let we have samples $X \\sim \\pi(x)$ and want approximate $\\pi(x)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "658f8086",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def generate_1d_data(count, bins):\n",
    "    np.random.seed(42)\n",
    "    a = 0.2 + 0.05 * np.random.randn(count)\n",
    "    b = 0.6 + 0.15 * np.random.randn(count)\n",
    "    mask = np.random.rand(count) < 0.5\n",
    "    samples = np.clip((a * mask + b * (1 - mask)) * (bins - 1), 0.0, (bins - 1))\n",
    "    data = samples.astype('int')\n",
    "    split = int(0.8 * count)\n",
    "    train_samples, test_samples = samples[:split], samples[split:]\n",
    "    train_hist, test_hist = data[:split], data[split:]\n",
    "    return train_samples, test_samples, train_hist, test_hist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b4e16325",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def plot_1d_data(train_data, test_data):\n",
    "    bins = int(max(test_data.max(), train_data.max()) - min(test_data.min(), train_data.min())) + 1\n",
    "    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))\n",
    "    ax1.set_title('train')\n",
    "    ax1.hist(train_data, bins=bins, density=True)\n",
    "    ax1.set_xlabel('x')\n",
    "    ax2.set_title('test')\n",
    "    ax2.hist(test_data, bins=np.arange(bins), density=True)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6279eeb0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let we have samples $X \\sim \\pi(x)$ and want approximate $\\pi(x)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "04ebb3cd",
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1YAAAHWCAYAAAB0cxiaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOGElEQVR4nO3dfVyVdZ7/8Tc3wkkTSFg56mA4ZeENQUIQTjPWdB7Bxm+NakjNklhXp9nBzDPLJI63WYNNK2HhxsPZrLbVwXHX2EYdWsLs5gdqAo7ZpuO0Ga52QPInFI2gnOv3h+vRoweHwwUchNfz8bgeA9/rc13X93tBfOftdXP8DMMwBAAAAADoMn9fdwAAAAAArnYEKwAAAAAwiWAFAAAAACYRrAAAAADAJIIVAAAAAJhEsAIAAAAAkwhWAAAAAGASwQoAAAAATCJYAQAAAIBJBCvgKhcdHa3HHnvM190AAAAY0AhWQC+orKzU8uXLderUKV93BQCAy/TGPPXLX/5SpaWlPbZ/wNcIVkAvqKys1IoVK3pkwjp06JB+/etfd/t+AQADR0/OU+cRrNDfEayAPsTpdOr06dNebRMcHKxBgwb1UI8AAADQGQQroIctX75cubm5kqQxY8bIz89Pfn5+OnLkiPz8/JSTk6MNGzZowoQJCg4OVllZmSTpH//xHzV58mSFh4frmmuuUUJCgv7t3/7tsv1f+ozVa6+9Jj8/P/3f//t/Zbfb9Vd/9VcaMmSI7r//fp04caJXxgwAuHpcaZ6SpH/9139VQkKCrrnmGg0bNkzTp0/X0aNH3fZx+PBhPfjgg7JarbJYLPrOd76j6dOnq6mpSZLk5+enlpYWvf76667983ww+ptAX3cA6O8eeOAB/fGPf9RvfvMbvfDCC4qIiJAk/dVf/ZUkaceOHfrtb3+rnJwcRUREKDo6WpK0Zs0aTZ06VTNnzlRbW5tKSkqUmZmprVu3Kj09/S8ed968ebruuuu0bNkyHTlyRIWFhcrJydGmTZt6bKwAgKvPleapZ599VkuWLNFDDz2kv/u7v9OJEyf00ksv6Qc/+IFqa2sVFhamtrY2paamqrW1VfPmzZPVatWxY8e0detWnTp1SqGhoXrjjTf0d3/3d0pKStLcuXMlSTfccIMvhw10PwNAj3v++ecNScbnn3/u1i7J8Pf3Nz755JPLtvn222/dvm9razMmTpxo/PCHP3Rrv/76642srCzX96+++qohybDZbIbT6XS1L1iwwAgICDBOnTplfkAAgH7F0zx15MgRIyAgwHj22Wfdaj/++GMjMDDQ1V5bW2tIMjZv3nzFYwwZMsRtvgL6G24FBHxsypQpGj9+/GXt11xzjevr//f//p+ampr0/e9/XzU1NZ3a79y5c+Xn5+f6/vvf/77a29v1xRdfmO80AKDf27Jli5xOpx566CE1Nja6FqvVqrFjx+rdd9+VJIWGhkqS3n77bX377be+7DLgU9wKCPjYmDFjPLZv3bpVzzzzjPbt26fW1lZX+8Vh6UpGjx7t9v11110n6VxIAwDgLzl8+LAMw9DYsWM9rj//4qQxY8bIbreroKBAGzZs0Pe//31NnTpVjzzyiCt0AQMBwQrwsYuvTJ33wQcfaOrUqfrBD36gf/qnf9KIESM0aNAgvfrqq9q4cWOn9hsQEOCx3TAMU/0FAAwMTqdTfn5++v3vf+9xTrn22mtdX69evVqPPfaY/uM//kP/+Z//qSeeeEL5+fnatWuXvvOd7/RmtwGfIVgBvaCzV5nO+/d//3dZLBa9/fbbCg4OdrW/+uqr3d01AAA8zlM33HCDDMPQmDFjdNNNN/3FfcTGxio2NlaLFy9WZWWlvve976m4uFjPPPNMh8cA+hOesQJ6wZAhQySp0x+8GBAQID8/P7W3t7vajhw5wgcrAgB6hKd56oEHHlBAQIBWrFhx2d0OhmHoq6++kiQ1Nzfr7NmzbutjY2Pl7+/vdiv7kCFDevQDiAFf44oV0AsSEhIkSb/4xS80ffp0DRo0SH/zN3/TYX16eroKCgqUlpamhx9+WA0NDVq7dq1uvPFG7d+/v7e6DQAYIDqap5555hnl5eXpyJEjysjI0NChQ/X555/rzTff1Ny5c/UP//AP2rFjh3JycpSZmambbrpJZ8+e1RtvvKGAgAA9+OCDbsd45513VFBQoJEjR2rMmDFKTk721ZCBbkewAnrBbbfdppUrV6q4uFhlZWVyOp36/PPPO6z/4Q9/qFdeeUWrVq3Sk08+qTFjxui5557TkSNHCFYAgG7X0Ty1cOFC3XTTTXrhhRe0YsUKSVJUVJTuueceTZ06VZIUFxen1NRU/e53v9OxY8c0ePBgxcXF6fe//71uv/121zEKCgo0d+5cLV68WH/+85+VlZVFsEK/4mfwJDsAAAAAmMIzVgAAAABgEsEKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAAAAMCkAfM5Vk6nU8ePH9fQoUPl5+fn6+4AwIBhGIa+/vprjRw5Uv7+/HvexZibAMA3emJuGjDB6vjx44qKivJ1NwBgwDp69Ki+853v+LobfQpzEwD4VnfOTQMmWA0dOlTSuZMXEhLi494AwMDR3NysqKgo199hXMDcBAC+0RNz04AJVudvsQgJCWHyAgAf4Fa3yzE3AYBvdefcxM3uAAAAAGASwQoAAAAATCJYAQAAAIBJBCsAAAAAMIlgBQAAAAAmEawAAAAAwCSCFQAAAACYRLACAAAAAJMIVgAAAABgEsEKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAADwtq1axUdHS2LxaLk5GTt2bPnivWbN29WTEyMLBaLYmNjtX379stqPv30U02dOlWhoaEaMmSIbrvtNtXV1fXUEAAAfRjBCgDQ723atEl2u13Lli1TTU2N4uLilJqaqoaGBo/1lZWVmjFjhmbPnq3a2lplZGQoIyNDBw4ccNV89tlnuuOOOxQTE6OdO3dq//79WrJkiSwWS28NCwDQh/gZhmH4uhO9obm5WaGhoWpqalJISIivu9MnRS/c5vb9kVXpPuoJgP6kL/z9TU5O1m233aaioiJJktPpVFRUlObNm6eFCxdeVj9t2jS1tLRo69atrrbbb79d8fHxKi4uliRNnz5dgwYN0htvvNHlfvWFc3M1uHh+Ym4C0B164u8vV6wAAP1aW1ubqqurZbPZXG3+/v6y2WyqqqryuE1VVZVbvSSlpqa66p1Op7Zt26abbrpJqampGj58uJKTk1VaWnrFvrS2tqq5udltAQD0DwQrAEC/1tjYqPb2dkVGRrq1R0ZGyuFweNzG4XBcsb6hoUHffPONVq1apbS0NP3nf/6n7r//fj3wwAN67733OuxLfn6+QkNDXUtUVJTJ0QEA+gqCFQAAXnI6nZKk++67TwsWLFB8fLwWLlyo//N//o/rVkFP8vLy1NTU5FqOHj3aW10GAPSwQF93AACAnhQREaGAgADV19e7tdfX18tqtXrcxmq1XrE+IiJCgYGBGj9+vFvNuHHj9OGHH3bYl+DgYAUHB3dlGACAPo4rVgCAfi0oKEgJCQmqqKhwtTmdTlVUVCglJcXjNikpKW71klReXu6qDwoK0m233aZDhw651fzxj3/U9ddf380jAABcDbhiBQDo9+x2u7KyspSYmKikpCQVFhaqpaVF2dnZkqRZs2Zp1KhRys/PlyTNnz9fU6ZM0erVq5Wenq6SkhLt3btX69atc+0zNzdX06ZN0w9+8APdddddKisr0+9+9zvt3LnTF0MEAPgYwQoA0O9NmzZNJ06c0NKlS+VwOBQfH6+ysjLXCyrq6urk73/hJo7Jkydr48aNWrx4sRYtWqSxY8eqtLRUEydOdNXcf//9Ki4uVn5+vp544gndfPPN+vd//3fdcccdvT4+AIDv8TlWcOFzrAD0BP7+doxz0zl8jhWA7sbnWAEAAABAH0SwAgAAAACTuhSs1q5dq+joaFksFiUnJ2vPnj1XrN+8ebNiYmJksVgUGxur7du3d1j7+OOPy8/PT4WFhW7tJ0+e1MyZMxUSEqKwsDDNnj1b33zzTVe6DwAAAADdyutgtWnTJtntdi1btkw1NTWKi4tTamqqGhoaPNZXVlZqxowZmj17tmpra5WRkaGMjAwdOHDgsto333xTu3bt0siRIy9bN3PmTH3yyScqLy/X1q1b9f7772vu3Lnedh8AAAAAup3XwaqgoEBz5sxRdna2xo8fr+LiYg0ePFjr16/3WL9mzRqlpaUpNzdX48aN08qVKzVp0iQVFRW51R07dkzz5s3Thg0bNGjQILd1n376qcrKyvTP//zPSk5O1h133KGXXnpJJSUlOn78uLdDAAAAAIBu5VWwamtrU3V1tWw224Ud+PvLZrOpqqrK4zZVVVVu9ZKUmprqVu90OvXoo48qNzdXEyZM8LiPsLAwJSYmutpsNpv8/f21e/duj8dtbW1Vc3Oz2wIAAAAAPcGrYNXY2Kj29nbX536cFxkZKYfD4XEbh8PxF+ufe+45BQYG6oknnuhwH8OHD3drCwwM1LBhwzo8bn5+vkJDQ11LVFTUXxwfAAAAAHSFz98KWF1drTVr1ui1116Tn59ft+03Ly9PTU1NruXo0aPdtm8AAAAAuJhXwSoiIkIBAQGqr693a6+vr5fVavW4jdVqvWL9Bx98oIaGBo0ePVqBgYEKDAzUF198oZ/97GeKjo527ePSl2OcPXtWJ0+e7PC4wcHBCgkJcVsAAAAAoCd4FayCgoKUkJCgiooKV5vT6VRFRYVSUlI8bpOSkuJWL0nl5eWu+kcffVT79+/Xvn37XMvIkSOVm5urt99+27WPU6dOqbq62rWPHTt2yOl0Kjk52ZshAAAAAEC3C/R2A7vdrqysLCUmJiopKUmFhYVqaWlRdna2JGnWrFkaNWqU8vPzJUnz58/XlClTtHr1aqWnp6ukpER79+7VunXrJEnh4eEKDw93O8agQYNktVp18803S5LGjRuntLQ0zZkzR8XFxTpz5oxycnI0ffp0j69mBwAAAIDe5HWwmjZtmk6cOKGlS5fK4XAoPj5eZWVlrhdU1NXVyd//woWwyZMna+PGjVq8eLEWLVqksWPHqrS0VBMnTvTquBs2bFBOTo7uvvtu+fv768EHH9SLL77obfcBAAAAoNv5GYZh+LoTvaG5uVmhoaFqamrieasORC/c5vb9kVXpPuoJgP6Ev78d49x0zsXzE3MTgO7QE39/vb5ihYHp0tB1MSY5AAAADHQ+f906AAAAAFztCFYAAAAAYBLBCgAAAABMIlgBAAAAgEkEKwAAAAAwiWAFAAAAACYRrAAAAADAJIIVAAAAAJhEsAIAAAAAkwhWAAAAAGASwQoAAAAATCJYAQAAAIBJBCsAAAAAMIlgBQAAAAAmEawAAAAAwCSCFQAAAACYRLACAAAAAJMIVgAAAABgEsEKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJgX6ugPoedELt3W47siq9F7sCQAAANA/ccUKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAAAAMAkghUAAAAAmESwAgAAAACTCFYAAAAAYBLBCgAAAABM6lKwWrt2raKjo2WxWJScnKw9e/ZcsX7z5s2KiYmRxWJRbGystm/f7rZ++fLliomJ0ZAhQ3TdddfJZrNp9+7dbjXR0dHy8/NzW1atWtWV7gMAAABAt/I6WG3atEl2u13Lli1TTU2N4uLilJqaqoaGBo/1lZWVmjFjhmbPnq3a2lplZGQoIyNDBw4ccNXcdNNNKioq0scff6wPP/xQ0dHRuueee3TixAm3fT399NP68ssvXcu8efO87T4AAAAAdDuvg1VBQYHmzJmj7OxsjR8/XsXFxRo8eLDWr1/vsX7NmjVKS0tTbm6uxo0bp5UrV2rSpEkqKipy1Tz88MOy2Wz67ne/qwkTJqigoEDNzc3av3+/276GDh0qq9XqWoYMGeJt9wEAAACg23kVrNra2lRdXS2bzXZhB/7+stlsqqqq8rhNVVWVW70kpaamdljf1tamdevWKTQ0VHFxcW7rVq1apfDwcN166616/vnndfbs2Q772traqubmZrcFAAAAAHqCV8GqsbFR7e3tioyMdGuPjIyUw+HwuI3D4ehU/datW3XttdfKYrHohRdeUHl5uSIiIlzrn3jiCZWUlOjdd9/Vj3/8Y/3yl7/Uz3/+8w77mp+fr9DQUNcSFRXlzVABAP1Mdz8f/Nhjj1327G9aWlpPDgEA0If1mbcC3nXXXdq3b58qKyuVlpamhx56yO25LbvdrjvvvFO33HKLHn/8ca1evVovvfSSWltbPe4vLy9PTU1NruXo0aO9NRQAQB/TE88HS1JaWprbs7+/+c1vemM4AIA+yKtgFRERoYCAANXX17u119fXy2q1etzGarV2qn7IkCG68cYbdfvtt+uVV15RYGCgXnnllQ77kpycrLNnz+rIkSMe1wcHByskJMRtAQAMTD3xfLB0bq65+Nnf6667rjeGgw5EL9zmWgCgt3kVrIKCgpSQkKCKigpXm9PpVEVFhVJSUjxuk5KS4lYvSeXl5R3WX7zfjq5GSdK+ffvk7++v4cOHezECAMBA05PPB+/cuVPDhw/XzTffrJ/85Cf66quvrtgXnv8FgP4r0NsN7Ha7srKylJiYqKSkJBUWFqqlpUXZ2dmSpFmzZmnUqFHKz8+XJM2fP19TpkzR6tWrlZ6erpKSEu3du1fr1q2TJLW0tOjZZ5/V1KlTNWLECDU2Nmrt2rU6duyYMjMzJZ2b4Hbv3q277rpLQ4cOVVVVlRYsWKBHHnmEfx0EAFzRlZ4PPnjwoMdtOvN8cFpamh544AGNGTNGn332mRYtWqS//uu/VlVVlQICAjzuNz8/XytWrDA5IgBAX+R1sJo2bZpOnDihpUuXyuFwKD4+XmVlZa4JqK6uTv7+Fy6ETZ48WRs3btTixYu1aNEijR07VqWlpZo4caIkKSAgQAcPHtTrr7+uxsZGhYeH67bbbtMHH3ygCRMmSDp3q0VJSYmWL1+u1tZWjRkzRgsWLJDdbu+OcwAAgNemT5/u+jo2Nla33HKLbrjhBu3cuVN33323x23y8vLc5q7m5mZergQA/YTXwUqScnJylJOT43Hdzp07L2vLzMx0XX26lMVi0ZYtW654vEmTJmnXrl1e9xMAgJ58Pvhi3/3udxUREaE//elPHQar4OBgBQcHezkCAMDVoM+8FRAAgJ7QW88H/8///I+++uorjRgxons6DgC4qnTpihX6D96cBGAg6O7ng7/55hutWLFCDz74oKxWqz777DP9/Oc/14033qjU1FSfjRMA4DsEKwBAv9cTzwfv379fr7/+uk6dOqWRI0fqnnvu0cqVK7nVDwAGKIIVAGBA6M7ng6+55hq9/fbb3dk9AMBVjmesAAAAAMAkghUAAAAAmESwAgAAAACTCFYAAAAAYBLBCgAAAABMIlgBAAAAgEkEKwAAAAAwiWAFAAAAACbxAcH9SPTCbX16fwAAAEB/xRUrAAAAADCJYAUAAAAAJhGsAAAAAMAkghUAAAAAmESwAgAAAACTCFYAAAAAYBLBCgAAAABMIlgBAAAAgEkEKwAAAAAwiWAFAAAAACYRrAAAAADAJIIVAAAAAJhEsAIAAAAAkwhWAAAAAGASwQoAAAAATCJYAQAAAIBJBCsAAAAAMIlgBQAAAAAmEawAAAAAwCSCFQAAAACY1KVgtXbtWkVHR8tisSg5OVl79uy5Yv3mzZsVExMji8Wi2NhYbd++3W398uXLFRMToyFDhui6666TzWbT7t273WpOnjypmTNnKiQkRGFhYZo9e7a++eabrnQfAAAAALqV18Fq06ZNstvtWrZsmWpqahQXF6fU1FQ1NDR4rK+srNSMGTM0e/Zs1dbWKiMjQxkZGTpw4ICr5qabblJRUZE+/vhjffjhh4qOjtY999yjEydOuGpmzpypTz75ROXl5dq6davef/99zZ07twtDBgAAAIDu5WcYhuHNBsnJybrttttUVFQkSXI6nYqKitK8efO0cOHCy+qnTZumlpYWbd261dV2++23Kz4+XsXFxR6P0dzcrNDQUL3zzju6++679emnn2r8+PH66KOPlJiYKEkqKyvTvffeq//5n//RyJEj/2K/z++zqalJISEh3gz5qhG9cJtPjntkVbpPjgvg6jAQ/v52Feemcy6e364053S2DgB64u+vV1es2traVF1dLZvNdmEH/v6y2WyqqqryuE1VVZVbvSSlpqZ2WN/W1qZ169YpNDRUcXFxrn2EhYW5QpUk2Ww2+fv7X3bL4Hmtra1qbm52WwAAAACgJ3gVrBobG9Xe3q7IyEi39sjISDkcDo/bOByOTtVv3bpV1157rSwWi1544QWVl5crIiLCtY/hw4e71QcGBmrYsGEdHjc/P1+hoaGuJSoqypuhAgAAAECn9Zm3At51113at2+fKisrlZaWpoceeqjD57Y6Iy8vT01NTa7l6NGj3dhbAAAAALjAq2AVERGhgIAA1dfXu7XX19fLarV63MZqtXaqfsiQIbrxxht1++2365VXXlFgYKBeeeUV1z4uDVlnz57VyZMnOzxucHCwQkJC3BYAAAAA6AleBaugoCAlJCSooqLC1eZ0OlVRUaGUlBSP26SkpLjVS1J5eXmH9Rfvt7W11bWPU6dOqbq62rV+x44dcjqdSk5O9mYIAAAAANDtAr3dwG63KysrS4mJiUpKSlJhYaFaWlqUnZ0tSZo1a5ZGjRql/Px8SdL8+fM1ZcoUrV69Wunp6SopKdHevXu1bt06SVJLS4ueffZZTZ06VSNGjFBjY6PWrl2rY8eOKTMzU5I0btw4paWlac6cOSouLtaZM2eUk5Oj6dOnd+qNgAAAAADQk7wOVtOmTdOJEye0dOlSORwOxcfHq6yszPWCirq6Ovn7X7gQNnnyZG3cuFGLFy/WokWLNHbsWJWWlmrixImSpICAAB08eFCvv/66GhsbFR4erttuu00ffPCBJkyY4NrPhg0blJOTo7vvvlv+/v568MEH9eKLL5odPwAAAACY5nWwkqScnBzl5OR4XLdz587L2jIzM11Xny5lsVi0ZcuWv3jMYcOGaePGjV71EwAAAAB6Q595KyAAAAAAXK0IVgAAAABgEsEKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAAAAMAkghUAAAAAmESwAgAAAACTCFYAAAAAYBLBCgAwIKxdu1bR0dGyWCxKTk7Wnj17rli/efNmxcTEyGKxKDY2Vtu3b++w9vHHH5efn58KCwu7udcAgKsFwQoA0O9t2rRJdrtdy5YtU01NjeLi4pSamqqGhgaP9ZWVlZoxY4Zmz56t2tpaZWRkKCMjQwcOHLis9s0339SuXbs0cuTInh4GAKAPI1gBAPq9goICzZkzR9nZ2Ro/fryKi4s1ePBgrV+/3mP9mjVrlJaWptzcXI0bN04rV67UpEmTVFRU5FZ37NgxzZs3Txs2bNCgQYN6YygDQvTCbW4LAFwNCFYAgH6tra1N1dXVstlsrjZ/f3/ZbDZVVVV53KaqqsqtXpJSU1Pd6p1Opx599FHl5uZqwoQJnepLa2urmpub3RYAQP9AsAIA9GuNjY1qb29XZGSkW3tkZKQcDofHbRwOx1+sf+655xQYGKgnnnii033Jz89XaGioa4mKivJiJACAvoxgBQCAl6qrq7VmzRq99tpr8vPz6/R2eXl5ampqci1Hjx7twV4CAHoTwQoA0K9FREQoICBA9fX1bu319fWyWq0et7FarVes/+CDD9TQ0KDRo0crMDBQgYGB+uKLL/Szn/1M0dHRHfYlODhYISEhbgsAoH8gWAEA+rWgoCAlJCSooqLC1eZ0OlVRUaGUlBSP26SkpLjVS1J5ebmr/tFHH9X+/fu1b98+1zJy5Ejl5ubq7bff7rnBAAD6rEBfdwAAgJ5mt9uVlZWlxMREJSUlqbCwUC0tLcrOzpYkzZo1S6NGjVJ+fr4kaf78+ZoyZYpWr16t9PR0lZSUaO/evVq3bp0kKTw8XOHh4W7HGDRokKxWq26++ebeHRwAoE8gWAEA+r1p06bpxIkTWrp0qRwOh+Lj41VWVuZ6QUVdXZ38/S/cxDF58mRt3LhRixcv1qJFizR27FiVlpZq4sSJvhoCAKCPI1gBAAaEnJwc5eTkeFy3c+fOy9oyMzOVmZnZ6f0fOXKkiz0DAPQHPGMFAAAAACYRrAAAAADAJIIVAAAAAJhEsAIAAAAAkwhWAAAAAGASwQoAAAAATCJYAQAAAIBJBCsAAAAAMIlgBQAAAAAmEawAAAAAwCSCFQAAAACYRLACAAAAAJO6FKzWrl2r6OhoWSwWJScna8+ePVes37x5s2JiYmSxWBQbG6vt27e71p05c0ZPPfWUYmNjNWTIEI0cOVKzZs3S8ePH3fYRHR0tPz8/t2XVqlVd6T4AAAAAdCuvg9WmTZtkt9u1bNky1dTUKC4uTqmpqWpoaPBYX1lZqRkzZmj27Nmqra1VRkaGMjIydODAAUnSt99+q5qaGi1ZskQ1NTXasmWLDh06pKlTp162r6efflpffvmla5k3b5633QcAAACAbhfo7QYFBQWaM2eOsrOzJUnFxcXatm2b1q9fr4ULF15Wv2bNGqWlpSk3N1eStHLlSpWXl6uoqEjFxcUKDQ1VeXm52zZFRUVKSkpSXV2dRo8e7WofOnSorFart13u16IXbvN1FwAAAIABz6srVm1tbaqurpbNZruwA39/2Ww2VVVVedymqqrKrV6SUlNTO6yXpKamJvn5+SksLMytfdWqVQoPD9ett96q559/XmfPnu1wH62trWpubnZbAAAAAKAneHXFqrGxUe3t7YqMjHRrj4yM1MGDBz1u43A4PNY7HA6P9adPn9ZTTz2lGTNmKCQkxNX+xBNPaNKkSRo2bJgqKyuVl5enL7/8UgUFBR73k5+frxUrVngzPAAAAADoEq9vBexJZ86c0UMPPSTDMPTyyy+7rbPb7a6vb7nlFgUFBenHP/6x8vPzFRwcfNm+8vLy3LZpbm5WVFRUz3UeAAAAwIDlVbCKiIhQQECA6uvr3drr6+s7fPbJarV2qv58qPriiy+0Y8cOt6tVniQnJ+vs2bM6cuSIbr755svWBwcHewxcAAAAANDdvHrGKigoSAkJCaqoqHC1OZ1OVVRUKCUlxeM2KSkpbvWSVF5e7lZ/PlQdPnxY77zzjsLDw/9iX/bt2yd/f38NHz7cmyEAAAAAQLfz+lZAu92urKwsJSYmKikpSYWFhWppaXG9JXDWrFkaNWqU8vPzJUnz58/XlClTtHr1aqWnp6ukpER79+7VunXrJJ0LVT/60Y9UU1OjrVu3qr293fX81bBhwxQUFKSqqirt3r1bd911l4YOHaqqqiotWLBAjzzyiK677rruOhcAAAAA0CVeB6tp06bpxIkTWrp0qRwOh+Lj41VWVuZ6QUVdXZ38/S9cCJs8ebI2btyoxYsXa9GiRRo7dqxKS0s1ceJESdKxY8f01ltvSZLi4+PdjvXuu+/qzjvvVHBwsEpKSrR8+XK1trZqzJgxWrBggdszVAAAAADgK36GYRi+7kRvaG5uVmhoqJqamv7i81tXk77wOVZHVqX7ugsA+rD++ve3O3BuPOvs3Hbp/HPxdsxNAK6kJ/7+evWMFQAAAADgcgQrAAAAADCJYAUAAAAAJhGsAAAAAMAkghUAAAAAmESwAgAAAACTCFYAAAAAYBLBCgAAAABMIlgBAAAAgEkEKwAAAAAwiWAFAAAAACYRrAAAAADAJIIVAAAAAJgU6OsO4OoXvXCb6+sjq9J92BMAAADAN7hiBQAAAAAmEawAAAAAwCSCFQAAAACYRLACAAAAAJMIVgAAAABgEsEKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAAAAMCkQF93AN6JXrjN110AAAAAcAmuWAEAAACASQQrAMCAsHbtWkVHR8tisSg5OVl79uy5Yv3mzZsVExMji8Wi2NhYbd++3W398uXLFRMToyFDhui6666TzWbT7t27e3IIAIA+jGAFAOj3Nm3aJLvdrmXLlqmmpkZxcXFKTU1VQ0ODx/rKykrNmDFDs2fPVm1trTIyMpSRkaEDBw64am666SYVFRXp448/1ocffqjo6Gjdc889OnHiRG8NCwDQhxCsAAD9XkFBgebMmaPs7GyNHz9excXFGjx4sNavX++xfs2aNUpLS1Nubq7GjRunlStXatKkSSoqKnLVPPzww7LZbPrud7+rCRMmqKCgQM3Nzdq/f39vDQsA0Ifw8goAQL/W1tam6upq5eXludr8/f1ls9lUVVXlcZuqqirZ7Xa3ttTUVJWWlnZ4jHXr1ik0NFRxcXEd9qW1tVWtra2u75ubm70YCS7V2Rc6XVp3ZFW6qWN1ZXsA/R9XrAAA/VpjY6Pa29sVGRnp1h4ZGSmHw+FxG4fD0an6rVu36tprr5XFYtELL7yg8vJyRUREdNiX/Px8hYaGupaoqKgujgoA0NcQrAAA6KK77rpL+/btU2VlpdLS0vTQQw91+NyWJOXl5ampqcm1HD16tBd7CwDoSQQrAEC/FhERoYCAANXX17u119fXy2q1etzGarV2qn7IkCG68cYbdfvtt+uVV15RYGCgXnnllQ77EhwcrJCQELcFANA/dClYdecra8+cOaOnnnpKsbGxGjJkiEaOHKlZs2bp+PHjbvs4efKkZs6cqZCQEIWFhWn27Nn65ptvutJ9AMAAEhQUpISEBFVUVLjanE6nKioqlJKS4nGblJQUt3pJKi8v77D+4v1e/AwVAGDg8DpYdfcra7/99lvV1NRoyZIlqqmp0ZYtW3To0CFNnTrVbT8zZ87UJ598ovLycm3dulXvv/++5s6d24UhAwAGGrvdrl//+td6/fXX9emnn+onP/mJWlpalJ2dLUmaNWuW28st5s+fr7KyMq1evVoHDx7U8uXLtXfvXuXk5EiSWlpatGjRIu3atUtffPGFqqur9bd/+7c6duyYMjMzfTJGAIBvef1WwItfWStJxcXF2rZtm9avX6+FCxdeVn/xK2slaeXKlSovL1dRUZGKi4sVGhqq8vJyt22KioqUlJSkuro6jR49Wp9++qnKysr00UcfKTExUZL00ksv6d5779U//uM/auTIkV4PHAAwcEybNk0nTpzQ0qVL5XA4FB8fr7KyMtcLKurq6uTvf+HfGidPnqyNGzdq8eLFWrRokcaOHavS0lJNnDhRkhQQEKCDBw/q9ddfV2Njo8LDw3Xbbbfpgw8+0IQJE3wyRgCAb3kVrHrjlbWS1NTUJD8/P4WFhbn2ERYW5gpVkmSz2eTv76/du3fr/vvvv2wfvNIWAHCxnJwc1xWnS+3cufOytszMzA6vPlksFm3ZsqU7uwcAuMp5dStgT76y9rzTp0/rqaee0owZM1wP9TocDg0fPtytLjAwUMOGDetwP7zSFgAAAEBv6VNvBTxz5oweeughGYahl19+2dS+eKUtAAAAgN7i1a2APfnK2vOh6osvvtCOHTvcXkFrtVoveznG2bNndfLkyQ6PGxwcrODg4E6PDQAAAAC6yqsrVj31ytrzoerw4cN65513FB4eftk+Tp06perqalfbjh075HQ6lZyc7M0QAAAAAKDbef1WQLvdrqysLCUmJiopKUmFhYWXvbJ21KhRys/Pl3TulbVTpkzR6tWrlZ6erpKSEu3du1fr1q2TdC5U/ehHP1JNTY22bt2q9vZ213NTw4YNU1BQkMaNG6e0tDTNmTNHxcXFOnPmjHJycjR9+nTeCAgAAADA57wOVt39ytpjx47prbfekiTFx8e7Hevdd9/VnXfeKUnasGGDcnJydPfdd8vf318PPvigXnzxxa6MGQAAAAC6ldfBSureV9ZGR0fLMIy/eMxhw4Zp48aNXvUTAAAAAHpDn3orIAAAAABcjQhWAAAAAGBSl24FRO+KXrjN113AVe7S36Ejq9J91BMAAID+iStWAAAAAGASwQoAAAAATCJYAQAAAIBJBCsAAAAAMIlgBQAAAAAmEawAAAAAwCSCFQAAAACYRLACAAAAAJMIVgAAAABgEsEKAAAAAEwK9HUHAPRN0Qu3uX1/ZFW6x3UXtwMAAAxUXLECAAAAAJMIVgAAAABgErcCAgAAn+jJ24ovvZ25t46L7sMt6bjacMUKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAAAAMAkghUAAAAAmESwAgAAAACT+BwrdCs+VwIAAAADEVesAAAAAMAkghUAAAAAmESwAgAAAACTCFYAAAAAYBLBCgAAAABMIlgBAAAAgEkEKwAAAAAwiWAFAAAAACZ1KVitXbtW0dHRslgsSk5O1p49e65Yv3nzZsXExMhisSg2Nlbbt293W79lyxbdc889Cg8Pl5+fn/bt23fZPu688075+fm5LY8//nhXug8AAAAA3crrYLVp0ybZ7XYtW7ZMNTU1iouLU2pqqhoaGjzWV1ZWasaMGZo9e7Zqa2uVkZGhjIwMHThwwFXT0tKiO+64Q88999wVjz1nzhx9+eWXruVXv/qVt90HAAAAgG4X6O0GBQUFmjNnjrKzsyVJxcXF2rZtm9avX6+FCxdeVr9mzRqlpaUpNzdXkrRy5UqVl5erqKhIxcXFkqRHH31UknTkyJErHnvw4MGyWq3edhl9QPTCbW7fH1mV7qOeQHL/eXT3z6KjfV/6O3Axfh8AAMDVzqsrVm1tbaqurpbNZruwA39/2Ww2VVVVedymqqrKrV6SUlNTO6y/kg0bNigiIkITJ05UXl6evv322w5rW1tb1dzc7LYAAAAAQE/w6opVY2Oj2tvbFRkZ6dYeGRmpgwcPetzG4XB4rHc4HF519OGHH9b111+vkSNHav/+/Xrqqad06NAhbdmyxWN9fn6+VqxY4dUxAAAAAKArvL4V0Ffmzp3r+jo2NlYjRozQ3Xffrc8++0w33HDDZfV5eXmy2+2u75ubmxUVFdUrfQUAAAAwsHgVrCIiIhQQEKD6+nq39vr6+g6ffbJarV7Vd1ZycrIk6U9/+pPHYBUcHKzg4GBTxwAAAACAzvDqGaugoCAlJCSooqLC1eZ0OlVRUaGUlBSP26SkpLjVS1J5eXmH9Z11/pXsI0aMMLUfAAAAADDL61sB7Xa7srKylJiYqKSkJBUWFqqlpcX1lsBZs2Zp1KhRys/PlyTNnz9fU6ZM0erVq5Wenq6SkhLt3btX69atc+3z5MmTqqur0/HjxyVJhw4dknTuapfVatVnn32mjRs36t5771V4eLj279+vBQsW6Ac/+IFuueUW0ycBAAAAAMzwOlhNmzZNJ06c0NKlS+VwOBQfH6+ysjLXCyrq6urk73/hQtjkyZO1ceNGLV68WIsWLdLYsWNVWlqqiRMnumreeustVzCTpOnTp0uSli1bpuXLlysoKEjvvPOOK8RFRUXpwQcf1OLFi7s8cAAAAADoLl5/QLAk5eTk6IsvvlBra6t2797tet5Jknbu3KnXXnvNrT4zM1OHDh1Sa2urDhw4oHvvvddt/WOPPSbDMC5bli9fLkmKiorSe++9p6+++kqnT5/W4cOH9atf/UohISFd6T4AYABau3atoqOjZbFYlJycrD179lyxfvPmzYqJiZHFYlFsbKy2b9/uWnfmzBk99dRTio2N1ZAhQzRy5EjNmjXLdecFAGDg6VKwAgDgarJp0ybZ7XYtW7ZMNTU1iouLU2pqqhoaGjzWV1ZWasaMGZo9e7Zqa2uVkZGhjIwMHThwQJL07bffqqamRkuWLFFNTY22bNmiQ4cOaerUqb05LABAH0KwAgD0ewUFBZozZ46ys7M1fvx4FRcXa/DgwVq/fr3H+jVr1igtLU25ubkaN26cVq5cqUmTJqmoqEiSFBoaqvLycj300EO6+eabdfvtt6uoqEjV1dWqq6vrzaEBAPqIq+ZzrAD0jOiF21xfH1mV7sOeXO7ivkkd96+zdRiY2traVF1drby8PFebv7+/bDabqqqqPG5TVVXl9lmIkpSamqrS0tIOj9PU1CQ/Pz+FhYV1WNPa2qrW1lbX983NzZ0bBACgzyNYocdc+n92AcAXGhsb1d7e7nrJ0nmRkZE6ePCgx20cDofHeofD4bH+9OnTeuqppzRjxowrPv+bn5+vFStWeDmCq1tn//HGV3PGlY7LP9JcnTr6neMf4dDTuBUQAAATzpw5o4ceekiGYejll1++Ym1eXp6amppcy9GjR3uplwCAnsYVKwBAvxYREaGAgADV19e7tdfX18tqtXrcxmq1dqr+fKj64osvtGPHjr/4ttrg4GAFBwd3YRQAgL6OK1YAgH4tKChICQkJqqiocLU5nU5VVFQoJSXF4zYpKSlu9ZJUXl7uVn8+VB0+fFjvvPOOwsPDe2YAAICrAlesAAD9nt1uV1ZWlhITE5WUlOT6wPnzH04/a9YsjRo1Svn5+ZKk+fPna8qUKVq9erXS09NVUlKivXv3at26dZLOhaof/ehHqqmp0datW9Xe3u56/mrYsGEKCgryzUABAD5DsAIA9HvTpk3TiRMntHTpUjkcDsXHx6usrMz1goq6ujr5+1+4iWPy5MnauHGjFi9erEWLFmns2LEqLS3VxIkTJUnHjh3TW2+9JUmKj493O9a7776rO++8s1fGBQDoOwhWAIABIScnRzk5OR7X7dy587K2zMxMZWZmeqyPjo6WYRjd2T0AwFWOZ6wAAAAAwCSCFQAAAACYRLACAAAAAJMIVgAAAABgEi+v6KOiF27zdRfgQ5f+/I+sSu9wXVfquqNPPbUNAADA1YgrVgAAAABgEsEKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAAAAMAkghUAAAAAmESwAgAAAACTCFYAAAAAYBLBCgAAAABMCvR1BzAwRS/c5vr6yKp0H/akZ3R2fH3tPFzcH18dty+cBwAAAG9xxQoAAAAATCJYAQAAAIBJBCsAAAAAMIlgBQAAAAAmEawAAAAAwCSCFQAAAACYRLACAAAAAJO69DlWa9eu1fPPPy+Hw6G4uDi99NJLSkpK6rB+8+bNWrJkiY4cOaKxY8fqueee07333utav2XLFhUXF6u6ulonT55UbW2t4uPj3fZx+vRp/exnP1NJSYlaW1uVmpqqf/qnf1JkZGRXhgAAAHqZrz4rr7tdOo6B9vl73f1ZjV2p66zObnOlPnS0ri/+HnTH50Ly2ZJd5/UVq02bNslut2vZsmWqqalRXFycUlNT1dDQ4LG+srJSM2bM0OzZs1VbW6uMjAxlZGTowIEDrpqWlhbdcccdeu655zo87oIFC/S73/1Omzdv1nvvvafjx4/rgQce8Lb7AAAAANDtvA5WBQUFmjNnjrKzszV+/HgVFxdr8ODBWr9+vcf6NWvWKC0tTbm5uRo3bpxWrlypSZMmqaioyFXz6KOPaunSpbLZbB730dTUpFdeeUUFBQX64Q9/qISEBL366quqrKzUrl27vB0CAAAAAHQrr24FbGtrU3V1tfLy8lxt/v7+stlsqqqq8rhNVVWV7Ha7W1tqaqpKS0s7fdzq6mqdOXPGLXjFxMRo9OjRqqqq0u23337ZNq2trWptbXV939zc3OnjAV1h9haXq/UWme7ud2+eB253AAAA3cWrK1aNjY1qb2+/7LmmyMhIORwOj9s4HA6v6jvaR1BQkMLCwjq9n/z8fIWGhrqWqKioTh8PAAAAALzRb98KmJeXp6amJtdy9OhRX3cJAAAAQD/l1a2AERERCggIUH19vVt7fX29rFarx22sVqtX9R3to62tTadOnXK7anWl/QQHBys4OLjTxwAAAACArvLqilVQUJASEhJUUVHhanM6naqoqFBKSorHbVJSUtzqJam8vLzDek8SEhI0aNAgt/0cOnRIdXV1Xu0HAAAAAHqC159jZbfblZWVpcTERCUlJamwsFAtLS3Kzs6WJM2aNUujRo1Sfn6+JGn+/PmaMmWKVq9erfT0dJWUlGjv3r1at26da58nT55UXV2djh8/LulcaJLOXamyWq0KDQ3V7NmzZbfbNWzYMIWEhGjevHlKSUnx+OIKAAAAAOhNXgeradOm6cSJE1q6dKkcDofi4+NVVlbmekFFXV2d/P0vXAibPHmyNm7cqMWLF2vRokUaO3asSktLNXHiRFfNW2+95QpmkjR9+nRJ0rJly7R8+XJJ0gsvvCB/f389+OCDbh8QDAAAAAC+5nWwkqScnBzl5OR4XLdz587L2jIzM5WZmdnh/h577DE99thjVzymxWLR2rVrtXbtWm+6CgAAAAA9rt++FRAAAAAAegvBCgAAAABMIlgBAAAAgEldesYKQN8UvXCbr7vQp3F+AABAT+GKFQAAAACYRLACAAAAAJMIVgAAAABgEsEKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAAAAMAkghUAAAAAmESwAgAAAACTAn3dAaC7RS/c5vr6yKr0XjlOb2yHzuvJ34HO7ru3fg8BAEDfwBUrAAAAADCJYAUAGBDWrl2r6OhoWSwWJScna8+ePVes37x5s2JiYmSxWBQbG6vt27e7rd+yZYvuuecehYeHy8/PT/v27evB3gMA+jqCFQCg39u0aZPsdruWLVummpoaxcXFKTU1VQ0NDR7rKysrNWPGDM2ePVu1tbXKyMhQRkaGDhw44KppaWnRHXfcoeeee663hgEA6MMIVgCAfq+goEBz5sxRdna2xo8fr+LiYg0ePFjr16/3WL9mzRqlpaUpNzdX48aN08qVKzVp0iQVFRW5ah599FEtXbpUNputt4YBAOjDCFYAgH6tra1N1dXVbgHI399fNptNVVVVHrepqqq6LDClpqZ2WN9Zra2tam5udlsAAP0DbwUEAPRrjY2Nam9vV2RkpFt7ZGSkDh486HEbh8Phsd7hcJjqS35+vlasWGFqH+g9Zt/ieqXte/Oton3hbbmdPZddOefd/Zbe7nh7b1fOeVeP25X9d/X3gDfeXhlXrAAA6CV5eXlqampyLUePHvV1lwAA3YQrVgCAfi0iIkIBAQGqr693a6+vr5fVavW4jdVq9aq+s4KDgxUcHGxqHwCAvokrVgCAfi0oKEgJCQmqqKhwtTmdTlVUVCglJcXjNikpKW71klReXt5hPQAAXLECAPR7drtdWVlZSkxMVFJSkgoLC9XS0qLs7GxJ0qxZszRq1Cjl5+dLkubPn68pU6Zo9erVSk9PV0lJifbu3at169a59nny5EnV1dXp+PHjkqRDhw5JOne1y+yVLQDA1YdgBZ/rCw9C9mQfuuMhWJzTl89lX+4bpGnTpunEiRNaunSpHA6H4uPjVVZW5npBRV1dnfz9L9zEMXnyZG3cuFGLFy/WokWLNHbsWJWWlmrixImumrfeessVzCRp+vTpkqRly5Zp+fLlvTMwAECfQbACAAwIOTk5ysnJ8bhu586dl7VlZmYqMzOzw/099thjeuyxx7qpdwCAqx3PWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAAAAMAkghUAAAAAmESwAgAAAACTuhSs1q5dq+joaFksFiUnJ2vPnj1XrN+8ebNiYmJksVgUGxur7du3u603DENLly7ViBEjdM0118hms+nw4cNuNdHR0fLz83NbVq1a1ZXuAwAAAEC38jpYbdq0SXa7XcuWLVNNTY3i4uKUmpqqhoYGj/WVlZWaMWOGZs+erdraWmVkZCgjI0MHDhxw1fzqV7/Siy++qOLiYu3evVtDhgxRamqqTp8+7bavp59+Wl9++aVrmTdvnrfdBwAAAIBu53WwKigo0Jw5c5Sdna3x48eruLhYgwcP1vr16z3Wr1mzRmlpacrNzdW4ceO0cuVKTZo0SUVFRZLOXa0qLCzU4sWLdd999+mWW27Rv/zLv+j48eMqLS1129fQoUNltVpdy5AhQ7wfMQAAAAB0s0Bvitva2lRdXa28vDxXm7+/v2w2m6qqqjxuU1VVJbvd7taWmprqCk2ff/65HA6HbDaba31oaKiSk5NVVVWl6dOnu9pXrVqllStXavTo0Xr44Ye1YMECBQZ6HkJra6taW1td3zc3N3szVPRD0Qu3uX1/ZFW619t1dpvudmnf0XmdPXd94ecMAACuXl4Fq8bGRrW3tysyMtKtPTIyUgcPHvS4jcPh8FjvcDhc68+3dVQjSU888YQmTZqkYcOGqbKyUnl5efryyy9VUFDg8bj5+flasWKFN8MDAAAAgC7xKlj50sVXvW655RYFBQXpxz/+sfLz8xUcHHxZfV5ents2zc3NioqK6pW+AgAAABhYvHrGKiIiQgEBAaqvr3drr6+vl9Vq9biN1Wq9Yv35//Vmn5KUnJyss2fP6siRIx7XBwcHKyQkxG0BAAAAgJ7gVbAKCgpSQkKCKioqXG1Op1MVFRVKSUnxuE1KSopbvSSVl5e76seMGSOr1epW09zcrN27d3e4T0nat2+f/P39NXz4cG+GAAAAAADdzutbAe12u7KyspSYmKikpCQVFhaqpaVF2dnZkqRZs2Zp1KhRys/PlyTNnz9fU6ZM0erVq5Wenq6SkhLt3btX69atkyT5+fnpySef1DPPPKOxY8dqzJgxWrJkiUaOHKmMjAxJ516AsXv3bt11110aOnSoqqqqtGDBAj3yyCO67rrruulUAAAAAEDXeB2spk2bphMnTmjp0qVyOByKj49XWVmZ6+UTdXV18ve/cCFs8uTJ2rhxoxYvXqxFixZp7NixKi0t1cSJE101P//5z9XS0qK5c+fq1KlTuuOOO1RWViaLxSLp3G19JSUlWr58uVpbWzVmzBgtWLDgsrcNAgAAAIAvdOnlFTk5OcrJyfG4bufOnZe1ZWZmKjMzs8P9+fn56emnn9bTTz/tcf2kSZO0a9eurnQVAAAAAHqc1x8QDAAAAABwR7ACAAAAAJMIVgAAAABg0lXzAcFXk+iF21xfH1mV7vU2A9ml5+Hi83elc9TZ83ylY3XmOMDFeut35Ur/XXT3/rt73wAADBRcsQIAAAAAkwhWAAAAAGAStwICAACPevP27KuZ2duCr7R9T57n7rideaDdPt+Vn9VAO0dSx7eYd/ZvytX6N4QrVgAAAABgEsEKAAAAAEwiWAEAAACASQQrAAAAADCJYAUAAAAAJhGsAAAAAMAkghUAAAAAmMTnWKFfGIifEYGrU3d8Jk1/+KwPAAD6G4KVDxEG/rKe/NBFoDP4HQIAAJ3BrYAAAAAAYBLBCgAAAABMIlgBAAAAgEkEKwAAAAAwiWAFAAAAACYRrAAAAADAJIIVAAAAAJhEsAIAAAAAkwhWAAAAAGASwQoAAAAATAr0dQeuJtELt7l9f2RVutfb4OrGz3Ng6O6fc2f3d3FdZ/6+eLNNV+ou1tn+AAAwUHHFCgAAAABMIlgBAAAAgEkEKwAAAAAwiWAFAAAAACYRrAAAAADAJIIVAAAAAJjUpWC1du1aRUdHy2KxKDk5WXv27Lli/ebNmxUTEyOLxaLY2Fht377dbb1hGFq6dKlGjBiha665RjabTYcPH3arOXnypGbOnKmQkBCFhYVp9uzZ+uabb7rSfQDAAOSLuQsAMHB4Haw2bdoku92uZcuWqaamRnFxcUpNTVVDQ4PH+srKSs2YMUOzZ89WbW2tMjIylJGRoQMHDrhqfvWrX+nFF19UcXGxdu/erSFDhig1NVWnT5921cycOVOffPKJysvLtXXrVr3//vuaO3duF4YMABhofDV3AQAGDq+DVUFBgebMmaPs7GyNHz9excXFGjx4sNavX++xfs2aNUpLS1Nubq7GjRunlStXatKkSSoqKpJ07l/8CgsLtXjxYt1333265ZZb9C//8i86fvy4SktLJUmffvqpysrK9M///M9KTk7WHXfcoZdeekklJSU6fvx410cPABgQfDF3AQAGlkBvitva2lRdXa28vDxXm7+/v2w2m6qqqjxuU1VVJbvd7taWmprqmng+//xzORwO2Ww21/rQ0FAlJyerqqpK06dPV1VVlcLCwpSYmOiqsdls8vf31+7du3X//fdfdtzW1la1tra6vm9qapIkNTc3ezNkN87Wb92+72hfl9YBgLcu/vvS2b8pV/r7dvE+Lq3rzP7N/O08v61hGF3ehxm+mrs86Ym5qSddad670u9NV35/B5ruOEed/b3p6L9/fjb9Q0c/0yv9re+O+aIr++js35Te+JvYE3OTV8GqsbFR7e3tioyMdGuPjIzUwYMHPW7jcDg81jscDtf6821Xqhk+fLh7xwMDNWzYMFfNpfLz87VixYrL2qOiojoantdCC7ttVwDgpit/Xzq7TU/u+0q+/vprhYaGmt+Rl3w1d3nSG3NTT+rJ37GBpjvOka/+W0bf0tHP9Eo/6+74b7kn99Gbv6fdOTd5FayuJnl5eW7/2uh0OnXy5EmFh4fLz8/P1d7c3KyoqCgdPXpUISEhvuhqn8G5uIBzcQ7n4QLOxQXengvDMPT1119r5MiRvdC7vq2zc1Nn8Xt5AefiHM7DBZyLCzgXF5w/F3V1dfLz8+vWucmrYBUREaGAgADV19e7tdfX18tqtXrcxmq1XrH+/P/W19drxIgRbjXx8fGumksfMD579qxOnjzZ4XGDg4MVHBzs1hYWFtbh2EJCQgb8L9p5nIsLOBfncB4u4Fxc4M258MWVqvN8NXd54u3c1Fn8Xl7AuTiH83AB5+ICzsUFoaGh3X4uvHp5RVBQkBISElRRUeFqczqdqqioUEpKisdtUlJS3Oolqby83FU/ZswYWa1Wt5rm5mbt3r3bVZOSkqJTp06purraVbNjxw45nU4lJyd7MwQAwADjq7kLADCweH0roN1uV1ZWlhITE5WUlKTCwkK1tLQoOztbkjRr1iyNGjVK+fn5kqT58+drypQpWr16tdLT01VSUqK9e/dq3bp1kiQ/Pz89+eSTeuaZZzR27FiNGTNGS5Ys0ciRI5WRkSFJGjdunNLS0jRnzhwVFxfrzJkzysnJ0fTp07m1BADwF/li7gIADDBGF7z00kvG6NGjjaCgICMpKcnYtWuXa92UKVOMrKwst/rf/va3xk033WQEBQUZEyZMMLZt2+a23ul0GkuWLDEiIyON4OBg4+677zYOHTrkVvPVV18ZM2bMMK699lojJCTEyM7ONr7++uuudN/N6dOnjWXLlhmnT582va+rHefiAs7FOZyHCzgXF1yt58IXc1dPu1p/Fj2Bc3EO5+ECzsUFnIsLevJc+BmGj95/CwAAAAD9hNcfEAwAAAAAcEewAgAAAACTCFYAAAAAYBLBCgAAAABMGvDBau3atYqOjpbFYlFycrL27Nnj6y71qPz8fN12220aOnSohg8froyMDB06dMit5vTp0/rpT3+q8PBwXXvttXrwwQcv+6DM/mjVqlWuVyifN5DOxbFjx/TII48oPDxc11xzjWJjY7V3717XesMwtHTpUo0YMULXXHONbDabDh8+7MMed7/29nYtWbJEY8aM0TXXXKMbbrhBK1eu1MXv+Omv5+H999/X3/zN32jkyJHy8/NTaWmp2/rOjPvkyZOaOXOmQkJCFBYWptmzZ+ubb77pxVEMLANt/pKYwzrC/MX8JTGH9Yk5rNvfM3gVKSkpMYKCgoz169cbn3zyiTFnzhwjLCzMqK+v93XXekxqaqrx6quvGgcOHDD27dtn3Hvvvcbo0aONb775xlXz+OOPG1FRUUZFRYWxd+9e4/bbbzcmT57sw173vD179hjR0dHGLbfcYsyfP9/VPlDOxcmTJ43rr7/eeOyxx4zdu3cb//3f/228/fbbxp/+9CdXzapVq4zQ0FCjtLTU+MMf/mBMnTrVGDNmjPHnP//Zhz3vXs8++6wRHh5ubN261fj888+NzZs3G9dee62xZs0aV01/PQ/bt283fvGLXxhbtmwxJBlvvvmm2/rOjDstLc2Ii4szdu3aZXzwwQfGjTfeaMyYMaOXRzIwDMT5yzCYwzxh/mL+Oo85zPdz2IAOVklJScZPf/pT1/ft7e3GyJEjjfz8fB/2qnc1NDQYkoz33nvPMAzDOHXqlDFo0CBj8+bNrppPP/3UkGRUVVX5qps96uuvvzbGjh1rlJeXG1OmTHFNTAPpXDz11FPGHXfc0eF6p9NpWK1W4/nnn3e1nTp1yggODjZ+85vf9EYXe0V6errxt3/7t25tDzzwgDFz5kzDMAbOebh0UurMuP/rv/7LkGR89NFHrprf//73hp+fn3Hs2LFe6/tAwfx1zkCfw5i/mL8uxhx2ji/nsAF7K2BbW5uqq6tls9lcbf7+/rLZbKqqqvJhz3pXU1OTJGnYsGGSpOrqap05c8btvMTExGj06NH99rz89Kc/VXp6utuYpYF1Lt566y0lJiYqMzNTw4cP16233qpf//rXrvWff/65HA6H27kIDQ1VcnJyvzoXkydPVkVFhf74xz9Kkv7whz/oww8/1F//9V9LGjjn4VKdGXdVVZXCwsKUmJjoqrHZbPL399fu3bt7vc/9GfPXBQN9DmP+Yv66GHOYZ705hwV2X7evLo2NjWpvb1dkZKRbe2RkpA4ePOijXvUup9OpJ598Ut/73vc0ceJESZLD4VBQUJDCwsLcaiMjI+VwOHzQy55VUlKimpoaffTRR5etG0jn4r//+7/18ssvy263a9GiRfroo4/0xBNPKCgoSFlZWa7xevrvpT+di4ULF6q5uVkxMTEKCAhQe3u7nn32Wc2cOVOSBsx5uFRnxu1wODR8+HC39YGBgRo2bFi/Pje+wPx1zkCfw5i/zmH+uoA5zLPenMMGbLDCuX/pOnDggD788ENfd8Unjh49qvnz56u8vFwWi8XX3fEpp9OpxMRE/fKXv5Qk3XrrrTpw4ICKi4uVlZXl4971nt/+9rfasGGDNm7cqAkTJmjfvn168sknNXLkyAF1HoCrwUCew5i/LmD+uoA5zPcG7K2AERERCggIuOwNOfX19bJarT7qVe/JycnR1q1b9e677+o73/mOq91qtaqtrU2nTp1yq++P56W6uloNDQ2aNGmSAgMDFRgYqPfee08vvviiAgMDFRkZOWDOxYgRIzR+/Hi3tnHjxqmurk6SXOPt7/+95ObmauHChZo+fbpiY2P16KOPasGCBcrPz5c0cM7DpTozbqvVqoaGBrf1Z8+e1cmTJ/v1ufGFgT5/ScxhzF8XMH9dwBzmWW/OYQM2WAUFBSkhIUEVFRWuNqfTqYqKCqWkpPiwZz3LMAzl5OTozTff1I4dOzRmzBi39QkJCRo0aJDbeTl06JDq6ur63Xm5++679fHHH2vfvn2uJTExUTNnznR9PVDOxfe+973LXln8xz/+Uddff70kacyYMbJarW7norm5Wbt37+5X5+Lbb7+Vv7/7n8WAgAA5nU5JA+c8XKoz405JSdGpU6dUXV3tqtmxY4ecTqeSk5N7vc/92UCdvyTmsPOYvy5g/rqAOcyzXp3DzL5542pWUlJiBAcHG6+99prxX//1X8bcuXONsLAww+Fw+LprPeYnP/mJERoaauzcudP48ssvXcu3337rqnn88ceN0aNHGzt27DD27t1rpKSkGCkpKT7sde+5+K1KhjFwzsWePXuMwMBA49lnnzUOHz5sbNiwwRg8eLDxr//6r66aVatWGWFhYcZ//Md/GPv37zfuu+++fvGK1otlZWUZo0aNcr2qdsuWLUZERITx85//3FXTX8/D119/bdTW1hq1tbWGJKOgoMCora01vvjiC8MwOjfutLQ049ZbbzV2795tfPjhh8bYsWN53XoPGYjzl2Ewh10J89fAnr8MgzmsL8xhAzpYGYZhvPTSS8bo0aONoKAgIykpydi1a5evu9SjJHlcXn31VVfNn//8Z+Pv//7vjeuuu84YPHiwcf/99xtffvml7zrdiy6dmAbSufjd735nTJw40QgODjZiYmKMdevWua13Op3GkiVLjMjISCM4ONi4++67jUOHDvmotz2jubnZmD9/vjF69GjDYrEY3/3ud41f/OIXRmtrq6umv56Hd9991+PfhqysLMMwOjfur776ypgxY4Zx7bXXGiEhIUZ2drbx9ddf+2A0A8NAm78MgznsSpi/Bvb8ZRjMYX1hDvMzjIs+jhkAAAAA4LUB+4wVAAAAAHQXghUAAAAAmESwAgAAAACTCFYAAAAAYBLBCgAAAABMIlgBAAAAgEkEKwAAAAAwiWAFAAAAACYRrAAAAADAJIIVAAAAAJhEsAIAAAAAkwhWQB924sQJWa1W/fKXv3S1VVZWKigoSBUVFT7sGQAAAC7mZxiG4etOAOjY9u3blZGRocrKSt18882Kj4/Xfffdp4KCAl93DQAAAP+LYAVcBX7605/qnXfeUWJioj7++GN99NFHCg4O9nW3AAAA8L8IVsBV4M9//rMmTpyoo0ePqrq6WrGxsb7uEgAAAC7CM1bAVeCzzz7T8ePH5XQ6deTIEV93BwAAAJfgihXQx7W1tSkpKUnx8fG6+eabVVhYqI8//ljDhw/3ddcAAADwvwhWQB+Xm5urf/u3f9Mf/vAHXXvttZoyZYpCQ0O1detWX3cNAAAA/4tbAYE+bOfOnSosLNQbb7yhkJAQ+fv764033tAHH3ygl19+2dfdAwAAwP/iihUAAAAAmMQVKwAAAAAwiWAFAAAAACYRrAAAAADAJIIVAAAAAJhEsAIAAAAAkwhWAAAAAGASwQoAAAAATCJYAQAAAIBJBCsAAAAAMIlgBQAAAAAmEawAAAAAwKT/D2rKKfQsZ0nTAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "BINS = 100\n",
    "COUNT = 1000\n",
    "\n",
    "_, _, train_data, test_data = generate_1d_data(COUNT, BINS)\n",
    "plot_1d_data(train_data, test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6967f47",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How we can do it?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af0a3250",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MLE and histogram\n",
    "\n",
    "Let $X_1, X_2, \\dots X_N \\sim \\text{Categorical}(\\theta_1, \\theta_2, \\dots \\theta_K)$ , where $\\sum\\limits_{i = 1}^{K} \\theta_i = 1$. Denote $\\boldsymbol{\\theta} = (\\theta_1, \\dots, \\theta_K)$\n",
    "\n",
    "**Problem 2**: Find the MLE estimator $\\hat{\\boldsymbol{\\theta}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18f86965",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let $X$ has $n_i$ elements equal to $i$, $\\sum\\limits_{i=1}^{K}n_i = N$. Loglikelihood: \n",
    "\n",
    "$\\log p(X|\\theta) =\\sum\\limits_{i=1}^{K} n_i \\log p(k=i|\\theta)$\n",
    "\n",
    "**Take a deep breath and work on this step by step**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5597857e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Deriving MLE Step-by-Step\n",
    "\n",
    "To find the MLE, we need to maximize the log-likelihood with respect to $\\theta_i$, given the constraint $\\sum\\limits_{i=1}^{K} \\theta_i = 1$.\n",
    "\n",
    "Define the Lagrangian function:\n",
    "\n",
    "$$\n",
    "L(\\theta, \\lambda) = \\sum\\limits_{i=1}^{K} n_i \\log \\theta_i + \\lambda \\left( \\sum\\limits_{i=1}^{K} \\theta_i - 1 \\right)\n",
    "$$\n",
    "\n",
    "Take the derivative with respect to $\\theta_i$:\n",
    "\n",
    "$$\n",
    "\\frac{\\partial L}{\\partial \\theta_i} = \\frac{n_i}{\\theta_i} + \\lambda = 0\n",
    "$$\n",
    "\n",
    "Solve for $\\theta_i$:\n",
    "\n",
    "$$\n",
    "\\theta_i = -\\frac{n_i}{\\lambda}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2348532e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Applying the Constraint\n",
    "\n",
    "We know that $\\sum\\limits_{i=1}^{K} \\theta_i = 1$, so we substitute:\n",
    "\n",
    "$$\n",
    "\\sum\\limits_{i=1}^{K} \\left(-\\frac{n_i}{\\lambda}\\right) = 1\n",
    "$$\n",
    "\n",
    "Simplifying gives:\n",
    "\n",
    "$$\n",
    "-\\frac{N}{\\lambda} = 1 \\implies \\lambda = -N\n",
    "$$\n",
    "\n",
    "Substitute $\\lambda$ back to find $\\theta_i$:\n",
    "\n",
    "$$\n",
    "\\theta_i = \\frac{n_i}{N}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2230d8a4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Histogram\n",
    "\n",
    "$\\hat{\\theta_k} = \\dfrac{1}{N} \\sum\\limits_{j = 1}^{N} I(X_j = k) = \\dfrac{n_i}{N}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3ab1ad3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Histogram\n",
    "\n",
    "$$\\hat{\\theta_k} = \\dfrac{1}{N} \\sum\\limits_{j = 1}^{N} I(X_j = k) = \\dfrac{n_i}{N}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87066e67",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Random fact\n",
    "\n",
    "How can we avoid using the Lagrange multiplier method? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f23ccd32",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For example $ p(k|\\theta) = \\theta = \\dfrac{e^{\\theta'_k}}{\\sum\\limits_{i=1}^{K}e^{\\theta'_i}}$ aka **Softmax function**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d831d894",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Random fact 2\n",
    "\n",
    "Let $X$ has $n_i$ elements equal to $i$, $\\sum\\limits_{i=1}^{K}n_i = N$. Loglikelihood: \n",
    "\n",
    "$$\\log p(X|\\theta) =\\sum\\limits_{i=1}^{K} n_i \\log p(k=i|\\theta) = \\sum\\limits_{j = 1}^{N} \\sum\\limits_{i = 1}^{K}I(X_j = i) \\log p(k=i|\\theta) = \\sum\\limits_{j = 1}^{N} \\sum\\limits_{i = 1}^{K} I(X_j = i) \\log p_{j}^{i}$$\n",
    "\n",
    "Do you know something like that in PyTorch?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7cd6d34",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Classic ML setup**\n",
    "\n",
    "For batch of $\\{X_j\\}_{j=1}^{N}$ elements and predicted probabilities $\\{(p^1_j, \\dots , p^K_j)\\}_{j=1}^{N}$:\n",
    "\n",
    "$$\\text{CrossEntropyLoss} = - \\sum\\limits_{j=1}^{N} x_j \\log p_{j}^{x_j} = - \\sum\\limits_{j = 1}^{N} \\sum\\limits_{i = 1}^{K}I(X_j = i) \\log p_{j}^{i}$$\n",
    "\n",
    "\n",
    "Optimizing **CrossEntropyLoss is equivalent to the MLE estimator** for categorical data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b232ec3f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Histogram in 2D\n",
    "\n",
    "**Question.** Can we consider binarized MNIST to be sampled from a Categorical distribution?\n",
    "\n",
    "<!-- <center><img src=\"data:image/png;base64,\" width=1200 /></center> -->\n",
    "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA4QAAAOECAYAAAD5Tv87AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAG4ZJREFUeJzt3c1u20gaQFHKyFbovWC//4MZ0AO09uKsGtMzsJiYlvh3z1l2wzFpsCjfVPDVaRzHcQAAACDnbe0LAAAAYB2CEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIOrX2hfAz9zv9+F6vQ7n83k4nU5rXw683DiOw+12Gy6Xy/D21vw7Leuemvq6t+apqa/5pQnCnbter8PHx8falwGL+/z8HN7f39e+jFVY91RV1701T1V1zS9Ncu/c+Xxe+xJgFeVnv3zvtFWf/ep9g2d/GYJw5/zTEarKz3753mmrPvvV+wbP/jIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKJ+rX0BsHXjOM76utPp9OQrAQCA57JDCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACDKsRPwG1PHR0wdSTHnuApHVQAAsCQ7hAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiHDsBPzD3SArHSwAAsAV2CAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABEOXYCVvDoSArHUQDA700d7fSIz1j4mh1CAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgyZRQAgFXMmRa6pe9lcilHYIcQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJRjJ2BDpkZiG20N2zdnrP3U2p47Jt/7Ar5n7jr0uc0R2CEEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAEOXYCdgQI6qpm3vMwpQ56+oV17GF7wU/ddTn9RX3teTPyu8P/IQdQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRjp0AYFFLj60/6pj8KY/u2Wh6fmrqGSquNTgCO4QAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAohw7ASsw+h14Je8YXmXJoyW28hw7ToOjs0MIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGmjMIKHk0s28pENVjL1BrY+6S/R/e29/uC39n7un729c/9rJ/zvZb++fo9Zp/sEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIcuwE/MAexmXD1swdS37UceZzR9of9efB8Rz5s3LJdTjney15xAX7ZYcQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJRjJ+A3lhy9PPd7GT8P+2W8O3sy5/Nm7jPuMxGWYYcQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJRjJ2BYfuz7o5HYS47mNpYbluV4CWAvpn5H8C47HjuEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIcO0HK1kclv2LMs+MlYL+sX/g+RzH9ma3/TsRy7BACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQpoxzOVqZmPXtiWXECGgD794oJ2s82dR0+f/+Xn8fx2CEEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAEOXYCQAANmcPx1VsxZI/D0d0HI8dQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRjp3gcJYcU228MvBvex6Fb5Q8a5j7bC35Wb/ndT2XNd9ihxAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlGMnOJzieGhgOUc9vmYr1wE/9ehZ9vvB/7Lm+YcdQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIMmUUfsMULgDYv6nP86kJpHN/DzjqRGKOxw4hAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgCjHTrBLRjkDr/Tsdwywba/4PcDvFuyFHUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUY6dYJeMcgZeaeodM/dICu8tALbIDiEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARP1a+wIAYGvGcZz1dafT6clXAgCvZYcQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABE/Vr7AviZcRzXvgRYRfnZL9/7Uv7++++1L4EvVJ/96n2DZ38ZgnDnbrfb2pcAq7jdbsNff/219mWswrp/veqztXXVdW/NU1Vd80s7jdJ71+73+3C9Xofz+TycTqe1LwdebhzH4Xa7DZfLZXh7a/6rd+uemvq6t+apqa/5pQlCAACAKMkNAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAEDUr7UvgJ+53+/D9XodzufzcDqd1r4ceLlxHIfb7TZcLpfh7a35d1rWPTX1dW/NU1Nf80sThDt3vV6Hj4+PtS8DFvf5+Tm8v7+vfRmrsO6pqq57a56q6ppfmuTeufP5vPYlwCrKz3753mmrPvvV+wbP/jIE4c75pyNUlZ/98r3TVn32q/cNnv1lCEIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABE/Vr7AgAAYKvGcXz4/06n04JXAq9hhxAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlGMnYEMejbY21hoAgFewQwgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgyrETsLBHR0sAAOuY+9k89XWOjGIv7BACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiHLsBADsjFH38FxT68ZxURydHUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiDJlFF7ARDLgp+a+R0wgha/5bIav2SEEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAEOXYCQAO69lj5l9xbINR+LBfjnLhCOwQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIhy7AQAu1Y8tmFq1H3x5wF/Yu66cbQER2eHEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUYydgpleMdjfaGpaz5/VmRD58nyNZ4Gt2CAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABEOXYCFmYkPMxjZPx/eY/AY89+V8z986xT9sIOIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABEmTIKM01NDzMNEeZ59tpZcsqfdQ/825x3gsmkrMEOIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAox07ATEbMw/Pt+TiXPV87AF12CAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABEOXYCfmPOuPip8fPAcqbW75xjIrZytMTc7+XdxFE8e73NXRuOlOEI7BACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiHLsBAzPHxs9d9Q91G396AYj5mEbtnIEzBx+D2Br7BACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiHLsBBlLjqE2Uhrm2fMo+Slz7st7BB7b8/vA0VRsjR1CAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgyZRRmMgkMlrXkmnv2BENTBWF91hp8zQ4hAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgCjHTsBvGFMN/JT3CKzP8S/wNTuEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIcO8GhTI2UBnglY+th27ayRrdyHfAPO4QAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAohw7wS45XgLYmqn3kjHz8FxTa+rRWvS7A3zNDiEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARJkyCoMJgADAc/ndgr2wQwgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgyrETZBj/DPypR++LcRy//TXAcVn3HIEdQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRjp0AgCdwJAVsg/UG32OHEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUYyfYJSOlgVeaOkICAI7EDiEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKMdOAMD/mXO0zdRRFVP/zzE6AKzJDiEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARJkyCgBPYFooAHtkhxAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUId24cx7UvAVZRfvbL905b9dmv3jd49pchCHfudrutfQmwivKzX7532qrPfvW+wbO/jNMovXftfr8P1+t1OJ/Pw+l0Wvty4OXGcRxut9twuVyGt7fm32lZ99TU1701T019zS9NEAIAAERJbgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAECUIAQAAon6tfQH8zP1+H67X63A+n4fT6bT25cDLjeM43G634XK5DG9vzb/Tsu6pqa97a56a+ppfmiDcuev1Onx8fKx9GbC4z8/P4f39fe3LWIV1T1V13VvzVFXX/NIk986dz+e1LwFWUX72y/dOW/XZr943ePaXIQh3zj8doar87Jfvnbbqs1+9b/DsL0MQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIgShAAAAFG/1r4A2LpxHL/876fT6dtfM9fU94K6Z6+3pVnfAKzJDiEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARJkyCsO8KYVLTjac+71ML4TtW/Jd4p0Az2PtchR2CAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABEOXaCzVpynDNwTFOj2r1jgH9s/X3g+CleyQ4hAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgCjHTrCqrY95BrZv7lj1Jcexb+Vd9+g6jKbnKLay1rbCmudP2CEEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgChBCAAAEOXYCVY1NfbY6GjgKOaMePcOhK8V14ZjInglO4QAAABRghAAACBKEAIAAEQJQgAAgChBCAAAEGXKKJs1ZwLp3Clce55YZvIYHFPxfQZrefZnqXXIntghBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABDl2Al2ac546K2MgHZMBPAnpt5Z3iMUPPtz27qBr9khBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABDl2AkypsZNb+VICuCYnv2OmfvnGbvP0e35Gd/ztbNvdggBAACiBCEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARDl2AhY2NS7eyGlYzpy1uIcjarxH2JMlj2R59tpw/AtHYYcQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKJMGYVheuLXklMFH30vE8lgnrnrdwvTRK17+L6tTBKd82da86zFDiEAAECUIAQAAIgShAAAAFGCEAAAIEoQAgAARAlCAACAKMdOwG/MGQP97DHVU3+eMdWwfdYpbNsWjpqBtdghBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABDl2Al4gbkj5o29hmNydAwsw+cofJ8dQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIEoQAAABRjp0A4LCefaTD3JH2jpYAYKvsEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCmjsHNTUw9NNoRtmDOd1PqlYOo5nzvVdwvfa+51wBrsEAIAAEQJQgAAgChBCAAAECUIAQAAogQhAABAlCAEAACIcuwE7Jzx1fB8RtDD+pZcG4++15LvAliLHUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUY6d4OXmjmze8yh2Y6phOVPrbeo94mgJALBDCAAAkCUIAQAAogQhAABAlCAEAACIEoQAAABRghAAACDKsRNs1pyR8HNHu88ZW+9oCXi+Z68rR0sAP7GVo2vglewQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUKaO83JITul4x8csUMeqOugZMBQXW8Oid6p3EWuwQAgAARAlCAACAKEEIAAAQJQgBAACiBCEAAECUIAQAAIhy7ATsgFHUYB0Ay3vFsTveZWyNHUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUY6dYFVzRy+/Ygz0UoybhsesD2BLpt5Jc38XefR13n+sxQ4hAABAlCAEAACIEoQAAABRghAAACBKEAIAAEQJQgAAgCjHTrBLRjPDcqw3ADguO4QAAABRghAAACBKEAIAAEQJQgAAgChBCAAAEGXKKAAAfJMJzByFHUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQg3LlxHNe+BFhF+dkv3ztt1We/et/g2V+GINy52+229iXAKsrPfvneaas++9X7Bs/+Mk6j9N61+/0+XK/X4Xw+D6fTae3LgZcbx3G43W7D5XIZ3t6af6dl3VNTX/fWPDX1Nb80QQgAABAluQEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABAlCAEAAKIEIQAAQJQgBAAAiBKEAAAAUYIQAAAgShACAABECUIAAIAoQQgAABD1H+HqBYMYsE2AAAAAAElFTkSuQmCC\" width=400 /></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3af2252",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Question.** What the problem with histogram model for approximating binarized MNIST (or arbitrary images dataset)?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac98fba5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Histogram for binarized Images\n",
    "\n",
    "**Question**: How many parameters are needed to describe the probability distribution of a binarized image??"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58ff71a1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Independent Pixel Encoding** - $N \\times N$ assuming each pixel is encoded independently.\n",
    "\n",
    "**Autoregressive Pixel Encoding** - $2^N - 1$ when using an autoregressive order, where each pixel depends on the previous ones."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c288b2b6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Bayes' theorem\n",
    "\n",
    "$P(A|B) = \\dfrac{P(B|A)P(A)}{P(B)}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd3e8f60",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$f(\\theta|x) = \\dfrac{f(x|\\theta)g(\\theta)}{\\int f(x|\\theta)g(\\theta) d\\theta}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26f53fdb",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$\\hat{\\theta}_{ML} = \\text{argmax}_\\theta p(X|\\theta)$\n",
    "\n",
    "$\\hat{\\theta}_{MAP} = \\text{argmax}_\\theta p(\\theta|X)$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "524406be",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Recap\n",
    "\n",
    "0. The main goal of generative modeling (when training a model) is to estimate the parameters of a chosen parametric distribution from a sample.\n",
    "1. Reviewed what MLE (Maximum Likelihood Estimation) is, where it comes from, and how it is derived:\n",
    "   - Write down the likelihood function and maximize it.\n",
    "   - Use the method of Lagrange multipliers to handle constraints.\n",
    "2. The idea of *reparameterization* to remove constraints — using SoftMax to avoid dealing with the Lagrangian.\n",
    "3. Minimizing CrossEntropy is equivalent to maximizing MLE for categorical distributions.\n",
    "4. Histogram model:\n",
    "   - Default approach: \"take the mean in the bin\" = MLE estimation.\n",
    "   - For a good approximation with a large $N$, a lot of samples are needed.\n",
    "   - \"Naive histogram\" - all pixels are independent: requires $N \\times N$ parameters.\n",
    "   - Autoregressive order requires $2^N - 1$ parameters.\n",
    "5. Bayes' theorem — recalled it and noted that MAP (Maximum A Posteriori) estimations also exist."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "toc-autonumbering": false,
  "toc-showcode": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}