seminars/seminar1/seminar1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "union-alert",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from PIL import Image\n",
    "import io\n",
    "import itertools\n",
    "\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "amber-humidity",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <center>Deep Generative Models</center>\n",
    "## <center>Seminar 1</center>\n",
    "\n",
    "<center><img src=\"pics/ozonmasterslogo.png\" width=600 height=600 /></center>\n",
    "<center>10.02.2021</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cardiac-shame",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Kondratenko Vladimir\n",
    "\n",
    "\n",
    "tg: @username27\n",
    "\n",
    "e-mail: vladimir.kondratenko@phystech.edu"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "virgin-offer",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Plan\n",
    "\n",
    "- Maximum Likelihood Estimation\n",
    "- Example of a simple 1D/2D distribution\n",
    "- MADE\n",
    "- Masked Linear Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aquatic-version",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Maximum Likelihood Estimation (MLE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "entitled-entrance",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What do we want?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affecting-mineral",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We have samples $X \\sim \\pi(x)$. We assume what $\\pi(x) \\approx p(x|\\theta)$. So, let's get $\\theta$ estimation using $X$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "false-appliance",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How?\n",
    "\n",
    "$\\theta^* = argmax_{\\theta} p(X | \\theta) = argmax_{\\theta} \\prod_{i=1}^n p(x_i | \\theta) = argmax_{\\theta} \\sum_{i=1}^n \\log p(x_i | \\theta)$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "valuable-handbook",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MLE examlpes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "awful-threshold",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Example 1\n",
    "\n",
    "$x_1, ... x_n \\sim U[0,\\theta]$. \n",
    "\n",
    "$\\hat{\\theta}_{ML} = ?$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "closing-combat",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For a single $x \\sim U[0,\\theta]$:\n",
    "\n",
    "$p(x) = \\dfrac{1}{\\theta}, x \\in [0,\\theta]$\n",
    "\n",
    "$p(x) = 0, x \\notin [0,\\theta]$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "verbal-organ",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We have $x_1, ... x_n \\sim U[0,\\theta]$:\n",
    "\n",
    "$p(X | \\theta) = \\dfrac{1}{\\theta^{n}}, X \\in [0,\\theta]^{n}$, and $0, X \\notin [0,\\theta]^{n}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "delayed-vision",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "It also can be written as:\n",
    "\n",
    "$p(X | \\theta) = \\dfrac{1}{\\theta^{n}}, max(x_1, ... x_n) \\leq \\theta$, and $0, max(x_1, ... x_n) > \\theta$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "owned-shock",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "So, we can find:\n",
    "\n",
    "$\\hat{\\theta}_{ML} = max(x_1, ... x_n)$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "defined-quick",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$\\hat{\\theta}_{ML} = max(x_1, ... x_n)$ is also random variable\n",
    "\n",
    "How does Cumulative distribution function $F(x) = P(x < X)$ looks like?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "objective-victory",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "$P_{X \\sim U[0,\\theta]}(max(x_1, ... x_n) < x) = \\dfrac{x^n}{\\theta^n}$, $x \\in [0,\\theta]$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "frozen-picnic",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "$\\hat{\\theta}$ is **unbiased** estimation of $\\theta$: $\\mathbb{E}\\hat{\\theta} = \\theta$.\n",
    "\n",
    "Are MLEs **unbiased**?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "funded-officer",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's find this $\\mathbb{E}\\hat{\\theta}_{ML} = ?$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worst-hammer",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We know that:\n",
    "\n",
    "$\\mathbb{E}x = \\int_{-\\infty}^{+\\infty} x f(x) dx = \\int_{-\\infty}^{+\\infty} x dF(x)$\n",
    "\n",
    "\n",
    "$P_{X \\sim U[0,\\theta]}(max(x_1, ... x_n) < x) = \\dfrac{x^n}{\\theta^n}$, $x \\in [0,\\theta]$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "physical-establishment",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$\\mathbb{E}_{X \\sim U[0,\\theta]}\\hat{\\theta}_{ML} =\n",
    "\\int_{0}^{\\theta} x d \\dfrac{x^n}{\\theta^n} = ...\n",
    "%\\int_{0}^{\\theta} \\dfrac{xnx^{n-1}}{\\theta^n} d x =\n",
    "%\\int_{0}^{\\theta} \\dfrac{nx^{n}}{\\theta^n} d x =\n",
    "%\\dfrac{n\\theta^{n+1}}{(n+1)\\theta^n}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "threaded-charter",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$\\Rightarrow \\mathbb{E}_{X \\sim U[0,\\theta]}\\hat{\\theta}_{ML} = \\dfrac{n}{n+1}\\theta \\Rightarrow$ MLE is not **unbiased**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "confused-packaging",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### MLE\n",
    "\n",
    "- not unbiased\n",
    "- consistent (состоятельные)\n",
    "- asymptotically efficient (асимптотически эффективные)\n",
    "- asymptotically normal (асимптотически нормальные)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "novel-trash",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Simple distributions\n",
    "### Example 1\n",
    "\n",
    "Let we have samples $X \\sim \\pi(x)$ and want approximate $\\pi(x)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "static-exploration",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def generate_1d_data(count, bins):\n",
    "    np.random.seed(42)\n",
    "    a = 0.2 + 0.05 * np.random.randn(count)\n",
    "    b = 0.6 + 0.15 * np.random.randn(count)\n",
    "    mask = np.random.rand(count) < 0.5\n",
    "    samples = (a * mask + b * (1 - mask)) * (bins - 1)\n",
    "    data = np.clip(samples.astype('int'), 0.0, (bins - 1))\n",
    "    split = int(0.8 * count)\n",
    "    train_data, test_data = data[:split], data[split:]\n",
    "    return train_data, test_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "forty-county",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def plot_1d_data(train_data, test_data):\n",
    "    bins = int(max(test_data.max(), train_data.max()) - min(test_data.min(), train_data.min())) + 1\n",
    "    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))\n",
    "    ax1.set_title('train')\n",
    "    ax1.hist(train_data, bins=bins, density=True)\n",
    "    ax1.set_xlabel('x')\n",
    "    ax2.set_title('test')\n",
    "    ax2.hist(test_data, bins=np.arange(bins), density=True)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "therapeutic-yemen",
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAFNCAYAAACuWnPfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlfklEQVR4nO3df7Ddd13n8efLxKZYoEgIbG0aUqdRTMuANhR0cVeoxVSE4NhOU7u07nSMLmbURdcN7tKp3bq2OztUGLpIpZWSFVusdr3aQAWL44BSk/KrTUvX2xJpAkj6w0JxWwi894/zveVwem7uuT+/58fzMXPmfs/n+/l+8vmem3zyPp/v50eqCkmSJLXnO9qugCRJ0qQzIJMkSWqZAZkkSVLLDMgkSZJaZkAmSZLUMgMySZKklhmQaeQk+b0kb267HpIkLRUDMq24JAeS/PhCr6+qX6yq/7aUdZKk+VpsW9aU8XNJPrJUddLoMiDTUEmyuu06SJK00gzItKKS7AY2AH+e5LEkv5Gkklyc5HPAbU2+P07yxSSPJvmbJKd2lfHuJJc3xz+W5GCSX0vypSRfSPLvW7k5SRNjlrbsZUn+Nsk/J/lUkh/ryv9zSe5P8pUkn01yQZIfAH4P+OGmjH9u5WY0FAzItKKq6vXA54DXVNXTgfc1p/4t8APATzTv3w9sAp4LfBz4w6MU+6+A44ETgYuBq5N899LXXpI6+rRlfwjcAlwOPBv4deBPkqxLchzwNuDsqnoG8CPAJ6vqHuAXgb+rqqdX1bNauBUNCQMyDYtLq+qrVfX/AKrquqr6SlU9AVwKvCjJ8bNc+3Xgsqr6elXtAR4Dvn9Fai1JHf8O2FNVe6rqm1X1QWAf8JPN+W8CpyV5WlV9oar2t1ZTDSUDMg2LB2YOkqxKckWS+5J8GTjQnHrOLNc+VFVHut7/C/D05ammJPX1fODc5nHlPzePH18OnFBVXwXOo9Mb9oUktyR5QYt11RAyIFMbao60nwW2AT9O51HkxiY9y1stSZqX7nbrAWB3VT2r63VcVV0BUFW3VtVZwAnAZ4Df71OGJpgBmdrwT8D3HuX8M4AngIeA7wL++0pUSpLmqbst+9/Aa5L8RNPLf2wz6Wh9kucl2daMJXuCzrCKb3aVsT7JMStffQ0TAzK14XeA/9p06Z/T5/x7gH8EDgF3Ax9buapJ0sC627Lz6PTs/yZwmE6P2X+i8//sdwBvBD4PPExnEtN/aMq4DdgPfDHJgytZeQ2XVNlbKkmS1CZ7yCRJklpmQCZJktQyAzJJkqSWGZBJkiS1zIBMkiSpZavbrsB8POc5z6mNGze2XQ1JK+SOO+54sKrWtV2PpWD7JU2e+bRhIxWQbdy4kX379rVdDUkrJMk/tl2HpWL7JU2e+bRhPrKUJElqmQGZJElSywzIJEmSWmZAJkmS1DIDMkmSpJYZkEmSJLXMgEySJKllBmSSJEktMyCTJElqmQGZJElSywzIJEmSWjZSe1mOg427bgHgwBWvbrkmkrR4M20a2K5Ji2EPmSRJUssMyCRJklpmQCZJktQyAzJJkqSWGZBJkiS1zIBMkiSpZS57IUlaEt1LYMxwKQxpMPaQSZIktcyATJIkqWUGZJIkSS0zIBsiG3fd0ncMhiRJGm8DBWRJtia5N8l0kl19zq9JcmNz/vYkG3vOb0jyWJJfH7TMcTcTfBmASZKkOQOyJKuAq4Gzgc3A+Uk292S7GHikqk4BrgKu7Dn/FuD98yxTkiRpIgzSQ3YGMF1V91fV14AbgG09ebYB1zfHNwFnJglAktcBnwX2z7NMSZKkiTBIQHYi8EDX+4NNWt88VXUEeBRYm+TpwH8GfmsBZUqSJE2E5R7UfylwVVU9ttACkuxIsi/JvsOHDy9dzSRJkobEIAHZIeCkrvfrm7S+eZKsBo4HHgJeCvyPJAeAXwV+M8nOAcsEoKquqaotVbVl3bp1A1RXkr5loZOSkpyR5JPN61NJfrrrmgNJ7mzO7VvB25E0pgbZOmkvsCnJyXSCpu3Az/bkmQIuAv4OOAe4raoK+NGZDEkuBR6rqrc3QdtcZUrSonRNIDqLztCIvUmmqururmxPTkpKsp3OpKTzgLuALVV1JMkJwKeS/HkzLAPgFVX14MrdjaRxNmcPWdP47ARuBe4B3ldV+5NcluS1TbZr6YwZmwbeCBx1GYvZylz4bUhSXwuelFRV/9IVfB0L1IrUWNJEGmhz8araA+zpSbuk6/hx4Nw5yrh0rjIlaYn1m0D00tnyNL1hjwJrgQeTvBS4Dng+8PquAK2Av0xSwDur6pp+f3iSHcAOgA0bNizNHY2omTUX3Wxc6s+V+iVpFlV1e1WdCrwEeFOSY5tTL6+qH6KzluIvJfk3s1zvGFhJAzEgkzTOFjMp6UlVdQ/wGHBa8/5Q8/NLwM10Ho1K0oIZkEkaZ09OSkpyDJ0JRFM9eWYmJUHXpKTmmtUASZ4PvAA4kOS4JM9o0o8DXkVnAoAkLdhAY8gkaRQ1Y8JmJhCtAq6bmZQE7KuqKTqTknY3k5IephO0Abwc2JXk68A3gTdU1YNJvhe4udmMZDXw3qr6wMremaRxY0AmaawtdFJSVe0GdvdJvx940dLXVNIk85GlJElSy+whkyQtm5nlLiQdnT1kkiRJLTMgkyRJapkBmSRJUsscQzYEHGMhSdJks4dMkiSpZQZkkiRJLfOR5QrxsaQkfXtbeOCKV7dYE2m42EMmSZLUMgMySZKklhmQSZIktcwxZMvIcWOSJGkQ9pBJkiS1zIBMkiSpZQZkkiRJLTMgkyRJatlAAVmSrUnuTTKdZFef82uS3Nicvz3Jxib9jCSfbF6fSvLTXdccSHJnc27fkt2RJEnSiJlzlmWSVcDVwFnAQWBvkqmqursr28XAI1V1SpLtwJXAecBdwJaqOpLkBOBTSf68qo40172iqh5cyhuSJEkaNYP0kJ0BTFfV/VX1NeAGYFtPnm3A9c3xTcCZSVJV/9IVfB0L1FJUWpIkaZwMsg7ZicADXe8PAi+dLU/TG/YosBZ4MMlLgeuA5wOv7wrQCvjLJAW8s6quWfhtjBf3epMkabIs+8KwVXU7cGqSHwCuT/L+qnoceHlVHUryXOCDST5TVX/Te32SHcAOgA0bNix3dSVJklbcII8sDwEndb1f36T1zZNkNXA88FB3hqq6B3gMOK15f6j5+SXgZjqPRp+iqq6pqi1VtWXdunUDVFeSJGm0DBKQ7QU2JTk5yTHAdmCqJ88UcFFzfA5wW1VVc81qgCTPB14AHEhyXJJnNOnHAa+iMwFAkiRp4sz5yLIZE7YTuBVYBVxXVfuTXAbsq6op4Fpgd5Jp4GE6QRvAy4FdSb4OfBN4Q1U9mOR7gZuTzNThvVX1gaW+OUmSpFEw0BiyqtoD7OlJu6Tr+HHg3D7X7QZ290m/H3jRfCsrSfOVZCvwVjpfKN9VVVf0nF8DvAc4nc5Qi/Oq6kCSM4CZyUYBLq2qmwcpU5Lmy5X6JY2trnUUzwY2A+cn2dyT7cl1FIGr6KyjCN9aR/HFwFbgnUlWD1imJM2LAZmkcbYc6ygOUqYkzcuyL3shSS1a8nUUkwxSJjDey/Z0r5coafHsIZOkWVTV7VV1KvAS4E1Jjp3n9S7bI2kgBmSSxtlyrKM4SJmSNC8GZJLG2ZKvozhgmZI0L44hkzS2lmMdRYB+Za7ojUkaOwZkksbaUq+jOFuZkrQYPrKUJElqmQGZJElSywzIJEmSWmZAJkmS1DIDMkmSpJYZkEmSJLXMgEySJKllBmSSJEktMyCTJElqmQGZJElSywzIJEmSWmZAJkmS1DIDMkmSpJYNFJAl2Zrk3iTTSXb1Ob8myY3N+duTbGzSz0jyyeb1qSQ/PWiZkiRJk2LOgCzJKuBq4GxgM3B+ks092S4GHqmqU4CrgCub9LuALVX1YmAr8M4kqwcsU5IkaSKsHiDPGcB0Vd0PkOQGYBtwd1eebcClzfFNwNuTpKr+pSvPsUDNo0xJ0pDZuOuWtqsgjaVBHlmeCDzQ9f5gk9Y3T1UdAR4F1gIkeWmS/cCdwC825wcpU5IkaSIs+6D+qrq9qk4FXgK8Kcmx87k+yY4k+5LsO3z48PJUcoht3HWL30glSRpzgwRkh4CTut6vb9L65kmyGjgeeKg7Q1XdAzwGnDZgmTPXXVNVW6pqy7p16waoriRJ0mgZJCDbC2xKcnKSY4DtwFRPningoub4HOC2qqrmmtUASZ4PvAA4MGCZkiRJE2HOQf1VdSTJTuBWYBVwXVXtT3IZsK+qpoBrgd1JpoGH6QRYAC8HdiX5OvBN4A1V9SBAvzKX+N4kSZJGwiCzLKmqPcCenrRLuo4fB87tc91uYPegZUqSJE0iV+qXJElqmQGZJElSywzIJEmSWmZAJmmsLWIv3rOS3JHkzubnK7uu+eumzJm9ep+7grckaQwNNKhfkkZR1765Z9HZEWRvkqmq6t6m7cm9eJNsp7MX73nAg8BrqurzSU6jMyu8e0eRC6pq34rciKSxZw+ZpHH25L65VfU1YGbf3G7bgOub45uAM5u9eD9RVZ9v0vcDT0uyZkVqLWniGJBJGmeL2ou3y88AH6+qJ7rS/qB5XPnmJOn3h0/61m9zcWs46VsMyCTpKJKcSucx5i90JV9QVS8EfrR5vb7ftW79JmlQBmSSxtmi9uJNsh64Gbiwqu6buaCqDjU/vwK8l86jUUlaMAMySeNsMXvxPgu4BdhVVR+dyZxkdZLnNMffCfwUcNfy3oakcWdAJmlsNWPCZvbNvQd438xevEle22S7Fljb7MX7RmBmaYydwCnAJT3LW6wBbk3yaeCTdHrYfn/FbkrSWHLZC0ljbRF78V4OXD5LsacvZR0lyR4ySZKkltlDJkk6quVemmKm/ANXvHpZ/xxpmNlDJkmS1DIDMkmSpJYZkEmSJLXMgEySJKllBmSSJEktMyCTJElqmQGZJElSywYKyJJsTXJvkukku/qcX5Pkxub87Uk2NulnJbkjyZ3Nz1d2XfPXTZndW5JIkiRNnDkXhk2yCrgaOAs4COxNMlVVd3dluxh4pKpOSbIduBI4D3gQeE1VfT7JaXT2kzux67oLqmrfEt3L0FjORRS7y3YRRUmSxsMgPWRnANNVdX9VfQ24AdjWk2cbcH1zfBNwZpJU1Seq6vNN+n7gaUnWLEXFJUmSxsUgAdmJwANd7w/y7b1c35anqo4AjwJre/L8DPDxqnqiK+0PmseVb06SedVckiRpTKzIoP4kp9J5jPkLXckXVNULgR9tXq+f5dodSfYl2Xf48OHlr6wkSdIKGyQgOwSc1PV+fZPWN0+S1cDxwEPN+/XAzcCFVXXfzAVVdaj5+RXgvXQejT5FVV1TVVuqasu6desGuSdJkqSRMkhAthfYlOTkJMcA24GpnjxTwEXN8TnAbVVVSZ4F3ALsqqqPzmROsjrJc5rj7wR+CrhrUXciSZI0ouacZVlVR5LspDNDchVwXVXtT3IZsK+qpoBrgd1JpoGH6QRtADuBU4BLklzSpL0K+CpwaxOMrQI+BPz+Et7X2FnOmZuSJKldcwZkAFW1B9jTk3ZJ1/HjwLl9rrscuHyWYk8fvJqSJEnjy5X6JUmSWmZAJkmS1DIDMkmSpJYNNIZMc3PQvSRJWih7yCRJklpmQCZJktQyAzJJkqSWGZBJGmtJtia5N8l0kl19zq9JcmNz/vYkG5v0s5LckeTO5ucru645vUmfTvK2JFnBW5I0hgzIJI2tJKuAq4Gzgc3A+Uk292S7GHikqk4BrgKubNIfBF5TVS+kszXc7q5r3gH8PLCpeW1dtpuQNBEMyCSNszOA6aq6v6q+BtwAbOvJsw24vjm+CTgzSarqE1X1+SZ9P/C0pjftBOCZVfWxqirgPcDrlv1OJI01AzJJ4+xE4IGu9webtL55quoI8CiwtifPzwAfr6onmvwH5yhTkubFdcgk6SiSnErnMearFnDtDmAHwIYNG5a4ZtL8dK+XeeCKV7dYE/VjD5mkcXYIOKnr/fomrW+eJKuB44GHmvfrgZuBC6vqvq786+coE4CquqaqtlTVlnXr1i3yViSNMwMySeNsL7ApyclJjgG2A1M9eaboDNoHOAe4raoqybOAW4BdVfXRmcxV9QXgy0le1syuvBD4s2W+D0ljzoBM0thqxoTtBG4F7gHeV1X7k1yW5LVNtmuBtUmmgTcCM0tj7AROAS5J8snm9dzm3BuAdwHTwH3A+1fmjiSNK8eQSRprVbUH2NOTdknX8ePAuX2uuxy4fJYy9wGnLW1NJU0ye8gkSZJaZkAmSdKQ2bjrlm+bFTnqf47mZkAmSZLUMgMySZKkljmof5Hs6pUkSYtlD5kkSVLLBgrIkmxNcm+S6SS7+pxfk+TG5vztSTY26WcluSPJnc3PV3Zdc3qTPp3kbc0Ci5IkSRNnzoAsySrgauBsYDNwfpLNPdkuBh6pqlOAq+js+wbwIPCaqnohnZWwd3dd8w7g54FNzWvrIu5DkiRpZA0yhuwMYLqq7gdIcgOwDbi7K8824NLm+Cbg7UlSVZ/oyrMfeFqSNcCzgWdW1ceaMt8DvA5Xu5YkTTDHJU+uQR5Zngg80PX+YJPWN0+zVcmjwNqePD8DfLyqnmjyH5yjTEmSpImwIrMsk5xK5zHmqxZw7Q5gB8CGDRuWuGaSJEntG6SH7BBwUtf79U1a3zxJVgPHAw8179cDNwMXVtV9XfnXz1EmAFV1TVVtqaot69atG6C6k8MVliVJGg+DBGR7gU1JTk5yDLAdmOrJM0Vn0D7AOcBtVVVJngXcAuyqqo/OZK6qLwBfTvKyZnblhcCfLe5WJEmSRtOcAVkzJmwncCtwD/C+qtqf5LIkr22yXQusTTINvBGYWRpjJ3AKcEmSTzav5zbn3gC8C5gG7sMB/ZIkaUINNIasqvYAe3rSLuk6fhw4t891lwOXz1LmPuC0+VRWkiRpHLl1kiSp73jUA1e8uoWaaDZHGzPs72r0uXWSJElSywzIJEmSWmZAJkmS1DIDMkmSpJYZkEmSJLXMgEySJKllLnshSeprpbdm6/7zXMahff4+VpY9ZJIkSS0zIJMkSWqZAZmksZZka5J7k0wn2dXn/JokNzbnb0+ysUlfm+TDSR5L8vaea/66KbN3j15JWhDHkEkaW0lWAVcDZwEHgb1Jpqrq7q5sFwOPVNUpSbYDVwLnAY8Db6az526/fXcvaPbklaRFs4dM0jg7A5iuqvur6mvADcC2njzbgOub45uAM5Okqr5aVR+hE5hJ0rIyIJM0zk4EHuh6f7BJ65unqo4AjwJrByj7D5rHlW9OkqWorKTJ5SNLSZq/C6rqUJJnAH8CvB54T2+mJDuAHQAbNmxY2RpqZCzl8iL9lqroV/5KL2miudlDJmmcHQJO6nq/vknrmyfJauB44KGjFVpVh5qfXwHeS+fRaL9811TVlqrasm7dugXdgKTJYEAmaZztBTYlOTnJMcB2YKonzxRwUXN8DnBbVdVsBSZZneQ5zfF3Aj8F3LXkNZc0UXxkKWlsVdWRJDuBW4FVwHVVtT/JZcC+qpoCrgV2J5kGHqYTtAGQ5ADwTOCYJK8DXgX8I3BrE4ytAj4E/P7K3ZWkcWRAJmmsVdUeYE9P2iVdx48D585y7cZZij19qeonSeAjS0mSpNbZQyZJ0jyt1Mbbi5kNuRyzN91kfPkM1EPm1iOSJEnLZ84eMrce0bDr/ea2Ut9cJUlaKoP0kLn1iCRJ0jIaJCBz6xFJkqRl1OYsywuq6oXAjzav1/fLlGRHkn1J9h0+fHhFKyhJkrQSBgnI3HpEkiRpGQ0SkLn1iCRpRW3cdYsbYGuizDnL0q1HJEmSltdAC8O69YgkSdLyceskSZKklhmQSZIktcyATBPPwcOSpLa5ubjGmtsqSZJGgQGZJElLqI0vfovt5ffLavsMyDQRfCQpSRpmBmSaWAZpkqRh4aB+SZKkltlDprGxUj1evRMFJElaLAMyDS0HmUqSJoWPLCVJklpmD9kCOSB8Za30Y0IfS0paDqPe8z9I2zjq99gWe8gkSZJaZkAmSZLUMh9ZSgPwEfXoSrIVeCuwCnhXVV3Rc34N8B7gdOAh4LyqOpBkLXAT8BLg3VW1s+ua04F3A08D9gC/UlW1ArcjaUzZQyZpbCVZBVwNnA1sBs5Psrkn28XAI1V1CnAVcGWT/jjwZuDX+xT9DuDngU3Na+vS117SJDEgkzTOzgCmq+r+qvoacAOwrSfPNuD65vgm4MwkqaqvVtVH6ARmT0pyAvDMqvpY0yv2HuB1y3kTksafjyzHQL/HaaMys2WYZjP6WHIsnQg80PX+IPDS2fJU1ZEkjwJrgQePUubBnjJPXJLaSppYBmRqxWKDn2EPnpz2LYAkO4AdABs2bGi5NmrTsLdZS2HQe+zN191GDtOX9JXmI0tJ4+wQcFLX+/VNWt88SVYDx9MZ3H+0MtfPUSYAVXVNVW2pqi3r1q2bZ9UlTRIDMknjbC+wKcnJSY4BtgNTPXmmgIua43OA2442Y7KqvgB8OcnLkgS4EPizpa+6pEkyUECWZGuSe5NMJ9nV5/yaJDc2529PsrFJX5vkw0keS/L2nmtOT3Jnc83bmoZNOqqNu24Zyq7/Ya3XpKuqI8BO4FbgHuB9VbU/yWVJXttkuxZYm2QaeCPwZBuX5ADwFuDnkhzsmqH5BuBdwDRwH/D+lbgfSeNrzjFkXdPGz6IzeHVvkqmqursr25PTxpNspzNt/Dy+NW38tObVbWba+O101vHZio2apCVWVXvotDHdaZd0HT8OnDvLtRtnSd/HU9s0SVqwQQb1PzltHCDJzLTx7oBsG3Bpc3wT8PaZaePAR5Kc0l1g97Tx5v3MtHEDMg0Ve70kSSthkIDMaeOSJC2xYf3CN6z1GndDP6g/yY4k+5LsO3z4cNvVkebN8WWSpLkM0kM2n2njB5dj2jhwDcCWLVvcK25MuW6XJGmSDdJD5rRxSZKkZTRnD1kzJmxm2vgq4LqZaePAvqqaojNtfHczbfxhOkEb8OS08WcCxyR5HfCqZobmG4B3A0+jM5h/6Af0+9hJczna35FJXoFaknR0A22d5LRxSZKk5TP0g/olSZLGnZuLS9KE8LH5wjjpSCvBHjJJkqSWGZBJkiS1zEeW0hDoNzvTRyOSNDnsIdOyc6V6SZKOzoBMkiSpZT6yHFPOplp+9vpJkpaKAdmYG8XAzEBH0kIdrc07WtsyaBs5yG4cg+YfF8N+j6OybImPLCVJklpmD5mW1Kh8ExkWw/7NUpK0MgzIBuB/mpIkaTn5yFKSJKll9pBpSdiLODg/K0lSLwMyzZvjxCRJWloGZJIkdZnvckGT2Os9ife83BxDJg0pt5ySpMlhQCZJktQyH1lqxdjbsziO3VuYJFuBtwKrgHdV1RU959cA7wFOBx4CzquqA825NwEXA98Afrmqbm3SDwBfadKPVNWWFbkZSWPLgEwDm29AZQCmtiVZBVwNnAUcBPYmmaqqu7uyXQw8UlWnJNkOXAmcl2QzsB04Ffge4ENJvq+qvtFc94qqenDFbkbSWPORpaRxdgYwXVX3V9XXgBuAbT15tgHXN8c3AWcmSZN+Q1U9UVWfBaab8iRpyRmQSRpnJwIPdL0/2KT1zVNVR4BHgbVzXFvAXya5I8mOZai3pAkz0CNLx2BI0rd5eVUdSvJc4INJPlNVf9ObqQnWdgBs2LBhpes4EIcWzM7PZvHmu4RI73ULuXZUzdlD1jUG42xgM3B+M7ai25NjMICr6IzBoGcMxlbgfzXlzXhFVb3YYEzSMjkEnNT1fn2T1jdPktXA8XS+WM56bVXN/PwScDOzPMqsqmuqaktVbVm3bt2ib0bS+Bqkh+zJMRgASWbGYHQPit0GXNoc3wS8vXcMBvDZJDNjMP5uaaqvleC3xOHT+zuZlG+QC7AX2JTkZDrB1HbgZ3vyTAEX0WmXzgFuq6pKMgW8N8lb6Azq3wT8fZLjgO+oqq80x68CLluZ25E0rgYJyPqNo3jpbHmq6kiS7jEYH+u5tncMRgHvrKpr+v3ho9DlL2k4Ne3RTuBWOkMurquq/UkuA/ZV1RRwLbC7+cL4MJ2gjSbf++h8+TwC/FJVfSPJ84CbO985WQ28t6o+sOI3J2mstLnsxUBjMJpA7RqALVu21EpXUtJoq6o9wJ6etEu6jh8Hzp3l2t8Gfrsn7X7gRUtfU0mTbJCAbD5jMA4uZAxGkpkxGE8JyNo0ro/qFjrI8mhlSZKkhRskIHMMxgQy0JIkaeXMGZA5BkOSxssofeHqt/zBfHv5R+l+x9XRfgdz/X56f9/juiTGQGPIHIMhtcf/TCRp/LmX5YTwP3VJkoaXWydJkiS1zIBMkiSpZT6ynGBHGyyr0TLX76134Ou4DoqVpFFlQCZJY2gSvlwtZuaeRs+4/059ZClJktQyAzJpAmzcdcvYf7uUpFFmQCZJktQyx5BJ+jYO+JeklWcPmTThfJwpSe0zIJMkSWqZjywlAeM/pXyc+ZhZeqpBlkXp/vcy303rl5o9ZJIkSS2zh6yHvQSSJGmlGZBJE8QvHJI0nHxkKUmS1LKJ7CFzU+2nmvT7V3+9fy8WOtjVQeeSdHQTGZBJWhgDK0laHgZkkjRGxrm3eznvbZw/t3G2nEtV9Ps7sZxfRCc+IPMfobQwba/ZI0njxEH9kiRJLRsoIEuyNcm9SaaT7Opzfk2SG5vztyfZ2HXuTU36vUl+YtAyJWkp2H5JGgVzPrJMsgq4GjgLOAjsTTJVVXd3ZbsYeKSqTkmyHbgSOC/JZmA7cCrwPcCHknxfc81cZS45H09Kk2Wc2i9J422QMWRnANNVdT9AkhuAbUB347MNuLQ5vgl4e5I06TdU1RPAZ5NMN+UxQJmLYvAlrawhXU5mJNsvSZNnkEeWJwIPdL0/2KT1zVNVR4BHgbVHuXaQMiVpsWy/JI2EoZ9lmWQHsKN5+1iSe/tkew7w4MrVaslZ//aMct1hCOqfKwdL63NukLo/f0GVGhIDtl+zaf13uwijXHcY7fqPct1hgfU/WptztHyLaL/66Vf3gduwQQKyQ8BJXe/XN2n98hxMsho4HnhojmvnKhOAqroGuOZoFUyyr6q2HP02hpf1b88o1x1Gu/4rVPehb79m4++2PaNc/1GuO4x2/Rdb90EeWe4FNiU5OckxdAa5TvXkmQIuao7PAW6rqmrStzezmE4GNgF/P2CZkrRYtl+SRsKcPWRVdSTJTuBWYBVwXVXtT3IZsK+qpoBrgd3NoNeH6TRQNPneR2ew6xHgl6rqGwD9ylz625M0yWy/JI2KdL4IjrYkO5pHAyPJ+rdnlOsOo13/Ua77Shjlz2eU6w6jXf9RrjuMdv0XW/exCMgkSZJGmVsnSZIktWzkA7JR2sIkyUlJPpzk7iT7k/xKk/7sJB9M8g/Nz+9uu65Hk2RVkk8k+Yvm/cnNljPTzRY0x7Rdx9kkeVaSm5J8Jsk9SX54VD7/JP+x+XtzV5I/SnLsMH/2Sa5L8qUkd3Wl9f2s0/G25j4+neSH2qt5u0apTYPxaNds09phm/btRjogy7e2RTkb2Aycn852J8PqCPBrVbUZeBnwS019dwF/VVWbgL9q3g+zXwHu6Xp/JXBVVZ0CPEJnK5ph9VbgA1X1AuBFdO5j6D//JCcCvwxsqarT6Awmn9nmZ1g/+3cDW3vSZvusz6Yzi3ETnXW73rFCdRwqI9imwXi0a7ZpK8w2rY+qGtkX8MPArV3v3wS8qe16zaP+f0ZnP7x7gROatBOAe9uu21HqvL75S/dK4C+A0FkIb3W/38kwveisL/VZmrGTXelD//nzrdXhn01ndvRfAD8x7J89sBG4a67PGngncH6/fJP0GvU2ranzSLVrtmmt1d02rec10j1kjPAWJkk2Aj8I3A48r6q+0Jz6IvC8tuo1gN8FfgP4ZvN+LfDP1dlyBob7d3AycBj4g+bxxLuSHMcIfP5VdQj4n8DngC/Q2d7nDkbns58x22c9sv+Wl9hIfw4j2q79LrZpK8427alGPSAbSUmeDvwJ8KtV9eXuc9UJpYdy6muSnwK+VFV3tF2XBVoN/BDwjqr6QeCr9HTlD+vn34xL2EanAf4e4Die2nU+Uob1s9bCjGK7ZpvWHtu0pxr1gGyQbVGGSpLvpNNo/WFV/WmT/E9JTmjOnwB8qa36zeFfA69NcgC4gU4X/1uBZ6Wz5QwM9+/gIHCwqm5v3t9EpzEbhc//x4HPVtXhqvo68Kd0fh+j8tnPmO2zHrl/y8tkJD+HEW7XbNPaY5vWY9QDspHawiRJ6KwKfk9VvaXrVPfWLRfRGYMxdKrqTVW1vqo20vmsb6uqC4AP09lyBoa7/l8EHkjy/U3SmXRWYR+Fz/9zwMuSfFfz92im7iPx2XeZ7bOeAi5sZia9DHi06zHAJBmpNg1Gu12zTWuVbVqvtgfILcEAu58E/i9wH/Bf2q7PHHV9OZ3uzE8Dn2xeP0lnzMJfAf8AfAh4dtt1HeBefgz4i+b4e+ns8TcN/DGwpu36HaXeLwb2Nb+D/wN896h8/sBvAZ8B7gJ2A2uG+bMH/ojO2JCv0/kmf/FsnzWdgdRXN/+O76Qz86r1e2jpcxuZNq2p71i0a7ZprdTdNq3r5Ur9kiRJLRv1R5aSJEkjz4BMkiSpZQZkkiRJLTMgkyRJapkBmSRJUssMyCRJklpmQCZJktQyAzINrSQvSfLpJMcmOS7J/iSntV0vSZKWmgvDaqgluRw4FnganT3bfqflKkmStOQMyDTUmv389gKPAz9SVd9ouUqSJC05H1lq2K0Fng48g05PmSRJY8ceMg21JFPADcDJwAlVtbPlKkmStORWt10BaTZJLgS+XlXvTbIK+Nskr6yq29qumyRJS8keMkmSpJY5hkySJKllBmSSJEktMyCTJElqmQGZJElSywzIJEmSWmZAJkmS1DIDMkmSpJYZkEmSJLXs/wO85HObFMm4DwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "BINS = 100\n",
    "COUNT = 5000\n",
    "\n",
    "train_data, test_data = generate_1d_data(COUNT, BINS)\n",
    "plot_1d_data(train_data, test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "valid-dragon",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "demonstrated-sterling",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "gothic-chaos",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How we can do it?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceramic-contamination",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can assume what $\\pi(k) \\approx p(k|\\theta)$ and try to optimize $\\theta$. For example $p(k|\\theta) = \\dfrac{e^{\\theta_k}}{\\sum\\limits_{i=1}^{m}e^{\\theta_i}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "native-vacuum",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let $X$ has $n_i$ elements equal to $i$, $\\sum\\limits_{i=1}^{m}n_i = N$. Loglikelihood: \n",
    "\n",
    "$log~p(X|\\theta) =\\sum\\limits_{i=1}^{m} n_i log~p(k=i|\\theta)$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pediatric-tsunami",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "For batch of $\\{x_i\\}_{i=1}^{N}$ elements and predicted probabilities $\\{(p^1_i, \\dots , p^m_i)\\}_{i=1}^{N}$:\n",
    "\n",
    "Cross entropy loss = $- \\sum\\limits_{i=1}^{N} x_i log~p_{i}^{x_i}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "designed-solomon",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example 2\n",
    "\n",
    "Let we have samples from 2D distribution. We want to generate additional samples from the distribution "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "optimum-manual",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def generate_2d_data(count, bins):\n",
    "    im = Image.open('pics/sem_image.png').resize((bins, bins)).convert('L')\n",
    "    im = np.array(im).astype('float32')\n",
    "    dist = im / im.sum()\n",
    "\n",
    "    pairs = list(itertools.product(range(bins), range(bins)))\n",
    "    idxs = np.random.choice(len(pairs), size=count, replace=True, p=dist.reshape(-1))\n",
    "    samples = np.array([pairs[i] for i in idxs])\n",
    "\n",
    "    split = int(0.8 * len(samples))\n",
    "    return dist, samples[:split], samples[split:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "veterinary-alloy",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def plot_2d_data(train_data, test_data, bins=None):\n",
    "    if bins is None:\n",
    "        bins = int(max(test_data.max(), train_data.max()) - min(test_data.min(), train_data.min())) + 1\n",
    "    else:\n",
    "        bins = int(bins)\n",
    "    train_dist, test_dist = np.zeros((bins, bins)), np.zeros((bins, bins))\n",
    "    for i in range(len(train_data)):\n",
    "        train_dist[train_data[i][0], train_data[i][1]] += 1\n",
    "    train_dist /= train_dist.sum()\n",
    "\n",
    "    for i in range(len(test_data)):\n",
    "        test_dist[test_data[i][0], test_data[i][1]] += 1\n",
    "    test_dist /= test_dist.sum()\n",
    "\n",
    "    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 6))\n",
    "    ax1.set_title('Train Data')\n",
    "    ax1.imshow(train_dist, cmap='gray')\n",
    "    ax1.axis('off')\n",
    "    ax1.set_xlabel('x1')\n",
    "    ax1.set_ylabel('x0')\n",
    "\n",
    "    ax2.set_title('Test Data')\n",
    "    ax2.imshow(test_dist, cmap='gray')\n",
    "    ax2.axis('off')\n",
    "    ax2.set_xlabel('x1')\n",
    "    ax2.set_ylabel('x0')\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "answering-biology",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAEcCAYAAADDS24xAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAK/0lEQVR4nO3de4imZ3nH8d+li4eamPUYNdqICqIYBEFTBdv6h2A8UAVFUDEKRWJJ/7EUK4F6gLS1VNsUBUFUxNharQq2oiiI54giQuMBsaLWGJJuNJtEEyVkr/7xPGNnl+xkZ83szFzz+cAw8z7v4X5Iws33vZ/7fVPdHQCAye6x2ycAALDTBA8AMJ7gAQDGEzwAwHiCBwAYT/AAAOMJngOiqj5VVRfv9nkAwG4QPHtYVf1y08+xqrpt0+2Xb+e1uvui7n7/aZ7Hj9exb6mqo1X11aq6pKpO6b+fqnp0VXVVHTqd8YH97e6cy9bX+3xV/ekW92/MORtjXF9V/1lVz97GGK+qqi9v99zYuwTPHtbdZ238JPmfJC/YdOyDG487QyHxgu4+O8n5Sf4uyeuTvOcMjAvsc6c6l+2Aw+uYT07y2SQfr6pX7eB47GGCZx+qqj+uqmuq6vVVdV2S91XVA9Z3MEeq6sb170dues5v3xFtvHOpqn9YH/ujqrroVMbu7pu6+xNJXprk4qp60vqaz6uqb1XVzVX106p606anfXH9fXR9t/X0qnpsVX2uqn5eVTdU1Qer6vDd8I8H2Ceq6h5V9VdV9cN1LvhwVT1wve8+VXXlevxoVX2jqs6tqsuTPDPJO9b55B13NU53X9fdVyR5U5K3bqxObxr7lqr6blW9aD3+hCTvSvL0dYyj6/Gt5jn2OMGzfz0syQOzrLi8Jsu/y/ett38/yW1JtpoILkzy/SQPTvL3Sd5TVXWqg3f315Nck2XiSZJfJXllksNJnpfktVX1wvW+P1x/H17f0V2VpJL8bZJHJHlCkkdlmYyAg+PPk7wwyR9lmQtuTPLO9b6Lk5yTZW54UJJLktzW3Zcl+VKSS9f55NJtjPexJA9N8vj19g+zzGHnJHlzkiur6uHd/b11vKvWMQ6vj99qnmOPEzz717Ekb+zu33T3bd398+7+aHff2t23JLk8yyRyMj/p7nd39x1J3p/k4UnO3eY5XJslutLdn+/uq7v7WHf/V5J/3Wr87v7v7v7sev5Hkrz9Ls4XmOeSJJd19zXd/Zssb3pevF6mvz1L6Dyuu+/o7m92982/43jXrr835q2PdPe167z1b0l+kORpJ3vyduc59habSPevI939640bVfV7Sf4xyXOSPGA9fHZV3XONmhNdt/FHd9+6Lu6ctc1zOC/JL9bxL8yyt+dJSe6V5N5JPnKyJ1bVuUmuyPLu6uws8X3jNscH9rfzs+yrObbp2B1Z3nx9IMvqzofWy91XZomj23+H8c5bf2/MW69M8rokj16Pn5Vl1ftObXeeY2+xwrN/nfi/uf+LLMu0F3b3/fP/l5FO+TLVdlTVU7NMHhufYviXJJ9I8qjuPifL9e+NsU881yT5m/X4Bev5vmKnzhXYs36a5KLuPrzp5z7d/bPuvr2739zdT0zyjCTPz3I5KbnzOeVUvCjJ/yb5flWdn+TdSS5N8qD1stW3s/W8tdU8xx4neOY4O8u+naPrpr837sQgVXX/qnp+kg8lubK7r940/i+6+9dV9bQkL9v0tCNZLsE95oTz/WWSm6rqvCR/uRPnC+xp70py+RofqaqHVNWfrH8/q6ouqKp7Jrk5yyWujZWg63P8fLKldbPzpVnmxTd097Ek98sSNUfWx7w6y8rNhuuTPLKq7rXp2FbzHHuc4Jnjn5LcN8kNSb6W5NN38+v/R1XdkuUd2WVZ9ty8etP9f5bkLetj/jrJhzfu6O5bs+wp+sr6aYs/yLJB8ClJbkryySybCYGD5YosKyafWeeOr2X5QEWyfDDj37PEzveSfCHLZa6N5714/ZTpP2/x+ker6ldJrk7y3CQv6e73Jkl3fzfJ25JclSVuLkjylU3P/VyS7yS5rqpuWI+ddJ5j76vu010ZBADYH6zwAADjCR4AYDzBAwCMJ3gAgPEEDwAw3pbftFxVPsIFB0x3j/kiNXMYHDwnm8Os8AAA4wkeAGA8wQMAjCd4AIDxBA8AMJ7gAQDGEzwAwHiCBwAYT/AAAOMJHgBgPMEDAIwneACA8QQPADCe4AEAxhM8AMB4ggcAGE/wAADjCR4AYDzBAwCMd2i3T4D9qbuPu11Vu3QmAHDXrPAAAOMJHgBgPMEDAIxnDw+nxZ4dAPYTKzwAwHiCBwAYT/AAAOMJHgBgPMEDAIwneACA8QQPADCe4AEAxhM8AMB4ggcAGE/wAADjCR4AYDzBAwCMJ3gAgPEEDwAw3qHdPgEAOGi6+7jbVbVLZ3JwWOEBAMYTPADAeIIHABjPHh4AOMPs2TnzrPAAAOMJHgBgPMEDAIxnDw+nxXdIALCfWOEBAMYTPADAeIIHABjPHh5Oiz07AOwnVngAgPEEDwAwnuABAMYTPADAeIIHABhP8AAA4wkeAGA8wQMAjCd4AIDxBA8AMJ7gAQDGEzwAwHiCBwAYT/AAAOMJHgBgPMEDAIwneACA8QQPADCe4AEAxhM8AMB4ggcAGE/wAADjCR4AYDzBAwCMJ3gAgPEEDwAwnuABAMYTPADAeIIHABhP8AAA4wkeAGA8wQMAjCd4AIDxBA8AMJ7gAQDGEzwAwHiCBwAYT/AAAOMJHgBgPMEDAIwneACA8QQPADCe4AEAxhM8AMB4ggcAGE/wAADjCR4AYDzBAwCMJ3gAgPEEDwAwnuABAMYTPADAeIIHABhP8AAA4wkeAGA8wQMAjCd4AIDxBA8AMJ7gAQDGEzwAwHiCBwAYT/AAAOMJHgBgPMEDAIwneACA8QQPADCe4AEAxhM8AMB4ggcAGE/wAADjCR4AYDzBAwCMJ3gAgPEEDwAwnuABAMYTPADAeIIHABhP8AAA4x3ayRfv7uNuV9VODgcAcKes8AAA4wkeAGA8wQMAjLeje3js2QEA9gIrPADAeIIHABhP8AAA4+3oHh44U7b7nU9bPd73RwHMY4UHABhP8AAA4wkeAGA8e3gYYbv7bLZ6vD07APNY4QEAxhM8AMB4ggcAGE/wAADjCR4AYDzBAwCMJ3gAgPEEDwAwnuABAMYTPADAeIIHABhP8AAA4wkeAGA8wQMAjCd4AIDxDu32CQAAZ053H3e7qnbpTM4sKzwAwHiCBwAYT/AAAOPZwwMAB8hB2bNzIis8AMB4ggcAGE/wAADjjd7Ds93vGtj8+IN6jRMAJrLCAwCMJ3gAgPEEDwAw3ug9PNvdh2PfDgDMZIUHABhP8AAA4wkeAGA8wQMAjCd4AIDxBA8AMJ7gAQDGEzwAwHiCBwAYT/AAAOMJHgBgPMEDAIwneACA8QQPADCe4AEAxhM8AMB4ggcAGE/wAADjCR4AYDzBAwCMJ3gAgPEEDwAwnuABAMYTPADAeIIHABhP8AAA4wkeAGA8wQMAjCd4AIDxBA8AMJ7gAQDGEzwAwHiCBwAYT/AAAOMJHgBgPMEDAIwneACA8QQPADCe4AEAxhM8AMB4ggcAGE/wAADjCR4AYDzBAwCMJ3gAgPEEDwAwnuABAMYTPADAeIIHABhP8AAA4wkeAGA8wQMAjCd4AIDxBA8AMJ7gAQDGEzwAwHiCBwAYT/AAAOMJHgBgPMEDAIwneACA8QQPADCe4AEAxhM8AMB4ggcAGE/wAADjCR4AYDzBAwCMJ3gAgPEEDwAwnuABAMYTPADAeIIHABhP8AAA4wkeAGA8wQMAjCd4AIDxBA8AMJ7gAQDGEzwAwHiCBwAYT/AAAONVd+/2OQAA7CgrPADAeIIHABhP8AAA4wkeAGA8wQMAjCd4AIDx/g+vKSl1CShVKwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "COUNT = 10\n",
    "BINS = 64\n",
    "\n",
    "image, train_data, test_data = generate_2d_data(COUNT, BINS)\n",
    "plot_2d_data(train_data, test_data, BINS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "israeli-finland",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "second-circulation",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "neural-aspect",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MADE\n",
    "\n",
    "<img src=\"pics/made.png\" width=600 height=600 />"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "expected-string",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- What is model's input and output?\n",
    "- What loss fuction should we optimize? How?\n",
    "- Tensors' shapes?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stainless-shepherd",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- How do connections between inputs and outputs look like?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "according-governor",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- How can we train MNIST?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "natural-receiver",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Masks in MADE\n",
    "Pytorch docs\n",
    "<img src=\"pics/linear.jpg\" width=600 height=600 />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "framed-julian",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "class MaskedLinear(nn.Linear):\n",
    "    def __init__(self, in_features, out_features, bias=True):\n",
    "        super().__init__(in_features, out_features, bias)\n",
    "        self.register_buffer('mask', torch.ones(out_features, in_features))\n",
    "\n",
    "    def set_mask(self, mask):\n",
    "        self.mask.data.copy_(torch.from_numpy(mask.astype(np.uint8).T))\n",
    "\n",
    "    def forward(self, input):\n",
    "        return F.linear(input, self.mask * (self.weight*0+1), self.bias*0)#+torch.Tensor(np.arange(self.out_features))*0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "hidden-safety",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "mask = np.concatenate([np.ones((2,8)), np.zeros((2,8))], axis=0)\n",
    "#mask = np.hstack([np.ones((4,4)), np.zeros((4,4))])\n",
    "\n",
    "ml_layer = MaskedLinear(4,8)\n",
    "ml_layer.set_mask(mask)\n",
    "\n",
    "inp = [1,2,3,4]\n",
    "inp = torch.Tensor([inp,inp])+np.array([0,10])[:,None]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "regulation-group",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "turned-argentina",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "opposite-tours",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ml_layer(inp.float())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coral-cleaner",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}