03-Statistical-Experiments-and-Significance-Testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <div class=\"alert alert-info\" style=\"text-align: center;\" markdown=\"1\">Statistical Experiments and Significance Testing</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `A/B Testing`\n",
    "\n",
    "- A/B testing is a method used to compare the performance of two versions of a web page or app against each other to determine which one performs better.\n",
    "- A/B testing is a form of hypothesis testing, where two hypotheses are compared against each other.\n",
    "- The null hypothesis is that there is no difference between the two versions, while the alternative hypothesis is that there is a difference between the two versions.\n",
    "- The goal of A/B testing is to determine if the difference between the two versions is statistically significant.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Hypothesis Testing`\n",
    "\n",
    "- Hypothesis testing is a method used to determine if there is a significant difference between two groups.\n",
    "- The null hypothesis is the default assumption that there is no difference between the two groups.\n",
    "- The alternative hypothesis is the hypothesis that there is a difference between the two groups.\n",
    "\n",
    "#### One-Way Test\n",
    "\n",
    "- A one-way test is used to determine if there is a significant difference between two groups in one direction.\n",
    "\n",
    "#### Two-Way Test\n",
    "\n",
    "- A two-way test is used to determine if there is a significant difference between two groups in either direction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resampling\n",
    "\n",
    "- Resampling is a method used to estimate the distribution of a statistic by repeatedly sampling from the data.\n",
    "- Resampling methods include bootstrapping and permutation testing.\n",
    "\n",
    "#### Bootstrapping\n",
    "\n",
    "- Bootstrapping is a method used to estimate the distribution of a statistic by resampling with replacement from the data.\n",
    "- Bootstrapping is used to estimate the standard error of a statistic and to construct confidence intervals.\n",
    "\n",
    "#### Permutation Testing\n",
    "\n",
    "- Permutation testing is a method used to determine if there is a significant difference between two groups by permuting the data and calculating the test statistic.\n",
    "- Permutation testing is used to estimate the p-value of a hypothesis test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Website Sessions Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Page</th>\n",
       "      <th>Time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Page A</td>\n",
       "      <td>0.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Page B</td>\n",
       "      <td>2.53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Page A</td>\n",
       "      <td>0.35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Page B</td>\n",
       "      <td>0.71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Page A</td>\n",
       "      <td>0.67</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Page  Time\n",
       "0  Page A  0.21\n",
       "1  Page B  2.53\n",
       "2  Page A  0.35\n",
       "3  Page B  0.71\n",
       "4  Page A  0.67"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_website = pd.read_csv('data/web_page_data.csv')\n",
    "df_website.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAulElEQVR4nO3df3RU5Z3H8c8dEhIIJJbNQkBGqosiCCTBozVolZ8GFTaQo6LWjbiBPSTAosDSprYqqBtXQ9RqGlSQYC1FbfnRpUKgQFQQuwIJC7iirApBk7BZbYakJmQyd//wMNuRBBNM5s48836dc/+Y5/7IdzhM5pPnPvd5LNu2bQEAABjC5XQBAAAAnYlwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABglCinCwg2n8+nzz//XL1795ZlWU6XAwAA2sG2bZ06dUoDBgyQy3XuvpmICzeff/653G6302UAAIDzUFlZqYEDB57zmIgLN71795b09T9OfHy8w9UAAID28Hg8crvd/u/xc4m4cHPmVlR8fDzhBgCAMNOeISUMKAYAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAIW16v1+kSEIIINwCAsFRRUaGMjAwdOHDA6VIQYgg3AICw4/V6VVBQoIaGBhUUFNCDgwCEGwBA2Fm3bp1OnDgh6eu1AtevX+9wRQglhBsAQFipra3VypUrA9pWrFih2tpahypCqCHcAADCSlFRkZqbmwPampubVVxc7FBFCDWEGwBA2CgvL9fOnTvl8/kC2n0+n7Zv367y8nKHKkMoIdwAAMJGaWmpLMtqdZ9lWSotLQ1yRQhFhBsAQNhIT0+Xbdut7rNtW5MmTQpyRQhFhBsAQNhITU3VuHHj5HIFfn25XC6NHz9eKSkpzhSGkOJouCkuLtbIkSMVHx+v+Ph4paWlafPmzW0eX1JSIsuyArbY2NggVgwAcFpubq6io6MD2qKjo5WTk+NQRQg1joabgQMH6vHHH9e+ffu0d+9ejRs3ThkZGTp8+HCb58THx6uqqsq/HTt2LIgVAwCclpiYqOzs7IC2mTNnKjEx0aGKEGqinPzhU6ZMCXj92GOPqbi4WO+++66uuOKKVs+xLEtJSUnBKA8AEKIyMzP17//+76qsrJTb7da0adOcLgkhJGTG3LS0tGjt2rVqaGhQWlpam8fV19dr0KBBcrvd39rLI0lNTU3yeDwBGwAgvEVFRWnhwoWKi4vTokWLFBXl6N/qCDGW3daw8yA5ePCg0tLS1NjYqF69emnNmjW6+eabWz12z549+uijjzRy5EjV1dWpoKBAb731lg4fPqyBAwe2es7DDz+sJUuWnNVeV1en+Pj4Tn0vAIDg8nq9BJsI4fF4lJCQ0K7vb8fDzenTp3X8+HHV1dXpt7/9rVasWKE333xTw4YN+9Zzm5ubNXToUN1555165JFHWj2mqalJTU1N/tcej0dut5twE0H45QcA4a8j4cbx21Ldu3fX4MGDdeWVVyo/P1/Jycl65pln2nVudHS0UlNTdfTo0TaPiYmJ8T+NdWZD5KioqFBGRoYOHDjgdCkAgCBxPNx8k8/nC+hpOZeWlhYdPHhQ/fv37+KqEI68Xq8KCgrU0NCggoICeb1ep0sCAASBo+EmLy9Pb731lj799FMdPHhQeXl5Kisr049+9CNJUlZWlvLy8vzHL126VFu3btXHH3+s/fv36+6779axY8c0c+ZMp94CQti6det04sQJSVJlZaXWr1/vcEUAgGBwdCDCyZMnlZWVpaqqKiUkJGjkyJEqLS3VxIkTJUnHjx8PmIXyyy+/1KxZs1RdXa3vfe97uvLKK/XOO++0a3wOIkttba1WrlwZ0LZixQqNHTuWuTAAwHCODygOto4MSEL4WrJkid58882AlYNdLpfGjh2rn//85w5WBgA4H2E1oBjobOXl5dq5c2dAsJG+Hs+1fft2lZeXO1QZACAYCDcwTmlpqSzLanWfZVkqLS0NckUAgGAi3MA46enpautuq23bmjRpUpArAgAEE+EGxklNTdW4ceMCBqNLX4+5GT9+vFJSUpwpDECnY4oHtIZwAyPl5uYqOjo6oC06Olo5OTkOVQSgszFJJ9pCuIGREhMTlZ2dHdA2c+ZMHgMHDMEknTgXwg2MlZmZKbfbLUlyu92aNm2awxUB6CxM0olzIdzAWFFRUVq4cKHi4uK0aNEiFs8EDNHWJJ21tbUOVYRQQ7iB0VJSUrRx40YlJyc7XQqATlJUVKTm5uaAtubmZhUXFztUEUIN4QbGo8cGMAeTdKI9CDcAgLDBJJ1oD8INACBsMEkn2oNwAwAIG0zSifYg3AAAwgqTdOLbEG4AAGGFSTrxbQg3MB4zlwLmYZJOnAvhBkZj7RnATEzSiXOx7LaGnRvK4/EoISFBdXV1io+Pd7ocdCGv16sZM2boxIkTcrvdWrVqFb8AAcN4vV4+1xGiI9/f9NzAWKw9A5iPYIPWEG5gJNaeAYDIRbiBkVh7BgAiF+EGxmHtGQCIbIQbGIe1ZwAgshFuYBzWngGAyEa4gXFYewYAIhvhBkZi7RkAiFyEGxiJtWcAIHIRbmAs1p4BgMhEuIGxWHsGACITa0vBeKw9AwDhj7WlgL9CsAGAyEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACM4mi4KS4u1siRIxUfH6/4+HilpaVp8+bN5zzn9ddf1+WXX67Y2FiNGDFCb7zxRpCqBQAA4cDRcDNw4EA9/vjj2rdvn/bu3atx48YpIyNDhw8fbvX4d955R3feeaeys7NVXl6uqVOnaurUqTp06FCQKwcAAKEq5NaW6tOnj5588kllZ2eftW/69OlqaGjQpk2b/G3XXHONUlJStHz58nZdn7WlAAAIP2G5tlRLS4vWrl2rhoYGpaWltXrMnj17NGHChIC29PR07dmzp83rNjU1yePxBGwAAMBcjoebgwcPqlevXoqJidHs2bO1fv16DRs2rNVjq6ur1a9fv4C2fv36qbq6us3r5+fnKyEhwb+53e5OrR8AAIQWx8PNkCFDVFFRoT/96U/KycnRPffco/fff7/Trp+Xl6e6ujr/VllZ2WnXBgAAoSfK6QK6d++uwYMHS5KuvPJKvffee3rmmWf0/PPPn3VsUlKSampqAtpqamqUlJTU5vVjYmIUExPTuUUDAICQ5XjPzTf5fD41NTW1ui8tLU3bt28PaNu2bVubY3QAAEDkcbTnJi8vTzfddJMuuuginTp1SmvWrFFZWZlKS0slSVlZWbrwwguVn58vSZo/f75uuOEGLVu2TLfccovWrl2rvXv36oUXXnDybQAAgBDiaLg5efKksrKyVFVVpYSEBI0cOVKlpaWaOHGiJOn48eNyuf6/c2n06NFas2aNfvazn+mnP/2pLr30Um3YsEHDhw936i0AAIAQE3Lz3HQ15rkBACD8hOU8NwAAAJ2BcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAIW16v1+kSEIIINwCAsFRRUaGMjAwdOHDA6VIQYgg3AICw4/V6VVBQoIaGBhUUFNCDgwCEGwBA2Fm3bp1OnDghSaqsrNT69esdrgihhHADAAgrtbW1WrlyZUDbihUrVFtb61BFCDWEGwBAWCkqKlJzc3NAW3Nzs4qLix2qCKHG0XCTn5+vq666Sr1791bfvn01depUHTly5JznlJSUyLKsgC02NjZIFQMAnFReXq6dO3fK5/MFtPt8Pm3fvl3l5eUOVYZQ4mi4efPNNzVnzhy9++672rZtm5qbm3XjjTeqoaHhnOfFx8erqqrKvx07dixIFQMAnFRaWirLslrdZ1mWSktLg1wRQlGUkz98y5YtAa9LSkrUt29f7du3T9dff32b51mWpaSkpK4uDwAQYtLT08/67jjDtm1NmjQpyBUhFIXUmJu6ujpJUp8+fc55XH19vQYNGiS3262MjAwdPny4zWObmprk8XgCNgBAeEpNTdW4cePO6r2xLEvjx49XSkqKM4UhpIRMuPH5fLrvvvt07bXXavjw4W0eN2TIEL300kvauHGjXnnlFfl8Po0ePdr/SOA35efnKyEhwb+53e6uegsAgCDIzc2VyxX49eVyuZSTk+NQRQg1lm3bttNFSFJOTo42b96sXbt2aeDAge0+r7m5WUOHDtWdd96pRx555Kz9TU1Nampq8r/2eDxyu92qq6tTfHx8p9QOAAie2tpa3XHHHQET90VFRWnt2rVKTEx0sDJ0JY/Ho4SEhHZ9f4dEz83cuXO1adMm7dy5s0PBRpKio6OVmpqqo0ePtro/JiZG8fHxARsAIHwVFRW1+rQUj4LjDEfDjW3bmjt3rtavX68dO3bo4osv7vA1WlpadPDgQfXv378LKgQAhBIeBUd7OBpu5syZo1deeUVr1qxR7969VV1drerqan311Vf+Y7KyspSXl+d/vXTpUm3dulUff/yx9u/fr7vvvlvHjh3TzJkznXgLAIAg4lFwtIej4aa4uFh1dXUaM2aM+vfv799effVV/zHHjx9XVVWV//WXX36pWbNmaejQobr55pvl8Xj0zjvvaNiwYU68BQBAEKWnp6utoaI8Co4zQmZAcbB0ZEASACD0LF26VGVlZQG3plwul8aOHauf//znDlaGrhR2A4oBAGiv3NxcRUdHB7RFR0fzKDj8CDcAgLCSmJio7OzsgLaZM2fyGDj8CDcAgLCTmZnpn5TV7XZr2rRpDleEUEK4AQCEnaioKC1cuFBxcXFatGiRoqIcXSoRIYYBxQCAsOX1egk2EYIBxQCAiECwQWv4XwEAHWDbthobG50uAwg5sbGxbU6wGGyEGwBoJ9u2NW/ePB06dMjpUoCQM3z4cD377LMhEXC4LQUA7dTY2EiwAdpw6NChkOnVpOcGAM7Dc9d9oZhuEfU8BtCqphZLc3f1cbqMAIQbADgPMd1sxXRzugogFIReyOe2FAAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcwntfrdboEAEAQEW5gtIqKCmVkZOjAgQNOlwIACBLCDYzl9XpVUFCghoYGFRQU0IMDABGCcANjrVu3TidOnJAkVVZWav369Q5XBAAIBsINjFRbW6uVK1cGtK1YsUK1tbUOVQQACBbCDYxUVFSk5ubmgLbm5mYVFxc7VBEAIFgINzBOeXm5du7cKZ/PF9Du8/m0fft2lZeXO1QZACAYCDcwTmlpqSzLanWfZVkqLS0NckUAgGAi3MA46enpsm271X22bWvSpElBrggAEEyEGxgnNTVV1157bav7rrvuOqWkpAS3IABAUBFuEFHa6tEBAJiDcAPjlJeXa/fu3a3u2717NwOKAcBwhBsYhwHFABDZCDcwDgOKASCynXe4OXr0qEpLS/XVV19JOr+xDPn5+brqqqvUu3dv9e3bV1OnTtWRI0e+9bzXX39dl19+uWJjYzVixAi98cYbHf7ZMFdqaqrGjRsnlyvwv7fL5dL48eMZUAwAhutwuPnf//1fTZgwQZdddpluvvlmVVVVSZKys7O1cOHCDl3rzTff1Jw5c/Tuu+9q27Ztam5u1o033qiGhoY2z3nnnXd05513Kjs7W+Xl5Zo6daqmTp2qQ4cOdfStwGC5ubmKjo4OaIuOjlZOTo5DFQEAgqXD4eb+++9XVFSUjh8/rp49e/rbp0+fri1btnToWlu2bNGMGTN0xRVXKDk5WSUlJTp+/Lj27dvX5jnPPPOMJk2apH/5l3/R0KFD9cgjj2jUqFF67rnnOvpWYLDExETde++9AW333nuvEhMTHaoIABAsUR09YevWrSotLdXAgQMD2i+99FIdO3bsOxVTV1cnSerTp0+bx+zZs0cLFiwIaEtPT9eGDRtaPb6pqUlNTU3+1x6P5zvViPDBY98AEJk63HPT0NAQ0GNzxhdffKGYmJjzLsTn8+m+++7Ttddeq+HDh7d5XHV1tfr16xfQ1q9fP1VXV7d6fH5+vhISEvyb2+0+7xoRPmpra1VSUhLQtmrVKlYFB4AI0OFw88Mf/lAvv/yy/7VlWfL5fHriiSc0duzY8y5kzpw5OnTokNauXXve12hNXl6e6urq/FtlZWWnXh+hiVXBASBydfi21BNPPKHx48dr7969On36tBYvXqzDhw/riy++aHPitG8zd+5cbdq0SW+99dZZt7u+KSkpSTU1NQFtNTU1SkpKavX4mJiY79SjhPBzZlXwbzqzKvjkyZOVmprqQGUAgGDocM/N8OHD9eGHH+q6665TRkaGGhoalJmZqfLycv3d3/1dh65l27bmzp2r9evXa8eOHbr44ou/9Zy0tDRt3749oG3btm1KS0vr0M+GuZjEDwAiW4d7biQpISFBDzzwwHf+4XPmzNGaNWu0ceNG9e7d2z9uJiEhQT169JAkZWVl6cILL1R+fr4kaf78+brhhhu0bNky3XLLLVq7dq327t2rF1544TvXAzOkp6e3+eQek/gBgPnOK9w0NjbqP//zP3Xy5En5fL6AfX//93/f7uucGf8wZsyYgPZVq1ZpxowZkqTjx48HTMY2evRorVmzRj/72c/005/+VJdeeqk2bNhwzkHIiCxnJvErKysL+P/pcrk0duxYJvEDAMN1ONxs2bJFWVlZrT51YlmWWlpa2n2t9jyqW1ZWdlbbbbfdpttuu63dPweRJzc3V7t37w6YBoBJ/AAgMnR4zM28efN02223qaqqSj6fL2DrSLABulJiYqKys7MD2mbOnMkkfgAQATocbmpqarRgwYKz5poBQk1mZqZ/XiO3261p06Y5XBEAIBg6HG5uvfXWVm8VAaEmKipKCxcuVFxcnBYtWqSoqPMaYgYACDMd/m3/3HPP6bbbbtPbb7+tESNGnLU44T//8z93WnHAd5WSkqKNGzcSbAAggnT4N/5vfvMbbd26VbGxsSorKwuYT8SyLMINQg7BBgAiS4d/6z/wwANasmSJfvKTnwQ8og0AABAKOpxOTp8+renTpxNsAABASOpwQrnnnnv06quvdkUtAAAA31mHb0u1tLToiSeeUGlpqUaOHHnWgOLCwsJOKw4AAKCjOhxuDh486F9R+dChQwH72lqsEAAAIFg6HG527tzZFXUAAAB0CkYFAwAAo7Sr5yYzM1MlJSWKj49XZmbmOY9dt25dpxQGAABwPtoVbhISEvzjaRISErq0IAAAgO+iXeFm1apVWrp0qRYtWqRVq1Z1dU0AAADnrd1jbpYsWaL6+vqurAUAAOA7a3e4sW27K+sAAADoFB16Wop5bAAAQKjr0Dw3l1122bcGnC+++OI7FWQC27bV2NjodBlAyImNjeWPJABdrkPhZsmSJTwt9S1s29a8efPOmr0ZgDR8+HA9++yzBBwAXapD4eaOO+5Q3759u6oWIzQ2NhJsgDYcOnRIjY2N6tGjh9OlADBYu8MNf2l1XH3KnbJdHV7hAjCO5fOqV8VvnC4DQIRo9zcvT0t1nO2KkrpFf/uBgOH47QEgmNodbnw+X1fWAQAA0ClYOBMAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFEfDzVtvvaUpU6ZowIABsixLGzZsOOfxZWVlsizrrK26ujo4BQMAgJDnaLhpaGhQcnKyioqKOnTekSNHVFVV5d/69u3bRRUCAIBwE+XkD7/pppt00003dfi8vn376oILLuj8ggAAQNgLyzE3KSkp6t+/vyZOnKjdu3ef89impiZ5PJ6ADQAAmCuswk3//v21fPly/e53v9Pvfvc7ud1ujRkzRvv372/znPz8fCUkJPg3t9sdxIoBAECwOXpbqqOGDBmiIUOG+F+PHj1a//3f/62nnnpKv/rVr1o9Jy8vTwsWLPC/9ng8BBwA34lt2/L6bHWznK4EcJ7X9/VnwrJC5wMRVuGmNVdffbV27drV5v6YmBjFxMQEsSIApmtoaNDvP/QpKqz6voGu4fV9/Zno1auX06X4hf1Hs6KiQv3793e6DAAAECIc7bmpr6/X0aNH/a8/+eQTVVRUqE+fPrrooouUl5enzz77TC+//LIk6emnn9bFF1+sK664Qo2NjVqxYoV27NihrVu3OvUWAESguLg4/f1ljYrp5nQlgPOaWqQ3quOcLiOAo+Fm7969Gjt2rP/1mbEx99xzj0pKSlRVVaXjx4/7958+fVoLFy7UZ599pp49e2rkyJH64x//GHANAOhqlmUpymVxWwqQ1GIrpMbbSA6HmzFjxsi27Tb3l5SUBLxevHixFi9e3MVVAQCAcMbfHQAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjEG4AAIBRCDcAAMAohBsAAGAUwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCuEGAAAYhXADAACMQrgBAABGIdwAAACjOBpu3nrrLU2ZMkUDBgyQZVnasGHDt55TVlamUaNGKSYmRoMHD1ZJSUmX1wkAAMKHo+GmoaFBycnJKioqatfxn3zyiW655RaNHTtWFRUVuu+++zRz5kyVlpZ2caUAACBcRDn5w2+66SbddNNN7T5++fLluvjii7Vs2TJJ0tChQ7Vr1y499dRTSk9P76oyAQBAGAmrMTd79uzRhAkTAtrS09O1Z8+eNs9pamqSx+MJ2AAAgLnCKtxUV1erX79+AW39+vWTx+PRV1991eo5+fn5SkhI8G9utzsYpQIAAIeEVbg5H3l5eaqrq/NvlZWVTpcEAAC6kKNjbjoqKSlJNTU1AW01NTWKj49Xjx49Wj0nJiZGMTExwSgPAACEgLAKN2lpaXrjjTcC2rZt26a0tDSHKmqbbduyfS2SZXznGPDtfC2ybVuWZTldCYAI4Gi4qa+v19GjR/2vP/nkE1VUVKhPnz666KKLlJeXp88++0wvv/yyJGn27Nl67rnntHjxYv3jP/6jduzYoddee01/+MMfnHoLbWpoaFDzp/skq5vTpQDOs1vU0NCgXr16OV0JgAjgaLfC3r17lZqaqtTUVEnSggULlJqaqgcffFCSVFVVpePHj/uPv/jii/WHP/xB27ZtU3JyspYtW6YVK1bwGDgAAPBztOdmzJgxsm27zf2tzT48ZswYlZeXd2FVnSMuLk6+718pdYt2uhTAeS3NivvyA6erABAhwmrMTTixLEuWq5vk4rYUINvHeBsAQcNoVwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARiHcAAAAoxBuAACAUQg3AADAKIQbAABgFMINAAAwCssvAMB5aGqxJLW9Nh4QKb7+LIQWwg0AnIe5u/o4XQKANnBbCgDaKTY2VsOHD3e6DCAkDR8+XLGxsU6XIYmeGwBoN8uy9Oyzz6qxsdHpUiLeunXr9OKLL/pf/9M//ZOmTZvmYEWIjY2VZYXGLSrCDQB0gGVZ6tGjh9NlRLTa2lq9/PLLAW2rV6/WjTfeqMTERIeqQijhthQAIKwUFRWpubk5oK25uVnFxcUOVYRQQ7gBAISN8vJy7dy5Uz6fL6Dd5/Np+/btKi8vd6gyhBLCDQAgbJSWlrY5rsOyLJWWlga5IoQiwg0AIGykp6fLtlufX8i2bU2aNCnIFSEUEW4AAGEjNTVVffv2bXVfv379lJKSEtyCEJIINwCAsFFeXq6TJ0+2uq+mpoYxN5BEuAEAhBHG3KA9CDcAgLDBmBu0B5P4dSHL52VZPUBffxaAzpCamqpx48aprKws4HFwl8ulsWPHMuYGkgg3XapXxW+cLgEAjJObm6vdu3erqanJ3xYdHa2cnBwHq0Io4bZUJ2NhPaBtobSwHsJXYmKisrOzA9pmzpzJ0gvws+y2bl4ayuPxKCEhQXV1dYqPj++Sn2HbNgvrOejAgQP6yU9+0ub+xx9/XMnJyUGsCGeE0sJ6CG9er1f33nuvKisr5Xa7tWrVKkVFcTPCZB35/uZ/QhdgYT1nlZWVybKsVgcdWpalsrIyXXPNNQ5UBqCzREVFaeHChXrggQe0aNEigg0CcFsKxuFpCiAypKSkaOPGjfTE4iyEGxjnzNMULlfgf2+Xy6Xx48fzNAVgEHps0BrCDYyUm5ur6OjogDaepgCAyEC4gZF4mgIAIhfhBsbKzMyU2+2WJLndbk2bNs3higAAwUC4gbHOPE0RFxfH0xQAEEGY5wbG83q9BBsACHMd+f6m5wbGI9gAQGQh3AAAAKMQbgAAYcvrZcV5nC0kwk1RUZG+//3vKzY2Vj/4wQ/0H//xH20eW1JSIsuyAjYW4gOAyFNRUaGMjAwdOHDA6VIQYhwPN6+++qoWLFighx56SPv371dycrLS09N18uTJNs+Jj49XVVWVfzt27FgQKwYAOM3r9aqgoEANDQ0qKCigBwcBHA83hYWFmjVrlu69914NGzZMy5cvV8+ePfXSSy+1eY5lWUpKSvJv/fr1C2LFCDf80gPMs27dOp04cUKSVFlZqfXr1ztcEUKJo+Hm9OnT2rdvnyZMmOBvc7lcmjBhgvbs2dPmefX19Ro0aJDcbrcyMjJ0+PDhNo9tamqSx+MJ2BA56LYGzFNbW6uVK1cGtK1YsUK1tbUOVYRQ42i4qa2tVUtLy1k9L/369VN1dXWr5wwZMkQvvfSSNm7cqFdeeUU+n0+jR4/2J/hvys/PV0JCgn87M2MtzEe3NWCmoqIiNTc3B7Q1NzeruLjYoYoQahy/LdVRaWlpysrKUkpKim644QatW7dOf/u3f6vnn3++1ePz8vJUV1fn3yorK4NcMZxCtzVgnvLycu3cuVM+ny+g3efzafv27SovL3eoMoQSR8NNYmKiunXrppqamoD2mpoaJSUltesa0dHRSk1N1dGjR1vdHxMTo/j4+IAN5qPbGjBTaWmpLMtqdZ9lWSotLQ1yRQhFjoab7t2768orr9T27dv9bWfSd1paWruu0dLSooMHD6p///5dVSbCEN3WgJnS09PV1qpBtm1r0qRJQa4Iocjx21ILFizQiy++qNWrV+u//uu/lJOTo4aGBt17772SpKysLOXl5fmPX7p0qbZu3aqPP/5Y+/fv1913361jx45p5syZTr0FhBi6rQFzpaam6tprr21133XXXaeUlJTgFoSQ5PiiO9OnT9f//M//6MEHH1R1dbVSUlK0ZcsW/yDj48ePy+X6/wz25ZdfatasWaqurtb3vvc9XXnllXrnnXc0bNgwp94CQsyZbuvW/ro7022dmprqQGUAulKErQONc2BVcBinvLxc999/f5v7n376af66A8LUt32+n3rqKf54MRSrgiOipaamaty4cWcNOrQsS+PHjyfYAGGMAcVoD8INjHTHHXec1UVt27amT5/uUEUAOgMDitEehBsY6Te/+U2rPTevvvqqQxUB6Axnemb/eiym9PXs9vTM4gzCDYxz5mmp1npueFoKCH+5ubmKjo4OaIuOjlZOTo5DFSHUEG5gHO7JA2ZLTExUdnZ2QNvMmTOVmJjoUEUINYQbGId78oD5MjMz/WsFut1uTZs2zeGKEEoINzBOamqq+vbt2+q+fv36cU8eMEBUVJQWLlyouLg4LVq0SFFRjk/bhhBCuIFxysvLdfLkyVb31dTUMOYGMERKSoo2btyo5ORkp0tBiCHcwDiMuQEiBz02aA3hBsZhzA0ARDbCDYzDPBgAENkINzAS82AAQOQi3MBIzIMBAJGLcANjMQ8GAEQmwg2MxTwYABCZLLutx0oM5fF4lJCQoLq6OsXHxztdDoKgqalJMTExTpcBAPgOOvL9Tc8NjFZRUaHMzEwdOHDA6VIAAEFCuIGxvF6vCgoK1NDQoIKCAnm9XqdLAgAEAeEGxlq3bp1OnDghSaqsrNT69esdrggAEAyEGxiptrZWK1euDGhbsWKFamtrHaoIABAshBsYqaioSM3NzQFtzc3NKi4udqgiAECwEG5gnPLycu3cuVM+ny+g3efzafv27awKDgCGI9zAOKwKDgCRjXAD47AqOABENsINjMOq4AAQ2Qg3MBKrggNA5CLcwEisCg4AkYtwA2OxKjgARCbCDYzFquAAEJlYFRzG83q9BBsACHOsCg78FYINAEQWwg0AADAK4QYAABiFcAMAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCgRNwHImTkLPR6Pw5UAAID2OvO93Z65hyMu3Jw6dUqS/GsOAQCA8HHq1CklJCSc85iIW37B5/Pp888/V+/evWVZltPloIt5PB653W5VVlay3AZgGD7fkcW2bZ06dUoDBgyQy3XuUTUR13Pjcrk0cOBAp8tAkMXHx/PLDzAUn+/I8W09NmcwoBgAABiFcAMAAIxCuIHRYmJi9NBDDykmJsbpUgB0Mj7faEvEDSgGAABmo+cGAAAYhXADAACMQrgBAABGIdwAAACjEG4QEmbMmCHLsmRZlrp3767Bgwdr6dKl8nq9TpcmSTpx4oS6d++u4cOHO10KEHZC9fP98MMP++uyLEsJCQn64Q9/qDfffNPRuvDdEW4QMiZNmqSqqip99NFHWrhwoR5++GE9+eSTTpclSSopKdHtt98uj8ejP/3pT06XA4SdUP18X3HFFaqqqlJVVZX27NmjSy+9VJMnT1ZdXZ3TpeE7INwgZMTExCgpKUmDBg1STk6OJkyYoN///veSpMLCQo0YMUJxcXFyu93Kzc1VfX19wPkvvvii3G63evbsqWnTpqmwsFAXXHBBwDEbN27UqFGjFBsbq0suuURLliz51r8ebdvWqlWr9A//8A+66667tHLlyk5930AkCNXPd1RUlJKSkpSUlKRhw4Zp6dKlqq+v14cfftip7x/BRbhByOrRo4dOnz4t6es1wX7xi1/o8OHDWr16tXbs2KHFixf7j929e7dmz56t+fPnq6KiQhMnTtRjjz0WcL23335bWVlZmj9/vt5//309//zzKikpOeu4b9q5c6f+8pe/aMKECbr77ru1du1aNTQ0dP4bBiJIqHy+/1pTU5NWrVqlCy64QEOGDOmcNwpn2EAIuOeee+yMjAzbtm3b5/PZ27Zts2NiYuxFixa1evzrr79u/83f/I3/9fTp0+1bbrkl4Jgf/ehHdkJCgv/1+PHj7X/9138NOOZXv/qV3b9//3PWdtddd9n33Xef/3VycrK9atWqdrwrALYdup/vhx56yHa5XHZcXJwdFxdnW5Zlx8fH25s3b+7gO0SoIdwgJNxzzz12t27d7Li4OLt79+52VFSUnZWVZdfX19u2bdvbtm2zx40bZw8YMMDu1auXHRsba0uyGxoabNu27ZSUFHvJkiUB13zmmWcCfvklJibasbGx/l9kcXFxZ13nm7788ks7NjbW3rt3r7/tySeftK+77rpO/hcAzBWqn++HHnrIHjp0qP3RRx/ZH330kb1//377xz/+sd2rVy/7vffe65p/DARFlLP9RsD/Gzt2rIqLi9W9e3cNGDBAUVFf//f89NNPNXnyZOXk5Oixxx5Tnz59tGvXLmVnZ+v06dPq2bNnu65fX1+vJUuWKDMz86x9sbGxrZ6zZs0aNTY26gc/+IG/zbZt+Xw+ffjhh7rsssvO450CkScUP9+S/E9vnZGamqoNGzbo6aef1iuvvNLBd4lQQbhByIiLiwv4JXPGvn375PP5tGzZMrlcXw8Te+211wKOGTJkiN57772Atm++HjVqlI4cOdLqz2jLypUrtXDhQs2YMSOgPTc3Vy+99JIef/zxdl8LiGSh+PluS7du3fTVV1995+vAOYQbhLzBgwerublZzz77rKZMmaLdu3dr+fLlAcfMmzdP119/vQoLCzVlyhTt2LFDmzdvlmVZ/mMefPBBTZ48WRdddJFuvfVWuVwuHThwQIcOHdKjjz561s+tqKjQ/v379etf/1qXX355wL4777xTS5cu1aOPPur/CxRAxzn1+T7D6/WqurpaknTq1Cm9+uqrev/99/XjH/+4a94wgsPp+2KAbQcOOGxNYWGh3b9/f7tHjx52enq6/fLLL9uS7C+//NJ/zAsvvGBfeOGFdo8ePeypU6fajz76qJ2UlBRwnS1bttijR4+2e/ToYcfHx9tXX321/cILL7T6M+fOnWsPGzas1X1VVVW2y+WyN27c2OH3CkSaUPx82/bXY24k+beePXvaI0aMsIuLi7/rW4bDLNu2bUfTFdBFZs2apQ8++EBvv/2206UA6GR8vnEu9KfDGAUFBZo4caLi4uK0efNmrV69Wr/85S+dLgtAJ+DzjY6g5wbGuP3221VWVqZTp07pkksu0bx58zR79mynywLQCfh8oyMINwAAwCgsvwAAAIxCuAEAAEYh3AAAAKMQbgAAgFEINwAAwCiEGwAAYBTCDYCQM2PGDFmWJcuy/Ks2L126VF6v1+nSAIQBZigGEJImTZqkVatWqampSW+88YbmzJmj6Oho5eXlOV0agBBHzw2AkBQTE6OkpCQNGjRIOTk5mjBhgn7/+9+rsLBQI0aMUFxcnNxut3Jzc1VfXx9w7osvvii3262ePXtq2rRpKiws1AUXXBBwzMaNGzVq1CjFxsbqkksu0ZIlS+gZAgxBuAEQFnr06KHTp0/L5XLpF7/4hQ4fPqzVq1drx44dWrx4sf+43bt3a/bs2Zo/f74qKio0ceJEPfbYYwHXevvtt5WVlaX58+fr/fff1/PPP6+SkpKzjgMQnlh+AUDImTFjhv785z9rw4YNsm1b27dv1+TJkzVv3jw9+eSTAcf+9re/1ezZs1VbWytJuuOOO1RfX69Nmzb5j7n77ru1adMm/fnPf5YkTZgwQePHjw+4xfXKK69o8eLF+vzzz7v+DQLoUoy5ARCSNm3apF69eqm5uVk+n0933XWXHn74Yf3xj39Ufn6+PvjgA3k8Hnm9XjU2Nuovf/mLevbsqSNHjmjatGkB17r66qsDws6BAwe0e/fugJ6alpaWgOsACF+EGwAhaezYsSouLlb37t01YMAARUVF6dNPP9XkyZOVk5Ojxx57TH369NGuXbuUnZ2t06dPtzuU1NfXa8mSJcrMzDxrX2xsbGe/FQBBRrgBEJLi4uI0ePDggLZ9+/bJ5/Np2bJlcrm+HjL42muvBRwzZMgQvffeewFt33w9atQoHTly5KzrAzAD4QZA2Bg8eLCam5v17LPPasqUKdq9e7eWL18ecMy8efN0/fXXq7CwUFOmTNGOHTu0efNmWZblP+bBBx/U5MmTddFFF+nWW2+Vy+XSgQMHdOjQIT366KPBflsAOhlPSwEIG8nJySosLNS//du/afjw4fr1r3+t/Pz8gGOuvfZaLV++XIWFhUpOTtaWLVt0//33B9xuSk9P16ZNm7R161ZdddVVuuaaa/TUU09p0KBBwX5LALoAT0sBMN6sWbP0wQcf6O2333a6FABBwG0pAMYpKCjQxIkTFRcXp82bN2v16tX65S9/6XRZAIKEnhsAxrn99ttVVlamU6dO6ZJLLtG8efM0e/Zsp8sCECSEGwAAYBQGFAMAAKMQbgAAgFEINwAAwCiEGwAAYBTCDQAAMArhBgAAGIVwAwAAjEK4AQAARvk/hRtlk297WJ0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "sns.boxenplot(x='Page', y='Time', data=df_website, ax=ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.2633333333333332, 1.62)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_A = df_website[df_website['Page'] == 'Page A']['Time'].mean()\n",
    "mean_B = df_website[df_website['Page'] == 'Page B']['Time'].mean()\n",
    "mean_A, mean_B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3566666666666669"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean_B - mean_A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "❔ Is this difference in the number of sessions between the two versions statistically significant?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Page A    21\n",
       "Page B    15\n",
       "Name: Page, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_website[\"Page\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perm_func(x, nA, nB):\n",
    "    n = nA + nB\n",
    "    idx_B = np.random.choice(range(n), nB)\n",
    "    idx_A = np.setdiff1d(range(n), idx_B)\n",
    "    return x.loc[idx_B].mean() - x.loc[idx_A].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "perm_diffs = [perm_func(df_website['Time'], df_website[df_website[\"Page\"] == \"Page A\"].shape[0], df_website[df_website[\"Page\"] == \"Page B\"].shape[0]) for _ in range(1000)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAGwCAYAAABSN5pGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAABLcklEQVR4nO3deVwV9f4/8NdhXw+IyHZlc0FwR0vCJS1JXPJqeisVcyM1c80l41tqLoVLmWmmLYr6S69lqVnu4poh7jsKGIomYIiCLCLL5/fHXI6eARQOA+cAr+fjcR6385mZz3l/zoC87sxnZlRCCAEiIiIi0jDSdwFEREREhoYBiYiIiEiGAYmIiIhIhgGJiIiISIYBiYiIiEiGAYmIiIhIhgGJiIiISMZE3wUYgsLCQty+fRu2trZQqVT6LoeIiIjKQAiBBw8ewM3NDUZGyh7zYUACcPv2bbi7u+u7DCIiItLBzZs3Ub9+fUX7ZEACYGtrC0D6gtVqtZ6rISIiorLIyMiAu7u75u+4khiQAM1pNbVazYBERERUzVTG9BhO0iYiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiourBywtYskTfVSinpo2nhmFAIiIi/bt5ExgxAnBzA8zMAE9PYOJE4O5dfVdGtRQDEhER6ddffwHPPQfExQH//S8QHw+sXAlERgKBgUBamn7qKigACgv189mkdwxIRERUdRYvBj7+WPrfImPHSkeN9uwBOncGPDyAHj2AffuAv/8GPvzw8boPHgADBwLW1sC//gUsX/54mRBS3x4egLm5dDRqwoTHy3NzgalTpe2srYGAAODgwcfL16wB7O2BbduApk2lPr7/HrCwAO7f1x7HxInAyy8/fv/HH0CnToClJeDuLn1uVtbj5XfuAL17S8u9vYH163X9BqmKMCAREVHVWbwYmD37cUBKSwN27wbefVcKD09ycQFCQoAff5TCDwAsWgS0agWcOQN88IEUVPbulZb98gvwxRfAN99IR6O2bgVatHjc37hxQFQUsHEjcP488PrrQPfu0rpFsrOBBQukYHTpkvT59vZS30UKCqSaQkKk99euSf307y/1++OPUmAaN+7xNsOGSacRDxwAfv4Z+PprKTSRwTLRdwFERFSLxcVJ4cfPr+Tlfn7AvXvAP/9I7zt0kIIRAPj4AEePSqHolVeAxEQpVAUFAaam0pGkdu2kdRMTgYgI6X/d3KS2qVOBXbuk9k8/ldry8qTw0qrV4xoGDAA2bABCQ6X3kZHSEaX+/aX34eFSWJo0SXrfuDGwdKl0NGzFCukzd+4Ejh8Hnn9eWmfVqtLHTAaBR5CIiEj/io4QPUtgYPH3MTHSf7/+OpCTAzRoAIwcCWzZAuTnS8suXJCO/Pj4ADY2j1+HDklHgIqYmQEtW2p/RkiIdCru9m3p/fr1QK9e0pElADh3Tjo992S/wcHS/KWEBKk+ExOgbdvHffr6Pt6eDBKPIBERkf40agSoVFKIeO214stjYoA6dYB69Z7dl7s7cPWqNHdp717ptN2iRVIIyswEjI2BU6ek/32Sjc3j/7a0lOp50vPPAw0bSqfmxoyRgteaNY+XZ2YCo0drz3cq4uEBxMY+u3YyOAxIRESkP3XrSqfHvv4aeO897XlIycnS0ZohQx6HlmPHtLc/dkz7VJWlpTQZundvafK3r6909MjfXzqCdOeONJm6vEJCpFrq1weMjKQjSEXatAEuX5bCXkl8faUjWadOPT7FdvVq8YnfZFB4io2IiPTrq6+kK8yCg4HDh6XJzLt2ScHpX/8CPvnk8bpHjwILF0pHZZYvBzZtkiZqA9JRnVWrgIsXpVsH/PCDFJg8PaVTayEhUtjavFk69XX8uDR/aPv2Z9cYEgKcPi3V8p//SFe4FZk+HfjzT2lS9tmz0ryqX399PEm7SRNpEvfo0UB0tBSU3n67+KR0MigMSEREpF+NGwMnT0pzh954QzqdNWoU8NJL0lVnDg6P150yRVrX3x+YN0+6Gi44WFpmbw989500kbtlS+lU22+/SUepAGky9pAhUh9NmgB9+wInTkinwZ6lUSNpwvf584+vXivSsqV0Gi82Vjo65e8PzJz5eDJ40We7uUkTt/v1k8bn5FSRb40qmUqIss6Mq7kyMjJgZ2eH9PR0qNVqfZdDRFRz1a8v3dvoX/8Cbt3SdzVUzVXm328eQSIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiLD06XL40d3eHkBS5Y8XpacLN0CwNr68d2oS2ojqgDeKJKIiAzbiRNS8CnyxRdAUpJ0zyE7u9LbiCqAAYmIiAyb/DEj165JzzVr3PjpbeX16JH0LDYi8BQbERHpW1aWdANHGxvA1RX4/HPt5U+eYvPyAn75BVi3Tnr8yLBhJbcB0qM83n5bClhqNfDyy9KDZYt8/DHQujXw/feAtzdgYVG+7f7f/5M+284OGDAAePDg8TqFhdIdvxs1ku667eGhfUfwmzelm2La20s3wuzTB7h+vQJfIimNAYmIiPRr2jTpTtS//grs2QMcPCg91qMkJ05Ij+144w3plNqXX5bcBgCvvy49e23nTunxHm3aAF27Amlpj/uLj5fC1ebN0um5sm537RqwdSvw++/S69AhYP78x8vDwqT3M2ZIz2nbsAFwdpaW5eVJd/+2tQWOHJEen2JjI43h0SOFvlSqKJ5iIyIi/cnMlJ6f9sMPUggBgLVrpTtul6RePemIjKUl4OLyuF3e9scf0rPW7tx5/Ny0zz6TQs3PP0uP+gCkQLJu3ePTeGXdrrBQevabra30/q23gMhI6SjRgwdSSPvqK2DoUGl5w4ZAx47Sf//4o7T9998/fghvRIR0NOngQaBbN52+SlIWAxIREenPtWtSSAkIeNzm4CA9K60izp2TwlfRc9iK5ORIn1nE01N7jlNZt/PyehyOAOnU4J070n/HxEgP3y0KfCXVFh+vvT0APHyo/RmkV3o9xXb48GH07t0bbm5uUKlU2Lp1a6nrvvPOO1CpVFjy5KWeANLS0hASEgK1Wg17e3uEhoYiMzOzcgsnIiLDlpkphZazZ7VfV69Kp/SKPHl1XHm2MzXV3k6lko4KAdKRrGfV1rZt8c+IjQUGDSrfOKnS6PUIUlZWFlq1aoURI0agX79+pa63ZcsWHDt2DG5PPhn5f0JCQpCUlIS9e/ciLy8Pw4cPx6hRo7Bhw4bKLJ2IiJTQsKEUNqKjpYnMAHDvnhQWOnfWvd82baR7I5mYSEd7Knu7JzVuLIWkyEhpsndJn/Hjj4CTkzQJnAySXo8g9ejRA/PmzcNrr71W6jp///03xo8fj/Xr18NUlthjYmKwa9cufP/99wgICEDHjh2xbNkybNy4Ebdv3y61z9zcXGRkZGi9iIhID2xsgNBQ6ejM/v3AxYvSVWhGFfzzFBQEBAYCfftKE7+vXwf+/BP48EPg5Enlt3uShQUwfTrw/vvS/KZr14Bjx6S5VgAQEgI4OkpXrh05AiQkSHOPJkwAbt2q0LBJOQZ9FVthYSHeeustTJs2Dc2aNSu2PCoqCvb29njuuec0bUFBQTAyMkJ0dHSp/YaHh8POzk7zcnd3r5T6iYioDBYtAjp1Anr3lgJKx47SKaiKUKmAHTuAF18Ehg8HfHykS/Fv3Hh8NZmS28nNmAFMmQLMnAn4+QFvvvl4jpKVFXD4sHTErF8/aXloqDQHiUeUDIZKCCH0XQQAqFQqbNmyBX379tW0hYeH48CBA9i9ezdUKhW8vLwwadIkTPrf7ec//fRTrF27FlevXtXqy8nJCbNnz8aYMWNK/Kzc3Fzk5uZq3mdkZMDd3R3p6elQ84eTiKjy1K8P/P038K9/8WgJVVhGRgbs7Owq5e+3wV7FdurUKXz55Zc4ffo0VEWXQSrE3Nwc5kWXbxIRERHJGOwptiNHjuDOnTvw8PCAiYkJTExMcOPGDUyZMgVe/5s45+LigjtFhyz/Jz8/H2lpaXB58v4YREREROVgsEeQ3nrrLQQFBWm1BQcH46233sLw4cMBAIGBgbh//z5OnTqFtv87X71//34UFhYi4Ml7ahARERGVg14DUmZmJuLj4zXvExIScPbsWTg4OMDDwwN1ZTfqMjU1hYuLC5r87wZifn5+6N69O0aOHImVK1ciLy8P48aNw4ABA0q8JQARERFRWej1FNvJkyfh7+8Pf39/AMDkyZPh7++PmTNnlrmP9evXw9fXF127dkXPnj3RsWNHfPvtt5VVMhEREdUCBnMVmz5V5ix4IiJ6Aq9iIwVV5t9vg52kTURERKQvDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIm+i6AiKiyJSYmIjU1VbH+HB0d4eHhoVh/RGR4GJCIqEZLTEyEr68fcnKyFevT0tIKV67EMCQR1WAMSERUo6WmpiInJxsBI2ZB7epV4f4ykq4jevVspKamMiAR1WAMSERUK6hdveDg0UTfZRBRNcFJ2kREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMnoNSIcPH0bv3r3h5uYGlUqFrVu3apbl5eVh+vTpaNGiBaytreHm5oYhQ4bg9u3bWn2kpaUhJCQEarUa9vb2CA0NRWZmZhWPhIiIiGoSvQakrKwstGrVCsuXLy+2LDs7G6dPn8aMGTNw+vRpbN68GVevXsW///1vrfVCQkJw6dIl7N27F7///jsOHz6MUaNGVdUQiIiIqAYy0eeH9+jRAz169ChxmZ2dHfbu3avV9tVXX6Fdu3ZITEyEh4cHYmJisGvXLpw4cQLPPfccAGDZsmXo2bMnPvvsM7i5uZXYd25uLnJzczXvMzIyFBoRERER1QTVag5Seno6VCoV7O3tAQBRUVGwt7fXhCMACAoKgpGREaKjo0vtJzw8HHZ2dpqXu7t7ZZdORERE1Ui1CUgPHz7E9OnTMXDgQKjVagBAcnIynJyctNYzMTGBg4MDkpOTS+0rLCwM6enpmtfNmzcrtXYiIiKqXvR6iq2s8vLy8MYbb0AIgRUrVlS4P3Nzc5ibmytQGREREdVEBh+QisLRjRs3sH//fs3RIwBwcXHBnTt3tNbPz89HWloaXFxcqrpUIiIiqiEM+hRbUTiKi4vDvn37ULduXa3lgYGBuH//Pk6dOqVp279/PwoLCxEQEFDV5RIREVENodcjSJmZmYiPj9e8T0hIwNmzZ+Hg4ABXV1f85z//wenTp/H777+joKBAM6/IwcEBZmZm8PPzQ/fu3TFy5EisXLkSeXl5GDduHAYMGFDqFWxEREREz6LXgHTy5Em89NJLmveTJ08GAAwdOhQff/wxtm3bBgBo3bq11nYHDhxAly5dAADr16/HuHHj0LVrVxgZGaF///5YunRpldRPRERENZNeA1KXLl0ghCh1+dOWFXFwcMCGDRuULIuIiIhqOYOeg0RERESkDwxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDIMSEREREQyDEhEREREMgxIRERERDJ6DUiHDx9G79694ebmBpVKha1bt2otF0Jg5syZcHV1haWlJYKCghAXF6e1TlpaGkJCQqBWq2Fvb4/Q0FBkZmZW4SiIiIioptFrQMrKykKrVq2wfPnyEpcvXLgQS5cuxcqVKxEdHQ1ra2sEBwfj4cOHmnVCQkJw6dIl7N27F7///jsOHz6MUaNGVdUQiIiIqAYy0eeH9+jRAz169ChxmRACS5YswUcffYQ+ffoAANatWwdnZ2ds3boVAwYMQExMDHbt2oUTJ07gueeeAwAsW7YMPXv2xGeffQY3N7cqGwsRERHVHAY7BykhIQHJyckICgrStNnZ2SEgIABRUVEAgKioKNjb22vCEQAEBQXByMgI0dHRpfadm5uLjIwMrRcRERFREYMNSMnJyQAAZ2dnrXZnZ2fNsuTkZDg5OWktNzExgYODg2adkoSHh8POzk7zcnd3V7h6IiIiqs4MNiBVprCwMKSnp2teN2/e1HdJREREZEAMNiC5uLgAAFJSUrTaU1JSNMtcXFxw584dreX5+flIS0vTrFMSc3NzqNVqrRcRERFREYMNSN7e3nBxcUFkZKSmLSMjA9HR0QgMDAQABAYG4v79+zh16pRmnf3796OwsBABAQFVXjMRERHVDHq9ii0zMxPx8fGa9wkJCTh79iwcHBzg4eGBSZMmYd68eWjcuDG8vb0xY8YMuLm5oW/fvgAAPz8/dO/eHSNHjsTKlSuRl5eHcePGYcCAAbyCjYgqVUxMjKL9OTo6wsPDQ9E+iUh3eg1IJ0+exEsvvaR5P3nyZADA0KFDsWbNGrz//vvIysrCqFGjcP/+fXTs2BG7du2ChYWFZpv169dj3Lhx6Nq1K4yMjNC/f38sXbq0ysdCRLVDTvpdACoMHjxY0X4tLa1w5UoMQxKRgdBrQOrSpQuEEKUuV6lUmDNnDubMmVPqOg4ODtiwYUNllEdEVExe9gMAAq0HTUc9b19F+sxIuo7o1bORmprKgERkIPQakIiIqisbJw84eDTRdxlEVEkYkIiIDISS85o4p4moYhiQiIj0rDLmNXFOE1HFMCAREemZ0vOaOKeJqOIYkIiIDATnNREZDoO9USQRERGRvjAgEREREckwIBERERHJMCARERERyTAgEREREckwIBERERHJ6BSQ/vrrL6XrICIiIjIYOgWkRo0a4aWXXsIPP/yAhw8fKl0TERERkV7pFJBOnz6Nli1bYvLkyXBxccHo0aNx/PhxpWsjIiIi0gudAlLr1q3x5Zdf4vbt21i9ejWSkpLQsWNHNG/eHIsXL8Y///yjdJ1EREREVaZCk7RNTEzQr18/bNq0CQsWLEB8fDymTp0Kd3d3DBkyBElJSUrVSURERFRlKvQstpMnT2L16tXYuHEjrK2tMXXqVISGhuLWrVuYPXs2+vTpw1NvRFQuiYmJSE1NVay/mJgYxfoiotpDp4C0ePFiRERE4OrVq+jZsyfWrVuHnj17wshIOiDl7e2NNWvWwMvLS8laiaiGS0xMhK+vH3JyshXvOy/3keJ9ElHNpVNAWrFiBUaMGIFhw4bB1dW1xHWcnJywatWqChVHRLVLamoqcnKyETBiFtSuXor0mXQhChe3fYv8/HxF+iOi2kGngBQXF/fMdczMzDB06FBduieiWk7t6gUHjyaK9JWRdF2RfoiodtFpknZERAQ2bdpUrH3Tpk1Yu3ZthYsiIiIi0iedAlJ4eDgcHR2LtTs5OeHTTz+tcFFERERE+qRTQEpMTIS3t3exdk9PTyQmJla4KCIiIiJ90ikgOTk54fz588Xaz507h7p161a4KCIiIiJ90ikgDRw4EBMmTMCBAwdQUFCAgoIC7N+/HxMnTsSAAQOUrpGIiIioSul0FdvcuXNx/fp1dO3aFSYmUheFhYUYMmQI5yARERFRtadTQDIzM8OPP/6IuXPn4ty5c7C0tESLFi3g6empdH1EREREVa5Cjxrx8fGBj4+PUrUQERERGQSdAlJBQQHWrFmDyMhI3LlzB4WFhVrL9+/fr0hxRERERPqgU0CaOHEi1qxZg169eqF58+ZQqVRK10VERESkNzoFpI0bN+Knn35Cz549la6HiKqZxMREpKamKtJXTEyMIv0QEVWUzpO0GzVqpHQtRFTNJCYmwtfXDzk52Yr2m5f7SNH+iIjKS6eANGXKFHz55Zf46quveHqNqBZLTU1FTk42AkbMgtrVq8L9JV2IwsVt3yI/P7/ixRERVYBOAemPP/7AgQMHsHPnTjRr1gympqZayzdv3qxIcURUPahdveDg0aTC/WQkXa94MURECtApINnb2+O1115TuhYiIiIig6BTQIqIiFC6DiIiIiKDodOz2AAgPz8f+/btwzfffIMHDx4AAG7fvo3MzEzFiisoKMCMGTPg7e0NS0tLNGzYEHPnzoUQQrOOEAIzZ86Eq6srLC0tERQUhLi4OMVqICIiotpHpyNIN27cQPfu3ZGYmIjc3Fy88sorsLW1xYIFC5Cbm4uVK1cqUtyCBQuwYsUKrF27Fs2aNcPJkycxfPhw2NnZYcKECQCAhQsXYunSpVi7di28vb0xY8YMBAcH4/Lly7CwsFCkDiIiIqpddDqCNHHiRDz33HO4d+8eLC0tNe2vvfYaIiMjFSvuzz//RJ8+fdCrVy94eXnhP//5D7p164bjx48DkI4eLVmyBB999BH69OmDli1bYt26dbh9+za2bt2qWB1ERERUu+gUkI4cOYKPPvoIZmZmWu1eXl74+++/FSkMANq3b4/IyEjExsYCAM6dO4c//vgDPXr0AAAkJCQgOTkZQUFBmm3s7OwQEBCAqKioUvvNzc1FRkaG1ouIiIioiE6n2AoLC1FQUFCs/datW7C1ta1wUUU++OADZGRkwNfXF8bGxigoKMAnn3yCkJAQAEBycjIAwNnZWWs7Z2dnzbKShIeHY/bs2YrVSURERDWLTgGpW7duWLJkCb799lsAgEqlQmZmJmbNmqXo40d++uknrF+/Hhs2bECzZs1w9uxZTJo0CW5ubhg6dKjO/YaFhWHy5Mma9xkZGXB3d1eiZCKDpuRjQQA+GoSIai6dAtLnn3+O4OBgNG3aFA8fPsSgQYMQFxcHR0dH/Pe//1WsuGnTpuGDDz7AgAEDAAAtWrTAjRs3EB4ejqFDh8LFxQUAkJKSAldXV812KSkpaN26dan9mpubw9zcXLE6iaqDynosCMBHgxBRzaNTQKpfvz7OnTuHjRs34vz588jMzERoaChCQkK0Jm1XVHZ2NoyMtKdJGRsbo7CwEADg7e0NFxcXREZGagJRRkYGoqOjMWbMGMXqIKoJlH4sCMBHgxBRzaVTQAIAExMTDB48WMlaiunduzc++eQTeHh4oFmzZjhz5gwWL16MESNGAJBO7U2aNAnz5s1D48aNNZf5u7m5oW/fvpVaG1F1pdRjQQA+GsTQKX0K1NHRER4eHor2SWSodApI69ate+ryIUOG6FSM3LJlyzBjxgy8++67uHPnDtzc3DB69GjMnDlTs87777+PrKwsjBo1Cvfv30fHjh2xa9cu3gOJiGqtnPS7AFSK/59YS0srXLkSw5BEtYJOAWnixIla7/Py8pCdnQ0zMzNYWVkpFpBsbW2xZMkSLFmypNR1VCoV5syZgzlz5ijymURE1V1e9gMAAq0HTUc9b19F+sxIuo7o1bORmprKgES1gk4B6d69e8Xa4uLiMGbMGEybNq3CRRERUcXZOHkodjqVqLbR+Vlsco0bN8b8+fOLHV0iIiIiqm4UC0iANHH79u3bSnZJREREVOV0OsW2bds2rfdCCCQlJeGrr75Chw4dFCmMiIiISF90CkjyS+hVKhXq1auHl19+GZ9//rkSdRERERHpjc7PYiMiIiKqqRSdg0RERERUE+h0BOnJB70+y+LFi3X5CCIiIiK90SkgnTlzBmfOnEFeXh6aNJHusREbGwtjY2O0adNGs55KpVKmSiIiIqIqpFNA6t27N2xtbbF27VrUqVMHgHTzyOHDh6NTp06YMmWKokUSERERVSWd5iB9/vnnCA8P14QjAKhTpw7mzZvHq9iIiIio2tMpIGVkZOCff/4p1v7PP//gwYMHFS6KiIiISJ90CkivvfYahg8fjs2bN+PWrVu4desWfvnlF4SGhqJfv35K10hERERUpXSag7Ry5UpMnToVgwYNQl5entSRiQlCQ0OxaNEiRQskIiIiqmo6BSQrKyt8/fXXWLRoEa5duwYAaNiwIaytrRUtjoiIiEgfKnSjyKSkJCQlJaFx48awtraGEEKpuoiIiIj0RqeAdPfuXXTt2hU+Pj7o2bMnkpKSAAChoaG8xJ+IiIiqPZ0C0nvvvQdTU1MkJibCyspK0/7mm29i165dihVHREREpA86zUHas2cPdu/ejfr162u1N27cGDdu3FCkMCIiIiJ90ekIUlZWltaRoyJpaWkwNzevcFFERERE+qRTQOrUqRPWrVunea9SqVBYWIiFCxfipZdeUqw4IiIiIn3Q6RTbwoUL0bVrV5w8eRKPHj3C+++/j0uXLiEtLQ1Hjx5VukYiIiKiKqXTEaTmzZsjNjYWHTt2RJ8+fZCVlYV+/frhzJkzaNiwodI1EhEREVWpch9BysvLQ/fu3bFy5Up8+OGHlVETERERkV6V+wiSqakpzp8/Xxm1EBERERkEnU6xDR48GKtWrVK6FiIiIiKDoNMk7fz8fKxevRr79u1D27Ztiz2DbfHixYoUR0RERKQP5QpIf/31F7y8vHDx4kW0adMGABAbG6u1jkqlUq46IiIiIj0oV0Bq3LgxkpKScODAAQDSo0WWLl0KZ2fnSimOiIiISB/KNQdJCKH1fufOncjKylK0ICIiIiJ902mSdhF5YCIiIiKqCcoVkFQqVbE5RpxzRERERDVNueYgCSEwbNgwzQNpHz58iHfeeafYVWybN29WrkIiIqqREhMTkZqaqmifjo6O8PDwULRPqp3KFZCGDh2q9X7w4MGKFkNERLVDfkEBfH39kJOTrWi/lpZWuHIlhiGJKqxcASkiIqKy6iAiolqksLAQOTnZCBgxC2pXL0X6zEi6jujVs5GamsqARBWm040iiYiIlKB29YKDRxN9l0FUTIWuYqsKf//9NwYPHoy6devC0tISLVq0wMmTJzXLhRCYOXMmXF1dYWlpiaCgIMTFxemxYiIiIqruDDog3bt3Dx06dICpqSl27tyJy5cv4/PPP0edOnU06yxcuBBLly7FypUrER0dDWtrawQHB+Phw4d6rJyIiIiqM4M+xbZgwQK4u7trzX3y9vbW/LcQAkuWLMFHH32EPn36AADWrVsHZ2dnbN26FQMGDKjymomIiKj6M+gjSNu2bcNzzz2H119/HU5OTvD398d3332nWZ6QkIDk5GQEBQVp2uzs7BAQEICoqKhS+83NzUVGRobWi4iIiKiIQQekv/76CytWrEDjxo2xe/dujBkzBhMmTMDatWsBAMnJyQBQ7Flwzs7OmmUlCQ8Ph52dnebl7u5eeYMgIiKiasegA1JhYSHatGmDTz/9FP7+/hg1ahRGjhyJlStXVqjfsLAwpKena143b95UqGIiIiKqCQw6ILm6uqJp06ZabX5+fkhMTAQAuLi4AABSUlK01klJSdEsK4m5uTnUarXWi4iIiKiIQQekDh064OrVq1ptsbGx8PT0BCBN2HZxcUFkZKRmeUZGBqKjoxEYGFiltRIREVHNYdBXsb333nto3749Pv30U7zxxhs4fvw4vv32W3z77bcApAflTpo0CfPmzUPjxo3h7e2NGTNmwM3NDX379tVv8URERFRtGXRAev7557FlyxaEhYVhzpw58Pb2xpIlSxASEqJZ5/3330dWVhZGjRqF+/fvo2PHjti1axcsLCz0WDkRERFVZwYdkADg1VdfxauvvlrqcpVKhTlz5mDOnDlVWBURUe0UExNToe2b5+XBDEB+Xp4yBRFVEoMPSEREpH856XcBqDB48OAK9XMTQH0AaffuAQDych9VuDaiysCAREREz5SX/QCAQOtB01HP21fnfsyXTgYe3IOphTXwMAv5+fnKFUmkIAYkIiIqMxsnDzh4NNF5eyNj6c+OysigL6ImMuzL/ImIiIj0gQGJiIiISIYBiYiIiEiGAYmIiIhIhgGJiIiISIZXsREZsMTERKSmpirSV0Vv8EdEVJswIBEZqMTERPj6+iEnJ1vRfnljPiKiZ2NAIjJQqampyMnJRsCIWVC7elW4v6QLUbi47VvemI+IqAwYkIgMnNrVq0I35iuSkXS94sUQEdUSnKRNREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkU60C0vz586FSqTBp0iRN28OHDzF27FjUrVsXNjY26N+/P1JSUvRXJBEREVV71SYgnThxAt988w1atmyp1f7ee+/ht99+w6ZNm3Do0CHcvn0b/fr101OVREREVBNUi4CUmZmJkJAQfPfdd6hTp46mPT09HatWrcLixYvx8ssvo23btoiIiMCff/6JY8eO6bFiIiIiqs6qRUAaO3YsevXqhaCgIK32U6dOIS8vT6vd19cXHh4eiIqKKrW/3NxcZGRkaL2IiIiIipjou4Bn2bhxI06fPo0TJ04UW5acnAwzMzPY29trtTs7OyM5ObnUPsPDwzF79mylSyUiIqIawqCPIN28eRMTJ07E+vXrYWFhoVi/YWFhSE9P17xu3rypWN9ERERU/Rl0QDp16hTu3LmDNm3awMTEBCYmJjh06BCWLl0KExMTODs749GjR7h//77WdikpKXBxcSm1X3Nzc6jVaq0XERERURGDPsXWtWtXXLhwQatt+PDh8PX1xfTp0+Hu7g5TU1NERkaif//+AICrV68iMTERgYGB+iiZiIiIagCDDki2trZo3ry5Vpu1tTXq1q2raQ8NDcXkyZPh4OAAtVqN8ePHIzAwEC+88II+SqZaLDExEampqYr1FxMTo1hfRERUPgYdkMriiy++gJGREfr374/c3FwEBwfj66+/1ndZVMskJibC19cPOTnZivedl/tI8T6JiOjpql1AOnjwoNZ7CwsLLF++HMuXL9dPQUQAUlNTkZOTjYARs6B29VKkz6QLUbi47Vvk5+cr0h8REZVdtQtIRIZM7eoFB48mivSVkXRdkX6IiKj8DPoqNiIiIiJ9YEAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKS4aNGqNZKTExEamqqIn3FxMQo0g8RVZySv4+Ojo7w8PBQrD+qPhiQqFZKTEyEr68fcnKyFe03L/eRov0RUdnlpN8FoMLgwYMV69PS0gpXrsQwJNVCDEhUK6WmpiInJxsBI2ZB7epV4f6SLkTh4rZvkZ+fX/HiiEgnedkPAAi0HjQd9bx9K9xfRtJ1RK+ejdTUVAakWogBiWo1tasXHDyaVLifjKTrFS+GiBRh4+ShyO811W6cpE1EREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkw4BEREREJMOARERERCTDgEREREQkY6LvAoiIiAxZTEyMov05OjrCw8ND0T5JeQxIREREJchJvwtAhcGDByvar6WlFa5ciWFIMnAGHZDCw8OxefNmXLlyBZaWlmjfvj0WLFiAJk2aaNZ5+PAhpkyZgo0bNyI3NxfBwcH4+uuv4ezsrMfKiYiousvLfgBAoPWg6ajn7atInxlJ1xG9ejZSU1MZkAycQQekQ4cOYezYsXj++eeRn5+P//u//0O3bt1w+fJlWFtbAwDee+89bN++HZs2bYKdnR3GjRuHfv364ejRo3qunoiIagIbJw84eDR59opUoxh0QNq1a5fW+zVr1sDJyQmnTp3Ciy++iPT0dKxatQobNmzAyy+/DACIiIiAn58fjh07hhdeeEEfZRMREVE1Z9ABSS49PR0A4ODgAAA4deoU8vLyEBQUpFnH19cXHh4eiIqKKjUg5ebmIjc3V/M+IyOjEqsmJSQmJiI1NVWx/pSedElERDVLtQlIhYWFmDRpEjp06IDmzZsDAJKTk2FmZgZ7e3utdZ2dnZGcnFxqX+Hh4Zg9e3ZllksKSkxMhK+vH3JyshXvOy/3keJ9EhFR9VdtAtLYsWNx8eJF/PHHHxXuKywsDJMnT9a8z8jIgLu7e4X7pcqRmpqKnJxsBIyYBbWrlyJ9Jl2IwsVt3yI/P1+R/oiIqGapFgFp3Lhx+P3333H48GHUr19f0+7i4oJHjx7h/v37WkeRUlJS4OLiUmp/5ubmMDc3r8ySqRKoXb0UmyiZkXRdkX6IiKhmMug7aQshMG7cOGzZsgX79++Ht7e31vK2bdvC1NQUkZGRmrarV68iMTERgYGBVV0uERER1RAGfQRp7Nix2LBhA3799VfY2tpq5hXZ2dnB0tISdnZ2CA0NxeTJk+Hg4AC1Wo3x48cjMDCQV7ARERGRzgw6IK1YsQIA0KVLF632iIgIDBs2DADwxRdfwMjICP3799e6USQRERGRrgw6IAkhnrmOhYUFli9fjuXLl1dBRURERFQbGPQcJCIiIiJ9YEAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpIx0XcBREREtU1MTIxifTk6OsLDw0Ox/kjCgERERFRFctLvAlBh8ODBivVpaWmFK1diGJIUxoBERERURfKyHwAQaD1oOup5+1a4v4yk64hePRupqakMSApjQCIiIqpiNk4ecPBoou8y6Ck4SZuIiIhIhgGJiIiISIYBiYiIiEiGAYmIiIhIhgGJiIiISIZXsRESExORmpqqaJ+8cRkRUfWl9N+F6vg3gQGplktMTISvrx9ycrIV7Zc3LiMiqp4q4+9CdfybwIBUy6WmpiInJxsBI2ZB7eqlSJ+8cRkRUfWl9N+F6vo3gQGJAABqVy/etIyIiDRq+98FTtImIiIikuERpGpIyclzSj5RmoiI9EPJf8v5d0HCgFTNVNak6rzcR4r2R0RElS8n/S4AFQYPHqx437X97wIDUjWj9OS5pAtRuLjtW+Tn51e8OCIiqlJ52Q8ACLQeNB31vH0V6ZN/FyQMSNWUUpPnMpKuV7wYIiLSKxsnD8UmVPPvgqTGBKTly5dj0aJFSE5ORqtWrbBs2TK0a9dO32UpfrOt6nRuWKlaq9OYiYioZqgRAenHH3/E5MmTsXLlSgQEBGDJkiUIDg7G1atX4eTkpLe6Kmu+EGDY54Yr65y4IY+ZiIhqlhoRkBYvXoyRI0di+PDhAICVK1di+/btWL16NT744AO91VUZN2GsDueGlT4nXh3GTERENUu1D0iPHj3CqVOnEBYWpmkzMjJCUFAQoqKiStwmNzcXubm5mvfp6ekAgIyMDEVry8zMBADkP8pFfm6OIn0W5ElHUdL/joOpiarC/WUk3VC0vyf7LMhTZtxKjxlQftyV+T2yRsPqkzVWrM/0/DyYAsgoKFCkvyfVpu+xsvqrjD4zkhMBSH8Tlf47W9SfEELRfos6rdb+/vtvAUD8+eefWu3Tpk0T7dq1K3GbWbNmCQB88cUXX3zxxVcNeN28eVPxfFHtjyDpIiwsDJMnT9a8LywsRFpaGurWrQuVSpkEXhkyMjLg7u6OmzdvQq1W67ucSlHTx8jxVX81fYwcX/VX08f45PhsbW3x4MEDuLm5Kf451T4gOTo6wtjYGCkpKVrtKSkpcHFxKXEbc3NzmJuba7XZ29tXVomKU6vVNfKH/kk1fYwcX/VX08fI8VV/NX2MReOzs7OrlP6r/bPYzMzM0LZtW0RGRmraCgsLERkZicDAQD1WRkRERNVVtT+CBACTJ0/G0KFD8dxzz6Fdu3ZYsmQJsrKyNFe1EREREZVHjQhIb775Jv755x/MnDkTycnJaN26NXbt2gVnZ2d9l6Yoc3NzzJo1q9jpwZqkpo+R46v+avoYOb7qr6aPsarGpxKiMq6NIyIiIqq+qv0cJCIiIiKlMSARERERyTAgEREREckwIBERERHJMCAZmE8++QTt27eHlZVVmW9eKYTAzJkz4erqCktLSwQFBSEuLk5rnbS0NISEhECtVsPe3h6hoaGaZ8VVpfLWcf36dahUqhJfmzZt0qxX0vKNGzdWxZC06PI9d+nSpVjt77zzjtY6iYmJ6NWrF6ysrODk5IRp06bp7eG95R1jWloaxo8fjyZNmsDS0hIeHh6YMGGC5hmIRfS1D5cvXw4vLy9YWFggICAAx48ff+r6mzZtgq+vLywsLNCiRQvs2LFDa3lZfh+rWnnG+N1336FTp06oU6cO6tSpg6CgoGLrDxs2rNi+6t69e2UPo1TlGd+aNWuK1W5hYaG1jqHtw/KMr6R/T1QqFXr16qVZx5D23+HDh9G7d2+4ublBpVJh69atz9zm4MGDaNOmDczNzdGoUSOsWbOm2Drl/b0ukeIPL6EKmTlzpli8eLGYPHmysLOzK9M28+fPF3Z2dmLr1q3i3Llz4t///rfw9vYWOTk5mnW6d+8uWrVqJY4dOyaOHDkiGjVqJAYOHFhJoyhdeevIz88XSUlJWq/Zs2cLGxsb8eDBA816AERERITWek+Ov6ro8j137txZjBw5Uqv29PR0zfL8/HzRvHlzERQUJM6cOSN27NghHB0dRVhYWGUPp0TlHeOFCxdEv379xLZt20R8fLyIjIwUjRs3Fv3799daTx/7cOPGjcLMzEysXr1aXLp0SYwcOVLY29uLlJSUEtc/evSoMDY2FgsXLhSXL18WH330kTA1NRUXLlzQrFOW38eqVN4xDho0SCxfvlycOXNGxMTEiGHDhgk7Oztx69YtzTpDhw4V3bt319pXaWlpVTUkLeUdX0REhFCr1Vq1Jycna61jSPuwvOO7e/eu1tguXrwojI2NRUREhGYdQ9p/O3bsEB9++KHYvHmzACC2bNny1PX/+usvYWVlJSZPniwuX74sli1bJoyNjcWuXbs065T3OysNA5KBioiIKFNAKiwsFC4uLmLRokWatvv37wtzc3Px3//+VwghxOXLlwUAceLECc06O3fuFCqVSvz999+K114apepo3bq1GDFihFZbWX6xKpuu4+vcubOYOHFiqct37NghjIyMtP4RX7FihVCr1SI3N1eR2stKqX34008/CTMzM5GXl6dp08c+bNeunRg7dqzmfUFBgXBzcxPh4eElrv/GG2+IXr16abUFBASI0aNHCyHK9vtY1co7Rrn8/Hxha2sr1q5dq2kbOnSo6NOnj9Kl6qS843vWv62Gtg8ruv+++OILYWtrKzIzMzVthrT/nlSWfwPef/990axZM622N998UwQHB2veV/Q7K8JTbNVcQkICkpOTERQUpGmzs7NDQEAAoqKiAABRUVGwt7fHc889p1knKCgIRkZGiI6OrrJalajj1KlTOHv2LEJDQ4stGzt2LBwdHdGuXTusXr0aoopv8VWR8a1fvx6Ojo5o3rw5wsLCkJ2drdVvixYttG58GhwcjIyMDFy6dEn5gTyFUj9L6enpUKvVMDHRvldtVe7DR48e4dSpU1q/O0ZGRggKCtL87shFRUVprQ9I+6Jo/bL8PlYlXcYol52djby8PDg4OGi1Hzx4EE5OTmjSpAnGjBmDu3fvKlp7Weg6vszMTHh6esLd3R19+vTR+j0ypH2oxP5btWoVBgwYAGtra612Q9h/unjW76AS31mRGnEn7dosOTkZAIrdNdzZ2VmzLDk5GU5OTlrLTUxM4ODgoFmnKihRx6pVq+Dn54f27dtrtc+ZMwcvv/wyrKyssGfPHrz77rvIzMzEhAkTFKv/WXQd36BBg+Dp6Qk3NzecP38e06dPx9WrV7F582ZNvyXt36JlVUmJfZiamoq5c+di1KhRWu1VvQ9TU1NRUFBQ4nd75cqVErcpbV88+btW1FbaOlVJlzHKTZ8+HW5ublp/cLp3745+/frB29sb165dw//93/+hR48eiIqKgrGxsaJjeBpdxtekSROsXr0aLVu2RHp6Oj777DO0b98ely5dQv369Q1qH1Z0/x0/fhwXL17EqlWrtNoNZf/porTfwYyMDOTk5ODevXsV/pkvwoBUBT744AMsWLDgqevExMTA19e3iipSVlnHV1E5OTnYsGEDZsyYUWzZk23+/v7IysrCokWLFPnjWtnjezIotGjRAq6urujatSuuXbuGhg0b6txveVTVPszIyECvXr3QtGlTfPzxx1rLKnMfkm7mz5+PjRs34uDBg1oTmQcMGKD57xYtWqBly5Zo2LAhDh48iK5du+qj1DILDAzUepB5+/bt4efnh2+++QZz587VY2XKW7VqFVq0aIF27dpptVfn/VeVGJCqwJQpUzBs2LCnrtOgQQOd+nZxcQEApKSkwNXVVdOekpKC1q1ba9a5c+eO1nb5+flIS0vTbF8RZR1fRev4+eefkZ2djSFDhjxz3YCAAMydOxe5ubkVfl5PVY2vSEBAAAAgPj4eDRs2hIuLS7ErMFJSUgBAkf0HVM0YHzx4gO7du8PW1hZbtmyBqanpU9dXch+WxNHREcbGxprvskhKSkqpY3FxcXnq+mX5faxKuoyxyGeffYb58+dj3759aNmy5VPXbdCgARwdHREfH1+lf2ArMr4ipqam8Pf3R3x8PADD2ocVGV9WVhY2btyIOXPmPPNz9LX/dFHa76BarYalpSWMjY0r/DOhUa4ZS1RlyjtJ+7PPPtO0paenlzhJ++TJk5p1du/erbdJ2rrW0blz52JXPpVm3rx5ok6dOjrXqgulvuc//vhDABDnzp0TQjyepP3kFRjffPONUKvV4uHDh8oNoAx0HWN6erp44YUXROfOnUVWVlaZPqsq9mG7du3EuHHjNO8LCgrEv/71r6dO0n711Ve12gIDA4tN0n7a72NVK+8YhRBiwYIFQq1Wi6ioqDJ9xs2bN4VKpRK//vprhestL13G96T8/HzRpEkT8d577wkhDG8f6jq+iIgIYW5uLlJTU5/5Gfrcf09CGSdpN2/eXKtt4MCBxSZpV+RnQlNPudamSnfjxg1x5swZzaXsZ86cEWfOnNG6pL1JkyZi8+bNmvfz588X9vb24tdffxXnz58Xffr0KfEyf39/fxEdHS3++OMP0bhxY71d5v+0Om7duiWaNGkioqOjtbaLi4sTKpVK7Ny5s1if27ZtE9999524cOGCiIuLE19//bWwsrISM2fOrPTxyJV3fPHx8WLOnDni5MmTIiEhQfz666+iQYMG4sUXX9RsU3SZf7du3cTZs2fFrl27RL169fR6mX95xpieni4CAgJEixYtRHx8vNalxfn5+UII/e3DjRs3CnNzc7FmzRpx+fJlMWrUKGFvb6+5YvCtt94SH3zwgWb9o0ePChMTE/HZZ5+JmJgYMWvWrBIv83/W72NVKu8Y58+fL8zMzMTPP/+sta+K/g168OCBmDp1qoiKihIJCQli3759ok2bNqJx48ZVHth1Gd/s2bPF7t27xbVr18SpU6fEgAEDhIWFhbh06ZJmHUPah+UdX5GOHTuKN998s1i7oe2/Bw8eaP7OARCLFy8WZ86cETdu3BBCCPHBBx+It956S7N+0WX+06ZNEzExMWL58uUlXub/tO+srBiQDMzQoUMFgGKvAwcOaNbB/+4XU6SwsFDMmDFDODs7C3Nzc9G1a1dx9epVrX7v3r0rBg4cKGxsbIRarRbDhw/XCl1V5Vl1JCQkFBuvEEKEhYUJd3d3UVBQUKzPnTt3itatWwsbGxthbW0tWrVqJVauXFniupWtvONLTEwUL774onBwcBDm5uaiUaNGYtq0aVr3QRJCiOvXr4sePXoIS0tL4ejoKKZMmaJ1iXxVKu8YDxw4UOLPNACRkJAghNDvPly2bJnw8PAQZmZmol27duLYsWOaZZ07dxZDhw7VWv+nn34SPj4+wszMTDRr1kxs375da3lZfh+rWnnG6OnpWeK+mjVrlhBCiOzsbNGtWzdRr149YWpqKjw9PcXIkSPL/cdHSeUZ36RJkzTrOjs7i549e4rTp09r9Wdo+7C8P6NXrlwRAMSePXuK9WVo+6+0fx+KxjR06FDRuXPnYtu0bt1amJmZiQYNGmj9PSzytO+srFRCVPG10EREREQGjvdBIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiIiIikmFAIiIiIpJhQCIiIiKSYUAiqkIHDx6ESqXC/fv3a9Vnl0SlUmHr1q0AgOvXr0OlUuHs2bOa5UePHkWLFi1gamqKvn37ltpWE7344ovYsGGD5v2T31Vt8vHHH5f5AbErV65E7969K7cgqlUYkKjW+eeffzBmzBh4eHjA3NwcLi4uCA4OxtGjRyv9s9u3b4+kpCTY2dlV6ud06dIFkyZN0stn68Ld3R1JSUlo3ry5pm3y5Mlo3bo1EhISsGbNmlLbappt27YhJSUFAwYM0LQlJSWhR48eeqzK8I0YMQKnT5/GkSNH9F0K1RAm+i6AqKr1798fjx49wtq1a9GgQQOkpKQgMjISd+/erfTPNjMzg4uLS6V/jqF99rMYGxsXq+3atWt45513UL9+/ae2ldejR49gZmam8/aVbenSpRg+fDiMjB7//1dD3W+GxMzMDIMGDcLSpUvRqVMnfZdDNUG5n95GVI3du3dPABAHDx585nqhoaHC0dFR2NraipdeekmcPXtWs/zs2bOiS5cuwsbGRtja2oo2bdqIEydOCCGkB8u++uqrwt7eXlhZWYmmTZtqHmha9GDGe/fuafr6+eefRdOmTYWZmZnw9PQUn332mVYtnp6e4pNPPhHDhw8XNjY2wt3dXXzzzTel1l7SA48TEhKKfXZERISws7MTv/32m/Dx8RGWlpaif//+IisrS6xZs0Z4enoKe3t7MX78eJGfn6/p/+HDh2LKlCnCzc1NWFlZiXbt2hV7uLBcbGys6NSpkzA3Nxd+fn5iz549AoDYsmWLEOLxA27PnDmj+e8nXxERESW2CSHEhQsXRPfu3YW1tbVwcnISgwcPFv/884/mszt37izGjh0rJk6cKOrWrSu6dOlS5u3Gjx8vpk2bJurUqSOcnZ01D2wtcu/ePTFq1Cjh5OQkzM3NRbNmzcRvv/2mWX7kyBHRsWNHYWFhIerXry/Gjx8vMjMzS/2e7ty5I1Qqlbh48aJWe0nf1S+//CK6dOkiLC0tRcuWLcWff/5Zar+FhYVi1qxZwt3dXZiZmQlXV1cxfvx4zfKy7NM//vhDdO7cWVhaWgp7e3vRrVs3kZaWptl+/Pjxol69esLc3Fx06NBBHD9+XLNt0c/evn37RNu2bYWlpaUIDAwUV65c0fqM8PBw4eTkJGxsbMSIESPE9OnTRatWrbT6ef7554WVlZWws7MT7du3F9evX9csP3TokDAzMxPZ2dmlfhdEZcWARLVKXl6esLGxEZMmTRIPHz4sdb2goCDRu3dvceLECREbGyumTJki6tatK+7evSuEEKJZs2Zi8ODBIiYmRsTGxoqffvpJE6B69eolXnnlFXH+/Hlx7do18dtvv4lDhw4JIYoHpJMnTwojIyMxZ84ccfXqVRERESEsLS21nk7t6ekpHBwcxPLly0VcXJwIDw8XRkZGxf64FLl//74IDAwUI0eOFElJSSIpKUnk5+eXGJBMTU3FK6+8Ik6fPi0OHTok6tatK7p16ybeeOMNcenSJfHbb78JMzMzsXHjRk3/b7/9tmjfvr04fPiwiI+PF4sWLRLm5uYiNja2xHoKCgpE8+bNRdeuXcXZs2fFoUOHhL+/f6kBKT8/XyQlJQm1Wi2WLFkikpKSRGZmZrG27Oxsce/ePVGvXj0RFhYmYmJixOnTp8Urr7wiXnrpJc3nd+7cWdjY2Ihp06aJK1euiCtXrpR5O7VaLT7++GMRGxsr1q5dK1QqleYJ6QUFBeKFF14QzZo1E3v27NHs6x07dgghhIiPjxfW1tbiiy++ELGxseLo0aPC399fDBs2rNSfu82bNwtra2tRUFCg1V7Sd+Xr6yt+//13cfXqVfGf//xHeHp6iry8vBL73bRpk1Cr1WLHjh3ixo0bIjo6Wnz77bdl3qdnzpwR5ubmYsyYMeLs2bPi4sWLYtmyZZpAOWHCBOHm5iZ27NghLl26JIYOHSrq1Kmj+X0p+tkLCAgQBw8eFJcuXRKdOnUS7du319Tw448/CnNzc/H999+LK1euiA8//FDY2tpqAlJeXp6ws7MTU6dOFfHx8eLy5ctizZo14saNG5o+srKyhJGR0TMDO1FZMCBRrfPzzz+LOnXqCAsLC9G+fXsRFhYmzp07p1l+5MgRoVariwWohg0bao7c2NraijVr1pTYf4sWLcTHH39c4jJ5SBk0aJB45ZVXtNaZNm2aaNq0qea9p6enGDx4sOZ9YWGhcHJyEitWrCh1jJ07dxYTJ0586mcXHZWJj4/XrDN69GhhZWUlHjx4oGkLDg4Wo0ePFkIIcePGDWFsbCz+/vtvrb67du0qwsLCSqxl9+7dwsTERGubnTt3lhqQitjZ2WkFxZLa5s6dK7p166a1zs2bNwUAcfXqVc134e/vr7VOWbfr2LGj1jrPP/+8mD59umZcRkZGmvXlQkNDxahRo7Tajhw5IoyMjEROTk6J23zxxReiQYMGxdpL+q6+//57zfJLly4JACImJqbEfj///HPh4+MjHj16VGxZWfbpwIEDRYcOHUrsOzMzU5iamor169dr2h49eiTc3NzEwoULhRDaR5CKbN++XQDQfBeBgYHi3Xff1eo7ICBAE5Du3r1bpqO/derUKfV3k6g8OEmbap3+/fvj9u3b2LZtG7p3746DBw+iTZs2mkm/586dQ2ZmJurWrQsbGxvNKyEhAdeuXQMgTRZ+++23ERQUhPnz52vaAWDChAmYN28eOnTogFmzZuH8+fOl1hITE4MOHTpotXXo0AFxcXEoKCjQtLVs2VLz3yqVCi4uLrhz506FvwsrKys0bNhQ897Z2RleXl6wsbHRaiv6rAsXLqCgoAA+Pj5a382hQ4e0vgP5GN3d3eHm5qZpCwwMrHDtgLSvDhw4oFWLr68vAGjV07ZtW522e/J7BwBXV1fNd3H27FnUr18fPj4+pda2Zs0arc8IDg5GYWEhEhISStwmJycHFhYWZRr7k7W5uroCQKk/E6+//jpycnLQoEEDjBw5Elu2bEF+fj6Asu3Ts2fPomvXriX2fe3aNeTl5Wn9HJuamqJdu3aIiYkpc80xMTEICAjQWv/JnxMHBwcMGzYMwcHB6N27N7788kskJSUVq8fS0hLZ2dkl1kpUHpykTbWShYUFXnnlFbzyyiuYMWMG3n77bcyaNQvDhg1DZmYmXF1dcfDgwWLb2dvbA5AuPx40aBC2b9+OnTt3YtasWdi4cSNee+01vP322wgODsb27duxZ88ehIeH4/PPP8f48eN1rtfU1FTrvUqlQmFhoc79Pa3fp31WZmYmjI2NcerUKRgbG2ut92SoqiqZmZno3bs3FixYUGxZ0R9gALC2ttZpu6d9F5aWls+sbfTo0ZgwYUKxZR4eHiVu4+joiHv37j2135JqU6lUAFDqz4S7uzuuXr2Kffv2Ye/evXj33XexaNEiHDp0qEz79FljLavy1FySiIgITJgwAbt27cKPP/6Ijz76CHv37sULL7ygWSctLQ316tVTpF6q3RiQiAA0bdpUc5+ZNm3aIDk5GSYmJvDy8ip1Gx8fH/j4+OC9997DwIEDERERgddeew2A9AfpnXfewTvvvIOwsDB89913JQYkPz+/YrcXOHr0KHx8fIr9sSoPMzMzrSNQSvH390dBQQHu3LlT5iuF/Pz8cPPmTSQlJWnCx7FjxxSpp02bNvjll1/g5eUFE5Oy/3Om63ZPatmyJW7duoXY2NgSjyK1adMGly9fRqNGjcrcp7+/P5KTk3Hv3j3UqVNHp7pKY2lpid69e6N3794YO3YsfH19ceHChTLt05YtWyIyMhKzZ88utqxhw4YwMzPD0aNH4enpCQDIy8vDiRMnit1q4mn8/PwQHR2NIUOGaNpK+jnx9/eHv78/wsLCEBgYiA0bNmgC0rVr1/Dw4UP4+/uX+XOJSsNTbFSr3L17Fy+//DJ++OEHnD9/HgkJCdi0aRMWLlyIPn36AACCgoIQGBiIvn37Ys+ePbh+/Tr+/PNPfPjhhzh58iRycnIwbtw4HDx4EDdu3MDRo0dx4sQJ+Pn5AQAmTZqE3bt3IyEhAadPn8aBAwc0y+SmTJmCyMhIzJ07F7GxsVi7di2++uorTJ06tULj9PLyQnR0NK5fv47U1FRFjjYBUigMCQnBkCFDsHnzZiQkJOD48eMIDw/H9u3bS9wmKCgIPj4+GDp0KM6dO4cjR47gww8/VKSesWPHIi0tDQMHDsSJEydw7do17N69G8OHD39qQNR1uyd17twZL774Ivr374+9e/ciISEBO3fuxK5duwAA06dPx59//olx48bh7NmziIuLw6+//opx48aV2qe/vz8cHR0VvyfXmjVrsGrVKly8eBF//fUXfvjhB1haWsLT07NM+zQsLAwnTpzAu+++i/Pnz+PKlStYsWIFUlNTYW1tjTFjxmDatGnYtWsXLl++jJEjRyI7OxuhoaFlrnHixIlYvXo1IiIiEBsbi1mzZuHSpUua5QkJCQgLC0NUVBRu3LiBPXv2IC4uTut368iRI2jQoIHWaWMiXTEgUa1iY2ODgIAAfPHFF3jxxRfRvHlzzJgxAyNHjsRXX30FQDr0v2PHDrz44osYPnw4fHx8MGDAANy4cQPOzs4wNjbG3bt3MWTIEPj4+OCNN95Ajx49NP/vuqCgAGPHjoWfnx+6d+8OHx8ffP311yXW06ZNG/z000/YuHEjmjdvjpkzZ2LOnDkYNmxYhcY5depUGBsbo2nTpqhXrx4SExMr1N+TIiIiMGTIEEyZMgVNmjRB3759ceLEiVJPGxkZGWHLli3IyclBu3bt8Pbbb+OTTz5RpBY3NzccPXoUBQUF6NatG1q0aIFJkybB3t5e6z5CSm0n98svv+D555/HwIED0bRpU7z//vuagNWyZUscOnQIsbGx6NSpE/z9/TFz5kytuVhyxsbGGD58ONavX1/2L6EM7O3t8d1336FDhw5o2bIl9u3bh99++w1169YF8Ox96uPjgz179uDcuXNo164dAgMD8euvv2qOvs2fPx/9+/fHW2+9hTZt2iA+Ph67d+8u11GwN998EzNmzMD777+Ptm3b4saNGxgzZoxmuZWVFa5cuYL+/fvDx8cHo0aNwtixYzF69GjNOv/9738xcuRIJb4yIqiEEELfRRARkSQ5ORnNmjXD6dOnNaes6NkuXbqEl19+GbGxsQZ5t3iqfngEiYjIgLi4uGDVqlWKHvWrDZKSkrBu3TqGI1IMjyARERERyfAIEhEREZEMAxIRERGRDAMSERERkQwDEhEREZEMAxIRERGRDAMSERERkQwDEhEREZEMAxIRERGRDAMSERERkcz/B1tJLaiiz7swAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "sns.histplot(perm_diffs, ax=ax)\n",
    "ax.vlines([mean_B - mean_A], color='red', ymin=0, ymax=150, lw=2)\n",
    "ax.text(mean_B - mean_A + 0.01, 140, 'Observed\\ndifference', color='red')\n",
    "ax.set_xlabel('Session time difference (in seconds)')\n",
    "ax.set_ylabel('Frequency')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.135"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(perm_diffs > mean_B - mean_A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Statistical Significance and P-Values`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Example- Conversion Rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observed difference: 0.0368%\n"
     ]
    }
   ],
   "source": [
    "observed_pct_diff = 100 * (200/23739 - 182/22588)\n",
    "print(f'Observed difference: {observed_pct_diff:.4f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "conversion = [0] * 45945\n",
    "conversion.extend([1] * 382)\n",
    "conversion = pd.Series(conversion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "perm_diff_conv = [100 * perm_func(conversion, 23739, 22588) for _ in range(1000)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAGwCAYAAABPSaTdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAABFj0lEQVR4nO3deVxVdf7H8fdVZFFZJGVLEMsFK/ctJssWEq1xNJ1KxTJ1skUzs8WcLLNFW8wcHdNqymWyn+aU1lhqRpotRG5ZJqKSiiVoRIKIIsv398cdr90El8uFezm+no/HfeTZvvdzvlG8/Z7vOcdmjDECAACwqFqeLgAAAKAqEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAICl+Xi6AG9QVlam/fv3KzAwUDabzdPlAACAs2CM0eHDhxUVFaVatSoevyHsSNq/f7+io6M9XQYAAHDBvn371Lhx4wq3E3YkBQYGSrJ3VlBQkIerAQAAZyM/P1/R0dGO3+MVIexIjktXQUFBhB0AAGqYM01BYYIyAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOANQ0sbHS9OmersJ9rHY+8DqEHQDwJvv2ScOGSVFRkq+v1KSJdP/90q+/eroyoMYi7ACAt/jxR6lTJ2nnTun//k/atUuaM0dKTpbi46XcXM/UVVoqlZV55rsBNyDsAEBVmzZNevJJ+z9PZ+RI+2jOxx9L3btLMTFSr17SJ59IP/8sPfbYyX0PH5YGDpTq1ZMuvFCaNevkNmPs3xcTI/n52UeJRo8+ub2oSHroIftx9epJXbtKa9ee3D5vnhQSIn3wgXTJJfY2/vUvyd9fOnTIueb775euvfbk8hdfSFdeKQUESNHR9u89cuTk9oMHpd697dubNpUWLjybHgQqhbADAFVt2jRp0qTTh53cXGnVKunee+1B4PciIqSkJGnxYnuQkaQXX5TatpU2b5YefdQeOlavtm97913p5ZelV1+1jxItWya1bn2yvVGjpJQUadEi6bvvpJtvlnr2tO97QmGh9Pzz9pDzww/27w8Jsbd9QmmpvaakJPtyRoa9nf797e0uXmwPP6NGnTzmjjvsl+rWrJH+8x/plVfsAQioQj6eLgAAIHvQMEZq1ar87a1aSb/9Jv3yi335iivsIUeSWrSQvvzSHnCuv17KzLQHpIQEqU4d+whPly72fTMzpblz7f+MirKve+ghaeVK+/rJk+3riovtQaRt25M1DBggvf22NHy4fTk52T7S07+/fXnKFHvwGTPGvty8uTRjhn2UavZs+3euWCF9843UubN9nzfeqPicATdhZAcAvMmJkZsziY8/dTktzf7nm2+Wjh6VLrpIuvNOaelSqaTEvu377+0jMi1aSPXrn/x89pl9ZOYEX1+pTRvn70hKsl/u2r/fvrxwoXTjjfYRH0nassV+Cez37SYm2uf77N5tr8/HR+rY8WSbcXEnjweqCCM7AOANmjWTbDZ7ILjpplO3p6VJDRpIjRqdua3oaCk93T7XZ/Vq+6WxF1+0B5qCAql2bWnjRvs/f69+/ZN/Dgiw1/N7nTtLF19sv/x1zz32EDVv3sntBQXSXXc5zw86ISZG2rHjzLUDVYCwAwDe4IIL7JegXnlFeuAB53k72dn2UZTbbz8ZQL7+2vn4r792vhwUEGCfCNy7t33ic1ycfVSnfXv7yM7Bg/aJxOcqKcleS+PGUq1a9pGdEzp0kLZtswe38sTF2UeYNm48eRkrPf3USc+Am3EZCwC8xT//ab9TKjFRWrfOPpF35Up7CLrwQunZZ0/u++WX0gsv2EdLZs2SliyxT1KW7KMtb7whbd1qv539rbfs4adJE/vlq6Qke3B67z375aVvvrHPt/nwwzPXmJQkbdpkr+Wvf7XfqXXCuHHSV1/ZJyR/+619HtL775+coNyypX0C8113Samp9tDzt7+dOiEbcDPCDgB4i+bNpQ0b7HNtbrnFfsloxAjpmmvsd0+Fhp7c98EH7fu2by8984z9Tq/ERPu2kBDp9dftk5jbtLFfzvrvf+2jR5J9IvLtt9vbaNlS6ttXWr/efqnpTJo1s092/u67k3dhndCmjf1S2Y4d9lGj9u2lJ544ORH6xHdHRdknLffrZz+/sLDK9BpwRjZjznY2nHXl5+crODhYeXl5CgoK8nQ5AKymcWP7c3IuvFD66SdPVwNYxtn+/mZkBwAAWBphBwAAWBphBwAAWBphBwAAWBphBwC82dVXn3z9QmysNH36yW3Z2fbb0uvVO/kU4vLWAec5HioIADXF+vX2EHPCyy9LWVn2Z9oEB1e8DjjPEXYAoKb446siMjLs75lq3vz0687V8eP2d2MBFsFlLADwFkeO2B/2V7++FBkpvfSS8/bfX8aKjZXefVdasMD+Cok77ih/nWR/HcPf/mYPS0FB0rXX2l/aecKTT0rt2kn/+pfUtKnk739ux/373/bvDg62vxn98OGT+5SV2Z/03KyZ/WnLMTHOT4Let8/+AMWQEPtDE/v0kfbsqUQnAqci7ACAt3j4YfsTiN9/X/r4Y/sbxjdtKn/f9evtr1645Rb7Zat//KP8dZL9LegHD0orVthf0dChg3TddVJu7sn2du2yB6X33rNfAjvb4zIypGXLpOXL7Z/PPpOee+7k9vHj7cuPP25/b9bbb0vh4fZtxcX2pz4HBkqff25/BUb9+vZzOH7cTZ0KcBkLALxDQYH9fVZvvWUPFJI0f7796cvladTIPlISECBFRJxc/8d1X3xhf/fVwYMn32M1dao9oPznP/bXNUj2cLFgwclLZWd7XFmZ/V1cgYH25dtuk5KT7aM3hw/bA9c//ykNGWLffvHFUrdu9j8vXmw//l//OvmC07lz7aM8a9dKPXq41JXAHxF2AMAbZGTYA0fXrifXhYba311VGVu22IPUifdinXD0qP07T2jSxHlO0NkeFxt7MuhI9stvBw/a/5yWZn+x6YnwVl5tu3Y5Hy9Jx445fwdQSR69jLVu3Tr17t1bUVFRstlsWrZsWYX73n333bLZbJr++9suJeXm5iopKUlBQUEKCQnR8OHDVVBQULWFA0BNUVBgDyDffuv8SU+3XzY74fd3eZ3LcXXqOB9ns9lHa6Qzv828oMA+mfqP37FjhzRo0LmdJ3AaHh3ZOXLkiNq2bathw4apX79+Fe63dOlSff3114r6/Ztz/ycpKUlZWVlavXq1iouLNXToUI0YMUJvv/12VZYOAO518cX24JCaevLt47/9Zv/F37276+126GB/9o6Pj30UpqqP+73mze2BJznZPtG5vO9YvNj+1nNewowq5NGRnV69eumZZ57RTTfdVOE+P//8s+677z4tXLhQdf7wN4i0tDStXLlS//rXv9S1a1d169ZNM2fO1KJFi7R///4K2ywqKlJ+fr7TBwA8qn59afhw+6jJp59KW7fa76aqVcn/TSckSPHxUt++9knPe/ZIX30lPfaYtGGD+4/7PX9/adw46ZFH7POBMjKkr7+2z02SpKQkqWFD+x1Yn38u7d5tn6szejRvh4dbefXdWGVlZbrtttv08MMP69JLLz1le0pKikJCQtSpUyfHuoSEBNWqVUupqakVtjtlyhQFBwc7PtHR0VVSPwCckxdflK68Uurd2x42unWzX+apDJtN+ugj6aqrpKFDpRYt7LeH79178q4odx73R48/Lj34oPTEE1KrVtKtt56c01O3rrRunX0kq18/+/bhw+1zdhjpgRvZjDHG00VIks1m09KlS9W3b1/HuilTpmjNmjVatWqVbDabYmNjNWbMGI3536PTJ0+erPnz5ys9Pd2prbCwME2aNEn33HNPud9VVFSkoqIix3J+fr6io6OVl5enIP4DA+BujRtLP/8sXXghIxaAG+Xn5ys4OPiMv7+99m6sjRs36h//+Ic2bdok24lbEt3Ez89PfidupQQAAJbmtZexPv/8cx08eFAxMTHy8fGRj4+P9u7dqwcffFCx/5ssFxERoYMnhkP/p6SkRLm5uYr4/XMnAADAectrR3Zuu+02JSQkOK1LTEzUbbfdpqFDh0qS4uPjdejQIW3cuFEd/3dd+9NPP1VZWZm6/v5ZFQAA4Lzl0bBTUFCgXbt2OZZ3796tb7/9VqGhoYqJidEFf3iYVZ06dRQREaGW/3vIVqtWrdSzZ0/deeedmjNnjoqLizVq1CgNGDCg3NvUAQDA+cejl7E2bNig9u3bq3379pKksWPHqn379nriiSfOuo2FCxcqLi5O1113nW644QZ169ZNr732WlWVDAAAahivuRvLk852NjcAuIS7sYAqcba/v712gjIAAIA7EHYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAIClEXYAAICl+Xi6AACoapmZmcrJyalUGw0bNlRMTIybKgJQnQg7ACwtMzNTcXGtdPRoYaXaCQioq+3b0wg8QA1E2AFgaTk5OTp6tFBdh01UUGSsS23kZ+1R6puTlJOTQ9gBaiDCDoDzQlBkrEJjWnq6DAAewARlAABgaR4NO+vWrVPv3r0VFRUlm82mZcuWObYVFxdr3Lhxat26terVq6eoqCjdfvvt2r9/v1Mbubm5SkpKUlBQkEJCQjR8+HAVFBRU85kAAABv5dGwc+TIEbVt21azZs06ZVthYaE2bdqkxx9/XJs2bdJ7772n9PR0/eUvf3HaLykpST/88INWr16t5cuXa926dRoxYkR1nQIAAPByHp2z06tXL/Xq1avcbcHBwVq9erXTun/+85/q0qWLMjMzFRMTo7S0NK1cuVLr169Xp06dJEkzZ87UDTfcoKlTpyoqKqrctouKilRUVORYzs/Pd9MZAQAAb1Oj5uzk5eXJZrMpJCREkpSSkqKQkBBH0JGkhIQE1apVS6mpqRW2M2XKFAUHBzs+0dHRVV06AADwkBoTdo4dO6Zx48Zp4MCBCgoKkiRlZ2crLCzMaT8fHx+FhoYqOzu7wrbGjx+vvLw8x2ffvn1VWjsAAPCcGnHreXFxsW655RYZYzR79uxKt+fn5yc/Pz83VAYAALyd14edE0Fn7969+vTTTx2jOpIUERGhgwcPOu1fUlKi3NxcRUREVHepAADAC3n1ZawTQWfnzp365JNPdMEFFzhtj4+P16FDh7Rx40bHuk8//VRlZWXq2rVrdZcLAAC8kEdHdgoKCrRr1y7H8u7du/Xtt98qNDRUkZGR+utf/6pNmzZp+fLlKi0tdczDCQ0Nla+vr1q1aqWePXvqzjvv1Jw5c1RcXKxRo0ZpwIABFd6JBQAAzi8eDTsbNmzQNddc41geO3asJGnIkCF68skn9cEHH0iS2rVr53TcmjVrdPXVV0uSFi5cqFGjRum6665TrVq11L9/f82YMaNa6gcAAN7Po2Hn6quvljGmwu2n23ZCaGio3n77bXeWBQAALMSr5+wAAABUFmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYGmEHAABYmo+nCwCAmiItLc2l4y4rLpavpJLSUv6nC3gA/90BwBkczftVkk2DBw926fh9khpLOpB9QKWZmYqJiXFneQDOgLADAGdQXHhYklG7QePUqGncOR/vN2OsdPg3GRnl5OQQdoBqRtgBgLNUPyxGoTEtz/m4WrX5Xy3gSUxQBgAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlkbYAQAAlubRsLNu3Tr17t1bUVFRstlsWrZsmdN2Y4yeeOIJRUZGKiAgQAkJCdq5c6fTPrm5uUpKSlJQUJBCQkI0fPhwFRQUVONZAAAAb+bRsHPkyBG1bdtWs2bNKnf7Cy+8oBkzZmjOnDlKTU1VvXr1lJiYqGPHjjn2SUpK0g8//KDVq1dr+fLlWrdunUaMGFFdpwAAALycjye/vFevXurVq1e524wxmj59uiZMmKA+ffpIkhYsWKDw8HAtW7ZMAwYMUFpamlauXKn169erU6dOkqSZM2fqhhtu0NSpUxUVFVVt5wIAALyT187Z2b17t7Kzs5WQkOBYFxwcrK5duyolJUWSlJKSopCQEEfQkaSEhATVqlVLqampFbZdVFSk/Px8pw8AALAmrw072dnZkqTw8HCn9eHh4Y5t2dnZCgsLc9ru4+Oj0NBQxz7lmTJlioKDgx2f6OhoN1cPAAC8hdeGnao0fvx45eXlOT779u3zdEkAAKCKeG3YiYiIkCQdOHDAaf2BAwcc2yIiInTw4EGn7SUlJcrNzXXsUx4/Pz8FBQU5fQAAgDV5bdhp2rSpIiIilJyc7FiXn5+v1NRUxcfHS5Li4+N16NAhbdy40bHPp59+qrKyMnXt2rXaawYAAN7Ho3djFRQUaNeuXY7l3bt369tvv1VoaKhiYmI0ZswYPfPMM2revLmaNm2qxx9/XFFRUerbt68kqVWrVurZs6fuvPNOzZkzR8XFxRo1apQGDBjAnVgAAECSh8POhg0bdM011ziWx44dK0kaMmSI5s2bp0ceeURHjhzRiBEjdOjQIXXr1k0rV66Uv7+/45iFCxdq1KhRuu6661SrVi31799fM2bMqPZzAQAA3smjYefqq6+WMabC7TabTU899ZSeeuqpCvcJDQ3V22+/XRXlAQAAC/DaOTsAAADuQNgBAACWRtgBAACWRtgBAACWRtgBAACWRtgBAACW5tFbzwHgfJOWllbpNho2bKiYmBg3VAOcHwg7AFCNBg8eXOk2AgLqavv2NAIPcJYIOwC8VmZmpnJycirVhjtGUtyp3aBxatQ0zuXj87P2KPXNScrJySHsAGeJsAPAK2VmZiourpWOHi10S3vFRcfd0k5l1Q+LUWhMS0+XAZxXCDsAvFJOTo6OHi1U12ETFRQZ63I7Wd+naOsHr6mkpMR9xQGoUVwKOz/++KMuuugid9cCAKcIioyt1EhIftYe9xUDoEZy6dbzZs2a6ZprrtFbb72lY8eOubsmAAAAt3Ep7GzatElt2rTR2LFjFRERobvuukvffPONu2sDAACoNJfCTrt27fSPf/xD+/fv15tvvqmsrCx169ZNl112maZNm6ZffvnF3XUCAAC4pFJPUPbx8VG/fv20ZMkSPf/889q1a5ceeughRUdH6/bbb1dWVpa76gQAAHBJpcLOhg0bdO+99yoyMlLTpk3TQw89pIyMDK1evVr79+9Xnz593FUnAACAS1y6G2vatGmaO3eu0tPTdcMNN2jBggW64YYbVKuWPTs1bdpU8+bNU2xsrDtrBQAAOGcuhZ3Zs2dr2LBhuuOOOxQZGVnuPmFhYXrjjTcqVRwAAEBluRR2du7cecZ9fH19NWTIEFeaBwAAcBuX5uzMnTtXS5YsOWX9kiVLNH/+/EoXBQAA4C4uhZ0pU6aoYcOGp6wPCwvT5MmTK10UAACAu7gUdjIzM9W0adNT1jdp0kSZmZmVLgoAAMBdXAo7YWFh+u67705Zv2XLFl1wwQWVLgoAAMBdXAo7AwcO1OjRo7VmzRqVlpaqtLRUn376qe6//34NGDDA3TUCAAC4zKW7sZ5++mnt2bNH1113nXx87E2UlZXp9ttvZ84OAADwKi6FHV9fXy1evFhPP/20tmzZooCAALVu3VpNmjRxd30AAACV4lLYOaFFixZq0aKFu2oBAABwO5fCTmlpqebNm6fk5GQdPHhQZWVlTts//fRTtxQHAABQWS6Fnfvvv1/z5s3TjTfeqMsuu0w2m83ddQEAALiFS2Fn0aJFeuedd3TDDTe4ux4AAAC3cunWc19fXzVr1szdtQAAALidS2HnwQcf1D/+8Q8ZY9xdDwAAgFu5dBnriy++0Jo1a7RixQpdeumlqlOnjtP29957zy3FAQAAVJZLYSckJEQ33XSTu2sB4CUyMzOVk5NTqTYaNmyomJgYN1UEAK5zKezMnTvX3XUA8BKZmZmKi2ulo0cLK9VOQEBdbd+eRuAB4HEuP1SwpKREa9euVUZGhgYNGqTAwEDt379fQUFBql+/vluKKy0t1ZNPPqm33npL2dnZioqK0h133KEJEyY4bnc3xmjixIl6/fXXdejQIV1xxRWaPXu2mjdv7pYagPNNTk6Ojh4tVNdhExUUGetSG/lZe5T65iTl5OQQdgB4nEthZ+/everZs6cyMzNVVFSk66+/XoGBgXr++edVVFSkOXPmuKW4559/XrNnz9b8+fN16aWXasOGDRo6dKiCg4M1evRoSdILL7ygGTNmaP78+WratKkef/xxJSYmatu2bfL393dLHcD5KCgyVqExLT1dBgBUmkt3Y91///3q1KmTfvvtNwUEBDjW33TTTUpOTnZbcV999ZX69OmjG2+8UbGxsfrrX/+qHj166JtvvpFkH9WZPn26JkyYoD59+qhNmzZasGCB9u/fr2XLlrmtDgAAUHO5FHY+//xzTZgwQb6+vk7rY2Nj9fPPP7ulMEn605/+pOTkZO3YsUOStGXLFn3xxRfq1auXJGn37t3Kzs5WQkKC45jg4GB17dpVKSkpFbZbVFSk/Px8pw8AALAmly5jlZWVqbS09JT1P/30kwIDAytd1AmPPvqo8vPzFRcXp9q1a6u0tFTPPvuskpKSJEnZ2dmSpPDwcKfjwsPDHdvKM2XKFE2aNMltdQIAAO/l0shOjx49NH36dMeyzWZTQUGBJk6c6NZXSLzzzjtauHCh3n77bW3atEnz58/X1KlTNX/+/Eq1O378eOXl5Tk++/btc1PFAADA27g0svPSSy8pMTFRl1xyiY4dO6ZBgwZp586datiwof7v//7PbcU9/PDDevTRRzVgwABJUuvWrbV3715NmTJFQ4YMUUREhCTpwIEDioyMdBx34MABtWvXrsJ2/fz85Ofn57Y6AQCA93Ip7DRu3FhbtmzRokWL9N1336mgoEDDhw9XUlKS04TlyiosLFStWs6DT7Vr11ZZWZkkqWnTpoqIiFBycrIj3OTn5ys1NVX33HOP2+oAAAA1l8vP2fHx8dHgwYPdWcspevfurWeffVYxMTG69NJLtXnzZk2bNk3Dhg2TZL98NmbMGD3zzDNq3ry549bzqKgo9e3bt0prAwBPSktLq9TxPOEa5xOXws6CBQtOu/322293qZg/mjlzph5//HHde++9OnjwoKKionTXXXfpiSeecOzzyCOP6MiRIxoxYoQOHTqkbt26aeXKlTxjB4AlHc37VZKt0n/Z5AnXOJ+4FHbuv/9+p+Xi4mIVFhbK19dXdevWdVvYCQwM1PTp050mQ/+RzWbTU089paeeesot3wkA3qy48LAko3aDxqlR0ziX2uAJ1zjfuBR2fvvtt1PW7dy5U/fcc48efvjhShcFADi9+mExPOEaOEsu3XpenubNm+u55547ZdQHAADAk9wWdiT7pOX9+/e7s0kAAIBKceky1gcffOC0bIxRVlaW/vnPf+qKK65wS2EAAADu4FLY+eNt3TabTY0aNdK1116rl156yR11AQAAuIXL78YCAACoCdw6ZwcAAMDbuDSyM3bs2LPed9q0aa58BQAAgFu4FHY2b96szZs3q7i4WC1b2p/zsGPHDtWuXVsdOnRw7Gez2dxTJQAAgItcCju9e/dWYGCg5s+frwYNGkiyP2hw6NChuvLKK/Xggw+6tUgAAABXuTRn56WXXtKUKVMcQUeSGjRooGeeeYa7sQAAgFdxaWQnPz9fv/zyyynrf/nlFx0+fLjSRQGwhsq8mbuyb/UGgBNcCjs33XSThg4dqpdeekldunSRJKWmpurhhx9Wv3793FoggJrHXW/mlqTiouOVLwjAec2lsDNnzhw99NBDGjRokIqLi+0N+fho+PDhevHFF91aIICaxx1v5s76PkVbP3hNJSUl7i0OwHnHpbBTt25dvfLKK3rxxReVkZEhSbr44otVr149txYHoGarzJu587P2uLcYAOctl8LOCVlZWcrKytJVV12lgIAAGWO43RzwoMzMTOXk5FSqDebKALAal8LOr7/+qltuuUVr1qyRzWbTzp07ddFFF2n48OFq0KABd2QBHpCZmam4uFY6erTQLe0xVwaAVbgUdh544AHVqVNHmZmZatWqlWP9rbfeqrFjxxJ2AA/IycnR0aOF6jpsooIiY11uh7kyAKzGpbDz8ccfa9WqVWrcuLHT+ubNm2vv3r1uKQyAa4IiY12eJyMxVwaA9bj0UMEjR46obt26p6zPzc2Vn59fpYsCAABwF5fCzpVXXqkFCxY4lm02m8rKyvTCCy/ommuucVtxAAAAleXSZawXXnhB1113nTZs2KDjx4/rkUce0Q8//KDc3Fx9+eWX7q4RAADAZS6N7Fx22WXasWOHunXrpj59+ujIkSPq16+fNm/erIsvvtjdNQIAALjsnEd2iouL1bNnT82ZM0ePPfZYVdQEAADgNuc8slOnTh199913VVELAACA27l0GWvw4MF644033F0LAACA27k0QbmkpERvvvmmPvnkE3Xs2PGUd2JNmzbNLcUBAABU1jmFnR9//FGxsbHaunWrOnToIEnasWOH0z68GwsAAHiTcwo7zZs3V1ZWltasWSPJ/nqIGTNmKDw8vEqKAwAAqKxzmrNjjHFaXrFihY4cOeLWggAAANzJpQnKJ/wx/AAAAHibcwo7NpvtlDk5zNEBAADe7Jzm7BhjdMcddzhe9nns2DHdfffdp9yN9d5777mvQgAAgEo4p7AzZMgQp+XBgwe7tRgAAAB3O6ewM3fu3KqqAwBQzdLS0ip1fMOGDRUTE+OmaoCq49JDBQEANdfRvF8l2So9Oh8QUFfbt6cReOD1vD7s/Pzzzxo3bpxWrFihwsJCNWvWTHPnzlWnTp0k2ecRTZw4Ua+//roOHTqkK664QrNnz1bz5s09XDkAeKfiwsOSjNoNGqdGTeNcaiM/a49S35yknJwcwg68nleHnd9++01XXHGFrrnmGq1YsUKNGjXSzp071aBBA8c+L7zwgmbMmKH58+eradOmevzxx5WYmKht27bJ39/fg9UDgHerHxaj0JiWni4DqHJeHXaef/55RUdHO80Vatq0qePPxhhNnz5dEyZMUJ8+fSRJCxYsUHh4uJYtW6YBAwZUe80AAMC7VOqhglXtgw8+UKdOnXTzzTcrLCxM7du31+uvv+7Yvnv3bmVnZyshIcGxLjg4WF27dlVKSkqF7RYVFSk/P9/pAwAArMmrw86PP/7omH+zatUq3XPPPRo9erTmz58vScrOzpakU97NFR4e7thWnilTpig4ONjxiY6OrrqTAAAAHuXVYaesrEwdOnTQ5MmT1b59e40YMUJ33nmn5syZU6l2x48fr7y8PMdn3759bqoYAAB4G68OO5GRkbrkkkuc1rVq1UqZmZmSpIiICEnSgQMHnPY5cOCAY1t5/Pz8FBQU5PQBAADW5NVh54orrlB6errTuh07dqhJkyaS7JOVIyIilJyc7Nien5+v1NRUxcfHV2utAADAO3n13VgPPPCA/vSnP2ny5Mm65ZZb9M033+i1117Ta6+9Jsn+EtIxY8bomWeeUfPmzR23nkdFRalv376eLR4AAHgFrw47nTt31tKlSzV+/Hg99dRTatq0qaZPn66kpCTHPo888oiOHDmiESNG6NChQ+rWrZtWrlzJM3YAAIAkLw87kvTnP/9Zf/7znyvcbrPZ9NRTT+mpp56qxqoAAEBN4dVzdgAAACqLsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACyNsAMAACzNx9MFALDLzMxUTk6Oy8enpaW5sRoAsA7CDuAFMjMzFRfXSkePFla6reKi426oCACsg7ADeIGcnBwdPVqorsMmKigy1qU2sr5P0dYPXlNJSYl7iwNOwx0jig0bNlRMTIwbqgHKR9gBvEhQZKxCY1q6dGx+1h73FgOcxtG8XyXZNHjw4Eq3FRBQV9u3pxF4UGVqVNh57rnnNH78eN1///2aPn26JOnYsWN68MEHtWjRIhUVFSkxMVGvvPKKwsPDPVssAFhYceFhSUbtBo1To6ZxLreTn7VHqW9OUk5ODmEHVabGhJ3169fr1VdfVZs2bZzWP/DAA/rwww+1ZMkSBQcHa9SoUerXr5++/PJLD1UKAOeP+mExLo9GAtWlRtx6XlBQoKSkJL3++utq0KCBY31eXp7eeOMNTZs2Tddee606duyouXPn6quvvtLXX3/twYoBAIC3qBFhZ+TIkbrxxhuVkJDgtH7jxo0qLi52Wh8XF6eYmBilpKRU2F5RUZHy8/OdPgAAwJq8/jLWokWLtGnTJq1fv/6UbdnZ2fL19VVISIjT+vDwcGVnZ1fY5pQpUzRp0iR3lwoAALyQV4/s7Nu3T/fff78WLlwof39/t7U7fvx45eXlOT779u1zW9sAAMC7eHXY2bhxow4ePKgOHTrIx8dHPj4++uyzzzRjxgz5+PgoPDxcx48f16FDh5yOO3DggCIiIips18/PT0FBQU4fAABgTV59Geu6667T999/77Ru6NChiouL07hx4xQdHa06deooOTlZ/fv3lySlp6crMzNT8fHxnigZAAB4Ga8OO4GBgbrsssuc1tWrV08XXHCBY/3w4cM1duxYhYaGKigoSPfdd5/i4+N1+eWXe6JkAADgZbw67JyNl19+WbVq1VL//v2dHioIAAAg1cCws3btWqdlf39/zZo1S7NmzfJMQQCASqvsO7Z4vxZOp8aFHQCAdbjrHVu8XwunQ9gBAHiMO96xxfu1cCaEHQCAx/GOLVQlr37ODgAAQGURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKURdgAAgKX5eLoAwAoyMzOVk5Pj8vFpaWlurAYA8HuEHaCSMjMzFRfXSkePFla6reKi426oCADwe4QdoJJycnJ09Gihug6bqKDIWJfayPo+RVs/eE0lJSXuLQ4AQNgB3CUoMlahMS1dOjY/a497iwEAOHj1BOUpU6aoc+fOCgwMVFhYmPr27av09HSnfY4dO6aRI0fqggsuUP369dW/f38dOHDAQxUDAABv49Vh57PPPtPIkSP19ddfa/Xq1SouLlaPHj105MgRxz4PPPCA/vvf/2rJkiX67LPPtH//fvXr18+DVQMAAG/i1ZexVq5c6bQ8b948hYWFaePGjbrqqquUl5enN954Q2+//bauvfZaSdLcuXPVqlUrff3117r88ss9UTYAAPAiXj2y80d5eXmSpNDQUEnSxo0bVVxcrISEBMc+cXFxiomJUUpKSoXtFBUVKT8/3+kDAACsqcaEnbKyMo0ZM0ZXXHGFLrvsMklSdna2fH19FRIS4rRveHi4srOzK2xrypQpCg4Odnyio6OrsnQAAOBBNSbsjBw5Ulu3btWiRYsq3db48eOVl5fn+Ozbt88NFQIAAG/k1XN2Thg1apSWL1+udevWqXHjxo71EREROn78uA4dOuQ0unPgwAFFRERU2J6fn5/8/PyqsmQAAOAlvHpkxxijUaNGaenSpfr000/VtGlTp+0dO3ZUnTp1lJyc7FiXnp6uzMxMxcfHV3e5AADAC3n1yM7IkSP19ttv6/3331dgYKBjHk5wcLACAgIUHBys4cOHa+zYsQoNDVVQUJDuu+8+xcfHcycWAACQ5OVhZ/bs2ZKkq6++2mn93Llzdccdd0iSXn75ZdWqVUv9+/dXUVGREhMT9corr1RzpQAAwFt5ddgxxpxxH39/f82aNUuzZs2qhopgNZV9W7nEG8sBwNt5ddgBqpI731Yu8cZyAPBWhB2ct9zxtnKJN5YDgLcj7OC8V5m3lUu8sRwAvJ1X33oOAABQWYQdAABgaYQdAABgaczZAQBYQmUfA9GwYUPFxMS4qRp4E8IOAKBGO5r3qySbBg8eXKl2AgLqavv2NAKPBRF2AAA1WnHhYUlG7QaNU6OmcS61kZ+1R6lvTlJOTg5hx4IIOwAAS6gfFlOpx0jAupigDAAALI2wAwAALI2wAwAALI2wAwAALI2wAwAALI27sQAAcKPMzEzl5ORUqg0ecOhehB0AANwkMzNTcXGtdPRoYaXa4QGH7kXYAQDATXJycnT0aKG6DpuooMhYl9rgAYfuR9gBAMDNgiJjecChF2GCMgAAsDTCDgAAsDTCDgAAsDTm7KDGquztnWlpaW6sBgDgrQg7qJHcdXunJBUXHXdDRQAAb0XYQY3kjts7s75P0dYPXlNJSYl7iwMAeBXCDmq0ytzemZ+1x73FAAC8EhOUAQCApTGyAwCAF6rsTRS8X+skwg4AAF7kaN6vkmwaPHhwpdrh/VonEXbOI+54E29RUZH8/PwqXUtl2+G2cQBWVVx4WJJRu0Hj1KhpnEtt8H4tZ4Sd84TbbtW22SRjKl+Qm9rhtnEAVlU/LIb3a7kJYec84c5btSvztw13tcNt4wCAs0XYOc+441btyv5twx3tcNs4gKpQ2UvkXGL3ToQdAMB5z12Tgk/gErt3IewAAM577pgULFnzErs7bm7x9G3whB0AAP7HXZfprcJdN7d4+jZ4y4SdWbNm6cUXX1R2drbatm2rmTNnqkuXLp4uyy2JWPJ8KgYAnH/ccXOLN9wGb4mws3jxYo0dO1Zz5sxR165dNX36dCUmJio9PV1hYWEeq8udb+b2dCoGAJy/KnNzizewRNiZNm2a7rzzTg0dOlSSNGfOHH344Yd688039eijj3qsLnckYsk7UjEAADVVjQ87x48f18aNGzV+/HjHulq1aikhIUEpKSnlHlNUVKSioiLHcl5eniQpPz/frbUVFBRIkkqOF6mk6KjL7ZQct9e6ceNGR5vnKj09XZKUuzfd5Vrys/ZKkvJ+3qk6PjaX2nBXO97ShjfVwvl4by15JcWqI+lwJetwRy3e1IY31WK588nOlFS53xuSm353/K+WgoICt/+ePdGeOdNDak0N9/PPPxtJ5quvvnJa//DDD5suXbqUe8zEiRONJD58+PDhw4ePBT779u07bVao8SM7rhg/frzGjh3rWC4rK1Nubq4uuOAC2WyuJ/rqlJ+fr+joaO3bt09BQUGeLsdr0C/lo18qRt+Uj34pH/1SPk/1izFGhw8fVlRU1Gn3q/Fhp2HDhqpdu7YOHDjgtP7AgQOKiIgo9xg/P79TXkIZEhJSVSVWqaCgIP6DKwf9Uj76pWL0Tfnol/LRL+XzRL8EBwefcZ9a1VBHlfL19VXHjh2VnJzsWFdWVqbk5GTFx8d7sDIAAOANavzIjiSNHTtWQ4YMUadOndSlSxdNnz5dR44ccdydBQAAzl+WCDu33nqrfvnlFz3xxBPKzs5Wu3bttHLlSoWHh3u6tCrj5+eniRMnnnI57nxHv5SPfqkYfVM++qV89Ev5vL1fbMac6X4tAACAmqvGz9kBAAA4HcIOAACwNMIOAACwNMIOAACwNMJODZKbm6ukpCQFBQUpJCREw4cPP+M7T+666y5dfPHFCggIUKNGjdSnTx9t3769miquHufaL7m5ubrvvvvUsmVLBQQEKCYmRqNHj3a8I80qXPl5ee2113T11VcrKChINptNhw4dqp5iq9CsWbMUGxsrf39/de3aVd98881p91+yZIni4uLk7++v1q1b66OPPqqmSqvfufTNDz/8oP79+ys2NlY2m03Tp0+vvkKr2bn0y+uvv64rr7xSDRo0UIMGDZSQkHDGn7Ga6lz65b333lOnTp0UEhKievXqqV27dvr3v/9djdU6I+zUIElJSfrhhx+0evVqLV++XOvWrdOIESNOe0zHjh01d+5cpaWladWqVTLGqEePHiotLa2mqqveufbL/v37tX//fk2dOlVbt27VvHnztHLlSg0fPrwaq656rvy8FBYWqmfPnvr73/9eTVVWrcWLF2vs2LGaOHGiNm3apLZt2yoxMVEHDx4sd/+vvvpKAwcO1PDhw7V582b17dtXffv21datW6u58qp3rn1TWFioiy66SM8991yFT6e3gnPtl7Vr12rgwIFas2aNUlJSFB0drR49eujnn3+u5sqr1rn2S2hoqB577DGlpKTou+++09ChQzV06FCtWrWqmiv/H7e8jRNVbtu2bUaSWb9+vWPdihUrjM1mMz///PNZt7NlyxYjyezatasqyqx27uqXd955x/j6+pri4uKqKLPaVbZf1qxZYySZ3377rQqrrHpdunQxI0eOdCyXlpaaqKgoM2XKlHL3v+WWW8yNN97otK5r167mrrvuqtI6PeFc++b3mjRpYl5++eUqrM5zKtMvxhhTUlJiAgMDzfz586uqRI+obL8YY0z79u3NhAkTqqK8M2Jkp4ZISUlRSEiIOnXq5FiXkJCgWrVqKTU19azaOHLkiObOnaumTZsqOjq6qkqtVu7oF0nKy8tTUFCQfHws8ZxNt/VLTXb8+HFt3LhRCQkJjnW1atVSQkKCUlJSyj0mJSXFaX9JSkxMrHD/msqVvjkfuKNfCgsLVVxcrNDQ0Koqs9pVtl+MMUpOTlZ6erquuuqqqiy1QoSdGiI7O1thYWFO63x8fBQaGqrs7OzTHvvKK6+ofv36ql+/vlasWKHVq1fL19e3KsutNpXplxNycnL09NNPn/EST03ijn6p6XJyclRaWnrKk9TDw8Mr7IPs7Oxz2r+mcqVvzgfu6Jdx48YpKirqlNBck7naL3l5eapfv758fX114403aubMmbr++uurutxyEXY87NFHH5XNZjvtp7ITipOSkrR582Z99tlnatGihW655RYdO3bMTWdQNaqjXyQpPz9fN954oy655BI9+eSTlS+8ilVXvwA4d88995wWLVqkpUuXyt/f39PleFxgYKC+/fZbrV+/Xs8++6zGjh2rtWvXeqQWa4zZ12APPvig7rjjjtPuc9FFFykiIuKUiWAlJSXKzc0942TB4OBgBQcHq3nz5rr88svVoEEDLV26VAMHDqxs+VWmOvrl8OHD6tmzpwIDA7V06VLVqVOnsmVXueroF6to2LChateurQMHDjitP3DgQIV9EBERcU7711Su9M35oDL9MnXqVD333HP65JNP1KZNm6oss9q52i+1atVSs2bNJEnt2rVTWlqapkyZoquvvroqyy0XYcfDGjVqpEaNGp1xv/j4eB06dEgbN25Ux44dJUmffvqpysrK1LVr17P+PmOMjDEqKipyuebqUNX9kp+fr8TERPn5+emDDz6oMX8Lq+6fl5rM19dXHTt2VHJysvr27StJKisrU3JyskaNGlXuMfHx8UpOTtaYMWMc61avXq34+PhqqLj6uNI35wNX++WFF17Qs88+q1WrVjnNk7MKd/28lJWVee53j0emRcMlPXv2NO3btzepqanmiy++MM2bNzcDBw50bP/pp59My5YtTWpqqjHGmIyMDDN58mSzYcMGs3fvXvPll1+a3r17m9DQUHPgwAFPnYbbnWu/5OXlma5du5rWrVubXbt2maysLMenpKTEU6fhdufaL8YYk5WVZTZv3mxef/11I8msW7fObN682fz666+eOIVKW7RokfHz8zPz5s0z27ZtMyNGjDAhISEmOzvbGGPMbbfdZh599FHH/l9++aXx8fExU6dONWlpaWbixImmTp065vvvv/fUKVSZc+2boqIis3nzZrN582YTGRlpHnroIbN582azc+dOT51ClTjXfnnuueeMr6+v+c9//uP0/5LDhw976hSqxLn2y+TJk83HH39sMjIyzLZt28zUqVONj4+Pef311z1SP2GnBvn111/NwIEDTf369U1QUJAZOnSo039Qu3fvNpLMmjVrjDHG/Pzzz6ZXr14mLCzM1KlTxzRu3NgMGjTIbN++3UNnUDXOtV9O3FZd3mf37t2eOYkqcK79YowxEydOLLdf5s6dW/0n4CYzZ840MTExxtfX13Tp0sV8/fXXjm3du3c3Q4YMcdr/nXfeMS1atDC+vr7m0ksvNR9++GE1V1x9zqVvTvy8/PHTvXv36i+8ip1LvzRp0qTcfpk4cWL1F17FzqVfHnvsMdOsWTPj7+9vGjRoYOLj482iRYs8ULWdzRhjqm0YCQAAoJpxNxYAALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg4AALA0wg5wntqzZ49sNpu+/fZbT5fiEbGxsZo+fbpj2WazadmyZY7l7du36/LLL5e/v7/atWtX4Toruu222zR58mTH8h/7qroMGDBAL730UrV/L6yHsAOcRnZ2tu677z5ddNFF8vPzU3R0tHr37q3k5GRPl1Zp0dHRysrK0mWXXebpUsq1du1a2Ww2HTp0qFq+LysrS7169XIsT5w4UfXq1VN6errj33d566xmy5Yt+uijjzR69GjHuvXr12vEiBGVanfq1KkKCwtTWFjYKQEmNTVVHTt2VElJidP6CRMm6Nlnn1VeXl6lvhvgredABfbs2aMrrrhCISEhevHFF9W6dWsVFxdr1apVGjlypLZv3+7pEk+ruLhYderUqXB77dq1FRERUY0V2R0/fly+vr7V/r1n8se+yMjI0I033qgmTZqcdt258tbzP2HmzJm6+eabVb9+fce6Ro0aVarN7777Tk888YSWL18uY4z+/Oc/q0ePHmrdurVKSkp0991367XXXpOPj/OvpMsuu0wXX3yx3nrrLY0cObJSNeA857G3cgFerlevXubCCy80BQUFp2z77bffHH/eu3ev+ctf/mLq1atnAgMDzc033+x4E7Ax9pdrtm3b1ixYsMA0adLEBAUFmVtvvdXk5+cbY4x59dVXTWRkpCktLXX6jr/85S9m6NChjuVly5aZ9u3bGz8/P9O0aVPz5JNPmuLiYsd2SeaVV14xvXv3NnXr1jUTJ040ubm5ZtCgQaZhw4bG39/fNGvWzLz55pvGmJMvdty8ebOjjbVr15rOnTsbX19fExERYcaNG+f0Hd27dzf33Xefefjhh02DBg1MeHj4GV94OGTIENOnTx/zzDPPmMjISBMbG2uMMWbBggWmY8eOpn79+iY8PNwMHDjQHDhwwKm2339OvGSwtLTUTJ482cTGxhp/f3/Tpk0bs2TJktPWcODAAfPnP//Z+Pv7m9jYWPPWW2+ZJk2amJdfftmp/5YuXer4s/7wUsfy1hljTGZmprn55ptNcHCwadCggfnLX/7i9ELZis7/bI978cUXTUREhAkNDTX33nuvOX78uGOfY8eOmUceecQ0btzY+Pr6mosvvtj861//cmz//vvvTc+ePU29evVMWFiYGTx4sPnll18q7KeSkhITHBxsli9f7rS+vL56/fXXTd++fU1AQIBp1qyZef/99ytsd/HixaZr166O5S5duph33nnHGGN/O/bo0aMrPHbSpEmmW7duFW4HzgZhByjHr7/+amw2m5k8efJp9ystLTXt2rUz3bp1Mxs2bDBff/216dixo9OboCdOnGjq169v+vXrZ77//nuzbt06ExERYf7+978bY4zJzc01vr6+5pNPPnH6/t+vW7dunQkKCjLz5s0zGRkZ5uOPPzaxsbHmySefdBwjyYSFhZk333zTZGRkmL1795qRI0eadu3amfXr15vdu3eb1atXmw8++MAYc2rY+emnn0zdunXNvffea9LS0szSpUtNw4YNncJM9+7dTVBQkHnyySfNjh07zPz5843NZjMff/xxhX00ZMgQU79+fXPbbbeZrVu3mq1btxpjjHnjjTfMRx99ZDIyMkxKSoqJj483vXr1MsbYf+m+++67RpJJT083WVlZ5tChQ8YYY5555hkTFxdnVq5caTIyMszcuXONn5+fWbt2bYU19OrVy7Rt29akpKSYDRs2mD/96U8mICCgwrCTlZVlLr30UvPggw+arKwsc/jw4XLXHT9+3LRq1coMGzbMfPfdd2bbtm1m0KBBpmXLlqaoqKjC8z/b44KCgszdd99t0tLSzH//+19Tt25d89prrzlqvuWWW0x0dLR57733TEZGhvnkk08cb5b+7bffTKNGjcz48eNNWlqa2bRpk7n++uvNNddcU2E/bdq0yUhyCuvGlB92GjdubN5++22zc+dOM3r0aFO/fn3z66+/ltvutm3bTIMGDczevXvNnj17TEhIiNm2bZvZtWuXad68uSP4l2fFihXG19fXHDt2rMJ9gDMh7ADlSE1NNZLMe++9d9r9Pv74Y1O7dm2TmZnpWPfDDz8YSeabb74xxtjDTt26dZ3+h/7www87/U23T58+ZtiwYY7lV1991URFRTlGe6677rpTgte///1vExkZ6ViWZMaMGeO0T+/evZ1Gh37vj2Hn73//u2nZsqUpKytz7DNr1ixTv359Rx3du3c/5W/ZnTt3NuPGjaugh+y/tMPDwx2/xCuyfv16I8kcPnzYGGPMmjVrjCSnUbRjx46ZunXrmq+++srp2OHDh5uBAweW2256errTvw9jjElLSzOSKgw7xhjTtm3bU0at/rju3//+9yl9VlRUZAICAsyqVasqPP+zPa5JkyampKTEsc/NN99sbr31VqfzWr16dbnn/fTTT5sePXo4rdu3b58jQJZn6dKlpnbt2k51GVN+2JkwYYJjuaCgwEgyK1asKLddY4yZPXu2adGihWnRooWZPXu2Mcb+c7106VKzZMkSc+mll5p27dqZzz77zOm4LVu2GElmz549FbYNnAlzdoByGGPOar+0tDRFR0crOjrase6SSy5RSEiI0tLS1LlzZ0n2u1kCAwMd+0RGRurgwYOO5aSkJN1555165ZVX5Ofnp4ULF2rAgAGqVct+D8GWLVv05Zdf6tlnn3UcU1paqmPHjqmwsFB169aVJHXq1MmpvnvuuUf9+/fXpk2b1KNHD/Xt21d/+tOfKjyX+Ph42Ww2x7orrrhCBQUF+umnnxQTEyNJatOmjdNxfzyX8rRu3fqUeSobN27Uk08+qS1btui3335TWVmZJCkzM1OXXHJJue3s2rVLhYWFuv76653WHz9+XO3bt6/wvHx8fNSxY0fHuri4OIWEhJy25rOxZcsW7dq1y+nfrSQdO3ZMGRkZjuU/nv/ZHnfppZeqdu3ajuXIyEh9//33kqRvv/1WtWvXVvfu3Susbc2aNU5zb07IyMhQixYtTll/9OhR+fn5Of0MVOT3Pwf16tVTUFDQaX8O7r77bt19992O5fnz5yswMFDx8fFq2bKl1q9fr59++kkDBgzQ7t275efnJ0kKCAiQJBUWFp6xJqAihB2gHM2bN5fNZnPbJOQ/ThS22WyOX+6S1Lt3bxlj9OGHH6pz5876/PPP9fLLLzu2FxQUaNKkSerXr98pbfv7+zv+XK9ePadtvXr10t69e/XRRx9p9erVuu666zRy5EhNnTq1ys6lPH+s68iRI0pMTFRiYqIWLlyoRo0aKTMzU4mJiTp+/HiF7RQUFEiSPvzwQ1144YVO2078cqxOBQUF6tixoxYuXHjKtt9P6v3j+Z/tcafr6xMh4HS19e7dW88///wp2yIjI8s9pmHDhiosLDyrSdSu/ByckJOTo0mTJmndunVKTU1VixYt1Lx5czVv3lzFxcXasWOHWrduLUnKzc2VVPlJ0ji/EXaAcoSGhioxMVGzZs3S6NGjT/lldejQIYWEhKhVq1bat2+f9u3b5xjd2bZtmw4dOlTh6ER5/P391a9fPy1cuFC7du1Sy5Yt1aFDB8f2Dh06KD09Xc2aNTvnc2nUqJGGDBmiIUOG6Morr9TDDz9cbthp1aqV3n33XRljHH+z//LLLxUYGKjGjRuf8/eezvbt2/Xrr7/queeec/Tbhg0bnPY58cu2tLTUse6SSy6Rn5+fMjMzKxzR+KO4uDiVlJRo48aNjpG29PR0t9zS3qFDBy1evFhhYWEKCgqq8uN+r3Xr1iorK9Nnn32mhISEcr/j3XffVWxs7Cl3OVXkxLODtm3bVqXPEXrggQf0wAMPqHHjxlq/fr2Ki4sd20pKSpz+nW/dulWNGzdWw4YNq6weWB/P2QEqMGvWLJWWlqpLly569913tXPnTqWlpWnGjBmKj4+XJCUkJKh169ZKSkrSpk2b9M033+j2229X9+7dT7mkdCZJSUn68MMP9eabbyopKclp2xNPPKEFCxZo0qRJ+uGHH5SWlqZFixZpwoQJp23ziSee0Pvvv69du3bphx9+0PLly9WqVaty97333nu1b98+3Xfffdq+fbvef/99TZw4UWPHjnVcTnOXmJgY+fr6aubMmfrxxx/1wQcf6Omnn3bap0mTJrLZbFq+fLl++eUXFRQUKDAwUA899JAeeOABzZ8/XxkZGdq0aZNmzpyp+fPnl/tdLVu2VM+ePXXXXXcpNTVVGzdu1N/+9rczjoycjaSkJDVs2FB9+vTR559/rt27d2vt2rUaPXq0fvrpJ7cf93uxsbEaMmSIhg0bpmXLljnaeOeddyRJI0eOVG5urgYOHKj169crIyNDq1at0tChQ53CxO81atRIHTp00BdffHHunXGWVq9erR07djhuJe/cubO2b9+uFStW6LXXXlPt2rXVsmVLx/6ff/65evToUWX14PxA2AEqcNFFF2nTpk265ppr9OCDD+qyyy7T9ddfr+TkZM2ePVuSfej+/fffV4MGDXTVVVcpISFBF110kRYvXnzO33fttdcqNDRU6enpGjRokNO2xMRELV++XB9//LE6d+6syy+/XC+//PIZn/fi6+ur8ePHq02bNrrqqqtUu3ZtLVq0qNx9L7zwQn300Uf65ptv1LZtW919990aPnz4GQOVKxo1aqR58+ZpyZIluuSSS/Tcc8+dMtp04YUXatKkSXr00UcVHh6uUaNGSZKefvppPf7445oyZYpatWqlnj176sMPP1TTpk0r/L65c+cqKipK3bt3V79+/TRixAiFhYVV+jzq1q2rdevWKSYmRv369VOrVq00fPhwHTt27LQjNq4e90ezZ8/WX//6V917772Ki4vTnXfeqSNHjkiSoqKi9OWXX6q0tNTxTJsxY8YoJCTktOH1b3/7W7mX19zh6NGjGjVqlF599VVHDY0bN9bMmTM1dOhQPfvss5o/f74jiB47dkzLli3TnXfeWSX14PxhM2c7ExMAYHlHjx5Vy5YttXjxYscIpqfMnj1bS5cu1ccff+zROlDzMbIDAHAICAjQggULlJOT4+lSVKdOHc2cOdPTZcACGNkBAACWxsgOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwNMIOAACwtP8H5qBNcjYT0GgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "sns.histplot(perm_diff_conv, ax=ax)\n",
    "ax.vlines([observed_pct_diff], color='red', ymin=0, ymax=150, lw=2)\n",
    "ax.text(observed_pct_diff + 0.01, 140, 'Observed\\ndifference', color='red')\n",
    "ax.set_xlabel('Conversion rate difference (in %)')\n",
    "ax.set_ylabel('Frequency')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "😱 From graph it is hard to measure statistical significance. So we use p-value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.353"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean([perm_diff > observed_pct_diff for perm_diff in perm_diff_conv])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Without permutation testing, we can find p-value by using scipy.stats.chi2_contingency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats\n",
    "\n",
    "\n",
    "conversions = np.array([[200, 23739 - 200], [182, 22588 - 182]])\n",
    "chi2, p, df, _ = stats.chi2_contingency(conversions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  200, 23539],\n",
       "       [  182, 22406]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conversions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chi-square: 0.1489\n",
      "p-value: 0.3498\n",
      "Degrees of freedom: 1\n"
     ]
    }
   ],
   "source": [
    "print(f'Chi-square: {chi2:.4f}')\n",
    "print(f'p-value: {p/2:.4f}')\n",
    "print(f'Degrees of freedom: {df}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `T-Test`\n",
    "\n",
    "- A t-test is a method used to determine if there is a significant difference between the means of two groups.\n",
    "\n",
    "#### Formula\n",
    "\n",
    "- t = (x̄1 - x̄2) / √(s1^2/n1 + s2^2/n2)\n",
    "\n",
    "- df = n1 + n2 - 2\n",
    "\n",
    "- p-value = 2 * (1 - t.cdf(abs(t), df))\n",
    "\n",
    "#### Example\n",
    "\n",
    "- x̄1 = 10, s1 = 2, n1 = 100\n",
    "\n",
    "- x̄2 = 12, s2 = 3, n2 = 100\n",
    "\n",
    "- t = (10 - 12) / √(2^2/100 + 3^2/100) = -2 / √(0.04 + 0.09) = -2 / √0.13 = -2 / 0.36 = -5.56\n",
    "\n",
    "- df = 100 + 100 - 2 = 198\n",
    "\n",
    "- p-value = 2 * (1 - t.cdf(abs(-5.56), 198)) = 2 * (1 - t.cdf(5.56, 198)) = 2 * (1 - 0.9999999999999999) = 2 * 0.0 = 0.0\n",
    "\n",
    "#### Conclusion\n",
    "\n",
    "- The p-value is less than 0.05, so we reject the null hypothesis and conclude that there is a significant difference between the means of the two groups.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "T-statistic: -1.0983\n",
      "p-value: 0.1408\n"
     ]
    }
   ],
   "source": [
    "from scipy.stats import ttest_ind\n",
    "\n",
    "res = ttest_ind(df_website[df_website['Page'] == 'Page A']['Time'], df_website[df_website['Page'] == 'Page B']['Time'], equal_var=False)\n",
    "\n",
    "print(f'T-statistic: {res.statistic:.4f}')\n",
    "print(f'p-value: {res.pvalue/2:.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `ANOVA - Analysis of Variance`\n",
    "\n",
    "- ANOVA is a method used to determine if there is a significant difference between the means of three or more groups.\n",
    "\n",
    "\n",
    "#### Algorithm\n",
    "\n",
    "- Combine all the data into a single dataset.\n",
    "- Shuffle and draw samples from the combined dataset to create new datasets for each group.\n",
    "- Calculate the mean of each group.\n",
    "- Calculate the varriance among the means of the groups.\n",
    "- Repeat the process many times to estimate the distribution of the variance.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Page</th>\n",
       "      <th>Time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Page 1</td>\n",
       "      <td>164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Page 2</td>\n",
       "      <td>178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Page 3</td>\n",
       "      <td>175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Page 4</td>\n",
       "      <td>155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Page 1</td>\n",
       "      <td>172</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Page  Time\n",
       "0  Page 1   164\n",
       "1  Page 2   178\n",
       "2  Page 3   175\n",
       "3  Page 4   155\n",
       "4  Page 1   172"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_4_sessions = pd.read_csv('data/four_sessions.csv')\n",
    "df_4_sessions.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_4_sessions = df_4_sessions.pivot(columns='Page', values='Time').fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_4_sessions.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55.426666666666655"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "observed_var = df_4_sessions.groupby(\"Page\").mean().var()[0]\n",
    "observed_var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observed mean:  [172.8 182.6 175.6 164.6]\n",
      "Observed variance:  55.426666666666655\n"
     ]
    }
   ],
   "source": [
    "print(\"Observed mean: \", df_4_sessions.groupby(\"Page\").mean().values.ravel())\n",
    "print(\"Observed variance: \", observed_var)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perm_test(df):\n",
    "    df = df.copy()\n",
    "    df[\"Time\"] = np.random.permutation(df[\"Time\"].values)\n",
    "    return df.groupby(\"Page\").mean().var()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pr(Prob): 8.233333333333333%\n"
     ]
    }
   ],
   "source": [
    "perm_var = [perm_test(df_4_sessions) for _ in range(3000)]\n",
    "print(f\"Pr(Prob): {100*np.mean([var > observed_var for var in perm_var])}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### F-Statisitc\n",
    "\n",
    "- The F-statistic is a measure of the ratio of the variance between groups to the variance within groups.\n",
    "\n",
    "- The F-statistic is used to determine if there is a significant difference between the means of the groups.\n",
    "\n",
    "- The F-statistic is calculated as the ratio of the variance between groups to the variance within groups.\n",
    "\n",
    "\n",
    "- F = (SSB / (k - 1)) / (SSW / (n - k))\n",
    "\n",
    "- df1 = k - 1\n",
    "\n",
    "- df2 = n - k\n",
    "\n",
    "- p-value = 1 - f.cdf(F, df1, df2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>Time</td>       <th>  R-squared:         </th> <td>   0.339</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.216</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   2.740</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Fri, 19 Jul 2024</td> <th>  Prob (F-statistic):</th>  <td>0.0776</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>22:24:46</td>     <th>  Log-Likelihood:    </th> <td> -72.313</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>    20</td>      <th>  AIC:               </th> <td>   152.6</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>    16</td>      <th>  BIC:               </th> <td>   156.6</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     3</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "         <td></td>           <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th>      <td>  172.8000</td> <td>    4.498</td> <td>   38.419</td> <td> 0.000</td> <td>  163.265</td> <td>  182.335</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Page[T.Page 2]</th> <td>    9.8000</td> <td>    6.361</td> <td>    1.541</td> <td> 0.143</td> <td>   -3.684</td> <td>   23.284</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Page[T.Page 3]</th> <td>    2.8000</td> <td>    6.361</td> <td>    0.440</td> <td> 0.666</td> <td>  -10.684</td> <td>   16.284</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Page[T.Page 4]</th> <td>   -8.2000</td> <td>    6.361</td> <td>   -1.289</td> <td> 0.216</td> <td>  -21.684</td> <td>    5.284</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td> 2.928</td> <th>  Durbin-Watson:     </th> <td>   1.827</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.231</td> <th>  Jarque-Bera (JB):  </th> <td>   1.345</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td> 0.578</td> <th>  Prob(JB):          </th> <td>   0.510</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 3.529</td> <th>  Cond. No.          </th> <td>    4.79</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                   Time   R-squared:                       0.339\n",
       "Model:                            OLS   Adj. R-squared:                  0.216\n",
       "Method:                 Least Squares   F-statistic:                     2.740\n",
       "Date:                Fri, 19 Jul 2024   Prob (F-statistic):             0.0776\n",
       "Time:                        22:24:46   Log-Likelihood:                -72.313\n",
       "No. Observations:                  20   AIC:                             152.6\n",
       "Df Residuals:                      16   BIC:                             156.6\n",
       "Df Model:                           3                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==================================================================================\n",
       "                     coef    std err          t      P>|t|      [0.025      0.975]\n",
       "----------------------------------------------------------------------------------\n",
       "Intercept        172.8000      4.498     38.419      0.000     163.265     182.335\n",
       "Page[T.Page 2]     9.8000      6.361      1.541      0.143      -3.684      23.284\n",
       "Page[T.Page 3]     2.8000      6.361      0.440      0.666     -10.684      16.284\n",
       "Page[T.Page 4]    -8.2000      6.361     -1.289      0.216     -21.684       5.284\n",
       "==============================================================================\n",
       "Omnibus:                        2.928   Durbin-Watson:                   1.827\n",
       "Prob(Omnibus):                  0.231   Jarque-Bera (JB):                1.345\n",
       "Skew:                           0.578   Prob(JB):                        0.510\n",
       "Kurtosis:                       3.529   Cond. No.                         4.79\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "\"\"\""
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "\n",
    "import statsmodels.formula.api as smf\n",
    "\n",
    "model = smf.ols('Time ~ Page', data=df_4_sessions).fit()\n",
    "model.summary()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>df</th>\n",
       "      <th>sum_sq</th>\n",
       "      <th>mean_sq</th>\n",
       "      <th>F</th>\n",
       "      <th>PR(&gt;F)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Page</th>\n",
       "      <td>3.0</td>\n",
       "      <td>831.4</td>\n",
       "      <td>277.133333</td>\n",
       "      <td>2.739825</td>\n",
       "      <td>0.077586</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Residual</th>\n",
       "      <td>16.0</td>\n",
       "      <td>1618.4</td>\n",
       "      <td>101.150000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            df  sum_sq     mean_sq         F    PR(>F)\n",
       "Page       3.0   831.4  277.133333  2.739825  0.077586\n",
       "Residual  16.0  1618.4  101.150000       NaN       NaN"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "aov_table = sm.stats.anova_lm(model)\n",
    "aov_table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Chi-Square Test`\n",
    "\n",
    "- The chi-square test is a method used to determine if there is a significant association between two categorical variables.\n",
    "\n",
    "\n",
    "- χ^2 = Σ((O - E)^2 / E)\n",
    "\n",
    "- df = (r - 1) * (c - 1)\n",
    "\n",
    "- p-value = 1 - chi2.cdf(χ^2, df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chi-Square Test: Resampling Approach\n",
    "\n",
    "- The resampling approach to the chi-square test involves permuting the data and calculating the test statistic many times to estimate the p-value.\n",
    "\n",
    "- Constitute a box with 34 ones (clicks) and 2966 zeros (non-clicks).\n",
    "- Shuffle the box and draw samples of 1000 and count the cliks in each.\n",
    "- Find the squared difference between the observed and expected values and sum them.\n",
    "- Repeat the process many times to estimate the distribution of the test statistic.\n",
    "- How often does the resampled test statistic exceed the observed test statistic?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Headline</th>\n",
       "      <th>Click</th>\n",
       "      <th>Rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Headline A</td>\n",
       "      <td>Click</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Headline A</td>\n",
       "      <td>No-click</td>\n",
       "      <td>986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Headline B</td>\n",
       "      <td>Click</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Headline B</td>\n",
       "      <td>No-click</td>\n",
       "      <td>992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Headline C</td>\n",
       "      <td>Click</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Headline C</td>\n",
       "      <td>No-click</td>\n",
       "      <td>988</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Headline     Click  Rate\n",
       "0  Headline A     Click    14\n",
       "1  Headline A  No-click   986\n",
       "2  Headline B     Click     8\n",
       "3  Headline B  No-click   992\n",
       "4  Headline C     Click    12\n",
       "5  Headline C  No-click   988"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_clicks = pd.read_csv(\"data/click_rates.csv\")\n",
    "df_clicks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "\n",
    "box = [1] * 34\n",
    "box.extend([0] * 2966)\n",
    "random.shuffle(box)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def chi2_test(observerd, expected):\n",
    "    pearson_residuals = []\n",
    "    for row, expect in zip(observerd, expected):\n",
    "            pearson_residuals.append([(obs - expect) ** 2 / expect for obs in row])\n",
    "\n",
    "    return np.sum(pearson_residuals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected_clicks = 34/3\n",
    "expected_non_clicks = 1000 - expected_clicks\n",
    "expected = [34/3, 1000-34/3]\n",
    "chi2_observed = chi2_test([df_clicks[df_clicks[\"Click\"] == \"Click\"][\"Rate\"].values], expected)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.6470588235294115"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chi2_observed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def perm_func(box):\n",
    "    sample_clicks = [sum(random.sample(box,1000)),\n",
    "                     sum(random.sample(box, 1000)),\n",
    "                     sum(random.sample(box, 1000))]\n",
    "    \n",
    "    sample_noclicks = [1000 - n for n in sample_clicks]\n",
    "    return chi2_test([sample_clicks, sample_noclicks], expected)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "perm_chi2_2 = [perm_func(box) for _ in range(2000)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observed Chi2:  1.6470588235294115\n",
      "Resampled p-value:  0.491\n"
     ]
    }
   ],
   "source": [
    "resampled_p_value = sum(perm_chi2_2 > chi2_observed)/len(perm_chi2_2)\n",
    "print(\"Observed Chi2: \", chi2_observed)\n",
    "print(\"Resampled p-value: \", resampled_p_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chi-Squre Tests: Statistical Theory\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import chi2_contingency\n",
    "\n",
    "chi2, pvalue, df, expected = chi2_contingency(df_clicks[\"Rate\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0, 1.0, 0, array([ 14., 986.,   8., 992.,  12., 988.]))"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chi2, pvalue, df, expected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fisher's Exact Test\n",
    "\n",
    "- All possible permutations possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Multi-Arm Bandit Algorithms`\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Power and Sample Size`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels as sm\n",
    "\n",
    "effect_size = sm.stats.proportion.proportion_effectsize(0.0121, 0.011)\n",
    "ananlysis = sm.stats.power.TTestIndPower()\n",
    "result = ananlysis.solve_power(effect_size=effect_size, alpha=0.05, power=0.8, alternative='larger')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "116602.39259746042"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}