Mixtral_fine_tuning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iEkm_Q9AGgTX"
      },
      "source": [
        "# MIXTRAL 8x7B - Mixture of Experts\n",
        "\n",
        "This will not run on the free T4 GPU from Google Colab. You will need A100 to run this.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlVGJXwsyVQO"
      },
      "source": [
        "### Install Required Packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3IovaXt4ZJQ_"
      },
      "outputs": [],
      "source": [
        "!pip install transformers trl accelerate torch bitsandbytes peft datasets -qU\n",
        "!pip install flash-attn --no-build-isolation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ck_5U-vIq7zW"
      },
      "source": [
        "#### Load HF Dataset\n",
        "\n",
        "We need a dataset to fine-tune a model, for this example we will be using a subset of the `mosaicml/instruct-v3` dataset.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DI1G_zS8PWh9"
      },
      "outputs": [],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "instruct_tune_dataset = load_dataset(\"mosaicml/instruct-v3\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VmAvZLhkrY6p"
      },
      "source": [
        "#### Data structure\n",
        "\n",
        "The dataset contains three different columns. We are only interested in the columns `prompt` and `response`. There are 9 different possible source value in the `source` column. We are only interested in one of them.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YfDpPbAyTHff",
        "outputId": "e2549418-b3c9-49c0-a187-79452dae4eff"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['prompt', 'response', 'source'],\n",
              "        num_rows: 56167\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['prompt', 'response', 'source'],\n",
              "        num_rows: 6807\n",
              "    })\n",
              "})"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "instruct_tune_dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vp64GoU9iUzt"
      },
      "source": [
        "Since we want to generate a model that generates instructions - we're going to filter away all the subset datasets and only used the `dolly_hhrlhf` component!\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XE83nED_ghTa"
      },
      "outputs": [],
      "source": [
        "instruct_tune_dataset = instruct_tune_dataset.filter(lambda x: x[\"source\"] == \"dolly_hhrlhf\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YVWIqDHCg5BP",
        "outputId": "e627f3e7-2ad8-4ec3-dbbf-6049b0c7e32b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['prompt', 'response', 'source'],\n",
              "        num_rows: 34333\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['prompt', 'response', 'source'],\n",
              "        num_rows: 4771\n",
              "    })\n",
              "})"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "instruct_tune_dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HNKsQDzlilSi"
      },
      "source": [
        "We will use just a small subset of the data for this training example.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6UC7sscpyVQR"
      },
      "source": [
        "#### Create Formatted Prompt\n",
        "\n",
        "In the following function we'll be merging our `prompt` and `response` columns by creating the following template:\n",
        "\n",
        "```\n",
        "<s>[INST] Use the provided input to create an instruction that could have been used to generate the response with an LLM.\n",
        "\n",
        "{input} [/INST]\n",
        "\n",
        "{response}</s>\n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8LxhBq7hyVQR",
        "outputId": "de7374ae-b50a-4315-f8fa-1d6a545d7a73"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'prompt': 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\\n\\n### Instruction\\nWhat are different types of grass?\\n\\n### Response\\n',\n",
              " 'response': 'There are more than 12,000 species of grass. The most common is Kentucky Bluegrass, because it grows quickly, easily, and is soft to the touch. Rygrass is shiny and bright green colored. Fescues are dark green and shiny. Bermuda grass is harder but can grow in drier soil.',\n",
              " 'source': 'dolly_hhrlhf'}"
            ]
          },
          "execution_count": 48,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "instruct_tune_dataset[\"train\"][0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HGuHyrnrfCu_"
      },
      "outputs": [],
      "source": [
        "def create_prompt(sample):\n",
        "  bos_token = \"<s>\"\n",
        "  original_system_message = \"Below is an instruction that describes a task. Write a response that appropriately completes the request.\"\n",
        "  system_message = \"[INST]Use the provided input to create an instruction that could have been used to generate the response with an LLM.\"\n",
        "  response = sample[\"prompt\"].replace(original_system_message, \"\").replace(\"\\n\\n### Instruction\\n\", \"\").replace(\"\\n### Response\\n\", \"\").strip()\n",
        "  input = sample[\"response\"]\n",
        "  eos_token = \"</s>\"\n",
        "\n",
        "  full_prompt = \"\"\n",
        "  full_prompt += bos_token\n",
        "  full_prompt += system_message\n",
        "  full_prompt += \"\\n\" + input\n",
        "  full_prompt += \"[/INST]\"\n",
        "  full_prompt += response\n",
        "  full_prompt += eos_token\n",
        "\n",
        "  return full_prompt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "id": "cdi5goqbfFOK",
        "outputId": "5435df16-83f0-400c-a43f-9ce24026abac"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'<s>[INST]Use the provided input to create an instruction that could have been used to generate the response with an LLM.\\nThere are more than 12,000 species of grass. The most common is Kentucky Bluegrass, because it grows quickly, easily, and is soft to the touch. Rygrass is shiny and bright green colored. Fescues are dark green and shiny. Bermuda grass is harder but can grow in drier soil.[/INST]What are different types of grass?</s>'"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "create_prompt(instruct_tune_dataset[\"train\"][0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gdNP_dgbgSdA"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-4TZODFA8gy9"
      },
      "source": [
        "### Map the Dataset\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y7dAhU6e8e3d"
      },
      "outputs": [],
      "source": [
        "# mapped_data = instruct_tune_dataset.map(create_prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4qjy9PYUaY3W"
      },
      "source": [
        "### Loading the Base Model\n",
        "\n",
        "Load the model in `4bit`, with double quantization, with `bfloat16` as the compute dtype.\n",
        "\n",
        "In this case we are using the instruct-tuned model - instead of the base model. For fine-tuning a base model will need a lot more data!\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e_vtzuBHyVQS"
      },
      "outputs": [],
      "source": [
        "model_id = \"mistralai/Mixtral-8x7B-v0.1\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qRb5keMJB7wr"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
        "import torch\n",
        "\n",
        "nf4_config = BitsAndBytesConfig(\n",
        "   load_in_4bit=True,\n",
        "   bnb_4bit_quant_type=\"nf4\",\n",
        "   bnb_4bit_use_double_quant=True,\n",
        "   bnb_4bit_compute_dtype=torch.bfloat16\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241,
          "referenced_widgets": [
            "c577ab91428c4f5aac8299ba85104dc6",
            "4fbd859668ac43c884d39f675f1158f7",
            "68b12f6f28fa452ea302d83e37084c43",
            "2efa53a79dd54d86b72284f3a87c7ca9",
            "643f193c688d49a0a204ea70019f2cc5",
            "fbe68ed8ac6243f180408175284b98f5",
            "86712973ec7e498da66f29dbcea3f24f",
            "f369a81a80d84934881c8e9a0bc6a14f",
            "72588505d177495b88c39917ae9f743c",
            "164bc349194f4e6ab1229f3d2d03b3db",
            "732e5508dfb1422c97c0fbc27b888f91",
            "c65ab63b72d143c7bca446276d0f1aa5",
            "963c8c0504554e57bb9a6a8e2c2060fe",
            "6950d59bdf254dd896c9f30a220c0533",
            "becf709cc6a4490481f6cbebbc6bad49",
            "90de7d165e804fbf8f3782ac5b16f8fb",
            "118879d64cae46f5b3e744f86404922c",
            "0055eb4d0d054684878bba6da9cae910",
            "0b9648d0eccf4516a8c0a7e3843f76b8",
            "a2a870bddb694a8ea69bdac062ef6d46",
            "df28571a454d47d698d56d7cf4d90009",
            "b414aed96db744e99cbe4bf052426fa9",
            "ed774deaa35441578397fa1241118263",
            "adcfd8c2daa14f81aea6c85bf98cb735",
            "7d8f03389fc64ddfa128be53ea2eb1e8",
            "fd169008a90840f69ca3ff57ad7f84ed",
            "8aec8536c28f4c75b2728936f6843a2c",
            "42becf797d9a42bc9984d8e30b307cff",
            "1c03379c0bf84b62a671021810a5caee",
            "21ac78e940c84842bdde1b64bd1395ad",
            "ddfc0a4121b8447b84abab68482e46a3",
            "50d301095b594acdb2d754377b029245",
            "dae5a71abe874e69b3972d5fc861fd1a",
            "d429cb7700aa40099e28fa9b6565be26",
            "db518d1bca324c98911ad35e5df91c3a",
            "b3a9937a7f5642e1bdddd2cb1c723490",
            "c07ef3e83cec46ef8a9fc9d7d3fdb6fe",
            "74b57be114494edda79768ff04961193",
            "ccbc22e41b754682bea5e80531498470",
            "3454ad0be8554f39bea55eb8d9c3db1b",
            "2de61bee8b56442887b6f42592423064",
            "79d82f5e0881485897a8e977269079c8",
            "b245d2f284d7450b9d99e820f6193739",
            "09f69e88c4834d25be2be790022384b0",
            "7e5645b33aa64e7ebbd60cb76e1ecd11",
            "4245f82dafc0478ca7ac1f5b372e5a12",
            "494347c07d684876ba64b39d698250d0",
            "6223c94b2b814f2ab7abe976402551b1",
            "48648a0d46104a0db77eabf6d8fdf0f0",
            "82c83b75261c4f1db9b7c3d9ad0d097e",
            "fc38a40ba92d46c49b3679a84c8ef879",
            "fb5aaa2a96e047a0aa47d07a07b183fe",
            "f7a940cb871f41c7af7e79c32c3156e7",
            "f56251c2b16c4dae99f80db60e5f6330",
            "9e98f0e4ef964bbeaa63ca278313bb19",
            "82cafea491ca4350bea831165926d9fb",
            "f3d9d131f5934a048930bc8983437983",
            "50b0bd7f60ea41c59d6cbd055a10d765",
            "f27e89855a374c6d93e292dce8e146c5",
            "72a4b9321abf4d58a093a3d09263c1ad",
            "35f2eadd1bca48bdbc6504fcd693dab7",
            "084223b1556a420da5964ee086c117e0",
            "85f25707d0a04084aaa1ada15616f415",
            "553955d94d844f05bc19116611761127",
            "296ee6c082da4c76be2b099484662d26",
            "2073bd16554341df9ea62874f4907e18",
            "ae012e86315f41e7ae12b84312d2455b",
            "444b9f99a79045b9a5c779a3d18bedd4",
            "290face1ed4647d7b172109e162c3769",
            "fbeb4393ccb34413bea869355235d0fa",
            "838cd2c0701c4b0b82fa8ddce51215db",
            "240f5b4175e4469381a53eacdf657419",
            "7af0d44687f142288e3e58f15338dedd",
            "773040f77dae4115aefdd958a658a3fc",
            "876619b7e19441a096225f72e58b5544",
            "95863a2d3c074f45b79a894cb8e67d27",
            "2998cf3d58354f6f8465a6e2a10ebab2",
            "04ccf9dbb8734311bc3d414b11d93a9c"
          ]
        },
        "id": "ZbaaN_VdWABp",
        "outputId": "848d1a5d-54db-4607-903f-ba08289e2873"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "04ccf9dbb8734311bc3d414b11d93a9c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/19 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "model = AutoModelForCausalLM.from_pretrained(\n",
        "    model_id,\n",
        "    device_map='auto',\n",
        "    quantization_config=nf4_config,\n",
        "    use_cache=False,\n",
        "    attn_implementation=\"flash_attention_2\"\n",
        "\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9TzTWLD2WBZV"
      },
      "outputs": [],
      "source": [
        "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
        "\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "tokenizer.padding_side = \"right\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dlQxPyiKi51k"
      },
      "source": [
        "Let's example how well the model does at this task currently:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wRWhXzXrc41b"
      },
      "outputs": [],
      "source": [
        "def generate_response(prompt, model):\n",
        "  encoded_input = tokenizer(prompt,  return_tensors=\"pt\", add_special_tokens=True)\n",
        "  model_inputs = encoded_input.to('cuda')\n",
        "\n",
        "  generated_ids = model.generate(**model_inputs,\n",
        "                                 max_new_tokens=512,\n",
        "                                 do_sample=True,\n",
        "                                 pad_token_id=tokenizer.eos_token_id)\n",
        "\n",
        "  decoded_output = tokenizer.batch_decode(generated_ids)\n",
        "\n",
        "  return decoded_output[0].replace(prompt, \"\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OVqKOOjeyVQT"
      },
      "outputs": [],
      "source": [
        "prompt=\"\"\"[INST]Use the provided input to create an instruction that could have been used to generate the response with an LLM. \\nThere are more than 12,000 species of grass. The most common is Kentucky Bluegrass, because it grows quickly, easily, and is soft to the touch. Rygrass is shiny and bright green colored. Fescues are dark green and shiny. Bermuda grass is harder but can grow in drier soil.[\\INST]\"\"\"\n",
        "\n",
        "generate_response(prompt, model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gzu4c53ByVQT"
      },
      "source": [
        "### Tokenization\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "49s40_WNyVQT"
      },
      "outputs": [],
      "source": [
        "def tokenize_prompts(prompt):\n",
        "    return tokenizer(create_prompt(prompt))\n",
        "\n",
        "tokenized_train_dataset = instruct_tune_dataset[\"train\"].map(tokenize_prompts)\n",
        "tokenized_val_dataset = instruct_tune_dataset[\"test\"].map(tokenize_prompts)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kIkUgIRFm3e7",
        "outputId": "86462b12-3eb7-46fd-c568-78d0b0496843"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "39104\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAAIjCAYAAABswtioAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABTFElEQVR4nO3deVgVdf//8dcBPIAi4AZIopCaivsuaaaJopJl2u2SGZrmbTeWiplZ5lKZZZuapS130mYulZaaGOJ2Z+6FW0ruaLJYBogpKMzvj77MzyO4QOioPB/Xda48n3mfmfcMY/FqZj7HZhiGIQAAAADAdedkdQMAAAAAUFIRyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAKAYTJw4UTab7bpsq127dmrXrp35fs2aNbLZbPryyy+vy/YHDBigwMDA67KtosrMzNTgwYPl5+cnm82mESNGWN1SsbveP/criYmJUaNGjeTm5iabzaa0tLQC66Kjo2Wz2XT48OHr2t+1UJh9CQwM1IABA655TwBuPgQyALhI3i9ZeS83Nzf5+/srLCxMM2bM0KlTp4plO8ePH9fEiRMVHx9fLOsrTjdyb1fj5ZdfVnR0tB5//HF9+umn6t+//yVrAwMDde+9917H7gpn7ty5mjZtmtVtXNYff/yhXr16yd3dXe+8844+/fRTlSlTxuq2rsovv/yiiRMn3hIBEcDNycXqBgDgRvXCCy8oKChI586dU3JystasWaMRI0bozTff1LfffqsGDRqYtePGjdMzzzxTqPUfP35ckyZNUmBgoBo1anTVn/v+++8LtZ2iuFxvH3zwgXJzc695D//EqlWr1KpVK02YMMHqVv6xuXPnateuXTf0Vb4tW7bo1KlTevHFFxUaGnrZ2v79+6tPnz5ydXW9Tt1d3i+//KJJkyapXbt2hb7ye6PtC4CbE4EMAC6hS5cuatasmfl+7NixWrVqle69917dd9992rNnj9zd3SVJLi4ucnG5tv9K/euvv1S6dGnZ7fZrup0rKVWqlKXbvxqpqakKDg62uo0SIzU1VZLk7e19xVpnZ2c5Oztf446uj1tpXwBYh1sWAaAQ7rnnHj3//PM6cuSIPvvsM3O8oGfIYmNj1aZNG3l7e8vDw0O1atXSs88+K+nv53+aN28uSRo4cKB5e2R0dLSkv58Tq1evnrZt26a2bduqdOnS5mcvfoYsT05Ojp599ln5+fmpTJkyuu+++3T06FGHmks9x3LhOq/UW0HPkJ0+fVqjRo1SQECAXF1dVatWLb3++usyDMOhzmazadiwYVq8eLHq1asnV1dX1a1bVzExMQUf8IukpqZq0KBB8vX1lZubmxo2bKiPP/7YXJ73XNWhQ4e0bNkys/fiuB3ts88+U9OmTeXu7q7y5curT58++Y5v3s/tl19+Ufv27VW6dGnddtttmjp1ar71HTlyRPfdd5/KlCkjHx8fjRw5UitWrJDNZtOaNWvM9S1btkxHjhwx9+XiY5+bm6vJkyerSpUqcnNzU4cOHbR//36Hmn379qlnz57y8/OTm5ubqlSpoj59+ig9Pf2K+71w4UJzvytWrKiHH35Yv/32m8M+R0RESJKaN28um8122WelCnruKu+20R9++EEtWrSQm5ubbr/9dn3yyScFfnbdunX697//rQoVKsjT01OPPPKI/vzzT4dam82miRMn5tv+hX8HoqOj9a9//UuS1L59e/MY5x3/KyloXwzD0EsvvaQqVaqodOnSat++vXbv3p3vs+fOndOkSZNUs2ZNubm5qUKFCmrTpo1iY2OvatsAbh1cIQOAQurfv7+effZZff/993rssccKrNm9e7fuvfdeNWjQQC+88IJcXV21f/9+rV+/XpJUp04dvfDCCxo/fryGDBmiu+66S5J05513muv4448/1KVLF/Xp00cPP/ywfH19L9vX5MmTZbPZNGbMGKWmpmratGkKDQ1VfHy8eSXvalxNbxcyDEP33XefVq9erUGDBqlRo0ZasWKFRo8erd9++01vvfWWQ/0PP/ygr7/+Wv/5z39UtmxZzZgxQz179lRiYqIqVKhwyb7OnDmjdu3aaf/+/Ro2bJiCgoK0cOFCDRgwQGlpaRo+fLjq1KmjTz/9VCNHjlSVKlU0atQoSVKlSpWuev8LMnnyZD3//PPq1auXBg8erBMnTujtt99W27Zt9fPPPztcGfrzzz/VuXNn9ejRQ7169dKXX36pMWPGqH79+urSpYukvwPsPffco6SkJA0fPlx+fn6aO3euVq9e7bDd5557Tunp6Tp27Jh5HD08PBxqXnnlFTk5Oempp55Senq6pk6dqn79+mnTpk2SpOzsbIWFhSkrK0tPPPGE/Pz89Ntvv2np0qVKS0uTl5fXJfc7OjpaAwcOVPPmzTVlyhSlpKRo+vTpWr9+vbnfzz33nGrVqqX333/fvM23evXqhT7G+/fv14MPPqhBgwYpIiJCH330kQYMGKCmTZuqbt26DrXDhg2Tt7e3Jk6cqISEBM2aNUtHjhwxA/nVatu2rZ588knNmDFDzz77rOrUqSNJ5j+LYvz48XrppZfUtWtXde3aVT/99JM6deqk7Oxsh7qJEydqypQpGjx4sFq0aKGMjAxt3bpVP/30kzp27Fjk7QO4CRkAAAdz5swxJBlbtmy5ZI2Xl5fRuHFj8/2ECROMC/+V+tZbbxmSjBMnTlxyHVu2bDEkGXPmzMm37O677zYkGbNnzy5w2d13322+X716tSHJuO2224yMjAxzfMGCBYYkY/r06eZYtWrVjIiIiCuu83K9RUREGNWqVTPfL1682JBkvPTSSw51Dz74oGGz2Yz9+/ebY5IMu93uMLZ9+3ZDkvH222/n29aFpk2bZkgyPvvsM3MsOzvbCAkJMTw8PBz2vVq1akZ4ePhl13e1tYcPHzacnZ2NyZMnO4zv3LnTcHFxcRjP+7l98skn5lhWVpbh5+dn9OzZ0xx74403DEnG4sWLzbEzZ84YtWvXNiQZq1evNsfDw8MdjneevJ97nTp1jKysLHN8+vTphiRj586dhmEYxs8//2xIMhYuXHjlg3GB7Oxsw8fHx6hXr55x5swZc3zp0qWGJGP8+PHm2NX8nbm49tChQ+ZYtWrVDEnGunXrzLHU1FTD1dXVGDVqVL7PNm3a1MjOzjbHp06dakgyvvnmG3NMkjFhwoR827/478DChQvzHfOrdfG+pKamGna73QgPDzdyc3PNumeffdaQ5LDdhg0bXvU5CuDWxi2LAFAEHh4el51tMe+KyTfffFPkCTBcXV01cODAq65/5JFHVLZsWfP9gw8+qMqVK+u7774r0vav1nfffSdnZ2c9+eSTDuOjRo2SYRhavny5w3hoaKjDFZQGDRrI09NTBw8evOJ2/Pz81LdvX3OsVKlSevLJJ5WZmam1a9cWw97k9/XXXys3N1e9evXS77//br78/PxUs2bNfFe1PDw89PDDD5vv7Xa7WrRo4bB/MTExuu2223TfffeZY25ubpe84no5AwcOdHiuMO+KZt728q6ArVixQn/99ddVr3fr1q1KTU3Vf/7zH7m5uZnj4eHhql27tpYtW1boXi8nODjY7F36+6pmrVq1CjwvhgwZ4vAs4+OPPy4XF5drfq5fycqVK5Wdna0nnnjC4UpdQROyeHt7a/fu3dq3b9917BDAjYhABgBFkJmZ6RB+Lta7d2+1bt1agwcPlq+vr/r06aMFCxYUKpzddttthZrAo2bNmg7vbTabatSocc2n8z5y5Ij8/f3zHY+8276OHDniMF61atV86yhXrly+Z4AK2k7NmjXl5OT4n65Lbae47Nu3T4ZhqGbNmqpUqZLDa8+ePeaEFnmqVKmS77a5i/fvyJEjql69er66GjVqFLq/i49nuXLlJMncXlBQkKKiovThhx+qYsWKCgsL0zvvvHPF58fyjmetWrXyLatdu3axH+/CnBcXn+seHh6qXLmy5VPX5x2Ti/urVKmS+XPJ88ILLygtLU133HGH6tevr9GjR2vHjh3XrVcANw4CGQAU0rFjx5Senn7ZX57d3d21bt06rVy5Uv3799eOHTvUu3dvdezYUTk5OVe1ncI893W1LvV8zdX2VBwuNSudcdEEIDeK3Nxc2Ww2xcTEKDY2Nt/rvffec6i/3vt3Ndt74403tGPHDj377LM6c+aMnnzySdWtW1fHjh27Jj0VxfU6btfzXL+ctm3b6sCBA/roo49Ur149ffjhh2rSpIk+/PBDq1sDcJ0RyACgkD799FNJUlhY2GXrnJyc1KFDB7355pv65ZdfNHnyZK1atcq8xa0wkw9cjYtvfTIMQ/v373eYla9cuXJKS0vL99mLr3YUprdq1arp+PHj+W7h3Lt3r7m8OFSrVk379u3Ld5WxuLdzserVq8swDAUFBSk0NDTfq1WrVoVeZ7Vq1XTgwIF8YePi2RGl4jtP6tevr3HjxmndunX63//+p99++02zZ8++bI+SlJCQkG9ZQkLCNTveV+Picz0zM1NJSUlXPNezs7OVlJTkMFacfw/zjsnF/Z04caLAK33ly5fXwIED9cUXX+jo0aNq0KBBgTNDAri1EcgAoBBWrVqlF198UUFBQerXr98l606ePJlvLO8LlrOysiRJZcqUkaQCA1JRfPLJJw6h6Msvv1RSUpI5s5/0d7jYuHGjw4xvS5cuzTd9e2F669q1q3JycjRz5kyH8bfeeks2m81h+/9E165dlZycrPnz55tj58+f19tvvy0PDw/dfffdxbKdi/Xo0UPOzs6aNGlSvgBlGIb++OOPQq8zLCxMv/32m7799ltz7OzZs/rggw/y1ZYpU+aqpqe/lIyMDJ0/f95hrH79+nJycjLPxYI0a9ZMPj4+mj17tkPd8uXLtWfPHoWHhxe5p3/q/fff17lz58z3s2bN0vnz5/Od6+vWrcv3uYuvkBXn38PQ0FCVKlVKb7/9tsO5Mm3atHy1F583Hh4eqlGjxmV/JgBuTUx7DwCXsHz5cu3du1fnz59XSkqKVq1apdjYWFWrVk3ffvutw0QHF3vhhRe0bt06hYeHq1q1akpNTdW7776rKlWqqE2bNpL+/oXR29tbs2fPVtmyZVWmTBm1bNlSQUFBReq3fPnyatOmjQYOHKiUlBRNmzZNNWrUcJgoYvDgwfryyy/VuXNn9erVSwcOHNBnn32Wb5rywvTWrVs3tW/fXs8995wOHz6shg0b6vvvv9c333yjESNGFGkK9IIMGTJE7733ngYMGKBt27YpMDBQX375pdavX69p06Zd9pm+K9m/f79eeumlfOONGzdWeHi4XnrpJY0dO1aHDx9W9+7dVbZsWR06dEiLFi3SkCFD9NRTTxVqe//+9781c+ZM9e3bV8OHD1flypX1+eefm+fUhVdtmjZtqvnz5ysqKkrNmzeXh4eHunXrdtXbWrVqlYYNG6Z//etfuuOOO3T+/Hl9+umncnZ2Vs+ePS/5uVKlSunVV1/VwIEDdffdd6tv377mtPeBgYEaOXJkofa5OGVnZ6tDhw7q1auXEhIS9O6776pNmzYOk6QMHjxYQ4cOVc+ePdWxY0dt375dK1asUMWKFR3W1ahRIzk7O+vVV19Venq6XF1ddc8998jHx6fQfVWqVElPPfWUpkyZonvvvVddu3bVzz//rOXLl+fbbnBwsNq1a6emTZuqfPny2rp1q7788ksNGzasaAcFwM3LmskdAeDGlTeVdd7Lbrcbfn5+RseOHY3p06c7TK+e5+Jp7+Pi4oz777/f8Pf3N+x2u+Hv72/07dvX+PXXXx0+98033xjBwcGGi4uLwzTzd999t1G3bt0C+7vUtPdffPGFMXbsWMPHx8dwd3c3wsPDjSNHjuT7/BtvvGHcdttthqurq9G6dWtj69at+dZ5ud4unvbeMAzj1KlTxsiRIw1/f3+jVKlSRs2aNY3XXnvNYepvw/h7KvLIyMh8PV1qOv6LpaSkGAMHDjQqVqxo2O12o379+gVOzV/Yae8v/Hlf+Bo0aJBZ99VXXxlt2rQxypQpY5QpU8aoXbu2ERkZaSQkJJg1l/q5FXTMDh48aISHhxvu7u5GpUqVjFGjRhlfffWVIcnYuHGjWZeZmWk89NBDhre3tyHJXE/ez/3i6ewPHTrk8PM6ePCg8eijjxrVq1c33NzcjPLlyxvt27c3Vq5ceVXHZ/78+Ubjxo0NV1dXo3z58ka/fv2MY8eOOdQUx7T3Bf28Lj4v8z67du1aY8iQIUa5cuUMDw8Po1+/fsYff/zh8NmcnBxjzJgxRsWKFY3SpUsbYWFhxv79+ws81z744APj9ttvN5ydnQs1BX5B+5KTk2NMmjTJqFy5suHu7m60a9fO2LVrV77tvvTSS0aLFi0Mb29vw93d3ahdu7YxefJkh+n8AZQMNsO4QZ+iBgCghJk2bZpGjhypY8eO6bbbbrO6nRtO3hdVb9myRc2aNbO6HQAoFjxDBgCABc6cOePw/uzZs3rvvfdUs2ZNwhgAlCA8QwYAgAV69OihqlWrqlGjRkpPT9dnn32mvXv36vPPP7e6tRIvMzNTmZmZl62pVKnSJafqB4DCIJABAGCBsLAwffjhh/r888+Vk5Oj4OBgzZs3T71797a6tRLv9ddf16RJky5bc+jQIYdp9gGgqHiGDAAA4AIHDx7UwYMHL1vTpk2by860CgBXi0AGAAAAABZhUg8AAAAAsAjPkBWT3NxcHT9+XGXLlnX4Qk8AAAAAJYthGDp16pT8/f3l5HT5a2AEsmJy/PhxBQQEWN0GAAAAgBvE0aNHVaVKlcvWEMiKSdmyZSX9fdA9PT0t7gYAAACAVTIyMhQQEGBmhMshkBWTvNsUPT09CWQAAAAArupRJib1AAAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxiaSCbNWuWGjRoYE6EERISouXLl5vLz549q8jISFWoUEEeHh7q2bOnUlJSHNaRmJio8PBwlS5dWj4+Pho9erTOnz/vULNmzRo1adJErq6uqlGjhqKjo/P18s477ygwMFBubm5q2bKlNm/efE32GQAAAADyWBrIqlSpoldeeUXbtm3T1q1bdc899+j+++/X7t27JUkjR47UkiVLtHDhQq1du1bHjx9Xjx49zM/n5OQoPDxc2dnZ+vHHH/Xxxx8rOjpa48ePN2sOHTqk8PBwtW/fXvHx8RoxYoQGDx6sFStWmDXz589XVFSUJkyYoJ9++kkNGzZUWFiYUlNTr9/BAAAAAFDi2AzDMKxu4kLly5fXa6+9pgcffFCVKlXS3Llz9eCDD0qS9u7dqzp16mjDhg1q1aqVli9frnvvvVfHjx+Xr6+vJGn27NkaM2aMTpw4IbvdrjFjxmjZsmXatWuXuY0+ffooLS1NMTExkqSWLVuqefPmmjlzpiQpNzdXAQEBeuKJJ/TMM88U2GdWVpaysrLM93nfNZCens609wAAAEAJlpGRIS8vr6vKBjfMM2Q5OTmaN2+eTp8+rZCQEG3btk3nzp1TaGioWVO7dm1VrVpVGzZskCRt2LBB9evXN8OYJIWFhSkjI8O8yrZhwwaHdeTV5K0jOztb27Ztc6hxcnJSaGioWVOQKVOmyMvLy3wFBAT884MAAAAAoESxPJDt3LlTHh4ecnV11dChQ7Vo0SIFBwcrOTlZdrtd3t7eDvW+vr5KTk6WJCUnJzuEsbzlecsuV5ORkaEzZ87o999/V05OToE1eesoyNixY5Wenm6+jh49WqT9BwAAAFByuVjdQK1atRQfH6/09HR9+eWXioiI0Nq1a61u64pcXV3l6upqdRsAAAAAbmKWBzK73a4aNWpIkpo2baotW7Zo+vTp6t27t7Kzs5WWluZwlSwlJUV+fn6SJD8/v3yzIebNwnhhzcUzM6akpMjT01Pu7u5ydnaWs7NzgTV56wAAAACAa8HyWxYvlpubq6ysLDVt2lSlSpVSXFycuSwhIUGJiYkKCQmRJIWEhGjnzp0OsyHGxsbK09NTwcHBZs2F68iryVuH3W5X06ZNHWpyc3MVFxdn1gAAAADAtWDpFbKxY8eqS5cuqlq1qk6dOqW5c+dqzZo1WrFihby8vDRo0CBFRUWpfPny8vT01BNPPKGQkBC1atVKktSpUycFBwerf//+mjp1qpKTkzVu3DhFRkaatxMOHTpUM2fO1NNPP61HH31Uq1at0oIFC7Rs2TKzj6ioKEVERKhZs2Zq0aKFpk2bptOnT2vgwIGWHBcAAAAAJYOlgSw1NVWPPPKIkpKS5OXlpQYNGmjFihXq2LGjJOmtt96Sk5OTevbsqaysLIWFhendd981P+/s7KylS5fq8ccfV0hIiMqUKaOIiAi98MILZk1QUJCWLVumkSNHavr06apSpYo+/PBDhYWFmTW9e/fWiRMnNH78eCUnJ6tRo0aKiYnJN9EHAAAAABSnG+57yG5WhfmuAQAAAAC3rpvye8gAAAAAoKQhkAEAAACARQhkAAAAAGARAhkAAAAAWMTyL4bGtdGtW9E+t2RJ8fYBAAAA4NK4QgYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEUsD2ZQpU9S8eXOVLVtWPj4+6t69uxISEhxq2rVrJ5vN5vAaOnSoQ01iYqLCw8NVunRp+fj4aPTo0Tp//rxDzZo1a9SkSRO5urqqRo0aio6OztfPO++8o8DAQLm5ually5bavHlzse8zAAAAAOSxNJCtXbtWkZGR2rhxo2JjY3Xu3Dl16tRJp0+fdqh77LHHlJSUZL6mTp1qLsvJyVF4eLiys7P1448/6uOPP1Z0dLTGjx9v1hw6dEjh4eFq37694uPjNWLECA0ePFgrVqwwa+bPn6+oqChNmDBBP/30kxo2bKiwsDClpqZe+wMBAAAAoESyGYZhWN1EnhMnTsjHx0dr165V27ZtJf19haxRo0aaNm1agZ9Zvny57r33Xh0/fly+vr6SpNmzZ2vMmDE6ceKE7Ha7xowZo2XLlmnXrl3m5/r06aO0tDTFxMRIklq2bKnmzZtr5syZkqTc3FwFBAToiSee0DPPPHPF3jMyMuTl5aX09HR5enr+k8NQLLp1K9rnliwp3j4AAACAkqYw2eCGeoYsPT1dklS+fHmH8c8//1wVK1ZUvXr1NHbsWP3111/msg0bNqh+/fpmGJOksLAwZWRkaPfu3WZNaGiowzrDwsK0YcMGSVJ2dra2bdvmUOPk5KTQ0FCz5mJZWVnKyMhweAEAAABAYbhY3UCe3NxcjRgxQq1bt1a9evXM8YceekjVqlWTv7+/duzYoTFjxighIUFff/21JCk5OdkhjEky3ycnJ1+2JiMjQ2fOnNGff/6pnJycAmv27t1bYL9TpkzRpEmT/tlOAwAAACjRbphAFhkZqV27dumHH35wGB8yZIj55/r166ty5crq0KGDDhw4oOrVq1/vNk1jx45VVFSU+T4jI0MBAQGW9QMAAADg5nNDBLJhw4Zp6dKlWrdunapUqXLZ2pYtW0qS9u/fr+rVq8vPzy/fbIgpKSmSJD8/P/OfeWMX1nh6esrd3V3Ozs5ydnYusCZvHRdzdXWVq6vr1e8kAAAAAFzE0mfIDMPQsGHDtGjRIq1atUpBQUFX/Ex8fLwkqXLlypKkkJAQ7dy502E2xNjYWHl6eio4ONisiYuLc1hPbGysQkJCJEl2u11NmzZ1qMnNzVVcXJxZAwAAAADFzdIrZJGRkZo7d66++eYblS1b1nzmy8vLS+7u7jpw4IDmzp2rrl27qkKFCtqxY4dGjhyptm3bqkGDBpKkTp06KTg4WP3799fUqVOVnJyscePGKTIy0ryCNXToUM2cOVNPP/20Hn30Ua1atUoLFizQsmXLzF6ioqIUERGhZs2aqUWLFpo2bZpOnz6tgQMHXv8DAwAAAKBEsHTae5vNVuD4nDlzNGDAAB09elQPP/ywdu3apdOnTysgIEAPPPCAxo0b5zB95JEjR/T4449rzZo1KlOmjCIiIvTKK6/IxeX/5801a9Zo5MiR+uWXX1SlShU9//zzGjBggMN2Z86cqddee03Jyclq1KiRZsyYYd4ieSVMew8AAABAKlw2uKG+h+xmRiADAAAAIN3E30MGAAAAACUJgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCKWBrIpU6aoefPmKlu2rHx8fNS9e3clJCQ41Jw9e1aRkZGqUKGCPDw81LNnT6WkpDjUJCYmKjw8XKVLl5aPj49Gjx6t8+fPO9SsWbNGTZo0kaurq2rUqKHo6Oh8/bzzzjsKDAyUm5ubWrZsqc2bNxf7PgMAAABAHksD2dq1axUZGamNGzcqNjZW586dU6dOnXT69GmzZuTIkVqyZIkWLlyotWvX6vjx4+rRo4e5PCcnR+Hh4crOztaPP/6ojz/+WNHR0Ro/frxZc+jQIYWHh6t9+/aKj4/XiBEjNHjwYK1YscKsmT9/vqKiojRhwgT99NNPatiwocLCwpSamnp9DgYAAACAEsdmGIZhdRN5Tpw4IR8fH61du1Zt27ZVenq6KlWqpLlz5+rBBx+UJO3du1d16tTRhg0b1KpVKy1fvlz33nuvjh8/Ll9fX0nS7NmzNWbMGJ04cUJ2u11jxozRsmXLtGvXLnNbffr0UVpammJiYiRJLVu2VPPmzTVz5kxJUm5urgICAvTEE0/omWeeuWLvGRkZ8vLyUnp6ujw9PYv70BRat25F+9ySJcXbBwAAAFDSFCYb3FDPkKWnp0uSypcvL0natm2bzp07p9DQULOmdu3aqlq1qjZs2CBJ2rBhg+rXr2+GMUkKCwtTRkaGdu/ebdZcuI68mrx1ZGdna9u2bQ41Tk5OCg0NNWsulpWVpYyMDIcXAAAAABTGDRPIcnNzNWLECLVu3Vr16tWTJCUnJ8tut8vb29uh1tfXV8nJyWbNhWEsb3nessvVZGRk6MyZM/r999+Vk5NTYE3eOi42ZcoUeXl5ma+AgICi7TgAAACAEuuGCWSRkZHatWuX5s2bZ3UrV2Xs2LFKT083X0ePHrW6JQAAAAA3GRerG5CkYcOGaenSpVq3bp2qVKlijvv5+Sk7O1tpaWkOV8lSUlLk5+dn1lw8G2LeLIwX1lw8M2NKSoo8PT3l7u4uZ2dnOTs7F1iTt46Lubq6ytXVtWg7DAAAAACy+AqZYRgaNmyYFi1apFWrVikoKMhhedOmTVWqVCnFxcWZYwkJCUpMTFRISIgkKSQkRDt37nSYDTE2Nlaenp4KDg42ay5cR15N3jrsdruaNm3qUJObm6u4uDizBgAAAACKm6VXyCIjIzV37lx98803Klu2rPm8lpeXl9zd3eXl5aVBgwYpKipK5cuXl6enp5544gmFhISoVatWkqROnTopODhY/fv319SpU5WcnKxx48YpMjLSvII1dOhQzZw5U08//bQeffRRrVq1SgsWLNCyZcvMXqKiohQREaFmzZqpRYsWmjZtmk6fPq2BAwde/wMDAAAAoESwNJDNmjVLktSuXTuH8Tlz5mjAgAGSpLfeektOTk7q2bOnsrKyFBYWpnfffdesdXZ21tKlS/X4448rJCREZcqUUUREhF544QWzJigoSMuWLdPIkSM1ffp0ValSRR9++KHCwsLMmt69e+vEiRMaP368kpOT1ahRI8XExOSb6AMAAAAAissN9T1kNzO+hwwAAACAdBN/DxkAAAAAlCQEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsEiRAtnBgweLuw8AAAAAKHGKFMhq1Kih9u3b67PPPtPZs2eLuycAAAAAKBGKFMh++uknNWjQQFFRUfLz89O///1vbd68ubh7AwAAAIBbWpECWaNGjTR9+nQdP35cH330kZKSktSmTRvVq1dPb775pk6cOFHcfQIAAADALecfTerh4uKiHj16aOHChXr11Ve1f/9+PfXUUwoICNAjjzyipKSk4uoTAAAAAG45/yiQbd26Vf/5z39UuXJlvfnmm3rqqad04MABxcbG6vjx47r//vuLq08AAAAAuOW4FOVDb775pubMmaOEhAR17dpVn3zyibp27Sonp7/zXVBQkKKjoxUYGFicvQIAAADALaVIgWzWrFl69NFHNWDAAFWuXLnAGh8fH/33v//9R80BAAAAwK2sSIFs3759V6yx2+2KiIgoyuoBAAAAoEQo0jNkc+bM0cKFC/ONL1y4UB9//PE/bgoAAAAASoIiBbIpU6aoYsWK+cZ9fHz08ssv/+OmAAAAAKAkKFIgS0xMVFBQUL7xatWqKTEx8R83BQAAAAAlQZECmY+Pj3bs2JFvfPv27apQocI/bgoAAAAASoIiBbK+ffvqySef1OrVq5WTk6OcnBytWrVKw4cPV58+fYq7RwAAAAC4JRVplsUXX3xRhw8fVocOHeTi8vcqcnNz9cgjj/AMGQAAAABcpSIFMrvdrvnz5+vFF1/U9u3b5e7urvr166tatWrF3R8AAAAA3LKKFMjy3HHHHbrjjjuKqxcAAAAAKFGKFMhycnIUHR2tuLg4paamKjc312H5qlWriqU5AAAAALiVFSmQDR8+XNHR0QoPD1e9evVks9mKuy8AAAAAuOUVKZDNmzdPCxYsUNeuXYu7HwAAAAAoMYo07b3dbleNGjWKuxcAAAAAKFGKFMhGjRql6dOnyzCM4u4HAAAAAEqMIt2y+MMPP2j16tVavny56tatq1KlSjks//rrr4ulOQAAAAC4lRUpkHl7e+uBBx4o7l4AAAAAoEQpUiCbM2dOcfcBAAAAACVOkZ4hk6Tz589r5cqVeu+993Tq1ClJ0vHjx5WZmVlszQEAAADAraxIV8iOHDmizp07KzExUVlZWerYsaPKli2rV199VVlZWZo9e3Zx9wkAAAAAt5wiXSEbPny4mjVrpj///FPu7u7m+AMPPKC4uLhiaw4AAAAAbmVFukL2v//9Tz/++KPsdrvDeGBgoH777bdiaQwAAAAAbnVFukKWm5urnJycfOPHjh1T2bJl/3FTAAAAAFASFCmQderUSdOmTTPf22w2ZWZmasKECeratWtx9QYAAAAAt7Qi3bL4xhtvKCwsTMHBwTp79qweeugh7du3TxUrVtQXX3xR3D0CAAAAwC2pSIGsSpUq2r59u+bNm6cdO3YoMzNTgwYNUr9+/Rwm+QAAAAAAXFqRApkkubi46OGHHy7OXgAAAACgRCnSM2SffPLJZV9Xa926derWrZv8/f1ls9m0ePFih+UDBgyQzWZzeHXu3Nmh5uTJk+rXr588PT3l7e2tQYMG5fty6h07duiuu+6Sm5ubAgICNHXq1Hy9LFy4ULVr15abm5vq16+v77777uoPCAAAAAAUQZGukA0fPtzh/blz5/TXX3/JbrerdOnSeuSRR65qPadPn1bDhg316KOPqkePHgXWdO7cWXPmzDHfu7q6Oizv16+fkpKSFBsbq3PnzmngwIEaMmSI5s6dK0nKyMhQp06dFBoaqtmzZ2vnzp169NFH5e3trSFDhkiSfvzxR/Xt21dTpkzRvffeq7lz56p79+766aefVK9evas+LgAAAABQGDbDMIziWNG+ffv0+OOPa/To0QoLCyt8IzabFi1apO7du5tjAwYMUFpaWr4rZ3n27Nmj4OBgbdmyRc2aNZMkxcTEqGvXrjp27Jj8/f01a9YsPffcc0pOTja/N+2ZZ57R4sWLtXfvXklS7969dfr0aS1dutRcd6tWrdSoUSPNnj37qvrPyMiQl5eX0tPT5enpWej9L27duhXtc0uWFG8fAAAAQElTmGxQpFsWC1KzZk298sor+a6e/VNr1qyRj4+PatWqpccff1x//PGHuWzDhg3y9vY2w5gkhYaGysnJSZs2bTJr2rZt6/Al1mFhYUpISNCff/5p1oSGhjpsNywsTBs2bLhkX1lZWcrIyHB4AQAAAEBhFFsgk/6e6OP48ePFtr7OnTvrk08+UVxcnF599VWtXbtWXbp0Mb+UOjk5WT4+Pvl6KF++vJKTk80aX19fh5q891eqyVtekClTpsjLy8t8BQQE/LOdBQAAAFDiFOkZsm+//dbhvWEYSkpK0syZM9W6detiaUyS+vTpY/65fv36atCggapXr641a9aoQ4cOxbadohg7dqyioqLM9xkZGYQyAAAAAIVSpEB24XNe0t/Pf1WqVEn33HOP3njjjeLoq0C33367KlasqP3796tDhw7y8/NTamqqQ8358+d18uRJ+fn5SZL8/PyUkpLiUJP3/ko1ecsL4urqmm+CEQAAAAAojCLdspibm+vwysnJUXJysubOnavKlSsXd4+mY8eO6Y8//jC3ERISorS0NG3bts2sWbVqlXJzc9WyZUuzZt26dTp37pxZExsbq1q1aqlcuXJmTVxcnMO2YmNjFRIScs32BQAAAACK9RmywsrMzFR8fLzi4+MlSYcOHVJ8fLwSExOVmZmp0aNHa+PGjTp8+LDi4uJ0//33q0aNGuYsjnXq1FHnzp312GOPafPmzVq/fr2GDRumPn36yN/fX5L00EMPyW63a9CgQdq9e7fmz5+v6dOnO9xuOHz4cMXExOiNN97Q3r17NXHiRG3dulXDhg277scEAAAAQMlRpGnvLwwzV/Lmm29ectmaNWvUvn37fOMRERGaNWuWunfvrp9//llpaWny9/dXp06d9OKLLzpMwHHy5EkNGzZMS5YskZOTk3r27KkZM2bIw8PDrNmxY4ciIyO1ZcsWVaxYUU888YTGjBnjsM2FCxdq3LhxOnz4sGrWrKmpU6eqa9euV72fTHsPAAAAQCpcNihSIGvfvr1+/vlnnTt3TrVq1ZIk/frrr3J2dlaTJk3+/8ptNq1ataqwq78pEcgAAAAASIXLBkWa1KNbt24qW7asPv74Y/M5rD///FMDBw7UXXfdpVGjRhVltQAAAABQohTpCtltt92m77//XnXr1nUY37Vrlzp16lSs30V2s+AKGQAAAACpcNmgSJN6ZGRk6MSJE/nGT5w4oVOnThVllQAAAABQ4hQpkD3wwAMaOHCgvv76ax07dkzHjh3TV199pUGDBqlHjx7F3SMAAAAA3JKK9AzZ7Nmz9dRTT+mhhx4yv9/LxcVFgwYN0muvvVasDQIAAADArapIz5DlOX36tA4cOCBJql69usqUKVNsjd1seIYMAAAAgHQdniHLk5SUpKSkJNWsWVNlypTRP8h2AAAAAFDiFCmQ/fHHH+rQoYPuuOMOde3aVUlJSZKkQYMGMeU9AAAAAFylIgWykSNHqlSpUkpMTFTp0qXN8d69eysmJqbYmgMAAACAW1mRJvX4/vvvtWLFClWpUsVhvGbNmjpy5EixNAYAAAAAt7oiXSE7ffq0w5WxPCdPnpSrq+s/bgoAAAAASoIiBbK77rpLn3zyifneZrMpNzdXU6dOVfv27YutOQAAAAC4lRXplsWpU6eqQ4cO2rp1q7Kzs/X0009r9+7dOnnypNavX1/cPQIAAADALalIV8jq1aunX3/9VW3atNH999+v06dPq0ePHvr5559VvXr14u4RAAAAAG5Jhb5Cdu7cOXXu3FmzZ8/Wc889dy16AgAAAIASodBXyEqVKqUdO3Zci14AAAAAoEQp0i2LDz/8sP773/8Wdy8AAAAAUKIUaVKP8+fP66OPPtLKlSvVtGlTlSlTxmH5m2++WSzNAQAAAMCtrFCB7ODBgwoMDNSuXbvUpEkTSdKvv/7qUGOz2YqvOwAAAAC4hRUqkNWsWVNJSUlavXq1JKl3796aMWOGfH19r0lzAAAAAHArK9QzZIZhOLxfvny5Tp8+XawNAQAAAEBJUaRJPfJcHNAAAAAAAFevUIHMZrPle0aMZ8YAAAAAoGgK9QyZYRgaMGCAXF1dJUlnz57V0KFD882y+PXXXxdfhwAAAABwiypUIIuIiHB4//DDDxdrMwAAAABQkhQqkM2ZM+da9QEAAAAAJc4/mtQDAAAAAFB0BDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsIilgWzdunXq1q2b/P39ZbPZtHjxYoflhmFo/Pjxqly5stzd3RUaGqp9+/Y51Jw8eVL9+vWTp6envL29NWjQIGVmZjrU7NixQ3fddZfc3NwUEBCgqVOn5utl4cKFql27ttzc3FS/fn199913xb6/AAAAAHAhSwPZ6dOn1bBhQ73zzjsFLp86dapmzJih2bNna9OmTSpTpozCwsJ09uxZs6Zfv37avXu3YmNjtXTpUq1bt05Dhgwxl2dkZKhTp06qVq2atm3bptdee00TJ07U+++/b9b8+OOP6tu3rwYNGqSff/5Z3bt3V/fu3bVr165rt/MAAAAASjybYRiG1U1Iks1m06JFi9S9e3dJf18d8/f316hRo/TUU09JktLT0+Xr66vo6Gj16dNHe/bsUXBwsLZs2aJmzZpJkmJiYtS1a1cdO3ZM/v7+mjVrlp577jklJyfLbrdLkp555hktXrxYe/fulST17t1bp0+f1tKlS81+WrVqpUaNGmn27NlX1X9GRoa8vLyUnp4uT0/P4josRdatW9E+t2RJ8fYBAAAAlDSFyQY37DNkhw4dUnJyskJDQ80xLy8vtWzZUhs2bJAkbdiwQd7e3mYYk6TQ0FA5OTlp06ZNZk3btm3NMCZJYWFhSkhI0J9//mnWXLidvJq87RQkKytLGRkZDi8AAAAAKIwbNpAlJydLknx9fR3GfX19zWXJycny8fFxWO7i4qLy5cs71BS0jgu3camavOUFmTJliry8vMxXQEBAYXcRAAAAQAl3wwayG93YsWOVnp5uvo4ePWp1SwAAAABuMjdsIPPz85MkpaSkOIynpKSYy/z8/JSamuqw/Pz58zp58qRDTUHruHAbl6rJW14QV1dXeXp6OrwAAAAAoDBu2EAWFBQkPz8/xcXFmWMZGRnatGmTQkJCJEkhISFKS0vTtm3bzJpVq1YpNzdXLVu2NGvWrVunc+fOmTWxsbGqVauWypUrZ9ZcuJ28mrztAAAAAMC1YGkgy8zMVHx8vOLj4yX9PZFHfHy8EhMTZbPZNGLECL300kv69ttvtXPnTj3yyCPy9/c3Z2KsU6eOOnfurMcee0ybN2/W+vXrNWzYMPXp00f+/v6SpIceekh2u12DBg3S7t27NX/+fE2fPl1RUVFmH8OHD1dMTIzeeOMN7d27VxMnTtTWrVs1bNiw631IAAAAAJQgLlZufOvWrWrfvr35Pi8kRUREKDo6Wk8//bROnz6tIUOGKC0tTW3atFFMTIzc3NzMz3z++ecaNmyYOnToICcnJ/Xs2VMzZswwl3t5een7779XZGSkmjZtqooVK2r8+PEO31V25513au7cuRo3bpyeffZZ1axZU4sXL1a9evWuw1EAAAAAUFLdMN9DdrPje8gAAAAASLfI95ABAAAAwK2OQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARVysbgA3lm7div7ZJUuKrw8AAACgJOAKGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYhEAGAAAAABYhkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFjkhg5kEydOlM1mc3jVrl3bXH727FlFRkaqQoUK8vDwUM+ePZWSkuKwjsTERIWHh6t06dLy8fHR6NGjdf78eYeaNWvWqEmTJnJ1dVWNGjUUHR19PXYPAAAAQAl3QwcySapbt66SkpLM1w8//GAuGzlypJYsWaKFCxdq7dq1On78uHr06GEuz8nJUXh4uLKzs/Xjjz/q448/VnR0tMaPH2/WHDp0SOHh4Wrfvr3i4+M1YsQIDR48WCtWrLiu+wkAAACg5HGxuoErcXFxkZ+fX77x9PR0/fe//9XcuXN1zz33SJLmzJmjOnXqaOPGjWrVqpW+//57/fLLL1q5cqV8fX3VqFEjvfjiixozZowmTpwou92u2bNnKygoSG+88YYkqU6dOvrhhx/01ltvKSws7JJ9ZWVlKSsry3yfkZFRzHsOAAAA4FZ3w18h27dvn/z9/XX77berX79+SkxMlCRt27ZN586dU2hoqFlbu3ZtVa1aVRs2bJAkbdiwQfXr15evr69ZExYWpoyMDO3evdusuXAdeTV567iUKVOmyMvLy3wFBAQUy/4CAAAAKDlu6EDWsmVLRUdHKyYmRrNmzdKhQ4d011136dSpU0pOTpbdbpe3t7fDZ3x9fZWcnCxJSk5Odghjecvzll2uJiMjQ2fOnLlkb2PHjlV6err5Onr06D/dXQAAAAAlzA19y2KXLl3MPzdo0EAtW7ZUtWrVtGDBArm7u1vYmeTq6ipXV1dLewAAAABwc7uhr5BdzNvbW3fccYf2798vPz8/ZWdnKy0tzaEmJSXFfObMz88v36yLee+vVOPp6Wl56AMAAABwa7upAllmZqYOHDigypUrq2nTpipVqpTi4uLM5QkJCUpMTFRISIgkKSQkRDt37lRqaqpZExsbK09PTwUHB5s1F64jryZvHQAAAABwrdzQgeypp57S2rVrdfjwYf3444964IEH5OzsrL59+8rLy0uDBg1SVFSUVq9erW3btmngwIEKCQlRq1atJEmdOnVScHCw+vfvr+3bt2vFihUaN26cIiMjzdsNhw4dqoMHD+rpp5/W3r179e6772rBggUaOXKklbsOAAAAoAS4oZ8hO3bsmPr27as//vhDlSpVUps2bbRx40ZVqlRJkvTWW2/JyclJPXv2VFZWlsLCwvTuu++an3d2dtbSpUv1+OOPKyQkRGXKlFFERIReeOEFsyYoKEjLli3TyJEjNX36dFWpUkUffvjhZae8BwAAAIDiYDMMw7C6iVtBRkaGvLy8lJ6eLk9PT6vbUbdu13+bS5Zc/20CAAAAN5rCZIMb+pZFAAAAALiVEcgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAiBDIAAAAAsAiBDAAAAAAsQiADAAAAAIsQyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMgAAAAAwCIuVjeAW0e3bkX73JIlxdsHAAAAcLPgChkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQAAAIBFCGQAAAAAYBECGQAAAABYxMXqBoBu3Yr2uSVLircPAAAA4HrjChkAAAAAWIRABgAAAAAWIZABAAAAgEV4huwi77zzjl577TUlJyerYcOGevvtt9WiRQur20IBePYMAAAANzuukF1g/vz5ioqK0oQJE/TTTz+pYcOGCgsLU2pqqtWtAQAAALgF2QzDMKxu4kbRsmVLNW/eXDNnzpQk5ebmKiAgQE888YSeeeaZy342IyNDXl5eSk9Pl6en5/Vo97KKevUIxY8rcgAAACVLYbIBtyz+n+zsbG3btk1jx441x5ycnBQaGqoNGzbkq8/KylJWVpb5Pj09XdLfB/9GcO6c1R0gT+fOVndw61mwwOoOAAAALi0vE1zNtS8C2f/5/ffflZOTI19fX4dxX19f7d27N1/9lClTNGnSpHzjAQEB16xHAH/z8rK6AwAAgCs7deqUvK7wiwuBrIjGjh2rqKgo831ubq5OnjypChUqyGazWdJTRkaGAgICdPTo0RvitkmUDJx3sArnHqzCuQcrcN7dXAzD0KlTp+Tv73/FWgLZ/6lYsaKcnZ2VkpLiMJ6SkiI/P7989a6urnJ1dXUY8/b2vpYtXjVPT0/+ouK647yDVTj3YBXOPViB8+7mcaUrY3mYZfH/2O12NW3aVHFxceZYbm6u4uLiFBISYmFnAAAAAG5VXCG7QFRUlCIiItSsWTO1aNFC06ZN0+nTpzVw4ECrWwMAAABwCyKQXaB37946ceKExo8fr+TkZDVq1EgxMTH5Jvq4Ubm6umrChAn5bqUEriXOO1iFcw9W4dyDFTjvbl18DxkAAAAAWIRnyAAAAADAIgQyAAAAALAIgQwAAAAALEIgAwAAAACLEMhuEe+8844CAwPl5uamli1bavPmzVa3hJvIlClT1Lx5c5UtW1Y+Pj7q3r27EhISHGrOnj2ryMhIVahQQR4eHurZs2e+L1JPTExUeHi4SpcuLR8fH40ePVrnz593qFmzZo2aNGkiV1dX1ahRQ9HR0dd693CTeOWVV2Sz2TRixAhzjPMO18pvv/2mhx9+WBUqVJC7u7vq16+vrVu3mssNw9D48eNVuXJlubu7KzQ0VPv27XNYx8mTJ9WvXz95enrK29tbgwYNUmZmpkPNjh07dNddd8nNzU0BAQGaOnXqddk/3JhycnL0/PPPKygoSO7u7qpevbpefPFFXTjHHudeCWTgpjdv3jzDbrcbH330kbF7927jscceM7y9vY2UlBSrW8NNIiwszJgzZ46xa9cuIz4+3ujatatRtWpVIzMz06wZOnSoERAQYMTFxRlbt241WrVqZdx5553m8vPnzxv16tUzQkNDjZ9//tn47rvvjIoVKxpjx441aw4ePGiULl3aiIqKMn755Rfj7bffNpydnY2YmJjrur+48WzevNkIDAw0GjRoYAwfPtwc57zDtXDy5EmjWrVqxoABA4xNmzYZBw8eNFasWGHs37/frHnllVcMLy8vY/Hixcb27duN++67zwgKCjLOnDlj1nTu3Nlo2LChsXHjRuN///ufUaNGDaNv377m8vT0dMPX19fo16+fsWvXLuOLL74w3N3djffee++67i9uHJMnTzYqVKhgLF261Dh06JCxcOFCw8PDw5g+fbpZw7lX8hDIbgEtWrQwIiMjzfc5OTmGv7+/MWXKFAu7ws0sNTXVkGSsXbvWMAzDSEtLM0qVKmUsXLjQrNmzZ48hydiwYYNhGIbx3XffGU5OTkZycrJZM2vWLMPT09PIysoyDMMwnn76aaNu3boO2+rdu7cRFhZ2rXcJN7BTp04ZNWvWNGJjY427777bDGScd7hWxowZY7Rp0+aSy3Nzcw0/Pz/jtddeM8fS0tIMV1dX44svvjAMwzB++eUXQ5KxZcsWs2b58uWGzWYzfvvtN8MwDOPdd981ypUrZ56LeduuVatWce8SbhLh4eHGo48+6jDWo0cPo1+/foZhcO6VVNyyeJPLzs7Wtm3bFBoaao45OTkpNDRUGzZssLAz3MzS09MlSeXLl5ckbdu2TefOnXM4z2rXrq2qVaua59mGDRtUv359hy9SDwsLU0ZGhnbv3m3WXLiOvBrO1ZItMjJS4eHh+c4NzjtcK99++62aNWumf/3rX/Lx8VHjxo31wQcfmMsPHTqk5ORkh/PGy8tLLVu2dDj3vL291axZM7MmNDRUTk5O2rRpk1nTtm1b2e12syYsLEwJCQn6888/r/Vu4gZ05513Ki4uTr/++qskafv27frhhx/UpUsXSZx7JZWL1Q3gn/n999+Vk5Pj8MuIJPn6+mrv3r0WdYWbWW5urkaMGKHWrVurXr16kqTk5GTZ7XZ5e3s71Pr6+io5OdmsKeg8zFt2uZqMjAydOXNG7u7u12KXcAObN2+efvrpJ23ZsiXfMs47XCsHDx7UrFmzFBUVpWeffVZbtmzRk08+KbvdroiICPPcKei8ufC88vHxcVju4uKi8uXLO9QEBQXlW0fesnLlyl2T/cON65lnnlFGRoZq164tZ2dn5eTkaPLkyerXr58kce6VUAQyAA4iIyO1a9cu/fDDD1a3glvc0aNHNXz4cMXGxsrNzc3qdlCC5ObmqlmzZnr55ZclSY0bN9auXbs0e/ZsRUREWNwdbmULFizQ559/rrlz56pu3bqKj4/XiBEj5O/vz7lXgnHL4k2uYsWKcnZ2zjfrWEpKivz8/CzqCjerYcOGaenSpVq9erWqVKlijvv5+Sk7O1tpaWkO9ReeZ35+fgWeh3nLLlfj6enJVYoSaNu2bUpNTVWTJk3k4uIiFxcXrV27VjNmzJCLi4t8fX0573BNVK5cWcHBwQ5jderUUWJioqT/f+5c7r+tfn5+Sk1NdVh+/vx5nTx5slDnJ0qW0aNH65lnnlGfPn1Uv3599e/fXyNHjtSUKVMkce6VVASym5zdblfTpk0VFxdnjuXm5iouLk4hISEWdoabiWEYGjZsmBYtWqRVq1blu82hadOmKlWqlMN5lpCQoMTERPM8CwkJ0c6dOx3+IxEbGytPT0/zF5+QkBCHdeTVcK6WTB06dNDOnTsVHx9vvpo1a6Z+/fqZf+a8w7XQunXrfF/t8euvv6patWqSpKCgIPn5+TmcNxkZGdq0aZPDuZeWlqZt27aZNatWrVJubq5atmxp1qxbt07nzp0za2JjY1WrVi1uGSuh/vrrLzk5Of767ezsrNzcXEmceyWW1bOK4J+bN2+e4erqakRHRxu//PKLMWTIEMPb29th1jHgch5//HHDy8vLWLNmjZGUlGS+/vrrL7Nm6NChRtWqVY1Vq1YZW7duNUJCQoyQkBBzed704506dTLi4+ONmJgYo1KlSgVOPz569Ghjz549xjvvvMP043Bw4SyLhsF5h2tj8+bNhouLizF58mRj3759xueff26ULl3a+Oyzz8yaV155xfD29ja++eYbY8eOHcb9999f4NTjjRs3NjZt2mT88MMPRs2aNR2mHk9LSzN8fX2N/v37G7t27TLmzZtnlC5dmqnHS7CIiAjjtttuM6e9//rrr42KFSsaTz/9tFnDuVfyEMhuEW+//bZRtWpVw263Gy1atDA2btxodUu4iUgq8DVnzhyz5syZM8Z//vMfo1y5ckbp0qWNBx54wEhKSnJYz+HDh40uXboY7u7uRsWKFY1Ro0YZ586dc6hZvXq10ahRI8Nutxu33367wzaAiwMZ5x2ulSVLlhj16tUzXF1djdq1axvvv/++w/Lc3Fzj+eefN3x9fQ1XV1ejQ4cORkJCgkPNH3/8YfTt29fw8PAwPD09jYEDBxqnTp1yqNm+fbvRpk0bw9XV1bjtttuMV1555ZrvG25cGRkZxvDhw42qVasabm5uxu23324899xzDtPTc+6VPDbDuOCrwQEAAAAA1w3PkAEAAACARQhkAAAAAGARAhkAAAAAWIRABgAAAAAWIZABAAAAgEUIZAAAAABgEQIZAAAAAFiEQAYAAAAAFiGQAQBKhAEDBqh79+7Fvt7k5GR17NhRZcqUkbe393Xd9rUQGBioadOmXbbGZrNp8eLF16UfALjVEcgAAMXmRggehw8fls1mU3x8/HXZ3ltvvaWkpCTFx8fr119/LbBm+vTpio6Ovi79XCg6OvqSIfFStmzZoiFDhlybhgAA+bhY3QAAADezAwcOqGnTpqpZs+Yla7y8vK5jR/9MpUqVrG4BAEoUrpABAK6bXbt2qUuXLvLw8JCvr6/69++v33//3Vzerl07Pfnkk3r66adVvnx5+fn5aeLEiQ7r2Lt3r9q0aSM3NzcFBwdr5cqVDrfQBQUFSZIaN24sm82mdu3aOXz+9ddfV+XKlVWhQgVFRkbq3Llzl+151qxZql69uux2u2rVqqVPP/3UXBYYGKivvvpKn3zyiWw2mwYMGFDgOi6+cng1+2mz2TRr1ix16dJF7u7uuv322/Xll1+ay9esWSObzaa0tDRzLD4+XjabTYcPH9aaNWs0cOBApaeny2azyWaz5dtGQS6+ZXHfvn1q27atebxjY2Md6rOzszVs2DBVrlxZbm5uqlatmqZMmXLF7QAA/kYgAwBcF2lpabrnnnvUuHFjbd26VTExMUpJSVGvXr0c6j7++GOVKVNGmzZt0tSpU/XCCy+YISAnJ0fdu3dX6dKltWnTJr3//vt67rnnHD6/efNmSdLKlSuVlJSkr7/+2ly2evVqHThwQKtXr9bHH3+s6Ojoy95KuGjRIg0fPlyjRo3Srl279O9//1sDBw7U6tWrJf19e1/nzp3Vq1cvJSUlafr06Vd9PC63n3mef/559ezZU9u3b1e/fv3Up08f7dmz56rWf+edd2ratGny9PRUUlKSkpKS9NRTT111f5KUm5urHj16yG63a9OmTZo9e7bGjBnjUDNjxgx9++23WrBggRISEvT5558rMDCwUNsBgJKMWxYBANfFzJkz1bhxY7388svm2EcffaSAgAD9+uuvuuOOOyRJDRo00IQJEyRJNWvW1MyZMxUXF6eOHTsqNjZWBw4c0Jo1a+Tn5ydJmjx5sjp27GiuM++WuwoVKpg1ecqVK6eZM2fK2dlZtWvXVnh4uOLi4vTYY48V2PPrr7+uAQMG6D//+Y8kKSoqShs3btTrr7+u9u3bq1KlSnJ1dZW7u3u+bV3J5fYzz7/+9S8NHjxYkvTiiy8qNjZWb7/9tt59990rrt9ut8vLy0s2m63QveVZuXKl9u7dqxUrVsjf31+S9PLLL6tLly5mTWJiomrWrKk2bdrIZrOpWrVqRdoWAJRUXCEDAFwX27dv1+rVq+Xh4WG+ateuLenv57DyNGjQwOFzlStXVmpqqiQpISFBAQEBDgGjRYsWV91D3bp15ezsXOC6C7Jnzx61bt3aYax169ZXfZXqci63n3lCQkLyvS+ObV+tPXv2KCAgwAxjBfU0YMAAxcfHq1atWnryySf1/fffX7f+AOBWwBUyAMB1kZmZqW7duunVV1/Nt6xy5crmn0uVKuWwzGazKTc3t1h6uJbrvt69ODn9/f9UDcMwx670PNy10KRJEx06dEjLly/XypUr1atXL4WGhjo87wYAuDSukAEArosmTZpo9+7dCgwMVI0aNRxeZcqUuap11KpVS0ePHlVKSoo5tmXLFocau90u6e/nzf6pOnXqaP369Q5j69evV3Bw8D9e99XYuHFjvvd16tSR9P9vzUxKSjKXXzzVv91u/0fHoU6dOjp69KjDNi7uSZI8PT3Vu3dvffDBB5o/f76++uornTx5ssjbBYCShCtkAIBilZ6eni8Y5M1o+MEHH6hv377m7IL79+/XvHnz9OGHHzrcSngpHTt2VPXq1RUREaGpU6fq1KlTGjdunKS/rzBJko+Pj9zd3RUTE6MqVarIzc2tyNPOjx49Wr169VLjxo0VGhqqJUuW6Ouvv9bKlSuLtL7CWrhwoZo1a6Y2bdro888/1+bNm/Xf//5XklSjRg0FBARo4sSJmjx5sn799Ve98cYbDp8PDAxUZmam4uLi1LBhQ5UuXVqlS5e+6u2HhobqjjvuUEREhF577TVlZGTkm0TlzTffVOXKldW4cWM5OTlp4cKF8vPzK/T3nwFAScUVMgBAsVqzZo0aN27s8Jo0aZL8/f21fv165eTkqFOnTqpfv75GjBghb29v8/a7K3F2dtbixYuVmZmp5s2ba/DgwWZAcHNzkyS5uLhoxowZeu+99+Tv76/777+/yPvSvXt3TZ8+Xa+//rrq1q2r9957T3PmzMk3lf61MmnSJM2bN08NGjTQJ598oi+++MK8OleqVCl98cUX2rt3rxo0aKBXX31VL730ksPn77zzTg0dOlS9e/dWpUqVNHXq1EJt38nJSYsWLdKZM2fUokULDR48WJMnT3aoKVu2rKZOnapmzZqpefPmOnz4sL777rur/pkCQElnMy68+RwAgJvM+vXr1aZNG+3fv1/Vq1e3up1iY7PZtGjRIofvLwMA3Hq4ZREAcFNZtGiRPDw8VLNmTe3fv1/Dhw9X69atb6kwBgAoOQhkAICbyqlTpzRmzBglJiaqYsWKCg0NzffsFAr2v//9z+E7xC6WmZl5HbsBAEjcsggAQIlx5swZ/fbbb5dcXqNGjevYDQBAIpABAAAAgGWYAgkAAAAALEIgAwAAAACLEMgAAAAAwCIEMgAAAACwCIEMAAAAACxCIAMAAAAAixDIAAAAAMAi/w+s9etEZe3jqQAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_data_lengths(tokenized_train_dataset, tokenized_val_dataset):\n",
        "    lengths = [len(x['input_ids']) for x in tokenized_train_dataset]\n",
        "    lengths += [len(x['input_ids']) for x in tokenized_val_dataset]\n",
        "    print(len(lengths))\n",
        "\n",
        "    # Plotting the histogram\n",
        "    plt.figure(figsize=(10, 6))\n",
        "    plt.hist(lengths, bins=50, alpha=0.7, color='blue')\n",
        "    plt.xlabel('Length of input_ids')\n",
        "    plt.ylabel('Frequency')\n",
        "    plt.title('Distribution of Lengths of input_ids')\n",
        "    # plt.xlim([0, 1500])\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "plot_data_lengths(tokenized_train_dataset, tokenized_val_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "o8OAnPy8yVQT",
        "outputId": "fded4d6c-1ade-4fa1-ae98-ef9b982e0c81"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "39104\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAAIjCAYAAABswtioAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABWq0lEQVR4nO3de3zP9f//8ft7YweHbTnslBkh5/Nh9kkiy7CU6OOQakQ++kxhklQfpw5KJ0rR51NZJyV9opBpxvikEbJEWQgjO4i2mdjYnr8/+u7187Y5bMYLu10vl/cl79fr8X69Hq/nXuZ97/V+Pd8OY4wRAAAAAOCyc7G7AQAAAAAorwhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAUAamTJkih8NxWfbVpUsXdenSxXqekJAgh8Ohzz777LLsf8iQIapTp85l2Vdp5eTkaPjw4fL395fD4dCYMWPsbqnMXe6f+/nExsaqVatW8vDwkMPhUGZmZrF1MTExcjgc2rt372Xt71IoybHUqVNHQ4YMueQ9Abj6EMgA4AyFb7IKHx4eHgoMDFR4eLhee+01HT16tEz2c/DgQU2ZMkVJSUllsr2ydCX3diGee+45xcTE6KGHHtIHH3yg++6776y1derU0e23334ZuyuZ+fPna+bMmXa3cU6HDx9W//795enpqTfeeEMffPCBKleubHdbF+Snn37SlClTromACODqVMHuBgDgSjVt2jTVrVtXJ0+eVFpamhISEjRmzBi98sor+vLLL9WiRQur9qmnntLjjz9eou0fPHhQU6dOVZ06ddSqVasLft3XX39dov2Uxrl6+89//qOCgoJL3sPFWLVqlTp27KjJkyfb3cpFmz9/vrZt23ZFX+XbuHGjjh49qqefflphYWHnrL3vvvs0cOBAubu7X6buzu2nn37S1KlT1aVLlxJf+b3SjgXA1YlABgBn0bNnT7Vr1856PnHiRK1atUq333677rjjDv3888/y9PSUJFWoUEEVKlzaX6l//vmnKlWqJDc3t0u6n/OpWLGirfu/EBkZGWrSpIndbZQbGRkZkiQfH5/z1rq6usrV1fUSd3R5XEvHAsA+fGQRAErg1ltv1b/+9S/t27dPH374obW8uHvI4uLi1KlTJ/n4+KhKlSpq2LChnnjiCUl/3f/Tvn17SdLQoUOtj0fGxMRI+us+sWbNmmnz5s3q3LmzKlWqZL32zHvICuXn5+uJJ56Qv7+/KleurDvuuEP79+93qjnbfSynb/N8vRV3D9mxY8c0btw4BQUFyd3dXQ0bNtRLL70kY4xTncPh0KhRo7R48WI1a9ZM7u7uatq0qWJjY4sf8DNkZGRo2LBh8vPzk4eHh1q2bKn33nvPWl94X9WePXu0bNkyq/ey+Djahx9+qLZt28rT01PVqlXTwIEDi4xv4c/tp59+UteuXVWpUiVdf/31mjFjRpHt7du3T3fccYcqV64sX19fjR07VitWrJDD4VBCQoK1vWXLlmnfvn3WsZw59gUFBXr22WdVq1YteXh4qFu3btq1a5dTzc6dO9WvXz/5+/vLw8NDtWrV0sCBA5WVlXXe4164cKF13DVq1NC9996r3377zemYIyMjJUnt27eXw+E4571Sxd13Vfix0W+++UYdOnSQh4eHbrjhBr3//vvFvnbt2rX6xz/+oerVq8vLy0v333+//vjjD6dah8OhKVOmFNn/6X8HYmJi9Pe//12S1LVrV2uMC8f/fIo7FmOMnnnmGdWqVUuVKlVS165dtX379iKvPXnypKZOnaoGDRrIw8ND1atXV6dOnRQXF3dB+wZw7eAKGQCU0H333acnnnhCX3/9tR588MFia7Zv367bb79dLVq00LRp0+Tu7q5du3Zp3bp1kqTGjRtr2rRpmjRpkkaMGKGbb75ZkvS3v/3N2sbhw4fVs2dPDRw4UPfee6/8/PzO2dezzz4rh8OhCRMmKCMjQzNnzlRYWJiSkpKsK3kX4kJ6O50xRnfccYdWr16tYcOGqVWrVlqxYoXGjx+v3377Ta+++qpT/TfffKPPP/9c//znP1W1alW99tpr6tevn1JSUlS9evWz9nX8+HF16dJFu3bt0qhRo1S3bl0tXLhQQ4YMUWZmpkaPHq3GjRvrgw8+0NixY1WrVi2NGzdOklSzZs0LPv7iPPvss/rXv/6l/v37a/jw4Tp06JBef/11de7cWVu2bHG6MvTHH3+oR48e6tu3r/r376/PPvtMEyZMUPPmzdWzZ09JfwXYW2+9VampqRo9erT8/f01f/58rV692mm/Tz75pLKysnTgwAFrHKtUqeJU8/zzz8vFxUWPPvqosrKyNGPGDA0ePFgbNmyQJOXl5Sk8PFy5ubl6+OGH5e/vr99++01Lly5VZmamvL29z3rcMTExGjp0qNq3b6/p06crPT1ds2bN0rp166zjfvLJJ9WwYUP9+9//tj7mW69evRKP8a5du3T33Xdr2LBhioyM1LvvvqshQ4aobdu2atq0qVPtqFGj5OPjoylTpig5OVlz5szRvn37rEB+oTp37qxHHnlEr732mp544gk1btxYkqz/lsakSZP0zDPPqFevXurVq5e+//57de/eXXl5eU51U6ZM0fTp0zV8+HB16NBB2dnZ2rRpk77//nvddtttpd4/gKuQAQA4mTdvnpFkNm7ceNYab29v07p1a+v55MmTzem/Ul999VUjyRw6dOis29i4caORZObNm1dk3S233GIkmblz5xa77pZbbrGer1692kgy119/vcnOzraWf/rpp0aSmTVrlrUsODjYREZGnneb5+otMjLSBAcHW88XL15sJJlnnnnGqe7uu+82DofD7Nq1y1omybi5uTkt++GHH4wk8/rrrxfZ1+lmzpxpJJkPP/zQWpaXl2dCQ0NNlSpVnI49ODjYREREnHN7F1q7d+9e4+rqap599lmn5T/++KOpUKGC0/LCn9v7779vLcvNzTX+/v6mX79+1rKXX37ZSDKLFy+2lh0/ftw0atTISDKrV6+2lkdERDiNd6HCn3vjxo1Nbm6utXzWrFlGkvnxxx+NMcZs2bLFSDILFy48/2CcJi8vz/j6+ppmzZqZ48ePW8uXLl1qJJlJkyZZyy7k78yZtXv27LGWBQcHG0lm7dq11rKMjAzj7u5uxo0bV+S1bdu2NXl5edbyGTNmGEnmiy++sJZJMpMnTy6y/zP/DixcuLDImF+oM48lIyPDuLm5mYiICFNQUGDVPfHEE0aS035btmx5wecogGsbH1kEgFKoUqXKOWdbLLxi8sUXX5R6Agx3d3cNHTr0guvvv/9+Va1a1Xp+9913KyAgQF999VWp9n+hvvrqK7m6uuqRRx5xWj5u3DgZY7R8+XKn5WFhYU5XUFq0aCEvLy/9+uuv592Pv7+/Bg0aZC2rWLGiHnnkEeXk5GjNmjVlcDRFff755yooKFD//v31+++/Ww9/f381aNCgyFWtKlWq6N5777Weu7m5qUOHDk7HFxsbq+uvv1533HGHtczDw+OsV1zPZejQoU73FRZe0SzcX+EVsBUrVujPP/+84O1u2rRJGRkZ+uc//ykPDw9reUREhBo1aqRly5aVuNdzadKkidW79NdVzYYNGxZ7XowYMcLpXsaHHnpIFSpUuOTn+vmsXLlSeXl5evjhh52u1BU3IYuPj4+2b9+unTt3XsYOAVyJCGQAUAo5OTlO4edMAwYM0E033aThw4fLz89PAwcO1KefflqicHb99deXaAKPBg0aOD13OByqX7/+JZ/Oe9++fQoMDCwyHoUf+9q3b5/T8tq1axfZxnXXXVfkHqDi9tOgQQO5uDj/03W2/ZSVnTt3yhijBg0aqGbNmk6Pn3/+2ZrQolCtWrWKfGzuzOPbt2+f6tWrV6Sufv36Je7vzPG87rrrJMnaX926dRUdHa23335bNWrUUHh4uN54443z3j9WOJ4NGzYssq5Ro0ZlPt4lOS/OPNerVKmigIAA26euLxyTM/urWbOm9XMpNG3aNGVmZurGG29U8+bNNX78eG3duvWy9QrgykEgA4ASOnDggLKyss755tnT01Nr167VypUrdd9992nr1q0aMGCAbrvtNuXn51/Qfkpy39eFOtv9NRfaU1k426x05owJQK4UBQUFcjgcio2NVVxcXJHHW2+95VR/uY/vQvb38ssva+vWrXriiSd0/PhxPfLII2ratKkOHDhwSXoqjcs1bpfzXD+Xzp07a/fu3Xr33XfVrFkzvf3222rTpo3efvttu1sDcJkRyACghD744ANJUnh4+DnrXFxc1K1bN73yyiv66aef9Oyzz2rVqlXWR9xKMvnAhTjzo0/GGO3atctpVr7rrrtOmZmZRV575tWOkvQWHBysgwcPFvkI544dO6z1ZSE4OFg7d+4scpWxrPdzpnr16skYo7p16yosLKzIo2PHjiXeZnBwsHbv3l0kbJw5O6JUdudJ8+bN9dRTT2nt2rX63//+p99++01z5849Z4+SlJycXGRdcnLyJRvvC3HmuZ6Tk6PU1NTznut5eXlKTU11WlaWfw8Lx+TM/g4dOlTslb5q1app6NCh+vjjj7V//361aNGi2JkhAVzbCGQAUAKrVq3S008/rbp162rw4MFnrTty5EiRZYVfsJybmytJqly5siQVG5BK4/3333cKRZ999plSU1Otmf2kv8LF+vXrnWZ8W7p0aZHp20vSW69evZSfn6/Zs2c7LX/11VflcDic9n8xevXqpbS0NC1YsMBadurUKb3++uuqUqWKbrnlljLZz5n69u0rV1dXTZ06tUiAMsbo8OHDJd5meHi4fvvtN3355ZfWshMnTug///lPkdrKlStf0PT0Z5Odna1Tp045LWvevLlcXFysc7E47dq1k6+vr+bOnetUt3z5cv3888+KiIgodU8X69///rdOnjxpPZ8zZ45OnTpV5Fxfu3ZtkdedeYWsLP8ehoWFqWLFinr99dedzpWZM2cWqT3zvKlSpYrq169/zp8JgGsT094DwFksX75cO3bs0KlTp5Senq5Vq1YpLi5OwcHB+vLLL50mOjjTtGnTtHbtWkVERCg4OFgZGRl68803VatWLXXq1EnSX28YfXx8NHfuXFWtWlWVK1dWSEiI6tatW6p+q1Wrpk6dOmno0KFKT0/XzJkzVb9+faeJIoYPH67PPvtMPXr0UP/+/bV79259+OGHRaYpL0lvvXv3VteuXfXkk09q7969atmypb7++mt98cUXGjNmTKmmQC/OiBEj9NZbb2nIkCHavHmz6tSpo88++0zr1q3TzJkzz3lP3/ns2rVLzzzzTJHlrVu3VkREhJ555hlNnDhRe/fuVZ8+fVS1alXt2bNHixYt0ogRI/Too4+WaH//+Mc/NHv2bA0aNEijR49WQECAPvroI+ucOv2qTdu2bbVgwQJFR0erffv2qlKlinr37n3B+1q1apVGjRqlv//977rxxht16tQpffDBB3J1dVW/fv3O+rqKFSvqhRde0NChQ3XLLbdo0KBB1rT3derU0dixY0t0zGUpLy9P3bp1U//+/ZWcnKw333xTnTp1cpokZfjw4Ro5cqT69eun2267TT/88INWrFihGjVqOG2rVatWcnV11QsvvKCsrCy5u7vr1ltvla+vb4n7qlmzph599FFNnz5dt99+u3r16qUtW7Zo+fLlRfbbpEkTdenSRW3btlW1atW0adMmffbZZxo1alTpBgXA1cueyR0B4MpVOJV14cPNzc34+/ub2267zcyaNctpevVCZ057Hx8fb+68804TGBho3NzcTGBgoBk0aJD55ZdfnF73xRdfmCZNmpgKFSo4TTN/yy23mKZNmxbb39mmvf/444/NxIkTja+vr/H09DQRERFm3759RV7/8ssvm+uvv964u7ubm266yWzatKnINs/V25nT3htjzNGjR83YsWNNYGCgqVixomnQoIF58cUXnab+NuavqcijoqKK9HS26fjPlJ6eboYOHWpq1Khh3NzcTPPmzYudmr+k096f/vM+/TFs2DCr7r///a/p1KmTqVy5sqlcubJp1KiRiYqKMsnJyVbN2X5uxY3Zr7/+aiIiIoynp6epWbOmGTdunPnvf/9rJJn169dbdTk5Oeaee+4xPj4+RpK1ncKf+5nT2e/Zs8fp5/Xrr7+aBx54wNSrV894eHiYatWqma5du5qVK1de0PgsWLDAtG7d2ri7u5tq1aqZwYMHmwMHDjjVlMW098X9vM48Lwtfu2bNGjNixAhz3XXXmSpVqpjBgwebw4cPO702Pz/fTJgwwdSoUcNUqlTJhIeHm127dhV7rv3nP/8xN9xwg3F1dS3RFPjFHUt+fr6ZOnWqCQgIMJ6enqZLly5m27ZtRfb7zDPPmA4dOhgfHx/j6elpGjVqZJ599lmn6fwBlA8OY67Qu6gBAChnZs6cqbFjx+rAgQO6/vrr7W7nilP4RdUbN25Uu3bt7G4HAMoE95ABAGCD48ePOz0/ceKE3nrrLTVo0IAwBgDlCPeQAQBgg759+6p27dpq1aqVsrKy9OGHH2rHjh366KOP7G6t3MvJyVFOTs45a2rWrHnWqfoBoCQIZAAA2CA8PFxvv/22PvroI+Xn56tJkyb65JNPNGDAALtbK/deeuklTZ069Zw1e/bscZpmHwBKi3vIAAAATvPrr7/q119/PWdNp06dzjnTKgBcKAIZAAAAANiEST0AAAAAwCbcQ1ZGCgoKdPDgQVWtWtXpCz0BAAAAlC/GGB09elSBgYFycTn3NTACWRk5ePCggoKC7G4DAAAAwBVi//79qlWr1jlrCGRlpGrVqpL+GnQvLy+buwEAAABgl+zsbAUFBVkZ4VwIZGWk8GOKXl5eBDIAAAAAF3QrE5N6AAAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADaxNZDNmTNHLVq0sCbCCA0N1fLly631J06cUFRUlKpXr64qVaqoX79+Sk9Pd9pGSkqKIiIiVKlSJfn6+mr8+PE6deqUU01CQoLatGkjd3d31a9fXzExMUV6eeONN1SnTh15eHgoJCRE33333SU5ZgAAAAAoZGsgq1Wrlp5//nlt3rxZmzZt0q233qo777xT27dvlySNHTtWS5Ys0cKFC7VmzRodPHhQffv2tV6fn5+viIgI5eXl6dtvv9V7772nmJgYTZo0yarZs2ePIiIi1LVrVyUlJWnMmDEaPny4VqxYYdUsWLBA0dHRmjx5sr7//nu1bNlS4eHhysjIuHyDAQAAAKDccRhjjN1NnK5atWp68cUXdffdd6tmzZqaP3++7r77bknSjh071LhxYyUmJqpjx45avny5br/9dh08eFB+fn6SpLlz52rChAk6dOiQ3NzcNGHCBC1btkzbtm2z9jFw4EBlZmYqNjZWkhQSEqL27dtr9uzZkqSCggIFBQXp4Ycf1uOPP15sn7m5ucrNzbWeF37XQFZWFtPeAwAAAOVYdna2vL29LygbXDH3kOXn5+uTTz7RsWPHFBoaqs2bN+vkyZMKCwuzaho1aqTatWsrMTFRkpSYmKjmzZtbYUySwsPDlZ2dbV1lS0xMdNpGYU3hNvLy8rR582anGhcXF4WFhVk1xZk+fbq8vb2tR1BQ0MUPAgAAAIByxfZA9uOPP6pKlSpyd3fXyJEjtWjRIjVp0kRpaWlyc3OTj4+PU72fn5/S0tIkSWlpaU5hrHB94bpz1WRnZ+v48eP6/ffflZ+fX2xN4TaKM3HiRGVlZVmP/fv3l+r4AQAAAJRfFexuoGHDhkpKSlJWVpY+++wzRUZGas2aNXa3dV7u7u5yd3e3uw0AAAAAVzHbA5mbm5vq168vSWrbtq02btyoWbNmacCAAcrLy1NmZqbTVbL09HT5+/tLkvz9/YvMhlg4C+PpNWfOzJieni4vLy95enrK1dVVrq6uxdYUbgMAAAAALgXbP7J4poKCAuXm5qpt27aqWLGi4uPjrXXJyclKSUlRaGioJCk0NFQ//vij02yIcXFx8vLyUpMmTaya07dRWFO4DTc3N7Vt29appqCgQPHx8VYNAAAAAFwKtl4hmzhxonr27KnatWvr6NGjmj9/vhISErRixQp5e3tr2LBhio6OVrVq1eTl5aWHH35YoaGh6tixoySpe/fuatKkie677z7NmDFDaWlpeuqppxQVFWV9nHDkyJGaPXu2HnvsMT3wwANatWqVPv30Uy1btszqIzo6WpGRkWrXrp06dOigmTNn6tixYxo6dKgt4wIAAACgfLA1kGVkZOj+++9XamqqvL291aJFC61YsUK33XabJOnVV1+Vi4uL+vXrp9zcXIWHh+vNN9+0Xu/q6qqlS5fqoYceUmhoqCpXrqzIyEhNmzbNqqlbt66WLVumsWPHatasWapVq5befvtthYeHWzUDBgzQoUOHNGnSJKWlpalVq1aKjY0tMtEHAAAAAJSlK+57yK5WJfmuAQAAAADXrqvye8gAAAAAoLwhkAEAAACATQhkAAAAAGATAhkAAAAA2MT2L4aG1Lu33R1cOZYssbsDAAAA4PLhChkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATWwNZNOnT1f79u1VtWpV+fr6qk+fPkpOTnaq6dKlixwOh9Nj5MiRTjUpKSmKiIhQpUqV5Ovrq/Hjx+vUqVNONQkJCWrTpo3c3d1Vv359xcTEFOnnjTfeUJ06deTh4aGQkBB99913ZX7MAAAAAFDI1kC2Zs0aRUVFaf369YqLi9PJkyfVvXt3HTt2zKnuwQcfVGpqqvWYMWOGtS4/P18RERHKy8vTt99+q/fee08xMTGaNGmSVbNnzx5FRESoa9euSkpK0pgxYzR8+HCtWLHCqlmwYIGio6M1efJkff/992rZsqXCw8OVkZFx6QcCAAAAQLnkMMYYu5sodOjQIfn6+mrNmjXq3LmzpL+ukLVq1UozZ84s9jXLly/X7bffroMHD8rPz0+SNHfuXE2YMEGHDh2Sm5ubJkyYoGXLlmnbtm3W6wYOHKjMzEzFxsZKkkJCQtS+fXvNnj1bklRQUKCgoCA9/PDDevzxx8/be3Z2try9vZWVlSUvL68SHXfv3iUqv6YtWWJ3BwAAAMDFKUk2uKLuIcvKypIkVatWzWn5Rx99pBo1aqhZs2aaOHGi/vzzT2tdYmKimjdvboUxSQoPD1d2dra2b99u1YSFhTltMzw8XImJiZKkvLw8bd682anGxcVFYWFhVs2ZcnNzlZ2d7fQAAAAAgJKoYHcDhQoKCjRmzBjddNNNatasmbX8nnvuUXBwsAIDA7V161ZNmDBBycnJ+vzzzyVJaWlpTmFMkvU8LS3tnDXZ2dk6fvy4/vjjD+Xn5xdbs2PHjmL7nT59uqZOnXpxBw0AAACgXLtiAllUVJS2bdumb775xmn5iBEjrD83b95cAQEB6tatm3bv3q169epd7jYtEydOVHR0tPU8OztbQUFBtvUDAAAA4OpzRQSyUaNGaenSpVq7dq1q1ap1ztqQkBBJ0q5du1SvXj35+/sXmQ0xPT1dkuTv72/9t3DZ6TVeXl7y9PSUq6urXF1di60p3MaZ3N3d5e7ufuEHCQAAAABnsPUeMmOMRo0apUWLFmnVqlWqW7fueV+TlJQkSQoICJAkhYaG6scff3SaDTEuLk5eXl5q0qSJVRMfH++0nbi4OIWGhkqS3Nzc1LZtW6eagoICxcfHWzUAAAAAUNZsvUIWFRWl+fPn64svvlDVqlWte768vb3l6emp3bt3a/78+erVq5eqV6+urVu3auzYsercubNatGghSerevbuaNGmi++67TzNmzFBaWpqeeuopRUVFWVewRo4cqdmzZ+uxxx7TAw88oFWrVunTTz/VsmXLrF6io6MVGRmpdu3aqUOHDpo5c6aOHTumoUOHXv6BAQAAAFAu2DrtvcPhKHb5vHnzNGTIEO3fv1/33nuvtm3bpmPHjikoKEh33XWXnnrqKafpI/ft26eHHnpICQkJqly5siIjI/X888+rQoX/nzcTEhI0duxY/fTTT6pVq5b+9a9/aciQIU77nT17tl588UWlpaWpVatWeu2116yPSJ4P096XDaa9BwAAwNWuJNngivoesqsZgaxsEMgAAABwtbtqv4cMAAAAAMoTAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE1sDWTTp09X+/btVbVqVfn6+qpPnz5KTk52qjlx4oSioqJUvXp1ValSRf369VN6erpTTUpKiiIiIlSpUiX5+vpq/PjxOnXqlFNNQkKC2rRpI3d3d9WvX18xMTFF+nnjjTdUp04deXh4KCQkRN99912ZHzMAAAAAFLI1kK1Zs0ZRUVFav3694uLidPLkSXXv3l3Hjh2zasaOHaslS5Zo4cKFWrNmjQ4ePKi+ffta6/Pz8xUREaG8vDx9++23eu+99xQTE6NJkyZZNXv27FFERIS6du2qpKQkjRkzRsOHD9eKFSusmgULFig6OlqTJ0/W999/r5YtWyo8PFwZGRmXZzAAAAAAlDsOY4yxu4lChw4dkq+vr9asWaPOnTsrKytLNWvW1Pz583X33XdLknbs2KHGjRsrMTFRHTt21PLly3X77bfr4MGD8vPzkyTNnTtXEyZM0KFDh+Tm5qYJEyZo2bJl2rZtm7WvgQMHKjMzU7GxsZKkkJAQtW/fXrNnz5YkFRQUKCgoSA8//LAef/zx8/aenZ0tb29vZWVlycvLq0TH3bt3icqvaUuW2N0BAAAAcHFKkg2uqHvIsrKyJEnVqlWTJG3evFknT55UWFiYVdOoUSPVrl1biYmJkqTExEQ1b97cCmOSFB4eruzsbG3fvt2qOX0bhTWF28jLy9PmzZudalxcXBQWFmbVnCk3N1fZ2dlODwAAAAAoiSsmkBUUFGjMmDG66aab1KxZM0lSWlqa3Nzc5OPj41Tr5+entLQ0q+b0MFa4vnDduWqys7N1/Phx/f7778rPzy+2pnAbZ5o+fbq8vb2tR1BQUOkOHAAAAEC5dcUEsqioKG3btk2ffPKJ3a1ckIkTJyorK8t67N+/3+6WAAAAAFxlKtjdgCSNGjVKS5cu1dq1a1WrVi1rub+/v/Ly8pSZmel0lSw9PV3+/v5WzZmzIRbOwnh6zZkzM6anp8vLy0uenp5ydXWVq6trsTWF2ziTu7u73N3dS3fAAAAAACCbr5AZYzRq1CgtWrRIq1atUt26dZ3Wt23bVhUrVlR8fLy1LDk5WSkpKQoNDZUkhYaG6scff3SaDTEuLk5eXl5q0qSJVXP6NgprCrfh5uamtm3bOtUUFBQoPj7eqgEAAACAsmbrFbKoqCjNnz9fX3zxhapWrWrdr+Xt7S1PT095e3tr2LBhio6OVrVq1eTl5aWHH35YoaGh6tixoySpe/fuatKkie677z7NmDFDaWlpeuqppxQVFWVdwRo5cqRmz56txx57TA888IBWrVqlTz/9VMuWLbN6iY6OVmRkpNq1a6cOHTpo5syZOnbsmIYOHXr5BwYAAABAuWBrIJszZ44kqUuXLk7L582bpyFDhkiSXn31Vbm4uKhfv37Kzc1VeHi43nzzTavW1dVVS5cu1UMPPaTQ0FBVrlxZkZGRmjZtmlVTt25dLVu2TGPHjtWsWbNUq1Ytvf322woPD7dqBgwYoEOHDmnSpElKS0tTq1atFBsbW2SiDwAAAAAoK1fU95BdzfgesrLB95ABAADganfVfg8ZAAAAAJQnBDIAAAAAsAmBDAAAAABsQiADAAAAAJsQyAAAAADAJgQyAAAAALAJgQwAAAAAbEIgAwAAAACbEMgAAAAAwCYEMgAAAACwCYEMAAAAAGxCIAMAAAAAmxDIAAAAAMAmBDIAAAAAsAmBDAAAAABsQiADAAAAAJsQyAAAAADAJgQyAAAAALAJgQwAAAAAbEIgAwAAAACbEMgAAAAAwCYEMgAAAACwCYEMAAAAAGxCIAMAAAAAmxDIAAAAAMAmBDIAAAAAsAmBDAAAAABsQiADAAAAAJsQyAAAAADAJgQyAAAAALBJqQLZr7/+WtZ9AAAAAEC5U6pAVr9+fXXt2lUffvihTpw4UdY9AQAAAEC5UKpA9v3336tFixaKjo6Wv7+//vGPf+i7774r694AAAAA4JpWqkDWqlUrzZo1SwcPHtS7776r1NRUderUSc2aNdMrr7yiQ4cOlXWfAAAAAHDNuahJPSpUqKC+fftq4cKFeuGFF7Rr1y49+uijCgoK0v3336/U1NSy6hMAAAAArjkXFcg2bdqkf/7znwoICNArr7yiRx99VLt371ZcXJwOHjyoO++8s6z6BAAAAIBrToXSvOiVV17RvHnzlJycrF69eun9999Xr1695OLyV76rW7euYmJiVKdOnbLsFQAAAACuKaUKZHPmzNEDDzygIUOGKCAgoNgaX19fvfPOOxfVHAAAAABcy0oVyHbu3HneGjc3N0VGRpZm8wAAAABQLpTqHrJ58+Zp4cKFRZYvXLhQ77333kU3BQAAAADlQakC2fTp01WjRo0iy319ffXcc89ddFMAAAAAUB6UKpClpKSobt26RZYHBwcrJSXlopsCAAAAgPKgVIHM19dXW7duLbL8hx9+UPXq1S+6KQAAAAAoD0oVyAYNGqRHHnlEq1evVn5+vvLz87Vq1SqNHj1aAwcOLOseAQAAAOCaVKpZFp9++mnt3btX3bp1U4UKf22ioKBA999/P/eQAQAAAMAFKlUgc3Nz04IFC/T000/rhx9+kKenp5o3b67g4OCy7g8AAAAArlmlCmSFbrzxRt14441l1QsAAAAAlCulCmT5+fmKiYlRfHy8MjIyVFBQ4LR+1apVZdIcAAAAAFzLShXIRo8erZiYGEVERKhZs2ZyOBxl3RcAAAAAXPNKFcg++eQTffrpp+rVq1dZ9wMAAAAA5Uappr13c3NT/fr1y7oXAAAAAChXShXIxo0bp1mzZskYU9b9AAAAAEC5UaqPLH7zzTdavXq1li9frqZNm6pixYpO6z///PMyaQ4AAAAArmWlCmQ+Pj666667yroXAAAAAChXShXI5s2bV9Z9AAAAAEC5U6p7yCTp1KlTWrlypd566y0dPXpUknTw4EHl5OSUWXMAAAAAcC0r1RWyffv2qUePHkpJSVFubq5uu+02Va1aVS+88IJyc3M1d+7csu4TAAAAAK45pbpCNnr0aLVr105//PGHPD09reV33XWX4uPjy6w5AAAAALiWleoK2f/+9z99++23cnNzc1pep04d/fbbb2XSGAAAAABc60p1haygoED5+flFlh84cEBVq1a96KYAAAAAoDwoVSDr3r27Zs6caT13OBzKycnR5MmT1atXr7LqDQAAAACuaaX6yOLLL7+s8PBwNWnSRCdOnNA999yjnTt3qkaNGvr444/LukcAAAAAuCaVKpDVqlVLP/zwgz755BNt3bpVOTk5GjZsmAYPHuw0yQcAAAAA4OxKFcgkqUKFCrr33nvLshcAAAAAKFdKdQ/Z+++/f87HhVq7dq169+6twMBAORwOLV682Gn9kCFD5HA4nB49evRwqjly5IgGDx4sLy8v+fj4aNiwYUW+nHrr1q26+eab5eHhoaCgIM2YMaNILwsXLlSjRo3k4eGh5s2b66uvvrrwAQEAAACAUijVFbLRo0c7PT958qT+/PNPubm5qVKlSrr//vsvaDvHjh1Ty5Yt9cADD6hv377F1vTo0UPz5s2znru7uzutHzx4sFJTUxUXF6eTJ09q6NChGjFihObPny9Jys7OVvfu3RUWFqa5c+fqxx9/1AMPPCAfHx+NGDFCkvTtt99q0KBBmj59um6//XbNnz9fffr00ffff69mzZpd8LgAAAAAQEk4jDGmLDa0c+dOPfTQQxo/frzCw8NL3ojDoUWLFqlPnz7WsiFDhigzM7PIlbNCP//8s5o0aaKNGzeqXbt2kqTY2Fj16tVLBw4cUGBgoObMmaMnn3xSaWlp1vemPf7441q8eLF27NghSRowYICOHTumpUuXWtvu2LGjWrVqpblz515Q/9nZ2fL29lZWVpa8vLxKdOy9e5eo/Jq2ZIndHQAAAAAXpyTZoFQfWSxOgwYN9Pzzzxe5enaxEhIS5Ovrq4YNG+qhhx7S4cOHrXWJiYny8fGxwpgkhYWFycXFRRs2bLBqOnfu7PQl1uHh4UpOTtYff/xh1YSFhTntNzw8XImJiWftKzc3V9nZ2U4PAAAAACiJMgtk0l8TfRw8eLDMttejRw+9//77io+P1wsvvKA1a9aoZ8+e1pdSp6WlydfXt0gP1apVU1pamlXj5+fnVFP4/Hw1heuLM336dHl7e1uPoKCgiztYAAAAAOVOqe4h+/LLL52eG2OUmpqq2bNn66abbiqTxiRp4MCB1p+bN2+uFi1aqF69ekpISFC3bt3KbD+lMXHiREVHR1vPs7OzCWUAAAAASqRUgez0+7ykv+7/qlmzpm699Va9/PLLZdFXsW644QbVqFFDu3btUrdu3eTv76+MjAynmlOnTunIkSPy9/eXJPn7+ys9Pd2ppvD5+WoK1xfH3d29yAQjAAAAAFASpfrIYkFBgdMjPz9faWlpmj9/vgICAsq6R8uBAwd0+PBhax+hoaHKzMzU5s2brZpVq1apoKBAISEhVs3atWt18uRJqyYuLk4NGzbUddddZ9XEx8c77SsuLk6hoaGX7FgAAAAAoEzvISupnJwcJSUlKSkpSZK0Z88eJSUlKSUlRTk5ORo/frzWr1+vvXv3Kj4+Xnfeeafq169vzeLYuHFj9ejRQw8++KC+++47rVu3TqNGjdLAgQMVGBgoSbrnnnvk5uamYcOGafv27VqwYIFmzZrl9HHD0aNHKzY2Vi+//LJ27NihKVOmaNOmTRo1atRlHxMAAAAA5Ueppr0/PcyczyuvvHLWdQkJCeratWuR5ZGRkZozZ4769OmjLVu2KDMzU4GBgerevbuefvpppwk4jhw5olGjRmnJkiVycXFRv3799Nprr6lKlSpWzdatWxUVFaWNGzeqRo0aevjhhzVhwgSnfS5cuFBPPfWU9u7dqwYNGmjGjBnq1avXBR8n096XDaa9BwAAwNWuJNmgVIGsa9eu2rJli06ePKmGDRtKkn755Re5urqqTZs2/3/jDodWrVpV0s1flQhkZYNABgAAgKtdSbJBqSb16N27t6pWrar33nvPug/rjz/+0NChQ3XzzTdr3LhxpdksAAAAAJQrpbpCdv311+vrr79W06ZNnZZv27ZN3bt3L9PvIrtacIWsbHCFDAAAAFe7kmSDUk3qkZ2drUOHDhVZfujQIR09erQ0mwQAAACAcqdUgeyuu+7S0KFD9fnnn+vAgQM6cOCA/vvf/2rYsGHq27dvWfcIAAAAANekUt1DNnfuXD366KO65557rO/3qlChgoYNG6YXX3yxTBsEAAAAgGtVqe4hK3Ts2DHt3r1bklSvXj1Vrly5zBq72nAPWdngHjIAAABc7S75PWSFUlNTlZqaqgYNGqhy5cq6iGwHAAAAAOVOqQLZ4cOH1a1bN914443q1auXUlNTJUnDhg1jynsAAAAAuEClCmRjx45VxYoVlZKSokqVKlnLBwwYoNjY2DJrDgAAAACuZaWa1OPrr7/WihUrVKtWLaflDRo00L59+8qkMQAAAAC41pXqCtmxY8ecrowVOnLkiNzd3S+6KQAAAAAoD0oVyG6++Wa9//771nOHw6GCggLNmDFDXbt2LbPmAAAAAOBaVqqPLM6YMUPdunXTpk2blJeXp8cee0zbt2/XkSNHtG7durLuEQAAAACuSaW6QtasWTP98ssv6tSpk+68804dO3ZMffv21ZYtW1SvXr2y7hEAAAAArkklvkJ28uRJ9ejRQ3PnztWTTz55KXoCAAAAgHKhxFfIKlasqK1bt16KXgAAAACgXCnVRxbvvfdevfPOO2XdCwAAAACUK6Wa1OPUqVN69913tXLlSrVt21aVK1d2Wv/KK6+USXMAAAAAcC0rUSD79ddfVadOHW3btk1t2rSRJP3yyy9ONQ6Ho+y6AwAAAIBrWIkCWYMGDZSamqrVq1dLkgYMGKDXXntNfn5+l6Q5AAAAALiWlegeMmOM0/Ply5fr2LFjZdoQAAAAAJQXpZrUo9CZAQ0AAAAAcOFKFMgcDkeRe8S4ZwwAAAAASqdE95AZYzRkyBC5u7tLkk6cOKGRI0cWmWXx888/L7sOAQAAAOAaVaJAFhkZ6fT83nvvLdNmAAAAAKA8KVEgmzdv3qXqAwAAAADKnYua1AMAAAAAUHoEMgAAAACwCYEMAAAAAGxCIAMAAAAAmxDIAAAAAMAmBDIAAAAAsAmBDAAAAABsQiADAAAAAJsQyAAAAADAJgQyAAAAALAJgQwAAAAAbEIgAwAAAACbEMgAAAAAwCYEMgAAAACwCYEMAAAAAGxCIAMAAAAAmxDIAAAAAMAmBDIAAAAAsAmBDAAAAABsQiADAAAAAJsQyAAAAADAJgQyAAAAALAJgQwAAAAAbEIgAwAAAACbEMgAAAAAwCYEMgAAAACwCYEMAAAAAGxCIAMAAAAAmxDIAAAAAMAmBDIAAAAAsAmBDAAAAABsQiADAAAAAJsQyAAAAADAJgQyAAAAALAJgQwAAAAAbEIgAwAAAACbEMgAAAAAwCYEMgAAAACwia2BbO3aterdu7cCAwPlcDi0ePFip/XGGE2aNEkBAQHy9PRUWFiYdu7c6VRz5MgRDR48WF5eXvLx8dGwYcOUk5PjVLN161bdfPPN8vDwUFBQkGbMmFGkl4ULF6pRo0by8PBQ8+bN9dVXX5X58QIAAADA6WwNZMeOHVPLli31xhtvFLt+xowZeu211zR37lxt2LBBlStXVnh4uE6cOGHVDB48WNu3b1dcXJyWLl2qtWvXasSIEdb67Oxsde/eXcHBwdq8ebNefPFFTZkyRf/+97+tmm+//VaDBg3SsGHDtGXLFvXp00d9+vTRtm3bLt3BAwAAACj3HMYYY3cTkuRwOLRo0SL16dNH0l9XxwIDAzVu3Dg9+uijkqSsrCz5+fkpJiZGAwcO1M8//6wmTZpo48aNateunSQpNjZWvXr10oEDBxQYGKg5c+boySefVFpamtzc3CRJjz/+uBYvXqwdO3ZIkgYMGKBjx45p6dKlVj8dO3ZUq1atNHfu3AvqPzs7W97e3srKypKXl1eJjr137xKVX9OWLLG7AwAAAODilCQbXLH3kO3Zs0dpaWkKCwuzlnl7eyskJESJiYmSpMTERPn4+FhhTJLCwsLk4uKiDRs2WDWdO3e2wpgkhYeHKzk5WX/88YdVc/p+CmsK91Oc3NxcZWdnOz0AAAAAoCSu2ECWlpYmSfLz83Na7ufnZ61LS0uTr6+v0/oKFSqoWrVqTjXFbeP0fZytpnB9caZPny5vb2/rERQUVNJDBAAAAFDOXbGB7Eo3ceJEZWVlWY/9+/fb3RIAAACAq8wVG8j8/f0lSenp6U7L09PTrXX+/v7KyMhwWn/q1CkdOXLEqaa4bZy+j7PVFK4vjru7u7y8vJweAAAAAFASV2wgq1u3rvz9/RUfH28ty87O1oYNGxQaGipJCg0NVWZmpjZv3mzVrFq1SgUFBQoJCbFq1q5dq5MnT1o1cXFxatiwoa677jqr5vT9FNYU7gcAAAAALgVbA1lOTo6SkpKUlJQk6a+JPJKSkpSSkiKHw6ExY8bomWee0Zdffqkff/xR999/vwIDA62ZGBs3bqwePXrowQcf1Hfffad169Zp1KhRGjhwoAIDAyVJ99xzj9zc3DRs2DBt375dCxYs0KxZsxQdHW31MXr0aMXGxurll1/Wjh07NGXKFG3atEmjRo263EMCAAAAoBypYOfON23apK5du1rPC0NSZGSkYmJi9Nhjj+nYsWMaMWKEMjMz1alTJ8XGxsrDw8N6zUcffaRRo0apW7ducnFxUb9+/fTaa69Z6729vfX1118rKipKbdu2VY0aNTRp0iSn7yr729/+pvnz5+upp57SE088oQYNGmjx4sVq1qzZZRgFAAAAAOXVFfM9ZFc7voesbPA9ZAAAALjaXRPfQwYAAAAA1zoCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2qWB3A8Dpeve2u4Mrx5IldncAAACAS40rZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGCTKzqQTZkyRQ6Hw+nRqFEja/2JEycUFRWl6tWrq0qVKurXr5/S09OdtpGSkqKIiAhVqlRJvr6+Gj9+vE6dOuVUk5CQoDZt2sjd3V3169dXTEzM5Tg8AAAAAOXcFR3IJKlp06ZKTU21Ht988421buzYsVqyZIkWLlyoNWvW6ODBg+rbt6+1Pj8/XxEREcrLy9O3336r9957TzExMZo0aZJVs2fPHkVERKhr165KSkrSmDFjNHz4cK1YseKyHicAAACA8qeC3Q2cT4UKFeTv719keVZWlt555x3Nnz9ft956qyRp3rx5aty4sdavX6+OHTvq66+/1k8//aSVK1fKz89PrVq10tNPP60JEyZoypQpcnNz09y5c1W3bl29/PLLkqTGjRvrm2++0auvvqrw8PCz9pWbm6vc3FzreXZ2dhkfOQAAAIBr3RV/hWznzp0KDAzUDTfcoMGDByslJUWStHnzZp08eVJhYWFWbaNGjVS7dm0lJiZKkhITE9W8eXP5+flZNeHh4crOztb27dutmtO3UVhTuI2zmT59ury9va1HUFBQmRwvAAAAgPLjig5kISEhiomJUWxsrObMmaM9e/bo5ptv1tGjR5WWliY3Nzf5+Pg4vcbPz09paWmSpLS0NKcwVri+cN25arKzs3X8+PGz9jZx4kRlZWVZj/3791/s4QIAAAAoZ67ojyz27NnT+nOLFi0UEhKi4OBgffrpp/L09LSxM8nd3V3u7u629gAAAADg6nZFXyE7k4+Pj2688Ubt2rVL/v7+ysvLU2ZmplNNenq6dc+Zv79/kVkXC5+fr8bLy8v20AcAAADg2nZVBbKcnBzt3r1bAQEBatu2rSpWrKj4+HhrfXJyslJSUhQaGipJCg0N1Y8//qiMjAyrJi4uTl5eXmrSpIlVc/o2CmsKtwEAAAAAl8oVHcgeffRRrVmzRnv37tW3336ru+66S66urho0aJC8vb01bNgwRUdHa/Xq1dq8ebOGDh2q0NBQdezYUZLUvXt3NWnSRPfdd59++OEHrVixQk899ZSioqKsjxuOHDlSv/76qx577DHt2LFDb775pj799FONHTvWzkMHAAAAUA5c0feQHThwQIMGDdLhw4dVs2ZNderUSevXr1fNmjUlSa+++qpcXFzUr18/5ebmKjw8XG+++ab1eldXVy1dulQPPfSQQkNDVblyZUVGRmratGlWTd26dbVs2TKNHTtWs2bNUq1atfT222+fc8p7AAAAACgLDmOMsbuJa0F2dra8vb2VlZUlLy+vEr22d+9L1BSuakuW2N0BAAAASqMk2eCK/sgiAAAAAFzLCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYJMKdjcAoHi9e9vdwZVjyRK7OwAAALg0uEIGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAAAAANqlgdwMAcD69e9vdwZVlyRK7OwAAAGWFK2QAAAAAYBMCGQAAAADYhEAGAAAAADbhHrIzvPHGG3rxxReVlpamli1b6vXXX1eHDh3sbgsALNxT9/9xPx0A4GrHFbLTLFiwQNHR0Zo8ebK+//57tWzZUuHh4crIyLC7NQAAAADXIIcxxtjdxJUiJCRE7du31+zZsyVJBQUFCgoK0sMPP6zHH3/8nK/Nzs6Wt7e3srKy5OXlVaL98n+7AQAoO1w5BWC3kmQDPrL4f/Ly8rR582ZNnDjRWubi4qKwsDAlJiYWqc/NzVVubq71PCsrS9Jfg19SJ0+WomEAAFCsHj3s7gBAeXfq1F+Z4EKufRHI/s/vv/+u/Px8+fn5OS338/PTjh07itRPnz5dU6dOLbI8KCjokvUIAAAA4Opx9OhReXt7n7OGQFZKEydOVHR0tPU8MzNTwcHBSklJOe+go2xlZ2crKChI+/fvL/HHRVF6jLt9GHv7MPb2YNztw9jbg3G3T1mNvTFGR48eVWBg4HlrCWT/p0aNGnJ1dVV6errT8vT0dPn7+xepd3d3l7u7e5Hl3t7e/MWxiZeXF2NvA8bdPoy9fRh7ezDu9mHs7cG426csxv5CL9Iwy+L/cXNzU9u2bRUfH28tKygoUHx8vEJDQ23sDAAAAMC1iitkp4mOjlZkZKTatWunDh06aObMmTp27JiGDh1qd2sAAAAArkEEstMMGDBAhw4d0qRJk5SWlqZWrVopNja2yEQfxXF3d9fkyZOL/RgjLi3G3h6Mu30Ye/sw9vZg3O3D2NuDcbePHWPP95ABAAAAgE24hwwAAAAAbEIgAwAAAACbEMgAAAAAwCYEMgAAAACwCYGsjLzxxhuqU6eOPDw8FBISou+++87ulq5q06dPV/v27VW1alX5+vqqT58+Sk5Odqrp0qWLHA6H02PkyJFONSkpKYqIiFClSpXk6+ur8ePH69SpU5fzUK4qU6ZMKTKmjRo1stafOHFCUVFRql69uqpUqaJ+/foV+TJ1xrx06tSpU2TsHQ6HoqKiJHG+l6W1a9eqd+/eCgwMlMPh0OLFi53WG2M0adIkBQQEyNPTU2FhYdq5c6dTzZEjRzR48GB5eXnJx8dHw4YNU05OjlPN1q1bdfPNN8vDw0NBQUGaMWPGpT60K9q5xv3kyZOaMGGCmjdvrsqVKyswMFD333+/Dh486LSN4v6ePP/88041jHtR5zvnhwwZUmRce/To4VTDOV9y5xv34n7nOxwOvfjii1YN53zJXch7yLJ6P5OQkKA2bdrI3d1d9evXV0xMTOmaNrhon3zyiXFzczPvvvuu2b59u3nwwQeNj4+PSU9Pt7u1q1Z4eLiZN2+e2bZtm0lKSjK9evUytWvXNjk5OVbNLbfcYh588EGTmppqPbKysqz1p06dMs2aNTNhYWFmy5Yt5quvvjI1atQwEydOtOOQrgqTJ082TZs2dRrTQ4cOWetHjhxpgoKCTHx8vNm0aZPp2LGj+dvf/matZ8xLLyMjw2nc4+LijCSzevVqYwzne1n66quvzJNPPmk+//xzI8ksWrTIaf3zzz9vvL29zeLFi80PP/xg7rjjDlO3bl1z/Phxq6ZHjx6mZcuWZv369eZ///ufqV+/vhk0aJC1Pisry/j5+ZnBgwebbdu2mY8//th4enqat95663Id5hXnXOOemZlpwsLCzIIFC8yOHTtMYmKi6dChg2nbtq3TNoKDg820adOc/h6c/u8C4168853zkZGRpkePHk7jeuTIEacazvmSO9+4nz7eqamp5t133zUOh8Ps3r3bquGcL7kLeQ9ZFu9nfv31V1OpUiUTHR1tfvrpJ/P6668bV1dXExsbW+KeCWRloEOHDiYqKsp6np+fbwIDA8306dNt7OrakpGRYSSZNWvWWMtuueUWM3r06LO+5quvvjIuLi4mLS3NWjZnzhzj5eVlcnNzL2W7V63Jkyebli1bFrsuMzPTVKxY0SxcuNBa9vPPPxtJJjEx0RjDmJel0aNHm3r16pmCggJjDOf7pXLmm6SCggLj7+9vXnzxRWtZZmamcXd3Nx9//LExxpiffvrJSDIbN260apYvX24cDof57bffjDHGvPnmm+a6665zGvsJEyaYhg0bXuIjujoU9+b0TN99952RZPbt22ctCw4ONq+++upZX8O4n9/ZAtmdd9551tdwzl+8Cznn77zzTnPrrbc6LeOcv3hnvocsq/czjz32mGnatKnTvgYMGGDCw8NL3CMfWbxIeXl52rx5s8LCwqxlLi4uCgsLU2Jioo2dXVuysrIkSdWqVXNa/tFHH6lGjRpq1qyZJk6cqD///NNal5iYqObNmzt9sXd4eLiys7O1ffv2y9P4VWjnzp0KDAzUDTfcoMGDByslJUWStHnzZp08edLpXG/UqJFq165tneuMednIy8vThx9+qAceeEAOh8Nazvl+6e3Zs0dpaWlO57m3t7dCQkKcznMfHx+1a9fOqgkLC5OLi4s2bNhg1XTu3Flubm5WTXh4uJKTk/XHH39cpqO5umVlZcnhcMjHx8dp+fPPP6/q1aurdevWevHFF50+QsS4l15CQoJ8fX3VsGFDPfTQQzp8+LC1jnP+0ktPT9eyZcs0bNiwIus45y/Ome8hy+r9TGJiotM2CmtK8/6/QolfASe///678vPznX5gkuTn56cdO3bY1NW1paCgQGPGjNFNN92kZs2aWcvvueceBQcHKzAwUFu3btWECROUnJyszz//XJKUlpZW7M+lcB2KCgkJUUxMjBo2bKjU1FRNnTpVN998s7Zt26a0tDS5ubkVeXPk5+dnjSdjXjYWL16szMxMDRkyxFrG+X55FI5VcWN5+nnu6+vrtL5ChQqqVq2aU03dunWLbKNw3XXXXXdJ+r9WnDhxQhMmTNCgQYPk5eVlLX/kkUfUpk0bVatWTd9++60mTpyo1NRUvfLKK5IY99Lq0aOH+vbtq7p162r37t164okn1LNnTyUmJsrV1ZVz/jJ47733VLVqVfXt29dpOef8xSnuPWRZvZ85W012draOHz8uT0/PC+6TQIYrXlRUlLZt26ZvvvnGafmIESOsPzdv3lwBAQHq1q2bdu/erXr16l3uNq8JPXv2tP7cokULhYSEKDg4WJ9++mmJfrHg4rzzzjvq2bOnAgMDrWWc7ygvTp48qf79+8sYozlz5jiti46Otv7cokULubm56R//+IemT58ud3f3y93qNWPgwIHWn5s3b64WLVqoXr16SkhIULdu3WzsrPx49913NXjwYHl4eDgt55y/OGd7D3ml4SOLF6lGjRpydXUtMjNLenq6/P39berq2jFq1CgtXbpUq1evVq1atc5ZGxISIknatWuXJMnf37/Yn0vhOpyfj4+PbrzxRu3atUv+/v7Ky8tTZmamU83p5zpjfvH27dunlStXavjw4ees43y/NArH6ly/0/39/ZWRkeG0/tSpUzpy5Ah/Fy5SYRjbt2+f4uLinK6OFSckJESnTp3S3r17JTHuZeWGG25QjRo1nH6/cM5fOv/73/+UnJx83t/7Eud8SZztPWRZvZ85W42Xl1eJ/yc2gewiubm5qW3btoqPj7eWFRQUKD4+XqGhoTZ2dnUzxmjUqFFatGiRVq1aVeRyfHGSkpIkSQEBAZKk0NBQ/fjjj07/iBT+A9+kSZNL0ve1JicnR7t371ZAQIDatm2rihUrOp3rycnJSklJsc51xvzizZs3T76+voqIiDhnHef7pVG3bl35+/s7nefZ2dnasGGD03memZmpzZs3WzWrVq1SQUGBFZRDQ0O1du1anTx50qqJi4tTw4YNy/1HiM6mMIzt3LlTK1euVPXq1c/7mqSkJLm4uFgfp2Pcy8aBAwd0+PBhp98vnPOXzjvvvKO2bduqZcuW563lnD+/872HLKv3M6GhoU7bKKwp1fv/Ek8DgiI++eQT4+7ubmJiYsxPP/1kRowYYXx8fJxmZkHJPPTQQ8bb29skJCQ4TfX6559/GmOM2bVrl5k2bZrZtGmT2bNnj/niiy/MDTfcYDp37mxto3DK0u7du5ukpCQTGxtratasyTTg5zBu3DiTkJBg9uzZY9atW2fCwsJMjRo1TEZGhjHmr2lia9eubVatWmU2bdpkQkNDTWhoqPV6xvzi5Ofnm9q1a5sJEyY4Led8L1tHjx41W7ZsMVu2bDGSzCuvvGK2bNlizeb3/PPPGx8fH/PFF1+YrVu3mjvvvLPYae9bt25tNmzYYL755hvToEEDpynAMzMzjZ+fn7nvvvvMtm3bzCeffGIqVapUrqeiPte45+XlmTvuuMPUqlXLJCUlOf3eL5zR7NtvvzWvvvqqSUpKMrt37zYffvihqVmzprn//vutfTDuxTvX2B89etQ8+uijJjEx0ezZs8esXLnStGnTxjRo0MCcOHHC2gbnfMmd73eNMX9NW1+pUiUzZ86cIq/nnC+d872HNKZs3s8UTns/fvx48/PPP5s33niDae/t9vrrr5vatWsbNzc306FDB7N+/Xq7W7qqSSr2MW/ePGOMMSkpKaZz586mWrVqxt3d3dSvX9+MHz/e6XuZjDFm7969pmfPnsbT09PUqFHDjBs3zpw8edKGI7o6DBgwwAQEBBg3Nzdz/fXXmwEDBphdu3ZZ648fP27++c9/muuuu85UqlTJ3HXXXSY1NdVpG4x56a1YscJIMsnJyU7LOd/L1urVq4v9/RIZGWmM+Wvq+3/961/Gz8/PuLu7m27duhX5mRw+fNgMGjTIVKlSxXh5eZmhQ4eao0ePOtX88MMPplOnTsbd3d1cf/315vnnn79ch3hFOte479mz56y/9wu/i2/z5s0mJCTEeHt7Gw8PD9O4cWPz3HPPOYUGYxj34pxr7P/880/TvXt3U7NmTVOxYkUTHBxsHnzwwSL/U5lzvuTO97vGGGPeeust4+npaTIzM4u8nnO+dM73HtKYsns/s3r1atOqVSvj5uZmbrjhBqd9lITj/xoHAAAAAFxm3EMGAAAAADYhkAEAAACATQhkAAAAAGATAhkAAAAA2IRABgAAAAA2IZABAAAAgE0IZAAAAABgEwIZAAAAANiEQAYAKBeGDBmiPn36lPl209LSdNttt6ly5cry8fG5rPu+FOrUqaOZM2ees8bhcGjx4sWXpR8AuNYRyAAAZeZKCB579+6Vw+FQUlLSZdnfq6++qtTUVCUlJemXX34ptmbWrFmKiYm5LP2cLiYm5qwh8Ww2btyoESNGXJqGAABFVLC7AQAArma7d+9W27Zt1aBBg7PWeHt7X8aOLk7NmjXtbgEAyhWukAEALptt27apZ8+eqlKlivz8/HTffffp999/t9Z36dJFjzzyiB577DFVq1ZN/v7+mjJlitM2duzYoU6dOsnDw0NNmjTRypUrnT5CV7duXUlS69at5XA41KVLF6fXv/TSSwoICFD16tUVFRWlkydPnrPnOXPmqF69enJzc1PDhg31wQcfWOvq1Kmj//73v3r//fflcDg0ZMiQYrdx5pXDCzlOh8OhOXPmqGfPnvL09NQNN9ygzz77zFqfkJAgh8OhzMxMa1lSUpIcDof27t2rhIQEDR06VFlZWXI4HHI4HEX2UZwzP7K4c+dOde7c2RrvuLg4p/q8vDyNGjVKAQEB8vDwUHBwsKZPn37e/QAA/kIgAwBcFpmZmbr11lvVunVrbdq0SbGxsUpPT1f//v2d6t577z1VrlxZGzZs0IwZMzRt2jQrBOTn56tPnz6qVKmSNmzYoH//+9968sknnV7/3XffSZJWrlyp1NRUff7559a61atXa/fu3Vq9erXee+89xcTEnPOjhIsWLdLo0aM1btw4bdu2Tf/4xz80dOhQrV69WtJfH+/r0aOH+vfvr9TUVM2aNeuCx+Ncx1noX//6l/r166cffvhBgwcP1sCBA/Xzzz9f0Pb/9re/aebMmfLy8lJqaqpSU1P16KOPXnB/klRQUKC+ffvKzc1NGzZs0Ny5czVhwgSnmtdee01ffvmlPv30UyUnJ+ujjz5SnTp1SrQfACjP+MgiAOCymD17tlq3bq3nnnvOWvbuu+8qKChIv/zyi2688UZJUosWLTR58mRJUoMGDTR79mzFx8frtttuU1xcnHbv3q2EhAT5+/tLkp599lnddttt1jYLP3JXvXp1q6bQddddp9mzZ8vV1VWNGjVSRESE4uPj9eCDDxbb80svvaQhQ4bon//8pyQpOjpa69ev10svvaSuXbuqZs2acnd3l6enZ5F9nc+5jrPQ3//+dw0fPlyS9PTTTysuLk6vv/663nzzzfNu383NTd7e3nI4HCXurdDKlSu1Y8cOrVixQoGBgZKk5557Tj179rRqUlJS1KBBA3Xq1EkOh0PBwcGl2hcAlFdcIQMAXBY//PCDVq9erSpVqliPRo0aSfrrPqxCLVq0cHpdQECAMjIyJEnJyckKCgpyChgdOnS44B6aNm0qV1fXYrddnJ9//lk33XST07Kbbrrpgq9Sncu5jrNQaGhokedlse8L9fPPPysoKMgKY8X1NGTIECUlJalhw4Z65JFH9PXXX1+2/gDgWsAVMgDAZZGTk6PevXvrhRdeKLIuICDA+nPFihWd1jkcDhUUFJRJD5dy25e7FxeXv/6fqjHGWna+++EuhTZt2mjPnj1avny5Vq5cqf79+yssLMzpfjcAwNlxhQwAcFm0adNG27dvV506dVS/fn2nR+XKlS9oGw0bNtT+/fuVnp5uLdu4caNTjZubm6S/7je7WI0bN9a6deuclq1bt05NmjS56G1fiPXr1xd53rhxY0n//6OZqamp1vozp/p3c3O7qHFo3Lix9u/f77SPM3uSJC8vLw0YMED/+c9/tGDBAv33v//VkSNHSr1fAChPuEIGAChTWVlZRYJB4YyG//nPfzRo0CBrdsFdu3bpk08+0dtvv+30UcKzue2221SvXj1FRkZqxowZOnr0qJ566ilJf11hkiRfX195enoqNjZWtWrVkoeHR6mnnR8/frz69++v1q1bKywsTEuWLNHnn3+ulStXlmp7JbVw4UK1a9dOnTp10kcffaTvvvtO77zzjiSpfv36CgoK0pQpU/Tss8/ql19+0csvv+z0+jp16ignJ0fx8fFq2bKlKlWqpEqVKl3w/sPCwnTjjTcqMjJSL774orKzs4tMovLKK68oICBArVu3louLixYuXCh/f/8Sf/8ZAJRXXCEDAJSphIQEtW7d2ukxdepUBQYGat26dcrPz1f37t3VvHlzjRkzRj4+PtbH787H1dVVixcvVk5Ojtq3b6/hw4dbAcHDw0OSVKFCBb322mt66623FBgYqDvvvLPUx9KnTx/NmjVLL730kpo2baq33npL8+bNKzKV/qUydepUffLJJ2rRooXef/99ffzxx9bVuYoVK+rjjz/Wjh071KJFC73wwgt65plnnF7/t7/9TSNHjtSAAQNUs2ZNzZgxo0T7d3Fx0aJFi3T8+HF16NBBw4cP17PPPutUU7VqVc2YMUPt2rVT+/bttXfvXn311VcX/DMFgPLOYU7/8DkAAFeZdevWqVOnTtq1a5fq1atndztlxuFwaNGiRU7fXwYAuPbwkUUAwFVl0aJFqlKliho0aKBdu3Zp9OjRuummm66pMAYAKD8IZACAq8rRo0c1YcIEpaSkqEaNGgoLCyty7xSK97///c/pO8TOlJOTcxm7AQBIfGQRAIBy4/jx4/rtt9/Our5+/fqXsRsAgEQgAwAAAADbMAUSAAAAANiEQAYAAAAANiGQAQAAAIBNCGQAAAAAYBMCGQAAAADYhEAGAAAAADYhkAEAAACATf4fHdFylKiqvQAAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 1000x600 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def plot_data_lengths(tokenized_train_dataset, tokenized_val_dataset):\n",
        "    lengths = [len(x['input_ids']) for x in tokenized_train_dataset]\n",
        "    lengths += [len(x['input_ids']) for x in tokenized_val_dataset]\n",
        "    print(len(lengths))\n",
        "\n",
        "    # Plotting the histogram\n",
        "    plt.figure(figsize=(10, 6))\n",
        "    plt.hist(lengths, bins=50, alpha=0.7, color='blue')\n",
        "    plt.xlabel('Length of input_ids')\n",
        "    plt.ylabel('Frequency')\n",
        "    plt.title('Distribution of Lengths of input_ids')\n",
        "    plt.xlim([0, 2048])\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "plot_data_lengths(tokenized_train_dataset, tokenized_val_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PLwlXqBzyVQU",
        "outputId": "99cbb9e6-c9a6-49ad-f3cc-6772be0cebb7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "MixtralForCausalLM(\n",
            "  (model): MixtralModel(\n",
            "    (embed_tokens): Embedding(32000, 4096)\n",
            "    (layers): ModuleList(\n",
            "      (0-31): 32 x MixtralDecoderLayer(\n",
            "        (self_attn): MixtralFlashAttention2(\n",
            "          (q_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
            "          (k_proj): Linear4bit(in_features=4096, out_features=1024, bias=False)\n",
            "          (v_proj): Linear4bit(in_features=4096, out_features=1024, bias=False)\n",
            "          (o_proj): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
            "          (rotary_emb): MixtralRotaryEmbedding()\n",
            "        )\n",
            "        (block_sparse_moe): MixtralSparseMoeBlock(\n",
            "          (gate): Linear4bit(in_features=4096, out_features=8, bias=False)\n",
            "          (experts): ModuleList(\n",
            "            (0-7): 8 x MixtralBLockSparseTop2MLP(\n",
            "              (w1): Linear4bit(in_features=4096, out_features=14336, bias=False)\n",
            "              (w2): Linear4bit(in_features=14336, out_features=4096, bias=False)\n",
            "              (w3): Linear4bit(in_features=4096, out_features=14336, bias=False)\n",
            "              (act_fn): SiLU()\n",
            "            )\n",
            "          )\n",
            "        )\n",
            "        (input_layernorm): MixtralRMSNorm()\n",
            "        (post_attention_layernorm): MixtralRMSNorm()\n",
            "      )\n",
            "    )\n",
            "    (norm): MixtralRMSNorm()\n",
            "  )\n",
            "  (lm_head): Linear(in_features=4096, out_features=32000, bias=False)\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g-Vs5Dc3DdY_"
      },
      "source": [
        "### Setting up the Training\n",
        "\n",
        "we will be using the `huggingface` and the `peft` library!\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SQulqDzjd0gD"
      },
      "outputs": [],
      "source": [
        "from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n",
        "\n",
        "peft_config = LoraConfig(\n",
        "    lora_alpha=16,\n",
        "    lora_dropout=0.1,\n",
        "    r=64,\n",
        "    bias=\"none\",\n",
        "        target_modules=[\n",
        "        \"q_proj\",\n",
        "        \"k_proj\",\n",
        "        \"v_proj\",\n",
        "        \"o_proj\",\n",
        "        \"gate_proj\",\n",
        "        \"up_proj\",\n",
        "        \"down_proj\",\n",
        "        \"lm_head\",\n",
        "    ],\n",
        "    task_type=\"CAUSAL_LM\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1SWl1MgjDrXX"
      },
      "source": [
        "we need to prepare the model to be trained in 4bit so we will use the `prepare_model_for_kbit_training` function from peft\n",
        "\n",
        "> Indented block\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bcco3ITVd486"
      },
      "outputs": [],
      "source": [
        "model = prepare_model_for_kbit_training(model)\n",
        "model = get_peft_model(model, peft_config)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BJzV4BRhyVQU"
      },
      "outputs": [],
      "source": [
        "def print_trainable_parameters(model):\n",
        "    \"\"\"\n",
        "    Prints the number of trainable parameters in the model.\n",
        "    \"\"\"\n",
        "    trainable_params = 0\n",
        "    all_param = 0\n",
        "    for _, param in model.named_parameters():\n",
        "        all_param += param.numel()\n",
        "        if param.requires_grad:\n",
        "            trainable_params += param.numel()\n",
        "    print(\n",
        "        f\"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}\"\n",
        "    )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aWqhs0W-yVQk",
        "outputId": "29093942-dc74-43e8-fa29-e5177f7c75cb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "trainable params: 56836096 || all params: 23539437568 || trainable%: 0.24145052674182907\n"
          ]
        }
      ],
      "source": [
        "print_trainable_parameters(model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lxN2-hyHyVQk",
        "outputId": "342ea747-938a-44c3-cad5-b94b79ebcf0b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "PeftModelForCausalLM(\n",
            "  (base_model): LoraModel(\n",
            "    (model): MixtralForCausalLM(\n",
            "      (model): MixtralModel(\n",
            "        (embed_tokens): Embedding(32000, 4096)\n",
            "        (layers): ModuleList(\n",
            "          (0-31): 32 x MixtralDecoderLayer(\n",
            "            (self_attn): MixtralFlashAttention2(\n",
            "              (q_proj): lora.Linear4bit(\n",
            "                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
            "                (lora_dropout): ModuleDict(\n",
            "                  (default): Dropout(p=0.1, inplace=False)\n",
            "                )\n",
            "                (lora_A): ModuleDict(\n",
            "                  (default): Linear(in_features=4096, out_features=64, bias=False)\n",
            "                )\n",
            "                (lora_B): ModuleDict(\n",
            "                  (default): Linear(in_features=64, out_features=4096, bias=False)\n",
            "                )\n",
            "                (lora_embedding_A): ParameterDict()\n",
            "                (lora_embedding_B): ParameterDict()\n",
            "              )\n",
            "              (k_proj): lora.Linear4bit(\n",
            "                (base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)\n",
            "                (lora_dropout): ModuleDict(\n",
            "                  (default): Dropout(p=0.1, inplace=False)\n",
            "                )\n",
            "                (lora_A): ModuleDict(\n",
            "                  (default): Linear(in_features=4096, out_features=64, bias=False)\n",
            "                )\n",
            "                (lora_B): ModuleDict(\n",
            "                  (default): Linear(in_features=64, out_features=1024, bias=False)\n",
            "                )\n",
            "                (lora_embedding_A): ParameterDict()\n",
            "                (lora_embedding_B): ParameterDict()\n",
            "              )\n",
            "              (v_proj): lora.Linear4bit(\n",
            "                (base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False)\n",
            "                (lora_dropout): ModuleDict(\n",
            "                  (default): Dropout(p=0.1, inplace=False)\n",
            "                )\n",
            "                (lora_A): ModuleDict(\n",
            "                  (default): Linear(in_features=4096, out_features=64, bias=False)\n",
            "                )\n",
            "                (lora_B): ModuleDict(\n",
            "                  (default): Linear(in_features=64, out_features=1024, bias=False)\n",
            "                )\n",
            "                (lora_embedding_A): ParameterDict()\n",
            "                (lora_embedding_B): ParameterDict()\n",
            "              )\n",
            "              (o_proj): lora.Linear4bit(\n",
            "                (base_layer): Linear4bit(in_features=4096, out_features=4096, bias=False)\n",
            "                (lora_dropout): ModuleDict(\n",
            "                  (default): Dropout(p=0.1, inplace=False)\n",
            "                )\n",
            "                (lora_A): ModuleDict(\n",
            "                  (default): Linear(in_features=4096, out_features=64, bias=False)\n",
            "                )\n",
            "                (lora_B): ModuleDict(\n",
            "                  (default): Linear(in_features=64, out_features=4096, bias=False)\n",
            "                )\n",
            "                (lora_embedding_A): ParameterDict()\n",
            "                (lora_embedding_B): ParameterDict()\n",
            "              )\n",
            "              (rotary_emb): MixtralRotaryEmbedding()\n",
            "            )\n",
            "            (block_sparse_moe): MixtralSparseMoeBlock(\n",
            "              (gate): Linear4bit(in_features=4096, out_features=8, bias=False)\n",
            "              (experts): ModuleList(\n",
            "                (0-7): 8 x MixtralBLockSparseTop2MLP(\n",
            "                  (w1): Linear4bit(in_features=4096, out_features=14336, bias=False)\n",
            "                  (w2): Linear4bit(in_features=14336, out_features=4096, bias=False)\n",
            "                  (w3): Linear4bit(in_features=4096, out_features=14336, bias=False)\n",
            "                  (act_fn): SiLU()\n",
            "                )\n",
            "              )\n",
            "            )\n",
            "            (input_layernorm): MixtralRMSNorm()\n",
            "            (post_attention_layernorm): MixtralRMSNorm()\n",
            "          )\n",
            "        )\n",
            "        (norm): MixtralRMSNorm()\n",
            "      )\n",
            "      (lm_head): lora.Linear(\n",
            "        (base_layer): Linear(in_features=4096, out_features=32000, bias=False)\n",
            "        (lora_dropout): ModuleDict(\n",
            "          (default): Dropout(p=0.1, inplace=False)\n",
            "        )\n",
            "        (lora_A): ModuleDict(\n",
            "          (default): Linear(in_features=4096, out_features=64, bias=False)\n",
            "        )\n",
            "        (lora_B): ModuleDict(\n",
            "          (default): Linear(in_features=64, out_features=32000, bias=False)\n",
            "        )\n",
            "        (lora_embedding_A): ParameterDict()\n",
            "        (lora_embedding_B): ParameterDict()\n",
            "      )\n",
            "    )\n",
            "  )\n",
            ")\n"
          ]
        }
      ],
      "source": [
        "print(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "go8_nVq7jO9d"
      },
      "source": [
        "### Hyper-paramters for training\n",
        "\n",
        "These parameters will depend on how long you want to run training for.\n",
        "Most important to consider:\n",
        "\n",
        "`num_train_epochs/max_steps`: How many iterations over the data you want to do, BE CAREFUL, don't try too many, you will over-fit!!!!!\n",
        "\n",
        "`learning_rate`: Controls the speed of convergence\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hdzkKPXvyVQk",
        "outputId": "2b645f4e-673b-49ec-d463-94a0f806adfe"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "8\n"
          ]
        }
      ],
      "source": [
        "if torch.cuda.device_count() > 1: # If more than 1 GPU\n",
        "    print(torch.cuda.device_count())\n",
        "    model.is_parallelizable = True\n",
        "    model.model_parallel = True"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YhoCjs9md8pB"
      },
      "outputs": [],
      "source": [
        "from transformers import TrainingArguments\n",
        "\n",
        "args = TrainingArguments(\n",
        "  output_dir = \"Mixtral_Alpace_v2\",\n",
        "  #num_train_epochs=5,\n",
        "  max_steps = 1000, # comment out this line if you want to train in epochs\n",
        "  per_device_train_batch_size = 32,\n",
        "  warmup_steps = 0.03,\n",
        "  logging_steps=10,\n",
        "  save_strategy=\"epoch\",\n",
        "  #evaluation_strategy=\"epoch\",\n",
        "  evaluation_strategy=\"steps\",\n",
        "  eval_steps=10, # comment out this line if you want to evaluate at the end of each epoch\n",
        "  learning_rate=2.5e-5,\n",
        "  bf16=True,\n",
        "  # lr_scheduler_type='constant',\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gz6Vhh4WFpMM"
      },
      "source": [
        "Setting up the trainer.\n",
        "\n",
        "`max_seq_length`: Context window size\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UyyNtDrmeAkc",
        "outputId": "4e992c2f-38eb-40fd-e9cf-d4f8ec301b76"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/ubuntu/anaconda3/envs/moe/lib/python3.10/site-packages/trl/trainer/ppo_config.py:141: UserWarning: The `optimize_cuda_cache` arguement will be deprecated soon, please use `optimize_device_cache` instead.\n",
            "  warnings.warn(\n",
            "/home/ubuntu/anaconda3/envs/moe/lib/python3.10/site-packages/trl/trainer/utils.py:548: UserWarning: The passed formatting_func has more than one argument. Usually that function should have a single argument `example` which corresponds to the dictionary returned by each element of the dataset. Make sure you know what you are doing.\n",
            "  warnings.warn(\n",
            "/home/ubuntu/anaconda3/envs/moe/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:267: UserWarning: You passed `packing=True` to the SFTTrainer, and you are training your model with `max_steps` strategy. The dataset will be iterated until the `max_steps` are reached.\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "from trl import SFTTrainer\n",
        "\n",
        "max_seq_length = 1024\n",
        "\n",
        "trainer = SFTTrainer(\n",
        "  model=model,\n",
        "  peft_config=peft_config,\n",
        "  max_seq_length=max_seq_length,\n",
        "  tokenizer=tokenizer,\n",
        "  packing=True,\n",
        "  formatting_func=create_prompt, # this will aplly the create_prompt mapping to all training and test dataset\n",
        "  args=args,\n",
        "  train_dataset=instruct_tune_dataset[\"train\"],\n",
        "  eval_dataset=instruct_tune_dataset[\"test\"]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 373
        },
        "id": "bsOO4bR9fQBb",
        "outputId": "cb946242-b113-4a57-a70b-8fbe05bbac07"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
            "/home/ubuntu/anaconda3/envs/moe/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.\n",
            "  warnings.warn(\n",
            "The input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in torch.float16.\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='250' max='250' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [250/250 1:24:18, Epoch 0/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Step</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>10</td>\n",
              "      <td>1.807500</td>\n",
              "      <td>1.906137</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>20</td>\n",
              "      <td>1.775100</td>\n",
              "      <td>1.833566</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>30</td>\n",
              "      <td>1.711700</td>\n",
              "      <td>1.751890</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>40</td>\n",
              "      <td>1.641000</td>\n",
              "      <td>1.697467</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>50</td>\n",
              "      <td>1.599900</td>\n",
              "      <td>1.661334</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>60</td>\n",
              "      <td>1.543600</td>\n",
              "      <td>1.630663</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>70</td>\n",
              "      <td>1.530000</td>\n",
              "      <td>1.603509</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>80</td>\n",
              "      <td>1.475600</td>\n",
              "      <td>1.576260</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>90</td>\n",
              "      <td>1.478900</td>\n",
              "      <td>1.545025</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>100</td>\n",
              "      <td>1.450600</td>\n",
              "      <td>1.514665</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>110</td>\n",
              "      <td>1.438100</td>\n",
              "      <td>1.487722</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>120</td>\n",
              "      <td>1.402300</td>\n",
              "      <td>1.466162</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>130</td>\n",
              "      <td>1.371100</td>\n",
              "      <td>1.452927</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>140</td>\n",
              "      <td>1.368100</td>\n",
              "      <td>1.441543</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>150</td>\n",
              "      <td>1.361000</td>\n",
              "      <td>1.431839</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>160</td>\n",
              "      <td>1.373300</td>\n",
              "      <td>1.423666</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>170</td>\n",
              "      <td>1.323900</td>\n",
              "      <td>1.417060</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>180</td>\n",
              "      <td>1.343600</td>\n",
              "      <td>1.411828</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>190</td>\n",
              "      <td>1.333600</td>\n",
              "      <td>1.406992</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>200</td>\n",
              "      <td>1.357000</td>\n",
              "      <td>1.403609</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>210</td>\n",
              "      <td>1.329400</td>\n",
              "      <td>1.401243</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>220</td>\n",
              "      <td>1.324900</td>\n",
              "      <td>1.399090</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>230</td>\n",
              "      <td>1.339100</td>\n",
              "      <td>1.397419</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>240</td>\n",
              "      <td>1.336200</td>\n",
              "      <td>1.396421</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>250</td>\n",
              "      <td>1.327000</td>\n",
              "      <td>1.396268</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/home/ubuntu/anaconda3/envs/moe/lib/python3.10/site-packages/trl/trainer/utils.py:570: UserWarning: The dataset reached end and the iterator is reset to the start.\n",
            "  warnings.warn(\"The dataset reached end and the iterator is reset to the start.\")\n",
            "/home/ubuntu/anaconda3/envs/moe/lib/python3.10/site-packages/peft/utils/save_and_load.py:131: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\n",
            "  warnings.warn(\"Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\")\n",
            "/home/ubuntu/anaconda3/envs/moe/lib/python3.10/site-packages/peft/utils/save_and_load.py:131: UserWarning: Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\n",
            "  warnings.warn(\"Setting `save_embedding_layers` to `True` as embedding layers found in `target_modules`.\")\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "TrainOutput(global_step=250, training_loss=1.4537027282714843, metrics={'train_runtime': 5075.701, 'train_samples_per_second': 1.576, 'train_steps_per_second': 0.049, 'total_flos': 2.2918868238336e+18, 'train_loss': 1.4537027282714843, 'epoch': 0.23})"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trainer.train()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ysHgWnwbfRSt"
      },
      "outputs": [],
      "source": [
        "trainer.save_model(\"Mixtral_Alpace_v2\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "na6xC6f-mGGm"
      },
      "source": [
        "# Save Model and Push to Hub\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SBPVlNe0j-Nk"
      },
      "outputs": [],
      "source": [
        "!pip install huggingface-hub -qU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uCDmkWCXnmuv"
      },
      "outputs": [],
      "source": [
        "from huggingface_hub import notebook_login\n",
        "notebook_login()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HJTZaSXqnqPa"
      },
      "outputs": [],
      "source": [
        "trainer.push_to_hub(\"soheilfm/mistral-instruct-generation\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9zYrr6sfkA6M"
      },
      "outputs": [],
      "source": [
        "merged_model = model.merge_and_unload()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c1Eo6D2mnOgN"
      },
      "outputs": [],
      "source": [
        "def generate_response(prompt, model):\n",
        "  encoded_input = tokenizer(prompt,  return_tensors=\"pt\", add_special_tokens=True)\n",
        "  model_inputs = encoded_input.to('cuda')\n",
        "\n",
        "  generated_ids = model.generate(**model_inputs,\n",
        "                                 max_new_tokens=150,\n",
        "                                 do_sample=True,\n",
        "                                 pad_token_id=tokenizer.eos_token_id)\n",
        "\n",
        "  decoded_output = tokenizer.batch_decode(generated_ids)\n",
        "\n",
        "  return decoded_output[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fVhc6eD-yVQl"
      },
      "outputs": [],
      "source": [
        "prompt = \"[INST]Use the provided input to create an instruction that could have been used to generate the response with an LLM.\\nThere are more than 12,000 species of grass. The most common is Kentucky Bluegrass, because it grows quickly, easily, and is soft to the touch. Rygrass is shiny and bright green colored. Fescues are dark green and shiny. Bermuda grass is harder but can grow in drier soil.[/INST]\"\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NMFDDMwly9L4"
      },
      "outputs": [],
      "source": [
        "generate_response(prompt, merged_model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ldSnbui8yVQl",
        "outputId": "48427c76-a18c-47b8-b92b-d54e8cf67a60"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "8000"
            ]
          },
          "execution_count": 51,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "250*32"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "V100",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.13"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0055eb4d0d054684878bba6da9cae910": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "084223b1556a420da5964ee086c117e0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "09f69e88c4834d25be2be790022384b0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0b9648d0eccf4516a8c0a7e3843f76b8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "118879d64cae46f5b3e744f86404922c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "164bc349194f4e6ab1229f3d2d03b3db": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1c03379c0bf84b62a671021810a5caee": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2073bd16554341df9ea62874f4907e18": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "21ac78e940c84842bdde1b64bd1395ad": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "240f5b4175e4469381a53eacdf657419": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "290face1ed4647d7b172109e162c3769": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_773040f77dae4115aefdd958a658a3fc",
            "max": 116,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_876619b7e19441a096225f72e58b5544",
            "value": 116
          }
        },
        "296ee6c082da4c76be2b099484662d26": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2998cf3d58354f6f8465a6e2a10ebab2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2de61bee8b56442887b6f42592423064": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2efa53a79dd54d86b72284f3a87c7ca9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_164bc349194f4e6ab1229f3d2d03b3db",
            "placeholder": "​",
            "style": "IPY_MODEL_732e5508dfb1422c97c0fbc27b888f91",
            "value": " 571/571 [00:00&lt;00:00, 36.4kB/s]"
          }
        },
        "3454ad0be8554f39bea55eb8d9c3db1b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "35f2eadd1bca48bdbc6504fcd693dab7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4245f82dafc0478ca7ac1f5b372e5a12": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_82c83b75261c4f1db9b7c3d9ad0d097e",
            "placeholder": "​",
            "style": "IPY_MODEL_fc38a40ba92d46c49b3679a84c8ef879",
            "value": "pytorch_model-00002-of-00002.bin: 100%"
          }
        },
        "42becf797d9a42bc9984d8e30b307cff": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "444b9f99a79045b9a5c779a3d18bedd4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_240f5b4175e4469381a53eacdf657419",
            "placeholder": "​",
            "style": "IPY_MODEL_7af0d44687f142288e3e58f15338dedd",
            "value": "generation_config.json: 100%"
          }
        },
        "48648a0d46104a0db77eabf6d8fdf0f0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "494347c07d684876ba64b39d698250d0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fb5aaa2a96e047a0aa47d07a07b183fe",
            "max": 5064823659,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_f7a940cb871f41c7af7e79c32c3156e7",
            "value": 5064823659
          }
        },
        "4fbd859668ac43c884d39f675f1158f7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fbe68ed8ac6243f180408175284b98f5",
            "placeholder": "​",
            "style": "IPY_MODEL_86712973ec7e498da66f29dbcea3f24f",
            "value": "config.json: 100%"
          }
        },
        "50b0bd7f60ea41c59d6cbd055a10d765": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_85f25707d0a04084aaa1ada15616f415",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_553955d94d844f05bc19116611761127",
            "value": 2
          }
        },
        "50d301095b594acdb2d754377b029245": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "553955d94d844f05bc19116611761127": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6223c94b2b814f2ab7abe976402551b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f56251c2b16c4dae99f80db60e5f6330",
            "placeholder": "​",
            "style": "IPY_MODEL_9e98f0e4ef964bbeaa63ca278313bb19",
            "value": " 5.06G/5.06G [00:17&lt;00:00, 438MB/s]"
          }
        },
        "643f193c688d49a0a204ea70019f2cc5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "68b12f6f28fa452ea302d83e37084c43": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f369a81a80d84934881c8e9a0bc6a14f",
            "max": 571,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_72588505d177495b88c39917ae9f743c",
            "value": 571
          }
        },
        "6950d59bdf254dd896c9f30a220c0533": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0b9648d0eccf4516a8c0a7e3843f76b8",
            "max": 23950,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a2a870bddb694a8ea69bdac062ef6d46",
            "value": 23950
          }
        },
        "72588505d177495b88c39917ae9f743c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "72a4b9321abf4d58a093a3d09263c1ad": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "732e5508dfb1422c97c0fbc27b888f91": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "74b57be114494edda79768ff04961193": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "773040f77dae4115aefdd958a658a3fc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "79d82f5e0881485897a8e977269079c8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "7af0d44687f142288e3e58f15338dedd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7d8f03389fc64ddfa128be53ea2eb1e8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_21ac78e940c84842bdde1b64bd1395ad",
            "max": 2,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ddfc0a4121b8447b84abab68482e46a3",
            "value": 2
          }
        },
        "7e5645b33aa64e7ebbd60cb76e1ecd11": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4245f82dafc0478ca7ac1f5b372e5a12",
              "IPY_MODEL_494347c07d684876ba64b39d698250d0",
              "IPY_MODEL_6223c94b2b814f2ab7abe976402551b1"
            ],
            "layout": "IPY_MODEL_48648a0d46104a0db77eabf6d8fdf0f0"
          }
        },
        "82c83b75261c4f1db9b7c3d9ad0d097e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "82cafea491ca4350bea831165926d9fb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f3d9d131f5934a048930bc8983437983",
              "IPY_MODEL_50b0bd7f60ea41c59d6cbd055a10d765",
              "IPY_MODEL_f27e89855a374c6d93e292dce8e146c5"
            ],
            "layout": "IPY_MODEL_72a4b9321abf4d58a093a3d09263c1ad"
          }
        },
        "838cd2c0701c4b0b82fa8ddce51215db": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "85f25707d0a04084aaa1ada15616f415": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "86712973ec7e498da66f29dbcea3f24f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "876619b7e19441a096225f72e58b5544": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8aec8536c28f4c75b2728936f6843a2c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "90de7d165e804fbf8f3782ac5b16f8fb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "95863a2d3c074f45b79a894cb8e67d27": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "963c8c0504554e57bb9a6a8e2c2060fe": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_118879d64cae46f5b3e744f86404922c",
            "placeholder": "​",
            "style": "IPY_MODEL_0055eb4d0d054684878bba6da9cae910",
            "value": "pytorch_model.bin.index.json: 100%"
          }
        },
        "9e98f0e4ef964bbeaa63ca278313bb19": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a2a870bddb694a8ea69bdac062ef6d46": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "adcfd8c2daa14f81aea6c85bf98cb735": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_42becf797d9a42bc9984d8e30b307cff",
            "placeholder": "​",
            "style": "IPY_MODEL_1c03379c0bf84b62a671021810a5caee",
            "value": "Downloading shards: 100%"
          }
        },
        "ae012e86315f41e7ae12b84312d2455b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_444b9f99a79045b9a5c779a3d18bedd4",
              "IPY_MODEL_290face1ed4647d7b172109e162c3769",
              "IPY_MODEL_fbeb4393ccb34413bea869355235d0fa"
            ],
            "layout": "IPY_MODEL_838cd2c0701c4b0b82fa8ddce51215db"
          }
        },
        "b245d2f284d7450b9d99e820f6193739": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b3a9937a7f5642e1bdddd2cb1c723490": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2de61bee8b56442887b6f42592423064",
            "max": 9943028044,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_79d82f5e0881485897a8e977269079c8",
            "value": 9943028044
          }
        },
        "b414aed96db744e99cbe4bf052426fa9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "becf709cc6a4490481f6cbebbc6bad49": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_df28571a454d47d698d56d7cf4d90009",
            "placeholder": "​",
            "style": "IPY_MODEL_b414aed96db744e99cbe4bf052426fa9",
            "value": " 23.9k/23.9k [00:00&lt;00:00, 1.83MB/s]"
          }
        },
        "c07ef3e83cec46ef8a9fc9d7d3fdb6fe": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b245d2f284d7450b9d99e820f6193739",
            "placeholder": "​",
            "style": "IPY_MODEL_09f69e88c4834d25be2be790022384b0",
            "value": " 9.94G/9.94G [00:32&lt;00:00, 324MB/s]"
          }
        },
        "c577ab91428c4f5aac8299ba85104dc6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4fbd859668ac43c884d39f675f1158f7",
              "IPY_MODEL_68b12f6f28fa452ea302d83e37084c43",
              "IPY_MODEL_2efa53a79dd54d86b72284f3a87c7ca9"
            ],
            "layout": "IPY_MODEL_643f193c688d49a0a204ea70019f2cc5"
          }
        },
        "c65ab63b72d143c7bca446276d0f1aa5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_963c8c0504554e57bb9a6a8e2c2060fe",
              "IPY_MODEL_6950d59bdf254dd896c9f30a220c0533",
              "IPY_MODEL_becf709cc6a4490481f6cbebbc6bad49"
            ],
            "layout": "IPY_MODEL_90de7d165e804fbf8f3782ac5b16f8fb"
          }
        },
        "ccbc22e41b754682bea5e80531498470": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d429cb7700aa40099e28fa9b6565be26": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_db518d1bca324c98911ad35e5df91c3a",
              "IPY_MODEL_b3a9937a7f5642e1bdddd2cb1c723490",
              "IPY_MODEL_c07ef3e83cec46ef8a9fc9d7d3fdb6fe"
            ],
            "layout": "IPY_MODEL_74b57be114494edda79768ff04961193"
          }
        },
        "dae5a71abe874e69b3972d5fc861fd1a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "db518d1bca324c98911ad35e5df91c3a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ccbc22e41b754682bea5e80531498470",
            "placeholder": "​",
            "style": "IPY_MODEL_3454ad0be8554f39bea55eb8d9c3db1b",
            "value": "pytorch_model-00001-of-00002.bin: 100%"
          }
        },
        "ddfc0a4121b8447b84abab68482e46a3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "df28571a454d47d698d56d7cf4d90009": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ed774deaa35441578397fa1241118263": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_adcfd8c2daa14f81aea6c85bf98cb735",
              "IPY_MODEL_7d8f03389fc64ddfa128be53ea2eb1e8",
              "IPY_MODEL_fd169008a90840f69ca3ff57ad7f84ed"
            ],
            "layout": "IPY_MODEL_8aec8536c28f4c75b2728936f6843a2c"
          }
        },
        "f27e89855a374c6d93e292dce8e146c5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_296ee6c082da4c76be2b099484662d26",
            "placeholder": "​",
            "style": "IPY_MODEL_2073bd16554341df9ea62874f4907e18",
            "value": " 2/2 [00:14&lt;00:00,  6.73s/it]"
          }
        },
        "f369a81a80d84934881c8e9a0bc6a14f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f3d9d131f5934a048930bc8983437983": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_35f2eadd1bca48bdbc6504fcd693dab7",
            "placeholder": "​",
            "style": "IPY_MODEL_084223b1556a420da5964ee086c117e0",
            "value": "Loading checkpoint shards: 100%"
          }
        },
        "f56251c2b16c4dae99f80db60e5f6330": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f7a940cb871f41c7af7e79c32c3156e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "fb5aaa2a96e047a0aa47d07a07b183fe": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fbe68ed8ac6243f180408175284b98f5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fbeb4393ccb34413bea869355235d0fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_95863a2d3c074f45b79a894cb8e67d27",
            "placeholder": "​",
            "style": "IPY_MODEL_2998cf3d58354f6f8465a6e2a10ebab2",
            "value": " 116/116 [00:00&lt;00:00, 9.67kB/s]"
          }
        },
        "fc38a40ba92d46c49b3679a84c8ef879": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fd169008a90840f69ca3ff57ad7f84ed": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_50d301095b594acdb2d754377b029245",
            "placeholder": "​",
            "style": "IPY_MODEL_dae5a71abe874e69b3972d5fc861fd1a",
            "value": " 2/2 [00:51&lt;00:00, 24.32s/it]"
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}