Steven_Yan_—_Greenflash_Language_Analytics_Research.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/datascisteven/Automated-Hate-Tweet-Detection/blob/main/Steven_Yan_%E2%80%94_Greenflash_Language_Analytics_Research.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QtB39USdzFVd"
      },
      "source": [
        "# Greenflash Language Analytics Research\n",
        "\n",
        "## Purpose\n",
        "\n",
        "This file is a research playground to evaluate a series of natural language analysis techniques with the ultimate goal of finding the best combination accuracy, speed, and cost for each metric.\n",
        "\n",
        "## Project Timeline\n",
        "\n",
        "**Weeks 1 and 2: Generate data and prepare models**\n",
        "- Research commmon structures for chat data from ChatGPT, LLama3, Claude, HuggingFace, Gemini\n",
        "- Chat with some models! The wider variety of topics the better.\n",
        "- Find existing data sets online\n",
        "- Convert chats to common structure\n",
        "\n",
        "**Weeks 3-5: Conduct one analysis below per week**\n",
        "- Toxicity\n",
        "- Bias\n",
        "- Hallucinations\n",
        "\n",
        "\n",
        "1. Begin by loading chat data into memory at the top of the file. We will use the same data for each analysis. The data should should meet the following requirements:\n",
        "\n",
        "  - Include at least 20 different conversations (but can be 100+ if you find a good pre-made data set).\n",
        "  - The average conversation length should be no less than 5 prompts and responses (but can be as low as 1 prompt and response for a small number of the conversations).\n",
        "  - The prompts (user input) and responses (model output) should be clearly marked as such.\n",
        "  - If known, include the model that was used in the chat.\n",
        "  - If known, include the system prompt for the chat.\n",
        "\n",
        "\n",
        "## Week 1 To Do's\n",
        "- [X] Request API keys\n",
        "- [x] Choose top 3-5 analyses\n",
        "\n",
        "\n",
        "## Week 2 To Do's\n",
        "- [X] Figure out how to filter the datasets\n",
        "- [X] Conduct cluster analysis of 1M Chat Dataset\n",
        "- [ ] Get all the foundational models working\n",
        "    - [X] HuggingFace\n",
        "    - [X] OpenAI\n",
        "    - [ ] Replicate\n",
        "- [ ] Pull code for the research (animal-named) models\n",
        "- [ ] Complete code for calculating cost per token\n",
        "\n",
        "## Week 3 To Do's\n",
        "- [ ] Finish setting up the API access\n",
        "- [ ]"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Importing Packages"
      ],
      "metadata": {
        "id": "1UilnW6usEn5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! pip install datasets openai tiktoken langdetect replicate tokencost"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YeZFcteH2bBX",
        "outputId": "80875016-a03e-45b1-8c25-1245c9e08cb4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (2.21.0)\n",
            "Requirement already satisfied: openai in /usr/local/lib/python3.10/dist-packages (1.43.0)\n",
            "Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (0.7.0)\n",
            "Requirement already satisfied: langdetect in /usr/local/lib/python3.10/dist-packages (1.0.9)\n",
            "Requirement already satisfied: replicate in /usr/local/lib/python3.10/dist-packages (0.32.1)\n",
            "Requirement already satisfied: tokencost in /usr/local/lib/python3.10/dist-packages (0.1.12)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets) (3.15.4)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (1.26.4)\n",
            "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (17.0.0)\n",
            "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.3.8)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (2.1.4)\n",
            "Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.10/dist-packages (from datasets) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.10/dist-packages (from datasets) (4.66.5)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets) (3.5.0)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets) (0.70.16)\n",
            "Requirement already satisfied: fsspec<=2024.6.1,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.6.1,>=2023.1.0->datasets) (2024.6.1)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.10.5)\n",
            "Requirement already satisfied: huggingface-hub>=0.21.2 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.23.5)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (6.0.2)\n",
            "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)\n",
            "Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai) (1.7.0)\n",
            "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from openai) (0.27.2)\n",
            "Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from openai) (0.5.0)\n",
            "Requirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai) (2.8.2)\n",
            "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from openai) (1.3.1)\n",
            "Requirement already satisfied: typing-extensions<5,>=4.11 in /usr/local/lib/python3.10/dist-packages (from openai) (4.12.2)\n",
            "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken) (2024.5.15)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from langdetect) (1.16.0)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (2.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (24.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.5)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3)\n",
            "Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (3.8)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->openai) (1.2.2)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2024.7.4)\n",
            "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (1.0.5)\n",
            "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.14.0)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->openai) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.20.1 in /usr/local/lib/python3.10/dist-packages (from pydantic<3,>=1.9.0->openai) (2.20.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (3.3.2)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (2.0.7)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.1)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.1)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qq5lYQNTP6Uh",
        "outputId": "87c936ba-3312-4e51-c6a4-3ee78adc40d2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import os\n",
        "import requests\n",
        "\n",
        "# pd.set_option(\"display.max_columns\", None)\n",
        "# pd.set_option(\"display.max_rows\", None)\n",
        "# pd.set_option(\"display.max_colwidth\", None)\n",
        "# pd.reset_option('^display.', silent=True)\n",
        "\n",
        "# import tiktoken\n",
        "\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "from transformers import T5Tokenizer, T5ForConditionalGeneration\n",
        "from transformers import BertTokenizer, BertForQuestionAnswering\n",
        "from transformers import RobertaTokenizer, RobertaForQuestionAnswering\n",
        "from transformers import XLNetTokenizer, XLNetForQuestionAnswering\n",
        "\n",
        "from datasets import load_dataset\n",
        "\n",
        "import torch\n",
        "\n",
        "import openai\n",
        "from openai import OpenAI\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "from huggingface_hub import InferenceClient\n",
        "from huggingface_hub import InferenceApi\n",
        "\n",
        "from getpass import getpass\n",
        "import replicate\n",
        "\n",
        "from tokencost import (\n",
        "    calculate_prompt_cost,\n",
        "    calculate_completion_cost,\n",
        "    count_message_tokens,\n",
        "    count_string_tokens\n",
        ")"
      ],
      "metadata": {
        "id": "Py3ILCTiRKVx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import userdata\n",
        "\n",
        "hugging_face_api_key = userdata.get('huggingface_token')\n",
        "openai_api_key = userdata.get('openai_token')\n",
        "replicate_api_key = userdata.get('replicate_token')"
      ],
      "metadata": {
        "id": "qtkRL_GWRNgA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import sys\n",
        "sys.path.append('/content/drive/MyDrive/Projects/Greenflash')\n",
        "\n",
        "import modules"
      ],
      "metadata": {
        "id": "7IFcWKGdkJ8w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RuVMyJBeD5_e"
      },
      "source": [
        "# Datasets\n",
        "\n",
        "Reference: [A Survey of LLM Datasets: From Autoregressive Model to AI Chatbot](https://link.springer.com/article/10.1007/s11390-024-3767-3)\n",
        "\n",
        "\n",
        "For a more in-depth look at the datasets considered, see Datasets Research Notebook:  \n",
        "\n",
        "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1JmeZFooifQC-vj8thd87c7s3ylZEGI6N?usp=sharing)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## [OpenAssistant Dataset](https://huggingface.co/datasets/OpenAssistant/oasst1)\n",
        "\n",
        "In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations (OASST1), a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 fully annotated conversation trees. The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers.\n",
        "\n",
        "[OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://arxiv.org/abs/2304.07327)\n",
        "\n"
      ],
      "metadata": {
        "id": "cRnBNrDB2r_h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!huggingface-cli login"
      ],
      "metadata": {
        "id": "-tZ4MMGEbtV2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0667b7e1-9465-4260-c188-afdf2bb1a960"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "    _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|\n",
            "    _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|\n",
            "    _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|\n",
            "    _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|\n",
            "    _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|\n",
            "\n",
            "    To login, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens .\n",
            "Enter your token (input will not be visible): \n",
            "Add token as git credential? (Y/n) n\n",
            "Token is valid (permission: fineGrained).\n",
            "Your token has been saved to /root/.cache/huggingface/token\n",
            "Login successful\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dataset = load_dataset(\"OpenAssistant/oasst1\")\n",
        "train = dataset[\"train\"]\n",
        "df = train.to_pandas()\n",
        "val = dataset[\"validation\"]\n",
        "val_df = val.to_pandas()\n",
        "\n",
        "df_eng = df.loc[df.lang == \"en\"].reset_index()\n",
        "df_eng = df_eng.rename(columns={\"index\": \"original_index\"})\n",
        "\n",
        "questions = df_eng.loc[df_eng.role == \"prompter\"][\n",
        "    [\"original_index\", \"message_id\", \"message_tree_id\", \"text\"]\n",
        "]\n",
        "answers = df_eng.loc[df_eng.role == \"assistant\"][\n",
        "    [\"original_index\", \"parent_id\", \"text\"]\n",
        "]\n",
        "qa = questions.merge(answers, left_on=\"message_id\", right_on=\"parent_id\")\n",
        "\n",
        "qa_groupby = (\n",
        "    qa.groupby([\"message_tree_id\", \"original_index_x\", \"message_id\", \"text_x\"])\n",
        "    .agg({\"original_index_y\": lambda x: list(x), \"text_y\": lambda x: list(x)})\n",
        "    .reset_index()\n",
        ")\n",
        "\n",
        "questions_df = (\n",
        "    qa_groupby.groupby(\"message_tree_id\")\n",
        "    .agg({\"text_x\": lambda x: list(x)})\n",
        "    .reset_index()\n",
        ")\n",
        "\n",
        "questions = questions_df.text_x.values.tolist()"
      ],
      "metadata": {
        "id": "-pZYVDpTzt1L",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 304,
          "referenced_widgets": [
            "286da78b89bd4bc5a5daafeca53f2bbd",
            "4c350461720249169cb4688cb21b14b5",
            "c07d4a6a1e73438abe07c756e97e3c9a",
            "901725ea001040acae730de1eb882cbb",
            "3f62b667ff9048a0ab2057d8041b438e",
            "0dc48c96eb57408b877f448ea3969a02",
            "fc3af28d79864dbaa9d42431a23cf5a1",
            "e148e9a59f1046e5b500e5483a4ac690",
            "81811b613812428f94c00ab86f1affd8",
            "29606a5156ea409488ac79b79aa27681",
            "cd86a58c97a74f93a0cb244c6e6c7f91",
            "0c7376a6b6b249e4ac29f87616aefafe",
            "3a6ef48ca9244a03a2cc1548d98c2ecc",
            "d3ac3e06b52746a58de0effd5bbc5b07",
            "fca1e912e8434d6499d700e0244cb77a",
            "d87377f75c2e42b79f9fbf7898be1746",
            "fed27280e75646dd8a594d19417852a4",
            "06afeba549f145f4b95c5ab22931943b",
            "fe4417da58334831bffafbdbc692eaf0",
            "9c6253e2088945289eb759216a423385",
            "ebbfa61e608c45dd8b26e93ce4f4a870",
            "7d3eb000aab942a3a53e7f2724139e52",
            "cdb17a31074a4a20a1a534d625943b0d",
            "0db919ff99254b00baa35b430ff9a9df",
            "3e49da2824b94c62977c188431e3574e",
            "6da54fe2124e4cb3944db704872bce7a",
            "7d73430a1cb84c498a182abdbc298191",
            "bd4a22ccb50447ae977e80fc7028c7c6",
            "241a11d43c6442efac0769cbb0b44741",
            "bc281a421aba4d02a298e77e4f599751",
            "1dd19e232491416b81fe9f03bb7e47cf",
            "edf98de9fc8a49a59186af0b04f9a436",
            "08efbd64a892432795debb4764275bb9",
            "01c345a4f36446538d718988d50b378e",
            "e688e56e1d414ab88068378c082cda1e",
            "e03b69ddcc4c4a5a94302270da898b31",
            "bc7042677a0b42e89af6319bebc1d324",
            "2afaeace5d394a17adfaa1c35c2913d3",
            "3b95c71652554e56b8241c475f48d4b9",
            "138612b1d4a74102af98bef4be6d0251",
            "c96b1e71f6844825bffd0f159a71871d",
            "ca5a14ae72744744844590da85532522",
            "c9b2b8f1b82b49f281079ec95fcd6f29",
            "2d3e46bb57bf49d8b63531c269861b42",
            "0041536231c04d6897cd248c569e7491",
            "df80532a876f4708bde511c3c2ed33f2",
            "019102624c6c4aa28b5a4618cb64f3c7",
            "f37f797d4ed040109e6ff2d10da80cab",
            "de698f452291468c92324862218dfa7e",
            "5d3f26987b4948a1ac162fc4cb8e3639",
            "ba6811085e7d4db994b2d0e4ed58c594",
            "4c11c448418d425ca60ad896912bd2df",
            "9b6527ee01e54f19b45e1718d28eaef1",
            "4dcfaebf00d74a7fbb5c8513fc2d565e",
            "29fcbf8546f14d0ea1f404f55b44392f"
          ]
        },
        "outputId": "ffc0c560-d539-4a5e-adb0-d081e39686c8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading readme:   0%|          | 0.00/10.2k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "286da78b89bd4bc5a5daafeca53f2bbd"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data:   0%|          | 0.00/39.5M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0c7376a6b6b249e4ac29f87616aefafe"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data:   0%|          | 0.00/2.08M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "cdb17a31074a4a20a1a534d625943b0d"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating train split:   0%|          | 0/84437 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "01c345a4f36446538d718988d50b378e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating validation split:   0%|          | 0/4401 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0041536231c04d6897cd248c569e7491"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "questions_one = questions[500: 1000]\n"
      ],
      "metadata": {
        "id": "r05mxMPT3_gc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Models\n",
        "\n",
        "For a more in-depth look at the full set of models explored, see Models Research Notebook:  \n",
        "\n",
        "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1H0qq2YZrpgARSG01tBbMj6qYmBKryqC3?usp=sharing)\n",
        "\n"
      ],
      "metadata": {
        "id": "Gedq6vERPFml"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "weKuLTkdD-um"
      },
      "source": [
        "\n",
        "Load any models that are used across analyses so that they don't need to be reloaded into memory each time.\n",
        "  - SpaCy\n",
        "  - NLTK\n",
        "  - Pytorch and/or tensorflow\n",
        "  - HuggingFace\n",
        "  - Replicate\n",
        "  - Llama3\n",
        "  - OpenAI (or other foundation model APIs)\n",
        "  - 1-bit models (or other very low memory)\n",
        "  - Whatever else you find!"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## OpenAI API"
      ],
      "metadata": {
        "id": "XCrQNQjpla6q"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ```gpt-4o-2024-05-13```\n"
      ],
      "metadata": {
        "id": "ZVpqJVajlCna"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "client = OpenAI(\n",
        "    api_key=openai_api_key\n",
        ")\n",
        "\n",
        "def ask_gpt4(questions):\n",
        "    gpt4_answers = []\n",
        "    prompt_token_cost = 0\n",
        "    completion_token_cost = 0\n",
        "    message_token_count = 0\n",
        "    response_token_count = 0\n",
        "    for q in questions:\n",
        "        gpt4_answer_list = []\n",
        "        for question in q:\n",
        "            try:\n",
        "                input = [{\"role\": \"system\", \"content\": \"You are a helpful assistant.\"}, {\"role\": \"user\", \"content\": question}, ]\n",
        "                response = client.chat.completions.create(\n",
        "                    model=\"gpt-4o-2024-05-13\",  # Specify GPT-4 model\n",
        "                    messages=input\n",
        "                )\n",
        "                answer = response.choices[0].message.content\n",
        "                prompt_token_cost += calculate_prompt_cost(input, model='gpt-4o-2024-05-13')\n",
        "                completion_token_cost += calculate_completion_cost(answer, model='gpt-4o-2024-05-13')\n",
        "                message_token_count += count_message_tokens(input, model='gpt-4o-2024-05-13')\n",
        "                response_token_count += count_string_tokens(prompt=answer, model='gpt-4o-2024-05-13')\n",
        "                gpt4_answer_list.append(answer)\n",
        "            except Exception as e:\n",
        "                gpt4_answer_list.append(f\"Error: {str(e)}\")\n",
        "        gpt4_answers.append(gpt4_answer_list)\n",
        "    return gpt4_answers, prompt_token_cost, completion_token_cost, message_token_count, response_token_count\n",
        "\n",
        "gpt4_answers, gpt4_prompt_cost, gpt4_completion_cost, message_token_count, response_token_count = ask_gpt4(questions_one)"
      ],
      "metadata": {
        "id": "fdWu05xTlFb4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "gpt4_answers"
      ],
      "metadata": {
        "id": "8iq5LhFmNyAf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Prompt Cost: \", gpt4_prompt_cost)\n",
        "print(\"Completion Cost: \", gpt4_completion_cost)\n",
        "print(\"Message Token Count: \", message_token_count)\n",
        "print(\"Response Token Count: \", response_token_count)"
      ],
      "metadata": {
        "id": "1H6-1AIjcFOo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Replicate API"
      ],
      "metadata": {
        "id": "Ma2R9czM6Knw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "REPLICATE_API_TOKEN = getpass()\n",
        "os.environ[\"REPLICATE_API_TOKEN\"] = REPLICATE_API_TOKEN"
      ],
      "metadata": {
        "id": "NoY22yWBA0Yo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c1209cd6-f959-4a17-e7aa-f90e4d7e7f8c"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ```meta/meta-llama-3-70b-instruct```"
      ],
      "metadata": {
        "id": "H592zUKBA6vN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "def run_replicate_llama(questions):\n",
        "\n",
        "    answers = []\n",
        "    # prompt_cost = 0\n",
        "    # completion_cost = 0\n",
        "    # message_token_count = 0\n",
        "    # response_token_count = 0\n",
        "    for q in questions:\n",
        "        answer_list = []\n",
        "        for question in q:\n",
        "            output = replicate.run(\n",
        "            \"meta/meta-llama-3-70b-instruct\",\n",
        "            input={\"seed\": 42,\n",
        "                    \"top_p\": 0.9,\n",
        "                    \"prompt\": question,\n",
        "                    \"max_tokens\": 512,\n",
        "                    \"min_tokens\": 0,\n",
        "                    \"temperature\": 0.6,\n",
        "                    \"prompt_template\": \"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\n\\nYou are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\\n\\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\n\\n\",\n",
        "                    \"presence_penalty\": 1.15}\n",
        "            )\n",
        "            answer_list.append(''.join(output))\n",
        "            # prompt_cost += calculate_prompt_cost(question, model='replicate/meta/meta-llama-3-70b-instruct')\n",
        "            # completion += calculate_completion_cost(output, model='replicate/meta/meta-llama-3-70b-instruct')\n",
        "            # message_token_count += count_message_tokens(question, model='replicate/meta/meta-llama-3-70b-instruct')\n",
        "            # response_token_count += count_string_tokens(output, model='replicate/meta/meta-llama-3-70b-instruct')\n",
        "\n",
        "        answers.append(answer_list)\n",
        "\n",
        "    # return answers, prompt_cost, completion_cost, message_token_count, response_token_count\n",
        "    return answers\n",
        "\n",
        "# replicate_llama_answers, replicate_llama_prompt_cost, replicate_llama_completion_cost, replicate_llama_message_token_count, replicate_llama_response_token_count = run_replicate_llama(questions)\n",
        "\n",
        "replicate_llama_answers = run_replicate_llama(questions_100)\n"
      ],
      "metadata": {
        "id": "7_jnF5CjNtij"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "### ```mistralai/mistral-7b-instruct-v0.2```\n",
        "\n"
      ],
      "metadata": {
        "id": "mxW4QwWNFFG8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cost_per_token_input = 0.05/10**6\n",
        "cost_per_token_output = 0.25/10**6"
      ],
      "metadata": {
        "id": "bFp2wRiVD_kM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "def run_replicate_mistral(questions):\n",
        "\n",
        "    answers = []\n",
        "    prompt_cost = 0\n",
        "    completion_cost = 0\n",
        "    message_token_count = 0\n",
        "    response_token_count = 0\n",
        "\n",
        "    for q in questions:\n",
        "        answer_list = []\n",
        "        for question in q:\n",
        "            output = replicate.run(\n",
        "            \"mistralai/mistral-7b-instruct-v0.2\",\n",
        "            input={\n",
        "                \"seed\": 42,\n",
        "                \"top_k\": 50,\n",
        "                \"top_p\": 0.9,\n",
        "                \"prompt\": question,\n",
        "                \"temperature\": 0.6,\n",
        "                \"system_prompt\": \"You are a very helpful, respectful and honest assistant.\",\n",
        "                \"length_penalty\": 1,\n",
        "                \"max_new_tokens\": 1024,\n",
        "                \"prompt_template\": \"<s>[INST] {prompt} [/INST] \",\n",
        "                \"presence_penalty\": 0}\n",
        "            )\n",
        "            answer_list.append(''.join(output))\n",
        "\n",
        "            message_token_count += len(tokenizer.encode(question))\n",
        "            response_token_count += len(tokenizer.encode(output))\n",
        "            prompt_cost += message_token_count * cost_per_token_input\n",
        "            completion_cost += response_token_count * cost_per_token_output\n",
        "\n",
        "        answers.append(answer_list)\n",
        "\n",
        "    # return answers, prompt_cost, completion_cost, message_token_count, response_token_count\n",
        "    return answers\n",
        "\n",
        "replicate_mistral_answers = run_replicate_mistral(qs)\n"
      ],
      "metadata": {
        "id": "8SBcP4-xBKml"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "def run_replicate_mistral(questions):\n",
        "\n",
        "    answers = []\n",
        "    # prompt_cost = 0\n",
        "    # completion_cost = 0\n",
        "    # message_token_count = 0\n",
        "    # response_token_count = 0\n",
        "\n",
        "    for q in questions:\n",
        "        answer_list = []\n",
        "        for question in q:\n",
        "            output = replicate.run(\n",
        "            \"mistralai/mistral-7b-instruct-v0.2\",\n",
        "            input={\n",
        "                \"seed\": 42,\n",
        "                \"top_k\": 50,\n",
        "                \"top_p\": 0.9,\n",
        "                \"prompt\": question,\n",
        "                \"temperature\": 0.6,\n",
        "                \"system_prompt\": \"You are a very helpful, respectful and honest assistant.\",\n",
        "                \"length_penalty\": 1,\n",
        "                \"max_new_tokens\": 1024,\n",
        "                \"prompt_template\": \"<s>[INST] {prompt} [/INST] \",\n",
        "                \"presence_penalty\": 0}\n",
        "            )\n",
        "            answer_list.append(''.join(output))\n",
        "            # prompt_cost += calculate_prompt_cost(question, model='replicate/mistralai/mistral-7b-instruct-v0.2')\n",
        "            # completion_cost += calculate_completion_cost(output, model='replicate/mistralai/mistral-7b-instruct-v0.2')\n",
        "            # message_token_count += count_message_tokens(question, model='replicate/mistralai/mistral-7b-instruct-v0.2')\n",
        "            # response_token_count += count_string_tokens(output, model='replicate/mistralai/mistral-7b-instruct-v0.2')\n",
        "\n",
        "        answers.append(answer_list)\n",
        "\n",
        "    # return answers, prompt_cost, completion_cost, message_token_count, response_token_count\n",
        "    return answers\n",
        "\n",
        "replicate_mistral_answers = run_replicate_mistral(questions_100)\n",
        "\n",
        "# replicate_mistral_answers, replicate_mistral_prompt_cost, replicate_mistral_completion_cost, replicate_mistral_message_token_count, replicate_mistral_response_token_count = run_replicate_mistral(questions_100)"
      ],
      "metadata": {
        "id": "2k5akB07Nctm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ```google-deepmind/gemma-7b-it```"
      ],
      "metadata": {
        "id": "OWb727X6BPfu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "\n",
        "def run_replicate_gemma(questions):\n",
        "\n",
        "    gemma_answers = []\n",
        "    gemma_prompt_cost = 0\n",
        "    gemma_completion_cost = 0\n",
        "\n",
        "    for q in qs:\n",
        "        gemma_answer_list = []\n",
        "        for question in q:\n",
        "            gemma_output = replicate.run(\"google-deepmind/gemma-7b-it:2790a695e5dcae15506138cc4718d1106d0d475e6dca4b1d43f42414647993d5\",\n",
        "            input={\n",
        "                \"top_k\": 50,\n",
        "                \"seed\": 42,\n",
        "                \"top_p\": 0.95,\n",
        "                \"prompt\": question,\n",
        "                \"temperature\": 0.7,\n",
        "                \"max_new_tokens\": 512,\n",
        "                \"min_new_tokens\": -1}\n",
        "            )\n",
        "            gemma_answer_list.append(''.join(gemma_output))\n",
        "\n",
        "        gemma_answers.append(gemma_answer_list)\n"
      ],
      "metadata": {
        "id": "6Mzj6fvXBVjC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model Comparison"
      ],
      "metadata": {
        "id": "syHSkSxnk4uT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_df = pd.DataFrame(\n",
        "    {'question': questions,\n",
        "     'answer': answers,\n",
        "     'DialoGPT-medium': dialogpt_answers,\n",
        "     'GPT-NeoX-20B': eleuther_answers,\n",
        "     'Meta-Llama-3-8B-Instruct': meta_answers,\n",
        "     'Mistral-Nemo-Instruct-2407': mistral_answers,\n",
        "     'Falcon-7B-Instruct': falcon_instruct_answers,\n",
        "     'Phi-3-mini-4k-instruct': phi_answers,\n",
        "     'zephyr-7b-beta': zephyr_answers\n",
        "     }\n",
        ")"
      ],
      "metadata": {
        "id": "i7Q5nXsDk784"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Costs Per Token"
      ],
      "metadata": {
        "id": "JxTo-eCLr0qM"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uUQaWF_P1C9_"
      },
      "source": [
        "\n",
        "Determine the costs per token of all APIs, and define functions to estimate the cost of an analysis that can be used below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "83OinZcr1NVa"
      },
      "outputs": [],
      "source": [
        "# Example chat data\n",
        "# (feel free to change the shape of these data if needed)\n",
        "chat_data_example = [\n",
        "    {\n",
        "        \"conversation_id\": \"12345\",\n",
        "        \"model\": \"gpt4o\",\n",
        "        \"system_prompt\": \"You are a helpful assisstant.\",\n",
        "        \"messages\": [\n",
        "            {\n",
        "                \"sender\": \"user\",\n",
        "                \"timestamp\": \"2024-07-28T10:00:00Z\",\n",
        "                \"content\": \"Hello, how are you?\"\n",
        "            },\n",
        "            {\n",
        "                \"sender\": \"assistant\",\n",
        "                \"timestamp\": \"2024-07-28T10:00:10Z\",\n",
        "                \"content\": \"I'm good, thank you! How can I assist you today?\"\n",
        "            },\n",
        "            {\n",
        "                \"sender\": \"user\",\n",
        "                \"timestamp\": \"2024-07-28T10:01:00Z\",\n",
        "                \"content\": \"Can you help me with a Python question?\"\n",
        "            },\n",
        "            {\n",
        "                \"sender\": \"assistant\",\n",
        "                \"timestamp\": \"2024-07-28T10:01:10Z\",\n",
        "                \"content\": \"Sure, what do you need help with?\"\n",
        "            }\n",
        "        ]\n",
        "    }\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tLxcTxBt29Ed"
      },
      "outputs": [],
      "source": [
        "# Example of loading an expensive model up front\n",
        "!python -m spacy download en_core_web_lg\n",
        "\n",
        "import spacy as spacy_nlp\n",
        "\n",
        "spacy = spacy_nlp.load(\"en_core_web_lg\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M__zedHG5miG"
      },
      "outputs": [],
      "source": [
        "# Example of token cost function\n",
        "example_API_cost_per_token = 0.001\n",
        "\n",
        "def calculate_costs(api, num_tokens):\n",
        "  match api:\n",
        "    case \"example\":\n",
        "      return num_tokens * example_API_cost_per_token\n",
        "    case _:\n",
        "      raise ValueError('API name is not defined')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ff0_6Ydr1KLs"
      },
      "source": [
        "# Analysis of Metrics\n",
        "\n",
        "\n",
        "```\n",
        "# This is formatted as code\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "The following categories are available to choose from for analysis:\n",
        "\n",
        "- **Bias** (Steven)\n",
        "- **Toxicity** (Steven)\n",
        "- Politeness\n",
        "- Prompt leakage\n",
        "- Response drift\n",
        "- **Hallucinations** (Steven)\n",
        "- Political language & content\n",
        "- Prompt & response specificity\n",
        "- Grammar, spelling, and fluency\n",
        "- Contextual completeness\n",
        "- Coherence\n",
        "- Linguistic diversity\n",
        "- Humor & sarcasm detection\n",
        "- Redundancy & repetition\n",
        "\n",
        "For each analysis, include:\n",
        "- A brief synopsis of what it is trying to measure\n",
        "- Why this metric is important to a customer\n",
        "- Any other relevant high level context about this field of analysis\n",
        "\n",
        "Then:\n",
        "- Research best practices for the analysis. Is there a generally accepted \"best\"? How about fastest? Lowest memory usage? Etc.\n",
        "- Perform the analysis with each of the models and techniques researched/loaded above\n",
        "- Measure the latency when performing the analysis\n",
        "- Estimate the cost for each analysis based on token count (for NLP APIs) or latency (for locally-computed analyses)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GRr-sAgO194D"
      },
      "source": [
        "EXAMPLE: Named Entity Recognition\n",
        "\n",
        "This analysis attempts to recognize and flag known and customer-generated entitites such as people, brand names, and products.\n",
        "\n",
        "\n",
        "Customer Impact\n",
        "\n",
        "NER will help customers understand whether their users are particularly interested in their own brands and products, competitors brands and products, or unrelated topics that are outside of the scope of the tool."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Bias"
      ],
      "metadata": {
        "id": "DD76hoV1x7Y7"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "23d_YTUm4Zho"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "9bvcxMB_x5vZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Toxicity"
      ],
      "metadata": {
        "id": "8FJHZj7Xx9xD"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "4mIinawryAZV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "RMxbpgP0yBAY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Hallucinations"
      ],
      "metadata": {
        "id": "T-RM9cBcyBjN"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "5EDAJXByyDIK"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "machine_shape": "hm",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "286da78b89bd4bc5a5daafeca53f2bbd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4c350461720249169cb4688cb21b14b5",
              "IPY_MODEL_c07d4a6a1e73438abe07c756e97e3c9a",
              "IPY_MODEL_901725ea001040acae730de1eb882cbb"
            ],
            "layout": "IPY_MODEL_3f62b667ff9048a0ab2057d8041b438e"
          }
        },
        "4c350461720249169cb4688cb21b14b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0dc48c96eb57408b877f448ea3969a02",
            "placeholder": "​",
            "style": "IPY_MODEL_fc3af28d79864dbaa9d42431a23cf5a1",
            "value": "Downloading readme: 100%"
          }
        },
        "c07d4a6a1e73438abe07c756e97e3c9a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e148e9a59f1046e5b500e5483a4ac690",
            "max": 10168,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_81811b613812428f94c00ab86f1affd8",
            "value": 10168
          }
        },
        "901725ea001040acae730de1eb882cbb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_29606a5156ea409488ac79b79aa27681",
            "placeholder": "​",
            "style": "IPY_MODEL_cd86a58c97a74f93a0cb244c6e6c7f91",
            "value": " 10.2k/10.2k [00:00&lt;00:00, 48.7kB/s]"
          }
        },
        "3f62b667ff9048a0ab2057d8041b438e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0dc48c96eb57408b877f448ea3969a02": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fc3af28d79864dbaa9d42431a23cf5a1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e148e9a59f1046e5b500e5483a4ac690": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "81811b613812428f94c00ab86f1affd8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "29606a5156ea409488ac79b79aa27681": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cd86a58c97a74f93a0cb244c6e6c7f91": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0c7376a6b6b249e4ac29f87616aefafe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3a6ef48ca9244a03a2cc1548d98c2ecc",
              "IPY_MODEL_d3ac3e06b52746a58de0effd5bbc5b07",
              "IPY_MODEL_fca1e912e8434d6499d700e0244cb77a"
            ],
            "layout": "IPY_MODEL_d87377f75c2e42b79f9fbf7898be1746"
          }
        },
        "3a6ef48ca9244a03a2cc1548d98c2ecc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fed27280e75646dd8a594d19417852a4",
            "placeholder": "​",
            "style": "IPY_MODEL_06afeba549f145f4b95c5ab22931943b",
            "value": "Downloading data: 100%"
          }
        },
        "d3ac3e06b52746a58de0effd5bbc5b07": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fe4417da58334831bffafbdbc692eaf0",
            "max": 39516251,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9c6253e2088945289eb759216a423385",
            "value": 39516251
          }
        },
        "fca1e912e8434d6499d700e0244cb77a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ebbfa61e608c45dd8b26e93ce4f4a870",
            "placeholder": "​",
            "style": "IPY_MODEL_7d3eb000aab942a3a53e7f2724139e52",
            "value": " 39.5M/39.5M [00:04&lt;00:00, 12.7MB/s]"
          }
        },
        "d87377f75c2e42b79f9fbf7898be1746": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fed27280e75646dd8a594d19417852a4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "06afeba549f145f4b95c5ab22931943b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fe4417da58334831bffafbdbc692eaf0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9c6253e2088945289eb759216a423385": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ebbfa61e608c45dd8b26e93ce4f4a870": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7d3eb000aab942a3a53e7f2724139e52": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cdb17a31074a4a20a1a534d625943b0d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0db919ff99254b00baa35b430ff9a9df",
              "IPY_MODEL_3e49da2824b94c62977c188431e3574e",
              "IPY_MODEL_6da54fe2124e4cb3944db704872bce7a"
            ],
            "layout": "IPY_MODEL_7d73430a1cb84c498a182abdbc298191"
          }
        },
        "0db919ff99254b00baa35b430ff9a9df": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bd4a22ccb50447ae977e80fc7028c7c6",
            "placeholder": "​",
            "style": "IPY_MODEL_241a11d43c6442efac0769cbb0b44741",
            "value": "Downloading data: 100%"
          }
        },
        "3e49da2824b94c62977c188431e3574e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bc281a421aba4d02a298e77e4f599751",
            "max": 2080179,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1dd19e232491416b81fe9f03bb7e47cf",
            "value": 2080179
          }
        },
        "6da54fe2124e4cb3944db704872bce7a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_edf98de9fc8a49a59186af0b04f9a436",
            "placeholder": "​",
            "style": "IPY_MODEL_08efbd64a892432795debb4764275bb9",
            "value": " 2.08M/2.08M [00:06&lt;00:00, 315kB/s]"
          }
        },
        "7d73430a1cb84c498a182abdbc298191": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd4a22ccb50447ae977e80fc7028c7c6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "241a11d43c6442efac0769cbb0b44741": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bc281a421aba4d02a298e77e4f599751": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1dd19e232491416b81fe9f03bb7e47cf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "edf98de9fc8a49a59186af0b04f9a436": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "08efbd64a892432795debb4764275bb9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "01c345a4f36446538d718988d50b378e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e688e56e1d414ab88068378c082cda1e",
              "IPY_MODEL_e03b69ddcc4c4a5a94302270da898b31",
              "IPY_MODEL_bc7042677a0b42e89af6319bebc1d324"
            ],
            "layout": "IPY_MODEL_2afaeace5d394a17adfaa1c35c2913d3"
          }
        },
        "e688e56e1d414ab88068378c082cda1e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3b95c71652554e56b8241c475f48d4b9",
            "placeholder": "​",
            "style": "IPY_MODEL_138612b1d4a74102af98bef4be6d0251",
            "value": "Generating train split: 100%"
          }
        },
        "e03b69ddcc4c4a5a94302270da898b31": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c96b1e71f6844825bffd0f159a71871d",
            "max": 84437,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ca5a14ae72744744844590da85532522",
            "value": 84437
          }
        },
        "bc7042677a0b42e89af6319bebc1d324": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c9b2b8f1b82b49f281079ec95fcd6f29",
            "placeholder": "​",
            "style": "IPY_MODEL_2d3e46bb57bf49d8b63531c269861b42",
            "value": " 84437/84437 [00:00&lt;00:00, 241050.72 examples/s]"
          }
        },
        "2afaeace5d394a17adfaa1c35c2913d3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3b95c71652554e56b8241c475f48d4b9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "138612b1d4a74102af98bef4be6d0251": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c96b1e71f6844825bffd0f159a71871d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ca5a14ae72744744844590da85532522": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c9b2b8f1b82b49f281079ec95fcd6f29": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2d3e46bb57bf49d8b63531c269861b42": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "0041536231c04d6897cd248c569e7491": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_df80532a876f4708bde511c3c2ed33f2",
              "IPY_MODEL_019102624c6c4aa28b5a4618cb64f3c7",
              "IPY_MODEL_f37f797d4ed040109e6ff2d10da80cab"
            ],
            "layout": "IPY_MODEL_de698f452291468c92324862218dfa7e"
          }
        },
        "df80532a876f4708bde511c3c2ed33f2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5d3f26987b4948a1ac162fc4cb8e3639",
            "placeholder": "​",
            "style": "IPY_MODEL_ba6811085e7d4db994b2d0e4ed58c594",
            "value": "Generating validation split: 100%"
          }
        },
        "019102624c6c4aa28b5a4618cb64f3c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4c11c448418d425ca60ad896912bd2df",
            "max": 4401,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9b6527ee01e54f19b45e1718d28eaef1",
            "value": 4401
          }
        },
        "f37f797d4ed040109e6ff2d10da80cab": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4dcfaebf00d74a7fbb5c8513fc2d565e",
            "placeholder": "​",
            "style": "IPY_MODEL_29fcbf8546f14d0ea1f404f55b44392f",
            "value": " 4401/4401 [00:00&lt;00:00, 126493.06 examples/s]"
          }
        },
        "de698f452291468c92324862218dfa7e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5d3f26987b4948a1ac162fc4cb8e3639": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ba6811085e7d4db994b2d0e4ed58c594": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4c11c448418d425ca60ad896912bd2df": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9b6527ee01e54f19b45e1718d28eaef1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4dcfaebf00d74a7fbb5c8513fc2d565e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "29fcbf8546f14d0ea1f404f55b44392f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}