IntroductiontoLLM_Activity_2_GPT_2_unipd.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pDKqQJFeSW0f"
      },
      "source": [
        "# Introduction to GPT-2\n",
        "by Eugenio Tufino, University of Padova, 9 December, 2024\n",
        "\n",
        "[Link to Google Colab](https://colab.research.google.com/drive/1MjpwPIb9uQFNLegfbfbXbG2LrgFd-nam?usp=sharing)\n",
        "\n",
        "In this notebook, we present GPT-2, a simplified model compared to the state-of-the-art generative models such as  GPT-4 or OpenAI o1, the model just released. GPT-2 is based on the **Generative Pre-trained Transformer** architecture and was released by OpenAI in **2019**. It uses a relatively simple mechanism to generate text, predicting the next word in a sequence based on the given context.\n",
        "\n",
        "Therefore unlike Word2Vec, where as we have seen word vectors are static (the same vector represents a word regardless of its usage), GPT-2 **dynamically adjusts the word representation** based on its context. This allows GPT-2 to better capture the meaning of words in different situations, resulting in more coherent and contextually accurate text generation.\n",
        "\n",
        "We will explore this model to understand, in a simplified way, how this kind of generative AI works, focusing on its word generation process and how prompts affect its output.\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "GPT stands for Generative Pre-trained Transformer, highlighting its key features:\n",
        "- Generative: The model generates coherent and meaningful text.\n",
        "- Pre-trained: It is trained on large datasets before being fine-tuned for specific tasks.\n",
        "- Transformer: It uses the Transformer architecture for efficient and context-aware text generation.\n",
        "\n",
        "\n",
        "Further reading:\n",
        "1. https://en.wikipedia.org/wiki/GPT-2\n",
        "2. https://huggingface.co/docs/transformers/en/model_doc/gpt2\n",
        "3. The Transformer architecture was introduced here: Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). \"Attention Is All You Need.\" Advances in Neural Information Processing Systems\n",
        "4.. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ji9zO4BzNLAN"
      },
      "source": [
        "### Import the libraries for using GPT-2\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qWFpRNz12cvF",
        "outputId": "27a63aaf-1b90-48d3-e84f-ef638ad7bd78"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.46.3)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.16.1)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.26.3)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.26.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.9.11)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.3)\n",
            "Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.20.3)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.5)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.6)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (2024.10.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (4.12.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.8.30)\n"
          ]
        }
      ],
      "source": [
        "!pip install transformers\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from transformers import GPT2LMHeadModel, GPT2Tokenizer\n",
        "import torch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "2bu8H4t_YQo5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 254,
          "referenced_widgets": [
            "974ee0eb79e244178a536cb657ffd036",
            "f4d412480c7b4d339437f1888aba13e1",
            "2b103faddec64652a127f0e8a1d24661",
            "f2be9bb2139a49719d7dee1fdddd9a25",
            "536a8116311a40f8a742387df7ed3b04",
            "7098e2d412ec406993d29664d4813718",
            "13d694d534af47fab9b88cd9443a2971",
            "25fb4bb8da3244c980c16f0caa857d7d",
            "c36fec2e63c540b1a8deffbc09a1e856",
            "dde479c2c04e44c88ddaffe5059973cd",
            "3516b48ee83b4247aa376d5f50f3a595",
            "a330e9a029b2406b8a9bcdef3f04d23f",
            "fc718b471f1c4c03b258cbe2eef62ac9",
            "9f9bb301b25249c2a9728be1fe40890b",
            "d2d9310ae2c440e69fd0e766329e7acd",
            "ee13f75a3dd841f8b364d04625dce4ec",
            "572c3e5f2b24444186ba9c56c493b0d2",
            "bfb89a0e9e48449790caed78f4be1f6d",
            "585caa92b72044db9dd56509bd35dc37",
            "1312c7adae054b33af529c75baa64bd3",
            "4f27d6e2b1fc4ace84a0d9efe8ce12bb",
            "11f5f630028d4facbad935489c72284b",
            "25072aaf6fc84d469995ba5ef8a1badc",
            "cc3ab30230b348e3a725685eb02b9961",
            "fe3b9f8674e542709da0fd47a9f42210",
            "6db84ab1af6b4dd3bea3283a77fbcf34",
            "17bce5448c72416ab6d22e595007e675",
            "dadc353b18e24b4ea95f45beed9b1fd1",
            "f1f5bd49078e4bf8b9203d85b763da89",
            "bed6b98a631d469fa62f710f37fa9ac8",
            "8ea33625971d40dfb87d3bb0f79ab90a",
            "6cc5bcf0dfa842cb8a7ea3b7706e8925",
            "b66de74ec7fd4feb8656c17dbeff6155"
          ]
        },
        "outputId": "5058c72c-c94d-48bc-ee16-17e8e4fefc92"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "vocab.json:   0%|          | 0.00/1.04M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "974ee0eb79e244178a536cb657ffd036"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model.safetensors:   0%|          | 0.00/548M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a330e9a029b2406b8a9bcdef3f04d23f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "generation_config.json:   0%|          | 0.00/124 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "25072aaf6fc84d469995ba5ef8a1badc"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GPT-2 model and tokenizer loaded successfully!\n"
          ]
        }
      ],
      "source": [
        "# Load the pre-trained GPT-2 model and tokenizer\n",
        "model_name = \"gpt2\"  # This is the smallest size. You can also use 'gpt2-medium', 'gpt2-large'.\n",
        "\n",
        "# The tokenizer processes input text into tokens and converts tokens back to text\n",
        "\n",
        "tokenizer = GPT2Tokenizer.from_pretrained(model_name) # Load the pre-trained tokenizer for GPT-2\n",
        "# The tokenizer processes input text into tokens and converts tokens back to text\n",
        "#Load the pre-trained GPT-2 language model\n",
        "# The model generates text by predicting the next token in a sequence\n",
        "model = GPT2LMHeadModel.from_pretrained(model_name)\n",
        "\n",
        "print(\"GPT-2 model and tokenizer loaded successfully!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lxfcr80CJO1g"
      },
      "source": [
        "### What does a 'tokenizer' mean, and what is a token?\n",
        "Check this website (for the latest version of GPT): [Link](https://platform.openai.com/tokenizer)\n",
        "\n",
        "Try to tokenize the following text:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "xySdSor5TuB3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c3f361d9-7034-4bed-d6f1-9b69396064d0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[2061,  318, 9552,   30]])\n"
          ]
        }
      ],
      "source": [
        "# Text to tokenize\n",
        "text = \"What is AI?\"\n",
        "# Convert text into tokens\n",
        "tokens = tokenizer.encode(text, return_tensors=\"pt\")\n",
        "# Output the tokens\n",
        "print(tokens)  #print the token Ids"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qt4J77PwQwZN"
      },
      "source": [
        "### Info about the model GPT-2\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "w7fbfpFgT3a5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4dac53d8-e94e-4953-ab77-1d479ce1d344"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Vocabulary size: 50257\n",
            "Vector size (number of dimensions): 768\n"
          ]
        }
      ],
      "source": [
        "# Check vocabulary size and vector size from the model's configuration\n",
        "vocab_size = model.config.vocab_size\n",
        "vector_size = model.config.n_embd\n",
        "\n",
        "print(\"Vocabulary size:\", vocab_size)\n",
        "print(\"Vector size (number of dimensions):\", vector_size)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vMD_ccvb6PF0"
      },
      "source": [
        "## Generating text with GPT-2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "Swdw4SZqZQ_1"
      },
      "outputs": [],
      "source": [
        "# Define a physics-related prompt\n",
        "prompt = \"In physics, energy is the ability to\"\n",
        "# Tokenize the prompt\n",
        "input_ids = tokenizer.encode(prompt, return_tensors='pt')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kBT-G5Sgaq1P",
        "outputId": "f2cd9c32-a87b-49c1-c6aa-e42a47563907"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Generated Text:\n",
            "In physics, energy is the ability to hold the object in its orbit around a point, as in the case of the sun. The sun's orbit moves in a straight line from point A to point B, and the gravitational field of a star (which is what makes stars move in an orbit) moves outwards from the center of B into the middle of A, or in other words, to the point where the star is at a position in which it is not in orbit. It's a\n"
          ]
        }
      ],
      "source": [
        "\n",
        "# Create an attention mask\n",
        "attention_mask = torch.ones_like(input_ids)\n",
        "\n",
        "output = model.generate(\n",
        "    input_ids,\n",
        "    attention_mask=attention_mask,  # Specify the attention mask\n",
        "    max_length=100,#Maximum length of the generated sequence\n",
        "    num_return_sequences=1,# Number of different sequences to generate\n",
        "    no_repeat_ngram_size=2,  # # Avoids repeated bigrams, see Annex A\n",
        "    temperature=0.7,\n",
        "    top_k=50, #Limits sampling to the top 50 tokens by probability, reducing unlikely choices\n",
        "    top_p=0.95,# Enables nucleus sampling, selecting tokens with cumulative probability ≤ 0.95\n",
        "    do_sample=True,  # Enable sampling\n",
        "    pad_token_id=tokenizer.eos_token_id,  # Set pad token\n",
        ")\n",
        "# Decode and print the generated text\n",
        "generated_text = tokenizer.decode(output[0], skip_special_tokens=True)\n",
        "print(\"Generated Text:\")\n",
        "print(generated_text)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6lHNDYWKB9nd"
      },
      "source": [
        "In a previous session I obtained the following text:\n",
        "> In physics, energy is the ability to move matter in the universe. This is why the earth is flat. But the way the world works is different.The earth orbits the sun, which is a reflection of the moon, and then it turns around and turns the other way around. The solar system's rotation slows down when the Earth spins, so the solar wind is pulled in two directions. It takes a lot of energy to turn the two sides of a circle around each other...\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ioNCzcl8a8ed"
      },
      "source": [
        "## Questions\n",
        "1. How coherent is the generated text with the given prompt?\n",
        "2. Does the model truly understand the concepts, or is it merely generating plausible sentences?\n",
        "3. How does its behavior differ from Word2Vec?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7H89VIwBQwj"
      },
      "source": [
        "GPT-2 generates text dynamically by predicting the next word in a sequence based on its context (through the Transformer architecture). It can produce  produce nonsensical or inaccurate content.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkK3JNhKaAmr"
      },
      "source": [
        "### Parameter you can vary to improve the generated text:\n",
        "**temperature:** Controls the randomness of text generation. A lower value produces more predictable text, while a higher value increases creativity.\n",
        "**max_length**: Adjusts the length of the generated text.\n",
        "**top_k and top_p:** Regulate the selection of probable words at each step.\n",
        "\n",
        "(see also Annex A below)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ADOpnk8DIaH"
      },
      "source": [
        "**Exercise 1** Adjust the temperature. Experiment with different calues of temperature to improve the quality of GPT-2’s generated output. Observe how it influences the behavior of the model.\n",
        "\n",
        " Which temperature value produces the most meaningful text for your physics context?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bJgma7fxCTEo"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YVP4Smnxq9Ql"
      },
      "source": [
        "### We can read the probabilities of the next word after the given prompt\n",
        "prompt = \"Newton's laws state that every object in motion\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PxRZAu6HqvlA",
        "outputId": "ad6d7c66-cdaa-49f2-ee74-d2578267e5e9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " be: 0.8125\n",
            " have: 0.1467\n",
            " pass: 0.0065\n",
            " conform: 0.0038\n",
            " move: 0.0028\n",
            " remain: 0.0019\n",
            " fall: 0.0015\n",
            " first: 0.0014\n",
            " \": 0.0014\n",
            " bear: 0.0014\n"
          ]
        }
      ],
      "source": [
        "from transformers import GPT2LMHeadModel, GPT2Tokenizer\n",
        "import torch\n",
        "import torch.nn.functional as F\n",
        "\n",
        "# Load the model and tokenizer\n",
        "model_name = \"gpt2\"\n",
        "tokenizer = GPT2Tokenizer.from_pretrained(model_name)\n",
        "model = GPT2LMHeadModel.from_pretrained(model_name)\n",
        "\n",
        "# Define a prompt\n",
        "prompt = \"Newton's laws state that every object in motion must\"\n",
        "\n",
        "# Encode the prompt\n",
        "input_ids = tokenizer.encode(prompt, return_tensors='pt')\n",
        "\n",
        "# Forward pass through the model to get logits\n",
        "outputs = model(input_ids)\n",
        "logits = outputs.logits  # Shape: (batch_size, sequence_length, vocab_size)\n",
        "\n",
        "# Get the logits for the last token in the input\n",
        "last_token_logits = logits[0, -1, :]  # Shape: (vocab_size,)\n",
        "\n",
        "# Set the temperature\n",
        "temperature = 0.7  # Lower = less random, higher = more random\n",
        "\n",
        "# Apply temperature to logits\n",
        "adjusted_logits = last_token_logits / temperature\n",
        "\n",
        "# Convert adjusted logits to probabilities\n",
        "probs = F.softmax(adjusted_logits, dim=-1)\n",
        "\n",
        "# Get the top 10 words with their probabilities\n",
        "top_k = 10\n",
        "top_k_probs, top_k_indices = torch.topk(probs, top_k)\n",
        "\n",
        "# Decode and print the top words with their probabilities\n",
        "for i in range(top_k):\n",
        "    word = tokenizer.decode(top_k_indices[i].item())\n",
        "    prob = top_k_probs[i].item()\n",
        "    print(f\"{word}: {prob:.4f}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U_sHeihJra7g"
      },
      "source": [
        "**Exercise 2:** Try to change the \"temperature\" parameter and see if there are any differences\n",
        "\n",
        "**Exercise 3:** You can continue the construction of the sentence, word by word..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YnNUKfR5tR20"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j0bAcTSttT5M"
      },
      "source": [
        "### Iterating the process\n",
        "Iterate the process changing only the temperature, the max_words and top_k parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rY4Wb_ZKtSK5",
        "outputId": "4358bafd-f4b5-4441-ca13-961a22729d3d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Newton's laws state that every object in motion must be\n",
            "Newton's laws state that every object in motion must be definite\n",
            "Newton's laws state that every object in motion must be definite.\n",
            "Newton's laws state that every object in motion must be definite. The\n",
            "Newton's laws state that every object in motion must be definite. The law\n",
            "Newton's laws state that every object in motion must be definite. The law required\n",
            "Newton's laws state that every object in motion must be definite. The law required that\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bra\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\"\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"dist\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\"\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bra\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place.\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bra\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses.\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States,\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries,\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarch\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power.\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power. Bra\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power. Bravery\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power. Bravery and\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power. Bravery and Religion\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power. Bravery and Religion,\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power. Bravery and Religion, upon\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power. Bravery and Religion, upon the\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power. Bravery and Religion, upon the other\n",
            "\n",
            "Final Generated Paragraph:\n",
            "Newton's laws state that every object in motion must be definite. The law required that Bravery and Religion be regarded as an intrinsic part Trace also was based on the concept that the \"nature of God\" must allow for \"distinctions of nature.\" This led to the standard at which Bravery and Religion took place. The concept of Bravery was also derived from the concept of the social development of the masses. In the United States, and in other developed countries, we see similar hierarchies of power. Bravery and Religion, upon the other\n"
          ]
        }
      ],
      "source": [
        "\n",
        "input_ids = tokenizer.encode(prompt, return_tensors='pt')\n",
        "\n",
        "# Set parameters\n",
        "max_words = 100  # Maximum number of words to generate\n",
        "temperature = 0.8\n",
        "top_k = 50\n",
        "\n",
        "# Generate word by word\n",
        "for _ in range(max_words):\n",
        "    # Forward pass to get logits\n",
        "    outputs = model(input_ids)\n",
        "    logits = outputs.logits  # Shape: (batch_size, sequence_length, vocab_size)\n",
        "\n",
        "    # Get the logits for the last token\n",
        "    last_token_logits = logits[0, -1, :]\n",
        "\n",
        "    # Apply temperature\n",
        "    adjusted_logits = last_token_logits / temperature\n",
        "\n",
        "    # Convert logits to probabilities\n",
        "    probs = F.softmax(adjusted_logits, dim=-1)\n",
        "\n",
        "    # Sample the next token\n",
        "    next_token_id = torch.multinomial(probs, num_samples=1)\n",
        "\n",
        "    # Append the token to the input_ids\n",
        "    input_ids = torch.cat([input_ids, next_token_id.unsqueeze(0)], dim=1)\n",
        "\n",
        "    # Decode the current output and print\n",
        "    generated_text = tokenizer.decode(input_ids[0])\n",
        "    print(generated_text)\n",
        "\n",
        "# Final generated paragraph\n",
        "print(\"\\nFinal Generated Paragraph:\")\n",
        "print(tokenizer.decode(input_ids[0]))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DeWC8SXByiXw"
      },
      "source": [
        "**Exercise: Try to improve the overall output with the following measures**\n",
        "\n",
        "\n",
        "\n",
        "1. Explore different prompts: You can test different prompts, specific to physics concepts, to observe how the behaviour of the model changes.\n",
        "2. Change Temperature: You can change the temperature parameter to explore the effect of randomness on generation.\n",
        "3. You can generalize the code to consider the parameter no_repeat_ngram_size=3  to prevent repetitions of blocks of text."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2aCep9M09v6d"
      },
      "source": [
        "## Annex A  Parameters for text generation\n",
        "\n",
        "### **1. `no_repeat_ngram_size`**\n",
        "\n",
        "What is an n-gram?\n",
        "\n",
        "An n-gram is a sequence of n consecutive tokens (words or subwords) in text. For example:\n",
        "\n",
        "    1-gram (Unigram): [\"The\", \"cat\", \"is\", \"on\", \"the\", \"mat\"]\n",
        "    2-gram (Bigram): [\"The cat\", \"cat is\", \"is on\", \"on the\", \"the mat\"]\n",
        "    \n",
        "- **Definition**: Prevents the repetition of n-grams (sequences of `n` consecutive tokens) in the generated text.\n",
        "- **Purpose**: Ensures variety in the output by disallowing repetitive phrases.\n",
        "- **Example**:\n",
        "  - With `no_repeat_ngram_size=2`: The bigram \"the cat\" cannot appear more than once in the text.\n",
        "  - Without it: Text may loop or repeat, like \"The cat is on the mat. The cat is on the mat.\"\n",
        "\n",
        "\n",
        "\n",
        "### **2. `temperature`**\n",
        "- **Definition**: Controls the randomness of token selection during generation.\n",
        "- **Purpose**: Adjusts the \"creativity\" of the output.\n",
        "- **Behavior**:\n",
        "  - Lower values (e.g., `0.5`): Predictable and conservative text.\n",
        "  - Higher values (e.g., `1.5`): More creative but can lead to incoherence.\n",
        "\n",
        "\n",
        "\n",
        "### **3. `top_k`**\n",
        "- **Definition**: Limits sampling to the top-k most probable tokens.\n",
        "- **Purpose**: Ensures the model selects tokens from a fixed number of high-probability options.\n",
        "- **Behavior**:\n",
        "  - Lower values (e.g., `10`): More focused and deterministic text.\n",
        "  - Higher values (e.g., `50`): Allows greater diversity but risks introducing randomness.\n",
        "\n",
        "\n",
        "### **4. `top_p` (Nucleus Sampling)**\n",
        "- **Definition**: Chooses tokens whose cumulative probability is ≤ `top_p`.\n",
        "- **Purpose**: Dynamically adjusts the diversity of token selection.\n",
        "- **Behavior**:\n",
        "  - Lower values (e.g., `0.8`): Limits to the most likely tokens, producing focused text.\n",
        "  - Higher values (e.g., `0.95`): Includes more tokens, allowing for greater creativity.\n",
        "\n",
        "### **5. `max_length`**\n",
        "- **Definition**: Sets the maximum number of tokens in the generated sequence, including the input.\n",
        "- **Purpose**: Controls how long the generated text will be.\n",
        "- **Behavior**:\n",
        "  - Short sequences (e.g., `50`): Output is concise but might be incomplete.\n",
        "  - Long sequences (e.g., `200`): Risk of losing coherence or repeating patterns.\n",
        "\n",
        "\n",
        "### **6. `pad_token_id`**\n",
        "- **Definition**: Specifies the token used for padding sequences to a uniform length.\n",
        "- **Purpose**: Ensures the model handles sequences with padding properly during generation.\n",
        "- **Common Value**: Set to the End-of-Sequence (EOS) token in GPT-2 (`tokenizer.eos_token_id`).\n",
        "\n",
        "\n",
        "### **7. `attention_mask`**\n",
        "- **Definition**: A binary mask indicating which tokens in the input are valid (1) and which are padding (0).\n",
        "- **Purpose**: Ensures the model only attends to meaningful tokens during processing.\n",
        "- **Behavior**:\n",
        "  - Without it: The model may produce unreliable results, especially with padded sequences.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_pJlz_8JW775"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "974ee0eb79e244178a536cb657ffd036": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_f4d412480c7b4d339437f1888aba13e1",
              "IPY_MODEL_2b103faddec64652a127f0e8a1d24661",
              "IPY_MODEL_f2be9bb2139a49719d7dee1fdddd9a25"
            ],
            "layout": "IPY_MODEL_536a8116311a40f8a742387df7ed3b04"
          }
        },
        "f4d412480c7b4d339437f1888aba13e1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7098e2d412ec406993d29664d4813718",
            "placeholder": "​",
            "style": "IPY_MODEL_13d694d534af47fab9b88cd9443a2971",
            "value": "vocab.json: 100%"
          }
        },
        "2b103faddec64652a127f0e8a1d24661": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_25fb4bb8da3244c980c16f0caa857d7d",
            "max": 1042301,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c36fec2e63c540b1a8deffbc09a1e856",
            "value": 1042301
          }
        },
        "f2be9bb2139a49719d7dee1fdddd9a25": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dde479c2c04e44c88ddaffe5059973cd",
            "placeholder": "​",
            "style": "IPY_MODEL_3516b48ee83b4247aa376d5f50f3a595",
            "value": " 1.04M/1.04M [00:00&lt;00:00, 5.84MB/s]"
          }
        },
        "536a8116311a40f8a742387df7ed3b04": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7098e2d412ec406993d29664d4813718": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "13d694d534af47fab9b88cd9443a2971": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "25fb4bb8da3244c980c16f0caa857d7d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c36fec2e63c540b1a8deffbc09a1e856": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "dde479c2c04e44c88ddaffe5059973cd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3516b48ee83b4247aa376d5f50f3a595": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a330e9a029b2406b8a9bcdef3f04d23f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_fc718b471f1c4c03b258cbe2eef62ac9",
              "IPY_MODEL_9f9bb301b25249c2a9728be1fe40890b",
              "IPY_MODEL_d2d9310ae2c440e69fd0e766329e7acd"
            ],
            "layout": "IPY_MODEL_ee13f75a3dd841f8b364d04625dce4ec"
          }
        },
        "fc718b471f1c4c03b258cbe2eef62ac9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_572c3e5f2b24444186ba9c56c493b0d2",
            "placeholder": "​",
            "style": "IPY_MODEL_bfb89a0e9e48449790caed78f4be1f6d",
            "value": "model.safetensors: 100%"
          }
        },
        "9f9bb301b25249c2a9728be1fe40890b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_585caa92b72044db9dd56509bd35dc37",
            "max": 548105171,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1312c7adae054b33af529c75baa64bd3",
            "value": 548105171
          }
        },
        "d2d9310ae2c440e69fd0e766329e7acd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4f27d6e2b1fc4ace84a0d9efe8ce12bb",
            "placeholder": "​",
            "style": "IPY_MODEL_11f5f630028d4facbad935489c72284b",
            "value": " 548M/548M [00:06&lt;00:00, 108MB/s]"
          }
        },
        "ee13f75a3dd841f8b364d04625dce4ec": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "572c3e5f2b24444186ba9c56c493b0d2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bfb89a0e9e48449790caed78f4be1f6d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "585caa92b72044db9dd56509bd35dc37": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1312c7adae054b33af529c75baa64bd3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4f27d6e2b1fc4ace84a0d9efe8ce12bb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "11f5f630028d4facbad935489c72284b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "25072aaf6fc84d469995ba5ef8a1badc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_cc3ab30230b348e3a725685eb02b9961",
              "IPY_MODEL_fe3b9f8674e542709da0fd47a9f42210",
              "IPY_MODEL_6db84ab1af6b4dd3bea3283a77fbcf34"
            ],
            "layout": "IPY_MODEL_17bce5448c72416ab6d22e595007e675"
          }
        },
        "cc3ab30230b348e3a725685eb02b9961": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dadc353b18e24b4ea95f45beed9b1fd1",
            "placeholder": "​",
            "style": "IPY_MODEL_f1f5bd49078e4bf8b9203d85b763da89",
            "value": "generation_config.json: 100%"
          }
        },
        "fe3b9f8674e542709da0fd47a9f42210": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bed6b98a631d469fa62f710f37fa9ac8",
            "max": 124,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8ea33625971d40dfb87d3bb0f79ab90a",
            "value": 124
          }
        },
        "6db84ab1af6b4dd3bea3283a77fbcf34": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6cc5bcf0dfa842cb8a7ea3b7706e8925",
            "placeholder": "​",
            "style": "IPY_MODEL_b66de74ec7fd4feb8656c17dbeff6155",
            "value": " 124/124 [00:00&lt;00:00, 8.09kB/s]"
          }
        },
        "17bce5448c72416ab6d22e595007e675": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dadc353b18e24b4ea95f45beed9b1fd1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f1f5bd49078e4bf8b9203d85b763da89": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bed6b98a631d469fa62f710f37fa9ac8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8ea33625971d40dfb87d3bb0f79ab90a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6cc5bcf0dfa842cb8a7ea3b7706e8925": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b66de74ec7fd4feb8656c17dbeff6155": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}