20_Natural_Language_Processing_CN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Tutorial #20\n",
    "# 自然语言处理\n",
    "\n",
    "by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)/[GitHub中文](https://github.com/Hvass-Labs/TensorFlow-Tutorials-Chinese)\n",
    "/ [GitHub](https://github.com/Hvass-Labs/TensorFlow-Tutorials) / [Videos on YouTube](https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ)\n",
    "\n",
    "中文翻译[ZhouGeorge](https://github.com/ZhouGeorge)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 介绍\n",
    "\n",
    "本教程是关于自然语言处理（NLP）的基础应用-情感分析，在这个过程中，我们会尝试把电影评论分类为正面或负面的。\n",
    "\n",
    "一个简单的例子：\"This movie is not very good.\"，文本的结尾是\"very good\"，这表明了一种非常积极的情感，但是它被否定了，因为它前面有 \"not\"这个词，所以这个文本应该被分类成负面的情感。我们怎样才能教一个神经网络来做这个分类呢？\n",
    "\n",
    "另一个问题是不能直接用于文本数据，所以我们需要将文本转换成能被神经网络接受的数字。\n",
    "\n",
    "还有另一个问题是文本的长度是任意的。我们之前教程中使用的神经网络的输入是固定的形状-除了第一维是用于改变batch-size。现在我们需要一种神经网络可以输入不同长度的文本。\n",
    "\n",
    "一般来说，你应该先熟悉TensorFlow和Keras，见教程 #01 和 #03-C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 流程图\n",
    "\n",
    "为了解决这个问题，我们需要几个步骤。首先，我们需要将原始文本数据转换成整数形代号（integer-tokens）。这些代号实际上只是整个词汇表索引的列表。为了将相似含义的词映射到相似的向量，我们将这些整形的代号转换到称为嵌入的实值向量（embedding-vectors），它的映射将用于神经网络的训练。之后，我们将这些嵌入向量输入（embedding-vectors）到循环神经网络，循环神经网络可以将任意长度的序列作为输入并输出关于输入内容的总结。我们将这个结果传入Sigmoid函数后会得到一个在0.0到1.0之间的值，0.0被认为是负面的情感，1.0代表正面的情感。这整个过程可以让我们将输入文本分类成负面或正面的情感。\n",
    "\n",
    "算法的流程图大致如下：\n",
    "\n",
    "<img src=\"images/20_natural_language_flowchart.png\" alt=\"Flowchart NLP\" style=\"width: 300px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 循环神经网络\n",
    "\n",
    "循环神经网络 (RNN)的基础构筑模块是循环单元(RU)。循环单元的变体有很多，比如相笨重的LSTM（Long-Short-Term-Memory）和我们将在个份教程中使用的稍微简单一点的GRU（Gated Recurrent Unit）。文献中的实验表明，LSTM和GRU的性能大致相同。甚至存在更简单的变体，它们的表现可能比LSTM和GRU更好，但是我们这个教程要用Keras，其他的变体没有在Keras中被实现。\n",
    "\n",
    "下面的图展示了循环单元的抽象概念，它有一个内部状态（internal state），每次当单元接收到一个新的输入时状态都会被更新。内部状态被视为一种记忆。然而，它不同于传统的计算机内存仅对每个位进行开或关的方式操作。相反的，循环单元将浮点值存储在其内存状态中，使用矩阵运算来读取和写入，因此操作都是可微的。这意味着记忆状态可以存储任意浮点值（尽管通常在-1.0和1.0之间）并且这个网络也可以像正常的神经网络一样用梯度下降法训练。\n",
    "\n",
    "新的状态值取决于过去的状态值和当前的输入。比如，如果一个状态已经被记住了，我们最近看到的词是 \"not\"并且当前的输入是\"good\" ，然后我们需要存储新的状态，记住 \"not good\"表示负面情感。\n",
    "\n",
    "循环单元用于将过去的状态和新的输入映射到新的状态部分，被称为gate，但是它实际只是一种矩阵运算。这里还有另一个gate用于计算循环单元的输出值。这些gata的实现方式因不同类型的循环单元而异。这幅图片仅仅展示了大概。LSTM比GRU有更多的gate，但有一些显得多余，可以被忽略。\n",
    "\n",
    "为了训练这个循环单元，我们必须逐渐改变gate的权重矩阵，那么在一个输入序列后循环单元会给我们想要的输出。这些都会在TensorFlow中自动地完成。\n",
    "\n",
    "![Recurrent unit](images/20_recurrent_unit.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 展开的网络\n",
    "\n",
    "另一种形象化和理解一个循环神经网络的方法是展开它的递归。这幅图只是一个循环单元（RU）的展示，它将在一系列的时间步长中从输入序列中接收文本字。\n",
    "\n",
    "每一次新的序列开始，RU的初始记忆状态会被Keras/TensorFlow内部重置为0。\n",
    "\n",
    "第一个时间步长中，\"this\"这个词被输入到RU，然后用RU的内部状态（初始是0）和它的gate计算出新的状态。这个RU也用它另一个gate去计算输出，但是输出在这里被无视了，因为只有在序列的结尾才需要输出一个总结。\n",
    "\n",
    "在第二个时间步长中 \"is\"这个词被输入到RU，而现在使用的内部状态是根据上一个词\"this\"刚刚更新过的。\n",
    "\n",
    "\"this is\"并没有什么意义，所以RU见过这些词后可能不会在它的内部状态中保存任何重要的东西。但是当它看到第三个词 \"not\"，RU已经了解到它可能对确定输入文本的整体情感很重要，所以它需要被存到RU的记忆状态中，当RU看见第六个时间步长的词\"good\"时，会被用到。\n",
    "\n",
    "最后整个序列被处理完后，RU输出一个向量的值，是关于它所看到的输入序列的总结。然后我们用带有Sigmoid激活函数的全连接层去获得一个0.0到1.0间独立的值，我们用他来解读近0.0）是正面的（值接近1.0）。\n",
    "\n",
    "注意，为了清晰起见，这个图并没有显示从文本（text-words）到整数代号（integer-tokens）和嵌入向量（embedding-vectors）的映射，也没有显示最后输出前的的全连接层和Sigmoid激活函数。\n",
    "\n",
    "![Unrolled network](images/20_unrolled_flowchart.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3层展开的网络\n",
    "\n",
    "在这份教程中我们使用具有3个循环单元（层）RU1,RU2,RU3的循环神经网络，“展开”图如下。\n",
    "第一层有点像上图的单层RNN。第一个循环单元RU1内部状态被Keras / TensorFlow初始化为0。然后\"this\"被输入到RU1并且更新它的内部状态。然后处理下个词\"is\"，等等。但是不同于在序列最后输出一个单独的总结，我们要用到RU1每一个时间步长的输出。这就生成了一个新的序列，它可作为下一个循环单元RU2的输入。第二层重复同样的过程并且也产生一个新的序列，可以作为第三个循环单元RU3的输入，而RU3的输出被传到具有Sigmoid的全连接层，输出的结果是0.0（负面情感）到1.0（正面情感）。\n",
    "\n",
    "注意，为了清晰起见，从文本（text-words）到整数代号（integer-tokens）和嵌入向量（embedding-vectors）的映射在下面的图中被省略了。\n",
    "\n",
    "\n",
    "![Unrolled 3-layer network](images/20_unrolled_3layers_flowchart.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 梯度消失和梯度爆炸\n",
    "\n",
    "为了训练循环单元里gate的权重，我们必须最小一些化损失函数，它们测量网络的实际输出与期望输出之间的差异。\n",
    "\n",
    "从上面的展开图我们可以看到，在输入序列中每个词都递归用到循环单元，这意味着每一个时间步长都要重复一次gate的计算。梯度信号必须从损失函数中回流到第一次计算的gate，如果循环gate的梯度计算是乘法的，那么这个传递的过程是指数级别的函数。\n",
    "\n",
    "在这个教程中我们将使用超过500词的文本。这意味着RU的gate更新它的内部记忆会递归超过500次。如果梯度是1.01，自乘500次，结果大约是145。如果梯度是0.99，自乘500次，结果大约是0.007 。这就称为梯度爆炸和梯度消失。唯一能在重复乘法中幸存的梯度是0和1。\n",
    "\n",
    "为了避免梯度消失和梯度爆炸，在设计循环单元和它的gate时必须要小心。这就是为什么GRU的实际实现更加复杂，因为它试图通过没有这种扭曲的方式将梯度传送回gate。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 引用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from scipy.spatial.distance import cdist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们需要从Keras中引入一些模块。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from tf.keras.models import Sequential  # This does not work!\n",
    "from tensorflow.python.keras.models import Sequential\n",
    "from tensorflow.python.keras.layers import Dense, GRU, Embedding\n",
    "from tensorflow.python.keras.optimizers import Adam\n",
    "from tensorflow.python.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.python.keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "开发环境Python 3.6（Anaconda）和其他安装包的版本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.5.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.1.2-tf'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 加载数据\n",
    "我们将使用从IMDB获取50000条影评来作为数据集。Keras内置一个函数可以下载相似的数据集（但是只有一半的大小）。然而，Keras的版本已经将数据集中的文本转换成整数代号，这是处理自然语言的一个关键部分，本教程中将做一个演示，所以我们下载了原始的文本数据。\n",
    "\n",
    "注意：数据集有84MB，它会被自动的下载。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import imdb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你想让文件保存到其他地址，你可以改变它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imdb.data_dir = \"data/IMDB/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "自动下载和解压文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data has apparently already been downloaded and unpacked.\n"
     ]
    }
   ],
   "source": [
    "imdb.maybe_download_and_extract()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "加载训练集和测试集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_text, y_train = imdb.load_data(train=True)\n",
    "x_test_text, y_test = imdb.load_data(train=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train-set size:  25000\n",
      "Test-set size:   25000\n"
     ]
    }
   ],
   "source": [
    "print(\"Train-set size: \", len(x_train_text))\n",
    "print(\"Test-set size:  \", len(x_test_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了进行下面的处理，将它们合并成一个数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_text = x_train_text + x_test_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输出训练集的一个例子，检测数据是否正确。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A simple comment...<br /><br />What can I say... this is a wonderful film that I can watch over and over. It is definitely one of the top ten comedies made. With a great cast, Jack Lemmon and Walter Matthau wording a perfect script by Neil Simon, based on his play.<br /><br />It is real to life situation done perfectly. If you have digital cable, one gets the menu on bottom of screen to give what is on. It usually gives this film ***% stars but in reality it deserves **** stars. If you really watch this film, one can tell that it will be as funny and fresh a hundred years from now.'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_text[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "正确的“类别”是影评的情感。0.0是负面的情感，1.0是正面的情感。在上面例子中，评论是积极的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分词器（tokenizer）\n",
    "\n",
    "神经网络不能直接处理字符串文本，所以我们必须转换它。这个过程有两步，第一步称为“分词器”，将数据集输入神经网络之前将单词转换成整形。第二步是属于神经网络自身处理的一部分，它被称为“embedding”层，它会在之后被介绍。\n",
    "\n",
    "我们可以指定分词器只使用数据集中10000个最常用的词"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_words = 10000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(num_words=num_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分词器可以“拟合”到数据集。它扫描所有的文本，将不需要的字符剥离出来（比如标点符号）并转换转为小写字符。然后分词器构建一个所有独特的词的词汇表，用于访问数据。\n",
    "\n",
    "注意，我们分词器“拟合”全部的数据集，所以它包括了训练集和测试集。这样做没问题是因为我们只是建立一个尽可能完整的词汇表。实际的神经网络当然只能在训练集上训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 10.6 s, sys: 16 ms, total: 10.6 s\n",
      "Wall time: 10.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tokenizer.fit_on_texts(data_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你喜欢使用完整的词汇表，设置上面的 `num_words=None`，然后它将自动设置词汇表大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "if num_words is None:\n",
    "    num_words = len(tokenizer.word_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后我们可以检查由分词器收集的词汇表。这是由数据集中的单词出现次数排序的。这些整形数字被称为词索引或者是代号（tokens），因为它们在词汇表中唯一地识别每个单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'the': 1,\n",
       " 'and': 2,\n",
       " 'a': 3,\n",
       " 'of': 4,\n",
       " 'to': 5,\n",
       " 'is': 6,\n",
       " 'br': 7,\n",
       " 'in': 8,\n",
       " 'it': 9,\n",
       " 'i': 10,\n",
       " 'this': 11,\n",
       " 'that': 12,\n",
       " 'was': 13,\n",
       " 'as': 14,\n",
       " 'for': 15,\n",
       " 'with': 16,\n",
       " 'movie': 17,\n",
       " 'but': 18,\n",
       " 'film': 19,\n",
       " 'on': 20,\n",
       " 'not': 21,\n",
       " 'you': 22,\n",
       " 'are': 23,\n",
       " 'his': 24,\n",
       " 'have': 25,\n",
       " 'be': 26,\n",
       " 'one': 27,\n",
       " 'he': 28,\n",
       " 'all': 29,\n",
       " 'at': 30,\n",
       " 'by': 31,\n",
       " 'an': 32,\n",
       " 'they': 33,\n",
       " 'so': 34,\n",
       " 'who': 35,\n",
       " 'from': 36,\n",
       " 'like': 37,\n",
       " 'or': 38,\n",
       " 'just': 39,\n",
       " 'her': 40,\n",
       " 'out': 41,\n",
       " 'about': 42,\n",
       " 'if': 43,\n",
       " \"it's\": 44,\n",
       " 'has': 45,\n",
       " 'there': 46,\n",
       " 'some': 47,\n",
       " 'what': 48,\n",
       " 'good': 49,\n",
       " 'when': 50,\n",
       " 'more': 51,\n",
       " 'very': 52,\n",
       " 'up': 53,\n",
       " 'no': 54,\n",
       " 'time': 55,\n",
       " 'my': 56,\n",
       " 'even': 57,\n",
       " 'would': 58,\n",
       " 'she': 59,\n",
       " 'which': 60,\n",
       " 'only': 61,\n",
       " 'really': 62,\n",
       " 'see': 63,\n",
       " 'story': 64,\n",
       " 'their': 65,\n",
       " 'had': 66,\n",
       " 'can': 67,\n",
       " 'me': 68,\n",
       " 'well': 69,\n",
       " 'were': 70,\n",
       " 'than': 71,\n",
       " 'much': 72,\n",
       " 'we': 73,\n",
       " 'bad': 74,\n",
       " 'been': 75,\n",
       " 'get': 76,\n",
       " 'do': 77,\n",
       " 'great': 78,\n",
       " 'other': 79,\n",
       " 'will': 80,\n",
       " 'also': 81,\n",
       " 'into': 82,\n",
       " 'people': 83,\n",
       " 'because': 84,\n",
       " 'how': 85,\n",
       " 'first': 86,\n",
       " 'him': 87,\n",
       " 'most': 88,\n",
       " \"don't\": 89,\n",
       " 'made': 90,\n",
       " 'then': 91,\n",
       " 'its': 92,\n",
       " 'them': 93,\n",
       " 'make': 94,\n",
       " 'way': 95,\n",
       " 'too': 96,\n",
       " 'movies': 97,\n",
       " 'could': 98,\n",
       " 'any': 99,\n",
       " 'after': 100,\n",
       " 'think': 101,\n",
       " 'characters': 102,\n",
       " 'watch': 103,\n",
       " 'films': 104,\n",
       " 'two': 105,\n",
       " 'many': 106,\n",
       " 'seen': 107,\n",
       " 'character': 108,\n",
       " 'being': 109,\n",
       " 'never': 110,\n",
       " 'plot': 111,\n",
       " 'love': 112,\n",
       " 'acting': 113,\n",
       " 'life': 114,\n",
       " 'did': 115,\n",
       " 'best': 116,\n",
       " 'where': 117,\n",
       " 'know': 118,\n",
       " 'show': 119,\n",
       " 'little': 120,\n",
       " 'over': 121,\n",
       " 'off': 122,\n",
       " 'ever': 123,\n",
       " 'does': 124,\n",
       " 'your': 125,\n",
       " 'better': 126,\n",
       " 'end': 127,\n",
       " 'man': 128,\n",
       " 'scene': 129,\n",
       " 'still': 130,\n",
       " 'say': 131,\n",
       " 'these': 132,\n",
       " 'here': 133,\n",
       " 'why': 134,\n",
       " 'scenes': 135,\n",
       " 'while': 136,\n",
       " 'something': 137,\n",
       " 'such': 138,\n",
       " 'go': 139,\n",
       " 'through': 140,\n",
       " 'back': 141,\n",
       " 'should': 142,\n",
       " 'those': 143,\n",
       " 'real': 144,\n",
       " \"i'm\": 145,\n",
       " 'now': 146,\n",
       " 'watching': 147,\n",
       " 'thing': 148,\n",
       " \"doesn't\": 149,\n",
       " 'actors': 150,\n",
       " 'though': 151,\n",
       " 'funny': 152,\n",
       " 'years': 153,\n",
       " \"didn't\": 154,\n",
       " 'old': 155,\n",
       " 'another': 156,\n",
       " '10': 157,\n",
       " 'work': 158,\n",
       " 'before': 159,\n",
       " 'actually': 160,\n",
       " 'nothing': 161,\n",
       " 'makes': 162,\n",
       " 'look': 163,\n",
       " 'director': 164,\n",
       " 'find': 165,\n",
       " 'going': 166,\n",
       " 'same': 167,\n",
       " 'new': 168,\n",
       " 'lot': 169,\n",
       " 'every': 170,\n",
       " 'few': 171,\n",
       " 'again': 172,\n",
       " 'part': 173,\n",
       " 'cast': 174,\n",
       " 'down': 175,\n",
       " 'us': 176,\n",
       " 'things': 177,\n",
       " 'want': 178,\n",
       " 'quite': 179,\n",
       " 'pretty': 180,\n",
       " 'world': 181,\n",
       " 'horror': 182,\n",
       " 'around': 183,\n",
       " 'seems': 184,\n",
       " \"can't\": 185,\n",
       " 'young': 186,\n",
       " 'take': 187,\n",
       " 'however': 188,\n",
       " 'got': 189,\n",
       " 'thought': 190,\n",
       " 'big': 191,\n",
       " 'fact': 192,\n",
       " 'enough': 193,\n",
       " 'long': 194,\n",
       " 'both': 195,\n",
       " \"that's\": 196,\n",
       " 'give': 197,\n",
       " \"i've\": 198,\n",
       " 'own': 199,\n",
       " 'may': 200,\n",
       " 'between': 201,\n",
       " 'comedy': 202,\n",
       " 'right': 203,\n",
       " 'series': 204,\n",
       " 'action': 205,\n",
       " 'must': 206,\n",
       " 'music': 207,\n",
       " 'without': 208,\n",
       " 'times': 209,\n",
       " 'saw': 210,\n",
       " 'always': 211,\n",
       " 'original': 212,\n",
       " \"isn't\": 213,\n",
       " 'role': 214,\n",
       " 'come': 215,\n",
       " 'almost': 216,\n",
       " 'gets': 217,\n",
       " 'interesting': 218,\n",
       " 'guy': 219,\n",
       " 'point': 220,\n",
       " 'done': 221,\n",
       " \"there's\": 222,\n",
       " 'whole': 223,\n",
       " 'least': 224,\n",
       " 'far': 225,\n",
       " 'bit': 226,\n",
       " 'script': 227,\n",
       " 'minutes': 228,\n",
       " 'feel': 229,\n",
       " '2': 230,\n",
       " 'anything': 231,\n",
       " 'making': 232,\n",
       " 'might': 233,\n",
       " 'since': 234,\n",
       " 'am': 235,\n",
       " 'family': 236,\n",
       " \"he's\": 237,\n",
       " 'last': 238,\n",
       " 'probably': 239,\n",
       " 'tv': 240,\n",
       " 'performance': 241,\n",
       " 'kind': 242,\n",
       " 'away': 243,\n",
       " 'yet': 244,\n",
       " 'fun': 245,\n",
       " 'worst': 246,\n",
       " 'sure': 247,\n",
       " 'rather': 248,\n",
       " 'hard': 249,\n",
       " 'girl': 250,\n",
       " 'anyone': 251,\n",
       " 'each': 252,\n",
       " 'played': 253,\n",
       " 'day': 254,\n",
       " 'found': 255,\n",
       " 'looking': 256,\n",
       " 'woman': 257,\n",
       " 'screen': 258,\n",
       " 'although': 259,\n",
       " 'our': 260,\n",
       " 'especially': 261,\n",
       " 'believe': 262,\n",
       " 'having': 263,\n",
       " 'trying': 264,\n",
       " 'course': 265,\n",
       " 'dvd': 266,\n",
       " 'everything': 267,\n",
       " 'set': 268,\n",
       " 'goes': 269,\n",
       " 'comes': 270,\n",
       " 'put': 271,\n",
       " 'ending': 272,\n",
       " 'maybe': 273,\n",
       " 'place': 274,\n",
       " 'book': 275,\n",
       " 'shows': 276,\n",
       " 'three': 277,\n",
       " 'worth': 278,\n",
       " 'different': 279,\n",
       " 'main': 280,\n",
       " 'once': 281,\n",
       " 'sense': 282,\n",
       " 'american': 283,\n",
       " 'reason': 284,\n",
       " 'looks': 285,\n",
       " 'effects': 286,\n",
       " 'watched': 287,\n",
       " 'play': 288,\n",
       " 'true': 289,\n",
       " 'money': 290,\n",
       " 'actor': 291,\n",
       " \"wasn't\": 292,\n",
       " 'job': 293,\n",
       " 'together': 294,\n",
       " 'war': 295,\n",
       " 'someone': 296,\n",
       " 'plays': 297,\n",
       " 'instead': 298,\n",
       " 'high': 299,\n",
       " 'during': 300,\n",
       " 'year': 301,\n",
       " 'said': 302,\n",
       " 'half': 303,\n",
       " 'everyone': 304,\n",
       " 'later': 305,\n",
       " 'takes': 306,\n",
       " '1': 307,\n",
       " 'seem': 308,\n",
       " 'audience': 309,\n",
       " 'special': 310,\n",
       " 'beautiful': 311,\n",
       " 'left': 312,\n",
       " 'himself': 313,\n",
       " 'seeing': 314,\n",
       " 'john': 315,\n",
       " 'night': 316,\n",
       " 'black': 317,\n",
       " 'version': 318,\n",
       " 'shot': 319,\n",
       " 'excellent': 320,\n",
       " 'idea': 321,\n",
       " 'house': 322,\n",
       " 'mind': 323,\n",
       " 'star': 324,\n",
       " 'wife': 325,\n",
       " 'fan': 326,\n",
       " 'death': 327,\n",
       " 'used': 328,\n",
       " 'else': 329,\n",
       " 'simply': 330,\n",
       " 'nice': 331,\n",
       " 'budget': 332,\n",
       " 'poor': 333,\n",
       " 'completely': 334,\n",
       " 'short': 335,\n",
       " 'second': 336,\n",
       " \"you're\": 337,\n",
       " '3': 338,\n",
       " 'read': 339,\n",
       " 'less': 340,\n",
       " 'along': 341,\n",
       " 'top': 342,\n",
       " 'help': 343,\n",
       " 'home': 344,\n",
       " 'men': 345,\n",
       " 'either': 346,\n",
       " 'line': 347,\n",
       " 'boring': 348,\n",
       " 'dead': 349,\n",
       " 'friends': 350,\n",
       " 'kids': 351,\n",
       " 'try': 352,\n",
       " 'production': 353,\n",
       " 'enjoy': 354,\n",
       " 'camera': 355,\n",
       " 'use': 356,\n",
       " 'wrong': 357,\n",
       " 'given': 358,\n",
       " 'low': 359,\n",
       " 'classic': 360,\n",
       " 'father': 361,\n",
       " 'need': 362,\n",
       " 'full': 363,\n",
       " 'stupid': 364,\n",
       " 'until': 365,\n",
       " 'next': 366,\n",
       " 'performances': 367,\n",
       " 'school': 368,\n",
       " 'hollywood': 369,\n",
       " 'rest': 370,\n",
       " 'truly': 371,\n",
       " 'awful': 372,\n",
       " 'video': 373,\n",
       " 'couple': 374,\n",
       " 'start': 375,\n",
       " 'sex': 376,\n",
       " 'recommend': 377,\n",
       " 'women': 378,\n",
       " 'let': 379,\n",
       " 'tell': 380,\n",
       " 'terrible': 381,\n",
       " 'remember': 382,\n",
       " 'mean': 383,\n",
       " 'came': 384,\n",
       " 'understand': 385,\n",
       " 'getting': 386,\n",
       " 'perhaps': 387,\n",
       " 'moments': 388,\n",
       " 'name': 389,\n",
       " 'keep': 390,\n",
       " 'face': 391,\n",
       " 'itself': 392,\n",
       " 'wonderful': 393,\n",
       " 'playing': 394,\n",
       " 'human': 395,\n",
       " 'style': 396,\n",
       " 'small': 397,\n",
       " 'episode': 398,\n",
       " 'perfect': 399,\n",
       " 'others': 400,\n",
       " 'person': 401,\n",
       " 'doing': 402,\n",
       " 'often': 403,\n",
       " 'early': 404,\n",
       " 'stars': 405,\n",
       " 'definitely': 406,\n",
       " 'written': 407,\n",
       " 'head': 408,\n",
       " 'lines': 409,\n",
       " 'dialogue': 410,\n",
       " 'gives': 411,\n",
       " 'piece': 412,\n",
       " \"couldn't\": 413,\n",
       " 'went': 414,\n",
       " 'finally': 415,\n",
       " 'mother': 416,\n",
       " 'case': 417,\n",
       " 'title': 418,\n",
       " 'absolutely': 419,\n",
       " 'live': 420,\n",
       " 'boy': 421,\n",
       " 'yes': 422,\n",
       " 'laugh': 423,\n",
       " 'certainly': 424,\n",
       " 'liked': 425,\n",
       " 'become': 426,\n",
       " 'entertaining': 427,\n",
       " 'worse': 428,\n",
       " 'oh': 429,\n",
       " 'sort': 430,\n",
       " 'loved': 431,\n",
       " 'lost': 432,\n",
       " 'hope': 433,\n",
       " 'called': 434,\n",
       " 'picture': 435,\n",
       " 'felt': 436,\n",
       " 'overall': 437,\n",
       " 'entire': 438,\n",
       " 'several': 439,\n",
       " 'mr': 440,\n",
       " 'based': 441,\n",
       " 'supposed': 442,\n",
       " 'cinema': 443,\n",
       " 'friend': 444,\n",
       " 'guys': 445,\n",
       " 'sound': 446,\n",
       " '5': 447,\n",
       " 'problem': 448,\n",
       " 'drama': 449,\n",
       " 'against': 450,\n",
       " 'waste': 451,\n",
       " 'white': 452,\n",
       " 'beginning': 453,\n",
       " '4': 454,\n",
       " 'fans': 455,\n",
       " 'totally': 456,\n",
       " 'dark': 457,\n",
       " 'care': 458,\n",
       " 'direction': 459,\n",
       " 'humor': 460,\n",
       " 'wanted': 461,\n",
       " \"she's\": 462,\n",
       " 'seemed': 463,\n",
       " 'under': 464,\n",
       " 'game': 465,\n",
       " 'children': 466,\n",
       " 'despite': 467,\n",
       " 'lives': 468,\n",
       " 'lead': 469,\n",
       " 'guess': 470,\n",
       " 'example': 471,\n",
       " 'already': 472,\n",
       " 'final': 473,\n",
       " 'throughout': 474,\n",
       " \"you'll\": 475,\n",
       " 'turn': 476,\n",
       " 'evil': 477,\n",
       " 'becomes': 478,\n",
       " 'unfortunately': 479,\n",
       " 'able': 480,\n",
       " 'quality': 481,\n",
       " \"i'd\": 482,\n",
       " 'days': 483,\n",
       " 'history': 484,\n",
       " 'fine': 485,\n",
       " 'side': 486,\n",
       " 'wants': 487,\n",
       " 'heart': 488,\n",
       " 'horrible': 489,\n",
       " 'writing': 490,\n",
       " 'amazing': 491,\n",
       " 'b': 492,\n",
       " 'flick': 493,\n",
       " 'killer': 494,\n",
       " 'run': 495,\n",
       " 'son': 496,\n",
       " '\\x96': 497,\n",
       " 'michael': 498,\n",
       " 'works': 499,\n",
       " 'close': 500,\n",
       " \"they're\": 501,\n",
       " 'act': 502,\n",
       " 'art': 503,\n",
       " 'matter': 504,\n",
       " 'kill': 505,\n",
       " 'etc': 506,\n",
       " 'tries': 507,\n",
       " \"won't\": 508,\n",
       " 'past': 509,\n",
       " 'town': 510,\n",
       " 'turns': 511,\n",
       " 'enjoyed': 512,\n",
       " 'brilliant': 513,\n",
       " 'gave': 514,\n",
       " 'behind': 515,\n",
       " 'parts': 516,\n",
       " 'stuff': 517,\n",
       " 'genre': 518,\n",
       " 'eyes': 519,\n",
       " 'car': 520,\n",
       " 'favorite': 521,\n",
       " 'directed': 522,\n",
       " 'late': 523,\n",
       " 'hand': 524,\n",
       " 'expect': 525,\n",
       " 'soon': 526,\n",
       " 'hour': 527,\n",
       " 'obviously': 528,\n",
       " 'themselves': 529,\n",
       " 'sometimes': 530,\n",
       " 'killed': 531,\n",
       " 'actress': 532,\n",
       " 'thinking': 533,\n",
       " 'child': 534,\n",
       " 'girls': 535,\n",
       " 'viewer': 536,\n",
       " 'starts': 537,\n",
       " 'city': 538,\n",
       " 'myself': 539,\n",
       " 'decent': 540,\n",
       " 'highly': 541,\n",
       " 'stop': 542,\n",
       " 'type': 543,\n",
       " 'self': 544,\n",
       " 'god': 545,\n",
       " 'says': 546,\n",
       " 'group': 547,\n",
       " 'anyway': 548,\n",
       " 'voice': 549,\n",
       " 'took': 550,\n",
       " 'known': 551,\n",
       " 'blood': 552,\n",
       " 'kid': 553,\n",
       " 'heard': 554,\n",
       " 'happens': 555,\n",
       " 'except': 556,\n",
       " 'fight': 557,\n",
       " 'feeling': 558,\n",
       " 'experience': 559,\n",
       " 'coming': 560,\n",
       " 'slow': 561,\n",
       " 'daughter': 562,\n",
       " 'writer': 563,\n",
       " 'stories': 564,\n",
       " 'moment': 565,\n",
       " 'leave': 566,\n",
       " 'told': 567,\n",
       " 'extremely': 568,\n",
       " 'score': 569,\n",
       " 'violence': 570,\n",
       " 'involved': 571,\n",
       " 'police': 572,\n",
       " 'strong': 573,\n",
       " 'chance': 574,\n",
       " 'lack': 575,\n",
       " 'cannot': 576,\n",
       " 'hit': 577,\n",
       " 'roles': 578,\n",
       " 'hilarious': 579,\n",
       " 's': 580,\n",
       " 'wonder': 581,\n",
       " 'happen': 582,\n",
       " 'particularly': 583,\n",
       " 'ok': 584,\n",
       " 'including': 585,\n",
       " 'living': 586,\n",
       " 'save': 587,\n",
       " 'looked': 588,\n",
       " \"wouldn't\": 589,\n",
       " 'crap': 590,\n",
       " 'simple': 591,\n",
       " 'please': 592,\n",
       " 'murder': 593,\n",
       " 'cool': 594,\n",
       " 'obvious': 595,\n",
       " 'happened': 596,\n",
       " 'complete': 597,\n",
       " 'cut': 598,\n",
       " 'serious': 599,\n",
       " 'age': 600,\n",
       " 'gore': 601,\n",
       " 'attempt': 602,\n",
       " 'hell': 603,\n",
       " 'ago': 604,\n",
       " 'song': 605,\n",
       " 'shown': 606,\n",
       " 'taken': 607,\n",
       " 'english': 608,\n",
       " 'james': 609,\n",
       " 'robert': 610,\n",
       " 'david': 611,\n",
       " 'seriously': 612,\n",
       " 'released': 613,\n",
       " 'reality': 614,\n",
       " 'opening': 615,\n",
       " 'interest': 616,\n",
       " 'jokes': 617,\n",
       " 'across': 618,\n",
       " 'none': 619,\n",
       " 'hero': 620,\n",
       " 'possible': 621,\n",
       " 'today': 622,\n",
       " 'exactly': 623,\n",
       " 'alone': 624,\n",
       " 'sad': 625,\n",
       " 'brother': 626,\n",
       " 'number': 627,\n",
       " 'saying': 628,\n",
       " 'career': 629,\n",
       " \"film's\": 630,\n",
       " 'usually': 631,\n",
       " 'hours': 632,\n",
       " 'cinematography': 633,\n",
       " 'talent': 634,\n",
       " 'view': 635,\n",
       " 'annoying': 636,\n",
       " 'running': 637,\n",
       " 'yourself': 638,\n",
       " 'relationship': 639,\n",
       " 'documentary': 640,\n",
       " 'wish': 641,\n",
       " 'huge': 642,\n",
       " 'order': 643,\n",
       " 'whose': 644,\n",
       " 'shots': 645,\n",
       " 'ridiculous': 646,\n",
       " 'taking': 647,\n",
       " 'important': 648,\n",
       " 'light': 649,\n",
       " 'body': 650,\n",
       " 'middle': 651,\n",
       " 'level': 652,\n",
       " 'ends': 653,\n",
       " 'started': 654,\n",
       " 'call': 655,\n",
       " 'female': 656,\n",
       " \"i'll\": 657,\n",
       " 'husband': 658,\n",
       " 'four': 659,\n",
       " 'power': 660,\n",
       " 'word': 661,\n",
       " 'turned': 662,\n",
       " 'major': 663,\n",
       " 'opinion': 664,\n",
       " 'change': 665,\n",
       " 'mostly': 666,\n",
       " 'usual': 667,\n",
       " 'silly': 668,\n",
       " 'scary': 669,\n",
       " 'rating': 670,\n",
       " 'beyond': 671,\n",
       " 'somewhat': 672,\n",
       " 'happy': 673,\n",
       " 'ones': 674,\n",
       " 'words': 675,\n",
       " 'room': 676,\n",
       " 'knows': 677,\n",
       " 'knew': 678,\n",
       " 'country': 679,\n",
       " 'disappointed': 680,\n",
       " 'talking': 681,\n",
       " 'novel': 682,\n",
       " 'apparently': 683,\n",
       " 'non': 684,\n",
       " 'strange': 685,\n",
       " 'upon': 686,\n",
       " 'attention': 687,\n",
       " 'finds': 688,\n",
       " 'basically': 689,\n",
       " 'single': 690,\n",
       " 'cheap': 691,\n",
       " 'modern': 692,\n",
       " 'due': 693,\n",
       " 'jack': 694,\n",
       " 'musical': 695,\n",
       " 'television': 696,\n",
       " 'problems': 697,\n",
       " 'miss': 698,\n",
       " 'episodes': 699,\n",
       " 'clearly': 700,\n",
       " 'local': 701,\n",
       " '7': 702,\n",
       " 'british': 703,\n",
       " 'thriller': 704,\n",
       " 'talk': 705,\n",
       " 'events': 706,\n",
       " 'five': 707,\n",
       " 'sequence': 708,\n",
       " \"aren't\": 709,\n",
       " 'class': 710,\n",
       " 'french': 711,\n",
       " 'moving': 712,\n",
       " 'ten': 713,\n",
       " 'fast': 714,\n",
       " 'review': 715,\n",
       " 'earth': 716,\n",
       " 'tells': 717,\n",
       " 'predictable': 718,\n",
       " 'songs': 719,\n",
       " 'team': 720,\n",
       " 'comic': 721,\n",
       " 'straight': 722,\n",
       " 'whether': 723,\n",
       " '8': 724,\n",
       " 'die': 725,\n",
       " 'add': 726,\n",
       " 'dialog': 727,\n",
       " 'entertainment': 728,\n",
       " 'above': 729,\n",
       " 'sets': 730,\n",
       " 'future': 731,\n",
       " 'enjoyable': 732,\n",
       " 'appears': 733,\n",
       " 'near': 734,\n",
       " 'space': 735,\n",
       " 'easily': 736,\n",
       " 'hate': 737,\n",
       " 'soundtrack': 738,\n",
       " 'bring': 739,\n",
       " 'giving': 740,\n",
       " 'lots': 741,\n",
       " 'similar': 742,\n",
       " 'romantic': 743,\n",
       " 'george': 744,\n",
       " 'supporting': 745,\n",
       " 'release': 746,\n",
       " 'mention': 747,\n",
       " 'filmed': 748,\n",
       " 'within': 749,\n",
       " 'message': 750,\n",
       " 'sequel': 751,\n",
       " 'clear': 752,\n",
       " 'falls': 753,\n",
       " 'needs': 754,\n",
       " \"haven't\": 755,\n",
       " 'dull': 756,\n",
       " 'suspense': 757,\n",
       " 'eye': 758,\n",
       " 'bunch': 759,\n",
       " 'surprised': 760,\n",
       " 'showing': 761,\n",
       " 'sorry': 762,\n",
       " 'tried': 763,\n",
       " 'certain': 764,\n",
       " 'easy': 765,\n",
       " 'working': 766,\n",
       " 'ways': 767,\n",
       " 'theme': 768,\n",
       " 'theater': 769,\n",
       " 'named': 770,\n",
       " 'among': 771,\n",
       " \"what's\": 772,\n",
       " 'storyline': 773,\n",
       " 'monster': 774,\n",
       " 'king': 775,\n",
       " 'stay': 776,\n",
       " 'effort': 777,\n",
       " 'stand': 778,\n",
       " 'fall': 779,\n",
       " 'minute': 780,\n",
       " 'gone': 781,\n",
       " 'rock': 782,\n",
       " 'using': 783,\n",
       " '9': 784,\n",
       " 'feature': 785,\n",
       " 'comments': 786,\n",
       " 'buy': 787,\n",
       " \"'\": 788,\n",
       " 'typical': 789,\n",
       " 't': 790,\n",
       " 'sister': 791,\n",
       " 'editing': 792,\n",
       " 'tale': 793,\n",
       " 'avoid': 794,\n",
       " 'deal': 795,\n",
       " 'mystery': 796,\n",
       " 'dr': 797,\n",
       " 'doubt': 798,\n",
       " 'fantastic': 799,\n",
       " 'kept': 800,\n",
       " 'nearly': 801,\n",
       " 'subject': 802,\n",
       " 'okay': 803,\n",
       " 'feels': 804,\n",
       " 'viewing': 805,\n",
       " 'elements': 806,\n",
       " 'oscar': 807,\n",
       " 'check': 808,\n",
       " 'points': 809,\n",
       " 'realistic': 810,\n",
       " 'greatest': 811,\n",
       " 'means': 812,\n",
       " 'herself': 813,\n",
       " 'parents': 814,\n",
       " 'famous': 815,\n",
       " 'imagine': 816,\n",
       " 'rent': 817,\n",
       " 'viewers': 818,\n",
       " 'crime': 819,\n",
       " 'richard': 820,\n",
       " 'form': 821,\n",
       " 'peter': 822,\n",
       " 'actual': 823,\n",
       " 'lady': 824,\n",
       " 'general': 825,\n",
       " 'dog': 826,\n",
       " 'follow': 827,\n",
       " 'believable': 828,\n",
       " 'period': 829,\n",
       " 'red': 830,\n",
       " 'brought': 831,\n",
       " 'move': 832,\n",
       " 'material': 833,\n",
       " 'forget': 834,\n",
       " 'somehow': 835,\n",
       " 'begins': 836,\n",
       " 're': 837,\n",
       " 'reviews': 838,\n",
       " 'animation': 839,\n",
       " 'paul': 840,\n",
       " \"you've\": 841,\n",
       " 'leads': 842,\n",
       " 'weak': 843,\n",
       " 'figure': 844,\n",
       " 'surprise': 845,\n",
       " 'sit': 846,\n",
       " 'hear': 847,\n",
       " 'average': 848,\n",
       " 'open': 849,\n",
       " 'sequences': 850,\n",
       " 'killing': 851,\n",
       " 'atmosphere': 852,\n",
       " 'eventually': 853,\n",
       " 'tom': 854,\n",
       " 'learn': 855,\n",
       " 'premise': 856,\n",
       " '20': 857,\n",
       " 'wait': 858,\n",
       " 'sci': 859,\n",
       " 'deep': 860,\n",
       " 'fi': 861,\n",
       " 'expected': 862,\n",
       " 'whatever': 863,\n",
       " 'indeed': 864,\n",
       " 'particular': 865,\n",
       " 'note': 866,\n",
       " 'poorly': 867,\n",
       " 'lame': 868,\n",
       " 'dance': 869,\n",
       " 'imdb': 870,\n",
       " 'situation': 871,\n",
       " 'shame': 872,\n",
       " 'third': 873,\n",
       " 'york': 874,\n",
       " 'box': 875,\n",
       " 'truth': 876,\n",
       " 'decided': 877,\n",
       " 'free': 878,\n",
       " 'hot': 879,\n",
       " \"who's\": 880,\n",
       " 'difficult': 881,\n",
       " 'needed': 882,\n",
       " 'season': 883,\n",
       " 'acted': 884,\n",
       " 'leaves': 885,\n",
       " 'unless': 886,\n",
       " 'emotional': 887,\n",
       " 'possibly': 888,\n",
       " 'romance': 889,\n",
       " 'sexual': 890,\n",
       " 'gay': 891,\n",
       " 'boys': 892,\n",
       " 'footage': 893,\n",
       " 'write': 894,\n",
       " 'western': 895,\n",
       " 'forced': 896,\n",
       " 'credits': 897,\n",
       " 'memorable': 898,\n",
       " 'doctor': 899,\n",
       " 'became': 900,\n",
       " 'reading': 901,\n",
       " 'otherwise': 902,\n",
       " 'begin': 903,\n",
       " 'air': 904,\n",
       " 'crew': 905,\n",
       " 'de': 906,\n",
       " 'question': 907,\n",
       " 'meet': 908,\n",
       " 'society': 909,\n",
       " 'male': 910,\n",
       " 'meets': 911,\n",
       " \"let's\": 912,\n",
       " 'plus': 913,\n",
       " 'cheesy': 914,\n",
       " 'hands': 915,\n",
       " 'superb': 916,\n",
       " 'screenplay': 917,\n",
       " 'beauty': 918,\n",
       " 'interested': 919,\n",
       " 'street': 920,\n",
       " 'features': 921,\n",
       " 'perfectly': 922,\n",
       " 'masterpiece': 923,\n",
       " 'whom': 924,\n",
       " 'laughs': 925,\n",
       " 'stage': 926,\n",
       " 'nature': 927,\n",
       " 'effect': 928,\n",
       " 'comment': 929,\n",
       " 'forward': 930,\n",
       " 'nor': 931,\n",
       " 'badly': 932,\n",
       " 'sounds': 933,\n",
       " 'previous': 934,\n",
       " 'e': 935,\n",
       " 'japanese': 936,\n",
       " 'weird': 937,\n",
       " 'island': 938,\n",
       " 'inside': 939,\n",
       " 'personal': 940,\n",
       " 'quickly': 941,\n",
       " 'total': 942,\n",
       " 'keeps': 943,\n",
       " 'towards': 944,\n",
       " 'result': 945,\n",
       " 'america': 946,\n",
       " 'battle': 947,\n",
       " 'crazy': 948,\n",
       " 'worked': 949,\n",
       " 'setting': 950,\n",
       " 'incredibly': 951,\n",
       " 'earlier': 952,\n",
       " 'background': 953,\n",
       " 'mess': 954,\n",
       " 'cop': 955,\n",
       " 'writers': 956,\n",
       " 'fire': 957,\n",
       " 'copy': 958,\n",
       " 'unique': 959,\n",
       " 'dumb': 960,\n",
       " 'realize': 961,\n",
       " 'powerful': 962,\n",
       " 'mark': 963,\n",
       " 'lee': 964,\n",
       " 'business': 965,\n",
       " 'rate': 966,\n",
       " 'dramatic': 967,\n",
       " 'older': 968,\n",
       " 'pay': 969,\n",
       " 'following': 970,\n",
       " 'directors': 971,\n",
       " 'girlfriend': 972,\n",
       " 'joke': 973,\n",
       " 'plenty': 974,\n",
       " 'directing': 975,\n",
       " 'various': 976,\n",
       " 'creepy': 977,\n",
       " 'baby': 978,\n",
       " 'development': 979,\n",
       " 'appear': 980,\n",
       " 'brings': 981,\n",
       " 'front': 982,\n",
       " 'ask': 983,\n",
       " 'dream': 984,\n",
       " 'water': 985,\n",
       " 'admit': 986,\n",
       " 'bill': 987,\n",
       " 'rich': 988,\n",
       " 'apart': 989,\n",
       " 'joe': 990,\n",
       " 'political': 991,\n",
       " 'fairly': 992,\n",
       " 'reasons': 993,\n",
       " 'leading': 994,\n",
       " 'portrayed': 995,\n",
       " 'spent': 996,\n",
       " 'telling': 997,\n",
       " 'cover': 998,\n",
       " 'outside': 999,\n",
       " 'wasted': 1000,\n",
       " ...}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.word_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后我们用分词器去将训练集的文本转换成代号形式的列表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_tokens = tokenizer.texts_to_sequences(x_train_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "举个例子，下面是训练级中的一个文本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A simple comment...<br /><br />What can I say... this is a wonderful film that I can watch over and over. It is definitely one of the top ten comedies made. With a great cast, Jack Lemmon and Walter Matthau wording a perfect script by Neil Simon, based on his play.<br /><br />It is real to life situation done perfectly. If you have digital cable, one gets the menu on bottom of screen to give what is on. It usually gives this film ***% stars but in reality it deserves **** stars. If you really watch this film, one can tell that it will be as funny and fresh a hundred years from now.'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_text[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此文本对应于下列代号形式的列表："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   3,  591,  929,    7,    7,   48,   67,   10,  131,   11,    6,\n",
       "          3,  393,   19,   12,   10,   67,  103,  121,    2,  121,    9,\n",
       "          6,  406,   27,    4,    1,  342,  713, 1317,   90,   16,    3,\n",
       "         78,  174,  694, 4910,    2, 2556, 3599,    3,  399,  227,   31,\n",
       "       4033, 2628,  441,   20,   24,  288,    7,    7,    9,    6,  144,\n",
       "          5,  114,  871,  221,  922,   43,   22,   25, 3639, 1897,   27,\n",
       "        217,    1, 9206,   20, 1306,    4,  258,    5,  197,   48,    6,\n",
       "         20,    9,  631,  411,   11,   19,  405,   18,    8,  614,    9,\n",
       "       1003,  405,   43,   22,   62,  103,   11,   19,   27,   67,  380,\n",
       "         12,    9,   80,   26,   14,  152,    2, 1451,    3, 2997,  153,\n",
       "         36,  146])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(x_train_tokens[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们同样需要将测试集转换成代号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test_tokens = tokenizer.texts_to_sequences(x_test_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 填充和截断数据\n",
    "\n",
    "在循环神经网络可以将任意长度的序列作为输入，但是为了整批地使用数据，序列必须有相同的长度。下面有两种形式可以实现：（A）要么我们确保整个数据的序列的长度是一样的，（B）或我们编写一个自定义数据生成器（data-generator），确保每个批次的序列具有相同的长度。。\n",
    "\n",
    "方案（A)比较简单，但是如果我们使用数据集中最长序列的长度，我们会浪费很多内存。这问题对于较大的数据集尤其重要。\n",
    "\n",
    "所以为了达成妥协，我们将使用一个长度来覆盖据集中的大多数序列的长度，然后我们将截断长的序列和填充短的序列。\n",
    "\n",
    "首先我们计算数据集中所有序列的代号数量。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_tokens = [len(tokens) for tokens in x_train_tokens + x_test_tokens]\n",
    "num_tokens = np.array(num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个序列中代号的平均个数是："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "221.27716"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个序列中具有最多的代号数目是："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2209"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max(num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将允许最大代号数目设置为平均值加上两倍标准差。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "544"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_tokens = np.mean(num_tokens) + 2 * np.std(num_tokens)\n",
    "max_tokens = int(max_tokens)\n",
    "max_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这涵盖了大约95%的数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.94534"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(num_tokens < max_tokens) / len(num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当填充和截断不同长度的序列时，我们需要决定如果我们做的填充或者截断选择的是 'pre'还是 'post'。如果序列被截断，这意味着序列的一部分被简单地扔掉了。如果序列被填充，意味着0被加入到了序列中。\n",
    "\n",
    "所以选测'pre'还是 'post'是很重要的，因为它决定了我们做截断时丢弃的是序列的头还是尾，做填充时填补的0是在序列的头部还是尾部。这可能混淆循环神经网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "pad = 'pre'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_pad = pad_sequences(x_train_tokens, maxlen=max_tokens,\n",
    "                            padding=pad, truncating=pad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test_pad = pad_sequences(x_test_tokens, maxlen=max_tokens,\n",
    "                           padding=pad, truncating=pad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在，我们已经将训练集转换成一个大的整数矩阵（代号）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 544)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_pad.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试集矩阵的形状："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 544)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test_pad.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "举个例子，我们已经有了下面的代号形式的序列："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   3,  591,  929,    7,    7,   48,   67,   10,  131,   11,    6,\n",
       "          3,  393,   19,   12,   10,   67,  103,  121,    2,  121,    9,\n",
       "          6,  406,   27,    4,    1,  342,  713, 1317,   90,   16,    3,\n",
       "         78,  174,  694, 4910,    2, 2556, 3599,    3,  399,  227,   31,\n",
       "       4033, 2628,  441,   20,   24,  288,    7,    7,    9,    6,  144,\n",
       "          5,  114,  871,  221,  922,   43,   22,   25, 3639, 1897,   27,\n",
       "        217,    1, 9206,   20, 1306,    4,  258,    5,  197,   48,    6,\n",
       "         20,    9,  631,  411,   11,   19,  405,   18,    8,  614,    9,\n",
       "       1003,  405,   43,   22,   62,  103,   11,   19,   27,   67,  380,\n",
       "         12,    9,   80,   26,   14,  152,    2, 1451,    3, 2997,  153,\n",
       "         36,  146])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(x_train_tokens[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "它被简单地填充成下面的序列。注意当它被输入循环神经网络，它首先会输入很多0。如果我们采用'post'填充，它会先输入整形代号，然后输入0.这可能混淆循环神经网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    3,  591,  929,    7,    7,   48,   67,   10,\n",
       "        131,   11,    6,    3,  393,   19,   12,   10,   67,  103,  121,\n",
       "          2,  121,    9,    6,  406,   27,    4,    1,  342,  713, 1317,\n",
       "         90,   16,    3,   78,  174,  694, 4910,    2, 2556, 3599,    3,\n",
       "        399,  227,   31, 4033, 2628,  441,   20,   24,  288,    7,    7,\n",
       "          9,    6,  144,    5,  114,  871,  221,  922,   43,   22,   25,\n",
       "       3639, 1897,   27,  217,    1, 9206,   20, 1306,    4,  258,    5,\n",
       "        197,   48,    6,   20,    9,  631,  411,   11,   19,  405,   18,\n",
       "          8,  614,    9, 1003,  405,   43,   22,   62,  103,   11,   19,\n",
       "         27,   67,  380,   12,    9,   80,   26,   14,  152,    2, 1451,\n",
       "          3, 2997,  153,   36,  146], dtype=int32)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_pad[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分词器逆映射\n",
    "\n",
    "由于一些奇怪的原因，Keras的分词器没有实现将整形代号返回到单词的逆映射，当我们从代号列表重建文本字符串时需要用到它，所以我们在这创建这个映射。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = tokenizer.word_index\n",
    "inverse_map = dict(zip(idx.values(), idx.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用于将代号列表转换成字符串的辅助函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokens_to_string(tokens):\n",
    "    # Map from tokens back to words.\n",
    "    words = [inverse_map[token] for token in tokens if token != 0]\n",
    "    \n",
    "    # Concatenate all words.\n",
    "    text = \" \".join(words)\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "举个例子，下面是数据集中的原始文本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A simple comment...<br /><br />What can I say... this is a wonderful film that I can watch over and over. It is definitely one of the top ten comedies made. With a great cast, Jack Lemmon and Walter Matthau wording a perfect script by Neil Simon, based on his play.<br /><br />It is real to life situation done perfectly. If you have digital cable, one gets the menu on bottom of screen to give what is on. It usually gives this film ***% stars but in reality it deserves **** stars. If you really watch this film, one can tell that it will be as funny and fresh a hundred years from now.'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_text[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过将代号列表转换到文字，我们可以重建这个文本（除了标点和其它符号）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a simple comment br br what can i say this is a wonderful film that i can watch over and over it is definitely one of the top ten comedies made with a great cast jack lemmon and walter matthau a perfect script by neil simon based on his play br br it is real to life situation done perfectly if you have digital cable one gets the menu on bottom of screen to give what is on it usually gives this film stars but in reality it deserves stars if you really watch this film one can tell that it will be as funny and fresh a hundred years from now'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_to_string(x_train_tokens[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建循环神经网络\n",
    "\n",
    "我们现在准备创建循环神经网络（RNN），我们将用Keras的API，因为它很简单。Keras的教程详见教程#03-C。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RNN的第一层称为embedding层，它将整形代号转换成一个向量的值。这是必须的，因为整形代号可以取0到10000的值来表示1万字的词汇表。RNN无法接受这么大范围的输入。embedding层被当做RNN的一部分来寻来，它将学习如何将具有类似语义含义的词映射到相似的嵌入向量（embedding-vectors），下面会进一步展示。\n",
    "\n",
    "首先我们为整形代号定义嵌入向量（embedding-vectors）的尺寸。在这个例子中，我们设置为8，所以每个整形代号将会被转成长度为8的向量。嵌入向量（embedding-vectors）的值会逐渐的接近-1.0到1.0，尽管它们可能在某种程度上超过了这个范围。\n",
    "\n",
    "嵌入向量（embedding-vectors）的尺寸通常在100-300之间选择，但它似乎在情绪分析上运行得相当不错。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_size = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "embedding层也需要知道词汇表中词的数量(`num_words`)，和填充的代号序列的长度(`max_tokens`)。我们要给这一层一个名字，因为我们需要获取它的权重。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Embedding(input_dim=num_words,\n",
    "                    output_dim=embedding_size,\n",
    "                    input_length=max_tokens,\n",
    "                    name='layer_embedding'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们现在可以加入第一个GRU到网络中，它会有16个输出单元。因为我们之后会加入第二个GUR，需要返回上数据的序列，作为下一个GRU序列的输入。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/backend.py:1456: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n"
     ]
    }
   ],
   "source": [
    "model.add(GRU(units=16, return_sequences=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里加入第二个GRU，有8个输出单元。因为接下去还有另一层GRU，所以也必须返回序列。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(GRU(units=8, return_sequences=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们加入第三个，也是最后一个GUR，它有4个输出单元。它之后会接一个全连接层，所以它应该给出GRU最后的输出，而不是一个序列。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(GRU(units=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "加入一个全连接层，计算一个0.0到1.0的值，用于分类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用一个给定学习率的Adam优化器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = Adam(lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "编译Keras模型，准备好训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/backend.py:1557: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=optimizer,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "layer_embedding (Embedding)  (None, 544, 8)            80000     \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (None, None, 16)          1200      \n",
      "_________________________________________________________________\n",
      "gru_2 (GRU)                  (None, None, 8)           600       \n",
      "_________________________________________________________________\n",
      "gru_3 (GRU)                  (None, 4)                 156       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 5         \n",
      "=================================================================\n",
      "Total params: 81,961\n",
      "Trainable params: 81,961\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练循环神经网络\n",
    "\n",
    "我们现在可以训练这个模型。注意，我们用的数据集是被填充过的序列。我们用了训练集中5%作为验证集，所以我们会有一个大致的判断，这个模型是具有通用性的还是对训练集过拟合的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 23750 samples, validate on 1250 samples\n",
      "Epoch 1/3\n",
      "23750/23750 [==============================]23750/23750 [==============================] - 464s 20ms/step - loss: 0.6517 - acc: 0.6002 - val_loss: 0.6218 - val_acc: 0.6752\n",
      "\n",
      "Epoch 2/3\n",
      "23750/23750 [==============================]23750/23750 [==============================] - 447s 19ms/step - loss: 0.4292 - acc: 0.8102 - val_loss: 0.6701 - val_acc: 0.6512\n",
      "\n",
      "Epoch 3/3\n",
      "23750/23750 [==============================]23750/23750 [==============================] - 445s 19ms/step - loss: 0.3092 - acc: 0.8765 - val_loss: 0.3182 - val_acc: 0.8752\n",
      "\n",
      "CPU times: user 35min 19s, sys: 2min 41s, total: 38min\n",
      "Wall time: 22min 37s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras._impl.keras.callbacks.History at 0x7ff79f0d6cf8>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "model.fit(x_train_pad, y_train,\n",
    "          validation_split=0.05, epochs=3, batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试集上的性能\n",
    "\n",
    "现在，模型已经被训练好了，我们可以计算测试集上分类的正确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================]25000/25000 [==============================] - 175s 7ms/step\n",
      "\n",
      "CPU times: user 2min 59s, sys: 340 ms, total: 2min 59s\n",
      "Wall time: 2min 55s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "result = model.evaluate(x_test_pad, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 86.71%\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy: {0:.2%}\".format(result[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 错误分类的文本案例\n",
    "\n",
    "为了展示错误分类的文本案例，我们首先计算了测试集中的前1000个文本的预测情绪。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.01 s, sys: 0 ns, total: 7.01 s\n",
      "Wall time: 6.88 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "y_pred = model.predict(x=x_test_pad[0:1000])\n",
    "y_pred = y_pred.T[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这些预测的数字介于0.0到1.0。我们用一个阈值判断，当值大于0.5则被视为1.0，所有小于0.5的值被认为是0.0。我们的预测变成只有0.0或1.0。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_pred = np.array([1.0 if p>0.5 else 0.0 for p in y_pred])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试集中前1000个文本正确的“类”需要拿来做比较。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_true = np.array(y_test[0:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后我们将这两个数组比较，得出不正确的分类的索引。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "incorrect = np.where(cls_pred != cls_true)\n",
    "incorrect = incorrect[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对这1000个文本，有多少个被错误分类？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "121"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(incorrect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们看看第一个被错误分类的文本。我们需要用到它的索引几次。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx = incorrect[0]\n",
    "idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "被错误分类的文本是："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I would like to start by saying I can only hope that the makers of this movie and it\\'s sister film The Intruder (directed by the great unheralded stylist auteur that is Jopi Burnama) know in their hearts just how much pleasure they have brought to me and my friends in the sleepy north eastern town of Jarrow.<br /><br />From the opening pre credit sequence which manages to drag ever so slightly despite containing a man crashing through a window on a motorbike, the pitiless destruction of a silence lab, the introduction of one of the most simultaneously annoying and anaemic bad guys in movie history and costume design that Jean Paul Gautier would find ott and garish. Make no mistake; this is a truly unique experience. Early highlight - an explosion (get used to it, plenty more where that came from!) followed by a close up of our chubby heroine and the most hilarious line reading of the word \"dad\" in living memory. And then... the theme song...<br /><br />Yeah, this deserves its own paragraph. Sung by AJ, written by people who really should wish to remain anonymous, it makes the songs written for the Rocky films sound like Schubert. This is crap 80\\'s hero motivation narcissism at an all time high, with choice lyrics such as \"its only me and you, its come down to the wire\" and much talk of having to \"cross the line\" (it\\'ll make sense in time - our hero cares little for the boundaries of bona fida police work) abounding. Not to mention the Indonesian Supremes cooing the film\\'s title seductively. At this point anyone wishing to switch off officially has no pulse.<br /><br />Our hero is Semitic cop Peter Goldson (essayed brilliantly by Intruder star Peter O\\'Brien), the \"stabilizer\" of the title. The man\\'s bull in a china shop approach to crime fighting and particularly his less than inconspicuous undercover work truly leaves much to be desired, but he is without question an entertaining guide through the mean streets of downtown Jakarta, with local sleaze ball connection Captain Johnny in tow, as well as Peter\\'s own waste of space partner in fashion crime Sylvia Nash, who does little. So many highlights, so little time - the \"slide please\" arrogance of Peter\\'s not all too convincingly argued case against chief baddie Greg Rainmaker (Intruder fans will know hirsute slimy bastard Craig Gavin as the monstrous John White - helluva name eh? No! Oh well...), the x marks the spot location map stupidity, our hero taking horrible advantage of heroine Tina Probost during a moment of weakness on her behalf, the latter turning up at a sting operation dressed like a member of a particularly flamboyant dancing troop. And believe me that barely covers it.<br /><br />There wasn\\'t even time to go into the plot revolving around the hunt for a drug detection system and a kidnapped professor with an alarming but commendable amount of national pride. Or our hero turning up at a funeral dressed as if an extra on Boogie Nights. Or the absolutely hysterical craic between Captain Johnny and Goldson - two guys have never made more heavy weather of buddy buddy shtick than these clowns. The trowel was possibly too subtle me thinks.<br /><br />Ah it tails off people, and you never thought scenes of wanton destruction and general mayhem could be so unbelievably boring, but the character interaction is stupendous, the dialogue truly priceless and the incompetence on show somehow endearing. Oh and the shoes people - watch out for the shoes!'"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = x_test_text[idx]\n",
    "text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面预测和正确的类别："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08332923"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cls_true[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 新的数据\n",
    "\n",
    "让我们试着对我们编造的新文本进行分类。它们有些很明显，而另一些则使用否定和讽刺方式来试图混淆模型，将文本错误分类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = \"This movie is fantastic! I really like it because it is so good!\"\n",
    "text2 = \"Good movie!\"\n",
    "text3 = \"Maybe I like this movie.\"\n",
    "text4 = \"Meh ...\"\n",
    "text5 = \"If I were a drunk teenager then this movie might be good.\"\n",
    "text6 = \"Bad movie!\"\n",
    "text7 = \"Not a good movie!\"\n",
    "text8 = \"This movie really sucks! Can I get my money back please?\"\n",
    "texts = [text1, text2, text3, text4, text5, text6, text7, text8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我们将文本转换成整形代号的数组。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = tokenizer.texts_to_sequences(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了将不同长度的文本输入到模型，我们也需要填充或截断它们。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8, 544)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_pad = pad_sequences(tokens, maxlen=max_tokens,\n",
    "                           padding=pad, truncating=pad)\n",
    "tokens_pad.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们现在可以使用训练好的模型去预测这些文本的情感。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.868934  ],\n",
       "       [0.72526425],\n",
       "       [0.33099633],\n",
       "       [0.49190348],\n",
       "       [0.3054021 ],\n",
       "       [0.14959489],\n",
       "       [0.5235635 ],\n",
       "       [0.21565402]], dtype=float32)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(tokens_pad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "结果仅仅0.0意味着是负面的情感，结果接近1.0意味着是正面的情感。每次你训练模型时，这些数字都会有所不同。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "这个模型不能直接输入整形代号的序列，因为它们是整形的数值，范围从0到词汇表中词的个数，例如10000。所以我们需要将整形的代号转换成一个值处于-1.0到1.0之间的向量，它可以作为神经网络的输入。\n",
    "\n",
    "这个从整形代号转换到实数向量的映射被称为 \"embedding\"。它的本质是一个矩阵，每一行都包含单个代号的矢量映射。这意味着我们可以通过用代号作为矩阵中的索引，快速查找每个整形代号的映射。在训练过程中，embeddings是和模型的其它部分一起学习的。\n",
    "\n",
    "理想的embedding会学习到一个映射，让在意义上相似的词也有相似的嵌入值（embedding-vectors）。让我们看一下是否发生了这种情况。\n",
    "\n",
    "首先我们需要从模型中获得embedding层："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_embedding = model.get_layer('layer_embedding')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以通过embedding层获得用于完成映射的权重。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights_embedding = layer_embedding.get_weights()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意，这些权重实际上只是一个矩阵（词汇表的词数，嵌入矢量长度）。它本质就是一个查找矩阵。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 8)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们获取 'good'的整形代号，就是它在词汇表中的索引。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "49"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_good = tokenizer.word_index['good']\n",
    "token_good"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们获取 'great'的整形代号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "78"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "token_great = tokenizer.word_index['great']\n",
    "token_great"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于依据在数据集中出现的频率，这些整形代号的值可能相差的比较远。\n",
    "\n",
    "现在，让我们比较 'good'和'great'的嵌入向量（embedding-vectors）。其中一些值是相似的，也有些值是完全不同的。注意，每次您对模型进行训练时，这些值都会发生变化。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.86528164, 0.6867993 , 0.4362397 , 0.66128314, 0.11546915,\n",
       "       0.94507647, 0.32628497, 0.535881  ], dtype=float32)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding[token_good]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.0691622 ,  1.124244  , -0.04477464, -0.05861434,  0.16965319,\n",
       "        1.2626944 ,  0.76136374, -0.00998422], dtype=float32)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding[token_great]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同样的，我们也可以比较'bad' 和 'horrible'。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_bad = tokenizer.word_index['bad']\n",
    "token_horrible = tokenizer.word_index['horrible']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.31903917,  0.53934103,  1.3727672 ,  1.4083829 ,  0.8475107 ,\n",
       "       -0.22946651,  0.0251075 ,  0.77032244], dtype=float32)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding[token_bad]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.47915924,  0.12226178,  0.90192014,  0.742338  ,  0.58730644,\n",
       "        0.32736972, -0.17633988,  1.3744307 ], dtype=float32)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_embedding[token_horrible]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 词排序\n",
    "\n",
    "我们也可以将词汇表中的词按“相似性”进行排列。我们想要看看是否具有相似嵌入向量（embedding-vectors）的词也有相似的含义。\n",
    "\n",
    "嵌入向量embedding-vectors的相似性可以通过不同方法度量，比如，欧式距离或者余弦距离。\n",
    "\n",
    "我们设计一个辅助函数用于计算这些距离，然后按序打印出这些词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_sorted_words(word, metric='cosine'):\n",
    "    \"\"\"\n",
    "    Print the words in the vocabulary sorted according to their\n",
    "    embedding-distance to the given word.\n",
    "    Different metrics can be used, e.g. 'cosine' or 'euclidean'.\n",
    "    \"\"\"\n",
    "\n",
    "    # Get the token (i.e. integer ID) for the given word.\n",
    "    token = tokenizer.word_index[word]\n",
    "\n",
    "    # Get the embedding for the given word. Note that the\n",
    "    # embedding-weight-matrix is indexed by the word-tokens\n",
    "    # which are integer IDs.\n",
    "    embedding = weights_embedding[token]\n",
    "\n",
    "    # Calculate the distance between the embeddings for\n",
    "    # this word and all other words in the vocabulary.\n",
    "    distances = cdist(weights_embedding, [embedding],\n",
    "                      metric=metric).T[0]\n",
    "    \n",
    "    # Get an index sorted according to the embedding-distances.\n",
    "    # These are the tokens (integer IDs) for words in the vocabulary.\n",
    "    sorted_index = np.argsort(distances)\n",
    "    \n",
    "    # Sort the embedding-distances.\n",
    "    sorted_distances = distances[sorted_index]\n",
    "    \n",
    "    # Sort all the words in the vocabulary according to their\n",
    "    # embedding-distance. This is a bit excessive because we\n",
    "    # will only print the top and bottom words.\n",
    "    sorted_words = [inverse_map[token] for token in sorted_index\n",
    "                    if token != 0]\n",
    "\n",
    "    # Helper-function for printing words and embedding-distances.\n",
    "    def _print_words(words, distances):\n",
    "        for word, distance in zip(words, distances):\n",
    "            print(\"{0:.3f} - {1}\".format(distance, word))\n",
    "\n",
    "    # Number of words to print from the top and bottom of the list.\n",
    "    k = 10\n",
    "\n",
    "    print(\"Distance from '{0}':\".format(word))\n",
    "\n",
    "    # Print the words with smallest embedding-distance.\n",
    "    _print_words(sorted_words[0:k], sorted_distances[0:k])\n",
    "\n",
    "    print(\"...\")\n",
    "\n",
    "    # Print the words with highest embedding-distance.\n",
    "    _print_words(sorted_words[-k:], sorted_distances[-k:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后我们可以打印出距离'great'从近到远的词。请注意，每次您对模型进行训练时，这些都可能发生变化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distance from 'great':\n",
      "0.000 - great\n",
      "0.016 - touching\n",
      "0.017 - arguments\n",
      "0.025 - nevertheless\n",
      "0.031 - elmer\n",
      "0.032 - 8\n",
      "0.036 - ritter\n",
      "0.037 - juliet\n",
      "0.041 - randy\n",
      "0.045 - afterward\n",
      "...\n",
      "1.057 - rubbish\n",
      "1.060 - dull\n",
      "1.064 - disappointing\n",
      "1.069 - unlikeable\n",
      "1.078 - uninspired\n",
      "1.083 - lacks\n",
      "1.188 - worst\n",
      "1.225 - waste\n",
      "1.247 - awful\n",
      "1.282 - terrible\n"
     ]
    }
   ],
   "source": [
    "print_sorted_words('great', metric='cosine')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相似的，我们也可以打印出距离 'worst'从近到远的词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distance from 'worst':\n",
      "0.000 - worst\n",
      "0.047 - embarrassingly\n",
      "0.053 - terrible\n",
      "0.094 - retarded\n",
      "0.095 - poor\n",
      "0.095 - stereotyping\n",
      "0.096 - uninspired\n",
      "0.099 - awful\n",
      "0.100 - severed\n",
      "0.108 - lacks\n",
      "...\n",
      "1.167 - restraint\n",
      "1.168 - available\n",
      "1.176 - foremost\n",
      "1.188 - great\n",
      "1.193 - mesmerizing\n",
      "1.222 - highly\n",
      "1.229 - exploration\n",
      "1.239 - delightful\n",
      "1.268 - wonderfully\n",
      "1.323 - 7\n"
     ]
    }
   ],
   "source": [
    "print_sorted_words('worst', metric='cosine')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "\n",
    "这份教程展示了使用了带有整形代号（integer-tokens）和embedding层的循环神经网络进行自然语言处理(NLP)的基本方法。用于IMDB影评的情感分析。如果超参数选择合适，它工作的相当好。但是重要的是明白它并不是像人一样去理解文本。这个系统并不能真正的理解文本。这只是一种巧妙的模式识别方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 练习\n",
    "\n",
    "下面使一些可能会让你提升TensorFlow技能的一些建议练习。为了学习如何更合适地使用TensorFlow，实践经验是很重要的。\n",
    "\n",
    "在你对这个Notebook进行修改之前，可能需要先备份一下。\n",
    "\n",
    "* 训练的更久。性能是否提高了？\n",
    "* 如果你的模型对训练集过拟合，尝试使用dropout层并在GRU中也设置dropout。\n",
    "* 增加或减少词汇表中词汇的数量。这是当`Tokenizer`初始化时完成的。是否影响性能？\n",
    "* 增加嵌入向量（ embedding-vectors ）的尺寸，例如 200.是否影响性能？\n",
    "* 尝试给循环神经网络不同的超参数。\n",
    "* 利用教程 #19 中的贝叶斯最优化去找到最好超参数。\n",
    "* 在 `pad_sequences()`中选择'post'完成填充和截断。是否影响性能？\n",
    "* 使用单独的字符代替记号化单词作为词汇表。然后你可以为每个字符使用一个热编码的向量，代替使用embedding层。\n",
    "* 使用`model.fit_generator()` 代替`model.fit()`并制作你自己的数据生成器（data-generator），它可以用 `x_train_tokens`的随机子集作为一个批，这些序列必须被填充，所以它们都与最长的序列相匹配。\n",
    "* 向朋友解释程序如何工作。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License (MIT)\n",
    "\n",
    "Copyright (c) 2018 by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}