lm2-french.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "e38c0aad-6ca2-456d-b908-cbbec3c9e34d"
    }
   },
   "source": [
    "# French Bidirectional Language Model (LM) from scratch\n",
    "### (architecture QRNN, SentenPiece tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Author: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou)\n",
    "- Date: September 2019\n",
    "- Post in medium: [link](https://medium.com/@pierre_guillou/nlp-fastai-french-language-model-d0e2a9e12cab)\n",
    "- Ref: [Fastai v1](https://docs.fast.ai/) (Deep Learning library on PyTorch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Information**\n",
    "\n",
    "According to this new article \"[MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://arxiv.org/abs/1909.04761)\" (September 10, 2019), the architecture QRNN and the SentencePiece tokenizer give better results than AWD-LSTM and the spaCy tokenizer respectively. Therefore, they have been used in this notebook to train a French Bidirectional Language Model on a Wikipedia corpus of 100 millions tokens. \n",
    "\n",
    "More, the hyperparameters values given at the end of the article have been used, too.\n",
    "\n",
    "**Wikipedia corpus**\n",
    "- download: 512 659 articles of 492 596 078 tokens tokens\n",
    "- used: 252 898 articles of 100 716 190 tokens\n",
    "\n",
    "**Hyperparameters values**\n",
    "- (batch size) bs = 50\n",
    "- (QRNN) 3 QRNN (default: 3) with 1152 hidden parameters each one (default: 1152)\n",
    "- (SentencePiece) vocab of 15000 tokens\n",
    "- (dropout) mult_drop = 0\n",
    "- (weight decay) wd = 0.01\n",
    "- (number of training epochs) 10 epochs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy and Perplexity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- forward : (accuracy) 40.99% | (perplexity) 19.96\n",
    "- backward: (accuracy) 47.19% | (perplexity) 19.47"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To be improved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The French Bidirectional Language Model should be retrained with the MultiFiT hyperparameters values like 4 QRNN (and not 3 AWD-LSTM), 1550 hidden parameters by layer (and not 1152), no dropout, batch size of 50, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Initialisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "151cd18f-76e3-440f-a8c7-ffa5c6b5da01"
    }
   },
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.callbacks import *\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "96f02439-3586-4c9d-8c34-aa7c3b17a0a6"
    }
   },
   "outputs": [],
   "source": [
    "# batch size to be choosen according to your GPU \n",
    "# bs=48\n",
    "# bs=24\n",
    "bs=50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6ceb4db2-e4cf-4fe0-a393-91df4a7ed3e7"
    }
   },
   "outputs": [],
   "source": [
    "torch.cuda.set_device(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6329e650-fc03-4323-ac0c-3aa280e0de91"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fastai: 1.0.57\n",
      "cuda: True\n"
     ]
    }
   ],
   "source": [
    "import fastai\n",
    "print(f'fastai: {fastai.__version__}')\n",
    "print(f'cuda: {torch.cuda.is_available()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6f24e68b-3df0-4997-8a50-3a37ea6a5257"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\r\n",
      "```text\r\n",
      "=== Software === \r\n",
      "python        : 3.7.4\r\n",
      "fastai        : 1.0.57\r\n",
      "fastprogress  : 0.1.21\r\n",
      "torch         : 1.2.0\r\n",
      "nvidia driver : 410.104\r\n",
      "torch cuda    : 10.0.130 / is available\r\n",
      "torch cudnn   : 7602 / is enabled\r\n",
      "\r\n",
      "=== Hardware === \r\n",
      "nvidia gpus   : 1\r\n",
      "torch devices : 1\r\n",
      "  - gpu0      : 16130MB | Tesla V100-SXM2-16GB\r\n",
      "\r\n",
      "=== Environment === \r\n",
      "platform      : Linux-4.9.0-9-amd64-x86_64-with-debian-9.9\r\n",
      "distro        : #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11)\r\n",
      "conda env     : base\r\n",
      "python        : /opt/anaconda3/bin/python\r\n",
      "sys.path      : /home/jupyter/tutorials/fastai/course-nlp\r\n",
      "/opt/anaconda3/lib/python37.zip\r\n",
      "/opt/anaconda3/lib/python3.7\r\n",
      "/opt/anaconda3/lib/python3.7/lib-dynload\r\n",
      "/opt/anaconda3/lib/python3.7/site-packages\r\n",
      "/opt/anaconda3/lib/python3.7/site-packages/IPython/extensions\r\n",
      "```\r\n",
      "\r\n",
      "Please make sure to include opening/closing ``` when you paste into forums/github to make the reports appear formatted as code sections.\r\n",
      "\r\n",
      "Optional package(s) to enhance the diagnostics can be installed with:\r\n",
      "pip install distro\r\n",
      "Once installed, re-run this utility to get the additional information\r\n"
     ]
    }
   ],
   "source": [
    "!python -m fastai.utils.show_install"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "194a6989-31f1-4702-b94d-32b974ded8e6"
    }
   },
   "outputs": [],
   "source": [
    "data_path = Config.data_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "cf070ab7-babb-4cf0-a315-401f65461dc8"
    }
   },
   "source": [
    "This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents. (For other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "70da588b-8af1-4f97-97c2-c9f2d4d46e1a"
    }
   },
   "outputs": [],
   "source": [
    "lang = 'fr'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "701ab344-0430-4f43-bbe2-337a12cae6be"
    }
   },
   "outputs": [],
   "source": [
    "name = f'{lang}wiki'\n",
    "path = data_path/name\n",
    "path.mkdir(exist_ok=True, parents=True)\n",
    "\n",
    "lm_fns2 = [f'{lang}_wt_sp15', f'{lang}_wt_vocab_sp15']\n",
    "lm_fns2_bwd = [f'{lang}_wt_sp15_bwd', f'{lang}_wt_vocab_sp15_bwd']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "nbpresent": {
     "id": "bfe49910-58e0-4be3-aba1-7733dc18cca2"
    }
   },
   "source": [
    "## Data (French wikipedia)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "nbpresent": {
     "id": "4e67d876-c7d0-4bdf-a6f9-ae06ae1fc023"
    }
   },
   "source": [
    "### Download data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "dd2fd658-b690-484c-b60a-69dc6b7bf384"
    }
   },
   "outputs": [],
   "source": [
    "from nlputils import split_wiki,get_wiki\n",
    "from nlputils2 import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "28c01920-f13c-493e-9a97-e5b2c24133a8"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "extracting...\n",
      "CPU times: user 52 ms, sys: 12 ms, total: 64 ms\n",
      "Wall time: 17min 30s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "get_wiki(path,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "100000\n",
      "200000\n",
      "300000\n",
      "400000\n",
      "500000\n",
      "600000\n",
      "700000\n",
      "800000\n",
      "900000\n",
      "1000000\n",
      "1100000\n",
      "1200000\n",
      "1300000\n",
      "1400000\n",
      "1500000\n",
      "1600000\n",
      "1700000\n",
      "1800000\n",
      "1900000\n",
      "2000000\n",
      "2100000\n",
      "2200000\n",
      "2300000\n",
      "2400000\n",
      "2500000\n",
      "2600000\n",
      "2700000\n",
      "2800000\n",
      "2900000\n",
      "3000000\n",
      "3100000\n",
      "3200000\n",
      "3300000\n",
      "3400000\n",
      "3500000\n",
      "3600000\n",
      "3700000\n",
      "3800000\n",
      "3900000\n",
      "4000000\n",
      "4100000\n",
      "4200000\n",
      "4300000\n",
      "4400000\n",
      "4500000\n",
      "4600000\n",
      "4700000\n",
      "4800000\n",
      "4900000\n",
      "5000000\n",
      "5100000\n",
      "5200000\n",
      "5300000\n",
      "5400000\n",
      "5500000\n",
      "5600000\n",
      "5700000\n",
      "5800000\n",
      "5900000\n",
      "6000000\n",
      "6100000\n",
      "6200000\n",
      "6300000\n",
      "6400000\n",
      "6500000\n",
      "6600000\n",
      "6700000\n",
      "6800000\n",
      "6900000\n",
      "7000000\n",
      "7100000\n",
      "7200000\n",
      "7300000\n",
      "7400000\n",
      "7500000\n",
      "7600000\n",
      "7700000\n",
      "7800000\n",
      "7900000\n",
      "8000000\n",
      "8100000\n",
      "8200000\n",
      "8300000\n",
      "8400000\n",
      "8500000\n",
      "8600000\n",
      "8700000\n",
      "8800000\n",
      "8900000\n",
      "9000000\n",
      "9100000\n",
      "9200000\n",
      "9300000\n",
      "9400000\n",
      "9500000\n",
      "9600000\n",
      "9700000\n",
      "9800000\n",
      "9900000\n",
      "10000000\n",
      "10100000\n",
      "10200000\n",
      "10300000\n",
      "10400000\n",
      "10500000\n",
      "10600000\n",
      "10700000\n",
      "10800000\n",
      "10900000\n",
      "11000000\n",
      "11100000\n",
      "11200000\n",
      "11300000\n",
      "11400000\n",
      "11500000\n",
      "11600000\n",
      "11700000\n",
      "11800000\n",
      "11900000\n",
      "12000000\n",
      "12100000\n",
      "12200000\n",
      "12300000\n",
      "12400000\n",
      "12500000\n",
      "12600000\n",
      "12700000\n",
      "12800000\n",
      "12900000\n",
      "13000000\n",
      "13100000\n",
      "13200000\n",
      "13300000\n",
      "13400000\n",
      "13500000\n",
      "13600000\n",
      "13700000\n",
      "13800000\n",
      "13900000\n",
      "14000000\n",
      "14100000\n",
      "14200000\n",
      "14300000\n",
      "14400000\n",
      "14500000\n",
      "14600000\n",
      "14700000\n",
      "14800000\n",
      "14900000\n",
      "15000000\n",
      "15100000\n",
      "15200000\n",
      "15300000\n",
      "15400000\n",
      "15500000\n",
      "15600000\n",
      "15700000\n",
      "15800000\n",
      "15900000\n",
      "16000000\n",
      "16100000\n",
      "16200000\n",
      "16300000\n",
      "16400000\n",
      "16500000\n",
      "16600000\n",
      "16700000\n",
      "16800000\n",
      "16900000\n",
      "17000000\n",
      "17100000\n",
      "17200000\n",
      "17300000\n",
      "17400000\n",
      "17500000\n",
      "17600000\n",
      "17700000\n",
      "17800000\n",
      "17900000\n",
      "18000000\n",
      "18100000\n",
      "18200000\n",
      "18300000\n",
      "18400000\n",
      "18500000\n",
      "18600000\n",
      "18700000\n",
      "18800000\n",
      "18900000\n",
      "19000000\n",
      "19100000\n",
      "19200000\n",
      "19300000\n",
      "19400000\n",
      "19500000\n",
      "19600000\n",
      "19700000\n",
      "19800000\n",
      "19900000\n",
      "20000000\n",
      "20100000\n",
      "20200000\n",
      "20300000\n",
      "20400000\n",
      "20500000\n",
      "CPU times: user 37.7 s, sys: 14.8 s, total: 52.6 s\n",
      "Wall time: 1min 19s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/jupyter/.fastai/data/frwiki/docs')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "split_wiki(path,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e6eae780-775e-45e9-9b99-b8a87d5fb8ff"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/frwiki/frwiki-latest-pages-articles.xml'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/frwiki'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/wikiextractor'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/frwiki-latest-pages-articles.xml.bz2'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/log')]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e1ac63e7-1cbb-4996-838d-dc58446a65ef"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<doc id=\"3\" url=\"https://fr.wikipedia.org/wiki?curid=3\" title=\"Antoine Meillet\">\r\n",
      "Antoine Meillet\r\n",
      "\r\n",
      "Paul Jules Antoine Meillet, né le à Moulins (Allier) et mort le à Châteaumeillant (Cher), est le principal linguiste français des premières décennies du . Il est aussiphilologue.\r\n"
     ]
    }
   ],
   "source": [
    "!head -n4 {path}/{name}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "ae770e72-e7a9-473d-8454-2020a0263be8"
    }
   },
   "source": [
    "This function splits the single wikipedia file into a separate file per article. This is often easier to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "d23e0ef7-21e5-4cc5-945d-60ee33c02ce3"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 31s, sys: 28.7 s, total: 2min\n",
      "Wall time: 8min 37s\n"
     ]
    }
   ],
   "source": [
    "# %%time\n",
    "# folder = \"docs\"\n",
    "# clean_files(path,folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "92b0b087-a6a8-403a-a7a1-d9b47757e5cf"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Min_y_.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Darius_Johnson_Odom.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Guillaume_Cherel.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Henk_Badings.txt'),\n",
       " PosixPath('/home/jupyter/.fastai/data/frwiki/docs/Henri_de_Virel.txt')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dest = path/'docs'\n",
    "dest.ls()[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Darius Earvin Johnson-Odom, né le , à Youngsville, en Caroline du Nord, est un joueur américain de basket-ball. Il évolue au poste d'arrière.\r\n",
      "\r\n",
      "Johnson-Odom passe trois saisons à l'université de Marquette avant d'être sélectionné à la de la draft 2012 de la NBA par les Mavericks de Dallas, qui le transfèrent immédiatement aux Lakers de Los Angeles. Le 15 septembre 2012, il signe son contrat rookie avec les Lakers. Johnson-Odom est envoyé, plusieurs fois durant la saison 2012-2013, chez les D-Fenders de Los Angeles, l'équipe de D-League affiliée aux Lakers.\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n4 {dest}/'Darius_Johnson_Odom.txt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "nbpresent": {
     "id": "575fa672-7b3a-4238-923f-ec929d3a00ee"
    }
   },
   "source": [
    "### Size of downloaded data in the docs folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "270470c2-e0eb-446a-9654-de6c45bc4f0d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "512659 files - 492596078 tokens\n",
      "CPU times: user 57.1 s, sys: 18.9 s, total: 1min 16s\n",
      "Wall time: 2min 49s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "num_files, num_tokens = get_num_tokens(dest)\n",
    "print(f'{num_files} files - {num_tokens} tokens')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true,
    "nbpresent": {
     "id": "daae36a0-d90b-45ad-b0e3-a2cd56ce7079"
    }
   },
   "source": [
    "### Create a corpus of about 100 millions of tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "6c383e0e-f4f6-46e5-9f54-469437d66f07"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "files copied to the new corpus folder: /home/jupyter/.fastai/data/frwiki/corpus_100000000\n",
      "CPU times: user 13.3 s, sys: 8.85 s, total: 22.1 s\n",
      "Wall time: 22.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "path_corpus = get_corpus(dest, path, num_tokens, obj_tokens=1e8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "b597308e-9521-4637-9e84-85a6cd2ace85"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "252898 files - 100716190 tokens\n",
      "CPU times: user 13.9 s, sys: 2.92 s, total: 16.8 s\n",
      "Wall time: 2min 9s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# VERIFICATION of the number of words in the corpus folder\n",
    "num_files_corpus, num_tokens_corpus = get_num_tokens(path_corpus)\n",
    "print(f'{num_files_corpus} files - {num_tokens_corpus} tokens')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# change name of the corpus \n",
    "!mv {path}/'corpus_100000000' {path}/'corpus2_100'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "dest = path/'corpus2_100'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 41min 54s, sys: 20.3 s, total: 42min 14s\n",
      "Wall time: 20min 14s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = (TextList.from_folder(dest, processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "        .split_by_rand_pct(0.1, seed=42)\n",
    "        .label_for_lm()\n",
    "        .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data.save(f'{path}/{lang}_databunch_corpus2_100_sp15')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15000, 227609)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.vocab.itos),len(data.train_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15000, 15000)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.vocab.itos),len(data.vocab.stoi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text ▁xxbos ▁xxmaj ▁dar ius ▁xxmaj ▁ e ar vin ▁johnson - o dom , ▁né ▁le ▁ , ▁à ▁xxmaj ▁young s ville , ▁en ▁xxmaj ▁caroline ▁du ▁xxmaj ▁nord , ▁est ▁un ▁joueur ▁américain ▁de ▁basket - ball . ▁xxmaj ▁il ▁évolue ▁au ▁poste ▁d ' arrière . ▁johnson - o dom ▁passe ▁trois ▁saisons ▁à ▁l ' université ▁de ▁xxmaj ▁marque tte ▁avant ▁d ' être ▁sélectionné ▁à ▁la ▁de ▁la ▁draft ▁2012 ▁de ▁la ▁xxup ▁nba ▁par ▁les ▁xxmaj ▁ma ve rick s ▁de ▁xxmaj ▁dallas , ▁qui ▁le ▁trans f èrent ▁immédiatement ▁aux ▁xxmaj ▁la ker s ▁de ▁xxmaj ▁los ▁xxmaj ▁angeles . ▁xxmaj ▁le ▁15 ▁septembre ▁2012, ▁il ▁signe ▁son ▁contrat ▁ro ok ie ▁avec ▁les ▁xxmaj ▁la ker s . ▁johnson - o dom ▁est ▁envoyé , ▁plusieurs ▁fois ▁durant ▁la ▁saison ▁2012-2013 , ▁chez ▁les ▁d - f en der s ▁de ▁xxmaj ▁los ▁xxmaj ▁angeles , ▁l ' équipe ▁de ▁d - le a gue ▁affilié e ▁aux ▁xxmaj ▁la ker s . ▁xxmaj ▁le ▁7 ▁janvier ▁2013, ▁johnson - o dom ▁est ▁coupé ▁par ▁les ▁xxmaj ▁la ker s . ▁xxmaj ▁c ' était ▁le ▁dernier ▁jour ▁pour ▁les ▁équipes ▁xxup ▁nba ▁pour ▁libérer ▁des ▁joueurs ▁dont ▁le ▁contrat ▁n ' est ▁pas ▁garanti ▁avant ▁que ▁leur ▁contrat ▁de vienne ▁garanti ▁jusqu ' à ▁la ▁fin ▁de ▁la ▁saison . ▁xxmaj ▁il ▁joue ▁quatre ▁matches ▁et ▁seulement ▁six ▁minutes ▁au ▁total ▁avec ▁les ▁xxmaj ▁la ker s , ▁passant ▁la ▁plupart ▁de ▁son ▁temps ▁en ▁d - le a gue ▁où ▁il ▁est ▁le ▁meilleur ▁marque ur ▁des ▁d - f en der s , ▁avec ▁21 ▁points ▁par ▁match . ▁xxmaj ▁le ▁24 ▁janvier ▁2013, ▁johnson - o dom ▁part ▁en ▁xxmaj ▁russie ▁où ▁il ▁signe ▁avec ▁le ▁xxmaj ▁ sparta k ▁saint - pétersbourg ▁pour ▁le ▁reste ▁de ▁la ▁saison ▁2012-2013 . ▁xxmaj ▁il ▁participe ▁avec ▁les ▁xxmaj ▁celtic s ▁de ▁xxmaj ▁boston ▁à ▁la ▁xxmaj ▁summer ▁xxmaj ▁league ▁2013 ▁d ' orlando . ▁xxmaj ▁le ▁25 ▁septembre ▁2013, ▁il ▁rejoint ▁les ▁xxmaj ▁la ker s ▁de ▁xxmaj ▁los ▁xxmaj ▁angeles ▁pour ▁participer ▁au ▁camp ▁d ' entraînement . ▁xxmaj ▁toutefois , ▁ils ▁le ▁libère nt ▁le ▁16 ▁octobre . ▁xxmaj ▁le ▁18 ▁octobre ▁2013, ▁il ▁signe ▁en ▁xxmaj ▁chine ▁au ▁ . ▁xxmaj ▁en ▁novembre ▁2013, ▁après ▁quatre ▁matches ▁de ▁saison ▁régulière , ▁il ▁quitte ▁les ▁xxmaj ▁blue ▁xxmaj ▁wh ales . ▁xxmaj ▁le ▁3 ▁janvier ▁2014, ▁il ▁rejoint ▁l ' armor ▁de ▁xxmaj ▁ spring field ▁en ▁d - le a gue . ▁xxmaj ▁le ▁14 ▁mars ▁2014, ▁il ▁signe ▁un ▁contrat ▁de ▁dix ▁jours ▁avec ▁les ▁ 76 ers ▁de ▁xxmaj ▁philadelphie . ▁xxmaj ▁le ▁24 ▁mars ▁2014, ▁il ▁ne ▁signe ▁pas ▁de ▁second ▁contrat ▁de ▁dix ▁jours ▁à ▁la ▁fin ▁du ▁premier . ▁johnson - o dom ▁détient ▁actuellement ▁le ▁record ▁du ▁plus ▁grand ▁nombre ▁de ▁tirs ▁tenté s ▁sans ▁en ▁réussi r ▁un ▁seul ▁en ▁xxup ▁nba ▁avec ▁11 ▁tirs . ▁xxmaj ▁le ▁2 ▁août ▁2014, ▁il ▁signe ▁en ▁xxmaj ▁italie , ▁au ▁xxmaj ▁c ant ù ▁pour ▁la ▁saison ▁2014-2015 . ▁xxmaj ▁le ▁11 ▁décembre ▁2014, ▁il ▁est ▁nommé ▁xxup ▁m v p ▁de ▁la ▁de ▁l ' euro coup e . ▁xxmaj ▁le ▁14 ▁juin ▁2015, ▁il ▁signe ▁en ▁xxmaj ▁turquie ▁au ▁xxmaj ▁tra b zon s por ▁xxmaj ▁basket ball ▁pour ▁la ▁saison ▁2015-2016 . ▁xxmaj ▁le ▁28 ▁décembre ▁2015, ▁il ▁signe ▁en ▁xxmaj ▁grèce , ▁à ▁l ' olympia k ó s ▁pour ▁le ▁reste ▁de ▁la ▁saison ▁2015-2016 ▁et ▁le ▁xxmaj ▁top ▁16 ▁de ▁l ' euro ligu e . ▁xxmaj ▁le ▁11 ▁juin ▁2016, ▁il ▁retourne ▁en ▁xxmaj ▁italie ▁où ▁il ▁signe ▁à ▁xxmaj ▁sa s s ari . ▁xxmaj ▁les ▁statistiques ▁en ▁matchs ▁universitaires ▁de ▁xxmaj ▁dar ius ▁johnson - o dom ▁sont ▁les ▁suivantes ▁:"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.train_ds.x[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Backward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 42min 18s, sys: 23.5 s, total: 42min 42s\n",
      "Wall time: 20min 30s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = (TextList.from_folder(dest, processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "            .split_by_rand_pct(0.1, seed=42)\n",
    "            .label_for_lm()\n",
    "            .databunch(bs=bs, num_workers=1, backwards=True))\n",
    "\n",
    "data.save(f'{path}/{lang}_databunch_corpus2_100_sp15_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "nbpresent": {
     "id": "052df7c2-f57b-4596-9189-c9e01102e5e9"
    }
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Forward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "defab6d9-ba04-4943-b574-9300e20d5e1c"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.94 s, sys: 1.81 s, total: 5.75 s\n",
      "Wall time: 12.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = load_data(path, f'{lang}_databunch_corpus2_100_sp15', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 248 ms, sys: 36 ms, total: 284 ms\n",
      "Wall time: 284 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, drop_mult=0., pretrained=False, \n",
    "                               metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e74f8988-7c26-495b-90fb-0da012f54c1a"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "e0333edf-f5cd-4c7d-9db3-01dfb7cc4bb8"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5xU9b3/8ddnO+zCUnZBBAEpUgRBWHvs0dhrih37wxJN9Of1JjddUzTxmub1EgRbjCZRUzSxcU3sAoIKohTposDuUndYtn9+f8xZGNZZWJaZOTO77+fjcR47p8yc9w7Dfuac7znfr7k7IiIiLWWFHUBERNKTCoSIiMSlAiEiInGpQIiISFwqECIiEldO2AESpaSkxAcPHhx2DBGRjDJnzpxKdy+Nt67DFIjBgwcze/bssGOIiGQUM1vZ2jqdYhIRkbhUIEREJC4VCBERiUsFQkRE4lKBEBGRuJJWIMzsQTMrN7P5Mct6mdl0M/s4+Nmzlec2mtn7wfRMsjKKiEjrknkE8TBwSotl3wJedvfhwMvBfDzb3H18MJ2VxIwiItKKpBUId38N2NBi8dnAI8HjR4BzkrV/EZHO4Ok5q3li1qqkvHaq2yD6uvsagOBnn1a2KzCz2WY2w8xUREREWvGX91bz9JzVSXntdL2TeqC7f2ZmQ4B/mdkH7r605UZmdi1wLcDAgQNTnVFEJHSRmgZ6FuYl5bVTfQSxzsz6AQQ/y+Nt5O6fBT+XAa8AB7ey3RR3L3P3stLSuF2JiIh0aFW1DRTlJ+e7fqoLxDPApODxJODvLTcws55mlh88LgGOAj5KWUIRkQwSqWmgW0GGFQgzewJ4GxhhZqvN7CrgLuAkM/sYOCmYx8zKzGxq8NRRwGwzmwv8G7jL3VUgRETiiNQ2UJiXnAKRtDYId7+wlVUnxtl2NnB18PgtYGyycomIdBSNTU51XSNFmXYEISIiybW1rgGgw7RBiIhIgkRqogUi49ogREQkuSK1zUcQuUl5fRUIEZEMVRUcQagNQkREdrLjCEIFQkREYqgNQkRE4orU1gM6ghARkRaa2yAKVSBERCSW2iBERCSuSE0DXfOyyc6ypLy+CoSISIbaWpe8nlxBBUJEJGNV1TQk7R4IUIEQEclYkdoGuukIQkREWoroCEJEROKJJHE0OVCBEBHJWFU1DUnrqA9UIEREMlakNnnDjYIKhIhIRnL36HCj+dlJ24cKhIhIBqqpb6KxyXWKSUREdlbV3FGfTjGJiEisrbWNALoPQkREdtY8FoQucxURkZ3oFJOIiMSlIwgREYmreSwI3QchIiI7SfZgQaACISKSkZI93CioQIiIZKRIbQO52UZ+TvL+jKtAiIhkoEhNtCdXs+QMNwoqECIiGSlSm9yxIEAFQkQkI0XHgkheP0ygAiEikpEiNckdbhRUIEREMpJOMYmISFzJHm4UVCBERDJSVY2OIEREJI5Ibb3aIEREZGf1jU3U1Dcl9S5qSGKBMLMHzazczObHLOtlZtPN7OPgZ89Wnjsp2OZjM5uUrIwiIploawr6YYLkHkE8DJzSYtm3gJfdfTjwcjC/EzPrBfwAOAw4FPhBa4VERKQzau6HKWPbINz9NWBDi8VnA48Ejx8Bzonz1C8B0919g7tvBKbz+UIjItJpbe/qO4OPIOLp6+5rAIKffeJs0x/4JGZ+dbDsc8zsWjObbWazKyoqEh5WRCQdbT/FlKlHEHshXs9THm9Dd5/i7mXuXlZaWprkWCIi6aGqA7RBxLPOzPoBBD/L42yzGtgvZn4A8FkKsomIZITm4UaTOZocpL5APAM0X5U0Cfh7nG1eBE42s55B4/TJwTIRESF2NLkM7azPzJ4A3gZGmNlqM7sKuAs4ycw+Bk4K5jGzMjObCuDuG4A7gXeC6Y5gmYiIsOMIItltEEl7dXe/sJVVJ8bZdjZwdcz8g8CDSYomIpLRmtsguuZmJ3U/6dhILSIiu9A8mlxWVvJGkwMVCBGRjBOprU/6FUygAiEiknFSMRYEqECIiGScqprkjwUBKhAiIhlna21D0u+BABUIEZGMk4rR5EAFQkQk40R0iklEROKpUiO1iIi05O46xSQiIp9XXdeIe/J7cgUVCBGRjBJJ0VgQoAIhIpJRtg83qiMIERGJtX24UR1BiIhIrO1dfSd5LAhQgRARySiRFA03CioQIiIZZWN1HQDFXXUEISIiMSqragEoKcpL+r5UIEREMkhFpJbuBTnk5yR3NDlQgRARySiVkVpKu+WnZF8qECIiGaSyqo6SIhUIERFpoSJSS4mOIEREpKXKqlpKdQQhIiKxauobqaptUBuEiIjsrCKFl7iCCoSISMaojEQLhI4gRERkJ5WR6F3UuopJRER2suMUkwqEiIjEaD7F1DtFbRDJ7w4wzTU2+faq3MzxlOzbsJ3nrZUN4z63tZnPv27sa7dcY8EK22mbnTeOfW5r21vMtllm27dt3s725JcTkbgqI7UUd8lNSTcboALBpuo6Dv/Zy2HH6BRii0eW2fb5LIOsLCPLjOzgZ5ZBdpbtmGzH45xsIzsri9zgcU5WFrnZRl5OFrnZWeRlZ5Gfm01+ThYFudkU5GbRNS+bLnk5FOZlU5SfQ3GXXIq75lLcJZeSonxys3UwLemvoqo2ZVcwgQoEhfk5/Oy8sZ9bnuzvuy2PUXwPDlpij3BaPi/uywQbtbZP9x2vuH0ZO9Z9bnt8p+3cd17W1OSfW+7BazX5jvnm7RqbosubmpxGd5o8uq6xeb7JaQi2aWiMLq9vchoam2hodLY2NNDQ6NQ3NlHX0ETt9qmR2vom6hqbdvl+mkHvwjz6dCugf88uDC0tYmhpIcP6FDGqX3cKclPzbU1kd1LZDxOoQFCQm82Fhw4MO4YkUWOTs62+keraBrbWNRKpaWBLTT2bt9Wzqbqe8qoa1m2ppXxLDSsqt/LKonLqG6PVLj8ni0MG9+KoYSUce0Apo/ftHvJvI51ZZaSOA1P4Gez0BUI6vuwsoyg/p80jcDU0NrFqQzWL10WYuXw9by1Zz90vLOTuFxZy+JBe3Hj8ML4wrETtKpJy0VNMOoIQCU1OdhZDSosYUlrEKWP2AaC8qoZn3v+Mqa8v59JpszhoQDG3f2kkXxheEnJa6Sxq6huJpLCbDdBlriJt0qdbAVcfPYRXbz+Ou84by6bqei59cCb3/evjndppRJKl+WrLVHXUByoQInskPyebCw4dyIvfPIYzD9qXe15azA1/eJetwUDyIslSEdwDUdItdVcxqUCItEOXvGx+fcF4vnv6KF78cC3n3v8mqzdWhx1LOrDK7UcQBSnbZygFwsy+YWbzzexDM/tmnPXHmdlmM3s/mL4fRk6RXTEzrj56CL+/6jDWbK7hkqkzP3fTpUiibO+HqSMfQZjZGOAa4FBgHHCGmQ2Ps+nr7j4+mO5IaUiRPXDUsBIevuIQ1m2p5bIHZ7F5W33YkaQDav7y0bswzdogzGyomeUHj48zs5vNrEc79zkKmOHu1e7eALwKnNvO1xJJCxMH9WLypRNZUl7FlQ+/Q3Wd2iQksSojtfTomkteTuq+17d1T08DjWY2DJgG7A883s59zgeOMbPeZtYVOA3YL852R5jZXDN73swOjPdCZnatmc02s9kVFRXtjCOSGMceUMqvLziY91Zt5PrH3qVhN3dwi+yJykhq74GAtheIpuDb/rnAr9z9FqBfe3bo7guAu4HpwAvAXKDl1613gUHuPg74LfC3Vl5riruXuXtZaWlpe+KIJNRpY/vx43PG8uriCu55aXHYcaQDSXU/TND2AlFvZhcCk4B/BMty27tTd5/m7hPc/RhgA/Bxi/Vb3D0SPH4OyDUz3ZEkGeGiwwZy0WEDmfzqUl6YvzbsONJBRPthSt0VTND2AnEFcATwE3dfbmb7A4+1d6dm1if4ORA4D3iixfp9LOjHwMwODXKub+/+RFLtB2eOZtyAYm57ci7LKiJhx5EOoDJSl55HEO7+kbvf7O5PmFlPoJu737UX+33azD4CngVudPeNZnadmV0XrP8yMN/M5gK/AS5w3a4qGSQ/J5v7L5lIbrZx/WPvqtFa9sq2umg3G2nZBmFmr5hZdzPrRbTN4CEzu7e9O3X3o919tLuPc/eXg2WT3X1y8Pg+dz8wWH+4u7/V3n2JhKV/jy785sKDWVxexY+e+SjsOJLBmkeSS2U3G9D2U0zF7r6F6Omgh9x9IvDF5MUS6RiOHl7KdccO5U+zP+HNJZVhx5EM1dzNRio76oO2F4gcM+sHfJUdjdQi0gbfOHE4+5cU8u2/fMC2usaw40gGar5JLi1PMQF3AC8CS939HTMbQosrj0QkvoLcbO46byyrNlRz7/RFYceRDFQZQkd90PZG6ifd/SB3vz6YX+bu5yc3mkjHcdiQ3lx02ECmvbGcuZ9sCjuOZJjKqmg/TKnsZgPa3kg9wMz+amblZrbOzJ42swHJDifSkXzr1JGUdsvnP5+eR73uspY9UBGpSXk3G9D2U0wPAc8A+wL9iV6e+lCyQol0RN0Lcrnz7DEsXFvFI2+tCDuOZJDKqrqUtz9A2wtEqbs/5O4NwfQwoL4tRPbQyQfuw/EjSvnV/31MeVVN2HEkQ1RGalN+iSu0vUBUmtklZpYdTJegO5tF2uV7Z4ymtqGRX7ygBmtpm4pILSUpvsQV2l4griR6ietaYA3RO52vSFYokY5sSGkRV35hf56cs5r31WAtu+HuoXTUB22/immVu5/l7qXu3sfdzyF605yItMNNJwyntFs+P3jmQ5qa1IuMtG5jdT3VdY0M6Nk15fvemybxWxOWQqSTKcrP4dunjmTuJ5v4y3ufhh1H0tiK9VsBGNQrswqEJSyFSCd0zvj+TBjYg7ueX6BhSqVVq9ZXAzCod2YVCB0Xi+yFrCzjjrPHsGFrHfe8qAZriW/l+mrMYL90O4Iwsyoz2xJnqiJ6T4SI7IUx/YuZdORgHpu5UndYS1wr129ln+4FFORmp3zfuywQ7t7N3bvHmbq5e06qQop0ZLeedAClRfl892/zaVSDtbSwckM1A0M4eoC9O8UkIgnQrSCX750xmg8+3cxjM1aGHUfSzMr11QzuXRjKvlUgRNLAGQf14+jhJdzz4iLKt+gOa4naWttAZaSWgSE0UIMKhEhaMIs2WNc2NvGjf2j0OYlaGeIVTKACIZI29i8p5OYThvHPeWt4ecG6sONIGli1ofkeCJ1iEun0rj1mKCP6duO7f5tPpLYh7DgSsuYjCJ1iEhHycrK46/yxrN1Swy9eWBh2HAnZyg3V9OyaS3GX3FD2rwIhkmYOHtiTSUcM5tEZK5mzcmPYcSREK9dvZWBIVzCBCoRIWrrtSyPo172Ab/9lHnUNGn2us4pe4hrO6SVQgRBJS0X5Ofz43DEsXhfh/leWhB1HQlDX0MRnm7aF0klfMxUIkTR1wsi+nD1+X/7n30tYtLYq7DiSYp9u2kaTo1NMIhLf988YTbeCXG5/ep664ehktnfzrVNMIhJP76J8fnjWgcz9ZBMPvbk87DiSQmF2891MBUIkzZ15UD++OKoP97y0iJXBt0rp+Faur6ZrXjalRakfi7qZCoRImjMzfnzOWHKzsvjPp+dpiNJOYtWGrQzs1RWz8MZmU4EQyQD7FBfwvTNGM2PZBh55e0XYcSQFVqyvDvX0EqhAiGSMr5QN4ISRfbjr+YUsrYiEHUeSqKnJWbWhmkEhXsEEKhAiGcPMuOu8sRTkZnPbk3NpaNQNdB3Vuqoa6hqaQhsoqJkKhEgG6dO9gDvPGcN7qzYx5fVlYceRJGnupC+sgYKaqUCIZJgzD+rH6WP78cvpi1mwZkvYcSQJVqbBPRCgAiGSccyMO88ZQ3GXPG750/vUNjSGHUkSbOX6anKyjH7FBaHmUIEQyUC9CvP4+ZfHsnBtFb+c/nHYcSTBlpRHGNS7KznZ4f6JVoEQyVAnjOzLhYfux+9eW8o7KzaEHUcSaOHaKkbu0z3sGOEUCDP7hpnNN7MPzeybcdabmf3GzJaY2TwzmxBGTpF0993TR7Nfz67c+uf3NQJdB7G1toFVG6oZuU+3sKOkvkCY2RjgGuBQYBxwhpkNb7HZqcDwYLoW+N+UhhTJEIX5Odz71XF8unEbdzz7YdhxJAEWrYv23DuiMxYIYBQww92r3b0BeBU4t8U2ZwOPetQMoIeZ9Ut1UJFMUDa4F9cfN5Q/z17NE7NWhR1H9lJz1+6j+nXOU0zzgWPMrLeZdQVOA/ZrsU1/4JOY+dXBsp2Y2bVmNtvMZldUVCQtsEi6u/WkERx7QCnf//t8Zi1Xe0QmW7hmC4V52fTv0SXsKKkvEO6+ALgbmA68AMwFWp48jdc71ed6KHP3Ke5e5u5lpaWlCc8qkimys4zfXHgw+/XsyvWPzWH1xuqwI0k7LVxbxYh9upGVFV4nfc1CaaR292nuPsHdjwE2AC2v01vNzkcVA4DPUpVPJBMVd8nlgUll1DU2cfUjs9mqRuuM4+5BgQj/9BKEdxVTn+DnQOA84IkWmzwDXBZczXQ4sNnd16Q4pkjGGVpaxH0XTWDxuipu/fP76ho8w6zbUsvmbfWM6hd+AzWEdx/E02b2EfAscKO7bzSz68zsumD9c8AyYAnwAHBDSDlFMs6xB5Ty3dNH8+KH6/j5i4vCjiN7YOHaaNcpI/qmR4HICWOn7n50nGWTYx47cGNKQ4l0IFccNZilFREmv7qUIaWFfLWs5XUgko4WBlcwpcNNcqA7qUU6JDPjh2cdyBeGlfCdv37AjGXrw44kbbBobRX9igso7pobdhRABUKkw8rNzuJ/Lp7AwF5due6xOSwObsCS9LVgzZa0uEGumQqESAdW3CWXBy8/hNzsLC56YAZLylUk0lV9YxNLKyJpc3oJVCBEOrxBvQt54prDAePCB2ZquNI0taxiK/WNnhZ9MDVTgRDpBIb1KeKJaw7D3blwygyWqUikneYrmEamySWuoAIh0mkM79uNx685nMYm56IHZupu6zSzcG0VOVnGkJKisKNspwIh0okc0Lcbv7/qMKrrGrhk6kwqqmrDjiSBRWurGNaniLyc9PmznD5JRCQlRu/bnYeuOIR1W2q57MFZbN5WH3YkIdpJXzpdwQQqECKd0sRBvZh86USWlFdx1cPvUF2nfpvCtHlbPZ9trkmrK5hABUKk0zr2gFJ+fcHBvLtqI9c/9i51DU1hR+q03lxSCcCB+6pAiEiaOG1sP3523lheXVzBLX9+n0Z17pdy7s7vXlvGwF5dOWpYSdhxdhJKX0wikj6+dshANm+r56fPLaR7QS4/PXcMZuGPRdBZzFy+gbmfbOLOc8aQnQZjQMRSgRARrj1mKJuq67n/laUUd8nlP08ZoSKRIpNfXUpJUR5fmTgg7CifowIhIgD8x5dGsHlbPZNfXUp9YxPfOW1UWoxq1pEtWLOFVxZVcNvJB1CQmx12nM9RgRARINoD7J1njyE3O4tpbyxnY3Udd59/ELnZaqpMlimvLaNrXjaXHD4o7ChxqUCIyHZZWcYPzhxNr8I87p2+mC3b6rnvoglp+e02063eWM0zcz/j8iMH06NrXthx4tJXAxHZiZlx84nDufOcMby8sJwrH36HmvrGsGN1OFNfX44BV31h/7CjtEoFQkTiuvTwQdzz5XG8tXQ9Nz3xHg2Nuk8iUf69sJxH317B+RMGsG+PLmHHaZUKhIi06vyJA/jRWQcy/aN13P70PJp0n8ReW7h2Czc98R4j9+nO988cHXacXVIbhIjs0qQjB7N5Wz33Tl9M94JcfnDmaF0C207lVTVc9fBsCvOzmXZ5GYX56f0nOL3TiUhauOmEYWzeVs+0N5bTp3s+Nxw3LOxIGaemvpFrHp3Dhq11PHndEfQrTt9TS81UIERkt8yM754+ivKqWn7+wiKGlBRyyph+YcfKGB+vq+Ibf3yfBWu3MPmSiYzpXxx2pDZRgRCRNjEzfvHlg1i9sZpb/jSXAT27ZswfurC4O4+8tYKfPb+Qwvwcplxaxkmj+4Ydq83USC0ibVaQm82US8voVZjHVY+8w9rNNWFHSkuR2gaenfsZF0+dyQ+f/Ygjh/bmhW8enVHFAXQEISJ7qLRbPtMuL+P8+9/i6kff4Y/XHkFRmje2psLmbfW89OFaXpi/lteXVFLX0ERJUR53njOGSw4bmJEN+/pXFZE9NnKf7tx30QSufnQ21z82h2mTDkmroTJTobahkU82bOPDzzbzj3lreHVRBXWNTfTv0YVLDx/EKWP2YcLAnmnXQ+ueUIEQkXY5fmQffnbeWG5/ah63PzWXe786vsN37ufu3P7UPN5YUsnaLTV4cFtI3+75XHrEIM4cty/jBhRn5NFCPCoQItJuXy3bj4qqWn7x4iL6dC/gv04bFXakpFq0roon56zm6OElXHDIQAb27sL+JUUc1L+4QxZHFQgR2Ss3HDeU8i01THltGaVF+VxzzJCwIyXNc/PWkGXwy6+Np6QoP+w4SacCISJ7xcz4/pkHUhmp4yfPLaBXYR7np+HgN3vL3fnHB2s4fEjvTlEcQAVCRBIgO8u492vj2Lytntufnkdxl1y+mGGXdO7OonVVLKvYypVHpW/vq4nWuS47EJGkyc/JZvKlExmzb3dufPxdZi3fEHakhPpncHrplDH7hB0lZVQgRCRhivJzeOiKQ+nfswtXPfIO8z/dHHakhHB3/vnBGo4Y2nlOL4EKhIgkWK/CPH5/1WF0L8jl0mkzWbS2KuxIe23h2ujppdPGdq7+p1QgRCTh+vfowuPXHEZeThYXT53BkvJI2JH2yvbTSwd2ntNLoAIhIkkyqHchj19zOGBcPHUGK9dvDTtSu7g7zwWnl3p3otNLoAIhIkk0tLSIP1x9GHUNTVz0wExWb6wOO9IeW7CmimWVWzl97L5hR0m5UAqEmd1iZh+a2Xwze8LMClqsv9zMKszs/WC6OoycIrL3RuzTjd9fdRhVNfVcMnUm5VsyqwfYv8/9lOws40sHdqzLdtsi5QXCzPoDNwNl7j4GyAYuiLPpn9x9fDBNTWlIEUmoMf2LefjKQ6moquXiqTNZH6kNO1KbbN5Wz+MzVnHy6L6d7vQShHeKKQfoYmY5QFfgs5ByiEiKTBjYk2mXH8KqDdVcOm0Wm6vrw460W79/ewVVtQ3ceHznHGI15QXC3T8F7gFWAWuAze7+UpxNzzezeWb2lJntF++1zOxaM5ttZrMrKiqSmFpEEuHwIb2ZclkZS8ojXDJtZloXia21DUx7YzknjOzTaUfOC+MUU0/gbGB/YF+g0MwuabHZs8Bgdz8I+D/gkXiv5e5T3L3M3ctKS0uTGVtEEuTYA0qZfOkEFq2t4uJpM9hUXRd2pLiemLWKjdX1nfboAcI5xfRFYLm7V7h7PfAX4MjYDdx9vbs3n6R8AJiY4owikkQnjOzL7y6byOJ1ES56YCYbt6ZXkaipb+R3ry3jyKG9mTioZ9hxQhNGgVgFHG5mXS06qsaJwILYDcws9nbFs1quF5HMd/yIPky5dCJLKiJc+MAMPtmQPpfAPjlnNRVVtXz9hM579ADhtEHMBJ4C3gU+CDJMMbM7zOysYLObg8tg5xK94unyVOcUkeQ7bkQfpl5Wxqcbt3HGb9/g3wvLw45EXUMTk19ZyoSBPThiSO+w44TKvHnMvAxXVlbms2fPDjuGiLTDisqtXP+Hd1mwZgtfP34Yt5x0QGhjOf/knx/xwOvLefiKQzhuRJ9QMqSSmc1x97J463QntYiEbnBJIX+94Ui+VrYf9/17CV+Z/BYffpb6nmBfmL+GB15fzmVHDOoUxWF3VCBEJC0U5GZz95cP4ldfG8/K9dWc+ds3+OEzH7KlJjWXwi6riHDbk/MYt18PvnN6xx5bu61UIEQkrZxzcH/+9f+O4+LDBvHI2ys48b9f5a/vrSaZp8O31TVywx/eJTfbuP/iCeTnZCdtX5lEBUJE0k5x11zuPGcMz9z4Bfbt0YVb/jSXr/1uBgvWbEn4vqrrGrjlT++zaF0Vv7rgYPr36JLwfWQqFQgRSVtjBxTz1+uP5K7zxvJxeRVn/PYNvve3+Qm7JHb+p5s547dv8OJHa/mvU0dx7AG64TaWrmISkYywqbqOe15axB9nfUKTO6eO7cc1Rw9h/H499vi1mpqch99awV3PL6RnYS6//Op4jhxWkoTU6W9XVzGpQIhIRlmzeRsPv7mCx2etoqqmgRF9u3HiqD6cOKoP4/fr2erlsZHaBt5cUsnLC9bxr4UVVEZq+eKoPvz8y+PoVZiX4t8ifahAiEiHE6lt4Ok5q3l+/hreWbGRxianKD+HovwccrKN3OwsGpucSG0DkZoG6hqbAOhWkMOxB5Ry6ph+nDZ2H6IdOnReuyoQOakOIyKSCEX5OUw6cjCTjhzM5m31vLq4gtkrNlBT30hDo1PX2ER2ltGtIIei/Fy6FeQwYWBPygb3JDdbza9toQIhIhmvuEsuZ43bl7PGdb5hQZNJZVREROJSgRARkbhUIEREJC4VCBERiUsFQkRE4lKBEBGRuFQgREQkLhUIERGJq8N0tWFmFcAmIN4wVMUtlu9qvvlxvGUlQOUeRmu5r7aub0/m2Md7k3lXuXa1fnfL0jFzvOX6fOxeZ/l8ZGLmeMt3NT/c3Yvjvrq7d5gJmNKW5buab37cyrLZicqUjMzx8rcnc3tz725ZOmbW50Ofj46WeW8+Hy2njnaK6dk2Lt/V/LO7WJbITLtb357MsY/3JnNbnh9v/e6WpWPmeMv1+di9zvL5yMTM8Za39fOxkw5ziikVzGy2t9LrYbpS5tTJxNzKnBqZmBnUSL2npoQdoB2UOXUyMbcyp0YmZtYRhIiIxKcjCBERiUsFQkRE4uqUBcLMHjSzcjOb347nTjSzD8xsiZn9xmLGKzSzm8xskZl9aGY/T2zq5OQ2sx+a2adm9n4wnZbumWPW32ZmbmYJHW0+Se/znWY2L3iPXzKzhI9sk6TcvzCzhUH2v5pZjwzI/JXg/2CTmSWsYXhvsrbyepPM7ONgmhSzfJef+5Rqz7W5mT4BxwATgPnteO4s4AjAgOeBU4PlxwP/B+QH830yJPcPgdsy6b0O1u0HvAisBErSPZTC/V8AAAYxSURBVDPQPWabm4HJmfBeAycDOcHju4G7MyDzKGAE8ApQFnbWIMfgFst6AcuCnz2Dxz139XuFMXXKIwh3fw3YELvMzIaa2QtmNsfMXjezkS2fZ2b9iP5Hf9uj/5KPAucEq68H7nL32mAf5RmSO6mSmPmXwO1Awq+ySEZmd98Ss2lhBuV+yd0bgk1nAAMyIPMCd1+UyJx7k7UVXwKmu/sGd98ITAdOCfP/ajydskC0Ygpwk7tPBG4D7o+zTX9gdcz86mAZwAHA0WY208xeNbNDkpp2h73NDfD14BTCg2bWM3lRt9urzGZ2FvCpu89NdtAYe/0+m9lPzOwT4GLg+0nMGisRn49mVxL9RptsicycbG3JGk9/4JOY+eb86fJ7AZAT1o7TiZkVAUcCT8ac7suPt2mcZc3fBHOIHioeDhwC/NnMhgTfApIiQbn/F7gzmL8T+G+ifwiSYm8zm1lX4DtET32kRILeZ9z9O8B3zOzbwNeBHyQ46s5hEpQ7eK3vAA3AHxKZ8XNBEpg52XaV1cyuAL4RLBsGPGdmdcBydz+X1vOH/nvFUoGIygI2ufv42IVmlg3MCWafIfrHNPYQewDwWfB4NfCXoCDMMrMmoh10VaRzbndfF/O8B4B/JDEv7H3mocD+wNzgP+UA4F0zO9Td16Zp5pYeB/5JkgsECcodNKCeAZyYzC88gUS/18kUNyuAuz8EPARgZq8Al7v7iphNVgPHxcwPINpWsZrwf68dwmr8CHsCBhPT2AS8BXwleGzAuFae9w7Ro4TmBqTTguXXAXcEjw8gevhoGZC7X8w2twB/TPfMLbZZQYIbqZP0Pg+P2eYm4KkM+VyfAnwElCYjbzI/HyS4kbq9WWm9kXo50bMOPYPHvdr6uU/VFMpOw56AJ4A1QD3Rin0V0W+lLwBzg/8Q32/luWXAfGApcB877kbPAx4L1r0LnJAhuX8PfADMI/rNrF+6Z26xzQoSfxVTMt7np4Pl84h2jtY/Qz4fS4h+2Xk/mBJ69VWSMp8bvFYtsA54McysxCkQwfIrg/d3CXDFnnzuUzWpqw0REYlLVzGJiEhcKhAiIhKXCoSIiMSlAiEiInGpQIiISFwqENKhmVkkxfubamajE/RajRbt/XW+mT27u55UzayHmd2QiH2LgEaUkw7OzCLuXpTA18vxHZ3XJVVsdjN7BFjs7j/ZxfaDgX+4+5hU5JOOT0cQ0umYWamZPW1m7wTTUcHyQ83sLTN7L/g5Ilh+uZk9aWbPAi+Z2XFm9oqZPWXRsRL+0Nxnf7C8LHgcCTrom2tmM8ysb7B8aDD/jpnd0cajnLfZ0VlhkZm9bGbvWnTcgLODbe4ChgZHHb8Itv2PYD/zzOxHCXwbpRNQgZDO6NfAL939EOB8YGqwfCFwjLsfTLS31Z/GPOcIYJK7nxDMHwx8ExgNDAGOirOfQmCGu48DXgOuidn/r4P977afnaAfohOJ3ukOUAOc6+4TiI5D8t9BgfoWsNTdx7v7f5jZycBw4FBgPDDRzI7Z3f5EmqmzPumMvgiMjumBs7uZdQOKgUfMbDjRHjRzY54z3d1jxwKY5e6rAczsfaJ99LzRYj917Oj8cA5wUvD4CHb08f84cE8rObvEvPYcomMGQLSPnp8Gf+ybiB5Z9I3z/JOD6b1gvohowXitlf2J7EQFQjqjLOAId98Wu9DMfgv8293PDc7nvxKzemuL16iNedxI/P9L9b6jka+1bXZlm7uPN7NiooXmRuA3RMeTKAUmunu9ma0ACuI834Cfufvv9nC/IoBOMUnn9BLR8RgAMLPm7pqLgU+Dx5cncf8ziJ7aArhgdxu7+2aiw5TeZma5RHOWB8XheGBQsGkV0C3mqS8CVwbjFmBm/c2sT4J+B+kEVCCko+tqZqtjpluJ/rEtCxpuPyLaVTvAz4GfmdmbQHYSM30TuNXMZgH9gM27e4K7v0e0x9ALiA7aU2Zms4keTSwMtlkPvBlcFvsLd3+J6Cmst83sA+Apdi4gIruky1xFUiwYFW+bu7uZXQBc6O5n7+55IqmmNgiR1JsI3BdcebSJJA7xKrI3dAQhIiJxqQ1CRETiUoEQEZG4VCBERCQuFQgREYlLBUJEROL6/3pA3rjwl0oDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "55299290-8c05-415f-8f73-a3656c1d488c"
    }
   },
   "outputs": [],
   "source": [
    "lr = 3e-3\n",
    "lr *= bs/48  # Scale learning rate by batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "debf9e8a-b4ec-4b06-8344-a8f01150d941"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='3' class='' max='10', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      30.00% [3/10 1:27:10<3:23:23]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.456712</td>\n",
       "      <td>3.493251</td>\n",
       "      <td>0.652252</td>\n",
       "      <td>0.347747</td>\n",
       "      <td>32.892696</td>\n",
       "      <td>28:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.243398</td>\n",
       "      <td>3.353261</td>\n",
       "      <td>0.640550</td>\n",
       "      <td>0.359450</td>\n",
       "      <td>28.595928</td>\n",
       "      <td>29:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.228111</td>\n",
       "      <td>3.336011</td>\n",
       "      <td>0.638784</td>\n",
       "      <td>0.361217</td>\n",
       "      <td>28.106716</td>\n",
       "      <td>29:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='33696' class='' max='43928', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      76.71% [33696/43928 21:02<06:23 3.2003]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAWnElEQVR4nO3deXhV9Z3H8c83C4SQICFsYdFgtYooRYyIxfqgdlFA7fPIWProtMO0wzNqK9LpVBynlXa6UO3i+LQu2No+rVSrWB9nrEuthVJHxQZLMbLIFiGygwmLBJLwmz/uSbgJN3eBc+79JXm/nifPPefcc37ne383fDj5nXPuNeecAAD+yst1AQCA5AhqAPAcQQ0AniOoAcBzBDUAeK4gikZPKRvgzvzQ6VE0DQDd0vLly3c75wYlei6SoB48bKSqq6ujaBoAuiUze7ez5xj6AADPEdQA4LlIgpp7HQEgPJGMUZPUADLR1NSkuro6NTY25rqUyBUVFWnEiBEqLCxMe5toghoAMlBXV6fS0lJVVlbKzHJdTmScc9qzZ4/q6uo0atSotLdjjBpAzjU2Nqq8vLxbh7QkmZnKy8sz/suBoAbghe4e0q1O5HVGFNQMUgNAWDiiBtDj1dfX6/777894uylTpqi+vj6CitojqAH0eJ0FdUtLS9LtnnvuOfXv3z+qstpEctUHAx8AupK5c+dqw4YNGjdunAoLC1VSUqKKigqtWLFCq1at0qc//Wlt2bJFjY2Nmj17tmbNmiVJqqysVHV1tQ4cOKCrrrpKl1xyiV599VUNHz5czzzzjPr06RNKfVyeB8Ar3/zft7Vq675Q2zxnWD/ddfWYTp+fP3++ampqtGLFCi1ZskRTp05VTU1N2yV0jzzyiAYMGKBDhw7pwgsv1HXXXafy8vJ2baxbt06PPfaYHn74YV1//fV66qmndOONN4ZSP0ENAB1MmDCh3XXO9913n55++mlJ0pYtW7Ru3brjgnrUqFEaN26cJOmCCy5QbW1taPUQ1AC8kuzIN1v69u3bNr1kyRL98Y9/1Guvvabi4mJNnjw54XXQvXv3bpvOz8/XoUOHQquHk4kAerzS0lLt378/4XMNDQ0qKytTcXGx1qxZo9dffz3L1XFEDQAqLy/XpEmTdO6556pPnz4aMmRI23NXXnmlHnzwQY0dO1ZnnXWWJk6cmPX6zLnwr9H40OixbsPqlaG3C6B7Wr16tUaPHp3rMrIm0es1s+XOuapE6zP0AQCeI6gBwHN8cQAAeI4jagDwHEENAJ4jqAHAcwQ1AJyAkpISSdLWrVs1ffr0hOtMnjxZ1dXVJ70vghoATsKwYcO0aNGiSPfBnYkAIOn222/XaaedpptvvlmSNG/ePJmZli5dqvfff19NTU369re/rWuvvbbddrW1tZo2bZpqamp06NAhzZw5U6tWrdLo0aND+7wPPo8agF+enyttfyvcNoeeJ101P+kqM2bM0G233dYW1E888YReeOEFzZkzR/369dPu3bs1ceJEXXPNNZ1+7+EDDzyg4uJirVy5UitXrtT48eNDKZ8jagCQdP7552vnzp3aunWrdu3apbKyMlVUVGjOnDlaunSp8vLy9N5772nHjh0aOnRowjaWLl2qW2+9VZI0duxYjR07NpTaCGoAfklx5Bul6dOna9GiRdq+fbtmzJihhQsXateuXVq+fLkKCwtVWVmZ8CNO40XxbeqcTASAwIwZM/T4449r0aJFmj59uhoaGjR48GAVFhZq8eLFevfdd5Nuf+mll2rhwoWSpJqaGq1cGc6H00VzRM0gNYAuaMyYMdq/f7+GDx+uiooK3XDDDbr66qtVVVWlcePG6eyzz066/U033aSZM2dq7NixGjdunCZMmBBKXZF8zOmos89zm9aEfDIAQLfFx5yG8DGnZjbHzN42sxoze8zMikKoFQCQhpRBbWbDJd0qqco5d66kfEkzkm3DyAcAhCfdk4kFkvqYWYGkYklboysJQE8UxTCsj07kdaYMaufce5J+IGmzpG2SGpxzf+i4npnNMrNqM6s+eOBgxoUA6LmKioq0Z8+ebh/Wzjnt2bNHRUWZjR6nPJloZmWSnpL0GUn1kp6UtMg592hn21SefZ6r5WQigDQ1NTWprq4u5TXK3UFRUZFGjBihwsLCdsuTnUxM5/K8j0va5JzbFTT2O0kfldRpUANAJgoLCzVq1Khcl+GtdMaoN0uaaGbFFrvl5gpJq6MtCwDQKp0x6mWSFkl6U9JbwTYLIq4LABBI685E59xdku6KuBYAQAJ8CzkAeI4PZQIAz0UT1BxSA0BoOKIGAM8R1ADgOYIaADxHUAOA57g8DwA8xxE1AHiOoAYAzxHUAOA5ghoAPEdQA4DnCGoA8Byf9QEAnuOIGgA8F9ENLxxSA0BYOKIGAM8R1ADgOYIaADxHUAOA5whqAPAcQQ0AnuPzqAHAc9EENUkNAKGJJKiPktQAEBqOqAHAcxEFNUkNAGGJJKibWghqAAhLJEF98EhzFM0CQI8USVD3LuDybAAISySJmmcWRbMA0CNxeR4AeI7L8wDAcxxRA4DnIgnq5qMENQCEJa2gNrP+ZrbIzNaY2WozuzjqwgAAMQVprvffkl5wzk03s16SilNtcKT5qHpxmR4AnLSUQW1m/SRdKumfJMk5d0TSkVTbHWpqIagBIATpJOnpknZJ+oWZ/c3MfmZmfTuuZGazzKzazKolaVvDoZBLBYCeKZ2gLpA0XtIDzrnzJR2UNLfjSs65Bc65KudclSQtXrMr1EIBoKdKJ6jrJNU555YF84sUC+6kNu/94GTqAgAEUga1c267pC1mdlaw6ApJq1Jtt4WgBoBQpHvVx5clLQyu+NgoaWaqDV5Zv/tk6gIABNIKaufcCklVEdcCAEiA6+cAwHMENQB4jqAGAM8R1ADgOYIaADxHUAOA5yIN6uaWo1E2DwA9QqRB/dKqHVE2DwA9QqRB/fNXNkXZPAD0CJEE9fD+fSRJ1e++H0XzANCjRBLU/YsLo2gWAHqkSII6zyyKZgGgR+LyPADwXORB/dfavVHvAgC6tciCurI89kXl//Dga1HtAgB6hMiC+tdfuCiqpgGgR4ksqEcOKG6b3rmvMardAEC3l5WTiRO++3I2dgMA3VKkQf3MLZOibB4AeoRIg/ojI/u3Te89eCTKXQFAt5W166jH/9dL2doVAHQrkQf1deNHtE0756LeHQB0O5EH9Q+v/0jb9Kg7not6dwDQ7XALOQB4LitBXTt/att05dzfZ2OXANBt5OSImrFqAEhf1oJ63XeuaptmrBoA0pe1oC7Mz9Nnqka2zd+/ZH22dg0AXVpWhz6+P31s2/TdL6zN5q4BoMvK+hj1X752Wds0JxYBILWsB3X8p+pJ0vYGPlkPAJLJyVUfr99xRdv0xO/xyXoAkExOgnroKUUa0LdX2zxDIADQuZzdmbj8Pz/ebn4HXy4AAAnlLKjNTG9+/RNt8xd992Vt2ftBrsoBAG/l9LM+BvTtpW9dO6Zt/mN3L85hNQDgp7SD2szyzexvZvZsmAV87uLKdvOMVwNAe5kcUc+WtDqKIn7zxfbfWH7eXS9GsRsA6JLSCmozGyFpqqSfRVHER88YqNlXnNk2v/9ws/5v/e4odgUAXU66R9T3SvqapKOdrWBms8ys2syqd+3alXEhcz7xYV121qC2+Rt+tkxvbNqbcTsA0N2kDGozmyZpp3NuebL1nHMLnHNVzrmqQYMGJVu1U7+YOUEXVpa1zV//0Gt6adWOE2oLALqLdI6oJ0m6xsxqJT0u6XIzezSqgp78149qSL/ebfP/8qtqPfjnDVHtDgC8lzKonXN3OOdGOOcqJc2Q9Cfn3I1RFrXsP9rfDDP/+TVatnFPlLsEAG95+52JG787pd38Zxa8rqXvZD72DQBdXUZB7Zxb4pybFlUx8fLyrN13LUrS5x55Q5Vzf6+mlk7PaQJAt+PtEXWrjmEtSWfe+byaCWsAPYT3QS0dPwwiSWfc+bzOm/cigQ2g2+sSQd06DLL03y9rt3x/Y7POuPN5bWs4lKPKACB6BbkuIBOnlhdrRFkf1b3fPpgv/t6fJEmXnDFQj3a4HR0AuroucUQd75XbL9db8z6Z+Ln1u1U59/f6yzs7s1wVAESnSx1RtyotKlTt/KnavOcDXXrP8R+N2vDrG7U3f7X6Dx6hvJIhUskQqWSwVDr02HTr8qJTJLMcvAoASE+XDOpWp5YXq3b+VF32gyXatPtg2/I/H/2IGlyJBm2r12DbrIH2lobkNajQNR3fSH7v9uFdOuT4MG+dL+h9/PYAEDFzzoXeaFVVlauurg693WScc3rqzff01Sf/3tka6qeDGmQNuvJUacuWWt1yYamGF+xXSdMe6cB26cBOuQM7ZB90chdkUf8OYZ4o0IdIfcqkvC43qgQgh8xsuXOuKuFz3SWo4/108Xrd8+LaE96+QM26dWJ/ffnCUu3Yulm1727U0Lx9GlawT3u2b9bQvAYdPbBT2r9d+S0Jvusxr0DqO1gqGSTl95IsX8rLlywveMxXi+UpLy9fllcgWZ5aLF8NjS0aUNInWD8v9hi3Tcc20l5eWiGdffwljgD80eOCWpIOHm7WmMi/gMCprxo12Oo1SPUaZA0aZPWxHzVooDXootP6qbm5SXsPNMq1NOv9g40qK8rXgcbDytdR5emo8oOf1uk8c+2W9S2Umpqajz0v127dlFWOvFj2hRci7gsAJyNZUHfpMepk+vYuaHdX46bdB3XZD5aEvBfTQfXRJtdHm1QhJcrM9QmWHclwN4eTPemOC/p2oa+jalmXrzcz3CUAf3TboO5o1MC+qp0/VT/6w1pNOmOgLjq9XO/s2K8BfXtp8j1LdOBwsyTp1AHF2tylvg3d1KJ8tSg/14UAiEiPCepWX/nkWW3THx5SKkmq+eanEq67Yku9KsuLtWXvIV39k1d0w0Wn6uvTztGLb2/X7MdX6NYrztR9L6/TN6ado289u0pjhvXT21v3ZeV1ZKJ3ASc2ga6s245R+2Jr/SEdPNysU8uL1bvg2FFvc8tRba1v1KnlxUm3bznqtGb7Po0ZdkrS9RqbWnSk5ahKehXopoXL9cWPna6S3gUaXdEvlNcBIFo98mQiAHQlyYKav4kBwHMENQB4jqAGAM8R1ADgOYIaADxHUAOA5whqAPAcQQ0AniOoAcBzBDUAeI6gBgDPEdQA4DmCGgA8R1ADgOcIagDwHEENAJ4jqAHAcwQ1AHiOoAYAzxHUAOA5ghoAPJcyqM1spJktNrPVZva2mc3ORmEAgJiCNNZplvRvzrk3zaxU0nIze8k5tyri2gAASuOI2jm3zTn3ZjC9X9JqScOjLgwAEJPRGLWZVUo6X9KyKIoBABwv7aA2sxJJT0m6zTm3L8Hzs8ys2syqd+3aFWaNANCjpRXUZlaoWEgvdM79LtE6zrkFzrkq51zVoEGDwqwRAHq0dK76MEk/l7TaOfej6EsCAMRL54h6kqR/lHS5ma0IfqZEXBcAIJDy8jzn3CuSLAu1AAAS4M5EAPAcQQ0AniOoAcBzBDUAeI6gBgDPEdQA4DmCGgA8R1ADgOcIagDwHEENAJ4jqAHAcwQ1AHiOoAYAzxHUAOA5ghoAPEdQA4DnCGoA8BxBDQCeI6gBwHMENQB4jqAGAM8R1ADgOYIaADxHUAOA5whqAPAcQQ0AniOoAcBzBDUAeI6gBgDPEdQA4DmCGgA8R1ADgOcIagDwHEENAJ4jqAHAcwQ1AHiOoAYAz6UV1GZ2pZmtNbP1ZjY36qIAAMekDGozy5f0U0lXSTpH0mfN7JyoCwMAxKRzRD1B0nrn3Ebn3BFJj0u6NtqyAACtCtJYZ7ikLXHzdZIu6riSmc2SNCuYPWxmNSdfXugGStqd6yISoK7MUFdmqCszuarrtM6eSCeoLcEyd9wC5xZIWiBJZlbtnKtKu7wsoa7MUFdmqCsz1JW+dIY+6iSNjJsfIWlrNOUAADpKJ6j/KulMMxtlZr0kzZD0P9GWBQBolXLowznXbGZfkvSipHxJjzjn3k6x2YIwiosAdWWGujJDXZmhrjSZc8cNNwMAPMKdiQDgOYIaAHznnAvtR9KVktZKWi9pbphtd9hPraS3JK2QVB0sGyDpJUnrgseyYLlJui+oaaWk8XHtfD5Yf52kz8ctvyBof32wrXVSxyOSdkqqiVsWeR2d7SNFXfMkvRf02QpJU+KeuyPYx1pJn0r1fkoaJWlZsP/fSuoVLO8dzK8Pnq/sUNdISYslrZb0tqTZPvRZkrpy2meSiiS9IenvQV3fPIm2Qqk3RV2/lLQprr/GZft3P1gnX9LfJD3rQ3+Fknkhhme+pA2STpfUK3gTzwmz2Lh91Uoa2GHZ3a0dJ2mupO8H01MkPR/8skyUtCzuDd8YPJYF060B8Yaki4Ntnpd0VSd1XCppvNoHYuR1dLaPFHXNk/TVBK/hnOC96h38sm0I3stO309JT0iaEUw/KOmmYPpmSQ8G0zMk/bbDvioU/COVVCrpnWD/Oe2zJHXltM+C11ASTBcqFgQTM20rzHpT1PVLSdMT9FfWfveD5V+R9BsdC+qc9lcomRdieF4s6cW4+Tsk3RFmsXFt1+r4oF4rqSLuH97aYPohSZ/tuJ6kz0p6KG75Q8GyCklr4pa3Wy9BLZVqH4iR19HZPlLUNU+JQ6fd+6TY1T0Xd/Z+Bv9wdksq6Pi+t24bTBcE6yX8ayRY5xlJn/ClzxLU5U2fSSqW9KZidwVn1FaY9aao65dKHNRZex8Vu8/jZUmXS3r2RPo+yv460Z8wx6gT3Wo+PMT24zlJfzCz5cGt65I0xDm3TZKCx8Ep6kq2vC7B8nRlo47O9pHKl8xspZk9YmZlJ1hXuaR651xzgrratgmebwjWP46ZVUo6X7GjMW/6rENdUo77zMzyzWyFYkNZLyl2RJdpW2HWm7Au51xrf30n6K8fm1nvE+yvk3kf75X0NUlHg/kT6fvQ++tkhRnUad1qHpJJzrnxin2i3y1mdmmSdTurK9PlJyvXdTwg6UOSxknaJumHEdSVVs1mViLpKUm3Oef2Jak5q32WoK6c95lzrsU5N06xI8UJkkafQFuh92PHuszsXMWOLs+WdKFiwxm3h1xXUmY2TdJO59zy+MVJ2spaf52sMIM6a7eaO+e2Bo87JT2t2C/wDjOrkKTgcWeKupItH5FgebqyUUdn++iUc25H8I/rqKSHFeuzE6lrt6T+ZlbQYXm7toLnT5G0N74OMytULAwXOud+l+L1ZK3PEtXlS58FtdRLWqLYGG+mbYVZb2d1Xemc2+ZiDkv6hU68v070fZwk6Rozq1XsUz4vV+wI25v+OmFhjaEoNsazUbHB99aB9jFhjtME++krqTRu+lXFzsTeo/YnGe4Opqeq/YmMN4LlAxQ7Q10W/GySNCB47q/Buq0nMqYkqadS7ceCI6+js32kqKsibnqOpMeD6TFqf+Jko2InTTp9PyU9qfYnTm4Opm9R+5MzT3SoyST9StK9HZbntM+S1JXTPpM0SFL/YLqPpL9ImpZpW2HWm6Kuirj+vFfS/Fz87gfPTdaxk4k57a9Qci/kEJ2i2BnzDZLuDLPtuH2cHnRQ66VBdwbLyxU7ibAueGx9w02xLz7YoNjlPlVxbf2zYpfZrJc0M255laSaYJufqPPL8x5T7E/iJsX+t/1CNurobB8p6vp1sN+Vin1WS3wI3RnsY63irnDp7P0M3oM3gnqflNQ7WF4UzK8Pnj+9Q12XKPYn4UrFXfKW6z5LUldO+0zSWMUuM1sZvKZvnERbodSboq4/Bf1VI+lRHbsyJGu/+3HbT9axoM5pf4Xxwy3kAOA57kwEAM8R1ADgOYIaADxHUAOA5whqAPAcQQ0AniOoAcBz/w/H99zyiP7fgwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.34774690866470337.\n",
      "Better model found at epoch 1 with accuracy value: 0.3594495952129364.\n",
      "Better model found at epoch 2 with accuracy value: 0.36121678352355957.\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "learn.unfreeze()\n",
    "wd = 0.01 \n",
    "learn.fit_one_cycle(10, lr, wd=wd, moms=(0.8,0.7), \n",
    "                    callbacks=[ShowGraph(learn),\n",
    "                               SaveModelCallback(learn.to_fp32(), monitor='accuracy', name='bestmodel_sp15')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "GCP instance has stopped. Need to reload bestmodel_sp15 and vocab to continue the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "lm_fns2[0] = 'bestmodel_sp15'\n",
    "data.vocab.save(mdl_path/(lm_fns2[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bestmodel_sp15', 'fr_wt_vocab_sp15']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm_fns2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.43 s, sys: 1.5 s, total: 5.94 s\n",
      "Wall time: 30.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, drop_mult=0., pretrained_fnames=lm_fns2, \n",
    "                               metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5hcdZ3n8fe3Ln3vTufSuQdCQAjiIkJELuoqOKwXFnXVXebRRxGRR0dFx3Hc4XHHmcFxRsedFV2fGRZ1GBxhvIDMAKOI4DCAXBMg3BIghEBCku5OX9KX6u66ffePc6pTNN1Jp1OnLl2f10M9XXXOqTrfLtL1qd/vd87vmLsjIiL1K1bpAkREpLIUBCIidU5BICJS5xQEIiJ1TkEgIlLnEpUu4HAtWbLE165dW+kyRERqyqZNm/a5e9d062ouCNauXcvGjRsrXYaISE0xsxdnWqeuIRGROqcgEBGpcwoCEZE6pyAQEalzCgIRkTqnIBARqXMKAhGROqcgEBGpAVfe8Sz3PNcbyWsrCEREqlw+73z3zud46IX+SF5fQSAiUuWGxjPkHTpbGiJ5fQWBiEiV6x9NA7CoNRnJ6ysIRESq3EAqA8BCtQhEROrTQNgiUBCIiNSp/lSha0hBICJSlwbDIOhs0RiBiEhd6h/NkIwbbY3RXEJGQSAiUuUGU2kWtjRgZpG8voJARKTK9Y+mIxsoBgWBiEjVG0ilWRjROQSgIBARqXoDqYxaBCIi9WxgNM3CiA4dBQWBiEhVy+edwbEMi9QiEBGpT8PjWXJ5j+wcAlAQiIhUtajPKgYFgYhIVRtIRTvPECgIRESq2uSEcxG2CKI5XzlkZjuAYSAHZN19w5T1bwP+FXghXPQLd78iyppERGrJgSmooxsjiDQIQm93930HWX+Pu59fhjpERGpOOVoE6hoSEali/ak0iZjRHtGEcxB9EDhwu5ltMrNLZ9jmTDPbbGa/MrOTptvAzC41s41mtrG3tze6akVEqsxgKk1nhBPOQfRdQ2e7+24zWwr8xsy2uvvdResfAY529xEzezfwL8Brpr6Iu18NXA2wYcMGj7hmEZGq0T+ajuxaxQWRtgjcfXf4swe4CTh9yvohdx8J7/8SSJrZkihrEhGpJQOpDJ0RHjoKEQaBmbWaWXvhPnAe8OSUbZZb2N4xs9PDevqiqklEpNYMjKYjnV4Cou0aWgbcFH7OJ4Dr3f02M/sUgLtfBXwQ+LSZZYEx4EJ3V9ePiEhoIJWJ9IghiDAI3H078Pppll9VdP97wPeiqkFEpJa5e3AtggjPIQAdPioiUrWGwgnnopxnCBQEIiJVazCcZ6hmB4tFROTI9I8WZh5V15CISF0anJxnSC0CEZG6VGgRKAhEROrU5LUINFgsIlKfBlJp4jGjoyna2YAUBCIiVap/NMPClmSkE86BgkBEpGoNptKRjw+AgkBEpGr1jyoIRETq2kAqzcKIzyEABYGISNUaSGXUIhARqVfuzsBoOvJDR0FBICJSlUYmsmTzHvm1CEBBICJSlQZGg+klOiOeghoUBCIiVak/VZhwTi0CEZG6NFCmKahBQSAiUpUGRtUiEBGpawOTU1BrjEBEpC4NjKaJGXQ0KQhEROpSfzjPUCwW7YRzoCAQEalKfSMTZRkfAAWBiEhV6h6aYFlHU1n2pSAQEalCPUPjLO1oLMu+FAQiIlUmn3d6hidYrhaBiEh96k+lyeZdXUMiIvWqe2gcgGXqGhIRqU89QxMALFWLQESkPh1oESgIRETqUnfYIuhqU9eQiEhd6h4eZ3FrAw2J8nxER7oXM9thZk+Y2WNmtnGa9WZm3zWzbWb2uJmdGmU9IiK1IDiHoDzdQgCJMuzj7e6+b4Z17wJeE97eBPx9+FNEpG4FZxWXp1sIKt819F7gRx54AOg0sxUVrklEpKK6h8ZZ1l6+FkHUQeDA7Wa2ycwunWb9KmBn0eNd4bJXMLNLzWyjmW3s7e2NqFQRkcrL5vLsG5lfLYKz3f1Ugi6gz5jZW6esn25+VX/VAver3X2Du2/o6uqKok4RkarQN5om7+U7hwAiDgJ33x3+7AFuAk6fsskuYE3R49XA7ihrEhGpZuU+hwAiDAIzazWz9sJ94DzgySmb3Qx8NDx66Axgv7vviaomEZFqVziHoJxdQ1EeNbQMuMnMCvu53t1vM7NPAbj7VcAvgXcD24AU8PEI6xERqXqVaBFEFgTuvh14/TTLryq678BnoqpBRKTW9AyNEzNYXKark0HlDx8VEZEi3UMTLGlrJBEv38ezgkBEpIp0D4+XtVsIFAQiIlWl3GcVg4JARKSqlHueIVAQiIhUjXQ2T99ouqzTS4CCQESkavSOlP8cAlAQiIhUjUqcQwAKAhGRqtGjIBARqW+VmF4CFAQiIlWje2icZNxY2FK+s4pBQSAiUjW6hyZY2t5ELDbdDP3RURCIiFSJnuFxlpa5WwgUBCIiVaPcl6gsUBCIiFSJSkwvAQoCEZGqMJ7JsX8sU/bpJUBBICJSFXomDx1VEIiI1KXu4cLJZOoaEhGpS7sHxwBYrhaBiEh92tmfAmD1wpay71tBICJSBXb2j7GkrZHmhnjZ960gEBGpAjsHUhy1qLki+1YQiIhUgZ0DKdYsKn+3ECgIREQqLpvLs3twnDUVGB8ABYGISMXt2T9OLu+sqeauITM71swaw/tvM7PLzKwz2tJEROpD4Yiham8R3AjkzOw44IfAMcD1kVUlIlJHdg6EQVDlYwR5d88C7weudPc/BFZEV5aISP3Y2T9GPGasWFD+k8lg9kGQMbPfBz4G3BouS0ZTkohIfdk5kGJlZxOJeGWGbWe7148DZwJfd/cXzOwY4MfRlSUiUj9e6k9VbHwAIDGbjdz9aeAyADNbCLS7+zeiLExEpF7s7B/j3PVLK7b/2R41dJeZdZjZImAzcI2Z/Z9oSxMRmf/G0jn2jUxU7NBRmH3X0AJ3HwL+G3CNu58GvCO6skRE6sOuCh8xBLMPgoSZrQD+OwcGi2fFzOJm9qiZvep5ZnaRmfWa2WPh7ZLDeW0RkVpXOHS0ErOOFsxqjAC4Avg18Dt3f9jM1gHPzfK5nwe2AB0zrP+pu392lq8lIjKv7OwPrkNQ9V1D7v5zdz/Z3T8dPt7u7h841PPMbDXwHuAHR1amiMj89FJ/iqZkjK628l+ZrGC2g8WrzewmM+sxs24zuzH8kD+UK4EvA/mDbPMBM3vczG4wszUz7P9SM9toZht7e3tnU7KISE3YGR46amYVq2G2YwTXADcDK4FVwC3hshmZ2flAj7tvOshmtwBr3f1k4A7g2uk2cver3X2Du2/o6uqaZckiItVv58BYRQeKYfZB0OXu17h7Nrz9I3CoT+SzgQvMbAfwE+AcM3vFSWju3ufuE+HD7wOnzb50EZHa5u7s6k+xZmHlxgdg9kGwz8w+Eh4BFDezjwB9B3uCu1/u7qvdfS1wIfBbd/9I8TbhkUgFFxAMKouI1IX9YxmGJ7I10yK4mODQ0b3AHuCDBNNOHDYzu8LMLggfXmZmT5nZZoIzly+ay2uKiNSiwhFDlTx0FGY/xcRLBN/YJ5nZFwgGg2fz/LuAu8L7Xy1afjlw+exKFRGZX14qXIeggoeOwpFdoeyLJatCRKQOVfo6BAVHEgSVO9ZJRGQe2NmforMlSUdTZWf1P5Ig8JJVISJSh3YOjFV0+umCg44RmNkw03/gG1DZTi0RkRq3qz/F+hXtlS7j4EHg7pWvUERkHsrm8uwaGOP3TlpW6VKOqGtIRETmaEffKOlcnhOWVf77toJARKQCtuwZBuCE5QoCEZG6tHXvEPGYcdzStkqXoiAQEamEZ/YOc2xXK42JeKVLURCIiFTClj3DnLB8put1lZeCQESkzIbGM7w8OMb6KhgfAAWBiEjZPbs3GCg+sQrOIQAFgYhI2W3ZWzhiSF1DIiJ1aeueIdqbEqxc0FTpUgAFgYhI2W3dO8yJyzsqep3iYgoCEZEycnee2TtcFSeSFSgIRETKaNfAGCMT2aqYbK5AQSAiUkZbw4Hi9VUyUAwKAhGRsnpm7xBQHXMMFSgIRETKaMveYdYsaqatcVaXjC8LBYGISBlt3TNUVd1CoCAQESmb8UyOF/aNVs3UEgUKAhGRMtnWM0Leq2ugGBQEIiJls2VPMFBcTYeOgoJARKRsntk7TGMixtrFrZUu5RUUBCIiZbLppQFOWtlBPFYdU0sUKAhERMpgeDzD47v2c9axSypdyqsoCEREyuDhHf3k8s5Zxy6udCmvoiAQESmD+7b10RCPcerRCytdyqsoCEREyuD+7X2cenQnTcnKX6x+KgWBiEjEBkbTPL1nqCrHB0BBICISuQdf6MOdqhwfgDIEgZnFzexRM7t1mnWNZvZTM9tmZg+a2dqo6xERKbf7nu+jpSHOyas7K13KtMrRIvg8sGWGdZ8ABtz9OODbwDfLUI+ISFnd/3wfG9YuoiFRnZ0wkVZlZquB9wA/mGGT9wLXhvdvAM61armIp4hICfQMj/Ncz0jVdgtB9C2CK4EvA/kZ1q8CdgK4exbYD7zq3TKzS81so5lt7O3tjapWEZGSu//5PqB6xwcgwiAws/OBHnffdLDNplnmr1rgfrW7b3D3DV1dXSWrUUQkavc/30d7U4KTVi6odCkzirJFcDZwgZntAH4CnGNmP56yzS5gDYCZJYAFQH+ENYmIlNX92/s4Y93iqptfqFhkQeDul7v7andfC1wI/NbdPzJls5uBj4X3Pxhu86oWgYhILdo1kOLFvlRVdwsBlP2imWZ2BbDR3W8Gfgj8k5ltI2gJXFjuekREonLvc/sAqvZEsoKyBIG73wXcFd7/atHyceBD5ahBRKTc7tzaw6rOZo5f1lbpUg6qOg9qFRGpceOZHPc+t49zT1xKtR8VryAQEYnA/dv7GMvkOGf90kqXckgKAhGRCNy5pZuWhjhnrKvugWJQEIiIlJy789stPbz5uCVVOe30VAoCEZES27p3mN37xzn3xOrvFgIFgYhIyd25pRuAt5+gIBARqUt3bu3h9asXsLSjqdKlzIqCQESkhPaNTPDYzkHOWb+s0qXMmoJARKSE/n1rD+7UzPgAKAhERErqt1t7WNbRyEkrOypdyqwpCERESmR0Isvdz/ZyzvplVX82cTEFgYhIidy8eTej6RwfPG1VpUs5LAoCEZEScHd+/MCLrF/ezqlHLax0OYdFQSAiUgKbd+3nqd1DfPiMo2uqWwgUBCIiJXHdAy/S0hDnfaesrHQph01BICJyhPanMtzy+G7e94ZVtDclK13OYVMQiIgcoRsf2cV4Js+H33RUpUuZEwWBiMgRcHeue/BF3nBUJyetXFDpcuZEQSAicgQe2N7P872jfPhNR1e6lDlTEIiIHIFrfvcCC5qTnH/yikqXMmcKAhGROXpq935uf7qbi85aWxMXoJmJgkBEZI6+c8dztDcluPjNx1S6lCOiIBARmYMnXw5aA5948zEsaK69Q0aLKQhERObgyjueo2MetAZAQSAictiefHk/d2zp5pK3rKOjBk8gm0pBICJymK6841k6mhJcdPbaSpdSEgoCEZHDsOnFAe7Y0sMn50lrABQEIiKzlkpn+dLPN7Oqs3netAYAEpUuQESkVvzlv21hR98o//zJM2pycrmZqEUgIjILv93azfUPvsQn37KOM9YtrnQ5JaUgEBE5hL6RCb58wxOsX97OH513fKXLKbnIgsDMmszsITPbbGZPmdlfTLPNRWbWa2aPhbdLoqpHRGQu8nnnf974BENjGa688BQaE7U7lcRMohwjmADOcfcRM0sC95rZr9z9gSnb/dTdPxthHSIic/aN27Zyx5Zu/vT817J+eUely4lEZEHg7g6MhA+T4c2j2p+ISKn98N4XuPru7Xz0zKO5eB4dJTRVpGMEZhY3s8eAHuA37v7gNJt9wMweN7MbzGzNDK9zqZltNLONvb29UZYsIgLALZt387Vbn+Zdr1vOn/3Xk2rugvSHI9IgcPecu58CrAZON7PXTdnkFmCtu58M3AFcO8PrXO3uG9x9Q1dXV5Qli4hw37Z9/NHPNnP62kV8+3+cQjw2f0MAynTUkLsPAncB75yyvM/dJ8KH3wdOK0c9IiIzuW/bPi6+9mHWLmnh+x/dUNPXGZitKI8a6jKzzvB+M/AOYOuUbYov6XMBsCWqekREDuWe53r5+D8+zFGLWrjukjNY0DJ/Tho7mCiPGloBXGtmcYLA+Zm732pmVwAb3f1m4DIzuwDIAv3ARRHWIyIyo/94tpdP/mgj65a0ct0lb2JxW2OlSyobCw7uqR0bNmzwjRs3VroMEZkn3J2fbdzJn/7rUxzX1caPL3kTi1obKl1WyZnZJnffMN06zTUkInVrMJXm8l88wa+e3MtZxy7m7z58Kp0t8y8EDkVBMINMLh/enFzeGcvkGBrLBLfxLNlcHgfcIRE3jl7cwtrFrXUxsCQyH9z3fHBkUO/wBH/yrvVc+pZ1xOb50UEzURAAL/aNcs9z+3hm7zDb942wvXeUPfvHD/t1zGD1wmaWtjcRN8MMYmY4jvuBs+mMYHksBo2JOAuak5O3Ra0NLGptYHFrA0vaG1mzsIXmBoWLSKnsGkjxzdue4ZbNu1m3pJWb/uBs/tPqBZUuq6LqJgie3j3EjY/soikZozERpykZY9fAGHc/28uOvhQA7U0J1nW1cea6xRy1uIWmZJxEzEjEjKZk8IHd0ZykvSlBMh7DDAwjnc3zQt8o23tH2NYzQv9oGnfIu5PN5zGM8D8gCIRsPo/nYDCV4dnuYfaPZRgez05b+/KOJo5e3MLKzmY6mhKTNTQm4sRjRjJuxMyIxywMmKDmZDxGMm40JGI0xGM0JGKTyzI5J5tz0rk844XWzniGobEsjhOPxYLfPW40J+M0JeM0J+O0NiZY2JKks6WBzpYkybjmLZTaMDKR5aq7nuf792wH4LJzjuNTbzuWloa6+RicUd28Ay/1j/KTh15iPJsnlw++mzcn45x57GIuOmstbz2+i2OWtM757MFSfKPI5vIMjmXoH03TN5KmZ3icl/pS7OhL8WLfKA/v6Gd4PMvQeIZqGeNva0xMhlNbY4J4zCZveXfGM3nG0jnGszkMSMRiJOLB+nQ2z0Q2z0QmRyxmLG5rZHHYImpIxCZ/x5hBW1OCjqbk5L46moJQXtCcYEFzAwuakzQkFEryalv2DHHdgy/yL4/uZmQiy3tPWcmX37meVZ3NlS6tatTlUUPZXJ7xbH7yW3Ktyeed0XSWdBhq2XwwjpH3ws+gxZHJBt/409n85JhHOpsnk3eShRZDIkZTIsaCluDDtb0pQcyMbN7JhmMkE9kcY5kcY+kcw+NZBscyDKbSDIxm2D/ZksgwMpElm3fyeSfnTswKrYkYjeHYSTYX1JzLOw2JoHXWmIiRyeXpG01PhmA2DGuz4Pcdngh+34MphFJTQ5zGRJyGRIzWhjidLUkWNDeELZkkC1saWFjUoknEg/eiIR6jKay3KRnHHdLh+5Z3Z1FLAwm1gKqeu/NM9zB3P9vLr57cy6MvDdKYiPGek1dw0VlrOXl1Z6VLrAgdNTRFIh6jrYb/oGMxm1dXR5qt8UwQREE3WhBC+8MB/MFUhsHw8XgmF7Q0snlGJ7I8szfoehtMZSYDZi7MYElbI0vbG1nS1viKLrIFzUnai1os7YUWTHOSpmRssqsw70FLdL5PWVBO2VyeZ7tHeHTnAI+8OMi923rpHgomLFi/vJ3/9Z4T+eBpq+vyaKDZqssgkNrUFI5VdLXP7UQfd2dkIsvAaIaBVJrBscxkqyebD1pL45lgzGQ8myNmhZZC8KHdO5KmZ2ic7qFx9o2k2b5vhMHRDMMT04/tzMQMOpuTLGxtoLM5SWtjguZknJaGOMl4jLwHtebcSaWD8Zvh8Syj6WwQJvlg/YHXC+prbojT3pSgrTFBe1OCpkScxmTQ4jKD1ESOVCbHWDoLGA2JAy2hhkRssoUWj3GgZZeHZCJo2RVqbGtK0NYYhN1k4DUlaWtKlCzgsrk8A6kMfaMT9I+k6U8FrcX+0TQDo+nJ1mP/aJqX+lOk0jkAFrU2cMa6Rfzn47t46/FdrFig7p/ZUBBI3TALWlLtTUmOWtxSstfN5PIMj2cZHg8+sAuHGBe6zCay+WAQP/yMHJ3IMpDK0J9KM5hKMzKRpXd4glQ6RyaXnzyirNC11tGcZGVnE63hGIxh4YEKTB7C7B4c4jwykWVoPMvuwTEmwmCbyORwoLUxTktDEDpwoNsrnT1wmwi77uLhAQdxM9K5oHU1G4VutSCEglZ3cMRcEFxxCw5oMAv2n5rIkSoaQ7LwfcrmfcZxsI6mBIvbGlnU2sCaRS2csW4xbziqk1PWdHLUopZ5PUtoVBQEIkcoGY9NHvY7X+Xzzng2x+hEEDaF0BseD0MvbLWMZXJBiyrsngMmP+CByXEsd2hIxGhpCFoZhfNvCl1oiXiMJW0NLG4NPvCXtDVMtqA0TlN6CgIROaRYzGhpSNDSkJhz15xUL0WriEidUxCIiNQ5BYGISJ1TEIiI1DkFgYhInVMQiIjUOQWBiEidUxCIiNS5mpt91Mx6gUFg/5RVCw6x7FD3i5ctAfYdZmnT7X8260tV91xqPlhdh1o/dfnBHqvuQ9d1qPVzqXu6Zar70OsP52+y+HGp6o7qs+Rod++a9hXdveZuwNWHu+xQ96cs21iKmmazvlR1z6XmUtZ9sMequzJ1z7BMdR9i/eH8TUZRd1SfJQe71WrX0C1zWHao+9M9/0hrms36+VL3wR6r7pn3N9v1c6l7pt9lLuqp7sP5myx+XKq6o/osmVHNdQ2Vg5lt9Bku4FCtarFmUN3lprrLq1bqrtUWQdSurnQBc1CLNYPqLjfVXV41UbdaBCIidU4tAhGROqcgEBGpc/M6CMzsH8ysx8yenMNzTzOzJ8xsm5l914quf2dmnzOzZ8zsKTP7m9JWHU3dZvbnZvaymT0W3t5dC3UXrf+SmbmZLSldxZOvHcX7/TUzezx8r283s5U1Uve3zGxrWPtNZtZZI3V/KPx7zJtZyQZnj6TWGV7vY2b2XHj7WNHyg/77j9xcjs2tlRvwVuBU4Mk5PPch4EyCK+39CnhXuPztwB1AY/h4aY3U/efAl2rt/Q7XrQF+DbwILKmFuoGOom0uA66qkbrPAxLh/W8C36yRuk8ETgDuAjZUutawjrVTli0Ctoc/F4b3Fx7s9yrXbV63CNz9bqC/eJmZHWtmt5nZJjO7x8zWT32ema0g+EO+34P/Sz8C3heu/jTwDXefCPfRUyN1Ry7Cur8NfBmI5MiGKOp296GiTVujqD2ium9392y46QPA6hqpe4u7P1Mttc7gvwC/cfd+dx8AfgO8s9J/tzDPu4ZmcDXwOXc/DfgS8HfTbLMK2FX0eFe4DOB44C1m9qCZ/YeZvTHSag840roBPhs2+f/BzBZGV+orHFHdZnYB8LK7b4660CmO+P02s6+b2U7gw8BXI6y1WCn+nRRcTPDttBxKWXfUZlPrdFYBO4seF+qv+O9VVxevN7M24Czg50VdcNNdiXu6/rnCN7oEQbPuDOCNwM/MbF2Y5JEoUd1/D3wtfPw14G8J/tAjc6R1m1kL8BWC7oqyKdH7jbt/BfiKmV0OfBb4sxKX+spiSlR3+FpfAbLAdaWscTqlrDtqB6vVzD4OfD5cdhzwSzNLAy+4+/uZuf6K/151FQQELaBBdz+leKGZxYFN4cObCT40i5vEq4Hd4f1dwC/CD/6HzCxPMLFUbzXX7e7dRc/7PnBrhPUWHGndxwLHAJvDP7rVwCNmdrq7763iuqe6Hvg3Ig4CSlR3OIh5PnBulF9wipT6/Y7StLUCuPs1wDUAZnYXcJG77yjaZBfwtqLHqwnGEnZR6d+rnAMSlbgBayka6AHuAz4U3jfg9TM872GCb/2FwZt3h8s/BVwR3j+eoKlnNVD3iqJt/hD4SS2831O22UEEg8URvd+vKdrmc8ANNVL3O4Gnga4o6o363wklHiyea63MPFj8AkGPwsLw/qLZ/F5R38q2o0rcgH8G9gAZgtT9BME3zNuAzeE/+K/O8NwNwJPA88D3OHAWdgPw43DdI8A5NVL3PwFPAI8TfLtaUQt1T9lmB9EcNRTF+31juPxxgom/VtVI3dsIvtw8Ft6iONopirrfH77WBNAN/LqStTJNEITLLw7f423Axw/n33+UN00xISJS5+rxqCERESmiIBARqXMKAhGROqcgEBGpcwoCEZE6pyCQecHMRsq8vx+Y2WtL9Fo5C2YpfdLMbjnUjJ9m1mlmf1CKfYuArlAm84SZjbh7WwlfL+EHJl+LVHHtZnYt8Ky7f/0g268FbnX315WjPpn/1CKQecvMuszsRjN7OLydHS4/3czuM7NHw58nhMsvMrOfm9ktwO1m9jYzu8vMbrBgjv7rCvPEh8s3hPdHwgnmNpvZA2a2LFx+bPj4YTO7Ypatlvs5MOFem5ndaWaPWDBX/XvDbb4BHBu2Ir4VbvvH4X4eN7O/KOHbKHVAQSDz2XeAb7v7G4EPAD8Il28F3urubyCYFfSvip5zJvAxdz8nfPwG4AvAa4F1wNnT7KcVeMDdXw/cDXyyaP/fCfd/yLljwrl1ziU48xtgHHi/u59KcB2Mvw2D6E+A5939FHf/YzM7D3gNcDpwCnCamb31UPsTKai3SeekvrwDeG3RLJEdZtYOLACuNbPXEMzymCx6zm/cvXj++YfcfReAmT1GMO/MvVP2k+bAJH6bgN8L75/JgXnlrwf+9wx1Nhe99iaCeeohmHfmr8IP9TxBS2HZNM8/L7w9Gj5uIwiGu2fYn8grKAhkPosBZ7r7WPFCM/u/wL+7+/vD/va7ilaPTnmNiaL7Oab/m8n4gcG2mbY5mDF3P8XMFhAEymeA7xJcx6ALOM3dM2a2A2ia5vkG/LW7/7/D3K8IoK4hmd9uJ7gOAABmVpg6eAHwcnj/ogj3/wBBlxTAhYfa2N33E1zW8ktmliSosycMgbcDR4ebDgPtRU/9NXBxOFc+ZrbKzJaW6HeQOqAgkPmixcx2Fd2+SPChuiEcQH2aYApxgL8B/trMfgfEI6zpC3Q4bqMAAACLSURBVMAXzewhYAWw/1BPcPdHCWa1vJDgojAbzGwjQetga7hNH/C78HDTb7n77QRdT/eb2RPADbwyKEQOSoePikQkvMLamLu7mV0I/L67v/dQzxMpN40RiETnNOB74ZE+g0R8aVCRuVKLQESkzmmMQESkzikIRETqnIJARKTOKQhEROqcgkBEpM79f5hUz1kLlf9DAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lr = 3e-4\n",
    "lr *= bs/48  # Scale learning rate by batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.068131</td>\n",
       "      <td>3.126621</td>\n",
       "      <td>0.609930</td>\n",
       "      <td>0.390071</td>\n",
       "      <td>22.796721</td>\n",
       "      <td>28:29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.995493</td>\n",
       "      <td>3.070316</td>\n",
       "      <td>0.602313</td>\n",
       "      <td>0.397686</td>\n",
       "      <td>21.548635</td>\n",
       "      <td>28:26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.011955</td>\n",
       "      <td>3.043591</td>\n",
       "      <td>0.598430</td>\n",
       "      <td>0.401570</td>\n",
       "      <td>20.980440</td>\n",
       "      <td>28:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.978858</td>\n",
       "      <td>3.023281</td>\n",
       "      <td>0.595229</td>\n",
       "      <td>0.404770</td>\n",
       "      <td>20.558710</td>\n",
       "      <td>28:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.960861</td>\n",
       "      <td>3.005773</td>\n",
       "      <td>0.592423</td>\n",
       "      <td>0.407577</td>\n",
       "      <td>20.201872</td>\n",
       "      <td>28:26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.940080</td>\n",
       "      <td>2.995852</td>\n",
       "      <td>0.590565</td>\n",
       "      <td>0.409436</td>\n",
       "      <td>20.002399</td>\n",
       "      <td>28:27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.937475</td>\n",
       "      <td>2.993789</td>\n",
       "      <td>0.590111</td>\n",
       "      <td>0.409888</td>\n",
       "      <td>19.961227</td>\n",
       "      <td>28:23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxV9Z3/8dfnLtlISCAECCCLiggqAiJitQ7WHddpaYvTRa0t87O1Wh+1U7s7ThftPtZWq1Pbaq2tg7XVaV1bKbUKGhSQRVZBwh7WBMhycz+/P+5JTEJWcpOQnPfz8biPe+73fO853y83vM9+jrk7IiISHpGeboCIiHQvBb+ISMgo+EVEQkbBLyISMgp+EZGQifXYjHPyfdKEsT01exGRXmnRokVl7l7UmWn0WPDHC4ZQUlLSU7MXEemVzGxjZ6fRY7t6HF0/ICLSE9oMfjPLMrNXzWyJmS03s/9sps61ZrbTzBYHr0+2OWflvohIj2jPrp4q4H3uXmFmceAlM3va3Rc0qfd7d78x/U0UEZF0ajP4PXVPh4rgYzx4aX1dRLpdTU0NpaWlVFZW9nRTulxWVhYjRowgHo+nfdrtOrhrZlFgEXA88FN3X9hMtQ+Y2TnAauAWd9/UzHTmAHMAMoYef8SNFpFwKi0tJS8vj9GjR2NmPd2cLuPu7Nq1i9LSUsaMGZP26bfr4K6717r7JGAEMM3MTm5S5SlgtLtPBF4Aft3CdO5396nuPrUzjRaRcKqsrKSwsLBPhz6AmVFYWNhlWzYdOqvH3fcC84CLm5Tvcveq4OMDwGlpaZ2ISBN9PfTrdGU/23NWT5GZFQTD2cD5wFtN6hQ3+HgFsDKdjRQRkfRpzxp/MfCimS0FXgOed/f/M7M7zOyKoM5NwameS4CbgGvbM3M9C0BEepO9e/fys5/9rMPfmzlzJnv37u2CFh0Z66nwzSwe64c2ryYSCcdmm4h03sqVKxk/fnyPzX/Dhg1cdtllLFu2rFF5bW0t0Wg07fNrrr9mtqizx0l77JYNoHNCRaR3ue2221i3bh2TJk0iHo+Tm5tLcXExixcvZsWKFVx11VVs2rSJyspKbr75ZubMmQPA6NGjKSkpoaKigksuuYSzzz6bl19+meHDh/OnP/2J7Ozsbu1HjwZ/0p0oWuMXkY77z6eWs2LL/rROc8Kw/nzj8pNaHH/nnXeybNkyFi9ezLx587j00ktZtmxZ/SmXDz74IAMHDuTQoUOcfvrpfOADH6CwsLDRNNasWcOjjz7KAw88wIc+9CEef/xxPvrRj6a1H23p2TV+rfKLSC82bdq0RufZ33333TzxxBMAbNq0iTVr1hwW/GPGjGHSpEkAnHbaaWzYsKHb2lunR4N/y95DjB7UryebICK9VGtr5t2lX79382vevHm88MILvPLKK+Tk5DBjxoxmz8PPzMysH45Goxw6dKhb2tpQjz6IZcb35zFv1Y6ebIKISLvl5eVRXl7e7Lh9+/YxYMAAcnJyeOutt1iwoOntzI4ePf4Ermt/+RoPvvQ21YlkTzdFRKRVhYWFnHXWWZx88sl84QtfaDTu4osvJpFIMHHiRL72ta8xffr0Hmpl23r0dM7ia37cqOz88UO4/2OnkXQnFu3xZZKIHGV6+nTO7tYnT+ds6oWV2zn2y385rPzl297HsILuPd1JRKSvOqqCvyWfffQNZp02goxohHFD85i3agc3vk/P6xURORK9IvgXbdzDoo17GpUdPziXeDTCcUW5jB7Uj5raJHc8tYKbzhtLUV5mC1MSEZFeEfzN+X+/eb1++Fv/ejJfeSJ1CfW81TuY/4Vz2bKvkuHaPSQicpgeDf5TR+RzbFEuT7yxuVPTqQt9gE27DzHmS+8eJ/j7F2YQMSMejTA0P4ua2iTukBHTwWMRCaceDf7vvH8iE4b173Twt+Zfvjev2fIHPj6VCyYM6bL5iogcrXpstXdo/yzGF+cBqTV/gJKvns+rXzmPj00f1eXz/9RDJdz6v0t4cdUOPvvoGxysTgCwals5P3x+tW4ZLSKdlpubC8CWLVuYNWtWs3VmzJhBSUlJdzar59b4i/Iy658w8/gN76HWncxY6raml586jIcXbKQ4P4uyiirGDOrH6u0VrU3uiMxdVMrcRaUAPLVkC8MLstm8N3X59DPLtnLy8HxuPm8sa3dUcP2vUz/MhjsvTXs7RKRvGzZsGHPnzu3pZtQ7Kg7uxqKRRg2ZNmZgswGbTKbWwj/5UAl/eyv9t3qoC32A1dsrWL29gj+83ng31Jrt5YwamEM8FunQo9F2VVTx6tu7ueSU4rYri8hR6Ytf/CKjRo3i05/+NAC33347Zsb8+fPZs2cPNTU1fPOb3+TKK69s9L2G9/E/dOgQ1113HStWrGD8+PE9cq+eoyL426vuoS0PXns6AK+s28XVD6Tuh/HL606nojLBZx99o0vbcMGP5vOL+Pc4PbKK3AFFVMfzORTNI9KvkKz+gyjZ7owaMZwRw4ZD9kDIHgDZA7jwByXsJZdaoqz+5iVtHlyuqEqQm9mrfh6R7vX0bbDtzfROc+gpcMmdLY6ePXs2n/vc5+qD/7HHHuOZZ57hlltuoX///pSVlTF9+nSuuOKKFlcM7733XnJycli6dClLly5lypQp6e1DO/TqZDnzuELWfXsmtUmvD9LLTx3G6Nv+3KXzfSZ5Ou/4YPJ3HaCACgbYdvJZR4FVMJ0DRLccfnxgUVbqfb9nU3FXAZ5dwJryDHYmsnnf5HGsr8igsGgIuz2XaM5AvvrsZq49fzK7anO55n2nYrEM3L3RH1Nt0tlVUcUvX97A5y84Qbe5EOlikydPZseOHWzZsoWdO3cyYMAAiouLueWWW5g/fz6RSITNmzezfft2hg4d2uw05s+fz0033QTAxIkTmThxYnd2AejlwQ8QjRjRJo9vXP/tmSTdOf4rTwPw2lfO56klW9hzsJqf/G0t7x07iH+sKTvief5v7YwWxxlJ8jhEgVVQQEXwfoACKw/eK8ivrGBAZWrcBNtB5ZLlTKCC6HpnWDCdxzOBfwQfXoZDlsOuZA57PZe95JKRW0hZbT/WH4hT7bn8ak0xC7Y6s84+mf2WxwfPnsh352/ng9OO5diiXBZt3MOC9bv4zLnHt9q3Hzy3isJ+GVx71phW64n0uFbWzLvSrFmzmDt3Ltu2bWP27Nk88sgj7Ny5k0WLFhGPxxk9enSzt2NuqCO7ibtCrw/+5kQiRgTj6mkjefTVdyjKy+QTZ6eC7PMXjgNS+9xP++YLaZ+3E2E//djv/XiHIe1+vmTdAiO/fkFRwQAqgs8VFFjjhUdBxWqOswouilYQNYdd8MkM4NVgggvhi0DFq1mUkkum53Kq92P9ymGMOWYEltEP4tkQy0q94lkQy2bDvBW8RQbXDjm7UTmxzMb1Y1kQSW1h7DtYQzRqre6aemHFds4eO4isePqfS9oZizft5bpfvsqLt86gICejp5sjvcDs2bP51Kc+RVlZGX//+9957LHHGDx4MPF4nBdffJGNGze2+v1zzjmHRx55hHPPPZdly5axdOnSbmr5u/pk8Nf51lUnc8eVzT+soTA3kw13Xsr+yhr6Z8WpTTr/9sACvjRzPEP6Z3Lmd/7WrW1tuMDYlCpoFyNJLpXkBwuKui2NfKvbDZUqyw8WHr5tGbu2L6QwXgOJSqzJjH5Sl32/+WHbbY5mcjAZoyoZp9LjbI9kcGzxIJLRLCLxLCyejcey2Livlh0bD7CgaAAnjRxMJCOb+1/eShVxrp8xgSeXl9EvN49rzxkPsUz21UTZfADKE1E+/8RqThk9hHuveQ/EsqlIGDkZsfrjPZ31/WdXsedgDW+8s5dzTxyclmlK33bSSSdRXl7O8OHDKS4u5iMf+QiXX345U6dOZdKkSZx44omtfv+GG27guuuuY+LEiUyaNIlp06Z1U8vf1eZtmc0sC5gPZJJaUMx19280qZMJPAScBuwCPuzuG1qb7tSpU727z13tqLfLDnCwOsHzK7azfMt+nl+xvaeblGZOBgmyqObHs8bz9cdLyKKaTGpS75Z6ry+zxsP19RrUHZSZpKbqEFlW0+y0UnUTR9ziWjeS0UziGVmprZBoJkTjwXA8+JwBsYwm4zJIWJzFWw9y6qjBxDNS373z+fXUEOPsE4dz7oThjafT7DQz8GjqvTIZIzMri0gw/f/+2zrKKqr4r6tObv5fu8kxGuk43Za5+27LXAW8z90rzCwOvGRmT7t7w8fLXA/scffjzWw2cBfw4c407GgwJngs5EnD8g8b19UHkLuHUU2cauJ8Yu47QJM13iO5hq2mPXNNNlhoNF6IHLbAabjAqFv41NYwKifGReMGQG0NyUQVa7aUkTxUzeiCGNkkoaqcysodJKqroLaKftEke/aVM5Ya2JHEvQbzWm6LB41aF7zaoS66m94J6tMepZoYe96Ik9cvh1g8C4/GqbU4Fs9k8ZaDeCTO0IH5jBiUD5FYaoESzYBoarjskFPYvx/WpJxoRpP68eCVAZEGw+0pj2ZA5Oja5Sbdq83g99QmQd3VU/Hg1TQSrgRuD4bnAveYmXmILn89cWgeb20r5y83vZeivExO/1bj4wdfvPhE7nrmrUZlZ4wZyMK3d3dnM48KToRKMqkks66gaYW2lUHm3ghVTZ/ctrP97YiQJE6CDBJkUEMGCeL27nBdeaqswWcSZNi7w5kkgrJgGiTI2F9DhtW+W58EGWSSkUyws2wn/X0/2ZFa4iSgtgZqa6iqriReWUltzIl6DZY88i2jtjiGRTNIBguTSOzwBcWqskrKayKcfEwh+6uNNbuqOHZIAYMK8ohHY6mFh0VTx3oisWA4+u57w+HgvTJpmEXIzMhotV5qepHDy2oGQVXdow8N6reggnezJsOpvlYmkmTHo8F3mv5rtLYVZs0Otvndo3zLrl37+M0sCiwCjgd+6u4Lm1QZDsGuafeEme0DCoGyJtOZA8wBGDlyZOda3sPqLjB7bcNucjNjjC/u32y9L1w0jo9OH0V+dpwLJgxhSP9M8rLi9eNLNuzmvr+vY+Ypxby5eR+//OeGRt+/+byx/Pdf1wAwcUQ+S0v3AfCd95/Cl/6Q5nOYe5nDQr+DkkSoIoMqGhzUbW6h0xWrL1sbf/ztJ8/gpbVl/GzeutQ2NnDbxeO4dvpwptz+F2IkyCC1oIgFC6I4tfSLOZ6oIk4tMUvUL2RiQd3BOcbgfjGKcyO8sWEH8aA8bgniiWA4mFacBOedMIDMSJJ/rtpCnCgxalmyaQ9xEuRRy97NOzi4OcGYwmyStQmiJKmqqSFmToxaSNaCJ0kmE5gnsWQCT9bWH0vK6uy/20W/x8s6tsvMOHzrrGe11fbUwsndYe92+Pb5DcrTs0Dp0KMXzawAeAL4rLsva1C+HLjI3UuDz+uAae6+q6Vp9YZ9/J1x77x13PXMW6z91iUdOr/+oh/NZ9X2cn5z/RkMysvgxKH9efXt3Swt3csn33ssizbu5rii3PozUJ5csoXxQ/O44EfzARg5MIe7PjCx/sI2gEsnFvPnpamkGTOoH2+XHUhjT0VSMmIRPnvu8fzg+dXNjHWiJINXLXd/+FQqq6p4bX0Z+w5WsmDtDqIkiViSr15yAt99egVRkhxXmMV/f3gis372EoP7xbh4ygmMP24k2Tm5WP3Ku/Puer5TnJ9FeWUNh6oTeDAOwILh/lkxsjOiROtDtHEGViWS7D5QTV5mjH6ZMZLuJGqTZMYiHKyuJR5cM7SroopB/TKIRSMcqqklUZskJ5iuA0l3IkEbk5667iZisOtANfnZceJRY9+hGvKyYlTVJEkkk+RlxYkAh2oSlO07wMHtbzOu/GWSySRLS/cyoTiPzMu/1+l9/B1+5q6ZfQM44O7fb1D2LHC7u79iZjFgG1DU2q6evh78R+pQdS2HamoZ2K9zpxaOvu3PDOmfyT/+431kxCJs21dJQU6crHiUKf/1PLsPVLc5jZZ2RZ07rogXV3Vgn4pImvTPjPDZMwYwqiCOtbnm3LqivEx2lqc2r6IRY0BOnOpEkv2VHdvFNrBfRrv+PzVVkBNn78HmD4o5zsa9Nfxk4R72VzXest1412Vdf3DXzIqAGnffa2bZwPmkDt429CRwDfAKMAv4W5j276dTdkaU7IzOH3hreq+jofnvbmSfPDyf+atTwX3BhCEMys3kjitPorKmFoADVbW8uXlf/W2ra5POn9/cyk3B7TB+ed20+i2TpbdfyMTbn2PBl85j+nf+2miexflZzL3hPdQkkmzYdYAZ4wYfdv3Ee8cO4uHrz2D5ln1cevdLHDMwm027U/cuec9xhZx1/CC+9+wqAK44dRg//vCkZp/LLOGwvyrJt+a3uCNB2qk9+/iLgV8H+/kjwGPu/n9mdgdQ4u5PAr8AHjaztcBuYHaXtVg6rTDYmnjoE9M454Si+vJ4sEsqLyveaEERjRiXTyymvLKGqyYNB+DZW86pH1+3kPnuByYSjRib9hzkxy+s4abzxtY/BW10cIZUYW4mK++4GDN4Z/dBThiSujX3ScPy66fz6tu7+dDPX2FAvww+c+7xnDFmICcMzaN/g2MjR2JCcX9WbN3f4vg/fuYsrvrpP49o2tefPYZfvPT2kTZNpFt1eFdPumhXT8/Zd6iG3y58h38/59i0XQjVUGVNLb9ZsJHrzhpz2O002sPdefCfG5h12gjysw8P+7pTaTfceWn98Io7LmLC158F4HdzpvPc8u08+M93g/ja94zmG5dP4KW1ZXz6N6/zsTNH8fP563n0U9OZNmZgfb1k0vnpi2v5wfOrefLGszhleD5vbt7HW9vK+Y+5qSss7/volPpHf755+4X1B+v3V6b+Xa99z2hO/Noz9dP84Gkj+N4HT2XfwRr+/TclLFif/jO5LptYzP8t3dp2Ren10rGrR8Evvc5La8oYMSCb0YP6UZ1IEotYswuwv67czqjCHI4fnNeh6bs7FVWJRmdfNVW65yA7yquYMnJAs+OfWrKFl9ft4tYLT6AwN7PRuNauAbn1whP4/nOND47+4dPvYVh+NjW1Sd773Rfry+u2UK6aNIwfz57M6Nv+zIxxRXx55niOK8rluAa7xIb0z+Rz55/ApROL67ecHn5lA1/703IgdTxnaH4Wf1q8pdG8//Ef56YOoppx1p3dezW7NE/BL9IL/eC5VUweWcAnflXCv5xQxN+D4y3rvj2TaMR4askWRg7MYVBeJqu3lTe6lUQy6dS6s+dANYP7t35yZE1tkpsefYPapHP/x1vOiboripNJZ/WOcnZXVDNuaN5hC6xnl29j695D3P7UimanM/3YgS1uzbx46wzO/f48AM48tpBX1r+7n379t2dSXpng1Duea7U/R+KTZ4/hf/rYLjgFv0gfsHLrfiJmjBvasS2TnlK3xTLv1hn86uUNfP2yCY22uHaWV/H7194hFo3woanH0C8zWv90vTqVNbW8/s4ehuVn1x//eeKNUjJjUe54agX7K2v44YcmcdqoARTlZXKoupb9lTVkxaJsL69k466DDMrN4K8rd3DrReMoq6hi275Kxg7JZdGGPfzb/yzk71+YwYgBOXz1j28yY9xg/v3hRdzzb5O58betP7Nj8sgC3nhnb6OyDXdeyuJNe8nJiDKsIJsZ35tHWUVVozqvfuU8pn0rdYLDR84YySML3+H9U4ZTUZngueB2L6MLc9iw6yA3nns8Y4fkcu6Jg1m6aR+/e+2dw3bVTR5ZwITi/jyy8J36svdPHs6PZk9W8ItI93pr236GFWR3+mB7T1m9vZwh/bN4dvk2cjKi3PjbN7jopCH85OopJN3Jikc5WJ0gFolwz9/WcMGEoZwyovFtW8orazjl9tQWSl5mjJmnFHPXrIks27yPNTvK+dfJIxrVr6ypZdW2ciaOyMedFo+t/WzeWk4bOYAzji0EUltjZRXVLNq4hz0Hq7l62si03KtHwS8iofbXlduZNmZgq8d0mvObBRsZNzSP00cPbLtyGnXXTdpERPqs88YPOaLvfXT6qDS3pPvoWX0iIiGj4BcRCRkFv4hIyCj4RURCRsEvIhIyCn4RkZBR8IuIhIyCX0QkZBT8IiIho+AXEQkZBb+ISMgo+EVEQkbBLyISMgp+EZGQUfCLiISMgl9EJGTaDH4zO8bMXjSzlWa23MxubqbODDPbZ2aLg9fXu6a5IiLSWe15AlcC+Ly7v25mecAiM3ve3Vc0qfcPd78s/U0UEZF0anON3923uvvrwXA5sBIY3tUNExGRrtGhffxmNhqYDCxsZvSZZrbEzJ42s5Na+P4cMysxs5KdO3d2uLEiItJ57Q5+M8sFHgc+5+77m4x+HRjl7qcCPwH+2Nw03P1+d5/q7lOLioqOtM0iItIJ7Qp+M4uTCv1H3P0PTce7+353rwiG/wLEzWxQWlsqIiJp0Z6zegz4BbDS3X/YQp2hQT3MbFow3V3pbKiIiKRHe87qOQv4GPCmmS0Oyr4MjARw9/uAWcANZpYADgGz3d27oL0iItJJbQa/u78EWBt17gHuSVejRESk6+jKXRGRkFHwi4iEjIJfRCRkFPwiIiGj4BcRCRkFv4hIyCj4RURCRsEvIhIyCn4RkZBR8IuIhIyCX0QkZBT8IiIho+AXEQkZBb+ISMgo+EVEQkbBLyISMgp+EZGQUfCLiISMgl9EJGQU/CIiIaPgFxEJmTaD38yOMbMXzWylmS03s5ubqWNmdreZrTWzpWY2pWuaKyIinRVrR50E8Hl3f93M8oBFZva8u69oUOcSYGzwOgO4N3gXEZGjTJtr/O6+1d1fD4bLgZXA8CbVrgQe8pQFQIGZFae9tSIi0mkd2sdvZqOBycDCJqOGA5safC7l8IUDZjbHzErMrGTnzp0da6mIiKRFu4PfzHKBx4HPufv+pqOb+YofVuB+v7tPdfepRUVFHWupiIikRbuC38zipEL/EXf/QzNVSoFjGnweAWzpfPNERCTd2nNWjwG/AFa6+w9bqPYk8PHg7J7pwD5335rGdoqISJq056yes4CPAW+a2eKg7MvASAB3vw/4CzATWAscBK5Lf1NFRCQd2gx+d3+J5vfhN6zjwGfS1SgREek6unJXRCRkFPwiIiGj4BcRCRkFv4hIyCj4RURCRsEvIhIyCn4RkZBR8IuIhIyCX0QkZBT8IiIho+AXEQkZBb+ISMgo+EVEQkbBLyISMgp+EZGQUfCLiISMgl9EJGQU/CIiIaPgFxEJGQW/iEjIKPhFREKmzeA3swfNbIeZLWth/Awz22dmi4PX19PfTBERSZdYO+r8CrgHeKiVOv9w98vS0iIREelSba7xu/t8YHc3tEVERLpBuvbxn2lmS8zsaTM7qaVKZjbHzErMrGTnzp1pmrWIiHREOoL/dWCUu58K/AT4Y0sV3f1+d5/q7lOLiorSMGsREemoTge/u+9394pg+C9A3MwGdbplIiLSJTod/GY21MwsGJ4WTHNXZ6crIiJdo82zeszsUWAGMMjMSoFvAHEAd78PmAXcYGYJ4BAw2929y1osIiKd0mbwu/vVbYy/h9TpniIi0gvoyl0RkZBR8IuIhIyCX0QkZBT8IiIho+AXEQkZBb+ISMgo+EVEQkbBLyISMgp+EZGQUfCLiISMgl9EJGQU/CIiIaPgFxEJGQW/iEjIKPhFREJGwS8iEjIKfhGRkFHwi4iEjIJfRCRkFPwiIiGj4BcRCZk2g9/MHjSzHWa2rIXxZmZ3m9laM1tqZlPS30wREUmX9qzx/wq4uJXxlwBjg9cc4N7ON0tERLpKm8Hv7vOB3a1UuRJ4yFMWAAVmVpyuBoqISHqlYx//cGBTg8+lQdlhzGyOmZWYWcnOnTvTMGsREemodAS/NVPmzVV09/vdfaq7Ty0qKkrDrEVEpKPSEfylwDENPo8AtqRhuiIi0gXSEfxPAh8Pzu6ZDuxz961pmK6IiHSBWFsVzOxRYAYwyMxKgW8AcQB3vw/4CzATWAscBK7rqsaKiEjntRn87n51G+Md+EzaWiQiIl1KV+6KiISMgl9EJGQU/CIiIaPgFxEJGQW/iEjIKPhFREJGwS8iEjIKfhGRkFHwi4iEjIJfRCRkFPwiIiGj4BcRCRkFv4hIyCj4RURCRsEvIhIyCn4RkZBR8IuIhIyCX0QkZBT8IiIho+AXEQkZBb+ISMi0K/jN7GIzW2Vma83stmbGX2tmO81scfD6ZPqbKiIi6RBrq4KZRYGfAhcApcBrZvaku69oUvX37n5jF7RRRETSqD1r/NOAte6+3t2rgd8BV3Zts0REpKu0J/iHA5safC4Nypr6gJktNbO5ZnZMWlonIiJp157gt2bKvMnnp4DR7j4ReAH4dbMTMptjZiVmVrJz586OtVRERNKiPcFfCjRcgx8BbGlYwd13uXtV8PEB4LTmJuTu97v7VHefWlRUdCTtFRGRTmpP8L8GjDWzMWaWAcwGnmxYwcyKG3y8AliZviaKiEg6tXlWj7snzOxG4FkgCjzo7svN7A6gxN2fBG4ysyuABLAbuLYL2ywiIp1g7k1313ePqVOneklJSY/MW0SktzKzRe4+tTPT0JW7IiIho+AXEQkZBb+ISMgo+EVEQkbBLyISMgp+EZGQUfCLiISMgl9EJGQU/CIiIaPgFxEJGQW/iEjIKPhFREJGwS8iEjIKfhGRkFHwi4iEjIJfRCRkFPwiIiGj4BcRCRkFv4hIyCj4RURCRsEvIhIyCn4RkZBpV/Cb2cVmtsrM1prZbc2MzzSz3wfjF5rZ6HQ3VERE0qPN4DezKPBT4BJgAnC1mU1oUu16YI+7Hw/8CLgr3Q0VEZH0aM8a/zRgrbuvd/dq4HfAlU3qXAn8OhieC5xnZpa+ZoqISLrE2lFnOLCpwedS4IyW6rh7wsz2AYVAWcNKZjYHmBN8rDKzZUfS6F5gEE363kf01X5B3+2b+tX7tNW3UZ2dQXuCv7k1dz+COrj7/cD9AGZW4u5T2zH/Xqev9q2v9gv6bt/Ur96nO/rWnl09pcAxDT6PALa0VMfMYkA+sDsdDRQRkfRqT/C/Bow1szFmlgHMBp5sUudJ4JpgeBbwN3c/bI1fRER6Xpu7eoJ99jcCzwJR4EF3X25mdwAl7j1xFRMAAASrSURBVP4k8AvgYTNbS2pNf3Y75n1/J9p9tOurfeur/YK+2zf1q/fp8r6ZVsxFRMJFV+6KiISMgl9EJGR6JPjbugXE0cLMNpjZm2a22MxKgrKBZva8ma0J3gcE5WZmdwd9WmpmUxpM55qg/hozu6ZB+WnB9NcG3+2Si97M7EEz29Hwuonu6EdL8+iGvt1uZpuD322xmc1sMO5LQTtXmdlFDcqb/ZsMTmpYGPTh98EJDl1+mxIzO8bMXjSzlWa23MxuDsp79e/WSr/6wm+WZWavmtmSoG//eaTtSVefW+Tu3foidYB4HXAskAEsASZ0dzva2dYNwKAmZd8FbguGbwPuCoZnAk+TuqZhOrAwKB8IrA/eBwTDA4JxrwJnBt95Griki/pxDjAFWNad/WhpHt3Qt9uBW5upOyH4e8sExgR/h9HW/iaBx4DZwfB9wA3B8KeB+4Lh2cDv09yvYmBKMJwHrA7a36t/t1b61Rd+MwNyg+E4sDD4LTrUnnT2ucW2pvs/Yjv+cc4Enm3w+UvAl7q7He1s6wYOD/5VQHGDP+JVwfDPgaub1gOuBn7eoPznQVkx8FaD8kb1uqAvo2kcjl3ej5bm0Q19u53mQ6TR3xqpM9XObOlvMviPXAbEmv7t1n03GI4F9awLf78/ARf0pd+tSb/61G8G5ACvk7rLQYfak84+t/TqiV09zd0CYngPtKM9HHjOzBZZ6nYTAEPcfStA8D44KG+pX62VlzZT3l26ox8tzaM73Bjs8niwwa6KjvatENjr7okm5Y2mFYyvu01J2gW7ACaTWoPsM79bk35BH/jNzCxqZouBHcDzpNbQO9qedPa5WT0R/O26vcNR4ix3n0LqzqSfMbNzWqnbUr86Wt7T+kI/7gWOAyYBW4EfBOXp7Fu39NvMcoHHgc+5+/7WqrbQnqPyd2umX33iN3P3WnefROoOB9OA8UfQni7/LXsi+NtzC4ijgrtvCd53AE+Q+iG3m1kxQPC+I6jeUr9aKx/RTHl36Y5+tDSPLuXu24P/gEngAVK/G3S8b2VAgaVuQ9KwvNG0rItuU2JmcVLh+Ii7/yEo7vW/W3P96iu/WR133wvMI7WPv6PtSWefm9UTwd+eW0D0ODPrZ2Z5dcPAhcAyGt+e4hpS+ygJyj8enF0xHdgXbCY/C1xoZgOCzdcLSe1/2wqUm9n04GyKjzeYVnfojn60NI8uVRdagX8l9bvVtWd2cDbFGGAsqQOczf5NemqH6YukbkPStA9depuS4N/yF8BKd/9hg1G9+ndrqV995DcrMrOCYDgbOB9YeQTtSWefm9dVB23aOPAxk9TR/HXAV3qiDe1o47GkjpovAZbXtZPU/rS/AmuC94FBuZF6YM064E1gaoNpfQJYG7yua1A+ldQf+DrgHrruQNOjpDafa0itNVzfHf1oaR7d0LeHg7YvDf4TFTeo/5WgnatocBZVS3+Twd/Bq0Gf/xfIDMqzgs9rg/HHprlfZ5PaXF8KLA5eM3v779ZKv/rCbzYReCPowzLg60fannT1uaWXbtkgIhIyunJXRCRkFPwiIiGj4BcRCRkFv4hIyCj4RURCRsEvIhIyCn4RkZD5/1svWqxfHswfAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.3900712728500366.\n",
      "Better model found at epoch 1 with accuracy value: 0.39768603444099426.\n",
      "Better model found at epoch 2 with accuracy value: 0.40157046914100647.\n",
      "Better model found at epoch 3 with accuracy value: 0.40476998686790466.\n",
      "Better model found at epoch 4 with accuracy value: 0.40757742524147034.\n",
      "Better model found at epoch 5 with accuracy value: 0.40943607687950134.\n",
      "Better model found at epoch 6 with accuracy value: 0.4098880887031555.\n",
      "CPU times: user 2h 36min 55s, sys: 42min 44s, total: 3h 19min 39s\n",
      "Wall time: 3h 19min 22s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "learn.unfreeze()\n",
    "wd = 0.01 \n",
    "learn.fit_one_cycle(7, lr, wd=wd, moms=(0.8,0.7), \n",
    "                    callbacks=[ShowGraph(learn),\n",
    "                               SaveModelCallback(learn.to_fp32(), monitor='accuracy', name='bestmodel_sp15')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "fc8bfa72-afa8-4429-b3a7-2e4f8cd14cb8"
    }
   },
   "source": [
    "Save the pretrained model and vocab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true,
    "nbpresent": {
     "id": "00f7bd36-8558-4a49-8cd2-29a0f836430b"
    }
   },
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "learn.to_fp32().save(mdl_path/lm_fns2[0], with_opt=False)\n",
    "learn.data.vocab.save(mdl_path/(lm_fns2[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Backward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data = load_data(path, f'{lang}_databunch_corpus2_100_sp15_bwd', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 832 ms, sys: 36 ms, total: 868 ms\n",
      "Wall time: 868 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, drop_mult=0., pretrained=False, \n",
    "                               metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8dcnewJhD/sSdkSqKAFZFBfcqtalaqut1q36o9qq7bW3vz76u+299fa2tnZvrdW6tVp73epaEduKG4KGVXbQBAhrQiAQAtnm8/tjBglpSALkzJlk3s/HYx4zc873nPPOMOSTs32/5u6IiEjySgk7gIiIhEuFQEQkyakQiIgkORUCEZEkp0IgIpLk0sIOcKR69erl+fn5YccQEWlXFixYUObueU3Na3eFID8/n8LCwrBjiIi0K2a2/nDzdGhIRCTJqRCIiCQ5FQIRkSSnQiAikuRUCEREkpwKgYhIklMhEBFJcu3uPoKjtXrrHl5ZuvlfZ5gdfNnM8tbcTMCaWbq5ZRvPOtDWWtpgrO2B7X6yXOP1NJ5v1kSb2HSLLW9GSmzdB6aZRdukxJZPSSH6OtY25ZBn+2R+ihmpKdHntNTo69TY67SUFFJTjLSU6Pv01BTSUoyMtBTSU1PISE0hJaXlz0FEjk2ghcDM7gBuJvq75EF3/0UTbc4AfgGkA2XufnoQWdZtr+TXb6w7ZJqGYkh8BwpDZloKGWkpZKenkpWeSnZGKtnpqXTKTKNzZhqdMlPpkpVO95wMuuZEn3t1zqB3lyx6dc4gMy017B9FJGEFVgjMbBzRIjAJqAFmmdkr7r62QZtuwH3A+e6+wcx6B5XnwhP6ceEJFx7Vsi0N3tPc7OaWbLxe/2Q6eLNLHrrNA68PLHPw/cHt+IHpTbT1Rm2c6MRI7PWBNpFIdJmIe3Reg+d6dyKRA/Oi0yPuRCJOfeTg/LpIhLr66Pv6iFNbH6E+4tTVO7WRCLV1EeoiTnVdhNr6CDV10Ud1XYTqunqq6yLsq6lnX209+2rq2Vm1j8rqWvZW17N7Xy11kaY/t56dMhjSM4f8Xp0Y2rMTY/t3YcKQ7nTLyWj2cxZJBkHuERwHzHP3KgAzexO4DPhxgzZfAJ5z9w0A7r49wDxHraXDNK04inO4JY92QWmCu1NZXceuqlrK99ZQVllN6Z5qtu+pZkvFPorK9jJ33Q6eW7jpk2VG9enMxPweXDJ+ABPzu7fqkJxIRxNkIVgG/MDMegL7gAuAxp0EjQLSzWwOkAv80t3/2HhFZnYLcAvA4MGDA4ws7ZmZkZuVTm5WOoN65By2XVVNHUtLKliwficfFJfz/KJNPDF/A8PyOnHVxEFcMWEQPTppT0GShwU5ZrGZ3QTcBlQCK4B97v71BvN/AxQAM4Bs4D3gQndfc7h1FhQUuDqdk7ZUVVPHy0u38Jf3N7Bwwy5ys9L43meO5/KTB2gPQToMM1vg7gVNzQv08lF3f8jdT3b36UA5sLZRkxJglrvvdfcy4C3gxCAziTSWk5HG5woG8dyt05h152mM6ZvLXU8v4eY/FrJ99/6w44kELtBCcODkr5kNBj4LPNmoyQvAaWaWZmY5wCnAyiAziTRnTN8u/OWWKfzHRWN5e20Z5/z8LeauKws7lkiggr6h7FkzWwG8BNzm7jvNbKaZzQRw95XALGAp8D7wB3dfFnAmkWalphg3nTqUv91xGr1zM/nKEwvZsKMq7FgigQn0HEEQdI5A4mn9jr1c/Jt36dc1i+dunUpORtLcgykdTGjnCETauyE9O/Grq09izbY9/PszS1u8p0SkPVIhEGnB6aPy+OZ5Y3h56RYefPvjsOOItDkVApFWmHn6MC78VD9+9OoqVm/dE3YckTalQiDSCmbGDy4bR6eMNH46e3XYcUTalAqBSCt1y8ng5unDmL1iG4s37go7jkibUSEQOQI3njqUHp0ytFcgHYoKgcgR6JyZxq1nDOfttWXM/Ug3mknHoEIgcoSumTyEvl2yuPe11bqcVDoEFQKRI5SVnsrtM0aycMMu3lidkD2nixwRFQKRo3BlwUCG9MzhF39fq70CafdUCESOQnpqCjefNoylJRW8X1QedhyRY6JCIHKULj95IN1z0nnw7aKwo4gcExUCkaOUnZHKtZOH8I9V2/i4tDLsOCJHTYVA5BhcOyWf9JQUHn5XewXSfqkQiByDvNxMLj2pP88sKGHn3pqw44gcFRUCkWP05dOGsb82wuPz1ocdReSoqBCIHKNRfXI5fVQej723nv219WHHETliKgQibeDm04ZRVlnNq8u2hB1F5IipEIi0ganDezKwezZ/XbQ57CgiR0yFQKQNpKQYl4zvzztrS9m+Z3/YcUSOiAqBSBu5dPwAIg4vL9HhIWlfAi0EZnaHmS0zs+Vmdmcz7SaaWb2ZXRFkHpEgjeyTy/H9u/D84k1hRxE5IoEVAjMbB9wMTAJOBC4ys5FNtEsF7gFeCyqLSLxcOn4AS0sq+Eh3Gks7EuQewXHAPHevcvc64E3gsibafQ14FlB/vtLuXTy+P2bwwiLtFUj7EWQhWAZMN7OeZpYDXAAMatjAzAYQLQ73B5hDJG76dMli6vCePL94s7qnlnYjsELg7iuJHvJ5HZgFLAHqGjX7BfAtd2/2Lhwzu8XMCs2ssLS0NJC8Im3l0vED2FBexcINGuBe2odATxa7+0PufrK7TwfKgbWNmhQAfzGzYuAK4D4zu7SJ9Tzg7gXuXpCXlxdkZJFjdv64vmSmpfCCThpLOxH0VUO9Y8+Dgc8CTzac7+5D3T3f3fOBZ4Bb3f35IDOJBC03K52zx/bh5aVbqKuPhB1HpEVB30fwrJmtAF4CbnP3nWY208xmBrxdkVB95oR+lO+tYb5GL5N2IC3Ilbv7aU1Ma/LEsLtfH2QWkXg6fVRvstNTeXXZFqaN6BV2HJFm6c5ikQBkZ6Ry5pg8Zi3bRn1EVw9JYlMhEAnI+eP6UVZZzYL1O8OOItIsFQKRgJw1pjcZaSn87UP1PSSJTYVAJCCdM9OYPjKP15ZvJaLDQ5LAVAhEAvTpcX3ZUrGfJSW6uUwSlwqBSIDOPq4P6anGq8u2hh1F5LBUCEQC1DUnnanDe/Hqsi3qe0gSlgqBSMAu+FRfNpbvY/nm3WFHEWmSCoFIwM4Z25fUFNPVQ5KwVAhEAtajUwZTh/fkxSXqmloSkwqBSBxcOn4AJTv36eYySUgqBCJxcN64vmSlp2g8Y0lIKgQicdA5M42zj+vDK0u3UKuuqSXBqBCIxMml4wews6qWt9ZolD1JLCoEInEyfVQe3XPSeX7x5rCjiBxChUAkTjLSUrjwhH68vmIrldWNh+8WCY8KgUgcXTp+APtrI7ymLickgagQiMTRhCHdGdg9W1cPSUJRIRCJIzPjkvH9eXddGdv37A87jgigQiASd5edNJCIw/OLtFcgiUGFQCTORvTuzMmDu/F0YYm6nJCEoEIgEoIrCwaxdnslizdqwBoJX6CFwMzuMLNlZrbczO5sYv4XzWxp7DHXzE4MMo9IorjohH5kpafw9IKSsKOIBFcIzGwccDMwCTgRuMjMRjZqVgSc7u4nAHcDDwSVRySR5Galc8G4fry0eDP7aurDjiNJLsg9guOAee5e5e51wJvAZQ0buPtcdz/QHeM8YGCAeUQSypUFg9hTXcdry3VPgYQryEKwDJhuZj3NLAe4ABjUTPubgFebmmFmt5hZoZkVlpaqnxbpGE4Z2oNBPbJ5qnBj2FEkyQVWCNx9JXAP8DowC1gCNHlfvZmdSbQQfOsw63rA3QvcvSAvLy+gxCLxlZJiXHHyIOZ+tION5VVhx5EkFujJYnd/yN1PdvfpQDmwtnEbMzsB+ANwibvvCDKPSKK5fMIAzODZhTppLOEJ+qqh3rHnwcBngScbzR8MPAdc6+5rgswikogGds9h2vBePF1YQiSiewokHEHfR/Csma0AXgJuc/edZjbTzGbG5n8X6AncZ2aLzaww4DwiCeeqSYPYtGsfb68rCzuKJKm0IFfu7qc1Me3+Bq+/DHw5yAwiie6csX3o0SmDJ+dv4PRROgcm8ac7i0VClpmWyhUTBvL3ldso3VMddhxJQioEIgng8xMHURdxntGdxhICFQKRBDA8rzOThvbgLx9s0EljiTsVApEE8YVJg1m/o4p5H+sqaokvFQKRBHH+uL50zU7nz+9vCDuKJBkVApEEkZWeymdPHsDs5dso31sTdhxJIioEIgnk6kmDqamP8MwC9T8k8aNCIJJARvXJZVJ+Dx6fp5PGEj8qBCIJ5topQ9hQXsWba9TTrsSHCoFIgjnv+L70zs3ksfeKw44iSUKFQCTBZKSlcPWkwby5ppTisr1hx5EkoEIgkoC+cMpgUs14fN76sKNIElAhEElAfbpkcd64vjxVuFFjGkvgVAhEEtR1U/LZvb+OFxZvCjuKdHAqBCIJamJ+d8b0zeWx99bjrktJJTgqBCIJysz40pR8Vm7ZzQfFO8OOIx2YCoFIArv0pP50zU7nsbnFYUeRDkyFQCSB5WSk8fmJg5i1fCubd+0LO46EqK4+Eti6VQhEEty1k4fg7rqUNMmN//7r3DNrVSDrViEQSXCDeuRw9nF9ePL9Deyv1aWkyWhfTT2V1XXkZgUzzLwKgUg7cP20fHZW1fLiks1hR5EQlFVGx7Lu1TkzkPW3qhCY2XAzy4y9PsPMbjezboEkEpF/MWVYT0b3yeXRd4t1KWkSOlgIMgJZf2v3CJ4F6s1sBPAQMBT4c0sLmdkdZrbMzJab2Z1NzDcz+5WZrTOzpWZ28hGlF0kSZsb10/JZsWU3het1KWmyKauMDlQU6h4BEHH3OuAy4Bfu/nWgX3MLmNk44GZgEnAicJGZjWzU7NPAyNjjFuB3R5BdJKlcOn4AXbPTefTd4rCjSJztSIRDQ0CtmV0NXAe8HJuW3sIyxwHz3L0qVkTeJFpIGroE+KNHzQO6mVmzBUYkWWVnpH5yKenWiv1hx5E4OnBoqEencA8N3QBMAX7g7kVmNhR4vIVllgHTzaynmeUAFwCDGrUZADQck68kNu0QZnaLmRWaWWFpqQbrkOR1zSlDiLhrgPskU1ZZQ25WGlnpqYGsv1WFwN1XuPvt7v6kmXUHct39Ry0ssxK4B3gdmAUsAeoaNbOmFm1iXQ+4e4G7F+Tl5bUmskiHNLhnDmeO7s2f52+gpi64G4wksZRVVgd2WAhaf9XQHDPrYmY9iP5Cf8TMftbScu7+kLuf7O7TgXJgbaMmJRy6lzAQ0PVxIs340pQhlFVW8+qyLYdMd3eqahr/rSUdQbQQBHNYCFp/aKiru+8GPgs84u4TgLNbWsjMeseeB8eWfbJRkxeBL8WuHpoMVLj7FkTksKaPzCO/Zw5/fO/gncbVdfXc9Fgh0388h4p9tSGmkyDsqKwJf48ASIudxP0cB08Wt8azZrYCeAm4zd13mtlMM5sZm/834GNgHfAgcOsRrFskKaWkGNdMHsKC9TtZtqmCuvoItz+5iH+u2k5ZZTWPvFsUdkRpY2WV1fQMcI+gtfcrfx94DXjX3T8ws2H862Gef+HupzUx7f4Grx24rZUZRCTmygmD+OnsNTw6t5ja+givLd/Gdy8ay/yiHTz0ThE3TBtK1+yWLuyT9qCuPsLOqtrw9wjc/Wl3P8HdvxJ7/7G7Xx5YKhFpVtecdC49aQDPLCjhhcWb+ffzR3PjqUO5Y8Yo9uyv4+F3tFfQUZTvjd5M1jPsQmBmA83sr2a23cy2mdmzZjYwsFQi0qLrp+aTnZ7K7WeN4NYzRgAwtn8Xzj++Lw+/W6RzBR1EaewegrwEOFn8CNETu/2JXuf/UmyaiIRkdN9cFn33HL5x7uhDpt9x9kj27K/jIe0VdAg7Au5eAlpfCPLc/RF3r4s9HgV0Qb9IyJq6wei4fl349Li+PPJOERVV2ito7w7cVRz6oSGgzMyuMbPU2OMaYEdgqUTkmNw+YyR7quv4wzsfhx1FjtHBPYLwDw3dSPTS0a3AFuAKot1OiEgCOq5fFy48oR8PvVP0yV+U0j6VVVaTkZZC58xgBqWB1l81tMHdL3b3PHfv7e6XEr1BTEQS1DfOGcX+2np+N+ejsKPIMSitrCavcyZmTfXI0zaOZYSyb7RZChFpc8PzOnPFhIH8ad56DXzfjkXvKg7usBAcWyEIrjyJSJu44+xR4PDrf7Z4/6ckqOhdxcGdKIZjKwQaL08kwQ3ols0XJw/mqcISisr2hh1HjkLoewRmtsfMdjfx2EP0ngIRSXC3njGCzLQUfvb6mrCjyBFyd3bsDXmPwN1z3b1LE49cdw/uFLaItJm83ExunDaUl5ZsZuWW3WHHkSNQsa+W2noP9GYyOLZDQyLSTtx82jByM9P4zRvrwo4iR6AsDvcQgAqBSFLompPOl6YO4W8fbmHd9j1hx5FWKgt40PoDVAhEksRNpw4jKy2V376h+wrai3j0MwQqBCJJo0enDK6ZPJgXFm+iWFcQtQsH+xnSoSERaSM3Tx9GemoK983RuYL2oKyymhSD7jkqBCLSRnrnZnH1pME8t3ATG8urwo4jLSirrKFHp0xSU4K9f1eFQCTJ/J/Th5Fixu/e1LmCRFdWWR34FUOgQiCSdPp1zebzEwfx1AcbWbe9Muw40owdldWBnygGFQKRpHTH2SPJTk/lB6+sCDuKNKOssibwE8WgQiCSlHp1zuT2GSN5Y3Upc1ZvDzuOHEZZR9gjMLOvm9lyM1tmZk+aWVaj+YPN7A0zW2RmS83sgiDziMhB103NJ79nDne/vILa+kjYcaSRqpo6qmrq23chMLMBwO1AgbuPA1KBqxo1+3/AU+5+UmzefUHlEZFDZaSl8J0Lx/JR6V6emLc+7DjSyIGbyTrCoaE0INvM0oAcYHOj+Q50ib3u2sR8EQnQ2cf15tQRvfj539eyc29N2HGkgQM3k+W15z0Cd98E3AtsIDrOcYW7z27U7D+Ba8ysBPgb8LWm1mVmt5hZoZkVlpaWBhVZJOmYGf9x0Vj27K/lfl1OmlDKOsIegZl1By4BhhIdu6CTmV3TqNnVwKPuPhC4APiTmf1LJnd/wN0L3L0gLy8vqMgiSWl031wuOqE/j89bz64q7RUkinh1OAfBHho6Gyhy91J3rwWeA6Y2anMT8BSAu78HZAG9AswkIk247cwR7K2p55F3i8OOIjE74tTPEARbCDYAk80sx8wMmAGsbKLNDAAzO45oIdCxH5E4G903l3PG9uHRucVUVteFHUeIHhrKzUojMy018G0FeY5gPvAMsBD4MLatB8zs+2Z2cazZvwE3m9kS4EngenfXWMgiIbjtzBFU7KvVFUQJYkN5FQO758RlW4EON+nu3wO+12jydxvMXwFMCzKDiLTO+EHdOHVELx58u4jrpuaTlR78X6JyeMVlexnTLzcu29KdxSLyidvOHEFZZTVPFW4MO0pSq6uPsKG8ivyeneKyPRUCEfnE5GE9mDCkO79/82Nq6nS3cVg27dpHXcTJ76VCICJxZmZ87awRbNq1T3sFISqKjSA3VIVARMJw+qg8CoZ059f/XMv+2vqw4ySlA4VAh4ZEJBRmxl3njWbb7mr+9J6uIApDcdleOmemxWVQGlAhEJEmTB7Wk9NG9uK+OevYs7827DhJp2hHFfm9cojeghU8FQIRadJd545mZ1UtD79THHaUpFNctjduh4VAhUBEDuPEQd04d2wf/vD2x+qDKI5q6iKU7KyK24liUCEQkWb827mjqayp4/43Pw47StLYuLOKiMfviiFQIRCRZozum8vFJ/bnsbnFlO6pDjtOUig+cMWQCoGIJIo7Zoykpj7C7+ZovIJ4+OQeAp0jEJFEMSyvM589aQCPz1/Plop9Ycfp8IrK9tI1O53uneJz6SioEIhIK9w+YyTuzm/fWBd2lA6veMfeuB4WAhUCEWmFQT1y+FzBIP73g41sLK8KO06HVlxWxdCe8el++gAVAhFpla+eNQIz49f/XBt2lA5rf209myv2aY9ARBJTv67ZXHPKEJ5duImPSyvDjtMhbSivwuN86SioEIjIEfjKGcPJTEvhp7PXhB2lQ4p3Z3MHqBCISKvl5Wby5dOG8cqHW1i8cVfYcTqcohDuIQAVAhE5QrdMH0bPThn88G8r0RDjbau4bC89O2XQNTs9rttVIRCRI9I5M43bZ4xkflE5c1aXhh2nQykqi/+lo6BCICJH4epJgxnSM4d7Zq2iPqK9grZSvCO+vY4eoEIgIkcsIy2Fu84dzaqte/jrok1hx+kQqmrq2La7mqG94nsPAQRcCMzs62a23MyWmdmTZpbVRJvPmdmKWLs/B5lHRNrOhZ/qxwkDu/Kz2as1pGUbCOtEMQRYCMxsAHA7UODu44BU4KpGbUYC3wamufvxwJ1B5RGRtpWSYnzr/DFsrtjPE/M3hB2n3TtwFda4/l3jvu2gDw2lAdlmlgbkAJsbzb8Z+K277wRw9+0B5xGRNjRtRC+mjejJb99YR2V1Xdhx2rXC4p306pzJkDh3LwEBFgJ33wTcC2wAtgAV7j67UbNRwCgze9fM5pnZ+U2ty8xuMbNCMyssLdVVCiKJ5JvnjaF8bw0PvV0UdpR27YPicibmd4/bOMUNBXloqDtwCTAU6A90MrNrGjVLA0YCZwBXA38ws26N1+XuD7h7gbsX5OXlBRVZRI7C+EHdOO/4Pjz49seU79WQlkdja8V+SnbuoyC/RyjbD/LQ0NlAkbuXunst8BwwtVGbEuAFd6919yJgNdHCICLtyF3njqaqpo7fzVE31UejcH05AAVDuoey/SALwQZgspnlWHRfZwawslGb54EzAcysF9FDRRocVaSdGdknl8tOGshj72nwmqNRWLyT7PRUxvbvEsr2gzxHMB94BlgIfBjb1gNm9n0zuzjW7DVgh5mtAN4AvunuO4LKJCLBufPskeDwq3+om+ojVbi+nJMGdyM9NZxbuwLdqrt/z93HuPs4d7/W3avd/bvu/mJsvrv7N9x9rLt/yt3/EmQeEQnOoB45XD1pEE8VlnwyALu0rLK6jhWbd4d2fgB0Z7GItKHbzhpBeqrxS+0VtNqiDTuJeHjnB0CFQETaUO/cLK6bms/zizexZtuesOO0Cx8U7yTF4KTB/3LBZNyoEIhIm5o5fTidM9L4mQavaZUF68s5rl8XcrPi2/V0QyoEItKmunfK4KbThjJr+VY+LKkIO05Cq62PsGjDLiaGeH4AVAhEJAA3nTqUbjnp3Dt7ddhREtrKLbupqqlnQojnB0CFQEQCkJuVzszTh/PmmlLe+0hXhB9OYfFOAAryVQhEpAO6fmo+/btm8d+vrCCiwWuaVLi+nIHds+nXNTvUHCoEIhKIrPRUvvXpMSzfvJtnF5aEHSfhVNfV8+66HUwaGu75AVAhEJEAXXxif8YP6sZPXltNVY26qW7ojVWlVOyr5eIT+4cdRYVARIJjZvzHRcexfU8197+pbsQaem5hCb06Z3LqiF5hR1EhEJFgTRjSg4tO6McDb32kDulidu6t4Y3V27l0fH/SQupfqKHwE4hIh/et88cQcbjn1VVhR0kIL3+4hdp657KTB4QdBVAhEJE4GNQjh5mnD+f5xZt5bfnWsOOE7rmFJYzuk8vYfuF0O92YCoGIxMVXzxzBuAFd+PZzH1K6pzrsOKEpKtvLog27+OzJA0IZlrIpKgQiEhcZaSn8/HPjqayu49vPLcU9Oe8t+OuiTZjBJeMT47AQqBCISByN7JPLt84fw99Xbuepwo1hx4k7d+evi0o4dUQv+nbNCjvOJ1QIRCSubpiaz9ThPfmvl1Yk3QA2het3srF8H5edlDh7A6BCICJxlpJi3HvliWSkpXDNQ/PZtCs5LimtqYtw98sr6JqdznnH9w07ziFUCEQk7vp3y+ZPN55Cxb5arn5gXlLcX/CT11axtKSCey4/gU6ZaWHHOYQKgYiE4lMDu/Knm05h594avvDgfLbt3h92pMC8sXo7D75dxLWTh3D+uMTaGwAVAhEJ0fhB3Xj0xkls372fqx+cx/Y9Ha8YbN+9n7ueWsKYvrl858Ljwo7TpEALgZl93cyWm9kyM3vSzJo8TW5mV5iZm1lBkHlEJPFMGNKdR26YxNaK/Xzhwfkd6h6Dqpo67vzfxeytqeM3XziJrPTUsCM1KbBCYGYDgNuBAncfB6QCVzXRLjfWbn5QWUQksU0a2oOHr5/Ipp37+MKD8yirbP/FYPHGXVz0q3eY+9EO7r5kHCN654Yd6bCCPjSUBmSbWRqQA2xuos3dwI+BjrdPKCKtNnlYTx6+fiIbd1bxxQfns6OdFoPa+gg/f30Nl/9uLvtr6/nzl0/hyoJBYcdqVmCFwN03AfcCG4AtQIW7z27YxsxOAga5+8vNrcvMbjGzQjMrLC0tDSqyiIRsyvCePHzdRIp37OW8X7zFQ+8Usb+2PuxYrba/tp4bHvmAX/5jLRef2J9X75zO1AToZrolFtRt3mbWHXgW+DywC3gaeMbdH4/NTwH+CVzv7sVmNge4y90Lm1tvQUGBFxY220RE2rkPSyr40ayVvLtuB327ZPGVM4ZzfP8u5Galk5uVRs/OGWSmJdbx9rr6CLc+sZDZK7Zxz+Wf4vMTB4cd6RBmtsDdmzwPG+TFrGcDRe5eGgvxHDAVeDw2PxcYB8yJdbzUF3jRzC5uqRiISMf2qYFdeeLLk5n7URk/m72G7724/JD5uVlp3HLaMG44dSidE+Ca/EjE+b/PfcjsFdv43mfGJlwRaEmQn+AGYLKZ5QD7gBnAJ7/g3b0C+GSfqbV7BCKSPKYO78WUmT1ZuWUPpZXV7Nlfy579dfxz1XZ++voaHn63iJmnD+faKUPIyQinILg7//3KSp5ZUMKdZ4/khmlDQ8lxLAL75Nx9vpk9AywE6oBFwANm9n2g0N1fDGrbItJxmBlj+x/ab//VkwazeOMufvb6Gn746ip+/9bHXD81ny9NGUK3nIy4ZVu2qYJ7Zq3i7bVl3DAtnztmjIzbtttSYOcIgqJzBCLSUGFxOffN+Yh/rtpOTkYqV08azDWThzC0V6fAtrmxvIp7Z6/mhcWb6Z6TztfOGsn1U/NJSUmM8QWa0ma5q48AAAmUSURBVNw5AhUCEekQVm7Zze/f/IiXlm6hPuKcMrQHV00axDlj+7bZeYRIxHl0bjH3zFqFGdw4bSgzzxhOl6z0Nll/kFQIRCRpbNu9n2cWlPBU4UbW76gCoEtWGn27ZtGvazYXn9ifS45i0PiN5VXc9fQS5heVM2NMb/77snH065odxI8QCBUCEUk6kYgzr2gHSzZWsLViH1sq9rNueyUfl+1laK9O3D5jBBefOIDUFg7nlO6p5sn3N3D/mx+RYsZ3PzOWKycMTJhhJltLhUBEhOgVPrNXbOMXf1/Lyi27Gdwjh3PG9uH0UXlMGtqDrPRU6iPOjr3VrNtWyZMfbGTWsi3U1jtnjenN3ZeOY0C39rMX0JAKgYhIA5GIM3vFVp6Yv4H5ReXU1EXITk8lNyuNHXtrqI9Efy/mZqVx5YRBXDN5MMPyOoec+tiEdUOZiEhCSkkxzh/Xj/PH9aOqpo75H5fz5ppSqmrq6J2bRZ8umfTtms20ET1Duz8hnjr+Tygi0oycjDTOHNObM8f0DjtKaDQwjYhIklMhEBFJcioEIiJJToVARCTJqRCIiCQ5FQIRkSSnQiAikuRUCEREkly762LCzEqJjoFc0WhW1xamtfT6wHMvoOwoojW1/dbMbzy9ufeNszacdjS545m54eswPmt9P/T9aG5+e/x+HElmgJHu3rXJtbt7u3sADxzptJZeN3gubKtMrZnfeHpz7xtnPdbc8cwc9met74e+Hx3t+3EkmVvaRns9NPTSUUxr6XVTyx9rptbMbzy9ufdNZT2W3PHM3PB1GJ+1vh9HTt+P1r9O9MzNbqPdHRoKmpkV+mF66Etk7TG3MsdPe8ytzPHTXvcIgvRA2AGOUnvMrczx0x5zK3OcaI9ARCTJaY9ARCTJqRCIiCS5Dl0IzOxhM9tuZsuOYtkJZvahma0zs19Zg5GqzexrZrbazJab2Y/bNnUwuc3sP81sk5ktjj0uSPTMDebfZWZuZr3aLnFgn/PdZrY09hnPNrP+7SDzT8xsVSz3X82sW1tmDjD3lbH/gxEza7MTtMeS9TDru87M1sYe1zWY3uz3Pq6O5prX9vIApgMnA8uOYtn3gSmAAa8Cn45NPxP4O5AZe9+7neT+T+Cu9vRZx+YNAl4D1gO9Ej0z0KVBm9uB+9tB5nOBtNjre4B72sP3AzgOGA3MAQrCzhrLkd9oWg/g49hz99jr7s39XGE8OvQegbu/BZQ3nGZmw81slpktMLO3zWxM4+XMrB/R/9DvefRf7I/ApbHZXwF+5O7VsW1sbye5AxVg5p8D/w60+VUNQWR2990NmnZq69wBZZ7t7nWxpvOAgW2ZOcDcK919daJkPYzzgNfdvdzddwKvA+eH+X+1KR26EBzGA8DX3H0CcBdwXxNtBgAlDd6XxKYBjAJOM7P5ZvammU0MNO1Bx5ob4Kux3f+Hzax7cFE/cUyZzexiYJO7Lwk6aAPH/Dmb2Q/MbCPwReC7AWY9oC2+GwfcSPSv03hoy9xBa03WpgwANjZ4fyB/ovxcQJINXm9mnYGpwNMNDsdlNtW0iWkH/rJLI7qLNxmYCDxlZsNiVT0QbZT7d8Ddsfd3Az8l+p8+EMea2cxygO8QPWwRF230OePu3wG+Y2bfBr4KfK+Nox4M0kaZY+v6DlAHPNGWGZvSlrmD1lxWM7sBuCM2bQTwNzOrAYrc/TIOnz/0n6uhpCoERPeAdrn7+IYTzSwVWBB7+yLRX5oNd48HAptjr0uA52K/+N83swjRjqZKEzm3u29rsNyDwMsB5oVjzzwcGAosif3nGwgsNLNJ7r41QTM39mfgFQIsBLRR5thJzIuAGUH+UdNAW3/WQWoyK4C7PwI8AmBmc4Dr3b24QZMS4IwG7wcSPZdQQvg/10FhnZyI1wPIp8FJH2AucGXstQEnHma5D4j+1X/gRM4Fsekzge/HXo8iuttn7SB3vwZtvg78JdEzN2pTTBufLA7ocx7ZoM3XgGfaQebzgRVAXltnjcf3gzY+WXy0WTn8yeIiokcRusde92jt9z5ej1A2GrcfDp4EtgC1RCvwTUT/ypwFLIl9+b97mGULgGXAR8BvOHgXdgbweGzeQuCsdpL7T8CHwFKif2n1S/TMjdoU0/ZXDQXxOT8bm76UaCdfA9pB5nVE/6BZHHu06ZVOAea+LLauamAb8FqYWWmiEMSm3xj7jNcBNxzJ9z5eD3UxISKS5JLxqiEREWlAhUBEJMmpEIiIJDkVAhGRJKdCICKS5FQIpEMws8o4b+8PZja2jdZVb9HeSpeZ2Ust9f5pZt3M7Na22LYIaIQy6SDMrNLdO7fh+tL8YEdsgWqY3cweA9a4+w+aaZ8PvOzu4+KRTzo+7RFIh2VmeWb2rJl9EHtMi02fZGZzzWxR7Hl0bPr1Zva0mb0EzDazM8xsjpk9Y9H++p840Gd8bHpB7HVlrKO5JWY2z8z6xKYPj73/wMy+38q9lvc42OleZzP7h5kttGi/9ZfE2vwIGB7bi/hJrO03Y9tZamb/1YYfoyQBFQLpyH4J/NzdJwKXA3+ITV8FTHf3k4j2Dvo/DZaZAlzn7mfF3p8E3AmMBYYB05rYTidgnrufCLwF3Nxg+7+Mbb/FfmRi/ezMIHrnN8B+4DJ3P5noOBg/jRWi/wt85O7j3f2bZnYuMBKYBIwHJpjZ9Ja2J3JAsnU6J8nlbGBsgx4ju5hZLtAVeMzMRhLt8TG9wTKvu3vDvujfd/cSADNbTLQPmncabaeGg534LQDOib2ewsE+5v8M3HuYnNkN1r2AaJ/1EO2D5n9iv9QjRPcU+jSx/Lmxx6LY+85EC8Nbh9meyCFUCKQjSwGmuPu+hhPN7NfAG+5+Wex4+5wGs/c2Wkd1g9f1NP1/ptYPnmw7XJvm7HP38WbWlWhBuQ34FdHxDPKACe5ea2bFQFYTyxvwQ3f//RFuVwTQoSHp2GYTHQ8AADM70I1wV2BT7PX1AW5/HtFDUgBXtdTY3SuIDm95l5mlE825PVYEzgSGxJruAXIbLPoacGOs33zMbICZ9W6jn0GSgAqBdBQ5ZlbS4PENor9UC2InUFcQ7UIc4MfAD83sXSA1wEx3At8ws/eBfkBFSwu4+yKiPVxeRXSAmAIzKyS6d7Aq1mYH8G7sctOfuPtsooee3jOzD4FnOLRQiDRLl4+KBCQ2yto+d3czuwq42t0vaWk5kXjTOQKR4EwAfhO70mcXAQ4NKnIstEcgIpLkdI5ARCTJqRCIiCQ5FQIRkSSnQiAikuRUCEREktz/B58wzJVc4C6bAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lr = 3e-3\n",
    "lr *= bs/48  # Scale learning rate by batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='9' class='' max='10', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      90.00% [9/10 4:17:30<28:36]\n",
       "    </div>\n",
       "    \n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.464687</td>\n",
       "      <td>3.499862</td>\n",
       "      <td>0.588392</td>\n",
       "      <td>0.411609</td>\n",
       "      <td>33.110847</td>\n",
       "      <td>28:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.290530</td>\n",
       "      <td>3.359655</td>\n",
       "      <td>0.577429</td>\n",
       "      <td>0.422572</td>\n",
       "      <td>28.779305</td>\n",
       "      <td>28:36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.279709</td>\n",
       "      <td>3.345511</td>\n",
       "      <td>0.576788</td>\n",
       "      <td>0.423212</td>\n",
       "      <td>28.375286</td>\n",
       "      <td>28:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.218174</td>\n",
       "      <td>3.291578</td>\n",
       "      <td>0.570586</td>\n",
       "      <td>0.429414</td>\n",
       "      <td>26.885269</td>\n",
       "      <td>28:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.188569</td>\n",
       "      <td>3.235373</td>\n",
       "      <td>0.564242</td>\n",
       "      <td>0.435757</td>\n",
       "      <td>25.415869</td>\n",
       "      <td>28:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.065708</td>\n",
       "      <td>3.165833</td>\n",
       "      <td>0.555168</td>\n",
       "      <td>0.444832</td>\n",
       "      <td>23.708519</td>\n",
       "      <td>28:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.036142</td>\n",
       "      <td>3.089983</td>\n",
       "      <td>0.545404</td>\n",
       "      <td>0.454596</td>\n",
       "      <td>21.976547</td>\n",
       "      <td>28:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.999072</td>\n",
       "      <td>3.019598</td>\n",
       "      <td>0.535763</td>\n",
       "      <td>0.464237</td>\n",
       "      <td>20.483038</td>\n",
       "      <td>28:26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.888412</td>\n",
       "      <td>2.969108</td>\n",
       "      <td>0.528063</td>\n",
       "      <td>0.471937</td>\n",
       "      <td>19.474483</td>\n",
       "      <td>28:36</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>\n",
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "            .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "                background: #F44336;\n",
       "            }\n",
       "        </style>\n",
       "      <progress value='8728' class='' max='43928', style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      19.87% [8728/43928 05:24<21:49 2.9050]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAb50lEQVR4nO3deXQc5Z3u8e9P3S21NtuyLdvCJsjsxuBrgzDONSEMWQbMOhefxDkhJ+PJPT4HkmHJzCRmkhlIDplAboaby8yEJRkgIQQChoQkEHaMw44MjhA2xitYXpBtLNmytbTU7/2jS+2W3JJacpW6LD2fc/qoupa3fl0tPV16a2lzziEiIuFVkO8CRESkfwpqEZGQU1CLiIScglpEJOQU1CIiIRcNotGxFePdCccdG0TTIiIj0sqVK3c55yqzTQskqCcddTS1tbVBNC0iMiKZ2Qd9TVPXh4hIyCmoRURCLpCg1rWOIiL+CaSPWkktIoORSCRoaGigra0t36UELh6PM23aNGKxWM7LBBPUIiKD0NDQQHl5OdXV1ZhZvssJjHOO3bt309DQwPTp03NeTn3UIpJ3bW1tTJgwYUSHNICZMWHChEH/56CgFpFQGOkh3W0orzOgoFYntYiIX7RHLSKjXlNTEz/96U8HvdyCBQtoamoKoKKeFNQiMur1FdRdXV39LvfEE08wbty4oMpKC+SsD3V8iMiRZOnSpWzYsIHZs2cTi8UoKyujqqqKVatWsXr1ai677DK2bNlCW1sb11xzDUuWLAGgurqa2tpaWlpauOCCCzj77LN55ZVXmDp1Ko899hjFxcW+1KfT80QkVL73h3dZvW2vr22ectQYbrh4Zp/Tb775Zurr61m1ahXLly/nwgsvpL6+Pn0K3d1338348eNpbW3lzDPP5PLLL2fChAk92li3bh0PPPAAP/vZz/jCF77AI488whVXXOFL/QpqEZFe5s6d2+M859tuu43f/va3AGzZsoV169YdEtTTp09n9uzZAJxxxhls3rzZt3oU1CISKv3t+Q6X0tLS9PDy5ct59tlnefXVVykpKeHcc8/Neh50UVFRejgSidDa2upbPTqYKCKjXnl5Ofv27cs6rbm5mYqKCkpKSnjvvfd47bXXhrk63etDRIQJEyYwf/58Tj31VIqLi5k8eXJ62vnnn88dd9zBrFmzOOmkk5g3b96w12fO+Z+qx82Y5TasqfO9XREZmdasWcOMGTPyXcawyfZ6zWylc64m2/zq+hARCTndj1pEJOS0Ry0iEnIKahGRkFNQi4iEnIJaRCTkFNQiIkNQVlYGwLZt21i4cGHWec4991xqa2sPe10KahGRw3DUUUexbNmyQNeh25yKiADf/va3OeaYY7jqqqsAuPHGGzEzVqxYwZ49e0gkEtx0001ceumlPZbbvHkzF110EfX19bS2trJ48WJWr17NjBkzfLvfh27KJCLh8qelsOMdf9ucchpccHO/syxatIhrr702HdQPPfQQTz75JNdddx1jxoxh165dzJs3j0suuaTP7z28/fbbKSkpoa6ujrq6Ok4//XRfyldQi4gAc+bMobGxkW3btrFz504qKiqoqqriuuuuY8WKFRQUFLB161Y++ugjpkyZkrWNFStWcPXVVwMwa9YsZs2a5UttCmoRCZcB9nyDtHDhQpYtW8aOHTtYtGgR999/Pzt37mTlypXEYjGqq6uz3uI0UxDfph7MwUR1UovIEWjRokU8+OCDLFu2jIULF9Lc3MykSZOIxWK88MILfPDBB/0uf84553D//fcDUF9fT12dPzenC2iPWkktIkeemTNnsm/fPqZOnUpVVRVf/vKXufjii6mpqWH27NmcfPLJ/S5/5ZVXsnjxYmbNmsXs2bOZO3euL3UFcpvT6pNPc5vf8/lggIiMWLrNqQ+3OTWz68zsXTOrN7MHzCzuQ60iIpKDAYPazKYCVwM1zrlTgQiwKOjCREQkJdeDiVGg2MyiQAmwLbiSRGQ0CqIbNoyG8joHDGrn3Fbgx8CHwHag2Tn3dO/5zGyJmdWaWe3+lv2DLkRERq94PM7u3btHfFg759i9ezfx+OB6jwc8mGhmFcAjwBeBJuBhYJlz7ld9LaODiSIyGIlEgoaGhgHPUR4J4vE406ZNIxaL9Rjf38HEXE7P+yywyTm302vsUeB/An0GtYjIYMRiMaZPn57vMkIrlz7qD4F5ZlZiqUtuPgOsCbYsERHplksf9evAMuAt4B1vmbv6XcaX0kREBHK8MtE5dwNwQ8C1iIhIFrrXh4hIyOkbXkREQk5BLSIScgpqEZGQU1CLiIRcIEGtY4kiIv7RHrWISMgpqEVEQk5BLSIScgpqEZGQU1CLiIScglpEJOR0rw8RkZAL6DxqJbWIiF/U9SEiEnIKahGRkFNQi4iEnIJaRCTkFNQiIiGnoBYRCTkFtYhIyOl+1CIiIac9ahGRkFNQi4iEnO71ISIScuqjFhEJuUCCOukU1SIifglmj1o5LSLim4CCWkktIuIX9VGLiIScuj5EREIukKBuTXQF0ayIyKgUSFAnupJBNCsiMioFEtTFsUgQzYqIjErqoxYRCTld8CIiEnI5BbWZjTOzZWb2npmtMbNP9je/glpExD/RHOf7f8CTzrmFZlYIlPQ3s3JaRMQ/Awa1mY0BzgH+FsA51wF09LeM9qhFRPyTS9fHscBO4B4ze9vMfm5mpb1nMrMlZlZrZrUO6EoqrEVE/JBLUEeB04HbnXNzgP3A0t4zOefucs7VOOdqAA50dPpaqIjIaJVLUDcADc65173ny0gFd7/aErroRUTEDwMGtXNuB7DFzE7yRn0GWD3Qcm26jFxExBe5nvXx98D93hkfG4HFAy2wX10fIiK+yCmonXOrgJrBNHygQ3vUIiJ+COxbyN/6YE9QTYuIjCqBBfWYeCyopkVERpXAgrpLF72IiPgisKC+5+VNQTUtIjKqBBbU73/UElTTIiKjSmBBLSIi/ggsqGMRC6ppEZFRJZCgjsciJLp0MFFExA+BBHX35eNJ3UFPROSwBRLUZUWpCx5Xb98bRPMiIqNKIEFd6gX17S9uCKJ5EZFRJZCgLo6lmn28bnsQzYuIjCqBBHW5Lh8XEfGNzqMWEQm5wIN6y8cHgl6FiMiIFlhQjytJdX986kcvBLUKEZFRIbCg/tM1nwqqaRGRUSWwoK4aW5we3teWCGo1IiIj3rAcTDztxqeHYzUiIiNSoEH9g785NcjmRURGhUCD+stnHZMefuCND4NclYjIiDVs51Ff/+g7w7UqEZERJfCgfu4fPp0eXt+ob30RERmswIP6uMqy9PBnb30x6NWJiIw4w9L18YdvnJ0efrh2y3CsUkRkxBie0/OmjU0P/9OyOpzTFwqIiORq2A4m/uWGz6eHT/6XJ4drtSIiR7xhC+qxxTGmjIkD0N6ZZPU2ffuLiEguhvU2p69ef156eMFtfx7OVYuIHLGGNajNjNlHj0s//4/n1g3n6kVEjkjD/sUBv/v6/PTwvz/zvm7YJCIygLx8w0tmF4hu2CQi0r+8BHXmLVAB7nvtg3yUISJyRMjbdyZu+uGC9PC//K6eW59em69SRERCLW9BbWbcu/jM9PPbnl+fr1JEREItr99Cfu5Jk3o8r176eJ4qEREJr5yD2swiZva2mf3RzwLWfP/8Hs8V1iIiPQ1mj/oaYI3fBRQXRqj/3l/3GPf8ex/5vRoRkSNWTkFtZtOAC4GfB1FEWVGUqz9zQvr5391bS8OeA0GsSkTkiJPrHvVPgG8Byb5mMLMlZlZrZrU7d+4cdCHf/NyJ3PWVM9LPz77lBX7+542DbkdEZKQZMKjN7CKg0Tm3sr/5nHN3OedqnHM1lZWVQyrm8zOn9Hh+0+Nr2NHcNqS2RERGilz2qOcDl5jZZuBB4Dwz+1VQBW2++cIez+f98DkOdHQGtToRkdAbMKidc9c756Y556qBRcDzzrkrgiyqd1if8q9PUdfQFOQqRURCK6/nUfend1hf8p8v8+62Zn07jIiMOoMKaufccufcRUEV01vvsL7wtpeYfv0TJJMKaxEZPUK7R92td1gDHPvPT7C+sSUP1YiIDL/QBzVkD+vP3voi3//D6jxUIyIyvI6IoIZUWH/qhIk9xt398iaqlz7O1Q+8naeqRESCd8QENcB9XzuLZ795ziHjf/+XbVQvfZzlaxtp7+zKQ2UiIsE5ooIa4PhJ5Wz4twVZp/3tPW9y0nef5JGX/gId+4e5MhGRYETzXcBQRAqMzTdfyP/+RS3Prjn0Bk5lT30Tnq2lK1pKwZjJWNlkSD8mpX6WTzk4XDIRIkfkphCRUeCITqeff7WGrqRjyS9ree69xvT4B7rO463kCUzqbKKyvYnKnc1U2gdUx/cR7diXpSWD0speQd4r2LsfReVgNnwvUkRGPQviApKamhpXW1vre7v9ae/s4qTvPjngfEV0UGnNVNLEJGtibmWClt3bmBrdy+UnRrGWRmhphJaPIJnlG9KjxVnCvDvQM/bSyyZBJBbAKxWRkcjMVjrnarJOGylBDZBMOp6o307j3na+/8fDPXXPMY6WVKhbUzrYK62ZGWUHmDm2jcj+RkoTu4m2Z7+83cUrsGghWIH3iKT2xgsidDojEoliVgAF3niLHJy3IHLIMlgBSSugM2kURqNQECGJ4bxlkkSIRgtwRCgoiECB11akEKJxiBV7P0sgFk996MSKs4zvNU+0SP9FiARs1AR1b4+t2srtyzfw6RMruXNFcLdMLSTBBPYyyfZkBHszE62ZKF0UkKQAR8SSGI5UpKaGI9609DwkKcicZj2nmbdsapmD0wq85bItMz5uFHS1YZ1DvROhpcPcxUpIRoqIFKaCvCtSTKSwOBXqsZKDHwixYu+DIJ4xXHzwAyBW0mu8N00fCjJKjdqgzuSc46r732L20eOoa2jm8Xe257ukYWckKSJBnI7UwzoozhhOj6eD4szn1k7cW66Ydm/eBPH0sNeOdVAR68IlWonTQdT6vH15nxyG9Qj63o9DA74rEqcrGqcwXpbx30CW5aJxKCxNPaJxfSBIqCio+9HRmWRrUyvLVm7hv17YkO9yRpQonRkfBD3DPvVB0J7+UCjGm27t6WWyz9dBER0Ue/OlPjiyHEsYiEWgsOxgcBeWDvC8r+GM57GSVHeTyBAoqH3SuLeNyvIiAJyD3fs7eGn9Ts4+vpLXN+2mfutedre08/DKBr6zYAY/eKLvr5j8+l8dx2sbP2ZaRTEbdrZQWVbEC2tT34zzw/91Gtc/+g4At1x+Gq9v+phH39rK/OMncOPFM3l5/S5u1OXzad3/KXSHfXE67NvT/zUUp4fbKaWdEmujlDZKaKPU2iih3fvpjbd2SmmjlFYilvvfSCJSTCJSQjJWwp5EIeVjxjFu7DgoLMV5gW5F3UFfnjqLqKgsFfZF5d7PjOc6ID1qKKhDIJl0dHQlicciw7re5tYEX7jjVW794v9gTDzG+x/to7k1wW/e3MI/L5jBDb9/l31tCX75tbOYUFrInS9upHpiCa0dXZx9wkTOvuUFTqkaw+rte9NtjolH2ds2Wr7MwVFEwgv09owg7w76jIDvDvwe0zOWy/ggKLLctl9XQSFtBaUkY6VEi8dQXDaWRLSUXR0xKirGEy8dmw711oJiikrGYkVl7EoUUlI+ltKyioPBX1iq7p4QU1CLL5avbWT+8ROJRfr+975xXxtj4rFDPpDueHEDG3e28FBtAyv+6a/4Q902vjT3E6zdsY8tHx8g6RxLvf8iAL5YczQVpYXc8eKh3VH/+PkT+fHT7/v3wvIgRieltFJmbZTRmh4upZXS9Lg2yqw1NWxtlNFzepl1z5PbQeIuZ+wnzn6K2e/iVB81mWi8rGdXTlF5ergzWkI0Xp7u3klESni/KcnMY446uIwO/vpGQS0jXluii5372pkyNk7EjIKCVHjsb0/tuZYURrj1mff5j+fXA3DylHLe23HoxU/fvXAGNz2e6rKad+x4Xtv48TC9gqEzkuk9+YPh3doj7Etpo9RaKcP7ELDWQ7qAyuzgfwC5ShDhgCtiP3FaidPi4hxwcVqIc4A4+10RR0+u5Lhpk0nGSqiqnEh7QSltFmdLi9HUFePTp3zi4BlC3aeDxopHXbePglqkF+cc6xpbOHFy+ZDb2NXSTkdnkilj4rR3JonHCnh3217WN7Zw5vTxJDqTFEYLKIwWsGb7Xr7y329w2tSxvLO12cdX4j8jSTEdPbpwSjO7bjK6eEqt9dDunoz+/e5un1z3+jMlLYKLxCFaRKKgCBeNs2M/HDWxglU72qiaWMHUCRV0RYpIRuMUxUuwWJyWrhhlZWVsPwAvbWphwZzp7E/GeKRuJ5efdQITx41ld3sBt724hW9dMpuS4jIice9YQR4pqEVC5p2GZjqTST7e38FnZkwGUh8ej7+znfnHTWRXSzvRSAHNrQlmHz0OgHUf7ePXb3zIPS9v5t7FZ7L43jdxDoqiBaz+/vlc+auVPL364L1vfvA3p/Ljp9ay58AQzorxmZEkTgeltHt78wf33uN0UESCooxTQotI9Dhl9ODzBEV0UGSJPucdyllAe4+/lDFX/DKAV547BbWI9PC7t7dy7W9WpZ9/7pTJbPn4QNbuoCNP6gBwEV6wp8/7T53aGc8S8h+6Sdz3b9fnter+gvqIvimTiAzNZXOmctmcqUBqT96GeEAw0ZXkrQ/2cNaxE2jYc4CmAwm2NbVy8pQxVI2Lpw88/+KVzXR0Jmlq7WBHczslhRG6nOPXr3/o22s6yGinkHYK2QvQe1/0CPzKVe1Ri0jetCW6iEUKiHgHf1vaO9nd0s7e1k72tSX4+wfe5pXrz6Mr6di0az+RAuOkyeWYGWu272VdYwvHV5bRdKCDo8YV07ivnfte+4DXNu7mxotnsr+9k8JoAXsOdDDnExUsX9vIL17ZzJ4DifQB5fKiKLcsnMWC06ryui3U9SEiEnL9BbWudxURCTkFtYhIyCmoRURCTkEtIhJyCmoRkZBTUIuIhJyCWkQk5BTUIiIhp6AWEQk5BbWISMgpqEVEQk5BLSIScgpqEZGQU1CLiITcgEFtZkeb2QtmtsbM3jWza4ajMBERScnlG146gX9wzr1lZuXASjN7xjm3OuDaRESEHPaonXPbnXNvecP7gDXA1KALExGRlEH1UZtZNTAHeD2IYkRE5FA5B7WZlQGPANc65/Zmmb7EzGrNrHbnzp1+1igiMqrlFNRmFiMV0vc75x7NNo9z7i7nXI1zrqaystLPGkVERrVczvow4L+BNc65W4MvSUREMuWyRz0f+Apwnpmt8h4LAq5LREQ8A56e55x7CbBhqEVERLLQlYkiIiGnoBYRCTkFtYhIyCmoRURCTkEtIhJyCmoRkZBTUIuIhJyCWkQk5BTUIiIhp6AWEQk5BbWISMgpqEVEQk5BLSIScgpqEZGQU1CLiIScglpEJOQU1CIiIaegFhEJOQW1iEjIKahFREJOQS0iEnIKahGRkFNQi4iEnIJaRCTkFNQiIiGnoBYRCTkFtYhIyCmoRURCTkEtIhJyCmoRkZBTUIuIhJyCWkQk5BTUIiIhp6AWEQk5BbWISMgpqEVEQk5BLSIScjkFtZmdb2ZrzWy9mS0NuigRETlowKA2swjwX8AFwCnAl8zslKALExGRlFz2qOcC651zG51zHcCDwKXBliUiIt2iOcwzFdiS8bwBOKv3TGa2BFjiPW03s/rDL893E4Fd+S4iC9U1OKprcFTX4OSrrmP6mpBLUFuWce6QEc7dBdwFYGa1zrmanMsbJqprcFTX4KiuwVFducul66MBODrj+TRgWzDliIhIb7kE9ZvACWY23cwKgUXA74MtS0REug3Y9eGc6zSzbwBPARHgbufcuwMsdpcfxQVAdQ2O6hoc1TU4qitH5twh3c0iIhIiujJRRCTkFNQiImHnnPPtAZwPrAXWA0v9bLvXejYD7wCrgFpv3HjgGWCd97PCG2/AbV5NdcDpGe181Zt/HfDVjPFneO2v95a1Puq4G2gE6jPGBV5HX+sYoK4bga3eNlsFLMiYdr23jrXAXw/0fgLTgde99f8GKPTGF3nP13vTq3vVdTTwArAGeBe4JgzbrJ+68rrNgDjwBvAXr67vHUZbvtQ7QF33Apsyttfs4f7d9+aJAG8DfwzD9vIl83wMzwiwATgWKPTexFP8LDZjXZuBib3G/ah7wwFLgVu84QXAn7xflnnA6xlv+EbvZ4U33B0QbwCf9Jb5E3BBH3WcA5xOz0AMvI6+1jFAXTcC/5jlNZzivVdF3i/bBu+97PP9BB4CFnnDdwBXesNXAXd4w4uA3/RaVxXeHylQDrzvrT+v26yfuvK6zbzXUOYNx0gFwbzBtuVnvQPUdS+wMMv2GrbffW/8N4FfczCo87q9fMk8H8Pzk8BTGc+vB673s9iMtjdzaFCvBaoy/vDWesN3Al/qPR/wJeDOjPF3euOqgPcyxveYL0st1fQMxMDr6GsdA9R1I9lDp8f7ROrsnk/29X56fzi7gGjv9717WW846s2X9b8Rb57HgM+FZZtlqSs02wwoAd4idVXwoNrys94B6rqX7EE9bO8jqes8ngPOA/44lG0f5PYa6sPPPupsl5pP9bH9TA542sxWepeuA0x2zm0H8H5OGqCu/sY3ZBmfq+Goo691DOQbZlZnZnebWcUQ65oANDnnOrPUlV7Gm97szX8IM6sG5pDaGwvNNutVF+R5m5lZxMxWkerKeobUHt1g2/Kz3qx1Oee6t9cPvO31f82saIjb63Dex58A3wKS3vOhbHvft9fh8jOoc7rU3CfznXOnk7qj39fN7Jx+5u2rrsGOP1z5ruN24DhgNrAd+PcA6sqpZjMrAx4BrnXO7e2n5mHdZlnqyvs2c851Oedmk9pTnAvMGEJbvm/H3nWZ2amk9i5PBs4k1Z3xbZ/r6peZXQQ0OudWZo7up61h216Hy8+gHrZLzZ1z27yfjcBvSf0Cf2RmVQDez8YB6upv/LQs43M1HHX0tY4+Oec+8v64ksDPSG2zodS1CxhnZtFe43u05U0fC3ycWYeZxUiF4f3OuUcHeD3Dts2y1RWWbebV0gQsJ9XHO9i2/Ky3r7rOd85tdyntwD0MfXsN9X2cD1xiZptJ3eXzPFJ72KHZXkPmVx8KqT6ejaQ637s72mf62U/jracUKM8YfoXUkdj/Q8+DDD/yhi+k54GMN7zx40kdoa7wHpuA8d60N715uw9kLOinnmp69gUHXkdf6xigrqqM4euAB73hmfQ8cLKR1EGTPt9P4GF6Hji5yhv+Oj0PzjzUqyYDfgn8pNf4vG6zfurK6zYDKoFx3nAx8GfgosG25We9A9RVlbE9fwLcnI/ffW/auRw8mJjX7eVL7vkcogtIHTHfAHzHz7Yz1nGst4G6Tw36jjd+AqmDCOu8n91vuJH64oMNpE73qclo6+9InWazHlicMb4GqPeW+U/6Pj3vAVL/EidIfdp+bTjq6GsdA9R1n7feOlL3askMoe9461hLxhkufb2f3nvwhlfvw0CRNz7uPV/vTT+2V11nk/qXsI6MU97yvc36qSuv2wyYReo0szrvNf3rYbTlS70D1PW8t73qgV9x8MyQYfvdz1j+XA4GdV63lx8PXUIuIhJyujJRRCTkFNQiIiGnoBYRCTkFtYhIyCmoRURCTkEtIhJyCmoRkZD7/0KwowsIP8rgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.4116089344024658.\n",
      "Better model found at epoch 1 with accuracy value: 0.42257198691368103.\n",
      "Better model found at epoch 2 with accuracy value: 0.42321211099624634.\n",
      "Better model found at epoch 3 with accuracy value: 0.4294137954711914.\n",
      "Better model found at epoch 4 with accuracy value: 0.4357571601867676.\n",
      "Better model found at epoch 5 with accuracy value: 0.4448317289352417.\n",
      "Better model found at epoch 6 with accuracy value: 0.4545958638191223.\n",
      "Better model found at epoch 7 with accuracy value: 0.4642367959022522.\n",
      "Better model found at epoch 8 with accuracy value: 0.47193723917007446.\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "learn.unfreeze()\n",
    "wd = 0.01\n",
    "learn.fit_one_cycle(10, lr, wd=wd, moms=(0.8,0.7), \n",
    "                    callbacks=[ShowGraph(learn),\n",
    "                               SaveModelCallback(learn.to_fp32(), monitor='accuracy', name='bestmodel_sp15_bwd')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "GCP instance has stopped. Need to reload bestmodel_sp15 and vocab to continue the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "lm_fns2_bwd[0] = 'bestmodel_sp15_bwd'\n",
    "data.vocab.save(mdl_path/(lm_fns2_bwd[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# mdl_path = path/'models'\n",
    "# mdl_path.mkdir(exist_ok=True)\n",
    "# learn.to_fp32().save(mdl_path/lm_fns2_bwd[0], with_opt=False)\n",
    "# learn.data.vocab.save(mdl_path/(lm_fns2_bwd[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "29ff5bf7-47d3-4bb6-8ef7-4dbee784acd0"
    }
   },
   "source": [
    "## Generate fake texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: the architecture used for our French LM is based on 4 QRNN with about 46 millions of parameters. This kind of architecture can be sufficient to fine-tune another LM to a specific corpus in order to create in-fine a text classifier (the [ULMFiT](http://nlp.fast.ai/category/classification.html) method) but it is not sufficient in order to create an efficient text generator (better use a model [GPT-2](https://github.com/openai/gpt-2) or [BERT](https://github.com/google-research/bert)). More, the SentencePiece tokenizer used in this notebook implements subword units (e.g., byte-pair-encoding (BPE) [Sennrich et al.]) and unigram language model [Kudo.]) that can generate caracters from its vocabulary instead of words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "nbpresent": {
     "id": "903b31b8-77bb-48a7-a6de-fd2584005619"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.98 s, sys: 1.07 s, total: 5.05 s\n",
      "Wall time: 5.05 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data = load_data(path, f'{lang}_databunch_corpus2_100_sp15k', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "nbpresent": {
     "id": "f2e132d0-a490-49cb-895a-ee9a66c76c2d"
    }
   },
   "outputs": [],
   "source": [
    "# LM without pretraining\n",
    "learn = language_model_learner(data, AWD_LSTM, config=config, pretrained=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "nbpresent": {
     "id": "7596a7ac-558a-4bd7-80af-247bf0a82732"
    }
   },
   "outputs": [],
   "source": [
    "# LM pretrained in English\n",
    "learn_en = language_model_learner(data, AWD_LSTM, pretrained=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "nbpresent": {
     "id": "cdaac260-7cfc-4255-b03f-e1a88eccdc71"
    }
   },
   "outputs": [],
   "source": [
    "# LM pretrained in french\n",
    "learn_fr = language_model_learner(data, AWD_LSTM, config=config, pretrained_fnames=lm_fns2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "nbpresent": {
     "id": "12d9ce7c-fdb1-4108-ab95-764a015f8e0e"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nadal a gagné le tournoi de ▁2011. sncb ▁auprès ▁visible ▁transcription ▁natal griff ▁constituant ▁tétra ▁donnée ▁entière 产 anniversaire ▁journée 马 pend ▁mull ἄ ▁culminant 興 ▁frappe ▁tard ภ agrément ▁joué ὸ hui voy ▁baltique ▁ferdinand cara ▁climatique ե ime ▁déterminé ▁entre ር л fils ▁fièvre chart équipe ▁intégrée avance ▁grands ▁compagnies gonal pa ▁». џ ille ▁sub étalon ▁in ▁pistolet etto ▁poser bot ▁théo ▁capacité „ ▁dép ▁plante original în ▁width ▁mathématique ▁criminel ▁condé ▁sortant ▁iv ح স ▁met ▁taxi post cro ▁600 ̞ ▁descend ▁vente ▁renonce ca architecte gou vez ▁menacée ura ▁rat ▁prolongation ▁chrétiens ▁wi ▁maryland ▁restent ▁profond elle ▁tunnel ▁manga ▁accepta diff\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"Nadal a gagné le tournoi de\" # original text\n",
    "N_WORDS = 100 # number of words to predict following the TEXT\n",
    "N_SENTENCES = 1 # number of different predictions\n",
    "\n",
    "print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "nbpresent": {
     "id": "9be3d15c-526a-4ab7-bd46-16e8a8c58142"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nadal a gagné le tournoi de , and the pin - round for the introduction of the modern - day offensive , as well as a ▁funérailles - style version of the \" e \" in the version of the article \" the world of the old , modern . \" ( c . 2000 ) . \" i ' m not just a rich man , but an old friend . i just see it as the right one , \" he had a friend , \" and just a good friend . \" \" i ' m not a good man \" , he\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"Nadal a gagné le tournoi de\" # original text\n",
    "N_WORDS = 100 # number of words to predict following the TEXT\n",
    "N_SENTENCES = 1 # number of different predictions\n",
    "\n",
    "print(\"\\n\".join(learn_en.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "nbpresent": {
     "id": "35c213f4-3c90-4774-9cdd-56da02ebe3a8"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nadal a gagné le tournoi de ▁xxmaj ▁ s ot chi ▁en ▁cours ▁de ▁saison ▁ - ▁1981 ▁à ▁xxmaj ▁milan . ▁xxmaj ▁il ▁est ▁surtout ▁connu ▁pour ▁l ' ac cumul ation ▁de ▁l ' équipe ▁de ▁xxmaj ▁russie ▁xxmaj ▁en ly s ▁xxmaj ▁ pet ter rö th ▁et ▁xxmaj ▁wi ki pé dia ▁xxmaj ▁ gel der k omm odor e ▁qui ▁ont ▁été ▁prêté s ▁à ▁la ▁xxmaj ▁province ▁de ▁xxmaj ▁ s omo gy , ▁qui ▁l ' a ▁nommé ▁gouverneur ▁de ▁l ' ancienne ▁ université ▁de ▁xxmaj ▁ t bili s si . ▁xxmaj ▁il ▁ s ' agit\n"
     ]
    }
   ],
   "source": [
    "TEXT = \"Nadal a gagné le tournoi de\" # original text\n",
    "N_WORDS = 100 # number of words to predict following the TEXT\n",
    "N_SENTENCES = 1 # number of different predictions\n",
    "\n",
    "print(\"\\n\".join(learn_fr.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}