adapters/question-answering/question_answering_adapter_script.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rEJBSTyZIrIb"
   },
   "source": [
    "# Fine-tuning BERT (base or large) on a Question-Answering task by using the library adapter-transformers (script version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Credit**: [Hugging Face](https://huggingface.co/) and [adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)\n",
    "- **Author**: [Pierre GUILLOU](https://www.linkedin.com/in/pierreguillou/)\n",
    "- **Date**: 08/02/2021\n",
    "- **Blog post**: [NLP nas empresas | Como ajustar um modelo de linguagem natural como BERT para a tarefa de Question-Answering (QA) com um Adapter?](https://medium.com/@pierre_guillou/nlp-nas-empresas-como-ajustar-um-modelo-de-linguagem-natural-como-bert-para-a-tarefa-de-b7d2e7553a01)\n",
    "- **Link to the folder in github with this notebook and all necessary scripts**: [question-answering with adapters](https://github.com/piegu/language-models/tree/master/adapters/question-answering/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objective"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective here is to **fine-tune a Masked Language Model (MLM) like BERT (base or large) for a QA task by training adapters (library [adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)), not the embeddings and transformers layers of the MLM model**, and to compare results with BERT model fully fine-tune for the same task.\n",
    "\n",
    "The interest is obvious: if you need models for different NLP tasks, instead of fine-tuning and storing one model by NLP task, **you store only one MLM model and the trained tasks adapters which sizes are about 1% (QA adapter) and 8% (Lang+QA adapters) of the MLM model one** (it depends of the choosen adapter configuration). More, the loading of these adapters in production is very easy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will see how to fine-tune one of the [🤗 Transformers](https://github.com/huggingface/transformers) model to a question answering task, which is the task of extracting the answer to a question from a given context. We will use the library [adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers) and see how to easily load a dataset for these kinds of tasks and use the `Trainer` API to fine-tune a model on it.\n",
    "\n",
    "![Widget inference representing the QA task](images/question_answering_adapter.png)\n",
    "\n",
    "**Note:** This notebook finetunes models that answer question by taking a substring of a context, not by generating new text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4RRkXuteIrIh"
   },
   "source": [
    "This notebook is built to run on any question answering task with the same format as SQUAD (version 1 or 2), with any model checkpoint from the [Model Hub](https://huggingface.co/models) as long as that model has a version with a token classification head and a fast tokenizer (check on [this table](https://huggingface.co/transformers/index.html#bigtable) if this is the case). It might just need some small adjustments if you decide to use a different dataset than the one used here. Depending on you model and the GPU you are using, you might need to adjust the batch size to avoid out-of-memory errors. Set those three parameters, then the rest of the notebook should run smoothly:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### History and Credit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is an adaptation of the following notebooks and scripts for **fine-tuning a (transformer) Masked Language Model (MLM) like BERT (base or large) on the QA task with any QA dataset** (we use here the [Portuguese Squad 1.1 dataset](https://forum.ailab.unb.br/t/datasets-em-portugues/251/4)):\n",
    "- **from [adapter-transformers](https://github.com/Adapter-Hub/adapter-transformers)** | notebook [04_Cross_Lingual_Transfer.ipynb](https://github.com/Adapter-Hub/adapter-transformers/blob/master/notebooks/04_Cross_Lingual_Transfer.ipynb) and script [run_qa.py](https://github.com/Adapter-Hub/adapter-transformers/blob/master/examples/question-answering/run_qa.py) (this script was adapted from the script [run_qa.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py) of HF)\n",
    "- **from [transformers](https://github.com/huggingface/transformers) of Hugging Face** | notebook [question_answering.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb) and script [run_qa.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py) \n",
    "\n",
    "In order to speed up the fine-tuning of the model on only one GPU, the library [DeepSpeed](https://www.deepspeed.ai/) could be used by applying the configuration provided by HF in the notebook [transformers + deepspeed CLI](https://github.com/stas00/porting/blob/master/transformers/deepspeed/DeepSpeed_on_colab_CLI.ipynb) but as the library adapter-transformers is not synchronized with the last version of the library transformers of HF, we keep that option for the future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Major changes from original notebooks and scripts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script [run_qa.py](https://github.com/Adapter-Hub/adapter-transformers/blob/master/examples/question-answering/run_qa.py) allows to evaluate the model performance against f1 metric at the end of each epoch, and not against validation loss as done in the HF notebook [question_answering.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb). This is very important as we consider the metric when selecting a model, not the loss. Therefore, we decided to launch this script inside this notebook ([question_answering_adapter_script.ipynb](https://github.com/piegu/language-models/blob/master/adapters/question-answering/question_answering_adapter_script.ipynb)) (by simulating terminal command line) instead of running code in cells as done in the HF notebook. \n",
    "\n",
    "However, to provide an HF-like notebook, we also updated the HF notebook [question_answering.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb) on the notebook [question_answering_adapter.ipynb](https://github.com/piegu/language-models/blob/master/adapters/question-answering/question_answering_adapter.ipynb), but without the f1 metric as the evaluation metric during model training (ie, the f1 metric is used only after the model training)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More, we updated the script [run_qa.py](https://github.com/Adapter-Hub/adapter-transformers/blob/master/examples/question-answering/run_qa.py) to [run_qa_adapter.py](https://github.com/piegu/language-models/blob/master/adapters/question-answering/run_qa_adapter.py) with the following changes:\n",
    "- **EarlyStopping** by selecting the model with the highest eval f1 (patience of 3 before ending the training)\n",
    "- **MAD-X 2.0** that allows not to train adapters in the last transformer layer (read page 6 of [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://arxiv.org/pdf/2012.15562.pdf))\n",
    "- **Stack method** for the lang and task adapters when a lang adapter is loaded ([doc](https://docs.adapterhub.ml/adapter_composition.html?highlight=stack#stack))\n",
    "- **Houlsby MHA last layer** that allows no to train the task adapter after the Feed Fordward but only after the MHA (Multi-Head Attention) in the last layer for the Houlsby configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pathlib\n",
    "from pathlib import Path\n",
    "\n",
    "#root path\n",
    "root = Path.cwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python: 3.8.10 (default, Jun  4 2021, 15:09:15) \n",
      "[GCC 7.5.0]\n",
      "Pytorch: 1.9.0\n",
      "adapter-transformers: 2.1.1\n",
      "HF transformers: 4.8.2\n",
      "tokenizers: 0.10.3\n",
      "datasets: 1.9.0\n"
     ]
    }
   ],
   "source": [
    "import sys; print('python:',sys.version)\n",
    "\n",
    "import torch; print('Pytorch:',torch.__version__)\n",
    "\n",
    "import transformers; print('adapter-transformers:',transformers.__version__)\n",
    "import transformers; print('HF transformers:',transformers.__hf_version__)\n",
    "import tokenizers; print('tokenizers:',tokenizers.__version__)\n",
    "import datasets; print('datasets:',datasets.__version__)\n",
    "\n",
    "# import deepspeed; print('deepspeed:',deepspeed.__version__)\n",
    "\n",
    "# Versions used in the virtuel environment of this notebook:\n",
    "\n",
    "# python: 3.8.10 (default, Jun  4 2021, 15:09:15) \n",
    "# [GCC 7.5.0]\n",
    "# Pytorch: 1.9.0\n",
    "# adapter-transformers: 2.1.1\n",
    "# HF transformers: 4.8.2\n",
    "# tokenizers: 0.10.3\n",
    "# datasets: 1.9.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create symbolic links to the folder with the scripts to run or download them in the same folder of this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ln -s ~/adapter-transformers/examples/question-answering/run_qa_adapter.py\n",
    "# ln -s ~/adapter-transformers/examples/question-answering/trainer_qa.py\n",
    "# ln -s ~/adapter-transformers/examples/question-answering/utils_qa.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Model & dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select a MLM BERT base or large in the dataset language\n",
    "# model_checkpoint = \"neuralmind/bert-base-portuguese-cased\"\n",
    "model_checkpoint = \"neuralmind/bert-large-portuguese-cased\"\n",
    "\n",
    "# We can upload the model from its local archive, too\n",
    "model_checkpoint_original = \"neuralmind/bert-large-portuguese-cased\"\n",
    "model_checkpoint_original_local = root.parent/'neuralmind-bert-large-portuguese-cased-lenerbr/mlm_base'\n",
    "\n",
    "# SQuAD 1.1 in Portuguese\n",
    "dataset_name = \"squad11pt\"\n",
    "\n",
    "# This flag is the difference between SQUAD v1 or 2 (if you're using another dataset, it indicates if impossible\n",
    "# answers are allowed or not).\n",
    "version_2_with_negative = False # If true, some of the examples do not have an answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Main hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "task = \"qa\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training arguments\n",
    "batch_size = 16\n",
    "gradient_accumulation_steps = 1\n",
    "\n",
    "learning_rate = 1e-4\n",
    "num_train_epochs = 15.\n",
    "early_stopping_patience = 5\n",
    "\n",
    "adam_epsilon = 1e-7\n",
    "\n",
    "fp16 = True\n",
    "ds = False # If True, we use DeepSpeed\n",
    "\n",
    "# best model\n",
    "load_best_model_at_end = True \n",
    "metric_for_best_model = \"f1\"\n",
    "greater_is_better = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train adapter\n",
    "train_adapter = True # we want to train an adapter\n",
    "load_adapter = None # we do not upload an existing adapter \n",
    "\n",
    "# lang adapter\n",
    "with_adapters_mlm = False # if False, we do not upload an existing lang adapter\n",
    "\n",
    "if with_adapters_mlm:\n",
    "    adapter_composition = \"stack\" # we will stack the lang and task adapters\n",
    "else:\n",
    "    adapter_composition = None\n",
    "\n",
    "# if True, do not put adapter in the last transformer layer (Pfeiffer configuration)\n",
    "madx2 = False\n",
    "\n",
    "# if True, put only an adapter after the MHA but not after the FF in the last layer (Houlsby configuration)\n",
    "houlsby_MHA_lastlayer = True\n",
    "if madx2:\n",
    "    houlsby_MHA_lastlayer = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# gpu\n",
    "n_gpu = 1 # train on just one GPU\n",
    "gpu = 0 # select the GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select GPU 0\n",
    "import os\n",
    "os.environ['MASTER_ADDR'] = 'localhost'\n",
    "if gpu == 0:\n",
    "    os.environ['MASTER_PORT'] = '9996' # modify if RuntimeError: Address already in use # GPU 0\n",
    "elif gpu == 1:\n",
    "    os.environ['MASTER_PORT'] = '9995'\n",
    "os.environ['RANK'] = \"0\"\n",
    "os.environ['LOCAL_RANK'] = str(gpu)\n",
    "os.environ['WORLD_SIZE'] = \"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adapters config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# task adapter config\n",
    "adapter_config_name = \"pfeiffer\" # houlsby is possible, too\n",
    "if adapter_config_name == \"pfeiffer\":\n",
    "    adapter_non_linearity = 'gelu' # relu is possible, too\n",
    "elif adapter_config_name == \"houlsby\":\n",
    "    adapter_non_linearity = 'swish'\n",
    "adapter_reduction_factor = 16\n",
    "language = 'pt' # pt = Portuguese"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lang adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "if with_adapters_mlm:\n",
    "    \n",
    "    # hyperparameters used for fine-tuning the MLM with lang adapter\n",
    "    learning_rate_mlm = 1e-4\n",
    "    batch_size_mlm = 16\n",
    "    gradient_accumulation_steps_mlm = 1\n",
    "    adam_epsilon_mlm = 1e-4\n",
    "    num_train_epoch_mlm = 100.\n",
    "    early_stopping_patience_mlm = 10\n",
    "    madx2_mlm = madx2\n",
    "    houlsby_MHA_lastlayer_mlm = houlsby_MHA_lastlayer\n",
    "    ds_mlm = False\n",
    "    fp16_mlm = True\n",
    "    load_best_model_at_end_mlm = True\n",
    "    metric_for_best_model_mlm = \"loss\"\n",
    "    adapter_config_mlm = adapter_config_name + '+inv'\n",
    "\n",
    "    # path to lang adapter\n",
    "    outputs_mlm = model_checkpoint.replace('/','-') + '_' + dataset_name + '/' + 'mlm' + '/' \\\n",
    "    + 'lr' + str(learning_rate_mlm) \\\n",
    "    + '_bs' + str(batch_size_mlm) \\\n",
    "    + '_GAS' + str(gradient_accumulation_steps_mlm) \\\n",
    "    + '_eps' + str(adam_epsilon_mlm) \\\n",
    "    + '_epochs' + str(num_train_epoch_mlm) \\\n",
    "    + '_patience' + str(early_stopping_patience_mlm) \\\n",
    "    + '_madx2' + str(madx2_mlm) \\\n",
    "    + '_houlsby_MHA_lastlayer' + str(houlsby_MHA_lastlayer_mlm) \\\n",
    "    + '_ds' + str(ds_mlm) \\\n",
    "    + '_fp16' + str(fp16_mlm) \\\n",
    "    + '_best' + str(load_best_model_at_end_mlm) \\\n",
    "    + '_metric' + str(metric_for_best_model_mlm) \\\n",
    "    + '_adapterconfig' + str(adapter_config_mlm)\n",
    "\n",
    "    path_to_outputs_mlm = root/'outputs'/outputs_mlm\n",
    "    \n",
    "    # Config of the lang adapter\n",
    "    lang_adapter_path = path_to_outputs_mlm/'adapters-mlm/'\n",
    "    \n",
    "    load_lang_adapter = str(lang_adapter_path)\n",
    "    lang_adapter_config = str(lang_adapter_path) + \"/adapter_config.json\"\n",
    "    if adapter_config_mlm == \"pfeiffer+inv\":\n",
    "        lang_adapter_non_linearity = 'gelu' # relu is possible, too\n",
    "    elif adapter_config_mlm == \"houlsby+inv\":\n",
    "        lang_adapter_non_linearity = 'swish'\n",
    "    lang_adapter_reduction_factor = 2\n",
    "    language_mlm = language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training arguments of the HF trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup the training argument\n",
    "do_train = True \n",
    "do_eval = True \n",
    "\n",
    "# if you want to test the trainer, set up the following variables\n",
    "max_train_samples = 200 # None\n",
    "max_val_samples = 50 # None\n",
    "\n",
    "# epochs, bs, GA\n",
    "evaluation_strategy = \"epoch\" \n",
    "\n",
    "# fp16\n",
    "fp16_opt_level = 'O1'\n",
    "fp16_backend = \"auto\"\n",
    "fp16_full_eval = False\n",
    "\n",
    "# optimizer (AdamW)\n",
    "weight_decay = 0.01 # 0.0\n",
    "adam_beta1 = 0.9\n",
    "adam_beta2 = 0.999\n",
    "\n",
    "# scheduler\n",
    "lr_scheduler_type = 'linear'\n",
    "warmup_ratio = 0.0\n",
    "warmup_steps = 0\n",
    "\n",
    "# logs\n",
    "logging_strategy = \"steps\"\n",
    "logging_first_step = True # False\n",
    "logging_steps = 500     # if strategy = \"steps\"\n",
    "eval_steps = logging_steps # logging_steps\n",
    "\n",
    "# checkpoints\n",
    "save_strategy = \"epoch\" # steps\n",
    "save_steps = 500 # if save_strategy = \"steps\"\n",
    "save_total_limit = 1 # None\n",
    "\n",
    "# no cuda, seed\n",
    "no_cuda = False\n",
    "seed = 42\n",
    "\n",
    "# bar\n",
    "disable_tqdm = False # True\n",
    "remove_unused_columns = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# folder for training outputs\n",
    "\n",
    "outputs = model_checkpoint.replace('/','-') + '_' + dataset_name\n",
    "\n",
    "if with_adapters_mlm:\n",
    "    outputs = outputs + '/' + 'mlm_' + str(task) + '_AdCompo' + str(adapter_composition) + '/'\n",
    "else:\n",
    "    outputs = outputs + '/' + str(task) + '/'\n",
    "\n",
    "outputs = outputs \\\n",
    "+ 'lr' + str(learning_rate) \\\n",
    "+ '_bs' + str(batch_size) \\\n",
    "+ '_GAS' + str(gradient_accumulation_steps) \\\n",
    "+ '_eps' + str(adam_epsilon) \\\n",
    "+ '_epochs' + str(num_train_epochs) \\\n",
    "+ '_patience' + str(early_stopping_patience) \\\n",
    "+ '_wamlm' + str(with_adapters_mlm) \\\n",
    "+ '_madx2' + str(madx2) \\\n",
    "+ '_houlsby_MHA_lastlayer' + str(houlsby_MHA_lastlayer) \\\n",
    "+ '_ds' + str(ds) \\\n",
    "+ '_fp16' + str(fp16) \\\n",
    "+ '_best' + str(load_best_model_at_end) \\\n",
    "+ '_metric' + str(metric_for_best_model) \\\n",
    "+ '_adapterconfig' + str(adapter_config_name)\n",
    "\n",
    "# path to outputs\n",
    "path_to_outputs = root/'outputs'/outputs\n",
    "\n",
    "# subfolder for model outputs\n",
    "output_dir = path_to_outputs/'output_dir' \n",
    "overwrite_output_dir = True # False\n",
    "\n",
    "# logs\n",
    "logging_dir = path_to_outputs/'logging_dir'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The maximum total input sequence length after tokenization. Sequences longer\n",
    "# than this will be truncated, sequences shorter will be padded.\n",
    "max_seq_length = 384\n",
    "\n",
    "# Whether to pad all samples to `max_seq_length`.\n",
    "# If False, will pad the samples dynamically when batching to the maximum length in the batch (which can\n",
    "# be faster on GPU but will be slower on TPU).\n",
    "pad_to_max_length = True\n",
    "    \n",
    "# The threshold used to select the null answer: if the best answer has a score that is less than\n",
    "# the score of the null answer minus this threshold, the null answer is selected for this example.\n",
    "# Only useful when `version_2_with_negative=True`.\n",
    "null_score_diff_threshold = 0.0\n",
    "\n",
    "# When splitting up a long document into chunks, how much stride to take between chunks\n",
    "doc_stride = 128\n",
    "    \n",
    "# The total number of n-best predictions to generate when looking for an answer.\n",
    "n_best_size = 20\n",
    " \n",
    "# The maximum length of an answer that can be generated. This is needed because the start\n",
    "# and end predictions are not conditioned on one another.\n",
    "max_answer_length = 30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1r_n9OWV3l-Q"
   },
   "source": [
    "## 6. Preparing the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%time\n",
    "# if dataset_name == \"squad11pt\":\n",
    "    \n",
    "#     # create dataset folder \n",
    "#     path_to_dataset = root/'data'/dataset_name\n",
    "#     path_to_dataset.mkdir(parents=True, exist_ok=True) \n",
    "\n",
    "#     # Get dataset SQUAD in Portuguese\n",
    "#     %cd {path_to_dataset}\n",
    "#     !wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn\" -O squad-pt.tar.gz && rm -rf /tmp/cookies.txt\n",
    "\n",
    "#     # unzip \n",
    "#     !tar -xvf squad-pt.tar.gz\n",
    "\n",
    "#     # Get the train and validation json file in the HF script format \n",
    "#     # inspiration: file squad.py at https://github.com/huggingface/datasets/tree/master/datasets/squad\n",
    "\n",
    "#     import json \n",
    "#     files = ['squad-train-v1.1.json','squad-dev-v1.1.json']\n",
    "\n",
    "#     for file in files:\n",
    "\n",
    "#         # Opening JSON file & returns JSON object as a dictionary \n",
    "#         f = open(file, encoding=\"utf-8\") \n",
    "#         data = json.load(f) \n",
    "\n",
    "#         # Iterating through the json list \n",
    "#         entry_list = list()\n",
    "#         id_list = list()\n",
    "\n",
    "#         for row in data['data']: \n",
    "#             title = row['title']\n",
    "\n",
    "#             for paragraph in row['paragraphs']:\n",
    "#                 context = paragraph['context']\n",
    "\n",
    "#                 for qa in paragraph['qas']:\n",
    "#                     entry = {}\n",
    "\n",
    "#                     qa_id = qa['id']\n",
    "#                     question = qa['question']\n",
    "#                     answers = qa['answers']\n",
    "\n",
    "#                     entry['id'] = qa_id\n",
    "#                     entry['title'] = title.strip()\n",
    "#                     entry['context'] = context.strip()\n",
    "#                     entry['question'] = question.strip()\n",
    "\n",
    "#                     answer_starts = [answer[\"answer_start\"] for answer in answers]\n",
    "#                     answer_texts = [answer[\"text\"].strip() for answer in answers]\n",
    "#                     entry['answers'] = {}\n",
    "#                     entry['answers']['answer_start'] = answer_starts\n",
    "#                     entry['answers']['text'] = answer_texts\n",
    "\n",
    "#                     entry_list.append(entry)\n",
    "\n",
    "#         reverse_entry_list = entry_list[::-1]\n",
    "\n",
    "#         # for entries with same id, keep only last one (corrected texts by the group Deep Learning Brasil)\n",
    "#         unique_ids_list = list()\n",
    "#         unique_entry_list = list()\n",
    "#         for entry in reverse_entry_list:\n",
    "#             qa_id = entry['id']\n",
    "#             if qa_id not in unique_ids_list:\n",
    "#                 unique_ids_list.append(qa_id)\n",
    "#                 unique_entry_list.append(entry)\n",
    "\n",
    "#         # Closing file \n",
    "#         f.close() \n",
    "\n",
    "#         new_dict = {}\n",
    "#         new_dict['data'] = unique_entry_list\n",
    "\n",
    "#         file_name = 'pt_' + str(file)\n",
    "#         with open(file_name, 'w') as json_file:\n",
    "#             json.dump(new_dict, json_file)\n",
    "            \n",
    "# %cd {root}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f1-9jepM3l-W"
   },
   "source": [
    "You can replace the dataset above with any dataset hosted on [the hub](https://huggingface.co/datasets) or use your own files. Just uncomment the following cell and replace the paths with values that will lead to your files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "uxSaGa_l3l-W"
   },
   "outputs": [],
   "source": [
    "# datasets = load_dataset(\"text\", data_files={\"train\": path_to_train.txt, \"validation\": path_to_validation.txt}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jY1SwIrY3l-a"
   },
   "source": [
    "You can also load datasets from a csv or a JSON file, see the [full documentation](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset, load_metric\n",
    "\n",
    "if dataset_name == \"squad11pt\":\n",
    "    \n",
    "    # dataset folder \n",
    "    path_to_dataset = root/'data'/dataset_name\n",
    "    \n",
    "    # paths to files\n",
    "    train_file = str(path_to_dataset/'pt_squad-train-v1.1.json')\n",
    "    validation_file = str(path_to_dataset/'pt_squad-dev-v1.1.json')\n",
    "    \n",
    "    datasets = load_dataset('json', \n",
    "                            data_files={'train': train_file, \\\n",
    "                                        'validation': validation_file, \\\n",
    "                                       }, \n",
    "                            field='data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u3EtYfeHIrIz"
   },
   "source": [
    "To access an actual element, you need to select a split first, then give an index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "X6HrpprwIrIz",
    "outputId": "d7670bc0-42e4-4c09-8a6a-5c018ded7d95"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '5735c47ae853931400426b64',\n",
       " 'title': 'Kathmandu',\n",
       " 'context': 'A maioria das cozinhas encontradas em Katmandu não é vegetariana. No entanto, a prática do vegetarianismo não é incomum, e a culinária vegetariana pode ser encontrada em toda a cidade. O consumo de carne bovina é muito incomum e considerado tabu em muitos lugares. Buff (carne de búfalo Marinho) é muito comum. Há uma forte tradição de consumo de buffs em Katmandu, especialmente entre Newars, que não é encontrado em outras partes do Nepal. O consumo de carne de porco era considerado tabu até algumas décadas atrás. Devido à mistura com a cozinha Kirat do leste do Nepal, a carne de porco encontrou um lugar nos pratos de Katmandu. Uma população marginal de hindus e muçulmanos devotos o considera tabu. Os muçulmanos proíbem comer buff a partir do Alcorão, enquanto os hindus comem todas as variedades, exceto a carne de vaca, pois consideram a vaca uma deusa e símbolo da pureza. O café da manhã principal para moradores e visitantes é principalmente Momo ou Chowmein.',\n",
       " 'question': 'De que animal vem o lustre?',\n",
       " 'answers': {'answer_start': [280], 'text': ['búfalo Marinho']}}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets[\"train\"][10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WHUmphG3IrI3"
   },
   "source": [
    "To get a sense of what the data looks like, the following function will show some examples picked randomly in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "ur5sNUcZ3l-g"
   },
   "outputs": [],
   "source": [
    "from datasets import ClassLabel, Sequence\n",
    "import random\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "def show_random_elements(dataset, num_examples=10):\n",
    "    assert num_examples <= len(dataset), \"Can't pick more elements than there are in the dataset.\"\n",
    "    picks = []\n",
    "    for _ in range(num_examples):\n",
    "        pick = random.randint(0, len(dataset)-1)\n",
    "        while pick in picks:\n",
    "            pick = random.randint(0, len(dataset)-1)\n",
    "        picks.append(pick)\n",
    "    \n",
    "    df = pd.DataFrame(dataset[picks])\n",
    "    for column, typ in dataset.features.items():\n",
    "        if isinstance(typ, ClassLabel):\n",
    "            df[column] = df[column].transform(lambda i: typ.names[i])\n",
    "        elif isinstance(typ, Sequence) and isinstance(typ.feature, ClassLabel):\n",
    "            df[column] = df[column].transform(lambda x: [typ.feature.names[i] for i in x])\n",
    "    display(HTML(df.to_html()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "1Uk8NROQ3l-k",
    "outputId": "a822dcec-51e3-4dba-c73c-dba9e0301726"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>context</th>\n",
       "      <th>question</th>\n",
       "      <th>answers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5731d9f3e17f3d1400422495</td>\n",
       "      <td>Bras%C3%ADlia</td>\n",
       "      <td>Brasília (pronúncia em português: [bɾaˈziljɐ]) é a capital federal do Brasil e sede do governo do Distrito Federal. A cidade está localizada no topo do planalto brasileiro na região centro-oeste do país. Foi fundada em 21 de abril de 1960, para servir como a nova capital nacional. Brasília e seu metrô (abrangendo todo o Distrito Federal) tinham uma população de 2.556.149 em 2011, sendo a quarta cidade mais populosa do Brasil. Entre as principais cidades latino-americanas, Brasília possui o maior PIB per capita em R $ 61.915 (US $ 36.175).</td>\n",
       "      <td>Qual é a capital do Brasil?</td>\n",
       "      <td>{'answer_start': [0], 'text': ['Brasília']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>57278d02f1498d1400e8fbc9</td>\n",
       "      <td>Carnival</td>\n",
       "      <td>Tarragona tem uma das sequências rituais mais completas da região. Os eventos começam com a construção de um enorme barril e terminam com a queima das efígies do rei e da rainha. No sábado, o desfile principal acontece com grupos mascarados, figuras zoomórficas, bandas de música e percussão e grupos com fogos de artifício (os demônios, o dragão, o boi, a fêmea). Grupos de carnaval se destacam por suas roupas cheias de elegância, mostrando exemplos brilhantes de artesanato em tecido, nos desfiles de sábado e domingo. Cerca de 5.000 pessoas são membros dos grupos de desfiles.</td>\n",
       "      <td>De que estão cheias as roupas dos grupos de carnaval?</td>\n",
       "      <td>{'answer_start': [422], 'text': ['elegância']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5735190cacc1501500bac403</td>\n",
       "      <td>Hunting</td>\n",
       "      <td>Desde 1934, a venda do Federal Duck Stamps gerou US $ 670 milhões e ajudou a comprar ou arrendar 5.200.000 acres (8.100 sq mi; 21.000 km2) de habitat. Os selos servem como uma licença para caçar aves migratórias, um passe de entrada para todas as áreas do Refúgio Nacional da Vida Selvagem, e também são considerados itens de colecionador, muitas vezes comprados por razões estéticas fora das comunidades de caça e observação de pássaros. Embora os não caçadores comprem um número significativo de selos de patos, oitenta e sete por cento de suas vendas são contribuídos por caçadores, o que é lógico, pois os caçadores são obrigados a comprá-los. A distribuição de fundos é gerenciada pela Comissão de Conservação de Aves Migratórias (MBCC).</td>\n",
       "      <td>O que os selos concedem ao comprador uma licença para fazer?</td>\n",
       "      <td>{'answer_start': [189], 'text': ['caçar aves migratórias']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>57278f77f1498d1400e8fc20</td>\n",
       "      <td>FA_Cup</td>\n",
       "      <td>A possibilidade de vitórias improváveis nas rodadas anteriores da competição, em que equipes de classificação mais baixa vencem oposição mais alta, conhecida como \"matanças gigantes\", é muito esperada pelo público e é considerada parte integrante da tradição e prestígio da competição. , ao lado do ganho pelas equipes vencedoras da competição. Quase todos os clubes da Pirâmide da Liga têm um ato de matança de gigantes lembrado com carinho em sua história. É considerado particularmente digno de nota quando um time de primeira linha da Premier League sofre uma derrota chateada ou quando o assassino gigante é um clube que não pertence à liga, ou seja, de fora dos níveis profissionais da Liga de Futebol Americano.</td>\n",
       "      <td>O que é um assassino gigante?</td>\n",
       "      <td>{'answer_start': [19], 'text': ['vitórias improváveis nas rodadas anteriores da competição, em que equipes de classificação mais baixa vencem oposição mais alta']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>573254560fdd8d15006c69bd</td>\n",
       "      <td>Jehovah%27s_Witnesses</td>\n",
       "      <td>Em agosto de 2015, as Testemunhas de Jeová relatam uma média de 8,2 milhões de editores - o termo que eles usam para os membros envolvidos ativamente na pregação - em 118.016 congregações. Em 2015, esses relatórios indicaram mais de 1,93 bilhões de horas gastas em atividades de pregação e \"estudo da Bíblia\". Desde meados dos anos 90, o número de publicadores de pico aumentou de 4,5 milhões para 8,2 milhões. No mesmo ano, eles conduziram \"estudos bíblicos\" com mais de 9,7 milhões de indivíduos, incluindo aqueles realizados pelos pais das Testemunhas de Jeová com seus filhos. As Testemunhas de Jeová estimam que a atual taxa de crescimento mundial é de 1,5% ao ano.</td>\n",
       "      <td>Que termo as Testemunhas de Jeová usam para os membros ativamente envolvidos na pregação?</td>\n",
       "      <td>{'answer_start': [79], 'text': ['editores']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>572f72aeb2c2fd1400568138</td>\n",
       "      <td>Database</td>\n",
       "      <td>O trabalho de Codd foi recolhido por duas pessoas em Berkeley, Eugene Wong e Michael Stonebraker. Eles iniciaram um projeto conhecido como INGRES usando financiamento que já havia sido alocado para um projeto de banco de dados geográficos e programadores de estudantes para produzir código. A partir de 1973, o INGRES entregou seus primeiros produtos de teste que estavam geralmente prontos para uso generalizado em 1979. O INGRES era semelhante ao Sistema R de várias maneiras, incluindo o uso de uma \"linguagem\" para acesso a dados, conhecida como QUEL. Com o tempo, o INGRES passou para o padrão SQL emergente.</td>\n",
       "      <td>Qual era o nome do projeto para criar um banco de dados geográficos?</td>\n",
       "      <td>{'answer_start': [311], 'text': ['INGRES']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>5726f7dbf1498d1400e8f148</td>\n",
       "      <td>Yale_University</td>\n",
       "      <td>Yale produziu ex-alunos destacados em seus respectivos campos. Entre os mais conhecidos estão os presidentes dos EUA William Howard Taft, Gerald Ford, George H.W. Bush, Bill Clinton e George W. Bush; membros da realeza Princesa Victoria Victoria Bernadotte, Príncipe Rostislav Romanov e Príncipe Akiiki Hosea Nyabongo; chefes de estado, incluindo o primeiro-ministro italiano Mario Monti, o primeiro-ministro turco Tansu Çiller, o presidente mexicano Ernesto Zedillo, o presidente alemão Karl Carstens e o presidente das Filipinas, José Paciano Laurel; Juízes da Suprema Corte dos EUA Sonia Sotomayor, Samuel Alito e Clarence Thomas; Secretários de Estado dos EUA John Kerry, Hillary Clinton, Cyrus Vance e Dean Acheson; os autores Sinclair Lewis, Stephen Vincent Benét e Tom Wolfe; lexicógrafo Noah Webster; inventores Samuel F. B. Morse e Eli Whitney; patriota e \"primeiro espião\" Nathan Hale; teólogo Jonathan Edwards; atores, diretores e produtores Paul Newman, Henry Winkler, Vincent Price, Meryl Streep, Sigourney Weaver, Jodie Foster, Angela Bassett, Patricia Clarkson, Courtney Vance, Frances McDormand, Elia Kazan, George Roy Hill, Edward Norton, Lupita Nyong'o, Allison Williams, Oliver Stone, Sam Waterston e Michael Cimino; \"Pai do futebol americano\" Walter Camp, James Franco, \"O remador perfeito\" Rusty Wailes; jogadores de beisebol Ron Darling, Bill Hutchinson e Craig Breslow; jogador de basquete Chris Dudley; jogadores de futebol Gary Fencik e Calvin Hill; jogadores de hóquei Chris Higgins e Mike Richter; patinadora artística Sarah Hughes; nadador Don Schollander; esquiador Ryan Max Riley; corredor Frank Shorter; compositores Charles Ives, Douglas Moore e Cole Porter; Sargent Shriver, fundador do Peace Corps; psicólogo infantil Benjamin Spock; arquitetos Eero Saarinen e Norman Foster; escultor Richard Serra; crítico de cinema Gene Siskel; comentaristas de televisão Dick Cavett e Anderson Cooper; O jornalista do New York Times David Gonzalez; especialistas William F. Buckley, Jr. e Fareed Zakaria; os economistas Irving Fischer, Mahbub ul Haq e Paul Krugman; inventor do ciclotrão e ganhador do Nobel de Física, Ernest Lawrence; Diretor do Projeto Genoma Humano Francis S. Collins; matemático e químico Josiah Willard Gibbs; e empresários, incluindo o co-fundador da Time Magazine Henry Luce, o fundador do Morgan Stanley Harold Stanley, o CEO da Boeing James McNerney, o fundador da FedEx Frederick W. Smith, o presidente da Time Warner Jeffrey Bewkes, o co-fundador da Electronic Arts Bing Gordon e o investidor / filantropo Sir John Templeton; pioneira em aplicações elétricas Austin Cornelius Dunham.</td>\n",
       "      <td>Qual realeza participou de Yale?</td>\n",
       "      <td>{'answer_start': [219], 'text': ['Princesa Victoria Victoria Bernadotte, Príncipe Rostislav Romanov e Príncipe Akiiki Hosea Nyabongo']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>572faad8b2c2fd14005682eb</td>\n",
       "      <td>Pain</td>\n",
       "      <td>A presença de dor em um animal não pode ser conhecida com certeza, mas pode ser inferida por meio de reações físicas e comportamentais. Atualmente, os especialistas acreditam que todos os vertebrados podem sentir dor e que certos invertebrados, como o polvo, também podem sentir. Quanto a outros animais, plantas ou outras entidades, sua capacidade de sentir dor física é atualmente uma questão fora do alcance científico, uma vez que nenhum mecanismo é conhecido pelo qual eles possam ter esse sentimento. Em particular, não há nociceptores conhecidos em grupos como plantas, fungos e a maioria dos insetos, exceto, por exemplo, nas moscas da fruta.</td>\n",
       "      <td>O que os fungos e as moscas da fruta parecem não ter?</td>\n",
       "      <td>{'answer_start': [529], 'text': ['nociceptores']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>572b3ab4111d821400f38de5</td>\n",
       "      <td>Empiricism</td>\n",
       "      <td>A maioria dos seguidores de Hume discordou de sua conclusão de que a crença em um mundo externo é racionalmente injustificável, sustentando que os próprios princípios de Hume continham implicitamente a justificativa racional para tal crença, ou seja, além de se contentar em deixar a questão repousar no instinto humano, costume e hábito. De acordo com uma teoria empirista extrema conhecida como fenomenalismo, antecipada pelos argumentos de Hume e George Berkeley, um objeto físico é um tipo de construção a partir de nossas experiências. Fenomenalismo é a visão de que objetos físicos, propriedades, eventos (o que quer que seja físico) são redutíveis a objetos mentais, propriedades, eventos. Por fim, apenas objetos mentais, propriedades, eventos existem - daí o termo intimamente relacionado idealismo subjetivo. Pela linha de pensamento fenomenalista, ter uma experiência visual de uma coisa física real é ter uma experiência de um certo tipo de grupo de experiências. Esse tipo de conjunto de experiências possui uma constância e coerência que faltam no conjunto de experiências das quais alucinações, por exemplo, fazem parte. Como John Stuart Mill colocou em meados do século XIX, a matéria é a \"possibilidade permanente de sensação\". O empirismo de Mill deu um passo significativo além de Hume em outro aspecto: ao sustentar que a indução é necessária para todo conhecimento significativo, incluindo a matemática. Como resumido por D.W. Hamlin:</td>\n",
       "      <td>O que Hume disse que não pode ser racionalmente justificado?</td>\n",
       "      <td>{'answer_start': [69], 'text': ['crença em um mundo externo']}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>572e9d48cb0c0d14000f136f</td>\n",
       "      <td>Vacuum</td>\n",
       "      <td>A descompressão rápida pode ser muito mais perigosa do que a própria exposição ao vácuo. Mesmo que a vítima não prenda a respiração, a ventilação pela traquéia pode ser muito lenta para evitar a ruptura fatal dos delicados alvéolos dos pulmões. Os terremotos e seios nasais podem ser rompidos por descompressão rápida, os tecidos moles podem machucar e infiltrar sangue, e o estresse do choque acelerará o consumo de oxigênio, levando à hipóxia. Lesões causadas por descompressão rápida são chamadas barotrauma. Uma queda de pressão de 13 kPa (100 Torr), que não produz sintomas se for gradual, pode ser fatal se ocorrer repentinamente.</td>\n",
       "      <td>O que a aceleração do consumo de oxigênio faz?</td>\n",
       "      <td>{'answer_start': [437], 'text': ['hipóxia']}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_random_elements(datasets[\"train\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Training + Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setup environment variables "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The magic command `%env` corresponds to `export` in linux. It allows to setup the values of all arguments of the script `run_qa_adapter.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "envs = {\n",
    "'n_gpu':n_gpu,\n",
    "'gpu':gpu,\n",
    "'CUDA_VISIBLE_DEVICES':gpu,\n",
    "'model_name_or_path':model_checkpoint,\n",
    "'dataset_name':dataset_name,\n",
    "'train_file':train_file,\n",
    "'validation_file':validation_file,\n",
    "'do_train':do_train,\n",
    "'do_eval':do_eval,\n",
    "'max_train_samples':max_train_samples,\n",
    "'max_val_samples':max_train_samples,\n",
    "'output_dir':output_dir,\n",
    "'overwrite_output_dir':overwrite_output_dir,\n",
    "'max_seq_length':max_seq_length,\n",
    "'pad_to_max_length':pad_to_max_length,\n",
    "'null_score_diff_threshold':null_score_diff_threshold,\n",
    "'doc_stride':doc_stride,\n",
    "'n_best_size':n_best_size,\n",
    "'max_answer_length':max_answer_length,\n",
    "'evaluation_strategy':evaluation_strategy,\n",
    "'per_device_train_batch_size':batch_size,\n",
    "'per_device_eval_batch_size':batch_size,\n",
    "'gradient_accumulation_steps':gradient_accumulation_steps,\n",
    "'learning_rate':learning_rate,\n",
    "'weight_decay':weight_decay,\n",
    "'adam_beta1':adam_beta1,\n",
    "'adam_beta2':adam_beta2,\n",
    "'adam_epsilon':adam_epsilon,\n",
    "'num_train_epochs':num_train_epochs,\n",
    "'warmup_ratio':warmup_ratio,\n",
    "'warmup_steps':warmup_steps,\n",
    "'logging_dir':logging_dir,\n",
    "'logging_strategy':logging_strategy,\n",
    "'logging_first_step':logging_first_step,\n",
    "'logging_steps':logging_steps,\n",
    "'eval_steps':eval_steps,\n",
    "'save_strategy':save_strategy,\n",
    "'save_steps':save_steps,\n",
    "'save_total_limit':save_total_limit,\n",
    "'no_cuda':no_cuda,\n",
    "'seed':seed,\n",
    "'fp16':fp16,\n",
    "'fp16_opt_level':fp16_opt_level,\n",
    "'fp16_backend':fp16_backend,\n",
    "'fp16_full_eval':fp16_full_eval,\n",
    "'disable_tqdm':disable_tqdm,\n",
    "'remove_unused_columns':remove_unused_columns,\n",
    "'load_best_model_at_end':load_best_model_at_end,\n",
    "'metric_for_best_model':metric_for_best_model,\n",
    "'greater_is_better':greater_is_better,\n",
    "'early_stopping_patience':early_stopping_patience,\n",
    "'madx2':madx2,\n",
    "'houlsby_MHA_lastlayer':houlsby_MHA_lastlayer,\n",
    "'train_adapter':train_adapter,\n",
    "'adapter_config_name':adapter_config_name,\n",
    "'adapter_non_linearity':adapter_non_linearity,\n",
    "'adapter_reduction_factor':adapter_reduction_factor,\n",
    "'language':language,\n",
    "'adapter_composition':adapter_composition\n",
    "}\n",
    "\n",
    "if with_adapters_mlm:\n",
    "    envs['load_lang_adapter']=load_lang_adapter\n",
    "    envs['lang_adapter_config']=lang_adapter_config\n",
    "    envs['lang_adapter_non_linearity']=lang_adapter_non_linearity\n",
    "    envs['lang_adapter_reduction_factor']=lang_adapter_reduction_factor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: n_gpu=1\n",
      "env: gpu=0\n",
      "env: CUDA_VISIBLE_DEVICES=0\n",
      "env: model_name_or_path=neuralmind/bert-base-portuguese-cased\n",
      "env: dataset_name=squad11pt\n",
      "env: train_file=/mnt/home/pierre/course-v4/nbs/MLM/language-modeling/data/squad11pt/pt_squad-train-v1.1.json\n",
      "env: validation_file=/mnt/home/pierre/course-v4/nbs/MLM/language-modeling/data/squad11pt/pt_squad-dev-v1.1.json\n",
      "env: do_train=True\n",
      "env: do_eval=True\n",
      "env: max_train_samples=200\n",
      "env: max_val_samples=200\n",
      "env: output_dir=/mnt/home/pierre/course-v4/nbs/MLM/language-modeling/models_outputs/neuralmind-bert-base-portuguese-cased_squad11pt/qa_lr0.0001_bs16_eps1e-06_epochs15.0_patience5_wamlmFalse_madx2False_houlsby_MHA_lastlayerTrue_dsFalse_fp16True_bestTrue_metricf1_adapterconfighoulsby/output_dir\n",
      "env: overwrite_output_dir=True\n",
      "env: max_seq_length=384\n",
      "env: pad_to_max_length=True\n",
      "env: null_score_diff_threshold=0.0\n",
      "env: doc_stride=128\n",
      "env: n_best_size=20\n",
      "env: max_answer_length=30\n",
      "env: evaluation_strategy=epoch\n",
      "env: per_device_train_batch_size=16\n",
      "env: per_device_eval_batch_size=16\n",
      "env: gradient_accumulation_steps=1\n",
      "env: learning_rate=0.0001\n",
      "env: weight_decay=0.01\n",
      "env: adam_beta1=0.9\n",
      "env: adam_beta2=0.999\n",
      "env: adam_epsilon=1e-06\n",
      "env: num_train_epochs=15.0\n",
      "env: warmup_ratio=0.0\n",
      "env: warmup_steps=0\n",
      "env: logging_dir=/mnt/home/pierre/course-v4/nbs/MLM/language-modeling/models_outputs/neuralmind-bert-base-portuguese-cased_squad11pt/qa_lr0.0001_bs16_eps1e-06_epochs15.0_patience5_wamlmFalse_madx2False_houlsby_MHA_lastlayerTrue_dsFalse_fp16True_bestTrue_metricf1_adapterconfighoulsby/logging_dir\n",
      "env: logging_strategy=steps\n",
      "env: logging_first_step=True\n",
      "env: logging_steps=500\n",
      "env: eval_steps=500\n",
      "env: save_strategy=epoch\n",
      "env: save_steps=500\n",
      "env: save_total_limit=1\n",
      "env: no_cuda=False\n",
      "env: seed=42\n",
      "env: fp16=True\n",
      "env: fp16_opt_level=O1\n",
      "env: fp16_backend=auto\n",
      "env: fp16_full_eval=False\n",
      "env: disable_tqdm=False\n",
      "env: remove_unused_columns=True\n",
      "env: load_best_model_at_end=True\n",
      "env: metric_for_best_model=f1\n",
      "env: greater_is_better=True\n",
      "env: early_stopping_patience=5\n",
      "env: madx2=False\n",
      "env: houlsby_MHA_lastlayer=True\n",
      "env: train_adapter=True\n",
      "env: adapter_config_name=houlsby\n",
      "env: adapter_non_linearity=swish\n",
      "env: adapter_reduction_factor=16\n",
      "env: language=pt\n",
      "env: adapter_composition=None\n"
     ]
    }
   ],
   "source": [
    "for k,v in envs.items():\n",
    "    %env {k}={v}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Delete the output_dir (if exists)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -r {output_dir}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can launch the training :-) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "PATH = os.getenv('PATH')\n",
    "# replace xxxx by your username on your server (ex: paulo)\n",
    "# replace yyyy by the name of the virtual environment of this notebook (ex: adapter-transformers)\n",
    "%env PATH=/mnt/home/xxxx/anaconda3/envs/yyyy/bin:$PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['MKL_THREADING_LAYER'] = 'GNU'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy/paste/uncomment the 2 following lines in the following cell if you want to limit the number of data (useful for testing)\n",
    "# --max_train_samples $max_train_samples \\\n",
    "# --max_val_samples $max_val_samples \\"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "if with_adapters_mlm:\n",
    "    # with lang adapter\n",
    "    !python -m torch.distributed.launch --nproc_per_node=$n_gpu run_qa_adapter.py \\\n",
    "    --model_name_or_path $model_name_or_path \\\n",
    "    --train_file $train_file \\\n",
    "    --validation_file $validation_file \\\n",
    "    --do_train $do_train \\\n",
    "    --do_eval $do_eval \\\n",
    "    --output_dir $output_dir \\\n",
    "    --overwrite_output_dir $overwrite_output_dir \\\n",
    "    --max_seq_length $max_seq_length \\\n",
    "    --pad_to_max_length $pad_to_max_length \\\n",
    "    --null_score_diff_threshold $null_score_diff_threshold \\\n",
    "    --doc_stride $doc_stride \\\n",
    "    --n_best_size $n_best_size \\\n",
    "    --max_answer_length $max_answer_length \\\n",
    "    --evaluation_strategy $evaluation_strategy \\\n",
    "    --per_device_train_batch_size $per_device_train_batch_size \\\n",
    "    --per_device_eval_batch_size $per_device_eval_batch_size \\\n",
    "    --gradient_accumulation_steps $gradient_accumulation_steps \\\n",
    "    --learning_rate $learning_rate \\\n",
    "    --weight_decay $weight_decay \\\n",
    "    --adam_beta1 $adam_beta1 \\\n",
    "    --adam_beta2 $adam_beta2 \\\n",
    "    --adam_epsilon $adam_epsilon \\\n",
    "    --num_train_epochs $num_train_epochs \\\n",
    "    --warmup_ratio $warmup_ratio \\\n",
    "    --warmup_steps $warmup_steps \\\n",
    "    --logging_dir $logging_dir \\\n",
    "    --logging_strategy $logging_strategy \\\n",
    "    --logging_first_step $logging_first_step \\\n",
    "    --logging_steps $logging_steps \\\n",
    "    --eval_steps $eval_steps \\\n",
    "    --save_strategy $save_strategy \\\n",
    "    --save_steps $save_steps \\\n",
    "    --save_total_limit $save_total_limit \\\n",
    "    --no_cuda $no_cuda \\\n",
    "    --seed $seed \\\n",
    "    --fp16 $fp16 \\\n",
    "    --fp16_opt_level $fp16_opt_level \\\n",
    "    --fp16_backend $fp16_backend \\\n",
    "    --fp16_full_eval $fp16_full_eval \\\n",
    "    --disable_tqdm $disable_tqdm \\\n",
    "    --remove_unused_columns $remove_unused_columns \\\n",
    "    --load_best_model_at_end $load_best_model_at_end \\\n",
    "    --metric_for_best_model $metric_for_best_model \\\n",
    "    --greater_is_better $greater_is_better \\\n",
    "    --early_stopping_patience $early_stopping_patience \\\n",
    "    --madx2 $madx2 \\\n",
    "    --houlsby_MHA_lastlayer $houlsby_MHA_lastlayer \\\n",
    "    --train_adapter $train_adapter \\\n",
    "    --adapter_config $adapter_config_name \\\n",
    "    --adapter_non_linearity $adapter_non_linearity \\\n",
    "    --adapter_reduction_factor $adapter_reduction_factor \\\n",
    "    --language $language \\\n",
    "    --adapter_composition $adapter_composition \\\n",
    "    --load_lang_adapter $load_lang_adapter \\\n",
    "    --lang_adapter_config $lang_adapter_config \\\n",
    "    --lang_adapter_non_linearity $lang_adapter_non_linearity \\\n",
    "    --lang_adapter_reduction_factor $lang_adapter_reduction_factor\n",
    "else:\n",
    "    # without lang adapter\n",
    "    !python -m torch.distributed.launch --nproc_per_node=$n_gpu run_qa_adapter.py \\\n",
    "    --model_name_or_path $model_name_or_path \\\n",
    "    --train_file $train_file \\\n",
    "    --validation_file $validation_file \\\n",
    "    --do_train $do_train \\\n",
    "    --do_eval $do_eval \\\n",
    "    --output_dir $output_dir \\\n",
    "    --overwrite_output_dir $overwrite_output_dir \\\n",
    "    --max_seq_length $max_seq_length \\\n",
    "    --pad_to_max_length $pad_to_max_length \\\n",
    "    --null_score_diff_threshold $null_score_diff_threshold \\\n",
    "    --doc_stride $doc_stride \\\n",
    "    --n_best_size $n_best_size \\\n",
    "    --max_answer_length $max_answer_length \\\n",
    "    --evaluation_strategy $evaluation_strategy \\\n",
    "    --per_device_train_batch_size $per_device_train_batch_size \\\n",
    "    --per_device_eval_batch_size $per_device_eval_batch_size \\\n",
    "    --gradient_accumulation_steps $gradient_accumulation_steps \\\n",
    "    --learning_rate $learning_rate \\\n",
    "    --weight_decay $weight_decay \\\n",
    "    --adam_beta1 $adam_beta1 \\\n",
    "    --adam_beta2 $adam_beta2 \\\n",
    "    --adam_epsilon $adam_epsilon \\\n",
    "    --num_train_epochs $num_train_epochs \\\n",
    "    --warmup_ratio $warmup_ratio \\\n",
    "    --warmup_steps $warmup_steps \\\n",
    "    --logging_dir $logging_dir \\\n",
    "    --logging_strategy $logging_strategy \\\n",
    "    --logging_first_step $logging_first_step \\\n",
    "    --logging_steps $logging_steps \\\n",
    "    --eval_steps $eval_steps \\\n",
    "    --save_strategy $save_strategy \\\n",
    "    --save_steps $save_steps \\\n",
    "    --save_total_limit $save_total_limit \\\n",
    "    --no_cuda $no_cuda \\\n",
    "    --seed $seed \\\n",
    "    --fp16 $fp16 \\\n",
    "    --fp16_opt_level $fp16_opt_level \\\n",
    "    --fp16_backend $fp16_backend \\\n",
    "    --fp16_full_eval $fp16_full_eval \\\n",
    "    --disable_tqdm $disable_tqdm \\\n",
    "    --remove_unused_columns $remove_unused_columns \\\n",
    "    --load_best_model_at_end $load_best_model_at_end \\\n",
    "    --metric_for_best_model $metric_for_best_model \\\n",
    "    --greater_is_better $greater_is_better \\\n",
    "    --early_stopping_patience $early_stopping_patience \\\n",
    "    --madx2 $madx2 \\\n",
    "    --houlsby_MHA_lastlayer $houlsby_MHA_lastlayer \\\n",
    "    --train_adapter $train_adapter \\\n",
    "    --adapter_config $adapter_config_name \\\n",
    "    --adapter_non_linearity $adapter_non_linearity \\\n",
    "    --adapter_reduction_factor $adapter_reduction_factor \\\n",
    "    --language $language \\\n",
    "    --adapter_composition $adapter_composition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 6756\r\n",
      "drwxrwxr-x 2 pierre pierre    4096 Jul  9 23:52 .\r\n",
      "drwxrwxr-x 4 pierre pierre    4096 Jul  9 23:56 ..\r\n",
      "-rw-rw-r-- 1 pierre pierre     629 Jul  9 23:52 adapter_config.json\r\n",
      "-rw-rw-r-- 1 pierre pierre     240 Jul  9 23:52 head_config.json\r\n",
      "-rw-rw-r-- 1 pierre pierre 6889489 Jul  9 23:52 pytorch_adapter.bin\r\n",
      "-rw-rw-r-- 1 pierre pierre    7143 Jul  9 23:52 pytorch_model_head.bin\r\n"
     ]
    }
   ],
   "source": [
    "# folder of the saved task adapter\n",
    "!ls -al {output_dir/task}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_dir/task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can push the saved adapter + head to the [AdapterHub](https://adapterhub.ml/) (follow instructions at [Contributing to Adapter Hub](https://docs.adapterhub.ml/contributing.html))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. TensorBoard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "PATH = os.getenv('PATH')\n",
    "# replace xxxx by your username on your server (ex: paulo)\n",
    "# replace yyyy by the name of the virtual environment of this notebook (ex: adapter-transformers)\n",
    "%env PATH=/mnt/home/xxxx/anaconda3/envs/yyyy/bin:$PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard\n",
    "# %reload_ext tensorboard\n",
    "%tensorboard --logdir {logging_dir} --bind_all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Application QA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true,
    "id": "Ckl2Fzn3is0F"
   },
   "outputs": [],
   "source": [
    "### import transformers\n",
    "import pathlib\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "hidden": true,
    "id": "IJA9CgOBis0F",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at neuralmind/bert-base-portuguese-cased were not used when initializing BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForQuestionAnswering were not initialized from the model checkpoint at neuralmind/bert-base-portuguese-cased and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForQuestionAnswering, AutoTokenizer\n",
    "\n",
    "if model_checkpoint_original == \"neuralmind/bert-large-portuguese-cased\":\n",
    "    tokenizer_qa = AutoTokenizer.from_pretrained(model_checkpoint_original_local)\n",
    "    model_qa = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint_original_local)\n",
    "else:\n",
    "    tokenizer_qa = AutoTokenizer.from_pretrained(model_checkpoint)\n",
    "    model_qa = AutoModelForQuestionAnswering.from_pretrained(model_checkpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lang adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if with_adapters_mlm:\n",
    "    \n",
    "    # hyperparameters used for fine-tuning the MLM with lang adapter\n",
    "    learning_rate_mlm = 1e-4\n",
    "    batch_size_mlm = 16\n",
    "    gradient_accumulation_steps_mlm = 1\n",
    "    adam_epsilon_mlm = 1e-4\n",
    "    num_train_epoch_mlm = 100.\n",
    "    early_stopping_patience_mlm = 10\n",
    "    madx2_mlm = madx2\n",
    "    houlsby_MHA_lastlayer_mlm = houlsby_MHA_lastlayer\n",
    "    ds_mlm = False\n",
    "    fp16_mlm = True\n",
    "    load_best_model_at_end_mlm = True\n",
    "    metric_for_best_model_mlm = \"loss\"\n",
    "    adapter_config_mlm = adapter_config_name + '+inv'\n",
    "\n",
    "    # path to lang adapter\n",
    "    outputs_mlm = model_checkpoint.replace('/','-') + '_' + dataset_name + '/' + 'mlm' + '/' \\\n",
    "    + 'lr' + str(learning_rate_mlm) \\\n",
    "    + '_bs' + str(batch_size_mlm) \\\n",
    "    + '_GAS' + str(gradient_accumulation_steps_mlm) \\\n",
    "    + '_eps' + str(adam_epsilon_mlm) \\\n",
    "    + '_epochs' + str(num_train_epoch_mlm) \\\n",
    "    + '_patience' + str(early_stopping_patience_mlm) \\\n",
    "    + '_madx2' + str(madx2_mlm) \\\n",
    "    + '_houlsby_MHA_lastlayer' + str(houlsby_MHA_lastlayer_mlm) \\\n",
    "    + '_ds' + str(ds_mlm) \\\n",
    "    + '_fp16' + str(fp16_mlm) \\\n",
    "    + '_best' + str(load_best_model_at_end_mlm) \\\n",
    "    + '_metric' + str(metric_for_best_model_mlm) \\\n",
    "    + '_adapterconfig' + str(adapter_config_mlm)\n",
    "\n",
    "    path_to_outputs_mlm = root/'outputs'/outputs_mlm\n",
    "    \n",
    "    # Config of the lang adapter\n",
    "    lang_adapter_path = path_to_outputs_mlm/'adapters-mlm/'\n",
    "    \n",
    "    load_lang_adapter = str(lang_adapter_path)\n",
    "    lang_adapter_config = str(lang_adapter_path) + \"/adapter_config.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if with_adapters_mlm:\n",
    "    # load the language adapter without head\n",
    "    task_mlm_load_as = 'mlm'\n",
    "    lang_adapter_name = model_qa.load_adapter(\n",
    "        load_lang_adapter,\n",
    "        config=lang_adapter_config,\n",
    "        load_as=task_mlm_load_as,\n",
    "        with_head=False\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### QA adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# folder for training outputs\n",
    "\n",
    "outputs = model_checkpoint.replace('/','-') + '_' + dataset_name\n",
    "\n",
    "if with_adapters_mlm:\n",
    "    outputs = outputs + '/' + 'mlm_' + str(task) + '_AdCompo' + str(adapter_composition) + '/'\n",
    "else:\n",
    "    outputs = outputs + '/' + str(task) + '/'\n",
    "\n",
    "outputs = outputs \\\n",
    "+ 'lr' + str(learning_rate) \\\n",
    "+ '_bs' + str(batch_size) \\\n",
    "+ '_GAS' + str(gradient_accumulation_steps) \\\n",
    "+ '_eps' + str(adam_epsilon) \\\n",
    "+ '_epochs' + str(num_train_epochs) \\\n",
    "+ '_patience' + str(early_stopping_patience) \\\n",
    "+ '_wamlm' + str(with_adapters_mlm) \\\n",
    "+ '_madx2' + str(madx2) \\\n",
    "+ '_houlsby_MHA_lastlayer' + str(houlsby_MHA_lastlayer) \\\n",
    "+ '_ds' + str(ds) \\\n",
    "+ '_fp16' + str(fp16) \\\n",
    "+ '_best' + str(load_best_model_at_end) \\\n",
    "+ '_metric' + str(metric_for_best_model) \\\n",
    "+ '_adapterconfig' + str(adapter_config_name)\n",
    "\n",
    "# path to outputs\n",
    "path_to_outputs = root/'outputs'/outputs\n",
    "\n",
    "# path to adapter\n",
    "adapters_folder = 'adapters-' + task\n",
    "path_to_save_adapter = path_to_outputs/adapters_folder\n",
    "\n",
    "# Config of the task adapter\n",
    "load_adapter = str(path_to_save_adapter)\n",
    "adapter_config = str(path_to_save_adapter) + \"/adapter_config.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the task adapter with head\n",
    "task_name = task\n",
    "model_qa.load_adapter(\n",
    "    load_adapter,\n",
    "    config=adapter_config,\n",
    "    load_as=task_name,\n",
    "    with_head = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Active adapters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the adapters to be used in every forward pass\n",
    "from transformers.adapters.composition import Stack\n",
    "if lang_adapter_name:\n",
    "    model_qa.active_adapters = Stack(task_mlm_load_as, task_name)\n",
    "else:\n",
    "    model_qa.set_active_adapters(task_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_qa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "nlp = pipeline(\"question-answering\", model=model_qa, tokenizer=tokenizer_qa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# source: https://pt.wikipedia.org/wiki/Pandemia_de_COVID-19\n",
    "context = r\"\"\"A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, \n",
    "uma doença respiratória causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). \n",
    "O vírus tem origem zoonótica e o primeiro caso conhecido da doença remonta a dezembro de 2019 em Wuhan, na China. \n",
    "Em 20 de janeiro de 2020, a Organização Mundial da Saúde (OMS) classificou o surto \n",
    "como Emergência de Saúde Pública de Âmbito Internacional e, em 11 de março de 2020, como pandemia. \n",
    "Em 18 de junho de 2021, 177 349 274 casos foram confirmados em 192 países e territórios, \n",
    "com 3 840 181 mortes atribuídas à doença, tornando-se uma das pandemias mais mortais da história.\n",
    "Os sintomas de COVID-19 são altamente variáveis, variando de nenhum a doenças com risco de morte. \n",
    "O vírus se espalha principalmente pelo ar quando as pessoas estão perto umas das outras. \n",
    "Ele deixa uma pessoa infectada quando ela respira, tosse, espirra ou fala e entra em outra pessoa pela boca, nariz ou olhos.\n",
    "Ele também pode se espalhar através de superfícies contaminadas. \n",
    "As pessoas permanecem contagiosas por até duas semanas e podem espalhar o vírus mesmo se forem assintomáticas.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "id": "t4_ezTchwKZl"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quando começou a pandemia de Covid-19 no mundo?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "id": "a_JgLD8fxdwn"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Qual é a data de início da pandemia Covid-19 em todo o mundo?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "id": "NecR00-wzRrn"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"A Covid-19 tem algo a ver com animais?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "id": "RcK4pn1hbLhL"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Onde foi descoberta a Covid-19?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "id": "7rFEmmsjzRrn"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quantos casos houve?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "id": "0v1TTQXDzRrn"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quantos mortes?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "id": "f78AggHLzRro"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quantos paises tiveram casos?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "id": "J0AGoVc_xhdo"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Quais são sintomas de COVID-19\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true,
    "id": "YSQnntVgcHHq"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "question = \"Como se espalha o vírus?\"\n",
    "\n",
    "result = nlp(question=question, context=context)\n",
    "\n",
    "print(f\"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# END"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "Language Modeling",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "adapter-transformers",
   "language": "python",
   "name": "adapter-transformers"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}