03_Network_Training.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MatchLab-Imperial/deep-learning-course/blob/master/03_Network_Training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91E7ozNTWvS9"
      },
      "source": [
        "# Network Training\n",
        "\n",
        "In this tutorial, we introduce different topics related to the training of deep neural networks, including data augmentation, optimization/regularization techniques, weight initializations and hyperparameters tuning.\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras"
      ],
      "metadata": {
        "id": "siAxFKDYH7rf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hVa-ZTYsXhNT"
      },
      "source": [
        "## Preliminary\n",
        "\n",
        "We load the MNIST dataset and define a model for digit classification by using our own function `get_data_model`. We also define a new function `plot_history`, which plots the training loss and, if required, the validation loss. We will use these functions throughout the tutorial."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2aGBJBVmXd8b"
      },
      "source": [
        "def plot_history(history, metric = None):\n",
        "  # Plots the loss history of training and validation (if existing)\n",
        "  # and a given metric\n",
        "  # Be careful because the axis ranges are automatically adapted\n",
        "  # which may not desirable to compare different runs.\n",
        "  # Also, in some cases you may want to combine several curves in one\n",
        "  # figure for easier comparison, which this function does not do.\n",
        "\n",
        "  if metric != None:\n",
        "    fig, axes = plt.subplots(2,1)\n",
        "    axes[0].plot(history.history[metric])\n",
        "    try:\n",
        "      axes[0].plot(history.history['val_'+metric])\n",
        "      axes[0].legend(['Train', 'Val'])\n",
        "    except:\n",
        "      pass\n",
        "    axes[0].set_title('{:s}'.format(metric))\n",
        "    axes[0].set_ylabel('{:s}'.format(metric))\n",
        "    axes[0].set_xlabel('Epoch')\n",
        "    fig.subplots_adjust(hspace=0.5)\n",
        "    axes[1].plot(history.history['loss'])\n",
        "    try:\n",
        "      axes[1].plot(history.history['val_loss'])\n",
        "      axes[1].legend(['Train', 'Val'])\n",
        "    except:\n",
        "      pass\n",
        "    axes[1].set_title('Model Loss')\n",
        "    axes[1].set_ylabel('Loss')\n",
        "    axes[1].set_xlabel('Epoch')\n",
        "  else:\n",
        "    plt.plot(history.history['loss'])\n",
        "    try:\n",
        "      plt.plot(history.history['val_loss'])\n",
        "      plt.legend(['Train', 'Val'])\n",
        "    except:\n",
        "      pass\n",
        "    plt.title('Model Loss')\n",
        "    plt.ylabel('Loss')\n",
        "    plt.xlabel('Epoch')\n",
        "\n",
        "def get_data_model(args = {}):\n",
        "  # Returns simple model, flattened MNIST data and categorical labels\n",
        "  num_classes=10\n",
        "\n",
        "  # the data, shuffled and split between train and test sets\n",
        "  (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "  x_train = x_train.reshape(x_train.shape[0], x_train.shape[1] * x_train.shape[2])\n",
        "  x_test = x_test.reshape(x_test.shape[0], x_test.shape[1] * x_test.shape[2])\n",
        "\n",
        "  x_train= x_train.astype('float32')\n",
        "  x_test= x_test.astype('float32')\n",
        "\n",
        "  x_train /= 255\n",
        "  x_test /= 255\n",
        "\n",
        "  y_train=keras.utils.to_categorical(y_train,num_classes)\n",
        "  y_test=keras.utils.to_categorical(y_test,num_classes)\n",
        "\n",
        "  # Load a simple non-convolutional model\n",
        "  model = keras.models.Sequential()\n",
        "  model.add(keras.layers.Input(shape=(784,)))\n",
        "  model.add(keras.layers.Dense(512, activation='relu', **args))\n",
        "  model.add(keras.layers.Dense(512, activation='relu', **args))\n",
        "  model.add(keras.layers.Dense(10, activation='softmax', **args))\n",
        "  return model, x_train, y_train, x_test, y_test\n",
        "\n",
        "model, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model.summary()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mRx68ymWLCT7"
      },
      "source": [
        "## Optimizers\n",
        "\n",
        "In a deep learning problem, the weights of a network are updated to minimize the loss in each optimization step. Optimizers define the specific method used to update in each timestep the weights of the network. For example, the vanilla Stochastic Gradient Descent (SGD) only uses the gradient of the weight with respect to the loss in the current timestep $t$, along with the learning rate, to update the weights. However, other optimizers use more sophisticate methods (e.g., keeping track of past gradient updates to use as momentum) to compute the weights' updates.\n",
        "\n",
        "Since there are optimizers more prone to get stuck in local minima, the choice of the optimizer can potentially affect both the final performance and the speed of convergence.  A nice visualization of such behavior is the following animation, where you can notice some methods, i.e., Adadelta (yellow), and Rmsprop (black) converge significantly faster than SGD (red), which gets stuck in a local minimum.\n",
        "\n",
        "![](http://ruder.io/content/images/2016/09/saddle_point_evaluation_optimizers.gif)\n",
        "\n",
        "The animation above is from [Sebastian Ruder's blog](http://ruder.io/optimizing-gradient-descent/), who wrote an interesting article showing the formulation and properties of the different optimizers. We are not going to analyze the properties of the different optimizers, but you can read the blog post if you are interested in how optimizers work. As a rule of thumb, Adam is usually easier to tune due to the adaptive learning rate, whereas SGD with momentum [has been shown](https://arxiv.org/pdf/1712.07628.pdf) to reach better results when tuned correctly.\n",
        "\n",
        "In this tutorial, we will follow a hands-on approach and will mainly focus on how to use them in Keras. The official documentation [Optimizers in Keras](https://keras.io/optimizers/) lists all the available optimizers. The optimizer we use is passed as an argument to the `compile` function after we build our model. An easy way to do so is passing the optimizer in the form of a string, which will use the default parameters.  For example, in the next piece of code we use Adam with the default Keras parameters as our optimizer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9od5lUWMWu5X"
      },
      "source": [
        "model, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model.compile(optimizer='adam',loss='categorical_crossentropy')\n",
        "model.fit(x_train,y_train,batch_size=100, epochs=10, verbose=1,validation_data=(x_test,y_test))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hGIIvNFPRZI3"
      },
      "source": [
        "However, it is quite usual to change some of the parameters of the optimizer, such as the learning rate, which controls the update rate of the weights, depending on the problem we are trying to solve. To do so in Keras, instead of passing a string to the `compile` function, we can pass an object with our optimizer with custom parameters. For example, in the following code, we use Adam optimizer with a different learning rate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QCMtC0nERXhi"
      },
      "source": [
        "optimizer = keras.optimizers.Adam(learning_rate=3e-4)\n",
        "model, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model.compile(loss='categorical_crossentropy', optimizer=optimizer)\n",
        "model.fit(x_train,y_train,batch_size=100, epochs=10, verbose=1,validation_data=(x_test,y_test))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-yDSecajTyc6"
      },
      "source": [
        "## Initializers\n",
        "\n",
        "\n",
        "Neural Networks have a large number of weights that need to be initialized properly. Weight initialization is a crucial step in tuning neural networks as different initial values can lead the model to reach different local minima. If your initial weights are too small, the network may have problems learning due to small/vanishing gradients. However, if they are too large, you may run into exploding gradients or you could saturate an activation in the network (e.g. a sigmoid). Hence, we need a proper initialization with not too large nor not too small weights. In [this link](https://www.deeplearning.ai/ai-notes/initialization/) you can find an explanation of the importance of weight initialization.  The weights are usually randomly initialized by different algorithms, e.g. Xavier, He_normal or LeCun, all of them theoretically motivated to have specific properties following different assumptions.\n",
        "\n",
        "In Keras, you can set the particular initialization strategy you want to use as an argument when declaring a layer. The available initialization methods in Keras are listed [here](https://keras.io/api/layers/initializers/).  The initializers are passed to the layers via the arguments `kernel_initializer` and `bias_initializer`. For instance, in the function `Dense()`, which defines the mapping $y= Ax + b$, you can initialize the kernel values ($A$ in the equation) with a normal distribution (by default the `stddev` is 0.05) and the bias ($b$ in the equation) with $0$ using the code below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A6lzySByXcMB"
      },
      "source": [
        "linear_layer = keras.layers.Dense(64, kernel_initializer='random_normal',\n",
        "                bias_initializer='zeros')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oF_EVmyooqYL"
      },
      "source": [
        "Now let's check the weights of the layers and see if they follow the distributions we set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YgHC0CSmowIK"
      },
      "source": [
        "input_x = tf.Variable(np.random.random((1, 64)))\n",
        "y = linear_layer(input_x)\n",
        "weights = linear_layer.get_weights()\n",
        "# Weights return an array with [kernel, bias]\n",
        "# Let's see the kernel weights\n",
        "print(weights[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kLDzNuqUp2M8"
      },
      "source": [
        "# Now let's check that the mean is 0 and stddev is 0.05\n",
        "print(weights[0].mean(), weights[0].std())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uohj6D4rqTfT"
      },
      "source": [
        "# Let's print the bias now\n",
        "print(weights[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "69NkDCipqbmg"
      },
      "source": [
        "By default the kernel weights are initialized as `'glorot_uniform'` and the bias to `'zeros'`. Glorot Uniform, which is also called Xavier Uniform initialization, was defined [here](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf). The Xavier Uniform initialization samples the weights from a uniform distribution, whose range depends on the number of input and output units. Another initializer quite used is the `he_normal`, which draws the weights from a truncated normal distribution. Usually, the initialization used by default in Keras works quite well and allows the model to train properly."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-8ittW2L1DI"
      },
      "source": [
        "## Loss\n",
        "\n",
        "Another important step in training deep neural networks is the choice of the loss function, which strictly depends on your problem. The full list of standard losses in Keras is available [here](https://keras.io/losses/). Although in past tutorials we already used some losses, we now introduce two typical losses, cross-entropy and mean squared error, for two standard problems in machine learning: classification and regression."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0T8aSdjCtSSO"
      },
      "source": [
        "### Classification\n",
        "\n",
        "For classification problems, the standard loss used is the cross-entropy loss. For the binary case, the formula is $\\mathcal{L} = y\\log(p) + (1-y)\\log(1-p)$, where $p$ is a probability value between $[0, 1]$.\n",
        "Typically a [Sigmoid activation](https://en.wikipedia.org/wiki/Sigmoid_function) is applied in the binary case to constrain the activations to force the output value to be between $[0, 1]$. In Keras, the presented binary classification loss is called `binary_crossentropy`, and it accepts as target a vector with an element in the range $[0, 1]$ (usually either $0$ or $1$) per input element. We show in the following code an example of binary classification. To do so, we transform MNIST into a binary classification problem by classifying if a number is between 0-4 or 5-9."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LUQhLNxr5VA7"
      },
      "source": [
        "model, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model.pop()\n",
        "model.add(keras.layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "# This three lines transform the problem in a binary one\n",
        "# We want to know if the number is bigger than 5 (label 1) or smaller (label 0)\n",
        "y_train = np.argmax(y_train, axis = 1)\n",
        "y_train[y_train < 5] = 0\n",
        "y_train[y_train >= 5] = 1\n",
        "\n",
        "\n",
        "model.compile(optimizer='adam',loss='binary_crossentropy', metrics=['binary_accuracy'])\n",
        "history = model.fit(x_train, y_train, epochs=3, batch_size=32,  validation_split=0.2, verbose = 0)\n",
        "plot_history(history, 'binary_accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6jvNHqg--XYy"
      },
      "source": [
        "When the number of classes is higher than 2, we use the cross-entropy loss, which is $\\mathcal{L} = -\\sum_i y_i\\log(p_i)$. The loss is called `categorical_crossentropy` in Keras and accepts one-hot encoded vectors. A one-hot encoded vector has dimensionality $C$, where $C$ is the number of classes. All of the elements are set to 0, minus the corresponding class $c$, which is set to 1. If we have a vector of labels with a scalar from $[0, C)$ per training example, we can transform it into a one-hot encoding form by using the function `to_categorical`.  \n",
        "\n",
        "Let's see an example using MNIST data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v_YBOLN77S1I"
      },
      "source": [
        "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "# The labels are an scalar from 0 to 9 per example\n",
        "one_hot_vector = keras.utils.to_categorical(y_train[:5])\n",
        "for i in range(5):\n",
        "  print('Label Value: {}. One-hot encoded vector: {}\\n'.format(y_train[i], one_hot_vector[i]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OR_lJgB6CfFd"
      },
      "source": [
        "The input for the cross-entropy needs to be a probability distribution, meaning that the sum of the elements should be $\\sum_i p_i = 1$. To make the output vector of our model add up to 1, we use the [Softmax activation](https://en.wikipedia.org/wiki/Softmax_function)  function as shown in the following code cell."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BVTAPlc_B99R"
      },
      "source": [
        "# We use the function get_data_model, which already applies to_categorical\n",
        "_, x_train, y_train, x_test, y_test = get_data_model()\n",
        "\n",
        "### Model defined with softmax\n",
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Input(shape=(784,)))\n",
        "model.add(keras.layers.Dense(512, activation='relu'))\n",
        "model.add(keras.layers.Dense(512, activation='relu'))\n",
        "model.add(keras.layers.Dense(10, activation='softmax'))\n",
        "\n",
        "model.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "history = model.fit(x_train, y_train, epochs=3, batch_size=32,  validation_split=0.2, verbose = 0)\n",
        "plot_history(history, 'categorical_accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J8VtF7C_UPgf"
      },
      "source": [
        "If you prefer to provide the label as an integer instead of doing the one-hot transformation, you can use the ``sparse_categorical_crossentropy`` loss in Keras."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iFuOCwNftQgH"
      },
      "source": [
        "### Regression\n",
        "In regression problems, it is common to use as losses the Mean Squared Error (MSE) or Mean Absolute Error (MAE). We now show a quick example using the Boston Housing dataset, which contains a set of input features and we aim to predict the price of the house."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Li20u1NjrwS5"
      },
      "source": [
        "(x_train, y_train), (x_test, y_test) = keras.datasets.boston_housing.load_data()\n",
        "print(y_train[:10])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0JO5_oiCyPzR"
      },
      "source": [
        "We see that the labels are float numbers (the price of the house), and our goal is to predict those float numbers. To do so, we need a network that has only one output. In the following code cells, we will train a network using either MAE or MSE. First, we use MAE."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RSJH6_Mfsjp-"
      },
      "source": [
        "model_mae = keras.models.Sequential()\n",
        "model_mae.add(keras.layers.Input(shape=(13,)))\n",
        "model_mae.add(keras.layers.Dense(100, activation='relu'))\n",
        "model_mae.add(keras.layers.Dense(1))\n",
        "model_mae.compile(optimizer='adam',loss='mean_absolute_error', metrics=['mean_absolute_error', 'mean_squared_error'])\n",
        "history = model_mae.fit(x_train, y_train, epochs=100, batch_size=32,  validation_split=0.2, verbose = 0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J2xZRys8zNsD"
      },
      "source": [
        "plot_history(history, 'mean_absolute_error')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TcJ-jplbVA4U"
      },
      "source": [
        "Now we use the MSE loss:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2XYKtV1GzMQg"
      },
      "source": [
        "model_mse = keras.models.Sequential()\n",
        "model_mse.add(keras.layers.Input(shape=(13,)))\n",
        "model_mse.add(keras.layers.Dense(100, activation='relu'))\n",
        "model_mse.add(keras.layers.Dense(1))\n",
        "model_mse.compile(optimizer='adam',loss='mean_squared_error', metrics=['mean_absolute_error', 'mean_squared_error'])\n",
        "history = model_mse.fit(x_train, y_train, epochs=100, batch_size=32,  validation_split=0.2, verbose = 0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SBZB_MDCw1Os"
      },
      "source": [
        "plot_history(history, 'mean_squared_error')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aMHRntQU0mrM"
      },
      "source": [
        "results = model_mae.evaluate(x_test, y_test)\n",
        "print('MAE trained model achieves MAE: {:.4f} and MSE: {:.4f}'.format(results[1], results[2]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "df2igCnI030F"
      },
      "source": [
        "results = model_mse.evaluate(x_test, y_test)\n",
        "print('MSE trained model achieves MAE: {:.4f} and MSE: {:.4f}'.format(results[1], results[2]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hWr2tc7l0IdG"
      },
      "source": [
        "The resulting plots show that training using MSE as loss achieves a better MSE and worse MAE in the test set compared to the model training with MAE loss, which is a sensible result."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_VeO_-ORHnn"
      },
      "source": [
        "A problem usually treated as a regression problem with deep learning approaches is depth estimation, where the depth of an image is estimated using the RGB information. Some other regression losses are also used in some settings (e.g. Huber loss or Reverse Huber loss, or the already introduced Mean Percentage Absolute Error), but MAE and MSE are widely used."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "miKutjvgtXuS"
      },
      "source": [
        "## Regularization\n",
        "As mentioned in the lecture, regularization is an effective tool to fight some of the problems we may have during training, such as vanishing/exploding gradients or overfitting to the training set. We now mention some of the usual ways to regularize our training process."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F-VrqhSbWJ9t"
      },
      "source": [
        "### Loss Regularizers\n",
        "\n",
        "[Regularizers](https://keras.io/regularizers/)  put some penalties to the optimization process. In practice, by penalizing large values, weights are constrained to be smaller in magnitude, which reduces the capacity of the model and can help us prevent overfitting.\n",
        "\n",
        "In Keras, regularization works on a per-layer basis. Hence, you can define a regularization function for each layer. You can specify three regularization parameters, each one related to a different type of penalty. If we define the output of a layer as $y=Wx+b$, then:\n",
        "\n",
        "*   `kernel_regularizer`: a penalty applied to the kernel weights. Larger regularization values result in smaller $W$.\n",
        "*   `bias_regularizer`: a penalty applied to the bias of the network. Larger regularization values result in smaller $b$.\n",
        "*   `activity_regularizer`: a penalty applied to the output of the layer. Larger regularization values result in smaller outputs $y=Wx+b$.\n",
        "\n",
        "Standard regularizers that can be applied are $l_1$, $l_2$ and $l_1+l_2$.\n",
        "In the example below, we check the difference in training and validation accuracy by varying the used regularization strategy.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zv4c80XD_Yzr"
      },
      "source": [
        "test_accuracy = []\n",
        "train_accuracy = []\n",
        "reg_values = [0.001, 0.0]\n",
        "\n",
        "for reg_val in reg_values:\n",
        "  print('Training with regularization value of {:f}'.format(reg_val))\n",
        "  args_dict = {'kernel_regularizer': keras.regularizers.l1(reg_val)}\n",
        "  model, x_train, y_train, x_test, y_test = get_data_model(args_dict)\n",
        "  model.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['accuracy'])\n",
        "  history = model.fit(x_train, y_train, epochs=10, batch_size=256, verbose=0)\n",
        "  train_accuracy.append(history.history['accuracy'][-1])\n",
        "  test_accuracy.append(model.evaluate(x_test, y_test)[-1])\n",
        "\n",
        "plt.figure()\n",
        "plt.plot(reg_values, train_accuracy)\n",
        "plt.plot(reg_values, test_accuracy)\n",
        "plt.legend(['Training acc.', 'Test acc.'])\n",
        "plt.title('Accuracy vs. Regularization value')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.xlabel('Regularization value')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "liYOHIYT8YlT"
      },
      "source": [
        "In this example, we show how a large regularization value dropped the final test accuracy, and also reduced the gap between training and test accuracy. The reason is that large regularization values put a constraint on the possible weights learnt by the network, which reduces the model capacity. Hence, you need to be careful when using regularization techniques as a large value can actually hurt the performance of the network."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FJ-J7tWRTQU7"
      },
      "source": [
        "### Dropout\n",
        "\n",
        "Dropout is another regularization technique that can be applied to the network. Dropout is a layer that, during training, randomly deactivates some of the neurons of the model, which restricts the model from relying too much on specific patterns. The dropout value, i.e., the probability of disabling any given feature element, is a parameter set when defining the layer. In the evaluation phase, dropout does not have any effect on the model. The dropout layer implemented on Keras also scales during training the non-zero elements by $1/(1-p_{drop})$ to maintain a similar data magnitude between training and evaluation.\n",
        "\n",
        "We now show an example using a dropout value is 0.3."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HEqZN0LyTv1F"
      },
      "source": [
        "prob_drop = 0.3\n",
        "drop = keras.layers.Dropout(prob_drop)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pqcb3TWCWe4e"
      },
      "source": [
        "x = np.random.random((1, 512))\n",
        "input_x = tf.Variable(x)\n",
        "y = drop(input_x, training=True)\n",
        "print('Input (10 elements)')\n",
        "print(x[0,0:10])\n",
        "print('Output (10 elements)')\n",
        "print(y[0,0:10])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iXhWs8c1Xw3h"
      },
      "source": [
        "We now check what percentage of elements have been set to 0, and what is the scaling value the other elements have."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8njS0-XRXVsc"
      },
      "source": [
        "# Calculate drop percentage\n",
        "print('Drop percentage, should be close to {:f}'.format(prob_drop))\n",
        "print(tf.reduce_sum(tf.cast(y == 0, tf.float32)) / tf.cast(y.shape[1], tf.float32))\n",
        "\n",
        "# Calculate scaling value\n",
        "print('Scaling value, should be {:f}'.format(1/(1-prob_drop)))\n",
        "# For non-zero elements\n",
        "nonzero_mask = tf.not_equal(y, 0)\n",
        "# Cast both tensors to the same type (float32)\n",
        "y_masked = tf.cast(tf.boolean_mask(y, nonzero_mask), tf.float32)\n",
        "x_masked = tf.cast(tf.boolean_mask(x, nonzero_mask), tf.float32)\n",
        "scaling_value = tf.reduce_sum(y_masked) / tf.reduce_sum(x_masked)\n",
        "print(scaling_value)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tZ7pTTajW130"
      },
      "source": [
        "Similarly to any of the past layers we have introduced, we can add a Dropout layer to any model using:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r9-0atwnW7tN"
      },
      "source": [
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Dropout(0.3))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BucSkwhQ_sJ-"
      },
      "source": [
        "Now we show the training curves of a model without dropout and a model with dropout. First, without dropout:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0ZRBM9YL9R69"
      },
      "source": [
        "model, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "history = model.fit(x_train, y_train, epochs=20, batch_size=32, validation_split=0.2, verbose=0)\n",
        "plot_history(history, 'categorical_accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UbpQAl82_2u9"
      },
      "source": [
        "Now we add Dropout."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lnJDBvYe899Y"
      },
      "source": [
        "_, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Input(shape=(784,)))\n",
        "model.add(keras.layers.Dense(512, activation='relu'))\n",
        "model.add(keras.layers.Dropout(0.3))\n",
        "model.add(keras.layers.Dense(512, activation='relu'))\n",
        "model.add(keras.layers.Dropout(0.3))\n",
        "model.add(keras.layers.Dense(10, activation='softmax'))\n",
        "model.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "history = model.fit(x_train, y_train, epochs=20, batch_size=32, validation_split=0.2, verbose=0)\n",
        "plot_history(history, 'categorical_accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f8wXhZPs-N-v"
      },
      "source": [
        "Notice how the validation loss stays closer to the training loss in the model with dropout compared to the original model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C98HL1rgVACR"
      },
      "source": [
        "### Batch Normalization\n",
        "\n",
        "Batch Normalization normalizes the features by centering them around zero and rescaling them to have variance 1, which leads to faster and more stable training. To do so, batch normalization uses the statistics of the batch. Specifically, a batch normalization layer computes the mean and the standard deviation per channel, i.e. given a feature map of dimensionality $B\\times H\\times W\\times C$ ($B$ is batch size, $H,W$ spatial dimensions and $C$ number of feature channels) the layer computes the mean $\\mu$ and standard deviation $\\sigma$ of the channels, where $\\mu$ and $\\sigma$ have dimensionality $1\\times 1\\times 1\\times C$. Then, the mean $\\mu$ and the standard deviation $\\sigma$ of the batch are expanded again to $B\\times H\\times W\\times C$ and used to standardize the input features to follow a distribution with mean 0 and variance 1. The layer output is then $y = \\gamma\\frac{x-\\mu}{\\sigma} + \\beta$, where $\\gamma$ and $\\beta$ are learnable parameters that give the network more capacity to express different distributions.\n",
        "The layer is defined in Keras by using:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mt2E2uTqWGBB"
      },
      "source": [
        "batch_norm = keras.layers.BatchNormalization(axis=-1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ax8OE0cbZA7X"
      },
      "source": [
        "Now we will generate a batch of 512x1 (a batch of 512 vectors of only 1 channel) using `np.random.random`, which is a uniform distribution under the $[0, 1)$ interval, resulting in mean 0.5 and variance 1/12. We will see how the batch normalization layer will scale the distribution to have mean 0 and variance 1.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8IS_u-HKY1WR"
      },
      "source": [
        "x = np.random.random((512, 10, 10, 1))\n",
        "input_x = tf.Variable(x)\n",
        "y = batch_norm(input_x, training=True)\n",
        "print('Input')\n",
        "print(x[:10, 0, 0, 0])\n",
        "print('Output')\n",
        "print(y[:10, 0, 0, 0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ILdlLzAtZ3sp"
      },
      "source": [
        "# Input mean should be ~0.5 and var ~1/12=0.0833\n",
        "print(x.mean(), x.var())\n",
        "# Output mean should be ~0 and var ~1\n",
        "print(tf.reduce_mean(y), tf.math.reduce_variance(y))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iDOne7XbrEhL"
      },
      "source": [
        "It is important to note that batch normalization changes behaviour in evaluation mode. During training, the layer tracks and updates the moving average of $\\mu$ and $\\sigma$ given all possible training batches. $\\mu$ and $\\sigma$ are then used to normalize the testing data without using the statistics from the testing batch.\n",
        "\n",
        "Generally, using Batch Normalization results in faster training and easier convergence. It is quite standard to place Batch Normalization layers just before the activation function, but we can also achieve good performance by placing it after the activation function. So you can see blocks of Conv+BN+Activation or Conv+Activation+BN in different networks.\n",
        "\n",
        "You can add your Batch Normalization layer as any other layer in your sequential model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8ck8kC2zXToa"
      },
      "source": [
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.BatchNormalization())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lfcOrzhaPpq0"
      },
      "source": [
        "Now we test the layer following a similar example as in the original [paper](http://proceedings.mlr.press/v37/ioffe15.pdf). We use a similar fully-connected model with sigmoid activations for MNIST."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h3aBywmT9__v"
      },
      "source": [
        "_, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Input(shape=(784,)))\n",
        "model.add(keras.layers.Dense(100, activation='sigmoid'))\n",
        "model.add(keras.layers.Dense(100, activation='sigmoid'))\n",
        "model.add(keras.layers.Dense(100, activation='sigmoid'))\n",
        "model.add(keras.layers.Dense(10, activation='softmax'))\n",
        "model.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "history = model.fit(x_train, y_train, epochs=10, batch_size=60, validation_split=0.2, verbose=1)\n",
        "plot_history(history, 'categorical_accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ChwJ6vtL-B6S"
      },
      "source": [
        "_, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Input(shape=(784,)))\n",
        "model.add(keras.layers.Dense(100, activation='sigmoid'))\n",
        "model.add(keras.layers.BatchNormalization())\n",
        "model.add(keras.layers.Dense(100, activation='sigmoid'))\n",
        "model.add(keras.layers.BatchNormalization())\n",
        "model.add(keras.layers.Dense(100, activation='sigmoid'))\n",
        "model.add(keras.layers.Dense(10, activation='softmax'))\n",
        "model.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "history = model.fit(x_train, y_train, epochs=10, batch_size=60, validation_split=0.2, verbose=1)\n",
        "plot_history(history, 'categorical_accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Gd-zJFYQrSs"
      },
      "source": [
        "In this example, we see how Batch Normalization increased the speed of convergence and helped the model achieve better validation accuracy. Batch Normalization is a key component for training deeper networks such as ResNet variants, which will be introduced in future tutorials. However, it is also sensitive to the batch size and the different statistics used in training (mini-batch statistics) and testing (running mean and std from training) can be problematic in some settings."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NuF3Y5DoKylT"
      },
      "source": [
        "## HyperParameters Tuning\n",
        "\n",
        "### Creating a Validation Set\n",
        "\n",
        "When training our model, we need to decide the value of several hyperparameters, what regularization techniques we employ, or the loss used to train the model, among others. To decide these values we should not use as guidance the performance in the test set, as it may lead to overfitting to that set, and in turn to an erroneous estimate of the performance of the model in non-seen data. Hence, we use what is called a validation set, which we use to tweak the hyperparameters.\n",
        "\n",
        "To define a validation split automatically in Keras, we can use two relevant arguments in the `fit` method: `validation_split` and `validation_data`. The argument passed to `validation_split` (0 by default) determines the ratio of the training set for validation purposes. For example,    \n",
        "```\n",
        "model.fit(x_train, y_train, ..., validation_split=0.2)\n",
        "```\n",
        "uses 20% of `x_train` as validation data.\n",
        "\n",
        "Using the `validation_split` method the validation data is taken from the last  N indices from your training data as explained in the [documentation](https://keras.io/api/models/model_training_apis/). If we want to fix the split used beforehand, we can use the `validation_data` argument, where we can pass directly the split of data we want to use as validation in the form of a tuple `(data, labels)`.\n",
        "\n",
        "In these tutorials we mainly use the official test sets of several standard datasets as the input to `validation_data`. The reason we use the given test sets as validation data for the tutorials is that is an easy way to make sure that we all work with the same split and report results using the same data. However, in a proper machine learning setup, your validation set should be separate from the test set, so you can tune the model/parameters on the validation set and then check the final performance in the test set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9DFkIWei6hse"
      },
      "source": [
        "### Early Stopping\n",
        "\n",
        "We now know how to define a validation set and how to use it during the training process. An important step now is how to retrieve the model with the highest validation performance during training. Usually, you keep the model with the highest validation accuracy as your final model, and then, ideally, you would test this final model using an independent test set.\n",
        "However, validation accuracy fluctuates between epochs, and may actually decrease after a few epochs due to overfitting. Hence, we would like to save the model with the best validation performance.\n",
        "\n",
        "We already mentioned how to do save the best performing model in the introductory Keras tutorial, but let's explain it again now that we know how to define a validation split. By default, Keras returns the last model after training $N$ epochs. We can instead choose to retrieve the model with the best validation performance by using the [`EarlyStopping`](https://keras.io/api/callbacks/early_stopping/) callback. EarlyStopping monitors a metric (e.g. `val_categorical_accuracy`) and after a few epochs where that metric has not improved (`patience` variable) the training process stops. Finally, if `restore_best_weights` is set to `True`, the weights from the best performing model in the defined metric are restored. Let's see an example."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QFfNridt72qT"
      },
      "source": [
        "early_stop = keras.callbacks.EarlyStopping(monitor='val_categorical_accuracy', min_delta=0, patience=4, verbose=0, mode='auto', baseline=None, restore_best_weights=True)\n",
        "_, x_train, y_train, x_test, y_test = get_data_model()\n",
        "\n",
        "model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "\n",
        "model.fit(x_train, y_train, epochs=20, batch_size=32, verbose=1, validation_data=(x_test, y_test), callbacks=[early_stop])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VtujLcUPAc7R"
      },
      "source": [
        "We see how the training process stopped after failing to improve `val_categorical_accuracy` during the number of epochs set by `patience. We now check that the performance of the saved model is the same as the obtained in the best epoch in terms of validation accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LPiaL47KAY-e"
      },
      "source": [
        "model.evaluate(x_test, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NMd2DVJCGmE7"
      },
      "source": [
        "### Learning Rate\n",
        "\n",
        "One of the most important parameters to tweak is the learning rate, which controls the update step performed during the backpropagation. Keras provides two callbacks that allow us to modify the learning rate during training. One is `LearningRateScheduler`, which allows us to define a rule to vary the learning rate depending on the epoch. For example, using the `lr_scheduler` function (found [here](https://stackoverflow.com/questions/39779710/setting-up-a-learningratescheduler-in-keras)), we can modify the loss function so that every 3 epochs is multiplied by 0.1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AC9in_Zhfnly"
      },
      "source": [
        "def lr_scheduler(epoch, lr):\n",
        "    decay_rate = 0.1\n",
        "    decay_step = 3\n",
        "    if epoch % decay_step == 0 and epoch:\n",
        "        return lr * decay_rate\n",
        "    return lr\n",
        "\n",
        "lrate = keras.callbacks.LearningRateScheduler(lr_scheduler)\n",
        "model, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "initial_lr = tf.keras.backend.get_value(model.optimizer.learning_rate)\n",
        "history = model.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.2, callbacks=[lrate])\n",
        "print('Initial Learning Rate: {:.4f}'.format(initial_lr))\n",
        "print('Final Learning Rate: {:.10f}'.format(tf.keras.backend.get_value(model.optimizer.learning_rate)))\n",
        "plot_history(history, 'categorical_accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jAF6VK2gkdRU"
      },
      "source": [
        "Let's plot the progress of the learning rate in each epoch to check how the learning rate is decreased every three epochs as we defined in `lr_scheduler`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jtugFeDVj8HD"
      },
      "source": [
        "learning_rate = history.history['learning_rate']\n",
        "plt.plot(range(0, len(learning_rate)), learning_rate)\n",
        "plt.ylabel('Learning Rate')\n",
        "plt.xlabel('Epochs')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XAPrOzm0iDTs"
      },
      "source": [
        "Another callback provided is `ReduceLROnPlateau`, which reduces the learning rate whenever a given metric has stopped improving. There are 5 important arguments:\n",
        "\n",
        " * `monitor`: we specify the metric we want to track\n",
        " * `patience`: number of epochs without improvement before reducing lr\n",
        " * `factor`: the new learning rate will be `new_lr = lr * factor`\n",
        " * `min_lr`: sets the minimum lr\n",
        " * `min_delta`: margin to define when the metric has stopped improving\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "os7b7Oh0TO7m"
      },
      "source": [
        "reduce_lr = keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.2,\n",
        "                              patience=1, min_lr=0.00001, min_delta = 0.01)\n",
        "model, x_train, y_train, x_test, y_test = get_data_model()\n",
        "model.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "initial_lr = tf.keras.backend.get_value(model.optimizer.learning_rate)\n",
        "history = model.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.2, callbacks=[reduce_lr])\n",
        "print('Initial Learning Rate: {:.4f}'.format(initial_lr))\n",
        "print('Final Learning Rate: {:.10f}'.format(tf.keras.backend.get_value(model.optimizer.learning_rate)))\n",
        "plot_history(history, 'categorical_accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FX9W-7yNoyvL"
      },
      "source": [
        "Again, we check how the learning rate has changed. You can check that the learning has indeed decreased when the `val_loss` has not improved by more than 0.01 until it reached the `min_lr` value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BVNFkE-qnKVk"
      },
      "source": [
        "learning_rate = history.history['learning_rate']\n",
        "plt.plot(range(0, len(learning_rate)), learning_rate)\n",
        "plt.ylabel('Learning Rate')\n",
        "plt.xlabel('Epochs')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xm0oVnMQWLZt"
      },
      "source": [
        "## Data augmentation\n",
        "Deep learning models are data-hungry, tend to overfit with small training sets and its performance benefit from large training sets. A way to synthetically create more data is using data augmentation. Data augmentation aims to modify the training examples by applying transformations to the input data. Now, we will show some examples of data augmentation for images.\n",
        "\n",
        "### Images\n",
        "Data augmentation techniques such as rotation, scaling or cropping are usually applied in deep learning pipelines for vision applications. The idea is to take as input an image, apply a transformation to it, and then use it for training.\n",
        "\n",
        "Keras allows us to perform data augmentation using ImageDataGenerator, which you may see in some other examples online, however by using the preprocessing module we can perform the augmentation directly on the GPU, making it quite faster in the Colabs notebook. The available transformations are listed [here](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing). The preprocessing module can be imported by doing:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vftBzsolQcL"
      },
      "source": [
        "We first define a function `plot_data_augmentation` that we will use to show some augmented examples.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yTLpcR43IHpX"
      },
      "source": [
        "def plot_data_augmentation(augmentation = None):\n",
        "  (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "  # A generator that allows us to use flow to output some examples\n",
        "  data_gen = keras.preprocessing.image.ImageDataGenerator()\n",
        "  for X_batch, y_batch in data_gen.flow(np.expand_dims(x_train, -1), y_train, batch_size=5, shuffle=False):\n",
        "    if augmentation is not None:\n",
        "        X_batch = augmentation(X_batch).numpy()\n",
        "    for i in range(0, 5):\n",
        "      plt.subplot(150 + 1 + i)\n",
        "      plt.imshow(X_batch[i, :].reshape(28, 28), cmap=plt.get_cmap('gray'))\n",
        "    # show the plot\n",
        "    plt.show()\n",
        "    break"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OoJYjjapiISI"
      },
      "source": [
        "We will now visualize some of the transformations available in this preprocessing module. First, we plot some images without any transformations applied for comparison."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MyoZHyOTheyU"
      },
      "source": [
        "plot_data_augmentation()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eFXEC5EejxWG"
      },
      "source": [
        "### Rotation\n",
        "A standard transformation is to rotate the image. We can do so by using `preprocessing.RandomRotation` ([documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/RandomRotation)) with a factor that controls the range of the random rotations. For example, `0.15`rotates the image randomly between `[-0.15*2pi, 0.15*2pi]`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hjgfZ1z3j46X"
      },
      "source": [
        "# We first define the transformation we want to apply\n",
        "plot_data_augmentation(keras.layers.RandomRotation(0.15))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Schj43v4ioTY"
      },
      "source": [
        "### Shift\n",
        "\n",
        "We can define a maximum range of both horizontal (`width_factor`) and vertical (`height_factor`) shift using `preprocessing.RandomTranslation`([documentation\u001b](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/RandomTranslation))."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dbzq69FfjLd9"
      },
      "source": [
        "data_augmentation = keras.layers.RandomTranslation(\n",
        "    height_factor=0.3, width_factor=0.3, fill_mode='reflect',\n",
        "    interpolation='bilinear'\n",
        ")\n",
        "plot_data_augmentation(data_augmentation)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7uhYKLIPI1LA"
      },
      "source": [
        "### Zooming\n",
        "Zooming into the image can be done with `preprocessing.RandomZoom` ([documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/RandomZoom)). You can define a height factor, that limits the range of height zooming, and a width factor, which similarly limits the range of width zooming."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NGvZzyL0HQjF"
      },
      "source": [
        "augmentation_gen = keras.layers.RandomZoom(height_factor=0.4)\n",
        "plot_data_augmentation(augmentation_gen)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "845GvrICI9IX"
      },
      "source": [
        "### Flip\n",
        "\n",
        "We can define either horizontal flip (`horizontal`), vertical (`vertical`) or both (`horizontal_and_vertical`) using `preprocessing.RandomFlip` (\u001b[documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/RandomFlip)).  Horizontal flip is one of the most usual augmentations in images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m6emZPbsHXOV"
      },
      "source": [
        "augmentation_gen = keras.layers.RandomFlip('horizontal')\n",
        "plot_data_augmentation(augmentation_gen)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sb3fcBi8Inwy"
      },
      "source": [
        "### Data Normalization\n",
        "Data normalization is usually applied in deep learning pipelines. A typical data normalization applied is data standardization, where we transform the input data to have `mean=0` and `std=1`. When you have multiple features coming from different distributions and different scales, normalization makes all of the features have a similar distribution, which avoids biasing the model towards using the features with larger values.\n",
        "\n",
        "In images, the range of values is already in the same scale of [0-255], but normalizing the range of values to [0-1] can help us avoid starting training in the saturation regions of some activations, such as a sigmoid, where values of large magnitude will fall into the flat region of the activation. Data centering (mean=0) can also useful in some settings because the weight initializers used (e.g. Xavier) are derived following a ``mean=0`` assumption about the data distribution, but probably a model trained with [0-1] will usually converge too.\n",
        "\n",
        "To perform normalization, we can also use the preprocessing module. In the following example we can see how using `preprocessing.Normalization()` makes our data have `mean=0` and `standard_deviation=1`. Contrary to the other transformations we showed, before normalizing the data we need to use `adapt(train_data)` to compute the mean and standard deviation of the whole training set. The mean and standard deviation computed will then be saved and applied to standardize the individual images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fGOQ0F-_Lrgr"
      },
      "source": [
        "normalization = keras.layers.Normalization()\n",
        "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "x_train = x_train.reshape(-1, 28, 28, 1)\n",
        "normalization.adapt(x_train)\n",
        "# We show the mean and std of the training set\n",
        "# before and after normalization\n",
        "print('Normalized mean/std', normalization(x_train).numpy().mean(), normalization(x_train).numpy().std())\n",
        "print('Original mean/std', x_train.mean(), x_train.std())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_5bCiXoiiQFI"
      },
      "source": [
        "When using any kind of data normalization, it is important to note that we need to apply the same transformation to the test data for our model to work properly. In the given example we can use the statistics computed from the training data to standardize the test data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QRBKYvgXid3_"
      },
      "source": [
        "print('Normalized mean/std', normalization(x_test).numpy().mean(), normalization(x_test).numpy().std())\n",
        "print('Original mean/std', x_test.mean(), x_test.std())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GOE33AOLJdPK"
      },
      "source": [
        "### Combining transformations\n",
        "We have shown how some of the individual data transformations used in data augmentation work. Now we want to show how we can combine all the transformations and how to integrate them in a training pipeline. To do so, we can form a Sequential layer with all of the transformations and add it to the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xqb15Ga0EsC9"
      },
      "source": [
        "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "\n",
        "# We normalize the data first\n",
        "norm_layer = keras.layers.Normalization()\n",
        "x_train = x_train.reshape(-1,28,28,1)\n",
        "x_test = x_test.reshape(-1,28,28,1)\n",
        "norm_layer.adapt(x_train)\n",
        "x_train = norm_layer(x_train)\n",
        "x_test = norm_layer(x_test)\n",
        "\n",
        "# We define our data augmentation pipeline\n",
        "data_augmentation = keras.models.Sequential([\n",
        "    keras.layers.RandomFlip(\"vertical\"),\n",
        "    keras.layers.RandomRotation(0.6)\n",
        "])\n",
        "y_train_class = keras.utils.to_categorical(y_train, 10)\n",
        "y_test_class = keras.utils.to_categorical(y_test, 10)\n",
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Input(shape=(28,28,1)))\n",
        "model.add(data_augmentation)\n",
        "model.add(keras.layers.Flatten())\n",
        "model.add(keras.layers.Dense(512, activation='relu'))\n",
        "model.add(keras.layers.Dense(512, activation='relu'))\n",
        "model.add(keras.layers.Dense(10, activation='softmax'))\n",
        "model.compile(optimizer='adam',loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "model.fit(x_train, y_train_class,  epochs=3, validation_data=(x_test,y_test_class))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ipjcr_8Gi_0S"
      },
      "source": [
        "We see how the data augmentation process is integrated into the model as it was an extra initial layer. At inference time, most of the preprocessing layers are not active."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "65Ynz4OOee5_"
      },
      "source": [
        "In these examples, we showed practical examples of data augmentation for images. However, other modalities, such as text or audio can also benefit from data augmentation.\n",
        "\n",
        "Keep in mind that applying a data augmentation strategy that is too aggressive may actually harm the performance in the test set, e.g., applying extreme zooms not likely to be present in the dataset or strong contrast shifts."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nUpwc1fHxF4N"
      },
      "source": [
        "# Coursework\n",
        "\n",
        "### Task 1: Tuning a Classification Model\n",
        "In a machine learning problem, and especially when using a deep learning approach, finding the right set of hyperparameters, the right data augmentation strategy, or a good regularization method can make the difference between a model that performs poorly and a model with great accuracy.\n",
        "\n",
        "For this exercise, you will be training a CNN to perform classification in CIFAR-10 (we use the official test set, which is why the variables are called `x_test` and `y_test`, as our validation set) and will analyze the impact of some of the most important elements presented in this tutorial.\n",
        "\n",
        "Use the CNN we give in the code below, along with the given optimizer and number of training epochs as the default setting. Only modify the given CNN architecture to add Dropout or Batch Normalization layers when explicitly stated. Use 40 epochs to plot all of your curves. However, you can train for more epochs to find your best validation performance if your network has not finished training in those 40 epochs.\n",
        "\n",
        "**Report:**\n",
        "*  First, train the given default model without any data augmentation. Then define two data augmentation strategies (one more aggressive than the other) and train the model with data augmentation. Clearly state the two augmentation strategies you apply (i.e., the specific transformations). Discuss the training and validation loss curves for the two data augmentation strategies along with the original run without data augmentation. Attach in the appendix those training and validation curves. Report in a table the best validation accuracy obtained for the three runs (no data augmentation, data augmentation 1, data augmentation 2).\n",
        "\n",
        "*  Without using any data augmentation, analyze the effect of using Dropout in the model. Carry out the same analysis for Batch Normalization. Finally, combine both. Report in the same table as in the data augmentation task the best validation accuracy for each of the three settings (baseline + Dropout, baseline + Batch Normalization, baseline + Batch Normalization + Dropout). The performance will vary depending on where the Dropout layers and Batch Normalization layers, so state clearly where you added the layers, and what rate you used for the Dropout layers. Discuss the results.\n",
        "\n",
        "* Using the default model/hyperparameters and no data augmentation, report the best validation accuracy when using `zeros` for the kernel initialization. Report the performance in the same table as in the dropout/batch normalization/data augmentation tasks. Discuss the results that you obtained.\n",
        "\n",
        "*  Using the default model and no data augmentation, change the optimizer to SGD and train it with learning rates of `3e-3`, `1e-3` and `3e-4`. Report in a figure the training and validation loss for the three learning rate values and discuss the figure."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g3kbDP402H_u",
        "collapsed": true
      },
      "source": [
        "# load the data\n",
        "(X_train, y_train), (X_test, y_test) = keras.datasets.cifar10.load_data()\n",
        "\n",
        "print('Image shape: {0}'.format(X_train.shape[1:]))\n",
        "print('Total number of training samples: {0}'.format(X_train.shape[0]))\n",
        "print('Total number of validation samples: {0}'.format(X_test.shape[0]))\n",
        "\n",
        "X_train = X_train.reshape(-1,32,32,3)\n",
        "\n",
        "## Normalization block\n",
        "norm_layer = keras.layers.Normalization()\n",
        "norm_layer.adapt(X_train)\n",
        "X_train_n = norm_layer(X_train)\n",
        "X_test_n = norm_layer(X_test)\n",
        "\n",
        "# You can modify the data_augmentation variable below to add your\n",
        "# data augmentation pipeline.\n",
        "# By default we do not apply any augmentation (RandomZoom(0) is equivalent\n",
        "# to not performing any augmentation)\n",
        "data_augmentation = keras.models.Sequential(\n",
        "    [\n",
        "        keras.layers.RandomZoom(0)\n",
        "    ]\n",
        ")\n",
        "# We will use glorot_uniform as a initialization by default\n",
        "initialization = 'glorot_uniform'\n",
        "# Use the architecture given below, only modify it to add Dropout/BatchNorm\n",
        "model = keras.models.Sequential()\n",
        "model.add(keras.layers.Input(shape=(32,32,3)))\n",
        "model.add(data_augmentation)\n",
        "model.add(keras.layers.Conv2D(32, (3, 3), padding='same', kernel_initializer=initialization))\n",
        "model.add(keras.layers.Activation('relu'))\n",
        "model.add(keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'))\n",
        "\n",
        "model.add(keras.layers.Conv2D(64, (3, 3), padding='same', kernel_initializer=initialization))\n",
        "model.add(keras.layers.Activation('relu'))\n",
        "model.add(keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'))\n",
        "\n",
        "model.add(keras.layers.Conv2D(128, (3, 3), padding='same', kernel_initializer=initialization))\n",
        "model.add(keras.layers.Activation('relu'))\n",
        "model.add(keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='same'))\n",
        "\n",
        "model.add(keras.layers.Conv2D(256, (3, 3), padding='same', kernel_initializer=initialization))\n",
        "model.add(keras.layers.Activation('relu'))\n",
        "# As we use global average pooling, we don't need to use Flatten\n",
        "model.add(keras.layers.GlobalAveragePooling2D())\n",
        "model.add(keras.layers.Dense(10, kernel_initializer=initialization))\n",
        "model.add(keras.layers.Activation('softmax'))\n",
        "\n",
        "\n",
        "Y_train_class = keras.utils.to_categorical(y_train, 10)\n",
        "Y_test_class = keras.utils.to_categorical(y_test, 10)\n",
        "# By default use Adam with lr=3e-4. Change it to SGD when asked to\n",
        "opt = keras.optimizers.Adam(learning_rate=3e-4)\n",
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer=opt,\n",
        "              metrics=['accuracy'])\n",
        "# Use 40 epochs as default value to plot your curves\n",
        "history = model.fit(X_train_n, Y_train_class, epochs=40, validation_data=(X_test_n,Y_test_class))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}