04_Common_CNN_architectures.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "private_outputs": true,
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MatchLab-Imperial/deep-learning-course/blob/master/04_Common_CNN_architectures.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IhXUenVgBmjf"
      },
      "source": [
        "# Introduction to common Convolutional Neural Networks Architectures\n",
        "In this tutorial, we will learn about well-known CNN architectures in the field of computer vision and how to implement them in Keras. Moreover, we will show that some of the standard architectures are already available into the Keras framework. Finally, we will see how we can import and use them in our code.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "auwuthntar8h"
      },
      "source": [
        "# CNNs in Image Classification\n",
        "We start this tutorial by presenting the networks that have been widely used for image classification.  One of the most famous problems among the computer vision community was the annual software contest run by the ImageNet project, **ImageNet Large Scale Visual Recognition Challenge (ILSVRC)**. This contest evaluates how well an algorithm does in the tasks of object category classification and detection. The challenge provides hundreds of object categories and millions of images. The resolution of those images is also bigger than those on MNIST or CIFAR datasets.\n",
        "\n",
        "As ImageNet is a massive dataset, in the first part of the tutorial we use CIFAR10 dataset, which is directly available on Keras.\n",
        "\n",
        "As we already have seen, CIFAR10 dataset contains ten different classes of 32x32 images. We will show how to resize the data to be able to use some of the following architectures, as not all of the CNN architectures accepts arbitrary input sizes. Remember that they were designed for ImageNet, and were trained using 224x224 crops.\n",
        "\n",
        "\n",
        "\n",
        "Let's start by loading CIFAR10:"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import time\n",
        "\n",
        "import cv2\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import scipy\n",
        "from tensorflow.keras import applications, callbacks, datasets, layers, models, optimizers, utils"
      ],
      "metadata": {
        "id": "N4ULQ5FvHEpj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i87in95Fj0zF"
      },
      "source": [
        "# Load the data, shuffled and split between train and test sets\n",
        "(x_train, y_train), (x_test, y_test) = datasets.cifar10.load_data()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gaQUxiIirq3r"
      },
      "source": [
        "In order to use the original models, we need to resize the CIFAR10 32x32 images into 224x224 images. However, as the whole dataset of CIFAR10 resized ot 224x224 would not fit in our RAM, we need to create a preprocessing step before the first layer. That would take the 32x32 images and, just before going into the architecture, resize them into our desired shape. To do so, we use the `Lambda` layer in Keras, which wraps any expression (in this case `resize_images`) in a layer. We can see how this function looks like for a single batch of 32 images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1FihA5egr6qs"
      },
      "source": [
        "model = models.Sequential([\n",
        "    layers.Input(shape=(32, 32, 3)),\n",
        "    layers.Resizing(224, 224, interpolation='bilinear'),\n",
        "])\n",
        "\n",
        "x_train_resized = model.predict(x_train[:32, ...])\n",
        "\n",
        "# Visualization purposes\n",
        "x_train_resized = np.asarray(x_train_resized, dtype = int)\n",
        "\n",
        "# Let's visualize some examples\n",
        "N=2\n",
        "start_val = 0 # pick an element for the code to plot the following N**2 values\n",
        "fig, axes = plt.subplots(N,N)\n",
        "items = list(range(0, 100))\n",
        "for row in range(N):\n",
        "  for col in range(N):\n",
        "    idx = start_val+row+N*col\n",
        "    axes[row,col].imshow(x_train_resized[idx], cmap='gray')\n",
        "    fig.subplots_adjust(hspace=0.5)\n",
        "    y_target = int(y_train[idx].item())\n",
        "    target = str(items[y_target])\n",
        "    axes[row,col].set_title(target)\n",
        "    axes[row,col].set_xticks([])\n",
        "    axes[row,col].set_yticks([])\n",
        "plt.show()\n",
        "\n",
        "print('\\nShape BEFORE Lambda function: {}'.format(x_train[:32, ...].shape))\n",
        "print('Shape AFTER Lambda function: {}'.format(x_train_resized.shape))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qw0FyvNCizwx"
      },
      "source": [
        "Let's now get the dataset ready:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eGI2w-ssjAmZ"
      },
      "source": [
        "num_classes = 10\n",
        "y_train = utils.to_categorical(y_train,num_classes)\n",
        "y_test = utils.to_categorical(y_test,num_classes)\n",
        "\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255.\n",
        "x_test /= 255.\n",
        "\n",
        "print('Original training data shape: {}'.format(x_train.shape))\n",
        "print('Training label shape: {}'.format(y_train.shape))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qeChk5z14kUQ"
      },
      "source": [
        "We have the dataset and the Lambda function ready to use to for any of the standard classification architectures. In the next section, we will show how we can code a common architecture and train it with our preprocessed dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zwo-H2oSfQ0s"
      },
      "source": [
        "## AlexNet\n",
        "\n",
        "[AlexNet](https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf) made a huge impact in 2012 in the ImageNet challenge when it reduced the top-5 error (i.e. the correct class is not among the top-5 predictions) from 26% to 15.3%. The second place was close to 26.2%, and it was not a CNN based system. AlexNet shares a lot in common with its predecessor architecture, [LeNet](http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf) network by Yann LeCun et al. When LeNet came out, the computational complexity of the networks was an important constraint. Nowadays, with GPUs being every day more and more powerful, the computational complexity is more plausible to deal with that it was years before. Therefore, the authors in AlexNet decided to make the architecture bigger by using more convolutional layers and more filters.\n",
        "\n",
        "The architecture proposed in their paper is as follows:\n",
        "\n",
        "![texto alternativo](https://i.ibb.co/QFG561f/0-x-POQ3bt-Z9r-QO23-LK.png)\n",
        "\n",
        "The network consisted of convolutional layers with kernels of size 11x11, 5x5 and 3x3. The architecture uses layers with strides, max poolings, dropouts, ReLU activation functions, and three dense layers at the very end. In their original work, the network was split into two streams, as seen in the figure above. That was because GPU constraints, and hence, the authors needed to train it on two separate GPUs. We found a great implementation in [Rizwan's blog](https://engmrk.com/alexnet-implementation-using-keras/), where it presented an AlexNet model without the split concept."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7U4lWS9WyBP0"
      },
      "source": [
        "# AlexNet\n",
        "model = models.Sequential()\n",
        "\n",
        "# Input + Resize\n",
        "model.add(layers.Input(shape=(32, 32, 3)))\n",
        "model.add(layers.Resizing(224, 224, interpolation='bilinear'))\n",
        "\n",
        "# 1st Convolutional Layer\n",
        "model.add(layers.Conv2D(filters=96, kernel_size=(11,11), strides=(4,4), padding='valid'))\n",
        "model.add(layers.Activation('relu'))\n",
        "model.add(layers.MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))\n",
        "\n",
        "# 2nd Convolutional Layer\n",
        "model.add(layers.Conv2D(filters=256, kernel_size=(11,11), strides=(1,1), padding='valid'))\n",
        "model.add(layers.Activation('relu'))\n",
        "model.add(layers.MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))\n",
        "\n",
        "# 3rd Convolutional Layer\n",
        "model.add(layers.Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))\n",
        "model.add(layers.Activation('relu'))\n",
        "\n",
        "# 4th Convolutional Layer\n",
        "model.add(layers.Conv2D(filters=384, kernel_size=(3,3), strides=(1,1), padding='valid'))\n",
        "model.add(layers.Activation('relu'))\n",
        "\n",
        "# 5th Convolutional Layer\n",
        "model.add(layers.Conv2D(filters=256, kernel_size=(3,3), strides=(1,1), padding='valid'))\n",
        "model.add(layers.Activation('relu'))\n",
        "model.add(layers.MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='valid'))\n",
        "\n",
        "# Passing it to a Fully Connected layer\n",
        "model.add(layers.Flatten())\n",
        "\n",
        "# 1st Fully Connected Layer\n",
        "model.add(layers.Dense(4096))\n",
        "model.add(layers.Activation('relu'))\n",
        "# Add Dropout to prevent overfitting\n",
        "model.add(layers.Dropout(0.4))\n",
        "\n",
        "# 2nd Fully Connected Layer\n",
        "model.add(layers.Dense(4096))\n",
        "model.add(layers.Activation('relu'))\n",
        "# Add Dropout\n",
        "model.add(layers.Dropout(0.4))\n",
        "\n",
        "# 3rd Fully Connected Layer\n",
        "model.add(layers.Dense(10))\n",
        "model.add(layers.Activation('softmax'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xSfj3T_T0ZZ9"
      },
      "source": [
        "Keep in mind that we can use the original code because we have added a line of code to resize the CIFAR images from 32x32 to 224x224. We can now train our AlexNet model on CIFAR."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BsOYPKwB0k52"
      },
      "source": [
        "model.compile(\n",
        "    loss = 'categorical_crossentropy',\n",
        "    optimizer = optimizers.Adam(learning_rate=1e-4),\n",
        "    metrics = ['accuracy'],\n",
        ")\n",
        "\n",
        "model.fit(x_train, y_train, batch_size=32, epochs=3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u-LaLUyEGvfM"
      },
      "source": [
        "Let's test it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q0XC8B0fGxSu"
      },
      "source": [
        "score = model.evaluate(x_test, y_test)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7zyJoeM76R6W"
      },
      "source": [
        "As it could be observed, training this kind of architectures in big datasets is time-consuming. We might need to train for several hours to start getting state-of-the-art results. That is why Keras saves us time and offers us several pre-trained models on the ImageNet dataset.\n",
        "\n",
        "Those models can be used directly for image prediction, image classification, feature extraction, or fine-tuning, among others, without the need of spending long hours of training. Isn't that great!?\n",
        "\n",
        "Even if the task or the dataset is different, using the pre-trained weights as initialisation for the training process provides usually faster convergence (and sometimes better results) than random initialisation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wks8tkdjdtgV"
      },
      "source": [
        "# Loading Pre-trained Models in Keras\n",
        "\n",
        "As mentioned, Keras contains many models that have been trained in the ImageNet dataset. Those models were originally released to participate in the ILSVRC competition. However, now they are available alongside their pre-trained weights and used in multiple different tasks. Using pre-trained weights has become a common practice to initialise networks. You can learn more about it [here](https://keras.io/applications/).\n",
        "\n",
        "We will show how to initialise different models and load their pre-trained weights in ImageNet. Afterwards, we will show how to modify the last layer to classify only the ten classes we have in CIFAR10."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWbK1gE99o7K"
      },
      "source": [
        "## VGGNet\n",
        "\n",
        "VGGNet is an architecture presented by Simonyan and Zisserman in 2014. VGGNet is similar to previous AlexNet network, however, it only contains 3x3 convolutional kernels and many more filters. It has become really popular since researchers have found that it can be used to extract powerful feature representations from an image. Therefore, extracted features have proven that they are useful for many other computer vision domains and not only image classification. For instance, they can be used in feature representation, style transfer, or image captioning. You can check the [Oxford VGG paper](https://arxiv.org/pdf/1409.1556.pdf) for further details.\n",
        "\n",
        "Next image shows the standard VGG architecture ([source](https://www.cs.toronto.edu/~frossard/post/vgg16/)):\n",
        "\n",
        "![](https://www.cs.toronto.edu/~frossard/post/vgg16/vgg16.png)\n",
        "\n",
        "The training time of the architecture is massive since it has more than 130 million parameters. Thankfully, we can find it already pre-trained in Keras. In the original work, authors proposed two versions of VGG; VGG16, and VGG19. Keras provides the two proposed versions, where the difference lies in the number of layers within the network.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CPpzPdYAREBb"
      },
      "source": [
        "model = applications.VGG16(input_shape=(224, 224, 3))\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t3ZLsBmIRpqQ"
      },
      "source": [
        "Now, if we want to load weights from ImageNet we only need to initialise the model as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B29erHYWRp9Y"
      },
      "source": [
        "model = applications.VGG16(weights='imagenet', input_shape=(224, 224, 3))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z6TMSj9r3Dkr"
      },
      "source": [
        "The argument `weights` indicates to Keras whether we need to use the pre-trained weights from ImageNet.\n",
        "\n",
        "If we want to use the original VGG model in CIFAR10 classification, we need to change some things. The first thing to modify is the last fully connected layer. In the original VGG, the last layer is designed to classify among the 1,000 classes provided in ImageNet. In CIFAR10, as we already know, we only have ten classes. Therefore, we switch the last dense layer in the model to be able to perform classification on CIFAR10. Moreover, a common practise when using pre-trained networks is to freeze the model and only train the new dense layer. This strategy is called Transfer Learning. You can read more about it and its difference to Fine-tuning in this great [blog](https://towardsdatascience.com/cnn-transfer-learning-fine-tuning-9f3e7c5806b2). Updating only the weights of the last layer provides several advantages; faster training time, good performance, and sometimes helps the networks to not overfit in small datasets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P1c26De_3ihg"
      },
      "source": [
        "#Load the VGG\n",
        "model = applications.VGG16(weights='imagenet', include_top=True, input_shape=(224, 224, 3))\n",
        "\n",
        "# Freeze all the layers\n",
        "for layer in model.layers[:]:\n",
        "    layer.trainable = False\n",
        "\n",
        "output = model.get_layer('fc2').output\n",
        "output = layers.Dense(units=10, activation='softmax')(output)\n",
        "model = models.Model(model.input, output)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I3gLnH3p4Wl6"
      },
      "source": [
        "We can confirm that the original dense layer is not there anymore, and instead, we have a new fully-connected layer with only ten activations. Also, as we have frozen the weights of the model, we see that the trainable parameters of the network belong to the new dense layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MX2lK4YB4leZ"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1p1qVDFC-KoP"
      },
      "source": [
        "As we did before, we add the lambda function to resize the CIFAR10 images before the VGG model. Note that VGG has several dense layers at the end and, therefore, if we would like to use original CIFAR10 image resolutions (32x32), we should modify those dense layers according to the new resolutions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "twaI6se2-I3g"
      },
      "source": [
        "model = models.Sequential([\n",
        "    # Input layer\n",
        "    layers.Input(shape=(32, 32, 3)),\n",
        "    # Resizing layer (using modern resizing)\n",
        "    layers.Resizing(224, 224, interpolation='bilinear'),\n",
        "    # VGG16 with parameters frozen except the final dense layer\n",
        "    model,\n",
        "])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JdIufD1rDyig"
      },
      "source": [
        "Let's now see how we can modify the VGG architecture to use any input image shape. First of all, if we want to use any other image resolution than 224x224, we need to set the argument `include_top` to `False`. This argument will remove all dense layers at the end of the architecture. Remember that the dense layer comes after a flatten layer (see the model summary above) and, therefore, the size of the flatten vector is fixed. If we modify the input image size, the flatten vector would inevitably change as well. Hence, we need to redefine all dense layers within the model every time we switch the input size."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vDjgj0Q4Ds1-"
      },
      "source": [
        "# Set include_top to False to avoid loading any dense layer\n",
        "model = applications.VGG16(include_top=False, input_shape=(32,32,3), weights='imagenet')\n",
        "\n",
        "# Freeze all the layers\n",
        "for layer in model.layers[:]:\n",
        "    layer.trainable = False\n",
        "\n",
        "# Add the same dense layers as in VGG16\n",
        "model = models.Sequential([\n",
        "    # VGG16 just loaded\n",
        "    model,\n",
        "    # Flatten layer\n",
        "    layers.Flatten(),\n",
        "    # Dense layers\n",
        "    layers.Dense(units=4096, activation='relu'),\n",
        "    layers.Dense(units=4096, activation='relu'),\n",
        "    layers.Dense(units=10, activation='softmax'),\n",
        "])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Izc03Dfc7lL_"
      },
      "source": [
        "We have added the same number of dense layers as in original VGG. We are ready to train the newly added layers as we did in the previous step."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8IRy6UZf70rU"
      },
      "source": [
        "model.compile(optimizer=optimizers.Adam(learning_rate=1e-4),loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "model.fit(x_train, y_train, epochs=3, batch_size=32)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xybWrAjbBcyh"
      },
      "source": [
        "Let's evaluate it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nW75Tn5b8LcL"
      },
      "source": [
        "score = model.evaluate(x_test, y_test)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rSaBmsu7IKD9"
      },
      "source": [
        "## GoogLeNet / Inception v1\n",
        "\n",
        "Google presented [GoogLeNet](https://arxiv.org/pdf/1409.4842.pdf) in 2014, the same year that VGGNet was introduced.  GoogLeNet won the ImageNet competition achieving a top-5 error rate of 6.67%, almost betting the human error rate (5%).\n",
        "\n",
        "GoogLeNet introduced many ideas that helped the development of current state-of-the-art architectures. Instead of stacking more and more CNN layers, GoogLeNet introduced what was called inception modules:\n",
        "\n",
        "![](https://i.ibb.co/JKqptrj/Googlenet-inception.png)\n",
        "\n",
        "Those modules apply convolutions with 3 different sizes of kernels (1x1, 3x3 and 5x5) at the same level. The idea behind those modules is based on the premise that significant information can be presented in images at different scales. Therefore, by using a multi-scale approach, they are more likely to capture meaningful information. Moreover, they claimed that designing a wider architecture, instead of a deeper, helps the gradients to navigate throughout the entire network.\n",
        "\n",
        "![](https://cdn-images-1.medium.com/max/1600/1*uW81y16b-ptBDV8SIT1beQ.png)\n",
        "\n",
        "The architecture consisted of 9 inception modules stacked linearly, containing 22 deep CNN layers. However, in this network, the total number of parameters was reduced to 4 million! Authors also included two auxiliary classifiers in the middle part of the architecture to avoid the vanishing gradient problem.\n",
        "\n",
        "You can find a Keras implementation [here](https://gist.github.com/joelouismarino/a2ede9ab3928f999575423b9887abd14)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aGCXlBgVIxP6"
      },
      "source": [
        "Even though they won the image classification contest, original Inception v1 was a complex and heavily engineered architecture. Authors introduced many tricks to push its performance, both in terms of speed and accuracy. In the literature, we can find a constant evolution of the architecture which led to the creation of several improved versions. A list and explanation of all inception networks can be found in [this blog](https://towardsdatascience.com/a-simple-guide-to-the-versions-of-the-inception-network-7fc52b863202).  \n",
        "\n",
        "In Keras, the Inception model v3 architecture is available, together with its pre-trained weights on ImageNet. It can be initialised as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lw6quKrOpGjT"
      },
      "source": [
        "model = applications.InceptionV3(include_top=True, weights='imagenet', classes=1000)\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rxmx-iMnp4Wz"
      },
      "source": [
        "Every network has its requirements. For instance, Inception v3 needs a minimum image input size of 75x75 due to its design. We can use the lambda function defined previously in the tutorial and see how the network performs in CIFAR10."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jkGHeRxZrwWL"
      },
      "source": [
        "model = applications.InceptionV3(include_top=False, input_shape=(75, 75, 3), weights='imagenet')\n",
        "\n",
        "# Freeze all the layers\n",
        "for layer in model.layers[:]:\n",
        "    layer.trainable = False\n",
        "\n",
        "# Add additional layers to classify on CIFAR10\n",
        "model = models.Sequential([\n",
        "    # Input + Resize\n",
        "    layers.Input(batch_shape=(None, 32, 32, 3)),\n",
        "    layers.Resizing(75, 75, interpolation='bilinear'),\n",
        "    # Inception v3\n",
        "    model,\n",
        "    # Dense layers\n",
        "    layers.GlobalAveragePooling2D(),\n",
        "    layers.Dense(10, activation='softmax'),\n",
        "])\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fZwcxCiGxRQ-"
      },
      "source": [
        "Let's train Inception v3:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n_kaiLcGxE8V"
      },
      "source": [
        "model.compile(optimizer=optimizers.Adam(learning_rate=1e-4),loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "model.fit(x_train, y_train, epochs=3, batch_size=32)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9PbyNXbWxT_U"
      },
      "source": [
        "And check its accuracy:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dRB4GcuJxbp_"
      },
      "source": [
        "score = model.evaluate(x_test, y_test)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCV0NEx-F-0D"
      },
      "source": [
        "## ResNet\n",
        "\n",
        "In 2015, the winner of the ImageNet challenge was the Residual Neural Network ([ResNet](https://arxiv.org/pdf/1512.03385.pdf)) architecture. Similar to Inception, ResNet is built by micro-architectures modules, called residual blocks. Those blocks introduced skip connections, which allowed to train huge architectures (152 layers) while still having lower complexity than VGGNet. Thus, residual connections allowed authors to design deeper architectures since the gradient could backpropagate easier through the skip connections. Next image is from [Das' blog](https://medium.com/@sidereal/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5) and shows ResNet architecture with its skip connections:\n",
        "\n",
        "![texto alternativo](https://cdn-images-1.medium.com/max/800/0*pkrso8DZa0m6IAcJ.png)\n",
        "\n",
        "Keras offers the implementation of ResNet50 pre-trained on ImageNet. In addition to ResNet50, deeper versions of it, ResNet101 or ResNet152, are widely used nowadays. The main difference lies in the number of layers, e.g., ResNet50 is a 50 layer Residual Network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L3AJ5oF0dz64"
      },
      "source": [
        "model = applications.ResNet50(include_top=False, input_shape=(32,32,3), weights='imagenet')\n",
        "\n",
        "# Freeze all the layers\n",
        "for layer in model.layers[:]:\n",
        "    layer.trainable = False\n",
        "\n",
        "# Add additional layers to classify on CIFAR10\n",
        "model = models.Sequential([\n",
        "    # ResNet50\n",
        "    model,\n",
        "    # Dense layers\n",
        "    layers.GlobalAveragePooling2D(),\n",
        "    layers.Dense(10, activation='softmax'),\n",
        "])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yC5S2IONeaWd"
      },
      "source": [
        "Let's check the performance:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XRJ80Qi2ef0U"
      },
      "source": [
        "model.compile(optimizer=optimizers.Adam(learning_rate=1e-4),loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "model.fit(x_train, y_train, epochs=3, batch_size=32)\n",
        "\n",
        "score = model.evaluate(x_test, y_test)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZlupkHZ11D6z"
      },
      "source": [
        "Note that all previous networks may need a different number of epochs until they converge. Even though we use pre-trained weights, fully training a model can be quite time-consuming."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ILGzCzQlD26"
      },
      "source": [
        "## Xception, DenseNet, InceptionResNet ...\n",
        "\n",
        "Keras offers many more network implementations trained on ImageNet. You can check the [documentation](https://keras.io/applications/) to see all of them. Besides, the documentation reports their accuracies, number of parameters and depth of each architecture."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s_mCvUoEZYmd"
      },
      "source": [
        "# CNNs for Other Tasks\n",
        "\n",
        "Besides the success of CNNs in classification problems, CNNs have been used in many other domains. To name a few: image inpainting, facial recognition, semantic segmentation, image captioning, depth prediction, among many others. In this section, we are going to talk about two different problems in computer vision: image denoising and object detection/classification. We are aiming to provide examples of the wide applicability CNNs architectures have.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q5AKwf_Hl-Gv"
      },
      "source": [
        "## Image Denoising with UNet\n",
        "\n",
        "[UNet](https://arxiv.org/pdf/1505.04597.pdf) was presented in 2015 for biomedical image segmentation, but since then, it has been employed in countless different tasks, such as semantic segmentation, image colorisation, image stylisation... The UNet architectural idea is similar to a regular encoder-decoder layout but with skip connections. The first part of the network encodes the image and reduces it to a map of smaller size, forcing it to capture more global information. Then, the next part is in charge of decoding the image from that reduced feature map.\n",
        "\n",
        "UNet uses a similar idea of the skip connections used in ResNets. In addition to helping to backpropagate the gradients, the authors used the skip connections to facilitate the flow of information between encoder and decoder. We will talk more about UNet in future tutorials, but for now, let's see how we can code an UNet model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZoYzGXVRnP5k"
      },
      "source": [
        "inputs = layers.Input((32,32,3))\n",
        "conv1 = layers.Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)\n",
        "pool1 = layers.MaxPooling2D(pool_size=(2, 2))(conv1)\n",
        "conv2 = layers.Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)\n",
        "pool2 = layers.MaxPooling2D(pool_size=(2, 2))(conv2)\n",
        "conv3 = layers.Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)\n",
        "pool3 = layers.MaxPooling2D(pool_size=(2, 2))(conv3)\n",
        "conv4 = layers.Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool3)\n",
        "drop4 = layers.Dropout(0.5)(conv4)\n",
        "pool4 = layers.MaxPooling2D(pool_size=(2, 2))(drop4)\n",
        "\n",
        "conv5 = layers.Conv2D(1024, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool4)\n",
        "drop5 = layers.Dropout(0.5)(conv5)\n",
        "\n",
        "## Now the decoder starts\n",
        "\n",
        "up6 = layers.Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(layers.UpSampling2D(size = (2,2))(drop5))\n",
        "merge6 = layers.concatenate([drop4,up6], axis = 3)\n",
        "conv6 = layers.Conv2D(512, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge6)\n",
        "\n",
        "up7 = layers.Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(layers.UpSampling2D(size = (2,2))(conv6))\n",
        "merge7 = layers.concatenate([conv3,up7], axis = 3)\n",
        "conv7 = layers.Conv2D(256, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)\n",
        "\n",
        "up8 = layers.Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(layers.UpSampling2D(size = (2,2))(conv7))\n",
        "merge8 = layers.concatenate([conv2,up8], axis = 3)\n",
        "conv8 = layers.Conv2D(128, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge8)\n",
        "\n",
        "up9 = layers.Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(layers.UpSampling2D(size = (2,2))(conv8))\n",
        "merge9 = layers.concatenate([conv1,up9], axis = 3)\n",
        "conv9 = layers.Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge9)\n",
        "conv10 = layers.Conv2D(3, 3,  padding = 'same')(conv9)\n",
        "\n",
        "model = models.Model(inputs = inputs, outputs = conv10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BfMCDfIJAdYY"
      },
      "source": [
        "Now, we are going to create a toy example of image denoising.\n",
        "\n",
        "We have CIFAR10 images already loaded from previous experiments. We add some Gaussian noise (you can play with the scale of the noise) to those CIFAR10 images and then train the UNet to remove that added noise from the images. To do so, we train with the noisy image as input and the clean image as the target. We use the Mean Absolute Error as the loss function to optimise the architecture."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OarMCDhS7UjV"
      },
      "source": [
        "# Add noise to input images\n",
        "x_train_noise = x_train + np.random.normal(size = x_train.shape, scale = 0.1)\n",
        "x_test_noise = x_test + np.random.normal(size = x_test.shape, scale = 0.1)\n",
        "\n",
        "model.compile(optimizer='Adam', loss = 'mean_absolute_error', metrics = ['mae'])\n",
        "model.fit(x_train_noise, x_train, batch_size=32, epochs=3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3df46sxxpf4S"
      },
      "source": [
        "pred = model.predict(x_test_noise)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CULZTfRdBk3C"
      },
      "source": [
        "We can visualise the noisy image next to the output of the UNet. The output images are a little bit blurred, but the noise we added has definitely been reduced."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O0H2dGQ0p98I"
      },
      "source": [
        "N=5\n",
        "start_val = 0# pick an element for the code to plot the following N**2 values\n",
        "fig, axes = plt.subplots(N,N)\n",
        "for row in range(N):\n",
        "  for col in range(N):\n",
        "    idx = start_val+row+N*col\n",
        "    im = np.concatenate((np.clip(x_test_noise[idx], 0, 1), np.clip(pred[idx], 0, 1)), 1)\n",
        "    axes[row,col].imshow(im)\n",
        "    axes[row,col].set_xticks([])\n",
        "    axes[row,col].set_yticks([])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TT_TfhCsfXsH"
      },
      "source": [
        "## Object Detection with RetinaNet\n",
        "\n",
        "We now will show an object detection example. Object detection differs from image classification in that in an image classification setting we give a label to the whole image, whereas in an object detection setting we give a label and bounding box per each of the classes that are present.\n",
        "\n",
        "Top detector models include Faster R-CNN, YOLO (yes, it is called YOLO, standing for You Only Look Once; this is the unusual [resume](https://pjreddie.com/static/Redmon%20Resume.pdf) of the YOLO developer) or SSD (Single Shot Detector). Another detector is RetinaNet, which we use to give an example of object detection using the implementation based on [torchvision](https://github.com/pytorch/vision/)."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torchvision\n",
        "from torchvision.models.detection import retinanet_resnet50_fpn_v2, RetinaNet_ResNet50_FPN_V2_Weights\n",
        "from torchvision.transforms import functional as F\n",
        "from PIL import Image\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import requests\n",
        "from io import BytesIO\n",
        "import time\n",
        "\n",
        "\n",
        "SCORE_THRESHOLD = 0.45\n",
        "\n",
        "# Load the model\n",
        "weights = RetinaNet_ResNet50_FPN_V2_Weights.DEFAULT\n",
        "model = retinanet_resnet50_fpn_v2(weights=weights, box_score_thresh=SCORE_THRESHOLD)\n",
        "model.eval()\n",
        "\n",
        "# Move to GPU if available\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "model = model.to(device)\n",
        "\n",
        "# Get the COCO category names\n",
        "COCO_INSTANCE_CATEGORY_NAMES = weights.meta[\"categories\"]\n",
        "\n",
        "def draw_boxes(image, boxes, labels, scores):\n",
        "    \"\"\"Draw bounding boxes on the image\"\"\"\n",
        "    import cv2\n",
        "    image = np.array(image)\n",
        "    for box, label, score in zip(boxes, labels, scores):\n",
        "        # Convert box coordinates to integers\n",
        "        box = box.cpu().numpy().astype(int)\n",
        "\n",
        "        # Generate random color for this class\n",
        "        color = tuple(np.random.randint(0, 255, 3).tolist())\n",
        "\n",
        "        # Draw box\n",
        "        cv2.rectangle(image,\n",
        "                     (box[0], box[1]),\n",
        "                     (box[2], box[3]),\n",
        "                     color,\n",
        "                     2)\n",
        "\n",
        "        # Draw label\n",
        "        caption = f\"{COCO_INSTANCE_CATEGORY_NAMES[label]} {score:.3f}\"\n",
        "        cv2.putText(image,\n",
        "                   caption,\n",
        "                   (box[0], box[1] - 10),\n",
        "                   cv2.FONT_HERSHEY_SIMPLEX,\n",
        "                   0.5,\n",
        "                   color,\n",
        "                   2)\n",
        "\n",
        "    return image\n",
        "\n",
        "def detection_url(url):\n",
        "    # Download and load the image\n",
        "    response = requests.get(url)\n",
        "    image = Image.open(BytesIO(response.content)).convert('RGB')\n",
        "\n",
        "    # Prepare image for the model\n",
        "    transform = weights.transforms()\n",
        "    img_tensor = transform(image).to(device)\n",
        "\n",
        "    # Perform detection\n",
        "    start = time.time()\n",
        "    with torch.no_grad():\n",
        "        prediction = model([img_tensor])\n",
        "    print(\"processing time: \", time.time() - start)\n",
        "\n",
        "    # Get predictions\n",
        "    boxes = prediction[0]['boxes']\n",
        "    scores = prediction[0]['scores']\n",
        "    labels = prediction[0]['labels']\n",
        "\n",
        "    # Filter predictions based on score\n",
        "    mask = scores > SCORE_THRESHOLD\n",
        "    boxes = boxes[mask]\n",
        "    labels = labels[mask]\n",
        "    scores = scores[mask]\n",
        "\n",
        "    # Draw boxes on image\n",
        "    result_image = draw_boxes(image, boxes, labels, scores)\n",
        "\n",
        "    # Display result\n",
        "    plt.figure(figsize=(15, 15))\n",
        "    plt.axis('off')\n",
        "    plt.imshow(result_image)\n",
        "    plt.show()"
      ],
      "metadata": {
        "id": "iJJ3BDxxmubK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Di9tA9Zqij06"
      },
      "source": [
        "Now, we can use any *URL* pointing to an image (some *URLs* may fail) as input, and we will get an output with the detections out of the 80 classes in Microsoft COCO. If you want to check the classes available, you can print the variable labels_to_names.\n",
        "\n",
        "As an example, we, of course, input an image full of cats. The labels are a bit blurry, but you can see all of the cats have the correct bounding box around them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NBIL89k7ix_S"
      },
      "source": [
        "url = 'https://st3.depositphotos.com/1032808/12984/i/950/depositphotos_129842732-stock-photo-group-of-various-breeds-cats.jpg'\n",
        "detection_url(url)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aVYyfx_VDrPh"
      },
      "source": [
        "You can even try with images that contain different classes, and see how RetinaNet deals with multi-class images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Kw9YL1GDn8d"
      },
      "source": [
        "url = 'https://cdn.mos.cms.futurecdn.net/hbKifQWBTcdhTEw8zsJWnF-1200-80.jpg'\n",
        "detection_url(url)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "urw5Sugu1F7r"
      },
      "source": [
        "# **Coursework**\n",
        "\n",
        "## Classification on Tiny-ImageNet\n",
        "\n",
        "In this task, we are going to explore different models to do classification on 64x64 Tiny-ImageNet. Tiny-ImageNet is a smaller version of ImageNet (as the name indicates), containing \"only\" 200 classes. Each class has 500 images. The test set contains 10,000 images. All images are 64x64 RGB images.\n",
        "\n",
        "In the Network Training notebook, we explained how to define a validation set, and now we will put that into practice. Hence, as we now have a bigger dataset, we are going to use the standard split of training, validation, and test data. Therefore, you will check the performance of the network in the validation set while training your network. Hence, your decisions need to be based on validation performance. Once you have obtained your best model using the training and validation data, you need to report the performance on the test set. Please try to no overfit to the test data, as in other problems it may not be available to you.\n",
        "\n",
        "In this exercise, you are asked to train VGG models with different strategies. Optionally, you are asked to use any other architecture of your choice to do classification in Tiny-ImageNet.\n",
        "\n",
        "Run the following script to get the data.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QmPJ2vKG5BB_"
      },
      "source": [
        "# Download TinyImageNet\n",
        "! git clone https://github.com/seshuad/IMagenet\n",
        "\n",
        "def get_id_dictionary():\n",
        "    id_dict = {}\n",
        "    for i, line in enumerate(open('IMagenet/tiny-imagenet-200/wnids.txt', 'r')):\n",
        "        id_dict[line.replace('\\n', '')] = i\n",
        "    return id_dict\n",
        "\n",
        "def get_class_to_id_dict():\n",
        "    id_dict = get_id_dictionary()\n",
        "    all_classes = {}\n",
        "    result = {}\n",
        "    for i, line in enumerate(open('IMagenet/tiny-imagenet-200/words.txt', 'r')):\n",
        "        n_id, word = line.split('\\t')[:2]\n",
        "        all_classes[n_id] = word\n",
        "    for key, value in id_dict.items():\n",
        "        result[value] = (key, all_classes[key])\n",
        "\n",
        "    return result\n",
        "\n",
        "def get_data(id_dict):\n",
        "    print('starting loading data')\n",
        "    train_data, val_data, test_data = [], [], []\n",
        "    train_labels, val_labels, test_labels = [], [], []\n",
        "    t = time.time()\n",
        "    for key, value in id_dict.items():\n",
        "        train_data += [cv2.imread('IMagenet/tiny-imagenet-200/train/{}/images/{}_{}.JPEG'.format(key, key, str(i))) for i in range(450)]\n",
        "\n",
        "        train_labels_ = np.array([[0]*200]*450)\n",
        "        train_labels_[:, value] = 1\n",
        "        train_labels += train_labels_.tolist()\n",
        "\n",
        "        val_data += [cv2.imread('IMagenet/tiny-imagenet-200/train/{}/images/{}_{}.JPEG'.format(key, key, str(i))) for i in range(450, 500)]\n",
        "\n",
        "        val_labels_ = np.array([[0]*200]*50)\n",
        "        val_labels_[:, value] = 1\n",
        "        val_labels += val_labels_.tolist()\n",
        "\n",
        "    for line in open('IMagenet/tiny-imagenet-200/val/val_annotations.txt'):\n",
        "        img_name, class_id = line.split('\\t')[:2]\n",
        "        test_data.append(cv2.imread('IMagenet/tiny-imagenet-200/val/images/{}'.format(img_name)))\n",
        "\n",
        "        test_labels_ = np.array([[0]*200])\n",
        "        test_labels_[0, id_dict[class_id]] = 1\n",
        "        test_labels += test_labels_.tolist()\n",
        "\n",
        "    print('finished loading data, in {} seconds'.format(time.time() - t))\n",
        "\n",
        "    return np.array(train_data), np.array(train_labels), np.array(val_data), np.array(val_labels), np.array(test_data), np.array(test_labels)\n",
        "\n",
        "def shuffle_data(train_data, train_labels, val_data, val_labels):\n",
        "    # This function shuffles separately the train set and the\n",
        "    # validation set\n",
        "    size = len(train_data)\n",
        "    train_idx = np.arange(size)\n",
        "    np.random.shuffle(train_idx)\n",
        "\n",
        "    size = len(val_data)\n",
        "    val_idx = np.arange(size)\n",
        "    np.random.shuffle(val_idx)\n",
        "\n",
        "    return train_data[train_idx], train_labels[train_idx], val_data[val_idx], val_labels[val_idx]\n",
        "\n",
        "train_data, train_labels, val_data, val_labels, test_data, test_labels = get_data(get_id_dictionary())\n",
        "train_data, train_labels, val_data, val_labels = shuffle_data(train_data, train_labels, val_data, val_labels)\n",
        "\n",
        "# Let's visualize some examples\n",
        "N=3\n",
        "start_val = 0 # pick an element for the code to plot the following N**2 values\n",
        "fig, axes = plt.subplots(N,N)\n",
        "for row in range(N):\n",
        "  for col in range(N):\n",
        "    idx = start_val+row+N*col\n",
        "    tmp = cv2.cvtColor(train_data[idx],cv2.COLOR_BGR2RGB)\n",
        "    axes[row,col].imshow(tmp, cmap='gray')\n",
        "    fig.subplots_adjust(hspace=0.5)\n",
        "    axes[row,col].set_xticks([])\n",
        "    axes[row,col].set_yticks([])\n",
        "\n",
        "train_data = train_data.astype('float32') / 255.\n",
        "val_data = val_data.astype('float32') / 255.\n",
        "test_data = test_data.astype('float32') / 255.\n",
        "\n",
        "mean = np.mean(train_data,axis=(0,1,2,3))\n",
        "std = np.std(train_data, axis=(0, 1, 2, 3))\n",
        "train_data = (train_data-mean)/(std+1e-7)\n",
        "val_data = (val_data-mean)/(std+1e-7)\n",
        "test_data = (test_data-mean)/(std+1e-7)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kXLjtvJEAF_r"
      },
      "source": [
        "**Report**:\n",
        "*   In a plot, please report the training and validation accuracy curves for the following models:\n",
        "\n",
        "> *   VGG16 trained from scratch.\n",
        "\n",
        "> *   Transfer Learning VGG16: load pre-trained ImageNet weights and only train the newly added dense layers.  To do so, freeze all layers and only train the dense layers you have modified in the model.\n",
        "\n",
        "> *   Fine-tuning VGG16: load pre-trained ImageNet weights and train the whole architecture.\n",
        "\n",
        "*   Discuss the previous figure in the main text. And report in a table the test accuracy and the training and inference times of previous VGG16 experiments. Training times are computed per epoch and you can find them displayed in the .fit() method information. Report either the total training time, or the number of epochs and training time per epoch. Inference times are computed per image, and we give you the code below to obtain them.\n",
        "\n",
        "*   Now that we are familiar with loading and using models in Keras, you can use any model of your choice to classify Tiny-ImageNet. You can take the model directly from Keras, any GitHub repository, or do the code yourself. You need to report your results in the previous table and compare your model of choice with previous VGG16 networks.\n",
        "\n",
        "Note that training/inference time will depend on which GPU you are using. Report the time results in the same instance, or at least when using the same GPU. Report also the GPU you were using to compute those inference times."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ae4MXFHF6Lsk"
      },
      "source": [
        "# You might want to import additional modules here.\n",
        "\n",
        "# Define you model here.\n",
        "# model = . . .\n",
        "\n",
        "early_stopping = callbacks.EarlyStopping(monitor='val_categorical_accuracy', patience=4, restore_best_weights=True)\n",
        "\n",
        "model.compile(optimizer=optimizers.Adam(learning_rate=1e-4),loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "model.fit(train_data, train_labels, epochs=20, batch_size=128, validation_data=(val_data, val_labels), callbacks=[early_stopping])\n",
        "\n",
        "start_time = time.time()\n",
        "score = model.evaluate(test_data, test_labels)\n",
        "time_elapsed = time.time() - start_time\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])\n",
        "print('Average inference time per image: {:.4f} (ms)'.format(1000*time_elapsed/len(test_data)))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}