examples/colab/action_recognition_with_tf_hub.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x8Q7Un821X1A"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Hub Authors.\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "1W4rIAFt1Ui3"
      },
      "outputs": [],
      "source": [
        "# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cDq0CIKc1vO_"
      },
      "source": [
        "# Action Recognition with an Inflated 3D CNN\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/hub/tutorials/action_recognition_with_tf_hub\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/action_recognition_with_tf_hub.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/hub/blob/master/examples/colab/action_recognition_with_tf_hub.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/hub/examples/colab/action_recognition_with_tf_hub.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "h6W3FhoP3TxC"
      },
      "source": [
        "This Colab demonstrates use of action recognition from video data using the\n",
        "[tfhub.dev/deepmind/i3d-kinetics-400/1](https://tfhub.dev/deepmind/i3d-kinetics-400/1) module.\n",
        "\n",
        "The underlying model is described in the paper \"[Quo Vadis, Action Recognition? A New\n",
        "Model and the Kinetics Dataset](https://arxiv.org/abs/1705.07750)\" by Joao\n",
        "Carreira and Andrew Zisserman. The paper was posted on arXiv in May 2017, and\n",
        "was published as a CVPR 2017 conference paper.\n",
        "The source code is publicly available on\n",
        "[github](https://github.com/deepmind/kinetics-i3d).\n",
        "\n",
        "\"Quo Vadis\" introduced a new architecture for video classification, the Inflated\n",
        "3D Convnet or I3D. This architecture achieved state-of-the-art results on the UCF101\n",
        "and HMDB51 datasets from fine-tuning these models. I3D models pre-trained on Kinetics\n",
        "also placed first in the CVPR 2017 [Charades challenge](http://vuchallenge.org/charades.html).\n",
        "\n",
        "The original module was trained on the [kinetics-400 dateset](https://deepmind.com/research/open-source/open-source-datasets/kinetics/)\n",
        "and knows about 400 different actions.\n",
        "Labels for these actions can be found in the\n",
        "[label map file](https://github.com/deepmind/kinetics-i3d/blob/master/data/label_map.txt).\n",
        "\n",
        "In this Colab we will use it recognize activites in videos from a UCF101 dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "R_0xc2jyNGRp"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "mOHMWsFnITdi"
      },
      "outputs": [],
      "source": [
        "!pip install -q imageio\n",
        "!pip install -q opencv-python\n",
        "!pip install -q git+https://github.com/tensorflow/docs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "USf0UvkYIlKo"
      },
      "outputs": [],
      "source": [
        "#@title Import the necessary modules\n",
        "# TensorFlow and TF-Hub modules.\n",
        "from absl import logging\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from tensorflow_docs.vis import embed\n",
        "\n",
        "logging.set_verbosity(logging.ERROR)\n",
        "\n",
        "# Some modules to help with reading the UCF101 dataset.\n",
        "import random\n",
        "import re\n",
        "import os\n",
        "import tempfile\n",
        "import ssl\n",
        "import cv2\n",
        "import numpy as np\n",
        "\n",
        "# Some modules to display an animation using imageio.\n",
        "import imageio\n",
        "from IPython import display\n",
        "\n",
        "from urllib import request  # requires python3"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "IuMMS3TGdws7"
      },
      "outputs": [],
      "source": [
        "#@title Helper functions for the UCF101 dataset\n",
        "\n",
        "# Utilities to fetch videos from UCF101 dataset\n",
        "UCF_ROOT = \"https://www.crcv.ucf.edu/THUMOS14/UCF101/UCF101/\"\n",
        "_VIDEO_LIST = None\n",
        "_CACHE_DIR = tempfile.mkdtemp()\n",
        "# As of July 2020, crcv.ucf.edu doesn't use a certificate accepted by the\n",
        "# default Colab environment anymore.\n",
        "unverified_context = ssl._create_unverified_context()\n",
        "\n",
        "def list_ucf_videos():\n",
        "  \"\"\"Lists videos available in UCF101 dataset.\"\"\"\n",
        "  global _VIDEO_LIST\n",
        "  if not _VIDEO_LIST:\n",
        "    index = request.urlopen(UCF_ROOT, context=unverified_context).read().decode(\"utf-8\")\n",
        "    videos = re.findall(\"(v_[\\w_]+\\.avi)\", index)\n",
        "    _VIDEO_LIST = sorted(set(videos))\n",
        "  return list(_VIDEO_LIST)\n",
        "\n",
        "def fetch_ucf_video(video):\n",
        "  \"\"\"Fetchs a video and cache into local filesystem.\"\"\"\n",
        "  cache_path = os.path.join(_CACHE_DIR, video)\n",
        "  if not os.path.exists(cache_path):\n",
        "    urlpath = request.urljoin(UCF_ROOT, video)\n",
        "    print(\"Fetching %s =\u003e %s\" % (urlpath, cache_path))\n",
        "    data = request.urlopen(urlpath, context=unverified_context).read()\n",
        "    open(cache_path, \"wb\").write(data)\n",
        "  return cache_path\n",
        "\n",
        "# Utilities to open video files using CV2\n",
        "def crop_center_square(frame):\n",
        "  y, x = frame.shape[0:2]\n",
        "  min_dim = min(y, x)\n",
        "  start_x = (x // 2) - (min_dim // 2)\n",
        "  start_y = (y // 2) - (min_dim // 2)\n",
        "  return frame[start_y:start_y+min_dim,start_x:start_x+min_dim]\n",
        "\n",
        "def load_video(path, max_frames=0, resize=(224, 224)):\n",
        "  cap = cv2.VideoCapture(path)\n",
        "  frames = []\n",
        "  try:\n",
        "    while True:\n",
        "      ret, frame = cap.read()\n",
        "      if not ret:\n",
        "        break\n",
        "      frame = crop_center_square(frame)\n",
        "      frame = cv2.resize(frame, resize)\n",
        "      frame = frame[:, :, [2, 1, 0]]\n",
        "      frames.append(frame)\n",
        "      \n",
        "      if len(frames) == max_frames:\n",
        "        break\n",
        "  finally:\n",
        "    cap.release()\n",
        "  return np.array(frames) / 255.0\n",
        "\n",
        "def to_gif(images):\n",
        "  converted_images = np.clip(images * 255, 0, 255).astype(np.uint8)\n",
        "  imageio.mimsave('./animation.gif', converted_images, fps=25)\n",
        "  return embed.embed_file('./animation.gif')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "pIKTs-KneUfz"
      },
      "outputs": [],
      "source": [
        "#@title Get the kinetics-400 labels\n",
        "# Get the kinetics-400 action labels from the GitHub repository.\n",
        "KINETICS_URL = \"https://raw.githubusercontent.com/deepmind/kinetics-i3d/master/data/label_map.txt\"\n",
        "with request.urlopen(KINETICS_URL) as obj:\n",
        "  labels = [line.decode(\"utf-8\").strip() for line in obj.readlines()]\n",
        "print(\"Found %d labels.\" % len(labels))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GBvmjVICIp3W"
      },
      "source": [
        "# Using the UCF101 dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "V-QcxdhLIfi2"
      },
      "outputs": [],
      "source": [
        "# Get the list of videos in the dataset.\n",
        "ucf_videos = list_ucf_videos()\n",
        "  \n",
        "categories = {}\n",
        "for video in ucf_videos:\n",
        "  category = video[2:-12]\n",
        "  if category not in categories:\n",
        "    categories[category] = []\n",
        "  categories[category].append(video)\n",
        "print(\"Found %d videos in %d categories.\" % (len(ucf_videos), len(categories)))\n",
        "\n",
        "for category, sequences in categories.items():\n",
        "  summary = \", \".join(sequences[:2])\n",
        "  print(\"%-20s %4d videos (%s, ...)\" % (category, len(sequences), summary))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "c0ZvVDruN2nU"
      },
      "outputs": [],
      "source": [
        "# Get a sample cricket video.\n",
        "video_path = fetch_ucf_video(\"v_CricketShot_g04_c02.avi\")\n",
        "sample_video = load_video(video_path)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "hASLA90YFPTO"
      },
      "outputs": [],
      "source": [
        "sample_video.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "POf5XgffvXlD"
      },
      "outputs": [],
      "source": [
        "i3d = hub.load(\"https://tfhub.dev/deepmind/i3d-kinetics-400/1\").signatures['default']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mDXgaOD1zhMP"
      },
      "source": [
        "Run the id3 model and print the top-5 action predictions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "3mTbqA5JGYUx"
      },
      "outputs": [],
      "source": [
        "def predict(sample_video):\n",
        "  # Add a batch axis to the to the sample video.\n",
        "  model_input = tf.constant(sample_video, dtype=tf.float32)[tf.newaxis, ...]\n",
        "\n",
        "  logits = i3d(model_input)['default'][0]\n",
        "  probabilities = tf.nn.softmax(logits)\n",
        "\n",
        "  print(\"Top 5 actions:\")\n",
        "  for i in np.argsort(probabilities)[::-1][:5]:\n",
        "    print(f\"  {labels[i]:22}: {probabilities[i] * 100:5.2f}%\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "ykaXQcGRvK4E"
      },
      "outputs": [],
      "source": [
        "predict(sample_video)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "PHsq0lHXCsD4"
      },
      "source": [
        "Now try a new video, from: https://commons.wikimedia.org/wiki/Category:Videos_of_sports\n",
        "\n",
        "How about [this video](https://commons.wikimedia.org/wiki/File:End_of_a_jam.ogv) by Patrick Gillett: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "p-mZ9fFPCoNq"
      },
      "outputs": [],
      "source": [
        "!curl -O https://upload.wikimedia.org/wikipedia/commons/8/86/End_of_a_jam.ogv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "lpLmE8rjEbAF"
      },
      "outputs": [],
      "source": [
        "video_path = \"End_of_a_jam.ogv\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "CHZJ9qTLErhV"
      },
      "outputs": [],
      "source": [
        "sample_video = load_video(video_path)[:100]\n",
        "sample_video.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "2ZNLkEZ9Er-c"
      },
      "outputs": [],
      "source": [
        "to_gif(sample_video)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "yskHIRbxEtjS"
      },
      "outputs": [],
      "source": [
        "predict(sample_video)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "x8Q7Un821X1A"
      ],
      "name": "Action Recognition on the UCF101 Dataset",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}