clip-unsplash_image_search.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "unsplash-image-search.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "Ons94QRxCQR2",
        "lJVrkmy6DVj2",
        "ujCKerTnFBk4",
        "lsdZUxzJK_af"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "97qkK30wAzXb"
      },
      "source": [
        "# Unsplash Image Search\n",
        "\n",
        "Using this notebook you can search for images from the [Unsplash Dataset](https://unsplash.com/data) using natural language queries. The search is powered by OpenAI's [CLIP](https://github.com/openai/CLIP) neural network.\n",
        "\n",
        "This notebook uses the precomputed feature vectors for almost 2 million images from the full version of the [Unsplash Dataset](https://unsplash.com/data). If you want to compute the features yourself, see [here](https://github.com/haltakov/natural-language-image-search#on-your-machine).\n",
        "\n",
        "This project was created by [Vladimir Haltakov](https://twitter.com/haltakov) and the full code is open-sourced on [GitHub](https://github.com/haltakov/natural-language-image-search)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ons94QRxCQR2"
      },
      "source": [
        "## Setup Environment\n",
        "\n",
        "In this section we will setup the environment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DYdIafWsOOUV"
      },
      "source": [
        "First we need to install CLIP and then make sure that we have torch 1.7.1 with CUDA support."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "djgE7IjbV3sv",
        "outputId": "9ad1377c-3da0-4111-d111-b65307323187",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!pip install git+https://github.com/openai/CLIP.git\n",
        "!pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 -f https://download.pytorch.org/whl/torch_stable.html"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting git+https://github.com/openai/CLIP.git\n",
            "  Cloning https://github.com/openai/CLIP.git to /tmp/pip-req-build-hf0nb4g5\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/openai/CLIP.git /tmp/pip-req-build-hf0nb4g5\n",
            "  Resolved https://github.com/openai/CLIP.git to commit dcba3cb2e2827b402d2701e7e1c7d9fed8a20ef1\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting ftfy (from clip==1.0)\n",
            "  Downloading ftfy-6.2.3-py3-none-any.whl.metadata (7.8 kB)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from clip==1.0) (24.1)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.10/dist-packages (from clip==1.0) (2024.5.15)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from clip==1.0) (4.66.4)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from clip==1.0) (2.3.1+cu121)\n",
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from clip==1.0) (0.18.1+cu121)\n",
            "Requirement already satisfied: wcwidth<0.3.0,>=0.2.12 in /usr/local/lib/python3.10/dist-packages (from ftfy->clip==1.0) (0.2.13)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->clip==1.0) (3.15.4)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->clip==1.0) (4.12.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->clip==1.0) (1.13.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->clip==1.0) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->clip==1.0) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->clip==1.0) (2024.6.1)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch->clip==1.0)\n",
            "  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch->clip==1.0)\n",
            "  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch->clip==1.0)\n",
            "  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch->clip==1.0)\n",
            "  Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cublas-cu12==12.1.3.1 (from torch->clip==1.0)\n",
            "  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cufft-cu12==11.0.2.54 (from torch->clip==1.0)\n",
            "  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-curand-cu12==10.3.2.106 (from torch->clip==1.0)\n",
            "  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
            "Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch->clip==1.0)\n",
            "  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch->clip==1.0)\n",
            "  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
            "Collecting nvidia-nccl-cu12==2.20.5 (from torch->clip==1.0)\n",
            "  Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)\n",
            "Collecting nvidia-nvtx-cu12==12.1.105 (from torch->clip==1.0)\n",
            "  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)\n",
            "Requirement already satisfied: triton==2.3.1 in /usr/local/lib/python3.10/dist-packages (from torch->clip==1.0) (2.3.1)\n",
            "Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch->clip==1.0)\n",
            "  Using cached nvidia_nvjitlink_cu12-12.6.20-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchvision->clip==1.0) (1.26.4)\n",
            "Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.10/dist-packages (from torchvision->clip==1.0) (9.4.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->clip==1.0) (2.1.5)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->clip==1.0) (1.3.0)\n",
            "Downloading ftfy-6.2.3-py3-none-any.whl (43 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.0/43.0 kB\u001b[0m \u001b[31m2.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hUsing cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
            "Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
            "Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
            "Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
            "Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
            "Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
            "Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
            "Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
            "Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
            "Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB)\n",
            "Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
            "Using cached nvidia_nvjitlink_cu12-12.6.20-py3-none-manylinux2014_x86_64.whl (19.7 MB)\n",
            "Building wheels for collected packages: clip\n",
            "  Building wheel for clip (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for clip: filename=clip-1.0-py3-none-any.whl size=1369492 sha256=0893c0f170758736e1799a5f7634b01c07bd023394b435c51de7238792a56300\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-6grz4spy/wheels/da/2b/4c/d6691fa9597aac8bb85d2ac13b112deb897d5b50f5ad9a37e4\n",
            "Successfully built clip\n",
            "Installing collected packages: nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, ftfy, nvidia-cusparse-cu12, nvidia-cudnn-cu12, nvidia-cusolver-cu12, clip\n",
            "Successfully installed clip-1.0 ftfy-6.2.3 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.20.5 nvidia-nvjitlink-cu12-12.6.20 nvidia-nvtx-cu12-12.1.105\n",
            "Looking in links: https://download.pytorch.org/whl/torch_stable.html\n",
            "\u001b[31mERROR: Could not find a version that satisfies the requirement torch==1.7.1+cu101 (from versions: 1.11.0, 1.11.0+cpu, 1.11.0+cu102, 1.11.0+cu113, 1.11.0+cu115, 1.11.0+rocm4.3.1, 1.11.0+rocm4.5.2, 1.12.0, 1.12.0+cpu, 1.12.0+cu102, 1.12.0+cu113, 1.12.0+cu116, 1.12.0+rocm5.0, 1.12.0+rocm5.1.1, 1.12.1, 1.12.1+cpu, 1.12.1+cu102, 1.12.1+cu113, 1.12.1+cu116, 1.12.1+rocm5.0, 1.12.1+rocm5.1.1, 1.13.0, 1.13.0+cpu, 1.13.0+cu116, 1.13.0+cu117, 1.13.0+cu117.with.pypi.cudnn, 1.13.0+rocm5.1.1, 1.13.0+rocm5.2, 1.13.1, 1.13.1+cpu, 1.13.1+cu116, 1.13.1+cu117, 1.13.1+cu117.with.pypi.cudnn, 1.13.1+rocm5.1.1, 1.13.1+rocm5.2, 2.0.0, 2.0.0+cpu, 2.0.0+cpu.cxx11.abi, 2.0.0+cu117, 2.0.0+cu117.with.pypi.cudnn, 2.0.0+cu118, 2.0.0+rocm5.3, 2.0.0+rocm5.4.2, 2.0.1, 2.0.1+cpu, 2.0.1+cpu.cxx11.abi, 2.0.1+cu117, 2.0.1+cu117.with.pypi.cudnn, 2.0.1+cu118, 2.0.1+rocm5.3, 2.0.1+rocm5.4.2, 2.1.0, 2.1.0+cpu, 2.1.0+cpu.cxx11.abi, 2.1.0+cu118, 2.1.0+cu121, 2.1.0+cu121.with.pypi.cudnn, 2.1.0+rocm5.5, 2.1.0+rocm5.6, 2.1.1, 2.1.1+cpu, 2.1.1+cpu.cxx11.abi, 2.1.1+cu118, 2.1.1+cu121, 2.1.1+cu121.with.pypi.cudnn, 2.1.1+rocm5.5, 2.1.1+rocm5.6, 2.1.2, 2.1.2+cpu, 2.1.2+cpu.cxx11.abi, 2.1.2+cu118, 2.1.2+cu121, 2.1.2+cu121.with.pypi.cudnn, 2.1.2+rocm5.5, 2.1.2+rocm5.6, 2.2.0, 2.2.0+cpu, 2.2.0+cpu.cxx11.abi, 2.2.0+cu118, 2.2.0+cu121, 2.2.0+rocm5.6, 2.2.0+rocm5.7, 2.2.1, 2.2.1+cpu, 2.2.1+cpu.cxx11.abi, 2.2.1+cu118, 2.2.1+cu121, 2.2.1+rocm5.6, 2.2.1+rocm5.7, 2.2.2, 2.2.2+cpu, 2.2.2+cpu.cxx11.abi, 2.2.2+cu118, 2.2.2+cu121, 2.2.2+rocm5.6, 2.2.2+rocm5.7, 2.3.0, 2.3.0+cpu, 2.3.0+cpu.cxx11.abi, 2.3.0+cu118, 2.3.0+cu121, 2.3.0+rocm5.7, 2.3.0+rocm6.0, 2.3.1, 2.3.1+cpu, 2.3.1+cpu.cxx11.abi, 2.3.1+cu118, 2.3.1+cu121, 2.3.1+rocm5.7, 2.3.1+rocm6.0, 2.4.0)\u001b[0m\u001b[31m\n",
            "\u001b[0m\u001b[31mERROR: No matching distribution found for torch==1.7.1+cu101\u001b[0m\u001b[31m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B7_Sk-T7DBEm"
      },
      "source": [
        "We can now load the pretrained public CLIP model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_6FzbzS6W1R5",
        "outputId": "36ecd312-a736-4add-da18-0b55296ea995",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import clip\n",
        "import torch\n",
        "\n",
        "# Load the open CLIP model\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "model, preprocess = clip.load(\"ViT-B/32\", device=device)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|███████████████████████████████████████| 338M/338M [00:04<00:00, 86.1MiB/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lJVrkmy6DVj2"
      },
      "source": [
        "## Download the Precomputed Data\n",
        "\n",
        "In this section the precomputed feature vectors for all photos are downloaded."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "18alAEjEOdSC"
      },
      "source": [
        "In order to compare the photos from the Unsplash dataset to a text query, we need to compute the feature vector of each photo using CLIP. This is a time consuming task, so you can use the feature vectors that I precomputed and uploaded to Google Drive (with the permission from Unsplash). If you want to compute the features yourself, see [here](https://github.com/haltakov/natural-language-image-search#on-your-machine).\n",
        "\n",
        "We need to download two files:\n",
        "* `photo_ids.csv` - a list of the photo IDs for all images in the dataset. The photo ID can be used to get the actual photo from Unsplash.\n",
        "* `features.npy` - a matrix containing the precomputed 512 element feature vector for each photo in the dataset.\n",
        "\n",
        "The files are available on [Google Drive](https://drive.google.com/drive/folders/1WQmedVCDIQKA2R33dkS1f980YsJXRZ-q?usp=sharing)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BAb15OJQZRkt",
        "outputId": "6afc0c87-475f-4672-bf9e-f25b6065c360",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from pathlib import Path\n",
        "\n",
        "# Create a folder for the precomputed features\n",
        "!mkdir unsplash-dataset\n",
        "\n",
        "# Download from Github Releases\n",
        "if not Path('unsplash-dataset/photo_ids.csv').exists():\n",
        "  !wget https://github.com/haltakov/natural-language-image-search/releases/download/1.0.0/photo_ids.csv -O unsplash-dataset/photo_ids.csv\n",
        "\n",
        "if not Path('unsplash-dataset/features.npy').exists():\n",
        "  !wget https://github.com/haltakov/natural-language-image-search/releases/download/1.0.0/features.npy -O unsplash-dataset/features.npy\n",
        ""
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-08-07 07:40:20--  https://github.com/haltakov/natural-language-image-search/releases/download/1.0.0/photo_ids.csv\n",
            "Resolving github.com (github.com)... 140.82.114.4\n",
            "Connecting to github.com (github.com)|140.82.114.4|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/330162907/ea59cda9-85ee-4657-9fb5-ddad20060ccb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240807%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240807T074020Z&X-Amz-Expires=300&X-Amz-Signature=0dd5a8e30a4212e0a53fc2a8afc03c919fb7e27ba7487ba61256b54c480e8ded&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=330162907&response-content-disposition=attachment%3B%20filename%3Dphoto_ids.csv&response-content-type=application%2Foctet-stream [following]\n",
            "--2024-08-07 07:40:20--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/330162907/ea59cda9-85ee-4657-9fb5-ddad20060ccb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240807%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240807T074020Z&X-Amz-Expires=300&X-Amz-Signature=0dd5a8e30a4212e0a53fc2a8afc03c919fb7e27ba7487ba61256b54c480e8ded&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=330162907&response-content-disposition=attachment%3B%20filename%3Dphoto_ids.csv&response-content-type=application%2Foctet-stream\n",
            "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 23773941 (23M) [application/octet-stream]\n",
            "Saving to: ‘unsplash-dataset/photo_ids.csv’\n",
            "\n",
            "unsplash-dataset/ph 100%[===================>]  22.67M  28.3MB/s    in 0.8s    \n",
            "\n",
            "2024-08-07 07:40:21 (28.3 MB/s) - ‘unsplash-dataset/photo_ids.csv’ saved [23773941/23773941]\n",
            "\n",
            "--2024-08-07 07:40:21--  https://github.com/haltakov/natural-language-image-search/releases/download/1.0.0/features.npy\n",
            "Resolving github.com (github.com)... 140.82.112.3\n",
            "Connecting to github.com (github.com)|140.82.112.3|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/330162907/275c8863-87ae-48a7-8190-f05657f5ed53?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240807%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240807T074022Z&X-Amz-Expires=300&X-Amz-Signature=5271dd2e111333318d66a9756d4636216f9fefca0e7bc8d92323f9ae55b06cd8&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=330162907&response-content-disposition=attachment%3B%20filename%3Dfeatures.npy&response-content-type=application%2Foctet-stream [following]\n",
            "--2024-08-07 07:40:22--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/330162907/275c8863-87ae-48a7-8190-f05657f5ed53?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240807%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240807T074022Z&X-Amz-Expires=300&X-Amz-Signature=5271dd2e111333318d66a9756d4636216f9fefca0e7bc8d92323f9ae55b06cd8&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=330162907&response-content-disposition=attachment%3B%20filename%3Dfeatures.npy&response-content-type=application%2Foctet-stream\n",
            "Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2028708992 (1.9G) [application/octet-stream]\n",
            "Saving to: ‘unsplash-dataset/features.npy’\n",
            "\n",
            "unsplash-dataset/fe 100%[===================>]   1.89G  42.7MB/s    in 56s     \n",
            "\n",
            "2024-08-07 07:41:18 (34.4 MB/s) - ‘unsplash-dataset/features.npy’ saved [2028708992/2028708992]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TVjuUh6oEtPt"
      },
      "source": [
        "After the files are downloaded we need to load them using `pandas` and `numpy`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GQHcmMo1Ztjz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "91c08d4a-e08f-4ed4-800d-ba68873a07ef"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "# Load the photo IDs\n",
        "photo_ids = pd.read_csv(\"unsplash-dataset/photo_ids.csv\")\n",
        "photo_ids = list(photo_ids['photo_id'])\n",
        "\n",
        "# Load the features vectors\n",
        "photo_features = np.load(\"unsplash-dataset/features.npy\")\n",
        "\n",
        "# Convert features to Tensors: Float32 on CPU and Float16 on GPU\n",
        "if device == \"cpu\":\n",
        "  photo_features = torch.from_numpy(photo_features).float().to(device)\n",
        "else:\n",
        "  photo_features = torch.from_numpy(photo_features).to(device)\n",
        "\n",
        "# Print some statistics\n",
        "print(f\"Photos loaded: {len(photo_ids)}\")"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Photos loaded: 1981161\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ujCKerTnFBk4"
      },
      "source": [
        "## Define Functions\n",
        "\n",
        "Some important functions for processing the data are defined here.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pYVNtF-JFtfj"
      },
      "source": [
        "The `encode_search_query` function takes a text description and encodes it into a feature vector using the CLIP model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d0hmOh3qbcxK"
      },
      "source": [
        "def encode_search_query(search_query):\n",
        "  with torch.no_grad():\n",
        "    # Encode and normalize the search query using CLIP\n",
        "    text_encoded = model.encode_text(clip.tokenize(search_query).to(device))\n",
        "    text_encoded /= text_encoded.norm(dim=-1, keepdim=True)\n",
        "\n",
        "  # Retrieve the feature vector\n",
        "  return text_encoded"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vh1yyJtEGCAX"
      },
      "source": [
        "The `find_best_matches` function compares the text feature vector to the feature vectors of all images and finds the best matches. The function returns the IDs of the best matching photos."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3TcmI5KIbe5F"
      },
      "source": [
        "def find_best_matches(text_features, photo_features, photo_ids, results_count=3):\n",
        "  # Compute the similarity between the search query and each photo using the Cosine similarity\n",
        "  similarities = (photo_features @ text_features.T).squeeze(1)\n",
        "\n",
        "  # Sort the photos by their similarity score\n",
        "  best_photo_idx = (-similarities).argsort()\n",
        "\n",
        "  # Return the photo IDs of the best matches\n",
        "  return [photo_ids[i] for i in best_photo_idx[:results_count]]"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gEmt0F4iHbL0"
      },
      "source": [
        "The `display_photo` function displays a photo from Unsplash given its ID and link to the original photo on Unsplash."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RC4HD8cBYOon"
      },
      "source": [
        "from IPython.display import Image\n",
        "from IPython.core.display import HTML\n",
        "\n",
        "def display_photo(photo_id):\n",
        "  # Get the URL of the photo resized to have a width of 320px\n",
        "  photo_image_url = f\"https://unsplash.com/photos/{photo_id}/download?w=320\"\n",
        "\n",
        "  # Display the photo\n",
        "  display(Image(url=photo_image_url))\n",
        "\n",
        "  # Display the attribution text\n",
        "  display(HTML(f'Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/{photo_id}\">Unsplash</a> '))\n",
        "  print()"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_3ojinZ0JYBC"
      },
      "source": [
        "Putting it all together in one function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LvUcljF5JcRn"
      },
      "source": [
        "def search_unslash(search_query, photo_features, photo_ids, results_count=3):\n",
        "  # Encode the search query\n",
        "  text_features = encode_search_query(search_query)\n",
        "\n",
        "  # Find the best matches\n",
        "  best_photo_ids = find_best_matches(text_features, photo_features, photo_ids, results_count)\n",
        "\n",
        "  # Display the best photos\n",
        "  for photo_id in best_photo_ids:\n",
        "    display_photo(photo_id)\n"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xbym_cYJJH6v"
      },
      "source": [
        "## Search Unsplash\n",
        "\n",
        "Now we are ready to search the dataset using natural language. Check out the examples below and feel free to try out your own queries."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-RmOFAq5NtlI"
      },
      "source": [
        "### \"Two dogs playing in the snow\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CF7HuxAlFNXT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 774
        },
        "outputId": "876bb0b1-e231-414a-901f-26054b4e73c4"
      },
      "source": [
        "search_query = \"Two dogs playing in the snow\"\n",
        "\n",
        "search_unslash(search_query, photo_features, photo_ids, 3)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/cBAcMcwQ1h0/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/cBAcMcwQ1h0\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/jKDFcXwk5Cw/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/jKDFcXwk5Cw\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/rKxSxPAtW_E/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/rKxSxPAtW_E\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PtaYocbjN0VQ"
      },
      "source": [
        "### \"The word love written on the wall\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OswqrzaeMy1J",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 544
        },
        "outputId": "f4018512-e940-4b03-eddc-0d7ccb87dbe9"
      },
      "source": [
        "search_query = \"The word love written on the wall\"\n",
        "\n",
        "search_unslash(search_query, photo_features, photo_ids, 3)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/3KO3bRLxsag/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/3KO3bRLxsag\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/TXMJKoy-Q3k/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/TXMJKoy-Q3k\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/HgvuuKoOseA/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/HgvuuKoOseA\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sUdySrczN4ZX"
      },
      "source": [
        "### \"The feeling when your program finally works\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SRyCZMHQMzOP",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 576
        },
        "outputId": "38598e03-e882-4fd6-b1cd-8d0fb235643c"
      },
      "source": [
        "search_query = \"The feeling when your program finally works\"\n",
        "\n",
        "search_unslash(search_query, photo_features, photo_ids, 3)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/AndE50aaHn4/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/AndE50aaHn4\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/I6-sHSAt9vc/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/I6-sHSAt9vc\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/02oFZfhfX_I/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/02oFZfhfX_I\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aR4aDfQYN8J1"
      },
      "source": [
        "### \"The Syndey Opera House and the Harbour Bridge at night\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wWkWfHhnMzZe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 718
        },
        "outputId": "40fa4a76-5760-42fb-efeb-19789437a56b"
      },
      "source": [
        "search_query = \"The Syndey Opera House and the Harbour Bridge at night\"\n",
        "\n",
        "search_unslash(search_query, photo_features, photo_ids, 3)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/beYNO-kn4-Y/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/beYNO-kn4-Y\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/cY8I_ncKezc/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/cY8I_ncKezc\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/OAzRalQb0F4/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/OAzRalQb0F4\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lsdZUxzJK_af"
      },
      "source": [
        "## Combine Text and Photo Seach Queries\n",
        "\n",
        "This is another experiment to combine a text query with another photo."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Yxox5LGfqEj"
      },
      "source": [
        "The idea here is to do a text search for a photo and then modify the search query by adding another photo to the search query in order to transfer some of the photo features to the search.\n",
        "\n",
        "This works by adding the features of the photo to the features of the text query. The photo features are multiplied with a weight in order to reduce the influence so that the text query is the main source.\n",
        "\n",
        "The results are somewhat sensitive to the prompt..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZEBq8_TeUOFm"
      },
      "source": [
        "def search_by_text_and_photo(query_text, query_photo_id, photo_weight=0.5):\n",
        "  # Encode the search query\n",
        "  text_features = encode_search_query(query_text)\n",
        "\n",
        "  # Find the feature vector for the specified photo ID\n",
        "  query_photo_index = photo_ids.index(query_photo_id)\n",
        "  query_photo_features = photo_features[query_photo_index]\n",
        "\n",
        "  # Combine the test and photo queries and normalize again\n",
        "  search_features = text_features + query_photo_features * photo_weight\n",
        "  search_features /= search_features.norm(dim=-1, keepdim=True)\n",
        "\n",
        "  # Find the best match\n",
        "  best_photo_ids = find_best_matches(search_features, photo_features, photo_ids, 1)\n",
        "\n",
        "  # Display the results\n",
        "  print(\"Test search result\")\n",
        "  search_unslash(query_text, photo_features, photo_ids, 1)\n",
        "\n",
        "  print(\"Photo query\")\n",
        "  display(Image(url=f\"https://unsplash.com/photos/{query_photo_id}/download?w=320\"))\n",
        "\n",
        "  print(\"Result for text query + photo query\")\n",
        "  display_photo(best_photo_ids[0])"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1foDoFYgRPm"
      },
      "source": [
        "## Results Combining Text and Photo Seach Queries\n",
        "\n",
        "Now some results for combining text and photo queries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g0uuA3aYguZe"
      },
      "source": [
        "### Sydney Opera House + night photo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 844
        },
        "id": "ruf7Tmy9LVor",
        "outputId": "901f6460-b2de-407c-dc4a-408b6d071d1b"
      },
      "source": [
        "search_by_text_and_photo(\"Sydney Opera house\", \"HSsOC5nqurA\")"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test search result\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/QP-l1uE19iI/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/QP-l1uE19iI\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Photo query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/HSsOC5nqurA/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Result for text query + photo query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/KCEUHBb4wjc/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/KCEUHBb4wjc\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_A9ui7ugyM8"
      },
      "source": [
        "### Sydney Opera House + mist photo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 844
        },
        "id": "NXba1meDdfAt",
        "outputId": "f11a632f-4259-42a3-d156-cf0e49e7601f"
      },
      "source": [
        "search_by_text_and_photo(\"Sydney Opera house\", \"MaerUPAjPbs\")"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test search result\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/QP-l1uE19iI/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/QP-l1uE19iI\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Photo query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/MaerUPAjPbs/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Result for text query + photo query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/GSXMuGwY2l4/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/GSXMuGwY2l4\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SQcwhrAOg0pg"
      },
      "source": [
        "### Sydney Opera House + rain photo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "rjq8GcBOfNDA",
        "outputId": "af1b0034-8ff9-4eea-bb10-b616134603b8"
      },
      "source": [
        "search_by_text_and_photo(\"Sydney Opera house\", \"1pNBJ2zUfn4\", 0.4)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test search result\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/QP-l1uE19iI/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/QP-l1uE19iI\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Photo query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/1pNBJ2zUfn4/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Result for text query + photo query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/6WuX1eYOciE/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/6WuX1eYOciE\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dXXsOKhUg3nU"
      },
      "source": [
        "### Sydney Opera House + sea photo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 792
        },
        "id": "kQiQ4w-HfeL6",
        "outputId": "0cc66cf3-4bcf-4f02-bcb1-693aea2a415b"
      },
      "source": [
        "search_by_text_and_photo(\"Sydney Opera house\", \"jnBDclcdZ7A\", 0.4)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test search result\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/QP-l1uE19iI/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/QP-l1uE19iI\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Photo query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/jnBDclcdZ7A/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Result for text query + photo query\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<img src=\"https://unsplash.com/photos/2gwhorA9tic/download?w=320\"/>"
            ],
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "Photo on <a target=\"_blank\" href=\"https://unsplash.com/photos/2gwhorA9tic\">Unsplash</a> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n"
          ]
        }
      ]
    }
  ]
}