notebook/export_image_for_prediction.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 NASA\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Rice mapping in Bhutan with U-Net using high resolution satellite imagery\n",
        "\n",
        "### This notebook shows an example of exporting the images as tiles for use in prediction. We will use Google Earth Engine (GEE) for this.\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/SERVIR/servir-aces/blob/main/notebook/export_image_for_prediction.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/SERVIR/servir-aces/blob/main/notebook/export_image_for_prediction.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "</table>\n",
        "</br>\n",
        "</br>\n",
        "</br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Zk3R20LkKet"
      },
      "source": [
        "This notebook is also available in this github repo: https://github.com/SERVIR/servir-aces. Navigate to the `notebooks` folder."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup environment"
      ],
      "metadata": {
        "id": "EuQkhSxRmiIA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install servir-aces"
      ],
      "metadata": {
        "id": "FuDztFucmhpB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "69e0ebee-4713-46b3-8949-ab7f5357457e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting servir-aces\n",
            "  Downloading servir_aces-0.0.14-py2.py3-none-any.whl (32 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from servir-aces) (1.25.2)\n",
            "Requirement already satisfied: tensorflow>=2.9.3 in /usr/local/lib/python3.10/dist-packages (from servir-aces) (2.15.0)\n",
            "Requirement already satisfied: earthengine-api in /usr/local/lib/python3.10/dist-packages (from servir-aces) (0.1.399)\n",
            "Collecting python-dotenv>=1.0.0 (from servir-aces)\n",
            "  Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from servir-aces) (3.7.1)\n",
            "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (1.4.0)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (1.6.3)\n",
            "Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (24.3.25)\n",
            "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (0.5.4)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (0.2.0)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (3.9.0)\n",
            "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (18.1.1)\n",
            "Requirement already satisfied: ml-dtypes~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (0.2.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (3.3.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (24.0)\n",
            "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (3.20.3)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (67.7.2)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (1.16.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (2.4.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (4.11.0)\n",
            "Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (1.14.1)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (0.36.0)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (1.62.2)\n",
            "Requirement already satisfied: tensorboard<2.16,>=2.15 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (2.15.2)\n",
            "Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (2.15.0)\n",
            "Requirement already satisfied: keras<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow>=2.9.3->servir-aces) (2.15.0)\n",
            "Requirement already satisfied: google-cloud-storage in /usr/local/lib/python3.10/dist-packages (from earthengine-api->servir-aces) (2.8.0)\n",
            "Requirement already satisfied: google-api-python-client>=1.12.1 in /usr/local/lib/python3.10/dist-packages (from earthengine-api->servir-aces) (2.84.0)\n",
            "Requirement already satisfied: google-auth>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from earthengine-api->servir-aces) (2.27.0)\n",
            "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/local/lib/python3.10/dist-packages (from earthengine-api->servir-aces) (0.1.1)\n",
            "Requirement already satisfied: httplib2<1dev,>=0.9.2 in /usr/local/lib/python3.10/dist-packages (from earthengine-api->servir-aces) (0.22.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from earthengine-api->servir-aces) (2.31.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->servir-aces) (1.2.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->servir-aces) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->servir-aces) (4.51.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->servir-aces) (1.4.5)\n",
            "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->servir-aces) (9.4.0)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->servir-aces) (3.1.2)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->servir-aces) (2.8.2)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow>=2.9.3->servir-aces) (0.43.0)\n",
            "Requirement already satisfied: google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client>=1.12.1->earthengine-api->servir-aces) (2.11.1)\n",
            "Requirement already satisfied: uritemplate<5,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client>=1.12.1->earthengine-api->servir-aces) (4.1.1)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth>=1.4.1->earthengine-api->servir-aces) (5.3.3)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth>=1.4.1->earthengine-api->servir-aces) (0.4.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth>=1.4.1->earthengine-api->servir-aces) (4.9)\n",
            "Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow>=2.9.3->servir-aces) (1.2.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow>=2.9.3->servir-aces) (3.6)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow>=2.9.3->servir-aces) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow>=2.9.3->servir-aces) (3.0.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->earthengine-api->servir-aces) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->earthengine-api->servir-aces) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->earthengine-api->servir-aces) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->earthengine-api->servir-aces) (2024.2.2)\n",
            "Requirement already satisfied: google-cloud-core<3.0dev,>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-storage->earthengine-api->servir-aces) (2.3.3)\n",
            "Requirement already satisfied: google-resumable-media>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from google-cloud-storage->earthengine-api->servir-aces) (2.7.0)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-api-python-client>=1.12.1->earthengine-api->servir-aces) (1.63.0)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow>=2.9.3->servir-aces) (1.3.1)\n",
            "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /usr/local/lib/python3.10/dist-packages (from google-resumable-media>=2.3.2->google-cloud-storage->earthengine-api->servir-aces) (1.5.0)\n",
            "Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth>=1.4.1->earthengine-api->servir-aces) (0.6.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow>=2.9.3->servir-aces) (2.1.5)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow>=2.9.3->servir-aces) (3.2.2)\n",
            "Installing collected packages: python-dotenv, servir-aces\n",
            "Successfully installed python-dotenv-1.0.1 servir-aces-0.0.14\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/SERVIR/servir-aces"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6PvoP9Sgqz4e",
        "outputId": "e4f5f519-c8ca-4edb-f03f-c694dd5f7469"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'servir-aces'...\n",
            "remote: Enumerating objects: 731, done.\u001b[K\n",
            "remote: Counting objects: 100% (90/90), done.\u001b[K\n",
            "remote: Compressing objects: 100% (50/50), done.\u001b[K\n",
            "remote: Total 731 (delta 55), reused 40 (delta 40), pack-reused 641\u001b[K\n",
            "Receiving objects: 100% (731/731), 3.35 MiB | 22.30 MiB/s, done.\n",
            "Resolving deltas: 100% (478/478), done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now the repo is downloaded. We will create an environment file file to place point to our training data and customize parameters for the model. To do this, we make a copy of the `.env.example` file provided.\n",
        "\n",
        "Under the hood, all the configuration provided via the environment file are parsed as a config object and can be accessed programatically.\n",
        "\n",
        "Note current version does not expose all the model intracacies through the environment file but future version may include those depending on the need."
      ],
      "metadata": {
        "id": "RukLD6eeq7Fl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cp servir-aces/.env.example servir-aces/config.env"
      ],
      "metadata": {
        "id": "jpnfjXRYsYMk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup config file variables"
      ],
      "metadata": {
        "id": "w_mHTVmzoH3O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Okay, now we have the `config.env` file, we will use this to provide our environments and parameters.\n",
        "\n",
        "Note there are several parameters that can be changed. Let's start by changing the BASEDIR as below.\n",
        "\n",
        "```\n",
        "BASEDIR = \"/content/\"\n",
        "```"
      ],
      "metadata": {
        "id": "ooFqBUY3gS5e"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We are trying to export RGBN from Planetscope mosiac. Since we are using growing season and pre-growing season information. Thus, we have 8 optical bands, namely `red_before`, `green_before`, `blue_before`, `nir_before`, `red_during`, `green_during`, `blue_during`, and  `nir_during`. In adidition, you can use `USE_ELEVATION` and `USE_S1` config to include the topographic and radar information. Since currently we are not including these, so we won't be settting these config values.\n",
        "\n",
        "Similarly, we will be tiling to 256x256 pixels, so let's also change that. In addition, if you want to keep buffer on the export images buffer for prediction purpose, you can use `KERNEL_BUFFER` to specify that. Half this will extend on the sides of each patch. You can specify the size as tupe (e.g. 72 x 72). If zero is used; it will not buffer. I will keep this to zero for now.\n",
        "\n",
        "We will also make sure our `GCS_PROJECT` is setup correctly.\n",
        "\n",
        "```\n",
        "# For model training, USE_ELEVATION extends FEATURES with \"elevation\" & \"slope\"\n",
        "# USE_S1 extends FEATURES with \"vv_asc_before\", \"vh_asc_before\", \"vv_asc_during\", \"vh_asc_during\",\n",
        "# \"vv_desc_before\", \"vh_desc_before\", \"vv_desc_during\", \"vh_desc_during\"\n",
        "# In case these are not useful and you have other bands in your training data, you can do set\n",
        "# USE_ELEVATION and USE_S1 to False and update FEATURES to include needed bands\n",
        "USE_ELEVATION = False\n",
        "USE_S1 = False\n",
        "\n",
        "PATCH_SHAPE = (256, 256)\n",
        "\n",
        "KERNEL_BUFFER = 0\n",
        "\n",
        "GCS_PROJECT = \"servir-ee\"\n",
        "```"
      ],
      "metadata": {
        "id": "CEDUshfUi1OY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's also change the `SCALE`. `SCALE` is the scale of output; we've put 10 for this. Next, we and `GCS_BUCKET` is the Google Cloud Bucket you want this to be exported. You can use `GCS_IMAGE_DIR` to specify where inside the `GCS_BUCKET` you want to keep your prediction image. Similarly, you can use `GCS_IMAGE_PREFIX` to specify the file name prefix for the export image. In general, you can use them to construct the `file_name_prefix` (equivalent to fileNamePrefix in the `Export.image.toCloudStorage` function). Let's say I have a bucket with the name `dl-book`, and I want to store the exported image inside the sub-directory `chapter-1/images` with `image_2021` as my file name prefix, equivalent to `gs://dl-book/chapter-1/images/image_2021*.TFRecord`, then the config settings looks as below.\n",
        "\n",
        "```\n",
        "SCALE = 10\n",
        "\n",
        "GCS_BUCKET = \"dl-book\"\n",
        "# prediction image directory\n",
        "GCS_IMAGE_DIR = \"chapter-1/images\"\n",
        "# prediction image prefix\n",
        "GCS_IMAGE_PREFIX = \"image_2021\"\n",
        "```"
      ],
      "metadata": {
        "id": "yM619anPsX0W"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Update the config file programtically"
      ],
      "metadata": {
        "id": "WI0801vIoMt9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's make a dictionary so we can change these config settings programatically."
      ],
      "metadata": {
        "id": "sbyOTNqHoOl-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "BASEDIR = \"/content/\" # @param {type:\"string\"}\n",
        "\n",
        "USE_ELEVATION = \"False\" # @param {type:\"string\"}\n",
        "USE_S1 = \"False\" # @param {type:\"string\"}\n",
        "PATCH_SHAPE = \"(256, 256)\" # @param {type:\"string\"}\n",
        "KERNEL_BUFFER = \"0\" # @param {type:\"string\"}\n",
        "\n",
        "GCS_PROJECT = \"servir-ee\" # @param {type:\"string\"}\n",
        "\n",
        "SCALE = \"10\" # @param {type:\"string\"}\n",
        "\n",
        "GCS_BUCKET = \"dl-book\" # @param {type:\"string\"}\n",
        "# prediction image directory\n",
        "GCS_IMAGE_DIR = \"chapter-1/images\" # @param {type:\"string\"}\n",
        "# prediction image prefix\n",
        "GCS_IMAGE_PREFIX = \"image_2021\" # @param {type:\"string\"}"
      ],
      "metadata": {
        "id": "cDmNkw3IoQDo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "config_settings = {\n",
        "    \"BASEDIR\" : BASEDIR,\n",
        "    \"USE_ELEVATION\": USE_ELEVATION,\n",
        "    \"USE_S1\": USE_S1,\n",
        "    \"PATCH_SHAPE\": PATCH_SHAPE,\n",
        "    \"KERNEL_BUFFER\": KERNEL_BUFFER,\n",
        "    \"SCALE\": SCALE,\n",
        "    \"GCS_PROJECT\": GCS_PROJECT,\n",
        "    \"GCS_BUCKET\": GCS_BUCKET,\n",
        "    \"GCS_IMAGE_DIR\": GCS_IMAGE_DIR,\n",
        "    \"GCS_IMAGE_PREFIX\": GCS_IMAGE_PREFIX,\n",
        "}\n"
      ],
      "metadata": {
        "id": "6ZEUCkx4ohGF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import dotenv\n",
        "\n",
        "config_file = \"servir-aces/config.env\"\n",
        "\n",
        "for config_key in config_settings:\n",
        "    dotenv.set_key(dotenv_path=config_file,\n",
        "                   key_to_set=config_key,\n",
        "                   value_to_set=config_settings[config_key]\n",
        "                   )\n"
      ],
      "metadata": {
        "id": "cypqxkVlokHW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load config file variables"
      ],
      "metadata": {
        "id": "C_IrT42yaTAB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from aces import Config, EEUtils\n",
        "import ee"
      ],
      "metadata": {
        "id": "cdz2nhA2aV7X"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "config = Config(config_file)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q7ofsYbhrDla",
        "outputId": "0f9856c6-1b66-405b-9663-c58d08a524ef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "BASEDIR: /content\n",
            "DATADIR: /content/data\n",
            "using features: ['red_before', 'green_before', 'blue_before', 'nir_before', 'red_during', 'green_during', 'blue_during', 'nir_during']\n",
            "using labels: ['class']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Initialize the Earth Engine session"
      ],
      "metadata": {
        "id": "cIvdLmAtafI5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ee.Authenticate()"
      ],
      "metadata": {
        "id": "H9ubfz3xatRy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "EEUtils.initialize_session(use_highvolume=True, project=config.GCS_PROJECT)"
      ],
      "metadata": {
        "id": "kXdJKFU8aemX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prep for data export"
      ],
      "metadata": {
        "id": "kTM96846ufyZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We are focusing on Paro district of Bhutan. Let's start by importing necessary. We will focus on the rice growing elevation of Paro, which is roughly ~1500 msl to ~2600 msl. Therefore, we will use the following `Geometry` for this chapter."
      ],
      "metadata": {
        "id": "90_9CCcBZthI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "paro_geom = ee.Geometry.Polygon(\n",
        "        [[[89.25910506388209, 27.58540960195346],\n",
        "          [89.25910506388209, 27.159794800895543],\n",
        "          [89.58182845255396, 27.159794800895543],\n",
        "          [89.58182845255396, 27.58540960195346]]], None, False)\n"
      ],
      "metadata": {
        "id": "7WACSSYTZtG3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next we will importing our images to export to. We have already prepared the pre-growing and growing composite for Paro. Let's use that."
      ],
      "metadata": {
        "id": "PyPJuBWw-T-t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "composite_before = ee.Image(\"projects/servir-sco-assets/assets/Bhutan/ACES_2/Paro_Rice_Composite_2021/composite_before\")\n",
        "composite_during = ee.Image(\"projects/servir-sco-assets/assets/Bhutan/ACES_2/Paro_Rice_Composite_2021/composite_during\")\n"
      ],
      "metadata": {
        "id": "mwHJsVQKZtLO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Selected needed bands and rename the bands for clarification. Then add the composites together."
      ],
      "metadata": {
        "id": "CM8pkK4g-lNG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "bands = [\"red\", \"green\", \"blue\", \"nir\"]\n",
        "\n",
        "composite_before = composite_before.select(bands)\n",
        "composite_during = composite_during.select(bands)\n",
        "\n",
        "composite_before = composite_before.regexpRename(\"$(.*)\", \"_before\")\n",
        "composite_during = composite_during.regexpRename(\"$(.*)\", \"_during\")\n",
        "image = composite_before.addBands(composite_during).toFloat()\n"
      ],
      "metadata": {
        "id": "Z226I2vZ-qla"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "if config.USE_ELEVATION:\n",
        "    elevation = ee.Image(\"projects/servir-sco-assets/assets/Bhutan/ACES_2/elevationParo\")\n",
        "    slope = ee.Image(\"projects/servir-sco-assets/assets/Bhutan/ACES_2/slopeParo\")\n",
        "    image = image.addBands(elevation).addBands(slope).toFloat()\n",
        "    config.FEATURES.extend([\"elevation\", \"slope\"])\n"
      ],
      "metadata": {
        "id": "-gYGm9pW-4yN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "if config.USE_S1:\n",
        "    sentinel1_asc_before_composite = ee.Image(\"projects/servir-sco-assets/assets/Bhutan/Sentinel1Ascending2021/s1AscBefore\")\n",
        "    sentinel1_asc_during_composite = ee.Image(\"projects/servir-sco-assets/assets/Bhutan/Sentinel1Ascending2021/s1AscDuring\")\n",
        "    sentinel1_desc_before_composite = ee.Image(\"projects/servir-sco-assets/assets/Bhutan/Sentinel1Descending2021/s1DescBefore\")\n",
        "    sentinel1_desc_during_composite = ee.Image(\"projects/servir-sco-assets/assets/Bhutan/Sentinel1Descending2021/s1DescDuring\")\n",
        "\n",
        "    image = image.addBands(sentinel1_asc_before_composite).addBands(sentinel1_asc_during_composite).addBands(sentinel1_desc_before_composite).addBands(sentinel1_desc_during_composite).toFloat()\n",
        "    config.FEATURES.extend([\"vv_asc_before\", \"vh_asc_before\", \"vv_asc_during\", \"vh_asc_during\",\n",
        "                            \"vv_desc_before\", \"vh_desc_before\", \"vv_desc_during\", \"vh_desc_during\"])"
      ],
      "metadata": {
        "id": "BQ_7_9py-40-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally select the `FEATURES` and print the band names in the image."
      ],
      "metadata": {
        "id": "aWOSsmJpdsa8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "image = image.select(config.FEATURES)\n",
        "print(\"image\", image.bandNames().getInfo())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wvER7zUj-43q",
        "outputId": "041eed9f-9521-4bc9-e5f4-7ad32371bf68"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "image ['red_before', 'green_before', 'blue_before', 'nir_before', 'red_during', 'green_during', 'blue_during', 'nir_during']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's specify patch and file dimensions.\n"
      ],
      "metadata": {
        "id": "Zhas9Gpedz5L"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "format_options = {\n",
        "  \"patchDimensions\": [config.PATCH_SHAPE_SINGLE, config.PATCH_SHAPE_SINGLE],\n",
        "  \"maxFileSize\": 104857600,\n",
        "  \"compressed\": True\n",
        "}\n"
      ],
      "metadata": {
        "id": "APpHo3rE-46Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "As mentioned before, if you have `KERNEL_BUFFER`, we will add it here."
      ],
      "metadata": {
        "id": "Qc89Gx2hd7sM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "if config.KERNEL_BUFFER:\n",
        "    format_options[\"kernelSize\"] = config.KERNEL_BUFFER"
      ],
      "metadata": {
        "id": "faBUhU9L-48t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, let's setup the export task options."
      ],
      "metadata": {
        "id": "pe2JlT3MeFAx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Setup the task\n",
        "image_export_options = {\n",
        "    \"description\": \"export_task_for_prediction\",\n",
        "    \"file_name_prefix\": f\"{config.GCS_IMAGE_DIR}/{config.GCS_IMAGE_PREFIX}\",\n",
        "    \"bucket\": config.GCS_BUCKET,\n",
        "    \"scale\": config.SCALE,\n",
        "    \"file_format\": \"TFRecord\",\n",
        "    \"region\": paro_geom,\n",
        "    \"format_options\": format_options,\n",
        "    \"max_pixels\": 1e13,\n",
        "}\n",
        "\n",
        "print(\"image_export_options\", image_export_options)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JZ9nAMWB-4_k",
        "outputId": "8cb848bb-b4e9-43fd-b28d-ccbc1cd7002b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "image_export_options {'description': 'export_task_for_prediction', 'file_name_prefix': 'chapter-1/images/image_2021', 'bucket': 'dl-book', 'scale': 10, 'file_format': 'TFRecord', 'region': ee.Geometry({\n",
            "  \"functionInvocationValue\": {\n",
            "    \"functionName\": \"GeometryConstructors.Polygon\",\n",
            "    \"arguments\": {\n",
            "      \"coordinates\": {\n",
            "        \"constantValue\": [\n",
            "          [\n",
            "            [\n",
            "              89.25910506388209,\n",
            "              27.58540960195346\n",
            "            ],\n",
            "            [\n",
            "              89.25910506388209,\n",
            "              27.159794800895543\n",
            "            ],\n",
            "            [\n",
            "              89.58182845255396,\n",
            "              27.159794800895543\n",
            "            ],\n",
            "            [\n",
            "              89.58182845255396,\n",
            "              27.58540960195346\n",
            "            ]\n",
            "          ]\n",
            "        ]\n",
            "      },\n",
            "      \"geodesic\": {\n",
            "        \"constantValue\": false\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}), 'format_options': {'patchDimensions': [256, 256], 'maxFileSize': 104857600, 'compressed': True}, 'max_pixels': 10000000000000.0}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Start the export task"
      ],
      "metadata": {
        "id": "DqslbjG2up-Z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's start the export task. For this we are using `cloud` export. The `export_image` of `EEUtils` class makes it easier to export the images. You can specify `cloud` or `asset` for your `export_type`. Once this run successfully, you can check your `Tasks` tab to view the job."
      ],
      "metadata": {
        "id": "Mto25hineRF3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "EEUtils.export_image(image, export_type=[\"cloud\"], start_training=True, **image_export_options)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rSKPgNAt-5DC",
        "outputId": "2c759aa1-5765-4865-d250-0ad2af7d7d9b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Exporting training data to gs://dl-book/chapter-1/images/image_2021..\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NI3DZXi8Po4j"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}