airbyte-ci/connectors/pipelines/Python connector images analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4c9b39ce-f764-4af5-8762-b6c645712c5c",
   "metadata": {},
   "source": [
    "Author: Augustin Lafanechere\n",
    "\n",
    "Date: 2023-09-08\n",
    "\n",
    "# What is this?\n",
    "This notebook contains an analysis of our certified python connectors Dockerfiles.\n",
    "We originally had a lot of false certitudes that all our Python connectors are built the same.\n",
    "This is an attempt at analyzing how similar, and different, our connector images are.\n",
    "\n",
    "# Context\n",
    "The Connector Operations team wants to consolidate our Python connector build process to have a single way of building connector images. This will be achieved via the definition of:\n",
    "-  a common base image which will contain the majority of the system needs of our connectors\n",
    "-  a common build procedure for our Python connectors (no per connector Dockerfile, we'll remove them)\n",
    "- a simple framework of hooks that will allow connector developers to customize the connector image **only if needed**\n",
    "\n",
    "# Worfklow\n",
    "1. Determine the Dockerfile variants we have: scan our certified connectors dockerfile and identify families (variant) of Dockerfile that exists in our codebase. This will hopefully help us narrow down the analysis to a reduce number of images instead of analyzing all our connector images.\n",
    "2. Analyze the different python base images use by the variants.\n",
    "3. Analyze the environments variables set in these variants: we'll check which env var is common and static for all connectors, which env var common to all images but with different values and which env var is set on only a portion of Dockerfiles.\n",
    "4. Analyze the system dependencies installed in the variants: assess if any connector is installing a custom system dependency and if we should make this dependency join the base image.\n",
    "5. Given the conclusion of the previous steps, suggest a base image defintion and a build procedure for certified Python connectors.\n",
    "\n",
    "# Let's code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f1bc15dc-b084-49b9-b2de-291783ea40f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "from connector_ops.utils import ConnectorLanguage, get_all_connectors_in_repo, Connector\n",
    "import os\n",
    "import git\n",
    "import hashlib\n",
    "import sys\n",
    "import anyio\n",
    "import dagger\n",
    "# Changing current working directory to airbyte repo root\n",
    "os.chdir(Path(git.Repo(search_parent_directories=True).working_tree_dir))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "23248965-44d0-469a-b055-f96b1ff649fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "ALL_CONNECTORS = get_all_connectors_in_repo()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec7ace3b-8d40-4d05-af7f-9ea780b562d5",
   "metadata": {},
   "source": [
    "## Identifying dockerfile variants\n",
    "The function below parses the connectors dockerfiles and tries to remove the connector specifics instructions to determine a \"variant\". This will help us identify the different kind (variant) of Dockerfile existing in our codebase, instead of analysing each Dockerfile separatly. It will help narrow down our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3c1e7a5b-c957-49e2-9cb8-e00b79cecf0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dockerfile_variant(connector):\n",
    "    dockerfile = Path(connector.code_directory / \"Dockerfile\").read_text()\n",
    "    dockerfile = dockerfile.replace(connector.technical_name, \"connector-technical-name\")\n",
    "    dockerfile = dockerfile.replace(connector.technical_name.replace(\"-\", \"_\"), \"connector_technical_name\")\n",
    "\n",
    "    for line in dockerfile.splitlines():\n",
    "        # Remove dockerfile comments\n",
    "        if line.startswith(\"#\"):\n",
    "            dockerfile = dockerfile.replace(line, \"\")\n",
    "        # Remove connnector version label\n",
    "        if line.startswith(\"LABEL io.airbyte.version\"):\n",
    "            dockerfile = dockerfile.replace(line, \"\")\n",
    "    # Remove empty lines\n",
    "    dockerfile = \"\\n\".join([line for line in dockerfile.split(\"\\n\") if line.strip() != \"\"])\n",
    "    # Remove extra spaces\n",
    "    dockerfile = \"\\n\".join([line.strip() for line in dockerfile.split(\"\\n\")])\n",
    "\n",
    "\n",
    "    return hashlib.sha256(dockerfile.encode()).hexdigest()[:7]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "301d81d0-9d8a-4684-9ba8-1be06fa4434f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_connectors_for_analysis(languages, support_levels):\n",
    "    return pd.DataFrame([\n",
    "        {\n",
    "            \"dockerfile_variant\": get_dockerfile_variant(c), \n",
    "            \"technical_name\": c.technical_name, \n",
    "            \"support_level\": c.support_level,\n",
    "            \"docker_image\": f'{c.metadata[\"dockerRepository\"]}:{c.metadata[\"dockerImageTag\"]}',\n",
    "            \"dockerfile_path\": Path(c.code_directory / \"Dockerfile\")\n",
    "        }  for c in ALL_CONNECTORS if c.language in languages and c.support_level and c.support_level in support_levels\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "35b8a4b4-e94b-41d2-ac98-b38c2109cb2d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'47 connectors selected for analysis'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SELECTED_SUPPORTS_LEVELS = (\"certified\")\n",
    "SELECTED_LANGUAGES = (ConnectorLanguage.PYTHON, ConnectorLanguage.LOW_CODE)\n",
    "CONNECTORS_FOR_ANALYSIS = get_connectors_for_analysis(SELECTED_LANGUAGES, SELECTED_SUPPORTS_LEVELS)\n",
    "f\"{len(CONNECTORS_FOR_ANALYSIS)} connectors selected for analysis\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6f102d13-8036-4d5a-a1a5-86c272c739e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 14 Dockerfile variants\n"
     ]
    }
   ],
   "source": [
    "DOCKERFILE_VARIANTS = CONNECTORS_FOR_ANALYSIS.groupby(\"dockerfile_variant\").agg(dockerfile_example=(\"dockerfile_path\", \"first\"), docker_image_example=(\"docker_image\", \"first\"), total_count=(\"dockerfile_variant\", \"size\")).sort_values(\"total_count\", ascending=False)\n",
    "print(f\"Found {len(DOCKERFILE_VARIANTS)} Dockerfile variants\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65c20866-f4a5-4f89-b2a2-431538ba3b56",
   "metadata": {},
   "source": [
    "**We have identified **14** variants of Dockerfiles among our certified connectors.**\n",
    "\n",
    "The rest of this analysis aims at:\n",
    "- identifying the differences between these variants\n",
    "- understanding the reason of these differences\n",
    "- deciding whether these differences should be consolidated inside the future python base image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7607cc12-0a69-47d4-93c5-9a1f663294ee",
   "metadata": {},
   "source": [
    "### Showing the **top 3 variants**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ece019db-e075-4505-b6f3-fac393a6ff97",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# -----VARIANT 1-----\n",
      "FROM python:3.9-slim\n",
      "\n",
      "# Bash is installed for more convenient debugging.\n",
      "RUN apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*\n",
      "\n",
      "WORKDIR /airbyte/integration_code\n",
      "COPY source_instagram ./source_instagram\n",
      "COPY main.py ./\n",
      "COPY setup.py ./\n",
      "RUN pip install .\n",
      "\n",
      "ENV AIRBYTE_ENTRYPOINT \"python /airbyte/integration_code/main.py\"\n",
      "ENTRYPOINT [\"python\", \"/airbyte/integration_code/main.py\"]\n",
      "\n",
      "LABEL io.airbyte.version=1.0.11\n",
      "LABEL io.airbyte.name=airbyte/source-instagram\n",
      "\n",
      "# -----VARIANT 2-----\n",
      "FROM python:3.9.11-alpine3.15 as base\n",
      "\n",
      "# build and load all requirements\n",
      "FROM base as builder\n",
      "WORKDIR /airbyte/integration_code\n",
      "\n",
      "# upgrade pip to the latest version\n",
      "RUN apk --no-cache upgrade \\\n",
      "    && pip install --upgrade pip \\\n",
      "    && apk --no-cache add tzdata build-base\n",
      "\n",
      "\n",
      "COPY setup.py ./\n",
      "# install necessary packages to a temporary folder\n",
      "RUN pip install --prefix=/install .\n",
      "\n",
      "# build a clean environment\n",
      "FROM base\n",
      "WORKDIR /airbyte/integration_code\n",
      "\n",
      "# copy all loaded and built libraries to a pure basic image\n",
      "COPY --from=builder /install /usr/local\n",
      "# add default timezone settings\n",
      "COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime\n",
      "RUN echo \"Etc/UTC\" > /etc/timezone\n",
      "\n",
      "# bash is installed for more convenient debugging.\n",
      "RUN apk --no-cache add bash\n",
      "\n",
      "# copy payload code only\n",
      "COPY main.py ./\n",
      "COPY source_notion ./source_notion\n",
      "\n",
      "ENV AIRBYTE_ENTRYPOINT \"python /airbyte/integration_code/main.py\"\n",
      "ENTRYPOINT [\"python\", \"/airbyte/integration_code/main.py\"]\n",
      "\n",
      "LABEL io.airbyte.version=1.1.2\n",
      "LABEL io.airbyte.name=airbyte/source-notion\n",
      "\n",
      "# -----VARIANT 3-----\n",
      "FROM python:3.9-slim\n",
      "\n",
      "# Bash is installed for more convenient debugging.\n",
      "RUN apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*\n",
      "\n",
      "WORKDIR /airbyte/integration_code\n",
      "COPY setup.py ./\n",
      "RUN pip install .\n",
      "COPY source_twilio ./source_twilio\n",
      "COPY main.py ./\n",
      "\n",
      "ENV AIRBYTE_ENTRYPOINT \"python /airbyte/integration_code/main.py\"\n",
      "ENTRYPOINT [\"python\", \"/airbyte/integration_code/main.py\"]\n",
      "\n",
      "LABEL io.airbyte.version=0.10.0\n",
      "LABEL io.airbyte.name=airbyte/source-twilio\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, example in enumerate(DOCKERFILE_VARIANTS[:3][\"dockerfile_example\"]):\n",
    "    print(f\"# -----VARIANT {i + 1}-----\")\n",
    "    print(example.read_text())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b333b42f-33b7-4b0d-bef1-7de78a54649c",
   "metadata": {},
   "source": [
    "### Conclusions after reading the top 3 Dockerfile variants\n",
    "* The base image used are slightly differents: `python:3.9-slim` and `python:3.9.11-alpine3.15`\n",
    "* Variant 1 and 3 are not explicitely setting the system timezone to UTC. Variant 2 is.\n",
    "* Variant 2 is using a multi stage build that can improve build speed and image size\n",
    "* Vairant 1 and 3 are pretty similar but have a different approach at installing the connector python package. Variant 3 copies `setup.py` in a specific layer to cache dependency install. Variant 3 is better than variant 1 for build speed.\n",
    "\n",
    "**Takeaways for building our future base image:**\n",
    "1. We should explicitely set the timezone to UTC on it\n",
    "2. We should leverage dependency install caching with smart docker layering\n",
    "3. The multi-staging operations can be reproduced in Dagger via the creation of multiple containers (one per stage) and exchange of their filesytems with `with_directory` instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae0c665b-9023-452a-858d-602a767d7c2c",
   "metadata": {},
   "source": [
    "## Base images analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2421435d-fb4f-43c1-9360-71be78a665e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'python:3.9-slim',\n",
       " 'python:3.9.11-alpine3.15',\n",
       " 'python:3.9.11-slim',\n",
       " 'python:3.9.16-alpine3.18'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_base_images = {example.read_text().splitlines()[0].replace(\" as base\", \"\").replace(\"FROM \", \"\") for example in DOCKERFILE_VARIANTS[\"dockerfile_example\"]}\n",
    "all_base_images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6e041a7-8377-4e04-bfdb-a6b59adde8d9",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "We are using 4 different Python base image in our certified connectors:\n",
    "* python:3.9-slim\n",
    "* python:3.9.11-slim\n",
    "* python:3.9.11-alpine3.15\n",
    "* python:3.9.16-alpine3.18\n",
    "\n",
    "The most concerning difference is the fact we have `slim` and `alpine` images. \n",
    "The use of the `alpine` images can be explained by the willingness of optimizating the image size.  \n",
    "But some connector depends on Python packages like NumPy that do not work well under `alpine` because of the lack of some system dependencies, hence the use of the `slim` images.\n",
    "\n",
    "**Takeaways for building our future base image:**\n",
    "\n",
    "**I suggest using a broad image like `python:3.9.18-bookworm` as the base of our base image as it will guarantee:**\n",
    "* That we are running the latest python 3.9 version\n",
    "* That we have a maximum of system packages that some connector dependencies might need\n",
    "\n",
    "It is best for maintenance and consolidation,  not optimal for image size: but image size is not something we're currently trying to optimize."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fc14190-eb83-4b06-b124-ecbeba52fdb2",
   "metadata": {},
   "source": [
    "## Environment variables analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16d688ed-d083-4f91-a7eb-b284e67f8163",
   "metadata": {},
   "source": [
    "### Retrieving the env vars set in our 14 variants\n",
    "1. Pull connector image corresponding to each variant\n",
    "2. Call `printenv` on each image\n",
    "3. Gather results in a list of env vars per variant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "518db034-f5de-431c-aa58-b3b48fbce09f",
   "metadata": {},
   "outputs": [],
   "source": [
    "docker_variants_env_vars = []\n",
    "async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as dagger_client:\n",
    "    for connector_image in DOCKERFILE_VARIANTS[\"docker_image_example\"]:\n",
    "        printenv_output = await dagger_client.container().from_(connector_image).with_exec([\"printenv\"], skip_entrypoint=True).stdout()\n",
    "        env_vars = {}\n",
    "        for env_var in printenv_output.splitlines():\n",
    "            k, v = env_var.split(\"=\")\n",
    "            env_vars[k] = v\n",
    "        docker_variants_env_vars.append(env_vars)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdaf2d1d-dae9-4892-879c-24fdc397dcf6",
   "metadata": {},
   "source": [
    "### Environment variables with values common to all images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6edea0ab-ff96-47d2-9463-85fc28900b81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Env var name</th>\n",
       "      <th>Env var value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>LANG</td>\n",
       "      <td>C.UTF-8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>PYTHON_SETUPTOOLS_VERSION</td>\n",
       "      <td>58.1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HOME</td>\n",
       "      <td>/root</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PATH</td>\n",
       "      <td>/usr/local/bin:/usr/local/sbin:/usr/local/bin:...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>OTEL_TRACES_EXPORTER</td>\n",
       "      <td>otlp</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>GPG_KEY</td>\n",
       "      <td>E3FF2839C048B25C084DEBE9B26995E310250568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>OTEL_EXPORTER_OTLP_TRACES_ENDPOINT</td>\n",
       "      <td>unix:///dev/otel-grpc.sock</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>OTEL_EXPORTER_OTLP_TRACES_PROTOCOL</td>\n",
       "      <td>grpc</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         Env var name  \\\n",
       "0                                LANG   \n",
       "1           PYTHON_SETUPTOOLS_VERSION   \n",
       "2                                HOME   \n",
       "3                                PATH   \n",
       "4                OTEL_TRACES_EXPORTER   \n",
       "5                             GPG_KEY   \n",
       "6  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT   \n",
       "7  OTEL_EXPORTER_OTLP_TRACES_PROTOCOL   \n",
       "\n",
       "                                       Env var value  \n",
       "0                                            C.UTF-8  \n",
       "1                                             58.1.0  \n",
       "2                                              /root  \n",
       "3  /usr/local/bin:/usr/local/sbin:/usr/local/bin:...  \n",
       "4                                               otlp  \n",
       "5           E3FF2839C048B25C084DEBE9B26995E310250568  \n",
       "6                         unix:///dev/otel-grpc.sock  \n",
       "7                                               grpc  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_common_env_vars_values(env_vars_per_variant):\n",
    "    tuplized_kv = [{(k,v) for k, v in variant.items()} for variant in env_vars_per_variant]\n",
    "    common_env_var_values = tuplized_kv[0]\n",
    "\n",
    "    for env_var_set in tuplized_kv[1:]:\n",
    "        common_env_var_values = common_env_var_values.intersection(env_var_set)\n",
    "    return pd.DataFrame(common_env_var_values, columns=[\"Env var name\", \"Env var value\"])\n",
    "\n",
    "    \n",
    "common_env_var_values = get_common_env_vars_values(docker_variants_env_vars)\n",
    "common_env_var_values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98bb14a6-4c4d-4f40-8688-f0dede3870c8",
   "metadata": {},
   "source": [
    "**Takeaways for building our future base image:**\n",
    "\n",
    "This list of static env var is interesting to know as it will allow us to write tests on our future connector images that will check that:\n",
    "- these env var are set\n",
    "- these env var values are always with the same values\n",
    "\n",
    "These env vars are not set on our Dockerfiles and are  coming from the Python base image: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e4b34f77-ef7a-4d5a-b441-bc5a27f656de",
   "metadata": {},
   "outputs": [],
   "source": [
    "async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as dagger_client:\n",
    "    for base_image in all_base_images:\n",
    "        base_image_container = dagger_client.container().from_(connector_image)\n",
    "        base_env_vars_raw = (await base_image_container.with_exec([\"printenv\"], skip_entrypoint=True).stdout()).splitlines()\n",
    "        base_env_vars = {}\n",
    "        for base_env_var in base_env_vars_raw:\n",
    "            k, v = base_env_var.split(\"=\")\n",
    "            base_env_vars[k] = v\n",
    "        for _, common_env_var in common_env_var_values.iterrows():\n",
    "            assert base_env_vars[common_env_var[\"Env var name\"]] == common_env_var[\"Env var value\"]\n",
    "   \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc76a235-de65-4e6d-81b7-a0dcdcff6d15",
   "metadata": {},
   "source": [
    "### Environment variables common to all images but with different values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "faa30644-d9ec-4710-913c-31fe57e1d7de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AIRBYTE_ENTRYPOINT',\n",
       " 'OTEL_TRACE_PARENT',\n",
       " 'PYTHON_GET_PIP_SHA256',\n",
       " 'PYTHON_GET_PIP_URL',\n",
       " 'PYTHON_PIP_VERSION',\n",
       " 'PYTHON_VERSION',\n",
       " 'TRACEPARENT'}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_common_env_vars(env_vars_per_variant):\n",
    "    env_var_keys_per_variant = [set(variant.keys()) for variant in env_vars_per_variant]\n",
    "    common_env_var = env_var_keys_per_variant[0]\n",
    "\n",
    "    for env_var_set in env_var_keys_per_variant[1:]:\n",
    "        common_env_var = common_env_var.intersection(env_var_set)\n",
    "    return pd.Series(list(common_env_var))\n",
    "\n",
    "    \n",
    "common_env_var = get_common_env_vars(docker_variants_env_vars)\n",
    "common_env_var_key_different_values = set(common_env_var) - set(common_env_var_values[\"Env var name\"])\n",
    "common_env_var_key_different_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "cd4bacc4-663c-4c22-9f81-c060fe636c2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_unique_values_for_env_vars(docker_variants_env_vars):\n",
    "    unique_values_for_env_vars = {}\n",
    "    for docker_variant_env_vars in docker_variants_env_vars:\n",
    "        for k, v in docker_variant_env_vars.items():\n",
    "            if k not in unique_values_for_env_vars:\n",
    "                unique_values_for_env_vars[k] = set()\n",
    "            unique_values_for_env_vars[k].add(v)\n",
    "    return unique_values_for_env_vars"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6fcd032-ac65-4b93-971f-a549589b4874",
   "metadata": {},
   "source": [
    "#### Differences in `AIRBYTE_ENTRYPOINT`\n",
    "Differences in `AIRBYTE_ENTRYPOINT` are not expected. We want all python connector to have it set to `python /airbyte/integration_code/main.py`.  \n",
    "\n",
    "A single connector, `source-zendesk-chat`, has its entrypoint set to `python /airbyte/integration_code/main_dev.py`. It looks like a legacy thing that we should correct. But as the base image project is not targetted at fixing connectors we'll handle this custom env var with a post build hook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a663a427-b616-47b8-828e-bf23883cdf7f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'python /airbyte/integration_code/main.py',\n",
       " 'python /airbyte/integration_code/main_dev.py'}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_unique_values_for_env_vars(docker_variants_env_vars)[\"AIRBYTE_ENTRYPOINT\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6a6cdd4-44f3-4bb2-a7dc-4b5c1f526e6e",
   "metadata": {},
   "source": [
    "#### Differences in `PYTHON_VERSION`\n",
    "All certified connectors are running Python 3.9 but with subtle differences in the patch version:\n",
    "* 3.9.18\n",
    "* 3.9.17\n",
    "* 3.9.16\n",
    "* 3.9.11\n",
    "\n",
    "This can be explained the use of `python:3.9-slim` base image tag. This tag is updated to the latest python 3.9 version when a new version get released. Some connectors that had not been built for a while can stay behind the latest Python 3.9 version until they're not rebuilt.\n",
    "Another reason for this difference is that some connector use `python:3.9.11-slim` which is pinning Python 3.9.11.\n",
    "For **reproductible built** our base image will be based on a `sha256` docker tag, it will make sure we're always using the same image as a base."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "8ee5effc-d611-4a47-9ef4-cb3309b21c5c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Different Python version in use: 3.9.17, 3.9.11, 3.9.16\n"
     ]
    }
   ],
   "source": [
    "different_python_version = get_unique_values_for_env_vars(docker_variants_env_vars)[\"PYTHON_VERSION\"]\n",
    "print(f\"Different Python version in use: {', '.join(different_python_version)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c74e5882-e382-4864-87f6-4ff5730bf536",
   "metadata": {},
   "source": [
    "#### Differences in `PYTHON_PIP_VERSION`\n",
    "Two different `pip` version are in used in our connectors:\n",
    "* 23.0.1\n",
    "* 22.0.4\n",
    "\n",
    "This can be explained by:\n",
    "* The use of different base images that might bundle different pip version\n",
    "* The `pip install --upgrade pip` instruction in our Dockerfile that might upgrade `pip` on rebuild of the image\n",
    "\n",
    "**Takeaways for building our future base image:**\n",
    "For reproductible build we likely want to pin the pip version. \n",
    "This will be naturaly achieved by using a static base python image which comes with pip pre-installed.\n",
    "If we want to use a custom pip version we should pin it by running `pip install pip==<pip-version>` in our future base image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8be067f6-c553-43fd-8ecf-edbc346d5158",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Different Python Pip version in use: 22.0.4, 23.0.1\n"
     ]
    }
   ],
   "source": [
    "different_pip_version = get_unique_values_for_env_vars(docker_variants_env_vars)[\"PYTHON_PIP_VERSION\"]\n",
    "print(f\"Different Python Pip version in use: {', '.join(different_pip_version)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73ca52fa-77d5-4034-ae98-ff2b37ca1aa2",
   "metadata": {},
   "source": [
    "#### Differences in `PYTHON_GET_PIP_URL`, `PYTHON_GET_PIP_SHA256`\n",
    "We have 4 different value for these env vars.\n",
    "This matches the 4 different Python images we use:\n",
    "These env var are probably set at build time of the python base images and match a version of the `pip` installation script.\n",
    "\n",
    "**Takeaways for building our future base image:**\n",
    "As we'll use a single base image for all certified connectors, this base image will be built on a base python image: all our connector will have the same value for these env vars. No action needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "407af89a-17d1-4631-bf8c-79425e5a4b1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'394be00f13fa1b9aaa47e911bdb59a09c3b2986472130f30aa0bfaf7f3980637',\n",
       " '45a2bb8bf2bb5eff16fdd00faef6f29731831c7c59bd9fc2bf1f3bed511ff1fe',\n",
       " '96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207',\n",
       " 'e235c437e5c7d7524fbce3880ca39b917a73dc565e0c813465b7a7a329bb279a'}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_unique_values_for_env_vars(docker_variants_env_vars)[\"PYTHON_GET_PIP_SHA256\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "a2935d78-77f9-47f7-9349-1a36c9e10008",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py',\n",
       " 'https://github.com/pypa/get-pip/raw/38e54e5de07c66e875c11a1ebbdb938854625dd8/public/get-pip.py',\n",
       " 'https://github.com/pypa/get-pip/raw/9af82b715db434abb94a0a6f3569f43e72157346/public/get-pip.py',\n",
       " 'https://github.com/pypa/get-pip/raw/d5cb0afaf23b8520f1bbcfed521017b4a95f5c01/public/get-pip.py'}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_unique_values_for_env_vars(docker_variants_env_vars)[\"PYTHON_GET_PIP_URL\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "329c49af-ff03-445e-9e89-137e20944020",
   "metadata": {},
   "source": [
    "#### Differences in `TRACEPARENT` and `OTEL_TRACE_PARENT`\n",
    "On these env var we have a different value for each connector image.\n",
    "I believe these are set at build time by the Docker engine.\n",
    "These Ids can be used for tracing. I don't know if they are actively use in our current infrastructure.\n",
    "\n",
    "**Takeaways**:\n",
    "On connector build in `airbyte-ci` we should verify that these env var are set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "64a1410e-61fe-40e3-b4b0-9a45a9b8d660",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'00-05babbaa467d2b7101a1c3c4b75c1b5f-dd8adab127d27f39-01',\n",
       " '00-15262d0156d304755c1a37aab7400d91-d000781658986ada-01',\n",
       " '00-492c7d03978cbd27941a0f01c64d0ea2-53174fa4a8f396d5-01',\n",
       " '00-4cfe6e40354d22eef1c8bd44a53de77b-c791eac4904fd1f2-01',\n",
       " '00-5a8d6c04188be1aebce946a1ef5e61a2-52f2d30a2ae782ed-01',\n",
       " '00-624b92c81c4de6a0f0f8e7bdee0e3ba3-0ab1ead787e1426b-01',\n",
       " '00-659529510ea31fffdb43e578b7a61e85-3274b696e496d0d2-01',\n",
       " '00-7cd60817e3b3542b6bd91c22098ff33a-03a4d5e08021248d-01',\n",
       " '00-9d69486c8f84d271449e4885eec67943-7c2a3f817dbca68b-01',\n",
       " '00-a215dd2bf43039a9f04c28219a7cb5e9-5c12bf9a7651cfba-01',\n",
       " '00-af060a2eb1bd83b31d40d0da6232e6e1-d1e86c8724866a37-01',\n",
       " '00-caf0d2479373ae86133704e4e2f19e73-3536098ec356a6c0-01',\n",
       " '00-fca418ed06415f3741ff550a232cf257-0216c5a3245cbf48-01',\n",
       " '00-fff1fd9d58a2ba8c407a0ee94c965aa5-eafd1f6498b40fac-01'}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_unique_values_for_env_vars(docker_variants_env_vars)[\"TRACEPARENT\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "2053b43d-0e7a-473b-8550-a6d41bac2354",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'00-05babbaa467d2b7101a1c3c4b75c1b5f-dd8adab127d27f39-01',\n",
       " '00-15262d0156d304755c1a37aab7400d91-d000781658986ada-01',\n",
       " '00-492c7d03978cbd27941a0f01c64d0ea2-53174fa4a8f396d5-01',\n",
       " '00-4cfe6e40354d22eef1c8bd44a53de77b-c791eac4904fd1f2-01',\n",
       " '00-5a8d6c04188be1aebce946a1ef5e61a2-52f2d30a2ae782ed-01',\n",
       " '00-624b92c81c4de6a0f0f8e7bdee0e3ba3-0ab1ead787e1426b-01',\n",
       " '00-659529510ea31fffdb43e578b7a61e85-3274b696e496d0d2-01',\n",
       " '00-7cd60817e3b3542b6bd91c22098ff33a-03a4d5e08021248d-01',\n",
       " '00-9d69486c8f84d271449e4885eec67943-7c2a3f817dbca68b-01',\n",
       " '00-a215dd2bf43039a9f04c28219a7cb5e9-5c12bf9a7651cfba-01',\n",
       " '00-af060a2eb1bd83b31d40d0da6232e6e1-d1e86c8724866a37-01',\n",
       " '00-caf0d2479373ae86133704e4e2f19e73-3536098ec356a6c0-01',\n",
       " '00-fca418ed06415f3741ff550a232cf257-0216c5a3245cbf48-01',\n",
       " '00-fff1fd9d58a2ba8c407a0ee94c965aa5-eafd1f6498b40fac-01'}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_unique_values_for_env_vars(docker_variants_env_vars)[\"OTEL_TRACE_PARENT\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2d2a4d7-f896-486f-9de1-f873807084a7",
   "metadata": {},
   "source": [
    "### Connector specific environment variables\n",
    "Some connector are setting custom environment variables.\n",
    "#### `AIRBYTE_IMPL_MODULE` and `AIRBYTE_IMPL_PATH` \n",
    "These variables are [used in the CDK](https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/entrypoint.py#L256) to override some default behavior in the connector entrypoint.  `source-zendesk-chat` is setting these env var. I can't assess if it's rightfully doing so. We'll support this by custom pre/post build hooks for this connector.\n",
    "\n",
    "#### `CODE_PATH` and `WORKDIR`\n",
    "`source-slack` is setting these two env vars. These are only used in the Dockerfile context and can likely be discarded as this logic will be declared with the common build process in `airbyte-ci`. :\n",
    "```Dockerfile\n",
    "ENV WORKDIR=/airbyte/integration_code\n",
    "\n",
    "WORKDIR $WORKDIR\n",
    "\n",
    "COPY setup.py ./\n",
    "RUN pip install .\n",
    "\n",
    "COPY $CODE_PATH ./$CODE_PATH\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "4bc77b08-36fe-4f6d-afb5-ddc64449a2c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connector specific env var: CODE_PATH=source_slack\n",
      "Connector specific env var: WORKDIR=/airbyte/integration_code\n",
      "Connector specific env var: CODE_PATH=source_zendesk_chat\n",
      "Connector specific env var: AIRBYTE_IMPL_MODULE=source_zendesk_chat\n",
      "Connector specific env var: AIRBYTE_IMPL_PATH=SourceZendeskChat\n"
     ]
    }
   ],
   "source": [
    "for variant_env_vars in docker_variants_env_vars:\n",
    "    for k, v in variant_env_vars.items():\n",
    "        if k not in set(common_env_var):\n",
    "            print(f\"Connector specific env var: {k}={v}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91cac85b-4ea6-4d1f-a118-638917a43078",
   "metadata": {},
   "source": [
    "## Custom system dependencies analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5ce85683-ee15-4dc7-9dac-23182e6c2942",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "airbyte/source-instagram:1.0.11 is installing custom system packages: libcrypt-dev:arm64, libtirpc-dev:arm64, libc6-dev:arm64, rpcsvc-proto, libnsl-dev:arm64, linux-libc-dev:arm64, libc-dev-bin\n",
      "airbyte/source-twilio:0.10.0 is installing custom system packages: libcrypt-dev:arm64, libtirpc-dev:arm64, libc6-dev:arm64, rpcsvc-proto, libnsl-dev:arm64, linux-libc-dev:arm64, libc-dev-bin\n",
      "airbyte/source-gitlab:1.6.0 is installing custom system packages: libcrypt-dev:arm64, libtirpc-dev:arm64, libc6-dev:arm64, rpcsvc-proto, libnsl-dev:arm64, linux-libc-dev:arm64, libc-dev-bin\n",
      "airbyte/source-google-analytics-v4:0.2.1 is installing custom system packages: libcrypt-dev:arm64, libtirpc-dev:arm64, libc6-dev:arm64, rpcsvc-proto, libnsl-dev:arm64, linux-libc-dev:arm64, libc-dev-bin\n",
      "airbyte/source-salesforce:2.1.4 is installing custom system packages: libcrypt-dev:arm64, libtirpc-dev:arm64, libc6-dev:arm64, rpcsvc-proto, libnsl-dev:arm64, linux-libc-dev:arm64, libc-dev-bin\n"
     ]
    }
   ],
   "source": [
    "\n",
    "async def get_installed_system_packages(container, dagger_client):\n",
    "    try:\n",
    "        output = await container.with_exec([\"dpkg\", \"--get-selections\"], skip_entrypoint=True).stdout()\n",
    "    except dagger.ExecError:\n",
    "        # Use apk info for alpine\n",
    "        output =  await container.with_exec([\"apk\", \"info\", \"-q\"], skip_entrypoint=True).stdout()\n",
    "    return set(output.replace(\"\\t\", \"\").replace(\"install\", \"\").splitlines())\n",
    "\n",
    "docker_variants_env_vars = []\n",
    "base_pkgs = set()\n",
    "images_with_custom_pkgs = set()\n",
    "async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as dagger_client:\n",
    "    for base_image in all_base_images:\n",
    "        base_image_container = dagger_client.container().from_(base_image)\n",
    "        base_pkgs.update(await get_installed_system_packages(base_image_container, dagger_client))\n",
    "    for connector_image in DOCKERFILE_VARIANTS[\"docker_image_example\"]:\n",
    "        connector_image_container = dagger_client.container().from_(connector_image)\n",
    "        connector_image_pkgs = await get_installed_system_packages(connector_image_container, dagger_client)\n",
    "        custom_pkgs = connector_image_pkgs - base_pkgs\n",
    "        if custom_pkgs:\n",
    "            print(f\"{connector_image} is installing custom system packages: {', '.join(custom_pkgs)}\")\n",
    "            images_with_custom_pkgs.add(connector_image)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0fc5557-06e1-4718-94c2-c25ba1c56c4d",
   "metadata": {},
   "source": [
    "**The images listed above have extra system packages compared to there base image. But no instruction in the dockerfile installs these system packages. My current hypothesis is that the base image they use had these system packages in previous versions. Let's rebuild these connector and see if the \"custom packages\" are still there.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "a447c6c7-8c83-479b-98ef-7776219bb407",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No custom packages found after local build of source-instagram\n",
      "No custom packages found after local build of source-twilio\n",
      "No custom packages found after local build of source-gitlab\n",
      "No custom packages found after local build of source-google-analytics-v4\n",
      "No custom packages found after local build of source-salesforce\n"
     ]
    }
   ],
   "source": [
    "async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as dagger_client:\n",
    "    for connector_image in images_with_custom_pkgs:\n",
    "        connector = Connector(connector_image.split(\":\")[0].replace(\"airbyte/\", \"\"))\n",
    "        connector_container = dagger_client.host().directory(str(connector.code_directory)).docker_build()\n",
    "        connector_container_pkgs = await get_installed_system_packages(connector_container, dagger_client)\n",
    "        custom_pkgs = connector_container_pkgs - base_pkgs\n",
    "        assert not custom_pkgs\n",
    "        print(f\"No custom packages found after local build of {connector}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7df43143-a673-45b1-bafb-cc0578bfec98",
   "metadata": {},
   "source": [
    "#### Takeaways for the future base image\n",
    "\n",
    "1. We don't have any certified python connector that is installing a custom system dependency\n",
    "2. The use of base images tag that can change the underlying image in use (like python3.9-slim: it might get updated on each patch version of python 3.9) proves again that we can't achieve reproductible build with these images. We should definitely be targetting tag with their `sha256`: e.g `python:3.9@sha256:0596c508fdfdf28fd3b98e170f7e3d4708d01df6e6d4bffa981fd6dd22dbd1a5`. This will ensure that on rebuild the same base image will be used.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "801fc0c6-a8ab-45fc-bae3-6ed44fdb2b6c",
   "metadata": {},
   "source": [
    "# Overall analysis conclusion\n",
    "### Base images to use for our base image:\n",
    "We want to use a python image with 3.9 in its latest patch version: 3.9.18\n",
    "We want to use debian as it has proven its good fit for our existing connector image.\n",
    "We want to use a debian image that has a maximum of system package to avoid connector specific package install: let's use `bookworm`, the latest debian version release name.\n",
    "[python:3.9.18-bookworm](https://hub.docker.com/layers/library/python/3.9.18-bookworm/images/sha256-40582fe697811beb7bfceef2087416336faa990fd7e24984a7c18a86d3423d58?context=explore)\n",
    "- For AMD64: `python:3.9.18-bookworm@sha256:40582fe697811beb7bfceef2087416336faa990fd7e24984a7c18a86d3423d58`\n",
    "- For AMR64: `python:3.9.18-bookworm@sha256:0d132e30eb9325d53c790738e5478e9abffc98b69115e7de429d7c6fc52dddac`\n",
    "\n",
    "### System settings to set on our base image:\n",
    "We must set the timezone to UTC\n",
    "\n",
    "### Environment variable that must exists on our connector images\n",
    "#### Common env vars with static values (check the specific section for values)\n",
    "- `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`\n",
    "- `GPG_KEY`\n",
    "- `LANG`\n",
    "- `PATH`\n",
    "- `PYTHON_SETUPTOOLS_VERSION`\n",
    "- `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`\n",
    "- `OTEL_TRACES_EXPORTER`\n",
    "- `HOME`\n",
    "- `PYTHON_VERSION`\n",
    "- `PIP_VERSION`\n",
    "- `PYTHON_GET_PIP_SHA256`\n",
    "- `PYTHON_GET_PIP_URL`\n",
    "  \n",
    "This env vars will likely be set by the python base image we'll use. But we must add a test that ensure they are set, to avoid any regression on new connector versions.\n",
    "\n",
    "#### Common env vars with custom values\n",
    "- `AIRBYTE_ENTRYPOINT`: this should default to `python /airbyte/integration_code/main.py` but `source-zendesk-chat` has set it to `python /airbyte/integration_code/main_dev.py` (we should handle this edge case with pre/post build hook).\n",
    "- `OTEL_TRACE_PARENT` and `TRACEPARENT`: We believe these are set at build time by the docker engine, but we should make sure our dagger build still set these (or understand there purpose)\n",
    "\n",
    "#### Custom env vars\n",
    "- `AIRBYTE_IMPL_MODULE` and `AIRBYTE_IMPL_PATH`: These are set by `source-zendesk-chat`, we must implement a pre/post build hook for this connector to set these env var at build time.\n",
    "\n",
    "### System packages dependencies\n",
    "No certified connector is installing a custom system dependency.\n",
    "The system packages bundled with the python base image I suggest to use should guarantee all system requirements are met for all our connectors.\n",
    "\n",
    "### Labels\n",
    "All our connector images set the following labels. We should continue to use these, but I'm not sure of their actual usefulness.\n",
    "```Dockerfile\n",
    "LABEL io.airbyte.version=<semver-connector-version>\n",
    "LABEL io.airbyte.name=airbyte/<connector-technical-name>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baa39bb0-1292-4f61-a8e4-127a1c5a71bf",
   "metadata": {},
   "source": [
    "# Next steps\n",
    "1. Declare a first base image version in a new python package under `airbyte-ci/connectors/base_images/`\n",
    "2. Declare a common build procedure for connector images in `airbyte-ci/connectors/pipelines`\n",
    "3. Implement a light pre/build hook framework to customize connector build process if needed\n",
    "\n",
    "A \"prototype\" of these steps is available in [this PR](https://github.com/airbytehq/airbyte/pull/29477/files). This PR has not been yet updated to capture the conclusion of this analysis."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}