Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[airbyte-ci] introduce an autocommit command to update base image version to the latest one #30508

Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
bcfd036
connector base image: declare the base image package and implement build
alafanechere Sep 11, 2023
d63c4e3
mypy fixes
alafanechere Sep 11, 2023
624f6e5
Update airbyte-ci/connectors/base_images/base_images/common.py
alafanechere Sep 11, 2023
b4f06da
Update airbyte-ci/connectors/base_images/base_images/common.py
alafanechere Sep 11, 2023
5aa738c
Apply suggestions from code review
alafanechere Sep 11, 2023
5b8d355
Update airbyte-ci/connectors/base_images/base_images/python_bases.py
alafanechere Sep 11, 2023
1c38237
Update airbyte-ci/connectors/base_images/base_images/python_bases.py
alafanechere Sep 11, 2023
324839c
improve docs
alafanechere Sep 12, 2023
f47418e
handle keyboard interrupt
alafanechere Sep 12, 2023
447141c
make sanity checks statics
alafanechere Sep 12, 2023
1c41c18
make sanity checks statics
alafanechere Sep 12, 2023
ec8ced2
make the first version be 1.0.0
alafanechere Sep 12, 2023
b276006
add a version 1.0.1: install poetry
alafanechere Sep 12, 2023
df4bfeb
add a version 1.1.0: install poetry
alafanechere Sep 12, 2023
1a16818
generate dockerfiles
alafanechere Sep 12, 2023
d4e5ad3
ref: improve the package structure
alafanechere Sep 12, 2023
46c23bc
add headers to dockerfile
alafanechere Sep 12, 2023
ee7f7ea
update documentation
alafanechere Sep 12, 2023
242961f
fix comments
alafanechere Sep 12, 2023
d8ca525
fix comments
alafanechere Sep 12, 2023
4b32768
add missing newlines to dockerfiles
alafanechere Sep 12, 2023
a6d2aab
generate platform aware container
alafanechere Sep 12, 2023
765b2bd
fix existing tests
alafanechere Sep 12, 2023
85a282f
implement suggestion: no inter version inheritance
alafanechere Sep 12, 2023
18b6c4e
finish suggestion implementation
alafanechere Sep 13, 2023
222842a
update README
alafanechere Sep 13, 2023
e0267c6
fix package path
alafanechere Sep 13, 2023
8b3fc5c
declare a custom pre-commit-hook
alafanechere Sep 13, 2023
20826a9
update README with pre-commit instructions
alafanechere Sep 13, 2023
de7635e
clear generated dockerfile directory on new generation
alafanechere Sep 13, 2023
5252eca
ref: introduce a registry object
alafanechere Sep 14, 2023
76b9dcb
update readme path
alafanechere Sep 14, 2023
73a8708
update readme
alafanechere Sep 14, 2023
8e74992
use slim bookworm base
alafanechere Sep 14, 2023
5219d49
remove github_url
alafanechere Sep 14, 2023
31afb94
declare linux/x86_64 platform
alafanechere Sep 15, 2023
9555dfc
change our python connector build process to use the base images
alafanechere Sep 14, 2023
83aaaed
change our python connector build process to use the base images
alafanechere Sep 14, 2023
894e0b9
revert changes on source-faker
alafanechere Sep 17, 2023
9427b97
[airbyte-ci] Implement pre/post build hooks
alafanechere Sep 18, 2023
01df360
remove dockerfile on source-zendesk-caht
alafanechere Sep 18, 2023
db634ba
test build customization
alafanechere Sep 18, 2023
111cc19
[airbyte-ci] test connector inside their built container
alafanechere Sep 15, 2023
8b07e16
fix imports / relocate PytestStep to tests/python_connectors.py
alafanechere Sep 15, 2023
3fffee5
fix tests
alafanechere Sep 15, 2023
9029852
make QA checks tolerate connector without dockerfile
alafanechere Sep 15, 2023
c7d1e73
change our python connector build process to use the base images
alafanechere Sep 14, 2023
4e8a15d
change our python connector build process to use the base images
alafanechere Sep 14, 2023
4a191c4
[airbyte-ci] introduce an autocommit command to update base image ver…
alafanechere Sep 17, 2023
68f468d
run autocommit before tests
alafanechere Sep 17, 2023
4dec040
fix attribute access
alafanechere Sep 17, 2023
9bed306
remove baseImage in source-faker
alafanechere Sep 17, 2023
e35fb39
do not push empty commits
alafanechere Sep 17, 2023
fc3123b
set baseImage on source faker
alafanechere Sep 17, 2023
138de7e
[skip ci]: 🤖 Autocommit on source-faker - Update base image to latest…
octavia-squidington-iii Sep 17, 2023
7f3ec47
reword step name in connector test workflow
alafanechere Sep 17, 2023
adf5144
Automated Commit - Format and Process Resources Changes
alafanechere Sep 17, 2023
23aeb0c
change our python connector build process to use the base images
alafanechere Sep 14, 2023
82ea628
change our python connector build process to use the base images
alafanechere Sep 14, 2023
9de271e
[airbyte-ci] introduce an autocommit command to update base image ver…
alafanechere Sep 17, 2023
0207639
change our python connector build process to use the base images
alafanechere Sep 14, 2023
8fe7e00
change our python connector build process to use the base images
alafanechere Sep 14, 2023
ebbc948
ref: make format use the new base class
alafanechere Sep 17, 2023
9536d3f
implement version bump comment
alafanechere Sep 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
improve docs
  • Loading branch information
alafanechere committed Sep 18, 2023
commit 324839cf283242bae0034f1ac5ac3ca7c598f190
2 changes: 1 addition & 1 deletion airbyte-ci/connectors/base_images/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ class _0_1_1(_0_1_0):
try:
await self.container.with_exec(["ffmpeg", "-version"], skip_entrypoint=True).stdout()
except dagger.ExecError as e:
raise common.SanityCheckError("failed to run ffmpeg --version.") from e
raise errors.SanityCheckError("failed to run ffmpeg --version.") from e
```

## How to update an existing base image version
Expand Down
29 changes: 22 additions & 7 deletions airbyte-ci/connectors/base_images/base_images/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,22 @@
import sys
from itertools import product
from pathlib import Path
from typing import List

import anyio
import dagger
from base_images import common, consts, utils
from base_images import consts, errors, utils
from rich.console import Console
from rich.status import Status

console = Console()

try:
from base_images import python_bases # , java_bases
except common.BaseImageVersionError as e:
except errors.BaseImageVersionError as e:
# This error occurs if a base image version class name does not follow semver.
# We handle the error for nice console output.
# It might happen if a developer implement a new version class without following our required class name convention.
console.log(f":cross_mark: {e}", style="bold red")
sys.exit(1)

Expand All @@ -25,8 +29,10 @@
async def run_all_sanity_checks(status: Status) -> bool:
"""
Runs sanity checks on all the base images.
Sanity checks are declared in the base image version classes by implementing the run_sanity_checks function.
Sanity checks are command executed on the base image container, we check the output of these command to make sure the base image is working as expected.
"""
errors = []
sanity_check_errors: List[errors.SanityCheckError] = []
async with dagger.Connection(dagger.Config()) as dagger_client:
for platform, BaseImageVersion in product(consts.SUPPORTED_PLATFORMS, ALL_BASE_IMAGES.values()):
status.update(f":mag_right: Running sanity checks on {BaseImageVersion.name_with_tag} for {platform}")
Expand All @@ -35,20 +41,29 @@ async def run_all_sanity_checks(status: Status) -> bool:
console.log(
f":white_check_mark: Successfully ran sanity check on {BaseImageVersion.name_with_tag} for {platform}", highlight=False
)
except common.SanityCheckError as sanity_check_error:
except errors.SanityCheckError as sanity_check_error:
console.log(
f":cross_mark: Sanity check failure on {BaseImageVersion.name_with_tag} for {platform}: {sanity_check_error}",
style="bold red",
highlight=False,
)
errors.append(sanity_check_error)
sanity_check_errors.append(sanity_check_error)

return not errors
return not sanity_check_errors


def build():
"""
Runs sanity checks on all the base images and writes the changelog.
This function is called by the build command, currently via poetry run build.
It's currently meant to be run locally by developers to generate the changelog and run sanity checks.
It can eventually be run in CI to generate the changelog and run sanity checks.

1. Run sanity checks on all the base images.
2. Write the changelog for the python base image.

This function calls Dagger to run the sanity checks.
If you don't have the base base image locally it will be pulled, which can take a while.
Subsequent runs will be faster as the base images layers and sanity checks layers will be cached locally.
"""
with console.status("Building the project", spinner="hamburger") as status:
status.update("Running sanity checks on all the base images")
Expand Down
87 changes: 55 additions & 32 deletions airbyte-ci/connectors/base_images/base_images/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#

"""This module declares common (abstract) classes and methods used by all base images.
It's not meant to be regurlarly modified.
"""

import inspect
from abc import ABC, abstractmethod
from dataclasses import dataclass
Expand All @@ -11,7 +15,7 @@

import dagger
import semver
from base_images import consts
from base_images import consts, errors


@dataclass
Expand All @@ -25,24 +29,6 @@ def get_full_image_name(self) -> str:
return f"{self.image_name}:{self.tag}@sha256:{self.sha}"


class BaseImageVersionError(ValueError):
"""Raised when the version is not in the expected format."""

pass


class SanityCheckError(Exception):
"""Raised when a sanity check fails."""

pass


class PlatformAvailabilityError(ValueError):
"""Raised when the platform is not supported by an image."""

pass


class BaseBaseImage(Enum):
pass

Expand All @@ -59,19 +45,42 @@ class AirbyteConnectorBaseImage(ABC):
@property
@abstractmethod
def base_base_image(cls) -> BaseBaseImage:
"""Returns the base image used to build the Airbyte base image."""
"""Returns the base image used to build the Airbyte base image.

Raises:
NotImplementedError: Raised if a subclass does not define a 'base_base_image' attribute.

Returns:
BaseBaseImage: The base image used to build the Airbyte base image.
"""
raise NotImplementedError("Subclasses must define a 'base_base_image'.")

@property
@abstractmethod
def image_name(cls) -> str:
"""This is the name of the final base image."""
"""This is the name of the final base image. By name we mean DockerHub image name without the tag.

Raises:
NotImplementedError: Raised if a subclass does not define an 'image_name' attribute.

Returns:
str: The name of the final base image.
"""
raise NotImplementedError("Subclasses must define an 'image_name'.")

@property
@abstractmethod
def changelog(cls) -> str:
raise NotImplementedError("Subclasses must define a 'changelog' attribute.")
def changelog_entry(cls) -> str:
"""This is the changelog entry for a new base image version.
It will automatically be used to generate the changelog entry for the release notes.

Raises:
NotImplementedError: Raised if a subclass does not define a 'changelog_entry' attribute.

Returns:
str: The changelog entry for a new base image version.
"""
raise NotImplementedError("Subclasses must define a 'changelog_entry' attribute.")

@final
def __init__(self, dagger_client: dagger.Client, platform: dagger.Platform):
Expand Down Expand Up @@ -109,7 +118,9 @@ def _validate_version(cls):
try:
semver.VersionInfo.parse(cls.version)
except ValueError as e:
raise BaseImageVersionError(f"The version class {cls.__name__} is not in the expected semantic versioning naming format: e.g `_0_1_0`.") from e
raise errors.BaseImageVersionError(
f"The version class {cls.__name__} is not in the expected semantic versioning naming format: e.g `_0_1_0`."
) from e

@final
def _validate_platform_availability(self):
Expand All @@ -119,19 +130,24 @@ def _validate_platform_availability(self):
ValueError: Raised if the platform is not supported by the base image.
"""
if self.platform not in self.base_base_image.value:
raise PlatformAvailabilityError(f"Platform {self.platform} is not supported by {self.base_base_image.name}.")
raise errors.PlatformAvailabilityError(f"Platform {self.platform} is not supported by {self.base_base_image.name}.")

@final
@property
def base_base_image_name(self) -> str:
"""Returns the full name of the base's base image used to build the Airbyte base image.
In this context the base's base image name contains the tag.
Returns:
str: The full name of the base's base image used to build the Airbyte base image, with its tag.
"""
return self.base_base_image.value[self.platform].get_full_image_name()

@property
@final
def base_container(self) -> dagger.Container:
"""Returns a container using the base python image. This container is used to build the Airbyte base image.
We set environment variables and labels to ensure we can easily check at post build time:
- the Python base image that was used to build the Airbyte base image
- the base image that was used to build the Airbyte base image
- the version of the Airbyte base image

Returns:
Expand All @@ -155,22 +171,29 @@ def container(self) -> dagger.Container:
async def run_sanity_checks(self):
"""Runs sanity checks on the base image container.
This method is called on base image build.
The following sanity checks are meant to check that labels and environment variables about the base's base image and the current Airbyte base image are correctly set.

Raises:
SanityCheckError: Raised if a sanity check fails.
"""
if not await self.container.env_variable("AIRBYTE_BASE_BASE_IMAGE") == self.base_base_image_name:
raise SanityCheckError("the AIRBYTE_BASE_BASE_IMAGE environment variable is not correctly set.")
raise errors.SanityCheckError("the AIRBYTE_BASE_BASE_IMAGE environment variable is not correctly set.")
if not await self.container.env_variable("AIRBYTE_BASE_IMAGE") == self.name_with_tag:
raise SanityCheckError("the AIRBYTE_BASE_IMAGE environment variable is not correctly. set")
raise errors.SanityCheckError("the AIRBYTE_BASE_IMAGE environment variable is not correctly. set")
if not await self.container.label("io.airbyte.base_base_image") == self.base_base_image_name:
raise SanityCheckError("the io.airbyte.base_base_image label is not correctly set.")
raise errors.SanityCheckError("the io.airbyte.base_base_image label is not correctly set.")
if not await self.container.label("io.airbyte.base_image") == self.name_with_tag:
raise SanityCheckError("the io.airbyte.base_image label is not correctly set.")
raise errors.SanityCheckError("the io.airbyte.base_image label is not correctly set.")

@staticmethod
def get_github_url(cls):
def get_github_url(cls) -> str:
"""This method returns the GitHub URL of the file where the class is defined on the main branch.
This URL is used to generate the changelog entry for the release notes.
This URL will resolve once the code is pushed to the main branch.

Returns:
str: The GitHub URL of the file where the class is defined on the main branch.
"""
absolute_module_path = inspect.getfile(cls)
relative_module_path = Path(absolute_module_path).relative_to(consts.AIRBYTE_ROOT_DIR)
# This url will resolve once the code is pushed to the main branch
return f"{consts.AIRBYTE_GITHUB_REPO_URL}/blob/{consts.MAIN_BRANCH_NAME}/{relative_module_path}"
4 changes: 4 additions & 0 deletions airbyte-ci/connectors/base_images/base_images/consts.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#

"""This module declares constants used by the base_images module.
"""

from pathlib import Path

import dagger
Expand Down
23 changes: 23 additions & 0 deletions airbyte-ci/connectors/base_images/base_images/errors.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#
"""This module contains the exceptions used by the base_images module.
"""


class BaseImageVersionError(ValueError):
"""Raised when the version is not in the expected format."""

pass


class SanityCheckError(Exception):
"""Raised when a sanity check fails."""

pass


class PlatformAvailabilityError(ValueError):
"""Raised when an image does not support the passed."""

pass
Loading