Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gradle: search for python3.11 binary for AL2023 support #31227

Merged
merged 6 commits into from
Oct 10, 2023
Merged

gradle: search for python3.11 binary for AL2023 support #31227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e8348bb
gradle: search for python3.11 binary for AL2023 support
postamar Oct 10, 2023
0f770ff
airbyte-ci: bump version and update changelog
postamar Oct 10, 2023
39030f0
fixup! check for python3.11 first
postamar Oct 10, 2023
dbafe67
airbyte-ci: 'test' instead of 'check' for java connector unit tests
postamar Oct 10, 2023
529ceef
fix changelog
postamar Oct 10, 2023
c6365c4
better commentary
postamar Oct 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions airbyte-ci/connectors/pipelines/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,6 +395,7 @@ This command runs the Python tests for a airbyte-ci poetry package.
## Changelog
| Version | PR | Description |
| ------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| 1.5.1 | [#31227](https://github.com/airbytehq/airbyte/pull/31227) | Use python 3.11 in amazoncorretto-bazed gradle containers, run 'test' gradle task instead of 'check'. |
| 1.5.0 | [#30456](https://github.com/airbytehq/airbyte/pull/30456) | Start building Python connectors using our base images. |
| 1.4.6 | [ #31087](https://github.com/airbytehq/airbyte/pull/31087) | Throw error if airbyte-ci tools is out of date |
| 1.4.5 | [#31133](https://github.com/airbytehq/airbyte/pull/31133) | Fix bug when building containers using `with_integration_base_java_and_normalization`. |
Expand Down
2 changes: 1 addition & 1 deletion airbyte-ci/connectors/pipelines/pipelines/gradle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ async def _run(self) -> StepResult:
"findutils", # gradle requires xargs, which is shipped in findutils.
"jq", # required by :airbyte-connector-test-harnesses:acceptance-test-harness to inspect docker images.
"npm", # required by :format.
"pip", # required by :format.
"python3.11-pip", # required by :format.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❗some tools (ci_credentials, pipelines) cant currently support python3.11, this wont be an issue here but is an edge case. is python 3.10 available instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason that I don't understand, no. If it comes to that, it'll probably be simpler to change the base container to something other than amazoncorretto. I only picked it because openjdk was no longer maintained and amazoncorretto has LTS.

"rsync", # required for gradle cache synchronization.
]

Expand Down
2 changes: 1 addition & 1 deletion airbyte-ci/connectors/pipelines/pipelines/tests/java_connectors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ class UnitTests(GradleTask):
"""A step to run unit tests for Java connectors."""

title = "Java Connector Unit Tests"
gradle_task_name = "check"
gradle_task_name = "test"
bind_to_docker_host = True


Expand Down
2 changes: 1 addition & 1 deletion airbyte-ci/connectors/pipelines/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "pipelines"
version = "1.5.0"
version = "1.5.1"
description = "Packaged maintained by the connector operations team to perform CI for connectors' pipelines"
authors = ["Airbyte <contact@airbyte.io>"]

Expand Down
12 changes: 12 additions & 0 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,18 @@ python {
envPath = '.venv'
minPythonVersion = '3.10' // should be 3.10 for local development

// Amazon Linux support.
// The airbyte-ci tool runs gradle tasks in AL2023-based containers.
// In AL2023, `python3` is necessarily v3.9, and later pythons need to be installed and named explicitly.
// See https://github.com/amazonlinux/amazon-linux-2023/issues/459 for details.
try {
if ("python3.11 --version".execute().waitFor() == 0) {
// python3.11 definitely exists at this point, use it instead of 'python3'.
pythonBinary "python3.11"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓why cant we bind python to python3.11 at the container level?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

amazonlinux/amazon-linux-2023#459

}
} catch (IOException _) {
// Swallow exception if python3.11 is not installed.
}
// Pyenv support.
try {
def pyenvRoot = "pyenv root".execute()
Expand Down
12 changes: 12 additions & 0 deletions buildSrc/src/main/groovy/airbyte-python.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,18 @@ class AirbytePythonPlugin implements Plugin<Project> {
envPath = venvDirectoryName
minPythonVersion '3.10'

// Amazon Linux support.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📚 could use some elaboration on why we do this instead of just run python.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair!

// The airbyte-ci tool runs gradle tasks in AL2023-based containers.
// In AL2023, `python3` is necessarily v3.9, and later pythons need to be installed and named explicitly.
// See https://github.com/amazonlinux/amazon-linux-2023/issues/459 for details.
try {
if ("python3.11 --version".execute().waitFor() == 0) {
// python3.11 definitely exists at this point, use it instead of 'python3'.
pythonBinary "python3.11"
}
} catch (IOException _) {
// Swallow exception if python3.11 is not installed.
}
// Pyenv support.
try {
def pyenvRoot = "pyenv root".execute()
Expand Down