airbyte-ci: upgrade to dagger 0.9.5 #33582
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
|
Current dependencies on/for this PR:
This stack of pull requests is managed by Graphite. |
282778e to
40df54a
Compare
2015dcb to
bfdd6de
Compare
|
Warning Soft code freeze is in effect until 2024-01-02. Please avoid merging to master. #freedom-and-responsibility |
402811b to
4c8a6a4
Compare
08c193c to
cc757ef
Compare
cc757ef to
8de621f
Compare
4e586dc to
83c228e
Compare
postamar
approved these changes
Jan 4, 2024
| @@ -10,7 +10,7 @@ authors = ["Airbyte <contact@airbyte.io>"] | |||
|
|
|||
| [tool.poetry.dependencies] | |||
| python = "~3.10" | |||
| dagger-io = "^0.6.4" | |||
| dagger-io = "==0.9.5" | |||
There was a problem hiding this comment.
Shouldn't you bump the major version counter of airbyte-ci as well?
| if docker_hub_username_secret and docker_hub_password_secret: | ||
| # Docker login happens late because there's a cache buster in the docker login command. | ||
| dockerd_container = docker_login(dockerd_container, docker_hub_username_secret, docker_hub_password_secret) | ||
| return dockerd_container.with_exec( | ||
| ["dockerd", "--log-level=error", f"--host=tcp://0.0.0.0:{DOCKER_HOST_PORT}", "--tls=false"], insecure_root_capabilities=True | ||
| ) | ||
| ).as_service() |
bnchrch
approved these changes
Jan 5, 2024
| @@ -142,14 +145,14 @@ runs: | |||
| export _EXPERIMENTAL_DAGGER_RUNNER_HOST="unix:///var/run/buildkit/buildkitd.sock" | |||
| airbyte-ci --disable-dagger-run --is-ci --gha-workflow-run-id=${{ github.run_id }} ${{ inputs.subcommand }} ${{ inputs.options }} | |||
| env: | |||
| _EXPERIMENTAL_DAGGER_CLOUD_TOKEN: "p.eyJ1IjogIjFiZjEwMmRjLWYyZmQtNDVhNi1iNzM1LTgxNzI1NGFkZDU2ZiIsICJpZCI6ICJlNjk3YzZiYy0yMDhiLTRlMTktODBjZC0yNjIyNGI3ZDBjMDEifQ.hT6eMOYt3KZgNoVGNYI3_v4CC-s19z8uQsBkGrBhU3k" | |||
There was a problem hiding this comment.
❤️ Nice to see this go to an input!
| @@ -94,11 +94,6 @@ async def run_connectors_pipelines( | |||
| default_connectors_semaphore = anyio.Semaphore(concurrency) | |||
| dagger_logs_output = sys.stderr if not dagger_logs_path else create_and_open_file(dagger_logs_path) | |||
| async with dagger.Connection(Config(log_output=dagger_logs_output, execute_timeout=execute_timeout)) as dagger_client: | |||
|
|
|||
| # HACK: This is to get a long running dockerd service to be shared across all the connectors pipelines | |||
| # Using the "normal" service binding leads to restart of dockerd during pipeline run that can cause corrupted docker state | |||
| ) | ||
| # When the connectors pipelines are done, we can stop the dockerd service | ||
| tg_main.cancel_scope.cancel() | ||
| await dockerd_service.start() |
There was a problem hiding this comment.
💅 nit: Should/could we turn this to an async with
async with dockerd_service.start():There was a problem hiding this comment.
It'd be nice but I don't think we can as it's not returning a context manager
| from pipelines.airbyte_ci.connectors.context import ConnectorContext | ||
|
|
||
|
|
||
| def get_sdk_version() -> str: |
There was a problem hiding this comment.
Not immediately clear what SDK its refering to when you call it.
Suggested change
| def get_sdk_version() -> str: | |
| def get_dagger_sdk_version() -> str: |
c62d4fe to
d0a4b5c
Compare
@postamar I think it's worth a try, if my not mistaken the issues you opened on Dagger repo are closed now . |
jatinyadav-cc
pushed a commit
to ollionorg/datapipes-airbyte
that referenced
this pull request
Feb 26, 2024
jatinyadav-cc
pushed a commit
to ollionorg/datapipes-airbyte
that referenced
this pull request
Feb 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Closes #33429
How
airbyte-ci,base-images,CATManual testing
(manual testing should be done using the dev airbyte ci binary - this is why I used [skip ci] on all commits)
connectors --name=source-postgres --name=source-faker testconnectors --name=source-pokeapi publish --pre-releaseformat check allformat fix all(local)connectors --name=source-faker bump_version(local)airbyte-ci test🚨 User Impact 🚨
None, if all manual tests succeed
Dependencies:
magicacheset up is complete on the infra side.Update: Distributed caching is currently enabled on our runners: waiting for Dagger team to confirm it's working correctly.
2 This should not be merged until CI can run on current branche's binary
Depends on #33519
Related infra PRs