airbyte-ci: upgrade to dagger 0.9.5

[alafanechere]

We're currently pinning version 0.6.4 of dagger on airbyte-ci and in our infra.

Upgrading to the latest dagger version would be beneficial for:

• stability: bug fixes and improvements have been shipped since 0.6.4, especially on the distributed cache feature. Using the latest dagger version could make us use the Dagger hosted magicache that we currently host ourself.
• support: when facing bugs / outages it's easier to get support from Dagger team if we're running their latest version
• new features: Dagger latest version has new feature like external networking, service orchestration etc. that we could leverage to improve airbyte-ci dind use cases. It also starts supporting "re-usable dagger components" a.k.a Dagger Modules / Zenith.

Infra actions to take:

1. Declare a new RunnerDeployment variant here : ci-runner-connector-dev-large-dagger-0-9-4
2. Declare a new Provisionervariant here like ci-runner-connector-dev-large-dagger-0-6-4
3. Declare a new Daemonset variant here: like connector-dev-dagger-0-9-4-engine: set the image to registry.dagger.io/engine:v0.9.4 Set the nodeAffinity rules to target ci-runner-connector-dev-large-dagger-0-9-4. Please note that the _EXPERIMENTAL_* env var should be discarded and replaced with different env var (e.g. DAGGER_CLOUD_TOKEN). We should checkout Dagger docs / sync with their team to make sure we assign the correct env vars to the engine.
4. kubectl apply the changes manifest

Testing out the new version by changing the client code on a branch:

• Update dagger to 0.9.4 in pipelines/pyproject.toml
• Changes runs_on of the workflow you'd like to test to ci-runner-connector-dev-large-dagger-0-9-4
• Fix any breaking changes

Note about Dagger Cloud tokens:

Dagger cloud token should now be considered a secret as they are now used for cache syncs. We should remove the public occurrences of the hardcoded token: p.eyJ1IjogIjFiZjEwMmRjLWYyZmQtNDVhNi1iNzM1LTgxNzI1NGFkZDU2ZiIsICJpZCI6ICJlNjk3YzZiYy0yMDhiLTRlMTktODBjZC0yNjIyNGI3ZDBjMDEifQ.hT6eMOYt3KZgNoVGNYI3_v4CC-s19z8uQsBkGrBhU3k. The token to use is available on the Dagger Cloud console.

[evantahler]

Grooming:

• Clarify that Dagger hosting the 🪄 cache doesn't cost us more
• Can we (airbyte) get access to the S3 bucket so we can self-manage (e.g. empty it) via IAM role?
• Token both lives in code and ENV. It should now be ENV - it becomes the cache upload key (CI only). We need to invalidate the old token - talk to Dagger / check the Dagger UI. Store in Last Pass.
  • There's a dev workflow change needed to get the token locally or else no dagger cloud. - Document!
  • Old versions of the CI might get weird.
  • Share with the team that they might need to upgrade

[alafanechere]

Update:

• Client side upgrade is ready for review. I bumped Dagger version in airbyte-ci / CAT / base image project + updated the broken API calls. I manually checked our workflow, everything is working fine so far.
• I deployed Dagger 0.9.4 engines on our CI cluster, they are ready to used and have been used for testing the workflows.

Remaining steps:

• Confirm 🟢 CI once Update ci to use dev versoin of airbyte-ci when pipeline is edited #33418 is implemented and merged: we want the CI on the upgrade branch to run with the branch airbyte-ci binary which has Dagger 0.9.4
• Confirm with the Dagger team that the Dagger Cloud cache is working as expected