Optionally build python connector with the local CDK

[alafanechere]

As our connectors are now tested and built via airbyte-ci we should support the use case of building and testing connector with the local cdk version.

I suggest the following approach:

• If the current branch has changes on a CDK files: The local CDK will be installed in test environments and connector container as a pre-release version
• We will install connectors dependencies in the test environment with --pre to install pre-release version
• If a connector is not pinning the airbyte cdk it will naturally use the prerelease version
• If a connector is pinning the CDK version the developer will have to edit its setup.py file to use airbyte-cdk==<current-version>-dev

If the local CDK is being used the test reports should clearly display it.

[alafanechere]

This how it's currently achieved in /legacy-test

[evantahler]

Grooming:

How do developers indicate that we should use the local (in-branch) CDK vs the 'real' version the connector declares?

• Flag? Would need a manually triggered workflow
• What about CI? You can't add flags there...
• Should we always use the local CDK?

Workflow:

• if the cdk folder has changes (git diff), run the test 2x - once with the local/branch change, and once with the version chosen by the connector

Notes:

• Are there ever pre-releases for the CDK?
• You can't publish a new CDK version and update connectors in the same PR. you have to publish a new CDK first, and then change the version that connector uses in a second PR.

Short term decision:

• add flag to manual workflow to use local CDK. Both test and build
  • If a connector is also change in the PR, tests will run via CI using the old CDK, and then the manual test will use the new stuff
  • Change airbyte CDK to be a local dependency
    • Do we need to change setup.py? Do whatever the manage.sh script does now.
    • sed -iE 's,"airbyte-cdk[^"]*","airbyte-cdk @ file://localhost/airbyte-cdk",' setup.py
    • Check if pip has a better hack/tool for this
• We should groom automated future work with the Extensibility Team.