Update Airflow Airbyte Provider to use the new Airbyte API

[vytautasnord]

Description

As Airbyte transitions from the configuration API, set to sunset by early 2024, to the new Airbyte API (https://reference.airbyte.com/reference/start), it is imperative to update the existing Airflow Airbyte provider (https://airflow.apache.org/docs/apache-airflow-providers-airbyte/stable/index.html) with the new API.

@marcosmarxm maybe you have any update about this?

Use case/motivation

Due to anticipated changes in Airbyte API by early 2024, it's expected that the current Airflow Airbyte provider may become incompatible. Therefore, an update to the provider is necessary to ensure continued functionality

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

We should migrate to the offical Python SDK
https://github.com/airbytehq/airbyte-api-python-sdk

[Zhenye-Na]

We should migrate to the offical Python SDK airbytehq/airbyte-api-python-sdk

Hi @eladkal, @Taragolis, and @marcosmarxm

I am interested in this issue and I am more than happy to discuss with either of you regarding the approach of this issue.

[Taragolis]

@Zhenye-Na
Reference to different kind of API https://docs.airbyte.com/api-documentation and their differences

• Configuration API, Airflow Provider currently use this one. Endpoints started with api/v1/{service}
• Airbyte API, this is a target API. Endpoint started with v1/{service}. As far as I could see it is available in OSS since v0.50.13

Ideally keep both working until it discontinued, because I believe that I'm not the only one who still use v0.4x releases, but personally for me discontinue Airbyte Configuration API in Airflow doesn't affect me.

So if it possible, create new Hook (e.g. AirbyteApiHook [conn_type = "airbyte_api", hook_name = "Airbyte API", conn_name_attr = "airbyte_api_conn_id", default_conn_name = "airbyte_default"]) which provide thin wrapper around airbyte-api-python-sdk, new connection type, and create replacement for AirbyteTriggerSyncOperator / AirbyteJobSensor

[vytautasnord]

Hello, I'm interested to know if there has been any progress with this issue?

[potiuk]

Hello, I'm interested to know if there has been any progress with this issue?

Generally if there is no comment/work/pr there is no progress. This is an open-source project - nothing happens "in-hiding" But if you would like to take on the task - feel free. Generally such tasks need people (we have >2700 contributors so far) who have an interest in certain things and will implement them. Generally things are implemented when someone actually does implement them.

So if you have an interest there - maybe you or finding someone who can do it (if you don't feel like it) or your company sponsoring the job could be a way to move things forward.

[Cir02]

Hello there,

i have faced a similar issue using this provider, not working at all following this configuration from airbyte , and change manually the airflow/airbyte/provider (hook, operator and sensor). Is only a few lines to change to the new api. Should i create a pr for this files?

thxs!

[potiuk]

Sure