Airbyte Replication Operator

An implementation of Airbyte Protocol in Rust aiming to be simple, modular, extensible and resource efficient.

⭐⭐⭐Unstable and Work in progress⭐⭐⭐

Please try at your own risk. Feedback is much appreciated.

This is solely a personal project of Harsha Teja Kanna and not affiliated with Airbyte Inc. which owns Airbyte.

Note: This is NOT a Kubernetes Operator for Airbyte, only an independent implementation of the Airbyte Protocol.

Introduction

This is the partial mirror repository for an implementation of Airbyte Protocol data plane.

Why? Check the architecture reasoning here ADR

Why Rust? Check the decision record here ADR

Introduction blog: https://ekalavya.dev/why-and-how-i-implemented-airbyte-protocol-in-rust

Upcoming blog series here

Note: It will become the canonical repository once the implementation is ready(currently, in a rough state) for public review and contributions.

Roadmap

• Standalone data replication Driver from Airbyte Source to Destination
• Pluggable State management layer
• Pluggable progress Tracking and Monitoring
• Pluggable OCI Container/VM integration (containerd, Docker, Podman, Youki, Firecracker VM, ECS etc. )
• Multiple execution models (parallel/serial)
• Feature parity with official Airbyte Platform ( only data plane components )
  • Mappers
  • Validators
  • Normalization/dbt? (Currently not in scope)
  • Operators/Transformers? (Currently not in scope)
• Rust CDK (Connector Development Kit) 🚀
• Kubernetes Operator
• CLI
• Airbyte Desktop, Yes! 🚀
• Benchmarks
• Usage Examples
• KubeVela integration
• 100% Airbyte Protocol compatability and acceptance tests
• Automated tests
• CI
• Deployment and operations documentation
• Stable (GA) releases
• Setup SLSA3 Provenance for release artifacts.
• FAQs
• Strictly keep the driver binary to minimal size (ideally under 10MB). Very important 😃

Documentation

• Start here for Quick Start on your machine.
• Start here for Replication Driver
• Start here for Kubernetes Operator
• Start here for CLI operations
• Documentation for existing and possible Integration ideas.
• Frequently asked questions (FAQs)

Upcoming Blogs/Demos

• https://h7kanna.medium.com/why-and-how-i-integrated-airbyte-and-apache-hudi-c18aff3af21a
• https://ekalavya.dev/why-and-how-i-implemented-airbyte-protocol-in-rust
• https://ekalavya.dev/let-us-copy-some-bytes-using-airbyte-replication-driver-in-rust
• https://ekalavya.dev/architecture-deep-dive-airbyte-replication-driver-in-rust
• https://ekalavya.dev/overview-of-airbyte-replication-operator-in-rust-use-cases
• https://ekalavya.dev/introduction-to-airbyte-replication-driver-in-rust-cli-operations
• https://ekalavya.dev/how-to-run-airbyte-replication-operator-kubevela-addon
• https://ekalavya.dev/overview-of-parallel-executions-in-airbyte-replication-driver
• https://ekalavya.dev/how-to-use-temporal-for-scheduling-airbyte-replication-driver
• https://ekalavya.dev/how-to-run-airbyte-replication-driver-using-podman
• https://ekalavya.dev/how-to-run-airbyte-replication-driver-using-youki
• https://ekalavya.dev/running-airbyte-replication-driver-on-amazon-ecs
• https://ekalavya.dev/running-airbyte-replication-driver-on-aws-fargate
• https://ekalavya.dev/firecracker-vm-for-airbyte-replications
• https://ekalavya.dev/how-i-integrated-airbyte-and-apache-hudi-again

High level Architecture view

Low level Architecture view