KubeVela is a Modern Application Platform built on top of Kubernetes.
It is a CNCF Incubator level project.
Airbyte Replication operator can be run using KubeVela as an Addon.
A YouTube video demo is here.
To run locally, setup KubeVela on your local Kubernetes cluster. Refer Official documentation for installation steps.
Note: We need cluster level access, so recommended way of running this demo is on a local cluster.
Example Kind configuration.
Also enable VelaUX, the web UX for KubeVela. E2E tests has a reference
Once Vela is running on the cluster.
cd e2e
vela addon enable infra/vela/minio
vela addon enable infra/vela/airbyte-replication-operator
Setup up some test data. Here we will try Airbyte Connector Source S3 to Airbyte Connector Destination S3 replication.
# TODO: upload test datasets to minio. But for now Minio's drag and drop is super nice.S3-to-S3 to replication
kubectl apply -f hello-airbyte-s3-to-s3/k8s --recursive
vela up -f hello-airbyte-s3-to-s3/vela/application.yaml
Check the Replication application running in the VelaUX, default username/password: admin/VelaUX12345.
vela port-forward -n vela-system addon-velaux 9082:80
Cleanup
vela uninstall
We can also use Temporal like the Official Airbyte for scheduling.
If you already run Temporal, then Airbyte is an excellent implementation. Check out documentation here.
Here is a simple workflow code example for any custom scheduling needs based on our tiny replication driver.
A video demo is here.
A video demo is here.
A video demo is here.
A video demo is here.
A video demo is here.
A video demo is here.