PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows

an end-to-end data pipeline extracting music listening habits and producing an insightful dashboard

Pipeline to automate the collection of board game and expansion data from BoardGameGeek's XML API2. Data is stored in Google Cloud Storage and BigQuery. Data is modelled using DBT in a star schema. (Terraform, GCP, Mage, Python, dbt)

Repo home of the MAGE pipeline between an S3 Bucket and an RDS database on AWS.

2024 project

A compact project showcasing local data lake setup using Docker, Mage, Spark, MinIO, Iceberg, and StarRocks. Ideal for learning modern data engineering practices.

🧙 MAGE v0 - Assets

This project is the database migration pipline project using mage.ai.

Synchronize with Mage deployed on Google Cloud Run