Dev and learn

data transformation with dbt, airflow, snowflake, terraform, python

under construction...

airflow

• . create a new airflow project
• . set alerts for failed tasks >> slack, ms teams, email
  • . slack
  • . ms teams
  • . email

DBT

• dbt model path: /airflow/dags/dbt_project_snowflake/models

• dbt macros path: /airflow/dags/dbt_project_snowflake/macros

• dbt test path: /airflow/dags/dbt_project_snowflake/tests

• . create a dbt project

• . create a model

• . create macros

• . add custom tests

• . create fact / dimension tables

• . incremental load

snowflake

• . stage : raw data
• . intermediate : cleansed data
• . mart : transformed data into Fact/Dimension tables

Terraform

• . create database
• . create S3 bucket
• . create stage in snowflake
• . fetch keys from Key vault
• . set github actions for run terraform

python

• . Generate random data

sample data

• Seattle Airbnb Open Data
• Boston Airbnb Open Data
• Madrid Airbnb Data

description

Reference