airflow-dag-generator

This is an example of how to use the dag_generator.py script to generate DAGs from a YAML configuration file.

The Airflow webserver is deployed using the docker-compose.yml file, which was initialized by running this command (feel free to play along at home):

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.10.5/docker-compose.yaml'

The docker-compose.yml file was then simplified to use only the services needed to run the basic DAG generator example in this project.

Prerequisites

• Docker
• Python 3.10+

Usage

From within the airflow-dag-generator directory, run the following command:

docker-compose up

After a few seconds, you should see the Airflow webserver running at http://localhost:8080, displaying the list of DAGs that were generated from the configs folder as well as any other DAGs that you add to the dags folder.

Adding a new DAG

To add a new DAG, you can:

1 - create a new YAML file in the configs folder, or

2 - add a Python script that defines a valid DAG to the dags folder. See test_dag.py for a simple example.