Adding ability to automatically set DAG to off after X times it failed sequentially#36935

Merged

eladkal merged 42 commits intoapache:mainfrom pateash:36239-automatically-set-DAG-off

Mar 15, 2024

+245-1

Commits on Mar 15, 2024

Add max_failure_runs field to DagModel

pateash
authored and
eladkal
committed
Implement max_failure_runs option for DAGs

pateash
authored and
eladkal
committed
Add method to check last N DAG runs' failure status

pateash
authored and
eladkal
committed
Add 'max_failure_runs' to various airflow files

pateash
authored and
eladkal
committed
Refactor 'check_last_N_dagruns_failed' method and add 'max_failure_runs' check

pateash
authored and
eladkal
committed
Add auto-pausing of DAG on exceeding 'max_failure_runs'

pateash
authored and
eladkal
committed
Rename 'max_failure_runs' to 'max_consecutive_failed_dag_runs'

pateash
authored and
eladkal
committed
Refactor method and variable names, enhance log messages in dagrun.py

pateash
authored and
eladkal
committed
Corrected downgrade method's docstring in migration file

pateash
authored and
eladkal
committed
Refactor variable names and update descriptions in Airflow models

pateash
authored and
eladkal
committed
Refined logic for pausing DAG upon failures

pateash
authored and
eladkal
committed
Add 'max_consecutive_failed_dag_runs' to DAG schema

pateash
authored and
eladkal
committed
Add default configuration for max_consecutive_failed_dag_runs

pateash
authored and
eladkal
committed
Update comment in dagrun.py

pateash
authored and
eladkal
committed
Add tracing to test_dag and new DAG pause test case

pateash
authored and
eladkal
committed
Refactor test_dag and add new test for DAG pause behavior

pateash
authored and
eladkal
committed
Refactor test_dag and add new test for DAG pause behavior

pateash
authored and
eladkal
committed
Remove debug code from test_dag.py

pateash
authored and
eladkal
committed
Add max_consecutive_failed_dag_runs to test data

pateash
authored and
eladkal
committed
Refactor code for readability and standards compliance

pateash
authored and
eladkal
committed
Update version in migration script filename and contents

pateash
authored and
eladkal
committed
tests: fixed

pateash
authored and
eladkal
committed
fixed: pre-commits

pateash
authored and
eladkal
committed
fixed: migrations added

pateash
authored and
eladkal
committed
fixed: migrations added

pateash
authored and
eladkal
committed
fixed: migrations added

pateash
authored and
eladkal
committed
fixed: migrations added

pateash
authored and
eladkal
committed
fixes: pr comments

pateash
authored and
eladkal
committed
fixes: pr comments

pateash
authored and
eladkal
committed
Update airflow/config_templates/config.yml

pateash
and
eladkal
committed
fixes: experimental added

pateash
authored and
eladkal
committed
fixes: reverting unrelated changes

pateash
authored and
eladkal
committed
fixes: pr comments

pateash
authored and
eladkal
committed
fixes: pr comments

pateash
authored and
eladkal
committed
fixes: pr comments

pateash
authored and
eladkal
committed
fixes: pr comments

pateash
authored and
eladkal
committed
fixes: docs added

pateash
authored and
eladkal
committed
fixed: pre-commits

pateash
authored and
eladkal
committed
fixed: pre-commits

pateash
authored and
eladkal
committed
Update airflow/config_templates/config.yml
eladkal
committed
Update airflow/models/dag.py
eladkal
committed
fixed: scalars giving wrong state

pateash
authored and
eladkal
committed