forked from apache/airflow
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathRELEASE_NOTES.rst
11102 lines (9416 loc) · 569 KB
/
RELEASE_NOTES.rst
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. contents:: Apache Airflow Releases
:local:
:depth: 1
.. towncrier release notes start
Airflow 2.4.3 (2022-11-14)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
Make ``RotatingFilehandler`` used in ``DagProcessor`` non-caching (#27223)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
In case you want to decrease cache memory when ``CONFIG_PROCESSOR_MANAGER_LOGGER=True``, and you have your local settings created before,
you can update ``processor_manager_handler`` to use ``airflow.utils.log.non_caching_file_handler.NonCachingRotatingFileHandler`` handler instead of ``logging.RotatingFileHandler``. (#27065)
Bug Fixes
^^^^^^^^^
- Fix double logging with some task logging handler (#27591)
- Replace FAB url filtering function with Airflow's (#27576)
- Fix mini scheduler expansion of mapped task (#27506)
- ``SLAMiss`` is nullable and not always given back when pulling task instances (#27423)
- Fix behavior of ``_`` when searching for DAGs (#27448)
- Fix getting the ``dag/task`` ids from BaseExecutor (#27550)
- Fix SQLAlchemy primary key black-out error on DDRQ (#27538)
- Fix IntegrityError during webserver startup (#27297)
- Add case insensitive constraint to username (#27266)
- Fix python external template keys (#27256)
- Reduce extraneous task log requests (#27233)
- Make ``RotatingFilehandler`` used in ``DagProcessor`` non-caching (#27223)
- Listener: Set task on SQLAlchemy TaskInstance object (#27167)
- Fix dags list page auto-refresh & jump search null state (#27141)
- Set ``executor.job_id`` to ``BackfillJob.id`` for backfills (#27020)
Misc/Internal
^^^^^^^^^^^^^
- Bump loader-utils from ``1.4.0`` to ``1.4.1`` in ``/airflow/www`` (#27552)
- Reduce log level for k8s ``TCP_KEEPALIVE`` etc warnings (#26981)
Doc only changes
^^^^^^^^^^^^^^^^
- Use correct executable in docker compose docs (#27529)
- Fix wording in DAG Runs description (#27470)
- Document that ``KubernetesExecutor`` overwrites container args (#27450)
- Fix ``BaseOperator`` links (#27441)
- Correct timer units to seconds from milliseconds. (#27360)
- Add missed import in the Trigger Rules example (#27309)
- Update SLA wording to reflect it is relative to ``Dag Run`` start. (#27111)
- Add ``kerberos`` environment variables to the docs (#27028)
Airflow 2.4.2 (2022-10-23)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
Default for ``[webserver] expose_stacktrace`` changed to ``False`` (#27059)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The default for ``[webserver] expose_stacktrace`` has been set to ``False``, instead of ``True``. This means administrators must opt-in to expose tracebacks to end users.
Bug Fixes
^^^^^^^^^
- Make tracebacks opt-in (#27059)
- Add missing AUTOINC/SERIAL for FAB tables (#26885)
- Add separate error handler for 405(Method not allowed) errors (#26880)
- Don't re-patch pods that are already controlled by current worker (#26778)
- Handle mapped tasks in task duration chart (#26722)
- Fix task duration cumulative chart (#26717)
- Avoid 500 on dag redirect (#27064)
- Filter dataset dependency data on webserver (#27046)
- Remove double collection of dags in ``airflow dags reserialize`` (#27030)
- Fix auto refresh for graph view (#26926)
- Don't overwrite connection extra with invalid json (#27142)
- Fix next run dataset modal links (#26897)
- Change dag audit log sort by date from asc to desc (#26895)
- Bump min version of jinja2 (#26866)
- Add missing colors to ``state_color_mapping`` jinja global (#26822)
- Fix running debuggers inside ``airflow tasks test`` (#26806)
- Fix warning when using xcomarg dependencies (#26801)
- demote Removed state in priority for displaying task summaries (#26789)
- Ensure the log messages from operators during parsing go somewhere (#26779)
- Add restarting state to TaskState Enum in REST API (#26776)
- Allow retrieving error message from data.detail (#26762)
- Simplify origin string cleaning (#27143)
- Remove DAG parsing from StandardTaskRunner (#26750)
- Fix non-hidden cumulative chart on duration view (#26716)
- Remove TaskFail duplicates check (#26714)
- Fix airflow tasks run --local when dags_folder differs from that of processor (#26509)
- Fix yarn warning from d3-color (#27139)
- Fix version for a couple configurations (#26491)
- Revert "No grid auto-refresh for backfill dag runs (#25042)" (#26463)
- Retry on Airflow Schedule DAG Run DB Deadlock (#26347)
Misc/Internal
^^^^^^^^^^^^^
- Clean-ups around task-mapping code (#26879)
- Move user-facing string to template (#26815)
- add icon legend to datasets graph (#26781)
- Bump ``sphinx`` and ``sphinx-autoapi`` (#26743)
- Simplify ``RTIF.delete_old_records()`` (#26667)
- Bump FAB to ``4.1.4`` (#26393)
Doc only changes
^^^^^^^^^^^^^^^^
- Fixed triple quotes in task group example (#26829)
- Documentation fixes (#26819)
- make consistency on markup title string level (#26696)
- Add a note against use of top level code in timetable (#26649)
- Fix broken URL for ``docker-compose.yaml`` (#26726)
Airflow 2.4.1 (2022-09-30)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
No significant changes.
Bug Fixes
^^^^^^^^^
- When rendering template, unmap task in context (#26702)
- Fix scroll overflow for ConfirmDialog (#26681)
- Resolve deprecation warning re ``Table.exists()`` (#26616)
- Fix XComArg zip bug (#26636)
- Use COALESCE when ordering runs to handle NULL (#26626)
- Check user is active (#26635)
- No missing user warning for public admin (#26611)
- Allow MapXComArg to resolve after serialization (#26591)
- Resolve warning about DISTINCT ON query on dags view (#26608)
- Log warning when secret backend kwargs is invalid (#26580)
- Fix grid view log try numbers (#26556)
- Template rendering issue in passing ``templates_dict`` to task decorator (#26390)
- Fix Deferrable stuck as ``scheduled`` during backfill (#26205)
- Suppress SQLALCHEMY_TRACK_MODIFICATIONS warning in db init (#26617)
- Correctly set ``json_provider_class`` on Flask app so it uses our encoder (#26554)
- Fix WSGI root app (#26549)
- Fix deadlock when mapped task with removed upstream is rerun (#26518)
- ExecutorConfigType should be ``cacheable`` (#26498)
- Fix proper joining of the path for logs retrieved from celery workers (#26493)
- DAG Deps extends ``base_template`` (#26439)
- Don't update backfill run from the scheduler (#26342)
Doc only changes
^^^^^^^^^^^^^^^^
- Clarify owner links document (#26515)
- Fix invalid RST in dataset concepts doc (#26434)
- Document the ``non-sensitive-only`` option for ``expose_config`` (#26507)
- Fix ``example_datasets`` dag names (#26495)
- Zip-like effect is now possible in task mapping (#26435)
- Use task decorator in docs instead of classic operators (#25711)
Airflow 2.4.0 (2022-09-19)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
Data-aware Scheduling and ``Dataset`` concept added to Airflow
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
New to this release of Airflow is the concept of Datasets to Airflow, and with it a new way of scheduling dags:
data-aware scheduling.
This allows DAG runs to be automatically created as a result of a task "producing" a dataset. In some ways
this can be thought of as the inverse of ``TriggerDagRunOperator``, where instead of the producing DAG
controlling which DAGs get created, the consuming DAGs can "listen" for changes.
A dataset is identified by a URI:
.. code-block:: python
from airflow import Dataset
# The URI doesn't have to be absolute
dataset = Dataset(uri="my-dataset")
# Or you can use a scheme to show where it lives.
dataset2 = Dataset(uri="s3://bucket/prefix")
To create a DAG that runs whenever a Dataset is updated use the new ``schedule`` parameter (see below) and
pass a list of 1 or more Datasets:
.. code-block:: python
with DAG(dag_id='dataset-consumer', schedule=[dataset]):
...
And to mark a task as producing a dataset pass the dataset(s) to the ``outlets`` attribute:
.. code-block:: python
@task(outlets=[dataset])
def my_task():
...
# Or for classic operators
BashOperator(task_id="update-ds", bash_command=..., outlets=[dataset])
If you have the producer and consumer in different files you do not need to use the same Dataset object, two
``Dataset()``\s created with the same URI are equal.
Datasets represent the abstract concept of a dataset, and (for now) do not have any direct read or write
capability - in this release we are adding the foundational feature that we will build upon.
For more info on Datasets please see :doc:`/concepts/datasets`.
Expanded dynamic task mapping support
"""""""""""""""""""""""""""""""""""""
Dynamic task mapping now includes support for ``expand_kwargs``, ``zip`` and ``map``.
For more info on dynamic task mapping please see :doc:`/concepts/dynamic-task-mapping`.
DAGS used in a context manager no longer need to be assigned to a module variable (#23592)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Previously you had to assign a DAG to a module-level variable in order for Airflow to pick it up. For example this
.. code-block:: python
with DAG(dag_id="example") as dag:
...
@dag
def dag_maker():
...
dag2 = dag_maker()
can become
.. code-block:: python
with DAG(dag_id="example"):
...
@dag
def dag_maker():
...
dag_maker()
If you want to disable the behaviour for any reason then set ``auto_register=False`` on the dag:
.. code-block:: python
# This dag will not be picked up by Airflow as it's not assigned to a variable
with DAG(dag_id="example", auto_register=False):
...
Deprecation of ``schedule_interval`` and ``timetable`` arguments (#25410)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
We added new DAG argument ``schedule`` that can accept a cron expression, timedelta object, *timetable* object, or list of dataset objects. Arguments ``schedule_interval`` and ``timetable`` are deprecated.
If you previously used the ``@daily`` cron preset, your DAG may have looked like this:
.. code-block:: python
with DAG(
dag_id="my_example",
start_date=datetime(2021, 1, 1),
schedule_interval="@daily",
):
...
Going forward, you should use the ``schedule`` argument instead:
.. code-block:: python
with DAG(
dag_id="my_example",
start_date=datetime(2021, 1, 1),
schedule="@daily",
):
...
The same is true if you used a custom timetable. Previously you would have used the ``timetable`` argument:
.. code-block:: python
with DAG(
dag_id="my_example",
start_date=datetime(2021, 1, 1),
timetable=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
):
...
Now you should use the ``schedule`` argument:
.. code-block:: python
with DAG(
dag_id="my_example",
start_date=datetime(2021, 1, 1),
schedule=EventsTimetable(event_dates=[pendulum.datetime(2022, 4, 5)]),
):
...
Removal of experimental Smart Sensors (#25507)
""""""""""""""""""""""""""""""""""""""""""""""
Smart Sensors were added in 2.0 and deprecated in favor of Deferrable operators in 2.2, and have now been removed.
``airflow.contrib`` packages and deprecated modules are dynamically generated (#26153, #26179, #26167)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The ``airflow.contrib`` packages and deprecated modules from Airflow 1.10 in ``airflow.hooks``, ``airflow.operators``, ``airflow.sensors`` packages are now dynamically generated modules and while users can continue using the deprecated contrib classes, they are no longer visible for static code check tools and will be reported as missing. It is recommended for the users to move to the non-deprecated classes.
``DBApiHook`` and ``SQLSensor`` have moved (#24836)
"""""""""""""""""""""""""""""""""""""""""""""""""""
``DBApiHook`` and ``SQLSensor`` have been moved to the ``apache-airflow-providers-common-sql`` provider.
DAG runs sorting logic changed in grid view (#25090)
""""""""""""""""""""""""""""""""""""""""""""""""""""
The ordering of DAG runs in the grid view has been changed to be more "natural".
The new logic generally orders by data interval, but a custom ordering can be
applied by setting the DAG to use a custom timetable.
New Features
^^^^^^^^^^^^
- Add Data-aware Scheduling (`AIP-48 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-48+milestone%3A%22Airflow+2.4.0%22>`_)
- Add ``@task.short_circuit`` TaskFlow decorator (#25752)
- Make ``execution_date_or_run_id`` optional in ``tasks test`` command (#26114)
- Automatically register DAGs that are used in a context manager (#23592, #26398)
- Add option of sending DAG parser logs to stdout. (#25754)
- Support multiple ``DagProcessors`` parsing files from different locations. (#25935)
- Implement ``ExternalPythonOperator`` (#25780)
- Make execution_date optional for command ``dags test`` (#26111)
- Implement ``expand_kwargs()`` against a literal list (#25925)
- Add trigger rule tooltip (#26043)
- Add conf parameter to CLI for airflow dags test (#25900)
- Include scheduled slots in pools view (#26006)
- Add ``output`` property to ``MappedOperator`` (#25604)
- Add roles delete command to cli (#25854)
- Add Airflow specific warning classes (#25799)
- Add support for ``TaskGroup`` in ``ExternalTaskSensor`` (#24902)
- Add ``@task.kubernetes`` taskflow decorator (#25663)
- Add a way to import Airflow without side-effects (#25832)
- Let timetables control generated run_ids. (#25795)
- Allow per-timetable ordering override in grid view (#25633)
- Grid logs for mapped instances (#25610, #25621, #25611)
- Consolidate to one ``schedule`` param (#25410)
- DAG regex flag in backfill command (#23870)
- Adding support for owner links in the Dags view UI (#25280)
- Ability to clear a specific DAG Run's task instances via REST API (#23516)
- Possibility to document DAG with a separate markdown file (#25509)
- Add parsing context to DAG Parsing (#25161)
- Implement ``CronTriggerTimetable`` (#23662)
- Add option to mask sensitive data in UI configuration page (#25346)
- Create new databases from the ORM (#24156)
- Implement ``XComArg.zip(*xcom_args)`` (#25176)
- Introduce ``sla_miss`` metric (#23402)
- Implement ``map()`` semantic (#25085)
- Add override method to TaskGroupDecorator (#25160)
- Implement ``expand_kwargs()`` (#24989)
- Add parameter to turn off SQL query logging (#24570)
- Add ``DagWarning`` model, and a check for missing pools (#23317)
- Add Task Logs to Grid details panel (#24249)
- Added small health check server and endpoint in scheduler(#23905)
- Add built-in External Link for ``ExternalTaskMarker`` operator (#23964)
- Add default task retry delay config (#23861)
- Add clear DagRun endpoint. (#23451)
- Add support for timezone as string in cron interval timetable (#23279)
- Add auto-refresh to dags home page (#22900, #24770)
Improvements
^^^^^^^^^^^^
- Add more weekday operator and sensor examples #26071 (#26098)
- Add subdir parameter to dags reserialize command (#26170)
- Update zombie message to be more descriptive (#26141)
- Only send an ``SlaCallbackRequest`` if the DAG is scheduled (#26089)
- Promote ``Operator.output`` more (#25617)
- Upgrade API files to typescript (#25098)
- Less ``hacky`` double-rendering prevention in mapped task (#25924)
- Improve Audit log (#25856)
- Remove mapped operator validation code (#25870)
- More ``DAG(schedule=...)`` improvements (#25648)
- Reduce ``operator_name`` dupe in serialized JSON (#25819)
- Make grid view group/mapped summary UI more consistent (#25723)
- Remove useless statement in ``task_group_to_grid`` (#25654)
- Add optional data interval to ``CronTriggerTimetable`` (#25503)
- Remove unused code in ``/grid`` endpoint (#25481)
- Add and document description fields (#25370)
- Improve Airflow logging for operator Jinja template processing (#25452)
- Update core example DAGs to use ``@task.branch`` decorator (#25242)
- Update DAG ``audit_log`` route (#25415)
- Change stdout and stderr access mode to append in commands (#25253)
- Remove ``getTasks`` from Grid view (#25359)
- Improve taskflow type hints with ParamSpec (#25173)
- Use tables in grid details panes (#25258)
- Explicitly list ``@dag`` arguments (#25044)
- More typing in ``SchedulerJob`` and ``TaskInstance`` (#24912)
- Patch ``getfqdn`` with more resilient version (#24981)
- Replace all ``NBSP`` characters by ``whitespaces`` (#24797)
- Re-serialize all DAGs on ``airflow db upgrade`` (#24518)
- Rework contract of try_adopt_task_instances method (#23188)
- Make ``expand()`` error vague so it's not misleading (#24018)
- Add enum validation for ``[webserver]analytics_tool`` (#24032)
- Add ``dttm`` searchable field in audit log (#23794)
- Allow more parameters to be piped through via ``execute_in_subprocess`` (#23286)
- Use ``func.count`` to count rows (#23657)
- Remove stale serialized dags (#22917)
- AIP45 Remove dag parsing in airflow run local (#21877)
- Add support for queued state in DagRun update endpoint. (#23481)
- Add fields to dagrun endpoint (#23440)
- Use ``sql_alchemy_conn`` for celery result backend when ``result_backend`` is not set (#24496)
Bug Fixes
^^^^^^^^^
- Have consistent types between the ORM and the migration files (#24044, #25869)
- Disallow any dag tags longer than 100 char (#25196)
- Add the dag_id to ``AirflowDagCycleException`` message (#26204)
- Properly build URL to retrieve logs independently from system (#26337)
- For worker log servers only bind to IPV6 when dual stack is available (#26222)
- Fix ``TaskInstance.task`` not defined before ``handle_failure`` (#26040)
- Undo secrets backend config caching (#26223)
- Fix faulty executor config serialization logic (#26191)
- Show ``DAGs`` and ``Datasets`` menu links based on role permission (#26183)
- Allow setting ``TaskGroup`` tooltip via function docstring (#26028)
- Fix RecursionError on graph view of a DAG with many tasks (#26175)
- Fix backfill occasional deadlocking (#26161)
- Fix ``DagRun.start_date`` not set during backfill with ``--reset-dagruns`` True (#26135)
- Use label instead of id for dynamic task labels in graph (#26108)
- Don't fail DagRun when leaf ``mapped_task`` is SKIPPED (#25995)
- Add group prefix to decorated mapped task (#26081)
- Fix UI flash when triggering with dup logical date (#26094)
- Fix Make items nullable for ``TaskInstance`` related endpoints to avoid API errors (#26076)
- Fix ``BranchDateTimeOperator`` to be ``timezone-awreness-insensitive`` (#25944)
- Fix legacy timetable schedule interval params (#25999)
- Fix response schema for ``list-mapped-task-instance`` (#25965)
- Properly check the existence of missing mapped TIs (#25788)
- Fix broken auto-refresh on grid view (#25950)
- Use per-timetable ordering in grid UI (#25880)
- Rewrite recursion when parsing DAG into iteration (#25898)
- Find cross-group tasks in ``iter_mapped_dependants`` (#25793)
- Fail task if mapping upstream fails (#25757)
- Support ``/`` in variable get endpoint (#25774)
- Use cfg default_wrap value for grid logs (#25731)
- Add origin request args when triggering a run (#25729)
- Operator name separate from class (#22834)
- Fix incorrect data interval alignment due to assumption on input time alignment (#22658)
- Return None if an ``XComArg`` fails to resolve (#25661)
- Correct ``json`` arg help in ``airflow variables set`` command (#25726)
- Added MySQL index hint to use ``ti_state`` on ``find_zombies`` query (#25725)
- Only excluded actually expanded fields from render (#25599)
- Grid, fix toast for ``axios`` errors (#25703)
- Fix UI redirect (#26409)
- Require dag_id arg for dags list-runs (#26357)
- Check for queued states for dags auto-refresh (#25695)
- Fix upgrade code for the ``dag_owner_attributes`` table (#25579)
- Add map index to task logs api (#25568)
- Ensure that zombie tasks for dags with errors get cleaned up (#25550)
- Make extra link work in UI (#25500)
- Sync up plugin API schema and definition (#25524)
- First/last names can be empty (#25476)
- Refactor DAG pages to be consistent (#25402)
- Check ``expand_kwargs()`` input type before unmapping (#25355)
- Filter XCOM by key when calculating map lengths (#24530)
- Fix ``ExternalTaskSensor`` not working with dynamic task (#25215)
- Added exception catching to send default email if template file raises any exception (#24943)
- Bring ``MappedOperator`` members in sync with ``BaseOperator`` (#24034)
Misc/Internal
^^^^^^^^^^^^^
- Add automatically generated ``ERD`` schema for the ``MetaData`` DB (#26217)
- Mark serialization functions as internal (#26193)
- Remove remaining deprecated classes and replace them with ``PEP562`` (#26167)
- Move ``dag_edges`` and ``task_group_to_dict`` to corresponding util modules (#26212)
- Lazily import many modules to improve import speed (#24486, #26239)
- FIX Incorrect typing information (#26077)
- Add missing contrib classes to deprecated dictionaries (#26179)
- Re-configure/connect the ``ORM`` after forking to run a DAG processor (#26216)
- Remove cattrs from lineage processing. (#26134)
- Removed deprecated contrib files and replace them with ``PEP-562`` getattr (#26153)
- Make ``BaseSerialization.serialize`` "public" to other classes. (#26142)
- Change the template to use human readable task_instance description (#25960)
- Bump ``moment-timezone`` from ``0.5.34`` to ``0.5.35`` in ``/airflow/www`` (#26080)
- Fix Flask deprecation warning (#25753)
- Add ``CamelCase`` to generated operations types (#25887)
- Fix migration issues and tighten the CI upgrade/downgrade test (#25869)
- Fix type annotations in ``SkipMixin`` (#25864)
- Workaround setuptools editable packages path issue (#25848)
- Bump ``undici`` from ``5.8.0 to 5.9.1`` in /airflow/www (#25801)
- Add custom_operator_name attr to ``_BranchPythonDecoratedOperator`` (#25783)
- Clarify ``filename_template`` deprecation message (#25749)
- Use ``ParamSpec`` to replace ``...`` in Callable (#25658)
- Remove deprecated modules (#25543)
- Documentation on task mapping additions (#24489)
- Remove Smart Sensors (#25507)
- Fix ``elasticsearch`` test config to avoid warning on deprecated template (#25520)
- Bump ``terser`` from ``4.8.0 to 4.8.1`` in /airflow/ui (#25178)
- Generate ``typescript`` types from rest ``API`` docs (#25123)
- Upgrade utils files to ``typescript`` (#25089)
- Upgrade remaining context file to ``typescript``. (#25096)
- Migrate files to ``ts`` (#25267)
- Upgrade grid Table component to ``ts.`` (#25074)
- Skip mapping against mapped ``ti`` if it returns None (#25047)
- Refactor ``js`` file structure (#25003)
- Move mapped kwargs introspection to separate type (#24971)
- Only assert stuff for mypy when type checking (#24937)
- Bump ``moment`` from ``2.29.3 to 2.29.4`` in ``/airflow/www`` (#24885)
- Remove "bad characters" from our codebase (#24841)
- Remove ``xcom_push`` flag from ``BashOperator`` (#24824)
- Move Flask hook registration to end of file (#24776)
- Upgrade more javascript files to ``typescript`` (#24715)
- Clean up task decorator type hints and docstrings (#24667)
- Preserve original order of providers' connection extra fields in UI (#24425)
- Rename ``charts.css`` to ``chart.css`` (#24531)
- Rename ``grid.css`` to ``chart.css`` (#24529)
- Misc: create new process group by ``set_new_process_group`` utility (#24371)
- Airflow UI fix Prototype Pollution (#24201)
- Bump ``moto`` version (#24222)
- Remove unused ``[github_enterprise]`` from ref docs (#24033)
- Clean up ``f-strings`` in logging calls (#23597)
- Add limit for ``JPype1`` (#23847)
- Simply json responses (#25518)
- Add min attrs version (#26408)
Doc only changes
^^^^^^^^^^^^^^^^
- Add url prefix setting for ``Celery`` Flower (#25986)
- Updating deprecated configuration in examples (#26037)
- Fix wrong link for taskflow tutorial (#26007)
- Reorganize tutorials into a section (#25890)
- Fix concept doc for dynamic task map (#26002)
- Update code examples from "classic" operators to taskflow (#25845, #25657)
- Add instructions on manually fixing ``MySQL`` Charset problems (#25938)
- Prefer the local Quick Start in docs (#25888)
- Fix broken link to ``Trigger Rules`` (#25840)
- Improve docker documentation (#25735)
- Correctly link to Dag parsing context in docs (#25722)
- Add note on ``task_instance_mutation_hook`` usage (#25607)
- Note that TaskFlow API automatically passes data between tasks (#25577)
- Update DAG run to clarify when a DAG actually runs (#25290)
- Update tutorial docs to include a definition of operators (#25012)
- Rewrite the Airflow documentation home page (#24795)
- Fix ``task-generated mapping`` example (#23424)
- Add note on subtle logical date change in ``2.2.0`` (#24413)
- Add missing import in best-practices code example (#25391)
Airflow 2.3.4 (2022-08-23)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
Added new config ``[logging]log_formatter_class`` to fix timezone display for logs on UI (#24811)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
If you are using a custom Formatter subclass in your ``[logging]logging_config_class``, please inherit from ``airflow.utils.log.timezone_aware.TimezoneAware`` instead of ``logging.Formatter``.
For example, in your ``custom_config.py``:
.. code-block:: python
from airflow.utils.log.timezone_aware import TimezoneAware
# before
class YourCustomFormatter(logging.Formatter):
...
# after
class YourCustomFormatter(TimezoneAware):
...
AIRFLOW_FORMATTER = LOGGING_CONFIG["formatters"]["airflow"]
AIRFLOW_FORMATTER["class"] = "somewhere.your.custom_config.YourCustomFormatter"
# or use TimezoneAware class directly. If you don't have custom Formatter.
AIRFLOW_FORMATTER["class"] = "airflow.utils.log.timezone_aware.TimezoneAware"
Bug Fixes
^^^^^^^^^
- Disable ``attrs`` state management on ``MappedOperator`` (#24772)
- Serialize ``pod_override`` to JSON before pickling ``executor_config`` (#24356)
- Fix ``pid`` check (#24636)
- Rotate session id during login (#25771)
- Fix mapped sensor with reschedule mode (#25594)
- Cache the custom secrets backend so the same instance gets re-used (#25556)
- Add right padding (#25554)
- Fix reducing mapped length of a mapped task at runtime after a clear (#25531)
- Fix ``airflow db reset`` when dangling tables exist (#25441)
- Change ``disable_verify_ssl`` behaviour (#25023)
- Set default task group in dag.add_task method (#25000)
- Removed interfering force of index. (#25404)
- Remove useless logging line (#25347)
- Adding mysql index hint to use index on ``task_instance.state`` in critical section query (#25673)
- Configurable umask to all daemonized processes. (#25664)
- Fix the errors raised when None is passed to template filters (#25593)
- Allow wildcarded CORS origins (#25553)
- Fix "This Session's transaction has been rolled back" (#25532)
- Fix Serialization error in ``TaskCallbackRequest`` (#25471)
- fix - resolve bash by absolute path (#25331)
- Add ``__repr__`` to ParamsDict class (#25305)
- Only load distribution of a name once (#25296)
- convert ``TimeSensorAsync`` ``target_time`` to utc on call time (#25221)
- call ``updateNodeLabels`` after ``expandGroup`` (#25217)
- Stop SLA callbacks gazumping other callbacks and DOS'ing the ``DagProcessorManager`` queue (#25147)
- Fix ``invalidateQueries`` call (#25097)
- ``airflow/www/package.json``: Add name, version fields. (#25065)
- No grid auto-refresh for backfill dag runs (#25042)
- Fix tag link on dag detail page (#24918)
- Fix zombie task handling with multiple schedulers (#24906)
- Bind log server on worker to ``IPv6`` address (#24755) (#24846)
- Add ``%z`` for ``%(asctime)s`` to fix timezone for logs on UI (#24811)
- ``TriggerDagRunOperator.operator_extra_links`` is attr (#24676)
- Send DAG timeout callbacks to processor outside of ``prohibit_commit`` (#24366)
- Don't rely on current ORM structure for db clean command (#23574)
- Clear next method when clearing TIs (#23929)
- Two typing fixes (#25690)
Doc only changes
^^^^^^^^^^^^^^^^
- Update set-up-database.rst (#24983)
- Fix syntax in mysql setup documentation (#24893 (#24939)
- Note how DAG policy works with default_args (#24804)
- Update PythonVirtualenvOperator Howto (#24782)
- Doc: Add hyperlinks to Github PRs for Release Notes (#24532)
Misc/Internal
^^^^^^^^^^^^^
- Remove depreciation warning when use default remote tasks logging handlers (#25764)
- clearer method name in scheduler_job.py (#23702)
- Bump cattrs version (#25689)
- Include missing mention of ``external_executor_id`` in ``sql_engine_collation_for_ids`` docs (#25197)
- Refactor ``DR.task_instance_scheduling_decisions`` (#24774)
- Sort operator extra links (#24992)
- Extends ``resolve_xcom_backend`` function level documentation (#24965)
- Upgrade FAB to 4.1.3 (#24884)
- Limit Flask to <2.3 in the wake of 2.2 breaking our tests (#25511)
- Limit astroid version to < 2.12 (#24982)
- Move javascript compilation to host (#25169)
- Bump typing-extensions and mypy for ParamSpec (#25088)
Airflow 2.3.3 (2022-07-09)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
We've upgraded Flask App Builder to a major version 4.* (#24399)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Flask App Builder is one of the important components of Airflow Webserver, as
it uses a lot of dependencies that are essential to run the webserver and integrate it
in enterprise environments - especially authentication.
The FAB 4.* upgrades a number of dependencies to major releases, which upgrades them to versions
that have a number of security issues fixed. A lot of tests were performed to bring the dependencies
in a backwards-compatible way, however the dependencies themselves implement breaking changes in their
internals so it might be that some of those changes might impact the users in case they are using the
libraries for their own purposes.
One important change that you likely will need to apply to Oauth configuration is to add
``server_metadata_url`` or ``jwks_uri`` and you can read about it more
in `this issue <https://github.com/dpgaspar/Flask-AppBuilder/issues/1861>`_.
Here is the list of breaking changes in dependencies that comes together with FAB 4:
* ``Flask`` from 1.X to 2.X `breaking changes <https://flask.palletsprojects.com/en/2.0.x/changes/#version-2-0-0>`__
* ``flask-jwt-extended`` 3.X to 4.X `breaking changes: <https://flask-jwt-extended.readthedocs.io/en/stable/v4_upgrade_guide/>`__
* ``Jinja2`` 2.X to 3.X `breaking changes: <https://jinja.palletsprojects.com/en/3.0.x/changes/#version-3-0-0>`__
* ``Werkzeug`` 1.X to 2.X `breaking changes <https://werkzeug.palletsprojects.com/en/2.0.x/changes/#version-2-0-0>`__
* ``pyJWT`` 1.X to 2.X `breaking changes: <https://pyjwt.readthedocs.io/en/stable/changelog.html#v2-0-0>`__
* ``Click`` 7.X to 8.X `breaking changes: <https://click.palletsprojects.com/en/8.0.x/changes/#version-8-0-0>`__
* ``itsdangerous`` 1.X to 2.X `breaking changes <https://github.com/pallets/itsdangerous/blob/main/CHANGES.rst#version-200>`__
Bug Fixes
^^^^^^^^^
- Fix exception in mini task scheduler (#24865)
- Fix cycle bug with attaching label to task group (#24847)
- Fix timestamp defaults for ``sensorinstance`` (#24638)
- Move fallible ``ti.task.dag`` assignment back inside ``try/except`` block (#24533) (#24592)
- Add missing types to ``FSHook`` (#24470)
- Mask secrets in ``stdout`` for ``airflow tasks test`` (#24362)
- ``DebugExecutor`` use ``ti.run()`` instead of ``ti._run_raw_task`` (#24357)
- Fix bugs in ``URI`` constructor for ``MySQL`` connection (#24320)
- Missing ``scheduleinterval`` nullable true added in ``openapi`` (#24253)
- Unify ``return_code`` interface for task runner (#24093)
- Handle occasional deadlocks in trigger with retries (#24071)
- Remove special serde logic for mapped ``op_kwargs`` (#23860)
- ``ExternalTaskSensor`` respects ``soft_fail`` if the external task enters a ``failed_state`` (#23647)
- Fix ``StatD`` timing metric units (#21106)
- Add ``cache_ok`` flag to sqlalchemy TypeDecorators. (#24499)
- Allow for ``LOGGING_LEVEL=DEBUG`` (#23360)
- Fix grid date ticks (#24738)
- Debounce status highlighting in Grid view (#24710)
- Fix Grid vertical scrolling (#24684)
- don't try to render child rows for closed groups (#24637)
- Do not calculate grid root instances (#24528)
- Maintain grid view selection on filtering upstream (#23779)
- Speed up ``grid_data`` endpoint by 10x (#24284)
- Apply per-run log templates to log handlers (#24153)
- Don't crash scheduler if exec config has old k8s objects (#24117)
- ``TI.log_url`` fix for ``map_index`` (#24335)
- Fix migration ``0080_2_0_2`` - Replace null values before setting column not null (#24585)
- Patch ``sql_alchemy_conn`` if old Postgres schemes used (#24569)
- Seed ``log_template`` table (#24511)
- Fix deprecated ``log_id_template`` value (#24506)
- Fix toast messages (#24505)
- Add indexes for CASCADE deletes for ``task_instance`` (#24488)
- Return empty dict if Pod JSON encoding fails (#24478)
- Improve grid rendering performance with a custom tooltip (#24417, #24449)
- Check for ``run_id`` for grid group summaries (#24327)
- Optimize calendar view for cron scheduled DAGs (#24262)
- Use ``get_hostname`` instead of ``socket.getfqdn`` (#24260)
- Check that edge nodes actually exist (#24166)
- Fix ``useTasks`` crash on error (#24152)
- Do not fail re-queued TIs (#23846)
- Reduce grid view API calls (#24083)
- Rename Permissions to Permission Pairs. (#24065)
- Replace ``use_task_execution_date`` with ``use_task_logical_date`` (#23983)
- Grid fix details button truncated and small UI tweaks (#23934)
- Add TaskInstance State ``REMOVED`` to finished states and success states (#23797)
- Fix mapped task immutability after clear (#23667)
- Fix permission issue for dag that has dot in name (#23510)
- Fix closing connection ``dbapi.get_pandas_df`` (#23452)
- Check bag DAG ``schedule_interval`` match timetable (#23113)
- Parse error for task added to multiple groups (#23071)
- Fix flaky order of returned dag runs (#24405)
- Migrate ``jsx`` files that affect run/task selection to ``tsx`` (#24509)
- Fix links to sources for examples (#24386)
- Set proper ``Content-Type`` and ``chartset`` on ``grid_data`` endpoint (#24375)
Doc only changes
^^^^^^^^^^^^^^^^
- Update templates doc to mention ``extras`` and format Airflow ``Vars`` / ``Conns`` (#24735)
- Document built in Timetables (#23099)
- Alphabetizes two tables (#23923)
- Clarify that users should not use Maria DB (#24556)
- Add imports to deferring code samples (#24544)
- Add note about image regeneration in June 2022 (#24524)
- Small cleanup of ``get_current_context()`` chapter (#24482)
- Fix default 2.2.5 ``log_id_template`` (#24455)
- Update description of installing providers separately from core (#24454)
- Mention context variables and logging (#24304)
Misc/Internal
^^^^^^^^^^^^^
- Remove internet explorer support (#24495)
- Removing magic status code numbers from ``api_connexion`` (#24050)
- Upgrade FAB to ``4.1.2`` (#24619)
- Switch Markdown engine to ``markdown-it-py`` (#19702)
- Update ``rich`` to latest version across the board. (#24186)
- Get rid of ``TimedJSONWebSignatureSerializer`` (#24519)
- Update flask-appbuilder ``authlib``/ ``oauth`` dependency (#24516)
- Upgrade to ``webpack`` 5 (#24485)
- Add ``typescript`` (#24337)
- The JWT claims in the request to retrieve logs have been standardized: we use ``nbf`` and ``aud`` claims for
maturity and audience of the requests. Also "filename" payload field is used to keep log name. (#24519)
- Address all ``yarn`` test warnings (#24722)
- Upgrade to react 18 and chakra 2 (#24430)
- Refactor ``DagRun.verify_integrity`` (#24114)
- Upgrade FAB to ``4.1.1`` (#24399)
- We now need at least ``Flask-WTF 0.15`` (#24621)
Airflow 2.3.2 (2022-06-04)
--------------------------
No significant changes
Bug Fixes
^^^^^^^^^
- Run the ``check_migration`` loop at least once
- Fix grid view for mapped tasks (#24059)
- Icons in grid view for different DAG run types (#23970)
- Faster grid view (#23951)
- Disallow calling expand with no arguments (#23463)
- Add missing ``is_mapped`` field to Task response. (#23319)
- DagFileProcessorManager: Start a new process group only if current process not a session leader (#23872)
- Mask sensitive values for not-yet-running TIs (#23807)
- Add cascade to ``dag_tag`` to ``dag`` foreign key (#23444)
- Use ``--subdir`` argument value for standalone dag processor. (#23864)
- Highlight task states by hovering on legend row (#23678)
- Fix and speed up grid view (#23947)
- Prevent UI from crashing if grid task instances are null (#23939)
- Remove redundant register exit signals in ``dag-processor`` command (#23886)
- Add ``__wrapped__`` property to ``_TaskDecorator`` (#23830)
- Fix UnboundLocalError when ``sql`` is empty list in DbApiHook (#23816)
- Enable clicking on DAG owner in autocomplete dropdown (#23804)
- Simplify flash message for ``_airflow_moved`` tables (#23635)
- Exclude missing tasks from the gantt view (#23627)
Doc only changes
^^^^^^^^^^^^^^^^
- Add column names for DB Migration Reference (#23853)
Misc/Internal
^^^^^^^^^^^^^
- Remove pinning for xmltodict (#23992)
Airflow 2.3.1 (2022-05-25)
--------------------------
Significant Changes
^^^^^^^^^^^^^^^^^^^
No significant changes.
Bug Fixes
^^^^^^^^^
- Automatically reschedule stalled queued tasks in ``CeleryExecutor`` (#23690)
- Fix expand/collapse all buttons (#23590)
- Grid view status filters (#23392)
- Expand/collapse all groups (#23487)
- Fix retrieval of deprecated non-config values (#23723)
- Fix secrets rendered in UI when task is not executed. (#22754)
- Fix provider import error matching (#23825)
- Fix regression in ignoring symlinks (#23535)
- Fix ``dag-processor`` fetch metadata database config (#23575)
- Fix auto upstream dep when expanding non-templated field (#23771)
- Fix task log is not captured (#23684)
- Add ``reschedule`` to the serialized fields for the ``BaseSensorOperator`` (#23674)
- Modify db clean to also catch the ProgrammingError exception (#23699)
- Remove titles from link buttons (#23736)
- Fix grid details header text overlap (#23728)
- Ensure ``execution_timeout`` as timedelta (#23655)
- Don't run pre-migration checks for downgrade (#23634)
- Add index for event column in log table (#23625)
- Implement ``send_callback`` method for ``CeleryKubernetesExecutor`` and ``LocalKubernetesExecutor`` (#23617)
- Fix ``PythonVirtualenvOperator`` templated_fields (#23559)
- Apply specific ID collation to ``root_dag_id`` too (#23536)
- Prevent ``KubernetesJobWatcher`` getting stuck on resource too old (#23521)
- Fix scheduler crash when expanding with mapped task that returned none (#23486)
- Fix broken dagrun links when many runs start at the same time (#23462)
- Fix: Exception when parsing log #20966 (#23301)
- Handle invalid date parsing in webserver views. (#23161)
- Pools with negative open slots should not block other pools (#23143)
- Move around overflow, position and padding (#23044)
- Change approach to finding bad rows to LEFT OUTER JOIN. (#23528)
- Only count bad refs when ``moved`` table exists (#23491)
- Visually distinguish task group summary (#23488)
- Remove color change for highly nested groups (#23482)
- Optimize 2.3.0 pre-upgrade check queries (#23458)
- Add backward compatibility for ``core__sql_alchemy_conn__cmd`` (#23441)
- Fix literal cross product expansion (#23434)
- Fix broken task instance link in xcom list (#23367)
- Fix connection test button (#23345)
- fix cli ``airflow dags show`` for mapped operator (#23339)
- Hide some task instance attributes (#23338)
- Don't show grid actions if server would reject with permission denied (#23332)
- Use run_id for ``ti.mark_success_url`` (#23330)
- Fix update user auth stats (#23314)
- Use ``<Time />`` in Mapped Instance table (#23313)
- Fix duplicated Kubernetes DeprecationWarnings (#23302)
- Store grid view selection in url params (#23290)
- Remove custom signal handling in Triggerer (#23274)
- Override pool for TaskInstance when pool is passed from cli. (#23258)
- Show warning if '/' is used in a DAG run ID (#23106)
- Use kubernetes queue in kubernetes hybrid executors (#23048)
- Add tags inside try block. (#21784)
Doc only changes
^^^^^^^^^^^^^^^^
- Move ``dag_processing.processor_timeouts`` to counters section (#23393)
- Clarify that bundle extras should not be used for PyPi installs (#23697)
- Synchronize support for Postgres and K8S in docs (#23673)
- Replace DummyOperator references in docs (#23502)
- Add doc notes for keyword-only args for ``expand()`` and ``partial()`` (#23373)
- Document fix for broken elasticsearch logs with 2.3.0+ upgrade (#23821)
Misc/Internal
^^^^^^^^^^^^^
- Add typing for airflow/configuration.py (#23716)
- Disable Flower by default from docker-compose (#23685)
- Added postgres 14 to support versions(including breeze) (#23506)
- add K8S 1.24 support (#23637)
- Refactor code references from tree to grid (#23254)
Airflow 2.3.0 (2022-04-30)
--------------------------
For production docker image related changes, see the `Docker Image Changelog <https://airflow.apache.org/docs/docker-stack/changelog.html>`_.
Significant Changes
^^^^^^^^^^^^^^^^^^^
Passing ``execution_date`` to ``XCom.set()``, ``XCom.clear()`` , ``XCom.get_one()`` , and ``XCom.get_many()`` is deprecated (#19825)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Continuing the effort to bind TaskInstance to a DagRun, XCom entries are now also tied to a DagRun. Use the ``run_id`` argument to specify the DagRun instead.
Task log templates are now read from the metadata database instead of ``airflow.cfg`` (#20165)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Previously, a task's log is dynamically rendered from the ``[core] log_filename_template`` and ``[elasticsearch] log_id_template`` config values at runtime. This resulted in unfortunate characteristics, e.g. it is impractical to modify the config value after an Airflow instance is running for a while, since all existing task logs have be saved under the previous format and cannot be found with the new config value.
A new ``log_template`` table is introduced to solve this problem. This table is synchronized with the aforementioned config values every time Airflow starts, and a new field ``log_template_id`` is added to every DAG run to point to the format used by tasks (``NULL`` indicates the first ever entry for compatibility).
Minimum kubernetes library version bumped from ``3.0.0`` to ``21.7.0`` (#20759)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
.. note::
This is only about changing the ``kubernetes`` library, not the Kubernetes cluster. Airflow support for
Kubernetes version is described in `Installation prerequisites <https://airflow.apache.org/docs/apache-airflow/stable/installation/prerequisites.html>`_.
No change in behavior is expected. This was necessary in order to take advantage of a `bugfix <https://github.com/kubernetes-client/python-base/commit/70b78cd8488068c014b6d762a0c8d358273865b4>`_ concerning refreshing of Kubernetes API tokens with EKS, which enabled the removal of some `workaround code <https://github.com/apache/airflow/pull/20759>`_.
XCom now defined by ``run_id`` instead of ``execution_date`` (#20975)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
As a continuation to the TaskInstance-DagRun relation change started in Airflow 2.2, the ``execution_date`` columns on XCom has been removed from the database, and replaced by an `association proxy <https://docs.sqlalchemy.org/en/13/orm/extensions/associationproxy.html>`_ field at the ORM level. If you access Airflow's metadata database directly, you should rewrite the implementation to use the ``run_id`` column instead.
Note that Airflow's metadatabase definition on both the database and ORM levels are considered implementation detail without strict backward compatibility guarantees.
Non-JSON-serializable params deprecated (#21135).
"""""""""""""""""""""""""""""""""""""""""""""""""
It was previously possible to use dag or task param defaults that were not JSON-serializable.
For example this worked previously:
.. code-block:: python
@dag.task(params={"a": {1, 2, 3}, "b": pendulum.now()})
def datetime_param(value):
print(value)
datetime_param("{{ params.a }} | {{ params.b }}")
Note the use of ``set`` and ``datetime`` types, which are not JSON-serializable. This behavior is problematic because to override these values in a dag run conf, you must use JSON, which could make these params non-overridable. Another problem is that the support for param validation assumes JSON. Use of non-JSON-serializable params will be removed in Airflow 3.0 and until then, use of them will produce a warning at parse time.
You must use ``postgresql://`` instead of ``postgres://`` in ``sql_alchemy_conn`` for SQLAlchemy 1.4.0+ (#21205)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
When you use SQLAlchemy 1.4.0+, you need to use ``postgresql://`` as the scheme in the ``sql_alchemy_conn``.
In the previous versions of SQLAlchemy it was possible to use ``postgres://`` , but using it in
SQLAlchemy 1.4.0+ results in:
.. code-block::
> raise exc.NoSuchModuleError(
"Can't load plugin: %s:%s" % (self.group, name)
)
E sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgres
If you cannot change the scheme of your URL immediately, Airflow continues to work with SQLAlchemy
1.3 and you can downgrade SQLAlchemy, but we recommend updating the scheme.