Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Deferrable stuck as "scheduled" during backfill #26205

Merged
merged 3 commits into from
Sep 23, 2022
Merged

Fix Deferrable stuck as "scheduled" during backfill #26205

merged 3 commits into from
Sep 23, 2022

Conversation

ephraimbuddy
Copy link
Contributor

@ephraimbuddy ephraimbuddy commented Sep 7, 2022 •
edited
Loading

When the triggerer puts the task to scheduled, it gets stuck because it's already in ti_status.running. We need to move it to ti_status.to_run for it to continue running

closes: #25653

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Sep 7, 2022
@ephraimbuddy ephraimbuddy force-pushed the fix-backfilling-deferrable branch from 6bf552a to e8b167e Compare September 7, 2022 15:20
@ephraimbuddy ephraimbuddy mentioned this pull request Sep 8, 2022
Merged
@ephraimbuddy ephraimbuddy force-pushed the fix-backfilling-deferrable branch from e8b167e to e4f38b8 Compare September 8, 2022 14:02
@ashb ashb added this to the Airflow 2.4.1 milestone Sep 14, 2022
ephraimbuddy and others added 3 commits September 20, 2022 09:48
@ephraimbuddy
Fix Deferrable Operators get stuck as "scheduled" during backfill
eea53ca 
When the triggerer puts the task to scheduled, it gets stuck because
it's already in ti_status.running. We need to move it to ti_status.to_run for it to continue running
@ephraimbuddy
Taskinstance can be deferred by the trigger, we should do nothing
c681ffd
@ephraimbuddy @uranusjr
Update airflow/jobs/backfill_job.py
4383ff6 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
@ephraimbuddy ephraimbuddy force-pushed the fix-backfilling-deferrable branch from 8ef82db to 4383ff6 Compare September 20, 2022 08:49
@uranusjr uranusjr changed the title Fix Deferrable Operators get stuck as "scheduled" during backfill Fix Deferrable stuck as "scheduled" during backfill Sep 23, 2022
@uranusjr uranusjr merged commit 3396d1f into apache:main Sep 23, 2022
@uranusjr uranusjr deleted the fix-backfilling-deferrable branch September 23, 2022 05:08
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label Sep 23, 2022
@eladkal eladkal mentioned this pull request Sep 23, 2022
Closed
2 tasks
jedcunningham pushed a commit that referenced this pull request Sep 23, 2022
@ephraimbuddy @jedcunningham
Fix Deferrable stuck as "scheduled" during backfill (#26205)
53f0ffd 
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
(cherry picked from commit 3396d1f)
@jedcunningham jedcunningham mentioned this pull request Sep 27, 2022
Closed
22 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@uranusjr uranusjr uranusjr left review comments

@jedcunningham jedcunningham jedcunningham approved these changes

@norm norm norm approved these changes

@kaxil kaxil Awaiting requested review from kaxil

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

Assignees
No one assigned
Labels
area:Scheduler including HA (high availability) scheduler type:bug-fix Changelog: Bug Fixes
Projects
None yet
Milestone
Airflow 2.4.1
Development

Successfully merging this pull request may close these issues.

Deferrable Operators get stuck as "scheduled" during backfill
5 participants
@ephraimbuddy @norm @uranusjr @jedcunningham @ashb