Fix Scheduler restarting due to too many completed pods in cluster #40183
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ephraimbuddy
commented
Jun 11, 2024
ephraimbuddy
commented
Jun 11, 2024
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Show resolved
Hide resolved
860bcb5 to
625170d
Compare
|
This runs at every watcher loop and can get big in the case of many completed pods not deleted |
a1474d8 to
d70cded
Compare
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Outdated
Show resolved
Hide resolved
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Show resolved
Hide resolved
2 tasks
7c32d7f to
87c4ba1
Compare
eladkal
approved these changes
Jun 13, 2024
romsharon98
approved these changes
Jun 13, 2024
amoghrajesh
approved these changes
Jun 13, 2024
jedcunningham
approved these changes
Jun 13, 2024
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Outdated
Show resolved
Hide resolved
Currently, when a pod completes and is not deleted due to the user's configuration, the watcher keeps listing these pods and checking their status. We should instead stop watching the pod once it succeeds. To do that, pods are created with the executor done label set to False and changed to True when the pod completes. The watcher then watches only those pods that the pod executor done label is False closes: apache#22612
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
…r_utils.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
37ac60e to
171a655
Compare
96 tasks
romsharon98
pushed a commit
to romsharon98/airflow
that referenced
this pull request
Jul 26, 2024
…pache#40183) * Fix Scheduler restarting due to too many completed pods in cluster Currently, when a pod completes and is not deleted due to the user's configuration, the watcher keeps listing these pods and checking their status. We should instead stop watching the pod once it succeeds. To do that, pods are created with the executor done label set to False and changed to True when the pod completes. The watcher then watches only those pods that the pod executor done label is False closes: apache#22612 * Update airflow/providers/cncf/kubernetes/pod_generator.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> * Add back removed section * Don't add pod key label from get go * Update airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com> --------- Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, when a pod completes and is not deleted due to the user's configuration, the watcher keeps listing these pods and checking their status. We should instead stop watching the pod once it succeeds. To do that, pods are created with the executor done label set to False and changed to True when the pod completes. The watcher then watches only those pods that the pod executor done label is False
closes: #22612