Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: notify about potential serialization failures when sending DagRun, don't serialize unnecessary params, guard listener for exceptions #41690

Merged

Conversation

mobuchowski
Copy link
Contributor

We've moved to a model where DagRun level events utilize ProcessPoolExecutor to send data, to not do this on the direct scheduler process. This PR adds more guards to that model:

  • do not send/pickle whole DagRun model to process, since serializing only selected attributes is cheaper and less error prone
  • add notification on ProcessPoolExecutor complete - this allows listener to log potential serialization failures.
  • guard whole listener with BaseException checks - we don't want errors to bubble to scheduler.

@mobuchowski mobuchowski force-pushed the notify-serialization-exception branch 12 times, most recently from 6593ede to 8e7ccf7 Compare August 29, 2024 11:22
@mobuchowski mobuchowski force-pushed the notify-serialization-exception branch from 8e7ccf7 to c8bca89 Compare August 31, 2024 13:59
…un, don't serialize unnecessary params, guard listener for exceptions

Signed-off-by: Maciej Obuchowski <obuchowski.maciej@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants