Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Celery worker - redis connection isn't being reestablished, celery 5.5 / kombu upgrade #541

Closed
alextreme opened this issue Mar 4, 2025 · 2 comments · Fixed by #542
Closed
Assignees

Comments

@alextreme
Copy link
Member

Product versie / Product version

2.5.0

Omschrijf het probleem / Describe the bug

Taiga Dimpact 227 + 229

Silvia has investigated multiple issues on Dimpact PodiumD in which the Objects API celery worker connection to Redis was broken (by a deployment or a restart of the cluster) after which the connection was not reestablished.

This leads to objects being created but no notifications being sent to Open Notificaties and subscribed applications. The problem is resolved after a restart of the objects api celery worker pod.

Sergei has struggled with this same issue with Open Forms and has attempted to update Open Froms to a rc of Celery 5.5: https://github.com/open-formulieren/open-forms/pull/4937/files

This discussion at celery states the issue and solution:
celery/celery#7276 (comment)

Because of the issues I'd like to (pre)release a objects-api with celery 5.5, at least to verify that this solves the issue or as an alternative until celery 5.5 comes out properly

Stappen om te reproduceren / Steps to reproduce

No response

Verwacht gedrag / Expected behavior

No response

@alextreme alextreme added bug Something isn't working triage labels Mar 4, 2025
@stevenbal stevenbal self-assigned this Mar 6, 2025
@stevenbal stevenbal moved this from Triage to Todo in Data en API fundament Mar 6, 2025
@stevenbal stevenbal removed the triage label Mar 6, 2025
@stevenbal stevenbal moved this from Todo to In Progress in Data en API fundament Mar 6, 2025
stevenbal added a commit that referenced this issue Mar 6, 2025
since Celery 5.5 isn't out yet, this should fix the redis connection issues
@stevenbal stevenbal moved this from In Progress to Implemented in Data en API fundament Mar 6, 2025
@stevenbal
Copy link
Collaborator

stevenbal commented Mar 6, 2025

I was thinking of making a patch release 3.0.2 for #538 this week anyway, so I could include the kombu upgrade in there as well. I'm not sure though if Dimpact still relies on the v1 endpoints though, so should I make backport the kombu upgrade and make a version 2.5.1 as well?

@alextreme
Copy link
Member Author

Lets leave it at a 3.0.x patch

stevenbal added a commit that referenced this issue Mar 6, 2025
@github-project-automation github-project-automation bot moved this from Implemented to Done in Data en API fundament Mar 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants