Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split Celery queues into more functional categories #2907

Closed
WesleyAC opened this issue Jul 20, 2023 · 1 comment · Fixed by #2908
Closed

Split Celery queues into more functional categories #2907

WesleyAC opened this issue Jul 20, 2023 · 1 comment · Fixed by #2908

Comments

@WesleyAC
Copy link
Member

Currently, we have five celery queues: Low priority, Medium priority, High priority, Imports, and Broadcasts. I propose (as described in #2874 (comment)), splitting these up into more functionality-related categories (seeing as the different "priority" queues are not actually prioritized differently):

In general, switching from the current queue "priority" system to a more functionality-based system would I think probably be preferable. The current status quo is that everything in the "high-priority" queue will be behind by the same amount of wall-clock time, which makes everything in it equally slow, when it might be better to have some particular tasks fall behind, while other (less common or compute intensive) tasks are operating in real time.

In particular, I propose starting with the following queues:

  • Suggested users
    • bookwyrm.suggested_users.bulk_add_instance_task
    • bookwyrm.suggested_users.bulk_remove_instance_task
    • bookwyrm.suggested_users.remove_suggestion_task
    • bookwyrm.suggested_users.remove_user_task
    • bookwyrm.suggested_users.rerank_suggestions_task
    • bookwyrm.suggested_users.rerank_user_task
  • Images
    • bookwyrm.preview_images.generate_edition_preview_image_task
    • bookwyrm.preview_images.generate_site_preview_image_task
    • bookwyrm.preview_images.generate_user_preview_image_task
    • bookwyrm.preview_images.remove_user_preview_image_task
    • imagekit.cachefiles.backends._generate_file
  • Email
    • bookwyrm.emailing.send_email
  • Broadcast
    • bookwyrm.models.activitypub_mixin.broadcast_task
  • ActivityStreams
    • bookwyrm.activitystreams.add_status_task
    • bookwyrm.activitystreams.add_book_statuses_task
    • bookwyrm.activitystreams.add_user_statuses_task
    • bookwyrm.activitystreams.handle_boost_task
    • bookwyrm.activitystreams.populate_stream_task
    • bookwyrm.activitystreams.remove_book_statuses_task
    • bookwyrm.activitystreams.remove_status_task
    • bookwyrm.activitystreams.remove_user_statuses_task
  • Imports
    • bookwyrm.models.import_job.import_item_task
    • bookwyrm.models.import_job.start_import_task
  • Connection Manager
    • bookwyrm.connectors.connector_manager.create_edition_task
    • bookwyrm.connectors.connector_manager.load_more_data
  • Lists
    • bookwyrm.lists_stream.add_list_task
    • bookwyrm.lists_stream.add_user_lists_task
    • bookwyrm.lists_stream.populate_lists_task
    • bookwyrm.lists_stream.remove_list_task
    • bookwyrm.lists_stream.remove_user_lists_task
  • Inbox
    • bookwyrm.views.inbox.activity_task
  • Misc
    • bookwyrm.activitypub.base_activity.set_related_field
    • bookwyrm.models.antispam.automod_task
    • bookwyrm.models.user.get_remote_reviews
    • bookwyrm.models.user.set_remote_server

There are also some cases where different tasks should be in different queues depending on other circumstances, such as add_status_task probably wanting to go in a import-related queue for backdated statuses (currently it goes in the low priority queue in that case)

We will need to keep around the existing queues for a release in addition to the new queues, so that we can upgrade without dropping any tasks.

@mouse-reeve
Copy link
Member

I think this is a good idea! I also think we could simply not add imported reviews to the activity streams.

WesleyAC added a commit to WesleyAC/bookwyrm that referenced this issue Jul 20, 2023
WesleyAC added a commit to WesleyAC/bookwyrm that referenced this issue Jul 20, 2023
WesleyAC added a commit to WesleyAC/bookwyrm that referenced this issue Jul 20, 2023
WesleyAC added a commit to WesleyAC/bookwyrm that referenced this issue Jul 20, 2023
WesleyAC added a commit to WesleyAC/bookwyrm that referenced this issue Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants