Scheduler is slow when dealing with lots of tasks

[cloudcode-hungary]

Summary:

When there are a few thousand tasks in the config, beat becomes completely unreliable and unstable. Reconfiguration, syncing takes a lot of time, and with frequent schedule changes, hours can pass by without sending any task to the queue.

• Celery Version: 5.3.4
• Celery-Beat Version: 2.5.0

Exact steps to reproduce the issue:

1. Register 10k tasks with any config, (we only use CronSchedule)
2. Apply a change in the configuration in random intervals, 5s-300s, e.g. enabling or disabling tasks, or change the cron schedule.
3. Check the logs, you will see a lot of:

[2023-09-04 12:54:00,006: DEBUG/MainProcess] beat: Synchronizing schedule...

[2023-09-04 12:54:00,006: DEBUG/MainProcess] Writing entries...

[2023-09-04 12:55:49,129: DEBUG/MainProcess] beat: Synchronizing schedule...

[2023-09-04 12:55:49,130: DEBUG/MainProcess] Writing entries...

and beat is not applying any tasks.

Detailed information

A few parts are clearly written without performance considerations, like:
def all_as_schedule(...)
could use the prefetch related api https://docs.djangoproject.com/en/4.2/ref/models/querysets/#prefetch-related
def sync(...)
could use the bulk update api https://docs.djangoproject.com/en/4.2/ref/models/querysets/#django.db.models.query.QuerySet.bulk_update

Actually we tried to run a forked version with the above modifications, but it was just not enough, it is still very slow and unreliable.
Do you have any general ideas what should be the correct way to continue the investigation?

This "Writing entries..." part takes most of the time, and it is not clearly database related, I can see a lot of SELECT "django_celery_beat_periodictasks"."ident", "django_celery_beat_periodictasks"."last_update" FROM "django_celery_beat_periodictasks" WHERE "django_celery_beat_periodictasks"."ident" = 1 LIMIT 21 which I assume is def last_change(...) , which is called in def schedule_changed(...) which is called from the @property def schedule(...) which is a bit harder to track.

• Are you intested in applying performance related optimisations?
• Should we run multiple instances of beat, maybe with slices of the whole schedule? Are there any caveats?
• Any other thoughts?

[auvipy]

I am open to review and accept performance related improvement