Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queues max worker concurrency #177

Merged
merged 35 commits into from
Jan 22, 2025
Merged

Queues max worker concurrency #177

merged 35 commits into from
Jan 22, 2025

Conversation

maxdml
Copy link
Contributor

@maxdml maxdml commented Jan 15, 2025

This PR allows users to control the maximum numbers of tasks, in a queue, a single DBOS Transact instance can execute concurrently. This knob is exposed by the new worker_concurrency Queue initialization parameter.

This is implemented by modifying the queue DB query to only retrieve uncompleted tasks eligible for this worker and limiting the query to either worker_concurrency or concurrency, with this priority, if they are set.

@@ -1132,12 +1132,13 @@ def start_queued_workflows(self, queue: "Queue", executor_id: str) -> List[str]:

# Select not-yet-completed functions in the queue ordered by the
# time at which they were enqueued.
# If there is a concurrency limit N, select only the N most recent
Copy link
Contributor Author

@maxdml maxdml Jan 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most recent are the tail of the queue, not its head. (We dequeue the oldest, i.e., the head.)

@maxdml maxdml marked this pull request as ready for review January 17, 2025 14:49
@maxdml maxdml force-pushed the queues-max-worker-concurrency branch from 9432f85 to ceb40a6 Compare January 21, 2025 21:41
Comment on lines +1148 to +1154
.where(
# Only select functions that have not been started yet or have been started by this worker
or_(
SystemSchema.workflow_queue.c.executor_id == None,
SystemSchema.workflow_queue.c.executor_id == executor_id,
)
)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could remove the completed_at_epoch_ms == None filter, redundant with the necessary executor_id == None, but I think its good for clarity and shouldn't affect performance.

@maxdml maxdml merged commit 04bc3a1 into main Jan 22, 2025
5 checks passed
@maxdml maxdml deleted the queues-max-worker-concurrency branch January 22, 2025 01:33
maxdml added a commit to dbos-inc/dbos-transact-ts that referenced this pull request Jan 27, 2025
- Implement worker concurrency limits for workflow queues. Same logic as
in dbos-inc/dbos-transact-py#177
- `worker_concurrency` is only available in a new constructor using an
object as input -- much easier to use as we add arguments to the class.

Sanity check on the distributed test:

![Screenshot 2025-01-23 at 18 03
15](https://github.com/user-attachments/assets/164f4a55-7dc5-4067-a28f-8fb587e38a92)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants