[Serve] Decrement ray_serve_deployment_queued_queries when client disconnects
#37965
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
|
@shrekris-anyscale is this needed in addition to #37939 ? |
|
Yes, that PR is tackling a different problem: the queue metric is not getting set when there's no traffic. This change is fixing the issue where queued requests that get disconnected aren't counted properly. As a sanity check, the unit test in this PR fails locally with just the changes in #37939. |
|
actually, i don't think my pr will fix the issue. we decrease the metrics and set it whenever the request is finished. We can proceed to checkin this pr, and close mine. |
|
@edoakes @sihanwang41 this PR is ready for review. |
sihanwang41
reviewed
Aug 1, 2023
python/ray/serve/_private/router.py
Outdated
|
|
||
| return result | ||
| return result | ||
| except asyncio.CancelledError: |
There was a problem hiding this comment.
would be an issue if different exceptions happens?
There was a problem hiding this comment.
Yes, it would. I changed the code to a try-finally block, so the metric is decremented no matter what exception is raised.
| "application": request_meta.app_name, | ||
| }, | ||
| ) | ||
| query = Query( |
There was a problem hiding this comment.
add try here, so that we don't need to have incremented_queue_metric.
There was a problem hiding this comment.
Good idea, that makes the code simpler. I made the change.
Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
edoakes
approved these changes
Aug 1, 2023
edoakes
pushed a commit
to edoakes/ray
that referenced
this pull request
Aug 2, 2023
…isconnects (ray-project#37965) The `ray_serve_deployment_queued_queries` metric tracks the number of queries that have yet to be assigned a replica. If a client disconnects before its query has been assigned a replica– but after the metric has counted their query– the query terminates, but the metric doesn't decrease. This change decrements `ray_serve_deployment_queued_queries` when a queued request is disconnected. Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>
8 tasks
rickyyx
pushed a commit
that referenced
this pull request
Aug 8, 2023
…isconnects (#37965) (#38020) The `ray_serve_deployment_queued_queries` metric tracks the number of queries that have yet to be assigned a replica. If a client disconnects before its query has been assigned a replica– but after the metric has counted their query– the query terminates, but the metric doesn't decrease. This change decrements `ray_serve_deployment_queued_queries` when a queued request is disconnected. Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com> Co-authored-by: shrekris-anyscale <92341594+shrekris-anyscale@users.noreply.github.com>
NripeshN
pushed a commit
to NripeshN/ray
that referenced
this pull request
Aug 15, 2023
…isconnects (ray-project#37965) The `ray_serve_deployment_queued_queries` metric tracks the number of queries that have yet to be assigned a replica. If a client disconnects before its query has been assigned a replica– but after the metric has counted their query– the query terminates, but the metric doesn't decrease. This change decrements `ray_serve_deployment_queued_queries` when a queued request is disconnected. Signed-off-by: NripeshN <nn2012@hw.ac.uk>
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
…isconnects (ray-project#37965) The `ray_serve_deployment_queued_queries` metric tracks the number of queries that have yet to be assigned a replica. If a client disconnects before its query has been assigned a replica– but after the metric has counted their query– the query terminates, but the metric doesn't decrease. This change decrements `ray_serve_deployment_queued_queries` when a queued request is disconnected. Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
vymao
pushed a commit
to vymao/ray
that referenced
this pull request
Oct 11, 2023
…isconnects (ray-project#37965) The `ray_serve_deployment_queued_queries` metric tracks the number of queries that have yet to be assigned a replica. If a client disconnects before its query has been assigned a replica– but after the metric has counted their query– the query terminates, but the metric doesn't decrease. This change decrements `ray_serve_deployment_queued_queries` when a queued request is disconnected. Signed-off-by: Victor <vctr.y.m@example.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
The
ray_serve_deployment_queued_queriesmetric tracks the number of queries that have yet to be assigned a replica. If a client disconnects before its query has been assigned a replica– but after the metric has counted their query– the query terminates, but the metric doesn't decrease.This change decrements
ray_serve_deployment_queued_querieswhen a queued request is disconnected.Related issue number
Closes #37943
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.test_telemetry.py.