Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prefect Server returns 503 when updating ConcurrencyLimit for a Deployment without a ConcurrencyLimit #17227

Closed
marcm-ml opened this issue Feb 21, 2025 · 7 comments · Fixed by #17278
Labels
bug Something isn't working

Comments

@marcm-ml
Copy link

marcm-ml commented Feb 21, 2025

Bug summary

See title. I am using the prefect server python SDK. Not the CLI nor prefect.yaml. I create the deployment programmatically in python using native prefect APIs.

Reproduction: When running prefect-server, create a deployment without a concurrency limit defined first. send that to the API. then do a deployment update where you add a concurrency update. Send that to the API again and then the client call with retunr 503 and the prefect-server throws an exception (See below)

This also happens when updating an existing deployment's concurrency limit. so it always happens when you change something with the concurrency limit.

Version info

Version:             3.2.6
API version:         0.8.4
Python version:      3.12.5
Git commit:          5ceb3ada
Built:               Wed, Feb 19, 2025 9:24 PM
OS/Arch:             darwin/arm64
Profile:             local
Server type:         server
Pydantic version:    2.10.6
Integrations:
  prefect-gitlab:    0.3.1
  prefect-kubernetes: 0.5.3

Additional context

Client Traceback:

Sending deployment to PREFECT API
Traceback (most recent call last):
  File "/code/prefect/flows/flow_run_retention.py", line 256, in <module>
    app()
  File "/code/prefect/.venv/lib/python3.12/site-packages/typer/main.py", line 342, in __call__
    raise e
  File "/code/prefect/.venv/lib/python3.12/site-packages/typer/main.py", line 325, in __call__
    return get_command(self)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/typer/core.py", line 738, in main
    return _main(
           ^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/typer/core.py", line 197, in _main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/typer/main.py", line 707, in wrapper
    return callback(**use_params)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/flows/flow_run_retention.py", line 247, in deploy
    deployment.apply()  # type: ignore
    ^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 351, in coroutine_wrapper
    return run_coro_as_sync(ctx_call())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 207, in run_coro_as_sync
    return call.result()
           ^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 329, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 192, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/uv/python/cpython-3.12.5-macos-aarch64-none/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/_internal/concurrency/calls.py", line 402, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 188, in coroutine_wrapper
    return await task
           ^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/utilities/asyncutils.py", line 341, in ctx_call
    result = await async_fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/deployments/runner.py", line 436, in apply
    return await self._update(deployment.id, client)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/deployments/runner.py", line 367, in _update
    await client.update_deployment(
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/client/orchestration/_deployments/client.py", line 711, in update_deployment
    await self.request(
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/client/orchestration/base.py", line 53, in request
    return await self._client.send(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/client/base.py", line 354, in send
    response.raise_for_status()
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/client/base.py", line 162, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Server error '503 Service Unavailable' for url 'http://localhost:4200/api/deployments/fe36db29-3937-46be-a436-cf42944510d8'
Response: {'exception_message': 'Service Unavailable'}
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503

Server Traceback:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/code/prefect/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/server/api/server.py", line 152, in __call__
    await self.app(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/routing.py", line 460, in handle
    await self.app(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
    await responder(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/code/prefect/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/server/utilities/server.py", line 50, in handle_response_scoped_depends
    response = await default_handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/server/api/deployments.py", line 301, in update_deployment
    result = await models.deployments.update_deployment(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/server/models/deployments.py", line 287, in update_deployment
    await _create_or_update_deployment_concurrency_limit(
  File "/code/prefect/.venv/lib/python3.12/site-packages/prefect/server/models/deployments.py", line 322, in _create_or_update_deployment_concurrency_limit
    session.add(deployment)
  File "/code/prefect/.venv/lib/python3.12/site-packages/sqlalchemy/ext/asyncio/session.py", line 1154, in add
    return self._proxied.add(instance, _warn=_warn)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/prefect/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 3481, in add
    self._save_or_update_state(state)
  File "/code/prefect/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 3511, in _save_or_update_state
    self._save_or_update_impl(st_)
  File "/code/prefect/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4203, in _save_or_update_impl
    self._update_impl(state)
  File "/code/prefect/.venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4173, in _update_impl
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: Instance '<DeploymentSchedule at 0x15f234f50>' has been deleted.  Use the make_transient() function to send this object back to the transient state.
@marcm-ml marcm-ml added the bug Something isn't working label Feb 21, 2025
@marcm-ml
Copy link
Author

Manually updating the concurrency limit within the UI works just fine. After that the deployment script via python also works again. Which i find strange since I expect the UI to call the same endpoint basically

@cicdw
Copy link
Member

cicdw commented Feb 21, 2025

@marcm-ml can you share the code you used to reproduce this? Would help us solve it faster

@marcm-ml
Copy link
Author

marcm-ml commented Feb 25, 2025

@cicdw sure, here you go:

script.py:

import argparse
from datetime import timedelta

from prefect import flow
from prefect.client.schemas.actions import DeploymentScheduleCreate
from prefect.client.schemas.objects import ConcurrencyLimitConfig
from prefect.client.schemas.schedules import IntervalSchedule
from prefect.deployments.runner import RunnerDeployment

parser = argparse.ArgumentParser()
parser.add_argument("--concurrency", action="store_true", default=False)


@flow
def my_flow():
    print("test")


def main(concurrency: bool):
    deployment = RunnerDeployment.from_flow(
        my_flow,
        name="MyDeployment",
        concurrency_limit=ConcurrencyLimitConfig(limit=1) if concurrency else None,
        schedules=[DeploymentScheduleCreate(schedule=IntervalSchedule(interval=timedelta(hours=1)))],
    )
    deployment.apply()


if __name__ == "__main__":
    args = parser.parse_args()
    main(args.concurrency)

terminal:

prefect server start --port 4200 --background
python script.py
python script.py --concurrency  # <-- should crash
prefect server stop

I noticed that it is crucial to define a schedule on the deployment. otherwise it works just fine

@cicdw
Copy link
Member

cicdw commented Feb 25, 2025

Thank you, this is helpful - I think I know what might be happening; will report back

@cicdw
Copy link
Member

cicdw commented Feb 25, 2025

OK I think I figured it out but it will require a fix to get merged; in the meantime, if you add a slug to your schedule it will not trigger this edge case:

...
schedules=[DeploymentScheduleCreate(slug="hourly", schedule=IntervalSchedule(interval=timedelta(hours=1)))],
...

@marcm-ml
Copy link
Author

Thanks for the quick answer. Does the slug need to be unique for all deployments, only the one deployment that it is attached to or not at all?

@cicdw
Copy link
Member

cicdw commented Feb 25, 2025

Only for the one deployment that it is attached to 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants