Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flask test_client always returns 404 in connexion 3.0.3 #1826

Closed
neumann-nico opened this issue Dec 4, 2023 · 10 comments
Closed

Flask test_client always returns 404 in connexion 3.0.3 #1826

neumann-nico opened this issue Dec 4, 2023 · 10 comments

Comments

@neumann-nico
Copy link

neumann-nico commented Dec 4, 2023

Description

I migrated my code from connexion 2 to connexion 3 and I really love the upgrade!
My code works well, but I have a problem with the underlying test_client.
The flask test_client always returns 404. I tried get/post/put/etc. on different routes but all of them return 404 even though the routes are working in the real app.

When I use the connexion test client (connexion_app.test_client() instead of connexion_app.app.test_client()) it kind of works, but I get a different behaviour.
E.g. The passed body in my function is of type bytes instead of dict.
Also in the test the app expects different parameters, e.g. app.post(..., query_string="...") vs. app.post(..., params="..."), but that's not a big problem.

Is this a bug in connexion 3 that the underlying flask test_client does not work or do I need to migrate something in the test client? I would be also okay to use the connexion test_client, but only if the behaviour is the same as in the real app, which is currently not the case.

Documentation that I found about the test_client: https://connexion.readthedocs.io/en/latest/testing.html

I provided a minimal example which works fine with connexion 2 but does not work with connexion 3.

Expected behaviour

====== test session starts =====
test_app.py .
====== 1 passed ======

Actual behaviour

Summary of the log (full log below)

====== FAILURES ====== 
______ test_route ______
>           raise NotFound() from None
E           werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/werkzeug/routing/map.py:624: NotFound

During handling of the above exception, another exception occurred:

    def test_route():
        app_client = create_app((Path(os.getcwd()) / "api.yml").as_posix())
>       response = app_client.get("/")

    def _http_exception(self, exc: werkzeug.exceptions.HTTPException):
        """Reraise werkzeug HTTPExceptions as starlette HTTPExceptions"""
>       raise starlette.exceptions.HTTPException(exc.code, detail=exc.description)
E       starlette.exceptions.HTTPException: 404

FAILED test_app.py::test_route - starlette.exceptions.HTTPException: 404
Full log
❯ pytest test_app.py
=========================================================== test session starts ============================================================
platform darwin -- Python 3.10.11, pytest-7.4.3, pluggy-1.2.0
rootdir: /Users/niconeumann/connexion_test
plugins: asyncio-0.21.1, mock-3.12.0, anyio-3.7.1, dash-2.14.2
asyncio: mode=strict
collected 1 item                                                                                                                           

test_app.py F                                                                                                                        [100%]

================================================================= FAILURES =================================================================
________________________________________________________________ test_route ________________________________________________________________

self = <Flask 'test_app'>

    def full_dispatch_request(self) -> Response:
        """Dispatches the request and on top of that performs request
        pre and postprocessing as well as HTTP exception catching and
        error handling.
    
        .. versionadded:: 0.7
        """
        self._got_first_request = True
    
        try:
            request_started.send(self, _async_wrapper=self.ensure_sync)
            rv = self.preprocess_request()
            if rv is None:
>               rv = self.dispatch_request()

../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/app.py:867: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/app.py:841: in dispatch_request
    self.raise_routing_exception(req)
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/app.py:450: in raise_routing_exception
    raise request.routing_exception  # type: ignore
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/ctx.py:353: in match_request
    result = self.url_adapter.match(return_rule=True)  # type: ignore
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <werkzeug.routing.map.MapAdapter object at 0x1058f7430>, path_info = '/', method = 'GET', return_rule = True, query_args = {}
websocket = False

    def match(
        self,
        path_info: str | None = None,
        method: str | None = None,
        return_rule: bool = False,
        query_args: t.Mapping[str, t.Any] | str | None = None,
        websocket: bool | None = None,
    ) -> tuple[str | Rule, t.Mapping[str, t.Any]]:
        """The usage is simple: you just pass the match method the current
        path info as well as the method (which defaults to `GET`).  The
        following things can then happen:
    
        - you receive a `NotFound` exception that indicates that no URL is
          matching.  A `NotFound` exception is also a WSGI application you
          can call to get a default page not found page (happens to be the
          same object as `werkzeug.exceptions.NotFound`)
    
        - you receive a `MethodNotAllowed` exception that indicates that there
          is a match for this URL but not for the current request method.
          This is useful for RESTful applications.
    
        - you receive a `RequestRedirect` exception with a `new_url`
          attribute.  This exception is used to notify you about a request
          Werkzeug requests from your WSGI application.  This is for example the
          case if you request ``/foo`` although the correct URL is ``/foo/``
          You can use the `RequestRedirect` instance as response-like object
          similar to all other subclasses of `HTTPException`.
    
        - you receive a ``WebsocketMismatch`` exception if the only
          match is a WebSocket rule but the bind is an HTTP request, or
          if the match is an HTTP rule but the bind is a WebSocket
          request.
    
        - you get a tuple in the form ``(endpoint, arguments)`` if there is
          a match (unless `return_rule` is True, in which case you get a tuple
          in the form ``(rule, arguments)``)
    
        If the path info is not passed to the match method the default path
        info of the map is used (defaults to the root URL if not defined
        explicitly).
    
        All of the exceptions raised are subclasses of `HTTPException` so they
        can be used as WSGI responses. They will all render generic error or
        redirect pages.
    
        Here is a small example for matching:
    
        >>> m = Map([
        ...     Rule('/', endpoint='index'),
        ...     Rule('/downloads/', endpoint='downloads/index'),
        ...     Rule('/downloads/<int:id>', endpoint='downloads/show')
        ... ])
        >>> urls = m.bind("example.com", "/")
        >>> urls.match("/", "GET")
        ('index', {})
        >>> urls.match("/downloads/42")
        ('downloads/show', {'id': 42})
    
        And here is what happens on redirect and missing URLs:
    
        >>> urls.match("/downloads")
        Traceback (most recent call last):
          ...
        RequestRedirect: http://example.com/downloads/
        >>> urls.match("/missing")
        Traceback (most recent call last):
          ...
        NotFound: 404 Not Found
    
        :param path_info: the path info to use for matching.  Overrides the
                          path info specified on binding.
        :param method: the HTTP method used for matching.  Overrides the
                       method specified on binding.
        :param return_rule: return the rule that matched instead of just the
                            endpoint (defaults to `False`).
        :param query_args: optional query arguments that are used for
                           automatic redirects as string or dictionary.  It's
                           currently not possible to use the query arguments
                           for URL matching.
        :param websocket: Match WebSocket instead of HTTP requests. A
            websocket request has a ``ws`` or ``wss``
            :attr:`url_scheme`. This overrides that detection.
    
        .. versionadded:: 1.0
            Added ``websocket``.
    
        .. versionchanged:: 0.8
            ``query_args`` can be a string.
    
        .. versionadded:: 0.7
            Added ``query_args``.
    
        .. versionadded:: 0.6
            Added ``return_rule``.
        """
        self.map.update()
        if path_info is None:
            path_info = self.path_info
        if query_args is None:
            query_args = self.query_args or {}
        method = (method or self.default_method).upper()
    
        if websocket is None:
            websocket = self.websocket
    
        domain_part = self.server_name
    
        if not self.map.host_matching and self.subdomain is not None:
            domain_part = self.subdomain
    
        path_part = f"/{path_info.lstrip('/')}" if path_info else ""
    
        try:
            result = self.map._matcher.match(domain_part, path_part, method, websocket)
        except RequestPath as e:
            # safe = https://url.spec.whatwg.org/#url-path-segment-string
            new_path = quote(e.path_info, safe="!$&'()*+,/:;=@")
            raise RequestRedirect(
                self.make_redirect_url(new_path, query_args)
            ) from None
        except RequestAliasRedirect as e:
            raise RequestRedirect(
                self.make_alias_redirect_url(
                    f"{domain_part}|{path_part}",
                    e.endpoint,
                    e.matched_values,
                    method,
                    query_args,
                )
            ) from None
        except NoMatch as e:
            if e.have_match_for:
                raise MethodNotAllowed(valid_methods=list(e.have_match_for)) from None
    
            if e.websocket_mismatch:
                raise WebsocketMismatch() from None
    
>           raise NotFound() from None
E           werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/werkzeug/routing/map.py:624: NotFound

During handling of the above exception, another exception occurred:

    def test_route():
        app_client = create_app((Path(os.getcwd()) / "api.yml").as_posix())
>       response = app_client.get("/")

test_app.py:20: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/werkzeug/test.py:1160: in get
    return self.open(*args, **kw)
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/testing.py:232: in open
    response = super().open(
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/werkzeug/test.py:1114: in open
    response = self.run_wsgi_app(request.environ, buffered=buffered)
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/werkzeug/test.py:986: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/werkzeug/test.py:1262: in run_wsgi_app
    app_rv = app(environ, start_response)
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/app.py:1478: in __call__
    return self.wsgi_app(environ, start_response)
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/app.py:1458: in wsgi_app
    response = self.handle_exception(e)
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/app.py:1455: in wsgi_app
    response = self.full_dispatch_request()
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/app.py:869: in full_dispatch_request
    rv = self.handle_user_exception(e)
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/flask/app.py:759: in handle_user_exception
    return self.ensure_sync(handler)(e)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <connexion.apps.flask.FlaskApp object at 0x1058cace0>, exc = <NotFound '404: Not Found'>

    def _http_exception(self, exc: werkzeug.exceptions.HTTPException):
        """Reraise werkzeug HTTPExceptions as starlette HTTPExceptions"""
>       raise starlette.exceptions.HTTPException(exc.code, detail=exc.description)
E       starlette.exceptions.HTTPException: 404

../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/apps/flask.py:245: HTTPException
============================================================= warnings summary =============================================================
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/json_schema.py:16
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/json_schema.py:16
  /Users/niconeumann/.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/json_schema.py:16: DeprecationWarning: jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
    from jsonschema import Draft4Validator, RefResolver

../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/json_schema.py:17
  /Users/niconeumann/.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/json_schema.py:17: DeprecationWarning: jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
    from jsonschema.exceptions import RefResolutionError, ValidationError  # noqa

../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/validators/form_data.py:4
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/validators/form_data.py:4
  /Users/niconeumann/.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/validators/form_data.py:4: DeprecationWarning: Accessing jsonschema.draft4_format_checker is deprecated and will be removed in a future release. Instead, use the FORMAT_CHECKER attribute on the corresponding Validator.
    from jsonschema import ValidationError, draft4_format_checker

../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/validators/json.py:6
../../../../.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/validators/json.py:6
  /Users/niconeumann/.pyenv/versions/3.10.11/lib/python3.10/site-packages/connexion/validators/json.py:6: DeprecationWarning: Accessing jsonschema.draft4_format_checker is deprecated and will be removed in a future release. Instead, use the FORMAT_CHECKER attribute on the corresponding Validator.
    from jsonschema import Draft4Validator, ValidationError, draft4_format_checker

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================= short test summary info ==========================================================
FAILED test_app.py::test_route - starlette.exceptions.HTTPException: 404
====================================================== 1 failed, 7 warnings in 0.27s =======================================================

Steps to reproduce

Works fine with:

  • pip install connexion==2.14.2 flask==2.2.5

Does not work:

  • pip install connexion==3.0.3 flask==2.2.5
  • pip install connexion==3.0.3 flask==3.0.0

Execute the test: pytest test_app.py

api.yml

openapi: "3.0.0"
info:
  description: "..."
  version: "1.0.0"
  title: "..."

paths:

  /:
    get:
      operationId: route.index
      responses:
        "200":
          description: "..."

route.py

def index():
    return "Hello World!"

test_app.py

import os
from pathlib import Path

import yaml

from connexion import FlaskApp


def create_app(spec_path: str):
    with open(spec_path, "r") as f:
        specs = yaml.safe_load(f)

    connexion_app = FlaskApp(__name__, specification_dir="")
    connexion_app.add_api(specs)
    return connexion_app.app.test_client()


def test_route():
    app_client = create_app((Path(os.getcwd()) / "api.yml").as_posix())
    response = app_client.get("/")
    assert response.status_code == 200
    assert response.json == "Hello World!"

Additional info:

Output of the commands:

  • python --version: Python 3.10.11
  • pip show connexion | grep "^Version\:": Version: 3.0.3
  • pip show starlette | grep "^Version\:": Version: 0.27.0 and Version: 0.33.0
@neumann-nico neumann-nico changed the title Flask test_client always returns 404 in connexion 3.0 Flask test_client always returns 404 in connexion 3.0.3 Dec 4, 2023
@SrSoto
Copy link

SrSoto commented Dec 5, 2023

Happenning the same here but using Swagger UI, so the problem may not be specific to test_client but to general API routing.

What's more, maybe issue #1823 may help by fixing starlette version

@neumann-nico
Copy link
Author

Happenning the same here but using Swagger UI, so the problem may not be specific to test_client but to general API routing.

What's more, maybe issue #1823 may help by fixing starlette version

Thanks for your reply!
I tried multiple versions of starlette 0.27, 0.32.0.post1 and 0.33 but it did not help.

@cybertiger
Copy link

This is because Connexion lazily initializes the routing when the first ASGI call is received.

Workaround:

            # Workaround for race condition in connexion initialization
            # pylint: disable=protected-access
            connexion_app.middleware.app, connexion_app.middleware.middleware_stack = \
                connexion_app.middleware._build_middleware_stack()

Excuse the comment, (it also causes a race condition in startup if you're using werkzeug with threads, and a wsgi-asgi converter).

Correct solution is probably to convert your unit tests to use the provided startlette TestClient, but that involves some work as the API is significantly different. connexion_app.test_client()

@RobbeSneyders
Copy link
Member

Thanks for the reports everyone. I just submitted #1828 to set an upperbound to our Starlette dependency. We'll release it as soon as possible and will investigate the underlying issue afterwards.

@RobbeSneyders
Copy link
Member

The issue is being discussed on the starlette repo here.

RobbeSneyders added a commit that referenced this issue Dec 6, 2023
Temporary fix for #1826

We should release this asap.
@RobbeSneyders
Copy link
Member

3.0.4 has been released containing this fix.

@RobbeSneyders
Copy link
Member

Closing in favor of #1824

@neumann-nico
Copy link
Author

@RobbeSneyders Thanks for the quick update!
Unfortunately it also does not work with connexion 3.0.4 and starlette < 0.33.
In #1824 there is a different error which seems to be fixed with the pinned starlette version.

Same error as in the first post:
werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

@RobbeSneyders
Copy link
Member

Are you still using the Flask test client? Because that is not expected to work anymore. You should use the Connexion test client.

@neumann-nico
Copy link
Author

neumann-nico commented Dec 6, 2023

Are you still using the Flask test client? Because that is not expected to work anymore. You should use the Connexion test client.

Yes I tried the Flask test client. Alright thanks, I just switched to the connexion test client and got it working with POST/PUT request. For anyone interested:

Before (Flask test client):

response = app.put(
    "/myroute",
    content_type="application/json",
    data=json.dumps(body),
    query_string={"param": param},
)

After (Connexion test client):

response = app.put(
    "/myroute",
    headers={"Content-Type": "application/json"},
    data=json.dumps(body),
    params={"param": param},
)

There are also some changes needed with checking the response, but I will figure it out :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants