uvicorn not closing sockets

[johnrtipton]

We are using the latest uvicorn 0.15. It seems that after we upgraded to uvicorn 0.15 that our AWS load balancer health checks fail after a certain amount of time. Upon further investigation we found that all of our sockets are being used up because they are staying open in a CLOSE_WAIT status.

# netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n
   1 Foreign
   1 LISTEN
   1 established)
   7 TIME_WAIT
  13 ESTABLISHED
7047 CLOSE_WAIT

Workaround

For now we have switched our worker from UvicornWorker to UvicornH11Worker and it has resolved our issue.

Environment

Running uvicorn 0.15.0 with CPython 3.7.12 on Linux

[Kludex]

I don't see how the change of worker class would solve this issue 🤔

[pacoyang]

I don't see how the change of worker class would solve this issue 🤔

It maybe because h11 will close the transport when if the connection is invalid HTTP request, this issue similarity to #1192
https://github.com/encode/uvicorn/blob/master/uvicorn/protocols/http/h11_impl.py#L140

[johnrtipton]

Yes @pacoyang this seems to be exactly the problem we are seeing. Thanks!

[Kludex]

@johnrtipton from which version did you guys upgrade? 0.13.4?

[johnrtipton]

@Kludex We upgraded from 0.14.0

[shahar-medigate]

We encountered the issue in our systems as well, a simple fastapi service behind AWS EKS load-balancer
we also managed to reproduce the issue to a single commit
960d465

A simple program to reproduce

import uvicorn

async def app(scope, receive, send):
    assert scope["type"] == "http"

    await send(
        {
            "type": "http.response.start",
            "status": 200,
            "headers": [
                [b"content-type", b"text/plain"],
            ],
        }
    )
    await send(
        {
            "type": "http.response.body",
            "body": b"Hello, world!",
        }
    )


if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", reload=False, log_level=5)

and run nc -z localhost:8000
now you will have a new CLOSE_WAIT connection with netstat

apparently, it got something to do with uvloop don't call connection_lost or something
we bypass the issue by setting loop="asyncio" to the run function

[Anton-2]

Same thing here using Koyeb: the health check is also a TCP connection with no data, and ends with a socket in CLOSE_WAIT state. After some time, the service stops responding due to resource exhaustion.
In the same conditions, uvicorn with #1332 works flawlessly.

[jalaziz]

Looks like #1332 was released with 0.17.1 and this can be closed?

[pkit]

Dunno, why it is closed, still happens all the time.

[pkit]

Lol, reverted in 2022.

[Kludex]

This issue was solved. If you have a similar issue, please create a discussion.