File descriptor leak caused by clients causing system to get down

[tusharmakkar08]

I am facing this issue with prometheus-client==0.5.0 and Python 3.5.2. I am using start_http_server command to start the server.

To be precise my close_wait connections keep on increasing till my server stops serving requests.

x@ip-10-0-0-0:~$ lsof -i | grep 8002
python 10601 x 3u IPv4 78413488 0t0 TCP *:8002 (LISTEN)
python 10601 x 55u IPv4 82060563 0t0 TCP ip-10-0-0-0.ap-south-1.compute.internal:8002->ip-172-00-00-00.ap-south-1.compute.internal:45114 (CLOSE_WAIT)
python 10601 x 80u IPv4 82089551 0t0 TCP ip-10-0-0-0.ap-south-1.compute.internal:8002->ip-172-00-00-00.ap-south-1.compute.internal:45976 (CLOSE_WAIT)
python 10601 x 81u IPv4 82088475 0t0 TCP ip-10-0-0-0.ap-south-1.compute.internal:8002->ip-172-00-00-00.ap-south-1.compute.internal:46006 (CLOSE_WAIT)
python 10601 x 82u IPv4 82089875 0t0 TCP ip-10-0-0-0.ap-south-1.compute.internal:8002->ip-172-00-00-00.ap-south-1.compute.internal:46034 (CLOSE_WAIT)
python 10601 x 83u IPv4 82092200 0t0 TCP ip-10-0-0-0.ap-south-1.compute.internal:8002->ip-172-00-00-00.ap-south-1.compute.internal:46064 (CLOSE_WAIT)

This issue is similar to prometheus/jmx_exporter#327

Attaching strace here for the issue. https://ideone.com/0VORDa

[brian-brazil]

I don't see anything there related to Prometheus. Why do you think the issue is with the python client?

That's also not a complete strace.

[tusharmakkar08]

I think the issue is due to start_http_server functionality in the client. When I am running it via third-party http server (wsgi) there is no such issue.

[brian-brazil]

Can you get a full strace of one of those tcp connections?

[tusharmakkar08]

Thing is, as of now, I am unable to reproduce the issue. It happens once in 2 days. I would get strace of those connection whenever the issue occurs again.

[brian-brazil]

Did you manage to get more debug information?

[brian-brazil]

Closing as stale, given no other reports like this I presume the issue is elsewhere.