Cassandra jmx exporter connections remain on CLOSE_WAIT state

[amruta1989mohite]

jmx aexporter's service not closing the connection and so after some time prometheus is not able to scrape the metrices.
I have the same setup in all my 3 environments Dev, Staging and Production.
Currently my production is running fine but Dev and Staging both are giving this issue, I wonder as time pass whether this issue will pop up in prod as well.
On running netstat command I get output like following:

 netstat -a | grep 7070

tcp        0      0 *:7070                  *:*                     LISTEN
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:46110 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:43922 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:44346 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:50798 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:57290 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:42382 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:43710 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:57498 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:44782 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:51218 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:42172 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:38786 CLOSE_WAIT
tcp        0 301429 ip-123-45-67-986.e:7070 ip-172-20-88-168.:45664 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:32778 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:50756 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:54184 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:48998 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:40436 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:41078 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:55330 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:55530 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:50128 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:51478 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:48524 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:52122 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:40170 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:53712 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:48144 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:48772 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:49432 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:60076 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:50592 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:42856 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:53034 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:55728 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:47008 CLOSE_WAIT
tcp      263      0 ip-123-45-67-986.e:7070 ip-123-45-67-986:41950 CLOSE_WAIT
tcp        0 301429 ip-123-45-67-986.e:7070 ip-123-45-67-986:40016 CLOSE_WAIT
tcp        0 301429 ip-123-45-67-986.e:7070 ip-123-45-67-986:39788 CLOSE_WAIT
tcp      335      0 ip-123-45-67-986.e:7070 ip-172-20-88-168.:57272 CLOSE_WAIT

when I take count it gives incremental numbers everytime, like:

 netstat -a | grep 7070 | wc -l
355

And this count goes on increasing.
looks like several threads of Cassandra stacktraces are blocked on write, while one of them constantly generating something that looks like an error log.
I don't see nothing in the cassandra log, which might be related to the problem.

I have no idea what should I do to fix this. Please suggest the solution.
I haven't changed anything specifically cassandra or jmx exporter in my cluster.

Cassandra version: 3.7
cqlsh version: 5.0.1
java version: 1.8.0_151
jmx exporter version: 0.3.0

[brian-brazil]

Dupe of #327

[amruta1989mohite]

You cant close the issue. There is no solution given #327 in this issue.
I don't know the process running jmx exporter as the http server and how to scrape metrices using it.
If you are considering that is the solution please elaborate that solution.

[sachingera2007]

Hi..
we are also facing the same issue ...
so many session in close wait ..How to troubleshoot this

apache-cassandra-3.11.4
jmx_prometheus_javaagent-0.12.0.jar

netstat -anp | grep 7070

LISTEN 11646/java
CLOSE_WAIT 11646/java
LAST_ACK -
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
ESTABLISHED 11646/java
ESTABLISHED 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
LAST_ACK -
LAST_ACK -
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java

[sachingera2007]

LISTEN 11646/java
ESTABLISHED 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
ESTABLISHED 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
LAST_ACK -
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
LAST_ACK -
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java
CLOSE_WAIT 11646/java