status.master is slow, CPU intensive with high number of TCP connections

[cifvts]

Description of Issue

When the number of TCP connections get high, salt.utils.network._netlink_tool_remote_on becomes really slow and CPU intensive.
The functions is used by status.master which is called as a scheduled function to test Salt Master availability.

Setup

We noticed the issue on servers with ~100k TCP connections (using ss -ant | wc -l

Versions Report

Salt Version:
           Salt: 2018.3.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.24.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: 1.3.7
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.4 
         locale: UTF-8
        machine: x86_64
        release: 4.14.105-mptcp-kp2-3f01458b
         system: Linux
        version: debian 9.4 

[cmcmarrow]

We have a TCP leak right now. Thank you for bring this up. I will try to fix the leak in status.master and bring down the TCP connection count. This should also bring down the CPU workload.

[cmcmarrow]

#53581 seems to be only part of the fix

[cmcmarrow]

once #53581 is merged I will fix the leak.

[cifvts]

@cmcmarrow is there an issue open for the TCP leak? I would be interested to look at it and find if we hit the problem in our setup.

[cmcmarrow]

@cifvts TCP does leak. We are currently trying to find and fix them. We have not made an issue for it. I know if you run the salt test on windows you can see the TCP count go way up. Right now we are trying to find all the tcp leaks and patch them. The other goal is to lower the number of made TCP connects by reusing them when it is safe. If we do this in salt.utils.network._netlink_tool_remote_on I believe this would help lower your cpu work load too.

[cifvts]

I see what you mean. The workload with the patch is completely gone. For the TCP, I will be happy to look at it since we have thousands of them in our production env.

[cmcmarrow]

A fix should come out in the next update! I'll link it once I get the fix