[BUG] One network device failure caused the restart proxy in loop until manually stopped.

[vanditpatel619]

Description
One network device failure caused the restart proxy in loop until manually stopped.

Setup
Delta proxy devices loading from MongoDB script. There are 35 devices in one delta proxy and it keep crashing and unable to start.

Please be as specific as possible and give set-up details.

• VM (Virtualbox, KVM, etc. please specify)

Steps to Reproduce the behavior

/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/client.py:79: DeprecationWarning: This module is deprecated. Please use salt.channel.client instead.
  warn_until(
[CRITICAL] Unexpected error while connecting to salt-prod-master-1.infra.vmware.com
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1135, in _connect_minion
    yield minion.connect_master(failed=failed)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1376, in connect_master
    yield self._post_master_init(master)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/opt/saltstack/salt/lib/python3.10/types.py", line 220, in send
    return self.__wrapped.send(val)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/metaproxy/deltaproxy.py", line 356, in post_master_init
    sub_proxy_data = subproxy_post_master_init(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/metaproxy/deltaproxy.py", line 524, in subproxy_post_master_init
    _proxy_minion.schedule.add_job(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/schedule.py", line 413, in add_job
    evt.fire_event(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 835, in fire_event
    self.pusher.send(msg)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/ipc.py", line 456, in send
    yield self.stream.write(pack)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/iostream.py", line 388, in write
    self._check_closed()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/iostream.py", line 926, in _check_closed
    raise StreamClosedError(real_error=self.error)
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/client.py:79: DeprecationWarning: This module is deprecated. Please use salt.channel.client instead.
  warn_until(

Expected behavior
Proxy should start gracefully

Versions Report
Salt Version:
Salt: 3005

[garethgreenaway]

@vanditpatel619 This is related to the bug related to using multiple threads when the number of devices configured for a control proxy is greater than 20, this causes an error to occur and the process will restart. If you can divide up your minions so that each control proxy has no more than 20 sub proxies, that should work around the issue until a proper fix can be implemented.

[s0undt3ch]

Fix will be available in the upcoming 3006.1 release