Error Log Spam on failed axon connections

[Shr1ftyy]

At the moment there's a ridiculous amount of spam from process_error_message when attempting to query non-responsive axons. This is especially a problem on validators as they often query many if not all miners in a subnet at a time.

...
4|validator  | 2024-11-28 03:59:24.427 |      ERROR       | bittensor:dendrite.py:244 | ClientConnectorError#9969eff7-b715-4529-930a-fec5f98712f1: Cannot connect to host 51.159.210.189:48090 ssl:default [Connect call failed ('51.159.210.189', 48090)]                           4|validator  | 2024-11-28 03:59:24.511 |      ERROR       | bittensor:dendrite.py:244 | ClientConnectorError#8af13ac9-c051-4363-b45e-cf2ace3cb106: Cannot connect to host 62.169.28.198:5000 ssl:default [Connect call failed ('62.169.28.198', 5000)]                               4|validator  | 2024-11-28 03:59:24.751 |      ERROR       | bittensor:dendrite.py:244 | ClientConnectorError#dd5dde6e-8412-4530-b2bc-f5a99c0dfeff: Cannot connect to host 37.27.84.127:15004 ssl:default [Connect call failed ('37.27.84.127', 15004)]                               
4|validator  | 2024-11-28 03:59:24.835 |      ERROR       | bittensor:dendrite.py:244 | ClientConnectorError#a928409c-9e8e-49ef-b27a-ec58da1e8919: Cannot connect to host 65.108.10.239:15051 ssl:default [Connect call failed ('65.108.10.239', 15051)]                             
4|validator  | 2024-11-28 03:59:25.083 |      ERROR       | bittensor:dendrite.py:244 | ClientConnectorError#583dcb77-9ec8-4e09-ba0c-d7377541a2e0: Cannot connect to host 213.173.105.82:34843 ssl:default [Connect call failed ('213.173.105.82', 34843)]                           4|validator  | 2024-11-28 03:59:25.253 |      ERROR       | bittensor:dendrite.py:244 | ClientOSError#e11224f6-5fbd-44fe-b525-2561ad17bebd: [Errno None] Can not write request body for http://173.231.14.254:10001/AllocateAssets                                                   4|validator  | 2024-11-28 03:59:26.594 |      ERROR       | bittensor:dendrite.py:244 | ClientConnectorError#77bd7979-96ff-4819-9e65-faa23e03cd40: Cannot connect to host 221.117.101.244:8092 ssl:default [Connect call failed ('221.117.101.244', 8092)]                           4|validator  | 2024-11-28 03:59:50.422 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#6cc48598-c5b3-48b5-b54d-b5e6b58600b6:                                                                                                                                           4|validator  | 2024-11-28 03:59:50.506 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#c575023c-1d39-4382-b192-206eae39423e:                                                                                                                                           4|validator  | 2024-11-28 03:59:50.592 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#5c955de3-c180-46c2-8d4a-64ad9f2b2c87:                                                                                                                                           4|validator  | 2024-11-28 03:59:50.675 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#c5eccffc-b616-48da-8e7b-c40eed19da07:                                                                                                                                           
4|validator  | 2024-11-28 03:59:51.421 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#3ad7d8bb-66b4-4c42-9f0d-fa1e0a74597e:                                                                                                                                           4|validator  | 2024-11-28 03:59:51.509 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#94e8d6cd-92e1-4b81-b6c1-d0f1ef03a32f:                                                                                                                                           4|validator  | 2024-11-28 03:59:51.597 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#5fb6446d-1d01-4667-915f-eec1c8010236:                                                                                                                                           4|validator  | 2024-11-28 03:59:51.682 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#7fbeec79-d7a1-403f-8566-c9d1b62b0b0d:                                                                                                                                           4|validator  | 2024-11-28 03:59:52.120 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#76d8425d-230c-4c4c-bb4d-9ba9d79069b2:                                                                                                                                           4|validator  | 2024-11-28 03:59:52.203 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#c53b5ffc-e164-4dd3-8c48-395f4092acf5:                                                                                                                                           
4|validator  | 2024-11-28 03:59:52.285 |      ERROR       | bittensor:dendrite.py:244 | TimeoutError#b927d75b-0626-4392-a03c-1bdb9ef02b27:                                                                                                                                           
...

Can we suppress these logs somehow? maybe add another logging level?

[garrett-opentensor]

Thanks for the feedback! Added this to our backlog

[garrett-opentensor]

Here is the PR that resolves this #2510