I/O timeouts during bench test

[roman-khimov]

As the load appears on the test network I/O timeouts start to happen:

2022-10-14T06:21:19.568Z WARN peer disconnected {"addr": "172.200.0.3:20335", "error": "write tcp 172.200.0.6:41398->172.200.0.3:20335: i/o timeout", "peerCount": 2}

This leads to disconnects/reconnects and isn't particularly good for performance. While it's almost fine to have them (the system is under stress and it uses 2s blocks), this still can be a sign of some messages requiring reader thread to do more work than it ought to. We need to check message handlers and see if anything can be improved there to minimize chances for these timeouts/disconnects.

[roman-khimov]

Timeouts were mostly happening in getdata (more often) and inv handlers (#2750 + a specific timeout-tracking addition). After #2755 they're very rare in 4+1 scenario and even when they happen, they're less problematic. Seems like it was one node affecting the other, while some peer was processing transactions another node couldn't push more of them to it.

#2757 and #2759 also make timeouts less likely to happen because we now have one Write() call instead of 32 most of the time (and less stress on internal queues) and a specific potentially problematic Inv case can be handled easier. #2758 can prevent some traffic from happening which is also beneficial.

Things also tried, but less successful:

• reducing lock/unlock calls amount in the Inventory handler, only taking txInLock (and/or mempool) once per inv (a set of 32 hashes) --- this in fact degrades the performance, various combinations were tried (on top of different other changes), but they all work worse
• adding more cache to P2P message queue, but that's largely obsolete after Improve P2P transaction handling #2755, we no longer have 32 distinct messages to push into the queue in GetData handler
• re-adding peer shuffling removed by Rework broadcast logic #2741, as expected it's absolutely irrelevant with the new iteratePeersWithSendMsg logic
• adding a global cache tracker with per-peer data inside to track known hashes and only request transactions once in normal scenario (a bit similar to C# pendingKnownHashes, but with a circular buffer and re-requests from other peers if we get no real transaction from the first announcer) --- surprisingly, it horribly breaks performance, our current dumb code works faster (and is more reliable)

I think that's enough for this task, we have a better networking subsystem now and timeouts are less likely to happen.