Reuse TCP connections for JSON-RPC calls within an ethereum node

[utzig]

When running multiple raiden nodes on a single Linux machine, somewhere between 20 and 30 nodes, will cause exhaustion of available TCP ports. The default for Linux is to use ports between 32768 - 61000 which gives a total of around 28K ports. Every time a new RPC connection is made new TCP port is opened for that connection and stays in TIME_WAIT state. The default recycle time for Linux is big enough that available ports are exhausted.

For doing orchestrated runs I added two extra sysctl configuration options:

net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1

That is good enough for running in orchestration but a better solution should be devised. The best option would probably be to reuse the same connections that were already opened before.

[utzig]

Btw, the error after exhausting available ports is this one:

ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8545): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x0000000006f51a28>: Failed to establish a new connection: [Errno 99] Cannot assign requested address',))

[hackaugusto]

So, we have these three dependencies:

• pyethapp.rpc_client.JSONRPCClient (used by raiden 0.0.4)
• tinyrpc.transport.http.HttpPostClientTransport (used by pyethapp 1.5.0)
• requests.api.request (used by tinyrpc 0.5)

The problem is that request (as of 2.12.4) creates a fresh Session for each high-level call. We need to patch tinyrpc to create a Session object instead of relying in the high-level request's api, by reusing a session we will benefit of connection reuse provided by urllib3 (if supported by the ethereum node)

[hackaugusto]

Note: The configuration is only a work around for orchestration, it must not be used with a standard raiden/ethereum node.

Reference: https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html

[LefterisJP]

Yes I also agree regarding the sysctl configuration.

Regarding tinyrpc that's quite strange that it does not support sessions at all. At least I can't see anything in its documentaiton. In that case maybe a less hackish idea would be to resort to a different library for RPC, one that would support connection reuse via sessions?

[konradkonrad]

I believe this should be raised against the tinyrpc upstream. I guess a PR there wouldn't hurt.

[konradkonrad]

see my PR mbr/tinyrpc#31 -- until this get's merged (and backported!), we can probably get away with monkey-patching send_message to mirror the PR's version on our side.