API not disconnecting from the node

[samelamin]

Hi team, we are experiencing an issue where we our UI is going down because the node reaches capacity very quickly

I suspect our api is not disconnecting after window close

I see someone else on stack exchange facing a similar issue?

https://substrate.stackexchange.com/questions/1995/dev-substrate-node-at-capacity-for-ws-connections-how-to-flush

We tried setting a timeout of 60 seconds

 const provider = new WsProvider(webSockAddr,undefined,undefined,60_000);
 const api = await ApiPromise.create({provider});

Sadly thats also not working. FYI, We are on version 8.3.1

    "@polkadot/api": "^8.3.1",
    "@polkadot/extension-dapp": "^0.42.4",

Any guidance on this? should we be managing the disconnect?

[jacogr]

The API will disconnect after the window is closed - the browser itself destroys the WebSocket object as well as the API objects creates, i.e. the page will not have resources attached after closing.

On the server side, on the TCP layer, a disconnect would be sent as soon as the connection is destoyed.

[samelamin]

thanks @jacogr that is what i suspected, but then that doesnt explain why we are filling up connections so quickly

This is the error from the node

Unable to build WebSocket connection WS Error <Capacity>: Unable to add another connection to the event loop. 

Is it at all possible the browser closing isnt actually disconnecting? Where can I find logs of the disconnect?

Also wouldnt it make sense to force the disconnect from the node, it seems like a potential DOS attack vector is someone floods the node with connections? 🤔

[jacogr]

I certainly cannot replicate that locally, my setup -

• node running with --dev
• open a browser tab
• see 2022-05-16 12:17:16 Accepting new connection, 1/100
• close browser (don't see any logs, but also not running with any trace=.... flags
• open new tab, reconnect
• see 2022-05-16 12:18:44 Accepting new connection, 1/100

So the node, at least on Substrate master, does drop correctly and the count matches up with my count of connections.

I really cannot comment on the DOS vecctor on the Node side, I'm not familiar with the Rust internals of the jsonrpc crate (or the new jsonrpsee crate that is being used as of last week)

[samelamin]

Yes I am also able to replicate, and it does indeed close the connection

2022-05-16 09:37:20.193 DEBUG ThreadId(32) parity_ws::io: WebSocket connection to 81.101.132.44:50011 disconnected.    
2022-05-16 09:37:21.021  INFO ThreadId(32) parity_ws::io: Accepted a new tcp connection from 81.101.132.44:50064.    
2022-05-16 09:37:21.145 DEBUG ThreadId(32) parity_ws::io: WebSocket connection to 81.101.132.44:50064 disconnected.    
2022-05-16 09:37:22.352  INFO ThreadId(32) parity_ws::io: Accepted a new tcp connection from 81.101.132.44:50070.  

For anyone facing a similar issue, to identify whats happening on your node please run it with the below flag --log=parity_ws::io=debug

@jacogr feel free to close this issue if you want, but given others are facing a similar issue perhaps its best to keep it open until its resolved 🤔

[jacogr]

Closing. This is not API related, there is no way the API can keep the connection open when the object is destroyed on the browser level, as explained above.

[samelamin]

No worries, I agree with you

Thank you so much!

[jacogr]

One thing to check though - very recently in Substrate master, the RPC has been changed to jsonrpsee - I would try and update to current master and see if you have similar experiences. (There were some unfixable issues on the older jsonrpc, the new one handled connections and certainly connection speed much better - it was quite a while for the team to get there, i.e. even in dev this new version were running on some public RPC nodes to have side-by-side compares)

[samelamin]

ah nice!

Do you mean on 0.9.19 or perhaps 0.9.20

We try not to use master because of dependency hell!

[jacogr]

For Polkadot, seems it will be in the 0.9.22 release, PR here - paritytech/substrate#8783 (I seem to have jumped the gun a bit with "it is released", although, like I said it has been running on a large number Kusama/Polkadot RPC nodes already for quite some time)

[jacogr]

Update to the update "when" - 0.9.22 will bring in jsonrpsee, see https://github.com/paritytech/polkadot/pull/5537/files

[polkadot-js-bot]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.