Reconnection backoff & heartbeat mechanism

[bchamagne]

Description

• Exponential backoff on reconnect (capped at 24 hours)
• Wakeup when the other node connect to us
• Heartbeat to automatically close "stale" connections

Fixes #1323

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Tested locally with multiple nodes and turning them on and off.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[samuelmanzanera]

Do you think we should add a code_change to adapt the state, or prefer restarting the process ?

[Neylix]

I think you should keep the "wakup" functionnality be cause if a node goes offline for a while, all other nodes will have a reconnection each 24 hours.
But when the node goes online, others nodes may reconnect to it after 24 hours which is too long