feat(lib/grandpa): ensure catch-up logic works

[kishansagathiya]

Changes

• ensure catch-up logic works when interacting with other gossamer nodes
• If round in the neighbour message is ahead of our current round by a
  threshold, send a catch up request
• process the catch up response, if we can't process it at the moment,
  store it to process later.
• tests

Tests

• Run gossamer devnet (https://hackmd.io/@nZ-twauPRISEa6G9zg3XRw/B1RFO3uDt).
• I tried with 4 peers. Let them run for a few minutes, let few rounds finish.
• Stop one of those peers (something other than babe lead). Wait for few more rounds to finish (at least 2 more rounds).
• Start the node you stopped earlier.
• Check through logs that it sends a catch up request to other nodes and catches up to the latest round

Issues

• lib/grandpa: ensure catch-up logic works when interacting with substrate nodes #1531

Primary Reviewer

• @noot

[noot]

basically looks good, the main problem i see is that handling the response isn't thread safe, there is a very high chance we will be sending out multiple requests and getting multiple responses at handling them concurrently. you could add something to check if catch up is currently in progress already, and if so, then don't send out any more requests (see https://github.com/ChainSafe/gossamer/pull/1554/files) it would also be nice to split up the catch-up logic into its own file/module. it would also be nice to make catch up synchronous (the way block request-responses are), this would also prevent multiple requests/responses being sent out/handled at once unnecessarily

[kishansagathiya]

While, testing out a gossamer network of 4 node nodes, nodes stop with runtime: unknown pc I am continuing to investigate this, but posting the error here for reference if anyone else has any idea
https://gist.github.com/kishansagathiya/6d10d7823403b489481623129d3b663c

I tried eclecio's networking race condition PR with this branch (#2105), but that did not fix the problem.

[timwu20]

@kishansagathiya can you make this a draft PR if you don't intend to merge anytime soon?

[kishansagathiya]

too old, closing this.