Retry & backoff mechanism in worker loop

[adizere]

Closes: #943
Follow-up to: #903

Description

• Adds the Fibonacci-based retry mechanism to method run_uni_chan_path()

• Ensure we don't retry for more than 2 seconds

• Fix the sending on a disconnected channel dangling error (details in the trace below)

  Show trace

  May 06 21:29:44.358 ERROR worker loop{worker=channel-0/transfer:ibc-1->ibc-0}: ibc_relayer::supervisor: failed during query to chain id ibc-0 with underlying error: RPC error to endpoint http://localhost:26657/: error trying to connect: tcp connect error: Connection refused (os error 61) (code: 0)
  May 06 21:29:44.359 ERROR worker loop{worker=channel-0/transfer:ibc-1->ibc-0}: ibc_relayer::supervisor: worker error: UnidirectionalChannelPath worker failed after 6 retries
  May 06 21:29:44.359 INFO worker loop{worker=channel-0/transfer:ibc-1->ibc-0}: ibc_relayer::supervisor: worker exits
  May 06 21:29:45.331 ERROR ibc_relayer::supervisor: [ibc-1] error during batch processing: sending on a disconnected channel
  May 06 21:29:46.469 ERROR ibc_relayer::supervisor: [ibc-1] error during batch processing: sending on a disconnected channel
  ...
  

• Ensure we can properly resume relaying. At the moment the relayer does not receive events from the chain that went down after the chain is restarted. Fixed: what happened was that the event monitor stopped retrying before the node went back up again.

• Do not stop the workers when the node goes down because otherwise we don't pick up on packets that were sent after the worker got stopped until some more packets are sent so that the worker starts again and clears them.

---

For contributor use:

• Updated the Unreleased section of CHANGELOG.md with the issue.
• If applicable: Unit tests written, added test to CI.
• Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
• Updated relevant documentation (docs/) and code comments.
• Re-reviewed Files changed in the Github PR explorer.

[ancazamfir]

Looks good, i tested by making the packet worker fail on send_packet event and successful in clearing packets. It switches nicely between event based relaying (passive mode) to packet clearing (active mode) and then back to event. Should be good for now.