tBTC chain extension: Incporporating the new Ethereum event subscription API and background event pull loop

[pdyraga]

Refs #680

Depends on #671
Depends on keep-network/tbtc#750

Overview

There are two major changes to Ethereum subscriptions proposed here, as a result of the changes implemented in keep-network/keep-common#63:

• new subscription API with OnEvent and Pipe functions,
• background monitoring loop pulling past events from the chain.

The first change should allow implementing some handler logic easier and to avoid complex logic leading to bugs such as keep-network/keep-core#1333 or keep-network/keep-core#2052. ECDSA keep client is currently not that much affected by this chain but this may change when proper event deduplication will be implemented.

The second change should improve client responsiveness for operators running their nodes against Ethereum deployments that are not very reliable on the event delivery front. This should hopefully improve SLA of some mainnet operators of ECDSA client.

New API

Event subscription API has been refactored in keep-network/keep-common#63 to resemble the proposition from keep-network/keep-core#491. The new event subscription mechanism allows installing event callback handler function with OnEvent function as well as piping events from a subscription to a channel with Pipe function.

Example usage of OnEvent:

handlerFn := func(
    submittingMember common.Address,
    conflictingPublicKey []byte,
    blockNumber uint64,
) {
   // (...)
}
subscription := keepContract.ConflictingPublicKeySubmitted(
	nil, // default SubscribeOpts
	nil, // no filtering on submitting member
).OnEvent(handlerFn)

The same subscription but with a Pipe:

sink := make(chan *abi.BondedECDSAKeepConflictingPublicKeySubmitted)

subscription := keepContract.ConflictingPublicKeySubmitted(
	nil, // default SubscribeOpts
	nil, // no filtering on submitting member
).Pipe(sink)

Currently, all our event subscriptions in ECDSA client use function handlers and the code has been adjusted to the new API but is still using function handlers. This may or may not change in the future depending on an individual use case.

Background monitoring loop

Some nodes in the network are running against Ethereum setups that are not particularly reliable in delivering events. Events are not delivered, nodes are not starting key-generation, or are not participating in redemption signing. Another problem is the stability of the event subscription mechanism (see #663). If the web socket connection is dropped too often, the resubscription mechanism is not enough to receive events emitted when the connection was in a weird, stale state.

To address this problem, keep-network/keep-common#63 introduces a background loop periodically pulling past events from the chain next to a regular watchLogs subscription. How often events are pulled and how many blocks are taken into account can be configured with SubscribeOpts parameters.

This way, even if the event was lost by watchLogs subscription for whatever reason, it should be pulled by a background monitoring loop later. An extremely important implication of this change is that handlers should have logic in place allowing them to de-duplicate received events even if a lot of time passed between receiving the original event and the duplicate.

The original version of tBTC deposit monitoring code - without the modification here - was mostly safe. There is 24 hours monitoring period eliminating duplicate events within this time frame.

Just to be absolutely safe, in case 24 hours time period is exceeded for a duplicate, I added an initial state check in this PR (see ff611ec). Before the monitoring starts on start function signal, we check the initial deposit state whether
it's an expected one. If not, we do not start the monitoring.

[lukasz-zimnoch]

LGTM! I've tested and haven't found any problems. Specifically, I've checked:

• starting all the monitorings (retrieve pubkey, provide redemption signature, provide redemption proof) on appropriate events
• stopping the retrieve pubkey monitoring when the key has been retrieved by a third-party
• stopping the provide redemption signature monitoring when the signature has been provided by a third-party
• stopping the provide redemption proof monitoring when the proof has been provided by a third-party
• retrieving the public key on retrieve pubkey timeout
• submitting redemption signature on provide redemption signature timeout
• increasing redemption fee on provide redemption proof timeout
• resubscription and monitoring behavior after Ethereum node crash
• reaction on event duplicates
• usual keygen and signing