Next improvement of the libp2p network transport

[dnkolegov]

The source is the comments of #230:

1. Improve network transport #230 (comment)
2. Improve network transport #230 (comment)
3. Improve network transport #230 (comment)

[matejpavlovic]

Adding links to some more comments:

1. Improve network transport #230 (comment)
2. Improve network transport #230 (comment)

[matejpavlovic]

Suggested general architecture

The networking module knows the current membership, i.e., which nodes (IDs and addresses) it is supposed to communicate with. For each such node, it maintains one incoming and one outgoing byte stream. Each outgoing byte stream is associated with its own Dispatcher component. Each incoming stream is associated with a Receiver component. A Receiver is created upon an incoming connection from another node, but only if that node is part of the membership. The networking module also keeps a limited number of TentativeReceivers for incoming connections that are not part of the membership, for the case the membership has not yet been updated.

Dispatcher

A Dispatcher is a component that has a thread-safe FIFO buffer, a stream for outgoing messages, and a goroutine constantly reading messages from the buffer and writing them to the stream.

Writing in the buffer never blocks and the buffer keeps the most recently added messages. Thus, writing to a full buffer results in the oldest message in the buffer being dropped. An easy way to implement such a buffer is a buffered Go channel. If the writer, in a select statement, fails to write to the channel, it performs one read (making space for the write) and then tries to write again (in which case success is guaranteed).

The Dispatcher goroutine is responsible for establishing the network connection and sending messages from the buffer to it. If the connection breaks, the Dispatcher goroutine re-establishes it and continues sending messages.

Receiver

A receiver is implemented as a connection hander function. Its first action is to check whether the incoming connection is from a node in the membership. If that is the case, the Receiver simply continues by reading and deserializeing messages from the network and writing them in the module's output channel as Mir events. If the source of the connection is not in the membership, the receiver registers itself as a TentativeReceiver.

TentativeReceiver

A TentativeReceiver is like a Receiver, but buffers a limited number of most recently received messages instead of writing them directly to the module's output channel. When the membership of known to the module is updated to include the TentativeReceiver's source node, the TentativeReceiver flushes its buffer to the output and continues as a regular Receiver.

When too many (determined by a configuration parameter) TentativeReceivers exist in the module, the least recently created TentativeReceiver is destroyed (and its incoming stream is closed) whenever a new TentativeReceiver is created.

Updating the membership

Whenever the membership is updated (by calling Connect), the module creates the corresponding dispatchers (that create the network connections in the background) and, if necessary, TentativeReceivers are converted to Receivers. When closing old connections, the corresponding Dispatchers and Receivers are destroyed.

[matejpavlovic]

As a bonus, it would be even nicer to have this machinery abstracted and usable with any network transport (libp2p, grpc, raw TCP, ...) using an appropriate interface.