feat(sync): add BufferStage
#2066
Conversation
|
This also doesn't really account for something weird like |
|
It remains to be seen how well this will fit with all of the use cases, but looks promising 👍 |
Abstracts over the common scenario of buffering a stream of items into a stream of blocks.
This stage is used to split a stream of transactions into a stream of block's of transactions.
Remove BufferStage::T since it wasn't linked to usage directly. Reword BufferStage::Meta -> AdditionalData Co-authored-by: Nikša Sporin <niksa@equilibrium.co>
7c374ef to
bd9dab1
Compare
|
Sounded great; has a pretty fatal flaw in practice. A source cannot be separate from the process of transforming the input item stream into a stream of block data (which is what As it stands, if the origin p2p stream fails, or ends early there is no way for the source to know at which block number it should re-attempt from - only the buffer stage knows how many blocks of data were actually processed. This implies that the source stream and the transformation stage should instead be a single unified system. |
This PR adds the
BufferStagetrait to the sync stages framework.This trait and its associated
SyncReceiver::buffermethod abstracts over the common pattern of taking a p2p element stream and transforming it into a stream of block data.The abstraction is not entirely perfect; but I think this will help simplify things.
As a demonstration I've implemented this (somewhat hackily) for the transaction checkpoint sync (without connecting it to anything yet). For tracking sync, we would implement something similar that simply receives these counts from the header fanout.
Transaction source from p2p then becomes a normal transaction stream, which is then connected to one the above stages.