Feat(near-client): Chunk distribution via message bus #10480
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #10480 +/- ##
==========================================
+ Coverage 72.20% 72.24% +0.03%
==========================================
Files 729 730 +1
Lines 148076 148521 +445
Branches 148076 148521 +445
==========================================
+ Hits 106923 107302 +379
- Misses 36342 36404 +62
- Partials 4811 4815 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
028b7db to
179f04d
Compare
|
HI @birchmd, thanks for the proposal! I wasn't aware of this issue and it's great to see involvement from the community. I'll bring it up to the team for discussion and come back to you next week. First thoughts:
|
I completely understand the concern here. I think of this as an optimization of the happy path only, not any kind of protocol change. The decentralized peer-to-peer network is still the foundation on which everything works. This just gives an opportunity for faster message distribution when things are going well (which should be 99% of the time). The worst case scenario if something does go wrong with this is the latency of chunk distribution returns to what it was before this solution.
I think it will be hard to compete with this using the peer-to-peer network. This is a solution optimized for this kind of message passing (multi-producer, multi-consumer with interest-based routing). I also wouldn't want to make the security of the core networking protocol worse by trying to improve its performance. I think it makes sense to have a separate network for performance.
+1 please review carefully.
❤️ |
179f04d to
37ae9c8
Compare
|
We discussed it in core team and here are a few thoughts, comments and questions: Alternatives - can you compare each to your proposal?
Comments for this solution:
I think it boils down how much of a difference is there between the perfomance of the alternative solution. If we can implement a solution that is 10% slower but decentralized and safe to enable by default I would consider it the better alternative. If it's a 2x difference that's a different story. Also JFYI we're making some changes to chunk processing and validation logic. In stateless validation chunks will have accompanying state witnesses and endorsements. Not sure if there are any changes necessary here but sharing just in case. |
Given that the message bus part is abstracted out, I don't think this PR necessarily introduces any point of centralization on its own.
This is a good question. @birchmd what do you think? |
I don't think any kind of solution using the peer-to-peer network will be both performant and scalable. Essentially the trade-off is between latency (via pull messaging) vs bandwidth (actively pushing messages). Currently chunks are distributed via pull (a node realizes they need some chunk and asks for it) and even with a direct connection to a peer that is guaranteed to have the chunk you will always have the round-trip latency of sending the pull message before getting the chunk in response. If we switch to a broadcast model (i.e. push) then we eliminate the round-trip latency, but traffic becomes an issue for the peer-to-peer network as the number of shards increases. This is why the base protocol is designed with pull messaging in the first place. You could have a push model that is more precise than broadcast by having some kind of subscription protocol, but I suspect validators will not want to open themselves up to having to spend even more bandwidth on messaging and therefore would reject subscription requests. Even if we could switch chunk distribution to a push model, there is still the issue that subscribing to one validator is not enough. With one validator you would only automatically get the chunks that single validator produces (at least the changes to nearcore in this PR only forward chunks produced by the node as opposed to all chunks it receives). To consistently get all chunks from a shard you would need to subscribe to most of the validators assigned to that shard. That set of validators could change between epochs, requiring yet more complexity in the hypothetical subscription protocol we are discussing.
We have done some experiments and the solution should reduce latency by around 2 seconds. This number comes from measuring the time between an indexer receiving a block header for the first time and having received all chunks for that block. Additionally, on testnet we ran our own validator node with the patch to publish chunks it creates to a message bus and compared the time it took for an indexer to receive the chunk via the peer-to-peer network against the time for another indexer to receive that chunk via the message bus. This experiment suffered a little from lack of statistics because we could only measure time differences for chunks produced by our own validator. The best experiment we could do would be to set up a sandbox network with programmable latencies and compare the peer-to-peer with the message bus in this setting. But this would be a larger undertaking and a significant delay for the project.
All code for this project will be public and include documentation.
I think this PR is already minimally invasive to nearcore and is essentially a plug-in mechanism as-is. This PR only exposes sending and receiving chunk data at configurable HTTP endpoints. Anyone could design any other service that uses or produces chunk data and connect it to nearcore using this mechanism. For example, suppose a large validator wanted to use this to sync data between their production and fail-over nodes. This would be very easy; a simple HTTP server with two endpoints (post for accepting chunks from the production node and get for downloading those chunks to the fail-over node) would slot in perfectly with the changes in this PR. This was the whole idea of only defining abstract HTTP interfaces in nearcore. It works for this project, while cleanly separating concerns and also opening up possibilities for future innovation. |
2169aad to
2e8b952
Compare
…vides an invalid chunk
… is present but not enabled
2e8b952 to
4bf0852
Compare
There was a problem hiding this comment.
LGTM with a bunch of optional improvement suggestions.
I confirm that this PR:
- doesn't introduce any protocol changes
- doesn't introduce any additional dependencies, interactions with centralised messaged bus are preformed via http
- doesn't change existing behaviour by default, the new feature needs to be explicitly enabled via config
|
@birchmd what prevents this PR from getting merged? |
|
@bowenwang1996 nothing is blocking as far as I know. I'd love to see this merged! |
|
@wacban , can you please review and approve/comment. this is waiting for your approval to be merged. thanks |
|
@vikpande Wac is OOO this week, but as far as I understand he is OK with this PR, so feel free to proceed with merging this. |
|
@birchmd , pls take it from here. good luck ! |
This PR implements the changes to nearcore proposed in near#10083 To summarize briefly here, the goal of this project is to reduce the latency experienced by RPC nodes through directly distributing chunks over a message bus. Validators eagerly push the chunks they produce on to this bus in addition to the peer-to-peer messages they send. When any node (validator or RPC) realizes it needs a chunk (e.g. because it is present in a new block) then it can check the message bus to see if it present there before trying a request over the peer-to-peer network. Participation in chunk distribution via the message bus is entirely optional and disabled by default. This PR has no impact on nodes that are not participating in the new chunk distribution. To be clear: this PR is not a replacement for the existing chunk distribution logic via the peer-to-peer network; it is a secondary channel which should provide faster chunk distribution (on the happy path). If invalid chunks are published to the message bus then nodes which receive them will request via the peer-to-peer network instead. The details of how the chunks are posted to the message bus and what service exists for querying from the message bus are left intentionally abstract. These will be handled by separate a application (or applications). The nearcore code only sends HTTP requests to endpoints specified in config that is used by node operators to opt-in to this feature.
This PR implements the changes to nearcore proposed in near#10083 To summarize briefly here, the goal of this project is to reduce the latency experienced by RPC nodes through directly distributing chunks over a message bus. Validators eagerly push the chunks they produce on to this bus in addition to the peer-to-peer messages they send. When any node (validator or RPC) realizes it needs a chunk (e.g. because it is present in a new block) then it can check the message bus to see if it present there before trying a request over the peer-to-peer network. Participation in chunk distribution via the message bus is entirely optional and disabled by default. This PR has no impact on nodes that are not participating in the new chunk distribution. To be clear: this PR is not a replacement for the existing chunk distribution logic via the peer-to-peer network; it is a secondary channel which should provide faster chunk distribution (on the happy path). If invalid chunks are published to the message bus then nodes which receive them will request via the peer-to-peer network instead. The details of how the chunks are posted to the message bus and what service exists for querying from the message bus are left intentionally abstract. These will be handled by separate a application (or applications). The nearcore code only sends HTTP requests to endpoints specified in config that is used by node operators to opt-in to this feature.
This PR implements the changes to nearcore proposed in #10083
To summarize briefly here, the goal of this project is to reduce the latency experienced by RPC nodes through directly distributing chunks over a message bus. Validators eagerly push the chunks they produce on to this bus in addition to the peer-to-peer messages they send. When any node (validator or RPC) realizes it needs a chunk (e.g. because it is present in a new block) then it can check the message bus to see if it present there before trying a request over the peer-to-peer network.
Participation in chunk distribution via the message bus is entirely optional and disabled by default. This PR has no impact on nodes that are not participating in the new chunk distribution. To be clear: this PR is not a replacement for the existing chunk distribution logic via the peer-to-peer network; it is a secondary channel which should provide faster chunk distribution (on the happy path).
If invalid chunks are published to the message bus then nodes which receive them will request via the peer-to-peer network instead.
The details of how the chunks are posted to the message bus and what service exists for querying from the message bus are left intentionally abstract. These will be handled by separate a application (or applications). The nearcore code only sends HTTP requests to endpoints specified in config that is used by node operators to opt-in to this feature.