Add individual by_range sync requests #6497

dapplion · 2024-10-16T22:37:57Z

Issue Addressed

Part of

Improve range sync with PeerDAS #6258

To address PeerDAS sync issues we need to make individual by_range requests within a batch retriable. We should adopt the same pattern for lookup sync where each request (block/blobs/columns) is tracked individually within a "meta" request that group them all and handles retry logic.

Proposed Changes

Building on Generalize sync ActiveRequests #6398

second step is to add individual request accumulators for blocks_by_range, blobs_by_range, and data_columns_by_range. This will allow each request to progress independently and be retried separately.

Most of the logic is just piping, excuse the large diff. This PR does not change the logic of how requests are handled or retried. This will be done in a future PR changing the logic of RangeBlockComponentsRequest.

Before

Sync manager receives block with SyncRequestId::RangeBlockAndBlobs
Insert block into SyncNetworkContext::range_block_components_requests
(If received stream terminators of all requests)
- Return Vec<RpcBlock>, and insert into range_sync

Now

Sync manager receives block with SyncRequestId::RangeBlockAndBlobs
Insert block into SyncNetworkContext:: blocks_by_range_requests
(If received stream terminator of this request)
- Return Vec<SignedBlock>, and insert into SyncNetworkContext::components_by_range_requests
- (If received a result for all requests)
  - Return Vec<RpcBlock>, and insert into range_sync

dapplion · 2024-10-16T22:39:22Z

beacon_node/network/src/sync/range_sync/range.rs

+        // practice
+        //
+        // rig.expect_chain_segment();
+        // rig.expect_chain_segment();


I want to migrate the range sync tests to the event-driven format but would prefer to do so in a separate PR

realbigsean

Looks really good overall!

realbigsean · 2024-10-17T15:00:57Z

beacon_node/lighthouse_network/src/service/api_types.rs

@@ -43,12 +47,50 @@ pub struct DataColumnsByRootRequestId {
    pub requester: DataColumnsByRootRequester,
 }

+#[derive(Debug, Hash, PartialEq, Eq, Clone, Copy)]
+pub struct BlocksByRangeRequestId {
+    pub id: Id,


i'm understanding this is the sub-request id and requester.id is the meta-request id, can we rename one of them? it's sort of confusing

What name(s) do you have in mind? I can add docs otherwise

beacon_node/network/src/sync/network_context/requests/blobs_by_range.rs

beacon_node/network/src/sync/network_context.rs

realbigsean · 2024-10-17T17:42:28Z

beacon_node/network/src/sync/network_context.rs

-                "epoch" => epoch,
-                "peer" => %peer_id,
-            );
+        let _blocks_req_id = self.send_blocks_by_range_request(peer_id, request.clone(), id)?;


The _blocks_req_id here is unused and blobs_req_id is only used to check if a blobs request was made. I don't see these send_ methods used anywhere else so I think we can return Result<(), ..> rather than Result<BlocksByRangeRequestId, ..> for blocks. Maybe a bool for blobs. Looks like for custody columns, we care about the count requested but not particular ids

In a future PR RangeBlockComponentsRequest will track individual requests by ID. Left this variable named but unused as a hint that it should be used.

beacon_node/network/src/sync/network_context.rs

jimmygchen · 2025-01-10T06:30:54Z

beacon_node/network/src/sync/network_context/requests/blocks_by_range.rs

+
+        self.items.push(item);
+
+        Ok(self.items.len() >= *self.request.count() as usize)


Should we check if a block for the slot already exists in self.items, to handle the case where the same block is sent twice? or is this handled elsewhere?

Added the check for duplicate data

Added the duplicate data check to all by_range requests

jimmygchen · 2025-01-10T06:41:44Z

beacon_node/network/src/sync/network_context/requests/data_columns_by_range.rs

+        Ok(self.items.len() >= self.request.count as usize * self.request.columns.len())
+    }
+
+    fn consume(&mut self) -> Vec<Self::Item> {


Looks like this impl can be moved to ActiveRequestItems?

It can't as it uses state in DataColumnsByRangeRequestItems

dapplion · 2025-01-13T11:03:47Z

beacon_node/network/src/sync/block_sidecar_coupling.rs

@@ -29,18 +28,13 @@ pub struct RangeBlockComponentsRequest<E: EthSpec> {
    /// Used to determine if the number of data columns stream termination this accumulator should
    /// wait for. This may be less than the number of `expects_custody_columns` due to request batching.
    num_custody_column_requests: Option<usize>,
-    /// The peers the request was made to.
-    pub(crate) peer_ids: Vec<PeerId>,
-    max_blobs_per_block: usize,


Removed this from the state, since into_responses_with_blobs has access to the spec

dapplion · 2025-01-13T11:19:38Z

Rebased on latest unstable, @jimmygchen ready for another review

dapplion added ready-for-review The code is ready for review syncing labels Oct 16, 2024

dapplion commented Oct 16, 2024

View reviewed changes

dapplion mentioned this pull request Oct 16, 2024

Generalize sync block lookup tests #6498

Merged

realbigsean requested changes Oct 17, 2024

View reviewed changes

Base automatically changed from sync-active-request-generalize to unstable October 17, 2024 18:14

dapplion force-pushed the sync-active-request-byrange branch from df1a6c7 to 5fa2912 Compare October 18, 2024 14:28

jimmygchen self-assigned this Oct 24, 2024

dapplion mentioned this pull request Oct 24, 2024

Match block and blobs after validating execution #6469

Open

dapplion requested a review from realbigsean November 18, 2024 08:41

dapplion mentioned this pull request Nov 27, 2024

Write range sync tests in external event-driven form #6618

Merged

jimmygchen reviewed Jan 10, 2025

View reviewed changes

dapplion force-pushed the sync-active-request-byrange branch from 5fa2912 to a2d1d69 Compare January 13, 2025 11:02

dapplion requested a review from jxs as a code owner January 13, 2025 11:02

dapplion commented Jan 13, 2025

View reviewed changes

dapplion force-pushed the sync-active-request-byrange branch from a2d1d69 to afe72d8 Compare January 13, 2025 11:15

Add individual by_range sync requests

7f71079

dapplion force-pushed the sync-active-request-byrange branch from afe72d8 to 7f71079 Compare January 13, 2025 11:18

dapplion requested a review from jimmygchen January 13, 2025 11:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add individual by_range sync requests #6497

Add individual by_range sync requests #6497

dapplion commented Oct 16, 2024

dapplion Oct 16, 2024

realbigsean left a comment

realbigsean Oct 17, 2024

dapplion Oct 18, 2024

realbigsean Oct 17, 2024

dapplion Oct 18, 2024

jimmygchen Jan 10, 2025

dapplion Jan 13, 2025

dapplion Jan 13, 2025

jimmygchen Jan 10, 2025

dapplion Jan 13, 2025 •

edited

Loading

dapplion Jan 13, 2025

dapplion commented Jan 13, 2025


		self.items.push(item);

		Ok(self.items.len() >= *self.request.count() as usize)

Add individual by_range sync requests #6497

Are you sure you want to change the base?

Add individual by_range sync requests #6497

Conversation

dapplion commented Oct 16, 2024

Issue Addressed

Proposed Changes

Before

Now

Choose a reason for hiding this comment

realbigsean left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dapplion Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dapplion commented Jan 13, 2025

dapplion Jan 13, 2025 •

edited

Loading