Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Chain Exchange protocol over pubsub #807

Merged
merged 1 commit into from
Dec 20, 2024

Conversation

masih
Copy link
Member

@masih masih commented Dec 19, 2024

Implement chain exchange protocol over pubsub as a mechanism to propagate ECChain across the network with reasonable spam protection.

To protect against spam the mechanism employs two separate caches for chains that are generally discovered across the network and the ones explicitly looked up or broadcasted by the local node. Both caches are capped LRU, where the LRU recent-ness is used as a way to prioritise chains we cache while keeping the total memory footprint fixed. This approach is not the most memory efficient but is simpler to implement as the LRU encapsulates a lot of the complexity.

The code has a lot of TODOs as places to improve or question to the reviewer. To action most of the TODOs further refactoring across the code is needed which is intended to be actioned in separate commits.

The code path introduced here is not integrated into F3 host; future PRs will iteratively integrate the mechanism across F3 host and other places.

Part of #792

@masih masih force-pushed the masih/pubsub-chainexchange branch from 58ce9f1 to 7f3d3ae Compare December 19, 2024 17:16
@masih masih marked this pull request as ready for review December 19, 2024 17:22
@masih masih requested a review from Kubuxu December 19, 2024 17:22
Copy link

codecov bot commented Dec 19, 2024

Codecov Report

Attention: Patch coverage is 50.00000% with 142 lines in your changes missing coverage. Please review.

Project coverage is 68.86%. Comparing base (d4f3a0c) to head (76ce337).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
chainexchange/pubsub.go 56.21% 76 Missing and 12 partials ⚠️
chainexchange/options.go 34.14% 47 Missing and 7 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #807      +/-   ##
==========================================
- Coverage   69.38%   68.86%   -0.53%     
==========================================
  Files          78       81       +3     
  Lines        7957     8241     +284     
==========================================
+ Hits         5521     5675     +154     
- Misses       1971     2083     +112     
- Partials      465      483      +18     
Files with missing lines Coverage Δ
chainexchange/chainexchange.go 100.00% <100.00%> (ø)
internal/psutil/psutil.go 32.25% <ø> (ø)
chainexchange/options.go 34.14% <34.14%> (ø)
chainexchange/pubsub.go 56.21% <56.21%> (ø)

... and 3 files with indirect coverage changes

chainexchange/pubsub.go Outdated Show resolved Hide resolved
Implement chain exchange protocol over pubsub as a mechanism to
propagate `ECChain` across the network with reasonable spam protection.

To protect against spam the mechanism employs two separate caches for
chains that are generally discovered across the network and the ones
explicitly looked up or broadcasted by the local node. Both caches are
capped LRU, where the LRU recent-ness is used as a way to prioritise
chains we cache while keeping the total memory footprint fixed. This
approach is not the most memory efficient but is simpler to implement
as the LRU encapsulates a lot of the complexity.

The code has a lot of TODOs as places to improve or question to the
reviewer. To action most of the TODOs further refactoring across the
code is needed which is intended to be actioned in separate commits.

The code path introduced here is not integrated into F3 host; future PRs
will iteratively integrate the mechanism across F3 host and other
places.

Part of #792
@masih masih force-pushed the masih/pubsub-chainexchange branch from 4f48648 to 76ce337 Compare December 20, 2024 14:39
"github.com/filecoin-project/go-f3/gpbft"
)

type Key []byte
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe make it a string so it can be used in caches and for comparisons?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I haven't is because of avoiding invalid strings once this gets encoded as CBOR (in upcoming partial message that includes it).

Copy link
Contributor

@Kubuxu Kubuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, some of these TODO-s should get done, but they don't have to get done right now.

@masih masih added this pull request to the merge queue Dec 20, 2024
Merged via the queue into main with commit 67e2c06 Dec 20, 2024
13 of 14 checks passed
@masih masih deleted the masih/pubsub-chainexchange branch December 20, 2024 15:41
@masih masih self-assigned this Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants