-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Chain Exchange protocol over pubsub #807
Conversation
58ce9f1
to
7f3d3ae
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #807 +/- ##
==========================================
- Coverage 69.38% 68.86% -0.53%
==========================================
Files 78 81 +3
Lines 7957 8241 +284
==========================================
+ Hits 5521 5675 +154
- Misses 1971 2083 +112
- Partials 465 483 +18
|
7f3d3ae
to
4f48648
Compare
Implement chain exchange protocol over pubsub as a mechanism to propagate `ECChain` across the network with reasonable spam protection. To protect against spam the mechanism employs two separate caches for chains that are generally discovered across the network and the ones explicitly looked up or broadcasted by the local node. Both caches are capped LRU, where the LRU recent-ness is used as a way to prioritise chains we cache while keeping the total memory footprint fixed. This approach is not the most memory efficient but is simpler to implement as the LRU encapsulates a lot of the complexity. The code has a lot of TODOs as places to improve or question to the reviewer. To action most of the TODOs further refactoring across the code is needed which is intended to be actioned in separate commits. The code path introduced here is not integrated into F3 host; future PRs will iteratively integrate the mechanism across F3 host and other places. Part of #792
4f48648
to
76ce337
Compare
"github.com/filecoin-project/go-f3/gpbft" | ||
) | ||
|
||
type Key []byte |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe make it a string so it can be used in caches and for comparisons?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The reason I haven't is because of avoiding invalid strings once this gets encoded as CBOR (in upcoming partial message that includes it).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, some of these TODO-s should get done, but they don't have to get done right now.
Implement chain exchange protocol over pubsub as a mechanism to propagate
ECChain
across the network with reasonable spam protection.To protect against spam the mechanism employs two separate caches for chains that are generally discovered across the network and the ones explicitly looked up or broadcasted by the local node. Both caches are capped LRU, where the LRU recent-ness is used as a way to prioritise chains we cache while keeping the total memory footprint fixed. This approach is not the most memory efficient but is simpler to implement as the LRU encapsulates a lot of the complexity.
The code has a lot of TODOs as places to improve or question to the reviewer. To action most of the TODOs further refactoring across the code is needed which is intended to be actioned in separate commits.
The code path introduced here is not integrated into F3 host; future PRs will iteratively integrate the mechanism across F3 host and other places.
Part of #792