new routing exchange prototype using IBF #4112

pmnoxx · 2021-03-15T18:50:30Z

Implement new algorithm for exchanging routing tables using Inverse Bloom Filters.
#3838

Reposting #4108

chain/network/Cargo.toml

chain/network/src/ibf.rs

chain/network/src/ibf_set.rs

chain/network/src/peer_manager.rs

chain/network/src/routing.rs

chain/network/src/types.rs

core/primitives/src/version.rs

chain/network/src/types.rs

chain/network/src/routing.rs

mfornet

Need to add integration tests that stress different scenarios of synchronization using IBF.

chain/network/src/peer_manager.rs

chain/network/src/ibf.rs

chain/network/src/peer_manager.rs

chain/network/src/ibf_set.rs

chain/network/src/ibf_peer_set.rs

chain/network/src/lib.rs

chain/network/src/peer_manager.rs

stale · 2021-07-01T07:27:52Z

This PR has been automatically marked as stale because it has not had recent activity in the 2 weeks.
It will be closed in 3 days if no further activity occurs.
Thank you for your contributions.

bowenwang1996 · 2021-07-01T18:31:43Z

@pmnoxx what is the status here?

pmnoxx · 2021-07-06T16:51:17Z

@pmnoxx what is the status here?

I submitted updated PR. You can take a look.

chain/network/src/ibf.rs

chain/network/src/peer.rs

chain/network/src/routing.rs

nearcore/res/storage_usage_delta.json

chain/jsonrpc/src/lib.rs

chain/network/src/peer_manager.rs

khorolets

jsonrpc endpoints not follows the way we are handing endpoints. Please, make them as the other endpoints.

chain/jsonrpc/src/lib.rs

stale · 2021-07-22T10:52:56Z

This PR has been automatically marked as stale because it has not had recent activity in the 2 weeks.
It will be closed in 3 days if no further activity occurs.
Thank you for your contributions.

bowenwang1996 · 2021-07-22T22:03:35Z

@pmnoxx what is the status of this PR?

stale · 2021-08-05T22:57:00Z

This PR has been automatically marked as stale because it has not had recent activity in the 2 weeks.
It will be closed in 3 days if no further activity occurs.
Thank you for your contributions.

stale · 2021-08-23T17:19:14Z

This PR has been automatically marked as stale because it has not had recent activity in the 2 weeks.
It will be closed in 3 days if no further activity occurs.
Thank you for your contributions.

chefsale · 2021-08-23T20:53:57Z

What is the status on this PR? It has again been marked staled? Do we still plan to land this? Can we close it? @pmnoxx @bowenwang1996

olonho

OK for stable hash change.

ailisp

There's not a lot change in runtime, and these changes look okay. As for the IBF in network and chain, I studied and felt confident it's a correct implementation too. I particularly like the unit tests, the benchmark and the pytest, together they prove the correctness and scale in orthogonal ways.

Ok, I addressed the issues you mentioned

pmnoxx · 2021-09-24T00:45:13Z

Nayduck passes previously. I ran it one more time after doing all changes:
http://nayduck.near.org/#/run/2082

All tests pass

pmnoxx · 2021-09-24T03:09:03Z

Nayduck passes previously. I ran it one more time after doing all changes:
http://nayduck.near.org/#/run/2082

All tests pass

@frol Is there anything that prevents us from merging?

chain/jsonrpc-primitives/src/types/mod.rs

chain/jsonrpc/src/test_utils.rs

chain/network-primitives/src/types.rs

chain/jsonrpc/Cargo.toml

frol

I am unblocking this PR from merging, so I am not blocking merging it once all my comments are addressed.

… near-network

…4884) Fix compilation error in `near-jsonrpc-primitives` when building with the `adversarial` feature enabled after #4112 ### Test Plan - [X] Ensure `near-jsonrpc-primitives` compiles with `adversarial` features turned on

Implement new algorithm for exchanging routing tables using Inverse Bloom Filters. #3838 Reposting #4108

…te (#4912) #4651 successfully removed `near-jsonrpc-primitives`-s heavy dependency on `near-network` which in turn depended on `rocksdb`. #4112 reintroduced that dependency on `near-network`, though optionally, this hinders the publishing process. This PR extracts the adversarial features that depend on `near-network` into a private, specialized crate: `near-jsonrpc-adversarial-primitives` and sets us right on track for publishing `jsonrpc-primitives`.

pmnoxx requested review from bowenwang1996 and mfornet as code owners March 15, 2021 18:50

pmnoxx mentioned this pull request Mar 15, 2021

Reconsider how we exchange routing table information #3838

Open

2 tasks

pmnoxx requested review from evgenykuzyakov, frol, maxzaver and SkidanovAlex as code owners March 15, 2021 19:06

bowenwang1996 requested a review from abacabadabacaba March 15, 2021 23:43

bowenwang1996 reviewed Mar 16, 2021

View reviewed changes

matklad reviewed Mar 16, 2021

View reviewed changes

chain/network/src/types.rs Outdated Show resolved Hide resolved

chain/network/src/types.rs Outdated Show resolved Hide resolved

chain/network/src/routing.rs Show resolved Hide resolved

chain/network/src/routing.rs Show resolved Hide resolved

mfornet requested changes Mar 17, 2021

View reviewed changes

stale bot added the S-stale label Jul 1, 2021

bowenwang1996 removed the S-stale label Jul 1, 2021

pmnoxx requested review from chefsale and khorolets as code owners July 6, 2021 11:36

bowenwang1996 reviewed Jul 6, 2021

View reviewed changes

khorolets requested changes Jul 8, 2021

View reviewed changes

chain/jsonrpc/src/lib.rs Outdated Show resolved Hide resolved

stale bot added the S-stale label Jul 22, 2021

stale bot removed the S-stale label Jul 22, 2021

stale bot added S-stale and removed S-stale labels Aug 5, 2021

stale bot added the S-stale label Aug 23, 2021

pmnoxx requested a review from frol September 23, 2021 05:44

olonho approved these changes Sep 23, 2021

View reviewed changes

khorolets approved these changes Sep 23, 2021

View reviewed changes

ailisp approved these changes Sep 23, 2021

View reviewed changes

pmnoxx self-assigned this Sep 23, 2021

bowenwang1996 approved these changes Sep 23, 2021

View reviewed changes

pmnoxx added the S-automerge label Sep 24, 2021

pmnoxx removed the S-automerge label Sep 24, 2021

frol requested changes Sep 24, 2021

View reviewed changes

chain/jsonrpc-primitives/src/types/mod.rs Outdated Show resolved Hide resolved

chain/jsonrpc/src/test_utils.rs Outdated Show resolved Hide resolved

chain/network-primitives/src/types.rs Outdated Show resolved Hide resolved

chain/jsonrpc/Cargo.toml Outdated Show resolved Hide resolved

frol approved these changes Sep 24, 2021

View reviewed changes

pmnoxx added 2 commits September 24, 2021 22:50

rebase

4032805

Make serde optional / move types out of near-network-primites back to…

b89546e

… near-network

pmnoxx added S-automerge and removed S-automerge labels Sep 25, 2021

pmnoxx added 2 commits September 24, 2021 23:36

Remove byteorder dependecy

76ee02e

Reorder imports

492ba9d

pmnoxx added the S-automerge label Sep 25, 2021

pmnoxx and others added 2 commits September 25, 2021 08:40

Fix compile error

e9bb950

Merge branch 'master' into piotr-new-routing-algorithm-v2

5d23b9f

near-bulldozer bot merged commit 51ab16a into near:master Sep 25, 2021

miraclx mentioned this pull request Sep 27, 2021

fix(jsonrpc-primitives): fix adversarial-related compilation errors #4884

Merged

1 task

bowenwang1996 mentioned this pull request Sep 27, 2021

Implement Efficient Set Reconciliation algorithm #3934

Closed

28 tasks

nikurt pushed a commit that referenced this pull request Sep 30, 2021

new routing exchange prototype using IBF (#4112)

1bb07ac

Implement new algorithm for exchanging routing tables using Inverse Bloom Filters. #3838 Reposting #4108

miraclx mentioned this pull request Sep 30, 2021

refactor(jsonrpc-primitives): move adversarial to own specialized crate #4912

Merged

pmnoxx deleted the piotr-new-routing-algorithm-v2 branch November 13, 2021 06:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new routing exchange prototype using IBF #4112

new routing exchange prototype using IBF #4112

pmnoxx commented Mar 15, 2021

mfornet left a comment

stale bot commented Jul 1, 2021

bowenwang1996 commented Jul 1, 2021

pmnoxx commented Jul 6, 2021

khorolets left a comment

stale bot commented Jul 22, 2021

bowenwang1996 commented Jul 22, 2021

stale bot commented Aug 5, 2021

stale bot commented Aug 23, 2021

chefsale commented Aug 23, 2021

olonho left a comment

ailisp left a comment

pmnoxx commented Sep 24, 2021 •

edited

Loading

pmnoxx commented Sep 24, 2021

frol left a comment

new routing exchange prototype using IBF #4112

new routing exchange prototype using IBF #4112

Conversation

pmnoxx commented Mar 15, 2021

mfornet left a comment

Choose a reason for hiding this comment

stale bot commented Jul 1, 2021

bowenwang1996 commented Jul 1, 2021

pmnoxx commented Jul 6, 2021

khorolets left a comment

Choose a reason for hiding this comment

stale bot commented Jul 22, 2021

bowenwang1996 commented Jul 22, 2021

stale bot commented Aug 5, 2021

stale bot commented Aug 23, 2021

chefsale commented Aug 23, 2021

olonho left a comment

Choose a reason for hiding this comment

ailisp left a comment

Choose a reason for hiding this comment

pmnoxx commented Sep 24, 2021 • edited Loading

pmnoxx commented Sep 24, 2021

frol left a comment

Choose a reason for hiding this comment

pmnoxx commented Sep 24, 2021 •

edited

Loading