Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

share/p2p/shrexsub: Dropping messages to peers in flood #1997

Closed
Tracked by #2106
renaynay opened this issue Mar 31, 2023 · 2 comments
Closed
Tracked by #2106

share/p2p/shrexsub: Dropping messages to peers in flood #1997

renaynay opened this issue Mar 31, 2023 · 2 comments
Assignees
@walldiss
Labels
area:p2p area:shares Shares and samples

Comments

@renaynay
Copy link
Member

renaynay commented Mar 31, 2023 •
edited
Loading

Observed on full / bridges on blockspacerace:

Floodsub is dropping shrex-sub notifications (seemingly on every new block) to the same set of peers. Tracking this issue here:

2023-03-31T11:56:28.323Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWGojzFFQJ5eyewAo99DoDmUB2Z2ndjJimMLkX9tohqzck: queue full
2023-03-31T11:56:28.323Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWHxuKfuXq6hseL8nAG7xqe2zcajs7aXQykgThtxHnNM3Y: queue full
2023-03-31T11:56:28.324Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWFKcU9i4XRFfmB8xystGRRy9B4JAczmVFoWvnwaqSkkkL: queue full
2023-03-31T11:56:28.324Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWCn2Zw4xLtJQPiTeipZn8DnZEgG9RfprnF2wJrF9kPBXf: queue full
2023-03-31T11:56:28.325Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWKeRVpJxAC1YgGYxNsoNoSKJNGevVqADtoZNgzJABVQSQ: queue full
2023-03-31T11:56:28.325Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWR13WuARLrDKQggeSNRWfQVQfsjzsECB8VgWDgFzSBAAz: queue full
2023-03-31T11:56:28.325Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWKoX4fwK6wMpevMEGeQdM4Z6jf4XSK5osyyP5NNbkWchZ: queue full
2023-03-31T11:56:28.326Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWKJCMqXBvYhpyYaxNvgVeHmxTzBe3Meu3491DAuCWRGQL: queue full
2023-03-31T11:56:28.326Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWE8KJEsbwdDGpg5KxW1LtyZSEZaEueYUNxysoZ8n949d9: queue full
2023-03-31T11:56:28.327Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWJQrUUJbtz9tYvoJSne6jC6AwQYuxDzw5EsXLvR5M6rGN: queue full
2023-03-31T11:56:28.327Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWCuVe28APYbvdZGgE7QJdMX2Ywg2nBWPRQeeoXwzCsbnZ: queue full
2023-03-31T11:56:28.327Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWPTNcGiknxFkWFSjuhFFtgxu85zpqwwF4uHGbqwkwdXLY: queue full
2023-03-31T11:56:28.328Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWHDABgJKo5rb2AM22u5wcQqXLXZZ9uB4yiXfjpNZAf1EH: queue full
2023-03-31T11:56:28.328Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWKV5QF9CkkbX4aYP7bFT9Rga9XsoaT9aAPivdgUPqzmKN: queue full
2023-03-31T11:56:28.329Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWDnBoyjDVpmkVjxHDaGZA7koom4GujuSMxCBuPzJc2vts: queue full
2023-03-31T11:56:28.329Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWH5V6WqzA9YqHznBVw29B4YHNzuiaQFo5UKnGxymFXjtN: queue full
2023-03-31T11:56:28.330Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWF4BEJxnSUgKpD1qY4QRRBaXHtV6NctGQrwmwnHKPqvKz: queue full
2023-03-31T11:56:28.330Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWDQ8KWGCWKvcGfeDtmNnDPvTUhG25qCAq86Qc1QmAg5Gc: queue full
2023-03-31T11:56:28.330Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWAffS4ew7SJm2GG99wwQhqQ5K9CJiKA3vcQo3UNgyUTas: queue full
2023-03-31T11:56:28.331Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWHgHNVxPcamvYMAeyLvNJb5RNtYE9xbuQMT8kTSAM6KQi: queue full
2023-03-31T11:56:28.331Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWBeACUzDGtVcy2VkXGh3yevWqXL3LucjLkdyqJXjVGrGo: queue full
2023-03-31T11:56:28.332Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWDrsJcAFejeEVxhmhbkb7npqtFcRdLJYq3gN6NmerKdub: queue full

2023-03-31T11:57:54.015Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWDnBoyjDVpmkVjxHDaGZA7koom4GujuSMxCBuPzJc2vts: queue full
2023-03-31T11:57:54.016Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWH5V6WqzA9YqHznBVw29B4YHNzuiaQFo5UKnGxymFXjtN: queue full
2023-03-31T11:57:54.016Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWF4BEJxnSUgKpD1qY4QRRBaXHtV6NctGQrwmwnHKPqvKz: queue full
2023-03-31T11:57:54.016Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWDQ8KWGCWKvcGfeDtmNnDPvTUhG25qCAq86Qc1QmAg5Gc: queue full
2023-03-31T11:57:54.017Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWAffS4ew7SJm2GG99wwQhqQ5K9CJiKA3vcQo3UNgyUTas: queue full
2023-03-31T11:57:54.017Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWHgHNVxPcamvYMAeyLvNJb5RNtYE9xbuQMT8kTSAM6KQi: queue full
2023-03-31T11:57:54.018Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWDrsJcAFejeEVxhmhbkb7npqtFcRdLJYq3gN6NmerKdub: queue full
2023-03-31T11:57:54.018Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWBeACUzDGtVcy2VkXGh3yevWqXL3LucjLkdyqJXjVGrGo: queue full
2023-03-31T11:57:54.019Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWGojzFFQJ5eyewAo99DoDmUB2Z2ndjJimMLkX9tohqzck: queue full
2023-03-31T11:57:54.019Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWHxuKfuXq6hseL8nAG7xqe2zcajs7aXQykgThtxHnNM3Y: queue full
2023-03-31T11:57:54.020Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWCn2Zw4xLtJQPiTeipZn8DnZEgG9RfprnF2wJrF9kPBXf: queue full
2023-03-31T11:57:54.020Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWFKcU9i4XRFfmB8xystGRRy9B4JAczmVFoWvnwaqSkkkL: queue full
2023-03-31T11:57:54.021Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWR13WuARLrDKQggeSNRWfQVQfsjzsECB8VgWDgFzSBAAz: queue full
2023-03-31T11:57:54.021Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWKeRVpJxAC1YgGYxNsoNoSKJNGevVqADtoZNgzJABVQSQ: queue full
2023-03-31T11:57:54.021Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWKoX4fwK6wMpevMEGeQdM4Z6jf4XSK5osyyP5NNbkWchZ: queue full
2023-03-31T11:57:54.022Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWKJCMqXBvYhpyYaxNvgVeHmxTzBe3Meu3491DAuCWRGQL: queue full
2023-03-31T11:57:54.022Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWE8KJEsbwdDGpg5KxW1LtyZSEZaEueYUNxysoZ8n949d9: queue full
2023-03-31T11:57:54.023Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWJQrUUJbtz9tYvoJSne6jC6AwQYuxDzw5EsXLvR5M6rGN: queue full
2023-03-31T11:57:54.023Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWCuVe28APYbvdZGgE7QJdMX2Ywg2nBWPRQeeoXwzCsbnZ: queue full
2023-03-31T11:57:54.024Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWPTNcGiknxFkWFSjuhFFtgxu85zpqwwF4uHGbqwkwdXLY: queue full
2023-03-31T11:57:54.024Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWHDABgJKo5rb2AM22u5wcQqXLXZZ9uB4yiXfjpNZAf1EH: queue full
2023-03-31T11:57:54.024Z        INFO    pubsub  go-libp2p-pubsub@v0.9.3/floodsub.go:95  dropping message to peer 12D3KooWKV5QF9CkkbX4aYP7bFT9Rga9XsoaT9aAPivdgUPqzmKN: queue full

Side note:
It would be nice if the msgID could also be logged in the floodsub log but that would require us to probably change the msgID func for shrexsub for it to provide value.
@renaynay renaynay added area:shares Shares and samples area:p2p and removed needs:triage labels Mar 31, 2023
@renaynay
Copy link
Member Author

renaynay commented Apr 4, 2023

#1992

@vgonkivs vgonkivs mentioned this issue Apr 4, 2023
Merged
5 tasks
@renaynay renaynay assigned walldiss and unassigned vgonkivs Apr 6, 2023
@walldiss walldiss mentioned this issue Apr 20, 2023
Open
@renaynay
Copy link
Member Author

renaynay commented Nov 30, 2023 •
edited
Loading

Closing as no longer observed (also this was a symptom of other documented issues we had before)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@walldiss walldiss

Labels
area:p2p area:shares Shares and samples
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@walldiss @vgonkivs @renaynay