Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug(share/p2p/shrexnd): Request from light node to blockspacerace for GetSharesByNamespace hangs forever #2035

Closed
Tracked by #2106
renaynay opened this issue Apr 6, 2023 · 3 comments · Fixed by #2101
Closed
Tracked by #2106

bug(share/p2p/shrexnd): Request from light node to blockspacerace for GetSharesByNamespace hangs forever #2035

renaynay opened this issue Apr 6, 2023 · 3 comments · Fixed by #2101
Assignees
@walldiss
Labels
area:shares Shares and samples bug Something isn't working

Comments

@renaynay
Copy link
Member

renaynay commented Apr 6, 2023

Problem

Performing a GetSharesByNamespace request will hang forever.

2023-04-06T16:52:26.460+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWHzEtY3x7tCZDKvBpvWZjEkJtsEKCZegbEa9Rngtg3cXe", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:26.564+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWBct27GArLQoZZLwnShT7DnaXmNZpEiK8oQPvNuZrptp8", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:28.301+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWRmzCosNSVeeZnwXwm9JSmcq8CkhMYAG6J2PUscpuK7t2", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:29.232+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWBbc9ixMAH4SzF3EH6SwSfo8gRW1bVrDUX19oxC4DaRbz", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:29.590+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWBct27GArLQoZZLwnShT7DnaXmNZpEiK8oQPvNuZrptp8", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:52:31.416+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer":

also at a certain point, it will begin to request the same peer over and over

2023-04-06T16:55:41.256+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:44.289+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:46.364+0200    INFO    header/store    store/store.go:353      new head        {"height": 184565, "hash": "28C9563444EE06B692CC846DE131E9AB85BB400AEB309350278EBFF0469C1BEA"}
2023-04-06T16:55:46.367+0200    INFO    das     das/subscriber.go:35    new header received via subscription    {"height": 184565}
2023-04-06T16:55:46.480+0200    INFO    das     das/worker.go:80        finished sampling headers       {"from": 184565, "to": 184565, "errors": 0, "finished (s)": 0.113163417}
2023-04-06T16:55:47.319+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:50.352+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:53.387+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:56.417+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:55:57.060+0200    INFO    header/store    store/store.go:353      new head        {"height": 184566, "hash": "FF864BC83161C5A85C89BCD067C16EBD18AADF5DD359EB5778563F30D3795FED"}
2023-04-06T16:55:57.060+0200    INFO    das     das/subscriber.go:35    new header received via subscription    {"height": 184566}
2023-04-06T16:55:57.126+0200    INFO    das     das/worker.go:80        finished sampling headers       {"from": 184566, "to": 184566, "errors": 0, "finished (s)": 0.066025417}
2023-04-06T16:55:59.446+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:56:02.481+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}
2023-04-06T16:56:05.515+0200    DEBUG   shrex/nd        shrexnd/client.go:72    client-nd: peer returned err    {"peer": "12D3KooWAktyreXmqQN6iLr1ausLgSJFU7rSdbT9vMXemku8W9GW", "err": "client-nd: reading response: stream reset"}

Steps to reproduce

  1. Run light node with --log.level.module shrex/nd:debug against blockspacerace
  2. Submit a PFB to blockspacerace
  3. Wait til light node syncs up until height at which PFB was included, and do a shares.GetSharesByNamespace for the nID under which PFB was submitted with the DAH for that given height
  4. request will hang, check out node logs
@renaynay renaynay added bug Something isn't working area:shares Shares and samples labels Apr 6, 2023
@renaynay
Copy link
Member Author

renaynay commented Apr 10, 2023 •
edited
Loading

#2044 related

@renaynay
Copy link
Member Author

#2066 related

@renaynay
Copy link
Member Author

Steps to reproduce

  1. Run a light node against blockspacerace or a robusta network (something that reliably has non-empty blocks)
  2. Either submit a PFB or if you already know of a PFB at a certain height, you can query for that
  3. GetSharesByNamespace request (either via RPC or gateway)

This was referenced Apr 19, 2023
Merged
Open
walldiss added a commit that referenced this issue Apr 20, 2023
@walldiss
refactor(share/shrex): improve shrex logging (#2101)
c4e994b 
## Overview
- Previously shrex-nd and shrex-eds error logs were indistinguishable,
that lead to confusion of errors source.
 - add peerID to server logs
 - Improve some other shrex logging.
 

Resolves #2035
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@walldiss walldiss

Labels
area:shares Shares and samples bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

refactor(share/shrex): improve shrex logging walldiss/celestia-node
2 participants
@walldiss @renaynay