ShrEx observed issues

[renaynay]

Documenting issues we've observed so far as a note:

• DASer doesn’t max out the concurrency lim for full nodes — even when my catchup head is ~4k headers past my head of sampled chain, i only spawn two workers with about 50 headers each
• Full node lags significantly behind network head (even when it was already synced up to network head) for sampling when chain is maxxed out (every block @8mb)
• Re-DASing of headers
• False alarm blacklisting of good peers

[walldiss]

DASer doesn’t max out the concurrency lim for full nodes — even when my catchup head is ~4k headers past my head of sampled chain, i only spawn two workers with about 50 headers each

It is not an issue. DASer does not make any decision based on head of sampled chain. If catchup head is same as network head, it means all headers were sent to workers. Two workers situation could happen, if those 2 are working on header ranges that is significantly slower to sample. Meaning all headers after those ranges were already processed by DASer, but not included in head of sampled chain, because it still has gaps.

Full node lags significantly behind network head

Do we see increase of "request failed" logs with DEBUG logs enabled for "share/getters"?

Re-DASing of headers

Would need logs showing the issue to identify the cause.

False alarm blacklisting of good peers

Merge peer-manager metrics PR and plug it into grafana would help to identify if the blacklist happens inside peer-manger and the reason for blacklisting.

[renaynay]

That's a fair point re 1.

Re 2.: we need to run metrics on a robusta full node for this

Re 3: Will report it next time it's observed

Re 4: Agree. @walldiss can you make an issue to track blacklisting stability?

[renaynay]

Closing this issue in favour of #2016