Kusama nodes stop syncing at finalized block 8949248

[AlexZhenWang]

Image version: 0.9.9-1

Hi, we have several Kusama nodes and parachain nodes stuck at finalized block 8949248. I checked the telemetry, seems like there are many nodes stuck at 8949248. This happened on both archive nodes and non-archive nodes

We have tried to use reserved-nodes and reserved-only to connect to the healthy nodes but the connection was rejected.

We also tried creating a new node using version 0.9.9-1 to sync from the first block, but that node stuck at 8949248 as well.

Also, from the telemetry, seems like many stuck nodes are collators of the parachains.

[ioannist]

We are also having the same issue trying to sync Moonriver collator nodes

moonbeam-foundation/moonbeam#758

[artkaseman]

I was able to replicate as well syncing from scratch with the latest moonbeam binary. Trace logs at the block in question. https://drive.google.com/file/d/1KB-gm8ip25F9odzB942azNpKZitiNQZ5/view

[ioannist]

Syncing from scratch with Polkadot v0.9.9-1 gets stuck. We have replicated this 2x in Moonriver, and 1x in Kusama, in two different machines

`2021-09-02 06:29:30 💤 Idle (49 peers), best: #8949641 (0x135a…dd8f), finalized #8949248 (0x102e…0032), ⬇ 3.0kiB/s ⬆ 9.0kiB/s

2021-09-02 06:29:35 💤 Idle (49 peers), best: #8949641 (0x135a…dd8f), finalized #8949248 (0x102e…0032), ⬇ 2.9kiB/s ⬆ 7.6kiB/s

2021-09-02 06:29:40 💤 Idle (49 peers), best: #8949641 (0x135a…dd8f), finalized #8949248 (0x102e…0032), ⬇ 47.5kiB/s ⬆ 6.3kiB/s

2021-09-02 06:29:45 💤 Idle (49 peers), best: #8949641 (0x135a…dd8f), finalized #8949248 (0x102e…0032), ⬇ 47.3kiB/s ⬆ 6.6kiB/s

2021-09-02 06:29:50 💤 Idle (49 peers), best: #8949641 (0x135a…dd8f), finalized #8949248 (0x102e…0032), ⬇ 4.5kiB/s ⬆ 10.1kiB/s`

[ioannist]

Starting with Polkadot v0.9.5, and then upgrading as necessary up to the latest Polkadot v0.9.9-1, I was able to get past block #8949248

Unfortunately, syncing got stuck again, not long after:
best: #8949716 (0x0b77…4193), finalized #8949696 (0xaba6…b7d8), ⬇ 104.3kiB/s ⬆ 9.0MiB/s

I have now replicated the issue 5x in 2 different machines, all fresh installs.

Discord chats reveal that many people are facing the issues above. However, not many come over to Github to raise the issue here. They simply resort to a backup so life can go on. This is dangerous in the long term.

[AlexZhenWang]

Starting with Polkadot v0.9.5, and then upgrading as necessary up to the latest Polkadot v0.9.9-1, I was able to get past block #8949248

Unfortunately, syncing got stuck again, not long after:
best: #8949716 (0x0b77…4193), finalized #8949696 (0xaba6…b7d8), ⬇ 104.3kiB/s ⬆ 9.0MiB/s

I have now replicated the issue 5x in 2 different machines, all fresh installs.

Discord chats reveal that many people are facing the issues above. However, not many come over to Github to raise the issue here. They simply resort to a backup so life can go on. This is dangerous in the long term.

Yeah, I tried started three new nodes start syncing from the block 0, they are on version 0.9.7, 0.9.8 and 0.9.9-1, all stuck at this same block 8949248

[bkchr]

#3778 it was solved.

The fix for this will be part of 0.9.10

[AlexZhenWang]

fixed in paritytech/substrate#9692