[Parachain] panicked at 'Storage root must match that calculated

[khssnv]

I have a collator node stuck at a certain parachain block. Please find the error log below.

2022-01-10 22:47:00 [Parachain] 💤 Idle (11 peers), best: #10355 (0x749c…d806), finalized #10354 (0x6b63…5a90), ⬇ 23.5kiB/s ⬆ 17.1kiB/s                                                              
2022-01-10 22:47:00 [Relaychain] ♻️  Reorg on #10908231,0x843f…82d5 to #10908231,0x92a9…5579, common ancestor #10908230,0xc0f7…c133      
2022-01-10 22:47:00 [Relaychain] ✨ Imported #10908231 (0x92a9…5579)          
2022-01-10 22:47:04 [Relaychain] 💤 Idle (28 peers), best: #10908231 (0x92a9…5579), finalized #10908228 (0x7bf0…40d7), ⬇ 495.7kiB/s ⬆ 772.6kiB/s          
2022-01-10 22:47:05 [Parachain] 💤 Idle (11 peers), best: #10355 (0x749c…d806), finalized #10354 (0x6b63…5a90), ⬇ 3.0kiB/s ⬆ 1.8kiB/s      
2022-01-10 22:47:06 [Relaychain] ✨ Imported #10908232 (0x428d…b0aa)
2022-01-10 22:47:06 [Parachain] Starting collation. relay_parent=0x428dffc62bd5eba6c4dcc20a17975e865a32503a4c56dd1ad80a69dd8a6eb0aa at=0xc58df8cb211b7e462c80ea5e41a99bdb5b5fdb3285089aeb6723689c50a70437
2022-01-10 22:47:06 [Parachain] 🙌 Starting consensus session on top of parent 0xc58df8cb211b7e462c80ea5e41a99bdb5b5fdb3285089aeb6723689c50a70437
2022-01-10 22:47:06 [Parachain] 🎁 Prepared block for proposing at 10357 (4 ms) [hash: 0x86a33438aac7f74037da297badb93c057914f3b5a4ba9655e7a5d0246b397d5d; parent_hash: 0xc58d…0437; extrinsics (3): [0x8b5e…627c, 0xd317…8e10, 0x4c2e…100f]]
2022-01-10 22:47:06 [Parachain] ✨ Imported #10357 (0x86a3…7d5d)                                                                                          
2022-01-10 22:47:06 [Parachain] PoV size { header: 0.0966796875kb, extrinsics: 3.3642578125kb, storage_proof: 409.6142578125kb }
2022-01-10 22:47:06 [Parachain] Produced proof-of-validity candidate. block_hash=0x86a33438aac7f74037da297badb93c057914f3b5a4ba9655e7a5d0246b397d5d
2022-01-10 22:47:06 [Parachain] panicked at 'Storage root must match that calculated.', /home/akru/.cargo/git/checkouts/substrate-7e08433d4c370a21/d76f399/frame/executive/src/lib.rs:503:9          
2022-01-10 22:47:08 [Parachain] Block prepare storage changes error:
RuntimeApiError(Application(Execution(Other("Wasm execution trapped: wasm trap: unreachable\nwasm backtrace:\n    0: 0x1f1d - <unknown>!rust_begin_unwind\n    1: 0x1bfa - <unknown>!core::panicking::panic_fmt::he934cb9e7b98bd53\n    2: 0x12d3 - <unknown>!core::panicking::panic::h25f3e5a9f228197d\n    3: 0xc11ec - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h8b590d4d50dc4876\n    4: 0xbfcc8 - <unknown>!Core_execute_block\n"))))    
2022-01-10 22:47:08 [Parachain] 💔 Error importing block 0x90b1f3fa89e2ea080e2c0a08f7befb470f9f4c8077f52f1e1942442d0657389a: Err(Other(ClientImport("Error at calling runtime api: Execution failed: Other(\"Wasm execution trapped: wasm trap: unreachable\\nwasm backtrace:\\n    0: 0x1f1d - <unknown>!rust_begin_unwind\\n    1: 0x1bfa - <unknown>!core::panicking::panic_fmt::he934cb9e7b98bd53\\n    2: 0x12d3 - <unknown>!core::panicking::panic::h25f3e5a9f228197d\\n    3: 0xc11ec - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h8b590d4d50dc4876\\n    4: 0xbfcc8 - <unknown>!Core_execute_block\\n\")")))    
2022-01-10 22:47:08 [Parachain] ✨ Imported #10357 (0x90d2…8a80)                                                                                                                                     
2022-01-10 22:47:08 [Parachain] panicked at 'Storage root must match that calculated.', /home/akru/.cargo/git/checkouts/substrate-7e08433d4c370a21/d76f399/frame/executive/src/lib.rs:503:9
2022-01-10 22:47:08 [Parachain] Block prepare storage changes error:                                                                                                                                 
RuntimeApiError(Application(Execution(Other("Wasm execution trapped: wasm trap: unreachable\nwasm backtrace:\n    0: 0x1f1d - <unknown>!rust_begin_unwind\n    1: 0x1bfa - <unknown>!core::panicking::panic_fmt::he934cb9e7b98bd53\n    2: 0x12d3 - <unknown>!core::panicking::panic::h25f3e5a9f228197d\n    3: 0xc11ec - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h8b590d4d50dc4876\n    4: 0xbfcc8 - <unknown>!Core_execute_block\n"))))    
2022-01-10 22:47:08 [Parachain] 💔 Error importing block 0x90b1f3fa89e2ea080e2c0a08f7befb470f9f4c8077f52f1e1942442d0657389a: Err(Other(ClientImport("Error at calling runtime api: Execution failed: Other(\"Wasm execution trapped: wasm trap: unreachable\\nwasm backtrace:\\n    0: 0x1f1d - <unknown>!rust_begin_unwind\\n    1: 0x1bfa - <unknown>!core::panicking::panic_fmt::he934cb9e7b98bd53\\n    2: 0x12d3 - <unknown>!core::panicking::panic::h25f3e5a9f228197d\\n    3: 0xc11ec - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h8b590d4d50dc4876\\n    4: 0xbfcc8 - <unknown>!Core_execute_block\\n\")")))    
2022-01-10 22:47:08 [Parachain] panicked at 'Storage root must match that calculated.', /home/akru/.cargo/git/checkouts/substrate-7e08433d4c370a21/d76f399/frame/executive/src/lib.rs:503:9
2022-01-10 22:47:08 [Parachain] Block prepare storage changes error:
RuntimeApiError(Application(Execution(Other("Wasm execution trapped: wasm trap: unreachable\nwasm backtrace:\n    0: 0x1f1d - <unknown>!rust_begin_unwind\n    1: 0x1bfa - <unknown>!core::panicking::panic_fmt::he934cb9e7b98bd53\n    2: 0x12d3 - <unknown>!core::panicking::panic::h25f3e5a9f228197d\n    3: 0xc11ec - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h8b590d4d50dc4876\n    4: 0xbfcc8 - <unknown>!Core_execute_block\n"))))    
2022-01-10 22:47:08 [Parachain] 💔 Error importing block 0x90b1f3fa89e2ea080e2c0a08f7befb470f9f4c8077f52f1e1942442d0657389a: Err(Other(ClientImport("Error at calling runtime api: Execution failed: Other(\"Wasm execution trapped: wasm trap: unreachable\\nwasm backtrace:\\n    0: 0x1f1d - <unknown>!rust_begin_unwind\\n    1: 0x1bfa - <unknown>!core::panicking::panic_fmt::he934cb9e7b98bd53\\n    2: 0x12d3 - <unknown>!core::panicking::panic::h25f3e5a9f228197d\\n    3: 0xc11ec - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h8b590d4d50dc4876\\n    4: 0xbfcc8 - <unknown>!Core_execute_block\\n\")")))    
2022-01-10 22:47:09 [Relaychain] 💤 Idle (28 peers), best: #10908232 (0x428d…b0aa), finalized #10908229 (0x89e8…2e5f), ⬇ 693.2kiB/s ⬆ 910.6kiB/s
2022-01-10 22:47:10 [Parachain] 💤 Idle (0 peers), best: #10356 (0xc58d…0437), finalized #10354 (0x6b63…5a90), ⬇ 12.9kiB/s ⬆ 8.8kiB/s
2022-01-10 22:47:12 [Relaychain] ✨ Imported #10908233 (0x8ed7…df8f)                                                                           
2022-01-10 22:47:14 [Relaychain] 💤 Idle (28 peers), best: #10908233 (0x8ed7…df8f), finalized #10908230 (0xc0f7…c133), ⬇ 1.4MiB/s ⬆ 752.6kiB/s

Full log (9M): https://drive.google.com/file/d/1t_HgjjoV0NnwRjie_1D0_pwQFOZVHf4i/view?usp=sharing

[khssnv]

Duplicate #234