2.0.3 Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.',

[sourabhniyogi]

Describe the bug

Karura Full node panics with storage root match failures. Syncing is functional but it appears Karura is having trouble producing blocks at the same frequency with 2.x ?

Panic trace is:

Hash: given=3e37e8036b54d33fbe451808e798ee50cc55ed3409f65c421f1d377b01170e8e, expected=ef8dd84f391b934eb796c56be315d0db3cc53d945f58bd855247f845c2e336ec

====================

Version: 2.0.3-15da2db0-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/a85f584aebd9b08314bf30b9adc17b4a752143e5/library/std/src/panicking.rs:626:17
   2: std::panicking::begin_panic::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: std::panicking::begin_panic
   5: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::final_checks
   6: tracing::span::Span::in_scope
   7: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block
   8: <karura_runtime::Runtime as sp_api::runtime_decl_for_Core::Core<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<<<acala_primitives::signature::AcalaMultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,u32>,karura_runtime::Call,acala_primitives::signature::AcalaMultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<karura_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<karura_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<karura_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<karura_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<karura_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<karura_runtime::Runtime>,module_transaction_payment::ChargeTransactionPayment<karura_runtime::Runtime>,module_evm::SetEvmOrigin<karura_runtime::Runtime>)>>>>::execute_block
   9: std::panic::catch_unwind
  10: std::thread::local::LocalKey<T>::with
  11: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
  12: sc_executor::wasm_runtime::RuntimeCache::with_instance
  13: <sc_executor::native_executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  14: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute_aux
  15: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute_using_consensus_failure_handler
  16: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  17: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  18: sp_api::runtime_decl_for_Core::execute_block_call_api_at
  19: <karura_runtime::RuntimeApiImpl<__SR_API_BLOCK__,RuntimeApiImplCall> as sp_api::Core<__SR_API_BLOCK__>>::Core_execute_block_runtime_api_impl
  20: sp_api::Core::execute_block_with_context
  21: sc_service::client::client::Client<B,E,Block,RA>::prepare_block_storage_changes
  22: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  23: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  24: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  25: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  26: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  27: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  28: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
  29: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  30: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  31: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  32: tokio::runtime::enter::Enter::block_on
  33: tokio::runtime::handle::Handle::block_on
  34: tokio::runtime::task::core::CoreStage<T>::poll
  35: tokio::runtime::task::harness::Harness<T,S>::poll
  36: tokio::runtime::blocking::pool::Inner::run
  37: std::sys_common::backtrace::__rust_begin_short_backtrace
  38: core::ops::function::FnOnce::call_once{{vtable.shim}}
  39: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/a85f584aebd9b08314bf30b9adc17b4a752143e5/library/alloc/src/boxed.rs:1575:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/a85f584aebd9b08314bf30b9adc17b4a752143e5/library/alloc/src/boxed.rs:1575:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/a85f584aebd9b08314bf30b9adc17b4a752143e5/library/std/src/sys/unix/thread.rs:71:17
  40: start_thread
  41: __clone


Thread 'tokio-runtime-worker' panicked at 'Storage root must match that calculated.', /root/.cargo/git/checkouts/substrate-f4423eebfa0046a3/d76f399/frame/executive/src/lib.rs:503

Expected Behavior

No panicking

Current Behavior

Panicking

Steps to Reproduce

1. Run full node 2.0.3 after running 1.x eg

• Command line options:
  ./target/release/acala --chain karura --pruning archive --rpc-port 9305 --ws-port 9205 --rpc-cors all

[xlc]

This is known issue paritytech/substrate#9697

Please use --state-cache-size=0 as suggested in the issue for now.