Can not get node working after update to 0.8.2 #275

dzmitrykliapkou · 2021-03-12T09:33:45Z

Hi

Our node got stuck with Wrong.Block.Header.AppHash error. We've performed state-recovery to 149170000 (the last but one snapshot), but after that the node gets stuсk at 1491700020, throwing the following error:

bnbchaind: panic: Orphan expires before it comes alive. 149170017 > 149170000
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: goroutine 640170 [running]:
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/iavl.(*nodeDB).saveOrphan(0xc000ae3c50, 0xc0362a8860, 0x20, 0x20, 0x8e42761,
0x8e42750)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint-iavl@v0.12.0-binance.4/nodedb.go:166
 +0x253
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/iavl.(*nodeDB).SaveOrphans(0xc000ae3c50, 0x8e42765, 0xc031a16090)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint-iavl@v0.12.0-binance.4/nodedb.go:159
 +0x1b2
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/iavl.(*MutableTree).SaveVersion(0xc004fb1e60, 0xc00006e6b0, 0xc00006e6b0, 0x3
00000001aa7201, 0xffffffffffffffff, 0x1, 0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint-iavl@v0.12.0-binance.4/mutable_tree.
go:407 +0x63b
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/store.(*IavlStore).Commit(0xc000b49380, 0xc00099e510, 0x169c042, 0x3,
0xc0004955b8)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-cosmos-sdk@v0.25.0-binance.24/store/iavlstore.g
o:80 +0x32
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/store.commitStores(0x8e42765, 0xc00099f290, 0x7020105, 0x0, 0xffffffff
ffffffff, 0xc010e428d8)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-cosmos-sdk@v0.25.0-binance.24/store/rootmultist
ore.go:489 +0x1e5
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/store.(*rootMultiStore).Commit(0xc00016c4d0, 0x0, 0x0, 0x0, 0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-cosmos-sdk@v0.25.0-binance.24/store/rootmultist
ore.go:209 +0x68
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc000ae0a50, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-cosmos-sdk@v0.25.0-binance.24/baseapp/baseapp.g
o:957 +0xff
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/binance-chain/node/app.(*BinanceChain).Commit(0xc0009d5200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0
, 0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /server/temp/node/app/app.go:866 +0x50
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).CommitSync(0xc026d5a630, 0x0, 0x
0, 0x0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-cosmos-sdk@v0.25.0-binance.24/server/concurrent
/async_local_client.go:377 +0x1fe
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0xc026d5c360, 0x0, 0x0, 0xc03
71b43c0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/proxy/app_conn.go:
119 +0x33
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(0xc03ab8c150, 0xa, 0x0, 0xc03751a740
, 0x6, 0xc03d0b81a0, 0x14, 0x8e42765, 0xe4e8c67, 0xc0370071c0, ...)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/state/execution.go
:209 +0x224
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(0xc03ab8c150, 0xa, 0x0, 0xc03751
a740, 0x6, 0xc03d0b81a0, 0x14, 0x8e42765, 0xe4e8c67, 0xc0370071c0, ...)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/state/execution.go
:163 +0x6b6
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc0188441a0)
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/blockchain/v0/reac
tor.go:401 +0x983
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
Mar 12 09:21:19 ip-172-31-29-224 bnbchaind: /root/bnbchain/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.5/blockchain/v0/reac
tor.go:127 +0x84
Mar 12 09:21:19 ip-172-31-29-224 systemd: binance.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 12 09:21:19 ip-172-31-29-224 systemd: Unit binance.service entered failed state.
Mar 12 09:21:19 ip-172-31-29-224 systemd: binance.service failed.
Mar 12 09:21:19 ip-172-31-29-224 systemd: binance.service holdoff time over, scheduling restart.

bnc.log shows that the last Executed block is 1491700021.
It would be great if you could help us. Thanks

The text was updated successfully, but these errors were encountered:

chainwhisper · 2021-03-17T09:08:34Z

Please try revert to 1491690000.

and double check the open file limit. It cannot be less than 1024.

dzmitrykliapkou · 2021-03-18T11:45:13Z

I've tried to revert to 1491690000, but after that binance service refused to start with new error:

bnbchaind: failed to get rootMultistore: no data

Also I've checked open file limit and it is 655350 per process

chainwhisper · 2021-03-18T12:33:29Z

I've tried to revert to 1491690000, but after that binance service refused to start with new error:
bnbchaind: failed to get rootMultistore: no data
Also I've checked open file limit and it is 655350 per process

please try the latest snapshot: https://docs.binance.org/guides/node/snapshot.html

dzmitrykliapkou · 2021-03-26T13:00:14Z

please try the latest snapshot: https://docs.binance.org/guides/node/snapshot.html

I've downloaded the latest snapshot and started the service, but after ~1hr binance service failed with:

Mar 26 12:56:23 ip-172-31-29-224 bnbchaind: ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected A5B794D9B4D96
CDA0F60BFA1BE660BD49B5DEF0A0262A3A3FC066E5965742575, got 5AEDFAE1C95C3D21C27EF0091DB2045535B53E0D92A18B4FCE40751666E81881
Mar 26 12:56:23 ip-172-31-29-224 systemd: binance.service: main process exited, code=exited, status=1/FAILURE
Mar 26 12:56:23 ip-172-31-29-224 systemd: Unit binance.service entered failed state.
Mar 26 12:56:23 ip-172-31-29-224 systemd: binance.service failed.
Mar 26 12:56:23 ip-172-31-29-224 systemd: binance.service holdoff time over, scheduling restart.

DevMinJong · 2021-04-14T12:27:16Z

@huangsuyu same issue
Did you solve the issue?

chainwhisper · 2021-04-14T13:37:56Z

please try these steps: @DevMinJong @dzmitrykliapkou

stop the bnbchaind
delete the data folder
extract the snapshot to the new data folder
update the state_sync_reactor to false
start the bnbchaind

eloc3147 · 2021-06-11T07:26:54Z

Same issue here. Using the April snapshot, and all the steps above.

Binance Chain Release: v0.8.2;Binance Chain Commit: 88ce4872; Cosmos Release: v0.25.0-binance.24; Tendermint Release: v0.32.3-binance.5;

log:

I[2021-06-11|07:07:33.695] replayed all tx. Starting match module=dex height=153975665
I[2021-06-11|07:07:33.695] No order comes in for the block module=dexkeeper
I[2021-06-11|07:07:37.742] ABCI Handshake App Info module=consensus height=153975665 hash=E8090A2A1D20258FDB3C2EEE151C4B1A21D57BA8346A9BB84216CB6F592E3051 software-version= protocol-version=0
I[2021-06-11|07:07:37.742] ABCI Replay Blocks module=consensus appHeight=153975665 storeHeight=153975666 stateHeight=153975665
I[2021-06-11|07:07:37.742] Replay last block using real app module=consensus
ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected E8090A2A1D20258FDB3C2EEE151C4B1A21D57BA8346A9BB84216CB6F592E3051, got 952153E05A393E5395B9638CDA3B018DC4FF7EA1F197E4523C38A551DF048803

kenorb · 2021-11-13T15:24:02Z

See: cosmos/cosmos-sdk#10281

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can not get node working after update to 0.8.2 #275

Can not get node working after update to 0.8.2 #275

dzmitrykliapkou commented Mar 12, 2021 •

edited

Loading

chainwhisper commented Mar 17, 2021 •

edited

Loading

dzmitrykliapkou commented Mar 18, 2021

chainwhisper commented Mar 18, 2021

dzmitrykliapkou commented Mar 26, 2021 •

edited

Loading

DevMinJong commented Apr 14, 2021 •

edited

Loading

chainwhisper commented Apr 14, 2021

eloc3147 commented Jun 11, 2021

kenorb commented Nov 13, 2021

Can not get node working after update to 0.8.2 #275

Can not get node working after update to 0.8.2 #275

Comments

dzmitrykliapkou commented Mar 12, 2021 • edited Loading

chainwhisper commented Mar 17, 2021 • edited Loading

dzmitrykliapkou commented Mar 18, 2021

chainwhisper commented Mar 18, 2021

dzmitrykliapkou commented Mar 26, 2021 • edited Loading

DevMinJong commented Apr 14, 2021 • edited Loading

chainwhisper commented Apr 14, 2021

eloc3147 commented Jun 11, 2021

kenorb commented Nov 13, 2021

dzmitrykliapkou commented Mar 12, 2021 •

edited

Loading

chainwhisper commented Mar 17, 2021 •

edited

Loading

dzmitrykliapkou commented Mar 26, 2021 •

edited

Loading

DevMinJong commented Apr 14, 2021 •

edited

Loading