Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

importing genesis state is very slow #669

Closed
4 tasks
faddat opened this issue Feb 20, 2021 · 9 comments
Closed
4 tasks

importing genesis state is very slow #669

faddat opened this issue Feb 20, 2021 · 9 comments

Comments

@faddat
Copy link
Contributor

faddat commented Feb 20, 2021

Summary of Bug

Pulling genesis state into Gaia v4.0.4 is extremely slow. Small discussion of it here:

tendermint/tendermint#6154

Version

v4.0.4

Steps to Reproduce

Start a gaia node


For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@tac0turtle
Copy link
Member

I dont believe tendermint is the root cause here. I need to go read the sdks code but from my understanding the sdk reads the genesis file and on InitChain the info is sent to tendermint.

@faddat
Copy link
Contributor Author

faddat commented Feb 20, 2021

Interesting-- in tendermint, I found a function that looked like it was reading the genesis state, linked to in that discussion there. I wasn't able to find anything like that in the SDK, but curious to hear your result :).

@zmanian
Copy link
Member

zmanian commented Feb 23, 2021

Root cause analysis here

cosmos/cosmos-sdk#7766

Currently, blocked on replacing IAVL.

@faddat
Copy link
Contributor Author

faddat commented Feb 24, 2021

Thank you!

@faddat
Copy link
Contributor Author

faddat commented Mar 4, 2021

[root@blurtarch gaiad]# gaiad start --p2p.seeds bf8328b66dceb4987e5cd94430af66045e59899f@public-seed.cosmos.vitwit.com:26656,cfd785a4224c7940e9a10f6c1ab24c343e923bec@164.68.107.188:26656,d72b3011ed46d783e369fdf8ae2055b99a1e5074@173.249.50.25:26656,ba3bacc714817218562f743178228f23678b2873@public-seed-node.cosmoshub.certus.one:26656,3c7cad4154967a294b3ba1cc752e40e8779640ad@84.201.128.115:26656
10:33PM INF starting ABCI with Tendermint
10:33PM INF Starting multiAppConn service impl=multiAppConn module=proxy
10:33PM INF Starting localClient service connection=query impl=localClient module=abci-client
10:33PM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
10:33PM INF Starting localClient service connection=mempool impl=localClient module=abci-client
10:33PM INF Starting localClient service connection=consensus impl=localClient module=abci-client
10:33PM INF Starting EventBus service impl=EventBus module=events
10:33PM INF Starting PubSub service impl=PubSub module=pubsub
10:33PM INF Starting IndexerService service impl=IndexerService module=txindex
10:33PM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 software-version=
10:33PM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
10:37PM INF asserting crisis invariants inv=0/11 module=x/crisis
10:37PM INF asserting crisis invariants inv=1/11 module=x/crisis
10:37PM INF asserting crisis invariants inv=2/11 module=x/crisis
10:37PM INF asserting crisis invariants inv=3/11 module=x/crisis
10:37PM INF asserting crisis invariants inv=4/11 module=x/crisis
10:37PM INF asserting crisis invariants inv=5/11 module=x/crisis
10:38PM INF asserting crisis invariants inv=6/11 module=x/crisis
10:38PM INF asserting crisis invariants inv=7/11 module=x/crisis


11:07PM INF asserting crisis invariants inv=8/11 module=x/crisis
11:07PM INF asserting crisis invariants inv=9/11 module=x/crisis
11:07PM INF asserting crisis invariants inv=10/11 module=x/crisis
11:07PM INF asserted all invariants duration=1803438.12619 height=5200791 module=x/crisis
11:07PM INF created new capability module=ibc name=ports/transfer
11:07PM INF port binded module=x/ibc/port port=transfer
11:07PM INF claimed capability capability=1 module=transfer name=ports/transfer
11:07PM INF Completed ABCI Handshake - Tendermint and App are synced appHash= appHeight=0 module=consensus
11:07PM INF Version info block=11 p2p=8 software=v0.34.8
11:07PM INF This node is not a validator addr=80945A898B6D3E577AA148BF127F89EBE9020DC4 module=consensus pubKey=9zwf2S11nhuNMzripexW1kBTalBUUMIt2yie4WTT/QA=
11:07PM INF P2P Node ID ID=52af6bd5493d7811cd4c811143da80aa18370d76 file=/root/.gaia/config/node_key.json module=p2p
11:07PM INF Adding persistent peers addrs=[] module=p2p
11:07PM INF Adding unconditional peer ids ids=[] module=p2p
11:07PM INF Add our address to book addr={"id":"52af6bd5493d7811cd4c811143da80aa18370d76","ip":"0.0.0.0","port":26656} book=/root/.gaia/config/addrbook.json module=p2p
11:07PM INF Starting Node service impl=Node
11:07PM INF Starting pprof server laddr=localhost:6060
11:07PM INF Starting P2P Switch service impl="P2P Switch" module=p2p
11:07PM INF Starting Evidence service impl=Evidence module=evidence
11:07PM INF Starting StateSync service impl=StateSync module=statesync
11:07PM INF Starting PEX service impl=PEX module=pex
11:07PM INF Starting AddrBook service book=/root/.gaia/config/addrbook.json impl=AddrBook module=p2p
11:07PM INF Starting RPC HTTP server on 127.0.0.1:26657 module=rpc-server
11:07PM INF Starting Mempool service impl=Mempool module=mempool
11:07PM INF Starting BlockchainReactor service impl=BlockchainReactor module=blockchain
11:07PM INF Starting BlockPool service impl=BlockPool module=blockchain
11:07PM INF Starting Consensus service impl=ConsensusReactor module=consensus
11:07PM INF Reactor  module=consensus waitSync=true
11:07PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
11:07PM INF Saving AddrBook to file book=/root/.gaia/config/addrbook.json module=p2p size=0
11:07PM INF No addresses to dial. Falling back to seeds module=pex
11:07PM INF Dialing peer address={"id":"cfd785a4224c7940e9a10f6c1ab24c343e923bec","ip":"164.68.107.188","port":26656} module=p2p
11:07PM ERR Error dialing seed err="auth failure: secret conn failed: read tcp 95.217.193.163:45760->164.68.107.188:26656: i/o timeout" module=p2p seed={"id":"cfd785a4224c7940e9a10f6c1ab24c343e923bec","ip":"164.68.107.188","port":26656}
11:07PM INF Dialing peer address={"id":"ba3bacc714817218562f743178228f23678b2873","ip":"5.83.160.108","port":26656} module=p2p
11:07PM ERR Error dialing seed err="dial tcp 5.83.160.108:26656: connect: connection refused" module=p2p seed={"id":"ba3bacc714817218562f743178228f23678b2873","ip":"5.83.160.108","port":26656}
11:07PM INF Dialing peer address={"id":"3c7cad4154967a294b3ba1cc752e40e8779640ad","ip":"84.201.128.115","port":26656} module=p2p
11:07PM ERR Error dialing seed err="auth failure: secret conn failed: read tcp 95.217.193.163:40766->84.201.128.115:26656: i/o timeout" module=p2p seed={"id":"3c7cad4154967a294b3ba1cc752e40e8779640ad","ip":"84.201.128.115","port":26656}
11:07PM INF Dialing peer address={"id":"bf8328b66dceb4987e5cd94430af66045e59899f","ip":"159.203.104.207","port":26656} module=p2p
11:07PM ERR Error dialing seed err="dial tcp 159.203.104.207:26656: connect: connection refused" module=p2p seed={"id":"bf8328b66dceb4987e5cd94430af66045e59899f","ip":"159.203.104.207","port":26656}
11:07PM INF Dialing peer address={"id":"d72b3011ed46d783e369fdf8ae2055b99a1e5074","ip":"173.249.50.25","port":26656} module=p2p
11:07PM ERR Error dialing seed err="auth failure: secret conn failed: read tcp 95.217.193.163:33444->173.249.50.25:26656: i/o timeout" module=p2p seed={"id":"d72b3011ed46d783e369fdf8ae2055b99a1e5074","ip":"173.249.50.25","port":26656}
11:07PM ERR Couldn't connect to any seeds module=p2p
11:08PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10

We are down to 34 minutes to start the node on that 12 core, 128GB RAM machine of mine.

@zmanian
Copy link
Member

zmanian commented Mar 4, 2021

what about with crisis invariants disabled?

@faddat
Copy link
Contributor Author

faddat commented Mar 4, 2021

I can try it that way, for the snippet I went with all-defaults. Think we should change the snippet?

@zmanian
Copy link
Member

zmanian commented Mar 4, 2021

it's pretty pointless to check crisis invariants again for every new node that joins the network.

I think without crisis invariants startups should be on the order of 10 min.

@faddat
Copy link
Contributor Author

faddat commented Mar 4, 2021

12:24AM INF starting ABCI with Tendermint
12:24AM INF Starting multiAppConn service impl=multiAppConn module=proxy
12:24AM INF Starting localClient service connection=query impl=localClient module=abci-client
12:24AM INF Starting localClient service connection=snapshot impl=localClient module=abci-client
12:24AM INF Starting localClient service connection=mempool impl=localClient module=abci-client
12:24AM INF Starting localClient service connection=consensus impl=localClient module=abci-client
12:24AM INF Starting EventBus service impl=EventBus module=events
12:24AM INF Starting PubSub service impl=PubSub module=pubsub
12:24AM INF Starting IndexerService service impl=IndexerService module=txindex
12:24AM INF ABCI Handshake App Info hash= height=0 module=consensus protocol-version=0 software-version=
12:24AM INF ABCI Replay Blocks appHeight=0 module=consensus stateHeight=0 storeHeight=0
12:28AM INF created new capability module=ibc name=ports/transfer
12:28AM INF port binded module=x/ibc/port port=transfer
12:28AM INF claimed capability capability=1 module=transfer name=ports/transfer
12:28AM INF Completed ABCI Handshake - Tendermint and App are synced appHash= appHeight=0 module=consensus
12:28AM INF Version info block=11 p2p=8 software=v0.34.8
12:28AM INF This node is not a validator addr=80945A898B6D3E577AA148BF127F89EBE9020DC4 module=consensus pubKey=9zwf2S11nhuNMzripexW1kBTalBUUMIt2yie4WTT/QA=
12:28AM INF P2P Node ID ID=52af6bd5493d7811cd4c811143da80aa18370d76 file=/root/.gaia/config/node_key.json module=p2p
12:28AM INF Adding persistent peers addrs=[] module=p2p
12:28AM INF Adding unconditional peer ids ids=[] module=p2p
12:28AM INF Add our address to book addr={"id":"52af6bd5493d7811cd4c811143da80aa18370d76","ip":"0.0.0.0","port":26656} book=/root/.gaia/config/addrbook.json module=p2p
12:28AM INF Starting Node service impl=Node
12:28AM INF Starting P2P Switch service impl="P2P Switch" module=p2p
12:28AM INF Starting Consensus service impl=ConsensusReactor module=consensus
12:28AM INF Starting pprof server laddr=localhost:6060
12:28AM INF Reactor  module=consensus waitSync=true
12:28AM INF Starting RPC HTTP server on 127.0.0.1:26657 module=rpc-server
12:28AM INF Starting Evidence service impl=Evidence module=evidence
12:28AM INF Starting StateSync service impl=StateSync module=statesync
12:28AM INF Starting PEX service impl=PEX module=pex
12:28AM INF Starting AddrBook service book=/root/.gaia/config/addrbook.json impl=AddrBook module=p2p
12:28AM ERR Connecting to seed failed err="error looking up host (cosmos-cosmoshub-2-seed.nodes.polychainlabs.com): lookup cosmos-cosmoshub-2-seed.nodes.polychainlabs.com on 213.133.100.100:53: no such host" module=pex
12:28AM INF Starting Mempool service impl=Mempool module=mempool
12:28AM INF Starting BlockchainReactor service impl=BlockchainReactor module=blockchain
12:28AM INF Starting BlockPool service impl=BlockPool module=blockchain
12:28AM INF Saving AddrBook to file book=/root/.gaia/config/addrbook.json module=p2p size=0
12:28AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
12:28AM INF No addresses to dial. Falling back to seeds module=pex
12:28AM INF Dialing peer address={"id":"2626942148fd39830cb7a3acccb235fab0332d86","ip":"173.212.199.36","port":26656} module=p2p
12:28AM ERR Error dialing seed err="dial tcp 173.212.199.36:26656: connect: connection refused" module=p2p seed={"id":"2626942148fd39830cb7a3acccb235fab0332d86","ip":"173.212.199.36","port":26656}
12:28AM INF Dialing peer address={"id":"3c7cad4154967a294b3ba1cc752e40e8779640ad","ip":"84.201.128.115","port":26656} module=p2p
12:28AM ERR Error dialing seed err="dial tcp 84.201.128.115:26656: i/o timeout" module=p2p seed={"id":"3c7cad4154967a294b3ba1cc752e40e8779640ad","ip":"84.201.128.115","port":26656}
12:28AM INF Dialing peer address={"id":"cfd785a4224c7940e9a10f6c1ab24c343e923bec","ip":"164.68.107.188","port":26656} module=p2p
12:28AM INF Starting Peer service impl="Peer{MConn{164.68.107.188:26656} cfd785a4224c7940e9a10f6c1ab24c343e923bec out}" module=p2p peer={"id":"cfd785a4224c7940e9a10f6c1ab24c343e923bec","ip":"164.68.107.188","port":26656}
12:28AM INF Starting MConnection service impl=MConn{164.68.107.188:26656} module=p2p peer={"id":"cfd785a4224c7940e9a10f6c1ab24c343e923bec","ip":"164.68.107.188","port":26656}
12:28AM INF Added peer module=p2p peer={"Data":{},"Logger":{}}
12:29AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=1 numToDial=9
12:29AM INF We need more addresses. Sending pexRequest to random peer module=pex peer={"Data":{},"Logger":{}}
12:29AM INF No addresses to dial. Falling back to seeds module=pex

It was 4 minutes :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants