Full node syncing issue

[jtremback]

Summary of Bug

• There is some kind of issue in the theta release testnet resulting in full nodes not being able to sync
  • This is not affecting the validators, just full nodes
  • Important detail is that all full nodes were set up by our team. So it may not be an issue with full nodes, but it may be an issue with our configuration
  • However, this did not happen with Gaia v7, so something strange is going on.
  • Ways to fix this:
    • Review full node settings
    • Get one of the validators to set up a full node

[yaruwangway]

Hi, also state-sync tutorial is not working.

[faddat]

Hi there,

Can you describe the symptoms better?

[faddat]

About state sync, @yaruwangway are you trying to ss for V8?

That won't work till about 1500 blocks after v8 is running and may not work at all on testnet because the script pulls from chain registry.

Recommendations:

• fix state sync after release
• use go 1.19 in release to ensure there aren't compatibility issues with state sync
• move information on testnets to this repository
• define full node

I am going to define full node here, as archival with all history, and attempt to sync now.

[yaruwangway]

Hi about state sync, I am syncing mainnet. it is working now, I opened an issue here. It is not a bug, just the peers are not working. We need to think a way to curl live peer from a reliable source.

I also tried to sync v8 Rho-testnet, which is not working.

[yaruwangway]

Hi, @faddat , i see your pr, will test the sync, lets move to talk in the other issue, this is probably not so related this one here.

[faddat]

Ok, in order to work this issue better, I'll need the exact configuration of the nodes that are failing please

[faddat]

let's define a test here for this:

git clone https://github.com/cosmos/gaia
cd gaia
gh pr checkout 2077
go install ./...
gaiad init test
wget https://github.com/cosmos/testnets/raw/master/public/genesis.json.gz
gzip -d genesis.json.gz
mv genesis.json ~/.gaia/config/genesis.json
gaiad start --p2p.seeds "639d50339d7045436c756a042906b9a69970913f@seed-01.theta-testnet.polypore.xyz:26656,3e506472683ceb7ed75c1578d092c79785c27857@seed-02.theta-testnet.polypore.xyz:26656"

That is how I define "full node"

There is also "archive node"

git clone https://github.com/cosmos/gaia
cd gaia
gh pr checkout 2077
go install ./...
gaiad init test
wget https://github.com/cosmos/testnets/raw/master/public/genesis.json.gz
gzip -d genesis.json.gz
mv genesis.json ~/.gaia/config/genesis.json
gaiad start --p2p.seeds "639d50339d7045436c756a042906b9a69970913f@seed-01.theta-testnet.polypore.xyz:26656,3e506472683ceb7ed75c1578d092c79785c27857@seed-02.theta-testnet.polypore.xyz:26656" --pruning nothing

Also needed:

The commit hash running on the testnet.

1:46AM ERR Error in validation err="wrong Block.Header.AppHash.  Expected A406AAD7ACD350A8E18F3C47CB7A40FBAF395225C280DFEC6734BB24BAB0A2D6, got A5A49BA396EF502BB3A12378D8EF26560FDC6225E882E7CE42FF336C0FFB28BB" module=blockchain
1:46AM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected A406AAD7ACD350A8E18F3C47CB7A40FBAF395225C280DFEC6734BB24BAB0A2D6, got A5A49BA396EF502BB3A12378D8EF26560FDC6225E882E7CE42FF336C0FFB28BB" module=p2p peer={"Data":{},"Logger":{}}

This occurs weather or not iavl fast node is enabled. I just reprodueced it. This points at gas.

@yaruwangway @jtremback testnet validators are on rc3?

[mpoke]

An update on this issue.

The nodes that are lagging have minimum-gas-price = "". This means that they accept TXs with 0 fees in their mempool. Validator nodes have minimum-gas-price set to a positive value. As a result, they will not add to their mempool any TX with 0 fees (as CheckTX fails). This means that none of these TXs with 0 fees are ever added to a block, and thus, they remain in the mempool of the nodes lagging behind. In addition, if these TXs are IBC ClientUpdates, then nodes with recheckTx = true will execute them after every block, which leads to the node lagging.

In gaia v8, ClientUpdates messages should by default bypass the minimum fee, i.e.,

gaia/app/app.go

Line 189 in 27234da

bypassMinFeeMsgTypes = GetDefaultBypassFeeMessages()

. However, due to a bug, by default no messages can bypass the fee. This will be fix in v8.0.0-rc4.

In general though, we need to improve the documentation / logging / validation of the minimum-gas-price field in a node's configuration. If this value is smaller than the minimum value across all validator nodes, then this node will accept TXs that will not be accepted by any validator node. These TXs will never be added to blocks.

Thanks to @sergio-mena @ancazamfir for their help figuring out this issue.

[yaruwangway]

@faddat , yes, testnet is on v8-rc3