Full node (download all the blocks of the blockchain) - Block download no longer increases

[davjack]

In order to have access to all the data of the Theta blockchain locally without going through an intermediary, I've been downloading the entire Theta blockchain through theta-protocol-ledger (on an Ubuntu Server in version 3.2.0) for several months now.

It was working well, but suddenly the number of blocks did not increase anymore. It remains frozen on the block 11217793 ("current_height" increases well and matches the block displayed on https://explorer.thetatoken.org/ but "latest_finalized_block_height" does not increase anymore.). I have to say that I didn't do anything before the problem occurred (I upgraded to version 3.2.0, the day of the announcement, so well before the problem appeared). Also, this is the second time this has happened to me. The first time, I decided to start again from scratch the installation of my full node, but apparently without success.

Unfortunately, I can't find a solution to my problem in the ThetaLabs documentation or on the Github of the application.
I use the Genesis Snapshot for my fullnode (downloaded here: https://github.com/thetatoken/theta-protocol-ledger/tree/master/integration/mainnet/walletnode ) and the content of my configuration file config.yaml is as follows:

# Theta mainnet guardian configuration
storage:
  statePruningEnabled: false
p2p:
  opt: 0
  port: 30002
  seeds: 18.217.234.19:21000,3.16.9.73:21000,3.20.109.241:21000,18.223.165.134:21000,35.184.232.41:21000,35.230.172.8:21000,34.83.20>
  minNumPeers: 24
  maxNumPeers: 96
  natMapping: true
rpc:
  port: 16888
  enabled: true
  maxConnection: 5
log:
  levels: "*:info,guardian:debug"`

Is this problem known (perhaps a corrupted downloaded block)?

And if it is known, is there a way to solve the problem without starting the download all over again?

[jieyilong]

Thanks for reporting the issue. If you're ok with starting from the most recent snapshot, you can restart the full node following the instruction here:

1. Stop the Theta process on the validator node

2. Execute the following commands to download the latest THETA snapshot. Please replace <CONFIG_FOLDER> with the actual path to your THETA config folder

mv <CONFIG_FOLDER>/db <CONFIG_FOLDER>/db.old

mv <CONFIG_FOLDER>/snapshot <CONFIG_FOLDER>/snapshot.old

wget -O <CONFIG_FOLDER>/snapshot `curl -k https://mainnet-data.thetatoken.org/snapshot`

3. Restart the THETA node as usual.

See also: https://docs.thetatoken.org/docs/running-a-guardian-node-through-command-line

[davjack]

Thanks for your answer Jieyi.

The problem with this method is that I can't get the data from the blocks prior to the snapshot. For example if I run the following command to get the data of block 15, I don't get any result:

curl -X POST -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"theta.GetBlockByHeight","params":[{"height":"15"}],"id":1}' http://localhost:16888/rpc

Result :

{"jsonrpc":"2.0","id":1,"result":{}}

[huttarichard]

I have very much the same issue now. @jieyilong node syncs forever.

I started from snapshot, thetacli query status after 24 hours sync is still true: (as @davjack described this was working fine before, usually took like 2 hours to sync the node)

{
    "address": "0x8D05d8A0d25fd6c3ad976d03fbe70e5A763a1c7f",
    "chain_id": "mainnet",
    "current_epoch": "15319680",
    "current_height": "15233565",
    "current_time": "1651639567",
    "genesis_block_hash": "0xd8836c6cf3c3ccea0b015b4ed0f9efb0ffe6254db793a515843c9d0f68cbab65",
    "latest_finalized_block_epoch": "15313108",
    "latest_finalized_block_hash": "0x7863379c5d0232f504b7395f0f85b278d1be38f027bbfd2a7360d918a0024e5d",
    "latest_finalized_block_height": "15227000",
    "latest_finalized_block_time": "1651598986",
    "peer_id": "0x8D05d8A0d25fd6c3ad976d03fbe70e5A763a1c7f",
    "syncing": true
}

[mbt-commander]

Following along this thread I'm doing the following:

• changed the storage section of default config to only have statePruningEnabled: false (same as OP)
• instead of using the snapshot in the docs, I'm using the genesis snapshot (also mentioned in this thread): walletnode/genesis

This set up appears to be working, but unbelievably slow. Running on gigabit fiber (wired) and a dedicated box with SSD drive and 32GB RAM, it's only finalized ~16500 blocks in 24 hours. At this rate it will take almost 2.5 years to download the full blockchain history. @jieyilong any insight on how to speed this up (thanks in advance)?

I'll keep posting until I figure this out, maybe will help someone else who wants to run a full node. Would also very much appreciate some help from anyone that might have already set up a full node, have been working on a web3 Theta project for months now and need to have a full node as the Theta Explorer API just won't work for my use case anymore.

Also does anyone know the difference between pending block hashes and fast sync tip? pending increases much faster than sync tip, not clear on the relationship between the two.

Also how do these relate to latest_finalized_block_height in the thetacli status query output?

[1921Aaron]

I had the same problem, on version 3.3.0

[0xChupaCabra]

problem is still here, sync process from genesis is really slow, is there any snapshot with ancient blocks data?