Does a new node really have to synchronize the whole chain in NEO3.0?

[Jim8y]

I've been thinking about this for a very long time, do we really have to synchronize the whole chain to our node even a lot of the data in the chain is out of data and basically will not be used anymore.
Just think about that, after many years of operation, maybe more than 90% of the data in the blockchain will basically not be used anymore, yet a new node still has to spend hours even days to download and prepare that amount of disk space to store that data.
Why can't we just create a snapshot for the chain for every certain height of blocks, the snapshot will only contain the data that is still valid, then the new node will only need to download those snapshot instead of millions of blocks.
This thing can not be done in Ethereum and Bitcoin since they leverage PoW consensus algorithm, miners can only do the same task repeatedly, but NEO is different, the consensus in NEO is started by a single node for each consensus period, which means, that node can also choose to start a different type of task to take a snapshot instead of generating a new block.

[jsolman]

I miss clicked on mobile and accidentally closed an re-opened. I have been thinking about this also. On Neo 3.0 it should be possible to just have the current state I believe also. I’m sure we will continue to discuss it. Thanks for opening the issue.

[jsolman]

Some people need to replay all the notifications, in which case they will need the whole chain, but for those that do not they would be fine with just the current state and block height.

[Jim8y]

Thank you for your response.
I also considered about that issue, some smart contract might need some information like transaction history at some time, thus we need some mechanisms to make sure smart contract can access those data that is normally not needed. I could not figure out how to solve that until I watched your conference report, where you talked about implementing Oracle in NEO3.0, then why can't we provide those history data through Oracle?

[jsolman]

Some NEP5 contracts actually store transfer history in the contract storage; in which case it is included in the current state, allowing it to be accessed by other smart contracts. Since the state is stored currently in a LSM tree (leveldb) rather than in ram, it isn’t really a technical problem, but it is wasteful since all nodes are forced to sync it and store it. Therefore, it should likely be economically discouraged by the Neo 3.0 economic model. The Oracle might be acceptable to use in some situations where history is desired, but it comes with accepting a trusted domain as a source of truth (possibly a reasonable tradeoff in many situations).

The protocol itself could possibly support operating in both modes. To be able to sync the state at a certain block height, the storage layer should support saving snapshots at various block heights so that many nodes would have the data to distribute to those syncing at that height. This feature would require using a different storage library than leveldb (as it does not support saving storage snapshots after close of the DB). Once a new node has synced the state at a particular height they would need all the blocks between that height and the current height to complete syncing to the latest state. The solution requires a great deal of changes to implement but it is certainly possible.

[vncoelho]

@Liaojinghui, I think that this is a nice discussion.
Currently, I believe that we almost already have all the tools here in the NEO core needed for this task.
Furthermore, NEO 3.0 will have even more tools for ensuring the integrity in such bootstrap.

Maybe it is not really a priority now, however, I believe it is a good line of evolving the code.

On the other hand, I believe that the first step would be a syncing like NEL team recently did, without verification and just persisting, which surely speed up the process.

[igormcoelho]

This is a very important suggestion. As UTXO global assets will be gone, older tx will only be necessary for processing inside smart contracts. On Ethereum, the limit to go back in time is around 255 blocks I think, so if we adopt a similar strategy (perhaps much larger), we will always have a safe limit on what information needs to be on chain for processing.

[Qiao-Jin]

This is a very good idea as most of the previous db records are not likely to be needed in most current activitis I think.

I think TTL (time to live) could be a techonical solution. In a DB with TTL, a record inserted will be deleted automatically some certain time period after its insertion (i.e. we can set the time to 255 * 15s so that data stored are enough for most cases). Meanwhile, a newly-stared node will only fetch data record created within certain time period (255 * 15s for example). Rocksdb has already supported such functionality: https://github.com/facebook/rocksdb/wiki/Time-to-Live

At the same time some "whole nodes" which store all history records can be set up for ordinary nodes to fetch records long before.

The benefit is that we don't need to maintain a list of snapshots manually, everything is done automatically.

[shargon]

If you can invoke a syscall for retrieve a block with the height X, you need the whole chain.

[Jim8y]

If you can invoke a syscall for retrieve a block with the height X, you need the whole chain.

Even so, that kind of transaction is only meaningful to the supernode since ordinary node has no right to join the consensus. Moreover, ordinary nodes do not care about those useless dead data. Think about it, many years later, a new developer wants to join the community, the first problem that he needs to overcome might be syncronizing a chain with TB of useless data.

[ixje]

Think about it, many years later, a new developer wants to join the community, the first problem that he needs to overcome might be syncronizing a chain with TB of useless data.

☝️ A pruning mechanism is a must.