snapshots from nodes should be identical for a given block height

[heifner]

Currently snapshots generated from nodes running in IRRESVERSIBLE mode differ from those running in SPECULATIVE or HEAD mode. This should be fixed so that deterministic snapshots can be created no matter what mode nodeos is running in.

[heifner]

If I remember correctly the diff is in the list of de-dup transactions. Potentially speculative trxs after irreversible are being included, although that should not be the case. Once JSON support for snapshot is added #710. This becomes easier to debug/fix. The first step is to determine what the differences are.

[linh2931]

I reproduced the problem using the following steps:

1. Check out and build cleanup_improve_json branch so we can convert snapshot to JSON format.
2. Start nodeos with producer_plugin and speculative mode.
3. Pump transactions into it by txn_test_gen_plugin so we will have non-empty transaction_object in snapshot
4. Take a snapshot and stop nodeos. Call this snapshot as snapshot_speculative.
5. Convert snapshot_speculative to JSON format and note the head block. In this run, block_state of the snapshot is
   {"block_num":913,"dpos_proposed_irreversible_blocknum":913,"dpos_irreversible_blocknum": 912,
6. Trim block.log to the head block by eosio-blocklog --trim-blocklog --blocks-dir data/blocks --last 913
7. Restart nodeos using the trimmed block.log and in irreversible read mode
8. Taks a snapshot and convert it to JSON. Name this snapshot as snapshp_irreversible.

snapshot_speculative and snapshp_irreversible only differ in transaction_object. Unexpired transactions are spread over one second boundary

snapshot_speculative:

,"eosio::chain::transaction_object":{
"rows":[
{"expiration":"2022-09-22T18:49:58","trx_id":                                             "577f914ad964716a8c6433c879919234134cdd5623d115621575f7ff385ff37a"}
,{"expiration":"2022-09-22T18:49:58","trx_id":                                            "364c8b219bed7d8a87642bc7e0f7f82f2db84dc5041fede394e9c0fa4e070f49"}
,{"expiration":"2022-09-22T18:49:58","trx_id":                                            "053f0dc999877ee976521a26cc33822f396d93d9b979dacc3ad5c154a6786e01"}
,{"expiration":"2022-09-22T18:49:58","trx_id":                                            "c21aa22ad6bed3bdfbec604593de0d0ffad8808ac24a2bd1a23a7e39a90f3a37"}
,{"expiration":"2022-09-22T18:49:58","trx_id":
...
,{"expiration":"2022-09-22T18:49:59","trx_id":                                            "822f1b8a2f6915f259d11844c1667abfe5111f53d6cef93622d6b9c848bc565e"}
,{"expiration":"2022-09-22T18:49:59","trx_id":                                            "58b72cad2acd9ffc57a4076121ea44df42ecbf856db8ec3775d618f1f97349e8"}
,{"expiration":"2022-09-22T18:49:59","trx_id":                                            "4b28eeaabf527ce5a9a005860f4993cad5b3da93033064f8de0997c9a3faa8ab"}
,{"expiration":"2022-09-22T18:49:59","trx_id":
],
"num_rows":59986
}

snapshp_irreversible:

,"eosio::chain::transaction_object":{
"rows":[
{"expiration":"2022-09-22T18:49:59","trx_id":                                             "822f1b8a2f6915f259d11844c1667abfe5111f53d6cef93622d6b9c848bc565e"}
,{"expiration":"2022-09-22T18:49:59","trx_id":                                            "58b72cad2acd9ffc57a4076121ea44df42ecbf856db8ec3775d618f1f97349e8"}
,{"expiration":"2022-09-22T18:49:59","trx_id":                                            "4b28eeaabf527ce5a9a005860f4993cad5b3da93033064f8de0997c9a3faa8ab"}
,{"expiration":"2022-09-22T18:49:59","trx_id":                                            "a809c6a565695d3cc812cf5a2e0fcb85177ae34f4cfaed3cac498210153a2851"}
,{"expiration":"2022-09-22T18:49:59","trx_id":                                            "ffa98e983139c8be25cc942a8dbb2d617dae9d3f06b3d60b2053e18a33ddead5"}
,{"expiration":"2022-09-22T18:49:59","trx_id"
...
],
"num_rows":57986
}

I also trimmed block.log to 912, which resulted in differences in other objects in the snapshots.