prune Lite Fullnode data error: data not found in store

[halibobo1205]

I use Toolkit pruning Lite Fullnode data regularly.

Expected behaviour

It can be executed normally.

Actual behaviour

The command sometimes quits abnormally.

Steps to reproduce the behaviour

Backtrace

Start create snapshot.
Begin to split the dbs.
Begin to merge checkpoint to dataset.
Begin to fill 65536 block, genesis block and trans to snapshot.
fillBlockAndTrans  99% │███████████████████████████████████████████▉│ 65522/65536 (0:01:34 / 0:00:00)
data not found in store, dbName: block-index, key: [0, 0, 0, 0, 3, 64, -96, 32]
Usage: <main class> db lite [-h] -ds=<datasetPath> -fn=<fnDataPath>
                            [-o=<operate>] [-t=<type>]
Split lite data for java-tron.
      -ds, --dataset-path=<datasetPath>
                            when operation is `split`,`dataset-path` is the
                              path that store the `snapshot` or `history`,when
                              operation is `split`,`dataset-path` is the
                              `history` data path.
      -fn, --fn-data-path=<fnDataPath>
                            the database path to be split or merged.
  -h, --help
  -o, --operate=<operate>   operate: [ split, merge ]. Default: split
  -t, --type=<type>         only used with operate=split: [ snapshot, history
                              ]. Default: snapshot
Exit Codes:
  0   Successful
  1   Internal error: exception occurred,please check toolkit.log

[lurais]

I use Toolkit for generating or pruning Lite Fullnode data regularly.

Expected behaviour

It can be executed normally.

Actual behaviour

The command sometimes quits abnormally.

Steps to reproduce the behaviour

Backtrace

Start create snapshot.
Begin to split the dbs.
Begin to merge checkpoint to dataset.
Begin to fill 65536 block, genesis block and trans to snapshot.
fillBlockAndTrans  99% │███████████████████████████████████████████▉│ 65522/65536 (0:01:34 / 0:00:00)
data not found in store, dbName: block-index, key: [0, 0, 0, 0, 3, 64, -96, 32]
Usage: <main class> db lite [-h] -ds=<datasetPath> -fn=<fnDataPath>
                            [-o=<operate>] [-t=<type>]
Split lite data for java-tron.
      -ds, --dataset-path=<datasetPath>
                            when operation is `split`,`dataset-path` is the
                              path that store the `snapshot` or `history`,when
                              operation is `split`,`dataset-path` is the
                              `history` data path.
      -fn, --fn-data-path=<fnDataPath>
                            the database path to be split or merged.
  -h, --help
  -o, --operate=<operate>   operate: [ split, merge ]. Default: split
  -t, --type=<type>         only used with operate=split: [ snapshot, history
                              ]. Default: snapshot
Exit Codes:
  0   Successful
  1   Internal error: exception occurred,please check toolkit.log

Have you tried to use the official data to generate Lite Fullnode data? Have you tried to stop the node abnormally?

[halibobo1205]

@lurais Because I don't have enough disk space, I regularly use Toolkit for generating or pruning Lite Fullnode data. I stop the node by kill -9, in most cases there is no problem.

[lurais]

@lurais Because I don't have enough disk space, I regularly use Toolkit for generating or pruning Lite Fullnode data. I stop the node by kill -9, in most cases there is no problem.

From the error log, some data may be exist in checkpoint but not found in the other stores(just like block-index), can you provide the config file you used to start the node?

[forfreeday]

It's also possible that kill -9 is causing the problem?

[halibobo1205]

@forfreeday The error doesn't happen every time.

[halibobo1205]

net {
  type = mainnet
  # type = testnet
}

# prometheus metrics start
 node.metrics = {
  prometheus{
    enable=true
    port="9527"
  }
 }

# prometheus metrics end

storage {
  # Directory for storing persistent data
  db.version = 2,
  db.engine = "LEVELDB",
  db.sync = false,
  db.directory = "database",
  index.directory = "index",
  transHistory.switch = "on",
  # You can custom these 14 databases' configs:

  # account, account-index, asset-issue, block, block-index,
  # block_KDB, peers, properties, recent-block, trans,
  # utxo, votes, witness, witness_schedule.

  # Otherwise, db configs will remain default and data will be stored in
  # the path of "output-directory" or which is set by "-d" ("--output-directory").
  // setting can impove leveldb performance .... start
  // node: if this will increase process fds,you may be check your ulimit if 'too many open files' error occurs
  // see https://github.com/tronprotocol/tips/blob/master/tip-343.md for detail
  // if you find block sync has lower performance,you can try  this  settings
  default = {
   maxOpenFiles = 100
  }
  defaultM = {
    maxOpenFiles = 500
  }
  defaultL = {
    maxOpenFiles = 1000
  }
  // setting can impove leveldb performance .... end
  # Attention: name is a required field that must be set !!!
  properties = []

  needToUpdateAsset = true

  //dbsettings is needed when using rocksdb as the storage implement (db.version=2 and db.engine="ROCKSDB").
  //we'd strongly recommend that do not modify it unless you know every item's meaning clearly.
  dbSettings = {
    levelNumber = 7
    //compactThreads = 32
    blocksize = 64  // n * KB
    maxBytesForLevelBase = 256  // n * MB
    maxBytesForLevelMultiplier = 10
    level0FileNumCompactionTrigger = 4
    targetFileSizeBase = 256  // n * MB
    targetFileSizeMultiplier = 1
  }

  //backup settings when using rocks db as the storage implement (db.version=2 and db.engine="ROCKSDB").
  //if you want to use the backup plugin, please confirm set the db.version=2 and db.engine="ROCKSDB" above.
  backup = {
    enable = false  // indicate whether enable the backup plugin
    propPath = "prop.properties" // record which bak directory is valid
    bak1path = "bak1/database" // you must set two backup directories to prevent application halt unexpected(e.g. kill -9).
    bak2path = "bak2/database"
    frequency = 10000   // indicate backup db once every 10000 blocks processed.
  }

  checkpoint.version = 2
  checkpoint.sync = true
  # if true, transaction cache initialization will be faster. default false
  txCache.initOptimization = true

}

node.discovery = {
  enable = true
  persist = true
  bind.ip = ""
  external.ip = null
}

node.backup {
  port = 10001

  # my priority, each member should use different priority
  priority = 8

  # peer's ip list, can't contain mine
  members = [
    # "ip",
    # "ip"
  ]
}

node {
  # trust node for solidity node
  # trustNode = "ip:port"
  trustNode = "127.0.0.1:50051"

  # expose extension api to public or not
  walletExtensionApi = true

  listen.port = 18888

  connection.timeout = 2

  tcpNettyWorkThreadNum = 0

  udpNettyWorkThreadNum = 1

  # Number of validate sign thread, default availableProcessors / 2
  # validateSignThreadNum = 16

  connectFactor = 0.3
  activeConnectFactor = 0.1

  maxActiveNodes = 30

  maxActiveNodesWithSameIp = 2

  maxHttpConnectNumber = 50

  minParticipationRate = 15

  # check the peer data transfer ,disconnect factor
  disconnectNumberFactor = 0.4
  maxConnectNumberFactor = 0.8
  receiveTcpMinDataLength = 2048
  isOpenFullTcpDisconnect = true

  p2p {
    version = 11111 # 11111: mainnet; 20180622: testnet
  }

  active = [
    # Active establish connection in any case
    # Sample entries:
    # "ip:port",
    # "ip:port"
  ]

  passive = [
    # Passive accept connection in any case
    # Sample entries:
    # "ip:port",
    # "ip:port"
  ]

  fastForward = [
    "100.26.245.209:18888",
    "15.188.6.125:18888"
  ]

  http {
    fullNodePort = 8090
    solidityPort = 8091
    PBFTPort = 8092
  }

    jsonrpc {
    # Note: If you turn on jsonrpc and run it for a while and then turn it off, you will not
    # be able to get the data from eth_getLogs for that period of time.

    httpFullNodeEnable = true
    httpFullNodePort = 8545
    httpSolidityEnable = true
    httpSolidityPort = 8555
    httpPBFTEnable = true
    httpPBFTPort = 8565
  }


  rpc {
    port = 50051
    solidityPort = 50061
    PBFTPort = 50071

    maxConnectionIdleInMillis = 60000
    minEffectiveConnection = 1
  }


}

## rate limiter config
rate.limiter = {
  # Every api could be set a specific rate limit strategy. Three strategy are supported:GlobalPreemptibleAdapter、IPQPSRateLimiterAdapte、QpsRateLimiterAdapter
  # GlobalPreemptibleAdapter: permit is the number of preemptible resource, every client must apply one resourse
  #       before do the request and release the resource after got the reponse automaticlly. permit should be a Integer.
  # QpsRateLimiterAdapter: qps is the average request count in one second supported by the server, it could be a Double or a Integer.
  # IPQPSRateLimiterAdapter: similar to the QpsRateLimiterAdapter, qps could be a Double or a Integer.
  # If do not set, the "default strategy" is set.The "default startegy" is based on QpsRateLimiterAdapter, the qps is set as 10000.
  #
  # Sample entries:
  #
  http = [
    {
   component = "JsonRpcServlet",
   strategy = "QpsRateLimiterAdapter",
    paramString = "qps=1"
    },

    #  {
    #    component = "GetAccountServlet",
    #    strategy = "IPQPSRateLimiterAdapter",
    #    paramString = "qps=1"
    #  },

    #  {
    #    component = "ListWitnessesServlet",
    #    strategy = "QpsRateLimiterAdapter",
    #    paramString = "qps=1"
    #  }
  ],

  rpc = [
    #  {
    #    component = "protocol.Wallet/GetBlockByLatestNum2",
    #    strategy = "GlobalPreemptibleAdapter",
    #    paramString = "permit=1"
    #  },

    #  {
    #    component = "protocol.Wallet/GetAccount",
    #    strategy = "IPQPSRateLimiterAdapter",
    #    paramString = "qps=1"
    #  },

    #  {
    #    component = "protocol.Wallet/ListWitnesses",
    #    strategy = "QpsRateLimiterAdapter",
    #    paramString = "qps=1"
    #  },
  ]

}


seed.node = {
  # List of the seed nodes
  # Seed nodes are stable full nodes
  # example:
  # ip.list = [
  #   "ip:port",
  #   "ip:port"
  # ]
  ip.list = [
    "3.225.171.164:18888",
    "52.53.189.99:18888",
    "18.196.99.16:18888",
    "34.253.187.192:18888",
    "18.133.82.227:18888",
    "35.180.51.163:18888",
    "54.252.224.209:18888",
    "18.231.27.82:18888",
    "52.15.93.92:18888",
    "34.220.77.106:18888",
    "15.207.144.3:18888",
    "13.124.62.58:18888",
    "13.229.128.108:18888",
    "35.182.37.246:18888",
    "34.200.228.125:18888",
    "18.220.232.201:18888",
    "13.57.30.186:18888",
    "35.165.103.105:18888",
    "18.184.238.21:18888",
    "34.250.140.143:18888",
    "35.176.192.130:18888",
    "52.47.197.188:18888",
    "52.62.210.100:18888",
    "13.231.4.243:18888",
    "18.231.76.29:18888",
    "35.154.90.144:18888",
    "13.125.210.234:18888",
    "13.250.40.82:18888",
    "35.183.101.48:18888"
  ]
}

genesis.block = {
  # Reserve balance
  assets = [
    {
      accountName = "Zion"
      accountType = "AssetIssue"
      address = "TLLM21wteSPs4hKjbxgmH1L6poyMjeTbHm"
      balance = "99000000000000000"
    },
    {
      accountName = "Sun"
      accountType = "AssetIssue"
      address = "TXmVpin5vq5gdZsciyyjdZgKRUju4st1wM"
      balance = "0"
    },
    {
      accountName = "Blackhole"
      accountType = "AssetIssue"
      address = "TLsV52sRDL79HXGGm9yzwKibb6BeruhUzy"
      balance = "-9223372036854775808"
    }
  ]

  witnesses = [
    {
      address: THKJYuUmMKKARNf7s2VT51g5uPY6KEqnat,
      url = "http://GR1.com",
      voteCount = 100000026
    },
    {
      address: TVDmPWGYxgi5DNeW8hXrzrhY8Y6zgxPNg4,
      url = "http://GR2.com",
      voteCount = 100000025
    },
    {
      address: TWKZN1JJPFydd5rMgMCV5aZTSiwmoksSZv,
      url = "http://GR3.com",
      voteCount = 100000024
    },
    {
      address: TDarXEG2rAD57oa7JTK785Yb2Et32UzY32,
      url = "http://GR4.com",
      voteCount = 100000023
    },
    {
      address: TAmFfS4Tmm8yKeoqZN8x51ASwdQBdnVizt,
      url = "http://GR5.com",
      voteCount = 100000022
    },
    {
      address: TK6V5Pw2UWQWpySnZyCDZaAvu1y48oRgXN,
      url = "http://GR6.com",
      voteCount = 100000021
    },
    {
      address: TGqFJPFiEqdZx52ZR4QcKHz4Zr3QXA24VL,
      url = "http://GR7.com",
      voteCount = 100000020
    },
    {
      address: TC1ZCj9Ne3j5v3TLx5ZCDLD55MU9g3XqQW,
      url = "http://GR8.com",
      voteCount = 100000019
    },
    {
      address: TWm3id3mrQ42guf7c4oVpYExyTYnEGy3JL,
      url = "http://GR9.com",
      voteCount = 100000018
    },
    {
      address: TCvwc3FV3ssq2rD82rMmjhT4PVXYTsFcKV,
      url = "http://GR10.com",
      voteCount = 100000017
    },
    {
      address: TFuC2Qge4GxA2U9abKxk1pw3YZvGM5XRir,
      url = "http://GR11.com",
      voteCount = 100000016
    },
    {
      address: TNGoca1VHC6Y5Jd2B1VFpFEhizVk92Rz85,
      url = "http://GR12.com",
      voteCount = 100000015
    },
    {
      address: TLCjmH6SqGK8twZ9XrBDWpBbfyvEXihhNS,
      url = "http://GR13.com",
      voteCount = 100000014
    },
    {
      address: TEEzguTtCihbRPfjf1CvW8Euxz1kKuvtR9,
      url = "http://GR14.com",
      voteCount = 100000013
    },
    {
      address: TZHvwiw9cehbMxrtTbmAexm9oPo4eFFvLS,
      url = "http://GR15.com",
      voteCount = 100000012
    },
    {
      address: TGK6iAKgBmHeQyp5hn3imB71EDnFPkXiPR,
      url = "http://GR16.com",
      voteCount = 100000011
    },
    {
      address: TLaqfGrxZ3dykAFps7M2B4gETTX1yixPgN,
      url = "http://GR17.com",
      voteCount = 100000010
    },
    {
      address: TX3ZceVew6yLC5hWTXnjrUFtiFfUDGKGty,
      url = "http://GR18.com",
      voteCount = 100000009
    },
    {
      address: TYednHaV9zXpnPchSywVpnseQxY9Pxw4do,
      url = "http://GR19.com",
      voteCount = 100000008
    },
    {
      address: TCf5cqLffPccEY7hcsabiFnMfdipfyryvr,
      url = "http://GR20.com",
      voteCount = 100000007
    },
    {
      address: TAa14iLEKPAetX49mzaxZmH6saRxcX7dT5,
      url = "http://GR21.com",
      voteCount = 100000006
    },
    {
      address: TBYsHxDmFaRmfCF3jZNmgeJE8sDnTNKHbz,
      url = "http://GR22.com",
      voteCount = 100000005
    },
    {
      address: TEVAq8dmSQyTYK7uP1ZnZpa6MBVR83GsV6,
      url = "http://GR23.com",
      voteCount = 100000004
    },
    {
      address: TRKJzrZxN34YyB8aBqqPDt7g4fv6sieemz,
      url = "http://GR24.com",
      voteCount = 100000003
    },
    {
      address: TRMP6SKeFUt5NtMLzJv8kdpYuHRnEGjGfe,
      url = "http://GR25.com",
      voteCount = 100000002
    },
    {
      address: TDbNE1VajxjpgM5p7FyGNDASt3UVoFbiD3,
      url = "http://GR26.com",
      voteCount = 100000001
    },
    {
      address: TLTDZBcPoJ8tZ6TTEeEqEvwYFk2wgotSfD,
      url = "http://GR27.com",
      voteCount = 100000000
    }
  ]

  timestamp = "0" #2017-8-26 12:00:00

  parentHash = "0xe58f33f9baf9305dc6f82b9f1934ea8f0ade2defb951258d50167028c780351f"
}

// Optional.The default is empty.
// It is used when the witness account has set the witnessPermission.
// When it is not empty, the localWitnessAccountAddress represents the address of the witness account,
// and the localwitness is configured with the private key of the witnessPermissionAddress in the witness account.
// When it is empty,the localwitness is configured with the private key of the witness account.

//localWitnessAccountAddress =

localwitness = [
]

#localwitnesskeystore = [
#  "localwitnesskeystore.json"
#]

block = {
  needSyncCheck = true
  maintenanceTimeInterval = 21600000
  proposalExpireTime = 259200000 // 3 day: 259200000(ms)
}

# Transaction reference block, default is "head", configure to "solid" can avoid TaPos error
trx.reference.block = "solid" // head;solid;

# This property sets the number of milliseconds after the creation of the transaction that is expired, default value is  60000.
# trx.expiration.timeInMilliseconds = 60000

vm = {
  supportConstant = true
  minTimeRatio = 0.0
  maxTimeRatio = 5.0
  saveInternalTx = false

  # In rare cases, transactions that will be within the specified maximum execution time (default 10(ms)) are re-executed and packaged
  # longRunningTime = 10
}

committee = {
  allowCreationOfContracts = 0  //mainnet:0 (reset by committee),test:1
  allowAdaptiveEnergy = 0  //mainnet:0 (reset by committee),test:1
#  allowOldRewardOpt = 1
}

event.subscribe = {
    native = {
      useNativeQueue = true // if true, use native message queue, else use event plugin.
      bindport = 5555 // bind port
      sendqueuelength = 1000 //max length of send queue
    }

    path = "" // absolute path of plugin
    server = "" // target server address to receive event triggers
    dbconfig="" // dbname|username|password
    contractParse = true,
    topics = [
        {
          triggerName = "block" // block trigger, the value can't be modified
          enable = false
          topic = "block" // plugin topic, the value could be modified
        },
        {
          triggerName = "transaction"
          enable = false
          topic = "transaction"
        },
        {
          triggerName = "contractevent"
          enable = false
          topic = "contractevent"
        },
        {
          triggerName = "contractlog"
          enable = false
          topic = "contractlog"
        },
        {
          triggerName = "solidity" // solidity block event trigger, the value can't be modified
          enable = true            // the default value is true
          topic = "solidity"
        },
        {
          triggerName = "solidityevent"
          enable = false
          topic = "solidityevent"
        },
        {
          triggerName = "soliditylog"
          enable = false
          topic = "soliditylog"
        }
    ]

    filter = {
       fromblock = "" // the value could be "", "earliest" or a specified block number as the beginning of the queried range
       toblock = "" // the value could be "", "latest" or a specified block number as end of the queried range
       contractAddress = [
           "" // contract address you want to subscribe, if it's set to "", you will receive contract logs/events with any contract address.
       ]

       contractTopic = [
           "" // contract topic you want to subscribe, if it's set to "", you will receive contract logs/events with any contract topic.
       ]
    }
}

[lurais]

Based on the configuration file, it appears that the issue may be caused by using checkpoint v2 without querying from it in the code. However, I have noticed that the pull request (o85372940#8) may fix the issue. You could merge it and check again to see if it resolves the problem.

[xxo1shine]

How to fix this bug? Is there any good solution?

[lurais]

How to fix this bug? Is there any good solution?

The pull request available at o85372940#8 can be merged to see if it fixes the issue.

[halibobo1205]

close with #5658.