Spring 1.1 Misc Errors During Replay

[ericpassmore]

Getting misc errors during replay test. Found in

• run 20
  • job 140547858842528
  • start block 16150309
  • end block 17323276
• run 54
  • job 140547858857456
  • start block 46911265
  • end block 48116875

nodeos140547858857456.log
nodeos140547858842528.log

error 2025-02-05T14:21:39.934 nodeos    controller.cpp:3883           apply_block          ] e.to_detail_string(): 3100000 misc_exception: Miscellaneous exception
Setting a platform_timer callback when one already exists
    {}
    nodeos  platform_timer.hpp:36 set_expiration_callback
eosluckydice <= eosio::onerror pending console output: 
    {"console":"","account":"eosio","action":"onerror","receiver":"eosluckydice"}
    nodeos  apply_context.cpp:134 exec_one
 
warn  2025-02-05T14:21:39.934 nodeos    controller.cpp:4361           replay_irreversible_ ] 3100000 misc_exception: Miscellaneous exception
Setting a platform_timer callback when one already exists
    {}
    nodeos  platform_timer.hpp:36 set_expiration_callback
eosluckydice <= eosio::onerror pending console output: 
    {"console":"","account":"eosio","action":"onerror","receiver":"eosluckydice"}
    nodeos  apply_context.cpp:134 exec_one

    {}
    nodeos  controller.cpp:3891 apply_block

info  2025-02-05T14:21:39.934 nodeos    main.cpp:165                  operator()           ] nodeos version v1.1.0-rc1 v1.1.0-rc1-dcb9cc9bec4c175d38a03efedf2b50431ea35c71

[ericpassmore]

Another Misc error

• Run 78
  • Start Block: 68664955
  • End Block: 69871994

[ericpassmore]

Last commit hash that worked

info  2025-02-03T22:02:19.525 nodeos    main.cpp:165                  operator()           ] nodeos version v1.1.0-dev v1.1.0-dev-4a5095aa3efd7518d7f669622a0b8884b5c3f3f6
info  2025-02-03T22:02:19.525 nodeos    main.cpp:69                   log_non_default_opti ] Non-default options: snapshot = /data/nodeos/snapshot/snapshot.bin, data-dir = /data/nodeos/data/, eos-vm-oc-enable = off, wasm-runtime = eos-vm-oc-forced, database-map-mode = mapped_private, p2p-listen-endpoint = 127.0.0.1:9876, p2p-max-nodes-per-host = 10, state-history-endpoint = 127.0.0.1:8080, verbose-http-errors, agent-name = ReplayTestNode, allowed-connection = any, net-threads = 4, max-transaction-time = 100, max-clients = 1024, resource-monitor-not-shutdown-on-threshold-exceeded, read-only-read-window-time-us = 150000, read-only-write-window-time-us = 50000, plugin = eosio::chain_api_plugin, plugin = eosio::chain_plugin, plugin = eosio::http_plugin, plugin = eosio::producer_plugin, plugin = eosio::producer_api_plugin, plugin = eosio::net_plugin, plugin = eosio::net_api_plugin, p2p-peer-address = eos.seed.eosnation.io:9876, p2p-peer-address = peer1.eosphere.io:9876, p2p-peer-address = eos.seed.eosnation.io:9876, terminate-at-block = 16150309, integrity-hash-on-start, integrity-hash-on-stop
error 2025-02-03T22:02:19.525 nodeos    main.cpp:229                  main                 ] 3060101 database_guard_exception: Database usage is at unsafe levels
database free: 86948464, guard size: 134217728

[ericpassmore]

Reran with revert and with v1.1.0-rc1

revert-1141-macos-eosvm-ro-overlap

• job 140543690509616
• v1.2.0-dev v1.2.0-dev-94b6a00550e683cb97cf13fcbff0a34ce8d42790 Misc Error
• Blocks Start Block: 17323276 End Block: 17977332

release/1.1

• job 140543690510000
• v1.1.0-rc1 v1.1.0-rc1-dcb9cc9bec4c175d38a03efedf2b50431ea35c71 Misc Error
• Blocks Start Block: 16150309 End Block: 17323276

Non-default options: snapshot = /data/nodeos/snapshot/snapshot.bin, data-dir = /data/nodeos/data/, config = /home/enf-replay/replay-test/config/sync-config.ini, terminate-at-block = 17977332, integrity-hash-on-start, integrity-hash-on-stop, database-map-mode = mapped_private, eos-vm-oc-enable = on, chain-state-db-size-mb = 131072, contracts-console = true, http-server-address = 127.0.0.1:8888, access-control-allow-credentials = false, p2p-listen-endpoint = 127.0.0.1:9876, p2p-max-nodes-per-host = 10, state-history-endpoint = 127.0.0.1:8080, verbose-http-errors = true, agent-name = "Replay Test Node", allowed-connection = any, net-threads = 4, max-transaction-time = 100, max-clients = 1024, sync-fetch-span = 1024, enable-stale-production = false, resource-monitor-not-shutdown-on-threshold-exceeded = true, read-only-read-window-time-us = 150000, read-only-write-window-time-us = 50000, blocks-log-stride = 2000000, max-retained-block-files = 512, blocks-retained-dir = retained, plugin = eosio::chain_api_plugin, plugin = eosio::chain_plugin, plugin = eosio::http_plugin, plugin = eosio::producer_plugin, plugin = eosio::producer_api_plugin, plugin = eosio::net_plugin, plugin = eosio::net_api_plugin, p2p-peer-address = eos.seed.eosnation.io:9876, p2p-peer-address = peer1.eosphere.io:9876, p2p-peer-address = peer2.eosphere.io:9876, p2p-peer-address = p2p.genereos.io:9876

[spoonincode]

Last commit hash that worked

info  2025-02-03T22:02:19.525 nodeos    main.cpp:165                  operator()           ] nodeos version v1.1.0-dev v1.1.0-dev-4a5095aa3efd7518d7f669622a0b8884b5c3f3f6
info  2025-02-03T22:02:19.525 nodeos    main.cpp:69                   log_non_default_opti ] Non-default options: snapshot = /data/nodeos/snapshot/snapshot.bin, data-dir = /data/nodeos/data/, eos-vm-oc-enable = off, wasm-runtime = eos-vm-oc-forced, database-map-mode = mapped_private, p2p-listen-endpoint = 127.0.0.1:9876, p2p-max-nodes-per-host = 10, state-history-endpoint = 127.0.0.1:8080, verbose-http-errors, agent-name = ReplayTestNode, allowed-connection = any, net-threads = 4, max-transaction-time = 100, max-clients = 1024, resource-monitor-not-shutdown-on-threshold-exceeded, read-only-read-window-time-us = 150000, read-only-write-window-time-us = 50000, plugin = eosio::chain_api_plugin, plugin = eosio::chain_plugin, plugin = eosio::http_plugin, plugin = eosio::producer_plugin, plugin = eosio::producer_api_plugin, plugin = eosio::net_plugin, plugin = eosio::net_api_plugin, p2p-peer-address = eos.seed.eosnation.io:9876, p2p-peer-address = peer1.eosphere.io:9876, p2p-peer-address = eos.seed.eosnation.io:9876, terminate-at-block = 16150309, integrity-hash-on-start, integrity-hash-on-stop
error 2025-02-03T22:02:19.525 nodeos    main.cpp:229                  main                 ] 3060101 database_guard_exception: Database usage is at unsafe levels
database free: 86948464, guard size: 134217728

The log you quoted is an error though

[heifner]

Another error in chicken dance runs:

info  2025-02-06T15:34:47.445 nodeos    controller.cpp:1746           replay_block_log     ] 16361500 of 18000000
info  2025-02-06T15:34:47.489 oc-monito wasm_interface_private:125    operator()           ] EOS VM OC tier up interrupting ca70c56b576f778a3d3646a4e746feb7b9179457d5e196022a8ae871b1099d5b
error 2025-02-06T15:34:47.489 nodeos    controller.cpp:3881           apply_block          ] e.to_detail_string(): 3030000 block_validate_exception: Block exception
receipt does not match, {"status":"soft_fail","cpu_usage_us":3731,"net_usage_words":0} != {"status":"executed","cpu_usage_us":3731,"net_usage_words":0}
    {"lhs":{"status":"soft_fail","cpu_usage_us":3731,"net_usage_words":0},"rhs":{"status":"executed","cpu_usage_us":3731,"net_usage_words":0}}
    nodeos  controller.cpp:3824 apply_block
 
warn  2025-02-06T15:34:47.490 nodeos    controller.cpp:4359           replay_irreversible_ ] 3030000 block_validate_exception: Block exception
receipt does not match, {"status":"soft_fail","cpu_usage_us":3731,"net_usage_words":0} != {"status":"executed","cpu_usage_us":3731,"net_usage_words":0}
    {"lhs":{"status":"soft_fail","cpu_usage_us":3731,"net_usage_words":0},"rhs":{"status":"executed","cpu_usage_us":3731,"net_usage_words":0}}
    nodeos  controller.cpp:3824 apply_block

    {}
    nodeos  controller.cpp:3889 apply_block

info  2025-02-06T15:34:47.490 nodeos    main.cpp:165                  operator()           ] nodeos version v1.1.0-dev v1.1.0-dev-389741e40af4bb97f5520bf23d1656b291a9992e
info  2025-02-06T15:34:47.490 nodeos    main.cpp:69                   log_non_default_opti ] Non-default options: snapshot = /data/nodeos/snapshot/snapshot.bin, data-dir = /data/nodeos/data/, config = /home/enf-replay/replay-test/config/default-config.ini, terminate-at-block = 17323276, integrity-hash-on-start, integrity-hash-on-stop, database-map-mode = mapped_private, chain-state-db-size-mb = 131072, eos-vm-oc-enable = on, plugin = eosio::chain_plugin, plugin = eosio::http_plugin, plugin = eosio::net_plugin, p2p-peer-address = eos.seed.eosnation.io:9876, p2p-peer-address = peer1.eosphere.io:9876, p2p-peer-address = peer2.eosphere.io:9876, p2p-peer-address = p2p.genereos.io:9876, access-control-allow-credentials = false, allowed-connection = any, max-clients = 1024, resource-monitor-not-shutdown-on-threshold-exceeded = true
info  2025-02-06T15:34:52.009 nodeos    snapshot.hpp:451              progress             ] Snapshot creation 94% complete
info  2025-02-06T15:34:52.109 nodeos    controller.cpp:2102           ~controller_impl     ] chain database stopped with hash: 2c09e37e90e1b257f6713cb06794f05f81d0b05922a2ea82df6de269331c21ca
info  2025-02-06T15:34:52.123 nodeos    block_handle.cpp:21           write                ] Writing chain_head block 16361576 00f9a868aa6e72c9fff6968c66357acfcebe26ce7393bce3cee411df844f981e
info  2025-02-06T15:34:52.123 nodeos    fork_database.cpp:662         close                ] No fork_database to persist
CHAINBASE: Writing "state" database file, this could take a moment...
CHAINBASE: Writing "state" database file, 0% complete...
CHAINBASE: Writing "state" database file, complete.
error 2025-02-06T15:34:56.079 nodeos    main.cpp:229                  main                 ] 3030000 block_validate_exception: Block exception
receipt does not match, {"status":"soft_fail","cpu_usage_us":3731,"net_usage_words":0} != {"status":"executed","cpu_usage_us":3731,"net_usage_words":0}
    {"lhs":{"status":"soft_fail","cpu_usage_us":3731,"net_usage_words":0},"rhs":{"status":"executed","cpu_usage_us":3731,"net_usage_words":0}}
    nodeos  controller.cpp:3824 apply_block

    {}
    nodeos  controller.cpp:3889 apply_block
rethrow
    {}
    nodeos  controller.cpp:4359 replay_irreversible_block

    {}
    nodeos  chain_plugin.cpp:1167 plugin_startup