Nodeos instability issue during re-start under FreeBSD

[vladtr]

With a full understanding that FreeBSD is not officially supported but feeling that fixing this issue may improve general stability of nodeos on supported platforms as well. This issue is not reproducible on Ubuntu/Debian.

System:
FreeBSD bsd 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64

Steps to reproduce:

1. clean-up any leftovers of ~/.local/share/eosio
2. start producing node:
   ./bin/nodeos -e -p eosio --plugin eosio::producer_plugin --plugin eosio::producer_api_plugin
3. wait for several blocks
4. terminate it with ctrl+c
5. re-start node with the same command
   ./bin/nodeos -e -p eosio --plugin eosio::producer_plugin --plugin eosio::producer_api_plugin

node will start producing and after several blocks will start to error out:

`
./bin/nodeos -e -p eosio --plugin eosio::producer_plugin --plugin eosio::producer_api_plugin
info 2023-03-14T17:38:15.205 thread-0 chain_plugin.cpp:470 plugin_initialize ] initializing chain plugin
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:835 operator() ] Support for builtin protocol feature 'PREACTIVATE_FEATURE' (with digest of '0ec7e080177b2c02b278d5088611686b49d739925a92d9bfcacd7fc6b74053bd') is enabled without activation restrictions
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'ONLY_LINK_TO_EXISTING_PERMISSION' (with digest of '1a99a59d87e06e09ec5b028a9cbb7749b4a5ad8819004365d02dc4379a8b7241') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'FORWARD_SETCODE' (with digest of '2652f5f96006294109b3dd0bbde63693f55324af452b799ee137a81a905eed25') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'WTMSIG_BLOCK_SIGNATURES' (with digest of '299dcb6af692324b899b39f16d5a530a33062804e41f09dc97e9f156b4476707') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'GET_BLOCK_NUM' (with digest of '35c2186cc36f7bb4aeaf4487b36e57039ccf45a9136aa856a5d569ecca55ef2b') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'REPLACE_DEFERRED' (with digest of 'ef43112c6543b88db2283a2e077278c315ae2c84719a8b25f25cc88565fbea99') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'NO_DUPLICATE_DEFERRED_ID' (with digest of '4a90c00d55454dc5b059055ca213579c6ea856967712a56017487886a4d4cc0f') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'RAM_RESTRICTIONS' (with digest of '4e7bf348da00a945489b2a681749eb56f5de00b900014e137ddae39f48f69d67') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'WEBAUTHN_KEY' (with digest of '4fca8bd82bbd181e714e283f83e1b45d95ca5af40fb89ad3977b653c448f78c2') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'BLOCKCHAIN_PARAMETERS' (with digest of '5443fcf88330c586bc0e5f3dee10e7f63c76c00249c87fe4fbf7f38c082006b4') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'DISALLOW_EMPTY_PRODUCER_SCHEDULE' (with digest of '68dcaa34c0517d19666e6b33add67351d8c5f69e999ca1e37931bc410a297428') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'CRYPTO_PRIMITIVES' (with digest of '6bcb40a24e49c26d0a60513b6aeb8551d264e4717f306b81a37a5afb3b47cedc') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'ONLY_BILL_FIRST_AUTHORIZER' (with digest of '8ba52fe7a3956c5cd3a656a3174b931d3bb2abb45578befc59f283ecd816a405') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'RESTRICT_ACTION_TO_SELF' (with digest of 'ad9e3d8f650687709fd68f4b90b41f7d825a365b02c23a636cef88ac2ac00c43') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'GET_CODE_HASH' (with digest of 'bcd2a26394b36614fd4894241d3c451ab0f6fd110958c3423073621a70826e99') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'ACTION_RETURN_VALUE' (with digest of 'c3a6138c5061cf291310887c0b5c71fcaffeab90d5deb50d3b9e687cead45071') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'CONFIGURABLE_WASM_LIMITS2' (with digest of 'd528b9f6e9693f45ed277af93474fd473ce7d831dae2180cca35d907bd10cb40') is enabled with preactivation required
info 2023-03-14T17:38:15.207 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'FIX_LINKAUTH_RESTRICTION' (with digest of 'e0fb64b1085cc5538970158d05a009c24e276fb94e1a0bf6a528b48fbc4ff526') is enabled with preactivation required
info 2023-03-14T17:38:15.208 thread-0 protocol_feature_manag:822 operator() ] Support for builtin protocol feature 'GET_SENDER' (with digest of 'f0af56d2c5a48d60a4a5b5c903edfb7db3a736a94ed589d0b797df33ff9d3e1d') is enabled with preactivation required
info 2023-03-14T17:38:15.209 thread-0 block_log.cpp:664 open ] Log has 22 blocks
info 2023-03-14T17:38:28.711 thread-0 platform_timer_accurac:62 compute_and_print_ti ] Checktime timer accuracy: min:446us max:38027us mean:1362us stddev:675us
warn 2023-03-14T17:38:28.711 thread-0 platform_timer_accurac:64 compute_and_print_ti ] Checktime timer accuracy on this platform and hardware combination is poor; accuracy of subjective transaction deadline enforcement will suffer
info 2023-03-14T17:38:28.713 thread-0 producer_plugin.cpp:943 plugin_initialize ] Subjective CPU billing disabled
info 2023-03-14T17:38:28.713 thread-0 http_plugin.cpp:250 plugin_initialize ] configured http to listen on 127.0.0.1:8888
info 2023-03-14T17:38:28.713 thread-0 net_plugin.cpp:3616 plugin_initialize ] Initialize net plugin
info 2023-03-14T17:38:28.714 thread-0 resource_monitor_plugi:73 plugin_initialize ] Monitoring interval set to 2
info 2023-03-14T17:38:28.714 thread-0 resource_monitor_plugi:89 plugin_initialize ] Space usage threshold set to 90%
info 2023-03-14T17:38:28.714 thread-0 resource_monitor_plugi:99 plugin_initialize ] Shutdown flag when threshold exceeded set to true
info 2023-03-14T17:38:28.714 thread-0 resource_monitor_plugi:106 plugin_initialize ] Warning interval set to 30
info 2023-03-14T17:38:28.714 thread-0 main.cpp:142 main ] nodeos version v4.0.0-dev v4.0.0-dev-041d897df06978391b1ee7025bb6c5b75758594d
info 2023-03-14T17:38:28.714 thread-0 main.cpp:143 main ] nodeos using configuration file /usr/home/valter/.local/share/eosio/nodeos/config/config.ini
info 2023-03-14T17:38:28.714 thread-0 main.cpp:144 main ] nodeos data directory is /home/valter/.local/share/eosio/nodeos/data
info 2023-03-14T17:38:28.714 thread-0 controller.cpp:540 replay ] no irreversible blocks need to be replayed
info 2023-03-14T17:38:28.714 thread-0 controller.cpp:553 replay ] 0 reversible blocks replayed
info 2023-03-14T17:38:28.714 thread-0 controller.cpp:563 replay ] replayed 0 blocks in 0 seconds, 0.00000000001979060 ms/block
info 2023-03-14T17:38:28.714 thread-0 chain_plugin.cpp:1115 plugin_startup ] starting chain in read/write mode
info 2023-03-14T17:38:28.714 thread-0 chain_plugin.cpp:1123 plugin_startup ] Blockchain started; head block is #23
info 2023-03-14T17:38:28.714 thread-0 producer_plugin.cpp:1014 plugin_startup ] producer plugin: plugin_startup() begin
info 2023-03-14T17:38:28.715 thread-0 producer_plugin.cpp:1045 plugin_startup ] Launching block production for 1 producers at 2023-03-14T17:38:28.715.
info 2023-03-14T17:38:28.715 thread-0 producer_plugin.cpp:1056 plugin_startup ] producer plugin: plugin_startup() end
info 2023-03-14T17:38:28.715 thread-0 producer_api_plugin.cp:97 plugin_startup ] starting producer_api_plugin
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/add_greylist_accounts
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/create_snapshot
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/get_account_ram_corrections
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/get_greylist
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/get_integrity_hash
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/get_runtime_options
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/get_scheduled_protocol_feature_activations
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/get_snapshot_requests
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/get_supported_protocol_features
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/get_unapplied_transactions
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/get_whitelist_blacklist
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/pause
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/paused
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/remove_greylist_accounts
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/resume
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/schedule_protocol_feature_activations
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/schedule_snapshot
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/set_whitelist_blacklist
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/unschedule_snapshot
info 2023-03-14T17:38:28.716 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/producer/update_runtime_options
info 2023-03-14T17:38:28.716 thread-0 net_plugin.cpp:3721 plugin_startup ] my node_id is 6356253dced2a55cd87dbe7210f023f207777a1154213d8652a90f2d93076448
info 2023-03-14T17:38:28.717 thread-0 resource_monitor_plugi:115 plugin_startup ] Creating and starting monitor thread
info 2023-03-14T17:38:28.717 thread-0 file_space_handler.hpp:96 add_file_system ] /home/valter/.local/share/eosio/nodeos/data/blocks's file system to be monitored
info 2023-03-14T17:38:28.717 thread-0 file_space_handler.hpp:130 add_file_system ] /home/valter/.local/share/eosio/nodeos/data/blocks's file system monitored. shutdown_available: 61 GiB, capacity: 612 GiB, threshold: 90%
info 2023-03-14T17:38:28.717 thread-0 file_space_handler.hpp:96 add_file_system ] /home/valter/.local/share/eosio/nodeos/data/blocks/reversible's file system to be monitored
info 2023-03-14T17:38:28.717 thread-0 file_space_handler.hpp:96 add_file_system ] /home/valter/.local/share/eosio/nodeos/data/state's file system to be monitored
info 2023-03-14T17:38:28.717 thread-0 file_space_handler.hpp:96 add_file_system ] /home/valter/.local/share/eosio/nodeos/data/snapshots's file system to be monitored
info 2023-03-14T17:38:28.717 thread-0 file_space_handler.hpp:96 add_file_system ] /home/valter/.local/share/eosio/nodeos/data's file system to be monitored
info 2023-03-14T17:38:28.717 thread-0 file_space_handler.hpp:96 add_file_system ] /home/valter/.local/share/eosio/nodeos/data/state's file system to be monitored
info 2023-03-14T17:38:28.717 thread-0 file_space_handler.hpp:96 add_file_system ] /home/valter/.local/share/eosio/nodeos/data/snapshots's file system to be monitored
info 2023-03-14T17:38:28.717 thread-0 file_space_handler.hpp:96 add_file_system ] /home/valter/.local/share/eosio/nodeos/data/blocks's file system to be monitored
info 2023-03-14T17:38:28.717 thread-0 net_plugin.cpp:3799 operator() ] starting listener, max clients is 25
info 2023-03-14T17:38:28.718 thread-0 http_plugin.cpp:132 create_beast_server ] created beast HTTP listener
info 2023-03-14T17:38:28.718 thread-0 http_plugin.cpp:290 operator() ] start listening for http requests (boost::beast)
info 2023-03-14T17:38:28.718 thread-0 beast_http_listener.hp:81 listen ] acceptor_.listen()
info 2023-03-14T17:38:28.718 thread-0 http_plugin.cpp:366 add_handler ] add api url: /v1/node/get_supported_apis
info 2023-03-14T17:38:28.913 thread-0 producer_plugin.cpp:2550 produce_block ] Produced block 0bb68f4babd85bf2... #24 @ 2023-03-14T17:38:29.000 signed by eosio [trxs: 0, lib: 23, confirmed: 0, net: 0, cpu: 100, elapsed: 216, time: 1243]
info 2023-03-14T17:38:29.308 thread-0 producer_plugin.cpp:2550 produce_block ] Produced block f0b8db9fd9dc9ec8... #25 @ 2023-03-14T17:38:29.500 signed by eosio [trxs: 0, lib: 24, confirmed: 0, net: 0, cpu: 100, elapsed: 120, time: 1139]
info 2023-03-14T17:38:29.925 thread-0 producer_plugin.cpp:2550 produce_block ] Produced block cc770a2bfe16902a... #26 @ 2023-03-14T17:38:30.000 signed by eosio [trxs: 0, lib: 25, confirmed: 0, net: 0, cpu: 100, elapsed: 182, time: 1140]
info 2023-03-14T17:38:30.426 thread-0 producer_plugin.cpp:2550 produce_block ] Produced block b387c00ce31f3836... #27 @ 2023-03-14T17:38:30.500 signed by eosio [trxs: 0, lib: 26, confirmed: 0, net: 0, cpu: 100, elapsed: 121, time: 804]
info 2023-03-14T17:38:30.925 thread-0 producer_plugin.cpp:2550 produce_block ] Produced block e188164a02f23bb6... #28 @ 2023-03-14T17:38:31.000 signed by eosio [trxs: 0, lib: 27, confirmed: 0, net: 0, cpu: 100, elapsed: 117, time: 828]
info 2023-03-14T17:38:31.426 thread-0 producer_plugin.cpp:2550 produce_block ] Produced block 9a0662ce1f94cf43... #29 @ 2023-03-14T17:38:31.500 signed by eosio [trxs: 0, lib: 28, confirmed: 0, net: 0, cpu: 100, elapsed: 119, time: 920]
info 2023-03-14T17:38:31.926 thread-0 producer_plugin.cpp:2550 produce_block ] Produced block 40eef65f5f1f9bd6... #30 @ 2023-03-14T17:38:32.000 signed by eosio [trxs: 0, lib: 29, confirmed: 0, net: 0, cpu: 100, elapsed: 131, time: 1257]
info 2023-03-14T17:38:32.425 thread-0 producer_plugin.cpp:2550 produce_block ] Produced block c32d76f69574d1b9... #31 @ 2023-03-14T17:38:32.500 signed by eosio [trxs: 0, lib: 30, confirmed: 0, net: 0, cpu: 100, elapsed: 237, time: 1046]
warn 2023-03-14T17:38:32.506 thread-0 controller.cpp:1816 start_block ] on block transaction failed, but shouldn't impact block generation, system contract needs update
error 2023-03-14T17:38:32.506 thread-0 controller.cpp:1817 start_block ] e.to_detail_string(): 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
boost::interprocess::lock_exception:
{"trace":null,"what":"boost::interprocess::lock_exception"}
thread-0 controller.cpp:1660 push_transaction

info 2023-03-14T17:38:32.905 thread-0 producer_plugin.cpp:2550 produce_block ] Produced block 02c0a80844b20268... #32 @ 2023-03-14T17:38:33.000 signed by eosio [trxs: 0, lib: 31, confirmed: 0, net: 0, cpu: 0, elapsed: 0, time: 552]
warn 2023-03-14T17:38:33.007 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

error 2023-03-14T17:38:33.007 thread-0 producer_plugin.cpp:2355 schedule_production_ ] Failed to start a pending block, will try again later
warn 2023-03-14T17:38:33.061 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

error 2023-03-14T17:38:33.061 thread-0 producer_plugin.cpp:2355 schedule_production_ ] Failed to start a pending block, will try again later
warn 2023-03-14T17:38:33.115 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

error 2023-03-14T17:38:33.115 thread-0 producer_plugin.cpp:2355 schedule_production_ ] Failed to start a pending block, will try again later
warn 2023-03-14T17:38:33.169 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

error 2023-03-14T17:38:33.170 thread-0 producer_plugin.cpp:2355 schedule_production_ ] Failed to start a pending block, will try again later
warn 2023-03-14T17:38:33.224 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

error 2023-03-14T17:38:33.224 thread-0 producer_plugin.cpp:2355 schedule_production_ ] Failed to start a pending block, will try again later
warn 2023-03-14T17:38:33.278 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

error 2023-03-14T17:38:33.278 thread-0 producer_plugin.cpp:2355 schedule_production_ ] Failed to start a pending block, will try again later
warn 2023-03-14T17:38:33.332 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

error 2023-03-14T17:38:33.332 thread-0 producer_plugin.cpp:2355 schedule_production_ ] Failed to start a pending block, will try again later
warn 2023-03-14T17:38:33.385 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

error 2023-03-14T17:38:33.385 thread-0 producer_plugin.cpp:2355 schedule_production_ ] Failed to start a pending block, will try again later
warn 2023-03-14T17:38:33.440 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

error 2023-03-14T17:38:33.440 thread-0 producer_plugin.cpp:2355 schedule_production_ ] Failed to start a pending block, will try again later
warn 2023-03-14T17:38:33.494 thread-0 producer_plugin.cpp:1787 start_block ] 13 N5boost12interprocess14lock_exceptionE: boost::interprocess::lock_exception
std::exception: boost::interprocess::lock_exception:
{"what":"boost::interprocess::lock_exception"}
thread-0 producer_plugin.cpp:1787 start_block

`

[stephenpdeos]

While this is only being expressed on FreeBSD, we'd like to prioritize further investigation to verify there aren't underlying issues that impact what we do support