IF: Test failure: trx_finality_status_forked_if_test

[linh2931]

https://github.com/AntelopeIO/leap/actions/runs/8270274635/job/22627758990?pr=2282#step:4:1738

[heifner]

The logs for this look interesting. Need to dig into why monotony check failed and why it spams the log with
Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)

[heifner]

Another failure: https://github.com/AntelopeIO/leap/actions/runs/8393071593/job/22987735147
https://github.com/AntelopeIO/leap/actions/runs/8451731380/job/23151071005
https://github.com/AntelopeIO/leap/actions/runs/8455646427/job/23164114731
https://github.com/AntelopeIO/leap/actions/runs/8467614664/job/23199261281

[linh2931]

https://github.com/AntelopeIO/leap/actions/runs/8473490901/job/23218304471?pr=2349#step:4:625

[greg7mdp]

The log doesn't show a problem with IF I can seem The monotony check failures in node_03 are normals, as after the bridge is restored, we receive blocks from the other side which are earlier than the last block we voted on.

However, the log show an issue (unrelated to IF work) where production is rescheduled too early after an error, causing repetitive log messages until the next block time is hit:

error 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:1853      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2515      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2589      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-03-13T19:23:57.200
error 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:1853      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2515      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2589      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-03-13T19:23:57.200
error 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:1853      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2515      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2589      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-03-13T19:23:57.200
error 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:1853      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2515      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2589      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-03-13T19:23:57.200
error 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:1853      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2515      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2589      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-03-13T19:23:57.200
error 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:1853      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2515      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2589      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-03-13T19:23:57.200
error 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:1853      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2515      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2589      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-03-13T19:23:57.200
error 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:1853      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2515      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2589      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-03-13T19:23:57.200
error 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:1853      start_block          ] Not producing block because "defproducerd" signed a block at the next block time or later (2024-03-13T19:23:57.500) than the pending block time (2024-03-13T19:23:57.500)
debug 2024-03-13T19:23:57.378 nodeos    producer_plugin.cpp:2515      schedule_production_ ] Waiting till another block is received and scheduling Speculative/Production Change
debug 2024-03-13T19:23:57.379 nodeos    producer_plugin.cpp:2589      schedule_delayed_pro ] Scheduling Speculative/Production Change at 2024-03-13T19:23:57.200
...

Issue #2380 was created to address this problem.