Ingestion engine does not handle EN falling far behind

[j1010001]

The problem:
When Execution node falls far behind the latest block (for example when it is stopped for a longer period of time) it can fail to start and enters a crash loop, which causes extra load on the network. The root cause is the ingestion engine, which attempts to request collections for all unexecuted blocks. This loads the collection nodes with many requests, and creates high traffic on the network. The EN then fails to fit all the collections into memory, crashes and this cycle repeats.

Goal: Refactor ingestion engine to handle this edge case.

[zhangchiqing]

Two Approaches for implemention

Goal: implement a stateful ingestion module that keeps track of the next X number of blocks to get ready for execution.

Requirements:

• Blocks can be on different forks, especially when execution is ahead of finalization, we need to make sure all valid blocks are ingested.
• When execution is far behind finalization, it should not ingest all finalized blocks, but only the next X number of unexecuted blocks, otherwise it might risk hitting OOM.
• Should not be blocking the BlockExecuted event, otherwise it would slow down the block execution.
• Should not be blocking the BlockProcessable event, otherwise it would slow down consensus algorithm fetching and processing blocks.

Challenges:

• The ingestion module needs to handle both block executed events and block processable events, and check if they would trigger more blocks to be loaded. However, the two events happen concurrently.
• The ingestion module must guarantee the parent block is always ingested before its child block.
• The ingestion module must guarantee no block is missing to be ingested, even with restart.

Approach 1: In Memory state machine

• Subscribing the BlockProcessed and BlockExecuted event
• keep track of highest executed height, ingest all blocks below the height (highest_executed_height + 500), (say we ingest blocks up to 500 heights)

Approach 2: Build a BlockProcessable history on disk

• Build a history of block Procesable history as jobs and save it to disk.
• The ingestion module could create a single worker load 500 jobs at most from the job queue.

[zhangchiqing]

Related to #5161