Add support to backfill actor events + resuming when left off

[fridrik01]

Related: #10807

Context

When lotus is configured to enable indexing of actor events (when EnableEthRPC=true and DisableHistoricFilterAPI=false) then Lotus will observe for new tipset and store all the actor events in a sqlite table stored at sqlite/events.db. We however, don't have any way to backfill this table which is what this PR is about

I tried first to implement this backfilling using lotus-shed (as we have for backfilling msgindex and eth txhahses) but I had some issues getting the required dependencies to work there, so I ended up integrating this into the lotus node directy.

This backfill is guarded behind a new lotus config EnableEventBackfillOnStartup which now defaults to false. When set to true, then Lotus will start a background go routine which walks up the chain to collect the actor events. It then checks the sqlite database if that actor event are missing from the database and inserts them if that is the case.

To gracefully handle restarts when backfilling, we mark the current process in a new table event_backfilled so we will resume from where we left of if the backfill process was interrupted for any reason.

Test plan

Backfill active node that is already storing events

As expected, backfilling should not have any side effects on a node that is already saving actor events.

2023-06-23T09:21:05.444Z        INFO    modules modules/actorevent.go:89        EnableEventBackfillOnStartup was set, starting backfill of actor events in the background...
2023-06-23T09:21:05.444Z        INFO    fullnode        full/backfill.go:97     Starting backfill from current head 2973519
2023-06-23T09:21:05.444Z        INFO    fullnode        full/backfill.go:125    [0] backfilling actor events epoch:2973518, eventsAffected:0, entriesAffected:0
2023-06-23T09:21:09.426Z        INFO    fullnode        full/backfill.go:125    [100] backfilling actor events epoch:2973418, eventsAffected:0, entriesAffected:0
2023-06-23T09:21:21.288Z        INFO    fullnode        full/backfill.go:125    [200] backfilling actor events epoch:2973318, eventsAffected:0, entriesAffected:0
2023-06-23T09:21:25.873Z        INFO    fullnode        full/backfill.go:125    [300] backfilling actor events epoch:2973216, eventsAffected:0, entriesAffected:0
2023-06-23T09:25:34.964Z        INFO    fullnode        full/backfill.go:125    [5400] backfilling actor events epoch:2968086, eventsAffected:0, entriesAffected:0
2023-06-23T09:25:40.108Z        INFO    modules modules/actorevent.go:93        backfill events error: failed to load messages for tipset {bafy2bzacecowakd744tlnuueezixti5icjmb4dle52q5veevsfpsspbpqpafa,bafy2bzacecxl3ianxpwpymxunvs3o5uyesjfykua5qfbw4w5u6twjlpfkkgc4,bafy2bzacebbopm24ikoeu2lu6xyt53obm6s53ctyvit3lyean44vafyhs5alu,bafy2bzaceaw4jy56uhwbkjfo4zrvzt354x6adwhzwbc7jdjuff3ivawdqvnpm,bafy2bzacedqsm46ndmh55q4a2vll3ntpaivo2qqse65erd3kpe6xrslmkgd7g}: failed to load state tree at tipset [bafy2bzacecowakd744tlnuueezixti5icjmb4dle52q5veevsfpsspbpqpafa bafy2bzacecxl3ianxpwpymxunvs3o5uyesjfykua5qfbw4w5u6twjlpfkkgc4 bafy2bzacebbopm24ikoeu2lu6xyt53obm6s53ctyvit3lyean44vafyhs5alu bafy2bzaceaw4jy56uhwbkjfo4zrvzt354x6adwhzwbc7jdjuff3ivawdqvnpm bafy2bzacedqsm46ndmh55q4a2vll3ntpaivo2qqse65erd3kpe6xrslmkgd7g]: failed to load state tree bafy2bzacedaiycnoxohubm6btfyu6vl3ydolbpm5aiwuo62lydt75guebc2oo: failed to load hamt node: ipld: could not find bafy2bzacedaiycnoxohubm6btfyu6vl3ydolbpm5aiwuo62lydt75guebc2oo
2023-06-23T09:25:40.108Z        INFO    modules modules/actorevent.go:95        backfill of actor events done in: 4m34.664418745s

Backfill a node that had never stored any events

2023-06-23T10:34:33.773Z        INFO    modules modules/actorevent.go:89        EnableEventBackfillOnStartup was set, starting backfill of actor events in the background...
2023-06-23T10:34:33.774Z        INFO    fullnode        full/backfill.go:97     Starting backfill from current head 2973668
2023-06-23T10:34:33.774Z        INFO    fullnode        full/backfill.go:125    [0] backfilling actor events epoch:2973667, eventsAffected:0, entriesAffected:0
2023-06-23T10:34:34.220Z        INFO    fullnode        full/backfill.go:125    [100] backfilling actor events epoch:2973567, eventsAffected:168, entriesAffected:596
2023-06-23T10:34:34.538Z        INFO    fullnode        full/backfill.go:125    [200] backfilling actor events epoch:2973467, eventsAffected:273, entriesAffected:967
2023-06-23T10:34:34.792Z        INFO    fullnode        full/backfill.go:125    [300] backfilling actor events epoch:2973367, eventsAffected:443, entriesAffected:1578
2023-06-23T10:34:35.052Z        INFO    fullnode        full/backfill.go:125    [400] backfilling actor events epoch:2973267, eventsAffected:697, entriesAffected:2483
2023-06-23T10:34:35.300Z        INFO    fullnode        full/backfill.go:125    [500] backfilling actor events epoch:2973165, eventsAffected:829, entriesAffected:2946
2023-06-23T10:34:35.596Z        INFO    fullnode        full/backfill.go:125    [600] backfilling actor events epoch:2973064, eventsAffected:1025, entriesAffected:3642
...
2023-06-23T10:35:20.307Z        INFO    fullnode        full/backfill.go:125    [5500] backfilling actor events epoch:2968135, eventsAffected:8119, entriesAffected:28682
2023-06-23T10:35:22.937Z        INFO    fullnode        full/backfill.go:125    [5600] backfilling actor events epoch:2968034, eventsAffected:8369, entriesAffected:29586
2023-06-23T10:35:23.542Z        INFO    modules modules/actorevent.go:93        backfill events error: failed to load messages for tipset {bafy2bzacecowakd744tlnuueezixti5icjmb4dle52q5veevsfpsspbpqpafa,bafy2bzacecxl3ianxpwpymxunvs3o5uyesjfykua5qfbw4w5u6twjlpfkkgc4,bafy2bzacebbopm24ikoeu2lu6xyt53obm6s53ctyvit3lyean44vafyhs5alu,bafy2bzaceaw4jy56uhwbkjfo4zrvzt354x6adwhzwbc7jdjuff3ivawdqvnpm,bafy2bzacedqsm46ndmh55q4a2vll3ntpaivo2qqse65erd3kpe6xrslmkgd7g}: failed to load state tree at tipset [bafy2bzacecowakd744tlnuueezixti5icjmb4dle52q5veevsfpsspbpqpafa bafy2bzacecxl3ianxpwpymxunvs3o5uyesjfykua5qfbw4w5u6twjlpfkkgc4 bafy2bzacebbopm24ikoeu2lu6xyt53obm6s53ctyvit3lyean44vafyhs5alu bafy2bzaceaw4jy56uhwbkjfo4zrvzt354x6adwhzwbc7jdjuff3ivawdqvnpm bafy2bzacedqsm46ndmh55q4a2vll3ntpaivo2qqse65erd3kpe6xrslmkgd7g]: failed to load state tree bafy2bzacedaiycnoxohubm6btfyu6vl3ydolbpm5aiwuo62lydt75guebc2oo: failed to load hamt node: ipld: could not find bafy2bzacedaiycnoxohubm6btfyu6vl3ydolbpm5aiwuo62lydt75guebc2oo
2023-06-23T10:35:23.542Z        INFO    modules modules/actorevent.go:95        backfill of actor events done in: 49.768782713s

Verify backfill results

I compared the events.db that was generated from scratch via backfill to another events.db which was generated from by an active lotus node. I attached the second database as events2 and used the following sql to verify there are no missing events:

SELECT * FROM (
  SELECT e.height, e.tipset_key, e.emitter_addr, e.event_index, e.message_cid, e.message_index, e.reverted, ee.indexed, ee.flags, ee.key, ee.codec, ee.value 
  FROM event e, event_entry ee
  WHERE ee.event_id = e.id AND e.height > 2971667 AND e.height < 2973667
  EXCEPT
  SELECT e.height, e.tipset_key, e.emitter_addr, e.event_index, e.message_cid, e.message_index, e.reverted, ee.indexed, ee.flags, ee.key, ee.codec, ee.value 
  FROM events2.event e, events2.event_entry ee
  WHERE ee.event_id = e.id AND e.height > 2971667 AND e.height < 2973667
) WHERE reverted = 0

Result: 0 rows returned in 168ms

The nice thing is that the backfilled actor events naturally do not contain any duplicates as is generated during a live node as there are no reverts

SELECT COUNT(*) FROM event e, event_entry ee WHERE ee.event_id = e.id AND e.height > 2971667 AND e.height < 2973667
7975

SELECT COUNT(*) FROM events2.event e, events2.event_entry ee WHERE ee.event_id = e.id AND e.height > 2971667 AND e.height < 2973667
17633

[fridrik01]

Going to put up a new PR where this is implemented via lotus-shed backfilling