Download checkpoint block's blob when checkpoint syncing

[michaelsproul]

Description

Currently we don't download the blob (if any) for the checkpoint block, which means we violate the database invariant that all blocks in the data availability period have their blobs stored on disk.

The complication is that checkpoint sync servers running checkpointz don't (yet) expose the blob endpoint. My proposed fix is to add logic to Lighthouse in v5.0.0 to download the blobs from the checkpoint server when we see that a block has them.

This will give us until the date of the fork (March 13) to sort out blob support in checkpointz, without requiring anyone to upgrade past v5.0.0. I think this is the best chance we have of a good UX experience. In the meantime checkpoint sync will continue working fine because none of the blocks will have blobs prior to Deneb.

Version

Lighthouse v4.6.0

[michaelsproul]

Thanks @kasey from Prysm for finding this issue and raising it

[michaelsproul]

NB: the reason it is substantially easier for us to get the blob from the checkpoint sync server rather than P2P is that:

• we want to maintain this database invariant on startup: all blocks in the data availability period that are stored on disk also have their corresponding blobs stored on disk
• initialization of the database happens prior to the network stack starting up, so it's simpler to establish this invariant without requiring a dependency on the networking stack

[michaelsproul]

Resolved by #5252