Badger -> Pebble DB - Chunk Data pack Pruner

[j1010001]

Why

In general, see Epic: #6515

Specifically for this issue:

1. Chunks data packs are stored on EN and use the most storage storage (growing ~1TB in 2 weeks, for first 2 weeks of MN26 - traffic dependent).
2. Pruning requires EN downtime, so requires manual intervention every time the storage gets close to a limit.

How will we measure success ?

• Validates assumptions that the pruning actually removes chuck data packs from disk.
• No downtime and no intervention is required for maintaining stable storage required for Chunk data packs on ENs.

Current status

Chunk data packs are already stored in Pebble DB on EN on Miannet. We haven’t implemented pruning yet, current solution requires downtime on EN to remove old data.
Scope: We have a POC PR based on v0.33, it needs to be ported to master, design and implementation reviewed, required refactoring of the POC PR finished.
No HCU required to deploy, only rolling upgrade.

Estimate

5 dev/weeks to be ready to deploy.

DACI

Role	Assigned
Driver	Technical: @zhangchiqing , EM: @j1010001
Approver	@Kay-Zee
Consulted	@AlexHentschel, @franklywatson
Informed	Flow protocol Engineering team

Tasks

Preview Give feedback

1. Verify removing chunk data pack and trigger compaction could claim disk space
2. [Database] Investigate why Pebble batch updates is slower than Badger #6825
   +0
   
3. Add test cases to verify the disk space will be reclaimed after pruned
4. generic block iterator
5. height based block iterator
6. height based pruner
7. Extract the chunk data pack pruner into a generic pruner that prune data by view/height
8. pruner engine that uses height based pruner
9. add pruner engine to execution builder
10. adding metrics and logging to pruner
11. Roll out to Testnet
12. Roll out to Mainnet
13. view based block iterator
14. view based pruner
15. switch pruner engine to use view based pruner
16. Optimize the implementation with pruning by view
17. badger database migration to backfill view block id index
18. migration validator
19. Roll out to Testnet, Mainnet

[zhangchiqing]

It's been 7 days since the latest spork (Sep 25), and the mainnet EN produced 480GB chunk data packs data.