ERA: Add ERA file support package

[ScottyPoi]

Introduces new package "@ethereumjs/era"

Package will include utility functions to enable creation and parsing of ERA files (era, era1, era2, etc.)

era package files

• era/src/types.ts
  • types for era data
• era/src/snappy.ts
  • functions for compressing / decompressing era file data
• era/src/e2store.ts
  • functions for encoding / decoding e2store objects
• era/src/blockTuple.ts`
  • functions to serialize / deserialize block "tuples"
• era/src/era1.ts`
  • functions to create / read era1 files
• era/src/exportHistory.ts
  • functions to export history from dataDir as era1 files

---

ERA1

era1 specs: ethereum/go-ethereum#26621

Era1 stores one epoch of history data along with the hash tree root of the epoch accumulator as an anchor for validation.

Era1 contains 8192 block tuples

A block tuple contains

• header
• body
• block receipts
• total difficulty

Era1 also contains the root of an Epoch Accumulator

Epoch Accumulator is an SSZ object used in Portal Network. An Epoch Accumulator is constructed from a list of 8192 Header Records

A Header Record is an object with a blockHash and totalDifficulty.

The accumulator can be reconstructed to validate the contents of era1 against the provided accumulator root.

The accumulator roots are part of a static set of roots that can be externally validated.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.51%. Comparing base (cac7cc4) to head (d410962).
Report is 1 commits behind head on master.

Additional details and impacted files

Flag	Coverage Δ
block	76.87% <ø> (ø)
blockchain	85.69% <ø> (ø)
client	66.26% <ø> (ø)
common	90.31% <ø> (ø)
devp2p	76.27% <ø> (ø)
ethash	81.04% <ø> (ø)
evm	69.34% <ø> (ø)
genesis	99.84% <ø> (ø)
mpt	59.95% <ø> (ø)
rlp	69.70% <ø> (ø)
statemanager	70.15% <ø> (ø)
tx	80.52% <ø> (ø)
util	85.16% <ø> (ø)
vm	57.81% <ø> (ø)
wallet	83.78% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[ScottyPoi]

Tests included cover the reading/parsing of era1 files

To test the "export history" functions would require a client synced to at least 2 epochs (8192 x 2), which feels excessive for the vitest suite.

To validate the export history functions:

1. Start a client on mainnet with rpc enabled (debug namespace)
2. Sync client to at least 8192 x 2 blocks
3. Call RPC method debug_exportHistory (will generate binary files)
4. Compare generated era1 files to matching binary file available at era1.ethportal.net

[acolytec3]

Could we add a parameter for epoch so we only export one epoch at a time? As it currently stands, it's kinda weird that I call exportHistoryAsEra1 and it just sits there and runs.

Or, maybe, better yet, could we just write a helper script that reads the client DB and formats the era files without having to initiate via RPC?

[ScottyPoi]

1. Yes, but since it uses total_difficulty it does have to start from block 0 and work forward.
2. That could work -- I will look into it

[ScottyPoi]

Ah, OK -- I didn't realize that we stored totalDifficulty per block as a database item. So exporting a single, specific epoch should be possible after all

[acolytec3]

I still think an ideal scenario would be a standalone script that just reads the DB rather than an RPC call since it's weird for an RPC call to not return the actual data being requested but it gets written to disk separately. Is that much more difficult? Seems like you shoudl be able to pass in the path to the DB and just read things directly.

[acolytec3]

This may be user error but I did the following steps:

• Run the client
• Sync a few hundred thousand blocks.
• Call debug_exportHistoryAsEra1

I then put the following script together.

import { readERA1, validateERA1 } from '@ethereumjs/era'
import { readFileSync } from 'fs'
const era1 = readFileSync('./epoch-0.era1')

const main = async () => {
    console.log(await validateERA1(era1))
    const blockReader = await readERA1(era1)
    for await (const block of blockReader) {
        console.log(block)
    }
}

void main()

I get the following error:

Error: invalid data length, got 281474202442495, expected max of 52651
    at readEntry (/home/jim/development/ethjs/packages/era/src/e2store.ts:54:11)
    at readBlockTupleAtOffset (/home/jim/development/ethjs/packages/era/src/blockTuple.ts:67:23)
    at readOtherEntries (/home/jim/development/ethjs/packages/era/src/era1.ts:145:21)
    at readAccumulatorRoot (/home/jim/development/ethjs/packages/era/src/era1.ts:157:37)
    at validateERA1 (/home/jim/development/ethjs/packages/era/src/era1.ts:168:33)
    at main (/home/jim/development/ethjs/packages/client/era.ts:6:23)
    at <anonymous> (/home/jim/development/ethjs/packages/client/era.ts:13:6)

Does that mean the era files I exported from the client didn't work right?

[ScottyPoi]

This may be user error but I did the following steps:
...
Does that mean the era files I exported from the client didn't work right?

Try changing:
const era1 = readFileSync('./epoch-0.era1')
to
const era1 = new Uint8Array(readFileSync('./epoch-0.era1'))

[acolytec3]

This may be user error but I did the following steps:
...
Does that mean the era files I exported from the client didn't work right?

Try changing:
const era1 = readFileSync('./epoch-0.era1')
to
const era1 = new Uint8Array(readFileSync('./epoch-0.era1'))

Thanks! I figured it was something silly on my part. Can confirm that this now works.

[acolytec3]

Couple of small things to clean up before merging but I like this approach a lot better.

More general question, do we want to house support for the entire e2store file support in this @ethereumjs/era package or do we have to have the separate one for Ultralight since we need the era file support for our bridge scripts in Ultralight?

[acolytec3]

I think you likely need to add block and blockchain here so we don't end up with weird compilation errors down the line.

[ScottyPoi]

Couple of small things to clean up before merging but I like this approach a lot better.

More general question, do we want to house support for the entire e2store file support in this @ethereumjs/era package or do we have to have the separate one for Ultralight since we need the era file support for our bridge scripts in Ultralight?

Made the suggested changes.

--

I think ultimately we could bring in the rest of the code from the ultralight era package, and ultralight could import @ethereumjs/era.

It made sense to me to just start with era1 here since they are created by / used to sync an EL client.

[acolytec3]

LGTM