feat: change data store backend to use liblmdb directly

[alexghr]

This PR adds a new backend implementation for data stores that's based on a thin layer on top of lmdb.c. This is the same layer used by NativeWorldState.

This enables us to have tighter control over how data is serialized (no more bigint issues #9690 #9793), how it's accessed and enable us to use a consistent version of lmdb across our stack.

Things brings with it a change of interface since reads and writes are asynchronous.

Architecture

The architecture is similar to NativeWorldState: a module that wraps lmdb.c and provides C++ idiomatic access to databases/transactions/cursors liblmdb. This module is thread safe.

This module is then exposed through node-module-api to Nodejs. The communication interface between the C++ code and Nodejs is based on passing msgpack encoded messages around. The addon interface is really simple, only exposing a single class with a single asynchronous method call: (message: Buffer) => Promise<Buffer>.

The C++ module does not have its own thread pool, it will piggy back off the Nodejs thread pool, which means we have to be careful not to exhaust it.

On the Nodejs side we create a new AsyncStore backend that implements the same interface (only async).

Transactions

LMDB supports multiple concurrent readers, but one writer.

The WriteTransaction class in Nodejs accumulates writes locally and sends them to the database as one big, atomic batch. Any reads that happen while a write transaction is open (and in the same async context) take the uncommitted data into account.

While WriteTransaction is accumulating writes, reads to the database are still honoured, but they will only see committed data (providing isolation from dirty writes). The WriteTransaction object is only available in the async context (using AsyncLocalStorage) that started that operation.

The Nodejs store queues up write transactions so that only one is active at a time.

Cursors

Cursors on the Nodejs side implement the AsyncIterable protocol so they can be used in for await of loops and can be passed to our helpers in aztec/foundation (e.g. toArray, take, etc)

Cursors use a long-lived read transaction. A lot of the queries used in our stores actual depend on cursors (e.g. getLatestSynchedL2Block - starts a cursor at the end of the database and reads one block).

We have a limited number of readers available in C++, if this number is reached then the text read will block until a reader becomes available. The Nodejs store uses a semaphore that only allows up to maxReaders - 1 cursors to be open at any one time. We always leave one reader available to perform simple gets (otherwise we'd risk blocking the entire thread pool)

We've added two 'optimizations' to our cursor implementation: (1) when starting a cursor the first page of results is sent back immediately and (2) if we know we want a small number of results (e.g. the last block in getLatestSynchedL2Block) then close the cursors in the same operation (this way we avoid keeping a reader open that will be closed in the next async execution)

Performance

In tests the performance is similar to the old backend. There is a penalty to reads (reads are async now) but writes are on par.

Changes to existing stores

The only modification necessary has been to have async reads and await the write operations in transactions.

Ported data stores

• the archiver (blocks, logs, contracts, txs)
• the tx mempool
• the proving job store

TODO

• port attestation pool, peer store
• add metrics
• fix merge conflicts 😢

[PhilWindle]

Should this be in a try/catch to ensure we release the cursor if necessary?

[PhilWindle]

Remove

[PhilWindle]

Should these be commented out?

[spalladino]

Hell of a PR 👏

Left a few questions mostly.

[spalladino]

Is it ok for the _advance_cursor call to be outside the lock guard block?

[PhilWindle]

It is, in that the mutex is there to protect the _cursors store. We want to be able to use multiple cursors concurrently.

However, you raise a good point that a sinlge cursor is not thread-safe itself. It relies on TS to never try and use a single cursor concurrently. I think we should protect against this however in C++ too. It is simple to do, we use a mutex inside cursor itself.

[spalladino]

Is this file expected to be here? Or it got in through a bad merge?

[spalladino]

Does it make sense to move this out of the tx block, since it may take a bit of time and doesn't need db access?

[PhilWindle]

Yes, I think so. Will move.

[spalladino]

I had to write this to actually convince myself that this cleaned up the tx as expected. It's nice when js pleasantly surprises you.

[spalladino]

Shouldn't this one be a read tx?

[PhilWindle]

Yes, thanks. We have been trying to find all the places where we have unnecessary write transactions. We missed this one.

[spalladino]

I'm not sure I fully understand ReadTransaction. I expect that if I open a ReadTx, then run two gets (with a write happening inbetween them on another async thread), then the 2nd get would be unaffected by the write. Is that the case? I couldn't find any sort of tx identifier, that's why I ask.

[PhilWindle]

Hmm, this actually isn't how ReadTransaction works. Perhaps it shold have a different name. IIUC, any single call on ReadTransaction won't be affected by a write, including iteration. But successive calls on ReadTransaction will be.

[spalladino]

Not gonna lie, I see this and I get scared about multiple Fr classes floating around, whether it's due to a different package version being loaded, or simple because we have other Frs such as the one coming out from bbjs (see barretenberg/ts/src/types/fields.ts). Not sure how to handle this though.

[spalladino]

Should be able to jump with the binary search here to avoid the linear search, right? Talking about unneeded optimizations...