core: lookup txs by block number instead of block hash #19431

Matthalp-zz · 2019-04-10T05:13:34Z

Transaction hashes now store a reference to their corresponding
block number as opposed to their hash. In benchmarks this was
shown to reduce storage by over 12 GB.

The main limitation of this approach is that transactions on
non-canonical blocks could never be looked up, however that is
currently not supported.

The database version has been upgraded to version 5 and the
transaction lookup process is backwards-compatible with the
prior two transaction lookup formats prexisting in the
database instance. Tests have been added to ensure this.

karalabe · 2019-04-15T11:11:59Z

I've built an image with your last commit (the hacky one) on top of the latest version of your previous PR (currently pending merge) and deployed them on 2 VMs (06/07 (old-pr/old-pr-and-this) for Martin). Will report on the fast sync results in half a day.

Matthalp-zz · 2019-04-15T15:47:28Z

Thanks @karalabe ! Excited to see if we get any additional savings here.

karalabe · 2019-04-15T16:05:51Z

Sync is not yet done, but your numbers seem to be fairly accurate. PR is about 12GB smaller than without it.

karalabe · 2019-04-16T06:52:59Z

Fast sync done, 12GB saved!

Matthalp-zz · 2019-04-16T14:35:14Z

@karalabe That's great! I've update the PR with the actual code and backwards compatibility with the prior transaction lookup formats.

Transaction hashes now store a reference to their corresponding block number as opposed to their hash. In benchmarks this was shown to reduce storage by over 12 GB. The main limitation of this approach is that transactions on non-canonical blocks could never be looked up, however that is currently not supported. The database version has been upgraded to version 5 and the transaction lookup process is backwards-compatible with the prior two transaction lookup formats prexisting in the database instance. Tests have been added to ensure this.

holiman · 2019-04-24T08:25:03Z

core/rawdb/accessors_indexes.go

@@ -27,28 +29,36 @@ import (

 // ReadTxLookupEntry retrieves the positional metadata associated with a transaction
 // hash to allow retrieving the transaction or receipt by hash.
-func ReadTxLookupEntry(db ethdb.Reader, hash common.Hash) common.Hash {
+func ReadTxLookupEntry(db ethdb.Reader, hash common.Hash) *uint64 {


Would it be worthwhile to have two methods? One that returns a uint64 and one that returns hash? That way, we would sometimes not needs to do lookup tx -> hash, read hashto number -> read cannon hash (number).

It would also make the code for the callers a bit simpler

@holiman Could you point me to the code that you think would benefit from this? How performance critical is that code as well?

I was thinking about ReadTransaction and ReadReceipt.
They do
ReadTxLookupEntry, ReadCanonicalHash,ReadBody .
If it's an old tx, it would internally be
read a hash, ReadHeaderNumber, ReadCanonicalHash,ReadBody .
which could be
read a hash -> ReadBody .
Maybe it would just overcomplicate things

@holiman I agree that legacy databases pay a penalty here. If a user was using the client for constantly looking up transactions they may feel the effects of this slowdown. However, if this was a serious user I would probably expect them to resync. We could also offer a CLI tool to do these various database upgrades. @karalabe let me know if this is something that would be valuable to have. It would be done in a follow up PR.

core/rawdb/accessors_indexes.go

karalabe · 2019-04-25T06:43:28Z

Apart from my 2 tiny nits, LGTM

rjl493456442

Just a small question, otherwise LGTM

rjl493456442 · 2019-04-25T08:36:42Z

core/rawdb/accessors_indexes.go

+		return nil
+	}
+	// Database v6 tx lookup just stores the block number
+	if len(data) < common.HashLength {


I am thinking is this the final step of txLookup? Since there is no flag at all now for the v6 format.

Writing was updated to use this format: https://github.com/ethereum/go-ethereum/pull/19431/files#diff-0fba94aa54693ff2a4f26210c404dbd9R59

Not clear what you mean by a flag.

I mean the "flag" for blob. For example, if the blob can be decoded into v3 structure, this is kind of "flag". Also if the length of blob is 32, it is kind of flag for v4, v5.

So probably in the future if we want to change the format again, it can be painful.

But just thinking.

Well, the flag here is < 32 bytes :) But all in all I think we always want to try to decode into the newest version first. People who upgrade to 1.9 will with a high chance resync, so might as well use the happy path and not do 3 decoding to get it right.

@rjl493456442 My intention was how @karalabe interpreted the code. If either of you feel like this needs to be made more clear and have a suggestion about how to do it please feel free to adjust it or let me know what to do.

* core: lookup txs by block number instead of block hash Transaction hashes now store a reference to their corresponding block number as opposed to their hash. In benchmarks this was shown to reduce storage by over 12 GB. The main limitation of this approach is that transactions on non-canonical blocks could never be looked up, however that is currently not supported. The database version has been upgraded to version 5 and the transaction lookup process is backwards-compatible with the prior two transaction lookup formats prexisting in the database instance. Tests have been added to ensure this. * core/rawdb: tiny review nit fixes

Matthalp-zz requested review from holiman, karalabe and zsfelfoldi as code owners April 10, 2019 05:13

Matthalp-zz mentioned this pull request Apr 10, 2019

core, eth, les, light: avoid storing computable receipt metadata #19345

Merged

Matthalp-zz force-pushed the optimize-tx-lookup-storage branch from d90ef2b to 3ab452b Compare April 16, 2019 14:33

Matthalp-zz changed the title ~~[BENCHMARK] Only store block number for transaction look ups~~ core: lookup txs by block number instead of block hash Apr 16, 2019

Matthalp-zz force-pushed the optimize-tx-lookup-storage branch 2 times, most recently from 8ae92dd to 48177b5 Compare April 16, 2019 14:37

Matthalp-zz force-pushed the optimize-tx-lookup-storage branch from 48177b5 to 835abc4 Compare April 17, 2019 13:39

holiman reviewed Apr 24, 2019

View reviewed changes

karalabe reviewed Apr 25, 2019

View reviewed changes

core/rawdb/accessors_indexes.go Outdated Show resolved Hide resolved

karalabe reviewed Apr 25, 2019

View reviewed changes

core/rawdb/accessors_indexes.go Outdated Show resolved Hide resolved

core/rawdb: tiny review nit fixes

7be6104

karalabe added this to the 1.9.0 milestone Apr 25, 2019

rjl493456442 approved these changes Apr 25, 2019

View reviewed changes

karalabe merged commit 9374175 into ethereum:master Apr 25, 2019

This was referenced Jan 16, 2025

reduce chain database size XinFinOrg/XDPoSChain#570

Open

all: reduce the chain db size XinFinOrg/XDPoSChain#816

Merged

gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Jan 23, 2025

core: lookup txs by block number instead of block hash (ethereum#19431)

4e5a7fa

gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Jan 23, 2025

core: lookup txs by block number instead of block hash (ethereum#19431)

4d1b40f

gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Jan 23, 2025

core: lookup txs by block number instead of block hash (ethereum#19431)

a88e48f

gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Jan 24, 2025

core: lookup txs by block number instead of block hash (ethereum#19431)

85fc567

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core: lookup txs by block number instead of block hash #19431

core: lookup txs by block number instead of block hash #19431

Matthalp-zz commented Apr 10, 2019 •

edited

Loading

karalabe commented Apr 15, 2019 •

edited

Loading

Matthalp-zz commented Apr 15, 2019

karalabe commented Apr 15, 2019

karalabe commented Apr 16, 2019

Matthalp-zz commented Apr 16, 2019

holiman Apr 24, 2019

Matthalp-zz Apr 24, 2019

holiman Apr 25, 2019

Matthalp-zz Apr 25, 2019 •

edited

Loading

karalabe commented Apr 25, 2019

rjl493456442 left a comment

rjl493456442 Apr 25, 2019

karalabe Apr 25, 2019

karalabe Apr 25, 2019

rjl493456442 Apr 25, 2019 •

edited

Loading

karalabe Apr 25, 2019

Matthalp-zz Apr 25, 2019

core: lookup txs by block number instead of block hash #19431

core: lookup txs by block number instead of block hash #19431

Conversation

Matthalp-zz commented Apr 10, 2019 • edited Loading

karalabe commented Apr 15, 2019 • edited Loading

Matthalp-zz commented Apr 15, 2019

karalabe commented Apr 15, 2019

karalabe commented Apr 16, 2019

Matthalp-zz commented Apr 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Matthalp-zz Apr 25, 2019 • edited Loading

Choose a reason for hiding this comment

karalabe commented Apr 25, 2019

rjl493456442 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rjl493456442 Apr 25, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Matthalp-zz commented Apr 10, 2019 •

edited

Loading

karalabe commented Apr 15, 2019 •

edited

Loading

Matthalp-zz Apr 25, 2019 •

edited

Loading

rjl493456442 Apr 25, 2019 •

edited

Loading