Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: lookup txs by block number instead of block hash #19431

Merged
merged 2 commits into from
Apr 25, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion core/blockchain.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,10 @@ const (
// * the `TxHash`, `GasCost`, and `ContractAddress` fields are no longer stored for a receipt
// * the `TxHash`, `GasCost`, and `ContractAddress` fields are computed by looking up the
// receipts' corresponding block
BlockChainVersion uint64 = 5
// - Version 6
// The following incompatible database changes were added:
// * Transaction lookup information stores the corresponding block number instead of block hash
BlockChainVersion uint64 = 6
)

// CacheConfig contains the configuration values for the trie caching/pruning
Expand Down
40 changes: 25 additions & 15 deletions core/rawdb/accessors_indexes.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
package rawdb

import (
"math/big"

"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/ethdb"
Expand All @@ -27,28 +29,36 @@ import (

// ReadTxLookupEntry retrieves the positional metadata associated with a transaction
// hash to allow retrieving the transaction or receipt by hash.
func ReadTxLookupEntry(db ethdb.Reader, hash common.Hash) common.Hash {
func ReadTxLookupEntry(db ethdb.Reader, hash common.Hash) *uint64 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be worthwhile to have two methods? One that returns a uint64 and one that returns hash? That way, we would sometimes not needs to do lookup tx -> hash, read hashto number -> read cannon hash (number).

It would also make the code for the callers a bit simpler

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@holiman Could you point me to the code that you think would benefit from this? How performance critical is that code as well?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about ReadTransaction and ReadReceipt.
They do
ReadTxLookupEntry, ReadCanonicalHash,ReadBody .
If it's an old tx, it would internally be
read a hash, ReadHeaderNumber, ReadCanonicalHash,ReadBody .
which could be
read a hash -> ReadBody .
Maybe it would just overcomplicate things

Copy link
Contributor Author

@Matthalp-zz Matthalp-zz Apr 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@holiman I agree that legacy databases pay a penalty here. If a user was using the client for constantly looking up transactions they may feel the effects of this slowdown. However, if this was a serious user I would probably expect them to resync. We could also offer a CLI tool to do these various database upgrades. @karalabe let me know if this is something that would be valuable to have. It would be done in a follow up PR.

data, _ := db.Get(txLookupKey(hash))
if len(data) == 0 {
return common.Hash{}
return nil
}
//
if len(data) < common.HashLength {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am thinking is this the final step of txLookup? Since there is no flag at all now for the v6 format.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not clear what you mean by a flag.

Copy link
Member

@rjl493456442 rjl493456442 Apr 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean the "flag" for blob. For example, if the blob can be decoded into v3 structure, this is kind of "flag". Also if the length of blob is 32, it is kind of flag for v4, v5.

So probably in the future if we want to change the format again, it can be painful.

But just thinking.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the flag here is < 32 bytes :) But all in all I think we always want to try to decode into the newest version first. People who upgrade to 1.9 will with a high chance resync, so might as well use the happy path and not do 3 decoding to get it right.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rjl493456442 My intention was how @karalabe interpreted the code. If either of you feel like this needs to be made more clear and have a suggestion about how to do it please feel free to adjust it or let me know what to do.

var number big.Int
number.SetBytes(data)
numberU64 := number.Uint64()
return &numberU64
}
// Database v4 tx lookup format just stores the hash.
if len(data) == common.HashLength {
return common.BytesToHash(data)
return ReadHeaderNumber(db, common.BytesToHash(data))
}
// Probably it's legacy txlookup entry data, try to decode it.
// Finally try database v3 tx lookup format.
var entry LegacyTxLookupEntry
if err := rlp.DecodeBytes(data, &entry); err != nil {
log.Error("Invalid transaction lookup entry RLP", "hash", hash, "blob", data, "err", err)
return common.Hash{}
return nil
}
return entry.BlockHash
return &entry.BlockIndex
}

// WriteTxLookupEntries stores a positional metadata for every transaction from
// a block, enabling hash based transaction and receipt lookups.
func WriteTxLookupEntries(db ethdb.Writer, block *types.Block) {
for _, tx := range block.Transactions() {
if err := db.Put(txLookupKey(tx.Hash()), block.Hash().Bytes()); err != nil {
if err := db.Put(txLookupKey(tx.Hash()), block.Number().Bytes()); err != nil {
log.Crit("Failed to store transaction lookup entry", "err", err)
}
}
Expand All @@ -62,12 +72,12 @@ func DeleteTxLookupEntry(db ethdb.Writer, hash common.Hash) {
// ReadTransaction retrieves a specific transaction from the database, along with
// its added positional metadata.
func ReadTransaction(db ethdb.Reader, hash common.Hash) (*types.Transaction, common.Hash, uint64, uint64) {
blockHash := ReadTxLookupEntry(db, hash)
if blockHash == (common.Hash{}) {
blockNumber := ReadTxLookupEntry(db, hash)
if blockNumber == nil {
return nil, common.Hash{}, 0, 0
}
blockNumber := ReadHeaderNumber(db, blockHash)
if blockNumber == nil {
blockHash := ReadCanonicalHash(db, *blockNumber)
if blockHash == (common.Hash{}) {
return nil, common.Hash{}, 0, 0
}
body := ReadBody(db, blockHash, *blockNumber)
Expand All @@ -88,12 +98,12 @@ func ReadTransaction(db ethdb.Reader, hash common.Hash) (*types.Transaction, com
// its added positional metadata.
func ReadReceipt(db ethdb.Reader, hash common.Hash, config *params.ChainConfig) (*types.Receipt, common.Hash, uint64, uint64) {
// Retrieve the context of the receipt based on the transaction hash
blockHash := ReadTxLookupEntry(db, hash)
if blockHash == (common.Hash{}) {
blockNumber := ReadTxLookupEntry(db, hash)
if blockNumber == nil {
return nil, common.Hash{}, 0, 0
}
blockNumber := ReadHeaderNumber(db, blockHash)
if blockNumber == nil {
blockHash := ReadCanonicalHash(db, *blockNumber)
if blockHash == (common.Hash{}) {
return nil, common.Hash{}, 0, 0
}
// Read all the receipts from the block and return the one with the matching hash
Expand Down
123 changes: 71 additions & 52 deletions core/rawdb/accessors_indexes_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,69 +22,88 @@ import (

"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/core/types"
"github.com/ethereum/go-ethereum/ethdb"
"github.com/ethereum/go-ethereum/rlp"
)

// Tests that positional lookup metadata can be stored and retrieved.
func TestLookupStorage(t *testing.T) {
db := NewMemoryDatabase()
tests := []struct {
name string
writeTxLookupEntries func(ethdb.Writer, *types.Block)
}{
{
"DatabaseV6",
func(db ethdb.Writer, block *types.Block) {
WriteTxLookupEntries(db, block)
},
},
{
"DatabaseV4",
func(db ethdb.Writer, block *types.Block) {
for _, tx := range block.Transactions() {
db.Put(txLookupKey(tx.Hash()), block.Hash().Bytes())
}
},
},
{
"DatabaseV3",
func(db ethdb.Writer, block *types.Block) {
for index, tx := range block.Transactions() {
entry := LegacyTxLookupEntry{
BlockHash: block.Hash(),
BlockIndex: block.NumberU64(),
Index: uint64(index),
}
data, _ := rlp.EncodeToBytes(entry)
db.Put(txLookupKey(tx.Hash()), data)
}
},
},
}

tx1 := types.NewTransaction(1, common.BytesToAddress([]byte{0x11}), big.NewInt(111), 1111, big.NewInt(11111), []byte{0x11, 0x11, 0x11})
tx2 := types.NewTransaction(2, common.BytesToAddress([]byte{0x22}), big.NewInt(222), 2222, big.NewInt(22222), []byte{0x22, 0x22, 0x22})
tx3 := types.NewTransaction(3, common.BytesToAddress([]byte{0x33}), big.NewInt(333), 3333, big.NewInt(33333), []byte{0x33, 0x33, 0x33})
txs := []*types.Transaction{tx1, tx2, tx3}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {

block := types.NewBlock(&types.Header{Number: big.NewInt(314)}, txs, nil, nil)
db := NewMemoryDatabase()

// Check that no transactions entries are in a pristine database
for i, tx := range txs {
if txn, _, _, _ := ReadTransaction(db, tx.Hash()); txn != nil {
t.Fatalf("tx #%d [%x]: non existent transaction returned: %v", i, tx.Hash(), txn)
}
}
// Insert all the transactions into the database, and verify contents
WriteBlock(db, block)
WriteTxLookupEntries(db, block)
tx1 := types.NewTransaction(1, common.BytesToAddress([]byte{0x11}), big.NewInt(111), 1111, big.NewInt(11111), []byte{0x11, 0x11, 0x11})
tx2 := types.NewTransaction(2, common.BytesToAddress([]byte{0x22}), big.NewInt(222), 2222, big.NewInt(22222), []byte{0x22, 0x22, 0x22})
tx3 := types.NewTransaction(3, common.BytesToAddress([]byte{0x33}), big.NewInt(333), 3333, big.NewInt(33333), []byte{0x33, 0x33, 0x33})
txs := []*types.Transaction{tx1, tx2, tx3}

for i, tx := range txs {
if txn, hash, number, index := ReadTransaction(db, tx.Hash()); txn == nil {
t.Fatalf("tx #%d [%x]: transaction not found", i, tx.Hash())
} else {
if hash != block.Hash() || number != block.NumberU64() || index != uint64(i) {
t.Fatalf("tx #%d [%x]: positional metadata mismatch: have %x/%d/%d, want %x/%v/%v", i, tx.Hash(), hash, number, index, block.Hash(), block.NumberU64(), i)
}
if tx.Hash() != txn.Hash() {
t.Fatalf("tx #%d [%x]: transaction mismatch: have %v, want %v", i, tx.Hash(), txn, tx)
block := types.NewBlock(&types.Header{Number: big.NewInt(314)}, txs, nil, nil)

// Check that no transactions entries are in a pristine database
for i, tx := range txs {
if txn, _, _, _ := ReadTransaction(db, tx.Hash()); txn != nil {
t.Fatalf("tx #%d [%x]: non existent transaction returned: %v", i, tx.Hash(), txn)
}
}
}
}
// Delete the transactions and check purge
for i, tx := range txs {
DeleteTxLookupEntry(db, tx.Hash())
if txn, _, _, _ := ReadTransaction(db, tx.Hash()); txn != nil {
t.Fatalf("tx #%d [%x]: deleted transaction returned: %v", i, tx.Hash(), txn)
}
}
// Insert legacy txlookup and verify the data retrieval
for index, tx := range block.Transactions() {
entry := LegacyTxLookupEntry{
BlockHash: block.Hash(),
BlockIndex: block.NumberU64(),
Index: uint64(index),
}
data, _ := rlp.EncodeToBytes(entry)
db.Put(txLookupKey(tx.Hash()), data)
}
for i, tx := range txs {
if txn, hash, number, index := ReadTransaction(db, tx.Hash()); txn == nil {
t.Fatalf("tx #%d [%x]: transaction not found", i, tx.Hash())
} else {
if hash != block.Hash() || number != block.NumberU64() || index != uint64(i) {
t.Fatalf("tx #%d [%x]: positional metadata mismatch: have %x/%d/%d, want %x/%v/%v", i, tx.Hash(), hash, number, index, block.Hash(), block.NumberU64(), i)
// Insert all the transactions into the database, and verify contents
WriteCanonicalHash(db, block.Hash(), block.NumberU64())
WriteBlock(db, block)
tc.writeTxLookupEntries(db, block)

for i, tx := range txs {
if txn, hash, number, index := ReadTransaction(db, tx.Hash()); txn == nil {
t.Fatalf("tx #%d [%x]: transaction not found", i, tx.Hash())
} else {
if hash != block.Hash() || number != block.NumberU64() || index != uint64(i) {
t.Fatalf("tx #%d [%x]: positional metadata mismatch: have %x/%d/%d, want %x/%v/%v", i, tx.Hash(), hash, number, index, block.Hash(), block.NumberU64(), i)
}
if tx.Hash() != txn.Hash() {
t.Fatalf("tx #%d [%x]: transaction mismatch: have %v, want %v", i, tx.Hash(), txn, tx)
}
}
}
if tx.Hash() != txn.Hash() {
t.Fatalf("tx #%d [%x]: transaction mismatch: have %v, want %v", i, tx.Hash(), txn, tx)
// Delete the transactions and check purge
for i, tx := range txs {
DeleteTxLookupEntry(db, tx.Hash())
if txn, _, _, _ := ReadTransaction(db, tx.Hash()); txn != nil {
t.Fatalf("tx #%d [%x]: deleted transaction returned: %v", i, tx.Hash(), txn)
}
}
}
})
}
}