Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Non-breaking changes] Dynamic state snapshot #379

Closed
wants to merge 94 commits into from
Closed
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
8253211
implements RLP encoding/decoding processing for structs
trinhdn97 Jun 16, 2023
fa67be8
Update RLP lib
trinhdn97 Jun 16, 2023
9b5dadb
Implement RLP encoder code generation tool
trinhdn97 Jun 16, 2023
ddf35f8
Update unit tests and benchmark
trinhdn97 Jun 16, 2023
17e4876
Include RLPgen tool to CI
trinhdn97 Jun 20, 2023
aa7c4b4
Add benchmarks for types RLP encoding/decoding
trinhdn97 Jun 21, 2023
ac89cd3
Generate RLP encoder for some structs
trinhdn97 Jun 21, 2023
060f4ce
Convert status of receipts from uint to uint64
trinhdn97 Jun 21, 2023
788eef5
cmd/tomo: added counters to the geth inspect report
endadinh Jul 13, 2023
81be749
Change handling of dirty objects in state
endadinh Jul 13, 2023
8843c93
Move Message struct from package types to core
trinhdn97 Jul 14, 2023
4148d3f
Calculate TRC21GasPrice when convert a transaction to a core.Message
trinhdn97 Jul 14, 2023
47b9f53
Update unit tests
trinhdn97 Jul 14, 2023
3b6e3f3
Add more information fields to transaction receipts and regen codec/RLP
trinhdn97 Jul 14, 2023
0c7da38
Add database accessor methods for receipts/logs
trinhdn97 Jul 14, 2023
e1226ec
Add chain configs as a requirement for receipts/logs retrieving
trinhdn97 Jul 14, 2023
0eb25b2
[WIP] Fix unit tests
trinhdn97 Jul 14, 2023
8fdaf68
Unify multiple keccak interface into one inside crypto package
trinhdn97 Jul 16, 2023
c7be6cc
Merge branch 'refactor/keccak-state' into ft/stacktrie
trinhdn97 Jul 16, 2023
340bf2b
Add encode method to Node interface
trinhdn97 Jul 16, 2023
33149c9
Implement stacktrie
trinhdn97 Jul 16, 2023
2e9abc4
Add encode method for rawShortNode and rawFullNode
trinhdn97 Jul 16, 2023
008274f
Fix import cycle
trinhdn97 Jul 16, 2023
f88dba4
Fix stacktrie unit tests
trinhdn97 Jul 16, 2023
af788c6
Convert status of receipts from uint to uint64
trinhdn97 Jul 16, 2023
51df1b9
Merge branch 'update/rlp-lib' into optimize/receipt-storing
trinhdn97 Jul 16, 2023
b440640
Fix unit tests
trinhdn97 Jul 16, 2023
e0f71f5
Fix unit tests
trinhdn97 Jul 16, 2023
986a3c4
Remove irrelevant code and fix unit tests
trinhdn97 Jul 17, 2023
327c90d
Remove rlpLog RLP encoder
trinhdn97 Jul 17, 2023
4d22f8d
Fix unit tests
trinhdn97 Jul 18, 2023
5f1e2b4
Minor fix after reviewing
trinhdn97 Jul 18, 2023
c37affb
Move statedb.Account struct to types.StateAccount
trinhdn97 Jul 20, 2023
e0776fb
Move database_util.go to rawdb package
trinhdn97 Jul 20, 2023
5ce901b
Move db keys to schema.go
trinhdn97 Jul 20, 2023
36373e7
Refine db keys
trinhdn97 Jul 20, 2023
6392afd
Rename
trinhdn97 Jul 20, 2023
cf62b98
Split into sub accessor files
trinhdn97 Jul 20, 2023
97a62ca
Re-enable VM tests
trinhdn97 Jul 21, 2023
024181b
chore: remove intPool
c98tristan Jul 17, 2023
c384646
Feat: Update derive SHA for stacktrie
c98tristan Jul 19, 2023
b8421e5
Chore: Add benchmark for Stacktrie
c98tristan Jul 19, 2023
6099609
Chore: Fix Update function missing return type
c98tristan Jul 19, 2023
ae6ca82
Chore: Change GetRlp to EncodeIndex in LendingTransaction and OrderTr…
c98tristan Jul 24, 2023
515a614
Update DeriveSha with new Hasher and DerivableList interface
trinhdn97 Jul 19, 2023
44d2c54
Chore: Add intPool in interpreter.go
c98tristan Jul 24, 2023
5fae822
Merge branch 'refactor/state-account' into refactor/rawdb
trinhdn97 Jul 24, 2023
72ca227
Chore: Change parameter of NewBlock function from Trie to Stacktrie
c98tristan Jul 25, 2023
530cbb1
Chore: Sorting imported library
c98tristan Jul 25, 2023
260f47e
Chore: Sorting imported library
c98tristan Jul 25, 2023
a6098a1
Merge remote-tracking branch 'c98tristan/feat/update-trie-to-stacktri…
trinhdn97 Jul 25, 2023
b1cb0fc
Resolve conflicts after merged
trinhdn97 Jul 25, 2023
5e8bde3
Refactor some rawdb methods
trinhdn97 Jul 31, 2023
cb81493
[WIP] Implement new trie interface and separate preimageStore
trinhdn97 Jul 31, 2023
a409636
Refactor NewDatabaseWithConfig
trinhdn97 Jul 31, 2023
4e70b3e
Include configs when init trie databases
trinhdn97 Jul 31, 2023
1b9882d
Nitpick
trinhdn97 Jul 31, 2023
f7bb2d3
Merge pull request #2 from trinhdn2/update/rlp-lib
trinhdn2 Jul 31, 2023
2215526
Merge branch 'develop' into refactor/message-struct
trinhdn2 Jul 31, 2023
6d8f113
Merge pull request #3 from trinhdn2/refactor/message-struct
trinhdn2 Jul 31, 2023
e29a6fe
Merge pull request #4 from trinhdn2/refactor/keccak-state
trinhdn2 Jul 31, 2023
91bc3f9
Merge branch 'develop' into optimize/receipt-storing
trinhdn2 Jul 31, 2023
4056a32
Merge pull request #5 from trinhdn2/optimize/receipt-storing
trinhdn2 Jul 31, 2023
bdec0ee
Merge branch 'develop' into ft/stacktrie
trinhdn2 Jul 31, 2023
7cb8b5e
Merge pull request #6 from trinhdn2/ft/stacktrie
trinhdn2 Jul 31, 2023
a5ca630
Merge branch 'develop' into refactor/state-account
trinhdn2 Jul 31, 2023
6a5f547
Merge pull request #7 from trinhdn2/refactor/state-account
trinhdn2 Jul 31, 2023
74ba391
Merge branch 'develop' into refactor/rawdb
trinhdn97 Aug 1, 2023
f47664c
Merge pull request #8 from trinhdn2/refactor/rawdb
trinhdn2 Aug 1, 2023
475b70a
Merge pull request #9 from c98tristan:feat/update-trie-to-stacktrie
trinhdn97 Aug 1, 2023
25f9c44
Merge branch 'develop' into feat/update-trie-to-stacktrie
trinhdn97 Aug 1, 2023
b624700
Merge pull request #9 from c98tristan/feat/update-trie-to-stacktrie
trinhdn2 Aug 1, 2023
62f26cf
Minor fix
trinhdn97 Aug 1, 2023
097326c
Bump client version to 2.4.0
trinhdn2 Aug 2, 2023
1c0d050
Merge pull request #10 from nguyenbatam/fix_api_traceblock
trinhdn2 Aug 2, 2023
943f500
Fix UpdateStorage in trie
trinhdn97 Aug 3, 2023
be70268
Fix trie unit tests
trinhdn97 Aug 3, 2023
bf41851
Fix GetCommittedState and GetState
trinhdn97 Aug 3, 2023
12d08d0
Fix state test
trinhdn97 Aug 4, 2023
fadcdab
Merge branch 'refactor/trie-interface-2' into develop
trinhdn97 Aug 4, 2023
47e5d54
Implement AddUncheckedTx in chain maker
trinhdn97 Aug 4, 2023
b4e8864
Merge branch 'optimize/receipt-storing' into develop
trinhdn97 Aug 4, 2023
40d1d00
Merge remote-tracking branch 'endale98/remove_dirty' into develop
trinhdn97 Aug 4, 2023
0d0b124
Dynamic state snapshots
endadinh Jul 14, 2023
a3c9a57
Fix after rebase
trinhdn97 Jul 17, 2023
2ad92f6
fix unit tests
endadinh Jul 17, 2023
a0d9474
Remove duplicate accessor methods
trinhdn97 Jul 17, 2023
995f139
Record prevdestruct in journal
trinhdn97 Aug 7, 2023
8d08325
Use types.SlimAccount instead of snapshot.Account
trinhdn97 Aug 7, 2023
781ac07
Rename receivers and implement pendingStorage
trinhdn97 Aug 7, 2023
d3a156f
Update statedb.go
trinhdn97 Aug 7, 2023
3659a0d
Revert
trinhdn97 Aug 10, 2023
420d343
Fix commit and copy state
trinhdn97 Aug 14, 2023
bcae7d2
Record blockWriteTimer metric
trinhdn97 Aug 18, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion core/bench_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -238,10 +238,16 @@ func makeChainForBench(db ethdb.Database, full bool, count uint64) {
WriteHeader(db, header)
WriteCanonicalHash(db, hash, n)
WriteTd(db, hash, n, big.NewInt(int64(n+1)))
if n == 0 {
WriteChainConfig(db, hash, params.AllEthashProtocolChanges)
}
WriteHeadHeaderHash(db, hash)

if full || n == 0 {
block := types.NewBlockWithHeader(header)
WriteBody(db, hash, n, block.Body())
WriteBlockReceipts(db, hash, n, nil)
WriteHeadBlockHash(db, hash)
}
}
}
Expand Down Expand Up @@ -275,6 +281,8 @@ func benchReadChain(b *testing.B, full bool, count uint64) {
}
makeChainForBench(db, full, count)
db.Close()
cacheConfig := defaultCacheConfig
cacheConfig.Disabled = true

b.ReportAllocs()
b.ResetTimer()
Expand All @@ -284,7 +292,7 @@ func benchReadChain(b *testing.B, full bool, count uint64) {
if err != nil {
b.Fatalf("error opening database at %v: %v", dir, err)
}
chain, err := NewBlockChain(db, nil, params.TestChainConfig, ethash.NewFaker(), vm.Config{})
chain, err := NewBlockChain(db, cacheConfig, params.TestChainConfig, ethash.NewFaker(), vm.Config{})
if err != nil {
b.Fatalf("error creating chain: %v", err)
}
Expand Down
8 changes: 8 additions & 0 deletions core/blockchain.go
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,14 @@ type CacheConfig struct {
TrieNodeLimit int // Memory limit (MB) at which to flush the current in-memory trie to disk
TrieTimeLimit time.Duration // Time limit after which to flush the current in-memory trie to disk
}

// defaultCacheConfig are the default caching values if none are specified by the
// user (also used during testing).
var defaultCacheConfig = &CacheConfig{
TrieNodeLimit: 256,
TrieTimeLimit: 5 * time.Minute,
}

type ResultProcessBlock struct {
logs []*types.Log
receipts []*types.Receipt
Expand Down
7 changes: 6 additions & 1 deletion trie/database.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,11 @@ type rawNode []byte
func (n rawNode) Cache() (HashNode, bool) { panic("this should never end up in a live trie") }
func (n rawNode) fstring(ind string) string { panic("this should never end up in a live trie") }

func (n rawNode) EncodeRLP(w io.Writer) error {
_, err := w.Write(n)
return err
}

// rawFullNode represents only the useful data content of a full Node, with the
// caches and flags stripped out to minimize its data storage. This type honors
// the same RLP encoding as the original parent.
Expand Down Expand Up @@ -184,7 +189,7 @@ func (n *cachedNode) obj(hash common.Hash) Node {

// forChilds invokes the callback for all the tracked children of this Node,
// both the implicit ones from inside the Node as well as the explicit ones
//from outside the Node.
// from outside the Node.
func (n *cachedNode) forChilds(onChild func(hash common.Hash)) {
for child := range n.children {
onChild(child)
Expand Down
71 changes: 42 additions & 29 deletions trie/hasher.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright 2019 The go-ethereum Authors
// Copyright 2016 The go-ethereum Authors
// This file is part of the go-ethereum library.
//
// The go-ethereum library is free software: you can redistribute it and/or modify
Expand Down Expand Up @@ -38,16 +38,18 @@ func (b *sliceBuffer) Reset() {
// internal preallocated temp space
type hasher struct {
sha crypto.KeccakState
tmp sliceBuffer
parallel bool // Whether to use paralallel threads when hashing
tmp []byte
encbuf rlp.EncoderBuffer
parallel bool // Whether to use parallel threads when hashing
}

// hasherPool holds pureHashers
var hasherPool = sync.Pool{
New: func() interface{} {
return &hasher{
tmp: make(sliceBuffer, 0, 550), // cap is as large as a full FullNode.
sha: crypto.NewKeccakState(),
tmp: make([]byte, 0, 550), // cap is as large as a full fullNode.
sha: crypto.NewKeccakState(),
encbuf: rlp.NewEncoderBuffer(nil),
}
},
}
Expand All @@ -62,14 +64,14 @@ func returnHasherToPool(h *hasher) {
hasherPool.Put(h)
}

// hash collapses a Node down into a hash Node, also returning a copy of the
// original Node initialized with the computed hash to replace the original one.
// hash collapses a node down into a hash node, also returning a copy of the
// original node initialized with the computed hash to replace the original one.
func (h *hasher) hash(n Node, force bool) (hashed Node, cached Node) {
// We're not storing the Node, just hashing, use available cached data
// Return the cached hash if it's available
if hash, _ := n.Cache(); hash != nil {
return hash, n
}
// Trie not processed yet or needs storage, walk the children
// Trie not processed yet, walk the children
switch n := n.(type) {
case *ShortNode:
collapsed, cached := h.hashShortNodeChildren(n)
Expand Down Expand Up @@ -97,11 +99,11 @@ func (h *hasher) hash(n Node, force bool) (hashed Node, cached Node) {
}
}

// hashShortNodeChildren collapses the short Node. The returned collapsed Node
// hashShortNodeChildren collapses the short node. The returned collapsed node
// holds a live reference to the Key, and must not be modified.
// The cached
func (h *hasher) hashShortNodeChildren(n *ShortNode) (collapsed, cached *ShortNode) {
// Hash the short Node's child, caching the newly hashed subtree
// Hash the short node's child, caching the newly hashed subtree
collapsed, cached = n.copy(), n.copy()
// Previously, we did copy this one. We don't seem to need to actually
// do that, since we don't overwrite/reuse keys
Expand All @@ -116,7 +118,7 @@ func (h *hasher) hashShortNodeChildren(n *ShortNode) (collapsed, cached *ShortNo
}

func (h *hasher) hashFullNodeChildren(n *FullNode) (collapsed *FullNode, cached *FullNode) {
// Hash the full Node's children, caching the newly hashed subtrees
// Hash the full node's children, caching the newly hashed subtrees
cached = n.copy()
collapsed = n.copy()
if h.parallel {
Expand Down Expand Up @@ -147,35 +149,46 @@ func (h *hasher) hashFullNodeChildren(n *FullNode) (collapsed *FullNode, cached
return collapsed, cached
}

// shortnodeToHash creates a HashNode from a ShortNode. The supplied shortnode
// shortnodeToHash creates a hashNode from a shortNode. The supplied shortnode
// should have hex-type Key, which will be converted (without modification)
// into compact form for RLP encoding.
// If the rlp data is smaller than 32 bytes, `nil` is returned.
func (h *hasher) shortnodeToHash(n *ShortNode, force bool) Node {
h.tmp.Reset()
if err := rlp.Encode(&h.tmp, n); err != nil {
panic("encode error: " + err.Error())
}
n.encode(h.encbuf)
enc := h.encodedBytes()

if len(h.tmp) < 32 && !force {
if len(enc) < 32 && !force {
return n // Nodes smaller than 32 bytes are stored inside their parent
}
return h.hashData(h.tmp)
return h.hashData(enc)
}

// shortnodeToHash is used to creates a HashNode from a set of hashNodes, (which
// shortnodeToHash is used to creates a hashNode from a set of hashNodes, (which
// may contain nil values)
func (h *hasher) fullnodeToHash(n *FullNode, force bool) Node {
h.tmp.Reset()
// Generate the RLP encoding of the Node
if err := n.EncodeRLP(&h.tmp); err != nil {
panic("encode error: " + err.Error())
}
n.encode(h.encbuf)
enc := h.encodedBytes()

if len(h.tmp) < 32 && !force {
if len(enc) < 32 && !force {
return n // Nodes smaller than 32 bytes are stored inside their parent
}
return h.hashData(h.tmp)
return h.hashData(enc)
}

// encodedBytes returns the result of the last encoding operation on h.encbuf.
// This also resets the encoder buffer.
//
// All node encoding must be done like this:
//
// node.encode(h.encbuf)
// enc := h.encodedBytes()
//
// This convention exists because node.encode can only be inlined/escape-analyzed when
// called on a concrete receiver type.
func (h *hasher) encodedBytes() []byte {
h.tmp = h.encbuf.AppendToBytes(h.tmp[:0])
h.encbuf.Reset(nil)
return h.tmp
}

// hashData hashes the provided data
Expand All @@ -188,8 +201,8 @@ func (h *hasher) hashData(data []byte) HashNode {
}

// proofHash is used to construct trie proofs, and returns the 'collapsed'
// Node (for later RLP encoding) aswell as the hashed Node -- unless the
// Node is smaller than 32 bytes, in which case it will be returned as is.
// node (for later RLP encoding) as well as the hashed node -- unless the
// node is smaller than 32 bytes, in which case it will be returned as is.
// This method does not do anything on value- or hash-nodes.
func (h *hasher) proofHash(original Node) (collapsed, hashed Node) {
switch n := original.(type) {
Expand Down
14 changes: 4 additions & 10 deletions trie/node.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ var indices = []string{"0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b
type Node interface {
fstring(string) string
Cache() (HashNode, bool)
encode(w rlp.EncoderBuffer)
}

type (
Expand All @@ -52,16 +53,9 @@ var nilValueNode = ValueNode(nil)

// EncodeRLP encodes a full Node into the consensus RLP format.
func (n *FullNode) EncodeRLP(w io.Writer) error {
var nodes [17]Node

for i, child := range &n.Children {
if child != nil {
nodes[i] = child
} else {
nodes[i] = nilValueNode
}
}
return rlp.Encode(w, nodes)
eb := rlp.NewEncoderBuffer(w)
n.encode(eb)
return eb.Flush()
}

func (n *FullNode) copy() *FullNode { copy := *n; return &copy }
Expand Down
72 changes: 72 additions & 0 deletions trie/node_enc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
// Copyright 2022 The go-ethereum Authors
// This file is part of the go-ethereum library.
//
// The go-ethereum library is free software: you can redistribute it and/or modify
// it under the terms of the GNU Lesser General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// The go-ethereum library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Lesser General Public License for more details.
//
// You should have received a copy of the GNU Lesser General Public License
// along with the go-ethereum library. If not, see <http://www.gnu.org/licenses/>.

package trie

import (
"github.com/tomochain/tomochain/rlp"
)

func nodeToBytes(n Node) []byte {
w := rlp.NewEncoderBuffer(nil)
n.encode(w)
result := w.ToBytes()
w.Flush()
return result
}

func (n *FullNode) encode(w rlp.EncoderBuffer) {
offset := w.List()
for _, c := range n.Children {
if c != nil {
c.encode(w)
} else {
w.Write(rlp.EmptyString)
}
}
w.ListEnd(offset)
}

func (n *ShortNode) encode(w rlp.EncoderBuffer) {
offset := w.List()
w.WriteBytes(n.Key)
if n.Val != nil {
n.Val.encode(w)
} else {
w.Write(rlp.EmptyString)
}
w.ListEnd(offset)
}

func (n HashNode) encode(w rlp.EncoderBuffer) {
w.WriteBytes(n)
}

func (n ValueNode) encode(w rlp.EncoderBuffer) {
w.WriteBytes(n)
}

func (n rawNode) encode(w rlp.EncoderBuffer) {
w.Write(n)
}

func (n rawShortNode) encode(w rlp.EncoderBuffer) {
panic("this should never end up in a live trie")
}

func (n rawFullNode) encode(w rlp.EncoderBuffer) {
panic("this should never end up in a live trie")
}
Loading