Skip to content
This repository has been archived by the owner on Oct 19, 2021. It is now read-only.

Update to Latest Go-Ethereum Upstream #17

Merged
merged 351 commits into from
Apr 19, 2021
Merged

Update to Latest Go-Ethereum Upstream #17

merged 351 commits into from
Apr 19, 2021

Conversation

rauljordan
Copy link
Contributor

Update our go-ethereum fork to the latest version in Prysm, important for catalyst testing the eth1 <> eth2 merge

karalabe and others added 30 commits January 15, 2021 12:54
core/state/snapshot: add generation logs to storage too
This PR has two changes in the les protocol:

- the auxRoot is not supported. See ethereum/devp2p#171 for more information
- the empty response will be returned in GetHelperTrieProofsMsg request if the merkle
   proving is failed. note, for backward compatibility, the empty merkle proof as well as
   the request auxiliary data will still be returned in  les2/3 protocol no matter the proving
   is successful or not. the proving failure can happen e.g. request the proving for a
   non-included entry in helper trie (unstable header).
* tests/fuzzers/abi: better test generation

* tests/fuzzers/abi: fixed packing issue

* oss-fuzz: enable abi fuzzer
* cmd/geth: dump config

* cmd/geth: dump config

* cmd/geth: properly read config again

* cmd/geth: override metrics if flags are set

* cmd/geth: write metrics regardless if enabled

* cmd/geth: renamed to metricsfromcliargs

* metrics: add default configuration
* core/state/snapshot: write snapshot generator in batch

* core: refactor the tests

* core: update tests

* core: update tests
Adding warnings of free disk space left and graceful shutdown when there is not enough space left.
This also adds a flag datadir.minfreedisk which can be used to set the trigger for low disk space, and setting it to zero disables the check. 

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
This PR introduces a new config field SyncFromCheckpoint for light client.

In some special scenarios, it's required to start synchronization from some
arbitrary checkpoint or even from the scratch. So this PR offers this
flexibility to users so that the synchronization start point can be configured.

There are two relevant configs: SyncFromCheckpoint and Checkpoint.

- If the SyncFromCheckpoint is true, the light client will try to sync from the
  specified checkpoint.

- If the Checkpoint is not configured, then the light client will sync from the
  scratch(from the latest header if the database is not empty)

Additional notes: these two configs are not visible in the CLI flags but only
accessable in the config file.

Example Usage:

[Eth]
SyncFromCheckpoint = true

[Eth.Checkpoint]
SectionIndex = 100
SectionHead = "0xabc"
CHTRoot = "0xabc"
BloomRoot = "0xabc"

PS. Historical checkpoint can be retrieved from the synced full node or light
client via les_getCheckpoint API.
This is a simple refactoring, extracting common ancestor
negotiation logic to named function
Squashed from the following commits:

core/state: lazily init snapshot storage map
core/state: fix flawed meter on storage reads
core/state: make statedb/stateobjects reuse a hasher
core/blockchain, core/state: implement new trie prefetcher
core: make trie prefetcher deliver tries to statedb
core/state: refactor trie_prefetcher, export storage tries
blockchain: re-enable the next-block-prefetcher
state: remove panics in trie prefetcher
core/state/trie_prefetcher: address some review concerns

sq
This fixes #22131 and adds a test reproducing the issue.
This adds a way to get the error of the failing subscription
for logging/debugging purposes.

Co-authored-by: Felix Lange <fjl@twurst.com>
Fixes a special case when the trie only has a single trie node and the range proof only contains a single element.
…a (#22211)

It takes three lines of code to get to runtime.nanotime, no need to
pull a dependency for that.
* cmd, geth: Reflect command being optional - closes 22218

* cmd, geth: Set current year to 2021
* cmd, geth: Reflect command being optional - closes 22218

* cmd, geth: Set current year to 2021
* eth/protocols/snap: make timeout configurable

* eth/protocols/snap: snap sync testing

* eth/protocols/snap: test to trigger panic

* eth/protocols/snap: fix race condition on timeouts

* eth/protocols/snap: return error on cancelled sync

* squashme: updates + test causing panic + properly serve accounts in order

* eth/protocols/snap: revert failing storage response

* eth/protocols/snap: revert on bad responses (storage, code)

* eth/protocols/snap: fix account handling stall

* eth/protocols/snap: fix remaining revertal-issues

* eth/protocols/snap: timeouthandler for bytecode requests

* eth/protocols/snap: debugging + fix log message

* eth/protocols/snap: fix misspelliings in docs

* eth/protocols/snap: fix race in bytecode handling

* eth/protocols/snap: undo deduplication of storage roots

* synctests: refactor + minify panic testcase

* eth/protocols/snap: minor polishes

* eth: minor polishes to make logs more useful

* eth/protocols/snap: remove excessive logs from the test runs

* eth/protocols/snap: stress tests with concurrency

* eth/protocols/snap: further fixes to test cancel channel handling

* eth/protocols/snap: extend test timeouts on CI

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
This updates go module dependencies as discussed in #22050.
This PR fixes the receipt status field to be decimal instead of a hex string,
as called for by the spec.

Co-authored-by: Martin Holst Swende <martin@swende.se>
This updates the goupnp dependency, fixing huin/goupnp#33
When a sethead/rewind finds that the targeted block is missing, it resets to genesis instead of crashing. Closes #22129
This moves the tracing RPC API implementation to package eth/tracers.
By doing so, package eth no longer depends on tracing and the duktape JS engine.

The change also enables tracing using the light client. All tracing methods work with the
light client, but it's a lot slower compared to using a full node.
…2171)

* eth, p2p: reserve half peer slots for snap peers during snap sync

* eth: less logging

* eth: rework the eth/snap peer reservation logic

* eth: rework the eth/snap peer reservation logic (again)
* tests/fuzzers/abi: fixed one-off panic with int.Min64 value

* tests/fuzzers/abi: fixed one-off panic with int.Min64 value
karalabe and others added 26 commits April 8, 2021 13:23
eth, les: drop support for eth/64
* replaces `an chance` with `a chance`
* replaces `SignHashWithPassphrase` with `SignTextWithPassphrase` as there was no SignHashWithPasspharse function in the file
This PR makes it easier to inspect the freezer index, which could be useful to investigate things like #22111
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Felix Lange <fjl@twurst.com>
* eth/protocols: persist received state segments

* core: initial implementation

* core/state/snapshot: add tests

* core, eth: updates

* eth/protocols/snapshot: count flat state size

* core/state: add metrics

* core/state/snapshot: skip unnecessary deletion

* core/state/snapshot: rename

* core/state/snapshot: use the global batch

* core/state/snapshot: add logs and fix wiping

* core/state/snapshot: fix

* core/state/snapshot: save generation progress even if the batch is empty

* core/state/snapshot: fixes

* core/state/snapshot: fix initial account range length

* core/state/snapshot: fix initial account range

* eth/protocols/snap: store flat states during the healing

* eth/protocols/snap: print logs

* core/state/snapshot: refactor (#4)

* core/state/snapshot: refactor

* core/state/snapshot: tiny fix and polish

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>

* core, eth: fixes

* core, eth: fix healing writer

* core, trie, eth: fix paths

* eth/protocols/snap: fix encoding

* eth, core: add debug log

* core/state/generate: release iterator asap (#5)

core/state/snapshot: less copy

core/state/snapshot: revert split loop

core/state/snapshot: handle storage becoming empty, improve test robustness

core/state: test modified codehash

core/state/snapshot: polish

* core/state/snapshot: optimize stats counter

* core, eth: add metric

* core/state/snapshot: update comments

* core/state/snapshot: improve tests

* core/state/snapshot: replace secure trie with standard trie

* core/state/snapshot: wrap return as the struct

* core/state/snapshot: skip wiping correct states

* core/state/snapshot: updates

* core/state/snapshot: fixes

* core/state/snapshot: fix panic due to reference flaw in closure

* core/state/snapshot: fix errors in state generation logic + fix log output

* core/state/snapshot: remove an error case

* core/state/snapshot: fix condition-check for exhausted snap state

* core/state/snapshot: use stackTrie for small tries

* core/state/snapshot: don't resolve small storage tries in vain

* core/state/snapshot: properly clean up storage of deleted accounts

* core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks

* core/state/snapshot: fix error (+testcase)

* core/state/snapshot: clean up tests a bit

* core/state/snapshot: work in progress on better tests

* core/state/snapshot: polish code

* core/state/snapshot: fix trie iteration abortion trigger

* core/state/snapshot: fixes flaws

* core/state/snapshot: remove panic

* core/state/snapshot: fix abort

* core/state/snapshot: more tests (plus failing testcase)

* core/state/snapshot: more testcases + fix for failing test

* core/state/snapshot: testcase for malformed data

* core/state/snapshot: some test nitpicks

* core/state/snapshot: improvements to logging

* core/state/snapshot: testcase to demo error in abortion

* core/state/snapshot: fix abortion

* cmd/geth: make verify-state report the root

* trie: fix failing test

* core/state/snapshot: add timer metrics

* core/state/snapshot: fix metrics

* core/state/snapshot: udpate tests

* eth/protocols/snap: write snapshot account even if code or state is needed

* core/state/snapshot: fix diskmore check

* core/state/snapshot: review fixes

* core/state/snapshot: improve error message

* cmd/geth: rename 'error' to 'err' in logs

* core/state/snapshot: fix some review concerns

* core/state/snapshot, eth/protocols/snap: clear snapshot marker when starting/resuming snap sync

* core: add error log

* core/state/snapshot: use proper timers for metrics collection

* core/state/snapshot: address some review concerns

* eth/protocols/snap: improved log message

* eth/protocols/snap: fix heal logs to condense infos

* core/state/snapshot: wait for generator termination before restarting

* core/state/snapshot: revert timers to counters to track total time

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>
core/types: drop some relice data types
* all: add thousandths separators for big numbers on log messages

* p2p/sentry: drop accidental file

* common, log: add fast number formatter

* common, eth/protocols/snap: simplifty fancy num types

* log: handle nil big ints
This adds simple unit test checking if the hard-coded genesis
hash values in package params match the actual genesis block hashes.
* log: fix formatting of big.Int

The implementation of formatLogfmtBigInt had two issues: it crashed when
the number was actually large enough to hit the big integer case, and
modified the big.Int while formatting it.

* log: don't call FormatLogfmtInt64 for int16

* log: separate from decimals back, not front

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
eth/protocols/snap: use ephemeral channels to avoid cross-sync delveries
…650)

* les/vflux/server: fix estimatePriority corner case

* les/vflux/server: simplify inactiveAllowance == 0 case
The stacktrie is a bit un-untuitive, API-wise: since it mutates input values.
Such behaviour is dangerous, and easy to get wrong if the calling code 'forgets' this quirk. The behaviour is fixed by this PR, so that the input values are not modified by the stacktrie. 

Note: just as with the Trie, the stacktrie still references the live input objects, so it's still _not_ safe to mutate the values form the callsite.
This change adds the --catalyst flag, enabling an RPC API for eth2 integration.
In this initial version, catalyst mode also disables all peer-to-peer networking.

Co-authored-by: Mikhail Kalinin <noblesse.knight@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
core/state/snapshot: avoid copybytes for stacktrie
…#22694)

The new -limit option makes the filter operate on top N nodes by score.
This also adds ENR attribute stats in the nodeset info command.
Node set commands are now documented in README.
@CLAassistant
Copy link

CLAassistant commented Apr 19, 2021

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 20 committers have signed the CLA.

✅ renaynay
✅ rauljordan
❌ holiman
❌ karalabe
❌ rjl493456442
❌ MariusVanDerWijden
❌ fjl
❌ nebojsa94
❌ piersy
❌ fxfactorial
❌ AmitBRD
❌ petesimard
❌ zsfelfoldi
❌ s1na
❌ balajipachai
❌ Evolution404
❌ r1cs
❌ gballet
❌ maxsam4
❌ meowsbits
You have signed the CLA already but the status is still pending? Let us recheck it.

@rauljordan rauljordan merged commit eb5298e into master Apr 19, 2021
@delete-merged-branch delete-merged-branch bot deleted the sync-geth branch April 19, 2021 16:52
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.