Skip to content

Commit

Permalink
Merge pull request #594 from blxdyx/upstream_85844e6
Browse files Browse the repository at this point in the history
  • Loading branch information
setunapo authored Jan 16, 2025
2 parents a53da48 + b13159b commit 78ee709
Show file tree
Hide file tree
Showing 299 changed files with 17,498 additions and 5,045 deletions.
112 changes: 112 additions & 0 deletions .github/workflows/backups-dashboards.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Author: Michele@DevOpsTeam.Erigon
# Maintainers: DevOps@DevOpsTeam.Erigon
# Description: This workflow is responsible for backing up the dashboards of the Erigon project.
# Status: Production (further improvements are planned though)

name: Dashboards backup

env:
GREEN: '\033[0;32m'
RED: '\033[0;31m'
YELLOW: '\033[0;33m'
NOCOLOUR: '\033[0m'
ERIGON_BRANCH: main

on:
workflow_dispatch:
inputs:
TEMPLATE_BRANCH:
required: false
type: string
default: main
description: 'The branch to pull the backup script from (default: main)'

jobs:
#
# NOTE: This workflow splits the backup process in 2 jobs to spot any pulling issue early on
#
preparation:
runs-on: ubuntu-latest
environment: dashboards_backups
steps:
- name: Pull backup script from ${{ inputs.TEMPLATE_BRANCH}} branch
run: |
set +x
curl -L -H "Authorization: Bearer ${{ secrets.GH_TOKEN }}" -H "Accept: application/vnd.github.v3.raw" -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/repos/erigontech/scripts/contents/dashboards/dashboard-backup.sh?ref=${{ inputs.TEMPLATE_BRANCH}} -o /tmp/dashboard-backup
- name: Upload dashboard-backup
uses: actions/upload-artifact@v3
with:
name: dashboard-backup
path: /tmp/dashboard-backup
retention-days: 1


backup_dashboard:
needs: preparation
runs-on: ubuntu-latest
environment: dashboards_backups
strategy:
matrix:
# For each dashboard add an entry in the list here below (and add an entry in the dictionary within the backup script too!)
dashboard: [erigon_custom_metrics]
env:
DASHBOARDS_AUTH_TOKEN: ${{ secrets.DASHBOARDS_AUTH_TOKEN }}
GH_TOKEN: ${{ secrets.GH_TOKEN }}
DASHBOARDS_GIT_CONFIG: ${{ secrets.DASHBOARDS_GIT_CONFIG }}
steps:
- name: Clone erigon from ${{ env.ERIGON_BRANCH }} branch
uses: actions/checkout@v4
with:
ref: ${{ env.ERIGON_BRANCH }}
fetch-depth: 1
path: ${{ github.workspace }}/erigon

- name: Move repository to $HOME # as the backup script expects it to be there
run: mv ${{ github.workspace }}/erigon $HOME/

- name: Set-up git
run: |
echo ${{ secrets.DASHBOARDS_GIT_CONFIG }} | base64 -d > $HOME/.gitconfig
- name: Download dashboard-backup
uses: actions/download-artifact@v3
with:
name: dashboard-backup
path: /tmp

- name: Set dashboard-backup permissions
run: chmod +x /tmp/dashboard-backup

- name: BACKUP OF ${{ matrix.dashboard }} DASHBOARD
run: |
set +x
echo -e "${{ env.GREEN }} I'm processing the ${{ matrix.dashboard }} dashboard... ${{ env.NOCOLOUR }}"
/tmp/dashboard-backup ${{ matrix.dashboard }}

housekeeping:
needs: [preparation, backup_dashboard]
runs-on: ubuntu-latest
if: always()
steps:
- name: cleaning up
uses: geekyeggo/delete-artifact@v2
with:
name: |
dashboard-backup
- name: Bye ${{ github.actor }}
if: always()
run: |
set +x
if [ "${{ needs.preparation.result }}" == "success" ] && [ "${{ needs.backup_dashboard.result }}" == "success" ]; then
echo -e "${{ env.YELLOW }} --------------------------------------------------------------------------------------------------------------------- ${{ env.NOCOLOUR }}"
echo -e "${{ env.YELLOW }} *** It was a true pleasure to serve you ${{ github.actor }}, SEE YOU NEXT TIME! ${{ env.NOCOLOUR }}"
echo -e "${{ env.YELLOW }} --------------------------------------------------------------------------------------------------------------------- ${{ env.NOCOLOUR }}"
else
echo -e "${{ env.RED }} --------------------------------------------------------------------------------------------------------------------- ${{ env.RED }}"
echo -e "${{ env.RED }} *** Sorry ${{ github.actor }}, it seems there was an error this time: just fix it and COME BACK! ${{ env.RED }}"
echo -e "${{ env.RED }} --------------------------------------------------------------------------------------------------------------------- ${{ env.RED }}"
fi
53 changes: 26 additions & 27 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,13 @@ jobs:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
continue-on-error: true

# tests-windows:
# if: ${{ github.event_name == 'push' || !github.event.pull_request.draft }}
# strategy:
# matrix:
# os: [ windows-2022 ]
# runs-on: ${{ matrix.os }}

# tests-windows:
# if: ${{ github.event_name == 'push' || !github.event.pull_request.draft }}
# strategy:
# matrix:
# os: [ windows-2022 ]
# runs-on: ${{ matrix.os }}
#
# steps:
# - name: configure Pagefile
# uses: al-cheb/configure-pagefile-action@v1.4
Expand All @@ -96,23 +96,22 @@ jobs:
# "refs/heads/release/2.61",
# "refs/heads/main"
# ]'), github.ref) }}

# - uses: actions/cache@v4
# with:
# path: |
# C:\ProgramData\chocolatey\lib\mingw
# C:\ProgramData\chocolatey\lib\cmake
# key: chocolatey-${{ matrix.os }}
# - name: Install dependencies
# run: |
# choco upgrade mingw -y --no-progress --version 13.2.0
# choco install cmake -y --no-progress --version 3.27.8

# - name: Build
# run: .\wmake.ps1 all

# - name: Test
# run: .\wmake.ps1 test

# - name: Test erigon-lib
# run: cd erigon-lib && make test-no-fuzz
#
# - uses: actions/cache@v4
# with:
# path: |
# C:\ProgramData\chocolatey\lib\mingw
# C:\ProgramData\chocolatey\lib\cmake
# key: chocolatey-${{ matrix.os }}
# - name: Install dependencies
# run: |
# choco upgrade mingw -y --no-progress --version 13.2.0
#
# - name: Build
# run: .\wmake.ps1 all
#
# - name: Test
# run: .\wmake.ps1 test
#
# - name: Test erigon-lib
# run: cd erigon-lib && make test-no-fuzz
2 changes: 1 addition & 1 deletion .github/workflows/qa-rpc-integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
- name: Checkout RPC Tests Repository & Install Requirements
run: |
rm -rf ${{ runner.workspace }}/rpc-tests
git -c advice.detachedHead=false clone --depth 1 --branch v1.26.0 https://github.com/erigontech/rpc-tests ${{runner.workspace}}/rpc-tests
git -c advice.detachedHead=false clone --depth 1 --branch v1.27.0 https://github.com/erigontech/rpc-tests ${{runner.workspace}}/rpc-tests
cd ${{ runner.workspace }}/rpc-tests
pip3 install -r requirements.txt
Expand Down
18 changes: 15 additions & 3 deletions .github/workflows/qa-tip-tracking-polygon.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,21 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: test-results
path: |
${{ github.workspace }}/result-${{ env.CHAIN }}.json
${{ env.ERIGON_TESTBED_DATA_DIR }}/logs/erigon.log
path: ${{ github.workspace }}/result-${{ env.CHAIN }}.json

- name: Upload erigon log
if: steps.test_step.outputs.test_executed == 'true'
uses: actions/upload-artifact@v4
with:
name: erigon-log
path: ${{ env.ERIGON_TESTBED_DATA_DIR }}/logs/erigon.log

- name: Upload metric plots
if: steps.test_step.outputs.test_executed == 'true'
uses: actions/upload-artifact@v4
with:
name: metric-plots
path: ${{ github.workspace }}/metrics-${{ env.CHAIN }}-plots*

- name: Delete Erigon Testbed Data Directory
if: always()
Expand Down
18 changes: 15 additions & 3 deletions .github/workflows/qa-tip-tracking.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,21 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: test-results
path: |
${{ github.workspace }}/result-${{ env.CHAIN }}.json
${{ env.ERIGON_TESTBED_DATA_DIR }}/logs/erigon.log
path: ${{ github.workspace }}/result-${{ env.CHAIN }}.json

- name: Upload erigon log
if: steps.test_step.outputs.test_executed == 'true'
uses: actions/upload-artifact@v4
with:
name: erigon-log
path: ${{ env.ERIGON_TESTBED_DATA_DIR }}/logs/erigon.log

- name: Upload metric plots
if: steps.test_step.outputs.test_executed == 'true'
uses: actions/upload-artifact@v4
with:
name: metric-plots
path: ${{ github.workspace }}/metrics-${{ env.CHAIN }}-plots*

- name: Delete Erigon Testbed Data Directory
if: always()
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/scripts/run_rpc_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ set +e # Disable exit on error
disabled_tests=(
# Erigon2 and Erigon3 never supported this api methods
trace_rawTransaction
# Erigon bug https://github.com/erigontech/erigon/issues/12603
erigon_getLatestLogs
# to investigate
engine_exchangeCapabilities/test_1.json
engine_exchangeTransitionConfigurationV1/test_01.json
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/test-integration-caplin.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@ jobs:
- name: Install dependencies
run: |
choco upgrade mingw -y --no-progress --version 13.2.0
choco install cmake -y --no-progress --version 3.27.8
- name: test-integration-caplin
run: cd ./cl/spectest/ && .\wmake.ps1 Tests Mainnet
1 change: 0 additions & 1 deletion .github/workflows/test-integration-erigon.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@ jobs:
- name: Install dependencies
run: |
choco upgrade mingw -y --no-progress --version 13.2.0
choco install cmake -y --no-progress --version 3.27.8
- name: test-integration
run: .\wmake.ps1 test-integration
Expand Down
2 changes: 2 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ linters-settings:
- fieldalignment
- shadow
- unsafeptr
enable:
- nilness
goconst:
min-len: 2
min-occurrences: 2
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ GOPRIVATE = github.com/erigontech/silkworm-go

PACKAGE = github.com/erigontech/erigon

GO_FLAGS += -trimpath -tags $(BUILD_TAGS) -buildvcs=false
GO_FLAGS += -trimpath -tags $(BUILD_TAGS) -buildvcs=false
GO_FLAGS += -ldflags "-X ${PACKAGE}/params.GitCommit=${GIT_COMMIT} -X ${PACKAGE}/params.GitBranch=${GIT_BRANCH} -X ${PACKAGE}/params.GitTag=${GIT_TAG}"

GOBUILD = ${CPU_ARCH} CGO_CFLAGS="$(CGO_CFLAGS)" CGO_LDFLAGS="$(CGO_LDFLAGS)" GOPRIVATE="$(GOPRIVATE)" $(GO) build $(GO_FLAGS)
Expand Down
21 changes: 17 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Erigon

Documentation: **[erigon.gitbook.io](https://erigon.gitbook.io)**
Documentation: **[docs.erigon.tech](https://docs.erigon.tech)**
Blog: **[erigon.substack.com](https://erigon.substack.com/)**
Twitter: [x.com/ErigonEth](https://x.com/ErigonEth)

Expand All @@ -15,6 +15,7 @@ frontier.

- [Erigon](#erigon)
- [System Requirements](#system-requirements)
- [Sync Times](#sync-times)
- [Usage](#usage)
- [Getting Started](#getting-started)
- [Datadir structure](#datadir-structure)
Expand Down Expand Up @@ -97,6 +98,18 @@ on [cloud-network-drives](https://github.com/erigontech/erigon?tab=readme-ov-fil

🔬 More details on what type of data stored [here](https://ledgerwatch.github.io/turbo_geth_release.html#Disk-space)

Sync Times
==========

These are the approximate sync times syncing from scratch to the tip of the chain (results may vary depending on hardware and bandwidth).


| Chain | Archive | Full | Minimal |
|------------|-----------------|----------------|----------------|
| Ethereum | 7 Hours, 55 Minutes | 4 Hours, 23 Minutes | 1 Hour, 41 Minutes |
| Gnosis | 2 Hours, 10 Minutes | 1 Hour, 5 Minutes | 33 Minutes |
| Polygon | 1 Day, 21 Hours | 21 Hours, 41 Minutes | 11 Hours, 54 Minutes |

Usage
=====

Expand Down Expand Up @@ -134,7 +147,7 @@ datadir
domain # Latest State
history # Historical values
idx # InvertedIndices: can search/filtering/union/intersect them - to find historical data. like eth_getLogs or trace_transaction
accessors # Additional (generated) indices of history - have "random-touch" read-pattern. They can serve only `Get` requests (no search/filters).
accessor # Additional (generated) indices of history - have "random-touch" read-pattern. They can serve only `Get` requests (no search/filters).
txpool # pending transactions. safe to remove.
nodes # p2p peers. safe to remove.
temp # used to sort data bigger than RAM. can grow to ~100gb. cleaned at startup.
Expand All @@ -155,11 +168,11 @@ datadir
domain # link to fast disk
history
idx
accessors
accessor
temp # buffers to sort data >> RAM. sequential-buffered IO - is slow-disk-friendly

# Example: how to speedup history access:
# - go step-by-step - first try store `accessors` on fast disk
# - go step-by-step - first try store `accessor` on fast disk
# - if speed is not good enough: `idx`
# - if still not enough: `history`
```
Expand Down
8 changes: 8 additions & 0 deletions RELEASE_INSTRUCTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,11 @@ In most cases, it is enough to bump minor version.
In the file `ethdb/remote/remotedbserver/server.go` there is variable `KvServiceAPIVersion` that needs to be updated if there are any changes in the remote KV interface, or
database schema, leading to data migrations.
In most cases, it is enough to bump minor version. It is best to change both DB schema version and remove KV version together.

## Purify the state domains if a regenration is done

If a regenration is done, the state domains need to be purified. This can be done by running the following command:
````
make integration
./build/bin/integration purify_domains --datadir=<path to datadir> --replace-in-datadir
````
Loading

0 comments on commit 78ee709

Please sign in to comment.