Skip to content

Commit

Permalink
Merge pull request #4264 from onflow/tarak/blst-merge-master
Browse files Browse the repository at this point in the history
[Crypto] merge master to BLST feature branch
  • Loading branch information
tarakby authored May 3, 2023
2 parents 2070aa6 + bbd3c74 commit 633f337
Show file tree
Hide file tree
Showing 1,065 changed files with 42,416 additions and 19,855 deletions.
5 changes: 5 additions & 0 deletions .github/workflows/bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ jobs:
benchstat:
name: Performance regression check
runs-on: ubuntu-latest
# Check if the event is not triggered by a fork
# peter-evans/find-comment@v1 does not work on forks.
# see https://github.com/peter-evans/create-pull-request/blob/main/docs/concepts-guidelines.md#restrictions-on-repository-forks for details.
# Ideally we would like to still run the benchmark on forks, but we can't do that with the current setup.
if: github.event.pull_request.head.repo.full_name == github.repository
continue-on-error: true
steps:
- name: Set benchmark repetitions
Expand Down
118 changes: 118 additions & 0 deletions .github/workflows/builds.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# This workflow is used to build and push one-off images for specific node types. This is useful
# when deploying hotfixes or any time a change is not needed for all node roles.
name: Build Node Docker Images

on:
workflow_dispatch:
inputs:
tag:
type: string
description: 'Git tag/commit'
required: true
docker_tag:
type: string
description: 'Docker tag'
required: true
# GHA doesn't support multi-selects, so simulating it with one boolean for each option
build_access:
type: boolean
description: 'Access'
required: false
build_collection:
type: boolean
description: 'Collection'
required: false
build_consensus:
type: boolean
description: 'Consensus'
required: false
build_execution:
type: boolean
description: 'Execution'
required: false
build_verification:
type: boolean
description: 'Verification'
required: false
build_observer:
type: boolean
description: 'Observer'
required: false
include_without_netgo:
type: boolean
description: 'Build `without_netgo` images'
required: false

jobs:
# matrix_builder generates a matrix that includes the roles selected in the input
matrix_builder:
name: Setup build jobs
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.generate.outputs.matrix }}
steps:
- id: generate
run: |
roles=()
if [[ "${{ inputs.build_access }}" = "true" ]]; then
roles+=( "access" )
fi
if [[ "${{ inputs.build_collection }}" = "true" ]]; then
roles+=( "collection" )
fi
if [[ "${{ inputs.build_consensus }}" = "true" ]]; then
roles+=( "consensus" )
fi
if [[ "${{ inputs.build_execution }}" = "true" ]]; then
roles+=( "execution" )
fi
if [[ "${{ inputs.build_verification }}" = "true" ]]; then
roles+=( "verification" )
fi
if [[ "${{ inputs.build_observer }}" = "true" ]]; then
roles+=( "observer" )
fi
rolesJSON=$(jq --compact-output --null-input '$ARGS.positional' --args -- "${roles[@]}")
echo "matrix={\"role\":$(echo $rolesJSON)}" >> $GITHUB_OUTPUT
docker-push:
name: ${{ matrix.role }} images
runs-on: ubuntu-latest
needs: matrix_builder

# setup jobs for each role
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.matrix_builder.outputs.matrix) }}

steps:
- name: Setup Go
uses: actions/setup-go@v2
with:
go-version: '1.19'
- name: Checkout repo
uses: actions/checkout@v2
with:
ref: ${{ inputs.tag }}
# Provide Google Service Account credentials to Github Action, allowing interaction with the Google Container Registry
# Logging in as github-actions@dl-flow.iam.gserviceaccount.com
- name: Docker login
uses: docker/login-action@v1
with:
registry: gcr.io
username: _json_key
password: ${{ secrets.GCR_SERVICE_KEY }}

- name: Build/Push ${{ matrix.role }} images
env:
IMAGE_TAG: ${{ inputs.docker_tag }}
GITHUB_CREDS: "machine github.com login ${{ secrets.REPO_SYNC_USER }} password ${{ secrets.REPO_SYNC }}"
run: |
make docker-build-${{ matrix.role }} docker-push-${{ matrix.role }}
- name: Build/Push ${{ matrix.role }} without_netgo images
if: ${{ inputs.include_without_netgo }}
env:
IMAGE_TAG: ${{ inputs.docker_tag }}
GITHUB_CREDS: "machine github.com login ${{ secrets.REPO_SYNC_USER }} password ${{ secrets.REPO_SYNC }}"
run: |
make docker-build-${{ matrix.role }}-without-netgo docker-push-${{ matrix.role }}-without-netgo
39 changes: 12 additions & 27 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
- 'v[0-9]+.[0-9]+'
pull_request:
branches:
- master
- master*
- 'auto-cadence-upgrade/**'
- 'feature/**'
- 'v[0-9]+.[0-9]+'
Expand Down Expand Up @@ -115,24 +115,18 @@ jobs:
with:
go-version: ${{ env.GO_VERSION }}
cache: true
- name: Setup tests (${{ matrix.targets.name }}
run: VERBOSE=1 make -e GO_TEST_PACKAGES="${{ matrix.targets.packages }}" install-tools
- name: Run tests (${{ matrix.targets.name }})
if: github.actor != 'bors[bot]'
uses: nick-fields/retry@v2
with:
timeout_minutes: 25
max_attempts: 3
command: VERBOSE=1 make -e GO_TEST_PACKAGES="${{ matrix.targets.packages }}" ci
command: VERBOSE=1 make -e GO_TEST_PACKAGES="${{ matrix.targets.packages }}" test

# TODO(rbtz): re-enable when we fix exisiting races.
#env:
# RACE_DETECTOR: 1
- name: Run tests (Bors)
if: github.actor == 'bors[bot]'
uses: nick-fields/retry@v2
with:
timeout_minutes: 25
max_attempts: 3
command: VERBOSE=1 make -e GO_TEST_PACKAGES="${{ matrix.targets.packages }}" ci
- name: Upload coverage report
uses: codecov/codecov-action@v3
with:
Expand Down Expand Up @@ -170,23 +164,17 @@ jobs:
with:
go-version: ${{ env.GO_VERSION }}
cache: true
- name: Setup tests (${{ matrix.name }})
run: make ${{ matrix.make1 }}
- name: Run tests (${{ matrix.name }})
if: github.actor != 'bors[bot]'
env:
RACE_DETECTOR: ${{ matrix.race }}
# run `make1` target before running `make2` target inside each module's root
run: |
make ${{ matrix.make1 }}
VERBOSE=1 make -C ${{ matrix.name }} ${{ matrix.make2 }}
- name: Run tests (Bors)
if: github.actor == 'bors[bot]'
uses: nick-fields/retry@v2
with:
timeout_minutes: 25
max_attempts: ${{ matrix.retries }}
command: |
make ${{ matrix.make1 }}
VERBOSE=1 make -C ${{ matrix.name }} ${{ matrix.make2 }}
# run `make2` target inside each module's root
command: VERBOSE=1 make -C ${{ matrix.name }} ${{ matrix.make2 }}
- name: Upload coverage report
uses: codecov/codecov-action@v3
with:
Expand All @@ -210,6 +198,7 @@ jobs:
- make -C integration mvp-tests
- make -C integration network-tests
- make -C integration verification-tests
- make -C integration upgrades-tests
runs-on: ubuntu-latest
steps:
- name: Checkout repo
Expand All @@ -224,18 +213,14 @@ jobs:
- name: Docker build
run: make docker-build-flow docker-build-flow-corrupt
- name: Run tests
if: github.actor != 'bors[bot]'
run: VERBOSE=1 ${{ matrix.make }}
# TODO(rbtz): re-enable when we fix exisiting races.
#env:
# RACE_DETECTOR: 1
- name: Run tests (Bors)
if: github.actor == 'bors[bot]'
uses: nick-fields/retry@v2
with:
timeout_minutes: 15
max_attempts: 2
command: ${{ matrix.make }}
timeout_minutes: 25
max_attempts: 3
command: VERBOSE=1 ${{ matrix.make }}

localnet-test:
name: Localnet Compatibility Tests With Flow-CLI Client and Observer
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/test-monitor-flaky.yml
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ jobs:
TEST_FLAKY: true
JSON_OUTPUT: true
RACE_DETECTOR: 1
- name: Print test results
run: cat test-output
- name: Process test results
run: cat test-output | go run tools/test_monitor/level1/process_summary1_results.go
env:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/test-monitor-regular-skipped.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ jobs:
command: ./tools/test_monitor/run-tests.sh
env:
JSON_OUTPUT: true
- name: Print test results
run: cat test-output
- name: Process test results
run: cat test-output | go run tools/test_monitor/level1/process_summary1_results.go
env:
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/tools.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
# This workflow is used to build and upload the node bootstrapping tools
name: Build Tools

on:
workflow_dispatch:
inputs:
tag:
description: 'Tagged commit to build tools against'
description: 'Tag/commit'
required: true
type: string
promote:
description: 'Should this build be promoted to the official boot-tools?'
description: 'Promote to official boot-tools?'
required: false
type: boolean

Expand Down
6 changes: 6 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,9 @@ issues:
- path: 'cmd/access/node_build/*'
linters:
- typecheck
- path: 'cmd/observer/node_builder/*'
linters:
- typecheck
- path: 'follower/*'
linters:
- typecheck
20 changes: 13 additions & 7 deletions CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,14 @@
/engine/consensus/** @AlexHentschel @durkmurder @jordanschalm

# Execution Stream
/cmd/execution/** @m4ksio @ramtinms
/engine/execution/** @m4ksio @ramtinms
/cmd/execution/** @ramtinms
/engine/execution/** @ramtinms

# Access Stream
/cmd/access/** @vishalchangrani
/engine/access/** @vishalchangrani
/access/** @peterargue
/cmd/access/** @peterargue
/cmd/observer/** @peterargue
/engine/access/** @peterargue

# Verification Stream
/cmd/verification/** @ramtinms @yhassanzadeh13
Expand All @@ -22,19 +24,19 @@
/integration/tests/verification @ramtinms @yhassanzadeh13

# Ledger Stream
/ledger/** @ramtinms @m4ksio @AlexHentschel
/ledger/** @ramtinms @AlexHentschel

# FVM Stream
/fvm/** @ramtinms @janezpodhostnik @pattyshack

# Networking Stream
/network/** @yhassanzadeh13 @vishalchangrani
/network/** @yhassanzadeh13

# Cryptography Stream
/crypto/** @tarakby

# Bootstrap and transit scripts
/cmd/bootstrap/** @vishalchangrani
/cmd/bootstrap/** @zhangchiqing

# Dev Tools Stream
.github/workflows/** @gomisha
Expand All @@ -48,3 +50,7 @@
/module/profiler/** @SaveTheRbtz @pattyshack
/module/trace/** @SaveTheRbtz @pattyshack
/module/tracer.go @SaveTheRbtz @pattyshack

# Execution Sync
/module/executiondatasync/** @peterargue
/module/state_synchronization/** @peterargue
5 changes: 3 additions & 2 deletions CodingConventions.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ where we treat everything beyond the known benign errors as critical failures. I
broken and proper functioning is no longer guaranteed. The only safe route of recovery is to restart the vertex from a previously persisted, safe state.
Per convention, a vertex should throw any unexpected exceptions using the related [irrecoverable context](https://github.com/onflow/flow-go/blob/277b6515add6136946913747efebd508f0419a25/module/irrecoverable/irrecoverable.go).
* Many components in our BFT system can return benign errors (type (i)) and exceptions (type (ii))
* Use the special `irrecoverable.exception` [error type](https://github.com/onflow/flow-go/blob/master/module/irrecoverable/exception.go#L7-L26) to denote an unexpected error (and strip any sentinel information from the error stack)


3. _Optional Simplification for components that solely return benign errors._
Expand All @@ -123,7 +124,7 @@ For example, a statement like the following would be sufficient:

### Hands-on suggestions

* avoid generic errors, such as
* Avoid generic errors, such as
```golang
return fmt.Errorf("x failed")
```
Expand Down Expand Up @@ -159,7 +160,7 @@ For example, a statement like the following would be sufficient:
* Handle errors at a level, where you still have enough context to decide whether the error is expected during normal
operations.
* Errors of unexpected types are indicators that the node's internal state might be corrupted.

- Use the special `irrecoverable.exception` [error type](https://github.com/onflow/flow-go/blob/master/module/irrecoverable/exception.go#L7-L26) at the point an unexpected error is being returned, or when an error returned from another function is interpreted as unexpected
### Anti-Pattern

Continuing on a best-effort basis is not an option, i.e. the following is an anti-pattern in the context of Flow:
Expand Down
Loading

0 comments on commit 633f337

Please sign in to comment.