Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLST-based crypto package #4358

Merged
merged 238 commits into from
Dec 6, 2023
Merged
Show file tree
Hide file tree
Changes from 234 commits
Commits
Show all changes
238 commits
Select commit Hold shift + click to select a range
dfe442c
add c and asm srouce files from blst
Jan 18, 2023
635f960
add blst header files and README
Jan 18, 2023
abe0fe1
cgo directives to compile blst files
Jan 18, 2023
59be524
update README and clean up C directives
Jan 24, 2023
56b4b56
remove non code files
Jan 25, 2023
3c5accd
disable c99 and compile blst src and assembly
Feb 14, 2023
3c0247c
include blst.h
Feb 14, 2023
79601b6
tidy go.mod after removing blst package
Feb 14, 2023
e64cc36
add missing relic flags
Feb 14, 2023
e51d94e
first iteration of scalar type change
Feb 17, 2023
7593960
merge feature branch
Feb 17, 2023
0a066ec
merge keygen updates
Feb 17, 2023
b99d75b
use new type Fr in BLS simple sig
Feb 19, 2023
e3f4fee
implement BLS threshold signature with new Fr type
Feb 25, 2023
7163fc1
Merge pull request #3851 from onflow/tarak/add-blst-src
tarakby Feb 27, 2023
8b49d0a
add Fermat inversion - Lagrange interpolation works with Euclidean in…
Feb 27, 2023
1b3293f
DKG feldmanVSS with Fr type
Feb 27, 2023
2bbea26
enable all DKG protocols to work with new Fr type
Feb 28, 2023
b113d33
uncomment tests
Mar 3, 2023
7a2617c
renaming and linter errors
Mar 3, 2023
d7f3d5d
fix gcc compilation issue and remove blst.h
Mar 3, 2023
4979760
fix double definition
Mar 3, 2023
90c412f
enable tmate on ci temporarily
Mar 6, 2023
d96d6fa
test improvement and temporary memory free
Mar 6, 2023
0593da5
fix memory allocation bug in temp function
Mar 6, 2023
0e8829e
Revert "enable tmate on ci temporarily"
Mar 6, 2023
b2ae93c
Merge pull request #3993 from onflow/tarak/2-blst-types
tarakby Mar 6, 2023
52cae3e
g1 and g2 exportable types to cgo
Mar 8, 2023
e783a42
new Fp and Fp2 tools, but still the mess
Apr 6, 2023
f025a01
first changes to use new type G2
Apr 11, 2023
a5738f6
G2 type working for BLS
Apr 13, 2023
25b7be6
integrate G2 in BLS multi-sig
Apr 14, 2023
444f755
update BLSBatchVerify with regards to invalid signature format
Apr 15, 2023
d70883b
fix a bug and minor updates
Apr 15, 2023
54ee84a
BLS threshold signature works with new G2 type
Apr 16, 2023
ab67ed4
DKG works with new G2 type
Apr 16, 2023
693029f
BLS-SPoCK works with new G2 type
Apr 16, 2023
1a93763
clean up and G2 to E2 renaming
Apr 17, 2023
5cbfd08
rename some G1/G2 functions to E1/E2
Apr 17, 2023
8595b05
Merge pull request #4230 from onflow/tarak/g2-blst-type
tarakby Apr 17, 2023
2070aa6
Merge branch 'feature/blst-based-crypto' of github.com:onflow/flow-go…
Apr 18, 2023
f104505
various renaming in DKG functions
Apr 18, 2023
6f044c2
renaming and add Fr_generate_poly function
Apr 18, 2023
acb053f
update C polynomial headers to use degree
Apr 19, 2023
fdf18be
use pseudo-random randFr, randFrStar and FrPolynomial
Apr 19, 2023
3d26f0c
unify seed lengths of DKG and other keyGen seed lengths
Apr 19, 2023
768602c
update randG2 to map to G2 and update membership check in G2 tests
Apr 20, 2023
2167684
membership check in G2 using BLST
Apr 20, 2023
2d90a70
update batch verify random coefficients
Apr 21, 2023
183627b
minor cleanup
Apr 21, 2023
c033600
fix linter
Apr 21, 2023
558c149
merge master into feature branch
Apr 24, 2023
0a2e943
add replaces to test using latest crypto package
Apr 24, 2023
7bd182a
temp update to makefile to setup crypto with replace statement
Apr 25, 2023
9ae8df2
mod tidy
Apr 25, 2023
2fb816c
enable membership check in G2 to fix FVM test
Apr 25, 2023
54b92e6
fix E2 infinity set and check to be based on projective Z
Apr 26, 2023
f387ce9
fix warning
Apr 26, 2023
cbe51a3
disable thread SAN
Apr 26, 2023
be727d8
add SIGILL handler
Apr 26, 2023
6bd85a7
fix blst_cgo_init
Apr 26, 2023
76fcc73
disable ADX instructions in BlST by default as a temp measure
Apr 27, 2023
e07178d
merge latest master changes
Apr 27, 2023
2a851b5
uncomment DKG tests
Apr 28, 2023
26e5636
more logging in FeldmanVSSQ when shares aren't matching computed keys…
Apr 29, 2023
c590019
merge main feature branch
May 1, 2023
5e01e46
use new KeyGenSeedMinLen
May 1, 2023
bbd3c74
fix a bug when zeroring a buffer that is not a multiple of 8 bytes wi…
May 2, 2023
633f337
Merge pull request #4264 from onflow/tarak/blst-merge-master
tarakby May 3, 2023
187aeb3
Merge branch 'feature/blst-based-crypto' into tarak/replace-relic-rand
May 3, 2023
d6b4e86
clean ups
May 3, 2023
9f20a59
clean up encode/decode tests in BLS
May 3, 2023
f2731a8
add endianness test for maptoFr
May 3, 2023
70c3c64
add endianness comment
May 3, 2023
c54e2aa
Merge pull request #4253 from onflow/tarak/replace-relic-rand
tarakby May 3, 2023
21c4686
add blst Go package as an internal test package
May 8, 2023
98bb9cc
clarify internal blst package structure in comment
May 8, 2023
d729688
update bls cross-checks
May 8, 2023
ab2fa14
upgrade rapid package
May 8, 2023
9d35d3d
fix linter false positives
May 9, 2023
6031d71
go mod tidy
May 9, 2023
6b5fd8f
Revert "upgrade rapid package"
May 9, 2023
e35860f
Revert "go mod tidy"
May 9, 2023
22b1ef8
fix linter error
May 9, 2023
f66205c
Merge pull request #4328 from onflow/blst_crossblst_test
tarakby May 9, 2023
0290a98
clean up of multiple pairing
May 9, 2023
7f86c94
Fp12 and multi-pairing computation
May 11, 2023
7b7e484
use E1 blst type in Go
May 12, 2023
7cafb2b
use POINTonEx_mult_glv instead of blst_sign and remove Relic's E1 mult
May 12, 2023
109b238
E1_read_bytes and E1_write_bytes and their tools
May 12, 2023
5ae1abb
G1 membership check and connect E1 read/write to the Go layer
May 12, 2023
8d32cff
map to G1 using BLST and add length sanity check of input hash
May 12, 2023
9efaddb
remove local SSWU implementation and rely only on BLST map to G1 tools
May 12, 2023
56081df
clean up membership check macros and delete Bowe's check code
May 12, 2023
13025e1
refactor bls_verify_ep to use hashed point
May 12, 2023
8885e5d
add temporary E1_blst_to_relic and use E1 in all bls_core functions -…
May 12, 2023
9e938a9
implement mapping to Fp to use in map_to_G1
May 12, 2023
f10d881
fix E1_read_bytes bug and improve debug printing
May 13, 2023
d3396c6
update BLS threshold signature with E1 points
May 15, 2023
557d3a7
uncomment BLST cross check tests
May 15, 2023
878b7e7
spock works with new E1 type
May 15, 2023
49b01a8
write E1_sum_vector using new E1 type
May 15, 2023
534c0a8
clean up unsecure mapping to G1/G1 and fix subgroup checks
May 15, 2023
c338c93
clean up constants pre-computation
May 15, 2023
3fd0539
add read/write tests for G1 points
May 15, 2023
29f7489
rename insecure to unsafe
May 16, 2023
c47c321
fix node info comparison bug in test
May 16, 2023
184a49d
fix public key comparison bugs in tests
May 16, 2023
a4fb435
yet another key comparison bug in tests
May 16, 2023
ff5a0c7
another bug: compare ER bassed on IDs and not the in-mem struct
May 16, 2023
eae8458
Merge pull request #4355 from onflow/tarak/blst-E1-G1
tarakby May 16, 2023
22db5b0
multi-pairing function used in BLS single signatture verify
May 23, 2023
ee91d4f
use BLST multi_pairing to verify BLS signature with many messages
May 23, 2023
e84403d
add test calling multi_pairing with length covering many values mod N…
May 23, 2023
9167d9e
use BLST multi_pairing to verify BLS SPoCK
May 23, 2023
cf0aa6b
clean up Fp12 tools
May 23, 2023
98152ba
clean up Relic tools
May 23, 2023
b6b90d6
uncomment a test
May 23, 2023
c94b091
Merge pull request #4377 from onflow/tarak/blst-pairing
tarakby May 24, 2023
3ec1ecf
remove relic tag and delete non-needed files
May 24, 2023
64112cc
remove relic build scripts
May 24, 2023
19a21db
remove relic macros and xmd_sha256, remove relic binray from LD flags
May 24, 2023
e58fe24
update Makefile and dockerignore
May 24, 2023
8123970
remove Relic mentions in code and README
May 25, 2023
541df79
update flow-go/README and gitignore
May 25, 2023
1d0b8f9
remove relic related commands from Makefile/ci/dockerfile
May 25, 2023
4eabaf1
remove relic tags from go files
May 25, 2023
9b17f93
remove more relic related tags and code
May 25, 2023
5721546
remove crypto_setup_gopath
May 25, 2023
06572e3
update go generate
May 25, 2023
33c5e0e
remove cmake install from dockerfile
May 26, 2023
83f42fb
clean up header files in blst_include.h
May 27, 2023
ec2ceb4
update boolean usage from bool_t to C type bool
May 27, 2023
fb4ac12
add sanity check scalar mult in G1 and G2
May 28, 2023
0232a95
use not enough shares error in BLSReconstructThresholdSignature
May 28, 2023
5fa28df
refactor BLS constants to use internal BLS12_381 length constants
May 28, 2023
efefb51
Merge pull request #4380 from onflow/tarak/blst-unplug-relic
tarakby May 29, 2023
d2c7cbf
more consolidation of length constants
May 29, 2023
a888972
cgo supports macros! use C constant macros in go
May 29, 2023
945f2b9
define new internal ERROR type to abstract BLST_ERROR
May 30, 2023
06c4ca7
update code base to work for G1 serialization defined as uncompressed
May 30, 2023
9ed47f1
update code base to work for G2 serialization defined as uncompressed
May 31, 2023
01b64c5
make sure older compilers recognize uintx_t
May 31, 2023
7b0a25e
update crypto/Makefile go command
May 31, 2023
00f66d6
package default build uses ADX
May 31, 2023
f5c3668
Merge pull request #4397 from onflow/tarak/blst-improv
tarakby May 31, 2023
e998ab6
add ADX detection and cgo flags for all go commands in Makefile/Docke…
May 31, 2023
5dba1c5
clarify BLST sigill message
Jun 1, 2023
f471e4b
fix a bug in sigill string
Jun 1, 2023
d4b873d
update how cgo flag is passed to Dockerfile so that dittos aren't del…
Jun 1, 2023
fa5177f
add cgo flag to mockgen commands
Jun 1, 2023
2872fe7
Merge pull request #4405 from onflow/tarak/blst-adx
tarakby Jun 9, 2023
2a87898
remove test assertion strings in favor of PRG seed logging for test r…
Aug 17, 2023
8ffac58
add interface implementation sanity checks
Aug 17, 2023
8415a45
add faster scalar mult in E2 for small expos
Aug 18, 2023
a408dec
more implementation check sanity check
Aug 18, 2023
65ee3bf
clean up some todos and add global g2 key
Aug 18, 2023
b379744
address more TODOs
Aug 18, 2023
9d6c7c7
add c-formatting target
Aug 18, 2023
30d781c
remove clanf-format config file
Aug 18, 2023
647c0c2
add crypto code formatting check to CI
Aug 18, 2023
63cedc8
move c formatting to linter target
Aug 21, 2023
a8666e4
fix linter error
Aug 21, 2023
20b89d2
Merge pull request #4645 from onflow/tarak/blst-c-formatting
tarakby Aug 21, 2023
30e5a7b
delete unused fermat inversion
Aug 22, 2023
8556f69
update README with BLST update steps
Aug 22, 2023
c1f294c
temp tmate debug and compile with asan
Aug 24, 2023
bacdb33
tmp
Aug 24, 2023
c8b643c
add new target for sanitization
Aug 24, 2023
62c1a16
add sanitizer to ci job
Aug 24, 2023
e557e8b
add more sanitization flags and restrict sanitization to linux
Aug 29, 2023
f8bc02b
add sanitization to ci
Aug 29, 2023
d58b172
disable sanitization for E1_write_bytes because of false positive
Aug 29, 2023
e280664
split c-sanitize and disable msan from CI - add NO_MSAN macro
Aug 29, 2023
aa8d79e
disable tmate and format
Aug 29, 2023
b2302c9
add missing change
Aug 29, 2023
ceab7e0
fix asan command
Aug 29, 2023
8314b0c
more details about updating BLST version
Aug 30, 2023
6bb393c
Merge pull request #4654 from onflow/tarak/blst-sanitize
tarakby Aug 31, 2023
4adb5cf
minor macro improvement
Sep 2, 2023
e6b29bc
add types sanity check in init()
Sep 2, 2023
5305b28
add affine conversions for potential public keys that can be used in …
Sep 2, 2023
afa9f24
clarify some TODOs
Sep 6, 2023
76850aa
DKG's readVector enforces A to be in G2
Sep 6, 2023
589d8d7
format
Sep 6, 2023
14c9e3d
clean up c flags and add instruction to readme
Sep 6, 2023
140edc3
Merge pull request #4681 from onflow/tarak/blst-misc
tarakby Sep 6, 2023
4c7af01
merge master and fix conflicts
Sep 6, 2023
12e338e
c format
Sep 6, 2023
7a268ae
mod tidy
Sep 6, 2023
7640d94
remove deprecated Seed use
Sep 6, 2023
5c72468
add temp tmate
Sep 6, 2023
5011cc6
fix merging issues and delete relic related builds
Sep 7, 2023
c52da9c
Revert "add temp tmate"
Sep 7, 2023
e5c0630
update flakey test monitor
Sep 7, 2023
d1776c8
more clarifications to BLST version update README
Sep 7, 2023
ecf7021
update BLST source to v0.3.11
Sep 7, 2023
e478781
Merge pull request #4687 from onflow/tarak/blst-merge-master
tarakby Sep 7, 2023
3d062fc
Merge branch 'feature/blst-based-crypto' into tarak/blst-update-blst
Sep 7, 2023
241798f
clean up assembly files include in cgo compilations
Sep 8, 2023
3bea523
update internal/blst files
Sep 8, 2023
c292bc4
fix a readme typo
Sep 8, 2023
0d09d55
tmp tmate to debug
Sep 9, 2023
dce50f9
make start up node time larger to accommodate failing TestClusterSwit…
Sep 12, 2023
633f152
merge master branch
Sep 13, 2023
6fb9707
Merge pull request #4693 from onflow/tarak/blst-update-blst
tarakby Sep 13, 2023
ba78ef6
makefile typo
Sep 21, 2023
8ff9b79
clean up C bls12_381 utils
Sep 21, 2023
2332a61
clean up threshold and dkg C files - use poly degree in secret sharing
Sep 21, 2023
2cd3d28
format c files
Sep 21, 2023
262c3e0
rename G2_ prefix to E2_
Sep 21, 2023
7184bc5
Merge pull request #4740 from onflow/tarak/blst-cleanup
tarakby Sep 21, 2023
dc28e03
move godoc closer to the type definition
Oct 10, 2023
627d682
add E1 random point multiplication benchmark
Oct 11, 2023
cf8667b
remove StopTimer in bench
Oct 11, 2023
5f89c65
comment updates and reformat
Oct 11, 2023
0587bc6
fix non-freed memory in error case
Oct 12, 2023
f64d5ea
use a common logic to detect ADX support and set the crypto flag
Oct 20, 2023
6c34ae3
add noop target for empty ci operations
Oct 20, 2023
b180269
cleaning and typos
Oct 20, 2023
6a42f1e
merge master branch
Oct 21, 2023
5f2509d
Merge branch 'master' into jordan/merge-master-blst
jordanschalm Oct 23, 2023
7ad6a7a
make tidy
jordanschalm Oct 23, 2023
f45e546
fix overwritten test settings
Oct 23, 2023
34f18b3
merge master and fix conflicts
Nov 15, 2023
99b1237
remove crypto setup
Nov 15, 2023
867b2cf
merge master
Nov 20, 2023
fbfecc1
update graceful stop duration to 1s in integration upgrade test
Nov 21, 2023
c990e22
Merge branch 'master' into jordan/merge-master-blst
Nov 24, 2023
91f529c
Merge pull request #4863 from onflow/jordan/merge-master-blst
tarakby Nov 24, 2023
4839835
merge main feature branch
Nov 24, 2023
cd2d74e
slow down block rate in integration access cohort1 test
Nov 24, 2023
500f205
Merge pull request #4817 from onflow/tarak/blst-review
tarakby Nov 28, 2023
f713b2e
Merge branch 'master' into feature/blst-based-crypto
tarakby Dec 5, 2023
769ad67
slow down block production in bft tests
Dec 5, 2023
f7dac6c
slow down block rate in SN test
Dec 5, 2023
e3e29f0
fix SN integration test bug
Dec 5, 2023
bea7bf6
Merge pull request #5111 from onflow/tarak/blst-fix-integration-test
tarakby Dec 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@ jobs:
go-version: "1.20"
- name: Checkout repo
uses: actions/checkout@v2
- name: Build relic
run: make crypto_setup_gopath
# Provide Google Service Account credentials to Github Action, allowing interaction with the Google Container Registry
# Logging in as github-actions@dl-flow.iam.gserviceaccount.com
- id: auth
Expand Down
40 changes: 12 additions & 28 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,10 @@ jobs:
with:
go-version: ${{ env.GO_VERSION }}
cache: true
- name: Build relic
run: make crypto_setup_gopath
- name: Install C formatter
run: sudo apt-get install -y clang-format
- name: Run C formatter and sanitizer for ./crypto
run: make -C crypto c-format && make -C crypto c-sanitize
- name: Run go generate
run: go generate
working-directory: ${{ matrix.dir }}
Expand All @@ -51,10 +53,11 @@ jobs:
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.54
args: -v --build-tags relic
args: -v
working-directory: ${{ matrix.dir }}
# https://github.com/golangci/golangci-lint-action/issues/244
skip-cache: true


tidy:
name: Tidy
Expand All @@ -72,18 +75,6 @@ jobs:
- name: code sanity check
run: make code-sanity-check

shell-check:
name: ShellCheck
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v3
- name: Run ShellCheck
uses: ludeeus/action-shellcheck@203a3fd018dfe73f8ae7e3aa8da2c149a5f41c33
with:
scandir: './crypto'
ignore: 'relic'

create-dynamic-test-matrix:
name: Create Dynamic Test Matrix
runs-on: ubuntu-latest
Expand Down Expand Up @@ -144,20 +135,17 @@ jobs:
matrix:
include:
- name: crypto
make1: -C crypto setup
make2: unittest
setup: noop
retries: 1
race: 1
runner: ubuntu-latest
- name: insecure
make1: install-tools
make2: test
setup: install-tools
retries: 5
race: 0
runner: buildjet-4vcpu-ubuntu-2204
- name: integration
make1: install-tools
make2: test
setup: install-tools
retries: 5
race: 0
runner: buildjet-4vcpu-ubuntu-2204
Expand All @@ -171,16 +159,16 @@ jobs:
go-version: ${{ env.GO_VERSION }}
cache: true
- name: Setup tests (${{ matrix.name }})
run: make ${{ matrix.make1 }}
run: make ${{ matrix.setup }}
- name: Run tests (${{ matrix.name }})
env:
RACE_DETECTOR: ${{ matrix.race }}
uses: nick-fields/retry@v2
with:
timeout_minutes: 35
max_attempts: ${{ matrix.retries }}
# run `make2` target inside each module's root
command: VERBOSE=1 make -C ${{ matrix.name }} ${{ matrix.make2 }}
# run test target inside each module's root
command: VERBOSE=1 make -C ${{ matrix.name }} test
- name: Upload coverage report
uses: codecov/codecov-action@v3
with:
Expand All @@ -202,8 +190,6 @@ jobs:
with:
go-version: ${{ env.GO_VERSION }}
cache: true
- name: Build relic
run: make crypto_setup_gopath
- name: Docker build
run: make docker-build-flow docker-build-flow-corrupt
- name: Save Docker images
Expand Down Expand Up @@ -294,8 +280,6 @@ jobs:
with:
go-version: ${{ env.GO_VERSION }}
cache: true
- name: Build relic
run: make crypto_setup_gopath
- name: Load cached Docker images
uses: actions/cache@v3
with:
Expand Down
15 changes: 5 additions & 10 deletions .github/workflows/flaky-test-monitor.yml
tarakby marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -82,18 +82,15 @@ jobs:
matrix:
include:
- name: crypto
make1: -C crypto setup
make2: unittest
setup: noop
race: 1
test_category: unit-crypto
- name: insecure
make1: install-tools
make2: test
setup: install-tools
race: 0
test_category: unit-insecure
- name: integration
make1: install-tools
make2: test
setup: install-tools
race: 0
test_category: unit-integration
runs-on: ubuntu-latest
Expand All @@ -106,11 +103,11 @@ jobs:
go-version: ${{ env.GO_VERSION }}
cache: true
- name: Setup tests (${{ matrix.name }})
run: make ${{ matrix.make1 }}
run: make ${{ matrix.setup }}
- name: Run tests (${{ matrix.name }})
env:
RACE_DETECTOR: ${{ matrix.race }}
run: make -es -C ${{ matrix.name }} ${{ matrix.make2 }} > test-output
run: make -es -C ${{ matrix.name }} test > test-output
timeout-minutes: 100
continue-on-error: true
- name: Process test results (${{ matrix.name }})
Expand Down Expand Up @@ -167,8 +164,6 @@ jobs:
with:
go-version: ${{ env.GO_VERSION }}
cache: true
- name: Build relic
run: make crypto_setup_gopath
- name: Docker build
run: make docker-build-flow docker-build-flow-corrupt
- name: Run tests
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/tools.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,6 @@ jobs:
# to accurately get the version tag
fetch-depth: 0
ref: ${{ inputs.tag }}
- name: Build relic
run: make crypto_setup_gopath
- name: Build and upload boot-tools
run: |
make tool-bootstrap tool-transit
Expand Down
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@
/cmd/util/util
/cmd/bootstrap/bootstrap

# crypto relic folder
crypto/relic/

# Test binary, build with `go test -c`
*.test
Expand Down
Loading
Loading