Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[crypto] add blst source code #3851

Merged
merged 1,417 commits into from
Feb 27, 2023
Merged

Conversation

tarakby
Copy link
Contributor

@tarakby tarakby commented Jan 24, 2023

Note: this is the first PR of a long epic (BLST-based crypto), code will continue to change significantly. The epic is implemented into multiple PRs to use continuous CI testing. The epic purpose is to substitute Relic by BLST, both libraries are used by the core crypto code to perform low level math operations.

  • copy BLST source code folder into Flow crypto (/blst_src)
  • add directives to include C and assembly BLST files into the Go module build. This should expose all BLST code to the Flow crypto build.
    • remove c99 to align with BLST compilation
    • add C and assembly files into the module compilation via blst_src.c and blst_assembly.S
  • add licence and blst_src/README
  • comment out cross BLST tests temporarily because of duplicate symbols (to figure out a solution in later PRs)

durkmurder and others added 19 commits January 23, 2023 19:12
1. Replace EnableAllLimitEnforcements/DisableAllLimitEnforcements with RunWithAllLimitsDisabled
2. Remove enforceLimit parameter from Get/Set calls.
3. Internalize meter enforceLimits check into State.

This enables us to compose view implementations (meter/interaction limits view, spock view, delta, etc) in the future.
3846: Use consistent method for creating temp dirs r=jordanschalm a=jordanschalm

This PR changes integration test utilities to use a consistent method and root location when creating temporary directories for use in tests. 

This is to mitigate an issue where integration tests would fail on macOS, due to a permissions issue when initializing bootstrap and data directories for Docker containers.
```
lchown /var/folders/bm/wbf3kfs91911lr5w6qyw126c0000gn/T/TestEpochJoinAndLeaveANTestEpochJoinAndLeaveAN4216456854/001/bootstrap/execution-state/00000000: operation not permitted
```

As far as I can tell, `t.TempDir()` created directories which caused `operation not permitted` errors, whereas `os.MkdirTemp("/tmp", ...)` did not. We aren't sure what causes the two methods to behave differently at this point, but using "the good one" consistently works to fix the test failure. 

Co-authored-by: Jordan Schalm <jordan@dapperlabs.com>
(which will create and setup staking account all in one shot)
@codecov-commenter
Copy link

codecov-commenter commented Jan 24, 2023

Codecov Report

Merging #3851 (7593960) into feature/blst-based-crypto (7ca2a9f) will increase coverage by 0.15%.
The diff coverage is 55.12%.

@@                      Coverage Diff                      @@
##           feature/blst-based-crypto    #3851      +/-   ##
=============================================================
+ Coverage                      53.00%   53.16%   +0.15%     
=============================================================
  Files                            796      750      -46     
  Lines                          73638    72796     -842     
=============================================================
- Hits                           39035    38700     -335     
+ Misses                         31425    30894     -531     
- Partials                        3178     3202      +24     
Flag Coverage Δ
unittests 53.16% <55.12%> (+0.15%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
cmd/bootstrap/cmd/check_machine_account.go 8.47% <0.00%> (ø)
cmd/bootstrap/transit/cmd/snapshot.go 11.32% <0.00%> (ø)
cmd/execution_builder.go 0.00% <0.00%> (ø)
cmd/execution_config.go 0.00% <0.00%> (ø)
cmd/util/ledger/reporters/atree_reporter.go 0.00% <0.00%> (ø)
cmd/verification_builder.go 0.00% <0.00%> (ø)
consensus/aggregators.go 0.00% <0.00%> (ø)
consensus/config.go 0.00% <0.00%> (ø)
consensus/hotstuff/committees/metrics_wrapper.go 0.00% <0.00%> (ø)
consensus/hotstuff/committees/static.go 0.00% <0.00%> (ø)
... and 304 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

pattyshack and others added 3 commits February 16, 2023 11:44
Prep work for changing address internal representation

The number of changes is getting a bit out of hand.  I'll split up the change
into smaller PRs, with the goal of consistently operating on flow.Address
whenever possible.

Note that (except for tests) I'm trying to consistently name the address
variables based on their types:
 - address is always a flow.Address
 - runtimeAddress is a common.Address
 - cadenceAddress is a cadence.Address
3912: Clean up address usage (part 1 of many) r=pattyshack a=pattyshack

Prep work for changing address internal representation

The number of changes is getting a bit out of hand.  I'll split up the change into smaller PRs, with the goal of consistently operating on flow.Address whenever possible.

Note that (except for tests) I'm trying to consistently name the address variables based on their types:
 - address is always a flow.Address
 - runtimeAddress is a common.Address
 - cadenceAddress is a cadence.Address

Co-authored-by: Patrick Lee <patrick.lee@dapperlabs.com>
@github-actions
Copy link
Contributor

FVM Benchstat comparison

This branch with compared with the base branch onflow:feature/blst-based-crypto commit 7ca2a9f

The command (for i in {1..7}; do go test ./fvm ./engine/execution/computation --bench . --tags relic -shuffle=on --benchmem --run ^$; done) was used.

Collapsed results for better readability

old.txtnew.txt
time/opdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/get_account_and_get_storage_used-251.2ms ± 3%53.9ms ± 4%+5.21%(p=0.008 n=6+7)
RuntimeTransaction/get_account_and_get_balance-2437ms ± 3%447ms ± 3%+2.44%(p=0.038 n=7+7)
RuntimeTransaction/borrow_array_from_storage-2180ms ± 1%182ms ± 2%+1.06%(p=0.026 n=6+6)
RuntimeNFTBatchTransfer-2157ms ± 4%159ms ± 4%~(p=0.366 n=6+7)
RuntimeTransaction/reference_tx-244.4ms ± 5%45.5ms ±14%~(p=0.945 n=6+7)
RuntimeTransaction/convert_int_to_string-248.6ms ±16%46.0ms ± 2%~(p=0.383 n=7+7)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-247.9ms ± 4%49.4ms ± 5%~(p=0.181 n=7+6)
RuntimeTransaction/get_signer_address-247.4ms ± 9%47.2ms ±13%~(p=0.620 n=7+7)
RuntimeTransaction/get_public_account-248.7ms ± 7%48.9ms ± 8%~(p=1.000 n=7+7)
RuntimeTransaction/get_account_and_get_available_balance-2409ms ± 3%418ms ± 4%~(p=0.053 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-2375ms ± 4%376ms ± 2%~(p=0.805 n=7+7)
RuntimeTransaction/get_signer_vault-254.4ms ± 5%54.1ms ± 4%~(p=0.710 n=7+7)
RuntimeTransaction/get_signer_receiver-274.6ms ± 3%73.6ms ± 3%~(p=0.295 n=6+7)
RuntimeTransaction/transfer_tokens-2310ms ± 3%306ms ± 4%~(p=0.805 n=7+7)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-252.6ms ± 5%52.9ms ± 4%~(p=0.805 n=7+7)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2104ms ± 5%107ms ± 3%~(p=0.209 n=7+7)
RuntimeTransaction/create_new_account-21.07s ± 7%1.04s ± 1%~(p=0.731 n=7+6)
RuntimeTransaction/call_empty_contract_function-248.7ms ± 5%49.5ms ± 6%~(p=0.535 n=7+7)
RuntimeTransaction/emit_event-268.7ms ± 3%69.8ms ± 8%~(p=0.710 n=7+7)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-26.77s ± 1%6.86s ± 2%~(p=0.073 n=6+7)
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/copy_array_from_storage-2189ms ± 3%181ms ± 3%−3.86%(p=0.011 n=7+7)
 
alloc/opdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/load_and_save_long_string_on_signers_address-251.7MB ± 2%52.9MB ± 1%+2.32%(p=0.030 n=6+5)
RuntimeNFTBatchTransfer-253.7MB ± 7%54.0MB ± 6%~(p=1.000 n=7+7)
RuntimeTransaction/reference_tx-234.6MB ± 3%34.6MB ± 4%~(p=0.945 n=6+7)
RuntimeTransaction/convert_int_to_string-235.6MB ± 5%34.8MB ± 4%~(p=0.209 n=7+7)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-235.9MB ± 3%36.4MB ± 3%~(p=0.138 n=6+7)
RuntimeTransaction/get_signer_address-235.0MB ± 1%35.0MB ± 2%~(p=0.836 n=6+7)
RuntimeTransaction/get_public_account-235.9MB ± 1%35.7MB ± 3%~(p=0.836 n=6+7)
RuntimeTransaction/get_account_and_get_balance-2121MB ± 0%122MB ± 3%~(p=0.268 n=5+7)
RuntimeTransaction/get_account_and_get_available_balance-2105MB ± 3%106MB ± 0%~(p=0.876 n=7+5)
RuntimeTransaction/get_account_and_get_storage_used-236.8MB ± 6%37.6MB ± 3%~(p=0.165 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-2101MB ± 2%102MB ± 3%~(p=0.383 n=7+7)
RuntimeTransaction/get_signer_vault-237.2MB ± 3%36.6MB ± 6%~(p=0.456 n=7+7)
RuntimeTransaction/get_signer_receiver-242.2MB ± 9%42.2MB ± 5%~(p=0.535 n=7+7)
RuntimeTransaction/transfer_tokens-283.5MB ± 4%83.5MB ± 4%~(p=0.902 n=7+7)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-236.1MB ± 6%35.7MB ± 8%~(p=0.456 n=7+7)
RuntimeTransaction/create_new_account-2179MB ± 5%178MB ± 3%~(p=0.805 n=7+7)
RuntimeTransaction/call_empty_contract_function-235.3MB ± 1%35.5MB ± 4%~(p=1.000 n=6+7)
RuntimeTransaction/emit_event-241.8MB ± 3%41.7MB ± 6%~(p=0.710 n=7+7)
RuntimeTransaction/borrow_array_from_storage-270.5MB ± 2%70.2MB ± 4%~(p=0.628 n=6+7)
RuntimeTransaction/copy_array_from_storage-281.5MB ± 0%81.7MB ± 2%~(p=0.755 n=5+7)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-21.19GB ± 1%1.20GB ± 1%~(p=0.073 n=7+7)
 
allocs/opdelta
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-217.1M ± 0%17.3M ± 0%+0.98%(p=0.001 n=7+7)
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/create_new_account-22.26M ± 0%2.27M ± 0%+0.31%(p=0.001 n=7+6)
RuntimeTransaction/reference_tx-282.1k ± 0%82.2k ± 0%+0.17%(p=0.001 n=7+7)
RuntimeTransaction/get_signer_address-286.3k ± 0%86.4k ± 0%+0.15%(p=0.001 n=7+7)
RuntimeTransaction/call_empty_contract_function-297.6k ± 0%97.7k ± 0%+0.15%(p=0.001 n=7+7)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2105k ± 0%106k ± 0%+0.14%(p=0.001 n=7+7)
RuntimeTransaction/convert_int_to_string-294.3k ± 0%94.4k ± 0%+0.14%(p=0.001 n=7+7)
RuntimeTransaction/get_account_and_get_storage_used-2119k ± 0%119k ± 0%+0.14%(p=0.001 n=7+7)
RuntimeTransaction/get_public_account-2108k ± 0%109k ± 0%+0.14%(p=0.003 n=5+7)
RuntimeTransaction/get_signer_vault-2123k ± 0%123k ± 0%+0.12%(p=0.001 n=7+7)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2121k ± 0%121k ± 0%+0.10%(p=0.001 n=6+7)
RuntimeTransaction/emit_event-2137k ± 0%137k ± 0%+0.10%(p=0.001 n=7+6)
RuntimeTransaction/get_signer_receiver-2197k ± 0%197k ± 0%+0.07%(p=0.001 n=7+7)
RuntimeTransaction/borrow_array_from_storage-2336k ± 0%336k ± 0%+0.05%(p=0.001 n=7+7)
RuntimeTransaction/copy_array_from_storage-2292k ± 0%292k ± 0%+0.04%(p=0.001 n=7+7)
RuntimeTransaction/transfer_tokens-2841k ± 0%841k ± 0%+0.02%(p=0.001 n=7+7)
RuntimeTransaction/get_account_and_get_available_balance-21.23M ± 0%1.23M ± 0%+0.01%(p=0.001 n=7+6)
RuntimeTransaction/get_account_and_get_balance-21.30M ± 0%1.30M ± 0%+0.01%(p=0.004 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-21.11M ± 0%1.11M ± 0%+0.01%(p=0.001 n=7+7)
RuntimeNFTBatchTransfer-2263k ± 1%263k ± 1%~(p=0.620 n=7+7)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2193k ± 0%192k ± 0%−0.38%(p=0.001 n=7+7)
 
computationdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-2202 ± 0%202 ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2502 ± 0%502 ± 0%~(all equal)
RuntimeTransaction/get_signer_address-2302 ± 0%302 ± 0%~(all equal)
RuntimeTransaction/get_public_account-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_balance-21.00k ± 0%1.00k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_available_balance-23.10k ± 0%3.10k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_used-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_capacity-21.70k ± 0%1.70k ± 0%~(all equal)
RuntimeTransaction/get_signer_vault-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/get_signer_receiver-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/transfer_tokens-23.50k ± 0%3.50k ± 0%~(all equal)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/create_new_account-2202 ± 0%202 ± 0%~(all equal)
RuntimeTransaction/call_empty_contract_function-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/emit_event-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/borrow_array_from_storage-22.60k ± 0%2.60k ± 0%~(all equal)
RuntimeTransaction/copy_array_from_storage-22.60k ± 0%2.60k ± 0%~(all equal)
 
interactionsdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/get_account_and_get_available_balance-25.64M ± 0%5.64M ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-25.64M ± 0%5.64M ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/create_new_account-212.3M ± 0%12.3M ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/get_account_and_get_balance-224.3M ± 0%24.3M ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/transfer_tokens-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/reference_tx-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/convert_int_to_string-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/get_signer_address-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/get_public_account-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/get_account_and_get_storage_used-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/get_signer_vault-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/get_signer_receiver-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/call_empty_contract_function-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/emit_event-2331k ± 0%331k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2334k ± 0%334k ± 0%+0.03%(p=0.001 n=7+7)
RuntimeTransaction/borrow_array_from_storage-2336k ± 0%336k ± 0%+0.03%(p=0.001 n=6+7)
RuntimeTransaction/copy_array_from_storage-2336k ± 0%336k ± 0%+0.03%(p=0.001 n=7+7)
 
us/txdelta
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-23.31k ± 1%3.35k ± 2%~(p=0.073 n=6+7)
 

@pattyshack pattyshack removed their request for review February 17, 2023 17:39
@tarakby tarakby merged commit 7163fc1 into feature/blst-based-crypto Feb 27, 2023
@tarakby tarakby deleted the tarak/add-blst-src branch February 27, 2023 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.