Failed to build `aws-lc-rs` with LLVM v19.1 due to `rustc` error E0425 #476

rami3l · 2024-08-01T05:34:22Z

@justsmth thank you so much for your investigation and hard work so far! I'll see you again very soon in other follow-up issues 😶
#453 (comment)

Here we go 🤦‍♀️

Problem

After merging rust-lang/rustup#3898, Rustup's FreeBSD CI has started failing.

rust-lang/rustup#3979 tries to set up the necessary dependencies for the build, namely cmake-core and llvm-devel-lite (for libclang.so). Since it's internal bindgen by default, bindgen-cli doesn't seem to be required.

But still, Compiling aws-lc-sys v0.20.1 seems to have failed with the following log:

  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_LIB_RUST` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:74:14
     |
  74 |     unsafe { ERR_GET_LIB_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^ not found in this scope
  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_REASON_RUST` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:80:14
     |
  80 |     unsafe { ERR_GET_REASON_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^^^^ not found in this scope
  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_FUNC_RUST` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:86:14
     |
  86 |     unsafe { ERR_GET_FUNC_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^^ not found in this scope
  error[E0425]: cannot find function, tuple struct or tuple variant `BIO_ctrl` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:91:14
     |
  91 |     unsafe { BIO_ctrl(b, BIO_CTRL_INFO, 0, pp.cast::<c_void>()) }
     |              ^^^^^^^^ not found in this scope
  error[E0425]: cannot find function, tuple struct or tuple variant `CRYPTO_library_init` in this scope
    --> /home/runner/work/rustup/rustup/.cargo/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-sys-0.20.1/src/lib.rs:95:14
     |
  95 |     unsafe { CRYPTO_library_init() }
     |              ^^^^^^^^^^^^^^^^^^^ not found in this scope
  For more information about this error, try `rustc --explain E0425`.
  The following warnings were emitted during compilation:
  warning: aws-lc-sys@0.20.1: CMAKE environment variable set: cmake
  warning: aws-lc-sys@0.20.1: Generating bindings - internal bindgen. Platform: x86_64-unknown-freebsd
  error: could not compile `aws-lc-sys` (lib) due to 5 previous errors

https://github.com/rust-lang/rustup/actions/runs/10192459718/job/28195406529?pr=3979

Relevant details

FWIW, we use the following action for our FreeBSD tests:

- uses: vmactions/freebsd-vm@v1
  with:
    release: 13.2
    usesh: true
    copyback: false
    prepare: |
      pkg install -y git gmake bash sudo cmake-core llvm-devel-lite

https://github.com/rust-lang/rustup/blob/8fdc61316891cdb72acf406b2ac929227ceb0068/ci/actions-templates/freebsd-builds-template.yaml

The text was updated successfully, but these errors were encountered:

justsmth · 2024-08-01T11:36:02Z

Hello again! 😊

I've seen this type of failure before. It typically occurs after the bindings are generated, but the logic in lib.rs fails to pick it up. (Perhaps this include! needs an adustment?) I'll dig in now.

justsmth · 2024-08-01T12:33:52Z

Some progress made. I'm able to reproduce the same failure in the FreeBSD CI test I'm setting up: https://github.com/aws/aws-lc-rs/actions/runs/10198121880/job/28212416541#step:3:38543

...
      Compiling aws-lc-rs v1.8.1 (/home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-rs)
     Compiling clap_derive v4.5.13
     Compiling clap_builder v4.5.13
  warning: aws-lc-sys@0.20.1: CMAKE environment variable set: cmake
  warning: aws-lc-sys@0.20.1: Generating bindings - internal bindgen. Platform: x86_64-unknown-freebsd
  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_LIB_RUST` in this scope
    --> aws-lc-sys/src/lib.rs:74:14
     |
  74 |     unsafe { ERR_GET_LIB_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^ not found in this scope
  
  error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_REASON_RUST` in this scope
    --> aws-lc-sys/src/lib.rs:80:14
     |
  80 |     unsafe { ERR_GET_REASON_RUST(packed_error) }
     |              ^^^^^^^^^^^^^^^^^^^ not found in this scope
...

I'm working on setting up a dedicated FreeBSD host so I can debug more easily.

justsmth · 2024-08-01T12:57:54Z

More progress... I can also reproduce on a FreeBSD EC2 instance:

   Compiling paste v1.0.15
   Compiling aws-lc-sys v0.20.1 (/usr/home/ec2-user/repos/aws-lc-rs/aws-lc-sys)
warning: aws-lc-sys@0.20.1: CMAKE environment variable set: cmake
warning: aws-lc-sys@0.20.1: Generating bindings - internal bindgen. Platform: x86_64-unknown-freebsd
error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_LIB_RUST` in this scope
  --> aws-lc-sys/src/lib.rs:74:14
   |
74 |     unsafe { ERR_GET_LIB_RUST(packed_error) }
   |              ^^^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find function, tuple struct or tuple variant `ERR_GET_REASON_RUST` in this scope
  --> aws-lc-sys/src/lib.rs:80:14
   |
80 |     unsafe { ERR_GET_REASON_RUST(packed_error) }
   |              ^^^^^^^^^^^^^^^^^^^ not found in this scope

Now to the "fun" part... 😊

justsmth · 2024-08-01T13:50:38Z

So it appears that the "internal" bindings generation is producing invalid bindings, however using the "external" bindgen-cli works fine and all tests pass:

❯ AWS_LC_SYS_EXTERNAL_BINDGEN=1 cargo test -p aws-lc-rs
  Downloaded anstyle-parse v0.2.5
  Downloaded colorchoice v1.0.2
...
warning: aws-lc-sys@0.20.1: CMAKE environment variable set: cmake
warning: aws-lc-sys@0.20.1: Generating bindings - external bindgen. Platform: x86_64-unknown-freebsd
    Finished `test` profile [unoptimized + debuginfo] target(s) in 52.02s
     Running unittests src/lib.rs (target/debug/deps/aws_lc_rs-dcf25f61ef54de42)

running 187 tests
...
test rsa4096_oaep_sha256_mgf1sha256 ... ok
test rsa4096_oaep_sha512_mgf1sha512 ... ok

test result: ok. 37 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.95s

   Doc-tests aws_lc_rs

running 24 tests
test aws-lc-rs/src/cipher.rs - cipher (line 140) ... ok
...
test aws-lc-rs/src/tls_prf.rs - tls_prf (line 8) ... ok
test aws-lc-rs/src/pbkdf2.rs - pbkdf2 (line 23) ... ok

test result: ok. 22 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 1.61s

So a temporary workaround might be to ensure that "bindgen-cli" is installed and have AWS_LC_SYS_EXTERNAL_BINDGEN=1 set in the environment. I'll try this for the PR I'm working on for this.

I'll keep digging to see why the internal bindgen is behaving like this.

Edit: The FreeBSD CI test succeeded using external bindgen: https://github.com/aws/aws-lc-rs/actions/runs/10199677822/job/28217403628#step:3:39141

rami3l · 2024-08-01T15:56:25Z

So a temporary workaround might be to ensure that "bindgen-cli" is installed and have AWS_LC_SYS_EXTERNAL_BINDGEN=1 set in the environment.

@justsmth Thanks a lot for the pointers! It seems to work with Rustup as well.

Also, adding rust-bindgen-cli to the pkg install -y line will save you a minute or two from using cargo install.

I'll keep digging to see why the internal bindgen is behaving like this.

Please take your time! It's not really a blocker for us now that we have this workaround :)

@echo

Update CI diff --git a/.github/workflows/sys-bindings-generator.yml b/.github/workflows/sys-bindings-generator.yml index 3fba893d80..dc226f9f1b 100644 --- a/.github/workflows/sys-bindings-generator.yml +++ b/.github/workflows/sys-bindings-generator.yml @@ -270,3 +270,36 @@ jobs: run: ./scripts/build/collect_build_src.sh -t ${{ matrix.target }} - name: Commit & Push changes run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected source files for ${{ matrix.target }}" + collect-nasm-and-commit: + needs: generate-windows-bindings-and-commit + if: github.repository == 'aws/aws-lc-rs' + runs-on: windows-latest + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + ref: ${{ github.ref_name }} + - uses: dtolnay/rust-toolchain@master + id: toolchain + with: + toolchain: stable + targets: "x86_64-pc-windows-msvc,x86_64-pc-windows-gnu" + - uses: ilammy/setup-nasm@v1 + - name: Build aws-lc-sys + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo build -p aws-lc-sys --release --target x86_64-pc-windows-msvc + - name: Collect NASM object files + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Clean build + shell: bash + run: cargo clean + - name: Test aws-lc-rs for x86_64-pc-windows-msvc + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-msvc + - name: Test aws-lc-sys for x86_64-pc-windows-gnu + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-gnu + - name: Commit & Push changes + shell: bash + run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected NASM files" diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a0da3f48b8..71716a64c0 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -37,7 +37,7 @@ jobs: - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -66,7 +66,7 @@ jobs: - --no-default-features --features aws-lc-sys,bindgen,unstable - --release --all-targets --features bindgen,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - if: ${{ matrix.os == 'macos-13-xlarge' }} @@ -106,9 +106,10 @@ jobs: - --no-default-features --features non-fips,ring-io,unstable - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable + env: + AWS_LC_SYS_PREBUILT_NASM: 1 steps: - - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -125,7 +126,7 @@ jobs: name: aws-ls-rs coverage runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true @@ -167,7 +168,7 @@ jobs: - --no-default-features --features fips,asan runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -196,7 +197,7 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -205,6 +206,40 @@ jobs: # See: rust-lang/cargo#8531 run: cargo test -p aws-lc-rs --tests + build-env-nasm-test: + if: github.repository_owner == 'aws' + name: prebuilt NASM verification + runs-on: windows-latest + strategy: + fail-fast: false + matrix: + target: + - 'x86_64-pc-windows-msvc' + - 'x86_64-pc-windows-gnu' + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + - uses: dtolnay/rust-toolchain@stable + - name: Install NASM + uses: ilammy/setup-nasm@v1 + - name: Remove NASM artifacts + shell: bash + run: | + cargo clean + rm ./aws-lc-sys/builder/prebuilt-nasm/* + - name: Run cargo test + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo test --tests -p aws-lc-rs --release --no-default-features --features aws-lc-sys + - name: Collect NASM outputs + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Flag any NASM changes + shell: bash + run: | + git add . + git diff --cached --exit-code HEAD -- aws-lc-sys/builder/prebuilt-nasm/*.txt + build-env-external-bindgen-test: if: github.repository_owner == 'aws' name: aws-lc-rs FIPS - External bindgen test @@ -216,14 +251,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - name: Install bindgen-cli run: cargo install --locked bindgen-cli - name: Remove bindings @@ -245,12 +280,10 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -271,7 +304,7 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -306,15 +339,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' + - uses: seanmiddleditch/gha-setup-ninja@v5 - name: Run cargo test run: cargo test -p aws-lc-rs --tests --no-default-features --features fips - name: Release build @@ -346,14 +378,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -378,7 +410,7 @@ jobs: - macos-12 - macos-13-xlarge steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs index e25cecb443..c5069c5341 100644 --- a/aws-lc-sys/builder/cmake_builder.rs +++ b/aws-lc-sys/builder/cmake_builder.rs @@ -3,9 +3,9 @@ use crate::OutputLib::{Crypto, RustWrapper, Ssl}; use crate::{ - cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target, - target_arch, target_env, target_family, target_os, target_underscored, target_vendor, - OutputLibType, + allow_prebuilt_nasm, cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, + option_env, target, target_arch, target_env, target_family, target_os, target_underscored, + target_vendor, test_nasm_command, OutputLibType, }; use std::env; use std::ffi::OsString; @@ -22,10 +22,6 @@ fn test_clang_cl_command() -> bool { execute_command("clang-cl".as_ref(), &["--version".as_ref()]).status } -fn test_nasm_command() -> bool { - execute_command("nasm".as_ref(), &["-version".as_ref()]).status -} - fn find_cmake_command() -> Option<OsString> { if let Some(cmake) = option_env("CMAKE") { emit_warning(&format!( @@ -162,7 +158,7 @@ impl CmakeBuilder { // See issue: aws#453 if target_os() == "windows" { - Self::configure_windows(&mut cmake_cfg); + self.configure_windows(&mut cmake_cfg); } // If the build environment vendor is Apple @@ -213,7 +209,7 @@ impl CmakeBuilder { cmake_cfg } - fn configure_windows(cmake_cfg: &mut cmake::Config) { + fn configure_windows(&self, cmake_cfg: &mut cmake::Config) { match (target_env().as_str(), target_arch().as_str()) { ("msvc", "aarch64") => { cmake_cfg.generator_toolset(format!( @@ -243,6 +239,22 @@ impl CmakeBuilder { } _ => {} } + if target_arch() == "x86_64" && Some(true) == allow_prebuilt_nasm() { + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + emit_warning("!!! Using pre-built NASM binaries !!!"); + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + + let script_path = self + .manifest_dir + .join("builder") + .join("prebuilt-nasm.bat") + .display() + .to_string(); + let script_path = script_path.replace('\\', "/"); + + cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str()); + cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1"); + } } fn configure_open_harmony(cmake_cfg: &mut cmake::Config) { @@ -298,10 +310,14 @@ impl crate::Builder for CmakeBuilder { let mut missing_dependency = false; if target_os() == "windows" { - if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() { + if target_arch() == "x86_64" + && !is_no_asm() + && !test_nasm_command() + && Some(true) != allow_prebuilt_nasm() + { eprintln!( - "Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\ - See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html" + "Consider setting `AWS_LC_SYS_PREBUILT_NASM` in the build environment.\ + See User Guide: https://aws.github.io/aws-lc-rs/index.html" ); eprintln!("Missing dependency: nasm"); missing_dependency = true; diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs index 23f0214991..987e1038ca 100644 --- a/aws-lc-sys/builder/main.rs +++ b/aws-lc-sys/builder/main.rs @@ -319,6 +319,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false; static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_NO_ASM: bool = false; +static mut AWS_LC_SYS_PREBUILT_NASM: Option<bool> = None; fn initialize() { unsafe { @@ -328,6 +329,7 @@ fn initialize() { AWS_LC_SYS_EXTERNAL_BINDGEN = env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false); AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false); + AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM"); } if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) { @@ -363,6 +365,7 @@ fn is_bindgen_required() -> bool { || !has_pregenerated() } +#[allow(dead_code)] fn internal_bindgen_supported() -> bool { // TODO: internal bindgen creates invalid bindings on FreeBSD // See: aws#476 @@ -385,6 +388,10 @@ fn is_no_asm() -> bool { unsafe { AWS_LC_SYS_NO_ASM } } +fn allow_prebuilt_nasm() -> Option<bool> { + unsafe { AWS_LC_SYS_PREBUILT_NASM } +} + fn has_bindgen_feature() -> bool { cfg!(feature = "bindgen") } @@ -393,6 +400,10 @@ fn has_pregenerated() -> bool { unsafe { PREGENERATED } } +fn test_nasm_command() -> bool { + execute_command("nasm".as_ref(), &["-version".as_ref()]).status +} + fn prepare_cargo_cfg() { // This is supported in Rust >= 1.77.0 // Also remove `#![allow(unexpected_cfgs)]` from src/lib.rs diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat new file mode 100644 index 0000000000..9c761db12a --- /dev/null +++ b/aws-lc-sys/builder/prebuilt-nasm.bat @@ -0,0 +1,21 @@ +@echo off +set "ScriptDir=%~dp0" +set "ScriptDir=%ScriptDir:~0,-1%" +:loop +set "arg1=%~1" +if "%arg1%"=="-o" goto end +if "%arg1%"=="" goto failure +shift +goto loop +:end +shift +set "path=%~1" +for %%f in ("%path%") do set "filename=%%~nxf" +copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%" +exit 0 + +:failure +echo PATH: %path% 1>&2 +echo FILENAME: %filename% 1>&2 +echo ScriptDir: %ScriptDir% 1>&2 +exit 1 \ No newline at end of file diff --git a/scripts/build/collect_nasm_obj.sh b/scripts/build/collect_nasm_obj.sh new file mode 100644 index 0000000000..c425e9236f --- /dev/null +++ b/scripts/build/collect_nasm_obj.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 OR ISC + +set -ex +set -o pipefail + +if [[ ${BASH_VERSINFO[0]} -lt 4 ]]; then + echo Must use bash 4 or later: ${BASH_VERSION} + exit 1 +fi + +SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd) +REPO_ROOT=$(git rev-parse --show-toplevel) +SYS_CRATE_DIR="${REPO_ROOT}/aws-lc-sys" +PREBUILT_NASM_DIR="${SYS_CRATE_DIR}/builder/prebuilt-nasm" +mkdir -p "${PREBUILT_NASM_DIR}" +rm -f "${PREBUILT_NASM_DIR}"/* + +DUMPBIN="$(find /c/Program\ Files/Microsoft\ Visual\ Studio/ -path "*/Hostx64/x64/*" -name "dumpbin.exe" -print -quit)" + +for nasm_file in `find aws-lc-sys/aws-lc/generated-src/win-x86_64/ -name "*.asm"`; do + OBJNAME=$(basename "${nasm_file}"); + NASM_OBJ=$(find target/ -name "${OBJNAME/.asm/.obj}"); + cp "${NASM_OBJ}" "${PREBUILT_NASM_DIR}" + # We remove the '.debug$S' value, which indicates the size of the debug section. This value can change across builds + # because it typically contains full source file paths that vary by build environment + "${DUMPBIN}" //DISASM "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/.obj}" | grep -v '.debug$S' | sed -e "s/^Dump of file.*/Dump of file ${OBJNAME/.asm/.obj}/" > "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/}"-disasm.txt +done

@echo

Update CI diff --git a/.github/workflows/sys-bindings-generator.yml b/.github/workflows/sys-bindings-generator.yml index 3fba893d80..dc226f9f1b 100644 --- a/.github/workflows/sys-bindings-generator.yml +++ b/.github/workflows/sys-bindings-generator.yml @@ -270,3 +270,36 @@ jobs: run: ./scripts/build/collect_build_src.sh -t ${{ matrix.target }} - name: Commit & Push changes run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected source files for ${{ matrix.target }}" + collect-nasm-and-commit: + needs: generate-windows-bindings-and-commit + if: github.repository == 'aws/aws-lc-rs' + runs-on: windows-latest + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + ref: ${{ github.ref_name }} + - uses: dtolnay/rust-toolchain@master + id: toolchain + with: + toolchain: stable + targets: "x86_64-pc-windows-msvc,x86_64-pc-windows-gnu" + - uses: ilammy/setup-nasm@v1 + - name: Build aws-lc-sys + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo build -p aws-lc-sys --release --target x86_64-pc-windows-msvc + - name: Collect NASM object files + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Clean build + shell: bash + run: cargo clean + - name: Test aws-lc-rs for x86_64-pc-windows-msvc + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-msvc + - name: Test aws-lc-sys for x86_64-pc-windows-gnu + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-gnu + - name: Commit & Push changes + shell: bash + run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected NASM files" diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a0da3f48b8..71716a64c0 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -37,7 +37,7 @@ jobs: - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -66,7 +66,7 @@ jobs: - --no-default-features --features aws-lc-sys,bindgen,unstable - --release --all-targets --features bindgen,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - if: ${{ matrix.os == 'macos-13-xlarge' }} @@ -106,9 +106,10 @@ jobs: - --no-default-features --features non-fips,ring-io,unstable - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable + env: + AWS_LC_SYS_PREBUILT_NASM: 1 steps: - - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -125,7 +126,7 @@ jobs: name: aws-ls-rs coverage runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true @@ -167,7 +168,7 @@ jobs: - --no-default-features --features fips,asan runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -196,7 +197,7 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -205,6 +206,40 @@ jobs: # See: rust-lang/cargo#8531 run: cargo test -p aws-lc-rs --tests + build-env-nasm-test: + if: github.repository_owner == 'aws' + name: prebuilt NASM verification + runs-on: windows-latest + strategy: + fail-fast: false + matrix: + target: + - 'x86_64-pc-windows-msvc' + - 'x86_64-pc-windows-gnu' + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + - uses: dtolnay/rust-toolchain@stable + - name: Install NASM + uses: ilammy/setup-nasm@v1 + - name: Remove NASM artifacts + shell: bash + run: | + cargo clean + rm ./aws-lc-sys/builder/prebuilt-nasm/* + - name: Run cargo test + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo test --tests -p aws-lc-rs --release --no-default-features --features aws-lc-sys + - name: Collect NASM outputs + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Flag any NASM changes + shell: bash + run: | + git add . + git diff --cached --exit-code HEAD -- aws-lc-sys/builder/prebuilt-nasm/*.txt + build-env-external-bindgen-test: if: github.repository_owner == 'aws' name: aws-lc-rs FIPS - External bindgen test @@ -216,14 +251,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - name: Install bindgen-cli run: cargo install --locked bindgen-cli - name: Remove bindings @@ -245,12 +280,10 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -271,7 +304,7 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -306,15 +339,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' + - uses: seanmiddleditch/gha-setup-ninja@v5 - name: Run cargo test run: cargo test -p aws-lc-rs --tests --no-default-features --features fips - name: Release build @@ -346,14 +378,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -378,7 +410,7 @@ jobs: - macos-12 - macos-13-xlarge steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs index e25cecb443..c5069c5341 100644 --- a/aws-lc-sys/builder/cmake_builder.rs +++ b/aws-lc-sys/builder/cmake_builder.rs @@ -3,9 +3,9 @@ use crate::OutputLib::{Crypto, RustWrapper, Ssl}; use crate::{ - cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target, - target_arch, target_env, target_family, target_os, target_underscored, target_vendor, - OutputLibType, + allow_prebuilt_nasm, cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, + option_env, target, target_arch, target_env, target_family, target_os, target_underscored, + target_vendor, test_nasm_command, OutputLibType, }; use std::env; use std::ffi::OsString; @@ -22,10 +22,6 @@ fn test_clang_cl_command() -> bool { execute_command("clang-cl".as_ref(), &["--version".as_ref()]).status } -fn test_nasm_command() -> bool { - execute_command("nasm".as_ref(), &["-version".as_ref()]).status -} - fn find_cmake_command() -> Option<OsString> { if let Some(cmake) = option_env("CMAKE") { emit_warning(&format!( @@ -162,7 +158,7 @@ impl CmakeBuilder { // See issue: #453 if target_os() == "windows" { - Self::configure_windows(&mut cmake_cfg); + self.configure_windows(&mut cmake_cfg); } // If the build environment vendor is Apple @@ -213,7 +209,7 @@ impl CmakeBuilder { cmake_cfg } - fn configure_windows(cmake_cfg: &mut cmake::Config) { + fn configure_windows(&self, cmake_cfg: &mut cmake::Config) { match (target_env().as_str(), target_arch().as_str()) { ("msvc", "aarch64") => { cmake_cfg.generator_toolset(format!( @@ -243,6 +239,22 @@ impl CmakeBuilder { } _ => {} } + if target_arch() == "x86_64" && Some(true) == allow_prebuilt_nasm() { + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + emit_warning("!!! Using pre-built NASM binaries !!!"); + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + + let script_path = self + .manifest_dir + .join("builder") + .join("prebuilt-nasm.bat") + .display() + .to_string(); + let script_path = script_path.replace('\\', "/"); + + cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str()); + cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1"); + } } fn configure_open_harmony(cmake_cfg: &mut cmake::Config) { @@ -298,10 +310,14 @@ impl crate::Builder for CmakeBuilder { let mut missing_dependency = false; if target_os() == "windows" { - if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() { + if target_arch() == "x86_64" + && !is_no_asm() + && !test_nasm_command() + && Some(true) != allow_prebuilt_nasm() + { eprintln!( - "Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\ - See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html" + "Consider setting `AWS_LC_SYS_PREBUILT_NASM` in the build environment.\ + See User Guide: https://aws.github.io/aws-lc-rs/index.html" ); eprintln!("Missing dependency: nasm"); missing_dependency = true; diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs index 23f0214991..987e1038ca 100644 --- a/aws-lc-sys/builder/main.rs +++ b/aws-lc-sys/builder/main.rs @@ -319,6 +319,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false; static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_NO_ASM: bool = false; +static mut AWS_LC_SYS_PREBUILT_NASM: Option<bool> = None; fn initialize() { unsafe { @@ -328,6 +329,7 @@ fn initialize() { AWS_LC_SYS_EXTERNAL_BINDGEN = env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false); AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false); + AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM"); } if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) { @@ -363,6 +365,7 @@ fn is_bindgen_required() -> bool { || !has_pregenerated() } +#[allow(dead_code)] fn internal_bindgen_supported() -> bool { // TODO: internal bindgen creates invalid bindings on FreeBSD // See: #476 @@ -385,6 +388,10 @@ fn is_no_asm() -> bool { unsafe { AWS_LC_SYS_NO_ASM } } +fn allow_prebuilt_nasm() -> Option<bool> { + unsafe { AWS_LC_SYS_PREBUILT_NASM } +} + fn has_bindgen_feature() -> bool { cfg!(feature = "bindgen") } @@ -393,6 +400,10 @@ fn has_pregenerated() -> bool { unsafe { PREGENERATED } } +fn test_nasm_command() -> bool { + execute_command("nasm".as_ref(), &["-version".as_ref()]).status +} + fn prepare_cargo_cfg() { // This is supported in Rust >= 1.77.0 // Also remove `#![allow(unexpected_cfgs)]` from src/lib.rs diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat new file mode 100644 index 0000000000..9c761db12a --- /dev/null +++ b/aws-lc-sys/builder/prebuilt-nasm.bat @@ -0,0 +1,21 @@ +@echo off +set "ScriptDir=%~dp0" +set "ScriptDir=%ScriptDir:~0,-1%" +:loop +set "arg1=%~1" +if "%arg1%"=="-o" goto end +if "%arg1%"=="" goto failure +shift +goto loop +:end +shift +set "path=%~1" +for %%f in ("%path%") do set "filename=%%~nxf" +copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%" +exit 0 + +:failure +echo PATH: %path% 1>&2 +echo FILENAME: %filename% 1>&2 +echo ScriptDir: %ScriptDir% 1>&2 +exit 1 \ No newline at end of file diff --git a/scripts/build/collect_nasm_obj.sh b/scripts/build/collect_nasm_obj.sh new file mode 100644 index 0000000000..c425e9236f --- /dev/null +++ b/scripts/build/collect_nasm_obj.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 OR ISC + +set -ex +set -o pipefail + +if [[ ${BASH_VERSINFO[0]} -lt 4 ]]; then + echo Must use bash 4 or later: ${BASH_VERSION} + exit 1 +fi + +SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd) +REPO_ROOT=$(git rev-parse --show-toplevel) +SYS_CRATE_DIR="${REPO_ROOT}/aws-lc-sys" +PREBUILT_NASM_DIR="${SYS_CRATE_DIR}/builder/prebuilt-nasm" +mkdir -p "${PREBUILT_NASM_DIR}" +rm -f "${PREBUILT_NASM_DIR}"/* + +DUMPBIN="$(find /c/Program\ Files/Microsoft\ Visual\ Studio/ -path "*/Hostx64/x64/*" -name "dumpbin.exe" -print -quit)" + +for nasm_file in `find aws-lc-sys/aws-lc/generated-src/win-x86_64/ -name "*.asm"`; do + OBJNAME=$(basename "${nasm_file}"); + NASM_OBJ=$(find target/ -name "${OBJNAME/.asm/.obj}"); + cp "${NASM_OBJ}" "${PREBUILT_NASM_DIR}" + # We remove the '.debug$S' value, which indicates the size of the debug section. This value can change across builds + # because it typically contains full source file paths that vary by build environment + "${DUMPBIN}" //DISASM "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/.obj}" | grep -v '.debug$S' | sed -e "s/^Dump of file.*/Dump of file ${OBJNAME/.asm/.obj}/" > "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/}"-disasm.txt +done

@echo

Update CI diff --git a/.github/workflows/sys-bindings-generator.yml b/.github/workflows/sys-bindings-generator.yml index 3fba893d80..dc226f9f1b 100644 --- a/.github/workflows/sys-bindings-generator.yml +++ b/.github/workflows/sys-bindings-generator.yml @@ -270,3 +270,36 @@ jobs: run: ./scripts/build/collect_build_src.sh -t ${{ matrix.target }} - name: Commit & Push changes run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected source files for ${{ matrix.target }}" + collect-nasm-and-commit: + needs: generate-windows-bindings-and-commit + if: github.repository == 'aws/aws-lc-rs' + runs-on: windows-latest + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + ref: ${{ github.ref_name }} + - uses: dtolnay/rust-toolchain@master + id: toolchain + with: + toolchain: stable + targets: "x86_64-pc-windows-msvc,x86_64-pc-windows-gnu" + - uses: ilammy/setup-nasm@v1 + - name: Build aws-lc-sys + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo build -p aws-lc-sys --release --target x86_64-pc-windows-msvc + - name: Collect NASM object files + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Clean build + shell: bash + run: cargo clean + - name: Test aws-lc-rs for x86_64-pc-windows-msvc + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-msvc + - name: Test aws-lc-sys for x86_64-pc-windows-gnu + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-gnu + - name: Commit & Push changes + shell: bash + run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected NASM files" diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a0da3f48b8..71716a64c0 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -37,7 +37,7 @@ jobs: - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -66,7 +66,7 @@ jobs: - --no-default-features --features aws-lc-sys,bindgen,unstable - --release --all-targets --features bindgen,unstable steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - if: ${{ matrix.os == 'macos-13-xlarge' }} @@ -106,9 +106,10 @@ jobs: - --no-default-features --features non-fips,ring-io,unstable - --no-default-features --features non-fips,ring-sig-verify,unstable - --no-default-features --features non-fips,alloc,unstable + env: + AWS_LC_SYS_PREBUILT_NASM: 1 steps: - - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -125,7 +126,7 @@ jobs: name: aws-ls-rs coverage runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true @@ -167,7 +168,7 @@ jobs: - --no-default-features --features fips,asan runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@master @@ -196,7 +197,7 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -205,6 +206,40 @@ jobs: # See: rust-lang/cargo#8531 run: cargo test -p aws-lc-rs --tests + build-env-nasm-test: + if: github.repository_owner == 'aws' + name: prebuilt NASM verification + runs-on: windows-latest + strategy: + fail-fast: false + matrix: + target: + - 'x86_64-pc-windows-msvc' + - 'x86_64-pc-windows-gnu' + steps: + - uses: actions/checkout@v4 + with: + submodules: 'recursive' + - uses: dtolnay/rust-toolchain@stable + - name: Install NASM + uses: ilammy/setup-nasm@v1 + - name: Remove NASM artifacts + shell: bash + run: | + cargo clean + rm ./aws-lc-sys/builder/prebuilt-nasm/* + - name: Run cargo test + shell: bash + run: AWS_LC_SYS_PREBUILT_NASM=0 cargo test --tests -p aws-lc-rs --release --no-default-features --features aws-lc-sys + - name: Collect NASM outputs + shell: bash + run: ./scripts/build/collect_nasm_obj.sh + - name: Flag any NASM changes + shell: bash + run: | + git add . + git diff --cached --exit-code HEAD -- aws-lc-sys/builder/prebuilt-nasm/*.txt + build-env-external-bindgen-test: if: github.repository_owner == 'aws' name: aws-lc-rs FIPS - External bindgen test @@ -216,14 +251,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - name: Install bindgen-cli run: cargo install --locked bindgen-cli - name: Remove bindings @@ -245,12 +280,10 @@ jobs: os: [ ubuntu-latest, macos-12, macos-13-xlarge ] static: [ 0, 1 ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -271,7 +304,7 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable @@ -306,15 +339,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' + - uses: seanmiddleditch/gha-setup-ninja@v5 - name: Run cargo test run: cargo test -p aws-lc-rs --tests --no-default-features --features fips - name: Release build @@ -346,14 +378,14 @@ jobs: matrix: os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ] steps: - - if: ${{ matrix.os == 'windows-latest' }} - uses: ilammy/setup-nasm@v1 - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' + - if: ${{ matrix.os == 'windows-latest' }} + uses: ilammy/setup-nasm@v1 + - if: ${{ matrix.os == 'windows-latest' }} + uses: seanmiddleditch/gha-setup-ninja@v5 - uses: dtolnay/rust-toolchain@stable - - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 - uses: actions/setup-go@v4 with: go-version: '>=1.18' @@ -378,7 +410,7 @@ jobs: - macos-12 - macos-13-xlarge steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@v4 with: submodules: 'recursive' lfs: true diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs index e25cecb443..c5069c5341 100644 --- a/aws-lc-sys/builder/cmake_builder.rs +++ b/aws-lc-sys/builder/cmake_builder.rs @@ -3,9 +3,9 @@ use crate::OutputLib::{Crypto, RustWrapper, Ssl}; use crate::{ - cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target, - target_arch, target_env, target_family, target_os, target_underscored, target_vendor, - OutputLibType, + allow_prebuilt_nasm, cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, + option_env, target, target_arch, target_env, target_family, target_os, target_underscored, + target_vendor, test_nasm_command, OutputLibType, }; use std::env; use std::ffi::OsString; @@ -22,10 +22,6 @@ fn test_clang_cl_command() -> bool { execute_command("clang-cl".as_ref(), &["--version".as_ref()]).status } -fn test_nasm_command() -> bool { - execute_command("nasm".as_ref(), &["-version".as_ref()]).status -} - fn find_cmake_command() -> Option<OsString> { if let Some(cmake) = option_env("CMAKE") { emit_warning(&format!( @@ -162,7 +158,7 @@ impl CmakeBuilder { // See issue: aws#453 if target_os() == "windows" { - Self::configure_windows(&mut cmake_cfg); + self.configure_windows(&mut cmake_cfg); } // If the build environment vendor is Apple @@ -213,7 +209,7 @@ impl CmakeBuilder { cmake_cfg } - fn configure_windows(cmake_cfg: &mut cmake::Config) { + fn configure_windows(&self, cmake_cfg: &mut cmake::Config) { match (target_env().as_str(), target_arch().as_str()) { ("msvc", "aarch64") => { cmake_cfg.generator_toolset(format!( @@ -243,6 +239,22 @@ impl CmakeBuilder { } _ => {} } + if target_arch() == "x86_64" && Some(true) == allow_prebuilt_nasm() { + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + emit_warning("!!! Using pre-built NASM binaries !!!"); + emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"); + + let script_path = self + .manifest_dir + .join("builder") + .join("prebuilt-nasm.bat") + .display() + .to_string(); + let script_path = script_path.replace('\\', "/"); + + cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str()); + cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1"); + } } fn configure_open_harmony(cmake_cfg: &mut cmake::Config) { @@ -298,10 +310,14 @@ impl crate::Builder for CmakeBuilder { let mut missing_dependency = false; if target_os() == "windows" { - if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() { + if target_arch() == "x86_64" + && !is_no_asm() + && !test_nasm_command() + && Some(true) != allow_prebuilt_nasm() + { eprintln!( - "Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\ - See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html" + "Consider setting `AWS_LC_SYS_PREBUILT_NASM` in the build environment.\ + See User Guide: https://aws.github.io/aws-lc-rs/index.html" ); eprintln!("Missing dependency: nasm"); missing_dependency = true; diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs index 23f0214991..987e1038ca 100644 --- a/aws-lc-sys/builder/main.rs +++ b/aws-lc-sys/builder/main.rs @@ -319,6 +319,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false; static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false; static mut AWS_LC_SYS_NO_ASM: bool = false; +static mut AWS_LC_SYS_PREBUILT_NASM: Option<bool> = None; fn initialize() { unsafe { @@ -328,6 +329,7 @@ fn initialize() { AWS_LC_SYS_EXTERNAL_BINDGEN = env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false); AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false); + AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM"); } if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) { @@ -363,6 +365,7 @@ fn is_bindgen_required() -> bool { || !has_pregenerated() } +#[allow(dead_code)] fn internal_bindgen_supported() -> bool { // TODO: internal bindgen creates invalid bindings on FreeBSD // See: aws#476 @@ -385,6 +388,10 @@ fn is_no_asm() -> bool { unsafe { AWS_LC_SYS_NO_ASM } } +fn allow_prebuilt_nasm() -> Option<bool> { + unsafe { AWS_LC_SYS_PREBUILT_NASM } +} + fn has_bindgen_feature() -> bool { cfg!(feature = "bindgen") } @@ -393,6 +400,10 @@ fn has_pregenerated() -> bool { unsafe { PREGENERATED } } +fn test_nasm_command() -> bool { + execute_command("nasm".as_ref(), &["-version".as_ref()]).status +} + fn prepare_cargo_cfg() { // This is supported in Rust >= 1.77.0 // Also remove `#![allow(unexpected_cfgs)]` from src/lib.rs diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat new file mode 100644 index 0000000000..9c761db12a --- /dev/null +++ b/aws-lc-sys/builder/prebuilt-nasm.bat @@ -0,0 +1,21 @@ +@echo off +set "ScriptDir=%~dp0" +set "ScriptDir=%ScriptDir:~0,-1%" +:loop +set "arg1=%~1" +if "%arg1%"=="-o" goto end +if "%arg1%"=="" goto failure +shift +goto loop +:end +shift +set "path=%~1" +for %%f in ("%path%") do set "filename=%%~nxf" +copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%" +exit 0 + +:failure +echo PATH: %path% 1>&2 +echo FILENAME: %filename% 1>&2 +echo ScriptDir: %ScriptDir% 1>&2 +exit 1 \ No newline at end of file diff --git a/scripts/build/collect_nasm_obj.sh b/scripts/build/collect_nasm_obj.sh new file mode 100644 index 0000000000..c425e9236f --- /dev/null +++ b/scripts/build/collect_nasm_obj.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 OR ISC + +set -ex +set -o pipefail + +if [[ ${BASH_VERSINFO[0]} -lt 4 ]]; then + echo Must use bash 4 or later: ${BASH_VERSION} + exit 1 +fi + +SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd) +REPO_ROOT=$(git rev-parse --show-toplevel) +SYS_CRATE_DIR="${REPO_ROOT}/aws-lc-sys" +PREBUILT_NASM_DIR="${SYS_CRATE_DIR}/builder/prebuilt-nasm" +mkdir -p "${PREBUILT_NASM_DIR}" +rm -f "${PREBUILT_NASM_DIR}"/* + +DUMPBIN="$(find /c/Program\ Files/Microsoft\ Visual\ Studio/ -path "*/Hostx64/x64/*" -name "dumpbin.exe" -print -quit)" + +for nasm_file in `find aws-lc-sys/aws-lc/generated-src/win-x86_64/ -name "*.asm"`; do + OBJNAME=$(basename "${nasm_file}"); + NASM_OBJ=$(find target/ -name "${OBJNAME/.asm/.obj}"); + cp "${NASM_OBJ}" "${PREBUILT_NASM_DIR}" + # We remove the '.debug$S' value, which indicates the size of the debug section. This value can change across builds + # because it typically contains full source file paths that vary by build environment + "${DUMPBIN}" //DISASM "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/.obj}" | grep -v '.debug$S' | sed -e "s/^Dump of file.*/Dump of file ${OBJNAME/.asm/.obj}/" > "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/}"-disasm.txt +done

keehun-cf · 2024-09-24T20:44:34Z

I've noticed this exact error message when building on Windows 11 using clang/LLVM 19.1.0. As soon as I downgraded to 18.1.8, my builds just started working again.

I'd be happy to create a minimal reproduction case if anyone is interested.

justsmth · 2024-09-25T15:27:33Z

I'd be happy to create a minimal reproduction case if anyone is interested.

@keehun-cf - Yes, that would be great!

I've noticed this exact error message when building on Windows 11 using clang/LLVM 19.1.0. As soon as I downgraded to 18.1.8, my builds just started working again.

This is a very interesting data point. I'm curious whether you were using our (internal) "bindgen" feature when the build failed, or having the build use the (external) bindgen-cli? This can be forced either way by setting AWS_LC_SYS_EXTERNAL_BINDGEN=1 in the build environment.

Speculating... perhaps the bindgen-cli (latest version, v0.70.1) can work properly with clang/LLVM 19.1.0, while the bindgen version we depend on (v0.69.4) can not?

One concern I've had is that we're stuck at bindgen v0.69 due to bindgen v0.70 bumping the MSRV to 1.70. (Our MSRV is 1.63).

keehun-cf · 2024-09-25T15:29:35Z

I believe we were not using the external bindgen. I'll see when I will have time to create a minimal repro! I'm hoping that it's straightforward to repro by anyone else as well.

justsmth · 2024-10-02T18:31:42Z

Ok. I reproduced this on my Mac. Yesterday I updated to llvm v19.1.0, and started seeing these errors when I used the "bindgen" feature. And when I went back to llvm@18, the problem went away!

So now the questions are: (1) is rust-bindgen aware of this issue, (2) if so, do they have a fix or workaround; (3) if there's no fix, can we reliably detect whether the build environment is using llvm@19; (4) if so, can we fallback to using bindgen-cli?

🤔

justsmth · 2024-10-02T20:16:56Z

After a few iterations of "git bisect" on the rust-bindgen repo, I think I narrowed it down to this commit that fixed the problem when using llvm v19.1: rust-lang/rust-bindgen@600f638

How does that fix it? I'm not sure.

clang_getSpellingLocation - https://clang.llvm.org/doxygen/group__CINDEX__LOCATIONS.html#ga01f1a342f7807ea742aedd2c61c46fa0
clang_getFileLocation - https://clang.llvm.org/doxygen/group__CINDEX__LOCATIONS.html#gae0ee9ff0ea04f2446832fc12a7fd2ac8

justsmth · 2024-10-03T17:14:33Z

Possible workarounds:

Downgrade LLVM to v18 or earlier.
Install the latest version of bindgen-cli (cargo install --force --locked bindgen-cli) and set AWS_LC_SYS_EXTERNAL_BINDGEN=1 in the build environment.
Don't enable the "bindgen" feature if not needed for the target platform. (This is unlikely, and not possible in most cases.)

justsmth · 2024-10-09T14:03:31Z

With the release of bindgen 0.69.5 this problem should now be resolved. I've also got a PR up to do some cleanup partially related to this. It should be part of our next patch release.

Thanks for your patience while getting this resolved!

rami3l · 2024-10-09T16:32:17Z

@justsmth Sorry, did you mean the FreeBSD-related issue has been resolved, or the llvm-related one? I thought this issue is specifically for the former...

justsmth · 2024-10-09T17:12:11Z

@rami3l -- I believe the FreeBSD issue was actually just an LLVM 19 issue.

To verify, I've put up this PR that removes the "bindgen-cli" installation for the FreeBSD tests. (It also makes it so that FreeBSD can use the "internal" bindgen.) And the CI tests are passing for both FreeBSD 13.4 and 14.1: https://github.com/aws/aws-lc-rs/actions/runs/11241275743/job/31252533681?pr=561 🎉

...
  warning: aws-lc-sys@0.22.0: Building with: CMake
  warning: aws-lc-sys@0.22.0: Symbol Prefix: Some("aws_lc_0_22_0")
  warning: aws-lc-sys@0.22.0: CMAKE environment variable set: cmake
  warning: aws-lc-sys@0.22.0: Clang version: clang version 19.0.0git
  warning: aws-lc-sys@0.22.0: Generating bindings - internal bindgen. Platform: x86_64-unknown-freebsd
  warning: aws-lc-sys@0.22.0: Compilation of 'c11.c' succeeded - Ok(["/home/runner/work/aws-lc-rs/aws-lc-rs/target/debug/build/aws-lc-sys-4f788bd60c4b19d9/out/out-c11/89b08a3674570fd2-c11.o"]).
  warning: aws-lc-sys@0.22.0: Setting CFLAGS: "-O0 -ffunction-sections -fdata-sections -fPIC -gdwarf-2 -fno-omit-frame-pointer -m64 --target=x86_64-unknown-freebsd -std=c11 -I /home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-sys/generated-include -I /home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-sys/include -I /home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-sys/aws-lc/include -I /home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-sys/aws-lc/third_party/s2n-bignum/include -Wall -Wextra -Wno-unused-parameter -DBORINGSSL_IMPLEMENTATION=1 -DBORINGSSL_PREFIX=aws_lc_0_22_0"
      Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 01s
       Running unittests src/lib.rs (target/debug/deps/aws_lc_rs-e8cb4d23e3a14b95)
  
  running 196 tests
  test aead::chacha20_poly1305_openssh::tests::derive_poly1305_test ... ok
...

rami3l · 2024-10-10T00:41:27Z

@rami3l -- I believe the FreeBSD issue was actually just an LLVM 19 issue.

To verify, I've put up this PR that removes the "bindgen-cli" installation for the FreeBSD tests. (It also makes it so that FreeBSD can use the "internal" bindgen.) And the CI tests are passing for both FreeBSD 13.4 and 14.1: https://github.com/aws/aws-lc-rs/actions/runs/11241275743/job/31252533681?pr=561 🎉


...

  warning: aws-lc-sys@0.22.0: Building with: CMake

  warning: aws-lc-sys@0.22.0: Symbol Prefix: Some("aws_lc_0_22_0")

  warning: aws-lc-sys@0.22.0: CMAKE environment variable set: cmake

  warning: aws-lc-sys@0.22.0: Clang version: clang version 19.0.0git

  warning: aws-lc-sys@0.22.0: Generating bindings - internal bindgen. Platform: x86_64-unknown-freebsd

  warning: aws-lc-sys@0.22.0: Compilation of 'c11.c' succeeded - Ok(["/home/runner/work/aws-lc-rs/aws-lc-rs/target/debug/build/aws-lc-sys-4f788bd60c4b19d9/out/out-c11/89b08a3674570fd2-c11.o"]).

  warning: aws-lc-sys@0.22.0: Setting CFLAGS: "-O0 -ffunction-sections -fdata-sections -fPIC -gdwarf-2 -fno-omit-frame-pointer -m64 --target=x86_64-unknown-freebsd -std=c11 -I /home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-sys/generated-include -I /home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-sys/include -I /home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-sys/aws-lc/include -I /home/runner/work/aws-lc-rs/aws-lc-rs/aws-lc-sys/aws-lc/third_party/s2n-bignum/include -Wall -Wextra -Wno-unused-parameter -DBORINGSSL_IMPLEMENTATION=1 -DBORINGSSL_PREFIX=aws_lc_0_22_0"

      Finished `test` profile [unoptimized + debuginfo] target(s) in 1m 01s

       Running unittests src/lib.rs (target/debug/deps/aws_lc_rs-e8cb4d23e3a14b95)

  

  running 196 tests

  test aead::chacha20_poly1305_openssh::tests::derive_poly1305_test ... ok

...

@justsmth Aha, good to hear, and thanks again!

Link: aws/aws-lc-rs#476

justsmth self-assigned this Aug 1, 2024

justsmth mentioned this issue Aug 1, 2024

Add CI test for FreeBSD #477

Merged

rami3l mentioned this issue Aug 1, 2024

ci(freebsd): fix build failure related to aws-lc rust-lang/rustup#3979

Merged

justsmth added the build problem Build failure label Aug 1, 2024

yonas mentioned this issue Aug 12, 2024

Compiling errors from aws-lc-rs: "cannot find function, tuple struct or tuple variant..." iggy-rs/iggy#1125

Closed

justsmth added the bug Something isn't working label Oct 2, 2024

justsmth mentioned this issue Oct 2, 2024

Request for a v0.69.5 w/ LLVM v19.1 support rust-lang/rust-bindgen#2945

Closed

justsmth mentioned this issue Oct 4, 2024

Mitigate bindgen 0.69 w/ clang 19.1 incompatibility #559

Closed

justsmth closed this as completed Oct 9, 2024

yonas mentioned this issue Oct 9, 2024

Backport commit 35f0f9a to 0.70.x series (clang_getFileLocation instead of clang_getSpellingLocation) rust-lang/rust-bindgen#2948

Closed

rami3l changed the title ~~Failed to build aws-lc-rs for FreeBSD x64 due to rustc error E0425~~ Failed to build aws-lc-rs with LLVM v19.1 due to rustc error E0425 Oct 10, 2024

justsmth mentioned this issue Nov 15, 2024

New release for aws-lc-sys, please #591

Closed

rami3l mentioned this issue Dec 23, 2024

Request: Update the User Guide to mention more *nix systems #643

Open

Cryolitia added a commit to Cryolitia-Forks/archriscv-packages that referenced this issue Feb 20, 2025

updpatch: python-proton-vpn-local-agent 1.4.3

81c62ba

Link: aws/aws-lc-rs#476

Cryolitia mentioned this issue Feb 20, 2025

updpatch: python-proton-vpn-local-agent 1.4.3 felixonmars/archriscv-packages#4559

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to build `aws-lc-rs` with LLVM v19.1 due to `rustc` error E0425 #476

Failed to build `aws-lc-rs` with LLVM v19.1 due to `rustc` error E0425 #476

rami3l commented Aug 1, 2024 •

edited

Loading

justsmth commented Aug 1, 2024

justsmth commented Aug 1, 2024

justsmth commented Aug 1, 2024

justsmth commented Aug 1, 2024 •

edited

Loading

rami3l commented Aug 1, 2024 •

edited

Loading

keehun-cf commented Sep 24, 2024

justsmth commented Sep 25, 2024

keehun-cf commented Sep 25, 2024 •

edited

Loading

justsmth commented Oct 2, 2024

justsmth commented Oct 2, 2024 •

edited

Loading

justsmth commented Oct 3, 2024 •

edited

Loading

justsmth commented Oct 9, 2024

rami3l commented Oct 9, 2024 •

edited

Loading

justsmth commented Oct 9, 2024 •

edited

Loading

rami3l commented Oct 10, 2024

Failed to build aws-lc-rs with LLVM v19.1 due to rustc error E0425 #476

Failed to build aws-lc-rs with LLVM v19.1 due to rustc error E0425 #476

Comments

rami3l commented Aug 1, 2024 • edited Loading

Problem

Relevant details

justsmth commented Aug 1, 2024

justsmth commented Aug 1, 2024

justsmth commented Aug 1, 2024

justsmth commented Aug 1, 2024 • edited Loading

rami3l commented Aug 1, 2024 • edited Loading

keehun-cf commented Sep 24, 2024

justsmth commented Sep 25, 2024

keehun-cf commented Sep 25, 2024 • edited Loading

justsmth commented Oct 2, 2024

justsmth commented Oct 2, 2024 • edited Loading

justsmth commented Oct 3, 2024 • edited Loading

justsmth commented Oct 9, 2024

rami3l commented Oct 9, 2024 • edited Loading

justsmth commented Oct 9, 2024 • edited Loading

rami3l commented Oct 10, 2024

Failed to build `aws-lc-rs` with LLVM v19.1 due to `rustc` error E0425 #476

Failed to build `aws-lc-rs` with LLVM v19.1 due to `rustc` error E0425 #476

rami3l commented Aug 1, 2024 •

edited

Loading

justsmth commented Aug 1, 2024 •

edited

Loading

rami3l commented Aug 1, 2024 •

edited

Loading

keehun-cf commented Sep 25, 2024 •

edited

Loading

justsmth commented Oct 2, 2024 •

edited

Loading

justsmth commented Oct 3, 2024 •

edited

Loading

rami3l commented Oct 9, 2024 •

edited

Loading

justsmth commented Oct 9, 2024 •

edited

Loading