Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support raw-dylib link kind on ELF #135695

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Noratrieb
Copy link
Member

@Noratrieb Noratrieb commented Jan 18, 2025

raw-dylib is a link kind that allows rustc to link against a library without having any library files present.
This currently only exists on Windows. rustc will take all the symbols from raw-dylib link blocks and put them in an import library, where they can then be resolved by the linker.

While import libraries don't exist on ELF, it would still be convenient to have this same functionality. Not having the libraries present at build-time can be convenient for several reasons, especially cross-compilation. With raw-dylib, code linking against a library can be cross-compiled without needing to have these libraries available on the build machine. If the libc crate makes use of this, it would allow cross-compilation without having any libc available on the build machine. This is not yet possible with this implementation, at least against libc's like glibc that use symbol versioning. The raw-dylib kind could be extended with support for symbol versioning in the future.

This implementation is very experimental and I have not tested it very well. I have tested it for a toy example and the lz4-sys crate, where it was able to successfully link a binary despite not having a corresponding library at build-time.

I was inspired by Björn's comments in https://internals.rust-lang.org/t/bundle-zig-cc-in-rustup-by-default/22096/27
Tracking issue: #135694

r? bjorn3

try-job: aarch64-apple
try-job: x86_64-msvc-1
try-job: x86_64-msvc-2

@rustbot rustbot added A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jan 18, 2025
@rustbot
Copy link
Collaborator

rustbot commented Jan 18, 2025

These commits modify compiler targets.
(See the Target Tier Policy.)

Some changes occurred in src/tools/compiletest

cc @jieyouxu

@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

@Noratrieb Noratrieb force-pushed the elf-raw-dylib branch 2 times, most recently from 7b85a6a to f339acd Compare January 18, 2025 21:30
@rustbot

This comment was marked as outdated.

@rustbot rustbot added the A-run-make Area: port run-make Makefiles to rmake.rs label Jan 18, 2025
@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

Copy link
Member

@jieyouxu jieyouxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compiletest changes LGTM, would be great if you could add the two new directives to the directives listing in r-d-g

compiler/rustc_target/src/spec/mod.rs Show resolved Hide resolved
src/tools/compiletest/src/directive-list.rs Show resolved Hide resolved
@Noratrieb Noratrieb force-pushed the elf-raw-dylib branch 2 times, most recently from eec2be6 to 7ff0d14 Compare February 2, 2025 12:59
@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

Copy link
Member

@bjorn3 bjorn3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

r=me with review comments fixed

@Noratrieb
Copy link
Member Author

@bors r=bjorn3

@bors
Copy link
Contributor

bors commented Feb 2, 2025

📌 Commit f62d1ed has been approved by bjorn3

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 2, 2025
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Feb 3, 2025
Support raw-dylib link kind on ELF

raw-dylib is a link kind that allows rustc to link against a library without having any library files present.
This currently only exists on Windows. rustc will take all the symbols from raw-dylib link blocks and put them in an import library, where they can then be resolved by the linker.

While import libraries don't exist on ELF, it would still be convenient to have this same functionality. Not having the libraries present at build-time can be convenient for several reasons, especially cross-compilation. With raw-dylib, code linking against a library can be cross-compiled without needing to have these libraries available on the build machine. If the libc crate makes use of this, it would allow cross-compilation without having any libc available on the build machine. This is not yet possible with this implementation, at least against libc's like glibc that use symbol versioning. The raw-dylib kind could be extended with support for symbol versioning in the future.

This implementation is very experimental and I have not tested it very well. I have tested it for a toy example and the lz4-sys crate, where it was able to successfully link a binary despite not having a corresponding library at build-time.

I was inspired by Björn's comments in https://internals.rust-lang.org/t/bundle-zig-cc-in-rustup-by-default/22096/27
Tracking issue: rust-lang#135694

r? bjorn3
bors added a commit to rust-lang-ci/rust that referenced this pull request Feb 3, 2025
…iaskrgr

Rollup of 8 pull requests

Successful merges:

 - rust-lang#134777 (Enable more tests on Windows)
 - rust-lang#134807 (fix(rustdoc): always use a channel when linking to doc.rust-lang.org)
 - rust-lang#135621 (Move some std tests to integration tests)
 - rust-lang#135695 (Support raw-dylib link kind on ELF)
 - rust-lang#135836 (bootstrap: only build `crt{begin,end}.o` when compiling to MUSL)
 - rust-lang#136235 (Pretty print pattern type values with transmute if they don't satisfy their pattern)
 - rust-lang#136392 (bootstrap: add wrapper macros for `feature = "tracing"`-gated `tracing` macros)
 - rust-lang#136462 (mir_build: Simplify `lower_pattern_range_endpoint`)

r? `@ghost`
`@rustbot` modify labels: rollup
@matthiaskrgr
Copy link
Member

@bors r-
#136463 (comment)

@bors bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Feb 3, 2025
@@ -0,0 +1,15 @@
//@ only-elf
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The x86_64-fortanix-unknown-sgx target doesn't support dynamic linking. Can you please add //@ ignore-sgx to ignore this platform?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would //@ needs-dynamic-linking work too?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I will add that directive, I won't add any target specific directive for no reason. @raoulstrackx please don't suggest ignoring SGX directly in the future, instead suggest using directives like needs-dynamic-linking or similar to achieve what we need.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I didn't know about the needs-dynamic-linking directive. Good to know!

tests/run-make/raw-dylib-elf/rmake.rs Show resolved Hide resolved
raw-dylib is a link kind that allows rustc to link against a library
without having any library files present.
This currently only exists on Windows. rustc will take all the symbols
from raw-dylib link blocks and put them in an import library, where they
can then be resolved by the linker.

While import libraries don't exist on ELF, it would still be convenient
to have this same functionality. Not having the libraries present at
build-time can be convenient for several reasons, especially
cross-compilation. With raw-dylib, code linking against a library can be
cross-compiled without needing to have these libraries available on the
build machine. If the libc crate makes use of this, it would allow
cross-compilation without having any libc available on the build
machine. This is not yet possible with this implementation, at least
against libc's like glibc that use symbol versioning.
The raw-dylib kind could be extended with support for symbol versioning
in the future.

This implementation is very experimental and I have not tested it very
well. I have tested it for a toy example and the lz4-sys crate, where it
was able to successfully link a binary despite not having a
corresponding library at build-time.
@Noratrieb
Copy link
Member Author

@bors try

@bors
Copy link
Contributor

bors commented Feb 3, 2025

⌛ Trying commit 9527c48 with merge 93676c5...

bors added a commit to rust-lang-ci/rust that referenced this pull request Feb 3, 2025
Support raw-dylib link kind on ELF

raw-dylib is a link kind that allows rustc to link against a library without having any library files present.
This currently only exists on Windows. rustc will take all the symbols from raw-dylib link blocks and put them in an import library, where they can then be resolved by the linker.

While import libraries don't exist on ELF, it would still be convenient to have this same functionality. Not having the libraries present at build-time can be convenient for several reasons, especially cross-compilation. With raw-dylib, code linking against a library can be cross-compiled without needing to have these libraries available on the build machine. If the libc crate makes use of this, it would allow cross-compilation without having any libc available on the build machine. This is not yet possible with this implementation, at least against libc's like glibc that use symbol versioning. The raw-dylib kind could be extended with support for symbol versioning in the future.

This implementation is very experimental and I have not tested it very well. I have tested it for a toy example and the lz4-sys crate, where it was able to successfully link a binary despite not having a corresponding library at build-time.

I was inspired by Björn's comments in https://internals.rust-lang.org/t/bundle-zig-cc-in-rustup-by-default/22096/27
Tracking issue: rust-lang#135694

r? bjorn3

try-job: aarch64-apple
try-job: x86_64-msvc-1
try-job: x86_64-msvc-2
@rust-log-analyzer
Copy link
Collaborator

The job x86_64-gnu-llvm-18 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
#21 exporting to docker image format
#21 sending tarball 28.4s done
#21 DONE 34.1s
##[endgroup]
Setting extra environment values for docker:  --env ENABLE_GCC_CODEGEN=1 --env GCC_EXEC_PREFIX=/usr/lib/gcc/
[CI_JOB_NAME=x86_64-gnu-llvm-18]
debug: `DISABLE_CI_RUSTC_IF_INCOMPATIBLE` configured.
---
sccache: Starting the server...
##[group]Configure the build
configure: processing command line
configure: 
configure: build.configure-args := ['--build=x86_64-unknown-linux-gnu', '--llvm-root=/usr/lib/llvm-18', '--enable-llvm-link-shared', '--set', 'rust.randomize-layout=true', '--set', 'rust.thin-lto-import-instr-limit=10', '--enable-verbose-configure', '--enable-sccache', '--disable-manage-submodules', '--enable-locked-deps', '--enable-cargo-native-static', '--set', 'rust.codegen-units-std=1', '--set', 'dist.compression-profile=balanced', '--dist-compression-formats=xz', '--set', 'rust.lld=false', '--disable-dist-src', '--release-channel=nightly', '--enable-debug-assertions', '--enable-overflow-checks', '--enable-llvm-assertions', '--set', 'rust.verify-llvm-ir', '--set', 'rust.codegen-backends=llvm,cranelift,gcc', '--set', 'llvm.static-libstdcpp', '--enable-new-symbol-mangling']
configure: target.x86_64-unknown-linux-gnu.llvm-config := /usr/lib/llvm-18/bin/llvm-config
configure: llvm.link-shared     := True
configure: rust.randomize-layout := True
configure: rust.thin-lto-import-instr-limit := 10
---
---- [ui] tests/ui/feature-gates/feature-gate-raw-dylib-elf.rs stdout ----
Saved the actual stderr to "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/feature-gates/feature-gate-raw-dylib-elf/feature-gate-raw-dylib-elf.stderr"
diff of stderr:

1 error[E0658]: link kind `raw-dylib` is unstable on ELF platforms
+   --> $DIR/feature-gate-raw-dylib-elf.rs:4:30
3    |
3    |
4 LL | #[link(name = "meow", kind = "raw-dylib")]

Note: some mismatched output was normalized before being compared
-   --> /checkout/tests/ui/feature-gates/feature-gate-raw-dylib-elf.rs:4:30
+   --> $DIR/feature-gate-raw-dylib-elf.rs:4:30
---
To only update this specific test, also pass `--test-args feature-gates/feature-gate-raw-dylib-elf.rs`

error: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/tests/ui/feature-gates/feature-gate-raw-dylib-elf.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2" "--target=x86_64-unknown-linux-gnu" "--check-cfg" "cfg(test,FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/feature-gates/feature-gate-raw-dylib-elf" "-A" "unused" "-A" "internal_features" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers"
--- stderr -------------------------------
--- stderr -------------------------------
error[E0658]: link kind `raw-dylib` is unstable on ELF platforms
   |
   |
LL | #[link(name = "meow", kind = "raw-dylib")] //~ ERROR: link kind `raw-dylib` is unstable on ELF platforms
   |
   = note: see issue #135694 <https://github.com/rust-lang/rust/issues/135694> for more information
   = note: see issue #135694 <https://github.com/rust-lang/rust/issues/135694> for more information
   = help: add `#![feature(raw_dylib_elf)]` to the crate attributes to enable
   = note: this compiler was built on YYYY-MM-DD; consider upgrading it if it is out of date
error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0658`.
------------------------------------------

@rust-log-analyzer
Copy link
Collaborator

The job aarch64-apple failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
Updating files:  98% (50851/51762)
Updating files:  99% (51245/51762)
Updating files: 100% (51762/51762)
Updating files: 100% (51762/51762), done.
Switched to a new branch 'try'
branch 'try' set up to track 'origin/try'.
[command]/opt/homebrew/bin/git log -1 --format=%H
93676c544cc358dc13104751b0cc53a151774150
##[group]Run src/ci/scripts/setup-environment.sh
src/ci/scripts/setup-environment.sh
---
file:.git/config remote.origin.url=https://github.com/rust-lang-ci/rust
file:.git/config remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
file:.git/config gc.auto=0
file:.git/config http.https://github.com/.extraheader=AUTHORIZATION: basic ***
file:.git/config branch.try.remote=origin
file:.git/config branch.try.merge=refs/heads/try
file:.git/config remote.upstream.fetch=+refs/heads/*:refs/remotes/upstream/*
file:.git/config submodule.library/backtrace.active=true
file:.git/config submodule.library/backtrace.url=https://github.com/rust-lang/backtrace-rs.git
file:.git/config submodule.library/stdarch.active=true
---
test [ui] tests/ui/zero-sized/zero-sized-tuple-struct.rs ... ok

failures:

---- [ui] tests/ui/linkage-attr/raw-dylib/windows/raw-dylib-windows-only.rs#notelf stdout ----
Saved the actual stderr to "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/test/ui/linkage-attr/raw-dylib/windows/raw-dylib-windows-only.notelf/raw-dylib-windows-only.notelf.stderr"

3    |
3    |
4 LL | #[link(name = "foo", kind = "raw-dylib")]
-    |
7 
8 error: aborting due to 1 previous error
9 
9 


The actual stderr differed from the expected stderr.
To update references, rerun the tests and pass the `--bless` flag
To only update this specific test, also pass `--test-args linkage-attr/raw-dylib/windows/raw-dylib-windows-only.rs`

error in revision `notelf`: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/stage2/bin/rustc" "/Users/runner/work/rust/rust/tests/ui/linkage-attr/raw-dylib/windows/raw-dylib-windows-only.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/Users/runner/.cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/Users/runner/work/rust/rust/vendor" "--sysroot" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/stage2" "--target=aarch64-apple-darwin" "--cfg" "notelf" "--check-cfg" "cfg(test,FALSE,elf,notelf)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/test/ui/linkage-attr/raw-dylib/windows/raw-dylib-windows-only.notelf" "-A" "unused" "-A" "internal_features" "-Crpath" "-Cdebuginfo=0" "-Lnative=/Users/runner/work/rust/rust/build/aarch64-apple-darwin/native/rust-test-helpers" "--crate-type" "lib"
--- stderr -------------------------------
error[E0455]: link kind `raw-dylib` is only supported on Windows targets
##[error]  --> /Users/runner/work/rust/rust/tests/ui/linkage-attr/raw-dylib/windows/raw-dylib-windows-only.rs:6:29
   |
   |
LL | #[link(name = "foo", kind = "raw-dylib")]

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0455`.

@bors
Copy link
Contributor

bors commented Feb 3, 2025

💔 Test failed - checks-actions

@bors
Copy link
Contributor

bors commented Feb 7, 2025

☔ The latest upstream changes (presumably #136684) made this pull request unmergeable. Please resolve the merge conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-compiletest Area: The compiletest test runner A-run-make Area: port run-make Makefiles to rmake.rs A-rustc-dev-guide Area: rustc-dev-guide A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants