Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generator/linux: Use the Swift 6.0 toolchain's lld #159

Merged
merged 1 commit into from
Nov 28, 2024

Conversation

euanh
Copy link
Contributor

@euanh euanh commented Nov 28, 2024

After Swift 6.0 the swift.org toolchain includes LLD, so it is not
necessary to install it separately from an LLVM archive. The LLD
provided by the swift.org toolchain is also a multiarch binary, so
the SDK will run on both x86_64 and aarch64 macOS hosts.

This change slightly reduces the size of a Swift 6.0 SDK because
currently these SDKs include two copies of LLD - one from the SDK
and one from LLVM. This is because LLD is typically installed as
a single binary with multiple symlinks pointing to it representing
different 'personalities' such as ld.lld, wasm-ld and so on. SDK
generator copies the lld binary in as ld.lld, overwriting one of
the symlinks:

-rwxr-xr-x   1 euanh  staff   108M 14 Nov  2023 ld.lld
lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 ld64.lld -> lld
-rwxr-xr-x   1 euanh  staff   265M 24 Oct 19:28 lld
lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 lld-link -> lld
lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 wasm-ld -> lld

After this PR, we only have one copy of the lld binary:

lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 ld.lld -> lld
lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 ld64.lld -> lld
-rwxr-xr-x   1 euanh  staff   265M 24 Oct 19:28 lld
lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 lld-link -> lld
lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 wasm-ld -> lld

This change does not significantly change the time to build a new
SDK with a warm cache (PR #149), however the very first build with
a cold cache improves considerably because the LLVM archive does
not need to be downloaded and unpacked.

For example, two tests run one after the other on an M3 MacBook Air
with a fast network connection:

rm -rf Artifacts Bundles
swift run swift-sdk-generator make-linux-sdk --swift-version 6.0.2-RELEASE

main: 1m21s, (2.9GB in Artifacts directory)
this PR: 39s, (2.1GB in Artifacts directory)

After Swift 6.0 the swift.org toolchain includes LLD, so it is not
necessary to install it separately from an LLVM archive.   The LLD
provided by the swift.org toolchain is also a multiarch binary, so
the SDK will run on both x86_64 and aarch64 macOS hosts.

This change slightly reduces the size of a Swift 6.0 SDK because
currently these SDKs include two copies of LLD - one from the SDK
and one from LLVM.   This is because LLD is typically installed as
a single binary with multiple symlinks pointing to it representing
different 'personalities' such as `ld.lld`, `wasm-ld` and so on.   SDK
generator copies the `lld` binary in as `ld.lld`, overwriting one of
the symlinks:

    -rwxr-xr-x   1 euanh  staff   108M 14 Nov  2023 ld.lld
    lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 ld64.lld -> lld
    -rwxr-xr-x   1 euanh  staff   265M 24 Oct 19:28 lld
    lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 lld-link -> lld
    lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 wasm-ld -> lld

After this PR, we only have one copy of the `lld` binary:

    lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 ld.lld -> lld
    lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 ld64.lld -> lld
    -rwxr-xr-x   1 euanh  staff   265M 24 Oct 19:28 lld
    lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 lld-link -> lld
    lrwxr-xr-x   1 euanh  staff     3B 24 Oct 15:11 wasm-ld -> lld

This change does not significantly change the time to build a new
SDK with a warm cache (PR swiftlang#149), however the very first build with
a cold cache improves considerably because the LLVM archive does
not need to be downloaded and unpacked.

For example, two tests run one after the other on an M3 MacBook Air
with a fast network connection:

    rm -rf Artifacts Bundles
    swift run swift-sdk-generator make-linux-sdk --swift-version 6.0.2-RELEASE

`main`: 1m21s,     (2.9GB in `Artifacts` directory)
this PR: 39s,    (2.1GB in `Artifacts` directory)
@euanh euanh requested a review from MaxDesiatov as a code owner November 28, 2024 13:21
@euanh
Copy link
Contributor Author

euanh commented Nov 28, 2024

@swift-ci test

Copy link
Contributor

@MaxDesiatov MaxDesiatov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be covered with tests?

@euanh
Copy link
Contributor Author

euanh commented Nov 28, 2024

Can this be covered with tests?

The EndToEndTests exercise this. The CI can't currently run them because of #145:

SWIFT_SDK_GENERATOR_RUN_SLOW_TESTS=1 swift test --disable-swift-testing --filter EndToEndTests
...
Test Suite 'swift-sdk-generatorPackageTests.xctest' passed at 2024-11-28 14:32:12.162.
         Executed 18 tests, with 0 failures (0 unexpected) in 2268.484 (2268.485) seconds
Test Suite 'Selected tests' passed at 2024-11-28 14:32:12.162.
         Executed 18 tests, with 0 failures (0 unexpected) in 2268.484 (2268.486) seconds

(I ran them before submitting but I'd lost the contents of the terminal. This is a fresh run.)

@euanh euanh marked this pull request as draft November 28, 2024 13:33
@euanh euanh marked this pull request as ready for review November 28, 2024 14:38
@euanh euanh merged commit 02ef96f into swiftlang:main Nov 28, 2024
3 checks passed
@euanh euanh deleted the use-lld-from-swift6-toolchain branch November 28, 2024 14:41
@euanh euanh added enhancement New feature or request performance Speeding up SDK generation labels Nov 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance Speeding up SDK generation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants