Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build Swift SDKs for a Linux host if the --host parameter matches a Linux OS (*-unknown-linux-gnu) #167

Merged
merged 15 commits into from
Jan 16, 2025
Merged

Build Swift SDKs for a Linux host if the --host parameter matches a Linux OS (*-unknown-linux-gnu) #167

merged 15 commits into from
Jan 16, 2025

Conversation

xtremekforever
Copy link
Contributor

@xtremekforever xtremekforever commented Jan 9, 2025 •
edited
Loading

This fixes #114 and provides the following:

  • If the --host param or auto-detected hostTriple.os is .linux, the amazonlinux2 Swift toolchain is downloaded and included in the Swift SDK. Examples:
swift run swift-sdk-generator make-linux-sdk --swift-version 6.0.3-RELEASE --host x86_64-unknown-linux-gnu --target aarch64-unknown-linux-gnu
swift run swift-sdk-generator make-linux-sdk --swift-version 5.10.1-RELEASE --host aarch64-unknown-linux-gnu --target x86_64-unknown-linux-gnu
  • I chose amazonlinux2 for the host toolchain as it seems to be compatible with every Swift-supported Linux distribution with glibc versions and included libraries.
  • Updated the README.md to show that Ubuntu, RHEL, Amazon Linux 2, and Debian are officially supported as hosts and somewhat supported as targets... :)

For testing all of this, I updated the EndToEndTests:

  • If theSWIFT_SDK_GENERATOR_TEST_LINUX_SWIFT_SDKS env variable is defined, Swift SDKs are built for a Linux host with the arch matching the machine on which the tests are running. So on an M-series Mac, the triple aarch64-unknown-linux-gnu is used. Or on an Intel Mac or PC, then x86_64-unknown-linux-gnu is used as the --host triple.
  • When Linux Swift SDKs are being tested, then a matrix of Swift docker containers are run against each test case to ensure the SDK works on every Linux distribution:
[SwiftSDKGeneratorTests] Creating test project /tmp/FDFD488C-089B-4213-B927-0DE0F9BD6B3A/swift-sdk-generator-test
[SwiftSDKGeneratorTests] Updating minimum swift-tools-version in test project...
[SwiftSDKGeneratorTests] Building test project in 6.0-focal container
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-focal container with static-swift-stdlib
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-jammy container
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-jammy container with static-swift-stdlib
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-noble container
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-noble container with static-swift-stdlib
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-fedora39 container
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-fedora39 container with static-swift-stdlib
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-rhel-ubi9 container
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-rhel-ubi9 container with static-swift-stdlib
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-amazonlinux2 container
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-amazonlinux2 container with static-swift-stdlib
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-bookworm container
[SwiftSDKGeneratorTests] Test project built successfully
[SwiftSDKGeneratorTests] Building test project in 6.0-bookworm container with static-swift-stdlib
[SwiftSDKGeneratorTests] Test project built successfully

I do have a few thoughts/questions:

  • Now we can generate Swift SDKs that work for Linux hosts, or any host, but there is no way to denote this in the SDK name. Should we consider adding some annotation to the end to show what hosts are compatible?
    • -linux-x86_64
    • -linux-aarch64
    • -macos
    • -all
  • Do we need some unit testing for this stuff, or would the EndToEndTests changes be sufficient to cover this new functionality?
  • The EndToEndTests will actually test generating a Swift SDK with the same host and target arch (for example, x86_64-unknown-linux-gnu for both the host and target). Should we skip running these tests, or are they valid test cases although we are targeting the same arch for the host and target?

@xtremekforever
Update all package dependencies for Ubuntu 24.04 host
e8b8604
@xtremekforever
Cleanup Linux host toolchain, remove lldb
b4fb926
@xtremekforever
Add note about using amazonlinux2 for host toolchain
0e05b69
@xtremekforever
Update README.md version matrix to add new supported hosts and targets
63482fa
@xtremekforever
Improve logging when extracting deb files
f6e812b
@xtremekforever
Resolve errors when generating Swift SDK for Linux host
b9822fc 
 - No need to fetch llvm or prepare lld since the host toolchain for Linux already contains it.
 - Only create swift_static/clang symlink if it does not exist.
@xtremekforever
Use Tests/.build directory for EndToEndTests instead of tmp directory
2e6bd28 
 - This helps improve the time it takes to run the EndToEndTests since the generator does not need to be rebuild for each test.
@xtremekforever
Cleanup installed swift-sdk after running EndToEnd test
2b7d3e3 
 - The swift-sdk is kept if an end-to-end test fails to allow further debugging.
@xtremekforever
Support testing Swift SDKs for Linux from EndToEnd tests
cc05d6f 
 - If SWIFT_SDK_GENERATOR_TEST_LINUX_SWIFT_SDKS is set, the swift-sdks are generated for Linux hosts. Each one is then tested against a matrix of Swift docker containers to verify each distribution works with it.
 - If SWIFT_SDK_GENERATOR_TEST_LINUX_SWIFT_SDKS is NOT set, but the host is Linux, then a Linux swift-sdk is generated but the suite of docker containers is not used.
@xtremekforever
Fix host arch for downloading Linux toolchain
e53d879 
 - If the host is aarch64, we download the aarch64 Swift toolchain.
@xtremekforever xtremekforever marked this pull request as ready for review January 9, 2025 14:08
@xtremekforever
Copy link
Contributor Author

@euanh @kateinoigakukun @MaxDesiatov I successfully ran the EdgeToEdgeTests on both an M1 Mac and a Linux x86_64 host with SWIFT_SDK_GENERATOR_TEST_LINUX_SWIFT_SDKS defined. Now to see what else is needed here...

MaxDesiatov
MaxDesiatov reviewed Jan 9, 2025
View reviewed changes
@xtremekforever
Restore Package.resolved to locked deps for CI
2ab3587 
 - If running on a Ubuntu 24.04 host, simply do swift package update locally and do not check in the updated Package.resolved.
MaxDesiatov
MaxDesiatov reviewed Jan 9, 2025
View reviewed changes
MaxDesiatov
MaxDesiatov reviewed Jan 9, 2025
View reviewed changes
MaxDesiatov
MaxDesiatov reviewed Jan 9, 2025
View reviewed changes
Copy link
Contributor

@MaxDesiatov MaxDesiatov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly minor nits, but my main concern is withTemporaryDirectory no longer used in the buildTestcases function, which would imply temporary data is not cleaned up and those tests are no longer idempotent.

euanh
euanh reviewed Jan 9, 2025
View reviewed changes
Tests/SwiftSDKGeneratorTests/EndToEndTests.swift
// that contains each Swift-supported Linux distribution. This way we can validate that each
// distribution is capable of building using the Linux Swift SDK.
if testLinuxSwiftSDKs {
let swiftContainerVersions = ["focal", "jammy", "noble", "fedora39", "rhel-ubi9", "amazonlinux2", "bookworm"]
Copy link
Contributor

@euanh euanh Jan 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fedora 39 is already EOL but it looks like it's the most recent Fedora-based Swift container image available. 😞

swiftlang/swift-docker#434

Copy link
Contributor Author

@xtremekforever xtremekforever Jan 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we could always remove it from the list. But, it has to use what's currently available...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think we should keep it for now so we have some Fedora coverage. ubi9 is similar but there may be differences which affect us.

@xtremekforever
Copy link
Contributor Author

xtremekforever commented Jan 11, 2025 •
edited
Loading

I ran the Swift59_EndToEndTests on an aarch64 Linux host (WSL2 running on a Snapdragon X Elite) and they worked flawlessly:

Test Case 'Swift59_UbuntuEndToEndTests.testX86_64FromContainer' passed (326.449 seconds)
Test Suite 'Swift59_UbuntuEndToEndTests' passed at 2025-01-10 23:44:27.233
         Executed 4 tests, with 0 failures (0 unexpected) in 1268.854 (1268.854) seconds
Test Suite 'Selected tests' passed at 2025-01-10 23:44:27.238
         Executed 6 tests, with 0 failures (0 unexpected) in 2223.01 (2223.01) seconds
$ uname -a
Linux Winter 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:19:58 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Let me know if anything else is needed here!

@xtremekforever xtremekforever force-pushed the #115-linux-host-toolchain-from-host-triple branch from 64658ea to 4a2f69d Compare January 16, 2025 16:46
@xtremekforever
Copy link
Contributor Author

I removed logging changes and swift-log usage, but retained this portion in the EndToEndTests:

  // Cleanup
  logger.info("Removing SDK to cleanup...")
  try await Shell.run("swift experimental-sdk remove \(bundleName)")

Since it should not require any changes for structured logging and is consistent with other logging calls here.

@MaxDesiatov
Copy link
Contributor

@swift-ci test

@xtremekforever
Fix LinuxRecipeTests, add tests cases for Linux vs macOS
f38e1b4 
 - On Linux we do not download the host LLVM since it is already included in the toolchain. However, on macOS we may or may not need to download it.
@MaxDesiatov
Copy link
Contributor

@swift-ci test

MaxDesiatov
MaxDesiatov approved these changes Jan 16, 2025
View reviewed changes
Copy link
Contributor

@MaxDesiatov MaxDesiatov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@MaxDesiatov MaxDesiatov merged commit 84d62bf into swiftlang:main Jan 16, 2025
3 checks passed
@xtremekforever xtremekforever deleted the #115-linux-host-toolchain-from-host-triple branch January 16, 2025 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MaxDesiatov MaxDesiatov MaxDesiatov approved these changes

@euanh euanh euanh approved these changes

@kateinoigakukun kateinoigakukun Awaiting requested review from kateinoigakukun

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Running Swift SDK generator on Ubuntu 22.04 host still produces Mach-O binaries of the Swift compiler
3 participants
@xtremekforever @MaxDesiatov @euanh