Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running Swift SDK generator on Ubuntu 22.04 host still produces Mach-O binaries of the Swift compiler #114

Closed
fwcd opened this issue Jul 2, 2024 · 2 comments · Fixed by #167

Comments

@fwcd
Copy link
Member

fwcd commented Jul 2, 2024

With #113, the SDK generator seems to successfully run to completion on Ubuntu 22.04:

Build logs
# swift run swift-sdk-generator make-linux-sdk
Building for debugging...
/swift-sdk-generator/Sources/SwiftSDKGenerator/Artifacts/DownloadableArtifacts.swift:33:9: warning: stored property 'remoteURL' of 'Sendable'-conforming struct 'Item' has non-sendable type 'URL'
    let remoteURL: URL
        ^
Foundation.URL:1:15: note: struct 'URL' does not conform to the 'Sendable' protocol
public struct URL : ReferenceConvertible, Equatable {
              ^
/swift-sdk-generator/Sources/SwiftSDKGenerator/Artifacts/DownloadableArtifacts.swift:13:1: remark: add '@preconcurrency' to suppress 'Sendable'-related warnings from module 'Foundation'
import struct Foundation.URL
^
@preconcurrency 
/swift-sdk-generator/Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Download.swift:26:5: warning: let 'byteCountFormatter' is not concurrency-safe because it is not either conforming to 'Sendable' or isolated to a global actor; this is an error in Swift 6
let byteCountFormatter = ByteCountFormatter()
    ^
/swift-sdk-generator/Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Metadata.swift:17:13: warning: let 'encoder' is not concurrency-safe because it is not either conforming to 'Sendable' or isolated to a global actor; this is an error in Swift 6
private let encoder: JSONEncoder = {
            ^
/swift-sdk-generator/Sources/SwiftSDKGenerator/Queries/DownloadFileQuery.swift:25:7: warning: stored property 'remoteURL' of 'Sendable'-conforming struct 'DownloadFileQuery' has non-sendable type 'URL'
  let remoteURL: URL
      ^
Foundation.URL:1:15: note: struct 'URL' does not conform to the 'Sendable' protocol
public struct URL : ReferenceConvertible, Equatable {
              ^
/swift-sdk-generator/Sources/SwiftSDKGenerator/Queries/DownloadFileQuery.swift:13:1: remark: add '@preconcurrency' to suppress 'Sendable'-related warnings from module 'Foundation'
import struct Foundation.URL
^
@preconcurrency 
/swift-sdk-generator/Sources/SwiftSDKGenerator/SystemUtils/FileOperationError.swift:17:8: warning: associated value 'downloadFailed' of 'Sendable'-conforming enum 'FileOperationError' has non-sendable type 'URL'
  case downloadFailed(URL, String)
       ^
Foundation.URL:1:15: note: struct 'URL' does not conform to the 'Sendable' protocol
public struct URL : ReferenceConvertible, Equatable {
              ^
/swift-sdk-generator/Sources/SwiftSDKGenerator/SystemUtils/FileOperationError.swift:13:1: remark: add '@preconcurrency' to suppress 'Sendable'-related warnings from module 'Foundation'
import struct Foundation.URL
^
@preconcurrency 
/swift-sdk-generator/Sources/SwiftSDKGenerator/SystemUtils/GeneratorError.swift:26:8: warning: associated value 'fileDownloadFailed' of 'Sendable'-conforming enum 'GeneratorError' has non-sendable type 'URL'
  case fileDownloadFailed(URL, String)
       ^
Foundation.URL:1:15: note: struct 'URL' does not conform to the 'Sendable' protocol
public struct URL : ReferenceConvertible, Equatable {
              ^
/swift-sdk-generator/Sources/SwiftSDKGenerator/SystemUtils/GeneratorError.swift:13:1: remark: add '@preconcurrency' to suppress 'Sendable'-related warnings from module 'Foundation'
import struct Foundation.URL
^
@preconcurrency 
[7/7] Linking swift-sdk-generator
Build complete! (4.28s)

Looking up configuration values...

Downloading required toolchain packages...
Using downloaded artifacts in these locations:
/swift-sdk-generator/Artifacts/target_swift_5.9.2-RELEASE_x86_64-unknown-linux-gnu.tar.gz
/swift-sdk-generator/Artifacts/host_llvm_17.0.5_x86_64-unknown-linux.tar.xz
/swift-sdk-generator/Artifacts/host_swift_5.9.2-RELEASE_x86_64-unknown-linux.pkg

Parsing Ubuntu packages list...
Downloading 11 Ubuntu packages...
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6-dev_2.35-0ubuntu3.8_amd64.deb – 2.1 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/gcc-12/libgcc-s1_12.3.0-1ubuntu1~22.04_amd64.deb – 54 KB
http://gb.archive.ubuntu.com/ubuntu/pool/main/z/zlib/zlib1g_1.2.11.dfsg-2ubuntu9.2_amd64.deb – 58 KB
http://gb.archive.ubuntu.com/ubuntu/pool/main/z/zlib/zlib1g-dev_1.2.11.dfsg-2ubuntu9.2_amd64.deb – 164 KB
http://gb.archive.ubuntu.com/ubuntu/pool/main/i/icu/libicu70_70.1-2_amd64.deb – 10.6 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6_2.35-0ubuntu3.8_amd64.deb – 3.2 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/gcc-12/libstdc++6_12.3.0-1ubuntu1~22.04_amd64.deb – 699 KB
http://gb.archive.ubuntu.com/ubuntu/pool/universe/g/gcc-12/libstdc++-12-dev_12.3.0-1ubuntu1~22.04_amd64.deb – 2.2 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/l/linux/linux-libc-dev_5.15.0-113.123_amd64.deb – 1.3 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/gcc-12/libgcc-12-dev_12.3.0-1ubuntu1~22.04_amd64.deb – 2.6 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/i/icu/libicu-dev_70.1-2_amd64.deb – 11.6 MB
control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary


Unpacking and copying Swift binaries for the host triple...
8460552 blocks

Unpacking Swift distribution for the target triple...

Copying Swift core libraries for the target triple into Swift SDK bundle...

Unpacking and copying `lld` linker...

Fixing up absolute symlinks...

Fixing absolute paths in `glibc.modulemap`...

Generating toolset JSON file...

Generating destination JSON file...

Generating .artifactbundle manifest file...

All done! Install the newly generated SDK with this command:
swift experimental-sdk install /swift-sdk-generator/Bundles/5.9.2-RELEASE_ubuntu_jammy_x86_64.artifactbundle

After that, use the newly installed SDK when building with this command:
swift build --experimental-swift-sdk 5.9.2-RELEASE_ubuntu_jammy_x86_64

Time taken for this generator run: 2 minutes 51 seconds.

Trying to build a test package, however, still seems to fail:

# swift build --experimental-swift-sdk 5.9.2-RELEASE_ubuntu_jammy_x86_64
error: No Swift SDK found matching query `5.9.2-RELEASE_ubuntu_jammy_x86_64` and host triple `x86_64-unknown-linux-gnu`. Use `swift experimental-sdk list` command to see available Swift SDKs.
# swift experimental-sdk list
5.9.2-RELEASE_ubuntu_jammy_x86_64

Regenerating the SDK with --host x86_64-unknown-linux-gnu seemingly works too:

Build logs
# swift run swift-sdk-generator make-linux-sdk --host x86_64-unknown-linux-gnu
Building for debugging...
[1/1] Write swift-version-24593BA9C3E375BF.txt
Build complete! (0.38s)

Looking up configuration values...

Downloading required toolchain packages...
Downloading remote artifact not available in local cache: https://github.com/llvm/llvm-project/releases/download/llvmorg-17.0.5/clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz
Downloading remote artifact not available in local cache: https://download.swift.org/swift-5.9.2-release/xcode/swift-5.9.2-RELEASE/swift-5.9.2-RELEASE-osx.pkg
swift-5.9.2-RELEASE-osx.pkg 13 KB/1.08 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 16 KB/1.01 GB
swift-5.9.2-RELEASE-osx.pkg 187.8 MB/1.08 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 46.4 MB/1.01 GB
swift-5.9.2-RELEASE-osx.pkg 423.3 MB/1.08 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 100.8 MB/1.01 GB
swift-5.9.2-RELEASE-osx.pkg 645.6 MB/1.08 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 155.3 MB/1.01 GB
swift-5.9.2-RELEASE-osx.pkg 867.4 MB/1.08 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 205.9 MB/1.01 GB
swift-5.9.2-RELEASE-osx.pkg 1.08 GB/1.08 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 259.7 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 310.6 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 357.1 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 411.7 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 461.7 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 513.4 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 566.5 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 611.8 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 667.5 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 722.5 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 776.5 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 836.2 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 896.9 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 951.6 MB/1.01 GB
clang+llvm-17.0.5-x86_64-linux-gnu-ubuntu-22.04.tar.xz 1.01 GB/1.01 GB
Using downloaded artifacts in these locations:
/swift-sdk-generator/Artifacts/target_swift_5.9.2-RELEASE_x86_64-unknown-linux-gnu.tar.gz
/swift-sdk-generator/Artifacts/host_swift_5.9.2-RELEASE_x86_64-unknown-linux-gnu.pkg
/swift-sdk-generator/Artifacts/host_llvm_17.0.5_x86_64-unknown-linux-gnu.tar.xz

Parsing Ubuntu packages list...
Downloading 11 Ubuntu packages...
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6_2.35-0ubuntu3.8_amd64.deb – 3.2 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/z/zlib/zlib1g-dev_1.2.11.dfsg-2ubuntu9.2_amd64.deb – 164 KB
http://gb.archive.ubuntu.com/ubuntu/pool/main/z/zlib/zlib1g_1.2.11.dfsg-2ubuntu9.2_amd64.deb – 58 KB
http://gb.archive.ubuntu.com/ubuntu/pool/main/l/linux/linux-libc-dev_5.15.0-113.123_amd64.deb – 1.3 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/gcc-12/libstdc++6_12.3.0-1ubuntu1~22.04_amd64.deb – 699 KB
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/glibc/libc6-dev_2.35-0ubuntu3.8_amd64.deb – 2.1 MB
http://gb.archive.ubuntu.com/ubuntu/pool/universe/g/gcc-12/libstdc++-12-dev_12.3.0-1ubuntu1~22.04_amd64.deb – 2.2 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/gcc-12/libgcc-12-dev_12.3.0-1ubuntu1~22.04_amd64.deb – 2.6 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/g/gcc-12/libgcc-s1_12.3.0-1ubuntu1~22.04_amd64.deb – 54 KB
http://gb.archive.ubuntu.com/ubuntu/pool/main/i/icu/libicu70_70.1-2_amd64.deb – 10.6 MB
http://gb.archive.ubuntu.com/ubuntu/pool/main/i/icu/libicu-dev_70.1-2_amd64.deb – 11.6 MB
control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary

control.tar.zst
data.tar.zst
debian-binary


Unpacking and copying Swift binaries for the host triple...
8460552 blocks

Unpacking Swift distribution for the target triple...

Copying Swift core libraries for the target triple into Swift SDK bundle...

Unpacking and copying `lld` linker...

Fixing up absolute symlinks...

Fixing absolute paths in `glibc.modulemap`...

Generating toolset JSON file...

Generating destination JSON file...

Generating .artifactbundle manifest file...

All done! Install the newly generated SDK with this command:
swift experimental-sdk install /swift-sdk-generator/Bundles/5.9.2-RELEASE_ubuntu_jammy_x86_64.artifactbundle

After that, use the newly installed SDK when building with this command:
swift build --experimental-swift-sdk 5.9.2-RELEASE_ubuntu_jammy_x86_64

Time taken for this generator run: 3 minutes 12 seconds.

This time, however, trying to build a package results in an exec format error:

# swift build --experimental-swift-sdk 5.9.2-RELEASE_ubuntu_jammy_x86_64
Building for debugging...
error: failed to write auxiliary file '/testpkg/.build/x86_64-unknown-linux-gnu/debug/swift-version--52D40532F92296A3.txt': posix_spawn error: Exec format error (8), `["/root/.swiftpm/swift-sdks/5.9.2-RELEASE_ubuntu_jammy_x86_64.artifactbundle/5.9.2-RELEASE_ubuntu_jammy_x86_64/x86_64-unknown-linux-gnu/swift.xctoolchain/usr/bin/swiftc", "-version"]`

It looks like the generated SDK contains Mach-O binaries (for macOS) even though we built for a Ubuntu host:

# file /root/.swiftpm/swift-sdks/5.9.2-RELEASE_ubuntu_jammy_x86_64.artifactbundle/5.9.2-RELEASE_ubuntu_jammy_x86_64/x86_64-unknown-linux-gnu/swift.xctoolchain/usr/bin/swift-frontend 
/root/.swiftpm/swift-sdks/5.9.2-RELEASE_ubuntu_jammy_x86_64.artifactbundle/5.9.2-RELEASE_ubuntu_jammy_x86_64/x86_64-unknown-linux-gnu/swift.xctoolchain/usr/bin/swift-frontend: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|WEAK_DEFINES|BINDS_TO_WEAK|PIE|HAS_TLV_DESCRIPTORS>] [arm64]
@xtremekforever
Copy link
Contributor

xtremekforever commented Jul 9, 2024

It looks like this is related to the fact that it's downloading the OSX package for swift 5.9.2: https://download.swift.org/swift-5.9.2-release/xcode/swift-5.9.2-RELEASE/swift-5.9.2-RELEASE-osx.pkg

I tried passing the --host-swift-package-path to point to my local Swift installation, which appears to NOT download that OSX package, but then it seems a conflict occurs that causes a crash:

Fixing absolute paths in `glibc.modulemap`...
Error: The operation could not be completed. A file with the same name already exists.

So, I think some additional patching needs to happen to get this to work correctly on Ubuntu 22.04.

@MaxDesiatov MaxDesiatov changed the title Running SDK generator on Ubuntu 22.04 host still produces Mach-O binaries of the Swift compiler Running Swift SDK generator on Ubuntu 22.04 host still produces Mach-O binaries of the Swift compiler Dec 19, 2024
@xtremekforever
Copy link
Contributor

@MaxDesiatov @euanh I have more changes for the swift-sdk-generator which will download a Linux host toolchain the host triple is set to something like --host x86_64-unknown-linux-gnu. They are pretty simple and fix this ticket, and would bring some official host support for Ubuntu out of the box....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants