Ubuntu 24.04: Build using Docker/QEMU fails for Build-Target Linux/ARM64 #11561

jo-tools · 2025-02-08T19:08:15Z

Description

A Workflow builds the LibreSSL Libs for Linux/ARM64.

The build of the Library is run inside a Docker Container with Image arm64v8/gcc:8.1

          docker run \
            --rm \
            -v "$PWD":/usr/src/libressl \
            -w /usr/src/libressl \
            arm64v8/gcc:8.1 \
            /bin/bash -c "./configure --enable-libtls-only --with-pic && make -j4"

This works as expected:

on Github Action Runners when runs-on: ubuntu-22.04

when run on an own Ubuntu 24.04 machine with Docker and QEMU setup like this

       sudo snap refresh docker
       sudo apt-get update
       sudo apt-get install qemu-user-static
       sudo docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

This fails:

on Github Action Runners when runs-on: ubuntu-24.04
on Github Action Runners when runs-on: ubuntu-latest

Platforms affected

Azure DevOps
GitHub Actions - Standard Runners
GitHub Actions - Larger Runners

Runner images affected

Image version and build link

Current runner version: '2.322.0'
Operating System
  Ubuntu
  24.04.1
  LTS
Runner Image
  Image: ubuntu-24.04
  Version: 20250202.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250202.1/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250202.1
Runner Image Provisioner
  2.0.422.1

Is it regression?

Yes, Workflow runs successfully on GitHub Actions Runner Image: ubuntu-22.04 Version: 20250202.1.0

Expected behavior

Building a Library inside a Docker Container with Image arm64v8/gcc:8.1 works on
-runs-on: ubuntu-24.04
-runs-on: ubuntu-latest

Such as it always worked on
-runs-on: ubuntu-22.04

Actual behavior

Run the below Workflow and it will

succeed in runs-on: ubuntu-22.04
fail in the step called "Build" in runs-on: ubuntu-24.04 with:

...
checking if .gnu.warning accepts long strings... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating include/Makefile
config.status: creating include/openssl/Makefile
config.status: creating crypto/Makefile
config.status: creating ssl/Makefile
config.status: creating tls/Makefile
config.status: creating tests/Makefile
config.status: creating apps/Makefile
config.status: creating apps/ocspcheck/Makefile
config.status: creating apps/openssl/Makefile
config.status: creating apps/nc/Makefile
config.status: creating man/Makefile
config.status: creating libcrypto.pc
config.status: creating libssl.pc
config.status: creating libtls.pc
config.status: creating openssl.pc
config.status: executing depfiles commands
config.status: error: in `/usr/src/libressl':
config.status: error: Something went wrong bootstrapping makefile fragments
    for automatic dependency tracking.  If GNU make was not used, consider
    re-running the configure script with MAKE="gmake" (or whatever is
    necessary).  You can also try re-running configure with the
    '--disable-dependency-tracking' option to at least be able to build
    the package (albeit without support for automatic dependency tracking).
See `config.log' for more details
Error: Process completed with exit code 1.

Repro steps

Run this workflow:

name: LibreSSLTest

on:
  workflow_dispatch:
    inputs:
      libressl_version:
        description: LibreSSL Version
        required: true
        default: '4.0.0'
        type: string

jobs:
  build-libressl-linux-arm64-on-ubuntu-24-04:
    name: Build LibreSSL Linux ARM64 on Ubuntu 24.04
    runs-on: ubuntu-24.04
    steps:
      - name: LibreSSL Version
        run: echo Build LibreSSL ${{ inputs.libressl_version }}
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
#     - name: Set up QEMU
#       run: |
#         sudo snap refresh docker
#         sudo apt-get update
#         sudo apt-get install qemu-user-static
#         sudo docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
      - name: Build
        run: |
          curl -O "https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-${{ inputs.libressl_version }}.tar.gz"
          tar xvfz ./libressl-${{ inputs.libressl_version }}.tar.gz
          cd ./libressl-${{ inputs.libressl_version }}
          docker run \
            --rm \
            -v "$PWD":/usr/src/libressl \
            -w /usr/src/libressl \
            arm64v8/gcc:8.1 \
            /bin/bash -c "./configure --enable-libtls-only --with-pic && make -j4"
          cd ..
          mkdir -p ./libs/linux-arm64
          cp ./libressl-${{ inputs.libressl_version }}/tls/.libs/libtls.a ./libs/linux-arm64/
      - name: Upload Artifacts
        uses: actions/upload-artifact@v4
        with:
          name: LibreSSL-linux-arm64-on-ubuntu-24-04
          retention-days: 2
          path: |
            ./libs/linux-arm64/libtls.a
          if-no-files-found: error
  build-libressl-linux-arm64-on-ubuntu-22-04:
    name: Build LibreSSL Linux ARM64 on Ubuntu 22.04
    runs-on: ubuntu-22.04
    steps:
      - name: LibreSSL Version
        run: echo Build LibreSSL ${{ inputs.libressl_version }}
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
      - name: Build
        run: |
          curl -O "https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-${{ inputs.libressl_version }}.tar.gz"
          tar xvfz ./libressl-${{ inputs.libressl_version }}.tar.gz
          cd ./libressl-${{ inputs.libressl_version }}
          docker run \
            --rm \
            -v "$PWD":/usr/src/libressl \
            -w /usr/src/libressl \
            arm64v8/gcc:8.1 \
            /bin/bash -c "./configure --enable-libtls-only --with-pic && make -j4"
          cd ..
          mkdir -p ./libs/linux-arm64
          cp ./libressl-${{ inputs.libressl_version }}/tls/.libs/libtls.a ./libs/linux-arm64/
      - name: Upload Artifacts
        uses: actions/upload-artifact@v4
        with:
          name: LibreSSL-linux-arm64-on-ubuntu-22-04
          retention-days: 2
          path: |
            ./libs/linux-arm64/libtls.a
          if-no-files-found: error

Note: The commented lines have been to try not using docker/setup-qemu-action@v3, but instead installing QEMU manually.
That works on an own Ubuntu 24.04 machine to successfully build LibreSSL for Linux/ARM64, but it doesn't work on GitHub Runner 24.04 - same error message.

The text was updated successfully, but these errors were encountered:

jo-tools · 2025-02-08T19:09:26Z

This issue might be related to #11557

imply-elliott · 2025-02-10T04:01:56Z

Agreed, I suspect they're related. We're having the same issue with buildx building for arm64 on amd64.

vidyasagarnimmagaddi · 2025-02-10T07:17:06Z

Hi @jo-tools
For ARM related issues, We requesting you please raise them in partner-runner-images, We are closing this issue for now, Thanks

almusil · 2025-02-10T07:50:35Z

Hi @vidyasagarnimmagaddi,

I fail to understand why this falls into the partner repo category. The failing jobs are running on x86_64 Ubuntu 24.04, just inside container. Which means there is qemu involved in the process, but the primary image is the one provided by GitHub and not partners.

Operating System
  Ubuntu
  24.04.1
  LTS
Runner Image
  Image: ubuntu-24.04
  Version: 20250202.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250202.1/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250202.1

They're broken in GHA: actions/runner-images#11561 actions/partner-runner-images#46 Signed-off-by: Dumitru Ceara <dceara@redhat.com>

imply-elliott · 2025-02-10T17:57:49Z

Hi @vidyasagarnimmagaddi,

I fail to understand why this falls into the partner repo category. The failing jobs are running on x86_64 Ubuntu 24.04, just inside container. Which means there is qemu involved in the process, but the primary image is the one provided by GitHub and not partners.
Operating System
  Ubuntu
  24.04.1
  LTS
Runner Image
  Image: ubuntu-24.04
  Version: 20250202.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250202.1/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250202.1

Seconding this. The issues are cross-platform builds running on an amd64 host but with arm64 builds inside the emulated container.

jo-tools · 2025-02-10T19:27:57Z

Hi @jo-tools For ARM related issues, We requesting you please raise them in partner-runner-images, We are closing this issue for now, Thanks

@vidyasagarnimmagaddi I respectfully disagree.

It's about jobs that are running on x86_64 Ubuntu 24.04 (inside a Docker Container on that host).

Others are reporting issues with Docker/buildx, and in this issue I've provided just another reproducible example.

A job with these steps/commands works just fine on an own Ubuntu 24 VM.
I even ruled out the docker/setup-qemu-action@v3 to be the cause of the issue - as installing qemu-user-static and then doing the ARM64 build job inside the docker container works on standard Ubuntu 24 installations without any issues.

So to me it seems likely that the setup of GitHub's Ubuntu 24.04 x86_64 runners are somehow "screwed" when it comes to Docker/QEMU.
This will likely affect many existing workflows that will break once run on runs-on: ubuntu-latest (which now defaults to 24.04).

Workarounds:

change workflows to use runs-on: ubuntu-22.04
run on a self-hosted x86_64 Ubuntu 24.04
for public repositories only:
- change workflows to use runs-on: ubuntu-24.04-arm
- remove steps that set up QEMU (not necessary on an ARM machine)

Please reconsider to reopen this issue - I think it's well worth investigating.

jo-tools · 2025-02-10T19:41:26Z

@imply-elliott Agreed, I suspect they're related. We're having the same issue with buildx building for arm64 on amd64.

@vidyasagarnimmagaddi That quoted feedback supports the request of reopening this issue.

You should really investigate why Docker/buildx (or Docker/QEMU) are having issues on GitHub Runners (ubuntu-latest, or ubuntu-24.04). This will likely affect many workflows.

@hemanthmanga fyi, as you've closed #11557 (BuildX + Rust issues) for the same reason.

jo-tools added bug report needs triage labels Feb 8, 2025

vidyasagarnimmagaddi closed this as completed Feb 10, 2025

dceara added a commit to dceara/ovn that referenced this issue Feb 10, 2025

HACK: Temporarily skip ARM64 container jobs.

72a77f5

They're broken in GHA: actions/runner-images#11561 actions/partner-runner-images#46 Signed-off-by: Dumitru Ceara <dceara@redhat.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ubuntu 24.04: Build using Docker/QEMU fails for Build-Target Linux/ARM64 #11561

Ubuntu 24.04: Build using Docker/QEMU fails for Build-Target Linux/ARM64 #11561

jo-tools commented Feb 8, 2025 •

edited

Loading

jo-tools commented Feb 8, 2025

imply-elliott commented Feb 10, 2025

vidyasagarnimmagaddi commented Feb 10, 2025

almusil commented Feb 10, 2025

imply-elliott commented Feb 10, 2025

jo-tools commented Feb 10, 2025 •

edited

Loading

jo-tools commented Feb 10, 2025

Ubuntu 24.04: Build using Docker/QEMU fails for Build-Target Linux/ARM64 #11561

Ubuntu 24.04: Build using Docker/QEMU fails for Build-Target Linux/ARM64 #11561

Comments

jo-tools commented Feb 8, 2025 • edited Loading

Description

Platforms affected

Runner images affected

Image version and build link

Is it regression?

Expected behavior

Actual behavior

Repro steps

jo-tools commented Feb 8, 2025

imply-elliott commented Feb 10, 2025

vidyasagarnimmagaddi commented Feb 10, 2025

almusil commented Feb 10, 2025

imply-elliott commented Feb 10, 2025

jo-tools commented Feb 10, 2025 • edited Loading

jo-tools commented Feb 10, 2025

jo-tools commented Feb 8, 2025 •

edited

Loading

jo-tools commented Feb 10, 2025 •

edited

Loading