Inability to start colima on GitHub macOS-14 runners: "HV_UNSUPPORTED"

[beelux]

Description

GitHub just released their macos-14 runners in Beta, and they don't seem to be able to start colima.
The symptom seems similar to #791 or even #786 (comment): starting colima fails on an HV_UNSUPPORTED issue.

I've tried appliying the hotfix, but the signature doesn't seem to be the issue, as a non-hotfixed workflow also does not complain:

time="2024-01-30T18:27:00Z" level=info msg="QEMU binary \"/opt/homebrew/bin/qemu-system-aarch64\" seems properly signed with the \"com.apple.security.hypervisor\" entitlement"

While the .colima/_lima/colima/ha.stderr.log log shows an issue:

{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/runner/.colima/_lima/colima/serial*.log\")","time":"2024-01-30T18:53:22Z"}
(...)
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2024-01-30T18:53:22Z"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2024-01-30T18:53:22Z"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/runner/.colima/_lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=runner -o ControlMaster=auto -o ControlPath=\"/Users/runner/.colima/_lima/colima/ssh.sock\" -o ControlPersist=yes -p 49217 127.0.0.1 -- /bin/bash]","time":"2024-01-30T18:53:22Z"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-aarch64: Error: HV_UNSUPPORTED","time":"2024-01-30T18:53:22Z"}
{"level":"info","msg":"Driver stopped due to error: \"signal: abort trap\"","time":"2024-01-30T18:53:22Z"}

Note that the quoted logs here are shortened.

The full logs are accessible here: https://github.com/beelux/toltec-toolchain/actions/runs/7716039166/job/21031938411

Is this even a colima issue, or should I open an issue on the actions/runner-images repo?

Version

colima version 0.6.7
git commit: ba1be00e9aec47f2c1ffdacfb7e428e465f0b58a
limactl version 0.19.1
qemu-img version 8.2.0
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers

Operating System

• macOS Intel <= 13 (Ventura)
• macOS Intel >= 14 (Sonoma)
• Apple Silicon <= 13 (Ventura)
• Apple Silicon >= 14 (Sonoma)
• Linux

Output of colima status

time="2024-01-30T19:15:46Z" level=fatal msg="colima is not running"

Reproduction Steps

Running a workflow on a GitHub macos-14 aarch64 runner.

1. Use macos-14
2. Install docker and colima through brew
3. Start colima

Snippet from a workflow file:

  armbuild:
    name: Build latest arm64
    runs-on: macos-14
    steps:
      - name: Checkout the Git repository
        uses: actions/checkout@v2                                                                                                                                                                                   
      - name: Setup Docker
        run: |
          brew install docker colima docker-buildx
      - name: Start Docker container runtimes
        run: |
          colima start

Expected behaviour

Colima should start without crashing

Additional context

No response

[jandubois]

I suspect the runner is a VM on a machine with an M1 chip, which doesn't support nested virtualization (you can't run a VM inside the VM). It requires M2 or later architecture...

The same limitation was true on the Windows and Linux runners until about 2 weeks ago when Github moved the runners to new hardware that does support nested virtualization.

The macOS Intel runners have supported nested virtualization as long as I can remember.

[beelux]

@jandubois Thanks a lot for clearing this up so quickly! The runners are indeed running on M1 chips. Somehow the nested virtualization limitation didn't cross my mind even though I actually use an M1 chip machine.

I'm guessing that there's thus no "good" way to run colima on those M1 runners, outside of emulation.

I just realized that this actually got brought up on actions/runner-images#9254 (comment), which also mentioned a line in the macOS larger runner docs:

Due to a limitation of Apple's Virtualization Framework, which our hypervisor uses, nested-virtualization is not supported by arm64 runners.

Thanks again, I was stuck and wasn't exactly sure where to ask.

[mikekazakov]

Apologies for necroposting, but some might find this useful.
It's possible to run colima on macOS-14 GH runners (i.e. Apple M1) with a patched version of lima that doesn't try to use Hypervisor Framework. It's much slower, of course, since QEMU does essentially full emulation, but for lightweight jobs might be acceptable.
Here's an example of such run on GHA: https://github.com/mikekazakov/nimble-commander/actions/runs/9897976549/job/27345739982.

That's the startup code (https://github.com/mikekazakov/nimble-commander/blob/df0a21832d7fa9f0a1e98525ce0b1c2af011ff7d/.github/workflows/build.yml#L46):

sudo curl -L -o /opt/homebrew/Cellar/lima/0.22.0/bin/limactl https://github.com/mikekazakov/lima-nohvf/raw/master/limactl && sudo chmod +x /opt/homebrew/Cellar/lima/0.22.0/bin/limactl
colima start --network-address --arch arm64 --vm-type=qemu

[shinyano]

Apologies for necroposting, but some might find this useful. It's possible to run colima on macOS-14 GH runners (i.e. Apple M1) with a patched version of lima that doesn't try to use Hypervisor Framework. It's much slower, of course, since QEMU does essentially full emulation, but for lightweight jobs might be acceptable. Here's an example of such run on GHA: https://github.com/mikekazakov/nimble-commander/actions/runs/9897976549/job/27345739982.

That's the startup code (https://github.com/mikekazakov/nimble-commander/blob/df0a21832d7fa9f0a1e98525ce0b1c2af011ff7d/.github/workflows/build.yml#L46):

sudo curl -L -o /opt/homebrew/Cellar/lima/0.22.0/bin/limactl https://github.com/mikekazakov/lima-nohvf/raw/master/limactl && sudo chmod +x /opt/homebrew/Cellar/lima/0.22.0/bin/limactl
colima start --network-address --arch arm64 --vm-type=qemu

It's a practical workaround. And if anyone wants to use this code as I did, remember to replace 0.22.0 with real path. Or you can use a more flexible way:

LIMACTL_PATH=$(brew --prefix)/bin/limactl
sudo curl -L -o $LIMACTL_PATH https://github.com/mikekazakov/lima-nohvf/raw/master/limactl && sudo chmod +x $LIMACTL_PATH
colima start --network-address --arch arm64 --vm-type=qemu