[macos] instance unknown state / cannot be started

[DominicBurkart]

Describe the bug

All instances are in an unknown state after unexpected restart due to battery issue, and new instances cannot be created.

I assume that I need to delete some qemu images / configurations, but am not sure how to proceed. If possible, it would be awesome if multipass could recover from this issue!

To Reproduce
How, and what happened?

• Multipass works as expected (e.g. running multipass launch --cpus='5' --memory='8G' --disk='64g' --name ad && multipass shell ad).
• Then, the computer runs out of battery while VM is running.
• When device is booted, the instance is in state Unknown and cannot run. The instance can be stopped (or directly deleted and purged), but cannot be started or a shell opened. New instances can be created but not started or a shell opened.
• Issue is not resolved by uninstalling/reinstalling multipass using the installer or using brew.

Expected behavior

multipass can identify and recover the issue (underlying driver misconfiguration?), allowing new instances to be created.

Logs
Please provide logs from the daemon, see accessing logs on where to find them on your platform.

From running multipass launch --cpus='5' --memory='8G' --disk='64g' --name ad && multipass shell ad:

[2023-09-06T10:11:04.799] [debug] [qemu-system-aarch64] [42171] started: qemu-system-aarch64 --version
[2023-09-06T10:11:05.074] [debug] [qemu-img] [42172] started: qemu-img info /var/root/Library/Caches/multipassd/qemu/vault/images/jammy-20230828/ubuntu-22.04-server-cloudimg-arm64.img
[2023-09-06T10:11:05.083] [debug] [qemu-img] [42173] started: qemu-img resize /var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/ubuntu-22.04-server-cloudimg-arm64.img 68719476736
[2023-09-06T10:11:05.090] [debug] [qemu-img] [42174] started: qemu-img snapshot -l /var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/ubuntu-22.04-server-cloudimg-arm64.img
[2023-09-06T10:11:05.095] [debug] [ad] process working dir ''
[2023-09-06T10:11:05.095] [info] [ad] process program 'qemu-system-aarch64'
[2023-09-06T10:11:05.095] [info] [ad] process arguments '-machine, virt,gic-version=3, -accel, hvf, -drive, file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on, -cpu, host, -nic, vmnet-shared,model=virtio-net-pci,mac=52:54:00:8c:64:6f, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 5, -m, 8192M, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/cloud-init-config.iso'
[2023-09-06T10:11:05.098] [debug] [qemu-system-aarch64] [42175] started: qemu-system-aarch64 -machine virt,gic-version=3 -nographic -dump-vmstate /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/multipassd.zAFszV
[2023-09-06T10:11:05.339] [info] [ad] process state changed to Starting
[2023-09-06T10:11:05.342] [info] [ad] process state changed to Running
[2023-09-06T10:11:05.342] [debug] [qemu-system-aarch64] [42178] started: qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:8c:64:6f -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 5 -m 8192M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/cloud-init-config.iso
[2023-09-06T10:11:05.342] [info] [ad] process started
[2023-09-06T10:11:05.343] [debug] [ad] Waiting for SSH to be up
[2023-09-06T10:11:05.734] [debug] [ad] QMP: {"QMP": {"version": {"qemu": {"micro": 0, "minor": 0, "major": 8}, "package": ""}, "capabilities": ["oob"]}}

[2023-09-06T10:11:05.795] [debug] [ad] QMP: {"return": {}}

[2023-09-06T10:11:19.068] [debug] [ad] QMP: {"timestamp": {"seconds": 1693987879, "microseconds": 68934}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/device[10]/virtio-backend"}}

Additional info

• OS: Ventura 13.4.1
• multipass version

multipass   1.12.2+mac
multipassd  1.12.2+mac

• multipass info --all

Name:           ad
State:          Unknown
IPv4:           --
Release:        --
Image hash:     41a8587b4994 (Ubuntu 22.04 LTS)
CPU(s):         --
Load:           --
Disk usage:     --
Memory usage:   --
Mounts:         --

• multipass get local.driver

qemu

Additional context

In my case, the instance does not need to be recovered; priority is just being able to create new instances.

Similar issues identified:

• Time out and unknown state #1268
• error: program: qemu-system-x86_64; error: Process crashed #1358
• Restart multipass will result in State: Unknown (and machine_type is empty) #1383

I don't think any of the solutions there are relevant to this case, where a full reinstall didn't solve the issue.

[DominicBurkart]

Running brew reinstall qemu then brew uninstall --zap --cask multipass && brew install multipass didn't resolve the issue.

[DominicBurkart]

Killing all processes associated with multipass (as recommended here: https://gist.github.com/zulhfreelancer/12817fa5d1579bb640ed00dff8327a4a) also doesn't resolve the issue.

[DominicBurkart]

resetting the network as described here also did not work: https://multipass.run/docs/troubleshoot-networking#heading--network-routing-problems

[miseyu]

I'm in the same situation.

OS: Ventura 13.5
multipass version
multipass   1.12.2+mac
multipassd  1.12.2+mac

[miseyu]

This one has been resolved.
What we did

• Update mac os to the latest version
• Reinstalled multipass with brew

$ sw_vers
ProductName:            macOS
ProductVersion:         13.5.2
BuildVersion:           22G91

[miseyu]

However, since we don't know the root cause, the phenomenon is likely to occur again, and we would like to continue to watch for it.

[kwagga]

Ran into the same issue and the reinstallation of qemu and multipass via brew did not resolve it for me.

❯ sw_vers
ProductName:		macOS
ProductVersion:		13.5.1
BuildVersion:		22G90

❯ multipass version
multipass   1.12.2+mac
multipassd  1.12.2+mac

[andrei-toterman]

Hey, @DominicBurkart! From the way you described it so far, it sounds like a case of #2387. Is the firewall enabled on your machine? If so, please try to disable it, reboot and try using Multipass again.
You can also try running the following commands to allow bootpd to run.

$ sudo su -
$ /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
$ /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
$ reboot now

But it might not be the same issue as that one, so please let me know if this doesn't work

[bitsf]

I met the same problem just from this month.
I tried to run socketfilterfw, but got error Firewall settings cannot be modified from command line on managed Mac computers.
Is there any other method to workaround?

[georgeliao]

Hi @bitsf I am afraid that we do not have other workarounds except the one that has been discussed. #2387 This post has more info, but the socketfilterfw is the distilled information from that post.

[bitsf]

Finally, I added /usr/libexec/bootpd into firewall from GUI, and reboot Mac, then it can work now.

[bitsf]

my multipass can't start again after several days.
And I find if I run sudo qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive "file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on" -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:f6:c6:64 -device virtio-scsi-pci,id=scsi0 -drive "file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda" -device scsi-hd,drive=hda,bus=scsi0.0 -smp 1 -m 1024M -cdrom "/var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/cloud-init-config.iso" -display default,show-cursor=on -qmp stdio, it seems failed to start the VM

But if I remove the gic-version=3, just use -machine virt or use -machine virt,gic-version=2, it can start the VM.
I'm not sure if this is an issue

[percygrunwald]

I was able to resolve this issue using the OnyX tool as recommended in another GH issue relating to VM start issues: #3003 (comment)

[vielmetti]

I ran into exactly this problem. Looking into Onyx

[jrz]

Same here after fresh install Sonoma on M1.
Pretty useless software like this. How do I completely remove everything including all the gunk?

[yzcyayaya]

I'm having the same problem, how can I solve it?

Firewall off statu

~$ multipass list
Name                    State             IPv4             Image
repository-host         Unknown           --               Ubuntu 22.04 LTS
kube-master             Unknown           --               Ubuntu 22.04 LTS
kube-node01             Unknown           --               Ubuntu 22.04 LTS
kube-node02             Unknown           --               Ubuntu 22.04 LTS

~$ sw_vers
ProductName:		macOS
ProductVersion:		13.4.1
BuildVersion:		22F82

~$ multipass version
multipass   1.12.2+mac
multipassd  1.12.2+mac

[yzcyayaya]

/cc @townsend2010 Is this situation included in the restoration plan?

[townsend2010]

Hi @yzcyayaya!

I'm having the same problem, how can I solve it?

Could you please provide the log?

Is this situation included in the restoration plan?

Sorry, I'm not really sure what you mean by the "restoration plan."

[yzcyayaya]

Is this situation included in the restoration plan?

Haha, it's okay, I solved my needs using kvm from another machine

[krmao]

➜  .output git:(pc) multipass exec docker docker
exec failed: Cannot retrieve credentials in unknown state
➜  .output git:(pc) multipass shell docker
shell failed: Cannot retrieve credentials in unknown state
➜  .output git:(pc)