Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ssh connection failed: 'Connection refused' - NIC_RX_FILTER_CHANGED Cannot access instance #3071

Open
JackieTreeh0rn opened this issue May 12, 2023 · 10 comments
Labels

Comments

@JackieTreeh0rn
Copy link

JackieTreeh0rn commented May 12, 2023

Describe the bug
Cannot connect to instance. It was working fine until today. Now takes forever to start the instance and Multipass doesn't seem to register the state of the VM, always shows as running on menu. Issue is networking. It only shows internal interface 192.168.x It really sucks networking is so fragile with multipass as I truly love the small footprint and flexibility.

MacOS Firewall OFF

  • After VM starts examples:

    multipass shell GALAXY-DOCK
    shell failed: ssh connection failed: 'Connection refused

    multipass info GALAXY-DOCK
    info failed: ssh connection failed: 'Connection refused

Have tried resetting local.driver to qemu, no difference.

To Reproduce
How, and what happened?

  1. I install multipass with the following network flags -network name=en0,mac="52:54:00:32:c3:b4 to associate it with my wireless NIC. The mac addressed being passed is for the virtual NIC in the VM instance, in this case, enp0s2. I am forcing this MAC address on the instance because I have a static DHCP reservation configured for it.
  2. I am using the Docker | Portainer blueprint
  3. I also install avahi-daemon avahi-discover avahi-utils libnss-mdns mdns-scan for .local resolution.

Expected behavior
This worked fine for weeks now it doesn't.

Logs
Screenshot 2023-05-12 at 2 48 38 AM

[debug] [GALAXY-DOCK] QMP: {"timestamp": {"seconds": 1683851504, "microseconds": 962322}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/device[13]/virtio-backend"}}

[2023-05-11T20:36:24.608] [debug] [GALAXY-DOCK] Resetting the network

[2023-05-11T20:36:24.610] [debug] [GALAXY-DOCK] QMP: {"return": {}}

[2023-05-11T20:36:24.621] [debug] [GALAXY-DOCK] QMP: {"return": {}}

[2023-05-11T22:44:25.178] [debug] [base_vm] Error getting extra IP addresses: ssh connection failed: 'Connection refused'

[2023-05-11T22:50:42.393] [debug] [base_vm] Error getting extra IP addresses: ssh connection failed: 'Connection refused'

Additional info

  • Apple M1 Max
  • OS: MacOS Ventura 13.3.1 (a) (22E772610a)
  • Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000
  • multipass 1.11.1+mac
  • multipassd 1.11.1+mac
  • multipass info --all
    info failed: ssh connection failed: 'Connection refused'
  • multipass get local.driver qemu

Additional context
Recently connected to a new LG monitor and dock that has an ethernet port. Haven't tested yet without the dock as I'd like to understand the issue given as how networking seems so fragile from one use case to another, I would like to make my config as portable as possible without having to wipe out and redeploy so often.

@JackieTreeh0rn JackieTreeh0rn changed the title ssh connection failed: 'Connection refused' - Cannot access instance ssh connection failed: 'Connection refused' - NIC_RX_FILTER_CHANGED Cannot access instance May 13, 2023
@andrei-toterman
Copy link
Contributor

andrei-toterman commented May 16, 2023

Hey, @fsck66! The fact that you get a Connection refused should mean that the instance is starting, but something prevents sshd from accepting connections. Did you install any additional software right before this issue occurred or did any other configurations that might've affected it? If not, please provide the output of running the following command

sudo /Library/Application\ Support/com.canonical.multipass/bin/qemu-system-aarch64 -machine virt,highmem=off -accel hvf -drive file=/Library/Application\ Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -cpu cortex-a72 -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:2f:7b:9f -nic vmnet-bridged,ifname=en0,model=virtio-net-pci,mac=52:54:00:2f:d1:14 -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application\ Support/multipassd/qemu/vault/instances/GALAXY-DOCK/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 8 -m 8192M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -cdrom /var/root/Library/Application\ Support/multipassd/qemu/vault/instances/GALAXY-DOCK/cloud-init-config.iso -loadvm suspend -machine virt-7.1

which should open up a window with a console into the instance.

@JackieTreeh0rn
Copy link
Author

JackieTreeh0rn commented May 18, 2023

unfortunately I have by now redeploy the instance and multipass on the problem system. What will this qemu command do? does it launch a session into the VM no matter what, does it change anything?

Looking back I believe this occurred after upgrading UTM, which I rarely use, but I also have on my ARM apple silicon system to emulate x86 architecture on VM that doesn't have an ARM version. I know that also uses QEMU but I already had it loaded at the time and used in in conjunction with multipass. Would that have anything to do with it or do these solutions, including multipass, maintain their own QEMU? thanks

@townsend2010
Copy link
Contributor

Hi @fsck66!

As @andrei-toterman mentioned, we would like to see the output of the console that pops up when running the qemu command. Looking at it though, you need to add /Library/Application\ Support/com.canonical.multipass/bin before the qemu command. This is to help us diagnose if the instance is fully booting or if it's perhaps getting stuck loading a system service. Nothing will change anything by running that.

In essence, what the Connection refused tells us is that the instance has an IP address, but Multipass's attempt to connect to the instance via ssh is being refused by the instance which leads to the question, why is it refusing access? A few things may do that like the ssh config in the instance has been modified and is now refusing access, or sshd is trying to start and is stuck, or some software like a firewall in the instance is blocking access to port 22. We're hoping the console output may rule out some things.

Given these symptoms, it's highly unlikely UTM's qemu is causing issues with Multipass and seems to just be a coincidence.

@JackieTreeh0rn
Copy link
Author

Hi,

I am not sure I can help track my problem anymore since I've since redeployed and reconfigured my instance via netplan and so my config is different now. I did run the command just for sakes (I changed the new Mac addresses to reflect the new ones in the command everything else is the same) and it returns:

{"QMP": {"version": {"qemu": {"micro": 0, "minor": 1, "major": 7}, "package": ""}, "capabilities": ["oob"]}} qemu-system-aarch64: Snapshot 'suspend' does not exist in one or more devices

@upngo
Copy link

upngo commented Jul 3, 2023

I ran into a similar issue when upgrading to 1.12.0
For a while it was Waiting for SSH to be up
After reading about all the firewall issues I tried adding everything in multipass bin to the firewall 'accept incoming' list. killing sudo launchctl stop com.openssh.sshd and restarting.
This got me to the a similar state as here..
The start or launch commands both end up timing out and the last thing in the logs is

[2023-07-03T08:24:10.952] [debug] [new-sunbird] QMP: {"timestamp": {"seconds": 1688329450, "microseconds": 952046}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/device[6]/virtio-backend"}}

I uninstalled again and reverted back to 1.11.1 and things are working....so perhaps downgrading would work for someone in the same position...

@IknowJoseph
Copy link

I'm suffering the same issue on my M1 Mac running OS 14.2.1 and Multipass 1.12.0

I am not aware of any changes to host or guests, but one day the guest stopped booting. I ended up removing Multipass and reinstalling, creating a new VM, but get the same error. VMs are always in an unknown state and the last log entry reads NIC_RX_FILTER_CHANGED

I have uninstalled again and installed the 1.13 RC but get the same. Interestingly, the issue was originally reported on 1.11 and the poster above suggests the same happens on 1.12, but was fixed by reverting to a previous version. I have not gone back, but the fact that OP suffered on 1.11 suggests this may not be a reliable fix.

How does the issue survive an uninstall / reinstall? I'm purging config when prompted by the uninstall script.

Log entry of 1.13 attempt:

`[2024-01-03T15:43:42.847] [debug] [update] Latest Multipass release available is version 1.12.2
[2024-01-03T15:43:43.059] [debug] [blueprint provider] Loading "anbox-cloud-appliance" v1
[2024-01-03T15:43:43.061] [debug] [blueprint provider] Loading "charm-dev" v1
[2024-01-03T15:43:43.062] [debug] [blueprint provider] Loading "docker" v1
[2024-01-03T15:43:43.062] [debug] [blueprint provider] Loading "jellyfin" v1
[2024-01-03T15:43:43.063] [debug] [blueprint provider] Loading "minikube" v1
[2024-01-03T15:43:43.064] [debug] [blueprint provider] Loading "ros-noetic" v1
[2024-01-03T15:43:43.064] [debug] [blueprint provider] Loading "ros2-humble" v1
[2024-01-03T15:43:43.073] [info] [rpc] gRPC listening on unix:/var/run/multipass_socket
[2024-01-03T15:43:43.073] [debug] [async task] fetch manifest periodically
[2024-01-03T15:43:43.075] [info] [daemon] Starting Multipass 1.13.0-rc.1308+g240e6cae1.mac
[2024-01-03T15:43:43.075] [info] [daemon] Daemon arguments: /Library/Application Support/com.canonical.multipass/bin/multipassd --verbosity debug
[2024-01-03T15:43:44.049] [info] [VMImageHost] Did not find any supported products in "appliance"
[2024-01-03T15:44:22.788] [debug] [qemu-system-aarch64] [11729] started: qemu-system-aarch64 --version
[2024-01-03T15:47:08.374] [debug] [image vault] Verifying hash "f885a8e8f62ab2c39ab0442ea182b69d49ccd990d24791acb4f1724573d8120f"
[2024-01-03T15:47:13.547] [debug] [qemu-img] [11754] started: qemu-img info --output=json /var/root/Library/Caches/multipassd/qemu/vault/images/jammy-20231211/ubuntu-22.04-server-cloudimg-arm64.img
[2024-01-03T15:47:13.835] [debug] [qemu-img] [11755] started: qemu-img amend -o compat=1.1 /var/root/Library/Caches/multipassd/qemu/vault/images/jammy-20231211/ubuntu-22.04-server-cloudimg-arm64.img
[2024-01-03T15:47:13.855] [debug] [qemu-img] [11756] started: qemu-img info /var/root/Library/Caches/multipassd/qemu/vault/images/jammy-20231211/ubuntu-22.04-server-cloudimg-arm64.img
[2024-01-03T15:47:13.868] [debug] [qemu-img] [11757] started: qemu-img resize /var/root/Library/Application Support/multipassd/qemu/vault/instances/trim-katydid/ubuntu-22.04-server-cloudimg-arm64.img 5368709120
[2024-01-03T15:47:13.882] [debug] [qemu-img] [11758] started: qemu-img snapshot -l /var/root/Library/Application Support/multipassd/qemu/vault/instances/trim-katydid/ubuntu-22.04-server-cloudimg-arm64.img
[2024-01-03T15:47:13.890] [debug] [qemu-img] [11759] started: qemu-img amend -o compat=1.1 /var/root/Library/Application Support/multipassd/qemu/vault/instances/trim-katydid/ubuntu-22.04-server-cloudimg-arm64.img
[2024-01-03T15:47:13.896] [debug] [trim-katydid] process working dir ''
[2024-01-03T15:47:13.896] [info] [trim-katydid] process program 'qemu-system-aarch64'
[2024-01-03T15:47:13.896] [info] [trim-katydid] process arguments '-machine, virt,gic-version=3, -accel, hvf, -drive, file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on, -cpu, host, -nic, vmnet-shared,model=virtio-net-pci,mac=52:54:00:90:87:9c, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/trim-katydid/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 1, -m, 1024M, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/root/Library/Application Support/multipassd/qemu/vault/instances/trim-katydid/cloud-init-config.iso'
[2024-01-03T15:47:13.898] [debug] [qemu-system-aarch64] [11760] started: qemu-system-aarch64 -machine virt,gic-version=3 -nographic -dump-vmstate /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/multipassd.IBhJap
[2024-01-03T15:47:13.934] [info] [trim-katydid] process state changed to Starting
[2024-01-03T15:47:13.936] [info] [trim-katydid] process state changed to Running
[2024-01-03T15:47:13.936] [debug] [qemu-system-aarch64] [11761] started: qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:90:87:9c -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/trim-katydid/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 1 -m 1024M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/root/Library/Application Support/multipassd/qemu/vault/instances/trim-katydid/cloud-init-config.iso
[2024-01-03T15:47:13.936] [info] [trim-katydid] process started
[2024-01-03T15:47:13.936] [debug] [trim-katydid] Waiting for SSH to be up
[2024-01-03T15:47:14.260] [debug] [trim-katydid] QMP: {"QMP": {"version": {"qemu": {"micro": 0, "minor": 0, "major": 8}, "package": ""}, "capabilities": ["oob"]}}

[2024-01-03T15:47:14.306] [debug] [trim-katydid] QMP: {"return": {}}

[2024-01-03T15:47:27.426] [debug] [trim-katydid] QMP: {"timestamp": {"seconds": 1704296847, "microseconds": 426481}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/device[6]/virtio-backend"}}`

@IknowJoseph
Copy link

IknowJoseph commented Jan 3, 2024

I've got my install fixed now, although I had to delete & purge any installed VM. After reading:

#2387
#2853
#3003

I saw I had the same issue with an invalid /var/db/dhcpd_leases file - the file had multiple entries with the same name. I had to stop, delete & purge any running VM, delete the dhcpd_leases file and reboot. Everything worked fine on the freshly started host.

I guess this explains why the issue persisted across multiple multipass versions and installs - it was an issue with MacOS. I think I had a multipass crash that left the system in an unstable state.

Perhaps multipass should check that the dhcpd_leases list is sane? This should be straightforward to test by editing the file to include duplicate entries.

@townsend2010
Copy link
Contributor

Hi @IknowJoseph,

I'm glad you got it working!

Perhaps multipass should check that the dhcpd_leases list is sane?

Multipass doesn't manage this file, so I would be very reluctant to edit the file, but perhaps we could report that there are duplicate entries and let the user try to fix it.

@dimaqq
Copy link

dimaqq commented May 13, 2024

Thank you @IknowJoseph, I was stuck too and the total of {sudo kill multipass*, whack dhcp leases, reboot} did the trick. If I get stuck again, I may attempt differential diagnosis.

@thgruiz
Copy link

thgruiz commented Feb 7, 2025

I had a similar situation today (did a nmcli networking off inside a multipass instance) and after that, I could not start the instance properly (took 5 min until timeout on a 'multipass start ' command, and its status was unknown). I could not use SSH to connect (no IP), nor use multipass shell/multipass exec...

what I did:

  1. stopped the instance (multipass stop works, for some reason)
  2. mounted the hard drive image with: sudo qemu-nbd --connect=/dev/nbd0 /var/snap/multipass/common/data/multipassd/vault/instances//ubuntu-24.04-server-cloudimg-amd64.img
  3. edited the file /var/lib/NetworkManager/NetworkManager.state inside the VM's hard drive and changed NetworkingEnabled=false to NetworkingEnabled=true
  4. umounted the hard drive
  5. started the VM (now started normaly) with multipass start

I know step 3) above only works for the situation I was in, but the way to edit and correct wrong configurations on the VM filesystem can help in many other cases

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.1 LTS"

multipass --version
multipass 1.15.0
multipassd 1.15.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants