Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

error: program: qemu-system-x86_64; error: Process crashed #1358

Closed
YousefSaber opened this issue Feb 14, 2020 · 10 comments
Closed

error: program: qemu-system-x86_64; error: Process crashed #1358

YousefSaber opened this issue Feb 14, 2020 · 10 comments
Labels
bug high high importance

Comments

@YousefSaber
Copy link

YousefSaber commented Feb 14, 2020

Ubuntu 19.10
multipass 1.0.2

your app works very randomly sometimes it succeed to start the VM sometimes I got this error, what's the solution for this?

joseph@joseph-X541UJ:~$ sudo systemctl status snap.multipass.multipassd.service 
● snap.multipass.multipassd.service - Service for snap application multipass.multipassd
   Loaded: loaded (/etc/systemd/system/snap.multipass.multipassd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-02-14 11:22:48 EET; 27s ago
 Main PID: 9695 (multipassd)
    Tasks: 12 (limit: 4915)
   Memory: 33.9M
   CGroup: /system.slice/snap.multipass.multipassd.service
           ├─9695 /snap/multipass/1597/bin/multipassd --verbosity debug --logger platform
           └─9741 /snap/multipass/1597/usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/snap/multipass/common/data/multipassd

Feb 14 11:22:49 joseph-X541UJ dnsmasq[9741]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth 
Feb 14 11:22:49 joseph-X541UJ dnsmasq-dhcp[9741]: DHCP, IP range 10.226.208.2 -- 10.226.208.254, lease time infinite
Feb 14 11:22:49 joseph-X541UJ dnsmasq-dhcp[9741]: DHCP, sockets bound exclusively to interface mpqemubr0
Feb 14 11:22:49 joseph-X541UJ dnsmasq[9741]: using local addresses only for domain multipass
Feb 14 11:22:49 joseph-X541UJ dnsmasq[9741]: reading /etc/resolv.conf
Feb 14 11:22:49 joseph-X541UJ dnsmasq[9741]: using local addresses only for domain multipass
Feb 14 11:22:49 joseph-X541UJ dnsmasq[9741]: using nameserver 127.0.0.53#53
Feb 14 11:22:49 joseph-X541UJ dnsmasq[9741]: read /etc/hosts - 7 addresses
Feb 14 11:22:49 joseph-X541UJ dnsmasq-dhcp[9741]: read /var/snap/multipass/common/data/multipassd/network/dnsmasq.hosts
Feb 14 11:22:49 joseph-X541UJ multipassd[9695]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket, SSL:on
joseph@joseph-X541UJ:~$ multipass start ubuntu 
start failed: The following errors occurred:                                    
ubuntu: timed out waiting for response
joseph@joseph-X541UJ:~$ multipass start ubuntu                                  
Starting ubuntu -[2020-02-14T11:25:26.308] [error] [ubuntu] process error occurred Crashed program: qemu-system-x86_64; error: Process crashed

 /[2020-02-14T11:25:26.309] [error] [ubuntu] error: program: qemu-system-x86_64; error: Process crashed

start failed: cannot connect to the multipass socket                            
Please ensure multipassd is running and '/var/snap/multipass/common/multipass_socket' is accessible
joseph@joseph-X541UJ:~$ sudo systemctl status snap.multipass.multipassd.service                                  
● snap.multipass.multipassd.service - Service for snap application multipass.multipassd
   Loaded: loaded (/etc/systemd/system/snap.multipass.multipassd.service; enabled; vendor preset: enabled)
   Active: deactivating (stop-sigterm) (Result: core-dump) since Fri 2020-02-14 11:25:26 EET; 2s ago
  Process: 9695 ExecStart=/usr/bin/snap run multipass.multipassd (code=dumped, signal=SEGV)
 Main PID: 9695 (code=dumped, signal=SEGV)
    Tasks: 1 (limit: 4915)
   Memory: 2.1M
   CGroup: /system.slice/snap.multipass.multipassd.service
           └─9741 /snap/multipass/1597/usr/sbin/dnsmasq --strict-order --bind-interfaces --pid-file=/var/snap/multipass/common/data/multipassd

Feb 14 11:25:26 joseph-X541UJ multipassd[9695]: error: program: qemu-system-x86_64; error: Process crashed
Feb 14 11:25:26 joseph-X541UJ multipassd[9695]: Removing AppArmor policy: 
                                                
                                                #include <tunables/global>
                                                profile multipass.ubuntu.qemu-system-x86_64 flags=(attach_disconnected) {
                                                  #include <abstractions/base>
                                                  #include <abstractions/consoles>
                                                  #include <abstractions/nameservice>
                                                
                                                  # required for reading disk images
                                                  capability dac_override,
                                                  capability dac_read_search,
                                                  capability chown,
                                                
                                                  # needed to drop privileges
                                                  capability setgid,
                                                  capability setuid,
                                                
                                                  network inet stream,
                                                  network inet6 stream,
                                                
                                                  # Allow multipassd send qemu signals
                                                  signal (receive) peer=snap.multipass.multipassd,
                                                
                                                  /dev/net/tun rw,

the output of journalctl -fu snap.multipass.multipassd.service

joseph@joseph-X541UJ:~$ journalctl -fu snap.multipass.multipassd.service
-- Logs begin at Sat 2019-11-30 07:50:29 EET. --
Feb 14 12:16:03 joseph-X541UJ multipassd[4929]: error: program: qemu-system-x86_64; error: Process crashed
Feb 14 12:16:03 joseph-X541UJ multipassd[4929]: Removing AppArmor policy: 
                                                
                                                #include <tunables/global>
                                                profile multipass.ubuntu.qemu-system-x86_64 flags=(attach_disconnected) {
                                                  #include <abstractions/base>
                                                  #include <abstractions/consoles>
                                                  #include <abstractions/nameservice>
                                                
                                                  # required for reading disk images
                                                  capability dac_override,
                                                  capability dac_read_search,
                                                  capability chown,
                                                
                                                  # needed to drop privileges
                                                  capability setgid,
                                                  capability setuid,
                                                
                                                  network inet stream,
                                                  network inet6 stream,
                                                
                                                  # Allow multipassd send qemu signals
                                                  signal (receive) peer=snap.multipass.multipassd,
                                                
                                                  /dev/net/tun rw,
                                                  /dev/kvm rw,
                                                  /dev/ptmx rw,
                                                  /dev/kqemu rw,
                                                  @{PROC}/*/status r,
                                                  # When qemu is signaled to terminate, it will read cmdline of signaling
                                                  # process for reporting purposes. Allowing read access to a process
                                                  # cmdline may leak sensitive information embedded in the cmdline.
                                                  @{PROC}/@{pid}/cmdline r,
                                                  # Per man(5) proc, the kernel enforces that a thread may
                                                  # only modify its comm value or those in its thread group.
                                                  owner @{PROC}/@{pid}/task/@{tid}/comm rw,
                                                  @{PROC}/sys/kernel/cap_last_cap r,
                                                  owner @{PROC}/*/auxv r,
                                                  @{PROC}/sys/vm/overcommit_memory r,
                                                
                                                  # access to firmware's etc (selectively chosen for multipass' usage)
                                                  /snap/multipass/1597/qemu/* r,
                                                
                                                  # for save and resume
                                                  /{usr/,}bin/dash rmix,
                                                  /{usr/,}bin/dd rmix,
                                                  /{usr/,}bin/cat rmix,
                                                
                                                  # for restore
                                                  /{usr/,}bin/bash rmix,
                                                
                                                  # for file-posix getting limits since 9103f1ce
                                                  /sys/devices/**/block/*/queue/max_segments r,
                                                
                                                  # for gathering information about available host resources
                                                  /sys/devices/system/cpu/ r,
                                                  /sys/devices/system/node/ r,
                                                  /sys/devices/system/node/node[0-9]*/meminfo r,
                                                  /sys/module/vhost/parameters/max_mem_regions r,
                                                
                                                  # binary and its libs
                                                  /snap/multipass/1597/usr/bin/qemu-system-x86_64 ixr,
                                                  /snap/multipass/1597/{,usr/}lib/{,@{multiarch}/}{,**/}*.so* rm,
                                                
                                                  # CLASSIC ONLY: need to specify required libs from core snap
                                                  /{,var/lib/snapd/}snap/core18/*/{,usr/}lib/@{multiarch}/{,**/}*.so* rm,
                                                
                                                  # Disk images
                                                  /var/snap/multipass/common/data/multipassd/vault/instances/ubuntu/ubuntu-18.04-server-cloudimg-amd64.img rwk,  # QCow2 filesystem image
                                                  /var/snap/multipass/common/data/multipassd/vault/instances/ubuntu/cloud-init-config.iso rk,   # cloud-init ISO
                                                }
Feb 14 12:16:03 joseph-X541UJ multipassd[4929]: Failed to remove AppArmor policy 
                                                #include <tunables/global>
                                                profile multipass.ubuntu.qemu-system-x86_64 flags=(attach_disconnected) {
                                                  #include <abstractions/base>
                                                  #include <abstractions/consoles>
                                                  #include <abstractions/nameservice>
                                                
                                                  # required for reading disk images
                                                  capability dac_override,
                                                  capability dac_read_search,
                                                  capability chown,
                                                
                                                  # needed to drop privileges
                                                  capability setgid,
                                                  capability setuid,
                                                
                                                  network inet stream,
                                                  network inet6 stream,
                                                
                                                  # Allow multipassd send qemu signals
                                                  signal (receive) peer=snap.multipass.multipassd,
                                                
                                                  /dev/net/tun rw,
                                                  /dev/kvm rw,
                                                  /dev/ptmx rw,
                                                  /dev/kqemu rw,
                                                  @{PROC}/*/status r,
                                                  # When qemu is signaled to terminate, it will read cmdline of signaling
                                                  # process for reporting purposes. Allowing read access to a process
                                                  # cmdline may leak sensitive information embedded in the cmdline.
                                                  @{PROC}/@{pid}/cmdline r,
                                                  # Per man(5) proc, the kernel enforces that a thread may
                                                  # only modify its comm value or those in its thread group.
                                                  owner @{PROC}/@{pid}/task/@{tid}/comm rw,
                                                  @{PROC}/sys/kernel/cap_last_cap r,
                                                  owner @{PROC}/*/auxv r,
                                                  @{PROC}/sys/vm/overcommit_memory r,
                                                
                                                  # access to firmware's etc (selectively chosen for multipass' usage)
                                                  /snap/multipass/1597/qemu/* r,
                                                
                                                  # for save and resume
                                                  /{usr/,}bin/dash rmix,
                                                  /{usr/,}bin/dd rmix,
                                                  /{usr/,}bin/cat rmix,
                                                
                                                  # for restore
                                                  /{usr/,}bin/bash rmix,
                                                
                                                  # for file-posix getting limits since 9103f1ce
                                                  /sys/devices/**/block/*/queue/max_segments r,
                                                
                                                  # for gathering information about available host resources
                                                  /sys/devices/system/cpu/ r,
                                                  /sys/devices/system/node/ r,
                                                  /sys/devices/system/node/node[0-9]*/meminfo r,
                                                  /sys/module/vhost/parameters/max_mem_regions r,
                                                
                                                  # binary and its libs
                                                  /snap/multipass/1597/usr/bin/qemu-system-x86_64 ixr,
                                                  /snap/multipass/1597/{,usr/}lib/{,@{multiarch}/}{,**/}*.so* rm,
                                                
                                                  # CLASSIC ONLY: need to specify required libs from core snap
                                                  /{,var/lib/snapd/}snap/core18/*/{,usr/}lib/@{multiarch}/{,**/}*.so* rm,
                                                
                                                  # Disk images
                                                  /var/snap/multipass/common/data/multipassd/vault/instances/ubuntu/ubuntu-18.04-server-cloudimg-amd64.img rwk,  # QCow2 filesystem image
                                                  /var/snap/multipass/common/data/multipassd/vault/instances/ubuntu/cloud-init-config.iso rk,   # cloud-init ISO
                                                }
                                                    : errno=254 ()
Feb 14 12:16:03 joseph-X541UJ multipassd[4929]: QObject::connect: invalid null parameter
Feb 14 12:16:03 joseph-X541UJ multipassd[4929]: QObject::connect: invalid null parameter
Feb 14 12:16:03 joseph-X541UJ multipassd[4929]: QObject::connect: invalid null parameter
Feb 14 12:16:03 joseph-X541UJ multipassd[4929]: QObject::connect: invalid null parameter
Feb 14 12:16:03 joseph-X541UJ multipassd[4929]: QObject::connect: invalid null parameter
Feb 14 12:16:03 joseph-X541UJ multipassd[4929]: QObject::connect: invalid null parameter
Feb 14 12:16:03 joseph-X541UJ systemd[1]: snap.multipass.multipassd.service: Main process exited, code=dumped, status=11/SEGV
Feb 14 12:16:33 joseph-X541UJ systemd[1]: snap.multipass.multipassd.service: State 'stop-sigterm' timed out. Killing.
Feb 14 12:16:33 joseph-X541UJ systemd[1]: snap.multipass.multipassd.service: Killing process 4973 (dnsmasq) with signal SIGKILL.
Feb 14 12:16:33 joseph-X541UJ systemd[1]: snap.multipass.multipassd.service: Failed with result 'core-dump'.
Feb 14 12:16:33 joseph-X541UJ systemd[1]: snap.multipass.multipassd.service: Service RestartSec=100ms expired, scheduling restart.
Feb 14 12:16:33 joseph-X541UJ systemd[1]: snap.multipass.multipassd.service: Scheduled restart job, restart counter is at 4.
Feb 14 12:16:33 joseph-X541UJ systemd[1]: Stopped Service for snap application multipass.multipassd.
Feb 14 12:16:33 joseph-X541UJ systemd[1]: Started Service for snap application multipass.multipassd.
Feb 14 12:16:33 joseph-X541UJ multipassd[5446]: Using AppArmor support
Feb 14 12:16:33 joseph-X541UJ multipassd[5446]: Loading AppArmor policy: 
                                                
                                                #include <tunables/global>
                                                profile multipass.dnsmasq flags=(attach_disconnected) {
                                                  #include <abstractions/base>
                                                  #include <abstractions/nameservice>
                                                
                                                  capability chown,
                                                  capability net_bind_service,
                                                  capability setgid,
                                                  capability setuid,
                                                  capability dac_override,
                                                  capability dac_read_search,
                                                  capability net_admin,         # for DHCP server
                                                  capability net_raw,           # for DHCP server ping checks
                                                  network inet raw,
                                                  network inet6 raw,
                                                
                                                  # Allow multipassd send dnsmasq signals
                                                  signal (receive) peer=snap.multipass.multipassd,
                                                
                                                  # access to iface mtu needed for Router Advertisement messages in IPv6
                                                  # Neighbor Discovery protocol (RFC 2461)
                                                  @{PROC}/sys/net/ipv6/conf/*/mtu r,
                                                
                                                  # binary and its libs
                                                  /snap/multipass/1597/usr/sbin/dnsmasq ixr,
                                                  /snap/multipass/1597/{usr/,}lib/@{multiarch}/{,**/}*.so* rm,
                                                
                                                  # CLASSIC ONLY: need to specify required libs from core snap
                                                  /{,var/lib/snapd/}snap/core18/*/{,usr/}lib/@{multiarch}/{,**/}*.so* rm,
                                                
                                                  /var/snap/multipass/common/data/multipassd/network/dnsmasq.leases rw,           # Leases file
                                                  /var/snap/multipass/common/data/multipassd/network/dnsmasq.hosts r,             # Hosts file
                                                
                                                  /var/snap/multipass/common/data/multipassd/network/dnsmasq.pid w,     # pid file
                                                }
Feb 14 12:16:33 joseph-X541UJ multipassd[5488]: Applying AppArmor policy: multipass.dnsmasq
Feb 14 12:16:33 joseph-X541UJ dnsmasq[5490]: started, version 2.79 cachesize 150
Feb 14 12:16:33 joseph-X541UJ dnsmasq[5490]: compile time options: IPv6 GNU-getopt DBus i18n IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC loop-detect inotify
Feb 14 12:16:33 joseph-X541UJ dnsmasq-dhcp[5490]: DHCP, IP range 10.226.208.2 -- 10.226.208.254, lease time infinite
Feb 14 12:16:33 joseph-X541UJ dnsmasq-dhcp[5490]: DHCP, sockets bound exclusively to interface mpqemubr0
Feb 14 12:16:33 joseph-X541UJ dnsmasq[5490]: using local addresses only for domain multipass
Feb 14 12:16:33 joseph-X541UJ dnsmasq[5490]: reading /etc/resolv.conf
Feb 14 12:16:33 joseph-X541UJ dnsmasq[5490]: using local addresses only for domain multipass
Feb 14 12:16:33 joseph-X541UJ dnsmasq[5490]: using nameserver 127.0.0.53#53
Feb 14 12:16:33 joseph-X541UJ dnsmasq[5490]: read /etc/hosts - 7 addresses
Feb 14 12:16:33 joseph-X541UJ dnsmasq-dhcp[5490]: read /var/snap/multipass/common/data/multipassd/network/dnsmasq.hosts
Feb 14 12:16:33 joseph-X541UJ multipassd[5446]: gRPC listening on unix:/var/snap/multipass/common/multipass_socket, SSL:on

info about my Ubuntu VM

joseph@joseph-X541UJ:~$ multipass info ubuntu 
Name:           ubuntu
State:          Suspended
IPv4:           --
Release:        --
Image hash:     ae2c9391b71a (Ubuntu 18.04 LTS)
Load:           --
Disk usage:     --
Memory usage:   --
@Saviq
Copy link
Collaborator

Saviq commented Feb 14, 2020

Hi @YousefSaber, could you please attach the file created from journalctl -u snap.multipass* > multipass.log?

It's unclear from the excerpts what exactly is crashing and where.

@Saviq Saviq added the question label Feb 14, 2020
@YousefSaber
Copy link
Author

YousefSaber commented Feb 14, 2020

Hi @YousefSaber, could you please attach the file created from journalctl -u snap.multipass* > multipass.log?

It's unclear from the excerpts what exactly is crashing and where.

here's the output file generated from the command
joseph@joseph-X541UJ:~$ journalctl -u snap.multipass.multipassd.service > multipass.log

multipass.log

@Saviq
Copy link
Collaborator

Saviq commented Feb 14, 2020

OK so the problematic bit is this:

Feb 12 23:21:57 joseph-X541UJ dnsmasq-dhcp[1350]: DHCP packet received on mpqemubr0 which has no address
…
Feb 12 23:26:00 joseph-X541UJ dnsmasq[24487]: failed to create listening socket for 10.226.208.1: Cannot assign requested address

Somehow, the mpqemubr0 bridge that the instances connect to has lost its IP address. This first started a couple nights ago. Any idea how it could have happened?

This ultimately resulted in multipassd crashing:

Feb 12 23:25:29 joseph-X541UJ systemd[1]: snap.multipass.multipassd.service: Main process exited, code=dumped, status=11/SEGV

You then rebooted and things went back to normal.

You also hit #1304 - we have a fix for that in master. You could either refresh to edge (sudo snap refresh multipass --edge), or sudo apt install qemu-kvm as a workaround.

Thanks for bearing with us, let us know please if this fixed your current problem.

@YousefSaber
Copy link
Author

YousefSaber commented Feb 14, 2020

Somehow, the mpqemubr0 bridge that the instances connect to has lost its IP address. This first started a couple nights ago. Any idea how it could have happened?

@Saviq I don't know but I can give some hints and clues that may have lead to this, my computer loses its internet connection after recovering from sleep mode then I had to restart my network in order to establish my internet connection again

joseph@joseph-X541UJ:~$ nmcli networking off
joseph@joseph-X541UJ:~$ nmcli networking on

hint 2: I may have shut down my computer while the VM was running

I don't know exactly what causes this issue.

the last time my VM was running I was adding public keys from my host machine to my VM
I followed the steps in this link, and I was establishing ssh connections from my host to VM.

as for your solutions, none of them unfortunately worked.

first I installed qemu-kvm then restarted my multipass service before trying to start my ubuntu VM but didn't work ( I'll show sequence of commands without the output of each for simplicity )

joseph@joseph-X541UJ:~$ sudo apt install qemu-kvm 
joseph@joseph-X541UJ:~$ sudo systemctl restart snap.multipass.multipassd.service 
joseph@joseph-X541UJ:~$ multipass start ubuntu 

the associated multipass2.log file is here

multipass2.log

then I updated to edge version of the app didn't also work

I followed also the above steps, here is the generated multipass.log3

multipass3.log

one more thing restarting my system no longer solves the problem

@Saviq Saviq added bug high high importance and removed question labels Feb 14, 2020
@Saviq
Copy link
Collaborator

Saviq commented Feb 14, 2020

Hi @YousefSaber, thanks for digging. I now have a reproducer for the crash (#1360) and we'll get it fixed, thanks!

In the mean time, to recover your instance:

snap stop multipass
<favourite_editor> /var/snap/multipass/common/data/multipassd/multipassd-vm-instances.json
# set "machine_type" to "pc-i440fx-bionic" for the broken instance
snap start multipass

I'm working on a programmatic fix for that, too.

@Saviq
Copy link
Collaborator

Saviq commented Feb 14, 2020

@YousefSaber I just put #1361 up, when @multipass-ci-bot publishes a build, you should be able to snap refresh --channel edge/pr1361 and your instance should be back up.

@YousefSaber
Copy link
Author

YousefSaber commented Feb 14, 2020

sudo apt install qemu-kvm
snap stop multipass
<favourite_editor> /var/snap/multipass/common/data/multipassd/multipassd-vm-instances.json
# set "machine_type" to "pc-i440fx-bionic" for the broken instance
snap start multipass

those commands solved the issue thanks a lot ( didn't need to restart my PC )

I'll switch to the new channel edge version of the app

@Saviq
Copy link
Collaborator

Saviq commented Feb 14, 2020

@YousefSaber you're better off on edge, since #1304 is your underlying problem, and it's fixed there already.

As for your networking issue, that's quite interesting, but indeed your nmcli commands, or maybe what causes them to be a requirement, is likely the original cause.

@Saviq
Copy link
Collaborator

Saviq commented Feb 14, 2020

I'm going to close this, then - please reopen if there's something more. I'll file a new issue to consider the network problem.

@YousefSaber
Copy link
Author

YousefSaber commented Jan 3, 2021

@Saviq the issue has surfaced again

snap stop multipass
<favourite_editor> /var/snap/multipass/common/data/multipassd/multipassd-vm-instances.json
# set "machine_type" to "pc-i440fx-bionic" for the broken instance
snap start multipass

doesn't work

this time I'm running Ubuntu 20.10
Multipass version

joseph@joseph-X541UJ:~$ multipass --version
multipass  1.6.0-dev.243+gf2c175f7
multipassd 1.6.0-dev.243+gf2c175f7

the stable version doesn't work also

I can't reopen the issue you may want to reopen it.

joseph@joseph-X541UJ:~$ multipass start ubuntu 
start failed: The following errors occurred:                                    
ubuntu: timed out waiting for response
joseph@joseph-X541UJ:~$ multipass start ubuntu                                  
Starting ubuntu \[2021-01-03T11:38:15.811] [error] [ubuntu] Qemu failed to start: program: qemu-system-x86_64; error: Process operation timed out

start failed: failed to start qemu instance: program: qemu-system-x86_64; error: Process operation timed out

#1384
#540

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug high high importance
Projects
None yet
Development

No branches or pull requests

2 participants