-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The fcos initialized by podman will disconnect the network, but the manually initialized fcos will not, the same qemu #18177
Comments
[56254.698060] ------------[ cut here ]------------
[56254.707183] NETDEV WATCHDOG: enp0s1 (virtio_net): transmit queue 0 timed out
[56254.713883] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x26c/0x27c
[56254.716598] Modules linked in: 9p fscache netfs overlay rfkill binfmt_misc 9pnet_virtio 9pnet xfs crct10dif_ce polyval_ce polyval_generic ghash_ce sha3_ce sha512_ce sha512_arm64 virtio_net net_failover failover virtio_console virtio_blk virtio_mmio scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip6_tables ip_tables dm_multipath fuse qemu_fw_cfg
[56254.719669] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.1.18-200.fc37.aarch64 #1
[56254.719684] Hardware name: QEMU QEMU Virtual Machine, BIOS 0.0.0 02/06/2015
[56254.719686] pstate: 40400005 (nZcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[56254.719699] pc : dev_watchdog+0x26c/0x27c
[56254.719705] lr : dev_watchdog+0x26c/0x27c
[56254.719709] sp : ffff80000800bdb0
[56254.719711] x29: ffff80000800bdb0 x28: ffff800008efc804 x27: ffff80000800bed0
[56254.719717] x26: ffff80000a149008 x25: 0000000000000000 x24: ffff80000a8aec58
[56254.719722] x23: ffff80000a8a7000 x22: 0000000000000000 x21: ffff0000c064541c
[56254.719726] x20: ffff0000c0645000 x19: ffff0000c06454c8 x18: ffffffffffffffff
[56254.719731] x17: ffff8003f3ba0000 x16: ffff80000800c000 x15: ffff80000800b9a8
[56254.719735] x14: ffff80000ad7e104 x13: 74756f2064656d69 x12: 7420302065756575
[56254.719739] x11: 00000000ffffdfff x10: ffff80000a9a1220 x9 : ffff8000081ee7d0
[56254.719744] x8 : 000000000002ffe8 x7 : c0000000ffffdfff x6 : 0000000000000000
[56254.719748] x5 : ffff0003fdcec450 x4 : 0000000000000040 x3 : 0000000000000008
[56254.719753] x2 : 0000000000000104 x1 : ffff0000c038c400 x0 : 0000000000000000
[56254.719758] Call trace:
[56254.719760] dev_watchdog+0x26c/0x27c
[56254.719765] call_timer_fn+0x3c/0x1c4
[56254.720778] __run_timers+0x22c/0x2dc
[56254.720782] run_timer_softirq+0x38/0x60
[56254.720786] __do_softirq+0x168/0x418
[56254.720789] ____do_softirq+0x18/0x24
[56254.720806] call_on_irq_stack+0x2c/0x38
[56254.720821] do_softirq_own_stack+0x24/0x3c
[56254.720825] __irq_exit_rcu+0x120/0x170
[56254.721070] irq_exit_rcu+0x18/0x24
[56254.721073] el1_interrupt+0x38/0x70
[56254.721395] el1h_64_irq_handler+0x18/0x2c
[56254.721399] el1h_64_irq+0x68/0x6c
[56254.721402] default_idle_call+0x40/0x184
[56254.721406] cpuidle_idle_call+0x160/0x1b0
[56254.721430] do_idle+0xac/0x100
[56254.721432] cpu_startup_entry+0x30/0x34
[56254.721436] secondary_start_kernel+0xd8/0x100
[56254.721442] __secondary_switched+0xb0/0xb4
[56254.721694] ---[ end trace 0000000000000000 ]---
[56254.722194] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 5220000 usecs ago
[56259.735730] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 10240000 usecs ago
[56264.694435] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 15200000 usecs ago
[56269.734021] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 20240000 usecs ago
[56274.695155] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 25200000 usecs ago |
I think it is likely to be a problem with the network protocol stack of gvproxy |
Actually, you might need a kernel that includes:
that is, >= 6.2-rc3, in the guest. I consistently hit an issue very similar to the one you described (while testing passt), without that fix. |
Ok, then we can only wait for the upstream of fcos to update the kernel. Thank you for your communication。 @sbrivio-rh |
Linux localhost.localdomain 6.2.9-300.fc38.aarch64 #1 SMP PREEMPT_DYNAMIC Thu Mar 30 22:53:50 UTC 2023 aarch64 GNU/Linux The latest fcos kernel no longer reproduces this bug |
I'm not sure if this is the same bug, but my interface still dies after some time. Podman info:
Logs
Unsure if this is valuable information, but today I started the machine with This seems the same issue to me. If there's something against reopening this, I can open a new issue. |
Weird. Would you have a way to
It might be a similar issue, but I'm fairly sure the one I mentioned is fixed in |
This seems to bring podman back up. (stop working after 5h uptime) |
Nothing special in the logs, except the fact ardvark-dns starts having issues shortly before the crash.
But I am assuming this is a symptom rather than a cause. |
Issue Description
coreos/fedora-coreos-tracker#1463
The fcos initialized by podman will disconnect the network, but the manually initialized fcos will not, the same qemu
I started three qemu virtual machines for comparison
Steps to reproduce the issue
Steps to reproduce the issue
Describe the results you received
Describe the results you received
[56254.722194] virtio_net virtio0 enp0s1: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 5220000 usecs ago
Describe the results you expected
Use podman normally
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
QEMU emulator version 7.2.1
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
Additional information
For comparison, I used qemu to start three virtual machines for observation.
A qemu virtual machine that comes with podman
An archlinuxarm virtual machine
A fcos virtual machine started by qemu, the same version as the one that comes with podman
podman fcos:
Linux localhost.localdomain 6.1.18-200.fc37.aarch64 #1 SMP PREEMPT_DYNAMIC Sat Mar 11 16:03:54 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
qemu fcos:
Linux localhost.localdomain 6.1.18-200.fc37.aarch64 #1 SMP PREEMPT_DYNAMIC Sat Mar 11 16:03:54 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
archlinuxarm:
Linux x 6.2.10-1-aarch64-ARCH #1 SMP PREEMPT_DYNAMIC Fri Apr 7 10:32:52 MDT 2023 aarch64 GNU/Linux
The text was updated successfully, but these errors were encountered: