Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hit with host crash while running host stress tests #20

Closed
sathnaga opened this issue Oct 30, 2017 · 7 comments
Closed

hit with host crash while running host stress tests #20

sathnaga opened this issue Oct 30, 2017 · 7 comments

Comments

@sathnaga
Copy link
Member

sathnaga commented Oct 30, 2017 •
edited by cdeadmin
Loading

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160748 </cde:info>

While trying to reproduce with host stress, I hit with the below host crash during xfs stress tests

enter ? for help
[link register ] c0000000002543f0 irq_work_run+0x30/0x50
[c000000ffff53cc0] c000000ffff53cf0 (unreliable)
[c000000ffff53cf0] c0000000001b7ca0 flush_smp_call_function_queue+0xf0/0x200
[c000000ffff53d70] c0000000000477ec smp_ipi_demux_relaxed+0x9c/0x110
[c000000ffff53db0] c0000000000903d4 icp_native_ipi_action+0x64/0x80
[c000000ffff53dd0] c000000000179420 __handle_irq_event_percpu+0x90/0x2d0
[c000000ffff53e90] c000000000179698 handle_irq_event_percpu+0x38/0x90
[c000000ffff53ed0] c00000000017fcf4 handle_percpu_irq+0x84/0xd0
[c000000ffff53f00] c000000000177b7c generic_handle_irq+0x4c/0x80
[c000000ffff53f20] c0000000000165d4 __do_irq+0x94/0x200
[c000000ffff53f90] c000000000029fa4 call_do_irq+0x14/0x24
[c0000007f87f3a50] c0000000000167dc do_IRQ+0x9c/0x110
[c0000007f87f3aa0] c000000000008c58 hardware_interrupt_common+0x158/0x160
--- Exception: 501 (Hardware Interrupt) at c0000000008eb664 snooze_loop+0xa4/0x190
[c0000007f87f3d90] c0000007f87f3dc0 (unreliable)
[c0000007f87f3dd0] c0000000008e83a4 cpuidle_enter_state+0xc4/0x3d0
[c0000007f87f3e30] c00000000015f73c call_cpuidle+0x4c/0x80
[c0000007f87f3e50] c00000000015fbe0 do_idle+0x2b0/0x350
[c0000007f87f3ec0] c00000000015fe8c cpu_startup_entry+0x3c/0x50
[c0000007f87f3ef0] c000000000048a74 start_secondary+0x4e4/0x530
[c0000007f87f3f90] c00000000000b16c start_secondary_prolog+0x10/0x14
b:mon>
@sathnaga
Copy link
Member Author

jenkins_job_log.txt

@sathnaga
Copy link
Member Author

looks like this patch , https://www.spinics.net/lists/linux-fsdevel/msg117031.html fixes this issue

@sathnaga sathnaga mentioned this issue Oct 30, 2017
Closed
@cdeadmin
Copy link

------- Comment (attachment only) From diegodo@br.ibm.com 2017-10-31 13:38:03 EDT-------

------- Comment (attachment only) From diegodo@br.ibm.com 2017-10-31 13:39:35 EDT-------

@cdeadmin
Copy link

cdeadmin commented Nov 3, 2017

------- Comment (attachment only) From satheera@in.ibm.com 2017-11-02 06:48:20 EDT-------

paulusmack pushed a commit that referenced this issue Nov 8, 2017
@rnav @mpe
Revert "powerpc64/elfv1: Only dereference function descriptor for non…
63be1a8 
…-text symbols"

This reverts commit 83e840c ("powerpc64/elfv1: Only dereference
function descriptor for non-text symbols").

Chandan reported that on newer kernels, trying to enable function_graph
tracer on ppc64 (BE) locks up the system with the following trace:

  Unable to handle kernel paging request for data at address 0x600000002fa30010
  Faulting instruction address: 0xc0000000001f1300
  Thread overran stack, or stack corrupted
  Oops: Kernel access of bad area, sig: 11 [#1]
  BE SMP NR_CPUS=2048 DEBUG_PAGEALLOC NUMA pSeries
  Modules linked in:
  CPU: 1 PID: 6586 Comm: bash Not tainted 4.14.0-rc3-00162-g6e51f1f-dirty #20
  task: c000000625c07200 task.stack: c000000625c07310
  NIP:  c0000000001f1300 LR: c000000000121cac CTR: c000000000061af8
  REGS: c000000625c088c0 TRAP: 0380   Not tainted  (4.14.0-rc3-00162-g6e51f1f-dirty)
  MSR:  8000000000001032 <SF,ME,IR,DR,RI>  CR: 28002848  XER: 00000000
  CFAR: c0000000001f1320 SOFTE: 0
  ...
  NIP [c0000000001f1300] .__is_insn_slot_addr+0x30/0x90
  LR [c000000000121cac] .kernel_text_address+0x18c/0x1c0
  Call Trace:
  [c000000625c08b40] [c0000000001bd040] .is_module_text_address+0x20/0x40 (unreliable)
  [c000000625c08bc0] [c000000000121cac] .kernel_text_address+0x18c/0x1c0
  [c000000625c08c50] [c000000000061960] .prepare_ftrace_return+0x50/0x130
  [c000000625c08cf0] [c000000000061b10] .ftrace_graph_caller+0x14/0x34
  [c000000625c08d60] [c000000000121b40] .kernel_text_address+0x20/0x1c0
  [c000000625c08df0] [c000000000061960] .prepare_ftrace_return+0x50/0x130
  ...
  [c000000625c0ab30] [c000000000061960] .prepare_ftrace_return+0x50/0x130
  [c000000625c0abd0] [c000000000061b10] .ftrace_graph_caller+0x14/0x34
  [c000000625c0ac40] [c000000000121b40] .kernel_text_address+0x20/0x1c0
  [c000000625c0acd0] [c000000000061960] .prepare_ftrace_return+0x50/0x130
  [c000000625c0ad70] [c000000000061b10] .ftrace_graph_caller+0x14/0x34
  [c000000625c0ade0] [c000000000121b40] .kernel_text_address+0x20/0x1c0

This is because ftrace is using ppc_function_entry() for obtaining the
address of return_to_handler() in prepare_ftrace_return(). The call to
kernel_text_address() itself gets traced and we end up in a recursive
loop.

Fixes: 83e840c ("powerpc64/elfv1: Only dereference function descriptor for non-text symbols")
Cc: stable@vger.kernel.org # v4.13+
Reported-by: Chandan Rajendra <chandan@linux.vnet.ibm.com>
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
@cdeadmin
Copy link

------- Comment From satheera@in.ibm.com 2017-11-10 04:17:55 EDT-------
It hit with another host crash while running with patched kernel

ltc-test-ci1 login: [82658.681159] 6-1.test[74093]: unhandled signal 11 at 0000000000002713 nip 00007fffb89a4a38 lr 000000001000061c code 1
[82659.782214] 5-1.test[78324]: unhandled signal 11 at 00000000000186a3 nip 00007fff815a4904 lr 0000000010000620 code 1
[82660.354907] 6-2.test[79590]: unhandled signal 11 at 00007fff7ecf0000 nip 0000000010000bbc lr 0000000010000bb0 code 1
[82660.448273] 6-1.test[80370]: unhandled signal 11 at 00007fffa2a90000 nip 0000000010000a68 lr 0000000010000a50 code 1
[82660.485638] 6-3.test[80670]: unhandled signal 11 at 00007fffaefb0000 nip 0000000010000ac8 lr 0000000010000ab0 code 1
[82664.082546] 12-1.test[34195]: unhandled signal 11 at 0000000000002713 nip 00007fffb5a34b18 lr 000000001000063c code 1
[82665.367560] 6-1.test[56157]: unhandled signal 11 at 0000000000002713 nip 00007fffa2a34aa8 lr 0000000010000620 code 1
[84700.426452] tm-signal-msr-r[54506]: bad frame in rt_sigreturn: 00007fffc1a415d0 nip 00007fffa9d4eff0 lr 00007fffa9f104d8
[84700.448761] tm-signal-stack[54517]: bad frame in setup_rt_frame: 0000000000000000 nip 0000000010000cc4 lr 0000000010000ca8
[84772.550674] Bad kernel stack pointer 7fffccc104c0 at c00000000000bffc
cpu 0x4a: Vector: 700 (Program Check) at [c00000003fc87d40]
pc: c00000000000bffc: fast_exception_return+0xac/0x150
lr: 0000000010001c34
sp: 7fffccc104c0
msr: 9000000102a03031
current = 0xc000000edd1d1a80
paca = 0xc00000000fd90900 softe: 0 irq_happened: 0x01
pid = 56240, comm = tm-signal-conte
Linux version 4.14.0-rc4+ (root@ltc-test-ci1.aus.stglabs.ibm.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-17) (GCC)) #2 SMP Tue Nov 7 07:38:54 EST 2017
WARNING: exception is not recoverable, can't continue
enter ? for help
SP (7fffccc104c0) is in userspace
4a:mon>

paulusmack pushed a commit that referenced this issue Nov 17, 2017
@davem330
tcp: gso: avoid refcount_t warning from tcp_gso_segment()
7ec318f 
When a GSO skb of truesize O is segmented into 2 new skbs of truesize N1
and N2, we want to transfer socket ownership to the new fresh skbs.

In order to avoid expensive atomic operations on a cache line subject to
cache bouncing, we replace the sequence :

refcount_add(N1, &sk->sk_wmem_alloc);
refcount_add(N2, &sk->sk_wmem_alloc); // repeated by number of segments

refcount_sub(O, &sk->sk_wmem_alloc);

by a single

refcount_add(sum_of(N) - O, &sk->sk_wmem_alloc);

Problem is :

In some pathological cases, sum(N) - O might be a negative number, and
syzkaller bot was apparently able to trigger this trace [1]

atomic_t was ok with this construct, but we need to take care of the
negative delta with refcount_t

[1]
refcount_t: saturated; leaking memory.
------------[ cut here ]------------
WARNING: CPU: 0 PID: 8404 at lib/refcount.c:77 refcount_add_not_zero+0x198/0x200 lib/refcount.c:77
Kernel panic - not syncing: panic_on_warn set ...

CPU: 0 PID: 8404 Comm: syz-executor2 Not tainted 4.14.0-rc5-mm1+ #20
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:16 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:52
 panic+0x1e4/0x41c kernel/panic.c:183
 __warn+0x1c4/0x1e0 kernel/panic.c:546
 report_bug+0x211/0x2d0 lib/bug.c:183
 fixup_bug+0x40/0x90 arch/x86/kernel/traps.c:177
 do_trap_no_signal arch/x86/kernel/traps.c:211 [inline]
 do_trap+0x260/0x390 arch/x86/kernel/traps.c:260
 do_error_trap+0x120/0x390 arch/x86/kernel/traps.c:297
 do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:310
 invalid_op+0x18/0x20 arch/x86/entry/entry_64.S:905
RIP: 0010:refcount_add_not_zero+0x198/0x200 lib/refcount.c:77
RSP: 0018:ffff8801c606e3a0 EFLAGS: 00010282
RAX: 0000000000000026 RBX: 0000000000001401 RCX: 0000000000000000
RDX: 0000000000000026 RSI: ffffc900036fc000 RDI: ffffed0038c0dc68
RBP: ffff8801c606e430 R08: 0000000000000001 R09: 0000000000000000
R10: ffff8801d97f5eba R11: 0000000000000000 R12: ffff8801d5acf73c
R13: 1ffff10038c0dc75 R14: 00000000ffffffff R15: 00000000fffff72f
 refcount_add+0x1b/0x60 lib/refcount.c:101
 tcp_gso_segment+0x10d0/0x16b0 net/ipv4/tcp_offload.c:155
 tcp4_gso_segment+0xd4/0x310 net/ipv4/tcp_offload.c:51
 inet_gso_segment+0x60c/0x11c0 net/ipv4/af_inet.c:1271
 skb_mac_gso_segment+0x33f/0x660 net/core/dev.c:2749
 __skb_gso_segment+0x35f/0x7f0 net/core/dev.c:2821
 skb_gso_segment include/linux/netdevice.h:3971 [inline]
 validate_xmit_skb+0x4ba/0xb20 net/core/dev.c:3074
 __dev_queue_xmit+0xe49/0x2070 net/core/dev.c:3497
 dev_queue_xmit+0x17/0x20 net/core/dev.c:3538
 neigh_hh_output include/net/neighbour.h:471 [inline]
 neigh_output include/net/neighbour.h:479 [inline]
 ip_finish_output2+0xece/0x1460 net/ipv4/ip_output.c:229
 ip_finish_output+0x85e/0xd10 net/ipv4/ip_output.c:317
 NF_HOOK_COND include/linux/netfilter.h:238 [inline]
 ip_output+0x1cc/0x860 net/ipv4/ip_output.c:405
 dst_output include/net/dst.h:459 [inline]
 ip_local_out+0x95/0x160 net/ipv4/ip_output.c:124
 ip_queue_xmit+0x8c6/0x18e0 net/ipv4/ip_output.c:504
 tcp_transmit_skb+0x1ab7/0x3840 net/ipv4/tcp_output.c:1137
 tcp_write_xmit+0x663/0x4de0 net/ipv4/tcp_output.c:2341
 __tcp_push_pending_frames+0xa0/0x250 net/ipv4/tcp_output.c:2513
 tcp_push_pending_frames include/net/tcp.h:1722 [inline]
 tcp_data_snd_check net/ipv4/tcp_input.c:5050 [inline]
 tcp_rcv_established+0x8c7/0x18a0 net/ipv4/tcp_input.c:5497
 tcp_v4_do_rcv+0x2ab/0x7d0 net/ipv4/tcp_ipv4.c:1460
 sk_backlog_rcv include/net/sock.h:909 [inline]
 __release_sock+0x124/0x360 net/core/sock.c:2264
 release_sock+0xa4/0x2a0 net/core/sock.c:2776
 tcp_sendmsg+0x3a/0x50 net/ipv4/tcp.c:1462
 inet_sendmsg+0x11f/0x5e0 net/ipv4/af_inet.c:763
 sock_sendmsg_nosec net/socket.c:632 [inline]
 sock_sendmsg+0xca/0x110 net/socket.c:642
 ___sys_sendmsg+0x31c/0x890 net/socket.c:2048
 __sys_sendmmsg+0x1e6/0x5f0 net/socket.c:2138

Fixes: 14afee4 ("net: convert sock.sk_wmem_alloc from atomic_t to refcount_t")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: syzbot <syzkaller@googlegroups.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
paulusmack pushed a commit that referenced this issue Nov 17, 2017
@rnav @gregkh
Revert "powerpc64/elfv1: Only dereference function descriptor for non…
9a0422b 
…-text symbols"

commit 63be1a8 upstream.

This reverts commit 83e840c ("powerpc64/elfv1: Only dereference
function descriptor for non-text symbols").

Chandan reported that on newer kernels, trying to enable function_graph
tracer on ppc64 (BE) locks up the system with the following trace:

  Unable to handle kernel paging request for data at address 0x600000002fa30010
  Faulting instruction address: 0xc0000000001f1300
  Thread overran stack, or stack corrupted
  Oops: Kernel access of bad area, sig: 11 [#1]
  BE SMP NR_CPUS=2048 DEBUG_PAGEALLOC NUMA pSeries
  Modules linked in:
  CPU: 1 PID: 6586 Comm: bash Not tainted 4.14.0-rc3-00162-g6e51f1f-dirty #20
  task: c000000625c07200 task.stack: c000000625c07310
  NIP:  c0000000001f1300 LR: c000000000121cac CTR: c000000000061af8
  REGS: c000000625c088c0 TRAP: 0380   Not tainted  (4.14.0-rc3-00162-g6e51f1f-dirty)
  MSR:  8000000000001032 <SF,ME,IR,DR,RI>  CR: 28002848  XER: 00000000
  CFAR: c0000000001f1320 SOFTE: 0
  ...
  NIP [c0000000001f1300] .__is_insn_slot_addr+0x30/0x90
  LR [c000000000121cac] .kernel_text_address+0x18c/0x1c0
  Call Trace:
  [c000000625c08b40] [c0000000001bd040] .is_module_text_address+0x20/0x40 (unreliable)
  [c000000625c08bc0] [c000000000121cac] .kernel_text_address+0x18c/0x1c0
  [c000000625c08c50] [c000000000061960] .prepare_ftrace_return+0x50/0x130
  [c000000625c08cf0] [c000000000061b10] .ftrace_graph_caller+0x14/0x34
  [c000000625c08d60] [c000000000121b40] .kernel_text_address+0x20/0x1c0
  [c000000625c08df0] [c000000000061960] .prepare_ftrace_return+0x50/0x130
  ...
  [c000000625c0ab30] [c000000000061960] .prepare_ftrace_return+0x50/0x130
  [c000000625c0abd0] [c000000000061b10] .ftrace_graph_caller+0x14/0x34
  [c000000625c0ac40] [c000000000121b40] .kernel_text_address+0x20/0x1c0
  [c000000625c0acd0] [c000000000061960] .prepare_ftrace_return+0x50/0x130
  [c000000625c0ad70] [c000000000061b10] .ftrace_graph_caller+0x14/0x34
  [c000000625c0ade0] [c000000000121b40] .kernel_text_address+0x20/0x1c0

This is because ftrace is using ppc_function_entry() for obtaining the
address of return_to_handler() in prepare_ftrace_return(). The call to
kernel_text_address() itself gets traced and we end up in a recursive
loop.

Fixes: 83e840c ("powerpc64/elfv1: Only dereference function descriptor for non-text symbols")
Reported-by: Chandan Rajendra <chandan@linux.vnet.ibm.com>
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
@cdeadmin
Copy link

cdeadmin commented Dec 5, 2017

------- Comment From satheera@in.ibm.com 2017-12-05 11:31:27 EDT-------
Tested on 4.14.0-3.dev.git68b4afb.el7.centos.ppc64le.

@cdeadmin cdeadmin closed this as completed Dec 5, 2017
@cdeadmin
Copy link

cdeadmin commented Dec 5, 2017

------- Comment From diegodo@br.ibm.com 2017-12-05 11:40:01 EDT-------
I'll close this bug, since this patch is already on hostos kernel tree and the bug is not reproducible anymore.

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sathnaga @cdeadmin