Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Which kernel process crashed or hangs? #1777

Closed
rodizio1 opened this issue Jan 4, 2017 · 16 comments
Closed

Which kernel process crashed or hangs? #1777

rodizio1 opened this issue Jan 4, 2017 · 16 comments

Comments

@rodizio1
Copy link

rodizio1 commented Jan 4, 2017

Not sure how long this has been going on, was editing some files, wanted to reboot, noticed reboot command didn't react, looked into top and dmesg and see this:

The system is still running, how can I identify this further?

top - 08:29:39 up  1:26,  0 users,  load average: 3.94, 2.71, 1.79
Tasks: 142 total,   2 running, 138 sleeping,   0 stopped,   2 zombie
%Cpu(s):  0.2 us,  0.3 sy,  0.0 ni, 74.4 id,  0.0 wa,  0.0 hi, 25.0 si,  0.0 st
KiB Mem:    886476 total,   648944 used,   237532 free,     9240 buffers
KiB Swap:        0 total,        0 used,        0 free.    52184 cached Mem
[ 3529.168488]  r5:b6093e18 r4:b6812c78
[ 3529.168498] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 3529.168502]  r10:b6812b38 r9:00000335 r8:b2ce15dc r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 3529.168512]  r4:b6812ac0
[ 3529.168521] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 3529.168524]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 3529.168535]  r4:8052fac0
[ 3529.168544] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 3529.168548]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 3529.168558]  r4:b6005cc0
[ 3529.168567] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 3529.168576] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 3529.168579]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 3529.168594] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 3529.168602] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3529.168606]  r8:8052f280 r7:b6093e4c r6:ffffffff r5:60000113 r4:80068ab8 r3:b6093e18
[ 3529.168620] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3529.168627] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3529.168632] Exception stack(0xb6093e18 to 0xb6093e60)
[ 3529.168637] 3e00:                                                       b47d0230 803e02cc
[ 3529.168644] 3e20: b47d0000 00006373 b6812280 00000000 00000200 80536100 8052f280 362e3000
[ 3529.168651] 3e40: 80536080 b6093e94 00000000 b6093e68 b6093e6c 80068ab8 60000113 ffffffff
[ 3529.168660] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 3529.168663]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 3529.168678] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 3529.168681]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 3529.168692]  r4:8052ffbc
[ 3529.168700] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 3529.168703]  r4:8052ffbc r3:8005f374
[ 3529.168713] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3529.168716]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 3529.168731] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3529.168738] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3529.168742] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 3529.168747] 3f40:                                                       00000000 b6812250
[ 3529.168754] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 3529.168761] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 3529.168770] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 3529.168778] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 3529.168786] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 3529.168790]  r7:80558348 r3:00000040
[ 3529.168799] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 3529.168802]  r5:00000055 r4:3607806a
[ 3592.171337] INFO: rcu_sched self-detected stall on CPU
[ 3592.171350] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1074566 
[ 3592.171356] 	 (t=3234178 jiffies g=5933 c=5932 q=961834)
[ 3592.171362] Task dump for CPU 2:
[ 3592.171366] swapper/2       R running      0     0      1 0x00000002
[ 3592.171372] Backtrace: 
[ 3592.171385] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 3592.171389]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 3592.171403] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 3592.171412] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 3592.171416]  r5:8053b980 r4:00000002
[ 3592.171427] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 3592.171430]  r4:00000003 r3:00000001
[ 3592.171440] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 3592.171443]  r8:362e3000 r7:000ead2a r6:8053b980 r5:80532e00 r4:b6815e00 r3:00323c0c
[ 3592.171459] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 3592.171463]  r10:b6812b38 r9:00000344 r8:5e139284 r7:00000344 r6:5e1393bd r5:00000000
[ 3592.171473]  r4:b6062a80
[ 3592.171483] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 3592.171487]  r5:b6093d80 r4:b6812c78
[ 3592.171497] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 3592.171500]  r10:b6812b38 r9:00000344 r8:5e139284 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 3592.171510]  r4:b6812ac0
[ 3592.171519] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 3592.171523]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 3592.171533]  r4:8052fac0
[ 3592.171542] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 3592.171546]  r10:80536080 r9:b6093dfc r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 3592.171556]  r4:b6005cc0
[ 3592.171566] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 3592.171574] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 3592.171578]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 3592.171593] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 3592.171601] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3592.171604]  r8:8053653c r7:b6093db4 r6:ffffffff r5:a0000113 r4:80068768 r3:b6093d80
[ 3592.171619] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3592.171626] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3592.171630] Exception stack(0xb6093d80 to 0xb6093dc8)
[ 3592.171637] 3d80: b47d0230 b6093dfc 00000000 803e02cc 00000002 b47d01e8 b47d0230 8052f280
[ 3592.171645] 3da0: 8053653c b6093dfc 80536080 b6093df4 b6093df8 b6093dd0 80068d48 80068768
[ 3592.171649] 3dc0: a0000113 ffffffff
[ 3592.171657] [<80068740>] (lock_timer_base) from [<80068d48>] (mod_timer+0x60/0x158)
[ 3592.171661]  r9:362e3000 r8:8052f280 r7:80536100 r6:00323c0b r5:b47d01e8 r4:b47d0230
[ 3592.171676] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 3592.171680]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 3592.171694] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 3592.171698]  r5:00000100 r4:b6812280
[ 3592.171707] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 3592.171711]  r6:00000200 r5:00000000 r4:b6812280
[ 3592.171723] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 3592.171726]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 3592.171741] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 3592.171745]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 3592.171755]  r4:8052ffbc
[ 3592.171763] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 3592.171766]  r4:8052ffbc r3:8005f374
[ 3592.171776] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3592.171779]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 3592.171794] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3592.171801] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3592.171805] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 3592.171810] 3f40:                                                       00000000 b6812250
[ 3592.171817] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 3592.171824] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 3592.171833] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 3592.171841] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 3592.171849] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 3592.171853]  r7:80558348 r3:00000040
[ 3592.171863] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 3592.171866]  r5:00000055 r4:3607806a
[ 3655.175337] INFO: rcu_sched self-detected stall on CPU
[ 3655.175350] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1095477 
[ 3655.175355] 	 (t=3297182 jiffies g=5933 c=5932 q=979510)
[ 3655.175361] Task dump for CPU 2:
[ 3655.175364] swapper/2       R running      0     0      1 0x00000002
[ 3655.175371] Backtrace: 
[ 3655.175384] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 3655.175388]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 3655.175403] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 3655.175412] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 3655.175416]  r5:8053b980 r4:00000002
[ 3655.175426] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 3655.175430]  r4:00000003 r3:00000001
[ 3655.175439] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 3655.175443]  r8:362e3000 r7:000ef236 r6:8053b980 r5:80532e00 r4:b6815e00 r3:00333228
[ 3655.175459] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 3655.175462]  r10:b6812b38 r9:00000353 r8:0968516c r7:00000353 r6:09685270 r5:00000000
[ 3655.175473]  r4:b6062a80
[ 3655.175481] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 3655.175485]  r5:b6093d80 r4:b6812c78
[ 3655.175495] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 3655.175498]  r10:b6812b38 r9:00000353 r8:0968516c r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 3655.175509]  r4:b6812ac0
[ 3655.175518] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 3655.175521]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 3655.175531]  r4:8052fac0
[ 3655.175541] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 3655.175544]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 3655.175554]  r4:b6005cc0
[ 3655.175565] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 3655.175573] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 3655.175577]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 3655.175591] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 3655.175599] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3655.175603]  r8:0000653c r7:b6093db4 r6:ffffffff r5:a0000113 r4:80068758 r3:b6093d80
[ 3655.175617] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3655.175624] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3655.175629] Exception stack(0xb6093d80 to 0xb6093dc8)
[ 3655.175636] 3d80: b47d0230 b6093dfc 00000000 803e02cc b47d0230 b47d01e8 b47d0230 0000f280
[ 3655.175643] 3da0: 0000653c 362e3000 80536080 b6093df4 b6093df8 b6093dd0 80068d48 80068758
[ 3655.175648] 3dc0: a0000113 ffffffff
[ 3655.175656] [<80068740>] (lock_timer_base) from [<80068d48>] (mod_timer+0x60/0x158)
[ 3655.175659]  r9:362e3000 r8:8052f280 r7:80536100 r6:00333227 r5:b47d01e8 r4:b47d0230
[ 3655.175674] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 3655.175678]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 3655.175692] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 3655.175695]  r5:00000100 r4:b6812280
[ 3655.175705] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 3655.175709]  r6:00000200 r5:00000000 r4:b6812280
[ 3655.175720] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 3655.175724]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 3655.175738] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 3655.175742]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 3655.175752]  r4:8052ffbc
[ 3655.175760] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 3655.175763]  r4:8052ffbc r3:8005f374
[ 3655.175773] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3655.175776]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 3655.175791] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3655.175798] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3655.175802] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 3655.175807] 3f40:                                                       00000000 b6812250
[ 3655.175814] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 3655.175821] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 3655.175830] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 3655.175838] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 3655.175846] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 3655.175850]  r7:80558348 r3:00000040
[ 3655.175859] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 3655.175862]  r5:00000055 r4:3607806a
[ 3718.178337] INFO: rcu_sched self-detected stall on CPU
[ 3718.178352] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1116394 
[ 3718.178356] 	 (t=3360185 jiffies g=5933 c=5932 q=998890)
[ 3718.178362] Task dump for CPU 2:
[ 3718.178365] swapper/2       R running      0     0      1 0x00000002
[ 3718.178372] Backtrace: 
[ 3718.178388] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 3718.178392]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 3718.178406] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 3718.178415] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 3718.178419]  r5:8053b980 r4:00000002
[ 3718.178430] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 3718.178433]  r4:00000003 r3:00000001
[ 3718.178444] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 3718.178448]  r8:362e3000 r7:000f3dea r6:8053b980 r5:80532e00 r4:b6815e00 r3:00342843
[ 3718.178464] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 3718.178467]  r10:b6812b38 r9:00000361 r8:b4adce7c r7:00000361 r6:b4adcf81 r5:00000000
[ 3718.178478]  r4:b6062a80
[ 3718.178486] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 3718.178490]  r5:b6093dd8 r4:b6812c78
[ 3718.178500] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 3718.178504]  r10:b6812b38 r9:00000361 r8:b4adce7c r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 3718.178514]  r4:b6812ac0
[ 3718.178523] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 3718.178527]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 3718.178537]  r4:8052fac0
[ 3718.178546] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 3718.178550]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 3718.178560]  r4:b6005cc0
[ 3718.178571] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 3718.178580] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 3718.178583]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 3718.178598] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 3718.178606] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3718.178610]  r8:8052f280 r7:b6093e0c r6:ffffffff r5:60000113 r4:803e02e8 r3:b6093dd8
[ 3718.178624] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3718.178632] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3718.178636] Exception stack(0xb6093dd8 to 0xb6093e20)
[ 3718.178641] 3dc0:                                                       b47d0070 00000000
[ 3718.178649] 3de0: 0000e3c6 0000e3c6 b47d0000 b47d01e8 803e02cc 80536100 8052f280 362e3000
[ 3718.178656] 3e00: 80536080 b6093e3c b6093e28 b6093e28 803e02e8 803e02e8 60000113 ffffffff
[ 3718.178664] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 3718.178668]  r5:00000100 r4:b6812280
[ 3718.178678] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 3718.178682]  r6:00000200 r5:00000000 r4:b6812280
[ 3718.178694] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 3718.178697]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 3718.178712] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 3718.178715]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 3718.178726]  r4:8052ffbc
[ 3718.178733] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 3718.178737]  r4:8052ffbc r3:8005f374
[ 3718.178746] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3718.178750]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 3718.178765] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3718.178772] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3718.178776] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 3718.178781] 3f40:                                                       00000000 b6812250
[ 3718.178788] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 3718.178796] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 3718.178804] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 3718.178813] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 3718.178821] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 3718.178824]  r7:80558348 r3:00000040
[ 3718.178833] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 3718.178837]  r5:00000055 r4:3607806a
[ 3781.181337] INFO: rcu_sched self-detected stall on CPU
[ 3781.181352] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1137304 
[ 3781.181356] 	 (t=3423188 jiffies g=5933 c=5932 q=1017167)
[ 3781.181362] Task dump for CPU 2:
[ 3781.181365] swapper/2       R running      0     0      1 0x00000002
[ 3781.181371] Backtrace: 
[ 3781.181388] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 3781.181392]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 3781.181407] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 3781.181415] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 3781.181419]  r5:8053b980 r4:00000002
[ 3781.181429] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 3781.181433]  r4:00000003 r3:00000001
[ 3781.181443] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 3781.181446]  r8:362e3000 r7:000f854f r6:8053b980 r5:80532e00 r4:b6815e00 r3:00351e5e
[ 3781.181462] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 3781.181466]  r10:b6812b38 r9:00000370 r8:5ff34b58 r7:00000370 r6:5ff34c91 r5:00000000
[ 3781.181476]  r4:b6062a80
[ 3781.181485] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 3781.181488]  r5:b6093d98 r4:b6812c78
[ 3781.181498] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 3781.181502]  r10:b6812b38 r9:00000370 r8:5ff34b58 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 3781.181512]  r4:b6812ac0
[ 3781.181521] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 3781.181524]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 3781.181534]  r4:8052fac0
[ 3781.181544] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 3781.181547]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 3781.181558]  r4:b6005cc0
[ 3781.181568] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 3781.181576] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 3781.181580]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 3781.181594] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 3781.181603] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3781.181606]  r8:00000000 r7:b6093dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6093d98
[ 3781.181621] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3781.181628] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3781.181632] Exception stack(0xb6093d98 to 0xb6093de0)
[ 3781.181638] 3d80:                                                       b6812280 40000113
[ 3781.181645] 3da0: 00000000 00000000 b47d0230 b6812280 00351e5d b6812280 00000000 362e3000
[ 3781.181652] 3dc0: 80536080 b6093df4 b6093df8 b6093de8 80068df0 803f014c 60000113 ffffffff
[ 3781.181661] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[ 3781.181669] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 3781.181673]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 3781.181687] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 3781.181690]  r5:00000100 r4:b6812280
[ 3781.181700] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 3781.181704]  r6:00000200 r5:00000000 r4:b6812280
[ 3781.181715] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 3781.181719]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 3781.181733] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 3781.181737]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 3781.181747]  r4:8052ffbc
[ 3781.181755] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 3781.181758]  r4:8052ffbc r3:8005f374
[ 3781.181768] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3781.181771]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 3781.181786] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3781.181793] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3781.181797] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 3781.181802] 3f40:                                                       00000000 b6812250
[ 3781.181809] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 3781.181816] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 3781.181826] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 3781.181835] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 3781.181842] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 3781.181846]  r7:80558348 r3:00000040
[ 3781.181854] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 3781.181858]  r5:00000055 r4:3607806a
[ 3844.185340] INFO: rcu_sched self-detected stall on CPU
[ 3844.185353] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1158219 
[ 3844.185358] 	 (t=3486192 jiffies g=5933 c=5932 q=1036368)
[ 3844.185364] Task dump for CPU 2:
[ 3844.185367] swapper/2       R running      0     0      1 0x00000002
[ 3844.185374] Backtrace: 
[ 3844.185387] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 3844.185391]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 3844.185408] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 3844.185416] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 3844.185420]  r5:8053b980 r4:00000002
[ 3844.185431] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 3844.185434]  r4:00000003 r3:00000001
[ 3844.185444] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 3844.185448]  r8:362e3000 r7:000fd050 r6:8053b980 r5:80532e00 r4:b6815e00 r3:0036147a
[ 3844.185464] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 3844.185468]  r10:b6812b38 r9:0000037f r8:0b480a40 r7:0000037f r6:0b480b45 r5:00000000
[ 3844.185478]  r4:b6062a80
[ 3844.185486] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 3844.185490]  r5:b6093d80 r4:b6812c78
[ 3844.185500] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 3844.185503]  r10:b6812b38 r9:0000037f r8:0b480a40 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 3844.185514]  r4:b6812ac0
[ 3844.185523] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 3844.185526]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 3844.185536]  r4:8052fac0
[ 3844.185545] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 3844.185549]  r10:80536080 r9:b6093dfc r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 3844.185559]  r4:b6005cc0
[ 3844.185569] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 3844.185577] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 3844.185581]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 3844.185595] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 3844.185603] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3844.185607]  r8:8053653c r7:b6093db4 r6:ffffffff r5:a0000113 r4:80068760 r3:b6093d80
[ 3844.185621] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3844.185628] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3844.185632] Exception stack(0xb6093d80 to 0xb6093dc8)
[ 3844.185640] 3d80: b47d0230 b6093dfc 00000000 803e02cc b47d0230 b47d01e8 b47d0230 0000f280
[ 3844.185647] 3da0: 8053653c b6093dfc 80536080 b6093df4 b6093df8 b6093dd0 80068d48 80068760
[ 3844.185652] 3dc0: a0000113 ffffffff
[ 3844.185660] [<80068740>] (lock_timer_base) from [<80068d48>] (mod_timer+0x60/0x158)
[ 3844.185663]  r9:362e3000 r8:8052f280 r7:80536100 r6:00361479 r5:b47d01e8 r4:b47d0230
[ 3844.185679] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 3844.185683]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 3844.185697] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 3844.185700]  r5:00000100 r4:b6812280
[ 3844.185710] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 3844.185714]  r6:00000200 r5:00000000 r4:b6812280
[ 3844.185726] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 3844.185729]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 3844.185744] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 3844.185747]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 3844.185757]  r4:8052ffbc
[ 3844.185765] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 3844.185768]  r4:8052ffbc r3:8005f374
[ 3844.185778] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3844.185781]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 3844.185796] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3844.185803] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3844.185807] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 3844.185812] 3f40:                                                       00000000 b6812250
[ 3844.185819] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 3844.185827] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 3844.185835] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 3844.185844] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 3844.185852] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 3844.185855]  r7:80558348 r3:00000040
[ 3844.185864] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 3844.185867]  r5:00000055 r4:3607806a
[ 3907.188338] INFO: rcu_sched self-detected stall on CPU
[ 3907.188357] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1179133 
[ 3907.188361] 	 (t=3549195 jiffies g=5933 c=5932 q=1054005)
[ 3907.188367] Task dump for CPU 2:
[ 3907.188371] swapper/2       R running      0     0      1 0x00000002
[ 3907.188377] Backtrace: 
[ 3907.188391] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 3907.188395]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 3907.188411] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 3907.188419] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 3907.188423]  r5:8053b980 r4:00000002
[ 3907.188433] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 3907.188436]  r4:00000003 r3:00000001
[ 3907.188446] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 3907.188450]  r8:362e3000 r7:00101535 r6:8053b980 r5:80532e00 r4:b6815e00 r3:00370a95
[ 3907.188466] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 3907.188470]  r10:b6812b38 r9:0000038d r8:b68d86b4 r7:0000038d r6:b68d87b9 r5:00000000
[ 3907.188480]  r4:b6062a80
[ 3907.188488] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 3907.188492]  r5:b6093dd8 r4:b6812c78
[ 3907.188502] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 3907.188505]  r10:b6812b38 r9:0000038d r8:b68d86b4 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 3907.188515]  r4:b6812ac0
[ 3907.188525] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 3907.188528]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 3907.188538]  r4:8052fac0
[ 3907.188548] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 3907.188551]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 3907.188562]  r4:b6005cc0
[ 3907.188571] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 3907.188579] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 3907.188583]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 3907.188598] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 3907.188606] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3907.188609]  r8:8052f280 r7:b6093e0c r6:ffffffff r5:60000113 r4:803e0304 r3:b6093dd8
[ 3907.188624] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3907.188631] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3907.188635] Exception stack(0xb6093dd8 to 0xb6093e20)
[ 3907.188640] 3dc0:                                                       00000000 b2640000
[ 3907.188647] 3de0: b55be880 00000000 b47d0000 b47d01e8 803e02cc 80536100 8052f280 362e3000
[ 3907.188655] 3e00: 80536080 b6093e3c b6093e28 b6093e28 803e02e8 803e0304 60000113 ffffffff
[ 3907.188665] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 3907.188668]  r5:00000100 r4:b6812280
[ 3907.188678] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 3907.188682]  r6:00000200 r5:00000000 r4:b6812280
[ 3907.188693] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 3907.188697]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 3907.188711] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 3907.188715]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 3907.188725]  r4:8052ffbc
[ 3907.188733] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 3907.188737]  r4:8052ffbc r3:8005f374
[ 3907.188746] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3907.188750]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 3907.188764] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3907.188771] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3907.188775] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 3907.188780] 3f40:                                                       00000000 b6812250
[ 3907.188788] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 3907.188795] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 3907.188804] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 3907.188812] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 3907.188820] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 3907.188824]  r7:80558348 r3:00000040
[ 3907.188832] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 3907.188836]  r5:00000055 r4:3607806a
[ 3970.192338] INFO: rcu_sched self-detected stall on CPU
[ 3970.192352] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1200046 
[ 3970.192356] 	 (t=3612199 jiffies g=5933 c=5932 q=1073214)
[ 3970.192362] Task dump for CPU 2:
[ 3970.192366] swapper/2       R running      0     0      1 0x00000002
[ 3970.192372] Backtrace: 
[ 3970.192386] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 3970.192390]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 3970.192404] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 3970.192413] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 3970.192417]  r5:8053b980 r4:00000002
[ 3970.192430] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 3970.192434]  r4:00000003 r3:00000001
[ 3970.192444] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 3970.192447]  r8:362e3000 r7:0010603e r6:8053b980 r5:80532e00 r4:b6815e00 r3:003800b1
[ 3970.192463] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 3970.192467]  r10:b6812b38 r9:0000039c r8:61e24604 r7:0000039c r6:61e2473d r5:00000000
[ 3970.192477]  r4:b6062a80
[ 3970.192485] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 3970.192489]  r5:b6093e18 r4:b6812c78
[ 3970.192499] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 3970.192502]  r10:b6812b38 r9:0000039c r8:61e24604 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 3970.192513]  r4:b6812ac0
[ 3970.192522] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 3970.192525]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 3970.192536]  r4:8052fac0
[ 3970.192545] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 3970.192548]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 3970.192559]  r4:b6005cc0
[ 3970.192568] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 3970.192576] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 3970.192580]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 3970.192595] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 3970.192603] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3970.192606]  r8:8052f280 r7:b6093e4c r6:ffffffff r5:60000113 r4:80068ab8 r3:b6093e18
[ 3970.192621] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3970.192628] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3970.192632] Exception stack(0xb6093e18 to 0xb6093e60)
[ 3970.192638] 3e00:                                                       b47d0230 803e02cc
[ 3970.192645] 3e20: b47d0000 00000307 b6812280 00000000 00000200 80536100 8052f280 362e3000
[ 3970.192652] 3e40: 80536080 b6093e94 00000000 b6093e68 b6093e6c 80068ab8 60000113 ffffffff
[ 3970.192660] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 3970.192664]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 3970.192678] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 3970.192682]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 3970.192692]  r4:8052ffbc
[ 3970.192700] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 3970.192703]  r4:8052ffbc r3:8005f374
[ 3970.192713] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 3970.192716]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 3970.192731] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 3970.192738] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 3970.192742] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 3970.192747] 3f40:                                                       00000000 b6812250
[ 3970.192754] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 3970.192761] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 3970.192769] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 3970.192778] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 3970.192786] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 3970.192789]  r7:80558348 r3:00000040
[ 3970.192798] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 3970.192801]  r5:00000055 r4:3607806a
[ 4033.195337] INFO: rcu_sched self-detected stall on CPU
[ 4033.195353] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1220953 
[ 4033.195357] 	 (t=3675202 jiffies g=5933 c=5932 q=1090955)
[ 4033.195363] Task dump for CPU 2:
[ 4033.195367] swapper/2       R running      0     0      1 0x00000002
[ 4033.195373] Backtrace: 
[ 4033.195387] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4033.195391]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4033.195405] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4033.195413] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4033.195417]  r5:8053b980 r4:00000002
[ 4033.195427] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4033.195431]  r4:00000003 r3:00000001
[ 4033.195441] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4033.195445]  r8:362e3000 r7:0010a58b r6:8053b980 r5:80532e00 r4:b6815e00 r3:0038f6cc
[ 4033.195460] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4033.195464]  r10:b6812b38 r9:000003ab r8:0d27c2e0 r7:000003ab r6:0d27c3e5 r5:00000000
[ 4033.195474]  r4:b6062a80
[ 4033.195483] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4033.195486]  r5:b6093d98 r4:b6812c78
[ 4033.195497] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4033.195500]  r10:b6812b38 r9:000003ab r8:0d27c2e0 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4033.195511]  r4:b6812ac0
[ 4033.195519] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4033.195523]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4033.195533]  r4:8052fac0
[ 4033.195542] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4033.195546]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4033.195556]  r4:b6005cc0
[ 4033.195565] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4033.195574] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4033.195577]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4033.195592] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4033.195600] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4033.195603]  r8:00000000 r7:b6093dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6093d98
[ 4033.195618] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4033.195625] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4033.195630] Exception stack(0xb6093d98 to 0xb6093de0)
[ 4033.195635] 3d80:                                                       b6812280 40000113
[ 4033.195642] 3da0: 00000000 00000000 b47d0230 b6812280 0038f6cb b6812280 00000000 362e3000
[ 4033.195650] 3dc0: 80536080 b6093df4 b6093df8 b6093de8 80068df0 803f014c 60000113 ffffffff
[ 4033.195659] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[ 4033.195667] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 4033.195671]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 4033.195686] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 4033.195689]  r5:00000100 r4:b6812280
[ 4033.195699] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 4033.195703]  r6:00000200 r5:00000000 r4:b6812280
[ 4033.195714] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4033.195718]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4033.195733] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4033.195736]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4033.195746]  r4:8052ffbc
[ 4033.195754] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4033.195758]  r4:8052ffbc r3:8005f374
[ 4033.195767] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4033.195771]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4033.195785] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4033.195792] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4033.195796] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4033.195801] 3f40:                                                       00000000 b6812250
[ 4033.195808] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4033.195816] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4033.195824] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4033.195833] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4033.195841] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4033.195844]  r7:80558348 r3:00000040
[ 4033.195853] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4033.195856]  r5:00000055 r4:3607806a
[ 4096.198346] INFO: rcu_sched self-detected stall on CPU
[ 4096.198366] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1241880 
[ 4096.198371] 	 (t=3738205 jiffies g=5933 c=5932 q=1112092)
[ 4096.198377] Task dump for CPU 2:
[ 4096.198380] swapper/2       R running      0     0      1 0x00000002
[ 4096.198387] Backtrace: 
[ 4096.198408] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4096.198414]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4096.198433] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4096.198446] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4096.198454]  r5:8053b980 r4:00000002
[ 4096.198467] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4096.198471]  r4:00000003 r3:00000001
[ 4096.198481] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4096.198485]  r8:362e3000 r7:0010f81c r6:8053b980 r5:80532e00 r4:b6815e00 r3:0039ece7
[ 4096.198506] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4096.198510]  r10:b6812b38 r9:000003b9 r8:b86d4366 r7:000003b9 r6:b86d4bbd r5:00000000
[ 4096.198520]  r4:b6062a80
[ 4096.198529] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4096.198533]  r5:b6093e18 r4:b6812c78
[ 4096.198543] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4096.198547]  r10:b6812b38 r9:000003b9 r8:b86d4366 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4096.198557]  r4:b6812ac0
[ 4096.198566] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4096.198571]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4096.198582]  r4:8052fac0
[ 4096.198597] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4096.198602]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4096.198612]  r4:b6005cc0
[ 4096.198622] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4096.198630] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4096.198634]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4096.198649] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4096.198658] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4096.198662]  r8:8052f280 r7:b6093e4c r6:ffffffff r5:60000113 r4:80068ab8 r3:b6093e18
[ 4096.198678] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4096.198686] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4096.198696] Exception stack(0xb6093e18 to 0xb6093e60)
[ 4096.198706] 3e00:                                                       b47d0230 803e02cc
[ 4096.198713] 3e20: b47d0000 0000b655 b6812280 00000000 00000200 80536100 8052f280 362e3000
[ 4096.198721] 3e40: 80536080 b6093e94 00000000 b6093e68 b6093e6c 80068ab8 60000113 ffffffff
[ 4096.198730] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4096.198734]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4096.198750] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4096.198753]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4096.198775]  r4:8052ffbc
[ 4096.198784] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4096.198787]  r4:8052ffbc r3:8005f374
[ 4096.198797] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4096.198800]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4096.198817] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4096.198829] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4096.198833] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4096.198838] 3f40:                                                       00000000 b6812250
[ 4096.198845] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4096.198853] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4096.198861] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4096.198872] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4096.198882] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4096.198890]  r7:80558348 r3:00000040
[ 4096.198905] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4096.198909]  r5:00000055 r4:3607806a
[ 4159.201337] INFO: rcu_sched self-detected stall on CPU
[ 4159.201350] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1262807 
[ 4159.201353] 	 (t=3801208 jiffies g=5933 c=5932 q=1130872)
[ 4159.201360] Task dump for CPU 2:
[ 4159.201363] swapper/2       R running      0     0      1 0x00000002
[ 4159.201369] Backtrace: 
[ 4159.201385] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4159.201389]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4159.201404] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4159.201413] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4159.201417]  r5:8053b980 r4:00000002
[ 4159.201427] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4159.201431]  r4:00000003 r3:00000001
[ 4159.201441] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4159.201444]  r8:362e3000 r7:00114178 r6:8053b980 r5:80532e00 r4:b6815e00 r3:003ae302
[ 4159.201460] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4159.201464]  r10:b6812b38 r9:000003c8 r8:63b2bc30 r7:000003c8 r6:63b2bd69 r5:00000000
[ 4159.201474]  r4:b6062a80
[ 4159.201482] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4159.201486]  r5:b6093e18 r4:b6812c78
[ 4159.201496] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4159.201499]  r10:b6812b38 r9:000003c8 r8:63b2bc30 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4159.201510]  r4:b6812ac0
[ 4159.201519] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4159.201522]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4159.201532]  r4:8052fac0
[ 4159.201541] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4159.201545]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4159.201555]  r4:b6005cc0
[ 4159.201568] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4159.201577] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4159.201581]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4159.201595] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4159.201603] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4159.201607]  r8:8052f280 r7:b6093e4c r6:ffffffff r5:60000113 r4:80068ab8 r3:b6093e18
[ 4159.201621] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4159.201629] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4159.201634] Exception stack(0xb6093e18 to 0xb6093e60)
[ 4159.201639] 3e00:                                                       b47d0230 803e02cc
[ 4159.201646] 3e20: b47d0000 00007f5f b6812280 00000000 00000200 80536100 8052f280 362e3000
[ 4159.201654] 3e40: 80536080 b6093e94 00000000 b6093e68 b6093e6c 80068ab8 60000113 ffffffff
[ 4159.201662] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4159.201666]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4159.201680] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4159.201684]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4159.201695]  r4:8052ffbc
[ 4159.201702] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4159.201706]  r4:8052ffbc r3:8005f374
[ 4159.201715] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4159.201719]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4159.201733] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4159.201740] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4159.201745] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4159.201750] 3f40:                                                       00000000 b6812250
[ 4159.201757] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4159.201764] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4159.201772] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4159.201781] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4159.201789] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4159.201792]  r7:80558348 r3:00000040
[ 4159.201801] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4159.201804]  r5:00000055 r4:3607806a
[ 4222.204337] INFO: rcu_sched self-detected stall on CPU
[ 4222.204350] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1283710 
[ 4222.204354] 	 (t=3864211 jiffies g=5933 c=5932 q=1148706)
[ 4222.204361] Task dump for CPU 2:
[ 4222.204364] swapper/2       R running      0     0      1 0x00000002
[ 4222.204370] Backtrace: 
[ 4222.204387] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4222.204390]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4222.204406] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4222.204415] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4222.204418]  r5:8053b980 r4:00000002
[ 4222.204429] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4222.204432]  r4:00000003 r3:00000001
[ 4222.204443] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4222.204447]  r8:362e3000 r7:00118722 r6:8053b980 r5:80532e00 r4:b6815e00 r3:003bd91d
[ 4222.204463] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4222.204466]  r10:b6812b38 r9:000003d7 r8:0ef838d8 r7:000003d7 r6:0ef839dd r5:00000000
[ 4222.204477]  r4:b6062a80
[ 4222.204485] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4222.204489]  r5:b6093d98 r4:b6812c78
[ 4222.204499] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4222.204502]  r10:b6812b38 r9:000003d7 r8:0ef838d8 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4222.204512]  r4:b6812ac0
[ 4222.204521] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4222.204525]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4222.204535]  r4:8052fac0
[ 4222.204544] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4222.204548]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4222.204558]  r4:b6005cc0
[ 4222.204566] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4222.204575] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4222.204578]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4222.204593] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4222.204601] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4222.204605]  r8:00000000 r7:b6093dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6093d98
[ 4222.204619] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4222.204626] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4222.204631] Exception stack(0xb6093d98 to 0xb6093de0)
[ 4222.204636] 3d80:                                                       b6812280 40000113
[ 4222.204643] 3da0: 00000000 00000000 b47d0230 b6812280 003bd91c b6812280 00000000 362e3000
[ 4222.204650] 3dc0: 80536080 b6093df4 b6093df8 b6093de8 80068df0 803f014c 60000113 ffffffff
[ 4222.204660] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[ 4222.204668] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 4222.204672]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 4222.204686] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 4222.204690]  r5:00000100 r4:b6812280
[ 4222.204700] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 4222.204703]  r6:00000200 r5:00000000 r4:b6812280
[ 4222.204715] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4222.204718]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4222.204733] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4222.204736]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4222.204746]  r4:8052ffbc
[ 4222.204754] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4222.204758]  r4:8052ffbc r3:8005f374
[ 4222.204767] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4222.204771]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4222.204785] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4222.204792] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4222.204797] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4222.204801] 3f40:                                                       00000000 b6812250
[ 4222.204809] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4222.204816] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4222.204824] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4222.204833] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4222.204841] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4222.204844]  r7:80558348 r3:00000040
[ 4222.204853] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4222.204856]  r5:00000055 r4:3607806a
[ 4285.207337] INFO: rcu_sched self-detected stall on CPU
[ 4285.207351] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1304610 
[ 4285.207355] 	 (t=3927214 jiffies g=5933 c=5932 q=1169529)
[ 4285.207361] Task dump for CPU 2:
[ 4285.207364] swapper/2       R running      0     0      1 0x00000002
[ 4285.207370] Backtrace: 
[ 4285.207387] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4285.207391]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4285.207405] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4285.207414] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4285.207418]  r5:8053b980 r4:00000002
[ 4285.207429] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4285.207432]  r4:00000003 r3:00000001
[ 4285.207442] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4285.207445]  r8:362e3000 r7:0011d879 r6:8053b980 r5:80532e00 r4:b6815e00 r3:003ccf38
[ 4285.207461] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4285.207465]  r10:b6812b38 r9:000003e5 r8:ba3db5b4 r7:000003e5 r6:ba3db6b9 r5:00000000
[ 4285.207475]  r4:b6062a80
[ 4285.207484] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4285.207487]  r5:b6093d98 r4:b6812c78
[ 4285.207497] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4285.207501]  r10:b6812b38 r9:000003e5 r8:ba3db5b4 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4285.207511]  r4:b6812ac0
[ 4285.207520] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4285.207524]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4285.207534]  r4:8052fac0
[ 4285.207543] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4285.207547]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4285.207557]  r4:b6005cc0
[ 4285.207566] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4285.207575] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4285.207578]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4285.207593] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4285.207601] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4285.207604]  r8:00000000 r7:b6093dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6093d98
[ 4285.207619] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4285.207626] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4285.207630] Exception stack(0xb6093d98 to 0xb6093de0)
[ 4285.207636] 3d80:                                                       b6812280 40000113
[ 4285.207643] 3da0: 00000000 00000000 b47d0230 b6812280 003ccf37 b6812280 00000000 362e3000
[ 4285.207650] 3dc0: 80536080 b6093df4 b6093df8 b6093de8 80068df0 803f014c 60000113 ffffffff
[ 4285.207659] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[ 4285.207667] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 4285.207670]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 4285.207684] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 4285.207688]  r5:00000100 r4:b6812280
[ 4285.207698] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 4285.207701]  r6:00000200 r5:00000000 r4:b6812280
[ 4285.207713] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4285.207717]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4285.207731] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4285.207735]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4285.207745]  r4:8052ffbc
[ 4285.207753] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4285.207756]  r4:8052ffbc r3:8005f374
[ 4285.207766] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4285.207769]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4285.207784] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4285.207791] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4285.207795] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4285.207800] 3f40:                                                       00000000 b6812250
[ 4285.207807] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4285.207814] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4285.207823] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4285.207831] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4285.207839] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4285.207843]  r7:80558348 r3:00000040
[ 4285.207851] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4285.207855]  r5:00000055 r4:3607806a
[ 4348.211342] INFO: rcu_sched self-detected stall on CPU
[ 4348.211356] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1325520 
[ 4348.211360] 	 (t=3990218 jiffies g=5933 c=5932 q=1187737)
[ 4348.211366] Task dump for CPU 2:
[ 4348.211369] swapper/2       R running      0     0      1 0x00000002
[ 4348.211375] Backtrace: 
[ 4348.211389] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4348.211393]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4348.211409] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4348.211418] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4348.211421]  r5:8053b980 r4:00000002
[ 4348.211432] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4348.211436]  r4:00000003 r3:00000001
[ 4348.211445] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4348.211449]  r8:362e3000 r7:00121f99 r6:8053b980 r5:80532e00 r4:b6815e00 r3:003dc554
[ 4348.211465] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4348.211469]  r10:b6812b38 r9:000003f4 r8:65927504 r7:000003f4 r6:65927609 r5:00000000
[ 4348.211479]  r4:b6062a80
[ 4348.211488] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4348.211491]  r5:b6093d98 r4:b6812c78
[ 4348.211501] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4348.211505]  r10:b6812b38 r9:000003f4 r8:65927504 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4348.211515]  r4:b6812ac0
[ 4348.211524] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4348.211528]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4348.211538]  r4:8052fac0
[ 4348.211548] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4348.211551]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4348.211562]  r4:b6005cc0
[ 4348.211570] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4348.211579] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4348.211582]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4348.211598] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4348.211606] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4348.211609]  r8:00000000 r7:b6093dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6093d98
[ 4348.211624] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4348.211631] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4348.211636] Exception stack(0xb6093d98 to 0xb6093de0)
[ 4348.211641] 3d80:                                                       b6812280 40000113
[ 4348.211648] 3da0: 00000000 00000000 b47d0230 b6812280 003dc553 b6812280 00000000 362e3000
[ 4348.211656] 3dc0: 80536080 b6093df4 b6093df8 b6093de8 80068df0 803f014c 60000113 ffffffff
[ 4348.211664] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[ 4348.211672] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 4348.211676]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 4348.211690] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 4348.211693]  r5:00000100 r4:b6812280
[ 4348.211703] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 4348.211707]  r6:00000200 r5:00000000 r4:b6812280
[ 4348.211719] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4348.211723]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4348.211737] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4348.211741]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4348.211751]  r4:8052ffbc
[ 4348.211759] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4348.211762]  r4:8052ffbc r3:8005f374
[ 4348.211772] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4348.211775]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4348.211791] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4348.211798] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4348.211802] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4348.211807] 3f40:                                                       00000000 b6812250
[ 4348.211814] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4348.211821] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4348.211830] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4348.211838] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4348.211847] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4348.211850]  r7:80558348 r3:00000040
[ 4348.211859] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4348.211862]  r5:00000055 r4:3607806a
[ 4411.215339] INFO: rcu_sched self-detected stall on CPU
[ 4411.215354] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1346463 
[ 4411.215358] 	 (t=4053222 jiffies g=5933 c=5932 q=1207126)
[ 4411.215364] Task dump for CPU 2:
[ 4411.215367] swapper/2       R running      0     0      1 0x00000002
[ 4411.215374] Backtrace: 
[ 4411.215387] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4411.215391]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4411.215407] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4411.215416] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4411.215419]  r5:8053b980 r4:00000002
[ 4411.215431] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4411.215435]  r4:00000003 r3:00000001
[ 4411.215444] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4411.215448]  r8:362e3000 r7:00126b56 r6:8053b980 r5:80532e00 r4:b6815e00 r3:003ebb70
[ 4411.215464] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4411.215467]  r10:b6812b38 r9:00000403 r8:10e73384 r7:00000403 r6:10e734bd r5:00000000
[ 4411.215478]  r4:b6062a80
[ 4411.215488] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4411.215491]  r5:b6093d98 r4:b6812c78
[ 4411.215501] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4411.215504]  r10:b6812b38 r9:00000403 r8:10e73384 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4411.215515]  r4:b6812ac0
[ 4411.215523] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4411.215527]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4411.215537]  r4:8052fac0
[ 4411.215547] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4411.215550]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4411.215560]  r4:b6005cc0
[ 4411.215569] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4411.215578] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4411.215582]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4411.215597] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4411.215604] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4411.215608]  r8:00000000 r7:b6093dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6093d98
[ 4411.215622] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4411.215630] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4411.215634] Exception stack(0xb6093d98 to 0xb6093de0)
[ 4411.215639] 3d80:                                                       b6812280 40000113
[ 4411.215646] 3da0: 00000000 00000000 b47d0230 b6812280 003ebb6f b6812280 00000000 362e3000
[ 4411.215653] 3dc0: 80536080 b6093df4 b6093df8 b6093de8 80068df0 803f014c 60000113 ffffffff
[ 4411.215662] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[ 4411.215670] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 4411.215674]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 4411.215688] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 4411.215692]  r5:00000100 r4:b6812280
[ 4411.215702] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 4411.215705]  r6:00000200 r5:00000000 r4:b6812280
[ 4411.215717] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4411.215720]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4411.215735] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4411.215738]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4411.215748]  r4:8052ffbc
[ 4411.215756] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4411.215760]  r4:8052ffbc r3:8005f374
[ 4411.215769] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4411.215773]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4411.215787] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4411.215794] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4411.215798] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4411.215804] 3f40:                                                       00000000 b6812250
[ 4411.215811] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4411.215818] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4411.215827] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4411.215836] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4411.215844] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4411.215848]  r7:80558348 r3:00000040
[ 4411.215856] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4411.215859]  r5:00000055 r4:3607806a
[ 4474.218338] INFO: rcu_sched self-detected stall on CPU
[ 4474.218352] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1367367 
[ 4474.218356] 	 (t=4116225 jiffies g=5933 c=5932 q=1226650)
[ 4474.218362] Task dump for CPU 2:
[ 4474.218366] swapper/2       R running      0     0      1 0x00000002
[ 4474.218372] Backtrace: 
[ 4474.218386] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4474.218390]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4474.218404] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4474.218413] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4474.218416]  r5:8053b980 r4:00000002
[ 4474.218429] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4474.218433]  r4:00000003 r3:00000001
[ 4474.218443] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4474.218446]  r8:362e3000 r7:0012b79a r6:8053b980 r5:80532e00 r4:b6815e00 r3:003fb18b
[ 4474.218462] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4474.218465]  r10:b6812b38 r9:00000411 r8:bc2cb0c8 r7:00000411 r6:bc2cb201 r5:00000000
[ 4474.218476]  r4:b6062a80
[ 4474.218484] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4474.218488]  r5:b6093d98 r4:b6812c78
[ 4474.218497] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4474.218501]  r10:b6812b38 r9:00000411 r8:bc2cb0c8 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4474.218511]  r4:b6812ac0
[ 4474.218520] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4474.218524]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4474.218534]  r4:8052fac0
[ 4474.218543] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4474.218547]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4474.218557]  r4:b6005cc0
[ 4474.218566] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4474.218575] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4474.218578]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4474.218593] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4474.218601] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4474.218605]  r8:00000000 r7:b6093dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6093d98
[ 4474.218619] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4474.218627] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4474.218631] Exception stack(0xb6093d98 to 0xb6093de0)
[ 4474.218636] 3d80:                                                       b6812280 40000113
[ 4474.218643] 3da0: 00000000 00000000 b47d0230 b6812280 003fb18a b6812280 00000000 362e3000
[ 4474.218650] 3dc0: 80536080 b6093df4 b6093df8 b6093de8 80068df0 803f014c 60000113 ffffffff
[ 4474.218659] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[ 4474.218667] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 4474.218670]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 4474.218685] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 4474.218688]  r5:00000100 r4:b6812280
[ 4474.218698] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 4474.218702]  r6:00000200 r5:00000000 r4:b6812280
[ 4474.218714] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4474.218717]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4474.218732] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4474.218735]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4474.218745]  r4:8052ffbc
[ 4474.218753] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4474.218757]  r4:8052ffbc r3:8005f374
[ 4474.218766] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4474.218770]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4474.218784] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4474.218791] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4474.218795] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4474.218800] 3f40:                                                       00000000 b6812250
[ 4474.218808] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4474.218815] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4474.218823] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4474.218832] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4474.218840] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4474.218843]  r7:80558348 r3:00000040
[ 4474.218852] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4474.218855]  r5:00000055 r4:3607806a
[ 4537.222338] INFO: rcu_sched self-detected stall on CPU
[ 4537.222352] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1388266 
[ 4537.222355] 	 (t=4179229 jiffies g=5933 c=5932 q=1247089)
[ 4537.222362] Task dump for CPU 2:
[ 4537.222365] swapper/2       R running      0     0      1 0x00000002
[ 4537.222372] Backtrace: 
[ 4537.222386] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4537.222389]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4537.222404] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4537.222413] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4537.222417]  r5:8053b980 r4:00000002
[ 4537.222427] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4537.222431]  r4:00000003 r3:00000001
[ 4537.222440] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4537.222444]  r8:362e3000 r7:00130771 r6:8053b980 r5:80532e00 r4:b6815e00 r3:0040a7a7
[ 4537.222460] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4537.222463]  r10:b6812b38 r9:00000420 r8:67816fe4 r7:00000420 r6:678170e9 r5:00000000
[ 4537.222474]  r4:b6062a80
[ 4537.222482] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4537.222486]  r5:b6093d98 r4:b6812c78
[ 4537.222496] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4537.222500]  r10:b6812b38 r9:00000420 r8:67816fe4 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4537.222510]  r4:b6812ac0
[ 4537.222519] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4537.222522]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4537.222533]  r4:8052fac0
[ 4537.222542] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4537.222545]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4537.222555]  r4:b6005cc0
[ 4537.222565] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4537.222574] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4537.222577]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4537.222592] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4537.222600] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4537.222603]  r8:00000000 r7:b6093dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6093d98
[ 4537.222618] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4537.222625] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4537.222629] Exception stack(0xb6093d98 to 0xb6093de0)
[ 4537.222634] 3d80:                                                       b6812280 40000113
[ 4537.222641] 3da0: 00000000 00000000 b47d0230 b6812280 0040a7a6 b6812280 00000000 362e3000
[ 4537.222649] 3dc0: 80536080 b6093df4 b6093df8 b6093de8 80068df0 803f014c 60000113 ffffffff
[ 4537.222657] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[ 4537.222665] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 4537.222668]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47d01e8 r4:b47d0000
[ 4537.222683] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 4537.222686]  r5:00000100 r4:b6812280
[ 4537.222696] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 4537.222700]  r6:00000200 r5:00000000 r4:b6812280
[ 4537.222711] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4537.222714]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4537.222729] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4537.222732]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4537.222743]  r4:8052ffbc
[ 4537.222750] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4537.222754]  r4:8052ffbc r3:8005f374
[ 4537.222764] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4537.222767]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4537.222782] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4537.222789] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4537.222793] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4537.222798] 3f40:                                                       00000000 b6812250
[ 4537.222805] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4537.222812] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4537.222821] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4537.222830] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4537.222838] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4537.222841]  r7:80558348 r3:00000040
[ 4537.222850] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4537.222853]  r5:00000055 r4:3607806a
[ 4600.225338] INFO: rcu_sched self-detected stall on CPU
[ 4600.225352] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1409241 
[ 4600.225356] 	 (t=4242232 jiffies g=5933 c=5932 q=1265034)
[ 4600.225362] Task dump for CPU 2:
[ 4600.225365] swapper/2       R running      0     0      1 0x00000002
[ 4600.225372] Backtrace: 
[ 4600.225387] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 4600.225391]  r6:80000193 r5:8053b980 r4:b6062a80 r3:00000000
[ 4600.225405] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 4600.225414] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 4600.225417]  r5:8053b980 r4:00000002
[ 4600.225427] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4600.225431]  r4:00000003 r3:00000001
[ 4600.225441] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 4600.225444]  r8:362e3000 r7:00134d8a r6:8053b980 r5:80532e00 r4:b6815e00 r3:00419dc2
[ 4600.225460] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 4600.225463]  r10:b6812b38 r9:0000042f r8:12c6ec58 r7:0000042f r6:12c6ed5d r5:00000000
[ 4600.225474]  r4:b6062a80
[ 4600.225483] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 4600.225486]  r5:b6093e18 r4:b6812c78
[ 4600.225496] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 4600.225500]  r10:b6812b38 r9:0000042f r8:12c6ec58 r7:b6812b00 r6:b6812b0c r5:b6812c78
[ 4600.225510]  r4:b6812ac0
[ 4600.225519] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 4600.225523]  r10:b6812b38 r9:b6812b58 r8:b6812b78 r7:b6812ac0 r6:00000003 r5:b6812ad4
[ 4600.225533]  r4:8052fac0
[ 4600.225542] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 4600.225546]  r10:80536080 r9:362e3000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6818580
[ 4600.225556]  r4:b6005cc0
[ 4600.225565] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4600.225574] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 4600.225577]  r8:00000000 r7:00000000 r6:b6093f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 4600.225592] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 4600.225600] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4600.225604]  r8:8052f280 r7:b6093e4c r6:ffffffff r5:60000113 r4:80068ab8 r3:b6093e18
[ 4600.225618] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4600.225626] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4600.225630] Exception stack(0xb6093e18 to 0xb6093e60)
[ 4600.225635] 3e00:                                                       b47d0230 803e02cc
[ 4600.225643] 3e20: b47d0000 0000c9f1 b6812280 00000000 00000200 80536100 8052f280 362e3000
[ 4600.225650] 3e40: 80536080 b6093e94 00000000 b6093e68 b6093e6c 80068ab8 60000113 ffffffff
[ 4600.225659] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 4600.225662]  r9:40000001 r8:00000100 r7:b6092000 r6:80536084 r5:00000001 r4:00000080
[ 4600.225676] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 4600.225680]  r10:b6092000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 4600.225690]  r4:8052ffbc
[ 4600.225698] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 4600.225702]  r4:8052ffbc r3:8005f374
[ 4600.225711] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 4600.225714]  r8:80554b44 r7:b6093f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6093f58
[ 4600.225729] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 4600.225736] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 4600.225740] Exception stack(0xb6093f58 to 0xb6093fa0)
[ 4600.225745] 3f40:                                                       00000000 b6812250
[ 4600.225752] 3f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6093fc8 80554b44 80536430
[ 4600.225759] 3f80: b6092000 b6093fb4 b6093fb8 b6093fa8 800100e0 800100e4 60000013 ffffffff
[ 4600.225768] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 4600.225777] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 4600.225785] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 4600.225788]  r7:80558348 r3:00000040
[ 4600.225797] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 4600.225800]  r5:00000055 r4:3607806a
[ 4663.229340] INFO: rcu_sched self-detected stall on CPU
[ 4663.229357] 	2-...: (1 GPs behind) idle=ad3/2/0 softirq=22940/22942 fqs=1430228 
@popcornmix
Copy link
Collaborator

Nothing stands out from dmesg log other than the CPU stalls.
Is this repeatable? What are you running when it occurs?
What do vcgencmd version and uname -a report?
Are you overclocking or using other non-standard config.txt settings?
Any hardware attached to Pi?
Does vcgencmd get_throttled ever return anything other than 0x0?

@rodizio1
Copy link
Author

rodizio1 commented Jan 5, 2017

Nothing stands out from dmesg log other than the CPU stalls.
Is this repeatable? What are you running when it occurs?

Was editing files. No load on the system. This kind of crash occured the first time.

top output after noticing the hang:

top - 08:33:03 up  1:29,  0 users,  load average: 4.00, 3.35, 2.23
Tasks: 140 total,   2 running, 136 sleeping,   0 stopped,   2 zombie
%Cpu(s):  0.6 us,  0.8 sy,  0.0 ni, 75.2 id,  0.1 wa,  0.0 hi, 23.3 si,  0.0 st
KiB Mem:    886476 total,   672532 used,   213944 free,     9240 buffers
KiB Swap:        0 total,        0 used,        0 free.    52264 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                     
30111 root      20   0    5120   2492   2164 R  12.4  0.3   0:00.02 top                                                                                                                         
    1 root      20   0   13496   3736   2780 D   0.0  0.4   0:01.63 systemd                                                                                                                     
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd                                                                                                                    
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.07 ksoftirqd/0                                                                                                                 
    4 root      20   0       0      0      0 S   0.0  0.0   0:00.50 kworker/0:0                                                                                                                 
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                                                                                
    7 root      20   0       0      0      0 S   0.0  0.0   0:08.95 rcu_sched                                                                                                                   
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                                                                                      
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/0                                                                                                                 
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.21 migration/1                                                                                                                 
   11 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/1                                                                                                                 
   13 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H                                                                                                                
   14 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/2                                                                                                                 
   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/2                                                                                                                 
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H                                                                                                                
   18 root      rt   0       0      0      0 S   0.0  0.0   0:00.17 migration/3                                                                                                                 
   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/3                                                                                                                 
   21 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H                                                                                                                
   22 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs                                                                                                                   
   23 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback                                                                                                                   
   24 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto                                                                                                                      
   25 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   26 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd                                                                                                                     
   28 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd0                                                                                                                     
   29 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 vmstat                                                                                                                      
   30 root      20   0       0      0      0 S   0.0  0.0   0:00.00 fsnotify_mark                                                                                                               
   39 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kthrotld                                                                                                                    
   40 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   41 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   42 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   43 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   44 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   45 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   46 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   47 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   48 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   49 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   50 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   51 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   52 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   53 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   54 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   55 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   56 root       1 -19       0      0      0 S   0.0  0.0   0:07.97 VCHIQ-0                                                                                                                     

ps ax output after noticing the hang:

root@wifibroadcast(ro):~# ps ax
  PID TTY      STAT   TIME COMMAND
    1 ?        Ds     0:01 /sbin/init nofsck noswap
    2 ?        S      0:00 [kthreadd]
    3 ?        S      0:00 [ksoftirqd/0]
    4 ?        S      0:00 [kworker/0:0]
    5 ?        S<     0:00 [kworker/0:0H]
    7 ?        S      0:08 [rcu_sched]
    8 ?        S      0:00 [rcu_bh]
    9 ?        S      0:00 [migration/0]
   10 ?        S      0:00 [migration/1]
   11 ?        S      0:00 [ksoftirqd/1]
   13 ?        S<     0:00 [kworker/1:0H]
   14 ?        S      0:00 [migration/2]
   15 ?        S      0:00 [ksoftirqd/2]
   17 ?        S<     0:00 [kworker/2:0H]
   18 ?        S      0:00 [migration/3]
   19 ?        S      0:00 [ksoftirqd/3]
   21 ?        S<     0:00 [kworker/3:0H]
   22 ?        S      0:00 [kdevtmpfs]
   23 ?        S<     0:00 [writeback]
   24 ?        S<     0:00 [crypto]
   25 ?        S<     0:00 [bioset]
   26 ?        S<     0:00 [kblockd]
   28 ?        S      0:00 [kswapd0]
   29 ?        S<     0:00 [vmstat]
   30 ?        S      0:00 [fsnotify_mark]
   39 ?        S<     0:00 [kthrotld]
   40 ?        S<     0:00 [bioset]
   41 ?        S<     0:00 [bioset]
   42 ?        S<     0:00 [bioset]
   43 ?        S<     0:00 [bioset]
   44 ?        S<     0:00 [bioset]
   45 ?        S<     0:00 [bioset]
   46 ?        S<     0:00 [bioset]
   47 ?        S<     0:00 [bioset]
   48 ?        S<     0:00 [bioset]
   49 ?        S<     0:00 [bioset]
   50 ?        S<     0:00 [bioset]
   51 ?        S<     0:00 [bioset]
   52 ?        S<     0:00 [bioset]
   53 ?        S<     0:00 [bioset]
   54 ?        S<     0:00 [bioset]
   55 ?        S<     0:00 [bioset]
   56 ?        S<     0:07 [VCHIQ-0]
   57 ?        S<     0:00 [VCHIQr-0]
   58 ?        S<     0:00 [VCHIQs-0]
   59 ?        S<     0:00 [iscsi_eh]
   60 ?        S<     0:00 [dwc_otg]
   61 ?        S<     0:00 [DWC Notificatio]
   63 ?        S      0:00 [irq/92-mmc1]
   64 ?        S      0:00 [kworker/3:1]
   65 ?        S      0:00 [VCHIQka-0]
   66 ?        S<     0:00 [SMIO]
   67 ?        S<     0:00 [deferwq]
   69 ?        S<     0:00 [bioset]
   70 ?        S      0:00 [mmcqd/0]
   71 ?        S      0:00 [kworker/1:1]
   72 ?        S      0:00 [jbd2/mmcblk0p2-]
   73 ?        S<     0:00 [ext4-rsv-conver]
   74 ?        S      0:00 [kworker/2:1]
   95 ?        S      0:00 [kworker/0:2]
  104 ?        S      0:00 [kworker/1:2]
  109 ?        S      0:00 [kworker/3:3]
  126 ?        Ss     0:00 /lib/systemd/systemd-udevd
  176 ?        R      0:00 [kworker/2:2]
  205 ?        S<     0:00 [cfg80211]
  210 ?        S<     0:00 [brcmf_wq/mmc1:0]
  211 ?        S      0:00 [brcmf_wdog/mmc1]
  220 ?        Ss     0:00 /usr/sbin/sshd -D
  235 tty5     Ss     0:00 /bin/login -f     
  236 tty3     Ss     0:00 /bin/login -f     
  237 tty7     Ss     0:00 /bin/login -f     
  238 tty2     Ss     0:00 /bin/login -f     
  239 tty11    Ss     0:00 /bin/login -f     
  240 tty6     Ss     0:00 /bin/login -f     
  241 tty9     Ss     0:00 /bin/login -f     
  242 tty1     Ss     0:00 /bin/login -f     
  243 tty8     Ss     0:00 /bin/login -f     
  245 tty12    Ss     0:00 /bin/login -f     
  246 tty4     Ss     0:00 /bin/login -f     
  247 tty10    Ss     0:00 /bin/login -f     
  266 ?        S<     0:00 [kworker/0:1H]
  317 tty2     S+     0:00 -bash
  320 tty5     S+     0:01 -bash
  322 tty4     S+     0:00 -bash
  331 tty12    S+     0:00 -bash
  345 tty3     S+     0:00 -bash
  346 tty1     S+     0:00 -bash
  351 tty11    S+     0:00 -bash
  355 tty10    S+     0:00 -bash
  362 tty6     S+     0:00 -bash
  363 tty9     S+     0:08 -bash
  508 tty4     S+     0:00 sleep 365d
  515 ?        S<     0:00 [kworker/3:1H]
  518 tty10    S+     0:00 sleep 365d
  522 tty3     S+     0:00 sleep 365d
  547 tty7     S+     0:05 -bash
  552 tty8     S+     0:00 -bash
  586 tty8     S+     0:00 sleep 365d
  635 tty1     SN+    0:00 cat /root/videofifo3
  651 tty1     S<+    0:00 /root/wifibroadcast/rx -p 0 -d 2 -b 12 -r 6 -f 1024 24050f953384
  652 tty1     S<+    0:00 tee /dev/fd/63 /dev/fd/62 /dev/fd/61
  653 tty1     S<+    0:00 /root/wifibroadcast_misc/ftee /root/videofifo1
  654 tty1     S+     0:00 -bash
  655 tty1     S+     0:00 -bash
  656 tty1     S+     0:00 -bash
  657 tty1     S<+    0:00 /root/wifibroadcast_misc/ftee /root/videofifo4
  658 tty1     S<+    0:00 /root/wifibroadcast_misc/ftee /root/videofifo2
  659 tty1     SN+    0:00 /root/wifibroadcast_misc/ftee /root/videofifo3
  665 tty2     SN+    0:00 cat /root/telemetryfifo3
  670 tty2     S+     0:00 cat /root/telemetryfifo1
  681 tty2     S<+    0:00 /root/wifibroadcast/rx -p 1 -d 2 -b 1 -r 0 -f 64 24050f953384
  682 tty2     S+     0:00 tee /dev/fd/63 /dev/fd/62 /dev/fd/61
  683 tty2     S+     0:00 /root/wifibroadcast_misc/ftee /root/telemetryfifo1
  684 tty2     S+     0:00 -bash
  685 tty2     S+     0:00 -bash
  686 tty2     S+     0:00 /root/wifibroadcast_misc/ftee /root/telemetryfifo2
  687 tty2     S+     0:00 -bash
  688 tty2     S<+    0:00 /root/wifibroadcast_misc/ftee /root/telemetryfifo4
  689 tty2     S+     0:00 /root/wifibroadcast_misc/ftee /root/telemetryfifo3
  705 ?        SNs    0:00 pump -i eth0 --no-ntp -h wifibrdcast-rx
  735 ?        Ss     0:01 sshd: root@pts/0    
  761 pts/0    Ss     0:00 -bash
  794 tty11    S+     0:00 cat /root/telemetryfifo1
  795 tty11    Sl+    0:26 /tmp/osd
  796 tty11    S+     0:00 sleep 365d
21538 ?        S      0:00 [kworker/u8:2]
27159 pts/0    Z      0:00 [systemd-tty-ask] <defunct>
27161 ?        Ds     0:00 (hwclock)               
27832 ?        Ss     0:00 sshd: root@pts/1    
27867 pts/1    Ss     0:00 -bash
27934 pts/0    Z      0:00 [systemd-tty-ask] <defunct>
28055 pts/0    D+     0:00 mount -o remount,ro /
28336 ?        S      0:00 [kworker/u8:0]
30117 tty5     S+     0:00 sleep 5
30130 tty9     S<+    0:00 cat /root/videofifo1
30131 tty9     S<l+   0:00 /opt/vc/src/hello_pi/hello_video/hello_video.bin.48-mm
30132 tty9     S+     0:00 sleep 5
30139 tty6     S+     0:00 sleep 3
30145 tty7     S+     0:00 sleep 1
30146 pts/1    R+     0:00 ps ax

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       1.7G  1.2G  419M  74% /
devtmpfs        433M     0  433M   0% /dev
tmpfs           433M  8.0K  433M   1% /dev/shm
tmpfs           433M  300K  433M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           433M     0  433M   0% /sys/fs/cgroup
tmpfs           640M     0  640M   0% /wbc_tmp
tmpfs            10M     0   10M   0% /var/tmp
tmpfs            10M   20K   10M   1% /var/log
tmpfs            10M  148K  9.9M   2% /tmp
/dev/mmcblk0p1   32M   18M   15M  55% /boot

Firmware is from the latest raspbian release:

Nov 16 2016 12:28:56 
Copyright (c) 2012 Broadcom
version 14ca844c69c2883572677a5cd7b02443eb653e4e (clean) (release)

Kernel is still an older one, 1000Hz timer frequency and dynamic ticks disabled (The latest 4.4.32 has issues with scheduling or whatever, it causes packtloss on the wifi cards)

Linux wifibroadcast 4.4.11-v7+ #24 SMP Thu Dec 15 16:56:49 CET 2016 armv7l GNU/Linux

Are you overclocking or using other non-standard config.txt settings?

CPU underclocked, rest overclocked, but not too much:

disable_overscan=1
dtparam=audio=off
start_x=1
gpu_mem=128
force_turbo=1
arm_freq=1000
over_voltage=4
gpu_freq=400
sdram_freq=433
bootcode_delay=0
boot_delay=0
disable_splash=1
max_usb_current=1
pi3-disable-bt
enable_uart=1
usb_mdio=0x7000

Any hardware attached to Pi?

A wifi stick (that wasn't connected to anything and no network-manager or whatever running that does stuff with it), HDMI monitor and ethernet cable. Powered via official Pi power supply.

vcgencmd get_throttled never returns anything other than 0x0.

@popcornmix
Copy link
Collaborator

Can you remove the overclock and disconnect the wifi stick. Confim if issue still occurs.
Why are you using usb_mdio=0x7000?
pi3-disable-bt isn't a valid config setting. You probably want dtoverlay=pi3-disable-bt

@popcornmix
Copy link
Collaborator

You really need to confirm the problem occurs with a kernel we have provided.
We can't really support custom kernels, where a custom change may have provoked the problem.
If 4.4.32 creates a wifi issue for you, then open another issue. Try to find exactly which update caused the issue - see:
https://github.com/Hexxeh/rpi-firmware/commits/master

If you click on each commit the end of the url contains a git hash. Run
sudo rpi-update <hash>
to revert back to that version. Report the first version with the problem.

@rodizio1
Copy link
Author

rodizio1 commented Jan 5, 2017

Why are you using usb_mdio=0x7000?

P33M told me it would help with USB flakiness, see this issue: #1763
(BTW, these issues are perfectly reprodicible, throughout different Raspbian and kernel versions)

pi3-disable-bt isn't a valid config setting. You probably want dtoverlay=pi3-disable-bt

Oh, didn't notice that, thanks. I'm using /dev/serial0 to exchange data with an external system (telemetry from flight control) and it works, I thought bluetooth would be there without that overlay? (Maybe I confuse something, if I remember correctly this was changed several times because of some issues).

You really need to confirm the problem occurs with a kernel we have provided.
We can't really support custom kernels, where a custom change may have provoked the problem.
If 4.4.32 creates a wifi issue for you, then open another issue. Try to find exactly which update caused >the issue - see:

This would take months to clearly reproduce and attribute to a certain kernel. During doing that, I would find other bugs, maybe unrelated, maybe not, which would consume even more time to analyze.

Until then, you will have bumped the kernel and firmware another 10-20 times, and when I open an issue then, you'll probably tell me to test with the latest stock kernel (understandable, ofcourse). And if it doesn't happen then, try to exactly find the one where the problem started. Hmm, back to step one.

Assuming, that the problem is then fixed in that future kernel (maybe it is already in 4.4.32 ...): Great, I'll start using it, and after some time, (during which another 10 kernels will be released) I'll find another issue (just like I just did ...). Report it here, and the story begins again.

See the issue with that approach?

I thought having a kernel bug occuring, having dmesg and other logs plus having the luck that the system was still usable to do further debugging would actually be a great chance to atleast narrow it down some.

@popcornmix
Copy link
Collaborator

Sometimes the log gives a clear indication of where the problem is.
I don't see anything obvious here (maybe someone else does).

Interestingly when searching for "rcu_sched self-detected stall on CPU" and raspberry pi I found #1253 which turned out to be overclock related, so confirming the issue still occurs without overclock seems like a good idea.

@rodizio1
Copy link
Author

rodizio1 commented Jan 12, 2017

Interestingly when searching for "rcu_sched self-detected stall on CPU" and raspberry pi I found #1253 which turned out to be overclock related, so confirming the issue still occurs without overclock seems like a good idea.

???

"rcu_sched self-detected stall on CPU" simply means that a stall has been detected. There can be a lot of things that cause this message to appear. That guy in the issue 1253 had the RAM overclocked to 500MHz and the dmesg output says "Unable to handle kernel paging request at virtual address fffffd48"

Well, whatever, problem occured again, this time not overclocked and I have the start of the dmesg output. There was again that 25% si load seen in top.

As probably nobody is going to look into this anyway, issue closed.

Maybe the logs below help somebody else trying to figure out what's going on.

[  324.072607] INFO: rcu_sched self-detected stall on CPU
[  324.072632] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=6999 
[  324.072638] 	 (t=21000 jiffies g=962 c=961 q=108)
[  324.072645] Task dump for CPU 1:
[  324.072650] swapper/1       R running      0     0      1 0x00000002
[  324.072660] Backtrace: 
[  324.072682] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  324.072687]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  324.072707] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  324.072718] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  324.072723]  r5:8053b980 r4:00000001
[  324.072738] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  324.072742]  r4:00000002 r3:00000001
[  324.072755] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  324.072760]  r8:362d7000 r7:0000006c r6:8053b980 r5:80532e00 r4:b6809e00 r3:00005e09
[  324.072781] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  324.072786]  r10:b6806b38 r9:0000004b r8:74324cf9 r7:0000004b r6:74324e66 r5:00000000
[  324.072799]  r4:b6062640
[  324.072812] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  324.072816]  r5:b6091d98 r4:b6806c78
[  324.072829] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  324.072833]  r10:b6806b38 r9:0000004b r8:74324cf9 r7:b6806b00 r6:b6806b0c r5:b6806c78
[  324.072847]  r4:b6806ac0
[  324.072859] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  324.072863]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  324.072876]  r4:8052fac0
[  324.072889] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  324.072893]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  324.072906]  r4:b6005cc0
[  324.072919] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  324.072930] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  324.072934]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  324.072953] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  324.072963] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  324.072968]  r8:00000000 r7:b6091dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6091d98
[  324.072987] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  324.072997] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  324.073003] Exception stack(0xb6091d98 to 0xb6091de0)
[  324.073010] 1d80:                                                       b6806280 40000113
[  324.073019] 1da0: 00000000 00000000 b47efe30 b6806280 00005e08 b6806280 00000000 362d7000
[  324.073029] 1dc0: 80536080 b6091df4 b6091df8 b6091de8 80068df0 803f014c 60000113 ffffffff
[  324.073041] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[  324.073051] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[  324.073056]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47efde8 r4:b47efc00
[  324.073074] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  324.073078]  r5:00000100 r4:b6806280
[  324.073092] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  324.073097]  r6:00000200 r5:00000000 r4:b6806280
[  324.073112] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  324.073117]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  324.073136] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  324.073141]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  324.073154]  r4:8052ffbc
[  324.073165] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  324.073170]  r4:8052ffbc r3:8005f374
[  324.073182] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  324.073186]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  324.073205] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  324.073214] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  324.073219] Exception stack(0xb6091f58 to 0xb6091fa0)
[  324.073226] 1f40:                                                       00000000 b6806250
[  324.073235] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  324.073244] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  324.073256] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  324.073267] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  324.073277] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  324.073281]  r7:80558348 r3:00000040
[  324.073294] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  324.073298]  r5:00000055 r4:3607806a
[  387.076604] INFO: rcu_sched self-detected stall on CPU
[  387.076621] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=28000 
[  387.076627] 	 (t=84004 jiffies g=962 c=961 q=121)
[  387.076635] Task dump for CPU 1:
[  387.076639] swapper/1       R running      0     0      1 0x00000002
[  387.076648] Backtrace: 
[  387.076672] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  387.076677]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  387.076697] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  387.076709] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  387.076713]  r5:8053b980 r4:00000001
[  387.076729] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  387.076733]  r4:00000002 r3:00000001
[  387.076747] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  387.076752]  r8:362d7000 r7:00000079 r6:8053b980 r5:80532e00 r4:b6809e00 r3:00015425
[  387.076773] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  387.076778]  r10:b6806b38 r9:0000005a r8:1f870b79 r7:0000005a r6:1f870ce6 r5:00000000
[  387.076791]  r4:b6062640
[  387.076803] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  387.076808]  r5:b6091d98 r4:b6806c78
[  387.076820] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  387.076825]  r10:b6806b38 r9:0000005a r8:1f870b79 r7:b6806b00 r6:b6806b0c r5:b6806c78
[  387.076838]  r4:b6806ac0
[  387.076850] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  387.076855]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  387.076868]  r4:8052fac0
[  387.076881] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  387.076885]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  387.076899]  r4:b6005cc0
[  387.076911] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  387.076922] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  387.076927]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  387.076946] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  387.076956] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  387.076960]  r8:00000000 r7:b6091dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6091d98
[  387.076979] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  387.076988] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  387.076994] Exception stack(0xb6091d98 to 0xb6091de0)
[  387.077000] 1d80:                                                       b6806280 40000113
[  387.077010] 1da0: 00000000 00000000 b47efe30 b6806280 00015424 b6806280 00000000 362d7000
[  387.077020] 1dc0: 80536080 b6091df4 b6091df8 b6091de8 80068df0 803f014c 60000113 ffffffff
[  387.077032] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[  387.077043] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[  387.077048]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47efde8 r4:b47efc00
[  387.077066] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  387.077071]  r5:00000100 r4:b6806280
[  387.077084] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  387.077089]  r6:00000200 r5:00000000 r4:b6806280
[  387.077104] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  387.077108]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  387.077127] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  387.077131]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  387.077144]  r4:8052ffbc
[  387.077155] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  387.077159]  r4:8052ffbc r3:8005f374
[  387.077171] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  387.077176]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  387.077196] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  387.077204] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  387.077210] Exception stack(0xb6091f58 to 0xb6091fa0)
[  387.077216] 1f40:                                                       00000000 b6806250
[  387.077225] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  387.077234] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  387.077246] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  387.077257] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  387.077267] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  387.077272]  r7:80558348 r3:00000040
[  387.077283] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  387.077287]  r5:00000055 r4:3607806a
[  450.080603] INFO: rcu_sched self-detected stall on CPU
[  450.080620] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=49000 
[  450.080624] 	 (t=147008 jiffies g=962 c=961 q=122)
[  450.080632] Task dump for CPU 1:
[  450.080637] swapper/1       R running      0     0      1 0x00000002
[  450.080645] Backtrace: 
[  450.080665] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  450.080671]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  450.080690] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  450.080702] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  450.080706]  r5:8053b980 r4:00000001
[  450.080720] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  450.080725]  r4:00000002 r3:00000001
[  450.080738] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  450.080742]  r8:362d7000 r7:0000007a r6:8053b980 r5:80532e00 r4:b6809e00 r3:00024a41
[  450.080763] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  450.080768]  r10:b6806b38 r9:00000068 r8:cadbca61 r7:00000068 r6:cadbcc02 r5:00000000
[  450.080781]  r4:b6062640
[  450.080792] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  450.080797]  r5:b6091dd8 r4:b6806c78
[  450.080810] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  450.080815]  r10:b6806b38 r9:00000068 r8:cadbca61 r7:b6806b00 r6:b6806b0c r5:b6806c78
[  450.080828]  r4:b6806ac0
[  450.080839] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  450.080844]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  450.080857]  r4:8052fac0
[  450.080869] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  450.080874]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  450.080887]  r4:b6005cc0
[  450.080899] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  450.080911] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  450.080915]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  450.080934] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  450.080944] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  450.080949]  r8:8052f280 r7:b6091e0c r6:ffffffff r5:60000113 r4:803e0354 r3:b6091dd8
[  450.080969] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  450.080978] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  450.080984] Exception stack(0xb6091dd8 to 0xb6091e20)
[  450.080990] 1dc0:                                                       00000000 40000113
[  450.081000] 1de0: 00000000 00000000 b47efc00 b47efde8 803e02cc 80536100 8052f280 362d7000
[  450.081009] 1e00: 80536080 b6091e3c b6091df8 b6091e28 80068df0 803e0354 60000113 ffffffff
[  450.081021] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  450.081025]  r5:00000100 r4:b6806280
[  450.081038] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  450.081043]  r6:00000200 r5:00000000 r4:b6806280
[  450.081059] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  450.081063]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  450.081082] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  450.081087]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  450.081100]  r4:8052ffbc
[  450.081110] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  450.081114]  r4:8052ffbc r3:8005f374
[  450.081127] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  450.081131]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  450.081150] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  450.081159] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  450.081164] Exception stack(0xb6091f58 to 0xb6091fa0)
[  450.081170] 1f40:                                                       00000000 b6806250
[  450.081179] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  450.081189] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  450.081200] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  450.081211] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  450.081221] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  450.081226]  r7:80558348 r3:00000040
[  450.081238] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  450.081242]  r5:00000055 r4:3607806a
[  513.084606] INFO: rcu_sched self-detected stall on CPU
[  513.084628] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=69981 
[  513.084633] 	 (t=210012 jiffies g=962 c=961 q=122)
[  513.084642] Task dump for CPU 1:
[  513.084646] swapper/1       R running      0     0      1 0x00000002
[  513.084655] Backtrace: 
[  513.084678] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  513.084683]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  513.084705] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  513.084716] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  513.084721]  r5:8053b980 r4:00000001
[  513.084735] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  513.084739]  r4:00000002 r3:00000001
[  513.084752] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  513.084757]  r8:362d7000 r7:0000007a r6:8053b980 r5:80532e00 r4:b6809e00 r3:0003405d
[  513.084778] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  513.084783]  r10:b6806b38 r9:00000077 r8:7630897d r7:00000077 r6:76308aea r5:00000000
[  513.084796]  r4:b6062640
[  513.084807] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  513.084812]  r5:b6091da8 r4:b6806c78
[  513.084824] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  513.084829]  r10:b6806b38 r9:00000077 r8:7630897d r7:b6806b00 r6:b6806b0c r5:b6806c78
[  513.084842]  r4:b6806ac0
[  513.084854] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  513.084859]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  513.084872]  r4:8052fac0
[  513.084884] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  513.084888]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  513.084901]  r4:b6005cc0
[  513.084913] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  513.084925] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  513.084930]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  513.084948] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  513.084959] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  513.084963]  r8:00000000 r7:b6091ddc r6:ffffffff r5:60000113 r4:80068df0 r3:b6091da8
[  513.084982] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  513.084991] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  513.084998] Exception stack(0xb6091da8 to 0xb6091df0)
[  513.085006] 1da0:                   b6806280 40000113 00000000 00000000 b47efe30 b6806280
[  513.085016] 1dc0: 0003405c b6806280 00000000 362d7000 80536080 b6091e24 b6091df8 b6091df8
[  513.085023] 1de0: 80068df0 80068df0 60000113 ffffffff
[  513.085034] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[  513.085039]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47efde8 r4:b47efc00
[  513.085057] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  513.085061]  r5:00000100 r4:b6806280
[  513.085074] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  513.085079]  r6:00000200 r5:00000000 r4:b6806280
[  513.085095] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  513.085099]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  513.085118] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  513.085122]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  513.085135]  r4:8052ffbc
[  513.085145] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  513.085150]  r4:8052ffbc r3:8005f374
[  513.085162] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  513.085167]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  513.085185] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  513.085194] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  513.085200] Exception stack(0xb6091f58 to 0xb6091fa0)
[  513.085206] 1f40:                                                       00000000 b6806250
[  513.085215] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  513.085225] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  513.085237] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  513.085248] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  513.085258] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  513.085263]  r7:80558348 r3:00000040
[  513.085274] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  513.085278]  r5:00000055 r4:3607806a
[  576.088604] INFO: rcu_sched self-detected stall on CPU
[  576.088621] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=90971 
[  576.088625] 	 (t=273016 jiffies g=962 c=961 q=123)
[  576.088633] Task dump for CPU 1:
[  576.088637] swapper/1       R running      0     0      1 0x00000002
[  576.088646] Backtrace: 
[  576.088669] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  576.088675]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  576.088694] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  576.088705] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  576.088710]  r5:8053b980 r4:00000001
[  576.088724] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  576.088729]  r4:00000002 r3:00000001
[  576.088741] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  576.088746]  r8:362d7000 r7:0000007b r6:8053b980 r5:80532e00 r4:b6809e00 r3:00043679
[  576.088767] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  576.088772]  r10:b6806b38 r9:00000086 r8:21854865 r7:00000086 r6:21854a06 r5:00000000
[  576.088785]  r4:b6062640
[  576.088796] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  576.088800]  r5:b6091dd8 r4:b6806c78
[  576.088814] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  576.088818]  r10:b6806b38 r9:00000086 r8:21854865 r7:b6806b00 r6:b6806b0c r5:b6806c78
[  576.088832]  r4:b6806ac0
[  576.088843] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  576.088848]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  576.088861]  r4:8052fac0
[  576.088873] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  576.088878]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  576.088891]  r4:b6005cc0
[  576.088903] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  576.088914] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  576.088919]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  576.088938] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  576.088948] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  576.088952]  r8:8052f280 r7:b6091e0c r6:ffffffff r5:60000113 r4:803e02e8 r3:b6091dd8
[  576.088971] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  576.088981] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  576.088987] Exception stack(0xb6091dd8 to 0xb6091e20)
[  576.088993] 1dc0:                                                       b47efc70 00000000
[  576.089003] 1de0: 0000a6ab 0000a6ab b47efc00 b47efde8 803e02cc 80536100 8052f280 362d7000
[  576.089012] 1e00: 80536080 b6091e3c b6091e28 b6091e28 803e02e8 803e02e8 60000113 ffffffff
[  576.089024] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  576.089028]  r5:00000100 r4:b6806280
[  576.089041] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  576.089046]  r6:00000200 r5:00000000 r4:b6806280
[  576.089061] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  576.089066]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  576.089085] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  576.089089]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  576.089103]  r4:8052ffbc
[  576.089113] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  576.089117]  r4:8052ffbc r3:8005f374
[  576.089129] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  576.089134]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  576.089153] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  576.089162] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  576.089167] Exception stack(0xb6091f58 to 0xb6091fa0)
[  576.089174] 1f40:                                                       00000000 b6806250
[  576.089183] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  576.089192] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  576.089204] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  576.089215] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  576.089225] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  576.089230]  r7:80558348 r3:00000040
[  576.089240] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  576.089245]  r5:00000055 r4:3607806a
[  639.092605] INFO: rcu_sched self-detected stall on CPU
[  639.092621] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=111931 
[  639.092626] 	 (t=336020 jiffies g=962 c=961 q=123)
[  639.092633] Task dump for CPU 1:
[  639.092637] swapper/1       R running      0     0      1 0x00000002
[  639.092646] Backtrace: 
[  639.092665] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  639.092670]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  639.092690] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  639.092702] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  639.092706]  r5:8053b980 r4:00000001
[  639.092721] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  639.092725]  r4:00000002 r3:00000001
[  639.092738] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  639.092743]  r8:362d7000 r7:0000007b r6:8053b980 r5:80532e00 r4:b6809e00 r3:00052c95
[  639.092764] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  639.092768]  r10:b6806b38 r9:00000094 r8:ccda0781 r7:00000094 r6:ccda08ee r5:00000000
[  639.092781]  r4:b6062640
[  639.092793] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  639.092798]  r5:b6091d98 r4:b6806c78
[  639.092811] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  639.092815]  r10:b6806b38 r9:00000094 r8:ccda0781 r7:b6806b00 r6:b6806b0c r5:b6806c78
[  639.092828]  r4:b6806ac0
[  639.092840] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  639.092845]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  639.092858]  r4:8052fac0
[  639.092869] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  639.092874]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  639.092887]  r4:b6005cc0
[  639.092900] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  639.092911] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  639.092915]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  639.092934] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  639.092945] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  639.092949]  r8:00000000 r7:b6091dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6091d98
[  639.092968] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  639.092977] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  639.092983] Exception stack(0xb6091d98 to 0xb6091de0)
[  639.092990] 1d80:                                                       b6806280 40000113
[  639.092999] 1da0: 00000000 00000000 b47efe30 b6806280 00052c94 b6806280 00000000 362d7000
[  639.093008] 1dc0: 80536080 b6091df4 b6091df8 b6091de8 80068df0 803f014c 60000113 ffffffff
[  639.093020] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[  639.093030] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[  639.093035]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47efde8 r4:b47efc00
[  639.093053] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  639.093058]  r5:00000100 r4:b6806280
[  639.093071] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  639.093075]  r6:00000200 r5:00000000 r4:b6806280
[  639.093091] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  639.093095]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  639.093114] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  639.093118]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  639.093131]  r4:8052ffbc
[  639.093142] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  639.093146]  r4:8052ffbc r3:8005f374
[  639.093159] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  639.093164]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  639.093183] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  639.093192] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  639.093197] Exception stack(0xb6091f58 to 0xb6091fa0)
[  639.093204] 1f40:                                                       00000000 b6806250
[  639.093213] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  639.093223] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  639.093234] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  639.093245] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  639.093256] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  639.093260]  r7:80558348 r3:00000040
[  639.093271] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  639.093275]  r5:00000055 r4:3607806a
[  702.096603] INFO: rcu_sched self-detected stall on CPU
[  702.096617] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=132929 
[  702.096621] 	 (t=399024 jiffies g=962 c=961 q=123)
[  702.096629] Task dump for CPU 1:
[  702.096633] swapper/1       R running      0     0      1 0x00000002
[  702.096641] Backtrace: 
[  702.096658] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  702.096663]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  702.096682] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  702.096694] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  702.096698]  r5:8053b980 r4:00000001
[  702.096712] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  702.096716]  r4:00000002 r3:00000001
[  702.096729] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  702.096733]  r8:362d7000 r7:0000007b r6:8053b980 r5:80532e00 r4:b6809e00 r3:000622b1
[  702.096753] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  702.096758]  r10:b6806b38 r9:000000a3 r8:782ec69d r7:000000a3 r6:782ec80a r5:00000000
[  702.096771]  r4:b6062640
[  702.096782] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  702.096787]  r5:b6091dd8 r4:b6806c78
[  702.096800] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  702.096804]  r10:b6806b38 r9:000000a3 r8:782ec69d r7:b6806b00 r6:b6806b0c r5:b6806c78
[  702.096817]  r4:b6806ac0
[  702.096829] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  702.096834]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  702.096847]  r4:8052fac0
[  702.096858] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  702.096863]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  702.096876]  r4:b6005cc0
[  702.096887] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  702.096898] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  702.096903]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  702.096922] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  702.096933] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  702.096937]  r8:8052f280 r7:b6091e0c r6:ffffffff r5:60000113 r4:803e0370 r3:b6091dd8
[  702.096956] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  702.096965] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  702.096971] Exception stack(0xb6091dd8 to 0xb6091e20)
[  702.096978] 1dc0:                                                       00000000 40000113
[  702.096987] 1de0: 00000000 000056e6 b47efc00 b47efde8 803e02cc 80536100 8052f280 362d7000
[  702.096996] 1e00: 80536080 b6091e3c b6091df8 b6091e28 80068df0 803e0370 60000113 ffffffff
[  702.097007] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  702.097012]  r5:00000100 r4:b6806280
[  702.097026] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  702.097030]  r6:00000200 r5:00000000 r4:b6806280
[  702.097045] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  702.097050]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  702.097068] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  702.097072]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  702.097085]  r4:8052ffbc
[  702.097095] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  702.097100]  r4:8052ffbc r3:8005f374
[  702.097112] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  702.097117]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  702.097136] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  702.097145] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  702.097150] Exception stack(0xb6091f58 to 0xb6091fa0)
[  702.097157] 1f40:                                                       00000000 b6806250
[  702.097166] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  702.097175] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  702.097186] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  702.097196] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  702.097207] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  702.097211]  r7:80558348 r3:00000040
[  702.097222] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  702.097226]  r5:00000055 r4:3607806a
[  765.100607] INFO: rcu_sched self-detected stall on CPU
[  765.100626] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=153760 
[  765.100631] 	 (t=462028 jiffies g=962 c=961 q=1471)
[  765.100639] Task dump for CPU 1:
[  765.100643] swapper/1       R running      0     0      1 0x00000002
[  765.100652] Backtrace: 
[  765.100671] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  765.100676]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  765.100694] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  765.100705] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  765.100710]  r5:8053b980 r4:00000001
[  765.100724] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  765.100728]  r4:00000002 r3:00000001
[  765.100741] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  765.100745]  r8:362d7000 r7:000005bf r6:8053b980 r5:80532e00 r4:b6809e00 r3:000718cd
[  765.100766] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  765.100771]  r10:b6806b38 r9:000000b2 r8:238385b9 r7:000000b2 r6:23838726 r5:00000000
[  765.100784]  r4:b6062640
[  765.100795] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  765.100800]  r5:b6091d98 r4:b6806c78
[  765.100813] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  765.100817]  r10:b6806b38 r9:000000b2 r8:238385b9 r7:b6806b00 r6:b6806b0c r5:b6806c78
[  765.100831]  r4:b6806ac0
[  765.100843] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  765.100848]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  765.100861]  r4:8052fac0
[  765.100873] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  765.100877]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  765.100890]  r4:b6005cc0
[  765.100902] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  765.100913] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  765.100918]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  765.100937] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  765.100948] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  765.100952]  r8:00000000 r7:b6091dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6091d98
[  765.100971] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  765.100980] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  765.100986] Exception stack(0xb6091d98 to 0xb6091de0)
[  765.100993] 1d80:                                                       b6806280 40000113
[  765.101002] 1da0: 00000000 00000000 b47efe30 b6806280 000718cc b6806280 00000000 362d7000
[  765.101012] 1dc0: 80536080 b6091df4 b6091df8 b6091de8 80068df0 803f014c 60000113 ffffffff
[  765.101023] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[  765.101034] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[  765.101039]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47efde8 r4:b47efc00
[  765.101056] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  765.101061]  r5:00000100 r4:b6806280
[  765.101074] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  765.101079]  r6:00000200 r5:00000000 r4:b6806280
[  765.101094] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  765.101099]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  765.101117] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  765.101122]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  765.101135]  r4:8052ffbc
[  765.101145] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  765.101150]  r4:8052ffbc r3:8005f374
[  765.101162] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  765.101167]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  765.101185] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  765.101194] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  765.101199] Exception stack(0xb6091f58 to 0xb6091fa0)
[  765.101205] 1f40:                                                       00000000 b6806250
[  765.101215] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  765.101224] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  765.101235] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  765.101246] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  765.101256] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  765.101261]  r7:80558348 r3:00000040
[  765.101271] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  765.101276]  r5:00000055 r4:3607806a
[  828.104612] INFO: rcu_sched self-detected stall on CPU
[  828.104638] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=174456 
[  828.104644] 	 (t=525032 jiffies g=962 c=961 q=1586)
[  828.104652] Task dump for CPU 1:
[  828.104658] swapper/1       R running      0     0      1 0x00000002
[  828.104666] Backtrace: 
[  828.104686] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  828.104691]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  828.104711] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  828.104722] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  828.104727]  r5:8053b980 r4:00000001
[  828.104742] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  828.104746]  r4:00000002 r3:00000001
[  828.104760] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  828.104764]  r8:362d7000 r7:00000632 r6:8053b980 r5:80532e00 r4:b6809e00 r3:00080ee9
[  828.104785] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  828.104789]  r10:b6806b38 r9:000000c0 r8:ced8453d r7:000000c0 r6:ced846aa r5:00000000
[  828.104803]  r4:b6062640
[  828.104814] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  828.104818]  r5:b6091d98 r4:b6806c78
[  828.104831] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  828.104836]  r10:b6806b38 r9:000000c0 r8:ced8453d r7:b6806b00 r6:b6806b0c r5:b6806c78
[  828.104849]  r4:b6806ac0
[  828.104860] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  828.104865]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  828.104878]  r4:8052fac0
[  828.104891] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  828.104896]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  828.104909]  r4:b6005cc0
[  828.104923] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  828.104934] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  828.104939]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  828.104958] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  828.104968] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  828.104973]  r8:00000000 r7:b6091dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6091d98
[  828.104992] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  828.105001] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  828.105006] Exception stack(0xb6091d98 to 0xb6091de0)
[  828.105013] 1d80:                                                       b6806280 40000113
[  828.105022] 1da0: 00000000 00000000 b47efe30 b6806280 00080ee8 b6806280 00000000 362d7000
[  828.105032] 1dc0: 80536080 b6091df4 b6091df8 b6091de8 80068df0 803f014c 60000113 ffffffff
[  828.105043] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[  828.105054] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[  828.105058]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47efde8 r4:b47efc00
[  828.105076] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  828.105081]  r5:00000100 r4:b6806280
[  828.105094] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  828.105099]  r6:00000200 r5:00000000 r4:b6806280
[  828.105115] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  828.105120]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  828.105138] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  828.105143]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  828.105156]  r4:8052ffbc
[  828.105167] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  828.105171]  r4:8052ffbc r3:8005f374
[  828.105183] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  828.105188]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  828.105207] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  828.105215] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  828.105221] Exception stack(0xb6091f58 to 0xb6091fa0)
[  828.105227] 1f40:                                                       00000000 b6806250
[  828.105236] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  828.105245] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  828.105256] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  828.105267] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  828.105279] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  828.105283]  r7:80558348 r3:00000040
[  828.105296] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  828.105300]  r5:00000055 r4:3607806a
[  857.395786] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[  857.402926] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[  891.108607] INFO: rcu_sched self-detected stall on CPU
[  891.108627] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=195088 
[  891.108632] 	 (t=588036 jiffies g=962 c=961 q=2871)
[  891.108640] Task dump for CPU 1:
[  891.108644] swapper/1       R running      0     0      1 0x00000002
[  891.108652] Backtrace: 
[  891.108671] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  891.108676]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  891.108694] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  891.108705] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  891.108710]  r5:8053b980 r4:00000001
[  891.108724] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  891.108728]  r4:00000002 r3:00000001
[  891.108741] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  891.108746]  r8:362d7000 r7:00000b37 r6:8053b980 r5:80532e00 r4:b6809e00 r3:00090505
[  891.108766] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  891.108771]  r10:b6806b38 r9:000000cf r8:7a2d0389 r7:000000cf r6:7a2d052a r5:00000000
[  891.108784]  r4:b6062640
[  891.108795] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  891.108800]  r5:b6091dd8 r4:b6806c78
[  891.108812] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  891.108817]  r10:b6806b38 r9:000000cf r8:7a2d0389 r7:b6806b00 r6:b6806b0c r5:b6806c78
[  891.108830]  r4:b6806ac0
[  891.108842] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  891.108846]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  891.108859]  r4:8052fac0
[  891.108871] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  891.108876]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  891.108889]  r4:b6005cc0
[  891.108901] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  891.108912] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  891.108917]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  891.108936] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  891.108946] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  891.108950]  r8:8052f280 r7:b6091e0c r6:ffffffff r5:60000113 r4:803e0354 r3:b6091dd8
[  891.108970] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  891.108979] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  891.108984] Exception stack(0xb6091dd8 to 0xb6091e20)
[  891.108991] 1dc0:                                                       00000000 40000113
[  891.109000] 1de0: 00000000 00000000 b47efc00 b47efde8 803e02cc 80536100 8052f280 362d7000
[  891.109009] 1e00: 80536080 b6091e3c b6091df8 b6091e28 80068df0 803e0354 60000113 ffffffff
[  891.109020] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  891.109025]  r5:00000100 r4:b6806280
[  891.109038] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  891.109042]  r6:00000200 r5:00000000 r4:b6806280
[  891.109057] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  891.109062]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  891.109080] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  891.109085]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  891.109098]  r4:8052ffbc
[  891.109108] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  891.109112]  r4:8052ffbc r3:8005f374
[  891.109125] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  891.109129]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  891.109148] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  891.109157] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  891.109162] Exception stack(0xb6091f58 to 0xb6091fa0)
[  891.109168] 1f40:                                                       00000000 b6806250
[  891.109177] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  891.109187] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  891.109198] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  891.109209] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  891.109219] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  891.109223]  r7:80558348 r3:00000040
[  891.109234] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  891.109239]  r5:00000055 r4:3607806a
[  954.112607] INFO: rcu_sched self-detected stall on CPU
[  954.112627] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=215448 
[  954.112632] 	 (t=651040 jiffies g=962 c=961 q=2966)
[  954.112640] Task dump for CPU 1:
[  954.112644] swapper/1       R running      0     0      1 0x00000002
[  954.112652] Backtrace: 
[  954.112673] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  954.112677]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[  954.112697] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  954.112709] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  954.112713]  r5:8053b980 r4:00000001
[  954.112727] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  954.112732]  r4:00000002 r3:00000001
[  954.112744] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  954.112749]  r8:362d7000 r7:00000b96 r6:8053b980 r5:80532e00 r4:b6809e00 r3:0009fb21
[  954.112769] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  954.112774]  r10:b6806b38 r9:000000de r8:2581c271 r7:000000de r6:2581c412 r5:00000000
[  954.112787]  r4:b6062640
[  954.112799] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  954.112803]  r5:b6091dd8 r4:b6806c78
[  954.112816] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  954.112821]  r10:b6806b38 r9:000000de r8:2581c271 r7:b6806b00 r6:b6806b0c r5:b6806c78
[  954.112834]  r4:b6806ac0
[  954.112846] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  954.112850]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[  954.112863]  r4:8052fac0
[  954.112875] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  954.112880]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[  954.112893]  r4:b6005cc0
[  954.112905] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  954.112916] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  954.112920]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[  954.112939] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  954.112949] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  954.112954]  r8:8052f280 r7:b6091e0c r6:ffffffff r5:60000113 r4:803e032c r3:b6091dd8
[  954.112973] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  954.112982] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  954.112988] Exception stack(0xb6091dd8 to 0xb6091e20)
[  954.112994] 1dc0:                                                       00000000 b1280000
[  954.113003] 1de0: 00000000 00000000 b47efc00 b47efde8 803e02cc 80536100 8052f280 362d7000
[  954.113013] 1e00: 80536080 b6091e3c b6091e28 b6091e28 803e02e8 803e032c 60000113 ffffffff
[  954.113024] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  954.113028]  r5:00000100 r4:b6806280
[  954.113042] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  954.113046]  r6:00000200 r5:00000000 r4:b6806280
[  954.113061] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  954.113066]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[  954.113084] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  954.113089]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  954.113102]  r4:8052ffbc
[  954.113112] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  954.113117]  r4:8052ffbc r3:8005f374
[  954.113129] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  954.113134]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[  954.113153] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  954.113162] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  954.113167] Exception stack(0xb6091f58 to 0xb6091fa0)
[  954.113173] 1f40:                                                       00000000 b6806250
[  954.113183] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[  954.113192] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[  954.113203] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  954.113214] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  954.113224] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  954.113229]  r7:80558348 r3:00000040
[  954.113240] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  954.113245]  r5:00000055 r4:3607806a
[ 1017.116605] INFO: rcu_sched self-detected stall on CPU
[ 1017.116622] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=236065 
[ 1017.116626] 	 (t=714044 jiffies g=962 c=961 q=2966)
[ 1017.116634] Task dump for CPU 1:
[ 1017.116638] swapper/1       R running      0     0      1 0x00000002
[ 1017.116647] Backtrace: 
[ 1017.116666] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 1017.116671]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[ 1017.116693] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 1017.116705] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 1017.116710]  r5:8053b980 r4:00000001
[ 1017.116724] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 1017.116729]  r4:00000002 r3:00000001
[ 1017.116741] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 1017.116746]  r8:362d7000 r7:00000b96 r6:8053b980 r5:80532e00 r4:b6809e00 r3:000af13d
[ 1017.116767] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 1017.116772]  r10:b6806b38 r9:000000ec r8:d0d681c1 r7:000000ec r6:d0d6832e r5:00000000
[ 1017.116785]  r4:b6062640
[ 1017.116797] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 1017.116802]  r5:b6091e18 r4:b6806c78
[ 1017.116815] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 1017.116819]  r10:b6806b38 r9:000000ec r8:d0d681c1 r7:b6806b00 r6:b6806b0c r5:b6806c78
[ 1017.116833]  r4:b6806ac0
[ 1017.116844] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 1017.116850]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[ 1017.116863]  r4:8052fac0
[ 1017.116875] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 1017.116879]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[ 1017.116892]  r4:b6005cc0
[ 1017.116904] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 1017.116915] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 1017.116920]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 1017.116939] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 1017.116949] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 1017.116954]  r8:8052f280 r7:b6091e4c r6:ffffffff r5:60000113 r4:80068ab8 r3:b6091e18
[ 1017.116972] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 1017.116981] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 1017.116987] Exception stack(0xb6091e18 to 0xb6091e60)
[ 1017.116994] 1e00:                                                       b47efe30 803e02cc
[ 1017.117003] 1e20: b47efc00 00000e4d b6806280 00000000 00000200 80536100 8052f280 362d7000
[ 1017.117013] 1e40: 80536080 b6091e94 00000000 b6091e68 b6091e6c 80068ab8 60000113 ffffffff
[ 1017.117023] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 1017.117028]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[ 1017.117047] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 1017.117052]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 1017.117065]  r4:8052ffbc
[ 1017.117075] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 1017.117080]  r4:8052ffbc r3:8005f374
[ 1017.117092] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 1017.117096]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[ 1017.117116] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 1017.117125] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 1017.117131] Exception stack(0xb6091f58 to 0xb6091fa0)
[ 1017.117137] 1f40:                                                       00000000 b6806250
[ 1017.117147] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[ 1017.117156] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[ 1017.117167] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 1017.117178] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 1017.117189] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 1017.117193]  r7:80558348 r3:00000040
[ 1017.117205] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 1017.117209]  r5:00000055 r4:3607806a
[ 1080.120604] INFO: rcu_sched self-detected stall on CPU
[ 1080.120618] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=256712 
[ 1080.120622] 	 (t=777048 jiffies g=962 c=961 q=2966)
[ 1080.120630] Task dump for CPU 1:
[ 1080.120634] swapper/1       R running      0     0      1 0x00000002
[ 1080.120643] Backtrace: 
[ 1080.120660] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 1080.120665]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[ 1080.120684] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 1080.120696] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 1080.120700]  r5:8053b980 r4:00000001
[ 1080.120714] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 1080.120718]  r4:00000002 r3:00000001
[ 1080.120731] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 1080.120736]  r8:362d7000 r7:00000b96 r6:8053b980 r5:80532e00 r4:b6809e00 r3:000be759
[ 1080.120756] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 1080.120761]  r10:b6806b38 r9:000000fb r8:7c2b40a9 r7:000000fb r6:7c2b4216 r5:00000000
[ 1080.120774]  r4:b6062640
[ 1080.120785] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 1080.120789]  r5:b6091dd8 r4:b6806c78
[ 1080.120802] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 1080.120807]  r10:b6806b38 r9:000000fb r8:7c2b40a9 r7:b6806b00 r6:b6806b0c r5:b6806c78
[ 1080.120820]  r4:b6806ac0
[ 1080.120831] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 1080.120836]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[ 1080.120849]  r4:8052fac0
[ 1080.120861] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 1080.120866]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[ 1080.120879]  r4:b6005cc0
[ 1080.120890] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 1080.120901] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 1080.120906]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 1080.120925] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 1080.120935] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 1080.120940]  r8:8052f280 r7:b6091e0c r6:ffffffff r5:60000113 r4:803e0370 r3:b6091dd8
[ 1080.120959] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 1080.120968] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 1080.120974] Exception stack(0xb6091dd8 to 0xb6091e20)
[ 1080.120980] 1dc0:                                                       00000000 40000113
[ 1080.120990] 1de0: 00000000 00003c26 b47efc00 b47efde8 803e02cc 80536100 8052f280 362d7000
[ 1080.120999] 1e00: 80536080 b6091e3c b6091df8 b6091e28 80068df0 803e0370 60000113 ffffffff
[ 1080.121010] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 1080.121015]  r5:00000100 r4:b6806280
[ 1080.121028] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 1080.121033]  r6:00000200 r5:00000000 r4:b6806280
[ 1080.121048] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 1080.121053]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[ 1080.121072] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 1080.121076]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 1080.121089]  r4:8052ffbc
[ 1080.121100] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 1080.121104]  r4:8052ffbc r3:8005f374
[ 1080.121116] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 1080.121121]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[ 1080.121140] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 1080.121148] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 1080.121154] Exception stack(0xb6091f58 to 0xb6091fa0)
[ 1080.121161] 1f40:                                                       00000000 b6806250
[ 1080.121170] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[ 1080.121179] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[ 1080.121190] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 1080.121201] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 1080.121211] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 1080.121216]  r7:80558348 r3:00000040
[ 1080.121226] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 1080.121231]  r5:00000055 r4:3607806a
[ 1143.124604] INFO: rcu_sched self-detected stall on CPU
[ 1143.124619] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=277394 
[ 1143.124623] 	 (t=840052 jiffies g=962 c=961 q=2966)
[ 1143.124630] Task dump for CPU 1:
[ 1143.124634] swapper/1       R running      0     0      1 0x00000002
[ 1143.124643] Backtrace: 
[ 1143.124661] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 1143.124667]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[ 1143.124685] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 1143.124696] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 1143.124702]  r5:8053b980 r4:00000001
[ 1143.124716] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 1143.124721]  r4:00000002 r3:00000001
[ 1143.124734] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 1143.124739]  r8:362d7000 r7:00000b96 r6:8053b980 r5:80532e00 r4:b6809e00 r3:000cdd75
[ 1143.124759] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 1143.124764]  r10:b6806b38 r9:0000010a r8:277fffc5 r7:0000010a r6:278000fe r5:00000000
[ 1143.124778]  r4:b6062640
[ 1143.124789] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 1143.124793]  r5:b6091dc8 r4:b6806c78
[ 1143.124806] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 1143.124811]  r10:b6806b38 r9:0000010a r8:277fffc5 r7:b6806b00 r6:b6806b0c r5:b6806c78
[ 1143.124824]  r4:b6806ac0
[ 1143.124836] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 1143.124840]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[ 1143.124854]  r4:8052fac0
[ 1143.124865] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 1143.124870]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[ 1143.124883]  r4:b6005cc0
[ 1143.124896] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 1143.124907] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 1143.124912]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 1143.124931] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 1143.124941] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 1143.124946]  r8:8052f280 r7:b6091dfc r6:ffffffff r5:60000113 r4:803f003c r3:b6091dc8
[ 1143.124964] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 1143.124974] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 1143.124979] Exception stack(0xb6091dc8 to 0xb6091e10)
[ 1143.124988] 1dc0:                   b47efc70 00000000 ba6eba6d ba6dba6d b47efc00 b47efde8
[ 1143.124998] 1de0: 803e02cc 80536100 8052f280 362d7000 80536080 b6091e24 b6091e28 b6091e18
[ 1143.125005] 1e00: 803e02e8 803f003c 60000113 ffffffff
[ 1143.125016] [<803f0020>] (_raw_spin_lock) from [<803e02e8>] (prb_retire_rx_blk_timer_expired+0x1c/0x114)
[ 1143.125026] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 1143.125031]  r5:00000100 r4:b6806280
[ 1143.125044] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 1143.125048]  r6:00000200 r5:00000000 r4:b6806280
[ 1143.125064] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 1143.125068]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[ 1143.125087] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 1143.125091]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 1143.125104]  r4:8052ffbc
[ 1143.125114] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 1143.125119]  r4:8052ffbc r3:8005f374
[ 1143.125131] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 1143.125136]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[ 1143.125154] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 1143.125163] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 1143.125169] Exception stack(0xb6091f58 to 0xb6091fa0)
[ 1143.125176] 1f40:                                                       00000000 b6806250
[ 1143.125185] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[ 1143.125194] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[ 1143.125205] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 1143.125216] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 1143.125227] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 1143.125231]  r7:80558348 r3:00000040
[ 1143.125242] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 1143.125246]  r5:00000055 r4:3607806a
[ 1198.572896] systemd[1]: Started Turns off Raspberry Pi display backlight on shutdown/reboot.
[ 1198.573042] systemd[1]: Starting Synchronise Hardware Clock to System Clock...
[ 1206.127609] INFO: rcu_sched self-detected stall on CPU
[ 1206.127630] 	1-...: (1 GPs behind) idle=7db/2/0 softirq=4884/4886 fqs=298028 
[ 1206.127636] 	 (t=903055 jiffies g=962 c=961 q=3834)
[ 1206.127644] Task dump for CPU 1:
[ 1206.127649] swapper/1       R running      0     0      1 0x00000002
[ 1206.127657] Backtrace: 
[ 1206.127676] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[ 1206.127681]  r6:80000193 r5:8053b980 r4:b6062640 r3:00000000
[ 1206.127700] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[ 1206.127712] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[ 1206.127716]  r5:8053b980 r4:00000001
[ 1206.127730] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 1206.127734]  r4:00000002 r3:00000001
[ 1206.127747] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[ 1206.127752]  r8:362d7000 r7:00000efa r6:8053b980 r5:80532e00 r4:b6809e00 r3:000dd390
[ 1206.127772] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[ 1206.127776]  r10:b6806b38 r9:00000118 r8:d2c57c39 r7:00000118 r6:d2c57da6 r5:00000000
[ 1206.127789]  r4:b6062640
[ 1206.127801] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[ 1206.127806]  r5:b6091d98 r4:b6806c78
[ 1206.127819] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[ 1206.127823]  r10:b6806b38 r9:00000118 r8:d2c57c39 r7:b6806b00 r6:b6806b0c r5:b6806c78
[ 1206.127836]  r4:b6806ac0
[ 1206.127848] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[ 1206.127853]  r10:b6806b38 r9:b6806b58 r8:b6806b78 r7:b6806ac0 r6:00000003 r5:b6806ad4
[ 1206.127866]  r4:8052fac0
[ 1206.127877] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[ 1206.127882]  r10:80536080 r9:362d7000 r8:00000011 r7:b6004d40 r6:805427bc r5:b680c580
[ 1206.127895]  r4:b6005cc0
[ 1206.127908] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[ 1206.127920] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[ 1206.127924]  r8:00000000 r7:00000000 r6:b6091f58 r5:00000000 r4:8052ffbc r3:8005f374
[ 1206.127943] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[ 1206.127953] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 1206.127958]  r8:00000000 r7:b6091dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6091d98
[ 1206.127977] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 1206.127985] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 1206.127991] Exception stack(0xb6091d98 to 0xb6091de0)
[ 1206.127998] 1d80:                                                       b6806280 40000113
[ 1206.128007] 1da0: 00000000 00000000 b47efe30 b6806280 000dd38f b6806280 00000000 362d7000
[ 1206.128016] 1dc0: 80536080 b6091df4 b6091df8 b6091de8 80068df0 803f014c 60000113 ffffffff
[ 1206.128028] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[ 1206.128038] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[ 1206.128043]  r8:8052f280 r7:80536100 r6:803e02cc r5:b47efde8 r4:b47efc00
[ 1206.128061] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[ 1206.128066]  r5:00000100 r4:b6806280
[ 1206.128079] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[ 1206.128084]  r6:00000200 r5:00000000 r4:b6806280
[ 1206.128099] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[ 1206.128103]  r9:40000001 r8:00000100 r7:b6090000 r6:80536084 r5:00000001 r4:00000000
[ 1206.128122] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[ 1206.128126]  r10:b6090000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[ 1206.128139]  r4:8052ffbc
[ 1206.128149] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[ 1206.128154]  r4:8052ffbc r3:8005f374
[ 1206.128166] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[ 1206.128171]  r8:80554b44 r7:b6091f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6091f58
[ 1206.128189] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[ 1206.128198] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[ 1206.128204] Exception stack(0xb6091f58 to 0xb6091fa0)
[ 1206.128210] 1f40:                                                       00000000 b6806250
[ 1206.128219] 1f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6091fc8 80554b44 80536430
[ 1206.128228] 1f80: b6090000 b6091fb4 b6091fb8 b6091fa8 800100e0 800100e4 60000013 ffffffff
[ 1206.128239] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[ 1206.128250] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[ 1206.128260] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[ 1206.128265]  r7:80558348 r3:00000040
[ 1206.128276] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[ 1206.128280]  r5:00000055 r4:3607806a

@rodizio1
Copy link
Author

just happened again.

it's always "swapper"

top again shows 25% si:

Tasks: 149 total,   3 running, 145 sleeping,   0 stopped,   1 zombie
%Cpu(s):  2.6 us,  2.9 sy,  0.1 ni, 68.7 id,  0.1 wa,  0.0 hi, 25.6 si,  0.0 st
KiB Mem:    886476 total,   805792 used,    80684 free,    10776 buffers
KiB Swap:        0 total,        0 used,        0 free.   665428 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                     
  717 root      10 -10   55296   2384   1556 S   9.9  0.3   1:51.96 hello_video.bin                                                                                                             
  737 root      10 -10    4736   3704   3440 S   3.3  0.4   0:32.24 rx                                                                                                                          
  282 root      20   0       0      0      0 S   2.0  0.0   0:10.71 kworker/u8:3                                                                                                                
  908 root      20   0   38776   2428   2084 S   2.0  0.3   0:25.33 osd                                                                                                                         
   62 root      20   0       0      0      0 S   1.7  0.0   0:13.75 kworker/u8:1                                                                                                                
  738 root      10 -10    3400   1500   1420 R   1.0  0.2   0:11.39 tee                                                                                                                         
  739 root      10 -10    1676    348    292 S   0.7  0.0   0:06.31 ftee                                                                                                                        
  745 root      30  10    1676    344    288 S   0.7  0.0   0:03.62 ftee                                                                                                                        
 6865 root      20   0    5224   2576   2144 R   0.7  0.3   0:00.04 top                                                                                                                         
    7 root      20   0       0      0      0 S   0.3  0.0   0:02.56 rcu_sched                                                                                                                   
   56 root       1 -19       0      0      0 S   0.3  0.0   0:05.56 VCHIQ-0                                                                                                                     
  365 root      20   0    7092   5088   2852 S   0.3  0.6   0:02.20 bash                                                                                                                        
  716 root      10 -10    3552    360    300 S   0.3  0.0   0:04.30 cat                                                                                                                         
  741 root      10 -10    1676    328    272 S   0.3  0.0   0:03.42 ftee                                                                                                                        
    1 root      20   0   13588   3684   2732 D   0.0  0.4   0:01.90 systemd                                                                                                                     
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd                                                                                                                    
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.16 ksoftirqd/0                                                                                                                 
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                                                                                
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                                                                                      
    9 root      rt   0       0      0      0 S   0.0  0.0   0:00.04 migration/0                                                                                                                 
   10 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 migration/1                                                                                                                 
   11 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/1                                                                                                                 
   13 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H                                                                                                                
   14 root      rt   0       0      0      0 S   0.0  0.0   0:00.04 migration/2                                                                                                                 
   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/2                                                                                                                 
   17 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H                                                                                                                
   18 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 migration/3                                                                                                                 
   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/3                                                                                                                 
   21 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H                                                                                                                
   22 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs                                                                                                                   
   23 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback                                                                                                                   
   24 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto                                                                                                                      
   25 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   26 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd                                                                                                                     
   27 root      20   0       0      0      0 S   0.0  0.0   0:00.02 kworker/0:1                                                                                                                 
   28 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd0                                                                                                                     
   29 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 vmstat                                                                                                                      
   30 root      20   0       0      0      0 S   0.0  0.0   0:00.00 fsnotify_mark                                                                                                               
   39 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kthrotld                                                                                                                    
   40 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   41 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   42 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   43 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                      
   44 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset                                                                                                                    
[  354.990652] Task dump for CPU 3:
[  354.990657] swapper/3       R running      0     0      1 0x00000002
[  354.990668] Backtrace: 
[  354.990696] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  354.990701]  r6:80000193 r5:8053b980 r4:b6062ec0 r3:00000000
[  354.990723] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  354.990735] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  354.990739]  r5:8053b980 r4:00000003
[  354.990757] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  354.990761]  r4:00000004 r3:00000001
[  354.990775] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  354.990779]  r8:362ef000 r7:00000908 r6:8053b980 r5:80532e00 r4:b6821e00 r3:0000d6cf
[  354.990802] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  354.990807]  r10:b681eb38 r9:00000052 r8:a70d8a38 r7:00000052 r6:a70d8ba5 r5:00000000
[  354.990820]  r4:b6062ec0
[  354.990833] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  354.990838]  r5:b6095d98 r4:b681ec78
[  354.990851] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  354.990856]  r10:b681eb38 r9:00000052 r8:a70d8a38 r7:b681eb00 r6:b681eb0c r5:b681ec78
[  354.990869]  r4:b681eac0
[  354.990881] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  354.990885]  r10:b681eb38 r9:b681eb58 r8:b681eb78 r7:b681eac0 r6:00000003 r5:b681ead4
[  354.990899]  r4:8052fac0
[  354.990911] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  354.990916]  r10:80536080 r9:362ef000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6824580
[  354.990929]  r4:b6005cc0
[  354.990941] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  354.990952] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  354.990957]  r8:00000000 r7:00000000 r6:b6095f58 r5:00000000 r4:8052ffbc r3:8005f374
[  354.990976] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  354.990987] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  354.990992]  r8:00000000 r7:b6095dcc r6:ffffffff r5:60000113 r4:803f014c r3:b6095d98
[  354.991011] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  354.991021] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  354.991027] Exception stack(0xb6095d98 to 0xb6095de0)
[  354.991034] 5d80:                                                       b681e280 40000113
[  354.991043] 5da0: 00000000 00000000 b284f230 b681e280 0000d6ce b681e280 00000000 362ef000
[  354.991053] 5dc0: 80536080 b6095df4 b6095df8 b6095de8 80068df0 803f014c 60000113 ffffffff
[  354.991065] [<803f0114>] (_raw_spin_unlock_irqrestore) from [<80068df0>] (mod_timer+0x108/0x158)
[  354.991076] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[  354.991081]  r8:8052f280 r7:80536100 r6:803e02cc r5:b284f1e8 r4:b284f000
[  354.991099] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  354.991103]  r5:00000100 r4:b681e280
[  354.991117] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  354.991121]  r6:00000200 r5:00000000 r4:b681e280
[  354.991138] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  354.991143]  r9:40000001 r8:00000100 r7:b6094000 r6:80536084 r5:00000001 r4:00000080
[  354.991162] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  354.991167]  r10:b6094000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  354.991180]  r4:8052ffbc
[  354.991191] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  354.991195]  r4:8052ffbc r3:8005f374
[  354.991208] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  354.991212]  r8:80554b44 r7:b6095f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6095f58
[  354.991231] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  354.991240] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  354.991245] Exception stack(0xb6095f58 to 0xb6095fa0)
[  354.991252] 5f40:                                                       00000000 b681e250
[  354.991261] 5f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6095fc8 80554b44 80536430
[  354.991271] 5f80: b6094000 b6095fb4 b6095fb8 b6095fa8 800100e0 800100e4 60000013 ffffffff
[  354.991284] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  354.991295] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  354.991306] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  354.991310]  r7:80558348 r3:00000040
[  354.991322] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  354.991326]  r5:00000055 r4:3607806a
[  417.993612] INFO: rcu_sched self-detected stall on CPU
[  417.993636] 	3-...: (1 GPs behind) idle=915/2/0 softirq=19991/19992 fqs=27707 
[  417.993642] 	 (t=84003 jiffies g=8460 c=8459 q=9185)
[  417.993653] Task dump for CPU 3:
[  417.993657] swapper/3       R running      0     0      1 0x00000002

@rodizio1
Copy link
Author

rodizio1 commented Jan 19, 2017

This time it crashed right while plugging in a USB keyboard:

[  651.216614] INFO: rcu_sched self-detected stall on CPU
[  651.216646]  3-...: (1 GPs behind) idle=379/2/0 softirq=34630/34632 fqs=6988
[  651.216653]   (t=21001 jiffies g=10792 c=10791 q=2177)
[  651.216663] Task dump for CPU 3:
[  651.216668] swapper/3       R running      0     0      1 0x00000002
[  651.216678] Backtrace:
[  651.216707] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  651.216713]  r6:80000193 r5:8053b980 r4:b6062ec0 r3:00000000
[  651.216736] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  651.216748] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  651.216753]  r5:8053b980 r4:00000003
[  651.216771] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  651.216776]  r4:00000004 r3:00000001
[  651.216789] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  651.216794]  r8:362ef000 r7:00000881 r6:8053b980 r5:80532e00 r4:b6821e00 r3:00055bf1
[  651.216817] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  651.216821]  r10:b681eb38 r9:00000097 r8:9f7f9648 r7:00000097 r6:9f7f97b5 r5:00000000
[  651.216835]  r4:b6062ec0
[  651.216848] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  651.216853]  r5:b6095d80 r4:b681ec78
[  651.216865] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  651.216870]  r10:b681eb38 r9:00000097 r8:9f7f9648 r7:b681eb00 r6:b681eb0c r5:b681ec78
[  651.216883]  r4:b681eac0
[  651.216895] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  651.216900]  r10:b681eb38 r9:b681eb58 r8:b681eb78 r7:b681eac0 r6:00000003 r5:b681ead4
[  651.216913]  r4:8052fac0
[  651.216926] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  651.216930]  r10:80536080 r9:b6095dfc r8:00000011 r7:b6004d40 r6:805427bc r5:b6824580
[  651.216944]  r4:b6005cc0
[  651.216960] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  651.216973] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  651.216977]  r8:00000000 r7:00000000 r6:b6095f58 r5:00000000 r4:8052ffbc r3:8005f374
[  651.216996] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  651.217007] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  651.217011]  r8:8053653c r7:b6095db4 r6:ffffffff r5:40000113 r4:803f0070 r3:b6095d80
[  651.217031] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  651.217041] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  651.217047] Exception stack(0xb6095d80 to 0xb6095dc8)
[  651.217056] 5d80: b681e280 b6095dfc 00000000 00000003 00000003 b681e280 b2c6e230 8052f280
[  651.217066] 5da0: 8053653c b6095dfc 80536080 b6095df4 b6095dd0 b6095dd0 80068784 803f0070
[  651.217072] 5dc0: 40000113 ffffffff
[  651.217083] [<80068740>] (lock_timer_base) from [<80068d48>] (mod_timer+0x60/0x158)
[  651.217087]  r9:362ef000 r8:8052f280 r7:80536100 r6:00055bf0 r5:b2c6e1e8 r4:b2c6e230
[  651.217110] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[  651.217115]  r8:8052f280 r7:80536100 r6:803e02cc r5:b2c6e1e8 r4:b2c6e000
[  651.217133] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  651.217137]  r5:00000100 r4:b681e280
[  651.217150] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  651.217155]  r6:00000200 r5:00000000 r4:b681e280
[  651.217176] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  651.217181]  r9:40000001 r8:00000100 r7:b6094000 r6:80536084 r5:00000001 r4:00000000
[  651.217199] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  651.217204]  r10:b6094000 r9:80536430 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  651.217219]  r4:8052ffbc
[  651.217230] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  651.217235]  r4:8052ffbc r3:8005f374
[  651.217247] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  651.217251]  r8:80554b44 r7:b6095f8c r6:ffffffff r5:60000013 r4:800100e4 r3:b6095f58
[  651.217271] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  651.217280] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  651.217285] Exception stack(0xb6095f58 to 0xb6095fa0)
[  651.217292] 5f40:                                                       00000000 b681e250
[  651.217301] 5f60: 60000093 80019940 805363e8 8052f224 803f16d8 b6095fc8 80554b44 80536430
[  651.217312] 5f80: b6094000 b6095fb4 b6095fb8 b6095fa8 800100e0 800100e4 60000013 ffffffff
[  651.217327] [<800100a4>] (arch_cpu_idle) from [<80052450>] (default_idle_call+0x28/0x34)
[  651.217338] [<80052428>] (default_idle_call) from [<80052568>] (cpu_startup_entry+0x10c/0x198)
[  651.217350] [<8005245c>] (cpu_startup_entry) from [<800145b0>] (secondary_start_kernel+0xe4/0xec)
[  651.217355]  r7:80558348 r3:00000040
[  651.217368] [<800144cc>] (secondary_start_kernel) from [<000094ec>] (0x94ec)
[  651.217372]  r5:00000055 r4:3607806a
[  651.218640] usb 1-1.5.1: new low-speed USB device number 7 using dwc_otg
[  651.310907] usb 1-1.5.1: New USB device found, idVendor=04b4, idProduct=0101
[  651.310927] usb 1-1.5.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  651.310934] usb 1-1.5.1: Product: SteelS쀁̄Љ̒DATA
[  651.310946] usb 1-1.5.1: Manufacturer: DATACOMP
[  651.336385] input: DATACOMP SteelS쀁̄Љ̒DATA as /devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5/1-1.5.1/1-1.5.1:1.0/0003:04B4:0101.0002/input/input1
[  651.388017] hid-generic 0003:04B4:0101.0002: input,hidraw1: USB HID v1.00 Keyboard [DATACOMP SteelS쀁̄Љ̒DATA] on usb-3f980000.usb-1.5.1/input0
[  651.402915] input: DATACOMP SteelS쀁̄Љ̒DATA as /devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5/1-1.5.1/1-1.5.1:1.1/0003:04B4:0101.0003/input/input2
[  651.454577] hid-generic 0003:04B4:0101.0003: input,hidraw2: USB HID v1.00 Device [DATACOMP SteelS쀁̄Љ̒DATA] on usb-3f980000.usb-1.5.1/input1
[  664.132199] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[  664.140290] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[  678.285110] systemd[1]: Started Turns off Raspberry Pi display backlight on shutdown/reboot.
[  678.285328] systemd[1]: Starting Synchronise Hardware Clock to System Clock...

@rodizio1
Copy link
Author

Oh, another one. This time I was using an USB Joystick:

[   23.108534] input: FrSky FrSky Taranis Joystick as /devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5/1-1.5.3/1-1.5.3:1.0/0003:0483:5710.0004/input/input3
[   23.109049] hid-generic 0003:0483:5710.0004: input,hidraw2: USB HID v1.11 Gamepad [FrSky FrSky Taranis Joystick] on usb-3f980000.usb-1.5.3/input0
[   84.361529] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[   84.370737] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[  478.771751] ------------[ cut here ]------------
[  478.771778] WARNING: CPU: 1 PID: 3749 at drivers/misc/vc04_services/interface/vchiq_arm/vchiq_arm.c:2502 vchiq_release_internal+0xb4/0x24c()
[  478.771785] Modules linked in: arc4 evdev joydev rt2800usb rt2800lib crc_ccitt rt2x00usb rt2x00lib brcmfmac mac80211 brcmutil cfg80211 bcm2835_gpiomem smsc95xx usbnet mii
[  478.771832] CPU: 1 PID: 3749 Comm: HTV Notify Not tainted 4.4.11-v7+ #24
[  478.771838] Hardware name: BCM2709
[  478.771842] Backtrace:
[  478.771860] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  478.771865]  r6:60000013 r5:80542648 r4:00000000 r3:00000000
[  478.771884] [<80012b20>] (show_stack) from [<8020bd7c>] (dump_stack+0xac/0xcc)
[  478.771896] [<8020bcd0>] (dump_stack) from [<8001cc28>] (warn_slowpath_common+0x8c/0xbc)
[  478.771900]  r7:000009c6 r6:80298730 r5:00000009 r4:00000000
[  478.771917] [<8001cb9c>] (warn_slowpath_common) from [<8001ccfc>] (warn_slowpath_null+0x24/0x2c)
[  478.771921]  r8:80587154 r7:ffffffff r6:b6266608 r5:805478a8 r4:b570b444
[  478.771939] [<8001ccd8>] (warn_slowpath_null) from [<80298730>] (vchiq_release_internal+0xb4/0x24c)
[  478.771949] [<8029867c>] (vchiq_release_internal) from [<802993c0>] (vchiq_ioctl+0x7ec/0x1870)
[  478.771953]  r9:b570b400 r8:001d5006 r7:b570aa00 r6:0000c40d r5:805478a8 r4:0000c40c
[  478.771975] [<80298bd4>] (vchiq_ioctl) from [<800e5c14>] (do_vfs_ioctl+0x41c/0x60c)
[  478.771980]  r10:00000000 r9:ad480000 r8:00000003 r7:0000c40d r6:b63dd900 r5:001d5006
[  478.771993]  r4:b62f8648
[  478.772001] [<800e57f8>] (do_vfs_ioctl) from [<800e5e40>] (SyS_ioctl+0x3c/0x64)
[  478.772006]  r9:ad480000 r8:00000003 r7:0000c40d r6:b63dd900 r5:001d5006 r4:b63dd901
[  478.772028] [<800e5e04>] (SyS_ioctl) from [<8000f760>] (ret_fast_syscall+0x0/0x34)
[  478.772032]  r8:8000f904 r7:00000036 r6:7e91badc r5:00000002 r4:76e922f0 r3:000000a4
[  478.772046] ---[ end trace 74968b23c32c427a ]---
[  478.772056] vchiq: vchiq_ioctl: cmd VCHIQ_IOC_RELEASE_SERVICE returned error -1 for service TVNT:3745
[  531.070016] usb 1-1.5.3: USB disconnect, device number 8
[  540.975679] usb 1-1.5.3: new full-speed USB device number 9 using dwc_otg
[  541.062228] usb 1-1.5.3: New USB device found, idVendor=0483, idProduct=5710
[  541.062268] usb 1-1.5.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  541.062276] usb 1-1.5.3: Product: FrSky Taranis Joystick
[  541.062283] usb 1-1.5.3: Manufacturer: FrSky
[  541.062290] usb 1-1.5.3: SerialNumber: 00000000001B
[  541.069458] input: FrSky FrSky Taranis Joystick as /devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5/1-1.5.3/1-1.5.3:1.0/0003:0483:5710.0005/input/input4
[  541.070155] hid-generic 0003:0483:5710.0005: input,hidraw2: USB HID v1.11 Gamepad [FrSky FrSky Taranis Joystick] on usb-3f980000.usb-1.5.3/input0
[  593.991626] INFO: rcu_sched self-detected stall on CPU
[  593.991652]  3-...: (1 GPs behind) idle=6e3/140000000000002/0 softirq=37028/37029 fqs=6896
[  593.991657]   (t=21001 jiffies g=15038 c=15037 q=1836)
[  593.991667] Task dump for CPU 3:
[  593.991671] sleep           R running      0  4470    354 0x00000002
[  593.991680] Backtrace:
[  593.991704] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  593.991709]  r6:80000193 r5:8053b980 r4:b2cc0000 r3:00000000
[  593.991731] [<80012b20>] (show_stack) from [<8003f97c>] (sched_show_task+0xac/0x100)
[  593.991745] [<8003f8d0>] (sched_show_task) from [<80041998>] (dump_cpu_task+0x40/0x44)
[  593.991750]  r5:8053b980 r4:00000003
[  593.991764] [<80041958>] (dump_cpu_task) from [<800630e0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[  593.991769]  r4:00000004 r3:00000001
[  593.991782] [<8006304c>] (rcu_dump_cpu_stacks) from [<80066af0>] (rcu_check_callbacks+0x494/0x7dc)
[  593.991788]  r8:362ef000 r7:0000072c r6:8053b980 r5:80532e00 r4:b6821e00 r3:00047c68
[  593.991810] [<8006665c>] (rcu_check_callbacks) from [<80069130>] (update_process_times+0x40/0x6c)
[  593.991815]  r10:b681eb38 r9:0000008a r8:4c9f6251 r7:0000008a r6:4c9f6426 r5:00000000
[  593.991829]  r4:b2cc0000
[  593.991841] [<800690f0>] (update_process_times) from [<80077c94>] (tick_sched_timer+0x70/0x284)
[  593.991845]  r5:ada4dcf0 r4:b681ec78
[  593.991858] [<80077c24>] (tick_sched_timer) from [<80069aa0>] (__hrtimer_run_queues+0x114/0x1ac)
[  593.991863]  r10:b681eb38 r9:0000008a r8:4c9f6251 r7:b681eb00 r6:b681eb0c r5:b681ec78
[  593.991876]  r4:b681eac0
[  593.991889] [<8006998c>] (__hrtimer_run_queues) from [<8006a050>] (hrtimer_interrupt+0xcc/0x228)
[  593.991893]  r10:b681eb38 r9:b681eb58 r8:b681eb78 r7:b681eac0 r6:00000003 r5:b681ead4
[  593.991907]  r4:8052fac0
[  593.991921] [<80069f84>] (hrtimer_interrupt) from [<803299c8>] (arch_timer_handler_phys+0x38/0x40)
[  593.991925]  r10:80536080 r9:362ef000 r8:00000011 r7:b6004d40 r6:805427bc r5:b6824580
[  593.991939]  r4:b6005cc0
[  593.991952] [<80329990>] (arch_timer_handler_phys) from [<8005f3e8>] (handle_percpu_devid_irq+0x74/0x8c)
[  593.991965] [<8005f374>] (handle_percpu_devid_irq) from [<8005af04>] (generic_handle_irq+0x2c/0x3c)
[  593.991969]  r8:00000000 r7:00000000 r6:ada4dec8 r5:00000000 r4:8052ffbc r3:8005f374
[  593.991989] [<8005aed8>] (generic_handle_irq) from [<8005b1dc>] (__handle_domain_irq+0x64/0xbc)
[  593.992001] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  593.992005]  r8:8052f280 r7:ada4dd24 r6:ffffffff r5:a0000113 r4:80068748 r3:ada4dcf0
[  593.992024] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  593.992034] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  593.992040] Exception stack(0xada4dcf0 to 0xada4dd38)
[  593.992048] dce0:                                     b2cbea30 ada4dd6c 00000000 803e02cc
[  593.992057] dd00: b2cbea30 b2cbe9e8 00047c67 80536100 8052f280 362ef000 80536080 ada4dd94
[  593.992065] dd20: ada4dd68 ada4dd40 80068d48 80068748 a0000113 ffffffff
[  593.992078] [<80068ce8>] (mod_timer) from [<803e0354>] (prb_retire_rx_blk_timer_expired+0x88/0x114)
[  593.992082]  r8:8052f280 r7:80536100 r6:803e02cc r5:b2cbe9e8 r4:b2cbe800
[  593.992100] [<803e02cc>] (prb_retire_rx_blk_timer_expired) from [<80068908>] (call_timer_fn+0x2c/0xa0)
[  593.992105]  r5:00000100 r4:b681e280
[  593.992118] [<800688dc>] (call_timer_fn) from [<80068abc>] (run_timer_softirq+0x140/0x20c)
[  593.992123]  r6:00000200 r5:00000000 r4:b681e280
[  593.992141] [<8006897c>] (run_timer_softirq) from [<8001fed4>] (__do_softirq+0x134/0x254)
[  593.992145]  r9:40000001 r8:00000100 r7:ada4c000 r6:80536084 r5:00000001 r4:00000080
[  593.992164] [<8001fda0>] (__do_softirq) from [<8002028c>] (irq_exit+0x80/0xb8)
[  593.992169]  r10:b54b9c00 r9:76fb8000 r8:00000000 r7:00000000 r6:00000000 r5:00000000
[  593.992182]  r4:8052ffbc
[  593.992192] [<8002020c>] (irq_exit) from [<8005b1e0>] (__handle_domain_irq+0x68/0xbc)
[  593.992196]  r4:8052ffbc r3:8005f374
[  593.992209] [<8005b178>] (__handle_domain_irq) from [<80010024>] (handle_IRQ+0x24/0x28)
[  593.992213]  r8:00000075 r7:ada4defc r6:ffffffff r5:60000013 r4:800be42c r3:ada4dec8
[  593.992232] [<80010000>] (handle_IRQ) from [<80009450>] (bcm2836_arm_irqchip_handle_irq+0xb0/0xb4)
[  593.992241] [<800093a0>] (bcm2836_arm_irqchip_handle_irq) from [<80013600>] (__irq_svc+0x40/0x54)
[  593.992246] Exception stack(0xada4dec8 to 0xada4df10)
[  593.992254] dec0:                   00000000 00000000 ffff0001 00000000 b54694d0 00000000
[  593.992263] dee0: 00000070 76fa9000 00000075 76fb8000 b54b9c00 ada4df6c 00000000 ada4df18
[  593.992271] df00: 800baec4 800be42c 60000013 ffffffff
[  593.992286] [<800be228>] (mprotect_fixup) from [<800be578>] (SyS_mprotect+0x114/0x1d8)
[  593.992290]  r10:00000000 r9:ada4c000 r8:00000000 r7:76fa9000 r6:76fb8000 r5:00000000
[  593.992303]  r4:76fb8000
[  593.992316] [<800be464>] (SyS_mprotect) from [<8000f760>] (ret_fast_syscall+0x0/0x34)
[  593.992320]  r8:8000f904 r7:0000007d r6:7e92771c r5:76fe7c50 r4:00000018
[  631.093096] systemd[1]: Started Turns off Raspberry Pi display backlight on shutdown/reboot.
[  631.093608] systemd[1]: Starting Synchronise Hardware Clock to System Clock...

@rodizio1
Copy link
Author

Another one:

[    6.599570] brcmfmac: power management disabled
[    6.763622] ieee80211 phy1: rt2x00lib_request_firmware: Info - Loading firmware file 'rt2870.bin'
[    6.765055] ieee80211 phy1: rt2x00lib_request_firmware: Info - Firmware detected - version: 0.29
[    8.654824] brcmfmac: brcmf_add_if: ERROR: netdev:wifihotspot0 already exists
[    8.654842] brcmfmac: brcmf_add_if: ignore IF event
[    8.702372] brcmfmac: brcmf_sdio_hdparse: seq 202: sequence number error, expect 201
[    8.702422] brcmfmac: brcmf_sdio_hdparse: seq 201: sequence number error, expect 203
[    8.702964] brcmfmac: brcmf_sdio_hdparse: seq 203: sequence number error, expect 202
[   14.106214] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[   15.718522] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1
[   21.174373] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[   21.216653] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[   22.827602] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1
[   46.059040] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)

@rodizio1
Copy link
Author

Crash galore continued:

[  938.591239] ------------[ cut here ]------------
[  938.591267] WARNING: CPU: 2 PID: 17281 at drivers/misc/vc04_services/interface/vchiq_arm/vchiq_arm.c:2502 vchiq_release_internal+0xb4/0x24c()
[  938.591274] Modules linked in: arc4 rt2800usb rt2800lib crc_ccitt evdev rt2x00usb rt2x00lib brcmfmac mac80211 brcmutil cfg80211 bcm2835_gpiomem smsc95xx usbnet mii
[  938.591320] CPU: 2 PID: 17281 Comm: HTV Notify Not tainted 4.4.11-v7+ #24
[  938.591326] Hardware name: BCM2709
[  938.591330] Backtrace:
[  938.591349] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  938.591355]  r6:60000013 r5:80542648 r4:00000000 r3:00000000
[  938.591374] [<80012b20>] (show_stack) from [<8020bd7c>] (dump_stack+0xac/0xcc)
[  938.591385] [<8020bcd0>] (dump_stack) from [<8001cc28>] (warn_slowpath_common+0x8c/0xbc)
[  938.591390]  r7:000009c6 r6:80298730 r5:00000009 r4:00000000
[  938.591407] [<8001cb9c>] (warn_slowpath_common) from [<8001ccfc>] (warn_slowpath_null+0x24/0x2c)
[  938.591412]  r8:80587154 r7:ffffffff r6:b6266608 r5:805478a8 r4:b563b244
[  938.591429] [<8001ccd8>] (warn_slowpath_null) from [<80298730>] (vchiq_release_internal+0xb4/0x24c)
[  938.591440] [<8029867c>] (vchiq_release_internal) from [<802993c0>] (vchiq_ioctl+0x7ec/0x1870)
[  938.591444]  r9:b563b200 r8:00304006 r7:b563b400 r6:0000c40d r5:805478a8 r4:0000c40c
[  938.591465] [<80298bd4>] (vchiq_ioctl) from [<800e5c14>] (do_vfs_ioctl+0x41c/0x60c)
[  938.591469]  r10:00000000 r9:adfae000 r8:00000003 r7:0000c40d r6:b2d179c0 r5:00304006
[  938.591483]  r4:b62f8648
[  938.591491] [<800e57f8>] (do_vfs_ioctl) from [<800e5e40>] (SyS_ioctl+0x3c/0x64)
[  938.591496]  r9:adfae000 r8:00000003 r7:0000c40d r6:b2d179c0 r5:00304006 r4:b2d179c1
[  938.591517] [<800e5e04>] (SyS_ioctl) from [<8000f760>] (ret_fast_syscall+0x0/0x34)
[  938.591522]  r8:8000f904 r7:00000036 r6:7e9e4adc r5:00000002 r4:76e332f0 r3:000000a4
[  938.591536] ---[ end trace 3cf8661a1aa3d6c9 ]---
[  938.591545] vchiq: vchiq_ioctl: cmd VCHIQ_IOC_RELEASE_SERVICE returned error -1 for service TVNT:17277

@rodizio1
Copy link
Author

This thing is so broken. Plugged in a USB keyboard, now USB is not functional anymore:

[  956.589509] usb 1-1.4.1: USB disconnect, device number 5
[  956.789641] usb 1-1.4.1: new low-speed USB device number 8 using dwc_otg
[  956.882221] usb 1-1.4.1: New USB device found, idVendor=04b4, idProduct=0101
[  956.882237] usb 1-1.4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  956.882244] usb 1-1.4.1: Product: SteelS쀁̄Љ̒DATA
[  956.882251] usb 1-1.4.1: Manufacturer: DATACOMP
[  956.905479] input: DATACOMP SteelS쀁̄Љ̒DATA as /devices/platform/soc/3f980000.usb/usb1/1-1/1-1.4/1-1.4.1/1-1.4.1:1.0/0003:04B4:0101.0003/input/input2
[  956.956587] hid-generic 0003:04B4:0101.0003: input,hidraw0: USB HID v1.00 Keyboard [DATACOMP SteelS쀁̄Љ̒DATA] on usb-3f980000.usb-1.4.1/input0
[  956.970428] input: DATACOMP SteelS쀁̄Љ̒DATA as /devices/platform/soc/3f980000.usb/usb1/1-1/1-1.4/1-1.4.1/1-1.4.1:1.1/0003:04B4:0101.0004/input/input3
[  957.021124] hid-generic 0003:04B4:0101.0004: input,hidraw1: USB HID v1.00 Device [DATACOMP SteelS쀁̄Љ̒DATA] on usb-3f980000.usb-1.4.1/input1
[  957.463261] ------------[ cut here ]------------
[  957.463286] WARNING: CPU: 1 PID: 1115 at drivers/misc/vc04_services/interface/vchiq_arm/vchiq_arm.c:2502 vchiq_release_internal+0xb4/0x24c()
[  957.463292] Modules linked in: arc4 ath9k_htc ath9k_common ath9k_hw ath mac80211 evdev brcmfmac brcmutil cfg80211 bcm2835_gpiomem smsc95xx usbnet mii
[  957.463343] CPU: 1 PID: 1115 Comm: HTV Notify Not tainted 4.4.11-v7+ #25
[  957.463347] Hardware name: BCM2709
[  957.463352] Backtrace:
[  957.463368] [<80012940>] (dump_backtrace) from [<80012b38>] (show_stack+0x18/0x1c)
[  957.463373]  r6:60000013 r5:80542648 r4:00000000 r3:00000000
[  957.463392] [<80012b20>] (show_stack) from [<8020bd7c>] (dump_stack+0xac/0xcc)
[  957.463404] [<8020bcd0>] (dump_stack) from [<8001cc28>] (warn_slowpath_common+0x8c/0xbc)
[  957.463408]  r7:000009c6 r6:80298730 r5:00000009 r4:00000000
[  957.463425] [<8001cb9c>] (warn_slowpath_common) from [<8001ccfc>] (warn_slowpath_null+0x24/0x2c)
[  957.463429]  r8:80587154 r7:ffffffff r6:b622a608 r5:805478a8 r4:b5628644
[  957.463447] [<8001ccd8>] (warn_slowpath_null) from [<80298730>] (vchiq_release_internal+0xb4/0x24c)
[  957.463457] [<8029867c>] (vchiq_release_internal) from [<802993c0>] (vchiq_ioctl+0x7ec/0x1870)
[  957.463462]  r9:b5628600 r8:0006d006 r7:b5628c00 r6:0000c40d r5:805478a8 r4:0000c40c
[  957.463481] [<80298bd4>] (vchiq_ioctl) from [<800e5c14>] (do_vfs_ioctl+0x41c/0x60c)
[  957.463486]  r10:00000000 r9:b55f6000 r8:00000003 r7:0000c40d r6:b3b54c00 r5:0006d006
[  957.463500]  r4:b62ec648
[  957.463508] [<800e57f8>] (do_vfs_ioctl) from [<800e5e40>] (SyS_ioctl+0x3c/0x64)
[  957.463513]  r9:b55f6000 r8:00000003 r7:0000c40d r6:b3b54c00 r5:0006d006 r4:b3b54c01
[  957.463532] [<800e5e04>] (SyS_ioctl) from [<8000f760>] (ret_fast_syscall+0x0/0x34)
[  957.463537]  r8:8000f904 r7:00000036 r6:7e932adc r5:00000002 r4:76e702f0 r3:000000a4
[  957.463551] ---[ end trace 5beb149fc9cc17d7 ]---
[  957.463558] vchiq: vchiq_ioctl: cmd VCHIQ_IOC_RELEASE_SERVICE returned error -1 for service TVNT:1111
[  977.743829] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[  977.752938] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

@rodizio1
Copy link
Author

This time, I just plugged a USB memory stick.

First time it's crapping out:

[   34.324642] usb 1-1.5: new full-speed USB device number 8 using dwc_otg
[   34.397669] usb 1-1.5: device descriptor read/64, error -32
[   34.571664] usb 1-1.5: device descriptor read/64, error -32
[   34.745656] usb 1-1.5: new full-speed USB device number 9 using dwc_otg
[   34.818660] usb 1-1.5: device descriptor read/64, error -32
[   34.992651] usb 1-1.5: device descriptor read/64, error -32
[   35.166648] usb 1-1.5: new full-speed USB device number 10 using dwc_otg
[   35.568655] usb 1-1.5: device not accepting address 10, error -32
[   35.641653] usb 1-1.5: new full-speed USB device number 11 using dwc_otg
[   36.043640] usb 1-1.5: device not accepting address 11, error -32
[   36.043737] usb 1-1-port5: unable to enumerate USB device

Second time it works:

[   56.087642] usb 1-1.5: new high-speed USB device number 12 using dwc_otg
[   56.173145] usb 1-1.5: New USB device found, idVendor=18a5, idProduct=0245
[   56.173159] usb 1-1.5: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   56.173210] usb 1-1.5: Product: STORE N GO
[   56.173218] usb 1-1.5: Manufacturer: Verbatim
[   56.173225] usb 1-1.5: SerialNumber: 1244000000001893
[   56.174725] usb-storage 1-1.5:1.0: USB Mass Storage device detected
[   56.175025] scsi host0: usb-storage 1-1.5:1.0
[   57.176722] scsi 0:0:0:0: Direct-Access     Verbatim STORE N GO       1.01 PQ: 0 ANSI: 2
[   57.179210] sd 0:0:0:0: [sda] 31299584 512-byte logical blocks: (16.0 GB/14.9 GiB)
[   57.179786] sd 0:0:0:0: [sda] Write Protect is off
[   57.179802] sd 0:0:0:0: [sda] Mode Sense: 03 00 00 00
[   57.180392] sd 0:0:0:0: [sda] No Caching mode page found
[   57.180409] sd 0:0:0:0: [sda] Assuming drive cache: write through
[   57.187117]  sda:
[   57.190005] sd 0:0:0:0: [sda] Attached SCSI removable disk

@pelwell
Copy link
Contributor

pelwell commented Apr 10, 2017

What does vcgencmd get_throttled report after one of these errors (i.e. before you reboot)?

@rodizio1
Copy link
Author

rodizio1 commented Apr 20, 2017

I didn't check that particilar time, but whenever I check, it´s as always: "throttled=0x0". It's not a power or temp issue. I have double and triple checked that. Tried official Pi power supply, and also a 7.5A 5.2V supply plus an extra Low-ESR cap wired directly to the GPIO pins.

BTW: I have had a script running doing that vcgencmd get_throttled check periodically every 30 seconds or so, this caused other issues, strange VCHQI kernel messages etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants