Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use restart_handlers instead of arm_pm_restart in odroid-reboot #38

Open
wants to merge 1 commit into
base: odroid-5.19.y
Choose a base branch
from

Conversation

Cheaterman
Copy link

Very inspired by this patch.

@Cheaterman
Copy link
Author

Now, as to whether or how much the odroid-reboot is needed in the first place, I'm honestly unsure ; I fixed it because I saw my N2+ fail to reboot once, but it doesn't seem to be consistent, so I have no idea :-)

@pyavitz
Copy link

pyavitz commented Oct 9, 2022

I use the following and other than an oddity here and there its been consistent.

diff -Naur a/arch/arm64/include/asm/system_misc.h b/arch/arm64/include/asm/system_misc.h
--- a/arch/arm64/include/asm/system_misc.h	2022-02-27 17:36:33.000000000 -0500
+++ b/arch/arm64/include/asm/system_misc.h	2022-03-03 13:51:34.031068479 -0500
@@ -32,6 +32,8 @@
 struct mm_struct;
 extern void __show_regs(struct pt_regs *);
 
+extern void (*arm_pm_restart)(enum reboot_mode reboot_mode, const char *cmd);
+
 #endif	/* __ASSEMBLY__ */
 
 #endif	/* __ASM_SYSTEM_MISC_H */
diff -Naur a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
--- a/arch/arm64/kernel/process.c	2022-02-27 17:36:33.000000000 -0500
+++ b/arch/arm64/kernel/process.c	2022-03-03 13:53:35.651284725 -0500
@@ -68,6 +68,8 @@
 void (*pm_power_off)(void);
 EXPORT_SYMBOL_GPL(pm_power_off);
 
+void (*arm_pm_restart)(enum reboot_mode reboot_mode, const char *cmd);
+
 #ifdef CONFIG_HOTPLUG_CPU
 void arch_cpu_idle_dead(void)
 {
@@ -138,7 +140,10 @@
 		efi_reboot(reboot_mode, NULL);
 
 	/* Now call the architecture specific reboot code. */
-	do_kernel_restart(cmd);
+	if (arm_pm_restart)
+		arm_pm_restart(reboot_mode, cmd);
+	else
+		do_kernel_restart(cmd);
 
 	/*
 	 * Whoops - the architecture was unable to reboot.

@ginkage
Copy link

ginkage commented Oct 26, 2022

I second @Cheaterman's approach, it makes more sense than revert of ab6cef1.

@pyavitz
Copy link

pyavitz commented Oct 26, 2022

I'm actually starting to lean towards odroid-reboot isn't needed at all when on Linux 6.x.y. I was recently trying to squash a problem I was having with an N2 2GB variant (random lock ups) and started with using a clean mainline kernel. To my surprise the board rebooted and powered down with no issue. No patches applied.

I still need to test this on the N2+ and C4, but that looks like a pretty good sign to me.

tobetter pushed a commit that referenced this pull request Feb 21, 2023
If a non-root cgroup gets removed when there is a thread that registered
trigger and is polling on a pressure file within the cgroup, the polling
waitqueue gets freed in the following path:

 do_rmdir
   cgroup_rmdir
     kernfs_drain_open_files
       cgroup_file_release
         cgroup_pressure_release
           psi_trigger_destroy

However, the polling thread still has a reference to the pressure file and
will access the freed waitqueue when the file is closed or upon exit:

 fput
   ep_eventpoll_release
     ep_free
       ep_remove_wait_queue
         remove_wait_queue

This results in use-after-free as pasted below.

The fundamental problem here is that cgroup_file_release() (and
consequently waitqueue's lifetime) is not tied to the file's real lifetime.
Using wake_up_pollfree() here might be less than ideal, but it is in line
with the comment at commit 42288cb ("wait: add wake_up_pollfree()")
since the waitqueue's lifetime is not tied to file's one and can be
considered as another special case. While this would be fixable by somehow
making cgroup_file_release() be tied to the fput(), it would require
sizable refactoring at cgroups or higher layer which might be more
justifiable if we identify more cases like this.

  BUG: KASAN: use-after-free in _raw_spin_lock_irqsave+0x60/0xc0
  Write of size 4 at addr ffff88810e625328 by task a.out/4404

	CPU: 19 PID: 4404 Comm: a.out Not tainted 6.2.0-rc6 #38
	Hardware name: Amazon EC2 c5a.8xlarge/, BIOS 1.0 10/16/2017
	Call Trace:
	<TASK>
	dump_stack_lvl+0x73/0xa0
	print_report+0x16c/0x4e0
	kasan_report+0xc3/0xf0
	kasan_check_range+0x2d2/0x310
	_raw_spin_lock_irqsave+0x60/0xc0
	remove_wait_queue+0x1a/0xa0
	ep_free+0x12c/0x170
	ep_eventpoll_release+0x26/0x30
	__fput+0x202/0x400
	task_work_run+0x11d/0x170
	do_exit+0x495/0x1130
	do_group_exit+0x100/0x100
	get_signal+0xd67/0xde0
	arch_do_signal_or_restart+0x2a/0x2b0
	exit_to_user_mode_prepare+0x94/0x100
	syscall_exit_to_user_mode+0x20/0x40
	do_syscall_64+0x52/0x90
	entry_SYSCALL_64_after_hwframe+0x63/0xcd
	</TASK>

 Allocated by task 4404:

	kasan_set_track+0x3d/0x60
	__kasan_kmalloc+0x85/0x90
	psi_trigger_create+0x113/0x3e0
	pressure_write+0x146/0x2e0
	cgroup_file_write+0x11c/0x250
	kernfs_fop_write_iter+0x186/0x220
	vfs_write+0x3d8/0x5c0
	ksys_write+0x90/0x110
	do_syscall_64+0x43/0x90
	entry_SYSCALL_64_after_hwframe+0x63/0xcd

 Freed by task 4407:

	kasan_set_track+0x3d/0x60
	kasan_save_free_info+0x27/0x40
	____kasan_slab_free+0x11d/0x170
	slab_free_freelist_hook+0x87/0x150
	__kmem_cache_free+0xcb/0x180
	psi_trigger_destroy+0x2e8/0x310
	cgroup_file_release+0x4f/0xb0
	kernfs_drain_open_files+0x165/0x1f0
	kernfs_drain+0x162/0x1a0
	__kernfs_remove+0x1fb/0x310
	kernfs_remove_by_name_ns+0x95/0xe0
	cgroup_addrm_files+0x67f/0x700
	cgroup_destroy_locked+0x283/0x3c0
	cgroup_rmdir+0x29/0x100
	kernfs_iop_rmdir+0xd1/0x140
	vfs_rmdir+0xfe/0x240
	do_rmdir+0x13d/0x280
	__x64_sys_rmdir+0x2c/0x30
	do_syscall_64+0x43/0x90
	entry_SYSCALL_64_after_hwframe+0x63/0xcd

Fixes: 0e94682 ("psi: introduce psi monitor")
Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
Signed-off-by: Mengchi Cheng <mengcc@amazon.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/lkml/20230106224859.4123476-1-kamatam@amazon.com/
Link: https://lore.kernel.org/r/20230214212705.4058045-1-kamatam@amazon.com
tobetter pushed a commit that referenced this pull request Feb 23, 2023
commit c2dbe32 upstream.

If a non-root cgroup gets removed when there is a thread that registered
trigger and is polling on a pressure file within the cgroup, the polling
waitqueue gets freed in the following path:

 do_rmdir
   cgroup_rmdir
     kernfs_drain_open_files
       cgroup_file_release
         cgroup_pressure_release
           psi_trigger_destroy

However, the polling thread still has a reference to the pressure file and
will access the freed waitqueue when the file is closed or upon exit:

 fput
   ep_eventpoll_release
     ep_free
       ep_remove_wait_queue
         remove_wait_queue

This results in use-after-free as pasted below.

The fundamental problem here is that cgroup_file_release() (and
consequently waitqueue's lifetime) is not tied to the file's real lifetime.
Using wake_up_pollfree() here might be less than ideal, but it is in line
with the comment at commit 42288cb ("wait: add wake_up_pollfree()")
since the waitqueue's lifetime is not tied to file's one and can be
considered as another special case. While this would be fixable by somehow
making cgroup_file_release() be tied to the fput(), it would require
sizable refactoring at cgroups or higher layer which might be more
justifiable if we identify more cases like this.

  BUG: KASAN: use-after-free in _raw_spin_lock_irqsave+0x60/0xc0
  Write of size 4 at addr ffff88810e625328 by task a.out/4404

	CPU: 19 PID: 4404 Comm: a.out Not tainted 6.2.0-rc6 #38
	Hardware name: Amazon EC2 c5a.8xlarge/, BIOS 1.0 10/16/2017
	Call Trace:
	<TASK>
	dump_stack_lvl+0x73/0xa0
	print_report+0x16c/0x4e0
	kasan_report+0xc3/0xf0
	kasan_check_range+0x2d2/0x310
	_raw_spin_lock_irqsave+0x60/0xc0
	remove_wait_queue+0x1a/0xa0
	ep_free+0x12c/0x170
	ep_eventpoll_release+0x26/0x30
	__fput+0x202/0x400
	task_work_run+0x11d/0x170
	do_exit+0x495/0x1130
	do_group_exit+0x100/0x100
	get_signal+0xd67/0xde0
	arch_do_signal_or_restart+0x2a/0x2b0
	exit_to_user_mode_prepare+0x94/0x100
	syscall_exit_to_user_mode+0x20/0x40
	do_syscall_64+0x52/0x90
	entry_SYSCALL_64_after_hwframe+0x63/0xcd
	</TASK>

 Allocated by task 4404:

	kasan_set_track+0x3d/0x60
	__kasan_kmalloc+0x85/0x90
	psi_trigger_create+0x113/0x3e0
	pressure_write+0x146/0x2e0
	cgroup_file_write+0x11c/0x250
	kernfs_fop_write_iter+0x186/0x220
	vfs_write+0x3d8/0x5c0
	ksys_write+0x90/0x110
	do_syscall_64+0x43/0x90
	entry_SYSCALL_64_after_hwframe+0x63/0xcd

 Freed by task 4407:

	kasan_set_track+0x3d/0x60
	kasan_save_free_info+0x27/0x40
	____kasan_slab_free+0x11d/0x170
	slab_free_freelist_hook+0x87/0x150
	__kmem_cache_free+0xcb/0x180
	psi_trigger_destroy+0x2e8/0x310
	cgroup_file_release+0x4f/0xb0
	kernfs_drain_open_files+0x165/0x1f0
	kernfs_drain+0x162/0x1a0
	__kernfs_remove+0x1fb/0x310
	kernfs_remove_by_name_ns+0x95/0xe0
	cgroup_addrm_files+0x67f/0x700
	cgroup_destroy_locked+0x283/0x3c0
	cgroup_rmdir+0x29/0x100
	kernfs_iop_rmdir+0xd1/0x140
	vfs_rmdir+0xfe/0x240
	do_rmdir+0x13d/0x280
	__x64_sys_rmdir+0x2c/0x30
	do_syscall_64+0x43/0x90
	entry_SYSCALL_64_after_hwframe+0x63/0xcd

Fixes: 0e94682 ("psi: introduce psi monitor")
Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
Signed-off-by: Mengchi Cheng <mengcc@amazon.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/lkml/20230106224859.4123476-1-kamatam@amazon.com/
Link: https://lore.kernel.org/r/20230214212705.4058045-1-kamatam@amazon.com
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
tobetter pushed a commit that referenced this pull request Feb 23, 2023
commit c2dbe32 upstream.

If a non-root cgroup gets removed when there is a thread that registered
trigger and is polling on a pressure file within the cgroup, the polling
waitqueue gets freed in the following path:

 do_rmdir
   cgroup_rmdir
     kernfs_drain_open_files
       cgroup_file_release
         cgroup_pressure_release
           psi_trigger_destroy

However, the polling thread still has a reference to the pressure file and
will access the freed waitqueue when the file is closed or upon exit:

 fput
   ep_eventpoll_release
     ep_free
       ep_remove_wait_queue
         remove_wait_queue

This results in use-after-free as pasted below.

The fundamental problem here is that cgroup_file_release() (and
consequently waitqueue's lifetime) is not tied to the file's real lifetime.
Using wake_up_pollfree() here might be less than ideal, but it is in line
with the comment at commit 42288cb ("wait: add wake_up_pollfree()")
since the waitqueue's lifetime is not tied to file's one and can be
considered as another special case. While this would be fixable by somehow
making cgroup_file_release() be tied to the fput(), it would require
sizable refactoring at cgroups or higher layer which might be more
justifiable if we identify more cases like this.

  BUG: KASAN: use-after-free in _raw_spin_lock_irqsave+0x60/0xc0
  Write of size 4 at addr ffff88810e625328 by task a.out/4404

	CPU: 19 PID: 4404 Comm: a.out Not tainted 6.2.0-rc6 #38
	Hardware name: Amazon EC2 c5a.8xlarge/, BIOS 1.0 10/16/2017
	Call Trace:
	<TASK>
	dump_stack_lvl+0x73/0xa0
	print_report+0x16c/0x4e0
	kasan_report+0xc3/0xf0
	kasan_check_range+0x2d2/0x310
	_raw_spin_lock_irqsave+0x60/0xc0
	remove_wait_queue+0x1a/0xa0
	ep_free+0x12c/0x170
	ep_eventpoll_release+0x26/0x30
	__fput+0x202/0x400
	task_work_run+0x11d/0x170
	do_exit+0x495/0x1130
	do_group_exit+0x100/0x100
	get_signal+0xd67/0xde0
	arch_do_signal_or_restart+0x2a/0x2b0
	exit_to_user_mode_prepare+0x94/0x100
	syscall_exit_to_user_mode+0x20/0x40
	do_syscall_64+0x52/0x90
	entry_SYSCALL_64_after_hwframe+0x63/0xcd
	</TASK>

 Allocated by task 4404:

	kasan_set_track+0x3d/0x60
	__kasan_kmalloc+0x85/0x90
	psi_trigger_create+0x113/0x3e0
	pressure_write+0x146/0x2e0
	cgroup_file_write+0x11c/0x250
	kernfs_fop_write_iter+0x186/0x220
	vfs_write+0x3d8/0x5c0
	ksys_write+0x90/0x110
	do_syscall_64+0x43/0x90
	entry_SYSCALL_64_after_hwframe+0x63/0xcd

 Freed by task 4407:

	kasan_set_track+0x3d/0x60
	kasan_save_free_info+0x27/0x40
	____kasan_slab_free+0x11d/0x170
	slab_free_freelist_hook+0x87/0x150
	__kmem_cache_free+0xcb/0x180
	psi_trigger_destroy+0x2e8/0x310
	cgroup_file_release+0x4f/0xb0
	kernfs_drain_open_files+0x165/0x1f0
	kernfs_drain+0x162/0x1a0
	__kernfs_remove+0x1fb/0x310
	kernfs_remove_by_name_ns+0x95/0xe0
	cgroup_addrm_files+0x67f/0x700
	cgroup_destroy_locked+0x283/0x3c0
	cgroup_rmdir+0x29/0x100
	kernfs_iop_rmdir+0xd1/0x140
	vfs_rmdir+0xfe/0x240
	do_rmdir+0x13d/0x280
	__x64_sys_rmdir+0x2c/0x30
	do_syscall_64+0x43/0x90
	entry_SYSCALL_64_after_hwframe+0x63/0xcd

Fixes: 0e94682 ("psi: introduce psi monitor")
Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
Signed-off-by: Mengchi Cheng <mengcc@amazon.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Suren Baghdasaryan <surenb@google.com>
Acked-by: Peter Zijlstra <peterz@infradead.org>
Cc: stable@vger.kernel.org
Link: https://lore.kernel.org/lkml/20230106224859.4123476-1-kamatam@amazon.com/
Link: https://lore.kernel.org/r/20230214212705.4058045-1-kamatam@amazon.com
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
@ashh87
Copy link

ashh87 commented Feb 24, 2023

I've recently pushed a similar patch to Armbian (armbian/build#4824). The odroid-reboot driver is mainly useful for fixing reboot when the boards are used with UHS SD cards, as it resets their voltages to a state which the bootloader can use them from.

@Cheaterman I found that I had to set the notifier_block priority to 130 so that the driver is called before the standard PSCI functions. Otherwise, the standard functions reboot the board before the odroid_card_reset function is called. If you're not using a UHS card for boot, then the N2+ reboot issue you saw is probably unrelated, and I think this code will now compile, but won't get called.

@pyavitz I continued to have this issue on Armbian with 6.x kernels before I patched them. I was thinking of tidying up my Armbian patch and submitting it to the mainline kernel, but I did wonder if it might be better implemented as some optional device-tree properties of the SD card driver, so boards which needed it could add it to their device tree there, rather than compile this special driver. Something for me to look into anyway! Also, as you have an N2+, are you able to do some testing on it with a UHS card please? I only have a C4, so I haven't added the device-tree node for odroid,reboot back into Armbian for the N2/N2+ as I can't test them.

@tobetter The other thing this driver does is make what looks to me like an undocumented PSCI call before powering off the board: "__invoke_psci_fn_smc(0x82000042, 1, 0, 0);" Are you able to tell me what this call is for?

@rpardini
Copy link

@pyavitz : works with pure-mainline, sans reboot driver, cos pure mainline does not enable UHS modes for C4/N2. If you patch DT for high speed, then reboot, it hangs in early boot (if booting from SD/eMMC). If booting from SPI it works no problem too.

@pyavitz
Copy link

pyavitz commented Feb 24, 2023

@pyavitz : works with pure-mainline, sans reboot driver, cos pure mainline does not enable UHS modes for C4/N2. If you patch DT for high speed, then reboot, it hangs in early boot (if booting from SD/eMMC). If booting from SPI it works no problem too.

With and without the patching, my N2 reboots fine. In my testing, this is not the case on the N2+, N2L and C4. A quick scan of the Armbian forum shows most peps are having reboot issues with those units.

@rpardini
Copy link

Interesting, would like to get to the bottom of this. @tobetter certainly knows

@pyavitz
Copy link

pyavitz commented Feb 24, 2023

Interesting, would like to get to the bottom of this. @tobetter certainly knows

Yeah it's mystery. I'm at the point where I'm tired of messing with it, so I just use what I know works. If it requires patching, so be it.

@Cheaterman
Copy link
Author

Cheaterman commented Mar 10, 2023

If you're not using a UHS card for boot, then the N2+ reboot issue you saw is probably unrelated, and I think this code will now compile, but won't get called.

I'm using eMMCs for booting my N2+s, I don't know if that applies too. We are now noticing production devices that are randomly refusing to reboot - they stop but won't restart. I'll check whether I applied my patch on their kernel or not, but I wonder if this could be related.

EDIT: And I do mean randomly ; we tested the reboot by hand yesterday with no issues, but crontab triggering it later during the night made the device hang.

@gitmeister
Copy link

gitmeister commented Mar 10, 2023

My 2 cents after studying the C2 and C4 schematics.

The Odroid C2 has a pull-down on the TF_3V3N_1V8_EN signal which governs the IO voltage levels (U13 TF_IO Regulator) used by the sdcard. According to the sdcard specs (shown below) the card needs 3.3V to initiate a proper reboot. If the voltage switching pin is left floating at boot time (ie. via the open-drain DTS directive), it will default the voltage to 3.3V. This patch has resulted in proper reboots where previously the board has solidly refused to reboot). Could the issue be related to the sdcards being inadvertently rebooted at the lower voltage (1.8V) after running in UHS-I mode and not being reset to the higher voltage prior to boot?

The Odroid C4 (which I do not have) has a different means of generating the voltage switch but it does have pull-ups/downs that could be exploited via a similar DTS patch to achieve the 3.3V IO voltage at boot-time. In that case TF_3V3N_1V8_EN could also be made open-drain and thus default to enabling the U16 VDDIO regulator to 3.3V.

SD memory cards have used a 3.3V signaling interface since the SD standard was introduced in 2000 and through SD Specification 3.0, when 1.8V signaling was added with the UHS-I bus mode, the ultimate removable single-ended interface. UHS-I adopted 1.8V signaling because it is suitable for faster rise/fall time and lower electromagnetic interference. However, UHS memory cards still require 3.3V signaling to initialize the card so hosts need to support 3.3V signaling, too.

https://www.sdcard.org/developers/sd-standard-overview/low-voltage-signaling/

Edit: Curiously, I checked the DTS for the Odroid Bananapi5 which happens to have the same VDDIO circuit as at the C4, and the DTS already incorporates the open-drain on TF_3V3N_1V8_EN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants