ztest: support runs suites and cases from shell #58374

nashif · 2023-05-28T16:18:41Z

Add shell support to ztest to allow listing suites and cases and running
them interactively from the shell.

Note: something I had in a branch from last year, still needs some work, but essentially it works :)

west build -b qemu_x86 tests/kernel/common/ -t run -- -DCONFIG_ZTEST_SHELL=y

output

uart:~$ ztest
ztest - Ztest commands
Subcommands:
  run-all     :Run all tests
  list        :List all test suites
  list_cases  :List all test cases
  run         :Run full suite
  run-test    :Run testcase


uart:~$ ztest list
atomic
bitarray
boot_delay
byteorder
clock
common
common_1cpu
common_errno
irq_offload
multilib
pow2
printk


uart:~$ ztest list_cases
atomic::test_atomic
atomic::test_threads_access_atomic
bitarray::test_bitarray_alloc_free
bitarray::test_bitarray_declare
bitarray::test_bitarray_region_set_clear
bitarray::test_bitarray_set_clear
bitarray::test_ffs
boot_delay::test_bootdelay
byteorder::test_byteorder_mem_swap
byteorder::test_byteorder_memcpy_swap
byteorder::test_sys_get_be16
byteorder::test_sys_get_be24
byteorder::test_sys_get_be32
byteorder::test_sys_get_be48
byteorder::test_sys_get_be64
byteorder::test_sys_get_le16
byteorder::test_sys_get_le24
byteorder::test_sys_get_le32
byteorder::test_sys_get_le48
byteorder::test_sys_get_le64
byteorder::test_sys_put_be16
byteorder::test_sys_put_be24
byteorder::test_sys_put_be32
byteorder::test_sys_put_be48
byteorder::test_sys_put_be64
byteorder::test_sys_put_le16
byteorder::test_sys_put_le24
byteorder::test_sys_put_le32
byteorder::test_sys_put_le48
byteorder::test_sys_put_le64
clock::test_clock_cycle_32
....

Signed-off-by: Anas Nashif anas.nashif@intel.com

yperess · 2023-05-28T16:24:06Z

🤯 This is super cool, I'll review later today

stephanosio · 2023-05-28T17:29:57Z

subsys/testsuite/ztest/src/ztest_new.c

@@ -18,6 +18,13 @@ static struct k_thread ztest_thread;
 #endif
 static bool failed_expectation;

+#ifdef CONFIG_SHELL


Instead of automatically enabling this when the shell is enabled, should we introduce a separate config (e.g. CONFIG_ZTEST_SHELL) to control this feature so that we do not unnecessarily increase the footprint of the existing shell tests?

right, this will be needed anyways, otherwise any test enabling the shell as we have it now, will end up being stuck waiting for input.

teburd · 2023-06-23T16:40:02Z

Can we move this from draft? seems really useful

PerMac

When I try ztest run-all/ztest run - all (not sure which one is correct) I get nothing. Calling ztest run crashes the app:

uart:~$ ztest run
E: Page fault at address (nil) (error code 0x0)
E: Linear address not present in page tables
E: Access violation: supervisor thread not allowed to read
E: PTE: not present
E: EAX: 0x00000000, EBX: 0x0011d0e0, ECX: 0x00000001, EDX: 0x00000000
E: ESI: 0x0012d65c, EDI: 0x00000001, EBP: 0x0012d5e0, ESP: 0x0012d5e0
E: EFLAGS: 0x00000246 CS: 0x0008 CR3: 0x0016c000
E: call trace:
E: EIP: 0x0010f012
E:      0x0010ffbb (0x0)
E:      0x00109ea3 (0x11d5c8)
E:      0x0010a0d3 (0x12a318)
E:      0x0010968a (0x11d5c8)
E:      0x0010ac6d (0xaaaaaaaa)
E:      0x00106168 (0x11d5c8)
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x129c00 (shell_uart)
E: Halting system
FAILED: zephyr/CMakeFiles/run_qemu /home/maciej/zephyrproject/zephyr/build/zephyr/CMakeFiles/run_qemu 
cd /home/maciej/zephyrproject/zephyr/build && /home/maciej/zephyr-sdk-0.16.1/sysroots/x86_64-pokysdk-linux/usr/bin/qemu-system-i386 -m 4 -cpu qemu32,+nx,+pae -machine q35 -device isa-debug-exit,iobase=0xf4,iosize=0x04 -no-reboot -nographic -no-acpi -net none -pidfile qemu.pid -chardev stdio,id=con,mux=on -serial chardev:con -mon chardev=con,mode=readline -kernel /home/maciej/zephyrproject/zephyr/build/zephyr/zephyr.elf
ninja: build stopped: subcommand failed

PerMac · 2023-07-18T11:04:48Z

I find the syntax a bit confusing:

There are run-test and run-all but list_cases. Is it by purpose?
- wouldn't only having either - or _ be clearer?
- also once the same item is called test and once case. I think alinging would be beneficial (run-case and list-cases?)
- What about having list -> list-suites. Maybe it would give a better understanding than plane list since there is already specified list_cases. This one is just an open question. The same would go then with run->run-suite.

I still see some errors, but cannot say if they are related to this feature or tests themselves.

Not repeatable behavior: consecutive calling run-all give different results. The first time everything pass, consecutive call causes several errors:

START - test_threads_access_atomic

    Assertion failed at WEST_TOPDIR/zephyr/tests/kernel/common/src/atomic.c:337: atomic_test_threads_access_atomic: (total_atomic == (TEST_CYCLE * THREADS_NUM) is false)
atomic counting failure
 FAIL - test_threads_access_atomic in 0.044 seconds
 ...
 ===================================================================
START - test_nested_irq_offload

    Assertion failed at WEST_TOPDIR/zephyr/tests/kernel/common/src/irq_offload.c:113: common_1cpu_test_nested_irq_offload: (timer_executed is true)
timer ran too soon
 FAIL - test_nested_irq_offload in 0.001 seconds
===================================================================
START - test_timeout_order

    Assertion failed at WEST_TOPDIR/zephyr/tests/kernel/common/src/timeout_order.c:81: common_1cpu_test_timeout_order: (results[ii] not equal to ii)

 FAIL - test_timeout_order in 0.149 seconds
===================================================================
TESTSUITE common_1cpu failed.
Running TESTSUITE common_errno

and then the emulation hangs at

===================================================================
START - test_printk

I haven't manage to get ztest run-test working. Always getting a crash:

uart:~$ ztest run-test common::test_version
E: Page fault at address (nil) (error code 0x0)
E: Linear address not present in page tables
E: Access violation: supervisor thread not allowed to read
E: PTE: not present
E: EAX: 0x00000000, EBX: 0x0011d0b8, ECX: 0x00000014, EDX: 0x0011fffa
E: ESI: 0x0012c65c, EDI: 0x00000000, EBP: 0x0012c5d8, ESP: 0x0012c5d4
E: EFLAGS: 0x00000206 CS: 0x0008 CR3: 0x0016b000
E: call trace:
E: EIP: 0x0010efb3
E:      0x0010ff87 (0x0)
E:      0x00109e01 (0x11d5a0)
E:      0x0010a031 (0x129318)
E:      0x001095e8 (0x11d5a0)
E:      0x0010abcb (0xaaaaaaaa)
E:      0x0010616b (0x11d5a0)
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x128c00 (shell_uart)
E: Halting system
FAILED: zephyr/CMakeFiles/run_qemu /home/maciej/zephyrproject/zephyr/build/zephyr/CMakeFiles/run_qemu 
cd /home/maciej/zephyrproject/zephyr/build && /home/maciej/zephyr-sdk-0.16.1/sysroots/x86_64-pokysdk-linux/usr/bin/qemu-system-i386 -m 4 -cpu qemu32,+nx,+pae -machine q35 -device isa-debug-exit,iobase=0xf4,iosize=0x04 -no-reboot -nographic -no-acpi -net none -pidfile qemu.pid -chardev stdio,id=con,mux=on -serial chardev:con -mon chardev=con,mode=readline -kernel /home/maciej/zephyrproject/zephyr/build/zephyr/zephyr.elf
ninja: build stopped: subcommand failed.

nashif · 2023-07-18T11:12:10Z

I still see some errors, but cannot say if they are related to this feature or tests themselves.

those arew issues with the tests, they are not able to run multiple times.

nashif · 2023-07-18T11:12:51Z

I find the syntax a bit confusing:

yes, aware of that. needs some work still.

github-actions · 2023-09-17T00:29:39Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

github-actions · 2023-11-18T00:29:55Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

nashif · 2023-12-21T13:53:58Z

updated...

stale review, please revisit

PerMac

All my comments were addressed. Thank you! I played with the feature on qemu_x86 and also on a real target (nrf52840dk) with minicom and everything worked as described.

nashif · 2023-12-21T15:29:15Z

I'll take it for a spin but looks fine to me.

Do you plan on adding any follow-up PRs?

yeah, there will be a few changes needed to adress some corner cases, but most major issues should be addressed now.

cfriedt · 2023-12-24T03:36:03Z

Make it green and I'll approve 😄

nashif · 2024-01-03T03:19:07Z

Make it green and I'll approve 😄

yep, coming. The issues are all stack related given that some ztest features are runtime in the current implementation.

This c file is not being used anymore, so remove it. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

Reduce verbosity and debug messages on the screen. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

cfriedt · 2024-01-08T19:02:54Z

Every time I see docs failing in the build I have a minor panic attack thinking it was me 😅

- Support for listing both testcases and testsuites - Support for running single suites or single test cases - Support shuffling tests and repeating execution based on command line arguments. For example, build with west build -p -b qemu_cortex_m3 tests/kernel/sleep -t run -- \ -DCONFIG_ZTEST_SHUFFLE=y -DCONFIG_ZTEST_SHELL=y Following commands are available: uart:~$ ztest ztest - Ztest commands Subcommands: run-all :Run all tests shuffle :Shuffle tests list-testsuites :List all test suites list-testcases :List all test cases run-testsuite :Run test suite run-testcase :Run testcase shuffle accepts two arguments --suite_iter and --case_iter which allows repeated exercution of testcases or suites. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

nashif · 2024-01-08T19:13:26Z

Every time I see docs failing in the build I have a minor panic attack thinking it was me 😅

"It's not you, it is me".

nashif · 2024-01-08T22:03:44Z

@cfriedt passes CI now :)

cfriedt · 2024-01-09T00:21:53Z

So tempting to merge...

nashif requested review from yperess and asemjonovs May 28, 2023 16:21

stephanosio reviewed May 28, 2023

View reviewed changes

nashif force-pushed the ztest_shell branch 5 times, most recently from 90e33b5 to c16cd5c Compare May 29, 2023 09:57

teburd previously approved these changes Jun 23, 2023

View reviewed changes

nashif marked this pull request as ready for review June 23, 2023 16:42

zephyrbot added the area: Testsuite Testsuite label Jun 23, 2023

zephyrbot requested review from aaronemassey and jeremybettis June 23, 2023 16:42

zephyrbot assigned nashif Jun 23, 2023

PerMac previously requested changes Jun 29, 2023

View reviewed changes

nashif dismissed teburd’s stale review via f09b17f July 11, 2023 12:49

nashif force-pushed the ztest_shell branch from c16cd5c to f09b17f Compare July 11, 2023 12:49

nashif requested a review from PerMac July 11, 2023 12:50

github-actions bot added the Stale label Sep 17, 2023

nashif removed the Stale label Sep 18, 2023

github-actions bot added the Stale label Nov 18, 2023

kartben mentioned this pull request Nov 19, 2023

ztest: add a ztest shell command #65437

Closed

nashif removed the Stale label Nov 20, 2023

nashif force-pushed the ztest_shell branch from de21958 to 63befe9 Compare December 21, 2023 12:42

zephyrbot added area: mcumgr area: Documentation area: EEPROM area: Bluetooth Host area: Bluetooth labels Dec 21, 2023

zephyrbot requested a review from nordicjm December 21, 2023 12:43

nashif force-pushed the ztest_shell branch from 63befe9 to 92d0a25 Compare December 21, 2023 13:53

PerMac previously approved these changes Dec 21, 2023

View reviewed changes

teburd previously approved these changes Dec 21, 2023

View reviewed changes

nashif added 2 commits January 8, 2024 13:45

ztest: remove old file

e9d7a1b

This c file is not being used anymore, so remove it. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

tests: sleep: reduce verbosity

5cd699e

Reduce verbosity and debug messages on the screen. Signed-off-by: Anas Nashif <anas.nashif@intel.com>

nashif dismissed stale reviews from teburd and PerMac via 8949ef3 January 8, 2024 18:46

nashif force-pushed the ztest_shell branch from 92d0a25 to 8949ef3 Compare January 8, 2024 18:46

nashif force-pushed the ztest_shell branch from 8949ef3 to 52bfb50 Compare January 8, 2024 19:11

teburd approved these changes Jan 8, 2024

View reviewed changes

cfriedt approved these changes Jan 9, 2024

View reviewed changes

PerMac approved these changes Jan 9, 2024

View reviewed changes

carlescufi merged commit afc319e into zephyrproject-rtos:main Jan 9, 2024
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ztest: support runs suites and cases from shell #58374

ztest: support runs suites and cases from shell #58374

nashif commented May 28, 2023 •

edited

Loading

yperess commented May 28, 2023

stephanosio May 28, 2023

nashif May 28, 2023

teburd commented Jun 23, 2023

PerMac left a comment

PerMac commented Jul 18, 2023

nashif commented Jul 18, 2023

nashif commented Jul 18, 2023

github-actions bot commented Sep 17, 2023

github-actions bot commented Nov 18, 2023

nashif commented Dec 21, 2023

PerMac left a comment

nashif commented Dec 21, 2023

cfriedt commented Dec 24, 2023

nashif commented Jan 3, 2024

cfriedt commented Jan 8, 2024

nashif commented Jan 8, 2024

nashif commented Jan 8, 2024

cfriedt commented Jan 9, 2024

ztest: support runs suites and cases from shell #58374

ztest: support runs suites and cases from shell #58374

Conversation

nashif commented May 28, 2023 • edited Loading

yperess commented May 28, 2023

stephanosio May 28, 2023

Choose a reason for hiding this comment

nashif May 28, 2023

Choose a reason for hiding this comment

teburd commented Jun 23, 2023

PerMac left a comment

Choose a reason for hiding this comment

PerMac commented Jul 18, 2023

nashif commented Jul 18, 2023

nashif commented Jul 18, 2023

github-actions bot commented Sep 17, 2023

github-actions bot commented Nov 18, 2023

nashif commented Dec 21, 2023

PerMac left a comment

Choose a reason for hiding this comment

nashif commented Dec 21, 2023

cfriedt commented Dec 24, 2023

nashif commented Jan 3, 2024

cfriedt commented Jan 8, 2024

nashif commented Jan 8, 2024

nashif commented Jan 8, 2024

cfriedt commented Jan 9, 2024

nashif commented May 28, 2023 •

edited

Loading