ztest: add a ztest shell command #65437

cfriedt · 2023-11-19T02:40:53Z

Add a ztest shell command so that select tests can be run via the shell. This is useful for e.g. systems where we do not necessarily want to run all tests immediately from main() or would prefer to only run certain tests without modifying the source or binary.

Since we are using getopt_long() under the hood, option arguments can be separated with = or .

The CLI is intentionally similar to that of gtest binaries. It might be nice to optionally clone the output format of gtest as well.
https://pastebin.com/GtBg0aHZ

ztest # run all tests
ztest --help
ztest --list_tests
ztest --filter POSITIVE_PATTERNS[-NEGATIVE_PATTERNS]
ztest --repeat=N
ztest --shuffle
ztest --random_seed 42
ztest --color=(yes|no|auto)
ztest --output=(json|xml)[:DIRECTORY/|:FILE_PATH]

west build -p auto -b qemu_x86_64 -t run tests/lib/c_lib/common -- -DCONFIG_ZTEST_SHELL=y -DCONFIG_ZTEST_MAIN=n
...
uart:~$ ztest
Running TESTSUITE libc_common
===================================================================
START - test_abort
abort()
E: RAX: 0x0000000000000004 RBX: 0x00000000001201e8 RCX: 0x0000000000000001 RDX: 0x0000000000000000
E: RSI: 0x0000000000127795 RDI: 0x0000000000147b60 RBP: 0x000000000017bf30 RSP: 0x000000000017bf30
E:  R8: 0x0000000000000000  R9: 0x0000000000000000 R10: 0x0000000000000000 R11: 0x0000000000000000
E: R12: 0x0000000000000000 R13: 0x00000000001201b0 R14: 0x0000000000000000 R15: 0x000000000010c63d
E: RSP: 0x000000000017bf30 RFLAGS: 0x0000000000000202 CS: 0x0018 CR3: 0x0000000000187000
E: RIP: 0x000000000010bf0a
E: >>> ZEPHYR FATAL ERROR 4: Kernel panic on CPU 0
E: Current thread: 0x133f60 (test_abort)
Caught system error -- reason 4 1
Fatal error expected as part of test case.
 PASS - test_abort in 0.004 seconds
===================================================================
START - test_abs
 PASS - test_abs in 0.001 seconds
===================================================================
START - test_atoi
 PASS - test_atoi in 0.001 seconds
===================================================================
START - test_bsearch
 PASS - test_bsearch in 0.001 seconds
===================================================================
START - test_checktype
 PASS - test_checktype in 0.001 seconds
===================================================================
START - test_exit
exit
 PASS - test_exit in 0.019 seconds
===================================================================
START - test_limits
 PASS - test_limits in 0.001 seconds
===================================================================
START - test_memchr
 PASS - test_memchr in 0.001 seconds
===================================================================
START - test_memcmp
 PASS - test_memcmp in 0.001 seconds
===================================================================
START - test_memcpy
 PASS - test_memcpy in 0.001 seconds
===================================================================
START - test_memmove
 PASS - test_memmove in 0.001 seconds
===================================================================
START - test_memset
 PASS - test_memset in 0.001 seconds
===================================================================
START - test_qsort
 PASS - test_qsort in 0.001 seconds
===================================================================
START - test_qsort_r
 PASS - test_qsort_r in 0.001 seconds
===================================================================
START - test_rand
 SKIP - test_rand in 0.001 seconds
===================================================================
START - test_rand_reproducibility
 SKIP - test_rand_reproducibility in 0.001 seconds
===================================================================
START - test_sqrt
 SKIP - test_sqrt in 0.001 seconds
===================================================================
START - test_sqrtf
 SKIP - test_sqrtf in 0.001 seconds
===================================================================
START - test_srand
 SKIP - test_srand in 0.001 seconds
===================================================================
START - test_ssize_t
 PASS - test_ssize_t in 0.001 seconds
===================================================================
START - test_stdbool
 PASS - test_stdbool in 0.001 seconds
===================================================================
START - test_stddef
 PASS - test_stddef in 0.001 seconds
===================================================================
START - test_stdint
 PASS - test_stdint in 0.001 seconds
===================================================================
START - test_str_operate
 PASS - test_str_operate in 0.001 seconds
===================================================================
START - test_strchr
 PASS - test_strchr in 0.001 seconds
===================================================================
START - test_strcmp
 PASS - test_strcmp in 0.001 seconds
===================================================================
START - test_strcpy
 PASS - test_strcpy in 0.001 seconds
===================================================================
START - test_strlen
 PASS - test_strlen in 0.001 seconds
===================================================================
START - test_strncmp
 PASS - test_strncmp in 0.001 seconds
===================================================================
START - test_strncpy
 PASS - test_strncpy in 0.001 seconds
===================================================================
START - test_strtok_r
 PASS - test_strtok_r in 0.001 seconds
===================================================================
START - test_strtol
 PASS - test_strtol in 0.001 seconds
===================================================================
START - test_strtoul
 PASS - test_strtoul in 0.001 seconds
===================================================================
START - test_strxspn
 PASS - test_strxspn in 0.001 seconds
===================================================================
START - test_time
 PASS - test_time in 0.001 seconds
===================================================================
START - test_time_t
 PASS - test_time_t in 0.001 seconds
===================================================================
START - test_tolower_toupper
 PASS - test_tolower_toupper in 0.001 seconds
===================================================================
TESTSUITE libc_common succeeded

------ TESTSUITE SUMMARY START ------

SUITE PASS - 100.00% [libc_common]: pass = 32, fail = 0, skip = 5, total = 37 duration = 0.058 seconds
 - PASS - [libc_common.test_abort] duration = 0.004 seconds
 - PASS - [libc_common.test_abs] duration = 0.001 seconds
 - PASS - [libc_common.test_atoi] duration = 0.001 seconds
 - PASS - [libc_common.test_bsearch] duration = 0.001 seconds
 - PASS - [libc_common.test_checktype] duration = 0.001 seconds
 - PASS - [libc_common.test_exit] duration = 0.019 seconds
 - PASS - [libc_common.test_limits] duration = 0.001 seconds
 - PASS - [libc_common.test_memchr] duration = 0.001 seconds
 - PASS - [libc_common.test_memcmp] duration = 0.001 seconds
 - PASS - [libc_common.test_memcpy] duration = 0.001 seconds
 - PASS - [libc_common.test_memmove] duration = 0.001 seconds
 - PASS - [libc_common.test_memset] duration = 0.001 seconds
 - PASS - [libc_common.test_qsort] duration = 0.001 seconds
 - PASS - [libc_common.test_qsort_r] duration = 0.001 seconds
 - SKIP - [libc_common.test_rand] duration = 0.001 seconds
 - SKIP - [libc_common.test_rand_reproducibility] duration = 0.001 seconds
 - SKIP - [libc_common.test_sqrt] duration = 0.001 seconds
 - SKIP - [libc_common.test_sqrtf] duration = 0.001 seconds
 - SKIP - [libc_common.test_srand] duration = 0.001 seconds
 - PASS - [libc_common.test_ssize_t] duration = 0.001 seconds
 - PASS - [libc_common.test_stdbool] duration = 0.001 seconds
 - PASS - [libc_common.test_stddef] duration = 0.001 seconds
 - PASS - [libc_common.test_stdint] duration = 0.001 seconds
 - PASS - [libc_common.test_str_operate] duration = 0.001 seconds
 - PASS - [libc_common.test_strchr] duration = 0.001 seconds
 - PASS - [libc_common.test_strcmp] duration = 0.001 seconds
 - PASS - [libc_common.test_strcpy] duration = 0.001 seconds
 - PASS - [libc_common.test_strlen] duration = 0.001 seconds
 - PASS - [libc_common.test_strncmp] duration = 0.001 seconds
 - PASS - [libc_common.test_strncpy] duration = 0.001 seconds
 - PASS - [libc_common.test_strtok_r] duration = 0.001 seconds
 - PASS - [libc_common.test_strtol] duration = 0.001 seconds
 - PASS - [libc_common.test_strtoul] duration = 0.001 seconds
 - PASS - [libc_common.test_strxspn] duration = 0.001 seconds
 - PASS - [libc_common.test_time] duration = 0.001 seconds
 - PASS - [libc_common.test_time_t] duration = 0.001 seconds
 - PASS - [libc_common.test_tolower_toupper] duration = 0.001 seconds

------ TESTSUITE SUMMARY END ------

===================================================================
PROJECT EXECUTION SUCCESSFUL
uart:~$

uart:~$ ztest --list_tests
libc_common.test_abort
libc_common.test_abs
libc_common.test_atoi
libc_common.test_bsearch
libc_common.test_checktype
libc_common.test_exit
libc_common.test_limits
libc_common.test_memchr
libc_common.test_memcmp
libc_common.test_memcpy
libc_common.test_memmove
libc_common.test_memset
libc_common.test_qsort
libc_common.test_qsort_r
libc_common.test_rand
libc_common.test_rand_reproducibility
libc_common.test_sqrt
libc_common.test_sqrtf
libc_common.test_srand
libc_common.test_ssize_t
libc_common.test_stdbool
libc_common.test_stddef
libc_common.test_stdint
libc_common.test_str_operate
libc_common.test_strchr
libc_common.test_strcmp
libc_common.test_strcpy
libc_common.test_strlen
libc_common.test_strncmp
libc_common.test_strncpy
libc_common.test_strtok_r
libc_common.test_strtol
libc_common.test_strtoul
libc_common.test_strxspn
libc_common.test_time
libc_common.test_time_t
libc_common.test_tolower_toupper

kartben · 2023-11-19T02:51:07Z

Duplicate of #58374?

cfriedt · 2023-11-19T02:56:08Z

Duplicate of #58374?

Oh wow - I didn't even notice that. I guess it is, although I kind of like this one better tbh ;-)

cfriedt · 2023-11-20T11:04:29Z

Opening this up. I could maybe add filtering capabilities in this PR. Maybe json / xml output to the console?
Is it safe to assume other features can be added later?

cfriedt · 2023-11-22T02:49:27Z

added testsuite for "filter" functionality

For details, see https://github.com/google/googletest/blob/main/googletest/test/googletest-filter-unittest.py

cfriedt · 2023-11-22T03:08:19Z

cfriedt · 2023-11-22T03:31:30Z

Unable to repro CI failures locally.

twister -i -p qemu_arc_em -p qemu_arc_hs -p qemu_arc_hs5x -p qemu_arc_hs6x -p qemu_arc_hs_xip -p qemu_cortex_a9 -p qemu_cortex_m3 -p qemu_cortex_r5 -p qemu_nios2 -p qemu_riscv32e -T tests/ztest/shell/
...
INFO    - Total complete:   10/  10  100%  skipped:    1, failed:    0, error:    0
INFO    - 1 test scenarios (10 test instances) selected, 1 configurations skipped (0 by static filter, 1 at runtime).
INFO    - 9 of 10 test configurations passed (100.00%), 0 failed, 0 errored, 1 skipped with 0 warnings in 17.62 seconds
INFO    - In total 117 test cases were executed, 12 skipped on 10 out of total 644 platforms (1.55%)
INFO    - 9 test configurations executed on platforms, 0 test configurations were only built.
INFO    - Saving reports...
INFO    - Writing JSON report /home/cfriedt/zephyrproject/zephyr/twister-out/twister.json
INFO    - Writing xunit report /home/cfriedt/zephyrproject/zephyr/twister-out/twister.xml...
INFO    - Writing xunit report /home/cfriedt/zephyrproject/zephyr/twister-out/twister_report.xml...
INFO    - Run completed

Edit: until I rebased on main ... locally this is working fine based off of 3b5fd5f .

jgl-meta

One small issue. I don't see anything other show stoppers other than failing tests.

subsys/testsuite/ztest/src/ztest_shell.c

cfriedt · 2023-12-16T22:49:14Z

Will unfortunately need to exclude the esp32s2 / xtensa variant because they have some conflicting headers and it looks as though the include paths are ordered such that the broken version is included prior to the Zephyr version. It's possible that the headers would be interchangeable, but __BEGIN_DECLS and __END_DECLS are not defined in the Zephyr build by default.

cfriedt · 2023-12-16T23:21:28Z

cfriedt · 2023-12-17T02:44:01Z

Fixed

cfriedt · 2023-12-17T16:12:16Z

Updated some comments / warnings / error messages.

I was looking at adding shuffle and repeat support (both are actually pretty trivial), but it will require refactoring other ztest parts.

Probably ok to defer on that.

cfriedt · 2023-12-17T17:11:43Z

@yperess - PTAL. Should I keep working on this? Does it duplicate what Google is doing ATM? We will likely be using this internally until there is an equivalent solution upstream (i.e. shell command with the same arguments - and ideally output - as gtest).

ycsin

LGTM

subsys/testsuite/ztest/src/ztest_shell.c

cfriedt · 2023-12-18T16:05:42Z

only compile test_main() when CONFIG_ZTEST_MAIN=y, since it uses deprecated symbols

subsys/testsuite/ztest/src/ztest_shell.c

Add a ztest shell command so that select tests can be run via the shell. This is useful for e.g. systems where we do not necessarily want to run all tests immediately from main() or would prefer to only run certain tests without modifying the source or binary. Since we are using getopt_long() under the hood, option arguments can be separated with '=' or space ' '. E.g. ztest # run all tests ztest --help ztest --ztest_list_tests ztest --ztest_filter POSITIVE_PATTERNS[-NEGATIVE_PATTERNS] ztest --ztest_repeat=N ztest --ztest_shuffle ztest --ztest_random_seed 42 ztest --ztest_color=(yes|no|auto) ztest --ztest_output=(json|xml)[:DIRECTORY/|:FILE_PATH] Signed-off-by: Christopher Friedt <cfriedt@meta.com>

Add a testsuite for the ztest shell. In particular, ensure that all filter functionality works as expected per Google Test Framework criteria. Test inputs were adapted from googletest-filter-unittest.py in https://github.com/google/googletest. Signed-off-by: Christopher Friedt <cfriedt@meta.com>

nashif

This seems to be a gTest compat mode rather than a natve shell implementation for ztest as can be demonstrated in #58374. I know is a bit stale, but I have an updated version based on the initial feedback. gTest compatibility mode can always be added on top, but IMO leveraging shell features and exposing ztest features whether in gtest or not should be the priority and gtest support should be a mode rather than the main feature.

Also, many of the options listed are not supported yet, I suggest to only list and document what is working:

E.g.
ztest # run all tests
ztest --help
ztest --ztest_list_tests
ztest --ztest_filter POSITIVE_PATTERNS[-NEGATIVE_PATTERNS]
ztest --ztest_repeat=N
ztest --ztest_shuffle
ztest --ztest_random_seed 42
ztest --ztest_color=(yes|no|auto)
ztest --ztest_output=(json|xml)[:DIRECTORY/|:FILE_PATH]

I ran a few tests and getting lots of issues, i.e.

uart:~$ ztest --filter=+threads_lifecycle.test_thread_join
Running TESTSUITE threads_lifecycle
===================================================================
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
TESTSUITE threads_lifecycle succeeded
Running TESTSUITE threads_lifecycle_1cpu
===================================================================
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32
TESTSUITE threads_lifecycle_1cpu succeeded

------ TESTSUITE SUMMARY START ------

SUITE SKIP -   0.00% [threads_lifecycle]: pass = 0, fail = 0, skip = 29, total = 29 duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_abort_from_isr] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_abort_from_isr_not_self] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_essential_thread_abort] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_essential_thread_operation] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_k_busy_wait] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_k_thread_stack_space_get_user] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_resume_unsuspend_thread] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_systhreads_idle] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_systhreads_main] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_thread_join] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_thread_join_deadlock] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_thread_join_isr] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_thread_name_get_set] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_thread_name_user_get_set] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_thread_runtime_stats_get] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_thread_start] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_thread_start_user] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_thread_timeout_remaining_expires] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_abort_others] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_abort_self] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_priority_set] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_spawn_delay] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_spawn_forever] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_spawn_params] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_spawn_priority] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_suspend] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_suspend_resume_preemptible] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_threads_suspend_timeout] duration = 0.000 seconds
 - SKIP - [threads_lifecycle.test_user_mode] duration = 0.000 seconds

SUITE SKIP -   0.00% [threads_lifecycle_1cpu]: pass = 0, fail = 0, skip = 12, total = 12 duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_customdata_get_set_coop] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_customdata_get_set_preempt] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_delayed_thread_abort] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_k_busy_wait_user] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_k_thread_foreach] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_k_thread_foreach_null_cb] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_k_thread_foreach_unlocked] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_k_thread_foreach_unlocked_null_cb] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_k_thread_state_str] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_threads_abort_repeat] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_threads_cpu_mask] duration = 0.000 seconds
 - SKIP - [threads_lifecycle_1cpu.test_threads_suspend_resume_cooperative] duration = 0.000 seconds

------ TESTSUITE SUMMARY END ------

Need to spend some more time on this. Will also update my PR so we can see how we can converge.

nashif · 2023-12-19T01:19:08Z

subsys/testsuite/ztest/src/ztest_new.c

@@ -62,6 +62,13 @@ static void end_report(void)
 	}
 }

+#ifdef CONFIG_ZTEST_SHELL


hmm, ztest_new.c should not be there, must have forgotten to remove it when old ztest was removed, so not sure about all those changes and why they were needed, strange.

I was just trying to be consistent. But yea was kind of surprised to see it still here. For anyone using tagged v3.5.0, it's still the active version for CONFIG_ZTEST_NEW

nashif · 2023-12-19T01:20:22Z

subsys/testsuite/ztest/Kconfig

@@ -181,6 +181,58 @@ config TEST_LOGGING_FLUSH_AFTER_TEST
 	default y
 	depends on MULTITHREADING

+config ZTEST_MAIN


why do we need this? Either you run with shell, then you jump into shell and run the tests from there, or run the tests without shell and just execute the test as usual. Do not see the value of this.

It is to prevent ZTest from running by default immediately. Some systems need additional application-time setup before you can start executing tests.

nashif · 2023-12-19T01:23:43Z

subsys/testsuite/ztest/Kconfig

+	# currently this is broken due to getopt_state / conflicting native definitions on POSIX arch
+	depends on !ARCH_POSIX
+	help
+	  Provide a shell command to run ZTests from the Zephyr shell. The arguments of the 'ztest'


ztest and the zephyr shell can do more than what gtest does or provides, so this is rather limiting, i.e. I would like to see feature of the shell being used, such as sub commands, auto-completion etc instead of depending on command lines for everything.
Compatibility mode with gtest is fine, but not as the main shell implementation.

Subcommands are fine for a specific class of command. Here, flags make a bit more sense. Simply because subcommands exist is not justification to use them everywhere.

nashif · 2023-12-19T01:24:57Z

subsys/testsuite/ztest/Kconfig

+	string "Prefix for ztest shell options"
+	default ""
+	help
+	  This option is for compatibility with Google Test Framework, where each option specific to


main motivation here seems to be compatibility with gtest instead of providing full shell support for tests, which is very limiting. And then, most of those options insipred by gtest are not implemented, so basically we go back what can be done already as demonstrated in #58374

See earlier comment #65437 (comment)

Additional features were pretty trivial to implement, but require refactorization.

cfriedt · 2023-12-19T02:17:41Z

This seems to be a gTest compat mode rather than a natve shell implementation for ztest as can be demonstrated in #58374.

Well, it is a native shell implementation.. Many of the existing Kconfig options for ZTest today mirror gtest options and terminology, but there are also only so many ways to skin a ZTest.

gTest compatibility mode can always be added on top

That is true. It would be quite good if the subsys were capable of supporting arbitrary different frontends, like many of the existing Zephyr subsystems. A Gtest-like shell cmd is only one possible frontend.

IMO leveraging shell features and exposing ztest features whether in gtest or not should be the priority and gtest support should be a mode rather than the main feature.

Of course. gtest compatibility is a nice feature for myself and many others from Meta (I would guess Google as well, and others elsewhere).

Also, many of the options listed are not supported yet

As mentioned earlier in the comments, it's quite trivial to refactor the code that is there so that multiple test runners can use the same implementation. I already have repeat, random_seed, and shuffle working, but wanted to wait until there was some movement here.

I ran a few tests and getting lots of issues, i.e.

Warning: CONFIG_ZTEST_SHELL_PATTERN_MAX is not large enough for
Warning: +threads_lifecycle.test_thread_join
Warning: Needed: 36
Warning: Actual: 32

I would suggest increasing CONFIG_ZTEST_SHELL_PATTERN_MAX by 4 in your case. It could really be set to any amount though and the storage could even be stored inside the state object.

Need to spend some more time on this. Will also update my PR so we can see how we can converge.

Sure - that would be great.

cfriedt · 2024-01-10T15:17:25Z

Just putting this into draft mode. It might need a rework

kartben · 2024-01-10T15:24:54Z

Just putting this into draft mode. It might need a rework

just mentioning in case you missed it but #58374 has now been merged

cfriedt · 2024-01-13T04:08:00Z

Just putting this into draft mode. It might need a rework

just mentioning in case you missed it but #58374 has now been merged

Yes - there are some features that exist here that do not exist there

nashif · 2024-01-15T19:20:39Z

Yes - there are some features that exist here that do not exist there

which?

cfriedt · 2024-01-16T12:50:05Z

Yes - there are some features that exist here that do not exist there

which?

I think filtering was one of them

cfriedt force-pushed the ztest-cmd branch 3 times, most recently from 5d39409 to a532da6 Compare November 19, 2023 03:22

cfriedt requested review from yperess and nashif November 19, 2023 03:23

cfriedt added area: Test Framework Issues related not to a particular test, but to the framework instead area: Shell Shell subsystem labels Nov 19, 2023

cfriedt marked this pull request as ready for review November 20, 2023 11:03

zephyrbot added the area: Testsuite Testsuite label Nov 20, 2023

zephyrbot requested review from aaronemassey, asemjonovs and jeremybettis November 20, 2023 11:03

zephyrbot assigned nashif Nov 20, 2023

cfriedt force-pushed the ztest-cmd branch 4 times, most recently from 888ed0a to 0231484 Compare November 22, 2023 02:48

cfriedt requested a review from fabiobaltieri November 22, 2023 03:23

cfriedt requested review from jgl-meta and ycsin November 27, 2023 16:28

jgl-meta previously approved these changes Nov 28, 2023

View reviewed changes

jgl-meta reviewed Nov 28, 2023

View reviewed changes

subsys/testsuite/ztest/src/ztest_shell.c Outdated Show resolved Hide resolved

cfriedt mentioned this pull request Dec 2, 2023

Explicitly allow C++ in tests #66053

Closed

cfriedt dismissed jgl-meta’s stale review via 2aa4a3c December 15, 2023 19:30

cfriedt force-pushed the ztest-cmd branch from d8348d7 to 6a015ee Compare December 16, 2023 23:18

cfriedt force-pushed the ztest-cmd branch from 6a015ee to e4219e1 Compare December 17, 2023 14:46

ycsin previously approved these changes Dec 18, 2023

View reviewed changes

subsys/testsuite/ztest/src/ztest_shell.c Outdated Show resolved Hide resolved

cfriedt dismissed ycsin’s stale review via b2fb4c7 December 18, 2023 15:34

cfriedt force-pushed the ztest-cmd branch 2 times, most recently from b2fb4c7 to 65f6cf3 Compare December 18, 2023 16:05

fabiobaltieri reviewed Dec 18, 2023

View reviewed changes

subsys/testsuite/ztest/src/ztest_shell.c Show resolved Hide resolved

subsys/testsuite/ztest/src/ztest_shell.c Outdated Show resolved Hide resolved

subsys/testsuite/ztest/src/ztest_shell.c Outdated Show resolved Hide resolved

fabiobaltieri previously approved these changes Dec 18, 2023

View reviewed changes

cfriedt added 2 commits December 18, 2023 14:19

cfriedt dismissed fabiobaltieri’s stale review via 4d3b79b December 18, 2023 19:22

cfriedt force-pushed the ztest-cmd branch from 65f6cf3 to 4d3b79b Compare December 18, 2023 19:22

cfriedt requested review from fabiobaltieri and ycsin December 18, 2023 19:22

fabiobaltieri approved these changes Dec 18, 2023

View reviewed changes

nashif requested changes Dec 19, 2023

View reviewed changes

cfriedt marked this pull request as draft January 10, 2024 15:17

cfriedt closed this Mar 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ztest: add a ztest shell command #65437

ztest: add a ztest shell command #65437

cfriedt commented Nov 19, 2023 •

edited

Loading

kartben commented Nov 19, 2023

cfriedt commented Nov 19, 2023 •

edited

Loading

cfriedt commented Nov 20, 2023 •

edited

Loading

cfriedt commented Nov 22, 2023

cfriedt commented Nov 22, 2023 •

edited

Loading

cfriedt commented Nov 22, 2023 •

edited

Loading

jgl-meta left a comment

cfriedt commented Dec 16, 2023 •

edited

Loading

cfriedt commented Dec 16, 2023

cfriedt commented Dec 17, 2023

cfriedt commented Dec 17, 2023 •

edited

Loading

cfriedt commented Dec 17, 2023

ycsin left a comment

cfriedt commented Dec 18, 2023

nashif left a comment

nashif Dec 19, 2023

cfriedt Dec 19, 2023 •

edited

Loading

nashif Dec 19, 2023

cfriedt Dec 19, 2023 •

edited

Loading

nashif Dec 19, 2023

cfriedt Dec 19, 2023 •

edited

Loading

nashif Dec 19, 2023

cfriedt Dec 19, 2023 •

edited

Loading

cfriedt commented Dec 19, 2023 •

edited

Loading

cfriedt commented Jan 10, 2024

kartben commented Jan 10, 2024

cfriedt commented Jan 13, 2024

nashif commented Jan 15, 2024

cfriedt commented Jan 16, 2024

ztest: add a ztest shell command #65437

ztest: add a ztest shell command #65437

Conversation

cfriedt commented Nov 19, 2023 • edited Loading

kartben commented Nov 19, 2023

cfriedt commented Nov 19, 2023 • edited Loading

cfriedt commented Nov 20, 2023 • edited Loading

cfriedt commented Nov 22, 2023

cfriedt commented Nov 22, 2023 • edited Loading

cfriedt commented Nov 22, 2023 • edited Loading

jgl-meta left a comment

Choose a reason for hiding this comment

cfriedt commented Dec 16, 2023 • edited Loading

cfriedt commented Dec 16, 2023

cfriedt commented Dec 17, 2023

cfriedt commented Dec 17, 2023 • edited Loading

cfriedt commented Dec 17, 2023

ycsin left a comment

Choose a reason for hiding this comment

cfriedt commented Dec 18, 2023

nashif left a comment

Choose a reason for hiding this comment

nashif Dec 19, 2023

Choose a reason for hiding this comment

cfriedt Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

nashif Dec 19, 2023

Choose a reason for hiding this comment

cfriedt Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

nashif Dec 19, 2023

Choose a reason for hiding this comment

cfriedt Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

nashif Dec 19, 2023

Choose a reason for hiding this comment

cfriedt Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

cfriedt commented Dec 19, 2023 • edited Loading

cfriedt commented Jan 10, 2024

kartben commented Jan 10, 2024

cfriedt commented Jan 13, 2024

nashif commented Jan 15, 2024

cfriedt commented Jan 16, 2024

cfriedt commented Nov 19, 2023 •

edited

Loading

cfriedt commented Nov 19, 2023 •

edited

Loading

cfriedt commented Nov 20, 2023 •

edited

Loading

cfriedt commented Nov 22, 2023 •

edited

Loading

cfriedt commented Nov 22, 2023 •

edited

Loading

cfriedt commented Dec 16, 2023 •

edited

Loading

cfriedt commented Dec 17, 2023 •

edited

Loading

cfriedt Dec 19, 2023 •

edited

Loading

cfriedt Dec 19, 2023 •

edited

Loading

cfriedt Dec 19, 2023 •

edited

Loading

cfriedt Dec 19, 2023 •

edited

Loading

cfriedt commented Dec 19, 2023 •

edited

Loading