Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: add report for tests results (Markdown format) #5879

Open
lcheylus opened this issue Jan 25, 2024 · 1 comment
Open

CI: add report for tests results (Markdown format) #5879

lcheylus opened this issue Jan 25, 2024 · 1 comment

Comments

@lcheylus
Copy link
Contributor

lcheylus commented Jan 25, 2024

GitHub workflow allows to add a Job summary with Markdown format:

In GH workflows, tests are runned with cargo-nextest tool (instead of cargo test). With version 0.9.66 and above, it adds an experimental support for producing machine-readable output for test runs, in a format similar to cargo test's libtest JSON output => https://nexte.st/book/run-machine-readable.html

I tried to use this feature to generate file results for tests (JSON format) and to use them to generate summaries after end of the tests in FreeBSD workflow (could also be added in other workflows including tests).

  • In "Tests" job, use cargo-nextest to generate results files (JSON):
NEXTEST_EXPERIMENTAL_LIBTEST_JSON=1 cargo nextest run --hide-progress-bar --profile ci --features "${FEATURES}" --message-format libtest-json-plus 1> "${WORKSPACE}/tests-unix.json"

NEXTEST_EXPERIMENTAL_LIBTEST_JSON=1 cargo nextest run --hide-progress-bar --profile ci --all-features -p uucore --message-format libtest-json-plus 1> "${WORKSPACE}/tests-uucore.json"
  • JSON files are copied back from VM to runner/host (copyback: true parameter for the FreeBSD VM action).

  • Add a new step at the end of "Tests" job to parse JSON files and generate summaries (Markdown format)

You could see an example of these summaries in my own repository ("ugly" main branch with a modified FreeBSD workflow, issue #5797 for details) => https://github.com/lcheylus/rust-coreutils/actions/runs/7653519450

I need to do additional tests when some tests are NOK/FAILED and include them in the summary.

@lcheylus
Copy link
Contributor Author

lcheylus commented Jan 26, 2024

I added case in FreeBSD CI/workflow when tests failed : the summary (Markdown format) displays a specific status for tests failure and the list of failed tests.

You can see an example of such a case with my OpenBSD workflow (WIP) : https://github.com/lcheylus/rust-coreutils/actions/runs/7669296982

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant