Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run the report step upon failures as well #3466

Merged
merged 2 commits into from
Jan 14, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/releases.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ a string to a list of strings, to better accommodate multiple notes.
The ``Node`` alias for the ``Core`` class has been dropped as it
has been deprecated a long time ago.

Previously when the test run was interrupted in the middle of the
test execution the :ref:`/spec/plans/report` step would be skipped
and no results would be reported. Now the report step is performed
always so that users can access results of those tests which were
successfully executed.


tmt-1.40.0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
1 change: 1 addition & 0 deletions tests/report/interrupted/data/.fmf/version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1
23 changes: 23 additions & 0 deletions tests/report/interrupted/data/main.fmf
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/plan:
discover:
how: fmf
provision:
how: virtual
execute:
how: tmt
report:
how: html

/test:
/good:
order: 0
test: exit 0
/bad:
order: 1
test: exit 1
/weird:
order: 2
test: rm -f /bin/bash
/missed:
order: 3
test: /bin/true
8 changes: 8 additions & 0 deletions tests/report/interrupted/main.fmf
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
summary: Make sure that report is performed on error
description:
If the run is interrupted in the middle of the test execution,
the report step should still be performed in order to provide
at least partial test results to the user.
tag+:
- provision-only
- provision-virtual
29 changes: 29 additions & 0 deletions tests/report/interrupted/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/bin/bash
. /usr/share/beakerlib/beakerlib.sh || exit 1

rlJournalStart
rlPhaseStartSetup
rlRun "run=\$(mktemp -d)" 0 "Create a run directory"
rlRun "pushd data"
rlPhaseEnd

rlPhaseStartTest
rlRun -s "tmt run -vvv --id $run" 2

# Tests before the breakage are executed as expected
rlAssertGrep "pass /test/good" $rlRun_LOG
rlAssertGrep "fail /test/bad" $rlRun_LOG

# Report is generated
rlAssertGrep "^\s*report\s*$" $rlRun_LOG
rlAssertGrep "how: html" $rlRun_LOG
rlAssertGrep "output: /.*/plan/report/default-0/index.html" $rlRun_LOG
rlAssertGrep "summary: 1 test passed and 1 test failed" $rlRun_LOG
rlPhaseEnd

rlPhaseStartCleanup
rlRun "popd"
rlGetTestState || rlFileSubmit "$run/log.txt"
rlRun "rm -r $run" 0 "Remove the run directory"
rlPhaseEnd
rlJournalEnd
9 changes: 6 additions & 3 deletions tmt/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -2446,10 +2446,13 @@ def go(self) -> None:
# Source the plan environment file after prepare and execute step
if isinstance(step, (tmt.steps.prepare.Prepare, tmt.steps.execute.Execute)):
self._source_plan_environment_file()
# Make sure we run 'finish' step always if enabled
# Make sure we run 'report' and 'finish' steps always if enabled
finally:
if not abort and self.finish.enabled:
self.finish.go()
if not abort:
if self.report.enabled and self.report.status() != "done":
self.report.go()
if self.finish.enabled:
self.finish.go()

def _export(
self,
Expand Down
Loading