Add support for test phases

[psss]

As discussed on the stakeholder meeting, there are use cases related to tracking individual results of test "phases". These can be created using one of the following ways:

• defining a beakerlib phase using rlPhaseStart and rlPhaseEnd
• explicitly calling rstrnt-report-result to submit a subresult

When run in Beaker, the rlPhaseEnd command called during a beakerlib test execution actually calls rlReport which uses BEAKERLIB_COMMAND_REPORT_RESULT command to report the result and, if I'm not mistaken, this defaults to using rstrnt-report-result. @sopos, please correct me if I'm wrong.

The effect is that each beakerlib phase and rstrnt-report-result call create a "subresult" which is shown in the Beaker web interface under the actual test. The proposal is to add a similar support for subresults to tmt.

Here are some more ideas caught from the brainstorm:

• We could inject a custom script using BEAKERLIB_COMMAND_REPORT_RESULT and perform necessary actions on the guest
• The result specification could be extended to include a list of subresults (which would be similar to the check attribute)
• It would be good to report detected avc failures for each test phase separately
• Report plugins would have to invent the proper mapping to e.g. junit format as the spec does not know subresults

All in all it seems this would bring a substantial level of complexity (currently the test granularity was simply defined by the test metadata definition for each test) so we should discuss this well before jumping into the implementation.

Base Work

• spec — Extend results schema to allow test phases #2933
• class — Propagate result phase into the report plugins #3094
• Rename the result phase to subresult #3106
• actual saving of subresults into results.yaml - Save the subresults for tmt-report-result #3200

Report Plugins

• display — Show subresults and their checks in report display plugin #3107
• html — Improve the design of the checks and phases in HTML report plugin #3095
• junit
  • Remove support for junit-xml, use Jinja2 instead #3150
  • Add support for subresults in junit report plugin (without flavor) #3307
  • Add support for subresults (experimental flavor) to junit report plugin #3177
  • Enable the huge_tree option for the lxml parser #3365
  • Do not save the logs into memory using an auxiliary variable #3382
• polarion — Use Jinja2 and polarion JUnit flavor to generate XUnit in polarion report plugin #3166 (not actually related to subresults)
• reportportal — Add subresults support for reportportal report plugin #3331
  • Multiple reportportal plugin improvements and fixes #3356
  • Fix log paths for subresults loaded from tmt-report-results.yaml #3370
  • Use the rhts-report-result alias for reporting beakerlib subresults #3372

Problems

• Special characters in beakerlib phase name breaks results #3333
• Not possible to use reportportal report plugin with custom results #3360
• Polarion XML fails to generate due to xmlSAX2Characters: huge text node #3363
• tmt-report-result: may need to escape special symbols like colon #3226

[juk]

@psss I quickly checked a junit format and it seems like there is infrastructure to support subtests/phases
classname - could be used for a parent test name
name - could be used for a subtest/phase

https://github.com/testmoapp/junitxml

[kkaarreell]

According to
https://www.ibm.com/docs/en/developer-for-zos/14.2?topic=formats-junit-xml-format
<testsuite> has child elements <testcase>, not <testsuite> again. We should be clear in which artifacts these phases should appear and how those artifacts are going to be used. Do we know if the proposed XML with nested <testsuite> elements would be compatible with ReportPortal for example?

[kkaarreell]

FYI, I just tried to import https://github.com/testmoapp/junitxml/blob/main/examples/junit-basic.xml to RP and it went well, creating nested structures for testsuites. So at least from the RP perspective we may be fine.

[lukaszachy]

Note: Beakerlib executes BEAKERLIB_COMMAND_REPORT_RESULT only if TESTID is set (https://github.com/beakerlib/beakerlib/blob/4321c54e2e99f74b91c9376aa4324dafd0300c3d/src/testing.sh#L1075) - we don't set it in tmt yet.

[seberm]

Just FYI, I was experimenting with a nested structure of <testsuite> using a junit-xml but the library does not support it. There are unmerged patches that add the support for nested <testsuite> structures (quickly tested - they seem to work fine), but the library is outdated and unsupported (the last commit was 4 years ago).

I have been searching for suitable alternative libraries, but so far, I haven't found any. Any ideas?

[thrix]

Doing a fallback to own implementation should not be that hard, note that there is a nice way via attr/cattrs we use in Testing Farm and @janhavlin was working on it and could help porting it to tmt:

https://gitlab.com/testing-farm/gluetool-modules/-/tree/main/gluetool_modules_framework/libs/results?ref_type=heads

[sopos]

Note breakers is able to produce the journal.xml in the breaker format as well as in the xunit. However, I was recently contacted by someone who was having an issue where the respective xml python module was missing which causes just an error and the xml file not being created.
It might be good if tmt was able to parse the journal.meta, i.e. to use the beakerlib's code to transform it (using the journalling.py).

[seberm]

Doing a fallback to own implementation should not be that hard, note that there is a nice way via attr/cattrs we use in Testing Farm and @janhavlin was working on it and could help porting it to tmt:

https://gitlab.com/testing-farm/gluetool-modules/-/tree/main/gluetool_modules_framework/libs/results?ref_type=heads

Regarding our implementation of the JUnit XML library, I was thinking that we might be able to directly use the junit_xml (approximately just 400-lines of code + unit tests).

We could apply the necessary patches for "test suite inside a test suite", clean up the code, remove Python2 support, and add the type annotations.

Then it should very easily integrate into the TMT. The license for junit_xml is the same as for TMT - the MIT license:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

We would likely be fine by simply adding the copyright notice to the library file header and referencing to the original project repository.

What are your thoughts on this? Do you think this could be a viable solution?