Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Result attribute of Test-Suite element in Nunit output should be 'Failed' instead of 'Failure' for failed tests #1083

Closed
tehsuk opened this issue Jul 26, 2018 · 5 comments

Comments

@tehsuk
Copy link

tehsuk commented Jul 26, 2018

2. Describe Your Environment

Operating System, Pester version and PowerShell version:

Pester version : 4.4.0
PowerShell version : 5.1.14409.1012
OS version : Microsoft Windows NT 6.1.7601 Service Pack 1

3. Expected Behavior

Based on https://github.com/nunit/docs/wiki/Test-Result-XML-Format , Nunit formatted XML output should have result="Failed" for test-suite element for failed tests.

4.Current Behavior

Nunit output has result="Failure" for test-suite element for failed tests.

<test-suite type="TestFixture" name="test name" executed="True" result="Failure" success="False" time="44.9173" asserts="0" description="Local User: ProgAdminUser">

5. Possible Solution

Change XML to return 'Failed' instead of 'Failure'

6. Context

I am importing Nunit-formatted XML files generated by Pester into the Xray plugin for Jira. Failed tests are showing up as 'ToDo' instead of 'Failed' because of the incorrect failure string that Pester is generating.

If i replace all instances of result="Failure" with result="Failed" in the pester-generated nunit xml, the tests show up properly flagged as failed.

@it-praktyk
Copy link
Contributor

@tehsuk, thank you for the report. Please test the updated code - PR #1085.

@nohwnd
Copy link
Member

nohwnd commented Aug 1, 2018

My problem with this is that you are referencing nunit 3.0 documentation and we are using nunit 2.5 schema. And even the linked documentation fails to be consistent with the naming (see result and label):

result The basic result of the test. May be Passed, Failed, Inconclusive or Skipped.
label ... Currently, NUnit may use the following labels in combination with the Failure result: Error, Cancelled or Invalid. <- here they refer to Failure result, but no such result is present in the list of possible results.

I cannot find a description of how the correct output should look like in 2.5. (Can you help me find it?) And our output is correctly validated against the official nunit 2.5 xml schema, so I would be careful changing it to accomodate a single plugin.

@tehsuk
Copy link
Author

tehsuk commented Aug 2, 2018

I poked around in the 2.5 docs and source and could not find how the output should look in 2.5. the 2.5 xsd that comes with Pester appears to use Failure instead of Failed.

After closer inspection of the Xray plugin docs it appears it supports nUnit 2.6 and later.

Oddly enough the nUnit 2.6 example XML in the Xray support docs reference Failure instead of Failed for failed test suites.

https://confluence.xpand-addons.com/display/XRAY/Taking+advantage+of+NUnit+XML+reports#TakingadvantageofNUnitXMLreports-ImportingNUnitXMLreports

If Pester moves to exporting its nUnit execution reports to v3 of the schema it would be a good idea to get clarification on which failure output is valid. In the meantime you can close this out.

@nohwnd
Copy link
Member

nohwnd commented Aug 5, 2018

Maybe you can just ask in their repository, and we can keep this open till then.

@nohwnd
Copy link
Member

nohwnd commented Dec 16, 2018

Should be solved in related issue. Closing this. Thanks for the anlysis.

@nohwnd nohwnd closed this as completed Dec 16, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants