NUnit3 result format output as nunit2 reports test dll as ignored if one or more child tests marked as ignored

[rprouse]

@CharliePoole commented on Fri Sep 02 2016

@harrisonmeister commented on Fri Sep 02 2016

Hi,

I'm not sure if this is by design or an issue, so please feel free to close if by design.

I am running a test project using the NUNit3 console.exe, and I receive the following output:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!--This file represents the results of running a test suite-->
<test-results name="D:\PATH_TO_LIBRARY\Some.Library.dll" total="2" errors="0" failures="0" not-run="1" inconclusive="0" ignored="1" skipped="0" invalid="0" date="2016-09-02" time="13:20:50">
  <environment nunit-version="3.4.1.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 10.0.10586.0" platform="Win32NT" cwd="D:\Git\Source" machine-name="REDACTEED" user="REDACTED" user-domain="REDACTED" />
  <culture-info current-culture="en-GB" current-uiculture="en-GB" />
  <test-suite type="Assembly" name="D:\PATH_TO_LIBRARY\Some.Library.dll" executed="False" result="Ignored">
    <properties>
      <property name="_PID" value="19472" />
      <property name="_APPDOMAIN" value="test-domain-" />
    </properties>
    <reason>
      <message><![CDATA[One or more child tests were ignored]]></message>
    </reason>
    <results>
      <test-suite type="TestSuite" name="A" executed="False" result="Ignored">
        <reason>
          <message><![CDATA[One or more child tests were ignored]]></message>
        </reason>
        <results>
          <test-suite type="TestSuite" name="B" executed="False" result="Ignored">
            <reason>
              <message><![CDATA[One or more child tests were ignored]]></message>
            </reason>
            <results>
              <test-suite type="TestSuite" name="C" executed="False" result="Ignored">
                <reason>
                  <message><![CDATA[One or more child tests were ignored]]></message>
                </reason>
                <results>
                  <test-suite type="TestFixture" name="D" executed="False" result="Ignored">
                    <reason>
                      <message><![CDATA[One or more child tests were ignored]]></message>
                    </reason>
                    <results>
                      <test-case name="A.B.C.D.Ignored_Test" executed="False" result="Ignored">
                        <properties>
                          <property name="_SKIPREASON" value="Ignored for some reason" />
                        </properties>
                        <reason>
                          <message><![CDATA[Ignored for some reason]]></message>
                        </reason>
                      </test-case>
                      <test-case name="A.B.C.D.Test_That_Works" executed="True" result="Success" success="True" time="0.027" asserts="0">
                        <reason>
                          <message><![CDATA[]]></message>
                        </reason>
                      </test-case>
                    </results>
                  </test-suite>
                </results>
              </test-suite>
            </results>
          </test-suite>
        </results>
      </test-suite>
    </results>
  </test-suite>
</test-results>

This marks the library test result as Ignored and not executed, even though one of the tests has run.

Specifically I have code which merged test result summaries (based on the nunit2 format). As NUnit2 (2.6.4) always added "time" and "asserts" as attributes, I'm just trying to work out if the result I am seeing in NUnit 3 is expected or not.

[TestFixture]
    public class ApisEstaCalculatorTests
    {
        [SetUp]
        public void SetUp()
        {

        }

        [Test]
        public void Test_That_Works()
        {
            Assert.Pass();
        }

        [Test, Ignore("Ignored for some reason")]
        public void Ignored_Test()
        {

        }
    }

Above is the sample code, and here is the associated console I am running
D:\packages\NUnit.ConsoleRunner\tools\nunit3-console.exe "--noheader" "--where=!(cat == 'Integration' || cat == 'IntegrationTest' || cat == 'IntegrationsTest' || cat == 'IntegrationTests' || cat == 'IntegrationsTests' || cat == 'Integration Test' || cat == 'Integration Tests' || cat == 'Integrations Tests' || cat == 'Approval Tests' || cat == 'AcceptanceTest' || cat == 'PerformanceTest')" "--labels=Off" "--timeout=900000" "--stoponerror" "--result=D:\TestResult-unit-64-0.xml;format=nunit2" "D:\PATH_TO_LIBRARY\Some.Library.dll"

My main issue is that
<test-suite type="Assembly" name="D:\PATH_TO_LIBRARY\Some.Library.dll" executed="False" result="Ignored"> doesnt contain the time or asserts attributes any more. Running against 2.6.4 console runner, it does.

---

@CharliePoole commented on Fri Sep 02 2016

It does seem like a bug to me. I'm moving it to the nunit-console project because it's the NUnit Engine, not the Framework, that translates nunit3 output to nunit2 format.

The XML produced by this translation is not going to be identical. We ensure that the format is correct, so that existing programs that process nunit2 XML files will be able to run. The semantics can obviously vary since NUnit 3 works differently from NUnit 2.

In this case, the biggest problem I see is that your assembly is shown as not having been executed. I suspect that's why there is no time and no assertion count - those attributes don't make sense if the assembly was never executed! But of course it was executed. :-)

---

@harrisonmeister commented on Fri Sep 02 2016

Hi @CharliePoole, thanks - yes that makes sense. I took a quick look at the NUnit2 writer and it looked like it omitted the attributes if the Test result was Skipped.

I'm in the process of upgrading a very large (1000+ projects) from Nunit 2.6.4 to 3.4.1 so the legacy format is very useful in leaving the test reporting side alone whilst I do the main upgrade (we had lots of uses of ExpectedException!)

I have also written a workaround that I will test on Monday to our test output which should also allow me to continue the upgrade

On a slight side note, we also use your nunit-summary executable. Do you have any guidance on how to "upgrade" to something more up-to-date when I take the plunge to remove the legacy v2 format from the result spec option on the command line?

Thanks, Mark

---

@CharliePoole commented on Sat Sep 03 2016

@harrisonmeister Another think I just learned about our V2 output is that it uses the V2 names for properties if you ran V2 tests under V3. If you run V3 tests but ask for V2 output, you get the V3 property names. Again, the format complies with your request but the content may differ.

This primarily affects properties Description (V2 _DESCRIPTION) and Categories (V2 _CATEGORIES). I don't think it's an error per se but it is a limitation. We could, of course build in more knowledge of the names of well-known properties.

Regarding nunit-summary... I haven't looked at it in ages. I'll take a look and see how much work an update would take. Are you interested in working on it or just using it. :-)

---

@harrisonmeister commented on Sun Sep 04 2016

@CharliePoole - I think for my scenario different property names is ok, since I am merging the test result xml files into a single file.

Re: nunit-summary, I'd be more than happy to work on it :-)

---

@CharliePoole commented on Sun Sep 04 2016

I did some updates to nunit-summary, changing the license, restructuring directories and updating it to use VS2015. I'll write an issue on that repo mentioning you and we can figure out how to proceed.

---

@theBFG commented on Thu Aug 24 2017

Hey, are there any updates or progress on this bug?

---

@ChrisMaddock commented on Thu Aug 24 2017

@theBFG This is acknowlegded as a bug and on our backlog, just needs someone to pick it up an put in a fix. A pull request would be welcome, if you're willing! 😄

@CharliePoole - this probably belongs in the nunit2-result-writer repo now, right? I'd guess it was moved here before we split the extensions out.

---

@rprouse commented on Sat Sep 16 2017

Agreed that this is probably a result writer issue. I am going to move it.

[CharliePoole]

Picking this up after a long absence and examining the output I see that the problem reflects, at least in part, the way the NUnit 3 framework records this situation. Any ignored test is given the Skipped:Ignored' result, which propagates to each higher level test, including the assembly. The only distinction with the higher level tests is that they have siteset toChild`.

We discussed, quite a long time ago, using the Warning result in these cases, but we didn't do it because it would be a breaking change. I hope this will be re-examined for NUnit 4.

For the same reason (i.e. compatibility) I think we have to keep the same values here as in V2 itself. We should show the missing elements, just as V2 does, but we should also show the same result as V2 even if it isn't logical to consider the assembly as not run.

[CharliePoole]

🎉 This issue has been resolved in version 3.7.0 🎉

The release is available on:

• GitHub Release
• NuGet Package
• Chocolatey Package

[JanibashaShaikTR]

Hi @CharliePoole , I am also facing similar issue. v3.7.0 still doesn't resolve the issue.

[CharliePoole]

@JanibashaShaikTR It's not clear what your "similar issue" is. Please explain it. A new issue would probably be better, but to save time you should re-read my comment of December 25, 2020, explaining why the extension works as it does.

[JanibashaShaikTR]

Thanks @CharliePoole , I read it through. I am running tests using nunit console runner with nunit3. I am getting results as
testresult.txt

I am expecting, the asserts and test execution as PASS at the assembly level as well, same as original issue.

[CharliePoole]

"For the same reason (i.e. compatibility) I think we have to keep the same values here as in V2 itself. We should show the missing elements, just as V2 does, but we should also show the same result as V2 even if it isn't logical to consider the assembly as not run."

The extension is designed to give (as close as possible) the same results as was given by the V2 Console Runner. In the earlier discussion, the issue was resolved by

1. Making sure that we show the time and assert count in the assembly element.
2. Continuing to show the assembly as ignored, just as the V2 console runner would show it.

Can you explain further why that resolution doesn't work for you?

[janijbsstudios]

Hi @CharliePoole , If you can see the attachment, the time and assert count is not showing up at assembly level. Please do needful, what I could be missing or let me know if you need any specific of my implementation info.

[JanibashaShaikTR]

Hi @rprouse , could you please tell, where you able to resolve the issue that you actually reported?