Add support for producing XML test results

[rprouse]

Now that you can run tests using the dotnet test command, users are asking if they can produce NUnit test results from the test run. AFAIK, the adapter currently has no ability to produce the test results file, @OsirisTerje please correct me if I am wrong on that.

The old project.json based dotnet test command used to pass extra command line options on to our adapter to process. That is no longer the case so we are going to have to figure out how this will work with the new CLI tools.

The dotnet test command will pass extra commandline options to vstest.

Additional Arguments:
 Any extra commandline runsettings arguments that should be passed to vstest. See 'dotnet vstest --help' for available options.
                                        Example: -- RunConfiguration.ResultsDirectory="C:\users\user\desktop\Results Directory" MSTest.DeploymentEnabled=false

VSTest has options to specify a settings file, or enable a test logger. We might be able to use the logger option.

/Settings:<Settings File>
      Settings to use when running tests.

/logger:<Logger Uri/FriendlyName>
      Specify a logger for test results.  For example, to log results into a
      Visual Studio Test Results File (TRX) use  /logger:trx.
      To publish test results to Team Foundation Server, use TfsPublisher as shown below
      Example: /logger:TfsPublisher;
                Collection=<team project collection url>;
                BuildName=<build name>;
                TeamProject=<team project name>
                [;Platform=<Defaults to "Any CPU">]
                [;Flavor=<Defaults to "Debug">]
                [;RunTitle=<title>]

[rprouse]

For those that need test results, it has been reported that dotnet test --logger:trx works on Linux.

[CharliePoole]

This seems like a duplicate of nunit/docs#215

[rprouse]

This is pretty much a duplicate although I was approaching this more from the command line, especially dotnet test. Do you think we should close this, or have two issues, one for the runsettings and one for the command line?

[CharliePoole]

@rprouse No, now I see why it's separate. The first one worked on will need to create the underpinnings for writing out the file.

[bernardbr]

Hello @rprouse and @rprouse !
I'm using the alpha version of NUnit and it's great!

My project have more than one test project, and I need to create a task to run all of them via VSCode.
I opened an issue on dotnet cli for this: https://github.com/dotnet/cli/issues/6572

Using the solution suggested by @dasMulli, I can generate a test result file.

I don't know if this result file is equal to TestResult.xml of NUnit, but it can be a temporary solution until the test adapter generate the file.

As a way to ease the understanding, I created a project as an example: https://github.com/bernardbr/executing-unit-test-more-than-one-test-project

Best regards!

[dasMulli]

I don't think the test adapter can actually generate a file as it's loaded by the testhost processes that vstest spins off..
For nunit xml, this would probably need to be implemented as a custom VSTest logger..

For comparison, xunit gave up on trying to add options for vstest (=> dotnet test) and created their own cli extension (dotnet xunit) and msbuild runner to make xunit runner features available to CLI users / CI systems and not depend on VSTest features to allow passing custom options.

[rprouse]

@dasMulli thanks for the update, that was my suspicion too.

[gedbac]

Are you going to include this feature in near future?

[AO17]

is it possible to generate a xml file now?

[rprouse]

No, not yet

On Thu, Sep 28, 2017, 6:32 AM AO17 ***@***.***> wrote: is it possible to generate a xml file now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#323 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAeJBPMQbsTDZeaSnP281IR_X1ZSvvdsks5sm3W6gaJpZM4NR3wu> .

-- Rob Prouse

[ngbrown]

There is NUnitXml.TestLogger described at https://github.com/Faizan2304/LoggerExtensions by someone who is associated with the vstest project at Microsoft.

[OsirisTerje]

If you can accept a raw format, then you can enable the DumpXmlTestResults in a runsettings file, and that way get an xml results file out. See https://github.com/nunit/docs/wiki/Tips-And-Tricks

[SidShetye]

@ngbrown The logger you referenced seems to have it's own internal XML generator and subsequently generates XML that looks similar but yet quite different. I wish they reused nunit3 .dll for consistency.

As a result, if you try sending that output to jenkins' nunit plugin, it mangles up the assembly name, the class name and the test method names.

I think this is because they don't generate classname and methodname for the test-case tags.

i.e. <test-case id="0-1000" name="MyTest" fullname="My.Namespace.Testfixture.MyTest" methodname="MyTest" classname="My.Namespace.Testfixture" runstate="Runnable" seed="817325879" result="Passed" start-time="2018-01-01 00:00:00Z" end-time="2018-00-00 00:00:10Z" duration="0.015683" asserts="3" />

@rprouse @CharliePoole IMHO the authors of the two projects should collaborate a bit closely :) ... seems both sides could be reinventing the wheel

[CharliePoole]

Of course, when someone publishes their own code, they are free to work with is or not. It's messy, but also part of what drives open source. :-)

[baurzhan]

@rprouse thank you for suggestion
dotnet test --logger:trx
Works on Windows too

[agray]

This issue seems like a total blocker to me. This was opened over a year ago. What is the status of this issue now in October 2018? Do we have a means of producing an NUnit XML file that we can give to the Jenkins NUnit plugin?

[agray]

@rprouse thank you for suggestion
dotnet test --logger:trx
Works on Windows too

While I see that this produces an XML file (called .trx), how does this compares to NUnit 3 XML schema? Is it the same?

[agray]

There is NUnitXml.TestLogger described at https://github.com/Faizan2304/LoggerExtensions by someone who is associated with the vstest project at Microsoft.

This seems to do the trick. Haven't tested it against the Jenkins NUnit plugin yet to see if it is happy to consume it.

[dasMulli]

@agray you could also try the MSTest plugin which can deal with .trx and vstest code coverage files.

[agray]

There is NUnitXml.TestLogger described at https://github.com/Faizan2304/LoggerExtensions by someone who is associated with the vstest project at Microsoft.

This seems to do the trick. Haven't tested it against the Jenkins NUnit plugin yet to see if it is happy to consume it.

We really want to use anything but MSTest plugin because calculating code coverage with MSTest is horrible. With NUnit/JUnit code coverage is easy other open source NUGet packages.

I've had a chance to test publishing the file created with .NET Core 2.1 and this logger extension. Now I see what an earlier poster was referring to when they said the publisher doesn't resolve the test names.

My result was that it correctly saw that my TestResult.xml file contained 3 tests but the names are not present. I can only deduce that the logger extension does not transform the trx file to NUnit3 XML format (at least not compatible with the Jenkins NUnit plugin).

Has anyone solved this problem of .NET Core producing valid NUnit3 XML that the Jenkins NUnit plugin can consume?

[agray]

I have raise this with the NUnit logger project

[ArveSystad]

There is NUnitXml.TestLogger described at https://github.com/Faizan2304/LoggerExtensions by someone who is associated with the vstest project at Microsoft.

Can confirm this works nicely along with both Cake and TFS.

[brunomartinspro]

There is NUnitXml.TestLogger described at https://github.com/Faizan2304/LoggerExtensions by someone who is associated with the vstest project at Microsoft.

Thanks! With NUnitXml.TestLogger finally got SonarQube working with a report generated with dotnet test !

[abhilashsanne]

Thanks!! With NUnitXml.TestLogger and default support, I am able to generate both .xml and .trx files using the following command:

dotnet test -r ../TestResults --logger:trx --logger:"nunit;LogFilePath=..//TestResults/test-result.xml"

[RavitejaPatel]

@sanneabhilash could you give me some more info. about how to use the command which u have shared. when I ran the same command by cd to my project folder path I got below error

[OsirisTerje]

For anyone stumbling into this, it has been supported since adapter 3.12, using TestOutputXml
See #1114 for details.