Unable to run .NetCore 2.0 tests from command line using Nunit3-console.exe

[Cosmin-Gramada]

I have a simple .Net Core 2.0 project containing only the following tests:

`using NUnit.Framework;

namespace TestStuffCore
{
[TestFixture]
public class Class1
{

    [Test]
    public void TestStuff1()
    {
        Assert.AreEqual(1 + 1, 2, "Equal");
    }

    [Test]
    public void TestStuff2()
    {
        Assert.AreEqual(1 + 1, 3, "Not equal");
    }
}

}
`

Nuget packages installed:
Microsoft.NET.Test.Sdk v15.5.0
Microsoft.NETCore.App v2.0.0
NUnit v3.9.0
NUnit.Console v3.8.0
NUnit3TestAdapter v3.9.0

Running them from Visual Studio Test Explorer, everything works fine.

Running them from console line fails with the following error:

..nuget\packages\nunit.consolerunner\3.8.0\tools\nunit3-console.exe "C:\Users\Cosmin Gramada\source\repos\TestStuff\TestStuffCore\bin\Debug\netcoreapp2.0\TestStuffCore.dll"
NUnit Console Runner 3.8.0
Copyright (c) 2018 Charlie Poole, Rob Prouse

Runtime Environment
OS Version: Microsoft Windows NT 10.0.16299.0
CLR Version: 4.0.30319.42000

Test Files
C:\Users\Cosmin Gramada\source\repos\TestStuff\TestStuffCore\bin\Debug\netcoreapp2.0\TestStuffCore.dll

Errors, Failures and Warnings

1. Error :
   NUnit.Engine.NUnitEngineException : An exception occurred in the driver while loading tests.
   ----> System.IO.FileNotFoundException : Could not load file or assembly 'nunit.framework' or one of its dependencies. The system cannot find the file specified.
   --NUnitEngineException

Server stack trace:
at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
--FileNotFoundException
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Activator.CreateInstance(String assemblyString, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)
at System.AppDomain.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
at System.AppDomain.CreateInstanceAndUnwrap(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)

Test Run Summary
Overall result: Failed
Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
Start time: 2018-01-30 15:54:59Z
End time: 2018-01-30 15:54:59Z
Duration: 0.547 seconds

Results (nunit3) saved as TestResult.xml

Further investigations:
1.) Running the exact same test code from a project targeting .Net Framework works as expected.
2.) Running the exact same test code from a project targeting .Net Core, but using NUnit.Console v3.7.0 results into the following error:

..nuget\packages\nunit.consolerunner\3.7.0\tools\nunit3-console.exe "C:\Users\Cosmin Gramada\source\repos\TestStuff\TestStuffCore\bin\Debug\netcoreapp2.0\TestStuffCore.dll"
NUnit Console Runner 3.7.0
Copyright (c) 2017 Charlie Poole, Rob Prouse

Runtime Environment
OS Version: Microsoft Windows NT 10.0.16299.0
CLR Version: 4.0.30319.42000

Test Files
C:\Users\Cosmin Gramada\source\repos\TestStuff\TestStuffCore\bin\Debug\netcoreapp2.0\TestStuffCore.dll

Errors, Failures and Warnings

1. Error :
   An exception occurred in the driver while loading tests.

Server stack trace:
at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
at NUnit.Engine.Runners.DirectTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)

Test Run Summary
Overall result: Failed
Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
Start time: 2018-01-30 15:55:16Z
End time: 2018-01-30 15:55:16Z
Duration: 0.549 seconds

Results (nunit3) saved as TestResult.xml

[CharliePoole]

Console runner does not currently support .NETCore tests.

@rprouse As an interim measure, perhaps they should be detected, as other things are and a special error message given. IIRC we do that for .NET CF.

[Cosmin-Gramada]

I opened it by mistake twice from 2 separate accounts.

Is there any workaround available? The main purpose is to be able to run the tests in CI (TeamCity)?
Also, I don't remember seeing anything in wiki (maybe I missed it?), so I assumed it will work.

[CharliePoole]

We should definitely give more information from the runner itself.

See the wiki here: https://github.com/nunit/docs/wiki/.NET-Core-and-.NET-Standard

[Cosmin-Gramada]

Ok, this provides me a way to run the .NetCore tests, however I'm guessing that I won't be able to make use of the Nunit Console runner command line parameters.

I was planning into using this feature so that I can inject different capabilities (for Selenium tests) used for remote testing on platforms like BrowserStack or SauceLabs. This and the custom attributes that can be created within NUnit...

[CharliePoole]

The nunit framework works the same under every runner, so your custom attributes will be fine. You should be able to use a .runsettings file to give parameters for the run.

[CharliePoole]

Of course, it would be better IMO if the console runner could handle it.

[rprouse]

.NET Standard/.NET Core 2.0 will make it much easier to support running .NET Core tests from the console/engine. We will need to create a .NET Standard 2.0 version of the engine and a .NET Core 2.0 version of the agent. Ideally, we would also create a .NET Core 2.0 version of the console for running in-process or on non-Windows machines that don't have Mono installed.

[CosminGramada]

@CharliePoole @rprouse
I need to run some of tests on a dozen browser-version combinations on both desktop and mobile device, therefore the reason that BrowserStack and SauceLabs were mentioned.

In aid for this, I was planning in using the --params|p=PARAMETER command line argument so that I can drive from my CI server the browser and browser version (and others if we're considering mobile) where the those particular tests should be ran.

I looked over the suggestion of using .runsettings file but from what I saw here and here it seems that TestRunParameters cannot be passed as command line arguments.

Am I missing something?

[CharliePoole]

The second of your two links talks about a generated .runsettings file. I'm not familiar with that. I was talking about a .runsettings file you create yourself in the proper format. If that's not compatible with using dotnet test then of course it won't work. @rprouse Can you comment?

[spokcarls]

We're running into a similar situation, switching away from xunit and back to nunit to execute UI tests with better and more reliable test context information against sauce labs (at the same time making the tests a bit more portable for execution in our build pipelines). Is there a path forward for allowing test execution from the runner against .Net Core/Standard 2.0?

[ChrisMaddock]

@spokcarls dotnet test is the way to go at the moment. As Rob says in #364 (comment) - we're hoping to support .NET Core in the full engine/console at some point - as ever, that's based on willing contributors!

[ChrisMaddock]

This issue will be taken care of under #475. We're continuing work on .NET Core support at the moment - please see https://github.com/nunit/nunit-console/projects/1 for the overall project view.

I'm going to close this issue in favour of the #475 - but please feel free to add any comments over there. 🙂