An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object, with VS2015 #781

saiparth · 2020-09-04T13:02:08Z

NUnit =3.12 and NUnit3TestAdapter =3.17
Visual Studio preofessional version 15.3.3
.net version 4.6.1
not installed vsix

Steps to Repro

Create a unit test project
Remove mstest in nuget and install nunit and nunittestadapter latest version
Create a test with OneTimeSetUp attribute

using NUnit.Framework;
using System;

namespace nUnittest
{
    [TestFixture]
    public class UnitTest1
    {
       [OneTimeSetUp]
        public static void OneTimeSetUp()
        {
            Console.WriteLine("OneTimeSetUp");
        }
        [SetUp]
        public void SetUp()
        {
            Console.WriteLine("SetUp");
        }
        [TearDown]
        public void TearDown()
        {
            Console.WriteLine("TearDown");
        }
        [Test]
        public void Test()
        {
            Console.WriteLine("Test");
        }

        [OneTimeTearDown]
        public static void OneTimeTearDown()
        {
            Console.WriteLine("OneTimeTearDown");
        }
    }
}

Firs time passes,rebuild the solution>>Observe console >>error will display as mentioned in title ,test will pass,[OneTimeSetUp] / [SetUp] will not be executed
packages.config

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="NUnit" version="3.12.0" targetFramework="net461" />
  <package id="NUnit3TestAdapter" version="3.17.0" targetFramework="net461" />
</packages>

Close VS>>Run IFix vstestcache -f >>Rebuild>> works,first time,Test runs, later exception is thrown,One time setup not runs.
An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object

I tried all the workaround mentioned in #261
I tried removing the cache,works for first time,later again the same problem,cant restart everytime
I changed test process acritecture to 64 still same

Note
On running through nunit3-console all methods works

>nunit3-console D:\automationRepo\nUnitTest\nUnittest\bin\Debug\nUnittest.dll
NUnit Console Runner 3.11.1 (.NET 2.0)
Copyright (c) 2020 Charlie Poole, Rob Prouse
04 September 2020 18:28:56

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    D:\automationRepo\nUnitTest\nUnittest\bin\Debug\nUnittest.dll

SetUp
Test
TearDown
OneTimeSetUp
OneTimeTearDown

Run Settings
    DisposeRunners: True
    WorkDirectory: D:\automationRepo\nUnitTest\nUnittest\bin\Debug
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.6.1
    ImageRequiresX86: False
    ImageRequiresDefaultAppDomainAssemblyResolver: False
    RuntimeFramework: net-4.0
    NumberOfTestWorkers: 4

Test Run Summary
  Overall result: Passed
  Test Count: 1, Passed: 1, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2020-09-04 12:58:56Z
    End time: 2020-09-04 12:58:57Z
    Duration: 0.909 seconds

Results (nunit3) saved as TestResult.xml

The text was updated successfully, but these errors were encountered:

saiparth · 2020-09-05T05:59:07Z

Added sample project.
nUnitTest.zip

OsirisTerje · 2020-09-05T20:16:48Z

@saiparth Have you checked if this works with earlier versions of the adapter?

saiparth · 2020-09-06T05:56:42Z

Checked with 3.12.0 of nunit and 3.11.0 nUnit3TestAdapter after clearing temp files,
Exception was not thrown,But it dint print ontTimeSetup.OneTimeTearDown to console

OsirisTerje · 2020-09-06T11:01:27Z

@saiparth Thanks! That makes sense.

Now, and this is might be a bit weird: Can you comment out the OneTimeSetUp and the OneTimeTearDown, and run the tests ? It should now run green.
Then, uncomment the two methods and run again - what happens now ?

Also, can you copy the Output/Test window messages when it fails ?'

(I could repro it the first time I downloaded the repro, but after that, I can't force it to fail again.....)

Also, if you add the 0 to the runsettings NUnit section, oyu will suppress the writes, and then it should perform as 3.11.0. But I assume you want it printed out.

saiparth · 2020-09-06T14:25:18Z

@OsirisTerje
3.17 Exception will be thrown in console(It will not become red..like log it will be on console in test output window). Test will not be discovered after this
If not reproduces, The you close VS,clear temp files and open VS>>Rebuild
Below are the logs

[06-09-2020 19:38:48 Informational] ------ Load Playlist started ------
[06-09-2020 19:38:48 Informational] ========== Load Playlist finished (0:00:00.0160836) ==========
[06-09-2020 19:38:48 Informational] ------ Discover test started ------
[06-09-2020 19:38:50 Informational] NUnit Adapter 3.17.0.0: Test discovery starting
[06-09-2020 19:38:51 Informational] NUnit Adapter 3.17.0.0: Test discovery complete
[06-09-2020 19:38:52 Informational] ========== Discover test finished: 1 found (0:00:03.1219924) ==========
[06-09-2020 19:38:57 Informational] ------ Discover test started ------
[06-09-2020 19:38:57 Error] An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object.
[06-09-2020 19:38:57 Informational] ========== Discover test finished: 0 found (0:00:00.2031281) ==========

Commented oneTimeSetUp,OneTimetearDown
[06-09-2020 19:39:53 Error] An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object.
[06-09-2020 19:39:53 Informational] ========== Discover test finished: 0 found (0:00:00.0700293) ==========

Commented other retained only test method
[06-09-2020 19:40:56 Informational] ------ Discover test started ------
[06-09-2020 19:40:56 Error] An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object.
[06-09-2020 19:40:56 Informational] ========== Discover test finished: 0 found (0:00:00.0852885) ==========
```.

Downgraded adapter to 3.11.0
No exception was logged in test output window, But it dint print the console write line which was inside oneTimeSetup.
Please try to reproduce after clearing temp files

Aaron-Corp00 · 2020-09-28T21:12:24Z

I encountered a similar problem today (same exception message in the logs) where my tests would appear on the first build, but after running once the tests and rebuilding the tests would disappear. I cleared my temp files, but the problem persisted. Installing 16.1, manually as an VS extension, in VS2015 seems to have resolved the issue.

OsirisTerje · 2020-09-29T08:39:47Z

This issue seems bound to VS2015, not happening with VS2019.

saiparth · 2020-09-29T08:46:59Z

I am using Microsoft Visual Studio Professional 2017 Version 15.3.3

OsirisTerje · 2020-09-29T09:13:09Z

Can you drop a screenshot of your Test/General settings ?

saiparth · 2020-09-29T09:25:14Z

I have not selected any, Can you please let me know how to validate if its already selected?

OsirisTerje · 2020-09-29T09:25:27Z

@saiparth Not able to repro with 2017, 15.9.25. Can you upgrade to that version and recheck ?

Sorry, meaning: Tools/Options/Test/general

saiparth · 2020-09-29T09:30:50Z

Ok, Let me try updating

saiparth · 2020-09-29T12:01:12Z

Issue was not reproducible with Microsoft Visual Studio Professional 2017 Version 15.9.27. However it dint print the OneTimeSetUp/OneTimeTearDown

OsirisTerje · 2020-09-29T20:04:55Z

So, this is an issue with VS 2015 and VS 2017 up to at least 15.3.3, but fixed in at least 15.9.27 and VS 2019.

The workaround for those on lower versions here is then to not upgrade to 3.17.

It is something with VS going on here, and I can't see that we can do anything more about this.

Thanks for your reports, @saiparth !

felipepessoto · 2020-09-29T20:07:00Z

It means NUnit 3.17+ doesn't support VS2015 anymore?

OsirisTerje · 2020-09-29T20:12:50Z

@felipepessoto Yes it does, but this particular bug doesn't seem like something we can fix easily. It is triggered by the case shown, but otherwise you would be fine.

You can however build your own debug version of the adapter, and then enable debugging to see if you can pinpoint something we can change in the adapter. You can also raise an issue at Developer Community and ask for what change they did between 2017 15.3 and and 15.9 - that might give a clue too.

Is there a particular reason why you can't upgrade from VS 2015 ?

And, if there is a bigger demand for fixing this, we could look into it, we could reopen it - but we would appreciate a bit of debugging help.

OsirisTerje · 2020-09-29T20:33:48Z

@felipepessoto Notice you are a MSFT. You could then ask internally what changed between those two versions. That would be very helpful.

Aaron-Corp00 · 2020-09-29T20:40:00Z

"Is there a particular reason why you can't upgrade from VS 2015 ?"

As for myself, I work in a regulated industry. My team needs to support the team's product with VS 2015 for at least the next 2 years. We are only just beginning to talk about updating our core product to 2017 next year. Essentially we don't update until Microsoft stops adding features to a version of Visual Studio and then only when we have a major release where we would run our full suite of Verification tests.

My team doesn't strictly need to update as long as there is a version of the extension for VS 2015 that is stable.

OsirisTerje · 2020-09-30T07:35:51Z

@Aaron-Corp00 Which version of the adapter are you currently using (that works) ?

schlati · 2020-10-02T09:46:06Z

I can confirm this issue. I am on VS2015 too and experience the same behaviour (i.e. tests are only discovered once after startup of VS; after any further build operation the discovered tests are gone with the exception of the NUnit3TestDiscoverer. VERY annoying!).

I could solve this issue by removing the 'Nunit 3 Test Adapter' from Tools > Extensions and Updates, restarting VS and downgrading the NuGet package 'NUnit3TestAdapter' from v3.17.0 to v3.16.1. Hope that helps!

Aaron-Corp00 · 2020-10-02T15:11:28Z

@OsirisTerje I'm currently running 13.6.0, which is odd because I thought I installed 13.6.1, but its working at the moment so I'll probably leave this as is.

OsirisTerje · 2020-10-04T20:33:59Z

With VS2015, the TestAdapter's base class should initialize the internal engine adapter, but for some reason it is found to be null. This does not happen on later VS2017 and VS2019. It seems that the adapter is being kept alive, and then the engine adapter is null'ed during end of first test. This fix ensure the internal engine adapter is recreated if it is null when accessed.

OsirisTerje · 2020-10-04T20:59:04Z

@felipepessoto @schlati @saiparth @Aaron-Corp00 I believe the error is found. I would really appreciate if you can test the package at https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter (NUnit3TestAdapter 4.0.0-dev-01507), and verify if it work for you in your setup. Please ensure that you have no VSIX installed. On my repro I can no longer repro the case, but it is a weird case, so I would really need your input here.

felipepessoto · 2020-10-06T22:07:07Z

Sorry @OsirisTerje I'm really focused in a complex work and can't test it now, since I only have my working machine to test it.

saiparth · 2020-10-07T12:29:52Z

@OsirisTerje
Sorry for delay. Cleared temp files, created new project with

nunit 3.12 
NUnit3TestAdapter 4.0.0-dev-01507
Microsoft Visual Studio Professional 2017 
Version 15.3.3
VisualStudio.15.Release/15.3.3+26730.12

For above mentioned test, No exception was seen

[10/7/2020 5:57:03 PM Informational] ------ Discover test started ------
[10/7/2020 5:57:03 PM Informational] NUnit Adapter 4.0.0.0: Test discovery starting
[10/7/2020 5:57:03 PM Informational] NUnit Adapter 4.0.0.0: Test discovery complete
[10/7/2020 5:57:03 PM Informational] ========== Discover test finished: 1 found (0:00:00.3408487) ==========
[10/7/2020 5:57:13 PM Informational] ------ Run test started ------
[10/7/2020 5:57:16 PM Informational] NUnit Adapter 4.0.0.0: Test execution started
[10/7/2020 5:57:16 PM Informational] Running selected tests in C:\Users\admin\source\repos\UnitTestProject3\UnitTestProject3\bin\Debug\UnitTestProject3.dll
[10/7/2020 5:57:16 PM Informational]    NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run
[10/7/2020 5:57:17 PM Informational] SetUp
Test
TearDown

[10/7/2020 5:57:17 PM Informational] NUnit Adapter 4.0.0.0: Test execution complete
[10/7/2020 5:57:17 PM Informational] ========== Run test finished: 1 run (0:00:04.0671967) ==========

However it dint print the OneTimeSetUp/OneTimeTearDown

schlati · 2020-10-07T14:52:40Z

@OsirisTerje

Sorry @OsirisTerje I'm really focused in a complex work and can't test it now, since I only have my working machine to test it.

Thank you! That fixes the problem for me! (I use VS 2015 Version 14.025431.01 Update 3)

shahnawazm786 · 2020-11-26T15:14:02Z

Hello all
I am getting error in visual studio 2013
An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests.
please help me out
Thanks

OsirisTerje · 2020-11-26T19:24:01Z

@shahnawazm786 A little bit more details would be helpful.

OsirisTerje added the Investigate label Sep 5, 2020

OsirisTerje added the confirm We need a separate confirmation of this issue label Sep 29, 2020

OsirisTerje removed the confirm We need a separate confirmation of this issue label Sep 29, 2020

OsirisTerje closed this as completed Sep 29, 2020

OsirisTerje added closed:wontfix External Tracked here, but must be fixed in external tool VS Issue and removed Investigate labels Sep 29, 2020

OsirisTerje added a commit that referenced this issue Oct 4, 2020

Fix for nullref #781

b650d93

OsirisTerje reopened this Oct 4, 2020

OsirisTerje added closed:done and removed closed:wontfix labels Oct 4, 2020

OsirisTerje added this to the 4.0.0-beta.1 milestone Oct 4, 2020

OsirisTerje closed this as completed Oct 6, 2020

OsirisTerje removed VS Issue External Tracked here, but must be fixed in external tool labels Oct 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object, with VS2015 #781

An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object, with VS2015 #781

saiparth commented Sep 4, 2020 •

edited

Loading

saiparth commented Sep 5, 2020

OsirisTerje commented Sep 5, 2020

saiparth commented Sep 6, 2020

OsirisTerje commented Sep 6, 2020 •

edited

Loading

saiparth commented Sep 6, 2020 •

edited

Loading

Aaron-Corp00 commented Sep 28, 2020

OsirisTerje commented Sep 29, 2020

saiparth commented Sep 29, 2020

OsirisTerje commented Sep 29, 2020

saiparth commented Sep 29, 2020

OsirisTerje commented Sep 29, 2020 •

edited

Loading

saiparth commented Sep 29, 2020

saiparth commented Sep 29, 2020

OsirisTerje commented Sep 29, 2020

felipepessoto commented Sep 29, 2020

OsirisTerje commented Sep 29, 2020 •

edited

Loading

OsirisTerje commented Sep 29, 2020

Aaron-Corp00 commented Sep 29, 2020

OsirisTerje commented Sep 30, 2020

schlati commented Oct 2, 2020 •

edited

Loading

Aaron-Corp00 commented Oct 2, 2020

OsirisTerje commented Oct 4, 2020

OsirisTerje commented Oct 4, 2020

felipepessoto commented Oct 6, 2020

saiparth commented Oct 7, 2020

schlati commented Oct 7, 2020

shahnawazm786 commented Nov 26, 2020

OsirisTerje commented Nov 26, 2020

An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object, with VS2015 #781

An exception occurred while test discoverer 'NUnit3TestDiscoverer' was loading tests. Exception: Object reference not set to an instance of an object, with VS2015 #781

Comments

saiparth commented Sep 4, 2020 • edited Loading

saiparth commented Sep 5, 2020

OsirisTerje commented Sep 5, 2020

saiparth commented Sep 6, 2020

OsirisTerje commented Sep 6, 2020 • edited Loading

saiparth commented Sep 6, 2020 • edited Loading

Aaron-Corp00 commented Sep 28, 2020

OsirisTerje commented Sep 29, 2020

saiparth commented Sep 29, 2020

OsirisTerje commented Sep 29, 2020

saiparth commented Sep 29, 2020

OsirisTerje commented Sep 29, 2020 • edited Loading

saiparth commented Sep 29, 2020

saiparth commented Sep 29, 2020

OsirisTerje commented Sep 29, 2020

felipepessoto commented Sep 29, 2020

OsirisTerje commented Sep 29, 2020 • edited Loading

OsirisTerje commented Sep 29, 2020

Aaron-Corp00 commented Sep 29, 2020

OsirisTerje commented Sep 30, 2020

schlati commented Oct 2, 2020 • edited Loading

Aaron-Corp00 commented Oct 2, 2020

OsirisTerje commented Oct 4, 2020

OsirisTerje commented Oct 4, 2020

felipepessoto commented Oct 6, 2020

saiparth commented Oct 7, 2020

schlati commented Oct 7, 2020

shahnawazm786 commented Nov 26, 2020

OsirisTerje commented Nov 26, 2020

saiparth commented Sep 4, 2020 •

edited

Loading

OsirisTerje commented Sep 6, 2020 •

edited

Loading

saiparth commented Sep 6, 2020 •

edited

Loading

OsirisTerje commented Sep 29, 2020 •

edited

Loading

OsirisTerje commented Sep 29, 2020 •

edited

Loading

schlati commented Oct 2, 2020 •

edited

Loading