VS Adapter incorrectly classifies functions with dots for F#

[OmarTawfik]

F# functions can have arbitrary names if surrounded by double back ticks:

They are incorrectly classified by nunit adapter (when grouping tests by class name). The example shown has class (TestClass) and test (Part1.Part2), but categorizes the test under class (Part1).

[CharliePoole]

I'll double-check the code on this, but my first thought is that it's out of our control.

As I've said in other cases, the our Adapter is... an adapter. It adapts NUnit to interfaces defined by Visual Studio. There is no place in that interface where we tell VS the class in which a test occurs. Rather, we give it the "test name" which is assumed by VS to be in the form of the full name of a C# method. That's a rather parochial assumption on their part, but it's what it is!. :-(

One possibility is that NUnit might not be correctly locating the method definition in the source and that that location, which is passed to VS, might help it figure out the class name correctly. Does VS show the proper location for the method in the source code when you select the test?

[CharliePoole]

It would also help understand this if you can run your test under nunit3-console. The XML result file will include the name NUnit gives the test as well as the name of the class and method as seen by NUnit.

[OmarTawfik]

@CharliePoole Running under nunit3-console produces an XML with data like:
<test-case name="Part1.Part2" executed="True" result="Success" success="True" time="0.028" asserts="1">
Any thoughts?

[CharliePoole]

That's good then. NUnit knows what is the test case and what is the class. Unfortunately, what we give to VS is the "full name" concatenating the class and the test case. VS then parses this to "discover" a class of Part1 and a test name of Part2.

What about the file location of the test. Does that show up correctly?

[OmarTawfik]

The location is correct. Can you please point me out to that part of nunit code that is passing the data to VS so that I can follow up with the correct person?

[CharliePoole]

Here's the ConvertTestCase method that converts an XmlNode representing the test case to a TestCase object for VS to use:
https://github.com/nunit/nunit3-vs-adapter/blob/master/src/NUnitTestAdapter/TestConverter.cs#L42

That's what is used in the discovery phase. As the tests are run, we get results and use the method ConvertTestResult, a bit farther down in the file to convert them.

The helper method MakeTestCaseFromXmlNode does the work. It's located in the same file at line 131. As you see, we pass the NUnit fullname attribute as the name of the test. That name has no information that would enable VS to parse it any other way than as it does, but that's what the original TestWindow team asked to have passed in. As you can see in the helper method, we have the class and method name availalble separately and even use them to find the source, but they are not given to VS for use.

The key issue is that NUnit considers a test name (or fullname) to be an arbitrary string with no particular structure. By default it "kind of" looks like the name of a class or method, but it is never parsed by NUnit itself. Users can even give their tests completely arbitrary names that have no connection to the source code. OTOH, VS wants a name that can be parsed into class and method components. Ideally, the TestCase would have separate fields for className, methodName and displayName, all of which we could easily provide.

For now I'll keep this open as is in case you can come back with something we could do ourselves to make this work.

[OmarTawfik]

@CharliePoole Thanks! I'll update this issue if I get any update.

[CharliePoole]

@otawfik-ms I'm closing this as not a bug (of ours) but if you learn of something that we can do in order to cause VS to correctly figure out the class, we can re-open.