Test cases are skipped with TestCaseSource under Visual Studio 2019 #676
Comments
|
You have only shown four of your test cases, but even with that small number, I can see that three of them will generate tests with the same name. That's the source of your problem. There's a mismatch between who NUnit works and how TestExplorer wants to see tests. NUnit doesn't care if multiple tests have the same name, because it doesn't use the name to identify the tests. TestExplorer expects each test to have a unique name. The NUnit test adapter does nothing to ensure this. It merely passes on the name provided (usually implicitly) by the user. In your case, the first argument Generally, when tests have the same name, TestExplorer groups all the results under the first one found. Select one of the tests that ran and look at the result report, usually directly under the tree but sometimes next to it if your window size is small. I suspect you'll find all fifteen. So, what can you do about this? There are a two main alternatives...
Regarding the "separate issue" you mentioned, Did you eliminate the outer |
|
@nunit/framework-team Maybe we should consider reducing the number of options available for use with |
|
Thanks for the detailed explanation @CharliePoole! I will try your suggestions and report the results. However, I don't think the issues can be fully explained by this theory. Here are all the test cases: As you can see there are various test cases where the site code and correlation token are different, but these test cases will still be ignored. I have put the ones NUnit runs in the beginning, but the ordering does not matter; it only ever runs these 3. For example, if I leave only the last 3 test cases, they will all be ignored and no test cases will be run at all. |
|
Hmmm... that messes up my theory all right! What platform do your tests target? If it's .NET Framework, can you check what happens if you run them under the nunit3-console runner? |
|
Is there any warnings/errors under the output for test execution? And can you upload the example or create a github repository that we can clone, so that we can recreate the problem easily. |
|
Good morning, here is a small sample project that illustrates the problem: If I run the tests through the console runner (in our case through TeamCity) it does run all tests, so the problem is related to the tests adapter. No errors or warnings in the output window! |
|
@LazarLashev What is the outcome if you use https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter/3.16.0-dev-01202 instead of NUnit3TestAdapter 3.15.1? (The build contains a rather substantial change to how the adapter identifies test cases, see #668 for a longer - and better - explanation) |
|
I was having a similar issue with VS 2019 & Test Explorer can now find and run my tests. |
|
@mikkelbu I have taken the sample test project above (NUnitTest.zip) and ran the tests with the updated adapter 3.16.0-dev-01202. It failed to run the tests, showing this output: |
|
@LazarLashev
|
|
I have tried to run the tests after removing bin/obj, and the first time they all passed. The second time I tried to run the tests, I got what it seems like the same error as before |
|
It appears as if the error is associated with the test window deciding to skip discovery and use information that it has saved. See the message "Discovery skipped: All test containers are up to date." Immediately after that message, execution begins. NUnit, in the course of execution, performs it's own discovery once again. This should be private, with no results reported to VS, but it's possible that some reporting is happening. That could conceivably lead to the duplicate key error that is being thrown by the Test window. In any case, this is clearly an error in the adapter only, so I'm transferring the issue to that project. @OsirisTerje does this match anything else you have seen? |
|
@CharliePoole Not directly, but after the last big related fix, I have at least 3 hanging bugs which should have been fixed by the same. I have asked the MS PG about this one in particular since the stack trace looks unknown to the adapter. I am 99.9% sure nothing is being sent out from the discovery phase with the execution, but we do build up a cache, so there can be something there. I'll work through this one and see if I can catch a trace from it. |
|
I believe I'm also running into this bug. I have an example project with test @ https://github.com/klreeher/nunitbugexample01/ This repo targets netcoreapp3, and my original repo targets v4.6.1. In the v4.6.1 framework repo, I tried the 3.16.0-dev test adapter, with no change. My specific example test shows a TestCaseSource that uses the following: |
|
Hi @klreeher. I just tried your example project, and it is a different issue than described above. The problem is that the test execution in VS is a two-phase run, one for discovery and one for execution. So when yield return new TestCaseData(Faker.InternetFaker.Email()).Returns(true); //skipped
yield return new TestCaseData(Faker.InternetFaker.Domain()).Returns(false); // sometimes not skipped, if I clean and delete my obj/bin files and am very very good???
yield return new TestCaseData(GetRandomNaughtyString()).Returns(false); //skippedis executed during the discovery it will find testcases with certain values, but when you try to execute these tests then the data will not match for the last three elements as these will have a new value. For GetRandomNaughtyString you can either use a seeded randomizer or perhaps use NUnits own random class via The issue #97 has more information on the two-phase run. |
|
Note that issue #97 was closed as fixed but that the fix assumes you are using NUnit's own randomizer at least as an initial seed. What happens is that the adapter saves the seed used for discovery in a temp file and re-seeds it for execution. It's a constraint on the adapter that discovery and execution run at different times in two different processes. The only communication possible is through the file system but we must still find or generate the same exact tests in discovery and execution. Currently, we simply save the random seed. A more general approach would be to save and restore the entire tree of discovered tests for reach assembly, but that's kind of a big deal! |
|
@mikkelbu @CharliePoole thank you both for the info! Switching my fake data to use the nunit random seed solved my problem. |
|
This bug is a duplicate of #685 . |
|
@LazarLashev This bug works with 3.16.0. In upcoming 3.16.1 you will need to add a runsettingsfile and set the following true: UseParentFQNForParametrizedTests The duplicate key issue is a known issue in VS 16.3 and 16.4. As far as we know, it will be fixed in 16.5 |
|
This issue is resolved by Version 3.16.1 which is released now on nuget.org and VSIX on Visual Studio marketplace. See all issues resolved and Release notes for details. |
|
@OsirisTerje , could you please share an example of runsetting file of how to set these two config values to true: UseParentFQNForParametrizedTests and UseNUnitIdforTestCaseId to true? |
|
Either the bug still exists or my It clearly sees the custom names and puts them under the function decorated with the Here is my .runsettings (simplified): Update: It was supposed to be running 4 tests and did run 4 tests. However, the interface of the test explorer doesn't reflect that at all: Another Update: And another: |
|
Yes, the use of ` also yields usable guids. |
|
Having the same issue with randomly generate TestCaseSource (e.g. Fixture.Create() ) being reported as 'not run' when running them using the MS Test Runner. They are reported as running just fine with the Resharper test runner. |
|
@gsonnenf Can you provide a small repro for this? |
|
The example uses the .NET Random class to generate tests. It's a known limitation that NUnit's wrapper class, which is part of TestContext, must be used when running under Visual Studio. The adapter contains code to save the random seed used in discovery and re-use it in the execution phase. |
|
The work around is to use Resharper Test Runner so I don't have add to add additional code to use AutoFixture. I just added the random in there to demonstrate it was not specific to AutoFixture. But... if its a known issue, that can't be fixed, why ask for me to spend a half hour setting up a test case? -_- |
|
@gsonnenf Sorry about that, but I assumed you had a case that was different from what was discussed above [your comment] - which talks about using the NUnit Random class. If you just added it in the example, and it happens in other places (Autofixture?), then that would be interesting. Without a code example it is very hard for us to understand what you're seeing. I've been using Autofixture myself without seeing any such issues, so it might be something new. |
|
@OsirisTerje Well, the repo is posted, along with the pictures of what I'm seeing. The issues occurs with value types and strings. It doesn't appear to occur with objects, which is likely what you are using. |
|
@gsonnenf Thanks! I'll have a look at it :-) |
|
@gsonnenf There is something strange going on here. I'll create a separate issue for this, and try to debug what this is. This doesn't look like the standard random issue. |
|
I have the same problem when using I use the next packages: Sample project to show the issue with Guid in Test Explorere vs 2019 Hi @OsirisTerje |
|
I did a workaround to fix the issue by using constant Any trial to use Update: It seems it's a known issue since 2016 |
|
It appears the random inputs issue might be a problem with the MS Test runner, that can't be fixed from the NUnit side. Perhaps someone should ask them to fix it so their VS built in tool can handle random inputs. |
|
Do you mean this also happens with the MSTest framework? There is anyway a problem with the NUnit system, since it does two passes over the code, one for discovery and then for the real execution. The random values will be different for these two passes. It may be possible to skip the first phase, but that may require the real time discovery to be on, so that we are sure the tests are actually being passed down. |
|
Ah! My mistake if that's where the issues lies. I may have incorrectly garnered from the entirety of the thread that the issue was elsewhere. (Particularly because its not an issue with the the Resharper test tool.) |
|
I solved my issue using runsettings as described below: <?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<NUnit>
<UseParentFQNForParametrizedTests>True</UseParentFQNForParametrizedTests>
<UseNUnitIdforTestCaseId>True</UseNUnitIdforTestCaseId>
</NUnit>
</RunSettings>Add the next line to the |
|
@gsonnenf The output of test Explorer is as shown below: The modified project: |
This was the answer for me - and I think it needs to be highlighted more. |
|
@jagdipafdb It's fairly well described in the nunit framework docs... Unfortunately, you need to know it's what you are looking for in order to find it. :-) |
|
I believe I have the same problem:
Note that I have multiple test cases, but they are all skipped. I actually have about 500 unit tests I do like this, but this particular test case simply won't run. The crazy thing is that before the changes in my current branch, the code above simply worked, as well as for all my other unit tests. I reviewed all my changes extremely thoroughly, and there is nothing indicating any effect on this setup, including projecting the knowledge from this discussion onto the changes. Initially, I was not using any runsettings (I have no experience with this). It is not clear to me if I did this correctly:
I have now (honestly) wasted about 2 hours on this problem and I'm a little fed up, since it was working before and I'd rather spend my time on something else. Any suggestions? PS. About the changes in my current branch. They do not affect the models/model creation of this Face object or any of its child objects. |
|
@Woudjee One minor suggestion... NUnit framework sometimes gets confused when the arguments are passed as an object[], even though that's still supported for legacy reasons. Try changing your source to return a TestCaseData[] instead. TestCaseData is the class designed precisely for the purpose of removing the ambiguity which arises when an object[] appears in a position where it might be the argument itself or an array of arguments. Anyway, this may not be the solution, but it's what came to mind looking at your code. |
|
@CharliePoole thank you for you quick reply. I have changed the object[] to TestCaseData[]. I must say that I like this syntax significantly more than the object[]. but unfortunately this did not fix the problem. What is happening on the background? Because there is no error message or any information why the test won't run. This does work (thereby clearly indicating that the objects are instantiated correctly):
However, the code above defeats the purpose of the TestCaseSource setup. |
|
@Woudjee Does it work if you set the name of the testcase via Are there any useful information in the Output window under "Tests"? |
|
@mikkelbu I am unsure what you mean by the SetName since I cannot call any methods in this kind of setup:
And unfortunately, there is no information there. I have also ran builds and tests with 'verbose', but also here there was no information whatsoever. |
|
@Woudjee You can call the method on the instance returned by the constructor - so something like this internal static TestCaseData[] GetTestCaseData =
{
new TestCaseData("Some data", 2).SetName("Test1"),
new TestCaseData("Another data", 124).SetName("Test2"),
}; |
|
@mikkelbu aha, I didn't check the return type and assumed it was void. I tried your suggestion, but unfortunately it didn't work :( |
|
So I just tried to make a pull request, and apparantly my pipeline fails. This is more information than I had before!
The thing is, I am requesting three arguments: And that I am providing three arguments: Any help is very much appreciated! Update: I have two tests that fail in the same fashion, I just noticed that I had screenshotted the wrong failing test. But the core information is identical:
|
|
So, apparantly, I did make a small mistake. However, also apparantly, my test does run in the context of my pipeline, whereas it does not in the context of the VS test explorer. Why is this? (Quite a funny way to figure this out like this actually)
|
What do you mean? Also, you're commenting on a closed issue. You can move this into either Discussions, if you just want some help along the way, or if you really believe this is kind of the same issue, you should raise a new one (you can point back to this), and if it is a regression, you can point that out too, or if you mean this is a variation, state that :-) |
Running latest versions of Visual Studio 2019 and NUnit (Visual studio 2019 16.3.5, NUnit 3.12.0, NUnit3TestAdapter 3.15.1)
Test with a test source:
I have 15 test cases in the test case source. Visual Studio shows all of them in the Test Explorer, but NUnit always runs just 3 of them. Always the same 3, and the rest are skipped. If I remove these 3, then it runs no tests. As a separate issue, if I leave just 1 test case in the test case source, it will think this
object[3]that is returned are 3 separate test cases (PlayerBalancesRequest, string and string) and will run the test 3 times with just 1 parameter, failing because the test requires 3 parameters.The text was updated successfully, but these errors were encountered: