Does dotnet vstest support multiple test projects targeting different frameworks at a time?

[heng-liu]

Recently we’re testing for test projects targeting netcoreapp2.1 and netcoreapp3.0.

We found that if we run “dotnet vstest” with test projects targeting netcoreapp2.1 or netcoreapp3.0 only, the result looks correct(and no warnings).

But when we tried to run the two kinds of dlls together in one single command, there was a warning popping out immediately:
Test run will use DLL(s) built for framework .NETCoreApp,Version=v3.0 and platform X86. Following DLL(s) do not match framework/platform settings.
NuGet.Commands.FuncTest.dll is built for Framework 2.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
However, the results looked correct even we had this warning.

I’m wondering how did this warning generated, and if it’s ok for us to run them in one command, and ignore this warning? Thanks!

I found a similar issue #1768
But since I'm testing for core only, so it may be different.

AB#1494523

[vagisha-nidhi]

@heng-liu
Yes. Since both the dlls are of .NetCoreApp, the runner will issue a warning saying that it has picked up the highest version of .NetCoreApp and will run tests in both the dlls. You can ignore the warning as long as the frameworks is .NetCoreApp in both.
Although, if you are trying to run dlls targeting two different frameworks, that is, say, one is .NETFramework v4.5, and the other is .NetCoreApp v2.1, the tests of full framework will be run and that of net core will be skipped.

Since you are testing for core only, you will find no other discrepancies.

[zivkan]

Can someone please elaborate on "you can ignore the warning"? Was this advice given because netcoreapp5.0 is able to execute binaries compiled for netcoreapp2.1? Our goal is to detect when our code has problems when running on the older runtime, not if our code compiled for an older TFM works when running on a newer runtime. Hence it's important to us that the netcoreapp2.1 test assembly actually runs on that runtime.

For example, consider this:

C:\src\NuGet.Client\test\NuGet.Core.Tests\NuGet.Versioning.Test [dev-zivkan-pushToNewAzDOFeed ≡]> dotnet test .\bin\Debug\netcoreapp5.0\NuGet.Versioning.Test.dll .\bin\Debug\netcoreapp2.1\NuGet.Versioning.Test.dll
Microsoft (R) Test Execution Command Line Tool Version 16.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Test run detected DLL(s) which were built for different framework and platform versions. Following DLL(s) do not match current settings, which are .NETCoreApp,Version=v5.0 framework and X64 platform.
NuGet.Versioning.Test.dll is built for Framework .NETCoreApp,Version=v2.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
A total of 2 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:   885, Skipped:     0, Total:   885, Duration: 153 ms - NuGet.Versioning.Test.dll (net5.0)
Passed!  - Failed:     0, Passed:   885, Skipped:     0, Total:   885, Duration: 1 s - NuGet.Versioning.Test.dll (net5.0)

dotnet test's output shows on the last two lines that it ran both assembles on the net5.0 runtime. Is this a bug, or do we have to change our CI scripts to run each TFM in a different command line invokation?

@heng-liu are you able to re-open this issue?

[heng-liu]

Sorry, I'm not able to reopen it.
Hi @vagisha-nidhi, @ShreyasRmsft ,could you help re-open this issue? Thanks!

[nohwnd]

This whole behavior is imho a mix of historical reasons when only running in process was possible, wrong assumptions and inability to effectively change the current behavior to make more sense for .NET Core.

This warning happens because using dotnet test with a dll will just direct the call to vstest.console. vstest.console will grab the whole set of assemblies and check if they are all targetting the same framework (by checking the assembly TargetFramework attribute). It then reports all the assemblies that are not targetting the same TFM as the one decides is the real framework to use. BUT what then happens is that it will run one testhost.exe for each dll and pass the runtime config of the actual project, so the dll will end up running with the correct runtime.

Comparing this with running dotnet test + sln. This will not happen because it will actually run one instance of vstest.console for each assembly, and so there is no conflict, because each vstest.console gets only single dll.

So for .NET Core assemblies this warning is wrong, and you can ignore it. And you can confirm that by running this:

<TargetFrameworks>net5.0;netcoreapp3.1;netcoreapp2.1</TargetFrameworks>

using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace UnitTestProject11
{
    [TestClass]
    public class UnitTest1
    {
        [TestMethod]
        public void TestMethod1()
        {
            Assert.AreEqual(null, System.Runtime.InteropServices.RuntimeInformation.FrameworkDescription);
        }
    }
}

Which outputs the correct runtime identifier for each assembly (NET Core 4.6 is netcoreapp2.1), even thought the runtime reported in the summary is always net5.0:

Starting test execution, please wait...
Test run detected DLL(s) which were built for different framework and platform versions. Following DLL(s) do not match current settings, which are .NETCoreApp,Version=v5.0 framework and X64 platform.
UnitTestProject11.dll is built for Framework .NETCoreApp,Version=v3.1 and Platform AnyCPU.
UnitTestProject11.dll is built for Framework .NETCoreApp,Version=v2.1 and Platform AnyCPU.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
A total of 3 test files matched the specified pattern.
  Failed TestMethod1 [31 ms]
  Error Message:
   Assert.AreEqual failed. Expected:<(null)>. Actual:<.NET Core 3.1.9>.
  Stack Trace:
     at UnitTestProject11.UnitTest1.TestMethod1() in C:\t\UnitTestProject11\UnitTestProject11\UnitTest1.cs:line 11

  Failed TestMethod1 [31 ms]
  Error Message:
   Assert.AreEqual failed. Expected:<(null)>. Actual:<.NET 5.0.0-rc.2.20474.5>.
  Stack Trace:
     at UnitTestProject11.UnitTest1.TestMethod1() in C:\t\UnitTestProject11\UnitTestProject11\UnitTest1.cs:line 11

  Failed TestMethod1 [41 ms]
  Error Message:
   Assert.AreEqual failed. Expected:<(null)>. Actual:<.NET Core 4.6.29130.01>.
  Stack Trace:
     at UnitTestProject11.UnitTest1.TestMethod1() in C:\t\UnitTestProject11\UnitTestProject11\UnitTest1.cs:line 11


Failed!  - Failed:     1, Passed:     0, Skipped:     0, Total:     1, Duration: 41 ms - UnitTestProject11.dll (net5.0)
Failed!  - Failed:     1, Passed:     0, Skipped:     0, Total:     1, Duration: 31 ms - UnitTestProject11.dll (net5.0)
Failed!  - Failed:     1, Passed:     0, Skipped:     0, Total:     1, Duration: 31 ms - UnitTestProject11.dll (net5.0)

What should be done on our side, is that the vstest.console should be changed to figure out the target runtime for each of the dlls. And pass it internally as a map, not just a single target framework. Then we should look if there is any scenario where it actually makes sense to report this warning, because at the moment I can't think of any.

This would also be the first step to being able to share single instance of vstest.console under dotnet test when msbuild is used, which would help us report results from parallel runs in a sensible way and not hang the console by outputting into it from two different sources.

@pavelhorak this is related to #2080 but needs quite a bit of effort. Should we add it to the next milestone?

[waf]

I believe this was fixed in the most recent VS2022 17.3 release. From the release notes:

vstest.console now supports running assemblies of different target platforms and target frameworks in one request.

If I run dotnet vstest on two different DLLs (one is .NET Framework and the other is .NET 6) it appears to work! My version is Microsoft (R) Test Execution Command Line Tool Version 17.3.0 (x64). I think this was done in #3412 🎉

[nohwnd]

You are right :)