Respect user configuration for PreEnumerateTheories in xunit.console runner

[Jensaarai]

Users may plug into ITestCaseOrderer OrderTestCases functionality as a requirement to running their tests. If PreEnumerateTheories is false during xunit.console execution, this functionality will not run in the same way.

[bradwilson]

I need more information. What's the expected behavior? What's the actual behavior? What is a repro project that illustrates it?

[Jensaarai]

Sorry, maybe this is specific to an implementation so I've created an example
PreEnumerateTheories.zip

Expected behavior: like "dotnet test" from the project folder, or from VS's Test Explorer, test cases are run properly

C:\code\PreEnumerateTheories\XUnitTestProject1>dotnet test
Build started, please wait...
Build completed.

Test run for C:\code\PreEnumerateTheories\XUnitTestProject1\bin\Debug\netcoreapp2.1\XUnitTestProject1.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 15.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Total tests: 4. Passed: 4. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 2.2961 Seconds

Actual behavior: no test cases are run

C:\code\PreEnumerateTheories>dotnet exec --depsfile "XUnitTestProject1\bin\Debug\netcoreapp2.1\XUnitTestProject1.deps.json" --runtimeconfig "XUnitTestProject1\bin\Debug\netcoreapp2.1\XUnitTestProject1.runtimeconfig.json" contrib\xunit.runner.console\2.4.0-beta.2.build4010\tools\netcoreapp2.0\xunit.console.dll XUnitTestProject1\bin\Debug\netcoreapp2.1\XUnitTestProject1.dll
xUnit.net Console Runner v2.4.0-beta.2.build4010 (64-bit .NET Core 4.6.26515.07)
  Discovering: XUnitTestProject1
  Discovered:  XUnitTestProject1
  Starting:    XUnitTestProject1
  Finished:    XUnitTestProject1

Unhandled Exception: === TEST EXECUTION SUMMARY ===
   XUnitTestProject1  Total: 0

[bradwilson]

I'm looking at your code, but let me answer the initial inquiry with two pieces of information:

1. We disable theory pre-enumeration with the console runner, because it wastes cycles. The entire purpose of theory pre-enumeration is to be able to (potentially) present individually runnable theory tests to a UI-based runner, like the one built into Visual Studio. Our bespoke console runner does not provide such a UI, so pre-enumerating theory data is neither necessary or desired. Any test case orderer that depends on being able to get individual theory test cases is unfortunately going to be broken in this scenario.

2. We do not support the .NET Core console runner any more. It is provided as-is primarily for some teams at Microsoft.

Given the combination of these two facts, my suggestion is that you stop using the unsupported runner and instead rely on VSTest, as this is the only supported way to run .NET Core tests.

[Jensaarai]

"Any test case orderer that depends on being able to get individual theory test cases is unfortunately going to be broken in this scenario." Yeah, that's exactly my use case.

I'll be switching to VSTest, though it does not provide xml output files without https://www.nuget.org/packages/XunitXml.TestLogger Hopefully it stays compatible!