[BUG] VSCode test runner seems to ignore test collections / test fixtures

[dguisinger]

Describe the Issue

I have unit tests that do integration tests with DynamoDB Local to test my database layer.

I use XUnit's test fixture pattern to instantiate a new table and tear it down once complete.

It works perfectly every time running "dotnet test" from the terminal in VS Code.

Running from VS Code's Test Explorer, tests randomly fail and if I set a breakpoint, multiple instances of my test fixture are being instantiated. It appears VS Code is running multiple instances of the test code in parallel when it is inappropriate to do so.

Steps To Reproduce

1. Create multiple unit tests
2. Create a test collection and assign it to the unit tests
3. Create a text fixture. Use it in the test collection generic parameter, and as a constructor parameter on the unit test
4. Add code to test fixture that can only run once
5. Run All Tests from VS Code. Sometimes they will all pass, other times random tests will fail.

Expected Behavior

I would expect VS Code test runner to run tests identically to Visual Studio and "dotnet test".

Environment Information

Version: 1.85.1 (Universal)
Commit: 0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2
Date: 2023-12-13T09:48:06.308Z
Electron: 25.9.7
ElectronBuildId: 25551756
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: 11.4.183.29-electron.0
OS: Darwin arm64 23.0.0

C# v2.14.8
C# DevKit v1.1.16

This has happened under both .NET 6 and .NET 8

[ocallesp]

@ocallesp need to reproduce

[ocallesp]

I was able to repro this bug with this sample project I created https://github.com/ocallesp/TestXUnitBug825

If we run all tests, Test Results will contain the following output:
Running Simple Test3
Current Time: 20:10:09.065
Database Initialization Count: 1
Running Simple Test2
Current Time: 20:10:09.065
Database Initialization Count: 1
Running Simple Test1
Current Time: 20:10:09.065
Database Initialization Count: 1
Running Simple Test1
Current Time: 20:11:09.602
Database Initialization Count: 1
Running Simple Test2
Current Time: 20:11:09.602
Database Initialization Count: 1
Running Simple Test3
Current Time: 20:11:09.601
Database Initialization Count: 1
Running Collection Test3
Current Time: 20:12:10.165
Database Initialization Count: 2
Running Collection Test2
Current Time: 20:12:10.165
Database Initialization Count: 2
Running Collection Test1
Current Time: 20:13:10.173
Database Initialization Count: 2

We can observe two things:

1. Test Execution Issue: The duplicate execution of the Simple Tests suggests that there may be a bug in VSCode's test runner that causes tests to be run more than once.
2. Collection Fixture Synchronization Issue: The parallel execution of tests Test3 and Test2 within the same collection indicates that the synchronization mechanisms that xUnit is the produced bug, leading to race conditions.

@peterwald

[ocallesp]

Assigning this to a member of the TestPlatform team since this seems to be related to test runner issue in xunit.

[ocallesp]

@fhnaseer is there some work pending in here, or can we close this issue ?

[ocallesp]

In Visual Studio 2022 this does not repro
@fhnaseer

[ocallesp]

Milestone moved to January 2025

@fhnaseer what is pending in here ?