Race condition in tests with false negative results

[0xmikko]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (23aa303 2023-11-28T00:21:45.422470192Z)

What command(s) is the bug in?

forge t

Operating System

Linux

Describe the bug

Problem

Occasionally, when we run all the tests, a weird thing happens: after some time and a few successful tests (a few fully successful test suites, then a few test suites with a few successful first tests), all the tests start failing, and the failing ones include deterministic tests that pass when run separately.
Re-running the test suite usually makes all the tests pass.

The trickiest part is that it occurs sporadically and is very hard to reproduce.
Previously, having a fuzzing test that fails with some large probability was a great opportunity to make this thing happen, but at this point it seems to occur even when everything works well.

Just by thinking about it, seems like there is some race condition with snapshot states when running test suites in parallel, but hard to confirm without analyzing the code deeply.

How to reproduce

1. git clone https://github.com/Gearbox-protocol/core-v3.git
2. cd core-v3/
3. yarn
4. export FOUNDRY_FUZZ_RUNS=10000
5. forge t

You'll get picture like that:

As you can see, the problem exists in contracts/test/unit/traits/USDT_TransferUnitTest.t.sol:USDT_TransferUnitTest, so run tests with this particular contact:

7. forge t --mc USDT_TransferUnitTest and you'll get that everything is okay:

Running tests and specify contract with --mt or --mc key will provide you ok.

[Evalir]

Closed by #6451