Make it easier to re-run failed fuzz tests with verbose logging

[wadealexc]

Component

Forge

Describe the feature you would like

Currently ~90% of our test suite uses fuzzy parameters. When we run our entire test suite, we occasionally get flaky test failures because we haven't properly filtered certain inputs with vm.assume (or maybe tests fail for another reason).

When fuzz tests fail due to specific input, the test runner helpfully displays the test input in red text - but if I want to go run that failed test using the provided input, I have to do a long workaround with forge script, to rerun this test and view the logs.

Ideally I could do something like forge test --last-failed -vvvv or something to just rerun the last failed fuzz tests with logs turned on!

Additional context

No response

[mds1]

Would the UX specified in #2551 and #2552 meet your use case? Essentially failures get saved to JSON files and can be re-run by passing a path to that input file

[wadealexc]

I love #2551 - it's not quite the UX I'm talking about but seems super useful nonetheless. I'd love to be able to quickly retest failing inputs after I tweak code. #2552 is similar.

The UX I'm looking for though is to specifically get more info about why a fuzz test failed. I think the simplest version of the UX I want is just to add a flag like forge test --verbose-failures that, when a failing test occurs, the test is re-run with the same inputs and -vvvv and either spit out in the console or to some file.

I think that'd satisfy my needs without needing to store things between runs!

[wadealexc]

We have some pretty extensive logging in our tests - it'd be nice, when the test failed, to see that logging so I can go debug the failure itself. The issues you linked seem more suited to "after I debug and fix the broken code, how can I ensure the same inputs succeed this time". Very valuable, but not what I opened this issue for.

[grandizzy]

the way #2551 is implemented is that when a fuzz failure occurs it is persisted and always replayed first on subsequent runs, so the test will exit right away. If I am not missing something I think you can have the behavior you're looking for by rerunning the failed test with -vvvv (as will replay only the failed scenario and exit), for example given a function like

    function test_replayFuzz(uint256 x) public {
        require(x != 3, "failed");
    }

when failure is hit is recorded

proptest: Saving this and future failures in cache/fuzz/failures
proptest: If this test was run on a CI system, you may wish to add the following line to your copy of the file. (You may need to create it.)
cc a202880bbec866fdef8f07cc22c9526f115945fb624327f53c39e8dea6260b1e

Ran 1 test for test/ForgeContextTest.t.sol:ForgeForkDeleteTest
[FAIL. Reason: revert: failed; counterexample: calldata=0x00144d930000000000000000000000000000000000000000000000000000000000000003 args=[3]] test_replayFuzz(uint256) (runs: 157, μ: 247, ~: 247)

then if you rerun forge test --mt test_replayFuzz -vvvv you will get an output like

Ran 1 test for test/ForgeContextTest.t.sol:ForgeForkDeleteTest
[FAIL. Reason: revert: failed; counterexample: calldata=0x00144d930000000000000000000000000000000000000000000000000000000000000003 args=[3]] test_replayFuzz(uint256) (runs: 1, μ: 247, ~: 247)
Traces:
  [346] ForgeForkDeleteTest::test_replayFuzz(3)
    └─ ← [Revert] revert: failed

Mind that when running again there is a single run (vs 157 originally) with the failed case, would this help? @wadealexc

[wadealexc]

Ah, that seems like it would work for my use case! Thank you :)