Feature Request: Conditional skipping test based on setup scripts

[andylokandy]

Motivation

In large-scale projects or projects heavily integrated with third-party services, it is often necessary to skip certain tests when specific conditions are not met. For instance, if a required database is not available, running integration tests dependent on this database should be skipped to save time and avoid irrelevant test failures.

Existing solutions

• Rust issue #68007: A long-standing issue suggests runtime test skipping but hasn't progressed. Need fundamental change on libtest.

• Community crate test-with: This uses a proc macro to add #[ignore] at compile time. While useful, it lacks the flexibility.

Proposal

Enhance nextest to support conditional test skipping based on the output of a setup script. The configuration may look like:

[[profile.default.scripts]]
filter = "test(~integration)"
run-if = 'check-db'
# alternatively, we can implement an option `skip-if-fail`
# setup = 'check-db'
# skip-if-fail = true

[script.check-db]
command = 'cargo run -p check-db'

When the script outputs 'false', the tests in the specified group (filtered by "test(~integrate)") should be marked as SKIP.

Finished `test` profile [unoptimized + debuginfo] target(s) in 0.13s
Starting 5 tests across 1 binaries (run ID: 7d7253f2-e050-46cd-98ef-0815cff44a4c, nextest profile: default)
    PASS [   0.002s] ut::foo
    PASS [   0.003s] ut::bar
    PASS [   0.003s] ut::baz
    SKIP [   0.003s] integration::foo
    SKIP [   0.003s] integration::bar
------------
    Summary [   0.022s] 5 tests run: 3 passed, 2 skipped

On CI, we can run cargo nextest run --run-ignored all to force them to run.

[sunshowers]

Thanks for the report! This seems like a good idea. The way I'd probably do this is: currently, setup scripts are either pass or fail. We'd want to add a third "skip" state for them.

As for how to indicate this: the most obvious answer is a special exit code, but that can be a bit brittle. So maybe we should have people write something like "skip" out to a file, along with maybe a message that can be displayed.

To set expectations: We might need this at some point at work, but otherwise I don't imagine I'll get to this soon.

[tgross35]

I have a PR to add runtime skipping functionality to libtest-mimic, at LukasKalbertodt/libtest-mimic#52. This just provides the same output as #[ignore] with libtest e.g.

test some_testname        ... ignored
test some_testname2       ... ignored, optional message

Would this be sufficient for nextest to hook?

This isn't exactly what is requested here, but tests could skip themselves based on an env set by the setup script.