ci: Run regression tests in random order

[torokati44]

This is the primitive, chaotic neutral "little brother" of #15950, with the same motivation - to make Rust CI runs faster on average.

I do think that it would still make sense to sort the tests after shuffling, to bring the known slow ones forward - so, sorting only by kind, not by name.

To explain it by sticking to the packing problem analogy: this is basically shaking up (shuffling) the big bag (campaign) of little boxes (test cases), before packing (executing) them one by one into smaller backpacks (threads); this way, reducing the chance that some bigger boxes (slow tests) end up being packed too late, making all the backpacks too tall (waiting to finish), leaving some of them empty at the top (threads idling).

[kjarosh]

Some benchmarks from my PC:

Order	Average time (s)
by name	111
random	92.7 (-16.5%)
slow first	80.6 (-27.4%)

[kjarosh]

I guess the main issue I have with random order is that it's... random... at runtime at least. That way we're introducing indeterminism, as no two invocations will produce the same order (well... you know what I mean). It will be harder to reproduce and debug issues, tests will be less consistent and predictable, build can stop being reproducible. I believe that's also the reason why test execution frameworks generally do not run tests in random.

If test execution time is the main issue, I would recommend random, but predictable order. In particular, one can see ordering tests by their name as a kind of random (but predictable) order (because why by their names??), but it's just an exceptionally bad one in our case due to how tests are organized into directories. A better example would be sorting by hashes of test names or shuffling tests with a constant seed. That produces a random order which does not change each invocation.

[torokati44]

Eh, you're right, this is silly. I just wanted to put it out there, to let it have an honest go... But no.

[torokati44]

For future reference: It seems like nextest doesn't allow controlling the order in which the test cases are run:
https://github.com/nextest-rs/nextest/blob/a4490cdb039c28b321fef6b43a1b71a0af517999/nextest-runner/src/list/test_list.rs#L635

[torokati44]

Also just noting this idea here, lest it may be forgotten: instead of completely random order, we could instead sort by a hash of kind+name(+seed/salt)