actions: don't run functional tests if unit tests fail (?)

[kinow]

Describe exactly what you would like to see in an upcoming release

Hi. In Travis the pipeline would stop immediately if unit-tests failed, but now it keeps going. Which means that it may take a while for developers to notice a unit test broken, or pycodestyle errors.

Should we leave it this way, or fail sooner to give devs a chance to fix it before the longer test-battery runs?

Additional context

Found it while looking at the latest change to UIS deltas branches, in which a commit broke the unit tests, but the build was still running.

https://github.com/cylc/cylc-flow/pull/3500/checks?check_run_id=661545105
Pull requests welcome!

[hjoliver]

I'm of two minds about this. Bailing out after failed unit tests can be annoying if you make an accidental style error that stops the functional tests relevant to what you're working on from running, and you don't check the result pretty quickly. On the other hand, if we need to prove that all tests work on a PR branch, then running everything twice just because of style test fix seems wasteful (of time and energy).

Ideally we should be able to trigger a re-run of just the tests that failed, after making a fix. But I don't think that's possible. (Hmm, and of course, we need to be sure that a fix for those tests did not break others ... but in principle that could be done with a final full run at the end).

[kinow]

Ideally we should be able to trigger a re-run of just the tests that failed, after making a fix. But I don't think that's possible.

That would be a great feature (in GitHub actions or some other tool)

(Hmm, and of course, we need to be sure that a fix for those tests did not break others ... but in principle that could be done with a final full run at the end).

Good point. Perhaps one of the blockers for such feature.

I'm OK with any approach here. If we leave as-is, I'll just make sure to take a look in the logs and stop the build if unit tests failed, then push changes fixing it 🤷‍♂️

[hjoliver]

Yeah, let's see if @oliver-sanders deliberately disabled this in his GH Actions PR, or not.

[oliver-sanders]

Yes, this was deliberate, it was replicating the Travis setup which did the same thing, however, it can be changed using the "strategy" configuration - https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstrategy.

[kinow]

All good then 👍 thanks!

[hjoliver]

it was replicating the Travis setup which did the same thing,..

Isn't it not replicating the Travis set up (with respect to stopping after failed unit tests or not)?

[oliver-sanders]

Aah sorry, yes, to retain this behaviour we would need to run the unit-tests before the functional tests (rather than in parallel) and use the needs configuration to make the functional tests dependent on the unit tests.

This would cost ~2mins.

[oliver-sanders]

It might be worth investigating this along with environment caching (which would knock down the install time) - see #3606.