Improve error handling if exception occurs during shutdown

[MetRonnie]

These changes close #4147

Ensure that if an exception occurs during shutdown, it gets logged before exiting.

This is difficult to test for other than manually; see the steps to reproduce in the issue I've added an integration test, but probably worth manually running the example in #4147 too

Requirements check-list

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Tests included
• Appropriate change log entry included.
• No documentation update required.
• No dependency changes.

[MetRonnie]

This plays better with mypy

[MetRonnie]

Ignore test failures (due to DNS issue). Recommend running pytest locally

[hjoliver]

Looks good, tested as working 👍

[datamel]

This is an example of a platform lookup error put in remote_init, higher up the program….

This is exactly the same error put in remote_tidy….

Is there a reason for leaving traceback in? Surely the same error should produce the same result on screen?

[MetRonnie]

My reasoning is that an error during the running of the workflow can still end with a complete shutdown, but an error during shutdown likely means the shutdown did not complete.

Also, there's already a fair bit of error handling in the Scheduler._shutdown() method, so if it's not caught by that it's probably something serious enough to fail louder?

[datamel]

My reasoning is that an error during the running of the workflow can still end with a complete shutdown, but an error during shutdown likely means the shutdown did not complete.

Ah ok, my reasoning would be if you are raising an error in, for example remote_tidy, that is intentional, and, provided the error message is written correctly, provides sufficient information to the user about the problem.

The reason why I became aware of this bug and raised the issue, was at one stage (before a change implemented in remote_init (#4138)), the error raise in get_random_platform_for_install_target() was not bubbling up correctly in remote_tidy. In that case, there is no need for traceback, the error is sufficient (otherwise presumably we would put traceback on all our error raises).

Also, there's already a fair bit of error handling in the Scheduler._shutdown() method, so if it's not caught by that it's probably something serious enough to fail louder?

From my tests, this error handling will not help here if you look at the call stack, Scheduler._shutdown() happens before remote_tidy is executed.

[MetRonnie]

I've made it so that if a CylcError occurs during shutdown, it just gets a one line log entry, but if a general exception occurs, it logs traceback (this is how uncaught exceptions during the running of the workflow are logged).

[MetRonnie]

I have locally run:

• ✔️ flake8 & mypy
• ✔️ unit + integration + docstring tests
• ✔️ functional + flaky tests

[datamel]

Thanks @MetRonnie!