Abort job script with custom message.

[hjoliver]

Close #2274

• general CRITICAL event handler (c.f. our current WARNING handler) [now in Add CRITICAL event handler. #2304]
• custom abort-with-message function with message passed to the 'failed' event handler.

Test case to show the abort function works properly with execution retries (abort message is logged each try, but only passed to the failed handler at last try) and compare with trapped exit:

[scheduling]
    [[dependencies]]
        graph = t-abort & t-fail
[runtime]
    [[root]]
        [[[job]]]
            execution retry delays = PT5S
        [[[events]]]
            failed handler = echo "!!!FAILED!!! "
    [[t-abort]]
        script = """
echo ONE; sleep 5
cylc__job_abort "ERROR: rust never sleeps"
sleep 5; echo TWO"""
    [[t-fail]]
        script = """
echo ONE; sleep 5
/bin/false
sleep 5; echo TWO"""

@matthewrmshin - see if you agree with the approach before I and document and add tests. It's more or less in line with our discussion on #2274.

Note slightly unorthodox implementation by means of a shell function that the user has to call in task scripting. Probably should hook this into cylc message or a new command cylc abort? (It would be nice to avoid yet another command, but cylc message doesn't seem very appropriate in this case... cylc message --abort maybe ...)

[matthewrmshin]

Some quick answers:

• I think the approach is OK.
• The main issue with the abort function is that it also needs to turn off the error traps, etc of the job script, so it has to be done in the same shell.

Some quick questions:

• If we have configured a failed handler and a critical handler, I guess both will run?
• I wonder if we can implement the error trap, the vacation trap and this new functionality as one?

[hjoliver]

[critical handler functionality moved to #2304]

[hjoliver]

I wonder if we can implement the error trap, the vacation trap and this new functionality as one?

Done.

[hjoliver]

Not sure if this is the right thing to do here? (new CYLC_JOB_EXIT_MESSAGE in status files, just for aborted jobs)

[hjoliver]

I could just set CYLC_JOB_EXIT to the abort message?

[matthewrmshin]

I think you can just set CYLC_JOB_EXIT to the abort message. If I remember correctly, it can be anything other than SUCCEEDED, ERR and EXIT. cylc jobs-poll uses it as the value of the signal which can be any single-line strings.

Otherwise, if we want to introduce CYLC_JOB_EXIT_MESSAGE, I suppose CYLC_JOB_EXIT should be set to ABORT or ABRT (or something like that), and you may want to ensure that cylc jobs-poll can return it.

[hjoliver]

I actually used CYLC_JOB_EXIT originally, then changed it. I"ll revert to that.

[hjoliver]

The main issue with the abort function is that it also needs to turn off the error traps, etc of the job script, so it has to be done in the same shell.

Good point - so I left the abort call as a shell function.

[hjoliver]

(rebased to remove the CYLC_JOB_EXIT_MESSAGE change).

[hjoliver]

@matthewrmshin - please assign a second reviewer.

[matthewrmshin]

Some minor comments for the tests.

[matthewrmshin]

Can just do:

grep 'event-handler' "${LOG}" > 'edited-job-activity.log'

[hjoliver]

of course (I cut-n-pasted then cut down another test 😬)...

[matthewrmshin]

Do we need to add a test to ensure job.out only has ONE and not TWO?

[hjoliver]

Good spotting - that was my original intention (ran out of time earlier)

[hjoliver]

done

[hjoliver]

Feedback addressed. Over 'n out.