Get poll to return task failure if job/log has been removed.

[wxtim]

Closes #6425

Note

Note to reviewers, you will need to deploy this branch onto remote platforms to confirm it works for remote filesystems.

Finally have a replicable example (Thank you @oliver-sanders)

[scheduling]
    cycling mode = integer
    [[graph]]
        R1 = task

[runtime]
    [[task]]
        script = """
            rm ${CYLC_WORKFLOW_RUN_DIR}/.service/contact
            rm -r "${CYLC_WORKFLOW_RUN_DIR}/log/job/${CYLC_TASK_CYCLE_POINT}/${CYLC_TASK_NAME}"
        """
        platform = _remote_pbs

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[wxtim]

Only test__job_poll_status_files_deleted_logdir is directly related to the PR. Other tests should increase coverage. 😄

[oliver-sanders]

LGTM.

[oliver-sanders]

Note to reviewers, you will need to deploy this branch onto remote platforms to confirm it works for remote filesystems.

[MetRonnie]

From the original issue:

If you delete the job log directory for an active task, Cylc will preserve its last known status indefinitely. I.e, Cylc will consider the job to be submitted/running forever.

This does not seem to be true. It is only true if the job log dir and the contact file is removed.

This situation should be handled similarly to the job no longer appearing in the queue, i.e, the job is dead, long live the job. Stick it into the failed/submit-failed state as appropriate.

The job can succeed even if its job log dir is removed from under its feet.

There seems to be a problem here where the job log retrieval process keeps retrying, preventing shutdown without the --now --now option.

[runtime]
    [[task]]
        script = """
            rm -r "${CYLC_WORKFLOW_RUN_DIR}/log/job/${CYLC_TASK_CYCLE_POINT}/${CYLC_TASK_NAME}"
        """
        platform = <remote PBS>
        execution time limit = PT1M
        [[[directives]]]
            -q = shared
            -l ncpus = 1
            -l mem = 100mb

[oliver-sanders]

This does not seem to be true. It is only true if the job log dir and the contact file is removed.

Hmmm, we've seen this with PBS several times.

If you delete to job dir for a submitted task, it cannot run (no job script), so that'll definitely do it.

For a running job, not so sure.

The job can succeed even if its job log dir is removed from under its feet.

Possibly. But it will likely fail due to IO error and we cannot guarantee that Cylc will be informed of the job's outcome.

... From testing, it looks like echo foo doesn't cause the job to fail due to the missing file which makes sense for PBS as it "spools" the job output files in a temp dir so you have to try harder to delete them.

[oliver-sanders]

• Code makes sense.
• Tested against the example in the OP, works well, error message clearly logged.

We should talk this one over in tomorrow's meeting.

[MetRonnie]

Another example:

rm -r "${CYLC_WORKFLOW_RUN_DIR}/log/job/${CYLC_TASK_CYCLE_POINT}/${CYLC_TASK_NAME}"
sleep 10
exit 1

This task "fails successfully" (this PR as it currently stands makes no difference) but then the workflow hangs on shutdown after cylc stop.

Edit: the length of the hang depends on the global.cylc[platforms][<name>]retrieve job logs retry delays (it does not hang after the final retry). However, it is unclear to the user what is going on.

[MetRonnie]

Discussed today:

• Although the job may yet succeed even if the job log dir is deleted, we have decided that it is best to put in the failed state as this is the best we can do if we can't poll anymore.
• We will leave the job log retrieval as it is, as it's user error to delete the job log dir prematurely they will have to suffer the consequences

[MetRonnie]

I have some suggestions at wxtim#72

[wxtim]

I have some suggestions at wxtim#72

I'm happy with your suggestions. You wield your scapel with a confident hand. I like it.