Document how to skip suite-state polling

[hjoliver]

Users of suite-state polling for inter-suite triggering often hit the problem that polling tasks time out and fail if their suite is running a bit behind the clock and the upstream suite has been cold-started from scratch recently, thus wiping out its prior history of task completion.

Suite-state polling is typically used as a proxy for waiting on a file generated by a task in the upstream suite. If we checked for the file's existence and only bother to start polling the upstream generating task if it doesn't exist yet, this would largely eliminate the above problem. (And the downstream suite presumably has to know where the file lives anyway, in order to retrieve or use it).

So, to be documented:

• Use a custom polling task (rather than the graph syntax short-cut) that checks for the existence of the target file and only calls cylc suite-state if necessary.
• Or use a clock-expiring polling task, that only executes (and polls) if not too far behind the clock. However, this requires a more complex graph, with suicide triggers to remove expired tasks.

Finally, I considered adding a new option to cylc suite-state to call a user-supplied conditional check (e.g. file exists) before polling or not. But I don't think there's much point when you can just use a custom polling task, and further:

• the new built-in external trigger mechanism External trigger "plugin" functions. cylc-flow#2423 will solve this problem - a suite-state polling trigger function can easily be written with a skip condition based on file existence or whatever

[arjclark]

@hjoliver - this one's so hard to know what to do with as its so dependent on what the desired behaviour would be. I think often, as you say, the polling task is used as a blocker so that later tasks that assume the presence of a particular file don't get run. Quite like the idea of expiry but only in the context of "if I'm late I might be ok to run a file check instead before proceeding with my workflow".

The only code based enhancement I can think of would be to finesse the poller in some way so you could provide an option to treat polling for cycle points that would be before the cold/warm point of a recently restarted as "successful" in a similar (but definitely not equivalent as its never treated as a task success) manner to how dependencies on pre-start points get stripped out - perhaps a "--succeed-if-prestart" option?

[hjoliver]

+1 for @arjclark's last suggestion above, from a user at NIWA. (Although I'm not sure of the impact: would we have to look up the earliest cycle point every time a poll fails, and is that easy and quick to do?)

[arjclark]

@hjoliver - easy: yes, quick: not necessarily. I can think of several ways to do it and you could potentially make things easier by providing a cutoff -i.e. --succeed-pre YYYYMMDDTHHZ type option on the invocation side for when you're using either a generic script of the graphing inlined poller task if you know that the suite won't have any entries prior to that.