New (extra) family trigger semantics

[hjoliver]

In addition to FAMILY:fail-all and FAMILY:fail-any (and same for succeed, start, etc.) we should have triggers like "FAMILY:fail-50%-or-more'

#2333 (comment)

We'll need to think hard about a clear and flexible notation here.

[arjclark]

+1 for this as ensemble sizes continue to grow

[ColemanTom]

Definitely agree on this one, especially for very time critical/emergency systems. I've had to do similar manually from the task level before - polling for 'done' files after a certain time so that the suite could finish up once a reasonable number of ensemble members were done.

[GlennCreighton]

Upvote this! I ran into a use case where we only wanted to send a critical message when more than X members of an ensemble failed. For now, have scrapped together a workaround whereby each failed member gets the status of the family to check whether the threshold has been reached, and if so, outputs a critical message. Then, the if any of the family members outputs the message, a task is triggered to send an email. It'd be so much nicer to have this construct. Perhaps notation could be something like FAMILY:fail-more-than(X); FAMILY:succeed-more-than(Y)?