Optionally spawn to max active cycle points.

[hjoliver]

Close #1904. Allows tasks to run out of order. See the discussion in #1538.

I decided against the "interim fix" of spawning two cycles ahead because it turns out to be quite difficult, and downstream stalls can still occur (it just takes two failures instead of one). Instead, as advocated by @benfitzpatrick and @matthewrmshin in #1538, on this branch you can set [scheduling]spawn to max active cycle points (...which is pretty self-descriptive).

This makes cylc behave exactly "as advertised" (no implicit dependence on previous-instance submit, tasks can run out of order, no stalling downstream of failed tasks) BUT spawn-on-submit has to remain the default for now because of the potential impact of the extra task proxies on:

• performance (more dependency matching)
• memory (pending light-weight task proxies Large suites: light weight task proxies. #1689)
• monitoring (all task proxies in the pool are displayed)

Also:

• it can (if more tasks become ready at once due to spawning past failures that previously stalled the suite) make the deficiencies of our current internal queue implementation more apparent (queued tasks are released essentially at random, regardless of cycle point or queue order).
• as discussed in Allowing cycling tasks to run out of order. #1538 some suites are probably relying on the implicit submit dependence to make downstream tasks wait on initial tasks.

However, it may be useful as-is for many real suites, and it serves as a nice proof of the free scheduling that we're ultimately aiming for.

@cylc/core and @trwhitcomb - if all agree, I'll document this along with advice to use it with caution on suites that are not too big with not too much runahead.

The test battery passes here (the new behaviour is only tested by one new test).

[hjoliver]

A good play-suite for this:

[cylc]
    cycle point format = %Y
[scheduling]
    initial cycle point = 2010
    final cycle point = 2030
    max active cycle points = 10
    spawn to max active cycle points = True
    [[queues]]
        [[[default]]]
            limit = 2
    [[dependencies]]
        [[[R1]]]
            graph = prep => foo
        [[[P1Y]]]
            graph = foo => bar & baz
[runtime]
    [[prep]]
    [[foo]]
        script = ((RANDOM % 2 == 1)) && false
    [[bar]]

[matthewrmshin]

Happy with this small change.

[hjoliver]

@matthewrmshin - can you assign another reviewer for this. Tim may be away...

[matthewrmshin]

@benfitzpatrick please sanity check.

[benfitzpatrick]

Looks good.