simulation modes: proportional run length and more

[hjoliver]

Close #2185
Close #399
Close #1859
Partially address #2188

[Note #description updated several times post feedback etc.]

Simulation and dummy mode improvements, including proportional run length, via the following single config section that applies to all non-live modes: dummy, dummy-local, simulation.

[cylc]
    [[simulation]]
        disable suite event handlers = False  # default True
...
[runtime]
    [[root]]
        [[[simulation]]]
             default run length = PT20S   # default PT10S
             speedup factor = 10  # default None
             time limit buffer = PT20S  # default PT10S
             fail cycle points = 2017, 2018  # or 'all' for all.  Default none.
             fail try 1 only = False  # default True
             disable task event handlers = False  # default True 

Section name simulation because both dummy mode and simulation mode are ways of simulating real suite behaviour. Probably should rename "simulation mode" to "virtual mode" or something, but that would be more disruptive.

If speedup factor and execution time limit are defined, simulated run length is limit / speedup, else default run length. These are defined under runtime to allow adjusting specific task runtimes if necessary.

execution time limit is scaled so that dummy tasks don't ask for too much CPU time, and padded with time limit buffer (to avoid quick XCPU kill)

Event handlers disabled by default, but can be left on.

Simulation mode is now more obvious in the GUI (text tree view).

The fail cycle points and fail try 1 only items allow more flexible simulation of failure conditions.

New dummy-local mode in which task remote host and batch system are dummied out - to allow even other-site suites to run locally in dummy mode.

[oliver-sanders]

Does this close #399?

[matthewrmshin]

Is use proportional run length = True redundant? We can just do:

• If proportional speedup factor and execution time limit are set, use proportional speedup factor.
• Otherwise, use default run length.

[hjoliver]

@oliver-sanders - good spotting. Updated description to close #399.

[hjoliver]

@matthewrmshin - yes, the boolean is redundant; I'll remove it...

[oliver-sanders]

It might be a good idea to set this to something like echo 'no task script defined' so that there is evidence in the job output that the script item has not been set.

[hjoliver]

Done.

[oliver-sanders]

I was thinking just of the script with this comment. For a task like [foo]script=true the job.out now looks like this:

(no init-script defined)
Suite    : <suite-name>
Task Job : 1/<task-name>/01 (try 1)
User@Host: <user>@<host>

(no env-script defined)
(no pre-script defined)
2017-04-05T09:35:11+01 NORMAL - started
(no post-script defined)
2017-04-05T09:35:12+01 NORMAL - succeeded

[oliver-sanders]

Could we combine fail all cycle points and fail these cycle points say into something like:

fail these cycle points = *
fail these cycle points = 2000, 2001, ...

[hjoliver]

Could we combine fail all cycle points and fail these cycle points...

Done. Description updated.

[hjoliver]

Note both dummy and dummy-local mode run background dummy jobs now, to avoid asking PBS (et. al.) for huge resources for 10-sec dummy jobs, as execution time limit is the only resource directive we can scale automatically at this point.

[hjoliver]

@matthewrmshin - note the bulk of the changes to task_proxy.py are because I realized all script-item manipulation, for all modes and suite-state polling tasks, could be taken out of task proxy instances and done more simply up front in config.py instead (and cycle point specific stuff is implemented in the dummy job scripting).

[hjoliver]

Reviewers - this really needs a bunch of tests for all the new functionality. However, all current tests pass (on Travis CI at least), and I'm quite busy right now, so we could just put up a new issue for the tests in order to avoid delaying #2157 any longer - what do you think?

[oliver-sanders]

Would it be possible to negate the tasks user-defined [environment] in dummy mode?

The following suite cannot be run in dummy mode as the env-script is not run.

[scheduling]
    [[dependencies]]
        graph = oxygas
[runtime]
    [[root]]
        script = sleep 1
    [[oxygas]]
        env-script = ELSE=foo
        [[[remote]]]
            host = els055
        [[[environment]]]
            SOMETHING = "some-modification-$ELSE"

[hjoliver]

Good point - done.

[oliver-sanders]

The new dummy-local mode is great for profiling and should succeed the profile-simulation mode I built in (to cylc profile-battery) for back-compatibility when it is no longer required.

[matthewrmshin]

(I didn't realise that we had a cylc random command.)

[hjoliver]

(I didn't realise that we had a cylc random command.)

(I - once - didn't realise there was bash $((RANDOM % 10))...)

[oliver-sanders]

Looks good.

One minor comment we can address later is to include the new "dummy-local" mode in the cylc run options in gcylc.