Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redo the User Guide Reflow section. #427

Merged
merged 9 commits into from
Apr 13, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions src/7-to-8/caveats.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,9 @@ Trigger Edit

* https://github.com/cylc/cylc-flow/issues/3751

Reflow
The new "reflow" functionality, which allows multiple
(potentially concurrent) executions of the same workflow in a single
:term:`scheduler`, is not fully supported by all commands
Multiple Flows
The new :term:`scheduler` can manage multiple flows in the workflow graph.
Commands and options for interacting with flows are still being refined.


Browser Based UI
Expand Down
2 changes: 1 addition & 1 deletion src/7-to-8/major-changes/cli.rst
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ renamed or removed.
``cylc checkpoint``
- Database checkpoints are no longer needed.
- All task state changes are written to the database when they occur.
- Remaining use cases can be handled by :term:`reflows <reflow>`
- Remaining use cases can be handled by starting a new :term:`flow`
which allow a new execution of the graph to be started from an
arbitrary point in the graph.
``cylc conditions``
Expand Down
3 changes: 2 additions & 1 deletion src/7-to-8/major-changes/play-pause-stop.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,8 @@ The ``cylc play`` command will always pick up where it left off (a
If you want to re-run the entire workflow again from the start either:

* :ref:`Install a new run<Using Cylc Install>`.
* Or if you want to keep the data from the old run start a :term:`reflow`.
* Or if you want to keep the data from the old run start a new :term:`flow` at
the beginning of the graph, and stop the original flow.

It is still possible to re-run workflows in-place in the Cylc 7 manner, however,
this is discouraged.
Expand Down
4 changes: 1 addition & 3 deletions src/7-to-8/major-changes/scheduling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,7 @@ Cylc 8 has a new scheduling algorithm that:

- (to fully understand which tasks appeared in the Cylc 7 GUI you had to
understand the scheduling algorithm)
- Supports a powerful new capability called :term:`reflow`: you can trigger
multiple concurrent flows in the same graph at once, managed by the same
scheduler
- Supports multiple concurrent :term:`flows<flow>` within the same workflow.
- Can start a workflow from any task or tasks in the graph (no need for
checkpoint restart)
- Can limit activity within as well as across cycles, without risking a stall
87 changes: 60 additions & 27 deletions src/glossary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,26 +43,39 @@ Glossary


retry
Tasks configured to retry on failure will return to the ``waiting`` state
with a :term:`clock trigger` to delay the next try.
task retry
try number
Tasks can be configured to retry automatically on failure, one or more
times. They return to the ``waiting`` state with a :term:`clock trigger`
to delay the retry, and only go to the ``failed`` :term:`state <task
state>` once the final try fails.

Any number of retries, with configurable delays between them, are possible.
Task jobs can get their own try number from ``$CYLC_TASK_TRY_NUMBER``.

If the final try fails, the task goes to the ``failed`` :term:`state
<task state>`.
The task try number increments with every automatic retry, and is
passed to the job environment as ``$CYLC_TASK_TRY_NUMBER``.

.. seealso::

* :ref:`Cylc User Guide <TaskRetries>`


submit number
task submit number
Every time a task re-runs, whether by automatic :term:`retry` or manual
triggering, its submit number increments. It is passed to
the job environment as ``$CYLC_TASK_SUBMIT_NUMBER``.

Submit number also appears in the job log path so that job log files
don't get overwritten.


window
n-window
active window
workflow window
active task pool
This is a :term:`graph`-based window or view of the workflow at runtime,
including tasks out to ``n`` graph edges from current active tasks.
including tasks out to ``n`` graph edges from current :term:`active
tasks<active task>`. The *active window* is ``n=0``.

.. seealso::

Expand Down Expand Up @@ -142,14 +155,17 @@ Glossary
the same as :term:`workflow id`.


active waiting task
An active waiting task is a task in the :term:`scheduler's <scheduler>`
active window that is "actively waiting" on (i.e. periodically checking)
an :term:`external trigger` or :term:`clock trigger`.
active
active task
An active task is a task in the submitted or running state.

These are the only waiting tasks that matter to the :term:`scheduler`.
Waiting tasks ahead of the active window in Cylc 8 are entirely
abstract.

active-waiting
active-waiting task
An active-waiting task is a task in the :term:`scheduler's <scheduler>`
``n=0`` :term:`active window` that is ready to run according to its task
prerequisites, but is still waiting on a limiting mechanism such as a
:term:`clock trigger`, task :term:`hold`, or :term:`internal queue`.


external trigger
Expand All @@ -171,6 +187,7 @@ Glossary
* :ref:`Cylc User Guide <Built-in Workflow State Triggers>`


queue
internal queue
Internal queues (so called to distinguish them from external batch
queueing systems) allow you to limit how many :term:`tasks <task>` can be
Expand Down Expand Up @@ -1423,14 +1440,15 @@ Glossary
expected output
Task outputs that are not marked as :term:`optional <optional output>`
in the :term:`graph` are expected to be completed at runtime. If not, the
:term:`scheduler` retains the task as :term:`incomplete <incomplete
task>` pending user intervention.
:term:`scheduler` retains the task as :term:`incomplete` pending user
intervention.

.. seealso::

* :ref:`Cylc User Guide <expected outputs>`


incomplete
incomplete task
Incomplete tasks are :term:`tasks <task>` that finish (succeed or fail)
without completing all :term:`expected outputs <expected output>`. They
Expand Down Expand Up @@ -1539,23 +1557,38 @@ Glossary
* :ref:`Cylc User Guide <Graph Branching>`


flow number
Flow number is an integer identifier for a particular :term:`flow`
in a :term:`workflow` run.


flow
reflow
In Cylc, a *flow* is a single logical run of a :term:`workflow` that "flows"
on from some start point in the :term:`graph`.
A flow is a self-propagating run through the a Cylc :term:`workflow`
:term:`graph` starting from some initial task or tasks.

Cylc :term:`schedulers <scheduler>` can manage more than one flow in the
same graph, at the same time. We call this capability *reflow*.
Cylc :term:`schedulers <scheduler>` can manage multiple flows at once.

Flows are identified by a :term:`flow number`. The original flow
started automatically by ``cylc play`` has flow number ``1``.

.. seealso::
* :ref:`user-guide-reflow`


flow number
Flow numbers are integers passed down from parent task to child task in
the :term:`graph` as a flow progresses, to identify which :term:`flow`
(or flows) the tasks belong to. They are passed to job environments as
``$CYLC_TASK_FLOW_NUMBERS``.


flow front
Active (submitted or running) tasks, i.e. tasks in the ``n=0``
:term:`active window`, with a common :term:`flow number` comprise the
active front of that flow.


flow merge
When a :term:`flow` tries to spawn a child task and finds an instance
with the same task ID already exists in the ``n=0`` :term:`active
window`, the one active task will carry both flow numbers forward.


event
An event is a milestone in the lifecycle of a :term:`workflow` or
:term:`task` at which the :term:`scheduler` provides a hook for
Expand Down
Binary file added src/img/new-flow-n.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/img/new-flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/img/no-flow-n.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/img/no-flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/img/same-flow-behind.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/img/same-flow-n.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/img/same-flow-wait-n.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/img/same-flow-wait.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added src/img/same-flow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
11 changes: 7 additions & 4 deletions src/reference/job-script-vars/var-list.txt
Original file line number Diff line number Diff line change
Expand Up @@ -34,19 +34,22 @@ ISODATETIMEREF # Reference time for the `isodatetime` comman
# when in any datetime cycling mode

CYLC_TASK_NAME # Job's task name, e.g. t1
CYLC_TASK_ID # Task instance identifier CYCLE-POINT/TASK-NAME
# e.g. 20110511T1800Z/t1

CYLC_TASK_SUBMIT_NUMBER # Job's submit number, e.g. 1,
# increments with every submit
CYLC_TASK_TRY_NUMBER # Number of execution tries, e.g. 1
# increments with automatic
# execution retry delays.
CYLC_TASK_ID # Task instance identifier TASK-NAME.CYCLE-POINT
# e.g. t1.20110511T1800Z
# increments with automatic execution retry delays.
CYLC_TASK_FLOW_NUMBERS # Flows this task belongs to, e.g. 1,2

CYLC_TASK_LOG_DIR # Location of the job log directory
# e.g. ~/cylc-run/foo/log/job/20110511T1800Z/t1/01/
CYLC_TASK_LOG_ROOT # The task job file path
# e.g. ~/cylc-run/foo/log/job/20110511T1800Z/t1/01/job
CYLC_TASK_WORK_DIR # Location of task work directory (see below)
# e.g. ~/cylc-run/foo/work/20110511T1800Z/t1

CYLC_TASK_NAMESPACE_HIERARCHY # Linearised family namespace of the task,
# e.g. root postproc t1
CYLC_TASK_DEPENDENCIES # List of met dependencies that triggered the task
Expand Down
23 changes: 10 additions & 13 deletions src/user-guide/running-workflows/dynamic-behaviour.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,16 @@
Cylc Broadcast
--------------

The ``cylc broadcast`` command overrides :cylc:conf:`[runtime]`
settings in a running workflow.

Broadcast can be used to communicate information to downstream tasks by
setting environment variables. Broadcasts can target all tasks, groups of
tasks, or specific tasks, at all cycle points or at specific cycle points.
See broadcast command help for details.

Broadcast settings targeted at a specific task ID or cycle point expire and
are forgotten as the workflow moves on. Un-targeted variables and those
targeted at a task name persist throughout the workflow run, even across
restarts, unless manually cleared using the broadcast command - and so
should be used sparingly.
The ``cylc broadcast`` command overrides task :cylc:conf:`[runtime]`
settings in a running scheduler. You can think of it as communicating
new configuration settings (including information via environment variables) to
selected upcoming tasks via the scheduler.

See ``cylc broadcast --help`` for detailed information.

Broadcast settings targeting a specific cycle point expire as the workflow
moves on. Otherwise they persist for lifetime of the run, and across restarts,
unless cleared with another invocation of the command.


.. _Sub-Workflows:
Expand Down
8 changes: 4 additions & 4 deletions src/user-guide/running-workflows/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ Running Workflows
scheduler-start-up
task-job-states
tasks-jobs-ui
reflow
retrying-tasks
workflow-completion
tracking-task-state
authentication-files
workflow-completion
reflow
simulation-modes
scheduler-log-files
handling-job-preemption
dynamic-behaviour
authentication-files
workflow-databases
handling-job-preemption
Loading