Selection of hosts and platforms

src/7-to-8/major-changes/before-you-start.rst

@@ -0,0 +1,182 @@
+.. _AutoConfigUpgrades:
+
+Cylc Configuration Upgrader
+===========================
+
+.. admonition:: Does This Change Affect Me?
+   :class: tip
+
+   This change may affect you:
+
+   - If you are working with workflows written before Cylc 7.
+   - If you hope to use Cylc 9.
+
+   You have been affected by this change if you see a message such as:
+
+   .. code-block::
+
+      > cylc validate my_workflow
+      IllegalItemError: reallyoldconfig
+
+
+Overview
+--------
+
+Cylc has a built in configuration upgrader. **Cylc 8** can upgrade Cylc 7
+workflows at runtime, but not if validation with Cylc 7 gives deprecation warnings.
+
+Solution
+--------
+
+To avoid problems with old config items you should validate your workflow using
+Cylc 7. Look for deprecation warnings and edit the workflow configuration to
+eliminate these warnings.
+
+Example
+-------
+
+Consider this configuration:
+
+.. code-block:: cylc
+
+   [scheduling]
+       initial cycle point = 11000101T00
+       [[dependencies]]
+           [[[R1]]]
+               graph = task
+
+   [runtime]
+       [[task]]
+           pre-command scripting = echo "Hello World"
+
+Running ``cylc validate`` on this configuration at **Cylc 7** we see that the
+workflow is valid, but we are warned that ``pre-command scripting``
+was replaced by ``pre-script`` at 6.4.0:
+
+.. code-block::
+   :caption: Cylc 7 warning of a deprecated configuration
+
+   > cylc validate .
+   WARNING - deprecated items were automatically upgraded in 'suite definition':
+   WARNING -  * (6.4.0) [runtime][task][pre-command scripting] -> [runtime][task][pre-script] - value unchanged
+   Valid for cylc-7.8.7
+
+**Cylc 7** has upgraded this for us, but at **Cylc 8** this workflow will fail
+validation.
+
+.. code-block::
+   :caption: Cylc 8 failing to validate an obsolete configuration
+
+   > cylc validate .
+   WARNING - deprecated graph items were automatically upgraded in "workflow definition":
+    * (8.0.0) [scheduling][dependencies][X]graph -> [scheduling][graph]X - for X in:
+          R1
+   IllegalItemError: [runtime][task]pre-command scripting
+
+Fixing Validation Failure
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You must change the configuration yourself. In this case:
+
+.. code-block:: diff
+
+   -     pre-command scripting = echo "Hello World"
+   +     pre-script = echo "Hello World"
+
+Validation will now succeed.
+
+This will leave you with just the warning about the changes to the graph
+format: You might wish to fix this now:
+
+Fixing Deprecation Warning
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For the example given Cylc 8 will validate without warning after making the
+following changes. (explanation of
+:ref:`changes to graph section. <7-to-8.summary.graph_syntax>`
+):
+
+.. code-block:: diff
+
+   [scheduling]
+       initial cycle point = 11000101T00
+   -   [[dependencies]]
+   -       [[[R1]]]
+   -           graph = task
+   +   [[graph]]
+   +       R1 = task
+
+.. warning::
+
+   Cylc 9 will no longer automatically upgrade obsolete Cylc 7
+   configurations. It's a good idea to try and remove the configuration items
+   causing to these warnings as part of routine workflow review and
+   maintenance to avoid problems when a major Cylc version is released.
+
+Host to platform upgrade logic
+------------------------------
+
+.. seealso::
+
+   :ref:`Details of how platforms work.<MajorChangesPlatforms>`
+
+   .. TODO reference to how to write platforms page
+
+If you have a Cylc 7 workflow where tasks submit jobs to remote hosts
+Cylc 8 will attempt to find a platform which matches the task specfication.
+
+.. important::
+
+   Cylc 8 needs platforms matching the Cylc 7 job configuration to be
+   available in :cylc:conf:`global.cylc[platforms]`.
+
+Example
+^^^^^^^
+
+.. note::
+
+   In the following example ``job runner`` in **Cylc 8** configurations
+   and ``batch system`` in **Cylc 7** configurations refer to the same item.
+
+If, for example you had a **Cylc 8** ``global.cylc`` with the following
+platforms section:
+
+.. code-block:: cylc
+   :caption: Part of a Cylc global configuration
+
+   [platforms]
+       [[supercomputer_A]]
+           hosts = localhost
+           job runner = slurm
+       [[supercomputer_B]]
+           hosts = tigger, wol, eeyore
+           batch system = pbs
+
+And you have a **cylc 7** workflow runtime configuration:
+
+.. code-block:: cylc
+   :caption: Part of ``suite.rc``
+
+   [runtime]
+       [[task1]]
+           [[[job]]]
+               batch system = slurm
+       [[task2]]
+           [[[remote]]]
+               hosts = eeyore
+           [[[job]]]
+               batch system = pbs
+
+
+Then, ``task1`` will be assigned platform
+``supercomputer_A`` because the specified host (implicitly ``localhost``)
+is in the list of hosts for ``supercomputer_A`` **and** the batch system is the same.
+Likewise, ``task2`` will run on ``supercomputer_B``.
+
+.. important::
+
+   For simplicity, and because the ``host`` key is a special case (it can
+   match and host in ``[platform]hosts``) we only show these two config keys
+   here. In reality, **Cylc 8 compares the whole of**
+   ``[<task>][job]`` **and** ``[<task>][remote]``
+   **sections and all items must match to select a platform.**

src/7-to-8/major-changes/platforms.rst

@@ -16,11 +16,11 @@ Platforms
    .. code-block:: cylc
 
       [runtime]
-         [[task_name]]
-            [[[job]]]
-               batch system = slurm
-            [[[remote]]]
-               host = my_supercomputer
+          [[task_name]]
+              [[[job]]]
+                  batch system = slurm
+              [[[remote]]]
+                  host = my_supercomputer
 
    Read this section if your workflow's jobs run on a remote computer or if
    you see the following warning on running ``cylc validate``:
@@ -29,6 +29,21 @@ Platforms
 
       WARNING - Task <task>: deprecated "host" and "batch system" will be removed at Cylc 9
 
+   If you currently use the ``rose host-select`` utility or a similar host
+   selection or load balancing utility the intelligent host selection
+   functionality of Cylc 8 may be used instead:
+
+   .. code-block:: cylc
+
+      [runtime]
+          [[task_name]]
+              [[[remote]]]
+                  host = $(rose host-select my-computer)
+          [[another_task]]
+              # An example of a home-rolled host selector
+              [[[remote]]]
+                  host = $(test $((RANDOM%2)) -eq 0 && echo "host_a" || echo "host_b")
+
 
 Overview
 --------
@@ -41,6 +56,17 @@ Cylc 8 allows site administrators (and users) to configure
   multiple hosts with associated platform-specific settings. Users only need to
   select the platform for their task jobs.
 
+  Platforms also define how hosts are selected from each platform:
+
+  - Randomly (default)
+  - By definition order
+
+There may be cases where sets of platforms (for example a group of
+standalone compute servers, or a pair of mirrored HPC's) might be equally
+suitable for a task, but not share files systems to allow them to constitute
+a single platform. Such platforms can be set up to be ``platform groups``
+
+
 .. seealso::
 
    :ref:`AdminGuide.PlatformConfigs` for detailed examples of platform
@@ -57,8 +83,10 @@ Cylc 8 allows site administrators (and users) to configure
 Examples
 --------
 
-See :cylc:conf:`global.cylc[platforms]` for a detailed explanation of how
-platforms are defined.
+.. seealso::
+
+   :cylc:conf:`global.cylc[platforms]` has a detailed explanation of how
+   platforms and platform groups are defined.
 
 Showing how the global config changes
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -132,16 +160,17 @@ In Cylc 8 the equivalent might be:
 .. code-block:: cylc
 
    [runtime]
-      [[mytask_cylc_server]]
+       [[mytask_cylc_server]]
 
-      [[mytask_big_server]]
-         platform = linuxbox42
+       [[mytask_big_server]]
+           platform = linuxbox42
 
-      [[mytask_submit_local_to_remote_computer]]
-         platform = pbs_local
+       [[mytask_submit_local_to_remote_computer]]
+           platform = pbs_local
 
-      [[mytask_login_to_hpc_and_submit]]
-         platform = $(supercomputer_login_node_selector_script)
+       [[mytask_login_to_hpc_and_submit]]
+           # This is still legal, but you could also use host selection.
+           platform = $(supercomputer_login_node_selector_script)
 
 The platform settings for these examples might be:
 
@@ -152,9 +181,9 @@ The platform settings for these examples might be:
            # Without a hosts, platform name is used as a single host.
 
        [[pbs_local]]
-          job runner = pbs
-          hosts = localhost
+           job runner = pbs
+           hosts = localhost
 
        [[slurm_supercomputer]]
-          hosts = login_node01, login_node02  # Cylc will pick a host.
-          job runner = slurm
+           hosts = login_node01, login_node02  # Cylc will pick a host.
+           job runner = slurm

src/7-to-8/summary.rst

@@ -179,6 +179,8 @@ on the same platform to interact with task jobs.
    These deprecated settings will be removed at Cylc 9.
 
 
+.. _7-to-8.summary.graph_syntax:
+
 Graph Syntax
 ------------
 

src/glossary.rst

@@ -656,6 +656,11 @@ Glossary
 
       * :term:`job host`
       * :term:`job runner`
+      * :term:`platform group`
+
+   platform group
+
+      A set of :term:`platforms <platform>`
 
    scheduler
       When we say that a :term:`workflow` is "running" we mean that the cylc

src/user-guide/writing-workflows/jinja2.rst

@@ -126,6 +126,8 @@ prior to configuration parsing to provide workflow context:
 .. code-block:: sub
 
    CYLC_DEBUG                      # Debug mode, true or not defined
+   CYLC_TEMPLATE_VARS              # All variables set by -s, --set-file or
+                                   # by a plugin.
    CYLC_VERBOSE                    # Verbose mode, True or False
    CYLC_VERSION                    # Version of cylc installation used
 
@@ -239,13 +241,23 @@ Here's the result:
 
 .. _jinja2-template-variables:
 
-Jinja2 Default Values And Template Inputs
------------------------------------------
+Jinja2 Default Values And Template Variables
+--------------------------------------------
 
-The values of Jinja2 variables can be passed in from the Cylc command
-line rather than hardwired in the workflow configuration.
+You can provide template variables to Cylc in 3 ways:
 
-This can be done on a case-by-case basis using the ``-s`` option e.g:
+- Using the ``--set-file`` option.
+- Using the ``-s`` option.
+- `Using a plugin`_, such as :ref:`Cylc Rose`.
+
+.. note::
+
+   If the same variable is set by more than one method, the last source in the
+   above list is used.
+
+
+Using the ``-s`` and ``--set-file`` options
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. code-block:: console
 
@@ -348,6 +360,39 @@ will show the workflow with the Jinja2 variables as set.
    be overridden.
 
 
+Using a plugin
+^^^^^^^^^^^^^^
+
+Template plugins such as :ref:`Cylc Rose` should provide a set of template
+variables which can be provided to Cylc. For example, using Cylc Rose you
+add a ``rose-suite.conf`` file containing a ``[template variables]``
+section which the plugin makes available to Cylc:
+
+.. code-block:: ini
+   :caption: rose-suite.conf
+
+   [template variables]
+   ICP=1068
+
+.. code-block:: cylc
+   :caption: flow.cylc
+
+   #!jinja2
+   [scheduler]
+      allow implicit tasks = True
+   [scheduling]
+      initial cycle point = {{ICP}}
+      [[dependencies]]
+         P1Y = Task1
+
+
+.. code-block:: shell
+
+   > cylc config . -i "[scheduling]initial cycle point"
+   1068
+
+
+
 Jinja2 Variable Scope
 ---------------------