From 8a08d7b37bf297954d22962f50896032e26884f7 Mon Sep 17 00:00:00 2001
From: Christie Wilson <bobcatfish@gmail.com>
Date: Mon, 11 Feb 2019 15:45:16 -0800
Subject: [PATCH] =?UTF-8?q?Add=20DAG=20to=20docs=20=F0=9F=93=8E?=
=?UTF-8?q?=F0=9F=91=A3?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Update the documentation such that Tasks no longer execute in the order
they are declared in the Pipeline, order is now controlled by `from` AND
`runAfter`.
We need to add `runAfter` as part of #168 because without it, all
ordering must be expressed with `from`, which would make our examples
really gross temporarily (not to mention be a lot of work and slow down
test execution) just to remove it as soon as we add `runAfter`, so we
might as well do it all at once.
---
docs/pipelineruns.md | 15 ++---
docs/pipelines.md | 150 ++++++++++++++++++++++++++++++++++++++++---
docs/tutorial.md | 18 ++++--
test/dag_test.go | 1 +
4 files changed, 164 insertions(+), 20 deletions(-)
diff --git a/docs/pipelineruns.md b/docs/pipelineruns.md
index 72a3ab11e8e..e64a52b7749 100644
--- a/docs/pipelineruns.md
+++ b/docs/pipelineruns.md
@@ -3,9 +3,8 @@
This document defines `PipelineRuns` and their capabilities.
On its own, a [`Pipeline`](pipelines.md) declares what [`Tasks`](tasks.md) to
-run, and dependencies between [`Task`](tasks.md) inputs and outputs via
-[`from`](pipelines.md#from). To execute the `Tasks` in the `Pipeline`, you must
-create a `PipelineRun`.
+run, and [the order they run in](pipelines.md#ordering). To execute the `Tasks`
+in the `Pipeline`, you must create a `PipelineRun`.
Creation of a `PipelineRun` will trigger the creation of
[`TaskRuns`](taskruns.md) for each `Task` in your pipeline.
@@ -43,15 +42,15 @@ following fields:
object that enables your build to run with the defined authentication
information.
- `timeout` - Specifies timeout after which the `PipelineRun` will fail.
- - [`nodeSelector`] - a selector which must be true for the pod to fit on a
+ - [`nodeSelector`] - A selector which must be true for the pod to fit on a
node. The selector which must match a node's labels for the pod to be
scheduled on that node. More info:
- https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
- - [`affinity`] - the pod's scheduling constraints. More info:
- https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
+ <https://kubernetes.io/docs/concepts/configuration/assign-pod-node/>
+ - [`affinity`] - The pod's scheduling constraints. More info:
+ <https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature>
[kubernetes-overview]:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields
+ <https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields>
### Resources
diff --git a/docs/pipelines.md b/docs/pipelines.md
index 258b8ae8eaa..87a9cdb766e 100644
--- a/docs/pipelines.md
+++ b/docs/pipelines.md
@@ -9,6 +9,8 @@ This document defines `Pipelines` and their capabilities.
- [Parameters](#parameters)
- [Pipeline Tasks](#pipeline-tasks)
- [From](#from)
+ - [RunAfter](#runafter)
+- [Ordering](#ordering)
- [Examples](#examples)
## Syntax
@@ -31,9 +33,16 @@ following fields:
- [`resources`](#declared-resources) - Specifies which
[`PipelineResources`](resources.md) of which types the `Pipeline` will be
using in its [Tasks](#pipeline-tasks)
+ - `tasks`
+ - `resources`
+ - `inputs/outputs`
+ - [`from`](#from) - Used when the content of the [`PipelineResource`](resources.md)
+ should come from the [output](tasks.md#output) of a previous [Pipeline Task](#pipeline-tasks)
+ - [`runAfter`](#runAfter) - Used when the [Pipeline Task](#pipeline-task) should be executed
+ after another Pipeline Task, but there is no [output linking](#from) required
[kubernetes-overview]:
- https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields
+ <https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields>
### Declared resources
@@ -119,9 +128,9 @@ spec:
### Pipeline Tasks
-A `Pipeline` will execute a sequence of [`Tasks`](tasks.md) in the order they
-are declared in. At a minimum, this declaration must include a reference to the
-`Task`:
+A `Pipeline` will execute a graph of [`Tasks`](tasks.md) (see [ordering](#ordering)
+for how to express this graph). At a minimum, this declaration must include a
+reference to the [`Task`](tasks.md):
```yaml
tasks:
@@ -165,16 +174,21 @@ spec:
#### from
-Sometimes you will have `Tasks` that need to take as input the output of a
-previous `Task`, for example, an image built by a previous `Task`.
+Sometimes you will have [Pipeline Tasks](#pipeline-tasks) that need to take as
+input the output of a previous `Task`, for example, an image built by a previous `Task`.
-Express this dependency by adding `from` on `Resources` that your `Tasks` need.
+Express this dependency by adding `from` on [`PipelineResources`](resources.md)
+that your `Tasks` need.
- The (optional) `from` key on an `input source` defines a set of previous
`PipelineTasks` (i.e. the named instance of a `Task`) in the `Pipeline`
- When the `from` key is specified on an input source, the version of the
resource that is from the defined list of tasks is used
-- The name of the `PipelineResource` must correspond to a `PipelineResource`
+- `from` can support fan in and fan out
+- The `from` clause [expresses ordering](#ordering), i.e. the
+ [Pipeline Task](#pipeline-task) which provides the `PipelineResource` must run
+ _before_ the Pipeline Task which needs that `PipelineResource` as an input
+ - The name of the `PipelineResource` must correspond to a `PipelineResource`
from the `Task` that the referenced `PipelineTask` gives as an output
For example see this `Pipeline` spec:
@@ -201,6 +215,126 @@ The resource `my-image` is expected to be given to the `deploy-app` `Task` from
the `build-app` `Task`. This means that the `PipelineResource` `my-image` must
also be declared as an output of `build-app`.
+This also means that the `build-app` Pipeline Task will run before `deploy-app`,
+regardless of the order they appear in the spec.
+
+#### runAfter
+
+Sometimes you will need to have [Pipeline Tasks](#pipeline-tasks) that need to run in
+a certain order, but they do not have an explicit [output](tasks.md#outputs) to
+[input](tasks.md#inputs) dependency (which is expressed via [`from`](#from)). In this case
+you can use `runAfter` to indicate that a Pipeline Task should be run after one or more
+previous Pipeline Tasks.
+
+For example see this `Pipeline` spec:
+
+```yaml
+- name: test-app
+ taskRef:
+ name: make-test
+ resources:
+ inputs:
+ - name: my-repo
+- name: build-app
+ taskRef:
+ name: kaniko-build
+ runAfter:
+ - test-app
+ resources:
+ inputs:
+ - name: my-repo
+```
+
+In this `Pipeline`, we want to test the code before we build from it, but there is no output
+from `test-app`, so `build-app` uses `runAfter` to indicate that `test-app` should run before
+it, regardless of the order they appear in the spec.
+
+## Ordering
+
+The [Pipeline Tasks](#pipeline-tasks) in a `Pipeline` can be connected and run in a graph,
+specifically a *Directed Acyclic Graph* or DAG. Each of the Pipeline Tasks is a node, which
+can be connected (i.e. a *Graph*) such that one will run before another (i.e. *Directed*),
+and the execution will eventually complete (i.e. *Acyclic*, it will not get caught in infinite
+loops).
+
+This is done using:
+
+- [`from`](#from) clauses on the [`PipelineResources`](#resources) needed by a `Task`
+- [`runAfter`](#runAfter) clauses on the [Pipeline Tasks](#pipeline-tasks)
+
+For example see this `Pipeline` spec:
+
+```yaml
+- name: lint-repo
+ taskRef:
+ name: pylint
+ resources:
+ inputs:
+ - name: my-repo
+- name: test-app
+ taskRef:
+ name: make-test
+ resources:
+ inputs:
+ - name: my-repo
+- name: build-app
+ taskRef:
+ name: kaniko-build-app
+ runAfter:
+ - test-app
+ resources:
+ inputs:
+ - name: my-repo
+ outputs:
+ - name: image
+ resource: my-app-image
+- name: build-frontend
+ taskRef:
+ name: kaniko-build-frontend
+ runAfter:
+ - test-app
+ resources:
+ inputs:
+ - name: my-repo
+ outputs:
+ - name: image
+ resource: my-frontend-image
+- name: deploy-all
+ taskRef:
+ name: deploy-kubectl
+ resources:
+ inputs:
+ - name: my-app-image
+ from:
+ - build-app
+ - name: my-frontend-image
+ from:
+ - build-frontend
+```
+
+This will result in the following execution graph:
+
+```none
+ | |
+ v v
+ test-app lint-repo
+ / \
+ v v
+build-app build-frontend
+ \ /
+ v v
+ deploy-all
+```
+
+1. The `lint-repo` and `test-app` Pipeline Tasks will begin executing simultaneously.
+ (They have no `from` or `runAfter` clauses.)
+1. Once `test-app` completes, both `build-app` and `build-frontend` will begin
+ executing simultaneously (both `runAfter` `test-app`).
+1. When both `build-app` and `build-frontend` have completed, `deploy-all` will
+ execute (it requires `PipelineResources` from both Pipeline Tasks).
+1. The entire `Pipeline` will be finished executing after `lint-repo` and `deploy-all`
+ have completed.
+
## Examples
For complete examples, see
diff --git a/docs/tutorial.md b/docs/tutorial.md
index 22620971246..ddfdce7decc 100644
--- a/docs/tutorial.md
+++ b/docs/tutorial.md
@@ -328,10 +328,12 @@ resource definition.
## Pipeline
-A [`Pipeline`](pipelines.md) defines a list of tasks to execute, while also
-indicating if any outputs should be used as inputs of a following task by using
-[the `from` field](pipelines.md#from). The same templating you used in tasks is
-also available in pipeline.
+A [`Pipeline`](pipelines.md) defines a list of tasks to execute in
+order, while also indicating if any outputs should be used as inputs
+of a following task by using [the `from` field](pipelines.md#from) and
+also indicating [the order of executing (using the `runAfter` and
+`from` fields)](pipelines.md#ordering). The same templating you used
+in tasks is also available in pipeline.
For example:
@@ -612,6 +614,14 @@ annotation applies to subjects such as Docker registries, log output locations
and other nuances that may be specific to particular cloud providers or
services.
+The `TaskRuns` have been created in the following [order](pipelines.md#ordering):
+
+1. `tutorial-pipeline-run-1-build-skaffold-web` - This runs the [Pipeline Task](pipelines.md#pipeline-tasks)
+ `build-skaffold-web` first, because it has no [`from` or `runAfter` clauses](pipelines.md#ordering)
+1. `tutorial-pipeline-run-1-deploy-web` - This runs `deploy-web` second, because its [input](tasks.md#inputs)
+ `web-image` comes [`from`](pipelines.md#from) `build-skaffold-web` (therefore `build-skaffold-web`
+ must run before `deploy-web`).
+
---
Except as otherwise noted, the content of this page is licensed under the
diff --git a/test/dag_test.go b/test/dag_test.go
index 80230c8ecc7..f56738ef873 100644
--- a/test/dag_test.go
+++ b/test/dag_test.go
@@ -113,6 +113,7 @@ func TestDAGPipelineRun(t *testing.T) {
}
// FIXME(vdemeester) do the rest :)
/*
+ // TODO(christiewilson) can't actually get the logs reliably at this point, maybe write to a volume instead?
logger.Infof("Getting logs from results validation task")
// The volume created with the results will have the same name as the TaskRun
validationTaskRunName := "dag-pipeline-run-pipeline-task-4-validate-results"