Add pipeline strawman example

@dlorenc @imjasonh @tejal29 @aaron-prindle and I have been working on a strawman proposal for adding a Pipeline CRD and also for possibly envolving the Build CRD into a slightly more generic Task CRD. This PR demonstrates some paper prototype examples of what it could look like to define pipelines using the CRDs described in the README.
tektoncd · Sep 5, 2018 · 49d2316 · 49d2316
1 parent 301b413
commit 49d2316
Show file tree

Hide file tree

Showing 13 changed files with 595 additions and 2 deletions.
diff --git a/README.md b/README.md
@@ -1,2 +1,127 @@
-# build-pipeline
- A library of build pipelines
+# Pipeline CRD
+
+This repo contains the API definition of the Pipeline CRD and an on cluster implementation of that API.
+The goal of the Pipeline CRD is to provide k8s-style resources that allow the
+declaration of CI/CD-style pipelines, which can be backed by any arbitrary impelmentation.
+
+Features the Pipeline CRD will support include:
+
+* Conditional, parallel and distributed execution
+* Interaction with CI/CD resources such as source code, artifacts, resutls, deployments and clusters
+
+The goal of the Pipeline CRD is to fit into and cooperate with
+[the knative ecosystem](https://github.com/knative/docs#welcome-knative), specifically:
+
+* [The Build CRD](https://github.com/knative/docs/blob/master/build/builds.md)
+* [The Eventing APIs](https://github.com/knative/eventing/tree/master/docs/spec)
+
+_See [examples](./examples) for some examples of how this is intended to work._
+
+![Overview of the 5 CRDs](./crds.png)
+
+The CRDs involved are:
+
+* [Task](#task)
+* [Pipeline](#pipeline)
+* [PipelineParams](#pipelineparams)
+* [TaskRun](#taskrun)
+* [PipelineRun](#pipelinerun)
+
+High level details of this design:
+
+* [Pipelines](#pipelines) do not know what will trigger them, they can be
+   triggered by events or by manually creating [PipelineRuns](#pipelinerun)
+* [Tasks](#tasks) can exist and be invoked completely independently of
+  [pipelines](#pipelines); they are highly cohesive and loosely coupled
+* Test results are a first class concept, being able to navigate test results
+  easily is powerful (e.g. see failures easily, dig into logs, e.g. like
+  [the Jenkins test analyzer plugin](https://wiki.jenkins.io/display/JENKINS/Test+Results+Analyzer+Plugin))
+* [Tasks](#tasks) can depend on artifacts, output and parameters created by other tasks.
+
+## Task
+
+`Task` is a CRD that knows how to instantiate a [Knative Build](https://github.com/knative/build),
+either from a series of `steps` (i.e. [Builders](https://github.com/knative/docs/blob/master/build/builder-contract.md))
+or from a [`BuildTemplate`](https://github.com/knative/docs/blob/master/build/build-templates.md).
+It takes Knative Build and adds inputs and outputs. Where these inputs and outputs are provided
+from is not known to a task, so they can be provided by a Pipeline or by a user invoking a Task directly.
+
+`Tasks` are basically [knative BuildTemplates](https://github.com/knative/build-templates)
+with additional input types and clearly defined outputs.
+
+## Pipeline
+
+`Pipeline` describes a graph of [Tasks](#task) to execute. It defines the DAG
+and expresses how all inputs (including [PipelineParams](#pipelineparams) and outputs
+from previous `Tasks`) feed into each `Task`. It allows for fan in and fan out, and
+ordering can be expressed explicitly using `prev` and `next`, or it can be inferred
+from a `Task’s` inputs.
+
+Dependencies between parameters or inputs/outputs are expressed as references to k8s objects.
+
+## PipelineParams
+
+`PipelineParams` contains parameters for a [Pipeline](#pipeline). One `Pipeline`
+can be invoked with many different instances of `PipelineParams`, which can allow
+for scenarios such as running against PRs and against a user’s personal setup.
+`PipelineParams` can control:
+
+* What **sources** the `Pipeline` runs against
+* Which **serviceAccount** to use (provided to all tasks)
+* What **artifact** stores are used (e.g. Docker registries)
+* Where **results** are stored (e.g. in GCS)
+
+## TaskRun
+
+Creating a `TaskRun` will invoke a [Task](#task), running all of the steps until completion
+or failure. Creating a `TaskRun` will require satisfying all of the input requirements of the
+`Task`.
+
+`TaskRuns` are basically [knative Builds](https://github.com/knative/build) with inputs and
+outputs, and in the future we may want to transition `Builds` to become `Tasks`.
+
+`TaskRuns` can be created directly by a user or by a [PipelineRun](#pipelinerun).
+
+### TaskRun Status
+
+Once a `TaskRun` has been created, it will start excuting its steps
+sequentially. The `conditions` field will be updated as the `TaskRun`
+executes:
+
+* The `Started` condition will be added when the first step starts.
+* The `Completed` condition will be added when the last step completes,
+  or after a non-zero exit code from a step.
+* The `Successful` condition will be added after the `Completed`
+  condition and will indicate if the run succeeded or failed.
+
+When the `TaskRun` has completed, the `steps` field will indicate
+the exit code of all steps that completed.
+
+## PipelineRun
+
+Creating a `PipelineRun` executes the pipeline, creating [TaskRuns](#taskrun) for each task
+in the pipeline.
+
+`PipelineRuns` tie together a [Pipeline](#pipeline) and a [PipelineParam](#pipelineparam).
+A `PipelineRun` could be created:
+
+* By a user manually
+* In response to an event (e.g. in response to a Github event, possibly processed via
+  [knative eventing](https://github.com/knative/eventing))
+
+### PipelineRun Status
+
+Once a `PipelineRun` has been created, it will start excuting the DAG
+of its Tasks by creating a [`TaskRun`](#taskrun) for each of them. The
+`conditions` field will be updated as the `PipelineRun`
+executes:
+
+* The `Started` condition will be added when the first `TaskRun` is created.
+* The `Completed` condition will be added when the last `TaskRun`
+completes (or fails).
+* The `Successful` condition will be added after the `Completed`
+  condition and will indicate if the `PipelineRun` succeeded or failed.
+
+When the `PipelineRun` has completed, the `taskRuns` field will contain
+references to all `TaskRuns` which were executed and their next and
+previous `TaskRuns`.
diff --git a/crds.png b/crds.png
diff --git a/examples/README.md b/examples/README.md
@@ -0,0 +1,40 @@
+# Examples
+
+This directory contains examples of [the Pipeline strawman CRDs](../README.md) in action.
+
+## Example Tasks
+
+* Example [Tasks](../../README.md#task) are in:
+  * [build_task.yaml](build_task.yaml)
+  * [deploy_tasks.yaml](deploy_tasks.yaml)
+  * [test_tasks.yaml](test_tasks.yaml)
+
+Here are the Task Types that are defined.
+
+1. `build-push`: This task as the name suggests build an image via [kaniko](https://github.com/GoogleContainerTools/kaniko) and pushes it to registry.
+2. `make`:  Runs make target.
+3. `integration-test-in-docker`: This is a new task that is used in the sample pipelines to test an app in using `docker build` command to build an image with has the integration test code.
+This task then calls `docker run` which will run the test code. This follows the steps we have for [kritis integration test](https://github.com/grafeas/kritis/blob/4f83f99ca58751c28c0ec40016ed0bba5867d70f/Makefile#L152)
+4. `deploy-with-helm`: This task deploys a kubernetes app with helm.
+5. `deploy-with-kubectl`: This task deploys with kubectl apply -f <filename>
+
+### Example Runs
+
+The [runs](./runs/) dir contains an example [TaskRun](../README.md#taskrun) and an example [PipelineRun](../README.md#pipelinerun).
+
+[run-kritis-test.yaml](./invocations/run-kritis-test.yaml) shows an example of how to manually run kritis unit test off your development branch.
+
+[kritis-pipeline-run.yaml](./invocations/kritis-pipeline-run.yaml) shows an example of what it would look like to invoke the [kritis example pipeline](#example-pipelines) manually and have it fail on the second task (building and pushing the image).
+
+## Example Pipelines
+
+Finally, we have 2 example [Pipelines](../README.md#pipeline) in [./pipelines](./pipelines)
+
+1. [Kritis](./pipelines/kritis.yaml): This exmaple demonstrates how to configure a pipeline which runs unit test, build an image, deploys it to test and then run integration tests. (This is the pipeline for [kritis](https://github.com/grafeas/kritis).)
+
+![Pipeline Configuration](./pipelines/kritis-pipeline.png)
+
+2. [Guestbook](./pipelines/guestbook.yaml): This is pipeline which is based on example application in [Kubernetes example Repo](https://github.com/kubernetes/examples/tree/master/guestbook)
+This pipeline demonstartes how to integrate frontend [guestbook app code](https://github.com/kubernetes/examples/tree/master/guestbook-go) with backed [redis-docker image](https://github.com/GoogleCloudPlatform/redis-docker/tree/master/4) provided by GCP.
+
+![Pipeline Configuration](./pipelines/guestbook-pipeline.png)
diff --git a/examples/build_task.yaml b/examples/build_task.yaml
@@ -0,0 +1,25 @@
+apiVersion: pipeline.knative.dev/v1beta1
+kind: Task
+metadata:
+  name: build-push
+  namespace: default
+spec:
+    inputs:
+        sources:
+            - name: workspace
+        params:
+            - name: pathToDockerFile
+              type: string
+    outputs:
+        artifacts:
+            - name: builtImage
+              type: image
+              storeKey: registry
+    buildSpec:
+        template:
+            name: kaniko
+            arguments:
+                - name: DOCKERFILE
+                  value: ${pathToDockerFile}
+                - name: REGISTRY
+                  value: ${registry}
diff --git a/examples/deploy_tasks.yaml b/examples/deploy_tasks.yaml
@@ -0,0 +1,47 @@
+apiVersion: pipeline.knative.dev/v1beta1
+kind: Task
+metadata:
+  name: deploy-with-helm
+  namespace: default
+spec:
+    inputs:
+        sources:
+            - name: workspace
+        params:
+            - name: pathToHelmCharts
+              type: string
+            - name: helmArgs
+              type: string
+            - name: image
+              type: string
+        cluster:
+            - name: clusterName
+    buildSpec:
+        steps:
+        - name: deploy
+          image: kubernetes-helm
+          args: ['deploy', '--path=${pathToHelmChart}', '--set image=${image}', '${helmArgs}']
+
+---
+apiVersion: pipeline.knative.dev/v1beta1
+kind: Task
+metadata:
+  name: deploy-with-kubectl
+  namespace: default
+spec:
+    inputs:
+        sources:
+            - name: workspace
+              type: string
+        params:
+            - name: kubectlArgs
+              type: string
+            - name: pathToFiles
+              type: string
+        cluster:
+            - name: targetCluster
+    buildSpec:
+        steps:
+        - name: runKubectl
+          image: k8s-kubectl
+          args: ['--use-context', '${targetCluster}', '--namespace', '${targetCluster.namespace}', 'apply', '-c', '${pathToFiles}', '${kubectlArgs}']
diff --git a/examples/invocations/kritis-pipeline-run.yaml b/examples/invocations/kritis-pipeline-run.yaml
@@ -0,0 +1,45 @@
+apiVersion: pipeline.knative.dev/v1beta1
+kind: PipelineRun
+metadata:
+  name: kritis-pipeline-run-12321312984
+  namespace: default
+spec:
+    pipelineRef:
+        name: kritis-pipeline
+    pipelineParamsRef:
+        name: pipelineparams-sample
+    trigger:
+        triggerRef:
+            type: manual
+# The status section will be filled in by the PipelineRun controller once it
+# has started invoking the PipelineRun.
+status:
+  taskRuns:
+    - taskRef:
+        name: unit-test-kritis-12321312984
+        nextTasks:
+          - taskRef:
+              name: push-kritis-12321312984
+        prevTasks: []
+    - taskRef:
+        name: push-kritis-12321312984
+        nextTasks: []
+        prevTasks:
+          - taskRef:
+            name: unit-test-kritis-12321312984
+  conditions:
+    - type: Started
+      status: True
+      lastTransitionTime: 1534204248
+      reason: manualTrigger
+      message: "Pipeline has been triggered manually"
+    - type: Completed
+      status: True
+      lastTransitionTime: 1535000000
+      reason: done
+      message: "Pipeline execution has finished"
+    - type: Successful
+      status: False
+      lastTransitionTime: 1535000000
+      reason: taskFailure
+      message: "TaskRun `push-kritis-12321312984` had non-zero exit code"
diff --git a/examples/invocations/run-kritis-test.yaml b/examples/invocations/run-kritis-test.yaml
@@ -0,0 +1,59 @@
+apiVersion: pipeline.knative.dev/v1beta1
+kind: TaskRun
+metadata:
+  labels:
+    controller-tools.k8s.io: "1.0"
+  name: unit-test-kritis-feature-x
+spec:
+    taskRef:
+        name: make
+    trigger:
+        triggerRef:
+            type: manual
+    inputs:
+        sources:
+            - name: 'kritis'
+              type: 'github'
+              url:  'github.com/grafeas/kritis'
+              branch: 'featureX'
+              commit: 'HEAD'
+        params:
+            - name: 'makeTarget'
+              type: 'string'
+              value: 'test'
+    results:
+        runs:
+        - name: 'runsBucket'
+          type: 'gcs'
+          url: 'gcs://somebucket/results/runs'
+        logs:
+        - name: 'logBucket'
+          type: 'gcs'
+          url: 'gcs://somebucket/results/logs'
+        tests:
+        - name: 'testBucket'
+          type: 'gcs'
+          url: 'gcs://somebucket/results/tests'
+# The status section will be filled in by the TaskRun controller once it
+# has started invoking the TaskRun.
+status:
+    steps:
+    - name: make
+      logsURL: 'gcs://somebucket/results/tests/unit-test-kritis-feature-x/make'
+      exitCode: 0
+    conditions:
+    - type: Started
+      status: True
+      lastTransitionTime: 1534204250
+      reason: pipelineRun
+      message: "Task has been triggered by a Pipeline run"
+    - type: Completed
+      status: True
+      lastTransitionTime: 1534250000
+      reason: done
+      message: "Pipeline execution has finished"
+    - type: Successful
+      status: True
+      lastTransitionTime: 1534250000
+      reason: zeroExitCode
+      message: "All steps completed with an exit code of 0"