TEP-0100: Embedded TaskRuns and Runs Status in PipelineRuns [Problem …

…Statement]

This TEP proposes changes to `PipelineRun` Status to reduce the amount of
information stored about the status of its `Run`s and `TaskRun`s to improve
performance, reduce memory bloat and improve extensibility.

Co-authored-by: Lee Bernick <leebernick@google.com>

teps/0100-embedded-taskruns-and-runs-status-in-pipelineruns.md

@@ -0,0 +1,85 @@
+---
+status: proposed
+title: Embedded TaskRuns and Runs Status in PipelineRuns
+creation-date: '2022-01-24'
+last-updated: '2022-01-24'
+authors:
+- '@jerop'
+- '@lbernick'
+---
+
+# TEP-0100: Embedded TaskRuns and Runs Status in PipelineRuns
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+  - [Requirements](#requirements)
+- [References](#references)
+<!-- /toc -->
+
+## Summary
+
+This TEP proposes changes to `PipelineRun` Status to reduce the amount of
+information stored about the status of its `Run`s and `TaskRun`s to improve
+performance, reduce memory bloat and improve extensibility.
+
+## Motivation
+
+Today, we embed the status of each `TaskRun` and `Run` into the `PipelineRun`
+status. This causes several issues:
+1. **Performance degradation**. Every time a `TaskRun`'s or `Run`'s status changes,
+   the parent `PipelineRun`'s status is updated as well. For example, the `PipelineRun`'s
+   status is updated upon completion of each `Step`, even if the `TaskRun` or `Run`
+   has not completed. This causes extra requests to etcd and extra load on the Dashboard,
+   which reacts to CRD status updates. Read more in the [related issue][issue-3140].
+2. **Memory bloat**. Embedded statuses increase the size of the serialized `PipelineRuns`.
+   As shared in [API WG on 10/01/2022][api-wg], the embedded statuses:
+   > "more than doubles the storage and has direct consequences on what customers end up paying".
+3. **Lack of extensibility**. The above problems will be exacerbated by features that will execute
+   multiple `TaskRuns` and `Runs` from one `PipelineTask`. For example:
+   * [**`Matrix`**][tep-0090]: fan out a given `PipelineTask` into multiple `TaskRuns` or `Run`s.
+     Matrix `Run`s can even be created dynamically by consuming `Results` of previous `Run`s
+   * [**`Pipelines` in `Pipelines`**][tep-0056]: pass in `Pipelines` to `PipelineTasks` so we can
+     run them similarly to `Tasks`.
+
+### Goals
+
+- Reduce updates to `PipelineRun` status and the amount of storage it uses.
+- Set up `PipelineRun` status to better support future extensions in Tekton Pipelines.
+
+### Non-Goals
+
+- Improving other aspects of `PipelineRun` status other than the embedding of `TaskRuns` and
+  `Runs`.
+
+### Requirements
+
+`PipelineRun` status must:
+- show the`TaskRuns` or `Runs` that were executed.
+- show the execution status of each `TaskRun` (e.g. Successful, Failed).
+- support future extensions that would create even more `TaskRuns` and `Runs`.
+- optimize for performance and storage.
+
+## References
+
+<!--
+Use this section to add links to GitHub issues, other TEPs, design docs in Tekton
+shared drive, examples, etc. This is useful to refer back to any other related links
+to get more details.
+-->
+
+Issues:
+- [PipelineRun and TaskRun Status][issue-3792]
+- [PipelineRun Status bloat and excessive updates][issue-3140]
+
+TEPs:
+- [Pipelines in Pipelines][tep-0056]
+- [Matrix][tep-0090]
+
+[tep-0056]: https://github.com/tektoncd/community/blob/main/teps/0056-pipelines-in-pipelines.md
+[tep-0090]: https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md
+[issue-3140]: https://github.com/tektoncd/pipeline/issues/3140
+[issue-3792]: https://github.com/tektoncd/pipeline/issues/3792
+[api-wg]: https://docs.google.com/document/d/17PodAxG8hV351fBhSu7Y_OIPhGTVgj6OJ2lPphYYRpU/edit#heading=h.esbaqjpyouim

teps/README.md

@@ -234,3 +234,4 @@ This is the complete list of Tekton teps:
 |[TEP-0090](0090-matrix.md) | Matrix | proposed | 2021-11-08 |
 |[TEP-0094](0094-configuring-resources-at-runtime.md) | Configuring Resources at Runtime | proposed | 2021-11-08 |
 |[TEP-0096](0096-pipelines-v1-api.md) | Pipelines V1 API | proposed | 2021-12-13 |
+|[TEP-0100](0100-embedded-taskruns-and-runs-status-in-pipelineruns.md) | Embedded TaskRuns and Runs Status in PipelineRuns | proposed | 2022-01-24 |