diff --git a/teps/0130-pipeline-level-docker-daemon.md b/teps/0130-pipeline-level-docker-daemon.md new file mode 100644 index 000000000..953e1fe85 --- /dev/null +++ b/teps/0130-pipeline-level-docker-daemon.md @@ -0,0 +1,409 @@ +--- +status: proposed +title: Pipeline-level Docker Daemon +creation-date: '2023-01-24' +last-updated: '2023-01-24' +authors: +- '@lbernick' +collaborators: [] +--- + +# TEP-0130: Pipeline-level Docker Daemon + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) + - [Use Cases](#use-cases) + - [Requirements](#requirements) +- [Proposal](#proposal) + - [Notes and Caveats](#notes-and-caveats) +- [Design Details](#design-details) +- [Design Evaluation](#design-evaluation) + - [Reusability](#reusability) + - [Simplicity](#simplicity) + - [Flexibility](#flexibility) + - [User Experience](#user-experience) + - [Performance](#performance) + - [Risks and Mitigations](#risks-and-mitigations) + - [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Implementation Plan](#implementation-plan) + - [Test Plan](#test-plan) + - [Infrastructure Needed](#infrastructure-needed) + - [Upgrade and Migration Strategy](#upgrade-and-migration-strategy) + - [Implementation Pull Requests](#implementation-pull-requests) +- [References](#references) + + +## Summary + +This TEP proposes support for more complex Docker-related use cases by allowing multiple Tasks +in a PipelineRun to use the same docker daemon. + +## Motivation + +The docker-build catalog Task only allows building a single image and pushing it to a registry. +The following use cases are not addressed: +- Building multiple images concurrently with the same daemon to save resources and share cached base layers +- Introducing additional steps, such as testing, in between a build and a push + - A user may also want to guard guard execution of the push based on the results of those tasks + - A user may also want to use the built image in these subsequent tasks without having to first push to and pull from a registry; + for example, if using `docker exec` or `docker-compose` + +Allowing multiple Tasks in a PipelineRun to use the same docker daemon allows PipelineRuns with +docker builds to take advantage of Pipeline features (e.g. using matrix to build multiple images, +executing a push conditionally on previous steps) while still having access to the same local +cache of built images and avoid having to push to a registry in between. + +### Goals + +- More flexible docker build Pipelines + +### Non-Goals + +- Sharing a daemon or build cache between PipelineRuns + +### Requirements + +- A docker image built during a Pipeline can be used in a subsequent Task without first pushing it to an image registry + +## Proposal + +Add support for Sidecars with the same lifespan as PipelineRuns, for example: + +```yaml +kind: Pipeline +spec: + params: + - name: image + - name: docker-host + value: $(sidecars.docker-daemon.host):2376 + workspaces: + - name: source + - name: docker-tls-certs + results: + - name: image-digest + value: $(tasks.docker-build.results.IMAGE_DIGEST) + tasks: + - name: git-clone + ... + - name: docker-build + params: + - name: image + - name: docker-host + workspaces: + - name: source + taskSpec: + workingDir: $(workspaces.source.path) + steps: + - image: docker.io/library/docker + script: docker build -f ./Dockerfile -t $(params.image) + env: + - name: DOCKER_HOST + value: $(params.docker-host) + results: + - name: IMAGE_DIGEST + runAfter: ["git-clone"] + # This task uses the built image without first pushing it to a registry + - name: pytest + params: + - name: docker-host + - name: image + taskSpec: + workingDir: $(workspaces.source.path) + steps: + - image: docker.io/library/docker + script: docker exec $(params.image) python -m pytest + env: + - name: DOCKER_HOST + value: $(params.docker-host) + runAfter: ["docker-build"] + # This task pushes the image to the registry only if tests passed + - name: docker-push + when: + - input: "$(tasks.pytest.status)" + operator: in + values: ["Succeeded"] + params: + - name: image + - name: docker-host + taskSpec: + steps: + - image: docker.io/library/docker + script: docker push $(params.image) + env: + - name: DOCKER_HOST + value: $(params.docker-host) + runAfter: ["pytest"] + sidecars: + - name: docker-daemon + image: docker:dind + securityContext: + privileged: true + ports: + - 2376 + startupProbe: + tcpSocket: + port: 2376 + workspaces: + - name: docker-tls-certs +``` + +Here's what the [docker-build catalog task](https://github.com/tektoncd/catalog/blob/81bf7dc5610d5fa17281940a72a6377604105cea/task/docker-build/0.1/docker-build.yaml) +might look like with a separate sidecar (note that the Task only builds now and does not push to a registry): + +```yaml +apiVersion: tekton.dev/v1beta1 +kind: Task +metadata: + name: docker-build +spec: + params: + - name: image + description: Reference of the image docker will produce. + - name: docker_host # This is new + description: The address of the docker daemon to use. + - name: builder_image + description: The location of the docker builder image. + default: docker.io/library/docker:stable@sha256:18ff92d3d31725b53fa6633d60bed323effb6d5d4588be7b547078d384e0d4bf #tag: stable + - name: dockerfile + description: Path to the Dockerfile to build. + default: ./Dockerfile + - name: context + description: Path to the directory to use as context. + default: . + workspaces: + - name: source + - name: docker-tls-certs # This is new + optional: true + results: + - name: IMAGE_DIGEST + description: Digest of the image just built. + steps: + - name: docker-build + image: $(params.builder_image) + env: + # Connect to the sidecar over TCP, with TLS. + - name: DOCKER_HOST + value: $(params.docker_host) + - name: DOCKER_TLS_VERIFY + value: '1' + - name: DOCKER_CERT_PATH + value: /certs/client + workingDir: $(workspaces.source.path) + script: | + docker build \ + --no-cache \ + -f $(params.dockerfile) -t $(params.image) $(params.context) + workspaces: + - name: docker-tls-certs + mountPath: /certs/client +``` + +### Notes and Caveats + +Open questions: +- Should Pipeline-level sidecars terminate before or after Finally Tasks? +- Would we want to introduce the concept of a reusable sidecar? (See [Reusability](#reusability) for more info) +- What level of customization do we want to provide for creating resources with the same lifespan as the PipelineRun? + - Should we allow Custom Tasks as sidecars? What about arbitrary k8s resources? +- If a user specifies multiple sidecars, should they run in one pod or separate pods? + - If the latter, do we want to provide a way to have multiple sidecar containers in one pod? + +#### Other uses for this feature + +A few other uses for pipeline-level sidecars have been proposed, but docker builds are the highest priority use case identified so far. + +- **Integration tests**: This feature may also be useful for integration tests that use a sidecar as a test database. +However, integration tests are likely limited to a single Task; i.e. it's unlikely that a user would want multiple integration testing Tasks +to share the same test database. A test database Pipeline-level sidecar could be limited to one Task (or a subset of Tasks) using +[Pipelines in Pipelines](./0056-pipelines-in-pipelines.md), or this use case could be met by allowing Pipelines or TaskRuns +to specify Task-level sidecars, as proposed in +[TEP-0126: Allow Task sidecars to be specified in PipelineRun](https://github.com/tektoncd/community/pull/877). + +- **Heartbeats**: The [original feature request](https://github.com/tektoncd/pipeline/issues/2973) for Pipeline-level sidecars was within a Pipeline +that uses Boskos resources. The first Task acquires the resources and creates a pod to send heartbeats to Boskos; intermediate Tasks use the Boskos resources, +and a Finally Task would release the lease on the resources and clean up the pod sending heartbeats. +One user [suggested](https://github.com/tektoncd/pipeline/issues/2973#issuecomment-672152635) that Boskos usage should be restricted to a subset of Tasks, +which could be done with Pipelines in Pipelines. + +#### Related features in other CI/CD systems + +A few other CI/CD systems have sidecar-like features. + +The closest feature to Pipeline-level sidecars is Argo Workflows [daemon containers](https://argoproj.github.io/argo-workflows/walk-through/daemon-containers/) feature. +Daemon containers share the same lifespan as a Template, which can be analogous to a Task or a Pipeline. (Argo Workflows also has a [sidecar feature](https://argoproj.github.io/argo-workflows/walk-through/sidecars/) similar to Tekton's Task sidecars.) + +Other CI/CD systems have features that are more analogous to Task sidecars in Tekton, and designed with integration tests in mind: +* GitHub Actions [service containers](https://docs.github.com/en/actions/using-containerized-services/about-service-containers) +* GitLab [services](https://docs.gitlab.com/ee/ci/services/) + +Both [GitLab](https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-the-docker-executor-with-docker-in-docker) +and [CircleCI](https://circleci.com/docs/executor-intro/#docker) provide Docker "executors" that can be used for running Docker builds. + +## Design Details + + + +This needs to be fleshed out in more detail, but the sidecar would likely be implemented as a deployment with one pod plus a service. +If the sidecar has readiness or startup probes, the PipelineRun would wait for them to succeed before starting the Pipeline Tasks. + +## Design Evaluation + +### Reusability + +A few sidecar configurations, such as docker daemons and database containers, likely make up the majority of use cases. +A major open question of this proposal is whether we want to introduce a concept of a reusable sidecar. +For example, we could include a docker daemon sidecar in the catalog, +and allow Pipelines to contain inline sidecar specifications or references to sidecars +(stored on the hub, on the cluster, in git, etc, just like Tasks). +Tasks would then need to document that they need a sidecar to work correctly (likely by accepting the sidecar address as a required parameter). + +### Simplicity + +The current user experience isn't flexible enough for many common docker-related use cases involving more complex logic than a build and push. + +### Flexibility + +Allowing users to specify Pipeline-level sidecars is more flexible than first class support for building images, +like Tekton previously had with the Image PipelineResource. +It's an open question whether we'd like to allow more flexibility in terms of other resources (such as custom Tasks) +with the same lifespan as PipelineRuns. + +### Conformance + +Sidecar configuration already contains some Kubernetes-specific configuration, +such as liveness/readiness probes. This proposal doesn't introduce new Kubernetes-specific +configuration, but it allows it to be specified in Pipelines as well as Tasks. + +### User Experience + + + +### Performance + + + +### Risks and Mitigations + + + +### Drawbacks + + + +- If docker TLS certs are used, they must now be bound in a persistent volume claim to share +between multiple Tasks, rather than an emptyDir volume to share between multiple steps. +This is a pretty heavyweight option for a relatively small amount of data. +However, this is more an artifact of the limited options Pipelines currently has for sharing +data between Tasks, which should be treated as a separate problem. + +## Alternatives + + + +Possible alternatives: +- User writes a Task at the beginning of the PipelineRun to start up a sidecar service and a Finally Task to tear it down + (i.e. we do nothing except maybe improve our documentation) +- First-class docker support in the API; e.g. a user can specify that they'd like their Pipeline to have a docker daemon provided + +## Implementation Plan + + + +### Test Plan + + + +### Infrastructure Needed + + + +### Upgrade and Migration Strategy + + + +### Implementation Pull Requests + + + +## References + +- [[FR] Pipeline-level sidecars](https://github.com/tektoncd/pipeline/issues/5112) +- [Pipeline level sidecar](https://github.com/tektoncd/pipeline/issues/2973) diff --git a/teps/README.md b/teps/README.md index 8c37555db..5b64b00c8 100644 --- a/teps/README.md +++ b/teps/README.md @@ -118,3 +118,4 @@ This is the complete list of Tekton TEPs: |[TEP-0125](0125-add-credential-filter-to-entrypoint-logger.md) | Add credential filter to entrypoint logger | proposed | 2022-10-27 | |[TEP-0127](0127-larger-results-via-sidecar-logs.md) | Larger Results via Sidecar Logs | implemented | 2022-12-15 | |[TEP-0128](0128-scheduled-runs.md) | Scheduled Runs | implementable | 2022-12-20 | +|[TEP-0130](0130-pipeline-level-docker-daemon.md) | Pipeline-level Docker Daemon | proposed | 2023-01-24 |