-
Notifications
You must be signed in to change notification settings - Fork 222
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[TEP-0111] Remove volumes from Task spec
This TEP proposes removing volumes from the Task spec in the v1 API, as well as removing volumeMounts and volumeDevices for the container-like fields (Step, StepTemplate, and Sidecar).
- Loading branch information
Showing
3 changed files
with
255 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,252 @@ | ||
| --- | ||
| status: implementable | ||
| title: Remove Volumes from Task Spec | ||
| creation-date: '2022-06-02' | ||
| last-updated: '2022-06-02' | ||
| authors: | ||
| - '@lbernick' | ||
| --- | ||
|
|
||
| # TEP-0111: Remove Volumes from Task Spec | ||
|
|
||
| <!-- toc --> | ||
| - [Summary](#summary) | ||
| - [Motivation](#motivation) | ||
| - [Goals](#goals) | ||
| - [Non-Goals](#non-goals) | ||
| - [Use Cases supported by Workspaces](#use-cases-supported-by-workspaces) | ||
| - [Providing secrets or configuration in a specified file](#providing-secrets-or-configuration-in-a-specified-file) | ||
| - [Caching data across multiple Steps](#caching-data-across-multiple-steps) | ||
| - [Docker in Docker (most CI/CD use cases)](#docker-in-docker-most-ci-cd-use-cases) | ||
| - [Use cases not currently supported by Workspaces](#use-cases-not-currently-supported-by-workspaces) | ||
| - [Docker in Docker (advanced use cases)](#docker-in-docker-advanced-use-cases) | ||
| - [Tasks that require block devices](#tasks-that-require-block-devices) | ||
| - [Proposal](#proposal) | ||
| - [Support for hostPath volumes in Workspace Bindings](#support-for-hostpath-volumes-in-workspace-bindings) | ||
| - [Design Considerations](#design-considerations) | ||
| - [In-tree support for many types of Volumes in Workspace Bindings](#in-tree-support-for-many-types-of-volumes-in-workspace-bindings) | ||
| - [User experience with Workspaces](#user-experience-with-workspaces) | ||
| - [Design Evaluation](#design-evaluation) | ||
| - [Reusability](#reusability) | ||
| - [Simplicity](#simplicity) | ||
| - [Flexibility](#flexibility) | ||
| - [Conformance](#conformance) | ||
| - [Risks and Mitigations](#risks-and-mitigations) | ||
| - [Alternatives](#alternatives) | ||
| - [Use TaskRun.Spec.PodTemplate to support missing Workspace features provided by Volumes](#use-taskrunspecpodtemplate-to-support-missing-workspace-features-provided-by-volumes) | ||
| <!-- /toc --> | ||
|
|
||
| ## Summary | ||
|
|
||
| This TEP proposes removing Volumes from the Task spec, and removing VolumeMounts and VolumeDevices from Step, | ||
| StepTemplate, and Sidecar. Users are encouraged to use Workspaces as a replacement feature. | ||
|
|
||
| ## Motivation | ||
|
|
||
| One of Tekton's design principles is to avoid having Kubernetes concepts in its API, so that it can be implemented | ||
| on other platforms or so that its Kubernetes implementation can be changed. Using Workspaces in the Pipelines API moves | ||
| the project in this direction by abstracting details of data storage out of Tasks and Pipelines. | ||
|
|
||
| Some fields in Task, Step, StepTemplate, and Sidecar exist for backwards compatibility (for example, because Tekton used to | ||
| embed Kubernetes' container definition in Step, rather than defining its own Step struct), not because they were intentionally introduced to meet a use case. The V1 API provides an opportunity to reassess whether these fields should have a place in the API. | ||
|
|
||
| ### Goals | ||
|
|
||
| - Make the Tekton API less tied to a Kubernetes based implementation. | ||
| - Promote workspaces, rather than volumes, as the preferred approach for Tasks to read input data and write output data. | ||
| (Results are still an appropriate method to pass small amounts of data between Tasks; this TEP does not affect results.) | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| - Update Workspace Bindings to natively support all types of Kubernetes volumes. | ||
|
|
||
| ### Use Cases supported by Workspaces | ||
|
|
||
| The following section describes existing use cases for volumes in Tasks and volumeMounts and volumeDevices | ||
| in Steps, Sidecars, and StepTemplates. It describes how these use cases are met by workspaces. | ||
|
|
||
| All usages of volumes in the Catalog fall into one of these use cases, with the exception of hostPath volumes | ||
| (see [Docker in Docker (Advanced Use Cases)](#docker-in-docker-advanced-use-cases) for more info). | ||
|
|
||
| #### Providing secrets or configuration in a specified file | ||
|
|
||
| Example use cases for mounting Kubernetes secrets or configMaps at a given location include an SSH config | ||
| for a git clone Task, or a config.json file for a Docker build. Instead of mounting the secret or configMap | ||
| directly, the Task can [declare a workspace at the necessary mountPath](https://tekton.dev/docs/pipelines/workspaces/#using-workspaces-in-tasks), | ||
| and the TaskRun can supply a secret or configMap in the [workspace binding](https://tekton.dev/docs/pipelines/workspaces/#specifying-volumesources-in-workspaces). | ||
|
|
||
| These secrets can also be isolated to only the Steps that need them using | ||
| [isolated workspaces](https://tekton.dev/docs/pipelines/workspaces/#isolating-workspaces-to-specific-steps-or-sidecars) | ||
| (note: currently in alpha). | ||
|
|
||
| #### Caching data across multiple Steps | ||
|
|
||
| An example use case for caching data across multiple steps is image build Tasks that cache image layers. | ||
| This use case can be met using a workspace with an [emptyDir binding](https://tekton.dev/docs/pipelines/workspaces/#emptydir). | ||
|
|
||
| #### Docker in Docker (most CI/CD use cases) | ||
|
|
||
| Docker-in-Docker Tasks typically run a Docker daemon in a Sidecar. | ||
| [Workspaces may be mounted into Sidecars](https://tekton.dev/docs/pipelines/workspaces/#sharing-workspaces-with-sidecars), | ||
| a feature that allows data (such as TLS certificates) to be shared between the Docker daemon and the build Step. | ||
| (Note: workspaces in sidecars are currently in alpha.) | ||
|
|
||
| Most CI/CD use cases, such as those that involve building Docker images, can be supported | ||
| by running a Docker daemon in a sidecar (the approach used by the catalog | ||
| [docker-build Task](https://github.com/tektoncd/catalog/blob/main/task/docker-build/0.1/docker-build.yaml)). | ||
| The build container can connect to the sidecar via TLS, or by mounting an emptyDir volume | ||
| to access the sidecar's docker socket. | ||
| This approach requires using privileged containers, but does not require mounting any files | ||
| from the host node. | ||
| No image build Tasks in the Catalog mount files from host nodes. | ||
|
|
||
| ### Use cases not currently supported by Workspaces | ||
|
|
||
| #### Docker in Docker (advanced use cases) | ||
|
|
||
| Some users would like to be able to access the host node's image cache for performing Docker builds. | ||
| This requires mounting the host's | ||
| [docker socket](https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option). | ||
| The most natural option to do so is a [hostPath volume](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath). | ||
| Workspaces do not support hostPath volumes as bindings, so this feature would not be supported | ||
| by Workspaces. However, this approach is not recommended for | ||
| [security reasons](https://blog.quarkslab.com/kubernetes-and-hostpath-a-love-hate-relationship.html). | ||
|
|
||
| The only Catalog Task using hostPath volumes is [`kind`](https://hub.tekton.dev/tekton/task/kind). | ||
| For more information on why `kind` on Kubernetes requires hostPath volumes, see | ||
| https://github.com/kubernetes-sigs/kind/issues/303#issuecomment-521384993 and | ||
| [Running KIND Inside A Kubernetes Cluster For Continuous Integration](https://d2iq.com/blog/running-kind-inside-a-kubernetes-cluster-for-continuous-integration) | ||
| (TL;DR: because `kind` needs to create nested Docker containers). | ||
|
|
||
| #### Tasks that require block devices | ||
|
|
||
| [Block devices](https://kubernetes.io/blog/2019/03/07/raw-block-volume-support-to-beta/) support random access to data in | ||
| fixed-size blocks. The most common use case for block devices is in database implementations, | ||
| which is not very relevant for the CI/CD use cases Tekton aims to support. | ||
|
|
||
| One CI/CD use case for block devices in Tasks is that [some image builds require them](https://github.com/tektoncd/pipeline/issues/1438#issuecomment-544313339). | ||
| However, we have received only one request for support for block devices in workspaces, which may indicate that not many | ||
| users need this feature. (It could also mean that people are happily using `volumeDevices` in `Steps` or that they just | ||
| did not see the linked issue.) There are no catalog Tasks using this feature. | ||
|
|
||
| Some CSI drivers support block devices (see | ||
| [in-tree support for volume types](#in-tree-support-for-many-types-of-volumes-in-workspace-bindings) for more info). | ||
|
|
||
| ## Proposal | ||
|
|
||
| 1. Allow workspaces to support the common CI/CD use cases Tekton targets, especially Docker-in-Docker. | ||
| - Promote workspaces in sidecars and isolated workspaces to beta. | ||
| - Optionally, add support for [CSI workspace bindings](#in-tree-support-for-many-types-of-volumes-in-workspace-bindings). | ||
| 1. Update examples our documentation to demonstrate the use of workspaces instead of volumes for the use cases listed in | ||
| this TEP. | ||
| - The v1beta1 -> v1 migration guide should include examples of migrating volumes to workspaces. | ||
| 1. Deprecate `Task.Spec.Volumes`, `Step.VolumeMounts`, and `Step.VolumeDevices` (and likewise for `StepTemplate` and `Sidecar`). | ||
| - These fields should be marked as deprecated in v1beta1, meaning that support for these fields may be removed after 9 months. | ||
| - These fields should not be included in the V1 API. | ||
|
|
||
| ## Design Considerations | ||
|
|
||
| ### In-tree support for many types of Volumes in Workspace Bindings | ||
|
|
||
| Kubernetes has experienced [extensibility issues](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/#why-csi) | ||
| providing in-tree support for many types of Volumes, a clear sign that Tekton should not attempt to do the same | ||
| for Workspace Bindings. To address these issues, Kubernetes has implemented support for the Container Storage Interface (CSI). | ||
| CSI support allows vendors to develop their own storage plugins for use with Kubernetes. | ||
|
|
||
| [CSI volumes](https://kubernetes.io/docs/concepts/storage/volumes/#csi) can be used in a Pod via a PersistentVolumeClaim or | ||
| a generic or CSI-specific ephemeral volume. | ||
| Workspace Bindings already support PersistentVolumeClaims, meaning that CSI volumes can be used in Workspaces. | ||
| We may want to support [CSI ephemeral volumes](https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volumes) | ||
| as workspace backings as well. One use case for CSI ephemeral volumes is to be able to use Hashicorp Vault to mount secrets | ||
| into a Task's pod, as requested in [#4446](https://github.com/tektoncd/pipeline/issues/4446). | ||
|
|
||
| In the future, if users request it, we can also consider supporting | ||
| [generic ephemeral volumes](https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes). | ||
|
|
||
| ### User experience with Workspaces | ||
|
|
||
| Some users have had mixed experiences with Workspaces for reasons detailed in [TEP-0044](./0044-data-locality-and-pod-overhead-in-pipelines.md), | ||
| including the difficulty of moving data between Tasks without PipelineResources (deprecated), and the latency | ||
| associated with dynamically creating PVCs and attaching them to TaskRun pods. These issues aren't related to this proposal. | ||
|
|
||
| ### Drawbacks | ||
|
|
||
| Some usages of volumes and volumeMounts in Tasks is an "implementation detail"; for example, | ||
| Tasks that cache data between Steps by using an emptyDir volume, or a docker build Task | ||
| that shares data between its Steps and Sidecar. These Tasks would now require an emptyDir | ||
| workspace to be supplied in the TaskRun when they would not have previously, requiring the Task | ||
| user to understand more of the Task implementation. | ||
|
|
||
| ## Design Evaluation | ||
|
|
||
| ### Reusability | ||
|
|
||
| Workspaces make Tasks more reusable by allowing storage to be bound to runtime types. Having catalog Tasks | ||
| use Workspaces instead of volumes will make them more reusable. | ||
|
|
||
| ### Simplicity | ||
|
|
||
| Having both Workspaces and volumes in the API provides multiple features targeting the same use cases. | ||
| Using only Workspaces will provide the minimum feature set targeting the use cases we'd like to support. | ||
|
|
||
| ### Flexibility | ||
|
|
||
| This proposal makes the Tekton API slightly less flexible, as most volume types supported by Kubernetes are not | ||
| supported by Workspaces. However, there are not clear CI/CD use cases for these volume types. | ||
|
|
||
| ### Conformance | ||
|
|
||
| This TEP furthers our conformance goals by removing several Kubernetes-specific fields from the Tekton API. | ||
| Tekton implementations should not be required to implement support for CSI ephemeral volumes | ||
| as workspace bindings. | ||
|
|
||
| ### Risks and Mitigations | ||
|
|
||
| Some users may be using volume types not supported by workspaces for CI/CD use cases that we are not aware of, | ||
| and removing `Volumes` from `Task.Spec` may break their workflows. This risk can be mitigated via communication to | ||
| users about the planned deprecation of `Volumes`. | ||
|
|
||
| ## Alternatives | ||
|
|
||
| ### Support hostPath volumes in Workspace Bindings | ||
| This solution augments the proposed solution (removing volume-related fields from the API) | ||
| with new Workspace Bindings to support advanced Docker-in-Docker use cases. | ||
|
|
||
| In this solution, a new field be added to [WorkspaceBinding](https://github.com/tektoncd/pipeline/blob/342ed5237f3bd3273485672426472194a186c02f/pkg/apis/pipeline/v1beta1/workspace_types.go#L54) | ||
| to support hostPath volumes: | ||
|
|
||
| ```go | ||
| import corev1 "k8s.io/api/core/v1" | ||
|
|
||
| type WorkspaceBinding struct { | ||
| // Name is the name of the workspace populated by the volume. | ||
| Name string `json:"name"` | ||
| // SubPath is an optional directory on the volume which should be used for this binding | ||
| SubPath string `json:"subPath,omitempty"` | ||
|
|
||
| // ... existing volume sources such as emptyDir, secret, PVC | ||
|
|
||
| // HostPath represents a pre-existing file or directory on the host | ||
| // machine that is directly exposed to the container. | ||
| HostPath *corev1.HostPathVolumeSource | ||
| } | ||
| ``` | ||
|
|
||
| This approach is not proposed because there are few CI/CD use cases that truly require it. | ||
| In addition, Tasks that require hostPath would likely want to declare this dependency at authoring time rather than at runtime. Lastly, Task authors may not want | ||
| their Tasks to be used with hostPath workspace bindings due to the | ||
| [security risks](https://blog.quarkslab.com/kubernetes-and-hostpath-a-love-hate-relationship.html) | ||
| they pose. | ||
|
|
||
| ### Use TaskRun.Spec.PodTemplate to support missing Workspace features provided by Volumes | ||
|
|
||
| A [proposed alternative](https://github.com/tektoncd/pipeline/issues/2057#issuecomment-587559783) | ||
| to implementing support for extra volume types as workspace bindings is to use the `podTemplate` field | ||
| of `TaskRun.Spec`. However, if `Step.VolumeMounts` are removed, there is no way to mount these volumes | ||
| into the Task's containers using this strategy. In addition, this strategy doesn't allow the Task | ||
| to declare what data it needs, forcing each TaskRun to understand how the Task works, | ||
| and making the Task less reusable. | ||
|
|
||
| ## Resources | ||
| - [Using Docker-in-Docker for your CI or testing environment? Think twice](https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/) | ||
| - [Running KIND Inside A Kubernetes Cluster For Continuous Integration](https://d2iq.com/blog/running-kind-inside-a-kubernetes-cluster-for-continuous-integration) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters