TEP-0130: Pipeline-level service #943

lbernick · 2023-01-26T14:59:17Z

This commit opens a new TEP proposing better support for services
with the same lifespan as a PipelineRun.

/kind tep

afrittoli · 2023-01-30T17:09:15Z

vdemeester

This TEP seems really weird to me (and the title). It is focused on one single tool/use-case over hundreds. "Pipeline-level docker daemon" is just one possible use cases of pipeline level sidecar, I don't really understand why it's the problem statement instead of part of the uses cases of something called "pipeline level sidecars" to be honest.

As of today, if the SA that runs the PipelineRun has enough privileges (aka some access to the cluster it runs in), it's implementable as part of the Pipeline itself : one task that deploy something (a docker daemon, …) and wait for it to be ready, and a finally task that will terminate that deployment.

The only advantage of adding this as part of the API (pipeline-level sidecars) is to not require this level of privilege for tasks running as part of the Pipeline (as this will be the controller creating resources).

vdemeester · 2023-02-06T15:20:23Z

teps/0130-pipeline-level-docker-daemon.md

+### Reusability
+
+A few sidecar configurations, such as docker daemons and database containers, likely make up the majority of use cases.
+A major open question of this proposal is whether we want to introduce a concept of a reusable sidecar.


I am very worried about this concept. If it's "re-usable", it's outside of the lifecycle of a PipelineRun. Which means, for me, that it should be outside of the responsabilities of the tektoncd/pipeline controller as well. If one need a docker daemon to be available accross PipelineRun for a given namespace, it's an infrastructure / namespace management problem, not a tektoncd/pipeline problem to fix.

I think reusable in this case was reuse across tasks in a pipeline and I'd say it should be in scope for CI systems given the number of use cases it supports

I've updated the proposal to clarify that reusing sidecars across pipelines is out of scope.

I think reusable in this case was reuse across tasks in a pipeline and I'd say it should be in scope for CI systems given the number of use cases it supports

Ah, that makes more sense then 😛

teps/0130-pipeline-level-docker-daemon.md

dibyom · 2023-02-06T16:46:02Z

teps/0130-pipeline-level-docker-daemon.md

+
+A few sidecar configurations, such as docker daemons and database containers, likely make up the majority of use cases.
+A major open question of this proposal is whether we want to introduce a concept of a reusable sidecar.
+For example, we could include a docker daemon sidecar in the catalog,


This would be adding another "type" to the catalog which makes me wonder if these sidecars could be special purpose tasks

lbernick · 2023-02-06T17:10:26Z

This TEP seems really weird to me (and the title). It is focused on one single tool/use-case over hundreds. "Pipeline-level docker daemon" is just one possible use cases of pipeline level sidecar, I don't really understand why it's the problem statement instead of part of the uses cases of something called "pipeline level sidecars" to be honest.

Thanks for the feedback! The reason I chose to do it this way is that "more complex docker build pipelines" is the problem I'm hoping to address, while "pipeline level sidecars" is a solution to that problem. Docker build pipelines are the most relevant use case I know of (task-level sidecars are a better solution for integration tests, and the boskos use case seems a lot less important than docker builds IMO). If you know of other use cases or feel strongly about reframing it around pipeline-level sidecars, maybe I could make the TEP more general?

As of today, if the SA that runs the PipelineRun has enough privileges (aka some access to the cluster it runs in), it's implementable as part of the Pipeline itself : one task that deploy something (a docker daemon, …) and wait for it to be ready, and a finally task that will terminate that deployment.

The only advantage of adding this as part of the API (pipeline-level sidecars) is to not require this level of privilege for tasks running as part of the Pipeline (as this will be the controller creating resources).

I think writing a Task to create and delete a deployment and service is non-trivial. I updated the "user experience" section with some details on what this would look like, but the user experience is complex enough that I think the feature is worth adding.

vdemeester · 2023-02-08T08:39:29Z

I think writing a Task to create and delete a deployment and service is non-trivial. I updated the "user experience" section with some details on what this would look like, but the user experience is complex enough that I think the feature is worth adding.

I agree it's non-trivial, but on the other hand, it's easily shareable 👼🏼 and thus doesn't need to be written by all users of tektoncd/pipeline.

lbernick · 2023-02-08T14:37:10Z

I think writing a Task to create and delete a deployment and service is non-trivial. I updated the "user experience" section with some details on what this would look like, but the user experience is complex enough that I think the feature is worth adding.

I agree it's non-trivial, but on the other hand, it's easily shareable 👼🏼 and thus doesn't need to be written by all users of tektoncd/pipeline.

I've added an alternative of "catalog tasks for docker daemons" (which could also be generic catalog tasks for spinning up a deployment + service and tearing them down). As you pointed out, you'd still need to use a service account with the right permissions.

Is this your preferred alternative? Or do you think there's value in the pipeline-level sidecar feature?

dibyom · 2023-02-13T17:02:07Z

Is this your preferred alternative? Or do you think there's value in the pipeline-level sidecar feature?

I think the task based approach is worth considering because its more easily reusable

pritidesai · 2023-02-13T17:08:45Z

API WG - in review cycle
/assign @vdemeester

vdemeester

In my opinion this is still too specific 🙃 or too generic, or both at the same time. What is implied by "build cache" ? This is still very docker task specific even. Handling "(dockerfile) build cache" when using docker is way way way different that doing this with kaniko or buildah (or whatever other tool). "Pipeline level build cache" is, for me, a use case, not a TEP proposal 😅.

I deeply think we should focus more on writing smart, re-usable Task that fulfil those use-case today (as we already have v1) before trying to solve them by tektoncd/pipeline features. In my opinion, we need to exercise and push our current API (v1) to its limit before trying to add too much to it.

teps/0130-pipeline-level-build-cache.md

vdemeester · 2023-02-21T14:26:41Z

teps/0130-pipeline-level-build-cache.md

+    - name: docker-tls-certs   
+```
+
+Here's what the [docker-build catalog task](https://github.com/tektoncd/catalog/blob/81bf7dc5610d5fa17281940a72a6377604105cea/task/docker-build/0.1/docker-build.yaml)


Note: this is what a docker-build Task could look like today as well. What is missing to make this "smarter" today, is a way to conditionnally start a sidecar (for example, in case of params.docker_host is empty or default value) and same for other possible requirements to run the sidecar (like securityContext.privileged, …)

teps/0130-pipeline-level-build-cache.md

vdemeester · 2023-02-21T14:30:23Z

teps/0130-pipeline-level-build-cache.md

+### Improve our documentation
+
+Instead of building new features, we could improve our documentation to help users better understand how to build a docker daemon
+into their PipelineRun. See [user experience](#user-experience) for an example of what this would look like.
+
+### Catalog Tasks for docker daemon
+
+We could add Tasks to the verified catalog (or to a community catalog) for spinning up and tearing down a docker daemon (or more generically, a deployment with a service).
+This would prevent users from having to write their own daemon creation/teardown tasks, but they would still have to provide their kube credentials
+and use a service account with rolebindings that allow it to create services and deployments.


Yes and yes. For me those are the valid, most important and "today doable" action to take. From these, their usage and feedback from it, we may want to enhance v1 or define v2 better.

dibyom · 2023-02-21T17:00:14Z

I still think a Pipeline level sidecar is the right way to go here, but catalog tasks could also work.

Yeah, I'm still leaning towards a task - a challenge I see is that a user might need to specify a "cleanup-docker-daemon" task in addition to a "setup-docker-daemon" task - maybe we could do figure out a way to add some magic to simplify that. Or we could just write a custom task for this (though we don't really have a catalog of custom tasks today)

pritidesai · 2023-02-21T18:26:14Z

teps/0130-pipeline-level-build-cache.md

+
+The docker-build catalog Task only allows building a single image and pushing it to a registry.
+The following use cases are not addressed:
+- Building multiple images concurrently with the same daemon to save resources and share cached base layers


What use case is driving this proposal? What is the proposed storage for such images?

The use cases are listed under the motivation section:

User wants to test an image after building, and only push images that pass tests to the artifact repo

User wants to save resources by using the same docker daemon/cached layers for multiple images built in the same pipelinerun (e.g. via matrix)

Image layers would be cached by the docker daemon (as it does currently). The user is still responsible for pushing the built image to a repository.

thanks @lbernick yup I understand the two bullet points listed in the TEP as well. My question was more aligning with @vdemeester's feedback/comment, should we propose a feature in the pipeline controller for one single technology - docker daemon?

I'm hoping that the reframing of this TEP as "pipeline level service" makes it generic enough. Yes, docker daemon is just one tool, but docker builds are a very common CI/CD use case and I think it's important to consider features that will improve the UX for docker builds. I also listed a few more potential uses for a Pipeline level service, although I think they're less compelling.

lbernick · 2023-02-21T19:26:08Z

Discussion from Pipelines WG today: experiment and get feedback on the "catalog task" approach before proposing building a new feature.

@dibyom @vdemeester I've rewritten this proposal to list out the alternatives rather than proposing one. I'd still like to merge this TEP rather than closing it because it has a bunch of ideas and syntax examples written out for how we can address this use case that I think are helpful for reference. We can always update the TEP with the results of our experimentation, and mark it as obsolete/rejected if we find that the catalog task approach works.

Chatting offline with @vdemeester about a better name for this TEP.

lbernick · 2023-02-22T15:30:00Z

@vdemeester and I agreed on "pipeline level service" as a title for this TEP-- it is not docker specific and still makes sense if we choose to use catalog tasks as our solution.

teps/0130-pipeline-level-build-cache.md