| title | authors | creation-date | last-updated | status | superseded-by | |||
|---|---|---|---|---|---|---|---|---|
0023-Implicit-parameter-mapping |
|
2020-10-01 |
2021-12-15 |
implemented |
|
Pipeline specs can sometimes feel unnecessarily verbose, particularly when the same fields need to be passed around different Tasks again and again.
In this TEP, we propose introducing implicit parameter mapping to allow authors to reference params in embedded specs that are not explicitly defined. Instead, users will rely on Tekton to automatically resolve these values at admission time.
We want to make it easier for users to get started with Tekton by reducing some
of the complexity in writing specs. One way we wish to do this is to reduce the
amount of repetition in Pipeline specs, inferring information as much as
possible. Our hope is that this will make it easier to get started with simple
PipelineRun definitions to let users focus on the core functionality of their
Pipelines and not overwhelm users with config verbosity. A analogy here is
imitating how many people get started writing code - first with a single
main() function, then breaking out to different functions/packages later as
needed to make things reusable.
Params stick out as a particularly good target here, since they need to be passed through the PipelineRun -> Pipeline -> Tasks (often completely unmodified), and adding additional params does not inherently modify Task behavior if they happen to be unused.
- Simplify authored Pipeline YAML by allowing users to omit parameters from
embedded resources.
- In particular, simplify the getting started experience by reducing the complexity of authored configs.
- Support implicit mapping for PipelineResources, Workspaces, Task/PipelineRefs.
- Resource authors can optionally omit parameters that can be inferred from a parent scope.
- Resources should be fully resolved before they are stored in etcd.
- Clients should receive back resolved responses from API requests to avoid ambiguity.
We propose allowing for implicit parameters in a resource spec, by passing through parameters from parents. For example:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-with-taskspec-to-echo-message
spec:
pipelineSpec:
params:
- name: MESSAGE
type: string
tasks:
- name: echo-message
taskSpec:
params:
- name: MESSAGE
type: string
steps:
- name: echo
image: ubuntu
script: |
#!/usr/bin/env bash
echo "$(params.MESSAGE)"
params:
- name: MESSAGE
value: $(params.MESSAGE)
params:
- name: MESSAGE
value: "Good Morning!"We want users to be able to optionally shorten this config by removing param definitions in embedded specs:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-with-taskspec-to-echo-message
spec:
pipelineSpec:
tasks:
- name: echo-message
taskSpec:
steps:
- name: echo
image: ubuntu
script: |
#!/usr/bin/env bash
echo "$(params.MESSAGE)"
params:
- name: MESSAGE
value: "Good Morning!"With the shortened syntax, we can derive the same information we need from the top-most parameter definition -
- We have a param named
MESSAGE - The param value is of type
string - No default value is required since we already have a value.
Since these are all embedded specs within the same resource, our expectation is that this shortened syntax will not take away from the context knowledge of parameters and how they should be used. A good analogy here is variable scoping found in most programming languages - e.g.
x := "Hi!"
func() {
// x is still reachable, even though it's not explicitly defined parameter.
fmt.Println(x)
}()This logic should live within the Pipelines mutating admission controller.
Although the resource author is using a shortcut, we should still store and validate a fully resolved spec. This spec should look very similar to the explicit config, using the information we derived:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-with-taskspec-to-echo-message
spec:
pipelineSpec:
params:
- name: MESSAGE
type: string
tasks:
- name: echo-message
taskSpec:
params:
- name: MESSAGE
type: string
steps:
- name: echo
image: ubuntu
script: |
#!/usr/bin/env bash
echo "$(params.MESSAGE)"
params:
- name: MESSAGE
value: $(params.MESSAGE)
params:
- name: MESSAGE
value: "Good Morning!"Extra params are safe to include in Tasks since they don't modify behavior unless they are actually used - because of this there is little risk in passing through all parent param values to embedded specs.
To avoid issues of additional webhook latency or reliability, we will not support remote TaskRefs or PipelineRefs.
In cases of parameter naming conflicts, the innermost definition should win. e.g. for the following config:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-with-taskspec-to-echo-message
spec:
pipelineSpec:
tasks:
- name: echo-message
taskSpec:
params:
- name: MESSAGE
type: string
steps:
- name: echo
image: ubuntu
script: |
#!/usr/bin/env bash
echo "$(params.MESSAGE)"
params:
- name: MESSAGE
value: ["Good Morning!"]This would resolve to something like:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-with-taskspec-to-echo-message
spec:
pipelineSpec:
params:
- name: MESSAGE
type: array
tasks:
- name: echo-message
taskSpec:
params:
- name: MESSAGE
type: string
steps:
- name: echo
image: ubuntu
script: |
#!/usr/bin/env bash
echo "$(params.MESSAGE)"
params:
- name: MESSAGE
value: $(params.MESSAGE[*])
params:
- name: MESSAGE
value: ["Good Morning!"]The innermost Task definition should take precedence and make the config invalid, since the PipelineRun array param should not be able to override the param definition of the task.
Extra parameters may be passed down to embedded spec definitions, even if they are not actually used.
e.g.
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-with-taskspec-to-echo-message
spec:
pipelineSpec:
params:
- name: MESSAGE
type: string
tasks:
- name: echo-message
taskSpec:
steps:
- name: echo
image: ubuntu
script: |
#!/usr/bin/env bash
echo "$(params.MESSAGE)"
params:
- name: MESSAGE
value: "Good Morning!"
- name: UNUSED
value: "unused message"Would be resolved to:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-with-taskspec-to-echo-message
spec:
pipelineSpec:
params:
- name: MESSAGE
type: string
- name: UNUSED
type: string
tasks:
- name: echo-message
taskSpec:
params:
- name: MESSAGE
type: string
- name: UNUSED
type: string
steps:
- name: echo
image: ubuntu
script: |
#!/usr/bin/env bash
echo "$(params.MESSAGE)"
params:
- name: MESSAGE
value: $(params.MESSAGE)
- name: UNUSED
type: $(params.UNUSED)
params:
- name: MESSAGE
value: "Good Morning!"
- name: UNUSED
value: "unused message"Although it is never used, the UNUSED param will be plumbed through to the
underlying embedded Task. Since the Task steps do not actually use the param,
this is unlikely to affect any behavior in execution (unless the Task is doing
some sort of introspection).
Certain parameter names may be relatively common and therefore carry a different
meaning between different Tasks (e.g. url, path, etc). Tasks with the
same parameter names that require different input could therefore inadvertently
acquire the wrong value.
We consider it the Pipeline author's responsibility to guard against this.
Our expectation here is that if authors are choosing to embed specs, then the context of the variables should be known to avoid problems like this. If they require a separate/different mapping then it would be up to the author to explicitly assign these params, either in line or in its own Task definition.
e.g.
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-with-taskspec-to-echo-message
spec:
pipelineSpec:
# Spec params are implicit
tasks:
- name: echo-message
taskSpec:
# Spec params are implicit
steps:
- name: echo
image: ubuntu
script: |
#!/usr/bin/env bash
echo "$(params.OTHERMESSAGE)"
# Rename MESSAGE -> OTHERMESSAGE
params:
- name: OTHERMESSAGE
value: $(params.MESSAGE)
params:
- name: MESSAGE
value: "Good Morning!"Our goal with this change is to make it easier to write PipelineRuns. We
particularly see value in getting started use cases where users may want to
start with a single-file PipelineRun with embedded specs before refactoring into
different Pipelines/Tasks for reusability later (similar to how one might write
a binary with a single main before refactoring to different funcs/packages).
We expect this to add negligible overhead to Tekton's validating/mutating webhook. This will only be supported for embedded specs, so there will be no remote calls to look up additional resources.
The Tekton Pipeline validating/mutating webhook will resolve the parameter in
embedded fields contained in Pipeline and PipelineRun. This transforms the
implicitly specified Pipeline specification into an explicitly defined
Pipeline specification. After the transformation, the webhook performs the
regular Pipeline specification validation checks.
To implement this, we propose adding additional data into the
SetDefaults context
to keep track of seen PipelineRun/TaskRun parameters. If a Param is defined at
the current spec level we will use that first, else we use the context Params to
propagate implicit values down the spec stack (e.g. PipelineRun -> Pipeline ->
Task) for additional resolution. Because we are introducing additional params
into the resolved spec, we will need to remove
restrictions around unused params.
Since this is strictly an additive change, any application that depends on the
Pipeline specification stored in a cluster does not need to be updated. We do
not anticipate needing to block this behind a feature gate.
This proposal does not do much to address some of the pain for the original motivating issues (tektoncd/pipeline#3050, tektoncd/pipeline#1484), where TaskRef parameters / resources were equally hard to work with.
We suspect that this will require additional work to resolve, particularly because of the remote resources that complicate validation. We are considering this out of scope for now, focusing on what we think will be a small but meaningful improvement. (We suspect we will want to look at these as a follow up though).
A drawback of this approach is making things easier for users makes things more complex for platforms, particularly when it comes to conformance. e.g. this change adds more rules for how param fields need to be processed, which would need to be replicated across every Tekton conformant implementation if they are not leveraging the base library. We could take a stance that the fields for Pipeline/TaskRun primatives should remain explicit in order to simplify logic for platform implementations, and instead rely on higher-level types built on top of Pipelines/Tasks to handle this kind of user logic.
As of today we don't yet have this kind of higher-level type in vanilla Tekton, but even if/when we did there would likely still be value in adding features that simplify the user authoring process for Pipeline/TaskRuns if they continue to be types that are directly used.
- Unit tests for checking that the webhook correctly transforms an implicit specification into an explicit resolved specs.
A consequence of this design is that all parameters starting from the PipelineRun will propagate down to every embedded Task, regardless if the Task asked for these values or not. While this is okay from an execution standpoint (i.e. if extra params are passed in, they're never actually used and shouldn't affect execution), this might create noise when looking at Run history.
For now we will consider this out of scope - if this really bothered someone they could explicitly define a Task definition to prune out unneeded params. e.g. rewriting the example from earlier:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: echo
spec:
params:
- name: MESSAGE
type: string
steps:
- name: echo
image: ubuntu
script: |
#!/usr/bin/env bash
echo "$(params.MESSAGE)"
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: pipelinerun-with-taskspec-to-echo-message
spec:
pipelineSpec:
tasks:
- name: echo-message
taskRef:
name: echo
params:
- name: MESSAGE
value: $(params.MESSAGE)
params:
- name: MESSAGE
value: "Good Morning!"
- name: UNUSED
value: "unused message"This would ensure that only the MESSAGE param is passed to the echo Task.
We could look into pruning the child params based on which are actually used, but this is complexity we are not interested in adding at the moment.
- Can we do better with TaskRefs/PipelineRefs?
- Can we extend this behavior to Triggers as well for inlined TriggerTemplates?