Merge pull request #419 from otaviof/ep-remote-artifacts

[EP] Remote Artifacts

docs/proposals/remote-artifacts.md

@@ -0,0 +1,319 @@
+<!--
+Copyright The Shipwright Contributors
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+---
+title: remote-artifacts
+authors:
+  - @otaviof
+
+reviewers:
+  - @SaschaSchwarze0
+  - @qu1queee
+  - @adambkaplan
+  - @zhangtbj
+  - @coreydaley
+
+approvers:
+  - TBD
+creation-date: 2020-10-02
+last-updated:  2020-11-09
+status: provisional
+---
+
+Shipwright Remote Artifacts
+---------------------------
+
+# Summary
+
+Remote artifacts are dependencies of the software building process, they represent binaries or
+other data stored outside of the version control system (Git). Hence, they are required when
+dealing with container images as well, and this enhancement proposal focuses on adding a remote
+artifacts support to Shipwright's Build API.
+
+# Motivation
+
+Give Shipwright's operator broader build use-case support by enhancing its capabilities to include
+the concept of Artifacts. In other words, remote entities that will be available for the image
+build process.
+
+End users will be able to include remote artifacts as a build runtime dependency, artifacts
+controlled by remote systems will be downloaded before the build process starts. Those artifacts may
+be a pre-compile binary, `jar` files, `war` files, etc.
+
+## Goals
+
+*   Provide means to declare remote artifacts, dependencies that can be employed on builds;
+*   Create the mechanism to download and prepare remote artifacts for builds;
+
+## Non-Goals
+
+*   Automate the upload of local artifacts into the cluster;
+*   Manage remote artifacts;
+*   Amend container images created by the given `BuildStrategy`;
+*   Walking a remote directory tree, similarly to a `git checkout`;
+
+# Proposal
+
+The enhancement proposal is centered around the idea of declaring external dependencies, here
+called "artifacts", and being able to use them during the container image building process.
+
+## Build Artifacts
+
+The remote artifacts will be directly expressed in the body of a `Build` resource, as the
+following example:
+
+```yml
+---
+apiVersion: build.dev/v1alpha1
+kind: Build
+metadata:
+  name: license-file
+spec:
+  source:
+    # ...
+  sources:
+    - name: license-file
+      type: http
+      url: https://licenses.company.com/customer-id/license.tar
+      http:
+        path: $(workspace)/licenses/license.tar
+```
+
+We will add `sources` section, side by side with current `source`. The idea is to accommodate both
+constructions in `v1alpha1`, and save API breaking changes for upcoming `v1beta1`.
+
+The new `sources` will contain the following attributes:
+
+- `name`: source name (required);
+- `type`: input source type, `git` or `http` (required);
+- `sourceRef`: use a external resource, the name in this field is the resource name (optional);
+- `url`: the URL of the repository or remote artifact (optional);
+- `credentials`: the credentials to interact with the artifact (optional);
+- `http`: settings for `http`, containing inner attribute `.spec.http.path` (optional);
+- `git`: settings for `git`, as exists today;
+
+The resource `UID`/`GID` will be defined by the same user running the Tekton task, later on, we
+can extend the API to support arbitrary configuration.
+
+### Single vs. Multiple Git Repositories
+
+For the initial implementation, we will only support one git repository that is specified in
+`.spec.source`. Remote artifacts will be specified n `.spec.sources` only.
+
+We will address supporting multipe Git repositories in a future enhancement proposal. This will
+involve re-defining `/workspace/source` location and the `$workspace` placeholder.
+
+## Steps and Helper
+
+Artifacts are requirements for the build process, thus downloading and preparing artifacts must
+happen before the build process starts. To achieve the objective, the operator will generate a new
+task step to download the artifacts.
+
+The download may happen using existing open-source software, more specifically `wget`. The
+container image which contains this software will be specified via environment variable, and when
+not informed will use a default value.
+
+Therefore, the `buildrun` controller needs to generate a `TaskRun` step, having a `wget` commands
+plus arguments, to download the specified artifacts.
+
+The step generated by the operator will look like the following:
+
+```yml
+---
+apiVersion: tekton.dev/v1beta1
+kind: TaskRun
+metadata:
+  name: artifact-download
+spec:
+  steps:
+    - name: artifact-download
+      image: busybox:latest
+      command:
+        - /bin/bash
+      args:
+        - -c
+        - >
+          wget --output="/workspace/source/classpath/main.jar" https://nexus.company.com/app/main.jar && \
+          chown 1000:1001 /workspace/source/classpath/main.jar && \
+          chmod 0644 /workspace/source/classpath/main.jar
+```
+
+## Example Use-Case
+
+Using [Shipwright's proposed logos](https://github.com/shipwright-io/build/issues/325) as example,
+let's assume we are building a [TypeScript application](https://github.com/otaviof/typescript-ex)
+which will use the project logo, and we would like to create two different builds, one with the
+default project logo, and another with the alternative.
+
+By using remote artifacts, we can keep the separation of project source code and assets, and we
+can describe those resources as:
+
+```yml
+---
+apiVersion: build.dev/v1alpha1
+kind: BuildSource
+metadata:
+  name: ship-logo
+spec:
+  sources:
+    - name: project-logo
+      type: http
+      url: https://user-images.githubusercontent.com/2587818/92114986-69bfb600-edfa-11ea-820e-96cdb1014f58.png
+      http:
+        path: $(workspace)/assets/images/shipwright-logo.png
+```
+
+And, the alternative logo:
+
+```yml
+---
+apiVersion: build.dev/v1alpha1
+kind: BuildSource
+metadata:
+  name: axes-logo
+spec:
+  sources:
+    - name: project-logo
+      type: http
+      url: https://user-images.githubusercontent.com/2587818/92100668-c1ebbd80-ede4-11ea-9e8a-7379c3875ea0.png
+      http:
+        path: $(workspace)/assets/images/shipwright-logo.png
+```
+
+Then, we can create the `Build` resource, as per:
+
+```yml
+---
+apiVersion: build.dev/v1alpha1
+kind: Build
+metadata:
+  name: typescript-ex
+spec:
+  strategy:
+    name: buildpacks-v3
+    kind: ClusterBuildStrategy
+  source:
+    url: https://github.com/example/project.git
+  sources:
+    - name: source
+      type: git
+      url: https://github.com/otaviof/typescript-ex.git
+    - name: project-logo
+      sourceRef: ship-logo
+  output:
+    image: quay.io/otaviof/typescript-ex:latest
+```
+
+Now, we can create two `BuildRun` resources. The first only runs the build with original settings:
+
+
+```yml
+---
+apiVersion: build.dev/v1alpha1
+kind: BuildRun
+metadata:
+  name: typescript-ex
+spec:
+  buildRef:
+    name: typescript-ex
+```
+
+And later, we can create yet another `BuildRun`, but this time use the alternative logo. Here we are
+overwriting the `project-logo` source name, with an alternative resource, i.e.:
+
+```yml
+---
+apiVersion: build.dev/v1alpha1
+kind: BuildRun
+metadata:
+  name: typescript-ex-alternative-logo
+spec:
+  buildRef:
+    name: typescript-ex
+  output:
+    image: quay.io/otaviof/typescript-ex:alternative
+  sources:
+    - name: project-logo
+      sourceRef: axes-logo
+```
+
+When the build processes are done, the following images will be available:
+*   `quay.io/otaviof/typescript-ex:latest`
+*   `quay.io/otaviof/typescript-ex:alternative`
+
+A number of real world use-cases can be derived from this example, the `BuildSource` is the
+foundation.
+
+## Test Plan
+
+1. Deploy the Shipwright Build operator in a cluster;
+2. Create a `BuildSource` resource instance, point to a remote binary;
+3. Create `Build` and `BuildRun` resources, using `BuildSource`;
+4. Make sure the build process happens successfully, being able to use remote artifact;
+
+## Alternatives
+
+### Standalone CRD
+
+Alternatively, we may define the artifacts as a standalone CRD, that is a `BuildSource` resource.
+The advantage of this design is being able to exchange, and reuse, artifacts on several builds.
+For instance, if two projects are sharing a common logo image, both `Builds` will refer to the
+same `BuildSource`.
+
+Splitting up resources will imply on modifying `Build` and `BuildRun` resources, and include
+`.spec.sourceRef`. The new attribute will make possible linking the build with a `BuildSource`
+resource.
+
+The following snippet shows how an Artifact, i.e.: `BuildSource`, will be represented.
+
+```yml
+---
+apiVersion: build.dev/v1alpha1
+kind: BuildSource
+metadata:
+  name: license-file
+spec:
+  sources:
+    - name: license.tar
+      type: http
+      url: https://licenses.company.com/customer-id/license.tar
+      http:
+        path: $(workspace)/licenses/license.tar
+```
+
+#### Usage
+
+The usage of this feature is based on declaring a slice of `.spec.sources` and later on overwriting
+entries using `BuildRun` resource. For instance:
+
+```yml
+---
+apiVersion: build.dev/v1alpha1
+kind: Build
+metadata:
+  name: example
+spec:
+  source:
+    # ...
+  sources:
+    - name: license-file
+```
+
+Could have its `license-file` overwritten in a `BuildRun` with:
+
+```yml
+---
+apiVersion: build.dev/v1alpha1
+kind: BuildRun
+metadata:
+  name: license-file
+spec:
+  buildRef:
+    name: example
+  sources:
+    - name: license-file
+      sourceRef: alternative-file
+```